Inputs and Outputs in Multimodal Applications

We've previously looked at several illustrative applications of multimodal AI. Now, let's examine in more detail the kinds of data these systems process as input and what they generate as output. Every AI system, multimodal or not, takes in some form of data (input) to produce a result (output). For multimodal systems, the significant aspect is often the variety of data types involved on either the input side, the output side, or both.

You can think of this process like a sophisticated kitchen. The ingredients (inputs) can be varied, perhaps a visual scene (an image), a spoken query (audio), and a written instruction (text). The final dish (output) results from how these diverse ingredients are combined and transformed by the system, perhaps into a spoken answer accompanied by a generated image.

Let's break down the inputs and outputs for the applications we discussed earlier in this chapter.

1. Image Captioning Systems

These systems are designed to automatically generate a textual description for a given image. They look at an image and tell you what's in it, in words.

• Input: The primary input is an image. This is data in the visual modality.
• Output: The output is text, specifically a sentence or phrase that describes the content or context of the input image. This is data in the text modality.

An image (visual data) is fed into the system, and a textual caption is produced.

2. Visual Question Answering (VQA)

VQA systems go a step further than captioning. They answer natural language questions about an image. This means the system must understand both the visual content of the image and the semantics of the text-based question.

• Inputs: This type of system takes two kinds of input:
  1. An image (visual modality).
  2. A text question (text modality) related to the image, for example, "What color is the car?" or "How many people are in this picture?".
• Output: The output is text, which is the answer to the question based on the provided image.

The VQA system processes both an image and a text question to generate a text-based answer.

3. Text-to-Image Synthesis

This application is, in a way, the reverse of image captioning. You provide the system with a textual description, and it attempts to generate a new image that matches that description.

• Input: The primary input is text (text modality), a sentence or a collection of keywords describing the desired visual scene, for example, "A red apple on a wooden table" or "Impressionist painting of a cat wearing a hat."
• Output: The output is an image (visual modality), newly created by the AI system based on the input text.

4. Speech Recognition Enhanced by Visual Cues

Standard automatic speech recognition (ASR) systems convert spoken audio into written text. However, human speech understanding often benefits from visual cues, like lip movements, especially in noisy environments. Multimodal speech recognition incorporates this.

• Inputs:
  1. Audio data (audio modality), which is the speech itself.
  2. Video data (visual modality), typically focusing on the speaker's face, particularly lip movements.
• Output: The output is text (text modality), representing the transcribed speech. The system aims to improve transcription accuracy by using information from both audio and visual streams.

5. Multimodal Sentiment Analysis

Sentiment analysis aims to determine the emotional tone or opinion expressed in a piece of content. Multimodal sentiment analysis enhances this by considering information from various sources.

• Inputs: These can be varied and combined, depending on the application:
  • Text data (e.g., a product review, a comment on social media).
  • Audio data (e.g., the tone of voice, pitch, and speaking rate in a spoken review or customer service call).
  • Video data (e.g., facial expressions, gestures in a video blog or recorded interview).
• Output: The output is typically a classification label (like "positive," "negative," "neutral," or more granular emotions like "happy," "sad," "angry") or a numerical score representing the intensity or polarity of the sentiment. This output is often categorical or numerical.

Summarizing Inputs and Outputs

To make this clearer, let's look at a table summarizing the input and output modalities for the applications we've discussed:

Application	Primary Input(s)	Modality of Input(s)	Primary Output(s)	Modality of Output(s)
Image Captioning	An image	Visual	A textual caption	Text
Visual Question Answering (VQA)	An image, A text question	Visual, Text	A textual answer	Text
Text-to-Image Synthesis	A textual description or prompt	Text	An image	Visual
Enhanced Speech Recognition	Audio (speech), Video (lip movements)	Audio, Visual	Transcribed text	Text
Multimodal Sentiment Analysis	Text, Audio, and/or Video content	Text, Audio, Visual	A sentiment label or score	Categorical/Numerical

This table highlights the flow of data types for common introductory multimodal applications. Notice how different combinations of modalities are used for inputs and outputs depending on the task.

Understanding these input and output patterns is a fundamental step in working with multimodal AI. When you encounter a new multimodal AI application, one of the first questions to consider is: "What kind of data does it take in, and what kind of data does it produce?" Answering this helps clarify the system's purpose, its operational flow, and the challenges involved in its design. As you continue learning, you'll see these I/O patterns appear in various forms, often in more complex and combined ways.