Comparing Information Across Modalities

Once we have individual data types like text, images, or audio transformed into numerical representations and properly aligned (as discussed in "Aligning Data from Multiple Sources"), a significant question arises: How can we tell if the information coming from these different sources is related? For instance, how do we determine if a given piece of text accurately describes an image, or if the sound in a video matches the visual scene? This section introduces the basic ideas behind comparing information across different modalities.

The ability to compare information is fundamental to many multimodal AI tasks. Imagine searching your photo library using a text query like "sunset over mountains." The system needs to compare your text query (one modality) with the content of your images (another modality) to find the best matches. Or, consider a system that verifies if a person speaking in a video is the same person whose voice is heard. This also requires comparing information from visual and audio streams.

What Does It Mean to Compare?

At its core, comparing information across modalities means assessing how similar or different their content is. If an image shows a cat playing with a yarn ball, and a text description says "A feline engages with a woolen toy," we'd intuitively say these two pieces of information are very similar. If the text said "A dog barks at a car," it would be very different. AI systems aim to quantify this similarity or dissimilarity.

To do this, they work with the numerical representations we learned about earlier (e.g., vectors for text, feature sets for images). If two pieces of information from different modalities are semantically related, their numerical representations should also reflect this relationship in some way. This often means their representations might be "close" to each other in a mathematical sense or share certain predictable patterns.

Measuring Similarity

To make comparisons concrete, AI systems often compute a similarity score. This is typically a number that indicates how related two pieces of information are. A higher score might mean more similar, and a lower score less similar (or vice versa, depending on the specific measure). There are several ways to approach this.

Distance in Feature Space

One intuitive way to think about similarity is through distance. If we can represent both an image and a piece of text as points in some high-dimensional space (even if these spaces are initially different), we could, in principle, measure the "distance" between them. If the points are close, the items are considered more similar. For example, if $V_I$ is a vector representing an image and $V_T$ is a vector representing a text caption, a simple distance like the Euclidean distance could be used if they are in the same space and have the same dimensions. However, data from different modalities often live in very different types of numerical spaces initially.

Cosine Similarity: Comparing Directions

A very common and effective measure, especially when dealing with high-dimensional data like text embeddings or image features, is cosine similarity. Instead of just looking at the distance between two points (vectors), cosine similarity measures the cosine of the angle between them.

Imagine two vectors as arrows starting from the same point.

• If the arrows point in almost the same direction, the angle between them is small, and the cosine of that angle is close to 1 (highly similar).
• If the arrows are perpendicular (pointing in unrelated directions), the angle is 90 degrees, and the cosine is 0 (not similar).
• If they point in opposite directions, the angle is 180 degrees, and the cosine is -1 (very dissimilar, though this interpretation can vary).

The formula for cosine similarity between two vectors $A$ and $B$ is: $\text{similarity} = \cos(\theta) = \frac{A \cdot B}{\|A\| \|B\|}$ Here, $A \cdot B$ is the dot product of the vectors, and $\|A\|$ and $\|B\|$ are their magnitudes (or lengths). The beauty of cosine similarity is that it's less sensitive to the magnitude of the vectors and more focused on their orientation or the "direction" of their content. This is often useful because the length of a feature vector might not be as informative as the pattern of its values.

For instance, a short sentence and a long descriptive paragraph about a "dog" might have feature vectors of different magnitudes, but their cosine similarity could still be high if they point in a similar direction in the feature space, indicating they both relate to "dog-ness."

The Idea of a Shared Representation Space

Comparing information becomes much more direct if we can project data from different modalities into a shared representation space (sometimes called a common embedding space). In such a space, items that are semantically similar are positioned close to each other, regardless of their original modality.

For example, an image of a bicycle, the word "bicycle," and even the sound of a bicycle bell might all be mapped to nearby points in this shared space. Once data is in this common format, comparing them can be as straightforward as calculating distances or cosine similarities between their points in this shared space.

In this diagram, items related to "dog" from both text and image modalities are mapped to nearby points in a shared space. Similarly, items related to "cat" form another cluster. This makes it easier to see that the text 'dog playing' is more similar to the image of a dog playing than to the image of a cat sleeping.

Achieving such a shared space is a significant goal of many multimodal learning techniques, which we'll touch upon later in the course.

Simple Examples of Cross-Modal Comparison

Let's consider a couple of straightforward scenarios:

1. Image-Text Matching:

   • Image: A photograph of a sunny beach with palm trees.
   • Text Caption A: "Palm trees on a sunny beach."
   • Text Caption B: "A snowy mountain peak." The system would convert the image into a numerical feature vector. It would also convert Caption A and Caption B into their respective text vectors. Then, it would calculate the similarity (e.g., cosine similarity) between the image vector and Caption A's vector, and between the image vector and Caption B's vector. We would expect a higher similarity score for Caption A.

2. Audio-Visual Synchrony:

   • Video Clip: A person speaking, with clear lip movements.
   • Audio Track: The sound of their voice. A system might extract features from the lip movements (visual modality) and features from the audio waveform (audio modality). By comparing these feature streams over time, the system can determine if they are synchronized and correspond to each other, for example, by checking if the timing of lip closures matches plosive sounds (like 'p' or 'b') in the audio.

Challenges in Comparing Modalities

While the idea of comparing information seems intuitive, it comes with its own set of challenges:

• Heterogeneity of Data: Text, images, and audio are fundamentally different. Text is symbolic and sequential, images are spatial arrays of pixels, and audio is a temporal waveform. Finding common ground or a meaningful way to compare their raw representations can be difficult. This is why transforming them into suitable feature vectors or embeddings is a necessary first step.
• The Semantic Gap: Low-level features (like pixel values in an image or raw audio frequencies) don't directly correspond to high-level semantic meaning (like "this is a picture of a happy dog" or "this person sounds angry"). Bridging this gap, so that comparisons reflect true semantic similarity, is a major area of research.
• Granularity: What level of detail should the comparison target? Are we comparing an entire image to an entire paragraph, or specific objects in an image to specific phrases in a text?
• Context: The meaning of information can be highly context-dependent. A word or an image snippet can mean different things in different situations, making direct comparison tricky without considering the broader context.

Understanding how to represent, align, and then compare information from different modalities lays a critical foundation. These comparisons are not just an end goal; they are often a stepping stone towards more complex tasks where information from multiple sources is integrated to make a decision or generate new content, as we will see in later chapters.