Relation Networks for Few-Shot Learning

While Prototypical Networks compute class prototypes and use a fixed distance metric in the embedding space, Relation Networks (RNs) introduce a different approach. Proposed by Sung et al. (2018), RNs learn a deep non-linear metric function directly, aiming to determine the similarity or "relation" between query examples and support examples. Instead of assuming a simple metric like Euclidean distance is sufficient, RNs posit that a dedicated neural network can better capture the complex relationships needed for few-shot classification, especially when dealing with intricate visual or semantic features.

Architecture of Relation Networks

The core idea is to train a network that explicitly outputs a scalar similarity score between pairs of feature representations. A typical Relation Network consists of two main components:

1. Embedding Module ($f_\phi$): This module, often a convolutional neural network (CNN) for vision tasks or a transformer encoder for sequence data, maps raw inputs (both support set examples $x_s$ and query examples $x_q$) into feature embeddings.

   $$\text{embedding}_s = f_\phi(x_s)$$ $$\text{embedding}_q = f_\phi(x_q)$$

   When adapting foundation models, $f_\phi$ might be the pre-trained model itself (potentially frozen) or a part of it.

2. Relation Module ($g_\psi$): This module takes pairs of embeddings (typically a query embedding combined with support embeddings) and outputs a scalar relation score between 0 and 1, indicating the degree of similarity relevant to the classification task. A common practice for an $N$-way, $K$-shot task is to first aggregate the support embeddings for each class $c_i$ (e.g., by summing or averaging) to get a class representation $s_i$:

   $$s_i = \frac{1}{K} \sum_{k=1}^{K} f_\phi(x_{s}^{(i, k)})$$

   Then, the relation module processes the concatenation of the query embedding $q = f_\phi(x_q)$ and each class representation $s_i$:

   $$r_i = g_\psi(\text{Combine}(s_i, q))$$

   The $\text{Combine}$ function is often simple concatenation. The relation module $g_\psi$ itself is usually a smaller neural network, like a few convolutional layers followed by fully connected layers, or just a multi-layer perceptron (MLP). It's designed to learn a task-specific similarity function.

Flow of information in a Relation Network for one class $c_i$. Support and query images are embedded, support embeddings are aggregated, combined with the query embedding, and fed into the relation module to produce a similarity score. This process is repeated for all classes in the support set.

Training Relation Networks

RNs are trained episodically, similar to other meta-learning methods. In each episode, a task (e.g., a C-way, K-shot classification problem) is sampled. The network processes the support set and query examples for that task. The objective function typically aims to push the relation score $r_i$ towards 1 if the query example $x_q$ belongs to class $c_i$, and towards 0 otherwise. A common loss function is Mean Squared Error (MSE):

$$\mathcal{L} = \sum_{\text{task } \mathcal{T}} \sum_{(x_q, y_q) \in \text{Queries}(\mathcal{T})} \sum_{i=1}^{C} (r_i - \mathbf{1}_{y_q = c_i})^2$$

Here, $r_i$ is the predicted relation score for class $c_i$ given the query $x_q$, and $\mathbf{1}_{y_q = c_i}$ is an indicator function that is 1 if the true label $y_q$ matches class $c_i$, and 0 otherwise. This loss is backpropagated through both the relation module $g_\psi$ and the embedding module $f_\phi$, allowing both the feature representation and the similarity function to be learned jointly.

Comparison to Prototypical Networks

Relation Networks differ significantly from Prototypical Networks:

• Learned Metric: RNs explicitly learn the similarity function ($g_\psi$), whereas Prototypical Networks use a predefined metric (e.g., squared Euclidean distance) in the learned embedding space.
• Complexity: The relation module $g_\psi$ can model highly non-linear relationships, potentially offering more flexibility than fixed distance metrics, especially if the underlying class structure in the embedding space is complex.
• Computation: RNs require passing pairs of embeddings (or query-aggregate pairs) through the relation module for each class, which can be computationally more intensive than the simple distance calculations in Prototypical Networks, particularly as the number of ways ($C$) or shots ($K$) increases.

Utilizing Foundation Model Embeddings

When applying Relation Networks in the context of large foundation models, the embedding module $f_\phi$ is naturally replaced by the foundation model.

• Frozen Embeddings: One strategy is to use the foundation model as a fixed feature extractor. The embeddings $f_\phi(x)$ are pre-computed or generated on-the-fly, and only the relation module $g_\psi$ is trained during meta-learning. This is computationally efficient but relies entirely on the quality and suitability of the foundation model's default embedding space.
• Fine-tuning: Alternatively, parts of the foundation model $f_\phi$ could be fine-tuned along with $g_\psi$ during meta-training. This offers more flexibility but significantly increases computational complexity and requires careful consideration of stability, especially given the bilevel nature of meta-learning optimization. Parameter-efficient fine-tuning (PEFT) techniques could potentially be integrated here.

The high dimensionality of embeddings from foundation models (e.g., 768, 1024, or more dimensions) needs consideration when designing the relation module $g_\psi$. A simple MLP might struggle with the curse of dimensionality or become computationally burdensome. Techniques like dimensionality reduction (if feasible without losing critical information) or carefully structured relation modules (e.g., using attention mechanisms or factorization) might be necessary.

Strengths and Limitations

Strengths:

• Learns a flexible, potentially non-linear similarity measure tailored to the task distribution.
• Can capture complex relationships that might be missed by fixed distance metrics.
• Achieved state-of-the-art results on several few-shot benchmarks upon its introduction.

Limitations:

• Computationally more expensive than methods like Prototypical Networks, as it requires forward passes through the relation module for each query-class comparison.
• The performance is sensitive to the architecture design of both the embedding and relation modules.
• Training can sometimes be less stable than simpler metric-based approaches, requiring careful hyperparameter tuning.

Relation Networks provide a powerful alternative within metric-based meta-learning, shifting complexity from solely learning good embeddings to jointly learning embeddings and a flexible comparison mechanism. Their effectiveness with foundation models hinges on efficiently leveraging the pre-trained representations while managing the computational cost of the learned relation module.