Self-Attention Mechanisms in CNNs

Standard Convolutional Neural Networks (CNNs) build hierarchical representations of visual data primarily through layers of convolutions and pooling. While highly effective for learning local patterns and textures, the fixed, local receptive field of the convolution operation inherently limits the network's ability to model long-range dependencies and capture global context directly. For instance, understanding the relationship between distant objects in a scene or relating a small detail to the overall image structure can be challenging for standard CNNs without resorting to very deep networks or aggressive pooling, which can discard fine-grained information.

Self-attention mechanisms provide a powerful way to address this limitation by allowing the network to dynamically weight the importance of features across different spatial locations or channels based on the input itself. Instead of relying on fixed receptive fields, attention allows the model to selectively focus on the most informative parts of the feature map for the task at hand, effectively creating dynamic, content-dependent connections.

Channel Attention: Focusing on 'What'

One prominent application of self-attention in CNNs is channel attention, which aims to model the interdependencies between feature channels. The core idea is that different channels in a feature map often correspond to different semantic attributes or object detectors, and not all channels are equally important for the subsequent layers. Channel attention mechanisms learn to explicitly assign weights to each channel, amplifying useful features and suppressing less relevant ones.

Squeeze-and-Excitation (SE) Networks

The Squeeze-and-Excitation (SE) block is a computationally lightweight and effective implementation of channel attention. It can be readily integrated into existing CNN architectures. The SE block operates in three stages:

1. Squeeze: This stage aggregates global spatial information into a channel descriptor. For an input feature map $U \in \mathbb{R}^{H \times W \times C}$ (Height, Width, Channels), global average pooling (GAP) is typically used to produce a vector $z \in \mathbb{R}^{1 \times 1 \times C}$. The $c$-th element of $z$ is calculated as:

   $$z_c = F_{sq}(u_c) = \frac{1}{H \times W} \sum_{i=1}^{H} \sum_{j=1}^{W} u_c(i, j)$$

   Here, $u_c$ represents the $c$-th channel of the input feature map $U$. This squeezed representation $z$ contains channel-wise statistics, effectively summarizing the global information for each channel.

2. Excitation: This stage learns a non-linear, channel-specific activation function using the squeezed information. It aims to capture the complex dependencies between channels. A common approach uses two fully connected (FC) layers with a bottleneck structure:

   $$s = F_{ex}(z, W) = \sigma(W_2 \delta(W_1 z))$$

   Here, $\delta$ is the ReLU activation function, and $\sigma$ is the Sigmoid activation function. $W_1 \in \mathbb{R}^{\frac{C}{r} \times C}$ and $W_2 \in \mathbb{R}^{C \times \frac{C}{r}}$ are the weights of the two FC layers. The first FC layer reduces the channel dimension by a factor $r$ (the reduction ratio, a hyperparameter), creating a bottleneck that limits model complexity and aids generalization. The second FC layer restores the channel dimension to $C$. The final Sigmoid activation ensures the output weights $s \in \mathbb{R}^{1 \times 1 \times C}$ are normalized between 0 and 1. These weights represent the learned importance or 'excitation' for each channel.

3. Scale (Rescale): The final stage applies the learned channel attention weights $s$ to the original input feature map $U$. The output feature map $\tilde{X} \in \mathbb{R}^{H \times W \times C}$ is obtained by channel-wise multiplication:

   $$\tilde{x}_c = F_{scale}(u_c, s_c) = s_c \cdot u_c$$

   Each channel $u_c$ of the input feature map is scaled by its corresponding attention weight $s_c$. This adaptively recalibrates the feature responses channel by channel, emphasizing informative channels and diminishing less useful ones.

Data flow within a Squeeze-and-Excitation (SE) block. The input feature map is processed through the attention path (Squeeze and Excitation) to compute channel weights, which are then used to rescale the original input map.

SE blocks significantly enhance the representational capacity of CNNs with only a minor increase in computational cost and parameters. They can be easily inserted into various existing architectures, often placed after the convolutional layers within residual blocks (e.g., creating SE-ResNet).

Spatial Attention and Non-local Networks: Focusing on 'Where' and Long-Range Dependencies

While channel attention focuses on what features are important, spatial attention aims to determine where in the feature map the most relevant information resides. Some mechanisms compute a spatial attention map based on inter-channel relationships. However, a more direct approach to capturing long-range spatial dependencies, inspired by self-attention in natural language processing, is found in Non-local Networks.

Non-local Neural Networks

Non-local Networks introduce blocks that compute the response at a position as a weighted sum of features at all positions in the input feature map. This allows the network to capture dependencies between distant spatial locations directly, overcoming the limitations of local receptive fields.

The generic non-local operation can be defined as:

$$y_i = \frac{1}{\mathcal{C}(x)} \sum_{\forall j} f(x_i, x_j) g(x_j)$$

Let's break down this formulation:

• $x$ is the input feature map.
• $i$ is the index of the output position (e.g., a spatial location in the feature map).
• $j$ enumerates all possible positions in the input feature map.
• $f(x_i, x_j)$ is a pairwise function that computes a scalar representing the relationship (e.g., affinity or similarity) between position $i$ and position $j$.
• $g(x_j)$ is a unary function that computes a representation of the input signal at position $j$. Often, this is a simple linear embedding: $g(x_j) = W_g x_j$, where $W_g$ is a learned weight matrix.
• $y_i$ is the output response at position $i$.
• $\mathcal{C}(x)$ is a normalization factor, typically the sum of all pairwise affinities for position $i$: $\mathcal{C}(x) = \sum_{\forall j} f(x_i, x_j)$.

Different choices for the pairwise function $f$ lead to different variants of the non-local block. A common and effective choice is the Embedded Gaussian function:

$$f(x_i, x_j) = e^{\theta(x_i)^T \phi(x_j)}$$

Here, $\theta(x_i) = W_\theta x_i$ and $\phi(x_j) = W_\phi x_j$ are linear embeddings (learned transformations) of the input features at positions $i$ and $j$, respectively. This formulation closely resembles the dot-product attention used in Transformers. The exponential function computes the affinity based on the dot product between the embedded representations. The entire operation effectively computes a weighted average of transformed input features $g(x_j)$, where the weights are determined by the similarity between the target position $i$ and all other positions $j$.

Non-local blocks can be inserted at various depths within a CNN. When placed in deeper layers, they can operate on semantically richer features and capture complex spatial relationships. However, computing the pairwise interactions across all spatial locations ($H \times W$ positions) can be computationally intensive, scaling quadratically with the number of spatial locations. This cost often limits their application to feature maps that have already been downsampled spatially.

Integrating Attention into CNNs

Both SE blocks and Non-local blocks act as complementary enhancements to standard convolutional layers.

• SE Blocks: Typically added within or after convolutional blocks (like residual blocks) to recalibrate channel features based on global information. Their low overhead makes them widely applicable.
• Non-local Blocks: Inserted strategically, often in later stages of the network or interspersed between standard convolutional blocks, to explicitly model long-range spatial dependencies. Their computational cost requires careful consideration regarding placement and input feature map size.

By incorporating self-attention mechanisms, CNNs gain the ability to dynamically modulate their feature representations based on the global context of the input image. Channel attention helps the network focus on the most relevant feature types, while spatial attention and non-local operations allow it to explicitly model relationships between distant parts of the image. These techniques enable CNNs to build more powerful and context-aware representations, leading to improved performance on various computer vision tasks, especially those requiring an understanding of broader scene structure or relationships between objects.