Contractive Autoencoders Formulation

While Sparse Autoencoders encourage finding concise representations by zeroing out less important latent units, and Denoising Autoencoders learn by reconstructing data from corrupted versions, Contractive Autoencoders (CAEs) take a different approach to building representations. They explicitly aim to make the learned feature mapping, represented by the encoder function $f(x)$, insensitive to small variations in the input data $x$.

The central idea is contraction. We want the autoencoder to learn a mapping where small changes in the input space correspond to even smaller changes in the latent feature space. This encourages the model to capture features that represent significant factors of variation in the data, while effectively ignoring minor, irrelevant fluctuations or noise. Imagine the encoder mapping points from a high-dimensional input space to a lower-dimensional latent space; a contractive mapping ensures that neighborhoods in the input space shrink when mapped to the latent space.

Mathematical Formulation

To achieve this insensitivity, CAEs add a penalty term to the standard reconstruction loss. This penalty measures how much the encoder's output $h = f(x)$ changes in response to infinitesimal changes in the input $x$. This sensitivity is captured by the Jacobian matrix of the encoder function $f$ with respect to $x$.

The Jacobian matrix $J(x)$ is a matrix containing all the first-order partial derivatives of the encoder's output vector $h$ with respect to the input vector $x$:

$$J_{ij}(x) = \frac{\partial h_i}{\partial x_j}$$

where $h_i$ is the $i$-th element of the latent representation and $x_j$ is the $j$-th element of the input.

The Contractive Autoencoder adds a penalty proportional to the squared Frobenius norm of this Jacobian matrix. The Frobenius norm $||A||_F$ of a matrix $A$ is defined as $||A||_F = \sqrt{\sum_{i,j} A_{ij}^2}$. Therefore, the penalty term is:

$$\Omega_{CAE}(x) = ||J(x)||_F^2 = \sum_{i,j} \left( \frac{\partial h_i(x)}{\partial x_j} \right)^2$$

This term sums the squares of all partial derivatives of the latent features with respect to the inputs. Minimizing this term encourages these derivatives to be small, making the mapping $f(x)$ contractive.

The complete loss function for a Contractive Autoencoder is the sum of the reconstruction loss (e.g., Mean Squared Error or Binary Cross-Entropy) and this contractive penalty, weighted by a hyperparameter $\lambda$:

$$L_{CAE}(x, \hat{x}) = L_{rec}(x, \hat{x}) + \lambda \Omega_{CAE}(x)$$ $$L_{CAE}(x, \hat{x}) = L_{rec}(x, \hat{x}) + \lambda \sum_{i,j} \left( \frac{\partial h_i(x)}{\partial x_j} \right)^2$$

Here, $\hat{x} = g(f(x))$ is the reconstructed output, $L_{rec}$ is the reconstruction loss, and $\lambda > 0$ controls the strength of the contractive penalty. A larger $\lambda$ enforces stronger contraction but might compromise reconstruction quality if set too high. Tuning $\lambda$ allows balancing the trade-off between learning an insensitive representation and accurately reconstructing the input data.

Interpretation and Connection to Manifold Learning

Minimizing the Jacobian norm encourages the encoder $f(x)$ to be contractive primarily in the vicinity of the training data points. This has an interesting connection to manifold learning. If the data lies on or near a lower-dimensional manifold within the higher-dimensional input space, the CAE tends to learn features that capture variations along the manifold while contracting (being insensitive to) directions orthogonal to the manifold. Essentially, the penalty encourages the encoder to learn the tangent space of the data manifold locally.

An illustration of the contractive mapping. Small perturbations $\delta x$ in the input space should lead to smaller perturbations $\delta h$ in the latent space, achieved by minimizing the Jacobian norm of the encoder $f$.

Implementation Considerations

Calculating the full Jacobian matrix and its Frobenius norm can add significant computational overhead to the training process, especially for high-dimensional inputs and latent spaces. Fortunately, modern deep learning frameworks with automatic differentiation capabilities can compute this penalty efficiently. While the exact computation might still be slower than a simple reconstruction loss, it's often feasible for typical network sizes. Frameworks often provide optimized ways to compute sums of squared gradients or use approximations, avoiding the explicit construction of the full Jacobian matrix.

Comparison with Denoising Autoencoders

Both Contractive and Denoising Autoencoders aim to learn representations that are less sensitive to input variations. However, they achieve this goal differently:

• DAEs: Learn robustness implicitly by training the model to reconstruct the original input from a stochastically corrupted version. The model must learn features that are stable enough to ignore the specific type of noise introduced during training.
• CAEs: Learn robustness explicitly by adding a penalty term that directly minimizes the sensitivity (gradient) of the learned features with respect to all directions of input perturbations.

The choice between DAEs and CAEs might depend on the specific task and the nature of expected variations in the data. CAEs provide a more direct analytical way to enforce insensitivity, while DAEs might be more intuitive and sometimes easier to implement depending on the noise model chosen.

In summary, Contractive Autoencoders offer a mathematically grounded approach to regularization by penalizing the sensitivity of the encoder mapping. This encourages the model to learn representations that capture the essential structure of the data while remaining invariant to small, local perturbations, contributing to more feature learning.