Geometric Interpretation of SVD

While the Singular Value Decomposition (SVD) formula, $A = U\Sigma V^T$, might look abstract, it describes a fundamental geometric process. Understanding this geometry provides intuition for why SVD is so effective in applications like dimensionality reduction and understanding linear transformations.

Consider a linear transformation represented by an $m \times n$ matrix $A$. When we apply this transformation to a vector $x$ to get $y = Ax$, SVD tells us this transformation can be broken down into three distinct geometric steps:

1. Rotation/Reflection ($V^T$): The matrix $V^T$ (the transpose of $V$) is an orthogonal matrix. When applied to the input vector $x$, $V^T x$ performs a rotation and possibly a reflection of the input space $\mathbb{R}^n$. It doesn't change the lengths of vectors or the angles between them. Think of this step as aligning the input space along a special set of orthogonal directions, given by the columns of $V$ (the right singular vectors). These vectors $v_1, v_2, \dots, v_n$ form an orthonormal basis for the input space. $V^T$ essentially rotates the space so that these principal input directions align with the standard coordinate axes.

2. Scaling ($\Sigma$): The matrix $\Sigma$ is an $m \times n$ rectangular diagonal matrix. Its diagonal entries are the singular values $\sigma_1, \sigma_2, \dots, \sigma_r$ (where $r$ is the rank of $A$), and all other entries are zero. This matrix scales the coordinates of the rotated vector $V^T x$. Specifically, it scales the $i$-th coordinate (which corresponds to the direction of $v_i$) by the singular value $\sigma_i$. Directions corresponding to zero singular values are effectively squashed to zero. This step stretches or shrinks the space along the newly aligned axes.

3. Rotation/Reflection ($U$): The matrix $U$ is also an orthogonal matrix ($m \times m$). This final step takes the scaled vector $\Sigma V^T x$ and performs another rotation and possibly a reflection in the output space $\mathbb{R}^m$. The columns of $U$, $u_1, u_2, \dots, u_m$ (the left singular vectors), form an orthonormal basis for the output space. This step rotates the scaled vectors from the axes-aligned orientation (after step 2) into their final positions in the output space, aligning them with the principal output directions defined by the columns of $U$.

Visualizing the Transformation

Imagine applying the transformation $A$ to all the points on a unit circle in 2D (or a unit sphere in 3D). Here's how SVD breaks down the transformation geometrically:

1. Input: Start with a unit circle centered at the origin.
2. Apply $V^T$: This rotates the unit circle. Since it's a rigid transformation, the result is still a unit circle, just possibly oriented differently. The axes defined by the right singular vectors $v_1, v_2$ are rotated to align with the standard axes $e_1, e_2$.
3. Apply $\Sigma$: This scales the rotated circle along the standard axes by factors $\sigma_1$ and $\sigma_2$. The circle becomes an ellipse, with its major and minor axes aligned with the standard coordinate axes. The lengths of the semi-axes are $\sigma_1$ and $\sigma_2$.
4. Apply $U$: This rotates the ellipse without changing its shape. The final result is an ellipse whose axes are aligned with the left singular vectors $u_1, u_2$.

The SVD essentially tells us that any linear transformation $A$ maps orthonormal basis vectors in the input space (columns of $V$) to orthogonal vectors in the output space (columns of $U$ scaled by singular values $\sigma_i$). That is, $Av_i = \sigma_i u_i$ for $i = 1, \dots, r$.

The transformation defined by matrix $A = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ maps the unit circle (gray) to an ellipse (red). SVD decomposes this overall transformation into a sequence of rotation ($V^T$), axis-aligned scaling ($\Sigma$), and another rotation ($U$).

This geometric view is particularly insightful for understanding dimensionality reduction using SVD. The singular values $\sigma_i$ quantify the importance of each principal direction. Larger singular values correspond to directions where the data (or the transformation) has the most variance or "spread". By keeping only the components corresponding to the largest singular values, we retain the most significant geometric features of the transformation while potentially discarding dimensions associated with small singular values (which might represent noise or less important variations).