Adam Algorithm Breakdown

Having introduced Adam (Adaptive Moment Estimation) as an optimizer that combines the benefits of momentum and adaptive scaling like RMSprop, let's look closely at its update mechanism. Adam achieves this by maintaining two separate exponentially decaying moving averages: one for the past gradients (first moment estimate) and another for the past squared gradients (second moment estimate).

First and Second Moment Estimates

At each timestep $t$, after computing the gradients $g_t = \nabla_\theta J(\theta_{t-1})$ with respect to the parameters $\theta$, Adam updates these two moving averages:

1. First Moment Estimate (Mean): This is similar to the momentum term. It accumulates an exponentially decaying average of past gradients.

   $$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$

   Here, $m_t$ is the first moment vector, $g_t$ is the gradient at the current timestep, and $\beta_1$ is the exponential decay rate for the first moment estimates (typically close to 1, e.g., 0.9). $m_0$ is initialized as a vector of zeros.

2. Second Moment Estimate (Uncentered Variance): This is similar to the term used in RMSprop. It accumulates an exponentially decaying average of past squared gradients.

   $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$

   Here, $v_t$ is the second moment vector, $g_t^2$ represents the element-wise square of the gradient vector, and $\beta_2$ is the exponential decay rate for the second moment estimates (often set higher, e.g., 0.999). $v_0$ is also initialized as a vector of zeros.

The terms $m_t$ and $v_t$ are estimates of the mean and the uncentered variance of the gradients, respectively. The hyperparameters $\beta_1$ and $\beta_2$ control the decay rates of these moving averages. Values closer to 1 mean that past gradients have a longer influence.

Bias Correction

A potential issue arises because $m_t$ and $v_t$ are initialized to zero vectors. Especially during the initial timesteps of training, when $t$ is small, these estimates are biased towards zero. Imagine if $\beta_1 = 0.9$; then $m_1 = 0.1 g_1$. This is significantly smaller than the actual gradient.

Adam addresses this initialization bias by computing bias-corrected first and second moment estimates, denoted as $\hat{m}_t$ and $\hat{v}_t$:

$$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$$ $$\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$

Notice the terms $(1 - \beta_1^t)$ and $(1 - \beta_2^t)$ in the denominators. At the beginning of training (small $t$), $\beta_1^t$ and $\beta_2^t$ are close to 1, making the denominators small. This division effectively counteracts the initial zero bias. As training progresses and $t$ increases, $\beta_1^t$ and $\beta_2^t$ approach zero (since $\beta_1, \beta_2 < 1$), so the correction terms $(1 - \beta_1^t)$ and $(1 - \beta_2^t)$ approach 1, and the bias correction has less effect. This ensures that the estimates are more accurate throughout the training process.

Parameter Update Rule

Finally, Adam uses these bias-corrected estimates to update the model parameters $\theta$. The update rule looks quite similar to RMSprop but uses the corrected momentum estimate $\hat{m}_t$ instead of the raw gradient $g_t$:

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$$

Here:

• $\theta_t$ are the parameters at timestep $t$.
• $\alpha$ is the learning rate (or step size).
• $\hat{m}_t$ is the bias-corrected first moment estimate.
• $\sqrt{\hat{v}_t}$ is the element-wise square root of the bias-corrected second moment estimate. This term scales the update adaptively for each parameter, similar to RMSprop. Parameters with larger past squared gradients (larger $\hat{v}_t$) will have smaller updates.
• $\epsilon$ is a small constant (e.g., $10^{-8}$) added for numerical stability to prevent division by zero in cases where $\hat{v}_t$ might be very close to zero.

Summary of the Adam Algorithm

To summarize, the Adam update process at each timestep $t$ involves these steps:

1. Compute gradient: $g_t = \nabla_\theta J(\theta_{t-1})$
2. Update biased first moment estimate: $m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$
3. Update biased second moment estimate: $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$
4. Compute bias-corrected first moment estimate: $\hat{m}_t = m_t / (1 - \beta_1^t)$
5. Compute bias-corrected second moment estimate: $\hat{v}_t = v_t / (1 - \beta_2^t)$
6. Update parameters: $\theta_t = \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$

The authors of the original Adam paper suggest default values of $\beta_1 = 0.9$, $\beta_2 = 0.999$, and $\epsilon = 10^{-8}$. The learning rate $\alpha$ (often suggested around $0.001$) remains a hyperparameter that usually requires tuning.

This step-by-step process, combining adaptive scaling based on the second moment and momentum based on the first moment, along with the important bias correction step, makes Adam a widely used and often effective default optimizer for deep learning models.