Long Short-Term Memory (LSTM) Networks

As we saw in the previous section, simple Recurrent Neural Networks (RNNs) struggle to learn dependencies between elements that are far apart in a sequence. This is largely due to the vanishing gradient problem, where gradients become too small during backpropagation to effectively update the network's weights for earlier time steps. To overcome this limitation, a more sophisticated recurrent unit was developed: the Long Short-Term Memory (LSTM) network, introduced by Hochreiter and Schmidhuber in 1997.

LSTMs are explicitly designed to remember information for long periods. The central innovation is the cell state ($C_t$), often seen as the network's "memory," which can carry information relatively unchanged across many time steps. Unlike the single hidden state transformation in a simple RNN, LSTMs use a system of gates to carefully regulate the flow of information into and out of this cell state.

The LSTM Cell Structure

An LSTM unit processes data sequentially, taking the input at the current time step ($x_t$) and the hidden state from the previous time step ($h_{t-1}$) to compute the new hidden state ($h_t$) and update its internal cell state ($C_t$). This process is controlled by three primary gates:

1. Forget Gate ($f_t$): This gate decides what information to discard from the cell state. It looks at $h_{t-1}$ and $x_t$ and outputs a number between 0 and 1 for each number in the previous cell state $C_{t-1}$. A 1 represents "completely keep this," while a 0 represents "completely get rid of this." It uses a sigmoid activation function ($\sigma$). $f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$ Here, $[h_{t-1}, x_t]$ denotes the concatenation of the previous hidden state and the current input, $W_f$ are the weights, and $b_f$ is the bias for the forget gate.

2. Input Gate ($i_t$): This gate determines which new information will be stored in the cell state. It consists of two parts:

   • First, a sigmoid layer decides which values to update ($i_t$). $i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$
   • Second, a tanh layer creates a vector of new candidate values, $\tilde{C}_t$, that could be added to the state. $\tilde{C}_t = \tanh(W_C [h_{t-1}, x_t] + b_C)$ These two parts are combined to update the cell state.

3. Updating the Cell State: The old cell state $C_{t-1}$ is updated to the new cell state $C_t$. We multiply the old state by $f_t$, forgetting the things we decided to forget earlier. Then we add $i_t * \tilde{C}_t$. This is the new candidate information, scaled by how much we decided to update each state value. $C_t = f_t * C_{t-1} + i_t * \tilde{C}_t$ Note the use of addition here. This additive interaction is important for allowing gradients to flow better compared to the repeated multiplications in simple RNNs.

4. Output Gate ($o_t$): This gate decides what the next hidden state $h_t$ (the output for this time step) should be. The output will be based on the filtered cell state.

   • First, a sigmoid layer determines which parts of the cell state to output. $o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$
   • Then, the (now updated) cell state $C_t$ is passed through tanh (to push values between -1 and 1) and multiplied by the output of the sigmoid gate $o_t$. This ensures we only output the parts we decided to. $h_t = o_t * \tanh(C_t)$

The hidden state $h_t$ is then passed to the next time step (as $h_{t-1}$) and can also be used as the output of the model at the current step.

Visualizing the LSTM Cell

The structure of an LSTM cell, with its gates controlling the flow of information through the cell state, can be visualized as follows:

Data flow within a single LSTM cell. Arrows indicate data dependencies. Circles with 'X' denote element-wise multiplication, '+' denotes element-wise addition, 'σ' denotes the sigmoid function, and 'tanh' denotes the hyperbolic tangent function. The cell state acts as a conveyor belt, with gates controlling information removal and addition.

How LSTMs Mitigate Vanishing Gradients

The mechanism allowing LSTMs to handle long-range dependencies better than simple RNNs lies in the cell state and the gating mechanism.

• Additive Updates: The cell state $C_t$ is updated primarily through addition ($C_t = f_t * C_{t-1} + i_t * \tilde{C}_t$). This additive interaction makes it easier for gradients to flow backward through time without vanishing as quickly as they do through the repeated matrix multiplications found in simple RNNs.
• Gate Control: The gates act as regulators. The forget gate can learn to preserve important information from the distant past by keeping its activation close to 1 for certain elements of the cell state. Similarly, the input gate can learn to prevent irrelevant new information from corrupting the cell state. This selective memory allows the network to maintain context over long sequences.

Think of the cell state as a conveyor belt carrying information through time. The forget gate removes items from the belt, the input gate adds new items, and the output gate reads items off the belt to decide the immediate output (hidden state). This structure is much more effective at preserving signals over long durations.

LSTMs have been highly successful in various NLP tasks, including machine translation, sentiment analysis, and sequence labeling, precisely because of their ability to capture long-range context. While they are more complex than simple RNNs, their effectiveness often justifies the additional computational overhead.

In the next section, we'll look at Gated Recurrent Units (GRUs), a slightly simpler variant of the LSTM that often achieves comparable performance.