Chapter 1 highlighted the constraints of traditional sequence models, particularly their difficulty handling long-range dependencies and the limitations imposed by fixed-length context vectors. This chapter introduces the attention mechanism, a method designed to overcome these issues by allowing models to dynamically focus on relevant parts of the input sequence when producing an output.

You will learn the fundamental concepts behind attention, starting with the motivation for moving beyond fixed context representations. We will define the general attention framework using the Query ($Q$), Key ($K$), and Value ($V$) abstraction. The mathematical details of the widely used Scaled Dot-Product Attention will be examined, including the significance of the scaling factor: $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$ We will also analyze the role of the softmax function in generating attention weights and discuss how these operations are efficiently implemented using matrix calculations suitable for parallel processing. A practical exercise will guide you through implementing this core attention mechanism. By the end of this chapter, you will have a clear understanding of how attention works at its most fundamental level, preparing you for the more complex variations used in the Transformer architecture.