Understanding Context Window

Imagine you're having a conversation. You naturally remember what was said recently, but details from much earlier might fade. Large Language Models have something similar, a limited memory called the context window.

The context window is the amount of text the model can "see" or "remember" when generating its next response. This includes both your most recent prompt and the preceding parts of the conversation (or the document you provided). Think of it as the model's short-term memory. Everything you input and everything the model outputs contributes to filling this window.

How is Context Measured?

Remember back in Chapter 1, we discussed tokens? Tokens are the basic units of text that LLMs process, often representing parts of words, whole words, or punctuation. The size of the context window is typically measured in the number of tokens it can hold. A model might have a context window of 2048 tokens, 4096 tokens, or even much larger numbers like 32,000 tokens or more. A larger number means the model can consider more text at once.

Why Does the Context Window Matter?

The size of the context window directly impacts how well the model can handle certain tasks:

1. Maintaining Conversation: In a chat, the context window holds the history of your interaction. If the conversation becomes longer than the window size, the earliest parts are "forgotten" by the model. It loses the context from the beginning of the chat.
2. Processing Long Documents: If you ask an LLM to summarize a long article or answer questions about it, the entire relevant text needs to fit within the context window for the model to process it effectively.
3. Following Complex Instructions: Multi-part instructions or prompts that refer back to earlier information rely on the context window to keep track of everything.

An Example: The Sliding Window

Think of the context window like a fixed-size sliding window moving over your conversation history. As new text (your prompt or the model's response) is added, the window slides forward. If the total text exceeds the window size, the oldest text falls out of view.

Let's illustrate this with a simplified example. Imagine a model with a very small context window, say, enough for only 15 words.

Interaction:

1. You: Write a short poem about a cat. (7 words)
2. Model: A furry friend, so sleek and neat, naps in the sun, a cozy treat. (13 words)
   • Context Used: 7 (yours) + 13 (model's) = 20 words.

• Window Contains (last 15 words): "...so sleek and neat, naps in the sun, a cozy treat." (The word "cat" might already be pushed out).

3. You: Now write one about a dog. (6 words)
4. Model: With wagging tail and happy bark, playing fetch down at the park. (12 words)
   • Context Used: 6 (yours) + 12 (model's) = 18 words are added.

• Window Contains (last 15 words): "...happy bark, playing fetch down at the park." (Information about the cat and maybe even the word "neat" is gone).

5. You: What was the first animal I asked about? (8 words)
6. Model: Based on our recent conversation, we discussed a dog. (10 words)

In this simplified case, because "cat" fell out of the small context window, the model couldn't recall it. Real LLMs have much larger windows, but the principle is the same.

A simplified representation showing how, at Turn 3, the information from Turn 1 might fall outside the limited context window, leading the model to only recall information from Turn 2 and Turn 3.

Model Differences and Practical Tips

Different models come with different context window sizes. When you were learning about finding models (Chapter 3), you might have seen specifications like "4k context" or "32k context". This refers to the approximate number of tokens (4096 or 32768, respectively) the model supports. Larger context windows generally require more computational resources (especially RAM and VRAM, as discussed in Chapter 2).

For basic chats, even a moderately sized context window (e.g., 4096 tokens) is often sufficient. However, if you plan to work with very long documents or have extended, complex dialogues, choosing a model with a larger context window becomes more important.

You generally don't need to manually count tokens. The tools you use (like Ollama or LM Studio, covered in Chapter 4) typically manage the context window automatically, trimming the oldest parts of the conversation as needed. However, being aware of this limit helps you understand why a model might sometimes seem to "forget" things you mentioned earlier in a long chat. If you notice this happening, you might need to remind the model by re-stating significant information in your prompt.

Understanding the context window is fundamental to effective prompting. It helps explain the model's conversational memory limits and guides you in structuring interactions, especially as they become longer or more complex.