Stop 11 · Large Language ModelsTopics · 4/9

Context Window & Attention

In short

Paste a 300-page PDF into a chatbot and it may forget page 2. The context window is the hard limit on what the model can 'see', and attention is how it uses that space.

The maximum number of tokens an LLM can 'see' at once — a hard constraint on how much conversation history, document content, or retrieved context you can feed a model.

The blob writes on a whiteboard while wiping old doodles off one edge, with strings linking related doodles.

Think of it like this

The classroom whiteboard

The teacher's whiteboard has fixed space. When it fills up, the oldest notes get wiped. While explaining, she points back and draws lines between related notes. The board size is the context window. Those connecting lines are attention.

Watch it happen

A chat that outgrows an 8-token window

sys

start

order

late

sorry

where

rider

end

1/5Toy example: the window fits 8 tokens. Everything the model knows right now must fit here.

The big ideas

What you need to know

Input plus output

The window covers the prompt, the chat history, any documents and the reply. Long answers eat into it too.

Attention links tokens

Each token scores every earlier token and blends in the most relevant ones. That's how 'it' finds what it refers to.

Cost grows fast

Standard attention compares every pair of tokens, so compute grows with the square of length. Long contexts are slower and pricier.

Long isn't always better

Models can miss details buried in the middle of very long inputs. Retrieving the right chunks often beats pasting everything.

See it in code

Walk through the code

python

1import tiktoken2enc = tiktoken.get_encoding("cl100k_base")3 4def fit_history(system, turns, limit=8000, reserve=1000):5    budget = limit - reserve - len(enc.encode(system))6    kept = []7    for turn in reversed(turns):            # newest first8        cost = len(enc.encode(turn["content"]))9        if cost > budget:10            break11        kept.insert(0, turn)12        budget -= cost13    return [{"role": "system", "content": system}] + kept

1/3Start from the limit, subtract space for the answer and the system prompt.

Keep the system prompt, reserve room for the reply, and fill the rest with the newest chat turns.

Save yourself the pain

Common mistakes

Oops

Filling the window right up to the limit with input.

Do this instead

Leave room for the reply. If the input uses everything, the answer gets cut off or the call fails.

Oops

Pasting entire books because the model 'supports' a huge context.

Do this instead

Long prompts cost more, run slower and can bury key facts. Send only the relevant parts when you can.

Your mission

Find where it forgets

~30 min

Quick check

A model has a 16,000-token window. Your prompt is 15,800 tokens. What is the likely problem?

Go deeper

1Compare context window sizes across popular models (GPT, Claude, Gemini).
2Learn why larger context windows cost more compute (attention scales quadratically with sequence length).
3Understand strategies (summarization, retrieval) for working within a limited context window.