Context Window

The context window is the maximum number of tokens a model can attend over at once — every token of your prompt, plus everything it has generated so far, has to fit inside it. It is a hard budget, spent in tokens, not characters or words. When people say a model is “128K,” they mean the window holds about 128,000 tokens; ask for the 128,001st and the oldest tokens fall off the edge.

The instinct is to read that number as an arbitrary product decision — a cap someone could lift if they felt generous. It isn’t. The window is finite because the cost of attention grows with the square of the number of tokens, and that square is what runs you out of GPU memory. The window is wherever the hardware says “no more.”

That single fact — cost scales as N², not N — is the whole lesson. It explains why long context is expensive, why the workarounds look the way they do, and what they each give up to get there.

Recall what attention actually computes: for each token, a query vector is compared against the key vector of every token it can see, by taking a dot product. With N tokens, that is N queries × N keys = N² dot products — a full N×N score matrix — and it happens independently in every layer and every attention head. Nothing about the operation is linear in N; it is quadratic by construction, because it is all-pairs.

So when you double the context, you do not double the work. Each token now has twice as many partners to score, and there are twice as many tokens doing the scoring: 2N tokens × 2N tokens = 4N². Doubling the context quadruples the cost. That ×4-per-doubling is the engine behind every number on this page.

Drag the context length below. Watch the dot-product count and the memory for the score matrix climb as N², and notice that on the bar chart almost the entire cost sits in the largest window — everything smaller is a sliver. Then compare the three attention masks to see exactly what you buy by not computing the whole matrix.

The dense grid fills the entire causal triangle — that is the N² wall. Sliding window keeps only a fixed band near the diagonal, so its cost grows like N·w (linear), but a query can no longer reach a token more than w steps back. Sparse attention keeps the recent band, a few always-on “global” tokens, and a strided sample of the rest — near-linear cost that approximates long-range links instead of computing them exactly. Sparsity is never free; you are trading reach for scale.

Take N² literally. Using 1K = 1,000 tokens for round figures, here is the dot-product count per layer per head, and the memory to hold the N×N score matrix in fp16 (2 bytes per score):

Every step down the table is one doubling of context and a ×4 in both columns. Going from 4K to 128K is a 32× longer prompt but a 1,024× heavier attention matrix. And that 33 GB at 128K is for one head in one layer — a real model stacks dozens of layers × dozens of heads on top of it.

The arithmetic is a two-line loop — there is no hidden complexity, just N*N:

BYTES_FP16 = 2                              # one attention score, half precision
prev = None
for k in (4, 8, 16, 32, 64, 128):
    n = k * 1000
    dots = n * n                            # attention scores, per layer per head
    mem  = dots * BYTES_FP16                # bytes for the N×N fp16 matrix
    mult = "" if prev is None else f"  ×{dots // prev}"
    print(f"{k:>4}K  {dots/1e6:>9,.0f}M dots  {mem/1e9:6.2f} GB{mult}")
    prev = dots

#    4K       16M dots    0.03 GB
#    8K       64M dots    0.13 GB  ×4
#   16K      256M dots    0.51 GB  ×4
#   32K    1,024M dots    2.05 GB  ×4
#   64K    4,096M dots    8.19 GB  ×4
#  128K   16,384M dots   32.77 GB  ×4

This is why 128K context costs far more than 32× a 4K request: the attention term grew 1,024×, and it is that term — not the linear-in-N parts of the network — that decides where the window has to stop.

• You run out of VRAM before you run out of compute. The common guess is that long context makes the model “CPU-bound.” It doesn’t — the GPU’s memory fills first. The N×N score matrix (and the activations around it) have to live in VRAM, and at 128K a single head’s matrix is already tens of GB. Memory is the hard wall; the GPU is starved for space long before it is starved for math.
• Flash attention is how the wall gets pushed back — not by changing the math. A model reaches 128K not with a different attention formula but by never materializing the full N×N matrix: flash attention computes it in tiles and recomputes on the fly, turning O(N²) memory into roughly O(N) memory (the compute stays N²). The window grows because the memory term was tamed.
• The KV cache grows with every token, forever. During generation the model caches the key and value vectors of every past token so it doesn’t recompute them. That cache is linear in N — but at 128K it is still gigabytes, and it only grows as the conversation continues. Long sessions die from KV-cache memory even when flash attention has handled the score matrix.
• Sliding-window and sparse attention scale, but drop long-range links. Restrict each token to a local band and cost becomes linear — but the model literally cannot connect a fact on page 1 to a question on page 50. The long-range dependency isn’t approximated; for a pure sliding window it is gone.
• Coherence collapses when a document is truncated mid-stream. Overflow the window and the oldest tokens silently fall off. The model keeps generating fluently, but it is now reasoning over a document whose opening — definitions, constraints, the actual question — has vanished. The output stays grammatical and goes quietly wrong. This is exactly the failure RAG exists to avoid: retrieve the few relevant chunks instead of stuffing the whole corpus into a budget it can’t fit.