Positional Encoding

A transformer processes every token in parallel — there is no left-to-right scan, no memory of “what came before,” nothing in the attention arithmetic that knows about sequence. The attention score between two tokens is a dot product of a query and a key, and a dot product does not care where either vector came from. Shuffle the input tokens and the outputs shuffle identically. To the raw mechanism, “the king is here” and “here is the king” are the same bag of tokens.

That is not a small defect; it deletes syntax. Word order is most of English grammar, and the model literally cannot see it. So position has to be injected into the input — turned into numbers and mixed into each token’s vector so that the same token at position 3 and position 17 arrives at attention as two different vectors.

The vector comes from the same space the Embeddings topic describes — position is just added into the row, dimension by dimension. And the choice of how to encode position turns out to control how far past its training length a model can stretch, which is exactly the Context Window story.

The original transformer’s scheme is sinusoidal: position pos maps to a deterministic vector whose entries are sines and cosines at geometrically spaced frequencies,

PE(pos, 2i)   = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Words undersell what that looks like. Look at it instead — the whole encoding matrix at once, then two positions pulled out and compared, then the modern alternatives side by side:

Three things the picture is showing you:

• Each dimension pair is a clock running at its own speed. The leftmost pair cycles every ~6 positions (wavelength 2π); each pair to the right runs slower, out to a wavelength of thousands of positions. Fast clocks give fine position, slow clocks give coarse position — together the row pins down exactly one position, the same way hour, minute, and second hands pin down a time of day.
• Nothing is learned in the encoding itself. The matrix is fixed before training starts. What the model learns is how to read it — which attention heads should use which frequency bands.
• Relative position falls out for free. The dot product between two encodings depends only on the offset between them, not their absolute location — you verified this in the widget when the grey similarity curve refused to change shape as you dragged A around. Even better, PE(pos+k) is a fixed rotation of PE(pos) (each sin/cos pair turns by k·ω), so “attend 3 tokens back” is a linear operation the model can learn once and apply anywhere.

Where this happens in the stack matters: additive schemes (sinusoidal, learned) mix position into the embedding before the Q/K/V projections the Attention topic walks through. RoPE acts later, inside attention, on Q and K after projection. ALiBi acts later still, on the score matrix itself. Same goal, three different insertion points.

Shrink the model width to d = 8 so the whole vector fits on a line. That gives four sin/cos pairs with divisors 10000^(2i/8) = 1, 10, 100, 1000 — wavelengths of 6.3, 63, 628, and 6,283 positions. The first three rows of the encoding matrix:

Read a row left to right and you can see the clock speeds: between position 1 and 2 the first pair swings hard (0.8415 → 0.9093, 0.5403 → −0.4161), the second pair moves a little, and the last two pairs barely twitch. One step is a big deal to the fast clock and nothing to the slow ones.

The generator is a few lines:

import numpy as np

def pe(pos, d=8, base=10000):
    i = np.arange(d // 2)
    angles = pos / base ** (2 * i / d)   # divisors: 1, 10, 100, 1000
    out = np.empty(d)
    out[0::2] = np.sin(angles)
    out[1::2] = np.cos(angles)
    return out

np.round(pe(1), 4)
# array([0.8415, 0.5403, 0.0998, 0.995 , 0.01  , 1.    , 0.001 , 1.    ])

# similarity depends ONLY on the offset, not the location:
pe(7) @ pe(8),  pe(30) @ pe(31)   # (3.5353, 3.5353)   offset 1
pe(7) @ pe(9),  pe(30) @ pe(32)   # (2.5637, 2.5637)   offset 2
pe(7) @ pe(7)                     # 4.0  (= d/2, since sin² + cos² = 1)

That last block is the property the widget’s similarity curve plots: the dot product between PE(7) and PE(8) is 3.5353, and between PE(30) and PE(31) it is exactly the same 3.5353. The encoding hands attention a clean “how far apart are we?” signal at any absolute location. This is what the input to the model actually is: token embedding + this vector, summed elementwise, and only then projected to Q, K, and V.

• No positional encoding: syntax is gone. Without the injected vector, attention is permutation-equivariant — “the king is here” and “here is the king” produce identical representations up to reordering. The model can still do bag-of-words topic matching; it cannot do grammar, negation scope, or “who did what to whom.”
• Learned encodings hit a wall, hard. GPT-2 has exactly 1,024 position rows. Position 1,025 is not “poorly encoded” — the row does not exist. Feeding a longer sequence is an out-of-bounds lookup; the context length is a hard architectural ceiling. This is the worst extrapolation behavior of the four schemes.
• Sinusoidal encodings extrapolate in theory, badly in practice. The formula produces a perfectly valid vector at position 50,000 — but the model only learned to interpret the phase combinations it saw during training. Past the training length the vectors are novel inputs, and quality degrades fast. Defined ≠ understood.
• RoPE stretches — with tricks — and that is why it won. Because RoPE encodes only relative angles, positions can be rescaled after training: position interpolation, NTK-aware scaling, and YaRN squeeze longer sequences into the angle range the model already understands. That is the machinery behind stretching a model trained at 8K out to 128K — the long-context story in the Context Window topic. ALiBi extrapolates natively (the distance penalty just keeps growing) but gives up long-range precision to do it.
• A subtle one: additive encodings share the embedding space. The position vector is summed into the same dimensions carrying token meaning (the Embeddings topic’s space) — the model must learn to keep “what” and “where” separable inside one vector. RoPE sidesteps this entirely by never touching the values, which is part of why it composes better at scale.