What is Positional Encoding?
Also called position embedding.
Positional encoding is the technique of injecting information about token order into a transformer, which otherwise treats its input as an unordered set. The encoding is added to or applied over token representations so that the model can distinguish sequences that contain the same tokens in different arrangements. Several schemes exist, including fixed sinusoidal patterns, learned embeddings, and rotary methods.
Attention computes weighted sums over positions, and a weighted sum does not depend on the order of its terms. Without positional information, a transformer would produce identical representations for "the dog bit the man" and "the man bit the dog". Positional encoding restores the missing signal by making each position carry a distinct, learnable or deterministic marker.
Early transformers added fixed sinusoidal vectors of varying frequency, chosen so that relative offsets correspond to predictable transformations. Learned absolute embeddings, where each index gets its own trained vector, are simple but cannot generalize past the longest position seen in training. Both approaches encode where a token sits in absolute terms rather than how far it sits from another token.
Rotary and relative schemes became dominant because they encode distance rather than absolute index. Rotary methods rotate query and key vectors by an angle proportional to position, so attention scores depend on the difference between two positions. This aligns better with how language works, since the relationship between two words usually depends on their separation rather than their offset from the document start.
Positional encoding is central to extending context length. Techniques that stretch or reinterpret the position signal, sometimes called interpolation or frequency scaling, let a model trained at one length operate at a longer one with limited additional training. Quality typically degrades gradually rather than failing outright, so claims about very long contexts warrant task-specific verification.
Key points
- Transformers have no built-in notion of token order
- Sinusoidal, learned, relative, and rotary schemes all exist
- Relative and rotary methods encode distance, not absolute index
- Position handling limits and enables context length extension
- Long-context quality often degrades gradually rather than sharply
In practice
A model asked to compare "payment received before shipment" with "shipment received before payment" must distinguish two sentences built from nearly identical tokens. Positional encoding is what makes the representations differ: the same word embedding for "payment" is combined with a different position signal in each sentence, so attention resolves the ordering. Strip the positional component out and the model can no longer tell the two clauses apart.