What is Self-Attention?
Also called intra-attention.
Self-attention is an attention mechanism in which a sequence attends to itself, so every position computes its representation by weighing all other positions in the same sequence. Queries, keys, and values all come from one input rather than from separate sources. It allows a model to build context-sensitive representations of each token based on its surroundings.
In cross-attention, one sequence attends to a different sequence, such as a translation decoder attending to an encoded source sentence. Self-attention removes that split. Each token in a single sequence asks what else in that same sequence is relevant to it, so the representation of a word becomes a blend of the word itself and the words that inform its meaning in this particular context.
Practical implementations use multiple heads. The projections are split into several parallel sets, each computing its own attention pattern, and the results are concatenated. Different heads empirically specialize toward different relationships, such as syntactic dependency, coreference, or positional proximity, though these specializations are emergent tendencies rather than designed roles.
Language models that generate text left to right apply a causal mask, which sets the scores for future positions to negative infinity so they receive zero weight. This makes the model unable to look ahead and preserves the training objective of predicting the next token. Models built for understanding rather than generation often skip the mask and attend in both directions.
Cost is the central constraint. Standard self-attention compares every position with every other, so compute and memory scale with the square of sequence length. Long-context systems reduce this with sliding windows, sparse patterns, low-rank approximations, or kernel tricks, each trading some ability to attend globally for tractable cost.
Key points
- A sequence attends to itself rather than to a separate input
- Multiple heads capture different relationships in parallel
- Causal masking prevents generative models from seeing ahead
- Cost scales with the square of sequence length
- Produces context-sensitive representations of each token
In practice
Consider "the bank raised its rates" and "the bank flooded after the storm". The token "bank" starts from the same embedding in both. Self-attention lets it draw heavily on "rates" in the first sentence and "flooded" in the second, so the representation passed to later layers differs sharply. That context-dependent shaping is what allows one model to handle both meanings without any explicit word-sense lookup.