What is Attention Mechanism?
Also called attention.
An attention mechanism is a neural network component that lets a model weigh which parts of its input matter most when computing each part of its output. Instead of compressing a whole sequence into one fixed representation, attention computes a set of relevance scores and forms a weighted blend of the input positions. It is the core building block of transformer architectures.
Attention works by projecting each position into three vectors conventionally called query, key, and value. The query for the position being computed is compared against every key to produce a similarity score, the scores are normalized into weights that sum to one, and the values are averaged using those weights. The result is a representation that draws mostly from the positions judged most relevant.
Before attention, sequence models passed information along step by step through recurrent connections, so a detail near the start of a long input had to survive many intermediate steps to influence the end. Attention creates a direct path between any two positions. This shortens the distance information must travel and makes long-range dependencies far easier to learn.
Attention is also highly parallel. Every position can compute its scores against every other position at the same time, which suits modern accelerator hardware and is a major reason transformer training scaled as far as it did. The cost is that naive attention grows quadratically with sequence length, which is why long-context work focuses so heavily on cheaper attention variants.
Attention weights are sometimes presented as an explanation of model behavior, showing which words the model looked at. This interpretation is contested. Weights indicate where information was gathered from, but they do not reliably reveal why a particular output was produced, and different weight patterns can yield the same prediction.
Key points
- Scores how relevant each input position is to each output
- Built from query, key, and value projections
- Creates direct paths between distant positions in a sequence
- Parallelizes well but costs grow with sequence length
- Attention weights are weak evidence of model reasoning
In practice
In the sentence "the invoice the accountant approved was overdue", a model resolving what "was overdue" refers to must connect it back to "invoice", not the nearer word "accountant". Attention lets the position computing "was" score every earlier word directly and place most weight on "invoice", regardless of how many words sit between them. A purely sequential model would have to carry that information forward through every intervening step.