What is Transformer?
Also called transformer architecture.
The transformer is the neural network architecture behind nearly all current language models. Introduced in a 2017 research paper, it processes a whole sequence at once and uses a mechanism called attention to let every position weigh the relevance of every other position. That design parallelizes well on modern hardware, which is what made training on internet-scale text practical.
Attention is the core idea. For each token, the model computes how much it should draw on every other token in the sequence, then blends their representations accordingly. Stacking dozens or hundreds of such layers lets early layers capture local grammar while later layers capture longer-range meaning, such as which noun a pronoun refers to several paragraphs earlier in the same document.
The predecessor architectures, recurrent networks, read text strictly left to right and carried a running summary forward. That made them hard to parallelize across hardware and prone to losing detail over long spans. Transformers replace the running summary with direct access to the entire sequence, trading a memory bottleneck for a compute cost that grows quadratically with sequence length.
It is a common error to treat transformer as a synonym for language model. The architecture is modality-agnostic and now underpins image, audio, protein, and video models as well. Variants also differ substantially: encoder-only designs suit classification and embeddings, decoder-only designs suit text generation, and encoder-decoder designs remain common for translation and other sequence-to-sequence work.
The quadratic cost of attention is the practical constraint most people eventually meet, because it is a major reason context windows are limited and long inputs cost more. A large body of research targets this with sparse, linear, and cached attention schemes, while mixture-of-experts designs attack a related problem by activating only part of the network for each token.
Key points
- Introduced in a 2017 research paper on attention mechanisms.
- Attention lets every position weigh every other position in the sequence.
- Parallel processing made internet-scale training feasible.
- Used for images, audio, and proteins, not only for text.
- Attention cost grows quadratically with input length.
In practice
Consider the sentence 'The keys to the cabinet are on the table.' To choose 'are' over 'is', the model must connect the verb to 'keys' and not to the nearer word 'cabinet'. Attention does exactly this: when processing the verb position, it assigns high weight to 'keys' and low weight to the intervening words, and the correct plural form follows from that weighting.