What is Tokenization?
Also called tokenizer, subword tokenization.
Tokenization is the step that converts raw text into the sequence of tokens a model can process, and converts the model's output back into text. Modern systems use subword algorithms that learn a fixed vocabulary from training data, keeping frequent words whole and splitting rare ones into pieces. It happens before any model computation and is invisible in the final result.
Byte pair encoding and its relatives build the vocabulary by starting from individual characters and repeatedly merging the most frequent adjacent pair. The result is a fixed list, typically tens of thousands to a few hundred thousand entries, that can represent any possible input without an unknown-word failure. Rare strings simply decompose into more, smaller pieces than common ones do.
The design solves a real dilemma. A word-level vocabulary cannot cover names, typos, or newly coined terms, while a character-level one produces sequences far too long to process efficiently. Subword units strike a balance, giving short sequences for common text and graceful handling of anything unusual, including code, identifiers, URLs, and mixed-language input in a single document.
Tokenizer choice has consequences people rarely anticipate. Vocabularies built mostly from English text encode other languages inefficiently, so the same sentence in one language can consume several times more tokens than in another. Since usage is metered per token, this produces real differences in cost and in effective context length for users writing in less represented languages.
A tokenizer is bound to the model it was trained with and cannot be swapped for another. Counting tokens with a different family's tokenizer gives wrong numbers, sometimes by a wide margin. Whitespace also carries meaning here: a leading space is usually part of the token itself, so formatting changes can quietly change token counts and even output quality.
Key points
- Converts text into a fixed vocabulary of subword units before processing.
- Byte pair encoding merges frequent character pairs to build the vocabulary.
- Non-English text often needs more tokens for the same meaning.
- Each tokenizer belongs to its model and cannot be substituted.
In practice
Feed a tokenizer the sentence 'Tokenization is straightforward.' A common word like 'is' stays whole. The longer, less frequent words might split into 'Token' and 'ization', then 'straight' and 'forward'. The period becomes its own token. The model then receives a list of integer IDs, one per piece, and never sees the original characters at all.