What is Token?
Also called tokens.
A token is the unit of text a language model actually reads and writes. It is usually a common word, a word fragment, a punctuation mark, or a space plus a word. Models process sequences of tokens rather than characters or whole words, and usage is normally metered per token, so token counts determine both cost and how much fits in one request.
For ordinary English prose, one token averages roughly four characters, so a hundred tokens is about seventy-five words. The ratio shifts considerably with content. Code, unusual names, long numbers, and languages that are underrepresented in the training data all split into more tokens per unit of meaning, sometimes several times more than an equivalent English sentence would need.
Every limit and price in a model API is expressed in tokens. Context windows are token counts, output caps are token counts, and billing is typically split between input tokens and output tokens at different rates. Estimating in words rather than tokens is a common source of surprise when a long document silently exceeds a limit or a bill lands higher than expected.
Because tokens are fragments rather than characters, models can be oddly weak at character-level tasks such as counting the letters in a word, reversing a string, or judging rhyme. The model sees an opaque chunk, not the letters inside it. Errors on long numbers have a similar cause, since digits get grouped into tokens inconsistently.
Different model families use different tokenizers, so the same paragraph does not cost the same number of tokens everywhere. Comparing per-token prices across providers without accounting for tokenizer differences overstates the precision of the comparison. Counting with the tokenizer belonging to the exact model in use is the only reliable method, and most providers publish a tool for it.
Key points
- Roughly four characters of English text per token on average.
- Context limits, output caps, and metering are all counted in tokens.
- Code and non-English text usually consume more tokens per meaning.
- Character-level tasks are hard because letters hide inside tokens.
- Tokenizers differ by model family, so counts are not portable.
In practice
The phrase 'unbelievable results' looks like two words to a reader. A typical tokenizer might cut it into 'un', 'bel', 'ievable', and ' results', four tokens rather than two, because the whole word is rarer than its pieces. Ask a model how many times the letter 'e' appears in that phrase and errors become likely, since it never sees the individual letters.