What is Max Tokens?
Also called max output tokens, output limit.
Max tokens is a request parameter that caps how many tokens a model may generate in a single response. Reaching the cap stops generation abruptly, potentially mid-sentence, and the response is reported as truncated by length. It bounds output cost and latency but is distinct from the model's total context window.
The parameter limits output only. The context window is the combined budget for input and output, so the effective maximum output is the window minus everything already in the prompt. Requesting more output tokens than the remaining budget allows produces an error in some interfaces and silent clipping in others, which makes explicit prompt accounting worthwhile on long inputs.
Truncation at the cap is mechanical, not graceful. The model does not compress or wrap up when it approaches the limit; it simply stops mid-token-stream. Structured outputs are especially vulnerable, since a cut-off payload is often unparseable, and downstream code that assumes well-formed data will fail rather than degrade.
The parameter serves as a cost and latency guard. Since usage is metered per token and generation time scales with output length, a cap bounds the worst case for a runaway or looping response. Setting it far above realistic need weakens that guard; setting it too tightly truncates legitimate answers, so calibration against observed output lengths is the practical approach.
Length is better controlled through instruction than through the cap. Asking for a specific length in the prompt makes the model plan a complete answer of roughly that size, while the cap only chops. Sensible practice is to state the desired length in the prompt, set the cap above it as a safety bound, and always check the finish reason.
Key points
- Caps generated tokens, not total context usage
- Truncation is abrupt and can break structured output
- Bounds worst-case cost and latency per request
- Prompt instructions control length better than the cap does
- Always inspect the finish reason for length truncation
In practice
A service generates product descriptions and sets the output cap generously to avoid clipping. One malformed input sends the model into a repetition loop that runs to the full cap on every request, multiplying token usage across thousands of calls before anyone notices. After the incident the team lowers the cap to slightly above the longest legitimate description they have seen and alerts whenever the finish reason indicates a length cutoff.