# What is Model Routing?

Also called model cascade, LLM router.

Model routing directs each request to a suitable model instead of sending everything to one. A router may classify difficulty, apply rules by task type, or try a cheap model first and escalate when confidence is low. The goal is to cut cost and latency on easy requests while preserving quality on the hard ones.

Implementations range from trivial to elaborate. Static rules map a task type to a chosen model, a small classifier predicts difficulty from the request text, and a cascade attempts a cheap model first then escalates on a low confidence score, a failed schema validation, or an explicit refusal to answer. Cascades are the most common, because escalation triggers are measurable.

Savings can be substantial because request mixes are usually heavily skewed. In many applications the large majority of traffic is routine classification, extraction, or formatting that a small model handles correctly, while a minority genuinely requires a frontier model. Routing captures that skew instead of pricing every request as though it were the hardest one.

The dangerous failure mode is a router that is wrong in the expensive direction. Sending a hard request to a weak model produces a confident wrong answer that no downstream check catches, which costs far more than the savings it earned. Sound designs bias toward escalation, log every routing decision, and measure end-to-end quality rather than per-model accuracy.

Routing adds operational surface area. Each model has its own tokenizer, prompt conventions, and formatting habits, so a prompt tuned for one target may underperform on another and must be tested separately per target. Provider-side automatic routing exists too, but it reduces the caller's control over which model actually produced a given answer.

## Key points

- Sends each request to a suitable model instead of one model for everything.
- Cascades try a cheap model first and escalate on low confidence.
- Works because most traffic is easier than the hardest case.
- Bias toward escalation; a wrong cheap answer costs more than it saves.
- Prompts need testing per target model, since conventions differ.

## In practice

A document tool receives ten thousand requests a day. Roughly nine thousand ask which category a page belongs to, and a small model answers those correctly and quickly. The remaining thousand ask for comparisons across many pages, so the router sends them to a frontier model. Any small-model answer that fails schema validation is automatically retried on the larger one.

## Related terms

- [Small Language Model](/en/glossary/small-language-model)
- [Inference](/en/glossary/inference)
- [Reasoning Effort](/en/glossary/reasoning-effort)
- [Benchmark](/en/glossary/benchmark)
- [Quantization](/en/glossary/quantization)

[Back to the AI Glossary](/en/glossary)