# What is Throttling?

Also called request throttling.

Throttling is the deliberate slowing or deferral of work to keep a system within a safe operating range. Rather than refusing requests outright, a throttled system delays them, processes them at a reduced rate, or moves them to a lower priority lane. It trades latency for stability and cost control.

The distinction from outright rejection is the key one. Rejection frees resources immediately and gives the caller a clear signal, but discards work. Throttling preserves the work at the cost of holding it, which is appropriate when the work is valuable, the caller is patient, and the overload is expected to be brief. Holding too much work for too long simply converts a rejection problem into a queue problem.

Common mechanisms include token bucket schedulers that release work at a target rate, concurrency limiters that cap simultaneous execution, delay injection that spaces out calls, and priority lanes that let interactive traffic proceed while background traffic waits. Adaptive variants adjust the rate based on observed latency or error rates from the downstream dependency, tightening automatically as it degrades.

Throttling is frequently applied to the outbound side of an AI pipeline. Model providers publish limits on requests and tokens per interval, and exceeding them produces errors that waste both time and, in some billing arrangements, partial work. A client-side throttle that paces calls below the published ceiling converts a stream of failures into slightly slower but reliable progress, which is usually the better outcome.

Selective throttling by workload class is what makes the technique acceptable to users. Slowing bulk backfills, scheduled autonomous runs, and low-priority batch jobs while leaving interactive sessions untouched keeps perceived quality intact during pressure. This requires the system to know the class of each unit of work, which is a design decision that must be made before the pressure arrives.

## Key points

- Delays or paces work rather than rejecting it outright
- Implemented with token buckets, concurrency caps, or priority lanes
- Adaptive throttles react to downstream latency and errors
- Client-side pacing avoids provider limit errors on AI calls
- Throttle background classes first to protect interactive traffic

## In practice

A content pipeline runs thousands of generation jobs nightly. A token bucket paces outbound model calls to stay under the provider's per-minute ceiling, and a concurrency cap of 20 keeps memory bounded. When the provider's latency rises, the adaptive controller halves the release rate. The batch finishes 40 minutes later than usual with no failed calls, instead of failing a third of them.

## Related terms

- [Rate limit](/en/glossary/rate-limit)
- [Backpressure](/en/glossary/backpressure)
- [Queue](/en/glossary/queue)
- [Quota](/en/glossary/quota)
- [Concurrency](/en/glossary/concurrency)

[Back to the AI Glossary](/en/glossary)