# What is Rate limit?

Also called throttling, quota.

A rate limit caps how many requests or how much usage a caller may consume in a time window, for example requests per minute or tokens per minute. Limits protect shared capacity, enforce fair access, and bound cost. Callers that exceed a limit typically receive a rejection response that indicates when the request may be retried.

Limits are commonly implemented with a token bucket, which allows short bursts up to a stored allowance and refills at a fixed rate, or with a sliding window that counts requests over the recent past. A service may enforce several limits at once, for instance on requests per minute, on tokens per minute, and on concurrent requests, and any one of them can be the binding constraint.

Limits appear on both sides of a system. External providers impose them, so client code must expect rejection and slow itself down. A system serving many customers also needs its own inbound limits, so that one automation cannot consume the shared capacity that everyone else depends on. Published limits also make behavior predictable for the people integrating with it.

The damaging mistake is retrying a rejection immediately, which increases the load exactly when capacity is short and can keep a service pinned in failure long after the original burst. Another is treating a limit response as a bug rather than as normal backpressure to be absorbed. It is also easy to forget that request limits and token limits are separate, so a system can be within one while exceeding the other.

Rate limits are the external constraint that concurrency controls, queues, and backoff policies are tuned against. A client-side limiter that paces requests below the ceiling generally outperforms one that sprints and retries. When rejections persist rather than resolving, a circuit breaker is the appropriate next layer, since continuing to call a saturated dependency helps nobody.

## Key points

- Caps requests or usage per time window to protect shared capacity.
- Token bucket and sliding window are the common algorithms.
- Rejections signal backpressure, so retry with backoff, never immediately.
- Model providers often limit requests and tokens with separate counters.

## In practice

An integration allows sixty requests per minute per account. A sync job fires two hundred at once, receives one hundred and forty rejections, and retries them straight away, which produces another wave of rejections. Adding a client-side limiter that releases one request per second removes the rejections entirely, and the sync finishes sooner than it did with retries.

## Related terms

- [Exponential backoff](/en/glossary/exponential-backoff)
- [Retry policy](/en/glossary/retry-policy)
- [Concurrency](/en/glossary/concurrency)
- [Queue](/en/glossary/queue)
- [Circuit Breaker](/en/glossary/circuit-breaker)

[Back to the AI Glossary](/en/glossary)