# What is Rate Limiting?

Also called rate limit, throttling.

Rate limiting is a control that caps how many requests a client may make to a service within a period of time. It protects capacity, contains cost, and prevents one caller from degrading service for everyone else. Callers that exceed the limit receive a rejection, conventionally an HTTP 429 status, often accompanied by guidance on when to retry.

Several algorithms are common. A fixed window counts requests per calendar interval and is simple but allows bursts across a boundary. A sliding window smooths that edge. A token bucket refills an allowance steadily and permits controlled bursts, which is why it is the usual choice for public services. Limits may be applied per key, per user, per address, or per endpoint, often several at once.

Model providers usually enforce two limits simultaneously, one on requests per minute and one on tokens per minute, so a few very long prompts can exhaust an allowance that many short ones would not. Concurrency caps are common as well. Agent loops make this acute, because a single user request can fan out into many model and tool calls within seconds.

Correct client behavior is to read any retry guidance in the response, wait, and use exponential backoff with random jitter so that many clients do not retry in unison. Retrying immediately makes congestion worse for everyone. Queueing work, batching requests, and caching repeated results usually help more than asking the provider for a higher limit.

When building a service, publish the limits, return remaining allowance in response headers, and make the rejection clearly distinguishable from other errors so clients can react correctly. Silently dropping traffic or returning a generic failure teaches callers to retry blindly, which is exactly the behavior a limit exists to prevent.

## Key points

- Caps requests from a client over a defined time window.
- Token bucket, fixed window, and sliding window are the common algorithms.
- Model APIs often limit requests and tokens at the same time.
- Back off exponentially with jitter, and never retry immediately.
- Publish limits and make rejections clearly distinguishable.

## In practice

An agent processing a queue of five hundred documents fires calls as fast as it can and starts receiving 429 responses after forty. Rewritten, it runs five documents at a time, reads the retry guidance on any rejection, waits that long, and doubles its wait after repeated failures with a small random offset. The batch finishes slightly slower than the theoretical best and without losing a single document.

## Related terms

- [API](/en/glossary/api)
- [REST API](/en/glossary/rest-api)
- [Idempotency](/en/glossary/idempotency)
- [Webhook](/en/glossary/webhook)
- [SDK](/en/glossary/sdk)

[Back to the AI Glossary](/en/glossary)