# What is Backpressure?

Also called flow control.

Backpressure is the mechanism by which an overloaded component signals upstream producers to slow down or stop sending work. Instead of accepting more than it can process and collapsing, the component pushes resistance back through the pipeline. The result is degraded throughput under load rather than a cascading failure.

Without backpressure, an overloaded consumer accumulates unbounded queues. Memory grows, latency rises past any useful bound, and the system eventually fails in a way that discards everything in flight, including work it could have completed. The failure mode is worse than simply refusing the excess, because the wasted capacity was spent on requests that no longer had a waiting client.

Implementations vary by transport. Bounded queues that block or reject on insertion are the simplest form. Streaming protocols use credit or window schemes where a consumer advertises how much it can accept. HTTP services signal with a status code indicating overload plus a retry hint. Connection pools and semaphores that limit concurrent work create backpressure implicitly by making callers wait for a slot.

Signals must propagate all the way to a point that can genuinely slow down, or they merely relocate the queue. If a service rejects work and the caller retries immediately without limit, load increases rather than decreases. Effective designs pair backpressure with retry limits, jittered backoff, and load shedding that drops the least valuable work rather than a random slice.

AI systems face backpressure at an unusual boundary: the model provider. Provider rate limits are themselves a backpressure signal, and the correct response is to slow the local pipeline rather than to spin retries. Long generation times mean a single worker holds capacity for seconds, so a modest arrival rate can saturate a pool quickly, making bounded queues and explicit concurrency caps more important than in typical request-response services.

## Key points

- Overloaded consumers signal producers to slow or stop
- Prevents unbounded queues, memory growth, and cascading failure
- Implemented via bounded queues, credit schemes, or overload responses
- Useless if callers retry immediately without limits
- Provider rate limits act as backpressure on AI pipelines

## In practice

An ingestion service feeds documents to a summarization worker pool. The queue is bounded at 500 items; when full, the ingestion endpoint returns an overload status with a retry hint instead of accepting more. During a bulk upload the endpoint starts rejecting after ninety seconds, the uploader slows automatically, and worker latency stays near normal instead of climbing past the client timeout.

## Related terms

- [Queue](/en/glossary/queue)
- [Rate limit](/en/glossary/rate-limit)
- [Throughput](/en/glossary/throughput)
- [Concurrency](/en/glossary/concurrency)
- [Graceful Degradation](/en/glossary/graceful-degradation)

[Back to the AI Glossary](/en/glossary)