# What is Concurrency?

Also called parallelism, in-flight requests.

Concurrency is the number of units of work a system has in progress at the same time. It is a control knob rather than an outcome: raising it can increase throughput until a bottleneck is reached, after which it mainly increases queueing and latency. Concurrency limits are usually enforced per tenant, per worker pool, and per external dependency.

Concurrency is enforced with explicit limits: worker pool sizes, semaphores around a dependency, connection pool maximums, and per-tenant caps. Strictly, concurrency describes work that is in progress and interleaved, while parallelism describes work executing at the same instant on separate resources. For operational purposes the number that matters is how many requests a dependency has open at once.

Limits exist to protect things that break under pressure. An external API has its own capacity, a database has a finite connection pool, and a model provider enforces its own ceilings. Bounded concurrency converts an overload into a queue, which is recoverable, instead of a wave of failures. It also bounds spend, since a runaway loop cannot fan out indefinitely.

The most common failure in agent systems is unbounded fan-out, where a single request spawns one subtask per item in a list of unknown length. The second is having a global limit but no per-tenant limit, so one heavy workload consumes all capacity and everyone else waits. The third is raising limits to cure latency when the bottleneck is downstream, which only moves the queue.

Concurrency is the knob, throughput is the result, and latency is the cost of turning the knob too far. Queues and worker pools are where limits are usually implemented, rate limits are the external constraint they are tuned against, and per-tenant caps are how fairness is maintained when many customers share one pool.

## Key points

- Counts work in progress, distinct from work arriving or completing.
- Raising it past the bottleneck adds queueing, not throughput.
- Needs per-tenant caps so one workload cannot starve others.
- Unbounded agent fan-out is a common source of runaway concurrency.

## In practice

An agent is asked to summarize sixty documents. Without a limit it launches sixty model calls at once, immediately trips the provider rate limit, and every call fails or retries. With a concurrency limit of six, calls complete steadily, the job finishes faster overall, and the retry storm never happens. The limit made the system faster by making it do less at once.

## Related terms

- [Throughput](/en/glossary/throughput)
- [Rate limit](/en/glossary/rate-limit)
- [Queue](/en/glossary/queue)
- [Worker](/en/glossary/worker)
- [Latency](/en/glossary/latency)

[Back to the AI Glossary](/en/glossary)