What is Throughput?
Also called requests per second, tokens per second.
Throughput is the amount of work a system completes per unit of time, expressed as requests per second, tasks per hour, or tokens per second. It is a property of the system under a given load, and it is limited by the slowest shared resource. Throughput and latency are related but distinct, and improving one can worsen the other.
Throughput is measured over a window under a stated load, and it relates to latency and concurrency through Little's law: the amount of work in progress equals throughput multiplied by average time in the system. It is bounded by the tightest constraint in the path, which is often an external dependency's rate limit, a connection pool, or a single-threaded step, not the size of the compute fleet.
Capacity planning depends on it. Knowing that a pipeline sustains a given number of tasks per minute tells an operator how long a backlog takes to clear, how many workers a nightly batch needs, and when demand will exceed supply. For token-metered systems, throughput is often better expressed in tokens per minute, because that is the unit the provider limit is written in.
Two mistakes recur. One is confusing throughput with latency, and assuming that a faster individual request always means more total work completed, when batching often trades one for the other deliberately. The other is measuring at low load and extrapolating linearly. Real systems have a knee, past which added load increases queueing and error rates while completed work stops rising or falls.
Throughput, latency, and concurrency cannot be optimized independently, so a capacity decision means choosing which one to hold fixed. Queues absorb bursts that exceed short-term throughput but cannot raise the sustained ceiling. When the ceiling is imposed by a dependency's rate limit, the only remedies are a higher limit, fewer calls per unit of work, or accepting a longer completion time.
Key points
- Work completed per unit time, measured under a stated load.
- Little's law ties throughput, latency, and concurrency together.
- Batching usually raises token throughput and raises per-request latency.
- Limited by the slowest shared resource, not by average capacity.
In practice
A nightly job must enrich forty thousand records before morning. Each record takes about two seconds, so a single worker would need over twenty hours. Running twenty-five workers in parallel gives roughly twelve records per second and finishes in about an hour, until the enrichment API's rate limit caps the system at eight per second. The real ceiling is the dependency, not the worker count.