What is Cold Start Latency?
Also called cold start.
Cold start latency is the extra delay incurred when a request arrives at capacity that is not yet ready to serve, requiring initialization first. The work may include starting a container, loading model weights, establishing connections, or populating caches. It affects the first requests after a scale-up, a deploy, or an idle period.
The delay is the sum of several stages, and which stage dominates varies. Container scheduling and image pull can dominate when images are large or uncached on the node. Runtime and dependency initialization dominates for heavy frameworks. Loading model weights into accelerator memory dominates for self-hosted inference and can run into tens of seconds. Warming connection pools and caches adds a tail where the first requests are slow but not failing.
The user impact is concentrated and easy to miss in averages. A small share of requests hitting cold capacity may barely move the mean while producing very poor experiences at high percentiles. Cold starts also cluster exactly when the system is scaling up under load, so the users affected are disproportionately those arriving during a traffic spike, when tolerance is lowest.
Mitigations trade cost against readiness. Keeping a warm pool of pre-initialized capacity eliminates most of the delay but pays for idle resources. Provisioning ahead of predictable demand shifts the problem where traffic is cyclical. Lighter images, lazy loading of rarely used components, and readiness probes that only admit traffic after initialization completes reduce both the duration and the visible impact.
For systems that call hosted model APIs, the classic weight-loading cold start belongs to the provider and is not directly controllable. What remains local is application initialization, connection establishment, and any local retrieval index. Teams sometimes misattribute provider-side variability to their own cold starts, so separating the timings in traces is what distinguishes a fixable problem from an upstream characteristic.
Key points
- Extra delay when capacity must initialize before serving
- Dominant stage varies: image pull, runtime init, or weight loading
- Concentrated in high percentiles and during scale-up spikes
- Warm pools remove the delay but pay for idle capacity
- Separate local init from provider-side variability in traces
In practice
A workspace service scales from two pods to eight during a morning spike. New pods take 38 seconds before passing readiness, of which 24 is pulling an image not cached on the node. Requests routed during that window see the ninety-ninth percentile latency triple. Pre-pulling the image to every node and keeping one spare pod warm cuts the readiness time to nine seconds.