AI Agent Operations

Running agents in production is the discipline of keeping a nondeterministic, network bound and metered workload reliable and affordable. It borrows most of its vocabulary from ordinary distributed systems, including queues, retries, autoscaling and service level objectives, then adds concerns specific to language models: token usage, time to first token, cost per action, and the fact that one logical task may run for minutes across many tool calls. The bundle of practices around this is often called LLMOps.

It matters because agent workloads fail in ways older dashboards were not built to show. A run can succeed technically and still be useless, cost several times its usual amount, or stall on a third party interface that answers slowly instead of failing outright. If you are building, this area decides whether your system degrades gracefully or falls over. If you are buying, it is where questions about uptime, isolation and incident handling get concrete answers.

Start here

The two spines: execution and visibility

Almost every term here hangs off one of two spines. The execution spine moves work: a request enters a queue, a worker picks it up, a workflow keeps the steps in order, retries with exponential backoff absorb transient failures, and anything that keeps failing lands in a dead letter queue instead of disappearing. Durable execution is what lets a long run survive a process restart. The visibility spine describes what happened: traces group spans into one story, monitoring turns those into metrics, alerting decides what wakes someone, and objectives set the line between acceptable and not. Cost and token accounting ride along the same traces, which is why instrumenting once pays twice.

What depends on what

Order matters when building this out. Tracing comes first, because retries, rate limits and cost attribution are guesswork without a request you can follow end to end. Objectives come next, since alerting with no target produces noise nobody reads. Only then do the protective mechanisms make sense: rate limits and concurrency caps protect shared capacity, circuit breakers stop a failing dependency from consuming every worker, and whether each dependency fails open or fails closed is a decision to make deliberately rather than inherit. Release safety, meaning canary rollout and rollback, sits on top, because you need the visibility spine already working to tell whether a canary is healthy.

What people get wrong

Teams instrument latency and forget throughput, then are surprised when a system that feels fast handles very little concurrent work. They set aggressive retry policies with no budget, turning a slow dependency into a self inflicted outage. They treat uptime as the headline metric when users experience partial degradation far more often than total failure, which is what objectives and graceful degradation exist to describe. They log final outputs rather than trajectories, so when an agent takes a strange path there is no record of the intermediate steps. Cost is the newest blind spot: without per action accounting, a prompt change that doubles token usage looks free until the invoice arrives.

Every term in AI Agent Operations

Agent trajectory logging: Agent trajectory logging is the practice of recording the full ordered sequence of steps an autonomous agent took during a run, including the messages it received, the intermediate summaries it produced, the tools it called with their arguments and results, and the final outcome. The stored trajectory lets a run be replayed, audited, and evaluated after the fact.
Alerting: Alerting is the practice of notifying a responsible person or system when monitored signals indicate a condition that requires attention. An alert defines a condition, a severity, a destination, and ideally a documented response. The central design problem is precision: alerts that fire without a required response train recipients to ignore them, which is how genuine incidents get missed.
At-Least-Once Delivery: At-least-once delivery is a messaging guarantee in which every message is delivered to a consumer one or more times, never zero. The system retries until it receives an acknowledgment, which means a message whose acknowledgment is lost will be redelivered. Consumers must therefore tolerate seeing the same message more than once.
Autoscaling: Autoscaling is the automatic adjustment of computing capacity in response to observed load or a schedule. Horizontal autoscaling adds or removes instances, while vertical autoscaling changes the resources allocated to an existing instance. It matches capacity to demand without manual intervention, but reacts only after a signal appears, so it cannot absorb a spike faster than new capacity can start.
Backpressure: Backpressure is the mechanism by which an overloaded component signals upstream producers to slow down or stop sending work. Instead of accepting more than it can process and collapsing, the component pushes resistance back through the pipeline. The result is degraded throughput under load rather than a cascading failure.
Blast Radius: Blast radius is the scope of harm a failure, defect, or malicious action can reach before something stops it. It is described in terms of who and what is affected: how many users or tenants, which data, which downstream systems, and whether the effects can be reversed. Reducing it is a design goal independent of reducing failure likelihood.
Canary Release: A canary release is a deployment strategy that sends a new version to a small share of traffic first, compares its behavior against the existing version, and expands the rollout only if the metrics look healthy. It limits the number of users exposed to a defective release and provides real production signal before full exposure. Its usefulness depends entirely on having metrics good enough to detect the problem.
Circuit Breaker: A circuit breaker is a resilience pattern that stops calls to a failing dependency after errors cross a threshold, returning failures immediately instead of waiting on timeouts. After a cooling period it allows a small number of trial calls, and restores normal traffic if those succeed. The pattern protects a struggling service from load and keeps the caller responsive.
Cold Start Latency: Cold start latency is the extra delay incurred when a request arrives at capacity that is not yet ready to serve, requiring initialization first. The work may include starting a container, loading model weights, establishing connections, or populating caches. It affects the first requests after a scale-up, a deploy, or an idle period.
Compensating Transaction: A compensating transaction is an operation that semantically reverses the effect of a previously committed step, used when a multi-step process fails partway and a true rollback is impossible. Rather than erasing history, it applies a counteracting change: a refund offsets a charge, a cancellation offsets a booking, a correction notice offsets a sent message.
Concurrency: Concurrency is the number of units of work a system has in progress at the same time. It is a control knob rather than an outcome: raising it can increase throughput until a bottleneck is reached, after which it mainly increases queueing and latency. Concurrency limits are usually enforced per tenant, per worker pool, and per external dependency.
Cost Attribution: Cost attribution is the practice of assigning machine costs, such as model tokens, tool calls, and compute, to the entity that caused them: a tenant, a feature, an agent, or an individual run. It turns a single aggregate provider bill into a breakdown that can be acted upon, priced against, or optimized.
Cost per action: Cost per action is the total resource cost attributed to one completed unit of work, such as one answered ticket, one drafted document, or one enriched record. It aggregates model tokens, tool and API calls, storage, retries, and failed attempts. Because a single action often involves many model calls, cost per action is usually far higher than the cost of one call.
Cron: Cron is a time based scheduler originating in Unix, and by extension the expression syntax used to describe recurring schedules. A cron expression uses five or six fields covering minute, hour, day of month, month, day of week, and sometimes seconds. The syntax has become the common vocabulary for recurring schedules well beyond the original program.
Dead Letter Queue: A dead letter queue is a separate holding area for messages that could not be processed successfully after their retry attempts were exhausted. Moving failures aside keeps a poison message from blocking or endlessly recycling through the main queue, while preserving the payload for inspection. Items in a dead letter queue can be examined, fixed, and replayed once the underlying cause is resolved.
Durable Execution: Durable execution is a model in which a long running process survives crashes, deploys, and restarts without losing its place. The engine records the outcome of each completed step, then reconstructs the process state by replaying that history and skipping work already done. It lets code that waits minutes, days, or longer be written as ordinary sequential logic.
Error Budget: An error budget is the amount of unreliability a service is permitted over a period, derived directly from its reliability target. If the target is 99.9 percent of requests succeeding in a month, the budget is the remaining 0.1 percent. Spending within budget is normal; exhausting it triggers an agreed change in how the team operates.
Evaluation Harness: An evaluation harness is the software scaffolding that runs a set of test inputs through an AI system, collects the outputs, scores them against defined criteria, and reports aggregate results. It standardizes how quality is measured so that two runs, two prompts, or two model configurations can be compared on identical terms rather than on impressions.
Exponential backoff: Exponential backoff is a retry timing strategy in which the wait between attempts grows multiplicatively, for example one second, then two, then four. Random jitter is added so that many clients failing at once do not retry in synchronized waves. The approach gives an overloaded dependency time to recover instead of adding load during its worst moment.
Fail Open and Fail Closed: Fail open and fail closed describe the two ways a system can behave when a check or dependency is unavailable. Failing open allows the operation to proceed, favoring availability. Failing closed blocks the operation, favoring safety or security. The correct choice depends on what the failing component protects, and picking the wrong default is a recurring source of both outages and breaches.
Feature Flag: A feature flag is a runtime switch that determines whether a piece of behavior is active, letting teams change what a system does without deploying new code. Flags can be global, or evaluated per user, tenant, or request, which allows a change to be enabled for a subset of traffic and reversed immediately.
Golden Dataset: A golden dataset is a curated collection of inputs paired with agreed-upon correct outputs, used as the reference standard when evaluating an AI system. Its examples are deliberately chosen and reviewed rather than sampled at random, so that the set covers the behaviors a team has decided matter most, including known past failures.
Graceful Degradation: Graceful degradation is the practice of designing a system so that partial failures reduce functionality instead of causing a total outage. Non-essential features are disabled, stale data is served, or simpler fallbacks take over while the core path keeps working. The goal is a smaller and more predictable blast radius when a dependency becomes slow or unavailable.
Human Review Queue: A human review queue is a work list where AI outputs or pending actions wait for a person to approve, reject, or edit them before they take effect or reach a recipient. It converts an autonomous step into a supervised one for cases that are high risk, low confidence, or subject to policy requirements.
Incident: An incident is an unplanned disruption or degradation of a service that requires a coordinated response. Handling one typically follows a sequence of detection, triage, mitigation, resolution, and review. The review, often called a postmortem, documents the timeline and contributing causes so that improvements target the conditions that allowed the failure rather than the individuals involved.
Kill Switch: A kill switch is a single control that halts a system's activity immediately, used when continued operation is causing harm. Unlike a gradual rollback, it prioritizes stopping over preserving in-flight work. In agent systems it typically stops new runs, cancels running ones, and blocks pending actions from executing.
Latency: Latency is the elapsed time between a request being issued and its response being complete. It is reported as a distribution rather than a single number, usually with percentiles such as the median, the ninety-fifth, and the ninety-ninth. In agent systems the total is dominated by the number of sequential model and tool calls rather than by raw computation.
LLM as Judge: LLM as judge is an evaluation technique in which a language model scores the output of another system against a written rubric, standing in for a human reviewer. It makes open-ended qualities such as helpfulness, tone, or faithfulness measurable at scale, and is used where no exact reference answer exists.
LLMOps: LLMOps is the practice of deploying, monitoring, and maintaining applications built on large language models once they serve real traffic. It covers prompt and configuration versioning, evaluation, cost and token tracking, latency monitoring, safety filtering, and incident response. The term borrows from MLOps and DevOps, and its boundaries are loosely defined and vary between teams and vendors.
Mean Time to Recovery: Mean time to recovery is the average elapsed time between the start of a service impairment and the restoration of normal service. It is measured across a set of incidents over a period and is used to characterize how quickly an organization contains failures, as distinct from how often failures occur.
Monitoring: Monitoring is the continuous collection and evaluation of signals about a running system to determine whether it is behaving as expected. It focuses on known indicators such as error rate, latency, saturation, and traffic, checked against defined thresholds. Monitoring answers whether something is wrong, while broader observability practice is concerned with explaining why.
Multi-Tenancy: Multi-tenancy is an architecture in which one deployment of a system serves many independent customers, called tenants, whose data and activity must remain separated. Isolation can be enforced at the row, schema, database, or infrastructure level, with stronger separation costing more to operate. Every query, cache key, background job, and log line must carry tenant context or isolation fails.
Observability: Observability is the degree to which the internal state of a running system can be understood from the data it emits, mainly logs, metrics, and traces. A system is observable when an operator can answer new questions about a failure without shipping new code to collect more data. It is a property of the system, not a single tool.
Offline Evaluation: Offline evaluation measures the quality of an AI system against a fixed, pre-collected dataset, without exposing any real user to the version being tested. Because inputs and scoring criteria are held constant, it isolates the effect of a change to the prompt, model, or code. It is the standard gate before a change reaches production traffic.
Online Evaluation: Online evaluation measures the quality of an AI system using real production traffic as it happens, rather than a stored test set. Signals come from user behavior, explicit ratings, downstream outcomes, and automated scoring applied to live outputs. It captures the real input distribution that a fixed dataset cannot represent.
Orchestration Engine: An orchestration engine is infrastructure that coordinates multi step processes, deciding what runs next, tracking each step's state, and handling retries, timeouts, and recovery after a crash. It separates the definition of a process from the machinery that reliably executes it. Common examples include workflow engines for business processes, job schedulers for data pipelines, and coordinators for multi step agent runs.
Poison Message: A poison message is a queued item that a consumer cannot process successfully no matter how many times it retries, because the failure is caused by the message itself rather than by a transient condition. Left unhandled, it is redelivered indefinitely, consuming capacity and blocking progress for other work.
Postmortem: A postmortem is a structured written review conducted after an incident, recording what happened, the timeline, the contributing causes, the impact, and the specific changes that will reduce recurrence. Its output is a durable document and a set of owned action items, not a verdict on individual performance.
Prompt Cache Hit Rate: Prompt cache hit rate is the proportion of input tokens served from a provider's cached prefix rather than processed fresh. Many model providers cache the computed state of a repeated prompt prefix and meter those tokens differently from new ones. A higher hit rate typically reduces both latency and metered input cost.
Queue: A queue is a buffer that holds units of work between the component that produces them and the components that process them. It decouples arrival rate from processing rate, so bursts are absorbed rather than dropped, and it allows work to be retried or redistributed if a processor fails. Queues are a foundational building block for background jobs and asynchronous agent execution.
Quota: A quota is a fixed allowance of a resource granted to an account, tenant, or workload over a defined period, such as requests per day, tokens per month, or concurrent runs. Once consumed, further use is refused or degraded until the period resets. Quotas bound cost and enforce fair sharing between tenants.
Rate limit: A rate limit caps how many requests or how much usage a caller may consume in a time window, for example requests per minute or tokens per minute. Limits protect shared capacity, enforce fair access, and bound cost. Callers that exceed a limit typically receive a rejection response that indicates when the request may be retried.
Retry policy: A retry policy defines when a failed operation is attempted again, how many times, how long to wait between attempts, and which failures are eligible. Transient failures such as timeouts and rate limit rejections are usually retried, while validation errors and permission denials are not. Without a bounded policy, retries can multiply load and turn a small fault into an outage.
Rollback: A rollback is the act of returning a system to a previously known good version after a change causes problems. It is the primary mitigation during deployment related incidents because it restores service without requiring the cause to be understood first. Rollback is only genuinely available when every change in the release is reversible, which database migrations and irreversible side effects can prevent.
Runbook: A runbook is a written procedure for handling a specific operational situation, listing the steps to diagnose and resolve it in order. It exists so that a responder who did not build the system can act correctly under time pressure, without reconstructing knowledge that someone already has.
Scheduled Job: A scheduled job is work that runs automatically at defined times or intervals rather than in response to a user request. Typical uses include generating reports, synchronizing data, sending digests, cleaning up expired records, and triggering recurring agent tasks. Because no person is waiting on the result, failures are easy to miss unless the schedule is explicitly monitored.
Service Level Objective: A service level objective is a target value for a measured indicator of service quality over a defined period, such as a percentage of requests served successfully within a latency threshold each month. The indicator itself is the service level indicator, and the gap between the target and perfection is the error budget. Objectives convert vague reliability goals into numbers that guide decisions.
Shadow Deployment: A shadow deployment runs a new version of a system on real production traffic while discarding its output, so users continue to receive results from the existing version. It reveals how the candidate behaves on true inputs without exposing anyone to its mistakes. The two sets of outputs are then compared offline.
Span: A span is a single timed operation inside a trace, such as one model call, one database query, or one tool invocation. It records a start time, a duration, a name, a status, and structured attributes, and it points to its parent span. Spans nested under one another form the tree that makes up a complete trace.
Synthetic Monitoring: Synthetic monitoring runs scripted transactions against a system on a schedule, from outside it, to verify that critical paths still work. Because the checks run continuously whether or not real users are active, they detect breakage during quiet periods and cover flows that real traffic exercises too rarely to reveal a problem promptly.
Throttling: Throttling is the deliberate slowing or deferral of work to keep a system within a safe operating range. Rather than refusing requests outright, a throttled system delays them, processes them at a reduced rate, or moves them to a lower priority lane. It trades latency for stability and cost control.
Throughput: Throughput is the amount of work a system completes per unit of time, expressed as requests per second, tasks per hour, or tokens per second. It is a property of the system under a given load, and it is limited by the slowest shared resource. Throughput and latency are related but distinct, and improving one can worsen the other.
Time to first token: Time to first token is the delay between sending a request to a language model and receiving the first piece of the streamed response. It captures queueing, prompt processing, and any preliminary work, but not the time spent generating the rest of the output. It is the main determinant of how responsive a streaming interface feels.
Token usage: Token usage is the count of tokens a language model reads and writes for a request, normally reported as separate input and output totals. Tokens are subword units, so counts depend on the tokenizer as well as on text length. Usage is the main driver of both cost and latency in model-backed systems, and it is usually metered per request.
Tracing: Tracing records the path of a single request or task through a system as a tree of timed operations. Each trace carries an identifier that links every step, so a slow or failed run can be inspected end to end. In agent systems a trace typically covers model calls, tool calls, retries, and any handoffs between components.
Uptime: Uptime is the proportion of a period during which a system is available for use, usually stated as a percentage. It is commonly expressed in nines, where ninety nine point nine percent allows roughly forty three minutes of downtime per month. The number is only meaningful alongside its definition, since what counts as available and where it is measured vary widely.
Webhook Delivery: Webhook delivery is the mechanism by which one system notifies another of an event by sending an HTTP request to a receiver supplied URL. It replaces polling with push, so the receiver learns about changes promptly. Reliable delivery requires signature verification, retries with backoff, duplicate tolerance on the receiving side, and a record of attempts that can be inspected and replayed.
Worker: A worker is a process that pulls units of work from a queue or schedule and executes them outside the request and response path. Workers are typically run as an interchangeable pool, so capacity is adjusted by changing how many exist and how many items each handles at once. They are the standard place to run long jobs such as agent runs, report generation, and data synchronization.
Workflow: A workflow is a defined sequence of steps that accomplishes a process, including the order of steps, the conditions that route between them, and the handling of failures. It can be expressed as code, as a declarative graph, or as a visual diagram. Making a process an explicit workflow gives it a stable identity that can be versioned, executed repeatedly, monitored, and audited.

Back to the AI Glossary

AI Agent Operations

Start here

The two spines: execution and visibility

What depends on what

What people get wrong

Commonly confused

Monitoring vs Observability

Latency vs Time to first token

Retry policy vs Exponential backoff

Fail Open and Fail Closed vs Graceful Degradation

Every term in AI Agent Operations