What is LLMOps?
Also called large language model operations, GenAIOps.
LLMOps is the practice of deploying, monitoring, and maintaining applications built on large language models once they serve real traffic. It covers prompt and configuration versioning, evaluation, cost and token tracking, latency monitoring, safety filtering, and incident response. The term borrows from MLOps and DevOps, and its boundaries are loosely defined and vary between teams and vendors.
In practice LLMOps is a set of feedback loops around a system that is not deterministic. Every request is recorded with its inputs, model configuration, tool calls, outputs, token counts, and duration. Those records feed dashboards, evaluation suites, and alerts. When behavior drifts after a prompt change or a provider update, the recorded history is what makes the change visible and reversible.
The reason it is treated as its own discipline is that the usual software assumptions do not hold. The same input can produce different output, quality is graded rather than passed or failed, the dependency is often an external provider that changes underneath the application, and every request has a variable price attached. Those four properties break testing, monitoring, and budgeting practices designed for deterministic services.
A common mistake is to treat LLMOps as a purchased tool rather than a set of habits. Buying a tracing dashboard produces data that nobody reads unless someone owns the review. A second mistake is versioning code while leaving prompts, model settings, and retrieval sources unversioned, which makes a behavior change impossible to attribute. A third is measuring only averages, when the failures live in the tail.
LLMOps sits above observability, which supplies the raw signals, and next to evaluation, which judges quality on a fixed set of cases. Observability tells an operator what happened in production, evaluation tells them whether a proposed change is better, and release practices decide how that change reaches users. Treating any one of the three as the whole discipline leaves a predictable gap.
Key points
- Applies operational discipline to systems whose outputs vary run to run.
- Covers versioning, evaluation, cost tracking, latency, safety, and incident response.
- Borrows practices from MLOps and DevOps rather than replacing them.
- Definitions differ across teams, so scope should be stated explicitly.
In practice
A support automation team changes one instruction line in an agent prompt. Because every run is logged with its prompt version, the team compares the week before and after: resolution rate falls slightly, average tokens per conversation rises, and one tool starts failing. They revert the line, confirm the metrics recover, and add an evaluation case so the same regression is caught before release next time.