LLMOps is the set of practices for running applications built on large language models in production: managing prompts and model versions, evaluating output quality, tracking cost and latency, handling provider changes and keeping the whole thing reliable. It keeps the model layer healthy. It does not, by itself, govern what an agent built on that layer is allowed to do.
Why does LLMOps matter?
Language model applications break in ways ordinary software does not. A provider updates a model and answers shift. A prompt tweak improves one case and worsens ten others. Costs climb because a retrieval step pulls in too much context. Without disciplined versioning, evaluation and monitoring, teams cannot tell why quality changed or roll back to a version that worked.
For agents, LLMOps is the foundation. An agent whose model layer is unstable cannot be certified meaningfully, because the evidence describes a system that keeps moving underneath it.
What does LLMOps cover in practice?
- Prompt and brief versioning: every instruction change is tracked and can be reverted.
- Model management: which model and version each application uses, and how changes are tested.
- Evaluation: test sets and scoring to compare versions before release.
- Retrieval: quality and freshness of the documents fed to the model.
- Monitoring: latency, errors, token use and cost per task.
- Safety filters: input and output checks for injection, toxicity or leaked data.
- Provider resilience: fallbacks and the ability to switch providers.
What does an example look like?
A US hospital system uses a language model to draft replies to patients' billing questions. The LLMOps team pins the model version, keeps every prompt change in version control, and runs each change against a few hundred past questions before release. When the provider announces a model update, they evaluate the new version on the same set, find it shortens answers in a way that drops required billing information, and adjust the prompt before switching. Because the data includes health information, they also confirm the provider arrangement meets HIPAA requirements. The operations context is described in AI agents in US healthcare operations.
How is LLMOps different from AgentOps?
LLMOps asks whether the model application is working well. AgentOps asks whether the agent is doing the right things in the business, within its bounds, with valid authority. LLMOps measures answer quality and latency; AgentOps tracks verdicts, held queues, certificates and who approved what. An agent can pass every LLMOps check and still be acting outside its role, because LLMOps does not look at the role.
The two meet at change. A model version change is an LLMOps event and also an AgentOps event: the agent's certificate no longer describes the agent that is running, and it needs fresh evidence.
Where does it matter most?
Wherever model behaviour must be explainable after the fact. In US public sector work, for example, an agency using a model to draft responses to benefits enquiries needs to show which model and prompt produced a given answer, months later. That is a versioning and record-keeping task as much as a technical one; the operating context is in AI agents in US public sector operations.
Which terms sit close to it?
MLOps: the older discipline for predictive models, from which LLMOps grew. Model drift: a change in behaviour LLMOps should detect. AI observability: the visibility LLMOps relies on. See our guide to deploying AI agents for how the layers fit together.
Questions people ask
Does LLMOps involve retraining models?
Rarely. Most LLM applications use a provider's model as it is and change prompts, retrieval and configuration instead. Fine-tuning is possible but is the exception, and it brings its own versioning and testing burden.
Who usually owns LLMOps?
An engineering or platform team. That is appropriate for the model layer, but ownership of what agents do in the business should sit with the operation the agents serve.