Operational practice for shipping and running software. DevOps: deterministic code. MLOps: trained models (add data + retraining + drift). LLMOps: mostly-third-party foundation models used via prompts, RAG, and agents.
How it works
DevOps focuses on the continuous integration and continuous deployment (CI/CD) of deterministic code. It involves setting up automated tests to monitor whether the software behaves as expected, with clear pass/fail indicators. In contrast, MLOps introduces additional complexities by incorporating data pipelines, feature stores, model registries, and mechanisms for retraining models when data changes or performance drifts. The artifact here is a model that you have trained, which requires constant attention to the underlying data and its quality.
LLMOps takes this a step further by recognizing that you often work with pre-trained foundation models rather than training your own from scratch. The focus shifts to the iterative development of Prompts, context, and eval sets. Testing in this domain is inherently probabilistic; instead of relying on traditional unit tests, you evaluate outcomes through LLMs acting as judges, using evaluation suites that assess the quality of responses rather than checking for exact matches. This introduces new metrics, such as cost, latency, and token usage, which become critical for operational success.
As you work with LLMOps, you also encounter unique risks that are not present in traditional software development. Issues like hallucination, where the model generates plausible but incorrect information, prompt injection attacks that manipulate the model's behavior, and the non-deterministic nature of outputs complicate the operational landscape. These factors require new strategies for monitoring and ensuring quality, making LLMOps a distinct practice within the broader landscape of software operations.
- Use DevOps for deterministic code.
- Use MLOps for model training and data management.
- Use LLMOps when prompts and RAG are involved.
- Monitor cost and latency in LLMOps.
- Evaluate outputs probabilistically in LLMOps.
The software-engineering analogue
You can think of LLMOps as an extension of DevOps, but with progressively looser guarantees on the outputs. In traditional software, you can expect deterministic results from your code, where passing tests confirm correctness. With LLMOps, however, the outputs are often non-deterministic and open-ended. This means that what constitutes a 'correct' output is not a single answer, but rather a distribution of possible responses that can be evaluated based on quality metrics.
Why not the obvious alternative
A common mistake is to treat an LLM application like a standard DevOps project. This approach breaks down because you cannot simply diff prompt changes and expect exact-match tests to work. The variability in responses means that traditional testing methods fall short. Ignoring the cost associated with each request can lead to unexpected expenses, and failing to implement safeguards against prompt injection or drift can compromise the integrity of the application.
When you don't control or version the model, you face challenges that require a different mindset. The need for strong monitoring and evaluation frameworks becomes clear, as you must account for the unpredictable nature of LLM outputs. This is why LLMOps practices are essential; they provide the necessary structure to manage the unique complexities of working with large language models.
When to use which
In practice, all three operational frameworks coexist and complement each other. Your application will still require the foundational elements of DevOps for managing code and infrastructure. If you are training or fine-tuning models, MLOps becomes essential for handling data, model registries, and retraining processes. However, the moment you start integrating prompts, retrieval-augmented generation (RAG), or agents into your workflow, LLMOps practices become mandatory. This includes implementing eval-driven CI, tracing outputs, monitoring costs, and establishing guardrails to mitigate risks.
In production: when it breaks
Why it happens. This delay can occur due to inefficient prompt engineering or suboptimal retrieval strategies. If the prompts are poorly designed or the retrieval system is not effectively sourcing relevant information, the LLM may struggle to generate timely and accurate responses, resulting in increased latency.
- Check the average response time of the LLM under different load conditions.
- Log the prompt length and complexity to identify any correlations with response time.
- Analyze the retrieval process to see if relevant documents are being fetched efficiently.
What it delivers
Ultimately, adopting these frameworks gives you a clear mental model of the operational changes that come with working in the LLM space. You gain insights into managing probabilistic quality, understanding token economics, and addressing new attack surfaces. This understanding helps you avoid applying the wrong playbook from traditional software development, ensuring that you navigate the complexities of LLM applications effectively.