The promise of MLOps in production collides with a reality that notebooks do not prepare you for: a model that performs well in offline evaluation can fail in subtle and costly ways in real production. The problem is almost never the algorithm. It is the system surrounding it: the data arriving, the code calling it, the infrastructure executing it, and the process updating it when reality changes. The difference between a team that operates ML with control and one that operates with permanent anxiety is not in the model; it is in the maturity of the system around it.
In 2026, the MLOps space has grown more complex in a specific direction: LLMs and LLM-based agents have become real production components for many organizations, adding a layer of challenge that classic MLOps frameworks had not anticipated. Probabilistic behavior, variable latency, inference costs with their own logic, and the emerging security surfaces of agentic systems mean that operating ML in 2026 requires more judgment, not just more tooling.
This guide aggregates the articles and concepts that cover that reality end to end: from how to take a prototype to a governable system, to how to choose between self-hosted and API, how to rigorously evaluate models in production, and how to govern agents that interact with critical systems.
What this guide covers
The articles below cover specific areas with technical depth. This guide acts as an entry map and starting point for teams building or reviewing their MLOps practice.
- Self-hosted LLMs with Ollama, vLLM and TGI: how to choose and deploy: technical comparison of the three main projects, with selection criteria by use case and deployment guide on Kubernetes
- MLOps in production: from prototype to a governable system: the gap between ML experimentation and production, and how to close it with the right pieces without over-engineering
- Scalable model training and evaluation: rigor before compute: how to build training and evaluation pipelines that scale without losing scientific rigor
- Multimodal embeddings: search and AI beyond text: embedding models that combine text, image, and structured data, and the use cases where that changes the outcome
- GPT-5.1 in the enterprise: adaptive capabilities and adoption criteria: what GPT-5.1 adds over previous generations and how to evaluate whether the capability jump justifies the cost
- Gemini 3.0 in the enterprise: multimodal AI for real use cases: multimodal capabilities in real enterprise contexts and the criteria that should guide adoption
- MCP in production: security and agent governance: Model Context Protocol as agent infrastructure and the security risks it introduces in systems accessing real resources
From prototype to production
The gap between a model that works in a notebook and an ML system that operates in production is larger than most teams anticipate. Not because the transition is technically impossible, but because it requires solving problems that experimentation can ignore but production cannot.
The first problem is reproducibility. A notebook is inherently an exploration artifact. Dependencies are implicit, cell execution order may be non-deterministic, and the environment state is rarely documented with enough precision. When that same code needs to run reliably on demand, the first task is converting it into something any team member can reproduce with exactly the same results. That requires explicit dependency management, data versioning, experiment tracking, and a clear separation between exploration code and production code.
The second problem is data. In experimentation, data is static. In production, data changes. Schemas evolve, distributions drift, upstream systems fail, and ingestion pipelines have their own inconsistencies. A mature ML system does not just process data correctly when everything goes well: it detects anomalies in input data before they reach the model and produces data quality metrics the team can monitor the same way they monitor latency or error rate.
The third problem is model governance itself. In experimentation, any new version is better than the previous one by definition, because it was trained with accumulated learning. In production, a new version can degrade specific use cases even if it improves global metrics. That requires shadow mode, canary releases, A/B evaluation with business metrics, and a clear approval and rollback process that does not depend on a data scientist having time to manually review logs.
Tools like MLflow, Weights & Biases, or Vertex AI Experiments solve experiment tracking. Kubeflow Pipelines, Metaflow, or Prefect solve pipeline orchestration. Feast or Tecton solve the feature store. None of them solves the problem alone, and adopting all of them at once without criteria is another way of avoiding the real problem. The sensible starting point is the smallest governable system that lets you answer these questions: what version of the model is in production right now, on what data was it trained, and what metrics justify it? If you cannot answer that in two minutes, you have MLOps debt.
LLM inference and serving
The decision to run an LLM self-hosted or consume it via API is one of the most consequential a team can make in the ML stack, and one of the most frequently made without the right data.
A provider API is always the option with lowest initial friction. There is no infrastructure to operate, the model is kept current, latency is optimized by the provider, and cost scales with usage. The problem appears when volume grows: the per-token cost of frontier models like GPT-5.1 or Gemini 3.0 can be perfectly affordable for point use cases and completely unsustainable for large-scale document processing pipelines, embedding generation at scale, or real-time inference with thousands of requests per minute.
Self-hosted inverts that tradeoff. Marginal cost per token drops dramatically once infrastructure is amortized, but operational burden rises significantly. You need to manage the model lifecycle, monitor GPU, optimize serving configuration for your load pattern, and plan upgrades when a new version you want to adopt appears.
Among serving frameworks, vLLM is currently the highest-performance option for GPU inference. Its implementation of PagedAttention allows efficient KV cache management, which translates into higher throughput and lower latency especially in long contexts. TGI (Text Generation Inference from Hugging Face) is more mature in some aspects of Hugging Face ecosystem integration and easier to get started for teams already working with that ecosystem. Ollama is the simplest option for local development and use cases where operational simplicity matters more than raw performance.
The most common decision that makes sense in organizations scaling their LLM usage is a hybrid model: frontier models via API for use cases where quality matters more than cost, and open self-hosted models for high-volume pipelines or use cases where data privacy makes it impossible to send information to a third party.
The article Self-hosted LLMs with Ollama, vLLM and TGI covers that comparison with detailed technical and operational criteria.
Model evaluation and observability
Model evaluation in production is the area where we most consistently see the difference between organizations that operate ML with control and those that operate with uncertainty. The pattern that fails systematically is this: the model is extensively evaluated before deployment and barely monitored afterward. That pattern treats deployment as a terminal event when in reality it is the beginning of the model's real lifecycle.
Drift is the most important phenomenon to monitor and the easiest to ignore because it produces no explicit errors. Data drift occurs when the distribution of production input data diverges from the distribution the model was trained on. Concept drift occurs when the relationship between inputs and the desired output changes over time, even if the input distribution remains stable. A churn prediction model trained on 2023 data may continue receiving perfectly reasonable inputs in 2026 while producing predictions that no longer correspond to market reality.
Detecting drift requires continuous statistical monitoring: PSI (Population Stability Index) for data drift, performance metrics when ground truth is available (which in many cases has latency, because the real outcome of a decision may take days or weeks to be observable), and periodic evaluation on test sets that are updated to represent the current distribution.
For LLMs, evaluation is more complex because outputs are free text. Classic metrics like accuracy do not apply directly, and text metrics like BLEU or ROUGE do not capture semantic quality. Modern evaluation pipelines for LLMs combine automatic evaluation with LLM-as-judge (using another model to evaluate output quality), sampled human evaluation, and domain-specific benchmarks. Tools like LangSmith, Arize Phoenix, or Evidently AI allow you to build those evaluation pipelines systematically.
The article Scalable model training and evaluation covers how to build those pipelines with the rigor production requires.
Embeddings and semantic search
Embeddings have moved from an advanced NLP technique to de facto infrastructure in modern AI systems. RAG (Retrieval-Augmented Generation), semantic search, recommendation, and duplicate detection are use cases that in 2026 are built with embeddings almost universally. The question is no longer whether to use embeddings, but which ones and with what retrieval model.
Multimodal embeddings add an additional dimension: models like CLIP, ImageBind, or Vertex AI Multimodal Embeddings allow representing text, images, audio, and even video in the same vector space. That enables searches that combine modalities naturally: "find images similar to this description" or "given this document, find the most visually related products." In e-commerce, content search, or digital asset management, that capability changes the outcome in a meaningful way.
The choice of embedding model has direct consequences on retrieval quality and operational cost. Models like OpenAI's text-embedding-3-large or Google's Gecko offer high quality with minimal operational overhead. Open models like E5-large, BGE-M3, or Nomic Embed offer full lifecycle control and zero inference cost once deployed, in exchange for the corresponding operational burden.
The vector database that stores and queries those embeddings also deserves deliberate choice. pgvector turns PostgreSQL into a vector database and is the lowest-friction operational option for teams already running PostgreSQL. Qdrant, Weaviate, and Milvus are native options offering better performance at scale and more hybrid filtering options. Pinecone and cloud provider managed offerings eliminate operational burden in exchange for dependency and cost.
The article Multimodal embeddings covers current models, indexing patterns, and vector database selection criteria in depth.
Agent security and governance
LLM-based agentic systems introduce a risk surface that organizations frequently underestimate until they experience the first incident. An agent that can execute code, access APIs, read and write databases, or send emails on behalf of a user has access to real resources. That means model failures: hallucinations, prompt injections, or unexpected outputs: are not just text errors; they are actions with consequences.
The OWASP LLM Top 10 documents the most frequent vulnerabilities in LLM-based applications. Prompt injection is the most critical: an attacker can introduce malicious instructions in the data the agent processes, causing it to execute unauthorized actions. In an agent accessing company resources, that can mean data exfiltration, execution of unwanted commands, or bypass of access controls.
The Model Context Protocol (MCP), which has become the emerging standard in 2025-2026 for integrating agents with external tools, adds its own risk surface. MCP servers can serve as entry points for attacks if they are not properly isolated, if the permissions they expose are excessive, or if communication between the agent and the MCP server is not properly authenticated.
Guardrails are the primary mitigation mechanism: input validation before reaching the model, output validation before executing actions, explicit limits on what tools the agent can use and under what conditions, and complete logging of all actions for audit. Tools like LlamaGuard, Guardrails AI, or custom implementations on prompt engineering with validation allow you to codify those limits systematically.
Governance in this context is not bureaucracy: it is the difference between an agentic system you can audit and one you cannot. If an agent takes an incorrect action, you need to be able to answer: what instruction reached it, what tool did it use, what arguments did it pass, and what result did it obtain? Without structured logging of agent decisions, that investigation is practically impossible.
The article MCP in production: security and agent governance covers these risks in depth and provides criteria for designing agentic systems that can be operated with control.
Decision matrix: LLM self-hosted vs API vs hybrid
The decision is not binary. In organizations with multiple LLM use cases, the answer is usually a deliberate mix. This table summarizes the main tradeoffs by criterion:
| Criterion | Self-hosted | API (provider) | Hybrid |
|---|---|---|---|
| Cost at low volume | High (fixed infrastructure) | Low (pay-per-token) | Variable |
| Cost at high volume | Low (marginal) | High | Optimizable per use case |
| Latency | Controllable and predictable | Provider-dependent | Per use case |
| Data privacy | Full | Depends on provider DPA | Segmentable by sensitivity |
| Model quality | Limited to available open models | Access to frontier models | Best of both |
| Operational burden | High (GPU, serving, upgrades) | Minimal | Medium |
| Dependency risk | None | Vendor lock-in, price changes | Reducible |
| Version control | Full | Per provider | Per component |
| Time to production | Weeks | Days | Variable |
The pattern that most frequently maximizes ROI is: frontier models via API for high-quality, low-volume use cases (content generation, complex analysis, user interaction), and open self-hosted models for high-volume pipelines, sensitive data processing, or tasks where a smaller specialized model outperforms a large general-purpose model in quality.
When to act
If your team is building on LLMs or classic ML for use cases that impact the business, and the system lacks drift monitoring, continuous evaluation, or agent decision logging, the risk is present even if it is not yet visible. The moment to establish those practices is before usage scales, not after the problem scales.
A team with production MLOps experience can help make that diagnosis, prioritize the pieces that mitigate the most risk with the least friction, and support implementation without blocking product work.
If that is your situation, the AI in production and MLOps consulting page describes how we work and what types of projects we handle.
