LLMOps is the practice of running apps that call language models. Prompt versioning, evals, cost and latency budgets, and the OpenTelemetry signals behind them.

What Is LLMOps?

LLMOps (large language model operations) is the practice of running software whose behavior depends on a model you did not train, cannot pin, and pay for by the token. Your application code is deterministic. The dependency is not. That mismatch is why the term exists at all: the things you version and the things you measure are both different from a normal service.

This article covers what LLMOps includes, where MLOps (machine learning operations) habits stop transferring, and what the loop looks like once it lands in OpenTelemetry.

What LLMOps actually covers

Start with a concrete failure, because the abstract version of this teaches nothing.

A support team ships a ticket summarizer. Someone complains the tone is too curt, so an engineer edits the system prompt, adds two sentences of guidance, and deploys. Nothing breaks. Two weeks later, p95 on the summarize endpoint has doubled, the monthly model bill is up 40%, and a customer reports that a summary invented a refund policy. Nobody can say which of the four changes shipped in those two weeks caused any of it. The prompt lives in a Python string, nothing recorded which version produced which output, and no score is attached to any completion.

Every part of LLMOps answers some version of that story:

The prompt is a named, versioned artifact, not a string literal.
The context you assemble (retrieved documents, tool results, conversation history) is recorded next to the call, because context is usually what actually changed when quality drops.
Model choice is a routing decision with a fallback, not a hardcoded model ID.
Output quality is measured continuously, because there is no exception to catch when a model is confidently wrong.

The loop is short. Change a prompt, model, or retrieval step, then compare cost, latency, and quality per version before the change becomes the default.

Where MLOps habits stop transferring

LLMOps is usually filed under MLOps, and that's roughly right. Four practices break when you carry them across.

There is no training run to reproduce. The reproducibility unit is the prompt, its version, the sampling parameters, and the exact model snapshot. Note that gpt-4.1-mini is an alias which moves under you. The snapshot recorded in gen_ai.response.model is what actually served the request, and it's the value worth alerting on.

In classical ML the expensive event happens once, offline, during training. A hosted model bills per request instead, and the distribution has a long tail: one user pasting a 40-page contract into a summarizer can cost more than a thousand ordinary requests. Averages bury that, so token counts belong in histograms, broken down by model, prompt version, and tenant.

Correctness has no label, so it becomes a score. There is no ground truth to compare against, which is why evaluations exist: heuristic checks, a model grading another model's output, human review on a sample. A quality regression looks like a groundedness pass rate falling from 0.94 to 0.88, not a failed assertion.

Streaming changes what latency even means. Total span duration is the wrong number for a streaming UI; time to first chunk is what the user feels, and per-chunk time tells you whether generation got slower or the answer just got longer. Most p95 duration regressions on LLM endpoints turn out to be longer outputs, not a slower model.

What the loop looks like in telemetry

OpenTelemetry's GenAI (generative AI) semantic conventions cover this, and it's worth knowing where they live now. On 12 June 2026, semantic-conventions v1.42.0 deprecated every gen_ai.* definition in the main repository and moved them to open-telemetry/semantic-conventions-genai. That was an organizational split, not a promotion to stable. Every GenAI span, metric, and event in the new repository is still marked Development, so pin the version you instrument against and expect names to shift.

The shape of the data is settled enough to build on:

Each model call is a span named {gen_ai.operation.name} {gen_ai.request.model}, so chat gpt-4.1-mini, carrying the provider, the model requested and the one that answered, token counts, and finish reasons.
gen_ai.prompt.name and gen_ai.prompt.version are part of the spec and conditionally required when a named template is used. These two attributes are what make "which prompt version caused this" a question with an answer.
Metrics include the gen_ai.client.token.usage and gen_ai.client.operation.duration histograms, plus gen_ai.client.operation.time_to_first_chunk and time_per_output_chunk for streaming. Agent workloads add gen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls, which is how you catch an agent that reaches the right answer after 30 model calls instead of 3.
Quality arrives as a log-based event, gen_ai.evaluation.result, parented to the span it grades.

Most teams skip that last one, and it's the piece that turns telemetry into an LLMOps loop. The following emits both a model span and the evaluation that grades it, using the OpenTelemetry Python SDK 1.44.0 with providers configured elsewhere in the process:

python

12345678910111213141516171819202122232425262728
from opentelemetry import trace
from opentelemetry._logs import get_logger

tracer = trace.get_tracer("support.summarizer")
logger = get_logger("support.summarizer")

with tracer.start_as_current_span("chat gpt-4.1-mini") as span:
span.set_attributes({
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "openai",
"gen_ai.request.model": "gpt-4.1-mini",
"gen_ai.response.model": "gpt-4.1-mini-2025-04-14",
"gen_ai.prompt.name": "ticket-summary",
"gen_ai.prompt.version": "4.2.0",
"gen_ai.usage.input_tokens": 812,
"gen_ai.usage.cache_read.input_tokens": 640,
"gen_ai.usage.output_tokens": 137,
"gen_ai.response.finish_reasons": ["stop"],
})

logger.emit(
event_name="gen_ai.evaluation.result",
attributes={
"gen_ai.evaluation.name": "Groundedness",
"gen_ai.evaluation.score.value": 0.82,
"gen_ai.evaluation.score.label": "pass",
},
)

Exported through a console exporter, the event carries the same trace and span IDs as the call it grades:

text

123
"trace_id": "0xd82e51a3fbe6307b313f66b72848989d",
"span_id": "0xcc840890a853658a",
"event_name": "gen_ai.evaluation.result"

That correlation is the whole point. The score sits next to the prompt version, the model snapshot, and the token count that produced it, so "did prompt 4.2.0 save tokens by dropping context the model needed?" becomes a query instead of an investigation.

You rarely write this by hand. Instrumentation already exists for the OpenAI, Anthropic, and Bedrock SDKs, and for frameworks like LangChain and Spring AI. If you self-host inference, the server side has its own conventions, covered in our vLLM walkthrough. Hand instrumentation is for the parts no library knows about: your gateway, your retrieval step, your evaluator.

Common pitfalls

Input token counts already include cached tokens. The spec states that gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_creation.input_tokens are both included in gen_ai.usage.input_tokens. Sum them and a cost dashboard overstates spend, badly on a cache-heavy workload. gen_ai.usage.reasoning.output_tokens has the same relationship to output_tokens.

gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions are the most useful attributes for debugging, and the riskiest to enable everywhere: they carry user personally identifiable information (PII), and they're large. Turn content capture on for a sampled subset or a single route, and filter at the source rather than paying to ingest and then drop.

Old attribute names outlive the spec. gen_ai.system is gone from the registry, replaced by gen_ai.provider.name. Copy a dashboard from a 2025 blog post and it will render cleanly while matching nothing.

The same blindness shows up in evaluation scores. A groundedness score of 0.82 means nothing unless you can group it by prompt version and model snapshot, and when the evaluator writes to a separate tool that doesn't propagate trace context, you end up with two datasets and no join key.

In Python, EventLogger is the wrong API now. Events are log records with event_name set, and opentelemetry.sdk._events.EventLogger has been deprecated since SDK 1.39.0 in favor of Logger. The same shift is underway for span events, which OpenTelemetry announced it is deprecating in March 2026. Most GenAI tutorials still show the old APIs.

Final thoughts

LLMOps is less exotic than the name suggests. It's the discipline you'd apply to any dependency you don't control, aimed at one that happens to be expensive, slow, and occasionally wrong in fluent prose. The thing that makes it tractable is refusing to let model telemetry live in its own tool. A prompt regression is often a retrieval regression, and a retrieval regression is a slow database query, a cache miss, or a timeout, all of which are already in your traces.

Dash0 ingests GenAI spans, metrics, and evaluation events over OTLP (the OpenTelemetry protocol) alongside everything else, so a model call sits in the same trace as the HTTP request, database query, and logs around it. Token usage and cost break down by model, prompt version, and tenant using the same metrics queries you already write. For the observability layer specifically, see what LLM observability measures, and for the attribute naming rules underneath it all, OpenTelemetry semantic conventions explained. Start a free trial to see your model calls, token costs, and evaluation scores in one view. No credit card required.