LLM evaluation is how you find out whether your model's output is actually good, not just fast and well-formed. A completion can come back in 400ms with valid JSON and still be confidently wrong, and none of your existing alerts will catch that, because an HTTP 200 and a passing schema check have nothing to say about whether the answer is true.
That's the mechanical problem underneath all of this: normal software testing checks for one correct output per input, and LLMs don't have a single correct output. Run the same prompt twice and you can get two different, both defensible, answers. Evaluation is the discipline of scoring "how good" instead of asserting "was it exactly this," and it works differently depending on whether you're running it before you ship a change or against live production traffic.
This article covers the methods that actually hold up, how to score production traffic without evaluating every request, and where OpenTelemetry now has a real answer for tying an evaluation score back to the exact call that produced it.
LLM evaluation vs. LLM observability
These two terms get used interchangeably and they shouldn't be. LLM observability is the full picture: latency, token cost, error rates, and trace-level visibility into every model and tool call in a request. Evaluation is one layer inside that picture, specifically the one that answers "was this output good," which none of your operational metrics can tell you. A service can have perfect p99 latency and a 0% error rate while quietly hallucinating in every third response. Evaluation is what surfaces that.
Evaluate before you ship, and again after
Most teams need evaluation at two different moments, and they catch different problems.
Offline evaluation runs against a curated dataset of known-good inputs and expected outputs, before a prompt change, model swap, or fine-tune goes live. It's a regression test: does the new version still get the cases right that the old version got right? This is where you catch a prompt edit that fixes one failure mode and quietly breaks three others.
Online evaluation runs against real production traffic, continuously, after you've shipped. It catches what your offline dataset never anticipated: the phrasing a real user tried, the edge case in your retrieval index, the slow drift in output quality after an upstream model provider pushes a silent update to the model version behind your API alias.
You need both. Offline evaluation without online evaluation means you're blind to drift. Online evaluation without offline evaluation means every prompt change is a live experiment with no regression safety net.
How evaluation methods actually score an output
Reference-based evaluation compares an output against a known-correct answer: exact match for structured extraction, semantic similarity via embeddings for looser tasks, or a scoring rubric against a gold-standard response. It only works when a ground truth exists, which rules out a lot of open-ended generation.
Reference-free evaluation scores an output without a ground truth to compare against, which covers most of what actually flows through a production LLM app. This ranges from cheap deterministic checks (is it valid JSON, does it contain a blocked term, is the response non-empty) to embedding-based faithfulness checks in RAG pipelines that measure whether the answer stays grounded in the retrieved context rather than drifting into things the model just knows from training.
LLM-as-judge is the reference-free method doing most of the heavy lifting right now: a separate model call scores the output against a rubric. A typical judge call looks like this:
1234567{"task": "Answer the user's billing question using only the provided account context.","context": "<retrieved account data>","output": "<the candidate response>","rubric": "Score 1-5 on faithfulness to context. 5 = every claim is directly supported. 1 = contains claims not present in context.","format": "Return {\"score\": int, \"rationale\": string}"}
The judge returns a structured score and a rationale, not a pass/fail with no explanation, because the rationale is what you read when you're debugging why a score dropped.
Human evaluation stays the gold standard, but it doesn't scale to every request. Use it to build your initial golden dataset and to periodically spot-check whether your automated judge still agrees with a human reviewer. If judge and human scores start diverging, trust the human and recalibrate the rubric.
How to sample production traffic
Scoring every production request with an LLM-as-judge call doubles your inference cost and adds latency if it's inline, so almost nobody evaluates 100% of traffic. A reasonable starting point: score a random 1-5% of all traffic to get a baseline quality trend, and layer explicit triggers on top so you don't rely on chance to catch the outputs that matter most. Score every response a user thumbs-downed or regenerated. Score every response in a category you already know is risky, like anything touching numbers, dates, or account-specific data. Random sampling alone tends to dilute your worst failure modes into a sea of fine-but-boring responses.
Start with offline evals if you're pre-launch
If you don't have production traffic yet, don't build a continuous evaluation pipeline first. Build a golden dataset of 50-200 representative cases and run it against every prompt or model change using an eval harness like promptfoo or DeepEval before anything ships. Continuous production evaluation matters once you have traffic to continuously evaluate; before that, it's infrastructure with nothing to measure.
Common pitfalls
Your judge has opinions about itself. LLM-as-judge scoring carries well-documented biases: it tends to prefer longer answers regardless of quality (verbosity bias), it can favor whichever option appears first or last in a comparison (position bias), and a judge tends to rate outputs from its own model family more favorably than outputs from a different provider (self-preference bias). Mitigate this by rotating the order of compared options, using a judge from a different provider than the model under evaluation, and periodically checking judge scores against a small human-labeled sample to catch drift.
A single score means nothing. A trend means something. Judge models aren't perfectly deterministic, so the same output scored twice can come back with slightly different numbers. Don't treat one low score on one request as proof of a regression. Look at the score distribution across a few hundred samples before and after a change.
Evaluation data that lives outside your telemetry is useless during an incident. If your eval pipeline writes scores to a separate dashboard or spreadsheet, you can't answer "did quality drop at the same time as the latency spike, and was it the same deploy?" without manually cross-referencing timestamps across two systems. OpenTelemetry's GenAI semantic conventions now define a real answer for this: the gen_ai.evaluation.result event, currently at Development stability. It carries gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.score.label, and gen_ai.evaluation.explanation, and it's meant to be parented to the exact GenAI span it evaluated (or linked via gen_ai.response.id when a span isn't available). Emit your judge's verdict as this event instead of a row in a separate table, and the score lives on the same trace as the token counts and latency for that call. It's new enough that most observability vendors don't have dedicated UI for it yet, but the data model is worth adopting now regardless, since the payoff is being able to filter by trace ID and see the eval score sitting right next to the span it graded.
Final thoughts
Evaluation only earns its keep once you're watching the trend, not just running the pipeline. Once you have scores flowing, alert on a sustained drop the same way you'd alert on an error-rate spike, because a quiet quality regression is just as much an incident as a loud one.
Dash0 is an OpenTelemetry-native observability platform that treats GenAI signals as first-class telemetry, so if your evaluator emits gen_ai.evaluation.result events or standard gen_ai.* span attributes, they land in the same explorer as your traces, logs, and metrics, filterable by trace ID right alongside token usage and latency for that call. The vLLM observability guide walks through what that GenAI telemetry looks like in a real setup.
Start a free trial to see your LLM traces, token usage, and evaluation scores in one place. No credit card required.