LLM Observability: A Developer's Guide

Putting an LLM into production changes what it means to understand a software system.

In a conventional web service, a successful request is usually evidence that the system worked. The server accepted the request, application logic executed, dependencies responded, and the API returned a valid result. Errors, latency, throughput, and resource utilization provide a reasonably useful picture of system health.

An LLM application can return 200 OKCopy in 800 milliseconds and still fail completely.

The model may misunderstand the user's intent. A retrieval system may return the wrong documents. An agent may select the wrong tool. A tool may succeed technically while making an incorrect business decision. A model may produce a fluent answer unsupported by its context. A conversation may require five unnecessary model calls. A memory subsystem may surface stale information. A prompt change may silently reduce answer quality while improving latency.

None of these failures necessarily produce an exception.

That is why LLM observability cannot be reduced to prompt logging or traditional application performance monitoring. It requires connecting execution telemetry, model behavior, evaluation results, configuration, cost, security signals, and real-world outcomes.

A useful definition is:

LLM observability is the ability to reconstruct, measure, explain, and improve the behavior of an AI application from user intent through model execution, retrieval, tools, memory, policies, and downstream outcomes.

The industry is increasingly converging around this view. OpenTelemetry now maintains dedicated Generative AI semantic conventions covering model operations, agent operations, metrics, events, and provider-specific telemetry, and separate work is underway to extend OpenTelemetry conventions to Model Context Protocol (MCP) interactions. Those conventions are currently marked Development, so they should be treated as an important emerging standard rather than a completely frozen contract.

At the same time, modern LLM engineering platforms increasingly connect production traces with online evaluation, curated datasets, experiments, and regression testing. LangSmith, for example, explicitly describes a lifecycle in which production traces are evaluated, failures become dataset examples, fixes are validated offline, and improvements are then redeployed.

This guide explains how to build that kind of system.

1. Why LLM observability is different

Traditional observability normally revolves around three major signals:

• traces;
• metrics;
• logs.

Those signals remain essential, but they do not answer the most important question in an AI application:

Was the result good?

Consider a support assistant that returns this result:

Operationally, everything looks healthy.

But suppose the assistant confidently tells the customer that their subscription was cancelled when no cancellation occurred.

Traditional telemetry sees success.

The user sees failure.

An observable AI system therefore needs several connected planes of information.

A mature observability system connects all seven.

This leads to an important principle:

Do not define success at the model boundary. Define success at the task boundary.

If an agent's job is to create a support ticket, the most trustworthy indication of success is usually that the ticket exists, not that the model said, “Your ticket has been created.”

If the system is supposed to answer questions from company documentation, success involves more than grammatical correctness. The appropriate document must have been retrieved, the answer must be supported by it, and the response must satisfy the user's information need.

The deeper you move into RAG and agents, the less meaningful a single model response becomes as the unit of observation.

2. The observable AI stack

A useful mental model is to think of an AI request as a distributed workflow.

A relatively simple retrieval-augmented generation (RAG) application might execute:

12345678910111213141516
User request
    │
    ▼
API
    │
    ├── authenticate
    ├── classify intent
    ├── rewrite search query
    │
    ├── retrieve documents
    ├── rerank documents
    ├── construct context
    │
    ├── call LLM
    ├── validate output
    └── return answer

An agentic system can be considerably more complicated:

12345678910111213141516171819202122
User
 │
 ▼
Agent
 │
 ├── Model call
 │     │
 │     └── decides to use tool
 │
 ├── Tool: search_orders
 │
 ├── Model call
 │     │
 │     └── decides to issue refund
 │
 ├── Policy check
 │
 ├── Tool: create_refund
 │
 ├── Model call
 │
 └── Final response

Every box may fail independently.

And some failures propagate.

A bad retrieval result can cause a good model to produce a bad answer. A malformed tool result can cause a subsequent planning decision to fail. A stale memory can influence several model turns. A compromised document can indirectly inject instructions into an agent that later invokes a privileged tool.

The job of observability is to preserve the causal structure.

The core object for doing that is the distributed trace.

3. Traces, spans, sessions, and events

3.1 Traces

A trace represents one meaningful end-to-end operation.

For a chatbot, that may be one conversational turn.

For an agent, it may be one user task.

For a long-running workflow, it may span multiple services and workers.

A trace should have a globally unique identifier and contain a tree of child spans.

12345678910111213141516
trace: customer_support_request
│
├── span: classify_intent
│
├── span: retrieve_policy
│   ├── embedding
│   ├── vector_search
│   └── rerank
│
├── span: agent
│   ├── model_call
│   ├── tool_call
│   ├── model_call
│   └── guardrail
│
└── span: response_validation

3.2 Spans

A span represents one bounded operation.

Typical AI spans include:

3.3 Sessions

A trace is often too small to describe a user's actual experience.

Suppose a customer asks:

User: I need to change my delivery address. Agent: What is your order number? User: 48392. Agent: That order has already shipped. User: Can it be redirected? Agent: Let me check.

Each user turn may create a separate trace, but the quality of the interaction can only be understood across the session.

Session identifiers make it possible to measure:

• number of turns to resolution;
• repeated questions;
• conversational coherence;
• user frustration;
• escalation;
• eventual task completion;
• cumulative token cost;
• cumulative tool activity.

A useful hierarchy is therefore:

• Session
  • Trace / turn
    • Span
    • Span
    • Span
  • Trace / turn
    • ...
  • Trace / turn

3.4 Events

Not everything deserves its own span.

Events are useful for noteworthy moments inside an operation:

• stream.startedCopy
• fallback.triggeredCopy
• rate_limit.receivedCopy
• guardrail.blockedCopy
• human_approval.requestedCopy
• memory.write_rejectedCopy

Use spans when timing and parent-child structure matter. Use events when something significant happens inside an existing operation.

4. Use OpenTelemetry as the backbone

For most engineering teams, the safest long-term architecture is to treat AI observability as an extension of distributed observability rather than creating a completely separate telemetry system.

That generally means:

123456789101112131415161718
Application
   │
   ▼
OpenTelemetry SDK / instrumentation
   │
   ▼
OpenTelemetry Collector
   │
   ├── enrichment
   ├── filtering
   ├── redaction
   ├── sampling
   └── routing
        │
        ├── trace backend
        ├── metrics backend
        ├── logs
        └── AI evaluation platform

This provides several benefits.

First, an LLM call can remain connected to the HTTP request, database query, queue message, cache lookup, and downstream API calls that surround it.

Second, instrumentation is less tightly coupled to one observability vendor. An OTel-native backend, Dash0 among them, can consume this exact pipeline without a separate SDK or a proprietary agent bolted on top.

Third, existing operational practices, including trace propagation, collectors, sampling, service metadata, alerting, and incident response, can be reused.

A caution about semantic conventions

As of August 2026, OpenTelemetry's GenAI semantic conventions remain in Development. Teams should expect attributes to continue evolving.

Do not scatter raw convention strings throughout application code.

Instead, create a small instrumentation abstraction:

123456789101112
class AITrace:
    def model_call(self, **metadata):
        ...

    def retrieval(self, **metadata):
        ...

    def tool_call(self, **metadata):
        ...

    def agent_run(self, **metadata):
        ...

That gives you one location where semantic conventions can evolve without rewriting the application.

5. What should an LLM trace contain?

The answer is much more than the prompt and completion.

A useful production trace should make it possible to answer four questions:

1. What executed?
2. What information influenced it?
3. Which configuration produced it?
4. What happened as a result?

A canonical trace schema should contain the following categories.

Request identity

• trace_idCopy
• span_idCopy
• session_idCopy
• request_idCopy
• tenant_idCopy
• user_pseudonymous_idCopy

Avoid putting raw personally identifying user information into observability fields unless there is a compelling requirement.

Application identity

• service.nameCopy
• service.versionCopy
• deployment.environment.nameCopy
• cloud.regionCopy
• vcs.ref.head.revisionCopy
• release.idCopy
• feature_flagsCopy

Model configuration

• providerCopy
• requested_modelCopy
• served_modelCopy
• temperatureCopy
• max_output_tokensCopy
• reasoning_configurationCopy
• response_formatCopy
• tool_choiceCopy

Prompt configuration

• prompt.nameCopy
• prompt.versionCopy
• prompt.template_hashCopy
• system_instruction_versionCopy
• few_shot_dataset_versionCopy

Do not use the entire prompt as the version identifier. Version the prompt as an artifact.

Usage

• input_tokensCopy
• output_tokensCopy
• cached_input_tokensCopy
• other_billable_token_categoriesCopy

Streaming

• request_started_atCopy
• first_chunk_atCopy
• first_token_atCopy
• stream_completed_atCopy
• chunks_emittedCopy
• stream_cancelledCopy

Retrieval

• retriever.nameCopy
• embedding_modelCopy
• index.nameCopy
• index.versionCopy
• queryCopy
• filtersCopy
• top_kCopy
• document_idsCopy
• retrieval_scoresCopy
• reranker_modelCopy
• reranker_scoresCopy
• context_tokensCopy

Tools

• tool.nameCopy
• tool.versionCopy
• tool.argumentsCopy
• tool.resultCopy
• tool.durationCopy
• tool.statusCopy
• authorization_policyCopy
• side_effect_idCopy

OpenTelemetry's experimental GenAI conventions define close equivalents under a gen_ai.Copy prefix (gen_ai.tool.nameCopy, gen_ai.tool.call.idCopy, gen_ai.tool.call.argumentsCopy, gen_ai.tool.call.resultCopy), so use those instead if you're instrumenting against an OTel-native backend.

Agent state

• agent.nameCopy
• agent.versionCopy
• turn_numberCopy
• parent_agentCopy
• handoff_targetCopy
• tool_call_countCopy
• model_call_countCopy
• retry_countCopy
• termination_reasonCopy

Here too, gen_ai.agent.nameCopy, gen_ai.agent.idCopy, and gen_ai.agent.versionCopy are the current OTel GenAI equivalents.

Guardrails

• policy.nameCopy
• policy.versionCopy
• decisionCopy
• risk_categoryCopy
• confidenceCopy
• actionCopy

Evaluation

• evaluator.nameCopy
• evaluator.versionCopy
• scoreCopy
• labelCopy
• thresholdCopy
• evaluation_scopeCopy
• judge_modelCopy

Outcome

• task_completedCopy
• transaction_idCopy
• ticket_idCopy
• escalatedCopy
• user_feedbackCopy
• conversionCopy
• resolution_codeCopy

The final category is often the most important and the least instrumented.

6. Structural telemetry versus content telemetry

One of the most important architectural distinctions is between structural telemetry and payload telemetry.

Structural telemetry describes what happened:

Payload telemetry contains the actual content:

• user prompt
• system prompt
• retrieved passages
• model response
• tool arguments
• tool results
• conversation history

You may want to retain structural telemetry broadly while sampling or suppressing payload telemetry.

A reasonable production policy might therefore be:

The exact percentages depend on traffic, regulatory requirements, and risk tolerance.

7. Measure latency as a pipeline, not a single number

End-to-end latency is necessary but not diagnostic.

Suppose response latency increases from two seconds to five seconds.

The model might be slower.

Or retrieval.

Or a tool.

Or a rate-limit retry.

Or your request may be sitting in a queue.

Instrument the latency budget.

Now the problem is obvious: the tool call, not the model, owns most of the budget.

Streaming metrics

For interactive systems, users perceive latency differently from machines.

A ten-second generation that begins streaming after 300 milliseconds can feel faster than a four-second response that emits nothing until the end.

Useful streaming metrics include:

For voice systems, extend the latency decomposition into every step between speech and response:

1. Speech end detection
2. Transcription
3. Agent/model planning
4. Tool execution
5. Response generation
6. Text-to-speech
7. First audible response

8. Token and cost observability

Token usage is not just a billing metric. It is often a diagnostic signal.

A sudden increase in input tokens can indicate:

• prompt growth;
• excessive conversation history;
• RAG context bloat;
• duplicated documents;
• memory accumulation;
• tool outputs being passed back unnecessarily.

A sudden increase in output tokens may indicate:

• prompt regression;
• missing stop conditions;
• agent loops;
• verbosity changes;
• model migration effects.

Track costs at several levels:

• cost per model call
• cost per trace
• cost per session
• cost per customer
• cost per feature
• cost per successful task

The last metric is particularly useful.

That is why optimization should generally target a quality × latency × cost frontier, not the cheapest model call.

9. Evaluation is part of observability

Traditional monitoring measures deterministic behavior well.

Evaluation measures semantic behavior.

An AI production system usually needs both.

Suppose you observe an error rate of 0.2%, P95 latency of 1.9 seconds, and a mean cost of $0.014 per request.

Nothing appears wrong.

But your groundedness evaluator has fallen from 97% to 83%.

That is a production incident.

Without evaluation telemetry, you would probably miss it.

9.1 Evaluate at the correct scope

Evaluation can occur at several levels.

Do not evaluate everything at the final-answer level.

10. Build a hierarchy of evaluators

Not every quality problem requires another LLM.

Use the cheapest reliable evaluator.

Level 1: deterministic checks

1234
assert output["currency"] in SUPPORTED_CURRENCIES
assert response.status_code == 200
assert len(citations) > 0
assert refund_amount <= authorized_limit

Level 2: programmatic heuristics

Examples include:

• citation overlap
• retrieval precision
• schema completeness
• keyword coverage
• edit distance
• SQL execution correctness
• tool success state

Level 3: reference-based evaluation

Compare an output to a known expected value.

Level 4: model-based evaluation

Useful for:

• helpfulness
• groundedness
• instruction adherence
• tone
• completeness
• answer relevance

Level 5: human evaluation

Humans remain important for:

• subjective quality
• policy interpretation
• high-risk errors
• judge calibration
• new failure discovery

Level 6: real-world outcomes

Examples:

• Was the ticket created?
• Did the user accept the answer?
• Did the transaction complete?
• Did the user reopen the issue?
• Was the generated SQL query correct?
• Did the test suite pass?

Prefer deterministic truth over model judgment whenever deterministic truth exists.

11. Using LLM-as-a-judge responsibly

LLM judges are extremely useful but should not be treated as unquestionable ground truth.

A robust judge should have a precise rubric.

A weak rubric looks like this:

Is this a good answer? Score it from 1 to 10.

A better rubric looks like this:

Determine whether every factual claim in the answer is supported by the supplied context. Return one of: SUPPORTED, PARTIALLY_SUPPORTED, UNSUPPORTED.

Version the evaluator

An evaluator is software. Store:

• evaluator_nameCopy
• evaluator_versionCopy
• judge_modelCopy
• judge_prompt_versionCopy
• rubric_versionCopy
• thresholdCopy

Trace the evaluator

Evaluation itself has latency, token usage, failures, and drift.

Calibrate against people

Take a human-labelled sample and compare evaluator decisions.

Track metrics such as:

• agreement rate
• false positive rate
• false negative rate
• class-specific recall
• judge disagreement

12. Offline and online evaluation

You need both.

Offline evaluation

Use it for:

• prompt comparisons
• model migrations
• retriever changes
• embedding migrations
• regression testing
• parameter tuning
• tool-schema changes

Online evaluation

Use it to discover:

• unknown user behavior
• new edge cases
• domain drift
• provider changes
• long-tail failures
• adversarial inputs
• production-only integration problems

The mature lifecycle looks like this:

1. Production traffic generates traces.
2. Online evaluation scores those traces.
3. Interesting failures are identified.
4. Failures become a curated evaluation dataset.
5. The dataset drives offline experiments.
6. Experiments that pass become a CI regression gate.
7. Passing changes deploy, and the cycle returns to production.

13. RAG observability

Retrieval-augmented generation creates a dangerous debugging shortcut:

“The model hallucinated.”

Often, the model did exactly what it could with bad context.

An observable RAG pipeline separates the stages and captures each one independently:

1. User question
2. Query transformation
3. Embedding
4. Retrieval
5. Filtering
6. Reranking
7. Context construction
8. Generation

Query transformation

Record:

• original query
• rewritten query
• rewrite model/version
• filters inferred

Embeddings

Capture:

• embedding model
• embedding dimensions
• embedding version
• input hash
• latency

Retrieval

Record:

• index
• index snapshot/version
• namespace
• filters
• top_k
• document IDs
• scores
• metadata

Reranking

Capture:

• reranker model
• candidate count
• rank before
• rank after
• score
• top_k retained

Context construction

Record:

• selected documents
• context ordering
• context token count
• truncated documents
• deduplication

RAG evaluation

Useful measurements include:

MRR (mean reciprocal rank) and NDCG (normalized discounted cumulative gain) are standard information-retrieval metrics for how well the top results are ordered, not just whether the right document showed up at all.

14. Agent observability

Agents shift the unit of analysis from response to trajectory.

A typical agent may work through a sequence like this:

1. Interpret
2. Plan
3. Call model
4. Select tool
5. Execute tool
6. Observe result
7. Call model again
8. Select another tool
9. Verify
10. Respond

The trajectory itself is the behavior.

Instrument agent boundaries

Record:

• agent.nameCopy
• agent.versionCopy
• agent.invocation_idCopy
• parent_agentCopy
• inputCopy
• outputCopy
• termination_reasonCopy

Model and tool calls per task

Track model_calls_per_agent_invocationCopy and tool_calls_per_agent_invocationCopy for every agent run.

Useful agent efficiency metrics include:

• steps to completion
• model calls per task
• tool calls per task
• retries per task
• duplicate tool calls
• loop rate
• handoff count
• cost per successful task

Observe tool arguments

Knowing that a tool was called is not enough. You need to know:

• Why was it called?
• What arguments were supplied?
• Which authorization policy approved it?
• What result came back?
• Was a side effect actually created?

15. Outcome verification for agents

One of the most common mistakes in agent evaluation is asking another model whether the agent succeeded when the environment can provide the answer.

For agents that act on the world, outcome telemetry should generally have priority over conversational confidence.

Examples:

The model's final sentence is evidence of what the model believes happened.

The external system is evidence of what actually happened.

16. Memory observability

Persistent memory introduces a new form of state.

Trace operations such as:

• memory.searchCopy
• memory.readCopy
• memory.writeCopy
• memory.updateCopy
• memory.deleteCopy

Capture:

• memory namespace
• tenant scope
• memory IDs
• retrieval scores
• source/provenance
• created_atCopy
• updated_atCopy
• TTL
• writer

Then monitor for:

• stale memory
• contradictory memory
• duplicate memory
• cross-tenant leakage
• incorrect memory attribution
• unexpected persistence

A particularly useful observability field is memory provenance.

17. MCP and multi-agent observability

Model Context Protocol (MCP) expands the observability boundary.

An application may now involve:

1234567891011
Agent
   │
   ├── MCP client
   │      │
   │      └── MCP server
   │             │
   │             ├── tool
   │             ├── data source
   │             └── downstream API
   │
   └── second agent

Trace context must survive these boundaries.

For MCP, capture:

• client identity
• server identity
• server version
• operation
• tool/resource/prompt identifier
• session
• transport
• duration
• result
• error

For multi-agent systems, add:

• sender agent
• receiver agent
• handoff reason
• delegated goal
• delegated permissions
• shared context version
• result

The important thing is to preserve causality.

18. Do not confuse observability with hidden chain-of-thought

Good LLM observability does not require collecting a model's private internal reasoning.

What developers need is observable system behavior:

• input
• output
• retrieved context
• model request
• tool decision
• tool arguments
• tool result
• handoff
• memory interaction
• policy result
• timing
• configuration
• evaluation
• outcome

A trace should answer:

What did the system do?

rather than requiring:

Reveal every private internal reasoning step the model may have used.

19. Security observability

AI security should share the same trace model as AI reliability.

Observability should make events such as this reconstructable, in order:

1. An untrusted document is retrieved.
2. Its content reaches the agent's context.
3. The agent attempts a privileged tool.
4. The authorization policy denies the call.

A high-quality security trace might contain:

• source_document_idCopy
• source_trust_levelCopy
• retrieval_trace_idCopy
• agent_idCopy
• requested_toolCopy
• requested_actionCopy
• credential_identityCopy
• permission_scopeCopy
• policy_decisionCopy
• side_effectCopy

Signals worth monitoring

Track rates and anomalies around:

• guardrail blocks
• prompt-injection detections
• denied tool calls
• privileged tool attempts
• cross-tenant retrieval attempts
• unexpected network destinations
• suspicious memory writes
• tool schema violations
• unusually high token consumption
• agent loops
• unexpected code execution

20. Privacy and telemetry governance

LLM observability creates a paradox:

The data most useful for debugging is often the data most dangerous to retain.

Treat the observability pipeline as a sensitive data system.

Redact before export

Prefer routing data so it flows application → redaction/filtering → collector/backend, never straight from the application to a backend that has not seen the redaction step.

Separate fields by sensitivity

For example:

• PUBLIC
• INTERNAL
• CONFIDENTIAL
• RESTRICTED

Use identifiers rather than content where possible

Instead of storing entire documents, prefer document IDs, versions, and chunk IDs.

Control retention

Example:

21. Sampling at production scale

Capturing every detail of every trace can become expensive.

It can also increase privacy risk.

A good architecture distinguishes several sampling decisions.

Structural sampling

Keep basic timing, model, usage, status, and version data at a high rate.

Payload sampling

Sample prompts, responses, documents, and tool payloads much more conservatively.

Tail sampling

Tail sampling is particularly valuable because you can preferentially keep traces that exhibit interesting behavior, matching a rule such as:

123456
error == true
OR latency > threshold
OR cost > threshold
OR groundedness == fail
OR policy_violation == true
OR task_success == false

But do not keep only failures.

Retain an unbiased random sample as well.

A healthy strategy combines:

• a random baseline sample;
• a high-value tail sample;
• security-required retention.

22. High-cardinality telemetry

AI systems generate enormous numbers of unique values:

• user IDs
• prompts
• document IDs
• trace IDs
• conversation IDs
• tool arguments

Do not blindly turn these into metric dimensions.

Use metrics for bounded dimensions such as:

• model
• deployment
• environment
• region
• feature
• error_typeCopy
• tool_nameCopy
• prompt_versionCopy

Keep high-cardinality information in traces or logs.

23. Version everything that can change behavior

One of the most valuable capabilities of an observability system is answering:

Why did this request behave differently from yesterday?

At minimum, consider versioning:

• application build
• model
• prompt
• developer/system instructions
• tool definitions
• tool implementations
• guardrails
• embedding model
• retrieval index
• reranker
• memory schema
• feature flags
• evaluator

This is what makes incidents reproducible.

24. Define LLM SLOs

Traditional availability SLOs remain necessary.

But they are not sufficient.

You may need service-level objectives across several dimensions.

Segment your SLOs

Always be able to segment by dimensions such as:

• model
• prompt version
• deployment
• user intent
• language
• region
• customer tier
• agent
• tool
• retriever version

25. Alert on behavior, not merely infrastructure

Traditional alerts remain useful.

AI-specific alerts might include:

• groundedness pass rate falls more than 5%
• cost per successful task rises more than 30%
• tool failure rate exceeds 3%
• agent loop rate doubles
• retrieval no-hit rate increases
• safety evaluator failure spikes
• mean model calls per task changes sharply
• human escalation rate rises
• negative feedback increases

Be careful with evaluator noise.

Alert on sustained rate changes, significant cohort regressions, and high-severity individual events.

26. Incident response playbooks

Observability becomes valuable when something goes wrong.

Playbook: quality regression

1. Identify the first time of regression.
2. Correlate with deployments and configuration changes.
3. Segment by model, prompt, agent, intent, language, and region.
4. Compare successful and failed traces.
5. Determine the failing stage: retrieval, model, tool, memory, or policy.
6. Reproduce representative traces offline.
7. Add failures to the evaluation dataset.
8. Implement a fix.
9. Run the regression suite.
10. Deploy and confirm production recovery.

Playbook: cost spike

Inspect:

• traffic volume
• input token distribution
• output token distribution
• model mix
• cache hit rate
• retries
• model calls per task
• tool calls per task
• agent loops
• prompt changes
• context size

Playbook: latency spike

Break the request into its stages: queue → retrieval → model → tool → model → validation, and time each one.

Playbook: suspicious tool activity

Trace backward from the tool call:

1. Tool execution
2. Authorization
3. Agent decision
4. Model context
5. Retrieved content
6. Original request

27. Self-hosted model observability

When you host inference yourself, model-level telemetry is only part of the problem.

Observe:

• GPU utilization
• GPU memory
• queue depth
• batch size
• request concurrency
• KV-cache utilization
• cache hit rate
• prefill latency
• decode latency
• tokens/sec
• OOM events
• model loading
• scheduler decisions

KV-cache refers to the attention key-value cache that stores previously computed attention states so the model doesn't recompute them on every new token. OOM means out-of-memory, usually the first sign that batch size or context length has outgrown available GPU memory.

28. Reference instrumentation pattern

A minimal application-specific abstraction might look like this:

1234567891011121314151617181920212223242526272829303132333435363738
from contextlib import contextmanager
from opentelemetry import trace

tracer = trace.get_tracer("support-ai")


@contextmanager
def model_span(
    *,
    model,
    prompt_version,
    feature,
):
    with tracer.start_as_current_span("ai.model.generate") as span:
        span.set_attribute("ai.model", model)
        span.set_attribute("ai.prompt.version", prompt_version)
        span.set_attribute("app.feature", feature)

        state = {}

        try:
            yield state

            if "input_tokens" in state:
                span.set_attribute(
                    "ai.usage.input_tokens",
                    state["input_tokens"],
                )

            if "output_tokens" in state:
                span.set_attribute(
                    "ai.usage.output_tokens",
                    state["output_tokens"],
                )

        except Exception as exc:
            span.record_exception(exc)
            raise

Use it inside the application:

12345678910111213141516171819
with tracer.start_as_current_span("support.answer") as request_span:
    request_span.set_attribute("app.version", APP_VERSION)
    request_span.set_attribute("session.id", session_id)

    docs = retrieve(query)

    with model_span(
        model=MODEL,
        prompt_version="support:v17",
        feature="support",
    ) as usage:

        result = client.generate(
            query=query,
            context=docs,
        )

        usage["input_tokens"] = result.usage.input_tokens
        usage["output_tokens"] = result.usage.output_tokens

The important design decision is that instrumentation has a stable internal interface.

29. A reference production architecture

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354
                         ┌──────────────────────┐
                         │      End user        │
                         └──────────┬───────────┘
                                    │
                                    ▼
                         ┌──────────────────────┐
                         │ Application / API    │
                         └──────────┬───────────┘
                                    │
             ┌──────────────────────┼─────────────────────┐
             │                      │                     │
             ▼                      ▼                     ▼
      ┌────────────┐        ┌──────────────┐       ┌────────────┐
      │ Retrieval  │        │ Agent / LLM  │       │ Guardrail  │
      └─────┬──────┘        └──────┬───────┘       └─────┬──────┘
            │                      │                     │
            │              ┌───────┴───────┐             │
            │              │               │             │
            ▼              ▼               ▼             │
      Vector/search      Tools           Memory          │
            │              │               │             │
            └──────────────┴───────┬───────┴─────────────┘
                                   │
                                   ▼
                           OpenTelemetry SDK
                                   │
                                   ▼
                           OTel Collector
                                   │
                 ┌─────────────────┼─────────────────┐
                 │                 │                 │
                 ▼                 ▼                 ▼
             Redaction          Sampling         Enrichment
                 │                 │                 │
                 └─────────────────┼─────────────────┘
                                   │
                ┌──────────────────┼────────────────────┐
                │                  │                    │
                ▼                  ▼                    ▼
          Trace backend       Metrics/APM         Evaluation
                                                        │
                                                        ▼
                                              Quality annotations
                                                        │
                    ┌───────────────────────────────────┼───────┐
                    │                                   │       │
                    ▼                                   ▼       ▼
               Dashboards                            Alerts   Dataset
                                                                │
                                                                ▼
                                                           Experiments
                                                                │
                                                                ▼
                                                           CI regression

The important part is that evaluation results join back to the same trace/session identifiers. Dash0, for example, ingests exactly this shape over OTLP and keeps traces, metrics, and evaluation-adjacent signals joined by the same resource, instead of splitting AI telemetry into a separate store from everything else in the diagram.

30. Turning production failures into test cases

One of the highest-leverage workflows is converting real incidents into regression tests.

Suppose production produces this failure:

User: Can I cancel my annual plan and receive a refund? System: Yes. I have cancelled it and issued a refund. Actual state: No cancellation occurred.

The trace reveals that the model decided the refund was allowed, the refund tool was never called, and the model still claimed completion anyway.

Create a dataset example containing:

• input
• relevant context
• expected tool trajectory
• expected outcome

Add evaluators:

• the refund tool must be invoked;
• a transaction must exist;
• the response cannot claim completion otherwise.

Now every future prompt or model change runs against it.

The production bug has become a permanent test.

This is the core of eval-driven development.

31. Selecting an observability platform

Do not start by asking:

Which LLM observability vendor is best?

Start by asking:

What telemetry architecture do we need?

Then evaluate products.

Important capabilities include:

The tooling landscape broadly falls into several categories:

• Open standards and instrumentation: OpenTelemetry, OpenInference
• AI-native observability and evaluation: LangSmith, Phoenix, Braintrust, Langfuse, W&B Weave, and others
• General observability platforms: this is where Dash0 sits, built OTel-native from the ground up rather than adding OpenTelemetry support onto a proprietary agent, alongside the wider Grafana ecosystem and cloud APM systems
• Cloud AI platforms: AWS, Microsoft, Google, and others
• Provider and framework tracing: model and agent SDK tracing
• Gateways and proxies: cross-provider routing and usage layers

32. Common anti-patterns

“We log every prompt and response”

That is logging, not observability.

“We have a gateway, so we can observe the AI system”

A gateway usually cannot fully see retrieval, application routing, memory, internal tools, business outcomes, or agent state.

“Our dashboard shows average latency”

Averages hide tail behavior. Use percentile distributions.

“The LLM judge says we're at 92%”

A number without a rubric, evaluator version, sample definition, and human calibration is weak evidence.

“The final answer looked correct”

An agent can produce a plausible final answer after an incorrect trajectory.

“We can reproduce it from the prompt”

Not if the trace lacks the model, prompt version, tools, retrieval index, feature flags, memory, and evaluator.

“We'll store everything indefinitely”

That is often expensive, unnecessary, and risky.

33. An observability maturity model

Level 0: prompt logs

• prompt
• response
• timestamp

Level 1: model telemetry

• model
• latency
• tokens
• errors
• cost

Level 2: distributed AI traces

• retrieval
• tools
• orchestration
• trace propagation
• sessions
• versions

Level 3: quality observability

• automated evaluation
• human labels
• user feedback
• outcomes

Level 4: continuous evaluation

• production
• evaluation
• datasets
• experiments
• CI

Level 5: governed agent observability

• security signals
• identity
• permissions
• memory provenance
• agent trajectories
• MCP
• privacy controls
• business outcomes

34. A practical implementation sequence

Phase 1: establish traces

Instrument the end-to-end request plus model, retrieval, and tool spans.

Phase 2: add configuration provenance

Record application, prompt, model, index, tool, and policy versions.

Phase 3: add performance and economics

Measure latency decomposition, streaming latency, tokens, and cost.

Phase 4: instrument outcomes

Connect traces to real-world success signals.

Phase 5: add evaluation

Start with deterministic checks, then add selected model judges and human calibration.

Phase 6: build production feedback loops

Convert production failures into offline evaluation cases.

Phase 7: add governance

Implement redaction, sampling, retention, RBAC, and audit controls.

Phase 8: operationalize

Define quality SLOs, behavioral alerts, and incident-response procedures.

35. The metrics that matter

Do not implement every metric because it exists.

Each metric should answer an operational question.

36. The central operating principle

The purpose of LLM observability is not to collect the maximum amount of data.

It is to reduce uncertainty.

When something goes wrong, an engineer should be able to move from:

“The AI seems worse today.”

to:

“Task success fell 9% for Spanish refund requests after prompt v24. The new prompt caused the agent to skip lookup_refund_policyCopy in 31% of affected traces. Retrieval quality and model latency are unchanged. Reverting to v23 restores the previous trajectory and passes the production-derived regression dataset.”

That is observability.

Or:

“P95 latency increased 1.6 seconds, but model TTFT is unchanged. The slowdown is isolated to inventory_lookupCopy, which began retrying after the downstream API deployment.”

That is observability.

Or:

“The agent attempted a privileged file operation after retrieving an untrusted document containing injected instructions. The authorization layer blocked the action. The trace identifies the document, retrieval operation, agent invocation, attempted tool call, and policy decision.”

That is observability.

The goal is not more dashboards.

The goal is explainable system behavior.

Final thoughts

LLM observability is becoming a discipline of its own because probabilistic, tool-using AI systems fail differently from traditional software.

Logs and infrastructure metrics remain essential, but they cannot tell you whether an answer was grounded, whether an agent chose the correct tool, whether a RAG pipeline retrieved the correct evidence, whether memory introduced stale information, or whether the system actually achieved the user's intended outcome.

A production-grade observability architecture therefore needs to connect, in order: user intent, application execution, retrieval and context, the model, tools and memory and agents, guardrails, the response, evaluation, and the real-world outcome, and attach enough version information to reproduce that behavior later.

The most mature teams will therefore treat observability and evaluation as one continuous engineering system:

1. Instrument
2. Observe
3. Evaluate
4. Investigate
5. Capture failures
6. Build regression tests
7. Experiment
8. Deploy
9. Observe again, closing the loop back to step 1

That feedback loop is what turns an unpredictable AI prototype into an operable production system.

The key question for every instrumentation decision is simple:

When this AI system behaves badly in production, will the data we are collecting let an engineer determine what happened, whether it was good, which system state produced it, what real-world consequence followed, and how to prevent the same failure from recurring?

If the answer is yes, you are building observability.

If the answer is no, you are probably just collecting logs.

Most teams should not try to build all 36 sections of this at once. Start with distributed traces across the model, retrieval, and tool boundaries described in Section 3, add configuration and version provenance next, and only then layer in evaluation and cost. An LLM judge running on top of a system with no structural traces underneath it just produces a confident number nobody can act on.

This is also the architecture Dash0 is built around. It's OTel-native from the ground up, so the traces, spans, and gen_aiCopy attributes described throughout this guide show up next to your existing logs, metrics, and traces instead of in a separate "AI observability" product with its own agent and its own bill. It's built as an AI control plane for production: telemetry stays in one place, in the open OpenTelemetry format, whether it came from a database driver or a model call. If you're instrumenting agents specifically, Dash0's guide to agentic observability applies the trajectory and outcome-verification patterns from Section 14 in practice, and Agent0 is Dash0's own production AI agent, built to investigate incidents by reasoning over that same normalized telemetry rather than raw logs. You can try Dash0 on your own traces at dash0.com. No credit card required.