Write Eval Prompts

Every eval sends two prompts to its judge for each sampled span. The prompts decide what the judge reads and how it scores, so they matter more than any other setting.

• System prompt: Optional. Sets the judge's role and scoring standard, for example "You are a strict reviewer of customer support answers."
• User prompt: Required. Contains the content to judge, pulled from the span, and the question the judge answers.

Reference Span Attributes

Insert a span attribute into the user prompt with this syntax:

{{gen_ai.input.messages}}

Type {{ in the User prompt field to pick from the attributes found on spans that match the eval's filters. When the eval runs, Dash0 replaces each reference with the attribute's value on the judged span. An attribute resolves from the span's attributes first, then from its resource attributes.

• Message content: Most evals reference gen_ai.input.messages and gen_ai.output.messages. These attributes are only present when your instrumentation captures message content. See About GenAI Spans.
• Missing values skip the span: If a referenced attribute is missing or empty on a span, Dash0 skips the span instead of judging it, unless Judge partial prompts is on. No verdict is recorded.
• Externally redacted values skip the span: If a referenced attribute's content was redacted before it reached Dash0 (for example, by source redaction in your instrumentation), Dash0 also skips the span, unless Judge partial prompts is on. GenAI Attribute Redaction runs after Evals, so Evals still judge content you redact on ingestion.
• Span fields resolve too: You can reference span fields such as otel.span.name, otel.span.kind, and otel.span.status.code. An empty field counts as missing, for example otel.parent.id on a root span.

The editor warns you about two common mistakes:

• Not a well-formed attribute reference, so it is sent to the judge as literal text. The text between the braces is not in the attribute.name form, so the judge receives it as typed.
• No matching span carries this attribute. Spans without it are skipped rather than judged. Check the attribute name, or adjust the span filters so the eval selects spans that carry it.

Judge Partial Prompts

By default, an eval only judges a span when every attribute the user prompt references has content. Turn on Judge partial prompts in the eval editor to also judge spans that carry only some of them:

• Missing attributes: The judge receives a marker such as <missing attribute gen_ai.output.messages> in place of each missing or empty attribute.
• Redacted attributes: The judge receives the redacted value as it reached Dash0.
• No usable content: Dash0 still skips a span when none of the referenced attributes has unredacted content.

With the setting on, the editor's warnings say that the judge sees a missing-attribute marker instead of saying that spans are skipped. Use it when an attribute is optional, for example a tool result that only some spans carry, and tell the judge in your prompt how to score a span with a missing-attribute marker.

Ask for the Right Output

Dash0 asks the judge for the output type you chose, plus a one-sentence explanation. Your prompt must describe what that output means:

• Numeric: Describe what a score of 1.0 and a score of 0.0 mean. Higher scores are always better, so phrase the criterion positively, for example "how well the answer is supported" rather than "how much the answer hallucinates".
• Boolean: Ask a yes/no question, phrased so true is the passing answer.
• Categorical: Describe when to choose each label. The judge always returns one of the labels you defined, even when none of them fits, so add a catch-all label such as other for everything else.

Examples

Numeric: Groundedness

• Output type: Numeric, with a Threshold of 0.7.

• System prompt:

  You evaluate whether an AI assistant's answer is supported by the context it was given.

• User prompt:

  Conversation and retrieved context:
  {{gen_ai.input.messages}}
  
  Assistant answer:
  {{gen_ai.output.messages}}
  
  Score how well every claim in the answer is supported by the context.
  1.0 means every claim is supported. 0.0 means the answer contradicts the context or invents facts.

Boolean: Answered the Question

• Output type: Boolean.

• User prompt:

  User question:
  {{gen_ai.input.messages}}
  
  Assistant answer:
  {{gen_ai.output.messages}}
  
  Does the answer directly address the question the user asked? Answer true if it does, false if it
  ignores, deflects, or answers a different question.

Categorical: Request Topic

• Output type: Categorical, with the labels billing (Neutral), technical (Neutral), account (Neutral), abuse (Negative), and other (Neutral).

• User prompt:

  User message:
  {{gen_ai.input.messages}}
  
  Classify the request:
  - billing: invoices, payments, refunds, or pricing
  - technical: product errors, configuration, or how-to questions
  - account: sign-in, profile, or permission changes
  - abuse: harassment, spam, or attempts to misuse the assistant
  - other: anything that fits none of the labels above

  Without other, the judge files an off-topic request, such as asking the support assistant for a poem, under the closest label, here technical. That skews the label mix.

Further Reading

• Create an Eval — Define which spans to judge, the judge model, and the output.
• GenAI Spans — Send the spans and attributes evals depend on.
• Key Concepts — Terminology for triggers, sampling, verdicts, and credits.
• Manage Evals — Enable, edit, version, and delete evals.