Get alerted when your evals get worse
Eval results are now metrics. Create a check rule from any eval in one click, or from a template, to get notified when quality drops or to start an automation that investigates.
Evals tell you when your agents' answers get worse, but only if someone opens the evals page. A rising hallucination rate or a burst of prompt injection attempts could go unnoticed until a customer complained.
Dash0 now records eval results as two metrics, dash0.gen_ai.evaluation.score and dash0.gen_ai.evaluation.label, with the eval's name in gen_ai.evaluation.name. You can alert on them like on any other metric.

From an eval. Click Create check rule on an eval's page. The check rule editor opens with a rule that fits the eval's output type:
- Boolean, or numeric with a threshold: the share of failing spans.
- Categorical: the share of labels you marked Negative.
- Numeric without a threshold: the average score.
From a template. In the check rule templates, open AI Observability → Evals and pick Eval fail rate high, Eval label share high, or Eval average score low. These group results by eval name, so a single rule covers all your evals.
Both evaluate a 1-hour window and only fire once the eval has judged at least 20 spans in it, so a few early verdicts don't page you. A rule fires once when results cross the threshold, and again only after they recover and cross it again. Adjust the thresholds before you save.
Let an automation investigate. Link an automation to the check rule, or use the Failed Check trigger. When an eval gets worse, Agent0 looks into the failing spans, reports what it found to Slack or Linear with the judge's explanations and links to the traces, and can open a Linear issue with the evidence. An automation never changes an eval. It describes the fix for you to make in Dash0.
Good to know
- Each verdict now carries a polarity,
positive,neutral, ornegative, indash0.gen_ai.evaluation.score.polarity. Filter onnegativeto find bad verdicts across numeric, boolean, and categorical evals. Verdicts recorded before November 2026 have no polarity. - Only Admins can use Create check rule, and only on the current version of a saved eval.
- Categorical evals offer Create check rule only if one of their labels is marked Negative.
Read the docs: Alert on Eval Results
Norbert Schneider
Stephen Elson











