Beta

Evals is in beta. Eval runs are not charged during the beta. Behavior and UI may change.

The template catalog holds 28 predefined evals that cover common ways to judge GenAI spans. A template fills in the eval editor: the name and description, the span filters, the sample rate, the judge model class, the output type, and both prompts. You review the draft, adjust it, and save it like any other eval.

Later changes to a template do not update evals you already created from it.

Create an Eval from a Template

You need the same permissions as for any new eval. See Create an Eval.

Open AI Observability → Evals.
Click Create new. If the dataset has no evals yet, click New eval instead.
Find a template. Type in Search templates... to match template names and descriptions, or click a category to narrow the list. All categories shows every template. Each category shows how many templates match your search.
Click a template. The eval editor opens with the template's settings filled in.
Review the settings. See Review a Template Before You Save.
Click Save.

To start from an empty editor instead, click the arrow next to Create new and select Manually.

Review a Template Before You Save

A template is a starting point. Check these settings against your spans before you save:

Name: Each dataset requires a unique eval name. If the dataset already has an eval with the template's name, for example from the same template, change the name.
Span filters: Templates select spans by operation type, and most also require the attributes their prompts read, for example gen_ai.input.messages is set. Spans without captured message content do not match, so they are not sampled. Because these filters go beyond a trigger preset, the Trigger shows as customized, for example Any GenAI operation (customized). Add filters such as service.name or gen_ai.agent.name to judge only one agent. Use the Matching spans panel to confirm that the template selects spans in your dataset.
Sample rate: Every template starts at 20%. Raise it for low-volume agents, or lower it to reduce credit usage on high-volume agents.
Model class: Each template uses the model class listed in the Template Catalog. Templates that need closer reading, such as Hallucination and Prompt injection, use Standard. See Model Class.
Prompts: Each template's prompts tell the judge which labels to pick and how to break ties. Edit them to add rules specific to your application. See Write Eval Prompts.

You cannot change the output type after the eval is created, so pick a different template or start manually if you need a different one.

Template Catalog

The catalog groups templates into five categories. The Top picks group at the top contains commonly used evals.

Response Quality

Judge whether responses are relevant, correct, complete and follow instructions.

Template	Judges	Output	Model class
Answer relevance	Whether the response addresses what the user asked.	Categorical: `relevant`, `partially_relevant`, `irrelevant`	Light
Correctness	Whether the response is free of factual and logical errors, judged without a reference answer.	Boolean	Standard
Completeness	Whether the response handles every part of the request.	Categorical: `complete`, `partial`, `incomplete`	Light
Conciseness	Whether the response avoids filler and repetition.	Boolean	Light
Refusal	Whether the assistant answered or declined, and whether that was the right call.	Categorical: `answered`, `appropriate_refusal`, `unnecessary_refusal`, `inappropriate_compliance`, `empty`	Standard
Instruction adherence	Whether the response follows the system instructions.	Boolean	Standard
Language match	Whether the response is in the user's language.	Boolean	Light
Summarization quality	How well a summary captures its source.	Numeric, threshold 0.7	Standard
SQL generation	Whether generated SQL correctly answers the question.	Boolean	Standard
Code correctness	Whether generated code does what the user asked.	Boolean	Standard
Code readability	How easy generated code is to read and maintain.	Numeric, threshold 0.5	Light

SQL generation only matches responses that contain a fenced sql block or upper-case SQL statements. Code correctness and Code readability only match responses with a code block tagged with a common programming language, such as ```python or ```go.

Grounding / RAG

Check that responses and retrieval steps stay grounded in the context the model was given.

Template	Judges	Output	Model class
Hallucination	Whether the response's claims are supported by the context the model was given.	Categorical: `grounded`, `partially_grounded`, `hallucinated`, `no_context`	Standard
Retrieval relevance	Whether a retrieval step returned documents that help answer the query.	Categorical: `relevant`, `partially_relevant`, `irrelevant`	Light
Context sufficiency	Whether the context given to the model was enough to answer the request.	Categorical: `sufficient`, `partial`, `insufficient`	Light
Citation correctness	Whether the response's citations and links point to sources that support the cited statements.	Boolean	Standard

Retrieval relevance matches spans where gen_ai.operation.name is retrieval and that carry gen_ai.retrieval.query.text and gen_ai.retrieval.documents. Citation correctness only matches responses that contain a URL or a numbered reference such as [1].

User Experience

Read how users react to the assistant.

Template	Judges	Output	Model class
User frustration	How the user feels in their latest message.	Categorical: `satisfied`, `neutral`, `frustrated`	Light
User correction	Whether the user accepted the assistant's previous answer without correcting it.	Boolean	Light

Agents and Tools

Judge how agents pick and call tools, and whether they finish their task.

Template	Judges	Output	Model class
Tool selection	Whether the model called the right tools, or correctly called none.	Boolean	Standard
Tool arguments	Whether the arguments of a tool call are well formed and plausible.	Boolean	Light
Tool result handling	Whether the model used tool results correctly.	Boolean	Standard
Agent task completion	Whether an agent run achieved the task it was given.	Categorical: `completed`, `partial`, `failed`	Standard

Tool selection needs gen_ai.tool.definitions on the span. Tool arguments judges execute_tool spans and skips calls whose gen_ai.tool.call.arguments contain <REDACTED>. Agent task completion judges invoke_agent spans.

Safety and Security

Catch harmful content, leaked data and attacks on the model.

Template	Judges	Output	Model class
Harmful content	Whether the response contains harmful content, and of what kind.	Categorical: `none`, `harassment`, `hate`, `sexual`, `illicit`, `violence`, `self_harm`, `extremism`, `weapons_cbrn`, `sexual_minors`	Light
Bias	Whether the response is free of stereotypes and demeaning generalizations.	Boolean	Light
PII leakage	Whether the response is free of personal information the user didn't provide.	Boolean	Standard
Secrets leakage	Whether model responses and tool results are free of exposed credentials.	Boolean	Light
System prompt leakage	Whether the response keeps the system instructions confidential.	Boolean	Light
Prompt injection	Whether the input to the model contains an attempt to hijack it, directly in user messages or indirectly through tool results and documents.	Categorical: `none`, `suspicious`, `injection`, `jailbreak`	Standard
SQL injection	Whether a tool call's arguments are free of SQL injection.	Boolean	Light

Secrets leakage judges both model calls and execute_tool spans. SQL injection judges execute_tool spans and skips calls whose gen_ai.tool.call.arguments contain <REDACTED>.

Data Handling

Templates send span content to the judge model, the same as any other eval. PII leakage and Secrets leakage can only find sensitive data that your spans carry. See About Evals.

Eval Templates