Agent evaluation and observability glossary

Updated 2026-10-12

Use these definitions to connect a quality question to the evidence a tool collects. Each term links to a primary source; the feedback control lets you flag an unclear definition.

Evaluation datasetsource

A collection of input cases and any reference outputs or criteria used to run a repeatable evaluation experiment. Keep cases representative of the tasks your users actually perform.

Assertionsource

A rule checked against an output, such as a required JSON shape, a text condition or a scored rubric. Passing an assertion only establishes that the selected condition passed.

Tracesource

A record of one application execution, linking steps such as model calls and tool use so an unexpected response can be investigated in context.

LLM as a judgesource

Using a language model to assess an output against defined criteria. The resulting score is an evaluation signal that needs calibration against the task and human judgments.

Groundednesssource

Whether the claims in a generated answer are supported by the retrieved context. An answer can be grounded while the source itself is wrong or incomplete.

Context relevancesource

Whether retrieved material is relevant to the user question. Inspecting this separately from the answer helps distinguish retrieval problems from response-generation problems.

Red teamingsource

Deliberately testing adversarial scenarios to expose unwanted behavior, such as prompt injection or disclosure. A test campaign covers the scenarios it exercises rather than certifying all future behavior.