evaluators

Derive the grading task from your hypothesis, choose a small mutually exclusive label set, then write a judge prompt whose rubric ties each label to observable evidence in the run's output fields. Calibrate against several representative cases before saving the evaluator.

Use a deterministic code evaluator whenever the judgment can be computed, such as exact-match, regex, JSON distance, or structural tool_calls checks. Reserve LLM judges for reading-comprehension or open-ended quality judgments that primitives cannot settle.

A golden reference is a hand-labeled ideal output where deviation is a defect, warranting strict comparison. A baseline-snapshot is a captured current-state output used only to detect behavior change, so losing to it is a signal rather than a verdict.

Choose a small, mutually exclusive, collectively exhaustive set, often binary like correct/incorrect or pass/fail. Add labels only to distinguish failure modes that matter, since every extra label introduces boundary ambiguity.

Mislabeling usually comes from an ambiguous rubric, wrong field selection, or unrepresentative test cases. Change one thing at a time between rubric, logic, labels, and cases, and fix unrepresentative cases before blaming the evaluator logic.

Do not use it for manual prompt drafting, which belongs to the playground skill, for running or comparing experiments, which belongs to the experiments skill, or for cross-trace failure diagnosis with no evaluator in scope, which belongs to debug-trace.