agent-eval

Start by running realistic inputs through the system and reading every output to categorize observed failure modes. Turn those categories into eval dimensions, build a 50-100 row golden set including failures, then layer programmatic checks, LLM-as-judge graders, and human review.

Write one judge per dimension with a rubric of checkable criteria, explicit negative examples, and few-shot anchors spanning the score range. Ask for chain-of-thought before the verdict, emit structured JSON output, and calibrate against human labels to high-80s-90s percent agreement.

pass@k measures whether an agent succeeds in at least one of k trials, showing capability. pass^k measures success in all k trials, showing reliability, which is what users actually experience; a 75 percent per-trial agent has pass@4 near 99 percent but pass^4 near 32 percent.

Grade three layers separately: the trajectory layer for correct sub-agent and workflow invocation, the tool layer for invocation and parameter accuracy, and the outcome layer for final environment state. Run k independent trials per task in fresh environments with complete transcripts captured.

Disagreement usually comes from ambiguous rubric criteria, missing few-shot anchors for near-miss cases, or known biases like verbosity and position bias. Read every disagreement, classify it as judge error, label error, or rubric ambiguity, and iterate until agreement reaches the target.

Do not automate when the rubric is ambiguous and human experts themselves disagree on labels, since no judge can outperform the humans it imitates. Keep high-stakes ambiguous calls with human experts until the rubric stabilizes.