agent-evaluation

Run each test multiple times and analyze the distribution of results rather than expecting a single deterministic output. Define pass thresholds as statistical properties, such as success rate above a target percentage across N runs.

Behavioral contract testing defines invariants an agent must always satisfy, such as never leaking credentials or always citing sources, then verifies those properties hold across many inputs. It replaces brittle exact-output string matching with property-based assertions.

Benchmarks often differ from production inputs in distribution, phrasing, and edge cases, and agents can be overfit to the metric. Bridge the gap with production-like evaluation sets, adversarial tests, and multi-dimensional metrics instead of a single score.

Treat flakiness as a signal: quantify pass rates over repeated runs and set explicit reliability thresholds. Separate genuinely unstable agent behavior from poorly designed tests that depend on exact wording.

Exact string matching rejects correct answers phrased differently and accepts wrong answers that happen to match. Use semantic similarity, rubric-based grading, or behavioral invariants to judge correctness instead.