eval-harness

You run regression evals for Claude Code by applying an eval-driven framework that uses pass@k metrics and grader patterns to detect regressions and ensure quality. This automates systematic feature testing across versioned eval storage.

Eval-driven validation is a framework that automates systematic feature testing for Claude Code work. It applies capability and regression evals using pass@k metrics and grader patterns to validate changes and ensure quality.

Pass@k metrics in capability evaluations measure the probability that at least one of k generated attempts passes the defined grader patterns. This provides a systematic way to validate feature performance and detect regressions.

Yes, you can use code-based, model-based, and human graders together. The framework supports deterministic, model-based, and human-grade evaluations with defined workflows and versioned storage to satisfy diverse grading requirements.

No, you do not need external dependencies to run eval-driven testing. The framework operates independently to define and run evals for Claude Code changes, measuring capability and regression performance with integrated storage.