eval-harness

Define the expected behavior and evaluation criteria before implementing code changes to set up regression evals. This framework supports eval-driven development by applying checks across code modifications and prompts to ensure reliability.

Eval-driven development is the practice of defining expected behavior before implementation to track progress across iterations. It applies to capability evals by generating standardized reports with pass@k and pass^k metrics to quantify risk and ensure reliable AI development.

Yes, you can automate grading using deterministic code-based graders and model-based assessments. This automation applies across human reviews and code changes to generate standardized evaluation reports that quantify progress and risk.

Run your defined eval checks after implementing code changes to generate evaluation reports with pass@k and pass^k metrics. The framework automates grading and produces standardized reports to quantify progress across development iterations.

No, the evaluation framework requires no external dependencies to run capability evals. It operates independently within Claude Code sessions to define evals, implement checks, and generate standardized reports.