Eval Harness Skill

Eval-driven development (EDD) is a formal evaluation framework for AI development that defines expected AI behavior before implementation. It evaluates AI capabilities and ensures code quality through structured testing, akin to unit testing for AI development sessions.

Use the /eval define feature-name command to create an eval definition file in a structured project directory. This establishes capability evals for new functionalities and regression evals to ensure existing AI behaviors are not broken during development.

Yes, this formal evaluation framework supports code-based, model-based, and human graders for AI regression testing. It tracks reliability metrics like pass@k and pass^k to assess the correctness of AI capabilities across development sessions.

Track AI reliability metrics by implementing a formal evaluation framework that monitors pass@k and pass^k metrics. It integrates via commands for defining, checking, and reporting on evals to assess AI capability reliability throughout development.

No external dependencies are required to set up this AI evaluation framework. It operates independently using scripts and references to define, check, and report on evals within a structured project directory.