eval-harness

You can set up regression checks by defining structured capability and regression evaluations with pass/fail criteria. The framework allows you to track regressions across model versions and generate automated evaluation reports for your AI sessions.

Pass@k is a formal scoring metric supported by this evaluation framework to measure code generation reliability. It evaluates whether at least one successful outcome exists within k attempts, providing quantitative assessment for AI-generated code tasks.

You can run evaluations using the built-in /eval CLI workflow. Define your evaluation criteria, select from multiple grader types, and execute the evaluation to generate pass/fail metrics and formal scoring reports.

Yes, the framework supports tracking regressions across model versions. By defining consistent evaluation criteria and running deterministic evaluations, you can compare reliability metrics and identify performance degradations between different AI model versions.

No, the evaluation harness operates without external dependencies. It provides a self-contained eval storage pattern and integrated CLI workflow, allowing you to define, run, and report evaluations directly within your development environment.

The framework supports multiple grader types for deterministic evaluation runs. You can define custom pass/fail criteria for capability and regression evaluations, enabling flexible code assessment tailored to your specific AI testing requirements.