Eval Harness Skill

Use code-based, model-based, and human graders to evaluate AI performance. This framework supports running capability evals for new features and regression evals to ensure existing functionality remains intact during AI development.

Capability evals test new features while regression evals ensure existing functionality remains intact. You can define both types before implementation to establish success criteria, then execute them using code-based, model-based, or human graders.

Pass@k and pass^k are key metrics tracked during AI evaluation to quantify model reliability. Monitoring these metrics throughout development ensures that regressions are detected and prevented across capability and regression evals.

Yes, the formal AI evaluation framework supports code-based, model-based, and human graders simultaneously. Combining these grader types allows for comprehensive evaluation of both new capabilities and existing regressions.

Implement a formal evaluation framework that tracks pass@k and pass^k metrics. By differentiating between capability and regression evals, you can systematically measure AI model performance and prevent regressions.