Eval Harness Skill

Define a capability evaluation by running the command /eval define followed by your feature name. This creates a structured evaluation framework integrated with your project structure for eval storage and versioning.

Run regression evals to test existing AI functionalities against defined success criteria. This framework tracks pass@k and pass^k metrics to identify and prevent unintended changes introduced by new code modifications.

Use code-based, model-based, or human grading mechanisms for AI evaluation. These approaches provide comprehensive options for testing AI capabilities and ensuring formal evaluation criteria are met.

Yes, the framework monitors pass@k and pass^k metrics for reliability tracking. It integrates with your project structure for eval storage and versioning to maintain formal AI evaluation standards.

Yes, use capability evals for new AI functionalities and regression evals for existing features. The framework supports code-based, model-based, and human grading to ensure comprehensive code quality.