Automate eval-driven testing and regression verification for AI code evaluation.
npx skills add https://github.com/devsepnine/hibi_ai --skill eval-harness-devsepnine
Please help me install this Agent Skill. Skill: eval-harness Source: https://github.com/devsepnine/hibi_ai/tree/main/src/skills/eval-harness Command: npx skills add https://github.com/devsepnine/hibi_ai --skill eval-harness-devsepnine
Eval Harness provides a formal framework for eval-driven development, enabling capability and regression testing with pass@k metrics to assess AI outputs during Claude Code sessions.
Define a new eval plan with capability and regression tests and run the eval workflow.
High-intent search queries and answers about installing and using this skill.
Pass@k metrics quantify AI success rates across multiple generation attempts. Eval Harness calculates these metrics during testing to measure the probability of achieving a correct outcome within k tries, providing statistical confidence in model capabilities.
You can set up regression tests by defining an eval plan with specific regression evals. The framework automates regression verification across Claude Code sessions and deployment workflows to detect performance degradation after model iterations.
Yes, model evaluation supports flexible graders including code-based, model-based, and human evaluation. This allows you to combine automated verification with manual review for comprehensive quality assurance of AI outputs.
To detect regressions across model iterations, run regression evals through the eval storage workflow. This applies standardized tests against new model versions to identify and prevent performance drops before deployment.
Eval-driven testing integrates directly with deployment workflows to ensure stable performance. It applies automated capability and regression verification before integration, preventing regressions from reaching production environments.
LuohaothuDefine eval criteria and track pass@k metrics for AI development.
linnefromiceEvaluate Claude Code sessions with code-based, model-based, and human graders.
D0NMEGAAutomate agent reliability assessments for Claude Code sessions with eval-driven workflows.
TakMczkImplement Eval-Driven Development evaluations with code, model, and human graders.
