Define and run capability and regression evals for Claude Code sessions.
npx skills add https://github.com/fideguch/my_dotfiles --skill eval-harness-fideguch
Please help me install this Agent Skill. Skill: eval-harness Source: https://github.com/fideguch/my_dotfiles/tree/main/claude/skills/eval-harness Command: npx skills add https://github.com/fideguch/my_dotfiles --skill eval-harness-fideguch
Provides a formal evaluation framework to design, run, and interpret evaluations for Claude Code sessions, enabling reliable validation of AI behavior and progress.
Define your evals, run them with the harness, and review the resulting reports to guide development.
High-intent search queries and answers about installing and using this skill.
Pass@k and pass^k are standardized evaluation metrics used to track AI performance changes over time. They provide objective scoring for capability and regression evals, allowing you to benchmark progress and validate behavior across different model versions within your evaluation framework.
Capability evals test new features and validate AI behavior, while regression evals guard against performance degradation in existing workflows. Both use the formal evaluation framework's standardized metrics and structured storage to track changes and ensure objective pass/fail criteria are met.
Eval-driven development for AI-assisted workflows is a formal process to define, implement, evaluate, and report tests with objective pass/fail criteria. It enables reliable validation of AI behavior and progress using standardized metrics and structured storage to track changes over time.
You evaluate AI behavior changes across model versions by running capability and regression evals through the harness, applying standardized metrics like pass@k, and comparing results stored in the structured evaluation storage to track performance benchmarks over time.
jinyuanluDefine and run eval-driven development tests for Claude Code sessions.
LuohaothuDefine eval criteria and track pass@k metrics for AI development.
rongaredeEvaluate Claude Code sessions with capability and regression evals.
statsclawAutomate Monte Carlo simulation studies evaluating statistical estimator performance.
