Evaluate Claude Code sessions with capability and regression evals.
npx skills add https://github.com/choub00/transformer --skill eval-harness-choub00
Please help me install this Agent Skill. Skill: eval-harness Source: https://github.com/choub00/transformer/tree/main/.cursor/skills/eval-harness Command: npx skills add https://github.com/choub00/transformer --skill eval-harness-choub00
Formalized evaluation framework for Claude Code sessions enabling reliable, repeatable testing of prompts and models.
Initialize an eval harness for Claude Code sessions by defining your first capability and regression evals.
High-intent search queries and answers about installing and using this skill.
Run pass@k metrics for Claude Code prompts by using the framework's built-in graders and lightweight automation. It calculates pass@k scores across multiple model attempts to quantify prompt reliability and model capability.
Eval-driven development for Claude Code is a testing methodology where prompt and model changes are validated through formalized evaluation workflows. It uses capability and regression evals to generate repeatable checks and reports before deployment.
Set up an evaluation framework for Claude Code prompts by initializing structured eval definitions within the .claude/evals directory. Accessible commands then run automated checks and generate reports to test prompts and models reliably.
Yes, you can use eval-harness without external dependencies because it is a self-contained community framework. It relies on lightweight automation and native Claude Code session commands to execute capability and regression evaluations.
Capability evals test new prompt features and model abilities, while regression evals ensure existing functionalities remain intact after changes. Both evaluation types are supported through the structured .claude/evals definitions and pass@k grading system.
andrew-starosciakEvaluate Claude Code sessions with pass@k scoring and structured reports.
xu-xiangDefine, execute, and report capability and regression evals for Claude Code sessions.
paulinghamValidate internal-eval flows with a stubbed pipeline and managed phases.
ryasrkFormalize evaluation frameworks for AI code sessions with pass@k and pass^k metrics.
