Implements formal AI evaluation framework with graded capability and regression evals.
npx skills add https://github.com/XD3an/cc-plus --skill eval-harness-skill
Please help me install this Agent Skill. Skill: Eval Harness Skill Source: https://github.com/XD3an/cc-plus/tree/main/skills/eval-harness Command: npx skills add https://github.com/XD3an/cc-plus --skill eval-harness-skill
This Skill provides a structured framework for evaluating AI development, ensuring predictable and reliable AI behavior through formal evaluation processes.
Use the eval harness skill to define a new capability evaluation for the 'user-registration' feature.
High-intent search queries and answers about installing and using this skill.
Eval-driven development formalizes AI testing by defining expected behavior through capability and regression evals before coding, ensuring predictable behavior and measuring reliability with pass@k and pass^k metrics.
To set up capability and regression evals, define the expected behavior for new features as capability evals, while creating regression evals to ensure existing functionality remains unaffected during development.
You can evaluate AI behavior using three grading mechanisms: code-based graders for automated checks, model-based graders for AI-assisted evaluation, and human graders for comprehensive manual assessment.
The evaluation framework operates independently without requiring specific external dependencies, allowing you to implement formal AI evaluation processes directly within your existing development environment.
Regression testing within this framework measures AI reliability by tracking pass@k and pass^k metrics, providing quantitative data to ensure existing functionality remains intact across development iterations.
VjayRamDefine DSPy signatures and modules with deterministic evaluation across splits.
LincyawImplement eval-driven development with code, model, and human graders.
bugrabilgeDesign benchmark test suites for evaluating LLM agent reliability.
as4584Evaluate LLM applications with automated metrics and human feedback.
