Formalize AI model evaluation with pass@k and pass^k metrics.
npx skills add https://github.com/nubiv/my-nome --skill eval-harness-nubiv
Please help me install this Agent Skill. Skill: eval-harness Source: https://github.com/nubiv/my-nome/tree/main/nix-darwin/config/claude/skills/eval-harness Command: npx skills add https://github.com/nubiv/my-nome --skill eval-harness-nubiv
This Skill provides a structured framework for evaluating AI model performance and reliability, enabling a more rigorous and systematic approach to AI development and testing.
Use the eval-harness skill to define a new capability evaluation for the 'user-authentication' feature.
High-intent search queries and answers about installing and using this skill.
Eval-driven development for AI formalizes model output evaluation by defining pass/fail criteria and success metrics before implementation. It uses capability and regression evals to systematically track AI agent reliability and ensure continuous performance improvement.
Evaluate AI agent reliability by running capability and regression evals using code-based, model-based, and human grading mechanisms. This framework tracks performance against defined pass/fail criteria to identify regressions during continuous development.
Pass@k and pass^k are metrics used to measure AI model performance and reliability. Pass@k tracks success rates across generation attempts, while pass^k helps benchmark consistency for continuous improvement and formal evaluation.
Yes, you can use code-based, model-based, and human grading mechanisms together for AI testing. This multi-modal approach supports capability and regression evals to thoroughly formalize the evaluation of AI model outputs.
You need a formal evaluation framework when developing new AI features that require rigorous testing and benchmarking. It allows you to define expected behavior upfront, track regressions, and systematically ensure agent reliability using formal metrics.
Eval-driven development requires defining success metrics and pass/fail criteria before implementation, demanding upfront planning. The framework focuses on capability and regression evals, meaning informal or ad-hoc testing without structured pass@k tracking falls outside its scope.
its-BasudebaDefine and run pass/fail evaluations with regression testing for AI agents.
flatrickDefine, measure, and track AI behavior with pass@k metrics.
ryasrkFormalize evaluation frameworks for AI code sessions with pass@k and pass^k metrics.
linnefromiceEvaluate Claude Code sessions with code-based, model-based, and human graders.
