Define, measure, and track AI behavior with pass@k metrics.
npx skills add https://github.com/flatrick/mdt --skill eval-harness-flatrick
Please help me install this Agent Skill. Skill: eval-harness Source: https://github.com/flatrick/mdt/tree/main/skills/eval-harness Command: npx skills add https://github.com/flatrick/mdt --skill eval-harness-flatrick
This Skill provides a structured framework for evaluating AI workflows, enabling developers to implement eval-driven development (EDD) principles and ensure reliability.
Use the eval-harness skill to define a new capability evaluation for the user registration feature.
High-intent search queries and answers about installing and using this skill.
Eval-driven development formalizes AI behavior tracking by defining pass/fail criteria before implementation. You measure agent reliability using pass@k metrics and continuously run evaluations to ensure intended functionality and prevent regressions.
Preventing regressions in AI workflows requires creating targeted evaluation test suites. You define capability evaluations with specific expected behaviors, then continuously run these checks during development to catch unintended side effects from code changes.
You measure AI agent reliability using pass@k and pass^k metrics. These evaluations quantify performance by tracking how consistently an agent meets predefined pass/fail criteria across multiple attempts, enabling precise capability assessment.
Yes, capability evaluations support code-based, model-based, and human grading types. This allows you to flexibly define expected behavior and automate the grading process to measure agent performance within your development workflow.
Yes, eval-driven development supports regression testing for AI features. You define capability evaluations outlining expected behavior, then continuously run these test suites as you code to prevent unintended side effects and maintain reliability.
Predefined pass/fail criteria are needed for AI evaluation to clearly outline expected behavior before implementation begins. This formalizes the definition of success, allowing objective measurement of agent performance and prevention of regressions.
maltemdDesign evaluation frameworks with metrics and pass/fail thresholds for LLM-based agents.
cbbkrd-techAutomates execution of pytest, jest, vitest and go test suites to detect regressions.
nubivFormalize AI model evaluation with pass@k and pass^k metrics.
linnefromiceEvaluate Claude Code sessions with code-based, model-based, and human graders.
