Defines and runs pass/fail evaluations for AI-assisted coding tasks.
npx skills add https://github.com/vrcms/everything-qwen-code --skill eval-harness-vrcms
Please help me install this Agent Skill. Skill: eval-harness Source: https://github.com/vrcms/everything-qwen-code/tree/main/.qwen/skills/eval-harness Command: npx skills add https://github.com/vrcms/everything-qwen-code --skill eval-harness-vrcms
This skill addresses the lack of reliability in AI-assisted coding by providing a structured framework to define, run, and report on evaluation criteria before and during development.
Use the eval-harness skill to define a new evaluation suite for the current feature by running the command to initialize the evaluation definition file.
High-intent search queries and answers about installing and using this skill.
Eval-driven development establishes formal pass/fail criteria before and during AI-assisted coding sessions to ensure consistent agent performance and reliability. It provides a structured framework to define, run, and report on evaluation criteria.
Set up regression testing by initializing an evaluation definition file to establish pass/fail criteria. The framework tracks pass@k and pass^k metrics to measure agent stability and regression resistance over time.
Yes, model-based evaluation is supported for open-ended outputs. The framework facilitates multi-modal grading, including deterministic code-based checks, model-based evaluation, and human-in-the-loop adjudicated review.
Benchmark AI agent reliability by tracking pass@k and pass^k metrics within a formal evaluation framework. This measures agent stability and regression resistance to ensure high-quality output during development.
Human-in-the-loop review is needed for complex evaluation criteria requiring human adjudication. It complements deterministic code-based checks and model-based evaluation to ensure high-quality output and stability.
SeanChenRAutomate evaluation and regression tracking for AI-assisted coding sessions.
kk20300113-pngDefine pass/fail criteria and calculate pass@k metrics for Claude Code agent evals.
ysyecustCodify and automate Claude Code session evaluations with pass@k metrics.
synqingValidate KiCad PCB designs against DRC and DFM rules.
