skills.rest
malinovskiy-makar
malinovskiy-makarCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Defines and runs capability and regression evals for Claude Code workflows using pass@k metrics.

Updated Jun 13, 2026
Search Tags:#claude code#llm evaluation#regression testing#eval-driven development#grader#pass@k#agent reliability
One-click install
npx skills add https://github.com/malinovskiy-makar/qls --skill eval-harness-malinovskiy-makar
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/malinovskiy-makar/qls/tree/main/.claude/skills/eval-harness
Command: npx skills add https://github.com/malinovskiy-makar/qls --skill eval-harness-malinovskiy-makar

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? AI-assisted coding workflows lack a formal way to verify that Claude Code sessions actually accomplish their tasks and that prompt or agent changes do not break existing behavior. This Skill brings eval-driven development (EDD) to Claude Code, treating evals as the unit tests of AI development. ## Core Features & Use Cases - Capability and Regression Evals: Define pass/fail criteria before implementation and verify that changes do not break existing functionality. - Multiple Grader Types: Use deterministic code graders, model-based LLM-as-judge graders, rule graders, or human review depending on the output type. - Reliability Metrics: Track pass@1, pass@3, and pass^3 metrics with recommended thresholds for release-critical paths. - Use Case: Before shipping a new authentication feature built with Claude Code, define capability evals (user registration, login, session persistence) and regression evals (existing routes unchanged), run them, and generate an eval report confirming pass@3 of 100% before merging. ## Quick Start Ask the agent to define an eval for a feature you are about to build, for example: define capability and regression evals for the new authentication feature before writing any code.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up eval-driven development for Claude Code?▼

Define capability and regression evals before writing code, storing them in .claude/evals/<feature>.md. Run the evals during implementation, then generate an eval report tracking pass@k metrics to decide if the change is ready to ship.

What is the difference between pass@k and pass^k metrics?▼

pass@k measures whether at least one of k attempts succeeds, while pass^k requires all k attempts to succeed. Use pass@3 above 90% for capability evals and pass^3 at 100% for release-critical regression paths.

When should I use a code grader versus a model grader?▼

Use code graders for deterministic checks like grep patterns, test runs, or build success since they are reproducible. Use model graders only for open-ended outputs where quality cannot be captured by assertions, and human graders for security-sensitive changes.

Can evals catch regressions from prompt or model changes?▼

Yes, regression evals record a baseline SHA or checkpoint and re-run existing tests after each change. The eval report compares current results against the baseline to flag any broken functionality.

What are common eval anti-patterns to avoid?▼

Avoid overfitting prompts to known eval examples, measuring only happy-path outputs, ignoring cost and latency drift, and allowing flaky graders in release gates. Keep evals fast and version them alongside the code.

Related Skills

Explore More Software Engineering Skills→
The-Skyy-Rose-Collection-LLCThe-Skyy-Rose-Collection-LLC
3

eval-harness

Define, run, and report evaluations with /eval commands.

Official
Intermediate
pjherronpjherron

eval-harness

Implements eval-driven development with pass/fail criteria and pass@k metrics for AI coding sessions.

Community
Intermediate
BagelHoleBagelHole
46

agent-observability

Instrument AI agents with trace IDs and child spans for LLM calls.

Community
Intermediate
coreyjacoreyja
6

Testing Test Writing

Write behavior-focused unit, integration, and end-to-end tests with mocked dependencies.

Community
Basic