skills.rest
greglas75
greglas75COMMUNITY
Explore all skills by this author →
GitHub

skill-eval

Evaluates skill behavior by running eval corpora in isolated executor sub-agents and grading transcripts.

6|Updated Mar 27, 2026
Search Tags:#claude code#regression testing#skill evaluation#behavioral testing#sub-agent#transcript grading#eval corpus
One-click install
npx skills add https://github.com/greglas75/zuvo --skill skill-eval-greglas75
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: skill-eval
Source: https://github.com/greglas75/zuvo/tree/main/skills/skill-eval
Command: npx skills add https://github.com/greglas75/zuvo --skill skill-eval-greglas75

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? It measures whether a skill's own instructions actually lead a competent agent to the correct behavior, replacing guesswork with per-assertion pass/fail evidence from real tool-call transcripts. ## Core Features & Use Cases - Behavioral Evaluation: Runs each eval case from a skill's corpus (evals/<skill>.evals.json) in a fresh, isolated executor sub-agent, then grades the transcript against fixed assertions with an injection-hardened grader. - Old-vs-New Comparison: The --compare <ref> mode materializes the old skill version from git and diffs pass rates against the current version on the same benchmark. - Honest Reporting: Writes per-assertion pass/fail reports with verbatim evidence to zuvo/reports/, separating infra failures from behavioral results and flagging inconclusive runs. - Use Case: After editing a skill's SKILL.md, run the evaluator to confirm the new instructions still produce correct agent behavior before shipping, or compare against a previous git ref to detect regressions. ## Quick Start Run the skill-eval skill on the 'refactor' skill to grade its eval corpus and write a pass/fail report.

Frequently Asked Questions about skill-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate whether a Claude Code skill actually works?▼

Run skill-eval with the skill name; it loads the skill's eval corpus from evals/<skill>.evals.json, executes each case in a fresh isolated executor sub-agent, and grades the tool-call transcript against the corpus assertions, producing a per-assertion pass/fail report.

How do I compare a skill's behavior before and after an edit?▼

Use the --compare <ref> flag with a git ref. The evaluator materializes the old skill version via git clone and checkout, grades it against the current corpus with the same grader, and reports an old-vs-new pass-rate diff per assertion.

Can skill-eval run from an installed plugin cache?▼

No. It is dev-only and fails closed with BLOCKED_DEV_ONLY if no evals/ corpus directory exists in the working tree, since it needs the repository's eval corpus and optionally git history for comparisons.

Why does skill-eval stop with BLOCKED_NO_TRANSCRIPT_CAPTURE?▼

The grader scores real tool calls, so the evaluator runs a canary probe first to verify executor transcripts can be captured. If neither auto-capture nor ACTION_LOG mode surfaces the canary markers, it stops rather than emitting a vacuous all-false report.

How does the grader resist prompt injection from transcripts?▼

The transcript is wrapped in a random per-run nonce fence, closing-fence lookalikes are neutralized with a regex, and the grader has no tools and treats everything inside the fence as untrusted data, following only the rules outside it.

Related Skills

Explore More Software Engineering Skills→
whatswithaviwhatswithavi

skill-creator

Create, evaluate, and iteratively improve Claude skills with benchmarked test runs.

Community
Advanced
shellicarshellicar

skill-repair

Diagnose failing skill scripts, propose fixes, and re-run original steps.

Community
Intermediate
biwakonbubiwakonbu

plugin-validation

Validate Claude Code plugin structures and metadata for configuration errors.

Community
Basic
wan-huiyanwan-huiyan
3

task-framing-claims-need-subagent-grep-verify

Instructs write-content sub-agents to grep-verify and tag corrections to dispatcher task-framing claims.

Community
Basic