skill-eval

To benchmark Claude Code skills and detect regressions, you can snapshot a skill before editing to establish a baseline, then run realistic prompts against before and after versions with multiple runs per eval. The process uses isolated claude -p runs and assertion-based grading to produce reproducible results.

Skill evaluation measures and compares Claude Code skill behavior to detect regressions and improvement gaps. Blind comparison works by grading responses with cited evidence, then aggregating pass rates, timing, and variance to understand why one version performs better without bias.

You measure flakiness and variance in Claude Code skill responses by running multiple isolated claude -p executions against realistic prompts. The skill applies statistical aggregation to pass rates and timing data, producing reproducible benchmark results that highlight inconsistency.

Yes, you can use assertion-based grading to validate new Claude Code skills. The skill-eval workflow allows you to create realistic eval prompts, run before-and-after benchmarks, and grade each response with cited evidence to prove a new skill adds value.

The best way to compare before and after versions of a Claude Code skill is using isolated runs, statistical aggregation, and post-hoc analysis. This approach snapshots the baseline, applies realistic prompts, and generates reproducible benchmark results with evidence.

If your Claude Code skill feels inconsistent, you can apply skill-eval for variance analysis. It runs isolated benchmarks, grades responses with cited evidence, and uses post-hoc analysis to identify flakiness and understand the root causes of behavioral regressions.