skill-creator

To create and benchmark AI skills, you draft a SKILL.md, generate test prompts, run baseline and with-skill comparisons, grade outputs, and aggregate benchmark statistics to evaluate performance. This skill automates that entire evaluation workflow.

Yes, you need the claude CLI and Python environment with pyyaml installed to run the evaluation scripts and description optimization loops. These dependencies support local benchmarking utilities and packaging tools.

The best way to automate description optimization is running iterative improvement loops that refine SKILL.md descriptions, rerun test prompts, and analyze pass rates. This skill uses the claude CLI to optimize descriptions based on benchmark results.

Grading and benchmarking works by producing eval sets, running with-skill and baseline comparisons, grading outputs against assertions, and aggregating statistics to surface high-variance or nondiscriminating test cases for iterative skill improvement.

Yes, you can package validated skills for distribution after benchmarking by running the packaging tools that compile the tested SKILL.md, scripts, references, and assets into a distributable .skill file.

High variance in eval assertions occurs when test prompts lack specificity or the SKILL.md trigger conditions are ambiguous. This skill surfaces nondiscriminating assertions during benchmark aggregation to help refine and stabilize outputs.