skill-creator

To create and test AI skill definitions, you draft a SKILL.md file with frontmatter and body, generate realistic eval prompts, run with-skill and baseline comparisons, and grade outputs to evaluate reliability. The workflow supports iterative improvement through benchmarked pass-rate and token delta metrics.

The best way to benchmark AI skill performance is to spawn parallel with-skill and baseline runs, apply drafted assertions to grade outputs, and aggregate metrics to surface pass-rate, time, and token deltas. This comparison establishes a quantifiable performance baseline for the skill module.

Grading skill outputs works by applying drafted assertions to the results of parallel with-skill and baseline runs. The grader evaluates these outputs against the assertions, generating structured eval artifacts and analyzer outputs that surface pass-rate, time, and token deltas for comparison.

Yes, you can package validated skills and generate viewer artifacts using Python. The workflow relies on the pyyaml dependency to manage skill definitions and includes packaging utilities to distribute the validated skill modules after benchmarking and human feedback are satisfactory.

You need local CLI tools available in your environment to invoke the full skill evaluation loop. The workflow invokes these local CLI tools when available to execute the parallel runs, grade outputs, and generate the benchmark and viewer artifacts required for complete automation.

You should iterate on skill prompts and descriptions after running initial with-skill and baseline comparisons. The workflow improves skill prompts and descriptions automatically, runs held-out trigger tests, and allows you to repeat the eval loop until benchmark metrics and human feedback are satisfactory.