skill-creator

Skill evaluation and benchmarking involves generating test prompts, running parallel baseline comparisons, grading outputs, and aggregating results to iteratively improve skill descriptions. This Skill provides automated workflows to execute this full evaluation loop.

Creating test prompts for skill evaluation requires generating realistic queries, running trigger tests in parallel, and capturing timing and token data. This Skill includes scripted tools to generate eval sets and orchestrate parallel trigger evaluations automatically.

Optimizing skill descriptions to reduce false positives requires an automated loop that proposes description variants, tests them against baseline runs, and grades outputs. This Skill runs description optimization iterations to increase correct triggering rates.

Running parallel skill evaluations and comparing outputs is supported through included Python scripts and a local review server. The Skill captures timing, tokens, and produces benchmark.json files along with human-readable reports automatically.

Python and the pyyaml dependency are required to run the skill benchmarking, test generation, and description optimization scripts. The Skill relies on these Python scripts and local server components to execute automated evaluation workflows.

A SKILL.md file defines metadata and instructions for Claude-style AI skills, needed when developers or product teams build automated workflows. This Skill provides guided prompts and templates to author clear SKILL.md descriptions for intended queries.