skill-creator

To create and test an AI skill definition for CI workflows, you draft a SKILL.md file, generate test prompts, run trigger and behavior evaluations, and iterate until benchmarks meet acceptance criteria. This script-driven workflow supports both development and CI contexts for reliable skill performance.

A SKILL.md file is a structured metadata and body document that defines an AI skill's capabilities and trigger descriptions. Writing correct frontmatter and body text ensures the skill activates reliably when users submit matching prompts during automated evaluation runs.

You benchmark AI skill performance against a baseline by spawning with-skill and baseline runs, grading the outputs, and aggregating the results into a benchmark.json file. This evaluation loop allows direct comparison of skill-enhanced and standard responses for iterative improvement.

Yes, you need pyyaml installed to package AI skills into distributable .skill files. This dependency supports the parsing and orchestration of SKILL.md frontmatter required during the execution of eval and packaging scripts.

Yes, you can generate standalone HTML review reports for AI skill evaluations. The packaging and presentation utilities produce these HTML pages to visualize grading results and timing JSON, enabling human feedback and iterative description optimization.

The best way to iterate on failing AI skill descriptions is to inspect the generated review page, interpret the grading JSON, and apply agent guidance to refine the SKILL.md text. Re-running the evaluation loop validates improvements against baseline pass rates.