skill-eval

To validate skill structure, run the quick validation script via `python3 -m scripts.skill_eval.quick_validate <path/to/skill>`. This checks frontmatter fields, naming conventions, and description length to ensure readiness for empirical testing.

Output benchmarking compares with-skill versus without-skill outputs to quantify added value. The evaluation generates HTML reports and baseline comparisons to visualize measurable improvements in skill performance.

Trigger evaluation tests whether a skill's description reliably activates for a defined set of prompts. It automates train/test splits, compares results, and refines prompts to maximize generalization across different user inputs.

Yes, HTML reports are generated during the skill evaluation workflow to provide visual results. These reports aid quick review and sharing with stakeholders by displaying trigger, description, and benchmark outcomes.

No external dependencies are required. The skill evaluation package runs structural checks, trigger evaluation, and output benchmarking using only the included scripts and reference components without needing additional installations.

Structural validation fails when frontmatter fields, naming conventions, or description length do not meet consistency requirements. Running the quick validation script identifies specific readiness issues before proceeding to trigger and benchmark testing.