skill-eval-writer

What problem does it solve?

This tool solves the challenge of creating standardized evaluation tooling for any Skill by turning a Skill's SKILL.md and its references into a complete evaluation package. It outputs evals.json and a grade.py script, along with run guidelines, enabling automated validation of instruction quality across Skills.

Core Features & Use Cases

• Generates a complete eval suite (evals.json) with five well-scoped eval cases and corresponding assertions.
• Produces grade.py that can validate Skill outputs by collecting either textual responses or generated files.
• Validates eval definitions and provides a consistent JSON schema for scoring.
• Supports optional directories (scripts/, references/, assets/) as resources to enrich evaluations.

Quick Start

Run the Skill Eval Writer on a Skill directory to generate evals.json, grade.py, and a grading guide.