skill-eval-writer

Convert SKILL.md and references into evals.json, grade.py, and running guides.

1|Updated Mar 18, 2026

Search Tags:#automation #python #json #yaml #grading #evals

Ready to run

npx skills add https://github.com/cr330326/AgentSkill --skill skill-eval-writer

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: skill-eval-writer
Source: https://github.com/cr330326/AgentSkill/tree/main/validate/skill-eval-writer
Command: npx skills add https://github.com/cr330326/AgentSkill --skill skill-eval-writer

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This tool solves the challenge of creating standardized evaluation tooling for any Skill by turning a Skill's SKILL.md and its references into a complete evaluation package. It outputs evals.json and a grade.py script, along with run guidelines, enabling automated validation of instruction quality across Skills.

Core Features & Use Cases

Generates a complete eval suite (evals.json) with five well-scoped eval cases and corresponding assertions.
Produces grade.py that can validate Skill outputs by collecting either textual responses or generated files.
Validates eval definitions and provides a consistent JSON schema for scoring.
Supports optional directories (scripts/, references/, assets/) as resources to enrich evaluations.

Quick Start

Run the Skill Eval Writer on a Skill directory to generate evals.json, grade.py, and a grading guide.

Frequently Asked Questions about skill-eval-writer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I auto-generate evals and grade scripts for a Skill?▼

Skill Eval Writer creates automated eval packs by converting SKILL.md and references into evals.json, grade.py, and running guides. It applies to Skill units requiring deterministic grade scripts and evaluative checks across multiple evals.

What is included in an automated eval package for validating Skill outputs?▼

An automated eval package includes evals.json with five eval cases, a Python 3.9 compatible grade.py script for grading, optional asset directories, and a generated grading.json with a strict JSON schema for scoring.

Can I use custom scripts and references when creating evaluation workflows?▼

Your Skill directory must contain a documented SKILL.md and optional reference files. Providing these inputs allows the Skill Eval Writer to extract evaluation workflows and generate deterministic grade scripts and evaluative checks.

Does the generated grade script support strict JSON schema outputs?▼

After generating evals.json and grade.py, execute the grade.py script to collect Skill outputs, validate against the strict JSON schema, and output scoring results in grading.json for automated evaluation.

What Python version is required to run the generated grading scripts?▼

Use Skill documentation generators to structure SKILL.md and reference files, providing structured instruction content that the Skill Eval Writer consumes to generate evals.json and grade.py scripts.

How do I validate eval definitions before running automated grading?▼

Use standalone Python testing frameworks or manual QA scripts to validate Skill outputs, achieving custom evaluation logic and scoring without relying on auto-generated evals.json or grade.py packages.