skills.rest
tuliosouza99
tuliosouza99COMMUNITY
Explore all skills by this author →
GitHub

skill-creator

Create and iteratively improve SKILL.md-based agent skills with Python eval scripts.

Updated Apr 21, 2023
Search Tags:#benchmarking#evaluation#packaging#skill-development#skill-creator#trigger-optimization
One-click install
npx skills add https://github.com/tuliosouza99/CartolaPy --skill skill-creator-tuliosouza99
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: skill-creator
Source: https://github.com/tuliosouza99/CartolaPy/tree/main/.agents/skills/skill-creator
Command: npx skills add https://github.com/tuliosouza99/CartolaPy --skill skill-creator-tuliosouza99

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyyaml, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

Helps authors design, evaluate, and iteratively improve Claude-style SKILL.md skills so they trigger reliably and produce higher-quality, repeatable outputs. It removes manual trial-and-error by providing an opinionated loop for drafting SKILL.md, generating test prompts, running comparative runs (with and without the skill), and surfacing quantitative and qualitative feedback.

Core Features & Use Cases

  • Iterative skill development: Draft SKILL.md content, generate test prompts, run evals, grade outputs, and rewrite the skill based on feedback.
  • Benchmarking & analysis: Aggregate grading results, produce benchmark summaries, and surface variance and pass-rate diagnostics.
  • Tooling & packaging: Includes scripts to run eval loops, generate an HTML review viewer, aggregate benchmarks, improve descriptions, and package a skill into a distributable file.
  • Use Case: Create a new automation skill, run blind comparisons against a baseline, collect user reviews via the viewer, and apply suggested revisions until the pass rates stabilize.

Quick Start

Ask the skill-creator to draft a SKILL.md for your task, produce 2–3 realistic test prompts, run evals, and return the benchmark and suggested SKILL.md edits.

Frequently Asked Questions about skill-creator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate and improve AI skill trigger reliability?▼

Skill evaluation works by generating test prompts, running comparative eval loops with and without the skill, grading outputs, and iteratively rewriting SKILL.md based on quantitative pass-rate diagnostics and qualitative feedback.

What's the best way to benchmark Claude-style skills locally?▼

Benchmarking Claude-style skills involves running an evaluation loop that aggregates grading results, produces benchmark summaries, and surfaces variance and pass-rate diagnostics to guide iterative improvements.

How do I package a SKILL.md file for distribution?▼

Packaging a SKILL.md file for distribution uses Python scripts to bundle skill content and components into a distributable file, requiring PyYAML for frontmatter parsing.

Do I need PyYAML to create and parse SKILL.md frontmatter?▼

Yes, PyYAML is required to parse SKILL.md frontmatter containing name and description fields, and to run the Python scripts for eval orchestration and packaging.

Can I generate test prompts automatically when developing AI skills?▼

Yes, test prompts are generated automatically during skill development by requesting the tool to produce realistic test prompts, run evals, and return benchmark results with suggested SKILL.md edits.

Why does my AI skill trigger inconsistently during testing?▼

Inconsistent skill triggering happens when the SKILL.md description lacks optimization, requiring trigger-optimization scripts to run blind comparisons against a baseline and apply suggested revisions until pass rates stabilize.

Related Skills

Explore More Software Engineering Skills→
aws-samplesaws-samples
12

good-skill

Validate skill-eval pipelines with a standards-compliant test Skill Unit.

Official
Basic
gelowzzzgelowzzz

skill-creator

Create, validate, and package Claude Skills into .skill files.

Community
Advanced
DFly7DFly7

skill-creator

Create, evaluate, and iteratively improve AI skill definitions and workflows.

Community
Advanced
zhbonerzhboner

skill-creator

Draft SKILL.md files and refine them through prompt-based test evaluations.

Community
Advanced