Verified:GitHub

skill-creator

Create and iteratively improve AI skills with structured evaluation and benchmarking workflows.

1|Updated May 5, 2026

Search Tags:#testing #skill creation #benchmarking #ai agents #evaluation #iterative improvement #description optimization

Ready to run

npx skills add https://github.com/kollaborai/kollab --skill skill-creator-kollaborai

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: skill-creator
Source: https://github.com/kollaborai/kollab/tree/main/bundles/skills/skill-creator
Command: npx skills add https://github.com/kollaborai/kollab --skill skill-creator-kollaborai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires yaml, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

Creating effective AI skills requires more than just writing instructions—it demands systematic testing, measurement, and iterative refinement to ensure they trigger correctly and perform reliably across diverse user queries.

Core Features & Use Cases

Skill Creation & Iteration: Draft new skills from scratch or improve existing ones through a structured capture-intent, write, test, and refine loop.
Quantitative Evaluation: Run evals with baseline comparisons, grade outputs against assertions, and aggregate benchmark statistics with variance analysis.
Description Optimization: Automatically optimize skill descriptions for better triggering accuracy using train/test splits and iterative improvement loops.

Quick Start

Use the skill-creator skill to build a new skill for extracting invoice data from PDFs, then run the eval loop to optimize its description for reliable triggering.

Frequently Asked Questions about skill-creator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate and benchmark AI agents during skill creation?▼

You can evaluate AI agents by running baseline comparisons, grading outputs against assertions, and aggregating benchmark statistics with variance analysis. This skill provides Python scripts for quantitative evaluation and blind comparison to measure performance.

What's the best way to optimize skill descriptions for triggering accuracy?▼

The best way to optimize skill descriptions is using train/test splits and iterative improvement loops. This skill automates description loop generation to systematically refine and optimize triggering accuracy.

How do I generate test cases for iterative improvement of AI skills?▼

You can generate test cases for iterative improvement by applying a structured capture-intent, write, test, and refine loop. This workflow generates test cases and performs quantitative grading to refine skills end-to-end.

Do I need Python scripts to run quantitative evaluations on AI agents?▼

Yes, you need Python scripts to run quantitative evaluations. This skill requires Python scripts specifically for eval aggregation, benchmark generation, blind comparison, and description loop automation.

Can I perform blind comparisons between baseline AI skills and new versions?▼

Yes, you can perform blind comparisons between baseline AI skills and new versions. The skill provides benchmark generation and blind comparison workflows to measure quantitative improvements accurately.