skill-testing

Verifies SKILL.md behavior changes through baseline and GREEN test passes with a Haiku judge agent.

1|Updated Feb 25, 2026
One-click install
npx skills add https://github.com/cjthompson/claude-code-config --skill skill-testing-cjthompson
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: skill-testing
Source: https://github.com/cjthompson/claude-code-config/tree/main/packages/skill-testing
Command: npx skills add https://github.com/cjthompson/claude-code-config --skill skill-testing-cjthompson

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Writing a SKILL.md file does not guarantee it actually changes agent behavior. This Skill provides a structured methodology to prove a skill works: run a baseline pass without the skill to document failure modes, then a GREEN pass with the skill loaded, with an independent Haiku judge agent scoring every response against a checklist. ## Core Features & Use Cases - Baseline/GREEN test methodology: Run prompts with and without the skill to isolate its actual effect on agent output. - Scenario file format: Define tests with exact prompts, MUST Contain items drawn from the skill's teachings, and MUST NOT Contain items drawn from observed baseline failures. - Independent judge agent: A Haiku subagent evaluates responses against the checklist and returns PASS, FAIL, or PARTIAL verdicts, removing biased self-judgment. - Use Case: After writing a SKILL.md for a UI framework, create test scenarios, run the baseline to capture the wrong answers unguided agents give, then verify the GREEN run passes all MUST criteria and record results in test-results.md. ## Quick Start Create a test scenario for my skill and run a baseline pass without the SKILL.md, then a GREEN pass with it, using a Haiku judge to verify the results.

Frequently Asked Questions about skill-testing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test whether a SKILL.md changes agent behavior?

Run two passes: a baseline with only the prompt and no skill, then a GREEN pass with the full SKILL.md prepended to the subagent's system prompt. A separate Haiku judge agent compares each response against your MUST and MUST NOT checklist to produce the verdict.

How do I write test scenarios for an AI agent skill?

Create one file per skill containing multiple test blocks, each with an exact prompt, 3-5 MUST Contain items taken from facts the skill explicitly teaches, and MUST NOT Contain items capturing the wrong answers unguided agents actually give during the baseline run.

Why should a separate judge agent evaluate skill test results?

Self-judgment by the orchestrating agent is biased and unreliable. A Haiku subagent follows strict checklists mechanically, checking only presence or absence of each criterion in the response text, making its PASS, FAIL, or PARTIAL verdict the authoritative result.

What does a PARTIAL verdict mean in skill testing?

PARTIAL means no MUST NOT items appeared but some MUST items were missing. On a GREEN run this indicates the skill teaches the right direction but has content gaps, so you should add the missing material to the SKILL.md and re-run the test.

When is baseline testing not enough for a skill?

A GREEN-only pass is insufficient because without a baseline you cannot tell whether the skill caused the correct behavior or the agent already knew the answer. Baseline runs also supply the realistic MUST NOT items for your scenarios.