prompt-eval

Validate prompt changes by testing fresh Claude sessions against neutral tasks.

Updated Aug 5, 2026
One-click install
npx skills add https://github.com/harivansh-afk/loom-index-e2e --skill prompt-eval-harivansh-afk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: prompt-eval
Source: https://github.com/harivansh-afk/loom-index-e2e/tree/main/skills/prompt-eval
Command: npx skills add https://github.com/harivansh-afk/loom-index-e2e --skill prompt-eval-harivansh-afk

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Editing a CLAUDE.md, skill, agent definition, or system prompt gives no proof the change actually works, since your current session already contains the change and any in-session check trivially passes. This Skill verifies prompt changes by spawning a fresh, clean-context Claude agent that re-reads the prompt from disk and observing whether the intended behavior emerges. ## Core Features & Use Cases - Clean-context evaluation: Runs claude -p sessions that load the real production prompt hierarchy, avoiding false results from in-session checks or under-applying subagents. - Sampled pass rates with A/B attribution: Runs 3-5 neutral trigger tasks, compares against a pre-change baseline, and reports a ✅ / ❌ / 🤔 verdict with a pass rate like 4/5. - Use Case: After adding a ✅/❌/🤔 status convention to your global CLAUDE.md and running home-manager switch, use this Skill to spawn five fresh Opus sessions with neutral tasks and confirm the convention is applied before calling the change done. ## Quick Start Verify my CLAUDE.md change works by running five fresh claude -p sessions with a neutral task and reporting the pass rate.

Frequently Asked Questions about prompt-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test if a CLAUDE.md change actually works?▼

Test a CLAUDE.md change by spawning a fresh `claude -p` session with a neutral task that should organically trigger the new behavior, never in your current session. Run it 3-5 times and report the pass rate, since a single run is just noise.

How to A/B test a system prompt change with Claude Code?▼

A/B a system prompt change by running `claude -p` once as baseline and once with `--append-system-prompt` containing the new instruction, on the same model. The goal is attributing the output difference to the change, not just seeing acceptable output.

Can I use the Agent tool to test prompt changes?▼

The Agent tool is only appropriate for testing an agent or subagent definition itself, since its own prompt is the thing under test. For CLAUDE.md, skill, memory, or system-prompt changes, subagents under-apply house-style rules and produce false negatives, so use `claude -p` instead.

Why does my prompt change eval pass but fail in real sessions?▼

Evals mislead when run in the current session, when a store-symlinked global CLAUDE.md was not applied via `home-manager switch`, or when the eval model differs from production. Match the model (default Opus) and confirm the on-disk file contains the change first.

How many samples are needed to evaluate a prompt change?▼

Run 3-5 samples minimum, more for subtle changes, because model output is stochastic and one run is noise. A change firing only 2/5 of the time is flaky (🤔), not a pass, and the rate must be reported with a real sampled output.