empirical-prompt-tuning

Iteratively evaluates and refines agent prompts using unbiased subagent execution and dual-sided metrics.

2|1|Updated May 11, 2019
One-click install
npx skills add https://github.com/guitarrapc/dotfiles-win --skill empirical-prompt-tuning-guitarrapc
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: empirical-prompt-tuning
Source: https://github.com/guitarrapc/dotfiles-win/tree/main/HOME/.agents/skills/empirical-prompt-tuning
Command: npx skills add https://github.com/guitarrapc/dotfiles-win --skill empirical-prompt-tuning-guitarrapc

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Prompt authors cannot objectively judge their own instructions, so ambiguous wording in skills, slash commands, and task prompts goes undetected until agents misbehave. This Skill replaces guesswork with empirical evaluation: unbiased subagents execute the prompt, and both self-reported feedback and measured metrics drive iterative refinement. ## Core Features & Use Cases - Bias-Free Execution: Dispatches fresh subagents per iteration to run evaluation scenarios against the target prompt, avoiding author bias. - Dual-Sided Metrics: Combines executor self-reports (ambiguities, discretion gaps, retries) with measured signals (success rate, accuracy, tool_uses, duration) against a fixed requirements checklist. - Convergence & Overfitting Control: Stops after consecutive clean iterations, runs hold-out scenarios to detect overfitting, and flags divergence when the prompt design itself is flawed. - Use Case: After writing a new skill or heavily revising a CLAUDE.md section, run 2-3 realistic scenarios through fresh subagents, record ambiguities and accuracy per iteration, and apply one minimal fix per cycle until metrics plateau. ## Quick Start Evaluate my newly written skill prompt by running it through fresh subagents on two realistic scenarios and iterate on the ambiguities they report.

Frequently Asked Questions about empirical-prompt-tuning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test whether my prompt or skill instructions are clear enough?

Dispatch a fresh subagent that reads the prompt cold and executes 2-3 realistic scenarios against a fixed requirements checklist. Collect its self-reported ambiguities plus measured accuracy, tool_uses, and duration, then apply one minimal fix per iteration.

How to measure prompt quality for AI agents?

Measure success as all critical checklist items passing, accuracy as the percentage of requirements met, plus tool_uses, duration_ms, and retry counts from the subagent's usage metadata. Weight qualitative feedback like reported ambiguities above raw speed metrics.

Can I evaluate my own prompt by rereading it myself?

No, self-review is structurally biased because authors cannot objectively read text they just wrote. The method requires dispatching a new subagent each iteration; if dispatch is unavailable, skip empirical evaluation rather than substituting self-review.

When should I stop iterating on prompt improvements?

Stop after two consecutive iterations with zero new ambiguities, accuracy improvement under 3 points, and step and duration variance within thresholds. Also run one hold-out scenario to confirm accuracy does not drop more than 15 points, which would indicate overfitting.

When is empirical prompt evaluation not worth doing?

Skip it for one-off disposable prompts where evaluation cost exceeds the benefit, and when the goal is reflecting the author's subjective preference rather than improving success rate. It also does not apply in environments where subagent dispatch is unavailable.