empirical-prompt-tuning

Iteratively evaluates and refines agent prompts using unbiased subagent execution and dual-sided metrics.

Updated Jan 30, 2026
One-click install
npx skills add https://github.com/RyoMa99/chezmoi_dotfiles --skill empirical-prompt-tuning-ryoma99
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: empirical-prompt-tuning
Source: https://github.com/RyoMa99/chezmoi_dotfiles/tree/main/dot_claude/skills/empirical-prompt-tuning
Command: npx skills add https://github.com/RyoMa99/chezmoi_dotfiles --skill empirical-prompt-tuning-ryoma99

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Prompt authors cannot objectively judge their own instructions—what seems clear to the writer often confuses a fresh executor. This Skill removes author bias by dispatching fresh subagents to actually execute the prompt, then measuring results from both sides (executor self-reports plus instruction-side metrics) and iterating until improvements plateau. ## Core Features & Use Cases - Bias-Free Execution Testing: Dispatches new subagents per iteration to run evaluation scenarios against the target prompt, never reusing agents or relying on self-review. - Dual-Sided Metrics: Combines executor self-reports (ambiguities, discretion-filled gaps, retries) with measured metrics (success/failure, accuracy %, tool_uses, duration_ms) tracked in a fixed requirements checklist with [critical] tags. - Convergence & Overfitting Guards: Stops after 2 consecutive clean iterations with metric thresholds, and validates against a hold-out scenario to detect overfitting. - Use Case: After writing a new skill or slash command, run 2-3 realistic scenarios through fresh subagents, collect ambiguity reports, apply one minimal fix per iteration, and repeat until no new issues surface. ## Quick Start Use the empirical-prompt-tuning skill to evaluate and iteratively improve my newly written skill prompt with fresh subagent test runs.

Frequently Asked Questions about empirical-prompt-tuning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test whether my prompt or skill instructions are clear to an AI agent?

Dispatch a fresh subagent with the full prompt text, a realistic scenario, and a requirements checklist, then have it execute the task and report ambiguities and discretionary decisions. Repeat with new subagents after each fix until no new issues appear.

How to measure prompt quality with metrics?

Track success/failure on critical checklist items, accuracy as the percentage of satisfied requirements, tool_uses and duration_ms from the agent usage metadata, and retry counts from the executor's self-report. Weight qualitative feedback over speed metrics.

Why shouldn't I review my own prompt instead of using a subagent?

Self-review is structurally biased because you cannot objectively re-read text you just wrote. A fresh subagent reads the instructions with no prior context, exposing ambiguities the author cannot see.

When should I stop iterating on prompt improvements?

Stop after two consecutive iterations with zero new ambiguities, accuracy improvement under 3 points, and step/time variation within 10-15%. Also run one hold-out scenario to confirm accuracy does not drop more than 15 points, which would indicate overfitting.

What are the limitations of empirical prompt evaluation?

It requires an environment that can dispatch new subagents; without one, the method must be skipped rather than replaced by self-review. It is also not cost-effective for one-off disposable prompts, and single-scenario testing risks overfitting.