custom-empirical-prompt-tuning

Iteratively test and refine agent-facing prompts using blank-slate subagent execution and two-sided evaluation.

1|Updated Apr 3, 2021
One-click install
npx skills add https://github.com/NaoyaMiyagawa/dotfiles --skill custom-empirical-prompt-tuning-naoyamiyagawa
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: custom-empirical-prompt-tuning
Source: https://github.com/NaoyaMiyagawa/dotfiles/tree/main/.ai/skills/custom-empirical-prompt-tuning
Command: npx skills add https://github.com/NaoyaMiyagawa/dotfiles --skill custom-empirical-prompt-tuning-naoyamiyagawa

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Prompt authors cannot objectively judge their own instructions, so ambiguous wording in skills, slash commands, CLAUDE.md files, and code-generation prompts causes agents to misbehave in ways the author never anticipated. This Skill replaces guesswork with empirical testing: a fresh subagent actually executes the prompt, and both sides of the exchange are measured until improvements plateau. ## Core Features & Use Cases - Bias-free execution testing: Dispatches a blank-slate subagent to run the target prompt against 2-3 realistic scenarios, then collects structured self-reports on unclear points, discretionary fill-ins, and retries. - Quantified iteration loop: Tracks success/failure, accuracy against a fixed requirements checklist, step count, duration, and retry count per iteration, with explicit convergence, divergence, and overfitting criteria. - Failure pattern ledger: Accumulates class-level fix rules across iterations so recurring failure modes are detected instead of re-patched, plus an optional two-variant exploration mode for breaking plateaus. - Use Case: After writing a new skill whose description promises navigation, form filling, and data extraction, run this Skill to verify the body actually delivers, fix the gaps iteration by iteration, and stop only when two consecutive rounds show zero new unclear points. ## Quick Start Empirically evaluate and iteratively improve the prompt in my skill file by dispatching fresh subagents against realistic scenarios until the improvements converge.

Frequently Asked Questions about custom-empirical-prompt-tuning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test whether my prompt or skill instructions are clear to an AI agent?

Dispatch a blank-slate subagent to execute the prompt against 2-3 realistic scenarios with a fixed requirements checklist. Collect its structured self-report on unclear points and discretionary fill-ins, then measure accuracy, step count, and retries to decide what to fix.

How to iteratively improve a CLAUDE.md or slash command prompt?

Fix one theme per iteration based on the subagent's structured reflection, then re-run with a fresh subagent each round. Stop when two consecutive iterations produce zero new unclear points and metric improvements fall below the defined thresholds.

Can I evaluate my own prompt by rereading it instead of using a subagent?

No. Self-rereading is structurally biased because you cannot view text you just wrote objectively, and the skill explicitly forbids it. If subagent dispatch is unavailable, delegate evaluation to a separate session or skip empirical evaluation entirely.

When should I stop iterating on prompt improvements?

Stop after two consecutive rounds with zero new unclear points, accuracy improvement of 3 points or less, and step and duration variation within 10-15 percent. Also run one hold-out scenario to check for overfitting before declaring convergence.

What are the limitations of empirical prompt evaluation with subagents?

It requires an environment where subagent dispatch is available, and each iteration costs multiple agent runs. It is not suitable for one-off throwaway prompts, and pairwise variant comparisons must avoid direct A/B rating due to LLM position bias.