waxa-eval

Empirical prompt tuning is a methodology for iteratively evaluating skill prompts using structured scenarios. It codifies real iteration runs to guide grader selection, interpret unclear points, and manage a ledger to judge convergence across defined test cases.

To set up an iterative evaluation loop, scaffold the evaluation skeleton inside your skill directory. Author per-task scenarios under the evals/ directory using templates, configure the eval.yaml file, and run an iteration pass to start tracking results in ledger.yaml.

The four-stage iteration pattern consists of structural fixes, grader breadth expansion, surface-form coverage, and addressing residual unclear points. This progression systematically refines prompt quality and evaluation convergence across multiple scenarios.

You track convergence by recording per-iteration results in ledger.yaml. This ledger-based approach logs outcomes across defined scenarios, allowing you to judge convergence and extract general fix rules from the empirical evaluation runs.

You should run empirical evaluations explicitly when the user asks for evaluation or when you need to iterate on skill quality. The skill operates on invocation rules and does not run automatically outside of requested evaluation contexts.

You need the waxa CLI to coordinate the evaluation loops and apply the empirical-prompt-tuning methodology. The CLI provides the necessary tooling to select graders, run scenarios, and manage the ledger for tracking convergence.