prompt-optimize

Automated prompt refinement works by iteratively applying single quality edits to your prompt file and retaining them only if they increase the evaluation score. You need a prompt file and an evaluation command to measure performance against your custom metrics.

You need a prompt file to optimize and a custom evaluation command that outputs a score. This evaluation command acts as the sole source of feedback, while objective, target score, and iteration budget are optional settings.

No, this prompt optimization requires an evaluation command as the sole source of feedback. Without it, the system cannot measure performance improvement or decide which prompt edits to keep during iteration.

Prompt refinement proposes and applies one quality edit per iteration. It evaluates the updated prompt using your custom scoring command and only keeps the edit if it strictly increases the evaluation metric.

The main limitation is its strict dependency on your custom evaluation command's accuracy. It optimizes solely for the score returned by that command, meaning any blind spots or biases in your evaluation logic will directly impact the refined prompt.