beam-ape-optimizer

Automated prompt optimization uses a three-agent credit assignment system of Doer, Critic, and Editor to identify failing instructions and iteratively refine prompts. It runs test batches against a golden dataset to ensure improved agent accuracy before redeployment.

Fixing failing prompts in a Beam agent graph is best handled by running iterative Automated Prompt Engineering (APE). The system records reasoning traces during test batches to audit failures, automatically identifies problematic instructions, and redeploys the corrected prompts.

Credit assignment in automated prompt engineering works by using three specialized agents: a Doer to execute tasks, a Critic to evaluate reasoning traces, and an Editor to fix identified failing instructions. This isolates specific prompt errors for targeted optimization.

You can track prompt performance improvements over multiple iterations by running test batches against a golden dataset. The system tracks redeployed prompts and reasoning traces across iterations to ensure sustained accuracy gains for your Beam agents.

A golden dataset is required to run test batches for validating prompt improvements. It provides the baseline accuracy metrics needed for the Doer/Critic/Editor credit assignment loop to effectively identify and fix failing instructions during redeployment.