autoresearch

Automated prompt optimization uses an autonomous loop to iteratively refine prompts, templates, and configs. A four-way architecture separates the main agent, eval agent, test runner, and judge to isolate responsibilities and ensure reliable evaluation during the refinement process.

Iterative prompt refinement is best handled by an autonomous optimization loop that reduces manual tuning effort. This workflow applies a built-in eval and test loop to generate evaluation outputs and continuously improve prompts across creative and technical experiments.

An eval agent and judge script work together by separating evaluation execution from quality scoring. The eval agent runs deterministic tests via a test runner, while the judge applies optional hybrid LLM judging to evaluate subjective quality metrics for the generated outputs.

Deterministic prompt evaluation requires specific files like eval.py and test_cases.json to satisfy structural requirements. These files define the deterministic test runner logic and test cases needed to execute the sandboxed optimization workflow safely.

Subjective quality evaluation in automated prompt engineering is supported through optional hybrid LLM judging. This mode complements deterministic objective metrics by applying hybrid evaluation techniques to score subjective creative quality during the optimization loop.

Autonomous prompt optimization loops require predefined test cases and deterministic evaluation scripts to function properly. While they reduce manual tuning, subjective quality relies on optional hybrid LLM judging, meaning the loop's reliability is constrained by the initial test case design and sandboxed workflow boundaries.