autoresearch

Automates prompt optimization and evaluation using separated agent roles and deterministic tests.

53|25|Updated Jan 23, 2026

Search Tags:#automation #prompt-engineering #prompt-optimization #test-runner #LLM-evaluation #autoresearch #eval-agent

Ready to run

npx skills add https://github.com/naveedharri/benai-skills --skill autoresearch-naveedharri

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: autoresearch
Source: https://github.com/naveedharri/benai-skills/tree/main/plugins/autoresearch/skills/autoresearch
Command: npx skills add https://github.com/naveedharri/benai-skills --skill autoresearch-naveedharri

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Autonomous loop to iteratively refine prompts, templates, configs, or code, reducing manual tuning effort and accelerating experimentation.

Core Features & Use Cases

Four-way separation architecture: main agent, eval agent, test runner, and judge script to isolate responsibilities and ensure reliability during optimization.
Deterministic and optional hybrid evaluation modes for objective metrics and subjective quality scoring.
Use cases include prompt improvement, template optimization, and code/config refinement across AI-assisted workflows.

Quick Start

Run the AutoResearch workflow to iteratively optimize prompts, generate evaluation outputs, and review results through the built-in eval/test loop.

Frequently Asked Questions about autoresearch

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate prompt optimization and evaluation for LLM workflows?▼

Automated prompt optimization uses an autonomous loop to iteratively refine prompts, templates, and configs. A four-way architecture separates the main agent, eval agent, test runner, and judge to isolate responsibilities and ensure reliable evaluation during the refinement process.

What is the best way to iteratively refine prompts without manual tuning?▼

Iterative prompt refinement is best handled by an autonomous optimization loop that reduces manual tuning effort. This workflow applies a built-in eval and test loop to generate evaluation outputs and continuously improve prompts across creative and technical experiments.

How does an eval agent and judge script work together for LLM evaluation?▼

An eval agent and judge script work together by separating evaluation execution from quality scoring. The eval agent runs deterministic tests via a test runner, while the judge applies optional hybrid LLM judging to evaluate subjective quality metrics for the generated outputs.

Do I need specific files to run deterministic prompt evaluation workflows?▼

Deterministic prompt evaluation requires specific files like eval.py and test_cases.json to satisfy structural requirements. These files define the deterministic test runner logic and test cases needed to execute the sandboxed optimization workflow safely.

Can I evaluate subjective quality in automated prompt engineering workflows?▼

Subjective quality evaluation in automated prompt engineering is supported through optional hybrid LLM judging. This mode complements deterministic objective metrics by applying hybrid evaluation techniques to score subjective creative quality during the optimization loop.

What are the limitations of autonomous prompt optimization loops?▼

Autonomous prompt optimization loops require predefined test cases and deterministic evaluation scripts to function properly. While they reduce manual tuning, subjective quality relies on optional hybrid LLM judging, meaning the loop's reliability is constrained by the initial test case design and sandboxed workflow boundaries.