skills.rest
matt-riley
matt-rileyCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

agentic-eval

Design iterative evaluation loops with convergence guards for agent output refinement.

1|1|Updated May 3, 2026
Search Tags:#evaluation#iteration#convergence#reflection#rubric#LLM-as-judge
One-click install
npx skills add https://github.com/matt-riley/agent-skills --skill agentic-eval-matt-riley
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agentic-eval
Source: https://github.com/matt-riley/agent-skills/tree/main/skills/agentic-eval
Command: npx skills add https://github.com/matt-riley/agent-skills --skill agentic-eval-matt-riley

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) and assets (resource) components.

What problem does it solve?

This Skill helps you design and implement an evaluation loop that lets an agent assess and improve its own outputs through iterative generation, critique, and refinement.

Core Features & Use Cases

  • Supports reflection loops, evaluator-optimizer pipelines, LLM-as-judge scoring, and rubric-based evaluation to drive continuous improvement.
  • Provides a clear pattern: Generate → Evaluate → Critique → Refine → Output with guardrails like max iterations and convergence checks.
  • Applies to code, text, design, or decision outputs across domains requiring measurable improvement over multiple iterations.

Quick Start

Define evaluation criteria and run a simple reflection loop on a sample output.

Frequently Asked Questions about agentic-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an LLM evaluation loop for iterative output refinement?▼

An LLM evaluation loop lets an agent assess and improve its own outputs through iterative generation, critique, and refinement. You define structured evaluation criteria and guardrails like max iterations and convergence checks.

What is an evaluator-optimizer pipeline and when do I need convergence guards?▼

An evaluator-optimizer pipeline generates, critiques, and refines outputs iteratively. Convergence guards are required to stop the loop automatically when no further improvement is detected across iterations.

Can I use rubric-based scoring to compare multiple candidate outputs?▼

Yes, rubric-based scoring enables comparing and refining multiple candidate outputs. You define structured evaluation criteria, and the agent applies LLM-as-judge scoring to drive measurable improvement over iterations.

How do I set up a reflection loop with a defined max_iterations limit?▼

To set up a reflection loop, define your evaluation criteria and a strict max_iterations limit. The agent follows a Generate, Evaluate, Critique, Refine pattern, stopping when the iteration cap or convergence is reached.

Does this evaluation loop pattern work for both code and text generation?▼

Yes, the evaluation loop pattern applies to code, text, design, or decision outputs across domains. It requires structured evaluation results to drive measurable improvement over multiple iterations.

Why does my agent evaluation loop keep running without improving the output?▼

Without convergence guards, an evaluation loop runs indefinitely. You must define a max_iterations limit and implement convergence checks to stop the process when no improvement is detected between iterations.

Related Skills

Explore More Software Engineering Skills→
vchelaruvchelaru
11

agentic-eval

Coordinate generate-evaluate-critique-refine loops to improve AI agent outputs.

Community
Intermediate
OpenLAIROpenLAIR
1.0k

inno-idea-eval

Score research ideas across five dimensions with three reviewer personas and Area Chair synthesis.

Official
Advanced
selfagencyselfagency

agentic-eval

Evaluate and iteratively refine AI agent outputs using structured rubrics.

Community
Intermediate
synaptiaisynaptiai
6

conducting-deep-research

Produce iterative research reports with source scoring and contradiction handling.

Official
Advanced