agentic-eval

Implements iterative evaluation and refinement loops for improving AI agent outputs.

Updated Jun 14, 2026
One-click install
npx skills add https://github.com/ironkid90-s/lucky5-v7 --skill agentic-eval-ironkid90-s
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: agentic-eval
Source: https://github.com/ironkid90-s/lucky5-v7/tree/main/.github/skills/agentic-eval
Command: npx skills add https://github.com/ironkid90-s/lucky5-v7 --skill agentic-eval-ironkid90-s

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Single-shot LLM generation often produces outputs with quality issues that go undetected. This Skill provides patterns for building self-critique, evaluator-optimizer, and test-driven refinement loops so agent outputs are systematically evaluated and improved before delivery. ## Core Features & Use Cases - Reflection Loops: Generate output, self-critique against criteria with structured JSON feedback, and refine until all checks pass. - Evaluator-Optimizer Pipelines: Separate generation and evaluation components with score thresholds and dimension-based feedback. - Code-Specific Refinement: Test-driven loop that generates code and pytest tests, runs them, and fixes failures iteratively. - Evaluation Strategies: Outcome-based checks, LLM-as-judge comparisons, and weighted rubric scoring. - Use Case: When building a report-generation agent, wrap the generator in an evaluator-optimizer loop with a rubric covering accuracy, clarity, and completeness, stopping when the score exceeds 0.8 or after 3 iterations. ## Quick Start Add a reflection loop to my report generator that critiques the output against accuracy and clarity criteria and refines it up to three times.

Frequently Asked Questions about agentic-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add self-critique to an LLM generation pipeline?

Use a reflection loop: generate the output, prompt the LLM to evaluate it against your criteria returning structured JSON with PASS/FAIL per criterion, then refine the output using the failure feedback. Repeat up to 3 iterations or until all criteria pass.

What is the evaluator-optimizer pattern for AI agents?

The evaluator-optimizer pattern separates generation and evaluation into distinct components. A generator produces output, an evaluator scores it against dimensions like accuracy and clarity, and an optimizer refines it until the score exceeds a threshold such as 0.8.

How do I evaluate LLM outputs with a rubric?

Define weighted dimensions such as accuracy at 0.4, clarity at 0.3, and completeness at 0.3. Have the LLM rate the output 1-5 per dimension as JSON, then compute the weighted average normalized to a 0-1 score.

How do I prevent infinite loops in agent refinement?

Set a maximum iteration count of 3-5 and add convergence detection that stops the loop if the output score stops improving between iterations. Log the full trajectory so stalled refinements can be debugged.

When should I use test-driven refinement for code generation?

Use it when generating executable code with verifiable behavior. Generate the code and pytest tests together, run the tests, and feed failures back into the model to fix the code, repeating until tests pass or the iteration limit is reached.