skills.rest
forgivesam168
forgivesam168COMMUNITY
Explore all skills by this author →
GitHub

agentic-eval

Evaluate AI agent outputs using structured rubrics and JSON feedback loops.

2|1|Updated Feb 5, 2026
Search Tags:#robustness#quality-assurance#code-review#refine#self-critique#evaluate
One-click install
npx skills add https://github.com/forgivesam168/ai-dev-workflow --skill agentic-eval-forgivesam168
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agentic-eval
Source: https://github.com/forgivesam168/ai-dev-workflow/tree/main/skills/agentic-eval
Command: npx skills add https://github.com/forgivesam168/ai-dev-workflow --skill agentic-eval-forgivesam168

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluate AI agent outputs to identify quality gaps and inconsistencies and propose improvements.

Core Features & Use Cases

  • Self-critique loops: Agents critique their own outputs to surface issues and refine results.
  • Evaluator-optimizer pipelines: Separate generation and evaluation stages for quality-critical tasks.
  • Test-driven refinement: Generate tests and refine code or analyses based on test outcomes.
  • Rubric-based or LLM-as-judge evaluation: Use structured criteria to rank and improve results.
  • Iterative improvement: Apply repeated refinements to code, reports, or analyses until criteria are met.

Quick Start

Use an initial task to kick off an evaluation loop. For example: "Summarize the latest release notes and evaluate the summary for accuracy and coverage." Then run the loop:

  • Generate a draft output for the task.
  • Evaluate the draft against a defined rubric.
  • Critique the evaluation results.
  • Refine the draft based on critique.
  • Repeat until the rubric is satisfied.

Frequently Asked Questions about agentic-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate AI agent outputs for quality and refinement?▼

Evaluate AI agent outputs by applying structured evaluation schemas, rubric-based LLM-as-judge criteria, and JSON-formatted feedback to identify quality gaps and propose measurable improvements.

What is a self-critique loop for iterative code refinement?▼

A self-critique loop is an iterative improvement process where AI agents critique their own drafts against a defined rubric, refine the output, and repeat until convergence criteria are satisfied.

How do I set up an evaluator-optimizer pipeline for quality-critical generation?▼

Set up an evaluator-optimizer pipeline by separating generation and evaluation stages, applying structured criteria to rank results, and using JSON-formatted feedback to drive iterative refinements.

Can I use rubric-based evaluation to improve generated reports and analyses?▼

Yes, rubric-based evaluation applies structured criteria to rank and improve generated reports, code, and analyses, using loop control parameters and convergence criteria for repeatable, measurable improvements.

What are the limitations of LLM-as-judge evaluation for test-driven refinement?▼

LLM-as-judge evaluation relies on defined rubrics and convergence criteria; limitations include the dependency on structured evaluation schemas and the need for loop control parameters to ensure repeatable refinement.

Do I need specific dependencies to run agentic evaluation loops?▼

No specific dependencies are required to run agentic evaluation loops; the process uses structured schemas, JSON-formatted feedback, and convergence criteria to enable repeatable improvements across code and analyses.

Related Skills

Explore More Software Engineering Skills→
Sir-chawakornSir-chawakorn
9

llm-eval-harness

Builds an LLM evaluation harness using golden datasets and code-based scorers.

Community
Advanced
pradyumnasagarpradyumnasagar
6

agent-handoff

Automate inter-skill data transfer and provenance capture via JSON Material Passports.

Community
Advanced
githubgithub
37.6k

agentic-eval

Automate iterative evaluation and refinement of AI agent outputs with JSON feedback.

Official
Intermediate
MudassarAbrarMudassarAbrar

agentic-eval

Evaluate and refine AI agent outputs through iterative generate-evaluate-critique-refine loops.

Community
Advanced