skills.rest
skurekjakub
skurekjakubCOMMUNITY
Explore all skills by this author →
GitHub

agent-eval

Evaluate completed Ralph agent runs against scoring criteria and artifact contracts.

Updated Apr 1, 2026
Search Tags:#workflow#agent#evaluation#quality-assurance#transcripts#artifact#ralph
One-click install
npx skills add https://github.com/skurekjakub/docwriter-agent --skill agent-eval-skurekjakub
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-eval
Source: https://github.com/skurekjakub/docwriter-agent/tree/main/.github/skills/agent-eval
Command: npx skills add https://github.com/skurekjakub/docwriter-agent --skill agent-eval-skurekjakub

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Decompose and evaluate a completed Ralph agent run to provide a structured quality assessment, including tool selection, task ordering, error recovery, output accuracy, and adherence to the agent-as-function pattern.

Core Features & Use Cases

  • Break down a run into sub-tasks and score each against standardized dimensions.
  • Verify artifact discipline (status.json, output.md, manifest.json) and phase transitions.
  • Generate actionable findings reports, improvement recommendations, and support decision-making for run improvements.
  • Compare multiple runs (transcripts, logs, and PR references) to build evaluation checklists.

Quick Start

Run the skill on a finished Ralph agent run directory to produce an evaluation plan and scored report.

Frequently Asked Questions about agent-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate Ralph agent runs for quality assurance?▼

To evaluate Ralph agent runs, the skill decomposes completed runs into sub-tasks and scores them against standardized dimensions like tool selection, task ordering, and error recovery to produce structured feedback.

What does an agent evaluation scoring criteria checklist include?▼

An agent evaluation scoring criteria checklist includes tool selection accuracy, task ordering efficiency, error recovery capability, output accuracy, and adherence to the agent-as-function pattern for comprehensive quality assessment.

How do I verify artifact contracts and phase transitions in agent transcripts?▼

Verify artifact contracts and phase transitions by checking for required files like status.json, output.md, and manifest.json within the log and artifact directories to ensure orchestrator compliance.

Can I compare multiple agent transcripts and logs to build an evaluation checklist?▼

Yes, you can compare multiple agent runs by analyzing their transcripts, logs, and PR references to build standardized evaluation checklists and generate actionable improvement recommendations.

What is the best way to generate findings reports from agent audit trails?▼

Generate findings reports by running the skill on a finished agent run directory to apply the evaluation plan, score performance dimensions, and output actionable improvement recommendations.

Related Skills

Explore More Data & Analytics Skills→
pertain99pertain99
1

session-logs

Query JSONL session logs with jq and ripgrep for context reconstruction.

Community
Intermediate
richardrimanrichardriman
2

review

Identify code quality issues and improvement opportunities without modifying source files.

Community
Intermediate
skurekjakubskurekjakub

agent-as-function

Decompose monolithic prompts into per-phase skills with artifact handoffs.

Community
Advanced
amnotyoungamnotyoung
3

evaluate

Orchestrate multi-agent evaluations of ODA projects against OECD-DAC and KOICA standards.

Community
Advanced