skills.rest
ROYCE-8425
ROYCE-8425COMMUNITY
Explore all skills by this author →
WebsiteGitHub

agent-eval

Evaluates coding agents' pass rate, cost, consistency via YAML-defined custom tasks and git worktree isolation.

1|Updated Apr 21, 2026
Search Tags:#LLM evaluation#task definition#coding agents#performance comparison#git isolation
One-click install
npx skills add https://github.com/ROYCE-8425/ai-marketing-hub --skill agent-eval-royce-8425
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-eval
Source: https://github.com/ROYCE-8425/ai-marketing-hub/tree/main/skills/agent-eval
Command: npx skills add https://github.com/ROYCE-8425/ai-marketing-hub --skill agent-eval-royce-8425

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill allows for a systematic comparison of coding agents, aiding in the selection of the best tool or model for a specific task.

Core Features & Use Cases

  • Head-to-Head Comparison: Compare coding agents such as Claude Code, Aider, Codex, etc., on custom tasks.
  • Performance Metrics: Measure pass rate, cost, time, and consistency across agents.
  • Custom Task Definitions: Declaratively define tasks with YAML, specifying what to do, which files to touch, and how to judge success.
  • Reproducibility: Each agent run gets its own git worktree for isolated execution.
  • Use Case: Before adopting a new coding tool or model, compare its performance against others using the skill.

Quick Start

Run a comparison with 'agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider --runs 3'.

Frequently Asked Questions about agent-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate coding agents head-to-head for performance?▼

Evaluate coding agents head-to-head by running custom YAML task definitions against multiple agents to measure pass rate, cost, time, and consistency using this comparison tool.

How do I define custom tasks for LLM evaluation?▼

You define custom tasks for LLM evaluation declaratively using YAML files, specifying what the agent should do, which files to touch, and how an LLM judge should evaluate success.

Does agent-eval require Git for task execution isolation?▼

Yes, Git is required for task execution isolation. The tool creates a separate git worktree for each agent run to ensure reproducible and isolated performance comparisons.

Can I compare coding agents like Claude Code and Aider on custom tasks?▼

Yes, you can compare coding agents like Claude Code and Aider by specifying multiple agents in the run command to measure their performance and consistency on identical tasks.

What metrics are measured during coding agent performance comparison?▼

During coding agent performance comparison, the tool measures pass rate, cost, time, and consistency across the evaluated coding agents to help you select the best model.

Related Skills

Explore More Software Engineering Skills→
riftzen-bitriftzen-bit

agent-eval

Compare coding agents on YAML-defined repository tasks and report pass rate, cost, time, and consistency metrics.

Community
Advanced
klu-devklu-dev
2

agent-eval

Compare coding agents on custom tasks using YAML task definitions and git worktree isolation.

Community
Intermediate
samymitysamymity
1

agent-eval

Compare coding agents on YAML-defined tasks with pass/fail judging and metrics.

Community
Advanced
eduardoaranteseduardoarantes
2

worktree-manager-skill

Manage git worktrees with automatic port isolation and per-worktree configuration.

Community
Advanced