skills.rest
klu-dev
klu-devCOMMUNITY
Explore all skills by this author →
GitHub

agent-eval

Compare coding agents on custom tasks using YAML task definitions and git worktree isolation.

2|Updated Apr 14, 2026
Search Tags:#code comparison#coding agents#agent evaluation#performance comparison#metrics collection
One-click install
npx skills add https://github.com/klu-dev/porting-ecc-to-vscode --skill agent-eval-klu-dev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-eval
Source: https://github.com/klu-dev/porting-ecc-to-vscode/tree/main/.github/skills/agent-eval
Command: npx skills add https://github.com/klu-dev/porting-ecc-to-vscode --skill agent-eval-klu-dev

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of comparing coding agents, such as Claude Code, Aider, and Codex, on custom tasks by providing metrics like pass rate, cost, and consistency.

Core Features & Use Cases

  • Head-to-Head Comparison: Compare coding agents on custom tasks.
  • Performance Metrics: Measure pass rate, cost, time, and consistency.
  • Use Case: Determine the best coding agent for your codebase by comparing their performance on specific tasks.

Quick Start

Run the agent-eval skill on the 'add-retry-logic' task using the 'claude-code' and 'aider' agents.

Frequently Asked Questions about agent-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compare coding agents like Claude Code and Aider on my own tasks?▼

Coding agent comparison involves defining custom tasks in YAML and executing them within isolated git worktrees to measure pass rate, cost, and consistency for your codebase.

What metrics are used for evaluating coding agent performance?▼

Coding agent evaluation uses metrics like pass rate, cost, time, and consistency to measure performance on custom tasks and determine the best agent for your codebase.

Do I need to define tasks in YAML to compare coding agents?▼

Yes, comparing coding agents requires YAML task definitions and git worktree isolation to accurately measure pass rate, cost, and consistency during the evaluation.

When should I run a coding agent comparison?▼

Run a coding agent comparison before adopting a new agent or during model updates to verify performance metrics and select the best agent for your codebase.

How does git worktree isolation work when testing coding agents?▼

Git worktree isolation provides each coding agent a separate, clean environment during comparison, preventing interference while measuring pass rate and consistency metrics.

Related Skills

Explore More Software Engineering Skills→
ROYCE-8425ROYCE-8425
1

agent-eval

Evaluates coding agents' pass rate, cost, consistency via YAML-defined custom tasks and git worktree isolation.

Community
Intermediate
aayushsoamaayushsoam
1

agent-eval

Benchmark coding agents on reproducible tasks with pass rate, cost, time, and consistency metrics.

Community
Advanced
samymitysamymity
1

agent-eval

Compare coding agents on YAML-defined tasks with pass/fail judging and metrics.

Community
Advanced
simplysmartaisimplysmartai

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and LLM-as-judge patterns.

Community
Advanced