agent-eval

Benchmark coding agents on reproducible YAML tasks with git worktrees.

Updated Apr 2, 2026

Search Tags:#pytest #benchmarking #benchmark #reproducibility #cost-analysis #coding-agents #agent-eval

Ready to run

npx skills add https://github.com/richardnpaul/everything-vscode-copilot --skill agent-eval-richardnpaul

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: agent-eval
Source: https://github.com/richardnpaul/everything-vscode-copilot/tree/main/.github/skills/agent-eval
Command: npx skills add https://github.com/richardnpaul/everything-vscode-copilot --skill agent-eval-richardnpaul

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Head-to-head comparisons of coding agents (Claude Code, Aider, Codex, etc.) on custom tasks with reproducible results to inform tool adoption decisions.

Core Features & Use Cases

Lightweight CLI to run agent comparisons on reproducible tasks using git worktrees for isolation.
Metrics collected: pass rate, cost, time, and consistency to enable data-backed decisions.
Use cases include evaluating agent performance before adopting a new model, regression checks on agent updates, and team-wide tool selection.

Quick Start

Run agent-eval against a defined set of tasks defined in YAML with multiple agents to compare.

Frequently Asked Questions about agent-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I benchmark coding agents like Claude Code and Aider on custom tasks?▼

You can benchmark coding agents by running head-to-head comparisons on reproducible tasks using YAML definitions and isolated git worktrees to produce repeatable results.

What metrics are collected when comparing AI coding assistants?▼

When comparing AI coding assistants, the metrics collected include pass rate, cost, time, and consistency, enabling data-backed decisions for tool adoption.

How do I ensure reproducibility when testing coding agents against each other?▼

Reproducibility is ensured by using deterministic judges, isolated git worktrees, and YAML task definitions to run identical scenarios across different coding agents.

Can I use this benchmarking workflow to check for regressions after an agent update?▼

Yes, you can run regression checks on agent updates by re-running the head-to-head benchmarks on reproducible tasks to verify performance and cost consistency.

Does this agent evaluation tool support Codex and similar frameworks?▼

Yes, the agent evaluation tool applies across Claude Code, Aider, Codex, and similar agents to compare pass rate, cost, time, and consistency in realistic tasks.