skills.rest
richardnpaul
richardnpaulCOMMUNITY
Explore all skills by this author →
GitHub

agent-eval

Benchmark coding agents on reproducible YAML tasks with git worktrees.

Updated Apr 2, 2026
Search Tags:#pytest#benchmarking#benchmark#reproducibility#cost-analysis#coding-agents#agent-eval
One-click install
npx skills add https://github.com/richardnpaul/everything-vscode-copilot --skill agent-eval-richardnpaul
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-eval
Source: https://github.com/richardnpaul/everything-vscode-copilot/tree/main/.github/skills/agent-eval
Command: npx skills add https://github.com/richardnpaul/everything-vscode-copilot --skill agent-eval-richardnpaul

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Head-to-head comparisons of coding agents (Claude Code, Aider, Codex, etc.) on custom tasks with reproducible results to inform tool adoption decisions.

Core Features & Use Cases

  • Lightweight CLI to run agent comparisons on reproducible tasks using git worktrees for isolation.
  • Metrics collected: pass rate, cost, time, and consistency to enable data-backed decisions.
  • Use cases include evaluating agent performance before adopting a new model, regression checks on agent updates, and team-wide tool selection.

Quick Start

Run agent-eval against a defined set of tasks defined in YAML with multiple agents to compare.

Frequently Asked Questions about agent-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark coding agents like Claude Code and Aider on custom tasks?▼

You can benchmark coding agents by running head-to-head comparisons on reproducible tasks using YAML definitions and isolated git worktrees to produce repeatable results.

What metrics are collected when comparing AI coding assistants?▼

When comparing AI coding assistants, the metrics collected include pass rate, cost, time, and consistency, enabling data-backed decisions for tool adoption.

How do I ensure reproducibility when testing coding agents against each other?▼

Reproducibility is ensured by using deterministic judges, isolated git worktrees, and YAML task definitions to run identical scenarios across different coding agents.

Can I use this benchmarking workflow to check for regressions after an agent update?▼

Yes, you can run regression checks on agent updates by re-running the head-to-head benchmarks on reproducible tasks to verify performance and cost consistency.

Does this agent evaluation tool support Codex and similar frameworks?▼

Yes, the agent evaluation tool applies across Claude Code, Aider, Codex, and similar agents to compare pass rate, cost, time, and consistency in realistic tasks.

Related Skills

Explore More Software Engineering Skills→
ourinesourines
39

worktree-task

Launch background agent sessions for large coding tasks via git worktrees and tmux.

Community
Advanced
aayushsoamaayushsoam
1

agent-eval

Benchmark coding agents on reproducible tasks with pass rate, cost, time, and consistency metrics.

Community
Advanced
samymitysamymity
1

agent-eval

Compare coding agents on YAML-defined tasks with pass/fail judging and metrics.

Community
Advanced
PriyanshKuniyalPriyanshKuniyal

temporal-python-testing

Test Temporal workflows with pytest and mocked activities.

Community
Intermediate