agent-eval

Compare coding agents across YAML-defined tasks with git worktree isolation.

Updated Apr 4, 2026

Search Tags:#metrics #evaluation #comparison #git-worktree #yaml-tasks #agent-eval #code-agents

Ready to run

npx skills add https://github.com/mitul-bhatia/Vibes --skill agent-eval-mitul-bhatia

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: agent-eval
Source: https://github.com/mitul-bhatia/Vibes/tree/main/.github/skills/agent-eval
Command: npx skills add https://github.com/mitul-bhatia/Vibes --skill agent-eval-mitul-bhatia

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill enables automated, reproducible comparisons of coding agents (e.g., Claude Code, Aider, Codex) on custom tasks, helping teams choose the best tool based on objective metrics.

Core Features & Use Cases

YAML Task Definitions: Define tasks declaratively so evaluations are repeatable and auditable.
Git Worktree Isolation: Each run uses a fresh git worktree for reproducible isolation between agents.
Metrics Collected: Capture pass rate, cost, time, and consistency to compare agent performance across tasks.

Quick Start

Create a tasks/ directory with YAML task definitions and run agent-eval against multiple agents to generate a comparison report.

Frequently Asked Questions about agent-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I compare coding agents like Claude Code and Aider on custom tasks?▼

You can compare coding agents by defining tasks in YAML files and running an evaluation framework that generates reports on pass rate, cost, time, and consistency metrics for each agent.

What metrics are captured when evaluating code agents?▼

When evaluating code agents, the metrics captured include pass rate, cost, time, and consistency, providing objective data to help teams choose the best tool for their specific development tasks.

How do I ensure reproducible isolation between coding agents during evaluation?▼

Reproducible isolation between agents is ensured by using a fresh git worktree for each evaluation run, preventing cross-contamination of code changes and maintaining a clean environment for every task.

Can I use YAML task definitions for automated agent comparisons?▼

Yes, YAML task definitions can be used to declaratively specify tasks, making automated coding agent comparisons repeatable, auditable, and easy to manage across multiple tools.

Does agent-eval support regression checks after updating coding models?▼

Yes, agent-eval supports regression checks after updates by applying automated, reproducible comparisons of coding agents across tasks, ensuring new updates do not degrade performance.

What is the best way to evaluate agent performance before adopting new coding tools?▼

The best way to evaluate agent performance is to run automated comparisons using YAML task definitions and per-task judges, collecting objective metrics like pass rate and cost to guide adoption decisions.