What problem does it solve? Choosing between coding agents like Claude Code, Aider, or Codex usually relies on anecdotal impressions rather than data. This Skill replaces vibes-based comparisons with systematic, reproducible benchmarks run against your own codebase and tasks. ## Core Features & Use Cases - Declarative Task Definitions: Define benchmark tasks in YAML with prompts, target files, pinned commits, and judge criteria (pytest, grep patterns, or LLM-as-judge). - Isolated Execution: Each agent run executes in its own git worktree, so agents cannot interfere with each other or corrupt the base repository. - Metrics & Reporting: Collects pass rate, API cost, wall-clock time, and consistency across repeated runs, then generates comparison tables. - Use Case: Before adopting a new coding agent for your team, define 3-5 tasks representing your real workload, run each agent 3 times, and compare pass rates and costs side by side to make a data-backed decision. ## Quick Start Define a YAML task file describing a coding change with test-based judge criteria, then run agent-eval against multiple agents and generate a comparison report.
