Compare coding agents across YAML-defined tasks with git worktree isolation.
npx skills add https://github.com/mitul-bhatia/Vibes --skill agent-eval-mitul-bhatia
Please help me install this Agent Skill. Skill: agent-eval Source: https://github.com/mitul-bhatia/Vibes/tree/main/.github/skills/agent-eval Command: npx skills add https://github.com/mitul-bhatia/Vibes --skill agent-eval-mitul-bhatia
This skill enables automated, reproducible comparisons of coding agents (e.g., Claude Code, Aider, Codex) on custom tasks, helping teams choose the best tool based on objective metrics.
Create a tasks/ directory with YAML task definitions and run agent-eval against multiple agents to generate a comparison report.
High-intent search queries and answers about installing and using this skill.
You can compare coding agents by defining tasks in YAML files and running an evaluation framework that generates reports on pass rate, cost, time, and consistency metrics for each agent.
When evaluating code agents, the metrics captured include pass rate, cost, time, and consistency, providing objective data to help teams choose the best tool for their specific development tasks.
Reproducible isolation between agents is ensured by using a fresh git worktree for each evaluation run, preventing cross-contamination of code changes and maintaining a clean environment for every task.
Yes, YAML task definitions can be used to declaratively specify tasks, making automated coding agent comparisons repeatable, auditable, and easy to manage across multiple tools.
Yes, agent-eval supports regression checks after updates by applying automated, reproducible comparisons of coding agents across tasks, ensuring new updates do not degrade performance.
The best way to evaluate agent performance is to run automated comparisons using YAML task definitions and per-task judges, collecting objective metrics like pass rate and cost to guide adoption decisions.
klu-devCompare coding agents on custom tasks using YAML task definitions and git worktree isolation.
aayushsoamBenchmark coding agents on reproducible tasks with pass rate, cost, time, and consistency metrics.
richardnpaulBenchmark coding agents on reproducible YAML tasks with git worktrees.
cuoreinpaceEvaluate LLM applications with automated metrics, human feedback, and benchmarking.
