skills.rest
vrcms
vrcmsCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

agent-eval

Benchmark coding agents across isolated git worktrees with pass rate, cost, and time metrics.

1|Updated Apr 6, 2026
Search Tags:#automation#git#testing#performance#benchmarking#agent-eval
One-click install
npx skills add https://github.com/vrcms/everything-qwen-code --skill agent-eval-vrcms
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-eval
Source: https://github.com/vrcms/everything-qwen-code/tree/main/.qwen/skills/agent-eval
Command: npx skills add https://github.com/vrcms/everything-qwen-code --skill agent-eval-vrcms

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill removes the guesswork from selecting coding agents by providing a standardized, data-driven framework to compare their performance on your specific codebase.

Core Features & Use Cases

  • Head-to-Head Comparison: Run multiple agents against identical tasks to measure pass rates, cost, and time.
  • Reproducible Benchmarking: Uses Git worktree isolation to ensure tests are isolated and consistent across different agent runs.
  • Multi-Modal Judging: Supports deterministic code-based tests (pytest, build commands), pattern matching (grep), and LLM-as-a-judge evaluations.

Quick Start

Use the agent-eval skill to run a comparison between claude-code and aider on the tasks defined in your tasks directory.

Frequently Asked Questions about agent-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark coding agent performance on my codebase?▼

You can benchmark coding agent performance by executing reproducible tasks across isolated git worktrees to measure pass rates, API cost, and execution time. This provides a standardized, data-driven framework for comparison.

What evaluation criteria are supported for agent regression testing?▼

Agent regression testing supports deterministic code-based tests like pytest and build commands, pattern matching using grep, and LLM-as-a-judge evaluations. These multi-modal judging criteria ensure comprehensive performance measurement.

How can I run a head-to-head comparison between different coding agents?▼

Run head-to-head comparisons by executing multiple agents against identical tasks within isolated git worktrees. This measures pass rates, cost, and execution time to facilitate data-backed tool selection.

Does agent benchmarking require git worktrees for test isolation?▼

Yes, reproducible benchmarking uses git worktree isolation to ensure tests are isolated and consistent across different agent runs. This prevents side effects from impacting performance metrics or pass rates.

Related Skills

Explore More Software Engineering Skills→
josorio7122josorio7122
1

worktree

Create isolated Git worktrees with dependency installation and baseline tests.

Community
Advanced
sumeetonline90sumeetonline90

agent-eval

Compare coding agents using reproducible YAML tasks and quantitative metrics.

Community
Intermediate
oabdelmaksoudoabdelmaksoud
3

eval-harness

Implement evaluation-driven development with code and model-based evaluators.

Community
Advanced
samymitysamymity
1

dmux-workflows

Orchestrate parallel AI agent panes in tmux sessions with merge steps.

Community
Intermediate