skills.rest
samymity
samymityCOMMUNITY
Explore all skills by this author →
GitHub

agent-eval

Compare coding agents on YAML-defined tasks with pass/fail judging and metrics.

1|Updated Mar 3, 2026
Search Tags:#regression testing#performance metrics#git worktrees#coding agents#agent benchmarking#YAML tasks
One-click install
npx skills add https://github.com/samymity/bridge-ventures-backend --skill agent-eval-samymity
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-eval
Source: https://github.com/samymity/bridge-ventures-backend/tree/main/.claude/skills/agent-eval
Command: npx skills add https://github.com/samymity/bridge-ventures-backend --skill agent-eval-samymity

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Agent-eval replaces subjective “which coding agent is better?” decisions with repeatable, measurable head-to-head comparisons on your own tasks.

Core Features & Use Cases

  • Reproducible agent runs with git worktrees: isolates each agent attempt from others to reduce interference and corruption of the base repo.
  • YAML-defined tasks and judges: lets you specify prompts, target files, and success criteria using deterministic checks like pytest/commands and optionally model-based evaluation.
  • Actionable metrics reporting: captures pass rate, cost (when available), wall-clock time, and consistency across repeated runs for clearer trade-off decisions.

Quick Start

Define a YAML task in tasks/, then run comparisons across agents with agent-eval run --task tasks/<your-task>.yaml --agent <agent1> --agent <agent2> --runs 3.

Frequently Asked Questions about agent-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark coding agents like Claude Code or Aider on my own tasks?▼

Run coding agent regression testing by executing agent-eval across pinned YAML tasks with deterministic pytest or grep judges. It isolates attempts in git worktrees and reports pass rates, ensuring agents maintain expected performance on your codebase over time.

How do I run reproducible head-to-head comparisons between coding agents?▼

Run reproducible head-to-head comparisons by defining YAML tasks with pinned commits and deterministic judges, then executing agent-eval across multiple agents. It uses git worktrees to isolate attempts, capturing pass rate, cost, and wall-clock time for reliable trade-off analysis.

How do I define deterministic success criteria for coding agent evaluations?▼

Use agent-eval for pre-adoption evaluations by running target agents against your YAML-defined tasks. It measures pass rate, wall-clock time, and run-to-run consistency using deterministic judges, providing measurable metrics to validate agent capabilities before integration.

Does agent-eval require git worktrees to isolate agent coding attempts?▼

Agent-eval reports pass rate, cost when available, wall-clock time, and run-to-run consistency. These actionable metrics enable clear trade-off decisions by capturing both the deterministic success rate and the operational efficiency of the evaluated coding agents.

Related Skills

Explore More Software Engineering Skills→
riftzen-bitriftzen-bit

agent-eval

Compare coding agents on YAML-defined repository tasks and report pass rate, cost, time, and consistency metrics.

Community
Advanced
Michae2xlMichae2xl

agent-eval

Compare coding agents on custom tasks using pass rate, cost, time, and consistency metrics.

Community
Advanced
richardnpaulrichardnpaul

agent-eval

Benchmark coding agents on reproducible YAML tasks with git worktrees.

Community
Intermediate
eduardoaranteseduardoarantes
2

worktree-manager-skill

Manage git worktrees with automatic port isolation and per-worktree configuration.

Community
Advanced