agent-eval

Compare coding agents on custom tasks using pass rate, cost, time, and consistency metrics.

Updated Nov 19, 2025

Search Tags:#development workflow #code comparison #performance evaluation #coding agents #task-based analysis

Ready to run

npx skills add https://github.com/Sake-Team/SmartSake --skill agent-eval-sake-team

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: agent-eval
Source: https://github.com/Sake-Team/SmartSake/tree/main/backup/skills/agent-eval
Command: npx skills add https://github.com/Sake-Team/SmartSake --skill agent-eval-sake-team

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill unit helps evaluate and compare coding agents like Claude Code, Aider, and Codex on custom tasks, providing metrics like pass rate, cost, time, and consistency.

Core Features & Use Cases

Agent Comparison: Head-to-head comparison of various coding agents.
Custom Task Support: Evaluate agents on tasks tailored to specific needs.
Performance Metrics: Obtain pass rate, cost, time, and consistency data.
Use Case: Compare the performance of Claude Code and Aider on adding retry logic to an HTTP client.

Quick Start

Compare agents on adding retry logic to 'http_client.py' with 'agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider'.

Frequently Asked Questions about agent-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate coding agent performance on custom tasks?▼

Coding agent performance evaluation involves comparing agents like Claude Code and Aider on user-defined tasks to measure pass rate, cost, time, and consistency. You execute specific scripts to run task-based analysis for your development workflow.

Can I compare Claude Code and Aider on the same development task?▼

Yes, you can perform head-to-head comparison of coding agents like Claude Code and Aider on the same custom task. The evaluation script measures pass rate, cost, time, and consistency to help you choose the best agent.

What metrics are measured during coding agent comparisons?▼

During coding agent comparisons, the metrics measured are pass rate, cost, time, and consistency. These performance metrics provide a quantitative basis for evaluating agents on custom tasks in your development workflow.

How do I set up a task-based analysis for an HTTP client retry logic feature?▼

To set up task-based analysis for an HTTP client retry logic feature, define your task parameters in a YAML configuration file and execute the evaluation script with your selected coding agents. The script compares performance against your defined criteria.

Do I need specific scripts to run coding agent evaluations?▼

Yes, you need to execute specific scripts to run coding agent evaluations and compare performance against defined criteria. These scripts process your custom tasks and generate the performance metrics data for analysis.