agentforge-benchmark

What problem does it solve?

This Skill helps teams test and benchmark AI agents comprehensively, including end-to-end tests, regression suites, and acceptance criteria, to reduce production risk and improve reliability.

Core Features & Use Cases

• Level 1: Tool unit tests for deterministic components
• Level 2: Integration tests with Mock LLMs to verify orchestration
• Level 3: End-to-end task tests with statistical pass rates
• LLM-as-Judge for automated acceptance and calibration against gold standards
• Regression design, housekeeping of regression cases, and harness stability
• Industry benchmarks guidance (SWE-bench, τ-bench, WebArena) and custom benchmarks
• Custom benchmark design and gold dataset creation

Quick Start

Define your initial end-to-end task set, enable automated acceptance using LLM-as-Judge, and run SWE-bench alongside τ-bench to establish a baseline.