skill-benchmarking

What problem does it solve?

Benchmarking AI model skills across tasks by running tiered evaluations against evals-tiered.json, enabling scalable, repeatable performance measurement.

Core Features & Use Cases

• Supports with_skill and without_skill evaluation runs, producing benchmark-<model>-tiered.json for direct comparison
• Aggregates tiered results (simple, medium, complex) and surfaces discriminating assertions to guide improvements
• Enforces strict grader isolation and evidence-only passing, ensuring objective evaluation
• Model-agnostic workflow: the caller provides the model slug; no hardcoded model assumptions

Quick Start

Run the full benchmark workflow for a given model slug using the evals-tiered.json data