skill-benchmarking

Run reproducible model benchmarks by executing tiered evaluations against evals-tiered.json. The workflow runs with_skill and without_skill comparisons, generating benchmark-<model>-tiered.json files for direct performance measurement.

Strict grader isolation is an evaluation mechanism that enforces evidence-only passing for model benchmarks. It ensures objective, reproducible grading by preventing the grader from accessing external context beyond the provided evidence.

Yes, the benchmarking workflow is model-agnostic and operates across platforms. The caller provides the model slug, meaning no hardcoded model assumptions restrict compatibility with your specific platform.

Compare model performance by running with_skill and without_skill evaluation tiers. This generates benchmark-<model>-tiered.json files that aggregate simple, medium, and complex results to surface discriminating assertions.

Tiered model evaluation aggregates simple, medium, and complex results to surface discriminating assertions. These assertions highlight specific performance gaps, guiding targeted improvements in model skills.

Yes, you must provide a model slug to run tiered benchmarks. The workflow is model-agnostic, requiring the caller to supply the slug so it can generate the corresponding benchmark-<model>-tiered.json output.