Audit benchmark suites against the Agent Benchmark Criteria framework.
npx skills add https://github.com/sourcegraph/CodeScaleBench --skill benchmark-audit
Please help me install this Agent Skill. Skill: benchmark-audit Source: https://github.com/sourcegraph/CodeScaleBench/tree/main/skills/benchmark-audit Command: npx skills add https://github.com/sourcegraph/CodeScaleBench --skill benchmark-audit
This Skill ensures the quality and reliability of benchmark suites by auditing them against the Agent Benchmark Criteria (ABC) framework, identifying potential issues before they impact evaluations.
Use the benchmark-audit skill to audit all benchmark suites and display the results in a table format.
High-intent search queries and answers about installing and using this skill.
To audit benchmark quality, you evaluate benchmark suites against the Agent Benchmark Criteria framework, assessing task, outcome, and reporting validity to identify instruction quality, verifier correctness, and reproducibility issues across suites.
The Agent Benchmark Criteria framework is used for benchmark validation to comprehensively evaluate task validity, outcome validity, and reporting dimensions, assigning letter grades from A to F to identify specific issues in benchmark task definitions.
Yes, you need Python scripts to run a benchmark suite audit, as the execution and analysis of benchmark task definitions and execution artifacts require Python scripts to process and validate the benchmark data.
To check benchmark reproducibility and verifier correctness before launching evaluations, perform a pre-flight audit across all benchmark suites to catch critical flaws early, ensuring all tasks are well-defined, verifiable, and reportable.
The best way to identify instruction quality issues in benchmark task definitions is to audit them against the Agent Benchmark Criteria framework, which assesses task, outcome, and reporting validity while generating letter grades and specific issue reports.
masa-codehubAnalyze design docs and ADRs to create PR-ready specification plans.
kaeligExecutes an evaluation pipeline for AI-generated code with deterministic and LLM-based graders.
Supporter09Benchmark code generation models on HumanEval, MBPP, and other functional-correctness tasks.
zhaohui-yangCheck Chinese official documents for structure, quality, and factual accuracy.
