score-tasks

Evaluating benchmark task quality involves scoring individual tasks on instruction clarity, verifier quality, and reproducibility using Python scripts. The scripts apply weighted scoring to benchmark suites and identify tasks needing review based on defined quality thresholds.

Verifier quality in code evaluation benchmarks measures how reliably a task's verification mechanism confirms correct solutions. It is scored alongside instruction clarity and reproducibility to flag weak tasks falling below defined quality thresholds for review.

You can automatically score all tasks in a software development benchmark suite by running specific Python scripts that apply weighted scoring to instruction clarity, verifier quality, and reproducibility, then display the results in a table.

Yes, Python scripts can flag benchmark tasks falling below a specified quality threshold. The scripts score tasks on instruction clarity, verifier quality, and reproducibility, highlighting areas for improvement before tasks are added to the suite.

No external dependencies are required to run benchmark task scoring scripts. The skill uses self-contained Python scripts to perform weighted scoring on instruction clarity, verifier quality, and reproducibility for benchmark suite auditing.

You should audit a benchmark suite for task reproducibility and instruction clarity when curating new tasks or evaluating existing ones. Automated scoring ensures tasks meet required standards for verifiability and consistency before being added to the suite.