validate-tasks

To validate benchmark task definitions, run a pre-flight check using Python 3 to identify truncated instructions, missing test scripts, and configuration mismatches. This prevents execution failures by verifying task integrity before initiating lengthy runs.

Pre-flight validation for code generation tasks is the process of checking benchmark definitions for truncated instructions, non-executable scripts, and template placeholders. It ensures task configurations align with expected parameters to prevent mid-run failures.

You can check for missing test scripts and configuration mismatches by running an automated validation script against the repository structure. This error detection process identifies incorrect path references and verifies that all necessary scripts are executable.

Yes, you need Python 3 installed to execute the validation script for pre-flight checks. You also require access to the CodeScaleBench repository structure to perform the pre-run analysis on task definitions and selection registries.

Benchmark task execution fails mid-run due to critical issues like template placeholders, non-executable test scripts, or truncated instructions. Running a pre-flight validation check detects these configuration mismatches beforehand to prevent wasted resources.

The best way to prevent execution failures is to perform automated pre-flight validation on benchmark tasks. This verifies that all task instructions are complete, necessary test scripts are present and executable, and configurations match expected parameters.