refresh-task-taxonomy

To update task taxonomy without triggering a re-benchmark, you can automate the taxonomy refresh process to read task specifications, assign content-based tags, and update the taxonomy file independently, ensuring the leaderboard hash remains unchanged.

Automated task taxonomy refresh is the process of updating task categories and tags by reading task specifications and assigning facets from a controlled vocabulary based on content, ensuring benchmark task discoverability and analysis remain current without manual intervention.

Yes, you can maintain code benchmark task discoverability automatically by running a taxonomy refresh workflow that detects task additions, changes, or removals and updates the taxonomy file with content-based tags accordingly.

The best way to assign tags to benchmark tasks based on content is to use an automated workflow that evaluates task specifications and assigns facets from a controlled vocabulary, ensuring consistent categorization across the entire task suite.

No, refreshing the task taxonomy does not affect benchmark results or leaderboard rankings because the taxonomy refresh is decoupled from the benchmarking process, ensuring the leaderboard hash remains unchanged while only updating discoverability metadata.

When refreshing a task taxonomy, sanity checks include verifying group balance and detecting missing facets, ensuring the quality and completeness of the taxonomy for accurate per-category model analysis and task discoverability.