rebuild-leaderboard

Name: rebuild-leaderboard
Availability: InStock
Author: SURUS

Automate benchmark re-evaluation and publish results to HuggingFace datasets.

8|Updated Sep 12, 2025

Search Tags:#automation #benchmarking #reproducibility #cluster-management #huggingface #ai-evaluation

Ready to run

npx skills add https://github.com/surus-lat/benchy --skill rebuild-leaderboard

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: rebuild-leaderboard
Source: https://github.com/surus-lat/benchy/tree/main/.agent/skills/rebuild-leaderboard
Command: npx skills add https://github.com/surus-lat/benchy --skill rebuild-leaderboard

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill addresses the critical need to restore and re-verify benchmark results on the LatamBoard after data loss or cluster failure, ensuring consistent model performance tracking.

Core Features & Use Cases

Automated Re-evaluation: Orchestrates the full benchmark suite for multiple models across GPU clusters.
Incremental Publishing: Merges and publishes results to HuggingFace after each model run to prevent data loss.
Use Case: If a cluster wipe occurs, use this skill to systematically re-run all models, validate their configurations, and restore the leaderboard state without manual intervention for every step.

Quick Start

Use the rebuild-leaderboard skill to initiate a full re-evaluation of all models currently listed in the configuration directory.

Frequently Asked Questions about rebuild-leaderboard

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I restore a HuggingFace leaderboard after a GPU cluster failure?▼

Restore a HuggingFace leaderboard by orchestrating an automated re-evaluation of all models on the GPU cluster. This skill manages task execution and synchronizes results incrementally to HuggingFace datasets to prevent data loss during recovery.

What is the best way to automate AI benchmarking across multiple models?▼

Automate AI benchmarking by orchestrating the full benchmark suite across available GPU clusters. This skill handles execution and resource allocation while merging results to HuggingFace after each model run to ensure reproducible performance tracking.

Do I need a specific CLI tool to run automated benchmark recovery?▼

Yes, automated benchmark recovery requires the benchy CLI to manage task execution. You also need a configured GPU cluster and valid HuggingFace authentication tokens to successfully synchronize the restored evaluation results.

Can I prevent data loss when re-running large benchmark suites on a GPU cluster?▼

Prevent data loss when re-running benchmarks by using incremental publishing. This skill merges and publishes results to HuggingFace datasets immediately after each individual model run, ensuring progress is saved if a subsequent failure occurs.

Does this automated benchmarking workflow require manual intervention for each model?▼

No, the automated benchmarking workflow reduces manual intervention by systematically reading all model configurations from the directory and executing the full suite. It handles end-to-end execution and publication automatically for every listed model.