What problem does it solve? Running an already-integrated model through the full TabArena-v0.1 benchmark involves many error-prone steps: resolving the right venv, installing model extras, smoke-fitting locally, generating SLURM or SkyPilot job files, monitoring long cluster arrays, triaging failures, and producing evaluated leaderboards. This Skill automates that entire pipeline from a single model name. ## Core Features & Use Cases - End-to-end benchmark orchestration: Scaffolds one tmp_scripts/run_<model>.py script with smoke, setup, and eval subcommands sharing a single benchmark name and paths, so launch and evaluation never drift apart. - Cluster launch and monitoring: Generates sbatch or SkyPilot commands, tracks array progress with SLURM/SkyPilot progress scripts, classifies failures (OOM, timeouts, preemption), and relaunches only missing items via cache-aware setup. - Evaluated leaderboard reporting: Produces Elo leaderboards, winrate matrices, and Pareto figures for all default subsets with the new model highlighted, plus a benchmark log entry. - Use Case: A maintainer says "benchmark TabM on the cluster" and gets a smoke-tested run launched on the GPU partition, progress updates every 15 minutes, and a final leaderboard showing where TabM ranks. ## Quick Start Ask the assistant to benchmark a registered model by name, for example "benchmark TabM on TabArena", and choose whether it should drive the cluster run end-to-end or hand off the launch commands to you.
