watch-benchmarks

What problem does it solve?

This Skill provides real-time monitoring of benchmark runs, automatically classifying their status, identifying error patterns, and presenting clear summaries to help manage and debug the benchmarking process.

Core Features & Use Cases

• Automated Status Classification: Scans benchmark run directories to determine if tasks are running, passed, failed, errored, or timed out.
• Error Fingerprinting: Identifies common error types (e.g., token refresh, API errors, rate limits) for faster debugging.
• Structured Reporting: Generates summaries in JSON or table format, detailing overall status, per-suite performance, and error breakdowns.
• Use Case: A researcher running a large suite of AI coding benchmarks needs to quickly see which tasks have failed and why, without manually checking each one.

Quick Start

Scan the official benchmark runs and display a summary of their status.