watch-benchmarks

Name: watch-benchmarks
Availability: InStock
Author: Sourcegraph

Monitor benchmark run statuses and generate structured JSON or table summaries.

32|4|Updated Feb 1, 2026

Search Tags:#automation #monitoring #reporting #error analysis #benchmark #status

Ready to run

npx skills add https://github.com/sourcegraph/CodeScaleBench --skill watch-benchmarks

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: watch-benchmarks
Source: https://github.com/sourcegraph/CodeScaleBench/tree/main/skills/watch-benchmarks
Command: npx skills add https://github.com/sourcegraph/CodeScaleBench --skill watch-benchmarks

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill provides real-time monitoring of benchmark runs, automatically classifying their status, identifying error patterns, and presenting clear summaries to help manage and debug the benchmarking process.

Core Features & Use Cases

Automated Status Classification: Scans benchmark run directories to determine if tasks are running, passed, failed, errored, or timed out.
Error Fingerprinting: Identifies common error types (e.g., token refresh, API errors, rate limits) for faster debugging.
Structured Reporting: Generates summaries in JSON or table format, detailing overall status, per-suite performance, and error breakdowns.
Use Case: A researcher running a large suite of AI coding benchmarks needs to quickly see which tasks have failed and why, without manually checking each one.

Quick Start

Scan the official benchmark runs and display a summary of their status.

Frequently Asked Questions about watch-benchmarks

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor the status of benchmark runs across multiple directories?▼

To monitor benchmark runs, scan the specified directories to classify tasks into running, completed, failed, errored, or timed-out states. It generates structured JSON or table summaries for quick analysis.

How can I identify recurring errors in failed benchmark tasks?▼

Identify recurring errors in failed benchmark tasks using error fingerprinting, which detects common patterns like API errors and rate limits. This helps group similar failures for faster debugging.

Can I filter benchmark status reports by time, suite, or configuration?▼

You can filter benchmark status reports by time, suite, and configuration. This narrows the structured summaries to show only the specific task performance and error breakdowns relevant to your analysis.

What is the best way to generate structured summaries for AI coding benchmarks?▼

The best way to generate structured summaries for AI coding benchmarks is scanning the run directories to output JSON or table formats. These detail overall status, per-suite performance, and error breakdowns.

Does the benchmark monitoring tool support writing per-task status files?▼

The benchmark monitoring tool optionally writes per-task status files. This captures the running, completed, failed, errored, or timed-out state of individual tasks for granular debugging.

Why do my benchmark tasks keep timing out and how can I check their status?▼

Check timed-out benchmark tasks by scanning run directories to classify their status. It fingerprints recurring errors and generates table or JSON summaries to help identify why tasks exceed limits.