skills.rest
sourcegraph
sourcegraphOFFICIAL
Explore all skills by this author →
Website𝕏@sourcegraphGitHub

watch-benchmarks

Monitor benchmark run statuses and generate structured JSON or table summaries.

32|4|Updated Feb 1, 2026
Search Tags:#automation#monitoring#reporting#error analysis#benchmark#status
One-click install
npx skills add https://github.com/sourcegraph/CodeScaleBench --skill watch-benchmarks
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: watch-benchmarks
Source: https://github.com/sourcegraph/CodeScaleBench/tree/main/skills/watch-benchmarks
Command: npx skills add https://github.com/sourcegraph/CodeScaleBench --skill watch-benchmarks

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill provides real-time monitoring of benchmark runs, automatically classifying their status, identifying error patterns, and presenting clear summaries to help manage and debug the benchmarking process.

Core Features & Use Cases

  • Automated Status Classification: Scans benchmark run directories to determine if tasks are running, passed, failed, errored, or timed out.
  • Error Fingerprinting: Identifies common error types (e.g., token refresh, API errors, rate limits) for faster debugging.
  • Structured Reporting: Generates summaries in JSON or table format, detailing overall status, per-suite performance, and error breakdowns.
  • Use Case: A researcher running a large suite of AI coding benchmarks needs to quickly see which tasks have failed and why, without manually checking each one.

Quick Start

Scan the official benchmark runs and display a summary of their status.

Frequently Asked Questions about watch-benchmarks

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor the status of benchmark runs across multiple directories?▼

To monitor benchmark runs, scan the specified directories to classify tasks into running, completed, failed, errored, or timed-out states. It generates structured JSON or table summaries for quick analysis.

How can I identify recurring errors in failed benchmark tasks?▼

Identify recurring errors in failed benchmark tasks using error fingerprinting, which detects common patterns like API errors and rate limits. This helps group similar failures for faster debugging.

Can I filter benchmark status reports by time, suite, or configuration?▼

You can filter benchmark status reports by time, suite, and configuration. This narrows the structured summaries to show only the specific task performance and error breakdowns relevant to your analysis.

What is the best way to generate structured summaries for AI coding benchmarks?▼

The best way to generate structured summaries for AI coding benchmarks is scanning the run directories to output JSON or table formats. These detail overall status, per-suite performance, and error breakdowns.

Does the benchmark monitoring tool support writing per-task status files?▼

The benchmark monitoring tool optionally writes per-task status files. This captures the running, completed, failed, errored, or timed-out state of individual tasks for granular debugging.

Why do my benchmark tasks keep timing out and how can I check their status?▼

Check timed-out benchmark tasks by scanning run directories to classify their status. It fingerprints recurring errors and generates table or JSON summaries to help identify why tasks exceed limits.

Related Skills

Explore More Software Engineering Skills→
surus-latsurus-lat
8

synthesize-data

Generate synthetic benchmark datasets as JSONL input-output pairs from task specifications.

Official
Intermediate
changgengluchanggenglu

excel-expert

Parse and standardize Excel workbook data across multiple sheets into JSON or CSV.

Community
Advanced
idanmoradarthasidanmoradarthas
37

ds-utils-metrics

Analyze and visualize sklearn-compatible model performance with metrics and plots.

Community
Intermediate
mdmagnuson-creatormdmagnuson-creator

test-failure-handling

Log automated test failures to verification-failures.json and test-debt.json.

Community
Advanced