skills.rest
ruvnet
ruvnetCOMMUNITY
Explore all skills by this author →
Website𝕏@ruvGitHub

gaia-debugging

Diagnose failed GAIA benchmark tasks by classifying failure modes and proposing targeted fixes.

70.1k|8.4k|Updated Jun 2, 2025
Search Tags:#root cause#failure analysis#benchmark#Debugging#Agent#GAIA#Trace
One-click install
npx skills add https://github.com/ruvnet/claude-flow --skill gaia-debugging
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gaia-debugging
Source: https://github.com/ruvnet/claude-flow/tree/main/plugins/ruflo-workflows/skills/gaia-debugging
Command: npx skills add https://github.com/ruvnet/claude-flow --skill gaia-debugging

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

When a GAIA benchmark question fails, it is hard to tell whether the root cause is a missing tool, a reasoning error, an answer-extraction bug, a loop, or a timeout. This Skill provides a systematic workflow to load the failing task's trace, classify the failure mode, and apply a verified fix before resubmitting.

Core Features & Use Cases

  • Failure Mode Taxonomy: Classifies failures into six codes (Tool Gap, Reasoning Miss, Extraction Bug, Loop Issue, Dataset Shift, API Timeout) with symptom-to-fix mappings.
  • Trace-Based Diagnosis: Loads the task result from the latest run, inspects tool calls, and re-runs the single question with extended logging and higher max-turns.
  • Pattern Memory: Stores verified fixes in a persistent memory namespace so similar past failures can be searched and reused.
  • Use Case: A GAIA run reports task_id X answered incorrectly. You load its trace, discover the FINAL_ANSWER regex missed the answer, fix the extraction pattern, re-run the single task to verify, and store the pattern for future sessions.

Quick Start

Ask the assistant to diagnose why GAIA task_id abc123 failed in the latest benchmark run and propose a fix.

Frequently Asked Questions about gaia-debugging

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug a failed GAIA benchmark question?▼

Load the task's trace from the latest results file, inspect which tools were called and what they returned, then classify the failure using the six-mode taxonomy. Re-run the single task with extended logging and higher max-turns to confirm the root cause before applying a fix.

What are the common GAIA agent failure modes?▼

The taxonomy defines six modes: Tool Gap (missing capability like image OCR), Reasoning Miss (right data, wrong conclusion), Extraction Bug (FINAL_ANSWER regex fails), Loop Issue (repeated tool calls hitting max-turns), Dataset Shift (stale ground truth), and API Timeout.

How do I re-run a single GAIA task with more logging?▼

Run the gaia-bench CLI with --task-id for the specific question, --limit 1, --max-turns 20, and --output json. This isolates the failing question and produces a detailed trace for inspection without re-running the full benchmark.

Why does a GAIA task fail even when the correct answer appears in the trace?▼

This is an Extraction Bug: the FINAL_ANSWER regex fails to parse the agent's output, so the answer is marked wrong despite being present. Test the extraction regex against the trace manually and fix the pattern.

Can I reuse fixes from previous GAIA debugging sessions?▼

Yes. Verified fixes are stored in the gaia-debug-patterns memory namespace with the task_id, failure mode, and fix description. You can search this namespace for similar past failures before starting a new diagnosis.

Related Skills

Explore More Software Engineering Skills→
sourcegraphsourcegraph
32

watch-benchmarks

Monitor benchmark run statuses and generate structured JSON or table summaries.

Official
Intermediate
avifeneshavifenesh
951

perf-benchmarker

Run sequential benchmarks with 60-second minimum durations and JSON metrics output.

Community
Advanced
Darwin-AgentDarwin-Agent
415

gaia-playbook

Diagnose GAIA trajectory gaps and select intervention patterns for score improvements.

Official
Advanced
tokyo-megacorptokyo-megacorp

challenge

Benchmark debate agents' bug detection accuracy on curated code challenges with F1 scores.

Official
Intermediate