skills.rest
FutureMLS-Lab
FutureMLS-LabOFFICIAL
Explore all skills by this author →
GitHub

debug-distributed-hang

Diagnose deadlocked SGLang distributed inference runs using NCCL diagnostics and per-rank logs.

556|81|Updated May 19, 2026
Search Tags:#sglang#nccl#distributed inference#multi-gpu#hang debugging#rank divergence
One-click install
npx skills add https://github.com/FutureMLS-Lab/OSCAR --skill debug-distributed-hang-futuremls-lab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: debug-distributed-hang
Source: https://github.com/FutureMLS-Lab/OSCAR/tree/main/sglang-research/.claude/skills/debug-distributed-hang
Command: npx skills add https://github.com/FutureMLS-Lab/OSCAR --skill debug-distributed-hang-futuremls-lab

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps diagnose and resolve SGLang distributed inference runs that freeze, time out, or deadlock when GPU ranks diverge during collective operations.

Core Features & Use Cases

  • Hang Localization: Use watchdog traces, py-spy, NCCL logging, and CUDA coredumps to identify the blocked collective and code path.
  • Rank State Comparison: Add structured per-rank logging and tensor or token hashes to detect state divergence without unnecessarily disrupting execution.
  • Root-Cause Isolation: Diff event streams and binary-search upstream inputs to locate the first divergence, then apply fixes for nondeterminism, conditional paths, resource failures, and pipeline send or receive mismatches.
  • Use Case: When a multi-GPU SGLang serving job hangs during decoding, use this Skill to identify the stalled AllGather, compare rank state, trace the earliest mismatch, and verify the fix across repeated runs.

Quick Start

Use the debug-distributed-hang skill to investigate the current SGLang multi-GPU timeout, identify the blocked collective, and guide me through collecting and comparing per-rank evidence.

Frequently Asked Questions about debug-distributed-hang

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug a multi-GPU SGLang inference hang during collective operations?▼

Debug multi-GPU SGLang inference hangs by using watchdog traces, py-spy, and NCCL diagnostics to identify blocked collectives, then comparing per-rank logs and tensor hashes to pinpoint state divergence.

What causes distributed inference deadlocks in tensor and pipeline parallel workloads?▼

Distributed inference deadlocks occur when GPU ranks diverge during collective operations like AllGather, AllReduce, or Barrier, often caused by nondeterminism, conditional path mismatches, or failed point-to-point sends and receives.

How do I find the first rank state divergence causing an NCCL timeout?▼

Find the first rank state divergence by diffing structured per-rank event streams and binary-searching upstream inputs, using tensor hashing to detect the earliest mismatch without disrupting execution.

Does this hang debugging approach work for expert and data parallelism?▼

Yes, this hang debugging approach diagnoses freezing, timeouts, and deadlocks across tensor, pipeline, data, and expert parallel workloads that fail during GPU communication.

Can I use CUDA coredumps to diagnose stalled AllReduce operations?▼

Yes, you can use optional CUDA coredumps alongside NCCL logging and py-spy traces to localize the exact stalled AllReduce collective and identify the blocked code path.

Related Skills

Explore More Software Engineering Skills→
spacecat2002spacecat2002

debug-distributed-hang

Diagnose hanging SGLang distributed inference jobs by locating blocked NCCL collectives and divergent rank states.

Community
Advanced
abhishekdutta18abhishekdutta18

agent-introspection-debugging

Diagnose AI agent failures and generate introspection reports.

Community
Advanced
sqjiansqjian

debug-distributed-hang

Diagnose distributed SGLang hangs using py-spy traces and NCCL logs.

Community
Advanced
TorchedHatTorchedHat
6

distributed-hang-diagnosis

Diagnose PyTorch distributed training hangs and identify root causes.

Official
Advanced