debug-distributed-hang

Trace per-rank state and NCCL logs to diagnose multi-GPU SGLang inference hangs.

Updated Jun 29, 2026

Search Tags:#debugging #logs #distributed #diagnostics #cuda #py-spy #NCCL

Ready to run

npx skills add https://github.com/hey-kong/sglang --skill debug-distributed-hang-hey-kong

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: debug-distributed-hang
Source: https://github.com/hey-kong/sglang/tree/main/.claude/skills/debug-distributed-hang
Command: npx skills add https://github.com/hey-kong/sglang --skill debug-distributed-hang-hey-kong

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Debugging hangs in multi-GPU SGLang distributed inference can cause all ranks to stall during collective operations. This skill provides structured methods to locate the root cause and unify debugging workflows.

Core Features & Use Cases

Per-rank logging to observe state across GPUs and identify divergence points.
Watchdog and profiling tools guidance (py-spy, CUDA core dumps, NCCL debug logs) to surface where the hang occurs.
Resolution workflow that guides you from detection to verification, including binary-search of root causes and safe fixes.

Quick Start

Run the distributed hang diagnostic workflow on a failing multi-GPU run to locate the first divergence and apply a validated fix.

Frequently Asked Questions about debug-distributed-hang

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I debug multi-GPU distributed inference hangs in SGLang?▼

Debug multi-GPU distributed inference hangs in SGLang by tracing per-rank state, watchdog traces, and NCCL logs to identify divergence points and stalled collectives. You can then apply a structured workflow to binary-search the root cause and validate fixes.

Why do all ranks stall during NCCL collective operations?▼

All ranks stall during NCCL collective operations when there is a divergence in execution state across GPUs or misrouted communications in TP/PP/DP/EP configurations. Per-rank logging and NCCL debug logs help surface the specific rank where the hang originates.

How to use py-spy to diagnose a distributed multi-GPU hang?▼

Use py-spy alongside CUDA debugging tools and NCCL debug logs to profile and dump the state of stalled ranks during a multi-GPU hang. This combination surfaces exactly where the collective operation freezes and helps trace the root cause of the divergence.

Do I need CUDA debugging tools to fix multi-GPU SGLang inference stalls?▼

You need CUDA debugging tools, py-spy, and NCCL logs to reproduce and validate fixes for multi-GPU SGLang inference stalls. These tools are required to trace per-rank state, analyze core dumps, and confirm that the applied resolution prevents future hangs.

What is the best way to locate rank divergence in TP/PP/DP/EP configurations?▼

The best way to locate rank divergence in TP/PP/DP/EP configurations is to enable per-rank logging and analyze watchdog traces to find the first rank that deviates. This structured detection workflow isolates the exact point of misrouted communications or stalled collectives.