debug-distributed-hang

Diagnose distributed SGLang hangs across TP/PP/DP/EP with per-rank logging and binary-search root-cause analysis.

Updated May 30, 2026

Search Tags:#debugging #logging #distributed #inference #sglang #nccl

Ready to run

npx skills add https://github.com/QingHuan-6/SglangGen --skill debug-distributed-hang-qinghuan-6

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: debug-distributed-hang
Source: https://github.com/QingHuan-6/SglangGen/tree/main/.claude/skills/debug-distributed-hang
Command: npx skills add https://github.com/QingHuan-6/SglangGen --skill debug-distributed-hang-qinghuan-6

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Diagnoses and resolves hangs in distributed SGLang runs across TP/PP/DP/EP by guiding detection, logging, and root-cause analysis.

Core Features & Use Cases

Identify hang location using py-spy, watchdog, NCCL logs, and CUDA coredumps.
Per-rank logging and diffing to find divergence points across GPUs.
Binary-search root cause and practical fixes for common hang scenarios (size mismatches, branch divergence, pipeline issues).
Step-by-step verification of fixes by repeated runs.

Quick Start

Follow the guided diagnostics to locate and fix a multi-GPU hang in a distributed SGLang run.

Frequently Asked Questions about debug-distributed-hang

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I debug distributed SGLang hangs during multi-GPU inference?▼

Debug distributed SGLang hangs by using py-spy, watchdog, and NCCL logs to identify stall locations across TP/PP/DP/EP ranks. Analyze per-rank logs to find divergence points and apply binary-search workflows to validate fixes.

Why does my NCCL AllGather or AllReduce operation stall and timeout?▼

NCCL collectives like AllGather or AllReduce stall when ranks diverge across GPUs. Use watchdog-triggered timeouts and CUDA coredumps to detect the hang, then diff per-rank logs to pinpoint the exact divergence point causing the timeout.

How do I locate the first rank divergence in a multi-GPU training hang?▼

Locate the first rank divergence by enabling per-rank logging and diffing the outputs across GPUs. Apply the binary-search root cause workflow to systematically narrow down the initial point of failure during distributed runs.

What are common causes of distributed inference hangs across TP and PP ranks?▼

Common causes of distributed inference hangs include tensor size mismatches, branch divergence, and pipeline issues across TP and PP ranks. Step-by-step verification through repeated runs confirms whether applied fixes resolve the divergence.

Can I use py-spy and CUDA coredumps to diagnose SGLang watchdog timeouts?▼

You can use py-spy and CUDA coredumps alongside NCCL debugging to diagnose SGLang watchdog timeouts. These tools capture the exact hang location and state of multi-GPU processes when collectives stall during distributed execution.