skills.rest
TorchedHat
TorchedHatOFFICIAL
Explore all skills by this author →
GitHub

distributed-hang-diagnosis

Diagnose PyTorch distributed training hangs and identify root causes.

6|8|Updated May 7, 2026
Search Tags:#pytorch#distributed training#nccl#flight recorder#hang diagnosis#collective mismatch#processgroupnccl
One-click install
npx skills add https://github.com/TorchedHat/ai-marketplace --skill distributed-hang-diagnosis
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-hang-diagnosis
Source: https://github.com/TorchedHat/ai-marketplace/tree/main/torch-distributed/skills/distributed-hang-diagnosis
Command: npx skills add https://github.com/TorchedHat/ai-marketplace --skill distributed-hang-diagnosis

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps identify and resolve PyTorch distributed training freezes by distinguishing deadlocks, communicator initialization failures, collective mismatches, NCCL timeouts, and DDP or FSDP execution issues.

Core Features & Use Cases

  • Hang Classification: Classify distributed failures across barrier deadlocks, p2p hangs, store initialization timeouts, collective mismatches, NCCL timeouts, unused parameters, and forward-order violations.
  • Source-Level Diagnosis: Trace ProcessGroupNCCL communicator lifecycle, watchdog behavior, work queues, store-based initialization, timeout layers, and flight recorder data.
  • Evidence-Based Verification: Use debug logs, cross-rank sequence comparisons, minimal reproducers, and per-pattern checklists to validate fixes instead of relying on assumptions.
  • Use Case: When a multi-GPU training job freezes during barrier or send/receive operations, use this Skill to classify the hang, inspect the relevant NCCL and distributed traces, identify the likely root cause, and verify that every rank completes safely.

Quick Start

Use the distributed hang diagnosis skill to analyze the attached PyTorch logs and classify the freeze, likely root cause, recommended fix, and verification evidence.

Frequently Asked Questions about distributed-hang-diagnosis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
Why does my PyTorch distributed training hang during NCCL operations?▼

Diagnosing PyTorch distributed hangs involves classifying NCCL barrier deadlocks, collective mismatches, and DDP or FSDP execution violations. Analyzing cross-rank logs and flight recorder traces identifies the specific root cause of the freeze.

How do I diagnose a collective mismatch in PyTorch distributed training?▼

Diagnosing a collective mismatch requires comparing cross-rank debug logs and flight recorder traces to trace the ProcessGroupNCCL communicator lifecycle. This evidence-based approach validates the sequence discrepancies causing the hang.

Can I use flight recorder traces to debug NCCL timeouts?▼

Yes, flight recorder traces are used to debug NCCL operation timeouts by inspecting watchdog behavior and work queues. This source-level diagnosis traces the ProcessGroupNCCL timeout layers to find the exact failure point.

What is the best way to identify FSDP forward-order violations causing training freezes?▼

Identifying FSDP forward-order violations requires analyzing distributed traces and applying evidence-based verification using per-pattern checklists. This validates that every rank completes safely and confirms the execution order discrepancy.

Does this approach require debug configuration to resolve store initialization timeouts?▼

Yes, resolving store initialization timeouts requires debug configuration and cross-rank sequence comparisons. Tracing the store-based initialization lifecycle within ProcessGroupNCCL provides the evidence needed to verify the proposed fix.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
320

fr-analysis

Analyze PyTorch NCCL flight-recorder dumps to identify collective hangs and suspect ranks.

Official
Advanced
fegome90-cmdfegome90-cmd

superpowers-test-driven-development

Guide Red-Green-Refactor cycles for features and bugfixes.

Community
Advanced
eddieluengeddielueng
1

pytorch-fsdp

Configure and troubleshoot PyTorch FSDP2 fully_shard distributed training.

Community
Advanced
MarbleSodasMarbleSodas

pytorch-fsdp

Configure and debug PyTorch FSDP/FSDP2 distributed training with sharding and offloading.

Community
Advanced