What problem does it solve?
Profiling distributed GPU training with Nsight Systems often leads to wrong conclusions: summed kernel time is mistaken for wall time, NCCL duration is mistaken for a bottleneck, and rank 0 is treated as representative. This Skill turns .nsys-rep and exported .sqlite traces into a rigorous critical-path diagnosis with a ranked, evidence-backed optimization plan for Megatron Bridge training runs.
Core Features & Use Cases
- Critical-path time accounting: Builds per-iteration device budgets using interval unions, splits compute-absent time into launch-starved, blocking, and dependency-stalled buckets, and reconciles every millisecond.
- Multi-rank and communication analysis: Surveys all captured ranks, separates transfer proxy from jitter wait, distinguishes exposed versus blocking communication, and inventories non-NCCL backends like HybridEP, DeepEP, and NVSHMEM.
- Evidence-backed gain estimation: Computes recoverable-time ceilings for step time, throughput, and MFU without double counting, and maps proven bottlenecks to follow-up tuning skills (CUDA graphs, TP/DP/MoE communication overlap, CPU offloading, memory tuning).
- Use Case: Given traces from a PP=8 run where NCCL appears to dominate, use this Skill to prove whether collectives actually block compute, identify the true straggler rank after cross-host clock alignment, and produce a ranked optimization plan with defensible gain ceilings.
Quick Start
Use the nemo-mbridge-perf-nsys-analysis skill to analyze my Megatron Bridge trace file profile.nsys-rep and tell me the dominant bottleneck and ranked optimization opportunities.
