perf-nsight-systems

Nsight Systems profiling identifies GPU idle gaps by capturing a full system timeline with nsys profile, then using nsys stats and nsys analyze to quantify where time is spent across CPU, GPU, OS runtime, and NCCL communication overlap.

The best way to profile a CUDA timeline with NVTX annotations is to run nsys profile to generate a .nsys-rep report, then summarize it with nsys stats and nsys analyze to trace CUDA activity and attribute time spent to specific NVTX annotations.

You use nsys stats to diagnose slow data loading by analyzing the generated .nsys-rep timeline to identify synchronous memcpy operations, API overhead, and CPU bottlenecks that block GPU compute utilization and cause anti-patterns.

Yes, Nsight Systems profiling works for analyzing NCCL and compute overlap by capturing distributed communication alongside CUDA activity in the .nsys-rep report, allowing you to validate iteration consistency and diagnose communication bottlenecks.

You should use nsys profile when you need an end-to-end deep learning timeline investigation across CPU, GPU, OS runtime, and communication, rather than isolated kernel-level metrics, to identify complex overlap failures and validate iteration consistency.