capture-nsys-profile

You can capture Nsight Systems traces by running a short PithTrain profile from a released checkpoint. The process automatically sizes the global batch to steady-state and collects representative .nsys-rep files to analyze kernel timelines and all-to-all overhead.

Profiling PithTrain with Nsight Systems requires the Nsight Systems CLI on all nodes and a GPU-enabled environment where world_size >= PP*CP*EP. You must also provide a released checkpoint with benchmark inputs prepared by the setup-benchmark-inputs skill.

Yes, you can profile across adaptive pipeline (PP), expert (EP), and context (CP) parallelism configurations. The profile capture adjusts the global batch to steady-state while collecting representative .nsys-rep traces for each parallelism setting.

Generating .nsys-rep traces provides actionable performance data for kernel timelines and all-to-all communication overhead. This allows you to pinpoint parallelism inefficiencies and apply targeted optimizations across your MoE training configurations.

The best way to analyze all-to-all overhead is to capture Nsight Systems traces during a short PithTrain run. This generates detailed .nsys-rep profiles from a steady-state checkpoint, exposing kernel timelines and communication bottlenecks for optimization.