perf-cuda-graphs

To reduce kernel launch overhead during GPU training, you can use CUDA graph capture to record and replay static execution flows, minimizing host-driver overhead and significantly speeding up training iterations.

CUDA graph capture is a technique that records a sequence of GPU operations into a reusable graph. It optimizes deep learning workloads by eliminating per-kernel launch overhead, resulting in increased throughput and reduced iteration time for static-shape models.

To enable CUDA graphs for attention and MoE modules, set the implementation to 'transformer_engine' and configure the capture scope to target these specific modules during your training loop setup.

CUDA graph capture is not suitable for models with dynamic shapes. It is designed specifically for static-shape, large-scale deep learning tasks where execution flows remain constant, ensuring valid performance tuning and reduced iteration time.

Yes, CUDA graph capture works with Transformer Engine, supporting TE-scoped graphs for attention, MLP, and MoE modules to accelerate the training of large-scale language models on GPU clusters by minimizing host overhead.