Capture and replay GPU operations to reduce host overhead in Megatron Bridge training.
npx skills add https://github.com/NVIDIA/skills --skill cuda-graphs
Please help me install this Agent Skill. Skill: cuda-graphs Source: https://github.com/NVIDIA/skills/tree/main/skills/Megatron-Bridge/perf-techniques/cuda-graphs Command: npx skills add https://github.com/NVIDIA/skills --skill cuda-graphs
CUDA graphs enable stable capture and replay of GPU operations to reduce host overhead during Megatron Bridge training.
It supports two implementations: local full-iteration graphs via MCore and Transformer Engine scoped graphs for per-layer capture of attention, MLP, and MoE components, making it suitable for dense, MoE, and large-model training scenarios. Use cases include accelerating pretraining and fine-tuning by reducing iteration time and increasing throughput.
Enable CUDA graphs by selecting a graph implementation and scope, then start training as usual.
High-intent search queries and answers about installing and using this skill.
To reduce GPU host overhead during Megatron Bridge training, enable CUDA graphs to capture and replay GPU operations, which stabilizes execution and minimizes launch latency. This supports both full-iteration graphs and Transformer Engine scoped graphs for attention and MLP modules.
Local full-iteration graphs capture entire training iterations via MCore, while Transformer Engine scoped graphs perform per-layer capture for attention, MLP, and MoE components. Both reduce host overhead but differ in their capture granularity and applicable module scopes.
Yes, you can use CUDA graphs for MoE model training. The implementation supports Transformer Engine scoped graphs that specifically capture Mixture of Experts modules alongside attention and MLP components, making it suitable for both dense and MoE architectures.
To enable CUDA graphs, select a graph implementation and scope, then start training as usual. You must ensure your environment meets requirements like maintaining static tensor shapes and enabling the Transformer Engine RNG tracker before capture.
Limitations of GPU graph capture include the requirement for static tensor shapes and adherence to mutual exclusions with other runtime constraints. You must also enable the Transformer Engine RNG tracker to ensure correct execution during graph replay.
sayalinvidiaImplements CUDA graph capture for Megatron Bridge with Transformer Engine and RNG tracking.
NVIDIA-NeMoCapture CUDA graphs to reduce host overhead during GPU model training.
yo-stevenCapture and replay CUDA graphs in Megatron Bridge to reduce training step latency.
yo-stevenCaptures and replays PyTorch GPU workloads using CUDA Graphs to reduce launch overhead.
