nemo-mbridge-perf-memory-tuning

To resolve GPU out-of-memory errors during Megatron Bridge training, enable expandable_segments via PYTORCH_CUDA_ALLOC_CONF to reduce memory fragmentation and prevent OOMs with zero throughput cost.

The best way to reduce peak GPU memory for Llama3-70B-style workloads is to combine expandable_segments with memory planning strategies like activation recompute and tuning parallelism configurations to satisfy safety and performance goals.

CPU offloading is blocked when pipeline model parallel size is greater than 1. This compatibility constraint prevents offloading in multi-stage pipeline configurations due to CUDA graph considerations.

Use activation recompute for GPU memory management as a secondary option when expandable segments are insufficient to mitigate fragmentation-induced memory pressure during training with long sequences.

Tuning tensor and pipeline parallelism configurations helps prevent OOMs in Megatron by distributing the workload to satisfy safety and performance goals while balancing memory planning constraints like CPU offloading limits.