perf-memory-tuning

To fix GPU OOM errors in Megatron Bridge, set the environment variable PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before launching training to reduce memory fragmentation and resolve borderline out-of-memory crashes without changing model configuration.

GPU memory fragmentation occurs when allocator gaps prevent fitting tensors despite having free memory. Enabling PyTorch CUDA expandable segments addresses this fragmentation-induced OOM by allowing the allocator to manage memory regions dynamically.

To reduce activation memory peak, use selective activation recompute to trade extra compute for lower memory usage. This avoids changing parallelism configuration while lowering the activation memory footprint during training.

You cannot use CPU offloading with pipeline model parallel size greater than one. CPU offloading in Megatron Bridge is blocked when pipeline_model_parallel_size is greater than one due to compatibility constraints.

Adjust tensor, pipeline, or data parallelism to resolve out-of-memory errors when the model capacity truly does not fit in GPU memory. This shifts parameter and optimizer memory placement across devices after allocator tuning proves insufficient.