memory-tuning

Fix GPU OOM during Megatron-LM training by enabling expandable segments to reduce memory fragmentation and selectively recomputing activations to lower peak memory usage.

Expandable segments mitigate GPU memory fragmentation by using non-fixed memory blocks, allowing the allocator to dynamically resize segments and avoid peak OOM crashes during large-scale Megatron-LM training.

Use activation recompute to save GPU memory when training large transformer models under tight memory budgets, selectively dropping and recomputing forward activations during the backward pass to lower peak memory usage.

CPU offloading in Megatron-LM has parallelism constraints, requiring careful tuning of tensor, pipeline, and data parallelism trade-offs to ensure offloading decisions maintain training stability and throughput.

Configure PYTORCH_CUDA_ALLOC_CONF for expandable segments by setting the environment variable to expandable_segments:True before launching your Megatron-LM training script to enable non-fixed memory block allocation.

Limitations of CPU offloading for distributed training include strict parallelism compatibility constraints, where incorrect offloading decisions coupled with tensor or pipeline parallelism can degrade throughput and destabilize memory limits.