perf-memory-tuning

Fix CUDA out-of-memory errors by setting the environment variable PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before launching training to reduce memory fragmentation and eliminate OOM failures.

Reduce GPU peak memory by applying activation recompute strategies, resizing tensor parallelism or pipeline parallelism, and enabling expandable segments to fit large models within memory constraints.

Adjusting parallelism helps prevent OOM errors by resizing tensor parallelism or pipeline parallelism to fit models within GPU memory constraints, ensuring stable training on high-end GPU clusters.

Activation recompute reduces peak memory consumption by strategically recomputing activations during the backward pass, freeing up memory for large-scale model training without requiring hardware upgrades.

Configure expandable segments when experiencing GPU out-of-memory failures caused by memory fragmentation, as it maintains training stability and ensures compatibility with various parallelism configurations.

Optimizing GPU memory requires pycuda and nvidia-pyindex dependencies to support troubleshooting OOMs, managing activation memory, and maintaining training stability across parallelism configurations.