perf-memory-tuning

What problem does it solve?

This Skill addresses GPU out-of-memory (OOM) failures caused by memory fragmentation and inefficient resource utilization during model training.

Core Features & Use Cases

• Memory fragmentation mitigation through setting PYTORCH_CUDA_ALLOC_CONF to expandable segments.
• Parallelism adjustment techniques such as increasing tensor parallelism or pipeline parallelism to fit models within GPU memory constraints.
• Activation recompute strategies to reduce peak memory consumption.
• Use Case: When training large language models on H100 GPUs, users can prevent OOM errors by applying these techniques, ensuring stable training without hardware upgrades.

Quick Start

Set the environment variable PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before launching your training session to dramatically reduce memory fragmentation and eliminate OOM errors.