memory-tuning

Mitigate GPU memory pressure and OOM during Megatron training with fragmentation fixes.

2.8k|332|Updated Feb 25, 2026

Search Tags:#memory-management #oom #megatron-lm #gpu-memory #cpu-offloading #expandable_segments #activation-recompute

Ready to run

npx skills add https://github.com/NVIDIA/skills --skill memory-tuning

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: memory-tuning
Source: https://github.com/NVIDIA/skills/tree/main/skills/Megatron-Bridge/perf-techniques/memory-tuning
Command: npx skills add https://github.com/NVIDIA/skills --skill memory-tuning

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

GPU memory fragmentation and peak memory usage during Megatron training often cause OOM or reduced throughput. This memory-tuning guide provides proven fixes to stabilize training on large models.

Core Features & Use Cases

Expandable segments: reduce fragmentation by using non-fixed memory blocks.
Activation recompute: selectively recompute activations to save peak memory.
CPU offloading constraints: guidance on when offloading is compatible with parallelism.
Parallelism tuning: advise TP/PP/DP trade-offs to fit memory budgets for large-scale training.

Quick Start

Set PYTORCH_CUDA_ALLOC_CONF to expandable_segments:True before launching Megatron training.

Frequently Asked Questions about memory-tuning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I fix GPU OOM errors during Megatron-LM training?▼

Fix GPU OOM during Megatron-LM training by enabling expandable segments to reduce memory fragmentation and selectively recomputing activations to lower peak memory usage.

How does expandable_segments reduce GPU memory fragmentation?▼

Expandable segments mitigate GPU memory fragmentation by using non-fixed memory blocks, allowing the allocator to dynamically resize segments and avoid peak OOM crashes during large-scale Megatron-LM training.

When should I use activation recompute to save GPU memory?▼

Use activation recompute to save GPU memory when training large transformer models under tight memory budgets, selectively dropping and recomputing forward activations during the backward pass to lower peak memory usage.

Can I use CPU offloading with tensor and pipeline parallelism in Megatron-LM?▼

CPU offloading in Megatron-LM has parallelism constraints, requiring careful tuning of tensor, pipeline, and data parallelism trade-offs to ensure offloading decisions maintain training stability and throughput.

How do I configure PYTORCH_CUDA_ALLOC_CONF for expandable segments?▼

Configure PYTORCH_CUDA_ALLOC_CONF for expandable segments by setting the environment variable to expandable_segments:True before launching your Megatron-LM training script to enable non-fixed memory block allocation.

What are the limitations of CPU offloading for distributed training?▼

Limitations of CPU offloading for distributed training include strict parallelism compatibility constraints, where incorrect offloading decisions coupled with tensor or pipeline parallelism can degrade throughput and destabilize memory limits.