skills.rest
sayalinvidia
sayalinvidiaCOMMUNITY
Explore all skills by this author →
GitHub

nemo-mbridge-perf-cpu-offloading

Configure CPU offloading options for Megatron Bridge training to reduce GPU memory usage.

Updated Apr 16, 2026
Search Tags:#memory optimization#distributed training#gpu memory#cpu offloading#megatron bridge#optimizer offload#activation offloading
One-click install
npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-cpu-offloading
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-mbridge-perf-cpu-offloading
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/nemo-mbridge-perf-cpu-offloading
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-perf-cpu-offloading

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill helps Megatron Bridge users reduce GPU memory pressure by offloading computations to CPU and distributing optimizer states, enabling training of larger models or longer sequences and aiding diagnosis of OOM or crashes due to offload configuration.

Core Features & Use Cases

  • Layer-level activation offloading (cpu_offloading) to move activations to CPU and reduce GPU memory usage; suitable for small-to-medium models with PP constraints.
  • Optimizer CPU offloading (optimizer_cpu_offload) using HybridDeviceOptimizer to place optimizer states on CPU; supports configurable offload fractions for memory-speed tradeoffs.
  • Flexible offload_fraction and overlap options (optimizer_offload_fraction, overlap_cpu_optimizer_d2h_h2d) to tune performance versus memory savings.
  • Compatibility guidance and caveats: activation offloading requires PP=1 and cannot be combined with recompute or CUDA graphs; optimizer offloading has no PP restriction but may incur CPU overhead.

Quick Start

Enable optimizer CPU offload and optional activation offload in Megatron Bridge configuration to begin reducing GPU memory usage.

Frequently Asked Questions about nemo-mbridge-perf-cpu-offloading

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce GPU memory pressure during distributed training?▼

Optimizer CPU offloading places optimizer states on CPU memory using HybridDeviceOptimizer. You configure optimizer_cpu_offload and tune optimizer_offload_fraction to balance memory savings against training speed and CPU overhead.

Can I use activation offloading with pipeline parallelism in Megatron Bridge?▼

Activation offloading requires pipeline parallelism set to one (PP=1) and cannot be combined with recompute or CUDA graphs. It moves layer-level activations to CPU, suiting small-to-medium models under strict memory limits.

Does optimizer offloading have pipeline parallelism restrictions?▼

Optimizer offloading has no pipeline parallelism restrictions and works across PP configurations. It distributes optimizer states to CPU, though it may incur CPU overhead when exchanging data between host and device.

What are the tradeoffs between optimizer and activation offloading for large MoE models?▼

For large MoE models, optimizer offloading offers flexible fraction tuning without PP limits, while activation offloading provides layer-level memory reduction but requires PP=1 and forbids recompute. Choose based on pipeline constraints.

Why does my activation offload configuration fail when CUDA graphs are enabled?▼

Activation offloading fails with CUDA graphs because the two features are incompatible by design. You must disable CUDA graphs and recompute before enabling cpu_offloading for activations in Megatron Bridge.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

memory-tuning

Mitigate GPU memory pressure and OOM during Megatron training with fragmentation fixes.

Official
Advanced
KappTech88KappTech88

deepspeed

Configure DeepSpeed ZeRO, pipeline parallelism, and MoE for large-scale training.

Community
Advanced
NVIDIANVIDIA
2.8k

activation-recompute

Configure selective or full activation recompute to trade memory for compute in Megatron Bridge.

Official
Advanced
danmartinez78danmartinez78

serving-llms-vllm

Deploy vLLM servers with OpenAI-compatible endpoints and quantization.

Community
Advanced