cpu-offloading

What problem does it solve?

Reduces GPU memory pressure by moving transformer activations and optimizer states to CPU memory, enabling larger models and more aggressive memory strategies in Megatron Bridge.

Core Features & Use Cases

• Activation offloading: layer-level offload with per-transformer-layer control, PP constraints, and optional weight offload.
• Optimizer offloading: fractional offload of Adam states via HybridDeviceOptimizer with overlap between GPU and CPU transfers.
• Use cases include training/inference of large models with limited GPU memory and scenarios requiring memory-speed tradeoffs.

Quick Start

Enable optimizer CPU offload with a 0.5 fraction to start reducing GPU memory usage while preserving performance.