perf-cpu-offloading

You can reduce GPU memory usage by offloading either layer activations or Adam optimizer states to CPU, which relieves memory pressure and helps prevent out-of-memory errors during training.

Enable optimizer CPU offloading by setting optimizer.optimizer_cpu_offload=True and optimizer.optimizer_offload_fraction to a value like 0.5, optionally enabling overlap_cpu_optimizer_d2h_h2d for higher fractions to maintain throughput.

No, activation offloading requires pipeline_model_parallel_size = 1 and cannot be used with recompute or CUDA graphs, making it suitable only for small to medium models without pipeline parallelism.

Yes, optimizer offloading via HybridDeviceOptimizer supports pipeline parallelism with PP>1 and is validated for large MoE runs, offering a controllable memory-speed tradeoff for large-scale training.

CUDA graphs cannot be used with activation offloading, which also requires no recompute and pipeline_model_parallel_size = 1, restricting this specific offloading technique to small or medium model configurations.

Enable optimizer.overlap_cpu_optimizer_d2h_h2d=True when using higher optimizer offload fractions to balance memory savings against throughput and keep training numerically stable.