skills.rest
NVIDIA-NeMo
NVIDIA-NeMoOFFICIAL
Explore all skills by this author →
WebsiteGitHub

perf-cpu-offloading

Offload activations and optimizer states to CPU during large-scale model training.

852|445|Updated May 21, 2025
Search Tags:#deep learning#performance tuning#gpu#memory optimization#cpu offloading#optimizer states
One-click install
npx skills add https://github.com/NVIDIA-NeMo/Megatron-Bridge --skill perf-cpu-offloading
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: perf-cpu-offloading
Source: https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/skills/perf-cpu-offloading
Command: npx skills add https://github.com/NVIDIA-NeMo/Megatron-Bridge --skill perf-cpu-offloading

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps users reduce GPU memory usage and improve training stability by enabling CPU offloading of activations and optimizer states.

Core Features & Use Cases

  • Memory Management: Offload transformer layer activations and optimizer states to CPU to prevent OOM errors during large model training.
  • Performance Tuning: Adjust offloading fractions and configurations for balanced memory savings and training throughput.
  • Use Case: An engineer training a 30B model on limited GPU memory can offload 50% of optimizer states to enable larger batch sizes without upgrading hardware.

Quick Start

Use the perf-cpu-offloading skill to configure optimizer offloading with 50% fraction and enable activation offloading for small models that fit with PP=1.

Frequently Asked Questions about perf-cpu-offloading

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce GPU memory usage for large model training without upgrading hardware?▼

Reduce GPU memory usage by offloading transformer layer activations and optimizer states to CPU memory, preventing out-of-memory errors and allowing training larger models on existing hardware.

What is CPU offloading for optimizer states in deep learning?▼

CPU offloading for optimizer states is a memory optimization technique that transfers these states from GPU to CPU RAM during training, freeing up GPU memory for larger batch sizes and model parameters.

Can I offload only a fraction of optimizer states to balance memory and training speed?▼

Yes, you can configure a specific offloading fraction, such as 50%, to balance memory savings with training throughput and minimize the performance impact of CPU data transfers.

Does CPU offloading work with distributed optimizer setups?▼

Yes, CPU offloading ensures compatibility with distributed optimizer setups, allowing you to scale models efficiently across multiple GPUs while managing memory constraints.

When should I enable activation offloading during deep learning performance tuning?▼

Enable activation offloading when training small models that fit with pipeline parallelism of 1, or when facing GPU RAM limits, to prevent OOM errors without heavily impacting throughput.

Related Skills

Explore More Software Engineering Skills→
HermeticOrmusHermeticOrmus

GPU Optimization Patterns

Optimize PyTorch GPU performance with memory profiling and mixed-precision training.

Community
Advanced
NVIDIANVIDIA
2.8k

memory-tuning

Mitigate GPU memory pressure and OOM during Megatron training with fragmentation fixes.

Official
Advanced
KappTech88KappTech88

deepspeed

Configure DeepSpeed ZeRO, pipeline parallelism, and MoE for large-scale training.

Community
Advanced
matthiaskloftmatthiaskloft

bayesflow-memory

Manage GPU memory for BayesFlow training pipelines to prevent CUDA OOM errors.

Community
Advanced