skills.rest
yo-steven
yo-stevenCOMMUNITY
Explore all skills by this author →
GitHub

perf-activation-recompute

Configure selective activation recompute to prevent GPU OOM in Megatron Bridge models.

Updated May 23, 2026
Search Tags:#performance tuning#gpu memory#cuda graphs#activation recompute#megatron bridge#oom debugging#fp8 training
One-click install
npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-activation-recompute-yo-steven
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: perf-activation-recompute
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/Megatron-Bridge/perf-activation-recompute
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-activation-recompute-yo-steven

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Activation recompute helps you avoid GPU out-of-memory (OOM) errors by reducing peak activation memory during training, at the cost of additional compute.

Core Features & Use Cases

  • Selective activation recompute: recompute selected transformer submodules (e.g., core_attn, mlp, layernorm) to shrink memory usage with controlled throughput impact.
  • Full-layer recompute: recompute a chosen number of full transformer layers using recompute_granularity=full when selective recompute still does not fit.
  • Compatibility guidance: handles key constraints like TE-scoped CUDA graphs incompatibility with full-layer recompute and PP/CPU-offloading limitations.
  • Use case: When a commit changes recompute-related settings and your run starts OOM’ing, apply targeted recompute modules first (often core_attn, then layernorm, and finally mlp) and only fall back to full-layer recompute if needed.

Quick Start

Try setting recompute_granularity=selective with recompute_modules=[core_attn], then add layernorm if you are still near the memory limit and use mlp only as the last selective step.

Frequently Asked Questions about perf-activation-recompute

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fix activation OOM errors during training without changing batch size?▼

Activation recompute prevents GPU OOM errors by trading stored activations for recomputation during backprop, reducing peak memory usage at the cost of additional compute throughput.

What is the best way to apply selective recompute in Megatron Bridge models?▼

Set recompute_granularity=selective and target submodules sequentially: start with core_attn, add layernorm if near the memory limit, and use mlp only as the final selective step.

When should I use full-layer recompute instead of selective activation recompute?▼

Use full-layer recompute with recompute_granularity=full only when selective recompute modules are insufficient to prevent OOM and memory capacity failures persist during training.

Does full-layer recompute work with TE-scoped CUDA graphs?▼

No, TE-scoped CUDA graphs are incompatible with full-layer recompute, requiring careful configuration validation to avoid memory fragmentation or capacity failures during training.

Why does my training run OOM after changing recompute settings?▼

Changing recompute settings can induce memory fragmentation or capacity failures, requiring targeted diagnosis of recompute granularity and module selection to resolve the regression.

Can I use CPU offloading with full-layer recompute for large models?▼

CPU-offloading has specific limitations with full-layer recompute configurations, requiring validation against pipeline parallelism constraints to ensure stable training execution.

Related Skills

Explore More Software Engineering Skills→
sayalinvidiasayalinvidia

nemo-mbridge-perf-memory-tuning

Mitigate GPU memory pressure in Megatron Bridge training with expandable segments.

Community
Intermediate
NVIDIANVIDIA
2.8k

memory-tuning

Mitigate GPU memory pressure and OOM during Megatron training with fragmentation fixes.

Official
Advanced
NVIDIA-NeMoNVIDIA-NeMo
852

perf-memory-tuning

Reduce GPU peak memory usage during large-scale model training by adjusting parallelism and activation recompute strategies.

Official
Intermediate
NVIDIANVIDIA
2.8k

cpu-offloading

Offload GPU activations and optimizer states to CPU memory during Megatron Bridge training.

Official
Intermediate