nemo-mbridge-perf-activation-recompute

What problem does it solve?

Activation recompute trades GPU memory for compute by discarding intermediate activations during the forward pass and recomputing them during backward in Megatron Bridge, enabling training under tighter memory budgets.

Core Features & Use Cases

• Supports selective recompute to save memory by recomputing specific submodules (e.g., core_attn, layernorm) with moderate compute cost.
• Supports full-layer recompute to maximize memory savings when memory pressure is extreme, with guidance on when to apply.
• Provides compatibility guidance for TE-scoped CUDA graphs and related constraints during memory tuning for large models.

Quick Start

Configure selective recompute first (e.g., core_attn; optionally add layernorm), then escalate to full recompute with recompute_num_layers and recompute_method if memory pressure persists.