activation-recompute

Activation recompute reduces GPU memory by discarding intermediate activations during the forward pass and recomputing them during backward in Megatron Bridge, trading compute for memory relief.

Selective recompute targets specific submodules using recompute_modules, while full recompute operates per transformer layer using recompute_num_layers and recompute_method, offering different memory-compute tradeoffs.

Set recompute_granularity to selective and specify target submodules in recompute_modules to begin trading memory for compute in Megatron Bridge transformer training workloads.

Activation recompute in Megatron Bridge requires awareness of CUDA graph and CPU offloading constraints, providing clear enablement guidelines and compatibility notes for these features.

Use activation recompute when memory is a bottleneck during large transformer training, especially with models featuring large hidden dimensions and long sequence lengths, enabling larger models without hardware upgrades.