perf-activation-recompute

To fix out-of-memory (OOM) errors during transformer training, apply selective activation recompute to reduce GPU memory usage. This technique recalculates specific module activations on the fly, freeing up memory to fit larger models or batch sizes without crashing.

Selective activation recompute saves GPU memory by discarding intermediate activations during the forward pass and recomputing them during the backward pass. By targeting specific modules like attention and layernorm, it lowers memory requirements for large-scale training.

Activation recompute optimizes memory efficiency with minimal performance impact. While recomputing activations introduces a slight overhead, it enables successful training of large transformer models that would otherwise fail due to GPU memory constraints.

To configure selective recompute, update your training setup to enable the feature on attention modules. Running your training script with this configuration will selectively recompute those activations, directly reducing peak memory usage.

Yes, activation recompute helps investigate and fix out-of-memory issues caused by memory fragmentation or large model sizes. By lowering the memory footprint of activations, it mitigates fragmentation and allows training to proceed.

Use selective recompute when reducing batch size is not viable or degrades training quality. It allows you to maintain larger batch sizes and fit large transformer models into hardware constraints by strategically managing memory for modules like attention.