Gradient Checkpointing (Activation Recomputation)

Gradient checkpointing reduces PyTorch training memory by enabling selective recomputation of activations, trading compute for memory to allow larger models or bigger batches within fixed GPU limits.

Yes, gradient checkpointing integrates directly with FSDP and DDP distributed training wrappers to resolve activation memory bottlenecks across multiple GPUs during large-scale training.

Activation recomputation supports none, full, selective, and sequential checkpointing strategies, letting you wrap expensive submodules with CheckpointWrapper or apply SelectiveCheckpointer according to your memory budget.

Selective recomputation targets specific expensive submodules to balance memory savings and compute overhead, whereas full checkpointing recomputes all activations, making selective ideal for optimizing memory budgets without excessive slowdown.

Yes, per-timestep checkpointing supports SNN components during long unrolls to further reduce activation memory in hybrid brain-inspired modules and transformer-style architectures.