Gradient Checkpointing (Activation Recomputation)

What problem does it solve?

Gradient checkpointing addresses the memory bottleneck during training by enabling selective recomputation of activations, allowing larger models or bigger batches within fixed GPU memory.

Core Features & Use Cases

• Activation recomputation trades compute for memory to reduce peak activations during backpropagation.
• Supports multiple strategies (none, full, selective, sequential) and integrates with distributed wrappers like FSDP and DDP.
• Per-timestep checkpointing for SNN components to further reduce memory in long unrolls.
• Works with standard transformer-style architectures and hybrid brain-inspired modules.

Quick Start

Use this to reduce training memory by wrapping expensive submodules with CheckpointWrapper or applying SelectiveCheckpointer according to your memory budget.