pytorch-fsdp

PyTorch FSDP model sharding reduces GPU memory by distributing parameters, gradients, and optimizer states across processes. During forward and backward passes, it uses all-gather to reconstruct layers and reduce-scatter to synchronize gradients, enabling larger model training.

FSDP2 fully_shard uses DTensor-based per-parameter sharding and hook-driven unsharding/resharding instead of module-level wrapping. This approach provides finer control over parameter sharding and requires adhering to specific forward and backward execution contracts to ensure correct distributed training.

To configure FSDP2, apply fully_shard bottom-up through your model modules, set up the optimizer with sharded parameters, and ensure forward execution respects DTensor semantics. You must also manage mixed precision and CPU offloading configurations based on your mesh and world size.

PyTorch FSDP training hangs often stem from uneven input handling across process groups or mismatched collective operations. Using the Join context manager for uneven inputs and verifying communication backend synchronization helps identify and resolve these execution contract violations.

Yes, using FSDP2 requires knowledge of PyTorch distributed fundamentals including process groups, communication backends, and synchronization. Understanding these prerequisites is essential for correctly implementing DTensor parameter sharding and debugging execution contract issues.

Use CPU offloading with PyTorch FSDP when GPU memory is insufficient for your model size and sharding alone does not resolve the constraint. It offloads parameters or optimizer states to CPU RAM, but requires careful configuration to avoid communication bottlenecks during the forward and backward passes.