pytorch-fsdp

PyTorch FSDP shards parameters across ranks, performing all-gather to unshard for forward and backward passes, then reduce-scatter for gradients. This reduces memory while maintaining correct distributed training semantics across devices.

To apply FSDP2 with fully_shard, structure your Transformer sharding plan bottom-up using a 1D or 2D DeviceMesh. This hook-driven workflow uses DTensor-based sharding to manage unshard and reshard behavior efficiently during training.

Distributed training hangs during forward or backward passes are typically caused by mismatched collectives across ranks. You must verify that your sharding strategy and DDP collectives are correctly synchronized to prevent stalls and incorrect training behavior.

Configuring mixed precision for FSDP requires choosing execution settings that balance throughput and numerical stability. You need to select appropriate mixed-precision options to ensure stable training while optimizing runtime performance.

Yes, you can use CPU offloading with PyTorch FSDP to train large models on limited GPU memory. Proper offloading considerations help reduce memory consumption while maintaining correct distributed semantics during training.

FSDP2 fully_shard uses a hook-aware workflow with DTensor-based sharding, whereas FSDP1 relies on traditional parameter sharding. FSDP2 provides a more modern approach to managing unshard and reshard flows for large-scale models.