pytorch-fsdp

PyTorch distributed training shards parameters, gradients, and optimizer state across workers using FSDP to fit larger models on available hardware. This reduces memory pressure by keeping only shards in GPU memory during transformer-scale workloads.

Migrating to FSDP2 requires applying modern distributed workflows including device meshes, DTensor-aware optimizer handling, and hook-driven collectives. You must configure bottom-up module sharding to replace older FSDP patterns for production training.

PyTorch FSDP supports mixed precision and CPU offloading to optimize memory-efficient distributed learning. These features allow you to offload parameters to CPU and use lower precision computations to fit larger models on limited GPU memory.

Configuring device-mesh setup for bottom-up module sharding requires correct initialization of the distributed process group and mesh topology in PyTorch. This setup coordinates hook-based all-gather and reduce-scatter operations across workers for FSDP2 migration scenarios.

PyTorch FSDP training fails during sharding and synchronization when device-mesh setup is incorrect or modules are not sharded bottom-up. Debugging requires verifying hook-based all-gather and reduce-scatter coordination to ensure proper distributed communication.

You should not use FSDP for distributed training when models are small enough to fit entirely on a single GPU without memory constraints. FSDP's overhead from hook-based all-gather and reduce-scatter coordination is unnecessary for transformer workloads that do not require parameter sharding.