pytorch-fsdp

PyTorch FSDP shards model parameters, gradients, and optimizer states across distributed workers to reduce memory pressure. You apply bottom-up sharding to your layers and coordinate all-gather and reduce-scatter operations to enable large-scale training.

FSDP2 migration provides clearer parameter semantics and DTensor-aware parameter handling compared to FSDP1. Moving to FSDP2 improves how model states are managed and sharded across device meshes during distributed training.

Mixed precision and CPU offloading are fully supported in PyTorch FSDP workflows. Combining these techniques with prefetching and sharded state handling improves training throughput and reduces peak memory use.

Setting up a device mesh for PyTorch FSDP requires correct device-mesh placement to coordinate distributed workers. You configure the mesh based on your available CPU, CUDA, or XPU systems to ensure proper sharding and communication.

Distributed training failures often stem from incorrect device-mesh placement or misconfigured all-gather and reduce-scatter coordination. Diagnose hook behavior and verify safe integration with torch.distributed APIs to stabilize training.

PyTorch FSDP is necessary when your large-scale model no longer fits on a single GPU. By sharding parameters, gradients, and optimizer states across workers, FSDP makes distributed training practical and reduces memory pressure.