pytorch-fsdp

Fully Sharded Data Parallel (FSDP) is a memory-optimization technique that shards parameters across data-parallel workers to solve memory and scalability bottlenecks in large-model training across multi-GPU nodes.

To apply FSDP, use the fully_shard function on your model and run training with a mesh-based device layout to enable per-parameter sharding, overlapping communication with computation across distributed workers.

Yes, FSDP supports CPU offloading and mixed precision to balance compute and memory during distributed training, allowing larger models to fit within hardware constraints across multi-GPU nodes.

Yes, FSDP meets requirements for usage by torch>=2.0 and transformers, managing dependencies via a mesh-based device layout and pre/post hooks for param sharding and gradient reduction.

FSDP2 migration guidance provides future-proofing for large-scale models by updating the sharding approach, ensuring continued memory efficiency and scalability across distributed training workflows.

FSDP shards parameters across devices for memory efficiency, solving memory-constrained model scaling by distributing parameter storage and overlapping communication with computation during gradient reduction.