pytorch-fsdp

PyTorch FSDP enables memory-efficient distributed training by sharding parameters across data-parallel workers, reducing the memory footprint on each device so you can train large models with limited memory.

Standard distributed data parallelism replicates parameters on every GPU, while per-parameter sharding distributes model parameters across devices, drastically cutting memory usage while overlapping computation to maintain throughput.

Yes, PyTorch FSDP supports CPU offloading and mixed-precision training to maximize throughput for large-scale models, allowing you to offload parameters to CPU and optimize memory during distributed training.

You can configure 1D or 2D device meshes for multi-GPU setups in PyTorch FSDP, managing distributed initialization and overlapping computation across data-parallel workers for scalable training.

Yes, implementing per-parameter sharding with FSDP2 requires PyTorch version 2.0 or higher and the transformers library, along with a basic understanding of distributed data parallelism.

FSDP shards parameters across devices, so you must use an all-gather-before-forward setup to ensure correct parameter availability before running the forward-backward pass on your transformer model.