pytorch-fsdp

PyTorch FSDP enables memory-efficient distributed training by sharding parameters across workers. It reduces per-process memory usage by distributing model parameters, allowing large neural networks to train across multi-node clusters.

FSDP2 is the next generation of Fully Sharded Data Parallel in PyTorch, offering improved per-parameter sharding and integration with distributed workflows. This Skill supports FSDP2 compatibility alongside standard FSDP features.

Yes, PyTorch FSDP supports mixed precision training and CPU offloading to optimize compute and memory efficiency. These features manage memory bottlenecks by offloading parameters to CPU RAM during distributed training.

Yes, FSDP training requires torch>=2.0 and the transformers library. You must set up a device mesh and initialize PyTorch distributed processing before wrapping your model with fully_shard for memory-efficient training.

Use Fully Sharded Data Parallel when training large models on multi-node clusters where memory is the bottleneck. Standard data parallel replicates all parameters per process, while FSDP shards them across workers to reduce memory usage.

To apply parameter sharding, set up a device mesh and wrap your model with the fully_shard function. This begins memory-efficient distributed training by distributing parameters across the configured data-parallel workers.