pytorch-fsdp

PyTorch FSDP2 reduces memory pressure by disaggregating model parameters across data-parallel workers. It uses a device mesh to enable per-parameter sharding, ensuring each worker only holds a fraction of the parameters during large-model training.

To migrate from FSDP1, apply the fully_shard API to your model components instead of wrapping entire modules. This enables per-parameter sharding and integrates with device meshes for scalable distributed training.

The best way to shard transformer parameters is using PyTorch FSDP2's per-parameter sharding. It performs all-gather before forward and after backward to reconstruct unsharded parameters, then resharding after computation to maximize memory savings.

Yes, you can use mixed precision workflows with per-parameter sharding. PyTorch FSDP2 supports mixed precision alongside device mesh configurations to optimize memory usage and accelerate training for large transformer-like models.

You need fully sharded data parallel when memory constraints limit training large models. FSDP allows you to disaggregate parameters across distributed workers, bypassing single-device memory limits and enabling scalable transformer-like model training.