pytorch-fsdp

PyTorch FSDP is a distributed training technique that shards model parameters, gradients, and optimizer states across data-parallel workers. You need FSDP when training large-scale NLP or vision models that exceed single-GPU memory constraints.

To apply per-parameter sharding in FSDP2, use the fully_shard function on your model and execute a training step. This enables per-parameter sharding, mixed precision, and CPU offloading to maximize memory efficiency.

The best way to migrate from FSDP1 to FSDP2 involves adopting per-parameter sharding and applying fully_shard to your model. FSDP2 offers improved memory management and layer-by-layer grouping strategies for large-scale models.

Yes, PyTorch FSDP2 fully supports CPU offloading and mixed precision training. These features are integral to the implementation, allowing you to maximize memory efficiency during fully sharded data-parallel training.

PyTorch FSDP training runs out of memory when model layouts are poorly grouped or sharding is incomplete. Implementing layer-by-layer grouping, mixed precision, and CPU offloading resolves memory bottlenecks.

Yes, you can use PyTorch FSDP2 across multiple data-parallel workers to achieve memory optimization. It provides per-parameter sharding and tooling references specifically designed to enable scalable distributed training under memory constraints.