pytorch-fsdp

PyTorch FSDP enables distributed training by applying the fully_shard function to modules, sharding parameters and state across a device mesh to reduce peak memory usage and unlock larger batch sizes.

Per-parameter sharding in FSDP2 distributes individual model parameters across GPUs rather than entire modules. This granular memory management minimizes peak memory usage during scalable training of large models.

Yes, FSDP supports mixed-precision training and CPU offloading to maximize compute utilization. These features work together to manage memory constraints while maintaining training speed across multi-GPU clusters.

You need PyTorch version 2.0 or higher to apply fully_shard. The workflow also requires the Transformers library and setup of a device mesh for proper state dict handling and model initialization.

The best way to handle state dictionaries in FSDP is by managing them across devices and mesh topologies during initialization. Proper state dict handling ensures correct parameter sharding and synchronization across multi-node clusters.

FSDP requires a device mesh to define the topology for parameter sharding and distributed training. This mesh configuration dictates how parameters and state dictionaries are distributed and synchronized across GPUs or nodes.