pytorch-fsdp2

To train large PyTorch models exceeding single-GPU memory, distribute parameters, gradients, and optimizer states across multiple GPUs using fully sharded data parallelism. This partitions memory usage so models requiring over 100GB of VRAM can train efficiently.

DTensor-based sharding is a PyTorch abstraction that enables more inspectable and flexible per-parameter sharding during distributed training. It distributes model components across a DeviceMesh to manage memory and parallelism explicitly.

Configure distributed checkpointing for FSDP2 models by integrating PyTorch's Distributed Checkpoint (DCP) to save and load distributed model states. This ensures robust persistence of sharded parameters across multiple GPUs.

Yes, FSDP2 enables composition with Tensor Parallelism for large model training. By utilizing DTensor and DeviceMesh abstractions, you can shard parameters across data and tensor dimensions simultaneously to maximize GPU memory optimization.

Yes, PyTorch FSDP2 supports mixed precision and CPU offloading configurations. These features allow you to further reduce GPU memory footprint during distributed training by offloading states or reducing parameter precision.