pytorch-fsdp2

To train a PyTorch model exceeding single-GPU memory capacity, shard parameters, gradients, and optimizer states across multiple GPUs using Fully Sharded Data Parallel (FSDP2). This reduces peak memory per GPU by distributing the workload across a cluster.

FSDP2 parameter sharding uses the fully_shard API to distribute model parameters as DTensor objects across a DeviceMesh. This enables multi-dimensional parallelism, allowing efficient memory management and distributed training across GPUs.

Yes, you can configure mixed precision and CPU offloading with PyTorch FSDP2. These features provide further memory savings by offloading parameters to CPU and reducing the memory footprint of numerical computations during large-scale training.

Save and load distributed checkpoints in PyTorch FSDP2 using PyTorch Distributed Checkpoint (DCP). It integrates directly with the training script to robustly manage saving and loading model states across multiple GPUs.

Configure the optimizer directly on the DTensor parameters after applying fully_shard. This ensures the optimizer states are correctly sharded across the DeviceMesh, maintaining memory efficiency during large model training.

You need DeviceMesh for multi-dimensional parallelism when training massive models across a cluster of GPUs. It works with FSDP2 to coordinate parameter sharding and distributed training across multiple parallelism dimensions simultaneously.