pytorch-fsdp2

To train large PyTorch models exceeding single-GPU memory, you can use PyTorch FSDP2 to distribute parameters, gradients, and optimizer states across multiple GPUs. This Skill integrates fully_shard to manage sharding and reduce peak memory per device.

The best way to set up distributed checkpointing for FSDP2 is using PyTorch's Distributed Checkpoint (DCP). This Skill configures DCP to robustly save and load distributed model states, ensuring fault tolerance during multi-GPU training.

Bottom-up sharding with fully_shard works by applying the wrapper to a model's submodules before the root module. This approach correctly initializes distributed sharding using DTensor and DeviceMesh for advanced parallelism.

Yes, you can use mixed precision and CPU offloading with PyTorch distributed training. This Skill configures both features to provide further memory reductions and performance optimizations during large model training.

Yes, you need torchrun to launch your PyTorch training scripts for multi-GPU model parallelism. It ensures correct distributed initialization before applying the fully_shard wrapper for bottom-up sharding across your cluster.

PyTorch FSDP2 setup may fail to shard models if fully_shard is not applied bottom-up to submodules before the root, or if DeviceMesh and DTensor are not correctly initialized for distributed training across nodes.