pytorch-fsdp2

To set up PyTorch FSDP2 training, initialize a meta-model, apply fully_shard to submodules bottom-up, call fully_shard on the root, materialize weights to CUDA, and instantiate a DTensor-aware optimizer before training.

Per-parameter DTensor sharding uses DeviceMesh to distribute model parameters across devices. You need it when training large models that exceed a single-GPU memory and require scalable distributed training and checkpointing.

Yes, you can configure mixed precision and optional CPU offload when applying FSDP2 fully_shard to submodules bottom-up, ensuring the optimizer is created only after sharding is complete.

A DTensor-enabled optimizer fails if constructed before sharding. You must apply fully_shard bottom-up to submodules and the root, invoke model(inputs) to run hooks, then instantiate the optimizer.

The best way to checkpoint FSDP2 models is using distributed checkpointing workflows with DCP or distributed state-dict helpers, which properly manage DTensor state across devices.

Yes, PyTorch FSDP2 supports DeviceMesh-based 2D hybrid sharding, allowing you to combine DTensor sharding strategies for scalable distributed training across multiple devices.