pytorch-fsdp2

To shard a large PyTorch model that exceeds single-GPU memory, apply FSDP2 cross-device sharding to submodules. Use DTensor-based per-parameter sharding with DeviceMesh, enforcing bottom-up sharding before the root to form efficient communication groups.

The correct order for applying FSDP2 sharding is bottom-up. You must shard submodules before the root module to form efficient communication groups, then create the optimizer post-sharding to ensure distributed training works correctly.

Yes, PyTorch FSDP2 supports distributed checkpointing across workers. It provides flexible checkpointing via DCP and state-dict helpers to save and load distributed state across multiple devices during scalable training.

Yes, you can use DTensor and DeviceMesh for per-parameter sharding in PyTorch. FSDP2 integrates DTensor-based per-parameter sharding with DeviceMesh to provide scalable sharding for transformer-style models.

You need to create the optimizer after applying FSDP2 sharding. The skill enforces post-sharding optimizer creation, ensuring the optimizer operates on the sharded parameters managed by DTensor and DeviceMesh.

A limitation of using FSDP2 is that it requires PyTorch and careful bottom-up sharding of submodules. Users must manage mixed-precision and offload settings, and coordinate distributed checkpointing using DCP and state-dict workflows.