pytorch-fsdp2

You enable PyTorch FSDP2 fully-sharded training by applying the `fully_shard` function bottom-up to submodules, using DTensor per-parameter sharding and optional DeviceMesh integration to distribute model memory across multiple GPUs.

DTensor-based sharding is a per-parameter sharding mechanism used in PyTorch FSDP2 to enable memory-efficient, scalable multi-GPU training for large transformer models that exceed single-GPU capacity.

You configure mixed-precision and CPU offload during the FSDP2 sharding application process, allowing DTensor-based optimizer states to manage memory consumption efficiently across multi-node training environments.

Yes, robust distributed checkpointing via DCP or distributed state-dict helpers supports topology changes, ensuring fault-tolerant state management across multi-GPU and multi-node training configurations.

Bottom-up sharding with `fully_shard` is preferred when you need explicit per-submodule application and DTensor-based state management to train large transformer models that do not fit on a single GPU.

You must construct the optimizer after applying FSDP2 sharding to ensure the optimizer state correctly reflects the DTensor-based sharded parameters and distributed training layout.