pytorch-fsdp2

FSDP2 uses DTensor for inspectable, per-parameter sharding composable with DeviceMesh. This approach handles optimizer setup with DTensor parameters and applies mixed precision, overcoming single-GPU memory limits for large language model training.

FSDP2 implements distributed checkpointing using DCP to save and load sharded model states across multiple GPUs. This ensures correct state recovery for DTensor parameters during large model training.

DTensor-based sharding is needed when training large models exceeding single-GPU VRAM. FSDP2 leverages DTensor with DeviceMesh to shard parameters, gradients, and optimizer states across multiple GPUs or nodes efficiently.

Yes, PyTorch FSDP2 supports mixed precision training. The skill integrates mixed precision configuration into the fully_shard setup to optimize memory usage and accelerate large model training across multiple GPUs.

Yes, FSDP2 is composable with DeviceMesh. It leverages DTensor for per-parameter sharding across the device mesh, enabling inspectable and flexible distribution of model parameters over multiple GPUs.

FSDP2 requires models to be initialized correctly with bottom-up sharding and DTensor parameters. It is applicable specifically for models exceeding single-GPU memory and requires a multi-GPU environment with DeviceMesh.