pytorch-fsdp2

To shard a PyTorch model across multiple GPUs and prevent OOM errors, apply bottom-up fully_shard configuration to split parameters, gradients, and optimizer state. This FSDP2 retrofit enables memory-efficient training for workloads exceeding single-device memory limits.

The best way to set up distributed checkpointing for FSDP2 training is using DCP-compatible state dict handling. This pattern ensures safe saving and loading of distributed state, supporting resharding and multi-rank recovery without corrupting the sharded model layout.

Yes, you can configure mixed precision and CPU offload with PyTorch FSDP2. These settings are integrated into the distributed workflow setup alongside torchrun-based execution and device mesh configuration to optimize memory efficiency during training.

Yes, a DTensor-aware optimizer is required for FSDP2 sharding. Constructing the optimizer with DTensor awareness ensures that the optimizer state is correctly partitioned across the device mesh during distributed training.

To initialize torchrun for a PyTorch FSDP2 training script, set up the distributed execution environment before applying bottom-up fully_shard sharding and configuring the device mesh. This establishes the multi-rank process group required for distributed training.

Your transformer training script OOMs on a single GPU because the model size exceeds available device memory. Retrofitting the script with FSDP2 shards parameters, gradients, and optimizer state across multiple GPUs to resolve this memory constraint.