pytorch-fsdp

FSDP2 reduces memory pressure by distributing parameters and state across devices. It uses bottom-up parameter sharding to group model layers, enabling scalable memory savings for transformer-scale training on large clusters.

Set up distributed training with FSDP2 and DTensor by defining a DeviceMesh and applying the fully_shard frontend. This enables per-parameter DTensor sharding, pre-forward all-gather, and post-backward reduce-scatter for synchronized gradients across ranks.

FSDP2 distinguishes itself through per-parameter DTensor sharding and bottom-up grouping across model layers. This approach provides scalable memory savings by applying pre-forward all-gather and post-backward reduce-scatter for synchronized gradients.

Use FSDP2 for transformer-scale training when large models exceed single-GPU memory limits. It distributes parameters and state across devices by applying bottom-up parameter sharding to reduce memory pressure and enable scalable training.

FSDP2 uses a DTensor-based setup with the fully_shard frontend to shard parameters. You define a DeviceMesh, apply fully_shard to your model layers bottom-up, and run your training script to enable sharding across the cluster.