pytorch-fsdp

PyTorch FSDP enables memory-efficient distributed training by sharding model parameters, gradients, and optimizer states across GPUs. You apply it to your model to distribute memory usage, allowing you to train models that exceed a single GPU's memory capacity.

FSDP2 introduces per-parameter sharding and utilizes DTensors, moving away from FSDP1's module-level sharding. This newer approach offers finer memory control and simplifies advanced distributed training patterns compared to FSDP1.

Migrating to FSDP2 involves transitioning to per-parameter sharding using DTensors and device meshes. You must adapt your code to use the `fully_shard` API and adjust pre-forward and post-backward hooks for all-gather and reduce-scatter operations.

Yes, PyTorch 2.x is required to use FSDP2 features like per-parameter sharding and DTensors. A compatible multi-GPU or multi-node mesh setup is also necessary to correctly distribute parameters and execute the training pipeline.

Pre-forward hooks trigger all-gather operations to collect sharded parameters before computation, while post-backward hooks trigger reduce-scatter operations to sync gradients. This ensures parameters are fully materialized in memory only when actively needed.

Optimizing memory and compute balance in multi-GPU environments involves using FSDP2 with device meshes to carefully manage per-parameter sharding. This distributes memory pressure evenly while maintaining compute efficiency during forward and backward passes.