pytorch-fsdp

Configure PyTorch FSDP for distributed training by defining parameter sharding strategies, setting up mixed-precision policies, and aligning collective execution across ranks. This guidance covers practical implementation details for FSDP and FSDP2 to reduce memory usage and improve scalability.

PyTorch FSDP training hangs occur when collective calls and join or uneven-input handling are mismatched across ranks. Diagnose these distributed correctness issues by ensuring collective ordering and uneven-input join handling align properly across all participating ranks.

The best way to set up CPU offloading in FSDP2 is to configure offloading parameters alongside per-parameter sharding and mixed-precision settings. This approach targets memory reduction for massive models by offloading parameters during distributed training.

Per-parameter sharding in PyTorch distributed training is needed when scaling massive models to reduce memory usage. It shards individual parameters across ranks, requiring careful alignment of collective calls to prevent synchronization issues.

FSDP2 supports mixed precision training and uneven-input join handling to manage varying input sizes across ranks. Configure these features by following PyTorch distributed documentation patterns to ensure collective ordering and correctness during execution.