pytorch-fsdp

PyTorch Fully Sharded Data Parallel reduces GPU memory by sharding parameters across data-parallel workers. It performs all-gather of parameters before forward and reduce-scatter of gradients after backward, lowering peak memory consumption while preserving model accuracy.

To apply fully_shard for distributed training, you wrap your PyTorch model or specific modules with the FSDP orchestration API. This shards parameters across GPUs, executing all-gather before forward passes and reduce-scatter after backward passes.

Yes, PyTorch FSDP supports mixed precision and CPU offloading to scale large models on constrained hardware. These features allow you to train memory-intensive architectures by reducing the per-device memory footprint.

FSDP2 introduces per-parameter sharding, advancing the original FSDP module-level orchestration. This per-parameter approach aligns with modern PyTorch distributed APIs, offering finer memory control for memory-intensive transformer architectures.

Yes, you need PyTorch version 2.0 or higher to use FSDP for memory-efficient training. The implementation requires PyTorch distributed APIs and supports Linux and macOS environments alongside the transformers library.

Use FSDP instead of standard data parallel when GPU memory limits constrain your batch size or model depth. FSDP shards parameters across workers to reduce peak memory, enabling larger effective batch sizes for transformer-based architectures.