pytorch-fsdp

PyTorch FSDP overcomes single-GPU memory limits by sharding model parameters, gradients, and optimizer states across distributed GPUs. This Skill provides guidance for configuring FSDP1 and FSDP2 to scale large models without manual low-level distributed communication management.

FSDP2 is the modern implementation that offers per-parameter sharding and device mesh setup, while FSDP1 is the legacy version. This Skill provides configuration guidance for both versions to help you implement memory-efficient distributed training workflows.

You can set up mixed precision and CPU offloading in PyTorch to maximize training throughput and optimize memory usage. This Skill offers best practices for configuring these performance optimizations within Fully Sharded Data Parallel workflows.

Out-of-memory crashes and gradient synchronization hangs are common distributed training errors. This Skill helps you correctly configure sharding groups and optimize memory usage to prevent these issues when fine-tuning large parameter models across multiple GPUs.

Yes, PyTorch FSDP supports both single-node and multi-node distributed training scenarios. This Skill provides expert guidance for device mesh initialization and debugging, enabling production-ready, memory-efficient training that scales without manual low-level distributed communication management.

The best way to configure a device mesh for a 7B parameter transformer is by using FSDP2 with mixed precision setup. This Skill provides the necessary guidance to initialize device meshes and per-parameter sharding for efficient multi-GPU training.