pytorch-fsdp

To configure PyTorch FSDP for multi-node training, you need to set up parameter sharding, mixed precision, and CPU offloading. This skill automatically generates the correct torchrun commands and environment variables for NCCL or Gloo backends to launch your sessions.

Reducing GPU memory usage for large transformer models requires parameter sharding and mixed precision. FSDP shards model parameters across GPUs and optionally offloads them to CPU, keeping memory usage low during distributed training.

Migrating a DDP script to FSDP2 requires updating your training loop to support parameter sharding and mixed precision. This skill provides migration assistance to help move existing distributed data parallel scripts to FSDP2 with minimal code changes.

Yes, PyTorch FSDP supports CPU offloading and mixed precision. This skill automatically selects the appropriate mixed-precision mode and CPU offloading configuration based on your specific model and hardware environment constraints.

The best way to launch distributed training across multiple GPUs is using torchrun with FSDP configuration. It handles parameter sharding and communication backends efficiently, preventing out-of-memory errors in large-scale multi-GPU or multi-node environments.