pytorch-fsdp2

What problem does it solve?

This Skill enables the correct integration of PyTorch FSDP2 (Fully Sharded Data Parallelism version 2) into training scripts, addressing challenges with large models that exceed single-GPU memory and optimizing distributed training performance.

Core Features & Use Cases

• Model Parallelism: Distributes model parameters, gradients, and optimizer states across multiple GPUs and nodes.
• Memory Optimization: Reduces peak memory usage per GPU, allowing for training of larger models.
• DTensor Integration: Leverages PyTorch's DTensor for more flexible and inspectable sharding.
• Distributed Checkpointing: Integrates with PyTorch's Distributed Checkpoint (DCP) for robust saving and loading of distributed states.
• Use Case: Training a multi-billion parameter language model that cannot fit into the memory of a single GPU.

Quick Start

Integrate PyTorch FSDP2 into your existing training script by following the step-by-step procedure outlined in the skill's documentation.