pytorch-fsdp2

Integrate PyTorch FSDP2 with DTensor sharding and DCP checkpointing into training scripts.

11.5k|842|Updated Nov 3, 2025

Search Tags:#llm #checkpointing #pytorch #distributed training #gpu memory #fsdp2 #dtensor

Ready to run

npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill pytorch-fsdp2

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp2
Source: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/08-distributed-training/pytorch-fsdp2
Command: npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill pytorch-fsdp2

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables the correct integration of PyTorch FSDP2 (Fully Sharded Data Parallelism version 2) into training scripts, addressing challenges with large models that exceed single-GPU memory and optimizing distributed training performance.

Core Features & Use Cases

Model Parallelism: Distributes model parameters, gradients, and optimizer states across multiple GPUs and nodes.
Memory Optimization: Reduces peak memory usage per GPU, allowing for training of larger models.
DTensor Integration: Leverages PyTorch's DTensor for more flexible and inspectable sharding.
Distributed Checkpointing: Integrates with PyTorch's Distributed Checkpoint (DCP) for robust saving and loading of distributed states.
Use Case: Training a multi-billion parameter language model that cannot fit into the memory of a single GPU.

Quick Start

Integrate PyTorch FSDP2 into your existing training script by following the step-by-step procedure outlined in the skill's documentation.

Frequently Asked Questions about pytorch-fsdp2

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I train large models that exceed single-GPU memory using PyTorch?▼

To train large models exceeding single-GPU memory, you can use PyTorch FSDP2 to distribute model parameters, gradients, and optimizer states across multiple GPUs and nodes, reducing peak memory usage per device.

How does DTensor integration work with PyTorch distributed training?▼

DTensor integration in PyTorch distributed training provides more flexible and inspectable parameter sharding. It works with FSDP2 to manage correct initialization and distribute sharding configurations across a DeviceMesh.

What is the best way to save and load distributed checkpointing states for FSDP2?▼

The best way to save and load distributed checkpointing states for FSDP2 is by integrating PyTorch's Distributed Checkpoint (DCP), which ensures robust saving and loading of distributed optimizer states and sharded parameters.

Can I use FSDP2 to optimize optimizer state distribution for multi-billion parameter LLMs?▼

Yes, you can use FSDP2 to optimize optimizer state distribution for multi-billion parameter LLMs. It manages correct initialization and parameter sharding via DTensor to enable distributed training of models that cannot fit on a single GPU.

Do I need a DeviceMesh to configure advanced sharding with fully_shard in PyTorch?▼

Yes, you need a DeviceMesh to enable advanced sharding configurations with fully_shard in PyTorch. FSDP2 leverages DeviceMesh to optimize distributed training performance and manage parameter sharding across multiple nodes.