pytorch-fsdp

Apply fully_shard to PyTorch models for distributed training with FSDP2.

Updated Apr 23, 2026

Search Tags:#sharding #gpu #pytorch #memory-management #distributed-training #fsdp #cpu-offloading

Ready to run

npx skills add https://github.com/Chris-Chai-Minjae/hermes-agent-r1-bridge --skill pytorch-fsdp-chris-chai-minjae

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/Chris-Chai-Minjae/hermes-agent-r1-bridge/tree/main/optional-skills/mlops/pytorch-fsdp
Command: npx skills add https://github.com/Chris-Chai-Minjae/hermes-agent-r1-bridge --skill pytorch-fsdp-chris-chai-minjae

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Fully Sharded Data Parallel (FSDP) in PyTorch enables memory-efficient, scalable training of large models by sharding parameters and state, reducing peak memory usage and unlocking larger batch sizes across multiple GPUs or nodes.

Core Features & Use Cases

Parameter sharding and per-parameter distribution using FSDP2 to minimize memory usage.
Mixed-precision training and CPU offloading to maximize compute utilization.
Guidance for applying fully_shard to modules and managing state dictionaries across devices and mesh topologies.

Quick Start

Configure a minimal FSDP workflow on a toy model and run one training step across two GPUs.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I use PyTorch FSDP for distributed training across multiple GPUs?▼

PyTorch FSDP enables distributed training by applying the fully_shard function to modules, sharding parameters and state across a device mesh to reduce peak memory usage and unlock larger batch sizes.

What is per-parameter sharding in FSDP2 and how does it minimize memory usage?▼

Per-parameter sharding in FSDP2 distributes individual model parameters across GPUs rather than entire modules. This granular memory management minimizes peak memory usage during scalable training of large models.

Can I use CPU offloading and mixed precision with FSDP to maximize compute utilization?▼

Yes, FSDP supports mixed-precision training and CPU offloading to maximize compute utilization. These features work together to manage memory constraints while maintaining training speed across multi-GPU clusters.

Do I need a specific PyTorch version to apply fully_shard to my models?▼

You need PyTorch version 2.0 or higher to apply fully_shard. The workflow also requires the Transformers library and setup of a device mesh for proper state dict handling and model initialization.

What is the best way to handle state dictionaries when using PyTorch FSDP across multi-node clusters?▼

The best way to handle state dictionaries in FSDP is by managing them across devices and mesh topologies during initialization. Proper state dict handling ensures correct parameter sharding and synchronization across multi-node clusters.

Why does FSDP require setting up a device mesh before applying fully_shard?▼

FSDP requires a device mesh to define the topology for parameter sharding and distributed training. This mesh configuration dictates how parameters and state dictionaries are distributed and synchronized across GPUs or nodes.