Verified:GitHub

pytorch-fsdp

Apply Fully Sharded Data Parallel to large PyTorch models for distributed training.

Updated Apr 10, 2026

Search Tags:#pytorch #distributed-training #mixed-precision #fsdp #memory-optimization #cpu-offloading #parameter-sharding

Ready to run

npx skills add https://github.com/overviewlabs/WHOX --skill pytorch-fsdp-overviewlabs

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/overviewlabs/WHOX/tree/main/skills/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/overviewlabs/WHOX --skill pytorch-fsdp-overviewlabs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill provides expert guidance for implementing and debugging PyTorch Fully Sharded Data Parallel (FSDP) workflows, including parameter sharding, mixed precision, CPU offloading, and the newer FSDP2 paradigm.

Core Features & Use Cases

FSDP integration and configuration: Set up FSDP across large models and optimize memory usage.
Parameter sharding and bucket management: Efficiently shard parameters and manage communication buckets.
FSDP2 workflows and debugging tips: Modern FSDP2 features with practical debugging patterns for real-world training tasks.

Quick Start

Instantiate your PyTorch model and wrap it with FSDP fully_shard to begin training.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I use PyTorch FSDP to solve memory constraints in large models?▼

PyTorch FSDP solves memory constraints by applying parameter sharding across multi-GPU and multi-node distributed training. You instantiate your model and wrap it with the fully_shard API to begin training large models efficiently.

What is the difference between FSDP and FSDP2 workflows in PyTorch?▼

FSDP2 is the newer paradigm for distributed training that offers modern features and practical debugging patterns. Both handle parameter sharding and mixed precision, but FSDP2 provides updated workflows for real-world training tasks in PyTorch 2.x.

Can I use CPU offloading and mixed precision with PyTorch FSDP?▼

Yes, PyTorch FSDP supports both CPU offloading and mixed precision to optimize memory usage. These features allow you to manage communication buckets and efficiently shard parameters across large models during distributed training.

Do I need a specific PyTorch version to run FSDP2 workflows?▼

Yes, FSDP2 workflows require PyTorch 2.x (torch>=2.0) and compatible transformers integrations. This version ensures proper support for fully_shard wrapping, parameter sharding, and multi-node distributed training features.

What is the best way to configure parameter sharding and bucket management in FSDP?▼

The best way to configure parameter sharding in FSDP is by instantiating your PyTorch model and wrapping it with fully_shard. This allows efficient parameter sharding and communication bucket management across multi-GPU setups.

Why does my PyTorch FSDP training fail on multi-node setups?▼

Multi-node FSDP training issues often stem from incompatible PyTorch versions or missing transformers integrations. Ensure you are using torch>=2.0 and properly configure parameter sharding and communication buckets for multi-node environments.