pytorch-fsdp

Apply PyTorch FSDP with per-parameter sharding and mixed precision.

Updated Jun 19, 2026

Search Tags:#pytorch #memory-management #distributed-training #fsdp #cpu-offloading #fsdp2 #parameter-sharding

Ready to run

npx skills add https://github.com/AnandaAnugrahHandyanto/savarez_agent --skill pytorch-fsdp-anandaanugrahhandyanto

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/AnandaAnugrahHandyanto/savarez_agent/tree/main/optional-skills/mlops/pytorch-fsdp
Command: npx skills add https://github.com/AnandaAnugrahHandyanto/savarez_agent --skill pytorch-fsdp-anandaanugrahhandyanto

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.0, transformers, and includes references (resource) components.

What problem does it solve?

Training large neural networks often cannot fit in memory; Fully Sharded Data Parallel (FSDP) distributes parameters across data-parallel workers to dramatically reduce per-process memory usage.

Core Features & Use Cases

Parameter sharding across workers to enable training of very large models with limited memory.
Mixed precision training and CPU offloading to optimize compute and memory efficiency.
FSDP2 compatibility and integration with PyTorch distributed workflows for scalable training.
Use case: training a large language model on a multi-node cluster where memory is the bottleneck.

Quick Start

Set up a device mesh and wrap your model with fully_shard to begin memory-efficient distributed training.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I use PyTorch FSDP to train large models that cannot fit in memory?▼

PyTorch FSDP enables memory-efficient distributed training by sharding parameters across workers. It reduces per-process memory usage by distributing model parameters, allowing large neural networks to train across multi-node clusters.

What is the difference between FSDP and FSDP2 for distributed training?▼

FSDP2 is the next generation of Fully Sharded Data Parallel in PyTorch, offering improved per-parameter sharding and integration with distributed workflows. This Skill supports FSDP2 compatibility alongside standard FSDP features.

Can I use CPU offloading and mixed precision with PyTorch FSDP?▼

Yes, PyTorch FSDP supports mixed precision training and CPU offloading to optimize compute and memory efficiency. These features manage memory bottlenecks by offloading parameters to CPU RAM during distributed training.

Do I need torch 2.0 and a device mesh setup to start FSDP training?▼

Yes, FSDP training requires torch>=2.0 and the transformers library. You must set up a device mesh and initialize PyTorch distributed processing before wrapping your model with fully_shard for memory-efficient training.

When should I use Fully Sharded Data Parallel instead of standard data parallel?▼

Use Fully Sharded Data Parallel when training large models on multi-node clusters where memory is the bottleneck. Standard data parallel replicates all parameters per process, while FSDP shards them across workers to reduce memory usage.

How do I apply fully_shard to wrap a model for parameter sharding in PyTorch?▼

To apply parameter sharding, set up a device mesh and wrap your model with the fully_shard function. This begins memory-efficient distributed training by distributing parameters across the configured data-parallel workers.