pytorch-fsdp

Name: pytorch-fsdp
Availability: InStock
Author: tangzheng202202

Guide PyTorch FSDP parameter sharding, mixed precision, and CPU offloading.

1|Updated Apr 13, 2026

Search Tags:#pytorch #distributed-training #mixed-precision #fsdp #cpu-offloading #parameter-sharding #model-parallelism

Ready to run

npx skills add https://github.com/tangzheng202202/hermes-skills --skill pytorch-fsdp-tangzheng202202

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/tangzheng202202/hermes-skills/tree/main/03-mlops/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/tangzheng202202/hermes-skills --skill pytorch-fsdp-tangzheng202202

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill provides expert guidance for Fully Sharded Data Parallel training in PyTorch, helping teams implement parameter sharding, mixed precision, and CPU offloading to train large models efficiently.

Core Features & Use Cases

FSDP guidance: instructions for setting up FSDP, bucketing strategies, and device placement to optimize memory and throughput.
Debugging & optimization: tips for diagnosing distributed training bottlenecks, selecting backends, and using FSDP2 features.
Use Case: Train massive language models with limited GPU memory by shard-parameter training across nodes.

Quick Start

Ask to configure a PyTorch model with FSDP and run a sample distributed training step.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up PyTorch FSDP for large model training?▼

To set up PyTorch FSDP for large model training, configure parameter sharding, bucketing strategies, and device placement to optimize memory usage and throughput across distributed nodes.

What is parameter sharding in distributed PyTorch workflows?▼

Parameter sharding in distributed PyTorch workflows involves splitting model parameters across nodes to train massive models with limited GPU memory, ensuring memory efficiency through Fully Sharded Data Parallel training.

Can I use mixed precision and CPU offloading with PyTorch FSDP?▼

Yes, you can use mixed precision and CPU offloading with PyTorch FSDP to optimize memory efficiency by offloading parameters to CPU and applying mixed precision during distributed training.

How do I debug distributed training bottlenecks in FSDP2?▼

Debug distributed training bottlenecks in FSDP2 by utilizing specific debugging utilities, diagnosing sharding strategies, and selecting appropriate backends to resolve performance issues within PyTorch distributed workflows.

When do I need Fully Sharded Data Parallel training in PyTorch?▼

You need Fully Sharded Data Parallel training in PyTorch when training massive language models that exceed limited GPU memory, requiring parameter sharding across multiple nodes to maintain memory efficiency.