skills.rest
tangzheng202202
tangzheng202202COMMUNITY
Explore all skills by this author →
GitHub

pytorch-fsdp

Guide PyTorch FSDP parameter sharding, mixed precision, and CPU offloading.

1|Updated Apr 13, 2026
Search Tags:#pytorch#distributed-training#mixed-precision#fsdp#cpu-offloading#parameter-sharding#model-parallelism
One-click install
npx skills add https://github.com/tangzheng202202/hermes-skills --skill pytorch-fsdp-tangzheng202202
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/tangzheng202202/hermes-skills/tree/main/03-mlops/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/tangzheng202202/hermes-skills --skill pytorch-fsdp-tangzheng202202

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill provides expert guidance for Fully Sharded Data Parallel training in PyTorch, helping teams implement parameter sharding, mixed precision, and CPU offloading to train large models efficiently.

Core Features & Use Cases

  • FSDP guidance: instructions for setting up FSDP, bucketing strategies, and device placement to optimize memory and throughput.
  • Debugging & optimization: tips for diagnosing distributed training bottlenecks, selecting backends, and using FSDP2 features.
  • Use Case: Train massive language models with limited GPU memory by shard-parameter training across nodes.

Quick Start

Ask to configure a PyTorch model with FSDP and run a sample distributed training step.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up PyTorch FSDP for large model training?▼

To set up PyTorch FSDP for large model training, configure parameter sharding, bucketing strategies, and device placement to optimize memory usage and throughput across distributed nodes.

What is parameter sharding in distributed PyTorch workflows?▼

Parameter sharding in distributed PyTorch workflows involves splitting model parameters across nodes to train massive models with limited GPU memory, ensuring memory efficiency through Fully Sharded Data Parallel training.

Can I use mixed precision and CPU offloading with PyTorch FSDP?▼

Yes, you can use mixed precision and CPU offloading with PyTorch FSDP to optimize memory efficiency by offloading parameters to CPU and applying mixed precision during distributed training.

How do I debug distributed training bottlenecks in FSDP2?▼

Debug distributed training bottlenecks in FSDP2 by utilizing specific debugging utilities, diagnosing sharding strategies, and selecting appropriate backends to resolve performance issues within PyTorch distributed workflows.

When do I need Fully Sharded Data Parallel training in PyTorch?▼

You need Fully Sharded Data Parallel training in PyTorch when training massive language models that exceed limited GPU memory, requiring parameter sharding across multiple nodes to maintain memory efficiency.

Related Skills

Explore More Data & Analytics Skills→
pmcdowallpmcdowall

pytorch-fsdp

Optimize PyTorch model training with fully sharded data parallel techniques.

Community
Advanced
519lab519lab
2

huggingface-accelerate

Unify distributed training APIs for DeepSpeed, FSDP, Megatron, and DDP.

Official
Intermediate
xiaoquqixiaoquqi

pytorch-fsdp

Apply PyTorch FSDP memory sharding across data-parallel workers for distributed training.

Community
Advanced
ragnarokhaaragnarokhaa
4

pytorch-fsdp

Guide PyTorch Fully Sharded Data Parallel training with sharding and optimization strategies.

Community
Advanced