skills.rest
pmcdowall
pmcdowallCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

pytorch-fsdp

Optimize PyTorch model training with fully sharded data parallel techniques.

Updated May 9, 2026
Search Tags:#deep learning#pytorch#distributed training#fsdp#mixed precision#model sharding#multi gpu
One-click install
npx skills add https://github.com/pmcdowall/hermes-skills --skill pytorch-fsdp-pmcdowall
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/pmcdowall/hermes-skills/tree/main/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/pmcdowall/hermes-skills --skill pytorch-fsdp-pmcdowall

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps engineers train large-scale deep learning models that exceed single-device memory limits by providing expert guidance on PyTorch Fully Sharded Data Parallel techniques, distributed communication, and memory optimization.

Core Features & Use Cases

  • FSDP Training Guidance: Explains parameter sharding, mixed precision, CPU offloading, optimizer state management, and FSDP1/FSDP2 workflows for large model training.
  • Distributed Systems Support: Covers PyTorch distributed APIs, process groups, communication backends, collective operations, and debugging strategies for multi-GPU and multi-node environments.
  • Use Case: Help an ML engineer migrate a large Transformer training pipeline to FSDP2, configure sharding strategies, optimize GPU memory usage, and troubleshoot distributed training failures.

Quick Start

Use the pytorch-fsdp skill to help optimize my PyTorch Transformer training setup for multi-GPU training with FSDP2 and mixed precision.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train large PyTorch models that exceed single GPU memory limits?▼

You can train large PyTorch models exceeding single-device memory by applying fully sharded data parallel techniques, which shard parameters, optimizer states, and gradients across multi-GPU and multi-node environments for memory-efficient distributed execution.

What is the best way to migrate a Transformer training pipeline to FSDP2?▼

Migrating a Transformer training pipeline to FSDP2 involves configuring new sharding strategies, applying mixed precision, managing optimizer states, and adapting your existing PyTorch distributed APIs to the updated FSDP2 workflows for optimized GPU memory usage.

Does PyTorch FSDP support CPU offloading and mixed precision for multi-node training?▼

Yes, PyTorch FSDP supports multi-node training scenarios utilizing mixed precision and CPU offloading to optimize memory consumption and distributed communication across large-scale deep learning workloads.

How do I configure sharding strategies and process groups for distributed PyTorch training?▼

Configuring sharding strategies requires utilizing PyTorch distributed APIs to establish process groups, select communication backends, and implement collective operations that enable efficient parameter sharding across multi-GPU environments.

Why does my distributed training fail during collective operations and how do I debug it?▼

Distributed training failures during collective operations require debugging strategies specific to PyTorch distributed APIs, focusing on verifying process group initialization, communication backends, and multi-node synchronization to resolve the errors.

Related Skills

Explore More Software Engineering Skills→
InverterNetworkInverterNetwork

pytorch-lightning

Standardize PyTorch training workflows with Lightning for distributed strategies.

Official
Advanced
cxnaivecxnaive

pytorch-fsdp

Implement fully sharded data-parallel training for PyTorch models across multi-GPU clusters.

Community
Advanced
ovachieverovachiever
52

pytorch-fsdp

Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.

Community
Advanced
ldzhhxxldzhhxx
1

pytorch-fsdp

Manage fully sharded data parallelism in PyTorch with FSDP.

Community
Advanced