pytorch-fsdp

Name: pytorch-fsdp
Availability: InStock
Author: enantiomer-h

Implement and optimize PyTorch FSDP training with parameter sharding and mixed precision.

Updated Aug 27, 2026

Search Tags:#sharding #pytorch #distributed-training #mixed-precision #fsdp #cpu-offloading

Ready to run

npx skills add https://github.com/enantiomer-h/DotfilePub --skill pytorch-fsdp-enantiomer-h

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/enantiomer-h/DotfilePub/tree/main/claude-code/.claude/skills/distributed-training-pytorch-fsdp
Command: npx skills add https://github.com/enantiomer-h/DotfilePub --skill pytorch-fsdp-enantiomer-h

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Expert guidance for Fully Sharded Data Parallel training with PyTorch FSDP, including parameter sharding, mixed precision, CPU offloading, and FSDP2, to help teams train large models efficiently.

Core Features & Use Cases

Guidance on configuring FSDP across CPU and GPU backends, including parameter sharding and mixing precision for memory efficiency.
Best Practices for debugging and migrating from FSDP1 to FSDP2, with examples of common patterns in distributed training.
Use Case: Scaling a transformer model on multi-node clusters with memory constraints while maintaining training speed.

Quick Start

Run a PyTorch training script with Fully Sharded Data Parallel (FSDP) 2 across a multi-GPU cluster using the recommended setup.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scale a large transformer model with PyTorch FSDP on multi-node clusters?▼

To scale large transformer models with PyTorch FSDP on multi-node clusters, apply parameter sharding and mixed precision to maintain memory efficiency while preserving training speed. This approach uses official patterns for configuring distributed training workflows.

What is the difference between FSDP1 and FSDP2 for distributed training?▼

The difference between FSDP1 and FSDP2 involves workflow improvements for distributed training. Migrating to FSDP2 requires following specific best practices and common patterns to ensure proper parameter sharding and execution across CPU and GPU backends.

When do I need CPU offloading in PyTorch distributed training?▼

You need CPU offloading in PyTorch distributed training when facing memory constraints on multi-node clusters. It allows you to offload parameter states to the CPU, enabling the training of huge models that would otherwise exceed available GPU memory limits.

Do I need specific PyTorch versions to use fully sharded data parallelism?▼

Yes, you need specific PyTorch versions to use fully sharded data parallelism, requiring torch version 2.0 or higher. You also need the transformers library installed to properly support the distributed training setup and model configurations.

Why is my PyTorch FSDP training failing on multi-GPU setups?▼

PyTorch FSDP training failures on multi-GPU setups often stem from incorrect mixed precision configurations or parameter sharding issues. Debugging requires checking common patterns and ensuring proper FSDP2 workflow adoption across CPU and GPU backends.