skills.rest
AlvaroBiano
AlvaroBianoCOMMUNITY
Explore all skills by this author →
GitHub

pytorch-fsdp

Implement PyTorch FSDP sharding, mixed precision, and CPU offloading for distributed training.

Updated May 2, 2026
Search Tags:#sharding#pytorch#distributed-training#fsdp#memory-optimization#cpu-offloading#fsdp2
One-click install
npx skills add https://github.com/AlvaroBiano/hermes-agent --skill pytorch-fsdp-alvarobiano
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/AlvaroBiano/hermes-agent/tree/main/optional-skills/mlops/pytorch-fsdp
Command: npx skills add https://github.com/AlvaroBiano/hermes-agent --skill pytorch-fsdp-alvarobiano

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

PyTorch Fully Sharded Data Parallel (FSDP) enables memory-efficient distributed training by sharding parameters and coordinating communication, allowing training of very large models on limited hardware.

Core Features & Use Cases

  • Fully Sharded Data Parallel (FSDP) support including FSDP2, enabling memory savings for large-scale models
  • Parameter sharding, mixed precision, and CPU offloading to balance compute and memory
  • Use cases: training transformer-based models and other large architectures that exceed single-device memory

Quick Start

Configure your PyTorch model for FSDP2, enable parameter sharding, and begin distributed training.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement PyTorch FSDP for memory-efficient distributed training?▼

Implement PyTorch FSDP by configuring parameter sharding strategies, enabling mixed precision, and applying CPU offloading to coordinate communication and balance compute with memory for large-scale models.

What is parameter sharding and when do I need it for large model training?▼

Parameter sharding distributes model parameters across devices to enable memory-efficient distributed training. You need it when training large transformer-based architectures that exceed single-device memory limits.

Does PyTorch FSDP2 support CPU offloading and mixed precision on macOS?▼

Yes, PyTorch FSDP2 supports CPU offloading and mixed precision configurations across Linux and macOS environments, requiring torch>=2.0 and transformers dependencies to balance compute and memory.

What is the best way to configure sharding strategies for transformer models in PyTorch?▼

The best way to configure sharding strategies for transformer models is using PyTorch FSDP, which enables fully sharded data parallel training with parameter sharding and mixed precision to optimize memory usage.

Why does distributed training fail with out-of-memory errors on limited hardware?▼

Distributed training fails with out-of-memory errors when large models exceed single-device memory limits. PyTorch FSDP solves this by sharding parameters and coordinating communication to enable memory-efficient training on limited hardware.

Related Skills

Explore More Software Engineering Skills→
AnandaAnugrahHandyantoAnandaAnugrahHandyanto

pytorch-fsdp

Apply PyTorch FSDP with per-parameter sharding and mixed precision.

Community
Advanced
Clay-HHKClay-HHK
2

pytorch-fsdp2

Set up PyTorch FSDP2 training with per-parameter DTensor sharding across devices.

Community
Intermediate
tangzheng202202tangzheng202202
1

pytorch-fsdp

Guide PyTorch FSDP parameter sharding, mixed precision, and CPU offloading.

Community
Advanced
pmcdowallpmcdowall

pytorch-fsdp

Optimize PyTorch model training with fully sharded data parallel techniques.

Community
Advanced