skills.rest
afel6
afel6COMMUNITY
Explore all skills by this author →
GitHub

pytorch-fsdp

Guide PyTorch FSDP2 training with parameter sharding and mixed precision.

Updated Mar 25, 2026
Search Tags:#sharding#pytorch#model-optimization#distributed-training#fsdp#cpu-offloading#fsdp2
One-click install
npx skills add https://github.com/afel6/scal-ai-pipeline --skill pytorch-fsdp-afel6
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/afel6/scal-ai-pipeline/tree/main/hermes_skills_library/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/afel6/scal-ai-pipeline --skill pytorch-fsdp-afel6

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

PyTorch Fully Sharded Data Parallel (FSDP) training often struggles with memory bottlenecks and configuration complexity when scaling to large models. This skill provides expert guidance for implementing, debugging, and optimizing FSDP-based training with FSDP2, including parameter sharding, mixed precision, and CPU offloading.

Core Features & Use Cases

  • Guidance for configuring FSDP2, bucketing, and mesh layouts for large models
  • Debugging distributed training issues such as deadlocks, gradient all-reduces, and compatibility with mixed precision
  • Use Case: Training transformer-based architectures across multi-GPU/multi-node clusters with optional CPU offloading for memory efficiency

Quick Start

Provide a minimal PyTorch FSDP2 setup example for training a transformer model with parameter sharding and CPU offloading

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up PyTorch FSDP2 for training a transformer model?▼

To set up PyTorch FSDP2 for transformer training, configure parameter sharding, mesh layouts, and optional CPU offloading across a mesh-based device layout. This skill provides minimal setup examples for multi-GPU and multi-node execution.

What is the best way to fix PyTorch FSDP deadlocks during distributed training?▼

The best way to fix PyTorch FSDP deadlocks is to debug gradient all-reduce operations and mixed precision compatibility. This skill provides guidance for resolving distributed training issues like deadlocks and gradient synchronization failures.

Does FSDP2 support CPU offloading for memory-intensive networks?▼

Yes, FSDP2 supports CPU offloading for memory-intensive networks. This skill guides configuring optional CPU offloading to achieve memory efficiency when training large transformer architectures across multi-GPU clusters.

Do I need a specific PyTorch version to use FSDP2 with mixed precision?▼

Yes, you need PyTorch version 2.0 or higher and the transformers library to use FSDP2 with mixed precision. The implementation assumes a mesh-based device layout for distributed execution.

Why does my PyTorch FSDP training run out of memory on multi-node clusters?▼

PyTorch FSDP training runs out of memory on multi-node clusters due to memory bottlenecks when scaling large models. Configuring parameter sharding, bucketing, and CPU offloading helps optimize memory efficiency.

Related Skills

Explore More Software Engineering Skills→
Chris-Chai-MinjaeChris-Chai-Minjae

pytorch-fsdp

Apply fully_shard to PyTorch models for distributed training with FSDP2.

Community
Advanced
jacardljacardl

pytorch-fsdp

Apply Fully Sharded Data Parallel to shard parameters across multi-GPU workers.

Community
Advanced
tangzheng202202tangzheng202202
1

pytorch-fsdp

Guide PyTorch FSDP parameter sharding, mixed precision, and CPU offloading.

Community
Advanced
xiaoquqixiaoquqi

pytorch-fsdp

Apply PyTorch FSDP memory sharding across data-parallel workers for distributed training.

Community
Advanced