skills.rest
founderphantom
founderphantomCOMMUNITY
Explore all skills by this author →
GitHub

pytorch-fsdp

Configure PyTorch FSDP2 training with per-parameter sharding and mixed precision.

Updated Apr 1, 2026
Search Tags:#sharding#pytorch#distributed-training#mixed-precision#fsdp#fsdp2#cpu-offload
One-click install
npx skills add https://github.com/founderphantom/zola-agent --skill pytorch-fsdp-founderphantom
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/founderphantom/zola-agent/tree/main/skills/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/founderphantom/zola-agent --skill pytorch-fsdp-founderphantom

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Fully Sharded Data Parallel (FSDP) training is complex to implement and optimize at scale. This skill provides expert guidance to configure, tune, and troubleshoot FSDP workflows in PyTorch, helping engineers maximize memory efficiency and throughput.

Core Features & Use Cases

  • Per-parameter sharding with FSDP2 to reduce memory footprint while preserving model accuracy.
  • CPU offloading and mixed precision to enable training on modest hardware and improve throughput.
  • Mesh-based deployment guidance for multi-GPU/multi-node setups and multi-dimensional device layouts.
  • Use cases include training large transformer models, simulating low-memory environments, and debugging distributed setup issues.

Quick Start

Run a guided setup for a PyTorch model using FSDP2 with parameter sharding and mixed precision.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure PyTorch FSDP2 for per-parameter sharding to reduce memory footprint?▼

Configure PyTorch FSDP2 per-parameter sharding by applying sharding to individual parameters rather than layers, reducing memory footprint while preserving model accuracy during distributed training. This skill provides setup guidance for initializing FSDP2 workflows.

What is the best way to set up CPU offloading and mixed precision in PyTorch distributed training?▼

Set up CPU offloading and mixed precision in PyTorch distributed training to enable training on modest hardware and improve throughput. This skill provides guidance to configure these features within Fully Sharded Data Parallel workflows.

How does fully sharded data parallel handle parameter distribution across multiple GPUs and nodes?▼

Fully sharded data parallel handles parameter distribution by sharding model parameters, gradients, and optimizer states across multiple GPUs and nodes. This skill covers mesh-based deployment guidance for multi-dimensional device layouts.

Why is my PyTorch FSDP training experiencing memory inefficiency or throughput bottlenecks?▼

PyTorch FSDP training experiences memory inefficiency or throughput bottlenecks due to suboptimal bucketing, parameter distribution, or mixed precision settings. This skill provides performance troubleshooting guidance to tune and optimize FSDP workflows.

Can I use FSDP to train large transformer models across multi-node setups with low-memory environments?▼

You can use FSDP to train large transformer models across multi-node setups by simulating low-memory environments through CPU offloading and mixed precision. This skill offers initialization and memory management guidance for such deployments.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

pytorch-fsdp

Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.

Community
Advanced
gqf2008gqf2008

pytorch-fsdp

Apply FSDP2 per-parameter sharding to PyTorch models with device meshes.

Community
Advanced
SriRamkunamsettySriRamkunamsetty

slime-rl-training

Orchestrate Megatron-LM and SGLang for end-to-end RL post-training.

Community
Advanced
HermeticOrmusHermeticOrmus

Distributed Training Patterns

Implement distributed training patterns with PyTorch DDP, FSDP, and DeepSpeed ZeRO-3.

Community
Advanced