pytorch-fsdp

Name: pytorch-fsdp
Availability: InStock
Author: Rawgrowth-Consulting

Implement distributed PyTorch FSDP training with per-parameter sharding and CPU offloading.

Updated Apr 23, 2026

Search Tags:#sharding #pytorch #distributed-training #fsdp #memory-optimization #cpu-offloading

Ready to run

npx skills add https://github.com/Rawgrowth-Consulting/rawclaw-agent --skill pytorch-fsdp-rawgrowth-consulting

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/Rawgrowth-Consulting/rawclaw-agent/tree/main/optional-skills/mlops/pytorch-fsdp
Command: npx skills add https://github.com/Rawgrowth-Consulting/rawclaw-agent --skill pytorch-fsdp-rawgrowth-consulting

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

PyTorch FSDP enables memory-efficient, scalable training by sharding parameters and overlapping computation across data-parallel workers.

Core Features & Use Cases

Per-parameter sharding with FSDP2 to reduce memory footprint on each device.
CPU offloading and mixed-precision support to maximize throughput for large-scale models.
Guidance for configuring device meshes (1D/2D) and distributed initialization for multi-GPU setups.
Use cases include training transformers and large language models, fine-tuning with limited memory, and research prototyping.

Quick Start

Apply FSDP to a Transformer model and run a forward-backward pass with an all-gather-before-forward setup to ensure correct parameter availability.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I train large language models when GPU memory is insufficient?▼

PyTorch FSDP enables memory-efficient distributed training by sharding parameters across data-parallel workers, reducing the memory footprint on each device so you can train large models with limited memory.

What is the difference between per-parameter sharding and standard distributed data parallelism?▼

Standard distributed data parallelism replicates parameters on every GPU, while per-parameter sharding distributes model parameters across devices, drastically cutting memory usage while overlapping computation to maintain throughput.

Can I use CPU offloading and mixed-precision training with PyTorch FSDP?▼

Yes, PyTorch FSDP supports CPU offloading and mixed-precision training to maximize throughput for large-scale models, allowing you to offload parameters to CPU and optimize memory during distributed training.

How do I configure device meshes for multi-GPU training in PyTorch?▼

You can configure 1D or 2D device meshes for multi-GPU setups in PyTorch FSDP, managing distributed initialization and overlapping computation across data-parallel workers for scalable training.

Do I need PyTorch 2.0 to implement FSDP2 for transformer models?▼

Yes, implementing per-parameter sharding with FSDP2 requires PyTorch version 2.0 or higher and the transformers library, along with a basic understanding of distributed data parallelism.

Why does my PyTorch FSDP forward pass fail with missing parameters?▼

FSDP shards parameters across devices, so you must use an all-gather-before-forward setup to ensure correct parameter availability before running the forward-backward pass on your transformer model.