skills.rest
arsity
arsityCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

pytorch-fsdp2

Shard model parameters across devices with PyTorch FSDP2 for large-model training.

Updated Mar 16, 2026
Search Tags:#checkpointing#pytorch#distributed-training#fsdp2#dtensor#device-mesh
One-click install
npx skills add https://github.com/arsity/scholar-tools --skill pytorch-fsdp2-arsity
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp2
Source: https://github.com/arsity/scholar-tools/tree/main/vendor/ai-research-skills/08-distributed-training/pytorch-fsdp2
Command: npx skills add https://github.com/arsity/scholar-tools --skill pytorch-fsdp2-arsity

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

PyTorch FSDP2 enables training very large models by sharding parameters, gradients, and optimizer state across multiple devices and managing distributed checkpointing, overcoming single-GPU memory limits.

Core Features & Use Cases

  • Explicit bottom-up sharding with fully_shard for transformer-like modules to enable DTensor-based training.
  • DeviceMesh support for 1D/2D parallelism, with 2D hybrid sharding when using Tensor Parallel alongside data parallelism.
  • Mixed precision and optional offload policies, along with distributed checkpointing using DCP or state-dict helpers.
  • Use Case: training a model with hundreds of billions of parameters that would not fit on a single GPU.

Quick Start

Initialize distributed training with torchrun and apply fully_shard bottom-up to the submodules, then run the training loop.

Frequently Asked Questions about pytorch-fsdp2

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a PyTorch model that is too large to fit on a single GPU?▼

PyTorch distributed training scales models beyond single-GPU limits by sharding parameters, gradients, and optimizer state across multiple devices using FSDP2.

How do I shard transformer modules with PyTorch FSDP2?▼

Shard transformer modules by applying the fully_shard API bottom-up to submodules, enabling DTensor-based training across your configured DeviceMesh topology.

Can I use 2D parallelism with Tensor Parallel and data parallelism in FSDP2?▼

Yes, FSDP2 supports 1D and 2D DeviceMesh topologies, allowing 2D hybrid sharding when combining Tensor Parallelism with data parallelism.

Does FSDP2 support mixed precision training and CPU offloading?▼

Yes, FSDP2 supports mixed precision policies and optional CPU offload to manage memory constraints during distributed training.

How do I save distributed checkpoints when training with FSDP2?▼

Distributed checkpointing with FSDP2 uses DCP or state-dict helpers to save sharded model states robustly across multiple devices.

Do I need torchrun to launch distributed training with FSDP2?▼

Yes, distributed training with FSDP2 requires launching with torchrun to initialize the process group and manage DeviceMesh setup across devices.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

pytorch-fsdp

Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.

Community
Advanced
gagan114662gagan114662

pytorch-fsdp2

Integrate PyTorch FSDP2 with parameter sharding and distributed checkpointing.

Community
Advanced
MesferAliMesferAli

pytorch-fsdp2

Implement PyTorch FSDP2 distributed training with DTensor sharding and checkpointing.

Community
Advanced
PlaidmustachePlaidmustache

pytorch-fsdp

Apply PyTorch FSDP for memory-efficient distributed training across GPUs.

Community
Advanced