skills.rest
handsomelong922
handsomelong922COMMUNITY
Explore all skills by this author →
GitHub

pytorch-fsdp2

Integrate PyTorch FSDP2 into training scripts for per-parameter sharding.

Updated Apr 3, 2026
Search Tags:#checkpointing#sharding#pytorch#distributed-training#fsdp2#dtensor#device-mesh
One-click install
npx skills add https://github.com/handsomelong922/my-codex-skills --skill pytorch-fsdp2-handsomelong922
Or copy as Structured Prompt for Agentâ–¼
Please help me install this Agent Skill.
Skill: pytorch-fsdp2
Source: https://github.com/handsomelong922/my-codex-skills/tree/main/skills/pytorch-fsdp2
Command: npx skills add https://github.com/handsomelong922/my-codex-skills --skill pytorch-fsdp2-handsomelong922

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, and includes references (resource) components.

What problem does it solve?

Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision/offload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.

Core Features & Use Cases

  • Bottom-up sharding: explicitly apply fully_shard to submodules before the root module to maximize memory efficiency and performance.
  • DeviceMesh integration: leverage DeviceMesh to manage complex topology for multi-dimensional parallelism.
  • Mixed precision & offload: configure MP and CPU offload to balance speed and memory.
  • Checkpointing support: use DCP or distributed state-dict patterns for robust and scalable saves/loads.
  • Use Case: training very large transformer models that exceed single-GPU memory while enabling deterministic sharded state handling.

Quick Start

Initialize distributed training with torchrun, apply fully_shard bottom-up to submodules, materialize parameters on GPU, and create the optimizer after sharding to enable DTensor-based training.

Frequently Asked Questions about pytorch-fsdp2

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I shard large PyTorch models that exceed single-GPU memory?â–¼

To shard large PyTorch models exceeding single-GPU memory, you can integrate FSDP2 using the fully_shard API to apply per-parameter sharding over multi-GPU setups.

What is the correct order to apply fully_shard and create the optimizer in PyTorch distributed training?â–¼

The correct order for PyTorch distributed training is to apply fully_shard bottom-up to submodules before the root module, materialize parameters on GPU, and create the optimizer after sharding.

Does PyTorch FSDP2 support distributed checkpointing for multi-GPU training?â–¼

Yes, PyTorch FSDP2 supports robust and scalable distributed checkpointing for multi-GPU training by utilizing either DCP or distributed state-dict patterns to save and load sharded states.

Can I configure mixed precision and CPU offload with DTensor-based sharding?â–¼

Yes, you can configure mixed precision and CPU offload with DTensor-based sharding to effectively balance training speed and memory consumption across your distributed setup.

When do I need to use DeviceMesh for multi-dimensional parallelism in PyTorch?â–¼

You need to use DeviceMesh for multi-dimensional parallelism in PyTorch when managing complex multi-GPU topologies, allowing you to leverage DTensor-based sharding for scalable training.

Best way to handle per-parameter sharding for transformer models in PyTorch?â–¼

The best way to handle per-parameter sharding for transformer models in PyTorch is using FSDP2's bottom-up sharding policy, explicitly applying fully_shard to submodules before the root module.

Related Skills

Explore More Software Engineering Skills→
arsityarsity

pytorch-fsdp2

Shard model parameters across devices with PyTorch FSDP2 for large-model training.

Community
Advanced
tadod12tadod12

pytorch-fsdp2

Add PyTorch FSDP2 fully-sharded training with DTensor sharding and DCP checkpointing.

Community
Advanced
gagan114662gagan114662

pytorch-fsdp2

Integrate PyTorch FSDP2 with parameter sharding and distributed checkpointing.

Community
Advanced
hhhi21ghhhi21g

pytorch-fsdp2

Implement PyTorch FSDP2 for distributed training with parameter sharding and checkpointing.

Community
Advanced