skills.rest
HermeticOrmus
HermeticOrmusCOMMUNITY
Explore all skills by this author →
Website𝕏@dbodartmGitHub

Distributed Training Patterns

Implement distributed training patterns with PyTorch DDP, FSDP, and DeepSpeed ZeRO-3.

Updated Feb 25, 2026
Search Tags:#mlops#gpu#pytorch#distributed training#fsdp#ddp#deepspeed
One-click install
npx skills add https://github.com/HermeticOrmus/LibreMLOps-Claude-Code --skill distributed-training-patterns
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Distributed Training Patterns
Source: https://github.com/HermeticOrmus/LibreMLOps-Claude-Code/tree/main/plugins/distributed-training/skills/distributed-training-patterns
Command: npx skills add https://github.com/HermeticOrmus/LibreMLOps-Claude-Code --skill distributed-training-patterns

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of training large machine learning models that exceed the memory or computational capacity of a single GPU, enabling efficient scaling across multiple GPUs and nodes.

Core Features & Use Cases

  • Distributed Data Parallel (DDP): Standard PyTorch DDP setup for multi-GPU training.
  • Fully Sharded Data Parallel (FSDP): Advanced memory optimization for massive models, sharding parameters, gradients, and optimizer states.
  • DeepSpeed ZeRO-3: Configuration for extreme-scale training with advanced memory optimization and communication strategies.
  • Mixed Precision Training: Utilizes AMP and GradScaler for FP16 training to reduce memory usage and speed up computation.
  • Gradient Checkpointing: Trades compute for memory by recomputing activations during the backward pass.
  • Efficient Data Loading: Strategies for optimized data loading across distributed ranks.
  • Use Case: Train a multi-billion parameter LLM by distributing its layers and optimizer states across dozens of GPUs using FSDP or DeepSpeed ZeRO-3, while leveraging mixed precision and gradient checkpointing to fit within hardware constraints.

Quick Start

Use the distributed training skill to set up a PyTorch DDP training loop for a custom model and dataset.

Frequently Asked Questions about Distributed Training Patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a large PyTorch model across multiple GPUs when it exceeds a single GPU's memory capacity?▼

Distributed training scales large PyTorch models across multiple GPUs by sharding parameters, gradients, and optimizer states using FSDP or DeepSpeed ZeRO-3 to fit within hardware memory constraints.

What's the difference between PyTorch DDP and FSDP for distributed training?▼

PyTorch DDP replicates the full model on each GPU for standard multi-GPU training, while FSDP shards parameters, gradients, and optimizer states across GPUs to optimize memory for massive models.

How do I set up a PyTorch DDP training loop for a custom model and dataset?▼

You can set up a PyTorch DDP training loop by configuring distributed process groups, wrapping your custom model with DDP, and implementing optimized data loading strategies across distributed ranks.

Does mixed precision training with AMP and GradScaler reduce memory usage for multi-billion parameter LLMs?▼

Mixed precision training with AMP and GradScaler enables FP16 computation to significantly reduce memory usage and speed up calculations when training multi-billion parameter LLMs across distributed environments.

When should I use gradient checkpointing during distributed training?▼

Gradient checkpointing should be used during distributed training to trade compute for memory by recomputing activations during the backward pass, helping massive models fit within GPU memory constraints.

Can I use DeepSpeed ZeRO-3 for extreme-scale training with advanced memory optimization?▼

DeepSpeed ZeRO-3 configures extreme-scale training with advanced memory optimization and specialized communication strategies to efficiently manage computation across dozens of GPUs.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

pytorch-fsdp

Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.

Community
Advanced
anxiety135790anxiety135790

evaluating-llms-harness

Benchmarks language models using standardized academic and code evaluation suites.

Community
Advanced
icyzhicyzh
1

lambda-labs-gpu-cloud

Provision and manage on-demand GPU cloud instances for ML training.

Community
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

distributed-llm-pretraining-torchtitan

Orchestrate PyTorch-native 4D parallelism for distributed LLM pretraining with torchtitan.

Community
Advanced