skills.rest
hung-phan
hung-phanCOMMUNITY
Explore all skills by this author →
GitHub

distributed-data-parallelism

Analyze and recommend DDP, FSDP, PP, and TP strategies for multi-GPU deep learning training.

1|Updated Jun 4, 2026
Search Tags:#deep learning#pipeline parallelism#FSDP#tensor parallelism#DDP#multi-GPU training
One-click install
npx skills add https://github.com/hung-phan/ml-skills --skill distributed-data-parallelism
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-data-parallelism
Source: https://github.com/hung-phan/ml-skills/tree/main/skills/ml-review/references/ml-training/data-parallel
Command: npx skills add https://github.com/hung-phan/ml-skills --skill distributed-data-parallelism

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, torch.distributed, torch.nn, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of scaling deep learning training beyond a single GPU, providing strategies for efficient training across multiple GPUs.

Core Features & Use Cases

  • Distributed Data Parallelism (DDP): Enables parallel training across multiple GPUs by replicating the model and splitting data.
  • Fully Sharded Data Parallelism (FSDP): Suitable for models that don't fit in a single GPU, by sharding parameters, gradients, and optimizer states.
  • Pipeline Parallelism (PP): Sequentially splits model layers across GPUs to handle large models.
  • Tensor Parallelism (TP): Splits individual layers across GPUs for attention heads and other naturally parallelizable layers.
  • Use Case: Ideal for researchers and engineers scaling their models to larger sizes and handling more complex tasks.

Quick Start

Use the distributed-data-parallelism skill to analyze and choose the appropriate distributed training strategy for your deep learning model.

Frequently Asked Questions about distributed-data-parallelism

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose the right multi-GPU training strategy for my deep learning model?▼

This Skill analyzes model size and data volume to recommend distributed training strategies, comparing DDP for replicated models, FSDP for memory constraints, Pipeline Parallelism for sequential layers, and Tensor Parallelism for split layers.

What is the difference between DDP and FSDP in PyTorch?▼

DDP replicates the deep learning model across GPUs and splits data for parallel training, whereas FSDP shards parameters, gradients, and optimizer states to fit models that exceed a single GPU's memory capacity.

When should I use Pipeline Parallelism or Tensor Parallelism for multi-GPU training?▼

Use Pipeline Parallelism to sequentially split deep learning model layers across GPUs, and apply Tensor Parallelism to split individual layers, like attention heads, across GPUs for naturally parallelizable computations.

Does this distributed training analysis require specific PyTorch libraries?▼

Yes, analyzing and implementing distributed training strategies requires PyTorch along with torch.distributed and torch.nn libraries to execute multi-GPU training workflows for large deep learning models.

What is the best way to train deep learning models that do not fit in a single GPU's memory?▼

The best way to train models exceeding single GPU memory is applying Fully Sharded Data Parallelism to shard parameters, gradients, and optimizer states, or using Pipeline Parallelism to distribute sequential layers across multiple GPUs.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

pytorch-fsdp

Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.

Community
Advanced
519lab519lab
2

huggingface-accelerate

Unify distributed training APIs for DeepSpeed, FSDP, Megatron, and DDP.

Official
Intermediate
zeronx798zeronx798

pytorch-fsdp

Implement Fully Sharded Data Parallel training in PyTorch for multi-GPU clusters.

Community
Advanced
ovachieverovachiever
52

ray-train

Orchestrate distributed training across clusters with Ray Train and Ray Tune.

Community
Advanced