skills.rest
sayalinvidia
sayalinvidiaCOMMUNITY
Explore all skills by this author →
GitHub

nemo-automodel-distributed-training

Guides NeMo AutoModel distributed training strategy selection and YAML field references for FSDP2, Megatron FSDP, DDP, TP, PP, and CP.

Updated Apr 16, 2026
Search Tags:#pipeline#distributed-training#ddp#moe#fsdp2#nemo-automodel#megatron-fsdp
One-click install
npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-automodel-distributed-training
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-automodel-distributed-training
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/nemo-automodel-distributed-training
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-automodel-distributed-training

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Distributed training for NeMo AutoModel requires careful strategy selection and YAML configuration; this skill provides guidance to choose suitable distributed strategies and understand constraints.

Core Features & Use Cases

  • Strategy selection guidance for multi-GPU/multi-node setups, including FSDP2, Megatron FSDP, and DDP.
  • YAML field references and constraint explanations for TP, PP, CP, EP, and MoE configurations.
  • Practical examples and guardrails to avoid unsupported combinations and misconfigurations.

Quick Start

Ask the agent to propose a distributed-training plan for NeMo AutoModel using fsdp2 across the required GPUs.

Frequently Asked Questions about nemo-automodel-distributed-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure NeMo AutoModel for distributed training across multiple GPUs?▼

Configure NeMo AutoModel distributed training by selecting a strategy like FSDP2, Megatron FSDP, or DDP, and setting the corresponding YAML fields for tensor, pipeline, context, and expert parallelism across your multi-GPU setup.

What is the difference between FSDP2 and Megatron FSDP for NeMo AutoModel?▼

FSDP2 and Megatron FSDP are distributed training strategies for NeMo AutoModel. Megatron FSDP supports advanced parallelism configurations like tensor and pipeline parallelism, whereas FSDP2 provides a different sharding approach for multi-GPU environments.

Can I use tensor, pipeline, and expert parallelism simultaneously in NeMo AutoModel?▼

You can configure tensor, pipeline, context, and expert parallelism together for MoE options in NeMo AutoModel, but specific strategy combinations have constraints and unsupported configurations that require careful YAML field validation.

When should I use DDP instead of FSDP2 for multi-node training?▼

Use DDP for straightforward data-parallel training without model sharding, while FSDP2 is suited for larger models requiring memory optimization across multi-node setups in NeMo AutoModel.

What YAML fields are required to set up MoE options in NeMo AutoModel?▼

Configuring MoE options in NeMo AutoModel requires referencing specific YAML fields to define expert parallelism and related constraints, ensuring the distributed training strategy aligns with your multi-GPU architecture.

Related Skills

Explore More Software Engineering Skills→
sayalinvidiasayalinvidia

nemo-automodel-launcher-config

Configure NeMo AutoModel launches across interactive, Slurm, and SkyPilot environments.

Community
Advanced
Tommy-ywTommy-yw
539

huggingface-accelerate

Simplify distributed training of deep learning models with HuggingFace Accelerate.

Community
Advanced
davpatel605-beepdavpatel605-beep

distributed-llm-pretraining-torchtitan

Configure distributed LLM pretraining with PyTorch parallelism and optimization techniques.

Community
Advanced
Yangel-hideYangel-hide

pytorch-lightning

Automate PyTorch training workflows with distributed strategies like DDP, FSDP, and DeepSpeed.

Community
Advanced