skills.rest
yo-steven
yo-stevenCOMMUNITY
Explore all skills by this author →
GitHub

perf-moe-vlm-training

Guide MoE VLM training strategy selection for FSDP and 3D-parallel configurations.

Updated May 23, 2026
Search Tags:#training optimization#fsdp#recompute#moe vlm#3d parallelism#cuda graphs#multimodal performance
One-click install
npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-moe-vlm-training-yo-steven
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: perf-moe-vlm-training
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/Megatron-Bridge/perf-moe-vlm-training
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-moe-vlm-training-yo-steven

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It helps you choose practical training strategies for Mixture-of-Experts (MoE) Vision-Language Models (VLMs) when you hit throughput limits, OOMs, or unstable training behavior.

Core Features & Use Cases

  • FSDP vs 3D parallel decision guidance: Choose the approach that matches your bring-up constraints and your desired steady-state throughput ceiling.
  • Empirical tuning heuristics for MoE VLMs: Apply rounded findings for recompute strategy, MBS sensitivity, CUDA-graph scope, and when to freeze the vision stack.
  • Pitfall-aware performance interpretation: Avoid misleading conclusions from mock (image-free) runs and ensure fair comparisons by normalizing effective work.

Quick Start

Use the perf-moe-vlm-training Skill to decide whether to start with FSDP for bring-up and then move to 3D parallel once your PP layout and real-data vision pipeline are stable.

Frequently Asked Questions about perf-moe-vlm-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize MoE VLM training throughput and resolve OOM errors?▼

Optimizing MoE VLM training requires applying empirical tuning heuristics for recompute strategy, MBS sensitivity, and CUDA-graph scope to resolve throughput limits and OOMs.

What is the best way to choose between FSDP and 3D parallelism for multimodal training?▼

Choosing between FSDP and 3D parallelism depends on your bring-up constraints and desired steady-state throughput ceiling; FSDP is recommended for initial bring-up, followed by 3D parallelism for stable production.

Why does my MoE VLM training performance drop when using mock image-free runs?▼

MoE VLM training performance drops in mock image-free runs because they yield misleading conclusions; you must use real vision inputs and normalize effective work to ensure fair performance comparisons.

Can I freeze the vision stack during MoE VLM training to save memory?▼

Yes, you can freeze the vision stack during MoE VLM training, and this Skill provides compatibility decision guidance to help you determine the exact conditions for safely applying this optimization.

How do I configure recompute settings and CUDA-graph scope for multimodal performance?▼

Configuring recompute settings and CUDA-graph scope for multimodal performance involves applying rounded empirical findings tailored to MoE VLM workloads to address memory constraints and stabilize training behavior.

When should I transition my VLM training from FSDP to 3D parallelism?▼

You should transition VLM training from FSDP to 3D parallelism once your pipeline parallel layout and real-data vision pipeline are stable, moving from initial bring-up toward achieving higher steady-state throughput.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

moe-vlm-training

Provides MoE-VLM training strategy selection and tuning for Megatron Bridge, including FSDP vs 3D-parallel decisions.

Official
Advanced
yo-stevenyo-steven

perf-moe-hardware-configs

Select MoE training parallelism and tuning configurations for H100, B200, GB200, and GB300.

Community
Intermediate
sayalinvidiasayalinvidia

nemo-mbridge-perf-moe-vlm-training

Compare FSDP-first and 3D-parallel paths for MoE VLM training in Megatron Bridge.

Community
Advanced
yo-stevenyo-steven

perf-moe-long-context

Configure MoE long-context training with CP sizing and selective recompute.

Community
Intermediate