skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

moe-training

Train Mixture-of-Experts models with DeepSpeed and HuggingFace configurations.

2|Updated Apr 12, 2026
Search Tags:#routing#training#load-balancing#deepspeed#moe#mixture-of-experts#expert-parallelism
One-click install
npx skills add https://github.com/Clay-HHK/claude-config --skill moe-training-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: moe-training
Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/AI-research-SKILLs/19-emerging-techniques/moe-training
Command: npx skills add https://github.com/Clay-HHK/claude-config --skill moe-training-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

MoE Training enables scaling model capacity and specialization while reducing compute and training cost compared to dense models by activating only a subset of experts per token.

Core Features & Use Cases

  • Efficient Large-Scale Training: DeepSpeed and HuggingFace-ready patterns and configuration examples for training sparse Mixture-of-Experts models.
  • Routing & Load Balancing: Guidance on top-k routing (top-1, top-2), auxiliary losses, router z-loss, and capacity factor tuning to avoid token dropping and imbalance.
  • Expert Parallelism & Inference: Expert sharding, expert-parallel sizing, inference optimizations (quantization, fused kernels, expert pruning, speculative decoding).
  • Use Case: Reproduce Mixtral 8x7B-style top-2 MoE training or scale toward DeepSeek-V3 by configuring expert counts, capacity factors, and DeepSpeed expert parallelism for production training runs.

Quick Start

Use the moe-training skill to generate a DeepSpeed configuration and a training command for a Mixtral-style top-2, 8-expert model with 32 layers, hidden size 4096, global batch size 256, and fp16.

Frequently Asked Questions about moe-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a Mixture-of-Experts model using DeepSpeed?▼

To train a Mixture-of-Experts model using DeepSpeed, configure expert parallelism, top-k routing, and capacity factors. This skill provides DeepSpeed-compatible configuration examples to scale model capacity efficiently for architectures like Mixtral 8x7B and DeepSeek-V3.

How does routing and load balancing work in MoE training?▼

MoE training balances load using top-k routing (top-1 or top-2), auxiliary losses, and router z-loss. Proper capacity factor tuning is necessary to avoid token dropping and expert imbalance during large-scale language model training runs.

Can I use HuggingFace to train a Mixtral 8x7B style model?▼

Yes, you can use HuggingFace to train Mixtral 8x7B style models. This skill provides HuggingFace-compatible configuration examples for setting expert counts, routing strategies, and global batch sizes for sparse model training workflows.

What inference optimizations are available for Mixture-of-Experts models?▼

Inference optimizations for Mixture-of-Experts models include quantization, fused kernels, expert pruning, and speculative decoding. These techniques reduce compute cost per token after training large-scale sparse architectures.

How do I configure expert parallelism to prevent token dropping?▼

To prevent token dropping during expert parallelism, tune the capacity factor and apply router z-loss alongside auxiliary losses. This ensures balanced expert utilization across shards during DeepSpeed training runs.

Related Skills

Explore More Education & Research Skills→
519lab519lab
2

huggingface-accelerate

Unify distributed training APIs for DeepSpeed, FSDP, Megatron, and DDP.

Official
Intermediate
MesferAliMesferAli

speculative-decoding

Accelerate LLM inference with speculative decoding and draft models.

Community
Advanced
zeronx798zeronx798

pytorch-fsdp

Implement Fully Sharded Data Parallel training in PyTorch for multi-GPU clusters.

Community
Advanced
informatico-madridinformatico-madrid
1

moe-training

Train Mixture of Experts models using DeepSpeed and HuggingFace Transformers.

Community
Advanced