Train mixture-of-experts models with sparse routing and expert parallelism.
npx skills add https://github.com/ricable/mcai --skill moe-training-ricable
Please help me install this Agent Skill. Skill: moe-training Source: https://github.com/ricable/mcai/tree/main/.agents/skills/moe-training Command: npx skills add https://github.com/ricable/mcai --skill moe-training-ricable
MoE Training helps you build and optimize sparse expert-based models when dense training is too expensive, enabling far more model capacity without proportional compute growth.
Use the moe-training skill to design a sparse MoE training plan for your model, including expert count, routing strategy, capacity settings, and distributed training configuration.
High-intent search queries and answers about installing and using this skill.
Train mixture of experts models with expert parallelism by configuring DeepSpeed to handle sparse routing and distributed expert execution across available GPUs. This enables capacity scaling without proportional compute growth.
Configure load balancing for sparse routing in Mixtral by tuning auxiliary loss, router z-loss, and capacity factors. These settings keep expert utilization efficient and stabilize training dynamics.
Scaling model capacity without proportional compute growth requires sparse MoE training. By routing tokens to selected experts, you achieve far greater capacity than dense models at lower compute costs.
Yes, you can configure top-1, top-2, or expert-choice routing patterns for Switch Transformers using PyTorch. This Skill guides selecting the right routing strategy for your architecture.
Yes, DeepSpeed supports distributed expert execution for large-scale Mixtral variants. You can use this Skill to plan training runs, prune experts, and choose the right distributed configuration for your GPUs.
MoE training runs may fail during sparse inference if capacity factors, routing mechanisms, or auxiliary losses are not properly configured. Ensure Accelerate and Transformers support your specific expert parallelism setup.
MesferAliTrain Mixture of Experts models using DeepSpeed and HuggingFace Transformers.
tianhao909Train Mixture of Experts models using DeepSpeed and HuggingFace Transformers.
Supporter09Optimize distributed LLM training with Megatron-Core parallelism strategies.
tadod12Optimize Mixture of Experts training with DeepSpeed or HuggingFace.
