Train large-scale Mixture of Experts models with DeepSpeed or HuggingFace.
npx skills add https://github.com/hhhi21g/HealthCenter --skill moe-training-hhhi21g
Please help me install this Agent Skill. Skill: moe-training Source: https://github.com/hhhi21g/HealthCenter/tree/main/.codex/skills/moe-training Command: npx skills add https://github.com/hhhi21g/HealthCenter --skill moe-training-hhhi21g
This Skill allows you to train Mixture of Experts (MoE) models efficiently with limited compute resources, enabling you to scale model capacity without proportional increases in computation.
Use the moe-training skill to train a MoE model with the following configuration:
python train_moe.py \
--num-layers 24 \
--hidden-size 1024 \
--num-attention-heads 16 \
--seq-length 2048 \
--max-position-embeddings 2048 \
--micro-batch-size 4 \
--global-batch-size 256 \
--train-iters 500000 \
--lr 0.0001 \
--min-lr 0.00001 \
--lr-decay-style cosine \
--num-experts 128 \
--moe-expert-parallel-size 4 \
--moe-loss-coeff 0.01 \
--moe-train-capacity-factor 1.25 \
--moe-eval-capacity-factor 2.0 \
--fp16
High-intent search queries and answers about installing and using this skill.
Train Mixture of Experts models with limited compute by using DeepSpeed and HuggingFace transformers to scale model capacity without proportional increases in computation. This skill implements sparse architectures and expert parallelism to optimize training efficiency.
Implement sparse architectures like Mixtral 8x7B or DeepSeek-V3 by configuring MoE routing mechanisms, load balancing, and expert parallelism using the integrated deepspeed and transformers libraries for efficient large-scale model training.
DeepSpeed works with HuggingFace transformers for MoE training, providing the necessary infrastructure for expert parallelism, load balancing, and capacity factor optimization alongside torch and accelerate dependencies.
Configure load balancing and routing mechanisms for large-scale MoE models by setting parameters like moe-loss-coeff, moe-train-capacity-factor, and moe-expert-parallel-size in your training script to optimize expert utilization and inference performance.
Training MoE models compared to dense models requires careful tuning of routing mechanisms and load balancing to prevent expert underutilization, though it achieves better performance per compute budget by activating only a subset of parameters during inference.
OpenLAIRTrain Mixture-of-Experts models at scale with DeepSpeed or HuggingFace.
Lord1EgyptLoad, run, and fine-tune pre-trained transformer models for NLP, vision, and audio tasks.
zhuangbiaoweiTrain Mixture of Experts models using DeepSpeed and HuggingFace Transformers.
hung-phanImplements sparse Mixture of Experts routing for language models.