What problem does it solve? Training large language models with dense architectures demands compute budgets that scale linearly with parameter count, making billion-parameter models prohibitively expensive. This Skill provides the architectures, configurations, and tuning practices needed to train sparse Mixture of Experts models that activate only a fraction of parameters per token, cutting training cost by roughly 5× compared to dense equivalents. ## Core Features & Use Cases - MoE Architecture Implementation: Build sparse MoE layers with top-k routing, gating networks, load balancing losses, and router z-loss in PyTorch, matching designs from Mixtral 8x7B, DeepSeek-V3, and Switch Transformers. - DeepSpeed Training Configuration: Configure expert parallelism, capacity factors, token dropping, and ZeRO optimization for distributed MoE training across multiple GPUs. - Inference Optimization: Apply FP8/INT8 quantization, fused MoE kernels, speculative decoding, and expert pruning with vLLM for production serving. - Use Case: A team wants to train a Mixtral-style model with 8 experts and top-2 routing on 8 GPUs. This Skill supplies the DeepSpeed config, training script, learning rate guidelines, and load balancing coefficients to launch the run and avoid common instability pitfalls. ## Quick Start Ask the agent to set up a DeepSpeed MoE training run with 8 experts, top-2 routing, and expert parallelism across your available GPUs.
