sglang-mixtral-quark-int4fp8-moe-optimization

Quantize Mixtral MoE models to INT4FP8 with per-expert weight packing on AMD ROCm.

721|65|Updated Apr 1, 2026

Search Tags:#quantization #amd #fp8 #rocm #moe #int4 #mixtral

Ready to run

npx skills add https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS --skill sglang-mixtral-quark-int4fp8-moe-optimization

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: sglang-mixtral-quark-int4fp8-moe-optimization
Source: https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS/tree/main/skills/model-optimization/sglang/sglang-mixtral-quark-int4fp8-moe-optimization
Command: npx skills add https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS --skill sglang-mixtral-quark-int4fp8-moe-optimization

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a production-ready workflow for online INT4-FP8 MoE quantization of Mixtral MoE models on AMD ROCm hardware, enabling efficient deployment and testing.

Core Features & Use Cases

Online quantization path for Mixtral-8x7B MoE checkpoints on ROCm/AMD.
Quantizes high-precision MoE weights to packed INT4 and stores per-weight scales for FP8-style MoE execution.
Guides auditing and extension of Mixtral AMD quantization in PR workflows, including GSM8K regression tests and diff history.

Quick Start

Start by configuring the AMD online int4fp8_moe quantization workflow for Mixtral-8x7B and run the associated GSM8K regression test.

Frequently Asked Questions about sglang-mixtral-quark-int4fp8-moe-optimization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I quantize Mixtral MoE weights to INT4 and FP8 on AMD ROCm hardware?▼

Yes, the workflow applies online INT4FP8 quantization specifically to Mixtral-8x7B and related MoE checkpoints on AMD ROCm hardware, executing per-expert weight packing during runtime.

Does the Mixtral AMD quantization workflow support GSM8K regression testing?▼

Yes, the workflow guides auditing and extension of Mixtral AMD quantization in PR workflows and includes running associated GSM8K regression tests to track diff history and ensure model performance.

What is needed to run online weight quantization for Mixtral MoE on AMD?▼

Running online weight quantization for Mixtral MoE on AMD requires ROCm hardware setup and utilizes AMD-only quantization paths to enable per-expert weight packing and FP8-style scales in fused-MoE execution pipelines.

Can I track diff audits for Mixtral AMD quantization changes?▼

Yes, the workflow satisfies requirements for diff-audit tracking by guiding the auditing and extension of Mixtral AMD quantization directly within PR workflows, ensuring quantization changes are properly reviewed.