skills.rest
soulmachine
soulmachineCOMMUNITY
Explore all skills by this author →
𝕏@soulmachineGitHub

deploy-kimi-k26-on-rtx-pro-6000

Deploy Moonshot Kimi-K2.6 on NVIDIA RTX PRO 6000 GPUs with vLLM or SGLang.

3|1|Updated Mar 31, 2026
Search Tags:#Docker#AI model deployment#high performance#vLLM#SGLang#NVIDIA RTX PRO 6000
One-click install
npx skills add https://github.com/soulmachine/skills --skill deploy-kimi-k26-on-rtx-pro-6000
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: deploy-kimi-k26-on-rtx-pro-6000
Source: https://github.com/soulmachine/skills/tree/main/deploy-kimi-k26-on-rtx-pro-6000
Command: npx skills add https://github.com/soulmachine/skills --skill deploy-kimi-k26-on-rtx-pro-6000

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires docker, nvidia-container-toolkit, nvidia-persistenced, python3, curl, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the deployment and serving of the Moonshot Kimi-K2.6 AI model on NVIDIA RTX PRO 6000 Blackwell GPUs, offering a user-chosen quantization (INT4 QAT or NVFP4) and selecting between vLLM or SGLang engines for optimal performance.

Core Features & Use Cases

  • Deploy Kimi-K2.6: Automates the deployment of Kimi-K2.6 (1T MoE, MLA, 256K context, vision) on Linux servers with 8× NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs.
  • Quantization and Engine Selection: Allows the user to select INT4 QAT or NVFP4 quantization and choose between vLLM or SGLang engines based on hardware recommendation.
  • Official Image Docker Container: Serves an OpenAI-compatible API on port 30000 through an official-image Docker container.
  • High-Performance Serving: Achieves high performance with TP=8 and weights bind-mounted in VRAM.
  • Productionization: Provides a systemd service and reverse proxy setup for production deployment.

Quick Start

Deploy the Kimi-K2.6 model with INT4 QAT on vLLM on a Linux server with 8× NVIDIA RTX PRO 6000 Blackwell GPUs by running the following commands:

QUANT=int4 FRAMEWORK=vllm bash scripts/serve_docker_vllm.sh

Frequently Asked Questions about deploy-kimi-k26-on-rtx-pro-6000

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy the Kimi-K2.6 AI model on NVIDIA RTX PRO 6000 GPUs?▼

To deploy the Kimi-K2.6 AI model on NVIDIA RTX PRO 6000 GPUs, run the provided Docker script with your chosen quantization and framework, which automates serving an OpenAI-compatible API.

Can I use SGLang instead of vLLM for Kimi-K2.6 inference?▼

Yes, you can use SGLang instead of vLLM for Kimi-K2.6 inference by setting the FRAMEWORK variable to SGLang, allowing you to choose the optimal engine based on your hardware.

What is the difference between INT4 QAT and NVFP4 quantization for Kimi-K2.6?▼

INT4 QAT and NVFP4 are the supported quantizations for Kimi-K2.6, letting you optimize high-performance AI model serving on the NVIDIA RTX PRO 6000 Blackwell GPUs based on hardware recommendations.

Does the Kimi-K2.6 deployment require Docker and NVIDIA Container Toolkit?▼

Yes, deploying Kimi-K2.6 requires Docker and the NVIDIA Container Toolkit to manage the official-image Docker container and bind-mount weights in VRAM for high-performance serving.

How do I set up a production-grade API endpoint for Kimi-K2.6?▼

To set up a production-grade API endpoint for Kimi-K2.6, the deployment process includes configuring a systemd service and reverse proxy, serving an OpenAI-compatible API on port 30000.

Are 8 NVIDIA RTX PRO 6000 Blackwell GPUs necessary for Kimi-K2.6?▼

The deployment is optimized for Linux servers with 8× NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs to achieve high performance using TP=8 and weights bind-mounted in VRAM.

Related Skills

Explore More Software Engineering Skills→
sayalinvidiasayalinvidia

holoscan-install-container

Install Holoscan SDK in a Docker container and validate GPU-enabled execution.

Community
Advanced
frankxaifrankxai
2

nvidia-nim

Configure and run NVIDIA NIM inference via OpenAI-compatible APIs.

Community
Advanced
DevMHOneDevMHOne

omni-inference

Expose OpenAI-compatible inference endpoints for chat, embeddings, images, audio, and more.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced