deployment

Deploy HuggingFace model checkpoints as OpenAI-compatible endpoints with vLLM or SGLang.

2.8k|332|Updated Feb 25, 2026

Search Tags:#deployment #gpu #endpoint #inference #vllm #sglang #modelopt

Ready to run

npx skills add https://github.com/NVIDIA/skills --skill deployment-nvidia

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: deployment
Source: https://github.com/NVIDIA/skills/tree/main/skills/Model-Optimizer/deployment
Command: npx skills add https://github.com/NVIDIA/skills --skill deployment-nvidia

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires python3, curl, nvidia-smi, and includes scripts (resource) components.

What problem does it solve?

Simplifies turning model checkpoints into live OpenAI-compatible inference endpoints by handling quantization, framework selection, and deployment orchestration.

Core Features & Use Cases

Supports vLLM and SGLang serving of quantized and unquantized HuggingFace checkpoints.
Automatically detects quantization formats and chooses the appropriate deployment path.
Suitable for local development, staging deployments, and remote cluster deployments for scalable inference.
Quick-start guidance via the provided deploy.sh script or equivalent commands.

Quick Start

Run the deploy.sh script with --model pointing to your checkpoint to launch a live OpenAI-compatible endpoint.

Frequently Asked Questions about deployment

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I deploy a quantized model checkpoint as an OpenAI-compatible endpoint?▼

You can deploy quantized or unquantized HuggingFace checkpoints as OpenAI-compatible endpoints by running the deploy.sh script with the --model flag pointing to your local path or repository.

Does this deployment skill automatically detect quantization formats for vLLM and SGLang?▼

Yes, automatic quantization detection is supported. The skill identifies the quantization format of your checkpoint and selects the appropriate deployment path for vLLM or SGLang serving.

How do I determine tensor parallelism for GPU inference deployment?▼

The skill guides GPU availability checks and memory estimates to determine tensor parallelism. It enforces environment checks using nvidia-smi to ensure your GPUs can support the model.

Can I use this skill for local development and remote cluster deployments?▼

Yes, the skill supports local development, staging deployments, and remote cluster deployments for scalable inference. It selects the appropriate framework based on model format and environment.

What dependencies do I need to serve inference endpoints with this tool?▼

You need python3, curl, and nvidia-smi installed. These dependencies allow the skill to execute scripts, verify GPU availability, and interact with the launched OpenAI-compatible API.

What is the best way to serve unquantized HuggingFace checkpoints for inference?▼

The best way is using the provided deploy.sh script, which selects either vLLM or SGLang based on your model format. It handles environment checks and launches a live endpoint quickly.