skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

deployment

Deploy HuggingFace model checkpoints as OpenAI-compatible endpoints with vLLM or SGLang.

2.8k|332|Updated Feb 25, 2026
Search Tags:#deployment#gpu#endpoint#inference#vllm#sglang#modelopt
One-click install
npx skills add https://github.com/NVIDIA/skills --skill deployment-nvidia
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: deployment
Source: https://github.com/NVIDIA/skills/tree/main/skills/Model-Optimizer/deployment
Command: npx skills add https://github.com/NVIDIA/skills --skill deployment-nvidia

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires python3, curl, nvidia-smi, and includes scripts (resource) components.

What problem does it solve?

Simplifies turning model checkpoints into live OpenAI-compatible inference endpoints by handling quantization, framework selection, and deployment orchestration.

Core Features & Use Cases

  • Supports vLLM and SGLang serving of quantized and unquantized HuggingFace checkpoints.
  • Automatically detects quantization formats and chooses the appropriate deployment path.
  • Suitable for local development, staging deployments, and remote cluster deployments for scalable inference.
  • Quick-start guidance via the provided deploy.sh script or equivalent commands.

Quick Start

Run the deploy.sh script with --model pointing to your checkpoint to launch a live OpenAI-compatible endpoint.

Frequently Asked Questions about deployment

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy a quantized model checkpoint as an OpenAI-compatible endpoint?▼

You can deploy quantized or unquantized HuggingFace checkpoints as OpenAI-compatible endpoints by running the deploy.sh script with the --model flag pointing to your local path or repository.

Does this deployment skill automatically detect quantization formats for vLLM and SGLang?▼

Yes, automatic quantization detection is supported. The skill identifies the quantization format of your checkpoint and selects the appropriate deployment path for vLLM or SGLang serving.

How do I determine tensor parallelism for GPU inference deployment?▼

The skill guides GPU availability checks and memory estimates to determine tensor parallelism. It enforces environment checks using nvidia-smi to ensure your GPUs can support the model.

Can I use this skill for local development and remote cluster deployments?▼

Yes, the skill supports local development, staging deployments, and remote cluster deployments for scalable inference. It selects the appropriate framework based on model format and environment.

What dependencies do I need to serve inference endpoints with this tool?▼

You need python3, curl, and nvidia-smi installed. These dependencies allow the skill to execute scripts, verify GPU availability, and interact with the launched OpenAI-compatible API.

What is the best way to serve unquantized HuggingFace checkpoints for inference?▼

The best way is using the provided deploy.sh script, which selects either vLLM or SGLang based on your model format. It handles environment checks and launches a live endpoint quickly.

Related Skills

Explore More Software Engineering Skills→
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
RheasilviaRheasilvia

serving-llms-vllm

Deploy OpenAI-compatible LLM inference servers using vLLM with PagedAttention and continuous batching.

Community
Advanced
annealing-inversionannealing-inversion

sglang-diffusion-modelopt-quant

Quantize diffusion DiT models with NVIDIA ModelOpt for SGLang Diffusion.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced