skills.rest
StarryCod
StarryCodCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve LLMs via vLLM with OpenAI-compatible endpoints and tensor parallelism.

11|Updated May 17, 2026
Search Tags:#production#endpoints#quantization#inference#vllm#llm-serving#gpu-parallelism
One-click install
npx skills add https://github.com/StarryCod/cogitum --skill serving-llms-vllm-starrycod
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/StarryCod/cogitum/tree/main/cogitum/data/skills/mlops/inference/vllm
Command: npx skills add https://github.com/StarryCod/cogitum --skill serving-llms-vllm-starrycod

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

vLLM provides a high-performance serving backend that enables OpenAI-compatible endpoints and efficient quantization for large language models, reducing latency and increasing throughput in production deployments.

Core Features & Use Cases

  • High-throughput serving with OpenAI-compatible APIs
  • Quantization options (AWQ, GPTQ, FP8) for memory efficiency
  • Tensor parallelism and multi-GPU deployment
  • Production-grade metrics, prefix caching and monitoring Use cases include hosting chatbots, QA assistants, and internal copilots at scale.

Quick Start

Deploy a vLLM server to expose an OpenAI-compatible endpoint for production-scale inference.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve LLMs in production with vLLM?▼

Serve LLMs in production with vLLM by deploying its high-performance backend to expose OpenAI-compatible endpoints, enabling high-throughput inference for chatbots and copilots.

Can I use vLLM for multi-GPU tensor parallelism?▼

Yes, you can use vLLM for multi-GPU deployment by configuring tensor parallelism, which splits model execution across GPUs to handle production-grade inference workloads efficiently.

What quantization options does vLLM support for memory efficiency?▼

vLLM supports AWQ, GPTQ, and FP8 quantization options to optimize memory efficiency, reducing latency and increasing throughput for large language models in production deployments.

How do I expose OpenAI-compatible endpoints using vLLM?▼

Expose OpenAI-compatible endpoints using vLLM by deploying its server, which natively provides API compatibility for seamless tooling integration and production-scale inference workloads.

Does vLLM provide production-grade metrics and monitoring?▼

Yes, vLLM provides production-grade metrics exposure and prefix caching, enabling comprehensive monitoring and optimized performance for scalable LLM serving workloads.

What is the best way to scale LLM serving for internal copilots?▼

The best way to scale LLM serving for internal copilots is using vLLM, which offers high-throughput serving, GPU memory management, and OpenAI-compatible APIs for production deployments.

Related Skills

Explore More Software Engineering Skills→
GliteTechGliteTech
16

setup-remote-machine

Provision and manage remote GPU machines with SSH readiness checks and teardown.

Official
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced
hsliuustc0106hsliuustc0106
84

vllm-omni-serving

Launch and configure vLLM-Omni API servers for OpenAI-compatible models.

Community
Advanced
Quill-AgentQuill-Agent
3

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints and quantization using vLLM.

Community
Advanced