skills.rest
BagelHole
BagelHoleCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

vllm-server

Deploy and manage vLLM for high-throughput LLM inference with OpenAI-compatible API endpoints.

46|4|Updated Jan 27, 2026
Search Tags:#performance#api#llm#inference#vllm#serving
One-click install
npx skills add https://github.com/BagelHole/DevOps-Security-Agent-Skills --skill vllm-server
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: vllm-server
Source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/tree/main/infrastructure/local-ai/vllm-server
Command: npx skills add https://github.com/BagelHole/DevOps-Security-Agent-Skills --skill vllm-server

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables the deployment and management of vLLM for high-throughput LLM inference, optimizing serving performance for production environments.

Core Features & Use Cases

  • Deploy LLMs: Serve open-source LLMs like Llama, Mistral, and Gemma.
  • OpenAI-Compatible API: Expose self-hosted models via a familiar API endpoint.
  • Performance Optimization: Configure continuous batching, tensor parallelism, and quantization for reduced latency and increased throughput.
  • Use Case: Deploy a Llama-3.1-70B-Instruct model with tensor parallelism across two GPUs, serving requests via an OpenAI-compatible API for a customer-facing application.

Quick Start

Serve the meta-llama/Llama-3.1-8B-Instruct model using vLLM with an OpenAI-compatible API on port 8000.

Frequently Asked Questions about vllm-server

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an OpenAI-compatible API for self-hosted LLMs?▼

To deploy an OpenAI-compatible API for self-hosted LLMs, use vLLM to serve open-source models like Llama or Mistral, exposing familiar API endpoints on your designated port for production inference.

What's the best way to optimize high-throughput LLM inference across multiple GPUs?▼

Optimize high-throughput LLM inference across multiple GPUs by configuring vLLM with tensor parallelism, continuous batching, and quantization, which reduces latency and increases request throughput for large models.

Do I need NVIDIA GPUs with CUDA to run vLLM for production LLM serving?▼

Yes, running vLLM for production LLM serving requires NVIDIA GPUs with CUDA 12.1+ installed, alongside either Docker or Python 3.9+, to support high-throughput inference and tensor parallelism.

How does continuous batching improve LLM inference performance?▼

Continuous batching improves LLM inference performance by dynamically processing incoming requests without waiting for a batch to complete, significantly reducing latency and maximizing GPU utilization in production environments.

Can I serve a Llama-3.1-70B-Instruct model using vLLM with tensor parallelism?▼

Yes, you can serve the Llama-3.1-70B-Instruct model using vLLM with tensor parallelism across multiple GPUs, enabling high-throughput inference via an OpenAI-compatible API endpoint for customer-facing applications.

Related Skills

Explore More Software Engineering Skills→
superfhpsuperfhp

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT on NVIDIA GPUs.

Community
Advanced
ModelsLabModelsLab
10

modelslab-chat-generation

Chat with 60+ LLMs through one OpenAI-compatible endpoint.

Official
Advanced
zaiyemeiyou404zaiyemeiyou404

serving-llms-vllm

Serve LLM models with OpenAI-compatible endpoints using vLLM.

Community
Advanced
jayll1303jayll1303
17

vllm-tgi-inference

Deploy local vLLM or HuggingFace TGI inference servers with OpenAI-compatible APIs.

Community
Advanced