skills.rest
nyxoraAI
nyxoraAICOMMUNITY
Explore all skills by this author →
Website𝕏@Nyxora_AIGitHub

serving-llms-vllm

Serve OpenAI-compatible LLM APIs with quantization using vLLM and PyTorch/Torch transformers.

5|2|Updated May 26, 2026
Search Tags:#production#quantization#OpenAI API#high throughput#LLM serving
One-click install
npx skills add https://github.com/nyxoraAI/Nyxora --skill serving-llms-vllm-nyxoraai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/nyxoraAI/Nyxora/tree/main/packages/core/playbooks/mlops/inference/vllm
Command: npx skills add https://github.com/nyxoraAI/Nyxora --skill serving-llms-vllm-nyxoraai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenges of deploying high-throughput LLM APIs, optimizing inference latency and throughput, and serving large models with limited GPU memory.

Core Features & Use Cases

  • High Throughput LLM Serving: Achieves 24x higher throughput than standard transformers with PagedAttention and continuous batching.
  • OpenAI API Compatibility: Supports OpenAI-compatible endpoints for seamless integration.
  • Quantization: Offers quantization options (GPTQ/AWQ/FP8) for efficient memory utilization.
  • Use Case: Deploy vLLM in production environments to serve complex LLM APIs, such as chatbots, assistants, and automated assistants.

Quick Start

Install vLLM using pip and run the server with the desired model:

pip install vllm
vllm serve meta-llama/Llama-3-8B-Instruct

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve LLMs with high throughput in production?▼

Serve LLMs with high throughput using vLLM to achieve up to 24x higher inference performance than standard transformers via PagedAttention and continuous batching, optimized specifically for production environments.

Can I use an OpenAI compatible API endpoint for my own LLM deployment?▼

Yes, vLLM supports OpenAI-compatible API endpoints for your LLM deployment. This allows seamless integration of your served models into existing application infrastructures built for OpenAI APIs.

What's the best way to deploy large models with limited GPU memory?▼

Deploy large models with limited GPU memory using vLLM's quantization options like GPTQ, AWQ, or FP8. These enable efficient memory utilization, allowing large models to run on hardware with constrained VRAM.

How do I get started serving a model like Llama-3 using vLLM?▼

Get started serving Llama-3 models by installing vLLM via pip and running the command 'vllm serve meta-llama/Llama-3-8B-Instruct'. This quickly initializes a high-throughput API server for the specified model.

What dependencies do I need to run a vLLM server?▼

You need vllm, torch, and transformers installed in your Python environment to run a vLLM server. These dependencies provide the necessary framework support for high-throughput LLM inference and model handling.

Related Skills

Explore More Software Engineering Skills→
DonZzzillaDonZzzilla

llama-cpp

Run local GGUF inference for llama.cpp models with Hugging Face Hub discovery.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced
ovachieverovachiever
52

weights-and-biases

Automate ML experiment tracking and model registry management via the wandb API.

Community
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

serving-llms-vllm

Deploy an OpenAI-compatible LLM inference server using vLLM with tensor parallelism and quantization.

Community
Intermediate