skills.rest
kngender5
kngender5COMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve high-throughput LLM APIs with OpenAI-compatible endpoints and quantization.

Updated Jun 4, 2026
Search Tags:#quantization#OpenAI API#inference optimization#tensor parallelism#LLM serving
One-click install
npx skills add https://github.com/kngender5/hermes --skill serving-llms-vllm-kngender5
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/kngender5/hermes/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/kngender5/hermes --skill serving-llms-vllm-kngender5

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenges of deploying production LLM APIs, optimizing inference latency/throughput, and serving models with limited GPU memory, especially for OpenAI-compatible endpoints and quantization.

Core Features & Use Cases

  • High Throughput LLM Serving: Achieves 24x higher throughput than standard transformers.
  • OpenAI API Compatibility: Supports OpenAI-compatible endpoints.
  • Quantization: Offers quantization methods like GPTQ/AWQ/FP8 for memory-efficient model serving.
  • Tensor Parallelism: Utilizes tensor parallelism for large models.
  • Use Case: Ideal for deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory.

Quick Start

Use the vLLM skill to serve the 'meta-llama/Llama-3-8B-Instruct' model on port 8000.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference throughput for a production API?▼

To optimize LLM inference throughput, you can utilize high-performance serving engines that achieve up to 24x higher throughput than standard transformers, supporting OpenAI-compatible endpoints and tensor parallelism for large models.

How do I serve large LLMs with limited GPU memory?▼

To serve large LLMs with limited GPU memory, apply quantization methods like GPTQ, AWQ, or FP8 for memory-efficient model serving, and distribute the workload across multiple GPUs using tensor parallelism.

Does vLLM support OpenAI API compatible endpoints?▼

Yes, vLLM supports OpenAI API compatible endpoints, allowing you to directly replace OpenAI API calls with your own high-throughput, self-hosted LLM serving infrastructure without changing client application code.

What is the best way to deploy an OpenAI-compatible LLM API using vLLM?▼

The best way to deploy an OpenAI-compatible LLM API using vLLM is by running the serving script with your chosen model, such as Llama-3-8B-Instruct, and configuring it to serve requests on a designated port like 8000 for optimized inference performance.

When do I need tensor parallelism for LLM serving?▼

You need tensor parallelism for LLM serving when deploying large models that exceed the memory capacity of a single GPU, allowing you to distribute the model weights and computation across multiple GPUs for optimized inference performance.

Can I use GPTQ or AWQ quantization for memory-efficient model serving?▼

Yes, you can use GPTQ or AWQ quantization for memory-efficient model serving, which reduces the GPU memory footprint of large language models while maintaining optimized inference latency and high throughput in production environments.

Related Skills

Explore More Software Engineering Skills→
Mekann2904Mekann2904

research-ml-deep

Integrate PyTorch Lightning, Transformers, and PyTorch Geometric for deep learning tasks.

Community
Advanced
ax-llmax-llm
2.9k

ax-rust-ai

Generate Rust AxAI provider client code for axllm workflows.

Official
Intermediate
grahama1970grahama1970
1

runpod-ops

Provision, manage, and terminate RunPod GPU instances for LLM workflows.

Community
Intermediate
lxh755818-botlxh755818-bot

serving-llms-vllm

Serve LLMs with OpenAI API, quantization, and tensor parallelism.

Community
Advanced