Deploy vLLM to serve LLMs through an OpenAI-compatible API.
npx skills add https://github.com/devMoez/titan --skill serving-llms-vllm-devmoez
Please help me install this Agent Skill. Skill: serving-llms-vllm Source: https://github.com/devMoez/titan/tree/main/skills/mlops/inference/vllm Command: npx skills add https://github.com/devMoez/titan --skill serving-llms-vllm-devmoez
This Skill helps you deploy LLMs as production-grade inference services with high throughput and low latency, while also enabling model quantization to fit larger models into limited GPU memory.
Example: Deploy a Llama 3 model behind an OpenAI-compatible endpoint for a chatbot with sustained multi-user traffic, then tune GPU utilization, caching, and quantization to meet latency/throughput targets.
Start an OpenAI-compatible vLLM server for meta-llama/Llama-3-8B-Instruct on port 8000 with high GPU utilization using one command.
High-intent search queries and answers about installing and using this skill.
Deploy vLLM to serve LLMs through an OpenAI-compatible API for production chatbots. It optimizes multi-user throughput and latency using PagedAttention and continuous batching to maximize GPU utilization.
Reduce VRAM requirements and fit larger models into limited GPU memory by applying quantization techniques like AWQ, GPTQ, or FP8. This enables efficient LLM deployment without requiring massive hardware upgrades.
Yes, you can scale LLM serving across multiple GPUs using tensor parallelism. This distributes model weights across available hardware, enabling efficient GPU utilization for larger models that exceed single-device memory limits.
Continuous batching improves multi-user LLM inference latency by dynamically processing incoming requests together. Combined with PagedAttention, it optimizes time-to-first-token and tokens per second under sustained multi-user workload.
Yes, vLLM supports offline batch inference workloads alongside production chatbot serving. It optimizes throughput and GPU utilization for both real-time multi-user assistants and offline batch processing tasks.
Supporter09Compress large language models into 4-bit weights with activation-aware quantization.
ldzhhxxServe OpenAI-compatible LLM APIs with optimized throughput and quantization.
ToowireddGenerate TypeScript API clients with authentication, retry, and rate limiting.
gztcodeOptimizes LLM inference with PagedAttention and continuous batching for production API serving.
