Launches vLLM-based OpenAI-compatible endpoints with PagedAttention and continuous batching.
npx skills add https://github.com/graniet/kheish --skill vllm-graniet
Please help me install this Agent Skill. Skill: vllm Source: https://github.com/graniet/kheish/tree/main/skills/mlops/inference/vllm Command: npx skills add https://github.com/graniet/kheish --skill vllm-graniet
Standard LLM serving solutions suffer from low throughput, high latency, and inability to fit large models on limited GPU hardware, making production deployment costly and inefficient. This Skill eliminates those bottlenecks by providing vLLM-based serving with optimized memory management and continuous batching.
Use the vllm skill to launch a production-ready OpenAI-compatible LLM serving endpoint for your target model with optimized throughput and memory usage.
High-intent search queries and answers about installing and using this skill.
Serve large LLM models on limited GPUs using AWQ, GPTQ, or FP8 quantization to reduce memory constraints. This approach allows 70B+ parameter models to run efficiently on a single 40GB GPU without significant accuracy loss.
High throughput for OpenAI-compatible LLM serving is achieved using PagedAttention and continuous batching. This combination maximizes inference performance, enabling endpoints to handle 100+ requests per second with low time-to-first-token latency.
PagedAttention optimizes LLM inference memory management by implementing continuous batching and paged memory allocation. This mechanism eliminates standard serving bottlenecks, maximizing hardware utilization and reducing time-to-first-token latency.
Process large datasets of prompts for offline batch inference without manual batching overhead using continuous batching. This handles large-scale offline inference pipelines efficiently, maximizing hardware utilization throughout processing.
vLLM deployment supports multi-format quantization including FP8, AWQ, and GPTQ for production AI API hosting. This enables memory-efficient large model serving on memory-constrained GPU environments while maintaining OpenAI-compatible endpoints.
pmcdowallConvert Hugging Face models into GGUF format with 2-8 bit quantization.
anxiety135790Deploy and optimize vLLM serving with PagedAttention and continuous batching.
cornmanwtfConfigure API Gateway rate limits, authentication, transformations, caching, and load balancing.
bailynloveServe OpenAI-compatible LLM APIs with PagedAttention and continuous batching.
