vllm-caching
OfficialProduction KV caching for production-grade vLLM
Authorair-gapped
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Tiered KV caching for vLLM workloads enables production-grade long-context inference by offloading KV data to CPU DRAM, NVMe, or disaggregated backends, reducing re-fetches and latency.
Core Features & Use Cases
- Supports Native CPU offload, LMCache DRAM+NVMe, NixlConnector for disaggregated prefill, MooncakeConnector for RDMA, and MultiConnector to compose backends.
- Provides sizing guidance and version-gated recommendations to optimize throughput and hit rates for long-context prompts.
- Use case: operate a multi-GPU vLLM service with high-context sessions and daily re-use of prompts across requests.
Quick Start
Configure a vLLM deployment to enable tiered KV caching and verify a small benchmark.
Dependency Matrix
Required Modules
None requiredComponents
references
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: vllm-caching Download link: https://github.com/air-gapped/skills/archive/main.zip#vllm-caching Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 510,000+ vetted skills library on demand.