Coordinate vLLM KV caching across native offload, LMCache, Nixl, and Mooncake backends.
npx skills add https://github.com/air-gapped/skills --skill vllm-caching
Please help me install this Agent Skill. Skill: vllm-caching Source: https://github.com/air-gapped/skills/tree/main/.claude/skills/vllm-caching Command: npx skills add https://github.com/air-gapped/skills --skill vllm-caching
Tiered KV caching for vLLM workloads enables production-grade long-context inference by offloading KV data to CPU DRAM, NVMe, or disaggregated backends, reducing re-fetches and latency.
Configure a vLLM deployment to enable tiered KV caching and verify a small benchmark.
High-intent search queries and answers about installing and using this skill.
Configure vLLM KV caching by selecting native CPU offload, LMCache, or disaggregated backends like Mooncake. This offloads KV data to CPU DRAM or NVMe, reducing re-fetches and optimizing latency for long-context inference.
The best way to offload KV cache in multi-GPU vLLM deployments is using LMCache for DRAM+NVMe or MooncakeConnector for RDMA. MultiConnector can compose these backends to optimize throughput and hit rates.
vLLM supports disaggregated prefill with RDMA backends via the MooncakeConnector. NixlConnector also enables disaggregated prefill scenarios, coordinating KV cache transfers across H100 and H200 GPU clusters.
You can compose LMCache DRAM+NVMe with native CPU offload using the MultiConnector. This tiered KV caching approach allows vLLM to layer backends, balancing latency and hit rates for long-context prompts.
Size vLLM KV cache by applying the sizing math and version-gated recommendations provided for tiered offload. Properly sizing KV cache directly impacts latency and hit rates for daily re-use of long-context prompts.
Tiered KV offload for vLLM may not suit workloads lacking prompt re-use across requests or single-GPU setups without disaggregated prefill. Without high-context sessions, offloading KV data yields minimal latency benefits.
gqf2008Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and multi-GPU scaling.
flagos-aiBenchmark vLLM-served AI models across five workload profiles for throughput, latency, TTFT, and TPOT.
air-gappedCoordinate cross-node memory transfers and metadata exchange with NIXL.
air-gappedRun LMCache MP to share KV caches among vLLM pods in Kubernetes clusters.
