sglang-hicache

Name: sglang-hicache
Availability: InStock
Author: air-gapped

Configure SGLang HiCache tiered KV caching with per-rank sizing and eviction policies.

5|1|Updated Apr 19, 2026

Search Tags:#distributed-cache #kv-cache #mooncake #hybrid-models #nixl #hicache #hf3fs

Ready to run

npx skills add https://github.com/air-gapped/skills --skill sglang-hicache

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: sglang-hicache
Source: https://github.com/air-gapped/skills/tree/main/.claude/skills/sglang-hicache
Command: npx skills add https://github.com/air-gapped/skills --skill sglang-hicache

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires curl, python3, and includes scripts (resource) and references (resource) components.

What problem does it solve?

SGLang HiCache enables hierarchical KV caching to extend per-rank GPU memory with L2 host RAM and optional L3 storage, unlocking larger models and longer context.

Core Features & Use Cases

Three-tier KV cache (L1/L2/L3) with per-rank sizing, eviction policies, and configurable prefetch.
Supports multiple L3 backends (mooncake, hf3fs, nixl, aibrix, eic, simm, file) and runtime attach/detach for swapping backends.
Ideal for production workloads with long-context agents, multi-tenant inference, and hybrid-model deployments.

Quick Start

Start a SGLang server with hierarchical cache enabled and pick a backend (e.g., Mooncake) using per-rank sizing and a production-safe prefetch policy.

Frequently Asked Questions about sglang-hicache

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scale KV cache beyond GPU memory for long-context agent workloads?▼

SGLang HiCache extends per-rank KV storage by implementing a three-tier KV cache (L1/L2/L3), utilizing L2 host RAM and optional L3 storage to support larger models and longer contexts.

What distributed backends can I use for L3 KV cache storage?▼

You can use multiple L3 backends for distributed KV cache storage including mooncake, hf3fs, nixl, aibrix, eic, simm, and file, with support for runtime attach and detach to swap backends dynamically.

Can I configure per-rank sizing and eviction policies for multi-tenant inference?▼

Yes, SGLang HiCache supports per-rank sizing, configurable eviction policies, and prefetch controls to enforce tiered caching behavior for multi-tenant inference across distributed backends.

How do I start a SGLang server with hierarchical cache enabled?▼

Start a SGLang server with hierarchical cache enabled by selecting an L3 backend such as Mooncake, then configuring per-rank sizing and a production-safe prefetch policy using the provided recipes.

Do I need Python3 to configure SGLang HiCache for hybrid-model deployments?▼

Yes, Python3 and curl are required dependencies to configure SGLang HiCache for hybrid-model deployments, utilizing provided scripts and references for production deployment recipes.