skills.rest
sgl-project
sgl-projectOFFICIAL
Explore all skills by this author →
GitHub

compute-mamba-ratio

Computes the optimal memory split ratio between state and KV pools for hybrid Mamba model serving.

33.0k|8.4k|Updated Jan 8, 2024
Search Tags:#SGLang#kv cache#LLM serving#Concurrency#Mamba#memory ratio#hybrid model
One-click install
npx skills add https://github.com/sgl-project/sglang --skill compute-mamba-ratio
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: compute-mamba-ratio
Source: https://github.com/sgl-project/sglang/tree/main/.claude/skills/compute-mamba-ratio
Command: npx skills add https://github.com/sgl-project/sglang --skill compute-mamba-ratio

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Hybrid models combining attention layers with linear-attention layers (Mamba, GDN, KDA) split serving memory into two independently budgeted pools, and choosing the wrong --mamba-full-memory-ratio causes one pool to bottleneck first, clamping concurrency or wasting GPU memory. This Skill computes the ratio at which neither pool bottlenecks first for a given workload.

Core Features & Use Cases

  • Optimal ratio computation: Derives r* = (S + D) · token_equiv · dcp_size / L from workload context length, per-slot state bytes, per-token KV bytes, cache strategy, speculative decoding, and DCP configuration.
  • Concurrency clamp prediction: Predicts max_running_requests from the memory budget and identifies which pool (state vs KV) binds first.
  • Configuration guidance: Covers cache strategies (extra_buffer, extra_buffer_lazy, no_buffer), dtype effects (fp8 KV, fp32/bf16 SSM), DCP sharding, and when to pin --max-mamba-cache-size instead of dialing a ratio.
  • Use Case: An engineer serving a hybrid Mamba model on TP8 with DCP8 reads the boot log byte constants, provides an average context length of 9216 tokens, and receives r ≈ 14 with a predicted clamp of 125, matching measured behavior.

Quick Start

Ask the assistant to compute the optimal --mamba-full-memory-ratio for your hybrid Mamba model given your average context length, boot log memory figures, cache strategy, and DCP size.

Frequently Asked Questions about compute-mamba-ratio

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose --mamba-full-memory-ratio for a hybrid Mamba model?▼

Compute r = (S + D) · token_equiv · dcp_size / L, where token_equiv is state bytes per slot divided by KV bytes per token from your boot log, S comes from your cache strategy, D is speculative draft tokens, and L is average context length.

Why is my concurrency clamped when serving a Mamba hybrid model?▼

The state pool is worst-case reserved and fail-loud, so max_running_requests equals max_mamba_cache_size divided by slots per request S. If the ratio under-provisions the state pool relative to KV, concurrency clamps before KV fills.

How does DCP affect the mamba memory ratio?▼

DCP shards per-rank KV by dcp_size, making KV roughly dcp_size times cheaper per request, so the optimal ratio multiplies by dcp_size. Under DCP the practical recommendation is to pin --max-mamba-cache-size directly instead of dialing a large ratio.

Does fp8 KV cache change the optimal memory split?▼

Yes, fp8 KV halves bytes per token, doubling token_equiv and therefore doubling the optimal ratio. This is an accuracy-gated change, so validate output quality for your workload before using it in production.

When should I pin --max-mamba-cache-size instead of setting a ratio?▼

Pin it when the computed ratio would drop below roughly 0.15, which happens at very long context lengths or under DCP. Set it to target concurrency times S and let the remaining memory go to the KV pool.

How does speculative decoding change the state pool sizing?▼

Plain speculative decoding adds D draft-token intermediate states per running request, raising the ratio. With ReplaySSM spec-verify, those states move to a fixed ring buffer, so use D=0 and the NOSPEC ratio instead.

Related Skills

Explore More Software Engineering Skills→
ihatesea69ihatesea69
3

mamba-architecture

Implement the Mamba selective state-space model for linear O(n) sequence processing.

Community
Advanced
FutureMLS-LabFutureMLS-Lab
556

sglang-auto-benchmark

Search SGLang server configurations against throughput and latency objectives.

Official
Advanced
pauldotyupauldotyu

aks-nodepool-management

Create dedicated AKS user node pools and migrate workloads off the system pool.

Community
Advanced
PremModhaOfficialPremModhaOfficial

connection-pool-tuning

Tune motadatagosdk/core/pool min/max, idle timeout, and healthcheck cadence.

Community
Basic