skills.rest
WangWindow
WangWindowCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

model-infer-kvcache

Implements and optimizes KVCache for LLM inference on Ascend NPU with PyTorch.

Updated Sep 15, 2026
Search Tags:#pytorch#llm inference#paged attention#mla#kvcache#ascend npu#fused attention
One-click install
npx skills add https://github.com/WangWindow/CANN-BatchMatMulMaxsum --skill model-infer-kvcache-wangwindow
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: model-infer-kvcache
Source: https://github.com/WangWindow/CANN-BatchMatMulMaxsum/tree/main/.agents/skills/model-infer-kvcache
Command: npx skills add https://github.com/WangWindow/CANN-BatchMatMulMaxsum --skill model-infer-kvcache-wangwindow

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, torch_npu, and includes references (resource) components.

What problem does it solve? LLM inference on Ascend NPU requires careful KVCache management to control memory usage and latency. This Skill guides the analysis and code migration of KVCache implementations, covering Legacy contiguous caches, Paged Attention with fused FA operators, and MLA compressed caches, so models avoid OOM and performance regressions. ## Core Features & Use Cases - Mode Selection: Chooses between Legacy, Paged Attention (FA + TND layout), and MLA absorb paths based on model attention architecture (MHA/GQA, sliding window, MLA) and deployment mode (framework vs standalone). - Data Structure Construction: Explains and implements block_table, slot_mapping, actual_seq_lengths, and kv_len lifecycle for both framework-managed and runner-managed deployments. - FA Operator Integration: Details npu_fused_infer_attention_score v1/v2 parameter mapping, sparse_mode and atten_mask constraints, and cache write operators like npu_scatter_nd_update_ and npu_kv_rmsnorm_rope_cache. - Use Case: When migrating a Qwen3 or DeepSeek-style model to cann-recipes-infer, use this Skill to convert Legacy scatter_update_ caches into Paged mode with get_cache_info(), cache_entries, and per-attn_type block_table routing, then verify the initialization, runtime, and release links. ## Quick Start Ask the assistant to analyze your model's attention architecture and migrate its KVCache to Paged Attention mode on Ascend NPU.

Frequently Asked Questions about model-infer-kvcache

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement Paged Attention KVCache on Ascend NPU?▼

Paged Attention stores KV in fixed-size physical blocks indexed by a block_table, with slot_mapping computed each step for cache writes. Use npu_scatter_nd_update_ for writes and npu_fused_infer_attention_score with block_table and TND layout for reads.

What is the difference between FA v1 and v2 fused attention operators?▼

FA v1 uses parameters like scale, num_heads, and actual_seq_lengths, while v2 renames them to softmax_scale, num_query_heads, and actual_seq_qlen. Mixing names silently falls back to defaults and causes precision failures, so match names to the version.

How does MLA compressed cache reduce memory usage?▼

MLA caches only low-rank latent tensors (nope_cache at kv_lora_rank dimension plus rope_cache) instead of full per-head KV, drastically shrinking memory. The absorb path passes the same compressed cache as key and value to the FA operator.

Why does Prefill produce garbage output without errors on NPU?▼

The usual cause is sparse_mode=0 with atten_mask=None, which removes causal masking. Use sparse_mode=3 with a [2048, 2048] boolean causal mask; mask dtype must be bool, int8, or uint8.

Can this KVCache approach reduce training memory usage?▼

No, this Skill covers inference-stage KVCache optimization only, including paging, compression, and fused operators. Training memory optimization requires different techniques such as gradient checkpointing or optimizer sharding.

Who constructs block_table in framework versus standalone deployment?▼

In framework deployment, the executor's KVCacheManager and BlockPool dynamically allocate blocks and build block_table and slot_mapping. In standalone deployment, the Runner statically pre-allocates block_table and recomputes slot_mapping each step from kv_len.

Related Skills

Explore More Software Engineering Skills→
dongg622dongg622

boltzgen

Migrate protein generation workflows from CUDA to Ascend NPU hardware.

Community
Advanced
AevonixAevonix

serving-llms-vllm

Serve LLM inferences with vLLM using PagedAttention and continuous batching.

Official
Advanced
tuliang1024tuliang1024

kvcache-optimization

Optimize KVCache handling for LLM inference to reduce memory usage and improve throughput.

Community
Advanced
tankpkgtankpkg
1

@tank/local-llm-fit

Determine if an open-source LLM fits a PC's VRAM, RAM, and bandwidth.

Official
Advanced