infra-cache

Configure per-model prompt and KV cache settings in LocalAI YAML files.

Updated Mar 26, 2026

Search Tags:#cache #anthropic #inference #prompt-cache #localai #kv-cache

Ready to run

npx skills add https://github.com/cyberpunk042/devops-expert-local-ai --skill infra-cache

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: infra-cache
Source: https://github.com/cyberpunk042/devops-expert-local-ai/tree/main/.claude/skills/infra-cache
Command: npx skills add https://github.com/cyberpunk042/devops-expert-local-ai --skill infra-cache

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manage AICP's inference-specific caches to optimize prompt tokenization, KV quantization, and embedding cache performance.

Core Features & Use Cases

LocalAI prompt cache configuration per model (paths, enable/disable)
KV cache quantization tuning (k/v types per model)
Anthropic prompt caching for cloud backends
Embedding cache management with nomic-embed
Use cases: reduce tokenization time, fit larger contexts, cut cloud costs with reuse

Quick Start

Edit config/models/<name>.yaml to enable and tune prompt_cache_path, prompt_cache_all, cache_type_k, and cache_type_v, then restart LocalAI to apply changes.

Frequently Asked Questions about infra-cache

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I configure a prompt cache for LocalAI models?▼

To configure a prompt cache for LocalAI models, edit the per-model YAML file to set the prompt_cache_path and prompt_cache_all keys, then restart LocalAI to apply the changes and verify tokenization speed.

Can I tune KV cache quantization for specific models?▼

You can tune KV cache quantization per model by defining the cache_type_k and cache_type_v YAML keys in the model configuration, allowing you to fit larger contexts and optimize inference performance.

Does Anthropic prompt caching work with cloud backends?▼

Anthropic prompt caching works with cloud backends by enforcing specific cache controls per model, which helps cut cloud costs through token reuse without requiring local infrastructure.

What is the best way to manage embedding cache performance with nomic-embed?▼

Managing embedding cache performance with nomic-embed involves adjusting per-model cache configurations within AICP to optimize prompt tokenization and embedding retrieval for LocalAI workflows.

Why does my LocalAI configuration not apply cache changes?▼

LocalAI configuration changes to prompt_cache_path or cache_type keys require a full service restart to take effect, as the inference engine loads these YAML settings during initialization.

Are YAML configuration keys required for every model cache?▼

YAML configuration keys like prompt_cache_path, prompt_cache_all, cache_type_k, and cache_type_v are enforced on a per-model basis, ensuring isolated and granular cache control across different models.