skills.rest
cyberpunk042
cyberpunk042COMMUNITY
Explore all skills by this author →
GitHub

infra-cache

Configure per-model prompt and KV cache settings in LocalAI YAML files.

Updated Mar 26, 2026
Search Tags:#cache#anthropic#inference#prompt-cache#localai#kv-cache
One-click install
npx skills add https://github.com/cyberpunk042/devops-expert-local-ai --skill infra-cache
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: infra-cache
Source: https://github.com/cyberpunk042/devops-expert-local-ai/tree/main/.claude/skills/infra-cache
Command: npx skills add https://github.com/cyberpunk042/devops-expert-local-ai --skill infra-cache

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manage AICP's inference-specific caches to optimize prompt tokenization, KV quantization, and embedding cache performance.

Core Features & Use Cases

  • LocalAI prompt cache configuration per model (paths, enable/disable)
  • KV cache quantization tuning (k/v types per model)
  • Anthropic prompt caching for cloud backends
  • Embedding cache management with nomic-embed
  • Use cases: reduce tokenization time, fit larger contexts, cut cloud costs with reuse

Quick Start

Edit config/models/<name>.yaml to enable and tune prompt_cache_path, prompt_cache_all, cache_type_k, and cache_type_v, then restart LocalAI to apply changes.

Frequently Asked Questions about infra-cache

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure a prompt cache for LocalAI models?▼

To configure a prompt cache for LocalAI models, edit the per-model YAML file to set the prompt_cache_path and prompt_cache_all keys, then restart LocalAI to apply the changes and verify tokenization speed.

Can I tune KV cache quantization for specific models?▼

You can tune KV cache quantization per model by defining the cache_type_k and cache_type_v YAML keys in the model configuration, allowing you to fit larger contexts and optimize inference performance.

Does Anthropic prompt caching work with cloud backends?▼

Anthropic prompt caching works with cloud backends by enforcing specific cache controls per model, which helps cut cloud costs through token reuse without requiring local infrastructure.

What is the best way to manage embedding cache performance with nomic-embed?▼

Managing embedding cache performance with nomic-embed involves adjusting per-model cache configurations within AICP to optimize prompt tokenization and embedding retrieval for LocalAI workflows.

Why does my LocalAI configuration not apply cache changes?▼

LocalAI configuration changes to prompt_cache_path or cache_type keys require a full service restart to take effect, as the inference engine loads these YAML settings during initialization.

Are YAML configuration keys required for every model cache?▼

YAML configuration keys like prompt_cache_path, prompt_cache_all, cache_type_k, and cache_type_v are enforced on a per-model basis, ensuring isolated and granular cache control across different models.

Related Skills

Explore More Software Engineering Skills→
FlexNetOSFlexNetOS

AI Inference & Model Serving

Automate AI model inference and serving setup for LocalAI and vLLM.

Community
Advanced
heysuhasheysuhas
2

serving-llms-vllm

Deploy LLM inference servers with the vLLM engine and OpenAI-compatible APIs.

Community
Advanced
DFly7DFly7

coreml

Integrate Core ML models into iOS apps for on-device inference.

Community
Advanced
OnlyTerpOnlyTerp
113

aider-cache-default-on

Enable Aider's --cache-prompts flag by default for supported models.

Community
Intermediate