infra-cache

What problem does it solve?

Manage AICP's inference-specific caches to optimize prompt tokenization, KV quantization, and embedding cache performance.

Core Features & Use Cases

• LocalAI prompt cache configuration per model (paths, enable/disable)
• KV cache quantization tuning (k/v types per model)
• Anthropic prompt caching for cloud backends
• Embedding cache management with nomic-embed
• Use cases: reduce tokenization time, fit larger contexts, cut cloud costs with reuse

Quick Start

Edit config/models/<name>.yaml to enable and tune prompt_cache_path, prompt_cache_all, cache_type_k, and cache_type_v, then restart LocalAI to apply changes.