caching-architecture

Implement multi-tier caching with L1, L2, and L3 backends for LiteLLM-RS.

104|19|Updated Jul 15, 2025

Search Tags:#architecture #caching #latency #Redis #TTL #semantic-cache #LiteLLM-RS

Ready to run

npx skills add https://github.com/majiayu000/litellm-rs --skill caching-architecture

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: caching-architecture
Source: https://github.com/majiayu000/litellm-rs/tree/main/.claude/skills/caching-architecture
Command: npx skills add https://github.com/majiayu000/litellm-rs --skill caching-architecture

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

LiteLLM-RS users often incur unnecessary latency and higher costs due to repeated identical requests in high-traffic gateway scenarios. This Skill provides a structured multi-tier caching architecture to transparently reuse results and cut both latency and cost.

Core Features & Use Cases

In-Memory L1 Cache: microsecond latency with LRU eviction to store recent responses.
Redis L2 Cache: exact-match caching with TTL-based expiration for scalable persistence.
Semantic L3 Cache: vector-store backed caching (Qdrant/Weaviate/Pinecone) for near-similar results.
Use Case: a chat gateway that processes thousands of identical user queries per minute can reuse prior responses instead of recomputing.

Quick Start

Configure the LiteLLM-RS cache in your config, initialize the CacheManager with the cache settings, and wrap the request/response cycle to get from cache first, then populate caches after a successful response.

Frequently Asked Questions about caching-architecture

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I reduce latency for repeated LiteLLM-RS requests in a high-traffic gateway?▼

Caching reduces LiteLLM-RS latency by storing prior responses across sessions. A multi-tier architecture uses an in-memory L1 cache, Redis L2 cache, and semantic L3 cache to prevent recomputing identical queries, significantly lowering response times and operational costs.

What is semantic caching and how does it work with LLM gateway responses?▼

Semantic caching stores and retrieves near-similar LLM gateway responses using a vector-store backend. Unlike exact-match caches, it compares query embeddings in databases like Qdrant, Weaviate, or Pinecone to return relevant previously generated answers without hitting the model again.

How do I set up a multi-tier cache for LiteLLM-RS?▼

To set up multi-tier caching, configure your LiteLLM-RS cache settings, initialize the CacheManager, and wrap your request-response cycle. The system checks the in-memory L1, Redis L2, and semantic L3 caches sequentially before processing a new request and populates them after a successful response.

Does LiteLLM-RS caching work with Redis and vector databases like Qdrant?▼

Yes, LiteLLM-RS caching integrates with Redis for exact-match L2 persistence and vector databases like Qdrant, Weaviate, or Pinecone for L3 semantic caching. This multi-tier approach combines microsecond L1 memory access with scalable persistent storage for high-traffic workloads.

What is the best way to cache identical user queries across chat sessions?▼

The best way to cache identical user queries is a multi-tier architecture using deterministic keys and TTLs. It checks an in-memory L1 cache with LRU eviction first, then a Redis L2 cache for exact matches, ensuring scalable persistence without recomputing identical requests.

When should I not use semantic caching for LLM API responses?▼

You should avoid semantic caching when query accuracy is critical and near-similar results are unacceptable, as vector-store L3 caching returns approximate matches. Rely on exact-match L1 and L2 caches with TTLs to guarantee deterministic responses for sensitive workloads.