caching-architecture

Caching reduces LiteLLM-RS latency by storing prior responses across sessions. A multi-tier architecture uses an in-memory L1 cache, Redis L2 cache, and semantic L3 cache to prevent recomputing identical queries, significantly lowering response times and operational costs.

Semantic caching stores and retrieves near-similar LLM gateway responses using a vector-store backend. Unlike exact-match caches, it compares query embeddings in databases like Qdrant, Weaviate, or Pinecone to return relevant previously generated answers without hitting the model again.

To set up multi-tier caching, configure your LiteLLM-RS cache settings, initialize the CacheManager, and wrap your request-response cycle. The system checks the in-memory L1, Redis L2, and semantic L3 caches sequentially before processing a new request and populates them after a successful response.

Yes, LiteLLM-RS caching integrates with Redis for exact-match L2 persistence and vector databases like Qdrant, Weaviate, or Pinecone for L3 semantic caching. This multi-tier approach combines microsecond L1 memory access with scalable persistent storage for high-traffic workloads.

The best way to cache identical user queries is a multi-tier architecture using deterministic keys and TTLs. It checks an in-memory L1 cache with LRU eviction first, then a Redis L2 cache for exact matches, ensuring scalable persistence without recomputing identical requests.

You should avoid semantic caching when query accuracy is critical and near-similar results are unacceptable, as vector-store L3 caching returns approximate matches. Rely on exact-match L1 and L2 caches with TTLs to guarantee deterministic responses for sensitive workloads.