litellm-production

What problem does it solve?

This Skill addresses the common production issues of LiteLLM, such as TCP churn, file-descriptor exhaustion, high latency, and cost overruns, by implementing a shared aiohttp session for connection pooling, OTel tracing, and Redis/disk cache tiering.

Core Features & Use Cases

• Connection Pooling: Reduces TCP churn and file-descriptor exhaustion.
• Response Caching: Improves latency and reduces cost by caching LLM responses.
• OTel Tracing: Enables detailed monitoring of LLM calls.
• Use Case: Deploy a service that makes high-throughput LLM calls, integrate LiteLLM into a FastAPI app, or configure LLM response caching for production services.

Quick Start

Initialize litellm with the production configuration to enable connection pooling, caching, and OTel tracing.