llm-quota-orchestration

Manage multiple Gemini API keys by implementing quota-aware key rotation and a ledger system that parses rate-limit headers to trigger cooldowns. This approach selects keys based on remaining quota or LRU, preventing 429 errors during high request volumes.

Yes, you can failover from Gemini cloud keys to local vLLM instances using tiered resilience. This configuration provides a controlled fail-fast fallback option when cloud API quota exhaustion occurs.

Quota-aware routing parses rate-limit headers from LLM responses to update a quota ledger and manage cooldowns. It selects the optimal API key by evaluating session affinity, remaining quota, or LRU status to route traffic reliably.

Preserve prefix caching during API key rotation by using sticky sessions that pin workflows to a single key. The orchestration layer injects status updates into agent prompts to maintain session affinity and reduce latency.

Yes, quota management works directly with the @google/generative-ai dependency. It is designed for services operating with multiple Gemini API keys that need to avoid 429s and route parallel requests efficiently.

Use a fail-fast option when cloud API keys are exhausted and local vLLM fallbacks are unavailable. This controlled failure mechanism prevents cascading delays in tiered resilience routing during critical quota depletion.