fastapi-gateway-pattern

What problem does it solve?

This Skill solves the problem of exposing multiple OpenAI-compatible LLM backends behind a single stable endpoint while enforcing authentication, quotas, streaming, and usage logging without entangling route code with infrastructure concerns.

Core Features & Use Cases

• OpenAI-compatible gateway front-end: Provide /v1/chat/completions, /v1/models, health, metrics, and admin endpoints while routing requests to upstream backends (vLLM/TGI/llama.cpp/sglang).
• Five-layer separation: Split responsibilities into Auth middleware, RateLimit middleware, Logging middleware, Routes layer, and Services layer to keep the system extensible and safe.
• Quota enforcement with observability: Combine RPM (slowapi), concurrent in-flight limits (per-user Semaphore), and daily token caps (DB-based aggregation) plus structured usage inserts after response completion.
• Streaming SSE pass-through: Proxy upstream SSE responses to clients while intercepting usage chunks to update accounting without breaking client SDK event boundaries.
• Production-ready DB pool sizing: Apply a verified SQLAlchemy async pool configuration to survive ~50 concurrent connections with fewer QueuePool timeouts.

Quick Start

Use the fastapi-gateway-pattern skill to scaffold a FastAPI gateway project by following the provided SKILL.md structure and then start implementing the auth, quota, streaming proxy, and usage logging layers.