skills.rest
saintgo7
saintgo7COMMUNITY
Explore all skills by this author →
GitHub

fastapi-gateway-pattern

Implement a FastAPI OpenAI-compatible gateway with authentication, quotas, and SSE proxying.

Updated Apr 4, 2026
Search Tags:#fastapi#rate limiting#sse streaming#openai-compatible api#llm gateway#api key auth#quota accounting
One-click install
npx skills add https://github.com/saintgo7/claude-skills --skill fastapi-gateway-pattern
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: fastapi-gateway-pattern
Source: https://github.com/saintgo7/claude-skills/tree/main/fastapi-gateway-pattern
Command: npx skills add https://github.com/saintgo7/claude-skills --skill fastapi-gateway-pattern

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill solves the problem of exposing multiple OpenAI-compatible LLM backends behind a single stable endpoint while enforcing authentication, quotas, streaming, and usage logging without entangling route code with infrastructure concerns.

Core Features & Use Cases

  • OpenAI-compatible gateway front-end: Provide /v1/chat/completions, /v1/models, health, metrics, and admin endpoints while routing requests to upstream backends (vLLM/TGI/llama.cpp/sglang).
  • Five-layer separation: Split responsibilities into Auth middleware, RateLimit middleware, Logging middleware, Routes layer, and Services layer to keep the system extensible and safe.
  • Quota enforcement with observability: Combine RPM (slowapi), concurrent in-flight limits (per-user Semaphore), and daily token caps (DB-based aggregation) plus structured usage inserts after response completion.
  • Streaming SSE pass-through: Proxy upstream SSE responses to clients while intercepting usage chunks to update accounting without breaking client SDK event boundaries.
  • Production-ready DB pool sizing: Apply a verified SQLAlchemy async pool configuration to survive ~50 concurrent connections with fewer QueuePool timeouts.

Quick Start

Use the fastapi-gateway-pattern skill to scaffold a FastAPI gateway project by following the provided SKILL.md structure and then start implementing the auth, quota, streaming proxy, and usage logging layers.

Frequently Asked Questions about fastapi-gateway-pattern

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an OpenAI-compatible FastAPI gateway for multiple LLM backends?▼

To build an OpenAI-compatible FastAPI gateway, you can scaffold a project that exposes /v1/chat/completions and /v1/models endpoints to route requests to upstream backends like vLLM, TGI, or llama.cpp while enforcing authentication, quotas, and logging.

How do I proxy SSE streaming responses in FastAPI while tracking token usage?▼

Proxying SSE streaming responses in FastAPI involves intercepting upstream usage chunks during pass-through to update accounting without breaking client SDK event boundaries. This allows real-time token quota tracking while maintaining continuous streaming to the client.

What is the best way to enforce per-user rate limits and token quotas in an LLM gateway?▼

Enforcing per-user rate limits and quotas in an LLM gateway requires combining RPM limiting via slowapi, asyncio per-user concurrency control using Semaphores, and DB-backed daily token aggregation to cap usage across concurrent requests safely.

Can I use FastAPI to manage API key authentication with salted SHA256 hashing?▼

FastAPI can manage API key authentication by implementing middleware that verifies Bearer key prefixes and validates requests using salted SHA256 hash verification, securing gateway access before routing to upstream backends.

Does SQLAlchemy async engine pool sizing handle 50 concurrent LLM gateway requests?▼

Tuned SQLAlchemy async engine pool settings are specifically configured to survive approximately 50 concurrent connections with fewer QueuePool timeouts, ensuring stable database operations for high-concurrency LLM gateway workloads.

Why does my LLM gateway route code get tangled with authentication and rate limiting logic?▼

LLM gateway route code becomes tangled when infrastructure concerns are mixed with endpoint logic. Applying a five-layer separation of Auth, RateLimit, Logging middleware, Routes, and Services layers keeps the system extensible and safe.

Related Skills

Explore More Software Engineering Skills→
jayll1303jayll1303
17

vllm-tgi-inference

Deploy local vLLM or HuggingFace TGI inference servers with OpenAI-compatible APIs.

Community
Advanced
saintgo7saintgo7

prometheus-fastapi-metrics

Add Prometheus custom metrics to FastAPI applications with a /metrics endpoint.

Community
Intermediate
saintgo7saintgo7

quota-rate-limit-pattern

Implement three-layer API gateway quota and rate limiting with distinct 429 reason codes.

Community
Advanced
BagelHoleBagelHole
46

llm-gateway

Deploy a unified API gateway for LLM traffic with load balancing and rate limiting.

Community
Advanced