skills.rest
saintgo7
saintgo7COMMUNITY
Explore all skills by this author →
GitHub

multi-llm-routing-pattern

Route LLM requests across multiple backends via an OpenAI-compatible gateway.

Updated Apr 4, 2026
Search Tags:#fastapi#llm routing#canary deployment#api gateway#fallback#vllm#weighted load balancing
One-click install
npx skills add https://github.com/saintgo7/claude-skills --skill multi-llm-routing-pattern
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: multi-llm-routing-pattern
Source: https://github.com/saintgo7/claude-skills/tree/main/multi-llm-routing-pattern
Command: npx skills add https://github.com/saintgo7/claude-skills --skill multi-llm-routing-pattern

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill solves the operational problem of routing requests to multiple LLM backends (vLLM/OpenAI/TGI) through a single OpenAI-compatible gateway without client changes or fragile ad-hoc logic.

Core Features & Use Cases

  • Five routing modes: static model-to-upstream mapping, weighted load balancing, fallback chain, user-plan-based routing, and hash-sticky A/B canary.
  • Production-grade fallback behavior: retries only on 5xx and network/timeout errors while returning 4xx immediately to the client.
  • Operational correctness checks: ensures consistency between upstream_map and the gateway’s served model list to prevent “/v1/models lies” UI mismatches.
  • Use case examples: send free users to a smaller model, run canary traffic for a new model safely, and fail over to a warm spare backend when one instance degrades.

Quick Start

Use the multi-llm-routing-pattern skill to add static model routing for qwen2.5-coder-32b and qwen3-coder-30b behind a single OpenAI-compatible endpoint, then enable fallback for 5xx/timeout only.

Frequently Asked Questions about multi-llm-routing-pattern

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I route requests to multiple vLLM backends behind an OpenAI-compatible gateway?▼

Route requests to multiple vLLM backends by mapping request model identifiers or user plan attributes to upstream instances, enabling weighted load balancing and fallback chains without requiring client changes.

How does LLM API gateway fallback handle 4xx and 5xx errors?▼

LLM API gateway fallback retries exclusively on 5xx and network/timeout errors while returning 4xx client errors immediately to prevent invalid request failover loops.

Can I run canary deployments for a new LLM model through an API gateway?▼

Run canary deployments for a new LLM model through an API gateway using deterministic, hash-sticky A/B routing to safely distribute traffic weights without breaking session consistency.

How do I ensure my LLM gateway's served model list matches the upstream map?▼

Ensure your LLM gateway's served model list matches the upstream map through operational correctness checks that prevent UI mismatches when serving multiple backend instances.

What is the best way to route free users to a smaller LLM model?▼

Route free users to a smaller LLM model by applying user-plan-based routing attributes within your API gateway to statically map request paths to designated upstream backends.

Why does my weighted load balancing configuration fail when an upstream LLM degrades?▼

Weighted load balancing configurations fail when upstream LLMs degrade if fallback chains are not configured to automatically fail over traffic to a warm spare backend instance.

Related Skills

Explore More Software Engineering Skills→
saintgo7saintgo7

fastapi-gateway-pattern

Implement a FastAPI OpenAI-compatible gateway with authentication, quotas, and SSE proxying.

Community
Advanced
GhostOf0daysGhostOf0days

load-balancing

Implement load balancing workflows for quantitative research with deterministic diagnostics.

Community
Advanced
jaredcroxtonjaredcroxton

serving-llms-vllm

Deploy OpenAI-compatible LLM inference servers with vLLM and quantization.

Community
Intermediate
dmwin72015dmwin72015
1

golang-samber-slog-****

Compose samber/slog pipelines for sampling, sanitizing, and routing Go logs.

Community
Advanced