inference-gateway-routing

To configure cascading routing for LLM providers on EKS, you can use this Skill to set up a multi-tiered inference gateway with Kubernetes Gateway API, providing automated fallback options across providers. It applies configurations via kubectl.

Semantic routing for LLM inference allows the dynamic selection of models based on the intent detected in the request. This Skill implements semantic routing within a multi-tiered gateway to direct requests to the appropriate provider.

You propagate OpenTelemetry traces by configuring HTTPRoute resources to manage traffic routing to a Langfuse endpoint. This Skill sets up the necessary OTel trace propagation within the EKS inference gateway for auditing.

Yes, you need a pre-existing EKS cluster to use this Skill. It assumes you already have a Kubernetes and EKS environment deployed, and requires dependencies like kubectl, kgateway, and Bifrost to configure the gateway.

Yes, you can unify multiple LLM providers into a single endpoint while optimizing cost-performance. This Skill uses cascading and semantic routing to route traffic efficiently, which is ideal for e-commerce companies needing integrated LLM infrastructure.