inference-gateway-routing

What problem does it solve?

This Skill configures and routes traffic through a two-tier inference gateway on EKS, integrating multiple LLM providers, and ensuring efficient inference with cascading and semantic routing.

Core Features & Use Cases

• Cascading Routing: Implements a cascade of LLM providers to provide fallback options.
• Semantic Routing: Allows the selection of models based on the intent detected.
• HTTPRoute with Trace Propagation: Manages traffic routing to a Langfuse endpoint while enabling OpenTelemetry for trace propagation.
• Use Case: An e-commerce company using multiple LLM providers wants to integrate them into a unified endpoint while ensuring the best cost-performance and maintaining traces for auditing.

Quick Start

Create a routing configuration to route requests to LLM providers using 'kubectl apply -f path/to/config.yaml'