skills.rest
Riku-KANO
Riku-KANOCOMMUNITY
Explore all skills by this author →
GitHub

gke-inference-gateway

Configure and debug multi-model GKE Inference Gateway deployments with Body-Based Router and per-pool Endpoint Pickers.

Updated Apr 20, 2026
Search Tags:#gke#gateway-api#httproute#inference-gateway#bbr#inferencepool#endpoint-picker
One-click install
npx skills add https://github.com/Riku-KANO/gemma4-gke-demo --skill gke-inference-gateway
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gke-inference-gateway
Source: https://github.com/Riku-KANO/gemma4-gke-demo/tree/main/.claude/skills/gke-inference-gateway
Command: npx skills add https://github.com/Riku-KANO/gemma4-gke-demo --skill gke-inference-gateway

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill consolidates expert knowledge on GKE Inference Gateway, including the correct API groups (inference.networking.k8s.io), the role of BBR, per-pool Endpoint Pickers, and safe migration paths away from deprecated InferenceModel to modern constructs.

Core Features & Use Cases

  • API group guidance: explains the two API groups (stable v1 and alpha v1alpha2) and when to use each.
  • BBR & HTTPRoute-based routing: shows how body→pool dispatch is done by BBR and how to map models to HTTPRoute header rules.
  • Operational patterns: how to deploy multi-model stacks, per-pool EPP, health checks, timeouts, and troubleshooting steps.
  • Uses example sequences from the real Gemma + vLLM deployment to visualize end-to-end flow.

Quick Start

Follow this skill to configure or debug GKE Inference Gateway using BBR, HTTPRoute header-based routing, and per-pool EPP.

Frequently Asked Questions about gke-inference-gateway

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I route traffic to multiple models behind a single GKE Inference Gateway?▼

Multi-model routing on GKE Inference Gateway uses Body-Based Router with HTTPRoute header matching to dispatch traffic to separate InferencePool resources. You configure BBR via GCPRoutingExtension to map body payloads to specific pools, enabling per-model routing through a single gateway entry point.

What is the difference between InferenceModel and BBR with HTTPRoute on GKE?▼

InferenceModel is a deprecated construct for GKE Inference Gateway, while BBR with HTTPRoute header matching is the modern replacement. Migrating to BBR involves wiring it via GCPRoutingExtension and mapping models to HTTPRoute rules, replacing the older InferenceModel API with current body-based dispatch logic.

How do I configure per-pool Endpoint Pickers for vLLM on GKE?▼

Per-pool Endpoint Pickers on GKE Inference Gateway are configured within each InferencePool to enable prefix-cache-aware routing for vLLM. You must validate EPP usage, health checks, timeouts, and gateway naming consistency to ensure correct API groups under inference.networking.k8s.io.

When should I use v1 vs v1alpha2 API groups for GKE Inference Gateway?▼

Use stable v1 API groups for production GKE Inference Gateway deployments, and v1alpha2 for alpha features like advanced BBR configurations. The skill explains both API groups under inference.networking.k8s.io and provides guidance on when to apply each based on your routing requirements.

How do I troubleshoot HTTPRoute dispatch issues with multi-model vLLM deployments?▼

Troubleshooting HTTPRoute dispatch on GKE Inference Gateway involves verifying BBR wiring through GCPRoutingExtension, checking per-pool EPP health checks, and validating timeout configurations. Ensure gateway naming consistency and confirm API groups match inference.networking.k8s.io to resolve routing failures.

Related Skills

Explore More Software Engineering Skills→
fajshahfajshah
1

gke-multi-cloud-k8s

Provision and manage multi-cloud GKE clusters with kubectl connectivity.

Community
Advanced
jamesogunsanjamesogunsan

istio-traffic-management

Configure Istio VirtualService, DestinationRule, and Gateway policies for traffic routing.

Community
Advanced
thc1006thc1006

gateway-api

Configure and deploy Kubernetes Gateway API with Envoy Gateway.

Community
Intermediate
totallyGregtotallyGreg
2

kgateway

Configure Kubernetes Gateway API resources with kgateway for Envoy-based routing.

Community
Advanced