gke-inference

What problem does it solve?

Deploying AI/ML inference workloads on GKE requires deep expertise in GKE GPU/TPU resource configuration, Kubernetes manifest generation, and model server setup, creating significant manual effort and risk of misconfiguration for engineering teams. This Skill eliminates that friction by providing a guided, best-practice-driven workflow for end-to-end inference deployment.

Core Features & Use Cases

• Validated Manifest Generation: Automatically generate production-ready Kubernetes manifests for LLMs (Llama, Gemma, Mistral) and other AI models using Google's Inference Quickstart, with pre-validated model-accelerator-server combinations to avoid invalid configurations.
• Accelerator & Resource Configuration: Select and configure optimal GPU/TPU accelerators, GKE Autopilot ComputeClasses, and resource quotas for inference workloads based on model size, latency requirements, and cost constraints.
• Production Optimization & Autoscaling: Implement GPU-based autoscaling, quantization, batching, and KV cache optimization to maximize throughput and minimize latency for LLM serving, with troubleshooting guidance for common deployment issues. Real-world use case: An ML engineering team can use this Skill to deploy a Gemma 2 9B model for real-time customer support in under an hour, with automatic scaling to handle peak traffic and optimized GPU utilization to reduce cloud costs.

Quick Start

Use the gke-inference skill to generate a validated Kubernetes manifest for deploying your Llama 3 8B model on GKE with vLLM and an NVIDIA L4 accelerator optimized for low-latency inference.