coreweave-inference

What problem does it solve?

Host scalable LLM inference endpoints on Kubernetes using vLLM, TGI, and Triton without vendor-specific APIs, enabling flexible deployment models and cost-conscious operations.

Core Features & Use Cases

• Architecture-ready deployments: deploy standard Kubernetes Deployments/Services with optional Knative Serving for scale-to-zero and HPA/KEDA-based autoscaling.
• Flexible serving backends: choose vLLM, TGI, or Triton for multi-model inference with GPU acceleration and shared model storage.
• Operational excellence: robust health checks, traffic splitting, observability, and cost-optimization for GPU-backed inference workloads.

Quick Start

Deploy a Kubernetes-based vLLM/TGI/Triton endpoint and expose it via a Service to begin serving requests.