Deploy scalable LLM inference endpoints on Kubernetes with vLLM, TGI, or Triton.
npx skills add https://github.com/tomz/agent-skills --skill coreweave-inference
Please help me install this Agent Skill. Skill: coreweave-inference Source: https://github.com/tomz/agent-skills/tree/main/coreweave-inference Command: npx skills add https://github.com/tomz/agent-skills --skill coreweave-inference
Host scalable LLM inference endpoints on Kubernetes using vLLM, TGI, and Triton without vendor-specific APIs, enabling flexible deployment models and cost-conscious operations.
Deploy a Kubernetes-based vLLM/TGI/Triton endpoint and expose it via a Service to begin serving requests.
High-intent search queries and answers about installing and using this skill.
Host LLM inference endpoints on Kubernetes by deploying standard Deployments and Services with vLLM, TGI, or Triton, using persistent model storage and GPU acceleration. Configure Knative Serving for scale-to-zero capabilities.
Yes, you can use vLLM and Triton together for multi-model inference on Kubernetes. The Skill supports flexible serving backends, allowing you to choose between vLLM, TGI, and Triton based on your specific workload requirements.
Set up autoscaling for GPU-backed inference workloads using HPA or KEDA, optionally integrated with Knative Serving. This enables dynamic scaling to handle fluctuating request volumes while optimizing operational costs.
Optimize costs for Kubernetes LLM serving by utilizing HPA/KEDA autoscaling and Knative scale-to-zero features. Shared persistent model storage across multiple instances further reduces GPU resource overhead.
Yes, you need readiness, liveness, and startup probes for LLM inference deployments. These health checks ensure your Kubernetes Services route traffic only to fully loaded models, preventing request failures during initialization.
Traffic splitting with multi-model inference on Kubernetes routes requests across different model versions or serving backends. This is managed through standard Kubernetes Service configurations and optional Knative Serving integrations.
nebiusProvision Nebius cloud resources and manage IAM authentication for physical-AI workflows.
BagelHoleScale LLM inference clusters on Kubernetes using KEDA and GPU metrics.
iceflowerGuide Kubernetes autoscaling with HPA, VPA, KEDA, and Knative.
tangzheng202202Deploy GPU-based ML workloads on a serverless cloud platform.
