gpu-autoscaling-engineer

Author and validate KEDA ScaledObjects and GPU node-pool policies for Kubernetes inference workloads.

Updated Jul 16, 2026
One-click install
npx skills add https://github.com/Cloud-Byte-Consulting/plugins --skill gpu-autoscaling-engineer-cloud-byte-consulting
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: gpu-autoscaling-engineer
Source: https://github.com/Cloud-Byte-Consulting/plugins/tree/main/gpu-research-platform/skills/gpu-autoscaling-engineer
Command: npx skills add https://github.com/Cloud-Byte-Consulting/plugins --skill gpu-autoscaling-engineer-cloud-byte-consulting

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? GPU inference services on Kubernetes often scale on the wrong signals (CPU), thrash expensive GPU nodes, or leave pods Pending because the pod-scaling and node-scaling loops are misconfigured. This Skill designs both loops together so capacity follows demand without paying for idle accelerators. ## Core Features & Use Cases - Dual-trigger ScaledObjects: Authors KEDA ScaledObjects combining a demand metric (RPS, queue depth) with an experience or GPU metric (p95 latency, TTFT, DCGM_FI_DEV_GPU_UTIL), including scale-to-zero and asymmetric scale-up/scale-down behavior. - GPU node-pool policy: Designs tainted dedicated GPU pools with aggregate GPU-count limits, WhenEmpty consolidation warm pools, startup taints, and cold-start engineering via image pre-caching. - YAML review and validation: Reviews existing ScaledObject/HPA configurations for GPU services and validates the full chain from metric trigger to node provisioning. - Use Case: Your LLM inference pods scale up but new replicas sit Pending for minutes while GPU nodes provision. Use this Skill to tune stabilization windows, configure a warm pool with consolidateAfter, and add a queue-depth trigger so scale-up completes before users notice. ## Quick Start Use the gpu-autoscaling-engineer skill to author a KEDA ScaledObject with scale-to-zero and dual Prometheus triggers for my llm-inference deployment.

Frequently Asked Questions about gpu-autoscaling-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I autoscale GPU inference pods on Kubernetes with KEDA?

Create a KEDA ScaledObject targeting your inference Deployment with two Prometheus triggers: one demand metric like request rate and one experience metric like p95 latency or DCGM_FI_DEV_GPU_UTIL. Set asymmetric scaling behavior with instant scale-up and a stabilization window for scale-down.

What metrics should I use to scale LLM inference workloads?

Scale on request rate, p95 latency, TTFT, tokens-per-second throughput, or queue depth rather than CPU utilization. Use DCGM_FI_DEV_GPU_UTIL only as a secondary guard trigger since it lags demand and reads misleadingly on shared GPUs.

Why do my GPU pods scale up but stay Pending?

Pending pods mean the node autoscaler is not provisioning GPU nodes, usually due to missing tolerations for the GPU pool taint, exhausted aggregate GPU limits, or quota constraints. Validate the full chain from ScaledObject through node-pool configuration, not the YAML in isolation.

Does KEDA scale-to-zero work for HTTP inference services?

Scale-to-zero works but the first request after activation needs a buffer, otherwise callers get errors instead of latency. Use the KEDA HTTP add-on to intercept and hold traffic during 0-to-1 activation, or place a queue in front of the service.

How do I prevent GPU nodes from scaling down too aggressively?

Use Karpenter consolidationPolicy WhenEmpty with consolidateAfter set to around 5 minutes so just-emptied GPU nodes stay warm for the next pod. Keep the scale-down stabilization window longer than measured node provision time to avoid thrashing expensive nodes.

Should I use HPA or KEDA for GPU workload autoscaling?

KEDA is preferred because it natively supports Prometheus external metrics, scale-to-zero, and multiple triggers while still driving a standard HPA under the hood. Plain HPA alone cannot scale to zero and requires additional plumbing for custom metrics.