gke-workload-scaling-troubleshooting

Diagnose and resolve HorizontalPodAutoscaler failures and metric pipeline breakage on GKE.

23.7k|2.0k|Updated Mar 31, 2026
Ready to run
npx skills add https://github.com/google/skills --skill gke-workload-scaling-troubleshooting-google
💬 Prompt for ChatGPT / Claude Chat▼
Please help me install this Agent Skill.
Skill: gke-workload-scaling-troubleshooting
Source: https://github.com/google/skills/tree/main/skills/cloud/gke-workload-scaling-troubleshooting
Command: npx skills add https://github.com/google/skills --skill gke-workload-scaling-troubleshooting-google

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? When a GKE HorizontalPodAutoscaler (HPA) stops scaling workloads or reports metric errors like <unknown> values and FailedGet* events, debugging requires deep knowledge of the metrics pipeline, Pod resource requests, and HPA configuration—work that is time-consuming and error-prone. This Skill systematically diagnoses the root cause and guides you to the fix. ## Core Features & Use Cases - HPA Metric Diagnosis: Identifies whether failures stem from missing Pod resource requests, broken custom/external metrics adapters, or invalid HPA configuration. - Metrics Pipeline Validation: Tests connectivity from the HPA controller through the metrics API to Cloud Monitoring, exposing control-plane firewall blocks or adapter crashes. - Scale-to-Zero Troubleshooting: Resolves issues preventing workloads from scaling to or from zero replicas on GKE 1.37+. - Use Case: Your deployment won't scale up despite high CPU usage. The HPA shows ScalingActive: False with reason FailedGetResourceMetric. This Skill guides you through checking Pod resource requests, verifying the metrics server, and confirms the exact manifest correction needed. ## Quick Start Describe the HPA error you're seeing (for example, metrics show unknown, won't scale up, or FailedGetScale events), the cluster name, and the workload namespace, and this Skill will provide step-by-step diagnostics and a corrected HPA or workload manifest to review.

Frequently Asked Questions about gke-workload-scaling-troubleshooting

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
Why does my GKE HPA show metrics as unknown?▼

Unknown metrics typically result from missing Pod resource requests (for CPU/memory metrics), a broken custom-metrics adapter (not running or not registered as an APIService), or control-plane firewall rules blocking the adapter's port. Verify Pod requests exist, confirm the adapter Pods are running and healthy, and check that the APIService is marked AVAILABLE.

How do I fix FailedGetResourceMetric errors on an HPA?▼

This error means the HPA cannot compute utilization percentages because at least one container lacks a resource request for the metric being scaled (CPU or memory). Add resource.requests for the scaled metric to every container in the Pod spec, including sidecars, then redeploy.

What causes FailedGetScale events in HPA?▼

FailedGetScale occurs when the HPA's scaleTargetRef doesn't resolve to an existing scalable workload. Verify the target name, kind (Deployment, StatefulSet, ReplicaSet), and apiVersion match exactly, and confirm it exists in the same namespace as the HPA.

How do I test if my custom-metrics adapter is working?▼

Query the metrics API directly using kubectl get --raw to bypass the HPA and test the entire pipeline. If the adapter is unavailable (Service Unavailable error), the control-plane firewall is blocking it; add the adapter's targetPort to the control-plane firewall rule.

Why won't my HPA scale down even though the metric is low?▼

Scale-down failure often stems from multiple metrics (HPA uses the highest), a missing metric showing unknown (which conservatively halts scale-down), or the default 5-minute stabilization window. Verify all metrics report valid values, use gauge metrics instead of rate-based metrics that stop reporting at zero, and reduce stabilizationWindowSeconds if faster scale-down is needed.

How do I enable scale-to-zero for a GKE workload?▼

Scale-to-zero requires GKE 1.37+, minReplicas: 0 in the HPA, and at least one External or Object metric (CPU/memory metrics cannot drive zero scaling). All metrics must evaluate to zero before the workload scales down; any unknown metric pauses the transition.