gke-ai-troubleshooting-node-unresponsive-timeout

Diagnose GKE TPU and GPU nodes stuck in NotReady due to kernel panics or hardware lockups.

23.7k|2.0k|Updated Mar 31, 2026
Ready to run
npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-node-unresponsive-timeout-google
💬 Prompt for ChatGPT / Claude Chat▼
Please help me install this Agent Skill.
Skill: gke-ai-troubleshooting-node-unresponsive-timeout
Source: https://github.com/google/skills/tree/main/skills/cloud/gke-ai-troubleshooting-node-unresponsive-timeout
Command: npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-node-unresponsive-timeout-google

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? When TPU or GPU nodes in GKE stop heartbeating to the control plane, they report NodeStatusUnknown status and pods remain stuck in Terminating, blocking multi-host workload recovery. This Skill systematically identifies whether the failure is caused by kernel panics, hardware errors, preemption, or disabled auto-repair, then guides you to resolution. ## Core Features & Use Cases - Heartbeat Verification: Query Cloud Logging and Monitoring to confirm when the node lost contact with the control plane. - Hardware Diagnosis: Inspect serial port output for kernel panics, NMI lockups, or machine check exceptions. - Auto-Repair Assessment: Verify whether auto-repair is enabled on the node pool and check repair history for failures. - System Event Analysis: Correlate Compute Engine audit logs to identify host errors, preemption, or automatic restarts. - Use Case: A multi-host TPU JobSet workload fails because one node reports NodeStatusUnknown and won't drain. Use this Skill to determine if it's a kernel panic (inspect serial logs), a disabled auto-repair setting (enable it), or insufficient quota (escalate), then apply the appropriate fix. ## Quick Start Provide your GKE cluster name, node pool name, the unresponsive node name, the incident timestamp, and the cluster region or zone, then I will walk you through the five-step diagnostic workflow to identify the root cause and resolution.

Frequently Asked Questions about gke-ai-troubleshooting-node-unresponsive-timeout

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I verify a GKE node is stuck with NodeStatusUnknown status?▼

Query Kubernetes node conditions using kubectl or Cloud Logging to confirm the Ready condition is Unknown with Reason NodeStatusUnknown (Kubelet stopped posting node status). Use the provided LQL query to check Cloud Logging for heartbeat loss around the incident time.

What does a kernel panic look like in GKE node serial port output?▼

Serial port logs show patterns like Fatal Machine check, hung_task: blocked tasks, NMI: Not continuing, or similar kernel freeze indicators. Check serial port 1 output via gcloud or review system logs in Cloud Logging if serial-port-logging is enabled.

How do I enable auto-repair on an unresponsive GKE node pool?▼

For Standard clusters, verify autoRepair is enabled on the node pool using gcloud container node-pools describe. If disabled, enable it per the documentation; for TPU slice node pools, note that the entire node pool is re-created during repair.

Why is my GKE node stuck in NotReady when auto-repair is enabled?▼

Auto-repair can be enabled and still fail due to insufficient quota, pod disruption budgets blocking drain, or a very recent failure (repair waits up to one hour for drain completion before forcing shutdown). Check repair history in the operation logs.

Can I force-delete stuck Terminating pods on an unresponsive node?▼

No. Force deletion bypasses kubelet confirmation, so a replacement pod can start while the old one still runs. Instead, rely on node auto-repair to drain and re-create the node, which safely terminates all pods.

What Compute Engine events indicate hardware failure on a GKE node?▼

Look for compute.instances.hostError (host hardware or software issue), compute.instances.automaticRestart (VM restarted after host error), or compute.instances.preempted (Spot or preemptible VM preempted by Compute Engine).