skills.rest
google
googleOFFICIAL
Explore all skills by this author →
Website𝕏@GoogleOSSGitHub

gke-ai-troubleshooting-handle-disruption-gpu-tpu

Diagnoses and mitigates GKE node disruptions from host maintenance on GPU and TPU workloads.

19.1k|1.5k|Updated Mar 31, 2026
Search Tags:#GPU#Troubleshooting#PromQL#gke#TPU#node disruption#host maintenance
One-click install
npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-handle-disruption-gpu-tpu
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gke-ai-troubleshooting-handle-disruption-gpu-tpu
Source: https://github.com/google/skills/tree/main/skills/cloud/gke-ai-troubleshooting-handle-disruption-gpu-tpu
Command: npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-handle-disruption-gpu-tpu

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

GPU and TPU workloads on GKE can be unexpectedly interrupted by Compute Engine host maintenance events, and engineers struggle to determine whether a disruption came from the platform or from their own application. This Skill provides a structured diagnostic workflow to confirm or rule out host maintenance and apply the right mitigations.

Core Features & Use Cases

  • Scheduled Maintenance Detection: Checks node labels for upcoming maintenance windows using kubectl.
  • PromQL-Based Investigation: Queries node interruption metrics filtered by HW/SW Maintenance reason in Cloud Monitoring.
  • Log and Taint Analysis: Inspects Cloud Logging entries and node taints to confirm active maintenance and node cordoning.
  • Use Case: A training job on a TPU nodepool crashes overnight. Use this Skill to verify whether a host maintenance event caused the disruption, then configure graceful termination, opportunistic maintenance, and PodDisruptionBudgets to protect future runs.

Quick Start

Ask the agent to investigate why my GPU workload on GKE cluster my-cluster in us-central1 was disrupted at a specific timestamp.

Frequently Asked Questions about gke-ai-troubleshooting-handle-disruption-gpu-tpu

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if GKE node disruption was caused by host maintenance?▼

Check node labels for cloud.google.com/scheduled-maintenance-time using kubectl, then query the kubernetes_io:node_interruption_count metric filtered by interruption_reason HW/SW Maintenance in Cloud Monitoring. Confirm with Cloud Logging entries showing active-node-maintenance set to ONGOING.

How to protect GPU workloads on GKE from maintenance disruptions?▼

Apply three mitigations together: set spec.terminationGracePeriodSeconds up to 60 minutes for graceful checkpointing, enable opportunistic maintenance so updates trigger when nodes are idle, and configure PodDisruptionBudgets to maintain minimum available replicas during evictions.

What PromQL query shows GKE node interruption events?▼

Use sum by interruption_type and interruption_reason over kubernetes_io:node_interruption_count with monitored_resource k8s_node and interruption_reason HW/SW Maintenance. A nodepool-level variant uses kubernetes_io:node_pool_interruption_count filtered by node_pool_name.

Does this troubleshooting approach work for non-GPU GKE workloads?▼

The workflow targets GPU and TPU nodepools where host maintenance causes full VM interruptions. It is not intended for general cluster creation, network policy configuration, or standard non-accelerator workload deployment issues.

Why did my GKE node get the impending-node-termination taint?▼

GKE applies the cloud.google.com/impending-node-termination:NoSchedule taint to cordon a node before host maintenance terminates it, preventing new Pods from scheduling there. You should not tolerate this taint; instead configure graceful termination handling.

Related Skills

Explore More Software Engineering Skills→
primatrixprimatrix

apply-resource

Manage GKE TPU Nodepool resources via XPK commands.

Official
Intermediate
gke-labsgke-labs
42

gke-reliability

Configure and validate high availability settings for GKE clusters and Kubernetes workloads.

Official
Intermediate
lloydchanglloydchang
2

health-check-monitor

Monitor Kubernetes cluster health across nodes, pods, and quotas.

Community
Advanced
DoanNgocCuongDoanNgocCuong

skypilot-multi-cloud-orchestration

Orchestrate machine learning workloads across multiple cloud providers with SkyPilot.

Community
Advanced