skills.rest
google
googleOFFICIAL
Explore all skills by this author →
Website𝕏@GoogleOSSGitHub

gke-ai-troubleshooting-jobset-interruption

Diagnoses GKE JobSet interruptions, preemptions, and restarts for AI/ML training workloads.

19.1k|1.5k|Updated Mar 31, 2026
Search Tags:#kubernetes#Troubleshooting#Diagnostics#gke#preemption#AI training#jobset
One-click install
npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-jobset-interruption
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gke-ai-troubleshooting-jobset-interruption
Source: https://github.com/google/skills/tree/main/skills/cloud/gke-ai-troubleshooting-jobset-interruption
Command: npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-jobset-interruption

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Large-scale AI/ML training jobs on GKE fail silently or restart in loops due to Spot VM preemptions, faulty physical hosts, or worker communication hangs, and finding the root cause requires correlating metrics and logs across many layers.

Core Features & Use Cases

  • Structured Diagnostic Workflow: Walks through JobSet restart counts, nodepool interruption events, node-to-host topology correlation, and pod/worker log analysis using ready-made MQL, PromQL, and LQL queries.
  • Failure Signature Library: Provides real log and metric patterns for Spot preemptions, host hardware failures, NCCL timeouts, and unschedulable pods to speed up root-cause matching.
  • Resolution Guidance: Recommends concrete fixes such as switching to on-demand VMs, compact placement policies, or quarantining faulty GCE host VMs.
  • Use Case: A training JobSet keeps restarting every few hours. Use this Skill to determine whether Spot preemptions or a bad physical host are responsible, then cordon the node and recreate the VM.

Quick Start

Ask the agent to diagnose why my JobSet training workload on my GKE cluster keeps restarting and identify whether preemptions or host failures are the cause.

Frequently Asked Questions about gke-ai-troubleshooting-jobset-interruption

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I troubleshoot GKE JobSet restarts for AI training workloads?▼

Query the kube_jobset_restarts metric filtered by cluster and JobSet name to confirm restart loops, then check nodepool interruption events, node readiness, and worker container logs in sequence. The Skill provides ready-made MQL, PromQL, and LQL queries for each step.

How to detect Spot VM preemptions causing GKE training interruptions?▼

Query the kubernetes.io/node_pool/interruption_count metric grouped by interruption type and reason, and search gke_nodepool logs for PreemptionEvent entries. High preemption counts indicate Spot VMs are being reclaimed, and switching to on-demand or reserved VMs is recommended.

What prerequisites are needed for GKE JobSet troubleshooting?▼

The JobSet metrics package must be enabled in kube-state-metrics, and Cloud Logging plus Cloud Monitoring must be enabled for the Google Cloud project. Without these, the diagnostic queries cannot retrieve restart, interruption, or pod status data.

Why does my GKE training job hang with NCCL timeout errors?▼

NCCL collective communication timeouts typically indicate a failed worker, network packet drops, or a faulty physical host. Inspect the coordinator worker container logs for NCCL WARN timeout messages and correlate failing nodes with their GCE host IDs.

When should I not use this JobSet troubleshooting approach?▼

This workflow is specific to JobSet-based AI/ML workloads and does not apply to general GKE cluster creation, basic workload deployment, or non-JobSet application issues. Use standard GKE diagnostics for those scenarios.

Related Skills

Explore More Software Engineering Skills→
ssmleossmleo

gke-observability

Set up GKE logging, monitoring, and Prometheus integration.

Community
Advanced
wangx7wangx7

gke-compute-classes

Configure, optimize, and troubleshoot GKE ComputeClasses for workload scheduling.

Community
Advanced
AstorYHAstorYH
15

pod-status

Retrieve and report the current status of a specified Kubernetes Pod.

Community
Basic
ssmleossmleo

gke-batch-hpc

Automate batch and HPC workloads on GKE using Kubernetes jobs, JobSets, Kueue, and the MPI Operator.

Community
Advanced