gke-ai-troubleshooting-tpu-metrics-monitoring

Query the kubernetes.io/container/accelerator/duty_cycle metric in Cloud Monitoring to see the percentage of time TensorCores were actively processing. For GKE 1.28.1-gke.1066000 or later, tensorcore_utilization and memory_bandwidth_utilization metrics are also available.

Query kubernetes_io:node_status_condition with condition="Ready" and status="True" filtered by cluster and node name. This requires GKE version 1.32.1-gke.1357001 or later to expose node status condition metrics.

TPU runtime metrics require GKE version 1.27.4-gke.900 or later with system metrics enabled. The TPU container must expose port 8431 for Prometheus scraping, and JAX workloads need version 0.4.14 or later.

A duty_cycle or tensorcore_utilization value below 20% during active training indicates heavy underutilization, typically caused by data pipeline bottlenecks or small batch sizes. Check input data loading performance before investigating hardware.

Use PromQL to divide the sum of node_pool_accelerator_times_to_recover_sum by its count over 7 days for MTTR. For MTBI, divide total node uptime by the sum of node_interruption_count over the same window.

This approach does not cover general non-TPU GKE workload monitoring or non-metric TPU debugging such as log analysis or code-level profiling. Use it only for infrastructure-level diagnosis via GKE system metrics.