gke-ai-troubleshooting-tpu-performance-degradation

Diagnose GKE Cloud TPU training throughput drops using ML Diagnostics Workload Monitoring and system metrics.

23.7k|2.0k|Updated Mar 31, 2026
Ready to run
npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-tpu-performance-degradation-google
💬 Prompt for ChatGPT / Claude Chat▼
Please help me install this Agent Skill.
Skill: gke-ai-troubleshooting-tpu-performance-degradation
Source: https://github.com/google/skills/tree/main/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation
Command: npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-tpu-performance-degradation-google

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? When Cloud TPU training jobs on Google Kubernetes Engine experience sudden throughput drops or step-time regressions (15%+ TPU duty-cycle loss), identifying whether the issue stems from workload resource saturation, hardware throttling, or network fabric faults requires correlating multiple diagnostic sources. This Skill automates that investigation by analyzing ML Diagnostics monitored events and 1-minute Cloud Monitoring metrics to pinpoint the root cause and route remediation accordingly. ## Core Features & Use Cases - ML Diagnostics Workload Monitoring Integration: Query MonitoredEvent analyzer reports (HBM capacity, host memory, CPU utilization, thermal/power throttling, network fabric) to classify performance degradation by category. - System Metrics Correlation: Run PromQL queries over 1-minute Cloud Monitoring metrics (kubernetes_io:node_accelerator_duty_cycle, memory utilization ratios, CPU allocatable utilization) to corroborate analyzer findings. - Instance-to-Node Mapping: Convert Compute Engine instance IDs reported by infrastructure analyzers to GKE node names and physical topology blocks for targeted remediation. - Use Case: A multi-slice JAX training job running on GKE TPUs shows a 20% duty-cycle drop. Use this Skill to determine whether HBM memory is exhausted (requiring batch size reduction), host CPU is saturated (requiring loader optimization), or a TPU node is experiencing thermal throttling (requiring Google Cloud Support escalation). ## Quick Start Provide the project ID, GKE cluster name, workload name, and the timestamp when throughput degradation was observed, then follow the diagnostic workflow to query ML runs, list monitored events, and inspect analyzer reports to route to the correct remediation path.

Frequently Asked Questions about gke-ai-troubleshooting-tpu-performance-degradation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I query ML Diagnostics monitored events for TPU performance degradation?▼

Use `gcloud alpha mldiagnostics monitored-events list` to list `PERFORMANCE_DEGRADATION` events, then `gcloud alpha mldiagnostics monitored-events describe` to inspect the `analyzerReports` array, which contains the detector type (HBM, host memory, CPU, thermal, ICI, network), detection state, and recommended actions.

What Cloud Monitoring metrics should I check when TPU training throughput drops?▼

Query `kubernetes_io:node_accelerator_duty_cycle` to confirm the 15%+ drop, then check `kubernetes_io:node_accelerator_memory_used` divided by `kubernetes_io:node_accelerator_memory_total` for HBM saturation, and `kubernetes_io:node_memory_allocatable_utilization` and `kubernetes_io:node_cpu_allocatable_utilization` for host resource saturation.

How do I map Compute Engine instance IDs from analyzer reports to GKE nodes?▼

Run `kubectl get nodes -l cloud.google.com/gke-tpu-accelerator` with the `-o jsonpath` flag to extract `container.googleapis.com/instance_id`, `gce-topology-block`, `gce-topology-subblock`, and `gce-topology-host` labels for matching instance IDs to node names and physical topology.

When should I escalate a TPU performance issue to Google Cloud Support?▼

Escalate when an infrastructure analyzer (thermal, power, ICI, memory bandwidth, or network) reports `detectionState: "DETECTED"` on specific instances, as these indicate potential hardware faults that require vendor investigation and possible node replacement.

What workload optimizations address HBM memory saturation detected by ML Diagnostics?▼

Capture XProf on-demand profiles using `gcloud alpha mldiagnostics profiler-session capture` to analyze HBM usage patterns, then reduce batch size or adjust model parameters to fit within available HBM capacity without spilling to host memory.

Does ML Diagnostics Workload Monitoring support frameworks other than JAX on TPUs?▼

No, Workload Monitoring is currently limited to JAX on Google Cloud TPUs. PyTorch or other frameworks will not generate ML runs or monitored events in the ML Diagnostics system.