What problem does it solve?
TPU v6e nodes on GKE can suffer vbar_control_agent segfaults, memory cgroup OOM kills, and TPU device initialization failures caused by race conditions during device resets or high-frequency metrics polling. This Skill provides a systematic diagnostic and resolution workflow to identify the root cause from Cloud Logging and cluster state.
Core Features & Use Cases
- Serial Console OOM Detection: Query serial console logs for memory cgroup OOM kills of vbar_control_agent using prebuilt Cloud Logging filter templates.
- Metrics Corruption Analysis: Detect tpu-device-plugin checksum mismatch errors indicating vBAR memory corruption.
- Custom Metrics Conflict Detection: Identify custom TPU metrics collectors (e.g., libtpu.sdk.tpumonitoring) that trigger the race condition, then recommend mitigations.
- Use Case: A training job on a TPU v6e cluster fails after a node reboot. Use this Skill to compute the incident time window, query serial console and container logs, confirm the vbar_control_agent crash, and advise disabling the custom metrics collector until the GKE resiliency fix ships.
Quick Start
Diagnose why my TPU v6e node in cluster tpu-cluster-prod crashed around 2026-04-14T20:00:00Z by checking for vbar_control_agent OOMs and tpu-device-plugin errors.
