gke-ai-troubleshooting-tpu-partition-overlap

Diagnose and resolve GKE TPU slice creation failures caused by partition overlap conflicts.

23.7k|2.0k|Updated Mar 31, 2026
Ready to run
npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-tpu-partition-overlap-google
💬 Prompt for ChatGPT / Claude Chat▼
Please help me install this Agent Skill.
Skill: gke-ai-troubleshooting-tpu-partition-overlap
Source: https://github.com/google/skills/tree/main/skills/cloud/gke-ai-troubleshooting-tpu-partition-overlap
Command: npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-tpu-partition-overlap-google

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? When a GKE dynamic TPU slice fails to form with a SliceCreationFailed status, determining whether the conflict is caused by partition overlap requires manual investigation across multiple Kubernetes resources and Cloud Monitoring metrics. This Skill automates the diagnostic workflow and guides you to the resolution. ## Core Features & Use Cases - Confirm Failure Root Cause: Verify SliceCreationFailed status and rule out unrelated causes (missing nodes, topology mismatches, unhealthy partitions). - Identify Conflicting Slices: Cross-reference partition IDs across all Slice custom resources and query Cloud Monitoring metrics to find which slice is using the same nodes. - Route to Resolution: Determine whether Kueue manages the conflicting slice and provide appropriate remediation (wait for release, select alternative partitions, or re-create with retry annotations). - Use Case: Your TPU slice creation is stuck in SliceCreationFailed. Run this Skill to confirm partition overlap, identify the conflicting slice, and determine whether to wait for it to release or select different partitions. ## Quick Start Describe the failing Slice custom resource and provide the cluster name, location, and project ID so I can check partition availability and identify which other slice is blocking these nodes.

Frequently Asked Questions about gke-ai-troubleshooting-tpu-partition-overlap

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What causes GKE TPU slice creation to fail with SliceCreationFailed status?▼

SliceCreationFailed occurs when selected nodes don't exist, topology doesn't match partition count, or requested partitions are already used by another slice. This Skill addresses the partition overlap cause specifically.

How do I find which slice is using the same TPU partitions?▼

List all Slice custom resources with kubectl, describe each to read their Partition IDs, then compare against the failing slice's nodes. Verify using the kubernetes.io/accelerator/partition/state Cloud Monitoring metric queried with partition_id and slice_id labels.

Can I manually delete another slice to free up partitions?▼

No. Never delete a slice managed by Kueue or owned by another workload. Instead, wait for its owner to release it, or select available partitions whose nodes aren't used by any other slice.

How does Kueue handle TPU slice creation and deletion?▼

Kueue automatically creates and manages Slice custom resources for workloads. Kueue-managed slices have labels like kueue.gke.io/tpu-slice-used-by-workload. To release nodes, delete the associated JobSet, not the Slice directly.

What should I do if the slice still fails after the conflicting slice is released?▼

Verify the slice has the annotation slice.gke.io/retry-on-failure: "true" for automatic retry. If missing, delete and re-create the slice with this annotation, or contact Cloud Customer Care if the issue persists.

How are TPU partitions of different shapes allocated on the same physical nodes?▼

Dynamic sub-slicing partitions a 4x4x4 sub-block into smaller topologies (2x4x4, 2x2x4, 2x2x2, 2x2x1) on the same physical node pool. Partitions of different shapes share nodes, so overlap detection must check all partition shapes.