gke-ai-troubleshooting-tpu-dynamic-slices-monitoring

Run kubectl describe slice on the slice name and inspect Status.Conditions. A SliceCreationFailed reason means prerequisite validation failed, such as missing nodes or topology mismatch, while FAILED indicates nodes are not in the same reservation block.

A slice stuck in DEACTIVATING usually has a finalizer blocking deletion. You can remove it with a JSON patch on /metadata/finalizers, but this bypasses controller cleanup and may orphan VM or network resources, so confirm before applying.

Single-slice Pods need the cloud.google.com/gke-tpu-slice-topology annotation plus nodeSelectors for gke-tpu-topology, gke-tpu-accelerator, and gke-tpu-slice. Multi-slice JobSets use the alpha.jobset.sigs.k8s.io/exclusive-topology annotation instead of a manual slice selector.

Do not use it for generic GKE node pool creation or standard non-TPU workload management. Those scenarios are covered by general GKE cluster creation and basics skills instead.

First verify no slices remain with kubectl get slice -A, then run gcloud container clusters update with --no-enable-slice-controller, delete the slices.accelerator.gke.io CRD, and remove TPU slice labels from all nodes.