gpu-workload-troubleshooter

Insufficient nvidia.com/gpu means the scheduler sees no allocatable GPUs. Check node allocatable resources with kubectl get nodes; if a GPU node shows zero, the NVIDIA device plugin DaemonSet is missing, unscheduled, or crashed on that node.

Add a toleration for the GPU pool taint to your pod spec, typically key nvidia.com/gpu with operator Exists and effect NoSchedule. GPU pools taint nodes deliberately, so every GPU pod needs the matching toleration; never remove the taint itself.

OOMKilled with exit code 137 means the container exceeded its host RAM limit, fixed by raising memory limits sized for load-time peaks. CUDA out of memory is VRAM exhaustion inside the GPU, fixed with smaller batches, quantization, or a larger GPU or MIG slice.

The node likely never registered the nvidia.com/gpu resource, shown as Initialized False with gpu resource not registered on the NodeClaim. Causes include a node image without the NVIDIA driver or toolkit, or the device plugin DaemonSet not landing on the new node.

Exit 137 with liveness probe failures means the kubelet is killing a healthy but slow server while it loads large model weights. Add a startupProbe budgeted to worst-case load time, such as periodSeconds 10 with failureThreshold 60, while keeping liveness tight afterward.

On managed Kubernetes, conditions like AcceleratedHardwareReady False or DCGM XID errors indicate hardware faults the provider must replace. Cordon and drain the node, collect node name, XID errors, and timestamps, then open a ticket rather than reinstalling drivers yourself.