k8s_gpu_pod_troubleshooter

Kubernetes GPU pods get stuck in pending status when HAMi components fail, GPU resources are insufficient, or scheduling policies conflict. This diagnostic workflow checks HAMi scheduler health, device plugin status, and pending pod requests to identify the exact root cause.

Troubleshoot CrashLoopBackOff errors on Kubernetes GPU pods by systematically checking HAMi webhook mutations, device plugin configurations, container runtime settings, and driver mismatches. This workflow isolates component failures and provides actionable remediation guidance.

The HAMi webhook mutates pod specifications during Kubernetes GPU scheduling to enforce allocation policies. Webhook misconfigurations cause pod allocation failures, which this diagnostic workflow detects by verifying webhook status against expected mutation rules.

You can diagnose insufficient GPU resource allocation in HAMi managed clusters by analyzing GPU node capacity and allocatable resources against pending pod requests. This workflow assesses resource availability to determine if scheduling failures stem from true resource scarcity.

The best way to check HAMi device plugin health is running a comprehensive diagnostic workflow that verifies the device plugin status alongside the HAMi scheduler and webhooks. This identifies plugin errors causing GPU pod scheduling conflicts.

Scheduling policy conflicts for Kubernetes GPU pods arise from HAMi webhook misconfigurations, device plugin errors, or mismatched container runtime configurations. This diagnostic workflow systematically checks these components to resolve allocation failures.