skills.rest
ai-dynamo
ai-dynamoOFFICIAL
Explore all skills by this author →
WebsiteGitHub

aiperf-kube-triage

Diagnose stuck, failing, or erroring AIPerf benchmark jobs on Kubernetes from CR status, pod state, and logs.

631|166|Updated Apr 16, 2025
Search Tags:#benchmarking#kubernetes#diagnosis#kubectl#Troubleshooting#aiperf#kueue
One-click install
npx skills add https://github.com/ai-dynamo/aiperf --skill aiperf-kube-triage
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: aiperf-kube-triage
Source: https://github.com/ai-dynamo/aiperf/tree/main/.agents/skills/aiperf-kube-triage
Command: npx skills add https://github.com/ai-dynamo/aiperf --skill aiperf-kube-triage

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

AIPerf benchmarks running on Kubernetes can get stuck Pending, crash-loop, OOM-kill, stall mid-run, or fail with opaque errors, and finding the root cause requires knowing which of the many CR status fields, pod containers, and log streams actually matter for each failure phase.

Core Features & Use Cases

  • Phase-based triage routing: Classify an AIPerfJob or AIPerfSweep by status.phase (Pending, Queued, Initializing, Running, Failed) and follow the matching diagnostic branch with kubectl and aiperf kube debug commands.
  • Failure pattern tables: Map status.error strings, scheduling messages, exit codes, and error-rate bands to concrete root causes and fixes, including the nineteen immutable spec fields that force delete-and-recreate instead of patching.
  • Use Case: A benchmark shows Running but requestsCompleted is flat. The skill directs you to test endpoint reachability from inside the cluster, grep worker logs for the credit-return channel warning, and check for CPU starvation of the records-manager before blaming the endpoint.

Quick Start

Ask the assistant to triage why your AIPerfJob named my-benchmark in namespace aiperf is stuck and not making progress.

Frequently Asked Questions about aiperf-kube-triage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug a stuck AIPerf benchmark on Kubernetes?▼

First read the CR phase with kubectl get aiperfjob and run aiperf kube debug for pod states, events, and log slices. Then follow the branch for that phase: Pending means unschedulable pods, Queued means Kueue has not admitted the workload, and Running with flat requestsCompleted means a stalled benchmark.

Why is my AIPerfJob stuck in Pending or Queued?▼

Pending means pods are unschedulable; check the PodScheduled condition for Insufficient cpu/memory, GPU, taint, or quota messages. Queued means Kueue has not admitted the workload; inspect the Workload object's QuotaReserved and Admitted conditions against ClusterQueue quotas.

Can I patch an AIPerfJob spec to fix scheduling or resources?▼

No. Nineteen spec fields including image, benchmark, resourceMode, podTemplate, and scheduling are immutable after creation, so any placement, resource, or workload fix requires deleting and recreating the CR. Only timeoutSeconds and cancel accept in-place patches.

Why was my AIPerf pod OOMKilled and how do I fix it?▼

The default resourceMode is burstable with requests only and no limits, so an OOMKilled container indicates node memory pressure or kubelet eviction, not a too-small memory budget. Lower connectionsPerWorker, spread workers across more pods, or raise AIPERF_K8S_WORKER_POD_MEMORY on the operator deployment.

Why are results missing while my AIPerf benchmark is still running?▼

Results are harvested only on a terminal phase, so missing results mid-run is expected behavior. Use aiperf kube results with --from-pods to read the live controller, and stream logs with --follow because successful runs delete pods immediately after harvest.

Related Skills

Explore More Software Engineering Skills→
tankpkgtankpkg
1

@tank/kueue

Analyzes Kueue repository guidance to produce structured Kubernetes queuing knowledge for workload admission scenarios.

Official
Advanced
TheLineExpTheLineExp

perf-test

Benchmark HTTP performance and load testing across development, staging, and production environments.

Community
Advanced
hardw00thardw00t
90

container-security

Automates container security assessments across Docker images, Kubernetes clusters, and runtimes using Trivy, Grype, Kubescape, kube-bench, and Falco.

Community
Advanced
air-gappedair-gapped
5

aiperf

Benchmark latency, throughput, and telemetry across OpenAI-compatible inference servers.

Official
Advanced