skills.rest
ai-dynamo
ai-dynamoOFFICIAL
Explore all skills by this author →
WebsiteGitHub

aiperf-kube-setup

Installs and configures the aiperf-operator Helm chart on multi-node Kubernetes clusters.

631|166|Updated Apr 16, 2025
Search Tags:#benchmarking#kubernetes#HELM#cluster setup#aiperf#Operator#jobset
One-click install
npx skills add https://github.com/ai-dynamo/aiperf --skill aiperf-kube-setup
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: aiperf-kube-setup
Source: https://github.com/ai-dynamo/aiperf/tree/main/.agents/skills/aiperf-kube-setup
Command: npx skills add https://github.com/ai-dynamo/aiperf --skill aiperf-kube-setup

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Preparing a real multi-node Kubernetes cluster for AIPerf benchmarking involves many failure-prone steps: installing JobSet, wiring registry pull secrets, choosing node placement and storage classes, and scoping operator RBAC. This Skill guides the one-time cluster setup so the aiperf-operator installs correctly the first time instead of leaving every benchmark pod Pending.

Core Features & Use Cases

  • Cluster inventory and prerequisite checks: Verify Kubernetes version, JobSet CRDs, GPU device plugin, storage classes, node taints, and registry access before installing anything.
  • Guided Helm installation: Install JobSet v0.8.0, create operator and benchmark namespaces with pull secrets, and deploy the aiperf-operator chart from a versioned values file.
  • Pitfall avoidance and verification: Documents sharp edges around tolerations, storage classes, Kueue queue routing, RBAC scoping, and mutable image tags, plus helm test and aiperf kube preflight verification steps.
  • Use Case: A platform engineer receives a new DGX cluster with tainted GPU node pools and a private registry, and uses this Skill to install the operator, configure a 1Ti results PVC, and validate the setup with preflight checks before handing the cluster to benchmark users.

Quick Start

Use the aiperf-kube-setup skill to prepare my Kubernetes cluster for AIPerf by inventorying the nodes, installing JobSet and the aiperf-operator Helm chart with a values file, and verifying the install with preflight checks.

Frequently Asked Questions about aiperf-kube-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I install the aiperf-operator on a Kubernetes cluster?▼

Install JobSet v0.8.0 first, create operator and benchmark namespaces with registry pull secrets, then run helm upgrade --install against the chart in deploy/helm/aiperf-operator using a values file. Verify with kubectl get pods, helm test, and aiperf kube preflight.

How do I configure node placement and tolerations for the aiperf-operator?▼

Set operator.nodeSelector to your node pool label and replace the chart's default tolerations with entries matching your cluster's taints, or an empty list on untainted pools. The defaults target dedicated=user-workload taints and are inert on clusters tainted differently.

Does the aiperf-operator Helm chart install JobSet automatically?▼

No, the chart grants RBAC for jobset.x-k8s.io but does not install JobSet. You must apply the JobSet v0.8.0 manifests separately with kubectl apply --server-side before installing the operator.

Why are my AIPerf benchmark pods stuck in ImagePullBackOff?▼

The chart's imagePullSecrets value covers only the operator and helm test pods, not benchmark pods. Benchmark pods take pull secrets from the CR's spec.podTemplate.imagePullSecrets or the --image-pull-secrets flag, and the secret must exist in the benchmark namespace.

Why is the aiperf-operator results PVC pending after install?▼

The storage.storageClassName default is empty, so the PVC falls back to the cluster's default StorageClass; on clusters without one it pends forever. Name a storage class explicitly, or set storage.enabled=false to use an emptyDir for test clusters.

Can I use a local Kind cluster for AIPerf operator testing?▼

Yes, Kind works for correctness and lifecycle testing using kind load with image.pullPolicy=Never, no pull secret, and the default storage class. It is not suitable for validating placement, storage, or performance behavior of a real cluster.

Related Skills

Explore More Software Engineering Skills→
kettleofketchupkettleofketchup

rook-ceph

Deploy a Rook Ceph cluster across Kubernetes nodes using Helm.

Community
Advanced
air-gappedair-gapped
5

aiperf

Benchmark latency, throughput, and telemetry across OpenAI-compatible inference servers.

Official
Advanced
googlegoogle
19.1k

gke-workload-troubleshooting

Diagnoses GKE workload failures via pod status, events, and logs, then proposes GitOps manifest fixes.

Official
Advanced
YelpYelp
1.7k

paasta-playground

Test PaaSTA code changes against a local Kind Kubernetes cluster playground environment.

Official
Advanced