aiperf-kube-sweep

Runs parameter sweeps and Bayesian search benchmarks on Kubernetes via AIPerfSweep CRs.

631|166|Updated Apr 16, 2025
One-click install
npx skills add https://github.com/ai-dynamo/aiperf --skill aiperf-kube-sweep
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: aiperf-kube-sweep
Source: https://github.com/ai-dynamo/aiperf/tree/main/.agents/skills/aiperf-kube-sweep
Command: npx skills add https://github.com/ai-dynamo/aiperf --skill aiperf-kube-sweep

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Running large benchmark grids or adaptive searches against LLM inference endpoints on Kubernetes requires orchestrating many sequential benchmark jobs, enforcing cardinality caps, handling credentials safely, and aggregating results across variations — all of which is error-prone to do by hand.

Core Features & Use Cases

  • Sweep orchestration: Submit one AIPerfSweep CR with aiperf kube sweep and get one child AIPerfJob per (variation, trial), executed sequentially by a dedicated sweep-controller pod.
  • Grid, zip, scenario, Sobol, Latin Hypercube, and adaptive/Bayesian search: Author parameter axes with dotted paths rooted in the benchmark: block, or define a search space with objectives for Bayesian optimization.
  • Safety guardrails: Credential-bearing axes are rejected before creation, the 200-variation and 10-trial caps are enforced, and nested in-process sweeps are blocked under operator management.
  • Use Case: Benchmark a vLLM deployment by sweeping phases.profiling.concurrency over [4, 8, 16, 32], then run aiperf kube results <sweep-name> to download the aggregate and per-child exports.

Quick Start

Ask the AI to author a sweep.yaml sweeping concurrency over a few values and submit it with aiperf kube sweep against your benchmark namespace.

Frequently Asked Questions about aiperf-kube-sweep

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a parameter sweep with aiperf on Kubernetes?

Author a YAML file with a benchmark block and a sweep block defining parameter axes, then submit it with aiperf kube sweep --config sweep.yaml --image aiperf:latest. The operator creates one child AIPerfJob per variation and trial, run sequentially by a sweep-controller pod.

How do I run Bayesian optimization benchmarks on Kubernetes?

Set sweep.type to adaptive_search with planner bayesian or optuna, define a searchSpace with lo/hi bounds, objectives with metric and direction, and a required maxIterations between 2 and 200. The sweep-controller proposes one point per iteration and creates one child AIPerfJob each.

Why does aiperf kube sweep reject my credential parameters?

Credential-bearing axes such as API keys, tokens, passwords, and any endpoint.headers path are rejected before anything is created, for every sweep type. Keep credentials fixed and inject them via Secret-backed env vars like AIPERF_INJECTED_API_KEY using valueFrom.secretKeyRef in the pod template.

What are the limits on aiperf sweep variations and trials?

Sweeps are capped at 200 variations enforced by the operator at admission, and 10 trials per variation enforced as a client-side schema bound on multiRun.numRuns. Dry-run and validation pass on over-cap grids, so count the product of axis lengths before submitting.

Does aiperf kube sweep --dry-run show the variation list?

No. Dry-run only renders the AIPerfSweep CR as JSON and returns before touching the cluster; it does not expand variations, show child counts, or check cardinality caps. Expand the sweep locally or compute the grid product yourself to preview variation counts.

How do I cancel a running aiperf sweep on Kubernetes?

Run aiperf kube cancel <sweep-name> for the whole sweep, or add -v <index> to cancel one variation. The running child drains to a terminal phase and still contributes results, remaining iterations are skipped, and aggregation runs over completed children.