skills.rest
akashjpal
akashjpalCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

arize-experiment

Creates, runs, and compares Arize experiments for evaluating model performance via the ax CLI.

1|Updated Nov 9, 2025
Search Tags:#llm evaluation#model comparison#benchmark#experiment#dataset#arize#ax cli
One-click install
npx skills add https://github.com/akashjpal/question-generator --skill arize-experiment-akashjpal
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: arize-experiment
Source: https://github.com/akashjpal/question-generator/tree/main/.agents/skills/arize-experiment
Command: npx skills add https://github.com/akashjpal/question-generator --skill arize-experiment-akashjpal

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Evaluating and comparing LLM outputs across datasets requires manual tracking of runs, scores, and regressions. This Skill automates experiment creation, run export, and result comparison on the Arize platform using the ax CLI. ## Core Features & Use Cases - Experiment CRUD: List, get, create, and delete experiments tied to versioned datasets with structured run schemas including evaluations and metadata. - Bulk Export & Analysis: Export experiment runs via REST or Arrow Flight, then analyze scores, distributions, and regressions with jq pipelines. - Model Comparison Workflows: Run real model APIs (OpenAI, Anthropic, Gemini, or custom endpoints) against dataset examples and compare evaluation metrics across experiments. - Use Case: A team wants to A/B test GPT-4o against Claude on a support-question dataset. The Skill exports the dataset, runs each model, creates two experiments, and identifies regressions where one model fails but the other passes. ## Quick Start Ask the AI to create an Arize experiment named gpt-4o-baseline against your dataset and compare its correctness scores with a previous experiment.

Frequently Asked Questions about arize-experiment

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an Arize experiment from model outputs?▼

Export your dataset with ax datasets export, call the real model API for each example to build a runs file with example_id and output fields, then run ax experiments create with --name, --dataset, --space, and --file flags.

How to compare two experiments in Arize?▼

Export both experiments with ax experiments export --stdout, then use jq to compute average evaluation scores, label distributions, and find regressions where examples passed in one experiment but failed in the other.

What model providers can I use for experiment runs?▼

The inference template supports OpenAI, Anthropic, Google Gemini, and any OpenAI-compatible endpoint such as Azure OpenAI, NVIDIA NIM, or local Ollama. Install the provider SDK and set its API key as an environment variable.

Why does my experiment export return only 500 runs?▼

The default REST export is limited to 500 runs per page. Re-run the export with the --all flag to use Arrow Flight for bulk transfer of the complete experiment results.

How do I fix a 401 Unauthorized error with the ax CLI?▼

Run ax profiles show to inspect the current profile, then update the API key with ax profiles update --api-key $ARIZE_API_KEY. Get a scoped service key from https://app.arize.com/admin under API Keys.

What fields are required in an experiment runs file?▼

Each run requires example_id matching a dataset example and an output string. Optional fields include evaluations with label, score, or explanation, plus metadata such as model name and latency.

Related Skills

Explore More Data & Analytics Skills→
Arize-aiArize-ai
42

arize-experiment

Manage Arize AI experiments with CRUD operations and run analysis via the `ax` CLI.

Official
Advanced
ethan-rioethan-rio

arize-prompt-optimization

Optimize LLM prompts using Arize trace data and evaluation scores.

Community
Advanced
langwatchlangwatch
3.5k

eval-triage

Diagnose failing LangWatch experiment runs down to failing rows, evaluators, and root causes.

Official
Intermediate
githubgithub
38.5k

arize-evaluator

Creates and runs LLM-as-judge evaluators on Arize via the ax CLI.

Official
Advanced