Generate synthetic YAML evaluation datasets for the PXI eval harness.
npx skills add https://github.com/Arize-ai/phoenix --skill pxi-eval-dataset
Please help me install this Agent Skill. Skill: pxi-eval-dataset Source: https://github.com/Arize-ai/phoenix/tree/main/.agents/skills/pxi-eval-dataset Command: npx skills add https://github.com/Arize-ai/phoenix --skill pxi-eval-dataset
Writing evaluation datasets for the PXI agent harness by hand is slow and error-prone: ground truth annotations drift, coverage gaps go unnoticed, and failures get misattributed between the dataset, the agent, and the evaluators. This Skill produces small, well-targeted synthetic YAML datasets that drop into evals/pxi/datasets/ and run through the PXI experiment harness with deterministic scoring.
Ask the assistant to write evals covering a specific PXI tool such as set_spans_filter, and it will survey the evaluators, draft coverage-diverse queries, annotate ground truth via cross-model subprocesses, and validate the dataset through the experiment harness.
High-intent search queries and answers about installing and using this skill.
Invoke the skill with the target tool name; it reads the tool definition and parameters_json_schema, surveys existing evaluators, drafts coverage-diverse queries, and annotates expected outputs via three independent model subprocesses. The result is a YAML dataset saved to evals/pxi/datasets/ and validated through run_experiment.
Each example is annotated in three fresh subprocesses: Sonnet and Opus via the Agent tool, and Codex via the annotate_via_codex.sh Bash wrapper in read-only sandbox mode. A fourth orchestrator subprocess merges the anonymized candidates, drops spec-invalid values, and records an agreement level of high, medium, or low in the example metadata.
The skill stops and summarizes the gap, proposing a new evaluator with a name, the expected field it reads, and score semantics. With your approval it implements the evaluator under evals/pxi/evaluators/, adds unit tests, exports it, and then continues dataset generation.
Failures fall into three categories: dataset or annotation issues where the expectation is wrong or too strict, genuine agent issues where the harness caught a real bug, and harness or evaluator issues where matching logic is broken. The skill prescribes fixing harness issues first, dataset issues second, and escalating genuine agent issues.
The harness invokes the agent once per example, so multi-turn conversational behaviors are out of scope. Datasets must include a non-empty splits list, unique example ids, and only evaluators listed in the dataset's top-level evaluators field are run.
growthxaiAutomate offline evaluation testing for Output SDK workflows with @outputai/evals.
yiouliAutomate the QA loop for LLM applications using the pixie-qa Python package.
CarlosQ96Map test coverage to GDD critical paths and flag gaps.
marchattonGenerate diverse synthetic test inputs for LLM pipeline evaluation.
