skills.rest
Arize-ai
Arize-aiOFFICIAL
Explore all skills by this author →
Website𝕏@arizeaiGitHub

phoenix-evals-new-metric

Create built-in classification evaluators for Phoenix evals from YAML configs.

11.3k|1.1k|Updated Nov 9, 2022
Search Tags:#benchmark#LLM evaluation#Phoenix#yaml config#Code generation#LLM-as-a-Judge#classification evaluator
One-click install
npx skills add https://github.com/Arize-ai/phoenix --skill phoenix-evals-new-metric
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: phoenix-evals-new-metric
Source: https://github.com/Arize-ai/phoenix/tree/main/.agents/skills/phoenix-evals-new-metric
Command: npx skills add https://github.com/Arize-ai/phoenix --skill phoenix-evals-new-metric

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Adding a new LLM-as-a-judge metric to Phoenix evals requires touching many files across Python, TypeScript, benchmarks, and docs; this Skill provides the complete ordered pipeline so nothing is missed.

Core Features & Use Cases

  • YAML Config Authoring: Define the evaluator's prompt, labels, score choices, and optimization direction in a single source-of-truth YAML config.
  • Cross-Language Code Generation: Compile the config into Python and TypeScript evaluator classes, then wire up exports and vitest tests.
  • Benchmarking and Documentation: Build a 30-50 example benchmark with per-category accuracy and a failed-examples printer, then publish a docs page with navigation updates.
  • Use Case: You want a new hallucination detection metric in Phoenix. Follow the pipeline to create the YAML config, run make codegen-prompts, add the Python and TypeScript evaluators, validate accuracy above 80% with a benchmark, and ship the docs page.

Quick Start

Create a new built-in classification evaluator for Phoenix evals that measures whether an LLM answer is faithful to the provided context.

Frequently Asked Questions about phoenix-evals-new-metric

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a new built-in evaluator in Phoenix evals?▼

Create a YAML config under prompts/classification_evaluator_configs, run make codegen-prompts, then add a Python evaluator subclassing ClassificationEvaluator and a TypeScript factory wrapping createClassificationEvaluator. Finish with a benchmark and a docs page.

How do I add an LLM-as-a-judge metric to Phoenix?▼

Define the judge prompt and label-to-score choices in a YAML classification evaluator config, compile it with make codegen-prompts, and wrap the generated config in Python and TypeScript evaluator classes. Validate the metric with a benchmark targeting over 80 percent accuracy.

What labels and scores should a Phoenix classification evaluator use?▼

Binary evaluators typically map a positive label to 1.0 and a negative label to 0.0, with optimization_direction set to maximize. Use minimize only when the metric measures something undesirable, such as hallucination.

When should a Phoenix evaluator use substitutions in its config?▼

Substitutions are only needed when the evaluator is a promoted_dataset_evaluator working with structured span data like tool definitions or message arrays. Evaluators using simple text fields such as input, output, and reference do not need them.

Why is my Phoenix evaluator benchmark accuracy low?▼

Low accuracy usually comes from an unclear judge rubric or unrepresentative benchmark examples. Inspect the failed examples printer output, then either refine the prompt in the YAML config or improve the benchmark's synthetic examples and rerun.

Related Skills

Explore More Software Engineering Skills→
githubgithub
38.5k

phoenix-evals

Build and run evaluators for AI and LLM applications using Phoenix.

Official
Advanced
box755box755

phoenix-observability

Collect OpenTelemetry traces and run LLM evaluations in Phoenix.

Community
Intermediate
Arize-aiArize-ai
11.0k

arize-phoenix

Capture AI application traces, evaluations, and prompts with OpenTelemetry.

Official
Advanced
cuba6112cuba6112

eval-frameworks

Provides evaluation frameworks for LLM outputs in RAG systems.

Community
Intermediate