skills.rest
akashjpal
akashjpalCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

arize-annotation

Creates annotation configs and queues and applies human labels to Arize spans.

1|Updated Nov 9, 2025
Search Tags:#llm evaluation#labeling#annotation#spans#human review#arize#ax cli
One-click install
npx skills add https://github.com/akashjpal/question-generator --skill arize-annotation-akashjpal
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: arize-annotation
Source: https://github.com/akashjpal/question-generator/tree/main/.agents/skills/arize-annotation
Command: npx skills add https://github.com/akashjpal/question-generator --skill arize-annotation-akashjpal

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires arize, pandas, and includes references (resource) components.

What problem does it solve? Setting up human review workflows in Arize requires creating label schemas, routing records to reviewers, and applying annotations to spans, which involves many CLI commands and SDK calls that are easy to get wrong. ## Core Features & Use Cases - Annotation Config Management: Create, list, get, and delete categorical, continuous, and freeform label schemas in an Arize space via the ax CLI. - Annotation Queues: Build human review queues linked to configs, assign reviewers, submit annotations for records, and manage queue records. - Bulk Span Annotation: Apply labels to project spans programmatically with the Python SDK's spans.update_annotations using a pandas DataFrame. - Use Case: A team wants reviewers to label LLM responses as correct or incorrect. Create a categorical Correctness config, spin up a review queue with annotator emails, then bulk-apply reviewed labels back to spans. ## Quick Start Ask the assistant to create a categorical annotation config named Correctness with correct and incorrect labels in your Arize space.

Frequently Asked Questions about arize-annotation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an annotation config in Arize?▼

Run ax annotation-configs create with a name, space, and type. Categorical configs take repeated --value flags, continuous configs take --min-score and --max-score, and freeform configs need no extra flags.

How do I set up a human review queue in Arize?▼

Use ax annotation-queues create with a name, space, at least one --annotation-config-id, and reviewer emails via --annotator-email. You can add instructions and choose an assignment method of all or random.

How do I bulk annotate spans with the Arize Python SDK?▼

Build a pandas DataFrame with context.span_id plus annotation.<name>.label or annotation.<name>.score columns, then call ArizeClient spans.update_annotations with your space_id, project_name, and the DataFrame.

What are the limitations of Arize span annotations?▼

Annotations apply only to spans within 31 days prior to submission. Each annotation config name must be unique within the space, and deleting a config also removes its queue associations.

Why does my ax annotation command return 401 Unauthorized?▼

A 401 means the API key is missing, wrong, or lacks access to the space. Run ax profiles show to inspect the profile, then update it with ax profiles update --api-key $ARIZE_API_KEY using a key from the Arize admin API Keys page.

Related Skills

Explore More Data & Analytics Skills→
seldoseldo
5

arize-trace

Export and analyze Arize traces and spans for LLM application debugging.

Community
Advanced
KAIROS7-team3KAIROS7-team3

vla-finetuning

Fine-tune Vision-Language-Action models with LoRA and Q4 quantization.

Official
Advanced
yanochka11yanochka11

validate-evaluator

Calibrate LLM judges against human labels using TPR/TNR analysis.

Community
Advanced
hamelsmuhamelsmu
1.6k

write-judge-prompt

Design LLM-as-Judge evaluators for subjective criteria with binary pass/fail outcomes.

Community
Intermediate