x-eval

What problem does it solve?

This Skill provides a comprehensive solution for evaluating the quality of AI agent outputs, enabling users to score, compare, benchmark, and calibrate agent performance.

Core Features & Use Cases

• Multi-Rubric Scoring: Score agent outputs against predefined or custom rubrics to ensure quality and consistency.
• Benchmarking: Compare different strategies or models on the same task for performance analysis.
• A/B Testing: Conduct A/B experiments with prompt variations to assess their impact on agent output quality.
• Consistency Measurement: Ensure that the output of a plugin is consistent across repeated trials.
• Quality Comparison: Compare the quality of two outputs to determine the better one.
• Rubric Management: Create and manage custom rubrics for specific evaluation needs.
• Calibration: Validate AI judge scores against human judgment to identify and mitigate biases.

Quick Start

To evaluate the quality of an AI agent's output, use the x-eval skill and specify the content and rubric for evaluation. For example, to score a code snippet using the 'code-quality' rubric, you would say: /xm:eval score "function add(a,b){return a+b}" --rubric code-quality