skills.rest
EquiStamp
EquiStampOFFICIAL
Explore all skills by this author →
WebsiteGitHub

pairwise-compare-evals

Compare AI safety evaluations pairwise across predefined criteria to generate rankings.

Updated Aug 23, 2026
Search Tags:#automation#yaml#risk-analysis#agent-based#pairwise-comparison#ahp#evaluation-design
One-click install
npx skills add https://github.com/EquiStamp/evaluating-evaluations --skill pairwise-compare-evals
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pairwise-compare-evals
Source: https://github.com/EquiStamp/evaluating-evaluations/tree/main/.claude/skills/pairwise-compare-evals
Command: npx skills add https://github.com/EquiStamp/evaluating-evaluations --skill pairwise-compare-evals

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires PyYAML, and includes scripts (resource) and assets (resource) components.

What problem does it solve?

Pairwise-compare-evals provides a structured method to rank AI safety evaluations by systematically contrasting each eval across a fixed set of criteria using the Saaty scale, producing a transparent justification trail.

Core Features & Use Cases

  • Compares all assessed evals head-to-head on 10 criteria (7 rubric dimensions + 3 porting criteria) to generate per-pair scores and overall rankings.
  • Generates batch prompts and aggregates results into a matrix and summary rankings, enabling data-driven prioritization of evals.
  • Use Case: run the full evaluation comparison workflow to surface top-performing evals and identify gaps in coverage or feasibility.

Quick Start

Run the full pairwise evaluation workflow to generate batches and start processing.

Frequently Asked Questions about pairwise-compare-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I rank AI safety evaluations using pairwise comparison?▼

The Saaty scale is applied to perform pairwise comparisons across 10 criteria, generating per-pair scores and overall rankings with a transparent justification trail for each evaluated YAML file.

What is the best way to automate head-to-head evaluation design for YAML files?▼

Automating head-to-head evaluation design involves applying a fixed criteria.yaml set to assessed YAML files, generating batch prompts, a comparison matrix, and summary rankings for data-driven prioritization.

Does pairwise comparison require a specific risk-analysis criteria file to work?▼

Yes, pairwise comparison requires an assets/criteria.yaml file defining the 10 evaluation criteria, including 7 rubric dimensions and 3 porting criteria, to systematically score the assessed evals.

Can I use agent-based analysis to identify gaps in evaluation coverage?▼

Agent-based analysis runs the full evaluation comparison workflow to surface top-performing evals and identify gaps in coverage or feasibility by aggregating results into a summary ranking.

How do I generate batch prompts for pairwise comparison of evals?▼

Batch prompts are generated by running the full pairwise evaluation workflow, which applies comparison scripts to all assessed eval YAMLs and outputs prompts with candidate rankings to an assessments directory.

Related Skills

Explore More Data & Analytics Skills→
The-Skyy-Rose-Collection-LLCThe-Skyy-Rose-Collection-LLC
3

eval-harness

Define, run, and report evaluations with /eval commands.

Official
Intermediate
grasberggrasberg
3

research-synthesizer

Synthesize cross-source evidence to identify consensus and gaps.

Community
Advanced
anotbanotb

contract-gap-review

Analyze contract clauses against regulatory requirements and identify compliance gaps.

Community
Advanced
jsagirjsagir
2

eureka

Analyze portfolios of projects to detect weak-signal cross-domain opportunities using AHP scoring criteria.

Community
Intermediate