skills.rest
rohunvora
rohunvoraCOMMUNITY
Explore all skills by this author →
𝕏@rohunvoraGitHub

advanced-evaluation

Automate LLM-as-a-judge scoring, pairwise comparison, and rubric generation with bias mitigation.

5|Updated Dec 25, 2025
Search Tags:#evaluation#LLM#scoring#quality-assurance#pairwise#rubrics#bias-mitigation
One-click install
npx skills add https://github.com/rohunvora/my-claude-skills --skill advanced-evaluation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/rohunvora/my-claude-skills/tree/main/.claude/skills/advanced-evaluation
Command: npx skills add https://github.com/rohunvora/my-claude-skills --skill advanced-evaluation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Master LLM-as-a-Judge evaluation techniques including direct scoring, pairwise comparison, rubric generation, and bias mitigation. Use when building evaluation systems, comparing model outputs, or establishing quality standards for AI-generated content.

Core Features & Use Cases

  • Direct Scoring: Structured, rubric-driven scoring of a single response on objective criteria.
  • Pairwise Comparison: Relative quality judgments between two responses with bias-mitigation protocols.
  • Rubric Generation: Domain-specific scoring rubrics to reduce variance and improve reliability.
  • Bias Mitigation: Techniques to counter position, length, self-enhancement, verbosity, and authority biases.
  • Evaluation Pipeline Design: End-to-end architecture for scalable evaluation.

Quick Start

Run the included evaluation_example.py script to see examples of direct scoring and pairwise comparison, and consult the references for bias mitigation patterns. Then plug in your own prompts and responses to bootstrap an automated evaluation pipeline.

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I use LLMs to automatically score and evaluate AI-generated outputs?▼

LLM-as-a-Judge evaluation automates scoring by using language models as judges to assess outputs against defined criteria. This Skill provides structured rubrics, direct scoring methods, and bias-mitigation protocols to build reliable evaluation pipelines for quality assurance at scale.

What's the difference between direct scoring and pairwise comparison for LLM evaluation?▼

Direct scoring evaluates a single response against explicit criteria and rubrics; pairwise comparison ranks two responses relative to each other. Pairwise comparison reduces bias by forcing comparative judgment rather than absolute scoring, useful when criteria are subjective or calibration is difficult.

How do I reduce bias in LLM evaluation judgments?▼

This Skill addresses position, length, self-enhancement, verbosity, and authority biases through deterministic protocols and structured prompting. Bias mitigation includes controlling input order, normalizing response length, and removing cues that favor one output over another.

Can I generate domain-specific scoring rubrics automatically?▼

Yes. Rubric generation creates tailored evaluation criteria for your domain and use case, reducing scorer variance and improving consistency. The Skill includes patterns for LLMs to produce structured rubrics from problem descriptions.

What do I need before building an evaluation pipeline with this Skill?▼

You need explicit quality criteria, sample outputs to evaluate, and a preference for direct scoring or pairwise comparison methodology. The Skill requires deterministic output formats (JSON) and clear performance objectives for your evaluation task.

Related Skills

Explore More Software Engineering Skills→
Sir-chawakornSir-chawakorn
9

llm-eval-harness

Builds an LLM evaluation harness using golden datasets and code-based scorers.

Community
Advanced
vecearvecear

evaluation-harness

Automate LLM evaluation with golden datasets, rubrics, thresholds, and regression reports.

Community
Advanced
BoraPerusicBoraPerusic

llm-evaluation

Evaluate LLM outputs with automated metrics and human judgments.

Community
Advanced
JhabbigJhabbig

llm-evaluation

Design evaluation frameworks for LLM applications with automated and human scoring.

Community
Advanced