skills.rest
boazcstrike
boazcstrikeCOMMUNITY
Explore all skills by this author →
Website𝕏@boazcstrikeGitHub

advanced-evaluation

Implement LLM-as-a-Judge evaluation systems with pairwise comparison and bias mitigation.

3|Updated Dec 23, 2025
Search Tags:#llm-as-judge#evaluation#rubrics#quality assessment#bias mitigation#pairwise comparison
One-click install
npx skills add https://github.com/boazcstrike/opencode --skill advanced-evaluation-boazcstrike
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/boazcstrike/opencode/tree/main/skills/advanced-evaluation
Command: npx skills add https://github.com/boazcstrike/opencode --skill advanced-evaluation-boazcstrike

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of reliably evaluating LLM outputs, ensuring quality, mitigating bias, and establishing consistent assessment standards.

Core Features & Use Cases

  • LLM-as-a-Judge: Implement production-grade techniques for using LLMs to evaluate other LLM responses.
  • Bias Mitigation: Actively counter position bias, length bias, and other systematic LLM judge errors.
  • Evaluation Pipelines: Design and build automated systems for consistent quality assessment.
  • Use Case: You need to compare the outputs of two different LLMs on a set of prompts. This Skill provides the framework to set up a pairwise comparison, swap response positions to avoid bias, and determine a statistically sound winner.

Quick Start

Use the advanced-evaluation skill to compare two model responses based on clarity and accuracy.

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I use LLM-as-a-judge to evaluate model outputs consistently?▼

LLM-as-a-judge evaluates model outputs by applying structured rubrics and direct scoring methodologies to ensure consistent quality assessment. You can build automated evaluation pipelines that reliably score responses across different models and tasks.

What is the best way to compare two LLM responses without introducing bias?▼

Pairwise comparison mitigates bias by evaluating two LLM responses against each other and swapping response positions to counter systematic errors. This framework determines a statistically sound winner while actively avoiding position and length bias.

How do I create evaluation rubrics for automated quality assessment pipelines?▼

Evaluation rubrics are created by defining specific quality criteria like clarity and accuracy into structured methodologies for direct scoring. These rubrics integrate into automated evaluation pipelines to establish consistent assessment standards across different models.

Does LLM-as-a-judge suffer from systematic bias, and how can it be mitigated?▼

LLM-as-a-judge is susceptible to systematic errors like position and length bias, which can be actively mitigated using structured methodologies. Bias mitigation techniques involve swapping response positions during pairwise comparisons to ensure objective quality assessment.

When should I use pairwise comparison instead of direct scoring for quality assessment?▼

Pairwise comparison is ideal when you need to determine a statistically sound winner between two specific models, while direct scoring suits broader quality assessment across multiple tasks. Both methodologies integrate into automated evaluation pipelines to ensure consistent standards.

What limitations exist when building automated LLM evaluation systems?▼

Automated LLM evaluation systems face limitations from inherent LLM judge biases, requiring active mitigation strategies like position swapping in pairwise comparisons. Reliable quality assessment depends on structured rubrics to counter subjective evaluation errors across different models.

Related Skills

Explore More Software Engineering Skills→
vecearvecear

evaluation-harness

Automate LLM evaluation with golden datasets, rubrics, thresholds, and regression reports.

Community
Advanced
enigmaicon-engenigmaicon-eng
1

advanced-evaluation

Score or compare LLM outputs with rubric-based LLM-as-judge evaluation and structured JSON results.

Community
Advanced
maragudkmaragudk
10

llm-as-a-judge

Build and validate LLM-as-Judge evaluators for automated quality assessment.

Official
Advanced
iani-kuliiani-kuli

validate-evaluator

Calibrate LLM-based evaluators against human-labeled ground truth using TPR and TNR metrics.

Community
Advanced