skills.rest
marinvch
marinvchCOMMUNITY
Explore all skills by this author →
GitHub

advanced-evaluation

Design and run LLM evaluation workflows with structured rubrics and bias mitigation.

Updated Mar 13, 2026
Search Tags:#evaluation#confidence#ci-cd#pairwise#rubrics#bias-mitigation#llm-evaluation
One-click install
npx skills add https://github.com/marinvch/ai-os --skill advanced-evaluation-marinvch
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/marinvch/ai-os/tree/main/.agents/skills/context-engineering-collection/skills/advanced-evaluation
Command: npx skills add https://github.com/marinvch/ai-os --skill advanced-evaluation-marinvch

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Automates the design and deployment of robust LLM-evaluator systems, enabling reliable judging of model outputs and bias mitigation.

Core Features & Use Cases

  • Structured rubrics and scoring for direct and pairwise evaluations
  • Bias mitigation patterns like position swapping and anonymization to reduce bias
  • Pipeline integration with common evaluation tools for CI/CD and monitoring

Quick Start

Provide a basic evaluation for a pair of model responses using the included rubric and bias-mitigation flow

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an LLM evaluation pipeline with bias mitigation?▼

Build an LLM evaluation pipeline with bias mitigation by applying structured rubrics, position swapping, and anonymization patterns to reduce bias during automated judging of model outputs.

What is confidence calibration in LLM evaluation?▼

Confidence calibration in LLM evaluation adjusts automated judge scores to reflect true probability estimates, ensuring that rubric-based pairwise comparisons yield reliable and structured outputs across prompts and models.

How do I set up pairwise evaluation rubrics for model outputs?▼

Set up pairwise evaluation rubrics by loading explicit scoring criteria into the evaluation workflow, which then applies position swapping and anonymization to compare two model responses fairly.

Does this LLM evaluation approach integrate with CI/CD pipelines?▼

Yes, this LLM evaluation approach integrates with CI/CD pipelines by connecting automated judging and confidence calibration outputs to common evaluation tools for continuous monitoring and deployment.

What's the best way to detect bias in automated LLM judging?▼

The best way to detect bias in automated LLM judging involves applying bias mitigation patterns like position swapping and anonymization, then comparing structured rubric scores across multiple model outputs.

When should I use pairwise evaluation instead of direct scoring?▼

Use pairwise evaluation instead of direct scoring when comparing two model responses requires relative judgment, applying structured rubrics with position swapping to mitigate bias in the evaluation pipeline.

Related Skills

Explore More Software Engineering Skills→
samvanmesamvanme

advanced-evaluation

Evaluate LLM outputs with scoring rubrics and bias-aware comparison.

Community
Advanced
phonowellphonowell
1

advanced-evaluation

Automate LLM evaluation pipelines with structured rubrics and bias mitigation.

Community
Advanced
EngineerWithAIEngineerWithAI
4

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Official
Advanced
aldy505aldy505

advanced-evaluation

Implements LLM-as-a-Judge techniques for scoring, pairwise comparison, and bias mitigation.

Community
Advanced