skills.rest
wangyouan
wangyouanCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

advanced-evaluation

Evaluate LLM outputs with direct scoring, pairwise comparison, and bias mitigation.

Updated Jun 29, 2026
Search Tags:#evaluation#LLM#quality assessment#bias mitigation#metric selection
One-click install
npx skills add https://github.com/wangyouan/codex-personal-kit --skill advanced-evaluation-wangyouan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/wangyouan/codex-personal-kit/tree/main/skills/advanced-evaluation
Command: npx skills add https://github.com/wangyouan/codex-personal-kit --skill advanced-evaluation-wangyouan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pypdf, pdfplumber, pdf2image, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides advanced evaluation techniques for LLM-as-judge systems, enabling the creation of reliable and unbiased evaluation systems for LLM outputs.

Core Features & Use Cases

  • Direct Scoring: Evaluate LLM outputs against defined criteria with a calibrated scale.
  • Pairwise Comparison: Compare LLM outputs pairwise for subjective preferences.
  • Bias Mitigation: Implement strategies to mitigate position, length, self-enhancement, verbosity, and authority biases.
  • Metric Selection Framework: Match metrics to the evaluation task structure.
  • Evaluation Approaches: Detailed guidance on implementing direct scoring and pairwise comparison.
  • Rubric Generation: Instructions for creating rubrics for consistent evaluation.
  • Practical Guidance: Best practices and gotchas for building evaluation systems.
  • Integration: Guidance on integrating with other skills and tools for comprehensive evaluation.

Quick Start

Activate the advanced-evaluation skill and provide a response or set of responses to evaluate.

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is LLM-as-judge evaluation and how does it assess output quality?▼

LLM-as-judge evaluation assesses output quality using direct scoring and pairwise comparison. It applies calibrated scales and defined criteria to automate quality assessment for language model responses systematically.

How do I mitigate bias in automated LLM scoring systems?▼

To mitigate bias in automated LLM scoring, implement strategies targeting position, length, self-enhancement, verbosity, and authority biases. This ensures consistent and reliable evaluation results across different outputs.

How do I create evaluation rubrics for consistent LLM quality assessment?▼

Create evaluation rubrics for LLM quality assessment by defining calibrated scoring scales and specific criteria. This framework generates consistent evaluation standards for direct scoring and pairwise comparison tasks.

What is the best way to compare LLM outputs pairwise for subjective preferences?▼

The best way to compare LLM outputs pairwise is using structured comparison techniques against defined criteria. This approach evaluates subjective preferences while applying bias mitigation to ensure fair results.

Does this LLM evaluation approach require Python and model access to execute?▼

Yes, executing this LLM evaluation logic requires Python and LLM model access. You need this environment setup to run the direct scoring, pairwise comparison, and bias mitigation scripts.

How do I select the right metrics for my LLM evaluation task structure?▼

Select the right metrics for LLM evaluation by matching them to your specific task structure. The metric selection framework aligns evaluation approaches with task requirements to ensure accurate quality assessment.

Related Skills

Explore More Software Engineering Skills→
BagelHoleBagelHole
46

ai-security-hardening

Harden AI deployments against prompt injection and data exfiltration.

Community
Advanced
enigmaicon-engenigmaicon-eng
1

advanced-evaluation

Score or compare LLM outputs with rubric-based LLM-as-judge evaluation and structured JSON results.

Community
Advanced
guanyangguanyang
947

advanced-evaluation

Implement LLM-as-a-Judge scoring with pairwise comparison and bias mitigation.

Community
Advanced
rohunvorarohunvora
5

advanced-evaluation

Automate LLM-as-a-judge scoring, pairwise comparison, and rubric generation with bias mitigation.

Community
Advanced