skills.rest
Shakudo-io
Shakudo-ioOFFICIAL
Explore all skills by this author →
WebsiteGitHub

advanced-evaluation

Implements customizable checklists for evaluating LLM outputs and detecting hallucinations.

Updated Feb 14, 2026
Search Tags:#metrics#evaluation#pairwise#rubrics#bias-mitigation#llm-evaluation
One-click install
npx skills add https://github.com/Shakudo-io/opencode-skills --skill advanced-evaluation-shakudo-io
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/Shakudo-io/opencode-skills/tree/main/context-optimization/skills/advanced-evaluation
Command: npx skills add https://github.com/Shakudo-io/opencode-skills --skill advanced-evaluation-shakudo-io

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

The Advanced Evaluation skill provides a structured, production-grade approach to evaluating LLM outputs, enabling teams to design reliable evaluation pipelines, improve objectivity, and systematically address biases that affect judgments.

Core Features & Use Cases

  • Direct Scoring: Define objective criteria and obtain calibrated scores for individual responses.
  • Pairwise Comparison: Compare two responses to determine preference while mitigating position bias.
  • Rubric Generation: Create domain-specific rubrics to standardize judgments across tasks.
  • Bias Mitigation & Monitoring: Implement strategies to detect and reduce biases (position, length, self-enhancement).
  • Evaluation Pipelines: Assemble multi-stage workflows including input validation, scoring, bias mitigation, and output formatting.

Quick Start

Run a full evaluation on a sample prompt and two responses to generate per-criterion scores, rubrics, and bias diagnostics.

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a production-grade LLM evaluation pipeline?▼

Build an LLM evaluation pipeline by assembling multi-stage workflows that include input validation, direct scoring, bias mitigation, and output formatting. This framework supports production-grade evaluation pipelines to systematically assess model outputs.

How does pairwise comparison mitigate position bias in LLM outputs?▼

Pairwise comparison mitigates position bias by comparing two responses to determine preference while actively monitoring and reducing positional influences. This strategy ensures objective judgments across different prompt responses.

Can I generate domain-specific rubrics for evaluating LLM responses?▼

Yes, you can generate domain-specific rubrics to standardize judgments across tasks. The framework provides rubric generation capabilities to define objective criteria and obtain calibrated scores for individual responses.

What strategies detect and reduce bias in LLM evaluation?▼

Bias mitigation strategies detect and reduce position, length, and self-enhancement biases. The framework implements monitoring mechanisms to systematically address biases that affect judgments during LLM output evaluation.

Does the advanced evaluation framework require external dependencies?▼

No, the advanced evaluation framework operates without external dependencies. It provides scripts and references to apply evaluation strategies, build pipelines, and implement bias mitigation across prompts independently.

What is direct scoring for LLM outputs and when is it needed?▼

Direct scoring defines objective criteria and obtains calibrated scores for individual responses. It is needed when teams require production-grade evaluation pipelines to improve objectivity and systematically address biases in LLM outputs.

Related Skills

Explore More Data & Analytics Skills→
BoraPerusicBoraPerusic

advanced-evaluation

Automate LLM output evaluation with direct scoring and pairwise comparison.

Community
Advanced
leonanpereirapintoleonanpereirapinto
1

ai-prompt-engineering-safety-review

Analyzes prompts for safety, bias, privacy, and security issues.

Community
Advanced
roanbrasilroanbrasil
5

llm-evals

Implement LLM evaluation patterns with LangSmith, RAGAS, and PromptFoo.

Community
Advanced
arthtyagiarthtyagi
1

benchmark-sandbox

Provision Vercel Sandboxes to run benchmark scenarios and generate coverage reports.

Community
Advanced