skills.rest
agentscope-ai
agentscope-aiOFFICIAL
Explore all skills by this author →
𝕏@agentscope_aiGitHub

metric-design

Select, configure, and combine evaluation graders into executable OpenJudge pipeline code.

809|65|Updated Jul 8, 2025
Search Tags:#LLM evaluation#LLM-as-judge#rubric generation#evaluation pipeline#metric design#grader selection#openjudge
One-click install
npx skills add https://github.com/agentscope-ai/OpenJudge --skill metric-design
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: metric-design
Source: https://github.com/agentscope-ai/OpenJudge/tree/main/skills/eval_pipeline/02-metric-design
Command: npx skills add https://github.com/agentscope-ai/OpenJudge --skill metric-design

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires py-openjudge.

What problem does it solve?

Teams building AI applications often have evaluation principles or datasets but struggle to choose the right graders, write effective LLM-as-judge prompts, and combine multiple metrics into a trustworthy composite score. This Skill turns vague evaluation goals into a working, cost-aware grading pipeline.

Core Features & Use Cases

  • Grader Selection Decision Tree: Maps each evaluation dimension to the right grader type, from zero-cost deterministic checks (StringMatchGrader, JsonValidatorGrader) to LLM judges (CorrectnessGrader, HallucinationGrader) and agentic graders.
  • Custom Judge Prompt Engineering: Enforces a four-component LLMGrader template (task criterion, binary pass/fail definitions, few-shot examples including borderline cases, structured critique-before-verdict output) and scans for anti-patterns like Likert scales and missing few-shot examples.
  • Pipeline Assembly with Gates: Produces executable GradingRunner code with weighted aggregation plus a GatedWeightedSumAggregator pattern so hard requirements like PII or safety can never be masked by high scores elsewhere.
  • Use Case: You have a customer support chatbot and 20 labeled examples. Use this Skill to auto-generate a rubric with IterativeRubricsGenerator, add deterministic format checks, and assemble a weighted pipeline that scores correctness, relevance, and hallucination.

Quick Start

Ask the AI to design an evaluation pipeline for your chatbot using OpenJudge, choosing graders for correctness, relevance, and hallucination and combining them into one composite score.

Frequently Asked Questions about metric-design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose the right grader for an LLM evaluation metric?▼

Walk the decision tree in order: use deterministic graders like StringMatchGrader or JsonValidatorGrader when a rule can check the dimension, built-in LLM graders like CorrectnessGrader for semantic text quality, agent-specific graders for tool calls and trajectories, and AgenticGrader only when external verification is required. Exhaust free deterministic options before adding LLM judges.

How do I write a good LLM-as-judge prompt?▼

Use the four-component template: a single task criterion, concrete binary pass/fail definitions, few-shot examples including at least one pass, one fail, and one borderline case, and a structured JSON output with critique before verdict. Avoid Likert scales, which produce scores that cannot be calibrated.

Can I auto-generate evaluation rubrics from labeled data?▼

Yes. OpenJudge provides SimpleRubricsGenerator for zero-shot rubric creation from a task description, and IterativeRubricsGenerator for data-driven generation when you have 20 or more labeled examples with query, response, and score fields.

Why should the judge model differ from the model being evaluated?▼

Using the same model for both roles causes self-evaluation bias that inflates scores. Always configure a different model, or at minimum a different model version, as the judge when instantiating LLM-based graders.

How do I prevent a high score from hiding a safety or PII failure?▼

Implement a gate instead of a weight. The GatedWeightedSumAggregator pattern returns score 0.0 whenever a gate grader like no_pii falls below its threshold, so hard requirements can never be compensated by strong scores on other dimensions.

When should I use deterministic graders instead of LLM judges?▼

Use deterministic graders whenever the check is code-expressible, such as exact string match, JSON schema validation, regex format checks, or code execution. They cost nothing, are fully reproducible, and roughly 30-50% of evaluation dimensions can typically be checked this way.

Related Skills

Explore More Data & Analytics Skills→
ContextJet-aiContextJet-ai
29

build-eval-dataset

Create and verify layered evaluation datasets for LLM applications.

Official
Intermediate
adamatdevopsadamatdevops
1

validate-evaluator

Calibrate an LLM judge against human Pass/Fail labels using TPR/TNR metrics.

Community
Advanced
githubgithub
38.5k

phoenix-evals

Build and run evaluators for AI and LLM applications using Phoenix.

Official
Advanced
wshobsonwshobson
39.3k

eval-harness-first

Build evaluation harnesses with goldens, graders, and baselines that gate fine-tuning runs.

Community
Advanced