skills.rest
Tyler-R-Kendrick
Tyler-R-KendrickCOMMUNITY
Explore all skills by this author →
GitHub

judge-rubric

Generate formalized judging rubrics with locked dimensions, pass-partial-fail boundaries, and evidence requirements.

Updated Mar 25, 2026
Search Tags:#python#evaluation#evidence#scoring#rubric#grading#llm-judge
One-click install
npx skills add https://github.com/Tyler-R-Kendrick/copilot-auto-training --skill judge-rubric-tyler-r-kendrick
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: judge-rubric
Source: https://github.com/Tyler-R-Kendrick/copilot-auto-training/tree/main/skills/judge-rubric
Command: npx skills add https://github.com/Tyler-R-Kendrick/copilot-auto-training --skill judge-rubric-tyler-r-kendrick

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve? Judging tasks often start without a stable scoring rubric, leading to inconsistent, holistic, or biased evaluations. This Skill produces a formal, decision-ready rubric package before any scoring begins, so another judge can apply it without reinterpreting the task. ## Core Features & Use Cases - Locked Rubric Authoring: Defines 3 to 7 task-adaptive dimensions with explicit pass, partial, and fail boundaries, allowed evidence per dimension, aggregation rules, and tie-breakers. - Deterministic Helper Script: Renders or validates rubric packages from a structured JSON contract via scripts/render_rubric.py, including a --validate-only mode for incomplete contracts. - Robustness Controls: Builds in order-bias checks, benchmark-overfitting skepticism, confidence guidance, and low-trust treatment of unsupported chain-of-thought. - Use Case: When comparing two policy-assistant answers or evaluating agent runs where both trajectory and final answer matter, generate a hybrid rubric with evidence-ledger expectations before scoring starts. ## Quick Start Ask the agent to create a formal judging rubric for your evaluation task, providing the domain, task contract, artifacts to be judged, and any existing scoring criteria.

Frequently Asked Questions about judge-rubric

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a scoring rubric for judging LLM outputs?▼

Provide the task contract, domain context, and artifacts to be judged, and the skill locks 3 to 7 dimensions with pass, partial, and fail boundaries before scoring. It also defines allowed evidence, aggregation rules, and tie-breakers so another judge can apply the rubric directly.

How do I render a rubric from a structured JSON contract?▼

Run scripts/render_rubric.py with --input-file pointing to your contract JSON and optionally --output-file for the markdown result. Use --validate-only to check contract completeness, including the required 3 to 7 dimensions, before rendering.

What is the difference between outcome-focused, process-aware, and hybrid rubrics?▼

Outcome-focused rubrics judge final outputs against references and criteria, while process-aware rubrics rely on traces, logs, and intermediate artifacts. Hybrid rubrics split evidence expectations across both, which suits agent-run evaluations where trajectory and final answer both matter.

What happens when the judging domain is underspecified?▼

The skill stops and returns a blockers list naming missing inputs, weak evidence areas, and needed clarifications instead of inventing thresholds. The helper script also fails validation when required contract fields such as allowed evidence are absent.

Does the rubric handle evaluator bias and unreliable reasoning?▼

Yes, the rubric package includes order-bias checks, benchmark-overfitting skepticism, and confidence guidance for narrow margins. Unsupported chain-of-thought and self-explanations are treated as low-trust evidence unless corroborated by artifacts.

Related Skills

Explore More Software Engineering Skills→
TheycallmehollaTheycallmeholla

hot-seat

Validate project plans through iterative single-question interrogation and decision logs.

Community
Intermediate
AgLyx3AgLyx3

interview-system-designer

Design role-specific interview loops with competency matrices and calibration analytics.

Community
Advanced
Andreicr1Andreicr1

adversarial-reviewer

Identify failure modes in code changes through adversarial scenario construction.

Community
Advanced
21StarkCom21StarkCom

stark-story-judge

Grades long-form blog posts with cold cross-vendor LLM judges on an anchored rubric.

Official
Advanced