meta-rubric-gen

What problem does it solve?

Generate precise, atomic, and machine-readable evaluation rubrics for grading single test-case LLM outputs, ensuring objective, MECE coverage and explicit negative constraints to prevent hallucinations and unsafe recommendations.

Core Features & Use Cases

• Atomic Criteria: Transforms a single user input into mutually exclusive, testable rubric items that yield yes/no judgments.
• Safety & Negative Constraints: Enforces 1–2 mandatory negative items for hallucinations, dangerous advice, or policy violations and ensures penalty budgets.
• Domain Context Retrieval: Optionally incorporates agent-specific references and MCP/tool capabilities to produce fact-first checks.
• YAML Output & Validation: Emits YAML-formatted rubrics with integer points and runs structural checks (MECE, penalty budget).
• Use Case: Auto-generate judge rubrics for medical advice, code generation, legal summaries, or content-moderation testcases.

Quick Start

Generate an atomic YAML rubric for a single test case by providing the case Input and optional agent_name plus must_follow notes.