design-test-rubric

What problem does it solve?

Design-test-rubric provides a structured blueprint to craft rigorous evaluation rubrics for PROBE-like AI agent systems, ensuring consistency and comparability across runs.

Core Features & Use Cases

• Eight-category rubric with weights summing to 100, tailored to observed failure modes and verification needs.
• Comprehensive severity taxonomy (critical/major/minor) with explicit sub-score rules and a hard cap on critical violations.
• Fixed violation log schema, run-tagging conventions, and a reusable scorecard template for all rubric revisions.
• Clear versioning and changelog workflow for iterative rubric improvements.

Quick Start

Write a starter rubric by listing eight categories with weights, define severity rules, specify the violation fields, and lock in the scorecard template; then bump the minor version and add a changelog entry.