design-test-rubric

To build a repeatable evaluation rubric, define eight weighted categories totaling 100 points, establish a three-tier severity taxonomy for violations, and implement a canonical scorecard template to ensure consistent model-verification across runs.

A three-tier severity taxonomy in a quality assurance scorecard classifies model violations as critical, major, or minor. It includes explicit sub-score rules and applies a hard cap on outcomes when critical violations occur during evaluation.

Manage rubric versioning by bumping the minor version and adding a changelog entry whenever you update category weights. This revision workflow supports adding or removing categories and fingerprinting for new models.

Structure a violation log schema by specifying standardized violation fields, incorporating run-tagging conventions, and linking entries to the fixed scorecard template. This ensures consistent tracking across all rubric revisions and model-verification protocols.

Yes, this rubric design scales for PROBE-like AI agent systems by providing a comprehensive scoring blueprint with eight weighted categories, a fixed scorecard template, and a three-layer model-verification protocol to ensure comparability across multiple runs.

The best way to enforce a hard cap on critical violations is to define explicit sub-score rules within the severity taxonomy that automatically trigger an outcome cap, ensuring any critical failure heavily impacts the final 100-point scorecard.