What problem does it solve? Multiple-choice evaluations that parse answer letters from free-form model generations with regex silently mislabel 5-15% of rows, fabricating or destroying measured effects. This Skill replaces naive letter parsing with a three-way LLM-judge verdict and controls the position and token biases that corrupt MCQ metrics. ## Core Features & Use Cases - LLM-Judge Grading: Grades each generation with a three-way CORRECT / INCORRECT / OTHER verdict, using either extract-then-compare or gold-relative direct judge prompts, so refusals and off-topic outputs are never coerced into a letter. - Bias Controls: Rotates option order (A/B swap or fixed permutations) to expose position bias, and warns against log-prob-only scoring that is blind to output corruption and token bias. - Parser-Artifact Diagnosis: Provides concrete signals — orientation instability, rising OTHER rates, regex-vs-judge disagreement over 5% — that flag an effect as un-measured until the parser is fixed. - Use Case: After fine-tuning or steering a model, you run an A/B preference eval and the treated arm stops obeying "answer A or B only". This Skill grades the free-form outputs with a frozen judge across both orientations, revealing that the apparent 30% A-preference was a regex coercion artifact. ## Quick Start Use the multiple-choice-evaluation skill to grade my model's free-form answers on this A/B evaluation with an LLM judge instead of a letter regex, and report per-orientation results.
