What problem does it solve? Automated evaluation of LLM outputs suffers from inconsistent scoring, position bias, length bias, and poor correlation with human judgment, making it hard to trust quality assessments at scale. ## Core Features & Use Cases - Direct Scoring and Pairwise Comparison: Apply structured scoring prompts with justification-first reasoning, or compare two responses with position-swapping to detect and neutralize position bias. - Bias Mitigation Toolkit: Counter position, length, self-enhancement, verbosity, and authority biases using cross-model evaluation, anonymization, and length-normalized scoring. - Rubric Generation and Metric Selection: Generate domain-specific scoring rubrics with level descriptions and edge cases, and validate judges with Spearman's rho, Cohen's kappa, and agreement metrics. - Use Case: When A/B testing a new prompt, run pairwise comparisons with position swapping across both variants, then report win rates with position consistency to confirm the result is not an artifact of ordering. ## Quick Start Ask the AI to compare two model responses to a prompt using pairwise comparison with position swapping and return the winner with a calibrated confidence score.
