agentsop-metric-design

To stop LLM-as-judge optimization from gaming biased evaluation metrics, decompose holistic criteria into 3–6 orthogonal yes/no sub-judges and apply deterministic length penalties so verbosity cannot win the optimization.

Compile and eval semantics in DSPy metric functions require returning strict bool during compilation for stable optimizer search, and float during evaluation or reporting to provide continuous scoring granularity.

Calibrate RAGAS and LlamaIndex evaluation metrics against humans by running bias-probe checks for length, self-preference, position, and rubric-order on at least 20 spot-checks before compiling any optimizer.

Yes, this metric design approach supports RAG evaluation triads by aligning metrics to Faithfulness, Relevancy, and Retriever quality so systems do not look faithful while remaining wrong or irrelevant.

The best way to mitigate judge bias in LLM pipeline evaluators is to break judging into orthogonal sub-judges, apply explicit scalar length control, and run cross-family bias probes with human calibration.

Yes, you can pass decomposed evaluation metrics into DSPy or MIPRO optimizers by generating a metric function and calibration receipt, then feeding that metric into your optimizer loop to quantitatively gate and regress changes.