llm-eval-harness

Generate synthetic ground truth with Ragas TestsetGenerator, mixing simple, reasoning, and multi_context evolution types for question diversity. Use a strong critic LLM distinct from the generator to vet rows, and human-review every case before versioning the dataset.

BLEU and ROUGE are cheap directional signals across versions but miss coherence and hallucinations. Reserve LLM-as-judge for qualitative criteria like tone and safety, and add faithfulness scoring when hallucination detection against provided context matters.

Set temperature to 0, score one qualitative aspect per metric, and keep the same judge model across all runs. Use at least five diverse few-shot examples with chain-of-thought reasoning, and hand-verify 3-5 judged cases before trusting aggregates.

Yes, any reference-free metric such as faithfulness or answer relevance can be reused at runtime as a guardrail with the same threshold. The model responds only when the metric clears its threshold, and the live pass-rate becomes a monitored drift signal.

Model-level evaluation treats the checkpoint as the unit under test, scoring it against curated datasets before registry promotion. Application-level evaluation instruments a running system with tracing spans and production scoring, which belongs to a separate application-eval skill.

A green component suite with a red end-to-end suite indicates a wiring bug, not a model bug, such as a broken prompt template hand-off between pipeline stages. Always run both component and end-to-end evaluations to isolate where failures originate.