llm-evaluation

Automated metrics quantify LLM performance by comparing generated outputs against reference text or ground truth. ROUGE measures token overlap; BERTScore uses semantic embeddings for meaning-based comparison. Define your metric suite, prepare test cases with expected outputs, run evaluation against your model, and review aggregate scores to measure quality and detect regressions.

BLEU counts n-gram matches and favors exact word overlap; ROUGE measures recall of overlapping text spans; BERTScore compares semantic similarity using contextual embeddings, capturing meaning even with paraphrasing. Choose BLEU for strict phrasing tasks, ROUGE for summarization, and BERTScore for fluency-tolerant evaluation.

Yes. LLM-as-Judge workflows evaluate outputs without requiring reference text, using pointwise scoring (single output rated independently) or pairwise comparison (choosing between two outputs). Reference-free evaluation suits open-ended tasks where ground truth is unavailable; reference-based metrics work better for deterministic tasks with known correct answers.

Define evaluation dimensions (accuracy, coherence, relevance, fluency, safety), prepare identical test cases, run both variants through your metrics suite, and compare aggregate results. Benchmarking tracks baselines across development and production; A/B testing validates whether prompt changes or model swaps improve measurable performance.

Retrieval metrics measure ranking quality: MRR (Mean Reciprocal Rank) scores first-relevant-item position, NDCG penalizes relevant items ranked lower, Precision@K and Recall@K measure coverage within top-K results. Use these alongside generation metrics like BERTScore to evaluate both retrieval effectiveness and answer quality.

Automated metrics catch obvious failures but miss nuance. Human evaluation on dimensions like coherence, safety, usefulness, and relevance captures quality automated metrics miss. Combine both: use automated metrics for scale and regression detection, reserve human evaluation for validation, model selection, and edge cases.