llm-evaluation

Use automated metrics like BLEU and ROUGE for text overlap, BERTScore for semantic similarity, and custom metrics for groundedness or toxicity. Combine them in an evaluation suite that runs your model against test cases and aggregates mean scores per metric.

LLM-as-judge uses a stronger model to score or compare outputs from another model. Use it for qualities hard to compute automatically, such as helpfulness or clarity, via pointwise scoring, pairwise comparison, or reference-based evaluation with structured JSON ratings.

Store baseline metric results and compare new evaluation runs against them using a relative-change threshold, such as 5%. Flag any metric whose score drops beyond the threshold, and integrate this check into your CI/CD pipeline for continuous evaluation.

Yes. LangSmith lets you create datasets of test examples, attach built-in evaluators like qa and context_qa, and run experiments against your chain. Results include aggregate metrics and are versioned with experiment metadata for tracking over time.

Use Cohen's kappa to quantify inter-rater agreement on annotation scores. Values below 0.2 indicate slight agreement, 0.4-0.6 moderate, and above 0.8 almost perfect, helping you judge whether your annotation guidelines are reliable.

Common pitfalls include optimizing a single metric, using too few test examples, testing on training data, ignoring statistical variance, and position bias in pairwise comparisons. Use multiple metrics, representative data, and proper statistical tests like t-tests with effect sizes.