add-llm-evals

To evaluate LLM output quality in CI, you can automate an offline evaluation suite on a fixed dataset to measure correctness, faithfulness, and relevance. This ensures prompt and model quality on your pull requests.

RAG evaluation implements faithfulness, answer relevance, context precision, and context recall. These metrics measure how accurately the retrieval-augmented generation output aligns with the provided context.

Yes, offline LLM evaluation requires promptfoo, DeepEval, or Ragas. These dependencies provide the testing frameworks needed to measure output quality against fixed datasets.

Yes, online scoring optionally grades production LLM traffic for output quality. This integrates with observability platforms to measure correctness, safety, and relevance in real-time.

Reference-based LLM evaluation uses exact or fuzzy matching against ground truth, while reference-free evaluation uses semantic similarity and LLM-as-a-judge rubrics to score output quality without predefined correct answers.