libeval

You evaluate RAG agent performance automatically by running an evaluation suite that scores responses using LLM-as-judge criteria, validates retrieval recall, and analyzes execution traces. The suite orchestrates tests across multiple scenarios and models, persisting results for benchmarking and reporting.

LLM-as-judge evaluation measures response quality in a RAG system by scoring outputs against predefined rubrics. It uses an LLM to act as a judge, coordinating criteria-based scoring alongside retrieval recall and execution trace analyses to ensure comprehensive quality testing.

Recall analysis validates retrieval coverage in RAG pipelines by examining memory windows and stored subjects. It checks whether the agent retrieves necessary information, ensuring the memory component provides adequate context for accurate response generation.

Yes, you need specific service clients to run RAG evaluation and trace analysis. The process requires configured clients for the agent, memory, and trace services, an LLM token, a prompt loader, an evaluation index, and config-defined evaluation scenarios.

Yes, you can benchmark multiple models across different RAG evaluation scenarios. The evaluator orchestrates runs across various scenarios and models, and the result store persists these outcomes to enable comprehensive reporting and continuous benchmarking.

The best way to analyze execution traces for RAG safety and correctness is using a dedicated trace evaluator. This component examines execution traces to assess agent performance, safety parameters, and correctness, persisting results alongside recall and criteria scores for reporting.