Evaluate RAG pipelines with retrieval and generation metrics and bottleneck analysis.
npx skills add https://github.com/MantisWare/BizForge --skill eval-rag-mantisware
Please help me install this Agent Skill. Skill: eval-rag Source: https://github.com/MantisWare/BizForge/tree/main/library/skills/ai-patterns/eval-rag Command: npx skills add https://github.com/MantisWare/BizForge --skill eval-rag-mantisware
Quickly quantify the quality of Retrieval-Augmented Generation pipelines by evaluating both retrieval and generation components and tracing end-to-end bottlenecks.
Run the evaluation against your RAG setup by providing the pipeline config, queries, and golden answers.
High-intent search queries and answers about installing and using this skill.
To evaluate a Retrieval-Augmented Generation pipeline, you need a defined pipeline configuration, a set of queries, and golden answers to measure retrieval and generation components. This computes retrieval metrics like recall and precision, alongside generation metrics like faithfulness and relevance.
RAG evaluation uses retrieval metrics including recall, precision, MRR, and NDCG to verify fetched documents. Generation metrics include faithfulness, relevance, and completeness, which are measured against the retrieved context to ensure accurate responses.
To identify RAG pipeline bottlenecks, run an evaluation that analyzes failures in both retrieval and generation components. This structured analysis isolates whether issues arise from fetching irrelevant documents or from generation errors, providing recommendations to guide improvements.
You can compare different RAG configurations by running evaluations that compute per-query and aggregate metrics. This allows you to benchmark retrieval and generation performance across setups, helping you identify the optimal pipeline configuration.
Yes, golden answers are required to benchmark retrieval-augmented generation. Providing a set of queries with their corresponding golden answers enables the computation of generation metrics like faithfulness and completeness, ensuring accurate end-to-end evaluation.
Miosa-osaEvaluate RAG pipelines with retrieval and generation metrics plus bottleneck analysis.
sqfzyAutomate benchmarks and profiling to identify performance bottlenecks across commits.
HemantSudarshanOrchestrate document retrieval and generation for end-to-end RAG workflows.
chicanoandres702Implement distributed tracing across microservices using Jaeger and Tempo.
