Evaluate RAG systems using retrieval, generation, and latency metrics.
npx skills add https://github.com/floflo777/claude-rag-skills --skill rag-eval
Please help me install this Agent Skill. Skill: rag-eval Source: https://github.com/floflo777/claude-rag-skills/tree/main/rag-eval Command: npx skills add https://github.com/floflo777/claude-rag-skills --skill rag-eval
This Skill enables rigorous evaluation of RAG systems by measuring retrieval, generation, and latency metrics to ensure quality and reliability.
Run a local evaluation with a prepared test dataset, then review the metrics in the generated report. If you have an Ailog API key, enable the benchmark to compare results against Ailog.
High-intent search queries and answers about installing and using this skill.
RAG evaluation measures retrieval quality by calculating recall, precision, MRR, and NDCG against a local test dataset of queries and gold answers. You provide the dataset, and the evaluation generates a comprehensive metrics report for your system.
RAG generation evaluation assesses faithfulness, relevance, coherence, and conciseness to measure output quality. It compares your system's generated answers against gold answers in your test dataset to quantify how accurately and relevantly the model responds.
You can benchmark your RAG system against a production baseline by enabling optional Ailog benchmarking with an API key. This compares your local evaluation metrics against Ailog's production RAG API to provide a competitive baseline in the final report.
End-to-end latency profiling measures the full performance of your RAG pipeline from retrieval to generation. It identifies bottlenecks by tracking latency metrics alongside retrieval and generation quality scores in the comprehensive evaluation report.
No external API key is required for local RAG evaluation. You can run tests against your own dataset of queries and gold answers to obtain retrieval and generation metrics, with the Ailog API key being entirely optional for production benchmarking.
The best way to quantify RAG quality is using standardized evaluation metrics covering retrieval, generation, and latency. This approach combines local dataset testing with optional benchmarking to generate a comprehensive report ensuring system reliability.
floflo777Evaluate RAG systems using retrieval, generation, and latency metrics.
MantisWareEvaluate RAG pipelines with retrieval and generation metrics and bottleneck analysis.
michaelalberScaffold end-to-end RAG pipelines for local or cloud LLMs.
chicanoandres702Evaluate LLM outputs with automated metrics, human judgments, and A/B testing.
