Evaluate RAG pipelines with retrieval and generation metrics plus bottleneck analysis.
npx skills add https://github.com/Miosa-osa/canopy --skill eval-rag-miosa-osa
Please help me install this Agent Skill. Skill: eval-rag Source: https://github.com/Miosa-osa/canopy/tree/main/library/skills/ai-patterns/eval-rag Command: npx skills add https://github.com/Miosa-osa/canopy --skill eval-rag-miosa-osa
Assess retrieval and generation quality in Retrieval-Augmented Generation pipelines by measuring retrieval effectiveness (recall, precision, MRR, NDCG) and generation fidelity (faithfulness, relevance, completeness), then surfacing bottlenecks for targeted improvements.
Run /eval-rag with your pipeline configuration, queries, and golden answers to generate a comprehensive evaluation report.
High-intent search queries and answers about installing and using this skill.
To evaluate RAG pipeline quality, you measure retrieval effectiveness using metrics like recall, precision, MRR, and NDCG, alongside generation fidelity metrics for faithfulness, relevance, and completeness. This end-to-end evaluation requires a configured pipeline, test queries, and golden answers.
Diagnosing RAG bottlenecks involves running end-to-end evaluations with golden answers to perform per-query analysis. This separates retrieval issues from generation issues, allowing you to identify exactly which stage causes poor performance and target improvements effectively.
Yes, you can compare multiple RAG pipelines by applying cross-pipeline configuration comparisons. Supplying your queries and golden answers allows the evaluation to surface performance differences between configurations, highlighting variations in retrieval and generation metrics.
RAG retrieval evaluation uses Recall, Precision, MRR, and NDCG to measure document fetching effectiveness. Generation evaluation uses Faithfulness, Relevance, and Completeness metrics to assess the final output quality against provided golden answers.
Yes, you need golden answers and evaluation queries to run a comprehensive RAG evaluation. These inputs are required to accurately measure generation fidelity and calculate retrieval metrics across your configured pipeline.
kinnerkarmanishEvaluate RAG pipelines by scoring retrieval and generation against gold references.
MantisWareEvaluate RAG pipelines with retrieval and generation metrics and bottleneck analysis.
Adam-GuerinGuide RAG system implementation covering embeddings, vector databases, chunking, retrieval, and LLM integration.
hamelsmuDesign LLM-as-Judge evaluators for subjective criteria with binary pass/fail outcomes.
