skills.rest
latestaiagents
latestaiagentsOFFICIAL
Explore all skills by this author →
GitHub

rag-evaluation

Evaluate RAG systems with retrieval metrics and LLM-judged generation quality.

5|1|Updated Feb 4, 2026
Search Tags:#testing#performance#llm#rag#metrics#evaluation
One-click install
npx skills add https://github.com/latestaiagents/agent-skills --skill rag-evaluation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rag-evaluation
Source: https://github.com/latestaiagents/agent-skills/tree/main/plugins/rag-architect/skills/rag-evaluation
Command: npx skills add https://github.com/latestaiagents/agent-skills --skill rag-evaluation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires ragas, langchain-openai, datasets, numpy, json, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the critical need to rigorously evaluate and improve the performance of Retrieval Augmented Generation (RAG) systems by providing comprehensive metrics for retrieval, generation, and end-to-end quality.

Core Features & Use Cases

  • Retrieval Metrics: Calculate Mean Reciprocal Rank (MRR), Recall@k, Precision@k, NDCG@k, and Hit Rate to assess the quality of retrieved documents.
  • Generation Metrics: Utilize RAGAS or LLM-as-Judge to evaluate faithfulness, answer relevancy, context precision, context recall, and answer correctness.
  • End-to-End Testing: Simulate real-world usage to measure latency, cost, and overall system effectiveness.
  • Use Case: A RAG engineer needs to compare two different retrieval strategies. They use this skill to run both strategies against a golden dataset, generating detailed reports on which strategy yields better retrieval accuracy and more faithful, relevant answers.

Quick Start

Use the rag-evaluation skill to evaluate the RAG system's performance against the test dataset located at /path/to/test_dataset.json.

Frequently Asked Questions about rag-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate RAG system performance using RAGAS and LLM-as-Judge?▼

To evaluate RAG system performance, you can use this skill to calculate retrieval metrics like MRR and Recall@k, and assess generation quality by leveraging RAGAS and LLM-as-Judge for faithfulness and answer relevancy.

What metrics are needed to measure retrieval accuracy in a RAG pipeline?▼

Retrieval accuracy in a RAG pipeline is measured using metrics such as Mean Reciprocal Rank, Recall@k, Precision@k, NDCG@k, and Hit Rate to quantify the quality of retrieved documents.

Can I measure RAG latency and cost with end-to-end testing?▼

Yes, you can measure RAG latency and cost through end-to-end testing that simulates real-world usage, providing a comprehensive analysis of overall system effectiveness and operational expenses.

How do I compare different RAG retrieval strategies against a golden dataset?▼

You can compare different RAG retrieval strategies by running them against a golden dataset to generate detailed reports on retrieval accuracy, faithfulness, and answer relevancy.

Do I need langchain-openai and datasets dependencies for RAG evaluation?▼

Yes, this RAG evaluation framework requires dependencies including ragas, langchain-openai, datasets, numpy, and json to execute its automated testing pipelines and performance analysis.

Related Skills

Explore More Software Engineering Skills→
jackjin1997jackjin1997
11

langsmith-dataset

Generate and upload LangSmith trace-based evaluation datasets for LangChain agents.

Community
Intermediate
floflo777floflo777
33

rag-eval

Evaluate RAG systems using retrieval, generation, and latency metrics.

Community
Intermediate
manutejmanutej
61

langchain-orchestration

Orchestrate LangChain chains, agents, memory, and RAG patterns.

Community
Advanced
JeffallanJeffallan
10.9k

prompt-engineer

Design and optimize LLM prompts with structured outputs and evaluation frameworks.

Community
Advanced