skills.rest
BagelHole
BagelHoleCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

rag-observability-evals

Monitor RAG systems with retrieval and generation quality metrics.

46|4|Updated Jan 27, 2026
Search Tags:#llm#rag#evaluation#retrieval#observability#hallucination#groundedness
One-click install
npx skills add https://github.com/BagelHole/DevOps-Security-Agent-Skills --skill rag-observability-evals
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rag-observability-evals
Source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/tree/main/devops/ai/rag-observability-evals
Command: npx skills add https://github.com/BagelHole/DevOps-Security-Agent-Skills --skill rag-observability-evals

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of treating Retrieval-Augmented Generation (RAG) systems as black boxes, enabling users to monitor, evaluate, and ensure the reliability and quality of RAG outputs in production environments.

Core Features & Use Cases

  • Retrieval Quality Metrics: Tracks recall, MRR, citation coverage, and embedding drift.
  • Generation Quality Checks: Assesses groundedness, hallucination rates, and instruction adherence.
  • Reliability & Cost Monitoring: Analyzes latency, token usage, and cache performance.
  • Use Case: A team deploying a RAG-powered customer support bot can use this skill to continuously monitor if the bot's answers are factually supported by retrieved documents and to detect any increase in hallucinations or response latency.

Quick Start

Begin evaluating the RAG system by running the offline evaluation pipeline.

Frequently Asked Questions about rag-observability-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I measure hallucination and groundedness in RAG systems?▼

Hallucination and groundedness in RAG systems are measured by evaluating generation accuracy against retrieved documents. This skill assesses whether LLM outputs are factually supported and calculates hallucination rates to ensure reliability.

What metrics are used for RAG observability and retrieval quality?▼

RAG observability tracks retrieval quality using metrics like recall, MRR, citation coverage, and embedding drift. These metrics reveal how accurately the system fetches relevant context before generation.

How do I run continuous regression testing on my RAG pipeline?▼

Continuous regression testing for RAG pipelines runs offline evaluations to monitor system reliability. It alerts on performance degradation by analyzing latency, token usage, cache performance, and generation accuracy.

Can I monitor RAG system latency and token usage in production?▼

Yes, RAG system latency and token usage can be monitored in production. The skill analyzes reliability and cost metrics, tracking response times and cache performance to detect operational degradation.

Why does my RAG system return unsupported answers?▼

RAG systems return unsupported answers when generation lacks groundedness in retrieved documents. Evaluating instruction adherence and citation coverage helps detect factual inconsistencies and increased hallucination rates.

Does this RAG evaluation tool require external dependencies?▼

No external dependencies are required to use this RAG evaluation tool. It operates independently to measure retrieval quality, generation accuracy, and system reliability without needing additional frameworks.

Related Skills

Explore More Software Engineering Skills→
floflo777floflo777
33

rag-eval

Evaluate RAG systems using retrieval, generation, and latency metrics.

Community
Intermediate
bradtaylorsfbradtaylorsf
8

evaluation-frameworks

Evaluate AI models, code quality, and agent performance with structured frameworks.

Community
Advanced
JCETools-PetraJCETools-Petra
16

ai-llm-engineering

Design RAG pipelines with chunking, embedding, retrieval, and evaluation metrics.

Community
Advanced
roanbrasilroanbrasil
5

llm-observability

Trace LLM calls and track token costs across LangSmith, LangFuse, and Phoenix.

Community
Advanced