skills.rest
langwatch
langwatchOFFICIAL
Explore all skills by this author →
GitHub

generate-rag-dataset

Generate synthetic Q&A evaluation datasets from a RAG knowledge base for LangWatch experiments.

3.5k|362|Updated Sep 9, 2023
Search Tags:#question generation#synthetic data#csv export#evaluation dataset#LLM testing#rAG#langwatch
One-click install
npx skills add https://github.com/langwatch/langwatch --skill generate-rag-dataset
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: generate-rag-dataset
Source: https://github.com/langwatch/langwatch/tree/main/skills/_compiled/native/generate-rag-dataset
Command: npx skills add https://github.com/langwatch/langwatch --skill generate-rag-dataset

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas.

What problem does it solve?

Building test data for a RAG pipeline by hand is slow and often produces shallow, repetitive questions that miss edge cases. This Skill analyzes your actual knowledge base and generates a diverse, ready-to-use evaluation dataset with expected answers and supporting context.

Core Features & Use Cases

  • Knowledge Base Analysis: Reads your documents, database schemas, vector store configuration, and chunking strategy to understand what the knowledge base covers.
  • Diverse Question Generation: Creates factual recall, multi-hop reasoning, comparison, edge case, and negative ("I don't know") questions to thoroughly test retrieval and answer quality.
  • Dual Export Formats: Produces a pandas DataFrame/CSV for LangWatch SDK experiments and a platform-ready CSV with input, expected_output, context, and question_type columns.
  • Use Case: You have a RAG chatbot over agricultural documentation and need regression test data before changing your prompt. The Skill reads the docs, generates 50+ varied Q&A pairs with source context, and exports a CSV you import directly into LangWatch to baseline accuracy.

Quick Start

Ask the agent to analyze your RAG knowledge base and generate a diverse evaluation dataset with context, exported as a LangWatch-ready CSV.

Frequently Asked Questions about generate-rag-dataset

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate test data for a RAG pipeline?▼

Point the Skill at your knowledge base documents, vector store config, and chunking setup. It reads the content and generates diverse Q&A pairs with expected answers and source context, exported as CSV for LangWatch experiments.

What question types should a RAG evaluation dataset include?▼

Include factual recall, multi-hop reasoning, comparison, edge cases, and negative cases where the answer is not in the knowledge base. Negative cases verify the system says "I don't know" instead of hallucinating.

Can I import the generated dataset into LangWatch?▼

Yes. The Skill exports a platform-ready CSV with input, expected_output, context, and question_type columns that imports directly into LangWatch datasets, plus a pandas DataFrame for SDK-based experiments.

Why include context in each RAG evaluation row?▼

Including the relevant document chunk per row lets you evaluate answer quality independently of retrieval quality. You can run platform experiments and test different prompts against the same retrieved context.

What are common mistakes when building RAG evaluation datasets?▼

Common mistakes include generating questions without reading the knowledge base, skipping negative cases, using one repetitive question pattern, omitting context, and writing expected answers not grounded in the source documents.

Related Skills

Explore More Data & Analytics Skills→
JayRHaJayRHa
3

rag-pipeline-designer

Design end-to-end RAG pipelines with chunking, embeddings, retrieval, and evaluation.

Community
Advanced
Maximilien-aiMaximilien-ai

weave-eval

Benchmark RAG solutions by running datasets and evaluators.

Official
Advanced
omarluqomarluq
5

golang-observability

Instrument Go services with logs, metrics, traces, and profiling.

Community
Advanced
bobmatnycbobmatnyc
68

datadog-observability

Integrate Datadog APM, logs, metrics, synthetics, and RUM for full-stack observability.

Community
Advanced