skills.rest
Maximilien-ai
Maximilien-aiOFFICIAL
Explore all skills by this author →
WebsiteGitHub

weave-eval

Benchmark RAG solutions by running datasets and evaluators.

Updated Mar 25, 2026
Search Tags:#rag#benchmark#dataset#evaluate#llm-judge#evaluator
One-click install
npx skills add https://github.com/Maximilien-ai/weave-cli-skills --skill weave-eval
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: weave-eval
Source: https://github.com/Maximilien-ai/weave-cli-skills/tree/main/skills/weave-eval
Command: npx skills add https://github.com/Maximilien-ai/weave-cli-skills --skill weave-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Provides a structured framework to benchmark, evaluate, and compare Retrieval-Augmented Generation (RAG) solutions across datasets, evaluators, and workflows.

Core Features & Use Cases

  • Evaluation Datasets: create, version, and manage test datasets for RAG evaluation.
  • Evaluators & LLM Judge: built-in evaluators and a pattern to integrate custom judges for quality scoring.
  • Benchmarks & Iteration: run multi-agent benchmarks, compare results, and iterate improvements.
  • Use Case Scenarios: baseline vs production-ready evals, QA vs summarization evaluation, cross-model comparisons.

Quick Start

Create a baseline dataset and run an initial evaluation against a selected agent to see baseline results

Frequently Asked Questions about weave-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark RAG experiments with automated evaluations?▼

Benchmark RAG experiments by running datasets against multiple agents and applying built-in evaluators. It structures the process to compare baseline versus production-ready results, analyze outputs, and iterate improvements.

What is an LLM judge and how does it evaluate RAG quality?▼

An LLM judge is an automated evaluator that scores RAG output quality. This framework provides built-in evaluators and patterns to integrate custom judges, enabling automated scoring across test datasets.

How do I create and manage datasets for RAG evaluation?▼

Create, version, and manage RAG evaluation datasets within the framework. It supports building baselines and test scenarios for QA or summarization, allowing you to run evaluations against selected agents.

Can I compare multiple agents in a single RAG benchmark run?▼

Yes, you can run multi-agent benchmarks to compare results across different models. The framework supports cross-model comparisons and iteration workflows to drive improvements from baseline to production readiness.

Do I need weave-cli to run RAG evaluations?▼

The framework integrates with weave-cli to support iteration workflows and manage evaluation processes. This integration enables running datasets and evaluators to analyze results and drive improvements.

What are the limitations of using automated evaluators for RAG?▼

Automated RAG evaluators rely on LLM judges for quality scoring, which may introduce model biases. Use the benchmark and iteration workflow to compare results across datasets and validate improvements before production readiness.

Related Skills

Explore More Data & Analytics Skills→
ken-cavanagh-gleanken-cavanagh-glean
4

evaluation

Design evaluation frameworks with rubrics and LLM-as-judge workflows.

Community
Advanced
ginmp8ginmp8

skill-benchmark

Generate standardized benchmark reports for reusable agent skill packages.

Community
Advanced
PSkinnerTechPSkinnerTech

building-with-llms

Architect prompting, agent orchestration, and evaluation frameworks for LLM applications.

Community
Advanced
roanbrasilroanbrasil
5

llm-evals

Implement LLM evaluation patterns with LangSmith, RAGAS, and PromptFoo.

Community
Advanced