weave-eval

Name: weave-eval
Availability: InStock
Author: maximilien-ai

Benchmark RAG solutions by running datasets and evaluators.

Updated Mar 25, 2026

Search Tags:#rag #benchmark #dataset #evaluate #llm-judge #evaluator

Ready to run

npx skills add https://github.com/Maximilien-ai/weave-cli-skills --skill weave-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: weave-eval
Source: https://github.com/Maximilien-ai/weave-cli-skills/tree/main/skills/weave-eval
Command: npx skills add https://github.com/Maximilien-ai/weave-cli-skills --skill weave-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Provides a structured framework to benchmark, evaluate, and compare Retrieval-Augmented Generation (RAG) solutions across datasets, evaluators, and workflows.

Core Features & Use Cases

Evaluation Datasets: create, version, and manage test datasets for RAG evaluation.
Evaluators & LLM Judge: built-in evaluators and a pattern to integrate custom judges for quality scoring.
Benchmarks & Iteration: run multi-agent benchmarks, compare results, and iterate improvements.
Use Case Scenarios: baseline vs production-ready evals, QA vs summarization evaluation, cross-model comparisons.

Quick Start

Create a baseline dataset and run an initial evaluation against a selected agent to see baseline results

Frequently Asked Questions about weave-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I benchmark RAG experiments with automated evaluations?▼

Benchmark RAG experiments by running datasets against multiple agents and applying built-in evaluators. It structures the process to compare baseline versus production-ready results, analyze outputs, and iterate improvements.

What is an LLM judge and how does it evaluate RAG quality?▼

An LLM judge is an automated evaluator that scores RAG output quality. This framework provides built-in evaluators and patterns to integrate custom judges, enabling automated scoring across test datasets.

How do I create and manage datasets for RAG evaluation?▼

Create, version, and manage RAG evaluation datasets within the framework. It supports building baselines and test scenarios for QA or summarization, allowing you to run evaluations against selected agents.

Can I compare multiple agents in a single RAG benchmark run?▼

Yes, you can run multi-agent benchmarks to compare results across different models. The framework supports cross-model comparisons and iteration workflows to drive improvements from baseline to production readiness.

Do I need weave-cli to run RAG evaluations?▼

The framework integrates with weave-cli to support iteration workflows and manage evaluation processes. This integration enables running datasets and evaluators to analyze results and drive improvements.

What are the limitations of using automated evaluators for RAG?▼

Automated RAG evaluators rely on LLM judges for quality scoring, which may introduce model biases. Use the benchmark and iteration workflow to compare results across datasets and validate improvements before production readiness.