weave-eval

What problem does it solve?

Provides a structured framework to benchmark, evaluate, and compare Retrieval-Augmented Generation (RAG) solutions across datasets, evaluators, and workflows.

Core Features & Use Cases

• Evaluation Datasets: create, version, and manage test datasets for RAG evaluation.
• Evaluators & LLM Judge: built-in evaluators and a pattern to integrate custom judges for quality scoring.
• Benchmarks & Iteration: run multi-agent benchmarks, compare results, and iterate improvements.
• Use Case Scenarios: baseline vs production-ready evals, QA vs summarization evaluation, cross-model comparisons.

Quick Start

Create a baseline dataset and run an initial evaluation against a selected agent to see baseline results