agentsop-domain-eval-set

What problem does it solve?

It solves the risk of shipping an LLM/RAG/agent system based on public benchmarks or demos that do not predict performance on your actual users and data distribution.

Core Features & Use Cases

• Source real held-out examples from production traffic (tickets, logs, queries, transactions) with stratification and deliberate edge-case coverage.
• Label and curate with agreement using gold/reference/pass-fail judgments (including gold passages for RAG), while rejecting genuinely ambiguous items.
• Seal, size, and version the benchmark as a 50–200 example artifact with leak audit and refresh-on-drift so it remains a trustworthy regression gate.

Quick Start

Ask an AI agent to create a 50–200 example held-out domain benchmark from your most recent production traffic, label the hard cases with agreement, seal the test split, version it, and produce a runbook for leak auditing and refresh cadence.