retrieval-eval-designer

What problem does it solve?

It helps you systematically improve a skills retrieval system by making evaluation coverage measurable and failure cases easy to triage.

Core Features & Use Cases

• Benchmark expansion: Add realistic benchmark queries tied to the current set of skill targets.
• Failure bucketing: Cluster retrieval failures by likely root cause to guide the next engineering or data task.
• Metrics-driven iteration: Use runtime metrics and failure reports to compare changes before and after updates.
• Schema and labeling discipline: Enforce consistent labeling such as query_type and keep expected skill names grounded in the dataset.

Quick Start

Use the retrieval-eval-designer skill to expand your evaluation cases by adding new eval/*.json entries and updating runtime/failure_buckets.json based on runtime/metrics_report.json.