skills.rest
zhang-ming-hui
zhang-ming-huiCOMMUNITY
Explore all skills by this author →
GitHub

retrieval-eval-designer

Maintain evaluation datasets, failure buckets, and relevance metrics for skills retrieval.

Updated May 28, 2026
Search Tags:#acceptance criteria#retrieval evaluation#failure bucketing#relevance metrics#benchmark dataset#query type labeling#skills taxonomy
One-click install
npx skills add https://github.com/zhang-ming-hui/ackownledge --skill retrieval-eval-designer
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: retrieval-eval-designer
Source: https://github.com/zhang-ming-hui/ackownledge/tree/main/ir_system/.agents/skills/retrieval-eval-designer
Command: npx skills add https://github.com/zhang-ming-hui/ackownledge --skill retrieval-eval-designer

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It helps you systematically improve a skills retrieval system by making evaluation coverage measurable and failure cases easy to triage.

Core Features & Use Cases

  • Benchmark expansion: Add realistic benchmark queries tied to the current set of skill targets.
  • Failure bucketing: Cluster retrieval failures by likely root cause to guide the next engineering or data task.
  • Metrics-driven iteration: Use runtime metrics and failure reports to compare changes before and after updates.
  • Schema and labeling discipline: Enforce consistent labeling such as query_type and keep expected skill names grounded in the dataset.

Quick Start

Use the retrieval-eval-designer skill to expand your evaluation cases by adding new eval/*.json entries and updating runtime/failure_buckets.json based on runtime/metrics_report.json.

Frequently Asked Questions about retrieval-eval-designer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I bucket retrieval failures by root cause for my search system?▼

To bucket retrieval failures, ingest runtime metrics reports and cluster failure cases into buckets by likely root cause. This guides the next engineering or data task by making triage measurable and reproducible.

What is the best way to expand benchmark queries for a skills retrieval system?▼

Expanding benchmark queries involves adding realistic eval JSON entries tied to current skill targets. Proper labeling with query_type ensures consistent schema discipline and improves query-type coverage across the dataset.

How do I use relevance metrics to compare retrieval changes before and after updates?▼

Relevance metrics from runtime reports enable metrics-driven iteration by comparing system performance before and after updates. This approach uses failure reports to quantify improvements and maintain reproducible evaluation updates.

Do I need specific JSON formats to define acceptance criteria for retrieval evaluation?▼

Defining acceptance criteria requires ingesting eval JSON inputs and runtime reports. Cases are labeled by query_type and aligned with expected skill names to produce grounded, reproducible evaluation updates.

Why does my skills taxonomy retrieval evaluation lack query-type coverage?▼

Lack of query-type coverage happens when benchmark queries are not properly labeled by query_type. Enforcing schema and labeling discipline during benchmark expansion ensures measurable coverage across all skill targets.

Related Skills

Explore More Data & Analytics Skills→
zhang-ming-huizhang-ming-hui

project-completion-reviewer

Run retrieval checks and triage failure buckets for project completion decisions.

Community
Intermediate
aayushsoamaayushsoam
1

eval-harness

Evaluate Claude Code sessions with pass/fail criteria and regression metrics.

Community
Advanced
Miosa-osaMiosa-osa
226

eval-rag

Evaluate RAG pipelines with retrieval and generation metrics plus bottleneck analysis.

Official
Advanced
sunholo-datasunholo-data
33

Eval Analyzer

Analyze AILANG evaluation baselines to categorize failures and compare model performance.

Official
Intermediate