retrieval-judge

Evaluate recall_search results for relevance and log judgments via flight_recorder_log.

Updated Jan 30, 2026
One-click install
npx skills add https://github.com/jyang234/ai-engineering-framework --skill retrieval-judge
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: retrieval-judge
Source: https://github.com/jyang234/ai-engineering-framework/tree/main/edi/internal/assets/skills/retrieval-judge
Command: npx skills add https://github.com/jyang234/ai-engineering-framework --skill retrieval-judge

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

After recall_search, this Skill enforces a critical judgment step to filter results by relevance rather than treating them as authoritative, reducing noise in knowledge retrieval.

Core Features & Use Cases

  • Evaluation checklist: Checks title/type match, content relevance, and applicability to the current task.
  • Auditable decisions: Logs judgments and keeps a clean, reproducible trail for review.
  • Use Case: When you run recall_search in a knowledge-assembly workflow, apply this Skill to discard irrelevant results and keep only those that address the user's query.

Quick Start

After performing recall_search, apply the evaluation steps to each result, log the outcome with flight_recorder_log, and proceed with the kept results.

Frequently Asked Questions about retrieval-judge

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I filter recall search results for relevance?

Recall search returns many candidates, but not all are relevant. After recall_search, apply evaluation criteria to filter results—checking title match, content relevance, and task fit—then log decisions via flight_recorder_log to keep only directly addressing results and maintain an audit trail.

What's the best way to reduce noise in knowledge retrieval workflows?

Knowledge retrieval often surfaces irrelevant results. Apply a standardized evaluation checklist to each recall_search result, log judgments for traceability, and discard misaligned items. This critical filtering step ensures downstream processes work with high-confidence data.

How do I audit and trace relevance decisions in recall workflows?

Relevance judgments are auditable decisions. This Skill enforces standardized evaluation against a checklist and logs each judgment via flight_recorder_log, creating a reproducible trail for review and verification of which results were kept and why.

Can I apply relevance filtering across different domains?

Yes. The evaluation checklist—title/type match, content relevance, task applicability—generalizes across domains. After recall_search in any knowledge-assembly workflow, apply the same standardized judgment step to filter noise consistently.

Why should I filter recall results instead of treating them as authoritative?

Recall search prioritizes coverage over precision, returning both relevant and tangential matches. Filtering by relevance reduces noise, ensures only directly addressing results feed downstream tasks, and prevents low-confidence data from propagating through your workflow.

What evaluation criteria should I use to judge search result relevance?

The Skill applies three core checks: does the result's title or type match the query, is the content substantively relevant, and does it address the current task. These criteria filter out false positives and keep only results that directly serve the user's need.