Compute precision, recall, MRR, nDCG, and MAP for file retrieval comparisons.
npx skills add https://github.com/sourcegraph/CodeScaleBench --skill ir-analysis
Please help me install this Agent Skill. Skill: ir-analysis Source: https://github.com/sourcegraph/CodeScaleBench/tree/main/skills/ir-analysis Command: npx skills add https://github.com/sourcegraph/CodeScaleBench --skill ir-analysis
This Skill quantifies how effectively AI agents retrieve relevant files from a codebase, comparing different retrieval methods against a known ground truth.
Run the ir-analysis skill to compute retrieval metrics for all benchmarks.
High-intent search queries and answers about installing and using this skill.
You measure retrieval quality by comparing file access results against known ground truth data to calculate precision, recall, MRR, nDCG, and MAP metrics for different configurations.
Comparing retrieval methods involves evaluating file recall performance between baseline local tools and advanced semantic search configurations to see which finds correct files faster.
Evaluating ground truth data provides a known baseline to calculate recall and precision, verifying whether an AI agent successfully retrieves the relevant files needed for a task.
Yes, you can calculate MRR and nDCG natively, as the analysis operates without external dependencies to directly compute information retrieval metrics from benchmark ground truth data.
You run nDCG and MAP evaluations when you need to quantify how effectively an AI agent retrieves relevant files from a codebase across different retrieval configurations.
prashsubGenerate and validate Genie Space benchmark questions with ground truth SQL.
JoaquinCampoEvaluate sparse retrieval models on BEIR, MIRACL, and mMARCO benchmarks.
jasonmeansIntegrates with Sourcegraph to search code, commits, diffs, symbols, and DeepSearch across repositories.
SyntaxAsSpiralPersist AI agent context across sessions with vector stores and temporal knowledge graphs.
