Compares AI agent answers across three knowledge-access levels and scores them.
npx skills add https://github.com/blueraai/bluera-knowledge --skill eval-blueraai
Please help me install this Agent Skill. Skill: eval Source: https://github.com/blueraai/bluera-knowledge/tree/main/skills/eval Command: npx skills add https://github.com/blueraai/bluera-knowledge --skill eval-blueraai
This Skill rigorously evaluates and compares the performance of AI agents in answering questions, specifically highlighting the impact of different levels of access to a local knowledge base (BK).
Run an evaluation of agent quality for the query "How does Express handle errors?".
High-intent search queries and answers about installing and using this skill.
To benchmark agent performance with and without a knowledge base, you run a comparative evaluation across three modes: no knowledge base access, file system access only, and full vector search access. This provides detailed scoring on accuracy and token usage.
Evaluating AI agent quality uses a scoring rubric that measures accuracy, specificity, completeness, and source grounding. It also tracks performance metrics including token usage and duration to quantify improvements.
You test the impact of vector search by running predefined or arbitrary queries through parallel agents with differing access levels. Comparing the outputs reveals quantifiable improvements in answer quality when full knowledge base search is enabled.
Yes, you can evaluate agent responses using only file system access. The comparative analysis runs one agent specifically with grep, read, and glob commands to see how it performs against both no access and full vector search modes.
The best way to compare AI agents across different knowledge base modes is to run them in parallel on identical queries and assess the results using a standardized rubric for accuracy, specificity, completeness, and source grounding.
48Nauts-OperatorImplement hybrid search combining vector similarity with BM25 ranking.
PostalocityDesign and optimize RAG pipelines with embeddings, chunking, and vector stores.
oakossImplements RAG pipelines with hybrid search and re-ranking.
apassuelloEvaluate LLM outputs with automated metrics, human judgments, and benchmarking.
