Design and run evaluation harnesses for LLM agents and systems.
npx skills add https://github.com/yzavyas/claude-1337 --skill build-eval
Please help me install this Agent Skill. Skill: build-eval Source: https://github.com/yzavyas/claude-1337/tree/main/plugins/eval-1337/skills/build-eval Command: npx skills add https://github.com/yzavyas/claude-1337 --skill build-eval
This skill provides a structured framework to design, validate, and compare evaluations for LLM agents, multi-agent systems, skills, MCP servers, and prompts, reducing ambiguity in measurement and alignment.
Run a baseline evaluation harness against your codebase by assembling a dataset of test cases, choosing a framework, and executing the evaluator. For example: create a dataset with agent tasks, select DeepEval metrics, and run your evaluation harness.
High-intent search queries and answers about installing and using this skill.
Evaluating LLM agents requires building reproducible test harnesses with clearly defined activation criteria and labeled metrics. You can measure agent effectiveness and coordination using iterative metrics like TaskCompletion and ToolCorrectness.
Building a test suite for multi-agent systems involves assembling a dataset of test cases, selecting an evaluation framework like DeepEval or RAGAS, and executing the evaluator to generate traceable, reproducible results.
Yes, you can benchmark MCP server reliability by integrating DeepEval, RAGAS, Promptfoo, or Braintrust into your evaluation harness. These frameworks provide flexible scoring for measuring server performance.
Evaluating prompts requires labeled metrics to reduce ambiguity in alignment. You must choose a framework such as DeepEval, Braintrust, or RAGAS and establish reproducible harnesses with traceable results.
LLM agent evaluation uses end-to-end metrics including TaskCompletion, ToolCorrectness, and pass@k. These labeled metrics validate agent coordination and provide reproducible measurement of task effectiveness.
Reproducible harnesses are needed to ensure traceable results when validating LLM agents and prompts. They provide a structured framework to design, validate, and compare evaluations, reducing ambiguity in measurement.
Sir-chawakornBuilds an LLM evaluation harness using golden datasets and code-based scorers.
EntityProcessIteratively refine AI prompt files against AgentV evaluation datasets.
ContextJet-aiAdd offline and online evaluation scoring to LLM and agent applications.
hamelsmuEvaluate RAG pipelines by measuring retrieval metrics and generation quality.
