Evaluate LLM and agent outputs using spec, tests, and eval layers.
npx skills add https://github.com/thimslugga/agent-skills --skill spec-test-eval
Please help me install this Agent Skill. Skill: spec-test-eval Source: https://github.com/thimslugga/agent-skills/tree/main/skills/agent/spec-test-eval Command: npx skills add https://github.com/thimslugga/agent-skills --skill spec-test-eval
Defines a disciplined framework to evaluate LLM and agent outputs by establishing explicit specs, binary tests, and graded evaluation. It helps teams design clear evaluation rubrics, compare models, and align judgments across evaluators.
Define your spec, derive tests from it, and apply evaluation functions to score outputs.
High-intent search queries and answers about installing and using this skill.
To design an evaluation rubric for LLM and agent outputs, define an explicit three-layer structure: establish the upstream spec, derive binary test functions for pass/fail gating, and apply evaluation functions for graded quality scoring.
The best way to compare multiple AI models and calibrate judgments across evaluators is to apply an anchored scoring scale within a structured spec-test-eval framework, ensuring consistent grading and clear pass/fail gating mechanisms.
Set up pass/fail gating mechanisms for evaluating AI responses by deriving binary test functions directly from your upstream spec, ensuring outputs meet explicit baseline requirements before undergoing graded quality scoring.
Yes, you can use this three-layer framework to benchmark different agents iteratively. It supports calibrating judgments across multiple evaluators and refining prompts or agent logic by anchoring scores against a consistent spec.
You need an upstream spec before writing test functions for AI evaluation because the spec serves as the source of truth. Test functions and graded evaluation scales are derived directly from this spec to ensure disciplined, aligned judgments.
tyevansCondenses prior LLM outputs or conversations into bullets, a paragraph, or a topic filter.
yanochka11Design LLM-as-Judge prompts with criteria, examples, and binary verdicts.
xurenluEvaluate and iteratively refine AI agent outputs through self-critique.
rohunvoraAutomate LLM-as-a-judge scoring, pairwise comparison, and rubric generation with bias mitigation.
