Design AI/LLM feature evaluation plans with test sets, rubrics, and judging approaches.
npx skills add https://github.com/oldwinter/skills --skill ai-evals
Please help me install this Agent Skill. Skill: ai-evals Source: https://github.com/oldwinter/skills/tree/main/lenny-skills/ai-evals Command: npx skills add https://github.com/oldwinter/skills --skill ai-evals
This Skill provides a structured framework for designing and executing evaluations of AI/LLM features, ensuring quality, safety, and performance before deployment.
Use the ai-evals skill to design an evaluation plan for a new AI feature.
High-intent search queries and answers about installing and using this skill.
Designing an LLM evaluation requires a structured process to define evaluation requirements, create golden test sets, develop scoring rubrics, and plan judging harnesses for quality assurance.
An error taxonomy categorizes failure modes identified during LLM testing. You need it to structure error analysis and build targeted golden test sets that ensure AI quality and safety before deployment.
Creating a scoring rubric for LLM-as-judge involves defining specific evaluation criteria and acceptance thresholds. This structured approach ensures consistent quality gates and reliable automated judging harnesses.
The best way to build a golden test set is through failure analysis, mapping outputs to an error taxonomy. This operationalizes AI evaluation by providing a structured benchmark for measuring LLM quality.
Yes, you can define acceptance thresholds by creating an evaluation PRD. This document establishes clear scope and quality gates to confidently verify that AI features meet safety and performance standards.
LLM testing lacks consistent quality assurance without a structured evaluation framework. Operationalizing rubric design, error analysis, and judging harnesses establishes the necessary quality gates for confident shipping.
yu13130122297Identify uncertainties in research framing, datasets, metrics, and baselines.
liqiongyu...
abdullah1854Develop evaluation frameworks for non-deterministic agent systems.
shreyashankarAutomate error analysis and data review on structured datasets.
