skills.rest
Kaleb-Rupe
Kaleb-RupeCOMMUNITY
Explore all skills by this author →
𝕏@MightieMagsGitHub

Evals

Evaluates AI agent workflows using code, model, and human graders with pass@k scoring.

Updated Jul 10, 2026
Search Tags:#evaluation#regression testing#llm-judge#pass@k#graders#agent benchmarking#prompt comparison
One-click install
npx skills add https://github.com/Kaleb-Rupe/aurora --skill evals-kaleb-rupe
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Evals
Source: https://github.com/Kaleb-Rupe/aurora/tree/main/claude/skills/Utilities/Evals
Command: npx skills add https://github.com/Kaleb-Rupe/aurora --skill evals-kaleb-rupe

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires yaml, and includes scripts (resource) and references (resource) components.

What problem does it solve? It provides objective, repeatable measurement of AI agent and prompt quality, replacing subjective judgment with deterministic graders, LLM judges, and statistical metrics like pass@k and pass^k. ## Core Features & Use Cases - Multi-Grader Evaluation: Combines deterministic scorers (string match, binary tests, static analysis) with model-based graders (LLM rubrics, natural language assertions, pairwise comparison) and human review. - Suite & Failure Management: Converts real agent failures into regression test tasks, manages capability and regression suites, and detects saturation to graduate suites automatically. - Model and Prompt Comparison: Runs A/B prompt tests with position swapping and multi-model comparisons with statistical significance reporting. - Use Case: After an agent edits the wrong file in production, log the failure, convert it into a regression task, and run the suite on every future change to prevent recurrence. ## Quick Start Run the regression-core eval suite and update my ALGORITHM ISC row 3 with the results.

Frequently Asked Questions about Evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run an evaluation suite for an AI agent?▼

Run an eval suite with the AlgorithmBridge CLI: bun run Tools/AlgorithmBridge.ts -s <suite-name>. It loads the suite's YAML tasks, executes trials via TrialRunner, aggregates grader scores, and reports pass rate against the suite threshold.

How do I compare two prompt versions with A/B testing?▼

Use the ComparePrompts workflow: create a comparison YAML config listing both prompt variants, enable position_swap to counter ordering bias, and run via the CLI or web UI. Results include win rates, p-values, and confidence intervals for statistical significance.

What is the difference between pass@k and pass^k metrics?▼

pass@k measures the probability of at least one success across k trials, indicating capability. pass^k measures the probability that all k trials succeed, indicating consistency and reliability. Use pass@k for capability evals and pass^k for regression gates.

How do I convert agent failures into regression tests?▼

Log failures with FailureToTask.ts log "description" -c category -s severity, then run convert-all to generate YAML task definitions. Each failure becomes a regression task with inferred graders based on its category, such as tool_calls or llm_rubric.

When should I use code-based graders versus LLM judges?▼

Use code-based graders for deterministic checks like tests, linting, and tool-call verification since they are fast and reproducible. Use LLM judges for nuanced quality rubrics, with reasoning-first prompts, 1-5 scales, and a different judge model than the one being evaluated.

Why does my LLM judge give inconsistent scores?▼

Inconsistency usually comes from missing reasoning requirements, wide scoring scales, or position bias. Require reasoning before scoring, use a 1-5 scale instead of 0-100, enable position swapping for comparisons, and consider multi-judge panels for stability.

Related Skills

Explore More Software Engineering Skills→
cekura-aicekura-ai
7

cekura-flag-call-log-failures

Triage production call logs to flag agent failures and compute outcome distributions.

Official
Advanced
ylxaiylxai
2

evals

Evaluate AI agent transcripts with code-based, model-based, and human graders.

Community
Advanced
agenticgogolagenticgogol
1

cost-optimizer

Runs gated cost-reduction experiments on evaluated agents and writes eval/cost_report.md.

Community
Intermediate
samcdavidsamcdavid
4

my-eval-plan

Design evaluation plans for AI/LLM features with scorers, datasets, and baselines.

Community
Advanced