skills.rest
larsboes
larsboesCOMMUNITY
Explore all skills by this author →
GitHub

Evals

Evaluate agent workflows and prompts with deterministic tests and AI-based judges.

Updated Aug 23, 2026
Search Tags:#evaluation#framework#scoring#transcripts#model-comparison#ai-evals
One-click install
npx skills add https://github.com/larsboes/pai-marketplace --skill evals-larsboes
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Evals
Source: https://github.com/larsboes/pai-marketplace/tree/main/marketplace/plugins/utilities/skills/Evals
Command: npx skills add https://github.com/larsboes/pai-marketplace --skill evals-larsboes

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires yaml.

What problem does it solve?

Evaluations of AI agent workflows and prompts to ensure reliable, auditable behavior across diverse tasks and models.

Core Features & Use Cases

  • Manage evaluation use cases, test cases, prompts, golden outputs, and schemas for scoring.
  • Run deterministic tests alongside AI-based rubrics, aggregate results, and compare models or prompts.
  • Store, query, and visualize results for regression and capability assessments.

Quick Start

Create a new use case, add test cases and prompts, then run the evaluation to see results.

Frequently Asked Questions about Evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate AI agent workflows using deterministic tests and AI-based judges?▼

You can evaluate AI agent workflows by creating use cases, adding test cases, and running deterministic tests alongside AI-based rubrics to aggregate results for reliable, auditable scoring.

What is the best way to compare prompts and models for regression testing?▼

Model and prompt comparison for regression testing is done by applying structured evaluation suites, storing results, and analyzing pass@k and pass^k metrics to ensure consistent agent behavior.

Can I manage test cases and scoring schemas for AI evaluations at scale?▼

Yes, you can manage evaluation use cases at scale by defining test cases, prompts, golden outputs, and schemas for scoring to systematically assess agent capabilities.

Does this evaluation framework require YAML configuration?▼

Yes, the framework requires YAML for setup, utilizing it to define and manage the structured evaluation use cases, prompts, and test configurations.

How do pass@k and pass^k metrics work in model comparison?▼

Pass@k and pass^k metrics provide auditable model comparison results by measuring the probability of generating a correct response across multiple attempts within the evaluation framework.

When should I use an AI-based judge for agent evaluation?▼

AI-based judges should be used alongside deterministic tests when evaluating complex agent workflows where flexible rubric scoring is needed to measure regression and capability accurately.

Related Skills

Explore More Education & Research Skills→
selcukyucelselcukyucel
1

eval-suite

Generate structured evaluation datasets with binary rubric scoring for AI outputs.

Community
Advanced
gugug168gugug168

eval-harness

Define and run capability and regression evaluations for Claude Code sessions.

Community
Advanced
vobasevobase
59

skill-creator

Create and iteratively improve Claude-compatible skills with automated evaluations.

Official
Advanced
jordanhubbardjordanhubbard
9

evaluating-llms-harness

Benchmarks LLM academic-task performance using lm-eval harnesses across multiple platforms and metrics.

Community
Advanced