Evals

Name: Evals
Availability: InStock
Author: starlink-awaken

Combine deterministic graders, model-based rubrics, and human review into repeatable evaluation pipelines.

1|Updated Jun 10, 2026

Search Tags:#workflow #transcript #ai #evaluation #comparison #scoring #grading

Ready to run

npx skills add https://github.com/starlink-awaken/pai-universal --skill evals-starlink-awaken

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: Evals
Source: https://github.com/starlink-awaken/pai-universal/tree/main/templates/packs/Evals/src
Command: npx skills add https://github.com/starlink-awaken/pai-universal --skill evals-starlink-awaken

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires @ai-sdk/anthropic, @langwatch/scenario, yaml, ai.

What problem does it solve?

Evals provides a structured framework to evaluate AI agents across complex workflows by uniting deterministic code-based checks, model-based rubrics, and human review into a single, repeatable process.

Core Features & Use Cases

Deterministic codified graders for fast, repeatable checks
Model-based rubrics for nuanced evaluation of outputs
Workflow orchestration for RunEval, CompareModels, ComparePrompts, CreateJudge, CreateUseCase, RunScenario, CreateScenario, ViewResults
Scenario-driven multi-turn evaluation with transcripts and tool-calls
Saturation and regression monitoring to guide suite maturation

Use cases: Benchmark agent behavior across coding, conversational, research, and GUI domains; compare models and prompts; build custom judges and evaluation pipelines.

Quick Start

Run a single evaluation to bootstrap a basic use case with prompts, test cases, and rubrics.

Frequently Asked Questions about Evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate AI agent workflows with deterministic graders and model-based rubrics?▼

Evaluating AI agent workflows involves combining fast deterministic graders, nuanced model-based rubrics, and human review into a single repeatable pipeline. This validates agent behavior across coding, conversational, research, and GUI domains.

What is the best way to compare prompts and models for AI regression testing?▼

Comparing prompts and models for regression testing requires orchestrated workflows like ComparePrompts and CompareModels. These generate transcripts, tool-call tracking, and saturation analysis to establish reliable quality gates.

Does the AI evaluation pipeline support scenario-driven multi-turn conversations?▼

Yes, scenario-driven multi-turn evaluation is supported through CreateScenario and RunScenario workflows. They generate detailed transcripts and track tool-calls to benchmark agent behavior across complex conversational flows.

Can I use custom judges for nuanced evaluation of AI outputs?▼

Yes, you can build custom judges using the CreateJudge workflow. This allows you to define model-based rubrics for nuanced evaluation of AI outputs within your specific coding, research, or GUI domain.

How do I bootstrap a basic AI evaluation use case with test cases and rubrics?▼

Bootstrapping a basic use case requires running a single evaluation via RunEval. This quickly sets up your prompts, test cases, and rubrics to validate your AI agent workflow.