skills.rest
starlink-awaken
starlink-awakenCOMMUNITY
Explore all skills by this author →
GitHub

Evals

Combine deterministic graders, model-based rubrics, and human review into repeatable evaluation pipelines.

1|Updated Jun 10, 2026
Search Tags:#workflow#transcript#ai#evaluation#comparison#scoring#grading
One-click install
npx skills add https://github.com/starlink-awaken/pai-universal --skill evals-starlink-awaken
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Evals
Source: https://github.com/starlink-awaken/pai-universal/tree/main/templates/packs/Evals/src
Command: npx skills add https://github.com/starlink-awaken/pai-universal --skill evals-starlink-awaken

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires @ai-sdk/anthropic, @langwatch/scenario, yaml, ai.

What problem does it solve?

Evals provides a structured framework to evaluate AI agents across complex workflows by uniting deterministic code-based checks, model-based rubrics, and human review into a single, repeatable process.

Core Features & Use Cases

  • Deterministic codified graders for fast, repeatable checks
  • Model-based rubrics for nuanced evaluation of outputs
  • Workflow orchestration for RunEval, CompareModels, ComparePrompts, CreateJudge, CreateUseCase, RunScenario, CreateScenario, ViewResults
  • Scenario-driven multi-turn evaluation with transcripts and tool-calls
  • Saturation and regression monitoring to guide suite maturation

Use cases: Benchmark agent behavior across coding, conversational, research, and GUI domains; compare models and prompts; build custom judges and evaluation pipelines.

Quick Start

Run a single evaluation to bootstrap a basic use case with prompts, test cases, and rubrics.

Frequently Asked Questions about Evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate AI agent workflows with deterministic graders and model-based rubrics?▼

Evaluating AI agent workflows involves combining fast deterministic graders, nuanced model-based rubrics, and human review into a single repeatable pipeline. This validates agent behavior across coding, conversational, research, and GUI domains.

What is the best way to compare prompts and models for AI regression testing?▼

Comparing prompts and models for regression testing requires orchestrated workflows like ComparePrompts and CompareModels. These generate transcripts, tool-call tracking, and saturation analysis to establish reliable quality gates.

Does the AI evaluation pipeline support scenario-driven multi-turn conversations?▼

Yes, scenario-driven multi-turn evaluation is supported through CreateScenario and RunScenario workflows. They generate detailed transcripts and track tool-calls to benchmark agent behavior across complex conversational flows.

Can I use custom judges for nuanced evaluation of AI outputs?▼

Yes, you can build custom judges using the CreateJudge workflow. This allows you to define model-based rubrics for nuanced evaluation of AI outputs within your specific coding, research, or GUI domain.

How do I bootstrap a basic AI evaluation use case with test cases and rubrics?▼

Bootstrapping a basic use case requires running a single evaluation via RunEval. This quickly sets up your prompts, test cases, and rubrics to validate your AI agent workflow.

Related Skills

Explore More Software Engineering Skills→
selcukyucelselcukyucel
1

eval-suite

Generate structured evaluation datasets with binary rubric scoring for AI outputs.

Community
Advanced
summarybotngsummarybotng
2

quality-metrics

Define and measure DORA metrics, bug escape rates, and test effectiveness.

Official
Advanced
ylxaiylxai
2

evals

Evaluate AI agent transcripts with code-based, model-based, and human graders.

Community
Advanced
diegosouzapwdiegosouzapw
108

deploy-to-vercel

Package upstream Vercel deployment workflows into auditable skill units.

Community
Advanced