skills.rest
RHEcosystemAppEng
RHEcosystemAppEngOFFICIAL
Explore all skills by this author →
GitHub

run-evals

Automate execution and grading of AI skill evaluations for CI pipelines.

10|7|Updated Mar 18, 2026
Search Tags:#benchmarking#skill testing#sdlc#ai evaluation#ci
One-click install
npx skills add https://github.com/RHEcosystemAppEng/sdlc-plugins --skill run-evals-rhecosystemappeng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: run-evals
Source: https://github.com/RHEcosystemAppEng/sdlc-plugins/tree/main/plugins/sdlc-workflow/skills/run-evals
Command: npx skills add https://github.com/RHEcosystemAppEng/sdlc-plugins --skill run-evals-rhecosystemappeng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill automates the evaluation of AI skills with deterministic output and CI-compatible results, enabling consistent and reliable testing.

Core Features & Use Cases

  • Deterministic Output: Ensures consistent results for repeated evaluations.
  • CI-Compatible: Generates output suitable for continuous integration pipelines.
  • Use Case: Use this Skill to evaluate the performance of an AI skill across multiple test cases and generate a comprehensive evaluation report.

Quick Start

Run the evals for the 'plan-feature' skill with the provided evals.json file.

Frequently Asked Questions about run-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate AI skill evaluation for CI pipelines?▼

Automating AI skill evaluation requires execution scripts and grading rules to generate deterministic, CI-compatible structured results. This ensures consistent, reliable testing across multiple test cases within standard continuous integration workflows.

What is deterministic output in AI skill testing?▼

Deterministic output in AI skill testing ensures consistent, repeatable evaluation results across multiple runs. It eliminates variance in automated grading, providing reliable benchmark data necessary for continuous integration pipelines.

How do I run evals for an AI skill using an evals.json file?▼

Running evals for an AI skill requires providing an evals.json file containing test cases alongside execution scripts and grading rules. The automation processes these inputs to generate a comprehensive performance evaluation report.

Does automated skill testing require predefined grading rules?▼

Automated skill testing requires predefined grading rules and evaluation scripts to accurately assess AI performance. These rules establish the deterministic criteria needed to generate structured, CI-compatible evaluation results.

Related Skills

Explore More Software Engineering Skills→
AidenSbVevoAidenSbVevo
1

skill-creator

Create, evaluate, and optimize custom AI skill units with automated testing.

Community
Advanced
686f6c61686f6c61
105

ci-cd-pipeline

Generate CI/CD pipeline configurations for GitHub Actions or GitLab CI.

Community
Advanced
guchengzhongguchengzhong
3

skill-creator

Create, modify and optimize reusable terminal skills with benchmarking support.

Community
Advanced
dotnetdotnet
23.3k

verify-tests-fail-without-fix

Automate MAUI UI test verification in CI workflows using PowerShell and Git.

Official
Advanced