skills.rest
Atrium-Hermes
Atrium-HermesCOMMUNITY
Explore all skills by this author →
𝕏@atriumhermesGitHub

Skill Evals

Compare current skill outputs against stored assertions in evals.json to detect regressions.

Updated Jun 2, 2026
Search Tags:#automation#qa#evaluation#memory#regression#diff#issues
One-click install
npx skills add https://github.com/Atrium-Hermes/atrium-lighthouse --skill skill-evals-atrium-hermes
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Skill Evals
Source: https://github.com/Atrium-Hermes/atrium-lighthouse/tree/main/skills/skill-evals
Command: npx skills add https://github.com/Atrium-Hermes/atrium-lighthouse --skill skill-evals-atrium-hermes

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates the evaluation of per-skill outputs by comparing current results against stored assertions in evals.json, flagging regressions, and queueing concrete fixes via issue tracking.

Core Features & Use Cases

  • Load input manifests from evals.json, memory/cron-state.json, and memory/issues/INDEX.md to determine quality signals, regression status, and coverage.
  • Generate an Action Queue with concrete next steps (patches, investigations, re-runs, new evals entries) to be executed by operators.
  • Produce artifacts like articles and memory logs to communicate results and track progress over time.

Quick Start

Run Skill Evals to compare current outputs against evals.json, compute diffs since the prior run, and automatically queue fixes and issue creation.

Frequently Asked Questions about Skill Evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate validation of skill outputs and detect regressions?▼

Automate validation by comparing current skill outputs against stored assertions in evals.json to detect regressions. The system computes diffs since the prior run, flags quality changes, and automatically queues concrete fixes for operators to execute.

What is regression testing for skill outputs using evals.json?▼

Regression testing for skill outputs is the process of comparing current results against stored assertions in evals.json. It identifies quality changes by computing diffs since the prior run and preparing an action queue with concrete fixes.

Do I need an issue tracker to queue fixes for detected eval regressions?▼

You do not need an external issue tracker to queue fixes, as the system prepares an action queue with concrete next steps internally. However, it provides integration points for issue trackers to communicate results and track progress.

How do I compute coverage and quality changes from memory logs and articles?▼

Compute coverage and quality changes by analyzing memory, articles, and issues loaded from input manifests. The system evaluates these sources against current outputs to detect regressions and generate a detailed report with clear next steps.

What's the best way to automate QA evaluation for per-skill outputs?▼

The best way to automate QA evaluation is by running a process that compares current results against stored assertions in evals.json. This flags regressions, computes diffs, and automatically queues fixes via issue tracking.

Related Skills

Explore More Data & Analytics Skills→
Atrium-HermesAtrium-Hermes

Heartbeat

Analyze cron-state.json, memory logs, and run statuses to surface skill failures.

Community
Intermediate
hohai99hohai99

self-healing-debugger

Fix implementation bugs flagged by failing tests and re-run tests.

Community
Advanced
LamseyahEliasLamseyahElias

evaluating-llms-harness

Benchmark LLMs across 60+ tasks with HuggingFace, vLLM, and API backends.

Community
Advanced
apassuelloapassuello

llm-evaluation

Evaluate LLM outputs with automated metrics, human judgments, and benchmarking.

Community
Advanced