harness:certify

Run repeated evaluations and compute score stability statistics for evolved LLM agents.

43|5|Updated Mar 31, 2026

Search Tags:#automation #stability #evaluation #statistics #langsmith #llm-agent #evolver

Ready to run

npx skills add https://github.com/raphaelchristi/harness-evolver --skill harness-certify

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: harness:certify
Source: https://github.com/raphaelchristi/harness-evolver/tree/main/skills/certify
Command: npx skills add https://github.com/raphaelchristi/harness-evolver --skill harness-certify

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Ensure that an evolved LLM agent's reported evaluation score is consistent and statistically reliable by running repeated evaluations and computing summary statistics.

Core Features & Use Cases

Repeated Evaluation Runs: Automatically run the project's evaluation script multiple times against the merged code to observe score variance.
Statistical Certification: Aggregate combined_scores to compute mean, standard deviation, range, and a categorical verdict (STABLE, MARGINAL, UNSTABLE).
Actionable Guidance: Provide next steps based on stability: deploy when stable, add rubrics or rerun heavy evolutions when unstable.
Use Case: After an evolution run merges an improved agent, run certification to confirm the improvement holds across multiple independent evaluations before tagging and deploying.

Quick Start

Run the certify flow to evaluate the current merged agent with three independent evaluation runs and produce a mean ± std certification report.

Frequently Asked Questions about harness:certify

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I verify LLM agent evaluation score stability after evolution?▼

To verify LLM agent evaluation score stability after evolution, run repeated evaluations across the same dataset and codebase to compute summary statistics like mean and standard deviation. This confirms whether the reported improvement holds consistently before deployment.

What statistics are used to certify LLM agent reliability?▼

Certifying LLM agent reliability uses mean, standard deviation, range, and a categorical verdict of STABLE, MARGINAL, or UNSTABLE. These summary statistics are aggregated from combined_scores across multiple independent evaluation runs to determine score variance.

Do I need an evolver.json file to run agent stability evaluations?▼

Yes, you need the project's .evolver.json file to run agent stability evaluations. The certification process also requires the Evolver tools run_eval.py and read_results.py, along with a Python runtime to execute those scripts.

When should I run certification for an evolved LLM agent?▼

You should run certification for an evolved LLM agent after an evolution run merges an improved agent. Running multiple evaluations confirms that the improvement holds across independent runs before you tag and deploy the merged agent version.

What should I do if my agent evaluation score is unstable?▼

If your agent evaluation score is unstable, you should add rubrics or rerun heavy evolutions. The certification process provides actionable guidance based on the stability verdict, advising deployment only when scores are stable across repeated runs.