skills.rest
raphaelchristi
raphaelchristiCOMMUNITY
Explore all skills by this author →
GitHub

harness:health

Detect and auto-correct quality issues in LangSmith evaluation datasets.

43|5|Updated Mar 31, 2026
Search Tags:#evaluation#test-generation#data-quality#langsmith#auto-correct#dataset-health
One-click install
npx skills add https://github.com/raphaelchristi/harness-evolver --skill harness-health
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: harness:health
Source: https://github.com/raphaelchristi/harness-evolver/tree/main/skills/health
Command: npx skills add https://github.com/raphaelchristi/harness-evolver --skill harness-health

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Identifies and repairs dataset quality problems that undermine reliable evaluation of LLM agents by checking size, difficulty distribution, dead examples, coverage, and splits so that evolution runs and evaluations produce meaningful results.

Core Features & Use Cases

  • Health Diagnostic: Runs a dataset health check that reports a health score, example counts, and a list of issues with severities.
  • Automated Corrections: Applies corrections such as creating train/held_out splits, retiring dead examples, and invoking test-generation to rebalance or harden the dataset.
  • Integration & Reporting: Uses the langsmith Client and dataset_health.py to update examples and prints a final health summary and warnings for any unresolved critical issues.
  • Use Case: Run this before /harness:evolve to ensure the evaluation dataset is balanced, challenging, and free of dead or mis-split examples.

Quick Start

Run the harness:health check to analyze dataset quality, auto-apply suggested fixes, and print the final health report.

Frequently Asked Questions about harness:health

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check LangSmith evaluation dataset health before running evolution?▼

To check LangSmith evaluation dataset health, run a diagnostic that reports a health score, example counts, and issues with severities. This identifies size, difficulty distribution, dead examples, coverage, and train/held_out split problems before evolution runs.

What is dataset health evaluation for LLM agents?▼

Dataset health evaluation for LLM agents is the process of detecting and auto-correcting dataset quality issues like dead examples, poor difficulty distribution, and incorrect splits. It ensures evolution runs and evaluations produce meaningful, reliable results.

How do I auto-correct dead examples and bad splits in my evaluation dataset?▼

Auto-correct dead examples and bad splits by running the dataset health check, which uses the LangSmith Client to update example splits, retire dead examples, and spawn test-generation subagents to rebalance the dataset automatically.

Do I need LangSmith access to auto-fix dataset quality issues?▼

Yes, you need LangSmith access and a configured .evolver.json file to auto-fix dataset quality issues. These are required to execute dataset_health.py, update example splits, and spawn test-generation subagents for retiring or generating examples.

Why does my LLM evaluation dataset produce unreliable results?▼

Your LLM evaluation dataset may produce unreliable results due to unbalanced difficulty distribution, dead examples, or mis-split train/held_out data. Running a dataset health check identifies these severity-ranked issues so you can apply automated corrections.

Related Skills

Explore More Data & Analytics Skills→
joonlabjoonlab
8

langsmith-trace

Query and export LangSmith execution traces to analyze LLM application runs.

Community
Intermediate
andikarachmanandikarachman
14

split-strategy

Select train/validation/test split strategies based on data characteristics.

Community
Intermediate
raphaelchristiraphaelchristi
43

harness:setup

Configure Harness Evolver, LangSmith, and baseline evaluations for Python projects.

Community
Advanced
junchenghuojunchenghuo
3

ai-prompt-eval-kit

Construct evaluation datasets and generate comparison reports for offline AI prompt testing.

Community
Intermediate