skills.rest
Sir-chawakorn
Sir-chawakornCOMMUNITY
Explore all skills by this author →
GitHub

llm-eval-harness

Builds an LLM evaluation harness using golden datasets and code-based scorers.

9|3|Updated Jun 13, 2026
Search Tags:#testing#llm#evaluation#regression#quality-assurance#ai-agent
One-click install
npx skills add https://github.com/Sir-chawakorn/sanook-cli --skill llm-eval-harness
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-eval-harness
Source: https://github.com/Sir-chawakorn/sanook-cli/tree/main/skills/llm-eval-harness
Command: npx skills add https://github.com/Sir-chawakorn/sanook-cli --skill llm-eval-harness

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill solves the problem of non-deterministic output quality in AI agents and LLM applications by replacing subjective vibes-based testing with a structured, reproducible regression testing framework.

Core Features & Use Cases

  • Multi-Layered Scoring: Combines deterministic code-based scorers (regex, JSON schema) with LLM-as-judge evaluation for semantic quality.
  • Regression Gates: Prevents performance degradation by comparing new model/prompt outputs against a committed baseline dataset.
  • Use Case: When updating a RAG system's retrieval configuration, use this harness to ensure that the new configuration maintains or improves faithfulness and correctness scores across a golden dataset of production-derived test cases.

Quick Start

Use the llm-eval-harness skill to initialize a new evaluation suite by defining your golden dataset and configuring the code-based scorers for your specific agent output.

Frequently Asked Questions about llm-eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a regression gate for LLM outputs?▼

To build a regression gate for LLM outputs, define a golden dataset and rubric to compare candidate outputs against established baselines, preventing performance degradation when updating prompts or models.

What is the best way to test RAG configuration updates?▼

Testing RAG configuration updates is best done by running an evaluation harness across a golden dataset of production-derived test cases to ensure new retrievals maintain or improve faithfulness and correctness scores.

Do I need a specific data format to start LLM regression testing?▼

Yes, you need a structured JSONL or CSV dataset to initialize the evaluation suite, providing the baseline data required to compare new agent outputs against committed baselines.

Can I use code-based scorers instead of LLM-as-judge for evaluation?▼

Yes, you can configure deterministic code-based scorers like regex and JSON schema validation independently or combine them with LLM-as-judge mechanisms for multi-layered evaluation of agent outputs.

Related Skills

Explore More Software Engineering Skills→
vecearvecear

evaluation-harness

Automate LLM evaluation with golden datasets, rubrics, thresholds, and regression reports.

Community
Advanced
binfen1binfen1

canary

Monitor live apps for console errors, performance regressions, and page failures.

Community
Advanced
viktorbezdekviktorbezdek
10

agent-evaluation

Evaluate LLM agent outputs using multi-dimensional rubrics and bias mitigation.

Community
Advanced
AbsolutelySkilledAbsolutelySkilled
207

llm-app-development

Design and implement LLM applications with guardrails and evaluation pipelines.

Official
Advanced