eval-harness

Run eval harness sweeps against a live Obsidian vault and bless clean baselines.

515|64|Updated Oct 23, 2024

Search Tags:#automation #testing #obsidian #gemini #baseline #harness #eval

Ready to run

npx skills add https://github.com/allenhutchison/obsidian-gemini --skill eval-harness-allenhutchison

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/allenhutchison/obsidian-gemini/tree/main/.agents/skills/eval-harness
Command: npx skills add https://github.com/allenhutchison/obsidian-gemini --skill eval-harness-allenhutchison

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Executes the eval harness against a real Obsidian instance, monitors for known reliability issues (concurrent runs, CLI-bridge hangs, stale child processes), and blesses a result as baseline only when the run is clean.

Core Features & Use Cases

Orchestrates preflight setup and environment checks to ensure each harness run starts from a clean slate.
Supports full model sweeps, single-model runs, and baseline blessing across provider/model configurations.
Provides guardrails, logging, and cleanup to enable repeatable, trustworthy baselines.

Quick Start

Prepare a test Obsidian vault, ensure the CLI is reachable, and run npm run eval to start a single-model evaluation.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run an eval harness against a live Obsidian vault?▼

To run an eval harness against a live Obsidian vault, prepare a test vault, ensure the Obsidian CLI is on PATH, and execute npm run eval to start a single-model evaluation.

What is baseline blessing in automated testing?▼

Baseline blessing in automated testing validates reliability and baseline correctness, marking a result as a verified baseline only when the eval harness run is completely clean.

Do I need a Gemini API key to run eval sweeps?▼

Yes, you need a configured Gemini API key to run eval sweeps, or an EVAL_JUDGE_API_KEY specifically for judge-matcher tasks, along with a Node/npm environment.

Why does my eval harness hang during concurrent runs?▼

Eval harness hangs during concurrent runs are known reliability issues caused by CLI-bridge problems or stale child processes, which preflight setup and environment checks help prevent.

Can I measure the impact of code changes on solve rate across configurations?▼

Yes, you can measure the impact of changes on solve rate across configurations by running full model sweeps through the eval harness against your configured provider and model setups.

What's the best way to ensure repeatable baselines in an eval harness?▼

The best way to ensure repeatable baselines in an eval harness is to use orchestration guardrails, logging, and cleanup to enable repeatable, trustworthy baselines from a clean slate.