skills.rest
allenhutchison
allenhutchisonCOMMUNITY
Explore all skills by this author →
Website𝕏@allen_hutchisonGitHub

eval-harness

Run eval harness sweeps against a live Obsidian vault and bless clean baselines.

515|64|Updated Oct 23, 2024
Search Tags:#automation#testing#obsidian#gemini#baseline#harness#eval
One-click install
npx skills add https://github.com/allenhutchison/obsidian-gemini --skill eval-harness-allenhutchison
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/allenhutchison/obsidian-gemini/tree/main/.agents/skills/eval-harness
Command: npx skills add https://github.com/allenhutchison/obsidian-gemini --skill eval-harness-allenhutchison

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Executes the eval harness against a real Obsidian instance, monitors for known reliability issues (concurrent runs, CLI-bridge hangs, stale child processes), and blesses a result as baseline only when the run is clean.

Core Features & Use Cases

  • Orchestrates preflight setup and environment checks to ensure each harness run starts from a clean slate.
  • Supports full model sweeps, single-model runs, and baseline blessing across provider/model configurations.
  • Provides guardrails, logging, and cleanup to enable repeatable, trustworthy baselines.

Quick Start

Prepare a test Obsidian vault, ensure the CLI is reachable, and run npm run eval to start a single-model evaluation.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run an eval harness against a live Obsidian vault?▼

To run an eval harness against a live Obsidian vault, prepare a test vault, ensure the Obsidian CLI is on PATH, and execute npm run eval to start a single-model evaluation.

What is baseline blessing in automated testing?▼

Baseline blessing in automated testing validates reliability and baseline correctness, marking a result as a verified baseline only when the eval harness run is completely clean.

Do I need a Gemini API key to run eval sweeps?▼

Yes, you need a configured Gemini API key to run eval sweeps, or an EVAL_JUDGE_API_KEY specifically for judge-matcher tasks, along with a Node/npm environment.

Why does my eval harness hang during concurrent runs?▼

Eval harness hangs during concurrent runs are known reliability issues caused by CLI-bridge problems or stale child processes, which preflight setup and environment checks help prevent.

Can I measure the impact of code changes on solve rate across configurations?▼

Yes, you can measure the impact of changes on solve rate across configurations by running full model sweeps through the eval harness against your configured provider and model setups.

What's the best way to ensure repeatable baselines in an eval harness?▼

The best way to ensure repeatable baselines in an eval harness is to use orchestration guardrails, logging, and cleanup to enable repeatable, trustworthy baselines from a clean slate.

Related Skills

Explore More Software Engineering Skills→
DaFumDaFum
1

one-command-doctor

Run Node project healthchecks covering build, lint, and runtime issues.

Community
Intermediate
khetansarveshkhetansarvesh
1

eval-harness

Define and run structured evals for Claude Code sessions with pass@k metrics.

Community
Advanced
Agent-Pattern-LabsAgent-Pattern-Labs
4

obsidian

Automate Obsidian vault management and note organization via obsidian-cli.

Official
Basic
MOODMNKY-LLCMOODMNKY-LLC

obsidian-cli

Automate Obsidian vault operations via the official obsidian CLI binary.

Community
Intermediate