om-evolve-harness

Converts agent failures into versioned harness regression cases with semantic assertions and owner-scoped fixes.

1.7k|382|Updated Sep 10, 2025
One-click install
npx skills add https://github.com/open-mercato/open-mercato --skill om-evolve-harness
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: om-evolve-harness
Source: https://github.com/open-mercato/open-mercato/tree/main/packages/create-app/agentic/shared/ai/skills/om-evolve-harness
Command: npx skills add https://github.com/open-mercato/open-mercato --skill om-evolve-harness

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

When an AI agent harness routes a request incorrectly or loads the wrong context, teams often patch prose without a reproducible regression, so the same failure returns. This Skill turns each real failure into a versioned, schema-validated case with semantic assertions, a single knowledge owner, and before/after evaluation evidence.

Core Features & Use Cases

  • Failure-to-case workflow: Capture a prompt, transcript, or PR as untrusted evidence, classify and deduplicate it, and reproduce it in a fresh pinned standalone scaffold.
  • Semantic assertions: Reduce failures to routing, decision, and artifact assertions instead of whole-output or whole-file goldens, with calibrated context budgets.
  • Single-owner knowledge changes: Select exactly one smallest owner (root invariant, router row, guide, skill reference, facts extractor, hook) and update only it, replacing duplicates with references.
  • Gated release validation: Run catalog validation, writable-target generate/typecheck/lint/build gates, code review, an isolated judge lane, and the full harness release suite with sanitized reports.
  • Use Case: An agent picks the wrong module guide for a request. Use this Skill to add an OMH-NNN case that fails before the fix, correct the router row, and prove the fix with the release suite.

Quick Start

Ask the agent to add a harness case for the routing failure you just observed and evolve the harness with before and after evaluation evidence.

Frequently Asked Questions about om-evolve-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add a new regression case when an AI agent routes a request incorrectly?

Capture the prompt or transcript as untrusted evidence, reproduce the failure in a fresh pinned scaffold, and add a schema-valid OMH-NNN case with required router entries, context, decisions, and validators. The case must fail before the owner fix and pass after it.

How do I fix wrong context loading in an agent harness?

Reduce the failure to semantic routing or context assertions, then select exactly one smallest owner such as a router row, guide, or facts extractor. Update only that owner, replace duplicates with references, and rerun the case plus related tags and budget gates.

What is the difference between knowledge-contract and asset-sync changes?

The class is machine-derived from the diff, not declared intent. Knowledge-contract changes touch authoritative instructions, routing JSON, or evaluators and require nine mandatory steps; asset-sync covers regenerated copies with unchanged source hashes and needs only synchronization validation.

Can I use whole model outputs as golden files for harness tests?

No. Cases must use semantic routing, decision, and artifact assertions rather than whole model output or whole-file goldens. Budgets are calibrated from the case's own measured context footprint, never inherited from a neighboring case.

What validation gates must pass before releasing a harness change?

Run yarn harness:validate for the case, family, and full catalog, then generate, typecheck, lint, and build in a disposable writable target. Finish with code review, the isolated judge lane, and yarn harness:release requiring a passing sanitized release report.