test-plugin-harnessing

Measure and optimize one phase of an oddyssey run under a coding-agent CLI.

9|2|Updated Aug 16, 2026
One-click install
npx skills add https://github.com/using-system/oddyssey --skill test-plugin-harnessing-using-system
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: test-plugin-harnessing
Source: https://github.com/using-system/oddyssey/tree/main/.claude/skills/test-plugin-harnessing
Command: npx skills add https://github.com/using-system/oddyssey --skill test-plugin-harnessing-using-system

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Changes to an agent harness (prompts, skills, commands) are often asserted to be faster or cheaper without proof. This Skill measures one named phase of an oddyssey run — preflight, drive, observation, or the whole run — under opencode, claude, or copilot, against a baseline measured on main just before the work, so a harnessing change is proven rather than claimed. ## Core Features & Use Cases - Phase-level measurement: measure_phase.py launches a headless CLI run, watches for the phase's end marker (k6 drive, dispatch pattern, or a custom regex), stops the run, and writes a record with run id, timing, and command form. - Controlled A/B sampling: run_samples.py runs alternating base/after sample chains (ABBA), syncs deployed scopes into a fake HOME, resets the local stack's data between samples, and journals each sample's outcome. - Run analysis: analyze_run.py reads the run's own logs, transcripts, or events to report commands, turns, generation time, tokens, cost, silent gaps, and the behaviors a harnessing change should remove (authored scripts, stack resets, redundant questions, --help calls). - Use Case: Before opening a PR that changes anything on the path of /odd-observe, /odd-verify, or /odd-status, run two samples per side against a fresh main baseline and quote turns, tokens, cost, and wall clock per phase. ## Quick Start Ask the agent to measure the preflight phase of an oddyssey run under copilot against main using this skill's measure_phase.py with a drive-mode mission on the local stack.

Frequently Asked Questions about test-plugin-harnessing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I measure one phase of an agent CLI run?

Run measure_phase.py with --cli, --model, --tag, --phase, and a mission file. It launches the CLI headless, watches for the phase's end marker (k6 drive or a custom --end-pattern regex), stops the run, and writes a record with timing and run id.

How do I prove a harness change is faster before a PR?

Measure a fresh baseline on main with the same mission, machine, CLI, and model, then run at least two samples per side in alternating ABBA order using run_samples.py. Compare turns, tokens, cost, and wall clock per phase, and quote behavior counts over durations.

Which CLIs and models does the measurement support?

The scripts support opencode, claude, and copilot, each handed its own form of a canonical vendor/name model id. Prefer a model with a small median turn, such as openai/gpt-5.6-luna on copilot, so the measurement reflects the harness rather than provider latency.

Why is my wall-clock comparison between two runs invalid?

A silent gap far above the run's median turn is provider latency, not package work, and analyze_run.py flags it. In that case compare generation time and command counts instead, or re-measure later.

What are the limitations of two-sample A/B measurement?

Two samples per side can only detect changes of the order of the run itself, since main alone spreads about 25 percent on turns and more on tokens and cost. Differences inside main's own spread are neither gains nor losses and should not be quoted as either.