skills.rest
bgauryy
bgauryyCOMMUNITY
Explore all skills by this author →
GitHub

octocode-benchmark

Runs pairwise CLI research benchmarks comparing Octocode against gh-based baselines with blind judging.

921|75|Updated Jun 5, 2025
Search Tags:#benchmark#code research#gitHub#cli comparison#paired statistics#blind judging#character measurement
One-click install
npx skills add https://github.com/bgauryy/octocode-mcp --skill octocode-benchmark
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: octocode-benchmark
Source: https://github.com/bgauryy/octocode-mcp/tree/main/packages/octocode-benchmark/skills/octocode-benchmark
Command: npx skills add https://github.com/bgauryy/octocode-mcp --skill octocode-benchmark

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires headroom-ai, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Comparing code-research tools fairly is hard: self-reported token counts are unreliable, judges are biased, and one heavy question can dominate pooled totals. This Skill provides a rigorous, by-hand benchmark methodology that measures Octocode against baseline CLIs (gh+RTK, gh+Headroom, plain gh) using instrumented character logs, isolated runner agents, and blind judges.

Core Features & Use Cases

  • Pairwise matchups with isolated runners: A fresh agent per (question, arm, pass) answers markdown research questions, with every command routed through instrumented wrappers that log model-in and model-out characters to JSONL.
  • Blind, reasoning-first judging: One judge per question grades answers relabeled X/Y in randomized order, establishing ground truth before scoring correctness (0-10), research depth (1-5), and workflow (1-5).
  • Honest statistics: Aggregates per-question paired ratios using geometric mean, median, sign tests, and bootstrap CIs across at least 3 passes, with fairness gates against whole-tree/whole-file dumps.
  • Use Case: Run a 30-question campaign comparing Octocode vs gh+RTK, then produce a rollup report showing correctness parity and per-question character ratios with confidence intervals.

Quick Start

Run the octocode-benchmark skill to execute the preflight check and start a pairwise matchup campaign over the GitHub question set.

Frequently Asked Questions about octocode-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run the Octocode benchmark against a baseline CLI?▼

Run the preflight script check-prereqs.sh to verify all arms, then spawn one isolated runner agent per question and arm, build a blind packet with build_blind_packet.py, judge each question, and aggregate with validate_campaign.py and per_question_summary.py.

How does the benchmark measure tool efficiency?▼

It counts total characters through the model: model-in (tool output pulled into context) plus model-out (commands, args, and final answer), captured by instrumented wrapper scripts into JSONL logs. Counts are recomputed from the logs, never self-reported.

Which baseline tools can Octocode be compared against?▼

Three pairwise matchups are supported: Octocode versus plain gh (read-only GitHub CLI), gh plus RTK, and gh plus Headroom compression. Each baseline runs as a separate matchup with its own wrapper, never a single three-way run.

Why does the benchmark use geometric mean instead of summed characters?▼

Summing characters weights each question by its absolute size, letting one heavy question dominate the verdict. The geometric mean of per-question ratios, plus median and leaner win-rate with a sign test, gives an honest typical factor.

What makes a benchmark run invalid?▼

A failed preflight check, mixing arms in one runner agent, a judge starting before both answers exist, or a fairness violation such as dumping a whole file tree when a targeted read would answer. Any of these requires fixing the issue and rerunning.

Related Skills

Explore More Software Engineering Skills→
kamp-uskamp-us
25

doctor

Verify repository readiness for kampus-pipeline by checking labels, authentication, and CI workflows.

Official
Intermediate
mims-harvardmims-harvard
1.7k

tooluniverse-statistical-modeling

Fit regression, survival, and ANOVA models on biomedical data with Python and R scripts.

Official
Advanced
yanochka11yanochka11

validate-evaluator

Calibrate LLM judges against human labels using TPR/TNR analysis.

Community
Advanced
MesferAliMesferAli

evaluating-llms-harness

Evaluate LLMs across 60+ academic benchmarks using lm-evaluation-harness.

Community
Advanced