skills.rest
peachest
peachestCOMMUNITY
Explore all skills by this author →
GitHub

replay-compare

Compares search backends by replaying historical queries from pi session logs.

Updated Jul 20, 2026
Search Tags:#benchmarking#latency#session logs#stratified sampling#backend comparison#search evaluation#query replay
One-click install
npx skills add https://github.com/peachest/skills --skill replay-compare-peachest
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: replay-compare
Source: https://github.com/peachest/skills/tree/main/in-progress/replay-compare
Command: npx skills add https://github.com/peachest/skills --skill replay-compare-peachest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tsx, and includes scripts (resource) components.

What problem does it solve? Choosing between two interchangeable search or tool backends is hard because vendor benchmarks are generic and self-serving. This Skill replays your own real historical queries against each backend so the comparison is grounded in your actual workload, producing a cited quality and latency report. ## Core Features & Use Cases - Historical Query Extraction: Walks all pi session JSONL files and pulls query strings out of tool-call commands with configurable regexes and a built-in junk filter. - Stratified Sampling: Buckets the corpus by theme, samples proportionally with a fixed seed, and dedupes near-duplicates so the replay set is reproducible. - Resumable Replay Runner: A template-based Node runner writes one JSON result file per query and leg, records latency and errors, and skips existing files so crashes are free. - Use Case: You are debating whether to switch from your current search CLI to a new engine. Use this Skill to extract 30 representative past queries, run both backends on them, and get a side-by-side report with success rates, p50 latency, and cited raw results. ## Quick Start Ask the agent to compare two search backends by replaying my historical queries from pi sessions and producing a quality and latency report.

Frequently Asked Questions about replay-compare

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compare two search backends with real user queries?▼

Extract historical queries from pi session JSONL files, take a stratified sample, then run each backend on the same queries with a replay runner. The Skill records success rate, result counts, and average plus p50 latency per backend, and produces a cited comparison report.

How to extract tool call queries from pi session logs?▼

Run the bundled extract-queries.py script with a --match substring for the CLI name and repeatable --regex patterns whose first capture group is the query. It walks all session JSONLs, filters junk test invocations, and outputs a frequency-sorted corpus.

Why does the Node replay runner fail with fetch failed on corporate networks?▼

Node fetch ignores proxy environment variables by default, so every fetch-based leg fails in under a second while curl works. Set NODE_USE_ENV_PROXY=1 when running the runner so requests go through the corporate proxy.

Can I customize the sampling buckets for my query corpus?▼

Yes, pass --buckets buckets.json to sample-queries.py with entries of [name, regex] pairs. The default buckets target a coding-agent research workload, but any thematic regex set works, and sampling stays proportional to bucket size with a fixed seed.

What are the limitations of replay-based backend comparison?▼

Replay comparison measures latency, success rate, and eyeball-level result quality on a small sample, not full relevance judgments. Quality assessment stays human-in-the-loop on 4-6 representative queries, and results only reflect your historical workload distribution.

Related Skills

Explore More Software Engineering Skills→
zjykzkzjykzk

pi-history-ingest

Ingest Pi session history into an Obsidian wiki while redacting sensitive data.

Community
Advanced
zeropoint-foundationzeropoint-foundation
1

llm-council

Query multiple LLM backends in parallel and aggregate outputs for cross-model analysis.

Community
Advanced
prajapatimehulprajapatimehul
22

validating-aws-pricing

Cross-check cloud cost findings against live AWS Pricing API data.

Community
Advanced
woodfishhhhwoodfishhhh
32

dataset

Discover and download public datasets from Kaggle, UCI, HuggingFace, and Tianchi.

Community
Intermediate