Verified:GitHub

exp-pilot-eval

Evaluate pilot experiment results and write pass, fail, or inconclusive verdicts.

1.6k|208|Updated Apr 9, 2026

Search Tags:#report generation #success criteria #experiment results #status transition #research wiki #pilot evaluation

Ready to run

npx skills add https://github.com/skyllwt/AutoSci --skill exp-pilot-eval-skyllwt

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: exp-pilot-eval
Source: https://github.com/skyllwt/AutoSci/tree/main/.claude/skills/exp-pilot-eval
Command: npx skills add https://github.com/skyllwt/AutoSci --skill exp-pilot-eval-skyllwt

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Pilot experiments generate noisy, incomplete signals, and teams need a consistent way to interpret those results and update an idea’s status without manual back-and-forth.

Core Features & Use Cases

Verdict evaluation: Reads pilot results and pilot logs, applies success-criterion logic, and produces a pass/fail/inconclusive decision.
Wiki state updates: Updates the linked idea page fields (pilot_result, failure_reason when failed, and status transition to failed) while enforcing lifecycle rules.
Actionable reporting: Generates a persistent PILOT_VERDICT_REPORT and appends an execution log for auditing and next-step planning.

Quick Start

Run pilot evaluation for an idea by providing its slug, for example: call exp-pilot-eval with idea slug 'my-idea' in auto mode.

Frequently Asked Questions about exp-pilot-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate pilot experiment results to decide if an idea passes or fails?▼

Pilot experiment results are evaluated by reading seed-based JSON outputs and pilot logs, applying threshold logic against predefined success criteria, and producing a pass, fail, or inconclusive verdict for the idea.

How do I update my research wiki idea status after a pilot test?▼

Updating your research wiki idea status happens automatically during evaluation by writing updated fields like pilot_result and failure_reason, and transitioning the status to failed when success criteria are not met.

What is the best way to generate an actionable report from noisy pilot experiment data?▼

Generating an actionable report from pilot data involves computing metric summaries, applying baseline comparisons, and persisting a PILOT_VERDICT_REPORT with an appended execution log for auditing and next-step planning.

What format do pilot results need to be in for automated evaluation?▼

Automated pilot evaluation requires results stored as seed-based JSON outputs, accompanied by a pilot log and a pilot spec that defines success criteria and baseline comparisons.

Can I automatically transition an idea to failed status if the pilot results do not meet baseline comparisons?▼

Yes, when pilot results do not meet baseline comparisons and thresholds, the evaluation enforces lifecycle rules to transition the idea status to failed and records the failure reason in the wiki metadata.

What happens if pilot results are inconclusive for the next research step?▼

When pilot results are inconclusive, the evaluation mechanism computes the metric summaries and produces an inconclusive verdict report, allowing teams to plan the next research step without forcing a pass or fail transition.