exp-eval

Evaluates completed experiments and updates research wiki claims, ideas, and graph edges.

Updated Jul 15, 2026
One-click install
npx skills add https://github.com/1549171506/shizhengmubangai1 --skill exp-eval-1549171506
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: exp-eval
Source: https://github.com/1549171506/shizhengmubangai1/tree/main/Desktop/123/empiricalwiki/i18n/en/skills/exp-eval
Command: npx skills add https://github.com/1549171506/shizhengmubangai1 --skill exp-eval-1549171506

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? After running a research experiment, deciding how the results affect your claims and ideas is subjective and error-prone. This Skill turns completed experiment results into structured wiki knowledge updates through an independent cross-model verdict process. ## Core Features & Use Cases - Cross-Model Verdict: A Review LLM independently judges whether results support, partially support, refute, or leave a claim inconclusive, then Claude synthesizes both verdicts conservatively. - Automatic Wiki Updates: Updates claim confidence and evidence, idea status (validated/failed), experiment pages, and knowledge graph edges (supports/invalidates) via the research_wiki.py tool. - Four Verdict Paths: Supported raises confidence and validates ideas; partially supported suggests supplementary experiments; not supported records a specific failure_reason as anti-repetition memory; inconclusive triggers debugging guidance. - Use Case: After completing an experiment testing whether a new attention variant improves benchmark scores, run the evaluation to get an impartial verdict, update the claim's confidence from 0.4 to 0.65, mark the linked idea as validated, and add a supports edge to the research graph. ## Quick Start Evaluate the completed experiment with slug attention-variant-v1 and update my research wiki claims, ideas, and graph based on the verdict.

Frequently Asked Questions about exp-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate a completed research experiment against a claim?

Run the evaluation with the experiment slug once its status is completed. The workflow loads the experiment, target claim, and linked idea, gets an independent Review LLM verdict, synthesizes it with Claude's analysis, and updates the wiki accordingly.

What verdict types does experiment evaluation produce?

Four verdicts are possible: supported, partially_supported, not_supported, and inconclusive. Each path applies different wiki updates, from raising claim confidence and validating ideas to recording failure reasons or suggesting debugging steps.

What happens when Claude and the Review LLM disagree on a verdict?

The disagreement is explicitly flagged and the more conservative verdict is chosen, using the lower confidence value. Methodological concerns like data leakage from either judge take priority over both verdicts.

Can I evaluate an experiment that is not marked completed?

No, only experiments with status completed can be evaluated. The workflow refuses incomplete experiments and directs you to run the experiment execution step first.

What happens if the Review LLM is unavailable during evaluation?

The workflow falls back to a single-model verdict by Claude and notes in the report that cross-model verification was unavailable. You are advised to confirm the verdict later when the review service is restored.

Does a failed experiment delete the associated claim?

No, claims are never deleted. A not_supported verdict lowers confidence and may set status to challenged, and only repeated refutations with confidence near zero lead to a deprecated status.