result-to-claim

Evaluates experiment results against intended claims using an external LLM reviewer and routes next actions.

75|7|Updated May 2, 2026
One-click install
npx skills add https://github.com/zjunlp/Mechanist --skill result-to-claim-zjunlp
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: result-to-claim
Source: https://github.com/zjunlp/Mechanist/tree/main/skills/result-to-claim
Command: npx skills add https://github.com/zjunlp/Mechanist --skill result-to-claim-zjunlp

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? After experiments finish, it is hard to judge objectively whether the numbers actually support the intended research claim. This Skill removes post-hoc rationalization by delegating the verdict to an external LLM reviewer and routing the project to the right next step. ## Core Features & Use Cases - Result Collection: Gathers metrics from W&B runs, EXPERIMENT_LOG.md, EXPERIMENT_TRACKER.md, training logs, and the research contract. - External LLM Judgment: Sends results to a reviewer via the llm-chat MCP and parses a strict pass/fail verdict with evidence gaps and suggested claim revisions. - Integrity-Aware Routing: Checks EXPERIMENT_AUDIT.json for integrity warnings, then routes to claim narrowing, supplementary experiments, pivoting, ablation planning, or paper writing. - Use Case: After a mechanistic interpretability experiment completes, run this gate before writing the paper to confirm the claim is supported or to identify which supplementary experiments are still needed. ## Quick Start Run the result-to-claim gate on my latest W&B run to judge whether the results support the intended claim.

Frequently Asked Questions about result-to-claim

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate whether experiment results support my research claim?

Run this gate after experiments complete. It collects metrics from W&B, EXPERIMENT_LOG.md, and logs, then asks an external LLM reviewer for a strict pass/fail verdict with evidence gaps and suggested claim revisions.

How do I configure the external reviewer model for result evaluation?

Set LLM_MODEL, LLM_BASE_URL, and LLM_API_KEY under mcpServers.llm-chat.env in the project .mcp.json or ~/.claude/settings.json, or export them as shell environment variables. The skill aborts if none of these sources provides a model.

What happens when the result-to-claim verdict is fail?

A fail verdict covers refuted, partial, or borderline support. The skill records a postmortem in findings.md, then routes to narrowing the claim, running supplementary experiments, or pivoting to the next idea from IDEA_CANDIDATES.md.

Does the result-to-claim gate work if the llm-chat MCP is unavailable?

Yes. If the external reviewer call fails, the skill makes its own judgment and marks it as pending external review, so the research pipeline is not blocked.

How does experiment integrity auditing affect the verdict?

If EXPERIMENT_AUDIT.json reports a fail status, the verdict is flagged with an integrity concern and confidence is downgraded to low. A warn status appends a warning, and a missing audit marks the verdict as provisional without blocking.