Stress-test research claims by swapping method, dataset, and model variants.
npx skills add https://github.com/zjunlp/Mechanist --skill auto-verify-zjunlp
Please help me install this Agent Skill. Skill: auto-verify Source: https://github.com/zjunlp/Mechanist/tree/main/skills/auto-verify Command: npx skills add https://github.com/zjunlp/Mechanist --skill auto-verify-zjunlp
High-intent search queries and answers about installing and using this skill.
Invoke the skill with a claim id such as C1, or leave the argument empty to target all claims. It audits the main experiment, runs one swap variant per selected dimension (method, dataset, model), then computes a robustness score to assign PASS or FAIL.
Robustness equals the number of passing variants divided by eligible variants that survived the integrity audit. A claim passes when robustness meets the threshold, default 0.5, meaning at least half of eligible variants must agree with the main experiment verdict.
Yes, set swap-variants to false for audit-only mode. The main-experiment audit still runs per claim and writes an INTEGRITY_ONLY or INCONCLUSIVE result, skipping all variant execution and GPU usage.
INCONCLUSIVE means the main experiment failed its integrity audit, so variants never ran and iteration must fix the main experiment. ZERO_ELIGIBLE_VARIANTS means variants ran but all failed their own integrity audit, so only the variant evaluation needs fixing.
Re-invoke with resume set to true. Each phase checks whether its primary artifact already exists on disk and skips completed work, including per-claim audits, variant runs, and verdict files, without deleting prior outputs.
By default one claim per pass (MAX_VERIFY_CLAIMS=1) with one variant per listed dimension, so the default model-only dimension yields one variant run. Raise the cap or broaden dimensions to expand coverage.
raja21068Chain method refinement and experiment planning into a research proposal pipeline.
LambenthanJudge completed experiments and update wiki claims with verdicts.
chuwenWuCritically evaluate scientific claims for methodological rigor and bias.
ARAVINDAN20Plan experimental designs with baselines, datasets, hyperparameters, and evaluation metrics.
