pre-registering-eval-study

Lock a study's hypothesis, outcome, stopping rule, and falsification criteria before data observation.

2|Updated May 23, 2026

Search Tags:#hypothesis testing #research methodology #power analysis #study design #experimental design #preregistration #llm evals

Ready to run

npx skills add https://github.com/rocklambros/rcs --skill pre-registering-eval-study

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pre-registering-eval-study
Source: https://github.com/rocklambros/rcs/tree/main/skills/workflow/pre-registering-eval-study
Command: npx skills add https://github.com/rocklambros/rcs --skill pre-registering-eval-study

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill prevents HARKing, p-hacking, optional stopping, and outcome switching by forcing a study design to be specified before any data is observed.

Core Features & Use Cases

Hypothesis Locking: Turns vague ideas into a directional, falsifiable primary hypothesis with an operationalized outcome.
Analysis Planning: Fixes the statistical test, minimum effect of interest, power justification, stopping rule, and falsification criterion in advance.
Confirmatory Study Workflows: Supports LLM evals, jailbreak studies, A/B tests, clinical-style studies, observational analyses, and model comparisons that need defensible pre-registration.

Quick Start

Ask the skill to pre-register your study before any data is collected, specifying the hypothesis, primary outcome, effect threshold, stopping rule, and commit location.

Frequently Asked Questions about pre-registering-eval-study

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I preregister a study design to prevent p-hacking and HARKing?▼

Preregistration locks your study's hypothesis, primary outcome, stopping rule, and falsification criteria before data collection. This prevents p-hacking and HARKing by requiring a versioned document with a recorded commit hash and a power-aware analysis plan.

Can I use a preregistration workflow for LLM evals and A/B testing?▼

Yes, preregistration supports confirmatory study workflows for LLM evals, A/B tests, and model comparisons. You must specify a directional, falsifiable primary hypothesis and an operationalized outcome before observing any evaluation data.

What is hypothesis locking in experimental design?▼

Hypothesis locking transforms vague research ideas into a directional, falsifiable primary hypothesis with an operationalized outcome. It fixes your statistical test, minimum effect of interest, and falsification criterion in advance to ensure defensible study design.

How do I set up a power-aware analysis plan before data collection?▼

To set up a power-aware analysis plan, specify your primary outcome, fixed effect threshold, minimum effect of interest, and stopping rule. This plan must be recorded in a versioned preregistration document with a commit hash before any data is observed.

When do I need to lock my study's stopping rule and falsification criteria?▼

You must lock stopping rules and falsification criteria before any data is observed to prevent optional stopping and outcome switching. This is required for clinical, observational, or experimental studies that will support a confirmatory claim.

What are the limitations of preregistering an observational study?▼

Preregistration for observational studies requires a fixed effect threshold and pre-specified power-aware analysis plan. You cannot use it for exploratory analyses without a falsifiable primary hypothesis, an operationalized outcome, and a recorded commit hash.