statistical-design-audit

Audits statistical inference designs for p-values, estimands, clustering, and multiple comparisons.

1|Updated Dec 13, 2025
One-click install
npx skills add https://github.com/ZK-Theory/TDL --skill statistical-design-audit-zk-theory
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: statistical-design-audit
Source: https://github.com/ZK-Theory/TDL/tree/main/.agents/skills/statistical-design-audit
Command: npx skills add https://github.com/ZK-Theory/TDL --skill statistical-design-audit-zk-theory

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Statistical results can pass every software test while the underlying inference is wrong — a mis-specified p-value denominator, an unstated estimand, a non-exchangeable permutation null, or uncorrected multiple comparisons. This Skill audits the statistical design itself before you trust the numbers. ## Core Features & Use Cases - Nine-dimension audit: Checks denominator correctness, Monte Carlo p-value formula, FDR/multiple comparisons, bootstrap setup, estimand stability, eligibility rules, clustering/ICC, null exchangeability, and Markov order specification. - Structured verdicts: Reports PASS / CONCERN / FAIL per dimension with the specific statistic or code location and the required fix, separating machine-checkable items from human-review judgments. - Issue catalogue: Ships a reference guide covering recurring failures (pseudoreplication, circular analysis, MICE non-convergence, IPW instability, Manski bounds misuse) with identification cues and remedies. - Use Case: Before accepting a panel-study rerun that reports significant permutation p-values, run the audit to confirm the denominator is n = min(B, total_pairs), the estimand is unchanged, and Benjamini-Hochberg FDR is applied with a defined family. ## Quick Start Audit the statistical design of my permutation-based panel analysis and report PASS, CONCERN, or FAIL for each audit dimension.

Frequently Asked Questions about statistical-design-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if a Monte Carlo permutation p-value is correct?

Verify the two-sided bias-corrected formula p = (r + 1) / (n + 1), where n = min(B, total_pairs) is the number of null draws actually used, not a diagnostic cap. The minimum achievable p is 1/(n+1) and p must be monotone non-decreasing in r.

How to audit multiple comparisons with FDR correction?

Confirm Benjamini-Hochberg FDR is applied and that the correction family is defined explicitly — which tests, cells, or metrics belong to the family and why. Both raw and adjusted p-values should be reported.

What makes a permutation null non-exchangeable?

A null is non-exchangeable when the shuffle permutes something the null should hold fixed, or operates downstream of where the association lives — for example permuting rows after embedding so the persistence diagram is invariant to the shuffle. The permutation must break only the association under test.

When should I stop and escalate during a statistical audit?

Escalate when a p-value denominator or formula cannot be reconciled with the governing design, when the estimand changed between runs framed as routine reruns, or when a permutation null is invariant to the operation it is supposed to test.

Does this audit replace human statistical judgment?

No. The audit distinguishes machine-checkable items like denominators and formulas, which can bind to contracts, from human-review-only judgments such as estimand appropriateness and exchangeability rationale, which still require expert review.