statistics

Choosing a statistical test depends on the question type, number of groups, and data distribution. The Skill provides a decision table mapping designs to tests, such as Welch's t-test for two independent means, Mann–Whitney U for non-normal data, and chi-square for categorical associations.

Sample size calculation uses statsmodels.stats.power with your significance level (typically 0.05), power (typically 0.80), and minimum detectable effect. Compute it before launching the experiment to avoid underpowered results and invalid early stopping.

scipy.stats covers standalone hypothesis tests, distributions, and resampling methods like bootstrap and permutation tests. statsmodels is better for regression, GLMs, and ANOVA where you need full model summaries, diagnostics, and inference on coefficients.

Causal claims from observational data require an explicit identification strategy, not just correlation. Draw a DAG to identify confounders, then apply matching, inverse-probability weighting, difference-in-differences, instrumental variables, or regression discontinuity, and report the assumptions each requires.

Peeking inflates false positives because repeatedly testing until significant breaks the nominal error rate. Use a fixed horizon determined by upfront power analysis, or apply sequential, group-sequential, or always-valid testing methods if monitoring is required.

Always report the effect size and a confidence or credible interval alongside the p-value. A p-value alone says nothing about practical magnitude, and a non-significant result does not prove the absence of an effect.