P=0.05, CI95%

Jackson Pollock: Number 17A

The love of complexity without reductionism makes art; the love of complexity with reductionism makes science. E.O. Wilson

Sellke T, Bayarri MJ, Berger JO. Calibration of p values for testing precise null hypotheses. The American Statistician. 2001 Feb 1;55(1):62-71.

Unfortunately, they are [p values] often incorrectly viewed as an error probability for rejection of the hypothesis or, even worse, as the posterior probability that the hypothesis is true.

Table 1 presents various p values and their associated calibrations. Thus, p = .05 translates into odds B(.05) = .407 (roughly 1 to 2.5) of H0 to H1, and frequentist error probability α(.05) = .289 in rejecting H0. (The default posterior probability of H0 would also be .289.) Clearly p = .05 does not indicate particularly strong evidence against H0. Even p = .01 corresponds to only about 8 to 1 odds against H0.

Formula for calculating α:

Formula for Excel:

=1/(1-1/EXP(1)/A1/LN(A1))
Cumming G, Maillardet R. Confidence intervals and replication: where will the next mean fall?. Psychological methods. 2006 Sep;11(3):217.

On average, a 95% CI will include just 83.4% of future replication means.

Formula for Excel:

=200*(NORMSDIST(NORMSINV(A1/200+0,5)/SQRT(2))-0,5)

de Winter JC. Using the Student’s t-test with extremely small sample sizes. Practical Assessment, Research & Evaluation. 2013 Aug 1;18(10):1-2.

This study further shows that a paired t-test is feasible with extremely small Ns if the within-pair correlation is high. It is concluded that there are no principal objections to using a t-test with Ns as small as 2.

The fox has many tricks; the hedgehog only one great trick. Erasmus