Jackson Pollock: Number 17A
The love of complexity without reductionism makes art; the love of complexity with reductionism makes science. E.O. Wilson
Sellke T, Bayarri MJ, Berger JO. Calibration of p values for testing precise null hypotheses. The American Statistician. 2001 Feb 1;55(1):62-71.
Unfortunately, they are [p values] often incorrectly viewed as an error probability for rejection of the hypothesis or, even worse, as the posterior probability that the hypothesis is true.
Table 1 presents various p values and their associated calibrations. Thus, p = .05 translates into odds B(.05) = .407 (roughly 1 to 2.5) of H0 to H1, and frequentist error probability α(.05) = .289 in rejecting H0. (The default posterior probability of H0 would also be .289.) Clearly p = .05 does not indicate particularly strong evidence against H0. Even p = .01 corresponds to only about 8 to 1 odds against H0.
Formula for calculating α:
Formula for Excel:
=1/(1-1/EXP(1)/A1/LN(A1))
Cumming G, Maillardet R. Confidence intervals and replication: where will the next mean fall?. Psychological methods. 2006 Sep;11(3):217.
On average, a 95% CI will include just 83.4% of future replication means.
Formula for Excel:
=200*(NORMSDIST(NORMSINV(A1/200+0,5)/SQRT(2))-0,5)
de Winter JC. Using the Student’s t-test with extremely small sample sizes. Practical Assessment, Research & Evaluation. 2013 Aug 1;18(10):1-2.
This study further shows that a paired t-test is feasible with extremely small Ns if the within-pair correlation is high. It is concluded that there are no principal objections to using a t-test with Ns as small as 2.
