Wise Use of Null Hypothesis Tests

Chapter 1. The conventional method is a flawed fusion

1.1 Three statisticians, two methods, and the mess that should be banned

1.2 Wise use and testing nulls that must be false

1.3 Null hypothesis testing in perspective

Chapter 2. The point is to generalize beyond our results

2.1 Samples and populations

2.2 Real and hypothetical populations

2.3 Randomization

2.4 Know your population, and do not generalize beyond it

Chapter 3. Null hypothesis testing explained

3.1 The effect of sampling error

3.2 The logic of testing a null hypothesis

3.3 We should know from the start that many null hypotheses cannot be correct

3.4 The traditional explanation of how to use p

3.5 What use of α accomplishes

3.6 The flawed hybrid in action

3.7 Criticisms of the flawed hybrid

3.8 We should test nulls in a way that answers the criticisms

3.9 How to use p and α

3.10 Mouse preference, done right this time

3.11 More p-values in action

3.12 What were the nulls and predictions?

3.13 What if p50.05000?

3.14 A radical but wise way to use p

3.15 0.05 or .05? p or P?

Chapter 4. How often do we get it wrong?

4.1 Distributions around means4.2 Distributions of test statistics

4.3 Null hypothesis testing explained with distributions

4.4 Type I errors explained

4.5 Probabilities before and after collecting data

4.6 The null’s precision explained

4.7 The awkward definition of p explained

4.8 Errors in direction

4.9 Power and errors in direction

4.10 Manipulating power to lower p-values

4.11 Increasing power with one-tailed tests

4.12 Power and why we should we set α to 0.10 or higher

4.13 Power, estimated effect size, and type M errors

4.14 How can we know a population’s distribution?

Chapter 5. Important things to know about null hypothesis testing

5.1 Examples of null hypotheses in proper statistics books and what they really mean

5.2 Categories of null hypotheses?

5.3 What if is important to accept the null?

5.4 Never do this

5.5 Null hypothesis testing as never explained before

5.6 Effect size: what is it and when is it important?

5.7 We should provide all results, even those not statistically “significant”

Chapter 6. Common misconceptions

6.1 Null hypothesis testing is misunderstood by many

6.2 Statistical “significance” means a difference is large enough to be important—wrong!

6.3 p is the probability of a type I error—wrong!

6.4 If results are statistically “significant,” we should accept the alternative hypothesis that something other than the null is correct—wrong!

6.5 If results are not statistically “significant,” we should accept the null hypothesis—wrong!

6.6 Based on p we should either reject or fail to reject the null hypothesis—often wrong!

6.7 Null hypothesis testing is so flawed that we should use confidence intervals instead—wrong!

6.8 Power can be used to justify accepting the null hypothesis—wrong!

6.9 The null hypothesis is a statement of no difference—not always

6.10 The null hypothesis is that there will be no significant difference between the expected and observed values—very, very wrong!

6.11 A null hypothesis should not be a negative statement—wrong!

Chapter 7. The debate over null hypothesis testing and wise use as the solution

7.1 The debate over null hypothesis testing

7.2 Communicate to educate

7.3 Plan ahead

7.4 Test nulls when appropriate, not promiscuously

7.5 Strike the right balance between what is conventional and what is best

7.6 Think outside of the null hypothesis test

7.7 Encourage our audience to draw their own conclusions

7.8 Allow ourselves to draw our own conclusions

7.9 Strike the right balance when providing our results

7.10 Know the misconceptions and do not fall for them

7.11 Do not say that two groups “differ” or “do not differ”

7.12 Provide all results somehow

7.13 Other reformed methods of null hypothesis testing

Chapter 8. Simple principles behind the mathematics and some essential concepts

8.1 Why different types of data require different types of tests

8.1.1 Simple principles behind the mathematics

8.1.2 Numerical data exhibit variation

8.1.3 Nominal data do not exhibit variation

8.1.4 How to tell the difference between nominal and numerical data

8.2 Simple principles behind the analysis of groups of measurements and discrete numerical data

8.2.1 Variance: a statistic of huge importance

8.2.2 Incorporating sample size and the difference between our prediction and our outcome

8.3 Drawing conclusions when we knew all along that the null must be false

8.4 Degrees of freedom explained

8.5 Other types of t tests

8.6 Analysis of variance and t tests have certain requirements

8.7 Do not test for equal variances unless . . .

8.8 Simple principles behind the analysis of counts of observations within categories

8.8.1 Counts of observations within categories

8.8.2 When the null hypothesis specifies the prediction

8.8.3 When there is only one degree of freedom

8.8.4 When the null hypothesis does not specify the prediction

8.9 Interpreting p when the null hypothesis cannot be correct

8.10 232 Designs and other variations

8.11 The problem with chi-squared tests

8.12 The reasoning behind the mathematics

8.13 Rules for chi-squared tests

Chapter 9. The two-sample t test and the importance of pooled variance

Chapter 10. Comparing more than two groups to each other

10.1 If we have three or more samples, most say we cannot use two-sample t tests to compare them two samples at a time

10.2 Analysis of variance

10.3 The price we pay is power

10.4 Comparing every group to every other group

10.5 Comparing multiple groups to a single reference, like a control

10.6 Is all of this a load of rubbish?

Chapter 11. Assessing the combined effects of multiple independent variables

11.1 Independent variables alone and in combination

11.2 No, we may not use multiple t tests

11.3 We have a statistical main effect: now what?

11.4 We have a statistical interaction: things to consider

11.5 We have a statistical interaction and we want to keep testing nulls

11.6 Which is more important, the main effect or the interaction?

11.7 Designs with more than two independent variables

11.8 Use of analysis of variance to reduce variation and increase power

Chapter 12. Comparing slopes: analysis of covariance

12.1 Analysis of covariance

12.2 Use of analysis of covariance to reduce variation and increase power

12.3 More on the use of analysis of covariance to reduce variation and increase power

12.4 Use of analysis of covariance to limit the effects of a confound

Chapter 13. When data do not meet the requirements of t tests and analysis of variance

13.1 When do we need to take action?

13.2 Floor effects and the square root transformation

13.3 Floor and ceiling effects and the arcsine transformation

13.4 Not as simple as a floor or ceiling effect—the rank transformation

13.5 Making analysis of variance sensitive to differences in proportion—the logarithmic transformation

13.6 Nonparametric tests

13.7 Transforming data changes the question being asked

Chapter 14. Reducing variation and increasing power by comparing subjects to themselves

14.1 The simple principle behind the mathematics

14.2 Repeated measures analysis of variances

14.3 Multiple comparisons tests on repeated measures

14.4 When subjects are not organisms

14.5 When repeated does not mean repeated over time

14.6 Pretest-posttest designs illustrate the danger of measures repeated over time

14.7 Repeated measures analysis of variance versus t tests

14.8 The problem with repeated measures

14.8.1 The requirement for sphericity

14.8.2 Correcting for a lack of sphericity

14.8.3 Multiple comparisons tests when there is a lack of sphericity

14.8.4 The multivariate alternative to correction

Chapter 15. What do those error bars mean?

15.1 Confidence intervals

15.2 Testing null hypotheses in our heads

15.3 Plotting confidence intervals

15.4 Error bars and repeated measures

15.5 Plot comparative confidence intervals to make the overlap myth a reality

Bonus chapters:
Appendix A: Philosophical objections
A.1 Decades of bitter debate
A.2 We want to know when we are wrong, not how often
A.3 Setting α to 0.05 does not mean that 5% of all null-based decisions are wrong 158
A.4 There are better ways to analyze and interpret data
A.5 The fallacy of affirming the consequent
A.6 Some say our method cannot be used to determine direction
A.6.1. The return of one-tailed tests
A.6.2. Kaiser’s absurd directional two-tailed tests
A.6.3. Invoking power to justify Kaiser’s directional two-tailed tests
A.6.4. Fisher did not follow Kaiser’s rules
A.6.5. Still not convinced?

Appendix B: How Fisher used null hypothesis tests
B.1 Why follow my advice?
B.2 Fisher tested for direction
B.3 Others did too
B.4 Fisher believed α should vary according to the circumstances
B.5 Fisher came close to saying there should be no α at all
B.6 In practice, Fisher did not categorize outcomes
B.7 Fisher’s language answers many criticisms of null hypothesis testing
B.8 Except for Fisher’s use of “significant”
B.9 Fisher’s inconsistency explained
B.10 Fisher’s thinking expressed in one word
B.11 We have come a long way since Fisher, but the wrong way?

Appendix C: The method attributed to Neyman and Pearson
C.1 Neyman and Pearson with Pearson
C.2 Neyman and Pearson without Pearson
C.3 An important limitation
C.4 Alternatives are always infinitely numerically precise
C.5 The method step-by-step
C.6 The method’s influence on the flawed hybrid
C.7 The method’s fate in the world of the flawed hybrid
C.8 Power spreads its wings
C.9 Neyman et al.’s method has no place in science