MPH 560 Module 4 Hypothesis Testing and P Values Example

What a P Value Can and Cannot Tell You: Hypothesis Testing in Public Health

Student Name

Master of Public Health Program, Aspen University

MPH 560: Applied Biostatistics for Public Health

Instructor Name

Month Day, Year

What a P Value Can and Cannot Tell You: Hypothesis Testing in Public Health

Hypothesis tests are among the most used and most misunderstood tools in public health research. A P value can help judge whether data are compatible with a hypothesis, but it cannot tell whether a finding is important or true. This paper explains the logic of hypothesis testing through a composite example and reviews guidance on interpreting P values correctly.

The Logic of Testing

A hypothesis test begins with a null hypothesis, usually of no difference or no effect, and an alternative hypothesis. Researchers calculate a test statistic that measures how far the data depart from what the null predicts, then a P value: the probability of obtaining a result at least as extreme as the one observed if the null hypothesis and all other model assumptions were true (Whitley & Ball, 2002).

A Composite Example

A composite clinic reports that 174 of 300 adolescents, 58%, have started HPV vaccination. The state benchmark is 70%. The null hypothesis is that the clinic's true rate is 70%; the alternative is that it differs. Under the null, the standard error of a proportion from 300 is the square root of 0.70 times 0.30 divided by 300, or 0.0265. The z statistic is 0.58 minus 0.70, divided by 0.0265, or about -4.54. The two-sided P value is well below .001.

Decisions and Errors

With a significance level of .05, the result leads to rejecting the null hypothesis. Every decision risks one of two errors, as the table shows.

Power

Power is the probability of detecting a real effect of a given size. It increases with sample size, effect size and the significance level. Studies with low power often miss important effects. Planning sample size before collecting data ensures adequate power, commonly 80% or 90%, for the smallest effect worth detecting.

What the P Value Means Here

The very small P value means that a sample result as far from 70% as 58% would be extremely unlikely if the clinic's true rate were 70%. It supports the conclusion that the clinic is below the benchmark. It does not measure how far below; for that, the estimate and its confidence interval, roughly 52% to 64%, are more informative.

The ASA Statement

Concerned about misuse, the American Statistical Association issued principles on P values. In paraphrase, the principles say that a P value shows how poorly data fit a stated model; that it is not the chance a hypothesis is correct, nor the chance that results are a fluke; that conclusions should not hinge on crossing a cutoff; that honest inference needs complete reporting; that a small P value does not mean a large or important effect; and that a P value on its own is weak evidence (Wasserstein & Lazar, 2016).

Common Misinterpretations

A guide to misinterpretations lists many errors, including treating P as the probability the null is true, treating P above .05 as proof of no effect, treating statistical significance as practical importance and comparing studies by whether each is significant rather than by their estimates (Greenland et al., 2016). Each error can mislead public health decisions.

Absence of Evidence

A non-significant result does not show that there is no effect. A small study may simply lack power, and its confidence interval may include both no effect and a substantial effect. Statisticians have emphasized that absence of evidence is not evidence of absence (Altman & Bland, 1995). Reporting the interval makes this clear.

Multiple Testing

When many tests are run, some will be significant by chance. With 20 independent tests at the .05 level, one false positive is expected even if all null hypotheses are true. Analysts should plan primary hypotheses in advance and interpret additional findings cautiously.

Good Practice

Good practice is to state hypotheses before analysis, report estimates with confidence intervals and exact P values, describe all analyses conducted, interpret results in light of study design and prior evidence and avoid describing results only as significant or not. Where possible, analysts should also describe how the sample size was chosen and what effect the study was powered to detect, which helps readers interpret non-significant findings.

One-Sided and Two-Sided Tests

A two-sided test asks whether a value differs in either direction; a one-sided test asks about one direction only. Two-sided tests are standard in health research because unexpected effects in the opposite direction matter. A one-sided test should be chosen before seeing the data and justified.

Statistical and Practical Significance

With very large samples, tiny differences become statistically significant; with small samples, important differences may not. A difference of 0.5 mm Hg in blood pressure might be significant in a study of 100,000 people but meaningless for health. Judging practical significance requires subject knowledge, not just statistics.

Preregistration and Transparency

Deciding hypotheses and analyses before seeing the data, and recording them publicly, protects against selective reporting. Many journals and funders now expect registered protocols for trials. Transparency about all analyses performed lets readers judge how much weight to give each result.

Applying the Principles

For the clinic, the useful message is not simply that its rate is significantly below the benchmark, but that it is about 12 points lower, with a plausible range of roughly 6 to 18 points, and that closing the gap would protect many adolescents.

Hypothesis Tests in Program Evaluation

Program managers often test whether a rate changed after an intervention. The same principles apply: define the hypothesis in advance, choose an appropriate test, report the change with its interval and consider whether other factors could explain it. A significant change does not by itself show that the program caused it.

Conclusion

Hypothesis testing asks whether data are compatible with a null hypothesis, and the P value summarizes that compatibility. In the composite clinic example, a rate of 58% against a 70% benchmark produced a z of about -4.54 and a P value below .001. But P values do not measure the size or truth of effects, and non-significant results do not prove absence. Reporting estimates and intervals, and interpreting results in context, leads to sounder public health decisions.

References

Altman, D. G., & Bland, J. M. (1995). Absence of evidence is not evidence of absence. BMJ, 311(7003), 485. https://doi.org/10.1136/bmj.311.7003.485

Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337-350. https://doi.org/10.1007/s10654-016-0149-3

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108

Whitley, E., & Ball, J. (2002). Statistics review 3: Hypothesis testing and P values. Critical Care, 6(3), 222-225. https://doi.org/10.1186/cc1493