Statistical Power: When is it useful?

Statistical power is one of the most important ideas in statistics—and one of the most misunderstood. In this episode, we explain what statistical power really means, why researchers need to think about it before they collect their data, and what goes wrong when they calculate “post hoc power” after the study is over. Along the way, we hunt for UFOs, test Kristin’s psychic abilities, explore the ethics of underpowered studies, and discover why post hoc power is really just a p-value wearing a fake mustache. Then statistician and powerlifter Andrew Althouse joins us for the inside story of a years-long scientific feud over post hoc power, where we discuss what happens when being wrong about statistics turns into doubling down, angry letters to the editor, and debates about how science corrects itself.

Statistical topics

• Effect size
• False negatives
• P-values
• Post hoc power
• Power calculations
• Research ethics
• Sample size
• Scientific controversy
• Sensitivity analysis
• Statistical power
• Statistical significance
• Study design
• Type II error

Methodologic Morals

• “Big questions need studies powerful enough to answer them.”
• “Post hoc power is circular reasoning masquerading as new information.”
• “Know when to cut your losses. A mistake gets more expensive every time you double down.”

References

• Althouse AD, Chow ZR. Comment on “Post-hoc power: If you must, at least try to understand.” Ann Surg. 2019;270:e78-e79.
• Althouse AD. Post hoc power: Not empowering, just misleading. J Surg Res. 2021;259:A3-A6.
• Altman DG. Statistics and ethics in medical research: III. How large a sample? BMJ. 1980;281:1336-1338.
• Bababekov YJ, Hung YC, Hsu YT, et al. Is the power threshold of 0.8 applicable to surgical science?—empowering the underpowered study. J Surg Res. 2019;241:235-239.
• Bababekov YJ, Stapleton SM, Mueller JL, et al. A proposal to mitigate the consequences of type 2 error in surgical science. Ann Surg. 2018;267:621-622.
• Chang DC, Stapleton SM. Response: The proliferation and misinterpretation of “as safe as” statements in surgical science: A call for professional discourse to search for a solution. J Surg Res. 2021;259:A12-A15.
• Freiman JA, Chalmers TC, Smith H Jr, et al. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial: Survey of 71 negative trials. N Engl J Med. 1978;299:690-694. 
• Hoenig JM, Heisey DM. The abuse of power: The pervasive fallacy of power calculations for data analysis. Am Stat. 2001;55:19-24.
• Nuzzo RL. Statistical Power. PM&R. 2016;8(9):907-912. doi:10.1016/j.pmrj.2016.08.004
• Nuzzo RL. Post hoc Power. PM&R. 2021;13(4):422-424. doi:10.1002/pmrj.12476

Kristin and Regina’s online courses: 

Demystifying Data: A Modern Approach to Statistical Understanding  

Clinical Trials: Design, Strategy, and Analysis 

Medical Statistics Certificate Program  

Writing in the Sciences 

Epidemiology and Clinical Research Graduate Certificate Program 

Programs that we teach in:

Epidemiology and Clinical Research Graduate Certificate Program 

Find us on:

Kristin -  LinkedIn & Twitter/X

Regina - LinkedIn &ReginaNuzzo.com

• (00:00) - Intro
• (00:43) - What is Statistical Power?
• (03:22) - The UFO Telescope Analogy
• (06:19) - The Psychic Ability Experiment
• (10:36) - Planning Power Before You Study
• (12:46) - History and the Landmark 1978 Paper
• (17:42) - Enter Post-Hoc Power: The Problem
• (19:26) - Three Problems with Post-Hoc Power
• (24:33) - The Surgery Journal Disaster
• (29:59) - Guest Interview: Andrew Althouse
• (32:30) - The 2018 Paper and the Controversy Begins
• (39:54) - Doubling Down and the Cost of Being Right
• (53:03) - Rating the Claim