Moving to a World Beyond “p < 0.05”

How do scientists determine whether or not results are interesting, important, worth pursuing further? In part, this has to do with statistical analysis of the data collected. This project looks at one of the traditional aspects -- "statistical significance" and how to move away from it.

Professor of Statistics, Nicole Lazar, co-edited a Special Issue of "The American Statistician" (sort of "Scientific American" for statisticians - a general-interest, less technical journal) on how to move statistical inference into the 21st century, by moving beyond arbitrary thresholds used traditionally for establishing "statistical significance."

Statistics is used to make many decisions in science: efficacy of new cancer treatments, efficacy of treatments for new diseases (such as COVID-19), and many more questions of broad public interest. It's crucial that we make those decisions in a principled, transparent, and (hopefully) replicable way. This project aims to improve the use of statistics when we make these important decisions.

The Special Issue gathered papers from scientists and statisticians with a variety of perspectives, backgrounds, and opinions. Some of these were invited, and others were contributed after a general call to the statistics community for papers. There were articles on different aspects of science, different proposals for how to modernize statistical inference, how to change the way that we educate students. As co-editor of the Issue, Lazar was also co-author on the lead editorial that summarized the 40+ papers.

We came up with the following principles: Accept uncertainty; be transparent, open, and modest - think "ATOM"!

Nicole Lazar

Professor of Statistics

This work is a step in a major shift on how statistics is used in scientific inference. As such, the authors of all of the papers that were submitted to the Special Issue (whether or not they were ultimately accepted) were an important part of the process. Lazar and her colleagues gathered an expert team of associate editors to help in evaluating the submitted papers and improving them through the review process. Lazar and the other two co-editors worked together in a very strong collaboration to synthesize the papers into a coherent editorial - which was then reviewed and commented on by over a dozen colleagues and peers around the world. Without the collaboration from all of these different groups and people, the Special Issue would not have been possible. And it has already had a very large impact, a year-and-a-half after its publication.

Statistical inference is hard and there is a lot of uncertainty in science. That doesn't mean that if a conclusion changes from one study to the next, that "something has gone wrong." This is part of the scientific process. Much of this hinges, though, on the statistical analysis being performed correctly, and the conclusions being drawn appropriately. A finding that is "statistically significant" may or may not be scientifically meaningful (or "scientifically significant"); it is worthy of further exploration and follow-up with another study. A finding that is not "statistically significant" may or may not be scientifically meaningful. One study on its own can't tell us much; repeated studies that converge on a consistent story are more convincing.

We came up with the following principles: Accept uncertainty; be transparent, open, and modest - think "ATOM"!

The entire Special Issue of "The American Statistician" is available online (only) and open access for everyone, in perpetuity. This includes all of the 40+ articles and the lead editorial. The Special Issue generated a lot of press, with stories in many major scientific and lay outlets.

Nicole Lazar is a Professor of Statistics and Director of Online Programs at Penn State. Her research interests include the foundations of statistical inference and the analysis of functional neuroimaging data. In particular, she has worked on fundamental inferential topics such as model selection, multiple testing problems, and likelihood theory, specifically in the context of modern large-scale data analysis problems.