Many models are better than one for COVID-19 scenario projections, study finds
During the COVID-19 pandemic, the COVID-19 Scenario Modeling Hub generated look-ahead projections for COVID-19 cases, hospitalizations and deaths under specific, policy-relevant scenarios. Those projections were provided to federal agencies such as the Centers for Disease Control and Prevention, local health authorities and the public to help inform decisions like the release of COVID-19 vaccines for children and when booster shots were recommended.
A team of researchers in the hub, led by Penn State biologists, evaluated the accuracy and reliability of these nearly two million projections — including ensemble models, which aggregate multiple individual model projections for a given scenario — made over two years by retrospectively comparing the model projections with what actually happened. The models showed how various interventions would impact the progression of the pandemic given a specific starting scenario, which the researchers found remained close to reality for 22 weeks on average before the arrival, for example, of an unanticipated variant of the virus. They also found that the ensemble model outperformed individual projections, ranking in the top three models, out of four to nine individual models, 93% of the time.
The team published their findings in the journal Nature Communications. The results highlight the value of a collaborative modeling approach, researchers said, and could have implications for better predicting future disease outbreak scenarios, from the seasonal flu to widespread pandemics.
“The COVID-19 Scenario Modeling Hub solicits projections from multiple, independent modeling teams,” said Emily Howerton, a postdoctoral scholar in biology in the Penn State Eberly College of Science, member of the hub and leader of the research team. “These teams make projections for specific policy-relevant scenarios designed by the hub that aim to collectively answer key public health questions. The hub is a huge collaborative effort, which makes it really exciting.”
The modeling teams come from a range of backgrounds, including academic institutions, government agencies and the private sector to generate scenario projections.
“Unlike weather forecasts, which look at what will happen based on previous trends, scenario projections consider what might happen under a set of specific conditions,” Howerton said, explaining that such a projection could model what might happen if a new variant emerged or help evaluate intervention strategies. “This allows us to make projections further into the future and evaluate the impact of potential interventions.”
Power in numbers
The hub also combines the individual projections into an ensemble scenario projection that includes results from four to nine mathematical models, each developed by the various teams.
“The ensemble projection is a lot like asking a bunch of friends for advice,” Howerton said. “You hear a variety of recommendations, and you combine these recommendations in your mind, looking for similarities and differences as you decide.”
Starting in February 2021, the hub produced 16 rounds of projections, each of which predicted anywhere from three months to a year of pandemic outcomes. According to Howerton, each round was guided by ongoing discussions with state and federal public health partners and reflected the shifting understanding of the origin of and response to the virus. For each round of projections, the hub created four scenarios. For example, in round 2, the hub varied vaccination uptake by variant spread to create the four scenarios — high vaccination uptake with high variant spread, high vaccination with low spread and so on.
In this paper, for each of the 16 rounds, they retrospectively compared those scenario projections with what eventually occurred and evaluated them by two criteria: prospective purpose — whether the scenarios matched reality; and retrospective evaluation — for how long reality matched the scenario specifications. Based on these criteria, the ensemble model typically outperformed all individual models by being a top-two performer 69% of the time and a top-three performer 93% of the time.
“In the Scenario Modeling Hub context, we also saw clear performance improvements of the ensemble over individual models,” said Cécile Viboud, a senior scientist at the Fogarty International Center at the National Institutes of Health (NIH Fogarty) and one of the leaders of the research team. “Not only was the ensemble more accurate in predicting COVID-19 trends overall, compared to individual models, it was also more reliable across all 16 rounds than any one individual model, which is extremely important when it comes to decision making purposes.”
Hub history
The hub was formed in December 2020 and built upon other multi-model efforts aimed at supporting public health decision-making. According to co-author Katriona Shea, the idea to inform decision making with multiple independent models was inspired by an exercise she ran earlier in the pandemic, called Multiple Models for Outbreak Decision Support (MMODS) that combined 17 models to evaluate COVID-19 reopening strategies. Shea, professor of biology and Alumni Professor in the Biological Sciences in the Penn State Eberly College of Science and one of the lead researchers on this team, said she was inspired by her experience in other fields, where collecting and combining expert judgments is common practice.
“This paper shows the true power of an ensemble projection, not only in generating consensus, but also for identifying when there are important things we do not know,” Shea said. “This type of information is essential to decision makers as they manage ongoing disease threats and future pandemics.”
As an example of the power of the ensemble projections, the hub recently contrasted the emergence of a new variant over two years with three different vaccination strategies — no vaccines, vaccines only for those 65 years old or older and vaccines for everyone. These results were shared at the meeting of the Advisory Committee for Immunization Practices, which develops recommendations for vaccines, and were used to aid the decision to release updated vaccines in September.
“This work really emphasizes the importance of coming together as a ‘hub’, answering the same questions and using our collective power to provide more reliable information than any one team could provide alone,” said Justin Lessler, professor of epidemiology in the Gillings School of Global Public Health at the University of North Carolina and one of the leaders of the research team. “This multi-model ensemble framework has been essential to the hub’s impact throughout the course of the pandemic. Our reliable and accurate projections have been a key source of information in informing response to emerging variants and the decisions to release COVID-19 vaccines for children and bivalent booster shots for all age groups.”
In addition to Howerton, Shea, Viboud and Lessler, the COVID-19 Scenario Modeling Hub coordination team includes members from Johns Hopkins University, the University of Massachusetts at Amherst, the University of Pittsburgh, U.S. Geological Survey, Harvard University, NIH Fogarty and the University of North Carolina. The entire research team includes all COVID-19 Scenario Modeling Hub modeling groups from Johns Hopkins University, the National Institutes of Health, the University of Pittsburgh, Columbia University, the University of Florida, Northeastern University, the University of Victoria, the University of Southern California, the University of Virginia, Institute for Health Metrics and Evaluation, University of Notre Dame, North Carolina State University and the University of Texas at Austin. A full list of authors can be found here.
This research was funded by The National Science Foundation, National Institute of General Medical Sciences, Health and Human Services, U.S. Centers for Disease Control and Prevention, Council of State and Territorial Epidemiologists, National Institutes of Health, Defense Threat Reduction Agency, Applied Research Associates, Tuberculosis Research Advancement Centers, and the COVID-19 HPC Consortium funded the research. The Eberly College of Science and Huck Institutes of the Life Sciences at Penn State provided additional support.