Most of the general public had never heard of epidemic modeling before the coronavirus disease 2019 (COVID-19) pandemic. CovidSim1, developed by modelers at Imperial College London, has been one of the most influential epidemic models used to inform governments on the COVID-19 pandemic response. In February 2020, although data from China showed that nationwide lockdown was able to bring the epidemic under control within the country2, other countries were uncertain whether such a lockdown would be feasible or desirable in their populations. In early March 2020, CovidSim was used to simulate the effects of various non-pharmaceutical interventions (NPIs) on the spread of COVID-19 in the UK and the US. Those results helped inform the British and US governments that lockdown would be necessary to avoid hospital overloads and massive deaths3. Since then, several groups of prominent scientists and software engineers have sought to evaluate the reliability of CovidSim4,5. In this issue of Nature Computational Science, Edeling et al.6 present a comprehensive examination of CovidSim simulations to quantify the uncertainties in the model and better assess the impact of these uncertainties on the model output (Fig. 1).

Fig. 1: The sources of uncertainty of COVID-19 pandemic models.
figure 1

Edeling et al.6 present a comprehensive examination of CovidSim simulations to quantify the uncertainties from three sources, including parameter uncertainty, model structure uncertainty and scenario uncertainty. The study also compares the CovidSim output with actual data to evaluate the robustness of model predictions.

Full size image

Understanding and quantifying the uncertainty of predictions from epidemic models is essential, which will not only improve the quality of decision making with constructive scrutiny but also make us more open to revise our decisions when new data and evidence emerge7. Leveraging on the supercomputing power in their research facilities, Edeling et al. identified three major sources of uncertainties that might affect the robustness of CovidSim predictions, including parametric uncertainty, model structure uncertainty and scenario uncertainty. First, the authors assessed the 940 parameters used as input to CovidSim and showed that the majority of variance in the outputs was due to the variance in only 19 parameters. Three key parameters, namely the duration of latent period, reduction in contact rates given social distancing, and time between infection and case isolation, accounted for more than 50% of the variance in two selected pandemic scenarios. The results indicated that even small changes in these key parameters could increase the predicted number of deaths in the UK by up to 300%. These experiments highlighted that pandemic models would benefit substantially from sensitivity analyses with thorough probabilistic sampling from the parameter space, especially in the early stage of the pandemic when data are very limited.

In addition to parameter uncertainty, their work also emphasizes the importance of model structure uncertainty. Pandemic model structure should be updated to incorporate new knowledge about transmission. For example, evidence for disease transmission by pre-symptomatic or asymptomatic carriers was absent (or very limited) when CovidSim was first developed8. As more data become available, an epidemiological model should be modified to account for asymptomatic infections and the difficulty in detecting and isolating them, in addition to updating at least two of the key parameters (that is, the latent period and the time between infection and case isolation). The first version of CovidSim did not account for superspreading events, which are now known to be responsible for a high proportion of COVID-19 transmission. Moreover, in view of the recent emergence and spread of mutant SARS-CoV-2 lineages, the original CovidSim model structure could be strengthened to project how the threat of mutant emergence would scale with the size of uncontrolled epidemics9.

Considering the scenario uncertainty, model predictions can be further improved by incorporating behavioral responses to policy interventions that may change over the course of the pandemic. The study presents two suppression (or lockdown) scenarios that were closest to the actual NPIs implemented in the UK and one additional scenario to mimic the actual on/off triggers of NPIs based on the weekly number of new intensive care unit (ICU) cases. However, even retrospectively, the authors found it challenging to tune the CovidSim input to create a scenario that resembled what had actually happened in the UK. Due to the dynamic nature of the pandemic and the responses to it, CovidSim and other pandemic models need to be updated accordingly and should mostly be used for short-term predictions (<3 months). The effects of NPIs are not time-invariant either: evidence from behavioral studies indeed shows that the adherence of NPIs changed over time, and NPIs that were effective in earlier waves might not be as effective in later waves10,11.

Public health decision making benefits from the proposed evaluation of uncertainty and assessment of the performance of model predictions. In their study, Edeling et al. compared the cumulative number of deaths from the simulation output with the actual death toll. The comparison seems intuitive, because the public is most concerned about the risk of death imposed by COVID-19. However, it should be acknowledged that while some sources of uncertainty have been shown to have more marginal impacts than others, these uncertainties could collectively have a compounding and cascading effect on the robustness of predictions. Furthermore, future studies should clarify the functional aims of model prediction assessments in the context of pandemic response. For example, if the primary objective of the models is to make recommendations about NPIs, the prediction of the effective reproductive number, R, is likely to be more relevant than the prediction of pandemic deaths. R provides a more direct and immediate measure of transmissibility of COVID-19 and hence the effect of NPIs over time, which CovidSim predicts reasonably well. In contrast, comparing the predicted cumulative number of deaths with the actual figures seems less appropriate in this case because (1) it is not easy to evaluate the effect of implementation, adjustment or relaxation of any NPIs based on the death toll given the one-month delay between infection and death; and (2) the cumulative number of deaths reflects the effects of NPIs accumulated and amplified over time. It will be interesting in the future to also evaluate and compare the prediction of R and how this relates to the uncertainty of the models.

Routine evaluation of uncertainty of predictions from epidemic models is essential to inform public health decision making, which is an important area for future research in the preparedness of the next pandemic.