∎

11institutetext: 1 Institute of Public Health, Medical Decision Making and
Health Technology Assessment, Department of Public Health, Health Services
Research and Health Technology Assessment, UMIT – University for Health
Sciences, Medical Informatics and Technology, Austria
22institutetext: 2 Department of Medical Statistics, University Medical Center
Göttingen, Germany
33institutetext: 3 Zeppelin University Friedrichshafen, Germany
44institutetext: 4 Pädagogische Hochschule Ludwigsburg, Germany
55institutetext: 5 Cytel Inc, 675, Massachusetts Avenue, Cambridge, MA 02139
USA
66institutetext: 6 Economic and Social Statistics, Trier University, Germany
77institutetext: 7 Department of Statistics, TU Dortmund University, Germany
88institutetext: 8 Psychology and Methods, Jacobs University Bremen, Germany
99institutetext: 9 Department of Systems Biology & Bioinformatics, University
of Rostock and Leibniz-Institute for Food Systems Biology, Technical
University of Munich, Germany
1010institutetext: 10 Goethe University Frankfurt, Germany
1111institutetext: 11 Institute for Technology Assessment and Department of
Radiology; Massachusetts General Hospital; Harvard Medical School, Boston, MA,
USA
1212institutetext: 12 Center for Health Decision Science and Departments of
Epidemiology and Health Policy & Management, Harvard T.H. Chan School of
Public Health, Boston, MA, USA

# On the role of data, statistics and decisions in a pandemic

Beate Jahn1,† Sarah Friedrich2,† Joachim Behnke3 Joachim Engel4 Ursula
Garczarek5 Ralf Münnich6 Markus Pauly7 Adalbert Wilhelm8 Olaf Wolkenhauer9
Markus Zwick10 Uwe Siebert1,11,12 Tim Friede2111Corresponding author:  †
Shared first authorship<EMAIL_ADDRESS>
(Received: date / Accepted: date)

###### Abstract

A pandemic poses particular challenges to decision-making because of the need
to continuously adapt decisions to rapidly changing evidence and available
data. For example, which countermeasures are appropriate at a particular stage
of the pandemic? How can the severity of the pandemic be measured? What is the
effect of vaccination in the population and which groups should be vaccinated
first? The process of decision-making starts with data collection and modeling
and continues to the dissemination of results and the subsequent decisions
taken. The goal of this paper is to give an overview of this process and to
provide recommendations for the different steps from a statistical
perspective. In particular, we discuss a range of modeling techniques
including mathematical, statistical and decision-analytic models along with
their applications in the COVID-19 context. With this overview, we aim to
foster the understanding of the goals of these modeling approaches and the
specific data requirements that are essential for the interpretation of
results and for successful interdisciplinary collaborations. A special focus
is on the role played by data in these different models, and we incorporate
into the discussion the importance of statistical literacy, and of effective
dissemination and communication of findings.

###### Keywords:

COVID-19 SARS-CoV-2 Health-decision framework Decision-analytic modeling

##  1 Introduction

In December 2019, the first cases of coronavirus disease 2019 (COVID-19) were
reported in Wuhan, China  (Zhou et al., 2020; Wu et al., 2020) and the
outbreak of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) was
declared a pandemic in March 2020 by the World Health Organization. In order
to control the spread of the virus and limit the negative consequences of the
pandemic, important decisions had and still have to be made. These concern the
spread of the disease, its impact on health, the utilization of health care
resources or potential effects of counter measures and vaccination strategies,
to name some examples. Statistical modeling plays an important role in
different fields of COVID-19 research. This starts with the collection of
adequate data and the preprocessing of this data, a complex sequence of steps,
where input is required from the data users, taking into account their
questions and information needs. After this preprocessing, examples of
statistical models range from characterizing the disease (Küchenhoff et al.,
2021; Roy et al., 2021; Luo et al., 2021), investigating comorbidities (Gross
et al., 2021; Hadzibegovic et al., 2021; Evangelou et al., 2021), evaluating
new treatments and vaccines with respect to efficacy and safety (Horby et al.,
2020; RECOVERY Collaborative Group, 2020; Shinde et al., 2021; Flaxman et al.,
2020) as well as planning corresponding trials (Mütze and Friede, 2020;
Stallard et al., 2020; Beyersmann et al., 2021), assessing the spread of the
disease in potential scenarios – such as comparing lockdown or vaccination
strategies (Nussbaumer-Streit et al., 2020; Van Pelt et al., 2021; Jahn et
al., 2021) – and evaluating the impact of the pandemic on clinical trials
(Kunz et al., 2020; Anker et al., 2020). One important aspect in the special
situation of a pandemic with a novel pathogen is the incorporation of
sequential inference, that is, continuously updating the research as new data
becomes available.

In the course of the pandemic, the availability and quality of data, the
varying interpretations of modeling results, as well as apparently
contradicting statements by scientists, have caused confusion and fostered
intense debate. The role, use and misuse of modeling for infectious disease
policy making has been critically discussed (James et al., 2021; Holmdahl and
Buckee, 2020). Furthermore, the CODAG reports (COVID-19 Data Analysis Group,
2021) clarify why models can lead to conflicting conclusions and discuss the
purposes of modeling and the validity of the results. For instance, policies
to contain the pandemic were –  in the beginning – mainly guided by 7-day
incidence. Measures such as curfews, limited numbers of guests at events and
restricted opening hours of stores were driven by this figure. However,
considering the 7-day incidence alone does not provide a meaningful view of
the overall picture as discussed by Küchenhoff et al. (2021). As mentioned in
the series “Unstatistik” (RWI – Leibniz-Institut für Wirtschaftsforschung,
2020) a value of 50 cases per 100,000 inhabitants in October 2020 in Germany
had an entirely different meaning than six months earlier due to changes in
testing strategies, and improved treatments among other factors. Concerning
the expected number of intensive care patients and deaths, a value of
approximately 50 in October 2020 is likely to correspond to a value of 15 to
20 in April 2020, possibly even less (RWI – Leibniz-Institut für
Wirtschaftsforschung, 2020). Recently, the hospitalization and ICU incidences
have been considered as additional measures. While this provides a more
reliable picture of the severity of the situation and is less affected by
differing testing strategies, it is not without shortcomings. For example,
under-reporting and time-lags lead to large differences between reported and
actual numbers. Moreover, as the severity of COVID infections dropped with the
Omicron variant and prevalence increased, a new discussion of hospitalization
“with” or “because of” COVID-19 emerged. These examples highlight the need to
use statistical methods such as nowcasting (Günther et al., 2021; Schneble et
al., 2021; Salas, 2021; Altmejd et al., 2020) for more precise estimations.

As known from the field of evidence-based medicine and health data and
decision science, decisions should be underpinned by the best available
evidence. For evidence-based decision making, three components are important:
a) data, b) statistical, mathematical and decision-analytic models (which
reduce the amount and the complexity of the data to meaningful indices,
visualizations, and/or predictions), and c) a set of available decisions,
interventions or strategies with their consequences described through a
utility or loss function (decision-making framework) and the related
tradeoffs.  General international guidance on these assessments and decision
analysis is implemented in a country-specific manner, mainly by Health
Technology Assessment (HTA) organisations (Drummond et al., 2008; Gandjour,
2020). COVID-19 examples include the evaluation of vaccination strategies
(Kohli et al., 2021; Debrabant et al., 2021; Reddy et al., 2021) or treatment
(Sheinson et al., 2021). Scientists of the German Network for Evidence-Based
Medicine raised the question on “COVID-19: Where is the evidence?” (EbM-
Netzwerk, 2020) which motivated a discussion about the need of randomized
controlled trials to investigate the effectiveness of preventive measures,
feasibility of such studies and longitudinal, representative data generation.

In the pandemic, a multitude of models have been used but the systematic
comparison across different classes of models is lacking. The goal of this
paper is to provide an overview of the process from data collection (primary
and secondary) and modeling, up to communication and decision making and to
provide recommendations related to these areas. We discuss a range of modeling
techniques including mathematical, statistical and decision-analytic models
along with their application in the COVID-19 context. With this overview, we
aim to foster the understanding of the goals of these modeling approaches and
the specific data requirements that are essential for the interpretation of
results and a successful interdisciplinary collaboration. Model types less
known to statisticians, such as decision-analytic models, still require
statistical thinking. In particular, functional relationships and input
parameters for these models are often provided by statisticians and
epidemiologists. Our target audience, therefore, is broad. It includes data
scientists – such as statisticians – mathematicians, physicists,
epidemiologists, economists, social and computer scientists, and decision
scientists.

The paper is organized as follows. In Section 2 we give a short overview of
modeling purposes and approaches with a special focus on differences between
disciplines. In Section 3, we discuss requirements of data quality and why
this is fundamental for the entire process. We then move on to modeling, with
Section 4 dealing with the different purposes of modeling. In Section 5, we
explain how decisions can be informed based on these models. We discuss
aspects of the reporting and communication of results in Section 6 , provide
recommendations in Section 7 and a discussion in Section 8.

##  2 Overview of modelling approaches and their purposes

As a statistician, the process of gaining knowledge starts with a research
question and continues with the acquisition of data, which then enters into
the statistical model – see Figure 1 in Friedrich et al. (2021) for an
illustration. Data acquisition here might refer either to the design of an
adequate experiment or observation, or to the use of so-called secondary data,
which has been collected for a different purpose. Statistical principles of
design are relevant, even when using secondary data (Rubin, 2008). In Bayesian
statistics, other information can be incorporated as _a priori information_
(e.g. O’Hagan, 2004). This might stem from previous studies or might be based
on expert opinions. The prior information combined with the data (likelihood)
then results in a posterior distribution. In modeling contexts outside
statistics, data and/or information is used in different ways. Simulation
models use prior information (again based either on data or on other sources
such as expert opinion or beliefs) to determine the parameters of interest and
are usually validated on a data set. The formal representation of the
mathematical or decision-analytic model makes assumptions about the system
that generates the data (Roberts et al., 2012), and the (mis)match between
data and model then provides insights that can be used as basis for decisions.
In contrast to statistical models, the order of data and modeling is thus
reversed. An illustration is depicted in Figure 1. For the purpose of
illustration, Figure 1 depicts the process as a sequence of steps. As the
pandemic progresses, however, some steps such as data capturing and modeling
might be iterated.

From a mathematical perspective, a statistic $f$ is a quantity or function
defined on the sampling space. Note that the term statistic is used both for
the function $f$ as well as for the value $f({\bf x})$ of the function on a
given data set ${\bf x}$ (DeGroot and Schervish, 2014; Licker, 2003). Choices
of $f$ include very simple preprocessing steps, e.g., taking the mean where
$f(x_{1},\dots,x_{n})=\frac{1}{n}\sum_{i=1}^{n}x_{i}$, as well as estimates
obtained from complex statistical models, such as hierarchical Bayesian models
used for nowcasting (Günther et al., 2021) or sophisticated regression models
for prediction (Iwendi et al., 2020). Thus, the input to a statistical model
might either be ‘raw data’ or might have undergone some previous steps, like
scaling or transformation applied to variables before entering them in a
regression model. These previous steps are often referred to as preprocessing
(especially in the context of machine learning) and examples include
descriptive statistics and exploratory data analysis, see also Friedrich et
al. (2021) and the references cited therein. It should be noted, however, that
data can often not be analyzed directly, but is the product of a complex
sequence of processing steps. In particular, Desrosières (2010) separates
three aspects of statistics, namely “(1) that of quantification properly
speaking, the making of numbers, (2) that of the uses of numbers as variables,
and finally, (3) the prospective inscription of variables in more complex
constructions, models”. Throughout this paper, we refer to the model input
parameters as ‘data’ irrespective of whether some preprocessing took place or
not. Official statistics, for instance, usually refer to crude observations
when talking about ‘data’, while the information obtained by (pre-)processing
this data is called statistics. For this paper, however, we adapt a slightly
different view on the same aspect and do not distinguish between crude and
preprocessed data. In this sense, the measures of quality and trustworthiness
discussed in Section 3 similarly extend to preprocessing, descriptive
statistics and exploratory data analysis (Friedrich et al., 2021).

Figure 1: For evidence-based decision making, a complex process is often
necessary. In both statistical and mathematical/decision-analytic modeling,
the path starts with a research question and ends in guidance for decision
making. In statistics (upper path) using data as the basis for modeling is
common. In mathematical and decision-analytic modeling (lower path), models
are based on subject matter knowledge and validated or calibrated using data
sets. For the purpose of illustration, the process is depicted as a sequence
of steps here. In reality, however, these are more likely to be cyclic,
iterating steps such as data capturing and modeling and informing new
questions based on previous results.

As mentioned above, modeling is an integral part of evidence-based decision
making. Here, we distinguish three purposes of modeling, which are summarized
in Table 1. The first category contains models which aim to explain patterns
and trends in the data. The second category aims to predict the present (so-
called now-casting) or the future (forecasting). Finally, decision-analytic
models aim to inform decision makers by simulating the consequences of
interventions and their related tradeoffs (e.g. benefit-harm tradeoff, cost-
effectiveness tradeoff).

Table 1: Overview of modeling purposes and approaches. For each different
group of models (columns) we describe goals, approaches and challenges.

Purpose: |  Explanation |  Prediction |  Decision Analysis
---|---|---|---
Goal |  Explain patterns, trends and interactions. |  Predict present (e.g. $R$ value) or future (e.g. ICU bed capacity) |  Evaluate predefined alternative interventions, actions or technologies (e.g. vaccination strategies)
Focus |  Dynamic patterns |  Absolute numbers |  Ranking of size and direction of effect, quantifying tradeoffs
Statistical approaches (Examples) |  Factor analysis, cluster analysis, regression, contingency analyses, propensity score methods, g-computation, marginal structural models |  Time series analysis, repeated measure analysis, statistical and machine learning. |  Statistical decision theory
Simulation models (Examples) |  ABM∗, SD∗, differential equations, MSM∗ |  Differential equations, ABM∗, MSM∗ |  Differential equations, ABM∗, MSM∗, SD∗, DES∗, State-Transition Models, Decision Trees
Importance of correctly assessing quantitative uncertainty |  Minor role (+) |  Major role (+++) |  Intermediate (++)
Importance of correctly assessing qualitative uncertainty |  Intermediate (++) |  Minor role (+) |  Major role (+++)
Weaknesses/ Challenges |  High risk of overinterpretation w.r.t. causality |  Highly sensitive to context |  Highly dependent on comprehensive choice of options and outcomes

ABM Agent-based model, DES discrete event simulation, SD system dynamics, ICU
intensive-care unit, MSM Microsimulation Modeling

In medical decision making and health economics, this leads to a formal
decision framework, which relates to statistical decision theory (Siebert,
2003, 2005). In this framework, (Parmigiani and Inoue, 2009) the decision
maker has to choose among a set of different actions. The consequences of
these actions depend on an unknown “state of the world”. The basis for
decision making depends on the quantitative assessment of these uncertain
consequences. To this end, a loss or utility function must be defined which
allows the quantification of benefits, risk, cost or other consequences of
different actions. Minimizing the loss function (or maximizing the utility
function) then leads to optimal decisions.

##  3 Data availability and quality

An essential basis for research and for evidence-based policy is high quality
data. The mere presence of data is not enough, as the process of data
definition, collection, and processing determines the quality of the data in
reflecting the phenomena on which to provide evidence. Poor definition of data
concepts and variables as well as bad choices in their collection and
processing can lead to misleading data, that is data with severe bias, or to
an unacceptably large remaining level of uncertainty about the phenomena of
interest, so that any results generated with that data form an inadequate
basis for decision making. Below, we describe which quality characteristics
have to be considered when planning a data collection or when assessing the
quality of already existing data for a task at hand. The underlying concepts
are general and well-known. Despite this, they are regretfully often
neglected, and thus we summarize them as eight characteristics. in the context
of policy making.

  1. 1.

Suitability for a target: Data in itself is neither good nor bad, but only
more or less suitable for achieving a certain goal. In order to assess data,
it is first necessary to understand, agree on, and describe the goal that the
data are supposed to support.

  2. 2.

Relevance: Data must provide relevant information to achieve the goal. To do
this, the data must measure the characteristics needed (e.g. how to measure
population immunity?) on the right individuals (e.g. representative sample for
generalization, or high-resolution data for local action?).

  3. 3.

Transparency: The data collection process must be transparent in terms of
origin, time of data collection and nature of the data. Transparency is a
requirement for peer-review processes to ensure correctness of results, and
for an adequate modeling of uncertainties.

  4. 4.

Quality standards: Data are well suited for policy making requiring general
overviews and spatio-temporal trends if local data collection follows a clear
and uniform definition of what is recorded and how it has been recorded.
Standardization includes, for example, the harmonization of data processes,
adequate training of the persons involved in the collection, and monitoring of
the processes.

  5. 5.

Trustworthiness: To place trust in the data, these must be collected and
processed independently, impartially and objectively. In particular, conflicts
of interest should be avoided in order not to jeopardize their credibility.

  6. 6.

Sources of error: Most data contain errors, such as measurement errors, input
errors, transmission errors or errors that occur due to non-response. With a
good description of data collection and data processing (see ‘Transparency’
above), possible sources of error can be assessed and incorporated into the
modeling for the quantification of uncertainty and the interpretation of
results.

  7. 7.

Timeliness and accuracy: Ideally, data used for policy-making should meet all
quality criteria. However, information derived from data must additionally be
up-to-date and some decisions (e.g. contact restrictions) cannot be postponed
to wait until standardized processes have been defined and implemented and
optimal data has been collected. The greater uncertainty in the data
associated with this must be met with transparency and with great care in its
interpretation.

  8. 8.

Access to data for science: In order to achieve the overall goal of evidence-
based policy making, it is important to make good data available as a resource
to a wide scientific public. This allows for the data to be analyzed in
different contexts and with different methods, and enables the data to be
interpreted from the perspective of different social groups and scientific
disciplines.

These eight aspects are included in the European Statistics Code of Practice
(European Statistics Code of Practice, 2017). This Code of Practice, however,
goes beyond the above mentioned points by covering further aspects of
statistical processes and statistical outputs as well as an additional section
on the institutional environment. The aim is to provide a common quality
framework of the European Statistical System.

The items presented above mainly refer to a primary data generating process,
that is, when the data are directly generated to provide information on a pre-
defined target. Especially in the context of COVID-19, information from
_available sources_ often has to be considered, where the data generation does
not necessarily coincide with the aim of the study. One prominent example is
the number of infections, which are gathered by the local health authorities,
but are used for comparing regional incidences which are the basis of several
policy decisions. Particular attention in this case has to be paid to
selection bias. One way to assess (and thus address) selection bias would come
from accompanying information on asymptomatically infected persons gained
through representative studies. Other information to mitigate selection bias
comes from the number of tests and the reasons for testing, but these are not
appropriately reported in Germany. Both problems yield biased regional
incidences. Hence, modeling based on these data may cause misleading results
and has to be considered carefully. Additionally, the data generating process
may be subject to informative sampling (Pfeffermann and Sverchkov, 2009).

The above aspects always have to be seen in light of the research question.
Incidences and infection patterns need highly different data. Available data
are often inappropriate, or must be accompanied by additional data sources.
Due to the highly volatile character of COVID-19 infections, data gathering –
especially via additional samples – must be very carefully planned to foster
the necessary quality to provide the foundation for policy actions (Rendtel et
al., 2021).

Representativity implies drawing adequate conclusions from the sample on the
population or parameters of the population. To achieve this, known inclusion
probabilities on a complete list of elements must be given in order to allow
statistical inference. Nowadays, the term representativity is generalized to
cover regional smaller granularity, as well as in accordance with the time
scale. Further details, especially for subgroup representativity, can be drawn
from Gabler and Quatember (2013). In household or business surveys, the term
representativity has to be seen in the context of non-response and its
compensation (Schnell, 2019). In practice, the term representativity is often
recognized as a sufficiently high quality sample. This is entirely misleading.
Indeed, statistical properties, and especially accuracy, have to be
additionally considered (Münnich, 2020). Finally, it has to be pointed out
that these aspects have to be separately considered for each variable or
target of interest.

##  4 From data to insights – the purposes of modeling

Data and decisions are often linked using statistical models or simulations.
As noted in Section 2, we refer to data as the input to statistical models
irrespective of any preprocessing steps. As such, data preparation,
descriptive statistics and preprocessing are not the focus of this paper and
are thus not discussed in detail. Nonetheless, they are an essential step in
any statistical analysis and especially in a situation such as the COVID-19
pandemic, where data are, in particular in the early stages of a pandemic,
sparse and often disorganized. Exploratory data analysis is an important step
to check data quality and discover potential anomalies in the data. An
important aspect, however, is to keep in mind that data is a product of a
complex sequence of steps. Here, transparency concerning the origin of the
data and the whole process of data preparation is important and should be
included as meta-data. As outlined in the Section 2, modeling can serve three
purposes. Each of them can be approached from either a statistical or a
mathematical modeling perspective. In these models, data can play different
roles: while statistical models use data as the basis for the model itself,
simulations are based on parameters according to prior information and
predictions based on the simulation can be checked against real data to assess
the precision and validity of the constructed model.

An important aspect is handling and communicating uncertainty. In statistical
models, different types of uncertainty occur: sampling variation, model
uncertainty, incomplete data, applicability of information and confounding are
common examples (e.g. Altman and Bland, 2014; Abadie et al., 2020; Chatfield,
1995). In mathematical and decision-analytic models, there are usually
alternative approaches to determine the values for key parameters used in
simulations. For decision-making purposes, therefore, it is important to
compare different methods for determining the indicators. Consequently, in
most cases, not a single number but an interval or distribution has to be
considered. In the following, we will consider the three purposes of modeling
in more detail. For each of them, we provide some mathematical background,
explain the difference between statistical models and mathematical or
decision-analytic models and give some examples of how these approaches were
applied in the COVID-19 pandemic.

###  4.1 Modeling for explanation

The main goal of these models is to explain patterns, trends or interactions.
Statistical models for this purpose include, for example, regression models
(Fahrmeir et al., 2007) as well as factor, cluster or contingency analyses
(e.g. Fabrigar and Wegener, 2011; Duran and Odell, 2013). In this context,
associations are often misinterpreted as causal relationships. The discovery
of correlations and associations, however, cannot be equated to establishing
causal claims. In statistics and clinical epidemiology, for example, the
Bradford Hill criteria (Hill, 1965) can be used to define a causal effect.

From a statistical perspective, there are two possibilities to tackle this
issue. The gold standard is to design a randomized experiment, which enables
causal conclusions. In the context of the SARS-CoV-2 pandemic, randomized
controlled trials were used for assessment of COVID-19 treatments including
the RECOVERY platform trial leading to publications such as Horby et al.
(2020); Abani et al. (2021); RECOVERY Collaborative Group (2020). In the
development of vaccines, too, randomized controlled trials (RCT) played a
vital role (e.g. Baden et al., 2021; Shinde et al., 2021). However, randomized
experiments are not always feasible due to ethical considerations, cost
constraints, and other reasons. Moreover, RCTs have been criticised for a
number of problems including their lack of external validity (Rothwell, 2005)
or the Hawthorne effect (Mayo, 2004).

Where randomized experiments are not possible and observational data is used
instead, causal conclusions are harder to draw. In order to get valid
estimates in this situation, a common approach is the counterfactual framework
by Rubin (1974). For simplicity, assume that we are interested in the effect
of a binary “treatment” $A\in\\{0,1\\}$ (this could be an “ immediate
lockdown” vs. “no immediate lockdown”, for example) on some outcome $Y$ (e.g.
number of infections with COVID-19). Then we denote $Y^{a=1}$ as the outcome
that would have been observed under treatment $a=1$ and $Y^{a=0}$ the outcome
that would have been observed under no treatment ($a=0$). A causal effect of
$A$ on $Y$ is now present, if $Y^{a=1}\neq Y^{a=0}$ for an individual. In
practice, however, only one outcome can be observed for each individual. Thus,
it is only possible to estimate an average causal effect, i.e.
$E(Y^{a=1})–E(Y^{a=0})$ (Hernán and Robins, 2020). Different possibilities for
estimating a causal effect have been proposed, for example, propensity score
methods (Cochran and Rubin, 1973), the parametric g-formula (Robins et al.,
2004), marginal structural models (Robins et al., 2000), structural nested
models (Robins, 1998) and graphical models (Didelez, 2007). Recent works have
shown that these methods have difficulties when it comes to small sample
studies as in the context of COVID-19 (Friedrich and Friede, 2020). Note that
the methods explained here can also be applied to more complex situations such
as non-binary treatments. In the pandemic, for example, it might be relevant
to compare different time points for starting the lockdown, i.e., to include a
time dimension in the considerations above. Furthermore, the methods can also
be extended to more complicated outcome variables, e.g. time-to-event data.

Mathematical models and simulations can also be used to understand and explain
dynamic patterns. Examples are simulation studies for public health
interventions such as lockdown and exit strategies, where general consequences
of different measures can be compared. Seemingly simple simulation models have
played an important role in communicating the dynamics during a pandemic. In
these models, assumptions about the system that generates the data and
(causal) relationships are made. The (mis)match between data and model then
provides insights that can be used as basis for decisions. However, this
procedure does not establish causal relationships in the statistical sense
described above.

The main challenge in modeling for explanation is good communication,
irrespective of whether the model is based on statistical or mathematical
approaches. Therefore, we consider standards for good communication in detail
in Section 6. Anticipating the human bias for interpreting results causally,
clear statements need to be made to which extent (from “not at all” to
“plausible”) specific detected associations allow some causal interpretation
and why. The two extreme interpretations – on the one hand, the simple
disclaimer that “correlation is not causation”, on the other, blanket and
unqualified causal interpretations – do a disservice to the complexity of the
problem as outlined by the methods above.

###  4.2 Modeling for prediction

In statistical prediction models, the modeler can choose from large toolboxes
in (spatio)-time-series analysis as well as statistics and machine learning
(ML). Examples cover simple but interpretable ARIMA models (Benvenuto et al.,
2020; Roy et al., 2021), support vector machines (Rustam et al., 2020), joint
hierarchical Bayes approaches (Flaxman et al., 2020) or state-of-the art ML
methods, such as long short-term memory (LSTM) or extreme gradient boosting
(XGBoost) (Luo et al., 2021). A comprehensive overview is also given by
Kristjanpoller et al. (2021).

For predictions based on such models, one can distinguish different aims: now-
casting and forecasting. For _now-casting_ , information up to the current
date and state are used to estimate or predict key figures, like the $R$
value, for example, which estimates during a pandemic how many people an
infected person infects on average. In _forecasting_ , spatio-temporal
predictions or simulations are used to look ahead in time, as in a weather
forecast, or to estimate the required number of ICU beds. An important aspect
in this situation is that the behavior of people influences the process that
is being modeled. To be concrete, policy decisions are based on the
predictions of a statistical or mathematical model and by introducing certain
counter-measures, the original predictions of the model never come true. Thus,
these models are not prediction models in a classical sense but more like
projections, i.e. scenarios of what would happen if no intervention was taken.
A thorough discussion of this topic can be found in Hellewell (2021). In
forecasting models, a causal relationship between the predictors and the
outcome may be required, while now-casting can also be achieved with
predictive variables that do not necessarily have a causal effect on the
outcome. Several statistical models have been proposed for now-casting, for
example hierarchical Bayesian models (Günther et al., 2021) or trend
regression models (Küchenhoff et al., 2021). Related approaches are discussed
in Altmejd et al. (2020); Schneble et al. (2021); Salas (2021).

#### Dynamic Models/Time-variant Dynamics

A unique feature of pandemic assessment is the dynamic nature of the event. By
this we not only refer to the explosive (exponential) growth that may occur
but the fact that the properties of the processes that describe spatio-
temporal changes are a function of time themselves. This is due to the fact
that the behavior of the people continuously changes the properties of the
system that we are trying to understand and make predictions for. This
contrasts with other natural systems, like the current weather, and most
systems in the engineering and physical sciences.

Simple infectious disease compartmental models can be described by the stock
of susceptible $S$, infected $I$, and removed population $R$ (either by death
or recovery), the contact rate $\kappa$, the infection probability $\beta$,
the recovery rate $\gamma$, the death rate $\mu$ and the birth rate $\Lambda$
(Kermack and McKendrick, 1927; Hethcote, 2000; Andersson and Britton, 2012;
Grassly and Fraser, 2008). Here,

$\displaystyle\frac{dS}{dt}$ $\displaystyle=\Lambda-\mu S-\frac{\kappa\beta
IS}{N}$ $\displaystyle\frac{dI}{dt}$ $\displaystyle=\frac{\kappa\beta
IS}{N}-\gamma I-\mu I$ $\displaystyle\frac{dR}{dt}$ $\displaystyle=\gamma
I-\mu R,$

where $t$ denotes the time point and $N$ is the total number of individuals in
the population, i.e. $N=I+S+R$.

Assuming that the susceptible individual first goes through a latent period
after infection before becoming infectious $E$, adapted models such as SEI,
SEIR or SEIRS, depending on whether the acquired immunity is permanent or not
can be applied (Jit and Brisson, 2011). Modeling the COVID-19 pandemic,
applications include further approaches such as SIR-X accounting for the
removal (quarantine) of symptomatic infected individuals and various other
extensions and applications (Dehning et al., 2020; Dings et al., 2021)
including prediction of the impact of vaccination (Bubar et al., 2021).

In this deterministic compartment model, predictions are determined entirely
by their initial conditions, the set of underlying equations, and the input
parameter values. Deterministic compartmental models have the advantage of
being conceptually simple and easy to implement, but they lack for example
stochasticity inherent in infectious disease transmission. In stochastic
compartment models, the occurrence of events like transmission of infection or
recovery is determined by probability distributions. Therefore, the chain of
events (like an outbreak) is not exactly predictable. However, there are many
possible types of stochastic epidemic models (Britton, 2010; Kretzschmar et
al., 2020).

#### Agent-based models (ABM)

Agent-based modeling as an alternative approach uses individual-level
simulation (Karnon et al., 2012). ABMs have been used to model biological
processes, ecological systems, traffic management, customer flow management or
stock markets, and in recent years increasingly for decision analysis as
discussed later (Marshall et al., 2015a; Bonabeau, 2002; Macal and North,
2008). ABMs represent complex systems in which individual ‘agents’ act
autonomously and are capable of interactions (Miksch et al., 2019). These
agents can represent the heterogeneity of individuals, and the behavior of
individuals can be described by simple rules. Such rules include how agents
interact, move between geographical zones, form households or consume
resources (Chhatwal and He, 2015; Bruch and Atwell, 2015; Hunter et al.,
2017). ABMs are often applied to study “emergent behavior” as a result of
these predefined rules. In infectious disease modeling, agent behaviors
combined with transmission patterns and disease progression lead to
population-wide dynamics, such as disease outbreaks (Macal and North, 2010).
In agent-based models, either all affected individuals are simulated
individually, or specific networks of individuals are integrated into the
simulation.

#### Discrete Event Simulation (DES)

Discrete event simulation is an individual-level simulation (Pidd, 2004;
Karnon et al., 2012; Jun et al., 1999; Zhang, 2018). The core concepts of DES
are entities (e.g. patients), attributes (e.g. patient characteristics),
events, resources (i.e. physical resources such as medical staff and medical
equipment), queues and time (Pidd, 2004; Banks et al., 2005; Jahn et al.,
2010b). In addition to health outcomes, performance measures such as resource
use or waiting times can be calculated, as physical resources (e.g. hospital
beds) can be explicitly modeled (Jahn et al., 2010b). The term discrete refers
to the fact that DES moves forward in time at discrete intervals (i.e. the
model jumps from the time of one event to the time of the next) and that
events are discrete (mutually exclusive) (Karnon et al., 2012).

#### Microsimulation

Microsimulation methods, introduced by Orcutt (1957), are used to simulate
policy actions on real populations. Li and O’Donoghue (2013) describe
microsimulations as “a tool to generate synthetic micro-unit based data, which
can then be used to answer many “what-if” questions that, otherwise, cannot be
answered”. The main difficulty for microsimulation is considered to be the
choice of an appropriate data source on which these simulations can be
conducted. Often, survey data are used. Nowadays, the first step in
microsimulation is the realistic generation of data in the necessary
geographic depth (e.g. Li and O’Donoghue, 2013). A full-population approach is
described in Münnich et al. (2021). Thereafter, the scenario-based
microsimulation analysis yields the necessary information for building
conclusions for policy support. In microsimulation methods, we distinguish
between static and dynamic models. The latter can be divided into time-
continuous and time-discrete models. An overview of microsimulation methods is
given in Li and O’Donoghue (2013) and the references therein. For modeling
COVID-19, dynamic models have to be considered. Bock et al. (2020) presents a
continuous time SIR microsimulation approach as an example for a dynamic
transmission model. In contrast to ABM, other microsimulations are often based
on survey data, or on realistic but synthetically extended survey data. The
above-mentioned cohort simulations are usually deterministic simulations, in
which an initial cohort of interest is followed over different paths over
time, and thus leading to a distribution of outcomes after the analytic-time
horizon. Recently, the dividing line between these methods and the related
terminology has become blurred. Which method is ultimately used often depends
on the background of the research team.

A key objective of forecasting in this context is to obtain numerically
precise predictions, for variables such as the number of ICU beds. With this
goal in mind, the reliability or accuracy of the predictions highly depends on
the availability and quality of the data used to estimate the values of the
parameters in the underlying mathematical or statistical models.

It should be noted that these models are highly sensitive to context in the
following sense: changes in the underlying system in variables that are not
part of the model can lead to changes in the relationship between the selected
predictors and the predictions, rendering the predictions and their assumed
uncertainty meaningless.

While it is widely appreciated that, for example, weather forecasts are only
reliable for a couple of days, forecasting during a pandemic is even more
complicated since the behavior of people influences the process that is being
modeled. Forecasting during pandemics is, therefore, itself a continuous
process with time-varying parameters. For this reason, such modeling effort is
a complex undertaking requiring a range of data and expertise. Such activities
should therefore be realized and coordinated through cross-disciplinary teams.
To account for regional differences, one would expect a collective of modeling
groups that support decision making for different parts of a country.

###  4.3 Decision-analytic modeling

Depending on the research question, different modeling approaches are used for
decision-analytic modeling and development of computer simulations (IQWiG,
2020; Roberts et al., 2012; STIKO, 2016). These include decision tree models,
state-transition models, discrete event simulation models, agent-based models,
and dynamic transmission models. Some of them have been introduced already
since they are also commonly used for prediction. Models introduced in this
section are predominantly used for decision analysis but could potentially be
used for other purposes as well (Table 1).

The selection of the model type depends on the decision problem and the
disease. In general, decision trees are applied for simple problems, without
time-dependent parameters and with a fixed and comparatively short time
horizon. If the decision problem requires the evaluation over a longer time
period and if parameters are time or age dependent, a state-transition cohort
(Markov) models (STM) could be applied. STMs allow for the modeling of
different health states and transitions between these states, and thus also
for repeated events. They are applied when time to event is important. If the
decision problem can be represented in an STM “with a manageable number of
health states that incorporate all characteristics relevant to the decision
problem, including the relevant history, a cohort simulation should be chosen
because of its transparency, efficiency, ease of debugging, and ability to
conduct specific value of information analyses.” (Siebert et al., 2012). If
the representation of the decision problem would lead to an unmanageable
number of states, then an individual-level state-transition model is
recommended (Siebert et al., 2012). Especially in situations where
interactions of individuals among each other or the health-care system need to
be considered, that is, when we are confronted with scarce physical resources,
queuing problems and waiting lines (e.g., limited testing capacities),
discrete event simulation (DES) would be an appropriate modeling technique.
DES allows the modeler to incorporate time-to-event data (e.g., time to
progression) and physical resources are explicitly defined (Karnon et al.,
2012). Modeling types such as differential equation systems, agent-based
models and system dynamics account for the specific features of infectious
diseases such as the transmissibility from infected to susceptible individuals
and the uncertainties arising from complex natural history and epidemiology
(Pitman et al., 2012; Grassly and Fraser, 2008; Jit and Brisson, 2011).

#### Decision tree models

In a decision-tree model, the consequences of alternative interventions or
health technologies are described by possible paths. Decision trees start with
decision nodes, followed by alternative choices (interventions, technologies,
etc.) of the decision maker. For each alternative, the patients’ paths, which
are determined by chance and that are outside the decision maker’s control,
are then described by chance nodes. At the end of the paths, the respective
consequences of each path are shown. Consequences or outcomes may include
symptoms, survival, quality of life, number of deaths or costs. Finally, the
expected outcomes of each alternative choice are calculated by taking a
weighted average over all pathways (Hunink et al., 2001; Rochau et al., 2015),
such as in the evaluation of COVID-19 testing strategies for university
campuses in a decision-tree analysis (Van Pelt et al., 2021).

#### State-transition models

A state-transition model is conceptualized in terms of a set of (health)
states and transitions between these states. Time is represented in time
intervals. Transition probabilities, time cycle length, state values
(“rewards”) and termination criteria are defined in advance. During the
simulations, individuals can only be in one state in each cycle. Paths of
individuals determined by events during a cycle are modeled with a Markov
cycle tree that uses a set of random nodes. The average number of cycles in
which individuals are in each state can be used in conjunction with the
rewards (e.g. life years, health-related quality of life or costs) to estimate
the consequences in terms of life expectancy, quality-adjusted life
expectancy, and the expected costs of alternative interventions or health
technologies. There are two common types of analyses of state-transition
models: Cohort models (“Markov”) (Beck and Pauker, 1983; Sonnenberg and Beck,
1993) and individual-level models (“first order Monte Carlo” models)
(Spielauer, 2007; Groot Koerkamp et al., 2010; Weinstein, 2006). Simple cohort
models are defined in mathematical literature as discrete-time Markov chains.
A discrete-time Markov chain is a sequence of random variables ${\displaystyle
X_{0},X_{1},X_{2},...}$ representing health states with the Markov property,
namely that the probability of moving to the next health state depends only on
the present state and not on the previous states:

${\displaystyle\Pr(X_{n+1}=x\mid
X_{1}=x_{1},X_{2}=x_{2},\ldots,X_{n}=x_{n})=\Pr(X_{n+1}=x\mid X_{n}=x_{n})}$

Generalized models such as continuous time Markov chains with finite or
infinite state space are not commonly applied in health decision science.
Applications of state-transition models in the pandemic include evaluation of
treatments (Sheinson et al., 2021) and vaccination strategies (Kohli et al.,
2021). We also find hybrid models including the combination of decision trees
and STMs (see Figure 2).

Figure 2: Example: A Cost-Effectiveness Framework for COVID-19 Treatments for
Hospitalized Patients in the United States, (Sheinson et al., 2021)

#### Discrete Event Simulation (DES)

Similar to decision trees and state-transition models, health outcomes and
costs of alternative health technologies can be assessed. In addition to these
outcomes, as mentioned earlier, performance measures can be calculated as
additional information for decision makers and the impact of scarce resources
on costs and health outcomes can be evaluated (Jahn et al., 2010a). The
increased use of DES to support decision making under uncertainty is shown in
the review of Zhang (2018). Model applications in COVID-19 include
optimizations of processes with scarce resources such as bed capacities
(Melman et al., 2021) or testing stations (Saidani et al., 2021) and
laboratory processes (Gralla, 2020).

#### Dynamic Models/Time-variant Dynamics

The dynamic SIR type models explained in Section 4.2 can also be used in the
context of decision-analytic modeling (e.g., vaccination allocation, ECDC,
2020a; Sandmann et al., 2021) . This model type can be extended by further
compartments such as Death (D) and other states (X), reflecting, for example,
quarantine or other states relevant to the research question. Such SIRDX
models have been used frequently to model non-pharmaceutical intervention
effects during the COVID-19 pandemics (Nussbaumer-Streit et al., 2020). As in
Markov state-transition models, a deterministic cohort simulation approach is
used to model the distribution of compartments over time. Deterministic
compartment models are useful for modeling the average behavior of disease
epidemics in larger populations. When stochastic effects (e.g., the extinction
of disease in smaller populations), more complex interactions between disease
and individual behavior or distinctly nonrandom mixing patterns (e.g., the
spread of the disease in different networks) are relevant, stochastic agent-
based approaches can be used (see next section).

#### Agent-based models (ABM)

Agent-based models as introduced earlier, have been used for decision
analysis, for example for cost-effectiveness analyses in health care in the
recent years (Marshall et al., 2015a; Chhatwal and He, 2015). ABMs are also
used in public health studies to model noncommunicable diseases (Nianogo and
Arah, 2015). A comparison of ABM, DES and system dynamics can be found in
Marshall et al. (2015a, b) and Pitman et al. (2012). ABMs are increasingly
applied for COVID-19 evaluations including decision support for vaccination
allocation accounting explicitly for network structure and contact behaviour
(Bicher et al., 2021; Jahn et al., 2021).

#### Microsimulation

Microsimulation as a modeling approach based on survey data and combining
characteristics of above mentioned modeling approaches is used for prediction
and decision analysis in various fields, especially for policy support using
scenarios. Recently, MSM are also used for modelling diseases (Hennessy et
al., 2015) including infectious diseases.

Table 2 provides a short comparative overview of these commonly applied
modeling approaches with example applications for COVID-19 research, in
addition to our general comparison at the beginning of the section. Further
guidance on model selection for a given problem at hand exists (IQWiG, 2020;
Roberts et al., 2012; Siebert et al., 2012; Marshall et al., 2015a).

Table 2: Overview of differences and similarities of simulation models commonly used as the basis for health decision sciences. Decision-Analytic Models |  Decision trees |  State-transition models |  Discrete event simulation |  Dynamic models |  Agent-based models |  Micro- simulation
---|---|---|---|---|---|---
Mathe- matical / statistical background |  Operations research, decision analysis, machine learning |  Markov chains, Monte Carlo methods |  Monte Carlo methods |  Differential equations (deterministic or stochastic) |  Monte Carlo methods |  Monte Carlo methods
Components |  Decision nodes, chance nodes, end nodes, paths |  (Health) states, transition probabilities |  Individuals (Entities), resources, event states |  Compartments, transition rates |  Individuals (agents), attributes, learning rules, environment |  Individuals, attributes, learning rules, environment
Time |  Not relevant |  Time-dependent parameters |  Discrete time intervals |  Continuous |  Continuous |  Continuous
Role and type of data |  Model building/ Parameter estimation/ validation |  Parameter estimation/ validation |  Parameter estimation/ validation |  Parameter estimation/ validation |  Parameter estimation/ validation |  Survey data
Other information |  Expert opinion |  Expert opinion |  Expert opinion |  Expert opinion |  Expert opinion/ beliefs |
Prominent application areas |  Engineering, law, health care |  Health care strategies |  Queues |  Infectious diseases |  Biological processes |  Traffic flow
SARS-CoV-2 / COVID-19 examples |  Testing strategies for university campuses (Van Pelt et al., 2021) |  Evaluation of vaccination strategies (Kohli et al., 2021) |  Balancing scarce resources (Melman et al., 2021) |  Contact tracing strategies (Kretzschmar et al., 2020) |  Vaccination strategies (Jahn et al., 2021) |  Herd immunity (Bock et al., 2020)

##  5 Decision analysis

The models described in Section 4 are built to inform decision making.
Therefore, the so-called decision analysis framework is used. Decision
analysis aims to support decisions under uncertainty by means of systematic,
explicit and quantitative methods. In particular, computer simulations and
prediction models as described above are used to calculate the short-term and
long-term benefits and harms (as well as the costs) of alternative
interventions, technologies, or measures in health care (Schöffski and
Schulenburg, 2011; Richardson and Spiegelhalter, 2021). The decision-analytic
framework includes, among other things, the relevant health states and events
considered to describe possible disease trajectories, the type of analysis
(e.g., benefit-harm, cost-benefit, budget-impact analyses (Drummond et al.,
2005)), and the simulation method (cohort- or individual-based). In addition
to base-case analysis (using the most likely parameters), scenario and
sensitivity analyses (Briggs et al., 2012) should be performed to show the
robustness or uncertainty of the results. Value of information analysis can be
applied to assess the value of future research to reduce uncertainty (Fenwick
et al., 2020; Siebert et al., 2013).

###  5.1 Decision tradeoffs

A central idea in decision analysis is that tradeoffs in outcomes of
alternative choices are formalized and, if possible, quantified. In addition,
the tradeoff between such outcomes is explicitly expressed, usually in the
form of an incremental tradeoff ratio. In the context of a benefit-harm
analysis, for example, this relates to quantifying the benefits of COVID-19
vaccination in terms of (incremental) deaths avoided and the harms of
vaccination in terms of (incremental) potential side effects.Alternatively,
the tradeoff of different school closure strategies in a pandemic (e.g.,
according to incidence level) weighting benefits in terms of (incremental)
deaths or hospitalisations avoided and lost education time should be
considered. In general, two or more interventions can be compared in a
stepwise incremental fashion (Keeney and Raiffa, 1976). Benefit-harm analyses
are often applied in screening evaluations (Mandelblatt et al., 2016;
Sroczynski et al., 2020). To detect efficient strategies, so-called strongly
dominated strategies are first excluded. These are strategies that result in
higher harms (e.g. due to testing or invasive diagnostic work-up) and lower
benefits (e.g., cancer-cases avoided, life-years gained) than other
strategies. Second, weakly dominated strategies are excluded, that is
strategies that result in higher harms per additional benefit compared with
the next most harmful strategy, or in other words, strategies that are
strongly dominated by a linear combination of any two other strategies. Third,
the incremental harm-benefit ratios (IHBRs) are calculated for the non-
dominated strategies.

$\displaystyle IHBR$ $\displaystyle=\frac{\Delta harms}{\Delta benefits}$
$\displaystyle=\frac{harm(strategy_{i})-harm(strategy_{i+1})}{benefit(strategy_{i})-benfit(strategy_{i+1})}$

There is no general benchmark for how much additional harm individuals are
willing to accept per unit of additional benefit. Strategies are explored as a
function of willingness-to-accept thresholds and they are displayed as harm-
benefit acceptability curves on the efficiency frontier (Neumann et al.,
2016).

In this context, the choice of measures that are presented and discussed also
influences decision behavior (Ariely and Jones, 2008). The same applies to
changes in decision-making due to alternatives that are presented. Regarding
optimization of vaccination interventions, temporal aspects of availability
and effectiveness of vaccinations can be considered. Alternative strategies
can be evaluated, like in the comparison of immediate vaccination with lower
vaccine protection, against later vaccination with expected higher
effectiveness but risk of intermediate infection. Further non-pharmaceutical
measures like school-closure strategies can be evaluated depending on
incidence level but also accounting for additional measures to reduce the
spread of the disease.

###  5.2 Statistical decision theory

As a more general framework, statistical decision theory can help to make
decisions on a formal basis. In this framework, the decision maker has to
choose among a set of different actions $a$ by quantitatively assessing the
consequences of these actions. To this end, we consider a loss function
$L(\theta,a)$, where the unknown parameter $\theta$ refers to the “state of
the world”. The interesting question for the statistician is how to use the
data in order to make optimal decisions. Assume we observe an experimental
outcome $x$ with possible values in a set $\mathcal{X}$, which depends on the
unknown parameter $\theta$. Furthermore, let $f(x|\theta)$ be the
corresponding likelihood function. Then, we define a decision function
$\delta(x)$ which turns data into actions (Parmigiani and Inoue, 2009). To
choose between decision functions, we measure their performance by a risk
function

$R(\theta,\delta)=\int_{x}L(\theta,\delta)f(x|\theta)dx.$

These can be approached from either a frequentist perspective (e.g. the
minimax decision rule) or a Bayesian perspective, where the risk is associated
with a prior distribution $\pi(\theta)$. For a more thorough treatment of
these concepts, we refer to Parmigiani and Inoue (2009).

Examples for loss functions in the context of COVID-19 include the number of
avoided deaths (Bubar et al., 2021), negative reward functions in Markovian
decision models (Eftekhari et al., 2020) or the social loss function as
proposed in a recent discussion paper of the European Commission (Buelens et
al., 2021).

##  6 Reporting and Communication

For data analysis and modeling to have an impact as a component of decision
making, appropriate reporting and communication is key. There are numerous
standards and guidelines for study planning and statistical reporting in the
numerous application areas, such as the ESS standard for quality reporting,
the CONSORT, PRISMA, CHEERs guidelines, and others (see https://equator-
network.org). These standards are based on commonly accepted core quality
principles and values such as accuracy, relevance, timeliness, clarity,
coherence, and reproducibility. For measures to restrain and overcome an
epidemic effectively, communication among experts that follows highest
professional and ethical standards is not sufficient. In a democratic society,
policy measures can only be implemented if they are accepted by the wider
population. This puts high demand on skills associated with communicating
statistical evidence on the side of scientists, governments and media, and a
citizenry able to understand statistical messages.

In recent decades, there have been numerous publications, initiatives, and
ideas to improve the communication of quantitative and statistical
information, see Hoffrage et al. (2000); Tufte (2001); Rosling and Zhang
(2011); Otava and Mylona (2020), to name only a few. Data journalism has
recently taken off as an innovative component of news publishing, and COVID-19
provides numerous excellent examples, often using an interactive visual format
on the Internet, such as dashboards. A fundamental problem in assessing
probabilities, for example, lies in the intuitive conflation of subjective
risks (“how likely am I to become infected”) and general risks (“how likely is
it that some person will become infected”). Another issue is that of equating
sensitivity of a diagnostic test and the positive predictive value (Eddy,
1982; Gigerenzer et al., 2007; McDowell et al., 2019; Binder et al., 2020). In
particular, the prevalence (or base rate) is often neglected leading to this
confusion. Fact boxes combined with icon arrays are recommended for the
presentation of test results. Both representations are based on natural
frequencies (Gigerenzer, 2011; Krauss et al., 2020) and present case numbers
as simply and concretely as possible. Many scientific studies show that icon
arrays help people understand numbers and risks more easily (e.g. McDowell et
al., 2019). The Harding Center for Risk Literacy shows many other examples of
transparent communication of risks, including
COVID-19222https://www.hardingcenter.de/de/ and a collection of misleading or
wrong communication of statistics, such as for vaccination effects
(”Unstatistik”).

When communicating the results of an analysis to policy makers or the general
public, the following aspects must also be kept in mind. While human thinking
tends towards pattern simplification and political communication also prefers
a simple cause-effect relationship, real phenomena are often multivariate.
Thus, when studying COVID-19 and predicting its spread, it is important to
consider its symptomatology, the incidence and geographic distribution of
diseases, population behavior patterns, government policies and impacts on the
economy, on schools, on people in nursing homes and on social life as a whole.
However, it is also crucial to integrate these into data analyses and to
communicate results clearly and transparently; for example, it might be
important to state that associations observed in the data could be caused by
other, omitted variables (confounders). In addition, much of the data comes
from observational studies, which usually makes a robust causal attribution
problematic. As many of these phenomena cannot be studied other than by
observation (for ethical and feasibility reasons), causal attribution might be
achieved as a scientific consensus opinion among scientists from the relevant
disciplines that understand the complexity of the models and the subject
matter studied. Visual representations take a central position in public
communication and aim to represent the corresponding dynamics and contents in
a quickly understandable way. Usually either time-dependent parameters or data
with a spatial reference are visualized. For spatially distributed data,
choropleth maps are predominantly used, in which administrative regions
defined by the responsible health authorities are colored according to the
distribution density of the infection figures or variables derived from them
(see Figure 3). Their visual perception problems - such as the visual
dominance of the area of administrative regulatory frameworks that have no
direct relation to infection events - are well known but the effects of such
problems are still widespread. In addition, the use of ordinance thresholds as
the basis for color scaling is often at odds with color schemes that emphasize
real spatial distributional differences.

Figure 3: Choropleth map of the incidence figures on 2021-02-12 for Germany by
district. Source: Robert-Koch-Institute
https://app.23degrees.io/export/oCRP768wQ3mCswE7-choro-corona-faelle-
pro-100-000/image.

For time-dependent parameters, different variants of time series diagrams are
used, predominantly line and column diagrams. The use of logarithmic scales in
time series diagrams should be evaluated with caution (Romano et al., 2020).
On the one hand, they tempt superficial readers to underestimate dynamic
growth processes; on the other hand, they increase the demands on the
mathematical and statistical literacy of the readership without corresponding
advantages of visual representation. Figure 4 shows the time course of the
7-day incidence per 100,000 people between 24 January and 4 February 2021 for
some selected countries. While the differences appear relatively small on the
logarithmic scale, the linear scale shows considerable differences.

Figure 4: The 7-day incidence for different countries over time. On the
logarithmic scale ( top graphic), differences appear small. The linear scale
(bottom graphic), however, shows considerable differences. Source: Our World
in Data, https://ourworldindata.org/covid-cases?country=IND USA GBR CAN DEU
FRA, accessed 2022-02-22.

##  7 Recommendations

Reaching a decision based on data requires several steps, which we have
illustrated in this paper: Data provide the basis for different kinds of
models, which can be used for prediction, explanation and decision making.
This forms the basis for making decisions within a formal framework. The
results of these models must be communicated to non-scientists in order to
gain acceptance of and adherence to policy decisions. Each of these steps
comes with its own caveats and requires sound statistical knowledge.

Data: Lessons learned from the current pandemic about data, variables and
information that should be obtained are critically discussed for specific
countries (Grossmann et al., 2022; The Royal Statistical Society, 2021;
Rendtel et al., 2021) and on a European and international level (Kucharski et
al., 2021; ECDC, 2020b), (Dean, 2022; Mathieu, 2022). Examples include
establishing new vaccination registries, extending coronavirus registries with
further socio-demographic parameters and data sharing. We recommend the
implementation of standards (European Statistics Code of Practice, 2017) and
processes for data collection on a national and international level,
especially within Europe. These standards need to be refined, meeting the
requirements of relevance, transparency, truthfulness, timeliness and accuracy
to improve the handling of epidemics and pandemics in the coming years.
National and international strategies and systematic collection and sharing of
data allowing researchers to access the information is important to build
comparable statistical and decision-analytic models.

Modeling Methods: Depending on the modeling approach, the model can be fit to
one data set (e.g., regression model) or data from different sources with
different levels of evidence can be used to populate the model (e.g.,
decision-analytic models). In addition, several modeling approaches may be
applicable for one research question (e.g., differential equation model or ABM
for the prognosis of COVID-19 spread and consequences). We recommend clear
communication about 1) the purpose of the model, 2) how the model uses data,
3) the database or additional assumptions and their evidence basis, and 4)
risks and uncertainties. If applicable, several modeling approaches should be
applied. As a result, the model best fitting the data would be selected or
different modeling approaches could provide insights into uncertainty, like in
national forecasting consortia (e.g., as in the Austrian COVID Prognosis
Consortium (CPK, 2021)) and nowcast/forecast ensembles. The infrastructure of
comprehensive population or microsimulation models – including population,
disease and flexible intervention or policy modules – needs to be established
and maintained beyond the current crisis.

Data Aquisition: Models require data from various sources and different
modeling approaches allow data transformation and synthesis from different
sources. Data aquisition for scientific evaluations requires a further
improved infrastructure to speed up model development and to parameterize
models with high level evidence including vaccine effectiveness in real time.
We recommend the creation of a central national DataLab, collecting data in a
unified way and linking data from different sources as well as enabling
accessibility for experienced users. With regard to data sharing, it is
important that this is manageable from a practical point of view in terms of
the time frame and resources needed.

Transparency: In the wake of the COVID-19 pandemic, reporting of infection
numbers and derived epidemiological indicators boomed, demonstrating with
dramatic clarity the knowledge gap between experts, policymakers, and the
public. To increase the acceptance of decisions and associated measures, all
steps in the decision-making process must be disclosed. We recommend a
transparent decision-making process and communication of this process starting
with the data, continuing with the choice of models, relevant perspectives,
outcomes or metrics for several outcomes and an explicit discussion of
considered tradeoffs (see, e.g., Gigerenzer et al., 2007; Gigerenzer and
Edwards, 2003; Richardson and Spiegelhalter, 2021). In this context, the media
also play a crucial role.

Interdisciplinary cooperation: A pandemic poses particular challenges to
society as a whole. In order to tackle these as efficiently as possible,
interdisciplinary cooperation, such as that fostered by the DAGStat, is
essential. We recommend that experts act as a specialist group rather than as
individuals, broadly positioned and media-sensitive. These interdisciplinary
collaborations should consist of data scientists including statisticians,
epidemiologists, experts in public health, social sciences, and ethics, as
well as decision and communication scientists. The DAGStat as an umbrella
organization of various professional societies, the Competence Network Public
Health COVID-19 or the Society for Medical Decision Making (SMDM) are examples
for existing networks that can be built upon.

Statistical Literacy / Data Literacy: The COVID-19 crisis brought into the
general public’s awareness that our social interaction and political decisions
are essentially based on data, modeling, the weighing of risks and benefits,
and thus on probability estimates, expected values and incremental harm-
benefit ratios. Clearly, we need additional efforts to promote statistical or
data literacy at all levels of society. The ability to critically evaluate and
interpret data and to critically reflect on model outcomes serves to promote
maturity in a modern digitized world. We recommend promotion of statistical
literacy to be intensified at all levels of education (school/vocational
education/training) following the Data Literacy Charta (Schüller et al., 2021)
and including risk competency (Ball et al., 2020; Loss et al., 2021).
Therefore, collaboration between statisticians and all stakeholders involved
in statistical literacy is necessary.

##  8  Discussion

In our paper, we discussed all steps starting from data capturing to
statistics, modeling, decision making and communication which are important
aspects in the context of evidence-based decision making. The current pandemic
has shown that, in particular in Germany, we are still far from such an
evidence-based decision-making process. Aims of this process include the
following: First, it should result in the best possible decision given the
available evidence and it is necessary to explicitly consider the tradeoffs
involved with certain interventions. Second, gaining the public’s acceptance
of the decisions is fundamental. In order to achieve these goals, we need
reliable data, careful interpretation of the results and a clear
communication, especially concerning uncertainty, see, for instance, WHO
(2020).

It is important to note that the considerations described in our paper not
only apply to the current pandemic, but also extend to future
pandemics333https://www.statnews.com/2021/05/18/luck-is-not-a-strategy-the-
world-needs-to-start-preparing-now-for-the-next-pandemic/ and other (public or
political) challenges such as the climate change debate (Ritchie, 2021).

Our paper has several limitations. First, we refer to data as the input to the
statistical models irrespective of possible preprocessing steps. An upcoming
publication on data and data infrastructure in Germany will include more
details on data preparation, building upon the DAGStat white paper (Deutsche
Arbeitsgemeinschaft Statistik, 2021). Second, our overview of modeling
techniques does not provide a detailed discussion of all modeling approaches
and their advantages and limitations but it should foster interdisciplinary
collaboration among data scientists. References on guidance papers provide
valuable further readings. Third, the decision-making process involves a
variety of stakeholders, including politicians, government agencies and health
authorities, health care providers, citizens, patients and their relatives,
scientists, and they all take different perspectives. We have not discussed
this aspect in detail in our paper, but the original white paper included a
paragraph on political decision making (Deutsche Arbeitsgemeinschaft
Statistik, 2021).

The German Consortium in Statistics (DAGStat), a network of 13 statistical
associations and the German Federal Office of
Statistics444https://www.dagstat.de/en/about-us/cooperating-societies and the
Society for Medical Decision Making (SMDM), initiated a collaboration of
scientists with backgrounds in all areas of statistics as well as
epidemiology, decision analysis and political sciences to critically discuss
the role of data and statistics as a basis for decision-making motivated by
the COVID-19 pandemic. We found that similar concepts are often considered in
different areas, but different notation and wording can hinder
transferability. In this sense, this paper also aims to bridge the gaps
between disciplines, and to broaden the research focus of statistical
disciplines to prepare for future pandemics.

###### Acknowledgements.

We thank Lyndon James (Harvard T.H. Chan School of Public Health) for
proofreading and language editing.

## Funding

Tim Friede and Markus Pauly are grateful for support by the Volkswagen
Foundation (“Bayesian and Nonparametric Statistics - Teaming up two opposing
theories for the benefit of prognostic studies in Covid-19”). Research by
Beate Jahn and Uwe Siebert was also funded in part by the Austrian Federal
Ministry for Digital and Economic Affairs BMDW and handled by the Austrian
Research Promotion Agency (FFG) within the Emergency Call for research into
COVID-19 in response to the SARS-CoV-2 outbreak (CIDS—Concurrent Infectious
Disease Simulation) (881665) and by the Gordon and Betty Moore Foundation
through Grant (GBMF9634) to Johns Hopkins University to support the work of
the Society for Medical Decision Making COVID-19 Decision Modeling Initiative.

## Conflict of interest

The authors declare that they have no conflict of interest.

## Availability of data and material

Not applicable.

## Code availability

Not applicable.

## Authors’ contributions

## References

  * Abadie et al. (2020) Abadie A, Athey S, Imbens GW, Wooldridge JM (2020) Sampling-Based versus Design-Based Uncertainty in Regression Analysis. Econometrica 88(1):265–296
  * Abani et al. (2021) Abani O, Abbas A, Abbas F, Abbas M, Abbasi S, Abbass H, Abbott A, Abdallah N, Abdelaziz A, Abdelfattah M, et al. (2021) Convalescent plasma in patients admitted to hospital with COVID-19 (RECOVERY): a randomised controlled, open-label, platform trial. The Lancet 397(10289):2049–2059
  * Altman and Bland (2014) Altman DG, Bland JM (2014) Uncertainty beyond sampling error. BMJ 349:g7065
  * Altmejd et al. (2020) Altmejd A, Rocklöv J, Wallin J (2020) Nowcasting Covid-19 statistics reported with delay: a case-study of Sweden. arXiv preprint arXiv:200606840
  * Andersson and Britton (2012) Andersson H, Britton T (2012) Stochastic epidemic models and their statistical analysis, vol 151. Springer Science & Business Media
  * Anker et al. (2020) Anker SD, Butler J, Khan MS, Abraham WT, Bauersachs J, Bocchi E, Bozkurt B, Braunwald E, Chopra VK, Cleland JG, Ezekowitz J, Filippatos G, Friede T, Hernandez AF, Lam CSP, Lindenfeld J, McMurray JJV, Mehra M, Metra M, Packer M, Pieske B, Pocock SJ, Ponikowski P, Rosano GMC, Teerlink JR, Tsutsui H, Van Veldhuisen DJ, Verma S, Voors AA, Wittes J, Zannad F, Zhang J, Seferovic P, Coats AJS (2020) Conducting clinical trials in heart failure during (and after) the COVID-19 pandemic: an Expert Consensus Position Paper from the Heart Failure Association (HFA) of the European Society of Cardiology (ESC). European Heart Journal 41(22):2109–2117
  * Ariely and Jones (2008) Ariely D, Jones S (2008) Predictably irrational. Harper Audio New York, NY
  * Baden et al. (2021) Baden LR, El Sahly HM, Essink B, Kotloff K, Frey S, Novak R, Diemert D, Spector SA, Rouphael N, Creech CB, McGettigan J, Khetan S, Segall N, Solis J, Brosz A, Fierro C, Schwartz H, Neuzil K, Corey L, Gilbert P, Janes H, Follmann D, Marovich M, Mascola J, Polakowski L, Ledgerwood J, Graham BS, Bennett H, Pajon R, Knightly C, Leav B, Deng W, Zhou H, Han S, Ivarsson M, Miller J, Zaks T (2021) Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine. New England Journal of Medicine 384(5):403–416
  * Ball et al. (2020) Ball D, Humpherson E, Johnson B, McDowell M, Ng R, Radaelli C, Renn O, Seedhouse D, Spiegelhalter D, Uhl A, Watt J (2020) Improving Society’s Management of Risks - A Statement of Principles. Collaboration to explore new avenues to improve public understanding and management of risk (CAPUR). Atomium, EISMD
  * Banks et al. (2005) Banks J, Carson JS, Nelson BL, Nicol DM (2005) Discrete event system simulation. Pearson Education India
  * Beck and Pauker (1983) Beck JR, Pauker SG (1983) The Markov Process in Medical Prognosis. Medical Decision Making 3(4):419–458
  * Benvenuto et al. (2020) Benvenuto D, Giovanetti M, Vassallo L, Angeletti S, Ciccozzi M (2020) Application of the ARIMA model on the COVID-2019 epidemic dataset. Data in Brief 29:105340
  * Beyersmann et al. (2021) Beyersmann J, Friede T, Schmoor C (2021) Design aspects of COVID-19 treatment trials: Improving probability and time of favorable events. Biometrical Journal
  * Bicher et al. (2021) Bicher M, Rippinger C, Urach C, Brunmeir D, Siebert U, Popper N (2021) Evaluation of Contact-Tracing Policies against the Spread of SARS-CoV-2 in Austria: An Agent-Based Simulation. Medical decision making : an international Journal of the Society for Medical Decision Making pp 1–16
  * Binder et al. (2020) Binder K, Krauss S, Wiesner P (2020) A new visualization for probabilistic situations containing two binary events: the frequency net. Frontiers in Psychology 11:750
  * Bock et al. (2020) Bock W, Adamik B, Bawiec M, Bezborodov V, Bodych M, Burgard JP, Goetz T, Krueger T, Migalska A, Pabjan B, et al. (2020) Mitigation and herd immunity strategy for COVID-19 is likely to fail. medRxiv
  * Bonabeau (2002) Bonabeau E (2002) Agent-based modeling: Methods and techniques for simulating human systems. Proceedings of the National Academy of Sciences 99(suppl 3):7280–7287
  * Briggs et al. (2012) Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD (2012) Model Parameter Estimation and Uncertainty Analysis: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–6. Medical Decision Making 32(5):722–732
  * Britton (2010) Britton T (2010) Stochastic epidemic models: A survey. Mathematical Biosciences 225(1):24–35
  * Bruch and Atwell (2015) Bruch E, Atwell J (2015) Agent-based models in empirical social research. Sociological Methods & Research 44(2):186–221
  * Bubar et al. (2021) Bubar KM, Reinholt K, Kissler SM, Lipsitch M, Cobey S, Grad YH, Larremore DB (2021) Model-informed COVID-19 vaccine prioritization strategies by age and serostatus. Science 371(6532):916–921, DOI 10.1126/science.abe6959, URL https://www.science.org/doi/abs/10.1126/science.abe6959
  * Buelens et al. (2021) Buelens C, et al. (2021) Lockdown policy choices, outcomes and the value of preparation time - a stylised model. Tech. rep., Directorate General Economic and Financial Affairs (DG ECFIN), European Commission
  * Chatfield (1995) Chatfield C (1995) Model uncertainty, data mining and statistical inference. Journal of the Royal Statistical Society: Series A (Statistics in Society) 158(3):419–444
  * Chhatwal and He (2015) Chhatwal J, He T (2015) Economic evaluations with agent-based modelling: an introduction. Pharmacoeconomics 33(5):423–433
  * Cochran and Rubin (1973) Cochran WG, Rubin DB (1973) Controlling bias in observational studies: A review. Sankhyā: The Indian Journal of Statistics, Series A pp 417–446
  * COVID-19 Data Analysis Group (2021) COVID-19 Data Analysis Group (2021) CODAG Berichte. URL https://www.covid19.statistik.uni-muenchen.de/newsletter/index.html
  * CPK (2021) CPK (2021) Covid-prognose-konsortium. URL https://www.sozialministerium.at/Informationen-zum-Coronavirus/Neuartiges-Coronavirus-(2019-nCov)/COVID-Prognose-Konsortium.html
  * Dean (2022) Dean N (2022) Tracking COVID-19 infections: time for change. Nature 602(7896):185, DOI 10.1038/d41586-022-00336-8
  * Debrabant et al. (2021) Debrabant K, Grønbæk L, Kronborg C (2021) The cost-effectiveness of a covid-19 vaccine in a danish context. Clin Drug Investig 41:975–988
  * DeGroot and Schervish (2014) DeGroot MH, Schervish MJ (2014) Probability and Statistics. Pearson Education Limited
  * Dehning et al. (2020) Dehning J, Zierenberg J, Spitzner FP, Wibral M, Neto JP, Wilczek M, Priesemann V (2020) Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions. Science 369(6500)
  * Desrosières (2010) Desrosières A (2010) A politics of knowledge-tools: The case of statistics. Between Enlightenment and Disaster: Dimensions of the Political Use of Knowledge, Brussels: Peter Lang pp 111–129
  * Deutsche Arbeitsgemeinschaft Statistik (2021) Deutsche Arbeitsgemeinschaft Statistik (2021) Stellungnahme der DAGStat. Daten und Statistik als Grundlage für Entscheidungen: Eine Diskussion am Beispiel der Corona-Pandemie. URL https://www.dagstat.de/fileadmin/dagstat/documents/DAGStat_Covid_Stellungnahme.pdf
  * Didelez (2007) Didelez V (2007) Graphical models for composable finite Markov processes. Scandinavian Journal of Statistics 34(1):169–185
  * Dings et al. (2021) Dings C, Götz K, Och K, Sihinevich I, Selzer D, Werthner Q, Kovar L, Marok F, Schräpel C, Fuhr L, Türk D, Britz H, Smola S, Volk T, Kreuer S, Rissland J, Lehr T (2021) COVID-19 Simulator. URL https://covid-simulator.com/en
  * Drummond et al. (2005) Drummond M, Sculpher M, Torrance G, O’Brien B, Stoddart G (2005) Methods for the economic evaluation of health care programmes, 3rd edn, Oxford University Press, New York, USA, chap Chapter 2: Basic types of economic evaluation, pp 6–33
  * Drummond et al. (2008) Drummond MF, Schwartz JS, Jönsson B, Luce BR, Neumann PJ, Siebert U, Sullivan SD (2008) Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care 24:244–258
  * Duran and Odell (2013) Duran BS, Odell PL (2013) Cluster analysis: a survey, vol 100. Springer Science & Business Media
  * EbM-Netzwerk (2020) EbM-Netzwerk (2020) COVID-19: Wo ist die Evidenz? URL https://www.ebm-netzwerk.de/de/veroeffentlichungen/stellungnahmen-pressemitteilungen
  * ECDC (2020a) ECDC (2020a) Covid-19 vaccination and prioritisation strategies in theeu/eea. URL https://www.ecdc.europa.eu/sites/default/files/documents/COVID-19-vaccination-and-prioritisation-strategies.pdf
  * ECDC (2020b) ECDC (2020b) Strategic andperformance analysisof ecdc response tothe covid-19 pandemic. URL https://www.ecdc.europa.eu/sites/default/files/documents/ECDC_report_on_response_Covid-19.pdf
  * Eddy (1982) Eddy DM (1982) Probabilistic reasoning in clinical medicine: Problems and opportunities. In: Kahneman D, Slovic P, Tversky A (eds) Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press, Cambridge, pp 249–267
  * Eftekhari et al. (2020) Eftekhari H, Mukherjee D, Banerjee M, Ritov Y (2020) Markovian And Non-Markovian Processes with Active Decision Making Strategies For Addressing The COVID-19 Pandemic. arXiv preprint arXiv:200800375
  * European Statistics Code of Practice (2017) European Statistics Code of Practice (2017) URL europa.eu
  * Evangelou et al. (2021) Evangelou N, Garjani A, dasNair R, Hunter R, Tuite-Dalton KA, Craig EM, Rodgers WJ, Coles A, Dobson R, Duddy M, Ford DV, Hughes S, Pearson O, Middleton LA, Rog D, Tallantyre EC, Friede T, Middleton RM, Nicholas R (2021) Self-diagnosed covid-19 in people with multiple sclerosis: a community-based cohort of the uk ms register. Journal of Neurology, Neurosurgery & Psychiatry 92(1):107–109, URL https://jnnp.bmj.com/content/92/1/107, https://jnnp.bmj.com/content/92/1/107.full.pdf
  * Fabrigar and Wegener (2011) Fabrigar LR, Wegener DT (2011) Exploratory factor analysis. Oxford University Press
  * Fahrmeir et al. (2007) Fahrmeir L, Kneib T, Lang S, Marx B (2007) Regression. Springer
  * Fenwick et al. (2020) Fenwick E, Steuten L, Knies S, Ghabri S, Basu A, Murray JF, Koffijberg HE, Strong M, Sanders Schmidler GD, Rothery C (2020) Value of Information Analysis for Research Decisions-An Introduction: Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in health : the Journal of the International Society for Pharmacoeconomics and Outcomes Research 23:139–150
  * Flaxman et al. (2020) Flaxman S, Mishra S, Gandy A, Unwin HJT, Mellan TA, Coupland H, Whittaker C, Zhu H, Berah T, Eaton JW, et al. (2020) Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe. Nature 584(7820):257–261
  * Friedrich and Friede (2020) Friedrich S, Friede T (2020) Causal inference methods for small non-randomized studies: Methods and an application in COVID-19. Contemporary Clinical Trials 99:106213
  * Friedrich et al. (2021) Friedrich S, Antes G, Behr S, Binder H, Brannath W, Dumpert F, Ickstadt K, Kestler HA, Lederer J, Leitgöb H, Pauly M, Steland A, Wilhelm A, Friede T (2021) Is there a role for statistics in artificial intelligence? Advances in Data Analysis and Classification pp 1–24
  * Gabler and Quatember (2013) Gabler S, Quatember A (2013) Repräsentativität von Subgruppen bei geschichteten Zufallsstichproben. AStA Wirtschafts-und Sozialstatistisches Archiv 7(3-4):105–119
  * Gandjour (2020) Gandjour A (2020) Willingness to pay for new medicines: a step towards narrowing the gap between NICE and IQWiG. BMC Health Services Research 20
  * Gigerenzer (2011) Gigerenzer G (2011) What are natural frequencies? BMJ p 343:d6386
  * Gigerenzer and Edwards (2003) Gigerenzer G, Edwards A (2003) Simple tools for understanding risks: from innumeracy to insight. BMJ 327(7417):741–744
  * Gigerenzer et al. (2007) Gigerenzer G, Gaissmaier W, Kurz-Milcke E, Schwartz LM, Woloshin S (2007) Helping doctors and patients make sense of health statistics. Psychological Science in the Public Interest 8(2):53–96
  * Gralla (2020) Gralla E (2020) Discrete Event Simulation for COVID-19 Testing: Identifying Bottlenecks and Supporting Scale-Up. In: 42nd Annual Meeting of the Society for Medical Decision Making, SMDM
  * Grassly and Fraser (2008) Grassly NC, Fraser C (2008) Mathematical models of infectious disease transmission. Nature reviews Microbiology 6:477–487
  * Groot Koerkamp et al. (2010) Groot Koerkamp B, Weinstein MC, Stijnen T, Heijenbrok-Kal MH, Hunink MM (2010) Uncertainty and patient heterogeneity in medical decision models. Medical Decision Making 30(2):194–205, publisher: SAGE Publications Sage CA: Los Angeles, CA
  * Gross et al. (2021) Gross O, Moerer O, Rauen T, Böckhaus J, Hoxha E, Jörres A, Kamm M, Elfanish A, Windisch W, Dreher M, Floege J, Kluge S, Schmidt-Lauber C, Turner JE, Huber S, Addo MM, Scheithauer S, Friede T, Braun GS, Huber TB, Blaschke S (2021) Validation of a Prospective Urinalysis-Based Prediction Model for ICU Resources and Outcome of COVID-19 Disease: A Multicenter Cohort Study. Journal of Clinical Medicine 10(14)
  * Grossmann et al. (2022) Grossmann W, Hackl P, Richter J (2022) Corona: Concepts for an improved statistical database. Austrian Journal of Statistics 51(3), URL https://ajs.or.at/index.php/ajs/article/view/1350, 1–26
  * Günther et al. (2021) Günther F, Bender A, Katz K, Küchenhoff H, Höhle M (2021) Nowcasting the COVID-19 pandemic in Bavaria. Biometrical Journal 63(3):490–502
  * Hadzibegovic et al. (2021) Hadzibegovic S, Lena A, Churchill TW, Ho JE, Potthoff S, Denecke C, Rösnick L, Heim KM, Kleinschmidt M, Sander LE, Witzenrath M, Suttorp N, Krannich A, Porthun J, Friede T, Butler J, Wilkenshoff U, Pieske B, Landmesser U, Anker SD, Lewis GD, Tschöpe C, Anker MS (2021) Heart failure with preserved ejection fraction according to the HFA-PEFF score in COVID-19 patients: clinical correlates and echocardiographic findings. European Journal of Heart Failure 23(11):1891–1902
  * Hellewell (2021) Hellewell J (2021) Is COVID-19 forecasting bad, or are you just projecting? URL https://jhellewell14.github.io/2021/11/16/forecasting-projecting.html
  * Hennessy et al. (2015) Hennessy DA, Flanagan WM, Tanuseputro P, Bennett C, Tuna M, Kopec J, Wolfson MC, Manuel DG (2015) The population health model (pohem): an overview of rationale, methods and applications. Population health metrics 13:24, DOI 10.1186/s12963-015-0057-x
  * Hernán and Robins (2020) Hernán MA, Robins JM (2020) Causal Inference: What If. Boca Raton: Chapman & Hall/ CRC
  * Hethcote (2000) Hethcote HW (2000) The mathematics of infectious diseases. SIAM review 42(4):599–653
  * Hill (1965) Hill AB (1965) The environment and disease: association or causation? Proceedings of the Royal Society of Medicine 58(5):295–300
  * Hoffrage et al. (2000) Hoffrage U, Lindsey S, Hertwig R, Gigerenzer G (2000) Communicating statistical information. Science 290(5500):2261–2262
  * Holmdahl and Buckee (2020) Holmdahl I, Buckee C (2020) Wrong but useful - what Covid-19 epidemiologic models can and cannot tell us. New England Journal of Medicine 383(4):303–305
  * Horby et al. (2020) Horby PW, Mafham M, Bell JL, Linsell L, Staplin N, Emberson J, Palfreeman A, Raw J, Elmahi E, Prudon B, et al. (2020) Lopinavir–ritonavir in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial. The Lancet 396(10259):1345–1352
  * Hunink et al. (2001) Hunink M, Glasziou P, Siegel J, Weeks J, Pliskin J, Elstein A, Weinstein M (2001) Managing uncertainty. Decision Making in Health and Medicine: Integrating Evidence and Values. Cambridge University Press, New York, USA, URL https://ebm.bmj.com/content/10/1/30
  * Hunter et al. (2017) Hunter E, Mac Namee B, Kelleher JD (2017) A taxonomy for agent-based models in human infectious disease epidemiology. Journal of Artificial Societies and Social Simulation 20(3)
  * IQWiG (2020) IQWiG (2020) IQWiG: Allgemeine Methoden. Version 6.0 vom 05.11.2020. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, URL https://www.iqwig.de/methoden/allgemeine-methoden_version-6-0.pdf?rev=180500
  * Iwendi et al. (2020) Iwendi C, Bashir AK, Peshkar A, Sujatha R, Chatterjee JM, Pasupuleti S, Mishra R, Pillai S, Jo O (2020) COVID-19 patient health prediction using boosted random forest algorithm. Frontiers in public health 8:357
  * Jahn et al. (2010a) Jahn B, Pfeiffer KP, Theurl E, Tarride JE, Goeree R (2010a) Capacity Constraints and Cost-Effectiveness: A Discrete Event Simulation for Drug-Eluting Stents. Medical Decision Making 30(1):16–28
  * Jahn et al. (2010b) Jahn B, Theurl E, Siebert U, Pfeiffer KP (2010b) Tutorial in medical decision modeling incorporating waiting lines and queues using discrete event simulation. Value in Health 13(4):501–506
  * Jahn et al. (2021) Jahn B, Sroczynski G, Bicher M, Rippinger C, Mühlberger N, Santamaria J, Urach C, Schomaker M, Stojkov I, Schmid D, Weiss G, Wiedermann U, Redlberger-Fritz M, Druml C, Kretzschmar M, Paulke-Korinek M, Ostermann H, Czasch C, Endel G, Bock W, Popper N, Siebert U (2021) Targeted COVID-19 Vaccination (TAV-COVID) Considering Limited Vaccination Capacities-An Agent-Based Modeling Evaluation. Vaccines 9(5):434
  * James et al. (2021) James LP, Salomon JA, Buckee CO, Menzies NA (2021) The use and misuse of mathematical modeling for infectious disease Policymaking: lessons for the COVID-19 pandemic. Medical Decision Making 41(4):379–385
  * Jit and Brisson (2011) Jit M, Brisson M (2011) Modelling the epidemiology of infectious diseases for decision analysis. Pharmacoeconomics 29(5):371–386
  * Jun et al. (1999) Jun JB, Jacobson SH, Swisher JR (1999) Application of discrete-event simulation in health care clinics: A survey. Journal of the Operational Research Society 50(2):109–123, publisher: Springer
  * Karnon et al. (2012) Karnon J, Stahl J, Brennan A, Caro JJ, Mar J, Möller J (2012) Modeling Using Discrete Event Simulation: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force–4. Medical Decision Making 32(5):701–711, publisher: SAGE Publications Inc STM
  * Keeney and Raiffa (1976) Keeney RL, Raiffa H (1976) Decision analysis with multiple conflicting objectives. Wiley& Sons, New York
  * Kermack and McKendrick (1927) Kermack WO, McKendrick AG (1927) A contribution to the mathematical theory of epidemics. Proceedings of the Royal Society of London Series A, Containing papers of a mathematical and physical character 115(772):700–721
  * Kohli et al. (2021) Kohli M, Maschio M, Becker D, Weinstein MC (2021) The potential public health and economic value of a hypothetical COVID-19 vaccine in the United States: Use of cost-effectiveness modeling to inform vaccination prioritization. Vaccine 39:1157–1164
  * Krauss et al. (2020) Krauss S, Weber P, Binder K, Bruckmaier G (2020) Natürliche Häufigkeiten als numerische Darstellungsart von Anteilen und Unsicherheit–Forschungsdesiderate und einige Antworten. Journal für Mathematik-Didaktik 41(2):485–521
  * Kretzschmar et al. (2020) Kretzschmar ME, Rozhnova G, Bootsma MCJ, van Boven M, van de Wijgert JHHM, Bonten MJM (2020) Impact of delays on effectiveness of contact tracing strategies for COVID-19: a modelling study. The Lancet Public health 5:e452–e459
  * Kristjanpoller et al. (2021) Kristjanpoller W, Michell K, Minutolo MC (2021) A causal framework to determine the effectiveness of dynamic quarantine policy to mitigate COVID-19. Applied Soft Computing 104:107241
  * Kucharski et al. (2021) Kucharski AJ, Hodcroft EB, Kraemer MUG (2021) Sharing, synthesis and sustainability of data analysis for epidemic preparedness in europe. The Lancet Regional Health - Europe 9:100215, URL https://www.sciencedirect.com/science/article/pii/S2666776221001927
  * Küchenhoff et al. (2021) Küchenhoff H, Günther F, Höhle M, Bender A (2021) Analysis of the early COVID-19 epidemic curve in Germany by regression models with change points. Epidemiology & Infection 149
  * Kunz et al. (2020) Kunz CU, Jörgens S, Bretz F, Stallard N, Lancker KV, Xi D, Zohar S, Gerlinger C, Friede T (2020) Clinical Trials Impacted by the COVID-19 Pandemic: Adaptive Designs to the Rescue? Statistics in Biopharmaceutical Research 12(4):461–477
  * Küchenhoff et al. (2021) Küchenhoff H, Antes G, Berger U, Hoyer A, Brinks R, Kauermann G (2021) CODAG Bericht Nr. 18. Informationen zur Pandemiesteuerung: Welche Daten benötigen wir? URL https://www.covid19.statistik.uni-muenchen.de/newsletter/index.html
  * Li and O’Donoghue (2013) Li J, O’Donoghue C (2013) A survey of dynamic microsimulation models: uses, model structure and methodology. International Journal of Microsimulation 6(2):3–55
  * Licker (2003) Licker MD (ed) (2003) McGraw-Hill dictionary of mathematics. McGraw-Hill Companies, Inc.
  * Loss et al. (2021) Loss J, Boklage E, Jordan S, Jenny MA, Weishaar H, El Bcheraoui C (2021) Risikokommunikation bei der Eindämmung der COVID-19-Pandemie: Herausforderungen und Erfolg versprechende Ansätze. Bundesgesundheitsblatt \- Gesundheitsforschung - Gesundheitsschutz 64(3):294–303
  * Luo et al. (2021) Luo J, Zhang Z, Fu Y, Rao F (2021) Time series prediction of COVID-19 transmission in America using LSTM and XGBoost algorithms. Results in Physics p 104462
  * Macal and North (2008) Macal CM, North MJ (2008) Agent-based modeling and simulation: ABMS examples. In: 2008 Winter Simulation Conference, IEEE, pp 101–112
  * Macal and North (2010) Macal CM, North MJ (2010) Tutorial on agent-based modelling and simulation. Journal of Simulation 4(3):151–162
  * Mandelblatt et al. (2016) Mandelblatt JS, Stout NK, Schechter CB, van den Broek JJ, Miglioretti DL, Krapcho M, Trentham-Dietz A, Munoz D, Lee SJ, Berry DA, van Ravesteyn NT, Alagoz O, Kerlikowske K, Tosteson ANA, Near AM, Hoeffken A, Chang Y, Heijnsdijk EA, Chisholm G, Huang X, Huang H, Ergun MA, Gangnon R, Sprague BL, Plevritis S, Feuer E, de Koning HJ, Cronin KA (2016) Collaborative Modeling of the Benefits and Harms Associated With Different U.S. Breast Cancer Screening Strategies. Annals of Internal Medicine 164:215–225
  * Marshall et al. (2015a) Marshall DA, Burgos-Liz L, IJzerman MJ, Crown W, Padula WV, Wong PK, Pasupathy KS, Higashi MK, Osgood ND (2015a) Selecting a dynamic simulation modeling method for health care delivery research—Part 2: Report of the ISPOR Dynamic Simulation Modeling Emerging Good Practices Task Force. Value in Health 18(2):147–160
  * Marshall et al. (2015b) Marshall DA, Burgos-Liz L, IJzerman MJ, Osgood ND, Padula WV, Higashi MK, Wong PK, Pasupathy KS, Crown W (2015b) Applying dynamic simulation modeling methods in health care delivery research—the SIMULATE checklist: report of the ISPOR simulation modeling emerging good practices task force. Value in Health 18(1):5–16
  * Mathieu (2022) Mathieu E (2022) Commit to transparent COVID data until the WHO declares the pandemic is over. Nature 602(7898):549–549, DOI 10.1038/d41586-022-00424-9
  * Mayo (2004) Mayo E (2004) The human problems of an industrial civilization. Routledge
  * McDowell et al. (2019) McDowell M, Gigerenzer G, Wegwarth O, Rebitschek FG (2019) Effect of tabular and icon fact box formats on comprehension of benefits and harms of prostate cancer screening: a randomized trial. Medical Decision Making 39(1):41–56
  * Melman et al. (2021) Melman G, Parlikad A, Cameron E (2021) Balancing scarce hospital resources during the COVID-19 pandemic using discrete-event simulation. Health Care Management Science pp 1–19
  * Miksch et al. (2019) Miksch F, Jahn B, Espinosa KJ, Chhatwal J, Siebert U, Popper N (2019) Why should we apply ABM for decision analysis for infectious diseases?—An example for dengue interventions. PloS one 14(8):e0221564
  * Münnich (2020) Münnich R (2020) Qualität der regionalen Armutsmessung–vom Design zum Modell. In: Qualität bei zusammengeführten Daten, Springer, pp 7–25
  * Münnich et al. (2021) Münnich R, Schnell R, Brenzel H, Dieckmann H, Dräger S, Emmenegger J, Höcker P, Kopp J, Merkle H, Neufang K, Obersneider M, Reinhold J, Schaller J, Schmaus S, Stein P (2021) A Population Based Regional Dynamic Microsimulation of Germany: The MikroSim Model. Methods, Data, Analyses 15(2):241–264, URL https://mda.gesis.org/index.php/mda/article/view/2021.03
  * Mütze and Friede (2020) Mütze T, Friede T (2020) Data monitoring committees for clinical trials evaluating treatments of COVID-19. Contemporary Clinical Trials 98:106154
  * Neumann et al. (2016) Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG (2016) Cost-Effectiveness in Health and Medicine. Oxford University Press
  * Nianogo and Arah (2015) Nianogo RA, Arah OA (2015) Agent-based modeling of noncommunicable diseases: a systematic review. American Journal of Public Health 105(3):e20–e31
  * Nussbaumer-Streit et al. (2020) Nussbaumer-Streit B, Mayr V, Dobrescu AI, Chapman A, Persad E, Klerings I, Wagner G, Siebert U, Ledinger D, Zachariah C, et al. (2020) Quarantine alone or in combination with other public health measures to control covid-19: a rapid review. Cochrane Database of Systematic Reviews 4(4):CD013574, update in: Cochrane Database Syst Rev. 2020; 15;9:CD013574
  * Orcutt (1957) Orcutt GH (1957) A new type of socio-economic system. The Review of Economics and Statistics 39(2):116–123
  * Otava and Mylona (2020) Otava M, Mylona K (2020) Communicating statistical conclusions of experiments to scientists. Quality and Reliability Engineering International 36(8):2688–2698
  * O’Hagan (2004) O’Hagan A (2004) Bayesian statistics: principles and benefits. Frontis pp 31–45
  * Parmigiani and Inoue (2009) Parmigiani G, Inoue L (2009) Decision theory: Principles and approaches, vol 812\. John Wiley & Sons
  * Pfeffermann and Sverchkov (2009) Pfeffermann D, Sverchkov M (2009) Inference under informative sampling. In: Handbook of Statistics, vol 29, Elsevier, pp 455–487
  * Pidd (2004) Pidd M (2004) Computer simulation in management science. 5th, John Wiley and Sons Ltd
  * Pitman et al. (2012) Pitman R, Fisman D, Zaric GS, Postma M, Kretzschmar M, Edmunds J, Brisson M (2012) Dynamic transmission modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–5. Medical decision making 32(5):712–721
  * RECOVERY Collaborative Group (2020) RECOVERY Collaborative Group (2020) Effect of hydroxychloroquine in hospitalized patients with Covid-19. New England Journal of Medicine 383(21):2030–2040
  * Reddy et al. (2021) Reddy KP, Fitzmaurice KP, Scott JA, Harling G, Lessells RJ, Panella C, Shebl FM, Freedberg KA, Siedner MJ (2021) Clinical outcomes and cost-effectiveness of covid-19 vaccination in south africa. Nat Commun 12:6238
  * Rendtel et al. (2021) Rendtel U, Liebig S, Meister R, Wagner GG, Zinn S (2021) Die erforschung der dynamik der corona-pandemie in deutschland: Survey-konzepte und eine exemplarische umsetzung mit dem sozio-oekonomischen panel (soep). AStA Wirtschafts-und Sozialstatistisches Archiv pp 1–42
  * Richardson and Spiegelhalter (2021) Richardson S, Spiegelhalter D (2021) How ideas from decision theory can help guide our actions. URL https://rss.org.uk/news-publication/news-publications/2021/general-news/how-ideas-from-decision-theory-can-help-guide-our/
  * Ritchie (2021) Ritchie H (2021) COVID’s lessons for climate, sustainability and more from our world in data. Nature 598(7879):9–9, DOI 10.1038/d41586-021-02691-4
  * Roberts et al. (2012) Roberts M, Russell LB, Paltiel AD, Chambers M, McEwan P, Krahn M, Force I (2012) Conceptualizing a model: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-2. Medical decision making : an international Journal of the Society for Medical Decision Making 32:678–689
  * Robins (1998) Robins JM (1998) Structural nested failure time models. Encyclopedia of Biostatistics 6:4372–4389
  * Robins et al. (2000) Robins JM, Hernán MA, Brumback B (2000) Marginal structural models and causal inference in epidemiology. Epidemiology 11(5):550–560
  * Robins et al. (2004) Robins JM, Hernán MA, Siebert U (2004) Estimations of the effects of multiple interventions. In: Ezzati M, Lopez A, Rodgers A, Murray C (eds) Comparative Quantification of Health Risks: Global and Regional Burden of Disease Attributable to Selected Major Risk Factors, vol 1, World Health Organization, Geneva, pp 2191–2230
  * Rochau et al. (2015) Rochau U, Jahn B, Qerimi V, Burger EA, Kurzthaler C, Kluibenschaedl M, Willenbacher E, Gastl G, Willenbacher W, Siebert U (2015) Decision-analytic modeling studies: An overview for clinicians using multiple myeloma as an example. Critical Reviews in Oncology/Hematology 94(2):164–178
  * Romano et al. (2020) Romano A, Sotis C, Dominioni G, Guidi S (2020) The scale of covid-19 graphs affects understanding, attitudes, and policy preferences. Health economics 29(11):1482–1494
  * Rosling and Zhang (2011) Rosling H, Zhang Z (2011) Health advocacy with gapminder animated statistics. Journal of Epidemiology and Global Health 1:11–14
  * Rothwell (2005) Rothwell PM (2005) External validity of randomised controlled trials:“to whom do the results of this trial apply?”. The Lancet 365(9453):82–93
  * Roy et al. (2021) Roy S, Bhunia GS, Shit PK (2021) Spatial prediction of COVID-19 epidemic using ARIMA techniques in India. Modeling Earth Systems and Environment 7(2):1385–1391
  * Rubin (1974) Rubin DB (1974) Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66(5):688
  * Rubin (2008) Rubin DB (2008) For objective causal inference, design trumps analysis. The Annals of Applied Statistics 2(3):808–840
  * Rustam et al. (2020) Rustam F, Reshi AA, Mehmood A, Ullah S, On BW, Aslam W, Choi GS (2020) COVID-19 future forecasting using supervised machine learning models. IEEE access 8:101489–101499
  * RWI – Leibniz-Institut für Wirtschaftsforschung (2020) RWI – Leibniz-Institut für Wirtschaftsforschung (2020) Anti-corona measures - don’t just look at new infections. URL https://www.rwi-essen.de/unstatistik/108/
  * Saidani et al. (2021) Saidani M, Kim H, Kim J (2021) Designing optimal COVID-19 testing stations locally: A discrete event simulation model applied on a university campus. PloS one 16(6):e0253869
  * Salas (2021) Salas J (2021) Improving the estimation of the COVID-19 effective reproduction number using nowcasting. Statistical Methods in Medical Research p 09622802211008939
  * Sandmann et al. (2021) Sandmann FG, Davies NG, Vassall A, Edmunds WJ, Jit M, for the Mathematical Modelling of Infectious Diseases COVID-19 working group C (2021) The potential health and economic value of sars-cov-2 vaccination alongside physical distancing in the uk: a transmission model-based future scenario analysis and economic evaluation. The Lancet Infectious diseases 21:962–974, DOI 10.1016/S1473-3099(21)00079-7
  * Schneble et al. (2021) Schneble M, De Nicola G, Kauermann G, Berger U (2021) Nowcasting fatal COVID-19 infections on a regional level in Germany. Biometrical Journal 63(3):471–489
  * Schnell (2019) Schnell R (2019) Survey-Interviews: Methoden standardisierter Befragungen, 2nd edn. Springer VS
  * Schöffski and Schulenburg (2011) Schöffski O, Schulenburg JMGvd (2011) Gesundheitsökonomische Evaluationen. Springer Berlin Heidelberg
  * Schüller et al. (2021) Schüller K, Koch H, Rampelt F (2021) Data Literacy Charta. URL https://www.stifterverband.org/data-literacy-charter, accessed Dec 21, 2021
  * Sheinson et al. (2021) Sheinson D, Dang J, Shah A, Meng Y, Elsea D, Kowal S (2021) A Cost-Effectiveness Framework for COVID-19 Treatments for Hospitalized Patients in the United States. Advances in Therapy 38:1811–1831
  * Shinde et al. (2021) Shinde V, Bhikha S, Hoosain Z, Archary M, Bhorat Q, Fairlie L, Lalloo U, Masilela MS, Moodley D, Hanley S, Fouche L, Louw C, Tameris M, Singh N, Goga A, Dheda K, Grobbelaar C, Kruger G, Carrim-Ganey N, Baillie V, de Oliveira T, Lombard Koen A, Lombaard JJ, Mngqibisa R, Bhorat AE, Benadé G, Lalloo N, Pitsi A, Vollgraaff PL, Luabeya A, Esmail A, Petrick FG, Oommen-Jose A, Foulkes S, Ahmed K, Thombrayil A, Fries L, Cloney-Clark S, Zhu M, Bennett C, Albert G, Faust E, Plested JS, Robertson A, Neal S, Cho I, Glenn GM, Dubovsky F, Madhi SA (2021) Efficacy of NVX-CoV2373 Covid-19 Vaccine against the B.1.351 Variant. New England Journal of Medicine 384(20):1899–1909
  * Siebert (2003) Siebert U (2003) When should decision-analytic modeling be used in the economic evaluation of health care? The European Journal of Health Economics 4:143–150
  * Siebert (2005) Siebert U (2005) Using decision-analytic modelling to transfer international evidence from health technology assessment to the context of the german health care system. GMS Health Technol Assess 1:Doc03 1
  * Siebert et al. (2012) Siebert U, Alagoz O, Bayoumi AM, Jahn B, Owens DK, Cohen DJ, Kuntz KM (2012) State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3. Medical decision making : an international journal of the Society for Medical Decision Making 32:690–700
  * Siebert et al. (2013) Siebert U, Rochau U, Claxton K (2013) When is enough evidence enough?–Using systematic decision analysis and value-of-information analysis to determine the need for further evidence. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen 107(9-10):575–584
  * Sonnenberg and Beck (1993) Sonnenberg FA, Beck JR (1993) Markov models in medical decision making: a practical guide. Medical Decision Making 13(4):322–338, publisher: Sage Publications Sage CA: Thousand Oaks, CA
  * Spielauer (2007) Spielauer M (2007) Dynamic microsimulation of health care demand, health care finance and the economic impact of health behaviours: survey and review. International Journal of Microsimulation 1(1):35–53, publisher: International Microsimulation Association
  * Sroczynski et al. (2020) Sroczynski G, Esteban E, Widschwendter A, Oberaigner W, Borena W, von Laer D, Hackl M, Endel G, Siebert U (2020) Reducing overtreatment associated with overdiagnosis in cervical cancer screening—a model-based benefit–harm analysis for austria. International Journal of Cancer 147(4):1131–1142, DOI https://doi.org/10.1002/ijc.32849, URL https://onlinelibrary.wiley.com/doi/abs/10.1002/ijc.32849, https://onlinelibrary.wiley.com/doi/pdf/10.1002/ijc.32849
  * Stallard et al. (2020) Stallard N, Hampson L, Benda N, Brannath W, Burnett T, Friede T, Kimani PK, Koenig F, Krisam J, Mozgunov P, Posch M, Wason J, Wassmer G, Whitehead J, Williamson SF, Zohar S, Jaki T (2020) Efficient Adaptive Designs for Clinical Trials of Interventions for COVID-19. Statistics in Biopharmaceutical Research 12(4):483–497
  * STIKO (2016) STIKO (2016) Stiko (2016) methoden zur durchführung und berücksichtigung von modellierungen zur vorhersage epidemiologischer und gesundheitsökonomischer effekte von impfungen für die ständige impfkommission, version 1.0 (stand: 16.03.2016), berlin. URL https://www.rki.de/DE/Content/Kommissionen/STIKO/Aufgaben_Methoden/Methoden_Modellierung.pdf?__blob=publicationFile
  * The Royal Statistical Society (2021) The Royal Statistical Society (2021) Statistics, Data and Covid - ”Ten statistical lessons the government can learn from the past year. URL https://rss.org.uk/policy-campaigns/policy/covid-19-task-force/statistics,-data-and-covid/
  * Tufte (2001) Tufte ER (2001) The Visual Display of Quantitative Information, 2nd edn. Graphics Press, Cheshire, CT
  * Van Pelt et al. (2021) Van Pelt A, Glick HA, Yang W, Rubin D, Feldman M, Kimmel SE (2021) Evaluation of COVID-19 testing strategies for repopulating college and university campuses: a decision tree analysis. Journal of Adolescent Health 68(1):28–34
  * Weinstein (2006) Weinstein MC (2006) Recent developments in decision-analytic modelling for economic evaluation. Pharmacoeconomics 24(11):1043–1053, publisher: Springer
  * WHO (2020) WHO (2020) Communicating and Managing Uncertainty in the COVID-19 Pandemic: A quick guide. URL https://cdn.who.int/media/docs/default-source/searo/whe/coronavirus19/managing-uncertainty-in-covid-19-a-quick-guide.pdf
  * Wu et al. (2020) Wu F, Zhao S, Yu B, Chen YM, Wang W, Song ZG, Hu Y, Tao ZW, Tian JH, Pei YY, et al. (2020) A new coronavirus associated with human respiratory disease in China. Nature 579(7798):265–269
  * Zhang (2018) Zhang X (2018) Application of discrete event simulation in health care: a systematic review. BMC Health Services Research 18(1):1–11, publisher: BioMed Central
  * Zhou et al. (2020) Zhou P, Yang XL, Wang XG, Hu B, Zhang L, Zhang W, Si HR, Zhu Y, Li B, Huang CL, et al. (2020) A pneumonia outbreak associated with a new coronavirus of probable bat origin. Nature 579(7798):270–273