demand literature: the bootstrapping approach and the Overlapping/Non-
Overlapping aggregation Blocks approach (Boylan and Syntetos, 2021).

The bootstrapping approach relies upon a resampling (with or without
replacement) of the historical demand data to build the empirical distribution
of the demand over a specified interval. As discussed in §2.7.5, this approach
was initially introduced by Efron (1979). Since then, it has been developed by
Willemain et al. (2004) and Zhou and Viswanathan (2011) to deal with
intermittent demand items (Babai et al., 2020). Willemain et al. (2004) have
proposed a method that resamples demand data by using a Markov chain to switch
between no demand and demand periods. The empirical outperformance of this
method has been shown when compared to Simple Exponential Smoothing (SES) and
Croston’s method (see also §2.8.1). However, the findings of Willemain et al.
(2004)’s work have been challenged by Gardner and Koehler (2005) and some
limitations have been addressed by Syntetos et al. (2015b). Zhou and
Viswanathan (2011) have developed an alternative bootstrapping method. Their
method samples separately demand intervals and demand sizes and it has been
shown to be associated with a good performance for long lead-times. Teunter
and Duncan (2009) and Hasni et al. (2019a) have developed adjustments of the
bootstrapping methods, where the lead-time demand forecast is adjusted by
assuming that the first period in the lead-time bucket corresponds to a non-
zero demand. They have demonstrated the outperformance of the adjusted
bootstrapping methods in a periodic order-up-to-level inventory control
system. A review of the bootstrapping methods in the context of intermittent
demand is provided by Hasni et al. (2019b).

Porras and Dekker (2008) were the first to consider aggregation with
overlapping and non-overlapping blocks (OB and NOB) approach in forecasting
the demand of spare parts. In the NOB approach, a demand series is divided
into consecutive non-overlapping blocks of time, whereas in OB, at each period
the oldest observation is dropped and the newest is included (Rostami-Tabar et
al., 2013). Boylan and Babai (2016) have compared the statistical and
inventory performance of the OB and NOB methods. They found that, unless the
demand history is short, there is a clear advantage of using OB instead of
NOB. More recently, based on extreme value theory (EVT), Zhu et al. (2017)
have proposed an improvement of the OB method that models better the tail of
lead-time demand. They have shown that the empirical-EVT method leads to
higher achieved target cycle service levels when compared to the original
method proposed by Porras and Dekker (2008). Temporal aggregation is further
discussed in §2.10.2.

####  2.8.3 Classification methods666666This subsection was written by John E.
Boylan.

In many application areas, forecasts are required across a wide collection of
products, services or locations. In this situation, it is convenient to
introduce classification rules that allow subsets of time series to be
forecasted using the same approaches and methods. Categorisation rules, such
as the ABC inventory classification, serve the forecasting function only
coincidentally. They do not necessarily align to the selection of the best
forecasting method.

Within certain modelling frameworks, classification of time series is well
established. For example, within an ARIMA framework (Box et al., 2008, and
§2.3.4), or within a state-space framework for exponential smoothing (Hyndman
et al., 2002, and §2.3.1), series may be classified, for example based on the
AIC (Akaike, 1973). It is more challenging to classify series according to
their recommended forecasting method if some of the candidate methods, such as
Croston’s method (see §2.8.1), lack a fully satisfactory model base. In the
field of intermittent demand forecasting, Syntetos et al. (2005) proposed the
SBC classification scheme, enabling time series to be classified according to
their length of average demand interval and coefficient of variation of demand
sizes (when demand occurs). These rules were based on assumptions of
independent and identically distributed (iid) demand, and a comparison of
expected mean square error between methods. The scheme has been extended by
Kostenko and Hyndman (2006) and by Petropoulos and Kourentzes (2015). In an
empirical case-study, Boylan et al. (2008) examined series not necessarily
conforming to iid assumptions and found the rules to be robust to inexact
specification of cut-off values. Moon et al. (2013) used logistic regression
to classify time series of demand for spare parts in the South Korean Navy.
The classification was designed to identify superior performance (accuracy and
inventory costs) of direct and hierarchical forecasting methods, based on the
serial correlation of demands, the coefficient of variation of demand volume
of spare parts (see also §3.2.7), and the functionality of the naval
equipment.

Bartezzaghi et al. (1999) identified five factors that contribute towards
intermittence and ‘lumpiness’ (intermittence with highly variable demand
sizes): number of potential customers, frequency of customers’ requests,
heterogeneity of customers, variety of individual customer’s requests, and
correlations between customers’ requests. These may contribute towards useful
classifications, for example by the number of customers for an item. When this
number is low and some of the customers are large, then direct communication
with the customers can inform judgmental forecasts. Similarly, if a customer’s
requests are highly variable, then ‘advance demand information’ from customers
can help to improve judgmental estimates. These strategies can be very useful
in a business-to-business environment, where such strategies are feasible.

An alternative approach to classification is combination of forecasts (see
§2.6 for a review, and §2.6.1 in particular). Petropoulos and Kourentzes
(2015) investigated combining standard forecasting methods for intermittent
demand (e.g., SES, Croston, Syntetos-Boylan Approximation; see also §2.8.1).
They did not find this to improve accuracy directly, but obtained good results
from the use of combinations of forecasts at different temporal frequencies,
using methods selected from the extended SBC classification scheme.

####  2.8.4 Peak over the theshold676767This subsection was written by
Konstantinos Nikolopoulos.

In the forecasting literature, Nikolopoulos (2020) argues that great attention
has been given to modelling fast-moving time series with or without cues of
information available (Nikolopoulos et al., 2007). Less attention has been
given to intermittent/count series (see §2.8.1, §2.8.2, and §2.3.8), which are
more difficult to forecast given the presence of two sources of uncertainty:
demand volume, and timing.

Historically there have been few forecasting methods developed specifically
for such data (Syntetos et al., 2015b). We believe that through a time series
decomposition approach á la Leadbetter (1991) we can isolate ‘peaks over
threshold’ (POT) data points, and create new intermittent series from any time
series of interest. The derived series present almost identical
characteristics with the series that Croston (1972) analysed. In essence one
could use such decomposition forecasting techniques to tackle much more
difficult phenomena and problems coming from finance, politics, healthcare,
humanitarian logistics, business, economics, and social sciences.

Any time series can be decomposed into two sub-series: one containing the
baseline (white swans) and one containing the extreme values over an
arbitrary-set or rule-based-set threshold (grey and black swans) as proposed
by Taleb (2008); see also §2.3.22. Unfortunately, major decision-related risks
and most of the underlying uncertainty lay with these extremes. So, it is very
important to be able to effectively model and forecast them.

It is unlikely that any forecasting approach will accurately give the exact
timing of the forthcoming extreme event, but it will instead provide a
satisfactory cumulative forecast over a long period of time. The question
still stands what can one do with this forecast? For earthquake data, although
even if we know that a major earthquake is going to hit a region, it is almost
impossible to decide to evacuate cities, but still we can influence and
legislate the way structures are built and increase the awareness, training,
preparedness and readiness of the public; and also ensure enough capital on
hold to cope with the aftermath of the major event. For epidemics/pandemics
(see §3.6.2) there are clear implications, as we have evidenced with COVID-19,
on how proactively we can source and secure human resources, medical supplies,
etc.

What is the current doctrine when forecasting in such a context: advanced
probabilistic models. These methods typically require a lot of data and
reconstruct the distributions of the underlying phenomena. These come with
common successes and a plethora of constraints: big data sets needed for
training the models, high mathematical complexity, and invisibility to
practitioners how these methods do actually work and thus less acceptance in
practice. Yet again, forecasting accuracy is the name of the game and thus
these forecasting methods are serious contenders for the task in hand.

Extreme Value Theory (EVT) analyses extreme deviations from statistical
measures of central location to estimate the probability of events that are
more extreme than anything observed in the time series. This is usually done
in the following two ways (Nikolopoulos, 2020): (i) deriving maxima and/or
minima series as a first step and then having the Generalised Extreme Value
Distribution fitted (often the number of extreme events is limited), and (ii)
isolating the values that exceed a threshold (point over threshold) that can
also lead to only a few instances extracted – so a very intermittent series in
nature. The analysis involves fitting a Poisson distribution for the number of
events in a basic time period and a second distribution – usually a
Generalised Pareto Distribution – for the size of the resulting POT values.

###  2.9 Reasoning and mining

####  2.9.1 Fuzzy logic686868This subsection was written by Claudio Carnevale.

The “classical” Boolean logic is not able to handle for uncertainties and/or
vagueness that are necessary when dealing with many real world problems. This
is in part due to the fact that the Boolean logic is based on only two values
(i.e., a statement can only be true or false). Fuzzy logic tries to overcome
this issue by admitting that a statement/variable could be partially true or
partially false. Mathematically, the fuzzy logic framework is based on the
work of Zadeh (1965) who introduced the theory of fuzzy sets. The main point
of this theory is the definition of two kinds of sets:

  1. 1.

Crisp sets are the “classical” sets in the Boolean logic. An element can
belong (or not) to a certain set.

  2. 2.

Fuzzy sets, where an element can belong to the sets with a certain membership
grade, with a value that varies in the interval $[0,1]$.

The definition of the fuzzy sets allows the framework to take into account the
uncertainty and vagueness of information. An extension of this approach is
related to the fact that a certain variable can assume a crisp value
(classical theory) or can belong to different fuzzy sets with different
membership grade. For example, in a system implemented to forecast the daily
pollutant concentration in atmosphere, one of the inputs could relate to the
weather conditions, such as the wind speed. In the classical approach, the
system must have as an input the value of the wind speed at a certain day. In
the fuzzy approach, the input of the system could be the membership grade of
the input variable to three different fuzzy sets: (i) “Not windy”, (ii)
“average windy”, and (iii) “strong windy”. On the other hand, the user of the
forecasting system may be only interested in a classification of the output
variable instead of the crisp value. In this case, the fuzzy approach is
applied to the pollutant concentration which could belong with a certain
degree to the fuzzy sets (i) “not polluted day”, (ii) “medium polluted day”,
(iii) “high polluted day”, and (iv) “critical polluted day”.

In fuzzy theory, each fuzzy set is characterised by a (generally nonlinear)
function, called the membership function, linking crisp values to the
membership of the different sets. The association of a crisp value to its
membership for a set is called fuzzyfication, while the inverse operation
(from a membership value to a crisp value) is called defuzzification. As with
the logic theory, the inference system assumes a key role in the fuzzy theory.
A Fuzzy Inference System (FIS) allows the interpretation of the membership
grades of the input variable(s) and, given some sets of fuzzy rules, assigns
the corresponding values to the output variable(s). In the literature, two
main fuzzy inference systems are presented:

  1. 1.

Mamdani system (Mamdani and Assilian, 1975), where both the input and output
of the inference system are membership functions.

  2. 2.

Sugeno system (Sugeno, 1985), where the output of the inference system is a
crisp value, usually obtained by applying a linear function to the defuzzified
value of the input.

####  2.9.2 Association rule mining696969This subsection was written by
Daniele Apiletti.

Association rule mining is an exploratory data-driven approach which is able
to automatically and exhaustively extract all existing correlations in a data
set of categorical features. It is a powerful but computationally intensive
technique, successfully applied in different forecasting contexts (Acquaviva
et al., 2015; Apiletti and Pastor, 2020; Di Corso et al., 2018). Its results
are in a human-readable form.

The data set must be in the form of transactions, i.e., a collection of
events, each described by categorical features. If the phenomena under
analysis are modelled by continuous-valued variables, discretisation can be
applied to obtain a suitable data set.

Association rule mining core task is the frequent itemset extraction, which
consists in finding frequently-occurring relationships among items in a data
set (Han et al., 2011). Given a data set of records characterised by several
attributes, an item refers to a pair of (attribute $=$ value), while a set of
items is called itemset. The support count of an itemset is the number of
records $r$ containing that itemset. The support of an itemset is the
percentage of records containing it with respect to the total number of
records in the data set. An itemset is frequent when its support is greater
than or equal to a minimum support threshold.

An association rule is an implication in the form $A\rightarrow B$, where $A$
and $B$ are disjoint itemsets (i.e., $A\cap B=\emptyset$) (Tan et al., 2005).
$A$ is called rule body or antecedent and $B$ rule head or consequent.

To evaluate the quality of an association rule, the support, confidence, and
lift metrics are commonly exploited (Han et al., 2011). Rule support is the
fraction of records containing both $A$ and $B$, indicating the probability
that a record contains every item in these itemsets. The support of the rule
is computed as the support of the union of $A$ and $B$.

Rule confidence represents the strength of the implication, and is the
conditional probability that a transaction containing $A$ also contains $B$,
$P(B|A)$, i.e., the proportion of records that contain $A$ with respect to
those that also contain $B$.

Finally, the lift of a rule measures the correlation between antecedent and
consequent. It is defined as the ratio between the rule $A\rightarrow B$
confidence and the support of $B$. A lift ratio equal to 1.0 implies that
itemsets $A$ and $B$ are not correlated. A lift higher than 1.0 indicates a
positive correlation, meaning that the occurrence of $A$ likely leads to the
occurrence of $B$ with the given confidence. The greater the lift, the
stronger the association. Finally, a lift lower than 1.0 indicates a negative
correlation between $A$ and $B$.

The problem of association rule mining consists in the extraction of all the
association rules having rule support and confidence greater than the
respective support and confidence thresholds, $MinConf$ and $MinSup$, defined
as parameters of the mining process (Tan et al., 2005). These thresholds allow
to control the statistical relevance of the extracted rules.

The process of rule mining consists of two steps. The first step is the
computation of frequent itemsets, i.e., itemsets with support greater or equal
to $MinSup$. The second step is the extraction of association rules from
frequent itemsets. Let be $F$ a frequent itemset, hence having a support
higher than $MinSup$, pairs $A$ and $B=F-A$ are derived so that the confidence
of $A\rightarrow B$ is higher than $MinConf$. The first step of the process is
the most computationally expensive. Thus, several algorithms have been
proposed to solve the problem of frequent itemset extraction (Zaki, 2000),
some specifically addressing high-dimensionality issues (Apiletti et al.,
2017, 2015). Despite being computationally demanding, association rule mining
is an exhaustive approach, i.e., all and only statistically relevant
correlations are extracted. §3.8.11 offers an example of applying association
rule mining to forecast the quality of beverages.

####  2.9.3 Forecasting with text information707070This subsection was written
by Xiaojia Guo.

Text data, such as social media posts, scholar articles and company reports,
often contains valuable information that can be used as predictors in
forecasting models (Aggarwal and Zhai, 2012). Before extracting useful
features from the data, a text document needs to be cleaned and normalised for
further processing. The first step of preparing the data is to filter out stop
words – the words that do not add much meaning to a sentence, e.g., “a”, “is”,
and “me”. For grammatical reasons, it is necessary for documents to use
different forms of words. Stemming and lemmatisation can be applied to reduce
inflectional forms or relate different forms of a word to a common base form.
Stemming often chops off the end of a word, while lemmatisation uses
vocabularies and morphological analysis to return the base form (or the lemma)
of a word (Lovins, JB, 1968; Manning et al., 2008). For example, the word
“industries” will be turned into “industri” or “industry” if stemming or
lemmatisation is applied.

To model and analyse text data, we need to transform it into numerical
representations so the forecasting models can process them as predictors. One
way of transforming the data is through sentiment analysis. Sentiment analysis
is often applied to detect polarity within customer materials such as reviews
and social media posts (Das and Chen, 2007; Archak et al., 2011). An easy way
to obtain the sentiment score of a word is to look it up in a happiness
dictionary (for example, the hedonometer dictionary, Hedonometer, 2020).
Another common way of representing the sentiment is to use a vector of numeric
values that denote the word’s positivity, negativity and neutrality based on
existing lexical databases such as the WordNet (Godbole et al., 2007;
Baccianella et al., 2010). Once the sentiment of each word is calculated, we
can apply an aggregation algorithm (e.g., simple average) to measure the
sentiment of an entire sentence or paragraph.

In scholar articles and company reports, context features might be more
important than sentiments. The bag-of-words model and word embeddings are
often applied to generate numeric representations of such text. A bag-of-words
model simply returns a matrix that describes the occurrence of words within a
document (Goldberg, 2017). When we use this matrix as input to a forecasting
model, each word count can be considered as a feature. The Word2Vec method is
a widely used embedding method that is built based on the context of a word.
Specifically, it trains a two-layer neural network that takes each word as an
input to predict its surrounding words (see §2.7.8 for a discussion of neural
networks for forecasting). The weights from the input layer to the hidden
layer are then utilised as the numerical representation for the input word (Le
and Mikolov, 2014). Once the text is turned into numeric representations, they
can be used as predictors in any forecasting models. The most challenging part
in this process is to find the right technique to extract features from the
text.

In terms of software implementation, the Natural Language Toolkit (NLTK) and
SpaCy library in Python can be applied to remove stop words and stem or
lemmatise text (Loper and Bird, 2002; Honnibal, 2015). The bag-of-words
technique is also available in NLTK. A particular implementation of the
Word2Vec model is available on Google code (2013). Moreover, a public data set
of movie reviews that is commonly studied in literature is available from the
Stanford NLP Group (2013).

###  2.10 Forecasting by aggregation

####  2.10.1 Cross-sectional hierarchical forecasting717171This subsection was
written by Patrícia Ramos.

In many applications time series can be aggregated at several levels of
aggregation based on geographic or logical reasons to form hierarchical
structures. These are called hierarchical time series. In the retail industry
(see also §3.2.4), for example, individual sales of products at the bottom-
level of the hierarchy can be grouped in categories and families of related
products at increasing aggregation levels, with the total sales of the shop or
distribution centre at the top level (Pennings and van Dalen, 2017; Oliveira
and Ramos, 2019; Villegas and Pedregal, 2018). Similarly, cross-sectional
hierarchies can be used for spatial aggregation to help model housing prices
or traffic in transportation networks, or otherwise formed geographical
demarcations (for example, Athanasopoulos et al., 2009; Kourentzes and
Athanasopoulos, 2019). The forecasts of hierarchical time series produced
independently of the hierarchical structure generally will not add up
according to the aggregation constrains of the hierarchy, i.e., they are not
coherent. Therefore, hierarchical forecasting methods that generate coherent
forecasts should be considered to allow appropriate decision-making at the
different levels. Actually, by taking advantage of the relationships between
the series across all levels these methods have shown to improve forecast
accuracy (Athanasopoulos et al., 2009; Shang and Hyndman, 2017; Yagli et al.,
2019). One of the main reasons behind this improved performance is that
forecast reconciliation is effectively a special case of forecast combinations
(Hollyman et al., 2021); see also §2.6.

The most common approaches to hierarchical forecasting are bottom-up and top-
down. In the bottom-up approach forecasts for each time series at the bottom-
level are first produced and then these are added up to obtain forecasts for
all other series at the hierarchy (Dunn et al., 1976). Since forecasts are
obtained at the bottom-level no information is lost due to aggregation. In the
top-down approach forecasts for the top-level series are first generated and
then these are disaggregated generally using historical proportions to obtain
forecasts for the bottom-level series, which are then aggregated (Gross and
Sohl, 1990). Hyndman et al. (2011) claim that this approach introduces bias to
the forecasts, however Hollyman et al. (2021) showed that it is possible to
calculate unbiased top-down forecasts.

Recent research on hierarchical forecasting tackles the problem using a two-
stage approach. Forecasts for the series at all levels of the hierarchy are
first obtained independently without considering any aggregation constrains
(we refer to these as base forecasts). Then, base forecasts are adjusted so
that they become coherent (we refer to these as reconciled forecasts). This
adjustment is achieved by a matrix that maps the base forecasts into new
bottom-level forecasts which are then added up (Hyndman et al., 2011).

Wickramasuriya et al. (2019) found the optimal solution for this matrix, which
minimises the trace of the covariance matrix of the reconciled forecast errors
(hence MinT reconciliation). This optimal solution is based on the covariance
matrix of the base forecast errors which incorporates the correlation
structure of the hierarchy. Wickramasuriya et al. (2019) presented several
alternative estimators for this covariance matrix: (i) proportional to the
identity which is optimal only when base forecast errors are uncorrelated and
equivariant (referred to as OLS), (ii) proportional to the sample covariance
estimator of the in-sample one-step-ahead base forecast errors with off-
diagonal elements null accounts for the differences in scale between the
levels of the hierarchy (referred to as WLS), (iii) proportional to the
previous estimator unrestricted also accounts for the relationships between
the series (referred to as MinT-Sample), and (iv) proportional to a shrinkage
estimator based on the two previous estimators, parameterising the shrinkage
in terms of variances and correlations, accounts for the correlation across
levels (referred as MinT-Shrink). Other researchers focus on simple (equal-
weighted) combinations of the forecasts produced at different hierarchical
levels (Abouarghoub et al., 2018; Hollyman et al., 2021). Pritularga et al.
(2021) showed that more complex reconciliation schemes result in more
variability in the forecasts, due to the estimation of the elements in the
covariance matrix, or the implicit combination weights. They provide
approximations for the covariance matrix that balance this estimation
uncertainty with the benefits of more finely tuned weights.

More recently these techniques were extended to probabilistic forecasting (Ben
Taieb et al., 2020). When base forecasts are probabilistic forecasts
characterised by elliptical distributions, Panagiotelis et al. (2020) showed
that reconciled probabilistic forecasts also elliptical can be obtained
analytically. When it is not reasonable to assume elliptical distributions, a
non-parametric approach based on bootstrapping in-sample errors can be used.

####  2.10.2 Temporal aggregation727272This subsection was written by Fotios
Petropoulos.

Temporal aggregation is the transformation of a time series from one frequency
to another of lower frequency. As an example, a time series of length $n$ that
is originally sampled at a monthly frequency can be transformed to a quarterly
series of length $n/3$ by using equally-sized time buckets of three periods
each. It is usually applied in an non-overlapping manner, but overlapping
aggregation can also be considered. The latter is preferred in the case when
the original series is short, but has the disadvantage of applying lower
weights on the few first and last observations of the series and introducing
autocorrelations (Boylan and Babai, 2016, and §2.8.2).

Temporal aggregation is appealing as it allows to investigate the original
series through different lenses. By changing the original frequency of the
data, the apparent series characteristics also change. In the case of slow-
moving series, temporal aggregation leads to decrease of intermittence
(Nikolopoulos et al., 2011, see also §2.8). In the case of fast-moving series,
higher levels of aggregation (i.e., lower frequencies) allow for better
modelling of trend patterns, while lower aggregation levels (i.e., higher
frequencies) are more suitable for capturing seasonal patterns (Spithourakis
et al., 2014; Kourentzes et al., 2014).

Research has found evidence of improved forecasting performance with temporal
aggregation for both slow (Nikolopoulos et al., 2011) and fast (Spithourakis
et al., 2011) moving time series. This led to characterise temporal
aggregation as a “self-improving mechanism”. The good performance of temporal
aggregation was reconfirmed by Babai et al. (2012), who focused on its utility
performance rather than the forecast error. However, one challenge with single
levels of aggregation is the choice of a suitable aggregation level for each
series (Kourentzes et al., 2017).

Instead of focusing on a single aggregation level, Andrawis et al. (2011),
Kourentzes et al. (2014), Petropoulos and Kourentzes (2014), and Petropoulos
and Kourentzes (2015) suggested the use of multiple levels of aggregation,
usually abbreviated as MTA (multiple temporal aggregation). This not only
tackles the need to select a single aggregation level, but also partly
addresses the issue of model uncertainty, instead of relying on model
selection and parametrisation at a single aggregation level. Using this
property, Kourentzes et al. (2017) showed that MTA will typically lead to more
accurate forecasts, even if in theory suboptimal. Different frequencies allow
for better identification of different series patterns, so it is intuitive to
consider multiple temporal levels and benefit from the subsequent forecast
combination across frequencies. Kourentzes and Petropoulos (2016) showed how
multiple temporal aggregation can be extended to incorporate exogenous
variables (see also §2.2.5). However, forecasting at a single level of
aggregation can still result in better performance when the seasonal pattern
is strong (Spiliotis et al., 2019b, 2020c).

Athanasopoulos et al. (2017) expressed multiple temporal aggregation within
the hierarchical forecasting framework (see §2.10.1) using the term “temporal
hierarchies”. Temporal hierarchies allow for the application of established
hierarchical reconciliation approaches directly to the temporal dimension.
Jeon et al. (2019) show how temporal hierarchies can be used to obtain
reconciled probabilistic forecasts, while Spiliotis et al. (2019b) explored
empirical bias-adjustment strategies and a strategy to avoid excessive
seasonal shrinkage. Nystrup et al. (2020) proposed estimators for temporal
hierarchies suitable to account for autocorrelation in the data. Finally,
Kourentzes and Athanasopoulos (2020) applied temporal hierarchies on
intermittent data, and showed that higher aggregation levels may offer
structural information which can improve the quality of the forecasts.

####  2.10.3 Cross-temporal hierarchies737373This subsection was written by
Nikolaos Kourentzes.

In the last two subsections (§2.10.1 and §2.10.2), we saw two complimentary
hierarchical structures, cross-sectional and temporal. Although the machinery
behind both approaches is similar, often relying on the hierarchical framework
by Hyndman et al. (2011) and Athanasopoulos et al. (2009), and work that
followed from these (particularly Wickramasuriya et al., 2019), they address
different forecasting problems. Cross-sectional hierarchies change the unit of
analysis but are fixed in the period of analysis. For example, a manufacturer
may operate using a hierarchy across products. The different nodes in the
hierarchy will correspond to different products, product groups, super-groups,
and so on, but will all refer to the same period, for example, a specific
week. Temporal hierarchies do the opposite, where the unit of analysis is
fixed, but the period is not. For example, we may look at the sales of a
specific Stock Keeping Unit (SKU) at a daily, weekly, monthly, quarterly and
annual levels. However, one can argue that having annual forecasts at the SKU
level may not be useful. Similarly, having aggregate sales across an
organisation at a weekly frequency is also of little value.

In connecting these to organisational decisions, we can observe that there is
only a minority of problems that either cross-sectional or temporal
hierarchies are natural, as typically decisions can differ across both the
unit and the period (planning horizon) of analysis. In the latter case, both
hierarchical approaches are more akin to statistical devices that can improve
forecast accuracy through the use of forecast combinations, rather than
satisfy the motivating argument behind hierarchical forecasting that is to
provide coherent predictions for decisions at different levels of the
hierarchy.

Cross-temporal hierarchies attempt to overcome this limitation, providing
coherent forecasts across all units and periods of analysis, and therefore a
common outlook for the future across decision-makers at different functions
and levels within an organisation. The literature remains sparse on how to
construct cross-temporal forecasts, as the size of the hierarchy can easily
become problematic. Kourentzes and Athanasopoulos (2019) propose a heuristic
approach to overcome the ensuing estimation issues. The approach works by
compartmentalising the estimation. First, they obtain estimates of the cross-
sectional reconciliation weights for each temporal level of the hierarchy.
Then, these are combined across temporal levels, to a unique set that
satisfies all coherency constraints. Using these combined weights, they obtain
the reconciled bottom level forecasts, which can be aggregated as needed.
Although they recognise that their approach can result in suboptimal results
in terms of reconciliation errors, it guarantees coherent forecasts. Cross-
temporal forecasts are more accurate than either temporal or cross-sectional
hierarchical forecasts and provide a holistic view of the future across all
planning levels and demarcations. Spiliotis et al. (2020c) also identify the
problem, however, they do not focus on the coherency of forecasts and propose
a sequential reconciliation across the two dimensions. This is shown to again
be beneficial, but it does not achieve coherency. Arguably one can adapt the
iterative correction algorithm by Kourentzes and Athanasopoulos (2020) to
enforce coherency in this approach as well.

####  2.10.4 Ecological inference forecasting747474This subsection was written
by Jose M. Pavía.

Ecological inference forecasting (EIF) aims to predict the inner-cells values
of a set of contingency tables when only the margins are known. It defines a
fundamental problem in disciplines such as political science, sociology and
epidemiology (Salway and Wakefield, 2004). Cleave et al. (1995), Greiner
(2007) and Pavia et al. (2009) describe other areas of application. The
fundamental difficulty of EIF lies in the fact that this is a problem with
more unknowns than observations, giving rise to concerns over identifiability
and indeterminacy: many sets of substantively different internal cell counts
are consistent with a given marginal table. To overcome this issue, a
similarity hypothesis (and, sometimes, the use of covariates) is routinely
assumed. The basic hypothesis considers that either conditional row
(underlying) probabilities or fractions are similar (related) among
contingency tables (Greiner and Quinn, 2010). The covariations among row and
column margins of the different tables are then used to learn about the
internal cells.

The above hypothesis is not a cure-all to the main drawback of this approach.
EIF is exposed to the so-called ecological fallacy (Robinson, 1950): the
presence of inconsistencies in correlations and association measures across
different levels of aggregation. This is closely related to the well-known
Simpson’s Paradox (Simpson, 1951). In this setting, the ecological fallacy is
manifested through aggregation bias (Wakefield, 2004) due to contextual
effects and/or spatial autocorrelation (Achen and Phillips Shively, 1995).
This has led many authors to disqualify ecological inference forecasts (see,
for example, Freedman et al., 1998; Tam Cho, 1998; Anselin and Tam Cho, 2002;
Herron and Shotts, 2004) and many others to study under which circumstances
ecological inference predictions would be reliable (Firebaugh, 1978; Gelman et
al., 2001; Forcina and Pellegrino, 2019; Guseo, 2010). Despite the criticisms,
many algorithms for solving the EIF problem can be found in the literature,
mainly from the ecological regression and mathematical programming frameworks
(some of them available in functions of the R statistical software).

The ecological regression literature has been prolific since the seminal
papers of Goodman (1953, 1959) and Duncan and Davis (1953) and is undergoing a
renaissance after King (1997): new methods generalised from $2\times 2$ tables
to $R\times C$ tables have been proposed (King et al., 1999; Rosen et al.,
2001), the geographical dimension of the data is being explicitly considered
(Calvo and Escolar, 2003; Puig and Ginebra, 2015), and new procedures
combining aggregated and individual level data, including exit polls (see also
§3.8.5), are introduced (Glynn and Wakefield, 2010; Greiner and Quinn, 2010;
Klima et al., 2019). See King et al. (2004) for a wide survey and Klima et al.
(2016) and Plescia and De Sio (2018) for an extensive evaluation of
procedures. In mathematical programming exact and inequality constraints for
the inner-cell values are incorporated in a natural way. Hence, this approach
has shown itself to be a proper framework for generating ecological inference
forecasts. The proposals from this approach can be traced back to Hawkes
(1969) and Irwin and Meeter (1969). After them, some key references include
McCarthy and Ryan (1977), Tziafetas (1986), Corominas et al. (2015), Romero et
al. (2020), and Pavía and Romero (2021). Solutions based on other strategies,
for instance, entropy maximization, have been also suggested (see, for
example, Johnston and Pattie, 2000; Bernardini Papalia and Fernandez Vazquez,
2020).

###  2.11 Forecasting with judgment

####  2.11.1 Judgmental forecasting757575This subsection was written by Nigel
Harvey.

People may use judgment alone to make forecasts or they may use it in
combination with statistical methods. Here the focus is on pure judgmental
forecasting (for judgmental adjustments, see §2.11.2). Different types of
judgment heuristic (mental ‘rules of thumb’) can be used to make forecasts.
The heuristic used depends on the nature of the information available to the
forecaster (Harvey, 2007).

Consider cases where the only relevant information is held in the forecaster’s
memory. For example, someone might be asked whether Manchester United or
Burnley will win next week’s match. Here one memory-based heuristic that might
be applicable is the recognition heuristic: if one recognises one object but
not the other, then one should infer that the recognised object has higher
value (Goldstein and Gigerenzer, 2002). In the above example, most people who
recognise just one of the teams would be likely to make a correct forecast
that Manchester United will win (Ayton et al., 2011). The availability
heuristic is another memory-based heuristic that may be applicable: objects
that are most easily brought to mind are those which are more likely. Thus, if
we are asked which team is likely to come top of the premier league, we would
say Manchester United if that is the one that most easily comes to mind. The
availability heuristic is often effective because more likely events are
encountered more often and more recently and are hence better remembered.
However, it can be disrupted by, for example, greater media coverage of more
unlikely (and hence more interesting) events.

Consider next cases in which forecasters possess information about values of
one or more variables correlated with the variable to be forecast. For
example, teachers may wish to forecast the grades of their students in a final
examination on the basis of past records of various other measures. Kahneman
and Tversky (1973) suggested that people use the representativeness heuristic
to deal with this type of situation. Forecasters first select a variable that
they think is able to represent the one that must be predicted. For example, a
teacher may consider that frequency in attending voluntary revision classes
represents a student’s ability in the final examination. Thus, if a student
attended 15 of the 20 revision classes, they are likely to obtain 75% in the
final examination.

Finally, consider situations in which people forecast future values of a
variable on the basis of a record of previous values of that variable. There
is some evidence that, when forecasting from time series, people use anchor-
and-adjustment heuristics (Hogarth and Makridakis, 1981; Lawrence and
O’Connor, 1992). For example, (i) when forecasting from an upward trended
series, they anchor on the last data point and then make an upward adjustment
to take the trend into account and (ii) when forecasting from an untrended
series containing autocorrelation, they anchor on the last data point and make
an adjustment towards the mean to take the autocorrelation into account.

Kahneman (2011) and others have divided cognitive processes into those which
are intuitive (System 1) and those which are deliberative (System 2). We have
discussed only intuitive processes underlying judgmental forecasting
(Gigerenzer, 2007). However, they can be supplemented by deliberative (System
2) processes (Theocharis and Harvey, 2019) in some circumstances.

####  2.11.2 Judgmental adjustments of computer-based forecasts767676This
subsection was written by Paul Goodwin.

Judgmental adjustments to algorithmic computer-based forecasts can enhance
accuracy by incorporating important extra information into forecasts (McNees,
1990; Fahimnia et al., 2020; Perera et al., 2019). However, cognitive factors
(see, for example, §2.11.1), and motivational biases (see §3.2.2), can lead to
the inefficient use of information (Fildes et al., 2019a), unwarranted
adjustments and reductions in accuracy (Fildes et al., 2009; Franses and
Legerstee, 2009a).

People may ‘egocentrically discount’ a computer’s forecasts when its rationale
is less clear than their own reasoning (Bonaccio and Dalal, 2006). They can
also be less tolerant of errors made by algorithms than those made by humans
(Önkal et al., 2009; Dietvorst et al., 2015, and §3.7.4). The random errors
associated with algorithmic forecasts, and the salience of rare large errors,
can therefore lead to an unjustified loss of trust in computer forecasts.
Adjustments may also give forecasters a sense of ownership of forecasts or be
used to justify their role (Önkal and Gönül, 2005).

Computer-based forecasts are designed to filter randomness from time-series.
In contrast, humans tend to perceive non-existent systematic patterns in
random movements (O’Connor et al., 1993; Reimers and Harvey, 2011, and §3.7.3)
and apply adjustments to reflect them. This can be exacerbated by the
narrative fallacy (Taleb, 2008), where people invent stories to explain these
random movements, and hindsight bias (Fischhoff, 2007), where they believe, in
retrospect, that these movements were predictable. Recent random movements,
and events, are particularly likely to attract undue attention, so long-run
patterns identified by the computer are given insufficient weight (Bolger and
Harvey, 1993). Damaging interventions are also probable when they result from
political interference (Oliva and Watson, 2009) or optimism bias (Fildes et
al., 2009), or when they reflect information already factored into the
computer’s forecast, leading to double counting (Van den Broeke et al., 2019).

How can interventions be limited to occasions when they are likely to improve
accuracy? Requiring people to document reasons justifying adjustments can
reduce gratuitous interventions (Goodwin, 2000b). Explaining the rationale
underlying statistical forecasts also improved adjustment behaviour when
series had a simple underlying pattern in a study by Goodwin and Fildes
(1999). However, providing guidance on when to adjust was ineffective in an
experiment conducted by Goodwin et al. (2011), as was a restriction preventing
people from making small adjustments.

When determining the size of adjustments required, decomposing the judgment
into a set of easier tasks improved accuracy in a study by Webby et al.
(2005). Providing a database of past outcomes that occurred in circumstances
analogous to those expected in the forecast period also improved adjustments
in a study by Lee et al. (2007). Outcome feedback, where the forecaster is
informed of the most recent outcome is unlikely to be useful since it contains
noise and exacerbates the tendency to over-focus on recent events (Goodwin and
Fildes, 1999; Petropoulos et al., 2016). However, feedback on biases in
adjustments over several recent periods may improve judgments (Petropoulos et
al., 2017). Feedback will be less useful where interventions are rare so there
is insufficient data to assess performance.

The evidence for this section is largely based on laboratory-based studies of
adjustment behaviour. §3.7.3 gives details of research into forecast
adjustment in practice and discusses the role of forecasting support systems
in improving the effectiveness of judgmental adjustments.

####  2.11.3 Judgmental model selection777777This subsection was written by
Shari De Baets.

Forecasters – practitioners and researchers alike – use Forecast Support
Systems (FSS) in order to perform their forecasting tasks. Usually, such an
FSS allows the forecaster to load their historical data and they can then
apply many different types of forecasting techniques to the selected data. The
idea is that the forecaster selects the method which leads to highest forecast
accuracy. Yet, there is no universally ‘best’ method, as it depends on the
data that is being forecasted (see also §2.5.4). Thus, selection is important
in achieving high accuracy. But how does this selection occur?

Research on judgmental selection of statistical forecasts is limited in
quantity. Lawrence et al. (2002) found that participants were not very adept
at selecting good forecasting algorithms from a range offered to them by an
FSS and had higher error than those who were presented with the optimal
algorithm by an FSS. Petropoulos et al. (2018b) compared judgmental selection
of forecasting algorithms with automatic selection based on predetermined
information criteria. They found that judgmental selection was better than
automatic selection at avoiding the ‘worst’ models, but that automatic
selection was superior at choosing the ‘best’ ones. In the end, overall
accuracy of judgmental selection was better than that of algorithmic
selection. If their experiment had included more variation of the data
(trends, fractals, different autoregressive factors) and variation of proposed
models, this could possibly have led to better algorithmic than judgmental
performance (Harvey, 2019). Time series that are more complex will place a
higher cognitive load on judgmental selection. This was confirmed in a study
by Han et al. (2019), who used an electroencephalogram (EEG) for the
comparison of judgmental forecast selection versus (judgmental) pattern
identification. They found that pattern identification outperformed forecast
selection, as the latter required a higher cognitive load, which in turn led
to a lower forecasting accuracy.

It is likely that, in practice, judgmental selection is much more common than
automatic selection. This preference for human judgment over advice from an
algorithm has been shown in an experiment by Önkal et al. (2009). But how apt
are forecasters in distinguishing ‘good’ models from ‘bad’ models? This was
investigated by De Baets and Harvey (2020) in an experiment. People were asked
to select the best performing model out of a choice of two different qualities
(accuracies) of models (different combinations of good versus medium versus
bad). People’s choice outperformed forecasts made by averaging the model
outputs, lending credence to the views of Fifić and Gigerenzer (2014). The
performance of the participants improved with a larger difference in quality
between models and a lower level of noise in the data series. In a second
experiment, De Baets and Harvey (2020) found that participants adjusted more
towards the advice of what they perceived to be a good quality model than a
medium or bad quality one.

Importantly, in selecting an algorithm and seeing it err, people are quick to
abandon it. This phenomenon is known as ‘algorithm aversion’ (Dietvorst et
al., 2015, see also §2.11.6) and is due to a ‘perfection schema’ we have in
our heads where algorithms are concerned (Madhavan and Wiegmann, 2007). We do
not expect them to ‘fail’ and thus react strongly when they do. While a model
may not perform as it should for a particular dataset and may thus elicit
algorithm aversion for that particular method, one should not abandon it for
all datasets and future forecasts.

####  2.11.4 Panels of experts787878This subsection was written by Konstantia
Litsiou.

Panels of experts are often used in practice to produce judgmental forecasts
(see, for example, §3.2.6 and §3.8.5). This is especially true in cases with
limited available quantitative data and with the level of uncertainty being
very high. In this section, three methods for eliciting judgmental forecasts
from panels of experts are presented: the Delphi method, interaction groups
(IG), and structured analogies (SA).

The Delphi method is centred around organising and structuring group
communication (Rao et al., 2010), which aims to achieve a convergence of
opinion on a specific real-world issue. It is a multiple-round survey in which
experts participate anonymously to provide their forecasts and feedback (Rowe
and Wright, 2001). At the end of each round, the facilitator collects and
prepares statistical summaries of the panel of experts’ forecasts. These
summaries are presented as feedback to the group, and may be used towards
revising their forecasts. This loop continues until a consensus is reached, or
the experts in the panel are not willing to revise their forecasts further. In
some implementations of the Delphi method, justification of extreme positions
(forecasts) is also part of the (anonymous) feedback process. The Delphi
method results in a more accurate outcome in the decision-making process
(Dalkey, 1969; Steurer, 2011). Rowe and Wright (2001) mentioned that, by
adopting the Delphi method, groups of individuals can produce more accurate
forecasts than simply using unstructured methods. A drawback of the Delphi
method is the additional cost associated with the need to run multiple rounds,
extending the forecasting process as well as increasing the potential drop-out
rates. On the other hand, the anonymity in the Delphi method eliminates issues
such as groupthink and the ‘dominant personalities’ effects (Van de Ven and
Delbeco, 1971).

The IG method suggests that the members of the panel of experts actively
interact and debate their points to the extent they have to reach an agreement
on a common forecast (Litsiou et al., 2019). Sniezek and Henry (1989) found
that members of interacting groups provide more accurate judgments compared to
individuals. However, there is mixed evidence about the forecasting potential
of IG (Scott Armstrong, 2006; Boje and Murnighan, 1982; Graefe and Armstrong,
2011). Besides, the need for arranging and facilitating meetings for the IG
makes it a less attractive option.

Another popular approach to judgmental forecasting using panels of experts is
SA, which refers to the recollection of past experiences and the use analogies
(Green and Armstrong, 2007). In the SA method, the facilitator assembles a
panel of experts. The experts are asked to recall and provide descriptions,
forecasts, and similarities/differences for cases analogous to the target
situation, as well as a similarity ranking for each of these analogous cases.
The facilitator gathers the lists of the analogies provided by the experts,
and prepares summaries, usually using weighted averages of the recalled cases
based on their similarity to the target situation (see also §2.6.4). Semi-
structured analogies (sSA) have also been proposed in the literature, where
the experts are asked to provide a final forecasts based on the analogous
cases they recalled, which essentially reduces the load for the facilitator
(Nikolopoulos et al., 2015). Nikolopoulos et al. (2015) supported that the use
of SA and IG could result to forecasts that are 50% more accurate compared to
unstructured methods (such as unaided judgment). One common caveat of using
panels of experts is the difficulty to identify who a real expert is. Engaging
experts with high level of experience, and encouraging the interaction of
experts are also supported by Armstrong and Green (2018).

####  2.11.5 Scenarios and judgmental forecasting797979This subsection was
written by M. Sinan Gönül.

Scenarios provide exhilarating narratives about conceivable futures that are
likely to occur. Through such depictions they broaden the perspectives of
decision makers and act as mental stimulants to think about alternatives.
Scenarios enhance information sharing and provide capable tools for
communication within organisations. By virtue of these merits, they have been
widely used in corporate planning and strategy setting since 1960’s (Godet,
1982; Schoemaker, 1991; Wright and Goodwin, 1999, 2009; Goodwin and Wright,
2010). Even though utilisation of scenarios as decision advice to judgmental
forecasting has been proposed earlier (Schnaars and Topol, 1987; Bunn and
Salo, 1993), the research within this domain remained limited until recently
when the interest in the subject has rekindled (Önkal et al., 2013; Goodwin et
al., 2019b; Wicke et al., 2019).

The recent research has used behavioural experimentation to examine various
features of scenarios and their interactions with judgmental adjustments (see
§2.11.2) of model-based forecasts. Önkal et al. (2013) explored the ‘content’
effects of scenarios where through the narration either a bleak/negative
future (a pessimistic scenario) or a bright/positive future (an optimistic
scenario) was portrayed. On a demand forecasting context for mobile phones,
the participants first received time-series data, model-based forecasts and
then asked to generate point and interval forecasts as well as provide a
confidence measure. With respect to the existence of scenarios, there were
four conditions where the participants may receive: (i) no scenarios, (ii)
optimistic scenarios, (iii) pessimistic scenarios, and (iv) both scenarios.
Findings indicate that decision makers respond differently to optimistic and
pessimistic scenarios. Receiving optimistic scenarios resulted in making
larger adjustments to the model-based forecasts. At the same time, led to an
increased confidence of the participants in their predictions. On the other
hand, participants who received negative scenarios tend to lower their
predictions the most among the four groups. An intriguing finding was the
balancing effect of scenarios on the interval forecast symmetry. The lower
interval bounds were adjusted upwards the most towards the centre-point of the
interval (i.e., model-based predictions) when optimistic scenarios were
received. Similarly, the upper bounds were adjusted downwards the most towards
the centre-point of the interval in the presence of pessimistic scenarios.

The prospects of receiving a single scenario versus multiple scenarios were
further explored in Goodwin et al. (2019b). The researchers investigated
whether assimilation or contrast effects will occur when decision makers see
optimistic (pessimistic) forecasts followed by pessimistic (optimistic) ones
compared against receiving a single scenario in solitude. In case of
assimilation, a scenario presenting an opposing world view with the initial
one would cause adjustments in the opposite direction creating an offset
effect. On the other hand, in case of contrast, the forecasts generated after
the initial scenarios would be adjusted to more extremes when an opposing
scenario is seen. In two experiments conducted in different contexts the
researchers found resilient evidence for contrast effects taking place.
Interestingly, seeing an opposing scenario also increased the confidence of
the forecasters in their initial predictions.

In terms of the effects of scenario presence on the forecasting performance,
however, the experimental evidence indicates the benefits are only
circumstantial. Goodwin et al. (2019a) found that providing scenarios worsened
forecast accuracy and shifted the resultant production order decisions further
away from optimality. Despite this performance controversy, the decision
makers express their fondness in receiving scenarios and belief in their
merits (Önkal et al., 2013; Goodwin et al., 2019b). Therefore, we need more
tailored research on scenarios and judgmental forecasting to reveal the
conditions when scenarios can provide significant improvements to the
forecasting accuracy.

####  2.11.6 Trusting model and expert forecasts808080This subsection was
written by Dilek Önkal.

Defined as “firm belief in the reliability, truth, and ability of
someone/something” (Oxford English Dictionary), trust entails accepting
vulnerability and risk (Rousseau et al., 1998). Given that forecasts are
altered or even discarded when distrusted by users, examining trust is a
central theme for both forecasting theory and practice.

Studies examining individual’s trust in model versus expert forecasts show
that individuals often distrust algorithms (Meehl, 2013; Burton et al., 2020)
and place higher trust on human advice (Diab et al., 2011; Eastwood et al.,
2012, but also §2.11.2, §2.11.3, and §3.7.4). We live in an era where we are
bombarded with news about how algorithms get it wrong, ranging from COVID-19
forecasts affecting lockdown decisions to algorithmic grade predictions
affecting university admissions. Individuals appear to prefer forecasts from
humans over those from statistical algorithms even when those forecasts are
identical (Önkal et al., 2009). Furthermore, they lose trust in algorithms
quicker when they see forecast errors (Dietvorst et al., 2015; Prahl and Van
Swol, 2017). Such ‘algorithm aversion’ and error intolerance is reduced when
users have opportunity to adjust the forecasting outcome, irrespective of the
extent of modification allowed (Dietvorst et al., 2018). Feedback appears to
improve trust, with individuals placing higher trust in algorithms if they can
understand them (Seong and Bisantz, 2008). Overuse of technical language may
reduce understanding of the forecast/advice, in turn affecting perceptions of
expertise and trustworthiness (Joiner et al., 2002). Explanations can be
helpful (Goodwin et al., 2013b), with their packaging affecting judgments of
trustworthiness (Elsbach and Elofson, 2000). Algorithmic appreciation appears
to easily fade with forecasting expertise (Logg et al., 2019), emphasising the
importance of debiasing against overconfidence and anchoring on one’s own
predictions.

Trusting experts also presents challenges (Hendriks et al., 2015; Hertzum,
2014; Maister et al., 2012). Expert forecasts are typically seen as
predisposed to group-based preconceptions (Brennan, 2020; Vermue et al.,
2018), along with contextual and motivational biases (Burgman, 2016).
Misinformed expectations, distorted exposures to ‘forecast failures’, and
over-reliance on one’s own judgments may all contribute to distrusting experts
as well as algorithms.

Credibility of forecast source is an important determinant in gaining trust
(Önkal et al., 2019). Studies show that the perceived credibility of system
forecasts affects expert forecasters’ behaviours and trust (Alvarado-Valencia
and Barrero, 2014), while providing information on limitations of such
algorithmic forecasts may reduce biases (Alvarado-Valencia et al., 2017).
Previous experience with the source appears to be key to assessing credibility
(Hertzum, 2002) and trust (Cross and Sproull, 2004). Such ‘experienced’
credibility appears to be more influential on users’ acceptance of given
forecasts as opposed to ‘presumed’ credibility (Önkal et al., 2017). Source
credibility can be revised when forecast (in)accuracy is encountered
repetitively (Jiang et al., 1996), with forecaster and user confidence playing
key roles (Sah et al., 2013).

Trust is critical for forecasting efforts to be translated into sound
decisions (Choi et al., 2020; Özer et al., 2011). Further work on fostering
trust in individual/collaborative forecasting will benefit from how trusted
experts and models are selected and combined to enhance decision-making.

###  2.12 Evaluation, validation, and calibration

####  2.12.1 Benchmarking818181This subsection was written by Anastasios
Panagiotelis.

When a new forecasting model or methodology is proposed, it is common for its
performance to be benchmarked according to some measure of forecast accuracy
against other forecasting methods using a sub-sample of some particular time
series. In this process, there is the risk that either the measures of
accuracy, competing forecasting methods or test data, are chosen in a way that
exaggerates the benefits of a new method. This possibility is only exacerbated
by the phenomenon of publication bias (Dickersin, 1990).

A rigorous approach to benchmarking new forecasting methods should follow the
following principles:

  1. 1.

New methods should always be compared to a larger number of suitable benchmark
methods. These should at a minimum include naïve methods such as a random walk
and also popular general purpose forecasting algorithms such as ARIMA models,
Exponential Smoothing, Holt Winters and the Theta method (see §2.3 and
references therein).

  2. 2.

Forecasts should be evaluated using a diverse set of error metrics for point,
interval and probabilistic forecasts (see §2.12.2). Where the forecasting
problem at hand should be tailored to a specific problem, then appropriate
measures of forecast accuracy must be used. As an example, the literature on
Value at Risk forecasting has developed a number of backtesting measures for
evaluating the quality of quantile forecasts (see Zhang and Nadarajah, 2018,
and references therein).

  3. 3.

Testing should be carried out to discern whether differences between
forecasting methods are statistically significant. For discussion see §2.12.6.
However, there should also be a greater awareness of the debate around the use
of hypothesis testing both in forecasting (Armstrong, 2007) and more generally
in statistics (Wasserstein and Lazar, 2016).

  4. 4.

Sample sizes for rolling windows should be chosen with reference to the latest
literature on rolling window choice (see Inoue et al., 2017, and references
therein).

  5. 5.

All code used to implement and benchmark new forecasting methods should, where
possible, be written in open source programming languages (such as C, Python
and R). This is to ensure replicability of results (for more on the
replicablity crisis in research see Peng, 2015, and references therein)

  6. 6.

Methods should be applied to appropriate benchmark datasets.

Regarding the last of these points there are some examples in specific fields,
of datasets that already play a de facto role as benchmarks. In macroeconomic
forecasting, the U.S. dataset of (Stock and Watson, 2012, see §2.7.1) is often
used to evaluate forecasting methods that exploit a large number of
predictors, with Forni et al. (2003) and Panagiotelis et al. (2019) having
constructed similar datasets for the EU and Australia respectively. In the
field of energy, the GEFCom data (Hong et al., 2016) discussed in §3.4.3 and
the IEEE 118 Bus Test Case data (Peña et al., 2018) are often used as
benchmarks. Finally, the success of the M Forecasting competitions (Makridakis
et al., 2020c) provide a benchmark dataset for general forecasting
methodologies (see §2.12.7 and references therein).

A recent trend that has great future potential is the publication of websites
that demonstrate the efficacy of different forecasting methods on real data.
The Covid-19 Forecast Hub828282https://viz.covid19forecasthub.org/ and the
Business Forecast Lab838383https://business-forecast-lab.com/ provide notable
examples in the fields of epidemiology and macroeconomics and business
respectively.

####  2.12.2 Point, interval, and pHDR forecast error measures848484This
subsection was written by Stephan Kolassa.

Point forecasts are single number forecasts for an unknown future quantity
also given by a single number. Interval forecasts take the form of two point
forecasts, an upper and a lower limit. Finally, a less common type of forecast
would be a predictive Highest Density Region (pHDR), i.e., an HDR (Hyndman,
1996) for the conditional density of the future observable. pHDRs would be
interesting for multimodal (possibly implicit) predictive densities, e.g., in
scenario planning. Once we have observed the corresponding realisation, we can
evaluate our point, interval and pHDR forecasts.

There are many common point forecast error measures (PFEMs), e.g., the mean
squared error (MSE), mean absolute error (MAE), mean absolute scaled error
(MASE), mean absolute percentage error (MAPE) or many others (see section 3.4
in Hyndman and Athanasopoulos, 2018). Which one is most appropriate for our
situation, or should we even use multiple different PFEMs?

There are many common point forecast error measures (PFEMs), e.g., the MSE,
MAE, MASE, (s)MAPE, the quantile score or pinball loss or many others (e.g.,
sections 5.8 and 5.9 in Hyndman and Athanasopoulos, 2021). Assuming $n$
historical periods with observations $y_{1},\dots,y_{n}$ and a forecasting
horizon $H$ with observations $y_{n+1},\dots,y_{n+H}$ and point forecasts
$f_{n+1},\dots,f_{n+H}$, we have:

$\displaystyle\text{MSE}=\sum_{t=n+1}^{n+H}(y_{t}-f_{t})^{2},\quad\text{MAE}=\sum_{t=n+1}^{n+H}|y_{t}-f_{t}|,\quad\text{MASE}=\frac{\sum_{t=n+1}^{n+H}|y_{t}-f_{t}|}{\sum_{t=2}^{n}|y_{t}-y_{t-1}|}$
$\displaystyle\text{MAPE}=\sum_{t=n+1}^{n+H}\frac{|y_{t}-f_{t}|}{y_{t}},\quad\text{sMAPE}=\sum_{t=n+1}^{n+H}\frac{|y_{t}-f_{t}|}{\frac{1}{2}(y_{t}+f_{t})}$
$\displaystyle
Q_{\alpha}=\sum_{t=n+1}^{n+H}(1-\alpha)(f_{t}-y_{t})1_{y_{t}<f_{t}}+\alpha(y_{t}-f_{t})1_{y_{t}\geq
f_{t}}.$

Let us take a step back. Assume we have a full density forecast and wish to
“condense” it to a point forecast that will minimise some PFEM in expectation.
The key observation is that different PFEMs will be minimised by different
point forecasts derived from the same density forecast (Kolassa, 2020b).

  * •

The MSE is minimised by the expectation.

  * •

The MAE and MASE are minimised by the median (Hanley et al., 2001).

  * •

The MAPE is minimised by the $(-1)$-median (Gneiting, 2011a, p. 752 with
$\beta=-1$).

  * •

The sMAPE is minimised by an unnamed functional that would need to be
minimised numerically (Gonçalves, 2015).

  * •

The hinge/tick/pinball $Q_{p}$ loss is minimised by the appropriate
$p$-quantile (Gneiting, 2011b).

  * •

In general, there is no loss function that is minimised by the mode (Heinrich,
2014).

We note that intermittent demand (see §2.8) poses specific challenges. On the
one hand, the MAPE is undefined if there are zeros in the actuals. On the
other hand, the point forecasts minimising different PFEMs will be very
different. For instance, the conditional median (minimising the MAE) may well
be a flat zero, while the conditional mean (minimising the MSE) will usually
be nonzero. Our forecasting algorithm may not output an explicit density
forecast. It is nevertheless imperative to think about which functional of the
implicit density we want to elicit (Gneiting, 2011a), and tailor our error
measure – and forecasting algorithm! – to it. It usually makes no sense to
evaluate a point forecast with multiple PFEMs (Kolassa, 2020b).

Interval forecasts can be specified in multiple ways. We can start with a
probability coverage and require two appropriate quantiles – e.g., we could
require a 2.5% and a 97.5% quantile forecast, yielding a symmetric or equal-
tailed 95% interval forecast. Interval forecasts ($\ell_{t},u_{t}$) of this
form can be evaluated by the interval score (Winkler, 1972; Brehmer and
Gneiting, 2020), a proper scoring rule (section 6.2 in Gneiting and Raftery,
2007):

$\text{IS}_{\alpha}=\sum_{t=n+1}^{n+H}(u_{t}-\ell_{t})+\frac{2}{\alpha}(\ell_{t}-y_{t})1_{y_{t}<\ell_{t}}+\frac{2}{\alpha}(y_{t}-u_{t})1_{y_{t}>u_{t}}.$
(3)

We can also use the hinge loss to evaluate the quantile forecasts separately.

Alternatively, we can require a shortest interval subject to a specified
coverage. This interval is not elicitable relative to practically relevant
classes of distributions (Brehmer and Gneiting, 2020; Fissler et al., 2020).

Yet another possibility is to maximise the interval forecast’s probability
coverage, subject to a maximum length $\ell$. This modal interval forecast
$(f_{t},f_{t}+\ell)$ is elicitable by an appropriate $\ell$-zero-one-loss
(Brehmer and Gneiting, 2020)

$L_{\ell}=\sum_{t=n+1}^{n+H}1_{f_{t}<y_{t}<f_{t}+\ell}=\\#\big{\\{}t\in
n+1,\dots,n+H\,\big{|}\,f_{t}<y_{t}<f_{t}+\ell\big{\\}}.$ (4)

The pHDR is not elicitable even for unimodal densities (Brehmer and Gneiting,
2020). In the multimodal case, the analysis is likely difficult. Nevertheless,
a variation of the Winkler score has been proposed to evaluate pHDRs on an ad
hoc basis (Hyndman, 2020). One could also compare the achieved to the nominal
coverage, e.g., using a binomial test – which disregards the volume of the
pHDR (Kolassa, 2020a).

In conclusion, there is a bewildering array of PFEMs, which require more
thought in choosing among than is obvious at first glance. The difficulties
involved in evaluating interval and pHDR forecasts motivate a stronger
emphasis on full density forecasts (cf. Askanazi et al., 2018, and §2.12.4).

####  2.12.3 Scoring expert forecasts858585This subsection was written by Yael
Grushka-Cockayne.

Evaluating forecasting capabilities can be a difficult task. One prominent way
to evaluate an expert’s forecast is to score the forecast once the realisation
of the uncertainty is known. Scoring forecasts using the outcome’s
realisations over multiple forecasts offers insights into an individual’s
expertise. Experts can also use scoring information to identify ways to
improve future forecasts. In addition, scoring rules and evaluation measures
can be designed to match decision-making problems, incentivising forecasts
that are most useful in a specific situation (Winkler et al., 2019).

Scoring rules were first suggested for evaluate meteorological forecasts in
work by Brier (1950). Scoring rules have since been used in a wide variety of
settings, such as business and other applications. When forecasting a discrete
uncertainty with only two possible outcomes (e.g., a loan with be defaulted on
or not, a customer will click on an ad or not), the Brier score assigns a
score of $-(1-p)^{2}$, where $p$ is the probability forecast reported that the
event will occurs. The greater the probability reported for an event that
occurs, the higher the score the forecast receives. Over multiple forecasts,
better forecasters will tend to have higher average Brier scores. For discrete
events with more than two outcomes, a logarithmic scoring rule can be used.

The scoring rules are attractive to managers in practice since they are
considered proper. Proper scoring rules (see also §2.12.4) incentivise honest
forecasts from the experts, even prior to knowing the realisation of an
uncertainty, since ex ante the expected score is maximised only when reported
probabilities equals true beliefs (Winkler et al., 1996; O’Hagan et al., 2006;
Bickel, 2007; Gneiting and Raftery, 2007; Merkle and Steyvers, 2013). Examples
of a scoring rule that is not proper yet still commonly used are the linear
score, which simply equals the reported probability or density for the actual
outcome, or the skill score, which is the percentage improvement of the Brier
score for the forecast relative to the Brier score of some base line naive
forecast (Winkler et al., 2019).

For forecasting continuous quantities, forecasts could be elicited by asking
for an expert’s quantile (or fractile) forecast rather than a probability
forecast. For instance, the 0.05, 0.25, 0.50, 0.75 and 0.95 quantiles are
often elicited in practice, and in some cases every 0.01 quantile, between 0-1
are elicited (e.g., the 2014 Global Energy Forecasting Competition, Hong et
al., 2016). Proper scoring rules for quantiles are developed in Jose and
Winkler (2009).

When forecasts are used for decision-making, it is beneficial if the scoring
rule used relates in some manner to the decision problem itself. In certain
settings, the connection of the scoring rule to the decision context is
straight forward. For example, Jose et al. (2008) develop scoring rules that
can be mapped to decision problems based on the decision maker’s utility
function. Johnstone et al. (2011) develop tailored scoring rules aligning the
interest of the forecaster and the decision maker. Grushka-Cockayne et al.
(2017b) link quantile scoring rules to business profit-sharing situations.

####  2.12.4 Evaluating probabilistic forecasts868686This subsection was
written by Florian Ziel.

Probabilistic forecasting is a term that is not strictly defined, but usually
refers to everything beyond point forecasting (Gneiting, 2011a). However, in
this section we consider only the evaluation of full predictive distributions
or equivalent characterisations. For the evaluation of prediction of
quantiles, intervals and related objects, see §2.12.2.

One crucial point for evaluating probabilistic forecasts is the reporting,
which is highly influenced from meteorologic communities. From the theoretical
point of view, we should always report the predicted cumulative distribution
function $\widehat{F}$ of our prediction target $F$. Alternatively for
continuous data, reporting the probability density function is a popular
choice. For univariate prediction problems a common alternative is to report
quantile forecast on a dense grid of probabilities, as it approximates the
full distribution (Hong et al., 2016). For multivariate forecasts, it seems to
become standard to report a large ensemble (a set of simulated
trajectories/paths) of the full predictive distribution. The reason is that
the reporting of a multivariate distribution (or an equivalent
characterisation) of sophisticated prediction models is often not feasible or
practicable, especially for non-parametric or copula-based forecasting
methods.

In general, suitable tools for forecasting evaluation are proper scoring rules
as they address calibration and sharpness simultaneously (Gneiting and
Raftery, 2007; Gneiting and Katzfuss, 2014). Preferably, we consider strictly
proper scoring rules which can identify the true predicted distribution among
a set of forecast candidates that contains the true model.

In the univariate case the theory is pretty much settled and there is quite
some consensus about the evaluation of probabilistic forecasts (Gneiting and
Katzfuss, 2014). The continuous ranked probability score (CRPS) and
logarithmic scores (log-score) are popular strictly proper scoring rules,
while the quadratic and pseudospherical score remain strictly proper
alternatives. The CRPS can be well approximated by averaging across quantile
forecasts on an equidistant grid of probabilities (Nowotarski and Weron,
2018).

For multivariate forecast evaluation the situation is more complicated and
many questions remain open (Gneiting and Raftery, 2007; Meng et al., 2020).
The multivariate version of the log-score is a strictly proper scoring rule,
but it requires the availability of a multivariate density forecast. This
makes it impracticable for many applications. Gneiting and Raftery (2007)
discuss the energy score, a multivariate generalisation of the CRPS, that is
strictly proper. Still, it took the energy score more than a decade to
increase its popularity in forecasting. A potential reason is the limited
simulation study of Pinson and Tastu (2013) that concludes that the energy
score can not discriminate well differences in the dependency structure. In
consequence other scoring rules were proposed in literature, e.g., the
variogram score (Scheuerer and Hamill, 2015) which is not strictly proper.
Ziel and Berk (2019) consider a strictly proper scoring method for continuous
variables using copula techniques. In contrast to Pinson and Tastu (2013),
recent studies (Ziel and Berk, 2019; Lerch et al., 2020) show that the energy
score discriminates well when used together with significance tests like the
Diebold-Mariano (DM) test. In general, we recommended scoring be applied with
reliability evaluation (see §2.12.5) and significance tests (see §2.12.6).
Additionally, if we want to learn about the performance of our forecasts it is
highly recommended to consider multiple scoring rules and evaluate on lower-
dimensional subspaces. For multivariate problems, this holds particularly for
the evaluation of univariate and bivariate marginal distributions.

####  2.12.5 Assessing the reliability of probabilistic forecasts878787This
subsection was written by Thordis Thorarinsdottir.

Probabilistic forecasts in the form of predictive distributions are central in
risk-based decision making where reliability, or calibration, is a necessary
condition for the optimal use and value of the forecast. A probabilistic
forecast is calibrated if the observation cannot be distinguished from a
random draw from the predictive distribution or, in the case of ensemble
forecasts, if the observation and the ensemble members look like random draws
from the same distribution. Additionally, to ensure their utility in decision
making, forecasts should be sharp, or specific, see §2.12.2 and §2.12.4 as
well as Gneiting et al. (2007).

In the univariate setting, several alternative notions of calibration exist
for both a single forecast (Gneiting et al., 2007; Tsyplakov, 2013) and a
group of forecasts (Strähl and Ziegel, 2017). The notion most commonly used in
applications is probabilistic calibration (Dawid, 1984); the forecast system
is probabilistically calibrated if the probability integral transform (PIT) of
a random observation, that is, the value of the predictive cumulative
distribution function in the observation, is uniformly distributed. If the
predictive distribution has a discrete component, a randomised version of the
PIT should be used (Gneiting and Ranjan, 2013).

Probabilistic calibration is assessed visually by plotting the histogram of
the PIT values over a test set. A calibrated forecast system will return a
uniform histogram, a $\cap$-shape indicates overdispersion and a $\cup$-shape
indicates underdispersion, while a systematic bias results in a biased
histogram (e.g. Thorarinsdottir and Schuhen, 2018). The discrete equivalent of
the PIT histogram, which applies to ensemble forecasts, is the verification
rank histogram (Anderson, 1996; Hamill and Colucci, 1997). It shows the
distribution of the ranks of the observations within the corresponding
ensembles and has the same interpretation as the PIT histogram.

For small test sets, the bin number of a PIT/rank histogram must be chosen
with care. With very few bins, the plot may obscure miscalibration while with
many bins, even perfectly calibrated forecasts can yield non-uniformly
appearing histograms (Thorarinsdottir and Schuhen, 2018; Heinrich, 2020). The
bin number should be chosen based on the size of the test set, with the bin
number increasing linearly with the size of the test set (Heinrich, 2020).
More specifically, the uniformity of PIT/rank values can be assessed with
statistical tests (Delle Monache et al., 2006; Taillardat et al., 2016; Wilks,
2019), where the test statistics can be interpreted as a distance between the
observed and a flat histogram (Wilks, 2019; Heinrich, 2020). Testing
predictive performance is further discussed in §2.12.6.

Calibration assessment of multivariate forecasts is complicated by the lack of
a unique ordering in higher dimensions and the many ways in which the
forecasts can be miscalibrated (Wilks, 2019). Gneiting et al. (2008) propose a
general two-step approach where an ensemble forecast and the corresponding
observation are first mapped to a single value by a pre-rank function.
Subsequently, the pre-rank function values are ranked in a standard manner.
The challenge here is to find a pre-rank function that yields informative and
discriminative ranking (Wilks, 2004; Gneiting et al., 2008; Thorarinsdottir et
al., 2016), see Thorarinsdottir et al. (2016) and Wilks (2019) for comparative
studies. Alternatively, Ziegel and Gneiting (2014) propose a direct
multivariate extension of the univariate setting based on copulas.

####  2.12.6 Statistical tests of forecast performance888888This subsection
was written by Victor Richmond R. Jose.

A natural consequence of growth in forecasting methodologies was the
development of statistical tests for predictive ability in the last thirty
years. These tests provided forecasters some formal reassurance that the
predictive superiority of a leading forecast is statistically significant and
is not merely due to random chance.

One of the early papers that undoubtedly sparked growth in this field was
Diebold and Mariano (1995, DM hereafter). In their seminal paper, DM provided
a simple yet general approach for testing equal predictive ability, i.e., if
two forecasting sources ($f_{1,t}$ and $f_{2,t}$, $t=1,\ldots,h$) are equally
accurate on average. Mathematically, if we denote the error
$e_{i,t}=y_{t}-f_{i,t}$ for $i=1$, $2$ and $t=1,\ldots,h$, the hypotheses for
this DM test is $H_{0}$: $E[L(-e_{1,t})-L(-e_{2,t})]=0$ for all $t$ versus
$H_{1}$: $E[L(-e_{1,t})-L(-e_{2,t})]\neq 0$ under a loss function $L$. Their
population-level predictive ability test has very few assumptions (e.g.,
covariance stationary loss differential) and is applicable to a wide range of
loss functions, multi-period settings, and wide class of forecast errors
(e.g., non-Gaussian, serially and/or contemporaneously correlated). This test
though not originally intended for models has been widely used by others to
test forecasting models’ accuracy (Diebold, 2015).

Modifications were later introduced by Harvey et al. (1998) to improve small
sample properties of the test. Generalisations and extensions have emerged to
address issues that DM tests encountered in practice such as nested models
(Clark and McCracken, 2001, 2009), parameter estimation error (West, 1996),
cointegrated variables (Corradi et al., 2001), high persistence (Rossi, 2005),
and panel data (Timmermann and Zhu, 2019). Finite-sample predictive ability
tests also emerged from the observation that models may have equal predictive
ability in finite samples, which generated a class called conditional
predictive accuracy tests (Giacomini and White, 2006; Clark and McCracken,
2013).

An alternative approach to comparing forecast accuracy is through the notion
of forecast encompassing, which examines if a forecast encompasses all useful
information from another with respect to predictions (Chong and Hendry, 1986;
Harvey et al., 1998; Clark and McCracken, 2001). Though it has a few more
assumptions, forecast encompassing tests in certain contexts might be
preferable to the mean square prediction error tests à la Diebold-Mariano
(Busetti and Marcucci, 2013).

Another stream of available statistical tests looks at multiple forecasts
simultaneously instead of pairs. Addressing a need for a reality check on
“data snooping”, White (2000) later modified by Hansen (2005) developed a
multiple model test that uses a null hypothesis of “superior predictive
ability” instead of the equal predictive ability used in DM tests. These have
also been generalised to deal with issues such as cointegrated variables
(Corradi et al., 2001) and multi-horizon forecasts (Quaedvlieg, 2019).
Recently, Li et al. (2020d) proposed a conditional superior predictive ability
test similar to Giacomini and White (2006)’s innovation to the DM test. A
different approach for studying performance of multiple forecasting models is
through the use of multiple comparison tests such as multiple comparison with
a control and multiple comparison with the best (Hsu, 1981; Edwards and Hsu,
1983; Horrace and Schmidt, 2000). These tests often are based on jointly
estimated confidence intervals that measure the difference between two
parameters of interest such as the forecast accuracies of a model and a
benchmark. Koning et al. (2005) illustrates how they can be ex post used to
analyse forecasting performance in the M3 forecasting competition (Makridakis
and Hibon, 2000) using model ranking instead of forecast accuracy scores as
its primitives. The multiple comparison of the best was used in the analysis
of the subsequent M4 and M5 Competitions (Makridakis et al., 2020c, 2021b, and
§2.12.7).

####  2.12.7 Forecasting competitions898989This subsection was written by
Fotios Petropoulos.

Forecasting competitions provide a “playground” for academics, data
scientists, students, practitioners, and software developers to compare the
forecasting performance of their methods and approaches against others.
Organisers of forecasting competitions test the performance of the
participants’ submissions against some hidden data, usually the last window of
observations for each series. The benefits from forecasting competitions are
multifold. Forecasting competitions (i) motivate the development of innovative
forecasting solutions, (ii) provide a deeper understanding of the conditions
that some methods work and others fail, (iii) promote knowledge dissemination,
(iv) provide a much-needed, explicit link between theory and practice, and (v)
leave as a legacy usable and well-defined data sets. Participation in
forecasting competitions is sometimes incentivised by monetary prizes.
However, the stakes are usually much higher, including reputational benefits.

The most famous forecasting competitions are the ones organised by Spyros
Makridakis. Initially, the research question focused on the relative
performance of simple versus complex forecast. M and M3 competitions
(Makridakis et al., 1982; Makridakis and Hibon, 2000) empirically showed that
simple methods (such as exponential smoothing; see §2.3.1) are equally good
compared to other more complex methods and models (such as ARIMA and neural
networks; see §2.3.4 and §2.7.8 respectively) in point-forecast accuracy – if
not better. Moreover, the early Makridakis competitions showed the importance
of forecast combinations in increasing predictive accuracy. For example, the
winner of the M3 competition was the Theta method (see §2.3.3), a simple
statistical method that involved the combination of linear regression and
simple exponential smoothing forecasts (Assimakopoulos and Nikolopoulos,
2000).

The M4 competition (Makridakis et al., 2020c) challenged researchers and
practitioners alike with a task of producing point forecasts and prediction
intervals for 100 thousand time series of varied frequencies. This time, the
main hypothesis focused on the ability of machine learning and neural network
approaches in the task of time series forecasting. Machine learning approaches
(see §2.7.10) that focused on each series independently performed poorly
against statistical benchmarks, such as Theta, Damped exponential smoothing or
simple averages of exponential smoothing models. However, the best two
performing submissions in the M4 competition (Smyl, 2020; Montero-Manso et
al., 2020) used neural network and machine learning algorithms towards
utilising cross-learning. So, the main learning outcome from the M4
competition is that, if utilised properly, machine learning can increase the
forecasting performance. Similarly to previous competitions, M4 demonstrated
again the usefulness of combining across forecasts, with five out of the top
six submissions offering a different implementation of forecast combinations.

Several other forecasting competitions focused on specific contexts and
applications. For example, M2 competition (Makridakis et al., 1993) suggested
that the benefits from additional information (domain expertise) are limited;
see also §2.11.4. The tourism forecasting competition (Athanasopoulos et al.,
2011) also showed that exogenous variables do not add value, while naive
forecasts perform very well on a yearly frequency (for a discussion on tourism
forecasting applications, see §3.8.1). The NN3 competition (Crone et al.,
2011) confirmed the superior performance of statistical methods, but noted
that neural network approaches are closing the distance. Tao Hong’s series of
energy competitions (Hong et al., 2014, 2016, 2019) demonstrated best
practices for load, price, solar, and wind forecasting, with extensions to
probabilistic and hierarchical forecasts (for energy forecasting applications,
see §3.4). Finally, many companies have hosted forecasting challenges through
the Kaggle platform. Bojer and Meldgaard (2020) reviewed the Kaggle
competitions over the last five years, and concluded that access to
hierarchical information, cross-learning, feature engineering, and
combinations (ensembles) can lead to increased forecasting performance,
outperforming traditional statistical methods. These insights were a
forerunner to the results of the M5 competition, which focused on
hierarchically organised retail data (Makridakis et al., 2021b, c).

Makridakis et al. (2021a) provide a list of design attributes for forecasting
competitions and propose principles for future competitions.

###  2.13 The future of forecasting theory909090This subsection was written by
Pierre Pinson.

The theory of forecasting appears mature today, based on dedicated
developments at the interface among a number of disciplines, e.g., mathematics
and statistics, computer sciences, psychology, etc. A wealth of these
theoretical developments have originated from specific needs and challenges in
different application areas, e.g., in economics, meteorology and climate
sciences, as well as management science among others. In this section, many
aspects of the theory of forecasting were covered, with aspects related to
data, modelling and reasoning, forecast verification. Now, the fact that
forecasting is mature does not mean that all has been done – we aim here at
giving a few pointers at current and future challenges.

First of all, it is of utmost importance to remember that forecasting is a
process that involves both quantitative aspects (based on data and models) and
humans, at various levels, i.e., from the generation of forecasts to their use
in decision-making. A first consequence is that we always need to find,
depending on the problem at hand, an optimal trade-off between data-driven
approaches and the use of expert judgment. In parallel, forecasting is to be
thought of in a probabilistic framework in a systematic manner (Gneiting and
Katzfuss, 2014). This allows us to naturally convey uncertainty about the
future, while providing the right basis to make optimal decisions in view of
the characteristics of the decision problem, as well as the loss (or utility)
function and risk aversion of the decision maker. Another consequence is that
using forecasts as input to decision-making often affects the outcome to be
predicted itself – a problem known as self-negating forecasts (possibly also
self-fulfilling) or the prophet dilemma. With advances in the science of
dynamic systems and game theory, we should invest in modelling those systems
as a whole (i.e., forecasting and decision-making) in order to predict the
full range of possible outcomes, based on the decisions that could be made.

In parallel, it is clear that today, the amount of data being collected and
possibly available for forecasting is growing at an astounding pace. This
requires re-thinking our approaches to forecasting towards high-dimensional
models, online learning, etc. Importantly, the data being collected is
distributed in terms of ownership. And, due to privacy concerns and
competitive interests, some may not be ready to share their data. Novel
frameworks to learning and forecasting ought to be developed with that context
in mind, for instance focusing on distributed and privacy-preserving learning
– an example among many others is that of Google pushing forward federated
learning (Abadi et al., 2016), an approach to deep learning where the learning
process is distributed and with a privacy layer. Eventually the access and use
of data, as well as the contribution to distributed learning (and
collaborative analytics, more generally), may be monetised, bringing a
mechanism design component to the future theory of forecasting. A simple and
pragmatic example is that of forecast reconciliation: if asking various agents
to modify their forecasts to make them coherent within a hierarchy, such
modifications could be monetised to compensate for accuracy loss.

A large part of today’s modelling and forecasting approaches uses a wealth of
data to identify and fit models, to be eventually used to forecast based on
new data and under new conditions. Different approaches have been proposed to
maximise the generalisation ability of those models, to somewhat maximise
chances to do well out-of-sample. At the root of this problem is the effort to
go beyond correlation only, and to identify causality (see, e.g., Pearl (2009)
for a recent extensive coverage). While causality has been a key topic of
interest to forecasters for a long time already, new approaches and concepts
are being pushed forward for identification of and inference in causal models
(Peters et al., 2017), which may have a significant impact on the theory of
forecasting.

Eventually, the key question of _what a good forecast is_ will continue to
steer new developments in the theory of forecasting in the foreseeable future.
The nature of goodness of forecasts (seen from the meteorological application
angle) was theorised a few decades ago already (Murphy, 1993), based on
consistency, quality and value. We still see the need to work further on that
question – possibly considering these 3 pillars, but possibly also finding
other ways to define desirable properties of forecasts. This will, in all
cases, translates to further developing frameworks for forecast verification,
focusing on the interplay between forecast quality and value, but also better
linking to psychology and behavioural economics. In terms of forecast
verification, some of the most pressing areas most likely relate to
(multivariate) probabilistic forecasting and to the forecasting of extreme
events. When it comes to forecast quality and value, we need to go beyond the
simple plugging of forecasts into decision problems to assess whether this
yields better decisions, or not. Instead, we ought to propose suitable
theoretical frameworks that allow assessing whether certain forecasts are
fundamentally better (than others) for given classes of decision problems.
Finally, the link to psychology and behavioural economics should ensure a
better appraisal of how forecasts are to be communicated, how they are
perceived and acted upon.

Most of the advances in the science of forecasting have come from the
complementarity between theoretical developments and applications. We can then
only be optimistic for the future since more and more application areas are
relying heavily on forecasting. Their specific needs and challenges will
continue fuelling upcoming developments in the theory of forecasting.

##  3 Practice

###  3.1 Introduction to forecasting practice919191This subsection was written
by Michael Gilliland.

The purpose of forecasting is to improve decision making in the face of
uncertainty. To achieve this, forecasts should provide an unbiased guess at
what is most likely to happen (the point forecast), along with a measure of
uncertainty, such as a prediction interval (PI). Such information will
facilitate appropriate decisions and actions.

Forecasting should be an objective, dispassionate exercise, one that is built
upon facts, sound reasoning, and sound methods. But since forecasts are
created in social settings, they are influenced by organisational politics and
personal agendas. As a consequence, forecasts will often reflect aspirations
rather than unbiased projections.

In organisations, forecasts are created through processes that can involve
multiple steps and participants. The process can be as simple as executive
fiat (also known as evangelical forecasting), unencumbered by what the data
show. More commonly, the process begins with a statistical forecast (generated
by forecasting software), which is then subject to review and adjustment, as
illustrated in figure 5.

Figure 5: Multi-stage forecasting process.

In concept, such an elaborate multi-stage process allows “management
intelligence” to improve forecast quality, incorporating information not
accounted for in the statistical model. In reality, however, benefits are not
assured. Lawrence et al. (2006) reviewed more than 200 studies, concluding
that human judgment can be of significant benefit but is also subject to
significant biases. Among the many papers on this subject, there is general
agreement on the need to track and review overrides, and the need to better
understand the psychological issues around judgmental adjustments.

The underlying problem is that each human touch point subjects the forecast to
the interests of the reviewers – and these interests may not align with
creating an accurate, unbiased forecast. To identify where such problems are
occurring, Forecast Value Added (FVA) analysis is an increasingly popular
approach among practitioners.

FVA is defined as the change in a forecasting performance metric that can be
attributed to a particular step or participant in the forecasting process
(Gilliland, 2002). Any activity that fails to deliver positive FVA (i.e.,
fails to improve forecast quality) is considered process waste.

Starting with a naive forecast, FVA analysis seeks to determine whether each
subsequent step in the process improves upon the prior steps. The “stairstep
report” of table 1 is a familiar way of summarising results, as in this
example from Newell Rubbermaid (Schubert and Rickard, 2011).

Table 1: Stairstep report showing FVA results. Process Step | Forecast accuracy | FVA vs. Naive | FVA vs. Statistical
---|---|---|---
| (100%$-$MAPE) |  |
Naive forecast | 60% |  |
Statistical forecast | 65% | 5% |
Adjusted forecast | 62% | 2% | -3%

Here, averaged across all products, naive (random walk) achieved forecast
accuracy of 60%. The company’s statistical forecast delivered five percentage
points of improvement, but management review and adjustment delivered negative
value. Such findings – not uncommon – urge further investigation into causes
and possible process corrections (such as training reviewers or limiting
adjustments). Alternatively, the management review step could be eliminated,
providing the dual benefits of freeing up management time spent on forecasting
and, on average, more accurate forecasts.

Morlidge (2014c) expanded upon FVA analysis to present a strategy for
prioritising judgmental adjustments, finding the greatest opportunity for
error reduction in products with high volume and high relative absolute error.
Chase (2021) described a machine learning (ML) method to guide forecast
review, identifying which forecasts are most likely to benefit from adjustment
along with a suggested adjustment range. Baker (2021) used ML classification
models to identify characteristics of non-value adding overrides, proposing
the behavioural economics notion of a “nudge” to prompt desired forecaster
behaviour. Further, Goodwin et al. (2017) derived upper bounds for FVA
relative to naive forecasts. And de Kok (2017) created a Stochastic Value
Added (SVA) metric to assess the difference between actual and forecasted
distributions, knowledge of which is valuable for inventory management.

Including an indication of uncertainty around the point forecast remains an
uncommon practice. Prediction intervals in software generally underestimate
uncertainty, often dramatically, leading to unrealistic confidence in the
forecast. And even when provided, PIs largely go unused by practitioners.
Goodwin (2014) summarised the psychological issues, noting that the generally
poor calibration of the PIs may not explain the reluctance to utilise them.
Rather, “an interval forecast may accurately reflect the uncertainty, but it
is likely to be spurned by decision makers if it is too wide and judged to be
uninformative” (Goodwin, 2014, page 5).

It has long been recognised (Chatfield, 1986; Lawrence, 2000) that the
practice of forecasting falls well short of the potential exhibited in
academic research, and revealed by the M forecasting competitions. In the M4,
a simple benchmark combination method (the average of Single, Holt, and Damped
exponential smoothing) reduced the overall weighted average (OWA) error by
17.9% compared to naive. The top six performing methods in M4 further reduced
OWA by over 5% compared to the combination benchmark (Makridakis et al.,
2020c). But in forecasting practice, just bettering the accuracy of naive has
proven to be a surprising challenge. Morlidge’s (2014b) study of eight
consumer and industrial businesses found 52% of their forecasts failed to do
so. And, as shown, Newel Rubbermaid beat naive by just two percentage points
after management adjustments.

Ultimately, forecast accuracy is limited by the nature of the behaviour being
forecast. But even a highly accurate forecast is of little consequence if
overridden by management and not used to enhance decision making and improve
organisational performance.

Practitioners need to recognise limits to forecastability and be willing to
consider alternative (non-forecasting) approaches when the desired level of
accuracy is not achievable (Gilliland, 2010). Alternatives include supply
chain re-engineering – to better react to unforeseen variations in demand, and
demand smoothing – leveraging pricing and promotional practices to shape more
favourable demand patterns.

Despite measurable advances in our statistical forecasting capabilities
(Makridakis et al., 2020b), it is questionable whether forecasting practice
has similarly progressed. The solution, perhaps, is what Morlidge (2014a, page
39) suggests that “users should focus less on trying to optimise their
forecasting process than on detecting where their process is severely
suboptimal and taking measures to redress the problem”. This is where FVA can
help.

For now, the challenge for researchers remains: To prompt practitioners to
adopt sound methods based on the objective assessment of available
information, and avoid the “worst practices” that squander resources and fail
to improve the forecast.

###  3.2 Operations and supply chain management

####  3.2.1 Demand management929292This subsection was written by Yanfei Kang.

Demand management is one of the dominant components of supply chain management
(Fildes et al., 2006). Accurate demand estimate of the present and future is a
first vital step for almost all aspects of supply chain optimisation, such as
inventory management, vehicle scheduling, workforce planning, and distribution
and marketing strategies (Kolassa and Siemsen, 2016). Simply speaking, better
demand forecasts can yield significantly better supply chain management,
including improved inventory management and increased service levels. Classic
demand forecasts mainly rely on qualitative techniques, based on expert
judgment and past experience (e.g., Weaver, 1971), and quantitative
techniques, based on statistical and machine learning modelling (e.g., Taylor,
2003b; Bacha and Meyer, 1992). A combination of qualitative and quantitative
methods is also popular and proven to be beneficial in practice by, e.g.,
judgmental adjustments (Turner, 1990; Önkal and Gönül, 2005; Syntetos et al.,
2016b, and §2.11.2), judgmental forecast model selection (Petropoulos et al.,
2018b; Han et al., 2019, and §2.11.3), and other advanced forecasting support
systems (Baecke et al., 2017; Arvan et al., 2019, see also §3.7.1).

The key challenges that demand forecasting faces vary from domain to domain.
They include:

  1. 1.

The existence of intermittent demands, e.g., irregular demand patterns of
fashion products. According to Nikolopoulos (2020), limited literature has
focused on intermittent demand. The seminal work by Croston (1972) was
followed by other representative methods such as the SBA method by Syntetos
and Boylan (2001), the aggregate–disaggregate intermittent demand approach
(ADIDA) by Nikolopoulos et al. (2011), the multiple temporal aggregation by
Petropoulos and Kourentzes (2015), and the $k$ nearest neighbour ($k$NN) based
approach by Nikolopoulos et al. (2016). See §2.8 for more details on
intermittent demand forecasting and §2.10.2 for a discussion on temporal
aggregation.

  2. 2.

The emergence of new products. Recent studies on new product demand
forecasting are based on finding analogies (Wright and Stern, 2015; Hu et al.,
2019), leveraging comparable products (Baardman et al., 2018), and using
external information like web search trends (Kulkarni et al., 2012). See
§3.2.6 for more details on new product demand forecasting.

  3. 3.

The existence of short-life-cycle products, e.g., smartphone demand (e.g.,
Szozda, 2010; Chung et al., 2012; Shi et al., 2020).

  4. 4.

The hierarchical structure of the data such as the electricity demand mapped
to a geographical hierarchy (e.g., Athanasopoulos et al., 2009; Hyndman et
al., 2011; Hong et al., 2019, but also §2.10.1).

With the advent of the big data era, a couple of coexisting new challenges
have drawn the attention of researchers and practitioners in the forecasting
community: the need to forecast a large volume of related time series (e.g.,
thousands or millions of products from one large retailer: Salinas et al.,
2019a), and the increasing number of external variables that have significant
influence on future demand (e.g., massive amounts of keyword search indices
that could impact future tourism demand (Law et al., 2019)). Recently, to deal
with these new challenges, numerous empirical studies have identified the
potentials of deep learning based global models, in both point and
probabilistic demand forecasting (e.g., Wen et al., 2017; Rangapuram et al.,
2018; Salinas et al., 2019a; Bandara et al., 2020b). With the merits of cross-
learning, global models have been shown to be able to learn long memory
patterns and related effects (Montero-Manso and Hyndman, 2020), latent
correlation across multiple series (Smyl, 2020), handle complex real-world
forecasting situations such as data sparsity and cold-starts (Chen et al.,
2020), include exogenous covariates such as promotional information and
keyword search indices (Law et al., 2019), and allow for different choices of
distributional assumptions (Salinas et al., 2019a).

####  3.2.2 Forecasting in the supply chain939393This subsection was written
by Paul Goodwin.

A supply chain is ‘a network of stakeholders (e.g., retailers, manufacturers,
suppliers) who collaborate to satisfy customer demand’ (Perera et al., 2019).
Forecasts inform many supply chain decisions, including those relating to
inventory control, production planning, cash flow management, logistics and
human resources (also see §3.2.1). Typically, forecasts are based on an
amalgam of statistical methods and management judgment (Fildes and Goodwin,
2007). Hofmann and Rutschmann (2018) have investigated the potential for using
big data analytics in supply chain forecasting but indicate more research is
needed to establish its usefulness.

In many organisations forecasts are a crucial element of Sales and Operations
Planning (S&OP), a tool that brings together different business plans, such as
those relating to sales, marketing, manufacturing and finance, into one
integrated set of plans (Thomé et al., 2012). The purposes of S&OP are to
balance supply and demand and to link an organisation’s operational and
strategic plans. This requires collaboration between individuals and
functional areas at different levels because it involves data sharing and
achieving a consensus on forecasts and common objectives (Mello, 2010).
Successful implementations of S&OP are therefore associated with forecasts
that are both aligned with an organisation’s needs and able to draw on
information from across the organisation. This can be contrasted with the
‘silo culture’ identified in a survey of companies by Moon et al. (2003) where
separate forecasts were prepared by different departments in ‘islands of
analysis’. Methods for reconciling forecasts at different levels in both
cross-sectional hierarchies (e.g., national, regional and local forecasts) and
temporal hierarchies (e.g., annual, monthly and daily forecasts) are also
emerging as an approach to break through information silos in organisations
(see §2.10.1, §2.10.2, and §2.10.3). Cross-temporal reconciliation provides a
data-driven approach that allows information to be drawn from different
sources and levels of the hierarchy and enables this to be blended into
coherent forecasts (Kourentzes and Athanasopoulos, 2019).

In some supply chains, companies have agreed to share data and jointly manage
planning processes in an initiative known as Collaborative Planning,
Forecasting, and Replenishment (CPFR) (Seifert, 2003, also see §3.2.3). CPFR
involves pooling information on inventory levels and on forthcoming events,
like sales promotions. Demand forecasts can be shared, in real time via the
Internet, and discrepancies between them reconciled. In theory, information
sharing should reduce forecast errors. This should mitigate the ‘bullwhip
effect’ where forecast errors at the retail-end of supply chains cause
upstream suppliers to experience increasingly volatile demand, forcing them to
hold high safety stock levels (Lee et al., 2007). Much research demonstrating
the benefits of collaboration has involved simulated supply chains (Fildes,
2017). Studies of real companies have also found improved performance through
collaboration (e.g., Boone and Ganeshan, 2008; Hill et al., 2018; Eksoz et
al., 2019), but case study evidence is still scarce (Syntetos et al., 2016a).
The implementation of collaborative schemes has been slow with many not
progressing beyond the pilot stage (Panahifar et al., 2015; Galbreth et al.,
2015). Barriers to successful implementation include a lack of trust between
organisations, reward systems that foster a silo mentality, fragmented
forecasting systems within companies, incompatible systems, a lack of relevant
training and the absence of top management support (Fliedner, 2003; Thomé et
al., 2014).

Initiatives to improve supply chain forecasting can be undermined by political
manipulation of forecasts and gaming. Examples include ‘enforcing’: requiring
inflated forecasts to align them with sales or financial goals, ‘sandbagging’:
underestimating sales so staff are rewarded for exceeding forecasts, and
‘spinning’: manipulating forecasts to garner favourable reactions from
colleagues (Mello, 2009). Pennings et al. (2019) discuss schemes for
correcting such intentional biases.

For a discussion of the forecasting of returned items in supply chains, see
§3.2.9, while §3.9 offers a discussion of possible future developments in
supply chain forecasting.

####  3.2.3 Forecasting for inventories949494This subsection was written by
John E. Boylan.

Three aspects of the interaction between forecasting and inventory management
have been studied in some depth and are the subject of this review: the
bullwhip effect, forecast aggregation, and performance measurement.

The ‘bullwhip effect’ occurs whenever there is amplification of demand
variability through the supply chain (Lee et al., 2004), leading to excess
inventories. This can be addressed by supply chain members sharing downstream
demand information, at stock keeping unit level, to take advantage of less
noisy data. Analytical results on the translation of ARIMA (see §2.3.4) demand
processes have been established for order-up-to inventory systems (Gilbert,
2005). There would be no value in information sharing if the wholesaler can
use such relationships to deduce the retailer’s demand process from their
orders (see, for example, Graves, 1999). Such deductions assume that the
retailer’s demand process and demand parameters are common knowledge to supply
chain members. Ali and Boylan (2011) showed that, if such common knowledge is
lacking, there is value in sharing the demand data itself and Ali et al.
(2012) established relationships between accuracy gains and inventory savings.
Analytical research has tended to assume that demand parameters are known.
Pastore et al. (2020) investigated the impact of demand parameter uncertainty,
showing how it exacerbates the bullwhip effect.

Forecasting approaches have been developed that are particularly suitable in
an inventory context, even if not originally proposed to support inventory
decisions. For example, Nikolopoulos et al. (2011) proposed that forecasts
could be improved by aggregating higher frequency data into lower frequency
data (see also §2.10.2; other approaches are reviewed in §3.2.1). Following
this approach, Forecasts are generated at the lower frequency level and then
disaggregated, if required, to the higher frequency level. For inventory
replenishment decisions, the level of aggregation may conveniently be chosen
to be the lead time, thereby taking advantage of the greater stability of data
at the lower frequency level, with no need for disaggregation.

The variance of forecast errors over lead time is required to determine safety
stock requirements for continuous review systems. The conventional approach is
to take the variance of one-step-ahead errors and multiply it by the lead
time. However, this estimator is unsound, even if demand is independent and
identically distributed, as explained by Prak et al. (2017). A more direct
approach is to smooth the mean square errors over the lead time (Syntetos and
Boylan, 2006).

Strijbosch and Moors (2005) showed that unbiased forecasts will not
necessarily lead to achievement, on average, of target cycle service levels or
fill rates. Wallström and Segerstedt (2010) proposed a ‘Periods in Stock’
measure, which may be interpreted, based on a ‘fictitious stock’, as the
number of periods a unit of the forecasted item has been in stock or out of
stock. Such measures may be complemented by a detailed examination of error-
implication metrics (Boylan and Syntetos, 2006). For inventory management,
these metrics will typically include inventory holdings and service level
implications (e.g., cycle service level, fill rate). Comparisons may be based
on total costs or via ‘exchange curves’, showing the trade-offs between
service and inventory holding costs. Comparisons such as these are now
regarded as standard in the literature on forecasting for inventories and
align well with practice in industry.

####  3.2.4 Forecasting in retail959595This subsection was written by Stephan
Kolassa & Patrícia Ramos.

Retail companies depend crucially on accurate demand forecasting to manage
their supply chain and make decisions concerning planning, marketing,
purchasing, distribution and labour force. Inaccurate forecasts lead to
unnecessary costs and poor customer satisfaction. Inventories should be
neither too high (to avoid waste and extra costs of storage and labour force),
nor too low (to prevent stock-outs and lost sales, Ma and Fildes, 2017).

Forecasting retail demand happens in a three-dimensional space (Syntetos et
al., 2016a): the position in the supply chain hierarchy (store, distribution
centre, or chain), the level in the product hierarchy (SKU, brand, category,
or total) and the time granularity (day, week, month, quarter, or year). In
general, the higher is the position in the supply chain, the lower is the time
granularity required, e.g., retailers need daily forecasts for store
replenishment and weekly forecasts for DC distribution/logistics activities at
the SKU level (Fildes et al., 2019b). Hierarchical forecasting (see §2.10.1)
is a promising tool to generate coherent demand forecasts on multiple levels
over different dimensions (Oliveira and Ramos, 2019).

Several factors affect retail sales, which often increase substantially during
holidays, festivals, and other special events. Price reductions and promotions
on own and competitors’ products, as well as weather conditions or pandemics,
can also change sales considerably (Huang et al., 2019).

Zero sales due to stock-outs or low demand occur very often at the SKU
$\times$ store level, both at weekly and daily granularity. The most
appropriate forecasting approaches for intermittent demand are Croston’s
method (Croston, 1972), the Syntetos-Boylan approximation (SBA; Syntetos and
Boylan, 2005), and the TSB method (Teunter et al., 2011), all introduced in
§2.8.1. These methods have been used to forecast sales of spare parts in
automotive and aerospace industries but have not yet been evaluated in the
retail context.

Univariate forecasting models are the most basic methods retailers may use to
forecast demand. They range from simple methods such as simple moving averages
or exponential smoothing to ARIMA and ETS models (discussed in §2.3). These
are particularly appropriate to forecast demand at higher aggregation levels
(Ramos et al., 2015; Ramos and Oliveira, 2016). The main advantage of linear
causal methods such as multiple linear regression is to allow the inclusion of
external effects discussed above. There is no clear evidence yet that
nonlinear models and novel machine learning methods can improve forecast
accuracy (Fildes et al., 2019b).

To be effective, point estimates should be combined with quantile predictions
or prediction intervals for determining safety stock amounts needed for
replenishment. However, to the best of our knowledge this is an under-
investigated aspect of retail forecasting (Taylor, 2007; Kolassa, 2016).

The online channel accounts for an ever-increasing proportion of retail sales
and poses unique challenges to forecasting, beyond the characteristics of
brick and mortar (B&M) retail stores. First, there are multiple _drivers or
predictors_ of demand that could be leveraged in online retail, but not in
B&M:

  * •

Online retailers can fine-tune customer interactions, e.g., through the
landing page, product recommendations, or personalised promotions, leveraging
the customer’s purchasing, browsing or returns history, current shopping cart
contents, or the retailer’s stock position, in order to tailor the message to
one specific customer in a way that is impossible in B&M.

  * •

Conversely, product reviews are a type of interaction between the customer and
the retailer and other customers which drives future demand.

Next, there are differences in forecast _use_ :

  * •

Forecast use strongly depends on the retailer’s omnichannel strategy
(Armstrong, 2017; Sopadjieva et al., 2017; Melacini et al., 2018): e.g., for
“order online, pick up in store” or “ship from store” fulfillment, we need
separate but related forecasts for both total online demand and for the demand
fulfilled at each separate store.

  * •

Online retailers, especially in fashion, have a much bigger problem with
product returns. They may need to forecast how many products are returned
overall (e.g., Shang et al., 2020), or whether a _specific_ customer will
return a _specific_ product.

Finally, there are differences in the forecasting _process_ :

  * •

B&M retailers decouple pricing/promotion decisions and optimisation from the
customer interaction, and therefore from forecasting. Online, this is not
possible, because the customer has total transparency to competitors’
offerings. Thus, online pricing needs to react much more quickly to
competitive pressures – faster than the forecasting cycle.

  * •

Thus, the specific value of predictors is often not known at the time of
forecasting: we don’t know yet which customer will log on, so we don’t know
yet how many people will see a particular product displayed on their
personalised landing page. (Nor do we know today what remaining stock will be
displayed.) Thus, changes in drivers need to be “baked into” the forecasting
algorithm.

  * •

Feedback loops between forecasting and other processes are thus even more
important online: yesterday’s forecasts drive today’s stock position, driving
today’s personalised recommendations, driving demand, driving today’s
forecasts for tomorrow. Overall, online retail forecasting needs to be more
agile and responsive to the latest interactional decisions taken in the web
store, and more tightly integrated into the retailer’s interactional tactics
and omnichannel strategy.

Systematic research on demand forecasting in an online or omnnichannel context
is only starting to appear (e.g. Omar et al., 2021, who use basket data from
online sales to improve omnichannel retail forecasts).

####  3.2.5 Promotional forecasting969696This subsection was written by
Nikolaos Kourentzes.

Promotional forecasting is central for retailing (see §3.2.4), but also
relevant for many manufacturers, particularly of Fast Moving Consumer Goods
(FMCG). In principle, the objective is to forecast sales, as in most business
forecasting cases. However, what sets promotional forecasting apart is that we
also make use of information about promotional plans, pricing, and sales of
complementary and substitute products (Bandyopadhyay, 2009; Zhang et al.,
2008). Other relevant variables may include store location and format,
variables that capture the presentation and location of a product in a store,
proxies that characterise the competition, and so on (Van Heerde et al., 2002;
Andrews et al., 2008).

Three modelling considerations guide us in the choice of models. First,
promotional (and associated effects) are proportional. For instance, we do not
want to model the increase in sales as an absolute number of units, but
instead, as a percentage uplift. We do this to not only make the model
applicable to both smaller and larger applications, for example, small and
large stores in a retailing chain, but also to gain a clearer insight into the
behaviour of our customers. Second, it is common that there are synergy
effects. For example, a promotion for a product may be offset by promotions
for substitute products. Both these considerations are easily resolved if we
use multiplicative regression models. However, instead of working with the
multiplicative models, we rely on the logarithmic transformation of the data
(see §2.2.1) and proceed to construct the promotional model using the less
cumbersome additive formulation (see §2.3.2). Third, the objective of
promotional models does not end with providing accurate predictions. We are
also interested in the effect of the various predictors: their elasticity.
This can in turn provide the users with valuable information about the
customers, but also be an input for constructing optimal promotional and
pricing strategies (Zhang et al., 2008).

Promotional models have been widely used on brand-level data (for example,
Divakar et al., 2005). However, they are increasingly used on Stock Keeping
Unit (SKU) level data (Trapero et al., 2015; Ma et al., 2016), given advances
in modelling techniques. Especially at that level, limited sales history and
potentially non-existing examples of past promotions can be a challenge.
Trapero et al. (2015) consider this problem and propose using a promotional
model that has two parts that are jointly estimated. The first part focuses on
the time series dynamics and is modelled locally for each SKU. The second part
tackles the promotional part, which pools examples of promotions across SKUs
to enable providing reasonable estimates of uplifts even for new SKUs. To
ensure the expected heterogeneity in the promotional effects, the model is
provided with product group information. Another recent innovation is looking
at modelling promotional effects both at the aggregate brand or total sales
level, and disaggregate SKU level, relying on temporal aggregation (Kourentzes
and Petropoulos, 2016, and §2.10.2). Ma et al. (2016) concern themselves with
the intra-and inter-category promotional information. The challenge now is the
number of variables to be considered for the promotional model, which they
address by using sequential LASSO (see also §2.5.3). Although the
aforementioned models have shown very promising results, one has to recognise
that in practice promotions are often forecasted using judgmental adjustments,
with inconsistent performance (Trapero et al., 2013); see also §2.11.2 and
§3.7.3.

####  3.2.6 New product forecasting979797This subsection was written by Sheik
Meeran.

Forecasting the demand for a new product accurately has even more consequence
with regards to well-being of the companies than that for a product already in
the market. However, this task is one of the most difficult tasks managers
must deal with simply because of non-availability of past data (Wind, 1981).
Much work has been going on for the last five decades in this field. Despite
his Herculean attempt to collate the methods reported, Assmus (1984) could not
list all even at that time. The methods used before and since could be
categorised into three broad approaches (Goodwin et al., 2013a) namely
management judgment, consumer judgment and diffusion/formal mathematical
models. In general, the hybrid methods combining different approaches have
been found to be more useful (Hyndman and Athanasopoulos, 2018; Peres et al.,
2010). Most of the attempts in New product Forecasting (NPF) have been about
forecasting ‘adoption’ (i.e., enumerating the customers who bought at least
one time) rather than ‘sales’, which accounts for repeat purchases also. In
general, these attempts dealt with point forecast although there have been
some attempts in interval and density forecasting (Meade and Islam, 2001).

Out of the three approaches in NPF, management judgment is the most used
approach (Kahn, 2002; Gartner and Thomas, 1993; Lynn et al., 1999) which is
reported to have been carried out by either individual managers or group of
them. Ozer (2011) and Surowiecki (2005) articulated their contrasting benefits
and deficits. The Delphi method (see §2.11.4) has combined the benefits of
these two modes of operation (Rowe and Wright, 1999) which has been effective
in NPF. Prediction markets in the recent past offered an alternative way to
aggregate forecasts from a group of Managers (Wolfers and Zitzewitz, 2004;
Meeran et al., 2013) and some successful application of prediction markets for
NPF have been reported by Plott and Chen (2002) and Karniouchina (2011).

In the second category, customer surveys, among other methods, are used to
directly ask the customers the likelihood of them purchasing the product. Such
surveys are found to be not very reliable (Morwitz, 1997). An alternative
method to avoid implicit bias associated with such surveys in extracting
inherent customer preference is conjoint analysis, which makes implicit trade
off customers make between features explicit by analysing the customers’
preference for different variants of the product. One analysis technique that
attempts to mirror real life experience more is Choice Based Conjoint analysis