# Creating Probabilistic Forecasts from Arbitrary Deterministic Forecasts
using Conditional Invertible Neural Networks

Kaleb Phipps<EMAIL_ADDRESS>Benedikt Heidrich  Marian Turowski  Moritz
Wittig  Ralf Mikut  Veit Hagenmeyer

###### Abstract

In various applications, probabilistic forecasts are required to quantify the
inherent uncertainty associated with the forecast. However, numerous modern
forecasting methods are still designed to create deterministic forecasts.
Transforming these deterministic forecasts into probabilistic forecasts is
often challenging and based on numerous assumptions that may not hold in real-
world situations. Therefore, the present article proposes a novel approach for
creating probabilistic forecasts from arbitrary deterministic forecasts. In
order to implement this approach, we use a conditional Invertible Neural
Network (cINN). More specifically, we apply a cINN to learn the underlying
distribution of the data and then combine the uncertainty from this
distribution with an arbitrary deterministic forecast to generate accurate
probabilistic forecasts. Our approach enables the simple creation of
probabilistic forecasts without complicated statistical loss functions or
further assumptions. Besides showing the mathematical validity of our
approach, we empirically show that our approach noticeably outperforms
traditional methods for including uncertainty in deterministic forecasts and
generally outperforms state-of-the-art probabilistic forecasting benchmarks.

###### keywords:

probabilistic forecasting , conditional invertible neural networks

††journal: International Journal of Forecasting

[kit]organization=Karlsruhe Institute for Technology,addressline=Institute for
Automation and Applied Informatics, city=Eggenstein-Leopoldshafen,
postcode=76344, country=Germany

##  1 Introduction

To quantify the inherent uncertainty associated with any prediction of the
future, forecasts must be probabilistic (Gneiting et al., 2007; Raftery,
2016). These probabilistic forecasts are crucial for many applications such as
stabilising energy systems (Dannecker, 2015), managing congestion in traffic
systems (Liu et al., 2021), or sizing servers of web applications to cope with
a certain number of daily visits (Koochali et al., 2019). Despite this
necessity for probabilistic forecasts, many forecasting methods are still
deterministic (Petropoulos et al., 2022). Even state-of-the-art forecasting
methods such as N-BEATS (Oreshkin et al., 2019), N-HiTS (Challu et al., 2022),
and Temporal Fusion Transformers (Lim et al., 2021), are still designed only
to generate deterministic forecasts by default. Extending these models to
create probabilistic forecasts would significantly increase their
applicability but is challenging and may involve custom loss functions or
novel architectures.

One solution to overcome these challenges is to create probabilistic forecasts
based on these existing deterministic forecasts. For many years such forecasts
have been created by assuming a Gaussian distribution based on the mean and
variance of the deterministic forecast errors and using this assumption to
calculate resulting quantiles (Hyndman and Athanasopoulos, 2018). Moreover,
such probabilistic forecasts can be created by applying the Bayesian theory of
probability to a deterministic method (Krzysztofowicz, 1999) or considering
Monte-Carlo sampling methods (Caflisch, 1998). Whilst these methods have been
effective in the past, their successful implementation requires a fundamental
understanding of statistics and is based on a series of mathematical
assumptions that may not hold in a real-world setting. Ideally, such
probabilistic forecasts should be created directly from arbitrary
deterministic forecasts, without any further statistical assumptions, and
without modifying the existing deterministic base forecaster in any way.

Therefore, in the present article, we present an easy-to-use approach that
creates probabilistic forecasts from arbitrary deterministic forecasts by
using a Conditional Invertible Neural Network (cINN)to learn the underlying
distribution of the time series data. Since time series have an inherent
component of randomness (Hyndman and Athanasopoulos, 2018), we propose using
this uncertainty within the distribution of the time series data to create
probabilistic forecasts. With our approach, we first map the unknown
distribution of the underlying time series data to a known and tractable
distribution by applying a cINN. Then, we use the output of a trained
arbitrary deterministic forecast as an input to the cINN and consider the
representation of this forecast in the known and tractable distribution. We
then analyse the neighbourhood of this representation in the known and
tractable distribution to quantify the uncertainty associated with the
representation. Finally, we use the backward pass of the cINN to convert this
uncertainty information into the forecast.

Thus, the main contribution of the present article is threefold. First, we
provide a novel approach for creating probabilistic forecasts from arbitrary
deterministic forecasts without modifying this deterministic forecast in any
way. Second, we mathematically show the validity of the proposed approach.
Third, we empirically evaluate the approach using different data sets from
various domains. In this empirical evaluation, we consider the improvement
over a baseline that assumes a Gaussian distribution of deterministic errors
to create probabilistic forecasts, compare our approach to state-of-the-art
benchmarks, and recreate the Global Energy Forecasting Competition 2014
(GEFCom2014)competition setting.

The remainder of our article is structured as follows. First, we present
related work and highlight the research gap that the present article addresses
in Section 2. In Section 3, we then explain our approach in detail and provide
mathematical proof for its validity. Afterwards, we highlight how we use a
cINN to enable probabilistic forecasts from an arbitrary deterministic
forecast. We detail the experimental setup in Section 4, before presenting our
results in Section 5. In Section 6 we discuss our evaluation and key insights.
Finally, we conclude and suggest possible directions for future work in
Section 7.

##  2 Related Work

Our article is closely related to two research fields, namely previous work
that enables probabilistic forecasts from deterministic forecasts and previous
work focusing on probabilistic forecasts using a cINN. In this section, we
present and discuss related work from both fields and highlight the research
gap addressed by the present article.

#### Enabling Probabilistic Forecasts

Determining the uncertainty associated with a point prediction is one of the
key research areas of uncertainty quantification (Smith, 2013). Therefore,
multiple methods exist, including error analysis and the assumption of a
Gaussian distribution (Hyndman and Athanasopoulos, 2018), a Bayesian method
involving assumed priors (Krzysztofowicz, 1999; Kaplan and Huang, 2021;
Raftery et al., 1997), integrating uncertainty into the prediction via an
ensemble of predictions (Toth et al., 2003; Cramer et al., 2022b), and
considering uncertainty through Monte Carlo sampling approaches or similar
(Caflisch, 1998; Camporeale et al., 2017; Xiu, 2010). Another, more recent
method, uses a custom loss function that considers both calibration and
sharpness to generate probabilistic forecasts based on deterministic forecasts
(Camporeale et al., 2019).

Despite their effectiveness, these methods have several limitations. First,
they all rely on certain assumptions, either regarding the Gaussian
distribution, the prior distribution, or the belief that an ensemble
accurately represents the uncertainty in the forecast. Second, most methods
are based on statistical methods and, therefore, require a fundamental
understanding of statistics to be implemented. Third, these methods,
especially the ensemble and Monte Carlo approaches, may be computationally
expensive. Furthermore, none of the mentioned methods consider the uncertainty
associated with the underlying data distribution when creating probabilistic
forecasts.

#### Probabilistic Forecasts using cINNs

To create probabilistic forecasts, cINNs, also referred to as normalising
flows (Ardizzone et al., 2019), are combined with other machine learning
methods. Arpogaus et al. (2022), for example, apply normalising flows to learn
the parameters of Bernstein polynomials, which are in turn used to create a
probabilistic forecast. Moreover, Rasul et al. (2020) combine normalising
flows with recurrent neural networks to create probabilistic forecasts.
Normalising flows are also combined with quantile regression networks and
copulas (Wen and Torkkola, 2019), or used to create a conditional
approximation of a Gaussian mixture model (Jamgochian et al., 2022) to improve
the accuracy of the resulting probabilistic forecasts. Whilst these methods
are all effective, normalising flows are used to enrich existing complex
probabilistic forecasting methods, but not to provide probabilistic forecasts
themselves.

An alternative method, that directly uses normalising flows in the context of
probabilistic forecasts, is to learn multi-dimensional distributions of
electricity price differences to predict the trajectory of intraday
electricity prices (Cramer et al., 2022a). Similarly, normalising flows may be
applied multiple times to generate scenario-based probabilistic forecasts
(Dumas et al., 2022; Zhang and Zhang, 2019; Ge et al., 2020), or to create a
proxy for weather ensemble prediction systems based on numerical weather
prediction models (Fanfarillo et al., 2021). These methods use the generative
nature of normalising flows to create multiple predictions drawn from the same
distribution. However, the forecasts are only probabilistic as an ensemble and
each individual forecast is deterministic. Furthermore, these forecasts rely
on the assumption that the underlying learned distribution remains constant,
and they do not always take external factors into account. Finally, these
methods all focus on directly creating probabilistic forecasts, and not on
enabling probabilistic forecasts based on existing arbitrary deterministic
forecasts. Therefore, such methods cannot be applied to create probabilistic
forecast from existing, well designed, deterministic forecasts.

#### Research Gap

Altogether, we identify a lack of existing work that directly creates
probabilistic forecasts from arbitrary deterministic forecasts, without
modifying the existing deterministic forecasting method or making potentially
unrealistic statistical assumptions. In the present article, we, therefore,
aim to fill this research gap by presenting an easy-to-use approach which we
describe in the following section.

##  3 Creating Probabilistic Forecasts with a cINN

To create probabilistic forecasts from arbitrary deterministic forecasts, we
directly apply the uncertainty in the underlying time series. This uncertainty
usually reflects the inherent randomness or unpredictability of the measured
underlying system. However, this underlying system typically generates
observations of an unknown distribution. Thus, it is challenging to directly
include the corresponding uncertainty in a forecast.

To solve this challenge, we aim to find a bijective mapping from the unknown
distribution to a known and tractable distribution. This bijective mapping
should also consider exogenous features, as shown in Figure 1. With such a
mapping $g$, we can map a deterministic forecast from the unknown distribution
to its representation in the known and tractable distribution. In the known
and tractable distribution, we can analyse the neighbourhood of this
representation and can gain information about its uncertainty. Finally, we can
map this uncertainty information back to the unknown distribution using the
inverse mapping $g^{-1}$ to create probabilistic forecasts.

In this section, we first mathematically show the validity of our approach
that uses the distribution of the underlying data to include uncertainty in a
deterministic forecast. We then explain how to apply our approach with a cINN,
starting with the training of this cINN, before describing how we generate
probabilistic forecasts using arbitrary deterministic forecasts.

Historical dataExogenous featuresDeterministic base forecaster$g$Analyse
neighbourhood of forecast$g^{-1}$Probabilistic forecastBijectivemapping (cINN)
Figure 1: Overview of the proposed approach. Exogenous features and historical
data are used as inputs for an arbitrary deterministic base forecaster. The
resulting deterministic forecast is combined with the exogenous features as
inputs to a bijective mapping realised by a cINN. This mapping creates a
representation of the forecast in a known and tractable distribution. We
analyse the neighbourhood of this known and tractable representation to gain
information about its uncertainty. Finally, we map this representation back to
the unknown distribution to create a probabilistic forecast.

###  3.1 Including Uncertainty from the Underlying Distribution of the Data

To formally show how a bijective mapping from an unknown distribution in a
known and tractable distribution enables probabilistic forecasts using
arbitrary deterministic forecasts, we first show that such a mapping exists.
Given the existence of this mapping, we show that such a bijective mapping
guarantees the equivalence of the uncertainty in the image and the inverse
image of the considered mapping.

#### Bijective Mapping

To introduce the bijective mapping, let us consider a times series
$\mathbf{y}=\\{y_{t}\\}_{t\in T}$ consisting of $T$ observations as
realisations of a random variable $Y\sim f_{Y}(\mathbf{y})$ with a probability
density function (PDF)$f_{Y}(\mathbf{y})$ in the realisation space
$\mathbb{Y}$. Furthermore, we have a bijective mapping
$g:\mathbb{Y}\rightarrow\mathbb{Z}$ from the realisation space $\mathbb{Y}$ to
the space of the tractable distribution $\mathbb{Z}$ where $\mathbf{y}\mapsto
g(\mathbf{y},\circ)=\mathbf{z}$, and $g$ being a continuously differentiable
function.111The function $g$ can include further parameters apart from
$\mathbf{y}$, such as exogenous information. These further parameters are
indicated via $\circ$. To calculate the PDF $f_{Z}(\mathbf{z})$ in terms of
$f_{Y}(\mathbf{y})$, we can apply the change of variables formula (Murphy,
2023), i.e.

$f_{Z}(\mathbf{z})=f_{Y}(g^{-1}(\mathbf{z,\circ}))\left|\det\left(\frac{\partial
g^{-1}}{\partial\mathbf{z}}\right)\right|,$ (1)

where $\frac{\partial g^{-1}}{\partial\mathbf{z}}$ is the Jacobian matrix.
Since $g$ is bijective, this equation describes a bijective mapping from the
unknown distribution $f_{Y}(\mathbf{y})$ to the known and tractable
distribution $f_{Z}(\mathbf{z})$. Therefore, the change of variable formula
provides us with the required mapping.

#### Equivalence of Uncertainty

After introducing the bijective mapping, we need to show the equivalence of
the uncertainty in the unknown distribution and known tractable distribution
when applying Equation 1. More specifically, we show the equivalence of
quantiles in both the realisation space and the tractable distribution space,
since quantiles serve as a non-parametric representation of the uncertainty.

To show this equivalence, we first consider the cumulative distribution
function (CDF)of the random variable $Z=g(Y,\circ)\sim f_{Z}(\mathbf{z})$,
defined as

$F_{Z}(\mathbf{z})=\int_{-\infty}^{\mathbf{z}}f_{Z}(\mathbf{u})d\mathbf{u}.$
(2)

If we use the expression for $f_{Z}(\mathbf{z})$ from the change of variables
formula (Equation 1) in the definition of the CDF (Equation 2), we obtain

$F_{Z}(\mathbf{z})=\int_{-\infty}^{\mathbf{z}}f_{Y}(g^{-1}(\mathbf{u},\circ))\left|\det\left(\frac{\partial
g^{-1}}{\partial\mathbf{u}}\right)\right|d\mathbf{u},$ (3)

describing the CDF of $F_{Z}(\mathbf{z})$ in terms of the CDF
$F_{Y}(\mathbf{y})$. Since $g$ is per definition a continuously differentiable
function, we can apply integration by substitution for multiple variables to
rewrite Equation 3 as

$F_{Z}(\mathbf{z})=\int_{-\infty}^{g^{-1}(\mathbf{z})}f_{Y}(\mathbf{v})d\mathbf{v}=F_{Y}(g^{-1}(\mathbf{z},\circ)),$
(4)

which is simply the CDF of $Y$ evaluated at the inverse of $g$. Further, the
quantiles $\mathbf{z}_{\alpha}$ of $Z$ are defined by the inverse of the CDF,
i.e.

$\displaystyle\mathbf{z}_{\alpha}=F_{Z}^{-1}(\alpha)$
$\displaystyle=\inf{\mathbf{z}\mid F_{Z}(\mathbf{z})\geq\alpha}$
$\displaystyle=\inf{\mathbf{z}\mid F_{Y}(g^{-1}(\mathbf{z},\circ))\geq\alpha}$

where $\inf$ refers to the infimum, the smallest value of $\mathbf{z}$ that
fulfils the condition, and $\alpha\in[0,1]$ is the considered quantile.
Consequently, if we know that the $\alpha$ quantile of $F_{Z}$ is
$\mathbf{z}_{\alpha}$, then we can also calculate the $\alpha$ quantile of
$F_{Y}$ as $g^{-1}(\mathbf{z}_{\alpha},\circ)=\mathbf{y}_{\alpha}$. From this
follows an equivalence between the quantiles of $Z$ and the quantiles of $Y$,
which implies an equivalence in the uncertainty.

Given the mathematical equivalence of the uncertainty in the two considered
distributions, we can include uncertainty in a tractable and known
distribution $f_{Z}(\mathbf{z})$ and use the inverse mapping
$g^{-1}:\mathbb{Z}\rightarrow\mathbb{Y}$ to map this uncertainty to the
original distribution $f_{Y}(\mathbf{y})$. To realise this bijective mapping
$g$, we use a cINN (Ardizzone et al., 2019; Heidrich et al., 2022). A cINN is
a neural network that consists of multiple specially designed conditional
affine coupling blocks (Ardizzone et al., 2019). As shown by Ardizzone et al.
(2019), these coupling blocks ensure that the mapping
$g:\mathbb{Y}\rightarrow\mathbb{Z}$ learnt by the cINN is bijective.
Furthermore, with the conditional information the cINN is able to consider
additional information, such as exogenous features or calendar information,
when learning the mapping (Ardizzone et al., 2019; Heidrich et al., 2022). As
a result, the cINN learns an approximation of $f_{Z}(\mathbf{z})$ and a
mapping $g$, which is per definition bijective, thus ensuring we can apply
Equation 1 as described previously.

###  3.2 Applying our Approach

In the following, we describe how we realise the inclusion of uncertainty with
a cINN.222The implementation will be made available after the acceptance of
the paper. We first detail how we train a cINN that learns the distribution of
the underlying data. Second, we describe how we use this trained cINN to
generate probabilistic forecasts.

#### Training

We use a cINN to realise the continuous differentiable function $g$ described
above. In addition to the original realisation $\mathbf{y}$, we also consider
conditional information $\mathbf{c}$ as an input to the function $g$. This
conditional information always includes calendar features such as time of the
day, and day of the week, but depending on the time series may also include
additional exogenous features that are available for the forecast period. The
aim of the training is to ensure that the cINN learns the function $g$, so
that resulting realisations $\mathbf{z}=g(\mathbf{y},\mathbf{c})$ follow a
known and tractable latent space distribution $f_{Z}(\mathbf{z})$. Therefore,
we apply the change of variables formula in the maximum likelihood loss
function

$\mathcal{L}=\frac{\parallel
g(\mathbf{y};\mathbf{c},\theta)\parallel^{2}_{2}}{2}-\log\mid J\mid,$ (5)

where $J=\det(\partial g/\partial\mathbf{y})$ is the determinant of the
Jacobian, and $\theta$ is the set of all trainable parameters (Ardizzone et
al., 2019; Heidrich et al., 2022). Training a cINN with this loss function
results in a network with the maximum likelihood parameters
$\hat{\theta}_{\text{ML}}$ and ensures that the realised latent space
distribution $f_{Z}(\mathbf{z})$ follows a Gaussian distribution (Ardizzone et
al., 2019).

Note that to apply our approach, a deterministic base forecaster must also be
trained. However, since our approach simply enables probabilistic forecasts
based on arbitrary deterministic forecasts, this deterministic base forecaster
can be trained as usual. Thus, we refrain from a more detailed description of
the training of the deterministic base forecaster.

#### Forecasting

To generate probabilistic forecasts, we begin with the output of a
deterministic base forecaster $\hat{\mathbf{y}}$. We combine this output with
the associated conditional information $\mathbf{c}$ and pass it on through the
trained cINN to obtain a latent space representation of the output, i.e.

$\hat{\mathbf{z}}=g(\hat{\mathbf{y}},\mathbf{c},\hat{\theta}_{\text{ML}}).$
(6)

Given this latent space representation of the deterministic forecast, we
explore the uncertainty in the neighbourhood of the forecast with

$\tilde{\mathbf{z}}_{i}=\hat{\mathbf{z}}+\mathbf{r}_{i},\quad
i=1,\dots,I,\,\,\mathbf{r}_{i}\sim\mathcal{N}(0,\sigma).$ (7)

Using Equation 7, we select a random noise $\mathbf{r}_{i}$ from a standard
normal distribution with mean $0$ and variance $\sigma$ and add this noise to
the realisation $\hat{\mathbf{z}}$. We define the variance used for the
sampling process $\sigma$ as the _sampling hyperparameter_ , which must be
manually optimised using an evaluation metric. Based on the selected $\sigma$,
we repeat the sampling process $I$ times to obtain multiple realisations of
$\tilde{\mathbf{z}}_{i}$ that are all similar but not identical to the
original forecast in the known latent space distribution. This set of
realisations provides a representation of the uncertainty in the neighbourhood
of the forecast. Due to the equivalence of uncertainty in both spaces shown in
Section 3.1, we then process each of these samples through the backward pass
of the cINN to include uncertainty around the original representation of the
deterministic forecast. Finally, we calculate the quantiles of these samples
in the original realisation space. The resulting quantiles represent a
probabilistic forecast, derived from the original arbitrary forecast.

##  4 Experimental Setup

This section describes the experimental setup we use to evaluate our approach
of creating probabilistic forecasts based on arbitrary deterministic
forecasts. We first introduce the data used, before explaining the evaluation
metrics. Furthermore, we describe the selected base forecasters to generate
the deterministic forecasts, introduce the used state-of-the-art probabilistic
benchmarks, and detail the implementation of the used cINN.

###  4.1 Data

We evaluate our proposed approach on four different data sets. In this
section, we briefly introduce each of these data sets before we describe their
preprocessing.

The first considered data set is _Electricity_ , namely the UCI Electricity
Load
Dataset333https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
(Dua and Graff, 2019). From this data set, we select the time series _MT_158_
and resample it to an hourly resolution. The second considered data set,
_Bike_ , contains hourly records of rented bikes from the UCI Bikesharing
Dataset (Fanaee-T and Gama, 2014; Dua and Graff,
2019).444https://archive.ics.uci.edu/ml/datasets/bike+sharing+dataset The
third data set, _OPSD_ , is a subset of hourly electricity load values for the
state of Baden-Württemberg in Germany, taken from the Open Power Systems Data
portal.555https://open-power-system-data.org/ The fourth data set, _Price_ ,
is a subset of the data provided for Task 1 in the electricity price track of
the Global Energy Forecasting Competition 2014 (GEFCom2014). The data includes
zonal electricity price values recorded for a single location at an hourly
resolution (Hong et al., 2016). To enable a thorough comparison of all models,
we do not attempt to forecast the single day required for Task 1, but instead,
consider a longer period.

We normalise each of the above data sets before creating separate test,
validation, and training subsets for the training and testing of our approach.
An overview of these splits and the exogenous variables considered for each
data set is presented in Table 5.

###  4.2 Evaluation Metrics

To evaluate our approach comprehensively, we consider several evaluation
metrics, both for deterministic and probabilistic forecasts. In the following,
we briefly present these metrics in the order they appear in the evaluation.

#### Mean Absolute Error

To evaluate the quality of the deterministic base forecasters, we consider the
Mean Absolute Error (MAE). The MAE is given by

$\operatorname{MAE}(y,\hat{y})=\frac{1}{n}\sum\limits_{i=1}^{n}|y_{i}-\hat{y}_{i}|$
(8)

with a true value $y_{i}$, a forecast value $\hat{y}_{i}$, and $n$
observations.

#### Continuous Ranked Probability Score

To evaluate the quality of the probabilistic forecasts, we consider the
Continuous Ranked Probability Score (CRPS)(Matheson and Winkler, 1976). The
CRPS is a proper scoring rule that measures the calibration and sharpness of a
predictive cumulative distribution function $F$ (Gneiting and Raftery, 2007).
The CRPS is defined as

$\operatorname{CRPS}(F,y)=\int_{\mathbb{R}}\left(F(z)-\text{1}\\{y\leq
z\\}\right)^{2}dz,$ (9)

where $\text{1}\\{y\leq z\\}$ is the indicator function which is one if $y\leq
z$ and otherwise zero. Since our approach as well as the benchmarks provide
samples drawn from a distribution, we use the sample-based variant of the CRPS
implemented in the properscoring
library.666https://github.com/properscoring/properscoring

#### CRPS Skill Score

To easily determine whether our approach leads to better probabilistic
forecasts, we measure the improvement in the CRPS compared to a baseline. The
resulting CRPS skill score is defined as

$\text{CRPS Skill
Score}=\left(1-\frac{\operatorname{CRPS}_{\text{Forecast}}}{\operatorname{CRPS_{\text{Baseline}}}}\right)\cdot
100,$ (10)

where $\operatorname{CRPS}_{\text{Forecast}}$ is the CRPS of the considered
forecasting method and $\operatorname{CRPS_{\text{Baseline}}}$ the CRPS of a
selected baseline.

#### Normalised Prediction Interval Width

To measure the sharpness of the probabilistic forecasts, we consider the
average normalised width of various Prediction Intervals (PIs). The normalised
average width of the $\beta$-PI is defined as

$|\operatorname{PI}(\beta)|=\frac{1}{\bar{y}}\left(\frac{1}{n}\sum\limits_{i=1}^{n}|\hat{y}_{i,\frac{1+\beta}{2}}-\hat{y}_{i,\frac{1-\beta}{2}}|\right),$
(11)

where $\hat{y}_{i,\frac{1+\beta}{2}}$ is the predicted upper quantile,
$\hat{y}_{i,\frac{1-\beta}{2}}$ the predicted lower quantile for the forecast
value $\hat{y}_{i}$, and $\bar{y}$ the mean of the target time series. We
consider the normalised prediction interval width to enable a comparison
between different data sets.

#### Training Time

To evaluate the computational cost, we measure the training time. Thereby, we
measure the training time of the cINN excluding the training time of the
deterministic base forecaster. We compare this to the training time of the
probabilistic benchmark methods. The training time is measured on a PC with an
Intel 3.00 GHz i7-9700 CPU, 32 GB RAM, and no GPU.

#### Pinball Loss Improvement

In order to evaluate our approach in the recreated GEFCom2014 competition, we
consider the scoring mechanism used in this competition. This mechanism relies
on the Pinball Loss (PL), a scoring rule that minimises the loss when issuing
a point forecast for the $\alpha$-quantile (Gneiting et al., 2022). For a set
of considered quantiles $Q=[0.01,\dots,0.99]$, the PL is calculated with

$\operatorname{PL}=\frac{1}{n\mid Q\mid}\sum_{\alpha\in
Q}\sum\limits_{i=1}^{n}\begin{cases}(y_{i}-\hat{y}_{i,\alpha})\cdot\alpha\quad
y_{i}\geq\hat{y}_{i,\alpha}\\\
(\hat{y}_{i,\alpha}-y_{i})\cdot(1-\alpha)\quad\hat{y}_{i,\alpha}>y,\end{cases}$
(12)

where $y_{i}$ is the true value and $\hat{y}_{i,\alpha}$ is the quantile
forecast for the quantile $\alpha$. For the GEFCom2014, the relative
improvement of the PL compared to a given baseline forecast is considered,
i.e.

$\operatorname{PL}_{\%}=\frac{\operatorname{PL}_{\text{Forecast}}}{\operatorname{PL}_{\text{Baseline}}},$
(13)

where $\operatorname{PL}_{\text{Forecast}}$ is the PL for the considered
forecast and $\operatorname{PL}_{\text{Baseline}}$ the PL for the baseline
provided in the GEFCom2014.

###  4.3 Selected Base Forecasters

Our proposed approach can be applied to forecasts from arbitrary deterministic
forecasters. Thus, we evaluate our approach on four simple and two state-of-
the-art deterministic forecasting methods. As simple base deterministic
forecasters we consider a _Random Forest (RF)_, a _Linear Regression (LR)_, a
_Feed-Forward Neural Network (NN)_, and the _eXtreme Gradient Boosting
(XGBoost)_ Regressor. The two state-of-the-art deterministic base forecasters
are _Neural Hierarchical Interpolation for Time Series Forecasting (N-HITS)_
(Challu et al., 2022) and _Temporal Fusion Transformer (TFT)_ (Lim et al.,
2021). We provide implementation details for each of the selected
deterministic base forecasters in Table 6.

As inputs, all base forecasting methods receive $24$ hours of historical
information for the target time series and exogenous features for the forecast
horizon of $24$ hours. The exogenous features comprise calendar information
and, depending on the data set, further exogenous variables, as shown in Table
5.

When applying the base forecasters with the cINN to create probabilistic
forecasts, we manually select the sampling parameter $\sigma$ that minimises
the CRPS on the validation data set. An overview of the selected sampling
parameters is presented in Table 7. Furthermore, all selected base
deterministic forecasters are implemented in a pipeline with pyWATTS (Heidrich
et al., 2021)777https://github.com/KIT-IAI/pyWATTS.

###  4.4 Probabilistic Benchmarks

To assess the quality of the probabilistic forecasts created with our
approach, we compare them to three state-of-the-art probabilistic benchmark
forecasting methods.

The first considered benchmark method is _DeepAR_ (Salinas et al., 2020),
which is an autoregressive recurrent neural network-based approach for
probabilistic forecasting. We implement DeepAR using the PyTorch Forecasting
library888https://pytorch-forecasting.readthedocs.io/en/stable/api/
pytorch_forecasting.models.deepar.DeepAR.html.

The second benchmark method is a _Quantile Regression Neural Network (QRNN)_.
It trains a NN to directly forecast a selected quantile, or multiple
quantiles, instead of the mean or median (Koenker et al., 2017). To realise
the QRNN, we use a separate simple feed-forward NN to forecast each of the
selected quantiles training each NN with the appropriate pinball loss
function. The QRNN is implemented using
TensorFlow999https://www.tensorflow.org/ with the Keras101010https://keras.io/
library and the pinball loss function.

The third benchmark method uses the _Nearest Neighbour Quantile Filter (NNQF)_
proposed by González Ordiano, J. A. et al. (2020). Similar to the QRNN, this
method also forecasts quantiles. However, instead of using a custom quantile
loss function to directly learn the quantiles, the NNQF finds similar values
for each time step based on similarity in the target variable to determine
quantiles in the data. A forecasting method is then trained to predict these
calculated quantiles (González Ordiano, J. A. et al., 2020). To realise the
NNQF, we use a multi-layer feed-forward NN with one output per quantile, which
is implemented using sklearn (Pedregosa et al., 2011) and pyWATTS (Heidrich et
al., 2021).

###  4.5 Used cINN

In the evaluation, we use the same cINN architecture (see Table 8) for each of
the considered data sets. It is based on GLOW coupling layers that consider
conditional input (Kingma and Dhariwal, 2018). Similar to Ardizzone et al.
(2019); Heidrich et al. (2022), the conditional input is provided by a fully
connected NN, which uses the same exogenous information available to the base
forecaster as conditional information (see Table 5). We detail the
implementation information for the used cINN in A.4. We implement the cINN in
a pipeline with pyWATTS (Heidrich et al., 2021).

##  5 Evaluation

Given the experimental setup, we present the evaluation of the proposed
approach in this section. We first report the deterministic performance of
each base forecaster since they affect the quality of the resulting
probabilistic forecasts. We then consider the improvement of the proposed
approach over a baseline that assumes a Gaussian error distribution, its
performance compared to state-of-the-art probabilistic benchmark forecasts,
and how it would have placed in the GEFCom2014.

###  5.1 Deterministic Results

Since the quality of the deterministic base forecaster influences the
resulting probabilistic forecast when combined with a cINN, we briefly
evaluate the deterministic performance of the selected base forecasters. We
report the average MAE (Equation 8) over five runs in Table 1.

Table 1: The average MAE (Equation 8) on the test data sets across five runs for each of the considered base forecasters introduced in Section 4.3. The lowest error for each data set is highlighted in bold. Data set | RF | LR | NN | XGBoost | N-HITS | TFT
---|---|---|---|---|---|---
Electricity | 0.2566 | 0.3892 | 0.2535 | 0.2502 | 0.2724 | 0.2549
Bike | 0.3076 | 0.4223 | 0.2998 | 0.2885 | 0.4359 | 0.3834
OPSD | 0.1549 | 0.2098 | 0.1272 | 0.1273 | 0.1732 | 0.1514
Price | 0.2863 | 0.2585 | 0.2595 | 0.2639 | 0.2983 | 0.3316

We see that different deterministic base forecasters perform best depending on
the data set considered. However, either the NN or XGBoost achieve the lowest
error for three of the four data sets. Furthermore, the size of the error is
usually similar for all forecasters in one data set. Finally, the advanced
base forecasters such as NHiTS and the TFT, often perform similarly or worse
than the four simple base forecasters.

###  5.2 Improvement Over Baseline

To assess the improvement obtained with our approach, we compare it to a
baseline. The baseline assumes that the errors from the deterministic forecast
follow a Gaussian distribution and generates probabilistic forecasts based on
this assumption. We plot the CRPS Skill Score (Equation 10) for each base
forecaster and every data set in Figure 2 (see Table 10 for the absolute
CRPS).

RF LR NN XGBoost N-HITS TFT $0$$10$$20$$30$CRPS Skill Score in
%ElectricityBikeOPSDPrice Figure 2: The CRPS Skill Score (Equation 10) for all
base forecasters introduced in Section 4.3 and the four considered data sets.
The skill score indicates the average improvement on the test data set over
five runs compared to the baseline which assumes a Gaussian distribution with
the same base forecaster. The error bars indicate the variance across the
runs.

We observe that our approach noticeably improves the performance of all base
forecasters across all data sets. The improvement is most noticeable for the
electricity and price data sets, where the proposed approach achieves an
average improvement of $17.40\%$ and $20.54\%$ respectively. The largest
improvement of $38.60\%$ has the LR base forecaster on the price data set. The
improvement is the smallest for the OPSD data set, with our approach only
yielding an average improvement of $5.74\%$ and the smallest improvement of
$0.59\%$ for the RF base forecaster. Furthermore, we observe that the
improvement is dependent on both the data set and the base forecaster used.
The observed improvements in CRPS vary for a given base forecaster across the
data sets, but also across all base forecasters for each data set.

###  5.3 Comparison to Benchmarks

In addition to the comparison to baseline assuming a Gaussian distribution, we
also compare our approach to the selected state-of-the-art probabilistic
forecasting benchmarks with respect to the quality of the probabilistic
forecast, its sharpness, and its training time.

Table 2: The CRPS (Equation 9) calculated on the test data set and averaged
over five runs. The three base forecasters from Section 4.3 with the best
performance using the cINN in our approach are selected and compared against
the three probabilistic benchmark forecasting methods from Section 4.4 for
each data set. The overall best probabilistic forecast is highlighted in bold.

Data Set | cINN | Probabilistic Benchmark
---|---|---
| Best | 2nd Best | 3rd Best | DeepAR | QRNN | NNQF
Electricity | 0.19251 |  0.19602 |  0.19743 | 0.2794 | 0.2020 | 0.2298
Bike | 0.18561 |  0.22283 |  0.23082 | 0.2317 | 0.3324 | 0.3808
OPSD | 0.09773 |  0.10042 |  0.11591 | 0.1624 | 0.1554 | 0.1679
Price | 0.18331 |  0.18374 |  0.19325 | 0.2023 | 0.2144 | 0.2576

  * 1

cINN-TFT

  * 2

cINN-NN

  * 3

cINN-XGBoost

  * 4

cINN-N-HITS

  * 5

cINN-LR

#### Quality

Concerning the quality of the probabilistic forecast, we select the three
best-performing base forecasters in our approach and compare their performance
to the performance of the selected probabilistic benchmark forecasting
methods. Table 2 reports the resulting mean CRPS (Equation 9) on the test data
set averaged over five runs for each data set.

We observe that the best-performing base forecaster with a cINN outperforms
all probabilistic benchmarks for all data sets. Actually, in all four data
sets, the three best-performing base forecasters combined with a cINN
outperform all benchmarks.

Table 3: The mean normalised width of the 98%, 70%, and 40% PIs (Equation 11) for the cINN with the best-performing base forecaster from Section 4.3 and the probabilistic benchmark forecasting methods from Section 4.4 over five runs. A low value indicates a sharp prediction interval. Prediction Interval | Data Set | Prediction Interval Width
---|---|---
|  | cINN | DeepAR | QRNN | NNQF
| Electricity | 0.87 | 1.57 | 1.52 | 1.54
| Bike | 0.70 | 1.31 | 1.79 | 1.61
98% | OPSD | 0.09 | 0.22 | 0.18 | 0.19
| Price | 0.50 | 0.53 | 0.86 | 0.83
| Electricity | 0.42 | 0.65 | 0.65 | 0.68
| Bike | 0.32 | 0.57 | 0.59 | 0.76
70% | OPSD | 0.04 | 0.09 | 0.06 | 0.09
| Price | 0.22 | 0.23 | 0.30 | 0.37
| Electricity | 0.21 | 0.32 | 0.27 | 0.36
| Bike | 0.16 | 0.29 | 0.23 | 0.37
40% | OPSD | 0.02 | 0.04 | 0.03 | 0.04
| Price | 0.11 | 0.12 | 0.13 | 0.19

#### Sharpness

Regarding the sharpness of the probabilistic forecast, we compare normalised
PI width (Equation 11) of the best performing base forecaster combined with
the cINN and all probabilistic benchmark forecasting methods. Table 3 reports
the mean normalised width of the 98%, 70%, and 40% PIs on the test data set
averaged over five runs.

We observe that for all data sets and each of the three considered prediction
intervals, the forecasts generated with the cINN result in the narrowest PIs.
This observation is confirmed by considering an exemplary graphical depiction
of the PIs for a forecast period of $24$ hours from the bike data set in
Figure 3. In this example, the PIs from the cINN are narrower than those from
all probabilistic benchmark forecasting methods and also accurately follow the
true values.

$0$$5$$10$$15$$20$$0$$500$$1{,}000$Bikes SharedcINN-
TFT$0$$5$$10$$15$$20$$0$$500$$1{,}000$DeepAR$0$$5$$10$$15$$20$$0$$500$$1{,}000$Bikes
SharedQRNN$0$$5$$10$$15$$20$$0$$500$$1{,}000$NNQF Figure 3: True values
(blue), $98\%$ PI (light blue), $70\%$ PI (light yellow), and $40\%$ PI (light
green) for an exemplary $24$ hour forecast period on the test set of the bike
data set. cINN DeepAR QRNN NNQF $0$$500$$1{,}000$Training Time in
SecondsElectricityBikeOPSDPrice Figure 4: The average training time across
five training runs for the cINN used to create probabilistic forecasts and all
probabilistic benchmark forecasting methods from Section 4.4. The error bars
indicate the variance in the training time across the runs. Note that only the
training time for the cINN is recorded, the additional time required to train
a deterministic base forecaster is not included.

#### Training Time

Finally, we compare the training time of our approach and the probabilistic
benchmark forecasting methods. In Figure 4, we report the average training
time across all five training runs and the variance across the runs.
Importantly, we only report the training time for the cINN and not the
associated base forecaster.

We observe that the cINN requires significantly less training time than DeepAR
or the QRNN, and a similar time as the NNQF.

###  5.4 GEFCom2014 Probabilistic Price Forecasting

We lastly evaluate the proposed approach by retrospectively determining its
placement in the GEFCom2014. The GEFCom2014 was a probabilistic energy
forecasting competition with four different forecasting tracks, namely load,
price, wind, and solar forecasting (Hong et al., 2016). For the evaluation, we
recreate the setup of the GEFCom2014 price forecasting track in which 14 teams
competed and compare the performance of our approach to the leading entrants
from the competition. This comparison is based on the scoring mechanism from
GEFCom2014 that considers the final twelve of fifteen tasks, each creating
24-hour forecasts with 99% quantiles. Given the pinball loss improvement
$\operatorname{PL}_{\%}$ for each task, the final ranking is determined as the
average relative pinball loss across all tasks weighted by the task number
(Hong et al., 2016). For our approach, we apply all previous deterministic
base learners and select the best-performing sampling parameter over the first
three non-evaluated tasks and use this for all remaining tasks. The final
weighted relative pinball loss and the resulting rank of our approach are
shown in Table 4 (see Table 11 for the results per task).

Table 4: The overall weighted pinball loss improvement all tasks and final
rank in the GEFCom2014 price forecasting challenge for all base forecasters
from Section 4.3 combined with the cINN. The weighted pinball loss improvement
indicates the improvement of a given method over the GEFCom2014 baseline
forecast weighted by the task number. The rank indicates the placement in
GEFCom2014 according to the weighted relative pinball loss.

| Average weighted pinball loss improvement | Rank
---|---|---
cINN-RF | 65.9 | 41
cINN-LR | 65.7 | 41
cINN-NN | 61.5 | 91
cINN-XGBoost | 65.0 | 41
cINN-N-HITS | 62.9 | 61
cINN-TFT | 67.4 | 3
Tololo (Gaillard et al., 2016) | 71.7 | 1
Team Poland (Maciejowska and Nowotarski, 2016) | 67.7 | 2
GMD (Dudek, 2016) | 67.1 | 31

  * 1

This ranking is determined by how each individual method would have placed in
the GEFCom2014 price forecasting challenge in 2014 and, therefore, assumes
that only one of the presented methods is considered at a time when creating
the respective ranking.

Overall, our approach with a cINN and various base forecasters performs well.
With simple deterministic forecasting methods such as RF regression and LR, we
achieve an average weighted pinball loss improvement that would have placed
these methods within the top five of the competition. Furthermore, with a more
advanced TFT base forecaster, we achieve an average weighted pinball loss
improvement that would have resulted in a third-place finish.

##  6 Discussion

In this section, we first discuss our results and the implications these
present, before we highlight some of the key insights gained from the
evaluation.

###  6.1 Results

Regarding the results of our evaluation, we first discuss the forecasting
performance, before considering the training time and the results in the
GEFCom2014.

#### Forecasting Performance

Overall, our approach generates high-quality probabilistic forecasts. The
approach not only outperforms a baseline that assumes Gaussian distributed
errors, but also generally performs better in terms of CRPS than three
selected state-of-the-art probabilistic benchmark forecasting methods. Our
approach also results in the narrowest PIs. However, despite being optimal
according to CRPS, the PIs created by our approach are quite narrow, which may
not always be ideal. Therefore, it may be interesting to improve the
calibration of the PIs generated by our approach.

#### Training Time

The training time of our approach is negligible in comparison to DeepAR and
the QRNN. However, we only report the training time of the used cINN and not
the associated base forecasters. In practice, we expect that an already
existing trained deterministic base forecaster will be used. However, if this
is not the case, the training time of the base forecaster must be taken into
account, since these are highly dependent on the base forecaster used.
Furthermore, the training time depends on the implementation used. Therefore,
future work should consider how the reported training times vary if
alternative libraries or programming languages are considered.

#### GEFCom2014

Not only does our approach perform competitively in the considered track of
the GEFCom2014, but several factors undercut its true performance. All top-
placing contestants in the competition perform specialised operations to
improve forecasting performance, i.e. peak pre-processing (Gaillard et al.,
2016), filtering methods to weight certain days higher (Maciejowska and
Nowotarski, 2016), or tailored training data periods to improve performance
(Dudek, 2016). In contrast, we consider all available data for training,
refrain from complex pre-processing steps, and only use the default
hyperparameters for the base forecasters. Furthermore, the true value of our
approach is its ability to enable simple deterministic base forecasters such
as a LR or RF, to rank within the top five compared to the original entrants.
Finally, different base forecaster perform better for different tasks.
Therefore, we expect an ensemble method that automatically selects the best
base forecaster for a given task to deliver even better performance.

###  6.2 Insights

In addition to the results, there are a few insights regarding the sampling
parameter and flexible nature of our approach, which we discuss here.

#### Sampling Parameter

The sampling parameter $\sigma$ is currently manually selected to create
optimal forecasts according to CRPS. However, by varying this sampling
parameter, it is possible to create different probabilistic forecasts which
follow the same general shape but vary in the amount of uncertainty
considered. Therefore, it may be interesting to investigate methods to
automatically select an optimal sampling parameter given the observed data, a
selected base forecaster, and a specific evaluation metric.

#### Flexible Nature

In the present article, we focus on evaluating the probabilistic forecasting
performance of a single selected base forecaster combined with the cINN.
However, our proposed approach allows for a far more flexible application.
Given a data set, we must only train the cINN once before we apply it to
generate probabilistic forecasts based on any arbitrary deterministic
forecast. Furthermore, our proposed approach can be applied to create
probabilistic forecasts without extending the existing deterministic
forecasting method or applying extensive statistical knowledge. This is
advantageous if a tailored deterministic forecasting method for a given
scenario exists. In such a setting our approach will allow them to create
probabilistic forecasts based on this tailored method without any further
modification.

##  7 Conclusion

In the present article, we introduce an easy-to-use approach to create
probabilistic forecasts from arbitrary deterministic forecasts by using a cINN
to learn the underlying distribution of the time series data. Our approach
maps the underlying distribution of the data to a known and tractable
distribution before combining the uncertainty from this known and tractable
distribution with an arbitrary deterministic forecast to create probabilistic
forecasts.

We show that our approach outperforms a probabilistic baseline forecast, which
assumes that the errors of a deterministic forecast follow a Gaussian
distribution. Moreover, we compare our approach to three state-of-the-art
probabilistic benchmark forecasting methods, achieving the lowest CRPS and
probabilistic forecasts with the narrowest prediction intervals. Finally, we
recreate the GEFCom2014 and show that our approach enables simple
deterministic base forecasters to rank within the top five.

Our approach offers a solution to generate flexible probabilistic forecasts
based on arbitrary deterministic forecasts. In future work, this flexibility
should be further investigated by automating the selection of the sampling
parameter used to include uncertainty and considering how different metrics
for optimising this parameter affect the resulting forecasts. Furthermore, it
may be interesting to extend our approach to incorporate time series with
stochastic inputs such as ensemble prediction systems from numerical weather
prediction models.

## Acknowledgements

This project is funded by the Helmholtz Association’s Initiative and
Networking Fund through Helmholtz AI and the Helmholtz Association under the
Program “Energy System Design”.

##  Appendix A Implementation Details

###  A.1 Data Sets and Exogenous Features

Table 5: Overview of the data sets used including the exogenous features
considered, and the used train, validation, and test sets.

Data Set | Target |  Exogenous Features | Train | Validation | Test
---|---|---|---|---|---
Electricity | MT_158 |  Calendar1 | $\left[0,18000\right)$ | $\left[18000,20000\right)$ | $\left[20000,22000\right]$
Bike2 | cnt |  Calendar1, Temperature, Humidity, Windspeed, Weather Situation | $\left[0,12000\right)$ | $\left[12000,14000\right)$ | $\left[14000,17544\right]$
OPSD |  load_power _statistics |  Calendar1 | $\left[0,6000\right)$ | $\left[6000,7000\right)$ | $\left[7000,8400\right]$
Price | Zonal Price |  Calendar1, Forecast Total Load, Forecast Zonal Load | $\left[0,13000\right)$ | $\left[13000,15000\right)$ | $\left[15000,21528\right]$

  * 1

Sine- and cosine-encoded time of the day, the sine- and cosine-encoded month
of the year, and a Boolean that indicates whether the current day is a weekend
day or not.

  * 2

To create a time index for this data, we merge the columns _dteday_ and _hr_
and deal with missing values using linear interpolation.

###  A.2 Selected Base Forecasters

Table 6: Overview of the selected base forecasters from Section 4.3 used to
create deterministic forecasts.

Base Forecaster | Classification | Implementation Details | Library Used
---|---|---|---
RF | Statistical | Default Hyperparameters |  SKlearn1
LR | Statistical | Default Hyperparameters |  SKlearn1
NN |  Machine Learning |  Hidden Layers: 3 Layer Sizes: 90-64-32 Hidden Activation Function: relu Output Activation Function: linear Optimiser: Adam2 Batch Size: 100 Max Epochs: 100 |  Tensorflow3 Keras4
XGBoost |  Gradient Boosting | Default Hyperparameters |  XGBoost5
N-HITS | Deep Learning | Default Hyperparameters |  PyTorch Forecasting6
TFT | Deep Learning | Default Hyperparameters |  PyTorch Forecasting6

  * 1

(Pedregosa et al., 2011)

  * 2

(Kingma and Ba, 2015)

  * 3

https://www.tensorflow.org/

  * 4

https://keras.io/

  * 5

(Chen and Guestrin, 2016)

  * 6

https://pytorch-forecasting.readthedocs.io/en/stable/index.html

###  A.3 Selected Sampling Parameters for Evaluation

Table 7: The selected sampling parameter for each base forecaster from Section 4.3 and each data set used in the evaluation. Data Set | RF | LR | NN | XGBoost | N-HITS | TFT
---|---|---|---|---|---|---
Electricity | 0.600 | 0.500 | 0.500 | 0.500 | 0.600 | 0.600
Bike | 0.800 | 0.350 | 0.350 | 0.300 | 0.300 | 0.350
OPSD | 0.300 | 0.350 | 0.250 | 0.1250 | 0.250 | 0.300
Price | 0.750 | 0.750 | 0.700 | 0.450 | 0.850 | 0.850
GEFCom Competition | 0.950 | 1.050 | 0.900 | 0.500 | 0.750 | 1.050

###  A.4 Implementation details for the used cINN

When training the use cINN we use the Adam optimiser with a maximum of 100
Epochs. Furthermore, when sampling in the latent space to create probabilistic
forecasts we consider a sample size of 100.

Table 8: The architecture of the used cINN Parameter | Description
---|---
Layers per block | Glow coupling layer and random permutation
Subnetwork in block |  Fully connected, see Table 9
Number of blocks | 5
Conditional network |  Fully connected, see Table 9
Table 9: Implementation details of the subnetwork and the conditioning network
$q$ in the used cINN.

(a) Subnetwork Layer | Description
---|---
Input | [Output of previous coupling layer,
| conditional information]
1 | Dense 32 neurons; activation: tanh
2 | Dense 24 neurons; activation: linear

(b) Conditioning network Layer | Description
---|---
Input | [Calendar information,
| historical information,
| exogenous forecasts if available]
1 | Dense 8 neurons; activation: tanh
2 | Dense 4 neurons; activation: linear

##  Appendix B Improvement Over Baseline

Table 10: Comparison of the absolute CRPS between the baseline assuming a Gaussian distribution and the same base forecaster from Section 4.3 with the cINN. We report the mean CRPS averaged across five runs for the test data on all data sets. Base Forecaster | Variant | Electricity | Bike | OPSD | Price
---|---|---|---|---|---
RF | Baseline | 0.2128 | 0.2584 | 0.1259 | 0.2410
cINN | 0.1976 | 0.2322 | 0.1252 | 0.2223
LR | Baseline | 0.3487 | 0.3595 | 0.1853 | 0.3047
cINN | 0.3020 | 0.3265 | 0.1607 | 0.1932
NN | Baseline | 0.2496 | 0.2429 | 0.1034 | 0.2305
cINN | 0.1960 | 0.2308 | 0.1004 | 0.1954
XGBoost | Baseline | 0.2142 | 0.2390 | 0.0995 | 0.2240
cINN | 0.1960 | 0.2228 | 0.0977 | 0.2000
N-HITS | Baseline | 0.2950 | 0.2638 | 0.1434 | 0.2615
cINN | 0.2099 | 0.2534 | 0.1319 | 0.1837
TFT | Baseline | 0.2693 | 0.2150 | 0.1257 | 0.2389
cINN | 0.1925 | 0.1856 | 0.1159 | 0.1833

##  Appendix C Full GEFCom2014 Results

Table 11: GEFCom2014 results for each task. The performance of each base
forecaster from Section 4.3 combined with the cINN is reported, as well as the
performance of the best three entrants from 2014. For each task, pinball loss
improvement $\operatorname{PL}_{\%}$ compared to the GEFCom2014 baseline is
calculated. The final results are a linear weighted average of the pinball
loss improvement in each task, weighted by the task number.

Model | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 |  |
---|---|---|---|---|---|---|---|---
| PL | $\operatorname{PL}_{\%}$ | PL | $\operatorname{PL}_{\%}$ | PL | $\operatorname{PL}_{\%}$ | PL | $\operatorname{PL}_{\%}$ | PL | $\operatorname{PL}_{\%}$ | PL | $\operatorname{PL}_{\%}$ |  |
cINN-RF | 1.10 | 72.67 | 3.46 | 56.61 | 2.93 | 48.64 | 7.13 | 41.35 | 5.24 | 86.34 | 6.45 | 85.41 |  |
cINN-LR | 1.76 | 56.25 | 2.95 | 62.97 | 1.92 | 66.36 | 2.97 | 75.53 | 8.59 | 77.60 | 12.14 | 72.55 |  |
cINN-NN | 1.16 | 71.14 | 4.38 | 45.01 | 2.02 | 64.57 | 4.31 | 64.53 | 7.26 | 81.05 | 7.10 | 83.96 |  |
cINN-XGBoost | 1.70 | 57.69 | 3.53 | 55.76 | 2.74 | 51.90 | 5.21 | 57.12 | 6.08 | 84.14 | 7.65 | 82.71 |  |
cINN-N-HITS | 1.70 | 57.87 | 2.45 | 69.21 | 1.05 | 81.57 | 3.75 | 69.15 | 8.03 | 79.06 | 10.08 | 77.21 |  |
cINN-TFT | 1.22 | 69.80 | 2.50 | 68.64 | 1.16 | 79.67 | 3.08 | 74.65 | 6.30 | 83.57 | 9.20 | 79.21 |  |
Tololo2 | 1.71 | 57.62 | 1.45 | 81.80 | 1.10 | 80.65 | 2.02 | 83.40 | 9.16 | 76.11 | 4.68 | 89.41 |  |
Team Poland3 | 1.97 | 50.98 | 1.82 | 77.19 | 1.19 | 79.11 | 2.82 | 76.77 | 7.56 | 80.28 | 4.21 | 90.49 |  |
GMD4 | 3.73 | 7.48 | 1.78 | 77.63 | 0.92 | 83.84 | 5.09 | 58.12 | 6.21 | 83.79 | 3.83 | 91.35 |  |
| Task 7 | Task 8 | Task 9 | Task 10 | Task 11 | Task 12 | Overall
| PL | $\operatorname{PL}_{\%}$ | PL | $\operatorname{PL}_{\%}$ | PL | $\operatorname{PL}_{\%}$ | PL | $\operatorname{PL}_{\%}$ | PL | $\operatorname{PL}_{\%}$ | PL | $\operatorname{PL}_{\%}$ |  Total Weighted $\operatorname{PL}_{\%}$ | Rank
cINN-RF | 7.55 | 58.54 | 2.51 | 92.06 | 2.16 | 94.97 | 1.95 | 31.89 | 1.79 | 44.03 | 5.69 | 74.56 | _65.9_ | 41
cINN-LR | 12.66 | 30.53 | 1.85 | 94.14 | 2.39 | 94.44 | 1.50 | 47.36 | 1.66 | 48.33 | 7.54 | 66.32 | _65.7_ | 41
cINN-NN | 6.69 | 63.29 | 2.83 | 91.03 | 1.62 | 96.21 | 1.88 | 34.33 | 1.70 | 46.99 | 15.38 | 31.30 | _61.5_ | 91
cINN-XGBoost | 7.64 | 58.05 | 2.43 | 92.30 | 1.47 | 96.58 | 2.03 | 28.87 | 1.83 | 42.97 | 7.05 | 68.50 | _65.0_ | 41
cINN-N-HITS | 11.70 | 35.79 | 1.40 | 95.58 | 1.28 | 97.01 | 2.31 | 19.17 | 2.06 | 35.72 | 6.37 | 71.53 | _62.9_ | 61
cINN-TFT | 8.53 | 53.21 | 1.67 | 94.71 | 1.43 | 96.67 | 2.42 | 15.41 | 1.66 | 48.18 | 4.95 | 77.88 | _67.4_ | 3
Tololo2 | 1.60 | 91.24 | 0.75 | 97.61 | 2.46 | 94.27 | 2.96 | -3.70 | 1.35 | 57.98 | 3.56 | 84.10 | _71.7_ | 1
Team Poland3 | 2.60 | 85.75 | 1.05 | 96.68 | 1.24 | 97.11 | 4.06 | -42.17 | 1.08 | 66.15 | 3.07 | 86.31 | _67.7_ | 2
GMD4 | 4.93 | 72.92 | 1.48 | 95.32 | 1.66 | 96.14 | 2.06 | 27.82 | 2.12 | 33.72 | 6.85 | 69.41 | _67.1_ | 31

  * 1

This ranking is determined by how each individual model would have placed in
the GEFCom2014 price forecasting challenge in 2014 and therefore assumes that
the other models introduced in this article are not included.

  * 2

Gaillard et al. (2016)

  * 3

Maciejowska and Nowotarski (2016)

  * 4

Dudek (2016)

## References

  * Ardizzone et al. (2019) Ardizzone, L., Lüth, C., Kruse, J., Rother, C., Köthe, U., 2019.  Guided image generation with conditional invertible neural networks.  arXiv:1907.02392.
  * Arpogaus et al. (2022) Arpogaus, M., Voss, M., Sick, B., Nigge-Uricher, M., Dürr, O., 2022.  Short-term density forecasting of low-voltage load using Bernstein-polynomial normalizing flows.  arXiv:2204.13939.
  * Caflisch (1998) Caflisch, R.E., 1998.  Monte Carlo and quasi-Monte Carlo methods.  Acta numerica 7, 1–49.
  * Camporeale et al. (2017) Camporeale, E., Agnihotri, A., Rutjes, C., 2017.  Adaptive selection of sampling points for uncertainty quantification.  International Journal for Uncertainty Quantification 7, 1–22.
  * Camporeale et al. (2019) Camporeale, E., Chu, X., Agapitov, O., Bortnik, J., 2019\.  On the generation of probabilistic forecasts from deterministic models.  Space Weather 17, 455–475.
  * Challu et al. (2022) Challu, C., Olivares, K.G., Oreshkin, B.N., Garza, F., Mergenthaler, M., Dubrawski, A., 2022\.  N-HITS: Neural hierarchical interpolation for time series forecasting.  arXiv:2201.12886.
  * Chen and Guestrin (2016) Chen, T., Guestrin, C., 2016\.  XGBoost: A scalable tree boosting system, in: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM. pp. 785–794.
  * Cramer et al. (2022a) Cramer, E., Witthaut, D., Mitsos, A., Dahmen, M., 2022a.  Multivariate probabilistic forecasting of intraday electricity prices using normalizing flows.  arXiv:2205.13826.
  * Cramer et al. (2022b) Cramer, E.Y., Ray, E.L., Lopez, V.K., Bracher, J., Brennen, A., Rivadeneira, A.J.C., Gerding, A., Gneiting, T., House, K.H., Huang, Y., et al., 2022b.  Evaluation of individual and ensemble probabilistic forecasts of Covid-19 mortality in the US.  Proceedings of the National Academy of Sciences 119, e2113561119.
  * Dannecker (2015) Dannecker, L., 2015.  Energy time series forecasting: efficient and accurate forecasting of evolving time series from the energy domain.  1 ed., Springer Vieweg, Wiesbaden, Germany.
  * Dua and Graff (2019) Dua, D., Graff, C., 2019.  UCI machine learning repository.  http://archive.ics.uci.edu/ml.  (Accessed 10 March 2022).
  * Dudek (2016) Dudek, G., 2016.  Multilayer perceptron for GEFCom2014 probabilistic electricity price forecasting.  International Journal of Forecasting 32, 1057–1060.
  * Dumas et al. (2022) Dumas, J., Wehenkel, A., Lanaspeze, D., Cornélusse, B., Sutera, A., 2022.  A deep generative model for probabilistic energy forecasting in power systems: normalizing flows.  Applied Energy 305, 117871\.
  * Fanaee-T and Gama (2014) Fanaee-T, H., Gama, J., 2014\.  Event labeling combining ensemble detectors and background knowledge.  Progress in Artificial Intelligence 2, 113–127.
  * Fanfarillo et al. (2021) Fanfarillo, A., Roozitalab, B., Hu, W., Cervone, G., 2021\.  Probabilistic forecasting using deep generative models.  GeoInformatica 25, 127–147.
  * Gaillard et al. (2016) Gaillard, P., Goude, Y., Nedellec, R., 2016.  Additive models and robust aggregation for GEFCom2014 probabilistic electric load and electricity price forecasting.  International Journal of Forecasting 32, 1038–1050.
  * Ge et al. (2020) Ge, L., Liao, W., Wang, S., Bak-Jensen, B., Pillai, J.R., 2020\.  Modeling daily load profiles of distribution network for scenario generation using flow-based generative network.  IEEE Access 8, 77587–77597.
  * Gneiting et al. (2007) Gneiting, T., Balabdaoui, F., Raftery, A.E., 2007.  Probabilistic forecasts, calibration and sharpness.  Journal of the Royal Statistical Society: Series B (Statistical Methodology) 69, 243–268.
  * Gneiting and Raftery (2007) Gneiting, T., Raftery, A.E., 2007\.  Strictly proper scoring rules, prediction, and estimation.  Journal of the American Statistical Association 102, 359–378.
  * Gneiting et al. (2022) Gneiting, T., Wolffram, D., Resin, J., Kraus, K., Bracher, J., Dimitriadis, T., Hagenmeyer, V., Jordan, A.I., Lerch, S., Phipps, K., et al., 2022\.  Model diagnostics and forecast evaluation for quantiles.  Annual Review of Statistics and Its Application 10\.
  * González Ordiano, J. A. et al. (2020) González Ordiano, J. A., Gröll, L., Mikut, R., Hagenmeyer, V., 2020\.  Probabilistic energy forecasting using the nearest neighbors quantile filter and quantile regression.  International Journal of Forecasting 36, 310–323.
  * Heidrich et al. (2021) Heidrich, B., Bartschat, A., Turowski, M., Neumann, O., Phipps, K., Meisenbacher, S., Schmieder, K., Ludwig, N., Mikut, R., Hagenmeyer, V., 2021\.  pyWATTS: Python workflow automation tool for time series.  arXiv:2106.10157.
  * Heidrich et al. (2022) Heidrich, B., Turowski, M., Phipps, K., Schmieder, K., Süß, W., Mikut, R., Hagenmeyer, V., 2022.  Controlling non-stationarity and periodicities in time series generation using conditional invertible neural networks.  Applied Intelligence , 1–18.
  * Hong et al. (2016) Hong, T., Pinson, P., Fan, S., Zareipour, H., Troccoli, A., Hyndman, R.J., 2016.  Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond.  International Journal of Forecasting 32, 896–913.
  * Hyndman and Athanasopoulos (2018) Hyndman, R.J., Athanasopoulos, G., 2018\.  Forecasting: principles and practice.  2 ed., OTexts, Melbourne, Australia.
  * Jamgochian et al. (2022) Jamgochian, A., Wu, D., Menda, K., Jung, S., Kochenderfer, M.J., 2022.  Conditional approximate normalizing flows for joint multi-step probabilistic electricity demand forecasting.  arXiv:2201.02753.
  * Kaplan and Huang (2021) Kaplan, D., Huang, M., 2021\.  Bayesian probabilistic forecasting with large-scale educational trend data: A case study using naep.  Large-scale Assessments in Education 9, 1–31.
  * Kingma and Ba (2015) Kingma, D.P., Ba, J.L., 2015\.  Adam: A Method for Stochastic Optimization, in: Bengio, Y., LeCun, Y. (Eds.), 3rd International Conference on Learning Representations (ICLR 2015).
  * Kingma and Dhariwal (2018) Kingma, D.P., Dhariwal, P., 2018\.  Glow: Generative flow with invertible 1x1 convolutions, in: Advances in Neural Information Processing Systems, pp. 10215–10224.
  * Koenker et al. (2017) Koenker, R., Chernozhukov, V., He, X., Peng, L., 2017.  Handbook of quantile regression.  CRC press.
  * Koochali et al. (2019) Koochali, A., Schichtel, P., Dengel, A., Ahmed, S., 2019\.  Probabilistic forecasting of sensory data with generative adversarial networks–ForGAN.  IEEE Access 7, 63868–63880.
  * Krzysztofowicz (1999) Krzysztofowicz, R., 1999.  Bayesian theory of probabilistic forecasting via deterministic hydrologic model.  Water Resources Research 35, 2739–2750.
  * Lim et al. (2021) Lim, B., Arık, S.Ö., Loeff, N., Pfister, T., 2021\.  Temporal fusion transformers for interpretable multi-horizon time series forecasting.  International Journal of Forecasting 37, 1748–1764.
  * Liu et al. (2021) Liu, J., Wu, N., Qiao, Y., Li, Z., 2021.  A scientometric review of research on traffic forecasting in transportation.  IET Intelligent Transport Systems 15, 1–16.
  * Maciejowska and Nowotarski (2016) Maciejowska, K., Nowotarski, J., 2016\.  A hybrid model for GEFCom2014 probabilistic electricity price forecasting.  International Journal of Forecasting 32, 1051–1056.
  * Matheson and Winkler (1976) Matheson, J.E., Winkler, R.L., 1976\.  Scoring rules for continuous probability distributions.  Management science 22, 1087–1096.
  * Murphy (2023) Murphy, K.P., 2023.  Probabilistic Machine Learning: Advanced Topics.  MIT Press.  URL: http://probml.github.io/book2.
  * Oreshkin et al. (2019) Oreshkin, B.N., Carpov, D., Chapados, N., Bengio, Y., 2019\.  N-BEATS: Neural basis expansion analysis for interpretable time series forecasting.
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E., 2011\.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research 12, 2825–2830.
  * Petropoulos et al. (2022) Petropoulos, F., Apiletti, D., Assimakopoulos, V., Babai, M.Z., Barrow, D.K., Taieb, S.B., Bergmeir, C., Bessa, R.J., Bijak, J., Boylan, J.E., et al., 2022\.  Forecasting: theory and practice.  International Journal of Forecasting 38, 705–871.
  * Raftery (2016) Raftery, A.E., 2016.  Use and communication of probabilistic forecasts.  Statistical Analysis and Data Mining: The ASA Data Science Journal 9, 397–410.
  * Raftery et al. (1997) Raftery, A.E., Madigan, D., Hoeting, J.A., 1997.  Bayesian model averaging for linear regression models.  Journal of the American Statistical Association 92, 179–191.
  * Rasul et al. (2020) Rasul, K., Sheikh, A.S., Schuster, I., Bergmann, U., Vollgraf, R., 2020.  Multivariate probabilistic time series forecasting via conditioned normalizing flows.  arXiv:2002.06103.
  * Salinas et al. (2020) Salinas, D., Flunkert, V., Gasthaus, J., Januschowski, T., 2020\.  DeepAR: Probabilistic forecasting with autoregressive recurrent networks.  International Journal of Forecasting 36, 1181–1191.
  * Smith (2013) Smith, R.C., 2013.  Uncertainty quantification: theory, implementation, and applications. volume 12.  Siam.
  * Toth et al. (2003) Toth, Z., Talagrand, O., Candille, G., Zhu, Y., 2003\.  Probability and ensemble forecasts.  Forecast verification: A practitioner’s guide in atmospheric science 137, 163.
  * Wen and Torkkola (2019) Wen, R., Torkkola, K., 2019\.  Deep generative quantile-copula models for probabilistic forecasting.  arXiv:1907.10697.
  * Xiu (2010) Xiu, D., 2010.  Numerical methods for stochastic computations, in: Numerical Methods for Stochastic Computations. Princeton University Press.
  * Zhang and Zhang (2019) Zhang, L., Zhang, B., 2019.  Scenario forecasting of residential load profiles.  IEEE Journal on Selected Areas in Communications 38, 84–95.