# CAMul: Calibrated and Accurate Multi-view Time-Series Forecasting

Harshavardhan Kamarthi, Lingkai Kong, Alexander Rodríguez, Chao Zhang, B.
Aditya Prakash
College of Computing
Georgia Institute of Technology
<EMAIL_ADDRESS>

###### Abstract

> Probabilistic time-series forecasting enables reliable decision making
> across many domains. Most forecasting problems have diverse sources of data
> containing multiple modalities and structures. Leveraging information as
> well as uncertainty from these data sources for well-calibrated and accurate
> forecasts is an important challenging problem. Most previous work on multi-
> modal learning and forecasting simply aggregate intermediate representations
> from each data view by simple methods of summation or concatenation and do
> not explicitly model uncertainty for each data-view. We propose a general
> probabilistic multi-view forecasting framework CAMul, that can learn
> representations and uncertainty from diverse data sources. It integrates the
> knowledge and uncertainty from each data view in a dynamic context-specific
> manner assigning more importance to useful views to model a well-calibrated
> forecast distribution. We use CAMul for multiple domains with varied sources
> and modalities and show that CAMul outperforms other state-of-art
> probabilistic forecasting models by over 25% in accuracy and calibration.

##  1 Introduction

Time-series forecasting is a classic machine learning problem with
applications covering wide-ranging domains including retail, meteorology,
economics, epidemiology and energy. For many of these applications, we have a
wide variety of datasets representing different _views_ or perspectives of the
phenomena to forecast. These views may differ in their structure and modality,
and also in the quality and reliability associated due to collection and
processing differences. Due to the high-stakes decisions that these forecasts
may inform (e.g., hospital resource allocation for COVID-19), designing ML
models that can leverage these multiple data sources to provide not only
accurate but also well-calibrated forecast distributions is an important task.

State-of-the-art time series forecasting methods employ sequential neural
models that can naturally integrate multiple time-varying exogenous features
(Salinas et al. 2020; Li et al. 2021; Rangapuram et al. 2018). However, they
are not suitable to ingest multiple modalities of data such as networks,
sequences and fixed sized features together.

Effective integration of information from multiple views is challenging
because information from individual views may be noisy, conflicting, redundant
or sometimes unreliable. Recent works that seek to integrate multi-source and
multi-modal data for accurate sequence forecasting by using architectures like
Graph Neural networks and Convolutional networks (such as for disease
(Ramchandani, Fan, and Mostafavi 2020) and sales forecasting (Ekambaram et al.
2020)) do not comprehensively address this challenge. They propose to combine
embeddings from multiple data sources either through direct concatenation (Hu
et al. 2020) or simple summation (Ramchandani, Fan, and Mostafavi 2020). Other
recent methods (Chen et al. 2021a) learn a data-view-specific importance
parameter and use it to combine embeddings in a weighted manner. However,
these weights are learned over all data-points and are not context-sensitive:
they do not account for temporal variation in the importance of each view as
well as variance across time series.

Moreover, these multi-modal time-series forecasting methods do not focus on
learning a _well-calibrated_ forecast distribution. This is especially
challenging in a multi-view setting since the ambiguity and reliability of
views has to be integrated to better inform forecast uncertainty. Redundancy
and high confidence in beliefs across multiple data views could inform a more
confident forecast whereas conflicting information and lower confidence from
some data sources may suggest higher uncertainty. Past works on multi-view
probabilistic forecasting (Yan et al. 2021) do not integrate stochastic
information about beliefs from individual data views and instead use a
deterministic embedding from fusion of view-specific embeddings to learn
distribution parameters. As a result, these methods do not model the view-
specific uncertainty which may result in mis-calibrated forecasts.

Figure 1: CAMul models and integrates view-specific information and
uncertainty.

Hence in this paper, our work tackles the challenge of modelling as well as
integrating information and uncertainty from each data view to provide
accurate and well-calibrated time-series forecasts (Figure 1). We introduce a
general multi-view probabilistic time-series forecasting framework, CAMul
(Calibrated and Accurate Multi-view Forecasting, pronounced ‘camel’), that
jointly models uncertainty from multiple views independently using a latent
embedding distribution. It then combines them in a context-sensitive mechanism
accounting for unreliable data views specific to given sequence to provide a
well-calibrated and accurate forecasts. To learn a view-specific distribution
capturing relevant information for each data source, we use a non-parametric
modelling framework. We directly use latent embeddings of datapoints in the
training set of each view in the functional space to allow flexible
representation of the predictive distribution that rely on similar patterns
seen before. Our main contributions are:

a) Probabilistic Neural Framework jointly modeling multi-view uncertainty: We
propose a general framework CAMul for probabilistic time series forecasting on
multi-modal and multi-source data. We make no assumptions on the structure of
data. Our non-parametric generative model leverages probabilistic relations
learned between latent representations of data points for each data view
independently accounting for uncertainty from each view.

b) Integrating multi-view uncertainty towards calibrated forecasts: CAMul
leverages the latent information and uncertainty from each view and carefully
integrates the beliefs from multiple views together, dynamically weighting
each view’s importance based on input data, to learn well-calibrated
predictive distribution.

b) Evaluation of CAMul on multiple domains: We use the CAMul framework to
design models for multi-view time-series forecasting tasks from different
domains using diverse data sources and modalities (static features, sequences,
networks). We compare CAMul against state-of-art domain-specific as well as
general forecasting baselines and show that CAMul models clearly outperform
all baselines by over 25% in accuracy and calibration. We also show both
empirically and using case studies, that our method of modeling and
integrating uncertainty from individual data sources indeed causes these
significant improvements.

##  2 Related Work

Probabilistic Time-series Forecasting Classical time-series forecasting like
exponential smoothing, ARIMA-based models (Hyndman and Athanasopoulos 2018)
focus on univariate time-series with a small number of exogenous features and
learn model parameters independently for each sequence. Recent probabilistic
forecasting models leverage the representation power of neural sequential
modules like DeepAR (Salinas et al. 2020) which directly models the mean and
variance parameters of the forecast distribution, Bayesian Neural Networks
(Zhu and Laptev 2017) which require assigning useful priors to parameters and
require high computational resources for learning. Other recent works are
inspired by classic state-space models like Deep State Space models (Li et al.
2021; Rangapuram et al. 2018). EpiFNP (Kamarthi et al. 2021) a state-of-art
calibrated disease forecasting model, is closest to our work and leverages
Functional Neural Process (FNP) (Louizos et al. 2019), which uses stochastic
correlations between input data and datapoints to model a flexible non-
parametric distribution for univariate sequences. Our work leverages FNP for
uncertainty modeling of each of the individual views before we jointly model
the forecast distribution combining distributions from different views.

Multi-view time-series forecasting Recent advances in deep learning
architectures have allowed us to extract and combine representations from
multi-modal data sources in an end-to-end fashion. Most methods employ simple
summation or concatenation methods (Yan et al. 2021; Li, Yang, and Zhang 2018)
to combine representations from multiple sources (Ramchandani, Fan, and
Mostafavi 2020; Ekambaram et al. 2020), or use a learned data-source specific
weight (Chen et al. 2021a) which does not account for dynamic variation in the
importance of each view. In contrast, we model data-view specific uncertainty
by learning a latent distribution for each view which allows the model to
reason about source specific uncertainty as it integrates uncertainty-aware
stochastic representations of all views towards the forecast distribution.

Figure 2: Pipeline of our method CAMul describing all the components of the
generative process.

##  3 Problem Formulation

We describe the _Multi-view probabilistic forecasting_ problem, a
generalization of the probabilistic forecasting problem to multi-modal and
multi-source data. We denote the time-series dataset as
$\mathcal{Y}=\\{Y_{i}\\}_{i=1}^{N}$ where individual time-series is a
univariate sequence, $Y_{i}=\\{y[t]_{i}:y[t]_{i}\in\mathbf{R},1\leq t\leq
T\\}$. At each time-step $t$, we also have exogenous data sources that can be
used to forecast future values in $\mathcal{Y}$. We divide these features from
multiple sources and modalities into _views_. Assume we have $K$ views for a
specific problem representing $K$ sources/modalities of data. The dataset for
a view $j$ is denoted by $\mathcal{X}^{(j)}=\\{X_{i}^{(j)}\\}_{i=0}^{N}$. We
do not assume any specific data type for each view. However, all datapoints of
a view are assumed to have the same type. We can also have _static views_
whose values do not change across time: $\forall
t,x[t]_{i}^{(j)}=x[1]_{i}^{(j)}$. In some cases, we also denote as
$X[1:t]_{i}^{(j)}$ all the data till week $t$:
$X[1:t]_{i}^{(j)}=\\{x[t^{\prime}]_{i}^{(j)}:0\leq t^{\prime}\leq t\\}$.

Given the time-series values $\\{Y[1:t]_{i}\\}_{i=0}^{N}$ and exogenous
features from all views $\\{X[1:t]_{i}^{(j)}\\}_{i=1\dots N,j=1\dots K}$ till
current time $t$, our goal is to model a well-calibrated forecast probability
distribution

$P(\\{y[t+\tau]_{i}\\}_{i=1}^{N}|\\{Y[1:t]_{i}\\}_{i=1}^{N},\\{\\{X[1:t]_{i}^{(j)}\\}_{i=1\dots
N}\\}_{j=1\dots K})$

for forecasting time-series values $\\{y[t+\tau]_{i}\\}_{i=1}^{N}$, $\tau$
steps in future. We also assume that the view $1$ is the sequence dataset,
i.e, $Y_{i}=X_{i}^{(1)},\forall i$ and refer to it as the _default view_.

##  4 Methodology

We first give an overview of CAMul and then describe each of its modules in
more detail.

Model Overview: CAMul tackles the challenges related to jointly modeling
useful information and uncertainty from multiple data-sources. It jointly
models the predictive uncertainty by integrating information and uncertainty
for each view. It dynamically considers the importance of each view based on
input sequence to leverage the more important views and learn the output
distribution. We use the functional neural process (FNP) (Louizos et al. 2019)
framework to learn a non-parametric latent embedding distribution for each
view which enables a flexible method to model complex distribution by
leveraging latent similarity with past training data-points from each data
view. In contrast to related approaches that use FNP for univariate sequences
(Kamarthi et al. 2021), CAMul solves a more general and harder problem of
leveraging stochastic uncertainty from multi-source, multi-modal data and
tackles the challenge of jointly modeling and integrating stochastic
representations from each view.

At a high level, CAMul accomplishes our goals by performing the following
steps in an end-to-end manner: A) learn latent representations for each view
using Multi-view Latent Probabilistic Encoders to capture information and
uncertainty from each data source, B) leverage stochastic similarity between
data-points in latent space to learn relations between time-series for each
view via View Specific Correlation Graph and leverage these relations to
derive a view-aware embedding distribution, C) combine latent embedding
distributions from multiple views using learned importance of each view based
on input sequence via Context-sensitive Dynamic View Selection Module to
derive the output distribution. Figure 2 shows the full pipeline.

For each view $j$, the set of data-points on which we learn the stochastic
similarity w.r.t input point is a subset of the training set
$\mathcal{X}^{(j)}$ of the view $j$ that comprehensively represents the entire
training data. This set is called the _reference set_ , denoted by
$\mathcal{R}^{(j)}$. For example, for a view that represents sequence data
from the past, we consider each of the complete time-series sequences
$X[1:t]_{i}^{(j)}$ from past training data as a single data-point in the
reference set. Similarly consider views that represent static time-specific
data such as months. Each training point for a given month contains the same
static features. Therefore, the reference set contains a datapoint for each
unique month. Note that, during inference, we retain these reference sets and
use them to derive the output distribution.

###  4.1 Multi-view Probabilistic Latent Encoders

To capture useful information and uncertainty from each view, we learn a
latent embedding distribution for datapoints of a specific view. We leverage
appropriate neural encoders for each view based on the modality to learn
latent distribution parameters. The probabilistic encoder $g_{\theta_{j}}$ for
each view $j$ derives the parameters of latent embedding distribution for each
of the reference points $\mathcal{R}^{(j)}=\\{R_{k}^{(j)}\\}_{k=1}^{N_{j}}$
and input training point $X_{i}^{(j)}$ parameterized by a Gaussian
Distribution:

$\begin{split}\mu_{k}^{(j)},\sigma_{k}^{(j)}&=g_{\theta_{j}}(R_{k}^{(j)})]\\\
z_{k}^{(j)}&\sim\mathcal{N}(\mu_{k}^{(j)},\sigma_{k}^{(j)}).\end{split}$ (1)

$g_{\theta_{j}}$, a neural network module parameterized by $\theta_{j}$, is
chosen based on the modality of view $j$. For example, we use GRU for views
with sequence data, Graph Convolutional Network (Kipf and Welling 2016) to
encode relational data and Feed-Forward networks for static fixed-sized
features. The Appendix contains detailed discussions of encoder architectures
for different views used in this paper. The set of all latent encodings of
reference set $\mathcal{R}^{(j)}$ and training set $\mathcal{X}^{(j)}$ is
denoted as $\mathcal{Z}^{R,(j)}$ and $\mathcal{Z}^{X,(j)}$ respectively.

###  4.2 View Specific Correlation Graph

The latent embedding for input data $z_{i}^{(j)}\in\mathcal{Z}^{X,(j)}$
captures the probabilistic information of input data in view $j$ whereas
latent variables of $\mathcal{Z}^{R,(j)}$ capture stochastic information from
entirety of the data view $j$. To capture the information from entire the data
view $j$ conditioned on datapoint $i$, we use the non-parametric FNP framework
that allows modeling a flexible data-view aware latent distribution for given
input data by leveraging related reference points of the view.

We first learn probabilistic relations between input datapoint and reference
points in latent space using a similarity metric to create a _View Specific
Correlation Graph_ (VSCG) and then use the reference points connected to
input’s embedding in VSCG to learn the _view-aware latent embedding_.

We choose the radial basis function (RBF) as the similarity metric to relate
reference set $\mathcal{R}^{(j)}$ to training set $\mathcal{X}^{(j)}$:

$k(z_{i}^{(j)},z_{k}^{(j)})=\exp(-\rho||z_{k}^{(j)}-z_{i}^{(j)}||^{2}).$ (2)

for all $z_{k}^{(j)}\in\mathcal{Z}^{R,(j)}$ and
$z_{i}^{(j)}\in\mathcal{Z}^{X,(j)}$.

Next, we sample the VSCG $G^{(j)}$ as a bipartite graph between
$\mathcal{R}^{(j)}$ and $\mathcal{X}^{(j)}$, modeling each edge as a Bernoulli
distribution parameterized by RBF similarity

$p((i,k)\in G^{(j)})=\mathrm{Bern}(k(z_{i}^{(j)},z_{k}^{(k)})).$ (3)

Note that during training, we approximate sampling from the discrete Bernoulli
using a Gumbel-softmax distribution (Jang, Gu, and Poole 2017). Finally, we
aggregate the sampled neighbouring reference points $N(i)$ for each training
point $i$ to construct the _view-aware latent variable_ $u_{i}^{(j)}$ (refer
Figure 2) as:

$\begin{split}\mu(u_{i}^{(j)}),\hat{\sigma}(u_{i}^{(j)})=\sum_{k\in
N(i)}(l^{\mu}_{j}(z_{k}^{(j)}),l^{\sigma}_{j}(z_{k}^{(j)}))\\\
u_{i}^{(j)}\sim\mathcal{N}(\mu(u_{i}^{(j)}),\exp(\hat{\sigma}(u_{i}^{(j)}))),\end{split}$
(4)

where $l^{\mu}_{j}$ and $l^{\sigma}_{j}$ are linear layers. Thus the view-
aware embedding’s latent distribution is derived from sampled reference points
to capture view-specific information. The stochastic process of VSCG also
models view-specific uncertainty.

###  4.3 Context-Specific Dynamic Views Selection

Not all views are equally useful to learn a well-calibrated forecast
distribution. The importance of each view for predictive distribution also
varies for each time-series and across time. For instance, for disease
prediction, views representing features related to time (like months) may be
useful for predicting seasonal changes but its importance may decrease during
highly volatile and uncertain weeks near the peak where short-term sequence
history and real-time exogenous features are more important. The importance of
a view may also change dynamically when the features are corrupted for a small
period of time during collection or measurement.

Thus, we need a dynamic mechanism to weigh the importance of each view based
on the input sequence. We propose the _View Selection module_ to learn
importance weights for each view conditioned on input sequence and then
aggregate embeddings from multiple views in proportion to learned weights.
Given the _view-aware latent variables_ of all $K$ views for each datapoint
$i$ as $\\{u_{i}^{(j)}\\}_{j=1}^{K}$, we combine these multiple views’
knowledge towards the construction of final functional for predictive
distribution. We leverage the cross-attention mechanism from (Vaswani et al.
2017) to learn the importance of each view but also condition the weights on
input sequence’s representation from default view $1$ that encodes the time-
series sequence $z_{i}^{(1)}$ as:

$\\{\alpha_{i}^{(j)}\\}_{j=1}^{K}=\text{Softmax}_{j}(\\{h_{1}(z_{i}^{(1)})^{T}h_{2}(u_{i}^{(j)})\\}_{j=1}^{K}),$
(5)

where $h_{1}$ and $h_{2}$ are linear layers to transform both embeddings to
same dimensions. $\\{\alpha_{i}^{(j)}\\}_{j=1}^{K}$ denotes the importance of
all views for sequence $i$. Finally, we use the weights to combine the latent
representations for each view as

$\tilde{u}_{i}=\sum_{j=1}^{K}\alpha_{i}^{(j)}u_{i}^{(j)}.$ (6)

Thus, $\tilde{u}_{i}$, called the _combined view embedding_ represents
combined knowledge from correlations learned by all views weighted by their
importance for final prediction distribution. We denote the set of _combined
view embeddings_ for all sequences as
$\mathcal{U}=\\{\tilde{u}_{i}\\}_{i=1}^{N}$.

###  4.4 Forecast Distribution Decoder

Having extracted stochastic representations from each of the views leveraging
latent encoders, VSCGs and the View selection module, we learn the output
distribution via the _Decoder module_. The decoder module uses the combined
view embeddings $\mathcal{U}$ and the sequence embedding of default view
$\mathcal{Z}^{X,(1)}$ to learn the parameters of output distribution. While
$\mathcal{U}$ directly uses only the selectively aggregated embeddings of
reference sets, $\mathcal{Z}^{X,(1)}$ leverages local historical patterns of
input sequence including novel information that can be used to extrapolate
beyond information from reference sets of multiple views. The final decoder
process is described as:

$\begin{split}e_{i}&=z_{i}^{(1)}\oplus\tilde{u}_{i}\\\
\mu(y_{i}),\sigma_{(}y_{i})&=d_{1}(e_{i}),\exp(d_{2}(e_{i}))\\\
\hat{y}_{i}^{(t+\tau)}&\sim\mathcal{N}(\mu(y_{i}),\sigma(y_{i})),\end{split}$
(7)

where $d_{1}$ and $d_{2}$ and feed-forward layers and $\oplus$ is the
concatenation operator. Here, we used a multivariate Gaussian distribution to
parameterize $P(y_{i}^{(t+\tau)}|e_{i})$ since our target sequence has real
numbers. However, this framework can be extended to discrete and integer-
valued output by carefully choosing the appropriate distribution and their
relevant statistic to predict.

###  4.5 Model Training and Inference

The generative process of CAMul can be summarized as

$\begin{split}&P(\\{y_{i}^{t+\tau}\\}_{i=1}^{N}|\mathcal{X},\mathcal{R})=\int\sum_{j=1}^{K}[\underbrace{P(\mathcal{Z}^{X,(j)}|\mathcal{X}^{(j)})P(\mathcal{Z}^{R,(j)}|\mathcal{R}^{(j)})}_{\text{Stochastic
latent encoders}}\\\
&\underbrace{P(G^{(j)}|\mathcal{Z}^{X,(j)},\mathcal{Z}^{R,(j)})P(\mathcal{U}^{(j)}|G^{(j)})}_{\text{VSCG}}]\\\
&\underbrace{P(\mathcal{U}|\\{\mathcal{U}^{(j)}\\}_{j=1}^{K})}_{\text{View
selection}}\underbrace{P(\\{y_{i}^{t+\tau}\\}_{i=1}^{N}|\mathcal{U},\mathcal{Z}^{X,(1)})}_{\text{Output
distribution}}d\mathcal{\\{}{Z}^{(j)}\\}_{j}d\mathcal{U}.\end{split}$

Our objective is to increase the log-likelihood of above equation which is
intractable due to integrals over real-valued random variables. Therefore, we
use variational inference by approximating the posterior
$\prod_{j=1}^{K}P(\mathcal{Z}^{(j)},\mathcal{U}^{(j)},G^{(j)}|\mathcal{X}^{(j)},\mathcal{R}^{(j)})P(\mathcal{U}|\\{\mathcal{U}^{(j)}\\}_{j})$
with the variational distribution :

$\begin{split}q_{j}(\mathcal{U}^{(j)},\mathcal{Z}^{(j)},G^{(j)}|\mathcal{X}^{(j)}=P(\mathcal{Z}^{X,(j)}|\mathcal{X}^{(j)})P(\mathcal{Z}^{R,(j)}|\mathcal{R}^{(j)})\\\
P(G^{(j)}|\mathcal{Z}^{X,(j)},\mathcal{Z}^{R,(j)}q_{j}(\mathcal{U}^{(j)}|\mathcal{X}^{(1)}),\end{split}$

for each view $j$, where $q_{j}$ is a 2-layer network that parametrizes the
Gaussian distribution $q_{j}(\mathcal{U}[j]|\mathcal{X}^{(1)})$ with input
sequences $\mathcal{X}^{(1)}$. The ELBO is derived to be:

$\begin{split}\mathcal{L}&=-E_{q_{j}(\mathcal{Z}^{(j)},G^{(j)},\mathcal{U}^{(j)}|\mathcal{X}^{(j)})}[\log
P(\\{y_{i}^{t+\tau}\\}_{i=1}^{N}|\mathcal{U},\mathcal{Z}^{X,(1)})\\\
&+\sum_{j=1}^{K}\log P(\mathcal{U}^{(j)}|G^{(j)},\mathcal{Z}^{(j)})-\log
q_{j}(\mathcal{U}^{(j)}|\mathcal{X}^{(1)})].\end{split}$

The parameters of the inference distributions $q_{j}$ and the components of
the generative process are jointly learned via Stochastic Gradient Variational
Bayes to minimize the ELBO loss (Kingma and Welling 2013). We use the
reparametrization trick for all sampling processes. During inference, we
sample from the joint distribution
$P(\\{y_{i}^{t+\tau}\\}_{i=1}^{N},\\{\mathcal{Z}^{(j)},G^{(j)}\\}_{j},\mathcal{U}|\mathcal{X},\mathcal{R})$
to generate samples for the forecast distribution
$\\{y_{i}^{t+\tau}\\}_{i=1}^{N}$. The pseudo-code for training and inference
is available in Appendix.

Benchmark | Dataset | Target | Views(View type)
---|---|---|---
| google-symptoms
---
(Health 2021)
| Historical wILI sequences and
---
anonymized search volumes for 14 flu-related
symptoms for 10 HHS regions and national level.
| Future wILI values for each
---
HHS region’s sequence
at 1- to 4-weeks ahead
| Past sequences (Sequence)
---
HHS adjacency (Graph)
Months (Static)
| covid19
---
(Cramer et al. 2021)
| Weekly data related to Covid-19 surveillance including
---
from hospital records, mortality, hospitalization
counts, mobility, exposure and survey data collected
by (Kamarthi et al. 2021) for 50 US states
| Mortality for
---
50 US states at
1- to 4-weeks ahead
| Mortality (Sequence)
---
Line-list (Sequence)
Mobility (Sequence)
Demographics (Static)
State adjacency (Graph)
| power
---
(Hebrail and Berard 2012)
| Minute-wise Household power consumption
---
measurements for a year with a total of
260,640 measurements each with
7 features from 3 meters.
| Total active
---
power consumption
next minute
| Past sequence (Sequence)
---
Time-of-day (Static)
Month (Static)
| tweet
---
(Shi et al. 2019)
| Weekly Tweet topic distribution for tweets
---
related to Covid-19 pandemic with topics
extracted using LDA.
| 1-week ahead topic
---
distribution
| Past sequence (Sequence)
---
State adjacency (Graph)
Month (Static)
Demographics (Static)

Table 1: Description of benchmark tasks: datasets, target and views.

##  5 Experiments

We evaluated our models on a workstation that runs on Intel Xeon 64 core
processor with 128 GB memory on a single Nvidia Tesla V100 GPU. Our model
takes less than 6 GB of memory and typically takes 20-40 minutes of training
time for each of the benchmarks discussed below. We describe the baselines,
benchmark datasets and tasks as well as evaluation metrics with additional
details in the Appendix.

Baselines: We compare our model against the state-of-art probabilistic
forecasting baselines along with some domain-specific baselines. The chosen
forecasting baselines have shown state-of-art performance on a wide set of
probabilistic forecasting tasks such as power consumption, air quality,
traffic forecasting, health risk assessment, data center load estimation, etc.
SARIMA (Hyndman and Athanasopoulos 2018): A classic time-series forecasting
baseline based on ARIMA that accounts for seasonal shifts. DeepAR (Salinas et
al. 2020): A state-of-art, widely used RNN based probabilistic forecasting
model that learns a parametric distribution. Deep State Space Model (DSSM) (Li
et al. 2021): A space state based model that uses neural networks to model
transition and emission distribution of state space. Deep Graph Factors
(GraphDF) (Chen et al. 2021b): A deep probabilistic forecasting model that
integrates relational information from graphs across time-series. In absence
of explicit relational information, Chen et al. (2021b) proposes to use RBF
kernel over the euclidean distance as edge weights over pairs of time-series,
which we use as the GraphDF-RBF baseline. If explicit relational data in form
of adjacency graph is available (in case of all benchmarks except power), we
also evaluate using this adjacency graph as the GraphDF-Adj baseline.
Recurrent neural process (RNP) (Qin et al. 2019): Neural Process based method
for temporal data which uses attention over reference points as part of the
generative process. Multi-modal Gaussian Process (MMGP): Similar to (Chen et
al. 2021a) we first pre-train a deterministic model combining deterministic
embeddings from the encoders and aggregating embeddings with fixed learned
weights to predict the output. Then we aggregate the embeddings as features to
train a Gaussian Process to get probabilistic predictions.

For disease forecasting benchmarks, we also evaluate against state-of-art
domain-specific forecasting models that use the same set of exogenous features
as described in Table 1.We choose top performing deep learning models for flu-
forecasting google-symptoms task: EpiDeep (Adhikari et al. 2019) and EpiFNP
(Kamarthi et al. 2021). For the covid19 task, we also choose CMU-TS (Cramer et
al. 2021) and DeepCovid (Rodríguez et al. 2021b) as baselines; these leverage
the CovDS dataset and are the best statistical models at the Covid-19 Forecast
Hub organized by CDC (Cramer et al. 2021). We also evaluate against variants
of CAMul which enable us to examine the importance of Context-Specific View
Selection and probabilistic view-specific modeling (see Q2 of Results).

Benchmarks: We evaluate CAMul framework on time-series forecasting problem
from a variety of domains involving diverse types of data views as described
in Table 1.

Evaluation metrics: We evaluate our model and baselines using probabilistic
metrics that test for both accuracy and calibration of forecasts. We use the
Cumulative ranked probability score (CRPS), a standard probabilistic metric
(Gneiting and Katzfuss 2014). Interval Score (IS), which is a popular
probabilistic forecast metric in epidemic forecasting (Reich et al. 2019), is
also used. In order to measure calibration of forecasts, we use the
Calibration Score (CS) based on (Kamarthi et al. 2021; Kuleshov, Fenner, and
Ermon 2018).

| power |  | tweet
---|---|---|---
| CRPS | CS | IS |  | CRPS | CS | IS
SARIMA | 1.16 $\pm$ 0.018 | 0.23 $\pm$ 0.002 | 3.49 $\pm$0.021 | SARIMA | 1.41 $\pm$ 0.262 | 0.37$\pm$ 0.007 | 3.87 $\pm$ 0.74
DeepAR | 0.72 $\pm$ 0.002 | 0.04 $\pm$ 0.003 | 1.57 $\pm$ 0.046 | DeepAR | 1.06 $\pm$ 0.152 | 0.15 $\pm$ 0.003 | 1.35 $\pm$ 0.31
DSSM | 0.59 $\pm$ 0.018 | 0.02 $\pm$ 0.002 | 1.32 $\pm$ 0.054 | DSSM | 0.92 $\pm$ 0.173 | 0.07 $\pm$ 0.002 | 0.96 $\pm$ 0.34
RNP | 0.85 $\pm$ 0.015 | 0.19 $\pm$ 0.005 | 3.31 $\pm$ 0.018 | RNP | 1.16 $\pm$ 0.045 | 0.22 $\pm$ 0.006 | 3\. 05 $\pm$ 0.52
MMGP | 1.19 $\pm$ 0.041 | 0.15 $\pm$ 0.001 | 3.03 $\pm$ 0.013 | MMGP | 1.31 $\pm$ 0.162 | 0.13 $\pm$ 0.0026 | 1.65 $\pm$ 0.49
GraphDF-RBF | 0.92 $\pm$ 0.03 | 0.13 $\pm$ 0.004 | 1.45 $\pm$ 0.033 | GraphDF-RBF | 0.62 $\pm$ 0.052 | 0.08 $\pm$ 0.010 | 1.91 $\pm$ 0.76
|  |  |  | GraphDF-Adj | 0.91 $\pm$ 0.031 | 0.14$\pm$ 0.009 | 2.39 $\pm$ 0.26
CAMul | 0.46 $\pm$0.02 | 0.02 $\pm$ 0.001 | 0.93 $\pm$ 0.021 | CAMul | 0.42 $\pm$ 0.015 | 0.05 $\pm$ 0.004 | 0.68 $\pm$ 0.21
CAMul-C | 1.13$\pm$0.04 | 0.08$\pm$0.001 | 2.88$\pm$ 0.014 |  CAMul-C | 0.84$\pm$0.0027 | 0.07$\pm$0.002 | 1.03$\pm$0.17
CAMul-S | 0.68$\pm$0.05 | 0.04$\pm$0.002 | 1.27$\pm$0.027 |  CAMul-S | 0.58$\pm$0.081 | 0.05$\pm$0.006 | 1.41$\pm$0.25
CAMul-D | 1.36$\pm$0.07 | 0.14$\pm$0.001 | 2.96$\pm$0.012 |  CAMul-D | 1.15$\pm$0.020 | 0.19$\pm$0.001 | 1.14$\pm$0.15

| covid19 |  | google-symptoms
---|---|---|---
| CRPS | CS | IS |  | CRPS | CS | IS
SARIMA | 61 $\pm$ 3.40 | 0.42 $\pm$ 0.017 | 6.32 $\pm$ 0.042 | SARIMA | 1.14 $\pm$ 0.017 | 0.43 $\pm$ 0.008 | 3.11 $\pm$ 0.240
DeepAR | 32.6 $\pm$ 6.40 | 0.19 $\pm$ 0.015 | 2.21 $\pm$ 0.015 | DeepAR | 0.55 $\pm$ 0.013 | 0.14 $\pm$ 0.003 | 1.42 $\pm$ 0.130
DSSM | 49.6 $\pm$ 3.10 | 0.22 $\pm$ 0.009 | 2.13 $\pm$ 0.041 | DSSM | 0.52 $\pm$ 0.011 | 0.13 $\pm$ 0.003 | 2.11 $\pm$ 0.140
RNP | 51.2 $\pm$ 7.60 | 0.35 $\pm$ 0.018 | 4.33 $\pm$ 0.031 | RNP | 0.71 $\pm$ 0.047 | 0.43 $\pm$ 0.006 | 4.13 $\pm$ 0.150
MMGP | 25.2 $\pm$ 1.10 | 0.27 $\pm$ 0.012 | 3.19 $\pm$ 0.015 | MMGP | 0.68 $\pm$ 0.021 | 0.21 $\pm$ 0.008 | 1.87 $\pm$ 0.410
GraphDF-RBF | 31.4 $\pm$ 2.80 | 0.21 $\pm$ 0.017 | 2.75 $\pm$ 0.046 | GraphDF- RBF | 0.59 $\pm$ 0.016 | 0.08 $\pm$ 0.002 | 1.03$\pm$ 0.240
GraphDF-Adj | 37.6 $\pm$ 3.30 | 0.28 $\pm$ 0.013 | 2.44 $\pm$ 0.017 | GraphDF- Adj | 0.64 $\pm$ 0.057 | 0.08 $\pm$ 0.005 | 2.87 $\pm$ 0.610
DeepCovid | 30.5 $\pm$ 7.10 | 0.18 $\pm$ 0.011 | 3.41 $\pm$ 0.029 | EpiDeep | 0.93 $\pm$ 0.046 | 0.24 $\pm$ 0.007 | 4.26 $\pm$ 0.910
CMU-TS | 21.8 $\pm$ 4.80 | 0.14 $\pm$ 0.015 | 3.17 $\pm$ 0.064 | EpiFNP | 0.31 $\pm$ 0.051 | 0.04 $\pm$ 0.002 | 0.56 $\pm$ 0.180
CAMul | 17.4 $\pm$ 5.40 | 0.15 $\pm$ 0.007 | 1.42 $\pm$ 0.017 | CAMul | 0.24 $\pm$ 0.060 | 0.04 $\pm$ 0.004 | 0.43 $\pm$ 0.020
CAMul-C | 18$\pm$3.10 | 0.19$\pm$0.003 | 1.73$\pm$0.024 |  CAMul-C | 0.29$\pm$0.022 | 0.05 $\pm$ 0.001 | 0.48 $\pm$ 0.060
CAMul-S | 18.4$\pm$2.90 | 0.15$\pm$0.007 | 1.64 $\pm$ 0.037 |  CAMul-S | 0.31$\pm$0.064 | 0.04$\pm$0.006 | 0.46$\pm$0.020
CAMul-D | 27.3$\pm$7.10 | 0.31$\pm$0.012 | 3.26 $\pm$0.021 |  CAMul-D | 0.74$\pm$0.081 | 0.19$\pm$0.002 | 1.83$\pm$0.040

Table 2: Evaluation scores (over 20 runs) for CAMul and baselines for all
benchmarks. The best performing scores are shown in bold. We performed t-test
with $\alpha=1\%$. The score in bold are statistically significantly better
than other models.

###  5.1 Results

Q1: Does CAMul provide well-calibrated accurate probabilistic forecasts across
all benchmarks?

We evaluate our model and baselines on the four tasks described in the
experiment section above. We ran the experiments 20 times for each of the
models for all tasks and reported the mean scores. The results are summarized
in Tables 2. For covid19 and google-symptoms, we show 2-week ahead results
here and rest, which show similar trends, in Appendix. CAMul models
significantly outperform the baselines in all the probabilistic and
calibration scores. Specifically for the hard disease-forecasting tasks we see
over 25% improvement in CRPS score and 50% improvement in interval score over
best baselines. In case of power and tweet, we observe 28% and 24% improvement
in CRPS scores over second-best model and 56% and 35% improvement in interval
score. Performing significance test (t-test) with $\alpha=1\%$ shows that
CAMul is significantly better than other models in all scores except those
highlighted in bold in Table 2. Applying post-hoc calibration methods
(Kuleshov, Fenner, and Ermon 2018; Song et al. 2019) on baselines also does
not affect the significance of our results (Table 5 in Appendix).

Q2 : Effect of multi-source stochastic modelling and context-sensitive dynamic
view selection on CAMul’s performance

We evaluate the efficacy of Context-Specific Dynamic Views Selection and
probabilistic modeling of each data view via Multi-view Latent Probabilistic
Encoders and VSCG. We compare the attention based dynamic view selection of
CAMul with two other variants a) CAMul-C concatenates SVCG-based latent
embeddings from all views, b) CAMul-S learns a static weight $w^{(j)}$ for
each view $j$ and combines the VSCG-based embeddings:
$\tilde{u}_{i}=\sum_{j}w^{(j)}u_{i}^{(j)}$, c) To test the efficacy of
stochastic modelling of latent embeddings, the variant CAMul-D uses
deterministic view-specific latent variables: We use the latent encoders’ mean
rather than sampling from Gaussian as Eqn. 1 and use a cross-attention layer
over reference points instead of VSCG to derive the view specific latent
variables as a weighted summation of reference points. The evaluation scores
of the variants are shown in Table 2. We see that the original configuration
of CAMul with dynamic view selection and multi-source probabilistic modeling
provides the best scores. Using deterministic multi-source latent embeddings,
in particular, drastically decreases the scores showing the efficacy of
uncertainty modeling.

Q3: Does CAMul handle information and uncertainty from multiple views to
provide better performance?

| power | tweet | covid19 | google-symptoms
---|---|---|---|---
All Views | 0.46 | 0.42 | 17.4 | 0.24
Default view | 1.03 | 0.95 | 24.3 | 0.39
Two Best Views | 0.63 | 0.77 | 18.3 | 0.35
Best Baseline | 0.59 | 0.62 | 21.8 | 0.31

Table 3: Comparison of CRPS score of CAMul with all views, default view $1$, 2
best views and best baseline.

CAMul models useful patterns and uncertainty for each of the diverse set of
data views independently before combining them. This is in contrast to most
baselines that use simple aggregation at feature level or latent embedding
level where noisy or reliable data from some views may hinder performance.
CAMul however learns to weigh the importance of each view before combining
them for prediction. To test the efficacy of CAMul’s handling of multiple data
views, we evaluated the model with a single default view, two best views and
compared it with the original CAMul with all views and the best performing
baseline as shown in Table 3. Note that for all benchmarks, the best view is
always the default view. We see that model with all views is clearly the best
performing. Moreover, using only the best view sometimes leads to lower
performance compared to the best performing baseline which has access to data
from all views. Therefore, we conclude that _CAMul can handle multiple views
that in turn lead to significantly better performance._.

Q4: Does weights of each view from View Selection module correspond to
predictive utility of the view?

The attention weights learned by the view selection module in Eqn. 5 informs
the learned importance of views that are used for aggregating view-aware
latent embeddings. We observed that the default view $1$ containing time-
series sequences was almost always the most highly weighted views and was
necessary for model to perform reasonably.

Next, we study the correlation between attention weights of other views and
the efficacy of the views. We assess the efficacy of a view by training
variant of CAMul with two views: the default view and view we are interested
in. We compare the CRPS scores of these variants to the attention weights and
see that the CRPS scores are inversely proportional to the view selection
module weight (See above Figure for an example for covid19 task). This further
shows that the view selection module, indeed, selects the most informative
views on average to improve the performance of our CAMul models.

Q5: Does the dynamic view-selection module adapt across time to select
informative views?

We provide specific case-studies to show the efficacy of the view selection
module in selecting useful views specific to the input sequence by studying
the attention weights. We describe a case-study related to covid19 below and
one related to google-symptoms in the Appendix.

Obs 1: For the covid19 task, weights of mobility view are higher than default
sequence view during early pandemic (June and July) for many highly populous
US states.

State | Month | Seq. | Line-list | Mobility | Adj. | Demo.
---|---|---|---|---|---|---
TX | June | 0.32 | 0.22 | 0.37 | 0.03 | 0.06
July | 0.37 | 0.23 | 0.31 | 0.05 | 0.04
GA | June | 0.21 | 0.34 | 0.31 | 0.01 | 0.13
July | 0.24 | 0.40 | 0.26 | 0.02 | 0.08
MA | June | 0.32 | 0.19 | 0.36 | 0.04 | 0.09
July | 0.39 | 0.29 | 0.23 | 0.02 | 0.07
NY | June | 0.25 | 0.24 | 0.32 | 0.04 | 0.15
July | 0.26 | 0.34 | 0.27 | 0.01 | 0.12

Table 4: Avg. Attention weight of View selection module during June and July
for some populous US states.

The average attention weight on the default view is over 40% for all months
over all states. However, during the initial 2 months of the dataset (June and
July), 12 states observed higher weight for mobility view including populous
states such as TX, GA, MA and NY. We found that CAMul’s view selection module
adapted by relying more on the mobility view data during the initial months
for mortality forecasting. This concurs with studies that at the initial
stages of the pandemic, decrease in mobility was highly correlated with
decrease in the disease spread (Chang et al. 2021), but later on, this changed
(Rodríguez et al. 2021b). In addition, line-list data was error-prone during
the initial months of the pandemic because the reporting systems were not yet
in place (Rodríguez et al. 2021a) whereas the mobility features extracted from
digital sources were more accurate in real-time (Budd et al. 2020).

##  6 Conclusion

We introduced CAMul, a general non-parametric generative framework for multi-
source, multi-modal probabilistic forecasting. Our framework successfully
tackles the challenge of modeling probabilistic information and uncertainty
from diverse data views of multiple modalities across multiple domain
benchmarks. CAMul outperformed both state-of-art general probabilistic
baselines as well as top domain-specific baselines, resulting in over 25%
improvement in accuracy and calibration metrics. Our case-studies also
empirically showed the importance of joint probabilistic modeling and context-
sensitive integration from multiple views. It automatically adapts to select
more useful data views as seen in the covid19 and google-symptoms case
studies.

CAMul can easily be applied to any multi-source and multi-modal forecasting
task by using appropriate encoders to encode a variety of data where capturing
multi-source uncertainty is important. Analysis of view selection for specific
tasks can also help understand data reliability, the disparity in data quality
across sensitive parameters. As future work, our work can also be extended to
tasks such as anomaly detection, change point detection and time-series
segmentation where drastic variation in confidence intervals and view
selection weights can be a useful predictive indicator of important behaviors.

## References

  * Adhikari et al. (2019) Adhikari, B.; Xu, X.; Ramakrishnan, N.; and Prakash, B. A. 2019.  Epideep: Exploiting embeddings for epidemic forecasting.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 577–586.
  * Blei, Ng, and Jordan (2003) Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003.  Latent dirichlet allocation.  _the Journal of machine Learning research_ , 3: 993–1022.
  * Budd et al. (2020) Budd, J.; Miller, B. S.; Manning, E. M.; Lampos, V.; Zhuang, M.; Edelstein, M.; Rees, G.; Emery, V. C.; Stevens, M. M.; Keegan, N.; et al. 2020.  Digital technologies in the public-health response to COVID-19.  _Nature medicine_ , 26(8): 1183–1192.
  * Centers for Disease Control and Prevention (2021) Centers for Disease Control and Prevention, U. 2021.  Flu Symptoms & Complications.
  * Chang et al. (2021) Chang, S.; Pierson, E.; Koh, P. W.; Gerardin, J.; Redbird, B.; Grusky, D.; and Leskovec, J. 2021.  Mobility network models of COVID-19 explain inequities and inform reopening.  _Nature_ , 589(7840): 82–87.
  * Chen et al. (2021a) Chen, C.; Liang, J.; Ma, F.; Glass, L.; Sun, J.; and Xiao, C. 2021a.  UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Data.  In _Proceedings of the Web Conference 2021_ , 217–226.
  * Chen et al. (2021b) Chen, H.; Rossi, R. A.; Mahadik, K.; Kim, S.; and Eldardiry, H. 2021b.  Graph Deep Factors for Forecasting with Applications to Cloud Resource Allocation.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 106–116.
  * Cho et al. (2014) Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_.
  * Chung et al. (2015) Chung, J.; Kastner, K.; Dinh, L.; Goel, K.; Courville, A. C.; and Bengio, Y. 2015\.  A recurrent latent variable model for sequential data.  _Advances in neural information processing systems_ , 28: 2980–2988.
  * Cramer et al. (2021) Cramer, E. Y.; Lopez, V. K.; Niemi, J.; George, G. E.; Cegan, J. C.; Dettwiller, I. D.; England, W. P.; Farthing, M. W.; Hunter, R. H.; Lafferty, B.; et al. 2021.  Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US.  _medRxiv_.
  * Ekambaram et al. (2020) Ekambaram, V.; Manglik, K.; Mukherjee, S.; Sajja, S. S. K.; Dwivedi, S.; and Raykar, V. 2020.  Attention based Multi-Modal New Product Sales Time-series Forecasting.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 3110–3118.
  * Fraccaro et al. (2016) Fraccaro, M.; Sønderby, S. K.; Paquet, U.; and Winther, O. 2016.  Sequential neural models with stochastic layers.  _Advances in neural information processing systems_.
  * Garnelo et al. (2018) Garnelo, M.; Schwarz, J.; Rosenbaum, D.; Viola, F.; Rezende, D. J.; Eslami, S.; and Teh, Y. W. 2018.  Neural processes.  _arXiv preprint arXiv:1807.01622_.
  * Gneiting and Katzfuss (2014) Gneiting, T.; and Katzfuss, M. 2014.  Probabilistic forecasting.  _Annual Review of Statistics and Its Application_ , 1: 125–151.
  * Hamilton (2020) Hamilton, W. L. 2020.  Graph representation learning.  _Synthesis Lectures on Artifical Intelligence and Machine Learning_ , 14(3): 1–159.
  * Health (2021) Health, G. 2021.  Using symptoms search trends to inform COVID-19 research.
  * Hebrail and Berard (2012) Hebrail, G.; and Berard, A. 2012.  Individual household electric power consumption Data Set.  _UCI Machine Learning Repository_.
  * Hu et al. (2020) Hu, W.; Yang, Y.; Wang, J.; Huang, X.; and Cheng, Z. 2020.  Understanding electricity-theft behavior via multi-source data.  In _Proceedings of The Web Conference 2020_ , 2264–2274.
  * Hyndman and Athanasopoulos (2018) Hyndman, R. J.; and Athanasopoulos, G. 2018.  _Forecasting: principles and practice_.  OTexts.
  * Jang, Gu, and Poole (2017) Jang, E.; Gu, S.; and Poole, B. 2017.  Categorical reparameterization with gumbel-softmax.  _ICLR_.
  * Kamarthi et al. (2021) Kamarthi, H.; Kong, L.; Rodríguez, A.; Zhang, C.; and Prakash, B. A. 2021.  When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting.  _arXiv preprint arXiv:2106.03904_.
  * Kamarthi et al. (2021) Kamarthi et al., H. 2021.  Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future.  _arXiv preprint arXiv:2106.04420_.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kingma and Welling (2013) Kingma, D. P.; and Welling, M. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Kipf and Welling (2016) Kipf, T.; and Welling, M. 2016.  Semi-Supervised Classification with Graph Convolutional Networks.  _ICLR_.
  * Krishnan, Shalit, and Sontag (2017) Krishnan, R.; Shalit, U.; and Sontag, D. 2017.  Structured inference networks for nonlinear state space models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31.
  * Kuleshov, Fenner, and Ermon (2018) Kuleshov, V.; Fenner, N.; and Ermon, S. 2018.  Accurate uncertainties for deep learning using calibrated regression.  In _International Conference on Machine Learning_ , 2796–2804. PMLR.
  * Leon Barrett and O’Kane (2015) Leon Barrett, A. K., Stephan Hoyer; and O’Kane, D. 2015.  properscoring 0.1.  _The Climate Corporation_.
  * Leskovec, Rajaraman, and Ullman (2020) Leskovec, J.; Rajaraman, A.; and Ullman, J. D. 2020.  _Mining of massive data sets_.  Cambridge university press.
  * Li et al. (2021) Li, L.; Yan, J.; Yang, X.; and Jin, Y. 2021.  Learning interpretable deep state space model for probabilistic time series forecasting.  _International Joint COnference on Artificial Intellignece_.
  * Li, Hao, and Lei (2016) Li, Y.; Hao, Z.; and Lei, H. 2016.  Survey of convolutional neural network.  _Journal of Computer Applications_ , 36(9): 2508–2515.
  * Li, Yang, and Zhang (2018) Li, Y.; Yang, M.; and Zhang, Z. 2018.  A survey of multi-view representation learning.  _IEEE transactions on knowledge and data engineering_ , 31(10): 1863–1883.
  * Lim and Zohren (2021) Lim, B.; and Zohren, S. 2021.  Time-series forecasting with deep learning: a survey.  _Philosophical Transactions of the Royal Society A_ , 379(2194): 20200209\.
  * Louizos et al. (2019) Louizos, C.; Shi, X.; Schutte, K.; and Welling, M. 2019.  The Functional Neural Process.  In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Qin et al. (2019) Qin, S.; Zhu, J.; Qin, J.; Wang, W.; and Zhao, D. 2019.  Recurrent attentive neural process for sequential data.  _arXiv preprint arXiv:1910.09323_.
  * Ramchandani, Fan, and Mostafavi (2020) Ramchandani, A.; Fan, C.; and Mostafavi, A. 2020.  Deepcovidnet: An interpretable deep learning model for predictive surveillance of covid-19 using heterogeneous features and their interactions.  _IEEE Access_ , 8: 159915–159930.
  * Rangapuram et al. (2018) Rangapuram, S. S.; Seeger, M. W.; Gasthaus, J.; Stella, L.; Wang, Y.; and Januschowski, T. 2018.  Deep state space models for time series forecasting.  _Advances in neural information processing systems_ , 31: 7785–7794.
  * Reich et al. (2019) Reich, N. G.; Brooks, L. C.; Fox, S. J.; Kandula, S.; McGowan, C. J.; Moore, E.; Osthus, D.; Ray, E. L.; Tushar, A.; Yamana, T. K.; et al. 2019.  A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States.  _Proceedings of the National Academy of Sciences_ , 116(8): 3146–3154.
  * Rodríguez et al. (2021a) Rodríguez, A.; Muralidhar, N.; Adhikari, B.; Tabassum, A.; Ramakrishnan, N.; and Prakash, B. A. 2021a.  Steering a Historical Disease Forecasting Model Under a Pandemic: Case of Flu and COVID-19.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 4855–4863.
  * Rodríguez et al. (2021b) Rodríguez, A.; Tabassum, A.; Cui, J.; Xie, J.; Ho, J.; Agarwal, P.; Adhikari, B.; and Prakash, B. A. 2021b.  DeepCOVID: An Operational Deep Learning-driven Framework for Explainable Real-time COVID-19 Forecasting.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 15393–15400.
  * Salehinejad et al. (2017) Salehinejad, H.; Sankar, S.; Barfett, J.; Colak, E.; and Valaee, S. 2017.  Recent advances in recurrent neural networks.  _arXiv preprint arXiv:1801.01078_.
  * Salinas et al. (2020) Salinas, D.; Flunkert, V.; Gasthaus, J.; and Januschowski, T. 2020.  DeepAR: Probabilistic forecasting with autoregressive recurrent networks.  _International Journal of Forecasting_ , 36(3): 1181–1191.
  * Shi et al. (2019) Shi, H.; Zhang, C.; Yao, Q.; Li, Y.; Sun, F.; and Jin, D. 2019.  State-sharing sparse hidden markov models for personalized sequences.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 1549–1559.
  * Song et al. (2019) Song, H.; Diethe, T.; Kull, M.; and Flach, P. 2019.  Distribution calibration for regression.  In _International Conference on Machine Learning_ , 5897–5906. PMLR.
  * Stock and Watson (2001) Stock, J. H.; and Watson, M. W. 2001.  Vector autoregressions.  _Journal of Economic perspectives_ , 15(4): 101–115.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , 5998–6008.
  * Yan et al. (2021) Yan, X.; Hu, S.; Mao, Y.; Ye, Y.; and Yu, H. 2021.  Deep multi-view learning methods: a review.  _Neurocomputing_.
  * Zheng et al. (2017) Zheng, X.; Zaheer, M.; Ahmed, A.; Wang, Y.; Xing, E. P.; and Smola, A. J. 2017.  State space LSTM models with particle MCMC inference.  _arXiv preprint arXiv:1711.11179_.
  * Zhu and Laptev (2017) Zhu, L.; and Laptev, N. 2017.  Deep and confident prediction for time series at uber.  In _2017 IEEE International Conference on Data Mining Workshops (ICDMW)_ , 103–110. IEEE.

Appendix for the paper "CAMul: Calibrated and Accurate Multi-view Time-Series
Forecasting"

We provide the code for CAMul models and anonymized datasets at
https://github.com/AdityaLab/CAMul.

##  Appendix A Additional Related works

##### Probabilistic Time-series Forecasting

Classical methods of time-series forecasting including exponential smoothing,
ARIMA-based models (Hyndman and Athanasopoulos 2018) focus on univariate time-
series with small number of exogenous features and learn model parameters
independently for each sequence. Vector Auto-regression (VAR) models extend
ARIMA based models to multivatiate time series (Stock and Watson 2001).

Deep learning based time-series models (Lim and Zohren 2021) such as RNN, GRU
and LSTMs, transformers (Vaswani et al. 2017) have proven useful in modelling
for large number of features capturing long-range complex non-linear patterns
in sequences. However, these methods are trained for point-prediction tasks.
DeepAR (Salinas et al. 2020) directly models the mean and variance parameters
of the forecast distribution using RNN-based architecture. Some recent works
are inspired from the space-state models to explicitly model the transition
and emission components with deep learning modules such as in the case of Deep
Markov Models (Krishnan, Shalit, and Sontag 2017), and recent Deep State Space
models (Li et al. 2021; Rangapuram et al. 2018). Others introduce
stochasticity into state dynamics of recurrent neural networks such as
Stochastic RNN (Fraccaro et al. 2016), Variational RNN (Chung et al. 2015) and
State Space LSTM (Zheng et al. 2017). Bayesian Neural Networks have also been
used to model time-series forecast distributions (Zhu and Laptev 2017) but
require assigning useful priors to parameters and comes with high
computational cost of learning.

Neural Process (NP) (Garnelo et al. 2018) models a global latent variable for
entire dataset to capture uncertainty with is used with input data’s embedding
to model the distribution parameters. Recurrent neural Process (Qin et al.
2019) leverages NP for sequence data. Functional Neural Process (FNP) captures
stochastic correlations between input data and datapoints from training
distribution to provide a flexible non-parametric mechanism to model output
distribution using related training data points in functional space. EpiFNP
(Kamarthi et al. 2021) leveraged this flexible modelling towards sequential
data to propose a epidemic forecasting model on univariate sequences for
state-of-art calibration and uncertainty. In contrast, our work leverages FNP
for uncertainty modelling of each of the individual views and then CAMul
jointly model the total uncertainty by carefully combining distributions from
different views towards a well-calibrated final output distribution.

##### Multi-view time-series forecasting

Extracting useful information from multiple data sources and data modalities
for enriching prediction models has been a very important machine learning
problem with challenges that include noisy data sources, integrating
information from multiple sources and handling different levels of uncertainty
and reliability. Recent advances is deep learning architectures has allowed us
to extract representations from variety of data sources such as images (Li,
Hao, and Lei 2016), sequences (Salehinejad et al. 2017), graphs (Hamilton
2020), etc, and combine these modules’ representation for training in a end-
to-end fashion. In order to integrate these methods most methods employ simple
summation or concatenation methods (Yan et al. 2021; Li, Yang, and Zhang
2018). For instance DeepCovidNet (Ramchandani, Fan, and Mostafavi 2020) uses
spatio-temporal and static features using simple summation for Covid-19
prediction. Ekambaram et al. (2020) similarly integrate images, text and
static features for predicting consumer sales.

Moreover, most of these methods do not focus on probabilistic forecasting
unlike (Chen et al. 2021a) which uses EHR sequence data, static demographic
and location data and learn a data-source specific weight by pre-training the
aggregated embeddings on prediction task of deciding if patient requires a
treatment. Then, they employ a Gaussian Process approach to learn forecast
probability on these embeddings to capture uncertainty.

In contrast, we model data-view specific and modality specific uncertainty by
learning a latent distribution for each view which allows the model to reason
about data reliability, source specific uncertainty and inherent noise when it
integrates uncertainty-aware stochastic representations of all views towards
the final forecast distribution.

##  Appendix B Examples of Multi-view Probabilistic Latent Encoders

We provide examples of Probabilistic Latent Encoders for data views of
different modalities that are used in the paper. Due to generality of our
framework, variety of neural encoders can be used specific to data view and
type of task.

##### Static features:

We encode the static features of fixed dimensions of form
$R_{k}^{(j)}\in\mathbf{R}^{d}$. We employ a feed-forward network to capture
the parameters of distribution.

##### Sequences:

To capture the latent embedding of sequence
$R^{(j)}_{k}=\\{R[0]_{k}^{(j)},R[1]_{k}^{(j)},\dots,R[T]_{k}^{(j)}\\}$, such
as in the default view, we leverage neural sequence encoders GRU (Cho et al.
2014) as:

$\\{\tilde{z}[t]_{k}^{(j)}\\}_{t=0}^{T}=GRU(\\{R[t]_{k}^{(j)}\\}_{t=0}^{T})$
(8)

where $\\{\tilde{z}[t]_{k}^{(j)}\\}_{t=0}^{T}$ are the intermediate states of
GRU.

In order to capture capture long-term relations and prevent over-emphasis on
last terms of sequence we employ self-attention layer introduced in (Vaswani
et al. 2017):

$\begin{split}\\{\alpha[0]_{d},\alpha[1]_{d},\dots,\alpha[T]_{d}\\}&=\text{Self-
Atten}(\\{\tilde{z}[t]_{k}^{(j)}\\}_{t=0}^{T})\\\
\mu_{k}^{(j)},\sigma_{k}^{(j)}&=g^{\prime}_{j}(\sum_{t=0}^{T}\alpha[t]_{d}z[t]_{d}^{(j)})\end{split}$
(9)

where $g^{\prime}_{j}$ is a single feed-forward layer.

##### Graph data:

If the data type of view $j$ contain relations, i.e, the reference sets have
an inherent graph structure denoted by $(A^{(j)},F^{(j)})$, where
$A^{(j)}\in\mathbf{R}^{N_{j}\times N_{j}}$ is the adjacency matrix
(potentially with weights) and $F^{(j)}=\\{f_{i}^{(j)}\\}_{i=1}^{N_{j}}$ are
feature vectors of fixed dimensions, then we can use a Graph Neural network
architecture (Kipf and Welling 2016) to encode the relations in $A^{(j)}$:

$\\{\tilde{z}_{i}^{(j)}\\}_{i=1}^{N_{j}}=GCN(A^{(j)},\\{f_{i}^{(j)}\\}_{i=1}^{N_{j}})$
(10)

Then, we use feed forward layer to derive the distribution parameters:

$\mu_{i}^{(j)},\sigma_{i}^{(j)}=g^{\prime}_{j}(\tilde{z}_{i}^{(j)})$ (11)

##  Appendix C Implementation details

We used numpy for data processing and PyTorch for model training and
inference. We use the properscoring library (Leon Barrett and O’Kane 2015) to
implement the CRPS evaluation.

###  C.1 Data pre-processing

Scaling values: Since time-series and exogenous features can have wide range
of values, we normalize the values of of each features with mean 0 and
variance 1. We derive the scaling factors for training dataset and apply the
transformation to test set during inference.

Chunking time-series for training: The long training sequences are split using
the shingling technique (Leskovec, Rajaraman, and Ullman 2020; Li et al. 2021)
where we fix a window size $W=10$ and randomly sample chunks of $W$ size from
the full sequence over the interval $[1,t-W-1]$ and record the $\tau$ ahead
value as ground truth. Note that the reference set of default view $1$ still
contains the full length sequence of training set, we only use the split
sequence as input for training set.

###  C.2 Hyperparameters

We describe the hyperparameters for CAMul for all benchmarks.

##### Multi-view probabilistic encoders

For the feed-forward networks of static view, we used a 3 layer network with
60 hidden units. We also used 60 hidden units for GRU of sequence views and
used a bi-directional version of GRU. For graph views, we used a 2-layer GCN
network with 60 hidden units. The final layer outputs 120 dimensional vector
which we split into mean and variance. We apply exponentiation on the variance
vector to make it positive. The latent embeddings $z_{i}^{(j)}$ for all views
thus have dimension of 60.

##### View specific correlation graph

The networks $l_{j}^{\mu}$ and $l_{j}^{\sigma}$ is a 2 layer feed forward
network with 60 hidden units. Thus, the dimension of view-aware latent
embedding is 60.

##### Dynamic view-selection module

We pass the view-aware latent embeddings and $z_{i}^{(1)}$ each through a
single 60-hidden unit feed forward layer ($h_{1}$, $h_{2}$) before computing
the cross attention weights. The combined-view embedding $\tilde{u}_{i}$ is a
60 dimensional vector.

##### Forecast decoder

We use a 3 layer feed forward layer that inputs the latent embedding of
sequence $z_{i}^{(1)}$ and $\tilde{u}_{i}$ and has 60 hidden units in each
hidden layer with final layer outputting 2 scalars $\mu(y_{i}),\sigma(y_{i})$.

Throughout all layers of CAMul we apply exponentiation to variance
vector/scalar to make it positive. We also use ReLU activation for hidden
layers unless specified otherwise. We also use ADAM optimizer (Kingma and Ba
2014) for parameter updates.

##### Note on hyperparameter selection

We sampled a validation set containing 10% of randomly selected chunks of
sequences from training set. This was used for model hyperparameter tuning. We
describe splitting of dataset into training and test set in Section D below
for each benchmark.

We found that the performance was not significantly sensitive to model
architecture, batch size or learning rate. We searched over the space of
$\\{30,60,120,250\\}$ for hidden units and mostly optimized for faster
convergence. We also searched over $\\{10,20,50,80\\}$ for batch-size and
found $20,20,10,50$ to be most optimal for tweet, covid19, google-symptoms,
power respectively. We used $0.0005$ as learning rate.

For power and google-symptoms we observed that CAMul took less than 700 epochs
to converge whereas it took 2000 and 1000 epochs for covid19 and tweet.
Regarding seed selection, we initialized random seeds to 0 to 19 for numpy and
pytorch for the 20 trials done for each benchmarks and didn’t observe any
significant variation in scores.

1

Input : Training sequences
$\\{Y_{N}\\}_{i=1}^{N}=\\{X_{i}^{(1)}\\}_{i=1}^{N}$, target labels
$\\{y[t+\tau]_{i}\\}_{i=1}^{N}$, view data sources
$\\{X_{i}^{(j)}\\}_{i=1}^{N}$ for each view $j\in\\{2,\dots,K\\}$, reference
points $\\{R_{i}^{(j)}\\}_{i=1}^{N_{j}}$ for each view $j$

2

3for  _$i\sim\\{1,\dots,N\\}$_ do

4

5 for  _$j\in\\{1,\dots,K\\}$_ do

/* Stochastic embeddings from latent stochastic encoder */

6 Sample $z_{i}^{(j)}$ as Eqn 1 using encoder $g_{\theta_{j}}$;

7 for  _$R_{k}^{(j)}\in\mathcal{R}^{(j)}$_ do

8 Sample $z_{k}^{(j)}$ using encoder $g_{\theta_{j}}$;

9

10 end for

/* Sample edges for SVCG */

11 for  _$R_{k}^{(j)}\in\mathcal{R}^{(j)}$_ do

12 Add $(k,i)$ to $G^{(j)}$ with probability $k(z_{k}^{(j)},z_{i}^{(j)})$;

13

14 end for

15 Derive $\mu(u_{i}^{(j)}),\sigma(u_{i}^{(j)})$ from sampled edges
$\\{k:(k,i)\in G^{(j)}\\}$ as Eqn 4;

16

/* Sample from Variational distribution */

17 Sample $\hat{u}_{i}^{(j)}$ from variational distribution $q_{j}$;

18

19 end for

20

/* Aggregate all view-aware embeddings using Dynamic view-selection module */

21 Compute importance weights $\\{\alpha_{i}^{(j)}\\}_{j=1}^{K}$ using cross-
attention as Eqn 5 from $\\{\hat{u}_{i}^{(j)}\\}_{j=1}^{K}$;

22 Compute the combined view embedding
$\tilde{u}_{i}^{(j)}=\sum_{j=1}^{K}\alpha_{i}^{(j)}\hat{u}_{i}^{(j)}$;

23

/* Final output distribution */

24 Derive $\mu(y_{i}),\sigma(y_{i})$ from $z_{i}^{(1)}$ and
$\tilde{u}_{i}^{(j)}$ via the decoder (Eqn 7);

25

/* Sample ELBO Loss */

26 Compute $\mathcal{L}_{1}=\log P(y[t+\tau]_{i}|\mu(y_{i}),\sigma(y_{i}))$;

27 Compute $\mathcal{L}_{2}=\sum_{j=1}^{K}\log
P(\hat{u}_{i}^{(j)}|\mu(u_{i}^{(j)}),\sigma(u_{i}^{(j)}))-\log
q_{j}(\hat{u}_{i}^{(j)}|X_{i}^{(j)})$;

28 Accumulate gradient for loss
$\mathcal{L}=-(\mathcal{L}_{1}+\mathcal{L}_{2})$;

29

30 end for

31Periodically update the weights of all modules of CAMul from accumulated
gradients using ADAM;

32

Algorithm 1 Training Algorithm for CAMul

1

Input : Input sequence $Y_{i}=X_{i}^{(1)}$, view data sources $X_{i}^{(j)}$
for each view $j\in\\{2,\dots,K\\}$, reference points
$\\{R_{i}^{(j)}\\}_{i=1}^{N_{j}}$ for each view $j$

2

3for  _$j\in\\{1,\dots,K\\}$_ do

/* Stochastic embeddings from latent stochastic encoder */

4 Sample $z_{i}^{(j)}$ as Eqn 1 using encoder $g_{\theta_{j}}$;

5 for  _$R_{k}^{(j)}\in\mathcal{R}^{(j)}$_ do

6 Sample $z_{k}^{(j)}$ using encoder $g_{\theta_{j}}$;

7

8 end for

/* Sample edges for SVCG */

9 for  _$R_{k}^{(j)}\in\mathcal{R}^{(j)}$_ do

10 Add $(k,i)$ to $G^{(j)}$ with probability $k(z_{k}^{(j)},z_{i}^{(j)})$;

11

12 end for

13 Sample view-aware latent embedding $u_{i}^{(j)}$ from sampled edges
$\\{k:(k,i)\in G^{(j)}\\}$ as Eqn 4;

14

15 end for

16

/* Aggregate all view-aware embeddings using Dynamic view-selection module */

17 Compute importance weights $\\{\alpha_{i}^{(j)}\\}_{j=1}^{K}$ using cross-
attention as Eqn 5;

18 Compute the combined view embedding
$\tilde{u}_{i}^{(j)}=\sum_{j=1}^{K}\alpha_{i}^{(j)}u_{i}^{(j)}$;

19

/* Final output distribution */

20 Sample $y[t+\tau]_{i}$ from $z_{i}^{(1)}$ and $\tilde{u}_{i}^{(j)}$ via the
decoder (Eqn 7);

21

Algorithm 2 Inference Algorithm for CAMul

##  Appendix D Details of Benchmarks

####  google-symptoms: Flu forecasting from Google symptoms

We use aggregated and anonymized search counts of flu-related symptoms
searches from each US state to predict future incidence of influenza. Dataset:
The flu incidence rate is represented by wILI (weighted Influenza related
illness) values released weekly by CDC for 8 regions of USA. The aggregate
symptom counts for over 400 symptoms are anonymized and publicly released for
each week since 2017 at county and state level by Google (Health 2021). We
choose to extract 14 symptoms related to influenza referred to in the CDC
website (Centers for Disease Control and Prevention 2021). We aggregate these
counts for each HHS region and use it as exogenous features. We also use
spatial adjacency data between HHS regions and time-related data (month and
year) as features. For forecasting wILI values for a given year, we use the
training set from all past years for model training and hyperparameter tuning.

Views: We use following diverse views for this benchmarks: 1\. Default
Historical wILI view: This view contains reference points of symptoms features
for all previous forecasting seasons for all HHS regions. We use the GRU
encoder for this view. 2\. HHS adjacency view: This is a graph feature view.
We have 8 reference points corresponding to HHS regions. The features
$\mathcal{F}^{(j)}$ are one-hot encodings. The adjacency matrix $A^{(j)}$
encodes the neighbourhood information between HHS regions that share a border.
We aptly use the GCN based latent encoder. 3\. Month view: This is a static
feature view. We have 12 reference points for each month. We use an embedding
layer to learn encodings for each reference point from one-hot embedding and
use the feed-forward latent encoder.

####  covid19: Covid-19 mortality forecasting

We also evaluate our model on the challenging task of forecasting COVID-19
mortality for each of the 50 US states. Dataset: We use the mortality data and
exogenous features used in (Rodríguez et al. 2021b; Kamarthi et al. 2021). The
exogenous features contain multiple kinds of data including hospital records,
mobility, exposure and survey data. We evaluate for duration of 7 months from
June 2020 to December 2020 and again use only past weeks’ data to train before
forecasting for 1 to 4 weeks ahead mortality. We train the model separately
for each week over all states using data upto past week as training set.

Views: 1\. Default Mortality view: This is a sequence view where each
reference point is a time-series of past mortality. 2. Line-list view This is
a multivariate sequence view that contains 7 weekly-collected features from
traditional surveillance including hospitalization and testing. 3. Mobility
and exposure view Similarly we use the digitally collected signals measuring
aggregate mobility and individuals collected from smartphones. For views 1,2
and 3 we use a GRU based encoder. 4. Demographic view This is a static view
where we encode each of the 50 states using 8 demographic features including
average income, unemployment, population, average age and voting preferences.
We use a feed-forward network for encoder. 5. State adjacency view: Similar to
flu-symptoms we construct a graph view of 50 states and encode spatial
adjacency states. We use a GCN-based encoder.

####  power :Power consumption forecasting

We also evaluate on household power consumption data used as a standard
forecasting benchmark. (Hebrail and Berard 2012) Dataset: It contains 260,640
measurements of power consumption of a household for a year using measurements
from 3 meters with a total of 7 features. Our goal is to forecast the total
active power consumption for 1 minute in future. We use the data from the
first 9 months of measurement as training set and last 3 months as test set
for forecasting. Views: 1\. Default Past sequence view We randomly sample
single day sequence from each of the past months as reference sets. 2. Time of
Day view: This is a static view We divide the 24 hours of a day into 6 equal
intervals and assign each reference point an interval using one-hot encoding.
3. Month view: This is a static view. We assign a reference point for each
month using one-hot encodings.

####  tweet: Tweet Topics prediction

Our goal for this task is to evaluate the topic distribution of tweets in the
future given topic distributions of past weeks similar to Shi et al. (2019).
Dataset: We collect Covid-19 related tweets for 15 weeks that have a
geographical tag to identify US state of the user. We extract 30 topics using
LDA (Blei, Ng, and Jordan 2003). We also allocate each tweet from a given week
to each of the 30 topics it is most likely related to using the LDA model for
each state. Thus, we have a multivariate sequence dataset where for each of
the 50 states, we have sequences containing topic distribution of tweets for
each week. Similar to google-symptoms, for each forecasting for each year, we
use data from past year as training set.

Views: 1\. Default Past sequence view: Similar to other task the first view
contains the past sequences for each state. 2. State adjacency view: We
construct a state adjacency view similar to covid19 task. 3. Month view:
Similar to power task. 4. Demographics view Same as view 4 of covid19 task.

Setup Remarks: We train a separate decoder module for each of the 30 topics.
The scores reported are average over all topics.

##  Appendix E Evaluation metrics

Cumulative ranked probability score (CRPS): The CRPS is a standard metric used
for evaluation of probabilistic forecasts and is a proper scoring rule that
generalizes mean average error to probabilistic forecasting (Gneiting and
Katzfuss 2014). Given ground truth $y$ and the predicted probability
distribution $\hat{p}(Y)$, let $\hat{F}_{y}$ be the CDF. Then CRPS is defined
as:

$CRPS(\hat{F}_{y},y)=\int_{-\infty}^{\infty}(\hat{F}(\hat{y})-\mathbf{1}\\{\hat{y}>y\\})^{2}d\hat{y}$

.

Interval Score(IS): IS is a standard score used in evaluation of probabilistic
forecasts in epidemiology (Reich et al. 2019) which measures the negative log
likelihood of a fixed size interval around the ground truth under the
predictive distribution:

$IS(\hat{p}_{y},y)=-\int_{y-L}^{y+L}\log\hat{p}_{y}(\hat{y})d\hat{y}.$

Confidence Score (CS): CS (Kamarthi et al. 2021) measures the overall
calibration of predictions of a model $M$ by calculating fraction $k_{m}(c)$
of prediction distributions that cover the ground truth at each confidence
interval $c$. A perfectly calibrated model has $k_{m}(c)$ very close to c. We,
therefore define CS as:

$CS(M)=\int_{0}^{1}|k_{m}(c)-c|dc$

.

Calibration score is also similar to Calibration error introduced in Kuleshov,
Fenner, and Ermon (2018).

##  Appendix F Correlation between Attention weights of View selection Module
and CRPS scores of two view variants

As discussed in Q4 of Section 5.1, the attention weights of the view selection
module correlated with the predictive utility of a view which we tested using
a 2 view variant of CAMul containing the default view and the view of
interest. The inverse correlation between CRPS and attention weights for all
benchmarks is shown in Figure 3.

Figure 3: CRPS is inverseley correlated with attention weights of View
selection module

##  Appendix G Additional Observation on view selection weights

Obs 2: In relation to google-symptoms task, for the weeks around the peak
week, the weights of views other than sequence view decreases by over 30% on
average for all HHS regions.

The month and HHS adjacency view capture information about seasonal and
average region-specific patterns. However, near the peak weeks where wILI
values are more volatile, CAMul relies on the past wILI sequence patterns
including values from past weeks for interpolation which is observed by the
sudden decrease in attention weights of other views. Therefore, we observed an
average decrease of 32.4% in attention weights of Month and Adjacency view
during the 4 weeks around the peak week (the week with highest wILI value).

(a) wILI values for 2018 season

(b) Average attention weights of views for 2018 season

Figure 4: Sequence view is more important during peak weeks

An example is shown in Figure 4 for the 2018 flu season. We see that the peak
week is at week 40. The weights corresponding the past sequence view sees an
increase on weeks 38 - 42 whereas weights of other views decrease.

(a) power | Isotonic | DC
---|---|---
Model | CRPS | CS | IS | CRPS | CS | IS
SARIMA | 1.32 | 0.14 | 3.28 | 1.7 | 0.23 | 3.95
DeepAR | 0.67 | 0.04 | 1.55 | 0.71 | 0.08 | 1.64
DSSM | 0.62 | 0.04 | 1.62 | 0.81 | 0.09 | 1.84
RNP | 0.87 | 0.17 | 2.55 | 0.93 | 0.16 | 2.58
GP | 1.03 | 0.12 | 2.78 | 1.03 | 0.12 | 2.78
GraphDF-RBF | 0.92 | 0.13 | 1.45 | 1.04 | 0.18 | 1.31
CAMul | 0.43 | 0.02 | 0.91 | 0.47 | 0.04 | 1.02

(b) tweet | Isotonic | DC
---|---|---
Model | CRPS | CS | IS | CRPS | CS | IS
SARIMA | 1.18 | 0.22 | 3.39 | 1.27 | 0.33 | 3.25
DeepAR | 1.02 | 0.18 | 1.24 | 1.17 | 1.15 | 1.36
DSSM | 1.19 | 0.08 | 1.15 | 1.21 | 0.17 | 1.83
RNP | 1.09 | 0.15 | 1.89 | 1.17 | 0.13 | 2.05
GP | 1.25 | 0.15 | 1.86 | 1.27 | 0.15 | 2.38
GraphDF-RBF | 0.7 | 0.1 | 1.61 | 0.75 | 0.08 | 1.92
GraphDF-Adj | 1.15 | 0.18 | 2.63 | 1.17 | 0.18 | 2.39
CAMul | 0.55 | 0.06 | 0.68 | 0.67 | 0.07 | 0.86

(c) covid19 | Isotonic | DC
---|---|---
Model | CRPS | CS | IS | CRPS | CS | IS
SARIMA | 57.7 | 0.34 | 5.63 | 62.3 | 0.39 | 5.93
DeepAR | 30.4 | 0.16 | 2.11 | 34.2 | 0.21 | 2.23
DSSM | 44.1 | 0.2 | 1.73 | 54 | 0.28 | 2.55
RNP | 39.1 | 0.31 | 3.16 | 39.4 | 0.34 | 4.28
GP | 23.1 | 0.26 | 3.14 | 23.7 | 0.29 | 3.27
GraphDF-RBF | 36 | 0.23 | 2.96 | 36.9 | 0.27 | 2.98
GraphDF-Adj | 38.6 | 0.29 | 2.4 | 38.6 | 0.29 | 2.4
DeepCovid | 29.1 | 0.16 | 2.49 | 29.1 | 0.16 | 2.49
CMU-TS | 21.1 | 0.13 | 2.8 | 21.1 | 0.13 | 2.8
CAMul | 13.9 | 0.12 | 1.28 | 13.9 | 0.12 | 1.28

(d) google-symptoms | Isotonic | DC
---|---|---
Model | CRPS | CS | IS | CRPS | CS | IS
SARIMA | 0.93 | 0.39 | 2.56 | 0.97 | 0.35 | 2.62
DeepAR | 0.54 | 0.15 | 1.52 | 0.58 | 0.14 | 1.54
DSSM | 0.5 | 0.12 | 2.14 | 0.56 | 0.14 | 2.36
RNP | 0.65 | 0.21 | 2.34 | 0.67 | 0.27 | 1.65
GP | 0.64 | 0.09 | 1.86 | 0.71 | 0.1 | 1.84
GraphDF-RBF | 0.53 | 0.1 | 1.07 | 0.61 | 0.12 | 1.23
GraphDF-Adj | 0.64 | 0.07 | 2.87 | 0.67 | 0.1 | 2.73
EpiDeep | 0.87 | 0.11 | 2.46 | 0.85 | 0.14 | 2.11
EpiFNP | 0.32 | 0.09 | 0.56 | 0.34 | 0.07 | 0.55
CAMul | 0.25 | 0.05 | 0.41 | 0.23 | 0.06 | 0.47

Table 5: Evaluation scores of CAMul and baselines after applying post-hoc
calibration. We use Isotonic regression (Kuleshov, Fenner, and Ermon 2018) and
Distribution calibration (Song et al. 2019) methods. The evaluation scores of
CAMul is statistically significantly better and does not change much due to
post-hoc methods, implying that our approach produces well-calibrated
forecasts without need for such post-hoc correction methods.

| covid19 (1 week ahead) |  | google-symptoms (1 week ahead)
---|---|---|---
| CRPS | CS | IS |  | CRPS | CS | IS
SARIMA | 55 $\pm$ 3.7 | 0.36 $\pm$ 0.014 | 5.51 $\pm$ 0.034 | SARIMA | 1.08 $\pm$ 0.017 | 0.38 $\pm$ 0.008 | 2.84 $\pm$ 0.24
DeepAR | 34.6 $\pm$ 5.3 | 0.17 $\pm$ 0.018 | 1.97 $\pm$ 0.039 | DeepAR | 0.49 $\pm$ 0.013 | 0.16 $\pm$ 0.003 | 1.11 $\pm$ 0.11
DSSM | 45.6 $\pm$ 2.1 | 0.15 $\pm$ 0.028 | 2.14 $\pm$ 0.061 | DSSM | 0.51 $\pm$ 0.010 | 0.19 $\pm$ 0.003 | 1.53 $\pm$ 0.14
RNP | 46.4 $\pm$ 6.6 | 0.27 $\pm$ 0.013 | 3.70 $\pm$ 0.068 | RNP | 0.77 $\pm$ 0.043 | 0.32 $\pm$ 0.006 | 3.49 $\pm$ 0.15
GP | 24.9 $\pm$ 8.4 | 0.18 $\pm$ 0.004 | 2.64 $\pm$ 0.058 | GP | 0.59 $\pm$ 0.016 | 0.29 $\pm$ 0.008 | 1.25 $\pm$ 0.43
GraphDF-RBF | 27.5 $\pm$ 2.1 | 0.17 $\pm$ 0.016 | 2.77 $\pm$ 0.049 | GraphDF- RBF | 0.61 $\pm$ 0.016 | 0.13 $\pm$ 0.002 | 0.96$\pm$ 0.24
GraphDF-Adj | 29.3 $\pm$ 2.4 | 0.21 $\pm$ 0.012 | 2.10 $\pm$ 0.037 | GraphDF- Adj | 0.52 $\pm$ 0.055 | 0.16 $\pm$ 0.005 | 1.68 $\pm$ 0.61
DeepCovid | 26.4 $\pm$ 7.1 | 0.16 $\pm$ 0.009 | 2.83$\pm$0.037 | EpiDeep | 0.93 $\pm$ 0.049 | 0.28 $\pm$ 0.007 | 3.16 $\pm$ 0.91
CMU-TS | 19.4 $\pm$ 3.1 | 0.16 $\pm$ 0.017 | 2.63 $\pm$ 0.029 | EpiFNP | 0.31 $\pm$ 0.057 | 0.03 $\pm$ 0.002 | 0.37 $\pm$ 0.06
CAMul | 16.7 $\pm$ 8.8 | 0.09 $\pm$ 0.010 | 1.12 $\pm$ 0.035 | CAMul | 0.19 $\pm$ 0.064 | 0.04 $\pm$ 0.004 | 0.35 $\pm$ 0.05
CAMul-C | 17.5$\pm$3.5 | 0.12$\pm$0.005 | 1.71$\pm$0.021 |  CAMul-C | 0.26$\pm$0.022 | 0.05 $\pm$ 0.001 | 0.39 $\pm$ 0.08
CAMul-S | 17.2$\pm$4.6 | 0.14$\pm$0.004 | 1.52 $\pm$ 0.035 |  CAMul-S | 0.25$\pm$0.064 | 0.06 $\pm$0.006 | 0.41 $\pm$0.06
CAMul-D | 21.6$\pm$8.4 | 0.21$\pm$0.008 | 2.76 $\pm$0.038 |  CAMul-D | 0.62$\pm$0.081 | 0.23 $\pm$0.002 | 1.27$\pm$0.03

| covid19 (3 week ahead) |  | google-symptoms (3 week ahead)
---|---|---|---
| CRPS | CS | IS |  | CRPS | CS | IS
SARIMA | 75 $\pm$ 2.6 | 0.44 $\pm$ 0.015 | 7.06 $\pm$ 0.033 | SARIMA | 1.94 $\pm$ 0.015 | 0.43 $\pm$ 0.014 | 6.49 $\pm$ 0.32
DeepAR | 47.9 $\pm$ 5.2 | 0.24 $\pm$ 0.016 | 2.29 $\pm$ 0.027 | DeepAR | 1.43 $\pm$ 0.011 | 0.14 $\pm$ 0.008 | 3.38 $\pm$ 0.16
DSSM | 55.9 $\pm$ 4.6 | 0.19 $\pm$ 0.012 | 3.08 $\pm$ 0.060 | DSSM | 1.53 $\pm$ 0.016 | 0.13 $\pm$ 0.005 | 3.76 $\pm$ 0.12
RNP | 64.7 $\pm$ 7.1 | 0.34 $\pm$ 0.015 | 4.93 $\pm$ 0.051 | RNP | 1.05 $\pm$ 0.032 | 0.43 $\pm$ 0.003 | 3.72 $\pm$ 0.13
GP | 33.8 $\pm$ 2.8 | 0.26 $\pm$ 0.014 | 3.83 $\pm$ 0.033 | GP | 1.43 $\pm$ 0.026 | 0.21 $\pm$ 0.009 | 2.21 $\pm$ 0.47
GraphDF-RBF | 36.8 $\pm$ 1.6 | 0.19 $\pm$ 0.021 | 2.89 $\pm$ 0.036 | GraphDF- RBF | 1.47 $\pm$ 0.019 | 0.08 $\pm$ 0.005 | 2.53$\pm$ 0.31
GraphDF-Adj | 43.4 $\pm$ 4.7 | 0.24 $\pm$ 0.018 | 3.24 $\pm$ 0.010 | GraphDF- Adj | 1.58 $\pm$ 0.036 | 0.08 $\pm$ 0.004 | 3.15 $\pm$ 0.39
DeepCovid | 46.2 $\pm$ 6.6 | 0.26 $\pm$ 0.009 | 3.71 $\pm$ 0.031 | EpiDeep | 1.13 $\pm$ 0.031 | 0.24 $\pm$ 0.013 | 6.35 $\pm$ 0.71
CMU-TS | 32.3 $\pm$ 5.1 | 0.21 $\pm$ 0.016 | 3.80 $\pm$ 0.065 | EpiFNP | 0.72 $\pm$ 0.062 | 0.07 $\pm$ 0.003 | 0.83 $\pm$ 0.11
CAMul | 26.2 $\pm$ 6.3 | 0.14 $\pm$ 0.009 | 1.83 $\pm$ 0.014 | CAMul | 0.46 $\pm$ 0.042 | 0.05 $\pm$ 0.007 | 0.68 $\pm$ 0.22
CAMul-C | 29.5$\pm$4.6 | 0.17$\pm$0.009 | 2.19$\pm$0.022 |  CAMul-C | 0.57$\pm$0.030 | 0.06 $\pm$ 0.003 | 0.76 $\pm$ 0.12
CAMul-S | 26.3$\pm$1.4 | 0.19$\pm$0.005 | 1.99 $\pm$ 0.042 |  CAMul-S | 0.54$\pm$0.052 | 0.06$\pm$0.011 | 0.71$\pm$0.07
CAMul-D | 38.1$\pm$5.1 | 0.26$\pm$0.019 | 3.38 $\pm$0.035 |  CAMul-D | 1.15$\pm$0.077 | 0.17$\pm$0.007 | 2.86$\pm$0.11

| covid19 (4 week ahead) |  | google-symptoms (4 week ahead)
---|---|---|---
| CRPS | CS | IS |  | CRPS | CS | IS
SARIMA | 85.3 $\pm$ 3.9 | 0.43 $\pm$ 0.013 | 7.86 $\pm$ 0.038 | SARIMA | 2.31$\pm$ 0.013 | 0.35 $\pm$ 0.006 | 7.72 $\pm$ 0.24
DeepAR | 59.4 $\pm$ 5.0 | 0.19 $\pm$ 0.018 | 3.85 $\pm$ 0.021 | DeepAR | 1.42$\pm$ 0.016 | 0.17 $\pm$ 0.004 | 5.63 $\pm$ 0.13
DSSM | 68.2 $\pm$ 4.3 | 0.21 $\pm$ 0.014 | 3.16 $\pm$ 0.051 | DSSM | 1.57$\pm$ 0.015 | 0.15 $\pm$ 0.002 | 4.91 $\pm$ 0.14
RNP | 74.5 $\pm$ 6.4 | 0.33 $\pm$ 0.017 | 5.19 $\pm$ 0.044 | RNP | 1.27$\pm$ 0.053 | 0.45 $\pm$ 0.008 | 4.13 $\pm$ 0.15
GP | 58.7 $\pm$ 2.1 | 0.23 $\pm$ 0.016 | 4.11 $\pm$ 0.017 | GP | 1.37$\pm$ 0.032 | 0.31 $\pm$ 0.008 | 3.80 $\pm$ 0.41
GraphDF-RBF | 57.7 $\pm$ 2.3 | 0.21 $\pm$ 0.018 | 3.42 $\pm$ 0.036 | GraphDF- RBF | 1.53$\pm$ 0.013 | 0.13 $\pm$ 0.006 | 2.79$\pm$ 0.24
GraphDF-Adj | 65.3 $\pm$ 3.0 | 0.20 $\pm$ 0.010 | 3.79 $\pm$ 0.023 | GraphDF- Adj | 1.64 $\pm$ 0.058 | 0.17 $\pm$ 0.003 | 2.87 $\pm$ 0.61
DeepCovid | 56.3 $\pm$ 6.8 | 0.28 $\pm$ 0.014 | 4.57 $\pm$ 0.029 | EpiDeep | 1.27 $\pm$ 0.038 | 0.41 $\pm$ 0.011 | 8.41 $\pm$ 0.91
CMU-TS | 47.0 $\pm$ 4.6 | 0.32 $\pm$ 0.016 | 3.62 $\pm$ 0.069 | EpiFNP | 0.75 $\pm$ 0.052 | 0.04 $\pm$ 0.004 | 0.93 $\pm$ 0.18
CAMul | 24.6 $\pm$ 4.4 | 0.19 $\pm$ 0.018 | 2.26 $\pm$ 0.028 | CAMul | 0.48 $\pm$ 0.057 | 0.04 $\pm$ 0.002 | 0.71 $\pm$ 0.02
CAMul-C | 41.0$\pm$3.3 | 0.25$\pm$0.015 | 2.51$\pm$0.019 |  CAMul-C | 0.64$\pm$0.024 | 0.06 $\pm$ 0.006 | 0.75 $\pm$ 0.06
CAMul-S | 38.7$\pm$3.6 | 0.21$\pm$0.018 | 3.22 $\pm$ 0.043 |  CAMul-S | 0.56$\pm$0.043 | 0.08$\pm$0.008 | 0.83$\pm$0.02
CAMul-D | 52.3$\pm$8.3 | 0.34$\pm$0.036 | 4.14 $\pm$0.025 |  CAMul-D | 1.44$\pm$0.065 | 0.15$\pm$0.006 | 2.99$\pm$0.04

Table 6: Evaluation scores (over 20 runs) for CAMul and baselines for all
covid19 and google-symptoms for 1,3 and 4-week ahead targets. Similar to the
results discussed in Section 5.1 Q1, our CAMul significantly outperforms the
state-of-art baselines and variants without dynamic view selection and
stochastic view-specific uncertainty modelling.