#  PROFHiT: Probabilistic Robust Forecasting for Hierarchical Time-series

Harshavardhan Kamarthi Lingkai Kong Alexander Rodríguez
Chao Zhang B. Aditya Prakash
College of Computing
Georgia Institute of Technology
<EMAIL_ADDRESS>

###### Abstract

Probabilistic hierarchical time-series forecasting is an important variant of
time-series forecasting, where the goal is to model and forecast multivariate
time-series that have underlying hierarchical relations. Most methods focus on
point predictions and do not provide well-calibrated probabilistic forecasts
distributions. Recent state-of-art probabilistic forecasting methods also
impose hierarchical relations on point predictions and samples of distribution
which does not account for coherency of forecast distributions. Previous works
also silently assume that datasets are always consistent with given
hierarchical relations and do not adapt to real-world datasets that show
deviation from this assumption. We close both these gap and propose PROFHiT,
which is a fully probabilistic hierarchical forecasting model that jointly
models forecast distribution of entire hierarchy. PROFHiT uses a flexible
probabilistic Bayesian approach and introduces a novel Distributional
Coherency regularization to learn from hierarchical relations for entire
forecast distribution that enables robust and calibrated forecasts as well as
adapt to datasets of varying hierarchical consistency. On evaluating PROFHiT
over wide range of datasets, we observed 41-88% better performance in accuracy
and calibration. Due to modeling the coherency over full distribution, we
observed that PROFHiT can robustly provide reliable forecasts even if up to
10% of input time-series data is missing where other methods’ performance
severely degrade by over 70%.

##  1 Introduction

Figure 1: Coherency of point predictions used in previous works vs
Distributional Coherency (Ours)

Time-series forecasting is an important problem that impacts decision-making
in a wide range of applications and has attracted increasing interests. In
many real-world situations, the time-series have inherent hierarchical
relations and structures. The values of time-series follow these hierarchical
relations with varying degree of consistency. The data available at lower
levels of hierarchy usually contain complex patterns, and leveraging the given
hierarchy to provide coherent forecasts allows for better performance than
independently modeling individual time-series. Given a hierarchical time-
series dataset of varying consistency, the desirable goal of Hierarchical
Time-series Forecasting (HTSF) is to generate forecast for all time-series
such that they are coherent with the underlying hierarchical relations [20]
(forecasts are coherent if they satisfy the hierarchical relations). Examples
of HTSF include modeling dependency of time-series of employment numbers [24]
measured at different geographical scales, epidemic forecasting at county,
state and country, sales demand data at multiple spatial scales and so on.

Classical methods for HTSF [10, 9] typically use a bottom-up or a top-down
approach where all time-series for a single level of hierarchy (usually top-
most or bottom-most) are modeled independently and the values of time-series
of other levels are derived using the aggregation function governing the
hierarchy. This is because they silently assume that the datasets strictly
satisfy the hierarchical constraints. They typically model for datasets
generated [24] by first collecting data for time-series of the leaf level
nodes of hierarchy and deriving time-series for higher levels of hierarchy.
For example, the well-studied Wiki dataset [24] contains visit counts for 150
groups of articles and these values are further aggregated by topic similarity
to form a 4-level hierarchy. We call such data as strongly consistent. In
contrast, many real-world datasets are weakly consistent and do not follow the
strict constraints of the hierarchy. Such data have an underlying data
generation process that may follow a hierarchical set of constraints but may
contain some deviations. These deviations can be caused by multiple factors
such as noise, measurement or reporting error, asynchrony in data aggregation
and revision pipeline, etc. and are usually more pronounced at leaf nodes of
the hierarchy. For example, in the case of epidemic forecasting [1], data at
the county and state levels have higher noise and discrepancies compared to
data at the national level. Most of the state-of-art HTSF methods are designed
for strongly consistent datasets — they thus may not adapt to such deviations
and can provide poor forecasts in weakly consistent scenarios.

Additionally, most of these methods focus only on _point predictions_ and do
not provide probabilistic forecasts which quantify the uncertainty of the
forecasts. While recent works on probabilistic HTSF propose deep-learning
approaches for modeling all time-series of the hierarchy simultaneously, they
impose coherency on point predictions only by sampling from distributions [21]
or quantiles of distributions [8]. Therefore, the forecast distributions may
still be incoherent. This can subsequently lead to bad calibration [12, 15]
i.e., produce wildly wrong predictions (especially in novel/atypical
scenarios) and can even be confident in their mistakes. Poor calibration can
also effect the robustness of the models i.e., the ability to perform well in
presence noise and missing data as we also see in our experiments later.

Table 1: Comparison of the features of PROFHiT with recent state-of-art
methods.

|  | 2-step
---
methods [9]
|  MinT [26]
---
ERM [24]
| HierE2E
---
[21]
| SHARQ
---
[8]
| PROFHiT
---
(This paper)
| Leverages full
---
hierarchy
$\times$ | ✓ | ✓ | ✓ |  TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
| Probabilistic
---
Forecasts
$\times$ | $\times$ | ✓ | ✓ |  TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
| Robust to
---
missing data
$\times$ | $\times$ | $\times$ | $\times$ |  TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
| Strongly
---
Consistent
✓ | ✓ | ✓ | ✓ |  TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
| Weakly
---
Consistent
$\times$ | $\times$ | $\times$ | ✓ |  TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
| Distributional
---
Coherency
$\times$ | $\times$ | $\times$ | $\times$ |  TextRenderingMode=FillStroke, LineWidth=.5pt, ✓

In this work, we fill both these gaps. First, our method models calibrated
forecast distributions that are coherent to hierarchical relations. Therefore,
the forecast distribution of any parent node is similar to aggregation of
random variables of children nodes’ forecast distribution. We denote this as
Distributional Coherency (Figure 1). Second, our model is trained to adapt to
datasets with different levels of hierarchical consistency. To this end, we
propose PROFHiT (Probabilistic Robust Forecasting for Hierarchical Time-
series), a novel neural probabilistic HTSF method that provides a Bayesian
approach to model the distributions of forecasts of all time-series together.
PROFHiT first learns a raw forecast distribution using only past data of each
time-series. Using the raw forecasts as priors, PROFHiT produces a refined set
of distribution with a novel refinement module, which combines training on
patterns from individual time-series as well as leveraging hierarchical
information. This provides PROFHiT the flexibility to adapt to datasets with
varying levels of consistency towards hierarchical constraints. We optimize
the refined parameters of distribution towards coherency using the novel
Distributional Coherency Loss rather than fitting over samples from marginals.
This enables the coherent refined forecast distributions to be well-calibrated
and robust to noise and missing data. We compare the benefits of PROFHiT with
previous methods in Table 1. Our main contributions are summarized as follows:

(1) Robust Probabilistic Hierarchical Time-Series Forecasting: We propose
PROFHiT, a deep probabilistic framework for modeling the distributions of each
time-series of multi-variate forecasts in conjunction by enforcing underlying
hierarchical relations. Our method leverages probabilistic deep-learning
models [12] to model priors of individual time-series and then refine the
priors of all time-series leveraging hierarchical relations to provide robust
and well-calibrated forecasts.
(2) Distributional Coherency and Adaptation to Consistency: We introduce a
novel refinement module and leverage multi-task learning over shared
parameters that enable PROFHiT to perform well on different datasets that
enforce varied levels of hierarchical consistency. We also introduce
Distributional Coherency Loss regularization, which imposes soft hierarchical
constraints on the forecast at the distributional level to learn accurate and
calibrated forecast distribution.
(3) Evaluation Across Multiple Datasets and with Missing Data: PROFHiT is
effective across multiple kinds of hierarchical time-series data which follow
different aggregation functions with ranging levels of consistency. Our method
out-performs strong state-of-art baselines. We also show that by regularizing
the all forecast distributions together for coherency, PROFHiT provides robust
predictions that handle missing data values.

##  2 Problem Statement

Consider the dataset of $N$ time-series over the time horizon $1,2,\dots,T$.
We denote the time-series $i$ as $\mathbf{y}_{i}\in\mathrm{R}^{T}$ and the
value at time $t$ of time-series $i$ as $y_{i}^{(t)}$. The $N$ time-series
have a hierarchical relationship. We denote the hierarchy
$\mathcal{T}=(G_{\mathcal{T}},H_{\mathcal{T}})$ as a tree $G_{\mathcal{T}}$ of
$N$ nodes rooted at time-series $1$. For a non-leaf node (time-series) $i$, we
denote its children as $\mathcal{C}_{i}$. The parent and children nodes are
related via set of aggregation relations $H_{\mathcal{T}}$ of form
$H_{\mathcal{T}}=\\{\mathbf{y}_{i}=\sum_{j\in\mathcal{C}_{i}}\phi_{ij}\mathbf{y}_{j}:\forall
i\in\\{1,2,\dots,N\\}\\}$ where values of $\phi_{ij}$ are known and time-
independent constants,

Let current time-step be $t$. For any $1\leq t_{1}<t_{2}\leq t$, we denote
$\mathbf{y}_{i}^{(t_{1}:t_{2})}=\\{y_{i}^{(t_{1})},y_{i}^{(t_{1}+1)},\dots,y_{i}^{(t_{2})}\\}$.
Then the Probabilistic Hierarchical Time-series Forecasting problem is defined
as: Given the data
$\mathcal{D}_{t}=[\mathbf{y}_{1}^{1:t},\mathbf{y}_{2}^{1:t},\dots,\mathbf{y}_{N}^{1:t}]$
and hierarchical information $\mathcal{T}$, our goal is to predict accurate
and well-calibrated probability distributions
$\\{p(y_{1}^{(t+\tau)}|\mathcal{D}_{t}),p(y_{2}^{(t+\tau)}|\mathcal{D}_{t})\dots
p(y_{N}^{(t+\tau)}|\mathcal{D}_{t})\\}$ of forecasts at time $t+\tau$ for all
time-series.

##  3 Methodology

Figure 2: Overview of pipeline of PROFHiT. The input timeseries is ingested by
TSFNP, a Neural Gaussian Process based probabilistic forecasting model to
output the Raw Forecast Distribution. The parameters of Raw forecast
Distribution is refined by Refinement module leveraging predictions from all
time-series. The training of PROFHiT is driven by Likelihood loss that to
learn from ground truth prediction as well as the novel Distributional
Coherency Loss that regularizes the refined forecast distribution to follow
the hierarchical relations of the time-series.

#### Overview

PROFHiT models the forecast distributions
$\\{P(y_{i}^{(t+\tau)}|\mathcal{D}_{t})\\}_{i=1}^{N}$ of all time-series nodes
of the hierarchy by leveraging the relations from the hierarchy to provide
well-calibrated robust forecasts that are adaptable to noise and variation of
strictness to hierarchical relations. Most existing works do not attempt to
model entire probabilistic distribution but focus on coherency of point
forecasts [9, 24] or samples or fixed quantiles of the distribution [21, 8].
This approach does not capture uncertainty of the forecasts and in turn does
not provide calibrated and robust predictions. Moreover, most methods operate
on datasets where the datasets are strongly consistent over hierarchical
relations. However, many real-world datasets are weakly consistent with time-
series values of all nodes of hierarchy observed simultaneously and may not
follow the hierarchical relations strictly due to noise and discrepancies in
collecting data at different levels. We found that most previous works on HTSF
may not adapt well to such deviations from these constraints of ground truth.

PROFHiT, on the other hand, reconciles the need to model coherency between
entire forecast distributions as well as induce a soft adaptable constraint to
enforce coherency via a two-step stochastic process that is trained in an end-
to-end manner. PROFHiT first produces a raw forecast distribution for each
node parameterized by $\\{(\hat{\mu}_{i},\hat{\sigma}_{i})\\}_{i=1}^{N}$ by
using past values of time-series via a time-series forecasting model adapted
from EpiFNP [12] called TSFNP. Raw forecasts of all nodes are used as priors
to derive a refined set of forecast distributions via the refinement module.
The full probabilistic process of PROFHiT is depicted in Figure 2 and formally
summarized as:

$\begin{split}P(\\{y_{i}^{(t+\tau)}\\}_{i=1}^{N}|\mathcal{D}_{t})&=\int\underbrace{P(\mathbf{z}|\\{\mathbf{y_{i}^{(1:t)}}\\}_{i=1}^{N})\left(\prod_{i=1}^{N}P(\mathbf{z_{i},u_{i}}|\\{\mathbf{y_{i}^{(1:t)}}\\}_{i=1}^{N})P(\hat{\mu}_{i},\hat{\sigma}_{i}|\mathbf{z}_{i},\mathbf{z},\mathbf{u}_{i})\right)}_{\text{{TSFNP}
(Raw forecasts)}}\\\
&\underbrace{\prod_{i=1}^{N}P(\mu_{i},\sigma_{i}|\\{\hat{\mu}_{j},\hat{\sigma}_{j}\\}_{j=1}^{N})P(y_{i}^{(t+\tau)}|\mu_{i},\sigma_{i})}_{\text{Refinement
Module}}d\\{\mathbf{u_{i}}\\}_{i=1}^{N}d\\{\mathbf{z_{i}}\\}_{i=1}^{N}.\end{split}$
(1)

PROFHiT regularizes the refined distributional parameters
$\\{(\mu_{i},\sigma_{i})\\}_{i=1}^{N}$ towards coherency by using a soft
regularizer of form $\text{Dist}(P(y_{i}|\mathcal{D}_{t}),P(\sum_{j\in
C_{i}}\phi_{ij}y_{j}|\mathcal{D}_{t}))$, where Dist is an efficiently
computable distributional distance metric. Optimizing for distributional
coherency enforces forecast distributions to be well-calibrated while
adaptively adhering to hierarchical relations of the dataset.

###  3.1 Raw forecast distributions from NGPs

NGPs (Neural gaussian Process) [11, 12, 18] are novel class of probabilistic
neural models for accurate and calibrated predictions. NGP models such as
EpiFNP [12] and its variants [11] have shown state-of-art performance in
calibrated probabilistic forecasting across multiple benchmarks for single
time-series. We, therefore, use a modified form of EpiFNP called TSFNP to
derive raw forecast distributions for each time-series node.

We briefly describe TSFNP’s three components: 1) Probabilistic Neural Encoder
(PNE): PNE encodes the input univariate time-series into a latent stochastic
embedding via a GRU [4] followed by a self-attention layer [25]:

$\begin{split}\mathbf{\mu(u)}_{i},\mathbf{\sigma(u)_{i}}=\text{GRU}(\mathbf{y_{i}^{(t^{\prime}:t)}}),\quad\mathbf{u}_{i}\sim\mathcal{N}(\mathbf{\mu(u)}_{i},\mathbf{\sigma(u)_{i}}).\end{split}$
(2)

2) Stochastic Data Correlation Graph (SDCG): SDCG leverages similar patterns
of past time-series data and aggregates them as local latent variable. Unlike
EpiFNP which uses past time-series information from same node, in our multi-
variate case TSFNP uses past information from all nodes’ past time-series.
Formally, for input sequence $\mathbf{y_{i}^{(t^{\prime}:t)}}$ and each of the
past sequence $\mathbf{y_{j}}$ where $j\in\\{1,\dots,N\\}$, we sample
$\mathbf{y_{j}}$ with probability
$\exp(-\gamma||\mathbf{u_{i}}-\mathbf{u_{j}}||_{2}^{2})$ into set $N_{i}$.
Then, we derive the local latent variable as

$\mathbf{z_{i}}\sim\mathcal{N}\left(\sum_{j\in
N_{i}}NN_{1}(\mathbf{u_{j}}),\exp(\sum_{j\in
N_{i}}NN_{2}(\mathbf{u_{j}}))\right)$ (3)

where $NN_{1}$ and $NN_{2}$ are feed-forward networks.

3) Predictive Distribution Decoder (PDD): PDD combines the latent embedding of
input time-series, local latent variable and combined information of all past
sequences to derive the parameters of the output distribution via a simple
feed-forward network. We first derive a global latent variable that combines
the information from latent embeddings of all past sequences via self-
attention:

$\begin{split}\\{\beta_{i}\\}_{i=1}^{N}&=\text{Self-
Atten}(\\{\mathbf{u}_{i}\\}_{i=1}^{N}),\quad\mathbf{z}=\sum_{i=1}^{N}\beta_{i}\mathbf{u_{i}}\end{split}$
(4)

Finally, we combine the latent embedding of input time-series, local latent
variable and global latent variable to derive the raw forecast distribution
modelled as a Gaussian $\mathcal{N}(\hat{\mu}_{i},\hat{\sigma}_{i})$ as:

$\begin{split}\mathbf{e}=\text{concat}(\mathbf{u_{i},z_{i},z}),\quad\hat{\mu}_{i},\hat{\sigma}_{i}=NN_{3}(\mathbf{e})\end{split}$
(5)

where $NN_{3}$ is a feed forward network.

###  3.2 Refinement module

The refinement module leverages the raw distributions of all nodes of
hierarchy to produce refined forecast distributions using hierarchical
relations. Given the parameters of raw forecast distributions
$\\{\hat{\mu}_{i},\hat{\sigma}_{i}\\}_{i=1}^{N}$ derived from TSFNP for all
time-series $\\{\mathbf{y_{i}^{(t^{\prime}:t)}}\\}_{i=1}^{N}$, the refinement
module derives the refined forecast distributions denoted by parameters
$\\{\mu_{i},\sigma_{i}\\}_{i=1}^{N}$ as functions of parameters of raw
forecasts of all time-series. The refined forecasts are optimized to be more
coherent due to the regularization through distributional coherency loss.
Since we drive the full distributions of refined forecasts to be coherent,
rather than just the samples or mean statistics, the refined distributions’
calibration is also consistent with the hierarchical relations.

Let $\hat{\mathbf{\mu}}=[\hat{\mu}_{1}\dots,\hat{\mu}_{N}]$ and
$\mathbf{\hat{\sigma}}=[\hat{\sigma}_{1}\dots,\hat{\sigma}_{N}]$ be vectors of
means and standard deviations of raw distributions. We model the refined mean
as a function of the raw means of all the nodes. Formally, we derive the mean
$\mu_{i}$ of refined distribution as a weighted sum of two terms: a)
$\hat{\mu}_{i}$, the mean of raw time-series, and b) linear combination of all
raw mean of all time-series:

$\gamma_{i}=\text{sigmoid}(\hat{w}_{i}),\quad\mu_{i}=\gamma_{i}\hat{\mu_{i}}+(1-\gamma_{i})\mathbf{w_{i}}^{T}\hat{\mu}.$
(6)

$\\{\hat{w}_{i}\\}_{i=1}^{N}$ and $\\{\mathbf{w}_{i}\\}_{i=1:N}$ are both
learnable set of parameters of the model. $\text{sigmoid}()$ denotes the
sigmoid function. $\gamma_{i}$ models the tradeoff between the influence of
the raw distribution of node $i$ and the influence of the other nodes of the
hierarchy. This is useful for the model to automatically adapt to datasets
with varying hierarchical consistency.

Similarly, we assume the variance of the refined distribution depends on the
raw mean and variance of all the time-series. The variance parameter
$\sigma_{i}$ of the refined distribution is derived from the raw distribution
parameters $\mathbf{\hat{\mu}}$ and $\mathbf{\hat{\sigma}}$ as

$\sigma_{i}=c\hat{\sigma_{i}}\text{sigmoid}(\mathbf{v}_{1i}^{T}\mathbf{\hat{\mu}}+\mathbf{v}_{2i}^{T}\mathbf{\hat{\sigma}}+b_{i})$
(7)

where $\\{\mathbf{v}_{1i}\\}_{i=1}^{N}$, $\\{\mathbf{v}_{2i}\\}_{i=1}^{N}$ and
$\\{b_{i}\\}_{i=1}^{N}$ are parameters and $c$ is a positive constant
hyperparameter.

###  3.3 Likelihood and Coherency Loss

We optimize the probabilistic process of Equation 1 for accuracy, calibration
and coherency.

#### Likelihood Loss

To maximise the likelihood
$P(\\{y_{i}^{(t+\tau)}\\}_{i=1}^{N}|\mathcal{D}_{t})$ we use variational
inference similar to [12] by approximating the posterior
$\prod_{i=1}^{N}P(\mathbf{z_{i}},\mathbf{u_{i}}^{(j)}|\mathcal{D}_{t})$ with
the variational distribution
$\prod_{i=1}^{N}P(\mathbf{u_{i}}|\mathbf{y_{i}^{(t^{\prime}:t)}})q_{i}(\mathbf{u_{i}}|\mathbf{y_{i}^{(t^{\prime}:t)}})$
where $q_{i}$ is a feed-forward network over GRU hidden embeddings of PNE that
parameterizes the Gaussian distribution of
$q_{i}(\mathbf{u_{i}}|\mathbf{y_{i}^{(t^{\prime}:t)}})$. We derive the ELBO as

$\begin{split}\mathcal{L}_{1}=-E_{\prod_{i}q_{i}(\mathbf{z_{i}},\mathbf{u_{i}}|\mathcal{D}_{t})}[\log
P(\\{y_{i}^{(t+\tau)}\\}_{i=1}^{N}|\\{\mathbf{u_{i}},\mathbf{z_{i}}\\}_{i=1}^{N},\mathbf{z})&+\sum_{i=1}^{N}\log
P(\mathbf{z_{i}}|\mathbf{u_{i}},\\{\mathbf{u^{\prime}_{j}}_{j=1}^{N}\\})-\log
q_{i}(\mathbf{u_{i}}|\mathbf{y_{i}^{(t^{\prime}:t)}})].\end{split}$ (8)

#### Distributional Coherency Loss

PROFHiT leverages the hierarchy relations in $\mathcal{T}$ and regularizes the
refined distributions to be coherent. Since PROFHiT aims to leverage
hierarchical coherency for improved robustness and calibration, we regularize
over the full distributions by introducing distributional coherency loss.
Distributional Coherency loss uses a distance metric between the distribution
of a node and the distribution of aggregated random variables of its children
nodes in the hierarchy. We use the Jensen-Shannon Divergence [6] (JSD) as the
distance metric since it is a symmetric and bounded measure. We derive the
distributional coherency loss as

$\begin{split}&\mathcal{L}_{2}=2\left[\sum_{i=1}^{N}\text{JSD}\left(P(y_{i}^{(t+\tau)}|\hat{\mu}_{i},\hat{\sigma}_{i}),P\left(\sum_{j\in\mathbf{C}_{i}}\phi_{ij}y_{j}^{(t+\tau)}|\\{|\hat{\mu}_{j},\hat{\sigma}_{j}\\}_{j\in\mathbf{C}_{i}}\right)\right)+1\right]\end{split}$
(9)

Computation of JSD is generally intractable. However, in our case, due to
parameterization of each time-series distribution as a Gaussian we get a
closed-form differentiable expression:

######  Theorem 1.

The Distributional Coherency Loss (Equation 9) can be exactly expressed as:

$\begin{split}\mathcal{L}_{2}=\sum_{i=1}^{N}\frac{\sigma_{i}^{2}+\left(\mu_{i}-\sum_{j\in
C_{i}}\phi_{ij}\mu_{j}\right)^{2}}{2\sum_{j\in
C_{i}}\phi^{2}_{ij}\sigma_{j}^{2}}+\sum_{i=1}^{N}\frac{\sum_{j\in
C_{i}}\phi^{2}_{ij}\sigma_{j}^{2}+\left(\mu_{i}-\sum_{j\in
C_{i}}\phi_{ij}\mu_{j}\right)^{2}}{2\sigma_{i}^{2}}.\end{split}$ (10)

We provide the proof in the Appendix. The distributional coherency loss
regularizes over the full refined forecast distributions to follow the
constraints of hierarchy $\mathcal{T}$. Thus, the total loss is given as
$\mathcal{L}=\mathcal{L}_{1}+\lambda\mathcal{L}_{2}$ where the hyperparameter
$\lambda$ control the tradeoff between data likelihood and coherency. Tuning
$\lambda$ also allows PROFHiT to adapt to datasets of different levels of
adherence to aggregation function of proposed hierarchy (unlike [21] that
assumes strict adherence). We use the reparameterization trick to make the
sampling process differentiable and we learn the parameters of all training
modules via Stochastic Variational Bayes [13]. The full pipeline of PROFHiT is
summarized in Figure 2.

###  3.4 Details on Training

We provide additional important details for effective training of PROFHiT
here.

Parameter sharing across nodes Since PROFHiT’s TSFNP module models multiple
nodes, we leverage the hard-parameter sharing paradigm of multi-task learning
[2] and use different set of parameters for Predictive Distribution Decoder
(i.e., weights of $NN3$ for each time-series $i$ is different) whereas the
parameters of other components of TSFNP are shared across all nodes (Figure
2). Sharing parameters for PNE drastically lowers the number of learnable
parameters since datasets can have large number of nodes (up to 512 nodes in
our experiments).

Pre-training on individual time-series Before we start training for refined
forecasts, we pre-train the parameters of TSFNP on given training dataset to
model raw forecast distribution accurately. We train using only a log
likelihood loss to learn parameters
$\\{\hat{\mu}_{i},\hat{\sigma}_{i}\\}_{i=1}^{N}$ similar to Equation 8.

##  4 Experiments

In this section, we evaluate PROFHiT over multiple datasets and compare it
with state-of-art hierarchical forecasting and domain-specific baselines. We
evaluated all models on a system with Intel 64 core Xeon Processor with 128 GB
memory and Nvidia Tesla V100 GPU with 32 GB VRAM. We provide the link to code
and datasets in Appendix.

###  4.1 Setup

#### Baselines:

We compare against the following hierarchical forecasting baselines: We use
(1) TSFNP [12] and (2) DeepAR [23] as state-of-art deep probabilistic
forecasting models which do not exploit hierarchy relations. (3) MinT [26] and
(4) ERM [24] are methods that convert incoherent forecasts as post-processing
step by framing it as an optimization problem. Since TSFNP provided better
evaluation scores compared to DeepAR, we performed ERM and MinT on TSFNP’s
mean forecasts only. (5) HierE2E [21] is a recent state-of-art deep-learning
based approach that projects the raw predictions onto a space of coherent
forecasts and trains the model in an end-to-end manner. (6) SHARQ [8] is
another state-of-art deep learning based approach that reconciles forecast
distributions by using quantile regressions and making the quantile values
coherent. Since MinT and ERM produce point-forecasts, we train them 10 times
independently and create forecast distribution from independent forecasts
(similar to [16]). Note that we fine-tune the hyperparameters of PROFHiT and
each baseline specific to each benchmark. More details on hyperparameters are
in Appendix.

We also evaluate the efficacy and contribution of our various modeling choices
by performing an ablation study using the following variants of PROFHiT: (7)
P-Global: We study the effect of our multi-tasking hard-parameter sharing
approach (Section 3.4) by training a variant where all the parameters,
including PDD are shared across all the nodes. (8) P-FineTune: We also look at
the efficacy of our soft regularization using both losses that adapts to
optimize for both coherency and training accuracy by comparing it with a
variant where the PDD parameters are further fine-tuned for individual nodes
using only the likelihood loss. (9) P-DeepAR: We evaluate our choice of using
TSFNP, a previous state-of-art univariate forecasting model for accurate and
calibrated forecasts with DeepVAR [23], another popular probabilistic
forecasting model that was also used in HierE2E baseline [21]. (10)
P-NoCoherent: This variant is trained by completely removing the
distributional coherency loss from the training. Note that unlike P-FineTune
which was initially trained with coherency loss before fine-tuning,
P-NoCoherent never uses the coherency loss at any point of training routine.
Therefore P-NoCoherent measures the importance of explicitly regularizing over
the information from the hierarchy.

#### Datasets:

We evaluate our method and baselines on a diverse set of publicly available
datasets (Table 2) from different domains with varied hierarchical relations
and consistency. The bechmarking dataset and evaluation setup including
forecast horizon is replicated from recent and past literature related to
general HTSF [21, 24, 26] as well as epidemic forecasting [22]. (1) Labour
dataset contains monthly employment data from Feb 1978 to Dec 2020 collected
from Australian Bureau of Statistics. (2) Tourism-L [26] contains tourism
flows in different regions in Australia group via region and demographic. This
dataset contains two sets of hierarchy (with four and five levels), one for
the mode of travel and the other for geography with the top node being the
only common node of both hierarchies. (3) Wiki dataset collects the number of
daily views of 145000 Wikipedia articles aggregated into 150 groups [24].
These 150 groups are leaf nodes of a four-level hierarchy with groups of
similar topics aggregated together. (4) Flu-Symptoms contains flu incidence
values called weighted influenza-like incidence (wILI) values [22] at multiple
spatial scales for USA for period of 2004-2020. The scales used are states,
HHS and National level (US states are grouped into 10 HHS regions by CDC). (5)
FB-Survey provides aggregated and anonymized daily indicator for the
prevalence of Covid-19 symptoms based on online surveys conducted on Facebook
[5] collected from Dec 2020 to Aug 2021 for each state and national level. We
use the state-level values to find aggregates at HHS levels.

Table 2: Dataset Characteristics

Dataset | No. of Nodes |  | Levels of
---
Hierarchy
| $\tau$
---
| Obs.
---
per node
Tourism-L | 555 | 4,5 | 12 | 228
Labour | 57 | 4 | 8 | 514
Wiki | 207 | 5 | 1 | 366
Flu-Symptoms | 61 | 3 | 4 | 544
FB-Survey | 61 | 3 | 4 | 257

#### Evaluation Metrics

We evaluate our model and baselines using carefully chosen metrics that are
widely used in literature [21, 8, 22].

1\. Cumulative Ranked Probability Score (CRPS) is a widely used standard
metric for evaluation of probabilistic forecasts that measures both accuracy
and calibration. Given ground truth $y$ and the predicted probability
distribution $\hat{p}(Y)$, let $\hat{F}_{y}$ be the CDF. Then, CRPS is defined
as:
$CRPS(\hat{F}_{y},y)=\int_{-\infty}^{\infty}(\hat{F}_{y}(\hat{y})-\mathbf{1}\\{\hat{y}>y\\})^{2}d\hat{y}.$
We approximate $\hat{F}_{y}$ as a Gaussian distribution formed from samples of
model to derive CRPS.

2\. Interval Score (IS) is a standard score used in evaluation of accuracy of
probabilistic forecasts in epidemiology [22]. IS measures the negative log
likelihood of a fixed size interval around the ground truth under the
predictive distribution:
$IS(\hat{p}_{y},y)=-\int_{y-L}^{y+L}\log\hat{p}_{y}(\hat{y})d\hat{y}.$

###  4.2 Results

We comprehensively understand the efficacy of PROFHiT by evaluating it through
the following questions: Q1: Does PROFHiT predict accurate calibrated
forecasts across all benchmarks? Q2: Does PROFHiT provide consistently better
performance across all levels of the hierarchy? Q3: Does PROFHiT provide
robust accurate forecasts if input sequences have missing data? Q4: Does
PROFHiT adapt to different strictness of hierarchical consistency in datasets?

#### Accuracy and Calibration performance (Q1)

Table 3: Average CRPS and IS (accros 5 runs) across all levels for all
baselines, PROFHiT and its variants. PROFHiT performs significantly better
than all baselines (t-test with $\alpha=1\%$) with 41% - 88% better accuracy
and Calibration.

| Models/Data | Tourism-L | Labour | Wiki | Flu-Symptoms | FB-Survey
---|---|---|---|---|---|---
|  | CRPS | LS | CRPS | LS | CRPS | LS | CRPS | LS | CRPS | LS
Baselines | DeepAR | 0.17 | 0.61 | 0.045 | 0.75 | 0.232 | 0.83 | 0.610 | 3.25 | 7.32 | 5.32
| TSFNP | 0.21 | 1.19 | 0.071 | 1.41 | 0.287 | 0.86 | 0.460 | 0.93 | 5.53 | 7.84
| MinT | 0.5 | 0.58 | 0.045 | 4.12 | 0.243 | 0.78 | 0.630 | 3.18 | 5.39 | 6.35
| ERM | 0.56 | 0.53 | 0.045 | 3.63 | 0.221 | 0.74 | 0.620 | 2.75 | 6.14 | 4.23
| HierE2E | 0.15 | 0.38 | 0.034 | 0.51 | 0.211 | 0.46 | 0.420 | 0.81 | 4.12 | 1.13
| SHARQ | 0.17 | 0.41 | 0.054 | 0.47 | 0.241 | 0.52 | 0.470 | 1.42 | 3.12 | 0.81
|  PROFHiT (Ours) | 0.12 | 0.33 | 0.026 | 0.21 | 0.184 | 0.35 | 0.250 | 0.28 | 1.43 | 0.45
Ablation | P-FineTune | 0.16 | 0.39 | 0.031 | 0.36 | 0.216 | 0.39 | 0.240 | 0.3 | 1.18 | 0.72
| P-Global | 0.13 | 0.42 | 0.027 | 0.31 | 0.185 | 0.34 | 0.350 | 0.47 | 2.64 | 1.39
| P-DeepAR | 0.13 | 0.52 | 0.029 | 0.58 | 0.201 | 0.48 | 0.361 | 0.54 | 2.13 | 0.75
| P-NoCoherent | 0.18 | 0.37 | 0.043 | 0.45 | 0.227 | 0.47 | 0.248 | 0.36 | 1.17 | 0.84

We evaluate all baselines, PROFHiT and its variants for the datasets described
in Section 4 over 5 independent runs. We report the average CRPS and IS scores
across all levels hierarchy in Table 3 and standard error in Appendix. We
observe that PROFHiT significantly outperforms all baselines in CRPS score by
41%-88% and IS by 14%-550%. Thus, PROFHiT adapts well to varied kinds of
datasets and outperforms all baselines in both accuracy and calibration.
Performing t-test with significance $\alpha=1\%$ we find that all the scores
are statistically significant compared to baselines. On comparing the
performance of PROFHiT with the variants, we observe that PROFHiT is
comparable to or better than the best-performing variant in each dataset. This
shows that all the important model design choices (multi-task parameter
sharing, distributional coherency, and joint training on both losses) of
PROFHiT are important for its consistently superior performance.

#### Performance across the hierarchy (Q2)

Next, we look at the performance of all models across each level of hierarchy.
We compared the performance of PROFHiT with best performing baselines HierE2E
and SHARQ for all datasets. PROFHiT significantly outperforms the best
baselines. At the leaf nodes, which contains most data, PROFHiT outperforms
best baselines by 7% in Wiki to 100% in FB-Survey. For the top node of time-
series the performance improvement is largest at 35% (Wiki) to 962% (FB-
Survey). We show detailed results in Appendix. PROFHiT also performs better
than the variants in most higher levels of hierarchy and its performance is
comparable to the best variant (P-FineTune and P-Global) at leaf nodes as
well. P-NoCoherent performs most poorly compared to all variant and PROFHiT
proving that Distributional Coherency Loss is a very important contributor for
consistent performance across the hierarchy in all datasets.

#### Robust forecasting through data imputation (Q3)

(a) Tourism-L

(b) Labour

(c) Wiki

(d) Flu-Symptoms

(e) FB-Survey

Figure 3: % increase in CRPS for PROFHiT and baselines (lower the better) with
increase in proportion of missing data.

(a) Tourism-L

(b) Labour

(c) Wiki

(d) Flu-Symptoms

(e) FB-Survey

Figure 4: % increase in CRPS (lower the better) for PROFHiT and its variants
with increase in proportion of missing data.

During real-time forecasting in real-world applications such as Epidemic or
Sales forecasting, we encounter situations where the past few values of time-
series are missing or unreliable for some of the nodes. This is observed
specifically at lower levels, due to discrepancies or delays during reporting
and other factors [3]. Therefore, one approach to performing forecasting in
such a situation is first by imputation of missing values based on past data
and then using the predicted missing values as part of the input for
forecasting. Task: To simulate such scenarios of missing data and evaluate the
robustness of PROFHiT and all baselines, we design a task called Hierarchical
Forecasting with Missing Values (HFMV). Formally, at time-period $t$, we are
given full data for up to time $t-\rho$. We show results here for $\rho=5$
which is the average forecast horizon of all tasks. We also evaluate for other
values of $\rho$ in the Appendix with similar results. For sequence values in
time period between $t-\rho$ and $t$, we randomly remove $k\%$ of these values
across all time-series. The goal of HFMV task is to use the given partial
dataset from $t-\rho$ to $t$ as input along with complete dataset for time-
period before $t-\rho$ to predict future values at $t+\tau$. Therefore,
success in HFMV implies that models are robust to missing data from recent
past by effectively leveraging hierarchical relations. Setup: We first train
PROFHiT and baselines on complete dataset till time $t^{\prime}$ and then fill
in the missing values of input sequence using the trained model. Using the
predicted missing values, we again forecast the output distribution. For each
baseline and PROFHiT, we perform multiple iterations of Monte-Carlo sampling
for missing values followed by forecasting future values to generate the
forecast distribution. We estimate the evaluation scores using sample
forecasts from all sampling iterations. Result: We measure relative decrease
in performance with increase in percentage of missing data $k$ (Figures 3, 4).
We observe that PROFHiT’s performance decrease with a larger fraction of
missing values is much slower compared to other baselines. Even at $k=10\%$,
PROFHiT’s performance decreases by 10.45-26.8% compared to other baselines
that consistently decrease by over 70%. Thus, PROFHiT effectively leverages
coherency to model missing values. We observe that P-NoCoherent’s performance
deteriorates very rapidly in most benchmarks, showing the importance of
distributional coherency loss for learning provides robust calibrated coherent
forecasts. The second worse-performing variant across all datasets is
P-FineTune which also relies less on the hierarchical relations due to fine-
tuning of parameters for specific time-series. Finally, we observe that
PROFHiT and P-Global suffer the least degradation in performance since both
these models prioritize integrating hierarchical coherency information which
enables them to provide better estimates for imputed data for missing input
and use them to generate more accurate and calibrated forecasts.

#### Flexibility to hierarchical consistency (Q4)

We evaluate PROFHiT and baselines over datasets with varying hierarchical
consistency. Datasets like Tourism-L, Labour and Wiki are constructed by
collecting values of leaf nodes and leveraging the hierarchical relations to
derive the values of time-series of other nodes of the hierarchy. Therefore,
these datasets are strongly consistent, i.e, they strictly follow the
hierarchical relations. For other real-world datasets like Flu-Symptoms and
FB-Survey, the values of all nodes of hierarchy are directly collected or
measured. For example, the values of Flu-Symptoms dataset are directly
collected from public health agencies at the state, HHS and national levels
and aggregated by CDC. Due to various factors like reporting discrepancies and
noise they contain values in time-series that may deviate from the given
hierarchical relations [3]. Therefore, Flu-Symptoms and FB-Survey are weakly
consistent.

Since most previous state-of-art models assume datasets to be strongly
consistent, deviations to this assumptions can cause under-performance when
used with weakly consistent datasets. This is evidenced in Table 3 where some
of the baselines like MinT and ERM that explicitly optimize for hierarchical
coherency perform worse than even TSFNP, which does not leverage hierarchical
relations, in Flu-Symptoms and FB-Survey. Overall, we found that for weakly
consistent datasets, PROFHiT provides a much larger 93% average improvement in
CRPS scores over best HTSF baselines compared to 54% average improvement for
strongly consistent datasets. These improvements are more pronounced at non-
leaf nodes of hierarchy where PROFHiT improves by 2.8 times for Flu-Symptoms
and 9.2 times for FB-Survey. This is because HTSF baselines which assume
strong consistency do not adapt to noise at leaf nodes that compound to errors
at higher levels of hierarchy. We observed that soft coherency regularization
and parameter sharing across nodes was vital for PROFHiT’s adaptability to
varying levels of consistency. Due to the design of refinement module the
tradeoff between learning from raw forecasts and from hierarchical relations
was accurately captured by PROFHiT via parameters ${\gamma_{i}}_{i=1}^{N}$. We
provide detained analysis of these observations in the Appendix.

##  5 Conclusion and Discussion

We described PROFHiT, a novel probabilistic hierarchical forecasting model
that produces accurate and well-calibrated forecasts that is adaptable to
datasets with varying levels of adherence to assumed hierarchical relations
and is robust to missing values in training sequences. We evaluated PROFHiT
against previous state-of-art hierarchical forecasting baselines over wide
variety of datasets and observed 41-88% improvement average improvement in
accuracy and calibration. PROFHiT provided best performance across the entire
hierarchy as well as significantly outperformed other models in providing
robust predictions when it encountered missing data where other baselines’
performance degraded by over 70%. Since PROFHiT can be applicable to many
real-world forecasting applications, there is a potential for limited misuse
of our model and is not immune to systemic biases and other safety issues
inherent in the datasets used.

Our work opens new possibilities like extending to various domains where time-
series values may not be continuous real numbers or can not be modelled as
Gaussian distributions. We can also explore modeling more complex structures
between time-series with different aggregation relations. PROFHiT can also be
used to study anomaly detection in time-series, especially in time-periods
where there are deviations from assumed coherency relations. Similar to [11],
we can extend our work to include multiple sources of features and modalities
of data both specific to each time-series and global to the entire hierarchy.

## References

  * [1] Bijaya Adhikari, Xinfeng Xu, Naren Ramakrishnan and B Aditya Prakash  “Epideep: Exploiting embeddings for epidemic forecasting”  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, 2019, pp. 577–586
  * [2] Rich Caruana  “Multitask learning”  In _Machine learning_ 28.1 Springer, 1997, pp. 41–75
  * [3] Prithwish Chakraborty, Bryan Lewis, Stephen Eubank, John S Brownstein, Madhav Marathe and Naren Ramakrishnan  “What to know before forecasting the flu”  In _PLOS Computational Biology_ 14.10 Public Library of Science San Francisco, CA USA, 2018, pp. e1005964
  * [4] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau and Yoshua Bengio  “On the properties of neural machine translation: Encoder-decoder approaches”  In _arXiv preprint arXiv:1409.1259_ , 2014
  * [5] CMU Delphi Research Group  “COVID-19 Trends and Impact Survey”, 2021  URL: https://cmu-delphi.github.io/delphi-epidata/api/covidcast-signals/fb-survey.html
  * [6] Dominik Maria Endres and Johannes E Schindelin  “A new metric for probability distributions”  In _IEEE Transactions on Information theory_ 49.7 IEEE, 2003, pp. 1858–1860
  * [7] Tim van Erven and Jairo Cugliari  “Game-theoretically optimal reconciliation of contemporaneous hierarchical time series forecasts”  In _Modeling and stochastic learning for forecasting in high dimensions_ Springer, 2015, pp. 297–317
  * [8] Xing Han, Sambarta Dasgupta and Joydeep Ghosh  “Simultaneously Reconciled Quantile Forecasting of Hierarchically Related Time Series”  In _International Conference on Artificial Intelligence and Statistics_ , 2021, pp. 190–198  PMLR
  * [9] Rob J Hyndman, Roman A Ahmed, George Athanasopoulos and Han Lin Shang  “Optimal combination forecasts for hierarchical time series”  In _Computational statistics & data analysis_ 55.9 Elsevier, 2011, pp. 2579–2589
  * [10] Rob J Hyndman and George Athanasopoulos  “Forecasting: principles and practice”  OTexts, 2018
  * [11] Harshavardhan Kamarthi, Lingkai Kong, Alexander Rodriguez, Chao Zhang and B Aditya Prakash  “CAMul: Calibrated and Accurate Multi-view Time-Series Forecasting”  In _ACM The Web Conference (WWW)_ , 2022
  * [12] Harshavardhan Kamarthi, Lingkai Kong, Alexander Rodriguez, Chao Zhang and B Aditya Prakash  “When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting”  In _Thirty-fifth Conference on Neural Information Processing Systems_ , 2021
  * [13] Diederik P Kingma and Max Welling  “Auto-encoding variational bayes”  In _arXiv preprint arXiv:1312.6114_ , 2013
  * [14] Rahul Krishnan, Uri Shalit and David Sontag  “Structured inference networks for nonlinear state space models”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 31.1, 2017
  * [15] Volodymyr Kuleshov, Nathan Fenner and Stefano Ermon  “Accurate uncertainties for deep learning using calibrated regression”  In _International conference on machine learning_ , 2018, pp. 2796–2804  PMLR
  * [16] Balaji Lakshminarayanan, Alexander Pritzel and Charles Blundell  “Simple and scalable predictive uncertainty estimation using deep ensembles”  In _Advances in neural information processing systems_ 30, 2017
  * [17] Longyuan Li, Junchi Yan, Xiaokang Yang and Yaohui Jin  “Learning interpretable deep state space model for probabilistic time series forecasting”  In _arXiv preprint arXiv:2102.00397_ , 2021
  * [18] Christos Louizos, Xiahan Shi, Klamer Schutte and Max Welling  “The functional neural process”  In _arXiv preprint arXiv:1906.08324_ , 2019
  * [19] Frank Nielsen  “On the Jensen–Shannon symmetrization of distances relying on abstract means”  In _Entropy_ 21.5 Multidisciplinary Digital Publishing Institute, 2019, pp. 485
  * [20] Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang and Tim Januschowski  “Deep state space models for time series forecasting”  In _Advances in neural information processing systems_ 31, 2018
  * [21] Syama Sundar Rangapuram, Lucien D Werner, Konstantinos Benidis, Pedro Mercado, Jan Gasthaus and Tim Januschowski  “End-to-End Learning of Coherent Probabilistic Forecasts for Hierarchical Time Series”  In _International Conference on Machine Learning_ , 2021, pp. 8832–8843  PMLR
  * [22] Nicholas G Reich, Logan C Brooks, Spencer J Fox, Sasikiran Kandula, Craig J McGowan, Evan Moore, Dave Osthus, Evan L Ray, Abhinav Tushar and Teresa K Yamana  “A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States”  In _Proceedings of the National Academy of Sciences_ 116.8 National Acad Sciences, 2019, pp. 3146–3154
  * [23] David Salinas, Valentin Flunkert, Jan Gasthaus and Tim Januschowski  “DeepAR: Probabilistic forecasting with autoregressive recurrent networks”  In _International Journal of Forecasting_ 36.3 Elsevier, 2020, pp. 1181–1191
  * [24] Souhaib Ben Taieb, James W Taylor and Rob J Hyndman  “Coherent probabilistic forecasts for hierarchical time series”  In _International Conference on Machine Learning_ , 2017, pp. 3348–3357  PMLR
  * [25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser and Illia Polosukhin  “Attention is all you need”  In _Advances in neural information processing systems_ , 2017, pp. 5998–6008
  * [26] Shanika L Wickramasuriya, George Athanasopoulos and Rob J Hyndman  “Optimal forecast reconciliation for hierarchical and grouped time series through trace minimization”  In _Journal of the American Statistical Association_ 114.526 Taylor & Francis, 2019, pp. 804–819

Appendix for PROFHiT: Probabilistic Robust Forecasting for Hierarchical Time-
series

##  Appendix A Additional Related Work

Probabilistic time-series forecasting Classical probabilistic time-series
forecasting methods include exponential smoothing and ARIMA [10]. They are
simple but focus on univariate time-series and model each time-series sequence
independently. Recently, deep learning based methods have been successfully
applied in this area. DeepVAR [23] trains an auto-regressive recurrent network
model on a large number of related time series to directly output the mean and
variance parameters of the forecast distribution. Other works are inspired
from the space-state models and explicitly model the transition and emission
components with deep learning modules such as deep Markov models [14] and deep
state space models [17, 20] Recently, EpiFNP [12] has achieved state-of-art
performance in epidemic forecasting. It learns the stochastic correlations
between input data and datapoints to model a flexible non-parametric
distribution for univariate sequences.

Hierarchical time-series forecasting Classical works on hierarchical time-
series forecasting used a two-step approach [9, 10] and focus on point
predictions. They first forecast for time-series only at a single level of the
hierarchy and then derive the forecasts for other nodes using the hierarchical
relations.

Recent methods like MinT and ERM are post-processing steps applied on the set
of forecasts at all levels of hierarchy. MinT [26] assumes that the base level
forecasts are uncorrelated and unbiased and solves an optimization problem to
minimize the variance of forecast errors of past predictions. The unbiased
assumption is relaxed in ERM [24]. Another line of works projects the raw
forecasts of all time-series into a subspace of coherent forecasts. [7] use an
iterative Game-theoretic approach of minimizing forecast error and projection
error. A more recent work [21] uses a deep-learning based end-to-end approach
to directly train on the projected forecasts. SHARQ [8] is another recent
probabilistic deep-learning based method that uses quantile regression and
regularizes for coherency at different quantiles of forecast distribution.
However, unlike our approach, these methods do not regularize for coherency
over the entire distribution (Distributional Coherency) but only over fixed
quantiles. They also are not designed for cases where the hierarchical
constraints are not always consistently followed.

##  Appendix B Code and Dataset

We provide code of our implementation of PROFHiT along with the datasets used
at https://github.com/AdityaLab/PROFHiT/. We will release the code and data
publicly after acceptance.

##  Appendix C Hyperparameters

###  C.1 Data Preprocessing

Most datasets used in our work assume the aggregation function to be simple
summation (i.e, $\phi_{ij}=1$ for all weights). We first normalize the values
of leaf time-series training data to have 0 mean and variance of 1. Since the
aggregation of values at higher levels of the hierarchy can lead to very large
values in time-series, we instead divide each non-leaf time-series by the
number of children. Then the weights of hierarchical relations become
$\phi_{ij}=\frac{1}{|C_{i}|}$ where $C_{i}$ is the set of all children nodes
of time-series $i$. For the remaining datasets (Flu-Symptoms, FB-Symptoms) the
time-series values are normalized by default and thus require no extra pre-
processing.

###  C.2 Model Architecture

The architecture of TSFNP used in PROFHiT is similar to that used in the
original implementation [12]. The GRU unit contains 60 hidden units and is bi-
directional. Thus the local latent variable is also of dimension 60. $NN_{1}$
and $NN_{2}$ are both 2-layered neural networks with the first layer shared
between both. Both layers have 60 hidden units. Finally, $NN_{3}$ is a three-
layer neural network with the input layer having $180$ units (for the
concatenated input of three 60 dimensional vectors) and the last two layers
having 60 hidden units. We found that the value of $c$ in Equation 7 is not
very sensitive and usually set it to 5.

###  C.3 Training and Evaluation

Given the training dataset $\mathcal{D}_{t}$ we extract training dataset for
each node as the set of prefix sequences
$\\{(\mathbf{y_{i}^{(t1:t2)}},y_{i}^{(t2+1)}):1\leq t1\leq t2<t-\tau\\}$ and
train the full model (TSFNP and refinement module). We tune the hyperparameter
using backtesting by validating on window $t-\tau$ to $t$. Finally we train
for entire training set with best hyperparameters.

For each benchmark, we used the validation set to mainly find the optimal
batch size and learning rate. We searched over batch-size of
$\\{10,50,100,200\\}$ and the optimal learning rate was usually around 0.001.
We used early stopping with the patience of 150 epochs to prevent overfitting.
For each independent run of a model, we initialized the random seeds from 0 to
5 for PyTorch and NumPy. We didn’t observe large variations due to randomness
for PROFHiT and all baselines.

##  Appendix D Proof of Theorem 1

_The Distributional Coherency Loss (Equation 9) can be exactly expressed as:_

$\begin{split}\mathcal{L}_{2}=\sum_{i=1}^{N}\frac{\sigma_{i}^{2}+\left(\mu_{i}-\sum_{j\in
C_{i}}\phi_{ij}\mu_{j}\right)^{2}}{2\sum_{j\in
C_{i}}\phi^{2}_{ij}\sigma_{j}^{2}}+\sum_{i=1}^{N}\frac{\sum_{j\in
C_{i}}\phi^{2}_{ij}\sigma_{j}^{2}+\left(\mu_{i}-\sum_{j\in
C_{i}}\phi_{ij}\mu_{j}\right)^{2}}{2\sigma_{i}^{2}}.\end{split}$ (11)

To prove Theorem 10, we use the following well-known result for JSD of two
Gaussian Distributions [19]:

######  Lemma 1.

Given two univariate Normal distributions
$P_{1}=\mathcal{N}_{1}(\mu_{1},\sigma_{1})$ and
$P_{2}=\mathcal{N}_{2}(\mu_{2},\sigma_{2})$, the JSD is

$\text{JSD}(P_{1},P_{2})=\frac{1}{2}\left[\frac{\sigma_{1}^{2}+(\mu_{1}-\mu_{2})^{2}}{2\sigma_{2}^{2}}+\frac{\sigma_{2}^{2}+(\mu_{1}-\mu_{2})^{2}}{2\sigma_{1}^{2}}-1\right]$
(12)

###### Proof of Theorem 10.

Consider each JSD term

$\text{JSD}\left(P(y_{i}^{t+\tau}|\hat{\mu}_{i},\hat{\sigma}_{i}),P\left(\sum_{j\in\mathbf{C}_{i}}\phi_{ij}y_{j}^{t+\tau}|\\{|\hat{\mu}_{j},\hat{\sigma}_{j}\\}_{j\in\mathbf{C}_{i}}\right)\right)+1$

of the summation in Equation 9. Note that

$P(y_{i}^{t+\tau}|\hat{\mu}_{i},\hat{\sigma}_{i})=\mathcal{N}(\mu_{i},\sigma_{i})$
(13)

and
$P(\sum_{j\in\mathbf{C}_{i}}\phi_{ij}y_{j}^{t+\tau}|\\{|\hat{\mu}_{j},\hat{\sigma}_{j}\\}_{j\in\mathbf{C}_{i}}))$
is weighted sum of Gaussian variables
$\\{\mathcal{N}(\mu_{j},\sigma_{j})\\}_{j\in C_{i}}$. Therefore,

$P\left(\sum_{j\in\mathbf{C}_{i}}\phi_{ij}y_{j}^{t+\tau}|\\{\hat{\mu}_{j},\hat{\sigma}_{j}\\}_{j\in\mathbf{C}_{i}}\right)=\mathcal{N}\left(\sum_{j\in
C_{i}}\phi_{ij}\mu_{j},\sqrt{\sum_{j\in
C_{i}}\phi^{2}_{ij}\sigma_{j}^{2}}\right).$ (14)

Using Lemma 12 along with Equations 13,14 we get the desired result in
Equation 11. ∎

##  Appendix E Performance across each level of hierarchy

We compared the performance of PROFHiT with best performing baselines HierE2E
and SHARQ for each level of hierarchy of all datasets. PROFHiT significantly
outperforms the best baselines as well as the variants. At the leaf nodes,
which contains most data, PROFHiT outperforms best baselines by 7% in Wiki to
100% in FB-Survey. For the top node of time-series the performance improvement
is largest at 35% (Wiki) to 962% (FB-Survey). We show detailed results in
Table 4

Table 4: Average CRPS scores at each level of hierarchy. PROFHiT significantly
outperforms best baselines across all benchmarks. Note that P-Finetune’s
performance decreases at higher levels of hierarchy compared to other variants
whereas P-Global’s performance is worse at lower levels.

Models/Data | Tourism-L | Labour
---|---|---
Hierarchy Levels | 1 | 2(Travel) | 3(Travel) | 4(Travel) | 5(Travel) | 2(Geo) | 3(Geo) | 4(Geo) | 1 | 2 | 3 | 4
HierE2E | 0.081 | 0.103 | 0.141 | 0.205 | 0.272 | 0.103 | 0.136 | 0.175 | 0.031 | 0.034 | 0.034 | 0.038
SHARQ | 0.093 | 0.131 | 0.163 | 0.218 | 0.295 | 0.131 | 0.138 | 0.152 | 0.097 | 0.124 | 0.133 | 0.149
PROFHiT (Ours) | 0.051 | 0.095 | 0.12 | 0.17 | 0.264 | 0.083 | 0.106 | 0.142 | 0.023 | 0.019 | 0.023 | 0.029
P-FineTune | 0.072 | 0.136 | 0.083 | 0.16 | 0.278 | 0.124 | 0.124 | 0.158 | 0.024 | 0.022 | 0.026 | 0.035
P-Global | 0.093 | 0.113 | 0.122 | 0.13 | 0.261 | 0.093 | 0.113 | 0.147 | 0.021 | 0.027 | 0.028 | 0.027
P-DeepAR | 0.075 | 0.097 | 0.136 | 0.183 | 0.281 | 0.095 | 0.122 | 0.159 | 0.025 | 0.027 | 0.031 | 0.033
P-NoCoherent | 0.086 | 0.142 | 0.107 | 0.18 | 0.265 | 0.132 | 0.138 | 0.147 | 0.027 | 0.031 | 0.029 | 0.026
Models/Data | Wiki | Flu-Symptoms | FB-Survey |
Hierarchy Levels | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 1 | 2 | 3 |
HierE2E | 0.042 | 0.105 | 0.229 | 0.272 | 0.372 | 0.272 | 0.421 | 0.458 | 4.14 | 4.04 | 4.13 |
SHARQ | 0.039 | 0.136 | 0.235 | 0.291 | 0.378 | 0.258 | 0.376 | 0.381 | 3.08 | 3.21 | 3.13 |
PROFHiT (Ours) | 0.031 | 0.074 | 0.133 | 0.216 | 0.252 | 0.216 | 0.133 | 0.338 | 0.32 | 0.43 | 1.89 |
P-FineTune | 0.034 | 0.086 | 0.153 | 0.232 | 0.275 | 0.222 | 0.175 | 0.293 | 0.43 | 0.65 | 1.83 |
P-Global | 0.048 | 0.103 | 0.187 | 0.265 | 0.186 | 0.269 | 0.213 | 0.376 | 0.37 | 0.37 | 2.11 |
P-DeepAR | 0.035 | 0.094 | 0.193 | 0.251 | 0.285 | 0.242 | 0.217 | 0.328 | 0.44 | 0.61 | 2.01 |
P-NoCoherent | 0.49 | 0.117 | 0.93 | 0.258 | 0.167 | 0.227 | 0.193 | 0.381 | 0.42 | 0.36 | 2.18 |

##  Appendix F Adapting to varying dataset consistency

######  Observation 1.

The average improvement in performance of PROFHiT over best HTSF baselines is
72% higher for weakly consistent datasets over its improvement for strongly
consistent datasets.

Since most previous state-of-art models assume datasets to be strongly
consistent, deviations to this assumptions can cause under-performance when
used with weakly consistent datasets. This is evidenced in Table 3 where some
of the baselines like MinT and ERM that explicitly optimize for hierarchical
coherency perform worse than even TSFNP, which does not leverage hierarchical
relations, in Flu-Symptoms and FB-Survey. Overall, we found that for weakly
consistent datasets, PROFHiT provides a much larger 93% average improvement in
CRPS scores over best HTSF baselines compared to 54% average improvement for
strongly consistent datasets. These improvements are more pronounced at non-
leaf nodes of hierarchy where PROFHiT improves by 2.8 times for Flu-Symptoms
and 9.2 times for FB-Survey. This is because HTSF baselines which assume
strong consistency do not adapt to noise at leaf nodes that compound to errors
at higher levels of hierarchy.

######  Observation 2.

PROFHiT’s approach to parameter sharing and soft coherency regularization
helps adapt to varying hierarchical consistency.

We observe that that best performing variant for strongly consistent datasets
in P-Global which is trained with both likelihood and Distributional Coherency
loss (Table 3). But its performance severely degrades for weakly consistent
datasets since sharing all model parameters across all time-series makes it
inflexible to model patterns and deviations specific to individual nodes. In
contrast, P-FineTune and P-NoCoherent performs the best among variants for
weakly consistent datasets since they train separate sets of decoder
parameters for each node. But they perform poorly for strongly consistent
datasets since they don’t leverage Distributional Coherency effectively.
PROFHiT combines the flexible parameter learning of P-FineTune and leverage
Distributional Coherency to jointly optimize the parameters like P-Global
providing comparable performance to best variants over all datasets.

Table 5: Average value of $\gamma_{i}$ for all datasets. Note that weakly
coherent datasets have higher $\gamma_{i}$ (depends mode on past data of same
time-series) where as strongly-coherent data have lower $\gamma_{i}$
(leverages the hierarchical relations).

Consistency | Dataset | Average value of $\gamma_{i}$
---|---|---
Strong | Tourism-L | 0.420 $\pm$ 0.096
| Labour | 0.348 $\pm$ 0.091
| Wiki | 0.313 $\pm$ 0.057
Weak | Symp | 0.759 $\pm$ 0.152
| Fbsymp | 0.789 $\pm$ 0.180

######  Observation 3.

PROFHiT’s Refinement module automatically learns to adapt to varying
hierarchical consistency.

The design choices of the refinement module help PROFHiT to adapt to datasets
of different levels of hierarchical consistency. Specifically, by optimizing
for values of $\\{\gamma_{i}\\}_{i=1}^{N}$ of Equation 6, PROFHiT aims to
learn a good trade-off between leveraging prior forecasts for a time-series
and hierarchical relations of forecasts from the entire hierarchy. We study
the learned values of $\\{\gamma_{i}\\}_{i=1}^{N}$ of Equation 6 used to
derive refined mean. Note that higher values of $\gamma_{i}$ indicate larger
dependence on raw forecasts of node and smaller dependence of forecasts of the
entire hierarchy. We plot the average values of $\gamma_{i}$ for each of the
datasets in Table 5. We observe that strongly consistent datasets have lower
values of $\gamma_{i}$ indicating that PROFHiT’s refinement module
automatically learns to strongly leverage the hierarchy for these datasets
compared to weakly coherent datasets.

Table 6: Std. dev of CRPS and IS (accros 5 runs) across all levels for all
baselines, PROFHiT and its variants. PROFHiT performs significantly better
than all baselines as noted using t-test with $\alpha=1\%$.

| Models/Data | Tourism-L | Labour | Wiki | Flu-Symptoms | FB-Survey
---|---|---|---|---|---|---
|  | CRPS | LS | CRPS | LS | CRPS | LS | CRPS | LS | CRPS | LS
Baselines | DeepAR | 0.011 | 0.040 | 0.004 | 0.038 | 0.002 | 0.044 | 0.018 | 0.098 | 0.482 | 0.434
| TSFNP | 0.006 | 0.021 | 0.003 | 0.018 | 0.015 | 0.069 | 0.019 | 0.004 | 0.251 | 0.217
| MinT | 0.005 | 0.019 | 0.002 | 0.121 | 0.018 | 0.006 | 0.014 | 0.111 | 0.468 | 0.213
| ERM | 0.044 | 0.005 | 0.002 | 0.110 | 0.016 | 0.069 | 0.018 | 0.133 | 0.148 | 0.209
| HierE2E | 0.001 | 0.038 | 0.003 | 0.049 | 0.019 | 0.018 | 0.005 | 0.051 | 0.325 | 0.109
| SHARQ | 0.000 | 0.011 | 0.001 | 0.046 | 0.017 | 0.007 | 0.002 | 0.116 | 0.133 | 0.048
|  PROFHiT (Ours) | 0.001 | 0.017 | 0.001 | 0.003 | 0.001 | 0.030 | 0.005 | 0.009 | 0.040 | 0.008
Ablation | P-FineTune | 0.016 | 0.031 | 0.003 | 0.003 | 0.016 | 0.014 | 0.001 | 0.006 | 0.090 | 0.004
| P-Global | 0.012 | 0.033 | 0.000 | 0.013 | 0.002 | 0.001 | 0.033 | 0.024 | 0.248 | 0.119
| P-DeepAR | 0.006 | 0.026 | 0.001 | 0.028 | 0.005 | 0.043 | 0.035 | 0.030 | 0.103 | 0.065
| P-NoCoherent | 0.005 | 0.012 | 0.001 | 0.009 | 0.015 | 0.043 | 0.012 | 0.025 | 0.110 | 0.053


# Monetizing Currency Pair Sentiments
through LLM Explainability

Lior Limonad Corresponding Author. Email<EMAIL_ADDRESS>Fabiana Fournier
Juan Manuel Vera Díaz  Inna Skarbovsky  Shlomit Gur  Raquel Lazcano  IBM
Research, Israel  Atos IT Solutions and Services Iberia, Madrid, Spain

###### Abstract

Large language models (LLMs) play a vital role in almost every domain in
today’s organizations. In the context of this work, we highlight the use of
LLMs for sentiment analysis (SA) and explainability. Specifically, we
contribute a novel technique to leverage LLMs as a post-hoc model-independent
tool for the explainability of SA. We applied our technique in the financial
domain for currency-pair price predictions using open news feed data merged
with market prices. Our application shows that the developed technique is not
only a viable alternative to using conventional eXplainable AI but can also be
fed back to enrich the input to the machine learning (ML) model to better
predict future currency-pair values. We envision our results could be
generalized to employing explainability as a conventional enrichment for ML
input for better ML predictions in general.

123

##  1 Introduction

Explainability is the foundation for the adoption and trust of humans in AI-
based systems. Explanations are the vehicle via which users can understand and
act upon the various situations that evolve during their interaction with the
system. Although some Machine Learning (ML) models are inherently explainable
(e.g., decision trees and linear models) and their internal logic predictions
are interpretable (i.e., can be easily understood by humans), more complex
models (e.g., ensemble models and deep learning models) require external
explanation frameworks, namely eXplainable AI (XAI), to be human-
understandable [11, 8]. Probably most prominent recently is the employment of
Generative AI111www.gartner.com/en/information-technology/glossary/generative-
ai, and particularly its applicability to textual artifacts in the form of
Large Language Models (LLMs). This presents a promising instrumentation to
enable out-of-the-box model-independent explanations that are easy to
interpret by humans.

Gartner’s report [5] highlights the widespread adoption of LLMs across
industries for uses including text summarization, question-answering, and
document translation. LLMs can also be augmented by additional capabilities to
create more powerful systems and feature a growing ecosystem of tools. Among
these, we foresee the benefit of leveraging LLMs not only as a means for the
automation of explanations in AI-based systems but also as an enrichment
mechanism that could be fed back as input for the AI itself to improve its
performance. Our results in this work pave the way to conducting a more
rigorous study to support our claim.

As one possible application of XAI in the financial sector, we have developed
an application that leverages ML for __ Sentiment Analysis (SA) on news feeds
about selected currency pairs. SA has become essential for grasping market
dynamics and forecasting trends in the financial sector. This technique’s
ability to gauge the overall mood of the market offers critical insights,
thereby facilitating more informed and strategic decision-making. For each, a
collection of related news articles is gathered over time. Our contribution is
in showing not only that the series of sentiments extracted from such feeds
can be leveraged to predict currency-pair rate changes but also to populate
information that can be used to enrich and thus improve the accuracy of such
predictions.

The overall high-level approach is depicted in Figure 1. Our ultimate goal is
to explore the potential to predict financial projections about currency-pair
rates based on the underlying sentiment sequences. Thus, given an input of a
series of daily currency-pair rates accompanied by corresponding news ads, as
a first step, we employ sentiment classification over these news ads to derive
their corresponding sentiment. Our idea is that accompanying currency-pair
prices with their associated sentiments could improve price predictions. As a
second step, we developed a novel algorithm that employs LLMs to extract the
set of terms explaining each sentiment classification. As a last step, we
blend these terms with the sentiments and the original price stream as an
enriched input for the rate prediction model.

Figure 1: Overall approach

##  2 Background

Recent advancements in the ML field have increased the complexity of ML
models, often at the expense of their interpretability, leading to “black box”
ML models and hindering their full adoption. Therefore, we have also seen an
increase in the need to explain ML models and their predictions, fuelled in
part by legislation, but also by incentives from the user’s or stakeholder’s
point of view (e.g., justify the ML model and gain domain insight), and from
the developer’s point of view (e.g., evaluate and improve the ML model).

__ eXplainable AI (XAI) is a research field that aims to make AI systems
results more understandable to humans [1]. Many XAI frameworks are
predominately developed for post-hoc (i.e., after model training)
interpretations of ML models. In contemporary XAI techniques (e.g., LIME [22]
or SHAP [16]), the ML model serves as a surrogate model typically trained
using historical data. The predicted value for a single instance serves as
input for the XAI explainer to produce an explanation.

Leveraging LLMs for the sake of explainability eliminates the burden of
training such explainability ML models. __ LLMs are deep-learning models
trained on vast amounts of text data to perform various natural language
processing tasks. One of their strengths is their ability to perform few-shot
and zero-shot learning with prompt-based learning [15]. With the emergence of
LLMs, the practice of ‘prompt engineering’ has been explored recently as a new
field of research that focuses on designing, refining, and implementing
different instructions to guide the output of LLMs in various tasks to achieve
a more concise and effective utilization of the LLMs, adapting it to the
corresponding skills of the users and the context of the different tasks such
as analysis or reasoning [19]. This may include different techniques such as
one- or few-shot samples, the use of quotes to annotate different parts of a
prompt, employing methodologies such as “Chain of Thought” (CoT), and
configuration of LLM-specific settings like temperature and top-p sampling to
control the variability of the output.

In this work, we employ SA for a concrete financial application of predicting
currency-pair prices. SA is the computational study of opinions, sentiments,
and emotions expressed in text [14]. As a field of Natural Language Processing
(NLP), a variety of techniques may be employed nowadays to enable SA, ranging
from basic rule-based methods to ML and deep learning-based approaches, that
typically involve model training based on corpora of textual narratives that
are labeled by sentiment data. In this work, we present an LLM-based scheme
for explaining SA that is agnostic to the SA technique employed.

##  3 LLM for SA explainability

We developed a technique for the use of an LLM for model-agnostic
explainability, given as an input a narrative and its sentiments as determined
by some SA model (e.g., VADER or BERT). Our technique identifies the set of
keywords or terms in the narrative that support (i.e., explain) the sentiment
according to the inference model.

Our approach was developed as a post-hoc technique to enable the
identification of the ‘K’ most relevant set of keywords or terms (i.e., the
k-sufficient set) in the input narrative that is sufficient to influence the
prediction of the ML model, regardless of which specific ML technique is used
for the prediction of the sentiment. Concretely, a k-sufficient set is a
subset of words from the original narrative that when provided as an input to
the ML model retains the same sentiment output as the sentiment that was
originally generated for the entire narrative. Thus, given some model $M$ and
an input narrative text $T\ =\ w_{1},...,w_{n}$, we can determine its
sentiment as $M(T)$. Respectively, a k-sufficient set in this case is a subset
of $k$ terms $s_{k}=w_{1},...,w_{k}\subseteq T$, where $\left|s_{k}\right|=k$,
conforming to:

  1. 1.

$M\left(s_{k}\right)=M\left(T\right)$, i.e., the subset of terms retains the
same sentiment.

  2. 2.

$\bigl{[}$optional$\bigr{]}$ There is no subset $s\prime=w_{1}...w_{z}\subset$
$s_{k}$ where $M(s\prime)=M(T)$

The set $s_{k}$ is deemed sufficient in the sense that its inclusion in text
$T$ guarantees the sentiment classification of $T$ equals the original
sentiment classification as deemed by the ML model (i.e., $M(T)$). We
acknowledge that there could be several sets in $T$ conforming to the above
conditions. Our current implementation is indifferent among such subsets.

Step 1 in our method accounts for explanation sufficiency corresponding to the
input request for K terms. That is, such a set contains all needed terms to
ensure it preserves consistency with the sentiment classification of the
original narrative. This does not assure that such a set contains the fewest
terms necessary to retain the original sentiment classification (i.e., a
minimal set). If such a requirement is also deemed necessary, we include step
two as an optional extension to fulfill such a requirement. Without concerns
of performance, a straightforward realization will need to add to the given
implementation also an exhaustive scan of all $\left|k-1\right|$ subsets
$s\prime$ to ensure that for any such subset $M\left(s\prime\right)\neq M(T)$.
Another possible realization may directly leverage on the power of the LLM to
identify such a minimal subset directly.

A more advanced approach could also exploit the computation of weights that
signify the relative importance of each word in affecting the result of the ML
model. In such a case, our algorithm should be enhanced with a selection of a
subset that relies on the value of such weights (e.g., the one with the
highest weight average). To elicit a k-sufficient set of keywords from a given
narrative, we follow a zero-shot prompting approach. The overall flow of our
approach to classification explanation is depicted in Figure 2.

Figure 2: Explanation of SA with k-sufficient terms

Algorithm 1 formally elaborates on the realization of the classification
explanation, given $M$ as the model for sentiment elicitation, $X$ as a
textual narrative, $L$ is the employed LLM, $K$ as the number of keywords, and
$maxAttempts$ as a threshold limit to the number of iteration attempts. An
implementation of the algorithm is available here:
https://github.com/IBM/SAX/AlFin.

Algorithm 1 LLMsentimentXplain($M$, $X$, $L$, $K$, $maxAttempts$)

1:$result\leftarrow\text{``no explanation''}$

2:$s\leftarrow M.\text{getSentiment}(X)$

3:$attempts\leftarrow 0$

4:$prompt\leftarrow\text{``given text:''}+X+\text{``what are the top''}+K+$

5:$\text{``terms that support its sentiment classification as:''}+s$

6:do

7: $attempts\leftarrow attempts+1$

8: $W\leftarrow L.\text{apply}(prompt)$

9: $s\prime\leftarrow M.\text{getSentiment}(W)$

10: if $s\neq s\prime$ then

11: $prompt\leftarrow prompt+\text{``the sentiment classification
of:''}+W+\text{``is''}+s\prime$

12: end if

13:while $(s\neq s\prime)\And(attempts<maxAttempts)$

14:if $s=s\prime$ then

15: $result\leftarrow W$

16:end if

17:return $result$

##  4 Enriched feeds to predict currency-pair prices

Building on prior work [10], our first envisioned hypothesis is that enriching
an input feed of price closing rates with sentiment information that
corresponds to a series of news ads about a given currency pair may promote
not only a better prediction of future trends (i.e., increase or decrease in
price) but also to the improved predictive accuracy of future closing prices.
By enrichment of the input, we relate to extending the input with additional
term embeddings as further elaborated below.

Going beyond the first hypothesis, we further envision the potential monetary
benefit to be exploited from explanatory information about sentiment
classification. Leveraging the technique presented in section 3 for
associating each sentiment with a set of k-sufficient terms for a given news
ad, our next step is to use this technique to enrich a series of news-ad-
derived sentiments with their corresponding highlighted keywords. We envision
that such enriched traces may serve as a valuable input for the prediction of
future currency indicators, correlating with price movements such as currency-
pair closure prices (i.e., as in [10]) and that complementing such sentiment
traces with the k-sufficient keywords (along each sentiment label) may help to
improve the accuracy of such predictions. With the technique presented, we
also foresee the viability of deriving such information automatically with the
use of recently developed LLMs.

Hence we hypothesize that:

  * •

H1: News-ad-derived sentiment information can be used to improve the accuracy
of predictions about currency-pair future rate behaviors.

  * •

H2: Sentiment feeds enriched with explanatory information will achieve better
predictive accuracy than non-enriched sentiment feeds.

To evaluate the hypotheses, we used an open sentiment labeled dataset from
[10]. This data consists of 2291 news ads about five different currency pairs
collected between January and May 2023. This dataset was combined with daily
currency-pair historical prices for the same period as summarized in Table 1,
which was downloaded from the Yahoo Finance website. Note that the USDJPY
price is about two orders of magnitude greater than all other currency pair
prices.

Table 1: Currency-pair closing prices during the 3 months period

Currency-pair | AUDUSD | EURCHF | EURUSD | GBPUSD | USDJPY
---|---|---|---|---|---
Max close price | 0.7145 | 1.0069 | 1.1068 | 1.2630 | 140.8710
Min close price | 0.6498 | 0.9684 | 1.0522 | 1.1827 | 128.0260
Average Close Price | 0.6773 | 0.9876 | 1.0806 | 1.2278 | 133.5069
Close Price STDev | 0.0150 | 0.0090 | 0.0141 | 0.0198 | 2.9260

Utilizing the dataset, a Long Short-Term Memory (LSTM) model [12] was
structured with a 32-unit layer followed by a neuron that regresses the
currency price. An LSTM model is a recurrent neural network architecture well-
suited for sequential data. The model was trained as a baseline to predict the
next day’s closing price from the past 5 days. To test the first hypothesis,
we extended this input with three sentiment-derived features, denoting the
percentage of news per day related to positive, neutral, and negative
sentiment.

To test the second hypothesis, we first employed three different LLMs for the
elicitation of k-sufficient keywords: IBM-granite-13b-chat, GPT-4.0, and
GPT-3.5-turbo, the former model provided by IBM
watsonx.ai222https://www.ibm.com/products/watsonx-ai/foundation-models and the
latter two by openAI333https://platform.openai.com/. The use of GPT-3.5-turbo
was also manually inspected to ensure the output was properly formed. Further
to the elicitation of the keywords, we computed the embeddings of these
keywords with an NNLM encoder [2] and averaged it each day. This average was
added as an additional input feature to the LSTM model. We repeated the
process for each LLM model type separately.

##  5 Results

Figure 3: A test set example: 14 consecutive days for testing trend
predictions. Red dots denote a drop and green dots denote a rise in currency-
pair price with respect to the previous day.)

To evaluate future trend predictions the data consisted of 74 prediction
points for each currency pair. Given days must be consecutive to maintain the
temporal correlation, data was utilized in two separate instances: initially
with a train-to-test split ratio of 44:30 days (i.e., $60\%/40\%$), and
subsequently with a train-to-test split ratio of 60:14 days (i.e.,
$80\%/20\%$). An example of the latter case consisting of a test set of 14
consecutive days is shown in Figure 3.

Similar to prior work, we assessed the quality of the predictions with an
accuracy metric that captures the proportion of correct predictions out of the
total number of predictions. Results for trend predictions are shown in Tables
2 and 3. The results in the tables are color-coded according to the range of
each column’s values, with higher accuracy represented by greener colors and
lower accuracy indicated by yellower colors.

Corresponding to H1, on average the inclusion of the sentiment information on
top of the underlying pricing information, as the input to the LSTM model did
not yield any improvement in trend prediction accuracy. However, excluding the
EURCHF currency pair, the embedding of the sentiment yielded either equal or
improved accuracy in trend predictions. With respect to H2, the inclusion of
the explanation information was mostly effective for the GPT4.0 model, with
all accuracy results except one (for USDJPY in Table 2) being either the same
or improved compared to the accuracy without the explanation information. For
both remaining models, Granite and GPT-3.5 Curated, the inclusion of
explanation information led to improved results on average, with both being
mostly dominated by GPT4.0.

Table 2: Predicting future trends for currency pairs (accuracy metric), 44
consecutive days per currency for training and 30 consecutive days for testing

|  | AUDUSD | EURUSD | USDJPY | EURCHF | GBPUSD | Average
---|---|---|---|---|---|---|---
LSTM | ACC | 66,67% | 60,00% | 63,33% | 76,67% | 56,67% | 64,67%
LSTM+Sentiments | ACC | 70,00% | 66,66% | 66,67% | 43,35% | 56,67% | 60,67%
LSTM+Sentiments+Explanations (granite) | ACC | 70,00% | 66,67% | 80,00% | 53,33% | 66,67% | 67,33%
LSTM+Sentiments+Explanations (gpt) | ACC | 70,00% | 66,67% | 63,33% | 76,67% | 80,00% | 71,33%
LSTM+Sentiments+Explanations (gpt cur) | ACC | 70,00% | 80,00% | 66,67% | 53,33% | 66,67% | 67,33%

Table 3: Predicting future trends for currency pairs (accuracy metric), 60
consecutive days per currency for training and 14 consecutive days for testing

|  | AUDUSD | EURUSD | USDJPY | EURCHF | GBPUSD | Average
---|---|---|---|---|---|---|---
LSTM | ACC | 71,43% | 64,29% | 64,29% | 71,43% | 57,14% | 65,72%
LSTM+Sentiments | ACC | 71,43% | 64,29% | 71,43% | 28,57% | 71,43% | 61,43%
LSTM+Sentiments+Explanations (granite) | ACC | 71,43% | 64,29% | 85,71% | 71,43% | 57,14% | 70,00%
LSTM+Sentiments+Explanations (gpt) | ACC | 71,43% | 78,57% | 71,43% | 71,43% | 71,43% | 72,86%
LSTM+Sentiments+Explanations (gpt cur) | ACC | 71,43% | 92,86% | 64,29% | 71,43% | 57,14% | 71,43%

For the prediction of currency-pair prices, data was randomly split,
60%-training, 6%-validation, and 34%-testing. Post-model training and testing,
we present predictive accuracy for all models according to three common
metrics: Mean Squared Error (MSE), Mean Absolute Error (MAE), and Mean
Absolute Percentage Error (MAPE). Results for all metrics are presented in
table 4 and shown separately for each metric, color-coded similar to the
above, with higher accuracy represented by greener colors and lower accuracy
indicated by redder colors.

With the partial exclusion of the results for USDJPY, our results corroborate
both hypotheses. We thus also show the overall average for each metric with
and without the inclusion of USDJPY. Our first hypothesis about the inclusion
of sentiment information holds across all currency pairs considering MSE,
while for MAE and MAPE the same holds with only the exclusion of USDJPY.
Similarly, our second hypothesis about the additional inclusion of sentiment-
explanatory information holds among the majority of currency pairs,
demonstrating the dominance of the two GPT LLMs over Granite. Here also, the
USDJPY currency-pair shows a deviation from this pattern, for which the
Granite model performs better. On average, we also see that the inclusion of
k-sufficient keywords (explanations) yields further improvement in accuracy
that goes beyond the enrichment of the input with only sentiment information.

Table 4: Next-day price prediction for all five currency pairs.

|  | AUDUSD | EURUSD | USDJPY | EURCHF | GBPUSD | Average |  | Average
---
(-USDJPY)
LSTM | MSE | 2,58E-04 | 3,57E-04 | 2,30E+00 | 3,93E-04 | 2,77E-04 | 4,60E-01 | 3,21E-04
LSTM+Sentiments | MSE | 1,22E-04 | 7,82E-05 | 2,23E+00 | 2,19E-04 | 2,30E-04 | 4,46E-01 | 1,62E-04
LSTM+Sentiments+Explanations (granite) | MSE | 4,20E-05 | 1,87E-04 | 1,23E+00 | 9,33E-06 | 3,59E-04 | 2,45E-01 | 1,49E-04
LSTM+Sentiments+Explanations (gpt4.0) | MSE | 1,06E-04 | 5,09E-05 | 4,20E+00 | 3,86E-05 | 5,68E-05 | 8,40E-01 | 6,31E-05
LSTM+Sentiments+Explanations (gpt3.5) | MSE | 1,09E-04 | 5,99E-05 | 7,54E+00 | 1,18E-05 | 9,48E-05 | 1,51E+00 | 6,88E-05
LSTM | MAE | 1,47E-02 | 1,74E-02 | 1,08E+00 | 1,95E-02 | 1,48E-02 | 2,29E-01 | 1,66E-02
LSTM+Sentiments | MAE | 8,97E-03 | 7,24E-03 | 1,30E+00 | 1,30E-02 | 1,35E-02 | 2,69E-01 | 1,07E-02
LSTM+Sentiments+Explanations (granite) | MAE | 5,48E-03 | 1,22E-02 | 8,53E-01 | 2,63E-03 | 1,69E-02 | 1,78E-01 | 9,31E-03
LSTM+Sentiments+Explanations (gpt4.0) | MAE | 8,90E-03 | 5,84E-03 | 1,77E+00 | 5,75E-03 | 5,97E-03 | 3,60E-01 | 6,62E-03
LSTM+Sentiments+Explanations (gpt3.5) | MAE | 8,51E-03 | 6,93E-03 | 2,46E+00 | 2,62E-03 | 8,21E-03 | 4,97E-01 | 6,57E-03
LSTM | MAPE | 2,20% | 1,59% | 0,81% | 1,99% | 1,19% | 1,56% | 1,74%
LSTM+Sentiments | MAPE | 1,34% | 0,66% | 0,98% | 1,32% | 1,08% | 1,08% | 1,10%
LSTM+Sentiments+Explanations (granite) | MAPE | 0,82% | 1,11% | 0,64% | 0,27% | 1,35% | 0,84% | 0,89%
LSTM+Sentiments+Explanations (gpt4.0) | MAPE | 1,35% | 0,54% | 1,29% | 0,38% | 0,89% | 0,89% | 0,79%
LSTM+Sentiments+Explanations (gpt3.5) | MAPE | 1,29% | 0,64% | 1,79% | 0,27% | 0,66% | 0,93% | 0,72%

##  6 Related Work

We relate our work to the literature in the domain of explainability for SA.

Generative AI and transformer-based LLMs like BERT [6] and GPT [21] have
excelled in sentiment analysis among various tasks, with BERT significantly
advancing general language understanding tasks [20]. However, their large size
and weight count lead to slower training. GPT-3.5’s performance further
highlights LLMs’ superiority in current methods [13]. Our research extends
LLMs’ utility to explain other ML models used for SA, challenging conventional
post-hoc XAI tools like SHAP or LIME for explainability [13]. Recent studies,
including one on financial texts, indicate that LLMs can enhance sentiment
classification performance by 35% and similarly improve correlations with
market returns [10]. While this work shows the potential benefit of utilizing
LLMs for sentiment classification and in demonstrating that such sentiments
retain a level of about $0.6-0.7$ correlation with market trends, our work
extends this work further in two aspects. We show that the enrichment of a ML
model input with not only the core sentiments associated with market prices,
but also with the set of K-sufficient terms associated with each such
sentiment, could improve the accuracy of such a model in predicting market
trends. Subsequently, we also show that the model is fairly capable of better
predicting not only the market trends with respect to individual currency-
pairs, but also the actual future prices.

The work in [9] shows LLMs’ potential for explaining business process models,
offering a simpler alternative to traditional XAI tools [17]. LLMs, despite
their complex architecture, can improve result understanding and trust through
LLM-based explanations [7]. A survey by Zhao et al. [24] highlights various
LLM explainability techniques based on the training paradigms of LLMs.

Probably the closest to our work on using LLMs to explain sentiment analysis
results is the work in [3] where LLMs are being employed for generating
counterfactual explanations. It presents a systematic approach to identify a
minimal set of text elements that changes the classification outcomes.
Following the use of LLMs for explaining SA results, we can rely on their
application for the realization of the optional minimality step in our method
for SA explainability.

While the interpretability of explanations for specific users can only be
verified through empirical assessment, as pursued in [9], LLMs can also assist
in automating the fidelity check of explanations. As indicated in [18], LLMs
can contribute to the factuality evaluation of a given summary to assess its
faithfulness.

Our core innovation is in the post-hoc methodological approach for leveraging
an LLM to identify a sufficient set of K-terms that retains the original
classification of the SA model as a novel approach, inspired by counterfactual
explanations in adversarial machine-learning in which minimal sufficient sets
of terms may typically be sought to reverse a model’s classification [23]. Our
approach leverages LLMs to infer keywords based on their inherent generative
power derived from the richness of underlying corpora used in their
development, as opposed to employing some XAI algorithmic convergence approach
as in [4]. Furthermore, we exploit these keywords to improve the accuracy of
the ML model predictions of market prices.

##  7 Conclusions and Future Work

Our results support the two hypotheses. While we may have limited the scope of
our empirical investigation presented to a small set of currency pairs during
3 months, our results look promising concerning both envisioned concepts,
demonstrating the potential monetary benefit of leveraging recent LLM
technology for the financial domain and automated trading. Further
investigation should aspire to increase the size of the dataset to corroborate
our findings. Our results are not only consistent with recent findings that
argue for the predictive value of utilizing sentiment information for the
prediction of future prices but are also showing great promise in using
explainability information in a way that goes beyond its conventional intent
to make model predictions more interpretable to humans. As one such concrete
application, our results show the value of leveraging sentiment explainability
to automatically enrich the input into predictive financial trading models,
where such input promotes improving its predictive accuracy. While our second
hypothesis was validated only with one technique for explanation derivation
and enrichment, other realization methods may be employed, such as replacing
embeddings with SHAP values as the realization of the input enrichment. In
general, we argue that the presented application in the financial domain is
only one example domain in which this approach could be exploited, while in
general, the use of explainability could yield a beneficial input enrichment
in other application domains as well.

We also foresee how recent advances in LLMs developed for time-series
predictions present a viable alternative to using models such as LSTM employed
in our work and are likely to equip companies with complete out-of-the-box
tool suites that could be fairly easy to compose to automate such
applications.

This project has received funding from the European Union’s Horizon research
and innovation programme under grant agreement no 101092639 (FAME).

## References

  * Adadi and Berrada [2018] A. Adadi and M. Berrada.  Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI).  _IEEE Access_ , 6, 2018.  ISSN 21693536.  10.1109/ACCESS.2018.2870052.
  * Bengio et al. [2003] Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin.  A neural probabilistic language model.  _J. Mach. Learn. Res._ , 3(null):1137–1155, 3 2003.  ISSN 1532-4435.
  * Bhattacharjee et al. [2023] A. Bhattacharjee, R. Moraffah, J. Garland, and H. Liu.  Towards LLM-guided Causal Explainability for Black-box Text Classifiers.  _ArXiv_ , 9 2023.
  * Carter et al. [2020] B. Carter, J. Mueller, S. Jain, and D. Gifford.  What made you do this? Understanding black-box decisions with sufficient input subsets.  In _AISTATS 2019 - 22nd International Conference on Artificial Intelligence and Statistics_ , 2020.
  * Chandrasekaran and Ramos [2023] A. Chandrasekaran and L. Ramos.  Hype Cycle for Generative AI.  Technical report, Gartner, 9 2023.
  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _ArXiv_ , 10 2018.
  * Ding et al. [2024] Q. Ding, D. Ding, Y. Wang, C. Guan, and B. Ding.  Unraveling the landscape of large language models: a systematic review and future perspectives.  _Journal of Electronic Business & Digital Economics_, 3(1):3–19, 2 2024.  ISSN 2754-4214.  10.1108/JEBDE-08-2023-0015.
  * Dumas et al. [2023] M. Dumas, F. Fournier, L. Limonad, A. Marrella, M. Montali, J.-R. Rehse, R. Accorsi, D. Calvanese, G. De Giacomo, D. Fahland, A. Gal, M. La Rosa, H. Völzer, and I. Weber.  AI-augmented Business Process Management Systems: A Research Manifesto.  _ACM Transactions on Management Information Systems_ , 14(1), 2023.  ISSN 21586578.  10.1145/3576047.
  * Fahland et al. [2024] D. Fahland, F. Fournier, L. Limonad, I. Skarbovsky, and A. J. E. Swevels.  How well can large language models explain business processes?  _ArXiv_ , 1 2024.
  * Fatouros et al. [2023] G. Fatouros, J. Soldatos, K. Kouroumali, G. Makridis, and D. Kyriazis.  Transforming sentiment analysis in the financial domain with ChatGPT.  _Machine Learning with Applications_ , 14:100508, 12 2023.  ISSN 26668270.  10.1016/j.mlwa.2023.100508.
  * Guidotti et al. [2018] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi.  A survey of methods for explaining black box models.  _ACM Computing Surveys_ , 51(5), 2018.  ISSN 15577341.  10.1145/3236009.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter and J. Schmidhuber.  Long Short-Term Memory.  _Neural Computation_ , 9(8):1735–1780, 11 1997.  ISSN 0899-7667.  10.1162/neco.1997.9.8.1735.
  * Kheiri and Karimi [2023] K. Kheiri and H. Karimi.  SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning.  _ArXiv_ , 7 2023.
  * Liu [2012] B. Liu.  _Sentiment Analysis and Opinion Mining_.  Springer International Publishing, Cham, 2012.  ISBN 978-3-031-01017-0.  10.1007/978-3-031-02145-9.
  * Liu et al. [2023] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.  _ACM Computing Surveys_ , 55(9), 2023.  ISSN 15577341.  10.1145/3560815.
  * Lundberg and Lee [2017] S. M. Lundberg and S. I. Lee.  A unified approach to interpreting model predictions.  In _Advances in Neural Information Processing Systems_ , volume 2017-December, pages 4768–4777, 2017.
  * Mavrepis et al. [2024] P. Mavrepis, G. Makridis, G. Fatouros, V. Koukos, M. M. Separdani, and D. Kyriazis.  XAI for All: Can Large Language Models Simplify Explainable AI?  _ArXiv_ , 1 2024.
  * Meske et al. [2022] C. Meske, E. Bunde, J. Schneider, and M. Gersch.  Explainable Artificial Intelligence: Objectives, Stakeholders, and Future Research Opportunities.  _Information Systems Management_ , 39(1):53–63, 1 2022.  ISSN 1058-0530.  10.1080/10580530.2020.1849465.
  * Meskó [2023] B. Meskó.  Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial.  _Journal of Medical Internet Research_ , 25, 10 2023.  ISSN 1438-8871.  10.2196/50638.
  * Mutinda et al. [2023] J. Mutinda, W. Mwangi, and G. Okeyo.  Sentiment Analysis of Text Reviews Using Lexicon-Enhanced Bert Embedding (LeBERT) Model with Convolutional Neural Network.  _Applied Sciences (Switzerland)_ , 13(3), 2023.  ISSN 20763417.  10.3390/app13031445.
  * Radford et al. [2018] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever.  Improving Language Understanding by Generative Pre-Training.  _OpenAI.com_ , 2018.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin.  "Why should i trust you?" Explaining the predictions of any classifier.  In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , volume 13-17-August-2016, pages 1135–1144, 2016.  10.1145/2939672.2939778.
  * Watson et al. [2022] D. S. Watson, L. Gultchin, A. Taly, and L. Floridi.  Local Explanations via Necessity and Sufficiency: Unifying Theory and Practice.  _Minds and Machines_ , 32(1):185–218, 3 2022.  ISSN 0924-6495.  10.1007/s11023-022-09598-7.
  * Zhao et al. [2024] H. Zhao, H. Chen, F. Yang, N. Liu, H. Deng, H. Cai, S. Wang, D. Yin, and M. Du.  Explainability for Large Language Models: A Survey.  _ACM Transactions on Intelligent Systems and Technology_ , 15(2):1–38, 4 2024.  ISSN 2157-6904.  10.1145/3639372.