# Comparing Deep Learning Models for the Task of Volatility Prediction Using
Multivariate Data

Wenbo Ge<EMAIL_ADDRESS>0000-0002-0429-3061 The Australian National
UniversitySchool of Computing145 Science Road, ActonCanberraACT2601Australia ,
Pooia Lalbakhsh<EMAIL_ADDRESS>0000-0001-9267-2610 Monash
UniversityMonash Energy Institute
Department of Data Science & Artificial Intelligence20 Exhibition
WalkClaytonVictoria3168Australia ,  Leigh Isai<EMAIL_ADDRESS>0000-0003-3717-3477 Euler Capital63-67 Cemetery
RoadDrysdaleVictoria3222Australia ,  Artem Lensky<EMAIL_ADDRESS>0000-0002-4745-6756 Lensky AnalyticsBraddonACT2601Australia and  Hanna
Suominen<EMAIL_ADDRESS>0000-0002-4195-1641 The Australian
National UniversitySchool of Computing145 Science Road,
ActonCanberraACT2601Australia University of TurkuDepartment of
ComputingTurkuFI-20014Finland

###### Abstract.

This study aims to compare multiple deep learning-based forecasters for the
task of predicting volatility using multivariate data. The paper evaluates a
range of models, starting from simpler and shallower ones and progressing to
deeper and more complex architectures. Additionally, the performance of these
models is compared against naive predictions and variations of classical GARCH
models.

The prediction of volatility for five assets, namely S&P500, NASDAQ100, gold,
silver, and oil, is specifically addressed using GARCH models, Multi-Layer
Perceptrons, Recurrent Neural Networks, Temporal Convolutional Networks, and
the Temporal Fusion Transformer. In the majority of cases, the Temporal Fusion
Transformer, followed by variants of the Temporal Convolutional Network,
outperformed classical approaches and shallow networks. These experiments were
repeated, and the differences observed between the competing models were found
to be statistically significant, thus providing strong encouragement for their
practical application.

deep learning, financial volatility forecasting, machine learning, neural
networks, predictive analytics

††copyright: none

##  1\. Introduction

One of the most important tasks for a financial institution is to monitor the
volatility of its portfolio. Volatility is a key factor in security valuation,
risk management, and options pricing, as well as it affects investment choices
and valuation of public and corporate liabilities (Enow, 2023). Needless to
say, being able to forecast volatility and the many components affected by it
would be extremely beneficial.

However, given that price aggregates a myriad of factors including economic
fundamentals, market sentiment, market makers, as well as the internal market
dynamics affected by the orderbook and stop-losses (Liston et al., 2023),
sophisticated computational models are developed to support practitioners’
judgment and decision making in quantitative finance. The stylised facts of
volatility (Guillaume et al., 1997; Engle and Patton, 2007; Masset, 2011), the
efficient market hypothesis (Malkiel, 1989; Timmermann and Granger, 2004), and
the ephemeral nature of financial relationships (Clarida et al., 2000; Edwards
et al., 2003) are just a few of the reasons why designing such forecasting
models, developing their evaluation settings, and deploying the evaluated
models in the specifics of the task are complex.

By proceeding from simpler or shallower models (i.e., the GARCH models and
Multi-Layer Perceptrons (MLP)) to deeper and more complex NNs (i.e., the
Recurrent NNs (RNN), _Convolutional NNs_ (CNN), Temporal Convolutional
Networks (TCN), and Temporal Fusion Transformer (TFT)), we will forecast the
volatility of five assets (i.e., S&P500, NASDAQ100, gold, silver, and oil).
These carefully developed examples with performance evaluations and
statistical analyses should encourage practitioners to study and apply deep
learning (DL) in forecasting volatility as a way of improving the accuracy of
predicted volatility.

##  2\. Background and literature review

###  2.1. Defining volatility

Volatility is often referred to as latent and unobservable (Poon and Granger,
2003). Consequently, there are many different ways to quantify volatility
(Patton, 2011), such as historical volatility (HV, the standard deviation of
logarithmic returns over a time window) (Poon and Granger, 2003),111Note that
despite the name containing the word ‘historical’, it is not defined
exclusively for historical data. This can still be forecasted in the same way
that realized volatility can forecast. Alternatively, HV is sometimes referred
to as the close-to-close method. realised volatility (RV, the square root of
the sum of squared log-returns over a time window) (Andersen and Bollerslev,
1998), implied volatility (IV, backwards calculated from options prices via an
option pricing model, such as Black-Scholes) (Mayhew, 1995), and many more.
Each definition has different properties, discussed in more detail in (Ge et
al., 2022), and which definition to use should reflect the aspect of
volatility that needs to be captured and the purpose of forecasting
volatility. If we are trying to forecast options prices, IV is a natural
choice, as it is one of the underpinning variables in its pricing. IV can also
be good if we are trying to gain insight into the market sentiment of certain
assets, as the IV of the S&P index is commonly referred to as the ‘fear’ and
‘greed’ index (Vergili and Çelik, 2023). Alternatively, if we are more
interested in the changes in price or returns of a given asset, we may decide
to use a HV or RV.

In addition to the definition of volatility, we must also decide on the window
of time that this volatility captures, keeping in mind that the further into
the future we are trying to forecast, the more uncertain any forecast will be.
Again, this should depend on the reason we are forecasting volatility; for
example, market makers and day traders may want to monitor short-term
volatility in the span of minutes to watch for entry/exit signals. The
volatility in the very long term (such as over the next year) will hold almost
no useful information as the time span is so wide, and will also be very hard
to forecast as the forecast period is so far from the information period.
Volatility in the very short term may also not be useful for the investors, as
they do not intend to action anything over the seconds or minutes time frame.
However, predicting short-term volatility is of interest to market makers and
high-frequency arbitrage traders (Hao and Lensky, 2023).

###  2.2. Motivation

The efficient market hypothesis is sometimes interpreted to mean that, with a
good enough model, the market price of assets is the only data required as it
captures all public information (Nazlioglu et al., 2023). The reality is that
even if market prices were the culmination of all public information, not all
public information can be decoded from market prices, the relationship is not
injective, and thus not invertible. Despite this, it is often the case that
the market price of the asset of interest is enough information for a decent
forecast, especially in near and short-term volatility forecasts as trends and
patterns can be detected.

In many other cases, introducing additional data that may capture useful
information and that can be readily decoded is helpful. This is not
necessarily limited to market prices or even financial data. For example,
information on the economy of a country can be interpreted from the Gross
Domestic Product (GDP) or its unemployment rate, and the level of inflation
can be drawn from the Consumer Price Index (CPI), the purchasing power of a
currency can be derived from its exchange rates to several highly traded
currencies, and the market sentiment may be derived from the options prices
for given assets. The time of day or the day of the week may also act as a
conditional for some models, as trading patterns may change in different
periods. The public’s sentiment about the market and its volatility can be
decoded from Tweets and Google Search phrases (Kordonis et al., 2016; Ren et
al., 2019). Information regarding what the public might do, and thus how
markets will move, may be held in news articles or the social media of
influential individuals; a ‘follow-the-leader’ phenomenon (Dittmar et al.,
2012; Zhang et al., 2018). Insider information may be reflected in portfolio
changes of well-connected individuals (Lakonishok and Lee, 2001; Neupane et
al., 2021).

The time period for which the data is useful also needs to be considered. In
other words, we should consider the time frame of useful information to
provide to the model for training and inference. If the goal was to forecast
volatility in the short term, such as the next 15 minutes, it may be a waste
to use data from the previous 50 years, as the trading strategies and market
regimes in the early years are very likely to be different than in the later
years. However, using only a week’s worth of training data may also miss out
on information from past market regimes that may repeat. The amount of
information provided at inference time is also important as it impacts the
computation time, as well as may dilute the useful information. When inferring
the volatility of the next 30 days, there is likely to be useful information
held the day before, but that alone may not be enough. However, using all the
data from the beginning of time will dilute the useful information, as well as
cause the model to compute for a long time for inference.

How the data are framed and presented to the model is also a very important
factor. The same data (e.g., the daily closing market prices of several
assets) can be presented in many different ways to a model. In its simplest
form, a string of numbers, each representing the close price of a given day,
can be indexed by date or not. Often, this data representation is normalized
by using one or more of many available normalization algorithms (e.g., 0-1,
logarithmic, unit vector, or min-max) before model training and inference. The
data could also be transformed into a graphical representation e.g. an image
so that the dependencies and explainability are better incorporated into the
system (Hao and Lensky, 2023).

Finally, the timing of the data must be considered. If using daily prices,
some assets are not on the market every day, and as a result, when using
multiple data streams, a mismatch between the date and time of each point in
the time series is likely to be present; a simple cure is to fill in the
missing data between points by using interpolation. One aspect that should be
clarified is that future information should not be fed back for any kind of
testing (this would violate good practice; Leaving empty days and using a
forecasting model that is robust to this provides another alternative. To
account for all the aforementioned nuances, models of certain structures with
a high number of parameters are required. Deep learning models are a good fit
for the task and deserve investigation.

###  2.3. Forecasting models for financial volatility

One of the most widely used family of models is the auto-regressive (AR)
models, which encompasses the well-known GARCH model (Bollerslev, 1986) and
its many variants. Each variant captures different properties of volatility
that may be more important for each individual task and typically do not
include exogenous variables. Intelligent models from machine learning(ML) and
DL have also shown much success (Cavalcante et al., 2016; Ismail Fawaz et al.,
2019; Chen et al., 2022), rising in popularity due to their successful
application in many other fields. However, until recent years the intelligent
models used in the State-Of-The-Art (SOTA) financial volatility forecasting
tend to be much shallower and simpler than those typically used in DL (Ge et
al., 2022).

Of the many types of models that can be used to understand and forecast
volatility, none are as widespread as the AR family of models. The seminal
Auto-Regressive Conditional Heteroscedasticity (ARCH) model, initially
proposed in 1982, models future volatility conditioned on previous
observations (Engle, 1982). This was then adapted to the GARCH model
(Bollerslev, 1986) by including an Auto-Regressive Moving Average (ARMA)
component in the model (Bollerslev, 1986). Since then, there have been many
advancements that attempt to address the models’ inability to capture several
stylized facts of volatility (Engle and Patton, 2007). Several examples are
the Exponential, Threshold, and Glosten-Jagannathan-Runkle versions (EGARCH,
TGARCH, GJR-GARCH) which allow for asymmetric dependencies in volatility, and
the Integrated and Fractionally Integrated versions (IGARCH, FIGARCH) which
address volatility persistence, where an observed shock in the volatility
series seems to impact future volatility over a long horizon. Despite the many
variants of the GARCH model, several experiments have found that the simple
ARCH($1$) and GARCH($1$, $1$) forecasting models perform the best (Hansen and
Lunde, 2005; Orhan and Köksal, 2012).

Neural network (NN) based models are commonly used in various areas including
volatility prediction (Ge et al., 2022), and although they do not have the
same theoretical underpinnings as the aforementioned GARCH models, they are
extremely flexible, often referred to as universal function approximators
(Hornik et al., 1989), possessing the ability to learn any arbitrary mapping
$f$ from input X to output $y$; $y=f(\textbf{X})$ (White, 1990). In the
context of time series analysis, a Nonlinear Auto-Regressive (NAR) framework
is often adopted with the MLP, enforcing an AR property to the nonlinear
mapping (e.g., $\widehat{y}_{t+1}=f([y_{t},y_{t-1},...,y_{t-m}]^{T})$)
(Lapedes and Farber, 1987; Chow and Leung, 1996; Kumar P. and Patil, 2015).
This can be extended into the framework of the nonlinear autoregressive
network with exogenous inputs (NARX) by including such input variables as
those derived from several indices, exchange rates, and outputs of GARCH
models, thus providing more information to the model (Lin et al., 1996; Bucci,
2020) which has been beneficial for forecasting performance (Kristjanpoller
and Hernández, 2017). Other network architectures, such as the RNN originally
designed for sequential data, have also been used in volatility forecasting
(Hajiabotorabi et al., 2019). Variations of the RNN, such as Long Short-Term
Memory (LSTM) models, can also be used, for instance in (Kim and Won, 2018)
whereby an LSTM and GARCH model were combined to forecast HV. CNNs and other
NNs used in computer vision can also be exploited by first converting the time
series into an image; gold prices can be converted into a 3-channel RGB image
and then processed with a pre-trained vgg16 (a well-known and high-performing
CNN model (Simonyan and Zisserman, 2015)) (Vidal and Kristjanpoller, 2020).

Although RNN, LSTM, and CNN are considered to be deep models, they are not
considered SOTA for time-series processing in DL, a distinct gap highlighted
in a recent systematic literature review (Ge et al., 2022). This is reserved
for recent models that have extremely deep capacity and use complex models,
often adapted from other fields such as TCNs, which have been successful in
music generation, speech enhancement, and many other areas involving time
series (Oord et al., 2016; Lea et al., 2017; Pandey and Wang, 2019). The TCN
was an adaptation of the CNN, consisting of 1-dimensional convolutional blocks
structured in a way that does not violate the temporal ordering of data (that
is, only past data can be seen when forecasting), known as a causal
convolution (Oord et al., 2016). In conjunction with a progressively
increasing dilation size, the receptive field can be exponentially increased
as layers increase, thus allowing for the exploitation of long-term
relationships. These blocks also often use residual connections, layer
normalization, gradient clipping, and dropout, all of which have been shown to
improve learning and performance (Pascanu et al., 2013; Ba et al., 2016; He et
al., 2016; Bai et al., 2018; Zhang et al., 2020).

The Transformer was originally designed for natural language processing and
handles sequential data well (Vaswani et al., 2017). It has outperformed
several top models in different natural language processing tasks and also
performed well in unrelated tasks without being trained for them, leading to
the hypothesis that the Transformer is a general model (Brown et al., 2020).
The key building block in the Transformer is multiheaded self-attention, which
allows the data to be encoded in a more meaningful way. The decoder then makes
a prediction, conditional on the inputs given. Several variants of the
Transformer have been specifically adapted to time series data, such as the
TFT. The TFT builds on top of the Transformer by deploying a gating mechanism
to skip unused components of the network, variable selection networks to
select relevant input variables at each time step, static co-variate encoders
to provide context to the model, temporal processing to learn long and short-
term relationships, and quantile predictions to forecast with corresponding
confidence (Lim et al., 2020).

##  3\. Methodology

Table 1.  Description of data Asset | Start date | End date
---|---|---
S&P500 | 22/Sep/2003 | 31/Dec/2018
NASDAQ100 | 01/Jan/2003 | 31/Dec/2018
Gold | 03/Jun/2008 | 31/Dec/2018
Silver | 16/Mar/2011 | 31/Dec/2018
WTI Crude Oil | 10/May/2007 | 31/Dec/2018

###  3.1. Data

####  3.1.1. Main data

Volatility was forecast for five assets: S&P500, NASDAQ-100 (NDX), gold,
silver, and oil. The data for each, as well as the corresponding volatility
indices, was collected (Table 1). The data consisted of the daily closing
prices, as well as the open, high, and low prices for S&P500, NDX, gold, and
oil. Volume was available only for S&P500 and NDX. Each asset was restricted
to a starting date that corresponded with when the volatility index was
introduced, except for S&P500 and NDX. This was because the volatility index
for S&P500 was originally for the S&P100 and later changed on 22 September
2003 and because the volatility index for NDX began earlier than one of the
exogenous variables used. Additionally, the ending date was restricted to 31
December 2018.

####  3.1.2. Exogenous/extra data

Exogenous variables consisted of indices such as SZSE, BSE SENSEX, FTSE100 and
DJIA, exchange rates US-YEN, US-EURO and the US dollar trade-weighted index as
well as such United States fundamentals as Federal Reserve primary credit
rate, the mean and median duration of unemployment, consumer price index
inflation rate, government debt per GDP, gross Federal debt, and currency in
circulation. Table 2 lists all types of input variables used per model. All
variables were date matched with the underlying assets by bringing forward the
nearest historical value, as depicted in Fig. 1.

Table 2.  Additional input variables for forecasting models Model |  Input
---|---
NARX |  Log returns, squared log returns, inverse price of the underlying asset, naïve forecasts for HV, naïve forecasts for IV, US dollar trade-weighted index, primary credit rate, and median length of unemployment
ANN-GARCH |  Log returns of the indices: [BSE SENSEX, FTSE100, SZSE, and DJIA], log-returns of the exchange rates: [US-YEN and US-EURO], the naïve forecasts for HV, the naïve forecasts for IV, and the log-returns of the underlying asset.
TCN |  Log returns of underlying variable, naïve forecasts for HV, naïve forecasts for IV, the current direction of movement, US dollar trade-weighted index, Federal Reserve primary credit rate, mean duration of unemployment, the median duration of unemployment, consumer price index inflation rate, Government debt per GDP, gross Federal debt, and currency in circulation
TFT |  Open, high, low, close, volume (where possible), log returns, squared log returns, inverse price of the underlying asset, naïve forecasts for HV, naïve forecasts for IV, the US dollar trade-weighted index, Federal Reserve primary credit rate, mean duration of unemployment, the median duration of unemployment, consumer price index inflation rate, Government debt per GDP, gross Federal debt, currency in circulation, day of the week, month of the year, and number of days since previous observation
Figure 1. Date matching method for exogenous variables

The task was to forecast the volatility of the next month for the five assets;
more specifically, to forecast the month-long HV and IV (Fig. 2), starting
from 1 day ahead, for S&P500, NDX, gold, silver, and oil. The ground truth for
HV was the standard deviation of the logarithmic returns from 1 trading day
ahead to 22 trading days ahead (which is equivalent to one calendar month)
(Poon and Granger, 2003). In other words, with $t$ referring to the current
time, we defined over a certain period

$[\tau_{1},\,\tau_{2}]=[t+1,\,t+22]$

HV as the standard deviation ($\text{std}(\cdot)$) of log-returns as follows:

(1)
$\text{HV}=\sqrt{\frac{1}{N}\sum_{t=\tau_{1}}^{\tau_{2}}{\left(r_{t}-\frac{1}{N}\sum_{t=\tau_{1}}^{\tau_{2}}{r_{t}}\right)^{2}}}=\text{std}\left(\begin{bmatrix}r_{\tau_{2}}\\\
r_{\tau_{2}-1}\\\ r_{\tau_{2}-2}\\\ \vdots\\\
r_{\tau_{1}}\end{bmatrix}\right)$

where $N=\tau_{2}-\tau_{1}=21$ is the number of samples between the time
steps, $P_{t}$ is price at time $t$, and $r_{t}=\log(P_{t}/P_{t-1})\cdot 100$.

For IV (Mayhew, 1995), this meant that the ground truth was simply the value
of the volatility index for the next trading day, as the volatility index was
already defined for the next calendar month. The values of the volatility
indices were also adjusted by a factor of $1/\sqrt{252}$, deannualizing the
value to be on the same scale as HV.

(a) HV (b) IV

Figure 2. Groundtruth and naïve forecasts.

###  3.2. Methods

To engineer and evaluate the forecasting models using these five methods, a
70-15-15 train-validation-test split of the data was used (Bishop, 2006)
because it did not violate the temporal aspect of the data (Fig. 3). The
training set was used to train the models using different hyperparameters,
which were then evaluated against the validation set to determine the
performance with that given set of hyperparameters. Different combinations of
hyperparameters were searched (see Appendix A for details), and the best-
performing set of hyperparameters proceeded to the test phase. Here, the model
was re-initialized and trained again with the union of both the training and
validation set, then evaluated once using the test set. All performances were
quantified with the Mean Squared Error (MSE) (Bishop, 2006).

Figure 3. Depiction of a 70-15-15 train-validation-test split with the
evaluation process

####  3.2.1. Statistical significance testing

Statistical significance testing was also performed in order to distinguish if
competing models were statistically significantly different from each other.
After the hyperparameters of a model were chosen and the testing phase was
completed, the model was reinitialized with a random seed, re-trained, and re-
tested. This was repeated until ten MSE values were obtained for each model.
These values were then tested across different models in a pair-wise fashion
to determine if they were from the same distribution. If so, the models were
assumed to be equivalent, if not, then the model with the smaller mean MSE was
assumed to be superior. The Shapiro-Wilk (SW) test (Shapiro and Wilk, 1965)
was first applied to assess the normality of the distribution with
significance level $\alpha$ = 0.05. The choice of the SW test was justified by
the applicability of the test to the data with unspecified mean and variance,
as well as its appropriateness for small sample sizes. If both distributions
were normal then Student’s t-test (Student, 1908) was used, otherwise the
Kruskal-Wallis (KW) test (Kruskal and Wallis, 1952) was employed.

####  3.2.2. Description of traditional benchmark models

The naïve model simply repeated the most recent known value of volatility
(Fig. 2). For IV, this was the value of the corresponding volatility index at
the current time. For HV, this was the standard deviation of log returns for
the current day and previous 20 days, that is, Equation (1) with
$\tau_{1}=t-20$ and $\tau_{2}=t$. This assumed that at the time of
forecasting, the current trading day is over and observed, an assumption
maintained for all forecasting models.

Several GARCH variants were reviewed for the best-performing model, along with
the lag order of those models. Namely, the variants consisted of ARCH, GARCH,
EGARCH, GJR-GARCH, and GARCH with a Student’s t-distribution. Lag orders were
1 and 2, as informed by the auto-correlation and partial auto-correlation
functions.

Variants of the MLP were also searched over, using both a NAR and NARX
framework. For the NAR models, the number of lagged inputs, number of hidden
layers, number of neurons, activation function, learning rate, and
preprocessing method was tuned. Similarly, the NARX framework searched over
the same hyperparameters, with the addition of which exogenous variables to
include: the log returns, squared log returns, the inverse price of the
underlying asset, the naïve forecasts for HV and IV, the primary credit rate,
the median length of unemployment, and the US dollar trade-weighted index.

####  3.2.3. Description of benchmark models used for comparison

The first model picked up from the literature was the ANN-GARCH model
(Kristjanpoller and Hernández, 2017), a combination of the previously
mentioned GARCH and NARX MLP models. The GARCH model made a next-day
volatility forecast, which is used as an input for the NARX model, along with
several other variables that describe the asset and the market. Both the GARCH
and NARX models were trained on the prior 252 trading days (one calendar
year), and re-trained for every forecast, resulting in a rolling model. The
variant and lag order of the GARCH model was searched over as hyperparameters,
as well as a number of lagged inputs, the number of hidden layers and neurons,
activation functions, preprocessing method, and the set of additional input
variables for the NARX model. The possible input variables to include were:
the log returns of several indices (BSE SENSEX, FTSE100, SZSE, and DJIA), the
log returns of several exchange rates (US-YEN and US-EURO), the naïve
forecasts for HV and IV, as well as the log-returns of the underlying asset.

The second model was the CNN-LSTM (Vidal and Kristjanpoller, 2020). This used
the log-returns of the underlying asset, converted to a 3-channel RGB image
through Gramian Angular Fields and Markov Transition Fields (Campanharo et
al., 2011; Wang and Oates, 2015) (Fig. 4). Features from these images were
extracted using vgg16 (Simonyan and Zisserman, 2015), and then further
processed by several convolutional blocks. In addition to this, the log-
returns of the underlying asset were also processed by the LSTM, which was
then combined with the processed image features to forecast volatility. The
hyperparameters search was performed over the range of learning rates,
preprocessing methods, and the length of the sequence that the LSTM was
exposed to.

Figure 4. Images generated from Gramian Angular Fields and Markov Transition
Fields with 80 trading days of log returns

####  3.2.4. Experimental models

Several variations of two deep learning models that recently have been used in
predicting volatility are further investigated.

The first model was the vanilla Temporal Convolutional Network(TCN), as well
as the TCN with several modifications. The first modification was to leverage
the naïve model and forecast a residual, defined as either the difference
$y_{\text{groundtruth},t}-y_{\text{na\"{i}ve},t}$, or the log difference
$log(y_{\text{groundtruth},t}/y_{\text{na\"{i}ve},t})$. Another modification
was to include multiple tasks to the network, introducing another loss
function that will have a separate but related training effect (Ruder, 2016).
The additional task was to predict either the direction of the forecast (up or
down), or the change in direction (change or no change), depicted in Fig. 5.
The final modification was to include additional input channels, introducing
new information to the network (Wan et al., 2019), such as descriptors of the
underlying asset (log returns, naïve forecasts for HV and IV, and current
direction of movement), and variables that describe the market (US dollar
trade-weighted index, Federal Reserve primary credit rate, mean and median
duration of unemployment, consumer price index inflation rate, Government debt
per GDP, gross Federal debt, and currency in circulation) as there is
literature to suggest that this may improve performance (Kristjanpoller and
Hernández, 2017). The hyperparameters searched over were as follows: which
definition of the residual to use, the form of the additional task, the
weighting of the additional task, and the set of variables to include, as well
as the number of hidden layers, number of filters, kernel size, dropout,
gradient clipping threshold, learning rate, and preprocessing method.

Figure 5. Examples of direction and direction change

The second model was the Temporal Fusion Transformer (TFT), an adaptation of
the Transformer model. Additional variables such as open, high, low, close,
and volume were investigated, as well as log returns, squared log returns,
inverse price of the underlying asset, and naïve forecasts for both HV and IV.
Fundamentals such as the US dollar trade-weighted index, Federal Reserve
primary credit rate, the mean and median duration of unemployment, consumer
price index inflation rate, Government debt per GDP, gross Federal debt, and
currency in circulation we also considered, finally such time descriptors as
day of the week, month, and a number of days since previous observation, were
also included. The hyperparameter search explored the set of input variables,
as well as the gradient clipping threshold, hidden size, hidden continuous
size, number of attention heads, learning rate, dropout, encoder length, and
prediction length. The full details of the hyperparameter search for these
models were reported in Appendix A.

###  3.3. Result evaluation and analysis

####  3.3.1. Benchmark models

The ANN-GARCH model performed best overall in forecasting HV, achieving the
smallest error for three of the assets, as seen in Table 3. There was no clear
GARCH variant that was best among the ANN-GARCH models, however, a lag order
of 1 was preferred over all assets. Similarly, there was no dominant MLP
structure among the ANN-GARCH models, every specific task had its own best
performing variant. The CNN-LSTM and the MLP models also performed well,
achieving the smallest errors for silver and oil, respectively. The CNN-LSTM
model preferred the LSTM to be exposed to 42 days for all assets, except
S&P500 for which the model preferred 21 days. The MLP model did not have a
consistently preferred lag across the assets but benefited from the inclusion
of exogenous variables. For gold and oil, the best exogenous variables were
those derived from the underlying asset: inverse price, squared log returns,
and naïve forecasts of HV and IV. For S&P500, NDX, and silver, the best
exogenous variables were the market variables: risk-free rate, unemployment,
and the US dollar index. The naïve and GARCH models performed the worst
overall, being beaten by all other models. Of the GARCH model, a GARCH(1,1)
model consistently outperformed all other variants and lag orders.

For IV forecasting, the naïve model performed the best overall, achieving the
smallest error for all five assets (Table 3). Following this, the best models
were the CNN-LSTM and MLP. The optimal exposure for the LSTM in the CNN-LSTM
was 1 day, presenting its best performance for all assets. The MLP model
showed no clear optimal lag order, and the addition of variables did not seem
to help with forecasting the IV of gold, silver, and oil. However, including
the inverse price, squared log returns, and naïve forecasts resulted in better
performance for S&P500 and NDX. There was no clear optimal GARCH variant in
both the GARCH model and the ANN-GARCH model, but a lag order of 1 was optimal
for all assets. The MLP side of the ANN-GARCH model preferred a lag order of
3, and the inclusion of exogenous variables did not improve performance.

Table 3.  Performance (MSE) of benchmark models, forecasting on the test set. Smaller is better, bold is the best. Task |  Method |  S&P500 |  NDX |  Gold |  Silver |  Oil
---|---|---|---|---|---|---
HV |  Naïve |  0.1281 |  0.1887 |  0.0216 |  0.0745 |  0.2799
GARCH |  0.2662 |  0.9430 |  0.0191 |  0.0611 |  7.9448
MLP |  0.2935 |  0.3542 |  0.0146 |  0.0666 |  0.2080
ANN-GARCH |  0.1136 |  0.1735 |  0.0105 |  0.0563 |  0.2191
CNN-LSTM |  0.1550 |  0.1869 |  0.0554 |  0.0488 |  0.3870
IV |  Naïve |  0.0098 |  0.0084 |  0.0013 |  0.0023 |  0.0139
GARCH |  0.2296 |  0.5859 |  0.0263 |  0.0279 |  7.6927
MLP |  0.0172 |  0.0151 |  0.0021 |  0.0032 |  0.0278
ANN-GARCH |  0.0248 |  0.0159 |  0.0022 |  0.0036 |  0.0312
CNN-LSTM |  0.0141 |  0.0104 |  0.0064 |  0.0104 |  0.0246

####  3.3.2. Hyperparameter optimization

In testing two hyperparameter optimization methods, BOHB only produced a
better forecasting model for the HV of oil, and the IV of gold and silver,
whilst grid search resulted in a model with better performance for all other
assets (Table 4). Both methods were given roughly the same wall time and were
both tested using the un-modified TCN. Though it is difficult to say if one
method is superior to the other, the continued use of grid search is justified
and was the primary hyperparameter optimization method for the remaining
experimental TCN models.

Table 4.  Performance (MSE) of hyperparameter search for TCN, forecasting on the test set. Smaller is better, bold is the best. Task |  Method |  S&P500 |  NDX |  Gold |  Silver |  Oil
---|---|---|---|---|---|---
HV |  Grid search |  0.1467 |  0.1912 |  0.0331 |  0.0364 |  0.2614
BOHB |  0.1591 |  0.2010 |  0.0616 |  0.0800 |  0.2107
IV |  Grid search |  0.0098 |  0.0083 |  0.0029 |  0.0032 |  0.0136
BOHB |  0.0126 |  0.0093 |  0.0017 |  0.0025 |  0.0139
Table 5.  Performance (MSE) of experimental models, forecasting on test set. Smaller is better, bold is the best. Task |  Method |  S&P500 |  NDX |  Gold |  Silver |  Oil
---|---|---|---|---|---|---
HV |  Naïve |  0.1281 |  0.1887 |  0.0216 |  0.0745 |  0.2799
Best benchmark |  0.1136 |  0.1735 |  0.0105 |  0.0488 |  0.2080
TCN |  0.1467 |  0.1912 |  0.0331 |  0.0364 |  0.2614
TCN variants |  0.1300 |  0.1728 |  0.0153 |  0.0315 |  0.2050
TFT |  0.0294 |  0.0513 |  0.0116 |  0.0341 |  0.0864
IV |  Naïve |  0.0098 |  0.0084 |  0.0013 |  0.0023 |  0.0139
Best benchmark |  0.0141 |  0.0104 |  0.0021 |  0.0032 |  0.0246
TCN |  0.0098 |  0.0083 |  0.0029 |  0.0032 |  0.0136
TCN variants |  0.0093 |  0.0081 |  0.0013 |  0.0022 |  0.0127
TFT |  0.0094 |  0.0083 |  0.0013 |  0.0025 |  0.0134

####  3.3.3. Experimental models

The TFT model performed best overall for HV forecasting, achieving the
smallest errors for S&P500, NDX, and oil (Table 5). An encoder length of 21
days was optimal for all assets, with no set of input variables that were
consistently best. S&P500 and NDX performed best with the addition of
variables that describe time and the underlying asset, gold and silver
performed best with the addition of variables that describe time, and oil
performed best with the addition of market and time descriptors. TCN variants
also performed well, achieving the smallest error for silver when utilizing
the secondary task of predicting direction change. There was no consistent
variant that performed best: S&P500 and NDX preferred forecasting the
residuals, whilst gold, silver, and oil performed better when utilizing a
secondary task. The inclusion of exogenous variables only increased the
performance for forecasting gold HV. Additionally, TCN variants consistently
performed better than the un-modified TCN (Table 5). The smallest error for
gold was achieved by a benchmark model, specifically the ANN-GARCH.

The TCN variants were the best performing model for IV forecasting, achieving
the smallest errors for all assets (Table 5). The optimal modification was to
use a secondary task of predicting the direction, as well as forecasting the
residuals, consistent amongst all assets. S&P500, NDX, and gold also benefited
from the inclusion of the volatility index value and previous direction of
movements. For the TFT model, an encoder length of 10 days was preferred for
all assets, except for S&P500 which preferred a length of 126. There was no
consistent set of input variables that performed the best. Silver preferred
the addition of variables that describe time, gold preferred the addition of
variables that describe the underlying asset and time, whilst S&P500, NDX, and
oil performed best with the addition of all variables available: descriptors
of the market, the underlying asset, and time.

####  3.3.4. Statistical significance

Statistical testing results were compiled into ten charts in figures (6, 7),
one for each combination of an asset (S&P500, NDX, gold, silver, and oil) and
task (HV and IV forecasting). Each chart is comprised of a horizontal pane and
a 6$\times$6 grid. The horizontal pane shows the distribution of errors from a
given model over ten repetitions, indicated by the red and blue vertical
lines, relative to the naïve forecast, the black horizontal line.

(a) SPX (b) Oil (c) Gold

(d) Silver (e) Nasdaq

Figure 6. Model comparison of historical volatility prediction using MSE
obtained for ten random seed values.

(a) SPX (b) Oil (c) Gold

(d) Silver (e) Nasdaq

Figure 7. Model comparison of implied volatility prediction using MSE obtained
for ten random seed values.

The brackets shown in blue in figures (6, 7) denote that a statistically
significant difference could not be found between the corresponding model and
the naïve model, whilst the brackets in red indicate that a statistically
significant difference was found. The 6$\times$6 grid implies that a
statistical difference was detected between different models on the $x$ and
$y$ axes, with the benchmark models encased in a pink square, and the
experimental models outside the pink square. The diagonal of this grid also
contains text that shows the average MSE of ten repetitions, with the smallest
error indicated in red text. Grey cells indicate that the model either was not
found to be statistically different from the naïve model, or is worse than it.
Yellow cells indicate that no statistical difference was detected between
models. They appear on the diagonal as comparing the same models always leads
to no difference being found, and in the upper triangle, indicating that there
was no statistical difference between the two corresponding models. Green
cells indicate a statistically significant difference is detected. They only
appear in the upper triangle and indicate that the two corresponding models
are different, implying the model with a smaller error is better than the
other. The lower triangle of the grid is muted with black cells as the results
are mirrored in the upper triangle. Thus, from this set of charts, we can
determine if the model with the smallest error forecast (indicated in red text
on the diagonal) performs similarly (no statistical difference detected, that
is, yellow cells) or better (statistical difference detected, that is, green
cells) than other models.

Table 6.  Smallest error models (in bold), and models for which no statistically significant difference could be found from the smallest error models. Task |  Model |  S&P500 |  NDX |  Gold |  Silver |  Oil
---|---|---|---|---|---|---
HV |  Smallest error |  TFT |  TFT |  ANN-GARCH |  TCN var. |  TFT
Equivalent |  - |  - |  TFT |  TCN |  -
IV |  Smallest error |  TCN var. |  TCN var. |  TCN var. |  TCN var. |  TCN var.
Equivalent |  - |  - |  - |  - |  -

##  4\. Discussion

The evaluation results revealed that the experimental models performed better
than the benchmark ones, across both defined forecasting tasks, and across all
five assets. Statistical significance testing confirmed this, indicating that
for almost all combinations of assets and tasks, the best performance belonged
to an experimental model, with all of the benchmark models different (and
worse) in performance on the basis of statistical significance. The only
exception to this was the HV forecasting of gold, for which a statistically
significant difference could not be found for one benchmark model (ANN-GARCH)
and the experimental models.

As for why the experimental models perform better than the intelligent models
previously used, it is likely a combination of several different reasons. One
difference common amongst the experimental models is that they are deeper,
that is, they have significantly more neurons. This means that they have more
learning capacity, can decode more complex information, and process larger
amounts of data. However, this does not mean that simply increasing the number
of neurons in any NN-based model would yield the same performance
improvements. This is because structure plays an important role.

The structure of the TCN is similar to that of the CNN and thus shares many of
the same benefits. Translational invariance allows the model to detect the
same features regardless of where in time it occurs. Weight sharing and a
progressively increasing dilation imply that each neuron in the TCN is more
efficient than the neurons in an MLP, allowing the TCN to have a wider
receptive field and do more processing with the same number of neurons.
Hierarchical layering along with residual connections allows the TCN to avoid
several problems common to RNNs and LSTMs, such as vanishing gradients and
poor long-term memory. Furthermore, the modifications of the TCN variants also
play an important role. Forecasting a residual rather than the raw value
limits the non-stationarity of the data and encodes the naïve forecast
directly into the inputs, providing an additional source of information and a
good starting point.

The TFT also differs from previously used NN-based models in many ways. One
key difference is the use of multi-headed self-attention, allowing the model
to focus on different periods of time conditional on the current value. As an
example, this means that if current volatility is low, the model might pay
attention to long-term history, whilst if current volatility is high, the
model might pay attention to short-term history. Other models like the MLP and
TCN can only interpret dependencies from fixed relative time steps, akin to a
fixed attention mask. Additionally, the TFT makes better use of certain
context variables like cyclic temporal variables (such as the day of the week
or month of the year) and market variables (such as the strength of the US
dollar, and Federal Reserve credit rates), allowing the model to be better
conditioned when making a forecast. The gating mechanism of the TFT also
allows for depth in certain areas of the model where it is needed, and
shallowness where it is not. Previous models have predefined depth and neuron
count, potentially not having enough capacity to learn certain relationships,
or being overparameterized and simply remembering training examples rather
than generalizing.

These results, however, come with some limitations. The first of which was
that the implementation of several models (GARCH and TFT) was open source and
thus not necessarily under the same strict control as the other models used.
Second, because the purpose of this investigation was to compare different
models, rather than to find the best model, the hyperparameter search space
may be limited. This does not mean to say that the search space was small, but
rather that there likely exists another combination of hyperparameters that
would yield better results.

As it seems that DL may offer better volatility forecasting performance than
traditional methods, however further research is needed to confirm whether
this is true for other models and financial quantities. For instance, there
are already several time series adaptations of the Transformer (Huang et al.,
2018; Li et al., 2019; Wu et al., 2020). The most recent is Autoformer (Wu et
al., 2021) which is an extension of Performer, another transformer-based
model. Autoformer, achieves SOTA, with a 38% relative improvement on six
benchmarks, covering five practical applications: energy, traffic, economics,
weather and disease. For a survey of Autofromer follow-up models see (Zhu et
al., 2023). Another class of models is graph neural networks, which recently
have been applied to the volatility prediction task (Zhang et al., 2023).
Finally, the use of stochastic NNs (Adorf and Johnston, 1990) or Gaussian
process NNs (Lee et al., 2018) may have potential as they handle uncertainty
well, one of the defining features of volatility.

Another avenue to enhance the models is to make use of multi-modal data, that
is, extend to multiple modes, or types, of data. Each type of data may capture
different information, and when combined, could produce better forecasts.
Examples of extensions to text exist (e.g., by using Twitter posts (Kordonis
et al., 2016; Oliveira et al., 2017) or news articles (Robertson et al., 2007;
Malik, 2011)). Combined with the larger capacity of deeper models, this
approach may yield promising results. Furthermore, the use of different
paradigms, such as continual learning, could yield even more benefits,
especially in the arena of applied volatility forecasting for real-world use
(Shin et al., 2017). Further future works could be paved by specifically
designing models for financial volatility forecasting, explicitly defined to
take advantage of different properties in financial time series and data.

##  5\. Conclusions

In closing, the DL-based methods have much to offer for financial volatility
forecasting. This study implemented several deep learning-based and
traditional models and compared these to several experimental recently
proposed models for time series prediction. The results indicate that variants
of TCN and TFT were able to forecast volatility with less error across almost
all assets and tasks investigated, with statistical significance.

## Competing interest

Although this paper is a research outcome related to a PhD student internship,
the authors wish to declare that Euler Capital is a multi-manager platform
investing capital across a broad range of trading strategies, focusing on
investment, research, and technology by deploying quantitative, systematic,
and behavioural-based trading models to transform volatility into opportunity.
The authors have no other conflict of interest to declare.

###### Acknowledgements.

This work was supported by Euler Capital Pty Ltd, under APR Intern Agreement
(INT - 0804). We would also like to thank the APR.Intern and Australian
National University (ANU) for their support to this research through the
Australian Postgraduate Research Internship Program (APR.Intern) program. We
also gratefully acknowledge funding from the Australian Government Research
Training Program (AGRTP) Domestic Scholarship for the first author’s PhD
studies.

## References

  * (1)
  * Adorf and Johnston (1990) Hans-Martin Adorf and Mark D. Johnston. 1990.  A discrete stochastic neural network algorithm for constraint satisfaction problems. In _1990 IJCNN International Joint Conference on Neural Networks_. IEEE, 917–924 vol.3.  https://doi.org/10.1109/IJCNN.1990.137951
  * Andersen and Bollerslev (1998) Torben G. Andersen and Tim Bollerslev. 1998.  Answering the Skeptics: Yes, Standard Volatility Models do Provide Accurate Forecasts.  _International Economic Review_ 39, 4 (Nov. 1998), 885.  https://doi.org/10.2307/2527343
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016\.  Layer Normalization. In _Advances in NIPS 2016 Deep Learning Symposium_.  https://arxiv.org/abs/1607.06450v1
  * Bai et al. (2018) Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. 2018\.  An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling.  (April 2018).  http://arxiv.org/abs/1803.01271
  * Bishop (2006) Christopher Bishop. 2006\.  _Pattern Recognition and Machine Learning_.  Springer-Verlag, New York.  https://www.springer.com/gp/book/9780387310732
  * Bollerslev (1986) Tim Bollerslev. 1986\.  Generalized autoregressive conditional heteroskedasticity.  _Journal of Econometrics_ 31, 3 (April 1986), 307–327.  https://doi.org/10.1016/0304-4076(86)90063-1
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  _Advances in Neural Information Processing Systems_ 33 (2020), 1877–1901.  https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  * Bucci (2020) Andrea Bucci. 2020\.  Cholesky–ANN models for predicting multivariate realized volatility.  _Journal of Forecasting_ 39, 6 (2020), 865–876.  https://doi.org/10.1002/for.2664
  * Campanharo et al. (2011) Andriana S. L. O. Campanharo, M. Irmak Sirer, R. Dean Malmgren, Fernando M. Ramos, and Luís A. Nunes Amaral. 2011.  Duality between Time Series and Networks.  _PLOS ONE_ 6, 8 (Aug. 2011), e23378.  https://doi.org/10.1371/journal.pone.0023378
  * Cavalcante et al. (2016) Rodolfo C. Cavalcante, Rodrigo C. Brasileiro, Victor L. F. Souza, Jarley P. Nobrega, and Adriano L. I. Oliveira. 2016.  Computational Intelligence and Financial Markets: A Survey and Future Directions.  _Expert Systems with Applications_ 55 (Aug. 2016), 194–211.  https://doi.org/10.1016/j.eswa.2016.02.006
  * Chen et al. (2022) Wei-Jie Chen, Jing-Jing Yao, and Yuan-Hai Shao. 2022\.  Volatility forecasting using deep neural network with time-series feature embedding.  _Economic Research-Ekonomska Istraživanja_ (2022), 1–25.
  * Chow and Leung (1996) Tommy W. S. Chow and Chi-Tat Leung. 1996.  Neural network based short-term load forecasting using weather compensation.  _IEEE Transactions on Power Systems_ 11, 4 (Nov. 1996), 1736–1742.  https://doi.org/10.1109/59.544636
  * Clarida et al. (2000) Richard Clarida, Jordi Gali, and Mark Gertler. 2000\.  Monetary Policy Rules and Macroeconomic Stability: Evidence and Some Theory.  _The Quarterly Journal of Economics_ 115, 1 (2000), 147–180.  https://www.jstor.org/stable/2586937
  * Dittmar et al. (2012) Amy Dittmar, Di Li, and Amrita Nain. 2012.  It Pays to Follow the Leader: Acquiring Targets Picked by Private Equity.  _The Journal of Financial and Quantitative Analysis_ 47, 5 (2012), 901–931.  https://www.jstor.org/stable/23351940
  * Edwards et al. (2003) Sebastian Edwards, Javier Gómez Biscarri, and Fernando Pérez de Gracia. 2003.  Stock market cycles, financial liberalization and volatility.  _Journal of International Money and Finance_ 22, 7 (Dec. 2003), 925–955.  https://doi.org/10.1016/j.jimonfin.2003.09.011
  * Engle (1982) Robert F. Engle. 1982\.  Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation.  _Econometrica_ 50, 4 (July 1982), 987–1008.  https://doi.org/10.2307/1912773
  * Engle and Patton (2007) Robert F. Engle and Andrew J. Patton. 2007.  2 - What good is a volatility model?*.  In _Forecasting Volatility in the Financial Markets (Third Edition)_ , John Knight and Stephen Satchell (Eds.). Butterworth-Heinemann, Oxford, 47–63.  https://doi.org/10.1016/B978-075066942-9.50004-2
  * Enow (2023) Samuel Tabot Enow. 2023\.  Modelling and Forecasting volatility in International financial markets.  _International Journal of Research in Business and Social Science (2147-4478)_ 12, 2 (2023), 197–203.
  * Ge et al. (2022) Wenbo Ge, Pooia Lalbakhsh, Leigh Isai, Artem Lenskiy, and Hanna Suominen. 2022.  Neural Network–Based Financial Volatility Forecasting: A Systematic Review.  _Comput. Surveys_ 55, 1 (Jan. 2022), 14:1–14:30.  https://doi.org/10.1145/3483596
  * Guillaume et al. (1997) Dominique M. Guillaume, Michel M. Dacorogna, Rakhal R. Davé, Ulrich A. Müller, Richard B. Olsen, and Olivier V. Pictet. 1997.  From the bird’s eye to the microscope: A survey of new stylized facts of the intra-daily foreign exchange markets.  _Finance and Stochastics_ 1, 2 (April 1997), 95–129.  https://doi.org/10.1007/s007800050018
  * Hajiabotorabi et al. (2019) Zeinab Hajiabotorabi, Aliyeh Kazemi, Faramarz Famil Samavati, and Farid Mohammad Maalek Ghaini. 2019\.  Improving DWT-RNN model via B-spline wavelet multiresolution to forecast a high-frequency time series.  _Expert Systems with Applications_ 138 (Dec. 2019), 112842\.  https://doi.org/10.1016/j.eswa.2019.112842
  * Hansen and Lunde (2005) Peter R. Hansen and Asger Lunde. 2005.  A forecast comparison of volatility models: does anything beat a GARCH(1,1)?  _Journal of Applied Econometrics_ 20, 7 (2005), 873–889.  https://doi.org/10.1002/jae.800
  * Hao and Lensky (2023) Mingyu Hao and Artem Lensky. 2023.  Short-Term Volatility Prediction Using Deep CNNs Trained on Order Flow.  arXiv:2304.02472
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep Residual Learning for Image Recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 770–778.  https://doi.org/10.1109/CVPR.2016.90 ISSN: 1063-6919.
  * Hornik et al. (1989) Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989\.  Multilayer feedforward networks are universal approximators.  _Neural Networks_ 2, 5 (Jan. 1989), 359–366.  https://doi.org/10.1016/0893-6080(89)90020-8
  * Huang et al. (2018) Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. 2018.  Music Transformer.  _arXiv:1809.04281 [cs, eess, stat]_ (Dec. 2018).  http://arxiv.org/abs/1809.04281
  * Ismail Fawaz et al. (2019) Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. 2019.  Deep learning for time series classification: a review.  _Data Mining and Knowledge Discovery_ 33, 4 (July 2019), 917–963.  https://doi.org/10.1007/s10618-019-00619-1
  * Kim and Won (2018) Ha Young Kim and Chang Hyun Won. 2018.  Forecasting the volatility of stock price index: A hybrid model integrating LSTM with multiple GARCH-type models.  _Expert Systems with Applications_ 103 (Aug. 2018), 25–37.  https://doi.org/10.1016/j.eswa.2018.03.002
  * Kordonis et al. (2016) John Kordonis, Symeon Symeonidis, and Avi Arampatzis. 2016\.  Stock Price Forecasting via Sentiment Analysis on Twitter. In _Proceedings of the 20th Pan-Hellenic Conference on Informatics_ _(PCI ’16)_. Association for Computing Machinery, New York, NY, USA, 1–6.  https://doi.org/10.1145/3003733.3003787
  * Kristjanpoller and Hernández (2017) Werner Kristjanpoller and Esteban Hernández. 2017.  Volatility of main metals forecasted by a hybrid ANN-GARCH model with regressors.  _Expert Systems with Applications_ 84 (Oct. 2017), 290–300.  https://doi.org/10.1016/j.eswa.2017.05.024
  * Kruskal and Wallis (1952) William H. Kruskal and W. Allen Wallis. 1952.  Use of Ranks in One-Criterion Variance Analysis.  _J. Amer. Statist. Assoc._ 47, 260 (1952), 583–621.  https://doi.org/10.2307/2280779
  * Kumar P. and Patil (2015) Hemanth Kumar P. and S. Basavaraj Patil. 2015.  Estimation forecasting of volatility using ARIMA, ARFIMA and Neural Network based techniques. In _2015 IEEE International Advance Computing Conference (IACC)_. IEEE, 992–997.  https://doi.org/10.1109/IADCC.2015.7154853
  * Lakonishok and Lee (2001) Josef Lakonishok and Inmoo Lee. 2001.  Are Insider Trades Informative?  _The Review of Financial Studies_ 14, 1 (Jan. 2001), 79–111.  https://doi.org/10.1093/rfs/14.1.79
  * Lapedes and Farber (1987) Alan S. Lapedes and Robert Farber. 1987.  Nonlinear signal processing using neural networks: Prediction and system modelling. In _IEEE international conference on neural networks_. 52.  https://www.osti.gov/biblio/5470451
  * LaValle et al. (2004) Steven M. LaValle, Michael S. Branicky, and Stephen R. Lindemann. 2004.  On the Relationship between Classical Grid Search and Probabilistic Roadmaps.  _The International Journal of Robotics Research_ 23, 7-8 (Aug. 2004), 673–692.  https://doi.org/10.1177/0278364904045481
  * Lea et al. (2017) Colin Lea, Michael D. Flynn, Rene Vidal, Austin Reiter, and Gregory D. Hager. 2017.  Temporal Convolutional Networks for Action Segmentation and Detection. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, Honolulu, HI, 1003–1012.  https://doi.org/10.1109/CVPR.2017.113
  * Lee et al. (2018) Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. 2018.  Deep Neural Networks as Gaussian Processes. In _Sixth International Conference on Learning Representations (ICLR)_. 17.  http://arxiv.org/abs/1711.00165
  * Li et al. (2019) Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019.  Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. In _Advances in Neural Information Processing Systems_ , Vol. 32. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2019/hash/6775a0635c302542da2c32aa19d86be0-Abstract.html
  * Lim et al. (2020) Bryan Lim, Sercan O. Arik, Nicolas Loeff, and Tomas Pfister. 2020.  Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting.  _arXiv:1912.09363 [cs, stat]_ (Sept. 2020).  http://arxiv.org/abs/1912.09363
  * Lin et al. (1996) Tsungnan Lin, Bill G. Horne, Peter Tino, and C. Lee Giles. 1996\.  Learning long-term dependencies in NARX recurrent neural networks.  _IEEE Transactions on Neural Networks_ 7, 6 (Nov. 1996), 1329–1338.  https://doi.org/10.1109/72.548162
  * Liston et al. (2023) Patrick Liston, Charles Gretton, and Artem Lensky. 2023\.  Modeling the Effect of Cascading Stop-Losses and Its Impact on Price Dynamics.  In _AI for Financial Institutions_. AAAI Bridge 2023 FinST.
  * Malik (2011) Farooq Malik. 2011\.  Estimating the impact of good news on stock market volatility.  _Applied Financial Economics_ 21, 8 (April 2011), 545–554.  https://doi.org/10.1080/09603107.2010.534063
  * Malkiel (1989) Burton G. Malkiel. 1989\.  Efficient Market Hypothesis.  In _Finance_ , John Eatwell, Murray Milgate, and Peter Newman (Eds.). Palgrave Macmillan UK, London, 127–134.  https://doi.org/10.1007/978-1-349-20213-3_13
  * Masset (2011) Philippe Masset. 2011\.  _Volatility Stylized Facts_.  SSRN Scholarly Paper ID 1804070. Social Science Research Network, Rochester, NY.  https://doi.org/10.2139/ssrn.1804070
  * Mayhew (1995) Stewart Mayhew. 1995\.  Implied Volatility.  _Financial Analysts Journal_ 51, 4 (1995), 8–20.  https://doi.org/10.2469/faj.v51.n4.1916
  * Nazlioglu et al. (2023) Saban Nazlioglu, Sevket Pazarci, Asim Kar, and Osman Varol. 2023\.  Efficient market hypothesis in emerging stock markets: gradual shifts and common factors in panel data.  _Applied Economics Letters_ (2023), 1–7.
  * Neupane et al. (2021) Biwesh Neupane, Chandra Thapa, Andrew Marshall, and Suman Neupane. 2021.  Mimicking insider trades.  _Journal of Corporate Finance_ 68 (June 2021), 101940\.  https://doi.org/10.1016/j.jcorpfin.2021.101940
  * Oliveira et al. (2017) Nuno Oliveira, Paulo Cortez, and Nelson Areal. 2017\.  The impact of microblogging data for stock market prediction: Using Twitter to predict returns, volatility, trading volume and survey sentiment indices.  _Expert Systems with Applications_ 73 (May 2017), 125–144.  https://doi.org/10.1016/j.eswa.2016.12.036
  * Oord et al. (2016) Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016.  WaveNet: A Generative Model for Raw Audio.  _arXiv:1609.03499 [cs]_ (Sept. 2016).  http://arxiv.org/abs/1609.03499
  * Orhan and Köksal (2012) Mehmet Orhan and Bülent Köksal. 2012.  A comparison of GARCH models for VaR estimation.  _Expert Systems with Applications_ 39, 3 (Feb. 2012), 3582–3592.  https://doi.org/10.1016/j.eswa.2011.09.048
  * Pandey and Wang (2019) A. Pandey and D. Wang. 2019\.  TCNN: Temporal Convolutional Neural Network for Real-time Speech Enhancement in the Time Domain. In _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 6875–6879.  https://doi.org/10.1109/ICASSP.2019.8683634 ISSN: 2379-190X.
  * Pascanu et al. (2013) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013\.  On the difficulty of training recurrent neural networks. In _Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28_ _(ICML’13)_. JMLR.org, Atlanta, GA, USA, III–1310–III–1318.
  * Patton (2011) Andrew J. Patton. 2011\.  Volatility forecast comparison using imperfect volatility proxies.  _Journal of Econometrics_ 160, 1 (Jan. 2011), 246–256.  https://doi.org/10.1016/j.jeconom.2010.03.034
  * Poon and Granger (2003) Ser-Huang Poon and Clive W J Granger. 2003.  Forecasting Volatility in Financial Markets: A Review.  _Journal of Economic Literature_ 41, 2 (2003), 478–539.
  * Ren et al. (2019) R. Ren, D. D. Wu, and T. Liu. 2019.  Forecasting Stock Market Movement Direction Using Sentiment Analysis and Support Vector Machine.  _IEEE Systems Journal_ 13, 1 (March 2019), 760–770.  https://doi.org/10.1109/JSYST.2018.2794462
  * Robertson et al. (2007) Calum Robertson, Shlomo Geva, and Rodney Wolff. 2007\.  News Aware Volatility Forecasting: Is the Content of News Important?  In _Data Mining and Analytics 2007: Proceedings of the Sixth Australasian Data Mining Conference (AusDM ’07)_ , J. Li, P. Kennedy, P. Christen, G. Williams, and I. Kolyshkina (Eds.). Australian Computer Society Inc, Australia, 157–166.  http://dl.acm.org/citation.cfm?id=1378267
  * Ruder (2016) Sebastian Ruder. 2016\.  An overview of gradient descent optimization algorithms.  _arXiv:1609.04747 [cs]_ (Sept. 2016).  http://arxiv.org/abs/1609.04747
  * Shapiro and Wilk (1965) S. S. Shapiro and M. B. Wilk. 1965.  An analysis of variance test for normality (complete samples).  _Biometrika_ 52, 3-4 (Dec. 1965), 591–611.  https://doi.org/10.1093/biomet/52.3-4.591 Publisher: Oxford University Press (OUP).
  * Shin et al. (2017) Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017\.  Continual Learning with Deep Generative Replay.  In _Advances in Neural Information Processing Systems 30_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 2990–2999.  http://papers.nips.cc/paper/6892-continual-learning-with-deep-generative-replay.pdf
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  _arXiv:1409.1556 [cs]_ (April 2015).  http://arxiv.org/abs/1409.1556 arXiv: 1409.1556.
  * Student (1908) Student. 1908\.  The probable error of a mean.  _Biometrika_ (1908), 1–25.  Publisher: JSTOR.
  * Timmermann and Granger (2004) Allan Timmermann and Clive W. J. Granger. 2004.  Efficient market hypothesis and forecasting.  _International Journal of Forecasting_ 20, 1 (Jan. 2004), 15–27.  https://doi.org/10.1016/S0169-2070(03)00012-8
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is all you need. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ _(NIPS’17)_. Curran Associates Inc., Red Hook, NY, USA, 6000–6010.
  * Vergili and Çelik (2023) Gizem Vergili and Mehmet Sinan Çelik. 2023.  The Relationship Between the Indices of Volatility (VIX) and Sustainability (DJSEMUP): An ARDL Approach.  _Business and Economics Research Journal_ 14, 1 (2023), 19–29.
  * Vidal and Kristjanpoller (2020) Andrés Vidal and Werner Kristjanpoller. 2020.  Gold volatility prediction using a CNN-LSTM approach.  _Expert Systems with Applications_ 157 (Nov. 2020), 113481\.  https://doi.org/10.1016/j.eswa.2020.113481
  * Wan et al. (2019) Renzhuo Wan, Shuping Mei, Jun Wang, Min Liu, and Fan Yang. 2019.  Multivariate Temporal Convolutional Network: A Deep Neural Networks Approach for Multivariate Time Series Forecasting.  _Electronics_ 8, 8 (Aug. 2019), 876\.  https://doi.org/10.3390/electronics8080876
  * Wang and Oates (2015) Zhiguang Wang and Tim Oates. 2015.  Encoding Time Series as Images for Visual Inspection and Classification Using Tiled Convolutional Neural Networks. In _Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence_. 8.
  * White (1990) Halbert White. 1990\.  Connectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings.  _Neural Networks_ 3, 5 (Jan. 1990), 535–549.  https://doi.org/10.1016/0893-6080(90)90004-5
  * Wu et al. (2021) Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021\.  Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In _Adv. Neural Inf. Process. Syst._ , Vol. 34. Virtual Conference, 22419–22430.
  * Wu et al. (2020) Neo Wu, Bradley Green, Xue Ben, and Shawn O’Banion. 2020\.  Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case.  _arXiv:2001.08317 [cs, stat]_ (Jan. 2020).  http://arxiv.org/abs/2001.08317
  * Zhang et al. (2023) Chao Zhang, Xingyue Pu, Mihai Cucuringu, and Xiaowen Dong. 2023\.  Graph Neural Networks for Forecasting Multivariate Realized Volatility with Spillover Effects.  arXiv:https://ssrn.com/abstract=4375165 http://dx.doi.org/10.2139/ssrn.4375165
  * Zhang et al. (2020) Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. 2020\.  Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity. In _International Conference on Learning Representations_. 21.  https://openreview.net/forum?id=BJgnXpVYwS
  * Zhang et al. (2018) Xi Zhang, Jiawei Shi, Di Wang, and Binxing Fang. 2018\.  Exploiting investors social network for stock prediction in China’s market.  _Journal of Computational Science_ 28 (Sept. 2018), 294–303.  https://doi.org/10.1016/j.jocs.2017.10.013
  * Zhu et al. (2023) Qingbo Zhu, Jialin Han, Kai Chai, and Cunsheng Zhao. 2023\.  Time Series Analysis Based on Informer Algorithms: A Survey.  _Symmetry_ 15, 4 (2023).  https://doi.org/10.3390/sym15040951

##  Appendix A Model grid search and input details

This section outlines the details of the grid search used for hyperparameter
optimisation of the models used, including the list of input variables
(LABEL:appendix:_Table_for_gridsearch).

All benchmark models used grid search as the sole method for tuning
hyperparameters (LaValle et al., 2004; Bishop, 2006), as this was the standard
in the literature. This evaluated every combination of hyperparameters within
a defined search space against the validation set, that is, trained on the
training set, and assessed with the validation set. The hyperparameter
combination that performed the best was moved on to be evaluated against the
test set, that is, trained on the union of training and validation set, and
assessed with the test set. The unmodified TCN was tested with both grid
search and BOHB to tune its set of hyperparameters, each given approximately
the same wall time. The best model from each method was assessed against the
test set, and the search method that performed best overall was used for the
remaining TCN experimentation. This strategy could not be applied to the TFT
as training was computationally expensive due to the size of the network.
Thus, grid search was used in conjunction with the Optuna hyperparameter
search method.

Table 7. Full grid search values for forecasting models Model | Hyperparameter |  Values
---|---|---
GARCH | Variant |  ARCH, GARCH, EGARCH, GJR-GARCH, GARCH with Student’s t-distribution
Lag order |  1, 2
NAR-MLP | Lag order |  1, 3, 5, 10, 15, 21
Number of hidden layers |  1, 2, 4
Neurons per hidden layer |  3, 5, 10, 21
Activation function |  ReLU, Sigmoid, TanH
Learning rate |  0.002, 0.02
Preprocessing method |  Standard scaler, min-max scaler
NARX-MLP | Lag order |  1, 3, 10, 15, 21
Number of hidden layers |  1, 2, 3, 4
Neurons per hidden layer |  5, 10, 20, 40, 100
Activation function |  ReLU, Sigmoid, TanH
Learning rate |  0.002, 0.02
Preprocessing method |  Standard scaler, min-max scaler
Additional input variables |  See Table LABEL:appendix:_table_of_model_inputs
ANN-GARCH | Variant (GARCH) |  GARCH, FIGARCH, GJR-GARCH, TARCH, GARCH with Student’s t-distribution
Lag order (GARCH) |  1, 2
Lag order (ANN) |  3, 8, 21
Number of hidden layers |  1, 2, 3
Neurons per hidden layer |  4, 5, 10, 20, 30
Activation functions |  ReLU, Sigmoid
Preprocessing method |  Standard scaler, min-max scaler
Additional input variables |  See Table LABEL:appendix:_table_of_model_inputs
CNN-LSTM | Learning rate |  0.0001, 0.001
Preprocessing method |  Standard scaler, min-max scaler
Receptive field (LSTM) |  1, 5, 10, 21, 42
TCN | Residual definition |  Difference, or log difference
Additional tasks |  Predict direction, or predict change in direction
Additional task weight |  0, 0.25, 0.5, 0.75
Number of filters |  2, 4, 5, 7
Channels per filter |  20, 25, 30, 35, 40
Kernel size |  2, 5, 7
Dropout |  0.1, 0.3, 0.5
Gradient clipping threshold |  0.2, -1 (off)
Learning rate |  0.001, 0.0001
Preprocessing method |  Standard scaler, min-max scaler
Additional input variables |  See Table LABEL:appendix:_table_of_model_inputs
| Gradient clipping threshold* |  0.1, 0.8
| Hidden size* |  16, 128
| Hidden continuous size* |  16, 128
| Number of attention heads* |  1, 8
TFT | Learning rate* |  0.005, 0.5
| Dropout* |  0.1, 0.5
| Encoder length |  10, 21, 126
| Prediction length |  1, 5
| Additional input variables |  See Table LABEL:appendix:_table_of_model_inputs
*The values for these hyperparameters indicate the lower and upper boundaries of a range, rather then the discrete values, as the hyperparameter optimisation for TFT uses Optuna.