# Comparative Analysis of Oscillating Wave Energy Converter Performance in
Southern Coast of the Caspian Sea

Erfan Amini
Coastal and Offshore Structures Engineering Group
School of Civil Engineering,University of Tehran
Tehran,Iran
<EMAIL_ADDRESS>
&Rojin Asadi
Coastal and Offshore Structures Engineering Group
School of Civil Engineering,University of Tehran
Tehran,Iran
<EMAIL_ADDRESS>
&Danial Golbaz
Coastal and Offshore Structures Engineering Group
School of Civil Engineering,University of Tehran
Tehran,Iran
<EMAIL_ADDRESS>
&Mahdie Nasiri
School of Mechanical Engineering
Iran University of Science and Technology
Tehran,Iran
<EMAIL_ADDRESS>
&Seyed Taghi Omid Naeeni
Coastal and Offshore Structures Engineering Group
School of Civil Engineering,University of Tehran
Tehran,Iran
<EMAIL_ADDRESS>
&Fereidoun Amini
School of Civil Engineering
Iran University of Science and Technology
Tehran,Iran
<EMAIL_ADDRESS>

###### Abstract

The importance of renewable energy supplies, particularly ocean wave energy,
has become undeniable globally in the last two decades. Iran is not an
exception due to the long barrier with deep water in the north and south of
the country. Because of this potential, the first step is chosen in order to
locate a suitable site for installing wave energy converters (WECs). The next
step is an applicable design for converters based on conditional and
geometrical parameters. In this study, locations with maximum potential are
chosen, and conditional parameters are used as input data to design the
oscillating converter geometrically. Next, geometric parameters are simulated
by the ABAQUS program, and for verifying it, three case study sites in the
Caspian sea are considered and designed in the WEC-Sim module in MATLAB to
evaluate conditional and geometrical parameters simultaneously. Finally, both
results are compared with each other. It is found that the Nowshahr port has
more potential than Anzali and Amirabad port, and after considering the
proposed design, the converter’s productivity raised by 63 percent. It is also
written that the flap’s optimal width is recommended to be between 18 and 23
meters. Moreover, changes in damping incident forces have a direct
relationship with the system’s efficiency, and the value of this force is more
on Nowshahr in pitch degree of freedom. Finally, according to the software
outputs, it can be concluded that the force applied to the power take-off
system behind the converter flap, which is the most crucial force in
evaluating the performance of the converter as the leading indicator, is
derived and compared between studied sites.

_K_ eywords Wave Energy Conversion $\cdot$ Wave Models $\cdot$ ABAQUS $\cdot$
WEC-Sim $\cdot$ Renewable Energy.

##  1 Introduction

The limited supply of fossil fuels, the detrimental effect of such resources
on the environment, and the growing entail of energy have led to a significant
breakthrough in the searching process for other clean and renewable energy
resources such as ocean energy. The share of these sources has increased in
recent decades, which indicates the gradual and growing acceptance of this
energy source from the industry. Concerning the more attention to harnessing
renewable energy, especially with wave converters both in the scientific and
commercial field, the world faces different converters’ expending usage.

Wave energy converters have various categories based on the construction site
[1, 2], the energy extraction system[3, 4], the way it is positioned against
the wave [5, 6], or the type of motion absorbed by the system [7, 8]. In
general, many studies have been done using optimization algorithms to find a
suitable layout or optimum distance between converters by [9, 10, 11, 12, 13].
Since the beginning of the twentieth century, most studies have been focused
on wave energy converters, especially point absorbers such as [14, 15, 16, 17,
18]. Moreover, Some research projects surveyed oscillating WECs studied by
[19, 20, 19, 21, 22]. Henry [23] evaluated the effect of different parameters
affecting OWSCs performance through a laboratory study method. In his two
models, seven parameters were assessed to stimulate the absorbing energy
system’s input power. These parameters were the wave period and the wave
power, the flap’s relative density, water depth, free-board of the flap, the
gap between the tubes, the gap underneath the flap, and flap width. The latter
parameter was one of the most critical parameters affecting the device’s
performance, which indicates an optimal limit to increase the device’s
efficiency by increasing the flap’s width. Yu et al. [24] introduced WEC-Sim
as an open-source module for modeling WECs in different wave conditions. WEC-
Sim was submitted through various platforms by validation and accurate
evaluation through numerical modeling, laboratory studies, and boundary
elements method for open-source hydrodynamic modeling[25]. In this study,
multiple parts and tools of this module are used in detail. Renzi et al. [26]
Investigated and developed the fundamental mathematical and hydrodynamic
equations to describe the base-fixed rotary oscillator’s movements. They also
discussed previous theorems’ insufficiency because the equations developed
earlier were used to describe only wave energy converters’ movement of the
point absorber type. It was also shown that the mathematical governing
equations related to the Oyster differ from the governing equations of other
terminator converters. To improve the energy absorption performance, Tom et
al. [27] changed the geometry of OWSCs’ cross-sectional shape, which was named
a rotary oscillator converter with controlled geometry. Although this research
was done by considering linear waves, the capture factor had improved. In
addition to the design loads, the hydrodynamic loads were also reduced in this
model.

While in Iran, the use of renewable energy is still much lower than fossil
fuels, the access to the sea in north and south of Iran has turned the country
into one of the potentials of the region in using ocean energy (or marine
energy). So far, various studies have been conducted on the feasibility,
design, and evaluation of the construction of wave energy converter sites in
Iran. Wind data from 1986 to 1995 for 14 sites located north and south of Iran
were taken by Zabihian, and Fung [28] to predict the wave characteristics.
According to their studies, the Chabahar site had the highest wave energy
potential. This site is located on the shores of the Oman Sea, which is
connected to the Indian Ocean, where the average wave energy is between 10 and
15 kW per meter. Besides, the Persian Gulf islands were known to have very
high wave energy potential. Since these islands are not connected to the
national grid, energy production through tidal wave energy in these islands
can be economically viable. Kamranzad et al. [29, 30, 31] surveyed 18
different sites to determine the energy distribution of waves as well as the
average and maximum annual power. This study indicates that the highest
average power was at the Govater station in the Oman Sea, which is due to the
possibility of seasonal and tropical storms and the arrival of distant ocean
waves in this region. In the Persian Gulf and the Caspian Sea, Lavan Island
and Nowshahr’s stations had the highest average power. To assign the most
suitable converter for these possible convenient locations, Soleimani et al.
[32] chose Amirabad in the Caspian sea, Asaluyeh port, and Farvar port in the
Persian Gulf. Their research revealed that submerged floating elements suit
the condition of the mentioned locations. Furthermore, the Oscillating Water
Column was chosen for the Farvar site on wave and shoreline conditions.
Moreover, the Farvar site was the best location amongst the other two for
having a maximum power output on three different converters, which were WECs
connected to the seafloor. WECs included with bottom-fixed heave buoy array
and WECs with bottom-fixed oscillating flap.

In this study, an attempt has been made to model the converter flap’s
geometric design in ABAQUS software. Then, the WEC-Sim module has been used to
simulate the performance of the converter. In this project, we have tried to
facilitate the use of the WEC-Sim modeling module by using different toolboxes
of MATLAB software such as Simulink and the solver of a multi-body dynamic;
called SimMechanics[33]. By considering the waves’ hydrodynamic properties at
each site, the feasibility of constructing an energy converter system in the
three northern ports of Iran is evaluated, and a suitable design for the
converter is presented. Finally, the output power performance has been
investigated by a comparative study. One of the main decisions for exploiting
energy from waves is selecting a suitable and adaptive converter to the chosen
location with a high-efficiency level. In 2010, Antonio and Falcão [34]
presented a comprehensive classification of wave energy converters, which were
divided into three categories: oscillating water columns, oscillating body
systems, and over-topping converters, based on how they work. It can be said
that the general technology of the developed converters while floating or
fixed, can be in one of these given categories.A comprehensive classification
with regard to new technologies is presented in Figure 1.The given figure
reveals three types of wave energy converter including over-topping [35, 36,
37], oscillating bodies [38, 39, 40], and oscillating water column [41, 42,
43]. Each of them is divided into two subsets, such as Fixed structure,
Floating, or Submerged ones.

Figure 1: The various wave energy technologies. [34]

This paper attempts to design the energy converter’s geometric parameters
according to the environmental parameters obtained for three ports (Nowshahr,
Amirabad, and Anzali) by creating a geometric model of the energy converter
for each of the three areas in ABAQUS software. Finally, we evaluate the
oscillating surge wave energy converter’s performance using geometric
parameters and environmental parameters in WEC-Sim software. Figure 2
describes the process of our study.

Figure 2: Steps of this research process.

This paper is organized as follows. Section 2 represents the research method,
which is divided into six subsections. As mentioned earlier, this paper stands
on searching for a suitable site for installing WECs in the north of Iran, so
the study area is discussed in subsection 2.1. All the details about the wave
energy converter used in this model are introduced in subsection 2.2. The
governing equations will be explained in subsection 2.3, following a
description of the Boundary Element Method (BEM) in subsection 2.5. Setting up
the simulation of this research, which is by WEC-Sim module described in
subsection 2.6, follows the converter’s details and optimal dimensions in
subsection 2.4. Section 3 presents the output of the study properly for all
three sites. Finally, in the last section, section 4, some conclusions are
drawn.

##  2 Research method

In this section, we attempted to discuss collecting the data and performing
the model’s initial requirement. Furthermore, when data is processed,
governing equations were implied as numerical modeling. Furthermore, the
boundary element method provides the initial demands of the WEC-Sim module.
Finally, a description of the geometric design provided as follows.

###  2.1 Data Collection

Based on the research of Kamranzad et al., it can be concluded that Nowshahr
port (in the southern part of the Caspian sea) has the potential to install
WECs [29][30][31]. Thus Nowshahr port is chosen for the case study of this
research. Also, Anzali port in the southwest of Caspian sea and Amirabad port
in the southeast of Caspian sea are chosen to compare results based on their
wave energy capacity [44]. The location of these three ports is shown in
figure 3.

Figure 3: The locations of three ports in the south of the Caspian Sea. [45]

An extensive set of wave data is needed for the sites of research. Therefore
the unprocessed metadata from SWAN modeling in the Caspian Sea area in the
authority of the Atlas project of Iranian waves is given from the Iranian
National Institute of Oceanography and Atmospheric Science. This dataset
included Bathymetric data of the site ($h$), a period of waves ($T_{i}$), wave
height ($H_{i}$), and direction of wave ($\alpha$) in time series of 27 years
period from 1980 to 2007 [46]. The given time-series data for each point was
per hour, which can be seen in all the model’s 2700 points. It is noted that
almost 89000 data each year were used, which in total would be near 2400000. A
code is written in MATLAB to preprocess the data to analyze and process this
dataset. Then after giving the mentioned data within the area, it returns a
related dataset. Next, data is classified in terms of location. It is saved in
a separate file to undergo the prepossessing statistical method to eliminate
the unreliable data. Moreover, the annual average of the dataset is calculated
for aggregating time series [44]. In this study, data was chosen in sectors
with a radius of 0.2 degrees around the center of each port, which the table 1
reveals the details.

Port’s Name | Minimum Longitude | Maximum longitude | Minimum Latitude | Maximum Latitude
---|---|---|---|---
Anzali | 49.26 | 49.66 | 37.44 | 37.84
Nowshahr | 36.6 | 37.00 | 51.30 | 51.70
Amirabad | 53.17 | 53.57 | 36.81 | 37.21

Table 1: Range of Longitude and Latitude for each selected site

After extracting the data from three regions, the wave height data was
converted to the significant wave height with the following equations.
Finally, their annual average for each region was calculated.

After extracting the data from three regions, wave height data ($H_{i}$) must
be transformed into significant wave height ($H_{s}$) with the following
equations [47]. Then their annual average is calculated for each location.

$H_{rms}=\sqrt{\frac{1}{N}{\sum_{i=1}^{n}H_{i}^{2}}}$ (1)

$H_{s}=\sqrt{2}H_{rms}$ (2)

###  2.2 Wave Energy Model

To absorb the waves’ energy, a device is needed to absorb the maximum energy
contained in them when the waves collide in the right direction. This device
is called a wave energy converter and consists of several parts. On the other
hand, waves have different behavioral characteristics. Therefore, the
converter should be designed and implemented according to the type of shore
and the waves’ hydrodynamic characteristics, and the energy converter’s
operating limitations.

In general, wave energy converters are divided into three categories:
attenuators, point absorbers, and terminators according to their dimensions
(compared to the wavelength of the collision) as well as their type and
location (relative to the direction of wave propagation) [48]. Converters that
are long and parallel to the direction of wave propagation are known as
attenuators. Converters perpendicular to wave propagation, with equal to or
larger dimensions than the wavelength, are included in the terminators’
category. Also, adsorbents, whose dimensions are less than one wavelength, are
known as point absorbers [34].

Several factors for choosing the type of the wave energy converter are
scrutinized, including the prospect of research [49], maintenance and
installation cost [50], and the optimal degree of freedom [51]. The final
decision is made by considering previous factors. Due to the lack of accessing
enough information, the importance of these factors is assumed to be equal. In
sum, considering all of the mention factors, Oscillating Surge WEC is the
better converter. Thus, our decision is made based on the discussed results.

###  2.3 Governing Equations

The WEC-Sim software uses Cummins’ equations in the modeling process to
calculate the motion response function of the oscillator. In 1962, Cummins
presented equations for the motion response of floats and ships against waves
at 6 degrees of freedom [52]. The main Cummins’ equation is as follows:

$m\ddot{X}=F_{ext}+F_{rad}+F_{PTO}+F_{v}+F_{m}$ (3)

Where $\ddot{X}$ is the velocity vector (either displacement or rotation) of
the device, $m$ is the mass matrix, $F_{ext}$ is the wave excitation force
vector, $F_{rad}$ is the wave radiation force vector, $F_{PTO}$ is the power
take-off force vector, $F_{v}$ is the viscous damping force vector, and
$F_{m}$ is the resulting force vector from mooring system. $F_{ext}$ and
$F_{rad}$ are calculated using the values provided by the boundary element
method (BEM) solver in the frequency range. The radiation force consists of an
added mass and a wave attenuation associated with the acceleration and
velocity of the floating body, respectively. Suppose the floating body is
constrained in the direction of wave propagation. In that case, the
hydrodynamic force of the waves affecting the body is called the wave
excitation force, which in three directions of freedom will include three
forces and three wave excitation torques [53].

WEC-Sim can be used for regular and irregular wave simulations, but $F_{ext}$
and $F_{rad}$ are calculated separately for the sinusoidal steady-state
response scenario and stochastic wave simulations. The sinusoidal steady-state
response is often used for simple wave energy converter designs with regular
input waves. For stochastic wave simulations or any other simulation in which
the effects of fluid time history are required, the use of the integral
convolution method on a floating mass is recommended [54].

The convolution integral calculation based on Cummins’ equation is used to
include the effect of fluid time history on the system. Therefore, the
radiation force can be calculated as follows [33].

$F_{rad}=-A_{\infty}\ddot{X}-\\!\\!\int_{0}^{t}K(t-\tau)\ddot{X}_{\tau}\hskip
4.26773ptd\tau$ (4)

Where $A$ is the mass-added matrix at the infinite frequency, and $K$ is the
impulse response function.
In irregular waves, the free surface height is made up of several regular wave
components linear superposition, often determined using a wave spectrum. This
spectrum describes the wave energy distribution over a wide range of wave
frequencies, characterized by wave height $(H_{s})$ and a peak period
$(T_{P})$. Irregular excitation force can be calculated as follows [54]:

$F_{ext}=\Re[R_{f}\int_{0}^{\infty}\\!\\!\\!\sqrt{2S(\omega_{r})}F_{x}e^{i(\omega_{r}t+\phi)}\hskip
4.26773ptd\omega_{r}]=\int_{-\infty}^{+\infty}\eta_{\tau}f_{e}(t-\tau)\hskip
4.26773ptd_{(\tau)}$ (5)

Where $\Re$ denotes the real part of the equation, $R_{f}$ is the ramp
function, $F_{x}$ is the excitation vector consists of amplitude and phase of
the wave, $S$ is the wave spectrum, $\phi$ is the stochastic phase angle,
$\eta_{\tau}$ represents water elevation and $f_{e}$ is the element of the
force vector. A ramp function $(R_{f})$ is necessary to avoid initial
transient oscillations in the short initial simulation times and calculate the
wave’s excitation force. if $t$ is considered as the simulation time and
$t_{r}$ is the initial oscillation adjustment time, it is better to consider
the value of $t_{r}$ equal to $25T_{P}$ for a more stable numerical solution.
According to $T_{P}$ values in the various ports under study (which are about
4 seconds), the amount of ramp time in this study is considered equal to 100
seconds [24].
The absorbed power in the power take-off mechanism is represented as a mass
damping system as follows [52]:

$P_{PTO}=(K_{PTO}X_{rel}\dot{X}_{rel}+C_{PTO}\dot{X}_{rel}^{2})=C_{PTO}\dot{X}_{rel}^{2}$
(6)

In which $K_{PTO}$ is PTO stiffness, $C_{PTO}$ is PTO damping, $X_{rel}$ and
$\dot{X}_{rel}$ are relative displacement and velocities. It should be
mentioned that the relative motion and velocity have a $\dfrac{\pi}{2}$ phase
difference, That is the reason for eliminating the stiffness sentence in the
equation.

###  2.4 Geometric and Numerical Design

Up to this moment, four programs have been used to simulate WECs called WEC-
Sim, ANSYS-AQWA, OpenFOAM, and FLOW-3D. For simulating currents around point
absorber converters, ANSYS-AQWA is the most probable choice; however, the
other three are recognized for simulating oscillating WECs in recent
literature. In FLOW-3D, if a converter in the flume has been used, water
elevation will fall gradually over time, making output data errors in the long
term. Furthermore, In the OpenFOAM, the mesh motion for fluid around the flap
does not match precisely with real motions; therefore, a cylinder of moving
mesh is entailed, and it needs to implement further codes[55]. Moreover, this
program cannot read files with .stl format, which is the format of much common
design programs[56]. Given these mentioned deficiencies, WEC-Sim seemed to be
a more useful program for this study.

WEC-Sim, which is an open-source wave energy converter simulation tool, is
financed by the U.S Department of Energy and developed by NREL (National
Renewable Energy Laboratory) and SNL (Sandia National Laboratories). This code
is implemented in MATLAB and Simulink using SimMechanics, which is the solver
of a multi-body dynamic problem [33]. WEC-Sim can simulate rigid bodies, power
take-off (PTO) systems, and mooring systems. Hydrodynamic forces are modeled
based on the diffraction and radiation method. Simultaneously, the dynamic of
a time-dependant system is considered by solving motion governing equations in
every six degrees of freedom on WEC. The diffraction and radiation method
generally computes hydrodynamic forces from a boundary element method (BEM)
solver in frequency domain [57].
According to the project, necessary changes are required for using BEM codes,
input variables, and output. Next, this module was installed due to access to
the primary source code, added to Simulink’s library. By considering the
recommendation of recent literature as well as the technical reports of Oyster
in Scotland in 2007, the height of the converter’s flap, after determining the
depth of the optimal points, will be equal to the depth plus one-meter free-
board[33, 58, 23]. Optimal thickness is presumed 1.8 meters in the designing
process, and the oscillator’s width is determined from Folley, and Wittaker’s
graph [59].

Figure 4: Determining the width of the oscillator based on the free swing
period and water depth. [59]

It is figured out from the plot that two factors are needed to find the
oscillator’s width: water depth and free swing period. The first one can be
found in the location’s parameters, and the latter can be computed from (7)
equation based on radiation relations.

$T_{N}=2\pi\sqrt{I+K_{p}}$ (7)

Where $K_{p}$ is rotational stiffness induce from the PTO system and $I$ is
the rotational moment of inertia.

To introduce the converter’s operation types to WEC-Sim software,
specifications of the converter, types of its connectors, motion restriction,
bodies’ degrees of freedom, and PTO system should be given to Simulink’s
software module. There are four steps based on the software’s manual
recommendation, and the specification of motion in the rotational oscillator
is described to the Simulink module: (i) Two rigid body blocks for
identification of the oscillator’s flap and base (ii) A global reference block
(iii) A fixed constrained block in every six degrees of freedom for the
oscillator’s base (iv) A rotational PTO block for the converter’s flap.
Finally, after supplying blocks’ connection, the final Simulink model for
oscillating surge WEC is completed. It is needed to choose the flap’s width
and height according to the relations and environmental(sea-state) parameters
to find the optimal dimensions of the oscillator’s flap. Table 2 introduces
mentioned parameters.

Table 2: Designed Geometrical parameters of the Oscillator’s Base and Flap Dimension’s Name | Amount
---|---
Width | Height
Optimal geometry of the Oscillator’s flap in Nowshahr port | 21 | 7.2
Optimal geometry of the Oscillator’s flap in Anzali port | 18 | 8.7
Optimal geometry of the Oscillator’s flap in Amirabad port | 23 | 7
Thickness the of Oscillator’s Flap ($m$) | 1.8
Width of the Oscillator’s Base ($m$) | 18
Thickness of the Oscillator’s Base ($m$) | 1.8
Height of the Oscillator’s Base ($m$) | 1.8
The distance between center of rotation and seafloor**** ($m$) | 2
Moment of Rotational Inertia (Pitch) ($kg.m^{2}$) | 1850000
Damping coefficient of PTO system ($\frac{Nsm}{rad}$) | 12000
Mass ($kg$) | 127000
Distance between center of mass and surface of the flap ($m$) | -3.9

For simulating the Oscillating wave surge converter in WEC-Sim, firstly, it
needs to design the flap and base of the oscillator in CAD programs. These
programs’ output is taken in two separate files in .stl format as a
geometrical input for WEC-Sim. ABAQUS software has been chosen to design the
geometrical parameters of the converter. It is worth noting that the
dimensions of the designed model in ABAQUS can be seen in figure 5.

Figure 5: The designed geometrical shape of OSWEC for Nowshahr port. [33]

When the described procedure is done, the measures of width and height of the
flap are obtained in the three locations of research.

Next, the base and flap were designed in ABAQUS CAE 2019 software, released by
DS SIMULIA. Due to the specification of Oyster converters installed in Orkney
shores of Scotland, the material of flap is chosen to be Aluminium with a
density of 2700 kilogram per cubic meter with uniform mass distribution. Other
parameters such as elastic modulus and Poisson’s ratio are 69 gigapascals
(GPa) and is 0.33, respectively. When the flap and base design are completed
in CATPart form, these two models are assembled. And to generate mesh on them,
surface meshing with triangular elements is chosen regarding the BEM solver.
The relative results are revealed in section 3.

###  2.5 Boundary Element Method

In this method, the governing differential equations are converted into
integral unions applied to the surface or boundary. These integrals are
numerically integrated on the boundary. In this approach, the boundary is
divided into small parts (boundary elements). Also, because all approximations
are surface-specific, the boundary element method can model areas involving
drastic changes of variables with better accuracy than the finite element
method [60]. We have used this method to perform the necessary hydrodynamic
data calculations before starting the WEC-Sim operation. The software works
first to solve the boundary element method through the open-source code,
NEMOH.

This code applies fluid environmental conditions and meshes the converter flap
and base by receiving the converter geometry file. Then, it calculates the
hydrodynamic coefficients in the frequency domain, which include the velocity
potential and the pressure field, using the boundary integral equation method
(also called the panel method). Next, the values of the added mass matrix
$(A_{\infty})$, the impact response function $(K)$, and the wave excitation
vector $(F_{x})$ are obtained [54].

The functional steps of the boundary element method codes are: Receiving NEMOH
outputs, Calculation of Impulse response functions (IRFs) for wave radiation
and excitation, Calculation of state-space realization coefficients
(calibration) for wave radiation response functions, and finally, Data storage
in the form of hierarchical data format(HDF5) [54].

###  2.6 Performance Assessment Criteria

According to the given explanations, the general modeling process by WEC-Sim
software is shown in Figure 6.

Figure 6: Numerical simulating in WEC-Sim module.(Adapted from[24])

As shown in figure 6, the converter’s specifications and geometry and
environmental conditions (in the form of wave spectrum) are given as the
software’s input. The software calculates the structure’s hydrodynamic
parameters in the pre-processing section and models the waves according to the
input parameters. In the main processing section, the structure’s response
characteristics to the collision of waves in the time domain are calculated.
Finally, in the post-processing part, the required diagrams and analyzes are
extracted. The following equations are also used to calculate the efficiency
of converters [23]:

$P_{i}=\dfrac{\mathcal{F}}{C_{g}t}$ (8)

In equation (8), $P_{i}$ is the longitudinal power of the incident waves,
$\mathcal{F}$ is the wave energy flux, and $C_{g}$ is the wave group velocity.

$\mathcal{F}=\dfrac{\rho
g\omega_{i}A_{i}^{2}}{4k_{i}}(1+\dfrac{2k_{i}d_{i}}{\sin
h(2k_{i}d_{i})})\sin\alpha_{i}$ (9)

$C_{g}=\dfrac{\omega}{k_{0}}.\dfrac{1}{2}[1+\dfrac{2k_{0}h}{\sinh(2k_{0}h)}]$
(10)

Also, $k_{0}$ is wave number, $h$ is water depth, and $H$ is the height of
incident waves.

$C_{f}=\dfrac{P_{PTO}}{P_{i}}$ (11)

In equation (11) the capture factor or total efficiency per unit length
$(C_{f})$ indicates the length of the wave crest, all of which is extracted by
the device. This parameter is obtained by dividing the absorption power by the
power of the impact waves, and its unit is in $m^{-1}$[23].

##  3 Result and Discussion

###  3.1 Interaction Implied Forces

In this part, all encountered forces and torques are obtained from the WEC-Sim
module. Based on equation(3), the considered forces are excitation force,
radiation resistance (or wave damping), PTO forces, and added mass force. Each
of these forces is calculated both in Surge and Pitch degrees of freedom. It
is worth to note that Viscous, Buoyancy forces, and mooring forces are not
modeled because of their negligible effect on the converter’s displacement. In
studying the radiation damping force of a wave, this force will have different
values for the two modes of moving the converter flap backward (collision of a
wave crest) and moving the converter flap forward (collision of a wave
through). For each port in two degrees of freedom of Pitch and Surge, software
outputs are obtained, and the values are presented in Figure 7. As this shows,
firstly, in all three ports, the radiation damping force values are larger
than the radiation damping torque. Since the converters’ rotational speed
values are less than their linear velocities [61], this seems logical.

(a) Surge (b) Pitch

Figure 7: Comparison of the wave power of radiation damping force both in
surge and pitch in all locations. (from up to down: Amirabad, Anzali, and
Nowshahr)

It can also be seen that the radiation force of the Nowshahr port converter is
much greater than the other two ports, which shows a logical trend in the
outputs due to the larger oscillations of the Nowshahr port and its higher
oscillation speed. It should note that the maximum wave power of radiation
damping force in surge and torque is around 1.42 $MN$ and 1.29 $MN.m$,
respectively. Accordingly, it can be said that comparing the radiation force
caused by the movement of the converter due to the collision of waves can be
considered as a relatively suitable criterion for comparing the performance of
the converter. In the study of the added mass forces, the two modes of
movement of the converter flap backward (collision of a wave crest) and moving
the converter flap forward (collision of a wave through) of this force will
have different values. For each port in two degrees of freedom of Pitch and
Surge, software outputs are obtained, and the values are presented in Figure
8.

(a) Surge (b) Pitch

Figure 8: Comparison of the wave torque and power of added mass force both in
surge and pitch in both directions. (from up to down: Amirabad, Anzali, and
Nowshahr)

From the outputs of WEC-Sim software, it can be seen that the amount of force
and moment of added mass in the Nowshahr port affects the converter flap more
than the two other ports. The maximum wave torque and added mass force are
approximately 5.55 $MN.m$ and 1.1 $MN$, respectively. Furthermore, the
increase in added mass force in the Amirabad port compared to the Anzali port
is due to the larger area of the converter flap in this port.
Since the proposed optimal design for Anzali port is 18 meters by 8.7 meters
and the Amirabad port is 23 meters by 7 meters, the force and moment of added
mass (in a resistant form) in Amirabad port has been achieved more than Anzali
port, which indicates that the presence of added mass or moment force on the
converter flap is not directly related to the converter’s capture factor and
should be considered along with other parameters to evaluate the performance
of the converter. Besides, the presence of a more amount of force or moment of
added mass on a converter flap can not be considered a reduction in its
performance.

The two modes of movement of the converter’s flap have been shown to evaluate
the obtained torque outputs from the power take-off system. Since the power
take-off system is practically activated in an oscillating motion, for each
port in the pitch degree of freedom, the software outputs are obtained, and
the values are presented in Figure9.

Figure 9: Comparison of the wave torque of PTO force in all locations. (from
up to down: Amirabad, Anzali, and Nowshahr)

As shown in the software outputs, the torque amount applied to the energy
converter in the Nowshahr port is more than Anzali, and Anzali port is more
than Amirabad port. Therefore, it can be expected that this parameter,
considering the relationships expressed in the previous chapter, directly
affects the efficiency of the converter. Furthermore, according to the
transfer rate outputs, it can be expected that more power has been extracted
from the Nowshahr port energy converter than the other two ports.

Next, the total torque value, which is the sum of all the forces acting on the
converter (mentioned in Equation (3)), is evaluated and compared in the
converters considered in the three ports. Figure 10 illustrates the total
applied torque values of the converters.

(a) Nowshahr port (b) Anzali port (c) Amirabad port

Figure 10: Total applied torque to converters in studied ports.

As shown in the torque application diagrams, the torque amount applied to the
converter plate in Anzali and Amirabad ports is more than the Nowshahr port.
However, it should be noted that merely increasing the amount of torque
applied to the converter flap will not increase the energy extracted. The
displacement of the converter flap must be considered in conjunction with
other environmental conditions. The maximum values of the actuator torque
affecting the pressure center of the converter flap (when moving under
pressure due to the impact of the wave crest) in Nowshahr, Anzali, and
Amirabad ports will be 832 kN, 630 kN, and 621 kN, respectively, in each time
step. Also, the maximum values of return torque affecting the pressure center
of the transducer plate (during the return movement under reduced pressure due
to the wave through impact) in Nowshahr, Anzali, and Amirabad ports are equal
to 881 kN, 598 kN, and 588 kN, respectively, in each time step.

###  3.2 Comparative Results of Capture Factor

According to the application of irregular shortwave theory, the capture
factor’s value will change at each time step according to the proposed
equations. For this purpose, the capture factor’s values in all three ports
are calculated per time steps in a total run-time of 400 seconds and compared
in Figure 11.

Figure 11: Comparison of capture factor in all locations.

A comparison of the capture factor’s values in the form of the above diagram
indicates that this quantity in Nowshahr port is more than the other two
ports. It is worth mentioning that all these values have been calculated at
points of each port with the maximum impact wave energy. The exact values of
the capture factor in all three ports, the total energy values of the waves,
and the energy absorbed by the converter in each of the ports are examined in
the table 3.

Port’s Name | Capture Factor (%) | Incident wave Energy ($Kw$) | Absorbed Energy ($Kw$)
---|---|---|---
Nowshahr | 62.99 | 5376.66 | 3358.76
Anzali | 48.84 | 4471.66 | 2183.96
Amirabad | 42.17 | 4116.66 | 1735.98
Table 3: The results of absorbed power, capture factor and incident wave power
in studied sites.

It can be said that in the studied ports, Nowshahr port has the priority of
constructing a wave energy converter system, considering the comparison of the
capture factor values. Then, Anzali and Amirabad’s feasibility on the southern
shores of the Caspian Sea can be studied.

Simultaneous consideration of converter design parameters, including
environmental and geometric quantities, can help achieve maximum system power
and significant progress in using this type of energy in Iran’s marine
industry in the future. Concerning the WEC-Sim module’s suggestion to take 1,
2, 1.8, and 2.3 meters of transverse distance between elements in the $Y$ axis
in Nowshaher, Anzali, and Amirabad ports, respectively.

###  3.3 Flap Response

In this section, the amount of flap displacement or rotation responses to the
implied is investigated in Pitch and Surge degrees of freedom. It must be
noted that the total run-time is 400 second and the ramp time is 100 second,
and the time step is chosen 0.1 regarding the Courant number [54]. The amount
of fluctuation, which can be seen in figure 12 for Surge and Pitch, is given
in meter and radian, respectively.

(a) Nowshahr port (b) Anzali port (c) Amirabad port

Figure 12: The amount of displacement and rotation of the flap in all
locations

It is clear that oscillation in Nowshahr is by far higher than the other
sites, and this difference is also witnessed in the displacement of the flap
and the angle of rotation. The maximum force of the flap in the Nowshahr port
(during the collision of wave crest) is 2.624 meters with a 0.54-radiant
maximum rotation. These measures for Anzali and Amirabad ports are 1.78, 1.64
meters, and 0.34, 0.32 radiant, respectively. Furthermore, when the flap moves
because of wave trough, the maximum amount of the flap’s displacement is 2.96,
2.12, and 2.04 meters. The maximum angle of rotation is 0.619, 0.42, and 0.4
radiant in Nowshahr, Anzali, and Amirabad. Finally, the flap’s rotational
velocity is 0.025, 0.02, 0.021 radian per second in Nowshahr, Anzali, and
Amirabad, accordingly.

##  4 Conclusions

Firstly, the oscillator surge converter chose to expand the horizon of
development, installing and maintenance cost, converter movement ability, and
the converter industry’s popularity. Next, designing geometrical parameters of
the converter depend on environmental(sea-state) conditions. Thus, the
research locations are chosen carefully, then their parameters are assessed
for implementing a model to extract the most potential energy. Thirdly, the
WEC-Sim module seems to be a better choice than its rivals because it takes
the standard formats of geometrical and environmental files. It is an open-
source and user-friendly module in MATLAB software where itself is one of the
popular ones for implementing codes. According to the literature review,
Nowshahr, Anzali, and Amirabad are chosen to install an oscillating wave surge
converter(OWSC) in the Caspian sea. Due to the broad set of data in the
Caspian Sea, an ocean metadata searching algorithm is written to find desired
data. The height of the designed converter’s flap is from 7 to 8.7 meters, and
the width is between 18 and 23 meters. The converter’s fluctuation and applied
torque show that the flap’s total used torque is high when more wave height is
observed in the installation site. It can be said that total power
encountering the oscillator’s flap, together with the capture factor, can be
considered a determinative parameter for revealing the system’s efficiency.
Finally, the simulation results indicate that the Nowshahr port parameters
have the maximum potential for harnessing energy by oscillator wave surge
converter with an efficiency of 63 percent and with the highest absorption
coefficient.

## Acknowledgement

The authors would like to express their gratitude to the Iranian National
Institute for Oceanography and Atmospheric Science (INIO), which provided the
data needed for performing this project. Also, we shall more specifically
appreciate the support of Dr. Mehdi Neshat, who was of great help, valuable
comments, and useful suggestions in this research. This work has been
supported by the Iran Port and Maritime Organization under the grant number
TE178.

## References

  * [1] Phelype Haron Oleinik, Thaísa Beloti Trombetta, Ricardo Cardoso Guimarães, Eduardo de Paula Kirinus, and Wiliam Correa Marques.  Comparative study of the influence of a wave energy converter site on the wave field of laguna, sc, brazil.  Sustainable Energy Technologies and Assessments, 31:262–272, 2019\.
  * [2] Liguo Wang, Jan Isberg, and Elisabetta Tedeschi.  Review of control strategies for wave energy conversion systems and their validation: the wave-to-wire approach.  Renewable and Sustainable Energy Reviews, 81:366–379, 2018.
  * [3] Mehdi Neshat, Bradley Alexander, and Markus Wagner.  A hybrid cooperative co-evolution algorithm framework for optimising power take off and placements of wave energy converters.  Information Sciences, 534:218–244, 2020.
  * [4] Paula B Garcia-Rosa, Giorgio Bacelli, and John V Ringwood.  Control-informed optimal array layout for wave farms.  IEEE Transactions on Sustainable Energy, 6(2):575–582, 2015.
  * [5] Alan Henry, Olivier Kimmoun, Jonathan Nicholson, Guillaume Dupont, Yanji Wei, Frederic Dias, et al.  A two dimensional experimental investigation of slamming of an oscillating wave surge converter.  International Society of Offshore and Polar Engineers, 2014.
  * [6] Yanji Wei, Thomas Abadie, Alan Henry, and Frederic Dias.  Wave interaction with an oscillating wave surge converter. part ii: Slamming.  Ocean Engineering, 113:319–334, 2016.
  * [7] Gianmaria Giannini, Paulo Rosa-Santos, Victor Ramos, and Francisco Taveira-Pinto.  On the development of an offshore version of the ceco wave energy converter.  Energies, 13(5):1036, 2020.
  * [8] Claudio A Rodríguez, Paulo Rosa-Santos, and Francisco Taveira-Pinto.  Hydrodynamic optimization of the geometry of a sloped-motion wave energy converter.  Ocean Engineering, 199:107046, 2020.
  * [9] Mehdi Neshat, Bradley Alexander, Markus Wagner, and Yuanzhong Xia.  A detailed comparison of meta-heuristic methods for optimising wave energy converter placements.  In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’18, page 1318–1325, New York, NY, USA, 2018. Association for Computing Machinery.
  * [10] Mehdi Neshat, Bradley Alexander, Nataliia Y Sergiienko, and Markus Wagner.  New insights into position optimisation of wave energy converters using hybrid local search.  Swarm and Evolutionary Computation, 59:100744, 2020.
  * [11] Mehdi Neshat, Ehsan Abbasnejad, Qinfeng Shi, Bradley Alexander, and Markus Wagner.  Adaptive neuro-surrogate-based optimisation method for wave energy converters placement optimisation.  pages 353–366. Springer, 2019.
  * [12] Mehdi Neshat, Nataliia Y Sergiienko, Erfan Amini, Meysam Majidi Nezhad, Davide Astiaso Garcia, Bradley Alexander, and Markus Wagner.  A new bi-level optimisation framework for optimising a multi-mode wave energy converter design: A case study for the marettimo island, mediterranean sea.  Energies, 13(20):5498, 2020.
  * [13] Erfan Amini, Danial Golbaz, Fereidoun Amini, Meysam Majidi Nezhad, Mehdi Neshat, and Davide Astiaso Garcia.  A parametric study of wave energy converter layouts in real wave models.  Energies, 13(22):6095, 2020.
  * [14] Motohiko Murai, Qiao Li, and Junki Funada.  Study on power generation of single point absorber wave energy converters (pa-wecs) and arrays of pa-wecs.  Renewable Energy, 2020.
  * [15] Rezvan Alamian, Rouzbeh Shafaghat, and Mohammad Reza Safaei.  Multi-objective optimization of a pitch point absorber wave energy converter.  Water, 11(5):969, 2019.
  * [16] Tunde Aderinto and Hua Li.  Conceptual design and simulation of a self-adjustable heaving point absorber based wave energy converter.  Energies, 13(8):1997, 2020.
  * [17] Erfan Amini, Danial Golbaz, Fereidoun Amini, Meysam Majidinezhad, Mehdi Neshat, and David Astiaso Garcia.  A parametric study of wave energy converter layouts in real wave models.  Energies, 13:6095, 2020.
  * [18] Mehdi Neshat, Bradley Alexander, Nataliia Y. Sergiienko, and Markus Wagner.  Optimisation of large wave farms using a multi-strategy evolutionary framework.  GECCO ’20, page 1150–1158, New York, NY, USA, 2020. Association for Computing Machinery.
  * [19] Zhenqing Liu, Yize Wang, and Xugang Hua.  Prediction and optimization of oscillating wave surge converter using machine learning techniques.  Energy Conversion and Management, 210:112677, 2020.
  * [20] Erfan Amini.  Locating and evaluating the oscillating surge wave energy converter using grey wolf optimizer algorithm and wec-sim toolbox.  Master thesis, University of Tehran, 2019.
  * [21] Kelley Ruehl, Dominic D Forbush, Yi-Hsiang Yu, and Nathan Tom.  Experimental and numerical comparisons of a dual-flap floating oscillating surge wave energy converter in regular waves.  Ocean Engineering, 196:106575, 2020.
  * [22] N Tran, NY Sergiienko, BS Cazzolato, B Ding, MH Ghayesh, and M Arjomandi.  The impact of pitch-surge coupling on the performance of a submerged cylindrical wave energy converter.  Applied Ocean Research, 104:102377, 2020.
  * [23] AJ Henry.  The hydrodynamics of small seabed mounted bottom hinged wave energy conerverters in shallow water.  PhD thesis, Queen’s University of Belfast, 2009.
  * [24] Michael Lawson, Yi-Hsiang Yu, Kelley Ruehl, Carlos Michelen, et al.  Development and demonstration of the wec-sim wave energy converter simulation tool.  2014\.
  * [25] Kelley Ruehl, Carlos Michelen, Bret Bosma, and Yi-Hsiang Yu.  Wec-sim phase 1 validation testing: Numerical modeling of experiments.  In International Conference on Offshore Mechanics and Arctic Engineering, volume 49972, page V006T09A026. American Society of Mechanical Engineers, 2016.
  * [26] Emiliano Renzi, Kenneth Doherty, Alan Henry, and Frederic Dias.  How does oyster work? the simple interpretation of oyster mathematics.  European Journal of Mechanics-B/Fluids, 47:124–131, 2014.
  * [27] Nathan Tom, Michael Lawson, Yi-Hsiang Yu, and Alan Wright.  Preliminary analysis of an oscillating surge wave energy converter with controlled geometry.  Technical report, NREL (National Renewable Energy Laboratory (NREL), 2015\.
  * [28] Yonghong Kuang, Yongjun Zhang, Bin Zhou, Canbing Li, Yijia Cao, Lijuan Li, and Long Zeng.  A review of renewable energy utilization in islands.  Renewable and Sustainable Energy Reviews, 59:504–513, 2016.
  * [29] Bahareh Kamranzad, Amir Etemad-Shahidi, and Vahid Chegini.  Sustainability of wave energy resources in southern caspian sea.  Energy, 97:549–559, 2016.
  * [30] Bahareh Kamranzad, Amir Etemad-Shahidi, and Vahid Chegini.  Developing an optimum hotspot identifier for wave energy extracting in the northern persian gulf.  Renewable Energy, 114:59–71, 2017.
  * [31] Bahareh Kamranzad, Vahid Chegini, and Amir Etemad-Shahidi.  Temporal-spatial variation of wave energy and nearshore hotspots in the gulf of oman based on locally generated wind waves.  Renewable Energy, 94:341–352, 2016.
  * [32] Kaveh Soleimani, Mohammad Javad Ketabdari, and Farzan Khorasani.  Feasibility study on tidal and wave energy conversion in iranian seas.  Sustainable Energy Technologies and Assessments, 11:77–86, 2015\.
  * [33] Yi-Hsiang Yu, Ye Li, Kathleen Hallett, and Chad Hotimsky.  Design and analysis for a floating oscillating surge wave energy converter.  In International Conference on Offshore Mechanics and Arctic Engineering, volume 45547, page V09BT09A048. American Society of Mechanical Engineers, 2014.
  * [34] F de O Antonio.  Wave energy utilization: A review of the technologies.  Renewable and sustainable energy reviews, 14(3):899–918, 2010.
  * [35] Tunde Aderinto and Hua Li.  Ocean wave energy converters: Status and challenges.  Energies, 11(5):1250, 2018.
  * [36] L Margheritini, P Frigaard, and G Iglesias.  Technological and commercial comparison of owc and ssg wave energy converters built into breakwaters.  page 167. CRC Press, 2020.
  * [37] Mohammad Yaghoub Abdollahzadeh Jamalabadi and Eisa Ahmadi.  Design, construction and testing of a dragon wave energy converter.
  * [38] Giuseppe Giorgi, Rui PF Gomes, João CC Henriques, Luís MC Gato, Giovanni Bracco, and Giuliana Mattiazzo.  Detecting parametric resonance in a floating oscillating water column device for wave energy conversion: Numerical simulations and validation with physical model tests.  Applied Energy, 276:115421, 2020.
  * [39] Markel Penalba, Thomas Kelly, and John Ringwood.  Using nemoh for modelling wave energy converters: A comparative study with wamit.  2017\.
  * [40] Xiaofan Li, ChienAn Chen, Qiaofeng Li, Lin Xu, Changwei Liang, Khai Ngo, Robert G Parker, and Lei Zuo.  A compact mechanical power take-off for wave energy converters: Design, analysis, and test verification.  Applied Energy, 278:115459, 2020.
  * [41] António FO Falcão and Joao CC Henriques.  Oscillating-water-column wave energy converters and air turbines: A review.  Renewable Energy, 85:1391–1424, 2016.
  * [42] António FO Falcão and João CC Henriques.  Model-prototype similarity of oscillating-water-column wave energy converters.  International Journal of Marine Energy, 6:18–34, 2014.
  * [43] Helen Bailey, Bryson RD Robertson, and Bradley J Buckham.  Wave-to-wire simulation of a floating oscillating water column wave energy converter.  Ocean Engineering, 125:248–260, 2016.
  * [44] Erfan Amini, Sayyed Taghi Omid Naeeni, and Pedram Ghaderi.  Investigating wave energy potential in southern coasts of the caspian sea and evaluating the application of gray wolf optimizer algorithm.  arXiv preprint arXiv:1912.13201, 2019.
  * [45] REUK.  Oyster wave power.  http://www.reuk.co.uk/, 2009.
  * [46] Aliasghar Golshani, Soheila Taebi, and Vahid Chegini.  Wave hindcast and extreme value analysis for the southern part of the caspian sea.  Coastal engineering journal, 49(04):443–459, 2007.
  * [47] Robert George Dean and Robert A Dalrymple.  Water wave mechanics for engineers and scientists.  In Unknown Host Publication Title. Prentice-Hall Inc, 1984.
  * [48] Aurélien Babarit.  Ocean wave energy conversion: resource, technologies and performance, 2017\.
  * [49] Iraide López, Jon Andreu, Salvador Ceballos, Iñigo Martínez De Alegría, and Iñigo Kortabarria.  Review of wave energy technologies and the necessary power-equipment.  Renewable and sustainable energy reviews, 27:413–434, 2013.
  * [50] Hamid Sarlak Chivaee.  Design and construction of sea wave energy absorbers.  Master’s thesis, Sharif University of Technology, Tehran, 2009.
  * [51] Aure´ lien Babarit, Jorgen Hals, Adi Kurniawan, Torgeir Moan, and Jorgen Krokstad.  Power absorption measures and comparisons of selected wave energy converters.  volume 44373, pages 437–446, 2011.
  * [52] WE Cummins.  The impulse response function and ship motions (no. dtmb-1661).  David Taylor Model Basin Washington DC, 1962.
  * [53] Johannes Falnes and Adi Kurniawan.  Ocean waves and oscillating systems: linear interactions including wave-energy extraction, 2020.
  * [54] Michael Lawson, Yi-Hsiang Yu, Adam Nelessen, Kelley Ruehl, and Carlos Michelen.  Implementing nonlinear buoyancy and excitation forces in the wec-sim wave energy converter modeling tool.  volume 45547, page V09BT09A043. American Society of Mechanical Engineers, 2014.
  * [55] Pál Schmitt and Björn Elsaesser.  On the use of openfoam to model oscillating wave surge converters.  Ocean Engineering, 108:98–104, 2015.
  * [56] Zahra Mozhgani.  Modeling the flow around energy absorbers under the impact of regular waves.  Master’s thesis, Amirkabir University of Technology, Tehran, 2017.
  * [57] Ye Li and Yi-Hsiang Yu.  A synthesis of numerical methods for modeling wave energy converter-point absorbers.  Renewable and Sustainable Energy Reviews, 16(6):4352–4364, 2012\.
  * [58] Alan Henry, Kenneth Doherty, L Cameron, Trevor Whittaker, and Rina Doherty.  Advances in the design of the oyster wave energy converter.  RINA Marine and Offshore Renewable Energy, London, UK, 2010.
  * [59] M Folley, TJT Whittaker, and Alan Henry.  The effect of water depth on the performance of a small surging wave energy converter.  Ocean Engineering, 34(8-9):1265–1274, 2007.
  * [60] Heinz Antes.  A short course on boundary element methods.  2010\.
  * [61] Liguo Wang, Jan Isberg, and Elisabetta Tedeschi.  Review of control strategies for wave energy conversion systems and their validation: the wave-to-wire approach.  Renewable and Sustainable Energy Reviews, 81:366–379, 2018.


# A Comparison of LSTMs and Attention Mechanisms for Forecasting Financial
Time Series

S.E. Yi*
Department of Computer Science
University of Toronto
Toronto, ON M5S 3H7
<EMAIL_ADDRESS>
&A. Viscardi*
Department of Computer Science
University of Toronto
Toronto, ON M5S 3H7
<EMAIL_ADDRESS>
&T. Hollis*
Department of Computer Science
University of Toronto
Toronto, ON M5S 3H7
<EMAIL_ADDRESS>

###### Abstract

While LSTMs show increasingly promising results for forecasting Financial Time
Series (FTS), this paper seeks to assess if attention mechanisms can further
improve performance. The hypothesis is that attention can help prevent long-
term dependencies experienced by LSTM models. To test this hypothesis, the
main contribution of this paper is the implementation of an LSTM with
attention. Both the benchmark LSTM and the LSTM with attention were compared
and both achieved reasonable performances of up to 60% on five stocks from
Kaggle’s Two Sigma dataset. This comparative analysis demonstrates that an
LSTM with attention can indeed outperform standalone LSTMs but further
investigation is required as issues do arise with such model architectures.

##  1 Introduction

Financial Time Series (FTS) modelling is a practice with a long history which
first revolutionised algorithmic trading in the early 1970s. The analysis of
FTS was divided into two categories: fundamental analysis and technical
analysis. Fundamental analysis is the study of a stock or currency’s price
based on economic factors. These include the overall state of the business or
economy, revenue, interest rates and others. On the other hand, technical
analysis, as defined by J. Murphy in (26), is the study of market action to
forecast future trends. This is achieved through the analysis of shorter term
financial data, primarily price and volume. Both fundamental and technical
analysis are put into question by the Efficient Market Hypothesis (EMH). The
EMH, highly disputed since its initial publication in 1970, hypothesizes that
stock prices are ultimately unpredictable (25). This has not stopped research
attempting to model FTS through the use of linear, non-linear and ML-based
models, as mentioned hereafter. Together these approaches form the main
subcategories of existing solutions in FTS analysis.

In parallel, the origins of attention mechanisms initially came into
prominence in the field of Computer Vision (CV). These new models were
originally loosely inspired by human visual attention mechanisms present in a
region of the prefrontal cortex known as the inferior frontal junction.
Applications were later leveraged to tackle issues with context-dependent
inference in Natural Language Processing (NLP). In both cases, the core
principle of attention is achieved by setting attention weights to assign more
or less of the algorithm’s finite attention on different subsets of the input
feature space. In CV, this corresponds to focussing on particular features of
the input image while in NLP this represents focus on particular words in the
input sentence. In NLP, this attention mechanism allows inference to be made
about an entire sentence while remaining sensitive to its context. This
remains to this day a particularly challenging task for long sentences.

In this paper we propose a novel approach to FTS forecasting by combining
these two fields of research. The idea is to leverage the developments in
attention mechanisms to improve the performance of promising LSTM RNN
architectures currently in use for FTS forecasting. The main contribution in
this paper is the implementation of an LSTM that uses attention for parsing
both news headlines and financial data. The performance of this model is then
compared with that of a regular LSTM without attention. This performance is
then evaluated using five stocks from Kaggle’s Two Sigma dataset (17) and
using various methods of data preprocessing (3).

The ultimate goals of FTS forecasting, among others, is to help solve the
problem of volatility in speculative markets and to help foresee large
financial events such as the 2009 financial crisis to ensure better economic
preparation.

##  2 Related Work

As discussed in (16), the most rudimentary approach to modelling FTS is by
assuming that they follow the random walk model. The random walk model can be
simply expressed as a sum of a series of independent random variables (12). By
weighing these, the first Auto-Regressive (AR) model was developed. A set of
equations were developed by U. Yule and G. Walker in (29) to provide
quantitative methods for estimating parameters in AR models. This work was
subsequently expanded upon by J. P. Burg in (5) who provided an alternative
approach albeit with different stability properties. These AR models are often
accompanied by another type of linear model, the Moving Average (MA) model
which gives the Auto-Regressive Moving Average (ARMA) model. However, a
fundamental limitation of AR, MA and ARMA models is that they all assume the
process being modelled is stationary. Stationarity is a property of processes
whereby the probability distribution remains constant over time, thus variance
also remains constant. Indeed, this assumption is significant as FTS are often
non-stationary processes. Therefore, this model’s accuracy will suffer,
highlighting the need to take this problem of stationarity into consideration.
This is done by generalising the ARMA model into the Autoregressive Integrated
Moving Average (ARIMA) model (12). The ARIMA model solves the issue of non-
stationarity by exploiting the concept of returns (or degrees of
differencing). Non-stationary time series can therefore be made stationary by
differencing. The aforementioned linear models all suffer from the assumption
that FTS are homoscedastic processes. This is indeed often a poor assumption
to make, as shown in (8) by R.F. Engle. In (8), Engle states that by using a
more sophisticated model such as the Auto-Regressive Conditional
Heteroscedasticity (ARCH) model, the homoscedastic assumption can be avoided.
This ARCH model was later described by Bollerslev in (4) as a special case of
a more generalised model called the Generalised Auto-Regressive Conditional
Heteroscedasticity (GARCH) model. Many more variants of the GARCH model have
been published since its original publication in 1986. These include NAGARCH
(nonlinear asymmetric GARCH) (9), EGARCH (exponential GARCH) (27), GJR-GARCH
(Glosten-Jagannathan-Runkle GARCH) (13) and many others. These GARCH
derivatives are often nested under Hentschel’s fGARCH (Family GARCH) model
(14) but these all lie outside the scope of this paper. In the same time as
the ARCH and GARCH models, J. Leontaris and S. A. Billings published an
alternative in (7) known as is the Nonlinear Autoregressive Moving Average
model with exogenous inputs (NARMAX). This work, building on their own
previous work on ARMAX models, demonstrated that NARMAX models can
successfully be applied to model complex time series. More information on
these models can be found in (16), including equations and further
explanation.

These state space and stochastic models were however quickly overwhelmed by
advances in Machine Learning. A wave of ML approaches to modelling FTS
severely disrupted the field of algorithmic trading via stochastic modelling
in the last two decades. One of the earliest approaches to FTS forecasting
using ML built on work from Kohonen in (20). In (20), Kohonen introduced the
idea of Self-Organising Maps (SOM) which were subsequently successfully
applied to FTS forecasting (21). In 2003, still in the early days of ML for
FTS predictions, SVMs (both linear and non-linear) were shown by Kim in (18)
to be of significant predictive capabilities for FTS. In parallel, Zhang
showed in his 2003 paper (31) that, by combining Artificial Neural Networks
(ANNs) with the aforementioned ARIMA model, promising FTS forecasting can be
achieved.

Nevertheless, it was only by benefitting from the Neural Network boom of 2012
brought on by the AlexNet work of Krizhevsky, Sutskever and Hinton on the
ImageNet competition (22), that ANNs became some of the most mainstream
methods for FTS forecasting, in particular with the rise of RNNs (6). However,
another type of neural network that has also been widely lauded for its
performance in FTS forecasting is the Echo State Network (ESN). Indeed, Lin
et. al showed in (24) that ESNs combined with Principal Component Analysis
(PCA) can sometimes exceed or at least match the performance of conventional
RNNs while decreasing the computational costs. This is due to the techniques
of Reservoir Computing introduced in ESNs. In short, ESNs can bypass the issue
of the vanishing gradient problem and long computational times, present in
conventional RNNs, by creating a large ‘reservoir’ of sparsely connected
neurons. The connections are assigned at random and weights within the
reservoir do not get conventionally trained, reducing computational time and
allowing the network to echo past states (emulating the ‘memory’ of RNNs).

Another alternative to RNNs is the Deep Belief Network (DBN). Hinton and
Salakhutdinov’s DBNs are a type of probabilistic generative neural network
composed of layers of Restricted Boltzmann Machines (RBMs). These have also
successfully been leveraged for accurate time series prediction (23).

In the modern day however, LSTM RNNs remain amongst some of the most popular
and promising models for predicting FTS. While LSTMs originally came to light
in the seminal 1997 paper (15) by Hochreiter and Schmidhuber, they only
recently rose to prominence for FTS forecasting. Amongst the most successful
LSTM implementations, the pioneering paper by Bao et al. in (2) implements a
variety of the most modern LSTM architectures coupled with autoencoders and
applies them to stock data. The work presented here extends and builds on the
insight of this paper by exploring the impact of leveraging attention models
for sentiment analysis built on top of LSTMs.

However, a word of caution is worth mentioning here. It is true that academic
publications in the field of FTS forecasting are often misleading. Indeed,
many of the most performant models are developed by private companies and kept
away from the public, with the utmost secrecy for competitive reasons.
Academia seems to be struggling to shed light on the most modern techniques
which is one of the prime motivations for the investigation presented
hereafter. In addition, many FTS forecasting papers tend to inflate their
performance for recognition and overfit their models due to the heavy use
simulators. Many of the performances claimed in these papers are difficult to
replicate as they fail to generalise for future changes in the particular FTS
being forecast.

Having reviewed the field of FTS forecasting, in order to better situate our
paper amongst existing literature, it is important to now cover a brief
history of attention mechanisms.

Many early prominent papers using attention mechanisms in NLP initially used
the term “alignment” to refer to attention in the context neural machine
translation. One of the most foundational of these is the 2014 Bahdanau, Cho
and Bengio collaboration in (1). In (1), Bahdanau shows that a single neural
network can be jointly tuned to maximise translation performance using
attention-based encoder-decoders. In parallel, Graves, Wayne and Danihelka
showed in (11) that neural networks (in particular LSTMs) can be improved by
coupling them to attention processes. While the application in (11) was
helping Neural Turing Machines infer copying, sorting and recall, subsequent
CV and NLP applications of the very same concept were greatly inspired by
techniques presented in this paper (30). Building on the work above as well as
on their own work, in 2015 Firat, Cho and Bengio in (10) had a breakthrough in
multilingual Neural Machine Translation (NMT). They used single attention
mechanisms shared across language pairs to outperform existing NMT benchmarks.

Since then, a whole flurry of different types of attention have been developed
including but not limited to self-attention, shared attention, local
attention, hybrid attention and multi-headed attention. Indeed, the multi-
headed attention model presented in the Transformer (28) by Google Brain and
UofT alumni has been widely lauded as a promising novel architecture for
dealing with long range dependencies without the issues of LSTMs in NMT. For
this reason, that paper will serve as inspiration for our investigation on the
potential of attention in FTS forecasting.

From the history of attention mechanisms presented here we can indeed see that
attention mechanisms are currently accepted as a very promising approach to
many problems, especially in the field of machine translation. However, we
have yet to see if they present any promise in sentiment analysis for FTS
forecasting which is the purpose of the work presented here. Intuitively,
since attention mechanisms are designed to help address the issue of context
dependency this suggests promising potential for sentiment analysis in FTS
forecasting. Indeed, most sentiment analysis algorithms would fail to consider
the impact of such context for long lengths. A naive example of this, such as
“I am a compulsive liar but this company is fantastic, the stock is destined
to rise and I don’t understand why more people have not invested yet”, would
be wrongly detected as a positive sentiment by most algorithms. The following
investigation aims to confirm or refute the hypothesis that LSTM performance
can be improved with attention.

##  3 Model Architecture

The two models investigated in this paper are a vanilla LSTM (as a benchmark)
and an LSTM with an attention mechanism. A diagram of these models is shown in
figure 1 below.

Figure 1: System diagram of the LSTM and LSTM with attention

Both the LSTM model and the LSTM with attention model used in this paper are
implemented with mean squared error loss using an Adam optimiser (19). It is
nonetheless important to cover the mathematical foundations of these models
before comparing their performances.

In encoder-decoder RNNs, the encoder reads an input sequence
$\textbf{x}=(x_{1},$ … $,x_{t-1})$ into a vector $\textbf{c}=(c_{1},$ …
$,c_{t-1})$, as shown in figure 1. A common approach and the one we will be
following in this paper is to use an RNN such that (1):

$\displaystyle h_{t}=f(x_{t},h_{t-1})$ (1)

and

$\displaystyle\textbf{c}=q(\\{h_{0},\dots,h_{T_{x}}\\})$ (2)

where $h_{t}$ is a hidden state at time $t$, c is the context vector generated
by the hidden states and $f$ and $q$ are non-linear functions.

Subsequently to equations 1 and 2, prediction is done at the decoder by
defining a probability over the translation y through the following
decomposition (1):

$\displaystyle
p(\textbf{y})=\prod_{t=1}^{T}p(y_{t}|\\{y_{1},\dots,y_{t-1}\\},c)$ (3)

where $\textbf{y}=(y_{1},\dots,y_{T_{y}})$. For RNNs, each conditional
probability is modelled as:

$\displaystyle p(y_{t}|\\{y_{1},\dots,y_{t-1}\\},c)=g(y_{t-1},s_{t},c)$ (4)

where $s_{t}$ is the RNN’s hidden state and $g$ is a nonlinear function that
outputs the probability of $y_{t}$.

In attention, a particular context $c_{i}$ depends on a sequence of
annotations ($h_{1},\dots,h_{T_{x}}$) to which an encoder maps the input
sequence. While each annotation $h_{i}$ contains information about the input
sequence, we want to focus attention on a particular part of the input. Thus
the context vector is computed as a weighted sum as follows (1):

$\displaystyle c_{i}=\sum_{j=1}^{T_{x}}\alpha_{ij}h_{j}$ (5)

where each context weight $\alpha_{ij}$ for each annotation $h_{j}$ is
calculated as follows:

$\displaystyle\alpha_{ij}=\frac{\exp(e_{ij})}{\sum_{k=1}^{T_{x}}\exp(e_{ik})}$
(6)

where

$\displaystyle e_{ik}=a(s_{i-1},h_{j})$ (7)

is known as the alignment model. Alignment models score how well matched the
input at position $j$ and output at position $i$ are.

Equations 1 through 6 are used for the LSTM and attention mechanisms while
Adam is used as the optimiser and Mean Square Error (MSE) is used as the loss
function. Mathematically, MSE loss is as follows:

$\displaystyle MSE=\frac{1}{n}\sum_{t=1}^{n}(y-t)^{2}$ (8)

##  4 Comparison of LSTM and LSTM with attention

We can now move onto the implementation of the baseline LSTM. Both this
baseline LSTM and the LSTM with attention were implemented within the Kaggle
kernel environment in Python using the Keras library. It is worth noting here
that special care was taken for sensible crosschecks. For example, the Kaggle
kernel environment was set up to block the use of ‘future’ data when training,
preventing look-ahead bias. Indeed, look-ahead bias is a significant source of
malpractice in the field of FTS so was worthy of extra consideration here. In
addition, all implementations closely followed existing literature from the
related work discussed in section 2.

To gain a good benchmark for the performance of an LSTM on forecasting stock
prices from the Two Sigma dataset, we first consider a subset of stocks. This
subset of stocks contains three very large companies (Intel, Wells Fargo,
Amazon), one SME (Agilent Technologies) and one smaller company (Benchmark
Electronics). These stocks were carefully chosen to have a wide variety of
market cap, volatility and overall trend. Larger market cap stocks tend to be
less volatile compared to smaller stocks. In addition, tech companies (like
Amazon) tend to be less affected by the 2009 crash than finance companies
(like Wells Fargo). These stocks and their volatility are shown in figure 2 as
follows.

Figure 2: Price and volatility of small stocks (Agilent, Benchmark Electronics
- top) and large stocks (Intel, Wells Fargo, Amazon - bottom)

Figure 2 clearly shows one of the main challenges of FTS forecasting which is
the change of volatility over time. Indeed, looking at the volatility of
Amazon, one notices a significant increase in volatility with increasing size
of the company. This is the main justification behind picking a diverse set of
stocks that are bound by different statistical properties.

It is worth showing some loss curves for the benchmark LSTM in order to
examine how well the model generalises to the validation data with increasing
numbers of epochs. This is shown in Figure 3 as follows.

Figure 3: Typical LSTM loss for training and validation data per epoch

Figure 3 shows the typical LSTM training and validation losses through epochs.
It can be noted that loss decreases as expected and begins to overfit after a
certain number of epochs. It is worth noting one particularity from this plot
which is that it shows training loss greater than validation loss for early
epochs. While this may seem unusual, it is a documented artifact of using
Keras. Indeed, the training loss output is the average loss over the batch
while the validation loss is the final loss. It is therefore expected that the
first few iterations show a higher training loss than validation loss. This is
because the very first training iterations will have a higher loss than the
final iteration of the validation loss.

In order to tune this benchmark model, a grid search is undertaken. The LSTM
performance during this grid search hyperparameter tuning, evaluated on the
validation set, can be seen by the following 3D loss plots.

Figure 4: LSTM grid search loss plots for 3 different sizes

These plots in figure 4 show the performance impact of dropout and lag for
three different LSTM sizes. Indeed, it seems the LSTM with size 64 performs
best on average. In addition, dropout is clearly a useful regulariser for all
three LSTM sizes, particularly for values of 0.2 and higher as it leads to
lower losses. From this hyperparameter tuning, the best chosen configuration
was with $size=64$, $lag=15$ and $dropout=0.1$. This resulted in a loss of
0.000805 and an up-down accuracy of 0.572 (or 57%). Indeed, this consistently
outperforms random guessing, which would have a performance of 50%, and is in
line with top of the range algorithms which usually achieve around 60% up-down
accuracy.

Once the hyperparameters are tuned, a common FTS investigation worth pursuing
is that of dataset shuffling techniques. In FTS, the choice of which piece of
data to use as the validation set is not trivial. Indeed, there exist a myriad
of ways of doing this which must be carefully considered before comparing this
LSTM to an LSTM with attention. The three methods investigated in this paper
are visualised in figure 5.

Figure 5: Shuffling techniques visualised

The fixed origin method is the most naive and common method used. Given a
certain split size, the start of the data is the training set and the end is
the validation set. However, this is a particularly rudimentary method to
choose, especially for a high-growth stock like Amazon. The reason why this is
the case is that the Amazon’s stock price starts off with low volatility and,
as the stock grows, experiences increasingly volatile behaviour. We would
therefore be training a model on low volatility dynamics and expect it to deal
with unseen high volatility dynamics for its predictions. This has indeed
shown itself to be difficult and come at a cost in performance for these types
of stocks as we will see in table 1. Therefore our benchmark for validation
loss and performance may be misleading if we only consider this. However, for
stocks like Intel that are more constant in their volatility, this method is
reasonable.

The rolling origin recalibration method is slightly less vulnerable than fixed
origin as it allows the validation loss to be computed by taking the average
of various different splits of the data to avoid running into unrepresentative
issues with high volatility timeframes.

Finally, the rolling window method is usually one of the most useful methods
as it is particularly used for FTS algorithms being run for long timeframes.
Indeed, this model outputs the average validation error of multiple rolling
windows of data. This means the final values we get are more representative of
recent model performance, as we are less biased by strong or poor performance
in the distant past.

It is now important to show the model performance of our tuned benchmark LSTM
using these different shuffling techniques. Using regular fixed origin our
tuned LSTM achieved a loss of 0.000805 and an up-down accuracy of 57%. Let’s
compare these values with top six values of rolling window (RW) and rolling
origin recalibration (ROR) in table 1.

Loss (RW) | Accuracy (RW) | Loss (ROR) | Accuracy (ROR)
---|---|---|---
0.000692 | 0.538 | 0.000810 | 0.555
0.000693 | 0.530 | 0.000825 | 0.571
0.000725 | 0.575 | 0.000978 | 0.607
0.000755 | 0.551 | 0.000989 | 0.563
0.000780 | 0.514 | 0.001001 | 0.579
0.000788 | 0.583 | 0.001014 | 0.538

Table 1: Performance comparison of shuffling methods

What table 1 shows is that both RW and ROR describe very slightly better
performances (58% and 60%) than that of the simple fixed origin method. This
suggests that for stocks like Amazon, using these shuffling methods would be
inevitable.

Now that reasonable benchmarks have been ascertained, we can compare them to
the performance of our LSTM with attention. The LSTM models with attention
generated here had a strong tendency to overfit the training data for multiple
epochs as shown in figure 6.

Figure 6: Typical LSTM+A loss for training and validation data per epoch

As shown in the loss curves of figure 6, we can see that performance at train
time continually decreases while validation performance rises suggesting
overfitting. Due to the high number of weights used in our implementation of
the attention mechanism (with 16 cells, there are around 10 000 parameters),
there are more parameters than the number of data points themselves, which
inevitably leads to overfitting. Due to a large amount of parameters compared
to the number of our data points, the loss values are very low and the
training loss does not fluctuate a lot. This explains how the validation loss
seems to increase immediately after one or two epochs when looking at the
graphs. From this behaviour, we decided to limit the number of epochs for the
LSTMs with attention (grid search over 25 epochs, compared to 100 epochs with
LSTMs only). In order to help alleviate some of the overfitting errors and
lack of generalisation, a hyperparameter grid search was undertaken to tune
the model’s hyperparameters. In particular, dropout and varying the amount of
lag for given LSTM sizes were investigated, as done for the LSTM without
attention. Loss plots are shown in figure 7 below.

Figure 7: LSTM+A grid search loss plots for 3 different sizes

The LSTM with attention achieves a globally lower loss than the LSTM without
attention but is sensitive to different hyperparameters. Indeed, in the case
of the LSTM with attention, the amount of lag is a significant factor in
decreasing loss. This can be seen in figure 7 as lags larger than 30
drastically improve loss. It could be hypothesized that this is due to the
unique nature of attention that can focus on relevant information from the
past while the LSTM without attention is incapable of doing this due to its
long term dependencies. This seems to support our original hypothesis on the
impact of using attention on LSTMs for FTS. From this hyperparameter tuning,
the best chosen configuration was with $size=16$, $lag=60$ and $dropout=0.05$.
This resulted in a loss of 0.001511 and an up-down accuracy of 0.588 (or 59%).
Again, this is in line with top of the range algorithms and is slightly higher
than that of the LSTM without attention (58% accuracy).

It is now worth comparing the performance, loss and accuracy, of both the
tuned LSTM and the tuned LSTM with attention across all five stocks. The
optimal hyperparameters for both the LSTM and LSTM with attention were set as
detailed previously and the following performances were observed.

| Intel | Wells Fargo | Amazon | Agilent | BE
---|---|---|---|---|---
LSTM (loss) | 0.000805 | 0.001200 | 0.002804 | 0.001073 | 0.002300
LSTM (accuracy) | 0.573 | 0.457 | 0.490 | 0.457 | 0.470
LSTM+A (loss) | 0.001511 | 0.000357 | 0.003168 | 0.000955 | 0.001787
LSTM+A (accuracy) | 0.588 | 0.603 | 0.328 | 0.443 | 0.493

Table 2: LSTM and LSTM with attention performance comparison

From table 2 above, we can observe multiple interesting dynamics at play. The
first being that the optimal parameters chosen by grid search with the Intel
stock fail to generalise to other stocks for the LSTM and the LSTM with
attention. While this was somewhat expected it is interesting nonetheless to
observe this limitation. Further time would allow the tuning of
hyperparameters for each stock but this is outside the scope of this paper and
will be discussed in the following section. In addition, it seems that the
LSTM with attention has a higher variability in its performances. This makes
sense as we know that the model is very complex in terms of parameter number
with respect to the data input. Overall in this comparison, the LSTM with
attention has outperformed the regular LSTM. Indeed, it does seem like we can
tentatively confirm our original hypothesis albeit further work is certainly
required. In addition, the attention architecture used in this paper does
involve certain limitations and has a caveat of inherent complexity.

##  5 Limitations and Future Work

One particular limitation of this model is that we only considered the first
prediction in our accuracy score. This means we are effectively doing single
step ahead prediction even if the model is inherently capable of doing multi-
step ahead prediction. Indeed, multi-step ahead forecasting is historically a
much harder problem than single step ahead. The reason why this is the case
can be seen by the naive approach of converting single step ahead forecasting
to multi-step forecasting via the iterative method. The iterative method
simply takes the predicted output of the model for time $t+1$ and forecasts
output for $t+2$ from the existing forecast without feedback from the real
world. This causes the model error to cascade through iterations and grow out
of control for more than a handful of time steps. This is a fundamental issue
where most models fail for FTS forecasting. However, the model described in
this paper is capable of seq2seq which allows for forecasting an entire
sequence without the uncontrollable error growth experienced by iterative
methods. Indeed, this potential for seq2seq is key advantage of the models
used and constitutes a major avenue for future work and research.

Another limitation of this investigation is confidence interval encoding.
Indeed, there are various ways to do so but the approach used here was to make
the models output a value between -1 and 1 by using a $\tanh$ function.
However, the confidence output of our model was not considered in the up-down
accuracy detailed above. Indeed, a simple way to exploit confidence intervals
to improve performance would be to only execute trades beyond certain
threshold of confidence. Indeed, this has been leveraged successfully for most
models in FTS forecasting however it remains a non-trivial task. Indeed, the
higher the threshold is set for executing trades, the higher the overall up-
down accuracy will be. However, the higher the threshold the lower the number
of total trades executed. There is therefore a major trade-off here as perfect
accuracy for few trades may be less profitable than excellent accuracy for
many trades.

One major possible extension to the work presented here would be the use of
Bayesian optimisation. Bayesian optimisation is a technique commonly used in
financial time series forecasting for the tuning of hyperparameters. While
this was not done in this report for time constraints, it is essential for a
more detailed comparison of the potential of attention mechanisms in LSTMs for
FTS forecasting. Indeed, as revealed by this paper, hyperparameters have
different sensitivities in the benchmark LSTM and in the LSTM with attention.
Future work would consist of choosing an appropriate surrogate function and
acquisition function (such as expected improvement) in an attempt to explore
the hyperparameter space with a smart explore-exploit trade-off. This is
indeed still an open problem in the field of FTS forecasting.

Another possible extension would be to investigate how the LSTM with attention
performs compared to other benchmark models for FTS forecasting. One
particular interesting comparison could be done with cutting edge Temporal
Convolutional Networks (TCN) as they are currently showing promising
performances compared to RNNs in NLP applications.

##  6 Conclusions

This paper has demonstrated the performance of a benchmark LSTM.
Investigations were undertaken to avoid using a single number as a benchmark
and a wide range of experimentation was conducted. This includes the
investigation of data shuffling methods such as rolling window and rolling
origin recalibration, which showed the impact of volatility on estimating
model performance. Ultimately, the benchmark LSTM consistently performed
around the 58% range which is in line with the best models currently available
which usually reach performances around 60%.

In addition, the LSTM with attention was successfully implemented and
leveraged for FTS forecasting. This task, while totally novel, did nonetheless
closely follow methods used in other FTS papers and papers using attention for
NLP. This LSTM with attention achieved performances of around 60% and above,
albeit with a higher variability than the benchmark LSTM.

The final comparison of this LSTM with attention does indeed confirm the
investigated hypothesis that an LSTM with attention can improve the
performance of existing LSTMs in FTS. A slight improvement was indeed
highlighted in the final comparison table, albeit both models need to be re-
tuned in between stocks. A theoretical explanation for why this is the case
was suggested, developed and tested.

Finally, further work in this topic has been suggested and the main model
limitations were discussed.

## References

  * [1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [2] Wei Bao, Jun Yue, and Yulei Rao.  A deep learning framework for financial time series using stacked autoencoders and long-short term memory.  PloS one, 12(7):e0180944, 2017.
  * [3] Christoph Bergmeir and José M Benítez.  On the use of cross-validation for time series predictor evaluation.  Information Sciences, 191:192–213, 2012.
  * [4] Tim Bollerslev.  Generalized autoregressive conditional heteroskedasticity.  Journal of econometrics, 31(3):307–327, 1986.
  * [5] John Parker Burg.  A new analysis technique for time series data.  Paper presented at NATO Advanced Study Institute on Signal Processing, Enschede, Netherlands, 1968, 1968.
  * [6] Rohitash Chandra and Mengjie Zhang.  Cooperative coevolution of elman recurrent neural networks for chaotic time series prediction.  Neurocomputing, 86:116–123, 2012.
  * [7] Sheng Chen and Steve A Billings.  Representations of non-linear systems: the narmax model.  International Journal of Control, 49(3):1013–1032, 1989.
  * [8] Robert F Engle.  Autoregressive conditional heteroscedasticity with estimates of the variance of united kingdom inflation.  Econometrica: Journal of the Econometric Society, pages 987–1007, 1982.
  * [9] Robert F Engle and Victor K Ng.  Measuring and testing the impact of news on volatility.  The journal of finance, 48(5):1749–1778, 1993.
  * [10] Orhan Firat, Kyunghyun Cho, and Yoshua Bengio.  Multi-way, multilingual neural machine translation with a shared attention mechanism.  arXiv preprint arXiv:1601.01073, 2016.
  * [11] Alex Graves, Greg Wayne, and Ivo Danihelka.  Neural turing machines.  arXiv preprint arXiv:1410.5401, 2014.
  * [12] James Douglas Hamilton.  Time series analysis, volume 2.  Princeton university press Princeton, NJ, 1994.
  * [13] Nessrine Hamzaoui and Boutheina Regaieg.  The glosten-jagannathan-runkle-generalized autoregressive conditional heteroscedastic approach to investigating the foreign exchange forward premium volatility.  International Journal of Economics and Financial Issues, 6(4):1608–1615, 2016.
  * [14] Ludger Hentschel et al.  All in the family: Nesting symmetric and asymmetric garch models.  Journal of Financial Economics, 39(1):71–104, 1995.
  * [15] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [16] Thomas Hollis.  Deep learning algorithms applied to blockchain-based financial time series.  2018\.
  * [17] Kaggle.  Two sigma: Using news to predict stock movements.  https://www.kaggle.com/c/two-sigma-financial-news.  Accessed: 2018-09-30.
  * [18] Kyoung-jae Kim.  Financial time series forecasting using support vector machines.  Neurocomputing, 55(1-2):307–319, 2003.
  * [19] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [20] Teuvo Kohonen.  Self-organized formation of topologically correct feature maps.  Biological cybernetics, 43(1):59–69, 1982.
  * [21] Timo Koskela, Markus Varsta, Jukka Heikkonen, and Kimmo Kaski.  Time series prediction using recurrent som with local linear models.  Int. J. of Knowledge-Based Intelligent Engineering Systems, 2(1):60–68, 1998.
  * [22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [23] Takashi Kuremoto, Shinsuke Kimura, Kunikazu Kobayashi, and Masanao Obayashi.  Time series forecasting using a deep belief network with restricted boltzmann machines.  Neurocomputing, 137:47–56, 2014.
  * [24] Xiaowei Lin, Zehong Yang, and Yixu Song.  Short-term stock price prediction based on echo state networks.  Expert systems with applications, 36(3):7313–7317, 2009.
  * [25] Burton G Malkiel and Eugene F Fama.  Efficient capital markets: A review of theory and empirical work.  The journal of Finance, 25(2):383–417, 1970.
  * [26] John J Murphy.  Technical analysis of the financial markets: A comprehensive guide to trading methods and applications.  Penguin, 1999.
  * [27] Eileen F St Pierre.  Estimating egarch-m models: Science or art?  The Quarterly Review of Economics and Finance, 38(2):167–180, 1998\.
  * [28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems, pages 5998–6008, 2017.
  * [29] Gilbert Thomas Walker.  On periodicity in series of related terms.  Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character, 131(818):518–532, 1931.
  * [30] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In International conference on machine learning, pages 2048–2057, 2015.
  * [31] G Peter Zhang.  Time series forecasting using a hybrid arima and neural network model.  Neurocomputing, 50:159–175, 2003.