# Velocity bunching in travelling wave accelerator with low acceleration
gradient††thanks: Supported by National Natural Science Foundation of China
(11205152)

HUANG Rui-Xuan HE Zhi-Gang1) LI Wei-Wei JIA Qi-Ka<EMAIL_ADDRESS>National
Synchrotron Radiation Laboratory, University of Science and Technology of
China, Hefei, 230029, Anhui, China

###### Abstract

We present the analytical and simulated results concerning the influences of
the acceleration gradient in the velocity bunching process, which is a bunch
compression scheme that uses a traveling wave accelerating structure as a
compressor. Our study shows that the bunch compression application with low
acceleration gradient is more tolerant to phase jitter and more successful to
obtain compressed electron beam with symmetrical longitudinal distribution and
low energy spread. We also present a transverse emittance compensation scheme
to compensate the emittance growth caused by the increasing of the space
charge force in the compressing process that is easy to be adjusted for
different compressing factors.

###### keywords:

traveling wave accelerating structure, velocity bunching, acceleration
gradient, emittance compensation

###### pacs:

4

1.85.Ew, 41.85.Ct, 41.60.Cr

##  1 Introduction

In recent years, the demand for applications of high brightness - low
emittance, high current, with sub-picosecond pulse length electron beams has
increased dramatically. In the fourth generation synchrotron light source
community, high-brightness beams are needed for application to short
wavelength free electron lasers (FEL), as well as for inverse-Compton-
scattering (ICS) generation of short x-ray pulses. For studying novel
accelerating techniques such as plasma-based accelerators and generation of
coherent THz radiation, short electron bunches are also required.

Short bunches are commonly obtained by magnetic compression. In this scheme,
the bunch is compressed when drifting through a series of dipoles arranged in
a chicane configuration which can introduce an energy-dependent path length.
Therefore an electron bunch with the proper time-energy correlation can be
shortened in the chicane. The time-energy correlation along the bunch can be
tuned by means of an accelerating section upstream from the chicane. Great
progress has been made in this this field, but magnetic compression may
introduce momentum spread and transverse emittance dilution due to the bunch
self-interaction via coherent synchrotron radiation[2]. To obtain a smaller
and more symmetrical electron beam, a linear energy-time correlation is
required along the bunch, which can be realized by a accelerating structure at
a higher harmonic[3], with respect to the main accelerating linac RF.

Velocity bunching relies on the phase slippage between the electrons and the
rf wave that occurs during the acceleration of nonultrarelativistic electrons.
It was experimentally observed in photocathode rf gun[4] and proposed to
integrate the velocity bunching scheme in the next photoinjector designs using
a dedicated rf structure downstream of the rf electron source[5]. Previous
experimental works showed the compression ability of the velocity bunching
method[6, 7]. Furthermore, the emittance growth in the compressing process was
completely compensated by long solenoids[8] when the compression factor is 3.

This paper mainly focuses on the beam bunching in the traveling wave structure
with low acceleration gradient (4 MV/m) instead of high acceleration gradient
(normally 20 MV/m) in previous work. A brief analysis of the velocity bunching
mechanism is presented in the Section 2 firstly. In Section 3, the analytical
and simulated results of bunch compression within high and low acceleration
gradient accelerators are described, and the conclusion is figured out that a
traveling wave accelerating structure with low acceleration gradient is more
tolerant to phase jitter and easier to obtain compressed electron beam with
symmetrical longitudinal distribution and low energy spread in the velocity
bunching process. A transverse emittance compensation scheme is shown in
Section 4, which is easy to be adjusted for different compressing factors.
Section 5 presents a summary of this paper.

##  2 Velocity bunching mechanism

In the velocity bunching process, the longitudinal phase space rotation is
based on a correlated time-velocity chirp in the electron bunch, so that
electrons on the tail of the bunch are faster than electrons in the bunch
head. This rotation occurs inside a traveling rf wave of a long multicell rf
structure which applies an off crest energy chirp to the injected beam as well
as accelerates it.. This is possible if the injected beam is slightly slower
than the phase velocity of the rf wave so that when injected at the zero
crossing field phase it slips back to phases where the field is accelerating,
but is simultaneously chirped and compressed.

An electron in an rf traveling wave accelerating structure experiences the
longitudinal electric field:

$\displaystyle{E_{z}}={E_{0}}\sin(\phi)$ (1)

where $E_{0}$ is the peak field, $\phi=kz-\omega t+{\phi_{0}}$ is the phase of
the electron with respect to the wave and $\phi_{0}$ is the injection phase of
the electron with respect to the rf wave. The evolution of $\phi$ can be
expressed as a function of z:

$\displaystyle\frac{{d\phi}}{{dz}}=k-\omega\frac{{dt}}{{dz}}=k-\frac{\omega}{{\beta
c}}=k(1-\frac{\gamma}{{\sqrt{{\gamma^{2}}-1}}})$ (2)

The energy gradient can be written as[9]:

$\displaystyle\frac{{d\gamma}}{{dz}}=\alpha k\sin(\phi)$ (3)

where $\alpha\equiv e{E_{0}}/m{c^{2}}k$ is defined as dimensionless vector
potential amplitude of the wave. The equations (2) and (3) with the initial
conditions ${\gamma_{z=0}}={\gamma_{0}}$ and ${\phi_{z=0}}={\phi_{0}}$
describe the longitudinal motion of an electron in the rf structure. Using a
separation of variables approach, one can get

$\displaystyle\alpha\cos\phi+\gamma-\sqrt{{\gamma^{2}}-1}=C$ (4)

the $\phi$ can be expressed as a function of $\gamma$ :

$\displaystyle\phi(\gamma)=\arccos(\frac{{C-\gamma+\sqrt{{\gamma^{2}}-1}}}{\alpha})$
(5)

where the constant C is set by the initial conditions of the problem:
$C=\alpha\cos\phi_{0}+\gamma_{0}-\sqrt{{\gamma_{0}^{2}}-1}$ . The final phase
of electron at the exit of the accelerator is

$\displaystyle\phi_{e}(\gamma_{e})=\arccos(\frac{{C-\gamma_{e}+\sqrt{{\gamma_{e}^{2}}-1}}}{\alpha})$
(6)

If the $\gamma_{e}$ is high enough,
$\sqrt{{\gamma_{e}^{2}}-1}-\gamma_{e}\simeq 0$ . With the approximation
$\gamma_{0}-\sqrt{{\gamma_{0}^{2}}-1}\simeq 1/(2\gamma_{0})$ . Then, the final
phase becomes:

$\displaystyle\phi_{e}=\arccos(\cos\phi_{0}+1/(2\alpha\gamma_{0}))$ (7)

Expanding Eq. (7) to first order in the initial energy spread and initial
phase spread gives

$\displaystyle\Delta{\phi_{e}}=\frac{{\sin{\phi_{0}}}}{{\sin{\phi_{e}}}}\Delta{\phi_{0}}+\frac{1}{{2\alpha\gamma_{0}^{2}\sin{\phi_{e}}}}\Delta{\gamma_{0}}$
(8)

Hence depending upon the incoming energy and phase extents (initial bunch
length), the phase of injection in the rf structure $\phi_{0}$ can be tuned to
minimize the phase extent of extraction (final bunch length).

##  3 Bunch compression in low and high gradient accelerators

Based on equations (6) and (8), one can find that the compression factor is
less sensitive to the injection phase $\phi_{0}$ and more tolerant to the
phase jitter when the peak field $E_{0}$ is lower. Fig. 1 shows the simulated
phase scanning results at different acceleration gradients by using the code
ASTRA[10].

The rms bunch length as a function of relative phase (There is about 28.7
degrees phase shift between the two results. In the figure, we do a
translation for contrast).

The simulation shows a coincident result to the analytical conclusion. It
indicates that higher precision phase jitter control and power supply for the
solenoids to compensate the transverse emittance growth (The emittance
compensation scheme will be presented in next section) are needed in
experimental practice when the acceleration gradient is high. A lower
acceleration gradient in the velocity bunching process has been confirmed to
be more tolerant to phase jitter and power precision. Furthermore, with a
lower gradient, the compressed electron beam tends to develop more symmetrical
longitudinal distribution. Analytical and simulated results will be presented
in following paragraph.

The velocity bunching process can be shown as Fig. 2. To compress the electron
bunch, the injection phase should be set at the phase that is tens degrees off
the crest. The velocity of the initial beam is smaller than the rf wave (
$\beta_{\phi}=1$ ), there is phase slippage between the initial and compressed
beam. For a high gradient accelerator, the phase slippage is smaller, but to
obtain the same compression factor as in the low gradient accelerator, the
injection phase should be closer to the crest; when the injection phase is
farther from the crest, one can get a compressed beam with a single spike and
most of electrons are in the spike, but the compression factor is too large,
the emittance compensation is difficult at present. In this paper, the
accelerators are the normal S-band ( $2856~{}MHz$ ), $2/3\pi$ mode, $3~{}m$
long travelling wave accelerators. One also can set an appropriate injection
phase to make the phase slippage in the range of linearity within a short
accelerator, then the electron bunch is imposed a more linear negative energy
chirp (the speed of electrons in the head is slower than the speed of
electrons in the tail), and the electron bunch can be compressed enormously in
a drift space with appropriate length, which has been presented in
Reference[7] and named by the authors as “ballistic bunching”.

Velocity bunching process.

Eq. (1) can be expanded as:

$\displaystyle{E_{z}}={E_{0}}(\phi-\frac{{{\phi^{3}}}}{{3!}}+\frac{{{\phi^{5}}}}{{5!}}-\frac{{{\phi^{7}}}}{{7!}}+\frac{{{\phi^{9}}}}{{9!}}\cdots)$
(9)

If there is no higher order terms in the equation, the longitudinal
distribution state of the initial beam can be preserved during the velocity
bunching process when the initial energy spread is very small or it can be
treat as a linear one. Unfortunately, there are always higher order terms in
the equation. So reducing the $E_{0}$ may be an available way to prevent the
longitudinal distortion. By using the equations (2) and (3), the numerical
calculation results show the rationality of the analysis. However, the space
charge force and magnetic force are not considered. In the following content,
the simulation results will be presented.

Longitudinal distribution after compression by different $E_{0}$ should be
compared. The current profiles of the initial bunch and the compressed bunches
(with almost identical bunch length) are shown in Fig. 3.

The current profiles of the initial bunch (blue line) and the compressed
bunches (black line: compressed bunch in $4~{}MV/m$ gradient accelerator; red
line: compressed bunch in $20~{}MV/m$ gradient accelerator).

From the above figure it appears that a traveling wave accelerating structure
with low acceleration gradient is more successful to obtain compressed
electron with symmetrical longitudinal distribution. Fig. 4 shows the bunch
length evolutions along the beam direction when the acceleration gradients are
$E_{0}=4~{}MV/m$ and $E_{0}=20~{}MV/m$ .

RMS bunch length evolution along the longitudinal position.

As we know that to avoid the longitudinal distortion in magnetic compression,
an X-band accelerator system is normally used to linearize the energy chirp of
the electron beam[3], which is cost almost one million dollar. About this
problem in the velocity bunching process, theoretical and experimental works
are worth doing in detail.

When the $E_{0}$ is higher, the energy spread in the bunch is greater, because
the bunch compression is achieved by the velocity difference within the bunch.
Fig. 5 shows a contrast of the energy spread evolution when the gradient are
$E_{0}=4~{}MV/m$ and $E_{0}=20~{}MV/m$ .

RMS energy spread evolution at different acceleration gradients.

##  4 Emittance compensation

In this section, an emittance compensation scheme will be suggested during the
velocity bunching process.

The electric field of accelerators along the beam line.

The electric and magnetic fields along the beam line are shown in Fig. 6 and
Fig. 7. The peak acceleration gradients of the photocathode rf gun,
accelerator 1 and accelerator 2 are $80~{}MV/m$ , $4~{}MV/m$ and $20~{}MV/m$
respectively. Four solenoids are used for emittance compensation, $B_{0}$ is
located at $10~{}cm$ downstream from the photocathode, and the other three are
around the accelerators.

The magnetic field of solenoids along the beam line.

To prevent irreversible emittance growth during bunch compression, the crux is
to preserve the laminarity of the electron beam with an envelope propagated as
close as possible to a Brillouin-like flow, represented by an invariant
envelope[11] as generalized to the context of beam compression and thus
increasing $I$ during acceleration. Mismatches between the space charge forces
and the external focusing gradient produce slice envelope oscillations that
cause normalized emittance oscillations. It has been shown that in order to
keep such oscillations under control during the velocity bunching, the beam
has to be injected into the accelerator with a laminar envelope waist (
$\sigma^{{}^{\prime}}=0$ ) and the envelope has to be matched to the
accelerating and focusing gradients so that it can stay close to an
equilibrium mode[11]. Long solenoids around the accelerator are used to
provide the required focusing.

The transverse emittance and beam size evolutions along the beam direction are
shown in Fig. 8. For electron bunches with different compression factors at
different injection phases, one just need to scan the strength of the
solenoids to compensate the emittance growth as shown in Fig. 8.

The transverse emittance and beam size evolutions along the beam direction at
different injection phases, the optimal magnetic strengths of solenoids are
also shown in the figure.

Fig. 9 shows the current profiles for different compressed beam at different
injection phases. We should point out that the magnetic force of long
solenoids (around the accelerator) can affect the compression, which can be
found by contrasting the results in Fig. 9 and phase scanning result at
$E_{0}=4~{}MV/m$ without the long solenoids in Fig. 1.

The current profiles for different compressed beam at different injection
phases.

##  5 Summary and conclusion

Based on the analyses, we contrast the ASTRA simulated results of velocity
bunching process in travelling wave structure with low ( $4~{}MV/m$ ) and high
( $20~{}MV/m$ ) acceleration gradient. According to the results, the
conclusion is figured out that the bunch compression application with low
acceleration gradient is more tolerant to phase jitter and should be useful
for obtaining better performance beams with symmetrical longitudinal
distribution and low energy spread. Furthermore a successful improvement of
transverse emittance during compression is possible with optimized long
solenoids which is easily satisfied for different compressing factor.

###### Acknowledgements.

The authors thank Prof. PEI, Yuan-Ji for insightful discussions.

## References

  * [1]
  * [2] Bane K L F, et al. Phys. Rev. ST Accel. Beams, 2009, 12: 030704.
  * [3] Emma P, LCLS-TN-01-1, November 14, 2001.
  * [4] Wang X J, Qiu X, and Ben-Zvi I. Phys. Rev. E, 1996, 54: R3121.
  * [5] Serafini L and Ferrario M. AIP Conf. Proc, 2001, 581: 87.
  * [6] Piot P, Carr L, Graves W S and Loos H. Phys. Rev. ST Accel. Beams, 2003, 6: 033503.
  * [7] Anderson S G, et al. Phys. Rev. ST Accel. Beams, 2005, 8: 014401.
  * [8] Ferrarion M, et al. Phys. Rev. Lett, 2010, 104: 054801
  * [9] Kim K J, Nucl. Instr. and Meth. A, 1989, 275: 201.
  * [10] Flottmann K. ASTRA user manual, http://www.desy.de/~mpyflo/
  * [11] Serafini L and Rosenzweig J B, Phys. Rev. E, 1997, 55: 7565.


# The Importance of the Current Input in
Sequence Modeling

Christian Oliva & Luis F. Lago-Fernández
Departamento de Ingeniería Informática
Universidad Autónoma de Madrid
Madrid, 28049, Spain
christian.oliva<EMAIL_ADDRESS>
###### Abstract

The last advances in sequence modeling are mainly based on deep learning
approaches. The current state of the art involves the use of variations of the
standard LSTM architecture, combined with several tricks that improve the
final prediction rates of the trained neural networks. However, in some cases,
these adaptations might be too much tuned to the particular problems being
addressed. In this article, we show that a very simple idea, to add a direct
connection between the input and the output, skipping the recurrent module,
leads to an increase of the prediction accuracy in sequence modeling problems
related to natural language processing. Experiments carried out on different
problems show that the addition of this kind of connection to a recurrent
network always improves the results, regardless of the architecture and
training-specific details. When this idea is introduced into the models that
lead the field, the resulting networks achieve a new state-of-the-art
perplexity in language modeling problems.

##  1 Introduction

Deep learning models constitute the current state of the art in most
artificial intelligence applications, from computer vision to robotics or
medicine. When dealing with sequential data, Recurrent Neural Networks (RNNs),
specially those architectures with gating mechanisms such as the LSTM [7], the
GRU [3] and other variants, are usually the default choice. One of the most
interesting applications of RNNs is related to the field of Natural Language
Processing, where most tasks, such as machine translation, document
summarization or language modeling, involve the manipulation of sequences of
textual data. Of these, language modeling has been extensively used to test
different innovations in recurrent architectures, mainly due to the ease of
obtaining very large datasets that can be used to train neural networks with
millions of parameters.

Sequence modeling consists of predicting the next element in a sequence given
the past history. In language modeling, the sequence is a text, and hence the
task is to predict the next word or the next character. In this context, some
of the best performing architectures include the Mogrifier LSTM [12] and
different variations of the AWD-LSTM [13], usually combined with dynamic
evaluation and mixture of sofmaxes (MoS) [20, 6]. These models obtain the best
state-of-the-art performance with moderate size datasets, such as the Penn
Treebank [15] or the Wikitext-2 [14] corpora, when no additional data are used
during training. When larger datasets are considered, or when external data
are used to pre-train the networks, attention-based architectures usually
outperform other models [18, 2].

In this work we use moderate-scale language modeling datasets to explore the
effect of a mechanism recently proposed by [16], when combined with different
LSTM-based models in the language modeling context. The idea consists of
modifying a recurrent architecture by introducing a direct connection between
the input and the output of the recurrent module. This has been shown to
improve both the model’s generalization results and its readability in simple
tasks related to the recognition of regular languages.

In a standard RNN, the output depends only on the network’s hidden state,
${\displaystyle{\bm{h}}}_{t}$, which in turn depends on both the input,
${\displaystyle{\bm{x}}}_{t}$, and the recent past,
${\displaystyle{\bm{h}}}_{t-1}$. But there is no explicit dependence of the
network’s output on its input. In some cases this could be a shortcoming,
since the transformation of ${\displaystyle{\bm{x}}}_{t}$ needed to compute
the network’s internal state is not necessarily the most appropriate to
compute the output. However, an explicit dependence of the output on
${\displaystyle{\bm{x}}}_{t}$ can be forced by adding a dual connection that
skips the recurrent layers. We claim that this strategy may be of general
application in RNN models.

To test our hypothesis we perform a thorough comparison of several state-of-
the-art RNN architectures, with and without the dual connection, on the Penn
Treebank and the Wikitext-2 datasets. Our results show that, under all
experimental conditions, the dual architectures outperform their non-dual
counterparts. In addition, the Mogrifier-LSTM enhanced with a dual connection
establishes a new state-of-the-art word-level perplexity for the Penn Treebank
problem when no additional data are used to train the models.

The remainder of the article is organized as follows. First, in section 2, we
present the different models we have used and the two possible architectures,
the standard recurrent architecture and the dual architecture. In section 3,
we describe the datasets and the experimental setup. In section 4, we present
our results. And finally, in section 5, we extract some conclusions and
discuss further lines of research.

##  2 Models

We start by presenting the standard recurrent architecture which is common to
all the models. In absence of a dual connection, the basic architecture
involves an embedding layer, a recurrent layer and a fully-connected layer
with $softmax$ activation:

$\displaystyle{\displaystyle{\bm{e}}}_{t}$ $\displaystyle=$
$\displaystyle{\displaystyle{\bm{W}}}^{ex}{\displaystyle{\bm{x}}}_{t}$ (1)
$\displaystyle{\displaystyle{\bm{h}}}_{t}$ $\displaystyle=$ $\displaystyle
REC({\displaystyle{\bm{e}}}_{t},{\displaystyle{\mathbb{S}}}_{t-1})$ (2)
$\displaystyle{\displaystyle{\bm{y}}}_{t}$ $\displaystyle=$ $\displaystyle
softmax({\displaystyle{\bm{W}}}^{yh}{\displaystyle{\bm{h}}}_{t}+{\displaystyle{\bm{b}}}^{y}),$
(3)

where ${\displaystyle{\bm{W}}}^{**}$ and ${\displaystyle{\bm{b}}}^{*}$ are
weight matrices and biases, respectively, and ${\displaystyle{\bm{x}}}_{t}$ is
the input vector at time $t$. The $REC$ module represents an arbitrary
recurrent layer, with ${\displaystyle{\mathbb{S}}}_{t-1}$ being a set of
vectors describing its internal state at the previous time step. In the most
general case, this module will simply be an LSTM cell, but we consider other
possibilities as well, as described below.

The dual architecture introduces an additional layer, with ReLU activation,
which is fed with both the output of the embedding layer and the output of the
recurrent module:

$\displaystyle{\displaystyle{\bm{e}}}_{t}$ $\displaystyle=$
$\displaystyle{\displaystyle{\bm{W}}}^{ex}{\displaystyle{\bm{x}}}_{t}$ (4)
$\displaystyle{\displaystyle{\bm{h}}}_{t}$ $\displaystyle=$ $\displaystyle
REC({\displaystyle{\bm{e}}}_{t},{\displaystyle{\mathbb{S}}}_{t-1})$ (5)
$\displaystyle{\displaystyle{\bm{d}}}_{t}$ $\displaystyle=$ $\displaystyle
ReLU({\displaystyle{\bm{W}}}^{de}{\displaystyle{\bm{e}}}_{t}+{\displaystyle{\bm{W}}}^{dh}{\displaystyle{\bm{h}}}_{t}+{\displaystyle{\bm{b}}}^{d})$
(6) $\displaystyle{\displaystyle{\bm{y}}}_{t}$ $\displaystyle=$ $\displaystyle
softmax({\displaystyle{\bm{W}}}^{yd}{\displaystyle{\bm{d}}}_{t}+{\displaystyle{\bm{b}}}^{y}).$
(7)

This way the network’s input can reach the softmax layer following two
different paths, through the recurrent layer and through the dual connection.
In the following we consider different forms for the recurrent module in
equations 2 and 5.

###  2.1 The LSTM module

In the simplest approach the recurrent module consists of an LSTM cell, where
the internal state includes both the output and the memory,
${\displaystyle{\mathbb{S}}}_{t}=\\{{\displaystyle{\bm{h}}}_{t};{\displaystyle{\bm{c}}}_{t}\\}$,
which are computed as follows:

$\displaystyle{\displaystyle{\bm{f}}}_{t}$ $\displaystyle=$
$\displaystyle\sigma({\displaystyle{\bm{W}}}^{fe}{\displaystyle{\bm{e}}}_{t}+{\displaystyle{\bm{W}}}^{fh}{\displaystyle{\bm{h}}}_{t-1}+{\displaystyle{\bm{b}}}^{f})$
(8) $\displaystyle{\displaystyle{\bm{i}}}_{t}$ $\displaystyle=$
$\displaystyle\sigma({\displaystyle{\bm{W}}}^{ie}{\displaystyle{\bm{e}}}_{t}+{\displaystyle{\bm{W}}}^{ih}{\displaystyle{\bm{h}}}_{t-1}+{\displaystyle{\bm{b}}}^{i})$
(9) $\displaystyle{\displaystyle{\bm{o}}}_{t}$ $\displaystyle=$
$\displaystyle\sigma({\displaystyle{\bm{W}}}^{oe}{\displaystyle{\bm{e}}}_{t}+{\displaystyle{\bm{W}}}^{oh}{\displaystyle{\bm{h}}}_{t-1}+{\displaystyle{\bm{b}}}^{o})$
(10) $\displaystyle{\displaystyle{\bm{z}}}_{t}$ $\displaystyle=$
$\displaystyle
tanh({\displaystyle{\bm{W}}}^{ze}{\displaystyle{\bm{e}}}_{t}+{\displaystyle{\bm{W}}}^{zh}{\displaystyle{\bm{h}}}_{t-1}+{\displaystyle{\bm{b}}}^{z})$
(11) $\displaystyle{\displaystyle{\bm{c}}}_{t}$ $\displaystyle=$
$\displaystyle{\displaystyle{\bm{f}}}_{t}\odot{\displaystyle{\bm{c}}}_{t-1}+{\displaystyle{\bm{i}}}_{t}\odot{\displaystyle{\bm{z}}}_{t}$
(12) $\displaystyle{\displaystyle{\bm{h}}}_{t}$ $\displaystyle=$
$\displaystyle{\displaystyle{\bm{o}}}_{t}\odot
tanh({\displaystyle{\bm{c}}}_{t}),$ (13)

where, as before, ${\displaystyle{\bm{W}}}^{**}$ are weight matrices and
${\displaystyle{\bm{b}}}^{*}$ are bias vectors. The $\odot$ operator denotes
an element-wise product, and $\sigma$ is the logistic sigmoid function. For
convenience, we summarize the joint effect of equations 8-13 as:

${\displaystyle{\bm{h}}}_{t}=LSTM({\displaystyle{\bm{e}}}_{t},\\{{\displaystyle{\bm{h}}}_{t-1};{\displaystyle{\bm{c}}}_{t-1}\\}).$
(14)

In the literature it is quite common to stack several LSTM layers. Here we
consider a double-layer LSTM, where the output ${\displaystyle{\bm{h}}}_{t}$
of the recurrent module is obtained by the concatenated application of two
LSTM layers:

$\displaystyle{\displaystyle{\bm{h}}}^{\prime}_{t}$ $\displaystyle=$
$\displaystyle
LSTM_{1}({\displaystyle{\bm{e}}}_{t},\\{{\displaystyle{\bm{h}}}^{\prime}_{t-1};{\displaystyle{\bm{c}}}^{\prime}_{t-1}\\})$
(15) $\displaystyle{\displaystyle{\bm{h}}}_{t}$ $\displaystyle=$
$\displaystyle
LSTM_{2}({\displaystyle{\bm{h}}}^{\prime}_{t},\\{{\displaystyle{\bm{h}}}_{t-1};{\displaystyle{\bm{c}}}_{t-1}\\}).$
(16)

We refer to this double LSTM module as $dLSTM$:

$\displaystyle{\displaystyle{\bm{h}}}_{t}$ $\displaystyle=$ $\displaystyle
dLSTM({\displaystyle{\bm{e}}}_{t},\\{{\displaystyle{\bm{h}}}_{t-1};{\displaystyle{\bm{c}}}_{t-1};{\displaystyle{\bm{h}}}^{\prime}_{t-1};{\displaystyle{\bm{c}}}^{\prime}_{t-1}\\})$
(17) $\displaystyle=$ $\displaystyle
LSTM_{2}(LSTM_{1}({\displaystyle{\bm{e}}}_{t},\\{{\displaystyle{\bm{h}}}^{\prime}_{t-1};{\displaystyle{\bm{c}}}^{\prime}_{t-1}\\}),\\{{\displaystyle{\bm{h}}}_{t-1};{\displaystyle{\bm{c}}}_{t-1}\\}).$
(18)

###  2.2 The Mogrifier-LSTM module

The Mogrifier-LSTM [12] is one of the state-of-the-art variations of the
standard LSTM architecture achieving the highest perplexity scores in language
modeling tasks. It basically consists of a standard LSTM block, but the input
${\displaystyle{\bm{e}}}_{t}$ and the hidden state
${\displaystyle{\bm{h}}}_{t-1}$ are transformed before entering equations
8-13. The mogrifier transformation involves several steps where
${\displaystyle{\bm{e}}}_{t}$ and ${\displaystyle{\bm{h}}}_{t-1}$ modulate
each other:

$\displaystyle{\displaystyle{\bm{e}}}_{t}^{i}$ $\displaystyle=$ $\displaystyle
2\sigma({\displaystyle{\bm{Q}}}^{i}{\displaystyle{\bm{h}}}_{t-1}^{i-1})\odot{\displaystyle{\bm{e}}}_{t}^{i-2},\;\;\;\;\;\textrm{for
odd}\;i\in\\{1,2,...,r\\}$ (19)
$\displaystyle{\displaystyle{\bm{h}}}_{t-1}^{i}$ $\displaystyle=$
$\displaystyle
2\sigma({\displaystyle{\bm{R}}}^{i}{\displaystyle{\bm{e}}}_{t}^{i-1})\odot{\displaystyle{\bm{h}}}_{t-1}^{i-2},\;\;\;\;\;\textrm{for
even}\;i\in\\{1,2,...,r\\},$ (20)

where ${\displaystyle{\bm{Q}}}^{i}$ and ${\displaystyle{\bm{R}}}^{i}$ are
weight matrices and we have
${\displaystyle{\bm{e}}}_{t}^{-1}={\displaystyle{\bm{e}}}_{t}$ and
${\displaystyle{\bm{h}}}_{t-1}^{0}={\displaystyle{\bm{h}}}_{t-1}$. The linear
transformations
${\displaystyle{\bm{Q}}}^{i}{\displaystyle{\bm{h}}}_{t-1}^{i-1}$ and
${\displaystyle{\bm{R}}}^{i}{\displaystyle{\bm{e}}}_{t}^{i-1}$ can also
include the addition of a bias vector, which has been omitted for the sake of
clarity. The constant $r$ is a hyperparameter whose value defines the number
of rounds of the transformation. We refer to this recurrent module, including
the mogrifier transformation and the subsequent application of the LSTM layer,
as:

${\displaystyle{\bm{h}}}_{t}=mLSTM({\displaystyle{\bm{e}}}_{t},\\{{\displaystyle{\bm{h}}}_{t-1};{\displaystyle{\bm{c}}}_{t-1}\\})=LSTM({\displaystyle{\bm{e}}}_{t}^{*},\\{{\displaystyle{\bm{h}}}_{t-1}^{*};{\displaystyle{\bm{c}}}_{t-1}\\}),$
(21)

where ${\displaystyle{\bm{e}}}_{t}^{*}$ and
${\displaystyle{\bm{h}}}_{t-1}^{*}$ are the highest indexed
${\displaystyle{\bm{e}}}_{t}^{i}$ and ${\displaystyle{\bm{h}}}_{t-1}^{i}$ in
equations 19 and 20. Note that the choice $r=0$ recovers the standard LSTM
model.

[12] also used a double-layer LSTM enhanced with the mogrifier transformation.
This strategy can be summarized as follows:

$\displaystyle{\displaystyle{\bm{h}}}_{t}$ $\displaystyle=$ $\displaystyle
mdLSTM({\displaystyle{\bm{e}}}_{t},\\{{\displaystyle{\bm{h}}}_{t-1};{\displaystyle{\bm{c}}}_{t-1};{\displaystyle{\bm{h}}}^{\prime}_{t-1};{\displaystyle{\bm{c}}}^{\prime}_{t-1}\\})$
(22) $\displaystyle=$ $\displaystyle
mLSTM_{2}(mLSTM_{1}({\displaystyle{\bm{e}}}_{t},\\{{\displaystyle{\bm{h}}}^{\prime}_{t-1};{\displaystyle{\bm{c}}}^{\prime}_{t-1}\\}),\\{{\displaystyle{\bm{h}}}_{t-1};{\displaystyle{\bm{c}}}_{t-1}\\}).$
(23)

##  3 Experiments

###  3.1 Datasets

We perform experiments on two datasets: the Penn Treebank corpus [11], as
preprocessed by [15], and the WikiText-2 dataset [14]. In both cases, the data
are used without any additional preprocessing.

The Penn Treebank (PTB) dataset has been widely used in the literature to
experiment with language modeling. The standard data preprocessing is due to
[15], and includes transformation of all letters to lower case, elimination of
punctuation symbols, and replacement of all numbers with a special token. The
vocabulary is limited to the 10,000 most frequent words. The data is split
into a training set which contains almost 930,000 tokens, and validation and
test sets with around 80,000 words each.

The WikiText-2 (WT2) dataset, introduced by [14], is a more realistic
benchmark for language modeling tasks. It consists of more than 2 million
words extracted from Wikipedia articles. The training, validation and test
sets contain around 2,125,000, 220,000, and 250,000 words, respectively. The
vocabulary includes over 30,000 words, and the data retain capitalization,
punctuation, and numbers.

###  3.2 Experimental setup

All the considered models follow one of the two architectures discussed in
section 2, either the Embedding-Recurrent-Softmax (ERS) architecture
(equations 1-3) or the dual architecture (equations 4-7). In either case, the
recurrent module can be any of $LSTM$, $dLSTM$, or $mdLSTM$. Weight tying [8,
17] is used to couple the weight matrices of the embedding and the output
layers. This reduces the number of parameters and prevents the model from
learning a one-to-one correspondence between the input and the output [13].

We run two different sets of experiments. First, we analyze the effect of the
dual connection by comparing the performances of the two architectures (ERS vs
Dual), using each of the recurrent modules, on both the PTB and the WT2
datasets. In this setting the hyperparameters are tuned for the ERS
architecture, and then transferred to the dual case. Second, we search for the
best hyperparameters for the dual architecture using the $mdLSTM$ recurrence,
and compare the perplexity score with current state-of-the-art values. All the
experiments have been performed using the Keras library [4].

The networks are trained using the Nadam optimizer [5], a variation of Adam
[9] where Nesterov momentum is applied. The number of training epochs is
different for each experimental condition. On one hand, when the objective is
to perform a pairwise comparison between dual and non-dual architectures, we
train the models for 100 epochs. On the other hand, when the goal is to
compare the dual network with state of the art approaches, we let the models
run for 300 epochs. We use batch sizes of 32 and 128 for the PTB and the WT2
problems, respectively, and set the sequence length to 25 in all cases. The
remaining hyperparameters are searched in the ranges described in tables 3 and
4 in appendix A.

Finally, all the models are run twice, both with and without dynamic
evaluation [10]. Dynamic evaluation is a standard method commonly used to
adapt the model parameters, learned during training, using also the validation
data. This allows the networks to get adapted to the new evaluation
conditions, which in general improves their performance. In order to keep the
models as simple as possible, no additional modifications have been
considered.

##  4 Results

We first show the results of the comparative analysis ERS vs Dual, then we
focus on the search of the optimal hyperparameters for the dual architecture
with the $mdLSTM$ recurrence.

###  4.1 Dual vs non-dual architectures

Table 1 displays the validation and test perplexity scores obtained for each
of the experimental configurations on the PTB and the WT2 problems, both with
and without dynamic evaluation. To facilitate the comparison, each pair of
rows contain the results for one of the recurrent modules ($LSTM$, $dLSTM$ or
$mdLSTM$) using the two architectures ERS and Dual, with the best values shown
in bold. In each case, the hyperparameters are tuned for the standard ERS
architecture and then used within the dual networks without any additional
adaptation. The exceptions are hyperparameters, such as the dual dropout,
which do not exist in the ERS configuration (those marked with an asterisk in
tables 3 and 4 in appendix A). To give a measure of the model complexity,
table 1 contains also the approximate number of trainable parameters for each
configuration.

Table 1: Validation and test word-level perplexity obtained for each of the
experimental configurations on the PTB (top) and the WT2 (bottom) datasets.
Penn Treebank Dataset
---
|  | No Dyneval | Dyneval
MODEL | No. PARAMS | Val. | Test | Val. | Test
LSTM | 8.88 M | 67.37 | 64.91 | 62.31 | 61.17
Dual LSTM | 9.60 M | 61.22 | 59.39 | 55.26 | 54.69
dLSTM | 13.62 M | 63.44 | 61.03 | 57.18 | 56.01
Dual dLSTM | 13.94 M | 60.99 | 59.56 | 56.11 | 54.87
mdLSTM | 21.43 M | 56.04 | 54.17 | 49.62 | 48.87
Dual mdLSTM | 22.88 M | 54.16 | 52.16 | 46.91 | 46.09
WikiText-2 Dataset
|  | No Dyneval | Dyneval
MODEL | No. PARAMS | Val. | Test | Val. | Test
LSTM | 20.23 M | 92.84 | 88.28 | 74.98 | 69.42
Dual LSTM | 20.95 M | 85.88 | 82.48 | 61.94 | 57.61
dLSTM | 29.60 M | 78.65 | 75.60 | 63.26 | 59.42
Dual dLSTM | 30.32 M | 77.01 | 73.90 | 61.10 | 57.10
mdLSTM | 37.51 M | 72.05 | 69.06 | 57.42 | 53.93
Dual mdLSTM | 38.95 M | 71.78 | 70.83 | 53.48 | 50.71

As expected, dynamic evaluation improves the results regardless of the model
or the dataset. The main observation, however, is that networks enhanced with
the dual connection display lower perplexity scores for almost all the
training conditions on both the PTB and the WT2 datasets. The advantage of the
Dual vs the ERS architecture is larger for less complex models, and narrows as
the model complexity increases. Nevertheless, even for networks with $mdLSTM$
recurrence, the dual architectures outperform their non-dual counterparts in
more than 2 perplexity points on the test set, when dynamic evaluation is
used.

Finally, it is worth noting that all the results presented correspond to our
own implementation of the models, and that in most cases we are not including
some of the several training or validation adaptations frequently used in the
literature (such as AWD or MoS, for example). This can explain the difference
with respect to the results reported by [12] for the Mogrifier-LSTM model. We
would expect a further improvement of the results were these additional
mechanisms implemented.

###  4.2 Dual Mogrifier fine tuning

The second part of the experiments consists of searching for the best
hyperparameters in the configuration that provided the smallest perplexity in
the previous setup, that is the Dual mdLSTM architecture. We carry out this
experiment with the PTB problem. After an extensive search (see tables 3 and 4
in appendix A), the best performance is obtained with a model with 850 units
in the embedding layer, 850 units in each of the mogrifier LSTM layers, and
850 units also in the dual layer. The input, recurrent, internal, and output
dropout rates are all set to 0.5, the dual input and output dropout rates are
set to 0.5 and 0.4, respectively, and the mogrifier dropout rate is set to
0.15. Both the embedding and the dual L2 regularization parameters are set to
$10^{-5}$. The mogrifier number of rounds is set to 4, and the rank to 100.
All the remaining hyperparameters are set to 0.

After the training phase, we continue with a fine tuning of some additional
hyperparameters, using the validation data. First, we look for the best
sequence length in the range $[5,70]$, and then we fine-tune the softmax
temperature in the range $[0.9,1.3]$. When using dynamic evaluation, we also
look for the best gradient clipping value (in the range $[0.0,1.0]$) and,
following [12], we repeat the whole procedure with the $\beta_{1}$ parameter
of the Nadam optimizer set to $0$, which resembles the RMSProp optimizer
without momentum. The results are shown in table 2, together with the top
perplexity scores reported in the literature for the same problem.

Table 2: Best validation and test word-level perplexity scores reported in the literature for the Penn Treebank dataset, with and without dynamic evaluation. Missing values in the last two columns correspond to works where the dynamic evaluation approach was not considered. The last row in the table displays the results obtained with our Dual mdLSTM network. |  | No Dyneval | Dyneval
---|---|---|---
MODEL |  | Val. | Test | Val. | Test
AWD-LSTM [13] | 24 M | 60.00 | 57.30 | - | -
AWD-LSTM-DOC [19] | 23 M | 54.12 | 52.38 | - | -
AWD-LSTM [10] | 24 M | 59.80 | 57.70 | 51.60 | 51.10
AWD-LSTM+PDR [1] | 24 M | 57.90 | 55.60 | 50.10 | 49.30
AWD-LSTM+MoS [21] | 22 M | 56.54 | 54.44 | 48.33 | 47.69
AWD-LSTM+MoS+PDR [1] | 22 M | 56.20 | 53.80 | 48.00 | 47.30
AWD-LSTM-DOC x5 [19] | 185 M | 48.63 | 47.17 | - | -
AWD-LSTM+MoS+FRAGE [6] | 24 M | 55.52 | 53.51 | 47.38 | 46.54
AWD-LSTM+MoS+Adv [20] | 22 M | 54.98 | 52.87 | 47.15 | 46.52
AWD-LSTM+MoS+Adv+PS [20] | 22 M | 54.10 | 52.20 | 46.63 | 46.01
Mogrifier-LSTM [12] | 24 M | 51.40 | 50.10 | 44.90 | 44.80
Dual mdLSTM \- ours | 23 M | 52.87 | 51.19 | 45.13 | 44.61

The state-of-the-art is dominated by several variations of the AWD-LSTM
network [13], the most common being the inclusion of a Mixture of Softmaxes
(MoS) [21]. Other add-ons include Direct Output Connection (DOC) [19], which
is a generalization of MoS, Frequency Agnostic word Embedding (FRAGE) [6],
Past Decode Regularization (PDR) [1], or Partial Shuffling (PS) with
Adversarial Training (Adv) [20]. The mogrifier-LSTM described in section 2.2
combines many of these ideas with a mutual gating between the input and the
hidden state vectors to obtain the best results reported in the literature for
the PTB problem, among those obtained by networks that do not use additional
data during the training phase. Compared with all these models, our current
approach leads the ranking with a perplexity score of $44.61$, even though
most of the aforementioned tricks have not been considered.

##  5 Discussion

In this work, we have presented a new network design for the Language Modeling
task based on the dual network proposed by [16]. This network adds a direct
connection between the input and the output, skipping the recurrent module,
and can be adapted to any of the traditional Embedding-Recurrent-Softmax (ERS)
models, opening the way to new approaches for this task. We have based our
work on the Penn Treebank [15] and the WikiText-2 [14] datasets, comparing the
ERS approach and its dual alternative. Regardless of the configuration, the
dual version performs always better, even though it faces a slight
disadvantage, since most of the hyperparameters are tuned using the ERS model.
We can expect a much better performance if the complete set of hyperparameters
is properly tuned for the dual network.

This is in fact the case for the second experiment, where a Dual mdLSTM, which
includes a simplified version of the mogrifier LSTM [12] within a dual
architecture, is fine tuned for the Penn Treebank dataset. After a thorough
search of the hyperparameters space, we have found a network configuration
that establishes a new state-of-the-art score for this problem. Interestingly,
this new record has been obtained in spite of leaving aside many of the
standard features used in most state-of-the-art approaches, such as the
Averaged SGD Weight-Drop (AWD) [13] or the Mixture of Softmaxes (MoS) [21].
The incorporation of these features into the dual architecture can be expected
to further increase the model performance.

The dual architecture was firstly proposed as an alternative that reduces the
computational load on the recurrent layer, letting it concentrate on modeling
the temporal dependencies only. From a more abstract point of view, it has
been argued that the dual architecture can be understood as a sort of Mealy
machine, where the output explicitly depends on both the hidden state and the
input [16]. Our results show that this explicit dependence on the input can
indeed lead to better performance on language modeling tasks. This emphasizes
the importance of the current input in RNN models.

Finally, although the new approach has not been tested with large-scale
language corpora, we expect that our results scale well to larger datasets.
Work in progress contemplates this extension. The dual architecture also needs
further research concerning the deepness of the specific variations of
Language Modeling and other families of problems not necessarily related to
Natural Language Processing. This work opens a new line of research to be
considered when processing any sequence or time series. The utility of this
approach in more general problems will be addressed in future work.

## References

  * [1] S. Brahma.  Improved language modeling by decoding the past.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1468–1476, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [3] K. Cho, B. van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  In A. Moschitti, B. Pang, and W. Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724–1734. ACL, 2014.
  * [4] F. Chollet et al.  Keras.  https://keras.io, 2015.
  * [5] T. Dozat.  Incorporating nesterov momentum into adam.  In Proceedings of 4th International Conference on Learning Representations, Workshop Track, 2016.
  * [6] C. Gong, D. He, X. Tan, T. Qin, L. Wang, and T.-Y. Liu.  Frage: Frequency-agnostic word representation.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [7] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, Nov. 1997.
  * [8] H. Inan, K. Khosravi, and R. Socher.  Tying word vectors and word classifiers: A loss framework for language modeling.  In Proceedings of the 5th International Conference on Learning Representations, 2017.
  * [9] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [10] B. Krause, E. Kahembwe, I. Murray, and S. Renals.  Dynamic evaluation of neural sequence models.  In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2766–2775. PMLR, 10–15 Jul 2018.
  * [11] M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini.  Building a large annotated corpus of english: The penn treebank.  Comput. Linguist., 19(2):313–330, June 1993.
  * [12] G. Melis, T. Kočiský, and P. Blunsom.  Mogrifier LSTM.  In International Conference on Learning Representations, 2020.
  * [13] S. Merity, N. S. Keskar, and R. Socher.  Regularizing and optimizing LSTM language models.  In International Conference on Learning Representations, 2018.
  * [14] S. Merity, C. Xiong, J. Bradbury, and R. Socher.  Pointer sentinel mixture models.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
  * [15] T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and S. Khudanpur.  Recurrent neural network based language model.  In T. Kobayashi, K. Hirose, and S. Nakamura, editors, INTERSPEECH, pages 1045–1048. ISCA, 2010.
  * [16] C. Oliva and L. F. Lago-Fernández.  Separation of memory and processing in dual recurrent neural networks.  In I. Farkaš, P. Masulli, S. Otte, and S. Wermter, editors, Artificial Neural Networks and Machine Learning – ICANN 2021, pages 360–371, Cham, 2021. Springer International Publishing.
  * [17] O. Press and L. Wolf.  Using the output embedding to improve language models.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163, Valencia, Spain, Apr. 2017. Association for Computational Linguistics.
  * [18] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [19] S. Takase, J. Suzuki, and M. Nagata.  Direct output connection for a high-rank language model.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4599–4609, Brussels, Belgium, Oct.-Nov. 2018\. Association for Computational Linguistics.
  * [20] D. Wang, C. Gong, and Q. Liu.  Improving neural language modeling via adversarial training.  In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6555–6565. PMLR, 09–15 Jun 2019\.
  * [21] Z. Yang, Z. Dai, R. Salakhutdinov, and W. W. Cohen.  Breaking the softmax bottleneck: A high-rank RNN language model.  In International Conference on Learning Representations, 2018.

##  Appendix A Appendix

Table 3: List of all the hyperparameters and the search range associated with each of them. Those marked with an asterisk $(^{*})$ refer to the dual architectures only. Name | Description | Values
---|---|---
Num epochs | Number of training epochs. | {100, 300}
Learning rate | Learning rate. | [10-6, 10-3]
Batch size | Batch size. | {32, 128}
Seq len | Sequence length. | {10, 25, 50}
Embedding units | Size of the embedding layer. | {400, 850}
Recurrent units | Size of the recurrent layers. | {400, 850, 1150}
LSTM layers | Number of recurrent layers. | {1, 2, 3}
Dual units∗ | Size of the dual layer. | {400, 850}
Embedding L2reg | L2 regularization applied to | {0, $10^{-6}$, $10^{-5}$}
| the Embedding and output |
| layers. |
Rec. input L2reg | L2 regularization applied to | {0, $10^{-6}$, $10^{-5}$}
| the input weights of the |
| recurrent layer. |
Rec. L2reg | L2 regularization applied to | {0, $10^{-6}$, $10^{-5}$}
| the recurrent weights of the |
| recurrent layer. |
Activation L2reg | L2 regularization applied to | {0, $10^{-6}$, $10^{-5}$}
| the recurrent layers output. |
Dual L2reg∗ | L2 regularization applied to | {0, $10^{-6}$, $10^{-5}$}
|  dual layer. |
Table 4: List of all the hyperparameters and the search range associated with each of them. Those marked with an asterisk $(^{*})$ refer to the dual architectures only. Name | Description | Values
---|---|---
Rec. input Drop. | Dropout before the first | [0.0, 0.5]
| recurrent layer. |
Rec. Drop. | Dropout for the linear | [0.0, 0.5]
| transformation of the recurrent |
| state. |
Rec. internal Drop. | Dropout between the recurrent | [0.0, 0.5]
| layers. |
Rec. output Drop. | Dropout after the last recurrent | [0.0, 0.5]
| layer. |
Dual input Drop.∗ | Dropout before the dual layer. | [0.0, 0.5]
Dual output Drop.∗ | Dropout after the dual layer. | [0.0, 0.5]
Mogrifier deep | Mogrifier rounds. | {0, 2, 3, 4, 5, 6}
Mogrifier L2reg | L2 regularization applied to | {0, $10^{-6}$, $10^{-5}$}
| Mogrifier weights. |
Mogrifier rank | Weight factorization. ${\displaystyle{\bm{Q}}}^{i}={\displaystyle{\bm{Q}}}^{i}_{l}{\displaystyle{\bm{Q}}}^{i}_{r}$ | {0, 50, 100, 200}
| with ${\displaystyle{\bm{Q}}}^{i}_{l}\in\displaystyle\mathbb{R}^{m\times k},{\displaystyle{\bm{Q}}}^{i}_{r}\in\displaystyle\mathbb{R}^{k\times n}$. |
Mogrifier Drop. | Dropout between the Mogrifier | [0.0, 0.2]
| weights. |
Learning rate eval | Learning rate when Dynamic | [10-6, 10-3]
| evaluation. |
Seq len eval | Sequence length when Dynamic | [5, 50]
| evaluation. |
Clipnorm eval | Gradients clipping to a maximum | [0.0, 1.0]
| norm. |


# Learning to Detect: A Data-driven Approach for Network Intrusion Detection

Zachary Tauscher, Yushan Jiang, Kai Zhang, Jian Wang, Houbing Song  Department
of Electrical Engineering & Computer Science
Embry-Riddle Aeronautical University
Daytona Beach, FL 32114 USA
{tauschez, jiangy2, zhangk3<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

With massive data being generated daily and the ever-increasing
interconnectivity of the world’s Internet infrastructures, a machine learning
based intrusion detection system (IDS) has become a vital component to protect
our economic and national security. In this paper, we perform a comprehensive
study on NSL-KDD, a network traffic dataset, by visualizing patterns and
employing different learning-based models to detect cyber attacks. Unlike
previous shallow learning and deep learning models that use the single
learning model approach for intrusion detection, we adopt a hierarchy
strategy, in which the intrusion and normal behavior are classified firstly,
and then the specific types of attacks are classified. We demonstrate the
advantage of the unsupervised representation learning model in binary
intrusion detection tasks. Besides, we alleviate the data imbalance problem
with SVM-SMOTE oversampling technique in 4-class classification and further
demonstrate the effectiveness and the drawback of the oversampling mechanism
with a deep neural network as a base model.

###### Index Terms:

Intrusion Detection System, Machine Learning, Data Analytics, Computer
Networks, NSL-KDD

##  I Introduction

Global communication and networking are commonplace in the current era.
Everything from cell phones to thermostats is connected to the internet. A
large number of users and devices connected to the internet makes the security
risk to these networks only that much greater. The ability to detect and
prevent network attacks is vital to maintain the confidentiality, integrity,
and availability of our information and communication systems. Network
intrusion detection and prevention systems (IDS/IPS) are a critical part of
any network or system architecture designed to record and analyze connection
behavior to identify possible attacks and report such information to an
administrator or prevent the attack entirely.

IDPS technologies vary in their methodologies for detecting intrusions but
tend to fall into two specific categories, signature-based and anomaly-based
detection. Signature-based IDS systems, also known as misuse IDS, have been
the most widely used due to their simplicity and reliability. These types of
systems utilize pattern recognition to compare signatures of well-known
attacks to current connections [1]. Anomaly-based IDS technology analyzes
normal network traffic to develop models of normal behavior. Any connections
that then deviate from these models are flagged as an intrusion. Anomaly-based
IDS often produce a high volume of false positives as any activity that
deviates from the normal is flagged. So while signature-based IDS is more
often used, anomaly-based IDS has greater potential power, especially as
machine learning and AI models continue to develop and become a greater focus
in cybersecurity [2].

With the development of capable AI-driven IDS/IPS technologies, there have
been various studies investigating and developing data-driven methods. Besides
the data analytics and pattern presentations using traditional visualization
techniques [3] and unsupervised K-means clustering [4] , the method toward
detecting attacks can be mainly divided into two parts, the classical machine
learning classifiers and deep learning models. In classical learning methods,
several classifiers are utilized and modified for binary and multi-class
intrusion detection tasks [5, 6, 7, 8], including basic tree methods, Multi-
layer Perceptron, and Support Vector Machine, Naive Bayes, Random Forest, and
a sophisticated variant of boost-based classifiers. Besides, feature selection
is leveraged to choose the informative subset of features to facilitate the
performance of classifiers, which is based on different techniques including
Flexible Neural Tree [9], visualization techniques of distribution histograms,
scatter plots, and information gain [10]. In deep learning methods, besides
the deep neural networks [11], Convolutional Neural Networks [12], Recurrent
Neural Networks [13], and their integration [14] are applied to capture
certain spatial characteristics and temporal dependencies in an individual or
joint manner, for downstream intrusion classification task. Moreover, to
tickle the data insufficient issues in some datasets, transfer learning [15]
and Variational Autoencoder [16] are also considered in terms of
representation transferring and learning, which generalizes the learning-based
methods to a wider range of applications.

TABLE I: Attack Category and Its Statistics in NSL-KDD dataset Attack Category |  Description |  Attack Type & Count
---|---|---
Probe |  A process of probing target computer network to find weaknesses in its defense |  satan (3633), portsweep (2931), nmap (1493), jpsweep (3599)
R2L |  Unauthorized access from a remote machine to a local machine |  spy (2), phf (4), multihop (7), imap (11), guess_passwd (53), ftp_write (8), warezmaster (20), warezclient (890)
U2R |  Unauthorized access to local superuser privileges by a local unprivileged user |  rootkit (10), perl (3), loadmodule (9), buffer_overflow (30),
DoS |  Oversaturating connection bandwidth or depleting the target’s system resources |  teardrop (892), smurf (2646), pod (201), neptune (41214), land (18), back (956)
Figure 1: Distribution histograms of specified features in the NSL-KDD data
set.

In this paper, we aim at developing a data-driven intrusion detection
framework to analyze and classify the patterns of normal network status and
various malicious attacks. To be specific, we perform exploration on the NSL-
KDD dataset representing real-world network traffic, by visualizing and
analyzing potential patterns so that preliminary decision making and
manipulation can be taken. Moreover, we adopt a two-stage hierarchy strategy
based on machine learning models for intrusion detection tasks, where the
attacks with abnormal patterns are first detected from normal samples, then
further classified into different types. At the first stage, we adopt
supervised classifiers and a representation learning model to detect
anomalies. As the intrusion data suffers from a severe imbalance problem, we
first leverage an oversampling technique at the second stage, then utilize a
deep neural network to classify the attack type in a supervised manner.

##  II Data Exploration

In this section, we present the exploration of the NSL-KDD dataset, which
includes the description, pattern visualization, and analytic.

###  II-A Dataset Description

NSL-KDD is the refined version of the KDD’99 [17] data set to solve its
inherent problems. For example, it does not contain redundant records so that
the model training and evaluation is not biased by high-frequently duplicated
records. Moreover, the number of selected records from each difficulty level
group is inversely proportional to the percentage of records in the KDD’99
data set, which results in varying classification rates of different machine
learning and it facilitates the analysis of distinct learning techniques.

####  II-A1 Labels

The label of each instance in the NSL-KDD is assigned as either normal or an
attack, with exactly one specific attack type. The attacks fall in one of
following categories in Table I with the statistics summary of specific attack
types, where R2L represents the Remote-To-Local Attack, U2R represents the
User-To-Root Attack, and DoS represents the Denial-of-Service Attack.

####  II-A2 Features

There are 41 features in the NSL-KDD data set that describe the
characteristics of the cyber network, which can be further divided into three
groups, consisting of 9 basic features, 19 traffic features, and 13 content
features. The basic features involve all attributes that can be captured from
a TCP/IP connection; the traffic features contain ”same host” and ”same
service” features based on a connection window of 100 connections; the content
features are related to suspicious behavior in the data portion like the
number of failed login attempts. Generally speaking, traffic features are
useful patterns to identify the DoS and probing attacks since they need to
scan the hosts or send packets (many connections to some hosts) within a very
short period of time. On the contrary, R2L and U2R attacks do not have any
intrusion-frequent sequential patterns but are embedded in the data portions
of the packets in a single connection. Hence, the content features are better
patterns that can be used to detect these two attacks. A detailed explanation
of each attribute is described in [18].

###  II-B Data Visualization

To further understand and explore the NSL-KDD data set we used visualization
techniques. Data visualization is the practice of graphically representing
data. Using such methods we can gain insight and make better sense of large
data sets By visualizing the NSL-KDD data set we gain a greater understanding
of the features with relation to each other and attack type classification.
For this investigation, we visualized the training data set to split up by
attack type.

Our First steps in visualization are distribution histograms of the features
in the data set, shown in Fig. 1. Distribution histograms plot the value of a
feature against its occurrence in the data. From these graphs, we gain
valuable information on issues within the data set, redundant features, and
how features relate to different attack types. One of the first things we
noticed during our initial investigation was the lack of instances of U2R and
R2L attack types within distribution graphs. This is due to the small number
of examples of these attacks within the data set. Further, we were able to
discover some redundant features within the data set. Features 20 and 21
always have a value of zero, feature 20, the number of outbound commands. With
a closer investigation of individual features, we can gain some insight into
how specific features correlate to specific attack types. Fig. 1 shows some
examples as to how these features correlate to specific attack types. We can
see that most attacks use TCP. TCP has many vulnerabilities often exploited by
attackers. DOS attacks often take advantage of the TCP handshake protocol by
flooding a target host with incomplete connection and service requests in the
hopes to waste server resources. As the server or host is attempting to handle
a large number of connections from the attacker, it is not able to handle the
requests of legitimate users thus rending the host inaccessible. This is
reflected in the distribution graphs of connection count, which shows a large
number of connections, and Serror Rate, which shows if those connection
attempts had no further replies. Another example shown in Fig. 1 is with the
Dst Host Srv Diff Host Rate histogram, which shows a correlation to the probe
attack type. This graph shows the percent of connections to different
destination machines from the same port number. Probe attacks will provide
information on what each port is doing and what is using that port by sending
information and waiting for a response. This nature is reflected in this graph
as Probe attacks show up in greater numbers as this feature increases.

We further investigated the data set by calculating the correlation
coefficient of the features compared to each other which can be seen in Fig.
2. This provides us insight into the strength of the relationship between the
two figures. The greater the correlation the closer the value is to -1, or 1.
Fig. 2 shows a strong correlation between higher-level figures which shows
that these higher-level figures have a higher potential to provide
information. These higher-level figures are often based on each other which
can also explain why they have such high significance. Examples of these
relationships are shown in the scatter plots of Fig. 3. Here we can see
correlation between higher level features which provide information on he
probe attack type. We can see that the probe attack type often threw the REJ
flag, but this percentage was often affected by the number of services the
probe reached out to. The further a probe attacked reached the greater its
connection attempt was rejected. By observing these relations more directly we
can determine the importance of some feature pairings towards anomaly
detection and attack type classification.

Figure 2: Correlation heat map of features in the NSL-KDD data set. Figure 3:
Scatter plots of specific features in the NSL-KDD data set.

##  III Methodology

In this section, we present the pipeline of our detection mechanism. Firstly,
we present the preprocessing details and imbalanced nature of the explored
dataset. After that, we also present a detection strategy and briefly describe
the utilized machine learning algorithms. We adopt a hierarchy two-stage
detection method: a binary classification followed by a 4-class
classification. In the first stage, each input is classified into two normal
samples and anomalies. We will explore different learning algorithms,
including supervised learning classifiers and an unsupervised learning model,
autoencoder. In the second stage, anomalies are further classified into four
main attack categories (DoS, Probe, R2L, U2R), where supervised learning
models are leveraged.

###  III-A Preprocessing

Although the NSL-KDD data set is a cleansed data set, we still need
preliminary preprocessing feature engineering before the data is fed into the
model. To be specific, the categorical features should be converted into the
numerical form so that they can be thought of as a vector in the Euclidean
space: Three attributes (’protocol_type’,’service’,and ’flag’) are
categorical, we encode them by using a LabelCount encoder which sorts the
categories by the frequency of each category within the feature. LabelCount
has specific advantages at the outlier-insensitive nature and a reduction of
dimensionality when certain features have very large numbers of categories.

After assigning numerical values to each categorical feature, the next step is
to normalize each feature, as features that are measured at different scales
do not contribute equally to the analysis and can create a bias for models.
Therefore, the standardization as shown in Equation 1 is applied to transform
the data to comparable scales (around the center 0 with a standard deviation
of 1).

$Z=\frac{x-\mu}{\sigma}$ (1)

where $Z$ denotes the standardized feature, $x$ denotes each value within the
feature, $\mu,\sigma$ denotes the mean and standard deviation of all values,
respectively.

In this dataset, the number of examples across the classes for the binary
classification (normal vs. others) is roughly close. However, the 4-class
intrusion data suffers from a severe imbalance, as the ratio of each class is
approximately $920:220:20:1$. Most machine learning algorithms assume or
expect a balanced class distribution for pattern learning and downstream
classification tasks. When such data skewness exists, these algorithms fail to
properly represent the distributive characteristics of the data whose results
provide invalid accuracy across the classes of the data [19]. To alleviate the
negative effect of imbalanced data set, we employ SVM-SMOTE[20], a
sophisticated oversampling technique leveraging a Support Vector Machine
algorithm to detect sample to use for generating new synthetic samples under
the framework of Synthetic Minority Oversampling Technique (SMOTE) class [21].

Figure 4: Learning-based on hierarchy intrusion detection strategy.

###  III-B Learning to Detect Network Intrusion

In this paper, we adopt various learning models for binary and 4-class
intrusion detection. Fig. 4 depicts the overview of our IDS based on a
hierarchy machine learning strategy. For binary detection, we utilize
supervised learning models including Decision Tree, Random Forest, Naive
Bayes, Support Vector Machine (SVM), AdaBoost, Gradient Boosting, Multi-layer
Perceptron (MLP). Besides, we also consider an unsupervised representation
learning model, Autoencoder, and treat the conventional binary classification
problem as an anomaly detection problem. It learns the representation of
normal samples and uses the reconstruction error as the anomaly score. At the
testing stage, samples with high reconstruction (exceeding the threshold) are
considered anomalies, as it is assumed that anomalies are difficult to be
reconstructed [22]. The specific algorithm of Autoencoder-based detection is
shown in Algorithm 1. After anomalies are detected, we utilize a deep neural
network accompanied with the aforementioned oversampling technique to classify
the specific intrusion type in a more robust manner.

Algorithm 1 Autoencoder-based attack detection algorithm

0: Normal dataset $\mathbf{X}$, Anomalous dataset $x^{i}$, $i=1,...N$,
threshold $\alpha$ defined by validation loss

0: $f_{\theta}$ : Encoder, $g_{\phi}$ : Decoder

0: reconstruction errors, anomaly indicator

1: $g_{\phi},f_{\theta}\leftarrow$ train a Autoencoder with normal dataset
$\mathbf{X}$. LOOP Process

2: for $i=1$ to $N$ do

3: $\text{ reconstruction error
}(i)=\left\|x^{(i)}-g_{\theta}\left(f_{\phi}\left(x^{(i)}\right)\right)\right\|^{2}$

4: if reconstruction error $>\alpha$ then

5: $x^{i}$ is an anomaly (attack)

6: else if reconstruction error $<=\alpha$ then

7: $x^{i}$ is not an anomaly (attack)

8: end if

9: end for

##  IV Experiments

In this section, we evaluate the performance of different machine learning
algorithms for two hierarchic stages of our intrusion detection system. We
also explore the effectiveness of SVM-SMOTE oversampling technique toward a
more valid classification model.

###  IV-A Evaluation Metrics and Experiment Settings

In this paper, we adopt accuracy, precision, recall, and F1 score for a
comprehensive evaluation in the binary classification task. For 4-class
classification, we use accuracy, F1 score of each intrusion type (one vs.
all), with their macro-average (arithmetic mean) and micro-average (weighted
mean) to demonstrate the classification performance of our model with the
existence of imbalance [23].

$\text{Accuracy}=\frac{\mathrm{Number\;of\;correct\;predictions}}{\mathrm{Total\;number\;of\;predictions}}$
(2) $\text{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$ (3)
$\text{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$ (4)
$\mathrm{F1}=2\times\frac{\text{ Precision }\times\text{ Recall }}{\text{
Precision }+\text{ Recall }}$ (5)

where $\mathrm{TP}$ = True Positives, $\mathrm{TN}$ = True Negatives,
$\mathrm{FP}$ = False Positives, and $\mathrm{FN}$ = False Negatives.

Next, we present the settings of the experiment for Autoencoder in binary
classification and deep neural network in multi-class classification. The
Autoencoder has three layers with 15 neurons in hidden space and 0.15 Gaussian
Noise, and 0.05 Dropout rate in encoding layers for regularization. The
activation for both encoder and decoder is Scaled Exponential Linear Unit
(SeLU), and the loss function is Mean Squared Error (MSE). In a deep neural
network, three layers are used, where there are 80 neurons in the hidden layer
with Rectified Linear Unit (ReLU) as activation function, four neurons in the
output layer with Softmax as activation function. The loss function is cross-
entropy. For both tasks, the batch size is 32; 0.15 of training data are used
for validation; the optimizer is Adam with a learning rate 0.001; early
stopping is adopted with the patience of 6 steps.

###  IV-B Binary Classification

TABLE II: Performance Evaluation of Binary Classification in IDS Model | Accuracy | Precision | Recall | F1 Score
---|---|---|---|---
Decision Tree | 68.28% | 68.16% | 83.09% | 0.7489
Random Forest | 76.00% | 87.34% | 67.65% | 0.7624
Naive Bayes | 76.86% | 96.21% | 59.95% | 0.7387
SVM | 80.47% | 97.56% | 67.38% | 0.7971
AdaBoost | 79.40% | 86.90% | 75.14% | 0.8059
Gradient Boosting | 68.12% | 65.04% |  95.13% | 0.7726
MLP | 77.90% | 95.82% | 63.96% | 0.7671
Autoencoder | 87.52% | 93.20% | 84.22% | 0.8848

The evaluation of different models for binary classification is shown in Table
II, where the best result for each metric is indicated in bold while the
corresponding second-best result is underlined. Among these learning models,
it can be observed that SVM yields the highest precision score and the second
highest accuracy, while Gradient Boosting classifier demonstrates its
advantage on the highest recall score with a clear margin to the second
highest one. In terms of the F1 score, AdaBoost and SVM achieve similarly good
performance among supervised learning models. The above results illustrate
that SVM and boosting methods gain more favors at separating attacks from
normal samples within the scope of supervised learning.

On the other hand, the Autoencoder shows a huge advantage in the binary
classification task, as it yields the highest accuracy, F1 score, and the
second-highest recall score and. Moreover, the increase of accuracy and F1
score from the second-best result is 7.05% and 0.0789, respectively,
demonstrating the power of unsupervised representation learning. To evaluate
the classification performance with more details, the confusion matrix of
Autoencoder is also presented, as shown in Fig.5. It is clear that Autoencoder
performs slightly better on identifying normal behaviors than malicious
attacks.

In general, the implemented Autoencoder with Gaussian noise and dropout as the
regularization method is a good alternative in binary classification and
anomaly detection. It is able to extract the feature information and generate
salient and generalized vector representations for the reconstruction of
normal samples only, where the anomalies usually do not share the similar
representation space and fail to be reconstructed with a much higher loss
during the model inference.

Figure 5: Confusion matrix of Autoencoder based binary classifier.

###  IV-C 4-Class Intrusion Type Classification

At this stage, the samples identified as intrusion are further classified into
different types. Table III shows the performance evaluation using a deep
neural network for 4-class classification, without or with the aforementioned
SVM-SMOTE oversampling method. It can be observed that the oversampling method
has a minor impact on accuracy and micro F1 score, with the value around 80.4%
and 0.783, respectively. However, it is clear that the new synthetic samples
generated by SVM-SMOTE help the model to learn patterns and significantly
improves the F1 score of U2R and yields an increase of macro F1 score by 0.1,
with a slightly better F1 score of R2L and little inferior scores for DoS and
Probe.

From Fig. 6, the confusion matrix of a 4-class classifier, we can conclude
that despite the models are trained using the balanced data oversampled by
SVM-SMOTE, it can only provide relatively accurate prediction on DoS (label 0)
and Probe attack (label 1). In addition, a large portion of misclassified
samples in R2L (label 2) and U2R (label 3) fall in the DoS attack, which
suggests the inferior of our model at identifying the different patterns
between DoS and R2L/U2R. Besides, a certain amount of misclassified samples in
U2R fall in R2L, indicating a similar problem. One of the possible
explanations is that, even with SVM-SMOTE to alleviate the imbalance problem
for R2L and U2R, the patterns of these new synthetic samples appear to be
insufficient to represent the attack behaviors in the testing set.

TABLE III: Performance Evaluation of 4-class Classification in IDS Evaluation Metrics | w/o Oversampling | w/ Oversampling
---|---|---
Accuracy | 80.47% | 80.36%
F$1_{DoS}$ | 0.8790 | 0.8678
F$1_{Probe}$ | 0.8703 | 0.8560
F$1_{R2L}$ | 0.4789 | 0.5100
F$1_{U2R}$ | 0.2075 | 0.5741
Macro F1 | 0.6089 | 0.7020
Micro F1 | 0.7839 | 0.7836
Figure 6: Confusion matrix of deep neural network model for 4-class
classification.

##  V Conclusion and Future Work

In this paper, we proposed a data-driven intrusion detection framework based
on data analytics and hierarchical learning-based detecting strategies.
Firstly, we visualize potential patterns and discuss the relationships between
features and underlying attack behaviors based on domain knowledge. Then, we
leverage the supervised and unsupervised learning method to classify normal
network behaviors and malicious attacks, and demonstrate the advantage of the
unsupervised representation learning model in our task. Next, we focus on the
problem of imbalance and alleviate it with SVM-SMOTE oversampling technique.
We further demonstrate the effectiveness and the drawback of the oversampling
mechanism and in 4-class classification with a deep neural network as a base
model. In general, our framework yields satisfactory results on classifying
normal samples and samples of the DoS and Probe attacks. However, it still
shows certain inferiority in terms of minority class with an oversampling
technique. For future work, we consider using more sophisticated learning
models and techniques in terms of data imbalance, for example, cost-sensitive
learning.

## Acknowledgment

This research was supported by the National Science Foundation under Grant No.
1956193.

## References

  * [1] H. Holm, “Signature based intrusion detection for zero-day attacks: (not) a closed chapter?” in _2014 47th Hawaii International Conference on System Sciences_ , 2014, pp. 4895–4904.
  * [2] V. Jyothsna and V. V. R. Prasad, “Article: A review of anomaly based intrusion detection systems,” _International Journal of Computer Applications_ , vol. 28, no. 7, pp. 26–35, August 2011, full text available.
  * [3] M. J. Turcotte, A. D. Kent, and C. Hash, “Unified host and network data set,” in _Data Science for Cyber-Security_. World Scientific, 2019, pp. 1–22.
  * [4] W. Zong, Y.-W. Chow, and W. Susilo, “Dimensionality reduction and visualization of network intrusion detection data,” in _Australasian Conference on Information Security and Privacy_. Springer, 2019, pp. 441–455.
  * [5] L. Dhanabal and S. Shantharajah, “A study on nsl-kdd dataset for intrusion detection system based on classification algorithms,” _International journal of advanced research in computer and communication engineering_ , vol. 4, no. 6, pp. 446–452, 2015.
  * [6] P. Negandhi, Y. Trivedi, and R. Mangrulkar, “Intrusion detection system using random forest on the nsl-kdd dataset,” in _Emerging Research in Computing, Information, Communication and Applications_. Springer, 2019, pp. 519–531.
  * [7] A. Divekar, M. Parekh, V. Savla, R. Mishra, and M. Shirole, “Benchmarking datasets for anomaly-based network intrusion detection: Kdd cup 99 alternatives,” in _2018 IEEE 3rd International Conference on Computing, Communication and Security (ICCCS)_. IEEE, 2018, pp. 1–8.
  * [8] W. Hu, J. Gao, Y. Wang, O. Wu, and S. Maybank, “Online adaboost-based parameterized methods for dynamic distributed network intrusion detection,” _IEEE Transactions on Cybernetics_ , vol. 44, no. 1, pp. 66–82, 2013.
  * [9] Y. Chen, A. Abraham, and J. Yang, “Feature selection and intrusion detection using hybrid flexible neural tree,” in _International Symposium on Neural Networks_. Springer, 2005, pp. 439–444.
  * [10] R. C. Staudemeyer and C. W. Omlin, “Extracting salient features for network intrusion detection using machine learning methods,” _South African computer journal_ , vol. 52, no. 1, pp. 82–96, 2014.
  * [11] P. Toupas, D. Chamou, K. M. Giannoutakis, A. Drosou, and D. Tzovaras, “An intrusion detection system for multi-class classification based on deep neural networks,” in _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_. IEEE, 2019, pp. 1253–1258.
  * [12] P. Wu and H. Guo, “Lunet: A deep neural network for network intrusion detection,” in _2019 IEEE Symposium Series on Computational Intelligence (SSCI)_. IEEE, 2019, pp. 617–624.
  * [13] S. A. Althubiti, E. M. Jones, and K. Roy, “Lstm for anomaly-based network intrusion detection,” in _2018 28th International telecommunication networks and applications conference (ITNAC)_. IEEE, 2018, pp. 1–3.
  * [14] P. Sun, P. Liu, Q. Li, C. Liu, X. Lu, R. Hao, and J. Chen, “Dl-ids: extracting features using cnn-lstm hybrid network for intrusion detection system,” _Security and Communication Networks_ , vol. 2020, 2020.
  * [15] P. Wu, H. Guo, and R. Buckland, “A transfer learning approach for network intrusion detection,” in _2019 IEEE 4th international conference on big data analytics (ICBDA)_. IEEE, 2019, pp. 281–285.
  * [16] Y. Yang, K. Zheng, B. Wu, Y. Yang, and X. Wang, “Network intrusion detection based on supervised adversarial variational auto-encoder with regularization,” _IEEE Access_ , vol. 8, pp. 42 169–42 184, 2020.
  * [17] M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, “A detailed analysis of the kdd cup 99 data set,” in _2009 IEEE symposium on computational intelligence for security and defense applications_. IEEE, 2009, pp. 1–6.
  * [18] L. Dhanabal and S. Shantharajah, “A study on nsl-kdd dataset for intrusion detection system based on classification algorithms,” _International Journal of Advanced Research in Computer and Communication Engineering_ , vol. 4, no. 6, pp. 446–452, 2015.
  * [19] H. He and E. A. Garcia, “Learning from imbalanced data,” _IEEE Transactions on knowledge and data engineering_ , vol. 21, no. 9, pp. 1263–1284, 2009.
  * [20] H. M. Nguyen, E. W. Cooper, and K. Kamei, “Borderline over-sampling for imbalanced data classification,” _International Journal of Knowledge Engineering and Soft Data Paradigms_ , vol. 3, no. 1, pp. 4–21, 2011.
  * [21] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: synthetic minority over-sampling technique,” _Journal of artificial intelligence research_ , vol. 16, pp. 321–357, 2002.
  * [22] J. An and S. Cho, “Variational autoencoder based anomaly detection using reconstruction probability,” _Special Lecture on IE_ , vol. 2, no. 1, pp. 1–18, 2015.
  * [23] C. Liu, W. Wang, M. Wang, F. Lv, and M. Konan, “An efficient instance selection algorithm to reconstruct training set for support vector machine,” _Knowledge-Based Systems_ , vol. 116, pp. 58–73, 2017.


* (112) Maninis, K., Radosavovic, I., Kokkinos, I.: Attentive single-tasking of multiple tasks.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 1851–1860 (2019)
  * (113) Maslej-Krešň’akov’a, V., Sarnovsk‘y, M., Butka, P., Machov’a, K.: Comparison of deep learning models and various text pre-processing techniques for the toxic comments classification.  Applied Sciences 10(23), 8631 (2020)
  * (114) Grégoire, F., Langlais, P.: Extracting parallel sentences with bidirectional recurrent neural networks to improve machine translation.  In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 1442–1453. Association for Computational Linguistics, Santa Fe, New Mexico, USA (2018).  URL https://www.aclweb.org/anthology/C18-1122
  * (115) Nakov, P., Ritter, A., Rosenthal, S., Sebastiani, F., Stoyanov, V.: SemEval-2016 task 4: Sentiment analysis in Twitter.  In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pp. 1–18. Association for Computational Linguistics, San Diego, California (2016).  DOI 10.18653/v1/S16-1001.  URL https://www.aclweb.org/anthology/S16-1001