# The Magnetic Field Calibration of the Full-Disk Magnetograph onboard the
Advanced Space based Solar Observatory (ASO-S/FMG)

S. Liu11affiliationmark:  22affiliationmark:  33affiliationmark:  , J.T.
Su11affiliationmark:  22affiliationmark:  33affiliationmark:  , X.Y.
Bai11affiliationmark:  22affiliationmark:  33affiliationmark:  , Y.Y.
Deng11affiliationmark:  22affiliationmark:  33affiliationmark:  , J.
Chen11affiliationmark:  22affiliationmark:  33affiliationmark:  , Y.L.
Song11affiliationmark:  22affiliationmark:  33affiliationmark:  , X.F.
Wang11affiliationmark:  22affiliationmark:  33affiliationmark:  , H.Q.
Xu11affiliationmark:  22affiliationmark:  33affiliationmark:  , X.
Yang11affiliationmark:  22affiliationmark:  33affiliationmark:  1National
Astronomical bservatories, Chinese Academy of Science, Beijing, 100101, China
2Key Laboratory of Solar Activity and Space Weather, National Space Science
Center, Chinese Academy of Science, Beijing, 100190, China  3School of
Astronomy and Space Sciences, University of Chinese Academy of Sciences,
Beijing, 100101, China<EMAIL_ADDRESS>
###### Abstract

The Full-disk magnetograph is a main scientific payload onboard the Advanced
Space based Solar Observatory (ASO-S/FMG) that through Stokes parameters’
observation to measures the vector magnetic field. The accuracy of magnetic-
field values is an important aspect of checking the quality of the FMG
magnetic-field measurement. According to the design of the FMG, the linear
calibration method under the weak-field approximation is the preferred scheme
for magnetic-field calibration. However, the spacecraft orbital velocity can
affect the position of observed spectral lines, then result in a change of the
polarization-signal strength. Thus, the magnetic field is modulated by the
orbit velocity of the spacecraft. In this article, through cross calibration
between FMG and HMI (Helioseismic and Magnetic Imager onboard the Solar
Dynamic Observatory), the effects of spacecraft orbital velocity on the
coefficient of magnetic-field calibration are investigated. By comparing the
magnetic field of FMG and HMI with spacecraft orbital velocity as an auxiliary
reference, the revised linear-calibration coefficients that depend on
spacecraft orbital velocity are obtained. Magnetic field of FMG corrected by
the revised calibration coefficients removing the effect of spacecraft orbital
velocity will be more accurate and suitable for scientific research.

Solar Magnetic Field, Polarimetric Measurements, Radiative Transfer

##  1 Introduction

The solar magnetic field provides an essential force that dominates solar
activity. The energy and mass released from solar activity, such as solar
flares, filament eruptions, corona mass eruptions and small-scale activity
mostly originate from stored magnetic energy (Krall et al., 1982; Shibata et
al., 1995; Wang et al., 1994; Tsuneta, 1996; Forbes, 2000; Bond et al., 2001;
Priest & Forbes 2002; Nindos et al., 2012). The full-disk magnetograph onboard
the Advanced Space based Solar Observatory (ASO-S/FMG: Gan et al., 2019; Deng
et al., 2019) measures magnetic fields on the solar photosphere, which would
contribute to studies of solar activity (Gan et al., 2023). Fe i $\lambda$
5234.19 Å is chosen as a magnetic-sensitive line by FMG to measure the
photosphere magnetic field. FMG use a liquid-crystal variable retarder (LCVR)
to observe polarized Stokes images ($I$, $Q$, $U$, $V$), which are calibrated
to yield the magnetic field on the photosphere (Deng et al., 2019). During FMG
operation, one or a few limited positions in the range of the spectral line
are fixed to observe Stokes parameters.

The magnetic-field calibration is a necessary and important process for
producing reliable magnetic fields, and magnetic-field calibration methods
vary for different types of instruments. A Stokes polarimeter has ability to
measure the full spectra of Stokes parameters $I$, $Q$, $U$, $V$ of a spectral
line, then an inversion method is applied to derive the magnetic field under
reasonable modes with other necessary thermal parameters. Since the full
information of a spectral line is used by the Stokes polarimeter, it can
measure the magnetic fields with higher accuracy, but lower temporal
resolution than magnetograhp type instruments (Tsuneta et al., 2008; Ichimoto
et al., 2008). FMG obtain magnetograms at higher temporal resolutions than a
spectragraph-base magnetograph, hence magnetic fields with rapid evolution can
be studied by FMG observations. For FMG, very limited Stokes information (one
spectral position is fixed in routine observation) can be obtained, so the
calibration of the magnetic field needs to be better and more carefully
implemented. The magnetic-field calibration method applied by FMG is linear
calibration under the weak-field approximation (Jefferies et al., 1989;
Jefferies & Mickey 1991).

$B_{\mathrm{L}}=C_{\mathrm{L}}V,\\\ $ (1)
$B_{\mathrm{T}}=C_{\mathrm{T}}(Q^{2}+U^{2})^{1/4},\\\ $ (2)
$\theta={\mathrm{arctan}}(B_{\mathrm{L}}/B_{\bot}),\\\ $ (3)
$\phi=1/2{\mathrm{arctan}}(U/Q),\\\ $ (4)

where $B_{\mathrm{L}}$ and $B_{\mathrm{T}}$ are the line-of-sight (LOS) and
transverse photosphere magnetic field, respectively. $\theta$ is the
inclination between the vector magnetic field and the direction normal to the
solar surface and $\phi$ is the magnetic-field azimuth. $C_{\mathrm{L}}$ and
$C_{\mathrm{T}}$ are the calibration coefficients for the LOS and transverse
magnetic fields, respectively.

The calibration coefficients $C_{\mathrm{L}}$ and $C_{\mathrm{T}}$ are
decisive parameters for the determination of photosphere magnetic field,
besides $C_{\mathrm{L}}$ and $C_{\mathrm{T}}$ are spectral-position related
parameters (Unno 1956; Rachkovsky 1967; Landolfi & Landi DegĺInnocenti 1982).
The ASO-S spacecraft run in a Sun-synchronous orbit with an altitude of about
720 km, an orbital inclination angle of 98.27°, and orbital period of about
99.2 mintes, which result in Sun–FMG radial velocity changes -3.9 to +3.9 km
s-1. The maximum value of the radial velocity corresponds to a wavelength
shift from -0.07Å to +0.07Å , which results in the changes of the observed
spectral position. As a result, calibration coefficients $C_{\mathrm{L}}$ and
$C_{T}$ should be corrected at every moment in time based on the velocity of
the ASO-S spacecraft. Hence, to obtain and apply the calibration coefficients
$C_{\mathrm{L}}$ and $C_{\mathrm{T}}$ depending on the velocity of ASO-S
spacecraft would improve the FMG magnetic-field observation.

In this article, the longitudinal magnetic field of FMG and HMI are
investigated and used to calculate the calibration coefficients that related
to spacecraft orbit velocity, with the purpose of improving the accuracy and
reliability of the FMG magnetic-field measurement. We present the observations
in Section 2; the methods and results analysis are given in Section 3; the
discussions and conclusions are included in Section 4.

##  2 Observations

The FMG consists of the main elements of one birefringent Lyot filter (FWHM =
0.1 Å, wavelength stabilization $\textless$ 0.02Å) and two LCVR polarimeters
to measure the magnetic field using the spectral line Fe i $\lambda$5234.19 Å.
The aperture of FMG is 14 cm and its effective focus length is 5355.76 mm. The
spatial resolution of magnetic field observed by FMG is about 1.5 arcsec, for
full-disk images 4k$\times$4k the spatial sampling is about 0.55 arcsec pixel.
During routine observation its temporal resolution about two minutes and
longitudinal sensitivity 15 G. Stokes $Q$, $U$, $V$ are observed by various
combinations of azimuth and retardation of two LCVRs (cf. Table 1, Hou et al.,
2020; Deng et al., 2019). Data processing includes dark field, flat field,
polarization bias, and geometry correction, then the polarization signals are
calibrated to magnetic field under the weak-field approximation (Su et al.
2019; Xu et al., 20021; Liu et al., 2008). The active-region data are cut out
from full-disk images based on external auxiliary time and position
information obtained from NOAA (www.solarmonitor.org/data/2023/04/01/meta/).

Helioseismic and Magnetic Imager is one of the instruments onboard Solar
Dynamic Observatory (SDO/HMI: Scherrer et al.,, 2012; Schou et al.,, 2012).
HMI with a aperture of 14 cm and a 4k $\times$ 4k CCD observe full-disk
magnetograms in the photospheric absorption line Fe i centered at 6173.3 Å.
The spatial resolution of HMI is about 0.9 arcsec while the pixel size is
about 0.5 arcsec. The temporal resolutions is 45 seconds for the longitudinal
magnetic field, while for other observations there are different temporal
resolutions (jsoc.stanford.edu/ajax/lookdata.html). Stokes $Q$, $U$, $V$ are
observed at six fixed wavelength positions by HMI, then inversion algorithms
are applied to Stokes observations to deduce the photosphere magnetic field.
In this article, the longitudinal magnetic field of FMG and HMI with temporal
resolution of 45 seconds are employed to investigate the calibration of FMG,
with the aim to correct the coefficient of FMG modulated by and related to the
spacecraft velocity. Figure 1 shows an example of an original FMG and HMI
longitudinal magnetic field observe at 04 April 2023 03:24 UT quasi-
simultaneously. After geometric correction using position information during
the observation, these two longitudinal magnetic-field signals have very good
consistency, however there are still differences in the size and spatial
resolution for these two images.

Figure 1: The original FMG ($left$) and HMI ($right$) full-disk longitudinal
magnetic field observed 04 April 2023 at 04:24 UT quasi-simultaneously.

##  3 Methods and Results

In order to compare calibration coefficients of FMG and HMI, firstly these two
magnetic-field images are co-aligned by adjusting the size and pixel
resolution. Based on observational information, especially coordinate
information, the two magnetic field images are basically aligned when they are
converted to a heliospheric coordinate system. However, due to the exist of
various system errors, there may still be slight position deviations between
the two magnetic-field images of FMG and HMI. Therefore, an image-alignment
method of a scale invariant feature transform (SIFT: Lowe 1999) based on image
features was adopted to co-align FMG and HMI magnetic-field images. The three
main steps of SIFT for image matching include: i) the feature-point
identification; ii) the feature-point matching; iii) determination of image-
registration parameters. Feature-point identification searching for rotation,
scaling, and translation at different image spatial scales of invariant
feature points. Hence, SIFT is suitable to solve the image rotation, affine
transformations, intensity, and viewpoint changes in matching features. The
SIFT method has been successfully applied in solar image processing (Yang et
al., 2018; Ji et al., 2019). After co-alignment of the FMG and HMI magentic
field, the calibration coefficients of FMG corresponding to HMI can be
obtained pixel-to-pixel on the solar surface directly. Figure 2 shows the
full-disk distribution of the ratio of line-of-sight magnetic field between
the FGM and HMI for line-of-sight velocity $v_{r}$ = -1.9 m s-1 and $v_{r}$ =
323.9 m s-1, respectively. While Panels a and b in Figure 3 show the enlarged
images cut from Figure 2 indicated by a black rectangle to investigate the
distribution of coefficients of FMG comprising to HMI in detail, Panels c and
d in Figure 3 give the errors of these ratios correspond Panels a and b,
respectively. From Figure 2 and 3, it is can be found that the distribution of
ratio of line-of-sight magnetic field between the FGM and HMI within the
activity region is relatively flat and consistent, which can also be seen from
the errors of these ratios shown in Panels c and d in Figure 3, where the
relative errors are small in active regions. The fluctuation of these ratios
in the quiet region is relatively large, as some magnetic field with small
value in the quiet region maybe close to the noise level. Hence, the
comparison of calibration coefficients between FMG and HMI is focused on
active region, to obtain the reliable results. In Figure 2, three small sub-
regions with strong magnetic field labeled 1, 2, and 3 are chosen to
investigate the ratio of line-of-sight magnetic field between FGM and HMI. A
series of full-disk images co-aligned by SIFT, the same as Figure 2 are
rotated to a reference time, and the sub-regions are cut out from full-disk
images by their pixel positions, then the median values of ratio of line-of-
sight magnetic field between the FGM and HMI for these series of sub-regions,
which can indicate the effective values of these ratios on the whole, are
calculated. Figure 4 a shows the ratio of line-of-sight magnetic field between
the FGM and HMI of these time series for sub-regions 1, 2, and 3, and $v_{r}$
with the unit of m s-1 (line-of-sight component of spacecraft velocity) is
plotted as an additional left $y$-axis to investigate the relations between
calibration coefficients and the spacecraft velocity. Here the mean of errors
of these ratios for sub-regions are calculated and labeled $\delta_{R}$, with
the values of $\delta_{R}$=0.118, 0.158, and 0.064 for sub-regions of 1, 2,
and 3, respectively. Figure 4 b, c, and d show the scatter plots of ratio of
line-of-sight magnetic field between the FGM and HMI vs. $v_{r}$ for sub-
regions 1, 2, and 3 labeled. It can be found that the ratios of line-of-sight
magnetic field between the FGM and HMI have a consistent trend with the
longitudinal velocity of the spacecraft, and the monotonicity of the ratio and
spacecraft longitudinal velocity can also be clearly seen from b, c, and d. It
means that the spacecraft longitudinal velocity regularly modulates the
magnetic-field calibration coefficient due to the shift of position of the
observed spectral line by Doppler effect resulting from spacecraft velocity.

Figure 2: The distribution of the ratios of line-of-sight magnetic field
between the FGM and HMI for $v_{r}$ = -1.9 m s-1 ($left$) and $v_{r}$ = 323.9
m s-1 ($right$) labeled, respectively. The region indicated by a $black$
$rectangle$ is the sub-region shown in Figure 3. The sub-regions 1, 2, and 3
indicated by a $orange$ $rectangle$ are the active regions for calculating
pixel-to-pixel calibration coefficients in Figure 4.

Figure 3: Panels a and b are the ratio of line-of-sight magnetic field between
the FGM and HMI, which are same as Figure 2, but for a sub-region cut from
full-disk image in Figure 2 indicated by a $black$ $rectangle$. Panels c and d
are the errors of the ratio of line-of-sight magnetic field between the FGM
and HMI for Panels a and b, respectively.

Figure 4: (a) the time series of cross coefficents for sub-regions 1, 2, and 3
indicated in Figure 2, here $v{{}_{r}}$ with the unit of m s-1 (line-of-sight
component of spacecraft velocity) is plotted as an additional left $y$-axis.
(b), (c) and (d) are the scatter plots of cross coefficents vs. $v{{}_{r}}$
for sub-regions 1, 2, and 3 as in Figure 2, respectively.

The reliability of the ratio of line-of-sight magnetic field by pixel-to-pixel
comparison of FMG and HMI in Figures 2 and 3 is severely affected by signal
noise. An alternative method is fit the active region magnetic field as a
whole to obtain the ratios between FMG and HMI. Firstly, the active region or
sub-region with strong magnetic field of FMG and HMI are plotted as scatter
plots; then linear fitting ($y=kx+b$) is applied to these scatter plots to
calculate the slope of this linear formula, the slope fitted can be regarded
as ratio of the coefficient between FMG and HMI. Figure 5 shows the process to
calculate the whole fitting ratio between FMG and HMI, Panels a and b are the
longitudinal magnetic field of FMG and HMI observed quasi-simultaneously. In
Panel c the scatter plot of FMG at spacecraft velocity of $v_{r}$ of 707 m s-1
and HMI magnetic field are plotted, the yellow line is the fitting result,
here the slope $k$ with value of 2.002 $\pm$ 0.001 is regarded as the ratios
of coefficient for this sub-region on the whole when $v_{r}$ is 707 m s-1. For
this fitting method, the ratio of coefficients is fitted through overall
distributions of the scatter plot, additionally the erorrs both of FMG and HMI
are included in the fitting process. Hence the effects of signal noise and
size of region are avoided, which is the advantage of this fitting method.

Figure 5:  (a) and (b) active-region longitudinal magnetic field of FMG and
HMI. (c) the scatter plot of FMG Panel a at spacecraft velocity of $v_{r}$ of
707 m s-1 and HMI Panel b active-region magnetic field, the $yellow$ $line$ is
the fit $y=kx+b$ .

Figure 6 shows a time series of the ratios of the coefficient between FMG and
HMI using fitting method and $v_{r}$ during a period of 25 February 2023 for a
sub-region with strong magnetic field. It can be found that the ratios of
coefficient are modulated by the longitudinal velocity of the spacecraft
[$v_{r}$], with evident periodicity that is the same exactly as orbital
period. The scatter plot with a high concentration of points means that the
one-to-one correspondence between calibration coefficient and spacecraft
velocity is excellent during this observation with multiple spacecraft orbital
period lasting about one day.

Figure 6: (a) the time series of cross coefficient of HMI/FMG using fitting
method and ${v_{r}}$ as an additional left $y$-axis on 25 February 2023 for a
sub-region with strong magnetic field. (b) are the scatter plots of cross
coefficents vs. $v_{r}$.

To correct FMG magnetic-field values modulated/affected by spacecraft velocity
is the ultimate goal of obtaining ratios of calibration coefficients between
FMG and HMI. Figure 7 shows some examples of the evolution of magnetic flux
for active regions before and after correcting by fitting method, Panels a and
b are active region NOAA 13230 observed on 24 February 2023, Panels c and d
are NOAA 13236 observed on 25 February 2023, the evolution of positive and
negative magnetic flux before and after correction are plotted and labeled by
$fp_{\mathrm{orig}}$, $fp_{\mathrm{corr}}$, $fn_{\mathrm{orig}}$, and
$fn_{\mathrm{corr}}$, respectively. Additionally, the error bars are
calculated and plotted for the corrected magnetic flux $fp_{\mathrm{corr}}$
and $fn_{\mathrm{corr}}$ in each panel, here the errors of
$fp_{\mathrm{corr}}$ and $fn_{\mathrm{corr}}$ are about 0.45% comparing the
values of $fp_{\mathrm{corr}}$ and $fn_{\mathrm{corr}}$ . From Figure 7, it
can be found that the evolution of both positive and negative magnetic flux
for these active region before correction are modulated by spacecraft velocity
evidently with the same periodicity. While after the correction, the
periodicities of evolutions of the magnetic flux become weaker significantly.
Therefore, the coefficients corrected FMG magnetic field will be more suitable
for quantitative research on solar magnetic field.

Figure 7: The examples of the evolution of the magnetic flux for active
regions before and after cross-calibration coefficients correction. Here, the
cross-calibration coefficients are calculated by fitting. Panels a and c are
active region NOAA 13230 observed on 24 February 2023, Panels a and c are NOAA
13236 observed on 25 February 2023, positive and negative magnetic flux before
and after correction are indicated by $fp_{\mathrm{orig}}$,
$fp_{\mathrm{corr}}$, $fn_{\mathrm{orig}}$, and $fn_{\mathrm{corr}}$,
respectively. For plots of $fp_{\mathrm{corr}}$ and $fn_{\mathrm{corr}}$, the
error bars are added.

##  4 Discussions and Conclusions

The FMG onboard the ASO-S spacecraft operates in Sun-synchronous orbit with
orbital period about 99.2 minutes. Subsequently Sun–FMG radial velocities also
have periodicity with longitudinal velocity [$v_{r}$] between Sun and FMG
changes periodically with from -3.9 to +3.9 km s-1. Longitudinal velocity
[$v_{r}$] of FMG resulting Doppler velocity affects the calibration of
magnetic field obviously for routine FMG observations, when one spectral
position is fixed during an observation. The method of correcting the magnetic
field affected by spacecraft velocity can be investigated by comparing the
calibration coefficients with other instruments.

In this article, HMI longitudinal magnetic field with 45-second cadence
working as reference to calibrate FMG magnetic field is tested. From the test
results in Figure 7, it can be found that this type of calibration method can
largely eliminate the influence of spacecraft periodic velocity on magnetic
field of FMG. The corrected magnetic field is closer to the actual conditions,
which would definitely improve the reliability of the FMG magnetic field. The
consistency with HMI magnetic field is not the goal of FMG calibration, the
real goal is to eliminate the impact of spacecraft orbital velocity period on
the FMG magnetic field, and to have a precise measurement of magnetic field
without biases or systematic errors. For example Figure 7, the effect of
magnetic flux evolution being consistent with spacecraft velocity period
should be corrected, as it is a false signal. Since the linear calibration
method is adopted by FMG, and the consistency of FMG magnetic-field values
itself without the influence of spacecraft orbital period is very important.

FMG observes magnetic field over the full solar disk, and calibration
coefficient is very sensitive to the position on the solar surface. There
exists a large-scale background Doppler velocity field related to the solar
position, which inevitably participates in the influence of spacecraft
velocity on the FMG magnetic field. Therefore, a complete data sample, in
which the active-region observations exist positions on the solar surface, is
a necessity in order to fully and effectively correct the FMG magnetic field
influenced by the periodicity of spacecraft velocity. However in this article,
the focus of attention has been placed on the impact of spacecraft velocity.

Another aspect that needs to be mentioned is that calibration coefficients are
spectral position related, since the relation between Stokes signal strength
and magnetic field varies for different observation position of a spectral
line when periodic velocity affects the shape of spectral lines. In the
routine observations of FMG at present, the observed spectral line is fixed at
-0.08 Å away from the center of the line. So the dateset of correcting
calibration coefficients in this article are suitable to correct the FMG
magnetic that observed at line wing of -0.08 Å. Additionally, dateset of
calibration coefficients for the observation at line wing of -0.08 Å is not
complete, and more active regions try to cover solar disk are need.

An alternative method to correct FMG magnetic field is a post-processing
method based on the periodicity of spacecraft velocity with high certainty.
For example evolution of magnetic flux in Figure 7, filtering technology (such
as the FFT) can be applied to these curves to obtain the component with the
period of 99.2 minutes, then a reliable magnetic field can be deduced by
eliminating 99.2 minutes periodic component. Additionally, machine-learning
methods of artificial intelligence may also correct the FMG magnetic field
after the acquisition of a large observation sample with enough information of
observation spectral position, solar surface position, and a full range of
spacecraft velocity.

From the results in the article, it can be found that through cross comparison
with HMI, it is largely possible to correct the periodic influence of
spacecraft orbital velocity on the FMG magnetic field, and obtain a more
realistic magnetic field observed by FMG. However, it should be noted that HMI
is not the only sample used as a reference standard, since the self
consistency of the FMG magnetic field avoiding spacecraft velocity impact is
the main goal of correction. Another observation can be chosen as reference is
data obtained by SMAT ($Solar$ $Magnetism$ and $Activity$ $Telescope$ at the
Huairou Solar Observing Station), the advantage of FMG and SMAT is that they
have common design parameters. To correct FMG magnetic field more
comprehensively, the main aspect is to establish a more comprehensive
calibration database with adequate information of observation spectral
position, solar-surface position and full range of spacecraft velocity, all of
these need a continuous long-term stable observation of FMG.

This work is supported by National Key R&D Program of China 2022YFF0503000
(2022YFF0503001),the Strategic Priority Research Program on Space Science, the
Chinese Academy of Sciences (Grant No. XDA15320301, XDA15320302, XDA15052200),
Natural Science Foundation of China (Grant No 11203036,11703042,
U1731241,11427901, 11473039, 11427901 and 11178016), the Young Researcher
Grant of National Astronomical Observations, Chinese Academy of Sciences, and
the Key Laboratory of Solar Activity National Astronomical Observations,
Chinese Academy

## References

  * Krall et al., (1982) Krall, K. R., Smith, J. B., Jr., Hagyard, M. J., West, E. A., Cummings, N. P. 1982, Sol. Phys., 79, 59.DOI. ADS.
  * Shibata et al., (1995) Shibata, K., Masuda, S., Shimojo, M., Hara, H., Yokoyama, T., Tsuneta, S., Kosugi, T., Ogawara, Y. 1995, ApJ, 451, L83.DOI. ADS.
  * Wang et al., (1994) Wang, H. M., Ewell, M. W., Jr., Zirin, H., Ai, G. X. 1994, ApJ, 424, 436.DOI. ADS.
  * Tsuneta, (1996) Tsuneta, S. 1996, ApJ, 456, 840.DOI. ADS.
  * Forbes, (2000) Forbes, T. G. 2000, J. Geophys. Res., 105, A10.DOI. ADS.
  * Bond et al., (2001) Bond, H. E., Mullan, D. J., O’Brien, M., Sean, Sion, E. M. 2001, ApJ, 560, 919.DOI. ADS.
  * Priest & Forbes (2002) Priest, E. R., Forbes, T. G. 2002, A&A Rev., 10, 313.DOI. ADS.
  * Nindos et al., (2012) Nindos, A., Patsourakos, S., Wiegelmann, T. 2012, ApJ, 748, L6.DOI. ADS.
  * Gan et al. (2019) Gan, W.Q., Zhu,C., Deng, Y.Y., Li, H., Su, Y., Zhang, H.Y., Chen, B., Zhang, Z., Wu, J., Deng, L., Huang, Y., Yang, J.F., Cui, J.J, Chang, J., Wang, C., Wu, J., Shan, Z., Chen, W., Fang, C., Yan, Y.H., Lin, J., Xiong, W.M, Chen,B., Bao, H.C., Cao, C.X., Bai, Y.P., Wang, T., Chen, B.L., Li, X.Y., Zhang, Y., Feng, L., Su, J.T., Li, Y., Chen, W., Li, Y.P, Su, Y.N., Wu, H.Y., Gu, M., Huang, L., Tang, X.J. 2019, $RAA$, 19, 156. DOI. ADS.
  * Gan et al. (2023) Gan, Weiqun and Zhu, Cheng and Deng, Yuanyong and Zhang, Zhe and Chen, Bo and Huang, Yu and Deng, Lei and Wu, Haiyan and Zhang, Haiying and Li, Hui and Su, Yang and Su, Jiangtao and Feng, Li and Wu, Jian and Cui, Jijun and Wang, Chi and Chang, Jin and Yin, Zengshan and Xiong, Weiming and Chen, Bin and Yang, Jianfeng and Li, Fu and Lin, Jiaben and Hou, Junfeng and Bai, Xianyong and Chen, Dengyi and Zhang, Yan and Hu, Yiming and Liang, Yaoming and Wang, Jianping and Song, Kefei and Guo, Quanfeng and He, Lingping and Zhang, Guang and Wang, Peng and Bao, Haicao and Cao, Caixia and Bai, Yanping and Chen, Binglong and He, Tao and Li, Xinyu and Zhang, Ye and Liao, Xing and Jiang, Hu and Li, Youping and Su, Yingna and Lei, Shijun and Chen, Wei and Li, Ying and Zhao, Jie and Li, Jingwei and Ge, Yunyi and Zou, Ziming and Hu, Tai and Su, Miao and Ji, Haidong and Gu, Mei and Zheng, Yonghuang and Xu, Dezhen and Wang, Xing 2023, Sol. Phys., 298, 51. DOI. ADS.
  * Deng et al., (2019) Deng, Y.Y., Zhang, H,Y., Yang, J.F., Li, F., Lin, J.B., Hou, J.F., Wu, Z., Song, Q., Duan, W., Bai, X.Y., Wang, D.G., Lv, J., Ge, W., Wang, J.N., Zheng, Z.Y., Wang, C.J., Wang, N.G., Ni, H.K., Zeng, Y.Z., Zhang, Y, Yang, X., Sun, Y.Z., Zhang, Z.Y., Wang, X. 2019, $RAA$, 19, 157. DOI. ADS.
  * Su et al. (2019) Su,J.T., Bai, X.Y., Chen, J., Guo, J.J., Liu, S., Wang, X.F., Xu, H.Q., Yang, X., Song, Y.L., Deng, Y.Y., Ji, K.F., Deng, L., Huang, Y., Li, H., Gan, W.Q. 2019, $RAA$, 19, 161. DOI. ADS.
  * Hou et al., (2020) Hou, J. F., Wang, D. G., Deng, Y. Y., Zhang, Z. Y., Sun, Y. Z. 2020, $ChinesePhysicsB$, 29, 124211. DOI. ADS.
  * Xu et al., (20021) Xu, H. Q., Liu, S., Su, J. T., Deng, Y. Y., Plotnikov, A., Bai, X. Y., Chen, J., Yang, X., Guo, J. J., Wang, X. F., Song, Y. L. 2021, $RAA$, 21, 67.DOI. ADS.
  * Liu et al., (2008) Liu, S., Su, J. T., Bai, X. Y., Deng, Y. Y., Chen, J., Song, Y. L., Wang, X. F., Xu, H. Q., Yang, X. 2022, Sol. Phys., 297, 6.DOI. ADS.
  * Lowe (1999) Lowe,D.G. 1999, $Proc.SeventhIEEEInternat.Conf.Comp.Vis.$, 2, 1150-1157 vol.2.DOI.
  * Tsuneta et al., (2008) Tsuneta, S., Ichimoto, K., Katsukawa, Y., Nagata, S., Otsubo, M., Shimizu, T., Suematsu, Y., Nakagiri, M., Noguchi, M., Tarbell, T., Title, A., Shine, R., Rosenberg, W., Hoffmann, C., Jurcevich, B., Kushner, G., Levay, M., Lites, B., Elmore, D., Matsushita, T., Kawaguchi, N., Saito, H., Mikami, I., Hill, L. D., Owens, J. K. 2008, Sol. Phys., 10, 313.DOI. ADS.
  * Ichimoto et al., (2008) Ichimoto, K., Lites, B., Elmore, D., Suematsu, Y., Tsuneta, S., Katsukawa, Y., Shimizu, T., Shine, R., Tarbell, T., Title, A., Kiyohara, J., Shinoda, K., Card, G., Lecinski, A., Streander, K., Nakagiri, M., Miyashita, M., Noguchi, M., Hoffmann, C., Cruz, T. 2008, Sol. Phys., 748, L6.DOI. ADS.
  * Jefferies et al., (1989) Jefferies, J., Lites, B. W., Skumanich, A. 1989, ApJ, 343, 920.DOI. ADS.
  * Jefferies & Mickey (1991) Jefferies, J. T., Mickey, D. L. 1991, ApJ, 372, 694.DOI. ADS.
  * Ji et al., (2019) Ji, K. F., Liu, H., Jin, Z.Y., Zhang, Z.H, Qing, Z.P. 2019, $ChineseSciBull$, 64,16.DOI. ADS.
  * Unno (1956) Unno, W. 1956, PASJ, 8, 108. ADS.
  * Rachkovsky (1967) Rachkovsky, D. N.. 1967, $Izv.Krym.Astrofiz.Obs.$, 37, 56. ADS.
  * Landolfi & Landi DegĺInnocenti (1982) Landolfi, M, Landi DegĺInnocenti, E. 1982, Sol. Phys., 78, 355. DOI. ADS.
  * Scherrer et al., (2012) Scherrer, P. H., Schou, J., Bush, R. I., Kosovichev, A. G., Bogart, R. S., Hoeksema, J. T., Liu, Y., Duvall, T. L., Zhao, J., Title, A. M., Schrijver, C. J., Tarbell, T. D.,Tomczyk, S. 2012, Sol. Phys., 275, 207. DOI. ADS.
  * Schou et al., (2012) Schou, J., Scherrer, P. H., Bush, R. I., Wachter, R., Couvidat, S., Rabello-Soares, M. C., Bogart, R. S., Hoeksema, J. T., Liu, Y., Duvall, T. L., Akin, D. J., Allard, B. A., Miles, J. W., Rairden, R.,Shine, R. A., Tarbell, T. D., Title, A. M., Wolfson, C. J., Elmore, D. F., Norton, A. A., Tomczyk, S. 2012, Sol. Phys., 275, 229. DOI. ADS.
  * Yang et al., (2018) Yang, P., Zeng, S.G., Liu, S., Zheng, S., Lin, G. H., He, L. H. 2018, $AstronResTech$, 15, 1.


PANTONE PANTONE PANTONE3015C PANTONE3015C 1 0.3 0 0.2 PANTONE 2021

Date of publication

This work is supported by the Cyber Security Research Programme–Artificial
Intelligence for Automating Response to Threats from the Ministry of Business,
Innovation, and Employment (MBIE) of New Zealand as a part of the Catalyst
Strategy Funds under the grant number MAUX1912.

Corresponding author: Julian Jang-Jaccard (e-mail: j.jang-
jaccard@massey.ac.nz).

# Deep Q-Learning based Reinforcement Learning Approach for Network Intrusion
Detection

HOOMAN ALAVIZADEH1  JULIAN JANG-JACCARD1  and HOOTAN ALAVIZADEH2  Cyber
Security Lab, Comp Sci/Info Tech, Massey University, Auckland, New Zealand
(e-mail<EMAIL_ADDRESS> Imam Reza International
University, Mashhad, Iran (e-mail<EMAIL_ADDRESS>
###### Abstract

The rise of the new generation of cyber threats demands more sophisticated and
intelligent cyber defense solutions equipped with autonomous agents capable of
learning to make decisions without the knowledge of human experts. Several
reinforcement learning methods (e.g., Markov) for automated network intrusion
tasks have been proposed in recent years. In this paper, we introduce a new
generation of network intrusion detection method that combines a Q-learning
based reinforcement learning with a deep feed forward neural network method
for network intrusion detection. Our proposed Deep Q-Learning (DQL) model
provides an ongoing auto-learning capability for a network environment that
can detect different types of network intrusions using an automated trial-
error approach and continuously enhance its detection capabilities. We provide
the details of fine-tuning different hyperparameters involved in the DQL model
for more effective self-learning. According to our extensive experimental
results based on the NSL-KDD dataset, we confirm that the lower discount
factor which is set as 0.001 under 250 episodes of training yields the best
performance results. Our experimental results also show that our proposed DQL
is highly effective in detecting different intrusion classes and outperforms
other similar machine learning approaches.

###### Index Terms:

Network Security, Deep Q Networks, Deep Learning, Q-Learning, Reinforcement
Learning, Epsilon-greedy, Network Intrusion Detection, NSL-KDD, Artificial
Intelligence

=-15pt

##  I Introduction

New generation of Intrusion Detection Systems (IDSs) increasingly demands
automated and intelligent network intrusion detection strategies to handle
threats caused by an increasing number of advanced attackers in the cyber
environment [1, 2, 3]. In particular, there have been high demands for
autonomous agent-based IDS solutions that require as little human intervention
as possible while being able to evolve and improve itself (e.g., by taking
appropriate actions for a given environment), and to become more robust to
potential threats that have not seen before (e.g., zero-day attacks) [4].

Reinforcement Learning (RL) has become a popular approach in detecting and
classifying different attacks using automated agents. The agent is able to
learn different behavior of attacks launched to specific environments and
formulates a defense strategy to better protect the environment in the future.
An RL approach can improve its capability for protecting the environment by
rewarding or penalizing its action after receiving feedback from the
environment (e.g., in a trial-and-error interaction to identify what works
better with a specific environment). An RL agent is capable of enhancing its
capabilities over time. Due to its powerful conception, several RL-based
intrusion detection techniques have been proposed in recent years to provide
autonomous cyber defense solutions in various contexts and for different
application scenarios such as IoT, Wireless Networks [5, 6], or Cloud [7, 8].
The RL agent is able to implement the self-learning capabilities during the
learning process based on its observation without any supervision requirement
that typically involves the expert knowledge from human [9]. Many network
intrusion detection techniques have been proposed based on this RL concept
[10].

However, most of the existing approaches suffer from the uncertainty in
detecting legitimate network traffic with appropriate accuracy and further
lacks the capability to deal with a large dataset. This is because an RL agent
typically needs to deal with very large learning states and would encounter
the state explosion problem. In recent years, deep reinforcement learning
(DRL) techniques have been proposed that are capable of learning in an
environment with an unmanageable huge number of states to address the main
shortcoming of existing RL techniques. DRL techniques such as deep Q-learning
have shown to be a promising method to handle the state explosion problem by
leveraging deep neural networks during the learning process [11].

Many DRL-based IDS for network intrusion detection techniques have been
proposed in the existing literature leveraging different types of intrusion
datasets to train and evaluate their models [12, 13]. However, most of these
existing proposals only focus on enhancing their detection ability and
performance compared to other similar approaches. The majority of these
existing works do not offer comprehensive studies as to how best to develop
and implement a DRL-based IDS approach for a network environment without
providing the precise details such as how the DQL agent can be formulated
based on an RL theory or how to fine-tune hyperparameters for more effective
self-learning and interact with the underlying network environment. In this
paper, we address these shortcomings by introducing the details of design,
development, and implementation strategy for the next generation of the DQL
approach for network intrusion detection.

The main contributions of our work are summarized as follows:

  * •

We introduce a new generation of network intrusion detection methods that
combine a Q-learning based reinforcement learning with a deep feed forward
neural network method for network intrusion detection. Our proposed model is
equipped with the ongoing auto-learning capability for a network environment
it interacts and can detect different types of network intrusions. Its self-
learning capabilities allow our model to continuously enhance its detection
capabilities.

  * •

We provide intrinsic details of the best approaches involved in fine-tuning
different hyperparameters of deep learning-based reinforcement learning
methods (e.g., learning rates, discount factor) for more effective self-
learning and interacting with the underlying network environment for more
optimized network intrusion detection tasks.

  * •

Our experimental results based on the NSL-KDD dataset demonstrate that our
proposed DQL is highly effective in detecting different intrusion classes and
outperforms other similar machine learning approaches achieving more than 90%
accuracy in the classification tasks involved in different network intrusion
classes.

The rest of the paper is organized as follows. Section II presents the related
work. Section III discusses the essential concepts and background associated
with reinforcement learning and deep neural network. Section IV represents the
NSL-KDD dataset used for this paper. The proposed DQL-based anomaly detection
approach is given in V. The evaluation, experimental results, and analysis are
given in Section VI. Finally, we conclude the paper in Section VII.

##  II Related Work

Q-learning, considered to be a model-free method, has been hailed to be a
promising approach, especially when utilized in challenging decision
processes. This method is appropriate if the other techniques such as
traditional optimization methods and supervised learning approaches are not
applicable [14]. The advantages of Q-learning are its effective results,
learning capabilities, and the potential combination with other models.

The application of machine learning such as Deep Reinforcement Learning (DRL)
[15, 16, 17], supervised and unsupervised learning, in cybersecurity, has been
investigated in various studies [18, 13, 19, 14]. In [18], the authors studied
a comprehensive review of DRL for cybersecurity. They studied the papers based
on the live, real, and simulated environment. In [19], the authors showed the
applications of DRL models in cybersecurity. They mainly focused on the
adversarial reinforcement learning methods. They also presented the recent
studies on the applications of multi-agent adversarial RL models for IDS
systems. In [13], the authors studied several DRL algorithms such as Double
Deep Q-Network (DDQN), Deep Q-Network (DQN), Policy Gradient (PG), and Actor-
Critic (AC) to intrusion detection using NSL-KDD [20] and AWID [21] datasets.
Those datasets were used for train purposes and classifying intrusion events
using the supervised machine learning algorithms. They showed the advantages
of using DRL in comparison with the other machine learning approaches which
are the application of DRL on modern data networks that need rapid attention
and response. They showed that DDQN outperforms the other approaches in terms
of performance and learning.

TABLE I: List of features on NSL-KDD dataset

F# | Feature Name | F# | Feature Name | F# | Feature Name
---|---|---|---|---|---
F1 | Duration | F15 | Su attempted | F29 | Same srv rate
F2 | Protocol_type | F16 | Num root | F30 | Diff srv rate
F3 | Service | F17 | Num file creation | F31 | Srv diff host rate
F4 | Flag | F18 | Num shells | F32 | Dst host count
F5 | Src bytes | F19 | Num access files | F33 | Dst host srv count
F6 | Dst bytes | F20 | Num outbound cmds | F34 | Dst host same srv rate
F7 | Land | F21 | Is host login | F35 | Dst host diff srv rate
F8 | Wrong fragment | F22 | Is guest login | F36 | Dst host same srv port rate
F9 | Urgent | F23 | Count | F37 | Dst host srv fiff host rate
F10 | Hot | F24 | Srv count | F38 | Dst host serror rate
F11 | Num_failed_logins | F25 | Serror rate | F39 | Dst host srv serror rate
F12 | Logged_in | F26 | Srv serror rate | F40 | Dst host rerror rate
F13 | Num compromised | F27 | Rerror rate | F41 | Dst host srv rerror rate
F14 | Root shell | F28 | Srv rerror rate | F42 | Class label

In [22, 23], the authors proposed a deep reinforcement learning technique
based on stateful Markov Decision Process (MDP), Q-learning. They evaluated
the performance of their method based on different factors such as learning
episodes, execution time, and cumulative reward and compared the effectiveness
of standard planning-based with a deep reinforcement learning based approach.

In [24], the authors proposed a reinforcement learning agent installed on
routers to learn from traffic passing through the network and avoid traffic to
the victim server. Moreover, [25] proposed a machine learning method to detect
multi-step attacks using hidden Markov models to predict the next step of the
attacker. In [26], the authors proposed a decision-theoretic framework named
ADRS based on the cost-sensitive and self-optimizing operation to analyze the
behavior of anomaly detection and response systems in autonomic networks.

In [27], the authors combined the multi-objective decision problem with the
evolutionary algorithms to make an efficient intrusion response system. They
considered an intrusion response system as a multi-attribute decision making
problem that takes account of several aspects before responding to the threats
such as cost of implementation, resource restriction, and effectiveness of the
time, and modification costs. This multi-objective problem tried to find an
appropriate response that was able to reduce the values of these functions.

Most of the existing works focused only on the enhancement of their methods to
provide better performance and on the evaluation of the performance of the
proposed techniques through comparison with other similar machine learning
(ML)-based approaches.

##  III Background

###  III-A Reinforcement Learning

Modeling a system as a Markov Decision Process (MDP) so that an agent can
interact with the environment based on different discrete time steps is an
important aspect in designing many decision-making related problems. MDP can
be shown as a 5-tuple: M=(S,$\mathcal{A}$, $T$, $R$,$\gamma$) where S denotes
a set of possible states and $\mathcal{A}$ indicates a set of possible actions
that the agent can perform on the environment, and $T$ denotes the transition
function from a state to another state. $T$ defines the (stationary)
probability distribution on $S$ to transit and reach a new state $s^{\prime}$.
The value of $R$ demotes the reward function, and $\gamma=[0,1)$ indicates the
discount factor.

A policy $\pi$ can be defined to determine the conditional probability
distribution of selecting different actions depending on each state $s$. The
distribution of the reward sequence can be determined once a stationary policy
has opted. Then, policy $\pi$ can be evaluated by an action-value function
which can be defined under $\pi$ as the expected cumulative discounted reward
based on taking action from state s and following $\pi$ policy. By solving the
MDP, the optimal policy $\pi^{*}$ can be found that maximizes the expected
cumulative discounted reward based on all states. The corresponding optimal
action values satisfy $Q^{*}(s,a)=\max\limits_{\pi}Q^{\pi}(s,a)$, and the
uniqueness and existence of the fixed-point solution of Bellman optimality
equations can be obtained by Banach’s fixed-point theorem.

$Q^{*}(s,a)=R(s,a)+\gamma\int_{s^{\prime}}T(s^{\prime}|s,a)~{}\max\limits_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})$

Figure 1: DQN model based on agent-environment interaction

The essential cyclic process in the RL is agent-environment interaction. The
RL agent should interact with the environment to explore and learn from
different transition and reward functions obtained from the action taken. This
process makes the RL agent able to find out the optimal policy, see Figure 1.
During the interaction with the environment at time $t$, the RL agent observes
the information about the current state $s$, and then chooses an action $a$
based on a policy. Then, it receives a reward $r$ from the environment based
on the action taken and moves to a new state $s^{\prime}$. The RL agent
improves itself by experiences gained based on this cyclic agent-environment
interaction. The learning process could be based on either (i) approximating
the transition probabilities and reward functions to learn the MDP model and
then finding an optimal policy using planning in the MDP (i.e., known as the
model-based approach), or (ii) trying to learn the optimal value functions
directly without learning the model and deriving the optimal policy (e.g.,
model-free approach).

Q-learning can be considered as a model-free approach that updates the
Q-values estimation based on the experience samples on each time step as the
following equation.

$Q(s,a)\leftarrow
Q(s,a)+\alpha(r+\gamma\max\limits_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})-Q(s,a))$

in which $\alpha$ is the learning rate, and $Q(s,a)$ is simply the current
estimation.

###  III-B Feed Forward Neural Network

We utilized a feed forward neural network constructed based on a fully
connected neural network as the main module of the DRL model for approximating
the Q-values and training the model based on the NSL-KDD Dataset. The
intrusion datasets will be fed into a pre-processing module first for
cleansing and preparing the dataset and also extracting the related features
[28, 29, 30].

The fully connected neural network includes different fully connected layers
that link every single neuron in the layer to neurons of the previous layer.
The neural network output $f(x)$ or $y$ is represented in Equation (1) [31].

$y=f(x)=F_{|f|}(F_{|f|}-1(\dots F_{2}(F_{1}(x))))$ (1)

where $x$ is the input, $F_{i}$ is a transformation function, and $|f|$
denotes the total number of computational layers which can be either hidden
layers and the output layer in the neural network. The outputs from the
preceding layers are transferred using each perceptron by applying a non-
linear activation function. Finally, the $i^{th}$ perceptron in the $t^{th}$
layer can be represented as:

$o^{t}_{i}=\nu\bigg{(}w^{t}_{i*}.o^{t-1}+b^{t}_{i}\bigg{)}$ (2)

where $o^{t-1}$ is the output of the preceding layer, $w^{t}_{i*}$ is the
weight vector of the perceptron, $b^{t}_{i}$ is its bias and $\nu$ is the non-
linear activation function.

Activation functions play an essential role in the training process of a
neural network. Activation functions manage the computations to be more
effective and reasonable as there is the complexity between the input units
and the response variable in a neural network. The main role of the activation
function is to convert an input unit of a neural network to an output unit.
Different activation functions can be used in a neural network such as
Sigmoid, Tanh, and ReLU. However, the ReLU activation function is known as a
more effective one comparing with the other activation functions in many
detection problems with lower run-time and demands for less expensive
computation costs, see Equation 3 where $z$ is the input.

$ReLU(z)=\begin{cases}0\hskip 28.45274pt\text{if}z<0\\\ 1\hskip
28.45274pt\text{if}z\geq 0\\\ \end{cases}$ (3)

TABLE II: NSL-KDD data-record classes

Categories | Notation | Definitions | # of Samples
---|---|---|---
Normal | N | Normal activities based on the features | 148517
DoS | D |  | Attacker tries to avoid users of a service
---
Denial of Service attack
53385
Probe | P |  | Attacker tries to scan the target network to collect
---
information such as vulnerabilities
14077
U2R | U |  | attackers with local access to victim’s machine
---
tries to get user privileges
119
R2L | P |  | attacker without a local account tries to send
---
packets to the target host to get access
3882

##  IV Dataset

NSL-KDD dataset is a labeled network intrusion detection dataset that so far
has been used in many tasks to evaluate different deep learning-based
algorithms for devising different strategies for IDS [32, 33]. NSL-KDD dataset
contains 41 features labeled as a normal or specific attack type (i.e.,
class). We utilized the one-hot-encoding method for the dataset preprocessing
to change the categorical features to the corresponding numeral values as deep
learning models can only work with numerical or floating values. We normalized
the train and test datasets to the values between 0 and 1 using a mix-max
normalization strategy. The 41 features presented in the NSL-KDD dataset can
be grouped into four such as basic, content-based, time-based, and host-based
traffic features. The value of these features is mainly based on continuous,
discrete, and symbolic values. The NSL-KDD dataset contains five attack
classes such as Normal Denial-of-Service (DoS), Probe, Root to Local (R2L),
and Unauthorized to Root (U2R). These attack classes can be identified using
the features corresponding to each NSL-KDD data. Table II defines the attack
classes for NSL-KDD that we consider in our study.

##  V Anomaly Detection using Deep Q Learning

###  V-A Deep Q-Networks

One of the most effective types of RL is Q-learning in which a function
approximator such as either a neural network or a deep neural network is used
in RL as a Q-function to estimate the value of the function. The Q-function
integrated with a deep neural network can be called Deep Q-Learning (DQL). The
Q-learning agent in the DQL can be represented as $Q(s,a;\theta)$. The
Q-function consists of some parameters such as state $s$ of the model, action
$a$, and reward $r$ value. The DQL agent can select an action $a$ and
correspondingly receives a reward for that specific action. The neural network
weights related to each layer in the Q-network at time $t$ are denoted by the
$\theta$ parameter. Moreover, $s_{i+1}$ or $s^{\prime}$ represents the next
state for DQL model. The DQL agent moves to the next state based on the
previous state $s$ and the action $a$ performed in the previous state $s$. A
deep neural network is used as the deep Q-network to estimate and predict the
target Q-values. Then, the loss function for each learning activity can be
determined by Q-values obtained on the current and previous states. In some
cases, only one neural network is used for estimating the Q-value. In this
case, a feedback loop is constructed to estimate the target Q-value so that
the target weights of the deep neural network are periodically updated.

###  V-B Deep R-Learning Concepts

Here we define the important concepts related to DQL based on the environment
where the NSL-KDD dataset is used for network intrusion detection tasks.

Figure 2: DQN model prediction using states and deep neural network, the
outputs are Q-values, and actions are computed based on $argmax~{}Q_{i}$ for
the current state. Figure 3: State transition Markov diagram for DQN agent
training process based on current and next states prediction and training.
Figure 4: DQL agent training phase flowchart.

####  V-B1 Environment

The environment for this study is the one where the pre-processed and
normalized NSL-KDD dataset is used where the columns (features) of the NSL-KDD
dataset denote the states of the DQN. There are 42 features in NSK-KDD and we
utilize the first 41 features as states. Feature 42 is the label that will be
used for computing the award vectors based on model prediction. Note that in
this DQN model, the agent only obtains actions to compute the rewards vector,
and there is no real action performed to the environment.

####  V-B2 Agent

DQL agent is utilized in the DQL model based on the network structure so that
there is at least one agent for the context of a network. The agent interacts
with the environment and applies rewards based on the current state and the
selected action. A DQL agent could be defined as a value-based RL agent that
is able to trains the model to estimate the future rewards values. A DQN can
be trained by an agent interacting with the environment based on the
observation and possibles action spaces. In the DQL training process, the
agent needs to explore the action space by applying a policy such as epsilon-
greedy exploration. The exploration helps the agent to selects either a random
action with a probability of $\epsilon$ or an action greedily based on the
value function with the greatest value with probability $1-\epsilon$.

####  V-B3 States

States in DQL describe the input by the environment to an agent for taking
action. In the environment where the NSL-KDD dataset is used, the dataset
features (as in Table I) are used for state parameters for DQN. We use those
41 features as the inputs of DQN such that $s_{i}=F_{i}$ for training and
prediction using DQN.

####  V-B4 Actions

An action is considered as the decision chosen by the agent after processing
the environment during a given time window such as after finishing the process
of a mini-batch. The DQN agent generates a list of actions as an action vector
based on the given input of the neural network and input features. The final
Q-values are used to judge whether an attack was captured succesfully. It
feeds the state vector with the size of the mini-batch to the current DQN.
Then, the agent compares the output of the current DQN based on threshold
rates as Q-values and determined the Q-threshold value for classifying the
attack classes.

####  V-B5 Rewards

In DQL, the feedback from the environment for a corresponding action done by
an agent is called a reward. A reward vector can be defined based on the
output values of DQN and the size of the mini-batch. The DQL agent can
consider a positive reward when the classification result of DQN matches the
actual result based on the labels in the NSL-KDD. Otherwise, it may get a
negative reward. The reward value can be considered depending on the
probability of prediction by the classifier. This value can be adjusted based
on the Q-values obtained to enhance the classifier’s performance.

TABLE III: DQL agent and Neural Network parameters.

Parameters | Description | Values
---|---|---
num-episode | Number of episodes to train DQN | 200
num-iteration | Number of iteration to improve Q-values in DQN | 100
hidden_layers | Number of hidden layers: Setting weights, producing outputs, based on activation function | 2
num_units | number of hidden unit to improve the quality of prediction and training | $2\times 100$
Initial weight value | Normal Initialization | Normal
Activation function | Non-linear activation function | ReLU
Epsilon $\epsilon$ | Degree of randomness for performing actions | 0.9
Decoy rate | Reducing the randomness probability for each iteration | 0.99
Gamma $\gamma$ | Discount factor for target prediction | 0.001
Batch-size (bs) | A batch of records NSL-KDD dataset fetched for processing | 500

###  V-C Deep Q-Learning Process

The standard Q-learning and DQN can be differentiated based on the method of
estimating the Q-value of each state-action pair and the way in which this
value can be approximated using generalized state-action pair by the function
Q(s,q). The process of DQL is based on the flowchart represented in Figure 4.
The DQN agent deals with the environment where the NSL-KDD is used. In the
first step, the parameters of the algorithm and models are initialized based
on Table III. The values of the features of NSL-KDD (F1–F41 as in Table 1)
indicate the state’s variables ($s$) of the DQN. Note that the batch size
($bs$) for the DQN process is set as 500. This means that for each state the
amount of 500 records of NSL-KDD are fetched from memory and fed into one
state ($S$), see the batch table represented in Figure 2. However, there are
41 features as the state variables each of which can have various values.
Thus, as the number of state-value pairs becomes comparatively large, it is
not possible to keep them in a Q-table (or look-up table). Thus, the DQN agent
leverages a DNN as the function approximator to compute Q values based on the
states and actions.

Figure 4 presents the overall steps of DQL using an agent. First, the
normalized NSL-KDD dataset is fed into the environment and the DQL agent
initializes a vector for Q values, state’s variables, actions according to
batch size, the DNN parameters, and weights are initialized. Then, the
learning iterations train the DQN based on the epsilon-greedy approach. The
outer iteration represents different episodes of the learning process and
after each iteration, the value of states is initialized again. Note that the
parameters of the trained DNN are preserved and are not initialized for each
episode iteration. The state $S_{n}$ at each discrete state is given by the
training sample (Batchn), see Figure 2. At the end of each episode, a complete
sequence of states, rewards, and actions are obtained in the terminal state.
During the start of the training, the agents receive the first batch (500
records from the environment), and this is the starting state $S_{1}$ of the
environment.

Figure 5: Exploration strategy using Epsilon-greedy approach

In the inner iteration, the DQN agent performs exploration, action selection,
and model training based on DQL. Note that each iteration adjusts the
Q-function approximator which uses a DNN. In the standard Q-learning, a table
of values is kept and the agent separately updates each state-value pair.
However, DQN utilizes a deep-learning approach to estimate the Q-function. We
leverage a deep neural network as a function approximator for the Q-function.
We use a deep neural network consisting of 4-layers as represented in Figure
2, with ReLU activation for all layers, including the last one to ensure a
positive Q-value. Layer one is the input layer which includes 41 neurons and
is fed with the state variables on each iteration. There are two hidden layers
with the size of 100 each for training purposes, and one output layer with the
size of 5 which keeps the output layer corresponding to the related Q values
for each attack class. Note that after each training iteration based on the
states and batch size the q values predicted in the output later will be fed
into Q vectors (denoted as QV) as illustrated in Figure 2.

In the DQN learning procedure, there should be an appropriate trade-off
between exploitation and exploration. At the first rounds of learning, the
exploration rate should be set as a high probability with the value of
approximately 1, and gradually decreases using a decoy rate, see Figure 5. The
exploration is performed based on the epsilon-greedy approach. An epsilon-
greedy policy is implemented as a training strategy based on reinforcement
learning definition which helps the agent to explore all possible actions and
find the optimal policy as the number of explorations increases. The action is
chosen by applying the epsilon-greedy approach which selects a random action
with a probability of $\epsilon$ or predicts the action with a probability of
($1-\epsilon$).

Figure 6: Comparing the loss and reward values of DQN learning process based
on different discount factor values: (a) $\gamma=0.001$, (b) $\gamma=0.01$,
(c) $\gamma=0.1$, and (d) $\gamma=0.9$

As the batch size for each state is equivalent to $bs$ (defined in Table III),
$bs$ numbers of random actions will be fed into the action vector (AV) as
$AV_{i}=Rnd(0,5),~{}\forall i\in bs$, where 0–5 denotes Norman (N), Probe (P),
DoS (D), U2R (R), and R2L (R), respectively. In the first set of iterations,
the probability of choosing random actions is high, but as time pasts this
probability gets lower by the epsilon-greedy approach as in Figure 5. With the
probability of $1-\epsilon$, the DQL agent predicts the actions using the
current state (including the first batch with the size of $bs$ and state
variables). Note that, the amount of $bs$ records of the environment denotes
the current state. The features (i.e., variable states) of the current state
are fed into the input layer of DNN architecture and the Q-values are
predicted based on the DNN parameters and weights in the output layer. The
action having the highest Q values is selected for each record $i$ in the
current state as $AV_{i}=argmax(QV_{i}),~{}\forall i\in bs$ which can be
either normal or malicious based on the four attack types.

In the next step, the action vector (AV) filled by either random actions
(i.e., with a higher chance in the first iterations) or predicted actions
using DQN (i.e., with higher chance after a while) is fed into the reward
function for computing the rewards based on comparing the $AV_{i}$ with the
labels in the dataset for corresponding data-record, see the reward function
represented in Figure 4. Then, the DQL agent needs to compute the Q vectors
and action vectors of the next state denoted as $Q^{\prime}V_{i}$ and
$A^{\prime}V_{i}$ for all $i\in bs$ to complete the training process and DQL
principles as illustrated in Figure 3. Then, the target Q (denoted as QT) is
computed based on the rewards, discount factor for future rewards, and
predicted Q vectors as Equation 4.

$QT_{i}=RV_{i}+\gamma.Q^{\prime}V_{i}$ (4)

The results of $QT_{i}$ are further fed into the DQN for the training process
and computing the loss function as represented in the learning improvement
phase in Figure 4. The training of the neural network is performed with a Mean
Square Error (MSE) loss between the Q-value estimated by the neural network
for the current state and a target Q value obtained by summing the current
reward and the next state’s Q-value multiplied by the value of discount factor
($\lambda$). The computation of the loss function for the evaluation of the
DQN performance is critical. We compute the loss value after each iteration
episode for the DQN network based on the current states and target network.
The total loss can be denoted as Equation 5.

$Loss=\frac{1}{n}\sum_{n}{\bigg{(}\underbrace{Q(s,a)}_{\text{Prediction}}-\underbrace{r+\gamma
Q(s^{\prime},a^{\prime})}_{\text{Target}}\bigg{)}^{2}}$ (5)

Once the training of the model is completed, the trained NN is used for
prediction. For each state, the Q-function provides the associated Q-value for
each of the possible actions for that specific state. The predicted action is
determined based on the maximum Q-value. The model is trained for a number of
iterations and episodes which are enough for covering the complete dataset.

##  VI Evaluation of DQL Model

###  VI-A Experiment Setup and Parameters

We implemented our proposed model in Python using Tensorflow framework version
1.13.2. In our study, we analyzed the performance of our DQL model using NSL-
KDD datasets. The training portion of the NSL-KDD dataset includes various
samples for the network features and corresponding labels for intrusion with
different possible values such as binary or multiclass anomaly. In this paper,
we considered the network features as states and the label values as the
actions to adapt these elements to DQN concepts.

The parameters and values associated with the DQL model are shown in Table
III. For the fully connected architecture, we used a total of two hidden
layers with ‘relu’ activation function apart from input and output layers.
During the training, various major parameters should be determined and
examined in order to find the best values that are appropriate and ideal for
the model. At the beginning step for training, the exploration rate
$\epsilon$, is set to 0.9 for the agent to perform exploration based on some
degree of randomness with the decoy rate of 0.99. The initial values for other
values such as batch-size, discount factor are illustrated in Table III.
However, we also evaluate the performance of the DQL model based on different
values. We examined the behavior of the proposed DQL agent by varying the
discount factor values. This essentially determines how the DQL agent can
improve the performance of learning based on future awards.

Figure 7: Confusion Matrix based on the classification categories for our DQL
model for two different Discount Factors: (a) $\gamma=0.001$, (b)
$\gamma=0.9$.

Figure 6 demonstrates the loss and reward values obtained during the DQL
training process based on different values of the discount factor. Figure
LABEL:fig:LR001 and Figure LABEL:fig:LR01 show the reward and loss values by
setting the $\lambda$ as $0.001$ and $0.01$, respectively. As it shows, the
loss value is lower in $\lambda=0.001$. However, we increased the learning
rate significantly and evaluated the loss and reward values in Figures Figure
LABEL:fig:LR1 and Figure LABEL:fig:LR9. The results indicate that higher value
for discount factor leads to higher loss value. As it also shows, the loss
value in the worst-case reaches $1.7$ based on $\lambda=0.001$, while it
reaches $4$ in the higher discount factor value $\lambda=0.9$. However, as
demonstrated in Figure 6, we can observe that reward values have a sharp
increasing trend for all discount factor values.

Based on the results obtained during the DQN agent learning process, we
discovered that the lower discount factor yields a lower loss value that leads
to better results in terms of learning the model especially when the episode
numbers are smaller.

###  VI-B Performance Metrics

We use different measurements to evaluate the performance of our proposed DQL
model used for network intrusion detection such as Accuracy, Precision,
Recall, and F1 score. However, the performance of the model cannot rely only
on the accuracy values since it evaluates the percentages of the samples that
are correctly classified. It ignores the samples incorrectly classified. To
perform better evaluation, we analyzed the results based on the other
performance metrics as follows.

##### Accuracy

Accuracy is one of the most common metrics to evaluate and judge a model. it
measures the total number of correct predictions made out of all the
predictions made by the model. It can be obtained based on True Positive (TP)
value, True Negative (TN) rate, False Positive (FP) rate, and False Negative
(FN) value. Equation (6) shows the Accuracy metric.

$Accuracy=\frac{TP+TN}{TP+FP+TN+FN}$ (6)

##### Precision

Precision evaluates can be obtained based on the percentage of positive
instances against the total predicted positive instances. In this case, the
denominator is the sum of TP and FP denoting the model prediction performed as
positive from the whole dataset. Indeed, it indicates that ‘how much the model
is right when it says it is right’, see Equation (7).

$Precision=\frac{TP}{TP+FP}$ (7)

##### Recall

Recall (Sensitivity) shows the percentage of positive instances against the
total actual positive instances. The denominator is the sum of TP and FN
values which is the actual number of positive instances presented in the
dataset. It indicates that ‘how many right ones the model missed when it
showed the right ones’. See Equation (8).

$Recall=\frac{TP}{TP+FN}$ (8)

##### F1 score

The harmonic mean of precision and recall values is considered as the F1
score. It considers the contribution of both values. Thus, the higher the F1
score indicates the better results. Based on the numerator of Equation (9), if
either precision or recall value goes low, the final value of the F1 score
also decreases significantly. We can conclude a model as a good one based on
the higher value of the F1 score. Equation (9) shows how the F1 score is
computed based on both precision and recall values.

$F1~{}score=\frac{2\times Precision\times Recall}{Precision+Recall}$ (9)

###  VI-C Performance Evaluation

We evaluated the performance of the DQN on the testing phase based on the
parameters set in Table III and training based on 200 episodes.

The confusion matrix for the DQL model based on two different discount factors
of $0.001$ and $0.9$ are shown in Figure 7. The confusion matrix represented
the evaluation of our model for the test data set. The rows in the confusion
matrix are associated with the predicted class and the columns indicate the
true class. The confusion matrix cells on the main diagonal demonstrate the
correctly classified percentages such as those having been classified as TP or
TN. However, the incorrectly classified portion is located in the off-diagonal
cells such as FN and FP values. The values located on the last columns (most
right columns) indicate the percentages of incorrectly classified predictions
corresponding to each class.

Considering the graphs, we observe that the true-positive rate for normal,
DoS, and Probe classes for $\lambda=0.001$ has decreased from 0.96 to 0.82,
and 0.68 to 0.93, 0.89, and 0.57 for $\lambda=0.9$, respectively. This shows
that the DQL agent performs better for the smaller values of discount factor
$\gamma=0.001$ comparing to larger discount factor value such as $\gamma=0.9$.
However, the results for the minority class of R2L are very low because of
unbalanced distributions of the number of samples for each class (see Table
II).

Figure 8: # of samples against estimated attack types Figure 9: Performance of
DRL process based on different episodes

Figure 8 shows the performance of the proposed DQL model for the correct
estimations (TP and TN) together with FP and FN values based on the number of
samples. The results are captured after performing 200 episodes for agent’s
learning with the discount factor of $\lambda=0.001$. Considering the graph,
we observe higher correct estimations for Normal, DoS, and Probe classes
respectively, while these values are lower for the minority classes due to the
unbalanced distribution of class samples.

We evaluated the performance or DQL model based on both accuracy and time
against the different numbers of training episodes in Figure 9. We can observe
that the accuracy value has an ascending trend from 100 episodes to 250
episodes. However, this value decreased to 300 episodes, while training based
on 300 episodes lasts more than 20 minutes. It shows that the best number of
episodes for the DQL agent for training is 250 episodes which take a smaller
execution time of around 17 minutes based on our implementation.

Table IV compares the overall performance of the DQL based on two different
discount factors in the DQL training process. The results show that all
performance metrics have higher values for the smaller value of discount
factor $\lambda=0.001$. Table V shows the performance metrics for each class
separately while the DQL model is trained based on 200 episodes with the
discount factor $\lambda=0.001$.

TABLE IV: Performance evaluation of DQL based on various discount factor values. Metric | Discount Factors
---|---
$\gamma=0.001$ | $\gamma=0.1$ | $\gamma=0.9$
Precision | 0.7784 | 0.6812 | 0.6731
Recall | 0.7676 | 0.7466 | 0.758
F1 score | 0.8141 | 0.7063 | 0.6911
Accuracy | 0.7807 | 0.7473 | 0.7578
TABLE V: Evaluation metrics for DQL Model based on each classes Metric | Attack Categories
---|---
Normal | DoS | Probe | R2L
Accuracy | 0.8094 | 0.9247 | 0.9463 | 0.8848
F1 score | 0.8084 | 0.9237 | 0.9449 | 0.8370
Precision | 0.8552 | 0.9249 | 0.9441 | 0.8974
Recall | 0.8093 | 0.83 | 0.9247 | 0.8848
TABLE VI: Comparing our model’s accuracy and time with other studies in the literature Approach | Reference | Adaptive-learning | Dataset | Accuracy | Time
---|---|---|---|---|---
SOM |  Ibrahim et al. [34] | ✗ | NSL-KDD | 75% | NA
RF | Jiang et al. [35] | ✗ | NSL-KDD | 74% | NA
BiLSTM | ✗ | NSL-KDD | 79% | 115
CNN-BiLSTM | ✗ | NSL-KDD | 83% | 72
Naive Bayes | Yang et al. [36] | ✗ | NSL-KDD | 76% | NA
SVM | ✗ | NSL-KDD | 68% | NA
DQL | Our model | ✓ | NSL-KDD | 78% | 21
TABLE VII: Comparing our model’s performance in classification with other
studies in the literature

Method | Reference | Normal | DoS | Probe | R2L
---|---|---|---|---|---
RF |  Jiang et al. [35] | 0.7823 | 0.8695 | 0.7348 | 0.0412
CNN | Yang et al. [36] | 0.9036 | 0.9014 | 0.6428 | 0.1169
LSTM | 0.8484 | 0.8792 | 0.6374 | 0.0994
CNN-BiLSTM | 0.9215 | 0.8958 | 0.7111 | 0.3469
DQL | Our model | 0.8084 | 0.9237 | 0.9463 | 0.8848

###  VI-D Comparison With Other Approaches

In this section, we compare the results obtained from our proposed DRL models
with various common ML-based models based on NSL-KDD datasets. We compare the
results with Self-organizing Map (SOM), Support Vector Machine (SVM), Random
Forest (RF), Naive Bayes (NB), Convolutional Neural Network (CNN), and some
hybrid models such as BiLSTM and CNN-BiLSTM models presented in different
studies [34, 35, 36].

We compare the results based on the performance metrics such as Accuracy,
Recall, F1 Score, and Precision. Table VI compares the accuracy and training
time (in minutes) with other studies in the literature.

As it shows, our model has a higher accuracy comparing with the other
approaches while it has a lower training time. However, the worst accuracy
obtained by the SVM approach is about 68%. Both BiLSTM and CNN-BiLSTM hybrid
approaches have high accuracy of 79% and %83, respectively. However, those
approaches have a higher training time than our model.

Table VII compares our model’s performance in terms of F1-score for all
classifications with other studies in the literature. F1-score is known as a
balance point between recall and precision scores and can be considered as the
harmonic average of both recall and precision. Based on the results summarized
in Table VII, the F1-score for the Normal class reaches about $81\%$ in our
study while this value is higher for CNN and CNN-BiLSTM hybrid approaches with
the values of around $90\%$. However, it can be seen that our models perform
better in terms of other attack classes such as DoS, Probe, R2L comparing with
other ML-based and hybrid approaches in the literature. It can be seen that
the RF method has the lowest value comparing to the other approaches.

##  VII Conclusion

We present a Deep Q-learning based (DQL) reinforcement learning model to
detect and classify different network intrusion attack classes. The proposed
DQL model takes a labeled dataset as input, then provides a deep reinforcement
learning strategy based on deep Q networks.

In our proposed model, a Q-learning based reinforcement learning is combined
with a deep feed-forward neural network to interact with the network
environment where network traffic is captured and analyzed to detect malicious
network payloads in a self-learning fashion by DQL agents using an automated
trial-error strategy without requiring human knowledge. We present the
implementation of our proposed method in detail including the basic elements
of DQL such as the agent, the environment, together with the other concepts
such as the quality of actions (Q-values), epsilon-greedy exploration, and
rewards.

To enhance the learning capabilities of our proposed method, we analyzed
various (hyper) parameters of the DQL agent such as discount factor, batch
size, and the number of learning episodes to find the best fine-tuning
strategies to self-learn for network intrusion tasks.

Our experimental results demonstrated that the proposed DQL model can learn
effectively from the environment in an autonomous manner and is capable of
classifying different network intrusion attack types with high accuracy.
Through the extensive experiments on parameter fine-tuning, we confirmed that
the best discount factor for our proposed method should be 0.001 with 250
episodes of learning.

For future work, we plan to deploy our proposed method on a realistic cloud-
based environment to enable the DQL agent to improve its self-learning
capabilities and classify the threats with high accuracy in a real-time
manner. We plan to apply our proposed model in improving the self-learning
capabilities in detecting Android-based malware [37, 38] and ransomware [39,
40] to test the generalizability and practicability of our model. We also plan
to deploy our proposed model in other applications such as detecting
unauthorized access in the smart city application [41] and outlier detection
of indoor air quality applications [42, 43, 44].

## References

  * [1] M. P. Stoecklin, “Deeplocker: How AI can power a stealthy new breed of malware,” _Security Intelligence, August_ , vol. 8, 2018.
  * [2] J. Hou, Q. Li, S. Cui, S. Meng, S. Zhang, Z. Ni, and Y. Tian, “Low-cohesion differential privacy protection for industrial internet,” _The Journal of Supercomputing_ , vol. 76, no. 11, pp. 8450–8472, 2020.
  * [3] M. Brundage, S. Avin, J. Clark, H. Toner, P. Eckersley, B. Garfinkel, A. Dafoe, P. Scharre, T. Zeitzoff, B. Filar _et al._ , “The malicious use of artificial intelligence: Forecasting, prevention, and mitigation,” _arXiv preprint arXiv:1802.07228_ , 2018.
  * [4] D. Bodeau and R. Graubart, “Cyber resiliency design principles: selective use throughout the lifecycle and in conjunction with related disciplines,” _McClean (VA): The MITRE Corporation_ , pp. 2017–0103, 2017.
  * [5] K. Toyoshima, T. Oda, M. Hirota, K. Katayama, and L. Barolli, “A dqn based mobile actor node control in wsan: Simulation results of different distributions of events considering three-dimensional environment,” in _International Conference on Emerging Internetworking, Data & Web Technologies_. Springer, 2020, pp. 197–209.
  * [6] N. Saito, T. Oda, A. Hirata, Y. Hirota, M. Hirota, and K. Katayama, “Design and implementation of a dqn based aav,” in _International Conference on Broadband and Wireless Computing, Communication and Applications_. Springer, 2020, pp. 321–329.
  * [7] K. Sethi, R. Kumar, D. Mohanty, and P. Bera, “Robust adaptive cloud intrusion detection system using advanced deep reinforcement learning,” in _International Conference on Security, Privacy, and Applied Cryptography Engineering_. Springer, 2020, pp. 66–85.
  * [8] K. Sethi, R. Kumar, N. Prajapati, and P. Bera, “Deep reinforcement learning based intrusion detection system for cloud infrastructure,” in _2020 International Conference on COMmunication Systems & NETworkS (COMSNETS)_. IEEE, 2020, pp. 1–6.
  * [9] K. Sethi, E. S. Rupesh, R. Kumar, P. Bera, and Y. V. Madhav, “A context-aware robust intrusion detection system: a reinforcement learning-based approach,” _International Journal of Information Security_ , vol. 19, no. 6, pp. 657–678, 2020.
  * [10] Q.-V. Dang and T.-H. Vo, “Reinforcement learning for the problem of detecting intrusion in a computer system,” in _Proceedings of Sixth International Congress on Information and Communication Technology_. Springer, 2022, pp. 755–762.
  * [11] Q. Cappart, T. Moisan, L.-M. Rousseau, I. Prémont-Schwarz, and A. Cire, “Combining reinforcement learning and constraint programming for combinatorial optimization,” _arXiv preprint arXiv:2006.01610_ , 2020.
  * [12] X. Ma and W. Shi, “Aesmote: Adversarial reinforcement learning with smote for anomaly detection,” _IEEE Transactions on Network Science and Engineering_ , 2020.
  * [13] M. Lopez-Martin, B. Carro, and A. Sanchez-Esguevillas, “Application of deep reinforcement learning to intrusion detection for supervised problems,” _Expert Systems with Applications_ , vol. 141, p. 112963, 2020.
  * [14] Z. S. Stefanova and K. M. Ramachandran, “Off-policy q-learning technique for intrusion response in network security,” _World Academy of Science, Engineering and Technology, International Science Index_ , vol. 136, pp. 262–268, 2018.
  * [15] V. François-Lavet, P. Henderson, R. Islam, M. G. Bellemare, and J. Pineau, “An introduction to deep reinforcement learning,” _arXiv preprint arXiv:1811.12560_ , 2018.
  * [16] B. Hu and J. Li, “Shifting deep reinforcement learning algorithm towards training directly in transient real-world environment: A case study in powertrain control,” _IEEE Transactions on Industrial Informatics_ , 2021\.
  * [17] K. Sethi, Y. V. Madhav, R. Kumar, and P. Bera, “Attention based multi-agent intrusion detection systems using reinforcement learning,” _Journal of Information Security and Applications_ , vol. 61, p. 102923, 2021.
  * [18] T. T. Nguyen and V. J. Reddi, “Deep reinforcement learning for cyber security,” _arXiv preprint arXiv:1906.05799_ , 2019.
  * [19] G. Caminero, M. Lopez-Martin, and B. Carro, “Adversarial environment reinforcement learning algorithm for intrusion detection,” _Computer Networks_ , vol. 159, pp. 96–109, 2019.
  * [20] M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, “A detailed analysis of the kdd cup 99 data set,” in _2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications_ , 2009, pp. 1–6.
  * [21] C. Kolias, G. Kambourakis, A. Stavrou, and S. Gritzalis, “Intrusion detection in 802.11 networks: Empirical evaluation of threats and a public dataset,” _IEEE Communications Surveys Tutorials_ , vol. 18, no. 1, pp. 184–208, 2016.
  * [22] S. Iannucci, O. D. Barba, V. Cardellini, and I. Banicescu, “A performance evaluation of deep reinforcement learning for model-based intrusion response,” in _2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS* W)_. IEEE, 2019, pp. 158–163.
  * [23] S. Iannucci, V. Cardellini, O. D. Barba, and I. Banicescu, “A hybrid model-free approach for the near-optimal intrusion response control of non-stationary systems,” _Future Generation Computer Systems_ , 2020.
  * [24] K. Malialis and D. Kudenko, “Distributed response to network intrusions using multiagent reinforcement learning,” _Engineering Applications of Artificial Intelligence_ , vol. 41, pp. 270–284, 2015.
  * [25] P. Holgado, V. A. Villagrá, and L. Vazquez, “Real-time multistep attack prediction based on hidden markov models,” _IEEE Transactions on Dependable and Secure Computing_ , 2017.
  * [26] Z. Zhang, F. Naït-Abdesselam, P.-H. Ho, and Y. Kadobayashi, “Toward cost-sensitive self-optimizing anomaly detection and response in autonomic networks,” _computers & security_, vol. 30, no. 6-7, pp. 525–537, 2011\.
  * [27] B. Fessi, S. Benabdallah, N. Boudriga, and M. Hamdi, “A multi-attribute decision model for intrusion response system,” _Information Sciences_ , vol. 270, pp. 237–254, Jun. 2014. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0020025514002527
  * [28] W. Xu, J. Jang-Jaccard, A. Singh, Y. Wei, and F. Sabrina, “Improving performance of autoencoder-based network anomaly detection on nsl-kdd dataset,” _IEEE Access_ , vol. 9, pp. 140 136–140 146, 2021.
  * [29] J. Zhu, J. Jang-Jaccard, T. Liu, and J. Zhou, “Joint spectral clustering based on optimal graph and feature selection,” _Neural Processing Letters_ , vol. 53, no. 1, pp. 257–273, 2021.
  * [30] W.-C. Shi and H.-M. Sun, “Deepbot: a time-based botnet detection with deep learning,” _Soft Computing_ , vol. 24, pp. 16 605–16 616, 2020.
  * [31] K. Ganju, Q. Wang, W. Yang, C. A. Gunter, and N. Borisov, “Property inference attacks on fully connected neural networks using permutation invariant representations,” in _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ , 2018, pp. 619–633.
  * [32] K. A. da Costa, J. P. Papa, C. O. Lisboa, R. Munoz, and V. H. C. de Albuquerque, “Internet of things: A survey on machine learning-based intrusion detection approaches,” _Computer Networks_ , vol. 151, pp. 147–157, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1389128618308739
  * [33] M. M. Hassan, A. Gumaei, A. Alsanad, M. Alrubaian, and G. Fortino, “A hybrid deep learning model for efficient intrusion detection in big data environment,” _Information Sciences_ , vol. 513, pp. 386–396, 2020.
  * [34] L. M. Ibrahim, D. T. Basheer, and M. S. Mahmod, “A comparison study for intrusion database (kdd99, nsl-kdd) based on self organization map (som) artificial neural network,” _Journal of Engineering Science and Technology_ , vol. 8, no. 1, pp. 107–119, 2013.
  * [35] K. Jiang, W. Wang, A. Wang, and H. Wu, “Network intrusion detection combined hybrid sampling with deep hierarchical network,” _IEEE Access_ , vol. 8, pp. 32 464–32 476, 2020.
  * [36] K. Yang, J. Liu, C. Zhang, and Y. Fang, “Adversarial examples against the deep learning based network intrusion detection systems,” in _MILCOM 2018-2018 IEEE Military Communications Conference (MILCOM)_. IEEE, 2018, pp. 559–564.
  * [37] J. Zhu, J. Jang-Jaccard, and P. A. Watters, “Multi-loss siamese neural network with batch normalization layer for malware detection,” _IEEE Access_ , vol. 8, pp. 171 542–171 550, 2020.
  * [38] J. Zhu, J. Jang-Jaccard, A. Singh, P. A. Watters, and S. Camtepe, “Task-aware meta learning-based siamese neural network for classifying obfuscated malware,” _arXiv preprint arXiv:2110.13409_ , 2021.
  * [39] T. R. McIntosh, J. Jang-Jaccard, and P. A. Watters, “Large scale behavioral analysis of ransomware attacks,” in _International Conference on Neural Information Processing_. Springer, 2018, pp. 217–229.
  * [40] T. McIntosh, J. Jang-Jaccard, P. Watters, and T. Susnjak, “The inadequacy of entropy-based ransomware detection,” in _International Conference on Neural Information Processing_. Springer, 2019, pp. 181–189.
  * [41] F. Sabrina and J. Jang-Jaccard, “Entitlement-based access control for smart cities using blockchain,” _Sensors_ , vol. 21, no. 16, p. 5264, 2021.
  * [42] Y. Wei, J. Jang-Jaccard, F. Sabrina, and H. Alavizadeh, “Large-scale outlier detection for low-cost pm18 sensors,” _IEEE Access_ , vol. 8, pp. 229 033–229 042, 2020.
  * [43] Y. Wei, J. Jang-Jaccard, F. Sabrina, and T. McIntosh, “Msd-kmeans: A novel algorithm for efficient detection of global and local outliers,” _arXiv preprint arXiv:1910.06588_ , 2019.
  * [44] R. Weyers, J. Jang-Jaccard, A. Moses, Y. Wang, M. Boulic, C. Chitty, R. Phipps, and C. Cunningham, “Low-cost indoor air quality (iaq) platform for healthier classrooms in new zealand: Engineering issues,” in _2017 4th Asia-Pacific World Congress on Computer Science and Engineering (APWC on CSE)_. IEEE, 2017, pp. 208–215.