# Omni SCADA Intrusion Detection Using Deep Learning Algorithms

Jun Gao, Luyun Gan, Fabiola Buschendorf, Liao Zhang, Hua Liu, Peixue Li,
Xiaodai Dong and Tao Lu

###### Abstract

We investigate deep learning based omni intrusion detection system (IDS) for
supervisory control and data acquisition (SCADA) networks that are capable of
detecting both temporally uncorrelated and correlated attacks. Regarding the
IDSs developed in this paper, a feedforward neural network (FNN) can detect
temporally uncorrelated attacks at an F1 of 99.967${\pm}$0.005% but correlated
attacks as low as 58${\pm}$2%. In contrast, long-short term memory (LSTM)
detects correlated attacks at 99.56${\pm}$0.01% while uncorrelated attacks at
99.3${\pm}$0.1%. Combining LSTM and FNN through an ensemble approach further
improves the IDS performance with F1 of 99.68${\pm}$0.04% regardless the
temporal correlations among the data packets.

###### Index Terms:

Feedforward Neural Networks, Multilayer Perceptron, Intrusion detection,
Network security, SCADA systems, Supervised learning, LSTM, IDS, Modbus,
Denial of Service (DoS).

††Manuscript received June 14, 2019; revised XY, 2019.This work is supported
in part by the Nature Science and Engineering Research Council of Canada
(NSERC) Discovery Grant (Grant No. RGPIN-2015-06515), Mitacs globalink
program, and Nvidia Corporation TITAN-X GPU grant. (Corresponding author: Tao
Lu)††J. Gao, L. Gan, L. Zhang, X. Dong and T. Lu are with the Department of
Electrical and Computer Engineering, University of Victoria, EOW 448, 3800
Finnerty Rd., Victoria, British Columbia, V8P 5C2, Canada, (e-mail:
{jungao,luyun,liao,xdong,taolu}@uvic.ca)††F. Buschendorf was with the
Department of Computer Science, University of Goettingen, Germany, (e-mail:
fabiola.buschendorf@protonmail.com)††H. Liu and P. Li are with Fortinet
Technology Inc., 899 Kifer Road, Sunnyvale, California 94086, USA, (e-mail:
<EMAIL_ADDRESS>
##  I Introduction

Supervisory control and data acquisition (SCADA) is a well established
industrial system to automate/monitor processes and to gather data from remote
or local equipment such as programmable logic controller (PLC), remote
terminal units (RTU) and human-machine-interfaces (HMI), etc. SCADA became
popular in the 60’s for power plants, water treatment [1], and oil pipelines
[2], etc., which were usually disconnected from the Internet and made use of
hardware devices running proprietary protocols. The network was secured from
harmful attacks because of its obscurity, thus security means were barely
implemented. However, as more and more SCADA systems are adopting its Modbus
protocol over TCP and are accessible via the Internet, they are vulnerable to
cyberattacks. In 2010, Stuxnet [3] was spread over the world and damaged
Iranian nuclear power plants. Since then, the need for industrial network
security became urgent.

To safeguard SCADA networks, an intrusion detection system (IDS) needs to be
implemented. IDS can be signature-based or anomaly-based. Traditionally,
signature-based IDS is the mainstream to detect SCADA attacks. It identifies
specific patterns from traffic data to detect the malicious activities and can
be implemented as policy rules in IDS software such as Snort [4, 5]. Ref. [6]
investigates a set of attacks against Modbus and designs rules to detect
attacks. Ref. [7] proposes a state-relation-based IDS (SRID) to increase the
accuracy and decrease the false negative rate in denial-of-service (DoS)
detection. However, these detection methods are too complicated and only valid
for specific scenarios. Overall, as discovered in the previous research,
signature based IDS is only efficient at finding known attacks and its
performance relies heavily on the experts’ knowledge and experiences.

An anomaly-based IDS [8] overcomes these challenges by introducing machine
learning to identify attack patterns from data. It is also widely used in
other applications such as mobile data misuse detection [9], software [10] and
wireless sensor security [11]. Several machine learning algorithms are
proposed to develop anomaly-based IDS. Linda et al. [12] tailored a neural
network model with error-back propagation and Levenberg-Marquardt learning
rules in their IDS. Rrushi and Kang [13] combined logistic regression and
maximum likelihood estimation to detect anomalies in process control networks.
Poojitha et al. [14] trained a feedforward neural network (FNN) to classify
intrusions on the KDD99 dataset and the industrial control system dataset.
Zhang et al. [15] used support vector machine and artificial immune system to
identify malicious network traffic in the smart grid. Maglaras and Jiang [16]
developed a one-class support vector machine module to train network traces
off-line and detect intrusions on-line. All these machine learning algorithms
are excellent in observing the pattern of attacks from the in-packet features.
None of them, however, takes into account of the temporal features between
packets and thus will not perform well on attacks such as DoS which has strong
temporal dependence.

DoS attacks are among the most popular attacks to slow down or even crush the
SCADA networks. Most of the devices in SCADA operate in low power mode with
limited capacity and are vulnerable to DoS [17]. Up to date, various DoS
types, including spoofing [18], flooding and smurfing [19], etc., have been
reported. Among all types of DoS, flooding DoS is widely-exploited where
hackers send a massive number of packets to jam the target network. In [20],
the author exploits TCP syn flooding attack against the vulnerability of TCP
transmission using hping DoS attack tool. Flooding DoS, along with all other
DoS, is difficult to detect because the in-packet features extracted from each
data packet may not display any suspicious pattern [21].

Similar to DoS, man-in-the-middle (MITM) is another attack that is hard to
detect from observing the in-packet features. It will be more efficient to
detect them by observing the inter-packet patterns in time domain.

Anomaly-based IDS on DoS and MITM becomes popular along with the advances of
machine learning. For example, in [22], an auto-associative kernel regression
(AAKR) coupled with the statistical probability ratio test (SPRT) is
implemented to detect DoS. The result is not satisfactory because the
regression model does not take the temporal signatures of DoS into
consideration. In [23], FNN is used to classify abnormal packets in SCADA with
85% accuray for MITM-based random response injection and 90% accuracy for DoS-
based random response injection attacks but 12% at Replay-based attacks. The
author exploits various attacks including DoS attacks and man-in-the-middle
(MITM) attacks in the testbed built in Modbus/RTU instead of Modbus/TCP. In
[24], the authors propose one-class support vector machine (OCSVM) combined
with k-means clustering method to detect the DoS. They set flags on every 10
packets to reflect the relationships of time series. But the handcrafted
features may be easily by-passed by expert attackers.

To detect temporally correlated attacks such as flooding DoS and MITM, one
should capture the temporal anomaly from these attacks. However, those above
mentioned IDS are not designed to extract temporal patterns from packets
sequence. A more practical approach is to implement an IDS with the capacity
of time series analysis.

Recurrent neural networks (RNN) are the machine learning models that
incorporate the recognition of temporal patterns. Among all RNN models, long
short-term memory (LSTM) gains its popularity from speech recognition [25],
music composition [26] and to machine translation [27]. It is designed to
predict future events according to the information in the previous time steps
and suitable for detecting attacks with temporal correlation. For example,
Ref. [28] applied LSTM for distributed DoS with high success rate. In [29] the
authors also developed a time-series anomaly detector based on LSTM [30]
networks to enhance the performance of IDS and apply this framework to the
dataset in [31]. But the number of DoS attacks in the dataset is relatively
small and the time interval for the DoS attack in this dataset is too long,
making the detection inefficient.

Despite of the excellent performance in detecting temporally correlated
attacks such as DoS and MITM, the capacity of RNN to detect temporally
uncorrelated attacks is limited compared to other types of machine learning
algorithms such as FNN. In this paper, utilizing the advantages of both RNN
and FNN while avoiding their disadvantages, we implement an omni IDS that can
detect all attacks regardless of their temporal dependence. On a SCADA testbed
[17], we demonstrate that our IDS reaches the highest performance against all
attacks compared to those that employ RNN or FNN alone.

##  II SCADA Testbed and Data Synthesize

Our IDS is tested on a simulated SCADA testbed. A simulated network has the
advantage of being easy to maintain, change and operate and is less costly
than a real device network. A software testbed, which simulates a SCADA
industry network and emulates the attacks was built by L. Zhang [17] on the
basis work of T. Morris [32]. In the past, several preliminary researches on
SCADA security had been conducted on this testbed [33, 34]. The attack target
is a simple SCADA network, consisting of two tanks using Modbus over TCP. The
liquid level of the tanks is controlled by pumps and measured by sensors via
Modbus control information. The purpose of this network is to attract hackers
and study possible defense methods. Such a system is called Honeypot, as it
fools the attacker while exploiting his behaviour. This tank system is
developed by the MBLogic HMIBuilder and HMIServer toolkit [35] and has been
extended by L. Zhang in [17]. The HMI’s purpose is to pull data from the
sensor or send the desired pump speed to the motor periodically. The back end
of the HMI is a PLC while the front end is a web browser.

As this system is simulated, we make use of four virtual machines as shown in
Fig. 1. The SCADA system runs on a Modbus master and several slaves. On a
virtual host called Nova the HMI is deployed, thus we refer to this host as
Modbus master. In order to extend the network, some Modbus slaves such as PLCs
are simulated by the HoneyD software [36]. This will provide a more realistic
Honeypot. The role of a Modbus slave is to process commands from the master by
pulling sensory data about the tank system from the PLCs and sending it back
to the master.

Figure 1: Testbed architecture [17]

The data needed to feed the neural network is generated by an attack machine
using a virtual host named Kali. Kali is a Debian-derived Linux host used for
penetration testing and features many attack and defense tools. Additional to
the message exchange between the Modbus master (Nova) and its slaves we can
launch normal traffic mixed with various attacks from Kali. A command line
tool, Modpoll [37], is used to send Modbus instructions to the PLC which
controls sensible tank system variables. An example Modpoll instruction which
sends a pump speed of 5 to the system looks like this:

⬇

$ modpoll -0 -r 32210 10.0.0.5 5

The command addresses a simulated PLC with an IP address of 10.0.0.5 and a
register address which contains either a threshold value (register 42212 -
42215), the current pump speed (32210) or the tank level (42210,42211),
measured by the sensors. Modpoll will send Modbus requests with function code
16 to attempt a write action to the specified registers. By modifying the pump
speed the attackers can exceed the allowed tank level and create serious
damage to a system. A script on Kali will randomly choose between these normal
or malicious Modbus instructions and will launch a Modpoll instruction with
another randomly chosen parameter. This will ensure desired distribution of
attack/non-attack data.

The traffic will be recorded by the fourth virtual machine referred to as
“Defense Wall”, which operates in the bridge mode and thus is invisible to the
attacker. With PyShark we capture the traffic between Nova and Modbus slaves
and between the attacker machine Kali and the PLCs. During this process we can
label each packet as malicious or normal.

###  II-A Features extracted from the data packets

In our testbed, we use a self-developed IDS installed on “Defense Wall” to
extract 19 features from each data packet captured. They are listed below:

  1. 1.

Source IP address;

  2. 2.

Destination IP address;

  3. 3.

Source port number;

  4. 4.

Destination port number;

  5. 5.

TCP sequence number;

  6. 6.

Transaction identifier set by the client to uniquely identify each request;

  7. 7.

Function code identify the Modbus function used;

  8. 8.

Reference number of the specified register;

  9. 9.

Modbus register data;

  10. 10.

Modbus exception code;

  11. 11.

Time stamp;

  12. 12.

Relative time;

  13. 13.

Highest threshold;

  14. 14.

Lowest threshold;

  15. 15.

High threshold;

  16. 16.

Low threshold;

  17. 17.

Pump speed;

  18. 18.

Tank 1 water level;

  19. 19.

Tank 2 water level.

Here, the “Relative time” represents the time in seconds for packets relative
to the first packet in the same TCP session. To reduce the periodicity of this
feature, we reset it to zero when “Relative time” reaches 3,000 seconds.

In our IDS, we adopt feature scaling of each feature $x$ in the dataset
according to

$x^{\prime}=\frac{x-\bar{x}}{\sigma_{x}}$ (1)

where $\bar{x}$ and $\sigma_{x}$ are the mean and standard deviation of
original feature $x$ and $x^{\prime}$ is the re-scaled feature from $x$ with
zero mean and unity variance.

###  II-B Types of attacks in our datasets

Figure 2: Data packet types distribution in Dataset I, II and online script.
The ones with a superscript “*” are temporally correlated attacks.

Using our scripts, we created two datasets. As illustrated in Fig. 2, in
addition to “Normal” data packets, Dataset I contains attacks that are
uncorrelated in time domain while Dataset II contains temporally dependent
attacks. Here we have incorporated 10 attacks in our testbed. 7 of them are
temporally uncorrelated while the remaining 3 are correlated. The temporally
uncorrelated attacks include “Pump Speed” (Pump), “Tank 1 Level” (T1), “Tank 2
Level” (T2), “Threshold Highest” (HH), “Threshold Lowest” (LL), “Threshold
High” (H) and “Threshold Low” (L) whose detailed descriptions can be found in
[17, 32].

Among all temporally correlated attacks, two types of flooding DoS attacks are
included [31]. The first labelled as “Scan flooding” (SCAN) is to send massive
scan command, resulting in increasing latency of communications between the
HMI and the sensors in SCADA. The second type labelled as “Incorrect CRC”
(CRC) is sending massive packets with incorrect cyclic redundancy check (CRC)
to cause latency of master.

Another temporally correlated attack included in this testbed is “Man-in-the-
middle” (MITM) attack. It is an eavesdropping where the attacker monitors the
communication traffics between two parties secretly. Here, the MITM attack is
launched by Ettercap [38] using ARP spoofing [39]. One effective way to detect
ARP spoofing is identifying the Media Access Control (MAC) address in layer 2
of OSI model. However, most of Network IDSs (NIDS) do not support the
protocols in layer 2 such as ARP and MAC protocols. Even Snort requires an ARP
spoof preprocessor [40] to collect the MAC address information to detect ARP
spoofing. Besides, the victim host of ARP spoofing attack would experience
packets retransmissions. For SCADA networks, packet retrasmissions or delay
may cause great damages. Therefore, the IDS should raise alert when it detects
either MITM attack or packets retransmissions. To make the IDS robust in
detecting both MITM and packets retransmissions we remove the MAC address
feature which was used for labeling MITM attack from the datasets for training
neural networks.

At the first stage, FNN and LSTM IDSs will be trained as binary classifiers
that only predict attacks from normal traffic and tested on these datasets
separately for performance comparisons. In on-line phases, these two IDSs
along with our FNN-LSTM ensemble IDS will be trained as multi-class
classifiers by the combined datasets to predict various types of attacks from
normal traffics and implemented on the testbed. In addition, we also implement
a script that can launch realtime attacks for online testing. The online
script will randomly launch normal traffic, temporally uncorrelated and
correlated attacks with ratios shown in the table to examine the omni-
detection capability of different IDSs.

##  III IDS Implementation

In this paper, we implemented three IDSs: a conventional FNN, a LSTM and a
FNN-LSTM ensemble IDS. Here, we use Keras [41] to implement tensorflow [42]
based machine learning models with AdamOptimizer [43] to train our model. The
structure of these IDSs are detailed in the following subsections.

###  III-A FNN IDS

(a)

(b)

Figure 3: (a)The schematics of the FNN IDS (b) Details of each neuron in FNN

The basic structure of the FNN IDS is illustrated in Fig. 3. A typical FNN is
formed by an input layer, an output layer and one or more hidden layers in-
between. Each layer has a number of neurons that use the neuron outputs from
the previous layer as input and produces output to the neurons in next layer.
In our case, inputs are the scaled and normalized features extracted from the
data packets, and outputs are the predictions of attacks and normal events.
Mathematically, the FNN can be expressed as:

$\begin{array}[]{rcl}\textbf{z}^{(1)}&=&\textbf{W}^{(1)}\textbf{x}+\textbf{b}^{(1)},\textbf{h}_{1}=f_{h}(\textbf{z}^{(1)})\\\
\textbf{z}^{(2)}&=&\textbf{W}^{(2)}\textbf{h}_{1}+\textbf{b}^{(2)},\textbf{h}_{2}=f_{h}(\textbf{z}^{(2)})\\\
&...&\\\
\textbf{z}^{(N+1)}&=&\textbf{W}^{(N+1)}\textbf{h}_{N}+\textbf{b}^{(N+1)},\hat{\textbf{y}}=\textbf{z}^{(N+1)}\end{array}$
(2)

where $N$ is the number of hidden layers, $f_{h}$ is the ReLU activation
function, and $\textbf{W}^{(1)},\textbf{W}^{(2)},...,\textbf{W}^{(N+1)}$,
$\textbf{b}^{(1)},\textbf{b}^{(2)},...,\textbf{b}^{(N+1)}$ are the parameters
to be trained. Here we use softmax cross entropy as our loss function, which
can be expressed as

$f_{L}(\hat{\textbf{y}},\textbf{y})=-\sum_{i=1}^{C}\textbf{y}_{i}\log(f_{s}(\hat{\textbf{y}_{i}}))$
(3)

where $\hat{\textbf{y}}$ is the predicted label and y the ground truth. $C$ is
the number of all possible classes, $\textbf{y}_{i}$ and
$\hat{\textbf{y}_{i}}$ are the actual and predicted labels that belongs to
class $i$, and $f_{s}$ is the softmax function.

###  III-B LSTM IDS

The LSTM is built on a collection of single LSTM cells [31]. The structure of
single LSTM cells is as Fig. 4a. Each LSTM cell has 3 gates: input gate,
forget gate and output gate. The input gate selects useful information and
push it to the cell. The irrelevant information will be discarded in forget
gate. The output gate outputs the activation state $o_{t}$. A hidden state
vector $h_{t}$ is transferred to the next time steps.

(a)

(b)

Figure 4: The structure of (a) single LSTM cell, (b) LSTM Network.

The following equations represent the processes of a single LSTM cell:

$\begin{array}[]{rcl}\textbf{f}_{t}&=&\sigma(\textbf{W}_{f}x_{t}+\textbf{U}_{f}h_{t-1}+\textbf{b}_{f})\\\
\textbf{i}_{t}&=&\sigma(\textbf{W}_{i}x_{t}+\textbf{U}_{i}h_{t-1}+\textbf{b}_{i})\\\
\textbf{o}_{t}&=&\sigma(\textbf{W}_{o}x_{t}+\textbf{U}_{o}h_{t-1}+\textbf{b}_{o})\\\
\textbf{c}_{t}&=&\textbf{f}_{t}\circ\textbf{c}_{t-1}+\textbf{i}_{t}\circ\sigma_{g}(\textbf{W}_{c}x_{t}+\textbf{U}_{c}h_{t-1}+\textbf{b}_{c})\\\
\textbf{h}_{t}&=&\textbf{o}_{t}\circ\sigma_{g}(\textbf{c}_{t})\end{array}$ (4)

where $\sigma_{g}$ is hyperbolic tangent function and $\sigma$ is sigmoid
function. $\circ$ is the element-wise product notation. $W$, $U$, $b$ are the
weight matrix for the gates.

Shown in Fig. 4b, the LSTM IDS includes two LSTM layers with 10 LSTM cells in
each layer. An activation layer with sigmoid activation function is placed
after the last LSTM layer. The ${\\{x_{1},x_{2},...,x_{t}}\\}$ vector is the
input vector containing features of packets within $t$ time steps. The dataset
is reshaped in this format and fit into the LSTM model. In our model, we set
$t=10$. The loss function in this model is binary cross entropy and the
optimizer is Adam optimizer [44].

###  III-C FNN-LSTM Ensemble IDS

Figure 5: Ensemble Model.

Our FNN-LSTM ensemble IDS aims to combine the advantages of both FNN and LSTM
while avoiding their weaknesses [45]. The schematics of this model is as shown
in Fig. 8. In this model, the data packet features are fed into FNN and LSTM
simultaneously to predict attacks as a multi-class classifier. The output
labels of both are concatenated as the input of a multilayer perceptron, which
through training, is capable of voting for the best prediction of the data
packet under investigation.

##  IV Experiment and Result

To demonstrate their capability for detecting attacks with/without temporal
correlation, we first implement FNN and LSTM IDSs to establish references for
comparison. At this stage, the IDSs only conduct binary classification to
predict if the data packet under investigation is normal (labeled as “0”) or
attack (labeled as “1”). Consequently, sigmoid function

$\sigma(z)=\frac{e^{z}}{1+e^{z}}$ (5)

is selected as the activation function. Here, $z$ is the output of the
previous LSTM layer.

###  IV-A Hyper parameters tuning

Both IDSs are trained using 70% of the randomly chosen samples from the two
datasets and tested with the remaining 30% samples following the 10-fold
training/testing procedure so that the average and standard deviation of
figures of merits including precision, recall and $\mathrm{F_{1}}$ can be used
for evaluation.

To determine the number of hidden layers necessary for our FNN, we computed
$\mathrm{F_{1}}$ with 0, 1 and 2 hidden layers where the values of 99.22%,
99.96% and 99.97% are obtained respectively. As shown, employing 1 hidden
layers in FNN will increase the $\mathrm{F_{1}}$ by more than 7% while using 2
hidden layers the improvement is minimal. Therefore, we select 1 hidden layer
in our FNN implementation.

In addition, to circumvent overfitting, we further adopted early stop
procedure in FNN such that the optimization stops when the number of epochs
whose relative differences of loss between consecutive ones are less than
$\mathrm{10^{-6}}$ reaches 35 [46]. Similarly, LSTM adopts early stop if
either maximum epochs reach 3.

In implementation of LSTM, we connect 10 LSTM cells in input layer where the
features from 10 consecutive data packets are entered into the cells to
predict if the last packet is normal or an attack. In training, we adopt mini-
batch with a batch size of $\mathrm{1,000}$.

###  IV-B Detection of temporally uncorrelated attacks

We exploit the Dataset I described in Section II to compare the detection
capability of FNN and LSTM for temporally uncorrelated attacks. To verify the
models, learning curves are plotted in Fig. 6 where training and testing
losses as a function of training samples are plotted. Here the average value
and standard deviation after 10 fold training/testing are represented by
circle markers and error bars respectively. As shown, with training samples
exceeding 40,000, FNN training and testing losses (blue dashed lines) start to
converge while LSTM (red solid lines) converges at sample size larger than
60,000. Overall, it confirms that the number of samples in Dataset I is
sufficient for the training and testing of our IDS.

Figure 6: Learning Curves of FNN and LSTM using temporally-uncorrelated-attacks dataset (Dataset I). TABLE I: Comparison of the temporally-uncorrelated-attacks detection. | Precision | Recall | F1
---|---|---|---
FNN | $\mathrm{99.996{\pm}0.006}$ | $\mathrm{99.84{\pm}0.05}$ | $\mathrm{99.92{\pm}0.03}$
LSTM | $\mathrm{99.88{\pm}0.06}$ | $\mathrm{98.7{\pm}0.4}$ | $\mathrm{99.3{\pm}0.1}$
TABLE II: Confusion matrices of temporally-uncorrelated-attacks detection using Dataset I (averaged over 10 trials) | Predicted
---|---
| Normal | Attacks
Actual | Normal | FNN | $\mathrm{69,845.4}$ | $\mathrm{0.6}$
LSTM | $\mathrm{69,902.2}$ | $\mathrm{22.8}$
Attacks | FNN | $\mathrm{30.7}$ | $\mathrm{19,741.3}$
LSTM | $\mathrm{241.9}$ | $\mathrm{19,448.1}$

After the IDSs are trained, we use 30% of samples in Dataset I for 10 fold
testing. Also shown in Table II and I, on average, for FNN, only 0.6 of the
69,846 normal datapackets are mislabelled as attacks while only 30.7 out of
19,771 actual attacks are mislabelled as normal traffic, yielding the
precision, recall and $\mathrm{F_{1}}$ to be $\mathrm{99.996{\pm}0.006\%}$,
$\mathrm{99.84{\pm}0.05\%}$, and $\mathrm{99.92{\pm}0.03\%}$. In comparison,
LSTM mislabelled 22.8 normal packets as attacks and 241.9 attacks as normal
packets, resulting the figures of merits to be $\mathrm{99.88{\pm}0.06\%}$,
$\mathrm{98.7{\pm}0.4\%}$ and $\mathrm{99.3{\pm}0.1\%}$. The comparison
demonstrates that FNN outperformed LSTM in detecting temporally uncorrelated
attacks where recognition of the in-packet feature patterns is critical.

###  IV-C Detection of temporally correlated attacks

Figure 7: Learning Curves of FNN and LSTM using temporally-correlated-attacks dataset (Dataset II). TABLE III: Comparison of temporally correlated attacks $\mathrm{(\%)}$ | Precision | Recall | F1
---|---|---|---
FNN | $\mathrm{73{\pm}2}$ | $\mathrm{49{\pm}4}$ | $\mathrm{58{\pm}2}$
LSTM | $\mathrm{99.60{\pm}0.01}$ | $\mathrm{99.52{\pm}0.02}$ | $\mathrm{99.56{\pm}0.01}$
TABLE IV: Confusion matrix of temporally correlated attacks | Predicted
---|---
| Normal | Attacks
Actual | Normal | FNN | $\mathrm{28,668.3}$ | $\mathrm{5,044.7}$
LSTM | $\mathrm{33,504.0}$ | $\mathrm{105.0}$
Attacks | FNN | $\mathrm{13,510.4}$ | $\mathrm{13,169.6}$
LSTM | $\mathrm{128.4}$ | $\mathrm{26,652.6}$

In this subsection FNN and LSTM are re-trained and tested using Dataset II for
comparison of their temporally correlated attacks detection comparison. Again
the learning curves in Fig. 8 shows that both FNN (blue dashed lines) and LSTM
(red solid lines) converge at training samples exceeding 10,000 while LSTM
clearly shows lower testing loss. This confirms the sufficiency of our dataset
to generalize the IDS models.

The performance of each model is compared in Table III and IV. As shown, FNN
is inefficient in detecting temporally correlated attacks with precision,
recall and $\mathrm{F_{1}}$ scores as low as $\mathrm{73{\pm}2\%}$,
$\mathrm{49{\pm}4\%}$ and $\mathrm{58{\pm}2}$ respectively. In particular,
5,044.7 out of 33,713 normal packets are mislabelled to attacks while 13,510.4
out of 26,680 actual attacks are mislabelled to normal traffic. It is evident
that the poor performance of FNN is caused by its inability to inter-packet
features. In contrast, LSTM displays an outstanding performance on the
corresponding figures of merits to be $\mathrm{99.60{\pm}0.01\%}$,
$\mathrm{99.52{\pm}0.02\%}$ and $\mathrm{99.56{\pm}0.01\%}$ where only 105.0
normal packets are mislabelled as attacks and 128.4 attacks packets are
mislabelled as normal traffic. As expected, LSTM outperforms FNN in detecting
temporally correlated attacks due to its inherent nature to observe data
pattern in time domain.

###  IV-D Omni attacks detection

TABLE V: Macro-average comparison of omni-attacks detection | Precision | Recall | F1
---|---|---|---
FNN | $\mathrm{88{\pm}1}$ | $\mathrm{89.2{\pm}0.8}$ | $\mathrm{87.4{\pm}0.6}$
LSTM | $\mathrm{99.54{\pm}0.03}$ | $\mathrm{99.01{\pm}0.07}$ | $\mathrm{99.27{\pm}0.05}$
Ensemble | $\mathrm{99.76{\pm}0.05}$ | $\mathrm{99.57{\pm}0.03}$ | $\mathrm{99.68{\pm}0.04}$

(a)

(b)

(c)

Figure 8: (a) Precision, (b) Recall and (c) $\mathrm{F_{1}}$ of individual
attacks in omni-attacks detection.

Recognizing the mutual strength of FNN and LSTM IDSs in detecting temporally
correlated and uncorrelated attacks, we here combine the advantages of both
for an omni attacks detector through ensemble approach. The structure of FNN-
LSTM ensemble is described in Subsection III-C. To implement, we first
remodelled FNN and LSTM to multi-class classifiers so that different attacks
can be distinguished. Dataset I and II are combined and used to train FNN and
LSTM independently. The outputs of both are combined to form the input
features of a multilayer perceptron for training. After training, FNN, LSTM
and FNN-LSTM ensemble IDSs are integrated into our SCADA testbed to detect and
classify attacks. The traffic is generated online using the script that
generates a pre-determined ratio of normal, temporally correlated and
uncorrelated attacks as described in Fig 2. To estimate the figures of merits,
we evenly divide the predicted labels to 10 portions and compute the average
and standard deviation of macro-averaged precision, recall and
$\mathrm{F_{1}}$. As shown in Table V, among all the three IDSs, the FNN
achieve lowest performance with macro-averaged figures of merits of
$\mathrm{88{\pm}1\%}$, $\mathrm{89.2{\pm}0.8\%}$ and $\mathrm{87.4{\pm}0.6\%}$
while LSTM reaches $\mathrm{99.54{\pm}0.03\%}$, $\mathrm{99.01{\pm}0.07\%}$
and $\mathrm{99.27{\pm}0.05\%}$. In contrast, the FNN-LSTM ensemble IDS
further outperforms both with figures of merits to be
$\mathrm{99.76{\pm}0.05}$, $\mathrm{99.57{\pm}0.03}$ and
$\mathrm{99.68{\pm}0.04}$. Detailed analysis in Fig. 8 further confirms that
the under-performance of FNN (yellow bars) are due to the mislabels of
temporally correlated attacks (MITM, CRC and SCAN) while the performance of
LSTM (red bars) by temporally uncorrelated attacks (“Pump Speed (Pump)”, “Tank
1 Level (T1)”, and “Threshold High (H)”, etc.). Overall, the FNN-LSTM ensemble
demonstrates a consistent out-performance over them in all types of attacks.

##  V Conclusion

In this paper we demonstrated that the FNN-LSTM ensemble IDS can detect all
types of cyberattacks regardless of the their temporal relevance. In opposite,
FNN only performance well in temporally uncorrelated attacks and LSTM is
relatively weak in uncorrelated attacks. In future research we will further
improve our model through field trials.

## Acknowledgment

## References

  * [1] S. Adepu and A. Mathur, “An investigation into the response of a water treatment system to cyber attacks,” in _17th IEEE International Symposium on High Assurance Systems Engineering (HASE 2016)_ , pp. 141–148, 2016.
  * [2] H. Huang, W. Zhang, G. Qi, S. Ma, Y. Yang, F. Yan, and P. Chen, “Research on accident inversion and analysis method of the oil and gas pipeline SCADA system,” in _2014 Sixth International Conference on Measuring Technology and Mechatronics Automation_ , pp. 492–496, 2014.
  * [3] S. Karnouskos, “Stuxnet worm impact on industrial cyber-physical system security,” in _IEEE 37th Annual Conference of the Industrial Electronics Society （IECON 2011）_ , pp. 4490–4494, 2011.
  * [4] M. Roesch, “Snort - lightweight intrusion detection for networks,” in _Proceedings of the 13th USENIX Conference on System Administration_ , ser. LISA ’99. Berkeley, CA, USA: USENIX Association, pp. 229–238, 1999.
  * [5] M. A. Aydın, A. H. Zaim, and K. G. Ceylan, “A hybrid intrusion detection system design for computer network security,” _Computers & Electrical Engineering_, vol. 35, no. 3, pp. 517 – 526, 2009.
  * [6] W. Gao and T. Morris, “On cyber attacks and signature based intrusion detection for modbus based industrial control systems,” _Journal of Digital Forensics, Security and Law_ , Vol. 9, No. 1, Article. 3, 2014.
  * [7] Y. Wang, Z. Xu, J. Zhang, L. Xu, H. Wang, and G. Gu, “SRID: State relation based intrusion detection for false data injection attacks in SCADA,” in _Computer Security - ESORICS 2014_ , pp. 401–418, 2014.
  * [8] V. Jyothsna, V. V. Rama Prasad, and K. Munivara Prasad, “A review of anomaly based intrusion detection systems,” _International Journal of Computer Applications_ , vol. 28, pp. 26–35, 08 2011.
  * [9] D. Damopoulos, S. A. Menesidou, G. Kambourakis, M. Papadaki, N. Clarke, and S. Gritzalis, “Evaluation of anomaly-based IDS for mobile devices using machine learning classifiers,” _Security and Communication Networks_ , vol. 5, no. 1, pp. 3–14, 2012.
  * [10] G. Nascimento and M. Correia, “Anomaly-based intrusion detection in software as a service,” in _2011 IEEE/IFIP 41st International Conference on Dependable Systems and Networks Workshops (DSN-W)_ , pp. 19–24, June 2011.
  * [11] M. S. Islam and S. A. Rahman, “Anomaly intrusion detection system in wireless sensor networks: security threats and existing approaches,” _International Journal of Advanced Science and Technology_ , vol. 36, no. 1, pp. 1–8, 2011.
  * [12] O. Linda, T. Vollmer, and M. Manic, “Neural network based intrusion detection system for critical infrastructures,” in _2009 International Joint Conference on Neural Networks_ , pp. 1827–1834, 2009.
  * [13] J. Rrushi and K.-D. Kang, “Detecting anomalies in process control networks,” _Critical Infrastructure Protection III_ , pp. 151–165, 2009.
  * [14] G. Poojitha, K. N. Kumar, and P. J. Reddy, “Intrusion detection using artificial neural network,” in _Computing Communication and Networking Technologies (ICCCNT), 2010 International Conference on Computing, Communication and Networking Technologies_. pp. 1–7, 2010.
  * [15] Y. Zhang, L. Wang, W. Sun, R. C. Green II, and M. Alam, “Distributed intrusion detection system in a multi-layer network architecture of smart grids,” _IEEE Transactions on Smart Grid_ , vol. 2, no. 4, pp. 796–808, 2011.
  * [16] L. A. Maglaras and J. Jiang, “Intrusion detection in scada systems using machine learning techniques,” in _2014 Science and Information Conference_. London, pp. 626–631, 2014.
  * [17] L. Zhang, “An implementation of scada network security testbed,” _Master’s thesis, University of Victoria, Victoria, BC_ , 2015.
  * [18] L. T. Heberlein and M. Bishop, “Attack class: address spoofing,” in _Proceedings of the 19th National Information Systems Security Conference_ , 1997.
  * [19] S. Kumar, “Smurf-based distributed denial of service (ddos) attack amplification in internet,” in _Second International Conference on Internet Monitoring and Protection (ICIMP 2007)_ , San Jose, CA, pp. 25–25, 2007.
  * [20] B. Chen, N. Pattanaik, A. Goulart, K. Butler-Purry, and D. Kundur, “Implementing attacks for modbus/tcp protocol in a real-time cyber physical system test bed,” _Proceedings - CQR 2015: 2015 IEEE International Workshop Technical Committee on Communications Quality and Reliability_ , pp. 1–6, 2015.
  * [21] N. Sayegh, A. Chehab, I. H. Elhajj, and A. Kayssi, “Internal security attacks on scada systems,” in _2013 Third International Conference on Communications and Information Technology (ICCIT)_ , pp. 22–27, 2013.
  * [22] D. Yang, A. Usynin, and J. Hines, “Anomaly-based intrusion detection for SCADA systems,” in _Proceedings of the 5. International Topical Meeting on Nuclear Plant Instrumentation Controls, and Human Machine Interface Technology_ , vol. 43, no. 47, pp. 797–803, 2006.
  * [23] W. Gao, T. Morris, B. Reaves, and D. Richey, “On scada control system command and response injection and intrusion detection,” _2010 eCrime Researchers Summit_ , Dallas, TX, pp. 1-9, 2010.
  * [24] L. A. Maglaras, J. Jiang, and T. Cruz, “Integrated ocsvm mechanism for intrusion detection in SCADA systems,” _Electronics Letters_ , vol. 50, no. 25, pp. 1935–1936, 2014.
  * [25] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech Recognition with Deep Recurrent Neural Networks,” _arXiv e-prints_ , p. arXiv:1303.5778, Mar. 2013\.
  * [26] D. Eck and J. Schmidhuber, “A first look at music composition using lstm recurrent neural networks,” Technical Report. Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale, 2002.
  * [27] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” _arXiv e-prints_ , p. arXiv:1409.3215, Sep. 2014\.
  * [28] C. D. McDermott, F. Majdani, and A. V. Petrovski, “Botnet detection in the internet of things using deep learning approaches,” in _2018 International Joint Conference on Neural Networks (IJCNN)_ , Rio de Janeiro, pp. 1–8, 2018.
  * [29] C. Feng, T. Li, and D. Chana, “Multi-level anomaly detection in industrial control systems via package signatures and lstm networks,” in _2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)_ , Denver, CO, pp. 261–272, 2017.
  * [30] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Comput._ , vol. 9, no. 8, pp. 1735–1780, Nov. 1997.
  * [31] T. Morris and W. Gao, “Industrial control system traffic data sets for intrusion detection research,” in _Critical Infrastructure Protection VIII_ , J. Butts and S. Shenoi, Eds. ICCIP 2014. IFIP Advances in Information and Communication Technology, vol 441. pp 65–78,2014.
  * [32] T. Morris, A. Srivastava, B. Reaves, W. Gao, K. Pavurapu, and R. Reddi, “A control system testbed to validate critical infrastructure protection concepts,” _International Journal of Critical Infrastructure Protection_ , vol. 4, no. 2, pp. 88 – 103, 2011.
  * [33] H. Wang, T. Lu, X. Dong, P. Li, and M. Xie, “Hierarchical online intrusion detection for scada networks,” _arXiv_ , p. 1611.09418, 2016.
  * [34] S. Patel, “IEC-61850 Protocol Analysis and Online Intrusion Detection System for SCADA Networks using Machine Learning,” Master’s thesis, University of Victoria, Victoria, BC, 2017.
  * [35] MBLogic. Mblogic homepage. [Online]. Available: http://mblogic.sourceforge.net/index.html
  * [36] N. Provos. Honeyd. [Online]. Available: http://www.honeyd.org/
  * [37] ProconX Pty Ltd. Modpoll modbus master simulator. [Online]. Available: http://www.modbusdriver.com/modpoll.html
  * [38] Ettercap, a comprehensive suite for man in the middle attacks. [Online]. Available: http://openmaniak.com/ettercap.php
  * [39] Z. Trabelsi and W. El-Hajj, “Arp spoofing: A comparative study for education purposes,” in _2009 Information Security Curriculum Development Conference_ , ser. InfoSecCD ’09. New York, NY, USA: ACM, pp. 60–66, 2009.
  * [40] Snort ARP spoof preprocessor. [Online]. Available: http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node17.html
  * [41] F. Chollet _et al._ , “Keras,” https://keras.io, 2015.
  * [42] M. Abadi, et. al. “Tensorflow: A system for large-scale machine learning,” in _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)_ , pp. 265–283, 2016.
  * [43] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [44] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _CoRR_ , vol. abs/1412.6980, 2014.
  * [45] T. G. Dietterich, “Ensemble methods in machine learning,” in _Proceedings of the First International Workshop on Multiple Classifier Systems_ , ser. MCS ’00. London, UK, UK: Springer-Verlag, pp. 1–15, 2000.
  * [46] L. Prechelt, “Early stopping-but when?” in _Neural Networks: Tricks of the Trade_. London, UK, UK: Springer-Verlag, pp. 55–69, 2012.


# Forward-Looking Sonar Patch Matching: Modern CNNs, Ensembling, and
Uncertainty

Arka Mallick1 and Paul Plöger1 and Matias Valdenegro-Toro2 1Department of
Computer Science, Hochschule Bonn-Rhein-Sieg, 53757 Sankt Augustin, Germany
<EMAIL_ADDRESS>Research Center for Artificial Intelligence,
28359 Bremen, Germany<EMAIL_ADDRESS>
###### Abstract

Application of underwater robots are on the rise, most of them are dependent
on sonar for underwater vision, but the lack of strong perception capabilities
limits them in this task. An important issue in sonar perception is matching
image patches, which can enable other techniques like localization, change
detection, and mapping. There is a rich literature for this problem in color
images, but for acoustic images, it is lacking, due to the physics that
produce these images. In this paper we improve on our previous results for
this problem (Valdenegro-Toro et al, 2017), instead of modeling features
manually, a Convolutional Neural Network (CNN) learns a similarity function
and predicts if two input sonar images are similar or not. With the objective
of improving the sonar image matching problem further, three state of the art
CNN architectures are evaluated on the Marine Debris dataset, namely DenseNet,
and VGG, with a siamese or two-channel architecture, and contrastive loss. To
ensure a fair evaluation of each network, thorough hyper-parameter
optimization is executed. We find that the best performing models are DenseNet
Two-Channel network with 0.955 AUC, VGG-Siamese with contrastive loss at 0.949
AUC and DenseNet Siamese with 0.921 AUC. By ensembling the top performing
DenseNet two-channel and DenseNet-Siamese models overall highest prediction
accuracy obtained is 0.978 AUC, showing a large improvement over the 0.91 AUC
in the state of the art.

##  I Introduction

More than two-thirds of our planet’s surface is covered by oceans and other
water bodies. For a human, it is often impossible to explore it extensively.
The need for venturing into potentially dangerous underwater scenarios appear
regularly, for example, finding new energy sources, monitoring tsunamis,
global warming, wreckage search, or maybe just to learn about deep sea
ecosystems. This motivates design and deployment of robots in underwater
scenarios, and much research goes in this direction. Some exploration or
monitoring tasks require the robot to ”see” underwater, to make intelligent
decisions. But the underwater environment is very difficult for optical
cameras, as light is attenuated and absorbed by the water particles. And a lot
of real-life monitoring and mapping tasks take place in a cluttered and turbid
underwater scenario. The limited visibility range of an optical sensor is a
big challenge. Hence, sonar is a more practical choice for underwater sensing,
as acoustic waves can travel long distances with comparatively little
attenuation.

An underwater robot, equipped with sonar image sensors, regularly needs to
perform basic tasks such as object detection and recognition, navigation,
manipulation etc. In underwater scenarios, sonar patch matching functionality
is very useful in several applications such as data association in
simultaneous localization and mapping (SLAM), object tracking, sonar image
mosaicing [1] etc. Patch matching, in general, is heavily used in computer
vision and image processing applications for low-level tasks like image
stitching [2], deriving structure from motion [3], also in high-level tasks
such as object instance recognition [4], object classification [5], multi-view
reconstruction [6], image-retrieval etc.

Typical challenges in patch matching tasks are different viewing points,
variations in scene insonification, occlusion, and different sensor settings.
For sonar patch matching the common challenges with acoustic vision adds to
the overall complexity. For example, low signal-to-noise ratio, lower
resolution, unwanted reflections, less visibility etc. Because of these
challenges, the underlying object features might not be so prominent as in a
normal optical image. It has also been found that it is very challenging to
manually design features for sonar images, and popular hand designed features
such as SIFT [7] are not always very effective in sonar images [8]. For these
reasons, patch matching for sonar images remains a topic of research interest.

Figure 1: Use of convolutional network for learning general similarity
function for image patches. The patches in the image are samples taken from
the data used in this work. Inspired from Zagoruyko et al. [9]

##  II State of the Art

Sonar image patch matching is more difficult than normal optical matching
problem. This is because sonar images have additional challenges such as non-
uniform insonification, low signal-to-noise ratio, poor contrast [10], low
resolution, low feature repeatability [11] etc. But sonar image matching has
important applications like in sonar registration, mosaicing [12], [1] and
mapping of seabed surface [13] etc. While Kim et al. [12] used Harris corner
detection and matched key-points to register sonar images, Hurtos et al. [1]
incorporated Fourier-based features for registration of FLS images.
Negahdaripour et al. [13] estimated mathematical models from the dynamics of
object movements and it’s shadows. Vandrish et al. [14] used SIFT [7] for
sidescan sonar image registration. Even though these approaches achieve
considerable success in respective goals, were found to be most effective when
the rotation/translation between the frames of sonar images are comparatively
smaller. Block-matching was performed on segmented sonar images by Pham et al.
[15], using Self-Organizing Map for the registration and mosaicing task.

Recently CNNs have been applied for this problem, Zbontar et al[16] for stereo
matching in color images, and Valdenegro-Toro et al [8] for sonar images,
which is based on Zagoryuko et al [9], and is the state of the art for sonar
image patch matching at 0.91 AUC on the Marine Debris dataset. CNNs are
increasingly being used for sonar image processing [17]. The main reason
behind such a rise of CNN usage is that it can learn sonar-specific
information from the data directly. No complex manual feature design or
rigorous data pre-processing steps are needed, which makes the task less
complex and good prediction accuracy can be achieved.

##  III Matching as Binary Classification

We formulate the matching problem as learning a classifier. A classification
model is given two images, and it decides if the images match or not. This
decision can be modeled as a score in $y\in[0,1]$, or a binary output decision
$y=0,1$.

For this formulation, we use AUC, the area under the ROC curve (Receiver
Operating Characteristic) as the primary metric to assess performance, as we
are interested in how separable are the score distributions between matches
and non-matches.

TABLE I: Best hyper-parameter values for DenseNet Two-Channel (DTC).

Name | Value | Name | Value
---|---|---|---
Layers | 2-2-2 | Pooling | avg
Growth rate (gr) | 12 | Number of filter | 32
DenseNet dropout | 0.2 | Compression | 0.5
Bottleneck | False | Batch size | 128
Optimizer | Adadelta | Learning rate | 0.03

##  IV Matching Architectures

In this section we describe the neural network architectures we selected as
trunk for the meta-architectures like two-channel and siamese networks, which
are used for matching.

###  IV-A Hyper-Parameter Tuning

For each architecture, we tuned their hyper-parameters using a validation set,
in order to maximize accuracy. Each range of hyper-parameters was set
individually for each architecture, considering width, filter values at each
layer, drop probabilities, dense layer widths, etc. Overall, we performed 10
runs of different hyper-parameter combinations for each architecture. Details
of the hyper-parameter tuning are available at [18].

###  IV-B DenseNet Two-Channel Network

In DenseNet [19] each layer connects to every layer in a feed-forward fashion.
With the basic idea to enhance the feature propagation, each layer of DenseNet
blocks takes the feature-maps of the previous stages as input.

TABLE II: Best hyperparameter values for DenseNet Siamese (DS).

Name | Value | Name | Value
---|---|---|---
Number of filter | 16 | Layers | 2-2
Growth rate | 30 | DenseNet dropout | 0.4
Compression | 0.3 | Bottleneck | False
FC output | 512 | FC dropout | 0.7
Pooling | flatten | Batch size | 64
Optimizer | Adadelta | Learning rate | 0.07

In DenseNet two channel the the sonar patches are supplied as inputs in two
channels format, the network by itself divides each patch into one channel and
learn the features from the patches and then finally compare them using the
Sigmoid activation function at the end with FC layer of single output.

Hyper-parameters for this architecture are shown in Table I.

###  IV-C DenseNet Siamese Network

In this architecture the branches of the Siamese network are DenseNet.
Following the classic Siamese model each branch of the Siamese network shares
weights between them and gets trained simultaneously on two input patches and
then learns the features from the inputs. Through the shared neurons the
Siamese network is able to learn the similarity function and be able to
discriminate between the two input patches. The role of the DenseNet branches
are feature extraction, the decision making or prediction part is taken care
of by the Siamese network.

Figure 2: DenseNet Siamese architecture.

In Figure 2 the basic architecture is displayed for the DenseNet-Siamese
network. The two DenseNet branches are designed to share weights between them.
The extracted features are concatenated and connected through a FC layer,
followed by ReLU activation and where applicable Batch Normalization and
Dropout layers. The output is then connected to another FC layer with single
output, for binary prediction score of matching (1) or non-matching (0).
Sigmoid activation function and binary cross entropy loss function is used for
this final FC layer. As mentioned in Figure 2 the size of the output of the FC
layer and value of dropout probability etc. hyper-parameters are shown in
Table II.

###  IV-D Contrastive Loss

Using Contrastive loss [20] higher dimensional input data (e.g. a pair of
images) can be mapped in a much lower dimensional output manifold, where
similar pairs are placed closer to each other and the dissimilar pairs have
larger distances between them depending on their dissimilarity. Using this
loss function the distance between two input patches projected in the output
manifold can be predicted and if the distance is closer to 0 then the input
pairs are matching, otherwise its dissimilar (above threshold). The formulas
for this loss are shown in Equations 1 and 2.

$D_{W}(\vec{X}_{1},\vec{X}_{2})=\left\lVert
G_{W}(\vec{X}_{1})-G_{W}(\vec{X}_{2})\right\rVert$ (1)
$L=(1-Y)\frac{1}{2}(D_{W})^{2}+\frac{Y}{2}\\{\max(0,m-D_{W})\\}^{2}$ (2)

Here L is the loss term, the formula presented here is the most generalized
form of the loss function, suitable for batch training. $\vec{X_{1}}$,
$\vec{X_{2}}$ represents a pair of input image vectors. Y are the labels, 0
for similar pair and 1 for dissimilar pair. $D_{w}$ is the parameterized
distance function to be learned by the neural network. $m>0$ is the margin
that defines a radius around $G_{w}$. The dissimilar pairs only contribute to
the loss function if their distance is within the radius. We use $m=1$ for our
experiments. One of the ideas for evaluating this loss function is to use it
with a Siamese network, as the loss function takes a pair of images as input,
indicating their similarity, matching pairs having closer distances in the
learned embedding than non-matching ones, and the distance between pairs can
be used as a score with a threshold.

###  IV-E VGG Siamese Network

The VGG network [21] is a CNN which was conceptualized by K. Simonyan and A.
Zisserman from the University of Oxford (Visual Geometry Group). This network
performed very well in ImageNet challenge 2014. The architecture/s has very
small 3x3 Conv filters and depth varying from 16 to 19 weight layers. This
network generalizes very well with different kinds of data. VGG network has
been chosen as the branches of the Siamese (Figure 3) network It’s role is to
extract features, similar to the DenseNet-Siamese, the final decision making
and prediction is done by the Siamese network. The network is trained with
Contrastive loss. The output of this network is euclidean distance between the
two input sonar patches, projected into lower dimension using Contrastive
loss. The hyper-parameters of this network are shown in Table III.

Figure 3: VGG Siamese network with contrastive loss.

Since contrastive loss returns projected distance, close to zero means
similarity and higher values means dissimilarity. Although, in our original
data and matching formulation, labels close to one represents similarity
between patches. Hence the labels for train, validation and test data here are
all flipped:

$Y_{\text{new}}=1-Y_{\text{old}}$ (3)

Equation 3 is applied to all ground truth labels, meaning that for this
evaluation input label zero means similarity (match) between patches.

TABLE III: Best hyper-parameter values for VGG Siamese network with Contrastive loss (CL). Name | Value | Name | Value
---|---|---|---
Conv filters | 16 | Kernel size | 3
FC Layers | 1 | FC output | 2048
Batch normalization | False | Dropout | 0.6
Batch size | 256 | Optimizer | Nadam
Conv Initializer | random normal | FC Initializer | glorot normal
Learning rates | 0.0002 |  |

##  V Experimental Evaluation

###  V-A Dataset

We use the Marine Debris dataset, matching task, 111Available at
https://github.com/mvaldenegro/marine-debris-fls-datasets/releases/ to
evaluate our models. This dataset contains 47K labeled sonar image patch
pairs, captured using a ARIS Explorer 3000 Forward-Looking sonar, generated
from the original 2627 labeled object instances. We exclusively use the D
dataset, on which the training and testing sets were generated using different
sets of objects, with the purpose of testing a truly generic image matching
algorithm that is not object specific. The training set contains 39840 patch
pairs, while the test set contains 7440 patch pairs.

###  V-B Comparative Analysis of AUC

Our main results are presented in Table IV and Figure 4, where we present the
AUC and the ROC curves on the test set, correspondingly.

DenseNet two-channel has highest mean AUC (10 trials) of $0.955\pm 0.009$ with
max AUC of 0.966. With total parameters of only 51,430. DenseNet-Siamese has
highest mean AUC (10 trials) of $0.921\pm 0.016$, Max AUC 0.95 with total
parameters of 16,725,485. VGG-Siamese network with Contrastive loss have mean
AUC (10 trials) of $0.949\pm 0.005$ and highest AUC value in a single run as
0.956. With total number of parameters of 3,281,840. These AUC values are
considerably better than Valdenegro-Toro [8], with improvements from $0.910$
to $0.966$ (almost 5 AUC points).

It is notable that our best performing model is a two-channel network,
indicating that this meta-architecture is better suited for the matching
problem than a siamese one, and that there is a considerable reduction in the
number of parameters, from $1.8$M to $51$K, which hints at increased
generalization.

A comparison of predictions between all our three architectures is provided in
Figure 5.

TABLE IV: Comparative analysis on the AUC and total number of parameters in the best performing networks. Network | AUC | Best AUC | # of Params
---|---|---|---
Two-Channel DenseNet | $0.955\pm 0.009$ | 0.966 | 51K
Siamese DenseNet | $0.921\pm 0.016$ | 0.95 | 16.7M
Siamese VGG | $0.949\pm 0.005$ | 0.956 | 3.3M
Two-Channel CNN [8] | 0.910 | 0.910 | 1.8M
Siamese CNN [8] | 0.855 | 0.855 | 1.8M
Figure 4: Comparison of ROC curves for best hyper-parameter architecture
configurations and top AUC. Figure 5: Comparison of predictions across
multiple models, DenseNet Siamese (DS), DenseNet Two-Channel (DTC), and VGG
Siamese Contrastive Loss (CL). Note that Siamese VGG produce distances which
are not in the range $[0,1]$, while the other architectures give scores in the
$[0,1]$ range.

###  V-C Monte Carlo Dropout Analysis

Normally Dropout is only applied in the training phase, where it provides
regularization to avoid overfitting. In test time all the connections/nodes
remain present and dropout is not applied, though the weights are adjusted
according to the dropout ratio during training. So every time a prediction on
test data is obtained, they are deterministic. For Monte Carlo dropout the
dropout is also applied in the inference/test time, which introduces
randomness, as the connections are dropped randomly according to the dropout
probability. This prediction process is stochastic i.e the model could predict
different predictions for same test data. The main goal of Monte Carlo Dropout
[22] is to generate samples of the predictive posterior distribution of an
equivalent Bayesian Neural Wetwork, which quantifies epistemic uncertainty.

We would like to evaluate uncertainty for our best performing model, the
DenseNet two-channels (AUC 0.966). This model is trained with Dropout with
$p=0.2$. For this evaluation the MC-Dropout during inference time is enabled
explicitly. 20 forward passes for each of the test images are made and the
mean score and standard deviation is computed. The standard deviation is a
measure of uncertainty, with increasing value indicating more uncertainty.

Figures 6 and 7 present these results in terms of the most uncertain patch
pairs in Figure 6, and the most certain (least uncertain) images in Figure 7.
These results give insights on what the model thinks are its most difficult
samples (high uncertainty), and in particular, the most uncertain examples
(highest standard deviation) are the ones close to being out of distribution,
where the patches are positioned near the border of the FLS polar field of
view, which probably confuses the model.

The lowest uncertainty results in Figure 7 indicate the easiest patch pairs to
discriminate, either the same object in relatively similar poses, or radically
different objects or background in each pair. In both cases the model is quite
confident of these predictions.

Figure 8 shows a large selection of patch pairs and their uncertainty
estimates, showing that the model is not always confident, particularly for
predictions with scores in between zero and one, even for pairs that a human
would consider to be easy to match or reject.

Figure 6: MC-Dropout predictions of DTC with highest standard deviation over
20 forward passes. Ground truth label 1 indicated matching. It is clear that
the low signal-to-noise for sonar is affecting the predictions, and unwanted
reflections and occlusions are also challenging.

Figure 7: MC-Dropout predictions of DTC with lowest standard deviation over 20
forward passes. These results show that the network learned some of the
similarity functions with great confidence. For object-object non-matching
pairs usual std values are much higher than other categories. Figure 8: MC-
Dropout predictions of DTC with highest standard deviation over 20 forward
passes, presenting 20 sonar image patch pairs from the test dataset with index
1000 to 1019, and corresponding mean prediction and standard deviation.

###  V-D Ensemble

The performance of the DenseNet-Siamese(DS) is good for non-matching pair
predictions. DenseNet two-channel(DTC) is overall very good, but most
uncertain in object-object non matching pairs.

This observation led to the hypothesis that making an ensemble of these two
classifiers might improve overall predictive capability. For this experiment a
few of the previously trained models of DTC and DS are loaded, and their
predictions on the test data are averaged,i.e. same weights for DS and DTC
both. These evaluation results are displayed in Table V. The ROC AUC
calculated on the average prediction is found to be higher than the individual
scores each time.

TABLE V: After combining DS and DTC models, with AUC presented in first two
columns, the Ensemble is encoded and its prediction accuracy (AUC) gets much
improved, presented in the third column.

DS model AUC | DTC model AUC | Ensemble AUC
---|---|---
0.95 | 0.959 | 0.97
0.952 | 0.959 | 0.97
0.952 | 0.963 | 0.973
0.952 | 0.966 | 0.971
0.952 | 0.972 | 0.978

Ensemble accuracies (AUC) are consistently better than each model
individually. If the underlying models, which encode the ensemble, has low
AUC, the ensemble AUC is found to be much-improved. For example the first
result presented in Table V where the ensemble accuracy is much higher (0.97
AUC) than the underlying model predictions (0.95 and 0.959 AUCs). By encoding
an ensemble of the DenseNet-Siamese model with AUC 0.952 and the DenseNet two-
channel model with 0.972 AUC, the resulting ensemble AUC is found to be 0.978,
which is the highest AUC on test data obtained in any other experiment during
the scope of this work. This indicates that both DS and DTC models are
complementary and could be used together if higher AUC is required in an
application.

##  VI Conclusions and Future Work

In this work we present new neural network architectures for matching of sonar
image patches, including extensive hyper-parameter tuning, and explore their
performance in terms of area under the ROC curve, uncertainty as modeled by
MC-Dropout, and performance as multiple models are ensembles. The results in
this work are proven to be improvements over the state of the art on the same
dataset. Using DenseNet two-Channel network, average prediction accuracy
obtained is 0.955 area under ROC curve (AUC). VGG-Siamese (with Contrastive
loss function) and DenseNet-Siamese perform the prediction with an average AUC
of 0.949 and 0.921 respectively. All these results are an improvement over the
result of 0.910 AUC from Valdenegro-Toro [8]. Furthermore, by encoding an
ensemble of DenseNet two-channel and DenseNet-Siamese models with respective
highest AUC scores, prediction accuracy for the Ensemble obtained is 0.978
AUC, which is overall highest accuracy obtained in the Marine Debris Dataset
for the matching task.

We expect that our results motivate other researchers to build applications on
top of our matching networks.

## References

  * [1] N. Hurtós, Y. Petillot, J. Salvi _et al._ , “Fourier-based registrations for two-dimensional forward-looking sonar image mosaicing,” in _Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on_. Ieee, 2012, pp. 5298–5305.
  * [2] M. Brown and D. G. Lowe, “Automatic panoramic image stitching using invariant features,” _International journal of computer vision_ , vol. 74, no. 1, pp. 59–73, 2007.
  * [3] N. Molton, A. J. Davison, and I. D. Reid, “Locally planar patch features for real-time structure from motion.” in _Bmvc_. Citeseer, 2004, pp. 1–10.
  * [4] D. G. Lowe, “Object recognition from local scale-invariant features,” in _Computer vision, 1999. The proceedings of the seventh IEEE international conference on_ , vol. 2. Ieee, 1999, pp. 1150–1157.
  * [5] B. Yao, G. Bradski, and L. Fei-Fei, “A codebook-free and annotation-free approach for fine-grained image categorization,” in _Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on_. IEEE, 2012, pp. 3466–3473.
  * [6] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski, “A comparison and evaluation of multi-view stereo reconstruction algorithms,” in _null_. IEEE, 2006, pp. 519–528.
  * [7] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International journal of computer vision_ , vol. 60, no. 2, pp. 91–110, 2004\.
  * [8] M. Valdenegro-Toro, “Improving sonar image patch matching via deep learning,” in _Mobile Robots (ECMR), 2017 European Conference on_. IEEE, 2017, pp. 1–6.
  * [9] S. Zagoruyko and N. Komodakis, “Learning to compare image patches via convolutional neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4353–4361.
  * [10] S. Emberton, L. Chittka, and A. Cavallaro, “Underwater image and video dehazing with pure haze region segmentation,” _Computer Vision and Image Understanding_ , vol. 168, pp. 145–156, 2018.
  * [11] N. Hurtós, N. Palomeras, S. Nagappa, and J. Salvi, “Automatic detection of underwater chain links using a forward-looking sonar,” in _OCEANS-Bergen, 2013 MTS/IEEE_. IEEE, 2013, pp. 1–7.
  * [12] K. Kim, N. Neretti, and N. Intrator, “Mosaicing of acoustic camera images,” _IEE Proceedings-Radar, Sonar and Navigation_ , vol. 152, no. 4, pp. 263–270, 2005.
  * [13] S. Negahdaripour, M. Aykin, and S. Sinnarajah, “Dynamic scene analysis and mosaicing of benthic habitats by fs sonar imaging-issues and complexities,” in _Proc. OCEANS_ , vol. 2011, 2011, pp. 1–7.
  * [14] P. Vandrish, A. Vardy, D. Walker, and O. Dobre, “Side-scan sonar image registration for auv navigation,” in _Underwater Technology (UT), 2011 IEEE Symposium on and 2011 Workshop on Scientific Use of Submarine Cables and Related Technologies (SSC)_. IEEE, 2011, pp. 1–7.
  * [15] M. T. Pham and D. Guériot, “Guided block-matching for sonar image registration using unsupervised kohonen neural networks,” in _Oceans-San Diego, 2013_. IEEE, 2013, pp. 1–5.
  * [16] J. Zbontar and Y. LeCun, “Stereo matching by training a convolutional neural network to compare image patches,” _Journal of Machine Learning Research_ , vol. 17, no. 1-32, p. 2, 2016.
  * [17] M. Valdenegro-Toro, “Objectness scoring and detection proposals in forward-looking sonar images with convolutional neural networks,” in _IAPR Workshop on Artificial Neural Networks in Pattern Recognition_. Springer, 2016, pp. 209–219.
  * [18] A. Mallick, “Sonar patch matching via deep learning,” Master’s thesis, Bonn-Rhein-Sieg University of Applied Sciences, 2019.
  * [19] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks.” in _CVPR_ , vol. 1, no. 2, 2017, p. 3.
  * [20] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 2. IEEE, 2006, pp. 1735–1742.
  * [21] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [22] Y. Gal and Z. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,” _arXiv:1506.02142_ , 2015.