# Vision-Based Autonomous Navigation for Unmanned Surface Vessel in Extreme
Marine Conditions

Muhayyuddin Ahmed1, Ahsan Baidar Bakht1, Taimur Hassan1,2, Waseem Akram1,
Ahmed Humais1
Lakmal Seneviratne1, Shaoming He3, Defu Lin3, and Irfan Hussain1,∗ 1 Khalifa
University Center for Autonomous Robotic Systems (KUCARS), Khalifa University,
United Arab Emirates.2 Department of Electrical, Computer and Biomedical
Engineering, Abu Dhabi University, UAE.3 Beijing Institute of Technology,
China.∗ Corresponding Author, Email<EMAIL_ADDRESS>
###### Abstract

Visual perception is an important component for autonomous navigation of
unmanned surface vessels (USV), particularly for the tasks related to
autonomous inspection and tracking. These tasks involve vision-based
navigation techniques to identify the target for navigation. Reduced
visibility under extreme weather conditions in marine environments makes it
difficult for vision-based approaches to work properly. To overcome these
issues, this paper presents an autonomous vision-based navigation framework
for tracking target objects in extreme marine conditions. The proposed
framework consists of an integrated perception pipeline that uses a generative
adversarial network (GAN) to remove noise and highlight the object features
before passing them to the object detector (i.e., YOLOv5). The detected visual
features are then used by the USV to track the target. The proposed framework
has been thoroughly tested in simulation under extremely reduced visibility
due to sandstorms and fog. The results are compared with state-of-the-art de-
hazing methods across the benchmarked MBZIRC simulation dataset, on which the
proposed scheme has outperformed the existing methods across various metrics.

###### Index Terms:

Navigation, Marine Robotics, Visual Control

##  I Introduction

Marine robotics has experienced significant growth in recent years due to the
rapid growth of the maritime industry and the rising demand for ocean data
[1]. Marine robots, particularly, unmanned surface vessels (USVs), are used to
carry out a wide range of tasks, such as environmental monitoring and
autonomous surveillance for maritime security. USVs are also used for rescue
operations and for the irregular activity detection of vessels around ports.
To perform these tasks in the complex and dynamic marine environment, an
advanced perception system that accurately captures environmental details is
required. The robust and real time visual perception system improves the USV’s
autonomy and intelligence level [2].

Vision-based perception systems provide vital information about the
environment for USV intelligent control. Visual servoing is a popular USV
navigation technique used to implement vision-based intelligent control. This
method uses vision-based feedback to steer the USV toward the specified target
while avoiding collisions with other objects. Moreover, vision-based feedback
is crucial for activities involving autonomous inspection and monitoring. The
visual servoing-based approach often assumes that the camera can clearly view
the target object/vessel in the environment.

The United Arab Emirates (UAE), experiences extreme weather conditions, such
as sandstorms in the summer and extreme fog in the winter. During sandstorms
or extreme fog, the visibility in the air and water can be reduced
significantly, making it difficult for autonomous systems such as USVs to
perform navigation tasks effectively. Developing vision-based
navigation/tracking approaches for extreme weather conditions with reduced
visibility is still a challenging research problem. It is essential to develop
advanced perception techniques that allow USV to navigate under such
circumstances. The existing approaches for vision-based navigation, such as
[2] [3] [4], applied object detection algorithms for target detection.
However, in harsh marine conditions such as extreme fog, high winds, and
sandstorms along the coast, significantly affect the detector’s performance
because the camera input to the detector is insufficient. Improving the camera
input before passing it to the detector is necessary to enhance object
detection accuracy. Recent advancements in computer vision have developed deep
learning-based approaches for image enhancement and dehazing. These approaches
are helpful to enhance image quality in such a way that the detector can
easily perform the desired tasks. However, the use of these techniques in
marine robotics for navigation is very limited.

Figure 1: Overview of object detection and tracking in extreme conditions with
reduced visibility. Onboard camera images are passed to the detection module,
which performs image dehazing using GANs. The processed image passes to the
detector, which detects the target object in the image and uses this
information for USV navigation.

This study contributes in this direction and proposes a visual servoing-based
object tracking framework for extreme marine conditions such as sandstorms and
extreme fog.

Contributions: The core contribution of this paper is the vision-based
framework for target tracking in extreme conditions with highly reduced
visibility in a coastal environment. The framework incorporates; an integrated
object detecting system that consists of: a) generative adversarial networks
(GANs) model for image dehazing, that minimizes the noise from the image and
highlights the object features. b) YOLOv5-based object detection system that
takes an enhanced image as input to detect the target object, as depicted in
Fig. 1. In order to track the detected object using visual servoing, a PID-
based feedback control system is implemented that computes the control inputs
for USV navigation to track the target object. It is important to note that
the proposed framework is generalized; any image de-noising model and USV
control technique can be used within the framework.

The rest of the paper is structured as follows, Sec. II describes the related
work, Sec. III provides an overview of the solution and discusses the
framework. The proposed approach is discussed in Sec. IV, which explains the
dehazing method, target detection approach, and tracking method. The results
are discussed in Sec. V, and Sec. VI concludes the work.

##  II Related Work

This section describes the related work regarding object detection in marine
environments. USVs autonomously perceive their environment using the onboard
camera; the captured images are then used for detection. Object detection on
the sea surface is challenging due to various factors, such as, weather
conditions (fog and haze). In recent years, deep learning-based object
detection techniques have emerged and are being used in various applications.
Some of the commonly used detection techniques are the R-CNN series [5],
covering Faster RCNN [6], Cascade R-CNN [7], Mask R-CNN [8], SSD [9], and YOLO
[10]. Despite extensive research on object detection techniques, their
practical application in the marine environment is still very limited.

Object detection techniques are required for USV to successfully detect and
track the desired object. For instance, [2] used YOLOv4-based object detection
to detect the objects with the USV onboard camera. In this work, a cross stage
partial network (CSPNet) was deployed in a feature fusion network, which helps
to reduce the computation burden. Additionally, objects with different
resolutions were learned by tuning the weights during the training process.
The YOLOv4 model was then integrated with the feature pyramid transformer
(FPT) in order to obtain cross-space and cross-scale feature interactions. The
proposed approach was trained and tested on a sea surface buoy object
detection dataset collected in a marine environment.

A fusion-based detection and tracking scheme for USV is proposed in [3]. In
this work, high-resolution and deep semantic features are achieved by a cross-
stage partial network. In addition, anchor and convolution layers are improved
in the network model. The proposed model was tested on an image and video
dataset. The results demonstrated better results with the fusion-based model
compared with other detection and tracking algorithms. In [4], authors
proposed an object detection scheme using radar-photoelectric approach. In
this work, first-frame extraction is carried out, then region-of-interest
(ROI) is used to predict the target object. The developed algorithm was tested
in a marine environment with the help of a radar-guided target object.

Despite the fact that numerous studies have been conducted to perform object
detection and tracking in marine environments, the input data acquired from
the environment is often blurry and noisy due to harsh conditions like extreme
fog, winds, and sandstorms near the coast. In such conditions, detectors may
not be able to detect the target object accurately because the camera input is
inadequate. In such scenarios, improving the camera input before passing it to
the detector is necessary to enhance object detection accuracy. To this end,
deep learning-based image dehazing and enhancement techniques are helpful
because they provide image quality improvement so that the detector can easily
perform the desired tasks without mission failure.

Researchers, for example, have come up with a single image dehazing algorithm
that uses deep convolutional neural networks. A fully convolutional network is
used in the proposed method to turn a single hazy image into a clear image.
The model learns to estimate the transmission map and atmospheric light in a
data-driven manner [11]. In another approach, a multi-scale fusion network for
single image dehazing is proposed. This method uses a multi-scale fusion
strategy to estimate the transmission map and atmospheric light. The network
consists of an encoder-decoder architecture with multi-scale feature fusion
blocks to capture contextual information [12]. In [13], a feature pyramid
dehazing network is proposed for single-image dehazing. The network consists
of an encoder-decoder architecture with FPN layers, which can estimate the
transmission map and atmospheric light at multiple scales. The proposed method
also introduces a texture preservation loss to preserve the high-frequency
details of the dehazed image. Recently, the Transformer architecture has been
extended to the computer vision domain and has shown promising results in
image classification tasks. One of the early works that applied the
Transformer to image dehazing is proposed by [14]. The approach is a
transformer-based network called DehazeTransformer, which can estimate the
transmission map and atmospheric light simultaneously. The DehazeTransformer
network takes the hazy image as input and provides the dehazed image as
output. The self-attention mechanism in the Transformer allows the network to
capture long-range dependencies in the input image and to generate high-
quality dehazed images.

Figure 2: Framework for target tracking in extreme environment

##  III Solution Overview

The proposed framework consists of three main components: sensing system,
learning module, and  navigation module, as depicted in Fig. 2. The sensing
module consists of the USV’s onboard sensory system, which includes an RGBD
camera (used to perceive the environment), an IMU (used to get information
about the USV’s orientation), and a LiDAR (for collision avoidance).

The learning module consists of two main components: learning for object
detection and image dehazing. The set of vessels that USV needs to track is
used to prepare the dataset. The object detection model is trained over the
collected dataset, and the trained model is saved for online processing. The
vision pipeline for object detection is described in Sec. IV-A. The training
dataset for image dehazing is generated by navigating the USV in coastal
environments under clear visibility, and this data will be used as ground
truth. Then, another dataset is captured in low-visibility conditions caused
by a sandstorm or fog. These two recorded datasets will be used to train the
generative adversarial network. Sec. IV-B will explain the pipeline for
dehazing.

Navigation mode will receive the image from the onboard camera and pass it to
the image dehazing algorithm, which will reconstruct the image and pass it to
the object detection module (that was trained during the learning phase). Once
the object is detected, the heading correction module will compute the heading
error and pass it to the control algorithm. The control algorithm will compute
the appropriate controls to correct the USV orientation and start navigating
towards the target; the detail of the navigation mode is covered in Sec. IV-C.

##  IV Proposed Approach

###  IV-A Vision for target detection

Deep learning models are highly effective in object detection. In scientific
literature, numerous object detectors are proposed, such as [15] [16] [17]. In
this study, we used YOLOv5, which is a very efficient and commonly used model
for object detection. The model is deployed on a custom dataset consisting of
6 different types of vessels ( Fig. 3).

In general, the YOLOv5 network model is composed of three components: the
backbone, neck, and head. By combining gradient changes with the feature map,
the CSPDarknet [18], which is used for feature extraction in the backbone,
solves the issue of gradient information repetition during model training. It
results in fewer parameters while increasing detection accuracy. FPN [19] and
PANet [20] are fused to form the neck layer, which conducts multi-scale
prediction across many layers and is in charge of feature fusion. It improves
localization and semantic representation at various sizes. The convolutional
batch normalization layers are finally concatenated in the final component,
and subsequently, mask formation is performed by obtaining the pixel details.

The model uses a combined loss function in the prediction phase that combines
bounding box regression, classification, and confidence and can be expressed
as follows:

Figure 3: Dataset for vessel detection.

$L=L_{clc}+L_{box}+L_{conf}$ (1)

where $L_{clc}$ is the classification error, $L_{box}$ the bounding box
regression error, and $L_{conf}$ is the confidence error.

The errors $L_{clc}$, $L_{box}$ , and $L_{conf}$ are computed as follows:

$L_{clc}=\sum_{i=0}^{k^{2}}l_{i}\sum_{c=1}^{C}E(\hat{p}_{i}(c),p_{i}(c))$ (2)

where $l_{i}\in(1,0)$ for class objects as category c. The predicted and
actual probabilities are represented by $\hat{p}_{i}$ and $p_{i}$.

$L_{box}=1-IoU(A^{p},A^{g})+\frac{A^{c}-A^{p}-A^{g}+I}{A^{c}}$ (3)

The bounding box, bounded by the predicted box is represented by $A^{p}$ and
the actual box is represented by $A^{g}$. The desired and the actual areas are
represented by $A^{c}$ and $I$, respectively. The ratio of the intersection
between the estimated and actual areas within the image frame is denoted by
$IoU$.

$L_{conf}=\sum_{i=0}^{k^{2}}\sum_{j=0}^{M}I_{i,j}E(\hat{C}_{i},C_{i})-\lambda_{no}\;\chi$
(4)
$\chi=\text{obj}\sum_{i=0}^{k^{2}}\sum_{j=0}^{M}I_{i,j}(1.I_{i,j})E(\hat{C}_{i},C_{i})$
(5)

$k^{2}$ denotes the image partitioning into $k\times k$ grids, representing
$M$ candidate anchors. $I_{i,j}\in(1,0)$ distinguish positive or negative
samples. $\hat{Ci}$ and $Ci$ are used to represent the confidence levels of
the $i$th predicted bounding box and the actual bounding box, respectively.
$\lambda_{no}\text{obj}$ shows the probability where the object $X$ does not
exist. Additionally, E(.) is used to denote a binary cross-entropy loss, which
is defined as:

$E(\hat{X}_{i},X_{i})=\hat{X}_{i}ln(X_{i})+(1-\hat{X}_{i})ln(1-X_{i})$ (6)

This target object detection approach works with high detection accuracy in a
clear environment. However, in cases of reduced visibility due to sand or fog,
the performance of the detection algorithm is significantly reduced.

###  IV-B Vision pipeline for dehazing

The proposed method for image dehazing using GANs consists of a generator
network and a discriminator network. The generator network takes a hazy image
(i.e., a noisy image due to sandstorm or fog) as input and generates a clear
image as output. The discriminator network takes a pair of hazy and clear
images and distinguishes whether the clear image is a real or fake one, as
shown in Fig. 4. During the training process, the generator network is trained
to minimize the adversarial loss, which measures the difference between the
generated and ground truth images according to the discriminator’s feedback.

Figure 4: Architecture for image dehazing using generative adversarial
networks (GANs): Input hazy images are fed to a generator network composed of
an encoder-decoder architecture, which generates clear images. A discriminator
network, consisting of sequential layers of convolution and batch
normalization, is used to distinguish between real images and images generated
by the generator network. This GAN-based approach is used for effective
dehazing of images.

The generator network uses a pre-trained DenseNet-121 [21] model for feature
extraction and is modified to form an encoder-decoder structure with skip
connections. The encoder part uses a series of transition blocks, residual
blocks, and bottleneck block consisting of multiple convolutions and pooling
layers, to refine the feature maps from the encoder part. The input hazy image
is first downsampled through several convolutional layers in the encoder part
of the network, gradually reducing the spatial dimensions of the input. This
helps in capturing the essential features of the input image at multiple
scales. The encoded feature map is then fed into the decoder part of the
network, which upsamples the feature map through a series of transposed
convolutional layers, gradually increasing the spatial dimensions of the
output. This helps in reconstructing the dehazed image from the encoded
features.

Figure 5: Marine environments under different weather conditions used to
validate the proposed approach. Fig.-a represents the environment under clear
weather conditions. Fig-b represents sandstorm, the small images shows
different intensities of sandstorm. Fig-c represents the snapshot of extreme
fog; the small figures show different intensity of the fog. Figure 6: a1 to a3
represents the images captured by onboard camera from different distances
during the sandstorm. b1 to b3 show the output of the GANs, and c1 to c3 show
the output of the object detection module. Figure 7: a1 to a3 represent the
images captured by onboard camera from different distances during fog. b1 to
b3 show the output of the GANs, and c1 to c3 show the output of the object
detection module.

To train the generator model, we use multiple loss functions to calculate the
total loss. Specifically, we use a combination of L1, Mean Squared Error
(MSE), and Binary Cross Entropy (BCE) loss functions.

$L=L_{1}+L_{\mathrm{MSE}}+L_{\mathrm{BCE}}$ (7)

The $L_{1}$ loss denotes the absolute difference between the predicted output
and the ground truth, while the $L_{MSE}$ represents the mean squared
difference between the predicted output and the ground truth. $L_{BCE}$ is the
difference between the predicted and ground truth binary labels, where the
predicted labels are probabilities of the image being clear or hazy. The
$L_{1}$, $L_{MSE}$, and $L_{BCE}$ can be defined mathematically as follows:

$L_{1}(y,\hat{y})=\frac{1}{n}\sum_{i=1}^{n}|y_{i}-\hat{y_{i}}|$ (8)

$L_{MSE}(y,\hat{y})=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y_{i}})^{2}$ (9)

$L_{BCE}(y,\hat{y})=-\frac{1}{n}\sum_{i=1}^{n}\left[y_{i}\log\hat{y_{i}}+(1-y_{i})\log(1-\hat{y_{i}})\right]$
(10)

where, $y$ represents the ground truth label, $\hat{y}$ denotes the predicted
label, and $n$ is the number of images in the training dataset. The
combination of these loss functions ensures that the generator model learns to
produce images that are both visually similar to the ground truth images and
have similar pixel values.

The discriminator model consists of a series of convolutional layers followed
by batch normalization and LeakyReLU activation functions. The output of the
last convolutional layer is then flattened and passed through a fully
connected layer with a single output unit, this layer is followed by a sigmoid
activation function to produce the final output of the discriminator. This
output represents the probability, with values closer to 1 indicating a real
image, and values closer to 0 indicating a fake image.

To create a dataset for training and testing GANs for the purpose of dehazing,
a five-minute video was recorded in a simulation environment showing USV
movement towards a target vessel in extreme conditions (sandstorm and fog)
separately. The video was recorded using the USV onboard camera at a frame
rate of 30 frames per second (fps). From the recorded video, 2 frames per
second were extracted, resulting in a total of 600 images for each
environment. The dataset was split equally into two sets, a training set and a
testing set, with 300 images each. The training set of 300 images was used to
train the GANs, and the testing set of 300 images was used to evaluate the
performance of the trained model. A GAN was trained on a GPU machine for 100
epochs. The training system specifications include an Nvidia RTX A6000 GPU
with 48GB of memory and an Intel Xeon Gold CPU 2.1GHzx80 processor.

After training the GAN model for dehazing, a network is made that can map hazy
images to the ground truth that matches them. This generator network can be
considered a model that has learned the underlying distribution of clear
images given the hazy ones. The learned distribution captures the statistical
regularities of clear images, allowing the generator to produce convincing
clear images from hazy inputs. To apply this generator network for object
detection tasks, we used it to preprocess the hazy images before passing them
to the object detection model. This preprocessing step helps to remove the
haze from the images, thus improving the detection performance of the object
detection model.

###  IV-C Visual servoing control

Image-based visual servoing (IBVS) is a way to control the movement of
unmanned surface vessels by looking at images using on board camera. In IBVS,
the camera captures images of the surrounding environment, and these images
are processed to estimate the vessel’s position and orientation relative to a
desired trajectory or a target object. Based on this information, a control
signal is generated to adjust the vessel’s motion and bring it closer to the
desired trajectory or object. IBVS can be used for a variety of tasks,
including navigation, inspection, and object tracking. One of the main
advantages of IBVS is that it does not require accurate measurements of the
vessel’s position or orientation since these can be estimated directly from
the camera images. Additionally, IBVS can be used in environments where GPS or
other external sensors may be unavailable or unreliable, such as in shallow or
narrow waterways or in areas with poor satellite coverage.

To implement IBVS, a visual error function $e$ is defined that quantifies the
difference between the desired visual features $\mathcal{F}_{d}$ and the
observed visual features $\mathcal{F}_{o}$ in the image captured by the USV’s
onboard camera. Where, $\mathcal{F}_{d}$ represents that the detected target
vessel should be in the center of the image. $\mathcal{F}_{o}$ describes the
observed difference between the center of the image and the center of the
bounding box of the target vessel. The error function $e$ is represented as
follows:

$e(t)=\mathcal{F}_{d}-\mathcal{F}_{o}$ (11)

The $e(t)$ is computed in terms of pixel error. The control input $u$ for the
USV is computed using PID controller in the following way;

$u(t)=K_{\textrm{p}}e(t)+K_{\textrm{i}}\int_{0}^{t}e(t)\,dt+K_{\textrm{d}}\frac{de(t)}{dt}$
(12)

where $K_{\textrm{p}}$, $K_{\textrm{i}}$, and $K_{\textrm{d}}$ are all non-
negative, and represent the coefficients for the proportional, integral, and
derivative gain, respectively. The computed control input $u(t)$ applied to
the USV for correcting the heading and navigation towards the target. The
process of computing the error and applying the computed control input to the
USV will continue until the USV reaches the desired proximity of the target
vessel.

Figure 8: Training progress of the GAN-based architecture for image dehazing:
The plot shows the variation of generator and discriminator network loss
functions over 100 epochs of training.

##  V Results and Discussions

To validate the proposed approach, we used an open-source marine simulator
(https://github.com/osrf/mbzirc.git) developed by Open Robotics for the MBZIRC
Maritime Grand Challenge. It was developed in C++ and Python. It uses ROS2
Galactic for communication among modules. The hydrostatic and hydrodynamic
features are modeled using Ignition Gazebo. This simulator also provides the
simulation of advanced features such as the generation of sandstorms and fog.
Fig. 5 shows the different environments that are used to verify the proposed
approach, Fig. 5-a represents the clear environment with proper visibility,
Fig. 5-b shows the sandstorm, and Fig. 5-c shows the reduced visibility due to
fog. The implementation of the approach can be found in the GitHub
repository111Source code: https://github.com/Muhayyuddin/visual-servoing.git.

We navigate the USV towards the target vessel in a sandstorm or fog and apply
a simple YOLOv5-based object detector; however, it was not able to detect the
target vessel, particularly when the target was far from the USV. Under the
same weather conditions, the proposed approach detected the object correctly.
Fig 6 and Fig. 7 show the sample images of sandstorm and fog. Fig. 6-a1-3 and
Fig. 7-a1-3 show the sample images captured by the USV’s onboard camera from
different distances during sandstorm and fog respectively. Fig. 6-b1-3 and
Fig. 7-b1-3 depict the output of the proposed dehazing module (GANs). The
output of the GANs is forwarded to the YOLOv5 object detector, the detection
results are shown in Fig. 6-c1-3 and Fig. 7-c1-3.

Generator and discriminator loss curves over 100 epochs are shown in Fig. 8.
It shows that the generator loss is decreasing over the 100 epochs, indicating
that the model is learning to generate more realistic images as the training
progresses. On the other hand, the discriminator loss is increasing,
suggesting that the model is getting better at generating synthetic images,
which makes it difficult to distinguish between real and generated images.

The results presented in Table I indicate the performance of different
backbone models within a proposed dehazing framework. It is worth noting that
the choice of evaluation metrics is crucial when comparing the performance of
different models. In our case, we use PSNR, which is used to evaluate the
amount of signal distortion present in a dehazed image compared to a ground
truth image; SSIM, which is used to assess the dehazing performance by
measuring the contrast, luminance, and structure of both the ground truth and
dehazed images; and MSE, which calculates the error between the dehazed image
and the ground truth image. It can be seen that DenseNet-121 performs best in
terms of PSNR, SSIM, and MSE with values of 24.99, 0.95, and 205.65,
respectively. The second-best performance in terms of PSNR, SSIM, and MSE is
achieved by ViT, with values of 24.55, 0.93, and 228.19, respectively. These
results indicate that ViT and ResNet-50 are also viable options for dehazing
tasks, but they are not as effective as DenseNet-121.

The given Table II shows a performance comparison of the proposed dehazing
framework with other state-of-the art methods, including GW-FS, FDA,
DehazeFormer, and FFA-Net. The comparative analysis is based on three commonly
used image quality metrics: PSNR, SSIM, and MSE. The proposed dehazing block
with DenseNet-121 as the backbone model outperforms all the compared state-of-
the-art methods in terms of all three metrics. The proposed method achieves
the highest PSNR of 24.99 dB, followed by DehazeFormer with a PSNR of 24.34
dB. In terms of SSIM, the proposed method again achieves the highest score of
0.95, followed by DehazeFormer with a score of 0.93. For MSE, the proposed
method achieves the lowest value of 205.65, which is considerably lower than
the other methods.

TABLE I: Effect of varying different backbone models within the proposed dehazing block. Bold indicates the best performance while the second-best performance is underlined. BackBone | PSNR | SSIM | MSE
---|---|---|---
DenseNet-121 [21] | 24.99 | 0.95 | 205.65
ResNet-50 [22] | 24.08 | 0.92 | 253.84
ViT [23] | 24.55 | 0.93 | 228.19
MobileNet-v2 [24] | 23.64 | 0.89 | 281.06
VGG-16 [25] | 23.92 | 0.86 | 304.82
TABLE II: Performance evaluation of the proposed dehazing block with state-of-the-art dehazing schemes in terms of PSNR, SSIM, and MSE. Bold indicates the best performance, while the second-best performance is underlined. Framework | PSNR | SSIM | MSE
---|---|---|---
Proposed | 24.99 | 0.95 | 205.65
GW-FS [26] | 23.90 | 0.90 | 264.73
FDA [27] | 23.08 | 0.84 | 319.51
DehazeFormer [14] | 24.34 | 0.93 | 238.92
FFA-Net [28] | 23.41 | 0.87 | 296.08

##  VI Conclusion and Future Work

This paper presents a vision-based autonomous navigation framework for USV in
extreme marine conditions such as sandstorm and fog near the coast. The
proposed framework applies an integrated perception pipeline that uses a
generative adversarial network to remove the noise and highlight the object
features in the input image, captured by the USV’s onboard camera. The
enhanced image is then passed to the YOLOv5 model to accurately detect the
target object (vessel). The detection output is used by the visual servoing-
based controller for tracking the target. The proposed scheme is compared with
the state-of-the-art methods for dehazing on the benchmarked MBZIRC simulation
dataset, where it significantly outperformed them across various metrics. In
the future, the proposed approach will be tested in a real-world scenario, in
which we will also enhance the USV control to incorporate extreme conditions
that will include strong ocean currents and waves.

## Acknowledgement

This work is supported by the Khalifa University of Science and Technology
under Award No. MBZIRC-8434000194, CIRA-2021-085, FSU-2021-019,
RC1-2018-KUCARS.

## References

  * [1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robotics and autonomous systems, vol. 57, no. 5, pp. 469–483, 2009.
  * [2] X. Sun, T. Liu, X. Yu, and B. Pang, “Unmanned surface vessel visual object detection under all-weather conditions with optimized feature fusion network in yolov4,” Journal of Intelligent & Robotic Systems, vol. 103, pp. 1–16, 2021.
  * [3] Z. Zhou, X. Hu, Z. Li, Z. Jing, and C. Qu, “A fusion algorithm of object detection and tracking for unmanned surface vehicles,” Frontiers in Neurorobotics, vol. 16, 2022.
  * [4] Q. Yu, B. Wang, and Y. Su, “Object detection-tracking algorithm for unmanned surface vehicles based on a radar-photoelectric system,” IEEE Access, vol. 9, pp. 57529–57541, 2021.
  * [5] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587, 2014.
  * [6] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in neural information processing systems, vol. 28, 2015.
  * [7] Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6154–6162, 2018.
  * [8] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision, pp. 2961–2969, 2017.
  * [9] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pp. 21–37, Springer, 2016.
  * [10] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788, 2016.
  * [11] D. Huang, K. Chen, J. Lu, and W. Wang, “Single image dehazing based on deep neural network,” in 2017 international conference on computer network, electronic and automation (ICCNEA), pp. 294–299, IEEE, 2017.
  * [12] C. Zhang and C. Wu, “Multi-scale attentive feature fusion network for single image dehazing,” in 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1–7, 2022.
  * [13] D. Zhao, L. Xu, L. Ma, J. Li, and Y. Yan, “Pyramid global context network for image dehazing,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 8, pp. 3037–3050, 2020.
  * [14] Y. Song, Z. He, H. Qian, and X. Du, “Vision transformers for single image dehazing,” arXiv preprint arXiv:2204.03883, 2022.
  * [15] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10781–10790, 2020.
  * [16] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” arXiv preprint arXiv:2010.04159, 2020.
  * [17] M. A. Rosales, M. G. B. Palconit, V. J. D. Almero, R. S. Concepcion, J.-A. V. Magsumbol, E. Sybingco, A. A. Bandala, and E. P. Dadios, “Faster r-cnn based fish detector for smart aquaculture system,” in 2021 IEEE 13th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM), pp. 1–6, IEEE, 2021.
  * [18] C.-Y. Wang, H.-Y. M. Liao, Y.-H. Wu, P.-Y. Chen, J.-W. Hsieh, and I.-H. Yeh, “Cspnet: A new backbone that can enhance learning capability of cnn,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 390–391, 2020.
  * [19] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117–2125, 2017\.
  * [20] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path aggregation network for instance segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8759–8768, 2018.
  * [21] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708, 2017.
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [23] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  * [24] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017.
  * [25] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  * [26] T. Hassan, S. Akçay, M. Bennamoun, S. Khan, and N. Werghi, “Unsupervised anomaly instance segmentation for baggage threat recognition,” Journal of Ambient Intelligence and Humanized Computing, pp. 1–12, 2021.
  * [27] Y. Yang and S. Soatto, “Fda: Fourier domain adaptation for semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4085–4095, 2020.
  * [28] X. Qin, Z. Wang, Y. Bai, X. Xie, and H. Jia, “Ffa-net: Feature fusion attention network for single image dehazing,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 11908–11915, 2020.


# Knowledge Distillation for Singing Voice Detection

###### Abstract

Singing Voice Detection (SVD) has been an active area of research in music
information retrieval (MIR). Currently, two deep neural network-based methods,
one based on CNN and the other on RNN, exist in literature that learn
optimized features for the voice detection (VD) task and achieve state-of-the-
art performance on common datasets. Both these models have a huge number of
parameters (1.4M for CNN and 65.7K for RNN) and hence not suitable for
deployment on devices like smartphones or embedded sensors with limited
capacity in terms of memory and computation power. The most popular method to
address this issue is known as knowledge distillation in deep learning
literature (in addition to model compression) where a large pre-trained
network known as the teacher is used to train a smaller student network. Given
the wide applications of SVD in music information retrieval, to the best of
our knowledge, model compression for practical deployment has not yet been
explored. In this paper, efforts have been made to investigate this issue
using both conventional as well as ensemble knowledge distillation techniques.

Index Terms: Singing Voice Detection, Accelerating CNNs/RNNs, Knowledge
Distillation, Soft Labels, Ensemble Knowledge Distillation

##  1 Introduction & Related Work

Singing voice detection is a binary classification problem in music
information retrieval, where the task is to identify singing voice in an audio
segment of duration 100 to 200 ms. Efficient voice detection systems can also
aid in other MIR tasks such as melody extraction [1, 2] or artist recognition
[3]. Early voice detection approaches [4, 5, 6] usually relied on complex
hand-engineered audio features, which have now gone out of favour with the
advent of deep neural networks. Some examples of such features include MFCCs
(Mel-Frequency Cepstral Coefficients), PLPs (Perceptual Linear Predictive
Coefficients) and LFPCs (Log Frequency Power Coefficients). These were used as
inputs to classification systems like Support Vector Machines, Hidden Markov
Models, Random Forests or Artificial Neural Networks. Two of the most popular
and recent DNN based approaches [7] and [8] showed that neural networks have
the capacity to learn complex features relevant to a particular task such as
SVD, from very low-level audio representations such as STFT or
Melspectrograms. [9] performed evaluation of these two approaches under a
common benchmark protocol so that pre-processing procedures do not give any of
the two models an undue advantage in terms of performance.

While these models have considerably improved voice detection performance as
well as eliminated reliance on hand-crafted features, little attention has
been paid to the optimal structure of these models - whether we actually need
the huge memory and computation power needed to train these models. Also, from
a deployment point of view, these networks are all non-optimal owing to their
sufficiently large inference times. In this paper, we try to address this
issue using principles of knowledge distillation [10]. Our experiments show
that the optimal CNN-based model is actually more than 250 times smaller than
the model used by [8], and the optimal RNN-based model is at least 2.5 times
smaller than the one used by [7]. Moreover, we show that, with knowledge
distillation on smaller student models, it is possible to obtain accuracies
substantially higher than the current state-of-the-art.

##  2 Proposed Methods

###  2.1 Basics of Knowledge Distillation

The basic idea behind knowledge distillation is to have the student network
trained with not only the information provided by true labels (also called
hard targets) but also by using soft targets produced by the teacher network
(also referred to as cumbersome model) as a regularizer. This helps the
student network learn to mimic the teacher’s behaviour. In accordance with
[10], a scaling hyperparameter referred to as temperature is used to regulate
the softness of the targets from the teacher network.

The generalized formula for softmax is given by:

$p_{i}=\frac{\exp\left(s_{i}/\tau\right)}{\sum_{j}\exp\left(s_{j}/\tau\right)}$
(1)

where $s_{i}$ and $p_{i}$ are the logit produced for the $i^{th}$ class and
the corresponding class probability respectively and $\tau$ is the softness
regulating temperature.

A KL-divergence loss is taken between the softmax probabilities of the teacher
and the student, represented as $q$ and $p$ respectively, raised to
temperature $\tau$ (between 2-20 in our experiments) and is given by:

${L}_{KD}=\tau^{2}KLD\left(q,p\right)$ (2)

The combined loss is given by:.

${L}_{\text{Total}}=(1-\lambda){L}_{CE}+\lambda{L}_{KD}$ (3)

where ${L}_{CE}$ is the cross-entropy loss between the student’s softmax
probabilities (at $\tau=1$) and the correct labels (hard targets) and
$\lambda$ is a second hyperparameter controlling the trade-off between the 2
losses. Generally, better results are obtained by keeping $\lambda$ close to
1, i.e., weight on ${L}_{KD}$ higher.

###  2.2 Knowledge Distillation with Schluter CNN

Before starting the discussion specific to knowledge distillation, we first
describe the model architecture and input features in Schluter CNN (Teacher
model) [8]:

####  2.2.1 Input Features

[8] uses melspectrograms (80-D) of 115 consecutive frames (1.6 seconds) of an
audio signal as input feature. The label of the central frame is considered as
the label of a particular sample. The resulting input dimension is 80x115. The
Mel bank is normalized to have zero mean and unit variance over the training
data.

####  2.2.2 Model Architecture

The Schluter CNN employs three types of feedforward neural network layers: 2D
convolutional layers with 3x3 kernels and Leaky ReLU activations [11],
maxpooling layers with 3x3 kernels and dense layers. To describe the
architecture, we use the following shorthand notations: ’ConvX’ denotes a conv
layer with X channels and Leaky ReLU activation, ’Max’ stands for maxpool, and
’DenseX’ denotes a dense layer with X neurons. The architecture can then be
described as Conv64-Conv32-Max-Conv128-Conv64-Max-Dense256-Dense64-Dense2.
Additionally, there are two dropout layers with dropout probability 0.2 in
between the final 3 dense layers. Note that the original model was trained
with a sigmoid output activation function and binary cross entropy loss where
appropriate thresholds were used on the output probability (between 0 and 1)
to infer the presence or absence of a voice label. However, since the main
objective of this paper is knowledge distillation, we needed a softmax
activation on the output. Hence, in our modified model, the output dimension
is 2 with a softmax activation and the model is trained with categorical
cross-entropy loss. In Table 2, we compare the performance of our base model
with the benchmarks by [9] and show that the two sets of accuracies are almost
identical. This justifies the above mentioned modification to Schluter CNN.

####  2.2.3 Building Student Networks from Base Model

The base model (Schluter CNN) has a total of 14,08,290 ($\sim$ 1.4M)
parameters. To build student models, we focus on reducing the number of
channels per layer (except maxpool of course). The lowest number of channels
in any particular layer in the architecture mentioned above is 32. We define a
variable ”filter scale” (abbreviated as FS) that can have any of the values in
{2, 4, 8, 16, 32}. As the name suggests, we divide the number of channels in
each layer by FS to reduce the overall number of model parameters. Higher the
value of FS, lower the number of parameters. We denote each resulting student
network as FSX. Student networks trained with distillation are denoted as KD-
FSX. These notations are later used in Section 3.3. As shown in Table 1, the
highest reduction achieved with student networks in terms of parameters is
nearly 1000x with FS32. The process of knowledge distillation from teacher CNN
model to student CNN model is shown in Fig. 1. As shown in figure, teacher
model is the pre-trained Schluter CNN. During distillation, teacher model
parameters are frozen and the knowledge of the teacher model is transferred to
a small student CNN model with reduced parameters.

Table 1: Number of parameters in Student Models Model | Parameters
---|---
FS2 | 3,52,402
FS4 | 88,266
FS8 | 22,150
FS16 | 5,580
FS32 | 1417
Figure 1: Distilling Schluter CNN to student CNN Figure 2: Distilling Leglaive
RNN to student RNN Figure 3: Ensemble distillation of Schluter CNN and
Leglaive RNN to student CNN and RNN model

###  2.3 Knowledge Distillation with Leglaive RNN

Similar to the CNN case, we first describe the model architecture and feature
inputs in Leglaive RNN (Teacher model) [7]:

####  2.3.1 Input Features

Leglaive RNN [7] first applies a double stage harmonic-percussion source
separation (HPSS) [12] on the audio signal as pre-processing to extract
signals relevant to the singing voice. The main idea behind HPSS is to
decompose the spectrogram of the input signal into one spectrogram smooth in
time direction (associated to harmonic components), and another spectrogram
smooth in frequency direction (associated to percussive components). Features
are then extracted from a filter bank on a Mel scale and the resulting
melspectrograms of the harmonic and percussive components are concatenated as
input to the Bidirectional LSTM (BiLSTM) network. This gives 80-dimensional
feature vectors for 218 frames (3.5 seconds) per sample. Additionally and
similar to Schluter CNN, the mel bands are normalized to have zero mean and
unit standard deviation over the train set. However, unlike Schluter CNN,
Leglaive RNN infers the label of each of the 218 time frames.

####  2.3.2 Model Architecture

The model is made up of 3 layers of stacked BiLSTM units of sizes 30, 20, and
40. Finally, this is followed by a shared dense layer (also known as time
distributed dense) to yield predictions for the 218 output time frames (3.5
seconds). Similar to 2.2.2, the original model was trained with a sigmoid
output (output dimension = 1) for each of the 218 frames and binary cross
entropy loss. Keeping in mind our main objective of experimentation with
distilled student models, we re-implement Leglaive RNN with 2 output
dimensions with softmax activation, for each frame. Comparison with [9]’s
implementation is shown in Table 5. Almost identical numbers justify these
modifications to the teacher network.

####  2.3.3 Building Student Network from Base Model

Leglaive RNN has a total of 65,682 ($\sim$ 65.7K) parameters. Reducing only
the number of LSTM units would not have too much of an effect in terms of
parameter count reduction. So, we completely remove the last 2 BiLSTM layers
and that gives close to 2.5x reduction in size for the student model with a
total of 26,762 ($\sim$ 26.8K) parameters. For future reference, the teacher
is abbreviated as LRNN (Leglaive RNN) and the student as SRNN (small RNN).
Student network trained with distillation is denoted as KD-SRNN. Knowledge
distillation from teacher RNN to student RNN model is shown in Fig. 2. As
shown in figure, teacher model is the pre-trained Leglaive RNN. During
distillation, teacher model parameters are frozen and the knowledge of the
teacher model is transferred to a small student RNN model with reduced
parameters.

###  2.4 Ensemble Knowledge Distillation

In this section, we propose to train both CNN and RNN based student networks
using the two teachers Schluter CNN and Leglaive RNN. For this, we use the
input features representation used for SVD in [8] without any data
augmentation. However, for this feature representation, we do not have any
prior RNN based models in literature. To get around this problem, we train the
Leglaive RNN architecture on this data, the only difference being, the RNN now
predicts labels of the central time frame as in the case of Schluter CNN. The
parameter count remains unchanged from 2.3.3. We first benchmark the
performance of LRNN and SRNN on the new data in Table 7. The input dimension
of Schluter CNN is same as before (80x115); that of Leglaive RNN is 115x80
(time dimension comes first in RNNs). Similar to the proposition in [10], we
combine the predictions of the teachers in the ensemble (here 2) by taking the
arithmetic or geometric mean of their predictions as soft targets. The CNN
based students are denoted as ENKD-FSX and the RNN based one as ENKD-SRNN.
These notations are reused in Section 3.6. As shown in Table 8, we achieve
state-of-the-art performance on the Jamendo dataset with this formulation. The
ensemble knowledge from two teacher models to a student model is shown in Fig.
3. As shown in figure, during knowledge distillation, the teacher (Schluter
CNN and Leglaive RNN) model parameters are frozen and the knowledge of both
teacher models are transferred to student CNN and RNN models.

##  3 Experiments

###  3.1 Dataset and Protocol

For our experiments, we use the Jamendo dataset, a publicly available dataset
with singing voice annotations, that was also used by [7] and [8]. It contains
93 copyright-free songs, collected and annotated by [6]. For comparison to
existing results [9], we follow the official split of 61 files for training
and 16 files each for validation and testing. We use the validation set for
hyperparameter tuning and report test results corresponding to the highest
validation accuracy. In accordance with [9], for test set, we report accuracy,
precision, recall, F-measure, False Positive Rate (FPR) and False Negative
Rate (FNR). Our source code is built on top of a public repository by [9]
available here - https://github.com/kyungyunlee/ismir2018-revisiting-svd.

###  3.2 Reproducing Results of Schluter CNN

Here we compare the results of our implementation (output dimension 2 and
trained by categorical cross-entropy loss) with the implementation by [9].
Note that, similar to [9], we also do not apply data augmentation for a fair
comparison with [7]. As shown in Table 2, our numbers only differ by small
magnitudes.

Table 2: Comparison of our Schluter CNN implementation with [9]’s implementation Model | Acc. | Prec. | Recall | F-Measure | FPR | FNR
---|---|---|---|---|---|---
[9] | 86.8 | 83.7 | 89.1 | 86.3 | 15.1 | 10.9
Ours | 85.4 | 81.3 | 89.3 | 85.1 | 17.9 | 10.7
Table 3: Performance of Student Derivatives from Schluter CNN Model | Acc. | Prec. | Recall | F-Measure | FPR | FNR
---|---|---|---|---|---|---
FS2 | 87.0 | 81.8 | 92.8 | 86.9 | 18.0 | 7.2
FS4 | 85.2 | 80.4 | 90.1 | 85.0 | 19.1 | 9.9
FS8 | 87.0 | 84.0 | 89.1 | 86.5 | 14.8 | 10.9
FS16 | 86.5 | 84.5 | 86.6 | 85.7 | 13.5 | 13.4
FS32 | 83.7 | 85.8 | 77.9 | 81.7 | 11.2 | 22.1
Table 4: Performance of Student Models with Knowledge Distillation Model | Acc. | Prec. | Recall | F-Measure | FPR | FNR
---|---|---|---|---|---|---
KD-FS2 | 87.0 | 83.7 | 89.4 | 86.4 | 15.2 | 10.6
KD-FS4 | 87.3 | 84.5 | 88.3 | 86.6 | 13.6 | 11.7
KD-FS8 | 86.6 | 83.1 | 89.5 | 86.2 | 15.9 | 10.5
KD-FS16 | 85.0 | 80.0 | 90.4 | 84.9 | 19.7 | 9.6
KD-FS32 | 81.5 | 79.4 | 81.3 | 80.3 | 18.4 | 18.7
Table 5: Comparison of our Leglaive RNN implementation with [9] Model | Acc. | Prec. | Recall | F-Measure | FPR | FNR
---|---|---|---|---|---|---
[9] | 87.5 | 86.1 | 87.2 | 86.6 | 12.2 | 12.8
LRNN (Ours) | 88.2 | 85.7 | 89.7 | 87.8 | 13.2 | 10.3
Table 6: Performance Comparison of SRNN and KD-SRNN Model | Acc. | Prec. | Recall | F-Measure | FPR | FNR
---|---|---|---|---|---|---
SRNN | 87.6 | 84.5 | 90.6 | 87.5 | 14.6 | 9.4
KD-SRNN | 88.9 | 85.7 | 91.5 | 88.5 | 13.4 | 8.5
Table 7: Performance of LRNN and SRNN on data used by [8] Model | Acc. | Precision | Recall | F-Measure | FPR | FNR
---|---|---|---|---|---|---
LRNN | 82.3 | 78.2 | 85.8 | 81.8 | 20.9 | 14.2
SRNN | 76.1 | 70.5 | 83.8 | 76.6 | 30.6 | 16.2
Table 8: Performance Comparison of ENKD Variants | Acc. | Prec. | Recall | F-Measure | FPR | FNR
---|---|---|---|---|---|---
Model |  | AM
---
| GM
---
| AM
---
| GM
---
| AM
---
| GM
---
| AM
---
| GM
---
| AM
---
| GM
---
| AM
---
| GM
---
ENKD-FS2 | 87.9 | 87.7 | 86.3 | 86.5 | 87.9 | 87.1 | 87.1 | 86.8 | 12.1 | 11.8 | 12.1 | 12.9
ENKD-FS4 | 86.8 | 88.4 | 84.4 | 86.8 | 88.0 | 88.4 | 86.1 | 87.6 | 14.2 | 11.7 | 12.0 | 11.6
ENKD-FS8 | 87.7 | 87.1 | 86.2 | 81.9 | 87.6 | 92.8 | 86.9 | 87.0 | 12.2 | 17.8 | 12.4 | 7.2
ENKD-FS16 | 86.4 | 86.5 | 84.2 | 83.4 | 87.1 | 88.1 | 85.6 | 85.8 | 14.3 | 15.0 | 12.9 | 11.9
ENKD-FS32 | 84.3 | 84.2 | 82.3 | 82.8 | 84.4 | 83.3 | 83.3 | 83.1 | 15.8 | 15.0 | 15.6 | 16.7
ENKD-SRNN | 84.4 | 83.1 | 81.1 | 81.5 | 86.5 | 82.5 | 83.7 | 82.0 | 17.5 | 16.3 | 13.5 | 17.5

###  3.3 Knowledge Distillation with Schluter CNN

####  3.3.1 Performance of Student Networks without Distillation

Here we show the performance of the student networks from Table 1. These are
trained similar to the teacher network. As shown in Table 3, with our smaller
student networks, we get better results across all performance measures, with
respect to both ours and the original implementation in Table 2. Interestingly
enough, FS32, having 1000x lesser parameters than the teacher, succeeds in
improving on 2 performance measures, Precision and FPR, over Schluter CNN.
This clearly indicates, that smaller models can be equally effective in SVD
and require further attention.

####  3.3.2 Performance of Distilled Student Networks

Here we present the performance of the student networks from 3.3.1, trained
with distillation by the teacher network. As shown in Table 4, performance
falls on almost every measure with the student models. This is expected since,
as per Table 3, student networks already have a lesser tendency to overfit,
i.e., they actually perform better than Schluter CNN. So, additional
supervision from the teacher actually ends up hurting the student network.
However, as we show in Section 3.6, performance of all FSX models improve with
additional supervision from an LSTM-based [13] teacher network.

###  3.4 Reproducing Results of Leglaive RNN

Similar to Section 3.2, we compare our implementation of Leglaive RNN with
that of [9]. Here the output dimension is 2 for the time distributed dense
layer and the model is trained by categorical cross-entropy loss. As shown in
Table 5, our numbers only differ by small magnitudes.

###  3.5 Knowledge Distillation with Leglaive RNN

####  3.5.1 Performance of Student Network with and without Distillation

As shown in Table 6, KD-SRNN comprehensively outperforms SRNN on all
performance measures, showing the benefits of knowledge distillation in this
context. Although SRNN performs slightly worse than LRNN on some of the
measures, KD-SRNN surpasses LRNN (our implementation) on all measures and is
inferior to the original implementation [9], only on Precision and FPR.

###  3.6 Ensemble Knowledge Distillation

####  3.6.1 Benchmarking LSTM Performance

First we show the performance of LRNN and SRNN on the data used by Schluter
CNN in Table 7. We see that its performance is considerably lower than
Schluter CNN or any of the FSX derivatives. Despite this, including it as a
teacher in the ensemble considerably improves the performance of all ENKD-FSX
and ENKD-SRNN models, as shown in Table 8.

####  3.6.2 ENKD Models

Here we present the performance of all CNN and RNN based student networks
trained by Ensemble Knowledge Distillation shown in Table 8. AM (Arithmetic
Mean) and GM (Geometric Mean) denote method by which teacher predictions were
combined. With Ensemble Knowledge Distillation (ENKD), we get substantial
improvements over all CNN and RNN models trained by simple knowledge
distillation (KD). On accuaracy, ENKD improves KD-FSX variants by the
following amounts - 0.9% (FS2), 1.1% (FS4), 1.1% (FS8), 1.5% (FS16), 2.8%
(FS32). ENKD also gives a new state-of-the-art model on the features used by
[8], ENKD-FS4 (mainly based on accuracy) that improves current best reported
results (Table 2) by 1.6%. This technique also proves quite beneficial for
SRNN, improving its accuracy by 8.3% to 84.4% (higher than even LRNN).
However, this accuracy is still lower than that of ENKD-FS4 by a fair amount,
proving that CNN is still the better model for this kind of data.

In summary, smaller student networks built from Schluter CNN actually show
lesser overfitting tendencies and hence perform better than the base model in
several cases. That is why knowledge distillation experiments with the base
model as a teacher, fail to get any substantial improvement over the student
networks’ performance. Knowledge Distillation experiments with Leglaive RNN
produce new state-of-the-art results on the features used by [7], with a
smaller student model, SRNN. With Ensemble Knowledge Distillation (ENKD), we
get substantial improvements over all CNN and RNN models trained by simple
knowledge distillation (KD).

##  4 Conclusion and Future Work

In this paper, we have shown that application of knowledge distillation
techniques to compress the singing voice detection models can be a new
direction of research in the field of Music Information Retrieval. Our
experiments show that smaller models trained with distillation can achieve
comparable and in some cases, even higher accuracies than current state-of-
the-art models. With respect to knowledge distillation, another interesting
experiment can be training student models on a subset of the training data and
comparing their performance with student/teacher models that learn from the
entire data. We also plan to explore multi-step knowledge distillation
techniques in the future, like the Teacher Assistant model [14], where an
intermediate-sized network can supposedly help bridge the knowledge gap
between student and teacher better and hence aid in more efficient
distillation.

##  5 Acknowledgements

The second author would like to thank Google for supporting his PhD through
Google PhD Fellowship program.

## References

  * [1] C.-L. Hsu, L.-Y. Chen, J.-S. R. Jang, and H.-J. Li, “Singing pitch extraction from monaural polyphonic songs by contextual audio modeling and singing harmonic enhancement,” in _ISMIR_ , 2009.
  * [2] M. G. Reddy and K. S. Rao, “Predominant melody extraction from vocal polyphonic music signal by combined spectro-temporal method,” in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2016, pp. 455–459.
  * [3] A. Berenzweig, D. P. Ellis, and S. Lawrence, “Using voice segments to improve artist classification of music,” _Academicommons_ , 2002.
  * [4] B. Lehner, R. Sonnleitner, and G. Widmer, “Towards light-weight, real-time-capable singing voice detection.” in _ISMIR_ , 2013, pp. 53–58.
  * [5] B. Lehner, G. Widmer, and R. Sonnleitner, “On the reduction of false positives in singing voice detection,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2014, pp. 7480–7484.
  * [6] M. Ramona, G. Richard, and B. David, “Vocal detection in music with support vector machines,” in _2008 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2008, pp. 1885–1888.
  * [7] S. Leglaive, R. Hennequin, and R. Badeau, “Singing voice detection with deep recurrent neural networks,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2015, pp. 121–125.
  * [8] J. Schlüter and T. Grill, “Exploring data augmentation for improved singing voice detection with neural networks.” in _ISMIR_ , 2015, pp. 121–126.
  * [9] K. Lee, K. Choi, and J. Nam, “Revisiting singing voice detection: A quantitative review and the future outlook,” _arXiv preprint arXiv:1806.01180_ , 2018.
  * [10] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [11] B. Xu, N. Wang, T. Chen, and M. Li, “Empirical evaluation of rectified activations in convolutional network,” _arXiv preprint arXiv:1505.00853_ , 2015.
  * [12] N. Ono, K. Miyamoto, J. Le Roux, H. Kameoka, and S. Sagayama, “Separation of a monaural audio signal into harmonic/percussive components by complementary diffusion on spectrogram,” in _2008 16th European Signal Processing Conference_. IEEE, 2008, pp. 1–4.
  * [13] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [14] S.-I. Mirzadeh, M. Farajtabar, A. Li, and H. Ghasemzadeh, “Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher,” _arXiv preprint arXiv:1902.03393_ , 2019.