# Interpretable UAV Collision Avoidance using Deep Reinforcement Learning

Deepak-George Thomas  Daniil Olshanskyi  Karter Krueger  Tichakorn
Wongpiromsarn  Ali Jannesari

###### Abstract

> The significant components of any successful autonomous flight system are
> task completion and collision avoidance. Most deep learning algorithms
> successfully execute these aspects under the environment and conditions they
> are trained. However, they fail when subjected to novel environments. This
> paper presents an autonomous multi-rotor flight algorithm, using Deep
> Reinforcement Learning augmented with Self-Attention Models, that can
> effectively reason when subjected to varying inputs. In addition to their
> reasoning ability, they are also interpretable, enabling it to be used under
> real-world conditions. We have tested our algorithm under different weather
> conditions and environments and found it robust compared to conventional
> Deep Reinforcement Learning algorithms.

## Introduction

The use of AI-powered drones for various tracking problems has become widely
prevalent. They have been recently used for fighting forest fires, searching
for a target during disasters and observing wildlife. In addition, recent work
has focused on developing autonomous online algorithms that control the
trajectory of the Unmanned Aerial Vehicle (UAV) under different mission
scenarios. The ability of UAVs to adapt to diverse and novel constraints comes
from the path planning capability of the underlying algorithm controlling it.
[1, 2, 3, 4, 5, 6]. Deep Reinforcement Learning (DRL) is usually preferred for
this task over the conventional approaches such as A*, artifical potential
fields, Floyd warshall, dynamic programming and similar methods, since it does
not present the same disadvantages as other methods which make it infeasible
to implement in real world situations, [7, 8, 9, 10, 11, 12, 13].

Reinforcement Learning (RL) is associated with an agent or a set of agents,
placed within an environment, that take a series of actions to maximize their
expected reward [14]. A Model Free RL algorithm, Q-Learning, was developed to
obtain the action value for specific states [15]. Deep Q-Learning (DQN) (a
class of DRL algorithms) combined Q-Learning with a Deep Neural Network (DNN)
[16]. A DQN consists of multiple hidden layers, thereby allowing it to learn
and function well in situations involving high dimensional spaces. Therefore,
it can mitigate to some extent, the curse of dimensionality problem affecting
Q-Learning [17].
Furthermore, the training of robots under real-world conditions is an
expensive and time-consuming process. Therefore, it is beneficial to train the
algorithm under realistic simulators and then transfer the parameters learned
by the algorithm from the system into a real-world robot. Since simulated
training cannot capture all the aspects and nuances of the real world, it is
another reason why the training needs to be robust to novel environments and
changes in weather conditions and lightning. A further motivation for the
algorithm to be generalizable is that using all aspects of the real world is
not possible in simulation. Therefore, we can only train it with a subset of
such features.
Our objective is to improve the learning and generalization capability of our
multirotor using self-attention models. For collision avoidance tasks, a
Convolutional Neural Network (CNN) is generally used as a function
approximator. The CNN uses the robot camera as input and these inputs are then
considered to be the ‘State’ in the DQN algorithm. The output of the CNN is a
set of Q-values equaling the number of actions. Based on the Q value, the
algorithm chooses an action that is then used to maximize the expected reward
[18]. The disadvantage of using CNNs is that they are generally not invariant
to deformations or rotations. Therefore, the learning algorithm may not be
able to classify, similar objects within the environment under the same
category if they have been deformed or modified in any way. To enable the DNN
to generalize to modifications in the environment, it needs to have relational
properties, which allows it to classify objects while relating them with other
objects [19, 20]. Hence, we use a Self-Attention Model with Duelling Deep Q
Network (D3QN-SAM) for collision avoidance, while comparing its performance
with a Duelling Deep Q-Network that uses a CNN (D3QN-CNN).

## Related Work

### Attention Augmented Reinforcement Learning

[20] borrowed ideas from the Relational RL umbrella (RRL) [21, 22] that was
introduced more than 20 years prior. RRL uses relational representations to
represent states, policies and actions, and combines it with RL. In addition,
they implemented a self attention model to compute non-local interactions.
Their basis for choosing this model was that it wasn’t computationally
intensive and shared features with graph neural networks. Furthermore, this
model enabled obtaining relations between different entities. Therefore they
incorporated relational reasoning coupled with structured perception to
develop RL algorithms capable of interpretable reasoning. It could be
discerned that agents trained using such architectures tend to be more
generalized and providing better performance compared with baseline models.
Also, they found that the algorithm’s internal computations could be
visualized and the computations were in line with those architectures that
obtained task-specific relations [19].
[23] developed a novel architecture consisting of various recurrent cells that
had almost independent transition dynamics, known as Recurrent Independent
Mechanism (RIM). Similar to [20], they used multiple attention heads that
enabled communication between RIMs. In addition, these attention heads were
used to select which RIMs were required for the input at a particular time
step. They claimed that usually numerous processes consist of sub-processes
working independently. However, ML models assume that all of these processes
interact degrading the generalization of these algorithms. Therefore, to
address this issue, they used RIMs that contended for attention during all
time steps and were updated at only pertinent steps. The authors demonstrated
that this technique improved the generalization capability across a wide range
of tasks such as those having temporal patterns, objects and a combination of
patterns and objects.
[24] extended the idea of self-attention for visual RL tasks. They use neuro-
evolution to train their DRL agent that used self-attention to focus on
critical aspects within images in order to finish their tasks. It was observed
that the agent’s focusing ability enables it to generalize to new
environments, since it concentrated on few (necessary) aspects of the visual
stimuli and thereby was able to beat conventional algorithms with relatively
lower parameters. However, the authors noted that their algorithm still
couldn’t compete with human abilities of generalization since despite the
change in environment, the authors kept the properties of objects obtaining
high attention the same. They claimed that modifying those properties could
lead to performance degradation.

### Collision Avoidance using Reinforcement Learning

[25] used DRL coupled with Long Short Term Memory Networks to assist drone
navigation around city environments. Their algorithm was used to assist pilots
under safety critical scenarios. In addition, they implemented curriculum
learning to augment their algorithm, wherein training starts on a single task
whose complexity is increased as learning progresses [26]. However, the use of
attention based mechanisms including transformers have been used to replace
LSTMs and GRUs due to their inherent limitations [27].
Moreover, [18] developed an autonomous multi-rotor flight DRL algorithm that
enabled collision avoidance under novel scenarios while encouraging
exploration. The algorithm used depth images as input while flying across an
enclosed space with the intention of maximizing the number of steps taken
before a crash. Since the rewards obtained for this task was sparse, they
developed a guidance based approach to motivate their agent to explore [28].
In addition to the existing standard D3QN networks (online and target), an
additional CNN network was used to predict the future state and this
prediction was then compared to the states in the replay memory. Thereby, they
rewarded the agent if it moved to states that had least similarity with the
ones already present in the replay memory. A Bayesian Gaussian Mixture Model
was used to obtain a similarity value between the states. This technique
enabled the agent to explore the environment and address the sparsity in
rewards. The shortcoming of this algorithm is that the Gaussian Mixture Model
used for assessing similarity is computationally costly and training would
take a long time for large environments.
Also, [29] used Deep Recurrent Q-Networks augmented using Temporal attention
for the purpose of collision avoidance in multi-rotors. In addition, they use
Conditional Generative Adversarial Neural Networks to estimate depth images
from inputs obtained using a monocular depth camera. This algorithm was tested
in a simulated room full of people and their algorithm was able to beat
conventional baselines such as DQN and D3QN. A potential drawback of this
algorithm is that [30] experimentally showed that the addition of recurrent
layers did not help improve the performance of the algorithm relative to
stacking observations while using a CNN. Therefore, stacking multiple
observations together before passing it through a CNN could make the use of
Recurrent Q-Networks redundant while reducing the computational overhead
obtained from using RNNs.

## Background

### Duelling Double Deep Q Networks

Our work deals with a single RL agent that has to make sequential decisions in
discrete time steps while interacting with its environment, $\epsilon$. The
state space, $S$ contains M images, $s_{t}=(x_{t-M+1},...,x_{t})$, where $t$
refers to the time step. Also, we take our action space, $A$ to be discrete,
$a_{t}={1,...,\mid{A}\mid}$ and depending upon the action the agent obtains a
reward $r_{t}$. As previously described, the agent in a RL algorithm attempts
to maximize its expected return. In addition, the return is subject to
discounting, $\gamma$ which is used to determine the value difference between
future and current rewards.
$R_{t}=\sum_{\tau=t}^{\infty}\gamma^{\tau-1}r_{\tau}$, where $\gamma\in[0,1]$
The value of being in state $s$ or the state-action $(s,a)$, provided the
agent follows the policy $\pi$ is given by -
$Q^{\pi}(s,a)=\mathbb{E}[R_{t}|s_{t}=s,a_{t}=a,\pi]$
$V^{\pi}(s)=\mathbb{E}_{a\sim\pi(s)}[Q^{\pi}(s,a)]$
Dynamic programming can be used to compute the following Q function -
$Q^{\pi}(s,a)=\mathbb{E}_{s^{\prime}}[r+\gamma\mathbb{E}_{a^{\prime}\sim\pi(s^{\prime})}[Q^{\pi}(s^{\prime},a^{\prime})]|s,a,\pi]$
The optimal $Q^{*}(s,a)$ is given by, $Q^{*}(s,a)=max_{\pi}Q^{\pi}(s,a)$.
Also, we get $V^{*}s=max_{a}Q^{*}(s,a)$ from a deterministic policy,
$a=\arg\max_{a^{\prime}\in A}Q^{*}(s,a^{\prime})$. Therefore the optimal Q
function can be obtained as follows -
$Q^{*}(s,a)=\mathbb{E}_{s^{\prime}}[r+\gamma
max_{a^{\prime}}[Q^{*}(s^{\prime},a^{\prime})]|s,a]$
Moreover, based on the above equations, a new function can be defined that is
used to associate the Q and value functions;
$A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s)$, where $A$ is the advantage function
From the preceding equations, it is easy to discern that the ,
$\mathbb{E}_{a~{}\pi(s)}[A^{\pi}(s,a)]=0$. The relative significance of each
action can be found using the advantage function, since it is the difference
of the value of taking a specific action within a state and the value of being
in that state.
To approximate the value function, a deep Q-network can be employed:
$Q(s,a;\theta)$ having parameter $\theta$. Training the DNN involves
minimizing the following loss function;
$L_{i}(\theta_{i})=\operatorname{\mathbb{E}}_{s,a,r,s^{\prime}}[(y_{i}^{DQN}-Q(s,a;\theta_{i}))^{2}]$
where
$y_{i}^{DQN}=r+\gamma*max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta^{-}),$
and $i$ is the iteration at that time step. Here $\theta^{-}$ constitutes the
online and target network parameters. Having two networks, instead of one, has
been shown to improve the stability of the DQN. As per this technique, the
online network is updated every iteration, however the target network is
allowed to update its parameters only at specific iterations [16].
The downside of both DQN and Q-learning is that the values used to choose and
evaluate an action are the same, leading to value estimates that are
overoptimistic. Double Deep Q-Networks addressed this issue by replacing the
target $y_{i}^{DDQN}$ with $y_{i}^{DQN}$ as given below; [31]
$y_{i}^{DDQN}=r+\gamma
Q(s^{\prime},arg_{a^{\prime}}max(s^{\prime},a^{\prime};\theta_{i});\theta^{i})$
[32]
The motivation behind the Dueling Network Architecture was the observation
that calculating the action values for various states did not help the agent.
The authors demonstrated this while playing Enduro, the knowledge regarding
left or right movements did not affect the game unless there was a chance of
collision . The architecture consisted of a Q-network called the duelling
network. The lower Convolutional layers of the dueling network were then split
up into two fully connected layers estimating the advantage and value
functions. Ultimately, the two layers were joined to output a Q function
consisting of Q values for every action. [16]
Furthermore, the design of the Dueling Network Architecture consisted of a few
intricate details. We know that $Q^{\pi}(s,a)=V^{\pi}(s)+A^{\pi}(s,a)$ and
$V^{\pi}(s)=\mathbb{E}_{a\sim\pi(s)}[Q^{\pi}(s,a)]$, we can infer that
$\mathbb{E}_{a\sim\pi(s)}[A^{\pi}(s,a)]=0$
Also, in the case of a deterministic policy, $a^{*}=\arg\max_{a^{\prime}\in
A}Q(s,a^{\prime})$, we can infer that $Q(s,a^{*})=V(s)$ leading to
$A(s,a^{*})=0$ The two fully connected layers in the dueling network yield a
scalar $V(s;\theta,\beta)$ and the $\mid{A}\mid-$ dimensional vector
$A(s,a;\theta,\alpha)$ respectively. Here $\alpha,\beta$ represent the
parameters of the fully connected layers while $\theta$ denotes the
convolution parameters.
Moreover, the authors claimed that inferring the following equation from the
above steps would be wrong;
$Q(s,a;\theta,\alpha,\beta)=V(s;\theta,\beta)+A(s,a;\theta,\alpha)$
This is because $V(s;\theta,\beta)$ and $A(s,a;\theta,\alpha)$ might not be
good estimates of their functions. In addition, $Q(s,a;\theta,\alpha,\beta)$
is not the Q-function but only an estimate of it. Also, the authors found that
the above equation is unidentifiable, since one cannot find unique values of
$V$ and $A$ if they have $Q$. This problem can be solved by coercing the
estimator of the advantage function at the specified action to provide no
advantage. Therefore, the authors ran forward mapping using the network’s
final module leading to the advantage function estimator obtaining a zero
advantage for the specific action.
$Q(s,a;\theta,\alpha,\beta)=V(s;\theta,\beta)+(A(s,a;\theta,\alpha)-\\\
max_{a^{\prime}\in|A|}A(s,a^{\prime};\theta,\alpha))$
Also, from $a^{*}=\arg\max_{a^{\prime}\in
A}Q(a,a^{\prime};\theta,\alpha,\beta)=\\\ \arg\max_{a^{\prime}\in
A}A(s,a^{\prime};\theta,\alpha)$ leads to
$Q(a,a^{*};\theta,\beta)=V(s;\theta,\beta)$. Therefore from the layers
$A(s,a;\theta,\alpha)$ we get an estimate of the advantage function whereas
from the layers $V(s;\theta,\beta)$ we get the value function estimation.
In addition, to increase the stability of optimization the authors replaced
the max operator with average;
$Q(s,a;\theta,\alpha,\beta)=V(s,\theta,\beta)+(A(s,a;\theta,\alpha)-1/{|A|}\sum_{a^{\prime}}A(s,a^{\prime};\theta,\alpha))$
Finally, it was noted that learning algorithms using Q networks in general
could now be trained on the dueling architecture. [33]

### Self-Attention Models

Self-Attention Models (SAM) can be used to obtain attention weights among all
nodes within a set, $N:\mathbb{R}^{n\times f}$. Here $n$ and $f$ represent the
number of nodes and the features associated with it respectively, The result
of a SAM can be understood as an edge matrix with dimension $n\times n$,
$E:\mathbb{R}^{n\times n}$. The edge matrix enables the observer to discern
the relation between various nodes leading to a better understanding of how
the DNN ”reasons”. Nodes that are highly related can communicate and therefore
update their parameters. This entire process takes place within a single step
and is referred to as the relational module. The output of a relational
module, $N:\mathbb{R}^{n\times f}$, is then sent to the next relational
module.
SAM have been shown to work with great success on language models in the past.
However, in recent years it has also been used and demonstrated great promise
on vision tasks [34, 35]. However, the visual data needs to be pre-processed
before it can be used as input within a relational module. In essence, in
order to represent images as nodes we need to convert pixel data into
objects.To achieve this, a CNN is used to convert the input images into a
collection of feature maps. The feature maps are responsible for learning fine
and rough features that are present within the images. These features are then
concatenated together along its channels thereby creating a cuboid. This
cuboid can then be sliced along the horizontal and vertical dimensions to
create objects that can be viewed as nodes of a graph which is used as input
to the relational module.
The relational module architecture described below was presented in the paper
by [20]. The node matrix, $N:\mathbb{R}^{n\times f}$ is projected into key,
query and value matrices. The matrices responsible for projecting the initial
node matrix are set as learnable parameters during training and should
converge to those than can generate optimal attention weights from the key,
value and query matrices.
To facilitate a better understanding, the SAM for two nodes,
$a:\mathbb{R}^{10}$ and $b:\mathbb{R}^{10}$ has been described here. We get
key, value and query matrices from these nodes through their projection
matrices.
$a_{Q}=a^{T}Q$, $a_{K}=a^{T}K$, $a_{V}=a^{T}V$
To obtain the self similarity for the nodes, an inner product operation is
performed between the key and query matrices of the respective nodes.
Generally the similarity measure for a node with itself is expected to be
high. However, depending upon its key and query that might not be the case. An
un-normalized scalar attention weight is generated by the product of the key
and query matrices of a particular node, $w_{a,a}=<a_{Q},a_{K}>$. This process
can now be extended to both of the nodes a and b. This leads to four attention
weights, (two for the interaction between the nodes and two for interaction
with themselves). A softmax function is then used to normalize the attention
weights so that each of them are between $[0,1]$ and they sum to 1. This
normalization is crucial to ensuring that the attention weight focus only on
the necessary nodes.
The attention weights for the nodes a and b are stored in a $2\times 2$
matrix. Subsequently, the value matrix gets updated by the product of the
attention weights and the value matrix. Therefore, each node is modified
depending upon its connection with the other. This process can now be
generalized to $n$ nodes.
The process of generating an attention matrix by multiplying the key and query
together and then using it to update the value matrix along with normalization
can be combined and represented as a matrix operation -
$\hat{N}=softmax(QK^{T})V$
Here n, f, Q, K and V represent the number of nodes, node feature vector
dimension, query matrix, key matrix and value matrix respectively with the
following dimensions; $Q:\mathbb{R}^{n\times f}$, $K^{T}:\mathbb{R}^{f\times
n}$ and $V:\mathbb{R}^{n\times f}$. The attention weight matrix $(QK^{T})$
will have dimension $n\times n$ with the value at $(x,y)$ demonstrating the
strength of the interaction of node $x$ with the node at $y$. The product of
the normalized attention weight matrix, $A$ with the value matrix updates the
feature vector of all nodes corresponding to its interaction with others,
leading to a node matrix with dimensions; $\hat{N}:\mathbb{R}^{n\times f}$.
The last step to create a relational block is to run a linear layer over the
resulting node matrix before implementing non-linearity for augmenting the
learning process. Multiple relational blocks can be stacked together to
increase the model complexity leading to better learning of higher order
relations [19, 27].

### Multi-Head Self Attention

The use of softmax in SAM models restricts the the attention weights that can
be placed on individual objects, especially for cases where there are a large
number of nodes. Moreover, this leads to cases wherein the attention is being
placed on a limited number of nodes and others are being ignored. In order to
mitigate the amount of information lost, we can use multiple independent
attention heads and subsequently concatenate them together. Therefore, these
attention heads could concentrate on different areas of the image, thereby
increasing the amount of learning without affecting the relational modules.
Another advantage of attention heads is that nodes corresponding to respective
heads can concentrate on a smaller set of nodes, instead of providing a minute
amount of weights by trying to cover all the nodes, leading to increased
interpretability [19, 27].

## Problem Description

### Problem Statement

Our goal is to train a multi-rotor to avoid obstacles while flying around an
area using DRL algorithms augmented with SAMs and CNNs. In addition, we test
the performance of our trained algorithms on different weather conditions as
well as new environments. Finally, we visualize the attention weights in order
to interpret the decisions made by our algorithm.

### Microsoft Airsim

The simulation environment was generated using the Microsoft Airsim Simulator,
which provided the physics behind the control of the UAV Quadcopter used in
the project. It can be implemented like a plugin for Unreal and can work with
all Unreal environments. This is beneficial for robotic applications such as
collision avoidance since evaluating such algorithms in the real world is
expensive. Furthermore, Airsim can accommodate UAV movements such as pitch,
roll and yaw through its API calls. In addition, the communication between the
python RL program and the simulation occurs through the Airsim API. Figure 1,
below shows the 3-D environments developed for training our agent using the
Unreal Engine [36, 37].

Figure 1: Training Environment

### MDP Formulation

#### State Space

The state space $S$ for our algorithm is in the form of depth images taken by
the depth camera during training in Airsim (Figure 2). Similar to [16], we
stack 5 frames together to model the sequential movement of the drone.

Figure 2: Image from Depth Camera

#### Action Space

One of the following four actions is executed by the agent during every time
step.
$A=\\{a_{0},a_{1},a_{2},a_{3}\\}$
The linear velocity $(v)$ is controlled by two actions, $a_{0}$ and $a_{3}$
with a speed of 1m/s and 2m/s respectively. The actions $a_{1}$ and $a_{2}$
consist of a combination of angular and linear velocity. $a_{1}$ makes the
multi-rotor take a $None$ yaw-right and then move in the forward direction at
1m/s. Similarly, $a_{2}$ forces the agent to take a $None$ yaw-left and then
move forward at 1m/s.

#### Reward Function

The reward function was designed so that the UAV learns to avoid collisions
while staying away from walls and objects. Therefore it consists of two major
components. The first component gives it a negative reward of -10 in case it
collides against the wall. However, minor brushes against the wall are
ignored. The next component rewards the UAV for staying away from objects. At
each time step, it extracts the centre of the image feed using $20\%$ of the
width and height of depth frame. Next, it finds the minimum value within that
region $\beta$ and compares it against a minimum distance threshold,
$\delta=3$. If the minimum value of the center of the depth image is greater
than the threshold, we consider the UAV to be in a safe region and reward it
proportionally based on how far it is from the wall. Therefore under this
condition, the agent gets a reward of $3+\beta/5$. This gives the agent a
reward of 3 for maintaining that threshold along with a proportionally
increasing reward for finding the ”most-open” area centered in front of it.
For cases where $\beta$ is less than $\delta$, a negative reward proportional
to how close it is to the walls is placed on the agent, equalling
$\delta/(\beta+0.5)*-1$ . We add a $0.5$ to the denominator to account for
cases where $\beta=0$, when the multi-rotor is extremely close to the wall,
and prevent the expression from becoming $-\inf$.

$R=\begin{cases}\delta/(\beta+0.5)*-1&\text{$\beta<\delta$}\\\
3+\beta/5&\text{$\beta>\delta$}\\\ -10&\text{Collision Detected}\end{cases}$

#### Model Dynamics

The reinforcement learning model largely follows work done by [20]. It takes
in depth images as input and calculates the $Q(s_{t},a_{t})$ for all actions
while it is at the state $s_{t}$. After taking an action based on
$\arg\max_{a}Q(s_{t},a_{t})$ it reaches the next state $s_{t+1}$. The tuple
containing the information $(s_{t},a_{t},r_{t},s_{t+1})$ are stored in the
replay memory $M$. In the event of a crash, the environment resets and the UAV
returns to its initial position. The model uses a CNN to learn features
obtained from the simulation depth camera. Next, it converts these features
along the channel dimension to form individual nodes. This converts our two-
dimensional image into a graph-based representation. Moreover, it projects
these nodes into key, query and value matrices. This representation enables
the attention algorithm to compute the relation between different nodes and
generate an attention matrix. The last steps include running the attention
matrix through linear layers and applying pooling operations (similar to that
of CNN operations) to generate an output layer with nodes equaling the number
of actions of the agent.

## Results and Discussions

### Training Results

In order to compare the performance of the SAM model with a conventional
baseline, we trained two D3QN agents. One agent had a SAM (with 3 heads) as
its function approximator whereas the other had a CNN. Each agent was trained
for a total of 8000 epochs. While tuning the number of epochs, we found that
using a large number of epochs led to catastrophic forgetting [38], wherein
the agent’s performance would suddenly drop and it seemed to ’forget’ whatever
it had learned during previous iterations. Furthermore, the training metrics
are calculated based on episodes, which can be defined as the series of steps
between each episode reset. The average episode length (Figure 3) and the
average reward (Figure 4) obtained by the agent was used to discern how well
the learning process was going. The D3QN SAM and the D3QN CNN have quite
similar plots. Also, the mean episode length and episode reward follow the
same trend, which implies that that reward function is helping the agent with
its objective. Moreover, during the epsilon-decay process, wherein the
$\epsilon$ is reduced during the course of training, the final epsilon was
kept at 0.05. This makes the agent take rare, random actions even after the
$epsilon$ has completely decayed.
Furthermore, we found that it was important to view the training videos in
simulation. Initially, we set our actions $a_{1}$ and $a_{2}$ to consist only
of yawing actions minus the linear velocity. The agent increased its rewards
and steps just by staying at one place and rotating indefinitely.
While comparing our work with [18], we noted that the training times using the
Guidance Training algorithm were a lot larger and that it took days to train
before it attained a reasonable policy. In contrast, the SAM model was able to
consistently increase its rewards and reduce its crash rate in a few hours.
Furthermore, [29] used temporal attention with DRL that allocates weights to
past observations(images) based on its importance and then predicts a
trajectory while faced with moving obstacles. However, our SAM algorithm
instead detects the relationship between different objects in a single image,
which allows the algorithm to be robust in the presence of noise and ,to a
certain extent, invariant to changes in the obstacles geometry [39].

Figure 3: Average Episode Length Figure 4: Average Episode Reward

### Testing Results

To test the robustness of our algorithm, we added various obstacles as well
tested the performance of the algorithm under different weather conditions
including rain, snow and falling leaves for 1000 time steps. During this
evaluation, the parameters were fixed and the D3QN algorithms were only
allowed to exploit without doing any exploration. The algorithms were first
tested on the environment in which they were trained (Basic Environment).
Next, we developed a new environment consisting of cylindrical and rectangular
beams as shown in Figure 5 and referred to it as the Complex Environment.
Finally, we changed the weather conditions within the Basic environment using
a combination of snow, rain and leaves and called it the Turbulent Environment
(Figure 6). The results of the testing process are shown in Table I.

TABLE I: Number of Crashes during Testing Case | D3QN-SAM | D3QN-CNN
---|---|---
Basic Environment | 0 | 8
Complex Environment | 1 | 5
Turbulent Environment | 1 | 10

It is evident from Table I that the D3QN-SAM performs much better than the the
D3QN-CNN under all cases. The D3QN-SAM had 0 failures in the Basic
Environment, while the D3QN-CNN failed 9 times which is surprising as both
algorithms had similar results during training. The architecture of the
Complex Environment was drastically different from the Basic Environment for
we wanted to check whether our algorithms actually learned a collision
avoidance policy rather than just follow a fixed path that minimized contact
with obstacles in the training environment. In addition, this environment was
filled with beams and the agent would have to constantly make detours to avoid
crashing. Based on the low number of crashes we can confidently claim that the
algorithms learned the desired policy with the D3QN-SAM having just 1 failure
in this environment. The Turbulent Environment was designed so as to distort
the camera feed and therefore the input states provided to the DRL algorithm.
The change in the depth image sent to the algorithm can be viewed at the lower
left of Figure 6. The D3QN-SAM was robust to change in this feed for, based on
its relational modules, it could identify the relation between objects and
place attention on the pertinent aspects of the state.

TABLE II: Average Rewards during Testing Case | D3QN-SAM | D3QN-CNN
---|---|---
Basic Environment | 2.406 | 2.459
Complex Environment | 2.16 | 2.319
Turbulent Environment | -7.809 | -7.83

Table II shows the average rewards obtained by the agents of both algorithms
during testing. The process for calculating these reward values is different
when compared to the training. During training, the average reward per episode
was calculated. This table shows the average reward obtained during 1000 time
steps the algorithm was run. It is interesting to note that both algorithms
have similar values for each environment, while the D3QN-SAM had fewer crashes
compared to the D3QN-CNN. One way to explain this is that the former algorithm
learned a better and robust policy while obtaining the same rewards and
further investigation needs to be done regarding this. The rewards obtained by
the Complex Environment is slightly lower than that of the Basic Environment.
This can be attributed to the fact that it has many obstacles present in the
form of vertical beams and one aspect of the reward depends on how far the
agent is from any obstruction. Furthermore, the rewards present in the
Turbulent Environment seem inconsistent with the other environments. The
objects floating around in the Turbulent Environment, such as leaves, cause
the depth sensor to obtain low values of depth even when the agent is far from
walls or beams. These low values in turn generate negative reward values.

Figure 5: Complex Environment Figure 6: Turbulent Environment

### Interpreting the D3QN-SAM

A major use of attention models in DNN is the ability to visualize its
weights, enabling a better understanding of how the network makes its
decisions. The attention weights of the D3QN agent has been visualized,
presented in green color, while it was placed in the Basic and Turbulent
Environment (Figure 7). It can be seen that a relatively large amount of
attention is placed on walls that are farthest from the UAV, which can be
interpreted as the agent trying to focus on the optimal distance to increase
it rewards. However, the choice to place the attention weights in Figure 7(b)
is sub-par, for it would have been better to focus on the opening next to the
walls. Perhaps increasing the rewards for staying away from walls and
obstacles would lead to better placement of attention weights [19].

(a)

(b)

Figure 7: Visualizing Attention Weights in a) Turbulent and b) Basic
Environment

## Conclusion and Future Work

This paper investigates the integration of SAM with DRL for the purpose of UAV
Collision Avoidance, while comparing it with a conventional CNN based model.
While the training results were quite similar, the DRL-SAM outperformed the
DRL-CNN during testing. Also, we show that the DRL-SAM algorithm is
interpretable and therefore can be incorporated into safety-critical vision
algorithms.
The next steps will involve getting the algorithm ready to be transferred onto
a real drone. To further reduce the chances of collisions both in simulation
and real-life we propose, that along with an penalty of coming close to
objects, the episode should end with a large negative reward if the depth
values reduce below a threshold. This would further encourage the UAV to keep
a minimum distance from obstacles under all circumstances. Finally, based on
the work by [29] and [30], we suggest a study on understanding the efficacy of
stacking a larger number of observations as input to a CNN. While traversing
difficult environments, the UAV would need to memorize larger trajectories for
cases where they are moving obstacles present.

## References

  * [1] Samira Hayat, Evşen Yanmaz, Timothy X Brown and Christian Bettstetter  “Multi-objective UAV path planning for search and rescue”  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 5569–5574  IEEE
  * [2] Hao Jiang and Yueqian Liang  “Online path planning of autonomous UAVs for bearing-only standoff multi-target following in threat environment”  In _IEEE Access_ 6 IEEE, 2018, pp. 22531–22544
  * [3] Ravi N Haksar and Mac Schwager  “Distributed deep reinforcement learning for fighting forest fires with a network of aerial robots”  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 1067–1074  IEEE
  * [4] Chunxue Wu et al.  “UAV autonomous target search based on deep reinforcement learning in complex disaster scene”  In _IEEE Access_ 7 IEEE, 2019, pp. 117227–117245
  * [5] Mitchell B Lyons et al.  “Monitoring large and complex wildlife aggregations with drones”  In _Methods in Ecology and Evolution_ 10.7 Wiley Online Library, 2019, pp. 1024–1035
  * [6] AE Asli, Jeremy Roghair and Ali Jannesari  “Energy-aware Goal Selection and Path Planning of UAV Systems via Reinforcement Learning”  In _arXiv preprint arXiv:1909.12217_ , 2019
  * [7] Mohammadreza Radmanesh, Manish Kumar, Paul H Guentert and Mohammad Sarim  “Overview of path-planning and obstacle avoidance algorithms for UAVs: A comparative study”  In _Unmanned systems_ 6.02 World Scientific, 2018, pp. 95–118
  * [8] Farid Bounini, Denis Gingras, Herve Pollart and Dominique Gruyer  “Modified artificial potential field method for online path planning applications”  In _2017 IEEE Intelligent Vehicles Symposium (IV)_ , 2017, pp. 180–185  IEEE
  * [9] B Moses Sathyaraj, Lakhmi C Jain, Anthony Finn and S Drake  “Multiple UAVs path planning algorithms: a comparative study”  In _Fuzzy Optimization and Decision Making_ 7.3 Springer, 2008, pp. 257
  * [10] Chaymaa Lamini, Said Benhlima and Ali Elbekri  “Genetic algorithm based approach for autonomous mobile robot path planning”  In _Procedia Computer Science_ 127 Elsevier, 2018, pp. 180–189
  * [11] Alonica Villanueva and Arnel Fajardo  “Deep Reinforcement Learning with Noise Injection for UAV Path Planning”  In _2019 IEEE 6th International Conference on Engineering Technologies and Applied Sciences (ICETAS)_ , 2019, pp. 1–6  IEEE
  * [12] Subrahmanyam Vaddi, Chandan Kumar and Ali Jannesari  “Efficient object detection model for real-time UAV applications”  In _arXiv preprint arXiv:1906.00786_ , 2019
  * [13] Rahim Mammadli, Felix Wolf and Ali Jannesari  “The Art of Getting Deep Neural Networks in Shape”  In _ACM Transactions on Architecture and Code Optimization (TACO)_ 15.4, 2019, pp. 62:1–62:21  DOI: 10.1145/3291053
  * [14] Vincent François-Lavet et al.  “An introduction to deep reinforcement learning”  In _arXiv preprint arXiv:1811.12560_ , 2018
  * [15] Christopher JCH Watkins  “Technical note q-learning”  In _Reinforcement learning_ Kluwer Academic, 1993
  * [16] Volodymyr Mnih et al.  “Human-level control through deep reinforcement learning”  In _nature_ 518.7540 Nature Publishing Group, 2015, pp. 529–533
  * [17] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage and Anil Anthony Bharath  “Deep reinforcement learning: A brief survey”  In _IEEE Signal Processing Magazine_ 34.6 IEEE, 2017, pp. 26–38
  * [18] Jeremy Roghair, Kyungtae Ko, Amir Ehsan Niaraki Asli and Ali Jannesari  “A Vision Based Deep Reinforcement Learning Algorithm for UAV Obstacle Avoidance”  In _arXiv preprint arXiv:2103.06403_ , 2021
  * [19] Alexander Zai and Brandon Brown  “Deep reinforcement learning in action”  Manning Publications, 2020
  * [20] Vinicius Zambaldi et al.  “Relational deep reinforcement learning”  In _arXiv preprint arXiv:1806.01830_ , 2018
  * [21] Sašo Džeroski, Luc De Raedt and Hendrik Blockeel  “Relational reinforcement learning”  In _International Conference on Inductive Logic Programming_ , 1998, pp. 11–22  Springer
  * [22] Sašo Džeroski, Luc De Raedt and Kurt Driessens  “Relational reinforcement learning”  In _Machine learning_ 43.1 Springer, 2001, pp. 7–52
  * [23] Anirudh Goyal et al.  “Recurrent independent mechanisms”  In _arXiv preprint arXiv:1909.10893_ , 2019
  * [24] Yujin Tang, Duong Nguyen and David Ha  “Neuroevolution of self-interpretable agents”  In _Proceedings of the 2020 Genetic and Evolutionary Computation Conference_ , 2020, pp. 414–424
  * [25] Victoria J Hodge, Richard Hawkins and Rob Alexander  “Deep reinforcement learning for drone navigation using sensor data”  In _Neural Computing and Applications_ 33.6 Springer, 2021, pp. 2015–2033
  * [26] Yoshua Bengio, Jérôme Louradour, Ronan Collobert and Jason Weston  “Curriculum learning”  In _Proceedings of the 26th annual international conference on machine learning_ , 2009, pp. 41–48
  * [27] Ashish Vaswani et al.  “Attention is all you need”  In _arXiv preprint arXiv:1706.03762_ , 2017
  * [28] Stephen Zhen Gou and Yuyang Liu  “DQN with model-based exploration: efficient learning on environments with sparse rewards”  In _arXiv preprint arXiv:1903.09295_ , 2019
  * [29] Abhik Singla, Sindhu Padakandla and Shalabh Bhatnagar  “Memory-based deep reinforcement learning for obstacle avoidance in UAV with limited environment knowledge”  In _IEEE Transactions on Intelligent Transportation Systems_ IEEE, 2019
  * [30] Matthew Hausknecht and Peter Stone  “Deep recurrent q-learning for partially observable mdps”  In _arXiv preprint arXiv:1507.06527_ , 2015
  * [31] Hado Van Hasselt, Arthur Guez and David Silver  “Deep reinforcement learning with double q-learning”  In _arXiv preprint arXiv:1509.06461_ , 2015
  * [32] Hado Hasselt  “Double Q-learning”  In _Advances in neural information processing systems_ 23, 2010, pp. 2613–2621
  * [33] Ziyu Wang et al.  “Dueling network architectures for deep reinforcement learning”  In _International conference on machine learning_ , 2016, pp. 1995–2003  PMLR
  * [34] Prajit Ramachandran et al.  “Stand-alone self-attention in vision models”  In _arXiv preprint arXiv:1906.05909_ , 2019
  * [35] Hengshuang Zhao, Jiaya Jia and Vladlen Koltun  “Exploring Self-Attention for Image Recognition”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [36] Juan Gonzalo Cárcamo Zuluaga, Jonathan P Leidig, Christian Trefftz and Greg Wolffe  “Deep reinforcement learning for autonomous search and rescue”  In _NAECON 2018-IEEE National Aerospace and Electronics Conference_ , 2018, pp. 521–524  IEEE
  * [37] Shital Shah, Debadeepta Dey, Chris Lovett and Ashish Kapoor  “Airsim: High-fidelity visual and physical simulation for autonomous vehicles”  In _Field and service robotics_ , 2018, pp. 621–635  Springer
  * [38] James Kirkpatrick et al.  “Overcoming catastrophic forgetting in neural networks”  In _Proceedings of the national academy of sciences_ 114.13 National Acad Sciences, 2017, pp. 3521–3526
  * [39] Wenjie Pei, Tadas Baltrusaitis, David MJ Tax and Louis-Philippe Morency  “Temporal attention-gated model for robust sequence classification”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6730–6739


# A Deep Learning Approach To Dead-Reckoning Navigation For Autonomous
Underwater Vehicles With Limited Sensor Payloads

Ivar Bjørgo Saksvik  Department of Mechanical, Electronic
and Chemical Engineering
Oslo Metropolitan University
Oslo, Norway
<EMAIL_ADDRESS>Alex Alcocer  Department of Mechanical, Electronic
and Chemical Engineering
Oslo Metropolitan University
Oslo, Norway
<EMAIL_ADDRESS>Vahid Hassani  Department of Mechanical, Electronic
and Chemical Engineering
Oslo Metropolitan University
Oslo, Norway
<EMAIL_ADDRESS>
###### Abstract

This paper presents a deep learning approach to aid dead-reckoning (DR)
navigation using a limited sensor suite. A Recurrent Neural Network (RNN) was
developed to predict the relative horizontal velocities of an Autonomous
Underwater Vehicle (AUV) using data from an IMU, pressure sensor, and control
inputs. The RNN network is trained using experimental data, where a doppler
velocity logger (DVL) provided ground truth velocities. The predictions of the
relative velocities were implemented in a dead-reckoning algorithm to
approximate north and east positions. The studies in this paper were twofold
I) Experimental data from a Long-Range AUV was investigated. Datasets from a
series of surveys in Monterey Bay, California (U.S) were used to train and
test the RNN network. II) The second study explore datasets generated by a
simulated autonomous underwater glider. Environmental variables e.g ocean
currents were implemented in the simulation to reflect real ocean conditions.
The proposed neural network approach to DR navigation was compared to the on-
board navigation system and ground truth simulated positions.

###### Index Terms:

Underwater Navigation, Deep learning, Dead-reckoning, Autonomous Underwater
Vehicles (AUV)

##  I Introduction

Autonomous underwater vehicles (AUVs) have in the last decades become
important tools in ocean research. Untethered from umbilical cables, these
vehicles are suitable for a high variety of applications including bathymetric
mapping, water sampling and environmental monitoring. A notorious challenge
for AUVs is to navigate and georeference acquired sensor data during
operations as GPS signals can’t propagate trough water. Conventional solutions
to this issue involve adding acoustic navigational or/and positioning
instruments to the AUV payload. Due to the good propagation of sound in water,
doppler velocity loggers and acoustic baseline systems are considered the
backbone in AUV navigation and underwater positioning [10], [25]. However,
these traditional sensors are often expensive and consumes large amounts of
power. In AUV fleets, the cost of adding acoustic instruments is compounded
with the number of vehicles. In this paper we consider a limited sensor suite
consisting of an IMU sensor and a pressure transducer, where acoustic
instruments are partially available to collect experimental training data.
Collected DVL velocity measurements from only a few missions are used as a
reference in supervised neural network training. The aim for the trained
network is to complement DR navigation when the DVL sensor is inaccessible,
for example in AUV fleets with budget limitations.

The absence of acoustic navigational and positioning instruments has
traditionally been compensated by model-based observers like Extended Kalman
Filters (EKFs). These are derived from AUV dynamics to form an estimation
model [8], [9], [21], [22]. Unfortunately, model-based observers rely on
parameters that are difficult to obtain in practice. The dynamics of an AUV is
derived based on intricate hydrodynamic models. Experiments must be carried
out in a towing-tank facility or using expensive CFD (Computational Fluid
Dynamics) software to obtain hydrodynamic damping coefficients [28], [3]. If
the external geometry of the AUV changes, i.e. when making small modifications
to payload sections, the coefficients need to be updated.

To avoid deriving complex AUV models and conducting time consuming towing-tank
or CFD experiments, this paper presents a data-driven approach to dead-
reckoning navigation. Using experimental data from AUV missions and
simulations, a neural network is trained to learn and generalize relative AUV
motions. Data-driven neural network regression abolishes the need for
knowledge of a dynamic model, and avoids modelling and estimation errors
related to classical state observers [4], [5]. A recurrent neural network
(RNN) is developed to relax time-delayed effects in the AUV dynamics which
occurs due to vehicle inertia, under actuation and added mass effects [1],
[6]. With an input layer composed of standard sensory measurements (pressure
sensor, inertial measurement unit) and control actions, the RNN network aims
to predict relative surge $u_{r}$ and sway $v_{r}$ velocities. These are
further implemented in a dead-reckoning algorithm to approximate North and
East positions during operations.

###  I-A Related Work

Several articles have addressed artificial neural network state estimation for
marine crafts. In Zhang et al. [4] a Short-Term Long-Term-Memory (LSTM)
recurrent neural network is proposed to estimate the relative position of an
AUV. The LSTM network used data from a pressure sensor, an inertial
measurement unit (IMU), and an acoustic doppler velocity logger (DVL) to
predict the horizontal north and east positions. Training and validation data
were collected from a series of surface trajectories while logging GPS
locations, which were projected as ground truth measurements. A similar study
with the same AUV is presented in Mu et al. [5], where a bi-directional LSTM
network was used. A neural network approach to dead-reckoning navigation of
dynamically positioned ships is presented in Skulestad et al. [6]. Control
actions and commands from vessel thrusters combined with heading measurements
was used as input data in a RNN network to aid navigation during GNSS outages.
Experiments were conducted in a vessel simulator with time-varying
environmental disturbances such as wind forces, sea waves and ocean currents.
In Chen et al. [13] a neural network is presented to assist navigation during
DVL malfunction. A nonlinear autoregressive network with exogenous SINS
(Strapdown Inertial Navigation System) inputs was used. The network was tested
and validated on a ship with a DVL mounted on the vessel hull to provide
training and validation data.

The remaining parts of this paper are detailing the following segments -
Section II and III addresses the concept of dead-reckoning navigation and the
neural network velocity observer respectively. Section IV presents the AUV
platforms and datasets used to train and test the neural networks. The results
are detailed in section V and the conclusion and recommendations for further
work are presented in VI.

##  II Dead-Reckoning Navigation

In the absence of GNSS (Global Navigation Satellite Systems) systems, AUVs
enters a dead-reckoning mode while under water. The DR algorithm predicts the
position of the AUV based on estimates at the previous time-step. With a
reference of the heading and attitude combined with relative velocity
measurements, the position is determined by numerical integration. To compute
the relative position of the AUV, the measured/estimated relative velocities
must be rotated with respect to the inertial reference frame of the vehicle.
Following [1] the inertial frame of underwater vehicles is defined by North-
East-Down (NED) local tangent plane coordinates. The NED velocities
$\boldsymbol{\dot{\chi}}=[\dot{N},\dot{E},\dot{D}]^{T}$ of an AUV are derived
by an rotation matrix from the body frame $\\{b\\}$ to the inertial frame
$\\{n\\}$ [1]. An AUV influenced by ocean currents $\boldsymbol{\upsilon}_{c}$
will have a relative velocity $\boldsymbol{\upsilon}_{r}$. Assuming that the
ocean currents are irrotational they are derived following [1] as
$\boldsymbol{\upsilon}_{r}=[u_{r},v_{r},w_{r}]^{T}=[u-u_{c},v-v_{c},w-w_{c}]^{T}$.
Accordingly, the relationship between the relative body-fixed and inertial
velocities are given as

$\boldsymbol{\dot{\chi}}=\boldsymbol{R}_{b}^{n}(\boldsymbol{\Theta})\cdot\boldsymbol{\upsilon}_{r}$
(1)

Where $\boldsymbol{\Theta}=[\phi,\theta,\psi]^{T}$ is the attitude and heading
of the AUV provided by an inertial measurement unit (IMU). Equation 1 can be
written in expanded form as

$\begin{bmatrix}\dot{N}\\\ \\\ \dot{E}\\\ \\\
\dot{D}\end{bmatrix}=\begin{bmatrix}u_{r}\cdot
c(\psi)c(\theta)+v_{r}\cdot[c(\psi)s(\theta)s(\phi)-s(\psi)c(\phi)]\\\
+w_{r}\cdot[s(\psi)s(\phi)+c(\psi)c(\phi)s(\theta))]\\\ \\\ u_{r}\cdot
s(\psi)c(\theta)+v_{r}\cdot[c(\psi)c(\phi)+s(\phi)s(\theta)s(\psi)]\\\ \\\
-u_{r}\cdot s(\theta)+v_{r}\cdot c(\theta)s(\phi)+w_{r}\cdot
c(\theta)c(\phi)\\\ \\\ \end{bmatrix}$ (2)

where $c()=cos()$ and $s()=sin()$.
For AUVs that typically operate with a zero angle of attack the attitude can
be neglected in eq. 2. However, for other vehicles like underwater gliders
which can perform spiraling motions with non-zero attitude
$[\phi,\theta]^{T}\neq 0$, it persist. After rotating the relative velocities
according to the inertial frame of the vehicle, numerical integration is
performed to obtain the position. Given the previous predicted position
$\boldsymbol{\chi}_{t}=[N_{t},E_{t},D_{t}]^{T}$ the DR algorithm is derived
following [6]

$\boldsymbol{\chi}(t+1)=\boldsymbol{\chi}(t)+\boldsymbol{R}^{n}_{b}(\boldsymbol{\Theta})\cdot\boldsymbol{\upsilon_{r}}(t+1)\cdot\Delta
t$ (3)

Where $\Delta t$ is the time step between the predictions.

Figure 1: DR navigation Illustration

##  III Neural Network Aided Dead-Reckoning Navigation

###  III-A Sensor Inputs

On-board sensors like IMUs and pressure transducers contains valuable in-situ
information about an AUV. These are used as input variables in the RNN network
to predict relative horizontal velocities $[u_{r},v_{r}]^{T}$. The development
of MEMS (Micro-Electrical-Mechanical-Systems) based IMUs have led to more
affordable inertial measurements. They consist of three-axis gyroscopes and
accelerometers, typically confined in a silicon chip. The gyroscope, which
give measurements of the angular rates
$\boldsymbol{\omega^{b}_{IMU}}=[p_{b},q_{b},r_{b}]^{T}$ and the accelerometer
which provide measurements of the rate-change of velocities
$\boldsymbol{\dot{\upsilon}^{b}_{IMU}}=[\dot{u}_{b},\dot{v}_{b},\dot{w}_{b}]^{T}$
can be modelled as

$\begin{split}\boldsymbol{\omega}^{b}_{IMU}&=\boldsymbol{\omega}^{b}_{gyro}+\boldsymbol{b}^{b}_{gyro}\\\
\boldsymbol{\dot{\upsilon}}^{b}_{IMU}&=\boldsymbol{\dot{\upsilon}}^{b}_{acc}+\boldsymbol{b}^{b}_{acc}\end{split}$
(4)

Where $\boldsymbol{b}^{b}_{gyro}$ and $\boldsymbol{b}^{b}_{acc}$ are sensor
biases [33]. Combined with a three-axis compass, a Kalman Filter derived from
a kinematic model impart the euler angles
$\boldsymbol{\Theta}=[\phi,\theta,\psi]^{T}$. A key component in AUVs is the
pressure transducer. The relationship between pressure and depth are assumed
to be constant, thus the vertical position of the AUV can be approximated by
the pressure measurements. Given a measured hydrostatic pressure $\Delta p$,
water density $\rho$ and gravitation $g$, the vertical position $z$ and
relative heave velocity $w_{r}$ is derived as

$z=\rho g\Delta p\implies
w_{r}=\boldsymbol{R}^{-1}_{bn}(\boldsymbol{\Theta})\cdot\dot{z}$ (5)

Where $z$ is assumed to be inertial $\\{n\\}$ and the relative heave velocity
$w_{r}$ is represented in the body-fixed frame $\\{b\\}$

###  III-B AUV Control Actuators

To enforce the neural network state observer, control actions from the AUV
actuators are used together with the sensor data. In this paper two AUVs with
different actuator configurations are investigated. Conventional underactuated
AUVs are normally equipped with an aft thurster and external control surfaces.
The thruster is the propulsion system which generates a hydrodynamic force
$\tau$ to induce surge transnational motions, while control surfaces consist
of external airfoils that alter attitude and heading depending on their
deflection angles. The control surfaces typically consist of a rudder and dive
planes denoted $\delta_{R}$ and $\delta_{D}$ respectively. Control actions are
determined from feedback controllers which in these studies are decoupled into
vertical and horizontal manoeuvres. For thruster based AUVs, speed controllers
are used to maintain a desired velocity and reject ocean current disturbances.
In addition to the conventional AUV actuators, a variable buoyancy system and
internal moving masses are introduced by the AUVs investigated in this paper.
The simulated autonomous underwater glider uses buoyancy displacement to alter
vertical motions, while a set of fixed wings generates hydrodynamic lift
forces to induce forward motions. As gliders operates at low-speeds, control
surfaces are ineffective due to low dynamic pressure. Control moments from
internal moving masses are used to change the attitude and heading of the
vehicle.

###  III-C RNN Architecture

The RNN architecture is formed with feedback loops in the hidden layers of the
network, providing internal memory to capture AUV dynamics with time-delays.
To improve the estimation of the relative horizontal surge and sway velocities
$[u_{r},v_{r}]^{T}$, two independent neural networks are used for each
velocity vector. This is convenient as surge and sway dynamics are often non-
interacting or slightly interacting [1], [4]. Using input variables that holds
low dependence to the predicted output variables reduces the generalization of
the network and may lead to the notorious issue of overfitting [20], [15].

Figure 2: Neural network illustration

RNN networks are generalized feedforward neural networks, but differs as
recurring context layers are included in the hidden layers as illustrated in
figure 2. The hidden nodes $\boldsymbol{h}(t)$ summarize the inputs
$\boldsymbol{x}$ and weights $w_{xh}$ from the previous layer combined with
the recurring layer $\boldsymbol{h}(t-i)$. The hidden neurons
$\boldsymbol{h(t)}$ and output neurons $\boldsymbol{y(t)}$ is derived
mathematically as

$\begin{split}\boldsymbol{h}(t)&=\sigma_{h}(\sum
w_{xh}\cdot\boldsymbol{x}+\sum
w_{hh}\cdot\boldsymbol{h}(t-i)+\boldsymbol{b}_{n})\\\ \\\
\boldsymbol{y}(t)&=lin(\sum
w_{hy}\cdot\boldsymbol{h}(t)+\boldsymbol{b}_{y})\end{split}$ (6)

Trough sequential learning based on AUV datasets, the RNN network learns to
predict the relative horizontal velocities. Activation functions $\sigma_{h}$
in the hidden layers are the key to learning the nonlinearity between the
selected inputs and predicted outputs. The network is trained based on the
renowned concept of backpropogation introduced in [24] to tune the weights and
biases. The goal of neural network training is to optimize the network
parameters so that the error function $E$ is minimized

$E=\frac{1}{2}\sum_{i}(\hat{\upsilon}_{r}(i)-\upsilon_{r}(i))^{2}$ (7)

Where $\hat{\upsilon}_{r}(i)-\upsilon_{r}(i)$ is the difference between the
actual and predicted relative velocities.

###  III-D Navigational training data

In this work, experimental DVL data and simulated velocities are used as a
reference for the supervised neural network training. Alternative approaches
may involve using acoustic positioning systems which are not prone to
cumulative integration errors [10], thus providing more accurate ground truth
measurements. However, a disadvantage with these approaches is that the
DVL/acoustic modem must be replaced with ”dummy” sensor to avoid changing the
hydrodynamic properties and net weight of the AUV. Ideally, we want to have a
reference of the AUV velocities/positions without changing the geometry and
weight. A potential solution is to use visual based (machine vision) pose
estimation relative to an assisting AUV/ROV. Machine vision has proven to be
successful in autonomous docking operations for AUVs [29],[30] and may
possibly be extended to tracking applications.

##  IV AUV Platforms $\&$ Datasets

To train and validate the neural network approach to DR navigation, AUV
datasets are needed. In this paper two AUV platforms are investigated. The
first dataset originates from sea-trials of the Tethys Long-Range AUV (LRAUV),
while the second dataset is from a MATLAB simulation of an underwater glider.

###  IV-A Tethys AUV

The Tethys Long-Range AUV [2], [14] was developed by the Monterey Bay Research
Institute (MBARI) as a research AUV with long-range capabilities. It’s
characterized as a hybrid AUV as it shares similar control actuators to
underwater gliders. This allows it to operate both in undulating glider-like
trajectories and at fixed depths. Datasets from the constellation of
underwater vehicles at MBARI is available through their public data repository
[15].

Figure 3: Tethys AUV, courtesy of MBARI

A series of missions in Monterey bay, California (U.S) were used to train and
validate the neural networks. In-situ measurements from a Microstrain 3DM-
GX5-24 IMU, Neil Brown pressure sensor and the control actuators were used as
inputs to the neural network. Ground truth relative surge and sway velocities
$[u_{r},v_{r}]^{T}$ were provided by a LinkQuest 600 KHz micro DVL. With
periodic GPS fixes the overall navigation accuracy for DVL-aided inertial DR
is 3-4 $\%$ [2]. The navigational errors arise initially from sensor noise and
random walk errors from the inertial measurement sensor.
The parameters for the IMU and DVL sensors hosted on the Tethys is presented
in table I and II respectively. The maximum operating altitude refer to bottom
lock navigation, where the AUV measures it’s velocities relative to the
seafloor. When out of range the DVL can measure the velocities relative to the
water (water lock). However, water is often considered as a moving reference
frame due to ocean currents, which introduces estimation errors depending on
the magnitude of the ocean current vector [10].

TABLE I: Microstrain 3DM-GX5-25 IMU Parameters Error | value
---|---
Accelorometer Bias Instability |  $\pm$ 0.04 mg
Gyroscope Bias Instability |  $\pm$ $8^{\circ}/h$
Attitude Accuracy | EFK - $\pm$ $0.25^{\circ}$ RMS
Heading Accuracy | EFK - $\pm$ $0.8^{\circ}$ RMS
TABLE II: LinkQuest 600 KHz Micro DVL Parameters Parameter | Value
---|---
Max Altitude | 110 meters
Min Altitude | 0.3 meters
Accuracy | 1 $\%$ $\pm$ 1 mm/s
Ping rate | 5 Hz

A minor part of the training dataset is presented in figure 4. Sensor noise
has been filtered out with low-pass filters and Gaussian smoothing. We can
observe that the AUV performs saw-tooth trajectories by using the dive plane
control surface and internal moving mass actuator. The heading is mostly
constant which indicates that the AUV is on a course keeping path governed by
a heading controller.

Figure 4: Training data Tethys AUV

###  IV-B Underwater Glider

The second dataset is gathered from a MATLAB simulation of the Seawing
underwater glider [3], [11]. An underwater glider is characterized as a
slender body with fixed wings. A variable buoyancy system (VBS) is used to
manipulate the volume that the vehicle is displacing to alter vertical motions
in the water column. Underwater gliders exploit hydrodynamic properties using
fixed wings to generate a forward motion. To control the attitude and heading
of the vehicle, an internal moving mass system is used. This typically
consists of a battery-pack that can be translated and rotated inside the
vehicle housing. A mathematical model of the glider is derived using the 6DOF
vectorial marine craft dynamics presented in Fossen [1]. State variables can
be divided into two vectors following SNAME notation [17] \- The position in
the inertial frame $\\{n\\}$ defined as
$\boldsymbol{\eta}=[x,y,z,\phi,\theta,\psi]^{T}$ and the relative velocity in
the body frame $\\{b\\}$ $\boldsymbol{\nu}_{r}=[u_{r},v_{r},w_{r},p,q,r]^{T}$.
Accordingly, a 6DOF kinematic and maneuvering model of an AUV is derived by

$\begin{split}&\boldsymbol{\dot{\eta}}=\boldsymbol{J_{\theta}}(\boldsymbol{\eta})\boldsymbol{\nu_{r}}\\\
\\\
&\boldsymbol{M}\boldsymbol{\dot{\nu_{r}}}+\boldsymbol{C}(\boldsymbol{\nu_{r}})\boldsymbol{\nu}_{r}+\boldsymbol{D}(\boldsymbol{\nu_{r}})\boldsymbol{\nu_{r}}+\boldsymbol{g}(\boldsymbol{\eta})=\boldsymbol{\tau}\end{split}$
(8)

Where $\boldsymbol{M}=\boldsymbol{M}_{rb}+\boldsymbol{M}_{A}$ and
$\boldsymbol{C}(\boldsymbol{\nu})$ = $\boldsymbol{C}_{rb}(\boldsymbol{\nu})$
\+ $\boldsymbol{C}_{A}(\boldsymbol{\nu})$ are the transnational and rotational
rigid-body dynamics with correlating added mass effects. Hydrodynamic forces
and moments are described in the damping matrix
$\boldsymbol{D}(\boldsymbol{\nu})$ and the restoring forces are defined by
$\boldsymbol{g}(\boldsymbol{\eta})$. $\boldsymbol{\tau}$ is the vector
describing the control forces and moments which acts on the vehicle.

In presence of ocean currents, the $\boldsymbol{relative}$
$\boldsymbol{velocity}$ is defined by differentiating the body-fixed
velocities to the ocean current vector
$\boldsymbol{\nu_{r}}=\boldsymbol{\nu}-\boldsymbol{\nu}_{c}$. In the Matlab
simulation we consider a two-dimensional irrotational ocean current model.
Given an absolute velocity $V_{c}=\sqrt{u_{c}^{2}+v_{c}^{2}}$ we can define
the ocean currents in the body frame as

$\boldsymbol{\nu}^{b}_{c}=\begin{bmatrix}V_{c}\cdot cos(\beta_{c}-\psi)\\\
V_{c}\cdot sin(\beta_{c}-\psi)\\\ 0\end{bmatrix}$ (9)

Many chose to simplify the ocean current model to be constant in the body-
fixed frame $\\{b\\}$, thus $\boldsymbol{\dot{\upsilon^{b}_{c}}}$ = 0 [18].
However, this only yields during course keeping. A more realistic approach is
to assume that the ocean currents are time-varying with respect to rotational
motions of the glider. Consider a skew-symmetric matrix $\boldsymbol{S}$ that
satisfies $\boldsymbol{S(x)\cdot y}=\boldsymbol{x}\times\boldsymbol{y}$ and
the angular velocities in the body frame $\boldsymbol{\omega_{b}}$, the ocean
currents can be derived as

$\dot{\boldsymbol{\nu}^{b}_{c}}=-\boldsymbol{S}(\boldsymbol{\omega_{b}})\cdot\boldsymbol{\upsilon}^{b}_{c}$
(10)

In order to model ocean currents in the 6DOF manoeuvring model, the rotational
dynamics detailing Coriolis and centripetal forces must be derived using
velocity-independent parametrizations [1]. Given a center of gravity vector
relative to the center of origin
$\boldsymbol{r}_{cg}=[x_{cg},y_{cg},z_{cg}]^{T}$ the Coriolis and centripetal
matrix can be defined as

$\boldsymbol{C}_{rb}(\boldsymbol{\nu})=\begin{bmatrix}m\cdot\boldsymbol{S}(\boldsymbol{\omega_{b}})&-m\cdot\boldsymbol{S}(\boldsymbol{\omega_{b}})\cdot\boldsymbol{S}(\boldsymbol{r}_{cg})\\\
m\cdot\boldsymbol{S}(\boldsymbol{r_{cg}})\cdot\boldsymbol{S}(\boldsymbol{\omega_{b}})&-\boldsymbol{S}(\boldsymbol{I}_{b}\cdot\boldsymbol{\omega_{b}})\end{bmatrix}$
(11)

Where $\boldsymbol{I}_{b}$ and $m$ is the vehicle inertia and total mass
respectively. As demonstrated in eq.11 the rotational dynamics is derived only
using angular velocities $\boldsymbol{\omega_{b}}=[p,q,r]^{T}$, which
satisfies the following property [1]

$\boldsymbol{M}_{rb}\boldsymbol{\nu}+\boldsymbol{C}_{rb}(\boldsymbol{\nu})\boldsymbol{\nu}=\boldsymbol{M}_{rb}\boldsymbol{\nu}_{r}+\boldsymbol{C}_{rb}(\boldsymbol{\nu}_{r})\boldsymbol{\nu}_{r}$
(12)

Glider dynamics was simulated in Simulink for a 10-hour interval. Time-varying
control inputs and ocean currents were present during the simulation to
provide variance in the training dataset. The simulated trajectories of the
glider were a combination of undulating wings-levelled motions and
turning/spiral manoeuvres. To control the attitude and heading of the glider,
two decoupled PID controllers was implemented. The measured state variables
were logged and saved to workspace during the simulation. Each training
variable holds 81 000 samples, while the validation dataset resulted in 20 000
samples per variable. The test dataset is a combination of wings-levelled
manoeuvres and spiral trajectories which differed from the trajectories used
in the training data. Hence, we can validate how well the network is
generalized to untrained glider motions.

Figure 5: Simulated glider training data

##  V Experimental and Simulation Results

This paper presents a two-folded study which consists of simulated and
experimental datasets. Collected AUV data was allocated into Matlab and
further used to develop, train, and test the neural networks. The predicted
outputs was fed into the dead-reckoning algorithm derived in eq. 3.
The Deep Learning Toolbox was used to design network architectures and perform
backpropogation training. A Scaled Conjugate Gradient (SCG) algorithm was
chosen as the training function to deal with the large AUV datasets
effectively. The SCG algorithm [24] abolish the need for line-searches as
presented in its predecessor [25] which reduces the computational load. To
improve the generalization of the neural networks, early-stopping was
introduced. Early-stopping divides the AUV dataset into training and
validation batches. The training dataset is fed into the SCG algorithm to tune
the weights and biases, while the validation data is used to monitor and
detect if occurrences of overfitting is evident [32]. If the network starts to
overfit the dataset, the training is aborted, hence the name early-stopping.

###  V-A Case Study 1 - Tethys AUV

Experimental data from the Tethys AUV was investigated in the initial study.
Data from three individual surveys was concatenated as a time-series vector
and used as training data. Datasets from another mission is used to test the
neural network on unseen data. The duration of the test trajectory is
approximately 3 hours long. The mission, illustrated in figure 6, was
conducted in shallow waters were the on-board Link Quest Micro DVL was able to
get a bottom-lock, although some samples were out of reach for the operating
altitude of the DVL sensor. Outliners in the DVL data was removed and further
filtered with Gaussian smoothing.

Figure 6: Test trajectory in Monterey Bay, California

The test trajectories consist of undulating saw-tooth motions with non-zero
angle of attack $\alpha$ as showed in figure 8. Note that periodic GPS fixes
was not accounted for in the results presented in figure 8 and 7. Two RNN
networks was developed to isolate relative surge and sway predictions. The
following table presents the training parameters used for the Tethys AUV.

TABLE III: ANN Training parameters - Tethys AUV Backpropogation Optimizer | Scaled Conjugate Gradient (SCG)
---|---
MSE Surge Network | 0.0347
MSE Sway Network | 0.00588
Early Stopping Data Division | Randomly
Early Stopping index | Training 70 $\%$, Val. 15$\%$, Test 15$\%$
Hidden layers | 3
Hidden neurons per layer | 40
Regressors per context layer | 5

The results are presented in figure 7 and 8. The blue dotted line represents
the predicted position based on estimated surge and sway velocities from the
RNN network. The orange line is the estimated position based on measured DVL
velocities.

Figure 7: Neural network aided navigation - Top view

A 3D view of the same trajectory is presented in figure 8 where depth
measurements from the pressure sensor is used for the vertical z-axis.

Figure 8: Neural network aided navigation - 3D view

The positioning error between the predicted and DVL-aided horizontal positions
was used to evaluate the performance. The positioning error is derived as

$\begin{bmatrix}N_{error}\\\ E_{error}\end{bmatrix}=\begin{bmatrix}\hskip
1.42271pt||\hskip 1.42271ptN_{est}-\hat{N}\hskip 1.42271pt||\hskip
1.42271pt\\\ \hskip 1.42271pt||\hskip 1.42271ptE_{est}-\hat{E}\hskip
1.42271pt||\hskip 1.42271pt\end{bmatrix}$ (13)

Where $\hat{N}$ and $\hat{E}$ are ground truth north and east positions
respectively.

Figure 9: Positioning Errors - Tethys AUV

The positioning error relates to an approximate displacement of 2500 meters
north and 1500 meter displacement in the east direction. Note that the ground
truth north and east positions estimated based on DVL velocities also have
estimation errors with an 3-4 $\%$ navigational accuracy [2].

###  V-B Case Study 2 - Underwater Glider

A second case study was conducted with the simulated autonomous underwater
glider. Training and test datasets were generated by the Simulink simulation
of the glider dynamics. A two-dimensional ocean current model was added in the
dynamics to create a simulated environment that reflects real ocean
conditions. Simulated ocean currents are assumed to be constant, but time-
varying during glider rotations.
Due to decoupled attitude and heading controllers from the simulated
trajectories, the interaction between the surge and sway dynamics is assumed
to be neglectable. Thus, two isolated RNN networks was developed to predict
the relative surge and sway velocities.
Table IV presents the neural network training parameters for the two RNN
networks.

TABLE IV: ANN Training parameters - Glider Backpropogation Optimizer | Scaled Conjugate Gradient (SCG)
---|---
MSE Surge Network | 0.000212
MSE Sway Network | 0.00000459
Early Stopping Data Division | Randomly
Early Stopping index | Training 70 $\%$, Val. 15$\%$, Test 15$\%$
Hidden layers | 3
Hidden neurons per layer | 50
Regressors per context layer | 5

The test trajectory presented in figure 10 consists of two spiral motions and
a wings-levelled movement. Three different ocean current scenarios was
simulated with increasing magnitude, see figure 11. The plot illustrated in
figure 10 shows the test dataset in presence of low currents -
$u_{c}=-0.05\hskip 2.84544ptm/s$ and $v_{c}=-0.002\hskip 2.84544ptm/s$.

Figure 10: Estimated NED position vs ground truth

RNN velocity predictions is presented by the blue dotted line in figure 10.
The orange line represents ground truth simulated NED positions. The glider
trajectory was simulated for 2.7 hours. The remaining simulations with
increasing ocean currents are presented in figure 11 where the north and east
positioning errors are compared by the three different scenarios. Note that
the x-axis relates to total samples with a rate of 2 Hz. Accordingly, the real
simulation time was 10 000 sec.

Figure 11: Positioning error with increasing ocean currents

During the two first simulations with low and medium strong currents the
positioning error is slightly increased. The simulation with strongest ocean
currents induced a larger divergence for the east positioning error, while a
low increase in the north error.

##  VI Conclusions And Further Work

A neural network approach to aid dead-reckoning navigation for AUVs with a
limited sensor suite was proposed in this work. Experimental data from an IMU,
a pressure sensor and control actions were gathered from sea-trials and
simulations with correlating ground truth DVL and simulated velocities. The
objective for the trained RNN networks is to complement AUV navigation in
absence of acoustic navigational instruments. Results from the proposed method
show promising potential considering a limited sensor payload. Improvements
can be made by re-initializing the DR algorithm with GPS fixes when the AUVs
are surfacing. The positioning error for the underwater glider grows slightly
with increasing magnitude of ocean current disturbances as illustrated in
figure 11. Glider positioning errors are significantly lower compared to DVL-
less traditional navigation algorithms used in commercial gliders [31].
However, sensor noise and random walk errors were not present in the simulated
IMU measurements. Further iterations of the simulated environment will focus
on adding more realistic scenarios by introducing sensor errors, GPS fixes and
vertical decomposition of the ocean currents.

Recommendations for further work include investigating alternative methods to
obtaining experimental data. Vision based pose estimation is considered a
promising candidate which avoids replacing the DVL or acoustic modem with a
”dummy” sensor. Another interesting subject is to extend the deep learning
approach to other underwater robots like remotely operated vehicles (ROVs).
Compared to under-actuated AUVs, small/miniaturized ROVs are easy to deploy
and does not require large displacements to excite the ROV dynamics, making
the experimental procedures less time-consuming.

##  VII Acknowledgments

We thank the Monterey Bay Research Institute (MBARI) for granting access to
mission data from the Tethys AUV.
This work was supported by the OASYS project funded by the Research Council of
Norway (RCN), the German Federal Ministry of Economic Affairs and Energy
(BMWi) and the European Commission under the framework of the ERA-NET Cofund
MarTERA.

## References

  * [1] Fossen, T. I. (2011). Handbook of marine craft hydrodynamics and motion control. John Wiley $\&$ Sons.
  * [2] Hobson, B. W., Bellingham, J. G., Kieft, B., McEwen, R., Godin, M., $\&$ Zhang, Y. (2012, September). Tethys-class long range AUVs-extending the endurance of propeller-driven cruising AUVs from days to weeks. In 2012 IEEE/OES autonomous underwater vehicles (AUV) (pp. 1-8). IEEE.
  * [3] Zhang, S., Yu, J., Zhang, A., $\&$ Zhang, F. (2013). Spiraling motion of underwater gliders: Modeling, analysis, and experimental results. Ocean Engineering, 60, 1-13.
  * [4] Zhang, X., He, B., Li, G., Mu, X., Zhou, Y., $\&$ Mang, T. (2020). Navnet: AUV Navigation Through Deep Sequential Learning. IEEE Access, 8, 59845-59861.
  * [5] Mu, X., He, B., Zhang, X., Song, Y., Shen, Y., $\&$ Feng, C. (2019). End-to-end navigation for autonomous underwater vehicle with hybrid recurrent neural networks. Ocean Engineering, 194, 106602.
  * [6] Skulstad, R., Li, G., Fossen, T. I., Vik, B., $\&$ Zhang, H. (2019). Dead reckoning of dynamically positioned ships: Using an efficient recurrent neural network. IEEE Robotics $\&$ Automation Magazine, 26(3), 39-51.
  * [7] Buduma, N., $\&$ Locascio, N. (2017). Fundamentals of deep learning: Designing next-generation machine intelligence algorithms. ” O’Reilly Media, Inc.”.
  * [8] Wang, P., Singh, P. K., $\&$ Yi, J. (2013, May). Dynamic model-aided localization of underwater autonomous gliders. In 2013 IEEE International Conference on Robotics and Automation (pp. 5565-5570). IEEE.
  * [9] Snyder, W., Van Uffelen, L., $\&$ Renken, M. (2019, June). Effects of Incorporating Inertial Measurements on the Localization Accuracy of the Seaglider AUV. In OCEANS 2019-Marseille (pp. 1-6). IEEE.
  * [10] Penas, A. A. (2009). Positioning and navigation systems for robotic underwater vehicles. Doctor thesis, Instituto Superior Tcnico.
  * [11] Yu, J. C., Zhang, A. Q., Jin, W. M., Chen, Q., Tian, Y., $\&$ Liu, C. J. (2011). Development and experiments of the sea-wing underwater glider. China Ocean Engineering, 25(4), 721-736.
  * [12] Graver, J. G. (2005). Underwater gliders: Dynamics, control and design.
  * [13] Li, W., Chen, M., Zhang, C., Zhang, L., $\&$ Chen, R. (2020). A Novel Neural Network-Based SINS/DVL Integrated Navigation Approach to Deal with DVL Malfunction for Underwater Vehicles. Mathematical Problems in Engineering, 2020.
  * [14] Bellingham, J. G., Zhang, Y., Kerwin, J. E., Erikson, J., Hobson, B., Kieft, B., … $\&$ Banka, A. (2010, September). Efficient propulsion for the Tethys long-range autonomous underwater vehicle. In 2010 IEEE/OES Autonomous Underwater Vehicles (pp. 1-7). IEEE.
  * [15] Alpaydin, E. (2020). Introduction to machine learning. MIT press.
  * [16] MBARI.com, ’MBARI Data Repository’, 2014. [Online]. Available: https://www.mbari.org/products/data-repository/. [Accessed: 22- Jul- 2021].
  * [17] SNAME, T. (1950). Nomenclature for treating the motion of a submerged body through a fluid. The Society of Naval Architects and Marine Engineers, Technical and Research Bulletin, (1950), 1-5.
  * [18] Børhaug, E., Pivano, L., Pettersen, K. Y., $\&$ Johansen, T. A. (2007). A model-based ocean current observer for 6DOF underwater vehicles. IFAC Proceedings Volumes, 40(17), 169-174.
  * [19] Todd, R. E., Rudnick, D. L., Sherman, J. T., Owens, W. B., $\&$ George, L. (2017). Absolute velocity estimates from autonomous underwater gliders equipped with Doppler current profilers. Journal of Atmospheric and Oceanic Technology, 34(2), 309-333.
  * [20] May, R., Dandy, G., $\&$ Maier, H. (2011). Review of input variable selection methods for artificial neural networks. Artificial neural networks-methodological advances and biomedical applications, 10, 16004.
  * [21] Sabet, M. T., Daniali, H. M., Fathi, A., $\&$ Alizadeh, E. (2017). A low-cost dead reckoning navigation system for an AUV using a robust AHRS: Design and experimental analysis. IEEE Journal of Oceanic Engineering, 43(4), 927-939.
  * [22] Rypkema, N. R., Fischell, E. M., Forrest, A. L., Benjamin, M. R., $\&$ Schmidt, H. (2018, November). Implementation of a hydrodynamic model-based navigation system for a low-cost AUV fleet. In 2018 IEEE/OES Autonomous Underwater Vehicle Workshop (AUV) (pp. 1-6). IEEE.
  * [23] Møller, M. F. (1993). A scaled conjugate gradient algorithm for fast supervised learning. Neural networks, 6(4), 525-533.
  * [24] Johansson, E. M., Dowla, F. U., $\&$ Goodman, D. M. (1991). Backpropagation learning for multilayer feed-forward neural networks using the conjugate gradient method. International Journal of Neural Systems, 2(04), 291-301.
  * [25] Rumelhart, D. E., Hinton, G. E., $\&$ Williams, R. J. (1985). Learning internal representations by error propagation. California Univ San Diego La Jolla Inst for Cognitive Science.
  * [26] González-García, J., Gómez-Espinosa, A., Cuan-Urquizo, E., García-Valdovinos, L. G., Salgado-Jiménez, T., $\&$ Cabello, J. A. E. (2020). Autonomous underwater vehicles: localization, navigation, and communication for collaborative missions. Applied sciences, 10(4), 1256.
  * [27] Li, W., Zhang, L., Sun, F., Yang, L., Chen, M., $\&$ Li, Y. (2015). Alignment calibration of IMU and Doppler sensors for precision INS/DVL integrated navigation. Optik, 126(23), 3872-3876.
  * [28] Javanmard, E., Mansoorzadeh, S., Pishevar, A., $\&$ Mehr, J. A. (2020). Determination of drag and lift related coefficients of an auv using computational and experimental fluid dynamics methods. Transactions of the Royal Institution of Naval Architects Part A: International Journal of Maritime Engineering, 162(Part A2), A177-A191.
  * [29] Liu, S., Xu, H., Lin, Y., $\&$ Gao, L. (2019). Visual navigation for recovering an AUV by another AUV in shallow water. Sensors, 19(8), 1889.
  * [30] Li, Y., Jiang, Y., Cao, J., Wang, B., $\&$ Li, Y. (2015). AUV docking experiments based on vision positioning using two cameras. Ocean Engineering, 110, 163-173.
  * [31] Woithe, H. C., Boehm, D., $\&$ Kremer, U. (2011, September). Improving slocum glider dead reckoning using a doppler velocity log. In OCEANS’11 MTS/IEEE KONA (pp. 1-5). IEEE.
  * [32] Prechelt, L. (1998). Early stopping-but when?. In Neural Networks: Tricks of the trade (pp. 55-69). Springer, Berlin, Heidelberg.
  * [33] Rogne, R. H., Bryne, T. H., Fossen, T. I., $\&$ Johansen, T. A. (2020). On the usage of low-cost mems sensors, strapdown inertial navigation, and nonlinear estimation techniques in dynamic positioning. IEEE Journal of Oceanic Engineering, 46(1), 24-39.