# Deep Reinforcement Learning for Localizability-Enhanced Navigation in
Dynamic Human Environments

Yuan Chen, Quecheng Qiu, Xiangyu Liu, Guangda Chen, Shunyi Yao, Jie Peng,
Jianmin Ji∗ and Yanyong Zhang∗ Authors are with University of Science and
Technology of China, Hefei 230026, China. Jianmin Ji and Yanyong Zhang are
corresponding authors<EMAIL_ADDRESS>
###### Abstract

Reliable localization is crucial for autonomous robots to navigate efficiently
and safely. Some navigation methods can plan paths with high localizability
(which describes the capability of acquiring reliable localization). By
following these paths, the robot can access the sensor streams that facilitate
more accurate location estimation results by the localization algorithms.
However, most of these methods require prior knowledge and struggle to adapt
to unseen scenarios or dynamic changes. To overcome these limitations, we
propose a novel approach for localizability-enhanced navigation via deep
reinforcement learning in dynamic human environments. Our proposed planner
automatically extracts geometric features from 2D laser data that are helpful
for localization. The planner learns to assign different importance to the
geometric features and encourages the robot to navigate through areas that are
helpful for laser localization. To facilitate the learning of the planner, we
suggest two techniques: (1) an augmented state representation that considers
the dynamic changes and the confidence of the localization results, which
provides more information and allows the robot to make better decisions, (2) a
reward metric that is capable to offer both sparse and dense feedback on
behaviors that affect localization accuracy. Our method exhibits significant
improvements in lost rate and arrival rate when tested in previously unseen
environments.

##  I INTRODUCTION

The capability of a robot to achieve reliable localization is a fundamental
skill for autonomous navigation systems. In many applications, incorrect
localization results can lead to bad effects on navigation performance and
safety. For example, a delivery robot may fail to deliver its goods to the
correct location, and a shopping guide robot may fall down an escalator when
it runs out of its work region. Additionally, the effect can be much more
severe for search and rescue robots, where human lives may be at risk.
However, the robot will inevitably be subjected to localization errors or
uncertainties due to various factors, such as sensor noise, environmental
changes, and mechanical drift. These errors can accumulate over time in
complex dynamic environments and lead to _navigation lost_ , where the robot
loses track of its position and orientation in the environment [1]. To
overcome these issues, some navigation methods plan paths not only to be
collision-free but also to provide access to a sensor stream that can obtain
more reliable localization along the path. However, most of these
localizability-enhanced methods require prior knowledge, such as predefined
patterns of landmarks [2, 3] or pre-built maps that quantify the
localizability of each region in the environment with a specific metric (i.e.,
the entropy or the pose estimation covariance) [4, 5, 6]. Based on this prior
knowledge, these methods can achieve considerable performance in restricted
environments. However, it is challenging to generalize these methods well to
unforeseen scenarios with unfamiliar classes of landmarks or adapt these
methods well to dynamic changes that cannot be recorded in pre-built maps.

This paper aims to develop a novel localizability-enhanced navigation approach
in dynamic human environments without requiring extensive prior knowledge. To
achieve this goal, the main challenges we need to overcome are twofold: (1)
how to extract features conducive to localization from raw sensing data and
assign them appropriate weights for planning and (2) how to interact with
pedestrians and balance navigation efficiency, obstacle avoidance safety, and
localization robustness.

To address the above challenges, we propose a novel approach that utilizes
deep reinforcement learning (DRL). Our approach takes advantage of Deep Neural
Network (DNN) to automatically extract geometric features from 2D laser data
that are helpful for localization. The DNN-based policy adapts to the scene
online and assigns appropriate importance to the extracted geometric features,
achieving a balance between efficiency, safety, and localization robustness
through trial and error in Reinforcement Learning (RL). To the best of our
knowledge, this is the first navigation method that enhances localizability in
dynamic human environments without relying on predefined landmark features or
localizability maps. To facilitate the agent to learn the desired policy, we
meticulously designed the representation of the state space and reward
function. We augment the state space by incorporating pedestrian maps [7] to
track the movement of surrounding pedestrians and include pose estimation
variance to measure the quality of localization results. This allows the robot
to make more informed decisions that consider the dynamic changes in the
environment and the confidence in the localization results. Moreover, we
define a reward function that can provide sparse feedback on _navigation lost_
and dense feedback on behaviors affecting localization accuracy. It is
important to note that the ground truth of the robot pose is only available
during training, and the robot must rely on estimated poses during execution.
We also design specialized pedestrian scenarios and policy network
architectures for training and successfully train a localizability-enhanced
navigation policy that can constantly obtain more reliable localization
results.

For the purpose of evaluating the effectiveness of our proposed approach, we
established a set of previously unseen environments in a simulation as a
testing benchmark. A range of metrics was utilized to assess the performance
of our method. The experimental results show that our approach achieves a loss
rate that is one order of magnitude lower than that of other navigation
methods while simultaneously increasing the arrival rate by $25\%$. This
performance improvement comes at the cost of an $9\%$ increase in moving
distance. Additionally, we provide a visualization of the activation of the
first laser feature layer to demonstrate the efficacy of our approach in
automatically extracting features. Finally, we conducted a real-world
deployment of our trained policy on a physical robot to show its practicality.
Our main contributions are summarized as follows:

  * •

We propose a DRL-based navigation method that enhances localizability in
dynamic human environments.

  * •

To learn the behaviors that enhance localizability, it is essential to
incorporate information about localization quality into the robot’s state and
use a reward function that gives frequent feedback on the behaviors that
affect localization.

  * •

We visualize the activation of the first laser feature layer to show the
geometry features extracted to make localizability-enhanced decisions.

  * •

We verify the generalization of our method through simulation testing in
various scenarios and the deployment of a physical robot.

##  II Related Work

The _navigation lost_ problem occurs when the robot’s localization uncertainty
or error becomes too large during navigation, causing the robot to lose track
of its position and orientation in the environment [1]. Past solutions to this
problem have been primarily passive, assuming that the robot’s motion and
sensor direction cannot be controlled. These methods selectively use the
sensor stream to minimize localization uncertainty or error [8, 9, 10].
However, if the localization routine has partial or full control over the
robot’s movement and sensor direction, it can increase the efficiency and
robustness of localization [11, 12], particularly in dynamic environments
where moving people may occlude the sensors’ view [13, 14]. These active
localization algorithms focus on the global localization problem where the
initial position of the robot is unknown and usually works when _navigation
lost_ occurs.

With the intention of avoiding _navigation lost_ , previous work such as [4,
5, 6] consider the localizability in path planning. They generate a
localizability map that contains the information content of each position in
the environment. Path planning is then optimized over both conventional costs,
such as distance or time, and localization certainty using the localizability
map. Based on prior knowledge, they can make long-range plans with a global
view but have trouble adapting well to dynamic changes through real-time
perception.

To make use of real-time perception, other prior knowledge types have been
introduced. Prediction-based methods [15, 16] predict the future pose
estimated error and uncertainty based on specific pose propagation rules and
then use the predicted error or uncertainty as one of the optimization items
for path planning. Landmark-based methods recognize particular classes of
landmarks from sensor data with a pre-designed pattern [2] or DNN-based
semantic segmentation [3]. They can evaluate candidate paths based on landmark
concentration or the category of the landmark in view. However, these methods
rely heavily on prior knowledge and are often difficult to generalize to
unforeseen environments.

For the purpose of improving generalizability, DRL has been applied to
localizability-enhanced navigation. Taking visual images as inputs, [17, 18]
present semantic-aware path-planning pipelines in which RL agents utilize
semantic masks to generate actions. To the best of our knowledge, there is
currently only one DRL-based localizability-enhanced navigation algorithm that
directly feeds raw sensing data (used for localization) to the RL agent, which
is [19]. However, [19] does not consider the interaction with pedestrians,
which limits their usefulness in real-world scenarios. As the implementation
details of [19] are not presented, we did not compare them with us in our
experiments.

Figure 1: The architecture of the policy network combined with a third-party
localization module. The input of the network includes the pedestrian map
$\mathbf{o}_{t}^{ped}$, the scan measurements $\mathbf{o}_{t}^{scan}$, and the
belief over the relative goal position $\mathbf{b}_{t}^{goal}$, which is
calculated by the location of the goal $\mathbf{l}_{g}$, and the belief over
the robot current pose $\mathbf{b}_{t}^{pose}$. $\mathbf{b}_{t}^{pose}$ is
estimated by the localization module with scan measurements, odometer data,
global map, and $\mathbf{b}_{t-1}^{pose}$. The network uses a pedestrian map
encoder, a scan encoder, and a target encoder to process
$\mathbf{o}_{t}^{ped}$, $\mathbf{o}_{t}^{scan}$, and $\mathbf{b}_{t}^{goal}$,
respectively. Then, the network concatenates the outputs of the encoders, feed
them into two fully connected (FC) layers, and gets a pair of linear and
angular velocities, i.e, $\mathbf{a}_{t}$.

##  III Problem Formulation

The problem of localizability-enhanced navigation in dynamic human
environments can be viewed as a sequential decision-making problem that aims
to reach a target location $\mathbf{l}_{g}$ while satisfying both collision-
free and localization error constraints. At each time step $t$, given a frame
sensing data $\mathbf{o}^{sensor}_{t}$, odometer data $\mathbf{o}^{odom}_{t}$,
and the probability distribution of the robot pose estimated at the previous
time step $\mathbf{b}^{pose}_{t-1}$, a third-party localization module $f$
estimates the probability distribution of the robot pose at the current time
step $\mathbf{b}^{pose}_{t}$, i.e.,

$\mathbf{b}^{pose}_{t}=f(\mathbf{o}^{sensor}_{t},\mathbf{o}^{odom}_{t},\mathbf{b}^{pose}_{t-1}).$

The probability distribution of the goal position in the robot coordinate
system $\mathbf{b}^{goal}_{t}$ can be computed with $\mathbf{b}^{pose}_{t}$
and $\mathbf{l}_{g}$. Then, a navigation policy $\pi_{\theta}$ provides an
action command based on the motion data of the nearby pedestrians
$\mathbf{o}^{ped}_{t}$, $\mathbf{o}^{sensor}_{t}$, and $\mathbf{b}^{goal}_{t}$
as follows:

$\mathbf{a}_{t}=\pi_{\theta}(\mathbf{o}^{ped}_{t},\mathbf{o}^{sensor}_{t},\mathbf{b}^{goal}_{t}),$

where $\theta$ is the model parameters. Our goal is to find a policy to
minimize the expectation of the arrival time while accounting for the robot’s
safety by ensuring that it does not collide or get lost. We define getting
lost as:

$\|\mathbf{p}^{gt}_{t}-\mathop{arg\max}_{\mathbf{p}}\mathbf{b}^{pose}_{t}(\mathbf{p})\|>\varepsilon,$

where $\mathbf{p}^{gt}_{t}$ denotes the ground truth of the robot pose at $t$,
$\mathop{arg\max}_{\mathbf{p}}\mathbf{b}^{pose}_{t}(\mathbf{p})$ denotes the
most likely pose based on $\mathbf{b}^{pose}_{t}$, and $\varepsilon$ is a
threshold. Note that instead of minimizing the pose error, we only limit the
pose error below a threshold for navigation efficiency.

##  IV Approach

We begin this section by introducing the key ingredients of our reinforcement-
learning framework. Next, we describe the details of the architecture of our
policy network, which works with a third-party localization module. Finally,
we present the algorithm and the scenarios for training, along with the
uncertainty settings in the simulation.

###  IV-A Reinforcement Learning Ingredients

####  IV-A1 _State space_

A state $\mathbf{b}_{t}$ at time step $t$ consists of the probability
distribution of the relative goal position $\mathbf{b}_{t}^{goal}$, the
readings of the 2D laser range finder $\mathbf{o}_{t}^{scan}$, and the three-
channel pedestrian map $\mathbf{o}^{ped}$. In specific,
$\mathbf{b}_{t}^{goal}=(x_{t},y_{t},a_{t},\sigma_{t}^{x},\sigma_{t}^{y},\sigma_{t}^{a})$
consists of the estimated relative target position $(x_{t},y_{t})$, the
estimated relative target orientation $\alpha_{t}$, and
$(\sigma_{t}^{x},\sigma_{t}^{y},\sigma_{t}^{\alpha})$ are respectively the
estimated variance of $x_{t}$, $y_{t}$ and $\alpha_{t}$, which indicate the
localization module’s confidence (or uncertainty) in pose estimation [20].
$\mathbf{o}_{t}^{scan}$ are the measurements of the last frame from a
180-degree laser scanner that provides 720 distance values per scanning (i.e.,
$\mathbf{o}_{t}^{scan}\in\mathbb{R}^{1\times 720}$). $\mathbf{o}_{t}^{ped}$ is
a local grid map with three channels (i.e.,
$\mathbf{o}_{t}^{ped}\in\mathbb{R}^{3\times 48\times 48}$). As specified in
[7], the first channel uses occupied grids to specify the locations and the
shapes of surrounding pedestrians. The rest two channels use the value in the
occupied grids to specify the speeds of corresponding pedestrians for the
x-axis and y-axis, respectively. These motion data can be accessed directly in
the simulator and estimated in the real world with human tracking.

####  IV-A2 _Action space_

The action space is a set of permissible velocities in discrete space. The
action of a differential robot includes the translational and rotational
velocity, i.e., $\mathbf{a}_{t}=(v_{t},\omega_{t})$. In this work, considering
the real robots’ kinematics and the real-world applications, we set the range
of the translational velocity $v\in\\{0.0,0.2,0.4,0.6\\}$ and the rotational
velocity in $w\in\\{-0.9,-0.6,-0.3,0.0,0.3,0.6,0.9\\}$.

####  IV-A3 _Reward function_

The reward $\rho_{t}$ consists of six components, i.e.,

$\displaystyle\rho_{t}$
$\displaystyle=\rho^{app}_{t}+\rho^{pose}+\rho^{arr}_{t}+\rho^{col}_{t}+\rho_{t}^{lost}+\rho^{step}_{t},$
$\displaystyle\rho^{app}_{t}$
$\displaystyle=k_{a}\left(\left\|\mathbf{l}_{t-1}^{gt}-\mathbf{l}_{g}\right\|-\left\|\mathbf{l}^{gt}_{t}-\mathbf{l}_{g}\right\|\right)/\left(1+\sigma_{t}^{x}+\sigma_{t}^{y}+\sigma_{t}^{a}\right),$
$\displaystyle\rho^{pose}_{t}$
$\displaystyle=k_{p}\left(\left\|\mathbf{p}_{t-1}-\mathbf{p}_{t-1}^{gt}\right\|-\left\|\mathbf{p}_{t}-\mathbf{p}_{t}^{gt}\right\|\right),$
$\displaystyle\rho^{arr}_{t}$
$\displaystyle=\left\\{\begin{array}[]{ll}r_{arr}&\text{ if
}\left\|\mathbf{l}_{t}-\mathbf{l}_{g}\right\|<\varepsilon_{a},\\\ 0&\text{
otherwise,}\end{array}\right.$ $\displaystyle\rho^{coll}_{t}$
$\displaystyle=\left\\{\begin{array}[]{ll}r_{col}&\text{ if collision},\\\
0&\text{ otherwise,}\end{array}\right.$ $\displaystyle\rho^{lost}_{t}$
$\displaystyle=\left\\{\begin{array}[]{ll}r_{lost}&\text{
if}\left\|\mathbf{l}_{t}-\mathbf{l}_{t}^{gt}\right\|>\varepsilon_{l}\text{,
or}\left|{yaw}_{t}-yaw_{t}^{gt}\right|>\varepsilon_{yaw},\\\ 0&\text{
otherwise,}\end{array}\right.$ $\displaystyle\rho^{step}_{t}$
$\displaystyle=r_{step},$

where $\mathbf{p}_{t}$ denotes the estimated pose of the robot at the time
step $t$, $\mathbf{p}_{t}$ consists of the estimated location $\mathbf{l}_{t}$
and the estimated yaw angle $yaw_{t}$, $\mathbf{p}_{t}^{gt}$ denotes the
ground truth pose of the robot at the time step $t$, $\mathbf{p}_{t}^{gt}$
also consists of $\mathbf{l}_{t}^{gt}$ and $yaw_{t}^{gt}$, $\mathbf{l}_{g}$
denotes the location of the goal, $k_{a}$, $k_{p}$, $r_{arr}$, $r_{col}$,
$r_{lost}$, $r_{step}$, $\varepsilon_{a}$, $\varepsilon_{l}$, and
$\varepsilon_{yaw}$ are hyperparameters.

The reward functions defined above provide both sparse and dense feedback for
agents. The sparse feedback includes $\rho^{arr}_{t}$ for rewarding the agents
arriving at the target, $\rho_{col}$ for penalizing collisions, and
$\rho_{lost}$ for penalizing getting lost, these two types of feedback
regulate the robots to satisfy the hard constraints. On the other hand, the
dense feedback encourages the robots to approach the goal when the
localization uncertainty is low (i.e., $\rho^{app}_{t}$) and to increase the
localization accuracy (i.e., $\rho^{pose}_{t}$), and penalize the time
consumption (i.e., $\rho^{step}$), this feedback adjusts the behavior of the
robots to satisfy soft constraints. As hard constraints are often more crucial
than soft constraints, we assign large weights for hard constraints and small
weights for soft constraints, i.e, $k_{a}=200$, $k_{p}=400$, $r_{arr}=500$,
$r_{col}=-800$, $r_{lost}=-500$, $r_{step}<-5$, $\varepsilon_{a}=0.5m$,
$\varepsilon_{l}=2.0m$, and $\varepsilon_{yaw}=0.25\pi$.

(a) Hybrid

(b) Sparse

(c) Corridor

(d) Hall

(e) Irregular

Figure 2: We consider five scenarios in our experiments and their global top
views are shown as above, where the red lines connect the robot to the target,
blue dots denote the navigation paths of the robot, and green spots denote the
legs’ trajectories of pedestrians, the black blocks denote static obstacles,
the yellow boxes denote the random region of the starting position, and the
grey boxes denote the random region of the target position.

###  IV-B Network Architecture

As shown in Fig. 1, we define a deep convolutional neural network that embeds
a pedestrian map encoder, a scan encoder, and a target encoder to compute the
probability of choosing each action. In specific, the pedestrian map encoder
employs three convolutional layers and a fully connected (FC) layer to extract
the high-level features from the pedestrian maps. The first layer convolves 64
two-dimensional filters with kernel size = $3\times 3$ (Conv $3\times 3$),
stride = 1 over the three channels of the pedestrian maps followed by ReLU
nonlinearities (which are used by every layer in the entire network except the
last layer). The second and third layers have the same kernel size and stride
as the first layer, but the second layer has 128 filters and the third layer
has 256 filters. The last FC layer has 512 rectifier units. The scan encoder
employs two convolutional layers and an FC layer to extract the high-level
features from scan measurements. The first layer convolves 32 one-dimensional
filters (Conv1D) with kernel size = 5, stride = 3 over the scans. The second
layer convolves 32 one-dimensional filters with kernel size = 3, stride = 2.
The FC layer has 512 rectifier units. The target encoder has only one FC layer
with 64 rectifier units, which transforms the estimated relative position and
uncertainty of the target into a high-dimensional feature vector. Then the
network combines both the encoded vectors and feeds them to an FC layer with
512 units. At last, the network uses a softmax layer with 28 units to choose
the pair of linear and angular velocities from corresponding values.

###  IV-C Training under Localization Uncertainty

####  IV-C1 _Training algorithm_

To train the complex localizability-enhanced policy with multiple optimization
items in the reward function, we need a policy gradient method that strikes a
balance between ease of tuning, sample complexity, and sample efficiency.
Therefore, we adopt proximal policy optimization (PPO) [21]. The state-value
function in PPO is approximated with a neural network, which has the same
architecture as our policy network, except that it has only one linear
activation unit in its last layer.

####  IV-C2 _Training Scenario_

We train the robot in our customized simulator [22]. As shown in Fig. 2(a),
the _hybrid_ training scenario has a nonuniform density of spatial features
and diverse pedestrian distribution. The scenario’s size is $33\,m\times
20\,m$. The pedestrians in the simulator are driven by optimal reciprocal
collision avoidance (ORCA) [23], or social force model (SFM) [24] as [7]. The
starting and target positions of pedestrians have small random ranges, which
can generate various collision avoidance situations.

####  IV-C3 _Uncertainty settings_

Since we only need to obtain the relative position of the target point rather
than the global position in our task, both map-based and mapless localization
algorithms can be used. The uncertainty of localization is mostly derived from
the uncertainty of perception and travel distance measurement. To better
simulate the real world, we add perception uncertainty to the simulated laser
range scanner and add the distance measurement uncertainty to the simulated
odometer.

The perception uncertainty is simulated by _Beam range finder model_ [25],
which can be parameterized with
$\\{z_{hit},z_{short},z_{max},z_{rand},\sigma_{hit}\\}$. _Beam range finder
model_ incorporates four types of measurement errors: small measurement noise
(with weight $z_{hit}$ and Gaussian noisy
$\mathcal{N}\left(0,\sigma_{\mathrm{hit}}^{2}\right)$), errors due to
unexpected objects (with weight $z_{short}$), errors due to failures to detect
objects (with weight $z_{max}$), and random unexplained noise (with weight
$z_{rand}$). As we have pedestrians in the simulation, we can directly
generate errors from unexpected objects. Therefore, we set $z_{short}=0$,
$z_{hit}=0.98$, $z_{max}=0.01$, $z_{rand}=0.01$, and
$\sigma_{\mathrm{hit}}=0.02$. We assume that the odometer is calibrated well
and has no systematic error. Hence, we only add non-systematic errors to the
simulated odometer with dynamics randomization as [26, 27] by multiplying the
$a_{t}$ with $\mathcal{N}\left(1,0.01\right)$.

Table I: Performance metrics (as mean/standard deviation) evaluated for different methods. Method | #metrics |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
| AR | CR | LR | SR |  $\bar{t}$ (mean/std) |  $\bar{d}$ (mean/std) |  $\overline{e_{p}}$ (mean/std) |  $\overline{e_{\alpha}}$ (mean/std) |  $\overline{\sum_{v}}$ (mean/std)
DWA | 0.631 | 0.322 | 0.035 | 0.012 | 23.82/7.806 | 11.19/1.80 | 0.3083/0.3524 | 0.1257/0.1678 | 0.5144/2.8543
DRL_laser | 0.542 | 0.395 | 0.057 | 0.006 | 29.88/10.76 | 13.09/2.83 | 0.3265/0.6434 | 0.1928/0.2169 | 0.3018/2.6280
DRL_laser+ped | 0.641 | 0.153 | 0.054 | 0.152 | 30.76/12.79 | 16.18/5.57 | 0.3261/0.6654 | 0.2697/0.2277 | 0.2951/2.9579
LNDRL(ours) | 0.892 | 0.104 | 0.003 | 0.001 | 22.86/5.26 | 12.21/2.15 | 0.3071/0.2385 | 0.1887/0.1775 | 0.1062/1.1299
-variance | 0.790 | 0.188 | 0.021 | 0.001 | 23.03/3.87 | 12.49/1.79 | 0.3072/0.3821 | 0.2338/0.2189 | 0.2301/1.8215
-reward | 0.743 | 0.227 | 0.030 | 0.000 | 23.54/4.97 | 12.69/2.25 | 0.3725/0.5212 | 0.1873/0.2111 | 0.3153/2.7664

##  V Experiments

In this section, we will begin by briefly discussing our implementation
details. Next, we will define the evaluation metrics and introduce the testing
scenarios. Afterward, we will evaluate the performance of our method in
various simulation scenarios, comparing it with other approaches. To
specifically illustrate the consideration of localizability, we will provide
examples of trajectories generated by our method. In addition, we will display
the activation of the first laser feature layer to demonstrate the extracted
geometry features that contribute to enhanced localizability decisions.
Ablation studies will then be conducted to show the role of our augmented
state and reward metric. Finally, we will provide details regarding our
deployment of the trained policy on a physical robot.

###  V-A Implementation Details

We use adaptive Monte Carlo localization (AMCL) 111http://wiki.ros.org/amcl
for localization, as it can efficiently quantify the uncertainty of
localization results with a covariance matrix, which is very important for
training. AMCL receives $10\,Hz$ laser data with a maximum range of $12m$ and
$10\,Hz$ odometer data in the simulation, consistent with the sensors we use
in the real world. The detailed AMCL parameters can be found in the video
link. We train the localizability-enhanced policy following the PPO algorithm
with the same hyperparameters as [7]. Both the policy and value networks are
implemented in Pytorch [28] and trained with the Adam optimizer [29] on a
computer with AMD R9 5950X CPU and Nvidia RTX 3090 GPU. It takes around 16
hours to run about $1.2\times 10^{6}$ steps in PPO when the networks converge
well in the training scenario.

###  V-B Metrics and Testing Scenarios

We use the following metrics to evaluate the performance:

  * •

Arrival rate (AR): the ratio of the episodes that end with the robot reaching
its target without any collisions.

  * •

Collision rate (CR): the ratio of the episodes that collisions occur.

  * •

Lost rate (LR): the ratio of the episodes that the robot gets lost (the
threshold is defined in the reward function).

  * •

Stuck rate (SR): the ratio of the episodes exceeding the maximum episode
length.

  * •

Reach time ($\bar{t}$): the average time required for the robot to
successfully reach its target.

  * •

Moving distance ($\bar{d}$): the average moving distance required for the
robot to successfully reach their targets.

  * •

Positioning error ($\overline{e_{p}}$): the average positioning error after
each action is performed.

  * •

Angular error ($\overline{e_{\alpha}}$): the average yaw error after each
action is performed.

  * •

Sum of pose variance ($\overline{\sum_{v}}$): the average sum of
$\sigma^{x}_{t}$, $\sigma^{y}_{t}$ and $\sigma^{\alpha}_{t}$ after each action
is performed.

Figure 2 illustrates five different scenarios that are considered in our
experiments. Please note that we only use _hybrid_ scenario as the training
environment and evaluate the performance in these five scenarios. The
pedestrians in the testing scenarios are driven by the same algorithm as the
training scenario, and also have small random regions for choosing start and
target locations. As _hybrid_ scenario is introduced in section IV-C2, we only
introduce the other four scenarios here:

  * •

Sparse scenario is $33\,m\times 20\,m$ with 4 pedestrians in it. It seems
extremely simple, however, the robot may get lost in the middle open space due
to the lack of spatial features.

  * •

Corridor scenario is $30\,m\times 20\,m$ with 22 pedestrians in it. The robot
needs to interact with dense crowds of people heading in the same or the
opposite direction.

  * •

Hall scenario is $20\,m\times 20\,m$ with 18 pedestrians in it. The robot
needs to interact with pedestrians in all directions.

  * •

Irregular scenario is $20\,m\times 20\,m$ with irregular obstacles in it. The
robot needs to adapt to unseen shapes of obstacles.

###  V-C Simulation Experiments

####  V-C1 _Baselines_

We compare our LNDRL (Lacalizability-enhanced Navigation with Deep
Reinforcement Learning) with a traditional collision avoidance method and two
DRL-based methods, all of which don’t consider localizability. For the sake of
fair comparison, all the methods receive the same laser data and use the same
localization algorithm with the same parameters to estimate the robot pose. To
handle the unstructured raw laser data, we set Dynamic Window Approach (DWA)
[30] as the classical benchmark. DRL_laser is a DRL-based method with only raw
laser data as the observation, its network architecture refers to [31], which
directly predicts the motions of nearby pedestrians from three consecutive
frames of laser data. DRL_laser+ped is an enhanced version of DRL_laser, which
adds pedestrian maps to the inputs of the network as LNDRL. The main
difference between the two DRL-based methods and our LNDRL is that they assume
localization is perfectly accurate. In specific, there is no $\sigma_{t}^{x}$,
$\sigma_{t}^{y}$, and $\sigma_{t}^{a}$ in their state space, and no
$\rho_{t}^{pose}$, $\rho_{t}^{lost}$ in their reward function. The other
settings for the two DRL-based methods are the same as LNDRL.

####  V-C2 _Evaluation results_

For each method, we performed 500 tests under every scenario and calculate the
mean of each metric. The test results are summarized in the top half of Table
I. LNDRL outperforms other methods in terms of lost rate and has the highest
arrival rate. However, behaviors that prioritize localization can reduce
navigation efficiency, resulting in a longer moving distance. Thus, LNDRL’s
performance in terms of this metric is not the best. Nevertheless, LNDRL
strikes a balance between localization robustness and navigation efficiency,
as it achieves the shortest reach time. We found that sometimes sacrificing
part of the navigation efficiency for localization robustness can help the
robot reach the target faster. Inaccurate localization can cause the robot to
fail to reach the target and increase the stuck rate. A possible strategy is
to temporarily trade off localization accuracy for efficiency as long as the
pose error remains within acceptable thresholds. The penalty for increased
pose error can be compensated later by reducing the pose error. This is why
LNDRL does not significantly outperform the other methods in terms of average
positioning and angular errors but has relatively low standard deviations on
these two metrics. Moreover, LNDRL achieves the best performance in the sum of
pose variance metric, indicating that it maintains a more confident
localization algorithm most of the time.

In comparison with other methods, DWA achieves the shortest moving distance
and smallest angular error. We attribute this to two factors: (1) DWA has
little randomness and can head straight to the target in the exact direction.
In contrast, due to the randomness in training and execution, DRL policies in
our experiments may have slight oscillations in the travel direction. (2) DWA
has weak intentions to avoid pedestrians, resulting in fewer turns and lower
angular error. However, this strategy comes at the cost of a higher collision
rate, which is similar to that of DRL_laser. Although DRL_laser has a smaller
angular error, it has a higher collision rate than DRL_laser+ped, indicating
the benefits of incorporating pedestrian maps to avoid pedestrians.

####  V-C3 _Example trajectories_

(a) DRL_laser+ped in sparse

(b) LNDRL in sparse

(c) DRL_laser+ped in corridor

(d) LNDRL in corridor

Figure 3: Robot’s trajectories generated by DRL_laser+ped and LNDRL (ours) in
the _sparse_ and _corridor_ scenarios. The blue spots denote the trajectories
of the robot and other spots denote the trajectories of pedestrians. The line
charts below show the positioning and angular error at each time step on the
corresponding trajectories. We can observe that LNDRL prefers the routes with
more spatial features to reach the target and constrains its pose error at the
lower level.

As shown in Fig. 3, we illustrate the robot’s trajectories generated by LNDRL
and DRL_laser+ped (which is the method that achieves the highest arrival rate
among the other three methods). We can observe in the _sparse_ scenario that
our LNDRL drives the robot moves along a path towards the goal with more
geometry features than DRL_laser+ped and limits the positioning and angular
error in a relatively low range. In the _corridor_ scenario, our LNDRL takes a
route close to the wall to reach the target, which helps the robot avoid being
surrounded by pedestrians DRL_laser+ped and get more accurate localization. An
interesting detail is that after the robot finishes its route along the wall,
instead of heading straight for the end, it first takes a look in the
direction where the geometry features are denser, which significantly reduces
its localization error.

####  V-C4 _Activation visualization_

(a) Scenario

(b) Raw laser data

(c) Activation of the first laser feature layer

Figure 4:  How the first layer of scan encoder “sees” the irregular obstacles.
(a) an example scenario and the (b) raw laser data perceived by the robot in
it. (c) Activation of the first laser feature layer with 32 1D filters.

We create a visual representation of the numerical data output by the first 1D
convolutional layer by assigning colors to different values and mapping each
channel to its corresponding position in the original laser data. In Fig.
4(c), the activation maps clearly show the corners of obstacles, which we
believe to be important features for localization. Additionally, we observed
features related to the distance from the robot, which may be useful for
obstacle avoidance, since close obstacles are more urgent to avoid.

####  V-C5 _Ablation studies_

We conduct ablation studies to show the role of our augmented state and reward
metric. In specific, two experiments were conducted: one with the removal of
$\sigma_{t}^{x}$, $\sigma_{t}^{y}$, and $\sigma_{t}^{a}$ from the state space,
denoted as ‘-variance’, and another with the removal of $\rho_{t}^{pose}$
(which is the most distinct item in the reward function compared to other DRL
navigation methods) from the reward function, denoted as ‘-reward’. The
results of the ablation study can be found in the bottom half of Table I,
which reveals that both the incorporation of localization quality and the
frequent feedback on localization are important. It is worth noting that while
both components are related to localization, their removal significantly
increases the collision rate. Our guess for this is that, when less confident
about localization, the robot sometimes regards the target as unreachable and
thus tends to end the episode by collision so as to reduce the penalty in time
consumption.

###  V-D Real World Experiments

Figure 5: (a) We compare LNDRL with DWA in a long corridor scenario
($80m\times 22m$), the five-pointed star represents the target point. (b) We
use LNDRL to avoid pedestrians.

To deploy the trained LNDRL model in a real-world environment, we equip a
TurtleBot2 with an RPLIDAR-A2M12 laser range scanner and use a Realsense D455
depth camera as the pedestrian tracking sensor. We utilize the SPENCER people
tracking framework [32] to track the surrounding pedestrians and generate a
pedestrian map. To estimate the robot’s pose, we apply the Adaptive Monte
Carlo Localization (AMCL) algorithm. We then evaluate LNDRL’s performance in a
long corridor scenario where glass walls are on one side and sofas on the
other. As illustrated in Fig. 5, our approach enables the robot to navigate
close to the sofas and thus more fully use geometric features for more
accurate localization. Furthermore, LNDRL can also avoid pedestrians in the
real world. The video can be found at https://github.com/YohannnChen/LNDRL.

##  VI Conclusions

In this paper, we present a novel approach to localizability-enhanced
navigation in dynamic human environments using deep reinforcement learning.
Our proposed method addresses the limitations of prior navigation methods by
automatically extracting and assigning importance to geometric features that
help with laser localization. It balances navigation efficiency, obstacle
avoidance safety, and localization robustness by leveraging trial and error.
We have also introduced two techniques, an augmented state representation and
a reward metric, that facilitate the planner’s learning. Our experimental
results demonstrate significant improvements in lost rate and arrival rate
across various simulation environments. Furthermore, our method generalizes
well for real-world deployment on a physical robot.

Our limitations are mainly in the following two aspects. Firstly, we have not
considered the acceleration limits of physical robots, which could lead to a
gap between simulation and real-world experiments. Secondly, more stringent
collision-free guarantees are needed to be provided to further improve safety
for navigation in human environments. Therefore, we will try to solve these
two problems in our future work.

## References

  * [1] D. Fox, W. Burgard, F. Dellaert, and S. Thrun, “Monte carlo localization: Efficient position estimation for mobile robots,” _AAAI/IAAI_ , vol. 1999, no. 343-349, pp. 2–2, 1999.
  * [2] Z. Zhang and D. Scaramuzza, “Perception-aware receding horizon navigation for mavs,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 2534–2541.
  * [3] L. Bartolomei, L. Teixeira, and M. Chli, “Perception-aware path planning for uavs using semantic segmentation,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 5808–5815.
  * [4] N. Roy, W. Burgard, D. Fox, and S. Thrun, “Coastal navigation-mobile robot navigation with uncertainty in dynamic environments,” in _Proceedings 1999 IEEE international conference on robotics and automation (Cat. No. 99CH36288C)_ , vol. 1. IEEE, 1999, pp. 35–40.
  * [5] R. Schirmer, P. Biber, and C. Stachniss, “Efficient path planning in belief space for safe navigation,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 2857–2863.
  * [6] Y. Gao, J. Liu, M. Q. Hu, H. Xu, K. P. Li, and H. Hu, “A new path evaluation method for path planning with localizability,” _IEEE Access_ , vol. 7, pp. 162 583–162 597, 2019.
  * [7] S. Yao, G. Chen, Q. Qiu, J. Ma, X. Chen, and J. Ji, “Crowd-aware robot navigation for pedestrians with multiple collision avoidance strategies via map-based deep reinforcement learning,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 8144–8150.
  * [8] D. Fox, W. Burgard, and S. Thrun, “Markov localization for mobile robots in dynamic environments,” _Journal of artificial intelligence research_ , vol. 11, pp. 391–427, 1999.
  * [9] G. D. Tipaldi, D. Meyer-Delius, and W. Burgard, “Lifelong localization in changing environments,” _The International Journal of Robotics Research_ , vol. 32, no. 14, pp. 1662–1678, 2013.
  * [10] D. Sun, F. Geißer, and B. Nebel, “Towards effective localization in dynamic environments,” in _2016 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2016, pp. 4517–4523.
  * [11] W. Burgard, D. Fox, and S. Thrun, “Active mobile robot localization,” in _IJCAI_. Citeseer, 1997, pp. 1346–1352.
  * [12] G. Borghi and V. Caglioti, “Minimum uncertainty explorations in the self-localization of mobile robots,” _IEEE Transactions on Robotics and Automation_ , vol. 14, no. 6, pp. 902–911, 1998.
  * [13] A. Q. Li, M. Xanthidis, J. M. O’Kane, and I. Rekleitis, “Active localization with dynamic obstacles,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2016, pp. 1902–1909.
  * [14] T. Fan, X. Cheng, J. Pan, P. Long, W. Liu, R. Yang, and D. Manocha, “Getting robots unfrozen and unlost in dense pedestrian crowds,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1178–1185, 2019.
  * [15] R. Sim and N. Roy, “Global a-optimal robot exploration in slam,” in _Proceedings of the 2005 IEEE international conference on robotics and automation_. IEEE, 2005, pp. 661–666.
  * [16] G. Costante, J. Delmerico, M. Werlberger, P. Valigi, and D. Scaramuzza, “Exploiting photometric information for planning under uncertainty,” _Robotics Research: Volume 1_ , pp. 107–124, 2018.
  * [17] L. Bartolomei, L. Teixeira, and M. Chli, “Semantic-aware active perception for uavs using deep reinforcement learning,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 3101–3108.
  * [18] L. Bartolomei, Y. Kompis, L. Teixeira, and M. Chli, “Autonomous emergency landing for multicopters using deep reinforcement learning,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 3392–3399.
  * [19] F. Lin, Z. Ji, C. Wei, and R. Grech, “Localisation-safe reinforcement learning for mapless navigation,” in _2022 IEEE International Conference on Robotics and Biomimetics (ROBIO)_. IEEE, 2022, pp. 1327–1334.
  * [20] V. Fox, J. Hightower, L. Liao, D. Schulz, and G. Borriello, “Bayesian filtering for location estimation,” _IEEE pervasive computing_ , vol. 2, no. 3, pp. 24–33, 2003.
  * [21] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [22] G. Chen, S. Yao, J. Ma, L. Pan, Y. Chen, P. Xu, J. Ji, and X. Chen, “Distributed non-communicating multi-robot collision avoidance via map-based deep reinforcement learning,” _Sensors_ , vol. 20, no. 17, p. 4836, 2020\.
  * [23] J. v. d. Berg, S. J. Guy, M. Lin, and D. Manocha, “Reciprocal n-body collision avoidance,” in _Robotics research_. Springer, 2011, pp. 3–19.
  * [24] D. Helbing and P. Molnar, “Social force model for pedestrian dynamics,” _Physical review E_ , vol. 51, no. 5, p. 4282, 1995.
  * [25] S. Thrun, “Probabilistic robotics,” _Communications of the ACM_ , vol. 45, no. 3, pp. 52–57, 2002.
  * [26] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2018, pp. 3803–3810.
  * [27] J. Choi, K. Park, M. Kim, and S. Seok, “Deep reinforcement learning of navigation in a complex and crowded environment with a limited field of view,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 5993–6000.
  * [28] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  * [29] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [30] D. Fox, W. Burgard, and S. Thrun, “The dynamic window approach to collision avoidance,” _IEEE Robotics & Automation Magazine_, vol. 4, no. 1, pp. 23–33, 1997.
  * [31] T. Fan, P. Long, W. Liu, and J. Pan, “Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios,” _The International Journal of Robotics Research_ , vol. 39, no. 7, pp. 856–892, 2020.
  * [32] T. Linder, S. Breuers, B. Leibe, and K. O. Arras, “On multi-modal people tracking from mobile platforms in very crowded and dynamic environments,” in _2016 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2016, pp. 5512–5519.