# Topological Data Analysis of Ion Migration Mechanism

Ryuhei Sato  Advanced Institute for Materials Research, Tohoku University,
2-1-1 Katahira, Aoba-ku, Sendai 980-8577, Japan  Kazuto Akagi  Advanced
Institute for Materials Research, Tohoku University, 2-1-1 Katahira, Aoba-ku,
Sendai 980-8577, Japan  Shigeyuki Takagi  Institute for Materials Research,
Tohoku University, 2-1-1 Katahira, Aoba-ku, Sendai 980-8577, Japan.  Kartik
Sau  Advanced Institute for Materials Research, Tohoku University, 2-1-1
Katahira, Aoba-ku, Sendai 980-8577, Japan  Mathematics for Advanced Materials
Open Innovation Laboratory (MathAM-OIL), National Institute of Advanced
Industrial Science and Technology (AIST), c/o Advanced Institute for Materials
Research (AIMR), Tohoku University, Sendai 980-8577, Japan  Kazuaki Kisu
Institute for Materials Research, Tohoku University, 2-1-1 Katahira, Aoba-ku,
Sendai 980-8577, Japan.  Hao Li  Advanced Institute for Materials Research,
Tohoku University, 2-1-1 Katahira, Aoba-ku, Sendai 980-8577, Japan  Shin-ichi
Orimo  Advanced Institute for Materials Research, Tohoku University, 2-1-1
Katahira, Aoba-ku, Sendai 980-8577, Japan  Institute for Materials Research,
Tohoku University, 2-1-1 Katahira, Aoba-ku, Sendai 980-8577, Japan.
<EMAIL_ADDRESS>(R.S<EMAIL_ADDRESS>(S.O.)

###### Abstract

Topological data analysis based on persistent homology has been applied to the
molecular dynamics simulation for the fast ion-conducting phase
($\alpha$-phase) of AgI, to show its effectiveness on the ion-migration
mechanism analysis. Time-averaged persistence diagrams of $\alpha$-AgI, which
quantitatively records the shape and size of the ring structures in the given
atomic configurations, clearly showed the emergence of the four-membered rings
formed by two Ag and two I ions at high temperatures. They were identified as
common structures during the Ag ion migration. The averaged potential energy
change due to the deformation of four-membered ring during Ag migration agrees
well with the activation energy calculated from the conductivity Arrhenius
plot. The concerted motion of two Ag ions via the four-membered ring was also
successfully extracted from molecular dynamics simulations by our approach,
providing the new insight into the specific mechanism of the concerted motion.

††preprint: AIP/123-QED

##  I Introduction

To realize all-solid-state ion batteries, ion-conducting electrolytes with
high performance have been developed so farOrimo  _et al._ (2007); Murugan,
Thangadurai, and Weppner (2007); Kamaya  _et al._ (2011); Matsuo and Orimo
(2011); Seino  _et al._ (2014); Unemoto, Matsuo, and Orimo (2014); Mohtadi and
Orimo (2016); Ohara  _et al._ (2016); Kim  _et al._ (2019); Sau  _et al._
(2019); Kisu  _et al._ (2021); Sau and Ikeshoji (2022). These materials,
called superionic conductors, tend to exhibit complex ion migration
mechanisms, such as the ion migration with concerted motionc (13); He, Zhu,
and Mo (2017); Zhang  _et al._ (2019); Sau  _et al._ (2021), utilizing
rotating anionMohtadi and Orimo (2016); Kim  _et al._ (2019); Kisu  _et al._
(2021); Orimo  _et al._ (2007); Sau and Ikeshoji (2022); Sau  _et al._ (2019);
Matsuo and Orimo (2011); Unemoto, Matsuo, and Orimo (2014); Sau  _et al._
(2021); Zhang and Nazar (2022), and the ion migration in glass materialsOhara
_et al._ (2016), which deviates from a single-ion hopping in a uniform
lattice. Since atoms around mobile ions are arranged in a wide variety of
configurations in such materials, extracting representative ion migration
mechanisms and rate-limiting step is not easy. Even for bulk materials such as
Li7La3Zr2O12, the activation energy for each Li ion migration during molecular
dynamics (MD) simulations is widely distributed from 0 to 0.6 eVWood  _et al._
(2021). Therefore, it is challenging to construct a guideline for material
design based on the migration mechanism.

Topological data analysis (TDA) based on persistent homology has been used for
structural analysis of amorphous materialsHiraoka  _et al._ (2016); Hong and
Kim (2019); Stellhorn  _et al._ (2020); Morley, Salmon, and Wilson (2021),
glassesHirata, Matsue, and Chen (2016); Onodera  _et al._ (2019); Hosokawa
_et al._ (2019); Hirata  _et al._ (2020); Sorensen  _et al._ (2022), porous
materialsLee  _et al._ (2017), and complex molecular liquidsSasaki, Okajima,
and Yamashita (2018). This analysis can characterize point cloud data (ex.
atomic structures), by projecting them onto a two-dimensional density
distribution of the rings and pores in the system. Extracting representative
ring/pore structures from the distribution and performing inverse analysis
have contributed to the elucidation of the hidden order in the complex
materials. In recent years, persistence diagrams, the distribution of ring and
pore structures, have been combined with machine learning and other techniques
to predict physical properties such as thermal conductivity
propertiesMinamitani  _et al._ (2022a, b), stress-strain curvesWang, Ogawa,
and Adachi (2020), formation enthalpyJiang  _et al._ (2021), and so onSuzuki
_et al._ (2021).

TDA was also used for dynamic phenomena. For example, it was applied for glass
formationHirata  _et al._ (2020) and metal nucleation processesBecker  _et
al._ (2022). Based on defect chemistry, the ion migration is a phenomenon that
occurs around ion vacancies. By considering vacancies in ionic conductors as
large voids inside rings and pores in the persistence diagram, ion migration
can be analyzed using TDA. By performing the inverse analysis along the
ring/pore structures with high density, a representative ion migration
mechanism within the material is considered to be extracted from the
persistence diagram. Since the obtained rings/pores are composed of multiple
ions, it is possible to discuss ion migration as a many-body problem.

In this study, we analyzed the Ag migration mechanism of $\alpha$-AgINield
_et al._ (1993); Hull (2004); Shimojo and Aniya (2005); Shimojo  _et al._
(2006); Bitrián and Trullàs (2008); Morgan and Madden (2014); Yamamoto (2017);
Brenner  _et al._ (2020); Grasselli (2022), a classical superionic conductor,
as an example to elucidate complex migration mechanisms such as concerted
motion. To our best knowledge, it is the first report to apply topological
data analysis based on persistent homology to the ion migration study. The
advantage of this method over existing ones is that it does not require any
prior information like collective variables for constrained MD simulations and
ion migration paths based on the crystallography, once the sufficient MD
simulation trajectory is obtained. Note that the potential energy surface of
the whole system mainly used in transition state search does not necessarily
work for analyzing migration paths in these ionic conductors, because multiple
ion migrations could occur simultaneously in a simulation cell. In addition,
in the highly-disordered system, the activation energy obtained from nudged
elastic band (NEB) methodMills, Jonsson, and Schenter (1995) is strongly
affected by the position of mobile ions, because of the electrostatic
interaction between them. This suggests that plenty of NEB calculations with
different mobile-ion configurations are needed to obtain a reasonable ion
migration mechanism. Therefore, this work becomes a milestone to understand
the ion migration mechanism for highly-disordered system like glass
materialsOhara  _et al._ (2016) and compounds with complex-anions Mohtadi and
Orimo (2016); Kim  _et al._ (2019); Kisu  _et al._ (2021); Orimo  _et al._
(2007); Sau and Ikeshoji (2022); Sau  _et al._ (2019); Matsuo and Orimo
(2011); Unemoto, Matsuo, and Orimo (2014); Sau  _et al._ (2021); Zhang and
Nazar (2022).

##  II Method

### A. Moleucular dynamics simulations

NVT-MD simulations of $\alpha$-AgI were performed using MD calculation package
(Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS)Plimpton
(1995) (version 29 October 2020)). Note that we chose $\alpha$-AgI as a model
system to demonstrate the effectiveness of TDA for the ion migration mechanism
for the following reasons; one is its high ionic conductivityObayashi,
Nakamura, and Hiraoka (2022); Hull (2004) making it easy to sample the ion
migration during MD simulation. another is that the classical potentialShimojo
and Kobayashi (1991); Parrinello, Rahman, and Vashishta (1983) is available in
this system, reducing the calculation cost. The other reason is that
$\alpha$-AgI shows concerted motion of Ag ionsYamamoto (2017); Morgan and
Madden (2014), although it is consisted of ordered matrix, the bcc lattice of
I ions. Therefore, it is easy to verify whether the ion migration mechanism
obtained from TDA is reasonable or not, by comparing it with that suggested
from the analysis and discussion based on the crystal structure.

Figure 1: (a) Crystal structure of $\alpha$-AgI (white: Ag black: Ag vacancy,
purple: I). I ions show a bcc structure, while Ag sites are located at the
vertexes of the Voronoi polyhedron (black bold lines in the figure), which is
composed of the perpendicular bisectors of each I ion. Ag ions occupy two of
the twelve sites (12d sites). (b) An example of the structure of $\alpha$-AgI
when Ag ions are specifically placed in two of these sites. (c) Conductivity
Arrhenius plot of $\alpha$-AgI during classical MD simulationsc (42). The
dashed curve represents the fitting result of conductivity above 440K using
Arrhenius equation, ln$\sigma T$ = –$E_{A}/RT$+A.

The detail of the calculation condition is followings. Nosé–Hoover thermostat
and barostatHoover (1986) was employed for the temperature control in all MD
simulations. Periodic boundary conditions were applied along all the Cartesian
directions. Newton’s motion equations were integrated using the customary
Verlet’s algorithm with a time-step length of 0.005 ps for MD simulations.
3-ns MD simulations with a cubic cell, 20.42 $\times$ 20.42 $\times$ 20.42 Å,
consisting of 256 Ag and I ions (4$\times$4$\times$4 supercell of $\alpha$-AgI
(Fig. 1)) were performed at various temperatures from 360 to 1000K. Previous
study Grasselli (2022) have reported that the finite size effect is negligibly
small in AgI system, in terms of diffusion coefficients calculation. This
suggests that our selected cell size does not affect the ion migration process
during these MD simulations. We used the Parrinello-Rahman-Vashishta
potentialShimojo and Kobayashi (1991); Parrinello, Rahman, and Vashishta
(1983), for the interatomic potential for $\alpha$-AgI. It is known that
Parrinello-Rahman-Vashishta potential represents Ag-I and I-I interactions
well and can reproduce many physical properties such as diffusion
coefficientsShimojo and Kobayashi (1991) and phase transitionsParrinello,
Rahman, and Vashishta (1983). However, it has been reported that this
potential cannot fully reproduce Ag-Ag interactions due to the lack of
treatment for covalent interactions between Ag ionsShimojo  _et al._ (2006)
and atomic polarizationBitrián and Trullàs (2008). Nevertheless, the accuracy
of the potential is not a matter of concern in this study, since our goal is
to obtain the essential processes hidden in the complex ion migration dynamics
containing concerted motions. Also note that NVT-MD simulations are
intentionally conducted to maintain $\alpha$-AgI structure at low temperatures
(360-440K). This makes us easier to compare the persistence diagram obtained
at different temperatures since the matrix, I bcc lattice, and cell parameter
is constant.

To calculate time-averaged potential energy for ring-structures obtained from
TDA, we also conducted constrained MD simulations. In these calculations,
smaller cubic cell, 10.21 $\times$ 10.21 $\times$ 10.21 Å, consisting of 32 Ag
and I ions (2$\times$2$\times$2 supercell) was employed. In the simulation
cell, we introduced one four-membered ring consisting of two Ag and two I ions
obtained from TDA. Ions in this ring structure have been fixed during MD
simulations by setting their force and velocity to zero. We prepared 16
simulation cells with different four-membered rings along the distribution
obtained from the persistence diagram. Then, 4-ns MD simulations of these
cells were conducted at 1000K. The difference of time-averaged potential
energy between these simulations should be originated from the shape of four-
membered ring and its surrounding environment, since the sampling time was
long enough and ions except that in fixed four-membered ring could move
freely. Therefore, we can discuss the change in potential energy during Ag ion
migration via the four-membered ring based on time-averaged potential energy
of these MD simulations.

### B. Topological data analysis

Figure 2:  Schematic diagram of how ring structures in an atomic configuration
are analyzed by topological data analysis (TDA) based on persistent homology.
In this analysis, we place a sphere with same radius, $r$, at each atomic
coordinate in the system. By increasing $r$, some atoms are connected to each
other to form the ring. The diameter of the sphere when the ring is formed
(2$r_{b}$) is assigned to the x-coordinate as birth diameter. At larger radius
($r=r_{d}$), this ring structure is filled by the spheres. Here, the diameter
2$r_{d}$ is named as death diameter and assigned to y-coordinate.

Persistent homology is a mathematical framework for analyzing n-dimensional
holes in the given discrete data such as atomic coordinates. A software
package, HomcloudObayashi, Nakamura, and Hiraoka (2022); Obayashi (2018), is
employed for TDA based on persistent homology. Here, we only show a brief
outline of its concept. See also reference 19 and 27 for the theoretical
details and the application to materials. We will focus on one-dimensional
holes (i.e. ring structures) hereafter. Note that one-dimensional hole in
homology is defined as the states, which has a one-dimensional boundary (i.e.,
a space bounded by lines connecting atoms) but no two-dimensional boundary
(i.e., the surface created by connecting atoms is not entirely filled by a
sphere of radius $r$, assuming there is the sphere with a certain radius $r$
at each atomic coordinate). Figure 2 is a schematic diagram of how rings are
analyzed in TDA based on persistent homology. First, a sphere with the common
radius, $r$, is placed at each of atomic coordinates regardless of its
element. By increasing $r$, new edges are spanned one by one when two spheres
are overlapped. At a particular radius ($r=r_{b}$), some atoms are connected
to each other and form a ring (top right figure in Fig. 2). In this paper, we
name this diameter 2$r_{b}$ as the birth diameter and assign it to
x-coordinate on the persistence diagram. At larger radius ($r=r_{d}$), this
ring structure is filled by the sphere as shown in the bottom left figure.
Here, the diameter 2$r_{d}$ is named as death diameter and assigned to
y-coordinate. Thus, a persistence diagram is obtained as a two-dimensional
scatter plot of birth-death pairs. To make it easy to understand visually, We
limit the area to the square of [0.0, 12.0] $\times$ [0.0, 12.0] (Å2) and
divided it into 200 $\times$ 200 meshes. We evaluated the number of birth-
death pairs in each mesh as $log_{10}P(b,d)$.

##  III Results and discussion

### A. Change in the persistence diagram against ionic conductivity in
$\bm{\alpha}$-AgI

$\alpha$-AgI consists of a bcc I lattice with twelve stable Ag sites (12d
sites), as shown in fig. 1(a). These stable Ag sites correspond to the vertex
of the Voronoi polyhedron for the I bcc lattice, the polyhedron created by the
perpendicular bisectors between I ions. Ag ions only occupy two of these
stable sites like the structure in fig. 1(b). Therefore, once it is thermally
activated, Ag ion can utilize unoccupied vacancy sites for ion migration.
According to previous studiesHull (2004); Nield  _et al._ (1993), six out of
seven Ag ions migrate along the sides of the Voronoi polyhedron, the black
line in the figure. On the other hand, the rest of Ag ions is transported
through the face-center position (0.5 0.5 0) in the unit cell. Figure 1(c) is
a conductivity Arrhenius plot of $\alpha$-AgI during MD simulations at various
temperatures. We used Green-Kubo formula for the conductivity calculationc
(42); Marcolongo and Marzari (2017); Kubo (1957); French, Hamel, and Redmer
(2011); Federico and Stefano (2019); Pegolo, Grasselli, and Baroni (2020). Due
to the poor convergence of these calculations, the conductivity calculations
used the results of MD simulations extended to 25 ns. $\alpha$-AgI showed
superionic conduction at temperatures higher than 440 K. The obtained
conductivity above 440 K is in good agreement with the experimental value
($\approx 10^{0}$ (S/cm))Hull (2004).

Figure 3:  Time-averaged persistence diagram of the ring structure during MD
simulations of $\alpha$-AgI for several temperatures. This persistence diagram
represents a probability density ($P$) distribution normalized so that the
integrated value over the squared regions [0.0, 12.0] $\times$ [0.0, 12.0]
(Å2) becomes unity. The color in the figure is scaled according to
log${}_{10}P$ and represents the probability density per the mesh of 0.05
$\times$ 0.05 Å2.

The change in the local structure of $\alpha$-AgI during Ag migration has been
confirmed prominently as that in ring structures in the persistence diagram.
Figure 3 shows time-averaged persistence diagrams during MD simulations at
various temperatures. We conducted TDA for the atomic configurations every
0.05 ps during MD simulations and obtained the averaged persistence diagram of
6,000 configurations. Two types of ring structures are mainly distributed
around (Birth, Death)=(b,d)=(2.8,4.5) and (4.2,4.4) in the persistence
diagrams for MD simulations at 360K and 400K, where ionic conductivity is low.
On the other hand, the ring-structure distribution along the black dashed line
increases on the persistence diagrams at 550K and 1000K. Therefore, these ring
structures could correspond to the ones for Ag migration. Note that the same
change in the ring structure has been confirmed in the persistent diagrams
obtained from the NPT-MD simulations, taking into account the phase transition
to $\beta$-AgI (see section S1 in the supporting information).

### B. Inverse analysis for the persistence diagram of $\bm{\alpha}$-AgI

Figure 4:  (a) Persistence diagram only showing four-membered rings and the
distribution of averaged I-I, Ag-I, and Ag-Ag distances in each (b,d) for
these four-membered rings in the persistence diagram. (Calculated for each
0.05 $\times$ 0.05 Å2 meshes.) The color in top left figure is scaled
according to log${}_{10}P$ and represents the probability density per the mesh
of 0.05 $\times$ 0.05 Å2. On the other hand, that in other figures is scaled
according to the distance (Å) between ions. (b) Ag migration obtained by the
deformation of four-membered ring along the black dashed line on the
persistence diagram. Note that I ions except that in the four-membered ring
show ideal I bcc lattice positions. (white: Ag, purple: I)

From the inverse analysisObayashi (2018) of the persistence diagram, a model
of Ag ion migration via four-membered rings has been obtained. Here, the
inverse analysis means to calculate averaged property for each mesh in the
persistence diagram, recorded during the TDA. Inverse analysis of the
persistence diagram at 1000K confirmed that these ring structures form a four-
membered ring consisting of two Ag and I ions (Fig. S2). The persistence
diagram of the four-membered rings in the system and the results of their
inverse analysis (the average I-I, Ag-I, and Ag-Ag distances in the four-
membered rings at each mesh) are summarized in Fig. 4(a). As shown in the
figure, the distance between I ions is almost constant. On the other hand, the
length of longest Ag-I bond in the four-membered ring decreases as birth
decreases. Similarly, the distance between Ag ions was also shortening with
decreasing death. These results show that the four-membered ring deforms, as
the birth decreases along the dashed black line in the persistence diagram.
Figure 4(b) shows the structure of the four-membered ring reproduced based on
these average distances. Note that I ions other than that in the 4-membered
ring occupy the ideal positions of bcc structure in Fig. 4(b). The black line
is the Voronoi polyhedron of an ideal I bcc lattice ($\approx$Ag migration
path). Ag1 moved from one vertex to the other in the Voronoi polyhedron,
indicating that Ag hopping migration has been reproduced by the deformation of
the four-membered ring obtained from the persistence diagram.

In these four-membered rings, the interaction between Ag1 and Ag2 ions via the
I-lattice has been confirmed from bond valence analysis (Fig. S3). Such a
picture of Ag migration via interaction with I-lattice has been claimed in
previous studies. For example, covalent-bond formation between nearest
neighbour Ag and I ions has been reported from the bond-overlap population
analysis of $ab$ $initio$ MD simulation for molten AgIShimojo  _et al._
(2006). In addition, $ab$ $initio$ MD simulation and THz-frequency Raman
polarization-orientation measurement suggest that anharmonic vibrations in the
I lattice are important for Ag migrationBrenner  _et al._ (2020).

Figure 5:  Ag migration obtained by repeating the deformation of the four-
membered ring as shown in Fig. 4(b) (i.e., high (b,d) $\rightarrow$ low (b,d)
$\rightarrow$ high (b,d)); (a) backward Ag migration, (b) single Ag hopping,
(c) concerted motion of two Ag ions, and (d) concerted motion of Ag ions
maintaining the four-membered ring shape. The figure is created with (111) as
the normal vector, and (b,d) represents the values of birth and death of the
four-member ring. (white: Ag, purple: I)

Fig. 5 and the video in Supporting information summarize possible Ag
migrations via the four-membered ring. During Ag migration in figs. 5(a)-(c),
the four-membered ring deforms along the dashed black line in Fig. 4(a). On
the other hand, the four-membered ring shape is maintained in the latter part
of Ag migration in Fig. 5(d). There are four possible migrations. The first
one (Fig. 5(a)) shows Ag hopping and subsequent returning to its original
site. In the second one (Fig. 5(b)), Ag1 further migrates to the other site.
In this reaction, Ag2 returns to the almost same position. Therefore, this
migration is considered to be the single Ag1 hopping. The third one (Fig.
5(c)) is a concerted motion, where Ag1 migrates from one site to the other and
Ag2 hopping occurs immediately after Ag1 migration. Fig. 5(d) also shows a
concerted migration of Ag ions. However, it occurs without the deformation of
the four-membered ring. As shown in the figure, Ag ions migrate along the
hexagon on the Voronoi polyhedron of I bcc, maintaining the shape of the four-
membered ring.

From the analysis of Ag hopping directions between nearest neighbour sites,
the ratio of these four Ag migrations are estimated to be around (a) 42%, (b)
19%, (c) 26%, and (d) 13%, respectively (Fig. S4). This suggests that
concerted motions such as (c) and (d) occur with about 39% probability. The
existence of concerted motion in AgI and similar class of material (Ag2Se) has
been claimed based on the Haven ratio for ionic conductivityYamamoto (2017)
and the velocity correlation function of Ag ions during MD simulationsShimojo
and Aniya (2005). These results are consistent with the obtained concerted
motion in Fig. 5(c) and (d). The advantage of TDA from conventional studies is
that we can extract the concrete model of concerted ion migration as shown in
Fig. 5.

Figure 6:  Time-averaged energy during 4-ns MD simulations for
2$\times$2$\times$2 $\alpha$-AgI supercells with one fixed four-membered ring
obtained from a persistence diagram along the black dashed line in Fig. 4(a).

The Ag migration via the four-membered ring obtained from the persistence
diagram corresponds to the representative Ag migration in $\alpha$-AgI. Figure
6 shows the time-averaged potential energy during 4-ns MD simulations with
four-membered rings obtained from the persistence diagram fixed in a 2×2×2
$\alpha$-AgI supercell. Since ions other than that in the four-membered ring
moved freely and the sampling time was long enough, the energy difference in
these MD simulations originated from the four-membered ring shape. Therefore,
fig. 6 shows the energy profile for the deformation of the four-membered ring
along the black dashed line in the persistence diagram in Fig. 4(a). The
averaged potential energy change due to the deformation of four-membered ring
during Ag migration is about 0.12 eV. Note that the energy profile for three
types of Ag migration in fig. 5(a)-(c) should be the same as that of Fig. 6.
This is because these Ag migrations can be reproduced by a point-symmetric
operation of the four-membered ring on the black dashed line in Fig. 4(a)
using the middle point of two I ions in the ring. The averaged potential
energy change due to the deformation of four-membered ring, which can
reproduce Ag migration, is in good agreement with the activation energy
obtained from the conductivity Arrhenius plot. This shows that the persistence
diagram is able to extract a local but representative structure for Ag
migration.

### C. The comparison between independent and four-membered ring Ag migration

Figure 7:  (left) Persistence diagram of I bcc lattice with Ag ions randomly
placed on the migration path, (edges and vertexes of the Voronoi polyhedron in
fig. 1(a)). (right) Time-averaged persistence diagram during MD simulation of
$\alpha$-AgI at 550 K. The black open squares in the figure represent the
(b,d) positions of the ring structure obtained from I bcc lattice with Ag ions
randomly placed at the stable (12d) site (edges of the Voronoi polyhedron).The
color in the figure is scaled according to log${}_{10}P$ and represents the
probability density per the mesh of 0.05 $\times$ 0.05 Å2.

Independent migration of each Ag ion like a random-walk modelc (49) does not
form the four-membered rings specific to the reproduced Ag migration, showing
that the interaction between ions selectively creates them during MD
simulations. Figure 7(left) shows a persistence diagram of an ideal I bcc
lattice with Ag ions randomly placed on the migration path (i.e. the edges and
vertexes of the Voronoi polyhedron for I bcc lattice). Since Ag ions are
placed almost randomly along the migration pathc (48), the resulting
persistence diagram is considered to be equivalent to that for the random-walk
model. We created 6000 random configurations of 4$\times$4$\times$4 supercells
and calculated their averaged persistence diagram so that the calculation
condition for it matches that for the persistence diagram for MD simulations.
The black open squares in the figure represent the (b,d) positions of rings
obtained when Ag ions are only placed at the stable site (the vertexes of the
Voronoi polyhedron). The distribution among these black open squares was found
to increase, when Ag ions are randomly placed on the migration path in the I
bcc lattice. However, ring structures along the black dashed line observed in
MD simulations at 550 and 1000K were hardly obtained in the persistence
diagram of I bcc lattice with random Ag configurations. Therefore, we conclude
that independent migration of Ag ions cannot explain the ones during MD
simulations and the concerted motion of Ag ions induced by the interaction
between them actually contributes to Ag ion conductivity in $\alpha$-AgI.

##  IV Conclusion

To reveal the ion migration mechanism in the superionic conductor, we
performed classical molecular dynamics simulations for $\alpha$-AgI and made a
series of topological data analysis based on persistent homology. When
$\alpha$-AgI shows superionic conductivity, characteristic Ag-I ring
structures were newly detected on the persistence diagram. They were
identified as four-membered rings formed by two Ag and I ions, which are
common to the Ag ion migration. The averaged potential energy change due to
the deformation of four-membered ring during Ag migration is about 0.12 eV,
which is in good agreement with the activation energy obtained from the
conductivity Arrhenius plot. Therefore, we conclude that the Ag ion migration
via these four-membered rings is the representative Ag migration process in
$\alpha$-AgI. In this migration, two Ag ions showed concerted motion and the
ratio of such Ag ion migration was about 39%. On the other hand, the number of
the corresponding four-membered rings became significantly small, if random-
walk of Ag ions in I bcc lattice is assumed. Therefore, based on topological
data analysis of the MD trajectories, we conclude that it is successfully
elucidated that Ag ions migrate not independently but concertedly via the
four-membered ring in $\alpha$-AgI.

## SUPPLEMENTARY MATERIAL

See the supplementary material for the persistence diagram of AgI during 3-ns
NPT-MD simulations (section S1), the distribution of most frequent number of
atoms in the ring in persistence diagram during the MD simulation at 1000K
(section S2), bond valence analysis of four-membered rings obtained from
inverse analysis of persistence diagram (section S3), The result of the
direction analysis of single Ag hopping during MD simulation at 1000K (section
S4), and videos for Ag migrations through the deformation of four-membered
ring obtained from the topological data analysis based on persistent homology.

###### Acknowledgements.

This work was supported by JSPS KAKENHI Grant-in-Aid for Scientific Research
on Innovative Areas “Hydrogenomics”, No. JP18H05513 and Grant-in-Aid for
Challenging Research (Pioneering, No, 19H05514). The computation in this work
has been done using the facilities of the Supercomputer Center, the Institute
for Solid State Physics, the University of Tokyo.

## AUTHOR DECLARATIONS

## Conflict of Interest

The authors have no conflicts to disclose.

## Data Availability

The data that support the findings of this study are available from the
corresponding author upon reasonable request. Note that the input files for MD
simulations, representative output files, and the calculation codes used in
this paper are available on Materials Cloud platformLeopold  _et al._ (2020).
See DOI: https://doi.org/10.24435/materialscloud:zn-5t

## Reference

## References

  * Orimo  _et al._ (2007) S. Orimo, Y. Nakamori, J. R. Eliseo, A. Züttel, and C. M. Jensen, “Complex hydrides for hydrogen storage,” Chem. Rev. 107, 4111–4132 (2007).
  * Murugan, Thangadurai, and Weppner (2007) R. Murugan, V. Thangadurai, and W. Weppner, “Fast lithium ion conduction in garnet-type Li7La3Zr2O12,” Angew. Chem. Int. Ed. 46, 7778–7781 (2007).
  * Kamaya  _et al._ (2011) N. Kamaya, K. Homma, Y. Yamakawa, M. Hirayama, R. Kanno, M. Yonemura, T. Kamiyama, Y. Kato, S. Hama, K. Kawamoto, and A. Mitsui, “A lithium superionic conductor,” Nat. Mater. 10, 682–686 (2011).
  * Matsuo and Orimo (2011) M. Matsuo and S. Orimo, “Lithium fast-ionic conduction in complex hydrides: Review and prospects,” Adv. Energy Mater. 1, 161–172 (2011).
  * Seino  _et al._ (2014) Y. Seino, T. Ota, K. Takada, A. Hayashi, and M. Tatsumisago, “A sulphide lithium super ion conductor is superior to liquid ion conductors for use in rechargeable batteries,” Energy Environ. Sci. 7, 627–631 (2014).
  * Unemoto, Matsuo, and Orimo (2014) A. Unemoto, M. Matsuo, and S. Orimo, “Complex hydrides for electrochemical energy storage,” Adv. Func. Mater. 24, 2267–2279 (2014).
  * Mohtadi and Orimo (2016) R. Mohtadi and S. Orimo, “The renaissance of hydrides as energy materials,” Nat. Rev. Mater. 2, 16091 (2016).
  * Ohara  _et al._ (2016) K. Ohara, A. Mitsui, M. Mori, Y. Onodera, S. Shiotani, Y. Koyama, Y. Orikasa, M. Murakami, K. Shimoda, K. Mori, T. Fukunaga, H. Arai, Y. Uchimoto, and O. Z., “Structural and electronic features of binary Li2S-P2S5 glasses,” Sci. Rep. 6, 21302 (2016).
  * Kim  _et al._ (2019) S. Kim, H. Oguchi, N. Toyama, T. Sato, S. Takagi, T. Otomo, D. Arunkumar, N. Kuwata, J. Kawamura, and S. Orimo, “A complex hydride lithium superionic conductor for high-energy-density all-solid-state lithium metal batteries,” Nat. Commun. 10, 1081 (2019).
  * Sau  _et al._ (2019) K. Sau, T. Ikeshoji, S. Kim, S. Takagi, K. Akagi, and S.-i. Orimo, “Reorientational motion and Li+-ion transport in Li2B12H12 system: Molecular dynamics study,” Phys. Rev. Materials 3, 075402 (2019).
  * Kisu  _et al._ (2021) K. Kisu, S. Kim, T. Shinohara, K. Zhao, A. Züttel, and S. Orimo, “Monocarborane cluster as a stable fuorine-free calcium battery electrolyte,” Sci. Rep. 11, 7563 (2021).
  * Sau and Ikeshoji (2022) K. Sau and T. Ikeshoji, “Ring mechanism of fast Na+ ion transport in Na2LiFeTeO6: Insight from molecular dynamics simulation,” Phys. Rev. Materials 6, 045406 (2022).
  * c (13) The migration of multiple ions triggered by that of one ion. Here, the term, concerted motion, is defined in a broad sense. In this article, if other ions move in conjunction with the migration of one ion, it is considered to be a concerted motion, regardless of the direction of transport, etc.
  * He, Zhu, and Mo (2017) X. He, Y. Zhu, and Y. Mo, “Origin of fast ion diffusion in super-ionic conductors,” Nat. Commun. 8, 15893 (2017).
  * Zhang  _et al._ (2019) Z. Zhang, Z. Zou, K. Kaup, R. Xiao, S. Shi, M. Avdeev, Y.-S. Hu, D. Wang, B. He, X. Li H., Huang, L. F. Nazar, and L. Chen, “Correlated migration invokes higher Na+-ion conductivity in NaSICON-type solid electrolytes,” Adv. Energy Mater. 9, 1902373 (2019).
  * Sau  _et al._ (2021) K. Sau, T. Ikeshoji, S. Kim, S. Takagi, and S. Orimo, “Comparative molecular dynamics study of the roles of anion-cation and cation-cation correlation in cation diffusion in Li2B12H12 and LiCB11H12,” Chem. Mater. 33, 2357–2369 (2021).
  * Zhang and Nazar (2022) Z. Zhang and L. F. Nazar, “Exploiting the paddle-wheel mechanism for the design of fast ion conductors,” Nat. Rev. Mater. 7, 389–405 (2022).
  * Wood  _et al._ (2021) B. C. Wood, J. B. Varley, K. E. Kweon, P. Shea, A. T. Hall, A. Grieder, M. Ward, V. P. Aguirre, D. Rigling, V. E. Lopez, C. Stancill, and A. N., “Paradigms of frustration in superionic solid electrolytes,” Phil. Trans. R. Soc. A. 379, 20190467–20190467 (2021).
  * Hiraoka  _et al._ (2016) Y. Hiraoka, T. Nakamura, A. Hirata, E. G. Escolar, K. Matsue, and Y. Nishiura, “Hierarchical structures of amorphous solids characterized by persistent homology,” Proc. Natl. Acad. Sci. USA 113, 7035 (2016).
  * Hong and Kim (2019) S. Hong and D. Kim, “Medium-range order in amorphous ices revealed by persistent homology,” J. Phys. Condens. Mater. 31, 455403 (2019).
  * Stellhorn  _et al._ (2020) J. R. Stellhorn, B. Paulus, S. Hosokawa, W.-C. Pilgrim, N. Boudet, N. Blanc, H. Ikemoto, S. Kohara, and Y. Sutou, “Structure of amorphous Cu2GeTe3 and the implications for its phase-change properties,” Phys. Rev. B 101, 214110 (2020).
  * Morley, Salmon, and Wilson (2021) D. O. Morley, P. S. Salmon, and M. Wilson, “Persistent homology in two-dimensional atomic networks,” J. Chem. Phys. 154, 124109 (2021).
  * Hirata, Matsue, and Chen (2016) A. Hirata, K. Matsue, and M. Chen,  _Structural analysis of metallic glasses with computational homology._ (Springer, 2016).
  * Onodera  _et al._ (2019) Y. Onodera, Y. Takimoto, H. Hijiya, T. Taniguchi, S. Urata, S. Inaba, S. Fujita, I. Obayashi, Y. Hiraoka, and S. Kohara, “Origin of the mixed alkali effect in silicate glass.” NPG Asia Mater. 11, 75 (2019).
  * Hosokawa  _et al._ (2019) S. Hosokawa, J.-F. Bérar, N. Boudet, W.-C. Pilgrim, L. Pusztai, S. Hiroi, K. Maruyama, S. Kohara, H. Kato, H. E. Fischer, and A. Zeidler, “Partial structure investigation of the traditional bulk metallic glass Pd40Ni40P20,” Phys. Rev. B 100, 054204 (2019).
  * Hirata  _et al._ (2020) A. Hirata, T. Wada, I. Obayashi, and Y. Hiraoka, “Structural changes during glass formation extracted by computational homology with machine learning,” Commun. Mater. 1, 98 (2020).
  * Sorensen  _et al._ (2022) S. S. Sorensen, T. Du, C. A. Biscio, L. Fajstrup, and M. M. Smedskjaer, “Persistent homology: A tool to understand medium-range order glass structure,” Journal of Non-Crystalline Solids: X 16, 100123 (2022).
  * Lee  _et al._ (2017) Y. Lee, S. D. Barthel, P. Dłotko, S. M. Moosavi, K. Hess, and B. Smit, “Quantifying similarity of pore-geometry in nanoporous materials,” Nat. Commun. 8, 15396 (2017).
  * Sasaki, Okajima, and Yamashita (2018) K. Sasaki, R. Okajima, and T. Yamashita, “Liquid structures characterized by a combination of the persistent homology analysis and molecular dynamics simulation,” AIP Conference Proceedings 2040, 020015 (2018).
  * Minamitani  _et al._ (2022a) E. Minamitani, T. Shiga, M. Kashiwagi, and I. Obayashi, “Topological descriptor of thermal conductivity in amorphous Si,” J. Chem. Phys. 156, 244502 (2022a).
  * Minamitani  _et al._ (2022b) E. Minamitani, T. Shiga, M. Kashiwagi, and I. Obayashi, “Relationship between local coordinates and thermal conductivity in amorphous carbon,” J. Vacuum Sci. Technol. A 40, 033408 (2022b).
  * Wang, Ogawa, and Adachi (2020) Z.-L. Wang, T. Ogawa, and Y. Adachi, “Property predictions for dual-phase steels using persistent homology and machine learning,” Adv. Theory Simul. 3, 1900227 (2020).
  * Jiang  _et al._ (2021) Y. Jiang, D. Chen, X. Chen, T. Li, G.-W. Wei, and F. Pan, “Topological representations of crystalline compounds for the machine-learning prediction of materials properties,” npj Comput. Mater. 7, 28 (2021).
  * Suzuki  _et al._ (2021) A. Suzuki, M. Miyazawa, J. M. Minto, T. Tsuji, I. Obayashi, Y. Hiraoka, and T. Ito, “Flow estimation solely from image data through persistent homology analysis,” Sci. Rep. 11, 17948 (2021).
  * Becker  _et al._ (2022) S. Becker, E. Devijver, R. Molinier, and N. Jakse, “Unsupervised topological learning approach of crystal nucleation,” Sci. Rep. 12, 3195 (2022).
  * Nield  _et al._ (1993) V. M. Nield, D. A. Keen, W. Hayes, and R. L. McGreevy, “Structure and fast-ion conduction in $\alpha$-AgI,” Solid State Ionics 66, 247–258 (1993).
  * Hull (2004) S. Hull, “Superionics: crystal structures and conduction processes,” Rep. Prog. Phys. 67, 1233 (2004).
  * Shimojo and Aniya (2005) F. Shimojo and M. Aniya, “Diffusion mechanism of Ag ions in superionic conductor Ag2Se from $ab$ $initio$ molecular-dynamics simulations,” J. Phys. Soc. Jpn. 74, 1224 (2005).
  * Shimojo  _et al._ (2006) F. Shimojo, T. Inoue, M. Aniya, T. Sugahara, and Y. Miyata, “$Ab$ $Initio$ molecular-dynamics study of static structure and bonding properties of molten AgI,” J. Phys Soc. Jpn. 75, 114602 (2006).
  * Bitrián and Trullàs (2008) V. Bitrián and J. Trullàs, “Molecular dynamics study of polarization effects on AgI,” J. Phys. Chem. B 112, 1718 (2008).
  * Morgan and Madden (2014) B. J. Morgan and P. A. Madden, “Relationships between atomic diffusion mechanisms and ensemble transport coefficients in crystalline polymorphs,” Phys. Rev. Lett. 112, 145901 (2014).
  * Yamamoto (2017) O. Yamamoto, “Solid state ionics: a japan perspective,” Sci. Technol. Adv. Mater. 18, 504 (2017).
  * Brenner  _et al._ (2020) T. M. Brenner, C. Gehrmann, R. Korobko, T. Livneh, D. A. Egger, and O. Yaffe, “Anharmonic host-lattice dynamics enable fast ion conduction in superionic AgI,” Phys. Rev. Mater. 4, 115402 (2020).
  * Grasselli (2022) F. Grasselli, “Investigating finite-size effects in molecular dynamics simulations of ion diffusion, heat transport, and thermal motion in superionic materials,” The Journal of Chemical Physics 156, 134705 (2022).
  * Mills, Jonsson, and Schenter (1995) G. Mills, H. Jonsson, and G. K. Schenter, “Reversible work transition state theory: application to dissociative adsorption of hydrogen,” Surface Science 324, 305–337 (1995).
  * Plimpton (1995) S. Plimpton, “Fast parallel algorithms for short-range molecular dynamics,” J. Comput. Phys. 117, 1 (1995).
  * Obayashi, Nakamura, and Hiraoka (2022) I. Obayashi, T. Nakamura, and Y. Hiraoka, “Persistent homology analysis for materials research and persistent homology software: Homcloud,” J. Phys. Soc. Jpn. 91, 091013 (2022).
  * Shimojo and Kobayashi (1991) F. Shimojo and M. Kobayashi, “Molecular dynamics studies of molten AgI. I. structure and dynamical properties,” J. Phys. Soc. Jpn. 60, 3725 (1991).
  * Parrinello, Rahman, and Vashishta (1983) M. Parrinello, A. Rahman, and P. Vashishta, “Structural transitions in superionic conductors,” Phys. Rev. Lett. 50, 1073 (1983).
  * c (42) The following Green-Kubo formula was employed to calculate Ag-ion conductivity,  $\sigma=\frac{1}{3Vk_{b}T}\int_{0}^{\infty}\langle\sum_{i}q_{i}\bm{v_{i}}(t^{\prime})\cdot\sum_{i}q_{i}\bm{v_{i}}(t^{\prime}+t)\rangle_{t^{\prime}}dt\rm{,}$ (1) where $v_{i}$ and $q_{i}$ are is velocity and the formal charge of ith ion. $V$ is the volume of the calculation cell. $t$ and $t^{\prime}$ are MD time step, and MD time step for the reference velocity, respectively.The inside of the bracket represents the correlation function of the current for ions.
  * Hoover (1986) W. G. Hoover, “Constant-pressure equations of motion,” Phys. Rev. A 34, 2499–2500 (1986).
  * Obayashi (2018) I. Obayashi, “Volume-optimal cycle: Tightest representative cycle of a generator in persistent homology,” SIAM J. Appl. Algebra Geom. 2, 508–534 (2018).
  * Marcolongo and Marzari (2017) A. Marcolongo and N. Marzari, “Ionic correlations and failure of nernst-einstein relation in solid-state electrolytes,” Phys. Rev. Mater. 1, 025402 (2017).
  * Kubo (1957) R. Kubo, “Statistical-mechanical theory of irreversible processes. i. general theory and simple applications to magnetic and conduction problems,” Journal of the Physical Society of Japan 12, 570–586 (1957).
  * French, Hamel, and Redmer (2011) M. French, S. Hamel, and R. Redmer, “Dynamical screening and ionic conductivity in water from ab initio simulations,” Phys. Rev. Lett. 107, 185901 (2011).
  * Federico and Stefano (2019) G. Federico and B. Stefano, “Topological quantization and gauge invariance of charge transport in liquid insulators,” Nature Physics 15, 967–972 (2019).
  * Pegolo, Grasselli, and Baroni (2020) P. Pegolo, F. Grasselli, and S. Baroni, “Oxidation states, thouless’ pumps, and nontrivial ionic transport in nonstoichiometric electrolytes,” Phys. Rev. X 10, 041031 (2020).
  * c (49) Here, we presume that each Ag ion migrates independently along the edge of the Voronoi polyhedron for I bcc lattice without interacting each others.
  * c (48) In principle, the arrangement of Ag ions is random, but the following two constraints are imposed, (1) The number of Ag ions in a unit cell is limited to two to maintain the stoichiometry within a unit cell. (2) Each Ag ion in the unit cell must not be placed at least on the same surface of Voronoi polyhedron.
  * Leopold  _et al._ (2020) T. Leopold, K. Snehal, P. Elsa, V. Y. Aliaksandr, G. Valeria, G. Fernando, B. Marco, U. Martin, P. H. Sebastiaan, Z. Spyros, S. A. Carl, A. Casper, Welzel, S. Ole, A. P. Carlo, P. Daniele, V. Joost, C. S. Thomas, S. Berend, P. Giovanni, and M. Nicola, “Materials cloud, a platform for open computational science,” Sci. Data 7, 299 (2020).


# Adaptive and Explainable Deployment of Navigation Skills
via Hierarchical Deep Reinforcement Learning

Kyowoon Lee∗1, Seongun Kim∗2 and Jaesik Choi2 *Equal contribution1 Department
of Computer Science and Engineering, Ulsan National Institute of Science and
Technology, Ulsan, Republic of Korea<EMAIL_ADDRESS>Graduate School
of Artificial Intelligence, Korea Advanced Institute of Science and
Technology, 291 Daehak-ro, Daejeon 34141, Republic of Korea. {seongun,
<EMAIL_ADDRESS>Correspondence to: Jaesik Choi

###### Abstract

For robotic vehicles to navigate robustly and safely in unseen environments,
it is crucial to decide the most suitable navigation policy. However, most
existing deep reinforcement learning based navigation policies are trained
with a hand-engineered curriculum and reward function which are difficult to
be deployed in a wide range of real-world scenarios. In this paper, we propose
a framework to learn a family of low-level navigation policies and a high-
level policy for deploying them. The main idea is that, instead of learning a
single navigation policy with a fixed reward function, we simultaneously learn
a family of policies that exhibit different behaviors with a wide range of
reward functions. We then train the high-level policy which adaptively deploys
the most suitable navigation skill. We evaluate our approach in simulation and
the real world and demonstrate that our method can learn diverse navigation
skills and adaptively deploy them. We also illustrate that our proposed
hierarchical learning framework presents explainability by providing semantics
for the behavior of an autonomous agent.

##  I INTRODUCTION

Autonomous navigation of mobile robots has gained much interest due to a wide
variety of important applications in industry. These include assistive robots
[1], a last-mile delivery [2], a guidance at airports [3], and warehouse
navigation [4], among others. To navigate a robot to a desired goal in
complex, cluttered, and highly dynamic environments with multiple static and
dynamic obstacles, a reliable and robust navigation policy is essential.

Traditional approaches to address the navigation problem typically consist of
a series of modules, each of which is specifically designed for solving a
particular sub-task of navigation problems such as human detection, prediction
of future trajectories of humans, and path planning [5, 6]. However, these
approaches require extensive computational efforts and rely on manually
engineered parameters, which limit the ability of mobile robots to operate in
previously unseen environments, or across different robotic platforms, as the
modular system suffers from a lack of generalization and sub-optimal
performance [7]. In this regard, deep reinforcement learning (DRL) approaches
have recently been proposed as learning based solutions for autonomous
navigation, which directly map raw sensor observations to controls and have
shown remarkable results compared to traditional methods and robustness to
sensor noise [8, 9, 10, 11]. However, existing DRL-based approaches require
hand-engineered reward functions that can be exceedingly time consuming.
Moreover, in challenging scenarios, DRL-based approaches with a fixed reward
function easily get stuck in local optima which makes complex situations
including navigating across narrow corridors or dense crowds and turning
corners, unsolvable.

Figure 1: An overview of the proposed hierarchical framework of navigation
skills. A high-level policy invokes low-level navigation skills from a raw
sensory observation. A low-level policy is adopted from a continuous skill
vector among the infinite number of skills, which then drives a robot while
understanding the context.

Another potential problem of existing DRL-based robot navigation is that a
policy represented by a deep neural network often lacks transparency and
cannot provide explanations on decision-making reasons. Recent works
investigate DRL methods which are interpretable and provide decision-making
reasons [12, 13]. However, they are only applicable in video games or in
simple simulation environments. Another line of research provides explanations
in real-world robots by highlighting input features that a deep network most
refers to when making a decision [14]. Nevertheless, this approach has the
limitation that it is not directly relevant to increasing the performance of
the policy model.

To address the aforementioned problems, we develop a framework of learning to
adaptively deploy navigation skills that are explainable with hierarchical
reinforcement learning (HRL).111Source codes are available at
https://github.com/leekwoon/hrl-nav Specifically, our approach can be divided
into two phases. The first phase trains a family of low-level navigation
policies, each of which is optimized for a particular skill vector in a
continuous representation. This skill vector is associated with a
corresponding reward function. For example, a skill vector that imposes an
additional penalty for the robot to be more cautious with obstacles can
exhibit socially-aware navigation behavior, while a skill vector encouraging
the robot getting closer to its goal may result in more aggressive behavior.
In the second phase, we train the high-level policy which deploys the most
suitable navigation skill for every time step. The main contributions of this
work can be summarized as follows:

  * •

Proposal of a hierarchical reinforcement learning approach which learns
diverse navigation skills and deploys them.

  * •

Extensive evaluation of our approach on various scenarios including a real-
world robot navigation which demonstrates effectiveness and explainability of
our approach.

##  II RELATED WORK

###  II-A Reinforcement Learning for Navigation

Recent advances in deep reinforcement learning have enabled the solving of
complex navigation tasks from raw sensory measurements. Long et al. [8]
propose a DRL-based multi-agent navigation framework to train collision
avoidance policy with proximal policy optimization (PPO) [15] using 2D LiDAR
observation. This DRL-based collision avoidance policy is further integrated
into a hybrid control framework [16] and a conventional global planner from
robot operating system (ROS) [17]. A similar approach is proposed in [10],
however, additional ego-safety and social-safety rewards are used to consider
human-awareness. Apart from using 2D LiDAR observation, navigation that learns
from RGB images has also been studied [18, 19].

###  II-B Reinforcement Learning with Sparse Rewards

Existing approaches which handle sparse rewards in reinforcement learning (RL)
can be divided into two categories, curriculum reinforcement learning and
reward shaping. In the context of curriculum reinforcement learning, the
implicit curriculum uses goal relabeling techniques which randomly sample
goals from failed trajectories [20, 21]. The explicit curriculum considers the
difficulty of goals during training, e.g., by generating the goal further from
the start in navigation or increasing the number of obstacles [22, 8]. The
reward shaping technique modifies the reward signal by learning a
parameterized dense reward function [23]. Sibling rivalry [24] uses a model-
free, dynamic reward shaping method that preserves optimal policies on sparse-
rewards tasks. On the other hand, AutoRL [25] uses large-scale hyperparameter
optimization to shape the reward.

###  II-C Hybrid Control framework for Navigation

To further improve the robustness and effectiveness of navigation policy by
leveraging the strength of multiple local planners, a hybrid control framework
has been widely adopted which uses a high-level switching controller to manage
a set of low-level control rules. For example, Jin et al. [26] propose hand-
designed switching rules to combine goal-navigation and obstacle avoidance,
Shucker et al. [27] introduce switching rules to handle challenging cases
where collisions or noises cannot be handled appropriately, and Fan et al.
[16] classify the scenarios into three cases by considering a robot’s sensor
measurement and using both a PID and DRL-based controller. Most similar to our
work, Kastner et al. [28] propose a hierarchical navigation system integrating
model-based and learning-based local planners by training an agent which
decides between multiple local planners. However, the approach uses only one
DRL-based local planner with the traditional model-based planners. As a
result, navigation systems following this approach are often strongly
dependent on deployed model-based planners. Instead, we train a family of
policies that exhibit different behaviors and focus on how to adaptively
deploy them.

##  III PROBLEM FORMULATION

HRL decomposes a general RL into a hierarchy of multiple sub-problems which
themselves are RL problems [29]. Higher-level problems abstract an original RL
problem and adopt which lower-level problems to solve, while lower-level
problems are defined to solve the original problem given an abstraction from
the higher-level problems.

In this work, we decompose a problem of learning navigation skills into a
hierarchy of two sub-problems, a high-level problem and a low-level problem.
The high-level problem is to find a macro policy which understands the context
of a navigation environment and adopts which navigation skills to invoke. The
macro policy observes a current state and demonstrates the context that the
agent encounters. The low-level problem is to discover a set of navigation
skills that drives a mobile robot to reach a target position. Each low-level
navigation skill issues a command by giving primitive actions such as motor
velocities, conditioned on the context distinguished by the macro policy.

We formalize each sub-problem as a Markov decision process (MDP), in
particular, a goal-conditioned MDP [30] for learning general-purpose
navigation skills. A high-/low-level problem is modeled as an MDP
$\langle\mathcal{S}^{\mathrm{high/low}},\mathcal{A}^{\mathrm{high/low}},\mathcal{G},P^{\mathrm{high/low}},R_{g}^{\mathrm{high/low}},\gamma\rangle$,
where $\mathcal{S}$ is the set of states, $\mathcal{G}$ is the set of goals,
$\mathcal{A}$ is the set of actions,
$P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to[0,+\infty)$ is the
transition probability,
$R_{g}:\mathcal{S}\times\mathcal{A}\times\mathcal{A}\to\mathbb{R}$ is the
goal-conditioned reward function, $\gamma\in[0,1]$ is the discount factor, and
each superscript $\mathrm{high}$ and $\mathrm{low}$ represents a notation for
the high-level and the low-level problem respectively.

Figure 2: A two-phase training procedure of the proposed hierarchical
framework. A family of navigation skills is trained in the first phase, while
a high-level policy is trained in the second phase. To train the infinite
number of navigation skills in the first phase, a single policy network which
is conditioned on a continuous skill vector is utilized. The skill vector
$a^{\mathrm{high}}$ is sampled from a fixed distribution per episode to
construct $s^{\mathrm{low}}$ along with a scan observation. The high-level
agent is then trained by taking actions from the low-level skill deployed by
the high-level agent. Figure 3: Actor-critic architecture used for learning a
family of navigation skills (left) and learning to deploy skills (right).
$\mathrm{Conv}$ denotes a convolutional layer and $\mathrm{Fc}$ denotes a
fully-connected layer.

To build the hierarchy, an action from the high-level agent
$a^{\mathrm{high}}\in\mathcal{A}^{\mathrm{high}}$ is observed as a state by
the low-level agent, where this high-level action implies the context of the
environment. In addition to $a^{\mathrm{high}}$, the low-level agent observes
the same state as the high-level agent, which results in
$s^{\mathrm{low}}=(s^{\mathrm{high}},a^{\mathrm{high}})\in\mathcal{S}^{\mathrm{low}}$.
While interacting with the environment and observing the state
$s^{\mathrm{low}}$, each low-level agent receives a reward from a reward
function
$R_{g}^{\mathrm{low}}(s^{\mathrm{low}},a^{\mathrm{low}},a^{\mathrm{high}})$
that consists of multiple reward terms which we will describe in more detail
in the following section. To effectively train each low-level agent to present
discriminative primitive actions conditioned on $a^{\mathrm{high}}$, they are
trained with different reward functions by weighting each reward term in
$R_{g}^{\mathrm{low}}$ by $a^{\mathrm{high}}$.

Under this problem definition, the optimal low-level policies and the high-
level policy in the hierarchy are achieved by an off-the-shelf RL algorithm
which maximizes a corresponding value function formulated as follows:

$\displaystyle
V^{\mathrm{low}}(s^{\mathrm{low}},g|\pi^{\mathrm{high}})\triangleq\mathbb{E}_{\begin{subarray}{c}a_{t}^{\mathrm{low}}\sim\pi^{\mathrm{low}}(a_{t}^{\mathrm{low}}|s_{t}^{\mathrm{low}},g),\\\
a_{t}^{\mathrm{high}}\sim\pi^{\mathrm{high}}(a_{t}^{\mathrm{high}}|s_{t}^{\mathrm{high}},g),\\\
s_{t+1}^{\mathrm{low}}\sim
P(s_{t+1}^{\mathrm{low}}|s_{t}^{\mathrm{low}},a_{t}^{\mathrm{low}})\end{subarray}}$
$\displaystyle\quad\Bigg{[}\sum_{t=0}^{\infty}\gamma^{t}R_{g}^{\mathrm{low}}(s_{t}^{\mathrm{low}},a_{t}^{\mathrm{low}},a_{t}^{\mathrm{high}})\Big{|}s_{0}^{\mathrm{low}}=s^{\mathrm{low}}\Bigg{]},$
(1) $\displaystyle
V^{\mathrm{high}}(s^{\mathrm{high}},g|\pi^{\mathrm{low}})\triangleq\mathbb{E}_{\begin{subarray}{c}a_{t}^{\mathrm{high}}\sim\pi^{\mathrm{high}}(a_{t}^{\mathrm{high}}|s_{t}^{\mathrm{high}},g),\\\
a_{t}^{\mathrm{low}}\sim\pi^{\mathrm{low}}(a_{t}^{\mathrm{low}}|s_{t}^{\mathrm{low}},g),\\\
s_{t+1}^{\mathrm{high}}\sim
P(s_{t+1}^{\mathrm{high}}|s_{t}^{\mathrm{high}},a_{t}^{\mathrm{high}})\end{subarray}}$
$\displaystyle\quad\Bigg{[}\sum_{t=0}^{\infty}\gamma^{t}R_{g}^{\mathrm{high}}(s_{t}^{\mathrm{high}},a_{t}^{\mathrm{high}},a_{t}^{\mathrm{low}})\Big{|}s_{0}^{\mathrm{high}}=s^{\mathrm{high}}\Bigg{]}.$
(2)

##  IV APPROACH

In the hierarchy, the high-level agent and the low-level agent interact with
each other. To train agents in the hierarchy, we decompose a training
procedure into two phases. In this section, we describe how each component of
MDP is defined and how the agent is trained for each training phase.

###  IV-A Learning a Family of Navigation Skills

In the first phase, we learn a family of low-level navigation policies, each
optimized for a particular skill vector in continuous representation. We
present a detailed setup of phase 1 as follows:

####  IV-A1 State space

A state $s_{t}^{\mathrm{low}}$ at time step $t$ consists of four components:
the raw 2D LiDAR measurements $s_{t}^{l}\in\mathbb{R}^{512}$ casting 512 rays
over 360∘ with up to 6m range, the linear and angular velocity of the robot
$s_{t}^{v}\in\mathbb{R}^{2}$, the relative goal state
$s_{t}^{g}\in\mathbb{R}^{2}$ represented in the polar coordinate, and the
high-level action $a_{t}^{\mathrm{high}}\in\mathbb{R}^{5}$. We call
$a_{t}^{\mathrm{high}}$ a skill vector since it induces a specific behavior by
weighting a number of reward terms.

####  IV-A2 Action space

Considering nonholonomic kinematics constraints, an action
$a_{t}^{\mathrm{low}}\in\mathbb{R}^{2}$ is composed of the linear velocity
$v_{t}\in[0,0.5]$ and angular velocity $w_{t}\in[-0.64,0.64]$ which is
normalized into $[-1,1]$ in the neural network.

####  IV-A3 Reward function

Our goal is to learn a family of navigation policies with a diverse set of
behavioral characteristics. Thus, we set the reward function parameterized by
the skill vector $a^{\mathrm{high}}$ as follows:

$\displaystyle
R_{t}^{\mathrm{low}}(s_{t}^{\mathrm{low}},a_{t}^{\mathrm{low}},a_{t}^{\mathrm{high}})$
$\displaystyle=r_{\mathrm{success}}+a^{\mathrm{high}}[r_{\mathrm{collision}}\;r_{\mathrm{progress}}\;r_{v}\;r_{w}\;r_{\mathrm{safety}}]^{\top}.$

The first term, $r_{\mathrm{success}}$, is the success reward, which is 1 if
the goal is achieved and 0 otherwise. The collision reward,
$r_{\mathrm{collision}}$ is -1 when the robot collides with obstacles and 0
otherwise. The progress reward, $r_{\mathrm{progress}}$ is the difference
between the previous distance to the goal and the current distance to the
goal. The driving reward, $r_{v}$ is the positive linear speed which
encourages the robot to drive as fast as possible. The turning reward, $r_{w}$
is the negative angular speed for smooth driving. Finally, inspired by the
social-safety zone in [10], $r_{\mathrm{safety}}$ is the safety reward that
gives a penalty with value $1-\frac{d_{t}}{r+0.5}$ when the robot enters the
safety zone, where $d_{t}$ is the distance to the closest obstacle and $r$ is
the robot radius.

####  IV-A4 Network architecture

The neural network architecture used for learning a family of navigation
skills is illustrated in the right of Fig. 3. We follow the architecture in
the prior work [8], except that the policy and the value function are
additionally parameterized by the skill vector $a^{\mathrm{high}}$ which is
concatenated with the output of the third layer, the current velocity $s_{v}$,
and the relative goal state $s_{g}$.

Figure 4: Training environments used to train the policies, with both
randomized maps and obstacles. We randomize training environments so that the
agent learns diverse skills. To control the difficulty, we manipulate the
number of humans, obstacles, and corridor and the width of corridor.

####  IV-A5 Training procedure

As each individual policy is trained under the similar structure of reward
function, we use a single deep neural network to train the entire family of
low-level policies simultaneously. Inspired by a universal policy [31], we
train a policy
$\pi^{\mathrm{low}}:(s_{t}^{\mathrm{low}},a_{t}^{\mathrm{high}})\rightarrow
a_{t}^{\mathrm{low}}$ that is conditioned on both state of the robot
$s_{t}^{\mathrm{low}}$ and the skill vector $a_{t}^{\mathrm{high}}$. In
contrast to existing approaches which learn a single policy, at the beginning
of the episode, we sample a skill vector from predefined distribution and fix
it during the rollout to train corresponding policy
$\pi_{a_{t}^{\mathrm{high}}}:s_{t}^{\mathrm{low}}\rightarrow
a_{t}^{\mathrm{low}}$, which we call a skill.

###  IV-B Learning to Deploy Skills

Figure 5: Unseen environments used to evaluate the policies. The robot
navigates from the green square to the goal is located on the red square.
Figure 6: Quantitative results in three unseen environments with 100 episodes.
We evaluate the proposed method by comparing four metrics: success rate,
collision rate, time to reach the goal, and path length. Our hierarchical
method (blue bar) presents comparative performance in all evaluation
environments.

The second phase of our approach is to learn a high-level policy to adaptively
deploy the navigation skills learned in phase 1. The following provides a
detailed formulation.

####  IV-B1 State space

A state $s_{t}^{\mathrm{high}}$ composed of three components, $s_{t}^{l}$,
$s_{t}^{v}$ and $s_{t}^{g}$, which are used in the first phase.

####  IV-B2 Action space

An action $a_{t}^{\mathrm{high}}\in\mathbb{R}^{5}$ is a skill vector that
decides the learned low-level behavior characteristic.

####  IV-B3 Reward function

The objective of the second phase is to minimize the time to reach the desired
goal and we use the following sparse reward function:

$\displaystyle
R_{t}^{\mathrm{high}}(s_{t}^{\mathrm{high}},a_{t}^{\mathrm{high}},a_{t}^{\mathrm{low}})=\begin{cases}0&\text{if
reach the goal }\\\ -1&\text{otherwise},\end{cases}$

where the agent gets the reward of 0 when the goal is achieved, and -1
otherwise.

####  IV-B4 Network architecture

We parameterize each of the policies and value functions with the architecture
from the prior work [8], except that the policy predicts a particular skill
vector that will be used by the low-level policy. We further feed the output
of high-level policy as an input to the low-level policy to infer the command
velocity of the robot during a rollout as shown in Fig. 3.

####  IV-B5 Training procedure

After finishing training the set of navigation skills, we can further perform
hierarchical control by training the high-level policy. During a rollout, the
high-level policy predicts the skill vector which decides the behavior
characteristic. This skill vector is observed as an additional input to the
low-level policy which outputs the command velocity of a robot. Note that, in
the second phase, gradients flow only through the high-level policy, not the
low-level policy. Due to a problem with the sparse reward setting where a
positive reward is only provided when the goal is achieved, we use the
hindsight experience replay (HER) technique [20] which provides an effective
way to handle the sparse reward challenge by revisiting previous states in the
replay buffer and storing additional trajectories with hindsight goals for
generating reward signals.

##  V EXPERIMENTS AND RESULTS

Figure 7: Results of the qualitative analysis. We compare the average value of
the skill vector with standard deviation among three evaluation environments.
The high-level agent deploys slow but careful skills in the first environment,
while it deploys smooth but fast skills in the other environments.

###  V-A Setup

We train a policy using an OpenAI-gym-compatible simulator that we specially
design to integrate it into the robot operating system (ROS) with the goal of
open-sourcing it for easy comparison of various approaches including state-of-
the-art learning-based approaches and conventional ones (see Fig. 4). To
create a scenario with various difficulty levels, we adopt the map generation
method from [28] to generate indoor (50m$\times$50m) and outdoor
(20m$\times$20m) maps. We randomize the noise on sensors, and static and
dynamic obstacles where we control the motion of the dynamic obstacles using a
learning-based collision avoidance method that extends the prior work [8] with
$A^{*}$ global planner to make it follow the sub-goal generated from the
global path for more realistic long-range behavior. We terminate the
simulation when reaching the goal, colliding with an obstacle, or exceeding
1000 time steps.

Figure 8: Trajectories of the our approach in the unseen building environment
(top) and skill vectors in particular situations (bottom). Figure 9:
Demonstration of our approach in unseen real scenarios. The scenarios are
navigating in (a) cluttered environment, (b) narrow corridor, (c) dense crowd
and (d) open space. The skill vectors are predicted by transferred high-level
policy.

To train the policies, we use soft actor-critic (SAC) [32] with an
automatically tuned entropy coefficient, employing the Adam optimizer [33]
with a learning rate 1e-4 for both the actor and the critic, a target
smoothing coefficient of 5e-4, and a replay buffer size of 1M.

We compare our approach with a traditional model-based approach dynamic window
approach (DWA), two DRL-based approaches LONG [8] and GRING [17], and four
agents trained with a fixed skill vector which we term $\pi_{\mathrm{goal-
oriented}}$, $\pi_{\mathrm{socially-aware}}$, $\pi_{\mathrm{safe}}$, and
$\pi_{\mathrm{mild}}$ where each policy is trained with the skill vector of
$[0,0.2,0.1,0,0]$, $[0.1,0,0,0.1,0.2]$, $[0.2,0,0,0.1,0.1]$, and
$[0.1,0.1,0.1,0.2,0]$ respectively.

###  V-B Quantitative Analysis

We evaluate the trained policies in three unseen large environments (see Fig.
5). These scenarios include (a) a corridor with 4 dynamic obstacles, (b) a
building with 9 dynamic obstacles, and (c) a shopping mall with 8 dynamic
obstacles which are designed to evaluate the ability of the agent including
global planning, collision avoidance, and safe navigation, among other real-
world situations. We validate the effectiveness of our method with four
metrics: success rate, collision rate, arriving time, and path length. Fig. 6
shows the evaluation results with 100 episodes in each scenario. To solely
evaluate the performance of learned navigation policy, all approaches do not
use a global planner that is frequently used to handle a long-range navigation
challenge. A traditional model-based approach DWA fails in dense crowds
scenarios. Further, in all evaluation environments, our approach shows
comparative performance to other baselines including DRL-based agents and
agents trained with the fixed skill vector, except for goal-oriented policy.
LONG and GRING present comparative results in densely crowded environments
where a local navigation policy can solve but performance drops significantly
for a situation where a global navigation policy is required. An interesting
observation is that $\pi_{\mathrm{goal-oriented}}$ performs well in long-range
and dynamic environments without crash or safety rewards typically used in
prior works. We assume that this is due to the goal-oriented reward, which
encourages exploration and goal achievement. It would be an appealing research
direction to study the role of reward terms rigorously.

###  V-C Qualitative Analysis

To qualitatively analyze the effectiveness of the proposed approach, we
interpret the average skill vector along the trajectory for each unseen
evaluation environment in Fig. 7. In the corridor environment, there is a long
wall. To successfully reach the goal, the robot needs to detour a long range.
To this end, the high-level agent deploys skills that present slow but careful
motor commands with relatively high crash skill value. On the other hand, in
the other two environments, the building and the shopping mall, there are
little static obstacles on the straight line from the initial position of the
robot to the target position. In these environments, the skill vector has
large progress and forward values so that the robot presents agile movement.
In addition, the high-level agent demonstrates smooth motor command through
the smooth skill value. The average discomfort skill value in the last two
crowded environments is relatively large, which validates the effectiveness of
our approach.

The whole trajectories of our approach in the building environment are shown
in Fig. 8 where green lines represent trajectories of successes. The skill
vectors demonstrate the interpretable deployment of learned navigation skills.
At the beginning where there are few obstacles around the robot as shown in
(a) of Fig. 8, progress and forward skill values are huge for fast navigation
towards the goal. In the middle of the navigation shown in (b), though
progress and forward skill values are still large, the discomfort value
increases when the robot is surrounded by humans, which would lead the robot
to avoid collision with them. When the robot enters the narrow corridor in
(c), the smooth value increases to prevent oscillatory behaviors. Around the
goal which is close to obstacles in (d), crash is high to safely arrive the
goal. The skill vector presents the same tendency across the other evaluation
scenarios. We refer the reader to the videos in supplementary material for
further information.

###  V-D Real-World Experiment

To demonstrate the validity of the proposed approach, we collect the expert
trajectory with our robot which has two 270∘ LiDARs that are merged together
to cover 360∘. We then evaluate how well the high-level policy understands the
context in a real-world shopping mall. Fig. 9 shows the skill vector generated
by high-level policy for different scenarios. The high-level policy encourages
the robot to keep a safe distance from crowds and to decrease the speed in
narrow corridors. On the other hand, the goal-oriented navigation skill with
high speed is preferred in open space.

##  VI CONCLUSION

In this paper, we propose a hierarchical learning framework to train
reinforcement learning based navigation policies that can present
explainability by providing semantics of a behavior of an autonomous agent and
can reduce the effort to design hand-engineered reward functions. The
evaluation results on unseen environments including real-world cases
demonstrate the adaptive and explainable deployment of learning navigation
skills.

## ACKNOWLEDGMENT

This work was supported by the Industry Core Technology Development Project,
20005062, Development of Artificial Intelligence Robot Autonomous Navigation
Technology for Agile Movement in Crowded Space, funded by the Ministry of
Trade, Industry & Energy (MOTIE, Republic of Korea) and was supported by
Institute of Information & communications Technology Planning & Evaluation
(IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00984,
Development of Artificial Intelligence Technology for Personalized Plug-and-
Play Explanation and Verification of Explanation, No.2019-0-00075, Artificial
Intelligence Graduate School Program (KAIST)).

## References

  * [1] D. Feil-Seifer and M. J. Matarić, “Socially assistive robotics,” _IEEE Robotics & Automation Magazine_, vol. 18, no. 1, pp. 24–31, 2011\.
  * [2] T. Hoffmann and G. Prause, “On the regulatory framework for last-mile delivery robots,” _Machines_ , vol. 6, no. 3, p. 33, 2018.
  * [3] S. Kayukawa, K. Higuchi, J. Guerreiro, S. Morishima, Y. Sato, K. Kitani, and C. Asakawa, “Bbeep: A sonic collision avoidance system for blind travellers and nearby pedestrians,” in _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ , 2019, pp. 1–12.
  * [4] W. Xiao-Long, W. Chun-Fu, L. Guo-Dong, and C. Qing-Xie, “A robot navigation method based on rfid and qr code in the warehouse,” in _2017 Chinese Automation Congress (CAC)_. IEEE, 2017, pp. 7837–7840.
  * [5] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, “Social gan: Socially acceptable trajectories with generative adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2255–2264.
  * [6] L. Liu, D. Dugas, G. Cesari, R. Siegwart, and R. Dubé, “Robot navigation in crowded environments using deep reinforcement learning,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 5671–5677.
  * [7] A. J. Sathyamoorthy, U. Patel, T. Guan, and D. Manocha, “Frozone: Freezing-free, pedestrian-friendly navigation in human crowds,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 4352–4359, 2020.
  * [8] P. Long, T. Fan, X. Liao, W. Liu, H. Zhang, and J. Pan, “Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 6252–6259.
  * [9] T. Fan, X. Cheng, J. Pan, P. Long, W. Liu, R. Yang, and D. Manocha, “Getting robots unfrozen and unlost in dense pedestrian crowds,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1178–1185, 2019.
  * [10] J. Jin, N. M. Nguyen, N. Sakib, D. Graves, H. Yao, and M. Jagersand, “Mapless navigation among dynamics with social-safety-awareness: a reinforcement learning approach from 2d laser scans,” in _2020 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2020, pp. 6979–6985.
  * [11] D. Dugas, J. Nieto, R. Siegwart, and J. J. Chung, “Navrep: Unsupervised representations for reinforcement learning of robot navigation in dynamic human environments,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 7829–7835.
  * [12] D. Lee, H. Tang, J. O. Zhang, H. Xu, T. Darrell, and P. Abbeel, “Modular architecture for starcraft ii with deep reinforcement learning,” in _Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference_ , 2018.
  * [13] D. Trivedi, J. Zhang, S.-H. Sun, and J. J. Lim, “Learning to synthesize programs as interpretable and generalizable policies,” _Advances in neural information processing systems_ , vol. 34, pp. 25 146–25 163, 2021.
  * [14] S. Kim and J. Choi, “Explaining the decisions of deep policy networks for robotic manipulations,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 2663–2669.
  * [15] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [16] T. Fan, P. Long, W. Liu, and J. Pan, “Fully distributed multi-robot collision avoidance via deep reinforcement learning for safe and efficient navigation in complex scenarios,” _arXiv preprint arXiv:1808.03841_ , 2018.
  * [17] R. Guldenring, M. Görner, N. Hendrich, N. J. Jacobsen, and J. Zhang, “Learning local planners for human-aware navigation in indoor environments,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 6053–6060.
  * [18] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 3357–3364.
  * [19] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik, “Cognitive mapping and planning for visual navigation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2616–2625.
  * [20] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hindsight experience replay,” _Advances in neural information processing systems_ , vol. 30, 2017\.
  * [21] M. Fang, C. Zhou, B. Shi, B. Gong, J. Xu, and T. Zhang, “Dher: Hindsight experience replay for dynamic goals,” in _International Conference on Learning Representations_ , 2018.
  * [22] C. Florensa, D. Held, X. Geng, and P. Abbeel, “Automatic goal generation for reinforcement learning agents,” in _International conference on machine learning_. PMLR, 2018, pp. 1515–1528.
  * [23] A. Y. Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in _Icml_ , vol. 99, 1999, pp. 278–287.
  * [24] A. Trott, S. Zheng, C. Xiong, and R. Socher, “Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [25] H.-T. L. Chiang, A. Faust, M. Fiser, and A. Francis, “Learning navigation behaviors end-to-end with autorl,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 2007–2014, 2019.
  * [26] J. Jin, Y. Kim, S. Wee, D. Lee, and N. Gans, “A stable switched-system approach to collision-free wheeled mobile robot navigation,” _Journal of Intelligent & Robotic Systems_, vol. 86, no. 3, pp. 599–616, 2017.
  * [27] B. Shucker, T. Murphey, and J. K. Bennett, “Switching rules for decentralized control with simple control laws,” in _2007 American Control Conference_. IEEE, 2007, pp. 1485–1492.
  * [28] L. Kastner, J. Cox, T. Buiyan, and J. Lambrecht, “All-in-one: A drl-based control switch combining state-of-the-art navigation planners,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 2861–2867.
  * [29] B. Hengst, _Hierarchical Reinforcement Learning_. Boston, MA: Springer US, 2010, pp. 495–502.
  * [30] T. Schaul, D. Horgan, K. Gregor, and D. Silver, “Universal value function approximators,” in _International Conference on Machine Learning_. PMLR, 2015, pp. 1312–1320.
  * [31] W. Yu, J. Tan, C. K. Liu, and G. Turk, “Preparing for the unknown: Learning a universal policy with online system identification,” _arXiv preprint arXiv:1702.02453_ , 2017.
  * [32] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1861–1870.
  * [33] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.