Jan Chwedeńczuk

Faculty of Physics, University of Warsaw

ul. Pasteura 5, PL–02–093 Warszawa, Poland

# Many-body Bell inequalities for bosonic qubits

###### Abstract

Since John Bell formulated his paramount inequality for a pair of spin-$1/2$
particles, quantum mechanics has been confronted with the postulates of local
realism with various equivalent configurations. Current technology, with its
advanced manipulation and detection methods, allows to extend the Bell tests
to more complex structures. The aim of this work is to analyze a set of Bell
inequalities suitable for a possibly broad family of many-body systems with
the focus on bosonic qubits. We develop a method that allows for a step-by-
step study of the many-body Bell correlations, for instance among atoms
forming a two-mode Bose-Einstein condensate or between photons obtained from
the parametric-down conversion. The presented approach is valid both for cases
of fixed and non-fixed number of particles, hence it allows for a thorough
analysis of quantum correlations in a variety of many-body systems.

##  1 Introduction

The consequences of the quantum-mechanical description of the physical world
are often puzzling or even troubling. In their famous paper dating back to
1935 [1], Albert Einstein, Boris Podolsky and Nathan Rosen (EPR) investigated
one such counter-intuitive prediction, related to what Einstein called the
“spooky action at the distance”. The EPR trio asked if the theory that allows
for such bizarre phenomena should be extended to a more complete model of
reality. Later that year, Erwin Schrödinger introduced the concept of
entanglement, as a fundamental ingredient of those exotic and apparently non-
local effects [2]. Only 29 years later John Bell showed that there are quantum
states, such as a singlet state of two spin-$1/2$ particles, that contain
correlations too strong for a locally realistic theory. Hence, the extension
of quantum mechanics, suggested by the EPR trio, to a theory that would be
consistent with the postulates of local realism, is not possible [3].

The Bell’s achievement is important because of its implications for our
understanding of quantum mechanics, but also because it provides a rather
simple recipe for testing the nonlocality of the theory. Its experimentally-
friendly version [4], known as the CHSH inequality (John Clauser, Michael
Horne, Abner Shimony and Richard Holt) has been used in numerous
configurations, some prominent examples are [5, 6, 7, 8, 9, 10, 11, 12, 13,
14, 15, 16] while [17] contains more references. Recently, Bell tests closing
the loopholes, including that related to the spatio-temporal separation, have
been reported [18, 19, 20, 21, 22, 23].

These experiments operate on pairs of entangled qubits: photons obtained in
the parametric-down conversion [24, 25, 26], electrons, Josephson junctions
[27] or nitrogen vacancies [28]. Now that we have gained extensive control
over complex quantum systems, Bell correlations can be tested on the many-body
level. This calls for a new set of Bell inequalities, tailored for such
purposes, and a versatile method has been developed in [29, 30, 31]. It
assumes that the system is composed of a collection of qubits and allows to
investigate the many-body entanglement, EPR-steering and Bell correlations in
a systematic way [32, 33].

Here we adapt these inequalities to bosonic systems and provide a tool for
detecting the Bell correlations with correlation functions of any desired
order. The presented method could be implemented in Bose-Einstein condensates
(BECs) trapped in a double-well potential or in an optical lattice [34]. We
show how to extend this approach to systems where the number of particles is
not fixed. This way, it could be used to detect the entanglement and the Bell
correlations in a multi-photon parametric-down-conversion state, without the
need to restrict to a single pair of photons. Similarly, an ideal test ground
for these inequalities would be a collection of atomic pairs, scattered from a
BEC [35, 36, 37, 38, 39, 40, 41, 42].

This manuscript is organized as follows. We start with Section 2, where we
introduce the theoretical tools necessary to analyze many-body Bell
correlations in systems of bosonic qubits. We introduce a family of many-body
Bell inequalities in Secction 2.1. First, we focus on the case, when parties
can be addressed individually, see Section 2.1.1. Next, in Section 2.1.2 we
consider the permutationally invariant case. In Section 2.1.3 we discuss the
case when the parties can be labeled with additional degree(s) of freedom. In
Section 2.2 we show how these inequalities can be tested with quantum systems.
In particular, in Section 2.2.3 we show how to adapt these tools to the case
when the number of qubits is not fixed. Next, we move to Section 3, where we
apply these techniques to some prominent examples: a BEC confined in a double-
well potential (see Section 3.1) or a multi-photon parametric-down conversion
signal (Section 3.2). We present some closing remarks in Section 4 and
conclude with Section 5.

##  2 Theoretical toolbox

In this section we develop all the theoretical tools necessary to
systematically analyze Bell correlations in complex many-body systems of
bosonic qubits.

###  2.1 Many-body Bell inequalities

We begin with a discussion of a broad family of Bell inequalities that can be
used in systems of individually accessible qubits, such as spin chains. Later,
we generalize these inequalities to systems of identical particles that cannot
be addressed one-by-one.

####  2.1.1 Individually addressable parties

Consider a pair of binary physical quantities: $\sigma^{(k)}_{1}=\pm 1$ and
$\sigma^{(k)}_{2}=\pm 1$, independently measured for $m$ particles (thus
$k\in\\{1,\ldots.m\\}$). Take the product of these outcomes

$\displaystyle\Sigma_{m}=\prod_{k=1}^{m}\sigma^{(k)}_{s_{k}},$ (1)

where
$\sigma^{(k)}_{s_{k}}=\frac{1}{2}(\sigma_{1}^{(k)}+(s_{k})i\sigma_{2}^{(k)})$
and the sign $s_{k}=\pm 1$ is chosen by each party separately. If the average
of $\Sigma_{m}$ over many experimental realizations is consistent with a local
realistic theory, it can be reproduced by taking a mean over some probability
distribution $p(\lambda)$ of a “hidden variable” $\lambda$, namely

$\displaystyle\left\langle\Sigma_{m}\right\rangle=\int\\!d\lambda\,p(\lambda)\prod_{k=1}^{m}\sigma^{(k)}_{s_{k}}(\lambda).$
(2)

As $|\sigma^{(k)}_{s_{k}}|^{2}=\frac{1}{2}$, then using the Cauchy-Schwarz
inequality in the integral form, namely

$\displaystyle|\int d\lambda f(\lambda)g(\lambda)|^{2}\leqslant\int
d\lambda|f(\lambda)|^{2}\int d\lambda|g(\lambda)|^{2}$ (3)

and setting $f(\lambda)=\sqrt{p(\lambda)}$ and
$g(\lambda)=\sqrt{p(\lambda)}\,\Sigma_{m}$, we obtain

$\displaystyle\mathcal{E}_{m}\equiv|\left\langle\Sigma_{m}\right\rangle|^{2}\leqslant\int\\!d\lambda\,p(\lambda)\prod_{k=1}^{m}|\sigma^{(k)}_{s_{k}}(\lambda)|^{2}=2^{-m}.$
(4)

Since this bound holds for all correlators as in Eq. (2), it is a many-body
Bell inequality, and it will be our main tool for studying Bell correlations
in complex systems throughout this work. For the original derivation of this
inequality and its extended discussion, see [29, 43, 31]. The inequality from
Eq. (4) is a member of a broader family of $N$-body Bell inequalities for
qubits [44].

####  2.1.2 Permutationally invariant case

Next, we assume that the $m$ parties are a subset of a larger set of $N$,
$m\leqslant N$. Our aim is to derive a Bell inequality that would be invariant
upon the permutation of any pair of parties. To this end, consider a
correlator of $m$ outcomes

$\displaystyle\left\langle\Sigma_{\vec{k}}\right\rangle=\left\langle\sigma^{(k_{1})}_{s_{k_{1}}}\ldots\sigma^{(k_{m})}_{s_{k_{m}}}\right\rangle.$
(5)

Note that rather than labelling $\Sigma$ with a single number $m$, as in Eq.
(1), we now use $\vec{k}=(k_{1},\ldots,k_{m})$, which denotes one particular
choice of $m$ parties out of $N$. As previously, every party $k_{i}$ has a
freedom of choice of the sign $s_{k_{i}}$. Next, we introduce a
permutationally invariant correlator

$\displaystyle\left\langle\tilde{\Sigma}_{m}\right\rangle=\left\langle\hat{\mathcal{S}}^{(N)}_{\vec{k}}\Sigma_{\vec{k}}\right\rangle=\left\langle\hat{\mathcal{S}}^{(N)}_{\vec{k}}\sigma^{(k_{1})}_{s_{k_{1}}}\ldots\sigma^{(k_{m})}_{s_{k_{m}}}\right\rangle.$
(6)

The symmetrization operator $\hat{\mathcal{S}}^{(N)}_{\vec{k}}$ sums over all
possible $N\choose m$ choices of $m$ parties out of $N$. Note that the
symmetrization allows for all products
$\sigma^{(k_{1})}_{s_{k_{1}}}\ldots\sigma^{(k_{m})}_{s_{k_{m}}}$, including
those that are unordered. Hence, for every fixed $\vec{k}$ there are $m!$
equivalent correlators and the total number of permutations is
$\|\hat{\mathcal{S}}^{(N)}_{\vec{k}}\|={N\choose m}m!=\frac{N!}{(N-m)!}$. The
Bell inequality now reads

$\displaystyle\tilde{\mathcal{E}}_{m}=|\left\langle\tilde{\Sigma}_{m}\right\rangle|^{2}=|\left\langle\hat{\mathcal{S}}^{(N)}_{\vec{k}}\Sigma_{\vec{k}}\right\rangle|^{2}\leqslant\|\hat{\mathcal{S}}^{(N)}_{\vec{k}}\|^{2}2^{-m}=\left(\frac{N!}{(N-m)!}\right)^{2}2^{-m}.$
(7)

####  2.1.3 More degrees of freedom

We now generalize the above findings to situations, where parties, besides
yielding the binary outcomes, can be labeled with another degree(s) of
freedom. A good illustration of such setup would be a collection of photons
emitted in the spontaneous parametric down conversion (SPDC) process, where
apart from the two orthogonal polarizations (binary results), emitted
particles can occupy distinct momentum states. Another example is a collection
of atomic pairs scattered / emitted from an $F=1$, $m_{F}=0$ Bose-Einstein
condensate into states with opposite momenta in a spin-changing collision
($m_{F}=\pm 1$) [41].

Our aim is to adapt the correlator from Eq. (5) to account for these
additional degrees of freedom. To this end we first consider the simplest
possible case, namely parties marked with one label which is either $A$ or
$B$—denoting two separate regions of space, distinct momenta, etc. We will
address the question of extending these arguments to more complex
configurations later on. The goal is to detect the Bell correlations in the
system, but we focus on its particular form — such that extends over the
regions. Thus we propose the correlator between $m$ parties in $A$ and $k$ in
$B$ in the following form

$\displaystyle\tilde{\mathcal{E}}_{m,k}=|\left\langle\tilde{\Sigma}_{m}^{(A)}\tilde{\Sigma}_{k}^{(B)}\right\rangle|^{2},$
(8)

where $\tilde{\Sigma}_{m/m}^{(A/B)}$ is defined in Eq. (6). According to Eq.
(7), the Bell inequality now reads

$\displaystyle\tilde{\mathcal{E}}_{m,k}$
$\displaystyle=|\left\langle\tilde{\Sigma}_{m}^{(A)}\tilde{\Sigma}_{k}^{(B)}\right\rangle|^{2}=|\left\langle\hat{\mathcal{S}}^{(N_{A})}_{\vec{k}}\Sigma^{(A)}_{\vec{k}}\hat{\mathcal{S}}^{(N_{B})}_{\vec{k}^{\prime}}\Sigma^{(B)}_{\vec{k}^{\prime}}\right\rangle|^{2}\leqslant\|\hat{\mathcal{S}}^{(N_{A})}_{\vec{k}}\|^{2}\|\hat{\mathcal{S}}^{(N_{A})}_{\vec{k}}\|^{2}2^{-(m+k)}$
$\displaystyle=\left(\frac{N_{A}!}{(N_{A}-m)!}\frac{N_{B}!}{(N_{B}-m)!}\right)^{2}2^{-(m+k)},$
(9)

where $N_{A}$ and $N_{B}$ is the number of particles per region.

Finally, we note that the approach outlined in the above paragraphs can be
extended to more complicated situations where more regions / degrees of
freedom are in play. To detect the Bell correlations between $\mu$ regions
$A_{1}\ldots A_{\mu}$ one could consider a correlator

$\displaystyle\tilde{\mathcal{E}}_{m_{1},\ldots
m_{\mu}}=|\left\langle\tilde{\Sigma}_{m_{1}}^{(A_{1})}\ldots\tilde{\Sigma}_{m_{\mu}}^{(A_{\mu})}\right\rangle|^{2},$
(10)

and then proceed analogically to Eq. (2.1.3).

###  2.2 Testing with quantum systems

We now discuss in detail how these local realistic bounds can be tested in
complex many-body systems of qubits.

####  2.2.1 Quantum: single degree of freedom

The inequality (4) is well-suited to test Bell correlations in many-body
systems of individually addressable qubits, such as spin chains or optical
lattices with one two-level atom per site. With $\hat{\sigma}_{s_{k}}^{(k)}$
being the Pauli rising ($s_{k}=+1$) / lowering ($s_{k}=-1$) operator for the
$k$-th qubit, the correlator is

$\displaystyle\mathcal{E}^{(q)}_{m}=|\left\langle\hat{\sigma}_{s_{1}}^{(1)}\ldots\hat{\sigma}_{s_{m}}^{(m)}\right\rangle|^{2},$
(11)

where the symbol $\left\langle\,\cdot\,\right\rangle$ denotes the average
calculated over a quantum state and the upper index $(q)$ indicates a quantum-
mechanical correlator. Among states which violate the bound from Eq. (4), the
GHZ state

$\displaystyle|\psi\rangle=\frac{1}{\sqrt{2}}\left(|\uparrow\rangle^{\otimes
N}+|\downarrow\rangle^{\otimes N}\right)$ (12)

stands out, giving a maximal value of the correlator $\mathcal{E}^{(q)}_{m}$
for $m=N$ and either $s_{k}=1$ or $s_{k}=-1$ for all $k$ simultaneously, for
instance

$\displaystyle\mathcal{E}^{(q)}_{N}=|\left\langle\hat{\sigma}_{+}^{(1)}\ldots\hat{\sigma}_{+}^{(N)}\right\rangle|^{2}=\frac{1}{4}.$
(13)

On the other hand, the inequality (7) is tailored for bosonic systems of
qubits, where particles cannot be addressed individually and occupy a single
common mode (i.e., there are no other degrees of freedom apart from
$|\uparrow\rangle$ and $|\downarrow\rangle$ for each particle), given that all
$s_{k_{i}}$ are equal. The dimensionality of the Hilbert space reduces from
$2^{N}$ to $N+1$ and all the states can be represented in a form

$\displaystyle\hat{\varrho}=\sum_{n,m=0}^{N}\varrho_{nm}|n,N-n\rangle\\!\langle
m,N-m|,$ (14)

where $|n,N-n\rangle$ is a (symmetrized) state of $n$ qubits in
$|\uparrow\rangle$ and $N-n$ in $|\downarrow\rangle$ while $\varrho_{nm}$ is
the corresponding matrix element. Bosonic operators $\hat{a}^{\dagger}$ and
$\hat{b}^{\dagger}$ create a single excitation in either $|\uparrow\rangle$ or
$|\downarrow\rangle$ state and the symmetrized equivalent of the single-qubit
rising / lowering operator $\hat{\sigma}_{\pm}^{(k)}$ is

$\displaystyle\hat{J}_{+}=\hat{a}^{\dagger}\hat{b}=\sum_{k=1}^{N}\hat{\sigma}_{+}^{(k)},$
(15a)
$\displaystyle\hat{J}_{-}=\hat{a}\hat{b}^{\dagger}=\sum_{k=1}^{N}\hat{\sigma}_{-}^{(k)}.$
(15b)

Since only the collective operations are now allowed, we must replace the
product of $m$ operators addressing single qubits as in Eq. (11) with

$\displaystyle\hat{\sigma}_{s_{1}}^{(1)}\ldots\hat{\sigma}_{s_{m}}^{(m)}\longrightarrow\hat{J}_{+}^{m}$
(16)

(or $\hat{J}_{-}^{m}$). Note that

$\displaystyle\hat{J}_{+}^{m}=\left(\sum_{k=1}^{N}\hat{\sigma}_{+}^{(k)}\right)^{m}=\hat{\mathcal{S}}^{(N)}_{\vec{k}}\hat{\sigma}^{(k_{1})}_{+}\ldots\hat{\sigma}^{(k_{m})}_{+},$
(17)

which is exactly a quantum equivalent of the averaged product in Eq. (6) with
all $s_{k_{i}}=+1$. Hence the Bell inequality from Eq. (7) can be tested with
quantum systems of bosonic qubits by seeking for violations of the following
inequality

$\displaystyle\tilde{\mathcal{E}}^{(q)}_{m}=|\left\langle\hat{J}_{+}^{m}\right\rangle|^{2}\leqslant\left(\frac{N!}{(N-m)!}\right)^{2}2^{-m}.$
(18)

####  2.2.2 Quantum: more degrees of freedom

Analogically, the cross-region Bell correlations can be tested with the
quantum correlator as below

$\displaystyle\tilde{\mathcal{E}}^{(q)}_{m,k}=|\left\langle\hat{J}_{+}^{(A)m}\hat{J}_{+}^{(B)k}\right\rangle|^{2},$
(19)

where $\hat{J}_{+}^{(A/B)}$ are local operators for $A/B$. Using the Bell
inequality from Eq. (8), a quantum system reveals Bell correlations if it
violates

$\displaystyle\tilde{\mathcal{E}}^{(q)}_{m,k}\leqslant\left(\frac{N_{A}!}{(N_{A}-m)!}\right)^{2}\left(\frac{N_{B}!}{(N_{B}-k)!}\right)^{2}2^{-(m+k)},$
(20)

However, does the breaking of the bound (20) imply the $A/B$ nonlocality? To
answer this question, consider a following state

$\displaystyle|\psi\rangle=\left[\frac{|\uparrow\rangle^{\otimes
N}+|\downarrow\rangle^{\otimes
N}}{\sqrt{2}}\right]_{A}\otimes\left[\frac{|\uparrow\rangle+|\downarrow\rangle}{\sqrt{2}}\right]_{B}$
(21)

of $N$ qubits in $A$ forming the maximally entangled GHZ state and a single
particle in $B$, uncorrelated with the rest, which is in a coherent
superposition of $|\uparrow\rangle$ and $|\downarrow\rangle$. For this state,
we obtain

$\displaystyle\tilde{\mathcal{E}}^{(q)}_{N,1}=|\left\langle\hat{J}_{+}^{(A)N}\hat{J}_{+}^{(B)}\right\rangle|^{2}=(N!)^{2}2^{-4}.$
(22)

This should be compared with the local realistic bound from Eq. (20) for $m=N$
and $k=1$, which is
$|\left\langle\hat{J}_{+}^{(A)N}\hat{J}_{+}^{(B)}\right\rangle|^{2}\leqslant(N!)^{2}2^{-(N+1)}$
and is violated by the right-hand side of Eq. (22) for $N>3$. This violation
is a result of the Bell correlations between the qubits located at $A$, rather
than the $A/B$ nonlocality [45, 46, 47].

Thus one must use Eq. (20) with some precaution in order to detect the genuine
inter-region nonlocality: if the single-region correlators are Bell-bounded,
i.e.,

$\displaystyle\tilde{\mathcal{E}}^{(q)}_{m}=|\left\langle\hat{J}_{+}^{(A)m}\right\rangle|^{2}\leqslant\left(\frac{N_{A}!}{(N_{A}-m)!}\right)^{2}2^{-m}$
(23a)
$\displaystyle\tilde{\mathcal{E}}^{(q)}_{k}=|\left\langle\hat{J}_{+}^{(B)k}\right\rangle|^{2}\leqslant\left(\frac{N_{B}!}{(N_{B}-k)!}\right)^{2}2^{-k}$
(23b)

and simultaneously the inequality from Eq. (20) is violated, then the non-
locality stems from the cross-region correlations.

####  2.2.3 Quantum: non-fixed number of qubits

We now show how the correlator from Eq. (19) can detect the $A/B$ nonlocality
in a system where the number of qubits is not fixed. To illustrate this point,
we consider a process of scattering of pairs of qubits into $A$ and $B$, with
the perfect correlation in the $\uparrow/\downarrow$ degree of freedom, namely

$\displaystyle|\psi_{AB}\rangle=\sum_{n,m=0}^{\infty}C_{n,m}|n:\uparrow\rangle_{A}|n:\downarrow\rangle_{B}|m:\downarrow\rangle_{A}|m:\uparrow\rangle_{B}\equiv\sum_{n,m=0}^{\infty}C_{n,m}|n,m\rangle_{A}|m,n\rangle_{B},$
(24)

where the ket $|n,m\rangle_{A}$ denotes $n$ bosonic qubits in the
$|\uparrow\rangle$ state and $m$ in $|\downarrow\rangle$, located in $A$ (and
analogically for $B$). The coefficients $C_{n,m}$ are determined by the
details of the scattering process.

To account for the qubit-number fluctuations, we split the sum over $n$ and
$m$ in Eq. (24) in such a way so that first we introduce $N$, the number of
particles in $A$ or $B$, then we sum over $n$, with $m=N-n$, and finally over
all possible $N$’s, which gives

$\displaystyle|\psi_{AB}\rangle$
$\displaystyle=\sum_{N=0}^{\infty}\sum_{n=0}^{N}C_{n,N-n}|n,N-n\rangle_{A}|N-n,n\rangle_{B}$
(25)
$\displaystyle=\sum_{N=0}^{\infty}\sqrt{p_{N}}\sum_{n=0}^{N}\mathcal{A}_{n}^{N}|n,N-n\rangle_{A}|N-n,n\rangle_{B},$

where we used the probability for having $N$ particles either in $A$ or $B$,

$\displaystyle p_{N}=\sum_{n=0}^{N}|C_{n,N-n}|^{2}\ \ \ \mathrm{and}\ \ \
\mathcal{A}_{n}^{N}=\frac{C_{n,N-n}}{\sqrt{p_{N}}}.$ (26)

Since operators $\hat{J}_{+}^{(A/B)}$ do not locally modify the number of
particles, superpositions between different $N$’s are not revealed by the
correlator from Eq. (19). This means that in this context the state from Eq.
(25) can be replaced by an incoherent mixture

$\displaystyle\hat{\varrho}_{AB}=\sum_{N=0}^{\infty}p_{N}\hat{\varrho}_{AB}^{(N)},$
(27)

where the fixed-$N$ density matrix is

$\displaystyle\hat{\varrho}_{AB}^{(N)}=\sum_{n,m=0}^{N}\mathcal{A}_{n}^{N}\mathcal{A}_{m}^{N*}|n,N-n\rangle\\!\langle
m,N-m|_{A}\otimes|N-n,n\rangle\\!\langle N-m,m|_{B}.$ (28)

The fact that the pure state from Eq. (25) is, from the point of view of the
Bell test, equivalent with the mixture (27), is a manifestation of the
superselection rules, which render coherences between states with different
number of particles as nonphysical [48, 49, 50, 51, 52].

We now proceed to establish an upper bound for the correlator
$\tilde{\mathcal{E}}^{(q)}_{m,k}$ in absence of non-local Bell correlations.
Using Eqs (19), (27) and the Cauchy-Schwarz inequality, which in the discrete
case takes the form of

$\displaystyle|\sum_{N}a_{N}b_{N}|^{2}\leqslant\sum_{N}|a_{N}|^{2}\sum_{N}|b_{N}|^{2}$
(29)

and setting $a_{N}=\sqrt{p_{N}}$ and $b_{N}=\sqrt{p_{N}}\,{\rm
Tr}\left[\hat{\varrho}_{AB}^{(N)}\hat{J}_{+}^{(A)m}\hat{J}_{+}^{(B)k}\right]$,
we obtain

$\displaystyle\tilde{\mathcal{E}}^{(q)}_{m,k}$
$\displaystyle=|\sum_{N=0}^{\infty}p_{N}{\rm
Tr}\left[\hat{\varrho}_{AB}^{(N)}\hat{J}_{+}^{(A)m}\hat{J}_{+}^{(B)k}\right]|^{2}\leqslant\sum_{N=0}^{\infty}p_{N}|{\rm
Tr}\left[\hat{\varrho}_{AB}^{(N)}\hat{J}_{+}^{(A)m}\hat{J}_{+}^{(B)k}\right]|^{2}$
$\displaystyle\equiv\sum_{N=0}^{\infty}p_{N}\tilde{\mathcal{E}}_{m,k}^{(q),N}.$
(30)

This way, we upper-bounded $\tilde{\mathcal{E}}^{(q)}_{m,k}$, calculated with
a full state (27) with a probabilistic combination of correlators
$\tilde{\mathcal{E}}^{(q),N}_{m,k}$, where each such
$\tilde{\mathcal{E}}^{(q),N}_{m,k}$ is calculated in a fixed-$N$ subspace. If
in all fixed-$N$ sectors the correlators lie within the bound from Eq. (20),
then the Bell nonlocality for the full state (27) is tested with

$\displaystyle\tilde{\mathcal{E}}^{(q)}_{m,k}\leqslant 2^{-(m+k)}f_{mk},$ (31)

where

$\displaystyle
f_{mk}=\sum_{N=\max(m,k)}^{\infty}p_{N}\left(\frac{N!}{(N-m)!}\frac{N!}{(N-k)!}\right)^{2}.$
(32)

If the inequality (31) is violated, then the system, as a whole, is non-local.
Note however, that this criterion does not tell in which fixed-$N$ sectors,
the nonlocality resides. To answer this question, one would need to analyze
$\tilde{\mathcal{E}}_{m,k}^{(q),N}$’s separately. We will further elaborate on
this point in Section 3.2.

Though the argument presented in this Section was not entirely universal — we
considered a particular non-fixed-$N$ state, see Eq. (24) — it is now clear
how to proceed in general. One should consider the fixed-$N$ sectors of states
in $A$ and $B$, for each determine the Bell bound and finally incoherently mix
various contributions.

Figure 1: The test of Bell correlations in a bosonic systems. When (a): the
qubits cannot be addressed individually, the Bell correlations are detected by
means of a collective operator $\hat{J}_{+}$, which is replaced by local
operators $\hat{\sigma}_{+}^{(k_{i})}$ when (b): qubits are spatially
separated.

####  2.2.4 Additional remarks

A loophole-free Bell test requires the particles to be causally separated, so
that the light cones associated with each of them do not intersect during the
phase of local operations and measurements. However, for a collection of
bosonic qubits occupying a single mode, the only available operations are the
collective ones [see Fig. 1(a)], such as represented by the rising operator
(15). This means that while the violation of Eq. (18) implies the presence of
Bell correlations, to close the causality loophole these particles would need
to be separated and addressed individually, as shown in Fig. 1(b). A similar
approach, using collective spin operators to detect the Bell correlations in a
bosonic system, has been experimentally tested in Ref. [53] (see also [54] for
the theoretical background). While in [53] the two-body Bell correlations were
witnessed in a spin-squeezed sample, the current work extends the analysis to
the many-body correlations.

Note also that in a many-body system, like the Bose-Einstein condensate, it
would be very hard to spatially split atoms apart without disturbing their
quantum degrees of freedom. Nevertheless, the existing techniques, like the
light-sheet method [55] or the detection of single metastable 4He atoms with
micro-channel plates [56, 41] open the way for future complete Bell tests in
systems of many-body bosonic qubits.

##  3 Applications

We now illustrate how the methods introduced in Section 2 apply to some
prominent physical cases.

###  3.1 BEC in a double-well potential

First, we take a BEC in a double-well potential: an important example of a
system of $N$ interacting ultra-cold bosonic qubits, governed by the two-mode
Bose-Hubbard Hamiltonian,

$\displaystyle\hat{H}=-\hat{J}_{x}+\frac{U}{N}\hat{J}_{z}^{2}.$ (33)

Here, the collective operators $\hat{J}_{x}$ and $\hat{J}_{z}$ are two members
of a triad

$\displaystyle\hat{J}_{x}=\frac{1}{2}\sum_{i=1}^{n}\hat{\sigma}_{x}^{(i)}=\frac{1}{2}(\hat{a}^{\dagger}\hat{b}+\hat{a}\hat{b}^{\dagger}),$
(34a)
$\displaystyle\hat{J}_{y}=\frac{1}{2}\sum_{i=1}^{n}\hat{\sigma}_{y}^{(i)}=\frac{1}{2i}(\hat{a}^{\dagger}\hat{b}-\hat{a}\hat{b}^{\dagger}),$
(34b)
$\displaystyle\hat{J}_{z}=\frac{1}{2}\sum_{i=1}^{n}\hat{\sigma}_{z}^{(i)}=\frac{1}{2}(\hat{a}^{\dagger}\hat{a}-\hat{b}^{\dagger}\hat{b})$
(34c)

of the angular momentum operators satisfying the commutation relation
$[\hat{J}_{n},\hat{J}_{m}]=i\epsilon_{nmk}\hat{J}_{k}$, where $\epsilon_{nmk}$
is a matrix element of the Levi-Civita tensor (for brevity, we used the
Einstein summation convention here). The dimensionless parameter $U$ in Eq.
(33) determines the strengths of the two-body interactions in units of the
Josephson energy [27].

Figure 2: The Bell correlators $\tilde{\mathcal{E}}^{(q)}_{m}$ calculated with
the ground state of the Hamiltonian from Eq. (33) with $N=100$ qubits. The
correlators are drawn as a function of the interaction strength $U$ in the
vicinity of the critical point $U=-1$ (marked with a vertical dashed blue
line), for $m=70,80,90$ and 100. The correlators are normalized to the right-
hand side of inequality (18), so that the dashed horizontal line at a constant
value $\tilde{\mathcal{E}}^{(q)}_{m}=1$ marks the local realistic bound for
all $m$.

We focus on the negative-$U$ case (attractive interactions) and find the
ground state of the Hamiltonian (33) with $N=100$ qubits. This is calculated
by first fixing a value of $U$ and then by performing the exact
diagonalization of the above Hamiltonian decomposed in the basis of the eigen-
states of $\hat{J}_{z}$, see Eq. (14). First, we scrutinize the vicinity of
$U=-1$, since at this point (in the limit of $N\rightarrow\infty$) the system
undergoes a quantum phase transition [57, 58, 59, 60, 61, 33]. The
significance of this point is underlined by the observation that upon passing
$U=-1$, the correlators $\tilde{\mathcal{E}}^{(q)}_{m}$ [here shown for
$m=70,80,90$ and 100 and normalized to the right-hand-side of Eq. (18)], cross
the limit and start to witness the Bell correlations, see Fig. 2. This is in
line with prior works [57, 62].

When $U$ drops below this critical point, the system enters a deeply quantum
regime [62, 63], where the ground state of the Hamiltonian (33) is a
macroscopic superposition of two largely occupied states and tends to the
$N00N$ state, i.e.,

$\displaystyle|\psi\rangle\xrightarrow{U\rightarrow-\infty}\frac{1}{\sqrt{2}}\Big{(}|N,0\rangle+|0,N\rangle\Big{)}.$
(35)

For this maximally entangled state, the quantum features are present only in
the highest-order correlation function, thus only the full $N$-body correlator
is non-vanishing and reaches a maximal value

$\displaystyle\tilde{\mathcal{E}}^{(q)}_{N}=\frac{1}{4}(N!)^{2}.$ (36)

Figure 3 shows the same correlators ($m=70,80,90,100$) but in a wider range of
interaction strengths, $U\in[-50,0]$. Clearly, at highly negative $U$, only
the maximal correlator remains above the local realistic bound.

Figure 3: The Bell correlators $\tilde{\mathcal{E}}^{(q)}_{m}$ calculated with
the ground state of the Hamiltonian (33) with $N=100$ qubits, as a function of
the interaction strength $U\in[-50,0]$, and for $m=70,80,90$ and 100. The
correlators are normalized as in Fig. 2.

###  3.2 Four-mode SPDC

Next, our focus is on the $A/B$ configuration, as discussed in Section 2.2.2.
We consider a broad family of scattering processes (SPDC, emissions of atomic
pairs from a BEC, etc.), governed by a Hamiltonian (in units of coupling
energy $g$)

$\displaystyle\hat{H}=\hat{a}^{\dagger}_{\uparrow}\hat{b}^{\dagger}_{\downarrow}+\hat{a}^{\dagger}_{\downarrow}\hat{b}_{\uparrow}+\mathrm{h.c.}$
(37)

The mode-functions associated with operators $\hat{a}_{\uparrow/\downarrow}$
are localized in region $A$, and analogically for
$\hat{b}_{\uparrow/\downarrow}$ and $B$. Since the two parts of the
Hamiltonian from Eq. (37),
$\hat{a}^{\dagger}_{\uparrow}\hat{b}^{\dagger}_{\downarrow}$ and
$\hat{a}^{\dagger}_{\downarrow}\hat{b}_{\uparrow}$, commute, the coefficient
from Eq. (24) factorizes, $C_{nm}=C_{n}C_{m}$ , with

$\displaystyle C_{n}=\frac{1}{\cosh(t)}\left(-i\tanh(t)\right)^{n}.$ (38)

The resulting state is

$\displaystyle|\psi\rangle=\sum_{n,m}\frac{\left(-i\tanh(t)\right)^{n+m}}{\cosh^{2}(t)}|n,m\rangle_{A}\otimes|m,n\rangle_{B}.$
(39)

In the Heisenberg picture, we have

$\displaystyle\hat{a}_{\uparrow}(t)=\cosh(t)\hat{a}_{\uparrow}(0)-i\sinh(t)\hat{b}_{\downarrow}^{\dagger}(0)$
(40)

and analogically for the remaining three operators.

Figure 4: The correlator $\tilde{\mathcal{E}}^{(q)}_{m,m}$ from Eq. (41) for
$m=2,3,5,6$ (top to bottom curve) as a function of time and normalized to the
local realistic bound (31). The Figure shows that the values of all the
correlators lie well below the Bell limit.

Due to the $A/B$ symmetry of the Hamiltonian (37), we consider the correlator
$\tilde{\mathcal{E}}^{(q)}_{m,k}$ with $k=m$, and using Eq. (40) we obtain

$\displaystyle\tilde{\mathcal{E}}^{(q)}_{m,m}=|\left\langle\hat{J}_{+}^{(A)m}\hat{J}_{-}^{(B)m}\right\rangle|^{2}=(\sinh(t)\cosh(t))^{4m}(m!)^{4}.$
(41)

Next, we establish the local realistic bound, using the approach detailed in
Section 2.2.2. To this end, we express the state (39) as follows

$\displaystyle|\psi\rangle=\sum_{N=0}^{\infty}(-i)^{N}\sqrt{p_{N}}|\psi_{N}\rangle,$
(42)

where the fixed-$N$ state is

$\displaystyle|\psi_{N}\rangle=\sum_{n=0}^{N}\frac{|n,N-n\rangle_{A}\otimes|N-n,n\rangle_{B}}{\sqrt{N+1}},$
(43)

and $p_{N}=\frac{\tanh^{2N}(t)}{\cosh^{4}(t)}(N+1)$ is the probability for
having $N$ particles in $A/B$, while $N=n+m$. Since local operators
$\hat{J}_{+}^{(A)}/\hat{J}_{-}^{(B)}$ conserve the number of particles, this
state can be replaced by an incoherent mixture

$\displaystyle\hat{\varrho}=\sum_{N=0}^{\infty}p_{N}|\psi_{N}\rangle\\!\langle\psi_{N}|.$
(44)

Once the $p_{N}$ is determined, the local realistic bound is given by Eq. (31)
with $m=k$, i.e.,

$\displaystyle f_{mm}$
$\displaystyle=(m!)^{4}\sum_{N=m}^{\infty}p_{N}{N\choose m}^{4}$ (45)
$\displaystyle=\frac{(m!)^{4}}{\cosh^{2}(t)}\Bigg{(}{0\choose
m}^{4}{}_{5}F_{4}\Big{[}\vec{a}_{1},\vec{m}_{1},\tanh^{2}(t)\Big{]}+{1\choose
m}^{4}{}_{5}F_{4}\Big{[}\vec{a}_{2},\vec{m}_{2},\tanh^{2}(t)\Big{]}\tanh^{2}(t)\Bigg{)},$

where ${}_{5}F_{4}$ stands for a generalized hypergeometric function and

$\displaystyle\vec{a}_{i}=(i,i,i,i,i)^{\rm T},$ (46a)
$\displaystyle\vec{m}_{i}=(i-m,i-m,i-m,i-m)^{\rm T}.$ (46b)

In Fig. 4 we plot the $\tilde{\mathcal{E}}^{(q)}_{m,m}$ normalized to the
local realistic bound, i.e., divided by $f_{mm}2^{-2m}$ for $m=2,3,5$ and 6,
as a function of time. Clearly, all these correlators lie well below the
bound, thus neither of them detects any Bell correlations.

This however does not mean that the state from Eq. (39) contains no highly
non-classical correlations, though we claim that they reside in the fixed-$N$
subspaces and the corresponding states $|\psi_{N}\rangle$ defined in Eq. (43).
To show this, we calculate

$\displaystyle\tilde{\mathcal{E}}^{(q),N}_{m,m}=|\langle\psi_{N}|\hat{J}_{+}^{(A)m}\hat{J}_{-}^{(B)m}|\psi_{N}\rangle|^{2}$
(47)

and plot this correlator normalized to the Bell bound, i.e., the right-hand
side of the Bell inequality

$\displaystyle\tilde{\mathcal{E}}^{(q),N}_{m,m}\leqslant\left(\frac{N!}{(N-m)!}\right)^{4}2^{-2m}$
(48)

for $N=2,3,6$ and 12 and for each $N$ changing $m$ from 1 to $N$, see Fig. 5.
Clearly, in fixed-$N$ subspaces, the many-body Bell correlations is present in
the state (43) and it is genuinely a result of the cross-region nonlocality.
In a single region alone, say $A$, the qubits are uncorrelated because the
reduced state is a full mixture

$\displaystyle\hat{\varrho}_{N}^{(A)}={\rm
Tr}\left[|\psi_{N}\rangle\\!\langle\psi_{N}|\right]_{B}=\sum_{n=0}^{N}\frac{|n,N-n\rangle\\!\langle
n,N-n|_{A}}{N+1}=\frac{1}{N+1}\hat{\mathds{1}}_{A},$ (49)

which gives ${\rm Tr}\left[\hat{J}_{+}^{(A)m}\hat{\varrho}_{N}^{(A)}\right]=0$
for all $m>0$. Symmetrically, the same argument holds for the region $B$.

Figure 5: The correlator from Eq. (47) calculated with the fixed-$N$ part [see
Eq. (43)] of the full state [Eq. (39)] for $N=2,3,6$ and 12, as a function of
$m\in[1,\ldots N]$ and normalized to the local realistic bound from Eq. (48).
Contrary to the correlator $\tilde{\mathcal{E}}^{(q)}_{m,m}$ calculated with a
full state, which does not break the Bell inequality with $m=2,3,5$ and 6,
here the bound is surpassed (for instance $N=6$ and $m=5,6$).

By inspecting the Fig. 5 we notice that already at the lowest $N=2$ (i.e., two
particles per $A/B$), the correlator $\tilde{\mathcal{E}}^{(q),2}_{2,2}$
breaks the local realistic bound. However, for larger $N$, the
$\tilde{\mathcal{E}}^{(q),N}_{2,2}$ remains below the bound.

This explains why the Bell nonlocality is not detected in the full state from
Eq. (39). For various $N$’s, the nonlocality is witnessed by the correlators
of different orders $m$. Hence the mixing of contributions from all $N$’s but
with fixed $m$ adds together a few contributions
$\tilde{\mathcal{E}}^{(q),N}_{m,m}$ that detect the nonlocality with a
majority that lie below the Bell bound. In consequence, the averaged
correlator $\tilde{\mathcal{E}}^{(q)}_{m,m}$ does not break the nonlocality
limit. The conclusion from this Section is that the nonlocality can be
detected in such an $A/B$ configuration following either of two paths. First
is the standard approach, where one restricts to a very low-gain regime, where
the probability of having more than a single qubit per region is negligible.
In such case, the CHSH inequality will detect the Bell correlations. The
second is the many-body regime, and in this case one should apply the post-
selection, restrict to the fixed-$N$ subspace and use the correlators
$\tilde{\mathcal{E}}^{(q),N}_{m,m}$. Otherwise, the mixing of different $N$’s,
as in Eq. (44), washes out any multi-particle non-local effects. Note that the
influence of fluctuations of the number of particles on the strength of the
Bell correlations has been widely discussed in [53].

##  4 Discussion

Before we summarize, let us take a closer look at the correlator
$\tilde{\mathcal{E}}^{(q)}_{m}$ from Eq. (18) for $m=3$. Using
$\hat{J}_{+}=\hat{J}_{x}+i\hat{J}_{y}$ we obtain

$\displaystyle\tilde{\mathcal{E}}^{(q)}_{3}=|\left\langle\hat{J}_{+}^{3}\right\rangle|^{2}$
$\displaystyle=\left(\left\langle\hat{J}_{x}^{3}-\hat{J}_{y}\hat{J}_{x}\hat{J}_{y}-\hat{J}_{y}^{2}\hat{J}_{x}-\hat{J}_{x}\hat{J}_{y}^{2}\right\rangle\right)^{2}+$
$\displaystyle+\left(\left\langle\hat{J}_{y}^{3}-\hat{J}_{x}\hat{J}_{y}\hat{J}_{x}-\hat{J}_{x}^{2}\hat{J}_{y}-\hat{J}_{y}\hat{J}_{x}^{2}\right\rangle\right)^{2}.$
(50)

In the experiment, to know $\tilde{\mathcal{E}}^{(q)}_{3}$ is to measure
$2^{3}=8$ different 3-rd order correlation functions (like
$\left\langle\hat{J}_{x}^{2}\hat{J}_{y}\right\rangle$), which generalizes to
$2^{m}$ $m$-th order functions for $\tilde{\mathcal{E}}^{(q)}_{m}$. This is
very challenging already for $m=3$ and the possibility to empirically evaluate
the Bell correlator for high $m$’s is a rather distant perspective.

Nevertheless, the experimental progress in single-particle detection is
enormous. For photons, such devices are of almost every-day use in
laboratories. For atoms, various techniques have been developed, including the
aforementioned light-sheet method [55] or the luminescence from atoms trapped
in an optical lattice [64, 65]. Last but not least, spatial correlations of up
to 6-th order between meta-stable 4He atoms have been measured with ultra-high
accuracy [56].

This work is intended mostly as a theoretical study of Bell correlations in
many-body bosonic configurations. Such inquiry brings forward the fundamental
aspects of complex systems. Nevertheless, there is also an application-
oriented flavor of this inquiry, because the many-body Bell correlations, just
as the entanglement [66] and the EPR steering [67], is a resource for the
quantum-enhanced metrology [33, 68].

Finally, we would like to bring up the relation between the
$\tilde{\mathcal{E}}^{(q)}_{m}$ and those other (than the Bell-type) quantum
correlations: the entanglement and the EPR steering. For spin-chains, when
qubits can be addressed individually, it is justified to use the correlator
$\mathcal{E}^{(q)}_{m}$ from Eq. (11). It detects entanglement when
$\mathcal{E}^{(q)}_{m}>4^{-m}$ and Bell correlations if
$\mathcal{E}^{(q)}_{m}>2^{-m}$ [31, 32]. Between these two threshold, lies a
border, above which $\mathcal{E}^{(q)}_{m}$ detects the EPR steering, and its
position depends on the physical setup—namely what is the ratio between the
active (steering) and the passive (being steered) qubits [29]. The limits
attributed to ${\mathcal{E}}^{(q)}_{m}$ can be translated into limits for
$\tilde{\mathcal{E}}^{(q)}_{m}$ like in Eq. (18), allowing for an even deeper
and more systematic study of quantum correlations in many-body bosonic
systems.

Note also that the GHZ (12), (21) and the N00N states (35) are considered in
this work merely for illustration as the limiting cases to establish the
bounds for the witnesses of Bell correlations in quantum systems. Naturally,
these maximally-entangled states are mostly vulnerable to particle losses and
for higher $N$ should not be regarded as realistic testing tools for the
theory.

##  5 Conclusions

We studied the many-body Bell correlations in bosonic systems. To this end, we
adopted a Bell correlator from the case when qubits can be addressed
individually to the situation when only collective bosonic operations are
allowed. Next, we showed how to construct a witness of Bell correlations for
even more complex scenarios, when qubits, apart from their internal two-level
structure, can occupy a number of external modes. Afterwards, we generalized
these results to systems, where the number of particles is not fixed.

With all these tools at hand, we considered a number of examples, such as a
Bose-Einstein condensate in a double-well potential with attractive
interactions or a collection of photons obtained in a spontaneous parametric-
down conversion process. While in the former case, the many-body Bell
correlations naturally emerged in the ground state, for the latter a more
detailed analysis was necessary. This was due to the fact that the full state
contained a non-fixed number of photons. We showed that strong Bell
correlations reside in fixed-$N$ sectors, rather then in the photonic state
taken as a whole.

The presented method allows for a systematic analysis of all the members of
the triad of quantum correlations: entanglement, EPR steering and Bell
correlations. We hope that such study expands the knowledge on highly non-
classical effects in many-body systems. This has implications both for the
basic research but also for the application-oriented quantum technologies.

##  6 Acknowledgements

The support of the National Science Centre, Poland under the QuantERA
programme, Project no. 2017/25/Z/ST2/03039 is acknowledged.

## References

  * [1] A. Einstein, B. Podolsky and N. Rosen,  _Can Quantum-Mechanical Description of Physical Reality Be Considered Complete?_ ,  Phys. Rev. 47, 777 (1935),  10.1103/PhysRev.47.777.
  * [2] E. Schrödinger,  _Discussion of probability relations between separated systems_ ,  In _Mathematical Proceedings of the Cambridge Philosophical Society_ , vol. 31, pp. 555–563. Cambridge University Press,  10.1017/S0305004100013554 (1935).
  * [3] J. S. Bell,  _On the Einstein Podolsky Rosen paradox_ ,  Physics 1, 195 (1964),  10.1103/PhysicsPhysiqueFizika.1.195.
  * [4] J. F. Clauser, M. A. Horne, A. Shimony and R. A. Holt,  _Proposed Experiment to Test Local Hidden-Variable Theories_ ,  Phys. Rev. Lett. 23, 880 (1969),  10.1103/PhysRevLett.23.880.
  * [5] S. J. Freedman and J. F. Clauser,  _Experimental Test of Local Hidden-Variable Theories_ ,  Phys. Rev. Lett. 28, 938 (1972),  10.1103/PhysRevLett.28.938.
  * [6] A. Aspect, P. Grangier and G. Roger,  _Experimental Tests of Realistic Local Theories via Bell’s Theorem_ ,  Phys. Rev. Lett. 47, 460 (1981),  10.1103/PhysRevLett.47.460.
  * [7] A. Aspect, J. Dalibard and G. Roger,  _Experimental Test of Bell’s Inequalities Using Time- Varying Analyzers_ ,  Phys. Rev. Lett. 49, 1804 (1982),  10.1103/PhysRevLett.49.1804.
  * [8] W. Tittel, J. Brendel, B. Gisin, T. Herzog, H. Zbinden and N. Gisin,  _Experimental demonstration of quantum correlations over more than 10 km_ ,  Phys. Rev. A 57, 3229 (1998),  10.1103/PhysRevA.57.3229.
  * [9] W. Tittel, J. Brendel, H. Zbinden and N. Gisin,  _Violation of Bell Inequalities by Photons More Than 10 km Apart_ ,  Phys. Rev. Lett. 81, 3563 (1998),  10.1103/PhysRevLett.81.3563.
  * [10] G. Weihs, T. Jennewein, C. Simon, H. Weinfurter and A. Zeilinger,  _Violation of Bell’s Inequality under Strict Einstein Locality Conditions_ ,  Phys. Rev. Lett. 81, 5039 (1998),  10.1103/PhysRevLett.81.5039.
  * [11] J.-W. Pan, D. Bouwmeester, M. Daniell, H. Weinfurter and A. Zeilinger,  _Experimental test of quantum nonlocality in three-photon Greenberger-Horne-Zeilinger entanglement_ ,  Nature 403, 515 (2000),  10.1038/35000514.
  * [12] D. Kielpinski, V. Meyer, C. A. Sackett, W. M. Itano, C. Monroe and D. J. Wineland,  _Experimental violation of a Bell’s inequality with efficient detection_ ,  Nature 409, 791 (2001),  10.1038/35057215.
  * [13] S. Gröblacher, T. Paterek, R. Kaltenbaek, Č. Brukner, M. Żukowski, M. Aspelmeyer and A. Zeilinger,  _An experimental test of non-local realism_ ,  Nature 446, 871 (2007),  10.1038/nature05677.
  * [14] D. Salart, A. Baas, J. A. W. van Houwelingen, N. Gisin and H. Zbinden,  _Spacelike Separation in a Bell Test Assuming Gravitationally Induced Collapses_ ,  Phys. Rev. Lett. 100, 220404 (2008),  10.1103/PhysRevLett.100.220404.
  * [15] M. Ansmann, H. Wang, R. C. Bialczak, M. Hofheinz, E. Lucero, M. Neeley, A. D. O’Connell, D. Sank, M. Weides, J. Wenner, A. N. Cleland and J. M. Martinis,  _Violation of Bell’s inequality in Josephson phase qubits_ ,  Nature 461, 504 (2009),  10.1038/nature08363.
  * [16] M. Giustina, A. Mech, S. Ramelow, B. Wittmann, J. Kofler, J. Beyer, A. Lita, B. Calkins, T. Gerrits, S. W. Nam, R. Ursin and A. Zeilinger,  _Bell violation using entangled photons without the fair-sampling assumption_ ,  Nature 497, 227 (2013),  10.1038/nature12012.
  * [17] N. Brunner, D. Cavalcanti, S. Pironio, V. Scarani and S. Wehner,  _Bell nonlocality_ ,  Rev. Mod. Phys. 86, 419 (2014),  10.1103/RevModPhys.86.419.
  * [18] B. Hensen, H. Bernien, A. Dréau, A. Reiserer, N. Kalb, M. Blok, J. Ruitenberg, R. Vermeulen, R. Schouten, C. Abellán _et al._ ,  _Loophole-free Bell inequality violation using electron spins separated by 1.3 kilometres_ ,  Nature 526, 682 (2015),  10.1038/nature15759.
  * [19] M. Giustina, M. A. M. Versteegh, S. Wengerowsky, J. Handsteiner, A. Hochrainer, K. Phelan, F. Steinlechner, J. Kofler, J.-A. Larsson, C. Abellán, W. Amaya, V. Pruneri _et al._ ,  _Significant-loophole-free test of bell’s theorem with entangled photons_ ,  Phys. Rev. Lett. 115, 250401 (2015).
  * [20] M. Giustina, M. A. Versteegh, S. Wengerowsky, J. Handsteiner, A. Hochrainer, K. Phelan, F. Steinlechner, J. Kofler, J.-Å. Larsson, C. Abellán _et al._ ,  _Significant-loophole-free test of bell’s theorem with entangled photons_ ,  Phys. Rev. Lett. 115(25), 250401 (2015).
  * [21] L. K. Shalm, E. Meyer-Scott, B. G. Christensen, P. Bierhorst, M. A. Wayne, M. J. Stevens, T. Gerrits, S. Glancy, D. R. Hamel, M. S. Allman _et al._ ,  _Strong loophole-free test of local realism_ ,  Phys. Rev. Lett. 115(25), 250402 (2015).
  * [22] W. Rosenfeld, D. Burchardt, R. Garthoff, K. Redeker, N. Ortegel, M. Rau and H. Weinfurter,  _Event-ready bell test using entangled atoms simultaneously closing detection and locality loopholes_ ,  Phys. Rev. Lett. 119(1), 010402 (2017).
  * [23] W. Rosenfeld, D. Burchardt, R. Garthoff, K. Redeker, N. Ortegel, M. Rau and H. Weinfurter,  _Event-ready bell test using entangled atoms simultaneously closing detection and locality loopholes_ ,  Phys. Rev. Lett. 119, 010402 (2017),  10.1103/PhysRevLett.119.010402.
  * [24] L. Mandel and E. Wolf,  _Optical coherence and quantum optics_ ,  Cambridge University Press,  10.1017/CBO9781139644105 (1995).
  * [25] D. C. Burnham and D. L. Weinberg,  _Observation of Simultaneity in Parametric Production of Optical Photon Pairs_ ,  Phys. Rev. Lett. 25, 84 (1970),  10.1103/PhysRevLett.25.84.
  * [26] P. G. Kwiat, K. Mattle, H. Weinfurter, A. Zeilinger, A. V. Sergienko and Y. Shih,  _New High-Intensity Source of Polarization-Entangled Photon Pairs_ ,  Phys. Rev. Lett. 75, 4337 (1995),  10.1103/PhysRevLett.75.4337.
  * [27] A. A. Golubov, M. Y. Kupriyanov and E. IlIchev,  _The current-phase relation in Josephson junctions_ ,  Rev. Mod. Phys. 76, 411 (2004),  10.1103/RevModPhys.76.411.
  * [28] M. W. Doherty, N. B. Manson, P. Delaney, F. Jelezko, J. Wrachtrup and L. C. Hollenberg,  _The nitrogen-vacancy colour centre in diamond_ ,  Phys. Rep. 528, 1 (2013),  10.1016/j.physrep.2013.02.001.
  * [29] E. G. Cavalcanti, C. J. Foster, M. D. Reid and P. D. Drummond,  _Bell inequalities for continuous-variable correlations_ ,  Phys. Rev. Lett. 99, 210405 (2007),  10.1103/PhysRevLett.99.210405.
  * [30] Q. He, E. Cavalcanti, M. Reid and P. Drummond,  _Bell inequalities for continuous-variable measurements_ ,  Phys. Rev. A 81, 062106 (2010),  10.1103/PhysRevA.81.062106.
  * [31] E. Cavalcanti, Q. He, M. Reid and H. Wiseman,  _Unified criteria for multipartite quantum nonlocality_ ,  Phys. Rev. A 84, 032115 (2011),  10.1103/PhysRevA.84.032115.
  * [32] A. Niezgoda, M. Panfil and J. Chwedeńczuk,  _Quantum correlations in spin chains_ ,  Phys. Rev. A 102, 042206 (2020),  10.1103/PhysRevA.102.042206.
  * [33] A. Niezgoda and J. Chwedeńczuk,  _Many-body nonlocality as a resource for quantum-enhanced metrology_ ,  Phys. Rev. Lett. 126, 210506 (2021),  10.1103/PhysRevLett.126.210506.
  * [34] L. Masi, T. Petrucciani, G. Ferioli, G. Semeghini, G. Modugno, M. Inguscio and M. Fattori,  _Spatial Bloch Oscillations of a Quantum Gas in a “Beat-Note” Superlattice_ ,  Phys. Rev. Lett. 127, 020601 (2021),  10.1103/PhysRevLett.127.020601.
  * [35] J. M. Vogels, K. Xu and W. Ketterle,  _Generation of Macroscopic Pair-Correlated Atomic Beams by Four-Wave Mixing in Bose-Einstein Condensates_ ,  Phys. Rev. Lett. 89, 020401 (2002),  10.1103/PhysRevLett.89.020401.
  * [36] A. Perrin, H. Chang, V. Krachmalnicoff, M. Schellekens, D. Boiron, A. Aspect and C. I. Westbrook,  _Observation of atom pairs in spontaneous four-wave mixing of two colliding bose-einstein condensates_ ,  Phys. Rev. Lett. 99, 150405 (2007),  10.1103/PhysRevLett.99.150405.
  * [37] M. Bonneau, J. Ruaudel, R. Lopes, J.-C. Jaskula, A. Aspect, D. Boiron and C. I. Westbrook,  _Tunable source of correlated atom beams_ ,  Phys. Rev. A 87, 061603 (2013),  10.1103/PhysRevA.87.061603.
  * [38] T. Wasak, P. Szańkowski, R. Bücker, J. Chwedeńczuk and M. Trippenbach,  _Bogoliubov theory for atom scattering into separate regions_ ,  New J. Phys. 16, 013041 (2014),  10.1088/1367-2630/16/1/013041.
  * [39] B. Lücke, M. Scherer, J. Kruse, L. Pezzé, F. Deuretzbacher, P. Hyllus, J. Peise, W. Ertmer, J. Arlt, L. Santos _et al._ ,  _Twin matter waves for interferometry beyond the classical limit_ ,  Science 334, 773 (2011),  10.1126/science.1208798.
  * [40] R. Bücker, J. Grond, S. Manz, T. Berrada, T. Betz, C. Koller, U. Hohenester, T. Schumm, A. Perrin and J. Schmiedmayer,  _Twin-atom beams_ ,  Nat. Phys. 7, 608 (2011),  10.1038/nphys1992.
  * [41] D. K. Shin, B. M. Henson, S. S. Hodgman, T. Wasak, J. Chwedeńczuk and A. G. Truscott,  _Bell correlations between spatially separated pairs of atom_ ,  Nat. Comm. 10, 4447 (2019),  10.1038/s41467-019-12192-8.
  * [42] P. Zin, T. Wasak, D. Boiron and C. I. Westbrook,  _Pair correlation of atoms scattered from colliding Bose-Einstein quasicondensates_ ,  Phys. Rev. A 101, 033616 (2020),  10.1103/PhysRevA.101.033616.
  * [43] Q. He, P. Drummond and M. Reid,  _Entanglement, EPR steering, and Bell-nonlocality criteria for multipartite higher-spin systems_ ,  Phys. Rev. A 83, 032120 (2011),  10.1103/PhysRevA.83.032120.
  * [44] M. Żukowski and i. c. v. Brukner,  _Bell’s Theorem for General N-Qubit States_ ,  Phys. Rev. Lett. 88, 210401 (2002),  10.1103/PhysRevLett.88.210401.
  * [45] Y.-C. Liang, D. Rosset, J.-D. Bancal, G. Pütz, T. J. Barnea and N. Gisin,  _Family of Bell-like Inequalities as Device-Independent Witnesses for Entanglement Depth_ ,  Phys. Rev. Lett. 114, 190401 (2015),  10.1103/PhysRevLett.114.190401.
  * [46] A. Aloy, J. Tura, F. Baccari, A. Acín, M. Lewenstein and R. Augusiak,  _Device-Independent Witnesses of Entanglement Depth from Two-Body Correlators_ ,  Phys. Rev. Lett. 123, 100507 (2019),  10.1103/PhysRevLett.123.100507.
  * [47] P.-S. Lin, J.-C. Hung, C.-H. Chen and Y.-C. Liang,  _Exploring Bell inequalities for the device-independent certification of multipartite entanglement depth_ ,  Phys. Rev. A 99, 062338 (2019),  10.1103/PhysRevA.99.062338.
  * [48] G. C. Wick, A. S. Wightman and E. P. Wigner,  _The intrinsic parity of elementary particles_ ,  Phys. Rev. 88, 101 (1952),  10.1103/PhysRev.88.101.
  * [49] H. M. Wiseman and J. A. Vaccaro,  _Entanglement of Indistinguishable Particles Shared between Two Parties_ ,  Phys. Rev. Lett. 91, 097902 (2003),  10.1103/PhysRevLett.91.097902.
  * [50] F. Verstraete and J. I. Cirac,  _Quantum nonlocality in the presence of superselection rules and data hiding protocols_ ,  Phys. Rev. Lett. 91, 010404 (2003),  10.1103/PhysRevLett.91.010404.
  * [51] S. D. Bartlett and H. M. Wiseman,  _Entanglement Constrained by Superselection Rules_ ,  Phys. Rev. Lett. 91, 097903 (2003),  10.1103/PhysRevLett.91.097903.
  * [52] T. Wasak, A. Smerzi and J. Chwedeńczuk,  _Role of particle entanglement in the violation of Bell inequalities_ ,  Sci. Rep. 8, 1777 (2018),  10.1038/s41598-018-20034-8.
  * [53] R. Schmied, J.-D. Bancal, B. Allard, M. Fadel, V. Scarani, P. Treutlein and N. Sangouard,  _Bell correlations in a Bose-Einstein condensate_ ,  Science 352, 441 (2016),  10.1126/science.aad8665.
  * [54] J. Tura, R. Augusiak, A. B. Sainz, T. Vértesi, M. Lewenstein and A. Acín,  _Detecting nonlocality in many-body quantum states_ ,  Science 344, 1256 (2014),  10.1126/science.1247715.
  * [55] R. Bücker, A. Perrin, S. Manz, T. Betz, C. Koller, T. Plisson, J. Rottmann, T. Schumm and J. Schmiedmayer,  _Single-particle-sensitive imaging of freely propagating ultracold atoms_ ,  New J. Phys. 11, 103039 (2009),  10.1088/1367-2630/11/10/103039.
  * [56] R. Dall, A. Manning, S. Hodgman, W. RuGway, K. V. Kheruntsyan and A. Truscott,  _Ideal n-body correlations with massive particles_ ,  Nat. Phys. 9, 341 (2013),  10.1038/nphys2632.
  * [57] C. Invernizzi, M. Korbman, L. Campos Venuti and M. G. A. Paris,  _Optimal quantum estimation in spin systems at criticality_ ,  Phys. Rev. A 78, 042106 (2008),  10.1103/PhysRevA.78.042106.
  * [58] P. Zanardi, M. G. A. Paris and L. Campos Venuti,  _Quantum criticality as a resource for quantum estimation_ ,  Phys. Rev. A 78, 042105 (2008),  10.1103/PhysRevA.78.042105.
  * [59] G. Salvatori, A. Mandarino and M. G. A. Paris,  _Quantum metrology in Lipkin-Meshkov-Glick critical systems_ ,  Phys. Rev. A 90, 022111 (2014),  10.1103/PhysRevA.90.022111.
  * [60] M. Bina, I. Amelio and M. G. A. Paris,  _Dicke coupling by feasible local measurements at the superradiant quantum phase transition_ ,  Phys. Rev. E 93, 052118 (2016),  10.1103/PhysRevE.93.052118.
  * [61] L. Garbe, M. Bina, A. Keller, M. G. A. Paris and S. Felicetti,  _Critical Quantum Metrology with a Finite-Component Quantum Phase Transition_ ,  Phys. Rev. Lett. 124, 120504 (2020),  10.1103/PhysRevLett.124.120504.
  * [62] A. Piga, A. Aloy, M. Lewenstein and I. Frérot,  _Bell Correlations at Ising Quantum Critical Points_ ,  Phys. Rev. Lett. 123, 170604 (2019),  10.1103/PhysRevLett.123.170604.
  * [63] P. Hauke, M. Heyl, L. Tagliacozzo and P. Zoller,  _Measuring multipartite entanglement through dynamic susceptibilities_ ,  Nat. Phys. 12, 778 (2016),  10.1038/nphys3700.
  * [64] J. F. Sherson, C. Weitenberg, M. Endres, M. Cheneau, I. Bloch and S. Kuhr,  _Single-atom-resolved fluorescence imaging of an atomic Mott insulator_ ,  Nature 467, 68 (2010),  10.1038/nature09378.
  * [65] C. Weitenberg, M. Endres, J. F. Sherson, M. Cheneau, P. Schauß, T. Fukuhara, I. Bloch and S. Kuhr,  _Single-spin addressing in an atomic Mott insulator_ ,  Nature 471, 319 (2011),  10.1038/nature09827.
  * [66] L. Pezzé and A. Smerzi,  _Entanglement, nonlinear dynamics, and the Heisenberg limit_ ,  Phys. Rev. Lett. 102, 100401 (2009),  10.1103/PhysRevLett.102.100401.
  * [67] B. Yadin, M. Fadel and M. Gessner,  _Metrological complementarity reveals the Einstein-Podolsky-Rosen paradox_ ,  Nat. Comm. 12, 2410 (2021),  10.1038/s41467-021-22353-3.
  * [68] G. Tóth and T. Vértesi,  _Quantum States with a Positive Partial Transpose are Useful for Metrology_ ,  Phys. Rev. Lett. 120, 020506 (2018),  10.1103/PhysRevLett.120.020506.


#  HTRON: Efficient Outdoor Navigation with Sparse Rewards via Heavy Tailed
Adaptive Reinforce Algorithm

Kasun Weerakoon, Souradip Chakraborty∗, Nare Karapetyan, Adarsh Jagan
Sathyamoorthy,
Amrit Singh Bedi and Dinesh Manocha These authors contributed equally to this
work. Kasun Weerakoon, Souradip Chakraborty, Nare Karapetyan, Adarsh Jagan
Sathyamoorthy, Amrit Singh Bedi and Dinesh Manocha are with the University of
Maryland, College Park, MD, USA. Email: {kasunw, schkra, knare, asathyam,
amritbd, dmanocha} @umd.edu.

###### Abstract

We present a novel approach to improve the performance of deep reinforcement
learning (DRL) based outdoor robot navigation systems. Most, existing DRL
methods are based on carefully designed dense reward functions that learn the
efficient behavior in an environment. We circumvent this issue by working only
with sparse rewards (which are easy to design), and propose a novel adaptive
Heavy-Tailed Reinforce algorithm for Outdoor Navigation called HTRON. The key
idea in this work is to utilize heavy-tailed policy parametrizations which
implicitly induces exploration in sparse reward settings. We evaluate the
performance of HTRON against Reinforce, PPO and TRPO algorithms in three
different outdoor scenarios: goal-reaching, obstacle avoidance, and uneven
terrain navigation. We observe in average an increase of $34.41\%$ in terms of
success rate, $15.15\%$ decrease in the average time steps taken to reach the
goal, and $24.9\%$ decrease in the elevation cost compared to the navigation
policies obtained by the other methods. Further, we demonstrate that our
algorithm can be transferred directly into a Clearpath Husky robot to perform
real-world outdoor terrain navigation.

> Keywords: Navigation, Deep Reinforcement Learning, Heavy Tailed Policy

##  1 Introduction

Autonomous robot navigation in complex outdoor environments has been an active
area of research. The resulting navigation systems are used for different
applications, including delivery [1], search and rescue [2], planetary
explorations [3], etc. Such applications require robots to gather useful
information efficiently from the environment to make intelligent navigation
decisions [4]. To this end, Deep Reinforcement Learning (DRL) techniques have
been widely employed in recent robotic systems due to their inherent
exploration and exploitation capabilities to gather necessary information from
the learning environments [5, 6, 7, 8, 9, 10]. Nevertheless, a major challenge
for DRL based algorithms is dealing with sparse rewards in continuous state
and actions spaces. In particular, the rewards are sparse in navigation
settings because it’s only available on reaching a goal (positive reward) or
hitting an obstacle (negative rewards) [11]. Hence, training DRL policies
under sparse reward settings oftentimes lead to instabilities and convergence
issues that can significantly degrade consistent performance, especially in
navigation applications [12, 13].

To improve the training efficiency and deal with sparse rewards, two popular
methods in the literature are reward shaping [14] and demonstration-guided
learning [15, 16]. The primary objective of rewards shaping-based methods is
to induce an exploration strategy using curiosity-driven methods which provide
an additional pseudo reward for exploration in the environment [17]. However,
such methods require carefully designed intrinsic rewards which can introduce
expert-specific bias to the learning systems hindering the overall
performance. Similarly, demonstration-guided methods have a high dependence on
expert supervision which could be difficult to obtain in practice for many
outdoor navigation tasks [18]. Instead of modifying the rewards, a more
pragmatic approach is to try to work directly with the sparse rewards [19] and
make the DRL based policies to train for the outdoor navigation task at hand.
But the major challenge is how to induce inherent exploration into the
training without reshaping the rewards. Recent work in literature[20, 21]
suggest that one way to handle these is to use heavy-tailed policy (such as
Cauchy) parametrization techniques. Motivated by these factors, in this work,
we try to find optimal behaviors for outdoor navigation tasks while directly
operating under sparse reward settings.

Main Contributions:

  1. 1.

We present a heavy-tailed policy formulation with adaptive gradient tracking
to address three navigation challenges encountered in complex outdoor
environments (goal-reaching, obstacle avoidance, and navigating on uneven
terrains). Our proposed algorithm HTRON outperforms three state-of-the-art
algorithms: Reinforce [22], PPO [23], and TRPO [24] in terms of cumulative
reward return with improved sample efficiency for faster convergence.

  2. 2.

We propose a set of novel sparse reward functions that do not require careful
reward designs yet are capable of successfully navigating a differential drive
robot using an improved stochastic policy gradient method.

  3. 3.

We evaluate the navigation performance of HTRON in both simulated and real-
world complex outdoor environments using a Clearpath Husky robot. HTRON
results in an increase of up to $34.41\%$ in terms of navigation success rate,
$15.15\%$ decrease in the average time steps taken to reach the goal, and
$24.9\%$ decrease in the elevation cost compared to the navigation policies
obtained by the other methods.

##  2 Related Work

Learning-based approaches, in particular DRL have pushed the boundaries of
mobile robot navigation for unstructured and dynamically changing environments
[25, 26, 8]. Nevertheless, these methods suffer from sample efficiency under
sparse rewards settings because dense rewards play an important role in
learning for DRL methods. We summarize the related work next.

Outdoor Robot Navigation: Navigation, path planning and obstacle avoidance are
extensively-studied fundamental research problems in the field of robotics
[27, 28]. Number of deterministic [29, 30] and stochastic [31, 32] algorithms
have been proposed in the past three decades. However, robot navigation in
complex outdoor environments still remains a challenging problem given the
complexity and diversity of the setting. To perform robot-centric decision-
making, deep reinforcement learning (DRL) strategies have been employed in
recent studies for outdoor navigation [33, 34]. For example, deterministic
policies such as DDPG[26], A3C[35], and DQN[36] with dense rewards have been
incorporated for robot navigation on uneven terrains. An adaptive model that
directly learns control and environmental dynamics is presented in [8].
Moreover, segmentation[37] and self supervised [25] methods for identifying
navigable regions have also been utilized with a combination of DRL based
navigation methods to ensure stable and efficient navigation.

Sparse Reward Settings: In sparse reward settings, an environment rarely
produces a useful reward which significantly affects the reinforcement policy
leaning [11]. To deal with this issue, several reward shaping methods based on
intrinsic curiosity and information gain-based shaping have been proposed
[38]. For example, reward shaping based methods such as [39, 17] encourage the
agent to explore unvisited states by modifying the actual reward output.
However, such approaches require additional effort and expertise in reward
function design. Imitation learning and learning from demonstration methods
are another approach for dealing with sparse rewards in practice [40].
However, obtaining reliable expert demonstrations for continuous control tasks
is practically infeasible and challenging [41, 15].

Heavy-tailed Policy Parametrization: The possibility of incorporating heavy-
tailed distributions to RL tasks is theoretically analyzed in recent
literature [42, 21]. For example, in [43], beta distribution is utilized with
dense rewards for stochastic policy training to enhance the state exploration
capabilities. Further, a heavy-tailed policy gradient method is proposed in
[42] to find convergence to global maxima while minimizing the risk of local
maxima convergence. Inspired by these ideas, we propose a heavy-tailed
adaptive reinforce algorithm to deal with sparse rewards in complex outdoor
environments.

##  3 Problem Formulation and Our Approach

###  3.1 Outdoor Navigation via Reinforcement Learning

Mathematically, we formulate the outdoor navigation problem as a Markov
Decision Process (MDP) in continuous state and actions spaces:

$\displaystyle\mathcal{M}:=\\{\mathcal{S},\,\mathcal{A},\,\mathbb{P},\,r,\,\gamma\\},$
(1)

where $\mathcal{S}$ denotes the state space including distance from the goal,
heading, roll and pitch angle; $\mathcal{A}$ is the actions space (linear,
angular velocity); $\mathbb{P}(s^{\prime}|s,a)$ is the transition kernel;
$r(s,a)$ is the reward and $\gamma\in(0,1)$ denotes the discount factor. The
objective of the robot is to learn a navigation policy $\pi_{\theta}(a|s)$
parameterized by $\theta$ (which controls the probability of taking a
particular action $a$ in given state $s$) to perform efficient outdoor
navigation. This is achieved by solving the following optimization problem

$\displaystyle\max_{{\theta}}J({\theta}):=V^{{\pi}_{{\theta}}}(s_{0}),$ (2)

where
$V^{{\pi}_{{\theta}}}(s_{0})=\mathbb{E}\big{[}\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})~{}|~{}s_{0}=s,a_{t}\sim{\pi}_{{\theta}}(\cdot|s_{t})\big{]}$
is the average cumulative reward (or value function), and $s_{0}$ denotes the
initial state along a trajectory
$\\{s_{t},a_{t},r(s_{t},a_{t})\\}_{u=0}^{\infty}$.We note that since the goal
of the outdoor navigating robot is to reach a goal and to make sure that the
problem (2) is actually achieving that, we need to design the reward function
$r(s,a)$ such that upon maximizing its cumulative value, our goal is achieved.
This is one of the major challenges in applying RL ideas to outdoor
navigation. The majority of the RL results revolve around learning in standard
environments and platforms like Gym, Mujoco [44] or other simulated
environments where the rewards structures are already defined. However, in
practical outdoor navigation scenarios, it is extremely hard to generate dense
rewards due to the possibility of huge unknown trajectories a robot can take
while navigating in the environment [34, 8]. The problem is exacerbated in
continuous state-action spaces where one needs to define rewards even for
infinite possibilities, which could be impractical.

###  3.2 Our Approach

To deal with this issue and empower the application of RL to outdoor
navigation, we take a different route and advocate the use of sparse rewards.
The advantage of sparse rewards is the simplicity of their design as they need
to be defined only at specific goals/sub-goals to the robot. However, the
sparse reward makes the learning problem hard due to the non-trivial
estimation of value functions over the continuous state space, which can be
mitigated through the use of heavy-tailed policy gradient algorithms[42, 21].
In this work, we start with a construction of an experimental sparse reward
design methodology for outdoor navigation scenarios and proceed with the
description of our algorithm.

####  3.2.1 Outdoor Navigation using Sparse Rewards

We categorize the level of complexity in the reward structure using three
navigation scenarios in outdoor environments. In all scenarios, our agent is
modeled as a differential drive robot with a two-dimensional continuous action
space $a=(v,\omega)$ (i.e. linear and angular velocities), whereas the state
space dimension varies for each scenario. The state inputs are obtained in
real-time from the robot’s odometry, and LiDAR sensors. The list of state
inputs used throughout the three scenarios is as follows:

  * •

$d_{goal}\in\mathbb{R}^{+}$ \- current distance between the robot and its
goal;

  * •

$\alpha_{goal}\in[0,\pi]$ \- current angle between the robot’s heading
direction and the goal;

  * •

$(v_{t-1},\omega_{t-1})\in[-1,1]$ \- actions from the previous time step (i.e.
linear and angular velocities);

  * •

$(\theta_{roll},\theta_{pitch})\in[0,\pi]$ \- roll and pitch angle of the
robot respectively;

  * •

$D_{obs}\in[0,10]$ \- laser scan vector that includes distance to the
obstacles around the robot (i.e. $360^{\circ}$ laser scan data as a vector
with $\sim 720$ elements) at a given time.

Scenario 1 (Goal Reaching Baseline): In this scenario, the robot is placed in
an obstacle-free outdoor terrain with an objective to reach a given goal
location. We incorporate distance to the goal, heading angle to the goal and
previous actions to define a four-dimensional state space
$s=[d_{goal},\alpha_{goal},v_{t-1},\omega_{t-1}]$. We utilize a set of sparse
rewards that are easy to implement for policy training. For the goal reaching
task, we define $r_{heading}$ and $r_{dist}$ rewards to maintain the robot’s
heading direction towards the goal and to encourage reaching the goal
respectively. Hence,

$r_{heading}=\mathbbm{1}_{\\{|\alpha_{goal}|\leq\pi/4\\}},\ \ \text{and}\ \
r_{dist}=\frac{\beta_{g}}{2}\mathcal{N}(\frac{d_{goal}}{2},\sigma_{g}^{2})+\beta_{g}\mathcal{N}(0,\sigma_{g}^{2}),$
(3)

where $\beta_{g}$ is a constant used to adjust the reward amplitude for goal
reaching, $\sigma_{g}\in[0,0.1]$ is the variance of the normal distribution
$\mathcal{N}(.,\sigma_{g}^{2})$ which we maintain at a very low range. This
ensures, the robot receives a reward $r_{dist}$ only when it reaches either
$\frac{d_{goal}}{2}$ or the defined goal. Otherwise, $r_{dist}$ is zero. These
reward formulations intuitively motivate the robot to reach the goal by
rewarding it when the halfway or the complete distance to the goal is
traveled. The total reward is obtained as, $r_{tot}=r_{heading}+r_{dist}$. The
reward distribution w.r.t the state parametric space is visualized in Figure
1(a) to highlight the reward sparseness.

Scenario 2 (Obstacle Avoidance): In this scenario, the robot is placed in an
outdoor area with static obstacles such as trees, walls, and sharp hills. The
objective is to reach the goal while avoiding collisions with these obstacles.
Additionally laser scan vector is taken into account in the state space
$s=[d_{goal},\alpha_{goal},v_{t-1},\omega_{t-1},D_{obs}]$. In addition to the
two rewards defined in the previous scenario (i.e. $r_{heading}$ and
$r_{dist}$), we introduce an additional sparse reward $r_{obs}$ as,

$r_{obs}=\begin{cases}-100&\text{if $\min(D_{obs})\leq d_{collision}$},\\\
0&\text{otherwise},\end{cases}$ (4)

to penalize collisions. Here, $d_{collision}$ is the minimum safety clearance
that the robot should maintain with obstacles. The modified total reward is
obtained as, $r_{tot}=r_{heading}+r_{dist}+r_{obs}$.

Figure 1: Sparse reward surface visualization: Reward distribution with (a)
$d_{goal}$, $\alpha_{goal}$ and (b) $d_{goal}$, $D_{obs}$. We observe that the
sparse reward settings lead to chaotic and unstructured reward surfaces. This
plot shows that our reward formulations in the navigation scenarios are
significantly sparse and do not include uniquely identifiable maxima that can
be reached smoothly.

Scenario 3 (Navigating on Uneven Terrains): In this scenario, the robot is
placed in a highly elevated outdoor terrain where it might flip over or
experience instability in some regions. The objective is to navigate to a goal
location while avoiding steep elevations in the environment. The corresponding
state space is
$s=[d_{goal},\alpha_{goal},v_{t-1},\omega_{t-1},\theta_{roll},\theta_{pitch}]$.
To minimize navigating on steep elevations, we incorporate another sparse
reward term ($r_{stable}$) with the goal reaching reward pair. We consider the
states where the robots roll ($\theta_{roll}$) or pitch angle
($\theta_{pitch}$) exceeds $\pm\frac{\pi}{4}$ as steep elevations (i.e.
unstable robot orientations). Hence, the stability sparse reward is defined
as,

$r_{stable}=\mathbbm{1}_{\\{|\theta_{roll}|\geq\pi/4\\}}\cup\mathbbm{1}_{\\{|\theta_{pitch}|\geq\pi/4\\}}.$
(5)

Hence, the total reward function for this scenario is obtained as,
$r_{tot}=r_{heading}+r_{dist}+r_{stable}$.

####  3.2.2 Heavy-Tailed Reinforce Algorithm for Outdoor Navigation (HTRON)

Just operating with sparse rewards is not sufficient to solve the outdoor
navigation problem because the learning procedure won’t be able to explore the
environment in a required manner. For instance, continuous control robotics
research primarily relies on Gaussian policy parametrization given by
$\pi_{{\theta}}(a|s)=\mathcal{N}(a|\varphi(s)^{\top}{\theta},\sigma^{2})$
where $\theta$ controls the mean of the Gaussian, $\varphi(s)$ denotes the
states feature representation $\varphi:\mathcal{S}\rightarrow\mathbb{R}^{d}$
with $d\ll q$, and $\sigma^{2}$ is variance. However, the performance of
Gaussian policy parametrization under sparse rewards is shown to suffer badly
in the latest research [42, 21]. The primary reason for failure is due to the
light-tailed nature of Gaussian distribution which restricts the policy to
take action closer to its mean value and thereby restricts exploration.
Typical methods to deal with exploration includes adding intrinsic curiosity
[45] or entropy based exploration [17] which either requires learning the
global dynamics model or estimation of the occupancy density function which is
extremely expensive for continuous control robotics problem.

In this work, we leverage an alternate approach motivated by the latest
research by [42, 21] and focus on developing a heavy-tailed parametrization
based policy for outdoor navigation problems with additional robotics-driven
constraints as detailed in Algorithm 1.

1: Initialize : Initial policy network parameter $\theta={\theta}_{0}$,
discount factor $\gamma$, step-size $\eta$, Adaptive momentum parameters
$\beta_{1}$, $\beta_{2}$, $\epsilon$, $\delta$, and $\phi$ Repeat for
$k=1,\dots$

2: Collect the trajectory $\xi_{k}(\theta_{k})$ by utilizing policy
$\pi_{\theta_{k}}$ with robotic action driven constraints i.e the actions are
first sampled as $a\sim\pi_{\theta_{k}}(a|s)$ and then we evaluate projections
$\mathcal{P}_{\infty}(a)$ such that $\|a\|_{\infty}\leq\delta$

3: Estimate ${\nabla}J({\theta}_{k},\xi_{k}(\theta_{k}))$ (cf. (7)) and then
perform clipping such that
$\|{\nabla}J({\theta}_{k},\xi_{k}(\theta_{k}))\|_{\infty}\leq\phi$

4: Estimate $g_{k}$ and $\theta_{k+1}$ with adaptive moment estimation method
using (8)

5: $k\leftarrow k+1$ Until Convergence

6: Return: ${\theta}_{k}$

Algorithm 1 Heavy-Tailed Reinforce for Outdoor Navigation (HTRON)

We parameterize the policy by a heavy-tailed Cauchy distribution given by

$\displaystyle\pi_{{\theta}}(a|s)=\frac{1}{\sigma\pi(1+((a-\varphi(s)^{\top}{\theta})/\sigma)^{2})},$
(6)

where $\sigma$ is the fixed variance. Now, finally we write the stochastic
policy gradient [20] as

$\displaystyle{\nabla}J({\theta}_{k},$
$\displaystyle\xi_{k}(\theta_{k}))=\sum_{t=0}^{T_{k}}\gamma^{t/2}r(s_{t},a_{t})\cdot\bigg{(}\sum_{\tau=0}^{t}\nabla\log\pi_{{\theta}_{k}}(a_{\tau}|s_{\tau})\bigg{)},$
(7)

where ${\nabla}J({\theta}_{k},\xi_{k}(\theta_{k}))$ denotes the unbiased
estimator of gradient $\nabla J(\theta_{k})$ at $\theta_{k}$, and
$T_{k}\sim\text{Geom}(1-\gamma^{1/2})$. Although heavy-tailed parametrization
induces exploration which has proven to be extremely beneficial in sparse
reward settings [42, 21], it also induces instability in behaviour due to the
high probability of taking extreme action even near optimal regions. Hence,
[42] proposed gradient tracking to mitigate the above issue. However, instead
of momentum tracking with mirror ascent type update as proposed in [42], we
use an adaptive moment estimation based method as an optimizer with gradient
clipping to stabilize the instability induced owing to the heavy-tailed
parametrization which improves the time taken by gradient tracking based
methods.

$\displaystyle m_{k}=\beta_{1}m_{k-1}+(1-\beta_{1})g_{k},$ (8) $\displaystyle
v_{k}=\beta_{2}v_{k-1}+(1-\beta_{2})g_{k}^{2},$ (9)
$\displaystyle\theta_{k+1}=\theta_{k}+\frac{\eta}{\sqrt{v_{k}}+\epsilon}m_{k},$
(10)

where $m_{k}$, $v_{k}$ are the first and second moments respectively and
$\beta_{1}$, $\beta_{2}$, $\epsilon$ are hyperparamters for the optimizer and
$g_{k}=\nabla J({\theta}_{k},\xi_{k}(\theta_{k}))$ with the added constraint
of clipping given as $\|\nabla
J({\theta}_{k},\xi_{k}(\theta_{k}))\|_{\infty}\leq\phi$. Another important
challenge is to incorporate practical constraints into the learning system
which is specific to robotics based problems where the action is bounded. To
solve the same, we use a simple projection $\mathcal{P}_{\infty}(\cdot)$ based
technique where we project the output from the policy $\pi_{\theta}(a|s)$ into
an infinite norm ball which constraints the action space within the bound and
can be shown as $a\sim\pi_{\theta}(a|s)$ such that $\|a\|_{\infty}\leq\delta$.

##  4 Experiments

We conduct a detailed analysis and performance comparison of our proposed
method against three state-of-the-art stochastic policy gradient algorithms:
TRPO [24], PPO [23], and Reinforce [22]. We used identical policy distribution
parameters (e.g. distribution standard deviation $\sigma$) and neural network
architectures with all four algorithms during training and evaluation.The
heavy-tailed policies are implemented and trained with Pytorch using a Unity
based realistic outdoor simulator, Clearpath Husky robot model, and ROS
Melodic platform. Further, we deploy HTRON on a real Clearpath Husky robot on
outdoor terrains.

The Unity simulator includes diverse terrains and elevations for training and
testing. We restrict ourselves to a $100m\times 100m$ region during training.
However, the real and simulated testing were conducted in different outdoor
scenarios. Particularly, the real-world environments are reasonably different
in terms of terrain structure(i.e. elevation gradient) and surface
properties(i.e grass and tiny gravel regions with different levels of
friction) which are not included or modeled in the simulator (e.g. elevation
gain in the simulator is up to $\sim 4m$, however, we restricted it to $\sim
3m$ elevation gain in the real world for robot’s safety).

Figure 2: Performance comparison of HTRON with Reinforce [22], PPO [23], TRPO
[24],  PPO (with Cauchy) and TRPO (with Cauchy) in three scenarios: (a) Goal-
reaching Baseline, (b) Obstacle avoidance, and (c) Navigating on uneven
terrains. Each episode contains $300$ times steps (i.e. $\sim$ $30k$-$42k$
overall training time steps). All the policies are trained with a fixed
standard deviation $\sigma=0.25$. We note that HTRON achieves the highest
average reward returns. The plots are averaged over $6$ random seeds and shows
mean and confidence intervals. Figure 3: Performance comparison of HTRON with
Reinforce [22], PPO [23], and TRPO [24] policies with large standard
deviations $(\sigma=0.5)$ in three scenarios: (a) Goal-reaching Baseline, (b)
Obstacle avoidance, and (c) Navigating on uneven terrains. We observe that the
Gaussian policies cannot achieve faster policy convergence by simply
increasing the standard deviation of the policy distribution. Instead, the
Gaussian policies demonstrate higher instability during training.

###  4.1 Policy Learning Performance

Figure 4: Trajectories when navigating on different simulated outdoor terrains
using HTRON, PPO[23], TRPO[24], Reinforce[22], DWA[30] and Ego-graph[46, 47]:
(a) Goal-reaching baseline, (b) Obstacle avoidance environment with multiple
trees, walls, and unreachable steep elevations. (c) Uneven terrain environment
with different levels of elevations and flat terrains. HTRON takes the
shortest time steps to successfully reach the goal while maintaining the
lowest elevation cost on uneven terrains. Figure 5: Navigating in real world
using HTRON: (a) Goal-reaching scenario, (b) Obstacle avoidance. (c) Uneven
terrain navigation. We deploy our algorithm on Clearpath Husky robot
demonstrating the ease of transferring HTRON on real systems without
significant performance degradation. We observe comparable or better
performance from HTRON in compared to classical planners DWA [30] and Ego-
graph [46, 47], especially on uneven terrains. The robot is equipped with a
VLP16 LiDAR, and a laptop with an Intel i9 CPU and an Nvidia RTX 2080 GPU.

We evaluate the performance of all algorithms through the policy convergence
rate (see Figure 2). In the goal-reaching baseline scenario, we observed that
standard Gaussian policy-based algorithms cannot achieve the highest reward
return when we incorporate the sparse rewards. In contrast, our heavy-tailed
policy is able to obtain maximum rewards in a sample-efficient manner as shown
in Figure 2(a). This is primarily due to the improved exploratory behavior
achieved by the heavy-tailed policy in comparison to the standard Gaussian
policies.

In Scenario 2 with static obstacles, our experiments showed that the random
and sparse obstacle penalties significantly affect policy convergence and
stability. In particular, PPO converges to the halfway point towards the goal
with multiple collisions. However, our method successfully learns policies to
reach the goal within $\sim 40$ episodes (see Figure 2(b)). This demonstrates
the accelerated learning capabilities provided by the heavy-tailed formulation
compared to the traditional stochastic policy gradient methods. In Scenario 3,
we include $r_{stable}$ to penalize the actions that could navigate the robot
towards steep elevations. We observe that Reinforce and TRPO based policies
struggle to even reach the goal under this setting while PPO manages to reach
the goal in only a few episodes without proper convergence.

Further, we performed additional experiments on PPO, TRPO, and Reinforce
algorithms by increasing the standard deviation ($\sigma=0.5$) of their
Gaussian policy distribution(see Fig. 3). We observe that even with increasing
variance, our algorithm (HTRON) outperforms all the baselines including PPO,
TRPO, Reinforce, etc which demonstrates the superiority of our algorithm.
Additionally, We observe that the Gaussian policies cannot achieve faster or
better policy convergence by simply increasing the standard deviation of the
policy distribution. Instead, the Gaussian policies demonstrate higher
instability during training. Hence, we restricted to $\sigma=0.25$ as
presented in Fig. 2 during our experiments.

Also, an important point is to understand that PPO and TRPO both fall under
trust-region-based policy gradient algorithms which utilize constrained policy
optimization. In other words, the policies are constrained to be in a trust
region and explore less in the initial iterations which are definitely not
suitable for Sparse rewards scenarios. Hence, it has been seen in literature
[20, 21] that PPO and TRPO fail to perform optimally under sparse reward
scenarios as also can be seen from our experiments.

###  4.2 Navigation Performance

We compare our method’s navigation performance qualitatively in Figure 4 and
quantitatively in Table 1. The quantitative evaluations of the navigation
trajectories are based on the following metrics:

  * •

Success Rate: The percentage of successful goal reaching attempts without any
collisions or flip-overs out of the total experiments.

  * •

Avg. Trajectory Length: The average number of time steps taken by the robot to
reach the goal.

  * •

Elevation Cost: Norm of the elevation gradient experienced by the robot
throughout a trajectory (i.e. $||\nabla z_{r}||$, where $z_{r}$ is the vector
that includes vertical motions of the robot’s along a trajectory).

We would like to note that in this work rather than proposing a competitive
navigation algorithm, the primary objective is to highlight the importance of
heavy-tailed distributions for faster policy convergence under practical
sparse reward settings. However, in addition to comparing against the DRL-
based methods, we evaluate the navigation performance of two classical
algorithms: Dynamic Window Approach(DWA) and Ego-graph) under the same test
conditions. DWA only utilizes 2D scan data for obstacle avoidance. Hence, it
performs well during the goal reaching and obstacle avoidance tasks. Ego-graph
baseline method in [47] utilizes robot-centric elevation data to obtain the
actions that minimize the elevation gradient cost. Hence, the Ego-graph
planner is capable of navigating uneven terrains successfully. However, such
methods’ behavior is limited by the size of the action space which could lead
to inconsistent navigation performance on complex uneven terrains.

We observe that DWA and Ego-graph outperform all the DRL based methods in
terms of success rate during goal reaching and obstacle avoidance. However,
HTRON demonstrates comparable or better performance during uneven terrain
navigation in terms of all the evaluation metrics.

We observe that our method maintains the highest success rate in all three
scenarios while other DRL policies demonstrate a significantly low performance
in successful goal-reaching. This is primarily due to the poor convergence in
Gaussian policies under sparse reward settings. Further, HTRON reaches the
goal faster using fewer time steps than the other three methods. This
indicates that our policy has optimized to collect more goal completion
rewards by reaching the goal location quickly. Policy convergence plots
presented in Figure 2 further validates this argument.

Scenario | Algorithm | Success Rate | Avg. Trajectory Length | Elevation Cost
---|---|---|---|---
|  | (%)$\uparrow$ | (# time steps)$\downarrow$ | (m)$\downarrow$
Goal Reaching Baseline | TRPO[24] | 32.28 | 236 | 0.764
| PPO[23] | 45.76 | 198 | 0.648
| Reinforce[22] | 36.91 | 207 | 0.655
| DWA[30] | 100 | 172 | 0.645
| Ego-graph[47] | 88.86 | 186 | 0.637
|  HTRON(Ours) | 72.46 | 168 | 0.641
Obstacle Avoidance | TRPO[24] | 18.74 | 243 | 0.637
| PPO[23] | 37.56 | 224 | 0.652
| Reinforce[22] | 8.93 | 286 | 0.624
| DWA[30] | 100 | 218 | 0.645
| Ego-graph[47] | 58.79 | 238 | 0.642
|  HTRON(Ours) | 62.21 | 209 | 0.639
Navigating on Uneven Terrains | TRPO[24] | 25.87 | 273 | 1.842
| PPO[23] | 37.46 | 264 | 1.674
| Reinforce[22] | 18.33 | 270 | 1.336
| DWA[30] | 27.38 | 268 | 1.943
| Ego-graph[47] | 65.89 | 198 | 1.316
|  HTRON(Ours) | 71.87 | 244 | 1.257

Table 1:  Navigation Comparisons: HTRON outperforms the other DRL methods in
terms of success rate and the average time to reach the goal. It also
maintains the lowest elevation cost when navigating on uneven terrains.
Reinforce and TRPO obtain slightly lower elevation costs during goal-reaching
and obstacle avoidance scenarios by rotating around flat regions without
reaching the actual goal. Further, HTRON demonstrates comparable performance
in terms of trajectory length and elevation cost compared to classical methods
(DWA and Ego-graph) in goal reaching and obstacle avoidance scenarios.
Especially, HTRON outperforms all the other methods when navigating on uneven
terrains.

Navigation scenarios 1 and 2 are common in both indoor and outdoor
environments. Hence, we further investigate the importance of our algorithm to
handle uneven terrains encountered in complex outdoor environments. We observe
that our method is capable of avoiding steep elevations from the changing roll
and pitch angles before leading to robot flip-overs (see Figure 2(c) and 4(c)
). Hence, trajectories generated by our algorithm significantly minimize the
elevation cost experienced by the robot while other methods navigate along
steep terrains with higher elevation costs.

Finally, we integrate our algorithm into a real Clearpath Husky robot to
demonstrate navigation capabilities in real outdoor settings. We observe that
our method can successfully perform the navigation tasks we trained in the
simulator. Sample navigation trajectories from three real outdoor scenarios
are presented in Figure 5.

##  5 Limitations and Analysis

We discuss the limitations of our method that we expect to address in the
future. The robot cannot identify the uneven terrains without experiencing the
uneven region due to the orientation based elevation sensing. A 3D LiDAR
pointcloud based elevation map could be utilized in state space to identify
the elevation changes in the robot’s vicinity without explicitly experiencing
elevation changes from the orientation data. Formulating sparse rewards on
such elevation maps to make early decisions to avoid uneven terrains would be
an interesting research problem. Moreover, the heavy-tailedness of the policy
distribution could lead to convergence instabilities. Especially, we observe
considerable instability and non-converging behavior when we try to use PPO
and TRPO with heavy-tailed distributions. This could be due to the heavy-
tailed nature of the gradients of the PPO and TRPO as suggested in the
empirical study [48] (i.e. Heavy-tailedness of the gradient increases as the
agent’s policy diverges from the behavioral policy). Hence, further
investigations need to be conducted to develop robust and stable heavy-tailed
policy models for higher dimensional state spaces and more complex sparse
reward settings.

##  6 Conclusion

In this work, we presented a novel approach for adapting heavy-tailed policy
based control strategy for efficient outdoor mobile robot navigation – HTRON.
The proposed method is able to overcome the major issue of sparse rewards
inherent to DRL based approaches. We show that without hand-crafted reward
shaping methods we are able to accelerate learning capabilities and produce a
reliable navigation policy. We have assessed the performance of navigation
policies in three different scenarios based on their complexity and
corresponding objectives: 1) goal reaching, 2) obstacle avoidance in a field
with multiple trees, walls, and unreachable steep elevations, and 3)
navigating uneven elevations and flat terrains. All policies have been trained
in a ROS-based high-fidelity Unity simulator. We report a performance
comparison of our method against REINFORCE, PPO, and TRPO algorithms in terms
of learning rate and navigation quality. We also deployed HTRON on real
Clearpath Husky in outdoor terrains. And finally, we report the limitations,
possible improvements, and future work that this approach can lead to robust
outdoor navigation.

#### Acknowledgments

This work was supported in part by ARO Grants W911NF1910069, W911NF2110026,
and Army Cooperative Agreement W911NF2120076. We acknowledge the support of
the Maryland Robotics Center.

## References

  * Lee et al. [2021] D. Lee, G. Kang, B. Kim, and D. H. Shim.  Assistive delivery robot application for real-world postal services.  _IEEE Access_ , 9:141981–141998, 2021.  doi:10.1109/ACCESS.2021.3120618.
  * Liu and Nejat [2016] Y. Liu and G. Nejat.  Multirobot cooperative learning for semiautonomous control in urban search and rescue applications.  _Journal of Field Robotics_ , 33(4):512–536, 2016\.
  * Thoesen and Marvi [2021] A. Thoesen and H. Marvi.  Planetary surface mobility and exploration: A review.  _Current Robotics Reports_ , 2(3):239–249, 2021\.
  * Dhiman et al. [2018] V. Dhiman, S. Banerjee, B. Griffin, J. M. Siskind, and J. J. Corso.  A critical investigation of deep reinforcement learning for navigation.  _arXiv preprint arXiv:1802.02274_ , 2018.
  * Chen et al. [2017] Y. F. Chen, M. Liu, M. Everett, and J. P. How.  Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning.  In _2017 IEEE international conference on robotics and automation (ICRA)_ , pages 285–292. IEEE, 2017.
  * Zhu and Zhang [2021] K. Zhu and T. Zhang.  Deep reinforcement learning based mobile robot navigation: A review.  _Tsinghua Science and Technology_ , 26(5):674–691, 2021.
  * Li et al. [2019] H. Li, Q. Zhang, and D. Zhao.  Deep reinforcement learning-based automatic exploration for navigation in unknown environment.  _IEEE transactions on neural networks and learning systems_ , 31(6):2064–2076, 2019.
  * Liang et al. [2022] J. Liang, K. Weerakoon, T. Guan, N. Karapetyan, and D. Manocha.  Adaptiveon: Adaptive outdoor navigation method for stable and reliable motions.  _arXiv preprint arXiv:2205.03517_ , 2022.
  * Long et al. [2018] P. Long, T. Fan, X. Liao, W. Liu, H. Zhang, and J. Pan.  Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 6252–6259. IEEE, 2018.
  * Sathyamoorthy et al. [2020] A. J. Sathyamoorthy, J. Liang, U. Patel, T. Guan, R. Chandra, and D. Manocha.  Densecavoid: Real-time navigation in dense crowds using anticipatory behaviors.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 11345–11352. IEEE, 2020.
  * Zai and Brown [2020] A. Zai and B. Brown.  _Deep reinforcement learning in action_.  Manning Publications, 2020.
  * Jiang et al. [2019] N. Jiang, S. Jin, and C. Zhang.  Hierarchical automatic curriculum learning: Converting a sparse reward navigation task into dense reward.  _Neurocomputing_ , 360:265–278, 2019.
  * Wang et al. [2020] C. Wang, J. Wang, J. Wang, and X. Zhang.  Deep-reinforcement-learning-based autonomous uav navigation with sparse rewards.  _IEEE Internet of Things Journal_ , 7(7):6180–6190, 2020.
  * Chiang et al. [2019] H.-T. L. Chiang, A. Faust, M. Fiser, and A. Francis.  Learning navigation behaviors end-to-end with autorl.  _IEEE Robotics and Automation Letters_ , 4(2):2007–2014, 2019.
  * Vecerik et al. [2017] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller.  Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.  _arXiv preprint arXiv:1707.08817_ , 2017.
  * Karnan et al. [2021] H. Karnan, G. Warnell, X. Xiao, and P. Stone.  Voila: Visual-observation-only imitation learning for autonomous navigation.  _arXiv preprint arXiv:2105.09371_ , 2021.
  * Houthooft et al. [2016] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel.  Vime: Variational information maximizing exploration.  _Advances in neural information processing systems_ , 29, 2016.
  * Konar et al. [2021] A. Konar, B. H. Baghi, and G. Dudek.  Learning goal conditioned socially compliant navigation from demonstration using risk-based features.  _IEEE Robotics and Automation Letters_ , 6(2):651–658, 2021.
  * Hare [2019] J. Hare.  Dealing with sparse rewards in reinforcement learning.  _arXiv preprint arXiv:1910.09281_ , 2019.
  * Bedi et al. [2021] A. S. Bedi, A. Parayil, J. Zhang, M. Wang, and A. Koppel.  On the sample complexity and metastability of heavy-tailed policy search in continuous control.  _arXiv preprint arXiv:2106.08414_ , 2021.
  * Chakraborty et al. [2022] S. Chakraborty, A. S. Bedi, A. Koppel, P. Tokekar, and D. Manocha.  Dealing with sparse rewards in continuous control robotics via heavy-tailed policies.  _arXiv_ , 2022.
  * Williams [1992] R. J. Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ , 8(3):229–256, 1992.
  * Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Schulman et al. [2015] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz.  Trust region policy optimization.  In _International conference on machine learning_ , pages 1889–1897. PMLR, 2015.
  * Sathyamoorthy et al. [2022] A. J. Sathyamoorthy, K. Weerakoon, T. Guan, J. Liang, and D. Manocha.  Terrapn: Unstructured terrain navigation through online self-supervised learning.  _arXiv preprint arXiv:2202.12873_ , 2022.
  * Weerakoon et al. [2022] K. Weerakoon, A. J. Sathyamoorthy, U. Patel, and D. Manocha.  Terp: Reliable planning in uneven outdoor environments using deep reinforcement learning.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 9447–9453, 2022.
  * Patle et al. [2019] B. Patle, A. Pandey, D. Parhi, A. Jagadeesh, et al.  A review: On path planning strategies for navigation of mobile robot.  _Defence Technology_ , 15(4):582–606, 2019.
  * Xiao et al. [2022] X. Xiao, B. Liu, G. Warnell, and P. Stone.  Motion planning and control for mobile robot navigation using machine learning: a survey.  _Autonomous Robots_ , pages 1–29, 2022.
  * Dijkstra et al. [1959] E. W. Dijkstra et al.  A note on two problems in connexion with graphs.  _Numerische mathematik_ , 1(1):269–271, 1959\.
  * Fox et al. [1997] D. Fox, W. Burgard, and S. Thrun.  The dynamic window approach to collision avoidance.  _IEEE Robotics & Automation Magazine_, 4(1):23–33, 1997.
  * Kuffner and LaValle [2000] J. J. Kuffner and S. M. LaValle.  Rrt-connect: An efficient approach to single-query path planning.  In _Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065)_ , volume 2, pages 995–1001. IEEE, 2000.
  * Kavraki et al. [1996] L. E. Kavraki, P. Svestka, J.-C. Latombe, and M. H. Overmars.  Probabilistic roadmaps for path planning in high-dimensional configuration spaces.  _IEEE transactions on Robotics and Automation_ , 12(4):566–580, 1996.
  * Zhou et al. [2021] C. Zhou, B. Huang, and P. Fränti.  A review of motion planning algorithms for intelligent robots.  _Journal of Intelligent Manufacturing_ , pages 1–38, 2021.
  * Shah et al. [2021] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine.  Ving: Learning open-world navigation with visual goals.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 13215–13222. IEEE, 2021.
  * Zhang et al. [2018] K. Zhang, F. Niroui, M. Ficocelli, and G. Nejat.  Robot navigation of environments with unknown rough terrain using deep reinforcement learning.  In _2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR)_ , pages 1–7. IEEE, 2018.
  * Josef and Degani [2020] S. Josef and A. Degani.  Deep reinforcement learning for safe local planning of a ground vehicle in unknown rough terrain.  _IEEE Robotics and Automation Letters_ , 5(4):6748–6755, 2020.
  * Guan et al. [2021] T. Guan, D. Kothandaraman, R. Chandra, A. J. Sathyamoorthy, and D. Manocha.  Ganav: Group-wise attention network for classifying navigable regions in unstructured outdoor environments.  _arXiv preprint arXiv:2103.04233_ , 2021.
  * Hussein et al. [2017] A. Hussein, E. Elyan, M. M. Gaber, and C. Jayne.  Deep reward shaping from demonstrations.  In _2017 International Joint Conference on Neural Networks (IJCNN)_ , pages 510–517. IEEE, 2017.
  * Botteghi et al. [2020] N. Botteghi, B. Sirmacek, K. A. Mustafa, M. Poel, and S. Stramigioli.  On reward shaping for mobile robot navigation: A reinforcement learning and slam based approach.  _arXiv preprint arXiv:2002.04109_ , 2020.
  * Nair et al. [2018] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel.  Overcoming exploration in reinforcement learning with demonstrations.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pages 6292–6299. IEEE, 2018.
  * Ravichandar et al. [2020] H. Ravichandar, A. Polydoros, S. Chernova, and A. Billard.  Recent advances in robot learning from demonstration.  _Annual Review of Control, Robotics, and Autonomous Systems_ , 3(ARTICLE):297–330, 2020.
  * Bedi et al. [2022] A. S. Bedi, S. Chakraborty, A. Parayil, B. M. Sadler, P. Tokekar, and A. Koppel.  On the hidden biases of policy mirror ascent in continuous action spaces.  _CoRR_ , abs/2201.12332, 2022.  URL https://arxiv.org/abs/2201.12332.
  * Chou [2017] P.-W. Chou.  _The beta policy for continuous control reinforcement learning_.  PhD thesis, Master’s thesis. Pittsburgh: Carnegie Mellon University, 2017.
  * Chen et al. [2021] X.-H. Chen, S. Jiang, F. Xu, Z. Zhang, and Y. Yu.  Cross-modal domain adaptation for cost-efficient visual reinforcement learning.  _Advances in Neural Information Processing Systems_ , 34:12520–12532, 2021.
  * Pathak et al. [2017] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _International conference on machine learning_ , pages 2778–2787. PMLR, 2017.
  * Lacaze et al. [1998] A. Lacaze, Y. Moscovitz, N. DeClaris, and K. Murphy.  Path planning for autonomous vehicles driving over rough terrain.  In _Proceedings of the 1998 IEEE International Symposium on Intelligent Control (ISIC) held jointly with IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA) Intell_ , pages 50–55, 1998.  doi:10.1109/ISIC.1998.713634.
  * Josef and Degani [2020] S. Josef and A. Degani.  Deep reinforcement learning for safe local planning of a ground vehicle in unknown rough terrain.  _IEEE Robotics and Automation Letters_ , 5(4):6748–6755, 2020.  doi:10.1109/LRA.2020.3011912.
  * Garg et al. [2021] S. Garg, J. Zhanson, E. Parisotto, A. Prasad, Z. Kolter, Z. Lipton, S. Balakrishnan, R. Salakhutdinov, and P. Ravikumar.  On proximal policy optimization’s heavy-tailed gradients.  In _International Conference on Machine Learning_ , pages 3610–3619. PMLR, 2021.