\newcommand{\etalchar}[1]{$^{#1}$}
\begin{thebibliography}{HMM{\etalchar{+}}12}

[1]
Scott Aaronson.
\newblock On perfect completeness for qma, 2008.

[2]
Leonard~M. Adleman, Jonathan DeMarrais, and Ming-Deh~A. Huang.
\newblock Quantum computability.
\newblock {\em SIAM Journal on Computing}, 26(5):1524--1540, 1997.

[3]
Dorit Aharonov, Vaughan Jones, and Zeph Landau.
\newblock A polynomial quantum algorithm for approximating the jones
\newblock In {\em Proceedings of the thirty-eighth annual ACM symposium on
Theory of computing}, pages 427--436, 2006.

[4]
Micha{\l} Adamaszek and Juraj Stacho.
\newblock Complexity of simplicial homology and independence complexes of
chordal graphs.
\newblock {\em Computational Geometry}, 57:8--18, 2016.

[5]
Simon Apers, Sayantan Sen, and D{\'a}niel Szab{\'o}.
\newblock A (simple) classical algorithm for estimating betti numbers.
\newblock {\em arXiv preprint arXiv:2211.09618}, 2022.

[6]
Ismail~Yunus Akhalwaya, Shashanka Ubaru, Kenneth~L Clarkson, Mark~S Squillante,
Vishnu Jejjala, Yang-Hui He, Kugendran Naidoo, Vasileios Kalantzis, and Lior
\newblock Towards quantum advantage on noisy quantum computers.
\newblock {\em arXiv preprint arXiv:2209.09371}, 2022.

[7]
Dominic~W Berry, Graeme Ahokas, Richard Cleve, and Barry~C Sanders.
\newblock Efficient quantum algorithms for simulating sparse hamiltonians.
\newblock {\em Communications in Mathematical Physics}, 270:359--371, 2007.

[8]
Dominic~W Berry, Andrew~M Childs, Richard Cleve, Robin Kothari, and Rolando~D
\newblock Exponential improvement in precision for simulating sparse
\newblock In {\em Proceedings of the forty-sixth annual ACM symposium on Theory
of computing}, pages 283--292, 2014.

[9]
Bela Bauer, Liza Huijse, Erez Berg, Matthias Troyer, and Kareljan Schoutens.
\newblock Supersymmetric multicritical point in a model of lattice fermions.
\newblock {\em Physical Review B}, 87(16), apr 2013.

[10]
Fernando~GSL Brand{\~a}o.
\newblock Entanglement theory and the quantum simulation of many-body physics.
\newblock {\em arXiv preprint arXiv:0810.0026}, 2008.

[11]
Sergey Bravyi.
\newblock Efficient algorithm for a quantum analogue of 2-sat.
\newblock {\em Contemporary Mathematics}, 536:33--48, 2011.

[12]
Dominic~W Berry, Yuan Su, Casper Gyurik, Robbie King, Joao Basso, Alexander
Del~Toro Barba, Abhishek Rajput, Nathan Wiebe, Vedran Dunjko, and Ryan
\newblock Quantifying quantum advantage in topological data analysis.
\newblock {\em arXiv preprint arXiv:2209.13581}, 2022.

[13]
Chris Cade and P~Marcos Crichigno.
\newblock Complexity of supersymmetric systems and the cohomology problem.
\newblock {\em arXiv preprint arXiv:2107.00011}, 2021.

[14]
Nai-Hui Chia, Andr{\'a}s~Pal Gily{\'e}n, Tongyang Li, Han-Hsuan Lin, Ewin Tang,
and Chunhao Wang.
\newblock Sampling-based sublinear low-rank matrix arithmetic framework for
dequantizing quantum machine learning.
\newblock {\em Journal of the ACM}, 69(5):1--72, 2022.

[15]
Timothy~Y Chow.
\newblock You could have invented spectral sequences.
\newblock {\em Notices of the AMS}, 53:15--19, 2006.

[16]
Andrew~M Childs and Robin Kothari.
\newblock Simulating sparse hamiltonians with star decompositions.
\newblock In {\em Theory of Quantum Computation, Communication, and
Cryptography: 5th Conference, TQC 2010, Leeds, UK, April 13-15, 2010, Revised
Selected Papers 5}, pages 94--103. Springer, 2011.

[17]
Marcos Crichigno and Tamara Kohler.
\newblock Clique homology is qma1-hard.
\newblock {\em arXiv preprint arXiv:2209.11793}, 2022.

[18]
Toby~S Cubitt, Ashley Montanaro, and Stephen Piddock.
\newblock Universal quantum hamiltonians.
\newblock {\em Proceedings of the National Academy of Sciences},
115(38):9497--9502, 2018.

[19]
P.~Marcos Crichigno.
\newblock {Supersymmetry and Quantum Computation}.
\newblock 11 2020.

[20]
Bill Fefferman and Cedric Yen-Yu Lin.
\newblock A complete characterization of unitary quantum space, 2016.

[21]
Robin Forman.
\newblock Hodge theory and spectral sequences.
\newblock {\em Topology}, 33(3):591--611, 1994.

[22]
Paul Fendley, Kareljan Schoutens, and Jan de~Boer.
\newblock {Lattice models with N=2 supersymmetry}.
\newblock {\em Phys. Rev. Lett.}, 90:120402, 2003.

[23]
Casper Gyurik, Chris Cade, and Vedran Dunjko.
\newblock Towards quantum advantage via topological data analysis.
\newblock {\em Quantum}, 6:855, 2022.

[24]
Chad Giusti, Robert Ghrist, and Danielle~S Bassett.
\newblock Two's company, three (or more) is a simplex.
\newblock {\em Journal of computational neuroscience}, 41(1):1--14, 2016.

[25]
Sevag Gharibian, Yichen Huang, Zeph Landau, Seung~Woo Shin, et~al.
\newblock Quantum hamiltonian complexity.
\newblock {\em Foundations and Trends{\textregistered} in Theoretical Computer
Science}, 10(3):159--282, 2015.

[26]
Sam Gunn and Niels Kornerup.
\newblock Review of a quantum algorithm for betti numbers.
\newblock {\em arXiv preprint arXiv:1906.07673}, 2019.

[27]
Andr{\'a}s Gily{\'e}n, Seth Lloyd, and Ewin Tang.
\newblock Quantum-inspired low-rank stochastic regression with logarithmic
dependence on the dimension.
\newblock {\em arXiv preprint arXiv:1811.04909}, 2018.

[28]
David Gosset and Daniel Nagaj.
\newblock Quantum 3-sat is qma \_1-complete.
\newblock {\em SIAM Journal on Computing}, 45(3):1080--1128, 2016.

[29]
Timothy~E Goldberg.
\newblock Combinatorial laplacians of simplicial complexes.
\newblock {\em Senior Thesis, Bard College}, 6, 2002.

[30]
Ryu Hayakawa.
\newblock Quantum algorithm for persistent betti numbers and topological data
\newblock {\em Quantum}, 6:873, 2022.

[31]
L~Huijse, D~Mehta, N~Moran, K~Schoutens, and J~Vala.
\newblock Supersymmetric lattice fermions on the triangular lattice:
superfrustration and criticality.
\newblock {\em New Journal of Physics}, 14(7):073002, jul 2012.

[32]
Liza Huijse.
\newblock {\em A supersymmetric model for lattice fermions}.
\newblock PhD thesis, Amsterdam U., 2010.

[33]
Stephen~P. Jordan, Hirotada Kobayashi, Daniel Nagaj, and Harumichi Nishimura.
\newblock Achieving perfect completeness in classical-witness quantum
merlin-arthur proof systems, 2012.

[34]
A.~Yu. Kitaev.
\newblock Quantum measurements and the abelian stabilizer problem, 1995.

[35]
Julia Kempe, Alexei Kitaev, and Oded Regev.
\newblock The complexity of the local hamiltonian problem.
\newblock {\em Siam journal on computing}, 35(5):1070--1097, 2006.

[36]
Alexei~Yu Kitaev, Alexander Shen, and Mikhail~N Vyalyi.
\newblock {\em Classical and quantum computation}.
\newblock Number~47. American Mathematical Soc., 2002.

[37]
Guang~Hao Low and Isaac~L Chuang.
\newblock Optimal hamiltonian simulation by quantum signal processing.
\newblock {\em Physical review letters}, 118(1):010501, 2017.

[38]
Seth Lloyd, Silvano Garnerone, and Paolo Zanardi.
\newblock Quantum algorithms for topological and geometric analysis of data.
\newblock {\em Nature communications}, 7(1):10138, 2016.

[39]
A Mathematica file containing details of calculations is available at:
\url{https://www.dropbox.com/sh/p3nwoqacyj0gl1l/AADDb6aULOSn7ZChfSsq1NTba?dl=0}.

[40]
John McCleary.
\newblock {\em A user's guide to spectral sequences}.
\newblock Number~58. Cambridge University Press, 2001.

[41]
Sam McArdle, Andr{\'a}s Gily{\'e}n, and Mario Berta.
\newblock A streamlined quantum algorithm for topological data analysis with
exponentially fewer qubits.
\newblock {\em arXiv preprint arXiv:2209.12887}, 2022.

[42]
Tobias~J Osborne.
\newblock Hamiltonian complexity.
\newblock {\em Reports on progress in physics}, 75(2):022001, 2012.

[43]
Giovanni Petri, Paul Expert, Federico Turkheimer, Robin Carhart-Harris, David
Nutt, Peter~J Hellyer, and Francesco Vaccarino.
\newblock Homological scaffolds of brain functional networks.
\newblock {\em Journal of The Royal Society Interface}, 11(101):20140873, 2014.

[44]
Michael Reimann, Max Nolte, Martina Scolamiero, Katharine Turner, Rodrigo
Perin, Giuseppe Chindemi, Pawel Dlotko, Ran Levi, Kathryn Hess, and Henry
\newblock {Cliques of Neurons Bound into Cavities Provide a Missing Link
between Structure and Function}.
\newblock {\em {Frontiers in Computational Neuroscience}}, 11:1--16, June 2017.

[45]
Yaoyun Shi.
\newblock Both toffoli and controlled-not need little help to do universal
quantum computing.
\newblock {\em Quantum Information \& Computation}, 3(1):84--92, 2003.

[46]
Alexander Schmidhuber and Seth Lloyd.
\newblock Complexity-theoretic limitations on quantum algorithms for
topological data analysis.
\newblock {\em arXiv preprint arXiv:2209.14286}, 2022.

[47]
Ewin Tang.
\newblock A quantum-inspired classical algorithm for recommendation systems.
\newblock In {\em Proceedings of the 51st annual ACM SIGACT symposium on theory
of computing}, pages 217--228, 2019.

[48]
Shashanka Ubaru, Ismail~Yunus Akhalwaya, Mark~S Squillante, Kenneth~L Clarkson,
and Lior Horesh.
\newblock Quantum topological data analysis with linear depth and exponential
\newblock {\em arXiv preprint arXiv:2108.02811}, 2021.

[49]
Larry Wasserman.
\newblock Topological data analysis.
\newblock {\em Annual Review of Statistics and Its Application}, 5:501--532,

[50]
Edward Witten.
\newblock Supersymmetry and morse theory.
\newblock {\em Journal of differential geometry}, 17(4):661--692, 1982.

[51]
Edward Witten.
\newblock Quantum field theory and the jones polynomial.
\newblock {\em Communications in Mathematical Physics}, 121(3):351--399, 1989.

\end{thebibliography}

\end{document}

\begin{comment}
\begin{abstract}
We show that the problem of deciding if a vertex-weighted graph has a $k$-dimensional hole, given a suitable promise in the NO case, is $\QMA_1$-hard and contained in $\QMA$. This extends the computational relationship between homology, supersymmetry and quantum mechanics, initiated in [13, 17]. Our proof uses a powerful tool from algebraic topology known as \emph{spectral sequences}, exploiting a connection between spectral sequences and Hodge theory presented by Forman [21]. We discuss the implications for quantum advantage in topological data analysis problems.
\end{abstract}
\end{comment}


# On the Courant-Friedrichs-Lewy condition for numerical solvers of the
coagulation equation

Guillaume Laibe1,2, Maxime Lombart1
1Univ Lyon, Univ Lyon1, Ens de Lyon, CNRS, Centre de Recherche Astrophysique
de Lyon UMR5574, F-69230, Saint-Genis,-Laval, France.
2Institut Universitaire de France
guillaume.laibe<EMAIL_ADDRESS>
###### Abstract

Evolving the size distribution of solid aggregates challenges simulations of
young stellar objects. Among other difficulties, generic formulae for
stability conditions of explicit solvers provide severe constrains when
integrating the coagulation equation for astrophysical objects. Recent
numerical experiments have recently reported that these generic conditions may
be much too stringent. By analysing the coagulation equation in the Laplace
space, we explain why this is indeed the case and provide a novel stability
condition which avoids time over-sampling.

###### keywords:

methods: numerical — (ISM:) dust, extinction

††pagerange: On the Courant-Friedrichs-Lewy condition for numerical solvers of
the coagulation equation–Data availability††pubyear: 2021

##  1 Introduction

The coagulation equation – also called Smoluchowski equation – is one of the
fundamental equation of physics, since it describes mass conservation for a
distribution of interacting particles (Banasiak et al., 2019). It plays a
central role in the formation of planets, since solid bodies, originating from
the interstellar medium, have to grow over thirty orders of magnitude in mass
to form cores of planets (Chiang & Youdin, 2010). As they grow, grains undergo
a complex interplay between coagulation and dynamics since dust/gas
interaction depends strongly on the size of the dust grains (Testi et al.,
2014). Hence the necessity of performing three dimensional simulations of
young stellar objects that integrate the coagulation equation in a self-
consistent manner (Haworth et al., 2016). However, this task was long thought
to be computationally prohibitive, since no hydrodynamical code could handle
the large number of dust bins required to solve for the coagulation equation
without over-diffusion. Recently, Lombart & Laibe (2021) showed that over-
diffusivity at small bin numbers could be overcome by the mean of a
Discontinuous Galerkine algorithm of high spatial order (Liu et al., 2019).
Still, to maintain practical performance, the coagulation solver should not be
called too often per hydrodynamical time step (Drążkowska et al., 2014, 2019).
Stability condition for explicit schemes have been studied since the very
beginning of the numerical study of the coagulation equation (Filbet &
Laurencot, 2004; Dullemond & Dominik, 2005; Gabriel & Tine, 2010; Forestier-
Coste & Mancini, 2012; Liu et al., 2019). Time stepping is set by the so-
called Courant-Friedrichs-Lewy condition (or CFL condition, Courant et al.
1928), which is thought to become drastically small for planet formation,
making the solver of no practical use. An alternate solution consists of using
implicit solvers, an approach successfully used for fragmentation (Brauer et
al., 2008; Birnstiel et al., 2010; Mahoney & Ramkrishna, 2002; Jacobson, 2005;
Sandu, 2006), but at the cost of heavy linear algebra operations that increase
with the order of the scheme.

Remarkably, Liu et al. (2019) noticed that his numerical solver was
numerically stable for a time step that is orders of magnitude larger than the
one given by the generic CFL condition (Sect. 2) ’Only for $\Delta t\leq
0.005$ do we observe a stable solution without using a reconstruction step.
This is a significant restriction. With the use of the scaling limiter
presented above, we observe that no negative values are generated by the
scheme and therefore the solution remains stable, even when raising the time
step to $\Delta t=1.$’. The real CFL condition should therefore be less
draconian than the one generically used. Finding it is the goal of this study.

The generic CFL criterion for hyperbolic equations has been proven not only to
ensure stability, but also strict positivity of the mass distribution (Filbet
& Laurencot, 2004). In a Discontinous Galerkine solver, positivity is instead
enforced with a slope limiter, associated with an SSP integrator. Unlocking
this positivity constrain ensures numerical stability is actually set by the
shortest physical time on which mass transfers through the dust distribution.
A high-order scheme ensures then accuracy even when integration is performed
with large time steps. Since the coagulation flux is expressed as a double
integral over the mass distribution, this time should result from integral
considerations over the mass distribution, an information encoded in the
Laplace transform of the Smoluchowski equation. Looking at the physical
timescales that appear when decomposing the mass distribution into decaying
exponentials reveals an alternate and less stringent CFL condition than the
ones previously used (Sect. 3). We validate these findings by testing this
condition back in the mass space with the solver of Lombart & Laibe (2021). In
this study, we focus on constant and additive kernels, since they can be
associated with analytic solutions that are the most relevant for planet
formation.

##  2 Smoluchowski equation

The Smoluchowski equation is a mean-field, non-linear, integro-differential
equation that model mass conservation along a binary collisional process
(Smoluchowski, 1916). The evolution of the number density of particles per
unit mass $f$ is given by

$\displaystyle\frac{\partial f}{\partial t}=$
$\displaystyle\frac{1}{2}\int_{0}^{x}K\left(y,x-y\right)f\left(y\right)f\left(x-y\right)\mathrm{d}y$
$\displaystyle-f\left(x\right)\int_{0}^{\infty}K\left(y,x\right)f\left(y\right)\mathrm{d}y,$
(1)

where the kernel $K\left(x,y\right)$ is a symmetric function that gives the
collisional rate between particles of mass $x$ and $y$. The conservative form
of Eq. 1 is

$\frac{\partial g}{\partial t}+\frac{\partial F\left[g\right]}{\partial x}=0,$
(2)

where $g\equiv xf$ is the mass density distribution per unit mass, and

$F\left(x\right)=\int_{0}^{x}\int_{x-u}^{\infty}K\left(u,v\right)g\left(u\right)\frac{g\left(v\right)}{v}\mathrm{d}u\mathrm{d}v,$
(3)

is the coagulation flux (Tanaka et al., 1996). The usual CFL condition for
conservative equations of the form Eq. 2 is

$\frac{\Delta t}{\Delta x}\max_{g}\left|\frac{\partial F}{\partial
g}\right|\lesssim 1,$ (4)

For the Smoluchowski equation, the condition given by Eq. 4 may be stringent
when considering local individual contribution to the flux of each mass bin.
Fig. 1 shows indeed that the quantity $\left|\frac{\partial F}{\partial
g}\right|^{-1}$ can become extremely small, since small increments $\delta g$
may become very small at the location of the maximum of $g$, while $\delta F$
remains finite. A physical stability condition should instead consider the
cumulated contributions of every bins to the local flux, accounting for the
contribution of the mass distribution that generates the flux in the mass
space. A natural tool to handle these effects consists of determining a
stability condition for the time step in the dual Laplace space. We therefore
introduce the Laplace transform
$\hat{f}\left(p,t\right)\equiv\int_{0}^{\infty}\mathrm{e}^{-px}f\left(x,t\right)\mathrm{d}x$
of the number density distribution.

Figure 1: Analytic solution for the mass density distribution $g$ and the
coagulation flux $F$ for $K=1$ and $f\left(x,0\right)=\mathrm{e}^{-x}$ at
$t=1$. The maximum of $g$ does not correspond to the maximum of $F$.

##  3 Physical time stepping

###  3.1 Constant kernel

Figure 2: Numerical integration of the Smoluchowski equation with a
discontinuous Galerkine scheme of order 2. Solutions are displayed in log-log
scale for the main plot and lin-log scale when zooming-in close to the
maximum. Grey squares correspond to the generic stability condition Eq. 4
(initially, $\Delta t\simeq 1.10^{-6}$). Green, orange and red dotes
correspond to $\Delta t\simeq 0.8\Delta t_{\rm CFL}$, $\Delta t\simeq
1.5\Delta t_{\rm CFL}$, $\Delta t\simeq 3\Delta t_{\rm CFL}$, where $\Delta
t_{\rm CFL}=M_{0}^{-1}$ from Eq. 10. No oscillations develop when the novel
CFL condition is satisfied (green), numerical integration remain stable. Blue
dashed line: analytic solution.

We first consider the constant kernel $K=1$. Taking the Laplace transform of
Eq. 1 gives

$\partial_{t}\hat{f}+M_{0}\left(t\right)\hat{f}-\frac{1}{2}\hat{f}^{2}=0,$ (5)

where, for the unit kernel, $M_{0}\left(t\right)=\left(1+t/2\right)^{-1}$
(Müller, 1928). We first note that

$0\leq\hat{f}\left(p,t\right)\leq\hat{f}\left(0,t\right)=M_{0}\left(t\right).$
(6)

Eq. 6 shows that the non-linear contribution lightens the linear term in Eq.
5. Discretising Eq. 5 with a forward Euler scheme and performing a linear
stability analysis of the form $\hat{f}^{n}=\hat{f}_{0}^{n}+\epsilon^{n}$
gives

$\frac{\epsilon^{n+1}-\epsilon^{n}}{\Delta
t}+M_{0}^{n}\epsilon^{n}=\hat{f}_{0}^{n}\epsilon^{n},$ (7)

i.e.

$\epsilon^{n+1}=\epsilon^{n}\left[1-\Delta
t\left(M_{0}^{n}-\hat{f}_{0}^{n}\right)\right].$ (8)

Stability of the scheme is obtained at any time from the sufficient condition
$\Delta t\leq\min\left(M_{0}^{n}-\hat{f}_{0}^{n}\right)^{-1}$. A stringent
majorant for $\Delta t$ is therefore

$\Delta t\leq 1/M_{0}^{n}.$ (9)

A stability condition for an explicit scheme is therefore set by the typical
timescale $M_{0}^{-1}$ of the linear term, as one would expect from the
evolution of the moments of the equation (Banasiak et al., 2019). Generally,
the novel stability condition should be weighted by a safety coefficient
$C^{\rm cst}$

$\Delta t\leq\frac{C^{\rm
cst}}{M_{0}^{n}}=\mathcal{O}\left(M_{0}^{-1}\right).$ (10)

This basal Von Neumann analysis is checked by integrating Eq. 5 within the
Laplace space with a forward Euler scheme (Fig. 4). Numerical results are
compared to an analytic solution of Eq. 5 (integrating only with respect to
time in this particuliar case)

$\hat{f}\left(p,t\right)=\frac{2}{2+t},$ (11)

and $C^{\rm cst}=1$. Numerical results are in excellent agreement with the
theoretical analysis. The real test consists of testing the condition Eq. 10
in the mass space. We therefore solve the Smoluchowski equation with the
algorithm of Lombart & Laibe (2021). We use $9$ orders of magnitude in mass
and $n=15$ log-spaced bins to mimic the challenging integration conditions
encountered in practice. We find numerical stability for the same exact
condition (Fig. 2). For $\Delta t\leq\Delta t_{\rm CFL}$, the numerical
integration follows the analytic solution with an unexpected accuracy even
close to marginal stability, confirming the observation of Liu et al. 2019.
When $\Delta t\geq\Delta t_{\rm CFL}$, numerical solution breaks strongly at
small masses. In practice, the constant $C^{\rm cst}$ should be chosen to
provide the desired trade-off between computational efficiency and numerical
accuracy. We verified the criterion on distributions with other values of
$M_{0}$.

###  3.2 Additive kernel

Let us now consider the additive kernel $K=x+y$. For this kernel, $M_{1}$ is
constant and $\frac{\mathrm{d}M_{0}}{\mathrm{d}t}=-M_{1}M_{0}$ (Golovin,
1963), hence

$M_{0}\left(t\right)=M_{0}^{\rm i}\mathrm{e}^{-M_{1}t}.$ (12)

The Laplace transform of Eq. 1 is

$\partial_{t}\hat{f}=\left[M_{0}\left(t\right)-\hat{f}\right]\partial_{p}\hat{f}-M_{1}\hat{f}.$
(13)

Contrary to the constant case, the contribution of the term
$\left[M_{0}\left(t\right)-\hat{f}\right]\partial_{p}\hat{f}$ reinforces the
contribution of the term $-M_{1}\hat{f}$ and contributes to numerical
stability. The strategy of analysis consists now in looking at the
characteristics of the problem, to show the existence of a real number $C>0$
that does not depend on $p$, such that $-C\leq\partial_{t}\ln\hat{f}\leq 0$.
As such, $\hat{f}$ decreases slower that a decaying exponential for which the
stability condition is known. Eq. 13 becomes

$\left(\partial_{t}\hat{f},\partial_{p}\hat{f},-1\right)\cdot\left(1,\hat{f}-M_{0}\left(t\right),-M_{1}\hat{f}\right)=0.$
(14)

Eq. 14 is solved by a method of characteristics by setting
$t=t\left(r,s\right)$, $p=p\left(r,s\right)$,
$u=u\left(r,s\right)\equiv\hat{f}\left(t,p\right)$,
$u\left(t\left(0,s\right),p\left(0,s\right)\right)=\hat{f}_{0}\left(s\right)$,
following Banasiak et al. (2019). One has

$\displaystyle\partial_{r}t$ $\displaystyle=1,$ $\displaystyle
t\left(0,s\right)=0,$ (15) $\displaystyle\partial_{r}p$
$\displaystyle=u-M_{0}\left(r\right),$ $\displaystyle p\left(0,s\right)=s,$
(16) $\displaystyle\partial_{r}u$ $\displaystyle=-M_{1}u,$ $\displaystyle
u\left(0,s\right)=\hat{f}_{0}\left(s\right).$ (17)

Eq. 15 gives $t\left(r,s\right)=r$, Eq. 17 gives
$u\left(r,s\right)=\hat{f}_{0}\left(s\right)\mathrm{e}^{-M_{1}r}$. Integrating
Eq. 12, solving for Eq. 16 gives

$p\left(r,s\right)=s+\frac{\left(\hat{f}_{0}\left(s\right)-M_{0}^{\rm
i}\right)}{M_{1}}\left(1-\mathrm{e}^{-M_{1}r}\right).$ (18)

Consider now $z\left(r,p\right)$ the implicit solution of Eq. 18 where $r$ and
$p$ are seen as two independent variables, i.e.

$p=z\left(r,p\right)+\frac{\left(\hat{f}_{0}\left(z\left(r,p\right)\right)-M_{0}^{\rm
i}\right)}{M_{1}}\left(1-\mathrm{e}^{-M_{1}r}\right).$ (19)

Then,
$\hat{f}\left(t\left(r,s\right),p\left(r,s\right)\right)=u\left(t,s=z\left(t,p\right)\right)$,
and $\hat{f}\left(t,p\right)$ is expressed on the implicit form
$\hat{f}\left(t,p\right)=\hat{f}_{0}\left(z\left(t,p\right)\right)\mathrm{e}^{-M_{1}t}$.
Deriving with respect to time gives

$\partial_{t}\ln\hat{f}=-M_{1}+\partial_{t}z\left(t,p\right)\frac{\hat{f}^{\prime}_{0}\left(z\left(t,p\right)\right)}{\hat{f}_{0}\left(z\left(t,p\right)\right)},$
(20)

where we have denoted for convenience
$\hat{f}^{\prime}_{0}\left(p\right)=\partial_{p}\hat{f}_{0}\left(p\right)$.
Differentiating Eq. 18 with respect to $r$ gives

$\frac{\partial z}{\partial
r}=M_{1}\frac{\left(M_{0}^{i}-\hat{f}_{0}\right)\mathrm{e}^{-M_{1}r}}{M_{1}+\hat{f}^{\prime}_{0}\left(1-\mathrm{e}^{-M_{1}r}\right)}.$
(21)

The identity
$\hat{f}^{\prime}_{0}\left(p\right)=-\int_{0}^{\infty}x\mathrm{e}^{-px}f\left(x\right)\mathrm{d}x$
ensures that

$0\leq\partial_{t}z\leq
M_{1}\frac{M_{0}^{i}-\hat{f}_{0}}{M_{1}+\hat{f}^{\prime}_{0}},$ (22)

and that for any $p$,

$0\geq\partial_{t}\ln\hat{f}\geq-M_{1}\left(1+T\left(p\right)\right).$ (23)

where $T\left(p\right)\geq 0$ is given by

$T\left(p\right)\equiv\frac{\int_{0}^{\infty}x\mathrm{e}^{-px}f_{0}\left(x\right)\mathrm{d}x}{\int_{0}^{\infty}\mathrm{e}^{-px}f_{0}\left(x\right)\mathrm{d}x}\frac{\int_{0}^{\infty}\left(1-\mathrm{e}^{-px}\right)f_{0}\left(x\right)\mathrm{d}x}{\int_{0}^{\infty}\left(1-\mathrm{e}^{-px}\right)xf_{0}\left(x\right)\mathrm{d}x}.$
(24)

Therefore the physical solution decays more slowly than an enveloppe with
exponential decay and is associated with the stability condition

$\Delta t\lesssim\frac{C^{\rm
add}}{M_{1}\left(1+\displaystyle\sup_{p}T\left[f_{0}\right]\right)}.$ (25)

In Appendix A, we prove that $T\left(p\right)\leq 1$, allowing us to write the
condition of Eq. 25

$\Delta t\lesssim\frac{C^{\rm
add}}{2M_{1}}=\mathcal{O}\left(M_{1}^{-1}\right).$ (26)

A refined criterion can be obtained when $T\left[\hat{f}_{0}\right]$ is
actually a decreasing function of $p$. In this case, $T\left(p\right)\leq
T\left(0\right)=\frac{M_{1}^{2}}{M_{0}^{\rm i}M_{2}^{\rm i}}$, which would
provide the refined stability condition

$\Delta t\lesssim\frac{C^{\rm add}}{M_{1}\left(1+\frac{M_{1}^{2}}{M_{0}^{\rm
i}M_{2}^{\rm i}}\right)}.$ (27)

We obtain excellent agreement for the conditions given by Eqs. 25 – 27 in the
Laplace space (Fig. 4), against a numerical solution obtained at high
resolution. Fig. 3 shows very good applicability of this condition in the real
space (varying $M_{1}$ gives similar results). This validates the findings of
Liu et al. (2019). On this example, a factor $\sim 10$ in processing time is
gained with the novel condition.

The term $\sup_{p}T\left[f_{0}\right]$ of Eq. 25 is the mathematical
consequence of the fact that the contribution of
$\left[M_{0}\left(t\right)-\hat{f}\right]\partial_{p}\hat{f}$ reinforces the
the one of $-M_{1}\hat{f}$ in Eq. 13. Finding this correction to be of order
unity is physically consistent with fluxes of mass of similar intensities
generated by the two terms of the right-hand-side of Eq. 1. We note that the
CFL condition comes from the limit $p\to 0$, which corresponds to the limit
case of a constant mass distribution that is non-integrable over the mass
space. Mass fluxes are indeed expected to be more intense for this
distribution, since an additive kernel favour growth over the largest grains.
We conjecture that this CFL condition can, alternatively, be obtained from the
evolution of the moments of Eq. 1. The method presented here can be applied to
other relevant coagulation kernels.

Figure 3: Numerical integration of the Smoluchowski equation with a
discontinuous Galerkine scheme of order 2. Solutions are displayed in log-log
scale for the main plot and lin-log scale when zooming-in close to the
maximum. Grey squares correspond to the generic stability condition Eq. 4
(initially, $\Delta t\simeq 2.10^{-8}$). Green dotes and red dots correspond
to $\Delta t\simeq 0.9\Delta t_{\rm CFL}$, $\Delta t\simeq 1.5\Delta t_{\rm
CFL}$, where $\Delta t_{\rm CFL}=2/3$ from Eq. 27. No oscillations develop
when the novel CFL condition is satisfied (green), numerical integration
remain stable. Blue dashed line: analytic solution.

##  4 Conclusion

In this study, we revisit the derivation of the stability condition for
explicit numerical solvers of the Smoluchowski equation. Generic formulae are
too stringent since they also ensure positivity. Enforcing positivity by some
alternate way (e.g. with a slope limiter associated to an SSP integrator)
allows to improve stability condition by several orders of magnitude. Novel
conditions that involve moments of the mass distribution – $\Delta t_{\rm
CFL}\sim M_{0}^{-1}$ for the constant kernel, $\Delta t_{\rm CFL}\sim
M_{1}^{-1}$ for the additive kernel – are obtained by analysing dual problems
in the Laplace space, to account for the non-locality of the coagulation
equation and its different responses to different dust distributions.
Numerical simulations are in excellent agreement with the theory and validate
our novel CFL condition, confirming the observations of Liu et al. (2019).

## Acknowledgements

GL acknowledges funding from the ERC CoG project PODCAST No 864965. This
project has received funding from the European Union’s Horizon 2020 research
and innovation programme under the Marie Skłodowska-Curie grant agreement No
823823. This project was partly supported by the IDEXLyon project (contract
nANR-16-IDEX-0005) under the auspices University of Lyon. We acknowledge
financial support from the national programs (PNP, PNPS, PCMI) of CNRS/INSU,
CEA, and CNES, France. We thank E. Lynch for useful comments.

## Data availability

The data and supplementary material underlying this article are available in
the repository "growth" on GitHub at https://github.com/mlombart/growth.git.
Figures can be reproduced following the file README.md. The repository
contains data and Python scripts used to generate figures.

## References

  * Banasiak et al. (2019) Banasiak J., Lamb W., Laurencot P., 2019, Analytic Methods for Coagulation-Fragmentation Models, Volume I, doi:10.1201/9781315154428.
  * Birnstiel et al. (2010) Birnstiel T., Dullemond C. P., Brauer F., 2010, A&A, 513, A79
  * Brauer et al. (2008) Brauer F., Dullemond C. P., Henning T., 2008, A&A, 480, 859
  * Chiang & Youdin (2010) Chiang E., Youdin A. N., 2010, Annual Review of Earth and Planetary Sciences, 38, 493
  * Courant et al. (1928) Courant R., Friedrichs K., Lewy H., 1928, Mathematische Annalen, 100, 32
  * Drążkowska et al. (2014) Drążkowska J., Windmark F., Dullemond C. P., 2014, A&A, 567, A38
  * Drążkowska et al. (2019) Drążkowska J., Li S., Birnstiel T., Stammler S. M., Li H., 2019, ApJ, 885, 91
  * Dullemond & Dominik (2005) Dullemond C. P., Dominik C., 2005, A&A, 434, 971
  * Filbet & Laurencot (2004) Filbet F., Laurencot P., 2004, SIAM Journal on Scientific Computing, 25, 2004
  * Forestier-Coste & Mancini (2012) Forestier-Coste L., Mancini S., 2012, SIAM Journal on Scientific Computing, 34, B840
  * Gabriel & Tine (2010) Gabriel P., Tine L. M., 2010, ESAIM: Proc., 30, 53
  * Golovin (1963) Golovin A., 1963, Izv. Geophys. Ser, 5, 482
  * Haworth et al. (2016) Haworth T. J., et al., 2016, PASA, 33, e053
  * Jacobson (2005) Jacobson M. Z., 2005, Fundamentals of Atmospheric Modeling
  * Liu et al. (2019) Liu H., Gröpler R., Warnecke G., 2019, SIAM Journal on Scientific Computing, 41, B448
  * Lombart & Laibe (2021) Lombart M., Laibe G., 2021, MNRAS, 501, 4298
  * Mahoney & Ramkrishna (2002) Mahoney A. W., Ramkrishna D., 2002, Chemical Engineering Science, 57, 1107
  * Müller (1928) Müller H., 1928, Fortschrittsberichte über Kolloide und Polymere, 27, 223
  * Sandu (2006) Sandu A., 2006, Aerosol Science Technology, 40, 261
  * Smoluchowski (1916) Smoluchowski M. V., 1916, Zeitschrift fur Physik, 17, 557
  * Tanaka et al. (1996) Tanaka H., Inaba S., Nakazawa K., 1996, Icarus, 123, 450
  * Testi et al. (2014) Testi L., et al., 2014, Protostars and Planets VI, p. 339
  * Yang & Tian (2017) Yang Z.-H., Tian J., 2017, Journal of Inequalities and Applications, 2017

##  Appendix A Bounding of $T$

Following Yang & Tian (2017), let denote

$T\left(p\right)\equiv\underbrace{\frac{\int_{0}^{\infty}x\mathrm{e}^{-px}f_{0}\left(x\right)\mathrm{d}x}{\int_{0}^{\infty}\mathrm{e}^{-px}f_{0}\left(x\right)\mathrm{d}x}}_{T_{1}\equiv
I_{1}/I_{2}}\underbrace{\frac{\int_{0}^{\infty}\left(1-\mathrm{e}^{-px}\right)f_{0}\left(x\right)\mathrm{d}x}{\int_{0}^{\infty}\left(1-\mathrm{e}^{-px}\right)xf_{0}\left(x\right)\mathrm{d}x}}_{T_{2}\equiv
I_{3}/I_{4}}.$ (28)

Deriving $T_{1}$ with respect to $p$ and symmetrising $x\leftrightarrow y$
gives

$\displaystyle
I_{2}\left(p\right)^{2}\frac{\mathrm{d}T_{1}\left[f\right]\left(p\right)}{\mathrm{d}p}$
$\displaystyle\equiv-\left\\{\left(\int_{0}^{\infty}\mathrm{e}^{-px}f\left(x\right)\mathrm{d}x\right)\left(\int_{0}^{\infty}\mathrm{e}^{-px}x^{2}f\left(x\right)\mathrm{d}x\right)-\left(\int_{0}^{\infty}\mathrm{e}^{-px}xf\left(x\right)\mathrm{d}x\right)^{2}\right\\},$
(29)
$\displaystyle=-\left\\{\int_{0}^{\infty}\\!\\!\int_{0}^{\infty}\mathrm{d}x\mathrm{d}y\,\mathrm{e}^{-px}\mathrm{e}^{-py}f\left(x\right)\\!f\left(y\right)y^{2}-\int_{0}^{\infty}\\!\\!\int_{0}^{\infty}\mathrm{d}x\mathrm{d}y\,\mathrm{e}^{-px}\mathrm{e}^{-py}f\left(x\right)\\!f\left(y\right)xy,\right\\}$
(30)
$\displaystyle=-\left\\{\int_{0}^{\infty}\\!\\!\int_{0}^{\infty}\mathrm{d}x\mathrm{d}y\,\mathrm{e}^{-px}\mathrm{e}^{-py}f\left(x\right)\\!f\left(y\right)\frac{\left(x^{2}+y^{2}\right)}{2}-\int_{0}^{\infty}\\!\\!\int_{0}^{\infty}\mathrm{d}x\mathrm{d}y\,\mathrm{e}^{-px}\mathrm{e}^{-py}f\left(x\right)\\!f\left(y\right)xy\right\\},$
(31)
$\displaystyle=-\frac{1}{2}\int_{0}^{\infty}\\!\\!\int_{0}^{\infty}\mathrm{d}x\mathrm{d}y\,\mathrm{e}^{-px}\mathrm{e}^{-py}f\left(x\right)\\!f\left(y\right)\left(x-y\right)^{2}<0.$
(32)

Hence, $T_{1}\left[f\right]$ is a decreasing function for any $f$. As such,

$T_{1}\left[f_{0}\right]\left(p\right)\leq
T_{1}\left[f_{0}\right]\left(0\right)=\frac{M_{1}}{M_{0}}.$ (33)

Similarly, deriving $T_{2}$ with respect to $p$ gives

$\displaystyle
I_{4}\left(p\right)^{2}\frac{\mathrm{d}T_{2}\left[f\right]\left(p\right)}{\mathrm{d}p}$
$\displaystyle\equiv\left\\{\left(\int_{0}^{\infty}\left(1-\mathrm{e}^{-px}\right)xf\left(x\right)\mathrm{d}x\right)\left(\int_{0}^{\infty}x\mathrm{e}^{-px}f\left(x\right)\mathrm{d}x\right)\right.$
$\displaystyle\left.\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,-\left(\int_{0}^{\infty}x^{2}\mathrm{e}^{-px}f\left(x\right)\mathrm{d}x\right)\left(\int_{0}^{\infty}\left(1-\mathrm{e}^{-px}\right)f\left(x\right)\mathrm{d}x\right)\right\\}$
(34)
$\displaystyle=\left\\{\int_{0}^{\infty}\\!\\!\int_{0}^{\infty}\mathrm{d}x\mathrm{d}y\,f\left(x\right)\\!f\left(y\right)xy\left(1-\mathrm{e}^{-px}\right)\mathrm{e}^{-py}-\int_{0}^{\infty}\\!\\!\int_{0}^{\infty}\mathrm{d}x\mathrm{d}y\,f\left(x\right)\\!f\left(y\right)x^{2}\mathrm{e}^{-px}\left(1-\mathrm{e}^{-py}\right)\right\\},$
(35)
$\displaystyle=\frac{1}{2}\left\\{\int_{0}^{\infty}\\!\\!\int_{0}^{\infty}\mathrm{d}x\mathrm{d}y\,f\left(x\right)\\!f\left(y\right)\underbrace{\left(y-x\right)\left[x\mathrm{e}^{-px}\left(1-\mathrm{e}^{-py}\right)-y\mathrm{e}^{-py}\left(1-\mathrm{e}^{-px}\right)\right]}_{\geq
0}\right\\}.$ (36)

Hence, $T_{2}\left[f\right]$ is strictly increasing function for any $f$,

$T_{2}\left[f_{0}\right]\left(p\right)\leq
T_{2}\left[f_{0}\right]\left(\infty\right)=\frac{M_{0}}{M_{1}}.$ (37)

Finally, $T\left(p\right)=T_{1}\left(p\right)T_{2}\left(p\right)\leq 1$.

##  Appendix B Stability condition in the Laplace space

Figure 4: Left: Numerical solution of Eq. 5 obtained with a first order Euler
scheme. Small grey, large green and large red dots correspond to $\Delta
t=10^{-3}\Delta t_{\rm CFL}$, $\Delta t=1.\Delta t_{\rm CFL}$ and $\Delta
t=1.25\Delta t_{\rm CFL}$, where $\Delta t_{\rm CFL}$ is given by Eq. 25
($C^{\rm cst}=1$). $\Delta t$ varies with time. Black solid line: analytic
solution. Right: Numerical solution of Eq. 13 obtained with a first order
upwind scheme, under the condition $\hat{f}\left(0,t\right)=1$ with 20 grid
points. Grey, large green and large red dots correspond to $\Delta t\simeq
0.09\Delta t_{\rm CFL}$, $\Delta t\simeq 0.83\Delta t_{\rm CFL}$ and $\Delta
t\simeq 1.07\Delta t_{\rm CFL}$, where $\Delta t_{\rm CFL}=2/3$ from Eq. 27
($C^{\rm add}=1$). Black solid line: analytic solution (a better sampling in
mass makes the numerical solution closer to the analytic solution).

Fig. 4 show the validity of the stability conditions obtained for the
numerical integration of $\hat{f}$ (Laplace space) in Sect. 3 .


# JUNO: Jump-Start Reinforcement Learning-based Node Selection for UWB Indoor
Localization
††thanks: This Project was partially supported by Department of National
Defence’s Innovation for Defence Excellence & Security (IDEaS), Canada.

1sh Zohreh Hajiakhondi-Meybodi  Electrical & Computer Engineering,
Concordia University,
Montreal, Canada
<EMAIL_ADDRESS>2nd Ming Hou  Defence Research and Development
Canada (DRDC),
Toronto, Canada
<EMAIL_ADDRESS>3th Arash Mohammadi  Concordia Institute for Inf.
Systems Eng.,
Concordia University,
Montreal, Canada
<EMAIL_ADDRESS>
###### Abstract

Ultra-Wideband (UWB) is one of the key technologies empowering the Internet of
Thing (IoT) concept to perform reliable, energy-efficient, and highly accurate
monitoring, screening, and localization in indoor environments. Performance of
UWB-based localization systems, however, can significantly degrade because of
Non Line of Sight (NLoS) connections between a mobile user and UWB beacons. To
mitigate the destructive effects of NLoS connections, we target development of
a Reinforcement Learning (RL) anchor selection framework that can efficiently
cope with the dynamic nature of indoor environments. Existing RL models in
this context, however, lack the ability to generalize well to be used in a new
setting. Moreover, it takes a long time for the conventional RL models to
reach the optimal policy. To tackle these challenges, we propose the Jump-
start RL-based Uwb NOde selection (JUNO) framework, which performs real-time
location predictions without relying on complex NLoS identification/mitigation
methods. The effectiveness of the proposed JUNO framework is evaluated in term
of the location error, where the mobile user moves randomly through an ultra-
dense indoor environment with a high chance of establishing NLoS connections.
Simulation results corroborate the effectiveness of the proposed framework in
comparison to its state-of-the-art counterparts.

###### Index Terms:

Indoor Localization, Internet of Things, Anchor Selection, Jump-Start
Reinforcement Learning, Ultra-Wideband (UWB).

##  I Introduction

Ultra-WideBand (UWB) technology has been emerged as a solution to meet the
phenomenal growth of the need for localizing users in indoor environments [1].
The use of a wide radio spectrum in UWB technologies enables individual multi-
path components of the received signal to be efficiently resolved, resulting
in high accuracy indoor positioning [2]. To monitor/track users in indoor
environments, several localization techniques have been proposed such as
Received Signal Strength Indicator (RSSI) [3, 4], Angle of Arrival (AoA) [5,
6, 7], and Time Difference of Arrival (TDoA) [8, 9], among which time-based
solutions [8, 9, 10] can be considered as more efficient ones. In such
scenarios, the time of the received signal from a set of available UWB beacons
is required, which can only be estimated accurately if the first arrival path
has been properly identified. In this context, one key challenge is
susceptibility to the Non-Line-of-Sight (NLoS) error. In presence of an
obstacle between the UWB beacon and the mobile device, the time of the
received signal will be delayed, resulting in a positive bias and a
significant degradation in the positioning accuracy. Therefore, NLoS
mitigation/identification in UWB-based indoor localization is of paramount
importance. Conventional indoor localization frameworks reduced such location
error via parametric solutions [11, 12], the accuracy of which is dependent on
implementation of complex pre-processing techniques adding considerable
latency. Furthermore, using a large number of UWB beacons for localizing users
is inefficient from the energy consumption perspective. In this regard, Dai et
al. [13] analytically proved that tracking users’ locations through a subset
of active beacons offers several benefits, including mitigating the energy
consumption of beacons, and improving the location accuracy. Consequently, the
main focus of recent researches [14, 15, 16, 17] has been shifted to use
anchor node selection to achieve the best localization performance in terms of
location accuracy and resource management. The paper aims to further advance
this emerging field.

Related Work: Anchor node selection in the context of indoor localization is
utilized to improve the network’s performance by setting a set of criteria for
selecting a subset of beacons with the highest utilities. One of the most
important criteria in indoor localization is to mitigate the location error,
caused by NLoS connections. Towards this goal, LoS connections will be
selected for monitoring/tracking users’ locations instead of extracting
location information from all available beacons. Generally speaking, anchor
selection frameworks can be classified into two groups, i.e., analytical
solutions and Artificial Intelligence (AI) models. Conventionally, the focus
of anchor selection frameworks was on the former category, i.e., deriving
fixed mathematical/optimization models [20, 15, 17] to meet the acceptable
location accuracy.To alleviate the high complexity caused by using complicated
mathematical formulations, recent research works [21, 22, 24, 23] used AI and
Machine Learning (ML) models to localize/navigate mobile devices in indoor
environments. One of the most commonly used AI-based Los/NLoS identification
approaches is supervised learning models [21, 22]. For instance, Poulose et
al. [22] applied Long Short-Term Memory (LSTM) network to predict the location
of mobile devices using the Time of Arrival (ToA)-based UWB method. Despite
all the benefits that come from using supervised models, there are several key
challenges ahead. On the one hand, supervised models require labelled LoS/NLoS
data, which is both costly and time-consuming, limiting general applicability
of supervised models within this context. On the other hand, even minor
changes in the indoor environment would require updating the training dataset.
Unsupervised learning models [24], however, eliminate the necessity for
labelling of channel conditions, therefore, allowing generalization while
saving time and precious resources. Such models, however, are impractical due
to the dynamic nature of indoor environments, such as unknown/varying number
of users and adverse environmental conditions. Furthermore, energy consumption
efficiency is compromised in scenarios where all the beacons require to
transmit/receive signals for LoS/NLoS identification.

To tackle the above mentioned issues, the main focus of researchers has been
shifted to use Reinforcement Learning (RL) models [25, 26, 27, 28] for indoor
localization/navigation application. In our previous work [25], for instance,
the mobile user is autonomously trained via an RL model to be localized by a
set of UWB beacons with LoS connections. The main objective of the RL model
within the indoor localization domain is to learn an optimal or near-optimal
policy that maximizes the location accuracy. One of the most important
challenges of existing RL models [25], however, is that the optimal policy
should be learned by the interaction of the agent (mobile user) with the
environment (i.e., via trial and error), without any prior information,
especially when the model is just initialized. Consequently, it may take a
long time for the RL model to reach the optimal policy. Another challenge is
the generalization ability of the pre-trained RL model to be used in a new and
different environment, where the density/location of obstacles is changing
over the time/environment. To tackle these issues, Uchendu et al. [29]
proposed the Jump-Start RL (JSRL) model, where the agent use a guide-policy
instead of a random one at the beginning of the learning process.
Consequently, the learning process is accelerated and the RL generalization
ability is highly improved.

Contribution: Motivated by the above discussion, we introduce the Jump-start
RL-based Uwb NOde selection (JUNO) framework with the application to indoor
localization. The main novelty of this work is the design of an autonomous and
real-time anchor node selection, where the key objective is to accelerate the
location accuracy improvement. Towards this goal, a combination of the guide
and exploration policies is used, where the guide-policy significantly speeds
up the early learning phase of the RL model to converge to the optimal
location accuracy. Furthermore, since any random guide-policy can be used in
the JUNO framework, the generalization ability of the pre-trained RL
frameworks improves. Simulation results illustrate that the proposed JUNO
framework outperforms its state-of-the-art counterparts in terms of the
cumulative rewards and location error even in an ultra-dense indoor
environment.

The rest of this paper is organized as follows: In Section II, the system
model is provided. Section III introduces the proposed JUNO framework. Section
IV presents experimental results. Finally, Section V concludes the paper.

##  II System Model

In this section, we first introduce the UWB wireless signal model and the TDoA
localization formulation. Then, we present the required background on the RL
model.

###  II-A UWB Wireless Signal Model

In this study, we consider a multi-user indoor environment (e.g., an office or
a hotel building), consisting of $N$ synchronized UWB beacons, denoted by
$UWB_{i}$, $i=1,\ldots,N$, and several mobile users randomly moving through
the environment. To support multiple users, we use the Time Hopping (TH)
technique as one of the efficient Code Division Multiple Access (CDMA)
schemes, where different codes are assigned to distinct users. Given the Pulse
Amplitude Modulation (PAM) modulation, the transmitted signal $s_{u}$ from
user $u$ is obtained as

$s_{u}(t)=\sum\limits_{n=-\infty}^{\infty}p_{u}(n)\sum\limits_{s=0}^{N_{c}-1}c_{u}(s)w(t-nT_{s}-sT_{c}-\theta_{u})$
(1)

where $T_{s}$, and $N_{c}$ represent the symbol time, and the number of chips
with duration of $T_{c}$ in each symbol, respectively. Term
$c_{u}(s)\in\\{0,1\\}$ denotes the access code associated with the mobile user
$u$, and $p_{u}(n)\in\\{-1,1\\}$ is the Independent and Identically
Distributed (IID) information symbols. Furthermore, term $\theta_{u}$ is the
time asynchronism with uniform distribution $[0,T_{s}]$, and $w(t)$ is the
normalized impulse signal. The received signal $r_{i}(t)$ by $UWB_{i}$ is
expressed as

$r_{i}(t)=\sum\limits_{u=1}^{N_{u}}\sqrt{P_{u}}\sum_{k=1}^{N(t)}\rho_{u,k}(t,\tau)s_{u}(t-\tau_{u,k}(t)-\tau_{i})+n(t),$
(2)

where $\tau_{u,k}(t)$ and
$\rho_{u,k}(t,\tau)=\beta_{u,k}(t,\tau)\exp{(j\Phi_{u,k}(t,\tau))}$ represent
the phase delay and the attenuation associated with $k^{\text{th}}$ path,
where the total number of detachable paths is denoted by $N(t)$, and $N_{u}$
is the number of users in the experimental indoor environment. Terms
$\beta_{u,k}(t,\tau)$ and $\Phi_{u,k}(t,\tau)$ are the amplitude and phase of
$k^{\text{th}}$ path, respectively. Term $\beta_{u,k}(t,\tau)$ is a
Nakagami-$m$ random variable that reflects the LoS/NLoS link condition, with
$m=1$ representing the Rayleigh fading and $m>1$ illustrating the Rician
channel model. Term $n(t)\sim\mathcal{N}(0,\sigma^{2})$ is the Additive White
Gaussian Noise (AWGN) channel. Moreover, term $\tau_{i}=d_{i}/c$ is the time
of arrival of signal $s_{u}(t)$, which is equivalent to the delay of the first
path within the LoS condition. Finally, terms $d_{i}$ and $c=3\times 10^{8}$
$m/s$ represent the distance between the user $u$ and $UWB_{i}$, and the speed
of light, respectively. Given the first peak of the estimated Channel Impulse
Response (CIR) of at least two UWB beacons $UWB_{i}$ and $UWB_{j}$, denoted by
$\tau_{i}$ and $\tau_{j}$, the mobile user’s location at time slot $t$,
denoted by ($x_{t},y_{t}$), is calculated as follows

$\displaystyle~{}~{}~{}~{}~{}\tau_{i}-\tau_{j}=$ (3)
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\frac{\sqrt{\big{(}x_{t}-x_{i}\big{)}^{2}+\big{(}y_{t}-y_{i}\big{)}^{2}}-\sqrt{\big{(}x_{t}-x_{j}\big{)}^{2}+\big{(}y_{t}-y_{j}\big{)}^{2}}}{c},$

where ($x_{i},y_{i}$) and ($x_{j},y_{j}$) denote the locations of $UWB_{i}$
and $UWB_{j}$, respectively. This completes presentation of the UWB wireless
signal model, next, we introduce the RL background.

###  II-B RL Background

RL model is an area of learning paradigm, where an agent, interacting with the
environment, makes a sequence of decisions to achieve the maximum accumulated
rewards. Markov Decision Process (MDP) is used to mathematically describe an
RL environment, which includes a set of states $\mathcal{S}$, a set of actions
$\mathcal{A}$, a reward function $\mathcal{R}$, and a transition function
$\mathcal{T}$. Based on the current state $s_{t}\in\mathcal{S}$ at time slot
$t$, the agent takes an action $a_{t}\in\mathcal{A}$, resulting in a new state
$s_{t+1}\in\mathcal{S}$ at time slot $t+1$, which is shown by the transition
function $\mathcal{T}(s_{t},a_{t},s_{t+1})$. The optimum policy $\pi^{*}$,
which leads the agent to the maximum accumulated rewards is given by

$\displaystyle\pi^{*}=\arg\max\limits_{\begin{subarray}{c}\pi\end{subarray}}\mathbb{E}_{\pi}\Big{\\{}\sum\limits_{t=0}^{H-1}\gamma^{t}r_{t+1}|s_{0}=s\Big{\\}},$
(4)

where $\gamma\in[0,1]$ is the discount factor and $H$ is the total number of
steps in one episode. To learn the optimal action, the agent receives a
feedback after taking action $a_{t}$, known as the reward function
$r_{t}=\mathcal{R}(s_{t},a_{t})$. Q-learning is a value-based RL framework,
where the Q-value associated with the action $a_{t}$ and the state $s_{t}$ at
time slot $t$, denoted by $Q(s_{t},a_{t})$, is expressed as

$Q(s_{t},a_{t})=\mathbb{E}_{\pi}\Big{\\{}\sum\limits_{t=0}^{H-1}\gamma^{t}r_{t+1}|s_{0}=s,a_{0}=a,a_{t}=\pi({s_{t}})\Big{\\}}.$
(5)

where $\pi$ is the policy that leads to taking an action in a given state. The
updated Q-value at each time slot is calculated as

$Q(s_{t},a_{t})\leftarrow(1-\alpha)Q(s_{t},a_{t})+\lambda(r_{t}+\alpha\max
Q(s_{t+1},a_{t+1})),$ (6)

where $\alpha\in[0,1]$ represents the learning rate.

##  III The Proposed JUNO Framework

Figure 1: The block diagram of the proposed JUNO anchor selection framework.

In this section, we first introduce the JSRL model, and then present details
of the proposed JUNO framework.

###  III-A JSRL model

Conventionally, an RL-based agent selects a random action $a_{t}$ at the
beginning of the learning process, where there is no prior information
resulting in a long time to reach the optimal policy $\pi$. The main
difference between the conventional RL and JSRL model [29] is that the agent
in the JSRL framework has access to two policies, called guide-policy
$\pi^{g}(a|s)$, and the exploration policy $\pi^{e}(a|s)$. While the
exploration policy is the same policy used in conventional RL models, which
will be updated during the training, the guide-policy is a fixed prior policy,
learned via an RL model or manually/randomly constructed. Since the Q-table is
initialized with zeros, the agent of the JSRL model follows $\pi^{g}(a|s)$
instead of randomly selecting an action $a_{t}$ based on the untrained policy
$\pi^{e}(a|s)$. Considering the fact that the distribution of the data under
policy $\pi^{g}(a|s)$ is not exactly the same as the policy $\pi^{e}(a|s)$,
the state space of the policy $\pi^{e}(a|s)$ would be different. Therefore, it
is essential to gradually transit the data collection from the policy
$\pi^{g}(a|s)$ to policy $\pi^{e}(a|s)$. Consequently, given an RL model with
horizon $H$, we define guide-step $h\leq H$ as the number of steps where the
agent uses policy $\pi^{g}(a|s)$, initialized with $H$ and gradually decreases
over the course of training. More precisely, the action is selected based on
$\pi^{g}(a|s)$ for $h$ steps at the initial stage of each training episode,
continuing with $\pi^{e}(a|s)$ for remaining $(H-h)$ steps.

###  III-B JUNO Anchor Selection

Following Reference [29], we use the JSRL model in our proposed JUNO framework
to accelerate the learning process. Due to the dynamic nature of indoor
venues, caused by varying environmental conditions, the proposed JUNO
framework seeks to train the mobile user to autonomously find the optimal LoS
connections at any given time and place. The proposed JUNO framework consists
of the following main components:

(i) Agent: In the proposed JUNO framework, the mobile user operates as the
agent, interacting with the environment via a set of actions.

(ii) State-Space: The state $\mathbf{s}_{t}\in\mathcal{S}$ is defined as the
user’s location $(x_{t},y_{t})$ at time slot $t$. Following Reference [28], we
discretize the indoor environment into $N_{l}=N_{x}\times N_{y}$ points, where
$x_{t}$ and $y_{t}$ are obtained as

$\displaystyle x_{t}$ $\displaystyle=$ $\displaystyle
x_{t-1}+\zeta_{x},~{}~{}~{}~{}~{}0\leq x_{t}\leq N_{x}$ (7)
$\displaystyle\text{and }y_{t}$ $\displaystyle=$ $\displaystyle
y_{t-1}+\zeta_{y},~{}~{}~{}~{}~{}0\leq y_{t}\leq N_{y}.$ (8)

where $\zeta_{x},\zeta_{y}\in\\{-1,0,1\\}$ are random numbers, indicating the
user’s movement in $x$-axis and $y$-axis [28], respectively. It should be
noted that although RL models with a continuous and high-dimensional state-
space [30] provide higher resolution and precise localization, they suffer
from high complexity, making them inefficient for real-time applications
requiring low latencies. The benefits of the discrete RL models in the context
of indoor localization [28, 27], therefore, make us to choose discrete RL over
its continuous counterpart.

(iii) Action-Space: The action space is defined as a set of nearby UWB
beacons, where the user’s location can be determined by extracting the time
information from the received signal of the corresponding beacons. The
cardinality of the action space, denote by $N_{a}$, is given by

$N_{a}=\frac{N_{u}!}{(N_{u}-N_{r})!\enspace N_{r}!},$ (9)

where $N_{r}$ represents the number of required beacons for localization,
depending on some parameters, such as the indoor localization framework (i.e.,
ToA, TDoA, and Two Way Ranging (TWR)), and the dimension of the experimental
environment, i.e., 2D or 3D area. Considering the fact that at least two UWB
beacons are required for the TDoA-based localization scheme in a 2D
environment, the selected action is a vector, denoted by
$\mathbf{a}=[a_{i},a_{j}]$, where $a_{i}$, $a_{j}$ represent $UWB_{i}$ and
$UWB_{j}$, respectively.

(iv) Reward: As stated previously, the main objective of the proposed JUNO
framework is to minimize the location error caused by UWB beacons with NLoS
connections. Therefore, after taking action $\mathbf{a}_{t}$, the estimated
user’s location $(x_{es,t}^{(i,j)},y_{es,t}^{(i,j)})$ is calculated, where the
superscript $(i,j)$ indicates that the estimated location is obtained by the
received signals of $UWB_{i}$ and $UWB_{j}$. Considering the fact that even if
one of these two connections is NLoS, we will face with a remarkable location
error, the combination of $UWB_{i}$ and $UWB_{j}$ at any location/time is of
paramount importance. For this reason, the reward function
$\mathcal{R}(s_{t},a_{t})$ is defined as

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\mathcal{R}(s_{t},a_{t})$
$\displaystyle\\!\\!\\!\\!=\\!\\!\\!\\!\\!$
$\displaystyle\left\\{\begin{array}[]{ll}\dfrac{1}{\mathcal{E}_{t}},~{}~{}~{}~{}~{}~{}~{}~{}\mbox{$\mathcal{E}_{t}\leq\mathcal{E}_{th},$}\\\
-\mathcal{E}_{t},~{}~{}~{}~{}~{}~{}\mbox{o.w.}\end{array}\right.,$ (12)

where $\mathcal{E}_{th}$ is a pre-defined threshold value for the maximum
acceptable location error [27], and $\mathcal{E}_{t}$ denotes the location
error at time slot $t$, calculated as

$\mathcal{E}_{t}=\sqrt{(x_{t}-x_{es,t}^{(i,j)})^{2}+(y_{t}-y_{es,t}^{(i,j)})^{2}}.$
(13)

This completes presentation of the proposed JUMP anchor selection framework,
next, we will describe our testbed and simulation results.

Figure 2: Investigating the effect of (a) Guide-policy on the location error;
(b) Learning rate $\alpha$ on the cumulative rewards, and; (c) Learning rate
$\alpha$ on the location error.

Figure 3: Investigating the effect of (a) $\epsilon$ on the location error;
(b) $h_{max}$ on the location error, and; (c) ECDF on the location error.

Figure 4: Ground truth and estimated random trajectories by using: (a) Nearest
neighbor; (b) Random, and; (c) JUNO frameworks.

##  IV Simulation Results

To evaluate the performance of the proposed JUNO framework, we consider an
experimental indoor area such as an office building with the size of
($60\times 50$) $m^{2}$, which is compromised of several non-overlapping sub-
areas [20, 28]. Following Reference [28], each sub-area is discretized into
several square zones, where the dimension of each zone is ($1\times 1$)
$m^{2}$. Although the location resolution, i.e., the number of discretized
points in the indoor environment $N_{l}$, is proportional to the location
accuracy, it also results in higher state-space, complexity, and the respond-
time. Therefore, there should be a trade-off between the location resolution
and the respond-time of the learning model. Despite the recent RL-based
localization works [28, 27], where the environment is divided into a grid of
$(5\times 5)$ $m^{2}$ and $(3\times 3)$ $m^{2}$ cells, respectively, we assume
higher resolution of $(1\times 1)$ $m^{2}$ to improve the location accuracy.
Mobile users are randomly moving through the network in $8$ directions based
on Eqs. (7) and (8), where it is assumed that mobile users are placed at
zones’ center [28]. At each location, the transmitted signal by the mobile
user are received by a set of nearby UWB beacons. Due to the obstacles in the
environment, the received signal would be LoS or NLoS connections, where it is
assumed that the channel condition of $UWB_{i}$, for ($1\leq i\leq N_{u}$), is
determined randomly at each zone to initialize the environment.

Fig. 2(a) illustrates the effect of the guide-policy on the proposed JUNO
framework. As shown in Fig. 2(a), using a random Q-table or the one obtained
by an RL model as the guide-policy outperforms the conventional RL approach,
accelerating the learning process of the JUNO framework, improving the
location accuracy, and increasing the RL’s generalization capabilities. We
also investigate the effect of learning rate $\alpha$ on the proposed JUNO
framework to obtain the best value of $\alpha$. As shown in Figs. 2(b)-(c),
increasing the number of epochs decreases the location error and increases the
cumulative rewards, illustrating that the model is well-trained. Moreover, it
is evident that the learning rate has not a great impact on the location
accuracy.

Fig. 3(a) shows the effect of $\epsilon$ on the JUNO framework, as a parameter
to maintain a trade-off between exploration and exploitation, where the random
action $a_{t}$ is chosen with the probability of $\epsilon$. Note that
$\epsilon=[1,0.1]$ means that $\epsilon$ is initialized with $1$, gradually
decreasing with time by
$\Delta\epsilon=\dfrac{\epsilon_{max}-\epsilon_{min}}{N_{epoch}}$ to $0.1$,
with $N_{epoch}=100$, which is the best strategy according to the results of
Fig. 3(a). Moreover, Fig. 3(b) illustrates the maximum step $h_{max}$ that the
guide-policy is initially used, where $h_{max}=0$ represents the conventional
RL model with no guide-policy. As shown in Fig. 3(b), larger $h_{max}$ results
in the lower location error.

To illustrate the effectiveness of the proposed framework, we compare it with
four baseline models: (i) Weighted Least Square (WLS) anchor selection [15];
(ii) Geometric Dilution of Precision (GDOP) anchor selection [17]; (iii)
Nearest Neighbor Node Selection (NN-NS), where the mobile user is localized by
$N_{r}$ number of nearest beacons, and; (iv) Random Node Selection (RNS),
where a set of UWB beacons are randomly selected for localization. Fig. 3(c)
compares the Empirical Cumulative Distribution Function (ECDF) of different
frameworks. According to the results shown in Fig. 3(c), the location error
caused by the proposed JUNO framework is considerably lower than that of its
counterparts. Finally, in Fig. 4 we consider a random trajectory in a
$(24\times 15)$ $m^{2}$ rectangular indoor environment, and compare the ground
truth with the estimated trajectories by our proposed JUNO framework and two
other baselines. As shown in Fig. 4, the estimated path by our proposed
framework in the most points closely follows that of the ground truth.

Finally, we use the Root Mean Squared Error (RMSE), which is a generally used
performance metric within the localization domain, calculated as follows

$RMSE=\sqrt{\dfrac{\sum\limits_{t=1}^{N}(L_{t}-L_{est,t})^{2}}{N}},$ (14)

where $L_{t}=(x_{t},y_{t})$ and $L_{est,t}=(x_{es,t},y_{es,t})$ represent the
exact and the estimated location of the user at time $t$, and $N$ denotes the
total number of steps once the proposed JUNO framework reaches the steady
state. The proposed JUNO framework achieves the localization error of $0.32$
m, while the CNN [21] and LSTM-based [22] localization frameworks track a
mobile user with $0.58$, $0.48$ location errors, respectively.

##  V Conclusion

In this paper, we presented the Jump-start RL-based Uwb NOde selection (JUNO)
framework with application to indoor localization. The key objective was to
overcome several challenges in existing UWB-based anchor selection frameworks,
such as not adapting with time-varying environmental conditions, lack of
generalization, and taking long times to reach the optimal policy. Using the
proposed JUNO framework, the learning process is accelerated, where the mobile
user is autonomously trained to identify a set of UWB beacons with LoS
connections to be localized based on the 2-D TDoA technique. The effectiveness
of the proposed framework is evaluated in terms of the location error and
cumulative rewards. Simulation results illustrated that the proposed JUNO
framework reduced the location error. According to the simulation results, the
proposed framework outperformed its counterparts in tracking a mobile user
moving in a random trajectory. With the emphasis on non-uniform distribution
of UWB beacons, our future research direction is to propose an adaptive
version of the JUNO framework, where the cardinality of the action space can
be adjusted during the learning process.

## References

  * [1] R. Huang, J. Tao, L. Yang, Y. Xue and Q. Wu, “Robust TDoA Indoor Tracking Using Constrained Measurement Filtering and Grid-Based Filtering,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 4895-4899.
  * [2] F. Zafari, A. Gkelias and K. K. Leung, “A Survey of Indoor Localization Systems and Technologies,” IEEE Communications Surveys & Tutorials, vol. 21, no. 3, pp. 2568-2599, thirdquarter 2019.
  * [3] M. Atashi, M. S. Beni, P. Malekzadeh, Z. HajiAkhondi-Meybodi, K. N. Plataniotis, and A. Mohammadi, “Orientation-Matched Multiple Modeling for RSSI-based Indoor Localization via BLE Sensors,”  European Signal Processing Conference (EUSIPCO), 2020.
  * [4] M. Salimibeni, Z. HajiAkhondi-Meybodi, M. Atashi, P. Malekzadeh, K. N. Plataniotis, and A. Mohammadi, “IoT-TD: IoT Dataset for Multiple Model BLE-based Indoor Localization/Tracking,”  European Signal Processing Conference, 2020.
  * [5] Z. HajiAkhondi-Meybodi, M. S. Beni, K. N. Plataniotis, and A. Mohammadi  “Bluetooth Low Energy-based Angle of Arrival Estimation via Switch Antenna Array for Indoor Localization,”  International Conference on Information Fusion, July 2020.
  * [6] Z. HajiAkhondi-Meybodi, M. S. Beni, A. Mohammadi, and K. N. Plataniotis,  “Bluetooth Low Energy-based Angle of Arrival Estimation in Presence of Rayleigh Fading,”  IEEE International Conference on Systems, Man, and Sybernetics, 2020.
  * [7] Z. HajiAkhondi-Meybodi, M. Salimibeni, A. Mohammadi and K. N. Plataniotis,  “Bluetooth Low Energy and CNN-Based Angle of Arrival Localization in Presence of Rayleigh Fading,”  International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 7913-7917.
  * [8] M. Khalaf-Allah,  “Particle Filtering for Three-Dimensional TDoA-Based Positioning Using Four Anchor Nodes,”  Sensors, vol. 20, pp. 4516–4542, Jan. 2020.
  * [9] R. Huang, L. Yang, J. Tao and Y. Xue,  “KLD Minimization-Based Constrained Measurement Filtering For Two-Step TDoA Indoor Tracking,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)”, 2021, pp. 4635-4639.
  * [10] S. Zhao, X. P. Zhang, X. Cui and M. Lu,  “Optimal TOA Localization for Moving Sensor in Asymmetric Network,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 7928-7932.
  * [11] L. Zhang, C. Wang, M. Ma and D. Zhang,  “WiDIGR: Direction-Independent Gait Recognition System Using Commercial Wi-Fi Devices,”  IEEE Internet of Things Journal, vol. 7, no. 2, pp. 1178–1191, Feb. 2020.
  * [12] J. Zhang, J. Salmi, E. S. Lohan,  “Analysis of Kurtosis-based LOS/NLOS Identification using Indoor MIMO Channel Measurement,”  IEEE Trans. on Vehicular Technology, vol. 62, no. 6, pp. 2871–2874, July 2013.
  * [13] W. Dai, Y. Shen, and M. Z. Win,  “Sparsity-Inspired Power Allocation for Network Localization,”  IEEE International Conference on Communications (ICC), pp. 2785-2790, June 2013.
  * [14] C. Wang, Y. Ning, J. Wang, L. Zhang, J. Wan, Q. He,  “Optimized Deployment of Anchors based on GDOP Minimization for Ultra-Wideband Positioning,”  Journal of Spatial Science, pp. 1–18, Nov. 2020.
  * [15] A. Albaidhani, A. Morell, and J. L. Vicario,  “Anchor Selection for UWB Indoor Positioning,”  Transactions on Emerging Telecommunications Technologies vol. 30, no. 6, June 2019.
  * [16] A. Courtay, M. L. Gentil, O. Berder, P. Scalart, S. Fontaine and A. Carer,  “Anchor Selection Algorithm for Mobile Indoor Positioning using WSN with UWB Radio,”  IEEE Sensors Applications Symposium (SAS), 2019, pp. 1-5.
  * [17] A. Albaidhani, and A. Alsudani,  “Anchor Selection by Geometric Dilution of Precision for an Indoor Positioning System using Ultra‐Wide Band Technology,”  IET Wireless Sensor Systems, 2020.
  * [18] Y. Zhu, Y. Zhang, F. Yan, L. Shen, and Y. Wu,  “Node Selection for Cooperative Localization with NLOS Mitigation in Wireless Sensor Networks,”  Proc. in IEEE Globecom Workshops, Dec. 2016, pp. 1–6.
  * [19] H. Zhang, X. Qi, Q. Wei, Q., and L. Liu,  “TOA NLOS Mitigation Cooperative Localisation Algorithm based on Topological Unit,”  IET Signal Processing, vol. 14, no. 10, pp. 765–773, Jan. 2021.
  * [20] S. Wang, and Y. Zhang,  “Convex Hull based Node Selection NLoS Mitigation for Indoor Localization,”  Proc. in IEEE Wireless Communications and Networking Conference, Apr. 2016, pp. 1–5.
  * [21] D. T. A. Nguyen, et al.  “Deep learning-based localization for UWB systems,”  Electronics, val. 9, no. 10, pp. 1712, 2020.
  * [22] A. Poulose, D. S. Han,  “UWB Indoor Localization Using Deep Learning LSTM Networks,”  Appl. Sci. 2020, pp. 6290.
  * [23] M. Salimibeni, Z. Hajiakhondi-Meybodi, A. Mohammadi, and Y. Wang,  “TB-ICT: A Trustworthy Blockchain-Enabled System for Indoor COVID-19 Contact Tracing,”  arXiv preprint arXiv:2108.08275, 2021.
  * [24] J. Fan and A. S. Awan,  “Non-Line-of-Sight Identification Based on Unsupervised Machine Learning in Ultra-Wideband Systems,”  IEEE Access, vol. 7, pp. 32464-32471, 2019.
  * [25] Z. Hajiakhondi-Meybodi, A. Mohammadi, M. Hou, and K. N. Plataniotis,  “DQLEL: Deep Q-Learning for Energy-Optimized LoS/NLoS UWB Node Selection,” accepted in  IEEE Transactions on Signal Processing, 2022.
  * [26] D. Milioris,  “Efficient Indoor Localization via Reinforcement Learning,”  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 8350-8354.
  * [27] M. Mohammadi, A. Al-Fuqaha, M. Guizani and J. Oh,  “Semisupervised Deep Reinforcement Learning in Support of IoT and Smart City Services,”  IEEE Internet of Things Journal, vol. 5, no. 2, pp. 624-635, April 2018.
  * [28] Y. Li, X. Hu, Y. Zhuang, Z. Gao, P. Zhang and N. El-Sheimy,  “Deep Reinforcement Learning (DRL): Another Perspective for Unsupervised Wireless Localization,”  IEEE Internet of Things Journal, vol. 7, no. 7, pp. 6279-6287, July 2020.
  * [29] I. Uchendu, T. Xiao, Y. Lu, B. Zhu, M. Yan, J. Simon, M. Bennice, C. Fu, C. Ma, J. Jiao, and S. Levine,  “Jump-Start Reinforcement Learning,”  arXiv preprint arXiv:2204.02372, Apr. 2022.
  * [30] E. Marchesini et al.,  “Discrete Deep Reinforcement Learning for Mapless Navigation,”  IEEE Inter. Conf. on Robotics and Automation (ICRA), 2020, pp. 10688-10694.
  * [31] L. Gen, et al.  “Is Q-learning minimax optimal? A tight sample complexity analysis,”  arXiv preprint arXiv:2102.06548, 2021.