# High temperature AdS black holes are
low temperature quantum phonon gases

Xiangqing Kong, Tao Wang, and Liu Zhao
School of Physics, Nankai University, Tianjin 300071, China email:
<EMAIL_ADDRESS><EMAIL_ADDRESS>author, email<EMAIL_ADDRESS>
###### Abstract

We report a precise match between the high temperature $(D+2)$-dimensional
Tangherlini-AdS black hole and the low temperature quantum phonon gas in
$D$-dimensional nonmetallic crystals residing in $(D+1)$-dimensional flat
spacetime. The match is realized by use of the recently proposed restricted
phase space formalism for black hole thermodynamics, and the result can be
viewed as a novel contribution to the AdS/CMT correspondence on a quantitative
level.

Key words: AdS black hole, phonon gas, thermodynamics, restricted phase space
formalism

##  1 Introduction

Black holes are the most important objects predicted by the modern
relativistic theories of gravity. Since the 1970s it has been commonly
believed that black holes are thermal objects[1, 2, 3, 4, 5], and as such they
must contain a large number of microscopic degrees of freedom, and
understanding the nature of these microscopic degrees of freedom might provide
a window for inspecting the quantum nature of gravitation. Although the goal
for understanding quantum gravity is still far beyond the scope of our human’s
sight, there indeed has been a number of progresses and speculations toward
the understanding of the microscopic structure of black holes, among which the
most notable ideas come from the AdS/CFT correspondence[6, 7, 8] .

In the recent years, another line of thinking pumps up which attempts to
understand the black hole microstructure purely from thermodynamic
perspective, especially following the so-called extended phase space (EPS)
formalism [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]. The EPS formalism to
black hole thermodynamics is an approach which takes the negative cosmological
constant as a thermodynamic variable which is proportional to the pressure and
is accompanied by an associated conjugate variable known as the thermodynamic
volume. A significant amount of works have been published about the behavior
of various black holes under the EPS approach, mostly concentrated in the
critical phase transitions. Some authors also attempted to explore the
interaction potential between the microscopic degrees of freedom (also called
black hole molecules) of the black holes starting from the thermal equation of
states that arises from the EPS formalism [20, 21, 22].

Since the end of the last year, we proposed and kept working on an alternative
formalism of black hole thermodynamics called the restricted phase space (RPS)
formalism [23, 24, 25, 26]. The RPS formalism differs from the EPS formalism
in that the cosmological constant is no longer taken as a thermodynamic
variable but rather kept as a constant. Instead, we allow the gravitational
constant $G$ to be variable and introduced a pair of new thermodynamic
quantities $N=L^{D}/G,\mu=GTI_{E}/L^{D}$ which are respectively interpreted as
the effective number of microscopic degrees of freedom (or alternatively the
number of black hole molecules) and the chemical potential of the black hole,
wherein $D$ is the dimension of the black hole bifurcation horizon, $L$ is an
arbitrarily chosen constant length scale, $I_{E}$ is the Euclidean action of
the black hole spacetime and $T$ is the black hole temperature. The RPS
formalism is a restricted version of the holographic thermodynamics for
AdS/CFT systems proposed by Visser[28], but with the removal of cosmological
constant from the list of thermodynamic variables. This removal of
cosmological constant resolves a number of issues, including, but not limited
to, the following points: 1) The Euler homogeneity holds perfectly without
introducing rational coefficients in the mass formula; 2) The theory-changing
problem which we call the ensemble of theories issue which existed in the EPS
formalism is completely avoided, and the black hole mass restored its original
interpretation as internal energy rather than as enthalpy as did in the EPS
formalism; 3) The definition of $N,\mu$ are now independent of the holographic
duality, and thus the RPS formalism applies perfectly to the cases of non-AdS
black holes.

Subsequent works [27, 29, 30] revealed that the RPS formalism works perfectly
for black holes with different asymptotics in Einstein gravity and certain
higher curvature gravity models in diverse spacetime dimensions. Some
universal behaviors have also been found for black holes under this formalism,
e.g. the black holes in Einstein-Hilbert and Born-Infield like gravities
behave qualitatively the same, while black holes in Chern-Simons like
gravities behave completely different. Moreover, for certain AdS black holes,
the high temperature limit of the heat capacity has a power law dependence on
the temperature which is identical to the low temperature limit of the Debye
heat capacity of nonmetallic crystals. We believe that this similarity is not
a coincidence, and this letter is a further exploration on this similarity. As
will be shown in the main text below, the similarity between the high
temperature AdS black holes and the low temperature nonmetallic crystals is
much more profound. Not only the heat capacities, but also the Helmholtz free
energies, the internal energies and the entropies of the two drastically
different types of systems (high temperature AdS black hole and low
temperature nonmetallic crystals) behave almost identically in certain
temperature ranges. Since the microscopic description of nonmetallic crystal
is identical to a quantum phonon gas, we conclude that the high temperature
AdS black holes are actually equivalent to low temperature quantum phonon
gases. Although this link still does not reveal the nature of individual black
hole molecules, it indeed reveals their collective effects, i.e. the black
hole molecules are quantum, and their collective motion behaves like phonons
in certain temperature limit. We believe that this observation is a meaningful
progress towards the ultimate understanding of the black hole microstructure
as well as the quantum nature of gravitation.

##  2 Thermodynamics of Tangherlini-AdS black holes and the high temperature
limit

We will exemplify the connection between high temperature AdS black holes and
low temperature Bose gases by studying in detail the thermodynamics of
Tangherlini-AdS black holes in the RPS formalism. Throughout this letter we
work in units $c=1,k_{B}=1,\hbar=1$ but keep the Newton constant $G$ intact.

We take the spacetime dimension to be $D+2$, so that the bifurcation horizon
for the black hole is $D$. The $(D+2)$-dimensional Einstein-Hilbert action
with Gibbons-Hawking boundary term is given by

$\displaystyle I=\frac{1}{16\pi G}\int_{\mathcal{M}}(R-2\Lambda)\sqrt{g}\,{\rm
d}^{D+2}x+\frac{1}{8\pi G}\int_{\partial\mathcal{M}}K\sqrt{h}\,{\rm
d}^{D+1}x,$ (1)

where $g=|{\rm det}(g_{\mu\nu})|$, $h=|{\rm det}(h_{ab})|$, with $h_{ab}$
being the induced metric on the boundary $\partial\mathcal{M}$ of the
spacetime $\mathcal{M}$ and $K$ being the trace of the extrinsic curvature of
$\partial\mathcal{M}$ in $\mathcal{M}$. The inclusion of the Gibbons-Hawking
boundary term is important in order to obtain the correct value for the
Euclidean action $I_{E}$.

The metric of the $(D+2)$-dimensional Tangherlini-AdS black hole written in
spherical coordinates takes the form

$\displaystyle{\rm d}s^{2}$ $\displaystyle=-f(r){\rm d}t^{2}+f^{-1}(r){\rm
d}r^{2}+r^{2}{\rm d}\Omega_{D}^{2},$ (2) $\displaystyle f(r)$
$\displaystyle=1-\frac{16\pi
G}{D\mathcal{A}_{D}}\frac{M}{r^{D-1}}+\frac{r^{2}}{\ell^{2}},$ (3)

where ${\rm d}\Omega_{D}^{2}$ is the line element on a unit $D$-sphere with
area
$\mathcal{A}_{D}=\frac{2\pi^{(D+1)/2}}{\Gamma\left(\frac{D+1}{2}\right)}$, and
$\ell$ is the AdS radius which is related to the negative cosmological
constant via

$\Lambda=-\frac{D(D+1)}{2\ell^{2}}.$

The black hole event horizon is located at $r=r_{h}$ which is a root of the
equation $f(r)=0$. Accordingly, the mass of the black hole can be expressed in
terms of $r_{h}$ and $G$ as

$\displaystyle M=\frac{D\mathcal{A}_{D}r_{h}^{D-1}}{16\pi
G}\left(1+\frac{r_{h}^{2}}{\ell^{2}}\right)=\frac{D\pi^{(D-1)/2}r_{h}^{D-1}\left(\ell^{2}+r_{h}^{2}\right)}{8\ell^{2}\Gamma\left(\frac{D+1}{2}\right)G}.$
(4)

In the RPS formalism, the Tangherlini-AdS black hole has two independent
extensive variables, i.e. the entropy $S$ and the effective number of
microscopic degrees of freedom $N$,

$\displaystyle S$
$\displaystyle=\frac{\mathcal{A}_{D}r_{h}^{D}}{4G}=\frac{\pi^{(D+1)/2}r_{h}^{D}}{2\Gamma\left(\frac{D+1}{2}\right)G},$
(5) $\displaystyle N$ $\displaystyle=\frac{L^{D}}{G},$ (6)

where $r_{h}$ is the radius of the event horizon of the black hole which is a
root of the equation $f(r)=0$. The corresponding intensive variables are

$\displaystyle T$ $\displaystyle=\frac{1}{4\pi}\left(\frac{\partial
f}{\partial
r}\right)_{r=r_{h}}=\frac{(D-1)\ell^{2}+(D+1)r_{h}^{2}}{4\pi\ell^{2}r_{h}},$
(7) $\displaystyle\mu$
$\displaystyle=\frac{GTI_{E}}{L^{D}}=\frac{\pi^{(D-1)/2}r_{h}^{D-1}\left(\ell^{2}-r_{h}^{2}\right)}{8\ell^{2}L^{D}\Gamma\left(\frac{D+1}{2}\right)}.$
(8)

As expected, the two intensive variables are not independent of each other,
because the first law

${\rm d}M=T{\rm d}S+\mu{\rm d}N$

and the Euler homogeneity relation

$\displaystyle M=TS+\mu N.$ (9)

hold simultaneously, which implies the Gibbs-Duhem relation

$S{\rm d}T+N{\rm d}\mu=0.$

It is preferable to replace the geometric parameter $r_{h}$ and the coupling
coefficient $G$ in the above expressions for thermodynamic quantities by a set
of independent macro state parameters. The standard practice is to take the
extensive parameters $(S,N)$ as independent variables and re-express $M$
(understood as the internal energy) and the intensive variables $T,\mu$ as
functions in $(S,N)$: $M=M(S,N),T=T(S,N),\mu=\mu(S,N)$. However, since the
major goal of this work is to analyze the high temperature limit of various
thermodynamic quantities, we prefer to take $(T,N)$ as independent variables
and rewrite the other macro state functions in terms of these. From eqs.(6)
and (7) it is straightforward to get

$\displaystyle G$ $\displaystyle=\frac{L^{D}}{N},$ (10) $\displaystyle r_{h}$
$\displaystyle=\frac{\ell\left(2\pi\ell
T\pm\sqrt{4\pi^{2}\ell^{2}T^{2}-D^{2}+1}\right)}{D+1}.$ (11)

The two branched values for $r_{h}$ indicate that there are two black hole
states at the same temperature, of which the smaller one (i.e. the negative
branch) is unstable because it has smaller entropy. One can easily check that,
in the high temperature limit, the value of the negative branch of $r_{h}$
goes to zero. Therefore, in order to study the high temperature limit, we only
need to consider the positive branch of $r_{h}$. In this branch, the entropy
and the internal energy of the black hole can be rewritten as

$\displaystyle S$
$\displaystyle=\frac{N\pi^{(D+1)/2}}{2\Gamma\left(\frac{D+1}{2}\right)}\left(\frac{\ell\left(\sqrt{4\pi^{2}\ell^{2}T^{2}-D^{2}+1}+2\pi\ell
T\right)}{(D+1)L}\right)^{D},$ (12) $\displaystyle M$
$\displaystyle=\frac{ND\pi^{(D-1)/2}\left(\frac{\ell\left(\sqrt{4\pi^{2}\ell^{2}T^{2}-D^{2}+1}+2\pi\ell
T\right)}{D+1}\right)^{D-1}\left(\frac{\left(\sqrt{4\pi^{2}\ell^{2}T^{2}-D^{2}+1}+2\pi\ell
T\right)^{2}}{(D+1)^{2}}+1\right)}{8L^{D}\Gamma\left(\frac{D+1}{2}\right)}.$
(13)

Accordingly, the Helmholtz free energy $F$ and the heat capacity $C_{N}$ of
the black hole can be written as

$\displaystyle F$ $\displaystyle=M-TS$
$\displaystyle=\frac{N\pi^{(D-1)/2}\left(\frac{\ell\left(\sqrt{4\pi^{2}\ell^{2}T^{2}-D^{2}+1}+2\pi\ell
T\right)}{D+1}\right)^{D-1}\left(1-\frac{\left(\sqrt{4\pi^{2}\ell^{2}T^{2}-D^{2}+1}+2\pi\ell
T\right)^{2}}{(D+1)^{2}}\right)}{8L^{D}\Gamma\left(\frac{D+1}{2}\right)},$
(14) $\displaystyle C_{N}$ $\displaystyle=T\left(\frac{\partial S}{\partial
T}\right)_{N}=\frac{ND\pi^{(D+3)/2}\ell
T\left(\frac{\ell\left(\sqrt{4\pi^{2}\ell^{2}T^{2}-D^{2}+1}+2\pi\ell
T\right)}{(D+1)L}\right)^{D}}{\Gamma\left(\frac{D+1}{2}\right)\sqrt{4\pi^{2}\ell^{2}T^{2}-D^{2}+1}}.$
(15)

The temperature dependence of the above results appear to be very complicated.
However, if we consider the high temperature limit $T\to\infty$, these results
become much simplified,

$\displaystyle\lim_{T\to\infty}M$
$\displaystyle=\frac{N\pi^{1/2}D}{2(D+1)\Gamma\left(\frac{D+1}{2}\right)}\left(\frac{\pi^{3/2}(2\ell)^{2}}{(D+1)L}\right)^{D}T^{D+1},$
(16) $\displaystyle\lim_{T\to\infty}F$
$\displaystyle=-\frac{N\pi^{1/2}}{2(D+1)\Gamma\left(\frac{D+1}{2}\right)}\left(\frac{\pi^{3/2}(2\ell)^{2}}{(D+1)L}\right)^{D}T^{D+1},$
(17) $\displaystyle\lim_{T\to\infty}S$
$\displaystyle=\frac{N\pi^{1/2}}{2\Gamma\left(\frac{D+1}{2}\right)}\left(\frac{\pi^{3/2}(2\ell)^{2}}{(D+1)L}\right)^{D}T^{D},$
(18) $\displaystyle\lim_{T\to\infty}C_{N}$
$\displaystyle=\frac{N\pi^{1/2}D}{2\Gamma\left(\frac{D+1}{2}\right)}\left(\frac{\pi^{3/2}(2\ell)^{2}}{(D+1)L}\right)^{D}T^{D}.$
(19)

At this point we need to make it clear what is meant by the high temperature
limit $T\to\infty$. Since $T$ is a dimensionful quantity, it does not make
sense to say the temperature is high or low without comparing to a constant
characteristic temperature. It is evident that in the present case, the
characteristic temperature $T_{\rm{bh}}$ can be chosen as

$T_{\rm{bh}}=\frac{(D+1)L}{\pi^{3/2}(2\ell)^{2}}.$

With this choice we can rearrange the above high temperature (i.e.
$T\gg{T_{\rm{bh}}}$) values of thermodynamic quantities in the form

$\displaystyle M$
$\displaystyle\approx\frac{\pi^{1/2}}{2(D+1)\Gamma\left(\frac{D+1}{2}\right)}DNT\left(\frac{T}{T_{\rm{bh}}}\right)^{D},$
(20) $\displaystyle F$
$\displaystyle\approx-\frac{\pi^{1/2}}{2(D+1)\Gamma\left(\frac{D+1}{2}\right)}NT\left(\frac{T}{T_{\rm{bh}}}\right)^{D},$
(21) $\displaystyle S$
$\displaystyle\approx\frac{\pi^{1/2}}{2\Gamma\left(\frac{D+1}{2}\right)}N\left(\frac{T}{T_{\rm{bh}}}\right)^{D},$
(22) $\displaystyle C_{N}$
$\displaystyle\approx\frac{\pi^{1/2}}{2\Gamma\left(\frac{D+1}{2}\right)}DN\left(\frac{T}{T_{\rm{bh}}}\right)^{D}.$
(23)

##  3 Relation to low temperature phonon gases

The power law temperature dependence of the black hole thermodynamic
quantities $M,F,S,C_{N}$ reminds us of the quantum phonon gases that appear in
nonmetallic crystals. Let us recall the following known results [31] for the
internal energy $E$, Helmholtz free energy $F$, entropy $S$ and isochoric heat
capacity $C_{V}$ of the $D$-dimensional phonon gases in nonmetallic crystals
residing in $(D+1)$-dimensional flat spacetime, which can be obtained
straightforwardly by use of (grand)canonical ensemble and the Debye’s linear
dispersion relation $\epsilon(k)=v_{s}|k|$ for phonons,

$\displaystyle E$
$\displaystyle=DNT\mathscr{D}_{D}\left(\frac{T_{\mathrm{D}}}{T}\right),$ (24)
$\displaystyle F$ $\displaystyle=DNT\log\left(1-{\rm
e}^{-T_{\mathrm{D}}/T}\right)-NT\mathscr{D}_{D}\left(\frac{T_{\mathrm{D}}}{T}\right),$
(25) $\displaystyle S$ $\displaystyle=-DN\log\left(1-{\rm
e}^{-T_{\mathrm{D}}/T}\right)+(D+1)N\mathscr{D}_{D}\left(\frac{T_{\mathrm{D}}}{T}\right),$
(26) $\displaystyle C_{V}$
$\displaystyle=DN\mathscr{L}_{D}\left(\frac{T_{\mathrm{D}}}{T}\right),$ (27)

where $N$ is the number of crystal lattice atoms, $T_{D}$ is the Debye
temperature, $\mathscr{D}_{D}(x)$ is the $D$-dimensional Debye function

$\mathscr{D}_{D}(x)\equiv Dx^{-D}\int_{0}^{x}\frac{y^{D}}{{\rm e}^{y}-1}{\rm
d}{y},$

and $\mathscr{L}_{D}(x)$ is the $D$-dimensional Langevin function,

$\mathscr{L}_{D}(x)\equiv\mathscr{D}_{D}(x)-x\frac{{\rm d}{}}{{\rm
d}{x}}\mathscr{D}_{D}(x).$

At low temperature $T\ll T_{D}$, the above phonon gas functions behave as

$\displaystyle E$ $\displaystyle\approx
f(D)DNT\left(\frac{T}{T_{\mathrm{D}}}\right)^{D},$ (28) $\displaystyle F$
$\displaystyle\approx-f(D)NT\left(\frac{T}{T_{\mathrm{D}}}\right)^{D},$ (29)
$\displaystyle S$ $\displaystyle\approx
f(D)(D+1)N\left(\frac{T}{T_{\mathrm{D}}}\right)^{D},$ (30) $\displaystyle
C_{V}$ $\displaystyle\approx
f(D)D(D+1)N\left(\frac{T}{T_{\mathrm{D}}}\right)^{D},$ (31)

where $f(D)=D\zeta(D+1)\Gamma(D+1)$.

We can see a surprising similarity between the high temperature results
(20)-(23) for black holes and the low temperature results (28)-(31) for
quantum phonon gases. In fact, if we make the identification

$\displaystyle T_{\rm
D}=\left(\frac{2(D+1)\Gamma\left(\frac{D+1}{2}\right)f(D)}{\pi^{1/2}}\right)^{1/D}T_{\rm
bh},$ (32)

then there will be a precise quantitative match between eqs. (20)-(23) and
eqs. (28)-(31), provided we further identify $M$ with $E$ and $C_{N}$ with
$C_{V}$. We can even absorb the constant factor appearing on the right hand
side of eq.(32) by a simple redefinition of the length scale $L$,

$L\to\left(\frac{2(D+1)\Gamma\left(\frac{D+1}{2}\right)f(D)}{\pi^{1/2}}\right)^{1/D}L,$

which is allowed because of the arbitrariness in the choice of $L$. Then the
Debye temperature $T_{\mathrm{D}}$ for the phonon gas will be identical to the
black hole characteristic temperature $T_{\rm bh}$. We thus conclude that the
Tangherlini-AdS black holes at high temperature are nothing but quantum phonon
gases in nonmetallic crystals at low temperature. In this correspondence, the
number of black hole molecules is identified with the number of crystal
lattice atoms which are both denoted as $N$.

##  4 Concluding remarks and discussions

The major conclusion of this work can be summarized in a single sentence:
Tangherlini-AdS black holes at high temperature are equivalent to quantum
phonon gases in nonmetallic crystals at low temperature. Since we have already
evidence about the same power law dependences of heat capacities for different
black hole solutions in different gravity models[27], it is natural to expect
that this AdS/phonon gas correspondence may also hold for other asymptotically
AdS black holes. The AdS asymptotics is a necessary condition for the above
correspondence to hold, because for non-AdS black holes the heat capacities
are identically negative [26], which could not have direct relationship to
normal phonon gases.

Asymptotically AdS black holes have been extensively studied during the past
25 years or so, mostly in connection with AdS/CFT correspondence and its
various extensions such as AdS/CMT (condensed matter theory) [32, 33, 34],
AdS/QCD [35, 36, 37], etc. Although a countless number of theoretical results
have been obtained in these fields, most of the correspondences remain
qualitative. The present work adds some extra contribution to the field of
AdS/CMT correspondence with a precise quantitative match between the two
sides. Let us remark that the AdS/phonon gas correspondence described in this
work holds in generic spacetime dimension $D+2$, where $D$ is the dimension of
the bifurcation horizon of the black hole and $D+1$ is the dimension of the
boundary of the black hole spacetime in which the $D$-dimensional non-metallic
crystal resides. Let us stress that the AdS/phonon gas correspondence that we
report here is established purely on the level of thermodynamics. No attempts
on the microscopic description were made here, and there is no need to assume
broken translational symmetries from the beginning, as did in previous works
on similar topics in the context of AdS/CFT correspondence [38, 39, 40, 41].
On the level of thermodynamics, the correspondence is complete, because,
according to the famous Mathieu’s theorem, the thermodynamic properties of any
macroscopic system is completely determined by a single thermodynamic
potential with adapted independent variables. In the present context, the
Helmholtz free energy $F=F(T,N)$ plays the role of the thermodynamic potential
on both sides.

From the point of view of statistical physics, the power law dependence of
thermodynamic quantities for the phonon gas is solely determined by Debye’s
linear dispersion relation. The AdS/phonon gas correspondence seems to
indicate that the dispersion relation for black hole molecules at high
temperature is also linear, provided the underlying statistical physics
principles are the same. Although the black hole microstructure remains
unclear as we still cannot say anything about individual black hole molecules,
the result of the present work does reveal that the black hole molecules are
quantum, and collectively behave as phonons at sufficiently high temperature.

One may wonder why the AdS/phonon gas correspondence could hold, given that
the AdS black hole has a chemical potential whilst the phonon gas has not.
This is actually a misreading of the role of black hole chemical potential. As
we have already pointed out in the end of the last section, the number of
black hole molecules corresponds to the number of crystal lattice sites,
rather than the number of phonons — the latter is not conserved, and hence the
phonon gas has a vanishing chemical potential. On the other hand, the number
of lattice atoms is conserved and hence the crystal background does have a
nonvanishing chemical potential which is connected with the binding energy of
the crystal and is responsible for the lattice growth. Likewise, the black
hole chemical potential should also be connected with the binding energy of
the black hole. Using the Euler homogeneity relation (9), it is
straightforward to write down the high temperature behavior of the black hole
chemical potential,

$\displaystyle\mu$
$\displaystyle=\frac{F}{N}\approx-\frac{\pi^{1/2}}{2(D+1)\Gamma\left(\frac{D+1}{2}\right)}T\left(\frac{T}{T_{\rm
bh}}\right)^{D}.$ (33)

The negativity of the chemical potential implies that the black hole molecules
are attractive, which is required in order to make the black hole the stable.

## Acknowledgement

This work is supported by the National Natural Science Foundation of China
under the grant No. 12275138 and 11575088.

## Data Availability Statement

This manuscript has no associated data.

## Declaration of competing interest

The authors declare no competing interest.

## References

  * [1] Bekenstein, J.D. “Black holes and the second law,” _Lett. Nuovo Cim._ 1972, 4(15):737–740.
  * [2] Bekenstein, J.D. “Black holes and entropy,” Phys. Rev. D 7(8):2333–2346, 1973.
  * [3] Bardeen, J.M.; Carter, B.; Hawking, S.W. “The four laws of black hole mechanics,” Comm. Math. Phys. 31(2):161–170, 1973.
  * [4] Hawking, S.W. “Particle creation by black holes,”  Comm. Math. Phys. 43:199-220, 1975.
  * [5] Bekenstein, J,D. “Statistical black-hole thermodynamics,” Physical Review D, 12(10):3077, 1975.
  * [6] ’t Hooft, G. “Dimensional reduction in quantum gravity,” [arXiv:gr-qc/9310026]
  * [7] Susskind, L. “The world as a hologram,” J. Math. Phys. 36(11):6377–6396, 1995. [arXiv:hep-th/9409089].
  * [8] Maldacena, J. “The large $N$ limit of superconformal field theories and supergravity,” Adv. Theore. & Math. Phys., 2(2):231–252, 1998, [arXiv:hep-th/9711200].
  * [9] Kastor, D.; Ray, S.; Traschen, J. “Enthalpy and the mechanics of AdS black holes,” Class. Quant. Grav. 26(19):195011, 2009. [arXiv:0904.2765].
  * [10] Dolan, B.P. “The cosmological constant and the black hole equation of state,” Class. Quant. Grav. 28(12):125020, 2010. [arXiv:1008.5023].
  * [11] Dolan, B.P. “Pressure and volume in the first law of black hole thermodynamics,” Class. Quant. Grav. 28(23):235017, 2011. [arXiv:1106.6260].
  * [12] Dolan, B.P. “Compressibility of rotating black holes,” Phys. Rev. D 84(12):127503, 2011. [arXiv:1109.0198].
  * [13] Kubizňák, D.; Mann, R.B. “P-V criticality of charged AdS black holes,” J. High Energy Phys. 7:1–25, 2012. [arXiv:1205.0559].
  * [14] Cai, R.G.; Cao, L.M.; Li, L.; Yang, R.Q. “P-V criticality in the extended phase space of Gauss-Bonnet black holes in AdS space,” J. High Energy Phys. 9:1–22, 2013. [arXiv:1306.6233].
  * [15] Kubizňák, D.; Mann, R.B.; Teo, M. “Black hole chemistry: thermodynamics with Lambda,” Class. Quant. Grav. 34(6):063001, 2017. [arXiv:1608.06147].
  * [16] Xu, W.; Zhao, L. “Critical phenomena of static charged AdS black holes in conformal gravity,” Phys. Lett. B 736:214–220, 2014. [arXiv:1405.7665].
  * [17] Xu, W.; Xu, H.; Zhao, L. “Gauss–Bonnet coupling constant as a free thermodynamical variable and the associated criticality,” Euro. Phys. J. C 74(7):1–13, 2014. [arXiv:1311.3053].
  * [18] Lemos, J.P.S.; Zaslavskii, O.B. “Black hole thermodynamics with the cosmological constant as independent variable: Bridge between the enthalpy and the Euclidean path integral approaches,” Phys. Lett. B 786:296–299, 2018. [arXiv:1806.07910].
  * [19] Zhang, J.L.; Cai, R.G.; Yu, H. “Phase transition and thermodynamical geometry of Reissner-Nordström-AdS black holes in extended phase space,” Phys. Rev. D 91(4):044028, 2015. [arXiv:1502.01428].
  * [20] Wei, S.W.; Liu, Y.X. “Insight into the microscopic structure of an AdS black hole from a thermodynamical phase transition,” Phys. Rev. Lett., 115(11):111302, 2015. [arXiv:1502.00386].
  * [21] Wei, S.W.; Liu, Y.X. “Extended thermodynamics and microstructures of four-dimensional charged gauss-bonnet black hole in AdS space,” Phys. Rev. D, 101(10):104018, 2020. [arXiv:2003.14275].
  * [22] Dehyadegari, A.; Sheykhi, A.; Wei, S.W. “Microstructure of charged AdS black hole via $P-V$ criticality,” Phys. Rev. D, 102(10):104013, 2020. [arXiv:2006.12265].
  * [23] Gao, Z.; Zhao, L. “Restricted phase space thermodynamics for AdS black holes via holography,” Class. Quant. Grav. 39:075019, 2021. [arXiv:2112.02386].
  * [24] Gao, Z.; Kong, X.; Zhao, L. “Thermodynamics of Kerr-AdS black holes in the restricted phase space,” Euro. Phys. J. C 82(2):1–10, 2022. [arXiv:2112.08672].
  * [25] Wang, T.; Zhao, L. “Black hole thermodynamics is extensive with variable newton constant,” Phys. Lett. B 827:136935, 2022. [arXiv:2112.11236].
  * [26] Zhao, L. “Thermodynamics for higher dimensional rotating black holes with variable Newton constant,” Chin. Phys. C 46(5):055105, 2022. [arXiv:2201.00521].
  * [27] Kong, X.; Wang, T.; Gao, Z.; Zhao, L. “Restricted phase space thermodynamics for black holes in higher dimensions and higher curvature gravities,” Entropy, 24(8):1131, 2022. [arXiv:2208.07748].
  * [28] Visser, M.R. “Holographic thermodynamics requires a chemical potential for color. Phys. Rev. D 105(10):106014, 2022. [arXiv:2101.04145].
  * [29] Sadeghi, J.; Shokri, M.; Gashti, S. N.; Alipour, M. R. “RPS thermodynamics of Taub-NUT AdS black holes in the presence of central charge and the weak gravity conjecture,” [arXiv:2205.03648].
  * [30] Bai, Y.-Y.; Chen, X.-R.; Xu, Z.-M.; Wu, B. “Revisit on thermodynamics of BTZ black hole with variable Newton constant,” [arXiv:2208.11859].
  * [31] Zhao, L. “Statistical thermal physics,” Science Press, Beijing, 2019, ISBN: 978-7-03-060735-5 (textbook in Chinese).
  * [32] Hartnoll, S.A. “Lectures on holographic methods for condensed matter physics,” Class. Quant. Grav., 26(22):224002, 2009. [arXiv:0903.3246].
  * [33] Sachdev, S. “Condensed matter and AdS/CFT,” In From gravity to thermal gauge theories: the AdS/CFT correspondence, pages 273–311. Springer, 2011.
  * [34] Zaanen, J.; Liu, Y.; Sun, Y.W.; Schalm, K. Holographic duality in condensed matter physics. Cambridge University Press, 2015.
  * [35] Karch, A.; Katz, E.; Son, D.T.; Stephanov, M.A. “Linear confinement and AdS/QCD,” Phys. Rev. D, 74(1):015005, 2006. [arXiv:hep-ph/0602229].
  * [36] Kajantie, K.; Tahkokallio, T.; Yee, J.T. “Thermodynamics of AdS/QCD,” J. High Energy Phys., 2007(01):019, 2007. [arXiv:hep-ph/0609254].
  * [37] Panero, M. “Thermodynamics of the QCD plasma and the large-N limit,” Phys. Rev. Lett., 103(23):232001, 2009. [arXiv:0907.3719].
  * [38] Ling, Y.; Niu, C.; Wu, J.P.; Xian, Z.Y.; Zhang H.B. “Metal-insulator transition by holographic charge density waves. Phys. Rev. Lett. 113(9):091602, 2014. [arXiv:1404.0777].
  * [39] Cai, R.G.; Li, L.; Wang, Y.A.; Zaanen, J. “Intertwined order and holography: the case of parity breaking pair density waves,” Phys. Rev. Lett. 119(18):181601, 2017. [arXiv:1706.01470].
  * [40] Alberte, L.; Ammon, M.; Baggioli, M.; Jiménez, A.; Pujolàs, O. “Black hole elasticity and gapped transverse phonons in holography,” J. High Energy Phys. 2018(1):1–30, 2018. [arXiv:1708.08477].
  * [41] Alberte, L.; Ammon, M.; Jiménez-Alba, A.; Baggioli, M.; Pujolàs, O. “Holographic phonons,” Phys. Rev. Lett. 120(17):171602, 2018. [arXiv:1711.03100].


# MIX-MAB: Reinforcement Learning-based Resource Allocation Algorithm for
LoRaWAN

Farzad Azizi∗, Benyamin Teymuri∗, Rojin Aslani∗, Mehdi Rasti∗, Jesse
Tolvanen†, and Pedro H. J. Nardelli†
∗Department of Computer Engineering, Amirkabir University of Technology,
Tehran, Iran
†School of Energy Systems, Lappeenranta-Lahti University of Technology,
Lappeenranta, Finland
Email: {farzad.azizi, benyamin.teymuri, rojinaslani<EMAIL_ADDRESS>{jesse.tolvanen<EMAIL_ADDRESS>This paper is partly supported by
Academy of Finland via (a) FIREMAN consortium n.326270 as part of CHIST-ERA
grant CHIST-ERA-17-BDSI-003, and (b) EnergyNet Research Fellowship
n.321265/n.328869 and (c) n.339541, and by Jane and Aatos Erkko Foundation via
STREAM project.

###### Abstract

This paper focuses on improving the resource allocation algorithm in terms of
packet delivery ratio (PDR), i.e., the number of successfully received packets
sent by end devices (EDs) in a long-range wide-area network (LoRaWAN). Setting
the transmission parameters significantly affects the PDR. Employing
reinforcement learning (RL), we propose a resource allocation algorithm that
enables the EDs to configure their transmission parameters in a distributed
manner. We model the resource allocation problem as a multi-armed bandit (MAB)
and then address it by proposing a two-phase algorithm named MIX-MAB, which
consists of the exponential weights for exploration and exploitation (EXP3)
and successive elimination (SE) algorithms. We evaluate the MIX-MAB
performance through simulation results and compare it with other existing
approaches. Numerical results show that the proposed solution performs better
than the existing schemes in terms of convergence time and PDR.

###### Index Terms:

IoT, LPWAN, LoRaWAN, LoRa, distributed resource allocation, reinforcement
learning, multi-armed-bandit.

##  I Introduction

The maturity of internet of things (IoT) technology is already rapid.
According to projections for the next ten years, over $125\times 10^{9}$ IoT
devices are expected to be connected worldwide [1]. The Low-power wide-area
network (LPWAN) can provide the network connection for many end devices (EDs)
in a wide range consuming low battery power [2]. Many protocols for LPWAN
exist including long-range wide area network (LoRaWAN) [2], SigFox [3], and
NB-IoT. LoRaWAN is one of the most promising candidates for IoT, attracting
more attention and can support many applications such as electricity metering,
localization, and industrial applications [4-5].

The transmission parameters, i.e., radio resources, including spreading
factors (SFs), sub-channels (SCs), and transmission power (TP), have a
significant role in determining the throughput of LoRaWAN [6]. Legacy LoRa
runs an adaptive data rate (ADR) mechanism in the central controller, i.e., a
network server (NS), to compute the transmission parameters of EDs and send
them back as a MAC command [7]. However, frequent communication between EDs
and the NS is inefficient because of the EDs’ limited energy and duty cycle
(DC) restrictions. Furthermore, the NS cannot respond to all the messages due
to strict deadlines. Therefore, it is essential to design a distributed
resource allocation algorithm in LoRaWAN.

Machine learning makes it possible to use distributed algorithms.
Reinforcement learning (RL) is a machine learning technique that does not need
any training data sets, making it the best choice for LoRaWAN. The literature
review shows that RL techniques can improve resource allocation performance in
LoRaWAN by enabling each LoRa ED to select the most suitable configuration
settings through the self-learning process. More especially in [8-9], a non-
stationary RL-based resource allocation algorithm called LoRa-MAB is proposed,
using an adversarial environment similar to LoRaWAN’s circumstances. However,
LoRa-MAB suffers from high convergence time due to the long exploration
process. This paper shows that combining the non-stationary adversarial
algorithms, suitable for LoRaWAN environment, with stochastic ones, having the
advantage of a short exploration process, improves the convergence time and
increases the packet delivery ratio (PDR) in LoRaWAN. Contributions of this
paper are as follow:

  * •

We model the resource allocation problem in LoRaWAN as a Multi-armed bandit
(MAB) problem. Then we employ a mixture of exponential weights for exploration
and exploitation (EXP3) as a non-stationary adversarial scheme and the
successive elimination (SE) as a non-stationary stochastic scheme to propose
our solution called MIX-MAB.

  * •

Through simulation results, we compare the performance of our algorithm with
LoRa-MAB [9] and Legacy LoRa [11] approaches in terms of convergence time,
energy consumption (EC), and PDR. Simulation results show that the convergence
time of the MIX-MAB is half of the LoRa-MAB while achieving the higher PDR
than others in different scenarios.

  * •

We also present a simulation scenario in which EDs have the freedom to select
their SFs, SCs, and TP simultaneously. Our numerical results show that the
proposed MIX-MAB algorithm achieves high PDR with the lowest EC in such a
scenario compared to those where the EDs can configure only one or two
transmission parameters.

  * •

Finally, the effect of the number of packets sent by each ED is examined,
showing the more frequently transmitted packets leads to reduced the PDR and
the EC.

Figure 1: LoRaWAN network architecture.

Rest of this paper is organized as follows. Section II presents the background
and related works. Our proposed MIX-MAB algorithm is described in Section III.
Sections IV and V contain the numerical results and conclusion, respectively.

##  II Background and Related Works

This section reviews LoRa physical layer and transmission parameters in
LoRaWAN. Then, it examines the related works.

###  II-A LoRa and LoRaWAN overview

LoRaWAN is a long-range wireless interconnections system composed of two core
components: LoRa and LoRaWAN. LoRa is a physical layer using the chirp spread
spectrum (CSS) radio modulation developed by Semtech [3]. LoRaWAN, implemented
on top of the LoRa, includes data link and network layers using star network
topology for data transmission. LoRaWAN network architecture is shown in
Fig.1. As seen, the EDs send data to the NS through the gateways (GWs).
LoRaWAN specifications are documented by LoRa Alliance [7].

###  II-B LoRa transmission parameters

In LoRa physical layer, each transmission depends on the following parameters:

  * •

SFs: EDs can select an SF value from 7-12 based on the environmental condition
between the ED and the GW. Selecting SF creates a trade-off between the data
rate, communication range, and energy utilization.

  * •

SCs: Depending on the world region, LoRa communications can operate over
license-free sub-GHz radio frequency bands including
$\\{433,\leavevmode\nobreak\ 868,\leavevmode\nobreak\ 915\\}$ MHz.

  * •

TP: The LoRa radio TP is adjustable from $-4$ to $20$ dBm in steps of $1$ dBm.

Successful transmissions in LoRaWAN, leading to the raised PDR, rely on the
interference management performed by adequately adjusting the transmission
parameters depending on the conditions between the EDs and GWs. There are two
ways to control transmission parameters in LoRaWAN: link-based approach and
network-aware approach [2]. In the link-based approach, transmission
parameters are asynchronously configured in a centralized manner by commands
that ED receives from the NS [10]. In the network-aware method, each ED
configures its transmission parameters in a distributed way.

###  II-C Related Works

Several research efforts have improved LoRa/LoRaWAN performance, focusing on
optimization or performance analysis [7-11]. The ADR algorithm has been
proposed in [7] for setting the LoRa transmission parameters. The proposed
approaches in [2] and [10] worked on improving the original ADR algorithm by
using the history of received packets in the NS. However, the ADR mechanism as
a centralized link-based approach has two shortcomings. The first one is that
its PDR decreases by increasing the number of EDs. The second one is the
increased EC in dens deployment of IoT EDs. Current research has focused on
using machine learning techniques such as RL, enabling EDs to use innovative
and inherently distributed techniques, thus preventing them from evacuating
their limited power by permanently communicating with the NS [8-9], [12]. The
MAB [8-9] and Q-learning [12] are two RL algorithms used in the literature to
propose distributed radio resource allocation in LoRaWAN. In [12], authors
applied Q-learning to offer a resource allocation for LoRaWAN, aiming at
decreasing the collision rate and improving the network PDR. However, despite
increasing the EC using the method in [12], the PDR is still a function of
available channels. Moreover, the Q-Learning solution requires the database to
save its processing data. The LoRa-MAB algorithm proposed in [8] suffers from
a severe drawback of high convergence time (equal to $200$ kHours as discussed
in [9]). The SE is a MAB-based algorithm presented in [13] while the
adversarial environment of LoRa has not been taken into account.

##  III Our proposed MIX-MAB algorithm

This section proposes a distributed resource allocation algorithm in LoRa.
Machine learning techniques as a distributed approach are divided into
supervised, unsupervised, and reinforcement learning. There is no need to
train data sets in RL-based methods, where learning happens through
interaction with the environment. RL agent can perceive and analyze its
environment, take actions and learn through trial and error [14]. We employ RL
to propose our resource allocation algorithm. Therefore, there is no need to
provide predefined data to EDs, and they will learn through sending messages
on the network. Thus, our proposed algorithm imposes no computational overhead
on the network from this point of view.

In LoRaWAN, the agents are the LoRa EDs interacting with the environment,
including GWs and the other EDs, to take actions defined as selecting their
transmission parameters set. The agent learns the best actions based on the
received reward defined based on the acknowledgment (ACK) messages. More
specifically, a LoRa ED selects a set of transmission parameters and sends its
data packet based on the selected setting. If the NS receives the packet, it
sends a confirmation ACK message to the ED. Receiving the ACK, the ED assigns
a binary reward (defined later) to the selected action set and uses it for
subsequent transmission parameters index.

We model the transmission parameters configuration by LoRa EDs as a MAB
problem, an RL-based technique, and formulated it using k-armed bandits.
Accordingly, an agent selects from k-different actions and receives a reward
based on its chosen action.

Three general categories of stochastic, adversarial, and switching bandit
algorithms exists to address the MAB problems. EXP3 is a non-stationary
adversarial MAB problem. LoRa EDs are placed in this type of algorithm because
selecting the same parameter like SFs affects the other EDs. But the lack of a
short exploration process in this approach will expand the convergence time.
Stochastic algorithms such as SE are not suitable for the LoRaWAN individually
because of their adversarial nature. However, the long exploration process of
EXP3 results in a high convergence time, which is a critical performance
criterion in LoRaWAN. On the other hand, the SE algorithm has the advantage of
short convergence time due to a short exploration process. So, Inspired by the
benefits of EXP3 and SE algorithms used in [9] [13], respectively, we combine
these two approaches and propose a new algorithm called MIX-MAB.

We assume that there are $U$ EDs in the LoRaWAN network forming the set of
$\mathcal{U}=\\{1,2,\cdots,U\\}$. Each ED aims at maximizing its PDR in a
decentralized manner by learning to select the most appropriate transmission
parameters set.

Let $\mathcal{S}$, $\mathcal{C}$, and $\mathcal{P}$ denote the set of SFs,
SCs, and TP, respectively. Assuming that each action is a vector composed of
three parameters representing SF, SC, and TP, $I_{k}=\\{s_{k},c_{k},p_{k}\\}$
denote the $k$th vector of parameters, means $k$th action, in which
$s_{k}\in\mathcal{S}$, $c_{k}\in\mathcal{C}$, and $p_{k}\in\mathcal{P}$ are
the values of SF, SC and TP in $k$th action, respectively. We assume that
there are $K$ actions whose set denoted by
$\mathcal{K}=\\{I_{0},I_{1},\cdots,I_{K-2},I_{K-1}\\}$. Let
$A_{u}(t)\in\mathcal{K}$ describe the selected action of the $u$th ED at the
$t$th iteration. Each iteration corresponds to a packet arrival in the ED.
Taking the $k$th action, the $u$th ED receives the related reward presented by
$R^{u}_{k}(t)\in\\{0,1\\}$, where $R^{u}_{k}(t)=1$ for successful
transmissions, i.e., the ACK message is received, while receiving the NACK
message is denoted by $R^{u}_{k}(t)=0$ for fail transmissions. We assign a
probability of selection to each action. Let $P^{u}_{k}(t)$ be the probability
of taking the $k$th action by the $u$th ED at the $t$th iteration. To obtain
the probabilities, we assign a weight to each action. $W^{u}_{k}(t)$ is the
weight of the $k$th action selected by the $u$th ED at the $t$th iteration.
The goal of each ED $u$ is to update $P^{u}_{k}(t)$ to achieve the largest
reward during $T$ iterations.

Our proposed MIX-MAB algorithm presented in Algorithm 1 composed of two
phases. The first phase, called the pre-processing phase, only includes
exploration process. The second phase, called the main processing phase,
consists of exploration and exploitation processes. Fig. 2 illustrates the
proposed MIX-MAB scheme for the $u$th ED as explained in what follows.

Figure 2: Proposed two-phase solution scheme for the $u$th ED.

1 Initialization:  __

2 Set $u\in\mathcal{U}$ as a $u$th ED

3 Set learning rate
$\gamma\\!=\\!\min\Big{\\{}1,\sqrt{\frac{K\log(K)}{(e-1)T}}\Big{\\}}$,
$e\\!=\\!2.71$

4 Set $l^{\textnormal{EXP}}=5,\alpha_{u}=1,l^{\textnormal{EE}}=100$,

5 Set $P^{u}_{k}(0)=nan,N^{u}_{k}=0,W^{u}_{k}(0)=1,\forall k\in\mathcal{K}$

6

7

Result: $A^{u}(t),\forall t=\\{1,2,\cdots,T\\}$

8 for  _$t=1$ to $T$_ do

9 if  _$\min_{k\in\mathcal{K}}\\{N^{u}_{k}\\}\leq l^{\textnormal{EXP}}$_ then

10 Select action $k\in\mathcal{K}$ by round-robin selection

11 else

12 Select action $k\in\mathcal{K}$ by PDF selection

13 end if

14 Set $A^{u}(t)=I_{k}$ and perform the action

15 $R^{u}_{k}(t)=\begin{cases}1\quad&\textnormal{if ACK is received}\\\
0\quad&\textnormal{otherwise}.\end{cases}$

16
$P^{u}_{k}(t+1)=(1-\gamma)\Big{(}\frac{W^{u}\\_k(t)}{\sum\\_{k\in\mathcal{K}}{W^{u}\\_k(t)}}\Big{)}+\frac{\gamma}{K}$

17
$P^{u}_{k}(t+1)=\frac{P^{u}_{k}(t+1)}{\sum_{k\in\mathcal{K}}{P^{u}_{k}(t)}}$

18 $W^{u}_{k}(t+1)=W^{u}_{k}(t)\times exp\Big{(}\frac{\gamma
R^{u}_{k}(t)}{K\times P^{u}_{k}(t+1)}\Big{)}$

19 $N^{u}_{k}=N^{u}_{k}+1$

20 if  _$N^{u}_{k} >l^{\textnormal{EXP}}$ &&
$P^{u}_{k}(t+1)<\frac{1}{2}\max_{\forall k\in\mathcal{K}}\\{P^{u}_{k}(t)\\}$_
then

21 $P^{u}_{k}(t+1)=0$

22

23 end if

24 if  _${N^{u}_{k} >\alpha_{u}\times l^{\textnormal{EE}}}$_ then

25 ${N^{u}_{k}=0,\forall k\in\mathcal{K}}$

26 ${\alpha_{u}=\alpha_{u}+1}$

27 end if

28

29 end for

Algorithm 1 MIX-MAB

###  III-A The pre-processing phase of MIX-MAB algorithm

Let $N^{u}_{k}$ represent the number of times that $u$th ED selects the $k$th
action. At the first time, when the $u$th ED has a packet to send, it selects
the first configuration setting, i.e., $I_{0}$. The next packet of the $u$th
ED takes the action of $I_{1}$ of setting parameters in a round-robin manner
and so on (line 8 of Algorithm 1). This process, taken from the SE algorithm,
continues until the $u$th ED takes all actions once. Then, the exploration
process is repeated for $l^{\textnormal{EXP}}$ times starting from the
beginning by selecting $I_{0}$ to $I_{K-1}$ (line 7).

Based on our simulation results, setting $l^{\textnormal{EXP}}=5$ leads to a
short exploration, generating the best results. Each time $u$th ED takes
action, the weight and probability of that action are updated based on the
reception of the ACK message. Lines 14-16 in Algorithm 1 present the equations
for calculating the probability and weight of an especial index for the
following iteration. According to these equations, taken from the EXP3
algorithm, if $k$th action does not result in receiving the ACK message by the
$u$th ED, i.e., $R^{u}_{k}(t)=0$, the relevant weight and probability of that
action, i.e., $W^{u}_{k}$ and $P^{u}_{k}$ are not updated, so they remain as
the value in the previous iteration. Note that the summation of all
probabilities should be one. So, after calculating the probability using line
14, we normalize the probabilities in line 15. In Fig. 2, ✓ and ✗ represent
the reception and non-reception of ACK, respectively. As seen in Fig. 2(a),
the value of $N^{u}_{k}$ for all $k\in\mathcal{K}$ reaches
$l^{\textnormal{EXP}}$ at the end of the pre-processing phase. The whole
actions’ weights and probabilities are the first phase’s output, which will be
used as input in the second phase.

###  III-B The main processing phase of MIX-MAB algorithm

After finishing the first phase, the second phase, i.e., the main process,
starts as shown in Fig. 2(b). In this phase, the actions are selected based on
their probability density function (PDF) values (line 10). Based on this
condition, which originated from the EXP3 scheme, actions with a high
probability have a higher chance of being chosen. Taking individual action $k$
in each iteration $t$, the $u$th ED updates that action’s weight and
probability by receiving the ACK. Suppose the updated probability of the $k$th
action is smaller than half of the maximum probability of all actions in
$\mathcal{K}$. In that case, the $u$th ED removes the $k$th action from its
available actions not to be selected in subsequent steps (lines 18-20). This
removal process is done implicitly by setting the probability of the $k$th
action equal to zero. Note that the above threshold for removing an action is
obtained heuristically through the simulation results, leading to the best
performance. This process continues until the number of times one of the
actions is selected, e.g., $N^{u}_{k}$ reaches a threshold value (lines
21-24). Actually, the action with the greatest $N^{u}_{k}$ is the most
suitable one with the best transmission parameter configuration. One of our
novelties in this phase is considering dynamic values for the threshold. Let
$\alpha_{u}\times l^{\textnormal{EE}}$ denote the considered threshold, where
$l^{\textnormal{EE}}=100$ is a constant parameter and $\alpha_{u}$ is a
variable parameter. At the initialization, we set $\alpha_{u}=1$ making the
threshold be equal to $l^{\textnormal{EE}}=100$. When $N^{u}_{k}$ for special
action like $k$ reaches the threshold of $100$, the $\alpha_{u}$’s value is
increased by one leading to a grown threshold value to $2\times
l^{\textnormal{EE}}=200$ and so on. Whenever the value of $N^{u}_{k}$ of one
action achieves the threshold, the $N^{u}_{k}$ for all actions are set to zero
while the weight and probabilities values remain unchanged. Through resetting
the number of selected actions, the algorithm continues from line 7. So it
starts the exploration process, i.e., pre-processing phase, again and gives a
new chance to the actions that already have been removed in the second phase
and may have become an appropriate option by changing the environmental
conditions. Note that by not zeroing the weight and probabilities of the
actions in the reset process, the proposed algorithm does not eliminate the
previous experiences learned from the environment.

##  IV Numerical Results

This section evaluates our algorithm performance by simulation results and
compares it with other approaches.

###  IV-A Simulation Setup

We use the LoRa-MAB simulator proposed in [9] and customize it to evaluate our
algorithm performance111This framework is available in
https://github.com/Farzad-Azizi/MIX-MAB. We consider a LoRaWAN network
composed of one GW, located at the center of a disc-shaped cell of radius
$r=4.5$ km, with $N=100$ EDs uniformly distributed as in [9]. We use the log-
distance path loss model, presented in [11]. The $1\%$ DC limitation is
satisfied by setting the packet generation rate of each ED to $15$
packets/hour and the packet length of $50$ bytes generated through an
exponential distribution. We assume that there is no collision between the ACK
and uplink messages. The GW delivers the ACK message on a separate channel
with a higher DC. Thus, when the ED does not receive the ACK, it shows that
the packet has been lost. We also consider the inter-SF collision and capture
effect. We run the simulations for $72\times 10^{10}$ millisecond, which is
equal to $200$ KHours horizon time. Other parameters that affect the
performance of LoRaWAN, including bandwidth and coding rate, are set as
$125$KHz and $4/5$, respectively.

To evaluate the proposed algorithm performance and compare it with other
schemes, we use three following metrics:

  1. 1.

Convergence time: an RL algorithm converges when the learning curve gets flat
and no longer increases.

  2. 2.

PDR: described the total received packets by the GW divided by the total sent
packet from all EDs.

  3. 3.

EC: defined as the average EC per transmitted packet per ED.

We also consider five following scenarios:

  * •

Scenario 1: Each ED can select one SF $\in[7,12]$, SC and TP are fixed at SC
$=868100$ Hz and TP $=14$ dBm.

  * •

Scenario 2: Each ED can select one SF $\in[7,12]$, one SC
$\in\\{868100,868300,868500\\}$ Hz, and TP is fixed at TP $=14$ dBm.

  * •

Scenario 3: Each ED can select one SF $\in[7,12]$, one TP $\in\\{8,11,14\\}$
dBm, and SC is fixed at SC $868100$ Hz.

  * •

Scenario 4: Each ED can select one SF $\in[7,12]$, one SC
$\in\\{868100,868300,868500\\}$ Hz, and one TP $\in\\{8,11,14\\}$ dBm.

  * •

Scenario 5: Options are like the previous scenario, but the number of packets
sent by EDs changed from 1 packet per hour to 1 packet per day and 1 packet
per week.

Figure 3: PDR & EC in MIX-MAB, LoRa-MAB, and LoRa in Scenario 1.

###  IV-B Simulation Results

Now, we provide the simulation results comparing the performance of our
proposed algorithm with the LoRa-MAB [9] and the Legacy LoRa [11] algorithms
in five defined scenarios. The LoRa-MAB algorithm proposed in [9] is based on
RL that uses the EXP3 scheme. The EDs configure their transmission parameters
randomly in the Legacy LoRa method.

####  IV-B1 Scenario 1

Fig. 3 shows the PDR and EC in MIX-MAB, LoRa-MAB, and Legacy LoRa in Scenario
1. As we observe, the PDR of our proposed solution is higher than LoRa-MAB and
Legacy LoRa algorithms. This is due to the nature of our proposed algorithm,
which uses a combination of short-term exploration at the first phase and
long-term exploitation and exploration processes at the second phase together.
On the contrary, LoRa-MAB does not apply the short-term exploration initially,
and Legacy LoRa does not employ the learning process at all. Additional to
that, the probability initialization in LoRa-MAB is defined as a uniform
distribution, i.e., $P^{u}_{k}(0)=\frac{1}{K},\forall
k\in\mathcal{K},u\in\mathcal{U}$. However, we do not assume equal probability
initialization for each action and ED in MIX-MAB. Instead, we set the initial
probabilities as an undefined number, i.e., $nan$, and then update them based
on the ACK reception. Besides that, MIX-MAB gives a second chance to all
removed actions, but LoRa-MAB does not.

We also see in Fig. 3 that the EC of the MIX-MAB is the same as the Legacy
LoRa and LoRa-MAB. The reason is that each ED transmits with TP of 14 $dBm$ in
all algorithms leading to the same EC. Another important observation from Fig.
3 is that our proposed algorithm’s PDR converges after $100$ KHours in horizon
time while the LoRa-MAB converges after $200$ KHours. Therefore, the
convergence time of the proposed solution is half of the LoRa-MAB. Actually,
uniform probability initialization in LoRa-MAB causes wrong choices to take a
long time removing from the actions, leading to an increase in the convergence
time. The Legacy LoRa scheme is not a learning-based algorithm, so its PDR and
EC remained unchanged in time.

Figure 4: PDR & EC in MIX-MAB, LoRa-MAB, and LoRa in Scenario 2.

####  IV-B2 Scenario 2

In this scenario, we evaluate the effect of SCs selection on overall
performance. For this purpose, the EDs have the freedom of choosing one of
three SCs compared to Scenario 1. Fig. 4 shows the PDR and EC in MIX-MAB,
LoRa-MAB, and Legacy LoRa in Scenario 2. As seen, the PDR of the MIX-MAB is
higher than that in scenario 1. Thus, we can conclude that increasing the
number of SCs increases the PDR. We also see that in Scenario 2, the PDR of
the MIX-MAB is still higher than the Legacy LoRa and LoRa-MAB while their
consumed power is the same.

####  IV-B3 Scenario 3

In this scenario, we investigate the effect of TP on the proposed algorithm
performance. For this goal, EDs have the freedom of transmitting with one of
three TP values compared to Scenario 1. Fig. 5 shows the PDR and EC in MIX-
MAB, LoRa-MAB, and Legacy LoRa in Scenario 3. As observed, the EC of the
proposed algorithm is lower than that in Scenario 1, while its PDR is
decreased. The reason is that having different values of TP to select by the
EDs results in reduced TP, leading to lower EC. However, sending packets with
less TP reduces PDR compared to Scenario 1, where all packets are sent with
maximum TP. From this figure, we also see that the PDR of the MIX-MAB is still
higher than the Legacy LoRa and LoRa-MAB. However, the EC of the MIX-MAB is a
little bit more than Legacy LoRa and LoRa-MAB. This is due to the goal of MIX-
MAB, to achieve the maximum PDR; thus, the ED transmits data with higher TP
leading to higher EC.

Figure 5: PDR & EC in MIX-MAB, LoRa-MAB, and LoRa in Scenario 3.

####  IV-B4 Scenario 4

Here we evaluate the MIX-MAB performance when the EDs have the most freedom to
select transmission parameters. Fig. 6 shows PDR and EC in MIX-MAB, LoRa-MAB,
and Legacy LoRa in Scenario 4. As shown, the PDR of the MIX-MAB is higher than
that in Scenario 1, while the EC decreased. It concludes that the proposed
algorithm achieves a high PDR when the EDs configure all three parameters. At
the same time, it consumes lower energy than the other three previous
scenarios. We also see that, similar to Scenario 2, the PDR of MIX-MAB is
higher than the Legacy LoRa and LoRa-MAB at the cost of higher EC.

Figure 6: PDR & EC in MIX-MAB, LoRa-MAB, and LoRa in Scenario 4.

####  IV-B5 Scenario 5

This scenario evaluates the number of transmitted packets’ effects on EC and
PDR of the proposed algorithm. The setting is similar to Scenario 4, except
that packet transmission frequency varies. As seen in Fig. 7, increasing the
frequency of sent packets leads to both improving the EC and the PDR due to
receiving more feedback from the NS, which in turn leads to an improvement in
the learning process. So, based on observations and considering the trade-off
between results, we have the best outcome for 1 packet per day, which is more
consistent with the IoT application and fulfills its requirements.

##  V Conclusion and Future Works

This paper focused on improving LoRaWAN’s resource allocation algorithm in
terms of PDR. We presented an RL-based resource allocation algorithm enabling
the LoRa EDs to configure their transmission parameters in a distributed
manner. Our proposed MIX-MAB algorithm combines two MAB schemes, i.e., SE and
EXP3, making the LoRa more efficient in interference management, leading to
higher network throughput. We evaluated the performance of the MIX-MAB and
compared it with the Legacy LoRa and LoRa-MAB mechanisms in five different
scenarios. Our simulation results show that the convergence time of the MIX-
MAB is half of LoRa-MAB while our solution achieves higher PDR than Legacy
LoRa and LoRa-MAB in all scenarios. We will reduce the computational overhead
for future works by proposing a new version of the MIX-MAB algorithm on the NS
side. Furthermore, we will deploy and evaluate our proposed algorithm for QoS
requirements of various industrial applications, including unmanned aerial
vehicles.

Figure 7: number of transmitted packets’ effects on EC & PDR.

## References

  * [1] S. Nižetić, pp. Šolić, D. González-de-Artaza, and L. Patrono, “Internet of Things (IoT): Opportunities, issues and challenges towards a smart and sustainable future,” _Journal of Cleaner Production_ , vol. 274, pp. 122877, 2020.
  * [2] J. Babaki, M. Rasti, and R. Aslani, “Dynamic spreading factor and power allocation of LoRa networks for dense IoT deployments,” _IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications_ , pp. 1-6, 2020.
  * [3] Sigfox, [Online]. Available: https://www.sigfox.com/, 2017.
  * [4] F. H. C. dos Santos Filho et al., ”Performance of LoRaWAN for handling telemetry and alarm messages in industrial applications”, Sensors, vol. 20, no. 11, pp. 3061, 2020.
  * [5] M. de Castro Tomé, P. H. J. Nardelli and H. Alves, ”Long-Range Low-Power Wireless Networks and Sampling Strategies in Electricity Metering,” in IEEE Transactions on Industrial Electronics, vol. 66, no. 2, pp. 1629-1637, Feb. 2019.
  * [6] F. Adelantado, X. Vilajosana, P. Tuset-Peiro, B. Martinez, J. Melia-Segui, and T. Watteyne, “Understanding the limits of LoRaWAN,” _IEEE Communications Magazine_ , vol. 55, no. 9, pp. 34-40, 2017.
  * [7] L. Alliance, “LoRaWAN 1.1 Specification,” _technical specification_ , [online]. Available: https://lora-alliance.org/resource_hub/lorawan-specification-v1-1/, 2017.
  * [8] D. Ta, K. Khawam, S. Lahoud, C. Adjih, and S. Martin, “LoRa-MAB: Toward an intelligent resource allocation approach for LoRaWAN,” _IEEE Global Communications Conference (GLOBECOM)_ , pp. 1-6, 2019.
  * [9] D. Ta, K. Khawam, S. Lahoud, C. Adjih, and S. Martin, “LoRa-MAB: A flexible simulator for decentralized learning resource allocation in IoT networks,” _12th IFIP Wireless and Mobile Networking Conference (WMNC)_ , pp. 55-62, 2019.
  * [10] S. Li, U. Raza, and A. Khan, “How agile is the adaptive data rate mechanism of LoRaWAN?,” _IEEE Global Communications Conference (GLOBECOM)_ , pp. 206-212, 2018.
  * [11] M. Bor, T. Voigt, and J.M. Alonso, “Do LoRa low-power wide-area networks scale?,” _MSWiM ’16 Proceedings of the 19th ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems_ , ACM Press, pp. 59-67, 2016.
  * [12] N. Aihara, K. Adachi, O. Takyu, M. Ohta, and T. Fujii, “Q-learning aided resource allocation and environment recognition in LoRaWAN with CSMA/CA,” _IEEE Access_ , vol. 7, pp. 152126-152137, 2019.
  * [13] R. Allesiardo, R. Féraud, and O Maillard, “The non-stationary stochastic multi-armed bandit problem,” _International Journal of Data Science and Analytics_ , vol. 3, no. 4, pp. 267-283, 2017.
  * [14] R.S. Sutton and A.G. Barto, _Reinforcement learning: An introduction_ , MIT press, 2018.


# Why haven’t the observed high-frequency QPOs been produced by (MHD) computer
simulations of black hole accretion disks?

Robert V. Wagoner and Celia R. Tandon  1  Dept. of Physics and KIPAC, Stanford
University, USA<EMAIL_ADDRESS>
###### Abstract

We compare some predictions of Wagoner & Tandon (2021) (WT) with the results
of the hydrodynamic and magnetohydrodynamic (MHD) simulations of Reynolds &
Miller (2009) (RM). It appears that the MHD simulations were not run for long
enough and the numerical damping was not small enough to produce the observed
high-frequency QPOs (and the g-mode seen in the hydro simulations).

##  1 Introduction

We do not consider the low-frequency QPOs, since their frequencies are not
stable. As an example of a possible high-frequency QPO (HFQPO) in a black hole
accretion disk, we choose the fundamental (axisymmetric) g-mode seen in the
hydro simulations of RM. We shall employ the dimensionless time (g-mode phase)
$\tau=2\pi f_{0}t\approx(2.8\times 10^{-2}c^{3}/GM)t$, where $f_{0}\approx
4.5\times 10^{-3}c^{3}/GM$ is the frequency of the g-mode. The run time of the
canonical MHD simulation MHD_1 of RM was $\tau(\mathrm{run})=1.1\times
10^{3}$.

The quantity plotted in Fig. 2 of RM is of order the turbulent stresses
(typically greater than the magnetic stresses within the eigenfunction of the
mode). We have assumed that the divergence of the turbulent stress tensor is
the main driver of the mode. From that Figure, it appears that the damping
time of the hydro runs was $\tau(\mathrm{damp})\approx 11$. We (and RM) assume
that this is due to numerical dissipation, in which case
$\tau(\mathrm{damp})\approx(\mbox{grid size})^{-1/2}$. (WT have shown that
‘turbulent viscosity’ does not operate on modes of oscillation.) The damping
time of the MHD run was about 10 times longer. This is consistent with the
fact that the grid sizes of runs HD3d_1 and MHD_1 were similar.

Goldreich & Kumar (1988) showed that convective turbulence excites (drives)
the p-modes in the sun. However, the origin of the damping of these modes is
still unknown (references in WT). The possibilities quoted do not include
turbulent viscosity.

##  2 Results

We take the damping rate parameter $D_{*}$ introduced by WT to be equal to
$1/\tau(\mathrm{damp})$. The amplitude of the mode should grow until a time of
order 1/(damping rate), as WT found. WT also found that if equilibrium
(roughly constant energy $E$ of the mildly nonlinear oscillations) is reached,
the short time average is

$\langle E\rangle\approx\langle A^{2}\rangle\sim
0.1(\eta\mathcal{M}^{2})^{2}/D_{*}.$ (1)

The (dimensionless) oscillation amplitude $A(\tau)$ is approximately equal to
the fluid displacement divided by the size of the smallest dimension of the
eigenfunction (of order the thickness $h$ of the accretion disk). The
parameter $\eta$ is the volume of the maximally coupled edd(ies) divided by
the volume of the eigenfunction, and $\mathcal{M}$ is the Mach number of those
eddies. Like the MHD (but not the hydro) runs of RM, our evolutions were begun
with $A=dA/d\tau=0$. Since equilibrium was achieved by RM
$[\tau(\mathrm{run})\gg\tau(\mathrm{damp})]$, we find from equation (1) [with
$D_{*}=1/\tau(\mathrm{damp})\sim 0.01$ for the MHD run] that for
representative values $\eta\sim\mathcal{M}\sim 0.1$, $\langle A^{2}\rangle\sim
10^{-5}$. This amplitude would presumably be undetectable in the MHD
simulations.

Employing the observed PSDs of black-hole binaries (BHBs) and narrow-line
Seyfert 1 AGNs (NLS1s), WT found that the visibility of a mode requires that

$\langle A^{2}\rangle\gtrsim[10\mbox{(BHBs)}-10^{2}\mbox{(NLS1s)}]/Q\>,$ (2)

where $1/Q$ is the fractional width of the QPO. Most observed HFQPOs have
$Q\lesssim 4$. Also, WT verified that nonlinearity reduced the value of $Q$.
We have assumed an axisymmetric mode of extent $\delta r\sim\delta z\sim
h(r)\sim 10^{-2}r$. We note that leakage of a mode into a traveling wave
(O’Neill et al., 2009) can reduce the required amplitude $A$.

WT found that a ‘blowout’ (onset of the effects of the lowest order
nonlinearity of the restoring force, terminating the run) occurred if the
amplitude reached a value $|A|=1$ (at $E=1/6$), which from equation (1)
occurred if $D_{*}\lesssim 0.1(\eta\mathcal{M}^{2})^{2}$. Therefore, in order
to reach a (more observable) blowout during a run,

$\tau(\mathrm{damp})=1/D_{*}\gtrsim 10\langle
A^{2}\rangle(\eta\mathcal{M}^{2})^{-2}\sim 10^{7}.$ (3)

Therefore, much longer run times
[$\tau(\mathrm{run})\gtrsim\tau(\mathrm{damp})$] are required.

##  3 Conclusions

From these results, it would appear that in order for simulations of black
hole accretion disks to produce the observed high-frequency QPOs, much smaller
grid sizes and consequently much longer runs must be employed. This result
applies to QPOs produced by normal modes of oscillation whose minimum extent
is of order the local thickness of the accretion disk. Topics for future
research then include: (1) How can this finite grid size source of damping be
reduced? (2) Why is our predicted damping within black hole accretion disks
with observed HFQPOs so small, and what is its origin?

We thank Marek Abramowicz and Cole Miller for comments. CT acknowledges
financial support from the Stanford Physics Department Summer Undergraduate
Research Program. We would like to thank Stanford University and the Stanford
Research Computing Center for providing computational resources on the
Sherlock cluster and support.

## References

  * Goldreich & Kumar (1988) Goldreich, P. & Kumar, P. 1988, ApJ, 326, 462
  * O’Neill et al. (2009) O’Neill, S.M., Reynolds, C.S. & Miller, M.C. 2009, ApJ, 693, 1100
  * Reynolds & Miller (2009) Reynolds, C.S. & Miller, M.C. 2009, ApJ, 692, 869
  * Wagoner & Tandon (2021) Wagoner, R.V. & Tandon, C.R. 2021, ApJ, 920: 114