# Emergent spin and clock variable in Bianchi type-I
quantum cosmology

Vishal and Malay K. Nandy vishal.phiitg<EMAIL_ADDRESS>Department of Physics, Indian Institute of Technology Guwahati, Guwahati 781
039, India

(February 20, 2024)

###### Abstract

We consider the Bianchi type-I model of the universe in the Wheeler-DeWitt
quantization scheme with the matter degree of freedom represented by a scalar
field. As a consequence, the quantum mechanical equation of the universe is
obtained in the minisuperspace consisting of the Misner variables and the
scalar field. Employing Dirac factorization, we find that the volume parameter
makes a suitable choice for the clock variable whereas the matter clock leads
to various inconsistencies. We further find that the minisuperspace orbital
angular momentum operator does not commute with the Hamiltonian in the Dirac-
type equation. We therefore find the missing part of the angular momentum
whereby the total angular momentum commutes with the Hamiltonian. We interpret
this missing part as the spin of the quantum universe during its early stage
of evolution. The emergence of the three-component spin vector is owing to the
presence of anisotropy in the Bianchi type-I model of the universe which is
absent in the quantization of isotropic models.

##  1 Introduction

The observed large-scale distribution of matter in our universe is homogenous
as well as isotropic while the universe is expanding at the same rate in every
direction. The assumption of homogeneity and isotropy is further strengthened
by the observed cosmic microwave background (CMB) radiation signifying
homogeneity and isotropy even at the period of the last scattering. Thus a
first step towards modeling the universe involves the Friedmann-Lemaitre-
Robertson-Walker metric [1, 2, 3, 4],
$ds^{2}=-dt^{2}+a^{2}(t)\,(dx^{2}+dy^{2}+dz^{2})$ (for a flat universe), where
$a(t)$ is the scale factor and $t$ the cosmic time.

Although this isotropic model has been able to explain most of the macroscopic
features of the universe, the anisotropies in the CMB (as well as in the
large-scale matter distribution) call for anisotropic models. In particular,
the anisotropies in the CMB power spectrum have been obtained by considering
anisotropic fluctuations in the metric [5, 6] so that the CMB anisotropies are
explained as quantum mechanical perturbations in the inflationary era which
serve to play the role of seeds in forming the anisotropic large-scale
structure of the universe. The quantum nature of the primordial perturbations
is an important feature that motivates the question of what might be the role
of quantum gravity effects around the time of the creation of the universe.

Quantum gravitational corrections to the scalar perturbations were considered
in the leading order in a universe undergoing de Sitter expansion [7]. In
addition, the quantum gravitational corrections to the scalar and tensor
perturbations in the inflationary era were taken into account in the Wheeler-
DeWitt (WDW) quantization scheme. It was found that the largest scales undergo
an enhancement in the power spectrum [8].

This motivates the question as to what might be the role of anisotropies in
quantum gravity in the early stage of the universe. Anisotropies are expected
to become prominent at early times and they need to be taken care of while
constructing a theory of the universe.

Bianchi universes [9, 10, 11, 12] are the simplest (classical) anisotropic
models which are homogenous but do not respect spatial isotropy. In this work,
we consider the Bianchi type-I model of the universe described by the metric

$ds^{2}=-Ndt^{2}+a^{2}(t)dx^{2}+b^{2}(t)dy^{2}+c^{2}(t)dz^{2}$ (1)

where the three unequal scale factors imply expansion of the universe with
unequal rates in the three spatial directions.

We consider this anisotropic model in the minisuperspace representation of the
Wheeler-DeWitt framework of quantum gravity [13, 14] to study the behavior of
the universe in the early stage of evolution.

The Wheeler-DeWitt equation was obtained in the ADM formalism of general
relativity [15] which is based on decomposition of the spacetime into
constant-time spatial hypersurfaces. This allows for the study of time
evolution of the spacetime geometry as a Hamiltonian system. Quantization is
achieved by promoting conjugate variables to operators in the Hamiltonian
constraint to arrive at the WDW equation, $\hat{H}\Psi=0$. This quantum
gravity framework has been employed to investigate the initial quantum
tunneling of the universe [16, 17, 18] as well as the cosmological singularity
and CMB anisotropy in different models of the universe [19, 20].

The Wheeler-DeWitt equation for FLRW and Bianchi minisuperspace models takes
the form of the Klein-Gordon equation which suffers from the problem of
negative probability density. To avoid this problem, Dirac factorization was
employed to obtain a Dirac-type equation [21, 22]. Upon Dirac factorization of
the Wheeler-DeWitt equation for the vacuum Bianchi class A models, it was
shown that the universe had a spin-like degree of freedom in the early quantal
stage [22].

In the present work, we incorporate a scalar field as a matter degree of
freedom to the Bianchi type-I model in order to study the angular momentum of
the early quantum state of the universe.

We find that the equation of continuity for the probability density and
hermiticity of the Hamiltonian restrict the choice of the time variable. Upon
choosing a legitimate time variable, we show that the orbital angular
momentum, constructed from the minisuperspace variables, does not commute with
the Hamiltonian. This signifies the existence of a missing part of the angular
momentum. This missing part turns out to be originating from the spin degree
of freedom. We demonstrate that the total angular momentum, orbital plus spin,
does commute with the Hamiltonian. Thus the spin of the universe naturally
emerges from the law of conservation of angular momentum.

The outline of this paper is as follows. In section 2, we obtain the Wheeler-
DeWitt equation for the Bianchi type-I universe in terms of Misner variables.
In section 3, we factorize the WDW equation by Dirac factorization and obtain
Schrodinger-type equations for different choices for the clock variable. It is
shown that the volume parameter is suitable for the clock variable. In section
4, we calculate the commutation relation of the orbital angular momentum with
the Hamiltonian and identify the missing part of the angular momentum as the
spin of the quantum universe. In section 5, we summarise and discuss our
results.

##  2 Wheeler-DeWitt quantization

For a spatially homogenous and anisotropically expanding universe, the
Bianchi-I model is the simplest cosmological model. In order to obtain the
Wheeler-DeWitt equation in this anisotropic model, we start with the
Hamiltonian formalism of general relativity and write the Einstein-Hilbert
action in ADM decomposed form,

$S_{G}=\frac{1}{16\pi G}\int
dt\,d^{3}x\;N\sqrt{h}\;({}^{3}\\!R+K^{ij}K_{ij}-K^{2}),$ (2)

where the Ricci scalar of the four-dimensional spacetime is expressed in terms
of the extrinsic curvature $K_{ij}$ and the Ricci scalar ${}^{3}\\!R$ on
three-dimensional hypersurfaces [23]. Here $h$ is the determinant of the
metric $h_{ij}$ defined on three-dimensional hypersurfaces and
$K=K^{ij}h_{ij}$ is the trace of $K^{ij}$. For a Bianchi type-I universe,
$K_{ij}=-\frac{\dot{h}_{ij}}{2N}=-\frac{1}{N}{\rm
diag}(a\dot{a},b\dot{b},c\dot{c})$ and ${}^{3}\\!R=0$.

We use Misner variables $(\alpha,\beta_{+},\beta_{-})$, defined by
$a=e^{\alpha+\beta_{+}+\sqrt{3}\beta_{-}}$,
$b=e^{\alpha+\beta_{+}-\sqrt{3}\beta_{-}}$ and $c=e^{\alpha-2\beta_{+}}$ [24,
25] so that $e^{\alpha}=(abc)^{1/3}$ contains the volume information and the
anisotropy parameters $\beta_{+}$ and $\beta_{-}$ determine the shape of the
universe.

Integrating over the spatial part in the action generates the fiducial volume
$V_{0}$ of the three-dimensional space and we choose units such that
$3V_{0}/4\pi G=1$. We consider a massless scalar field $\phi$ to represent the
matter part in the universe. Consequently, the total action $S=S_{G}+S_{m}$,
comprising of the gravitational and matter parts, is given by

$S_{G}=\int
dt\;\frac{1}{2N}\;e^{3\alpha}(-\dot{\alpha}^{2}+\dot{\beta}_{+}^{2}+\dot{\beta}_{-}^{2})$
(3) $S_{m}=\int dt\;\frac{1}{2N}e^{3\alpha}\dot{\phi^{2}}$ (4)

Corresponding to the coordinate variables $(\alpha,\beta_{+},\beta_{-},\phi)$,
we obtain the total Hamiltonian

$H=\frac{Ne^{-3\alpha}}{2}(-p_{\alpha}^{2}+p_{+}^{2}+p_{-}^{2}+p_{\phi}^{2})$
(5)

in terms of the conjugate momenta $(p_{\alpha},p_{+},p_{-},p_{\phi})$. The
classical Hamiltonian constraint reads

$p_{\phi}^{2}-p_{\alpha}^{2}+p_{+}^{2}+p_{-}^{2}=0.$ (6)

Now we proceed to quantize the system by promoting the phase space variables
to operators,
($\hat{q}_{a}=q_{a}\;,\hat{p}_{a}=-i\hbar\frac{\partial}{\partial q_{a}})$ for
$a=(\alpha,\beta_{+},\beta_{-},\phi)$. The resulting constraint now acts on
the wavefunction $\Psi(\alpha,\beta_{+},\beta_{-},\phi)$ and yields the
Wheeler-DeWitt equation

$\left(\frac{\partial^{2}}{\partial\alpha^{2}}-\frac{\partial^{2}}{\partial\beta_{+}^{2}}-\frac{\partial^{2}}{\partial\beta_{-}^{2}}-\frac{\partial^{2}}{\partial\phi^{2}}\right)\;\Psi(\alpha,\beta_{+},\beta_{-},\phi)=0.$
(7)

This equation is of the Klein-Gordon type and we encounter the problem of
negative probability density.

##  3 Dirac factorization and clock variable

For the resolution of the negative probability density following the above
equation, we use Dirac factorization in this section.

###  3.1 Matter field $\phi$ as clock variable

Upon choosing the matter field $\phi$ as the clock variable we rewrite the
Wheeler-DeWitt equation (7) as

$-\frac{\partial^{2}}{\partial\phi^{2}}\;\Psi(\vec{r},\phi)=\Theta\;\Psi(\vec{r},\phi)$
(8) $\Theta=(p_{\alpha}^{2}-p_{+}^{2}-p_{-}^{2})$ (9)

where $\vec{r}=(\alpha,\beta_{+},\beta_{-})$.

To obtain a Dirac-type equation, we take the square root on both sides of the
equation (8), giving

$\pm
i\frac{\partial}{\partial\phi}\Psi(\vec{r},\phi)=\sqrt{\Theta}\;\Psi(\vec{r},\alpha)$
(10)

with the square root of the Hamiltonian $\Theta$ expressed as

$\sqrt{\Theta}=(i\sigma_{1}\hat{p}_{+}+i\sigma_{2}\hat{p}_{-}+\sigma_{3}\hat{p}_{\alpha})$
(11)

where $\sigma_{1},\sigma_{2},\sigma_{3}$ are the Pauli matrices.

We note that the matter field $\phi$ plays the role of time in the Schrodinger
type equation (10). However, it is clearly seen that the corresponding
Hamiltonian $\sqrt{\Theta}$ given by equation (11) does not meet the condition
of hermiticity.

To find the equation of continuity, we rewrite the above equation in the
coordinate representation

$\pm
i\frac{\partial\Psi}{\partial\phi}-\sigma_{1}\frac{\partial\Psi}{\partial\beta_{+}}-\sigma_{2}\frac{\partial\Psi}{\partial\beta_{-}}+i\sigma_{3}\frac{\partial\Psi}{\partial\alpha}=0,$
(12)

and its Hermitian conjugate

$\mp
i\frac{\partial\Psi^{\dagger}}{\partial\phi}-\frac{\partial\Psi^{\dagger}}{\partial\beta_{+}}\sigma_{1}-\frac{\partial\Psi^{\dagger}}{\partial\beta_{-}}\sigma_{2}-i\frac{\partial\Psi^{\dagger}}{\partial\alpha}\sigma_{3}=0.$
(13)

The above two equations lead to the equation of continuity:

$\displaystyle
i\frac{\partial}{\partial\phi}(\Psi^{\dagger}\Psi)+\left(\Psi^{\dagger}\sigma_{1}\frac{\partial\Psi}{\partial\beta_{+}}-\frac{\partial\Psi^{\dagger}}{\partial\beta_{+}}\sigma_{1}\Psi\right)$
$\displaystyle+\left(\Psi^{\dagger}\sigma_{2}\frac{\partial\Psi}{\partial\beta_{-}}-\frac{\partial\Psi^{\dagger}}{\partial\beta_{-}}\sigma_{2}\Psi\right)$
$\displaystyle-i\left(\Psi^{\dagger}\sigma_{3}\frac{\partial\Psi}{\partial\alpha}-\frac{\partial\Psi^{\dagger}}{\partial\alpha}\sigma_{3}\Psi\right)=0.$
(14)

This equation deviates from the equation of continuity since the last line
contains an imaginary contribution. This violates the law of conservation of
probability.

To obtain the dispersion relation, we consider the plane wave solution in the
form of a two-component spinor,

$\Psi(r,\phi)=e^{i\vec{K}.\vec{r}}e^{i\omega\phi}\left(\begin{array}[]{cc}\psi_{1}\\\
\psi_{2}\\\ \end{array}\right),$ (15)

where
$\vec{r}=(\beta_{+}\;\hat{e}_{+}+\beta_{-}\;\hat{e}_{-}+\alpha\;\hat{e}_{\alpha})$
and
$\vec{K}=(k_{+}\;\hat{e}_{+}+k_{-}\;\hat{e}_{-}+k_{\alpha}\;\hat{e}_{\alpha})$.
Employing this plane wave solution in Eq (10), we get the dispersion relation

$\omega=\pm\sqrt{k_{\alpha}^{2}-k_{+}^{2}-k_{-}^{2}}.$ (16)

Thus, for modes with $k_{+}^{2}+k_{-}^{2}>k_{\alpha}^{2}$, this dispersion
relation admits imaginary frequency solutions which is inconsistent with the
plane wave solution.

From the various inconsistencies encountered above, we conclude that the
scalar field $\phi$ is not a good choice for the clock variable in a Bianchi
type-I universe.

###  3.2 Volume parameter $\alpha$ as clock variable

We shall now choose the volume parameter $\alpha$ as the clock variable. We
rewrite the Wheeler-DeWitt equation (7) as

$-\frac{\partial^{2}}{\partial\alpha^{2}}\;\Psi(\vec{r},\alpha)=\Omega\;\Psi(\vec{r},\alpha)$
(17) $\Omega=(p_{+}^{2}+p_{-}^{2}+p_{\phi}^{2})$ (18)

where $\vec{r}=(\beta_{+},\beta_{-},\phi)$.

Repeating the steps as before, we factorize this equation using Pauli matrices
and obtain a Schrodinger-type equation

$\pm
i\frac{\partial}{\partial\alpha}\Psi(\vec{r},\alpha)=\sqrt{\Omega}\;\Psi(\vec{r},\alpha),$
(19)
$\sqrt{\Omega}=(\sigma_{1}\hat{p}_{+}+\sigma_{2}\hat{p}_{-}+\sigma_{3}\hat{p}_{\phi}).$
(20)

In the above Schrodinger equation, the operator $\sqrt{\Omega}$ acts as a
Hamiltonian operator which is Hermitian since the momentum operators are
themselves Hermitian and they commute with the Pauli matrices.

We can now proceed to obtain the continuity equation and check whether the law
of conservation of probability is respected.

Rewriting equation (19) as

$\pm
i\frac{\partial\Psi}{\partial\alpha}+i\sigma_{1}\frac{\partial\Psi}{\partial\beta_{+}}+i\sigma_{2}\frac{\partial\Psi}{\partial\beta_{-}}+i\sigma_{3}\frac{\partial\Psi}{\partial\phi}=0,$
(21)

and its Hermitian conjugate being

$\mp
i\frac{\partial\Psi^{\dagger}}{\partial\alpha}-i\frac{\partial\Psi^{\dagger}}{\partial\beta_{+}}\sigma_{1}-i\frac{\partial\Psi^{\dagger}}{\partial\beta_{-}}\sigma_{2}-i\frac{\partial\Psi^{\dagger}}{\partial\phi}\sigma_{3}=0,$
(22)

we obtain the continuity equation:

$\displaystyle\frac{\partial}{\partial\alpha}(\Psi^{\dagger}\Psi)+\frac{\partial}{\partial\beta_{+}}(\Psi^{\dagger}\sigma_{1}\Psi)$
$\displaystyle+\frac{\partial}{\partial\beta_{-}}(\Psi^{\dagger}\sigma_{2}\Psi)+\frac{\partial}{\partial\phi}(\Psi^{\dagger}\sigma_{3}\Psi)=0.$
(23)

This equation conforms with the continuity equation
$\frac{\partial\rho}{\partial t}+\vec{\nabla}\cdot\vec{J}=0$, with the
identification $\rho=\Psi^{\dagger}\Psi$ and
$\vec{J}=\Psi^{\dagger}\vec{\sigma}\Psi$. Thus this continuity equation meets
the requirement of conservation of probability.

To analyze further, we take a plane wave solution for the two-component spinor
$\Psi$ as

$\Psi(\vec{r},\alpha)=e^{i\vec{K}.\vec{r}}e^{i\omega\alpha}\left(\begin{array}[]{cc}\psi_{1}\\\
\psi_{2}\\\ \end{array}\right),$ (24)

with wave vector
$\vec{K}=(k_{+}\;\hat{e}_{+}+k_{-}\;\hat{e}_{-}+k_{\phi}\;\hat{e}_{\phi})$ and
$\vec{r}=(\beta_{+}\;\hat{e}_{+}+\beta_{-}\;\hat{e}_{-}+\phi\;\hat{e}_{\phi})$.
Employing this solution in equation (19) leads to the dispersion relation

$\omega=\pm\sqrt{k_{+}^{2}+k_{-}^{2}+k_{\phi}^{2}},$ (25)

which supports solutions with only real-valued frequencies.

From the above consistencies, we thus conclude that the volume parameter
$\alpha$ is a suitable clock parameter for the Bianchi type-I universe in the
early stage of its quantum evolution.

##  4 Angular momentum and spin

We define the orbital angular momentum according to
$\vec{L}=\vec{r}\times\vec{p}$ with “position” vector
$\vec{r}=(\beta_{+}\;\hat{e}_{+}+\beta_{-}\;\hat{e}_{-}+\phi\;\hat{e}_{\phi})$
and the “momentum” vector
$\vec{p}=(p_{+}\;\hat{e}_{+}+p_{-}\;\hat{e}_{-}+p_{\phi}\;\hat{e}_{\phi})$, so
that

$\displaystyle\vec{L}=(\beta_{-}p_{\phi}-p_{-}\phi)\;\hat{e}_{+}-(\beta_{+}p_{\phi}-p_{+}\phi)\;\hat{e}_{-}$
$\displaystyle+(\beta_{+}p_{-}-p_{+}\beta_{-})\;\hat{e}_{\phi}$ (26)

With the three components of this “orbital” angular momentum operator, we
obtain the following commutation relations with the Hamiltonian
$\sqrt{\Omega}$,

$\displaystyle[L_{+},\sqrt{\Omega}]$
$\displaystyle=i(\sigma_{2}p_{\phi}-\sigma_{3}p_{-}),$ (27)
$\displaystyle[L_{-},\sqrt{\Omega}]$
$\displaystyle=i(\sigma_{3}p_{+}-\sigma_{1}p_{\phi}),$
$\displaystyle[L_{\phi},\sqrt{\Omega}]$
$\displaystyle=i(\sigma_{1}p_{-}-\sigma_{2}p_{+}).$

Thus the “orbital” angular momentum $\vec{L}$ does not commute with the
Hamiltonian $\sqrt{\Omega}$. This implies that $\vec{L}$ is not a constant of
motion.

In order to find the missing part of the angular momentum, we compute the
commutation relations of the Pauli matrices with the Hamiltonian
$\sqrt{\Omega}$ accounting for the fact that the Pauli matrices anticommute
among themselves. We thus obtain

$\displaystyle[\sigma_{1},\sqrt{\Omega}]$
$\displaystyle=2i(\sigma_{3}p_{-}-\sigma_{2}p_{\phi}),$ (28)
$\displaystyle[\sigma_{2},\sqrt{\Omega}]$
$\displaystyle=2i(\sigma_{1}p_{\phi}-\sigma_{3}p_{+}),$
$\displaystyle[\sigma_{3},\sqrt{\Omega}]$
$\displaystyle=2i(\sigma_{2}p_{+}-\sigma_{1}p_{-}).$

Combining the above two sets of commutation relations, we can easily see that
the sum $\vec{L}+\frac{1}{2}\vec{\sigma}$ commutes with the Hamiltonian
$\sqrt{\Omega}$, that is

$[\vec{L}+\frac{1}{2}\vec{\sigma},\sqrt{\Omega}]=0.$ (29)

Since the additional contribution $\frac{1}{2}\vec{\sigma}$ to the angular
momentum originates from the Pauli matrices, it can be interpreted as the spin
of the universe. Thus the Bianchi type-I universe behaves like a quantum
particle with spin one-half in the early stage of its quantum evolution.

##  5 Discussion and conclusion

In this paper, we quantized the Bianchi type-I universe containing a matter
field in the framework of the Wheeler-DeWitt quantization scheme. This gives
rise to a Klein-Gordon type equation in the Misner variables which naturally
suffers from the problem of negative probability density.

To circumvent this problem, we first followed the usual choice of making the
matter field as a clock variable and factorized the WDW equation to obtain a
Dirac-type equation. However, this equation suffers from three difficulties:
(a) the Hamiltonian in the Dirac-type equation does not meet the required
condition of hermiticity, (b) the equation of continuity does not respect the
law of conservation of probability, and (c) the plane wave solution allows for
imaginary-valued frequencies. From these inconsistencies, we conclude that the
scalar field does not make a good choice for the clock variable.

On the other hand, when we regarded the volume parameter $\alpha$ as the clock
variable, all the above inconsistencies were found to be resolved: (a) the
Hamiltonian in the Dirac-type equation satisfies the required condition of
hermiticity, (b) the equation of continuity respects the law of conservation
of probability, and (c) the plane wave solution allows for only real-valued
frequencies. From these consistencies, we conclude that the volume parameter
$\alpha$ is a good choice for the clock variable.

With the choice of volume parameter $\alpha$ as the clock variable, we
obtained the commutation relation of the angular momentum operator $\vec{L}$
with the Hamiltonian $\sqrt{\Omega}$ of the Dirac type equation. However, we
found that the orbital angular momentum $\vec{L}$ is not a constant of motion
as it does not commute with the Hamiltonian $\sqrt{\Omega}$.

To find the missing angular momentum, we considered commutation relations of
the Pauli matrices with the Hamiltonian $\sqrt{\Omega}$. We found that the
combination $\vec{L}+\frac{1}{2}\vec{\sigma}$ commutes with Hamiltonian. The
additional contribution $\frac{1}{2}\vec{\sigma}$ to the angular momentum can
be interpreted as the spin of the universe since it originates from the Pauli
matrices. We may say that the universe behaves like a quantum particle with
spin one-half in the early stage of its quantum evolution.

The above conclusions rest on the Bianchi type-I model containing a matter
field where the anisotropy of the universe is built into the model a priori.
Since anisotropy is expected to be prominent during the early phase of the
universe owing to the dominance of quantum mechanical fluctuations, these
conclusions have the potential of realistic significance. It is clear that the
emergence of the three-component spin vector is owing to the presence of
anisotropy in the universe which is absent in the quantization of the
isotropic models.

## Acknowledgement

Vishal is supported through a Research Fellowship from the Ministry of
Education (MoE), Government of India.

## References

  * [1] Alexander Friedman.  Über die krümmung des raumes.  Zeitschrift für Physik, 10(1):377–386, 1922.
  * [2] Georges Lemaître.  Un univers homogène de masse constante et de rayon croissant rendant compte de la vitesse radiale des nébuleuses extra-galactiques.  Annales de la Société Scientifique de Bruxelles, A47, p. 49-59, 47:49–59, 1927.
  * [3] Howard Percy Robertson.  Kinematics and world-structure.  The Astrophysical Journal, 82:284, 1935.
  * [4] Arthur Geoffrey Walker.  On milne’s theory of world-structure.  Proceedings of the London Mathematical Society, 2(1):90–127, 1937\.
  * [5] Karim A Malik and David Wands.  Cosmological perturbations.  Physics Reports, 475(1-4):1–51, 2009.
  * [6] Viatcheslav F Mukhanov, Hume A Feldman, and Robert Hans Brandenberger.  Theory of cosmological perturbations.  Physics reports, 215(5-6):203–333, 1992.
  * [7] Alexander Yu Kamenshchik, Alessandro Tronconi, and Giovanni Venturi.  Quantum cosmology and the inflationary spectra from a nonminimally coupled inflaton.  Physical Review D, 101(2):023534, 2020.
  * [8] David Brizuela, Claus Kiefer, and Manuel Krämer.  Quantum-gravitational effects on gauge-invariant scalar and tensor perturbations during inflation: The de sitter case.  Physical Review D, 93(10):104035, 2016.
  * [9] LD Landau and EM Lifshitz.  The classical theory of fields, course of theoretical physics.  Pergamon Press, Oxford, 1971.
  * [10] George FR Ellis and Henk Van Elst.  Cosmological models: Cargese lectures 1998.  Theoretical and Observational Cosmology, pages 1–116, 1999.
  * [11] Michael P Ryan and Lawrence C Shepley.  Homogeneous relativistic cosmologies, volume 65.  Princeton University Press, 2015.
  * [12] CW Misner.  Magic without magic.  edited by, J. Klauder, Freeman, San Francisco, page 441, 1972.
  * [13] J. A. Wheeler.  Relativity groups and topology.  In Relativity groups and Topology, 1963 Les Holches Lectures. Gordon and Breach, New York, 1970.
  * [14] Bryce S DeWitt.  Quantum theory of gravity. i. the canonical theory.  Physical Review, 160(5):1113, 1967.
  * [15] Richard Arnowitt, Stanley Deser, and Charles W Misner.  Canonical variables for general relativity.  Physical Review, 117(6):1595, 1960.
  * [16] James B Hartle and Stephen W Hawking.  Wave function of the universe.  Physical Review D, 28(12):2960, 1983.
  * [17] Alexander Vilenkin.  Quantum cosmology and the initial state of the universe.  Physical Review D, 37(4):888, 1988.
  * [18] Alexander Vilenkin.  Creation of universes from nothing.  Physics Letters B, 117(1-2):25–28, 1982.
  * [19] Claus Kiefer, Nick Kwidzinski, and Dennis Piontek.  Singularity avoidance in Bianchi I quantum cosmology.  The European Physical Journal C, 79:1–12, 2019.
  * [20] Claus Kiefer and Manuel Kraemer.  Quantum gravitational contributions to the cosmic microwave background anisotropy spectrum.  Physical review letters, 108(2):021301, 2012.
  * [21] David A Craig and Parampreet Singh.  Consistent probabilities in Wheeler-Dewitt quantum cosmology.  Physical Review D, 82(12):123526, 2010.
  * [22] Hidetomo Yamazaki and Tetsuya Hara.  Dirac decomposition of Wheeler-Dewitt equation in the Bianchi class A models.  Progress of Theoretical Physics, 106(2):323–337, 2001.
  * [23] Eric Poisson.  An advanced course in general relativity, volume 174.  2002\.
  * [24] Charles W Misner.  Quantum cosmology. I.  Physical Review, 186(5):1319, 1969.
  * [25] Charles W Misner.  Classical and quantum dynamics of a closed universe.  In Relativity: Proceedings of the Relativity Conference in the Midwest, held at Cincinnati, Ohio, June 2–6, 1969, pages 55–79. Springer, 1970\.


Table 4.10: Comparison of our different curriculum learning approaches of our
previous work Schraner [2020], the average return over 100 episodes on the 11
vs 11 hard environment is reported.

In table 4.11 we report our results obtained in our previous work [Schraner,
2019] by training an agent with IMPALA in a single Google Football
environments. With our curriculum learning approach, we can match and surpass
the results of our previous work by only using a fifth of the samples.
Compared to the baselines published with the Google Football environment
[Kurach et al., 2019] we archived better results on the easy environments
(Empty Goal, Run to Score, Pass and Shoot with Keeper) but slightly worse on
the difficult environments (Run, Pass and Shoot with Keeper, Easy Counter-
Attack, Hard Counter-Attack, and 11 vs. 11 lazy).

Scenario | Ours@50M | Google Baseline@50M
---|---|---
Empty Goal Close | 0.99 | 1.0
Empty Goal | 0.84 | 0.86
Run to Score | 0.88 | 0.88
Pass and Shoot with Keeper | 0.0 | 0.66
Run, Pass and Shoot with Keeper | -0.05 | 0.18
Easy Counter-Attack | 0.0 | 0.5
Hard Counter-Attack | -0.02 | 0.2
11 vs. 11 lazy | 0.01 | 0.2
11 vs. 11 easy | -1.59 | -0.35
11 vs. 11 medium | -1.6 | -0.79
11 vs. 11 hard | -2.78 | -1.16
Table 4.11: Results on the Football Academy scenarios obtained in our previous
work [Schraner, 2019] and the Google baseline [Kurach et al., 2019] with the
IMPALA algorithm. The reported results is the average reward of 100 episodes
after a training on 50 million frames.

## Chapter 5 Discussion

Recent work in curriculum learning for rl has shown promising results in
improving sample efficiency and asymptotic performance in challenging
environments. Most of these works focused on automatic task generation e.g.,
by using procedural generated environments [Wang et al., 2019], changing the
initial state distribution to easy starts [Florensa et al., 2017], creating
additional synthetic goals to guide the student [Racanière et al., 2020] or
generating auxiliary tasks [Riedmiller et al., 2018]. Using self-play to
create a curriculum was a core idea in AlphaStar [Vinyals et al., 2019].
Methods to automatically select tasks in a teacher-student setting are also
popular. Most of those methods use heuristics or sampling-based approaches to
select the tasks [Matiisen et al., 2019, Foglino et al., 2019a, b, Narvekar et
al., 2017, Narvekar and Stone, 2019, Kanitscheider et al., 2021]. In our
thesis, we propose a new approach to formulate the task sequencing problem as
an MDP and train a teacher rl agent to perform the task sequencing while
training the student simultaneously.

Our experiments show that performing task sequencing with an rl teacher agent
is superior to heuristic-based task sequencing. On the grid world environment,
we were able to outperform all baselines significantly. Our approach failed to
increase the asymptotic performance on most of the MiniGrid when compared to
tablua-rasa rl. Training an agent directly in easy grid world environments
leads to better performance compared to curriculum learning approaches. Our
Google Football experiments improved the asymptotic performance on easy
environments such as Empty Goal, Run to Score, and Pass and Shoot with Keeper.
We believe that the root cause of failing on challenging environments lies in
the teacher’s reward signal. Curriculum learning with the source task reward
signal is in some cases beneficial for more challenging environments such as
DoorKey-16x16, but it is difficult to target the teacher agent towards
specific environments. Using a targeted reward signal to guide the teacher
agent towards more challenging environments is often insufficient, as the
teacher faces a sparse reward signal in that setting. When providing the
teacher with a sparse reward signal, our teacher-student settings collapses
towards a uniform sampling approach.

The teacher agent’s learning curves did not converge after 1’000 steps.
Training both agents beyond 1’000 CMDP steps might increase the asymptotic
performance and help the student agent solve more challenging environments.
This would increase the asymptotic performance at the cost of sample
efficiency. We fixed the CMDP and student steps to match the number of
training steps used in our no curriculum learning experiments to carry out a
fair comparison. When specifying the number of teacher and student steps, we
have to balance the teacher’s and student’s performance. Using fewer teacher
steps might not be sufficient for the teacher to learn how to perform task
sequencing but would give the student more time to converge in the selected
environment. Using more teacher steps is beneficial for the teacher, but on
the other hand, the student has less time to learn the selected task. Using
fewer student steps might lead to a noisy reward signal for the teacher agent.
In future work, it would be interesting to analyze the importance and effect
of those values. One could also train the student until convergence or make
the number of student steps part of the teacher’s action space.

When comparing the three transfer methods policy, reward and combined, it
turns out that the policy transfer leads to the best results. Reward transfer
is volatile for both the rl teacher agents and the baselines. Combining reward
transfer with policy transfer fails. After a few CMDP steps, the added reward
bonus covers the environment’s reward, forcing the student agent to maximize
its reward signal.

We compared six partial observable state representations and one approximately
fully observable state representation (PCA) for the CMDP state. The PCA
representation type failed utterly. There are other ways to represent neural
network weights in a compact embeddings, such as autoencoders or network
distillations. In future work, we can investigate other methods to represent
the student weights. The results of the six partial observable representations
did not differ too much from each other. Overall the PTR, LP, and EMA
representation resulted in the highest asymptotic rewards and the most general
student agents for the MiniGrid environments. In our Google Football
environment RH, PTR, LP and ALP achived the highest asymptotic reward and
percentage of solved environments. In that environment tuning the $\alpha$
value for the EMA representation might improve the results.

The proposed curriculum learning approach is beneficial for multi-task
reinforcement learning. The reward across all tasks and the overall percentage
of solved environments are significantly higher compared to our baselines or
when training an agent on a single environment.

We investigated the sample efficiency of our teacher-student setup in three
grid world environments. The sample efficiency improves in the Empty-16x16 and
DoorKey16x16 environment but remains similar in the FourRooms environment. We
discovered that training in the proposed curriculum setup is noisy regarding
the student’s performance throughout training. The sample efficiency on the
Google Football environments increased considerably. By only using one-fifth
of the samples, we almost matched or surpassed the performance on 8 out of 11
environments compared to directly training in those environments. Training
agents for the 11 vs. 11 football game remains difficult with our proposed
curriculum learning approach. The asymptotic performance remained the same
compared to training directly in an environment. Using a careful, manually
defined curriculum improves the asymptotic performance compared to our
teacher-student setup. Our previous work concluded that defining such a
curriculum by hand requires a lot of tweaking and domain knowledge.

In this work, we analyzed different knowledge transfer methods, teacher
observation types, and reward signals. Using policy transfer combined with the
PTR, LP, or EMA representation and the source task reward reward signal, we
observe the following:

  * •

The generality of our agent improves.

  * •

The asymptotic performance in some environments increases compared to using no
curriculum.

  * •

Training an rl agent to perform task sequencing is superior over our
heuristic-based baselines.

  * •

The sample efficiency compared to no curriculum does increase in one out of
three analyzed grid world environments.

  * •

The sample efficiency compared to no curriculum does increase by a factor of
five on 8 out of 11 Google Football environments.

The improvement in the generality and sample efficiency with curriculum has to
be investigated more carefully in future research. Another weakness of our
work is the limited hyperparameter search. The effect of training teachers or
students for more steps remains unclear. Other types of rl algorithms such as
DQN or traditional tabular methods for the teacher agent have to be tested.
The teacher network architecture should be tuned further. Our agents trained
without curriculum learning on the grid world environments results in zero
rewards for hard environments. To perform more meaningful analysis, we propose
to tune those baseline agents and then transfer them to the teacher-student
setting. It remains unclear if other PPO hyperparameters for the teacher
training would improve performance and how well these parameters transfer
across all methods.

## Chapter 6 Conclusion

In this thesis, we successfully layout a teacher-student curriculum learning
approach with automated online task sequencing to improve the generality and
sample efficiency of our rl agents. Our CMDP framework allows training a
teacher and a student agent simultaneously, where the teacher selects tasks
for the student. The student’s asymptotic performance does not increase for
most grid world and Google Football environments compared to non-cl agents. We
evaluated multiple transfer methods, teacher reward signals, and CMDP
observations on the MiniGrid and the Google Research Football environment.
This work is a step towards successful multi-task reinforcement learning
agents.

We found that knowledge transfer through policy transfer works well and is
robust across a variety of teachers. In future work, it might be interesting
to experiment with additional knowledge transfer methods such as macro actions
[Vezhnevets et al., 2016] and task models [Shao et al., 2019]. The choice of
teacher observation is not as important as the transfer method. PTR, LP, ALP,
and EMA proved to provide useful observations. The teacher’s reward signal is
essential for training the teacher fast. The source task reward provides the
teacher agent with a rich reward signal and encourages to find a general
student agent. With this reward signal, it is difficult to target specific
environments to solve, but using the target reward signal is too sparse. One
could use a weighted form of the source task reward to lay focus on a subset
of the tasks while still providing a rich reward signal. Another option would
be a combination of both reward signals, where the source task reward signal
is used at the beginning of training and slowly exchanged with the target
reward.

For future work, it would be interesting to experiment with different rl
algorithms to train the teacher. One could use traditional rl methods such as
Q learning or off-policy algorithms with experience replay such as algorithms
from the DQN family. More work must be dedicated to the hyperparameter tuning
of the rl algorithms, teacher-student step settings, and network
architectures.

Although our work did not increase the asymptotic performance on most of the
MiniGrid and Google Football environments, we increased the overall
performance across the tasks and the sample efficiency. Future work should
experiment with other domains such as robotics and further tune the
hyperparameters to better understand our findings.

## References

  * Silver et al. [2016] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” _Nature_ , vol. 529, no. 7587, pp. 484–489, 2016.
  * Silver et al. [2017] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, “Mastering the game of go without human knowledge,” _Nature_ , vol. 550, no. 7676, pp. 354–359, 2017.
  * Silver et al. [2018] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, “A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,” _Science_ , vol. 362, no. 6419, pp. 1140–1144, 2018.
  * Mnih et al. [2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * OpenAI et al. [2019a] OpenAI, :, C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tang, F. Wolski, and S. Zhang, “Dota 2 with large scale deep reinforcement learning,” 2019.
  * Vinyals et al. [2019] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver, “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” _Nature_ , vol. 575, no. 7782, pp. 350–354, 2019.
  * OpenAI et al. [2019b] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang, “Solving rubik’s cube with a robot hand,” 2019.
  * Badia et al. [2020] A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, D. Guo, and C. Blundell, “Agent57: Outperforming the atari human benchmark,” 2020.
  * Pomerleau [1989] D. A. Pomerleau, _ALVINN: An Autonomous Land Vehicle in a Neural Network_. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1989, p. 305–313.
  * Ross et al. [2011] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” ser. Proceedings of Machine Learning Research, G. Gordon, D. Dunson, and M. Dudík, Eds., vol. 15. Fort Lauderdale, FL, USA: JMLR Workshop and Conference Proceedings, 11–13 Apr 2011, pp. 627–635.
  * Scheller et al. [2020] C. Scheller, Y. Schraner, and M. Vogel, “Sample efficient reinforcement learning through learning from demonstrations in minecraft,” ser. Proceedings of Machine Learning Research, H. J. Escalante and R. Hadsell, Eds., vol. 123. Vancouver, CA: PMLR, 08–14 Dec 2020, pp. 67–76.
  * Sutton et al. [1998] R. S. Sutton, D. Precup, and S. P. Singh, “Intra-option learning about temporally abstract actions,” in _Proceedings of the Fifteenth International Conference on Machine Learning_ , ser. ICML ’98. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1998, p. 556–564.
  * Dayan and Hinton [1993] P. Dayan and G. E. Hinton, “Feudal reinforcement learning,” in _Advances in Neural Information Processing Systems 5_ , S. J. Hanson, J. D. Cowan, and C. L. Giles, Eds. Morgan-Kaufmann, 1993, pp. 271–278.
  * Sutton et al. [1999a] R. S. Sutton, D. Precup, and S. Singh, “Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,” _Artif. Intell._ , vol. 112, no. 1–2, p. 181–211, Aug. 1999.
  * Elman [1993] J. L. Elman, “Learning and development in neural networks: The importance of starting small,” _Cognition_ , vol. 48, no. 1, pp. 71–99, 1993.
  * Lazaric et al. [2008] A. Lazaric, M. Restelli, and A. Bonarini, “Transfer of samples in batch reinforcement learning,” in _Proceedings of the 25th International Conference on Machine Learning_ , ser. ICML ’08. New York, NY, USA: Association for Computing Machinery, 2008, p. 544–551.
  * Lazaric and Restelli [2011] A. Lazaric and M. Restelli, “Transfer from multiple mdps,” in _Advances in Neural Information Processing Systems_ , J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, Eds., vol. 24. Curran Associates, Inc., 2011.
  * Sutton et al. [1999b] R. Sutton, D. Precup, and S. Singh, “Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,” _Artificial Intelligence_ , vol. 112, pp. 181–211, 1999.
  * Soni and Singh [2006] V. Soni and S. P. Singh, “Using homomorphisms to transfer options across continuous reinforcement learning domains,” in _Proceedings, The Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, July 16-20, 2006, Boston, Massachusetts, USA_. AAAI Press, 2006, pp. 494–499.
  * Vezhnevets et al. [2016] A. S. Vezhnevets, V. Mnih, J. Agapiou, S. Osindero, A. Graves, O. Vinyals, and K. Kavukcuoglu, “Strategic attentive writer for learning macro-actions,” in _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , ser. NIPS’16. Red Hook, NY, USA: Curran Associates Inc., 2016, p. 3494–3502.
  * Fachantidis et al. [2013] A. Fachantidis, I. Partalas, G. Tsoumakas, and I. Vlahavas, “Transferring task models in reinforcement learning agents,” _Neurocomputing_ , vol. 107, pp. 23–32, 2013, timely Neural Networks Applications in Engineering.
  * Fernández et al. [2010] F. Fernández, J. García, and M. Veloso, “Probabilistic policy reuse for inter-task transfer learning,” _Robotics and Autonomous Systems_ , vol. 58, no. 7, pp. 866–871, 2010, advances in Autonomous Robots for Service and Entertainment.
  * Taylor et al. [2007] M. E. Taylor, P. Stone, and Y. Liu, “Transfer learning via inter-task mappings for temporal difference learning,” _Journal of Machine Learning Research_ , vol. 8, no. 1, pp. 2125–2167, 2007.
  * Taylor and Stone [2005] M. E. Taylor and P. Stone, “Behavior transfer for value-function-based reinforcement learning,” in _The Fourth International Joint Conference on Autonomous Agents and Multiagent Systems_ , F. Dignum, V. Dignum, S. Koenig, S. Kraus, M. P. Singh, and M. Wooldridge, Eds. New York, NY: ACM Press, July 2005, pp. 53–59.
  * Taylor and Stone [2009] ——, “Transfer learning for reinforcement learning domains: A survey,” _Journal of Machine Learning Research_ , vol. 10, no. 56, pp. 1633–1685, 2009\.
  * Lazaric [2012] A. Lazaric, “Transfer in reinforcement learning: A framework and a survey,” in _Adaptation, Learning, and Optimization_. Springer Berlin Heidelberg, 2012, pp. 143–173.
  * Bengio et al. [2009] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _International Conference on Machine Learning, ICML_ , 2009.
  * Schaul et al. [2016] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” in _International Conference on Learning Representations_ , Puerto Rico, 2016.
  * Andrychowicz et al. [2017] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hindsight experience replay,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017.
  * Ren et al. [2018] Z. Ren, D. Dong, H. Li, and C. Chen, “Self-paced prioritized curriculum learning with coverage penalty in deep reinforcement learning,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 29, no. 6, pp. 2216–2226, 2018.
  * Kim and Choi [2018] T. Kim and J. Choi, “Screenernet: Learning curriculum for neural networks,” _CoRR_ , vol. abs/1801.00904, 2018.
  * Florensa et al. [2017] C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel, “Reverse curriculum generation for reinforcement learning,” in _Proceedings of the 1st Annual Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, S. Levine, V. Vanhoucke, and K. Goldberg, Eds., vol. 78. PMLR, 13–15 Nov 2017, pp. 482–495.
  * Foglino et al. [2019a] F. Foglino, C. C. Christakou, and M. Leonetti, “An optimization framework for task sequencing in curriculum learning,” in _2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)_ , 2019, pp. 207–214.
  * Foglino et al. [2019b] F. Foglino, C. Coletto Christakou, R. Luna Gutierrez, and M. Leonetti, “Curriculum learning for cumulative return maximization,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_. International Joint Conferences on Artificial Intelligence Organization, 7 2019, pp. 2308–2314.
  * Narvekar et al. [2017] S. Narvekar, J. Sinapov, and P. Stone, “Autonomous task sequencing for customized curriculum design in reinforcement learning,” in _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17_ , 2017, pp. 2536–2542.
  * Matiisen et al. [2019] T. Matiisen, A. Oliver, T. Cohen, and J. Schulman, “Teacher-student curriculum learning,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–9, 2019.
  * Narvekar and Stone [2019] S. Narvekar and P. Stone, “Learning curriculum policies for reinforcement learning,” in _Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS)_ , May 2019.
  * Narvekar et al. [2016] S. Narvekar, J. Sinapov, M. Leonetti, and P. Stone, “Source task creation for curriculum learning,” in _Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016)_ , Singapore, May 2016.
  * Schmidhuber [2013] J. Schmidhuber, “Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem,” _Frontiers in Psychology_ , vol. 4, p. 313, 2013.
  * Jiang et al. [2020] M. Jiang, E. Grefenstette, and T. Rocktäschel, “Prioritized level replay,” 2020\.
  * Kurach et al. [2019] K. Kurach, A. Raichuk, P. Stanczyk, M. Zajac, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet, and S. Gelly, “Google research football: A novel reinforcement learning environment,” 2019.
  * Flet-Berliac et al. [2020] Y. Flet-Berliac, R. Ouhamma, O.-A. Maillard, and P. Preux, “Is standard deviation the new standard? revisiting the critic in deep policy gradients,” 2020\.
  * Schraner [2020] Y. Schraner, _ReinforcementLearning on the GoogleFootball environment_ , 2020\.
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_ , 2nd ed. The MIT Press, 2018.
  * Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _CoRR_ , vol. abs/1707.06347, 2017.
  * Schulman et al. [2015] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” ser. Proceedings of Machine Learning Research, F. Bach and D. Blei, Eds., vol. 37. Lille, France: PMLR, 07–09 Jul 2015, pp. 1889–1897.
  * Hsu et al. [2020] C. C. Hsu, C. Mendler-Dünner, and M. Hardt, “Revisiting design choices in proximal policy optimization,” _CoRR_ , vol. abs/2009.10897, 2020.
  * Narvekar et al. [2020] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone, “Curriculum learning for reinforcement learning domains: A framework and survey,” _J. Mach. Learn. Res._ , vol. 21, pp. 181:1–181:50, 2020.
  * Svetlik et al. [2017] M. Svetlik, M. Leonetti, J. Sinapov, R. Shah, N. Walker, and P. Stone, “Automatic curriculum graph generation for reinforcement learning agents,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, Feb. 2017.
  * Wang et al. [2019] R. Wang, J. Lehman, J. Clune, and K. O. Stanley, “Poet: Open-ended coevolution of environments and their optimized solutions,” in _Proceedings of the Genetic and Evolutionary Computation Conference_ , ser. GECCO ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 142–151.
  * F.R.S. [1901] K. P. F.R.S., “Liii. on lines and planes of closest fit to systems of points in space,” _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ , vol. 2, no. 11, pp. 559–572, 1901.
  * Portelas et al. [2019] R. Portelas, C. Colas, K. Hofmann, and P. Oudeyer, “Teacher algorithms for curriculum learning of deep RL in continuously parameterized environments,” in _3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings_ , ser. Proceedings of Machine Learning Research, L. P. Kaelbling, D. Kragic, and K. Sugiura, Eds., vol. 100. PMLR, 2019, pp. 835–853.
  * Kanitscheider et al. [2021] I. Kanitscheider, J. Huizinga, D. Farhi, W. H. Guss, B. Houghton, R. Sampedro, P. Zhokhov, B. Baker, A. Ecoffet, J. Tang, O. Klimov, and J. Clune, “Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft,” _CoRR_ , vol. abs/2106.14876, 2021.
  * Agarap [2018] A. F. Agarap, “Deep learning using rectified linear units (relu),” 2018, cite arxiv:1803.08375Comment: 7 pages, 11 figures, 9 tables.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * Chevalier-Boisvert et al. [2018] M. Chevalier-Boisvert, L. Willems, and S. Pal, “Minimalistic gridworld environment for openai gym,” https://github.com/maximecb/gym-minigrid, 2018\.
  * Schraner [2019] Y. Schraner, _IP7 - Reinforcement Learning on the Google Football environment_ , 2019.
  * Racanière et al. [2020] S. Racanière, A. K. Lampinen, A. Santoro, D. P. Reichert, V. Firoiu, and T. P. Lillicrap, “Automated curriculum generation through setter-solver interactions,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.
  * Riedmiller et al. [2018] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. van de Wiele, V. Mnih, N. Heess, and J. T. Springenberg, “Learning by playing solving sparse reward tasks from scratch,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 4344–4353.
  * Shao et al. [2019] K. Shao, Y. Zhu, and D. Zhao, “Starcraft micromanagement with reinforcement learning and curriculum transfer learning,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , vol. 3, no. 1, pp. 73–84, 2019\.

###### List of Figures

  1. 2.1 The agent-environment interaction in a Markov decision process [Sutton and Barto, 2018].
  2. 2.2 Performance metrics for transfer learning using (a) weak transfer and (b) strong transfer (figure by Narvekar et al. [2020]).
  3. (a) Weak transfer
  4. (b) Strong transfer
  5. 2.3 We visualize the interaction between the three key elements of curriculum learning. Task generation is concerned with generate a set of tasks for the curriculum. Task sequencing selects a task out of the task set for the agent. The agent use knowledge obtained by training on previously selected task through transfer learning. After or during training on the new task, the obtained knowledge is stored. The task generation and sequencing can be done online and dependent on the student’s current performance or offline before training.
  6. 3.1 Teacher-student interaction for task sequencing.
  7. 4.1 Visualization of a subset of the MiniGrid environments used in this work. The environment names from top left to bottom right: Empty-6x6, FourRooms, DoorKey-16x16, MultiRoom-N4S5, KeyCorridor-S3R2, KeyCorridor-S3R3, and KeyCorridor-S6R3.
  8. 4.2 A blue line illustrates the agent’s trajectory in the DoorKey-8x8 environment. The maximum number of steps in the DoorKey-8x8 environment is 640. If the agent takes the direct path, he takes 18 steps until he reaches the green goal square. Therefore the agents reward is $1-0.9*(18/640)=0.9747$. The agent on the right takes 28 steps, leaving him with a reward of $1-0.9*(28/640)=0.9606$.
  9. 4.3 Student neural network architecture for the MiniGrid environments. The input is a $25\times 25\times 3$ fully observable representation of the environments state. After every fully connected layer we use a ReLU activation function. The policy head uses a softmax activation function for the action probability distribution. The value head does not use an activation function.
  10. 4.4 Comparison of the average return over 100 episodes at the end of training between MLP and CNN models. The agent is trained with PPO for 10 million steps on a single environment.
  11. 4.5 Learning curves for different teacher observations and transfer methods. After each teacher step the sum of the students average return over 100 episodes for each environment in $\mathcal{V}$ is plotted. We plot the learning curve with the highest total reward at the end of training for every configuration.
  12. 4.6 We visualize the added reward, when using reward transfer (left column) and combined transfer (right column). For every state in the Empty-8x8 (top row) and Empty-16x16 (bottom row) grid world environment we plot the added reward. A light color encodes a high added reward, black equals to zero added reward. Dark tiles at the end of the grid are walls, the black tile in the bottom right corner is the green goal square.
  13. 4.7 We visualize the sample probability distribution and learning curve of one teacher agent over its training time. The colors in the legends are ordered upside down to the order in the plot.
  14. 4.8 Visualization of the target reward signal for each teacher observation type over the teachers training cycle.
  15. 4.9 The average learning curve of three runs with three different random seeds for each $\alpha$ value.
  16. 4.10 We visualize the learning curves of our agent when trained in the teacher-student setup and directly on the target task for the Empty-16x16, DoorKey16x16, and FourRooms environment. We use a teacher with policy transfer, source task reward signal, and the six partially observable inputs. The learning curve of the agent trained directly on the task is labeled with ”None”.
  17. 4.11 The Google Football Engine is a football simulation which supports the major football rules like kickoffs (top left), goals (top right), fouls, cards (bottom left), corner and penalty kicks (bottom right), and offside. [Kurach et al., 2019]
  18. 4.12 To approximate the Markov property we use a frame stack of the last 4 frames as an input for our neural networks.
  19. 4.13 CNN network architecture with the stacked Super Mini Map observation as input. The one-hot encoded previous action $a_{t-1}$ is used as an additional input. The policy and value function are two heads sharing the network torso.
  20. A.1 Student CNN neural network architecture for the MiniGrid environments.

###### List of Tables

  1. 1.1 Comparison of our different curriculum learning approaches, the average return over 100 episodes on the 11 vs 11 hard environment is reported.
  2. 4.1 Description of the MiniGrid environments used in this thesis. All environments impose a penalty for the number of steps taken until reaching the target.
  3. 4.2 Highest average return over 100 episodes for each teacher type at the end of 1000 teacher steps. The None results are the average return over 100 episodes obtained when training only the environment of that row.
  4. 4.3 We compare the different knowledge transfer methods for each teacher observation type. Each configuration is run with three different random seeds. The maximum return over 100 episodes at the end of 1’000 teacher steps is reported. Additionally, we provide the standard deviation between the three runs.
  5. 4.4 We compare the different knowledge transfer methods for each baseline. Each configuration is run with three different random seeds. The maximum the average students return of 100 episodes at the end of 1’000 teacher steps is reported. Additionally, we provide the standard deviation between the three runs.
  6. 4.5 We compare the different teacher reward signals for each teacher observation type. Each configuration is run with three different random seeds. The maximum return over 100 episodes at the end of 1’000 teacher steps is reported. Additionally, we provide the standard deviation between the three runs.
  7. 4.6 We compare the different $\alpha$ values for the EMA observation type. Each configuration is run with three different random seeds. The maximum return of 100 episodes at the end of 1’000 teacher steps is reported. Additionally, we provide the standard deviation between the three runs.
  8. 4.7 We report the median measurements obtained by evaluating each agent with five random seed for 100 episodes on every environment in $\mathcal{V}$.
  9. 4.8 Description of the Google Football environments taken directly from Kurach et al. [2019]. All scenarios end after 400 frames, or if the ball is lost, a team scores or the game stops (e.g., if the ball leaves the pitch or a free kick is awarded).
  10. 4.9 Google football results obtained on one random seed after training for 1’000 CMDP steps, equal to 10 million student steps. The average return over 100 episodes is reported. We use policy transfer between the source tasks and the source task reward signal.
  11. 4.10 Comparison of our different curriculum learning approaches of our previous work Schraner [2020], the average return over 100 episodes on the 11 vs 11 hard environment is reported.
  12. 4.11 Results on the Football Academy scenarios obtained in our previous work [Schraner, 2019] and the Google baseline [Kurach et al., 2019] with the IMPALA algorithm. The reported results is the average reward of 100 episodes after a training on 50 million frames.
  13. A.1 Hyperparameters used for training a student in the CMDP setting on the grid world environments.
  14. A.2 Important fixed student hyperparameters used in our Google Football CMDP experiments.
  15. A.3 Hyperparameters used for training a teacher in the CMDP setting used in both the grid world and Google Football experiments.
  16. A.4 Comparison of the MLP and LSTM teacher architecture. We report the total average return over 100 episodes on all environments in $\mathcal{V}$ at the end of training.

##  Appendix A Appendix

###  A.1 Grid World CNN Architecture

In fig. A.1 we depict the CNN network architecture for grid world
environments. The network input is a $25\times 25\times 3$ fully observable
representation of the environment’s state. We use four convolution blocks with
$[16,16,32,64]$ channels, a $[2\times 2,3\times 3,3\times 3,3\times 3]$ kernel
and a stride of $[1,1,1,3]$. A ReLU activation function follows each
convolution block. After the convolution blocks, we use a fully connected
layer with 128 units. The policy and value function are separate heads on top
of the fully connected layer. The policy head uses a softmax activation
function.

Figure A.1: Student CNN neural network architecture for the MiniGrid
environments.

###  A.2 Grid World Student Hyperparameters

We optimized a limited number of hyperparameters for our students in the grid
world environment. The hyperparameters were tuned by training an rl agent with
PPO on a single environment for 10 million steps. We evaluate all
hyperparameters on all environments in $\mathcal{V}$. The parameters are
reported in table A.1. Parameters with only a single ”range” value in table
A.1 were set to common values used for PPO.

Parameter | Range | Best
---|---|---
Learning rate | [0.01, 0.03, 0.001, 0.003, 0.0001, 0.0003] | 0.0003
Discount | [0.97, 0.99, 0.997, 0.999] | 0.999
Entropy loss coefficient | 0.01 | 0.01
Value loss coefficient | 0.5 | 0.5
Gradient norm clip | 0.5 | 0.5
GAE $\lambda$ | 0.95 | 0.95
Clipping range | 0.2 | 0.2
Normalize advantage | True | True
Minibatches | 1 | 1
Epochs | [1,2,4] | 2
Optimizer | adam | adam
Training steps | $10^{\prime}000$ | $10^{\prime}000$
Unroll length | [256, 512] | 256
Number of actors | 40 | 40
Table A.1: Hyperparameters used for training a student in the CMDP setting on
the grid world environments.

###  A.3 Google Football Student Hyperparameters

In table A.2 we report the hyperparameters used in our Google Football CMDP
experiments. These values were tuned in our previous work [Schraner, 2020].
Therefore we do not tune any of those parameters in this thesis.

Parameter | Value
---|---
Learning rate | 0.0011879
Discount | $0.997$
Entropy loss coefficient | $0.00155$
Gradient norm clip | $0.76$
Value loss coefficient | $0.5$
GAE $\lambda$ | $0.95$
Clipping range | 0.115
Normalize advantage | True
Minibatches | 4
Epochs | 2
Optimizer | adam
Reward | Scoring
Observation | SMM
Training steps | $10^{\prime}000$
Unroll length | 512
Number of actors | 40
Table A.2: Important fixed student hyperparameters used in our Google Football
CMDP experiments.

###  A.4 Teacher Hyperparameters

We optimized a limited number of hyperparameters for our teacher on the grid
world environment. The hyperparameters were tuned by training a teacher agent
for 1000 steps with PPO, using policy transfer, total average return and the
RH observation. The parameters are reported in table A.3 and used for both the
grid world and Google Football experiments. Parameters with only a single
”range” value in table A.3 were set to common values used for PPO.

Parameter | Range | Best
---|---|---
Learning rate | [0.1, 0.3, 0.01, 0.03] | 0.03
Linear learning rate schedule | True | True
Discount | 1 | 1
Entropy loss coefficient | 0.01 | 0.01
Value loss coefficient | 0.5 | 0.5
Gradient norm clip | 0.5 | 0.5
GAE $\lambda$ | 0.95 | 0.95
Clipping range | 0.2 | 0.2
Normalize advantage | True | True
Minibatches | 1 | 1
Epochs | 4 | 4
Optimizer | adam | adam
Training steps | 1000 | 1000
Unroll length | 4 | 4
Table A.3: Hyperparameters used for training a teacher in the CMDP setting
used in both the grid world and Google Football experiments.

###  A.5 MLP vs. LSTM Teacher Model

We evaluated teachers with an MLP network architecture and an LSTM
architecture as described in section 3.2.4. Executing experiments with the
LSTM architecture takes around 1.5 times as long as using the MLP
architecture. In table A.4 we compare both network architectures. We report
the total average return. Each teacher is trained for 1’000 CMDP steps, where
for each step, the student is trained for 10’000 frames in the selected
environment. The total average return reward signal and policy transfer are
used in all experiments. We do not report results for LP and EMA for the LSTM
model, as those observation types were developed after this evaluation has
been carried out. The MLP architecture is superior to the LSTM architecture
for all experiments except the RH observation type. Therefore, we use the MLP
architecture for the experiments reported in this thesis.

Model | RH | PTR | LP | ALP | EMA | FS-EMA
---|---|---|---|---|---|---
MLP | 2.34 | 4.44 | 4.17 | 3.35 | 4.35 | 3.72
LSTM | 3.12 | 2.97 | - | 2.95 | - | 2.76
Table A.4: Comparison of the MLP and LSTM teacher architecture. We report the
total average return over 100 episodes on all environments in $\mathcal{V}$ at
the end of training.


# Inhomogeneity of a rotating quark-gluon plasma from holography

Nelson R. F. Braga, Octavio. C. Junqueira
nrfbraga<EMAIL_ADDRESS>Universidade Federal do Rio de
Janeiro—Instituto de Física, CEP: 21941-909, Rio de Janeiro, RJ, Brazil

###### Abstract

Rotation affects the transition temperature between confined (hadronic) and
deconfined (quark-gluon plasma) phases of the strongly interacting matter
produced in non-central heavy ion collisions. A holographic description of
this effect was presented recently, considering an AdS black hole with
cylindrical symmetry in rotation. Here we extend this approach in order to
analyse the more realistic case of strongly interacting matter that, rather
than living in a cylindrical shell, spreads over a region around the
rotational axis. In this case, the confined and deconfined phases may coexist.
The holographic description of the plasma behaviour under rotation is shown to
be consistent with the concept of local temperature for rotating frames
developed by Tolman and Ehrenfest.

##  I Introduction

The Quark-Gluon Plasma (QGP) is a state of matter formed by deconfined partons
that interact strongly Busza:2018rrf . Such a state can be created in
relativistic heavy-ion collisions produced in particle accelerators. When the
collision is non-central, the resulting quantum chromodynamics (QCD) matter
acquires angular momentum. This effect can be indirectly measured since the
angular momentum affects the polarization of strange quarks, see Liang:2004ph
, and induce measurable spin polarization of hadrons Huang:2020dtn .
Experimental results of RHIC collaboration indicates that this angular
momentum is of the order of $10^{3}\hbar$, with angular velocity $\omega\sim
0.03\,\text{fm}^{-1}c$ STAR:2017ckg . Recent models predict that it could
reach even higher values, with $\omega\sim 0.1\,\text{fm}^{-1}c$ Deng:2016gyh
; Jiang:2016woz .

Rotation affects the properties of QCD matter, including the critical
temperature of the confinement/deconfinement transition between hadronic and
plasma phases. For previous studies about the rotation effects in the QGP, see
for example Miranda:2014vaa ; Jiang:2016wvv ; McInnes:2016dwk ; Mamani:2018qzl
; Wang:2018sur ; Chernodub:2020qah ; Arefeva:2020jvo ; Chen:2020ath ;
Zhou:2021sdy ; Braguta:2021jgn ; Braguta:2021ucr ; Golubtsova:2021agl ;
Fujimoto:2021xix ; Braga:2022yfe ; Chen:2022mhf ; Golubtsova:2022ldm ;
Zhao:2022uxc ; Chernodub:2022 . The rotating QCD matter can be investigated
using holographic models Zhou:2021sdy ; Braga:2022yfe , with a rotating AdS
black hole (BH) being the holographic dual of the plasma. Such models predict
that rotation influences the critical temperature of phase transition: it
decreases with increasing angular velocity, in agreement with recent studies
of rotating QGP using phenomenological models Chen:2020ath ; Chernodub:2020qah
; Fujimoto:2021xix . The same behaviour was found in the Nambu-Jona-Lansinio
model Jiang:2016wvv , in which the critical temperature decreases due to
suppression of the chiral condensate. In contrast, lattice simulations for the
phase transition in gluondynamics with relativistic rotation Braguta:2021jgn ;
Braguta:2021ucr  indicates that the critical temperature increases with
increasing angular velocity.

The inhomogeneity of rotating QCD matter was studied recently in
Chernodub:2020qah  considering a toy model based on the compact Quantum
Electrodynamics (QED) theory. An outcome of this work is that that the plasma
can present confined and deconfined phases, spatially separated according to
the rotational velocity.

In the present work we consider the holographic description of a plasma, with
arbitrary shape, in a rigid rotation around a fixed axis. The cylindrical
slices of the plasma, consisting of the points that share the same distance to
the axis, have the same velocity. Each of these slices can be represented by a
rotating anti-de Sitter black hole (AdS BH) with cylindrical symmetry, as it
was done in Ref. Braga:2022yfe . Considering a soft-wall AdS/QCD background we
analyse the confinement/deconfinement transition of the rotating plasma. It is
found that, depending on the temperature, the angular velocity and the maximum
radius of the plasma, the hadronic and plasma phases may coexist,
characterizing an inhomogeneity of the medium.

The paper is organized as follows: in Section II, we present the rotating AdS
BH geometry and discuss its properties, like temperature. In Section III, we
discuss the confinement/deconfinement phase transition in the holographic
soft-wall model. Section IV is devoted to describe the inhomogeneity of QCD
matter as an effect of rotation. Section V contains the comparison between the
approach based on the Tolman-Eherenfest law and the holographic description,
and, finally, Sections VI contains our conclusions.

##  II Rotating AdS black hole with cylindrical symmetry

The metric for an AdS black hole with cylindrical symmetry and radius $\rho$,
rotating with an angular velocity $\omega$, is given by Zhou:2021sdy

$\displaystyle ds^{2}$ $\displaystyle=$ $\displaystyle
g_{tt}dt^{2}+g_{t\phi}dtd\phi+g_{\phi t}d\phi dt+g_{\phi\phi}d\phi^{2}$ (2)
$\displaystyle+\,g_{zz}dz^{2}+g_{xx}\sum_{i=1}^{2}dx_{i}^{2}\;,$

with

$\displaystyle g_{tt}$ $\displaystyle=$
$\displaystyle\frac{\gamma^{2}(\omega)L^{2}}{z^{2}}\left(\omega^{2}\rho^{2}-f(z)\right)\;,$
(3) $\displaystyle g_{\phi\phi}$ $\displaystyle=$
$\displaystyle\frac{\gamma^{2}(\omega)L^{2}}{z^{2}}\rho^{2}\left(1-\omega^{2}\rho^{2}f(z)\right)\;,$
(4) $\displaystyle g_{t\phi}$ $\displaystyle=$ $\displaystyle g_{\phi
t}=\frac{\gamma^{2}(\omega)L^{2}}{z^{2}}\omega\rho^{2}\left(1-f(z)\right)\;,$
(5) $\displaystyle g_{zz}$ $\displaystyle=$
$\displaystyle\frac{L^{2}}{z^{2}f(z)}\;,$ (6) $\displaystyle g_{xx}$
$\displaystyle=$ $\displaystyle\frac{L^{2}}{z^{2}}\;,$ (7)

where $f(z)=1-z^{4}/z_{h}^{4}$, being $z_{h}$ the black hole horizon position,
$\gamma$ is the Lorentz factor,

$\gamma(\omega\rho)=\frac{1}{\sqrt{1-\omega^{2}\rho^{2}}}\;,$ (8)

and $\phi$ is the angular coordinate with periodicity $0<\phi<2\pi/\gamma$.

From gauge/gravity duality, this AdS black hole represents a rotating QGP with
the form of a cylindrical surface of radius $\rho$. This simple situation,
where the speed is the same for all points of the plasma, is an important tool
for representing a realistic plasma, by just considering a superposition of
BHs with different radius $\rho$.

Other solution of Einstein’s equation in the case with rotation is the thermal
AdS space, that has the same form of equations (3)-(7) but taking $f(z)=1$.
Such a space is used to eliminate ultraviolet divergencies in the Hawking-Page
method to determine the critical temperature of confinement/deconfinement
transition, as we shall see in the next section.

After a straightforward calculation, one can rewrite the rotating BH metric
(3)-(7) in the canonical form:

$\displaystyle ds^{2}$ $\displaystyle=$
$\displaystyle-N(z)dt^{2}+\frac{L^{2}}{z^{2}}\frac{dz^{2}}{f(z)}+R(z)\left(d\phi+P(z)dt\right)^{2}+\frac{L^{2}}{z^{2}}\sum_{i-1}^{2}dx_{i}^{2}\;,$
(9)

with

$\displaystyle N(z)$ $\displaystyle=$
$\displaystyle\frac{L^{2}}{z^{2}}\frac{f(z)(1-\omega^{2}\rho^{2})}{1-f(z)\omega^{2}\rho^{2}}\;,$
(10) $\displaystyle R(z)$ $\displaystyle=$
$\displaystyle\frac{L^{2}}{z^{2}}\left(\gamma^{2}\rho^{2}-f(z)\gamma^{2}\omega^{2}\rho^{4}\right)\;,$
(11) $\displaystyle P(z)$ $\displaystyle=$
$\displaystyle\frac{\omega(1-f(z))}{1-f(z)\omega^{2}\rho^{2}}\;.$ (12)

Thus, defining $h_{00}=-N(z)$, the Hawking temperature can be obtained from
the surface gravity formula Zhou:2021sdy :

$\displaystyle T$ $\displaystyle=$
$\displaystyle|\frac{\kappa_{G}}{2\pi}|=\bigg{|}\frac{\lim_{z\rightarrow
z_{h}}-\frac{1}{2}\sqrt{\frac{g^{zz}}{-h_{00}(z)}}h_{00,z}}{2\pi}\bigg{|}=\frac{1}{\pi
z_{h}}\sqrt{1-\omega^{2}\rho^{2}}\;,$ (13)

where $\kappa_{G}$ is the surface gravity, and $g^{zz}$ the $zz$ component of
the inverse of the rotating BH metric. Note that if one takes
$\omega\rho\rightarrow 0$, one obtains the temperature
$T(\omega\rho\rightarrow 0)=1/(\pi z_{h})$, which is the well-known Hawking
temperature of a non-rotating AdS black hole. In this limit, the rotating BH
metric (2)-(7) is reduced to

$ds^{2}=\frac{L^{2}}{z^{2}}\left(f(z)dt^{2}+d\overrightarrow{x}^{2}+\rho^{2}d\phi^{2}+\frac{dz^{2}}{f(z)}\right)\;,$
(14)

which is the (non-rotating) cylindrical AdS BH space.

One can associate this temperature $1/(\pi z_{h})$ to the local temperature,
observed in a frame that is rotating with the plasma and thus observes no
rotation. Defining $T_{\text{local}}=1/(\pi z_{h})$ and using the expression
for the Hawking temperature (13), one obtains

$T_{\text{local}}=\frac{T}{\sqrt{1-\omega^{2}\rho^{2}}}\;,$ (15)

where $T$ corresponds to the temperature as seen by a rest frame that observes
the plasma rotating with a rotational velocity $\omega\rho$.

##  III Hawking-Page transition with rotation in the soft-wall AdS/QCD model

In the soft-wall holographic AdS/QCD model Karch:2006pv , one introduces an
energy parameter in the AdS geometry, which works as an infrared (IR) cutoff
in the gauge theory side of the gauge/gravity duality. The five-dimensional
gravitational action for this model, at zero temperature, takes the form
Herzog:2006ra ; BallonBayona:2007vp

$I=-\frac{1}{2\kappa^{2}}\int_{0}^{\infty}dz\int
d^{4}x\sqrt{g}e^{-\Phi}\left(R-\Lambda\right)\;=\frac{4}{L^{2}\kappa^{2}}\int_{0}^{\infty}dz\int
d^{4}x\sqrt{g}e^{-cz^{2}},$ (16)

where $\kappa$ is the gravitational coupling associated with the Newton
constant, and $\Lambda$, the cosmological constant that is related to the
curvature and the AdS radius by $\Lambda=\frac{3}{5}R=-\frac{12}{L^{2}}$. The
scalar field $\Phi(z)$ is the dilaton background that introduces the IR energy
parameter $\sqrt{c}$ in the model.

Considering the metric (2) for the rotating BH case and a similar expression,
but with $f(z)=1$, for the thermal AdS one, the determinant of the metric in
both cases is $g=\frac{L^{10}}{z^{10}}$, such that the finite temperature
version of (16) reads

$I_{on-shell}=\frac{4L^{3}}{\kappa^{2}}\int_{0}^{z_{f}}dz\int
d^{4}xz^{-5}e^{-cz^{2}}\;\,,$ (17)

where $z_{f}=z_{h}$ for the black hole. For the thermal AdS space there is no
horizon, so $z_{f}\rightarrow\infty$.

As the integration over the spatial bulk coordinates is trivial, we define an
action density, $\mathcal{E}=\frac{1}{V_{3D}}I_{on-shell}$, where $V_{3D}$ is
the spatial volume factor of the bulk. For a compact time direction in the
Euclidean signature with period $\beta_{s}$ that depends on the space
considered, we have $0\leq t<\beta_{s}$, such that, including the ultraviolet
regulator $\varepsilon$, one obtains the action density

$\mathcal{E}_{s}(\varepsilon)=\frac{4L^{3}}{\kappa^{2}\gamma}\int_{0}^{\beta_{s}}dt\int_{\varepsilon}^{z_{f}}dz\,z^{-5}e^{-cz^{2}}\;,$
(18)

where, for the AdS with a black hole, one has: $\beta_{BH}=\beta=1/T$ . By
requiring that the asymptotic limits of the BH and thermal AdS geometries are
the same at $z=\varepsilon$ with $\varepsilon\rightarrow 0$, one finds that
for the thermal AdS space the temporal period $\beta_{s}$ is
$\beta_{AdS}=\beta^{\prime}=\beta\sqrt{f(\varepsilon)}$. Note that the factor
$1/\gamma$ appear in $\mathcal{E}_{s}$ because we are dividing by the volume
of the cylindrical surface, for which the angular coordinate has the range
$0<\phi<2\pi$, whereas for the rotating BH metric one has:
$0<\phi<2\pi/\gamma$.

For both geometries the energy densities are infinite in the limit
$\varepsilon\rightarrow 0$. For this reason one defines a regularized action
density for the rotating black hole as the difference between the energy
densities of the two geometries, namely,

$\bigtriangleup\mathcal{E}(\varepsilon)=\lim_{\varepsilon\rightarrow
0}\left[\mathcal{E}_{BH}(\varepsilon)-\mathcal{E}_{AdS}(\varepsilon)\right]\;,$
(19)

where

$\displaystyle\mathcal{E}_{BH}(\varepsilon)$ $\displaystyle=$
$\displaystyle\frac{4L^{3}}{\kappa^{2}\gamma}\beta\int_{\varepsilon}^{z_{h}}dz\,z^{-5}e^{-cz^{2}}\;,$
(20) $\displaystyle\mathcal{E}_{AdS}(\varepsilon)$ $\displaystyle=$
$\displaystyle\frac{4L^{3}}{\kappa^{2}\gamma}\beta^{\prime}\int_{\varepsilon}^{\infty}dz\,z^{-5}e^{-cz^{2}}\;.$
(21)

This regularized density is finite in the ultraviolet limit and works as a
measure of stability of the rotating black hole. When
$\bigtriangleup\mathcal{E}$ is positive, the BH is unstable since the action
density of the thermal AdS space is smaller that the black hole one. In
contrast, if it is negative, the BH is stable. From gauge/gravity duality,
such analysis of stability is related to the transition between the confined
hadronic phase and the deconfined plasma one, with the black hole
corresponding to the QGP plasma and the thermal AdS to the hadronic phase.

Replacing (20) and (21) into (19), one obtains

$\bigtriangleup\mathcal{E}(z_{h})\,=\,\frac{L^{3}\pi}{\kappa^{2}z_{h}^{3}}\bigg{[}e^{-cz_{h}^{2}}(-1+cz_{h}^{2})+\frac{1}{2}+c^{2}z_{h}^{4}\text{Ei}\left(-cz_{h}^{2}\right)\bigg{]}\,.$
(22)

The critical temperature for the confinement/deconfinement phase transition is
obtained from the condition $\bigtriangleup\mathcal{E}(\omega\rho,T_{c})=0$.
The dependence of $\bigtriangleup\mathcal{E}$ on the temperature and on the
rotational velocity $\,\omega\rho\,$ is obtained using the Hawking temperature
expression (13). In Figure 1, we plot $\bigtriangleup\mathcal{E}$ as a
function of $T$, at six fixed values of $\omega\rho$.

Figure 1: Action density of rotating plasma ($\bigtriangleup\mathcal{E}$) as a
function of $\bar{T}=T/\sqrt{c}$, at different rotational velocities
$\omega\rho$, and with normalization $L^{3}/\kappa^{2}=1$.

From Eqs. (13) and (22), one obtains the critical temperature

$T_{c}(\omega\rho)=\frac{0.491728\sqrt{c}}{\gamma(\omega\rho)}=T_{c}(0)\sqrt{1-\omega^{2}\rho^{2}}\,.$
(23)

Equation (23) establishes how the critical temperature $T_{c}$, as observed by
a rest frame, depends on the angular velocity and on the rotational radius. It
is interesting to note that the above equation implies that for the rotating
plasma the transition occurs at a temperature
$T=T_{c}(\omega\rho)=T_{c}(0)\sqrt{1-\omega^{2}\rho^{2}}$, or equivalently,
using Eq. (15), when

$\frac{T}{\sqrt{1-\omega^{2}\rho^{2}}}=T_{\text{local}}=T_{c}(0)\,,$ (24)

or, in other words, when the local temperature is equal to the critical
temperature of (non-rotating) QCD matter.

##  IV QGP inhomogeneity from holography

In order to represent the situation that appears in non-central heavy-ion
collisions, we consider a region of space with QCD matter rotating with
angular velicity $\omega$ around a fixed axis. Using the same notation as in
the last sections, we represent the distance to the axis as $\rho$. For a
continuous distribution one has $0<\rho<R$, where $R$ is the maximum distance
to the axis where there is QCD matter. Points located on the axis, for which
$\rho=0$, are in the non-rotating rest frame.

Each portion of QCD matter located at a distance $\rho$ from the axis is
represented holographically by a rotating black hole, as the one described in
the previous sections, with rotational velocity $\omega\rho$. From Eq. (15)
one notices that the local temperature of this matter distribution increases
with $\rho$, reaching a maximum at $\rho=R$,

$T_{\text{local}}(R)=\frac{T}{\sqrt{1-\omega^{2}R^{2}}}\;,$ (25)

such that the local temperature inside the region varies in the range $T\leq
T_{local}\leq\frac{T}{\sqrt{1-\omega^{2}R^{2}}}$. This result was obtained
before in Chernodub:2020qah  from a different method.

As expressed in Eq. (24), the phase transition occurs when
$T_{\text{local}}=T_{c}(0)$. From the previous results, one identifies three
possible situations for rotating QCD matter.

i) If $T>T_{c}(0)$, there is just one phase. The QCD matter will be composed
of only QGP.

ii) If $T_{c}(0)>T_{\text{local}}(R)$, again there is just one phase. In this
case, the QCD matter will be composed only of hadrons.

iii) If $T<T_{c}(0)<T_{\text{local}}(R)$, the phase transition will occur at a
value $\rho_{c}$ of the radius such that
$T_{\text{local}}(\rho_{c})=T_{c}(0)$. In this case there are two phases:
hadronic matter for $0<\rho<\rho_{c}$ and QGP for $\rho_{c}<\rho<R$.

These three configurations are represented in Figure 2.

Figure 2: Configurations of rotating QCD matter in the region $0<\rho<R$. i)
QGP without phase transition for $T>T_{c}(0)$. ii) Only hadronic matter for
$T_{c}(0)>T_{local}(R)$. iii) Phase transition at $\rho_{c}$ when
$T_{\text{local}}(\rho_{c})=T_{c}(0)$.

The description of the existence of such a mixed phase in the plasma in a
rigid rotation was presented before in Chernodub:2020qah  using a different
approach to determine the critical temperature. The relation between the
critical temperature in the rotating and non-rotating frames are given by the
same equation in both cases, according to Eq. (23). Essentially, Figure 2 is
equivalent to Figure 7 of reference Chernodub:2020qah .

Such a result shows how the holographic description describes the
inhomogeneity of the QCD matter, this phenomenon being driven by the effect of
rotation. The configuration (iii) with a phase transition is characterized by
the curve of Figure 3, where we have assumed $\omega=0.1\,\text{fm}^{-1}$. If
there is no rotation, the inhomogeneity is not produced. The configuration of
QCD matter in the mixed phase described by holography possesses a typical
signature: the hadrons dominate the inner region near the rotation axis.

Figure 3: Local temperature of rotating QCD matter versus its radius ($\rho$),
assuming $\omega=0.1\,\text{fm}^{-1}$, $T=0.4$ and $T_{c}=0.49$, with the
energy scale $\sqrt{c}=1$. The radius is varying from zero to
$R=8\,\text{fm}$.

##  V Comparison with the Tolman-Ehrenfest effect

It was argued by Tolman and Ehrenfest Tolman:1930zza ; Tolman:1930ona  that
for a medium in thermal equilibrium located in a general non-flat space, local
observers would measure a temperature that depends on the spacetime metric. In
particular, for the rotating metric of eq. (2) the local temperature of Tolman
and Ehrenfest takes the form Chernodub:2020qah

$\displaystyle T^{TE}(\rho)=\frac{T^{TE}_{0}}{\sqrt{1-\omega^{2}\rho^{2}}}\;,$
(26)

where $T^{TE}_{0}$ is a reference temperature, that in our case is simply $T$,
the temperature as observed by a non-rotating frame, or equivalently the
temperature on the rotational axis. Comparing the above result with Eq. (15),
one realizes that the local temperature obtained by the holographic
description is equivalent to the Tolman-Ehrenfest one.

##  VI CONCLUSIONS

We described the possible inhomogeneity of a region of rotating QCD matter
using the holographic soft-wall AdS/QCD model. The rotational radius varies in
the range $0<\rho<R$. For each portion of matter at a given radius $\rho$
there is a particular gravitational dual. The relation between temperature,
rotational velocity and horizon position was determined by the Hawking
temperature condition of eq. (13). From this result we found and expression
for the local temperature observed by a frame that is at rest with respect to
the rotating strongly interacting matter. As a consequence one can have an
inhomogeneous medium, when the temperature at the rotational axis is smaller
than the critical temperature in the non-rotating case $T_{c}(0)$ but the
temperature at $\rho=R$ is greater than $T_{c}(0)$. It was also shown that the
local temperature obtained from holography is equivalent to the Tolman-
Ehrenfest one.

The main results obtained here coincide with the ones obtained previously in
ref. Chernodub:2020qah  using compact electrodynamics in two spatial
dimensions. It is interesting to see that the holographic approach leads to
the same behaviour for the rotating plasma.

Acknowledgments: The authors are supported by FAPERJ — Fundação Carlos Chagas
Filho de Amparo à Pesquisa do Estado do Rio de Janeiro and CNPq - Conselho
Nacional de Desenvolvimento Científico e Tecnológico. This work received also
support from Coordenação de Aperfeiçoamento de Pessoal de Nível Superior -
Brasil (CAPES) - Finance Code 001.

## References

  * (1) W. Busza, K. Rajagopal, and W. van der Schee, “Heavy Ion Collisions: The Big Picture, and the Big Questions,” Ann. Rev. Nucl. Part. Sci., vol. 68, pp. 339–376, 2018.
  * (2) Z.-T. Liang and X.-N. Wang, “Globally polarized quark-gluon plasma in non-central A+A collisions,” Phys. Rev. Lett., vol. 94, p. 102301, 2005\.  [Erratum: Phys.Rev.Lett. 96, 039901 (2006)].
  * (3) X.-G. Huang, J. Liao, Q. Wang, and X.-L. Xia, “Vorticity and Spin Polarization in Heavy Ion Collisions: Transport Models,” Lect. Notes Phys., vol. 987, pp. 281–308, 2021.
  * (4) L. Adamczyk et al., “Global $\Lambda$ hyperon polarization in nuclear collisions: evidence for the most vortical fluid,” Nature, vol. 548, pp. 62–65, 2017.
  * (5) W.-T. Deng and X.-G. Huang, “Vorticity in Heavy-Ion Collisions,” Phys. Rev. C, vol. 93, no. 6, p. 064907, 2016.
  * (6) Y. Jiang, Z.-W. Lin, and J. Liao, “Rotating quark-gluon plasma in relativistic heavy ion collisions,” Phys. Rev. C, vol. 94, no. 4, p. 044910, 2016.  [Erratum: Phys.Rev.C 95, 049904 (2017)].
  * (7) A. S. Miranda, J. Morgan, A. Kandus, and V. T. Zanchin, “Separable wave equations for gravitoelectromagnetic perturbations of rotating charged black strings,” Class. Quant. Grav., vol. 32, no. 23, p. 235002, 2015.
  * (8) Y. Jiang and J. Liao, “Pairing Phase Transitions of Matter under Rotation,” Phys. Rev. Lett., vol. 117, no. 19, p. 192302, 2016.
  * (9) B. McInnes, “A rotation/magnetism analogy for the quark–gluon plasma,” Nucl. Phys. B, vol. 911, pp. 173–190, 2016.
  * (10) L. A. H. Mamani, J. Morgan, A. S. Miranda, and V. T. Zanchin, “From quasinormal modes of rotating black strings to hydrodynamics of a moving CFT plasma,” Phys. Rev. D, vol. 98, no. 2, p. 026006, 2018.
  * (11) X. Wang, M. Wei, Z. Li, and M. Huang, “Quark matter under rotation in the NJL model with vector interaction,” Phys. Rev. D, vol. 99, no. 1, p. 016018, 2019.
  * (12) M. N. Chernodub, “Inhomogeneous confining-deconfining phases in rotating plasmas,” Phys. Rev. D, vol. 103, no. 5, p. 054027, 2021.
  * (13) I. Y. Aref’eva, A. A. Golubtsova, and E. Gourgoulhon, “Holographic drag force in 5d Kerr-AdS black hole,” JHEP, vol. 04, p. 169, 2021.
  * (14) X. Chen, L. Zhang, D. Li, D. Hou, and M. Huang, “Gluodynamics and deconfinement phase transition under rotation from holography,” JHEP, vol. 07, p. 132, 2021.
  * (15) J. Zhou, X. Chen, Y.-Q. Zhao, and J. Ping, “Thermodynamics of heavy quarkonium in rotating matter from holography,” Phys. Rev. D, vol. 102, no. 12, p. 126029, 2021.
  * (16) V. V. Braguta, A. Y. Kotov, D. D. Kuznedelev, and A. A. Roenko, “Influence of relativistic rotation on the confinement-deconfinement transition in gluodynamics,” Phys. Rev. D, vol. 103, no. 9, p. 094515, 2021.
  * (17) V. V. Braguta, A. Y. Kotov, D. D. Kuznedelev, and A. A. Roenko, “Lattice study of the confinement/deconfinement transition in rotating gluodynamics,” in 38th International Symposium on Lattice Field Theory, 10 2021.
  * (18) A. A. Golubtsova, E. Gourgoulhon, and M. K. Usova, “Heavy quarks in rotating plasma via holography,” Nucl. Phys. B, vol. 979, p. 115786, 2022.
  * (19) Y. Fujimoto, K. Fukushima, and Y. Hidaka, “Deconfining Phase Boundary of Rapidly Rotating Hot and Dense Matter and Analysis of Moment of Inertia,” Phys. Lett. B, vol. 816, p. 136184, 2021.
  * (20) N. R. F. Braga, L. F. Faulhaber, and O. C. Junqueira, “Confinement-deconfinement temperature for a rotating quark-gluon plasma,” Phys. Rev. D, vol. 105, no. 10, p. 106003, 2022.
  * (21) Y. Chen, D. Li, and M. Huang, “Inhomogeneous chiral condensation under rotation in the holographic QCD,” Phys. Rev. D, vol. 106, no. 10, p. 106002, 2022.
  * (22) A. A. Golubtsova and N. S. Tsegel’nik, “nuclear the holographic model of $\mathcal{N}=4$ SYM rotating quark-gluon plasma,” arXiv e-prints, p. arXiv:2211.11722, Nov. 2022.
  * (23) Y.-Q. Zhao, S. He, D. Hou, L. Li, and Z. Li, “Phase diagram of holographic thermal dense QCD matter with rotation,” arXiv e-prints, p. arXiv:2212.14662, Dec. 2022.
  * (24) M. N. Chernodub, V. A. Goy, and A. V. Molochkov, “Inhomogeneity of a rotating gluon plasma and the Tolman-Ehrenfest law in imaginary time: Lattice results for fast imaginary rotation,” Phys. Rev. D, vol. 107, no. 11, p. 114502, 2023.
  * (25) A. Karch, E. Katz, D. T. Son, and M. A. Stephanov, “Linear confinement and AdS/QCD,” Phys. Rev. D, vol. 74, p. 015005, 2006.
  * (26) C. P. Herzog, “A Holographic Prediction of the Deconfinement Temperature,” Phys. Rev. Lett., vol. 98, p. 091601, 2007.
  * (27) C. A. Ballon Bayona, H. Boschi-Filho, N. R. F. Braga, and L. A. Pando Zayas, “On a Holographic Model for Confinement/Deconfinement,” Phys. Rev. D, vol. 77, p. 046002, 2008.
  * (28) R. C. Tolman, “On the Weight of Heat and Thermal Equilibrium in General Relativity,” Phys. Rev., vol. 35, pp. 904–924, 1930.
  * (29) R. Tolman and P. Ehrenfest, “Temperature Equilibrium in a Static Gravitational Field,” Phys. Rev., vol. 36, no. 12, pp. 1791–1798, 1930\.