# Thermomagnetic Properties of QCD

Christoph P. Hofmanna

a Facultad de Ciencias, Universidad de Colima
Bernal Díaz del Castillo 340, Colima C.P. 28045, Mexico

###### Abstract

We explore the low-energy regime of quantum chromodynamics subjected to an
external magnetic field by deriving the two-loop representations for the
entropy density and the magnetization within chiral perturbation theory
(CHPT). At fixed temperature, the entropy density drops when the magnetic
field becomes stronger. The magnetization induced at finite temperature is
negative in the entire parameter region accessible by CHPT. We also point out
that the enhancement of the finite-temperature part in the quark condensate is
correlated with the decrease of the entropy density.

##  1 Introduction

The thermodynamic properties of quantum chromodynamics in a homogeneous
external magnetic field have been explored by many authors. In the present
study we focus on the entropy density and the magnetization – and furthermore
shed light on the connection between the quark condensate and entropy density.
Articles that also have discussed the dependence of entropy density and
magnetization on temperature, magnetic field strength, and pion mass are:
Refs. [1, 2, 3, 4, 5] based on lattice QCD, Refs. [6, 7, 8, 9] based on the
Nambu-Jona-Lasinio model and extensions thereof, as well as Refs. [10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23] that rely on yet other methods.

Still, in the regime of low temperatures and weak magnetic fields, a
comprehensive investigation of entropy density and magnetization appears to be
lacking. Here, within the framework of two-flavor chiral perturbation theory,
we provide such a fully systematic analysis. Based on earlier work of the
author, Refs. [24, 25, 26], we derive the two-loop representations for the
entropy density and the magnetization.

We find that the entropy density, at fixed temperature, decreases when the
magnetic field becomes stronger and it also decreases when the masses of the
pions grow. The impact of the magnetic field is most pronounced in the chiral
limit. In the real world with pion masses fixed at $M_{\pi}=140\,\text{MeV}$,
the entropy density also drops in the presence of an external magnetic field
whose impact is most distinct around the temperature $T\approx
30\,\text{MeV}$. While the results for the dependence of entropy density on
magnetic field strength, temperature, and arbitrary pion mass are new to the
best of our knowledge, the magnetic-field induced decrease of the entropy
density at the physical point $M_{\pi}=140\,\text{MeV}$ has also been observed
in the hadron resonance gas model [13], and in the (2+1) flavor Polyakov-loop
quark-meson model [21]. However, the comparison is only qualitative because
the latter reference is based on three flavors, and the hadron resonance gas
model includes even more particles. But most importantly, our CHPT study is
fully systematic and model independent.

The magnetization induced at finite temperature is negative in the entire
parameter region accessible by CHPT ($T,M_{\pi},\sqrt{|qH|}\lessapprox
0.2\,\text{GeV}$), which includes the physically most relevant case
$M_{\pi}=140\,\text{MeV}$. The magnitude of the finite-temperature
magnetization grows as both magnetic field strength and temperature increase,
implying that the QCD vacuum behaves as a diamagnetic medium at low
temperatures and weak magnetic fields. This is fully consistent with
conclusions drawn from lattice QCD [27, 3, 28, 29], the (2+1) flavor Polyakov-
loop quark-meson model [21], and the three-flavor quark-meson model with
$U_{A}(1)$ anomaly [30]. Still, the region accessible by CHPT has not been
fully addressed in these references.

Finally we point out that the characteristics of the entropy density and the
finite-temperature quark condensate in a magnetic field are correlated. This
becomes most transparent when the $H$=0 portions in either quantity are
subtracted to unmask the effect of the magnetic field. We observe that for
arbitrary pion masses – including the physical point $M_{\pi}=140\,\text{MeV}$
– the enhancement of the finite-temperature quark condensate in a magnetic
field is reflected in a decrease of the entropy density.

The article is organized as follows. To set the stage for our discussion, in
Sec. 2 we provide the two-loop representation for the free energy density and
explain our notation. In Sec. 3 we derive the entropy density and explore its
dependence on temperature, magnetic field strength and arbitrary pion masses,
including the physical point $M_{\pi}=140\,\text{MeV}$. The two-loop
representation for the magnetization induced at finite temperature is derived
in Sec. 4 and its properties in magnetic fields and for arbitrary pion masses
is elucidated in various figures. Sec. 5 is devoted to the connection between
order parameter and entropy density. Finally, Sec. 6 contains our conclusions.

##  2 Preliminaries

Two-flavor chiral perturbation theory111Introductions to chiral perturbation
theory are given in Refs. [31, 32, 33, 34, 35]. subjected to a magnetic
background $H$ and at finite temperature has been used by various authors to
explore the low-energy regime of quantum chromodynamics [36, 37, 38, 39, 40,
41, 42, 43, 44, 45, 24, 25, 26]. The starting point of the present study is
the two-loop representation for the free energy density derived in Ref.
[25],222We confine ourselves to the isospin limit $m_{u}=m_{d}$.

$z=z_{0}+z^{T}\,.$ (2.1)

Here $z_{0}$ is the vacuum energy density (free energy density at $T$=0) and
$z^{T}$ represents the finite-temperature portion. The latter amounts to

$\displaystyle z^{T}$ $\displaystyle=$ $\displaystyle-
g_{0}(M^{\pm}_{\pi},T,0)-\mbox{$\frac{1}{2}$}g_{0}(M^{0}_{\pi},T,0)-{\tilde{g}}_{0}(M^{\pm}_{\pi},T,H)$
(2.2)
$\displaystyle+\frac{M^{2}_{\pi}}{2F^{2}}\,g_{1}(M^{\pm}_{\pi},T,0)\,g_{1}(M^{0}_{\pi},T,0)-\frac{M^{2}_{\pi}}{8F^{2}}\,{\Big{\\{}g_{1}(M^{0}_{\pi},T,0)\Big{\\}}}^{2}$
$\displaystyle+\frac{M^{2}_{\pi}}{2F^{2}}\,g_{1}(M^{0}_{\pi},T,0)\,{\tilde{g}}_{1}(M^{\pm}_{\pi},T,H)+{\cal
O}(p^{8})\,,$

and relies on the kinematical Bose functions

$\displaystyle g_{0}({\cal M},T,0)$ $\displaystyle=$ $\displaystyle
T^{4}\,{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\rho^{-3}\,\exp\Big{(}-\frac{{\cal
M}^{2}}{4\pi
T^{2}}\rho\Big{)}\Bigg{[}S\Big{(}\frac{1}{\rho}\Big{)}-1\Bigg{]}\,,$
$\displaystyle g_{1}({\cal M},T,0)$ $\displaystyle=$
$\displaystyle\frac{T^{2}}{{4\pi}}\,{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\rho^{-2}\,\exp\Big{(}-\frac{{\cal
M}^{2}}{4\pi
T^{2}}\rho\Big{)}\Bigg{[}S\Big{(}\frac{1}{\rho}\Big{)}-1\Bigg{]}\,,$
$\displaystyle{\tilde{g}}_{0}(M^{\pm}_{\pi},T,H)$ $\displaystyle=$
$\displaystyle\frac{T^{2}}{{4\pi}}\,|qH|{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\rho^{-2}\,\Bigg{(}\frac{1}{\sinh(|qH|\rho/4\pi
T^{2})}-\frac{4\pi T^{2}}{|qH|\rho}\Bigg{)}$
$\displaystyle\times\,\exp\Big{(}-\frac{{(M^{\pm}_{\pi})}^{2}}{4\pi
T^{2}}\rho\Big{)}\Bigg{[}S\Big{(}\frac{1}{\rho}\Big{)}-1\Bigg{]}\,,$
$\displaystyle{\tilde{g}}_{1}(M^{\pm}_{\pi},T,H)$ $\displaystyle=$
$\displaystyle\frac{1}{16\pi^{2}}\,|qH|{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\rho^{-1}\,\Bigg{(}\frac{1}{\sinh(|qH|\rho/4\pi
T^{2})}-\frac{4\pi T^{2}}{|qH|\rho}\Bigg{)}$ (2.3)
$\displaystyle\times\,\exp\Big{(}-\frac{{(M^{\pm}_{\pi})}^{2}}{4\pi
T^{2}}\rho\Big{)}\Bigg{[}S\Big{(}\frac{1}{\rho}\Big{)}-1\Bigg{]}\,,$

with $S(z)$,

$S(z)=\sum_{n=-\infty}^{\infty}\exp(-\pi n^{2}z)\,,$ (2.4)

as the Jacobi theta function. These kinematical Bose functions depend on the
masses of the charged ($M^{\pm}_{\pi}$) and neutral ($M^{0}_{\pi}$) pions in a
magnetic field, namely,

$\displaystyle{(M^{\pm}_{\pi})}^{2}$ $\displaystyle=$ $\displaystyle
M^{2}_{\pi}+\frac{{\overline{l}}_{6}-{\overline{l}}_{5}}{48\pi^{2}}\,\frac{{|qH|}^{2}}{F^{2}}\,,$
$\displaystyle{(M^{0}_{\pi})}^{2}$ $\displaystyle=$ $\displaystyle
M^{2}_{\pi}+\frac{M^{2}}{F^{2}}\,K_{1}\,,$ (2.5)

where $K_{1}$ corresponds to the integral

$K_{1}=\frac{|qH|}{16\pi^{2}}\,{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\,\rho^{-1}\,\exp\Big{(}-\frac{M^{2}_{\pi}}{|qH|}\rho\Big{)}\,\Big{(}\frac{1}{\sinh(\rho)}-\frac{1}{\rho}\Big{)}\,,$
(2.6)

$q$ is the electric charge, and ${\overline{l}}_{5},{\overline{l}}_{6}$ are
renormalized next-to-leading order (NLO) low-energy effective constants. The
mass $\cal M$ appearing in the kinematical Bose functions $g_{0}$ and $g_{1}$
can either stand for $M^{\pm}_{\pi}$ or $M^{0}_{\pi}$. Finally, the mass
$M_{\pi}$ is the renormalized pion mass in zero magnetic field,

$M^{2}_{\pi}=M^{2}-\frac{{\overline{l}}_{3}}{32\pi^{2}}\,\frac{M^{4}}{F^{2}}+{\cal
O}(M^{6})\,,$ (2.7)

where $M$ ($F$) is the tree-level pion mass (pion decay constant).

##  3 Entropy Density

In the previous section we have defined the kinematical functions $g_{r}$ and
${\tilde{g}}_{r}$ that are dimensionful. In what follows it is more convenient
to use the dimensionless functions $h_{r}$ and ${\tilde{h}}_{r}$,

$h_{0}=\frac{g_{0}}{T^{4}}\,,\quad{\tilde{h}}_{0}=\frac{{\tilde{g}}_{0}}{T^{4}}\,,\qquad
h_{1}=\frac{g_{1}}{T^{2}}\,,\quad{\tilde{h}}_{1}=\frac{{\tilde{g}}_{1}}{T^{2}}\,,\qquad
h_{2}=g_{2}\,,\quad{\tilde{h}}_{2}={\tilde{g}}_{2}\,.$ (3.1)

In addition, instead of using absolute values of temperature, pion mass and
magnetic field strength, we prefer to work with the normalized and
dimensionless quantities $t,m$, and $m_{H}$ defined as

$t=\frac{T}{4\pi F}\,,\qquad m=\frac{M_{\pi}}{4\pi F}\,,\qquad
m_{H}=\frac{\sqrt{|qH|}}{4\pi F}\,.$ (3.2)

The common denominator represents the chiral symmetry breaking scale
$\Lambda_{\chi}\approx 4\pi F\approx 1\text{GeV}$. In the low-energy region
where chiral perturbation theory operates, the parameters $t,m$, and $m_{H}$
are small. In subsequent plots, the value of the (tree-level) pion decay
constant is $F=85.6\,\text{MeV}$ (see Ref. [46]).

The entropy density $s$ can be extracted from the pressure via

$s=\frac{\mbox{d}P}{\mbox{d}T}\,.$ (3.3)

The pressure in a homogeneous medium, up to the sign, is nothing but the
finite-temperature piece in the free energy density,

$P=-z^{T}\,.$ (3.4)

Derivatives of the Bose functions $g_{r}$ and ${\tilde{g}}_{r}$ with respect
to temperature are easily obtained with the relations

$\displaystyle\frac{\mbox{d}}{\mbox{d}T}\,g_{r}({\cal M},T,0)$
$\displaystyle=$ $\displaystyle\frac{2{\cal M}^{2}}{T}\,g_{r+1}({\cal
M},T,0)+\frac{d-2r}{T}\,g_{r}({\cal M},T,0)\,,$
$\displaystyle\frac{\mbox{d}}{\mbox{d}T}\,{\tilde{g}}_{r}(M^{\pm}_{\pi},T,H)$
$\displaystyle=$
$\displaystyle\frac{2M^{2}}{T}\,{\tilde{g}}_{r+1}(M^{\pm}_{\pi},T,H)+\frac{d-2r-2}{T}\,{\tilde{g}}_{r}(M^{\pm}_{\pi},T,H)$
(3.5) $\displaystyle+{\tilde{g}}^{[1]}_{r}(M^{\pm}_{\pi},T,H)\,,$

where

$\displaystyle{\tilde{g}}^{[1]}_{r}(M^{\pm}_{\pi},T,H)$ $\displaystyle=$
$\displaystyle\frac{T^{d-2r-2}}{{(4\pi)}^{r+1}}\,|qH|{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\rho^{r-\frac{d}{2}}\,\Bigg{(}\frac{|qH|\rho\coth(|qH|\rho/4\pi
T^{2})}{2\pi T^{3}\sinh(|qH|\rho/4\pi T^{2})}-\frac{8\pi T}{|qH|\rho}\Bigg{)}$
(3.6) $\displaystyle\times\,\exp\Big{(}-\frac{{(M^{\pm}_{\pi})}^{2}}{4\pi
T^{2}}\rho\Big{)}\Bigg{[}S\Big{(}\frac{1}{\rho}\Big{)}-1\Bigg{]}\,.$

On the basis of the representation for $z^{T}$, Eq. (2.2), the two-loop
entropy density – scaled by $1/T^{3}$ – amounts to

$\displaystyle\frac{s}{T^{3}}$ $\displaystyle=$ $\displaystyle
2s_{\pm}h_{1}(M^{\pm}_{\pi},T,0)+4h_{0}(M^{\pm}_{\pi},T,0)+s_{0}h_{1}(M^{0}_{\pi},T,0)+2h_{0}(M^{0}_{\pi},T,0)$
(3.7)
$\displaystyle+2s_{\pm}{\tilde{h}}_{1}(M^{\pm}_{\pi},T,H)+2{\tilde{h}}_{0}(M^{\pm}_{\pi},T,H)+{\tilde{h}}^{[1]}_{0}(M^{\pm}_{\pi},T,H)$
$\displaystyle-8\pi^{2}m^{2}\Big{\\{}2s_{\pm}h_{2}(M^{\pm}_{\pi},T,0)h_{1}(M^{0}_{\pi},T,0)+4h_{1}(M^{\pm}_{\pi},T,0)h_{1}(M^{0}_{\pi},T,0)$
$\displaystyle+2s_{0}h_{1}(M^{\pm}_{\pi},T,0)h_{2}(M^{0}_{\pi},T,0)+2s_{0}h_{2}(M^{0}_{\pi},T,0){\tilde{h}}_{1}(M^{\pm}_{\pi},T,H)$
$\displaystyle+2h_{1}(M^{0}_{\pi},T,0){\tilde{h}}_{1}(M^{\pm}_{\pi},T,H)+2s_{\pm}h_{1}(M^{0}_{\pi},T,0){\tilde{h}}_{2}(M^{\pm}_{\pi},T,H)$
$\displaystyle+h_{1}(M^{0}_{\pi},T,0){\tilde{h}}^{[1]}_{1}(M^{\pm}_{\pi},T,H)-s_{0}h_{1}(M^{0}_{\pi},T,0)h_{2}(M^{0}_{\pi},T,0)$
$\displaystyle-h_{1}(M^{0}_{\pi},T,0)h_{1}(M^{0}_{\pi},T,0)\Big{\\}}\,,$

with coefficients

$\displaystyle s_{\pm}$ $\displaystyle=$
$\displaystyle\frac{m^{2}}{t^{2}}+\frac{({\overline{l}}_{6}-{\overline{l}}_{5})m^{4}_{H}}{3t^{2}}\,,$
$\displaystyle s_{0}$ $\displaystyle=$
$\displaystyle\frac{m^{2}}{t^{2}}+\frac{m^{2}m_{H}^{2}}{t^{2}}\,{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\,\rho^{-1}\,\exp\Big{(}-\frac{m^{2}}{m^{2}_{H}}\rho\Big{)}\,\Big{(}\frac{1}{\sinh(\rho)}-\frac{1}{\rho}\Big{)}\,.$
(3.8)

The dimensionless functions ${\tilde{h}}^{[1]}_{0}(M^{\pm}_{\pi},T,H)$ and
${\tilde{h}}^{[1]}_{1}(M^{\pm}_{\pi},T,H)$ are

${\tilde{h}}^{[1]}_{0}(M^{\pm}_{\pi},T,H)=\frac{{\tilde{g}}^{[1]}_{0}(M^{\pm}_{\pi},T,H)}{T^{3}}\,,\qquad{\tilde{h}}^{[1]}_{1}(M^{\pm}_{\pi},T,H)=\frac{{\tilde{g}}^{[1]}_{1}(M^{\pm}_{\pi},T,H)}{T}\,.$
(3.9)

Figure 1: [Color online] Scaled entropy density $s/T^{3}$: Sum of one- and
two-loop contributions for $T=54\,\text{MeV}$ (left) and $T=108\,\text{MeV}$
(right) in terms of the dimensionless parameters $m,m_{H}$.

In Fig. 1 we plot the scaled entropy density $s/T^{3}$ – sum of one- and two-
loop contributions given in Eq. (3.7) – in terms of magnetic field strength
($m_{H}$) and pion mass ($m$) at the fixed temperatures
$T=\\{54,108\\}\,\text{MeV}$ ($t=\\{0.05,0.1\\}$). The entropy density
decreases when the magnetic field becomes stronger and it also decreases when
the masses of the pions grow. The impact of the pion mass, however, is more
pronounced than the dependence on the magnetic field. The limits $M\to 0$
(chiral limit) and $H\to 0$ do not pose any problems. In particular, taking
the double limit $\\{M,H\\}\to 0$, the scaled entropy density tends to the
value describing the noninteracting Bose gas,

$\frac{s(T,0,0)}{T^{3}}=\frac{2\pi^{2}}{15}\approx 1.32\,.$ (3.10)

Figure 2: [Color online] Subtracted scaled entropy density $s_{H}/T^{3}$: Sum
of one- and two-loop contributions for $T=54\,\text{MeV}$ (left) and
$T=108\,\text{MeV}$ (right) in terms of the dimensionless parameters
$m,m_{H}$.

To better assess the effect of the magnetic field on entropy, we subtract the
$H$=0 contribution, i.e., consider the quantity

$\frac{s_{H}}{T^{3}}=\frac{s(T,M_{\pi},H)-s(T,M_{\pi},0)}{T^{3}}$ (3.11)

that measures the influence of the magnetic field. In Fig. 2 we plot
$s_{H}/T^{3}$ in terms of magnetic field strength ($m_{H}$) and pion mass
($m$) at the same fixed temperatures $T=\\{54,108\\}\,\text{MeV}$. Overall, in
presence of the magnetic field, the entropy density drops. For a given fixed
$m_{H}$, the effect becomes most pronounced when the chiral limit is
approached.

Let us now focus on the real world defined by the physical point
$M_{\pi}=140\,\text{MeV}$ ($m=0.130$). In Fig. 3, on the LHS, we plot the
scaled two-loop entropy density $s/T^{3}$ according to Eq. (3.7) in function
of magnetic field strength ($m_{H}$) and temperature ($t$). Clearly, the
impact of temperature is predominant and the slight decrease of the entropy
density caused by the magnetic field is hardly visible. Therefore, we rather
consider the quantity $s_{H}/T^{3}$ defined by Eq. (3.11) that isolates the
effect of the magnetic field. Indeed, as illustrated on the RHS of Fig. 3, the
presence of the magnetic field lowers the entropy density in an interesting
and nontrivial way: the effect is most pronounced around $t\approx 0.035$,
corresponding to $T\approx 40\,\text{MeV}$. As we discuss below, the increase
of order – as witnessed by the drop in entropy density – is reflected in the
behavior of the finite-temperature quark condensate.

Figure 3: [Color online] Scaled QCD entropy density $s/T^{3}$ (LHS) and scaled
subtracted QCD entropy density $s_{H}/T^{3}$ (RHS). Both quantities in terms
of the dimensionless parameters $m_{H}$ (magnetic field strength) and $t$
(temperature).

While our results regarding the dependence of entropy density on magnetic
field strength, temperature, and arbitrary pion mass are new to the best of
our knowledge, the behavior of the entropy density at the physical pion mass
has been explored in the hadron resonance gas model [13], as well as in the
(2+1) flavor Polyakov-loop quark-meson model [21]. Indeed, both references
also report a decrease of entropy density caused by the magnetic field in the
relevant region $T\lessapprox 100\,\text{MeV}$ that is accessible with chiral
perturbation theory. The comparison, however, is only qualitative as the
latter reference is based on three flavors and the hadron resonance gas model
also includes more particles than just the three pions.

##  4 Magnetization

The magnetization ${\mathfrak{M}}$ is defined as the (negative) derivative of
the free energy density with respect to the magnetic field,

${\mathfrak{M}}(T,M,H)=-\frac{\mbox{d}z}{\mbox{d}|qH|}\,.$ (4.1)

It contains both temperature-dependent and temperature-independent
contributions. In this study we focus on the former.

The finite-temperature free energy density $z^{T}$, Eq. (2.2), involves
various types of kinematical Bose functions. Corresponding derivatives with
respect to the magnetic field are readily obtained via

$\displaystyle\frac{\mbox{d}}{\mbox{d}|qH|}\,g_{r}(M^{\pm},T,0)$
$\displaystyle=$
$\displaystyle-\frac{{\overline{l}}_{6}-{\overline{l}}_{5}}{24\pi^{2}}\,\frac{|qH|}{F^{2}}\,g_{r+1}(M^{\pm},T,0)\,,$
$\displaystyle\frac{\mbox{d}}{\mbox{d}|qH|}\,g_{r}(M^{0},T,0)$
$\displaystyle=$
$\displaystyle-\frac{M^{2}}{F^{2}}\,\frac{\mbox{d}K_{1}}{\mbox{d}|qH|}\,g_{r+1}(M^{0},T,0)\,,$
$\displaystyle\frac{\mbox{d}}{\mbox{d}|qH|}\,{\tilde{g}}_{r}(M^{\pm}_{\pi},T,H)$
$\displaystyle=$
$\displaystyle\frac{1}{|qH|}\,{\tilde{g}}_{r}(M^{\pm}_{\pi},T,H)-\frac{{\overline{l}}_{6}-{\overline{l}}_{5}}{24\pi^{2}}\,\frac{|qH|}{F^{2}}\,{\tilde{g}}_{r+1}(M^{\pm}_{\pi},T,H)$
(4.2) $\displaystyle+{\tilde{g}}^{[H]}_{r}(M^{\pm}_{\pi},T,H)\,,$

where

$\displaystyle{\tilde{g}}^{[H]}_{r}(M^{\pm}_{\pi},T,H)$ $\displaystyle=$
$\displaystyle\frac{T^{d-2r-2}}{{(4\pi)}^{r+1}}\,|qH|{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\rho^{r-\frac{d}{2}}\,\Bigg{(}-\frac{\rho\coth(|qH|\rho/4\pi
T^{2})}{4\pi T^{2}\sinh(|qH|\rho/4\pi T^{2})}+\frac{4\pi
T^{2}}{{|qH|}^{2}\rho}\Bigg{)}$ (4.3)
$\displaystyle\times\,\exp\Big{(}-\frac{{(M^{\pm}_{\pi})}^{2}}{4\pi
T^{2}}\rho\Big{)}\Bigg{[}S\Big{(}\frac{1}{\rho}\Big{)}-1\Bigg{]}\,.$

The finite-temperature magnetization ${\mathfrak{M}}(T,M,H)$ – up to two loops
and scaled by $1/T^{2}$ – then takes the form

$\displaystyle\frac{\mathfrak{M}}{T^{2}}$ $\displaystyle=$ $\displaystyle
m_{\pm}h_{1}(M^{\pm}_{\pi},T,0)+\frac{m_{0}}{2}\,h_{1}(M^{0}_{\pi},T,0)+\frac{t^{2}}{m^{2}_{H}}\,{\tilde{h}}_{0}(M^{\pm}_{\pi},T,H)$
(4.4)
$\displaystyle+m_{\pm}{\tilde{h}}_{1}(M^{\pm}_{\pi},T,H)+{\tilde{h}}^{[H]}_{0}(M^{\pm}_{\pi},T,H)$
$\displaystyle-4\pi^{2}m^{2}\Big{\\{}2m_{\pm}h_{2}(M^{\pm}_{\pi},T,0)h_{1}(M^{0}_{\pi},T,0)+2m_{0}h_{1}(M^{\pm}_{\pi},T,0)h_{2}(M^{0}_{\pi},T,0)$
$\displaystyle+2m_{0}h_{2}(M^{0}_{\pi},T,0){\tilde{h}}_{1}(M^{\pm}_{\pi},T,H)+\frac{2t^{2}}{m^{2}_{H}}\,h_{1}(M^{0}_{\pi},T,0){\tilde{h}}_{1}(M^{\pm}_{\pi},T,H)$
$\displaystyle+2m_{\pm}h_{1}(M^{0}_{\pi},T,0){\tilde{h}}_{2}(M^{\pm}_{\pi},T,H)+2h_{1}(M^{0}_{\pi},T,0){\tilde{h}}^{[H]}_{1}(M^{\pm}_{\pi},T,H)$
$\displaystyle-m_{0}h_{2}(M^{0}_{\pi},T,0)h_{1}(M^{0}_{\pi},T,0)\Big{\\}}\,,$

with coefficients

$\displaystyle m_{\pm}$ $\displaystyle=$
$\displaystyle-\frac{2({\overline{l}}_{6}-{\overline{l}}_{5})m^{2}_{H}}{3}\,,$
$\displaystyle m_{0}$ $\displaystyle=$
$\displaystyle-m^{2}{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\,\rho^{-1}\,\exp\Big{(}-\frac{m^{2}}{m^{2}_{H}}\rho\Big{)}\,\Big{(}\frac{1}{\sinh(\rho)}-\frac{1}{\rho}\Big{)}$
(4.5)
$\displaystyle-\frac{m^{4}}{m_{H}^{2}}\,{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\,\exp\Big{(}-\frac{m^{2}}{m^{2}_{H}}\rho\Big{)}\,\Big{(}\frac{1}{\sinh(\rho)}-\frac{1}{\rho}\Big{)}\,.$

The dimensionless functions ${\tilde{h}}^{[H]}_{0}(M^{\pm}_{\pi},T,H)$ and
${\tilde{h}}^{[H]}_{1}(M^{\pm}_{\pi},T,H)$ are

${\tilde{h}}^{[H]}_{0}(M^{\pm}_{\pi},T,H)=\frac{{\tilde{g}}^{[H]}_{0}(M^{\pm}_{\pi},T,H)}{T^{2}}\,,\qquad{\tilde{h}}^{[H]}_{1}(M^{\pm}_{\pi},T,H)={\tilde{g}}^{[H]}_{1}(M^{\pm}_{\pi},T,H)\,.$
(4.6)

Figure 4: [Color online] Scaled finite-temperature magnetization
${\mathfrak{M}}/T^{2}$: Sum of one- and two-loop contributions for
$T=54\,\text{MeV}$ (left) and $T=108\,\text{MeV}$ (right) in terms of the
dimensionless parameters $m,m_{H}$.

In Fig. 4 we plot ${\mathfrak{M}}/T^{2}$ in terms of magnetic field strength
($m_{H}$) and pion mass ($m$) at the fixed temperatures
$T=\\{54,108\\}\,\text{MeV}$ ($t=\\{0.05,0.1\\}$). The finite-temperature
magnetization turns out to be negative in the entire parameter region under
consideration. Note that its dependence on magnetic field strength is
nontrivial: remarkably, the magnitude of the finite-temperature magnetization
initially grows as the magnetic field increases, but then declines in stronger
magnetic fields. For fixed magnetic field strength, the effect becomes more
pronounced as one approaches the chiral limit. On the other hand, the limit
$H\to 0$ is trivial: the finite-temperature magnetization simply vanishes as
it should. Accordingly, there is no need to subtract the $H$=0 contribution
(as for the entropy density): the quantity ${\mathfrak{M}}(T,M,H)$ already
measures the impact of the magnetic field.

In Fig. 5 we explore the dependence of ${\mathfrak{M}}/T^{2}$ on magnetic
field strength and temperature in the real world with physical pion mass
$M_{\pi}=140\,\text{MeV}$ ($m=0.130$). The induced finite-temperature
magnetization is negative and its magnitude grows as both magnetic field and
temperature increase. The dependence of ${\mathfrak{M}}/T^{2}$ on the magnetic
field – at fixed temperature – implies that the QCD vacuum behaves as a
diamagnetic medium at low temperatures and in weak magnetic fields.

Figure 5: [Color online] Scaled finite-temperature QCD magnetization
${\mathfrak{M}}/T^{2}$ in terms of the dimensionless parameters $m_{H}$
(magnetic field strength) and $t$ (temperature).

This conclusion fully agrees with studies based on lattice QCD [27, 3, 28,
29], on the (2+1) flavor Polyakov-loop quark-meson model [21], and on the
three-flavor quark-meson model with $U_{A}(1)$ anomaly [30]. In these
references the finite-temperature magnetic susceptibility $\chi_{B}(T)$,

$\chi_{B}(T)=\lim_{H\to
0}\,-\frac{{\mbox{d}}^{2}}{\mbox{d}{|qH|}^{2}}\,\Big{\\{}z-z_{0}\Big{\\}}\,,$
(4.7)

was considered. At lower temperatures where only pions are the relevant
degrees of freedom, the authors indeed find that the magnetic susceptibility
is negative. In particular, its value at $T=90\,\text{MeV}$, reported in Ref.
[3],

$\chi_{B}(T\\!=\\!90\,\text{MeV})=-0.002(2)\,,$ (4.8)

is perfectly consistent with our two-loop CHPT analysis where we extract333We
have used $F=85.6\,\text{MeV}$ from Ref.[46] and – in accordance with Ref.
[13] – $M_{\pi}=135\,\text{MeV}$.

$\chi_{B}(T\\!=\\!90\,\text{MeV})=-0.00245\,.$ (4.9)

##  5 Finite-Temperature Quark Condensate

It is illuminating to include the quark condensate
${\langle{\bar{q}}q\rangle}$ into discussion and compare its properties at low
temperatures and weak magnetic fields with the behavior of the entropy
density. The two-loop representation for the quark condensate, decomposed into
$T$=0 and finite-temperature contributions,

$\langle{\bar{q}}q\rangle=\langle
0|{\bar{q}}q|0\rangle+{\langle{\bar{q}}q\rangle}^{T}\,,$ (5.1)

has been derived in Ref. [26]. Here we just quote the result for the finite-
temperature portion ${\langle{\bar{q}}q\rangle}^{T}$ that is relevant to make
our point,444The quantity ${\langle 0|{\bar{q}}q|0\rangle}_{0}$ is the quark
condensate in the chiral limit at zero temperature and zero magnetic field.

$\frac{{\langle{\bar{q}}q\rangle}^{T}}{{\langle
0|{\bar{q}}q|0\rangle}_{0}}\,{\Big{(}1-\frac{M^{2}_{\pi}}{32\pi^{2}F^{2}}\,(2{\overline{l}}_{3}-1)\Big{)}}^{-1}=-\Big{\\{}\frac{q_{1}}{F^{2}}T^{2}+\frac{q_{2}}{F^{4}}T^{4}+{\cal
O}(T^{6})\Big{\\}}\,,$ (5.2)

with coefficients

$\displaystyle q_{1}$ $\displaystyle=$ $\displaystyle
h_{1}(M^{\pm}_{\pi},T,0)+\mbox{$\frac{1}{2}$}a_{0}h_{1}(M^{0}_{\pi},T,0)+{\tilde{h}}_{1}(M^{\pm}_{\pi},T,H)\,,$
$\displaystyle q_{2}$ $\displaystyle=$
$\displaystyle+\mbox{$\frac{1}{2}$}h_{1}(M^{\pm}_{\pi},T,0)h_{1}(M^{0}_{\pi},T,0)+\mbox{$\frac{1}{2}$}h_{1}(M^{0}_{\pi},T,0){\tilde{h}}_{1}(M^{\pm}_{\pi},T,H)$
(5.3)
$\displaystyle-\mbox{$\frac{1}{8}$}h_{1}(M^{0}_{\pi},T,0)h_{1}(M^{0}_{\pi},T,0)-\mbox{$\frac{1}{2}$}\frac{m^{2}}{t^{2}}h_{1}(M^{0}_{\pi},T,0)h_{2}(M^{\pm}_{\pi},T,0)$
$\displaystyle-\mbox{$\frac{1}{2}$}a_{0}\frac{m^{2}}{t^{2}}h_{1}(M^{\pm}_{\pi},T,0)h_{2}(M^{0}_{\pi},T,0)-\mbox{$\frac{1}{2}$}a_{0}\frac{m^{2}}{t^{2}}{\tilde{h}}_{1}(M^{\pm}_{\pi},T,H)h_{2}(M^{0}_{\pi},T,0)$
$\displaystyle+\mbox{$\frac{1}{4}$}a_{0}\frac{m^{2}}{t^{2}}h_{1}(M^{0}_{\pi},T,0)h_{2}(M^{0}_{\pi},T,0)-\mbox{$\frac{1}{2}$}\frac{m^{2}}{t^{2}}h_{1}(M^{0}_{\pi},T,0){\tilde{h}}_{2}(M^{\pm}_{\pi},T,H)\,.$

The NLO mass correction $a_{0}$ reads

$a_{0}=\frac{\mbox{d}{(M^{0}_{\pi})}^{2}}{\mbox{d}M^{2}_{\pi}}=1+\frac{K_{1}}{F^{2}}+\frac{M^{2}_{\pi}}{F^{2}}\,\frac{\mbox{d}K_{1}}{\mbox{d}M^{2}_{\pi}}\,,$
(5.4)

where the integral $\mbox{d}K_{1}/\mbox{d}M^{2}_{\pi}$ is

$\frac{\mbox{d}K_{1}}{\mbox{d}M^{2}_{\pi}}=-\frac{1}{16\pi^{2}}\,{\int}_{\\!\\!\\!0}^{\infty}\mbox{d}\rho\,\exp\Big{(}-\frac{M^{2}_{\pi}}{|qH|}\rho\Big{)}\,\Big{(}\frac{1}{\sinh(\rho)}-\frac{1}{\rho}\Big{)}\,.$
(5.5)

Figure 6: [Color online] Scaled finite-temperature quark condensate: Sum of
one- and two-loop contributions for $T=54\,\text{MeV}$ (left) and
$T=108\,\text{MeV}$ (right) in terms of the dimensionless parameters $m$ (pion
mass) and $m_{H}$ (magnetic field strength).

In Fig. 6 we plot the scaled finite-temperature quark condensate, i.e., the
dimensionless quantity

$-\Big{(}q_{1}+q_{2}\,\frac{T^{2}}{F^{2}}\Big{)}\,,$ (5.6)

in terms of magnetic field strength ($m_{H}$) and pion mass ($m$) for the two
temperatures $T=\\{54,108\\}\,\text{MeV}$. The condensate grows as magnetic
field strength or pion masses increase, but the dependence on pion mass is
more pronounced.

To better appreciate the impact of the magnetic field – as for the entropy
density – we subtract the $H$=0 portion and define the dimensionless quantity

${\langle{\bar{q}}q\rangle}^{T}_{H}=\frac{c_{0}}{T^{2}}\,\Big{\\{}{\langle{\bar{q}}q\rangle}^{T}-{{\langle{\bar{q}}q\rangle}^{T}|}_{H=0}\Big{\\}}\,,$
(5.7)

where

$c_{0}=\frac{F^{2}}{{\langle
0|{\bar{q}}q|0\rangle}_{0}}\,{\Big{(}1-\frac{M^{2}_{\pi}}{32\pi^{2}F^{2}}(2{\overline{l}}_{3}-1)\Big{)}}^{-1}\,.$
(5.8)

Notice that ${\langle{\bar{q}}q\rangle}^{T}_{H}$ captures the effect of the
magnetic field in the finite-temperature quark condensate.

Figure 7: [Color online] Scaled subtracted finite-temperature quark condensate
${\langle{\bar{q}}q\rangle}^{T}_{H}$: Sum of one- and two-loop contributions
for $T=54\,\text{MeV}$ (left) and $T=108\,\text{MeV}$ (right) in terms of the
dimensionless parameters $m$ (pion mass) and $m_{H}$ (magnetic field
strength).

Inspecting the corresponding Fig. 7, it now becomes obvious that
${\langle{\bar{q}}q\rangle}^{T}_{H}$ – i.e., the subtracted finite-temperature
order parameter – is correlated with the subtracted entropy density (cf. Fig.
2): the enhancement of the order parameter caused by the magnetic field is
reflected in the decrease of the entropy density. Note that in either
subtracted quantity the influence of the magnetic field is largest in the
chiral limit.

Figure 8: [Color online] Scaled finite-temperature QCD quark condensate in
terms of the dimensionless parameters $m_{H}$ and $t$. LHS: Full contribution
$-(q_{1}+q_{2}T^{2}/F^{2})$. RHS: Subtracted contribution
${\langle{\bar{q}}q\rangle}^{T}_{H}$.

Let us finally discuss the connection between quark condensate and entropy
density at the physical point $M_{\pi}=140\,\text{MeV}$. On the LHS of Fig. 8
we depict the scaled finite-temperature QCD quark condensate,

$-\Big{(}q_{1}+q_{2}\,\frac{T^{2}}{F^{2}}\Big{)}\,,$ (5.9)

in function of magnetic field strength and temperature. The influence of the
magnetic field is barely visible in comparison to the predominant dependence
on temperature. This motivates us to consider the subtracted QCD quark
condensate,

${\langle{\bar{q}}q\rangle}^{T}_{H}\,,$ (5.10)

that we show on the RHS of Fig. 8. Now the correlation between order parameter
and entropy density (cf. RHS of Fig. 3) becomes obvious: the magnetic field
enhances the finite-temperature quark condensate which is reflected in the
decrease of the entropy density. It should be noted that the correlation is
not one-to-one, but just is of qualitative nature. For example, as Fig. 8
illustrates, the effect of the magnetic field is most pronounced around
$t\approx 0.085$, or $T\approx 90\,\text{MeV}$, while for the entropy density
the most distinct drop occurs around $T\approx 40\,\text{MeV}$.

##  6 Conclusions

Within the framework of two-flavor chiral perturbation theory in a magnetic
background, we have derived the two-loop representations for the entropy
density and the magnetization. In various figures we have explored how these
quantities depend on temperature, magnetic field strength and pion mass.

We observe that the entropy density – at fixed temperature – drops when the
magnetic field becomes stronger and that its decrease is most pronounced in
the chiral limit. At the physical point $M_{\pi}=140\,\text{MeV}$, the
decrease of the entropy density is most distinct around $T\approx
40\,\text{MeV}$. The lowering of the entropy density in an external magnetic
field has also been reported in model-based studies. Here, however, we have
provided a fully systematic investigation.

The magnetization at finite temperature is negative in the entire parameter
domain accessible by CHPT, i.e., $T,M_{\pi},\sqrt{|qH|}\lessapprox
0.2\,\text{GeV}$. Its dependence on magnetic field strength is nontrivial: the
magnitude of the finite-temperature magnetization initially grows as the
magnetic field increases, but then declines in stronger magnetic fields. This
behavior is most distinct in the chiral limit. In the real world
($M_{\pi}=140\,\text{MeV}$) the QCD vacuum behaves as a diamagnetic medium at
low temperatures and weak magnetic fields, in accordance with lattice QCD and
model-based studies.

Finally we have addressed the connection between the behavior of the entropy
density and the finite-temperature quark condensate in a magnetic field. The
connection becomes most obvious when the $H$=0 pieces in either quantity are
subtracted, such that the effect of the magnetic field is revealed: we then
observe that the enhancement of the finite-temperature quark condensate in a
magnetic field is reflected in a decrease of the entropy density.

Although the thermomagnetic properties of quantum chromodynamics in a
homogeneous external magnetic field have been explored before by various
authors, a comprehensive investigation of the entropy density and the
magnetization in the regime of low temperatures and weak magnetic fields seems
to be lacking. Here we have provided a fully systematic analysis relying on
two-flavor chiral perturbation theory.

While we have focused on effects emerging at finite temperature, it would be
interesting to also discuss zero-temperature effects. This, however, requires
a detailed analysis of the zero-temperature vacuum polarization that involves
the ”unphysical” next-to-leading order low-energy constant
${\overline{h}}_{2}$ that yet has to be determined. A detailed investigation
including the magnetic susceptibility and the associated diamagnetic and
paramagnetic phases – as well as an extensive comparison with the pertinent
literature – will be given in a forthcoming study [47]. It would also be
interesting to extend the present case to three-flavor chiral perturbation
theory, i.e., to include kaons and the $\eta$-particle. An even more ambitious
task is to elevate the present two-loop analysis to the three-loop level.
Corresponding work is in progress.

## Acknowledgments

The author thanks J. Bijnens and H. Leutwyler for correspondence.

## References

  * Buividovich et al. [2010] P. V. Buividovich, M. N. Chernodub, E. V. Luschevskaya, and M. I. Polikarpov, Nucl. Phys. B 826, 313 (2010).
  * Bali et al. [2013] G. S. Bali, F. Bruckmann, G. Endrödi, F. Gruber, and A. Schäfer, JHEP 04, 130 (2013).
  * Bali et al. [2014] G. S. Bali, F. Bruckmann, G. Endrödi, S. D. Katz, and A. Schäfer, JHEP 08, 177 (2014).
  * Bali et al. [2014] G. S. Bali, F. Bruckmann, G. Endrödi, and A. Schäfer, Phys. Rev. Lett. 112, 042301 (2014).
  * Braguta et al. [2019] V. V. Braguta, M. N. Chernodub, A. Y. Kotov, A. V. Molochkov, and A. A. Nikolaev, Phys. Rev. D 100, 114503 (2019).
  * Scavenius et al. [2001] O. Scavenius, Á. Mócsy, I. N. Mishustin, and D. H. Rischke, Phys. Rev. C 64, 045202 (2001).
  * Fayazbakhsh and Sadooghi [2014] S. Fayazbakhsh and N. Sadooghi, Phys. Rev. D 90, 105030 (2014).
  * Farias et al [2014] R. L. S. Farias, V. S. Timóteo, S. S. Avancini, M. B. Pinto, and G. Krein, Eur. Phys. J. A 53, 101 (2017).
  * Avancini et al. [2020] S. S. Avancini, R. L. S. Farias, M. B. Pinto, T. E. Restrepo, and W. R. Tavares, arXiv:2008.10720.
  * Chakrabarty [1996] S. Chakrabarty, Phys. Rev. D 54, 1306 (1996).
  * Cohen and Werbos [2009] T. D. Cohen and E. S. Werbos, Phys. Rev. C 80, 015203 (2009).
  * Schaefer et al. [2010] B.-J. Schäfer, M. Wagner, and J. Wambach, Phys. Rev. D 81, 074013 (2010).
  * Endrodi [2013] G. Endrödi, JHEP 04, 023 (2013).
  * Tsue et al [2015] Y. Tsue, J. da Providencia, C. Providencia, M. Yamamura, and H. Bohr, Prog. Theor. Exp. Phys. 2015, 103D01.
  * Skalozub and Minaiev [2016] V. Skalozub and P. Minaiev, arXiv:1612.00216.
  * Pang et al. [2016] L.-G. Pang, G. Endrödi, and H. Petersen, Phys. Rev. C 93, 044919 (2016).
  * Tawfik et al [2016] A. N. Tawfik, A. M. Diab, N. Ezzelarab, and A. G. Shalaby, Adv. High Energy Phys. 2016, 1381479.
  * Rath and Patra [2017] S. Rath and B. K. Patra, JHEP 12, 098 (2017).
  * Tawfik et al [2018] A. N. Tawfik, A. M. Diab, and M. T. Hussein, J. Exp. Theor. Phys. 126, 620 (2018).
  * Karmakar et al. [2019] B. Karmakar, R. Ghosh, A. Bandyopadhyay, N. Haque, and M. G. Mustafa, Phys. Rev. D 99, 094002 (2019).
  * Li et al. [2019] X. Li, W. Fu, and Y. Liu, Phys. Rev. D 99, 074029 (2019).
  * Rath and Patra [2019] S. Rath and B. K. Patra, Eur. Phys. J. A 55, 220 (2019).
  * Koothottil and Bannur [2020] S. Koothottil and V. M. Bannur, Phys. Rev. C 102, 015206 (2020).
  * Hofmann [2019] C. P. Hofmann, Phys. Rev. D 99, 014030 (2019).
  * Hofmann [2020] C. P. Hofmann, Phys. Rev. D 101, 114031 (2020).
  * Hofmann [2020] C. P. Hofmann, Phys. Rev. D 102, 094010 (2020).
  * Bonati et al. [2014] C. Bonati, M. D’Elia, M. Mariti, F. Negro, and F. Sanfilippo, Phys. Rev. D 89, 054506 (2014).
  * Levkova and DeTar [2014] L. Levkova and C. DeTar, Phys. Rev. Lett. 112, 012002 (2014).
  * Bali et al. [2020] G. S. Bali, G. Endrödi, and S. Piemonte, JHEP 07, 183 (2020).
  * Kamikado and Kanazawa [2015] K. Kamikado and T. Kanazawa, JHEP 01, 129 (2015).
  * Leutwyler [1995] H. Leutwyler, in Hadron Physics 94 – Topics on the Structure and Interaction of Hadronic Systems, edited by V. E. Herscovitz, C. A. Z. Vasconcellos and E. Ferreira (World Scientific, Singapore, 1995), p. 1.
  * Manohar [1997] A. V. Manohar, in Perturbative and Nonperturbative Aspects of Quantum Field Theory, edited by H. Latal and W. Schweiger (Springer, New York, 1997), p.311.
  * Scherer [2003] S. Scherer, Adv. Nucl. Phys. 27, 277 (2003).
  * Goity [2004] J. L. Goity, Czech. J. Phys. 51, B35 (2001).
  * Brauner [2010] T. Brauner, Symmetry 2, 609 (2010).
  * Shushpanov and Smilga [1997] I. A. Shushpanov and A. V. Smilga, Phys. Lett. B 402, 351 (1997).
  * Agasian and Shushpanov [2000] N. O. Agasian and I. A. Shushpanov, Phys. Lett. B 472, 143 (2000).
  * Agasian [2000] N. O. Agasian, Phys. Lett. B 488, 39 (2000).
  * Agasian and Shushpanov [2001] N. O. Agasian and I. A. Shushpanov, JHEP 10, 006 (2001).
  * Cohen et al. [2007] T. D. Cohen, D. A. McGady, and E. S. Werbos, Phys. Rev. C 76, 055201 (2007).
  * Werbos [2000] E. S. Werbos, Phys. Rev. C 77, 065202 (2008).
  * Agasian and Fedorov [2008] N. O. Agasian and S. M. Fedorov, Phys. Lett. B 663, 445 (2008).
  * Andersen [2012a] J. O. Andersen, Phys. Rev. D 86, 025020 (2012).
  * Andersen [2012b] J. O. Andersen, JHEP 10, 005 (2012).
  * Brauner and Kadam [2017] T. Brauner and S. V. Kadam, JHEP 11, 103 (2017).
  * Aoki et al. [2020] S. Aoki et al., Eur. Phys. J. C 80, 113 (2020).
  * Hofmann [2020c] C. P. Hofmann, in preparation.


# Complex Robotic Manipulation via Graph-Based Hindsight Goal Generation

Zhenshan Bing∗
Department of Informatics
Technical University of Munich, Germany
<EMAIL_ADDRESS>
&Matthias Brucker∗
Munich School of Engineering
Technical University of Munich, Germany
<EMAIL_ADDRESS>
&Kai Huang
School of Data and Computer Science
Sun Yat-sen University, China
<EMAIL_ADDRESS>
&Fabrice O. Morin, Alois Knoll
Department of Informatics
Technical University of Munich, Germany
{morinf<EMAIL_ADDRESS>

###### Abstract

Reinforcement learning algorithms such as hindsight experience replay (HER)
and hindsight goal generation (HGG) have been able to solve challenging
robotic manipulation tasks in multi-goal settings with sparse rewards. HER
achieves its training success through hindsight replays of past experience
with heuristic goals, but under-performs in challenging tasks in which goals
are difficult to explore. HGG enhances HER by selecting intermediate goals
that are easy to achieve in the short term and promising to lead to target
goals in the long term. This guided exploration makes HGG applicable to tasks
in which target goals are far away from the object’s initial position.
However, HGG is not applicable to manipulation tasks with obstacles because
the euclidean metric used for HGG is not an accurate distance metric in such
environments. In this paper, we propose graph-based hindsight goal generation
(G-HGG), an extension of HGG selecting hindsight goals based on shortest
distances in an obstacle-avoiding graph, which is a discrete representation of
the environment. We evaluated G-HGG on four challenging manipulation tasks
with obstacles, where significant enhancements in both sample efficiency and
overall success rate are shown over HGG and HER. Videos can be viewed at
https://sites.google.com/view/demos-g-hgg/.

##  1 Introduction

In recent years, deep reinforcement learning (RL) has made significant
progress, with RL concepts being successfully applied to decision-making
problems in robotics, which include, but not limited to, helicopter control
[19], hitting a baseball [22], door opening [5], screwing a cap onto a bottle
[17], and object manipulation [1]. To train a meaningful policy for such
tasks, neural networks are used as function approximators for learning a value
function to optimize a long-term expected return. Estimation of the expected
return is based on a reward function, which is highly associated with the
tasks and therefore must be thoroughly shaped for policy optimization.

In most real-world applications, where a concrete representation of efficient
or even admissible behavior is unknown, the design of a reward function is
challenging and time-consuming, thereby hindering the wide applications of RL.
Consequently, algorithms that can support learning from sparse rewards are
highly desirable, e.g., a binary signal indicating successful task completion;
since sparse rewards are easy to derive from the task definition without
further engineering. Andrychowicz et al. [1] introduced an algorithm called “
hindsight experience replay (HER)”, which improves the success of off-policy
RL algorithms in multi-goal RL problems with sparse rewards. The concept
behind HER is to first learn with hand-crafted heuristic intermediate goals
that are easy to achieve, and then continue with more difficult goals.
Precisely, HER constructs hindsight goals from previously achieved states,
replays known trajectories with these hindsight goals, and trains the goal-
dependent value function based on the results. While HER has proven to work
efficiently in environments where goals can be easily reached through random
explorations [23], it fails to reach goals that are far away from initial
states and hard to reach. Due to random exploration and the heuristic choice
of hindsight goals from achieved states, hindsight goals keep being
distributed around the initial state, far away from the target goals, which
will never be reached since no reward signal is obtained.

Hindsight goal generation (HGG) [24] tackles the aforementioned problem by
using intermediate hindsight goals as an implicit curriculum to guide
exploration towards target goals. HGG aims at choosing hindsight goals that
are both easy to achieve and challenging enough to help the function
approximator learn how to achieve the target goals eventually. In HGG, the
choice of hindsight goals is based on two criteria: the current value function
(as much knowledge as possible about how to reach the hindsight goals) and the
Wasserstein distance between the target goal distribution and the distribution
of potential hindsight goals (goals as close as possible to the target goal
distribution). The resulting Wasserstein-Barycenter problem is discretely
solved using the euclidean distance metric between two goals sampled from the
potential hindsight goal distribution and the target goal distribution. While
HGG demonstrates higher sample efficiency than HER in environments where the
euclidean metric is applicable, it fails in environments with obstacles, where
the shortest obstacle-avoiding distance between two goals cannot be computed
with the euclidean metric.

In this paper, we introduce graph-based hindsight goal generation (G-HGG), an
extension of HGG, designed for sparse-reward RL in robotic object manipulation
environments with obstacles. To make G-HGG applicable to environments with
obstacles, the euclidean metric is replaced by a graph-based distance metric:
the environment’s goal space is represented by a graph consisting of discrete
goals as vertices and obstacle-avoiding connections between the vertices as
edges. The distance between two goals are then approximated by the shortest
path on the graph between the two vertices that are closest to the two goals.
For verification, we evaluated G-HGG on four different challenging object
manipulation environments. With the graph-based distance metric, a significant
enhancement in both sample efficiency and overall success rate is demonstrated
over HGG and HER. Ablation studies in the appendix (See supplementary
material) show the robustness of G-HGG in terms of discretization density.

##  2 Background

Goal-Conditioned RL  In goal-conditioned RL, an agent interacts with its
environment to reach some goals, which can be modeled as a goal-conditioned
Markov decision process (MDP) with a state space $\mathcal{S}$, an action
space $\mathcal{A}$, a goal space $\mathcal{G}$, a probabilistic transition
function $P:S\times\mathcal{A}\rightarrow\mathcal{S}$, a reward function
$r_{g}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and a discount
factor $\gamma$. At every time step $t$, the agent’s behavior $a_{t}$ is
defined by a probabilistic policy $\pi(s_{t}||g)$, given by the current state
$s_{t}$ and the goal $g$ (we use $||$ as a symbol for concatenation into
$\mathcal{S}\times\mathcal{G}$). The task is to find an optimal policy that
can maximize the expected curriculum reward starting from the initial state
sampled from the initial state distribution $s\in S_{0}$, which is defined as

$V^{\pi}(s||g)=\mathbb{E}_{s_{0}=s\sim
S_{0},\,a_{t}\sim\pi(s_{t}||g),\,s_{t+1}\sim
P(s_{t},a_{t})}\big{[}\sum_{t=0}^{\infty}\gamma^{t}r_{g}(s_{t},a_{t})\big{]}.$
(1)

Hindsight goal generation HGG [24] is an extension of HER [1] to scenarios in
which the target goal distribution differs a lot from the initial state
distribution. HGG can be briefly explained as follows. A value function of a
policy $\pi$ for a specific goal $g$ is assumed to have some generalizability
to another goal $g^{\prime}$ close to $g$ [2, 18]. This assumption is
mathematically characterized via Lipschitz continuity as

$|V^{\pi}(s||g)-V^{\pi}(s^{\prime}||g^{\prime})|\leq L\cdot
d(s||g,s^{\prime}||g^{\prime}),$ (2)

where $d(s||g,s^{\prime}||g^{\prime})$ is a metric defined by

$d((s||g),(s^{\prime}||g^{\prime}))=c||m(s)-m(s^{\prime})||_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}2}}+||g-g^{\prime}||_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color<EMAIL_ADDRESS>(3)

$m(\cdot)$ is a state abstraction to map from the state space to the goal
space. The hyper-parameter $c>0$ provides a trade-off between 1) the distance
between target goals and 2) the distance between the goal representation of
the initial states. Assuming the generalizability condition (2) holds for two
distributions
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}s^{(1)}_{0}}||{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}g^{(1)}}\sim\mathcal{T}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}(1)}}$
and
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}s^{(2)}_{0}}||g^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}(2)}}\sim\mathcal{T}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}(2)}}$,
Ren et al. [24] demonstrated that

$V^{\pi}(\mathcal{T}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}(2)}})\geq
V^{\pi}(\mathcal{T}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}(1)}})-L\cdot\mathcal{D}(\mathcal{T}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}(1)}},\mathcal{T}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}(2)}}),$
(4)

where $\mathcal{D}(\cdot,\cdot)$ is the Wasserstein distance [26] based on
$d(\cdot,\cdot)$, defined as

$\mathcal{D}(\mathcal{T}^{(1)},\mathcal{T}^{(2)}):=\inf_{\mu\in\Gamma(\mathcal{T}^{(1)},\mathcal{T}^{(2)})}\bigg{(}\mathbb{E}_{\mu}\Big{[}d(s_{0}^{(1)}||g^{(1)},s_{0}^{(2)}||g^{(2)})\Big{]}\bigg{)}.$
(5)

$\Gamma(\mathcal{T}^{(1)},\mathcal{T}^{(2)})$ denotes the collection of all
joint distributions $\mu(s_{0}^{(1)}||g^{(1)},s_{0}^{(2)}||g^{(2)})$, the
marginal probabilities of which are $\mathcal{T}^{(1)}$ and
$\mathcal{T}^{(2)}$. With $\mathcal{T}^{*}$ denoting the joint distribution
over initial state $s_{0}$ and goal $g$, the agent tries to find a policy
$\pi$ maximizing the expectation of the discounted cumulative reward based on
the state value function defined in (1):

$V^{\pi}(\mathcal{T}^{*}):=\mathbb{E}_{s_{0}||g\sim\mathcal{T}^{*}}\Big{[}V^{\pi}(s_{0}||g)\Big{]}.$
(6)

From (4), it can be derived that optimizing this expected cumulative reward
defined in (6) can be relaxed into the surrogate problem of finding

$\max_{\mathcal{T},\pi}V^{\pi}(\mathcal{T})-L\cdot\mathcal{D}(\mathcal{T},\mathcal{T}^{*}).$
(7)

Joint optimization of $\pi$ and $\mathcal{T}$ (7) is solved in a two-stage
iterative algorithm. First, standard policy optimization maximizes the value
function $V^{\pi}$ based on experience generated from the intermediate task
set $\mathcal{T}$. Second, the intermediate task set $\mathcal{T}$ is
optimized while the policy $\pi$ is kept constant. The second step is a
variant of the Wasserstein Barycenter problem with the value function as a
bias term for each initial state-goal pair, which can be solved as a bipartite
matching problem [8]. For this to work, $\mathcal{T}^{*}$ is approximated by
by $K$ initial state-goal pairs sampled from it, resulting in
$\hat{\mathcal{T}}^{*}=\\{(\hat{s}_{0}^{i},\hat{g}^{i})\\}_{i=1}^{K}$. Now,
for every $(\hat{s}_{0}^{i},\hat{g}^{i})\in\hat{\mathcal{T}}^{*}$, a
trajectory $\tau^{i}=\\{s_{t}^{i}\\}$ from the replay buffer is found that
minimizes

$w((\hat{s}_{0}^{i},\hat{g}^{i}),\tau^{i}):=c||m(\hat{s}_{0}^{i})-m(s_{0}^{i})||_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}2}}+\min_{t}\big{(}||\hat{g}^{i}-m(s_{t}^{i})||_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}2}}-\frac{1}{L}V^{\pi}(s_{0}^{i}||m(s_{t}^{i}))\big{)}.$
(8)

The Lipschitz constant $L$ is treated as a hyper-parameter. These $K$
trajectories $\tau^{i}$ minimize the sum

$\sum_{(\hat{s}_{0}^{i},\hat{g}^{i})\in\hat{\mathcal{T}^{*}}}w((\hat{s}_{0}^{i},\hat{g}^{i}),\tau^{i})$
(9)

Finally, from each of the $K$ selected trajectories $\tau^{i}$, the hindsight
goal $g^{i}$ is selected from the state $s_{t}^{i}\in\tau^{i}$, that minimized
(8). More formally,

$g^{i}=m\bigg{(}\operatorname*{arg\,min}_{s_{t}^{i}\in\tau_{i}}\Big{(}||\hat{g}^{i}-m(s_{t}^{i})||_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}2}}-\frac{1}{L}V^{\pi}(s_{0}^{i}||m(s_{t}^{i}))\Big{)}\bigg{)}.$
(10)

Combined with the idea of HER [1] and replacing $(s_{0},g)$ with
$(\hat{s}^{i}_{0},g^{i}$), the generated hindsight transition
$(s_{t}||g,a_{t},r_{t},s_{t+1}||g)$ can be then stored in the replay buffer
for training the policy. To this end, HGG is able to generate a curriculum of
meaningful hindsight goals rather than hand-crafted heuristic goals from HER,
guiding exploration towards target goals.

##  3 Methodology

Problem Statement: In this paper, we focus on solving robotic object
manipulation tasks with sparse rewards, where the goal is to move an object to
a certain point in 3D space with a robotic gripper arm. Even though the
general principle of HGG is applicable, there is one major limitation. Solving
the Wasserstein Barycenter problem in HGG consists of the computation of the
distance between a goal $g\in\mathcal{G}$ and the goal representation $m(s)$
of state $s\in\mathcal{S}$. However, in 3D space with obstacles, the euclidean
metric used in HGG is generally not applicable.

(a) Case A with euclidean norm distance: Shortest path goes through the
obstacle, $d_{L2}^{A}=0.210$

(b) Case B with euclidean norm distance: Shortest path goes though the
obstacle, $d_{L2}^{B}=0.283$

(c) Case A with alternative distance: Shortest path goes around the obstacle,
$d_{alt}^{A}=0.566$

(d) Case B with alternative distance: Shortest path goes around the obstacle,
$d_{alt}^{B}=0.301$

Figure 1: Shortest distances between two points (green dots) in 3D space with
obstacles.

To illustrate this point, we consider an example, where the goal is to pick up
an object with a robotic arm, lift it over an obstacle (blue box), and place
it at the target goal. Let us consider two cases: A (Figures 1(a) and 1(c))
and B (Figures 1(b) and 1(d)), where goals $g_{1}$ and $g_{2}$ are marked in
green. In Figures 1(a) and 1(b), the euclidean metric calculates the distance,
$d_{euc}=||g_{2}-g_{1}||$, which is marked with red lines. If the task is
completed in an environment without obstacles, this is an accurate distance
metric, and the task can be solved using HGG. In environments with obstacles,
however, the object cannot go through these obstacles, which means the
euclidean metric is not a suitable distance metric. Then, a suitable shortest-
path based distance measure similar to the ones in Figures 1(c) and 1(d) is
required in such a scenario.

The problem of finding the shortest path between two points without
intersecting an obstacle is known as the euclidean shortest path problem. In
this paper, we propose an algorithm called graph-based hindsight goal
generation (G-HGG), in which approximate shortest distances from discrete
points in 3D space are pre-computed and used to compare distances between
potential hindsight goals.

###  3.1 G-HGG

G-HGG is an extension of HGG to environments with obstacles, where the
euclidean metric is not applicable as a distance metric. Hence, we reformulate
(8) and (10), replacing the euclidean metric with the graph-based distance
metric $d_{G}$ .

With the new formulation, for every
$(\hat{s}_{0}^{i},\hat{g}^{i})\in\hat{\mathcal{T}}^{*}$, we can find a
trajectory from the replay buffer $\tau^{i}=\\{s_{t}^{i}\\}\in R$ that
minimizes

$w((\hat{s}_{0}^{i},\hat{g}^{i}),\tau^{i}):=c||m(\hat{s}_{0}^{i})-m(s_{0}^{i})||_{2}+\min_{t}\bigg{(}d_{G}(\hat{g}^{i}-m(s_{t}^{i}))-\frac{1}{L}V^{\pi}(s_{0}^{i}||m(s_{t}^{i}))\bigg{)}\text{,}$
(11)

where all these variables have the same meanings as in (8). Altogether, these
$K$ trajectories $\tau^{i}$ minimize the sum

$\sum_{(\hat{s}_{0}^{i},\hat{g}^{i})\in\hat{\mathcal{T}^{*}}}w((\hat{s}_{0}^{i},\hat{g}^{i}),\tau^{i})\text{.}$
(12)

Finally, from each of the $K$ selected trajectories $\tau^{i}$, the hindsight
goal $g^{i}$ is selected from the state $s_{t}^{i}\in\tau^{i}$, that minimizes
(11):

$g^{i}=m\bigg{(}\operatorname*{arg\,min}_{s_{t}^{i}\in\tau_{i}}\Big{(}d_{G}(\hat{g}^{i}-m(s_{t}^{i}))-\frac{1}{L}V^{\pi}(s_{0}^{i}||m(s_{t}^{i}))\Big{)}\bigg{)}.$
(13)

The distance metric $d_{G}$ is based on shortest paths in a graph with a
discrete representation of the goal space as vertices. The computation of
shortest path distances is done pre-training and consists of creating a graph
representing the environment and pre-computing shortest distances among
vertices.

Graph Construction

Let us consider an unbounded goal space $\mathcal{G}$ with an infinite number
of goals; let us further define a continuous but bounded subset of the goal
space, the accessible goal space $\mathcal{G}_{A}\subset\mathcal{G}$.
$\mathcal{G}_{A}$ contains all potential goals that the object can reach. We
then establish a representation of the accessible goal space $\mathcal{G}_{A}$
with an undirected graph. A graph $G=(P,E)$ consists of a set of vertices $P$
with a set of weighted edges $E$, and an assigned weight $w$.

$E\subset\\{(p_{1},p_{2},w)\mid(p_{1},p_{2})\in P^{2},p_{1}\neq
p_{2},w\in\mathbb{R}\\},$ (14)

where $p_{1}$ and $p_{2}$ are two possible vertices. In environments with
obstacles, goals $g_{obs}\in\mathcal{G}$ lying within an obstacle that are
blocked from being reached are not elements of the accessible goal space
$g_{obs}\notin\mathcal{G}_{A}$. Since $\mathcal{G}_{A}$ is bounded, it can be
enclosed in a parallelepipedic bounding box defined by values $x_{min}$,
$x_{max}$, $y_{min}$, $y_{max}$, $z_{min}$, $z_{max}$ $\in\mathbb{R}$,
describing the span of this box in each coordinate direction. We then use this
box to generate a finite set of vertices $\hat{P}$ spatially arranged as an
orthorhombic lattice. $\hat{P}$ is defined by the total number of vertices
$n=n_{x}\cdot n_{y}\cdot n_{z}$, with $n_{x},n_{y},n_{z}\in\mathbb{N}$ in each
coordinate direction of $\mathcal{G}_{A}$, or alternatively by the distance
between two adjacent grid-points in each coordinate direction given by

$\Delta_{x}=\frac{x_{max}-x_{min}}{n_{x}-1},\;\Delta_{y}=\frac{y_{max}-y_{min}}{n_{y}-1},\;\Delta_{z}=\frac{z_{max}-z_{min}}{n_{z}-1}.$
(15)

Finally, we define the set of vertices as $P=\hat{P}\cap\mathcal{G}_{A}$,
where

$\begin{split}\hat{P}:=\\{(x_{min}+\Delta_{x}\cdot i,\;y_{min}+\Delta_{y}\cdot
j,\;z_{min}+\Delta_{z}\cdot k)\;\mid\;&i\in[0,n_{x}-1],j\in[0,n_{y}-1],\\\
&k\in[0,n_{z}-1]\\}.\end{split}$ (16)

A set of vertices in a demo environment with an obstacle is illustrated in
Figure 2(a), $\mathcal{G}_{A}$ is defined as the cuboid space marked by the
blue balls, in which the obstacle is depicted by the blue box.
$n_{x}=n_{y}=n_{z}=4$. All the vertices are evenly distributed in
$\mathcal{G}_{A}$ and no one lies inside the obstacle.

(a) Creating vertices.

(b) Creating outgoing edges.

(c) Creating graph.

Figure 2: Graph construction in the demo obstacle environment.

As a next step, we connect two adjacent vertices
$p_{1}=(\hat{x}_{1},\hat{y}_{1},\hat{z}_{1})\in
P,\;p_{2}=(\hat{x}_{2},\hat{y}_{2},\hat{z}_{2})\in P$ with an edge of weight
$w$ considering the following:

$\begin{split}(p_{1},p_{2},w)\in
E\iff&|\hat{x}_{2}-\hat{x}_{1}|\leq\Delta_{x}\;\text{ and
}|\hat{y}_{2}-\hat{y}_{1}|\leq\Delta_{y}\;\text{ and
}|\hat{z}_{2}-\hat{z}_{1}|\leq\Delta_{z},\end{split}$ (17)

with
$w:=\sqrt{(\hat{x}_{2}-\hat{x}_{1})^{2}+(\hat{y}_{2}-\hat{y}_{1})^{2}+\cdot(\hat{z}_{2}-\hat{z}_{1})^{2}}\text{.}$

(a) A: $d_{g}(g_{1},g_{2})=\hat{d}_{G}(\nu(g_{1}),\nu(g_{2}))=1.085$

(b) B: $d_{G}(g_{1},g_{2})=\hat{d}_{G}(\nu(g_{1}),\nu(g_{2}))=0.718$

Figure 3: Graph-based distances and shortest paths (red lines) between two
goals (green dots).

In environments with obstacles, it is important to make sure that no edge in
the graph cuts through an obstacle. It should be noted that, for convex
obstacles, we simply replace them with a minimal axis-aligned cuboid that can
enclose the obstacle. We then remove this cuboid from $\mathcal{G}_{A}$; for a
non-convex obstacle, we decompose it into several convex obstacles and then
replace each of them with a minimal axis-aligned cuboid that can enclose the
convex obstacle. Consequently, we require every convex sub-obstacle to be
detected by at least one potential vertex $v\in\hat{P}$ but
$v\notin\mathcal{G}_{A}$. Let us define the space of one of the cuboid with
its edges $\alpha,\beta,\gamma$. Thus, the graph must satisfy the graph
density criterion (18) for every convex sub-obstacle, i.e., continuous set
goals not included in $\mathcal{G_{A}}$:

$\begin{split}\Delta_{x}<\alpha_{min}^{obs};~{}\Delta_{y}<\beta_{min}^{obs};~{}\Delta_{z}<\gamma_{min}^{obs}\end{split},$
(18)

where $\alpha_{min}^{obs}$, $\beta_{min}^{obs}$, $\gamma_{min}^{obs}$
$\in\mathbb{R}$, describing the infimum length of edges of all the convex sub-
obstacle. When vertices are connected according to (17), every vertex is
connected to at most 26 adjacent vertices (as illustrated in Figure 2(b)).
However, an edge only exists as long as the adjacent vertex is contained in
$\mathcal{G}_{A}$, thus, in case of an environment with obstacles, there is
never an edge cutting through an obstacle as long as the graph density
criterion (18) is satisfied. Figure 2(c) illustrates the final graph
consisting of edges and vertices in our demo environment.

Algorithm 1 Graph-Based Hindsight Goal Generation (G-HGG)

1:Given: off-policy algorithm $\mathbb{A}$, sampling strategy $\mathbb{S}$,
reward function $r_{g}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$

2:Construct a graph $G=(V,E)$ as a discrete representation of $\mathcal{G}$
$\triangleright$ section 3.1

3:Pre-compute shortest distances $\hat{d}_{G}$ between every pair of vertices
$(p_{1},p_{2})\in P^{2}$ with Dijkstra

4:Initialize $\mathbb{A}$ and replay buffer $R$

5:for $iteration$ do

6: Construct a set of $M$ intermediate tasks
$\\{(\hat{s}_{0}^{i},g^{i})\\}_{i=1}^{M}$: $\triangleright$ HGG  • Sample
target tasks
$\\{(\hat{s}_{0}^{i},\hat{g}^{i})\\}_{i=1}^{K}\sim\mathcal{T}^{*}$ • Find $K$
distinct trajectories $\\{\tau^{i}\\}_{i=1}^{K}$ that together minimize (12)
$\triangleright$ weighted bipartite matching, based on
$d_{G}\approx\hat{d}_{G}$ • Find $M$ intermediate tasks
$(\hat{s}_{0}^{i},g^{i})$ by selecting intermediate goal $g^{i}$ from each
$\tau^{i}$ according to (13) $\triangleright$ based on
$d_{G}\approx\hat{d}_{G}$

7: for $episode=1,M$ do

8: $(s_{0},g)\leftarrow(\hat{s}_{0}^{i},g^{i})$ $\triangleright$ hindsight
goal-oriented exploration

9: for $t=0,T-1$ do

10: Sample an action $a_{t}$ using the policy from $\mathbb{A}$ with noise:
$a_{t}\leftarrow\pi(s_{t}||g)+\mathcal{N}_{t}$

11: Execute the action $a_{t}$ and observe a new state $s_{t+1}$

12: for $t=0,T-1$ do

13: $r_{t}:=r_{g}(s_{t},a_{t})$; Store transition
$(s_{t}||g,a_{t},r_{t},s_{t+1}||g)$ in $R$; $\triangleright$ experience replay

14: Sample a set of additional goals for replay
$G:=\mathbb{S}(current\,episode)$

15: for $g^{\prime}\in G$ do

16: $r^{\prime}:=r_{g^{\prime}}(s_{t},a_{t})$; Store the transition
$(s_{t}||g^{\prime},a_{t},r^{\prime},s_{t+1}||g^{\prime})$ in $R$;
$\triangleright$ HER

17: for $t=1,N$ do

18: Sample a minibatch $B$ from the replay buffer $R$ $\triangleright$ HER or
EBP

19: Perform one step of optimization using $\mathbb{A}$ and minibatch $B$
using DDPG

Shorted Distance Computation Considering the created graph that represents the
environment, we can employ a shortest path algorithm such as Dijkstra’s
algorithm [7] to calculate shortest paths and shortest distances $\hat{d}_{G}$
between every possible pair of vertices
$(p_{1},p_{2})=((\hat{x}_{1},\hat{y}_{1},\hat{z}_{1}),(\hat{x}_{2},\hat{y}_{2},\hat{z}_{2}))\in
P^{2}$ in a graph $G=(P,E)$. All possible combinations of the resulting
shortest distance function $\hat{d}_{G}$ can be efficiently pre-computed with
Dijkstra and stored in an $n\times n$ table, where $n$ denotes the number of
vertices in $P$. An ablation study of $n$ can be found in the Appendix
LABEL:subsubsec_vertices.

Given two goals $g_{1}=(x_{1},y_{1},z_{1})\in\mathcal{G}$,
$g_{2}=(x_{2},y_{2},z_{2})\in\mathcal{G}$ and a graph $G=(P,E)$ with
representing the approximate goal space $\mathcal{G}_{A}\subset\mathcal{G}$
with $x_{min}$, $x_{max}$, $y_{min}$, $y_{max}$, $z_{min}$, $z_{max}$,
$\Delta_{x}$, $\Delta_{y}$, and $\Delta_{z}$, the graph-based distance
$d:\mathcal{G}^{2}\rightarrow\mathbb{R}$ is defined such that

$d_{G}(g_{1},g_{2})=\begin{cases}\hat{d}_{G}\Big{(}\nu(g_{1}),\nu(g_{2})\Big{)},&\text{if
}g_{1}\in\mathcal{G}_{A}\land g_{2}\in\mathcal{G}_{A}\\\
\infty,&\text{otherwise}\end{cases}$ (19)

where $\nu:\mathcal{G}_{A}\rightarrow P$ maps goals in $\mathcal{G}_{A}$ to
the closest vertex in $P$:

$\begin{split}&\nu(g)=\nu(x,y,z)=(\hat{x},\hat{y},\hat{z})=\\\
&\big{(}x_{min}+\Delta_{x}\cdot\Big{\lfloor}\frac{x-x_{min}}{\Delta_{x}}\Big{\rceil},~{}y_{min}+\Delta_{y}\cdot\Big{\lfloor}\frac{y-y_{min}}{\Delta_{y}}\Big{\rceil},~{}z_{min}+\Delta_{x}\cdot\Big{\lfloor}\frac{z-z_{min}}{\Delta_{z}}\Big{\rceil}\big{)}\end{split}$
(20)

$\lfloor a\rceil$ rounds any $a\in\mathbb{R}$ to the closest integer value.
Figure 3 illustrates a visualization of graph-based goal distance computation
between goals $g_{1}\in\mathcal{G}_{A}$, $g_{2}\in\mathcal{G}_{A}$. In both
Cases A (Figure 3(a)) and B (Figure 3(b)), both goals are contained in the
accessible goal space $\mathcal{G}_{A}$. Therefore, the graph-based distance
is approximated by the shortest distance between the two vertices $\nu(g_{1})$
and $\nu(g_{2})$ that are closest to the goals $g_{1}$ and $g_{2}$,
respectively. Mathematically,
$d(g_{1},g_{2})=\hat{d}_{G}(\nu(g_{1}),\nu(g_{2}))$.

Algorithm and Stop Condition: The overall G-HGG algorithm is provided as
Algorithm 1. The main differences between HGG and G-HGG are in steps 2,3 and
6. Step 2 (graph construction) and step 3 (shortest distance computation) are
performed pre-training. The pre-computed table of shortest distances
$\hat{d}_{G}$ is used in step 6. G-HGG guides exploration by hindsight goals.
This is especially useful at the beginning of training, where hindsight goals
are far from the target goals $\mathcal{G}_{T}$. When training progresses and
a certain fraction $\delta_{stop}\in[0,1]$ of the hindsight goal candidates
are very close to sampled target goals, stopping HGG and continuing with
vanilla HER will lead to faster learning and higher success rates. Because of
the page limit, we include the ablation study on $\delta_{stop}$ in Appendix
LABEL:chapter:ablation.

##  4 Experiments

To demonstrate the advantages of G-HGG over HGG and vanilla HER, we create new
experimental environments based on the standard robotic manipulation
environments from OpenAI Gym [23].

  * •

FetchPushLabyrinth (Figure 4(a)): the goal is to push the cube from its
initial position around the blue obstacles (labyrinth) to a target goal. The
gripper remains permanently closed.

  * •

FetchPickObstacle (Figure 4(b)): the goal is to pick up the cube from its
initial position, lift it over the obstacle, and place it at a target goal
position. Gripper control is enabled.

  * •

FetchPickNoObstacle (Figure 4(c)): the goal is to pick up the cube from its
initial position, lift it up, and place it at a target goal position. No
obstacle is present in this scenario, but the target goals are located in the
air. Gripper control is enabled.

  * •

FetchPickAndThrow (Figure 4(d)): the goal is to pick up the cube from its
initial position, lift it up, and throw it into one of the eight boxes
(obstacles). Gripper control is enabled.

(a) FetchPushLabyrinth

(b) FetchPickObstacle

(c) FetchPickNoObstacle

(d) FetchPickAndThrow

Figure 4: Robotic manipulation environments.

We tested G-HGG on the four environments to compare its performance to HGG and
HER. Since we know from [32] and [24] that energy-based prioritization (EBP)
[32] significantly enhances the performance of both HER and HGG, we used EBP
in all our trainings of HER, HGG, and G-HGG. The results clearly show that
G-HGG outperforms HGG by far in environments with obstacles, both in terms of
sample efficiency and maximum success rate. In the environment without
obstacles, the performance of G-HGG was still comparable to HGG in terms of
sample efficiency and maximum success rate.

Figure 5: Median test success rate (line) and interquartile range (shaded) of
G-HGG, HGG, and HER.

Figure 5 shows success rates of G-HGG, HGG, and HER (median and interquartile
range of five training runs each), plotted over training iterations in four
environments. The most remarkable results can be observed in the
FetchPushLabyrinth environment. While HER and HGG display no success over 400
iterations, G-HGG reaches a success rate of $80\%$ after 170 iterations,
increasing to over $90\%$ after 300 iterations. By comparing a sample of
hindsight goals from iterations 20, 40, 60, and 80 (Figure 6), it becomes
obvious that G-HGG outperforms HGG by far. While the graph-based distance
metric used in G-HGG leads to a choice of hindsight goals guiding the agent
around the obstacle towards the target goals , HGG repeatedly uses goals that
are closest to the target goals with respect to the euclidean metric. Since
this euclidean metric based shortest path is blocked by an obstacle, the agent
gets stuck trying to reach the hindsight goals (pushing the object against the
obstacle), never achieving more promising goals, and thus, never reaching the
target goals.

In the FetchPickObstacle environment, G-HGG outperforms HGG in terms of sample
efficiency, due to graph-based distances supporting goals that avoid the
obstacle. Since the euclidean distance metric is at least partly (in x and y
direction) valid, HGG eventually achieves a notable success rate as well.
However, the plots show that there is a large variance within the HGG
trainings, emphasizing the disadvantage of random exploration in HGG with non-
meaningful hindsight goals over guided exploration in G-HGG. Vanilla HER
cannot solve the task.

FetchPickNoObstacle is an environment where G-HGG has no advantage over HGG.
As no obstacle is present, the euclidean metric is applicable to compare
distances, allowing HGG to perform well. Since the euclidean metric used in
HGG is more exact than the graph-based distances in G-HGG, it is not
surprising that HGG yields slightly better training results. However, the
similarity of the curves shows that G-HGG’s performance is comparable to HGG
in terms of sample efficiency and success rate even in environments without
obstacles. Overall, we assume that G-HGG is generally applicable to all
environments where HGG yields good training results. HER still fails to reach
the target goals.

Figure 6: Hindsight goals in FetchPushLabyrinth after 20 (red), 40 (orange),
60 (yellow) and 80 (green) episodes of G-HGG (Left) / HGG (Right). One
iteration contains $M=50$ episodes.

FetchPickAndThrow is a difficult task, since target goals are not uniformly
sampled from a continuous target goal distribution, but from a discrete set of
eight goals. G-HGG yields better results than HGG in terms of success rate
after 600 iterations and is clearly more sample efficient. Both HGG and G-HGG
cannot achieve success rates above $60\%$ due to the difficulty of the task,
involving picking the object, lifting it up, and dropping it while giving it a
well-dosed push in the desired direction. Since the final policy trained to
achieve the task results from guided, but still random exploration, it is no
surprise that the agent cannot develop a perfect throwing motion from sparse
rewards.

##  5 Related Work

Informative and effective explorations are essential for solving goal-
conditioned RL tasks with sparse rewards. An amount of research has emerged
and we briefly discuss them from three main ideas.

Prioritized Experience Replay One major drawback of HER has been its
inefficient random replay of experience. Research has shown that prioritized
sampling of transitions from the replay buffer significantly increases sample
efficiency. Prioritized sampling can be based on the TD-error [27], reward-
weighted entropy [31], transition energy [32], and density of achieved goals
[33].
Curriculum Learning Another way to tackle the exploration problem in sparse-
reward multi-goal RL is curriculum learning (CL) [25], which presents problems
in a favorable order, the so-called curriculum. One CL approach to improve
exploration is augmenting the sparse learning problem with basic, easy-to-
learn auxiliary tasks [25, 9]. In the absence of extrinsic motivation due to
the sparsity of external rewards, intrinsic motivation can be used to create a
curriculum for improved exploration [28, 20, 14, 21, 30, 6, 3]. Another way of
constructing a meaningful curriculum is to predict high-reward states and
generate goals close to these meaningful states [16, 12, 13, 11].
Representation Learning [4] is a promising framework to solve goal-conditioned
RL tasks, in which representative abstractions are interpreted from high-
dimensional observations to low-dimensional latent states. With the
representations as a planner, model-free RL algorithms are able to perform
control tasks. Some work showed that learned representations can be used to
solve navigation and goal-reaching tasks for mobile agents. Robotic
manipulations tasks were implemented but limited to 2D space because most
representations were learned from images [29, 15, 10].

##  6 Conclusion

We proposed a novel automatic hindsight goal generation algorithm G-HGG on the
basis of the HGG for robotic object manipulation in environments with
obstacles, by which the selection of valuable hindsight goals is generated
with a graph-based distance metric. We formulated our solution as a graph
construction and shortest distance computation process as pre-training steps.
Experiments on four different challenging object manipulation tasks
demonstrated superior performance of G-HGG over HGG and HER in terms of both
maximum success rate and sample efficiency. Future research could concentrate
on improvement, extension, and real-world deployment of G-HGG.

## Broader Impact

## References

  * [1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba.  Hindsight experience replay.  In Advances in Neural Information Processing Systems, volume 2017-Decem, pages 5049–5059, 2017.
  * [2] Kavosh Asadi, Dipendra Misra, and Michael L. Littman.  Lipschitz continuity in model-based reinforcement learning.  35th International Conference on Machine Learning, ICML 2018, 1(1):419–435, 2018.
  * [3] A Aubret, L Matignon, and S Hassas.  A survey on intrinsic motivation in reinforcement learning.  2019\.
  * [4] Y. Bengio, A. Courville, and P. Vincent.  Representation learning: A review and new perspectives.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013.
  * [5] Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine.  Path integral guided policy search.  Proceedings - IEEE International Conference on Robotics and Automation, pages 3381–3388, 2017.
  * [6] Cédric Colas, Pierre Founder, Olivier Sigaud, Mohamed Chetouani, and Pierre Yves Oudeyer.  CURIOUS: Intrinsically motivated modular multi-goal reinforcement learning.  In 36th International Conference on Machine Learning, ICML 2019, volume 2019-June, pages 2372–2387, 2019.
  * [7] E.W. Dijkstra.  A Note on Two Problems in Connexion with Graphs.  Numerische Mathematik 1, 1959.
  * [8] Ran Duan and Hsin Hao Su.  A scaling algorithm for maximum weight matching in bipartite graphs.  Proceedings of the Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1413–1424, 2012.
  * [9] Manfred Eppe, Sven Magg, and Stefan Wermter.  Curriculum goal masking for continuous deep reinforcement learning.  2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019, pages 183–188, 2019.
  * [10] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine.  Search on the replay buffer: Bridging planning and reinforcement learning.  In Advances in Neural Information Processing Systems, pages 15220–15231, 2019.
  * [11] Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhang Zhengyou.  Curriculum-guided Hindsight Experience Replay.  NeurIPS 2019, (3):1–12, 2019.
  * [12] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel.  Automatic Goal Generation for Reinforcement Learning Agents.  Proceedings of the 35th International Conference on Machine Learning, PMLR 80, 2018.
  * [13] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel.  Reverse Curriculum Generation for Reinforcement Learning.  1st Conference on Robot Learning, CoRL 2017, pages 1–14, 2017.
  * [14] Sébastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer.  Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning.  2017\.
  * [15] Dibya Ghosh, Abhishek Gupta, and Sergey Levine.  Learning actionable representations with goal-conditioned policies.  ArXiv, abs/1811.07819, 2019.
  * [16] Anirudh Goyal, Philemon Brakel, William Fedus, Soumye Singhal, Timothy Lillicrap, Sergey Levine, Hugo Larochelle, and Yoshua Bengio.  Recall traces: Backtracking models for efficient reinforcement learning.  7th International Conference on Learning Representations, ICLR 2019, pages 1–19, 2019.
  * [17] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-end training of deep visuomotor policies.  Journal of Machine Learning Research, 17:1–40, 2016.
  * [18] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma.  Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees.  7th International Conference on Learning Representations, ICLR 2019, pages 1–27, 2019.
  * [19] A. Ng, A. Coates, M. Diel, V. Ganapathi, J. Schulte, B. Tse, E. Berger, and E. Liang.  Autonomous inverted helicopter flight via reinforcement learning.  Experimental Robotics IX, pages 363–372, 2006.
  * [20] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  34th International Conference on Machine Learning, ICML 2017, 6:4261–4270, 2017.
  * [21] Alexandre Péré, Sebastien Forestier, Olivier Sigaud, and Pierre Yves Oudeyer.  Unsupervised learning of goal spaces for intrinsically motivated goal exploration.  6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings, pages 1–26, 2018.
  * [22] Jan Peters and Stefan Schaal.  Reinforcement learning of motor skills with policy gradients.  Neural Networks, 21(4):682–697, 2008.
  * [23] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba.  Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research.  pages 1–16, feb 2018.
  * [24] Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng.  Exploration via Hindsight Goal Generation.  33rd Conference on Neural Information Processing Systems, NeurIPS 2019, 2019.
  * [25] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van De Wiele, Volodymyr Mnih, Nicolas Heess, and Tobias Springenberg.  Learning by playing - Solving sparse reward tasks from scratch.  35th International Conference on Machine Learning, ICML 2018, 10(2017):6910–6919, 2018.
  * [26] Ludger Rüschendorf.  The wasserstein distance and approximation theorems.  Probability Theory and Related Fields, 70(1):117–129, 1985.
  * [27] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  In 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings, pages 1–21, 2016.
  * [28] Satinder Singh, Andrew G. Barto, and Nuttapong Chentanez.  Intrinsically motivated reinforcement learning.  Advances in Neural Information Processing Systems, 2005.
  * [29] Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn.  Universal planning networks: Learning generalizable representations for visuomotor control.  In International Conference on Machine Learning, pages 4732–4741, 2018.
  * [30] Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus.  Intrinsic motivation and automatic curricula via asymmetric self-play.  6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings, (i):1–16, 2018.
  * [31] Rui Zhao, Xudong Sun, and Volker Tresp.  Maximum entropy-regularized multi-goal reinforcement learning.  In 36th International Conference on Machine Learning, ICML 2019, volume 2019-June, pages 13022–13035, 2019.
  * [32] Rui Zhao and Volker Tresp.  Energy-Based Hindsight Experience Prioritization.  2nd Conference on Robot Learning, CoRL 2018, 2018.
  * [33] Rui Zhao and Volker Tresp.  Curiosity-Driven Experience Prioritization via Density Estimation.  32nd Conference on Neural Information Processing Systems, NIPS 2018, 2019.