# Can we understand the decay width of the $T_{cc}^{+}$ state?

Xi-Zhe Ling  School of Physics, Beihang University, Beijing 102206, China
Ming-Zhu Liu<EMAIL_ADDRESS>School of Space and Environment, Beihang
University, Beijing 102206, China  School of Physics, Beihang University,
Beijing 102206, China  Li-Sheng Geng<EMAIL_ADDRESS>School of
Physics, Beihang University, Beijing 102206, China  Beijing Key Laboratory of
Advanced Nuclear Materials and Physics, Beihang University, Beijing, 102206,
China  School of Physics and Microelectronics, Zhengzhou University,
Zhengzhou, Henan 450001, China  En Wang<EMAIL_ADDRESS>School of Physics
and Microelectronics, Zhengzhou University, Zhengzhou, Henan 450001, China
Ju-Jun Xie<EMAIL_ADDRESS>Institute of Modern Physics, Chinese Academy
of Sciences, Lanzhou 730000, China  School of Nuclear Science and Technology,
University of Chinese Academy of Sciences, Beijing 101408, China  School of
Physics and Microelectronics, Zhengzhou University, Zhengzhou, Henan 450001,
China  Lanzhou Center for Theoretical Physics, Key Laboratory of Theoretical
Physics of Gansu Province, Lanzhou University, Lanzhou, Gansu 730000, China

###### Abstract

Inspired by the recent discovery of a doubly charmed tetraquark state
$T_{cc}^{+}$ by the LHCb Collaboration, we employ the effective Lagrangian
approach to investigate the decay width of $T_{cc}^{+}\to
D^{+}D^{0}\pi^{0}/D^{0}D^{0}\pi^{+}$ and $T_{cc}^{+}\to D^{0}D^{+}\gamma$ with
the assumption that $T_{cc}^{+}$ is an isoscalar $DD^{\ast}$ molecule. We show
that both the $T_{cc}\to DD\pi$ and $T_{cc}\to DD\gamma$ modes contribute to
the decay width of $T_{cc}$, with the former being dominant. The resulting
total decay width of about $\Gamma=63$ keV is smaller than the experimental
decay width obtained from the Breit-Wigner fit of the LHCb data,
$\Gamma=410\pm 165\pm 43^{+18}_{-38}$ keV, while close to the number obtained
from the alternative unitary analysis, $\Gamma=48\pm 2^{+0}_{-14}$ keV, which
supports the molecular nature of $T_{cc}$.

##  I Introduction

Mesons made of a pair of quark and anti-quark and baryons made of three quarks
can be well understood in the conventional quark model Gell-Mann (1964).
Although Quantum ChromoDynamics (QCD) allows for other quark configurations,
such as tetraquark, pentaquark, and hexaquark states, their existence were not
experimentally confirmed until the charged tetraquark state $Z_{c}(4430)$,
with the minimum quark content $c\bar{c}u\bar{d}$, was discovered in 2007 by
the Belle Collaboration Choi  _et al._ (2008). In 2015, the LHCb Collaboration
reported the first pentaquark states $P_{c}(4380)$ and $P_{c}(4450)$ Aaij  _et
al._ (2015), while the latter was shown to be a superposition of two states,
$P_{c}(4440)$ and $P_{c}(4457)$ Aaij  _et al._ (2019). In 2020, the LHCb
Collaboration discovered the first fully heavy tetraquark state $X(6900)$ Aaij
_et al._ (2020a). It should be noted that all of these exotic states carry
hidden charm number. The first open charm tetraquark states, $X_{0}(2866)$ and
$X_{1}(2904)$, were only discovered in 2020 by the LHCb Collaboration Aaij
_et al._ (2020b).

Since forty years ago, a series of pioneer works have already investigated the
likely existence of $QQ\bar{q}\bar{q}$ tetraquark states in the quark model,
which showed that the stability of a $QQ\bar{q}\bar{q}$ tetraquark depends on
the mass ratio of $m_{Q}/m_{\bar{q}}$ Ader  _et al._ (1982); Zouzou  _et al._
(1986); Lipkin (1986); Heller and Tjon (1987); Carlson  _et al._ (1988);
Silvestre-Brac and Semay (1993); Semay and Silvestre-Brac (1994). As the ratio
is larger, such a multiquark state is more stable. However, due to the
uncertainty of $m_{c}/m_{\bar{q}}$, whether the mass of the $cc\bar{q}\bar{q}$
tetraquark state is above or below the $DD^{\ast}$ mass threshold is
unsettled. Later, meson exchange potentials were employed to study the likely
existence of $DD^{\ast}$ molecules Manohar and Wise (1993); Pepin  _et al._
(1997), where due to the unknown parameters the existence of $DD^{\ast}$ bound
states is also uncertain. In 2002, a doubly charmed baryon was discovered by
the SELEX Collaboration Mattson  _et al._ (2002), which motivated further
theoretical studies on $cc\bar{q}\bar{q}$ tetraquark states Gelman and
Nussinov (2003); Vijande  _et al._ (2004). In 2003, $X(3872)$ was discovered
by the Belle Collaboration Choi  _et al._ (2003), which opened a new era in
hadron physics. One of the most promising interpretations of $X(3872)$ is a
$D\bar{D}^{\ast}$ bound state. This has further stimulated theoretical studies
on $D^{(*)}D^{(*)}$ molecules, i.e., $T_{cc}$ Janc and Rosina (2004); Navarra
_et al._ (2007); Vijande  _et al._ (2007); Ebert  _et al._ (2007); Yang  _et
al._ (2009); Molina  _et al._ (2010); Li  _et al._ (2013); Feng  _et al._
(2013); Luo  _et al._ (2017); Wang (2018); Junnarkar  _et al._ (2019); Maiani
_et al._ (2019); Liu  _et al._ (2019). In 2017, the LHCb collaboration
reported the doubly charmed baryon $\Xi_{cc}$ Aaij  _et al._ (2017), which
allows the $cc$-diquark mass precisely extracted by the mass of $\Xi_{cc}$ and
predicts a doubly charmed compact tetraquark state above the $DD^{*}$ mass
threshold by 8 MeV Karliner and Rosner (2017). Taking into account the heavy
quark symmetry between $\Xi_{cc}$ and $T_{cc}$, in Refs. Mehen (2017); Eichten
and Quigg (2017) a doubly charmed compact tetraquark state above the
$DD^{\ast}$ mass threshold was also predicted.

Very recently, the LHCb Collaboration reported the discovery of a doubly
charmed tetraquark state $T_{cc}^{+}$ with $I(J^{P})=0(1^{+})$, which is found
in the $D^{0}D^{0}\pi^{+}$ invariant mass spectrumAaij  _et al._ (2021a). Its
binding energy with respect to the $D^{\ast+}D^{0}$ mass threshold is found to
be $\delta=273\pm 61\pm 5^{+11}_{-14}$ keV and decay width is $\Gamma=410\pm
165\pm 43^{+18}_{-38}$ keV. In the unitarized Breit-Wigner profile that takes
into account the effect of mass threshold, the binding energy and decay width
change to $\delta=360\pm 40^{+4}_{-0}$ keV and $\Gamma=48\pm 2^{+0}_{-14}$
keV, respectively Aaij  _et al._ (2021b). The $T_{cc}$ state could be either a
compact tetraquark state or a hadronic molecule of $DD^{*}$. Since the
$T_{cc}$ mass is below the mass threshold of $D^{\ast+}D^{0}$ by 273 keV, the
molecular picture seems more appealing.

In the present work, we revisit the molecular picture for the $T_{cc}^{+}$
state. In particular, we study its hadronic and radiative decays to check
whether one can obtain a decay width in reasonable agreement with the LHCb
measurement. This can serve as a highly nontrivial check on its nature.

##  II Theoretical formalism

In our previous work Liu  _et al._ (2019), we predicted the existence of two
doubly charmed hadronic molecules, $DD^{\ast}$ and $D^{\ast}D^{\ast}$, with
$I(J^{P})=0(1^{+})$, in the One-Boson-Exchange(OBE) model where the cutoff
$\Lambda=$1.01 GeV was fixed by reproducing the binding energy of $X(3872)$,
assumed to be a $\bar{D}^{\ast}D$ bound state. With the cutoff $\Lambda=$1.01
GeV we obtained one $DD^{\ast}$ bound state below the $DD^{\ast}$ mass
threshold by 3 MeV, which is quite close to the mass of $T_{cc}$. Actually,
the experimental mass of $T_{cc}$ can be obtained with a slightly smaller
cutoff, i.e., $\Lambda=0.943$ GeV, as shown in Ref. Liu  _et al._ (2020).
Therefore, in the OBE model one can regard the newly observed doubly charmed
tetraquark state $T_{cc}$ as a $DD^{\ast}$ bound state. In the following, we
focus on its decay mechanism. Since $T_{cc}$ is doubly charmed, it cannot
decay into charmonium states and light mesons and the allowed strong decay
modes should contain open charm mesons. However, as the $T_{cc}$ mass is below
the $DD^{*}$ threshold, the decay can only proceed via an off-shell $D^{*}$,
which would heavily suppress its decay width. Weak and radiative decays are
also possible, but they are much smaller in comparison with the strong decay.

In this work, we assume that $T_{cc}^{+}$ is generated by couple channels
$D^{\ast+}D^{0}$ and $D^{\ast 0}D^{+}$, and it then can decay into
$D^{+}D^{0}\pi^{0}/D^{0}D^{0}\pi^{+}$ and $D^{+}D^{0}\gamma$ via the tree-
level diagrams, as shown in Fig. 1. In the following, we employ the effective
Lagrangian approach to calculate the partial decay widths of $T_{cc}\to
D^{+}D^{0}\pi^{0}/D^{0}D^{0}\pi^{+}$ and $D^{+}D^{0}\gamma$.

\begin{overpic}[scale={0.08}]{tree.eps} \put(10.0,25.0){$T_{cc}^{+}$} \par\put(41.0,3.0){$D^{0}$} \par\put(41.0,32.0){$D^{\ast+}$} \put(72.0,50.0){$D^{+}$} \put(70.0,21.0){$\pi^{0}$} \put(58.0,-6.0){(a)} \end{overpic} |  \begin{overpic}[scale={0.08}]{tree.eps} \put(10.0,25.0){$T_{cc}^{+}$} \par\put(41.0,3.0){$D^{0}$} \par\put(41.0,32.0){$D^{\ast+}$} \put(72.0,50.0){$D^{0}$} \put(70.0,21.0){$\pi^{+}$} \put(58.0,-6.0){(b)} \end{overpic} |  \begin{overpic}[scale={0.08}]{tree.eps} \put(10.0,25.0){$T_{cc}^{+}$} \par\put(41.0,3.0){$D^{+}$} \par\put(41.0,32.0){$D^{\ast 0}$} \put(72.0,50.0){$D^{0}$} \put(70.0,21.0){$\pi^{0}$} \put(58.0,-6.0){(c)} \end{overpic} |  \begin{overpic}[scale={.08}]{treeg.eps} \put(10.0,25.0){$T_{cc}^{+}$} \par\put(41.0,3.0){$D^{0}$} \par\put(41.0,32.0){$D^{\ast+}$} \put(72.0,50.0){$D^{+}$} \put(70.0,23.0){$\gamma$} \put(58.0,-6.0){(d)} \end{overpic} |  \begin{overpic}[scale={.08}]{treeg.eps} \put(10.0,25.0){$T_{cc}^{+}$} \par\put(41.0,3.0){$D^{+}$} \par\put(41.0,32.0){$D^{\ast 0}$} \put(72.0,50.0){$D^{0}$} \put(70.0,23.0){$\gamma$} \put(58.0,-6.0){(e)} \end{overpic}
---|---|---|---|---
Figure 1: Tree-level diagrams for strong decays of $T_{cc}^{+}\to
D^{+}\pi^{0}(D^{\ast+})D^{0}$ (a), $T_{cc}^{+}\to
D^{0}\pi^{+}(D^{\ast+})D^{0}$ (b) and $T_{cc}^{+}\to D^{0}\pi^{0}(D^{\ast
0})D^{+}$ (c) as well as radiative decays of $T_{cc}^{+}\to
D^{+}\gamma(D^{\ast+})D^{0}$ (d) and $T_{cc}^{+}\to D^{0}\gamma(D^{\ast
0})D^{+}$ (e).

The interaction between the $T_{cc}$ state and the $DD^{\ast}$ pair is
described by the following effective Lagrangian Dong  _et al._ (2008)

$\displaystyle\mathcal{L}_{T_{cc}}(x)=ig_{T_{cc}}T_{cc}^{\mu}(x)\int
dy\Phi(y^{2})D(x+\omega_{D^{\ast}}y)D^{\ast}_{\mu}(x-\omega_{D}y),$ (1)

where $\omega_{D^{\ast}}=\frac{m_{D^{\ast}}}{m_{D^{\ast}}+m_{D}}$ and
$\omega_{D}=\frac{m_{D}}{m_{D^{\ast}}+m_{D}}$ are the kinematic parameters
with $m_{D^{\ast}}$ and $m_{D}$ the masses of $D$ and $D^{*}$, and
$g_{T_{cc}}$ is the coupling between $T_{cc}$ and the ${D}^{\ast}D$ component.
The correlation function $\Phi(y^{2})$ is introduced to reflect the
distribution of the two constituent hadrons in a molecule, which also renders
the Feynman diagrams ultraviolet finite. Here we choose the Fourier
transformation of the correlation function in form of a Gaussian function

$\displaystyle\Phi(p^{2})=e^{-p_{E}^{2}/\Lambda^{2}},$ (2)

where $\Lambda$ is a size parameter, which is expected to be around 1 GeV
Huang  _et al._ (2020); Ling  _et al._ (2021), and $P_{E}$ is the Euclidean
momentum. The coupling of $g_{T_{cc}}$ can be estimated by reproducing the
binding energy of the $T_{cc}$ state via the compositeness condition Weinberg
(1963); Salam (1962); Hayashi  _et al._ (1967). The condition indicates that
the coupling constant can be determined from the fact that the renormalization
constant of the wave function of a composite particle should be zero.

For a spin-1 meson, the self energy can be divided into a transverse part and
a longitudinal part, i.e.,

$\displaystyle\Sigma^{\mu\nu}=g^{\mu\nu}_{\bot}\Sigma^{T}(k_{0}^{2})+\frac{p^{\mu}p^{\nu}}{p^{2}}\Sigma^{L}(k_{0}^{2}),$
(3)

with $g^{\mu\nu}_{\bot}=g^{\mu\nu}-\frac{p^{\mu}p^{\nu}}{p^{2}}$. The
compositeness condition can then be estimated from the transverse part of the
self energy

$Z_{T_{cc}}=1-\frac{d\Sigma_{T_{cc}}^{T}(k_{0}^{2})}{d{k}_{0}^{2}}|_{{{k}_{0}=m_{T_{cc}}}}=0.$
(4)

Table 1: Masses, quantum numbers and partial decay widths of relevant mesons used in this work Zyla  _et al._ (2020). Meson | $I(J^{P})$ | M (MeV) | Meson | $I(J^{P})$ | M (MeV)
---|---|---|---|---|---
$D^{0}$ | $\frac{1}{2}(0^{-})$ | $1864.84\pm 0.05$ | $D^{+}$ | $\frac{1}{2}(0^{-})$ | $1869.66\pm 0.05$
$D^{\ast 0}$ | $\frac{1}{2}(1^{-})$ | $2006.85\pm 0.05$ | $D^{\ast+}$ | $\frac{1}{2}(1^{-})$ | $2010.26\pm 0.05$
$\pi^{+}$ | $1(0^{-})$ | $139.57039\pm 0.00018$ | $\pi^{0}$ | $1(0^{-})$ | $134.9768\pm 0.0005$
Decay mode |  | Width (keV) | Decay mode |  | Width (keV)
$D^{*+}\to D^{0}\pi^{+}$$(D^{+}\pi^{0})$ |  |  $56.5\pm 1.2$($25.6\pm 0.6$) | $D^{*+}\to D^{+}\gamma$ |  | $1.33\pm 0.03$
$D^{*0}\to D^{0}\pi^{0}$ |  |  $34.658$ Rosner (2013) | $D^{*0}\to D^{0}\gamma$ |  |  $21.242$ Rosner (2013)

The $T_{cc}$ mass is below the $D^{\ast+}D^{0}$ mass threshold by 273 keV with
an uncertainty of 66 keV. We note that the threshold of $D^{\ast 0}D^{+}$ is
above that of $D^{\ast+}D^{0}$ by only about 1.4 MeV, and therefore we assume
that the couplings of $g_{T_{cc}D^{\ast 0}D^{+}}$ and
$g_{T_{cc}D^{\ast+}D^{0}}$ are the same, from SU(2)-isospin symmetry.
Therefore, we take the average mass of $D^{\ast 0}$ and $D^{\ast+}$ ($D^{0}$
and $D^{+}$) to calculate the coupling of $T_{cc}$ to its component
$g_{T_{cc}D^{\ast}D}$ (as a result, the binding energy is 0.978 MeV), then
obtain $g_{T_{cc}D^{\ast 0}D^{+}}$ and $g_{T_{cc}D^{\ast+}D^{0}}$ using
isospin symmetry. In the isospin symmetric limit, the $T_{cc}^{+}$ couplings
to $D^{\ast+}D^{0}$ and $D^{\ast 0}D^{+}$ satisfy the following relationship

$\displaystyle g_{T_{cc}^{+}D^{0}D^{\ast+}}=-g_{T_{cc}^{+}D^{\ast
0}D^{+}}=\frac{1}{\sqrt{2}}g_{T_{cc}DD^{\ast}}.$ (5)

Substituting the coupling $g_{T_{cc}DD^{\ast}}$ estimated by the compositeness
condition, the couplings $g_{T_{cc}^{+}D^{0}D^{\ast+}}$ and
$g_{T_{cc}^{+}D^{\ast 0}D^{+}}$ are determined, which turn out to be
consistent with the result of the chiral unitary approach Feijoo  _et al._
(2021). In Fig. 2 we present the dependence of the $T_{cc}$ coupling to
$DD^{\ast}$ on the $T_{cc}$ mass with the size parameter $\Lambda$ fixed at 1
and 2 GeV. The masses of the involved particles are given in Table 1. One can
see that the coupling gradually decreases as the $T_{cc}$ mass increases. Note
that the coupling is only weakly dependent on the size parameter $\Lambda$.

Figure 2: Coupling of $T_{cc}$ to $DD^{\ast}$ as a function of the $T_{cc}$
mass with $\Lambda=1$ and $\Lambda=2$ GeV. The vertical dashed line indicates
the experimental central value for the $T_{cc}$ mass.

The Lagrangian describing the $D^{\ast}$ decay into $D\pi$ and $D\gamma$ are

$\displaystyle\mathcal{L}_{DD^{\ast}\pi}$ $\displaystyle=$ $\displaystyle-
ig_{DD^{\ast}\pi}(D\partial^{\mu}\pi
D^{\ast{\dagger}}_{\mu}-D_{\mu}^{*}\partial^{\mu}\pi D^{{\dagger}}),$ (6)
$\displaystyle\mathcal{L}_{DD^{\ast}\gamma}$ $\displaystyle=$ $\displaystyle
eg_{DD^{\ast}\gamma}\varepsilon^{\mu\nu\alpha\beta}\partial_{\mu}A_{\nu}\partial_{\alpha}D^{\ast}_{\beta}D,$

where the fine structure constant $\frac{e^{2}}{4\pi}=\frac{1}{137}$, and
relevant couplings are determined as $g_{D^{\ast+}D^{0}\pi^{+}}=16.818$ and
$g_{D^{\ast+}D^{+}\gamma}=$0.468 GeV-1 by reproducing the decay widths of
$D^{\ast+}\to D^{0}\pi^{+}$ and $D^{\ast+}\to D^{+}\gamma$ Zyla  _et al._
(2020), respectively. Experimentally, there exists only an upper limit
$\Gamma<2.1$ MeV for the $D^{\ast 0}$ width. Thus we turn to the quark model
Rosner (2013), where the strong and radiative decay widths of $D^{\ast 0}$
were estimated to be $\Gamma_{D^{\ast 0}\to D^{0}\pi^{0}}=34.658$ keV and
$\Gamma_{D^{\ast 0}\to D^{0}\gamma}=21.242$ keV. 111We note that the lattice
QCD simulation Becirevic and Sanfilippo (2013) gave relatively larger values,
i.e., $\Gamma_{D^{\ast 0}\to D^{0}\pi^{0}}=53\pm 9$ keV and $\Gamma_{D^{\ast
0}\to D^{0}\gamma}=33\pm 6$ keV. From isospin symmetry, we expect that the
$D^{*0}$ strong decay width be smaller than the $D^{*+}$ strong decay width
because the $D^{*0}\to D^{+}\pi^{-}$ decay mode is kinematically forbidden. As
a result, we do not use these lattice QCD results. With these numbers, we
obtain the couplings $g_{D^{\ast 0}D^{0}\pi^{0}}=11.688$ and $g_{D^{\ast
0}D^{0}\gamma}=$1.843 GeV-1. It is clear that the strong couplings satisfy
approximately isospin symmetry while the electromagnetic couplings do not.

With the above Lagrangians the decay amplitudes of $T_{cc}\to DD\pi$ and
$T_{cc}\to DD\gamma$ are

$\displaystyle\mathcal{M}_{T_{cc}\to DD\pi}$ $\displaystyle=$ $\displaystyle
ig_{T_{cc}}g_{DD^{\ast}\pi}~{}p_{2\mu}\frac{-g^{\mu\nu}+k_{1}^{\mu}k_{1}^{\nu}/m_{D^{\ast}}^{2}}{k_{1}^{2}-m_{D^{\ast}}^{2}+im_{D^{\ast}}\Gamma_{m_{D^{\ast}}}}\varepsilon_{\nu}(p_{0}),$
(7) $\displaystyle\mathcal{M}_{T_{cc}\to DD\gamma}$ $\displaystyle=$
$\displaystyle
ig_{T_{cc}}g_{DD^{\ast}\gamma}\varepsilon^{\mu\nu\alpha\beta}~{}p_{2\mu}\varepsilon_{\nu}(p_{2})k_{1\alpha}\frac{-g_{\beta\sigma}+k_{1\beta}k_{1\sigma}/m_{D^{\ast}}^{2}}{k_{1}^{2}-m_{D^{\ast}}^{2}+im_{D^{\ast}}\Gamma_{m_{D^{\ast}}}}\varepsilon^{\sigma}(p_{0}),$
(8)

where $p_{2}$, $k_{1}$, and $p_{0}$ are the momentum of $\pi$($\gamma$),
$D^{\ast}$, and $T_{cc}$, respectively. The partial decay widths of $T_{cc}\to
DD\pi$ and $T_{cc}\to DD\gamma$ as a function of $m_{12}^{2}$ and $m_{23}^{2}$
Zyla  _et al._ (2020) read:

$d\Gamma=\frac{1}{(2\pi)^{3}}\frac{1}{2J+1}\frac{\overline{|\mathcal{M}|^{2}}}{32m_{T_{cc}}^{3}}dm_{12}^{2}dm_{23}^{2},$
(9)

with $m_{12}$ the invariant mass of $DD$ and $m_{23}$ the invariant mass of
$D\pi$ or $D\gamma$ for the $T_{cc}\to DD\pi$ or $T_{cc}\to DD\gamma$ decay,
respectively.

In principle, there exist three possible decay channels, $T_{cc}\to
D^{0}D^{0}\pi^{+}$, $T_{cc}\to D^{0}D^{+}\pi^{0}$, and $T_{cc}\to
D^{+}D^{+}\pi^{-}$. In Fig. 3(a/b), we show the decay width of $T_{cc}\to
DD\pi/\gamma$ as a function of the $T_{cc}$ mass, where we take the size
parameter $\Lambda=$ 1 and 2 GeV. With the $T_{cc}$ mass varying from 3874.751
to 3874.883 MeV, the decay width of $T_{cc}\to DD\pi$ is found to be about 46
to 62 keV with the size parameter $\Lambda=1$ GeV, which is smaller than the
Breit-Wigner width by one order of magnitude Aaij  _et al._ (2021a), but close
to the result yielded from the unitary analysis Aaij  _et al._ (2021b). One
should note that the decay $T_{cc}(D^{\ast 0}D^{+})\to D^{+}\pi^{-}D^{+}$ is
kinematically forbidden. The $T_{cc}(D^{\ast+}D^{0})\to D^{0}\pi^{+}D^{0}$
contribution accounts for 50% of the total decay width, and the remaining is
from $T_{cc}(D^{\ast+}D^{0})\to D^{+}\pi^{0}D^{0}$ and $T_{cc}(D^{\ast
0}D^{+})\to D^{0}\pi^{0}D^{+}$.

\begin{overpic}[scale={.34}]{Gammas.eps} \end{overpic}
\begin{overpic}[scale={.34}]{Gammag.eps} \end{overpic}

Figure 3:  Total decay width and partial decay widths of $T_{cc}\to DD\pi$(a)
and $T_{cc}\to DD\gamma$ (b) as a function of the $T_{cc}$ mass. The solid and
dashed lines represent the results obtained with a cutoff $\Lambda=$1 GeV and
$\Lambda=$2 GeV, respectively.

The radiative decay width of $T_{cc}\to DD\gamma$ is about $10$ keV, which
mainly originates from the $D^{*0}D^{+}$ component of the $DD^{*}$ molecule
because the radiative decay width of $D^{\ast 0}$ is about 16 times of that of
$D^{\ast+}$ Rosner (2013). Considering isospin breaking due to the mass
difference of the $D^{*+}D^{0}$ and $D^{*0}D^{+}$ channels, the radiative
decay width will decrease. As a result, what we obtained should be viewed as
an upper limit. The ratio of the decay widths of $T_{cc}\to DD\gamma$ and
$T_{cc}\to DD\pi$ is in agreement with the ratio of the decay widths of
$D^{\ast}\to D\gamma$ and $D^{\ast}\to D\pi$, reflecting the fact that the
decay width of $T_{cc}$ is mainly from the decay of $D^{\ast}$. Since the
decay width of a free $D^{\ast+}$ is 83.4 keV Zyla  _et al._ (2020), the decay
width of a weakly bound $DD^{*}$ molecule is not expected to be larger than
83.4 keV.

##  III Summary and outlook

We studied the decay width of the $T_{cc}^{+}$ state in the effective
Lagrangian approach. Assuming that the $T_{cc}^{+}$ state is a hadronic
molecule of $DD^{*}$, we obtained a partial decay width of $T_{cc}\to DD\pi$
of about 53 keV and a radiative decay width of about 10 keV. Their sum is much
smaller than the central experimental value of the Breit-Wigner fit, but
agrees with that of the unitary fit. We argued that although the experimental
binding energy favors a molecular interpretation for the $T_{cc}$ state, a
complete understanding of its decay width is still missing.

The discovery of the doubly charmed tetraquark state $T_{cc}^{+}$ may open up
another new era for hadron physics, in the same way that the discovery of
$X(3872)$ did Choi  _et al._ (2003). For open charm exotic baryons, it is very
reasonable to expect a complete multiplet of doubly charmed
$D^{(\ast)}\Sigma_{c}^{\ast}$ hadronic molecules, which are more bound than
their hidden charm $\bar{D}^{(\ast)}\Sigma_{c}^{\ast}$ counterparts Liu  _et
al._ (2020), consistent with Refs. Dong  _et al._ (2021); Chen  _et al._
(2021a, b). From SU(3) symmetry, one may expect the existence of
$D^{\ast}D_{s}$ or $D^{\ast}D_{s}$ molecules. However, these two systems are
found difficult to bind, at least from the perspective of OBE models Li  _et
al._ (2013). In a series of recent works Sanchez Sanchez  _et al._ (2018);
Martinez Torres  _et al._ (2019); Wu  _et al._ (2019, 2021), we predicted one
$DDK$ bound state with isospin 1/2, spin-parity $0^{-}$, and a minimum quark
content of $cc\bar{s}\bar{q}$, which can be regraded as the strangeness
partner of $T_{cc}$. All these remain to be further studied in more detail
both theoretically and experimentally.

Note added: After the $T_{cc}^{+}$ state was discovered by the LHCb
Collaboration, a series of works have been performed to investigate the
property of $T_{cc}$ Yan and Valderrama (2021); Fleming  _et al._ (2021); Meng
_et al._ (2021); Chen  _et al._ (2021c); Feijoo  _et al._ (2021); Dai  _et
al._ (2021); Huang  _et al._ (2021); Hu  _et al._ (2021); Albaladejo (2021);
Jin  _et al._ (2021); Ren  _et al._ (2021); Dong  _et al._ (2021). In Refs Yan
and Valderrama (2021); Fleming  _et al._ (2021), in addition to the tree level
contribution, the final $DD$ rescattering effect was also taken into account.
However, it only contributes several keV, which obeys the power counting of
effective field theory. In Refs. Meng  _et al._ (2021); Chen  _et al._
(2021c), the authors argued that in the molecular picture of $T_{cc}^{+}$ the
explicit breaking of isospin should be considered and it may lead to another
molecular state mainly coupling to the $D^{\ast 0}D^{+}$ channel. However, the
authors of Ref. Feijoo  _et al._ (2021) did not find another $T_{cc}$
denominated by the $D^{\ast 0}D^{+}$ channel in the chiral unitary model. In
Ref. Dai  _et al._ (2021), Dai et al. have considered the interaction of
$D^{\ast+}D^{0}$ and $D^{0}D^{0}\pi^{+}$ coupled channels by fitting to the
LHCb data, and interpreted the $T_{cc}^{+}$ state as a virtual state.
Moreover, several theoretical works investigated the production of $T_{cc}$
Huang  _et al._ (2021); Hu  _et al._ (2021). In Ref. Albaladejo (2021), M.
Albaladcjo predicted several $D^{\ast}D^{\ast}$ bound states based on heavy
quark spin symmetry and the molecular picture where the $T_{cc}$ state is a
$DD^{\ast}$ molecule.

##  IV Acknowledgments

MZL thank Mao-Jun Yan for useful discussions. This work is partly supported by
the National Natural Science Foundation of China under Grants No.12105007,
No.11735003, No.11975041, No.11961141004, No.11961141012, No.12075288, and
No.1210050997, and the fundamental Research Funds for the Central
Universities, the Youth Innovation Promotion Association CAS, the Key Research
Projects of Henan Higher Education Institutions under No. 20A140027, the
Project of Youth Backbone Teachers of Colleges and Universities of Henan
Province (2020GGJS017), and the Fundamental Research Cultivation Fund for
Young Teachers of Zhengzhou University (JC202041042).

## References

  * Gell-Mann (1964) M. Gell-Mann, Phys. Lett. 8, 214 (1964).
  * Choi  _et al._ (2008) S. K. Choi _et al._ (Belle), Phys. Rev. Lett. 100, 142001 (2008), arXiv:0708.1790 [hep-ex] .
  * Aaij  _et al._ (2015) R. Aaij _et al._ (LHCb), Phys. Rev. Lett. 115, 072001 (2015), arXiv:1507.03414 [hep-ex] .
  * Aaij  _et al._ (2019) R. Aaij _et al._ (LHCb), Phys. Rev. Lett. 122, 222001 (2019), arXiv:1904.03947 [hep-ex] .
  * Aaij  _et al._ (2020a) R. Aaij _et al._ (LHCb), Sci. Bull. 65, 1983 (2020a), arXiv:2006.16957 [hep-ex] .
  * Aaij  _et al._ (2020b) R. Aaij _et al._ (LHCb), Phys. Rev. Lett. 125, 242001 (2020b), arXiv:2009.00025 [hep-ex] .
  * Ader  _et al._ (1982) J. P. Ader, J. M. Richard, and P. Taxil, Phys. Rev. D 25, 2370 (1982).
  * Zouzou  _et al._ (1986) S. Zouzou, B. Silvestre-Brac, C. Gignoux, and J. M. Richard, Z. Phys. C 30, 457 (1986).
  * Lipkin (1986) H. J. Lipkin, Phys. Lett. B 172, 242 (1986).
  * Heller and Tjon (1987) L. Heller and J. A. Tjon, Phys. Rev. D 35, 969 (1987).
  * Carlson  _et al._ (1988) J. Carlson, L. Heller, and J. A. Tjon, Phys. Rev. D 37, 744 (1988).
  * Silvestre-Brac and Semay (1993) B. Silvestre-Brac and C. Semay, Z. Phys. C 57, 273 (1993).
  * Semay and Silvestre-Brac (1994) C. Semay and B. Silvestre-Brac, Z. Phys. C 61, 271 (1994).
  * Manohar and Wise (1993) A. V. Manohar and M. B. Wise, Nucl. Phys. B 399, 17 (1993), arXiv:hep-ph/9212236 .
  * Pepin  _et al._ (1997) S. Pepin, F. Stancu, M. Genovese, and J. M. Richard, Phys. Lett. B 393, 119 (1997), arXiv:hep-ph/9609348 .
  * Mattson  _et al._ (2002) M. Mattson _et al._ (SELEX), Phys. Rev. Lett. 89, 112001 (2002), arXiv:hep-ex/0208014 .
  * Gelman and Nussinov (2003) B. A. Gelman and S. Nussinov, Phys. Lett. B551, 296 (2003), arXiv:hep-ph/0209095 [hep-ph] .
  * Vijande  _et al._ (2004) J. Vijande, F. Fernandez, A. Valcarce, and B. Silvestre-Brac, Eur. Phys. J. A 19, 383 (2004), arXiv:hep-ph/0310007 .
  * Choi  _et al._ (2003) S. K. Choi _et al._ (Belle), Phys. Rev. Lett. 91, 262001 (2003), arXiv:hep-ex/0309032 .
  * Janc and Rosina (2004) D. Janc and M. Rosina, Few Body Syst. 35, 175 (2004), arXiv:hep-ph/0405208 [hep-ph] .
  * Navarra  _et al._ (2007) F. S. Navarra, M. Nielsen, and S. H. Lee, Phys. Lett. B 649, 166 (2007), arXiv:hep-ph/0703071 .
  * Vijande  _et al._ (2007) J. Vijande, E. Weissman, A. Valcarce, and N. Barnea, Phys. Rev. D76, 094027 (2007), arXiv:0710.2516 [hep-ph] .
  * Ebert  _et al._ (2007) D. Ebert, R. N. Faustov, V. O. Galkin, and W. Lucha, Phys. Rev. D76, 114015 (2007), arXiv:0706.3853 [hep-ph] .
  * Yang  _et al._ (2009) Y. Yang, C. Deng, J. Ping, and T. Goldman, Phys. Rev. D 80, 114023 (2009).
  * Molina  _et al._ (2010) R. Molina, T. Branz, and E. Oset, Phys. Rev. D 82, 014010 (2010), arXiv:1005.0335 [hep-ph] .
  * Li  _et al._ (2013) N. Li, Z.-F. Sun, X. Liu, and S.-L. Zhu, Phys. Rev. D88, 114008 (2013), arXiv:1211.5007 [hep-ph] .
  * Feng  _et al._ (2013) G. Q. Feng, X. H. Guo, and B. S. Zou, (2013), arXiv:1309.7813 [hep-ph] .
  * Luo  _et al._ (2017) S.-Q. Luo, K. Chen, X. Liu, Y.-R. Liu, and S.-L. Zhu, (2017), arXiv:1707.01180 [hep-ph] .
  * Wang (2018) Z.-G. Wang, Acta Phys. Polon. B 49, 1781 (2018), arXiv:1708.04545 [hep-ph] .
  * Junnarkar  _et al._ (2019) P. Junnarkar, N. Mathur, and M. Padmanath, Phys. Rev. D 99, 034507 (2019), arXiv:1810.12285 [hep-lat] .
  * Maiani  _et al._ (2019) L. Maiani, A. D. Polosa, and V. Riquer, Phys. Rev. D 100, 074002 (2019), arXiv:1908.03244 [hep-ph] .
  * Liu  _et al._ (2019) M.-Z. Liu, T.-W. Wu, M. Pavon Valderrama, J.-J. Xie, and L.-S. Geng, Phys. Rev. D99, 094018 (2019), arXiv:1902.03044 [hep-ph] .
  * Aaij  _et al._ (2017) R. Aaij _et al._ (LHCb), Phys. Rev. Lett. 119, 112001 (2017), arXiv:1707.01621 [hep-ex] .
  * Karliner and Rosner (2017) M. Karliner and J. L. Rosner, (2017), arXiv:1707.07666 [hep-ph] .
  * Mehen (2017) T. Mehen, Phys. Rev. D 96, 094028 (2017), arXiv:1708.05020 [hep-ph] .
  * Eichten and Quigg (2017) E. J. Eichten and C. Quigg, Phys. Rev. Lett. 119, 202002 (2017), arXiv:1707.09575 [hep-ph] .
  * Aaij  _et al._ (2021a) R. Aaij _et al._ (LHCb), (2021a), arXiv:2109.01056 [hep-ex] .
  * Aaij  _et al._ (2021b) R. Aaij _et al._ (LHCb), (2021b), arXiv:2109.01038 [hep-ex] .
  * Liu  _et al._ (2020) M.-Z. Liu, J.-J. Xie, and L.-S. Geng, (2020), arXiv:2008.07389 [hep-ph] .
  * Dong  _et al._ (2008) Y.-b. Dong, A. Faessler, T. Gutsche, and V. E. Lyubovitskij, Phys. Rev. D 77, 094013 (2008), arXiv:0802.3610 [hep-ph] .
  * Huang  _et al._ (2020) Y. Huang, M.-Z. Liu, Y.-W. Pan, L.-S. Geng, A. Martínez Torres, and K. P. Khemchandani, Phys. Rev. D101, 014022 (2020), arXiv:1909.09021 [hep-ph] .
  * Ling  _et al._ (2021) X.-Z. Ling, J.-X. Lu, M.-Z. Liu, and L.-S. Geng, (2021), arXiv:2106.12250 [hep-ph] .
  * Weinberg (1963) S. Weinberg, Phys. Rev. 130, 776 (1963).
  * Salam (1962) A. Salam, Nuovo Cim. 25, 224 (1962).
  * Hayashi  _et al._ (1967) K. Hayashi, M. Hirayama, T. Muta, N. Seto, and T. Shirafuji, Fortsch. Phys. 15, 625 (1967).
  * Zyla  _et al._ (2020) P. A. Zyla _et al._ (Particle Data Group), PTEP 2020, 083C01 (2020).
  * Rosner (2013) J. L. Rosner, Phys. Rev. D 88, 034034 (2013), arXiv:1307.2550 [hep-ph] .
  * Feijoo  _et al._ (2021) A. Feijoo, W. H. Liang, and E. Oset, (2021), arXiv:2108.02730 [hep-ph] .
  * Becirevic and Sanfilippo (2013) D. Becirevic and F. Sanfilippo, Phys. Lett. B 721, 94 (2013), arXiv:1210.5410 [hep-lat] .
  * Dong  _et al._ (2021) X.-K. Dong, F.-K. Guo, and B.-S. Zou,  (2021), 10.1088/1572-9494/ac27a2, arXiv:2108.02673 [hep-ph] .
  * Chen  _et al._ (2021a) R. Chen, N. Li, Z.-F. Sun, X. Liu, and S.-L. Zhu, Phys. Lett. B 822, 136693 (2021a), arXiv:2108.12730 [hep-ph] .
  * Chen  _et al._ (2021b) K. Chen, B. Wang, and S.-L. Zhu, Phys. Rev. D 103, 116017 (2021b), arXiv:2102.05868 [hep-ph] .
  * Sanchez Sanchez  _et al._ (2018) M. Sanchez Sanchez, L.-S. Geng, J.-X. Lu, T. Hyodo, and M. P. Valderrama, Phys. Rev. D 98, 054001 (2018), arXiv:1707.03802 [hep-ph] .
  * Martinez Torres  _et al._ (2019) A. Martinez Torres, K. P. Khemchandani, and L.-S. Geng, Phys. Rev. D99, 076017 (2019), arXiv:1809.01059 [hep-ph] .
  * Wu  _et al._ (2019) T.-W. Wu, M.-Z. Liu, L.-S. Geng, E. Hiyama, and M. P. Valderrama, Phys. Rev. D100, 034029 (2019), arXiv:1906.11995 [hep-ph] .
  * Wu  _et al._ (2021) T.-W. Wu, M.-Z. Liu, and L.-S. Geng, Phys. Rev. D 103, L031501 (2021), arXiv:2012.01134 [hep-ph] .
  * Yan and Valderrama (2021) M.-J. Yan and M. P. Valderrama, (2021), arXiv:2108.04785 [hep-ph] .
  * Fleming  _et al._ (2021) S. Fleming, R. Hodges, and T. Mehen, (2021), arXiv:2109.02188 [hep-ph] .
  * Meng  _et al._ (2021) L. Meng, G.-J. Wang, B. Wang, and S.-L. Zhu, Phys. Rev. D 104, 051502 (2021), arXiv:2107.14784 [hep-ph] .
  * Chen  _et al._ (2021c) R. Chen, Q. Huang, X. Liu, and S.-L. Zhu, (2021c), arXiv:2108.01911 [hep-ph] .
  * Dai  _et al._ (2021) L.-Y. Dai, X. Sun, X.-W. Kang, A. P. Szczepaniak, and J.-S. Yu, (2021), arXiv:2108.06002 [hep-ph] .
  * Huang  _et al._ (2021) Y. Huang, H. Q. Zhu, L.-S. Geng, and R. Wang, (2021), arXiv:2108.13028 [hep-ph] .
  * Hu  _et al._ (2021) Y. Hu, J. Liao, E. Wang, Q. Wang, H. Xing, and H. Zhang, (2021), arXiv:2109.07733 [hep-ph] .
  * Albaladejo (2021) M. Albaladejo, (2021), arXiv:2110.02944 [hep-ph] .
  * Jin  _et al._ (2021) Y. Jin, S.-Y. Li, Y.-R. Liu, Q. Qin, Z.-G. Si, and F.-S. Yu, (2021), arXiv:2109.05678 [hep-ph] .
  * Ren  _et al._ (2021) H. Ren, F. Wu, and R. Zhu, (2021), arXiv:2109.02531 [hep-ph] .


# To the Max: Reinventing Reward in Reinforcement Learning

Grigorii Veviurko  Wendelin Boehmer  Mathijs de Weerdt

###### Abstract

In reinforcement learning (RL), different rewards can define the same optimal
policy but result in drastically different learning performance. For some, the
agent gets stuck with a suboptimal behavior, and for others, it solves the
task efficiently. Choosing a good reward function is hence an extremely
important yet challenging problem. In this paper, we explore an alternative
approach to using rewards for learning. We introduce max-reward RL, where an
agent optimizes the maximum rather than the cumulative reward. Unlike earlier
works, our approach works for deterministic and stochastic environments and
can be easily combined with state-of-the-art RL algorithms. In the
experiments, we study the performance of max-reward RL algorithms in two goal-
reaching environments from Gymnasium-Robotics and demonstrate its benefits
over standard RL. The code is publicly available.111Placeholder for the GitHub
link.

Reinforcement Learning, Machine Learning, ICML

##  1 Introduction

Reinforcement Learning (RL) is a learning paradigm where an intelligent agent
solves sequential decision-making problems through trial and error. The main
objective that an RL agent learns to optimize is the cumulative return, i.e.,
a discounted sum of the rewards. This makes the reward a crucial element of
the problem, as it defines the optimal decision-making policy that the agent
tries to learn.

It is well known (Ng et al., 1999) that there are infinitely many ways to
define the reward function under which a desired policy is optimal.
Practically, however, these rewards often result in drastically different
learning processes. For example, many major successes of RL required
meticulous engineering of the reward: by hand, e.g. (Berner et al., 2019) or
by learning it from a human example (Vinyals et al., 2019). Hence, designing a
reward function that enables learning and corresponds to a certain optimal
policy is a challenging problem in modern reinforcement learning.

In many RL problems, the true reward function is sparse, i.e., only successful
completion of the task is rewarded. In particular, the sparse reward is
characteristic to goal-reaching problems where the agent needs to enter the
goal state (Plappert et al., 2018; Florensa et al., 2018; Ghosh et al., 2020).
Sparse reward problems are notoriously hard to solve with standard RL. A
popular and simple solution is to introduce a dense surrogate reward that
represents some sort of distance between the agent and the goal (Towers et
al., 2023; de Lazcano et al., 2023). However, this approach is very sensitive
and should be tailored to each problem individually, because otherwise the
agent can get stuck with a sub-optimal policy exploiting the dense reward. To
overcome this issue, we propose max-reward RL, where the agent optimizes the
maximum reward achieved in the episode rather than the cumulative return.
Under this paradigm, the reward design process becomes much more natural and
straightforward.

One of the key properties of the cumulative return is that it satisfies the
Bellman equation (Bellman, 1954) and hence can be efficiently approximated and
optimized by iteratively applying the Bellman operator. To make the max-reward
RL approach viable, an analogous learning rule is required. However, Cui & Yu
(2023) prove that naively changing summation into a max operator in the
standard Bellman update rule works only in a deterministic setting and hence
cannot be used in most RL problems and algorithms.

Inspired by results from stochastic optimal control theory (Kröner et al.,
2018), this paper introduces a theoretically justified framework for max-
reward RL in the general stochastic setting. We prove the stochastic and
deterministic policy gradient theorems and reformulate some of the state-of-
the-art algorithms (PPO, TD3) for the max-reward case. Using the Maze
environment (de Lazcano et al., 2023) with different dense rewards, we
experimentally demonstrate that max-reward algorithms outperform their
cumulative counterparts. Finally, experiments with a challenging Fetch
environment (de Lazcano et al., 2023) show the promise of max-reward RL in
more realistic goal-reaching problems.

##  2 Related work

The first attempt to formulate max-reward RL was made by Quah & Quek (2006),
where the authors derived a learning rule for the maximum reward state-action
value function. However, as it was shown later (Gottipati et al., 2020), that
work made a technical error of interchanging expectation and maximum
operators. Gottipati et al. (2020) corrected this error, but the value
functions learned via their approach differ from the expected maximum reward
if stochasticity is present. Independently, Wang et al. (2020) derived a
similar method in the context of planning in deterministic Markov Decision
Processes (MDPs). Later, Cui & Yu (2023) demonstrated that the presence of
stochasticity poses a problem not only for the max-reward RL but also for
other non-cumulative rewards.

There exists a parallel branch of research that (re)discovered maximum reward
value functions in the context of safe RL for reach-avoid problems (Fisac et
al., 2014). In their work, Fisac et al. (2019) considered a deterministic
open-loop dynamic system, where the agent’s goal is to avoid constraint
violations. The authors derived a contraction operator, similar to the one by
Gottipati et al. (2020), to learn the max-cost safe value function. Hsu et al.
(2021) extended this approach to reach-avoid problems, where the goal is to
reach the goal while not violating constraints. Later, max-cost value
functions were applied to the safe RL context to learn the best-performing
policy that does not violate the constraints (Yu et al., 2022). The main
limitation of the three aforementioned works is the same as for (Gottipati et
al., 2020) – their methods only apply to deterministic environments and
policies.

Effective reward design is a long-standing challenge in reinforcement learning
which dates back to at least as early as 1994 (Mataric, 1994). In this
paragraph, we briefly summarize the existing work related to the reward design
problem. For further reading, we refer the reader to Eschmann (2021). Some of
the big successes of RL utilize a hand-designed reward function, e.g., in the
game of DOTA (Berner et al., 2019) or robots playing soccer (Haarnoja et al.,
2023). However, manually designed rewards often lead to undesirable behavior
(Krakovna et al., 2020). Alternatively, the reward can be designed in an
automated fashion. For example, based on state novelty to encourage
exploration (Tang et al., 2017; Pathak et al., 2017; Burda et al., 2018), by
learning it from the experiences (Trott et al., 2019), or by using human data
(Ibarz et al., 2018).

To conclude, reward design and reward shaping remain challenging topics. In
this work, we propose a new way to think about the reward – the max-reward RL
framework. While self-sufficient in some cases, this approach can also be
combined with various existing methods for reward design.

##  3 Background

Figure 1: Five-state chain MDP and the training results of cumulative (in
green) and max-reward (in violet) Q-learning. The vertical axis is the number
of training epochs needed to recover the optimal policy; the horizontal axis
represents the values of the intermediate reward $x.$ Four panels correspond
to different probabilities of skipping transitions into $s_{4}$ when training.

We consider a standard reinforcement learning setup for continuous
environments. An agent interacts with an MDP defined by a tuple
$(\mathcal{S},\mathcal{A},R,P,p_{0},\gamma),$ where $\mathcal{S}$ is the
continuous state space, $\mathcal{A}$ is the continuous action space, and
${R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to[0,\bar{R}]}$ is a non-
negative and bounded reward function.222Non-negativity of reward can be
achieved in any MDP with bounded reward function. For each state-action pair,
$(s,a)\in\mathcal{S}\times\mathcal{A}$, the transition function
$P(\cdot|s,a)\in\mathcal{P}(\mathcal{S})$ is a probability density function
(PDF) of the next state $s^{\prime}$ and
$p_{0}(\cdot)\in\mathcal{P}(\mathcal{S})$ is the PDF of the initial state
$s_{0}$. Scalar $0\leq\gamma<1$ is the discount factor. We use
$\pi:\mathcal{S}\to\mathcal{P}(\mathcal{A})$ to denote a stochastic policy and
$\mu:\mathcal{S}\to\mathcal{A}$ to denote a deterministic policy. The time is
discrete and starts at zero, i.e., $t\in\mathbb{N}\cup\\{0\\}$. For each
timestep $t$, the state is denoted by $s_{t}$, the action by $a_{t}$, and the
reward by $r_{t+1}:=R(s_{t},a_{t},s_{t+1}).$ Everywhere in the text, the
expectation over policy, ${\mathbb{E}}_{\pi}$, denotes the expectation over
the joint distribution of $s_{t},a_{t},r_{t+1}$ for
$t\in\mathbb{N}\cup\\{0\\}$ induced by $\pi,P$, and $p_{0}.$ Sometimes, we use
such notation as ${\mathbb{E}}_{x\sim\pi}$ (or just ${\mathbb{E}}_{x}$) to
emphasize that the expectation is taken only over $x$.

In standard RL, the main quantity being optimized is the cumulative return,
defined as $G_{t}=\sum_{i=0}^{\infty}\gamma^{i}r_{t+1+i}$. To maximize
$\mathbb{E}_{\pi}\big{[}G_{t}\big{]}$, most RL algorithms learn state and/or
state-action value functions defined as follows:

$v^{\pi}(s)=\mathbb{E}_{\pi}\big{[}G_{t}\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\big{]}\,,\quad
v^{\ast}(s)=\max_{\pi}v^{\pi}(s).$
$q^{\pi}(s,a)=\mathbb{E}_{\pi}\big{[}G_{t}\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}\,,\quad q^{\ast}(s,a)=\max_{\pi}q^{\pi}(s,a).$

Crucially, these functions are solutions to the corresponding Bellman
equations:

$v^{\pi}(s)={{\mathbb{E}}}_{\begin{subarray}{c}a_{t}\\\
s_{t+1}\end{subarray}}\big{[}r_{t+1}+\gamma
v^{\pi}(s_{t+1})\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\big{]}$
$q^{\pi}(s,a)={{\mathbb{E}}}_{\begin{subarray}{c}s_{t+1}\\\
a_{t+1}\end{subarray}}\big{[}r_{t+1}+\gamma
q^{\pi}(s_{t+1},a_{t+1})\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$
$q^{\ast}(s,a)={{\mathbb{E}}}_{{s_{t+1}}}\big{[}r_{t+1}+\gamma\max_{a^{\prime}}q^{\ast}(s_{t+1},a^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$

The defining feature of these equations is that they can be solved by
repeatedly applying Bellman operators. These operators are contractions and
hence each of them has a unique fixed point that corresponds to one of the
value functions above. For example, the optimal state-action value function
$q^{\ast}(s,a)$ is the fixed point of the Bellman optimallity operator
$\mathcal{T}^{\ast}$:

$\big{(}\mathcal{T}^{\ast}q\big{)}(s,a)=\mathbb{E}_{s_{t+1}}\big{[}r_{t+1}+\gamma\max_{a^{\prime}}q(s_{t+1},a^{\prime})\big{|}\begin{subarray}{c}\\\
s_{t}=s\\\ a_{t}=a\end{subarray}\big{]}$ (1)

The Bellman equation is foundational for all state-of-the-art RL algorithms as
it allows training neural networks to approximate value functions. Therefore,
for the max-reward framework to be useable, it is necessary to derive an
analog of the Bellman equation. Below, we describe such an attempt made by
Gottipati et al. (2020) and demonstrate that it is limited to purely
deterministic problems.

###  3.1 Deterministic max-reward RL

Instead of cumulative return, max-reward RL aims at optimizing the max-reward
return:

$\hat{G}_{t}=\max\big{\\{}r_{t+1},\gamma
r_{t+2},\gamma^{2}r_{t+3}\ldots\big{\\}}$ (2)

To approximate $\mathbb{E}_{\pi}\big{[}\hat{G}_{t}\big{]}$, Gottipati et al.
(2020) introduced the following definition of the state-action value
functions:

$\hat{q}^{\pi}_{det}(s,a)=\mathbb{E}_{\begin{subarray}{c}a_{t}\\\
s_{t+1}\end{subarray}}\Big{[}r_{t+1}\lor\gamma
q(s_{t+1},a_{t+1})\Big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\Big{]}$
$\hat{q}^{\ast}_{det}(s,a)=\mathbb{E}_{s_{t+1}}\Big{[}r_{t+1}\lor\gamma\max_{a^{\prime}}q(s_{t+1},a^{\prime})\Big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\Big{]}$

where $\lor$ denotes the binary $\max$ operator, i.e., ${a\lor
b:=\max\\{a,b\\}.}$ By construction, $\hat{q}^{\ast}_{det}$ and
$\hat{q}^{\pi}_{det}$ satisfy Bellman-like recursive equations. In their work,
Gottipati et al. (2020) proved that the following operator is a contraction:

$\big{(}\hat{\mathcal{T}}^{\ast}_{det}q\big{)}(s,a)=\mathop{{\mathbb{E}}}_{{s_{t+1}}}\big{[}r_{t+1}\lor\gamma\max_{a^{\prime}}q(s_{t+1},a^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$ (3)

Therefore, $\hat{q}^{\ast}_{det}$ is the unique fixed point of
$\hat{\mathcal{T}}_{det}^{\ast}$ and can be learned, e.g., with Q-learning.

#### Chain environment example.

Before going into the limitations of the approach above, we conduct a simple
experiment to motivate the use of max-reward reinforcement learning. We show
that max-reward RL is a better approach in a goal-reaching problem where the
agent needs to learn to reach the goal state. Specifically, it dominates the
standard cumulative RL when transitions into the goal state occur infrequently
in the training data, which is often the case in larger-scale goal-reaching
problems.

Consider the five-state chain environment in Figure 1. Transitions leading
into $s_{4}$ have reward of $1,$ transitions into $s_{2}$ have a reward
parametrized with $x\in(0,1)$, and other rewards are zero. Hence, the optimal
policy, concerning both max-reward and cumulative returns, is to go to $s_{4}$
and stay there. We run tabular Q-learning using standard (Eq. (1)) and max-
reward (Eq. (3)) Bellman operators for different values of the intermediate
reward $x.$ In each training epoch, we iterate over all possible transitions.
For each transition, we compute the target value using one of the Bellman
operators and update the Q-table. Crucially, we _randomly skip some of the
transitions into $s_{4}$ with a certain probability_. In the experiment, we
consider four values for the skip probability –
$p_{skip}\in\\{0,0.3,0.6,0.9\\}.$ During training, when a transition into
$s_{4}$ is sampled, the Q-table is updated with probability $1-p_{skip}$ and
otherwise left unchanged. Transitions into other states are never skipped. In
this way, we can control how often the agent is exposed to the transitions
into the optimal state and thereby simulate problems where goal-reaching
transitions are rarely encountered.

The results in Figure 1 indicate that for larger values of the skip
probability, the max-reward approach converges to the optimal policy
significantly faster than standard Q-learning. We believe that this phenomenon
can be explained by differences in bootstrapping. In standard RL, the target
for the $q-$value is a sum of the immediate reward and the $q-$value at the
next timestep. Therefore, this target changes in each epoch until convergence.
In the max-reward case, on the other hand, the target in the max-reward state
is just the reward and does not change with time. This example suggests that
the max-reward approach is a better choice in environments where the task of
the agent is to reach the goal state.

Figure 2: A three-state MDP with two possible policies.

Unfortunately, the max-reward approach described above has a serious
theoretical drawback. Expanding the definition of $\hat{q}^{\ast}_{det}$ for
more timesteps, we obtain a nested sequence of non-interchangeable $\lor$ and
$\mathbb{E}$:

$\hat{q}^{\ast}_{det}(s,a)={\mathbb{E}}_{\pi^{\ast}}\Big{[}r_{t+1}\lor\gamma{\mathbb{E}}_{\pi^{\ast}}\big{[}r_{t+2}\lor\ldots\big{]}\Big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\Big{]}$

Using Jensen’s inequality (Jensen, 1906), we conclude the following:

$\hat{q}_{det}^{\ast}(s,a)\leq{\mathbb{E}}_{\pi^{\ast}}\big{[}\hat{G}_{t}|\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$ (4)

When both the policy and the transition model are deterministic, Eq. (4)
becomes an equality. However, if stochasticity is present, the value of
$\hat{q}_{det}^{\ast}(s,a)$ is merely a lower bound of the expected return.
Hence, it can induce suboptimal policies.

In Figure 2, we show an example where the policy maximizing
$\hat{q}^{\ast}_{det}$ is suboptimal. The figure demonstrates a three-state
MDP and two policies, $\pi_{1}$ (red arrows) and $\pi_{2}$ (blue arrows). Let
$\gamma=1$ for simplicity. For the state $s_{0}$, the expected max-reward
return is higher for the policy $\pi_{1}:$

$\displaystyle{\mathbb{E}}_{\pi_{1}}[\hat{G}_{0}]$
$\displaystyle={\mathbb{E}}_{\pi_{1}}[r_{1}\lor r_{2}]=9>$
$\displaystyle{\mathbb{E}}_{\pi_{2}}[\hat{G}_{0}]={\mathbb{E}}_{\pi_{2}}[r_{1}\lor
r_{2}]=8$

So $\pi_{1}$ is better in terms of the expected max-reward return, but the
value functions have the following values:

$\hat{q}_{det}^{\pi_{1}}(s_{0})={\mathbb{E}}_{\pi_{1}}\big{[}r_{1}\lor{\mathbb{E}}_{\pi_{1}}[r_{2}]\big{]}={\mathbb{E}}_{\pi_{1}}[r_{1}\lor
6]=6$
$\hat{q}_{det}^{\pi_{2}}(s_{0})={\mathbb{E}}_{\pi_{2}}\big{[}r_{1}\lor{\mathbb{E}}_{\pi_{2}}[r_{2}]\big{]}={\mathbb{E}}_{\pi_{2}}[r_{1}\lor
8]=8$

Based on the values of $\hat{q}_{det}^{\pi}$, we would conclude that $\pi_{2}$
is better, which we already showed to be incorrect. This example demonstrates
that even in a simple stochastic environment, the operator
$\hat{T}_{det}^{\ast}$ can lead to incorrect policies. Therefore, it is an
open question whether there exists a Bellman-like operator that would enable
learning max-reward returns in the stochastic setting.

##  4 Max-reward RL

In this section, we introduce a novel approach to max-reward RL that is
theoretically sound, works for both stochastic and deterministic cases, and
can be combined with state-of-the-art RL algorithms. First, we expand the
definition of the max-reward return given in Eq. (2):

${\mathbb{E}}_{\pi}[\hat{G}_{t}]={\mathbb{E}}_{\pi}\big{[}r_{t+1}\lor\gamma\hat{G}_{t+1}\big{]}$
(5)

Since ${\mathbb{E}}$ and $\lor$ do not commute, it is impossible to extract
the term ${\mathbb{E}}_{\pi}\big{[}G_{t+1}\big{]}$ on the right-hand side of
Eq. (5). Because of that, we cannot obtain an equation involving only
${\mathbb{E}}_{\pi}\big{[}G_{t}\big{]},\;{\mathbb{E}}_{\pi}\big{[}G_{t+1}\big{]}$,
and $r_{t+1}.$ Instead, we will utilize an approach from stochastic optimal
control theory (Kröner et al., 2018) and define the max-reward value function
using an auxiliary variable that allows propagating information between
timesteps:

######  Definition 4.1.

Let $y\in\mathbb{R}$ be an auxiliary real variable. The _max-reward value
functions_ are defined as follows:

$\hat{v}^{\pi}(s,y)={\mathbb{E}}_{\pi}[y\lor\hat{G}_{t}\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}]$
$\hat{q}^{\pi}(s,a,y)={\mathbb{E}}_{\pi}[y\lor\hat{G}_{t}\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}]$

Since reward is lower-bounded, $r_{t+1}\geq 0$, we can always recover the
expected max-reward return ${\mathbb{E}}_{\pi}\big{[}\hat{G}_{t}\big{]}$ by
substituting $y=0$ into the value functions:

$\displaystyle\hat{v}^{\pi}(s,0)={\mathbb{E}}_{\pi}[\hat{G}_{t}\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}]$
(6)
$\displaystyle\hat{q}^{\pi}(s,a,0)={\mathbb{E}}_{\pi}[\hat{G}_{t}\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}]$

Hence, if we find an efficient method of learning the max-reward value
functions, we will also be able to optimize ${\mathbb{E}}_{\pi}[\hat{G}_{t}]$.
By combining the definition of the max-reward value functions with Eq. (5), we
obtain the following recursive equations:

######  Lemma 4.2.

Let $y\in{\mathbb{R}}$ and let $y^{\prime}:=\frac{R(s,a,s_{t+1})\lor
y}{\gamma}$. Then, the max-reward value functions are subject to the following
Bellman-like equations:

$\hat{v}^{\pi}(s,y)=\gamma{\mathbb{E}}_{\begin{subarray}{c}a_{t}\\\
s_{t+1}\end{subarray}}\big{[}y^{\prime}\lor\hat{v}^{\pi}(s_{t+1},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\big{]}$
$\hat{q}^{\pi}(s,a,y)=\gamma{\mathbb{E}}_{\begin{subarray}{c}s_{t+1}\\\
a_{t+1}\end{subarray}}\big{[}y^{\prime}\lor\hat{q}^{\pi}(s_{t+1},a_{t+1},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$

Proof of this lemma, as well as all other proofs, can be found in Appendix A.
The auxiliary variable $y$ is crucial for Lemma 4.2, as it propagates
information between the value functions at timesteps $t$ and $t+1$. The extra
term ${y^{\prime}\lor}$ might seem redundant, but it is important since it
enforces the boundary conditions. Without it, the functions $v\equiv 0$ and
$q\equiv 0$ would be solutions to these equations. Using Lemma 4.2, we can
define Bellman-like operators for the max-reward value functions:

######  Definition 4.3.

Let
$v:\mathcal{S}\times\mathbb{R}\to\mathbb{R},\>q:\mathcal{S}\times\mathcal{A}\times\mathbb{R}\to\mathbb{R}$
be real-valued functions and let $y^{\prime}:=\frac{R(s,a,s_{t+1})\lor
y}{\gamma}$. Then, the _max-reward Bellman operator_ $\hat{\mathcal{T}}^{\pi}$
is defined as follows:

$\hat{\mathcal{T}}^{\pi}v(s,y):=\gamma{\mathbb{E}}_{\begin{subarray}{c}a_{t}\\\
s_{t+1}\end{subarray}}\big{[}y^{\prime}\lor
v(s_{t+1},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\big{]}$
$\hat{\mathcal{T}}^{\pi}q(s,a,y):=\gamma{\mathbb{E}}_{\begin{subarray}{c}s_{t+1}\\\
a_{t+1}\end{subarray}}\big{[}y^{\prime}\lor
q(s_{t+1},a_{t+1},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$

In the following theorem, we prove that this operator is a contraction and
that the max-reward state and state-action value functions are its fixed
points.

######  Theorem 4.4.

$\hat{\mathcal{T}}^{\pi}$ is a $\gamma-$contraction with respect to the
$L_{\infty}$ norm, and $\hat{v}^{\pi}$(or $\hat{q}^{\pi}$) is its fixed point.

Theorem 4.4 implies that the max-reward value functions can be learned in the
same way as the standard value functions – by sampling from the environment
and applying Bellman operators. In the next section, we define the objective
function of the max-reward RL problem and discuss how the presence of the
auxiliary variable $y$ impacts the notion of optimal policy.

###  4.1 Max-reward objective

Similarly to standard RL, the main objective in the max-reward RL problems is
to maximize the expected (max-reward) return from the initial state, defined
as follows:

$\hat{J}(\pi)={\mathbb{E}}_{s_{0}\sim
p_{0}}\big{[}\hat{v}^{\pi}(s_{0},0)\big{]}$ (7)

Then, the optimal policy is naturally defined as :

$\pi^{\ast}=\arg\max_{\pi}\hat{J}(\pi).$ (8)

To better understand the properties of the max-reward optimal policy, consider
again the MDP in Figure 2. Let $\gamma=1.$ Then, the values of the objective
function for $\pi_{1}$ and $\pi_{2}$ can be computed as follows:

$\hat{J}(\pi_{1})={\mathbb{E}}_{\pi_{1}}[6\lor r_{2}]=9$
$\hat{J}(\pi_{2})={\mathbb{E}}_{\pi_{2}}[6\lor r_{2}]=8$

Hence, $\pi_{1}$ is optimal. However, if we consider the max-reward return
from $t=1,$ we have

${\mathbb{E}}_{\pi_{1}}[G_{1}]=\frac{12+0}{2}=6\quad{\mathbb{E}}_{\pi_{2}}[G_{1}]=\frac{9+7}{2}=8$

and hence $\pi_{2}$ obtains higher expected max-reward return starting at
$s=s_{1}.$ Seemingly, there is a contradiction: $\pi_{1}$ is optimal but
$\pi_{2}$ is better from the state $s_{1}.$ However, the explanation is
simple: the maximum reward is the highest reward encountered anywhere along
the trajectory. An optimal decision thus not only depends on the current
state, as with the cumulative reward, but it also depends on the maximum
reward that has been acquired thus far. In the example, if we start from
$s_{1}$, then we haven’t encountered any reward yet. Hence, following
$\pi_{1}$, we will have $r_{2}=0$ as the maximum reward half of the time. If
we start from $s_{0}$, we receive a reward of $r_{1}=6$ when going to $s_{1}$.
Then, the maximum reward will not be lower than 6, even if we get $r_{2}=0$.
Thus, we conclude:

_In max-reward RL, the optimal policy $\pi^{\ast}$ maximizing $\hat{J}(\cdot)$
should depend not only on the current state, but also on the rewards obtained
so far._

To formalize this observation, we introduce additional notation. We define the
extended state space as $\hat{\mathcal{S}}:=\mathcal{S}\times{\mathbb{R}}$ and
we denote extended states by
${\hat{s}=(s,y),\;s\in\mathcal{S},\;y\in{\mathbb{R}}.}$ Then, for an extended
state $(s,y)\in\hat{\mathcal{S}}$ and for an action $a\in\mathcal{A},$ the
extended transition model $\hat{P}(\cdot,\cdot|s,y,a)$ is a PDF over
${(s^{\prime},y^{\prime})\in\hat{\mathcal{S}}},$ defined as

$\hat{P}(s^{\prime},y^{\prime}|s,y,a)=P(s^{\prime}|s,a)\,\delta\Big{(}y^{\prime}-\frac{R(s,a,s^{\prime})\lor
y}{\gamma}\Big{)}$

where $\delta(\cdot)$ is the Dirac delta function. The initial distribution of
$(s_{0},y_{0})$ is given by $\hat{p}_{0}(s_{0},y_{0})=p(s_{0})\delta(y_{0})$
thereby ensuring $y_{0}\equiv 0$. Combining everything, we introduce the
following definition:

######  Definition 4.5.

Let $M=(\mathcal{S},\mathcal{A},R,P,p_{0},\gamma)$ be an MDP. Then, the
extended max-reward MDP is an MDP $\hat{M}$ given by the tuple
$(\hat{\mathcal{S}},\mathcal{A},R,\hat{P},\hat{p_{0}},\gamma).$

Now, we can define the notion of policy for max-reward RL:

######  Definition 4.6.

Let $M$ be an MDP and let $\hat{M}$ be its induced extended max-reward MDP.
Then, any policy $\hat{\pi}$ in $\hat{M}$ is an extended max-reward policy.

After we have defined optimality in the max-reward sense, we can introduce the
max-reward Bellman optimality operator:

######  Definition 4.7.

Let $q:\mathcal{S}\times\mathcal{A}\times\mathbb{R}\to\mathbb{R}$ be a real-
valued function and let $y^{\prime}:=\frac{R(s,a,s_{t+1})\lor y}{\gamma}$.
Then, the _max-reward Bellman optimality operator_ $\hat{\mathcal{T}}^{\ast}$
is defined as follows:

$\hat{\mathcal{T}}^{\ast}q(s,a,y):=\gamma{\mathbb{E}}_{{s_{t+1}}}\big{[}y^{\prime}\lor\max_{a^{\prime}}q(s_{t+1},a^{\prime},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$

Similarly to $\hat{T}^{\pi},$ this operator is also a contraction:

######  Theorem 4.8.

$\hat{\mathcal{T}}^{\ast}$ is a $\gamma-$contraction with respect to the
$L_{\infty}$ norm, and $\hat{q}^{\ast}$ is its fixed point.

We have most of the pieces of the max-reward RL framework. We established that
it operates on the extended max-reward MDP $\hat{M}$, where the extended
states preserve information about the past rewards. Then, both the max-reward
optimal and on-policy value functions can be learned by sampling transitions
from $\hat{M}$. Therefore, all DQN-based methods (Mnih et al., 2013) can be
used under the max-reward RL paradigm directly. However, most state-of-the-art
RL algorithms utilize policies parametrized by neural networks. This is
possible due to the policy gradient theorems (Sutton et al., 1999; Silver et
al., 2014), as they allow estimating the objective function gradient with
respect to the policy parameters via sampling. In the next section, we
formulate and prove max-reward policy gradient theorems for both deterministic
and stochastic extended max-reward policies.

###  4.2 Policy gradient theorems

1: Initialize critic networks $\hat{q}_{\phi_{1}},\hat{q}_{\phi_{2}}$ and
actor network $\mu_{\theta}$

2: Initialize target networks
$\phi_{1}^{\prime}\leftarrow\phi_{1},\,\phi_{2}^{\prime}\leftarrow\phi_{2},\,\theta^{\prime}\leftarrow\theta$

3: Initialize replay buffer $\mathcal{D}$

4: for $episode=1,2,\ldots$ do

5: Initialize $s_{0},y_{0}\sim\hat{p}_{0}$

6: for $t=0,1,\ldots,T-1$ do

7: Sample exploration noise $\epsilon_{t}$

8: Execute $a_{t}=\mu(s_{t},y_{t})+\epsilon_{t}$ and get $s_{t+1},r_{t+1}$

9: Update $y_{t+1}=(y_{t}\lor r_{t+1})/{\gamma}$

10: Save $(s_{t},y_{t},a_{t},s_{t+1},r_{t+1},y_{t+1})$ into $\mathcal{D}$

11: if initial exploration is over then

12: Sample a mini-batch of size $N$ from $\mathcal{D}$

13: Sample target actions noise $\eta$

14: $\tilde{a}\leftarrow\mu_{\theta^{\prime}}(s^{\prime},y^{\prime})+\eta$

15: $z\leftarrow
y^{\prime}\lor\gamma\min_{i=1,2}\hat{q}_{\phi^{\prime}_{i}}(s^{\prime},\tilde{a},y^{\prime})$

16: Critic loss
$L_{c}=\frac{1}{N}\sum^{2}_{i=1}(z-\hat{q}_{\phi_{i}}(s,a,y))^{2}$

17: Perform gradient update step on $L_{c}$

18: if time to update policy then

19:
$L_{a}\leftarrow{\mathbb{E}}\big{[}\hat{q}_{\phi_{1}}(s,\mu_{\theta}(s,y),y)\big{]}$

20: Perform gradient update step on $L_{a}$

21:
$\phi_{i}^{\prime}\leftarrow\tau\phi_{i}+(1-\tau)\phi^{\prime}_{i},\;i=1,2$

22: $\theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta$

23: end if

24: end if

25: end for

26: end for

Algorithm 1 Max-reward TD3 Algorithm 2 Max-reward PPO

1: Initialize actor $\hat{\pi}_{\theta}$ and critic $\hat{v}_{\phi}$

2: for $iteration=1,2,\ldots$ do

3: Initialize trajectories buffer $\mathcal{D}$

4: for $actor=1,2,\ldots,N$ do

5: Initialize $s_{0},y_{0}\sim\hat{p}_{0}$

6: for $t=0,1,\ldots,T-1$ do

7: Execute $a_{t}\sim\pi_{\theta}(\cdot|s_{t},y_{t})$ and get
$s_{t+1},r_{t+1}$

8: Update $y_{t+1}=(y_{t}\lor r_{t+1})/{\gamma}$

9: Save $(s_{t},y_{t},a_{t},s_{t+1},r_{t+1},y_{t+1})$ into $\mathcal{D}$

10: end for

11:
$\hat{G}_{t}^{n}\leftarrow\gamma^{n}\hat{v}_{\phi}(s_{t+n},y_{t+n}),\;n=1,\ldots,T-t$

12:
$\hat{G}_{t}(\lambda)=(1-\lambda)\sum_{n=1}^{T-t}\lambda^{n-1}\hat{G}^{n}_{t}$

13: Compute advantages
$\hat{A}_{t}=\hat{G}_{t}(\lambda)-\hat{v}_{\phi}(s_{t},y_{t})$

14: end for

15: for k=$1,2\ldots K$ do

16: Critic loss:
$L_{c}\leftarrow\frac{1}{T}\sum_{t}(\hat{G}_{t}^{T-t}-\hat{v}_{\phi}(s,y))^{2}$

17: Actor loss: $L_{a}\leftarrow
L_{PPO}(\pi_{\theta},\\{\hat{A}_{t}\\}_{t=1}^{T})$

18: Perform gradient update step on $L_{a}+L_{c}$

19: end for

20: end for

Figure 3: Learning curves of TD3, max-reward TD3, PPO, and max-reward PPO on
two different mazes. The vertical axis is the success ratio, i.e., whether the
goal was reached during the episode. The shaded area is the standard error of
the mean. The horizontal axis is the total environmental timesteps in
millions. For each maze, we present results for six different reward functions
(columns). Figure 4: Left: Single-goal maze, where the goal (red ball) is
always in the same location. Right: Two-goals maze with two spawn locations of
the goal (red balls). In each episode, the location for the goal is chosen
randomly.

First, we define $\hat{p}^{\hat{\pi}}_{t}(s_{0},y_{0},s,y)$ – the probability
measure of arriving in the extended state $(s,y)$ after $t$ timesteps,
starting from $(s_{0},y_{0})$ and executing the extended policy $\hat{\pi}$.
Let

$\hat{P}^{\hat{\pi}}(s^{\prime},y^{\prime}|s,y)=\int_{a}\hat{\pi}(a|s,y)\hat{P}(s^{\prime},y^{\prime}|s,y,a)da$

be the “on-policy” transition model. Then,
$\hat{p}^{\hat{\pi}}_{t}(s_{0},y_{0},s,y)$ is defined as follows:

$\hat{p}_{0}(s_{0},y_{0},s,y)=\delta(s-s_{0})\delta(y-y_{0})$
$\displaystyle\hat{p}^{\hat{\pi}}_{t}(s_{0},y_{0},s,y)=\mathop{\int}_{\tilde{s},\tilde{y}}$
$\displaystyle\hat{p}^{\hat{\pi}}_{t-1}(s_{0},y_{0},\tilde{s},\tilde{y})\,\hat{P}^{\hat{\pi}}(s,y|\tilde{s},\tilde{y})\,d\tilde{s}\,d\tilde{y}$

The discounted stationary state distribution of an extended max-reward MDP is
then given by

$\hat{d}^{\hat{\pi}}(s,y)=\\!\\!\\!\mathop{\int}_{s_{0},y_{0}}\\!\\!\\!\hat{p}_{0}(s_{0},y_{0})\sum_{t=0}^{\infty}\gamma^{t}\,\hat{p}^{\hat{\pi}}_{t}(s_{0},y_{0},s,y)\,ds_{0}\,dy_{0}.$

As such, $\hat{d}^{\hat{\pi}}$ is not a distribution. However, it can be
normalized into one by dividing it by
$C=\int_{s,y}\hat{d}^{\hat{\pi}}(s,y)\,ds\,dy$.

Finally, we can formulate and prove the max-reward policy gradient theorems.
Consider a neural network with weights $\theta$ that represents a stochastic
policy. Then, we have the following result:

######  Theorem 4.9.

Let
$\hat{\pi}_{\theta}:\mathcal{S}\times{\mathbb{R}}\to\mathcal{P}(\mathcal{A})$
be a stochastic extended max-reward policy parameterized with $\theta$. Then,
the following holds for $\nabla_{\theta}\hat{J}(\theta)$:

$\nabla_{\theta}\;\hat{J}(\theta)\propto{\mathbb{E}}_{\begin{subarray}{c}(s,y)\sim\hat{d}^{\hat{\pi}}\\\
a\sim\hat{\pi}_{\theta}\end{subarray}}\big{[}\hat{q}^{\hat{\pi}_{\theta}}(s,a,y)\nabla_{\theta}\ln\hat{\pi}_{\theta}(a|s,y)\big{]}$

The deterministic max-reward policy gradient follows from the stochastic
version:

######  Corollary 4.10.

Let $\hat{\mu}_{\theta}:\mathcal{S}\times{\mathbb{R}}\to\mathcal{A}$ be a
deterministic extended max-reward policy parameterized with $\theta$. Then
$\nabla_{\theta}\hat{J}(\theta)$ can be computed as follows:

$\nabla_{\theta}J(\theta)\propto{\mathbb{E}}_{\hat{d}^{\hat{\mu}}}\big{[}\nabla_{\theta}\hat{\mu}_{\theta}(s,y)\nabla_{a}\hat{q}^{\hat{\mu}_{\theta}}(s,a,y)|_{a=\mu_{\theta}(s,y)}\big{]}$

The policy gradient theorems allow us to use various algorithms from standard
RL, such as REINFORCE, A2C, A3C, TRPO, PPO, DDPG, and TD3 to optimize maximum
rewards. In this work, we focus on PPO and TD3, as they are considered to be
the best-performing algorithms from their corresponding families. In
Algorithms 1 and 2 we provide descriptions of max-reward TD3 and PPO in
pseudocode.

##  5 Experiments

To practically evaluate the benefits of using maximum instead of cumulative
reward, we compare the max-reward TD3 and PPO with their cumulative
counterparts using two goal-reaching environments from Gymnasium-Robotics (de
Lazcano et al., 2023) under different dense reward functions.

###  5.1 Maze with shortest path rewards

First, we consider the Maze environment from Gym Robotics (de Lazcano et al.,
2023) illustrated in Figure 4, where the agent controls a ball by applying
acceleration in two dimensions. The objective is to reach the goal position in
the maze. Each episode lasts for 1000 timesteps and there are no terminal
states. We use two different mazes: single-goal maze, where the goal is always
spawned in the same location, and the two-goals maze where at each episode the
goal location is chosen randomly from the two possible options. The main
performance metric in the Maze environment is whether the goal was reached
during the episode. We refer to this quantity as success ratio.

In the experiment, we consider several reward functions that induce the same
optimal policy of reaching the goal state:

  * 1.

Sparse reward – only reaching the goal is rewarded with $r=1.$

  * 2.

Dense $l_{2}$ reward – default dense reward, defined as the exponent of the
negative of the $l_{2}$-distance to the goal. Reaching the goal is rewarded
with $r=1.$

  * 3.

Discrete shortest path (DSP) – our custom reward that represents the “true”
distance to the goal. To compute it, the maze is split into $n\times m$ cells.
Then, the distance matrix $D\in{\mathbb{R}}^{n\times m}$ is computed such that
for each cell $(i,j)$, $D[i,j]$ is the number of cells between $(i,j)$ and the
goal-containing cell. The DSP reward with parameter $k\in\mathbb{N}$ is then
defined as

$r^{k}_{dsp}(i,j)=\begin{cases}\beta^{D[i,j]+1},\;\text{ if }D[i,j]=0\mod k\\\
0,\quad\quad\quad\;\,\text{ otherwise }\end{cases}$

where $\beta$ is a hyperparameter. The value of $k$ controls the sparsity of
the reward, i.e., for larger $k$ fewer cells have a non-zero reward. Reaching
the goal is rewarded with $r=1.$

For the DSP reward, we first tune the value of $\beta$ by running standard TD3
and PPO on the single-goal maze. We set $k=1$ and run 10 random seeds for each
algorithm for $\beta\in\\{0.65,0.7,0.75,0.8,0.85,0.9,0.95\\}$. Additionally,
we test the negative version of the DSP reward, $r^{k}_{dsp}(i,j)-1$, which,
in theory, should cause better exploration. For TD3, the best-performing
reward was the negative DSP with $\beta_{TD3}=0.9$, and for PPO – negative DSP
with $\beta_{PPO}=0.95.$ In all other runs involving DSP reward we use these
values of $\beta.$

Next, we run hyperparameters tuning for cumulative TD3 and PPO. Finally, we
compare TD3, PPO, max-reward TD3, and max-reward PPO on the single-goal and
two-goals mazes using sparse, dense $l_{2}$, and DSP reward for
$k=1,2,3,4$.333For cumulative methods, negative DSP reward is used.

Figure 3 demonstrates the learning curves. The sparse reward performs
inconsistently due to insufficient exploration. Dense $l_{2}$ reward has local
maximums (especially in the single-goal maze) and its performance greatly
depends on the maze topology. The DSP reward, which represents the true
distance to the goal, overall performs better.

Importantly, we see that the max-reward approaches work for all values of $k,$
while the standard RL methods do not. For larger $k$, the reward becomes
sparser, and cumulative approaches tend to converge to suboptimal policies. We
believe that the nature of this phenomenon is the same as in the chain
environment example discussed in Section 3.1. Specifically, the Maze
environment can be seen as a larger chain with multiple intermediate rewards.
All methods quickly learn to stay in one of the cells with non-zero reward.
Then, to update the policy, samples of transitions with a higher reward are
necessary. For larger $k,$ these transitions become less frequent, as the
cells with non-zero reward become further from one another. In line with the
chain environment results, max-reward methods require fewer such transitions
and therefore perform more efficiently.

###  5.2 Fetch environment

In the second experiment, we consider a more challenging robotics problem.
Specifically, we study the Fetch-Slide environment depicted in Figure 5. The
agent controls a 7-DoF manipulator and its goal is to move the puck into the
target location. Each episode is truncated after 100 timesteps and there are
no terminal states. We use the standard dense reward for this problem defined
as $l_{2}-$distance between the puck and the goal. The performance metric for
this environment is again the success ratio – a binary value that indicates
whether the goal was reached during the episode. This environment is known to
be challenging for standard RL and it cannot be solved without special
approaches (Plappert et al., 2018).

The plot in Figure 5 demonstrates that the max-reward TD3 achieves a goal-
reaching policy while the standard version fails to learn completely. We
believe that this result shows the great potential of max-reward RL in such
environments.

##  6 Conclusions and future work

In this work, we provide a theoretical description of the max-reward
reinforcement learning paradigm and verify it experimentally. Our theoretical
contributions include a novel formulation of the max-reward value functions
and a Bellman-like contraction operator that allows for efficient learning.
Besides, we prove the policy gradient theorems for max-reward policies and
hence enable using the state-of-the-art RL algorithms in the context of max-
reward RL.

Our experiments on two different robotic problems show that max-reward RL is a
significantly better choice for goal-reaching environments with dense rewards.
Moreover, max-reward RL is capable of solving the Fetch-Slide environment,
which is known to be challenging for standard RL methods. These results
demonstrate the great potential of max-reward RL.

In future work, we aim to study how max-reward RL can be combined with the
existing methods for automated reward design and explore its potential in
other goal-reaching problems.

Figure 5: Left: Illustration of the Fetch-Slide environment. The agent’s goal
is to push the puck (black cylinder) into the goal location (red sphere).
Right: Success ratio for standard (light red) and max-reward (dark-red) TD3.

## References

  * Bellman (1954) Bellman, R.  Some applications of the theory of dynamic programming—a review.  _Journal of the Operations Research Society of America_ , 2(3):275–288, 1954.
  * Berner et al. (2019) Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Burda et al. (2018) Burda, Y., Edwards, H., Storkey, A., and Klimov, O.  Exploration by random network distillation, 2018.
  * Cui & Yu (2023) Cui, W. and Yu, W.  Reinforcement learning with non-cumulative objective.  _IEEE Transactions on Machine Learning in Communications and Networking_ , 2023.
  * de Lazcano et al. (2023) de Lazcano, R., Andreas, K., Tai, J. J., Lee, S. R., and Terry, J.  Gymnasium robotics, 2023.  URL http://github.com/Farama-Foundation/Gymnasium-Robotics.
  * Eschmann (2021) Eschmann, J.  Reward function design in reinforcement learning.  _Reinforcement Learning Algorithms: Analysis and Applications_ , pp. 25–33, 2021.
  * Fisac et al. (2014) Fisac, J. F., Chen, M., Tomlin, C. J., and Sastry, S. S.  Reach-avoid problems with time-varying dynamics, targets and constraints, 2014.
  * Fisac et al. (2019) Fisac, J. F., Lugovoy, N. F., Rubies-Royo, V., Ghosh, S., and Tomlin, C. J.  Bridging Hamilton-Jacobi safety analysis and reinforcement learning.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pp. 8550–8556, May 2019.
  * Florensa et al. (2018) Florensa, C., Held, D., Geng, X., and Abbeel, P.  Automatic goal generation for reinforcement learning agents, 2018.
  * Ghosh et al. (2020) Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C., Eysenbach, B., and Levine, S.  Learning to reach goals via iterated supervised learning, 2020.
  * Gottipati et al. (2020) Gottipati, S. K., Pathak, Y., Nuttall, R., Sahir, Chunduru, R., Touati, A., Subramanian, S. G., Taylor, M. E., and Chandar, S.  Maximum reward formulation in reinforcement learning, 2020.
  * Haarnoja et al. (2023) Haarnoja, T., Moran, B., Lever, G., Huang, S. H., Tirumala, D., Wulfmeier, M., Humplik, J., Tunyasuvunakool, S., Siegel, N. Y., Hafner, R., Bloesch, M., Hartikainen, K., Byravan, A., Hasenclever, L., Tassa, Y., Sadeghi, F., Batchelor, N., Casarini, F., Saliceti, S., Game, C., Sreendra, N., Patel, K., Gwira, M., Huber, A., Hurley, N., Nori, F., Hadsell, R., and Heess, N.  Learning agile soccer skills for a bipedal robot with deep reinforcement learning, 2023.
  * Hsu et al. (2021) Hsu, K.-C., Rubies-Royo, V., Tomlin, C. J., and Fisac, J. F.  Safety and liveness guarantees through Reach-Avoid reinforcement learning.  December 2021.
  * Ibarz et al. (2018) Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D.  Reward learning from human preferences and demonstrations in atari, 2018.
  * Jensen (1906) Jensen, J. L. W. V.  Sur les fonctions convexes et les inégalités entre les valeurs moyennes.  _Acta mathematica_ , 30(1):175–193, 1906.
  * Krakovna et al. (2020) Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J., and Legg, S.  Specification gaming: the flip side of ai ingenuity.  2020\.
  * Kröner et al. (2018) Kröner, A., Picarelli, A., and Zidani, H.  Infinite horizon stochastic optimal control problems with running maximum cost.  _SIAM J. Control Optim._ , 56(5):3296–3319, January 2018.
  * Mataric (1994) Mataric, M. J.  Reward functions for accelerated learning.  In _Machine learning proceedings 1994_ , pp. 181–189. Elsevier, 1994.
  * Mnih et al. (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Ng et al. (1999) Ng, A. Y., Harada, D., and Russell, S.  Policy invariance under reward transformations: Theory and application to reward shaping.  In _Icml_ , volume 99, pp. 278–287. Citeseer, 1999.
  * Pathak et al. (2017) Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T.  Curiosity-driven exploration by self-supervised prediction, 2017.
  * Plappert et al. (2018) Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej, M., Welinder, P., et al.  Multi-goal reinforcement learning: Challenging robotics environments and request for research.  _arXiv preprint arXiv:1802.09464_ , 2018.
  * Quah & Quek (2006) Quah, K. and Quek, C.  Maximum reward reinforcement learning: A non-cumulative reward criterion.  _Expert Systems with Applications_ , 31(2):351–359, 2006.  ISSN 0957-4174.  doi: https://doi.org/10.1016/j.eswa.2005.09.054.  URL https://www.sciencedirect.com/science/article/pii/S0957417405002228.
  * Silver et al. (2014) Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M.  Deterministic policy gradient algorithms.  In _International conference on machine learning_ , pp. 387–395. Pmlr, 2014.
  * Sutton et al. (1999) Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y.  Policy gradient methods for reinforcement learning with function approximation.  _Advances in neural information processing systems_ , 12, 1999.
  * Tang et al. (2017) Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., and Abbeel, P.  Exploration: A study of count-based exploration for deep reinforcement learning, 2017.
  * Towers et al. (2023) Towers, M., Terry, J. K., Kwiatkowski, A., Balis, J. U., Cola, G. d., Deleu, T., Goulão, M., Kallinteris, A., KG, A., Krimmel, M., Perez-Vicente, R., Pierré, A., Schulhoff, S., Tai, J. J., Shen, A. T. J., and Younis, O. G.  Gymnasium, March 2023.  URL https://zenodo.org/record/8127025.
  * Trott et al. (2019) Trott, A., Zheng, S., Xiong, C., and Socher, R.  Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards, 2019.
  * Vinyals et al. (2019) Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019.
  * Wang et al. (2020) Wang, R., Zhong, P., Du, S. S., Salakhutdinov, R. R., and Yang, L.  Planning with general objective functions: Going beyond total rewards.  _Advances in Neural Information Processing Systems_ , 33:14486–14497, 2020.
  * Yu et al. (2022) Yu, D., Ma, H., Li, S. E., and Chen, J.  Reachability constrained reinforcement learning.  May 2022.

##  Appendix A Proofs

###### Proof of Lemma 4.2.

First, we prove the equation for the state value function $\hat{v}^{\pi}:$

$\displaystyle\hat{v}^{\pi}(s,y)$
$\displaystyle={\mathbb{E}}_{\pi}\big{[}y\lor\hat{G}_{t}\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\big{]}={\mathbb{E}}_{\pi}\big{[}y\lor
r_{t+1}\lor\gamma\hat{G}_{t+1}\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\big{]}$
$\displaystyle=\gamma{\mathbb{E}}_{\pi}\big{[}y^{\prime}\lor\hat{G}_{t+1}\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\big{]}=\gamma{\mathbb{E}}_{\pi}\big{[}y^{\prime}\lor
y^{\prime}\lor\hat{G}_{t+1}\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\big{]}$
$\displaystyle=\gamma{\mathbb{E}}_{\begin{subarray}{c}a_{t}\\\
s_{t+1}\end{subarray}}\big{[}y^{\prime}\lor\hat{v}^{\pi}(s_{t+1},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\big{]}$

Then, for the state-action value function $\hat{q}^{\pi}:$

$\displaystyle\hat{q}^{\pi}(s,a,y)$
$\displaystyle={\mathbb{E}}_{\pi}\big{[}y\lor\hat{G}_{t}\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}={\mathbb{E}}_{\pi}\big{[}y\lor
r_{t+1}\lor\gamma\hat{G}_{t+1}\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$
$\displaystyle=\gamma{\mathbb{E}}_{\pi}\big{[}y^{\prime}\lor\hat{G}_{t+1}\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}=\gamma{\mathbb{E}}_{\pi}\big{[}y^{\prime}\lor
y^{\prime}\lor\hat{G}_{t+1}\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$
$\displaystyle=\gamma{\mathbb{E}}_{\begin{subarray}{c}s_{t+1}\\\
a_{t+1}\end{subarray}}\big{[}y^{\prime}\lor\hat{q}^{\pi}(s_{t+1}a_{t+1},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$

∎

###### Proof of Theorem 4.4.

First, we demonstrate a simple property of the $\lor$ operator that we will
use later. Let $a,x,y\in{\mathbb{R}}.$ Then, using equation $x\lor
y=0.5(x+y+|x-y|)$, we obtain the following:

$\displaystyle a\lor x-a\lor y$
$\displaystyle=0.5\big{(}a+x+|x-a|-a-y-|y-a|\big{)}$ (9)
$\displaystyle=0.5(x-y+|x-a|-|y-a|)$ $\displaystyle\leq 0.5(x-y+|x-a-(y-a)|)$
$\displaystyle=0.5(x-y+|x-y|)$ $\displaystyle\leq|x-y|$

Now, we can prove that $\hat{\mathcal{T}}^{\pi}$ is a contraction. We begin
with the state-action case. Let
$q,z:\mathcal{S}\times\mathcal{A}\times{\mathbb{R}}$ be two-real valued
functions. Then, we can expand
$\|{\hat{\mathcal{T}}}^{\pi}q-{\hat{\mathcal{T}}}^{\pi}z\|_{\infty}$ as
follows:

$\displaystyle\|{\hat{\mathcal{T}}}^{\pi}q-{\hat{\mathcal{T}}}^{\pi}z\|_{\infty}$
$\displaystyle=\gamma\sup_{s\in\mathcal{S},a\in\mathcal{A},y\in{\mathbb{R}}}\Big{|}{\mathbb{E}}_{\begin{subarray}{c}s_{t+1}\\\
a_{t+1}\end{subarray}}\Big{[}y^{\prime}\lor
q(s_{t+1},a_{t+1},y^{\prime})-y^{\prime}\lor
z(s_{t+1},a_{t+1},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\Big{]}\Big{|}$
$\displaystyle\leq\gamma\sup_{s\in\mathcal{S},a\in\mathcal{A},y\in{\mathbb{R}}}{\mathbb{E}}_{\begin{subarray}{c}s_{t+1}\\\
a_{t+1}\end{subarray}}\Big{[}\Big{|}y^{\prime}\lor
q(s_{t+1},a_{t+1},y^{\prime})-y^{\prime}\lor
z(s_{t+1},a_{t+1},y^{\prime})\Big{|}\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\Big{]}|$
$\displaystyle\overset{(\ast)}{\leq}\gamma\sup_{s_{t+1}\in\mathcal{S},a_{t+1}\in\mathcal{A},y^{\prime}\in{\mathbb{R}}}\Big{|}y^{\prime}\lor
q(s_{t+1},a_{t+1},y^{\prime})-y^{\prime}\lor
z(s_{t+1},a_{t+1},y^{\prime})\Big{|}$
$\displaystyle=\gamma\sup_{s\in\mathcal{S},a\in\mathcal{A},y\in{\mathbb{R}}}\Big{|}y\lor
q(s,a,y)-y\lor z(s,a,y)\Big{|}$
$\displaystyle\leq\gamma\sup_{s\in\mathcal{S},a\in\mathcal{A},y\in{\mathbb{R}}}\Big{|}q(s,a,y)-z(s,a,y)\Big{|}=\|q-z\|_{\infty}$

The first inequality follows from the fact that
$\Big{|}{\mathbb{E}}_{x}[x]\Big{|}\leq{\mathbb{E}}_{x}\Big{[}|x|\Big{]}$ and
the last inequality follows from Eq. (9). In $(\ast)$, we use the following
property of the expectation:
$\sup_{y}\big{\\{}{\mathbb{E}}[x|y]\big{\\}}\leq\sup_{x}\\{x\\}.$ Now, we
demonstrate the contraction property for the state value function: Let
$v,u:\mathcal{S}\times{\mathbb{R}}$ be two-real valued functions. Then, we
expand $\|{\hat{\mathcal{T}}}^{\pi}v-{\hat{\mathcal{T}}}^{\pi}u\|_{\infty}$ as
follows:

$\displaystyle\|{\hat{\mathcal{T}}}^{\pi}v-{\hat{\mathcal{T}}}^{\pi}u\|_{\infty}$
$\displaystyle=\gamma\sup_{s\in\mathcal{S},y\in{\mathbb{R}}}\Big{|}{\mathbb{E}}_{\begin{subarray}{c}a_{t}\\\
s_{t+1}\end{subarray}}\Big{[}y^{\prime}\lor
v(s_{t+1},y^{\prime})-y^{\prime}\lor
u(s_{t+1},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\Big{]}\Big{|}$
$\displaystyle\leq\gamma\sup_{s\in\mathcal{S},y\in{\mathbb{R}}}{\mathbb{E}}_{\begin{subarray}{c}a_{t}\\\
s_{t+1}\end{subarray}}\Big{[}\Big{|}y^{\prime}\lor
v(s_{t+1},y^{\prime})-y^{\prime}\lor
u(s_{t+1},y^{\prime})\Big{|}\big{|}\begin{subarray}{c}s_{t}=s\\\
\end{subarray}\Big{]}$
$\displaystyle\leq\gamma\sup_{s_{t+1}\in\mathcal{S},y^{\prime}\in{\mathbb{R}}}\Big{|}y^{\prime}\lor
v(s_{t+1},y^{\prime})-y^{\prime}\lor u(s_{t+1},y^{\prime})\Big{|}$
$\displaystyle=\gamma\sup_{s\in\mathcal{S},y\in{\mathbb{R}}}\Big{|}y\lor
v(s,y)-y\lor u(s,y)\Big{|}$
$\displaystyle\leq\gamma\sup_{s\in\mathcal{S},y\in{\mathbb{R}}}\Big{|}v(s,y)-u(s,y)\Big{|}=\|v-u\|_{\infty}$

Therefore, the max-reward Bellman operator is a contraction. Hence, by the
Banach fixed-point theorem, it has a unique fixed-point(s). From Lemma 4.2, we
conclude that this is the max-reward value function(s). ∎

###### Proof of Lemma 4.8.

$\displaystyle\|{\hat{\mathcal{T}}}^{\ast}q-{\hat{\mathcal{T}}}^{\ast}z\|_{\infty}$
$\displaystyle=\gamma\sup_{s\in\mathcal{S},a\in\mathcal{A},y\in{\mathbb{R}}}\Big{|}{\mathbb{E}}_{\begin{subarray}{c}s_{t+1}\\\
\end{subarray}}\Big{[}y^{\prime}\lor\max_{a^{\prime}}q(s_{t+1},a^{\prime},y^{\prime})-y^{\prime}\lor\max_{a^{\prime}}z(s_{t+1},a^{\prime},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\Big{]}\Big{|}$ (10)
$\displaystyle\leq\gamma\sup_{s\in\mathcal{S},a\in\mathcal{A},y\in{\mathbb{R}}}{\mathbb{E}}_{\begin{subarray}{c}s_{t+1}\\\
\end{subarray}}\Big{[}\Big{|}y^{\prime}\lor\max_{a^{\prime}}q(s_{t+1},a^{\prime},y^{\prime})-y^{\prime}\lor\max_{a^{\prime}}z(s_{t+1},a^{\prime},y^{\prime})\Big{|}\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\Big{]}$
$\displaystyle\leq\gamma\sup_{s\in\mathcal{S},a\in\mathcal{A},y\in{\mathbb{R}}}{\mathbb{E}}_{\begin{subarray}{c}s_{t+1}\\\
\end{subarray}}\Big{[}\max_{a^{\prime}}\Big{|}y^{\prime}\lor
q(s_{t+1},a^{\prime},y^{\prime})-y^{\prime}\lor
z(s_{t+1},a^{\prime},y^{\prime})\Big{|}\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\Big{]}$
$\displaystyle\leq\gamma\sup_{s_{t+1}\in\mathcal{S},a_{t+1}\in\mathcal{A},y^{\prime}\in{\mathbb{R}}}\Big{|}y^{\prime}\lor
q(s_{t+1},a_{t+1},y^{\prime})-y^{\prime}\lor
z(s_{t+1},a_{t+1},y^{\prime})\Big{|}$
$\displaystyle=\gamma\sup_{s\in\mathcal{S},a\in\mathcal{A},y\in{\mathbb{R}}}\Big{|}y\lor
q(s,a,y)-y\lor z(s,a,y)\Big{|}$
$\displaystyle\leq\gamma\sup_{s\in\mathcal{S},a\in\mathcal{A},y\in{\mathbb{R}}}\Big{|}q(s,a,y)-z(s,a,y)\Big{|}=\|q-z\|_{\infty}$

∎

###### Proof of Theorem 4.9.

First of all, we notice that the max-reward Bellman equation implies another
recursive equation for $\hat{v}^{\pi}$ and $\hat{q}^{\pi}:$

$\hat{q}^{\pi}(s,a,y)=\gamma{\mathbb{E}}_{\begin{subarray}{c}s_{t+1}\end{subarray}}\big{[}y^{\prime}\lor\hat{v}^{\pi}(s_{t+1},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\end{subarray}\big{]}=\gamma{\mathbb{E}}_{\begin{subarray}{c}a_{t}\\\
s_{t+1}\end{subarray}}\big{[}\hat{v}^{\pi}(s_{t+1},y^{\prime})\big{|}\begin{subarray}{c}s_{t}=s\\\
a_{t}=a\end{subarray}\big{]}$ (11)

As discussed in the main paper, we use the version with extra $\lor
y^{\prime}$ to enforce boundary conditions. However, we can still use the
equation above as it is a property of the max-reward value function.

Before proving the theorem, we introduce simplified notation to improve
readability – for all functions, we use subscripts to denote the input
variables. For example, $\hat{v}_{t}:=\hat{v}^{\hat{\pi}}(s_{t},y_{t})$. The
proof follows the one for the standard policy gradient theorem. We begin by
obtaining a recurrent equation for $\nabla_{\theta}\hat{v}_{0}:$

$\displaystyle\nabla_{\theta}\hat{v}_{0}$
$\displaystyle=\nabla_{\theta}\big{(}\int_{a_{0}}\hat{\pi}_{0}\hat{q}_{0}da_{0}\big{)}=\underbrace{\int_{a_{0}}(\nabla_{\theta}\hat{\pi}_{0})\hat{q}_{0}da_{0}}_{\phi_{0}}+\int_{a_{0}}\hat{\pi}_{0}(\nabla_{\theta}q)_{0}da_{0}$
$\displaystyle\overset{\text{Eq.\eqref{eq:simplified-
bellman}}}{=}\phi_{0}+\gamma\int_{a_{0}}\hat{\pi}_{0}\Big{(}\nabla_{\theta}\int_{s_{1},y_{1}}\hat{p}(s_{1},y_{1}|s_{0},y_{0},a_{0})\hat{v}_{1}ds_{1}dy_{1}\Big{)}da_{0}$
$\displaystyle=\phi_{0}+\gamma\int_{s_{1},y_{1}}\int_{a_{0}}\hat{\pi}_{0}\hat{p}(s_{1},y_{1}|s_{0},y_{0},a_{0})(\nabla_{\theta}\hat{v}_{1})ds_{1}dy_{1}da_{0}$
$\displaystyle=\phi_{0}+\gamma\int_{s_{1},y_{1}}\hat{p}_{1}^{\hat{\pi}}(s_{0},y_{0},s_{1},y_{1})(\nabla_{\theta}\hat{v}_{1})ds_{1}dy_{1}\,,$

where we introduced the shorthand
$\phi_{t}=\phi(s_{t},y_{t})=\int_{a}\nabla_{\theta}\hat{\pi}(a|s_{t},y_{t})\,q(s_{t},a,y_{t})\,da$.
Expanding this recurrence further, we obtain

$\displaystyle\nabla_{\theta}\hat{v}_{0}$
$\displaystyle=\sum_{t=0}^{\infty}\int_{s_{t},y_{t}}\gamma^{t}\hat{p}_{t}^{\hat{\pi}}(s_{0},y_{0},s_{t},y_{t})\phi(s_{t},y_{t})ds_{t}dy_{t}$
$\displaystyle=\int_{s,y}\Big{(}\sum_{t=0}^{\infty}\gamma^{t}\hat{p}_{t}^{\hat{\pi}}(s_{0},y_{0},s,y)\Big{)}\phi(s,y)dsdy$
$\displaystyle\propto\int_{s,y}\hat{d}^{\hat{\pi}}(s,y|s_{0},y_{0})\phi(s,y)dsdy$
$\displaystyle=\int_{s,y}\hat{d}^{\hat{\pi}}(s,y|s_{0},y_{0})\Big{(}\int_{a}\big{(}\nabla_{\theta}\hat{\pi}(a|s,y)\big{)}\hat{q}(s,a,y)da\Big{)}dsdy$

Above, $\hat{d}^{\hat{\pi}}(s,y|s_{0},y_{0})$ is the discounted stationary
distribution of $s,y$ for policy $\pi$ given $s_{0},y_{0}$. Finally, we
substitute this formula for $\nabla_{\theta}\hat{v}_{0}$ into the definition
of $\hat{J}(\theta)$ and conclude the proof:

$\displaystyle\nabla_{\theta}\hat{J}(\theta)$
$\displaystyle=\int_{s_{0},y_{0}}\hat{p}_{0}(s_{0},y_{0})\int_{s,y}\hat{d}^{\hat{\pi}}(s,y|s_{0},y_{0})\Big{(}\int_{a}\big{(}\nabla_{\theta}\hat{\pi}(a|s,y)\big{)}\hat{q}(s,a,y)da\Big{)}dsdyds_{0}dy_{0}$
(12)
$\displaystyle=\int_{s,y}\hat{d}^{\hat{\pi}}(s,y)\Big{(}\int_{a}\big{(}\nabla_{\theta}\hat{\pi}(a|s,y)\big{)}\hat{q}(s,a,y)da\Big{)}dsdy$
$\displaystyle={\mathbb{E}}_{\begin{subarray}{c}s,y\sim\hat{d}^{\hat{\pi}}\\\
a\sim\hat{\pi}(\cdot|s,y)\end{subarray}}\big{[}\hat{q}^{\hat{\pi}}(s,a,y)\nabla_{\theta}\ln\hat{\pi}(a|s,y)\big{]}$

∎

##  Appendix B Experimental details

For all experiments, we used our implementation of TD3 and PPO that we
verified on several MuJoco domains. The implementation of the max-reward
algorithms is similar to their cumulative versions except for the following
differences:

  * 1.

The input layer of all neural networks has an extra dimension to work with the
extended states $(s,y).$

  * 2.

The output layer of the value networks uses Tanh activation and is rescaled to
$u\in[0,\bar{R}]$. Then, it is transformed with $ReLU(u-y)+y$ to enforce
$\hat{v}^{\pi}(s,y)\geq y.$

Hyperparameters of all runs are reported in Tables 1-2.

Parameter |  PPO |  PPOMax |  TD3 |  TD3Max
---|---|---|---|---
Parallel environments |  16 |  16 |  16 |  16
Discount factor $\gamma$ |  0.99 |  0.999 |  0.99 |  0.995
Learning rate |  3e-4 |  3e-4 |  3e-4 |  3e-4
Lr. annealing |  No |  No |  No |  No
Entropy weight |  5e-2 |  5e-2 |  |
Value loss weight |  0.5 |  0.5 |  |
Clip coef. |  0.2 |  0.2 |  |
GAE $\lambda$ |  0.95 |  1 |  |
Policy update freq. |  |  |  2 |  2
Target soft update $\tau$ |  |  |  0.005 |  0.005
Expl. noise type |  |  |  pink |  pink
Expl. noise std |  |  |  0.7 |  0.7
Expl. noise clip |  |  |  0.5 |  0.5
Target noise scale |  |  |  0.2 |  0.2
Initial expl. steps |  |  |  25000 |  25000
Tr. epochs per rollout |  10 |  10 |  |
Rollout length |  1024 |  2048 |  |
Minibatch size |  32 |  32 |  256 |  256
Table 1: Hyperparameters for the experiments with Maze environment. Parameter |  TD3 |  TD3Max
---|---|---
Parallel environments |  16 |  16
Discount factor $\gamma$ |  0.99 |  0.995
Learning rate |  3e-4 |  3e-4
Lr. annealing |  No |  No
Policy update freq. |  2 |  2
Target soft update $\tau$ |  0.005 |  0.005
Expl. noise type |  pink |  pink
Expl. noise std |  0.1 |  0.1
Expl. noise clip |  0.5 |  0.5
Target noise scale |  0.2 |  0.2
Initial expl. steps |  25000 |  25000
Minibatch size |  256 |  256
Table 2: Hyperparameters for the experiments with Fetch environment.