# The possibility of $X(4014)$ as a $D^{\ast}\bar{D}^{\ast}$ molecular state

Man-Yu Duan1 Dian-Yong Chen1,3<EMAIL_ADDRESS>En Wang2<EMAIL_ADDRESS>1School of Physics, Southeast University, Nanjing 210094, China
2School of Physics and Microelectronics, Zhengzhou University, Zhengzhou,
Henan 450001, China
3Lanzhou Center for Theoretical Physics, Lanzhou University, Lanzhou 730000,
P. R. China

###### Abstract

Within the framework of the local hidden gauge approach, we have studied the
near-threshold interaction of the $D^{*}\bar{D}^{*}$ channel with quantum
numbers $I(J^{PC})=0(0^{++})$, $0(2^{++})$, $1(0^{++})$, and $1(2^{++})$,
respectively. The contact interaction is taken into account alone, since it is
expected to give the dominant contribution near the threshold. One pole, which
is found in the case of the quantum numbers $I(J^{PC})=0(0^{++})$, could be
associated to the $X(4014)$ recently observed by the Belle Collaboration.
Thus, we suggest that the $X(4014)$ may be an $D^{*}\bar{D}^{*}$ molecular
state with $I(J^{PC})=0(0^{++})$, and more precise information about
$X(4014)$, such as resonance parameters and decay properties, could be useful
to shed light on its structure.

##  I INTRODUCTION

In last two decades, a large amount of the charmonium-like states, named $XYZ$
states, have been observed experimentally Brambilla:2019esw ; Cleven:2015era ;
Ma:2014zva ; Liu:2019zoy ; Hosaka:2016pey . Many kinds of theoretical
interpretations to their structures have been proposed, such as hadro-
quarkonia Voloshin:2007dx ; Alberti:2016dru , tetraquarks Maiani:2014aja ;
Maiani:2013nmn , hadronic molecules Fleming:2007rp ; Mehen:2011ds ;
Wang:2013kra ; Gamermann:2009fv ; Wu:2021udi ; Wu:2021ezz , kinematic effect
Ikeno:2020mra ; Nakamura:2021dix ; Chen:2011pv ; Chen:2013wca , and the mixing
of different components. Since most of the $XYZ$ states appear at certain
hadronic threshold, hadronic molecule is one of the most promising
interpretations among the various exotic states, although there are still many
controversies. For instance, the hidden-charm state $X(3872)$ is quite close
to the $D\bar{D}^{*}$ threshold Belle:2008fma ; BaBar:2007cmo , $Z_{cs}(3985)$
is close to the $\bar{D}_{s}D^{*}/\bar{D}^{*}_{s}D$ threshold BESIII:2020qkh .
Recently, a $T_{cc}^{+}$ state reported by the LHCb Collaboration has the mass
very close to the $D^{*+}D^{0}$ threshold LHCb:2021vvq ; LHCb:2021auc , and
could well be interpreted as a $D^{*}D$ molecular state Ling:2021bir ;
Feijoo:2021ppq ; Ren:2021dsi ; Chen:2021vhg ; Du:2021zzh . In addition, one
loosely $D\bar{D}$ bound state with $I(J^{PC})=0(0^{++})$ predicted in Ref.
Gamermann:2006nm  is supported by the Belle and BABAR measurements of
$\gamma\gamma\to D\bar{D}$ and $e^{+}e^{-}\to J/\psi D\bar{D}$ Wang:2020elp ;
Wang:2019evy , and the $Z_{c}(4025)$ observed by the BESIII Collaboration
could be interpreted as the $D^{*}\bar{D}^{*}$ molecule with
$I(J^{P})=1(1^{+})$ Duan:2021pll . As pointed out in Ref. Dong:2020hxe , the
molecular structure should appear at any threshold of a pair of heavy-quark
and heavy-antiquark hadrons with attractive interaction at threshold, and
experimental information about such structures is crucial to deepen our
understanding of the interactions between heavy hadrons and the internal
structures of the exotic states.

In 2021, the Belle Collaboration has analyzed the two-photon process
$\gamma\gamma\to\gamma\psi(2S)$ from the threshold to $4.2$ GeV for the first
time Belle:2021nuv , and found two structures in the lineshape of the cross
section. The first one with a local significance of $3.1\sigma$ corresponds to
one resonance with the mass and width as,

$\displaystyle M_{1}$ $\displaystyle=$ $\displaystyle 3922.4\pm 6.5\pm
2.0~{}{\rm MeV},$ $\displaystyle\Gamma_{1}$ $\displaystyle=$ $\displaystyle
22\pm 17\pm 4~{}{\rm MeV},$ (1)

which may be the $X(3915)$, $\chi_{c2}(3930)$, or the mixing of them, and the
second one with a local significance of $2.8\sigma$ corresponds to one new
resonance $X(4014)$ with the mass and width as,

$\displaystyle M_{2}$ $\displaystyle=$ $\displaystyle 4014.3\pm 4.0\pm
1.5~{}{\rm MeV},$ $\displaystyle\Gamma_{2}$ $\displaystyle=$ $\displaystyle
4\pm 11\pm 6~{}{\rm MeV}.$ (2)

As discussed in Ref. Belle:2021nuv , the $X(4014)$ state has a mass in
agreement with the predicted mass 4012 MeV for the $J^{PC}=2^{++}$
$D^{*}\bar{D}^{*}$ molecule with the assumption of $X(3915)$ as
$J^{PC}=0^{++}$ $D^{*}\bar{D}^{*}$ molecule Guo:2013sya ; Nieves:2012tt .
However, the binding energy of $D^{*}\bar{D}^{*}$ is of the order of $100$ MeV
if one considers the $X(3915)$ as a $D^{\ast}\bar{D}$ molecular state, which
is much larger than the ones of $X(3872)$, $Z_{cs}(3985)$, $T^{+}_{cc}$, and
$Z_{c}(4025)$. It should also be pointed out that the molecular explanation of
$X(3915)$ is still in debate. For instance, in Refs. Liu:2009fe ; Duan:2020tsx
; Kher:2018wtv  the $X(3915)$ and $\chi_{c2}(3930)$ could be assigned as the
charmonia $\chi_{c0}(2P)$ and $\chi_{c2}(2P)$, respectively, and in Ref.
Li:2015iga  $X(3915)$ is suggested as an $S$-wave $D_{s}^{+}{D}_{s}^{-}$
molecular state. Thus, it implies that the nature of the newly reported
$X(4014)$ state is still unclear, and the study on the $D^{*}\bar{D}^{*}$
interaction is crucial to explore its possible internal structure.

There exist many studies about the $D^{*}\bar{D}^{*}$ interaction in
literatures Dong:2021juy ; Molina:2009ct . In our previous work, we have
analysed the BESIII measurement of
$e^{+}e^{-}\to(D^{*}\bar{D}^{*})^{\pm,0}\pi^{\mp,0}$ reactions by taking into
account the vector-vector $D^{*}\bar{D}^{*}$, $K^{*}\bar{K}^{*}$, and
$\rho\rho$ interactions within the framework of local hidden gauge formalism,
and conclude that the $Z_{c}(4025)$ structure could be a shallow
$D^{*}\bar{D}^{*}$ bound state Duan:2021pll . Recently, the interaction
between charmed hadrons have been described by constant contact terms, and 229
molecular states were predicted Dong:2021juy . In Ref. Dong:2021juy , the
masses of the $D^{*}\bar{D}^{*}$ bound states with
$I(J^{PC})=0(0^{++})$/$0(1^{+-})$/$0(2^{++})$ are $1.82\sim 36.6$ MeV lower
than the $D^{*}\bar{D}^{*}$ threshold, which favors the $D^{*}\bar{D}^{*}$
molecular explanation for the $X(4014)$ state. In the present work, we will
investigate the S -wave near-threshold $D^{*}\bar{D}^{*}$ interaction within
an alternative framework, i.e. the local hidden gauge approach, and the vector
exchanges in the crossed channels in the hidden gauge model provide further
insights into the contact interactions explored in Ref. Dong:2021juy . We also
discuss the possibility of the X(4014) as the $D^{*}\bar{D}^{*}$ molecular
state and its favored quantum numbers.

This paper is organized as follows. We will show the formalism in Sec. II, and
present the calculated results and related discussions in Sec. III. Finally,
Sec. IV will be devoted to a short summary.

Figure 1: The sketch diagrams for $D^{\ast}\bar{D}^{\ast}\to
D^{\ast}\bar{D}^{\ast}$. Diagrams (a) and (b) correspond to vector meson
exchanged interaction and contact interaction, respectively, where the
exchanged vector mesons can be $\rho,\ \omega$, and $J/\psi$.

##  II FORMALISM

In this section, we investigate the $D^{*}\bar{D}^{*}$ interaction following
the approach of Ref. Molina:2009ct . The Lagrangian is taken from the local
hidden gauge formalism describing the interaction of vector mesons
Bando:1987br ; Nagahiro:2008cv ,

$\mathcal{L}=-\frac{1}{4}\langle V_{\mu\nu}V^{\mu\nu}\rangle\ ,$ (3)

where the symbol $\langle~{}\rangle$ stands for the trace of SU(4), and the
tensor $V_{\mu\nu}$ is defined as

$V_{\mu\nu}=\partial_{\mu}V_{\nu}-\partial_{\nu}V_{\mu}-ig[V_{\mu},V_{\nu}]\ $
(4)

with $V_{\mu}$ to be

$V_{\mu}=\left(\begin{array}[]{cccc}\frac{\omega}{\sqrt{2}}+\frac{\rho^{0}}{\sqrt{2}}&\rho^{+}&K^{*+}&\bar{D}^{*0}\\\
\rho^{-}&\frac{\omega}{\sqrt{2}}-\frac{\rho^{0}}{\sqrt{2}}&K^{*0}&D^{*-}\\\
K^{*-}&\bar{K}^{*0}&\phi&D^{*-}_{s}\\\
D^{*0}&D^{*+}&D^{*+}_{s}&J/\psi\end{array}\right)_{\mu}\ .$ (5)

By expanding the effective Lagrangian in Eq. (3), one can obtain two types of
$D^{\ast}\bar{D}^{\ast}$ interactions, which are vector-meson exchange
interactions [Fig. 1-(a)] and contact interaction [Fig. 1-(b)], respectively.
As for the vector meson exchange interaction, the general form of the
amplitude corresponding to diagram Fig. 1-(a) reads,

$\displaystyle
i\mathcal{M}=ig_{1}g_{2}\epsilon_{1\mu}\epsilon_{2\nu}\Gamma^{\mu\alpha}\frac{-i(-g^{\nu\beta}+q^{\nu}q^{\beta}/m_{V}^{2})}{q^{2}-m_{V}^{2}+i\epsilon}\epsilon_{3\alpha}^{\ast}\epsilon_{4\beta}^{\ast}\
.$ (6)

where $q$ is the momentum of the exchanged vector meson, and $\Gamma^{\mu\nu}$
is a tensor function of $g^{\mu\nu}$ and $p_{i}^{\mu}p_{j}^{\nu}$. As shown in
Eq. (2), the mass of $X(4014)$ is very close to the threshold of
$D^{\ast}\bar{D}^{\ast}$ and the width is very small. Thus, when we discuss
the possibility of $X(4014)$ as a $D^{\ast}\bar{D}^{\ast}$ molecular state, we
should merely consider the energy range around the threshold. In this case,
the momenta of the involved mesons are very limited, i.e., $q^{2}\sim 0$. Then
the propagator of the exchanged meson can be approximated by,

$\displaystyle\frac{(-g^{\nu\beta}+q^{\nu}q^{\beta}/m_{V}^{2})}{q^{2}-m_{V}^{2}+i\epsilon}\simeq\frac{g^{\nu\beta}}{m_{V}^{2}}$
(7)

where the terms suppressed by $\mathcal{O}(q^{2}/m_{V}^{2})$ have been
neglected. With the above approximation, the amplitudes of the meson exchanged
diagrams are in the same form as the one of contact diagram. Thus, in the
following, we can only consider the contact interaction in the present work.

Moreover, the decay width of the molecular states are related to the
contributions of the coupled channels in the present theoretical models
Zyla:2020zbs . The uncertainty of the widths reported by Belle Collaborations
is very large, while the mass is relatively accurate. Thus as the first step,
we can mainly focus on the mass of $X(4014)$, which justifies including the
$D^{*}\bar{D}^{*}$ single channel in the present work. Our understanding of
molecular state will greatly benefit from a systematic spectrum of hadronic
molecules based on the one single channel Dong:2021juy .

To depict the contact diagram, one can expand the Lagrangian in Eq. (3) and
the effective Lagrangian for the contact term reads,

$\mathcal{L}^{(c)}=\frac{g^{2}}{2}\langle
V_{\mu}V_{\nu}V^{\mu}V^{\nu}-V_{\nu}V_{\mu}V^{\mu}V^{\nu}\rangle\ .$ (8)

With the above effective Lagrangian, one can obtain the amplitude
corresponding to Fig. 1-(b), which is,

$\displaystyle t^{(c)}_{D^{*+}D^{*-}\to D^{*+}D^{*-}}$ $\displaystyle=$
$\displaystyle
2g^{2}(\epsilon^{(1)}_{\mu}\epsilon^{(2)\,\mu}\epsilon^{\ast(3)}_{\nu}\epsilon^{\ast(4)\,\nu}+\epsilon^{(1)}_{\mu}\epsilon^{(2)}_{\nu}\epsilon^{\ast(3)\,\mu}\epsilon^{\ast(4)\,\nu}$
$\displaystyle-2\epsilon^{(1)}_{\mu}\epsilon^{(2)}_{\nu}\epsilon^{\ast(3)\,\nu}\epsilon^{\ast(4)\,\mu})\
,$ $\displaystyle t^{(c)}_{D^{*+}{D}^{*-}\to D^{*0}\bar{D}^{*0}}$
$\displaystyle=$ $\displaystyle
g^{2}(\epsilon^{(1)}_{\mu}\epsilon^{(2)\,\mu}\epsilon^{\ast(3)}_{\nu}\epsilon^{\ast(4)\,\nu}+\epsilon^{(1)}_{\mu}\epsilon^{(2)}_{\nu}\epsilon^{\ast(3)\,\mu}\epsilon^{\ast(4)\,\nu}$
(9)
$\displaystyle-2\epsilon^{(1)}_{\mu}\epsilon^{(2)}_{\nu}\epsilon^{\ast(3)\,\nu}\epsilon^{\ast(4)\,\mu})\
,$

where the indices 1, 2, 3, and 4 correspond to the particles with the momenta
$p_{1}$, $p_{2}$, $p_{3}$, and $p_{4}$ in Fig. 1-(b). By considering the spin
projection operators and the isospin doublets charmed and anticharmed mesons
as Refs. Molina:2009ct ; Molina:2008jw , we obtain the kernels of
$D^{*}\bar{D}^{*}$ interactions for different isospin and total angular
momentum, which are,

$\displaystyle V^{I=0,J=0}_{D^{*}\bar{D}^{*}\to D^{*}\bar{D}^{*}}$
$\displaystyle=$ $\displaystyle 6g_{D}^{2},\qquad
V^{I=0,J=2}_{D^{*}\bar{D}^{*}\to D^{*}\bar{D}^{*}}=-3g_{D}^{2},$
$\displaystyle V^{I=1,J=0}_{D^{*}\bar{D}^{*}\to D^{*}\bar{D}^{*}}$
$\displaystyle=$ $\displaystyle 2g_{D}^{2},\qquad
V^{I=1,J=2}_{D^{*}\bar{D}^{*}\to D^{*}\bar{D}^{*}}=-g_{D}^{2},$ (10)

With the above potentials, one can search the possible poles by solving the
Bethe-Salpeter equation, which is

$T=[1-VG]^{-1}V\ ,$ (11)

where the $G$ is the two-meson loop function given by

$G=i\int\frac{d^{4}q}{(2\pi)^{4}}\frac{1}{q^{2}-m_{1}^{2}+i\epsilon}\frac{1}{(q-P)^{2}-m_{2}^{2}+i\epsilon}\
,$ (12)

with $m_{1}$ and $m_{2}$ the masses of the two mesons. $q$ is the four-
momentum of the meson in the centre of mass frame, and $P$ is the total four-
momentum of the meson-meson system.

In the present work, we use the dimensional regularization method as indicated
in Refs. Molina:2009ct ; Duan:2020vye , and in this scheme, the two-meson loop
function $G$ can be expressed as,

$\displaystyle G$ $\displaystyle=$
$\displaystyle\frac{1}{16\pi^{2}}\left[\alpha+\log\frac{m_{1}^{2}}{\mu^{2}}+\frac{m_{2}^{2}-m_{1}^{2}+s}{2s}\log\frac{m_{2}^{2}}{m_{1}^{2}}\right.$
(13)
$\displaystyle+\frac{|\vec{q}\,|}{\sqrt{s}}\left(\log\frac{s-m_{2}^{2}+m_{1}^{2}+2|\vec{q}\,|\sqrt{s}}{-s+m_{2}^{2}-m_{1}^{2}+2|\vec{q}\,|\sqrt{s}}\right.$
$\displaystyle+\left.\left.\log\frac{s+m_{2}^{2}-m_{1}^{2}+2|\vec{q}\,|\sqrt{s}}{-s-m_{2}^{2}+m_{1}^{2}+2|\vec{q}\,|\sqrt{s}}\right)\right],$

where $\mu$ and $\alpha$ are model parameter, while $\vec{q}\,$ is the
momentum of the meson in the centre of mass frame, which reads,

$|\vec{q}\,|=\frac{\sqrt{\left[s-(m_{1}+m_{2})^{2}\right]\left[s-(m_{1}-m_{2})^{2}\right]}}{2\sqrt{s}}.$
(14)

In the complex plane for a general $\sqrt{s}$, the loop function in the second
Riemann sheet can be written as

$G^{II}(\sqrt{s})=G^{I}(\sqrt{s})+i\frac{|\vec{q}\,|}{4\pi\sqrt{s}},\qquad{\rm
Im}(|\vec{q}\,|)>0,$ (15)

where $G^{II}$ refers to the loop function in the second Riemann sheet, and
$G^{I}$ is the one in the first Riemann sheet as given by Eq. (13) for the
$D^{*}\bar{D}^{*}$ channel. When searching for the poles, we use $G^{I}$ for
Re$(\sqrt{s})<m_{1}+m_{2}$, and use $G^{II}$ for Re$(\sqrt{s})>m_{1}+m_{2}$.

##  III RESULTS AND DISCUSSIONS

Before we discuss the numerical results, the values of the relevant parameters
should be clarified, which include the coupling constants $g_{D}$, the
parameter $\mu$, and the subtraction constant $\alpha$. As for $\alpha$, it is
usually fixed to be $\mu=1$ GeV as indicated in Ref. Molina:2009ct . In
general, the coupling constant $g_{D}=m_{D^{\ast}}/(2f_{D})=6.9$ Molina:2009ct
, however, in the present work, the vector meson exchanged interactions have
been absorbed by the contact diagrams, thus, the value of $g_{D}$ will be
changed in the present estimation. Here we vary $g_{D}$ from 5 to 8 to check
the parameter dependence of the results. It should be clarified that the value
of the subtraction constant $\alpha$ can not be estimated by the first
principle in the present frame and usually be considered as a phenomenological
model parameter Gamermann:2006nm ; Duan:2021pll , which can be fixed by
reproducing some relevant experimental measurements. For example, in Ref.
Molina:2009ct  the value of $\alpha$ is fixed to be -2.07 in order to get a
pole around 3940 MeV in $I=0,\ J=0$ channel, which is associated to the
$X(3915)$. In the present work, we mainly focus on the mass of the $X(4014)$,
however, the experimental data for $\gamma\gamma\to\gamma\psi(2S)$ are very
inaccurate around $4014$ MeV Belle:2021nuv , thus, it is difficult to fit our
results with experimental data. Here, We will vary the subtraction constant
$\alpha$ in a large range to check whether one can reproduce the mass of
$X(4014)$ in a reasonable parameter range.

Figure 2: The modulus squared of the amplitude $|T_{D^{*}\bar{D}^{*}\to
D^{*}\bar{D}^{*}}|^{2}$ for the case of $I=0,\ J=0$ depending on the parameter
$\alpha$.

With the above formalisms, we can estimate the amplitudes for the
$D^{*}\bar{D}^{*}\to D^{*}\bar{D}^{*}$ transition with quantum numbers
$I(J^{P})=0(0^{+})$, $0(2^{+})$, $1(0^{+})$, $0(2^{+})$. Here, we take the
case of $I=0$, $J=0$ as an example to show the modulus squared of the
amplitude $|T_{D^{\ast}\bar{D}^{\ast}\to D^{\ast}\bar{D}^{\ast}}|$ depending
on the subtraction constant $\alpha$ in Fig. 2, where the coupling constants
$g_{D}=6.9$. From the figure, one can find when $\alpha$ is very small, such
as $\alpha=-2.0$, there is a pole around 3540 MeV, by increasing $\alpha$, the
pole moves to the threshold, then the modulus squared of the amplitude behaves
like a cusp when $\alpha$ is greater than -0.7, which will not correspond to
any bound state any more.

Figure 3: The pole mass depending on the parameter $\alpha$ for the case of $I=0$, $J=0$. The grey horizontal line and band indicates the $D^{\ast}\bar{D}^{\ast}$ threshold and the mass range of $X(4014)$ measured by Belle collaboration Belle:2021nuv . Table 1: The parameter range determined by reproducing the measured mass of $X(4014)$ with different $I(J^{PC})$ assignments.  $I(J^{PC})$ |  $g_{D}$ |  $\alpha$ Range
---|---|---
$0(0^{++})$ |  5.0 |  $-0.54\sim-0.34$
|  6.0 |  $-0.86\sim-0.66$
|  6.9 |  $-1.04\sim-0.84$
|  8.0 |  $-1.18\sim-0.98$
$1(0^{++})$ |  8.0 |  $-0.36\sim-0.16$

In Fig. 3, we present the estimated pole for the case of $I=0,\ J=0$ depending
on the coupling constants $g_{D}$ and the subtraction constants $\alpha$.
Here, the coupling constants $g_{D}$ are set to be 5.0, 6.0, 6.9, and 8.0,
respectively. From the figure, one can find that, for a certain $g_{D}$, the
pole mass increases with the increasing of $\alpha$, and it appears the
overlaps between the theoretical estimation and Belle data. In Table 1, we
collect the parameter range determined by reproducing the measured mass of
$X(4014)$, where one can find for different $g_{D}$, the mass of $X(4014)$
could be reproduced by adjusting the subtraction parameter $\alpha$.

Figure 4: The same as Fig. 3 but for the case of $I=1$, $J=0$.

In Fig. 4, we present the estimated pole mass for the case of $I=1$, $J=0$
depending on the subtraction constant $\alpha$. Similar to the case of $I=0$,
$J=0$, the pole mass increases with the increasing of $\alpha$ for a certain
$g_{D}$. However, our estimations indicate that when we take $g_{D}=6.9$, the
pole mass always far below the measured mass of $X(4014)$ even when
$\alpha=0$. When we take a larger $g_{D}$, for example $g_{D}=8.0$, we can
reproduce Belle data with $-0.36<\alpha<-0.16$. Besides the cases of $J=0$, we
also check the cases of $J=2$ to further evaluate the favored quantum numbers
of $X(4014)$. In Fig. 5, we present modulus square of the amplitude for the
cases of $I=0$, $J=2$ and $I=1$, $J=2$ depending on the parameter $\alpha$.
From the figure one can find in the considered parameter space, i.e.,
$5.0<g_{D}<8.0$, $-2.0<\alpha<0$, there is no pole in the modulus square of
the amplitudes, which indicates that the $D^{\ast}\bar{D}^{\ast}$ can not form
a bound state with $J=2$ based on our calculations.

Figure 5: The same as Fig. 2 but for the cases of $I=0$, $J=2$ and $I=1$,
$J=2$.

##  IV Summary

Inspired by recent observation of $X(4014)$ in the process
$\gamma\gamma\to\gamma\psi(2S)$, we have studied the hidden-charm
$D^{*}\bar{D}^{*}$ interaction within the framework of the local hidden gauge
approach, where the possible $I(J^{PC})$ quantum numbers could be
$0(0^{++}),0(2^{++}),1(0^{++})$, and $1(2^{++})$ respectively. For the case of
$I(J^{PC})=0(0^{++})$, our estimations indicate in a large coupling constant
range, i.e., $5.0<g_{D}<8.0$, one can reproduce the measured mass of $X(4014)$
by adjusting the subtraction parameter $\alpha$. As for the case of
$I(J^{PC})=1(0^{++})$, one can also reproduce the measured mass but only with
a large coupling constants $g_{D}$. As for the cases of $I(J^{PC})=0(2^{++})$
and $I(J^{PC})=1(2^{++})$, our calculations show that there is no pole in the
modulus square of the amplitudes in the considered parameter space.

To summarize, in the $D^{\ast}\bar{D}^{\ast}$ molecular scenario, our
estimations weakly favor the $I(J^{PC})=0(0^{++})$ assignment for $X(4014)$
comparing to $I(J^{PC})=1(0^{++})$, and disfavor the $J=2$ assignments.
However, it should be clarified that the local significance of $X(4014)$ is
only 2.8$\sigma$, more experimental information about this state is crucial to
shed light on its nature.

Before the end of this work, we should mention that there already exist some
hints of $X(4140)$ in the experimental data. For instance, the Belle
Collaboration has measured the inclusive process $e^{+}e^{-}\to J/\psi X$, and
one can observe a peak around 4014 MeV in the $M_{\rm rec}(J/\psi)$
distribution, as shown in Fig. 1 of Ref. Belle:2005lik . Later, the Belle
Collaboration reported the study of the processes $e^{+}e^{-}\to J/\psi
D^{(*)}\bar{D}^{(*)}$, and there is more events in the energy region
$4000<M_{D\bar{D}^{*}}<4025$ MeV of the $D\bar{D}^{*}$ invariant mass
distribution, as depicted in Fig. 2(b) of Ref. Belle:2007woe . In addition, in
the cross sections of the process $\gamma\gamma\to J/\psi\omega$ measured by
the Belle Collaboration, one can find the events concentrate in the region
$4010\sim 4020$ MeV Belle:2009and . All the above hints imply that one state
with a mass close to the $D^{*}\bar{D}^{*}$ threshold and a narrow width
exists, which coincides with the $X(4140)$ observed by the Belle Collaboration
Belle:2021nuv .

## Acknowledgement

The authors thank Prof. Zhi-Hui Guo for valuable discussion and suggestive
comments. This work is supported by the National Natural Science Foundation of
China under Grant No.11775050, 12175037. This work is also supported by the
Natural Science Foundation of Henan under Grand No. 222300420554, the Project
of Youth Backbone Teachers of Colleges and Universities of Henan Province
(2020GGJS017), the Youth Talent Support Project of Henan (2021HYTP002), and
the Open Project of Guangxi Key Laboratory of Nuclear Physics and Nuclear
Technology, No.NLK2021-08.

## References

  * (1) N. Brambilla, S. Eidelman, C. Hanhart, A. Nefediev, C. P. Shen, C. E. Thomas, A. Vairo and C. Z. Yuan, The $XYZ$ states: experimental and theoretical status and perspectives, Phys. Rept. 873, 1-154 (2020)
  * (2) M. Cleven, F. K. Guo, C. Hanhart, Q. Wang and Q. Zhao, Employing spin symmetry to disentangle different models for the XYZ states, Phys. Rev. D 92, no.1, 014005 (2015)
  * (3) L. Ma, X. H. Liu, X. Liu and S. L. Zhu, Strong decays of the $XYZ$ states, Phys. Rev. D 91, no.3, 034032 (2015)
  * (4) Y. R. Liu, H. X. Chen, W. Chen, X. Liu and S. L. Zhu, Pentaquark and Tetraquark states, Prog. Part. Nucl. Phys. 107, 237-320 (2019)
  * (5) A. Hosaka, T. Iijima, K. Miyabayashi, Y. Sakai and S. Yasui, Exotic hadrons with heavy flavors: X, Y, Z, and related states, PTEP 2016, no.6, 062C01 (2016)
  * (6) M. B. Voloshin, Charmonium, Prog. Part. Nucl. Phys. 61, 455-511 (2008)
  * (7) M. Alberti, G. S. Bali, S. Collins, F. Knechtli, G. Moir and W. Söldner, Hadroquarkonium from lattice QCD, Phys. Rev. D 95, no.7, 074501 (2017)
  * (8) L. Maiani, F. Piccinini, A. D. Polosa and V. Riquer, The Z(4430) and a New Paradigm for Spin Interactions in Tetraquarks, Phys. Rev. D 89, 114010 (2014)
  * (9) L. Maiani, V. Riquer, R. Faccini, F. Piccinini, A. Pilloni and A. D. Polosa, A $J^{PG}=1^{++}$ Charged Resonance in the $Y(4260)\to\pi^{+}\pi^{-}J/\psi$ Decay?, Phys. Rev. D 87, no.11, 111102 (2013)
  * (10) S. Fleming, M. Kusunoki, T. Mehen and U. van Kolck, Pion interactions in the $X(3872)$, Phys. Rev. D 76, 034006 (2007)
  * (11) T. Mehen and R. Springer, Radiative Decays $X(3872)\to\psi(2S)\gamma$ and $\psi(4040)->X(3872)\gamma$ in Effective Field Theory, Phys. Rev. D 83, 094009 (2011)
  * (12) Q. Wang, M. Cleven, F. K. Guo, C. Hanhart, U. G. Meißner, X. G. Wu and Q. Zhao, Y(4260): hadronic molecule versus hadro-charmonium interpretation, Phys. Rev. D 89, no.3, 034001 (2014)
  * (13) D. Gamermann and E. Oset, Isospin breaking effects in the X(3872) resonance, Phys. Rev. D 80, 014003 (2009)
  * (14) Q. Wu, D. Y. Chen and T. Matsuki, A phenomenological analysis on isospin-violating decay of $X(3872)$, Eur. Phys. J. C 81, no.2, 193 (2021)
  * (15) Q. Wu and D. Y. Chen, Exploration of the hidden charm decays of Zcs(3985), Phys. Rev. D 104, no.7, 074011 (2021)
  * (16) N. Ikeno, R. Molina and E. Oset, The $Z_{cs}(3985)$ as a threshold effect from the $\bar{D}_{s}^{*}D+\bar{D}_{s}D^{*}$ interaction, Phys. Lett. B 814, 136120 (2021)
  * (17) D. Y. Chen and X. Liu, Phys. Rev. D 84 (2011), 094003 doi:10.1103/PhysRevD.84.094003 [arXiv:1106.3798 [hep-ph]].
  * (18) S. X. Nakamura, A. Hosaka and Y. Yamaguchi, $P_{c}(4312)^{+}$ and $P_{c}(4337)^{+}$ as interfering $\Sigma_{c}\bar{D}$ and $\Lambda_{c}\bar{D}^{*}$ threshold cusps, Phys. Rev. D 104, no.9, L091503 (2021)
  * (19) D. Y. Chen, X. Liu and T. Matsuki, Phys. Rev. Lett. 110 (2013) no.23, 232001 doi:10.1103/PhysRevLett.110.232001 [arXiv:1303.6842 [hep-ph]].
  * (20) T. Aushev et al. [Belle], Study of the B —$>$ X(3872)(D*0 anti-D0) K decay, Phys. Rev. D 81, 031103 (2010)
  * (21) B. Aubert et al. [BaBar], Study of Resonances in Exclusive B Decays to anti-D(*) D(*) K, Phys. Rev. D 77, 011102 (2008)
  * (22) M. Ablikim et al. [BESIII], Observation of a Near-Threshold Structure in the $K^{+}$ Recoil-Mass Spectra in $e^{+}e^{-}\rightarrow K^{+}(D_{s}^{-}D^{*0}+D_{s}^{*-}D^{0}$), Phys. Rev. Lett. 126, no.10, 102001 (2021)
  * (23) R. Aaij et al. [LHCb], Observation of an exotic narrow doubly charmed tetraquark, [arXiv:2109.01038 [hep-ex]].
  * (24) R. Aaij et al. [LHCb], Study of the doubly charmed tetraquark $T_{cc}^{+}$, [arXiv:2109.01056 [hep-ex]].
  * (25) X. Z. Ling, M. Z. Liu, L. S. Geng, E. Wang and J. J. Xie, Can we understand the decay width of the Tcc+ state?, Phys. Lett. B 826 (2022), 136897
  * (26) A. Feijoo, W. H. Liang and E. Oset, D0D0$\pi$\+ mass distribution in the production of the Tcc exotic state, Phys. Rev. D 104 (2021) no.11, 114015
  * (27) H. Ren, F. Wu and R. Zhu, Hadronic Molecule Interpretation of Tcc+ and Its Beauty Partners, Adv. High Energy Phys. 2022 (2022), 9103031
  * (28) R. Chen, Q. Huang, X. Liu and S. L. Zhu, Predicting another doubly charmed molecular resonance Tcc’+ (3876), Phys. Rev. D 104 (2021) no.11, 114042
  * (29) M. L. Du, V. Baru, X. K. Dong, A. Filin, F. K. Guo, C. Hanhart, A. Nefediev, J. Nieves and Q. Wang, Coupled-channel approach to Tcc+ including three-body effects, Phys. Rev. D 105 (2022) no.1, 014024
  * (30) D. Gamermann, E. Oset, D. Strottman and M. J. Vicente Vacas, Dynamically generated open and hidden charm meson systems, Phys. Rev. D 76 (2007), 074016
  * (31) E. Wang, H. S. Li, W. H. Liang and E. Oset, Analysis of the ${\gamma\gamma\to D\bar{D}}$ reaction and the $D\bar{D}$ bound state, Phys. Rev. D 103 (2021), 054008.
  * (32) E. Wang, W. H. Liang and E. Oset, Analysis of the $e^{+}e^{-}\to J/\psi D\bar{D}$ reaction close to the threshold concerning claims of a $\chi_{c0}(2P)$ state, Eur. Phys. J. A 57 (2021), 38
  * (33) M. Y. Duan, G. Y. Wang, E. Wang, D. M. Li and D. Y. Chen, Revisiting the $Z_{c}$ (4025) structure observed by BESIII in $e^{+}e^{-}\to(D^{*}\bar{D}^{*})^{\pm,0}\pi^{\mp,0}$ at $\sqrt{s}$ =4.26 GeV, Phys. Rev. D 104, no.7, 074030 (2021)
  * (34) X. K. Dong, F. K. Guo and B. S. Zou, Explaining the Many Threshold Structures in the Heavy-Quark Hadron Spectrum, Phys. Rev. Lett. 126, no.15, 152001 (2021)
  * (35) X. L. Wang et al. [Belle], Study of $\gamma\gamma\to\gamma\psi(2S)$ at Belle, Phys. Rev. D 105, no.11, 112011 (2022)
  * (36) F. K. Guo, C. Hidalgo-Duque, J. Nieves and M. P. Valderrama, Consequences of Heavy Quark Symmetries for Hadronic Molecules, Phys. Rev. D 88 (2013), 054007
  * (37) J. Nieves and M. P. Valderrama, The Heavy Quark Spin Symmetry Partners of the X(3872), Phys. Rev. D 86 (2012), 056004
  * (38) X. Liu, Z. G. Luo and Z. F. Sun, X(3915) and X(4350) as new members in P-wave charmonium family, Phys. Rev. Lett. 104 (2010), 122001
  * (39) M. X. Duan, S. Q. Luo, X. Liu and T. Matsuki, Possibility of charmoniumlike state $X(3915)$ as $\chi_{c0}(2P)$ state, Phys. Rev. D 101 (2020) no.5, 054029
  * (40) V. Kher and A. K. Rai, Spectroscopy and decay properties of charmonium, Chin. Phys. C 42 (2018) no.8, 083101
  * (41) X. Li and M. B. Voloshin, $X(3915)$ as a $D_{s}\bar{D}_{s}$ bound state, Phys. Rev. D 91 (2015) no.11, 114014
  * (42) X. K. Dong, F. K. Guo and B. S. Zou, A survey of heavy-antiheavy hadronic molecules, Progr. Phys. 41, 65-93 (2021)
  * (43) R. Molina and E. Oset, The Y(3940), Z(3930) and the X(4160) as dynamically generated resonances from the vector-vector interaction, Phys. Rev. D 80, 114013 (2009).
  * (44) M. Bando, T. Kugo and K. Yamawaki, Nonlinear Realization and Hidden Local Symmetries, Phys. Rept. 164, 217-314 (1988)
  * (45) H. Nagahiro, L. Roca, A. Hosaka and E. Oset, Hidden gauge formalism for the radiative decays of axial-vector mesons, Phys. Rev. D 79, 014015 (2009)
  * (46) P.A. Zyla et al. [Particle Data Group], Review of Particle Physics, PTEP 2020, no.8, 083C01 (2020)
  * (47) R. Molina, D. Nicmorus and E. Oset, The rho rho interaction in the hidden gauge formalism and the f(0)(1370) and f(2)(1270) resonances, Phys. Rev. D 78, 114018 (2008)
  * (48) M. Y. Duan, J. Y. Wang, G. Y. Wang, E. Wang and D. M. Li, Role of scalar $a_{0}(980)$ in the single Cabibbo suppressed process $D^{+}\rightarrow\pi^{+}\pi^{0}\eta$, Eur. Phys. J. C 80, no.11, 1041 (2020)
  * (49) K. Abe et al. [Belle], Observation of a new charmonium state in double charmonium production in e+ e- annihilation at s**(1/2) ~ 10.6-GeV, Phys. Rev. Lett. 98 (2007), 082001
  * (50) P. Pakhlov et al. [Belle], Production of New Charmoniumlike States in e+ e- –$>$ J/psi D(*) anti-D(*) at s**(1/2) ~ 10. GeV, Phys. Rev. Lett. 100 (2008), 202001
  * (51) S. Uehara et al. [Belle], Observation of a charmonium-like enhancement in the gamma gamma —$>$ omega J/psi process, Phys. Rev. Lett. 104 (2010), 092001


Property (2.4) follows from similar lines as the proof of Proposition 2.1. To
show (2.5), note that

$\displaystyle
Q^{\pi}_{r}(s,a)=\max_{u\in\mathcal{U}}Q^{\pi}_{u}(s,a)=c(s,a)+\max_{u\in\mathcal{U}}\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u}(s^{\prime}|s,a)V^{\pi}_{u}(s^{\prime}),$

where the last inequality follows from the standard relation between
$V^{\pi}_{u}$ and $Q^{\pi}_{u}$. It suffices to note that by taking
$u=u_{\pi}$ defined in (2.2), we have $V^{\pi}_{u_{\pi}}=V^{\pi}_{r}$ and thus
$\max_{u\in\mathcal{U}}\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u}(s^{\prime}|s,a)V^{\pi}_{u}(s^{\prime})\geq\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u_{\pi}}(s^{\prime}|s,a)V^{\pi}_{r}(s^{\prime})=\max_{u\in\mathcal{U}}\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u}(s^{\prime}|s,a)V^{\pi}_{r}(s^{\prime})$,
where the last equality follows from (2.2). On the other hand, we have

$\displaystyle\max_{u\in\mathcal{U}}\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u}(s^{\prime}|s,a)V^{\pi}_{u}(s^{\prime})\leq\max_{u\in\mathcal{U}}\max_{u^{\prime}\in\mathcal{U}}\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u}(s^{\prime}|s,a)V^{\pi}_{u^{\prime}}(s^{\prime})=\max_{u\in\mathcal{U}}\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u}(s^{\prime}|s,a)V^{\pi}_{r}(s^{\prime}),$

hence we obtain

$\displaystyle
Q^{\pi}_{r}(s,a)=c(s,a)+\max_{u\in\mathcal{U}}\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u}(s^{\prime}|s,a)V^{\pi}_{u}(s^{\prime})=c(s,a)+\max_{u\in\mathcal{U}}\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u}(s^{\prime}|s,a)V^{\pi}_{r}(s^{\prime}).$

Thus (2.5) is proved. Moreover, (2.6) follows from taking expectation with
respect to $a\sim\pi(\cdot|s)$ on both sides of (2.5) and making use of (2.1).
Finally, from (2.5) and (2.6), it is also clear that

$\displaystyle
Q^{\pi}_{r}(s,a)=c(s,a)+\gamma\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u_{\pi}}(s^{\prime}|s,a)\sum_{a^{\prime}\in\mathcal{A}}\pi(a^{\prime}|s^{\prime})Q^{\pi}_{r}(s^{\prime},a^{\prime}),~{}~{}\forall(s,a)\in{\mathcal{S}}\times\mathcal{A},$

where $u_{\pi}$ is defined as in (2.2), then $Q^{\pi}_{r}=Q^{\pi}_{u_{\pi}}$
since $Q^{\pi}_{u_{\pi}}$ is the unique solution of the previous system. ∎

###### Proof of Lemma 2.3.

It suffices to consider the differentiability of $f_{\rho}$ inside
$\mathrm{ReInt}(\Pi)$, as its relative boundary is a zero-measure set when
taking the
$(\lvert\mathcal{A}\rvert-1){\lvert{\mathcal{S}}\rvert}$-dimensional Hausdorff
measure. We begin by noting that there exists
$\left\\{e_{i}\right\\}_{i=0}^{\lvert\mathcal{A}\rvert-1}\subset\mathbb{R}^{\lvert\mathcal{A}\rvert}$,
such that for any $\pi\in\Pi$, we have uniquely defined
$\left\\{a^{\pi}_{i}(s)\right\\}_{i=1,\ldots,\lvert\mathcal{A}\rvert-1,s\in{\mathcal{S}}}$
satisfying

$\displaystyle\pi(\cdot|s)=\sum_{i=1}^{\lvert\mathcal{A}\rvert-1}a_{i}^{\pi}(s)\cdot
e_{i}+e_{0}\coloneqq E\cdot a^{\pi}(s)+e_{0},~{}~{}\forall s\in{\mathcal{S}},$
(A.5)

where we denote
$a^{\pi}(s)=(a_{1}^{\pi}(s),\ldots,a_{\lvert\mathcal{A}\rvert-1}^{\pi}(s))$,
and $E=[e_{1},\ldots,e_{\lvert\mathcal{A}\rvert-1}]$ has independent columns.
We also write the above relation in short as $\pi=\mathcal{M}(a^{\pi})$. It is
clear that $\mathcal{M}$ is a Lipschitz continuous mapping, and we denote its
Lipschitz constant by $L_{\mathcal{M}}$. Alternatively, since $E$ has
independent columns, we also have

$\displaystyle a^{\pi}(s)=E^{\dagger}(\pi(\cdot|s)-e_{0}),~{}~{}\forall
s\in{\mathcal{S}},$ (A.6)

where $E^{\dagger}$ denotes the Moore–Penrose inverse of $E$. We will write
the above relation in short as $a^{\pi}=\mathcal{M}^{-1}(\pi)$. In addition,
we can write the objective (1.5) equivalently as

$\displaystyle f_{\rho}(\pi)=g(a^{\pi};\mathcal{E}),$ (A.7)

where
$a^{\pi}=\left(a^{\pi}(s)\right)_{s\in{\mathcal{S}}}=\mathcal{M}^{-1}(\pi)\subset\mathbb{R}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}$
is defined as in (A.6), and $\mathcal{E}=(E;e_{0})$. Now consider the set

$\displaystyle\mathcal{A}=\left\\{(a(1),\ldots,a(\lvert{\mathcal{S}}\rvert)):~{}E\cdot
a(s)+e_{0}\in\mathrm{ReInt}(\Delta_{\mathcal{A}}),~{}\forall
s\in{\mathcal{S}}\right\\}\subset\mathbb{R}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}.$

It is clearly that $\mathcal{A}$ is an open set in
$\mathbb{R}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}$. In
addition, for any $a,a^{\prime}\in\mathcal{A}$, by letting
$\pi=\mathcal{M}(a)$, $\pi^{\prime}=\mathcal{M}(a^{\prime})$, we have

$\displaystyle\lvert g(a;\mathcal{E})-g(a^{\prime};\mathcal{E})\rvert$
$\displaystyle=\lvert f_{\rho}(\pi)-f_{\rho}(\pi^{\prime})\rvert$
$\displaystyle\overset{(a)}{\leq}\tfrac{1}{1-\gamma}\sup_{s\in{\mathcal{S}}}\lVert\pi(\cdot|s)-\pi^{\prime}(\cdot|s)\rVert_{1}$
$\displaystyle\overset{(b)}{\leq}\tfrac{\sqrt{\lvert\mathcal{A}\rvert}}{1-\gamma}\sup_{s\in{\mathcal{S}}}\lVert
E\rVert_{2}\lVert a(s)-a^{\prime}(s)\rVert_{2}$
$\displaystyle\leq\tfrac{\sqrt{\lvert\mathcal{A}\rvert}}{1-\gamma}\lVert
E\rVert_{2}\lVert a-a^{\prime}\rVert_{2},$

where $(a)$ follows from Lemma B.6, and $(b)$ follows from the definition
(A.5). From the prior relation, we know that
$g(\cdot;\mathcal{E}):\mathcal{A}\to\mathbb{R}$ is a Lipschitz continuous
mapping. Combined with the fact that $\mathcal{A}$ is open, we conclude from
the Rademacher’s theorem [40] that $g(\cdot;\mathcal{E})$ is almost everywhere
differentiable in $\mathcal{A}$, when the measure is taken to be the
$\mathbb{R}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}$-dimensional
Lebesgue measure. Let us define
$\mathcal{A}_{z}\subset\mathcal{A}\subset\mathbb{R}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}$
as the set of non-differentiable points of $g(\cdot;\mathcal{E})$.
Accordingly, we define
$\Pi_{z}=\left\\{\pi\in\Pi:\pi=\mathcal{M}(a),a\in\mathcal{A}_{z}\right\\}\subset\mathbb{R}^{\lvert{\mathcal{S}}\rvert\lvert\mathcal{A}\rvert}$.
We proceed to show that $\Pi_{z}$ is a zero-measure set when taking the
measure to be the
$(\lvert\mathcal{A}\rvert-1){\lvert{\mathcal{S}}\rvert}$-dimensional Hausdorff
measure.

Recall that the $m$-dimensional Hausdorff measure of any set $A$ is defined as
(see [41])

$\displaystyle\mathcal{H}^{m}(A)$ $\displaystyle=\lim_{\delta\to
0,\delta>0}\mathcal{H}^{m}_{\delta}(A)=\sup_{\delta>0}\mathcal{H}^{m}_{\delta}(A),$
(A.8) $\displaystyle\mathcal{H}^{m}_{\delta}(A)$
$\displaystyle=w_{m}\inf_{\left\\{C_{j}\right\\}_{j=1}^{\infty}}\left\\{\sum_{j=1}^{\infty}\left(\tfrac{\mathrm{diam}(C_{j})}{2}\right)^{m}:~{}\mathrm{diam}(C_{j})<\delta,~{}A\subset\cup_{j=1}^{\infty}C_{j}\right\\},$
(A.9)

where $w_{m}=\pi^{m/2}/\Gamma(\tfrac{m}{2}+1)$. In addition, by letting
$\mathcal{L}^{m}$ denote the $m$-dimensional Lebesgure measure in
$\mathbb{R}^{m}$, we have the following relation [41],

$\displaystyle\mathcal{L}^{m}(A)=\mathcal{H}^{m}(A)=\mathcal{H}^{m}_{\delta}(A),~{}\forall\delta>0,~{}\forall
A\subset\mathbb{R}^{m}.$ (A.10)

Now fix $\delta>0$, for any collection of subset
$\left\\{C_{j}\right\\}_{j=1}^{m}\subset\mathbb{R}^{(\lvert\mathcal{A}\rvert-1){\lvert{\mathcal{S}}\rvert}}$
with $\mathrm{diam}(C_{j})<\delta$, and
$\mathcal{A}_{z}\subset\cup_{j=1}^{\infty}C_{j}$, we know that
$\Pi_{z}\subset\cup_{j=1}^{\infty}\mathcal{M}(C_{j})$, and
$\mathrm{diam}(\mathcal{M}(C_{j}))\leq L_{\mathcal{M}}\mathrm{diam}(C_{j})$.
Thus,

$\displaystyle\mathcal{H}^{(\lvert\mathcal{A}\rvert-1){\lvert{\mathcal{S}}\rvert}}_{(L_{\mathcal{M}}\delta)}(\Pi_{z})\leq\sum_{j=1}^{\infty}\left(\tfrac{\mathrm{diam}(\mathcal{M}(C_{j}))}{2}\right)^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}\leq\sum_{j=1}^{\infty}\left(\tfrac{\mathrm{diam}(C_{j})L_{\mathcal{M}}}{2}\right)^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}.$

Now by taking infimum over
$\left\\{C_{j}\right\\}_{j=1}^{m}\subset\mathbb{R}^{(\lvert\mathcal{A}\rvert-1){\lvert{\mathcal{S}}\rvert}}$
of the right hand side, we obtain

$\displaystyle\mathcal{H}^{(\lvert\mathcal{A}\rvert-1){\lvert{\mathcal{S}}\rvert}}_{(L_{\mathcal{M}}\delta)}(\Pi_{z})$
$\displaystyle\leq
L_{\mathcal{M}}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}\inf_{\left\\{C_{j}\right\\}_{j=1}^{\infty}}\sum_{j=1}^{\infty}\left(\tfrac{\mathrm{diam}(C_{j})}{2}\right)^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}$
$\displaystyle\overset{(a)}{=}L_{\mathcal{M}}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}\cdot\mathcal{H}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}_{\delta}(\mathcal{A}_{z})$
$\displaystyle\overset{(b)}{=}L_{\mathcal{M}}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}\cdot\mathcal{L}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}(\mathcal{A}_{z})$
$\displaystyle\overset{(c)}{=}0,$

where $(a)$ follows from the definition in (A.9), $(b)$ follows from
equivalence of
$\mathcal{L}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}$ and
$\mathcal{H}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}_{\delta}$
for any $\delta>0$ given (A.10), and the fact that
$\mathcal{A}_{z}\subset\mathbb{R}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}$.
Finally, $(c)$ follows from the fact that
$\mathcal{L}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}(\mathcal{A}_{z})=0$.
Thus, by letting $\delta\to 0$ on the left hand side, and making use of the
definition of Hausdorff measure (A.8), we obtain
$\mathcal{H}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}(\Pi_{z})=0$.

We then proceed to show that $f_{\rho}$ is differentiable within
$\Pi_{d}=\mathrm{ReInt}(\Pi)\setminus\Pi_{z}=\mathcal{M}(\mathcal{A}\setminus\mathcal{A}_{z})$,
where the differentiability is defined in the sense of Definition 2.1. To see
this, we first note that from the differentiability of $g(\cdot;\mathcal{E})$,
for any $a^{\prime}\in\mathcal{A}\setminus\mathcal{A}_{z}$,

$\displaystyle g(a;\mathcal{E})-g(a^{\prime};\mathcal{E})-\langle\nabla
g(a^{\prime};\mathcal{E}),a-a^{\prime}\rangle={{\scriptstyle\mathcal{O}}}(\lVert
a-a^{\prime}\rVert),~{}\forall a.$ (A.11)

Let us denote $\nabla g(a;\mathcal{E})[s]$ as the partial derivative of
$g(a;\mathcal{E})$ with respect to $a(s)$. Now given any
$\pi^{\prime}\in\Pi_{d}$, consider any policy $\pi$, we know that there exists
$a=\mathcal{M}^{-1}(\pi)$, $a^{\prime}=\mathcal{M}^{-1}(\pi^{\prime})$, with
$a^{\prime}\in\mathcal{A}\setminus\mathcal{A}_{z}$. Hence we obtain from the
differentiability of $g(\cdot;\mathcal{E})$ at $a^{\prime}$ that

$\displaystyle
f_{\rho}(\pi)-f_{\rho}(\pi^{\prime})-\sum_{s\in{\mathcal{S}}}\langle\nabla
g(a^{\prime};\mathcal{E})[s],E^{\dagger}(\pi(\cdot|s)-\pi^{\prime}(\cdot|s)\rangle$
$\displaystyle\overset{(a)}{=}$ $\displaystyle
g(a;\mathcal{E})-g(a^{\prime};\mathcal{E})-\sum_{s\in{\mathcal{S}}}\langle\nabla
g(a^{\prime};\mathcal{E})[s],a(s)-a^{\prime}(s)\rangle$
$\displaystyle\overset{(b)}{=}$
$\displaystyle{{\scriptstyle\mathcal{O}}}(\lVert a-a^{\prime}\rVert)$
$\displaystyle\overset{(c)}{=}$
$\displaystyle{{\scriptstyle\mathcal{O}}}(\lVert\pi-\pi^{\prime}\rVert),$

where $(a)$ follows from (A.6) and (A.7), $(b)$ follows from (A.11) and the
differentiability of $g(\cdot;\mathcal{E})$ at $a^{\prime}$, and $(c)$ follows
again from (A.6). Thus from the above relation and Definition 2.1, we know
that $f_{\rho}$ is differentiable at any $\pi\in\Pi^{d}$, whose $(s,a)$-entry
is given by

$\displaystyle\nabla f_{\rho}(\pi)[s,a]=\left[(E^{\dagger})^{\top}\nabla
g(\mathcal{M}^{-1}(\pi);\mathcal{E})[s]\right][a],~{}~{}\forall(s,a)\in{\mathcal{S}}\times\mathcal{A}.$

∎

###### Proof of Lemma 2.4.

Let us define
$e=\mathbf{1}/\lvert\mathcal{A}\rvert\in\mathbb{R}^{\lvert\mathcal{A}\rvert}$,
and let
$\mathbf{e}=e\otimes\mathbf{1}\in\mathbb{R}^{\lvert{\mathcal{S}}\rvert\lvert\mathcal{A}\rvert}$.
For any policy $\pi\in\Pi$, we can accordingly define $\pi_{e}$ where
$\pi_{e}(\cdot|s)=\pi(\cdot|s)-e$. Conversely, let
$\Pi_{e}=\left\\{\pi_{e}:\pi\in\Pi\right\\}$, note that the mapping
$\mathcal{M}:\Pi\to\Pi_{e}$ from $\pi$ to $\pi_{e}$ is one-to-one and onto.
Let us define function $g$ with domain $\Pi_{e}$ as

$\displaystyle g(\pi_{e})=f_{\rho}(\pi),~{}~{}\forall\pi_{e}\in\Pi_{e}.$

It should be clear that $\Pi_{e}\subset\mathcal{U}$ where $\mathcal{U}$ is a
$(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert$-dimensional subspace in
$\mathbb{R}^{\lvert{\mathcal{S}}\rvert\lvert\mathcal{A}\rvert}$. Let
$\lVert\cdot\rVert$ denote the euclidean-norm on
$\mathbb{R}^{\lvert{\mathcal{S}}\rvert\lvert\mathcal{A}\rvert}$, then it is
immediate that $(\mathcal{U},\lVert\cdot\rVert)$ is a Banach space, and
$g:\Pi_{e}\subset\mathcal{U}\to\mathbb{R}$.

For any $\pi\in\mathrm{ReInt}(\Pi)$ where $f_{\rho}$ is differentiable in the
sense of Definition 2.1, we have
$f_{\rho}(\pi^{\prime})-f_{\rho}(\pi)-\langle\nabla
f_{\rho}(\pi),\pi^{\prime}-\pi\rangle={{\scriptstyle\mathcal{O}}}(\lVert\pi-\pi^{\prime}\rVert_{2})$.
Thus, by letting $\mathrm{Int}_{\mathcal{U}}(X)$ denote the interior of set
$X$ inside the Banach space $(\mathcal{U},\lVert\cdot\rVert)$, we obtain for
$\pi_{e}=\mathcal{M}^{-1}(\pi)\in\mathrm{Int}_{\mathcal{U}}(\Pi_{e})$ that

$\displaystyle g(\pi_{e}^{\prime})-g(\pi_{e})-\langle\nabla
f(\pi),\pi_{e}^{\prime}-\pi_{e}\rangle={{\scriptstyle\mathcal{O}}}(\lVert\pi_{e}-\pi_{e}^{\prime}\rVert),~{}\forall\pi^{\prime}_{e}\in\Pi_{e}.$

That is, $g$ is Fréchet differentiable at $\pi_{e}$ with gradient $\nabla
g(\pi_{e})=\nabla f(\pi)$. Given Lemma 2.3, it should be clear that the set of
$\pi_{e}$ in $\Pi_{e}$ where $g$ is not Fréchet differentiable has a measure
of zero in the
$(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert$-dimensional Hausdorff
measure.

We next show that (2.12) defines a Fréchet subgradient of $g$. This is due to
the fact that for any $\pi_{e}\in\mathrm{Int}_{\mathcal{U}}(\Pi_{e})$,

$\displaystyle\liminf_{\pi^{\prime}_{e}\to\pi_{e},\pi_{e}^{\prime}\neq\pi_{e},\pi_{e}^{\prime}\in\Pi_{e}}\left(g(\pi^{\prime}_{e})-g(\pi_{e})-\tfrac{1}{1-\gamma}\sum_{s\in{\mathcal{S}}}\sum_{a\in\mathcal{A}}d_{\rho}^{\pi,u_{\pi}}(s)Q^{\pi}_{u_{\pi}}(s,a)\left(\pi^{\prime}_{e}(a|s)-\pi_{e}(a|s)\right)\right)\big{/}\lVert\pi_{e}^{\prime}-\pi_{e}\rVert$
$\displaystyle=$
$\displaystyle\liminf_{\pi^{\prime}\to\pi,\pi^{\prime}\neq\pi,\pi^{\prime}\in\Pi}\left(f_{\rho}(\pi^{\prime})-f_{\rho}(\pi)-\tfrac{1}{1-\gamma}\sum_{s\in{\mathcal{S}}}\sum_{a\in\mathcal{A}}d_{\rho}^{\pi,u_{\pi}}(s)Q^{\pi}_{u_{\pi}}(s,a)\left(\pi^{\prime}(a|s)-\pi(a|s)\right)\right)\big{/}\lVert\pi^{\prime}-\pi\rVert_{2}\geq
0,$

where the equality follows from the construction of $g$ and $\pi_{e}$, and the
inequality follows from Lemma 2.2.

By combining observations from the previous two paragraphs, it is clear that
the set of non-Fréchet differentiable points of $g$ form a zero-measure set in
$\mathcal{U}$, when taking the
$(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert$-dimensional Hausdorff
measure. The Fréchet subgradient at any point $\pi_{e}$ is given by (2.12).
Moreover, at any Fréchet differentiable point $\pi_{e}$ of $g$, we conclude
that its Fréchet gradient $\nabla g(\pi)$ is the only element in its Fréchet
subdifferential (Proposition 1.1, [18]), which is given by (2.12).

It remains to show that for any Fréchet differentiable point
$\pi_{e}\in\mathrm{Int}_{\mathcal{U}}(\Pi_{e})$ of $g$, with its derivative
denoted by $\nabla g(\pi)$, the corresponding $\pi\in\mathrm{ReInt}(\Pi)$ is
also a differentiable point of $f_{\rho}$ in the sense of Definition 2.1. To
see this, note that

$\displaystyle f_{\rho}(\pi^{\prime})-f_{\rho}(\pi)-\langle\nabla
g(\pi_{e}),\pi^{\prime}-\pi\rangle$ $\displaystyle=$ $\displaystyle
g(\pi^{\prime}_{e})-g(\pi_{e})-\langle\nabla
g(\pi_{e}),\pi^{\prime}_{e}-\pi_{e}\rangle={{\scriptstyle\mathcal{O}}}(\lVert\pi^{\prime}_{e}-\pi_{e}\rVert)={{\scriptstyle\mathcal{O}}}(\lVert\pi^{\prime}-\pi\rVert_{2}),~{}~{}\forall\pi^{\prime}\in\mathrm{ReInt}(\Pi).$

Thus we conclude that $f_{\rho}$ is also differentiable at $\pi$, and the
gradient $\nabla f_{\rho}(\pi)$ defined in Definition 2.1 coincides with the
Fréchet gradient of $g$ at $\pi_{e}$, given by (2.12). The proof is then
completed. ∎

###### Proof of Lemma 2.5.

The essential arguments can be understood as an application of Danskin’s
Theorem [5], but with additional care to handle the low-dimensional nature of
the domain $\Pi$. This is due to the reason that Danskin’s Theorem requires
the domain to be an open set in the euclidean space, which is full-
dimensional, and hence can not be directly applied in our setup, see Theorem
II of Chapter 3 in [5].

We will inherit the same notations and definitions as in the proof of Lemma
2.3. Let $\Pi_{\mathcal{M}}=\left\\{a^{\pi}:\pi\in\Pi\right\\}$. Note that the
mapping $\mathcal{M}:\Pi_{\mathcal{M}}\to\Pi$ is one-to-one and onto, and
$\mathrm{Int}(\Pi_{\mathcal{M}})$ is an open set in
$\mathbb{R}^{(\lvert\mathcal{A}\rvert-1)\lvert{\mathcal{S}}\rvert}$. To
proceed, we first show that $g(a;\mathcal{E})$ is differentiable inside
$\mathrm{Int}(\Pi_{\mathcal{M}})$. To this end, note that for any
$\pi_{a}=\mathcal{M}(a)$,

$\displaystyle
g(a;\mathcal{E})=f_{\rho}(\pi_{a})=\sum_{s\in{\mathcal{S}}}\max_{u\in\mathcal{U}}V^{\pi_{a}}_{u}(s)\rho(s).$

To apply Danskin’s Theorem, it suffices to show that

  * (O)

$V^{\pi}_{u}(s)$ is continuous in $(\pi,u)$.

  * (A)

For any $a\in\mathrm{Int}(\Pi_{\mathcal{M}})$ and $u\in\mathcal{U}$,
$V^{\pi_{a}}_{u}(s)$ is differentiable in $a$, and the partial gradient is
continuous in $(a,u)$.

  * (B)

For any $a\in\mathrm{Int}(\Pi_{\mathcal{M}})$, the worst-case uncertainty
$\left\\{u\in\mathcal{U}:V^{\pi_{a}}_{u}(s)=\max_{u\in\mathcal{U}}V^{\pi_{a}}_{u}(s)\right\\}$
is a singleton, denoted by $u_{\pi_{\alpha}}$.

Note that condition (O) is trivial to verify, and condition (B) is readily
implied by the precondition of the lemma. We then turn to show (A). For any
$a^{\prime},a\in\mathrm{Int}(\Pi_{\mathcal{M}})$, by letting
$\delta=\pi_{a}^{\prime}-\pi_{a}$,

$\displaystyle V^{\pi_{a^{\prime}}}_{u}(s)-V^{\pi_{a}}_{u}(s)$
$\displaystyle\overset{(a)}{=}\tfrac{1}{1-\gamma}\sum_{s^{\prime}\in{\mathcal{S}}}\sum_{\widetilde{a}\in\mathcal{A}}d_{s}^{\pi_{a},u}(s^{\prime})Q^{\pi_{a}}_{u}(s^{\prime},\widetilde{a})\delta(\widetilde{a}|s^{\prime})+{{\scriptstyle\mathcal{O}}}(\lVert\delta\rVert_{2}),$
$\displaystyle\overset{(b)}{=}\mathcal{L}^{\pi_{a}}_{u}(\delta)+{{\scriptstyle\mathcal{O}}}(\lVert
a^{\prime}-a\rVert_{2})$ (A.12)
$\displaystyle\overset{(c)}{=}\mathcal{L}^{\pi_{a}}_{u}(\mathcal{M}(a^{\prime}-a))+{{\scriptstyle\mathcal{O}}}(\lVert
a^{\prime}-a\rVert_{2}),$

where $(a)$ is due to Lemma 2.1, and $(b)$ and $(c)$ follow from the
definition in (A.5), and $\mathcal{L}^{\pi}_{u}$ denotes a linear operator of
$\delta$ mapping to $\mathbb{R}$ and implicitly defined via the first term in
equality $(a)$. Hence given $(c)$, since $\mathcal{L}_{u}^{a}\circ M$ is again
a linear operator, we know that $V^{\pi_{a}}_{u}(s)$ is differentiable at any
point $a\in\mathrm{Int}(\Pi_{\mathcal{M}})$. To show the continuity of the
gradient, it suffices to show that the operator $\mathcal{L}_{u}^{\pi}$ is
continuous, which simply follows from the fact that $Q^{\pi}_{u}$ and
$d^{\pi,u}_{s}$ is continuous in $(\pi,u)$ (following from similar arguments
of (2.11) and B.10). Thus term (A) is proved.

Applying the Danskin’s Theorem, we obtain that the robust value
$V^{\pi_{a}}_{r}(s)=\max_{u\in\mathcal{U}}V^{\pi_{a}}_{u}(s)$ is also
differentiable in $a$, for any $a\in\mathrm{Int}(\Pi_{\mathcal{M}})$.
Specifically, we have

$\displaystyle
V^{\pi_{a^{\prime}}}_{r}(s)-V^{\pi_{a}}_{r}(s)=\mathcal{L}^{\pi_{a}}_{u_{\pi_{\alpha}}}(\mathcal{M}(a^{\prime}-a))+{{\scriptstyle\mathcal{O}}}(\lVert
a^{\prime}-a\rVert_{2}),$ (A.13)

for any $a,a^{\prime}\in\mathrm{Int}(\Pi_{\mathcal{M}})$. Now given any
$\pi,\pi^{\prime}\in\mathrm{ReInt}(\Pi)$, it is clear that
$a=a^{\pi},a^{\prime}=a^{\pi^{\prime}}$ both belong to
$\mathrm{Int}(\Pi_{\mathcal{M}})$, hence

$\displaystyle
V^{\pi^{\prime}}_{r}(s)-V^{\pi}_{r}(s)=V^{\pi_{a^{\prime}}}_{r}(s)-V^{\pi_{a}}_{r}(s)$
$\displaystyle\overset{(a)}{=}\mathcal{L}^{\pi_{a}}_{u_{\pi_{\alpha}}}(\mathcal{M}(a^{\prime}-a))+{{\scriptstyle\mathcal{O}}}(\lVert
a^{\prime}-a\rVert_{2})$
$\displaystyle\overset{(b)}{=}\mathcal{L}^{\pi}_{u_{\pi}}(\pi^{\prime}-\pi)+{{\scriptstyle\mathcal{O}}}(\lVert\pi^{\prime}-\pi\rVert_{2}),$

where $(a)$ follows from (A.13), and $(b)$ follows from the definition of
$a,a^{\prime}$ and (A.5), (A.6). Since $\mathcal{L}^{\pi}_{u_{\pi}}$ is a
linear operator, we obtain that $V^{\pi}_{r}(s)$ is differentiable at
$\pi\in\mathrm{ReInt}(\Pi)$ in the sense of Definition 2.1. The concrete form
of gradient can be simplify read from the definition of operator
$\mathcal{L}^{\pi}_{u}$ in (A.12), the proof is then completed.

∎

##  Appendix B RPMD with a General Class of Bregman Divergences

In this section, we show that for a general class of Bregman divergences,
whenever the set of uncertain transitions $\mathcal{P}_{s,a}$ form a
relatively strongly convex set in $\mathbb{R}^{\lvert{\mathcal{S}}\rvert}$,
then RPMD converges with any constant stepsize for a general class of
divergences. To proceed, we first recall the following definition of strongly
convex sets.

######  Definition B.1 (Strongly Convex Set).

A set $\mathcal{C}$ in $\mathbb{R}^{d}$ is called strongly convex with respect
to $R>0$, if $\mathcal{C}$ is bounded, and for any $x,y\in\mathcal{C}$, any
$\lambda\in[0,1]$, we have $\mathcal{B}(\lambda
x+(1-\lambda)y),\delta)\subset\mathcal{C}$, where
$\delta=\tfrac{\lambda(1-\lambda)}{2R}\lVert x-y\rVert_{2}^{2}$. Here
$\mathcal{B}(z,r)=\left\\{z^{\prime}\in\mathbb{R}^{d}:\lVert
z^{\prime}-z\rVert_{2}\leq r\right\\}$ denotes the ball centered around $z$
with radius $r$.

It is well known that any level set of a strongly convex function is a
strongly convex set [44]. In particular, any full-dimensional ellipsoid is a
strongly convex set. A strongly convex convex set also satisfies the following
useful property.

######  Lemma B.1 (Theorem 1, [44]).

For any set $\mathcal{C}\subset\mathbb{R}^{d}$ that is strongly convex with
respect to $R>0$, let $\mathrm{Bd}(C)$ denote its boundary, and
$\mathcal{N}_{\mathcal{C}}(x)$ denote its normal cone at $x\in X$. Then for
any pair of vectors $(x_{i},p_{i})$, $i=1,2$, where
$x_{i}\in\mathrm{Bd}(\mathcal{C})$ and
$p_{i}\in\mathcal{N}_{\mathcal{C}}(x_{i})$ with $\lVert p_{i}\rVert_{2}=1$, we
have

$\displaystyle\lVert x_{1}-x_{2}\rVert_{2}\leq R\lVert p_{1}-p_{2}\rVert.$
(B.1)

Note that by definition, a strongly convex set must have full dimension, which
can not be satisfied by $\mathcal{P}_{s,a}$ as
$\mathcal{P}_{s,a}\subset\Delta_{\mathcal{A}}$ lies in a lower dimensional
hyperplane. Given this observation, we define the following strong convexity
of a set ${\mathcal{S}}$ restricted to its affine span.

######  Definition B.2 (Relatively Strongly Convex Set).

A set $\mathcal{C}$ in $\mathbb{R}^{d}$ is called relatively strongly convex
with respect to $R>0$, if $\mathcal{C}$ is bounded, and for any
$x,y\in\mathcal{C}$, any $\lambda\in[0,1]$, we have $\mathcal{B}(\lambda
x+(1-\lambda)y),\delta)\cap\mathcal{H}_{\mathcal{C}}\subset\mathcal{C}$, where
$\delta=\tfrac{\lambda(1-\lambda)}{2R}\lVert x-y\rVert_{2}^{2}$. Here
$\mathcal{B}(z,r)=\left\\{z^{\prime}\in\mathbb{R}^{d}:\lVert
z^{\prime}-z\rVert_{2}\leq r\right\\}$ denotes the ball centered around $z$
with radius $r$, and $\mathcal{H}_{\mathcal{C}}=\mathrm{Aff}(\mathcal{C})$
denotes the affine subspace spanned by $\mathcal{C}$.

Similar to Lemma B.1, the relatively strongly convex set also satisfies an
analogy to (B.1).

######  Lemma B.2.

For any set $\mathcal{C}\subset\mathbb{R}^{d}$ that is relatively strongly
convex with respect to $R>0$, let $\mathrm{ReBd}(C)$ denote its relative
boundary, and $\mathcal{N}^{r}_{\mathcal{C}}(x)$ denote its normal cone at
$x\in X$ restricted to $\mathcal{H}_{\mathcal{C}}$, i.e.,

$\displaystyle\mathcal{N}^{r}_{\mathcal{C}}(x)=\left\\{z\in\mathrm{Lin}(\mathcal{C}):(y-x)^{\top}z\leq
0,\forall y\in\mathcal{C}\right\\},$

where $\mathrm{Lin}(\mathcal{C})$ denotes the unique linear subspace defining
$\mathcal{H}_{\mathcal{C}}$, which satisfies
$\mathcal{H}_{\mathcal{C}}=x+\mathrm{Lin}(C)$ for any $x\in\mathcal{C}$. Then
for any pair of vectors $(x_{i},p_{i})$, $i=1,2$, where
$x_{i}\in\mathrm{ReBd}(\mathcal{C})$ and
$p_{i}\in\mathcal{N}^{r}_{\mathcal{C}}(x_{i})$ with $\lVert
p_{i}\rVert_{2}=1$, we have

$\displaystyle\lVert x_{1}-x_{2}\rVert_{2}\leq R\lVert p_{1}-p_{2}\rVert.$
(B.2)

###### Proof.

Note that $\mathcal{N}^{r}_{\mathcal{C}}(x)$ is invariant to translation of
set $C$. Moreover, the both sides of statement (B.2) are also invariant to
translation of set $C$, and hence we can without loss of generality assume
$\mathbf{0}\in\mathcal{C}$, and hence
$\mathcal{H}_{\mathcal{C}}=\mathrm{Lin}(\mathcal{C})$. The rest of the claim
follows directly from a change of coordinates and working in the lower-
dimensional space $\mathrm{Lin}(\mathcal{C})$, in which we can apply Lemma
B.1. ∎

We then show that if set of uncertain transitions $\mathcal{P}_{s,a}$ is a
relatively strongly convex set in $\mathbb{R}^{\lvert{\mathcal{S}}\rvert}$
with dimension $\lvert{\mathcal{S}}\rvert-1$, then the worst-case transition
kernel $\mathbb{P}_{u_{\pi}}$ is uniquely defined, and is also Lipschitz
continuous with respect to policy $\pi$. En route, we will make use of the
following simple fact.

######  Lemma B.3.

For $x,y\in\mathbb{R}^{d}$ and $x,y\neq\mathbf{0}$, we have

$\displaystyle\lVert\tfrac{x}{\lVert x\rVert_{2}}-\tfrac{y}{\lVert
y\rVert_{2}}\rVert_{2}\leq\lVert x-y\rVert_{2}/\min\left\\{\lVert
x\rVert_{2},\lVert y\rVert_{2}\right\\}.$ (B.3)

###### Proof.

Since both side of claim (B.3) are symmetric with respect to $(x,y)$, we can
without loss of generality assume $\lVert y\rVert_{2}\geq\lVert x\rVert_{2}$.
Note that $\lVert\tfrac{x}{\lVert x\rVert_{2}}-\tfrac{y}{\lVert
y\rVert_{2}}\rVert_{2}=\tfrac{1}{\lVert x\rVert_{2}}\lVert x-\tfrac{y}{\lVert
y\rVert_{2}}\lVert x\rVert_{2}\rVert_{2}$. We make the following observations.

Denote $\mathrm{Proj}_{y}:\mathbb{R}^{d}\to\mathbb{R}^{d}$ as the projection
operator onto $\mathrm{span}(\left\\{y\right\\})$. We know that
$\mathrm{Proj}_{y}(x)=\delta_{x}y$ for $\delta_{x}=\langle x,y\rangle/\lVert
y\rVert_{2}^{2}$ and $\mathrm{Proj}_{y}(x)\leq\lVert x\rVert_{2}$. On the
other hand, we have $x^{\prime}\coloneqq\tfrac{\lVert x\rVert_{2}}{\lVert
y\rVert_{2}}y=\delta_{x}^{\prime}y$ with $\lVert x^{\prime}\rVert_{2}=\lVert
x\rVert_{2}$, hence $\delta_{x}^{\prime}\geq\lvert\delta_{x}\rvert$. Since
$x^{\prime}\in\mathrm{span}(\left\\{y\right\\})$, we have

$\displaystyle\lVert x-x^{\prime}\rVert_{2}^{2}=\lVert
x-\mathrm{Proj}_{y}(x)\rVert_{2}^{2}+\lVert\mathrm{Proj}_{y}(x)-x^{\prime}\rVert_{2}^{2}=\lVert
x-\mathrm{Proj}_{y}(x)\rVert_{2}^{2}+(\delta_{x}^{\prime}-\delta_{x})^{2}\lVert
y\rVert_{2}^{2}.$

On the other hand, since $\delta_{x}^{\prime}\leq 1$, we also have

$\displaystyle\lVert
x-\mathrm{Proj}_{y}(x)\rVert_{2}^{2}+(\delta_{x}^{\prime}-\delta_{x})^{2}\lVert
y\rVert_{2}^{2}\leq\lVert
x-\mathrm{Proj}_{y}(x)\rVert_{2}^{2}+(1-\delta_{x})^{2}\lVert
y\rVert_{2}^{2}=\lVert x-y\rVert_{2}^{2}.$

Hence, we obtain $\lVert x-x^{\prime}\rVert_{2}^{2}\leq\lVert
x-y\rVert_{2}^{2}$. In conclusion, since we have assumed $\lVert
x\rVert_{2}\leq\lVert y\rVert_{2}$, we obtain

$\displaystyle\lVert\tfrac{x}{\lVert x\rVert_{2}}-\tfrac{y}{\lVert
y\rVert_{2}}\rVert_{2}\leq\lVert x-y\rVert_{2}/\min\left\\{\lVert
x\rVert_{2},\lVert y\rVert_{2}\right\\}.$

The proof is then completed. ∎

The next lemma provides a sufficient condition the uniqueness of the worst-
case transition kernel.

######  Lemma B.4.

Suppose $\mathcal{P}_{s,a}$ is relatively strongly convex with respect to
$R_{s,a}>0$ for all $(s,a)\in{\mathcal{S}}\times\mathcal{A}$, and
$\mathrm{dim}(\mathcal{P}_{s,a})=\lvert{\mathcal{S}}\rvert-1$. Let
$r_{*}=\min_{\pi\in\Pi}\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\rVert_{2}$,
where
$\mathcal{H}=\mathrm{Lin}(\mathcal{P}_{s,a})=\left\\{x\in\mathbb{R}^{\lvert{\mathcal{S}}\rvert}:\mathbf{1}^{\top}x=0\right\\}$.
Suppose $r_{*}>0$, then for any policy $\pi\in\Pi$, the worst-case environment
$\mathbb{P}_{u_{\pi}}$ defined in (2.2) is unique, and

$\displaystyle\lVert\mathbb{P}_{u_{\pi}}(\cdot|s,a)-\mathbb{P}_{u_{\pi^{\prime}}}(\cdot|s,a)\rVert_{2}\leq\tfrac{R_{s,a}}{r_{*}}\lVert
V^{\pi}_{r}-V_{r}^{\pi^{\prime}}\rVert_{2},~{}~{}\forall(s,a)\in{\mathcal{S}}\times\mathcal{A}.$
(B.4)

The result of Lemma B.4 clearly hinges upon the quantity $r_{*}>0$. We will
provide a verifiable necessary and sufficient condition that certifies this
requirement.

###### Proof of Lemma B.4.

We first recall from (2.2) that for any policy $\pi\in\Pi$, the worse case
transition $\mathbb{P}_{u_{\pi}}$ is given by

$\displaystyle\mathbb{P}_{u_{\pi}}(\cdot|s,a)\in\mathop{\mathrm{argmax}}_{u(\cdot|s,a)\in\mathcal{U}_{s,a}}\sum_{s^{\prime}\in{\mathcal{S}}}\mathbb{P}_{u}(s^{\prime}|s,a)V^{\pi}_{r}(s^{\prime})=\mathop{\mathrm{argmax}}_{p\in\mathcal{P}_{s,a}}p^{\top}V^{\pi}_{r},~{}~{}\forall(s,a)\in{\mathcal{S}}\times\mathcal{A}.$
(B.5)

It is worth mentioning that the solution to (B.5) remains unchanged when we
shift $V^{\pi}_{r}$ by $\delta\mathbf{1}$ for any $\delta\in\mathbb{R}$, where
$\mathbf{1}\in\mathbb{R}^{\lvert{\mathcal{S}}\rvert}$ denotes the all-one
vector. To see this, note that

$\displaystyle
p^{\top}(V^{\pi}_{r}+\delta\mathbf{1})=p^{\top}V^{\pi}_{r}+\delta$ (B.6)

due to $p\in\mathcal{P}_{s,a}$, and hence shifting $V^{\pi}_{r}$ only changes
the objective by a constant $\delta$ for every feasible $p$. Let
$\mathrm{Proj}_{\mathcal{H}}:\mathbb{R}^{\lvert{\mathcal{S}}\rvert}\to\mathbb{R}^{\lvert{\mathcal{S}}\rvert}$
denote the projection operator onto $\mathcal{H}$, then given observation
(B.6), we must have

$\displaystyle\mathbb{P}_{u_{\pi}}(\cdot|s,a)\in\mathop{\mathrm{argmax}}_{p\in\mathcal{P}_{s,a}}p^{\top}\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r}),~{}~{}\forall(s,a)\in{\mathcal{S}}\times\mathcal{A},~{}\forall\pi\in\Pi.$
(B.7)

Let $\mathrm{ReBd}(\mathcal{P}_{s,a})$ denote the relative boundary of
$\mathcal{P}_{s,a}$. We claim that for any solution $p^{*}$ of (B.5) (or
equivalently (B.7)), we must have $p^{*}\in\mathrm{ReBd}(\mathcal{P}_{s,a})$.
Suppose not, and $p^{*}$ is in the relative interior of $\mathcal{P}_{s,a}$,
then there exists $\delta>0$ such that
$\mathcal{B}(p^{*},\delta)\cap\mathcal{H}\subset\mathcal{P}_{s,a}$. Now it is
immediate to see that
$p^{*}+\delta\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})/\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\rVert_{2}$
is a strictly better solution than $p^{*}$ unless
$\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})=\mathbf{0}$, which can not happen
since $r_{*}>0$.

We proceed to show that $p^{*}$ is indeed unique. Suppose not, and $p_{1}\neq
p_{2}\in\mathcal{P}_{s,a}$ are two solutions of (B.5). From the relative
strong convexity of set $\mathcal{P}_{s,a}$, we know that for any
$\lambda\in(0,1)$, we have $p_{\lambda}=\lambda p_{1}+(1-\lambda)p_{2}$ in the
relative interior of $\mathcal{P}_{s,a}$, which contradicts with the
previously established fact that any solution of (B.5) is in
$\mathrm{ReBd}(\mathcal{P}_{s,a})$.

By reading the optimality condition of the (B.7), we have

$\displaystyle(p-\mathbb{P}_{u_{\pi}}(\cdot|s,a))^{\top}\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\leq
0,~{}~{}\forall p\in\mathcal{P}_{s,a}.$ (B.8)

Hence we immediately see that
$\mathbb{P}_{u_{\pi}}(\cdot|s,a)\in\mathrm{ReBd}(\mathcal{P}_{s,a})$, and
$\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\in\mathcal{N}_{\mathcal{P}_{s,a}}(\mathbb{P}_{u_{\pi}}(\cdot|s,a))$.
Then we can apply Lemma B.1 and obtain

$\displaystyle\lVert\mathbb{P}_{u_{\pi}}(\cdot|s,a)-\mathbb{P}_{u_{\pi^{\prime}}}(\cdot|s,a)\rVert_{2}\leq
R_{s,a}\lVert\overline{V}^{\pi}_{r}-\overline{V}_{r}^{\pi^{\prime}}\rVert_{2},~{}~{}\forall(s,a)\in{\mathcal{S}}\times\mathcal{A},$

where
$\overline{V}^{\pi}_{r}=\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})/\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\rVert_{2},~{}\overline{V}^{\pi^{\prime}}_{r}=\mathrm{Proj}_{\mathcal{H}}(V^{\pi^{\prime}}_{r})/\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi^{\prime}}_{r})\rVert_{2}$.
Now applying Lemma B.3, we obtain

$\displaystyle\lVert\mathbb{P}_{u_{\pi}}(\cdot|s,a)-\mathbb{P}_{u_{\pi^{\prime}}}(\cdot|s,a)\rVert_{2}$
$\displaystyle\leq
R_{s,a}\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})-\mathrm{Proj}_{\mathcal{H}}(V^{\pi^{\prime}}_{r})\rVert_{2}/r_{*}$
$\displaystyle\leq\tfrac{R_{s,a}}{r_{*}}\lVert
V^{\pi}_{r}-V^{\pi^{\prime}}_{r}\rVert_{2},$

where
$r_{*}=\min_{\pi\in\Pi}\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\rVert_{2}$,
and the last inequality uses the non-expansiveness of the projection operator.
∎

It should be noted that the conditions on the relative strong convexity of
$\mathcal{P}_{s,a}$ and
$\mathrm{dim}(\mathcal{P}_{s,a})=\lvert{\mathcal{S}}\rvert-1$ are readily
satisfied by many common uncertainty sets. For instance, one can take
$\mathcal{P}_{s,a}$ as the intersection of a full-dimensional ellipsoid with
$\Delta_{{\mathcal{S}}}$ (i.e., ellipsoidal uncertainty [36]). On the other
hand, Lemma B.4 relies on the key condition that
$r_{*}=\min_{\pi\in\Pi}\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\rVert_{2}>0$.
We next provide a simple necessary and sufficient condition that certifies
$r_{*}>0$.

######  Lemma B.5.

Suppose the cost function $c$ satisfies
$\cap_{s\in{\mathcal{S}}}\left[\min_{a\in\mathcal{A}}c(s,a),\max_{a\in\mathcal{A}}c(s,a)\right]=\emptyset$,
then we must have
$r_{*}=\min_{\pi\in\Pi}\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\rVert_{2}>0$,
where
$\mathcal{H}=\left\\{x\in\mathbb{R}^{\lvert{\mathcal{S}}\rvert}:\mathbf{1}^{\top}x=0\right\\}$.
The converse of the statement is also true.

###### Proof.

“$\Rightarrow$”: Suppose the cost function satisfies $c$ satisfies
$\cap_{s\in{\mathcal{S}}}\left[\min_{a\in\mathcal{A}}c(s,a),\max_{a\in\mathcal{A}}c(s,a)\right]=\emptyset$,
but $r_{*}=0$. Then we have $V^{\pi}_{r}=\lambda\mathbf{1}$ for some
$\lambda>0$ (since $V^{\pi_{r}}>0$ must hold) and $\pi\in\Pi$. Let us define
$\mathbb{P}^{\pi}_{u_{\pi}}:{\mathcal{S}}\times{\mathcal{S}}\to[0,1]$ by
$\mathbb{P}^{\pi}_{u_{\pi}}(s,s^{\prime})=\sum_{a\in\mathcal{A}}\mathbb{P}_{u_{\pi}}(s^{\prime}|s,a)\pi(a|s)$,
and $c^{\pi}:{\mathcal{S}}\to\mathbb{R}$ by
$c^{\pi}(s)=\sum_{a\in\mathcal{A}}\pi(a|s)c(a|s)$, then given observation
(2.3) and the standard Bellman condition of $V^{\pi}_{u_{\pi}}$, we know that
$V^{\pi}_{r}=\left(I-\gamma\mathbb{P}^{\pi}_{u_{\pi}}\right)^{-1}c^{\pi},$
which gives

$\displaystyle
c^{\pi}=\left(I-\gamma\mathbb{P}^{\pi}_{u_{\pi}}\right)V^{\pi}_{r}=V^{\pi}_{r}-\gamma\mathbb{P}^{\pi}_{u_{\pi}}V^{\pi}_{r}\overset{(a)}{=}\lambda\mathbf{1}-\gamma\lambda\mathbf{1}=(1-\gamma)\lambda\mathbf{1},$
(B.9)

where in term $(a)$ we uses the fact that
$\mathbb{P}^{\pi}_{u_{\pi}}\mathbf{1}=\mathbf{1}$. However, (B.9) must
contradict with the condition that
$\cap_{s\in{\mathcal{S}}}\left[\min_{a\in\mathcal{A}}c(s,a),\max_{a\in\mathcal{A}}c(s,a)\right]=\emptyset$,
since for each $s\in{\mathcal{S}}$, we have
$c^{\pi}(s)\in\left[\min_{a\in\mathcal{A}}c(s,a),\max_{a\in\mathcal{A}}c(s,a)\right]$.

“$\Leftarrow$”: On the other hand, suppose $r_{*}>0$ but
$\cap_{s\in{\mathcal{S}}}\left[\min_{a\in\mathcal{A}}c(s,a),\max_{a\in\mathcal{A}}c(s,a)\right]\neq\emptyset$,
then one can readily find a policy $\pi$ such that $c^{\pi}=\lambda\mathbf{1}$
for some $\lambda\in\mathbb{R}$. Thus we have

$\displaystyle
V^{\pi}_{r}=\left(I-\gamma\mathbb{P}^{\pi}_{u_{\pi}}\right)^{-1}c^{\pi}\overset{(b)}{=}\sum_{t=0}^{\infty}\gamma^{t}\left(\mathbb{P}^{\pi}_{u_{\pi}}\right)^{t}c^{\pi}=\lambda\sum_{t=0}^{\infty}\gamma^{t}\mathbf{1}=\tfrac{\lambda}{1-\gamma}\mathbf{1},$

where in term $(b)$ we use the fact that $\rho(\gamma
P^{\pi}_{u_{\pi}})\leq\gamma$, where $\rho(A)$ denotes the spectral radius of
matrix $A$. Note that the previous observation then implies
$r_{*}\leq\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\rVert_{2}=0$. ∎

We proceed to show that $V^{\pi}_{r}(s)$ is Lipschitz continuous in $\pi$.

######  Lemma B.6.

For any policies $\pi,\pi^{\prime}\in\Pi$, we have

$\displaystyle\lVert
V^{\pi}_{r}-V^{\pi^{\prime}}_{r}\rVert_{\infty}\leq\tfrac{1}{1-\gamma}\lVert\pi-\pi^{\prime}\rVert_{\infty},$
(B.10)

where we define
$\lVert\pi-\pi^{\prime}\rVert_{\infty}=\sup_{s\in{\mathcal{S}}}\lVert\pi(\cdot|s)-\pi^{\prime}(\cdot|s)\rVert_{1}$,
i.e., the matrix $\ell_{\infty}$-norm when viewing $\pi$ as a matrix in
$\mathbb{R}^{\lvert{\mathcal{S}}\rvert\times\lvert\mathcal{A}\rvert}$.

###### Proof.

Note that for any $u\in\mathcal{U}$, from the performance difference lemma of
standard MDPs [13, 21], we have

$\displaystyle\lvert
V^{\pi}_{u}(s)-V^{\pi}_{u}(s)\rvert=\lvert\mathbb{E}_{s^{\prime}\sim
d_{s}^{\pi,u}}\left[\langle
Q^{\pi}_{u},\pi-\pi^{\prime}\rangle_{s^{\prime}}\right]\rvert\leq\tfrac{1}{1-\gamma}\lVert\pi-\pi^{\prime}\rVert_{\infty},$

where in the last inequality we use the Cauchy-Schwarz inequality, and the
fact that $\lVert Q^{\pi}_{u}\rVert_{\infty}\leq\tfrac{1}{1-\gamma}$. Hence we
obtain

$\displaystyle\lvert
V^{\pi}_{r}(s)-V^{\pi^{\prime}}_{r}(s)\rvert=\lvert\sup_{u\in\mathcal{U}}V^{\pi}_{u}(s)-\sup_{u\in\mathcal{U}}V^{\pi}_{u}(s)\rvert\leq\sup_{u\in\mathcal{U}}\lvert
V^{\pi}_{u}(s)-V^{\pi}_{u}(s)\rvert\leq\tfrac{1}{1-\gamma}\lVert\pi-\pi^{\prime}\rVert_{\infty}.$

∎

By combining Lemma B.6 and Lemma B.4, we are ready to establish the Lipschitz
continuity of $d^{\pi^{*},u_{\pi}}$ with respect to policy $\pi$.

######  Lemma B.7.

Suppose $\mathcal{P}_{s,a}$ is relatively strongly convex with respect to
$R_{s,a}>0$ for all $(s,a)\in{\mathcal{S}}\times\mathcal{A}$ with
$R=\max_{s\in{\mathcal{S}},a\in\mathcal{A}}R_{s,a}$, and
$\mathrm{dim}(\mathcal{P}_{s,a})=\lvert{\mathcal{S}}\rvert-1$. Let
$r_{*}=\min_{\pi\in\Pi}\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\rVert_{2}$,
where
$\mathcal{H}=\left\\{x\in\mathbb{R}^{\lvert{\mathcal{S}}\rvert}:\mathbf{1}^{\top}x=0\right\\}$.
If $r_{*}>0$, then we have

$\displaystyle\lVert
d^{\pi^{*},u_{\pi}}_{\rho}-d^{\pi^{*},u_{\pi^{\prime}}}_{\rho}\rVert_{1}\leq\tfrac{\gamma\lvert\mathcal{A}\rvert\lvert{\mathcal{S}}\rvert
R}{(1-\gamma)^{2}r_{*}}\lVert\pi-\pi^{\prime}\rVert_{\infty}.$ (B.11)

###### Proof.

Let us define $\mathbb{P}^{\pi}_{u}:{\mathcal{S}}\times{\mathcal{S}}\to[0,1]$
by
$\mathbb{P}^{\pi}_{u}(s^{\prime},s)=\sum_{a\in\mathcal{A}}\mathbb{P}_{u}(s^{\prime}|s,a)\pi(a|s)$,
then for any $\pi\in\Pi$, and $\rho\in\Delta_{{\mathcal{S}}}$, we obtain

$\displaystyle
d_{\rho}^{\pi,u}=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}(\mathbb{P}^{\pi}_{u})^{t}\rho=(1-\gamma)\left(1-\gamma\mathbb{P}^{\pi}_{u}\right)^{-1}\rho,$
(B.12)

where in the last equality we use the fact that
$\rho(\gamma\mathbb{P}^{\pi}_{u})\leq\gamma<1$. Hence we have

$\displaystyle d^{\pi^{*},u_{\pi}}_{\rho}-d^{\pi^{*},u_{\pi^{\prime}}}_{\rho}$
$\displaystyle=(1-\gamma)\left(\left(I-\gamma\mathbb{P}^{\pi^{*}}_{u_{\pi}}\right)^{-1}-\left(I-\gamma\mathbb{P}^{\pi^{*}}_{u_{\pi}^{\prime}}\right)^{-1}\right)\rho$
$\displaystyle=(1-\gamma)\gamma\left(I-\gamma\mathbb{P}^{\pi^{*}}_{u_{\pi}}\right)^{-1}\underbrace{\left(\mathbb{P}^{\pi^{*}}_{u_{\pi}}-\mathbb{P}^{\pi^{*}}_{u_{\pi}^{\prime}}\right)}_{\Delta^{\pi}_{\pi^{\prime}}}\left(I-\gamma\mathbb{P}^{\pi^{*}}_{u_{\pi}^{\prime}}\right)^{-1}\rho,$

where the last equality uses the matrix identity
$A^{-1}-B^{-1}=A^{-1}(B-A)B^{-1}$ for any invertible matrix $A,B$. Note that
$\lVert\left(I-\gamma\mathbb{P}^{\pi^{*}}_{u_{\pi}}\right)^{-1}\rVert_{1}\leq(1-\gamma)^{-1}$
as we have shown in (2.10), it suffices to control the $\ell_{1}$-norm of
$\Delta^{\pi}_{\pi^{\prime}}=\mathbb{P}^{\pi^{*}}_{u_{\pi}}-\mathbb{P}^{\pi^{*}}_{u_{\pi}^{\prime}}$.
We now make the following observations

$\displaystyle\sum_{s^{\prime}\in{\mathcal{S}}}\lvert\mathbb{P}^{\pi^{*}}_{u_{\pi}}(s^{\prime},s)-\mathbb{P}^{\pi^{*}}_{u_{\pi}^{\prime}}(s^{\prime},s)\rvert$
$\displaystyle=\sum_{s^{\prime}\in{\mathcal{S}}}\lvert\sum_{a\in\mathcal{A}}\left(\mathbb{P}_{u_{\pi}}(s^{\prime}|s,a)-\mathbb{P}_{u_{\pi^{\prime}}}(s^{\prime}|s,a)\right)\pi^{*}(a|s)\rvert$
$\displaystyle\leq\sum_{a\in\mathcal{A}}\sum_{s^{\prime}\in{\mathcal{S}}}\lvert\mathbb{P}_{u_{\pi}}(s^{\prime}|s,a)-\mathbb{P}_{u_{\pi^{\prime}}}(s^{\prime}|s,a)\rvert\pi^{*}(a|s)$
$\displaystyle\leq\sup_{a\in\mathcal{A}}\lVert\mathbb{P}_{u_{\pi}}(\cdot|s,a)-\mathbb{P}_{u_{\pi^{\prime}}}(\cdot|s,a)\rVert_{1}$
$\displaystyle\overset{(a)}{\leq}\tfrac{R\sqrt{\lvert{\mathcal{S}}\rvert}}{r_{*}}\lVert
V^{\pi}_{r}-V_{r}^{\pi^{\prime}}\rVert_{2}$
$\displaystyle\overset{(b)}{\leq}\tfrac{R\lvert{\mathcal{S}}\rvert}{r_{*}(1-\gamma)}\lVert\pi-\pi^{\prime}\rVert_{\infty}$

where $(a)$ uses Lemma B.4 and the definition of $R$, and $(b)$ uses Lemma
B.6, From the prior inequality, we obtain
$\lVert\Delta^{\pi}_{\pi^{\prime}}\rVert_{1}\leq\tfrac{R\lvert{\mathcal{S}}\rvert}{(1-\gamma)r_{*}}\lVert\pi-\pi^{\prime}\rVert_{\infty}$.
Putting everything together, we conclude that

$\displaystyle\lVert
d^{\pi^{*},u_{\pi}}_{\rho}-d^{\pi^{*},u_{\pi^{\prime}}}_{\rho}\rVert_{1}$
$\displaystyle\leq(1-\gamma)\gamma\lVert\left(I-\gamma\mathbb{P}^{\pi^{*}}_{u_{\pi}}\right)^{-1}\rVert_{1}\lVert\Delta^{\pi}_{\pi^{\prime}}\rVert_{1}\lVert\left(I-\gamma\mathbb{P}^{\pi^{*}}_{u_{\pi}^{\prime}}\right)^{-1}\rVert_{1}\lVert\rho\rVert_{1}$
$\displaystyle\leq\tfrac{\gamma\lvert{\mathcal{S}}\rvert
R}{(1-\gamma)^{2}r_{*}}\lVert\pi-\pi^{\prime}\rVert_{\infty}.$

The proof is then completed. ∎

Combining elements established above, we are now ready to establish the main
result of this subsection.

######  Theorem B.1.

Suppose $\mathcal{P}_{s,a}$ is relatively strongly convex with respect to
$R_{s,a}>0$ for all $(s,a)\in{\mathcal{S}}\times\mathcal{A}$ with
$R=\max_{s\in{\mathcal{S}},a\in\mathcal{A}}R_{s,a}$, and
$\mathrm{dim}(\mathcal{P}_{s,a})=\lvert{\mathcal{S}}\rvert-1$. Let
$r_{*}=\min_{\pi\in\Pi}\lVert\mathrm{Proj}_{\mathcal{H}}(V^{\pi}_{r})\rVert_{2}$,
where
$\mathcal{H}=\left\\{x\in\mathbb{R}^{\lvert{\mathcal{S}}\rvert}:\mathbf{1}^{\top}x=0\right\\}$,
and suppose $r_{*}>0$. For any $\eta>0$, let $\eta_{k}=\eta>0$ for all $k\geq
0$. If

  1. (1)

Distance-generating function $w$ is $\mu$-strongly convex with respect to
$\lVert\cdot\rVert_{1}$-norm;

  2. (2)

$D_{w}^{*}=\sup_{\pi\in\Pi}D^{\pi^{*}}_{\pi}(s)<\infty$.

Then RPMD outputs policy $\pi_{k}$ satisfying

$\displaystyle
f_{\rho}(\pi_{k})-f_{\rho}(\pi^{*})\leq\tfrac{M}{(1-\gamma)k}\left(f_{\rho}(\pi_{0})-f_{\rho}(\pi^{*})\right)+\tfrac{D_{w}}{(1-\gamma)\eta
k}+\tfrac{2\gamma\lvert{\mathcal{S}}\rvert^{2}RD_{w}^{*}}{(1-\gamma)^{7/2}r_{*}\sqrt{\eta\mu
k}},$ (B.13)

where $M$ is defined as in Lemma 3.2.

###### Proof.

By summing up inequality (3.5) from $t=0$ to $k-1$, we obtain

$\displaystyle
f_{\rho}(\pi_{k})-f_{\rho}(\pi^{*})+\tfrac{1-\gamma}{M}\sum_{t=1}^{k-1}\left(f_{\rho}(\pi_{t})-f_{\rho}(\pi^{*})\right)$
$\displaystyle\leq$ $\displaystyle
f_{\rho}(\pi_{0})-f_{\rho}(\pi^{*})+\tfrac{1}{M\eta_{0}}\mathbb{E}_{s\sim
d_{\rho}^{\pi^{*},u_{0}}}D^{\pi^{*}}_{\pi_{0}}(s)+\underbrace{\sum_{t=1}^{k-1}\left(\tfrac{1}{M\eta_{t}}\mathbb{E}_{s\sim
d_{\rho}^{\pi^{*},u_{t}}}D^{\pi^{*}}_{\pi_{t}}(s)-\tfrac{1}{M\eta_{t-1}}\mathbb{E}_{s\sim
d_{\rho}^{\pi^{*},u_{t-1}}}D^{\pi^{*}}_{\pi_{t}}(s)\right)}_{(A)}.$ (B.14)

We will make use of Lemma B.7 to bound term $(A)$. Specifically, we have

$\displaystyle\lvert\mathbb{E}_{s\sim
d_{\rho}^{\pi^{*},u_{t}}}D^{\pi^{*}}_{\pi_{t}}(s)-\mathbb{E}_{s\sim
d_{\rho}^{\pi^{*},u_{t-1}}}D^{\pi^{*}}_{\pi_{t}}(s)\rvert$
$\displaystyle\overset{(a)}{\leq}\lVert
d_{\rho}^{\pi^{*},u_{t}}-d_{\rho}^{\pi^{*},u_{t-1}}\rVert_{1}D_{w}^{*}$
$\displaystyle\overset{(b)}{\leq}\tfrac{\gamma\lvert{\mathcal{S}}\rvert
RD_{w}^{*}}{(1-\gamma)^{2}r_{*}}\lVert\pi_{t}-\pi_{t-1}\rVert_{\infty},$
(B.15)

where $(a)$ uses Holder’s inequality, and $(b)$ uses Lemma B.7, and the
definition that $u_{t}=u_{\pi_{t}}$ for all $t\geq 0$. Since $w(\cdot)$ is
$\mu$-strongly convex with respect to $\lVert\cdot\rVert_{1}$-norm, we have

$\displaystyle\lVert\pi_{t}-\pi_{t-1}\rVert_{\infty}=\sup_{s\in{\mathcal{S}}}\lVert\pi_{t}(\cdot|s)-\pi_{t-1}(\cdot|s)\rVert_{1}\leq\sup_{s\in{\mathcal{S}}}\sqrt{2D^{\pi_{t}}_{\pi_{t-1}}(s)/\mu},$

which combined with (B.14) and (B.15), gives

$\displaystyle
f_{\rho}(\pi_{k})-f_{\rho}(\pi^{*})+\tfrac{1-\gamma}{M}\sum_{t=1}^{k-1}\left(f_{\rho}(\pi_{t})-f_{\rho}(\pi^{*})\right)$
$\displaystyle\leq$ $\displaystyle
f_{\rho}(\pi_{0})-f_{\rho}(\pi^{*})+\tfrac{1}{M\eta}\mathbb{E}_{s\sim
d_{\rho}^{\pi^{*},u_{0}}}D^{\pi^{*}}_{\pi_{0}}(s)+\tfrac{\gamma\lvert{\mathcal{S}}\rvert
RD_{w}^{*}}{(1-\gamma)^{2}r_{*}M\eta}\sum_{t=1}^{k-1}\sum_{s\in{\mathcal{S}}}\sqrt{\tfrac{2D^{\pi_{t}}_{\pi_{t-1}}(s)}{\mu}}.$

It follows from Lemma 3.3 that

$\displaystyle\left(\sum_{t=1}^{k-1}\sqrt{D^{\pi_{t}}_{\pi_{t-1}}(s)}\right)^{2}\leq
k\sum_{t=1}^{k-1}D^{\pi_{t}}_{\pi_{t-1}}(s)\leq
k\eta\left(V^{\pi_{0}}_{r}(s)-V^{\pi^{*}}_{r}(s)\right)\leq\tfrac{k\eta}{1-\gamma},~{}~{}\forall
s\in{\mathcal{S}}.$

By combining two prior relations, we obtain

$\displaystyle
f_{\rho}(\pi_{k})-f_{\rho}(\pi^{*})+\tfrac{1-\gamma}{M}\sum_{t=1}^{k-1}\left(f_{\rho}(\pi_{t})-f_{\rho}(\pi^{*})\right)$
$\displaystyle\leq$ $\displaystyle
f_{\rho}(\pi_{0})-f_{\rho}(\pi^{*})+\tfrac{1}{M\eta}\mathbb{E}_{s\sim
d_{\rho}^{\pi^{*},u_{0}}}D^{\pi^{*}}_{\pi_{0}}(s)+\tfrac{2\gamma\lvert{\mathcal{S}}\rvert^{2}RD_{w}^{*}\sqrt{k}}{(1-\gamma)^{5/2}r_{*}M\sqrt{\eta\mu}}.$

By (3.8), we know that $f_{\rho}(\pi_{t+1})\leq f_{\rho}(\pi_{t})$ for all
$t\geq 0$. The desired claim then follows immediately after combining this
observation with the above inequality. ∎

In view of Theorem B.1, the RPMD method with constant stepsize $\eta$ outputs
an $\epsilon$-optimal policy within
$\mathcal{O}(\max\left\\{1/\epsilon,1/(\eta\epsilon^{2})\right\\})$
iterations, for a general class of Bregman divergences. Moreover, conditions
(1) and (2) in Theorem B.1 are both satisfied by some common distance-
generating functions, including the squared $\ell_{2}$-norm $w(p)=\lVert
p\rVert_{2}^{2}$, and the negative Tsallis entropy
$w(p)=\tfrac{k}{q-1}\left(\sum_{i}p_{i}^{q}-1\right)$ for any entropic-index
$q\in(1,2]$.

In addition, the second term of (B.13) also establishes a link on geometry of
the set of uncertain probabilities $\mathcal{P}_{s,a}$ to the convergence
rate. In addition, the dependence of convergence on the size of the state
space is largely due to Lipschitz constant of $d_{\rho}^{\pi^{*},u_{\pi}}$
with respect to policy $\pi$. The current characterization of such Lipschitz
constant seems improvable with additional efforts. Nevertheless, it is worth
mentioning that one can simply get rid of such a dependence by using a large
stepsize.