#  $q$-Munchausen Reinforcement Learning

Lingwei Zhu Nara Institute of Science and Technology, Japan  Zheng Chen  Osaka
University, Japan  Eiji Uchibe  Advanced Telecommunication Research, Japan
Takamitsu Matsubara  Nara Institute of Science and Technology, Japan

###### Abstract

The recently successful Munchausen Reinforcement Learning (M-RL) features
implicit Kullback-Leibler (KL) regularization by augmenting the reward
function with logarithm of the current stochastic policy. Though significant
improvement has been shown with the Boltzmann softmax policy, when the Tsallis
sparsemax policy is considered, the augmentation leads to a flat learning
curve for almost every problem considered. We show that it is due to the
mismatch between the conventional logarithm and the non-logarithmic
(generalized) nature of Tsallis entropy. Drawing inspiration from the Tsallis
statistics literature, we propose to correct the mismatch of M-RL with the
help of $q$-logarithm/exponential functions. The proposed formulation leads to
implicit Tsallis KL regularization under the maximum Tsallis entropy
framework. We show such formulation of M-RL again achieves superior
performance on benchmark problems and sheds light on more general M-RL with
various entropic indices $q$.

##  1 Introduction

Temporal difference (TD) learning nowadays encompasses a large number of
important reinforcement learning (RL) algorithms that attempt to learn the
optimal value function by bootstrapping the current estimate [Sutton and
Barto, 2018]. Recently proposed modifications on the TD loss often introduce
additional information metrics of policies, e.g. Shannon entropy of the
current policy for encouraging better exploration of the state-action space
[Ziebart, 2010]; or Kullback-Leibler (KL) divergence for penalizing deviation
from some given baseline policy [Azar et al., 2012, Fox et al., 2016]. Such
augmentation trades off optimality bias for benefits such as multimodal
optimal policy (Shannon entropy) [Haarnoja et al., 2017] or average of past
values (KL) [Vieillard et al., 2020a].

To avoid computing the recursively defined KL regularized optimal policy that
could incur large error, Munchausen RL (M-RL) [Vieillard et al., 2020b]
proposes to implicitly perform KL regularization by adding logarithm of the
current stochastic policy of any kind to the TD loss. Empirically, significant
improvement upon Deep Q-Network [Mnih et al., 2015] has been shown by adding
the log-softmax-policy, derived as the result of Shannon entropy augmentation.
M-RL is an exemplar method of implicit KL regularization: in this setting a
new action value function is defined and iterated upon, and the greedy policy
with respect to which resembles the true explicitly regularized one [Kozuno et
al., 2019].

Motivated by the claim that arbitrary stochastic policy could be used for the
Munchausen log-policy augmentation, in many scenarios it is desired to exploit
other stochastic but more _compact_ policies assigning probabilities only to a
subset of actions. Such policies are more well-suited to problems that require
safety or parsimony on action selection: e.g. some actions under certain
circumstances must have strictly zero probabilities of being selected. This
formulation naturally invites Tsallis sparse entropy [Lee et al., 2018, Chow
et al., 2018]. However, when Tsallis sparse policy is used in place of the
softmax in the log-policy augmentation, confusing flat learning curves emerge
for almost every problem considered, contradicting the general claim. In this
paper, we view the problem as a result of the mismatch between the
conventional logarithm and the non-logarithmic (generalized) nature of Tsallis
entropy [Tsallis, 1988]: the behavior of Tsallis entropy is better described
by the deformed $q$-logarithm/exponential functions rather than their standard
counterparts [Yamano, 2002].

To correct the mismatch, we propose to use the $q$-log-policy drawing
inspiration from the Tsallis statistics literature. Maximum Tsallis entropy
framework [Lee et al., 2020] is employed under which we show the Munchausen
augmentation leads to implicit Tsallis KL regularization [Furuichi et al.,
2004] by the help of $q$-logarithm and its $2-q$ duality [Naudts, 2002]. The
corrected formulation ensures the consistency between the non-logarithmic
Tsallis policies and the Munchausen augmentation, again leading to improved
performance as the original M-RL did.

The contribution of this paper is mainly theoretical in introducing the
Tsallis KL regularization to bridge the gap between the maximum Tsallis
entropy framework and implicit KL regularization put forward by M-RL. Related
work and background are provided in Section 2 and 3, respectively. Section 4
introduces the proposed method to achieving implicit Tsallis KL
regularization. We validate the proposed method by several proof-of-concept
experiments on benchmark problems in Section 5, and discuss possible future
directions in Section 6.

##  2 Related Work

Entropy Regularization.  Recent successful RL algorithms often feature the use
of entropy regularization, which refers to augmenting the reward function with
the Shannon entropy of policies as bonus [Ziebart, 2010, Haarnoja et al.,
2017, 2018], or penalizing deviating from some baseline policy (which is often
the previous policy) via KL divergence [Azar et al., 2012, Fox et al., 2016].
It is known that the Shannon entropy augmentation renders the optimal policy
stochastic and smoothens the optimization landscape [Ahmed et al., 2019], and
KL regularization groups the error by summation, which could lead to
asymptotic cancellation of errors if mild assumptions such as the sequence of
errors is martingale difference [Azar et al., 2012, Vieillard et al., 2020a].
This result testified to prior work that empirically averaged action values
and showed to substantially improve upon the vanilla Deep Q-Network (DQN)
[Anschel et al., 2017, Lan et al., 2020].

However, a KL regularized policy recursively depends on the previous policies
which poses a challenge for implementation [Geist et al., 2019]. Exactly
computing such recursively defined policy is intractable in practice since
prohibitive memory complexity will be incurred by the need of storing all
previous policies. On other hand, naïvely evaluating only one previous policy
as a surrogate for the recursive dependence typically results in large greedy
step errors and underperformance [Nachum et al., 2017, Vieillard et al.,
2020b]. Recently, there is a trend on implicitly performing KL regularization
so its properties can be enjoyed without needing to care about its recursive
nature [Vieillard et al., 2020b, c], in which Munchausen DQN (MDQN) was one of
the state-of-the-art method, achieving comparable performance with
distributional methods. However, different with the claim that any stochastic
policy could be used, MDQN with Tsallis entropy induced policies showed flat
learning curves for even very simple control problems. In this paper, we show
that it is because of the mismatch between the logarithm and the non-
logarithmic Tsallis entropy.

Tsallis Statistics.  In the field of statistical mechanics, it is well known
that a wide range of phenomena in nature such as chaos or fractals can be
successfully described by power-law theories rather than the traditional
Gibbs-Boltzmann statistics [Suyari and Tsukada, 2005]. Tsallis entropy has
hence been proposed [Tsallis, 1988] and successfully applied to various
problems of which power law was of concern. Tsallis entropy can be seen as a
generalization of Shannon entropy, described by deformed logarithm and
exponential functions [Naudts, 2002, Yamano, 2002].

In RL, the introduction of Tsallis entropy was quite recent, benefited greatly
from the sparsity-inducing Tsallis sparse policy [Martins and Astudillo, 2016,
Lee et al., 2018, Chow et al., 2018]. It turns out that the Tsallis sparse
entropy and Shannon entropy are both special cases of the maximum general
Tsallis entropy framework considered in [Lee et al., 2020]. However, maximum
Tsallis entropy methods (excluding Shannon entropy) often suffer from various
errors: the closed-form sparsemax policy is inherently more sensitive to
errors than softmax [Lee et al., 2020], while other entropies do not possess
closed-form policy and entail approximation which could incur significant
greedy step error. Motivated by the error cancellation effect of KL, we
introduce Tsallis KL (relative entropy) regularization into the maximum
Tsallis entropy framework [Furuichi et al., 2004, Ohara, 2007] which has not
seen published results to the best of the authors’ knowledge.

##  3 Background

###  3.1 Reinforcement Learning

We focus on discrete-time discounted Markov Decision Processes (MDPs) which
are expressed by the quintuple $(\mathcal{S},\mathcal{A},P,r,\gamma)$, where
$\mathcal{S}$ and $\mathcal{A}$ denote state space and finite action space,
respectively. $P(\cdot|s,a)$ denotes transition probability over the state
space given state-action pair $(s,a)$, and $r(s,a)\in[0,r_{max}]$ defines the
reward associated with that transition. When time step $t$ is of concern, we
write $r_{t}:=r(s_{t},a_{t})$. $\gamma\in(0,1)$ is the discount factor. A
policy $\pi(\cdot|s)$ is a mapping from the state space to distributions over
actions. For control purposes, we define the state-action value function
following policy $\pi$ as
$Q_{\pi}\in\mathbb{R}^{|\mathcal{S}|\times|\mathcal{A}|}$:

$\displaystyle
Q_{\pi}(s,a)=\mathbb{E}\left[\sum_{t=0}^{\infty}r_{t}|s_{0}=s,a_{0}=a\right],$
(1)

where the expectation is with respect to the policy $\pi$ and transition
probability.

For notational convenience, we define the inner product for any two functions
$F_{1},F_{2}\in\mathbb{R}^{|\mathcal{S}|\times|\mathcal{A}|}$ as $\left\langle
F_{1},F_{2}\right\rangle\in\mathbb{R}^{|\mathcal{S}|}$. $\boldsymbol{1}$
denotes an all-one vector whose dimension should be clear from the context.
With the above notations, the Bellman operator acting upon any function
$Q\\!\in\\!\mathbb{R}^{|\mathcal{S}|\times|\mathcal{A}|}$ can be defined as:
$T_{\pi}Q:=r+\gamma P_{\pi}Q$, where
$P_{\pi}Q\\!\in\\!\mathbb{R}^{|\mathcal{S}|\times|\mathcal{A}|}\\!:=\\!\mathbb{E}_{s^{\prime}\sim
P(\cdot|s,a),a^{\prime}\sim\pi(\cdot|s^{\prime})}\\!\left[Q(s^{\prime},a^{\prime})\right]\\!=\\!P\\!\left\langle\pi,Q\right\rangle$,
and the definition should be understood as component-wise. Repeatedly applying
the Bellman operator renders $Q$ converges to the unique fixed point
$Q_{\pi}$, which also includes the optimal action value function
$Q_{*}:=Q_{\pi^{*}}$ [Bertsekas and Tsitsiklis, 1996]. In this paper we focus
on value iteration methods that perform the following loop to obtain the
optimal policy:

$\displaystyle\begin{cases}\pi_{k+1}=\operatorname*{arg\,max}_{\pi}\\!\left\langle\pi,Q_{k}\right\rangle,&\\\
Q_{k+1}=r+\gamma P\\!\left\langle\pi_{k+1},Q_{k}\right\rangle.&\end{cases}$
(2)

In practice, one often parametrizes $Q$ functions by weight vectors $\theta$
and optimizes the TD loss
$Q_{\theta}-r-\gamma\left\langle\pi,Q_{\bar{\theta}}\right\rangle$ to obtain
the new action value, where $\bar{\theta}$ denotes a fixed target of $\theta$
[Mnih et al., 2015].

###  3.2 Entropy Regularization

Recent RL algorithms often feature the trade-off between faster convergence
and optimality bias introduced by information metrics. Specifically, Shannon
entropy, KL divergence and Tsallis entropy are among the most employed,
respectively defined as
$\mathcal{H}\left(\pi\right)\\!:=\\!\left\langle-\pi,\ln\pi\right\rangle$,
$D_{\\!K\\!L}\\!\left(\pi||\bar{\pi}\right)\\!:=\\!\left\langle\pi,\ln\pi-\ln\bar{\pi}\right\rangle$
and
$S_{q}(\pi)\\!:=\\!\frac{k}{q-1}\left(1-\left\langle\boldsymbol{1},\pi^{q}\right\rangle\right)$.
Note that $S_{q}(\pi)$ converges to $\mathcal{H}\left(\pi\right)$ when
$k\\!=\\!\frac{1}{2},q\rightarrow 1$. In this paper we consider the case
$k\\!=\\!\frac{1}{2},q\\!=\\!2$ for $S_{q}(\pi)$ which corresponds to the
Tsallis sparse entropy
$S_{2}(\pi)\\!:=\\!\frac{1}{2}\\!\left\langle\pi,(1-\pi)\right\rangle$ [Lee et
al., 2018, Chow et al., 2018]. Let us associate a coefficient $\tau$ to
$\mathcal{H}\left(\pi\right)$, we can then succinctly write the soft value
iteration as:

$\displaystyle\begin{cases}\pi_{k+1}=\operatorname*{arg\,max}_{\pi}\\!\left\langle\pi,Q_{k}-\tau\ln\pi\right\rangle,&\\\
Q_{k+1}=r+\gamma
P\\!\left\langle\pi_{k+1},Q_{k}-\tau\ln\pi_{k+1}\right\rangle.&\end{cases}$
(3)

It is well known that $\pi_{k+1}=\exp\left(\tau^{-1}(Q_{k}-V_{k})\right)$,
where
$V_{k}\\!=\\!\tau\log\left\langle\boldsymbol{1},\tau^{-1}Q_{k}\right\rangle$
is the soft value function [Haarnoja et al., 2017]. If we replace
$\mathcal{H}\left(\pi\right)$ with $S_{2}(\pi)$ in Eq. (3), the greedy policy
becomes a _sparsemax_ policy:
$\pi_{k+1}\\!=\\!\left[\frac{Q_{k}}{\tau}-\psi\left(\frac{Q_{k}}{\tau}\right)\right]_{+}$,
where $[\cdot]_{+}=\max\\{\cdot,0\\}$, and $\psi$ is the normalization
ensuring the policy sums to 1:

$\displaystyle\begin{split}\psi\left(\frac{Q(s,\cdot)}{\tau}\right)=\frac{\sum_{a\in
S(s)}\frac{Q(s,a)}{\tau}-1}{|S{(s)}|},\end{split}$ (4)

where $S(s)$ is the set of actions satisfying
$1\\!+\\!i\frac{Q(s,a_{(i)})}{\tau}\\!>\\!\sum_{j=1}^{i}\frac{Q(s,a_{(j)})}{\tau}$,
$a_{(j)}$ indicates the action with $j$th largest action value, $|S(s)|$
denotes the cardinality of $S(s)$. It is hence clear that Tsallis entropy
augmentation renders policies sparse by truncating actions $a^{\prime}$ that
have lower values than the normalization
$\frac{Q(s,a^{\prime})}{\tau}<\psi\left(\frac{Q(s,a^{\prime})}{\tau}\right)$.
This truncation is shown in the bottom of Figure 2. In the rest of the paper,
we shall respectively call the Boltzmann softmax and Tsallis sparsemax policy
as softmax and sparsemax policy in short. We also term MDQN with sparsemax
augmentation as log-sparsemax MDQN.

###  3.3 Munchausen RL

The concept of M-RL can be best described by that the authors advocate for
“optimizing for the immediate reward augmented by the scaled log-policy of the
agent when using any TD scheme” [Vieillard et al., 2020b, p.1]. As a concrete
instantiation, M-RL is implemented via Munchausen Deep Q-Netwrok (MDQN):

$\displaystyle\begin{split}\begin{cases}\pi_{k+1}=\operatorname*{arg\,max}_{\pi}\left\langle\pi,Q_{k}\right\rangle+\tau\mathcal{H}\left(\pi\right)\\\
Q_{k+1}=r+{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\alpha\tau\ln\pi_{k+1}}}+\gamma
P\left\langle\pi_{k+1},Q_{k}-{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\tau\ln\pi_{k+1}}\right\rangle,\end{cases}\end{split}$
(5)

where the blue term comes from that M-RL requires a stochastic policy and
hence Shannon entropy augmentation is employed. The red term is the Munchausen
log-policy term, with $\alpha$ an additional coefficient.

The evaluation step of Eq. (5) has the following equivalence:

$\displaystyle\begin{split}&Q_{k+1}\\!-\\!{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\alpha\tau\ln\pi_{k+1}}}=\\\
&r\\!+\\!\gamma
P\\!\left\langle\\!\pi_{k+1},Q_{k}\\!-\\!{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\alpha\tau\ln\frac{\pi_{k+1}}{\pi_{k}}}\\!-\\!\alpha\tau\ln\pi_{k}\\!-\\!{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}(1\\!-\\!\alpha)\tau\ln\pi_{k+1}}\\!\\!\right\rangle\\\
&=r+\gamma
P\big{(}\\!\left\langle\pi_{k+1},Q_{k}-\alpha\tau\ln\pi_{k}\right\rangle-\\\
&\qquad\qquad\qquad\quad\alpha\tau
D_{\\!K\\!L}\\!\left(\pi_{k+1}||\pi_{k}\right)+(1-\alpha)\tau\mathcal{H}\left(\pi_{k+1}\right)\\!\big{)}.\end{split}$
(6)

By defining a new action value function
$Q^{\prime}_{k+1}:=Q_{k+1}\\!-\\!\alpha\tau\ln\pi_{k+1}$, we see that the
augmentation of $\alpha\tau\ln\pi_{k+1}$ led to implicit KL regularization.

##  4 Tsallis Munchausen RL

###  4.1 Logarithm Sparsemax Policy

Figure 1: MDQN with the standard $\ln\pi$ augmentation, with $\pi$ being the
Tsallis sparsemax policy induced by Tsallis sparse policy $S_{2}(\pi)$. Mean
and $\pm 1$ standard deviation are averaged over 50 independent trials.

While MDQN was shown to perform well with Shannon entropy, it exhibits an
almost flat learning curve for even simple benchmark problems such as
CartPole-v1 in the classic control library [Brockman et al., 2016], as can be
seen from Figure 1.

The failure of MDQN in solving the task can be interpreted from two
viewpoints: advantage and KL regularization. In the advantage perspective, one
may consider the Boltzmann policy _encapsulates_ information of the action gap
[Farahmand, 2011] since $\pi_{k+1}=\exp\left(\tau^{-1}(Q_{k}-V_{k})\right)$,
and the logarithm can hence be regarded as _decapsulating_ it. When the
Tsallis policy is adopted, the logarithm fails to decipher the information
contained in the power-law Tsallis entropy.

The KL regularization perspective is intuitive in that the equivalence in Eq.
(6) no longer holds with Tsallis entropy augmentation $\tau
S_{2}(\pi_{k+1})\\!=\\!-\tau\left\langle\pi_{k+1},(1-\pi_{k+1})\right\rangle$
($\tau$ absorbs the constant $\frac{1}{2}$). Since no logarithm term appears,
it is clear that implicit KL regularization can no longer be performed. It is
hence desired that the Tsallis entropy be written in logarithms to match the
derivation in Eq. (6):

$\displaystyle Q_{k+1}=r+\alpha\tau\ln_{q}\pi_{k+1}+\gamma
P\left\langle\pi_{k+1},Q_{k}-\tau\ln_{q}\pi_{k+1}\right\rangle,$ (7)

where $\ln_{q}\pi_{k+1}$ is the assumed logarithm function capable of
representing $(1-\pi_{k+1})$. There indeed exists such logarithm functions by
drawing inspiration from the Tsallis statistics literature, by which we can
describe the general Tsallis entropy $S_{q}(\pi)$ via the deformed
$q$-logarithm. We introduce it in the next sub-section.

###  4.2 $q$-logarithm

Tsallis entropy $S_{q}(\pi)$ has been heavily exploited in statistical physics
to describe phenomena that exhibit power-law behavior. To see it is a
generalization of Shannon entropy, $S_{q}(\pi)$ can be defined in terms of the
deformed $q$-logarithm as follows:

$\displaystyle
S_{q}(\pi)=\left\langle-\pi^{q},\ln_{q}\pi\right\rangle,\quad\ln_{q}\pi:=\begin{cases}\ln\pi&\text{if
}q=1\\\ \frac{\pi^{1-q}-1}{1-q}&\text{if }q\neq 1.\end{cases}$ (8)

$S_{q}(\pi)$ and $q$-logarithm have the following properties:

  * •

Convexity.  $S_{q}(\pi)$ is convex for $q\leq 0$, concave for $q\geq 0$, and
is a linear function for $q=0$.

  * •

Boundedness.  $0\leq S_{q}(\pi)\leq\ln_{q}|\mathcal{A}|$ for all $q$, where
$|\mathcal{A}|$ denotes the cardinality of the action set.

  * •

Monotonicity.  $\ln_{q}\pi$ is monotonically increasing with respect to $\pi$.

  * •

Pseudo-additivity.  $q$-logarithm satisfies:

$\displaystyle\ln_{q}{\pi\mu}=\ln_{q}\pi+\ln_{q}\mu+(1-q)\ln_{q}\pi\ln_{q}\mu.$
(9)

Here, convexity ensures the regularization properties analyzed in [Geist et
al., 2019, Li et al., 2019] hold. Boundedness allows for derivation of lower-
and upper-bounds on the optimal action values [Lee et al., 2018, 2020].

Maximizing the general Tsallis entropy $S_{q}(\pi)$ in RL has been considered
in [Lee et al., 2020]. However, taking one step further, employing general
Tsallis KL regularization (Tsallis relative entropy) has never seen published
results to the best of authors’ knowledge. Nonetheless, we believe its
introduction into RL is necessary for rendering MDQN compatible with Tsallis
entropy. Moreover, (generalized) KL divergence is fundamental in the sense
that it produces the entropy and the mutual information as special cases
[Furuichi et al., 2004].

With the help of $q$-logarithm , one might hope to correct the mismatch in Eq.
(7) by modifying the Munchausen term to its $\ln_{q}$ version, hence recover
implicit Tsallis KL regularization. However, this point is not straightforward
as the result of the pseudo-additivity Eq. (9):

$\displaystyle\ln_{q}\\!\frac{\pi_{k+1}}{\pi_{k}}$
$\displaystyle=\ln_{q}\pi_{k+1}\cdot\frac{1}{\pi_{k}}$
$\displaystyle=\ln_{q}\pi_{k+1}+\ln_{q}\frac{1}{\pi_{k}}+(1-q)\ln_{q}\pi_{k+1}\ln_{q}\frac{1}{\pi_{k}}.$

The term $\ln_{q}\\!\frac{\pi_{k+1}}{\pi_{k}}$ was crucial in forming the
implicit KL regularization in Eq. (6). Unfortunately for $q$-logarithm there
is a residual $(q-1)\ln_{q}\pi_{k+1}\ln_{q}\frac{1}{\pi_{k}}$, which destroys
the equivalence required. Furthermore, the residual involves both the current
policy $\pi_{k+1}$ and the base one $\pi_{k}$, hence one cannot define a new
action value same as $Q^{\prime}_{k+1}$.

Fortunately, by resorting to the $2-q$ duality from the Tsallis statistics
literature, we can define Tsallis KL divergence in another way, rendering
implicit Tsallis KL regularization possible.

###  4.3 Tsallis KL Regularization

Figure 2: (Top) Illustration of KL and Tsallis KL divergence components
$\pi_{1}\ln\frac{\pi_{1}}{\pi_{2}}$ and
$-\pi_{1}\ln_{q}{\frac{\pi_{2}}{\pi_{1}}}$ between Boltzmann policies and
Gaussian policies when $q=2$. (Bottom) Illustration of sparsemax acting upon
$\pi_{1}$ by truncating actions that have values lower than $\psi$ defined in
Eq. (4).

Before introducing the solution, we need a few definitions. For any two
policies $\pi$ and $\mu$, we define their Tsallis KL divergence as:

$\displaystyle
D^{q}_{\\!K\\!L}\\!\left(\pi\left|\\!\right|\mu\right):=-\left\langle\pi,\ln_{q}\frac{\mu}{\pi}\right\rangle,$
(10)

where it reduces to KL divergence when $q\rightarrow 1$. Examples of KL and
Tsallis KL of the frequently used Boltzmann and Gaussian policies are shown in
Figure 2. Tsallis KL divergence possesses the following properties similar to
that of KL [Furuichi et al., 2004]:

  * •

Nonnegativity.  $D^{q}_{\\!K\\!L}\\!\left(\pi\left|\\!\right|\mu\right)\geq
0$.

  * •

Convexity.  For $0\leq\lambda\leq 1$ and $q\geq 0$, consider policies
$\pi_{i},\mu_{i},i=1,2$ and their interpolation
$\tilde{\pi}:=\lambda\pi_{1}+(1-\lambda)\pi_{2}$,
$\tilde{\mu}:=\lambda\mu_{1}+(1-\lambda)\mu_{2}$, we have

$\displaystyle
D^{q}_{\\!K\\!L}\\!\left(\tilde{\pi}\left|\\!\right|\tilde{\mu}\right)\leq\lambda
D^{q}_{\\!K\\!L}\\!\left(\pi_{1}\left|\\!\right|\mu_{1}\right)+(1-\lambda)D^{q}_{\\!K\\!L}\\!\left(\pi_{2}\left|\\!\right|\mu_{2}\right).$
(11)

The above properties hold for both KL and Tsallis KL divergences, hence we can
expect their regularization have similar characteristics. For example, it
might be possible to replace the theoretical results [Kozuno et al., 2019,
Vieillard et al., 2020a] with their $q$-logarithm version.

The first step of deriving implicit Tsallis KL regularization is to rewrite
$S_{q}(\pi_{k+1})$ into a compatible form with the inner product
$\left\langle\pi_{k+1},Q_{k}-\tau\ln_{q^{*}}\\!\pi_{k+1}\right\rangle$, where
$q^{*}$ is some dual entropic index of $q$: since the original definition
$S_{q}(\pi)=\left\langle-\pi_{k+1}^{q},\ln_{q}\pi_{k+1}\right\rangle$ has
power expectation which does not easily translate to the regularization scheme
in Eq. (3). To this end, we adopt the $q^{*}=2-q$ duality [Naudts, 2002] which
has the equivalence [Suyari and Tsukada, 2005]:

$\displaystyle
S_{q}(\pi_{k+1})=\left\langle-\pi_{k+1}^{q},\ln_{q}\pi_{k+1}\right\rangle=\left\langle-\pi_{k+1},\ln_{q^{*}}\\!\pi_{k+1}\right\rangle,$
(12)

Eq. (12) defined a dual $q^{*}$-logarithm which is consistent with the one
used in recent RL literature [Lee et al., 2020]. The properties of Eq. (8)
still hold with by replacing $q=2-q^{*}$.

The second step concerns expressing Tsallis KL divergence by $q^{*}$-logarithm
without the residual term. By the $q$-power expectation formulation of Tsallis
KL [Ohara, 2007] we have

$\displaystyle
D^{q}_{\\!K\\!L}\\!\left(\pi_{k+1}\left|\\!\right|\pi_{k}\right):=-\left\langle\\!\pi_{k+1},\ln_{q}\frac{\pi_{k}}{\pi_{k+1}}\\!\right\rangle$
$\displaystyle=\left\langle\pi_{k+1}^{q},\ln_{q}\pi_{k+1}-\ln_{q}\pi_{k}\right\rangle=\left\langle\pi_{k+1}^{q},\frac{\pi_{k+1}^{1-q}-\pi_{k}^{1-q}}{1-q}\right\rangle.$

We now show the equivalence of
$\left\langle\pi_{k+1}^{q},\ln_{q}\pi_{k+1}-\ln_{q}\pi_{k}\right\rangle$ and
$\left\langle\pi_{k+1},\ln_{q^{*}}\\!\pi_{k+1}-\ln_{q^{*}}\\!\pi_{k}\right\rangle$
by exploiting the $2-q$ duality for the $S_{q}(\pi)$ [Suyari and Tsukada,
2005, Eq.(12)]:

$\displaystyle\begin{split}&\left\langle\pi_{k+1},\ln_{q^{*}}\\!\pi_{k+1}\\!-\\!\ln_{q^{*}}\\!\pi_{k}\right\rangle\\!=\\!\left\langle\\!\\!\pi_{k+1},\frac{\pi_{k+1}^{q^{*}-1}-1}{q^{*}-1}\\!-\\!\frac{\pi_{k}^{q^{*}-1}-1}{q^{*}-1}\\!\right\rangle\\\
&=\left\langle\\!\pi_{k+1},\frac{\pi_{k+1}^{q^{*}-1}-\pi_{k}^{q^{*}-1}}{q^{*}-1}\\!\right\rangle=\left\langle\\!-\pi_{k+1},\frac{\left(\frac{\pi_{k}}{\pi_{k+1}}\right)^{q^{*}-1}\\!-1}{\pi_{k+1}^{1-q^{*}}(q^{*}-1)}\\!\right\rangle\\\
&=\left\langle\\!-\pi_{k+1}^{2-q^{*}},\frac{\left(\frac{\pi_{k}}{\pi_{k+1}}\right)^{q^{*}-1}\\!-1}{q^{*}-1}\\!\right\rangle\\!=\\!\left\langle\\!\\!-\pi_{k+1}^{q},\frac{\left(\frac{\pi_{k}}{\pi_{k+1}}\right)^{1-q}\\!-1}{1-q}\\!\right\rangle\\\
&=D^{q}_{\\!K\\!L}\\!\left(\pi_{k+1}||\pi_{k}\right),\end{split}$ (13)

Given the above equation, we can now reformulate the Tsallis version of MDQN
as:

$\displaystyle\begin{split}&Q_{k+1}=r+{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\alpha\tau\ln_{q^{*}}\\!\pi_{k+1}}+\gamma
P\left\langle\pi_{k+1},Q_{k}-{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\tau\ln_{q^{*}}\\!\pi_{k+1}}\right\rangle,\\\
&\Leftrightarrow
Q_{k+1}-\alpha\tau\ln_{q^{*}}\\!\pi_{k+1}=\left\langle\pi_{k+1},Q_{k}-\alpha\tau\ln_{q^{*}}\\!\pi_{k}\right\rangle-\\\
&\qquad\left\langle\pi_{k+1},\alpha\tau(\ln_{q^{*}}\\!\pi_{k+1}-\ln_{q^{*}}\\!\pi_{k})-(1-\alpha)\tau\ln_{q^{*}}\\!\pi_{k+1}\right\rangle,\\\
&\Leftrightarrow Q^{\prime\prime}_{k+1}=\\\ &r\\!+\\!\gamma
P\\!\left\langle\pi_{k+1},Q^{\prime\prime}_{k}\right\rangle\\!-\\!\alpha\tau
D^{q}_{\\!K\\!L}\\!\left(\pi_{k+1}||\pi_{k}\right)\\!+\\!(1-\alpha)\tau
S_{q}(\pi_{k+1}),\end{split}$ (14)

where
$Q^{\prime\prime}_{k+1}\\!:=\\!Q_{k+1}-\alpha\tau\ln_{q^{*}}\\!\pi_{k+1}$ is
the new action value function. Hence it is clear that Eq. (14) generalizes
MDQN in Eq. (6) by letting $q=1$.

###  4.4 Tsallis Policies

Similar with the exponential policy form induced by Shannon entropy,
$q^{*}$-logarithm induces $q^{*}$-exponential $\exp_{q^{*}}\\!$ as its inverse
function:

$\displaystyle\exp_{q^{*}}\\!(Q)=\begin{cases}\exp(Q),&\text{if }q=1\\\
\left[1+(q^{*}-1)Q\right]^{\frac{1}{q^{*}-1}}_{+},&\text{if }q\neq
1.\end{cases}$ (15)

$q^{*}$-exponential has been exploited in [Lee et al., 2020] to yield the
maximum Tsallis entropy policies as
$\pi^{*}_{q^{*}}\\!=\\!\exp_{q^{*}}\\!(Q/q^{*}-\psi_{q^{*}}(Q)/q^{*})$, where
$\psi_{q^{*}}$ is the normalization term depending on specific choices of the
dual entropic index $q^{*}$. When $q^{*}\neq 1,2,\infty$, there might be no
closed-form expression for $\psi_{q^{*}}$ and hence the policy
$\pi^{*}_{q^{*}}$.

Motivated by the property of KL regularized policy
$\pi_{k+1}\\!\propto\\!\pi_{k}\exp(Q_{k})\\!\propto\\!\pi_{k-1}\exp(Q_{k}+Q_{k-1})\\!\propto\\!\dots\\!\propto\\!\exp(\sum_{j=1}^{k}Q_{j}),$
where we drop the regularization coefficient $\tau$ for convenience, it is
natural to expect the Tsallis KL regularized policies possess a similar
averaging effect. This indeeds holds for the $q^{*}$-exponential since we have
[Yamano, 2002]:

$\displaystyle\begin{split}\exp_{q^{*}}\\!\left(\sum_{j=1}^{k}Q_{j}\right)^{q^{*}-1}&\\!\\!\\!\\!\\!\\!=\left(\exp_{q^{*}}\\!Q_{1}\dots\exp_{q^{*}}\\!Q_{k}\right)^{q^{*}-1}\\!\\!\\\
&+\sum_{j=2}^{k}(q^{*}-1)^{j}\\!\\!\\!\sum_{i_{1}<\dots<i_{j}}^{k}\\!Q_{i_{1}}\dots
Q_{i_{j}},\end{split}$ (16)

When $q^{*}\\!=\\!2$, Eq. (16) reduces to:

$\displaystyle\exp_{q^{*}}\\!\left(\sum_{j=1}^{k}Q_{j}\right)\\!\\!\\!=\exp_{q^{*}}\\!Q_{1}\dots\exp_{q^{*}}\\!Q_{k}\\!+\\!\\!\\!\\!\\!\\!\sum_{\begin{subarray}{c}j=2\\\
i_{1}<\dots<i_{j}\end{subarray}}^{k}\\!\\!\\!\\!Q_{i_{1}}\dots Q_{i_{j}},$

in which the residual term
$\sum_{\begin{subarray}{c}i_{1}<\dots<i_{j}\end{subarray}}^{k}Q_{i_{1}}\dots
Q_{i_{j}}$ can be regarded as a special case of adaptively weighting the
contribution of each $Q_{j}$ function rather than the uniform weighting. One
can also define the $q^{*}$-product $\times_{q^{*}}$ [Suyari and Tsukada,
2005, Eq.(29)] so that
$\exp_{q^{*}}\\!Q_{1}\times_{q^{*}}\exp_{q^{*}}\\!Q_{2}=\exp_{q^{*}}\\!(Q_{1}+Q_{2})$.

###  4.5 Practical Implementation

Input: total number of steps $T$, update period $I$,

interaction period $C$, epsilon-greedy threshold $\epsilon$,

entropic index $q^{*}$, entropy coefficients $\alpha,\tau$;

Initialize: Network weights $\bar{\theta}=\theta$;

Replay buffer $B=\\{\\}$;

1 for  _$t=1,2,\dots,T$ _ do

2 Collect tuple $(s_{t},a_{t},r_{t},s_{t+1})$ in $\mathcal{B}$ with
$\pi_{q^{*},\epsilon}$

3 if  _mod $(t,C)=0$_ then

4 sample a minibatch $\mathcal{B}_{t}\subset\mathcal{B}$

5 compute $Q_{\bar{\theta}},\pi_{\bar{\theta}}$ using target network
$\bar{\theta}$

6 compute
$\ln_{q^{*}}\\!\pi_{\bar{\theta}}=\frac{\pi_{\bar{\theta}}^{q^{*}-1}-1}{q^{*}-1}$

7 update $\theta$ with one step of SGD on the loss $\mathcal{L}$

8

9 end if

10 if  _mod $(t,I)=0$_ then

11 update target network $\bar{\theta}\leftarrow\theta$

12

13 end if

14

15 end for

Algorithm 1 Tsallis Entropy Munchausen DQN

Given Eq. (14), we detail our proposed algorithm: Tsallis Entropy Munchausen
DQN (TEMDQN) by modifying the TD loss of MDQN. Specifically, we replace every
appearance of $\ln\pi_{k+1}$ in Eq. (5) by $q^{*}$-logarithm
$\ln_{q^{*}}\\!\pi_{k+1}$, which leads to the following loss function:

$\mathcal{L}(\theta)\\!:=\\!\hat{\mathbb{E}}_{\mathcal{B}}\\!\left[\\!\left(r\\!+\\!\alpha\tau\ln_{q^{*}}\\!\pi_{\bar{\theta}}\\!+\\!\gamma\left\langle\pi_{\bar{\theta}},Q_{\bar{\theta}}\\!-\\!\tau\ln_{q^{*}}\\!\pi_{\bar{\theta}}\right\rangle\\!-\\!Q_{\theta}\right)^{2}\\!\right],$

where $\hat{\mathbb{E}}_{\mathcal{B}}$ denotes empirical expectation w.r.t. to
the replay buffer $\mathcal{B}$. The policy $\pi_{\bar{\theta}}$ is computed
from $Q_{\bar{\theta}}$ as introduced in Section 3.2. Note that in this paper
we focus on the case $q^{*}=2$ which corresponds to the Tsallis sparse policy
since it enjoys closed-form policy expression. However, it is straightforward
to extend TEMDQN to other entropic indices by employing approximate policy
following [Chen et al., 2018, Lee et al., 2020].

The pseudo-code of TEMDQN is listed in Alg. 1. At every step $t$, the
environment is explored using the policy $\pi_{q^{*},\epsilon}$ to collect a
tuple of experience into the buffer $\mathcal{B}$, where $\epsilon$ denotes a
small value used for $\epsilon$-greedy exploration. Every $C$ steps, the
network is updated by running stochastic gradient descent on a randomly
sampled batch $\mathcal{B}_{t}$ on the loss function $\mathcal{L}$. The target
network is updated every $I$ steps. Implementation details such as
hyperparameters are provided in Appendix A.

##  5 Experiments

We aim to validate the proposed method on both simple and challenging
benchmark problems. Specifically, we show that MDQN can perform significantly
poor with Tsallis sparse policy due to the logarithm mismatch, while TEMDQN
still achieves superior performance.

Figure 3 shows the comparison between TEMDQN, log-sparsemax MDQN and
TsallisDQN on the classic control problems from the OpenAI Gym [Brockman et
al., 2016], with implementation details provided in Appendix A.1. It serves as
the basic validation for our claim, from which we see log-sparsemax MDQN
failed to solve the tasks, while TEMDQN improved upon TsallisDQN and
successfully converged to the optimum.

We also compared TEMDQN and log-sparsemax MDQN in more challenging
environments such as the MiniGrid [Chevalier-Boisvert et al., 2018], MinAtar
games [Young and Tian, 2019]. We are also interested in examining whether
TEMDQN can perform favorably with distributional methods by running it on the
Atari games [Bellemare et al., 2013]. For reliable and reproducible
evaluation, we implement all algorithms based on the Stable-Baselines3 library
[Raffin et al., 2021].

Figure 3: Comparison between TEMDQN, log-sparsemax MDQN and TsallisDQN on
LunarLander-v2 and Acrobot-v1 with the policy being Tsallis sparsemax. MDQN
$\ln_{q^{*}}\\!\pi$ denotes the TEMDQN augmentation. All algorithms are
averaged over 50 independent trials to plot the mean and $\pm 1$ standard
deviation.

###  5.1 MiniGrid Environments

We aim to further consolidate the effectiveness of TEMDQN on more challenging
tasks such as the MiniGrid environments featuring sparse reward, partial
observability, the need for efficient exploration and visual input [Chevalier-
Boisvert et al., 2018]. To better illustrate the effectiveness, we choose
Empty-16$\times$16 and Dynamic-Obstacles-16$\times$16 which are the biggest
environments of their kinds. The agent in Empty starts from a fixed position
and attempts to reach the goal given partial observation and negative reward
alone the way. In Dynamic-Obstacles, the environment is equipped with randomly
moving obstacles and running into them incurs a large penalty. Implementation
details are provided in Appendix A.2.

Figure 4:  Comparison on the selected MiniGrid environments. (Top) TEMDQN,
log-sparsemax MDQN and TsallisDQN. (Bottom) TEMDQN, SQL, original MDQN and
MoVI.

TsallisDQN Comparison.  We first compare TEMDQN against TsallisDQN and log-
sparsemax MDQN. For statistical significance, all algorithms are averaged over
10 independent trials to plot mean and $\pm 1$ standard deviation in Figure 4.
Consistent with our analysis, it is visible that log-sparsemax MDQN failed to
learn meaningful behaviors in Empty-16$\times$16, leading to a flat learning
curve, while TEMDQN benefited greatly from the implicitly Tsallis KL
regularization and quickly converged to the optimum. TsallisDQN also managed
to converge to the optimum, but at a slower rate than TEMDQN.

In Dynamic-Obstacles-16$\times$16, the $\ln\pi$ augmentation again failed to
lead MDQN to the goal due to the logarithm mismatch. On the other hand,
TsallisDQN converged to a sub-optimum as a result of dynamic obstacles, which
resulted in slower convergence and lower final score than TEMDQN.

Figure 5:  Comparison of TEMDQN against TsallisDQN and log-sparsemax MDQN on
MinAtar environments.  Figure 6: Comparison between distributional TEMDQN and
log-sparsemax MDQN on Atari games.

Averaging.  One of the most important features of KL regularization is the
averaging effect when facing errors. It is hence interesting to inspect
whether TEMDQN benefits from the _pseudo-averaging_ effect introduced in Eq.
(16). In this subsection we compare TEMDQN against existing state-of-the-art
algorithms featuring such averaging effect. Specifically, we compare against:

  * •

Speedy Q-learning (SQL) [Azar et al., 2011] which reduces the greediness of
Q-learning by using two value estimates. It is shown in [Vieillard et al.,
2020a] that SQL policy can be expressed as
$\operatorname*{arg\,max}_{\pi}\left\langle\pi,\frac{1}{k}\sum_{j=1}^{k}Q_{j}-\frac{\tau}{k}\ln\pi\right\rangle$.

  * •

Momentum Value Iteration (MoVI) [Vieillard et al., 2020c] which approximates
KL policy without the help of Shannon entropy. MoVI policy is expressed as
$\operatorname*{arg\,max}_{\pi}\left\langle\pi,\frac{1}{k}\sum_{j=1}^{k}Q_{j}\right\rangle$.

  * •

MDQN.  We compare with the original MDQN implemented based on soft Q-learning,
which achieved significant improvement.

The comparison is shown in the bottom of Figure 4. In Empty, it is surprising
to see that TEMDQN reached to the optimum faster than all other algorithms.
The slower convergence of MDQN and SQL might be due to the Shannon entropy
augmentation assigned probabilities to actions not helpful to reaching the
goal. We compare the the values of $\ln\pi$ and $\ln_{q^{*}}\\!\pi$ for the
original MDQN and TEMDQN in Figure 7 in Appendix A.2.

In Dynamic-Obstacles, MDQN and SQL converged faster thanks to both the
averaging as well as diversity, which is confirmed by the slightly inferior
MoVI deterministic policy. By contrast, TEMDQN achieved a good balance leading
to fast convergence in both domains.

###  5.2 MinAtar Games

Atari games are one of the standard challenging benchmark problems for
examining modern RL algorithms. In this subsection we aim to show TEMDQN can
also effectively learn in such problems. We adopt the MinAtar environments
[Young and Tian, 2019] which consist of five Atari games optimized
comprehensively e.g. frame-skipping, observation space, etc. TEMDQN is
compared against TsallisDQN and log-sparsemax MDQN for $10^{7}$ frames without
frame skip. Experimental details are provided in Appendix A. Results are
averaged over 5 seeds.

From Figure 5 it is visible that the trend in previous experiments continue to
hold: log-sparsemax MDQN led to flat learning curves for all games due to the
mismatch. On the other hand, TEMDQN was capable of improving upon TsallisDQN
by a large margin on hard exploration games Asterix, Freeway, Seaquest. The
underperformant behavior of TsallisDQN on those environments is expected since
the sparsemax policy can result in insufficient exploration which is critical
to Atari games [Lee et al., 2020]. Though the improvement for the other two
games was not so significant, it is relieving that TEMDQN never perform worse
than the TsallisDQN, rendering it a safe substitute for TsallisDQN.

###  5.3 Distributional Munchausen

We implemented TEMDQN and log-sparsemax MDQN on top of Quantile Regression DQN
(QR-DQN) [Dabney et al., 2018] and compare them on a subset of 10 full-fledged
Atari games [Bellemare et al., 2013]. Instead of learning the conditional
expectation $Q$ function, distributional RL attempts to learn the distribution
itself:

$\displaystyle\begin{split}&Q_{\pi}(s,a)=\mathbb{E}\left[Z_{\pi}(s,a)\right],\\\
&r+\gamma P\\!\left\langle\pi,Q_{Z}\right\rangle(s,a)=r(s,a)+\gamma
Z(s^{\prime},a^{\prime}).\end{split}$ (17)

Specifically, QR-DQN aims to approximate $Z$ by regressing state-action pairs
$(s,a)$ to a uniform probability distribution supported by $N$ fixed quantiles
$\varphi_{i}:Z_{\varphi}(s,a):=\frac{1}{N}\sum_{j=1}^{N}\delta_{\varphi_{j}(s,a)}$,
where $\delta_{\\{\cdot\\}}$ is a Delta function. The implementations of
distributional TEMDQN and log-sparsemax MDQN replace every appearance of $Q$
with $Z$ in Eqs. (5), (14). Implementation details are provided in Appendix
A.4.

We allow $3\times 10^{7}$ environment steps for training and the results are
averaged over three seeds and shown in Figure 6. It is visible that log-
sparsemax MDQN has the same trend as in prior experiments of yielding nearly
flat learning curves, which is outperformed by TEMDQN by a large margin.

##  6 Discussion and Conclusion

In this paper we showed that Munchausen RL with log-sparsemax-policy
augmentation exhibited significantly poor performance for almost every problem
considered, which contradicted the general claim of M-RL that arbitrary
stochastic policy could be used. We showed that the poor performance was due
to the mismatch between the conventional logarithm and the non-logarithmic
(power) nature of Tsallis entropy. Inspired by the Tsallis statistics
literature, we reformulated Tsallis entropy and rewrote the M-RL update rule
by $q$-logarithm. The resultant algorithm enjoyed implicit Tsallis KL
divergence regularization which has neven been considered in RL literature.
Experiments on various benchmark problems validated the effectiveness of the
proposed method.

One interesting future direction is to examine various ways of implicit
Tsallis KL regularization by following a similar idea to existing literature
such as MoVI, SQL, and compare their empirical performance.

## References

  * Ahmed et al. [2019] Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans.  Understanding the impact of entropy on policy optimization.  In _Proceedings of 36th International Conference on Machine Learning_ , volume 97, pages 151–160, 2019.
  * Anschel et al. [2017] Oron Anschel, Nir Baram, and Nahum Shimkin.  Averaged-DQN: Variance reduction and stabilization for deep reinforcement learning.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70, pages 176–185, 2017.
  * Azar et al. [2011] Mohammad Azar, Hilbert Kappen, Mohammad Ghavamzadeh, and Rémi Munos.  Speedy q-learning.  In _Advances in Neural Information Processing Systems_ , volume 24, pages 1–9, 2011.
  * Azar et al. [2012] Mohammad Gheshlaghi Azar, Vicenç Gómez, and Hilbert J Kappen.  Dynamic policy programming.  _Journal of Machine Learning Research_ , 13(1):3207–3245, 2012.
  * Bellemare et al. [2013] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47(1):253–279, 2013.  ISSN 1076-9757.
  * Bertsekas and Tsitsiklis [1996] Dimitri P Bertsekas and John N Tsitsiklis.  _Neuro-Dynamic Programming_.  Athena Scientific, 1st edition, 1996.  ISBN 1886529108.
  * Brockman et al. [2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Chen et al. [2018] Gang Chen, Yiming Peng, and Mengjie Zhang.  Effective exploration for deep reinforcement learning via bootstrapped q-ensembles under tsallis entropy regularization.  _arXiv:abs/1809.00403_ , 2018.  URL http://arxiv.org/abs/1809.00403.
  * Chevalier-Boisvert et al. [2018] Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal.  Minimalistic gridworld environment for openai gym.  https://github.com/maximecb/gym-minigrid, 2018.
  * Chow et al. [2018] Yinlam Chow, Ofir Nachum, and Mohammad Ghavamzadeh.  Path consistency learning in Tsallis entropy regularized MDPs.  In _Proceedings of 35th International Conference on Machine Learning_ , volume 80, pages 979–988, 2018.
  * Dabney et al. [2018] Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos.  Distributional reinforcement learning with quantile regression.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, pages 2892–2899, 2018.
  * Farahmand [2011] Amir-massoud Farahmand.  Action-gap phenomenon in reinforcement learning.  In _Advances in Neural Information Processing Systems_ , pages 1–9, 2011.
  * Flet-Berliac et al. [2021] Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist.  Adversarially guided actor-critic.  In _Proceedings of the International Conference on Representation Learning (ICLR 2021)_ , pages 1–12, 2021.
  * Fox et al. [2016] Roy Fox, Ari Pakman, and Naftali Tishby.  Taming the noise in reinforcement learning via soft updates.  In _Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence_ , pages 202–211, 2016.
  * Furuichi et al. [2004] S. Furuichi, K. Yanagi, and K. Kuriyama.  Fundamental properties of tsallis relative entropy.  _Journal of Mathematical Physics_ , 45(12):4868–4877, 2004.
  * Geist et al. [2019] Matthieu Geist, Bruno Scherrer, and Olivier Pietquin.  A theory of regularized Markov decision processes.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97, pages 2160–2169, 2019.
  * Haarnoja et al. [2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70, pages 1352–1361, 2017.
  * Haarnoja et al. [2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80, pages 1861–1870, 2018.
  * Kozuno et al. [2019] Tadashi Kozuno, Eiji Uchibe, and Kenji Doya.  Theoretical analysis of efficiency and robustness of softmax and gap-increasing operators in reinforcement learning.  In _Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics_ , volume 89, pages 2995–3003, 2019.
  * Lan et al. [2020] Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White.  Maxmin q-learning: Controlling the estimation bias of q-learning.  In _8th International Conference on Learning Representations (ICLR)_ , pages 1–10, 2020.
  * Lee et al. [2018] Kyungjae Lee, Sungjoon Choi, and Songhwai Oh.  Sparse markov decision processes with causal sparse tsallis entropy regularization for reinforcement learning.  _IEEE Robotics and Automation Letters_ , 3:1466–1473, 2018\.
  * Lee et al. [2020] Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, Mineui Hong, Jae In Kim, Yong-Lae Park, and Songhwai Oh.  Generalized tsallis entropy reinforcement learning and its application to soft mobile robots.  In _Robotics: Science and Systems XVI_ , pages 1–10, 2020.
  * Li et al. [2019] Xiang Li, Wenhao Yang, and Zhihua Zhang.  A regularized approach to sparse optimal policy in reinforcement learning.  In _Advances in Neural Information Processing Systems 32_ , pages 1–11, 2019.
  * Martins and Astudillo [2016] André F. T. Martins and Ramón F. Astudillo.  From softmax to sparsemax: A sparse model of attention and multi-label classification.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16, page 1614–1623, 2016.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, and Others.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 2015.
  * Nachum et al. [2017] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans.  Bridging the gap between value and policy based reinforcement learning.  In _Advances in Neural Information Processing Systems 30_ , pages 2775–2785. 2017.
  * Naudts [2002] Jan Naudts.  Deformed exponentials and logarithms in generalized thermostatistics.  _Physica A-statistical Mechanics and Its Applications_ , 316:323–334, 2002.
  * Ohara [2007] Atsumi Ohara.  Geometry of distributions associated with tsallis statistics and properties of relative entropy minimization.  _Physics Letters A_ , 370(3):184–193, 2007.
  * Raffin et al. [2021] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann.  Stable-baselines3: Reliable reinforcement learning implementations.  _Journal of Machine Learning Research_ , 22(268):1–8, 2021.
  * Sutton and Barto [2018] Richard S. Sutton and Andrew G. Barto.  _Reinforcement Learning: An Introduction_.  A Bradford Book, Cambridge, MA, USA, 2018.
  * Suyari and Tsukada [2005] H. Suyari and M. Tsukada.  Law of error in tsallis statistics.  _IEEE Transactions on Information Theory_ , 51(2):753–757, 2005.
  * Tsallis [1988] Constantino Tsallis.  Possible generalization of boltzmann-gibbs statistics.  _Journal of Statistical Physics_ , 52:479–487, 1988.
  * Vieillard et al. [2020a] Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist.  Leverage the average: an analysis of regularization in rl.  In _Advances in Neural Information Processing Systems 33_ , pages 1–12, 2020a.
  * Vieillard et al. [2020b] Nino Vieillard, Olivier Pietquin, and Matthieu Geist.  Munchausen reinforcement learning.  In _Advances in Neural Information Processing Systems 33_ , pages 1–11. 2020b.
  * Vieillard et al. [2020c] Nino Vieillard, Bruno Scherrer, Olivier Pietquin, and Matthieu Geist.  Momentum in reinforcement learning.  In _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , volume 108, pages 2529–2538, 2020c.
  * Yamano [2002] Takuya Yamano.  Some properties of q-logarithm and q-exponential functions in tsallis statistics.  _Physica A: Statistical Mechanics and its Applications_ , 305(3):486–496, 2002.
  * Young and Tian [2019] Kenny Young and Tian Tian.  Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments.  _arXiv preprint arXiv:1903.03176_ , 2019.
  * Ziebart [2010] Brian D. Ziebart.  _Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy_.  PhD thesis, Carnegie Mellon University, 2010.

##  Appendix A Experimental Setting

Due to their respective characteristics, we exploited three different
architectures for running experiments on gym environments, MiniGrid and
MinAtar games and Atari games, detailed in Sections A.1, A.2, A.3 and A.4,
respectively.

###  A.1 Gym Classic Control

The Gym classic control environments are an ideal testbed of evaluating the
correctness of the proposed method. We select the environments LunarLander-v2
and Acrobot-v1 for comparison which are reasonably complicated and can be
solved by fully connected networks. TEMDQN is compared against TsallisDQN
(sparsemaxDQN) [Lee et al., 2018] and log-sparsemax MDQN, where $\pi$ is the
Tsallis sparsemax policy induced by $S_{2}(\pi)$. Totally $5\times 10^{5}$
steps are allowed for solving both tasks, and we perform 50 independent trials
to plot the mean and $\pm 1$ standard deviation in Figure 3.

The hyperparameters used for the LunarLander-v2 and Acrobot-v1 are detailed in
Table 1. The epsilon threshold is fixed at $0.01$ from the beginning of
learning. Since the sparsemax policy assigns $0$ to some actions, the $\ln\pi$
augmentation might be undefined. To this end, we add a small value $\Delta$ to
the policy to prevent ill-defined $\ln\pi$. FC$\,n$ refers to the fully
connected layer with $n$ activation units. To find a performant set of entropy
coefficient $\tau$ and Munchausen augmentation coefficient $\alpha$, we
performed a grid search over the sets
$\tau=\\{10^{-4},10^{-3},10^{-2},10^{-1},10^{0},10^{1},10^{2}\\}$ and
$\alpha=\\{10^{-4},10^{-3},0.01,0.1,0.5,0.9\\}$.

Table 1: Parameters used for TEMDQN, log-sparsemax MDQN and TsallisDQN on LunarLander-v2 and Acrobot-v1.  Parameter | Value
---|---
$T$ (total steps) | $5\times 10^{5}$
$C$ (interaction period) | 4
$|\mathcal{B}|$ (buffer size) | $5\times 10^{4}$
$\mathcal{B}_{t}$ (batch size) | 128
$\gamma$ (discount rate) | 0.99
$I$ (update period) | $2500$
$\epsilon$ (epsilon greedy threshold) | 0.01
$\Delta$ (ill-defined $\ln\pi$ prevention number) | $10^{-8}$
$\tau$ (Tsallis entropy coefficient) | 10
$\alpha$ (Munchausen coefficient) | 0.0025
Q-network architecture | FC512 - FC512
activation units | ReLU
optimizer | Adam
optimizer learning rate | $10^{-3}$

###  A.2 MiniGrid Environments

The algorithms performed on the MiniGrid environments took images of the
gridworld as input and requires sophisticated network architecture for
processing. We follow the architecture in [Flet-Berliac et al., 2021] and show
the details in Table 2. To find a performant set of parameters $\tau$ and
$\alpha$, we performed a grid search over $\tau=\\{2.5\times 10^{-4},2.5\times
10^{-3},0.025,0.25,2.5\\}$ and
$\alpha=\\{10^{-4},10^{-3},10^{-2},0.1,0.5,0.9\\}$. We compared the the values
of $\ln\pi$ and $\ln_{q^{*}}\\!\pi$ for the original MDQN and TEMDQN in Figure
7. It is clear that the magnitude of $\ln_{q^{*}}\\!\pi$ tends to be large
while MDQN prefers small log-policy, which could be interpreted as TEMDQN
prefers larger Tsallis KL regularization effect.

Figure 7:  Comparison between $\ln\pi$ and $\ln_{q^{*}}\\!\pi$ on MiniGrid environments. TEMDQN log-policy tends to be of larger magnitude than MDQN.  Table 2: Parameters used for the MiniGrid environments.  Parameter | Value
---|---
$T$ (total steps) | $1\times 10^{6}$
$C$ (interaction period) | 4
$|\mathcal{B}|$ (buffer size) | $1\times 10^{6}$
$\mathcal{B}_{t}$ (batch size) | 256
$\gamma$ (discount rate) | 0.99
$I$ (update period) | $2048$
$\epsilon$ (epsilon greedy threshold) | $1.0\rightarrow 0.01|_{10\%}$
$\Delta$ (ill-defined $\ln\pi$ prevention number) | $10^{-8}$
$\tau$ (Tsallis entropy coefficient) | 0.25
$\alpha$ (Munchausen coefficient) | 0.0001
$\beta$ (SQL/MoVI mixture steps) | $2.5\times 10^{4}$
Q-network architecture |
FC512 - $\text{Conv}^{2}_{3,3}32$ \- $\text{Conv}^{2}_{3,3}32$ \-
$\text{Conv}^{2}_{3,3}32$ \- FC
activation units | ELU
optimizer | Adam
optimizer learning rate | $3\times 10^{-4}$

###  A.3 MinAtar Games

For MinAtar games we employed the configuration recommended by Young and Tian
[2019]. The hyperparameters are listed in Table 3.

Since MinAtar games optimize different aspects of Atari games, no frame
skipping is necessary and hence the for every frame we update the network. The
epsilon greedy threshold $\epsilon:1.0\rightarrow 0.05|_{10\%}$ denotes that
$\epsilon$ is initialized as 1.0 and gradually decays to $0.05$ through the
first 10% of learning. The network architecture consists of only one
convolutional layer where $\text{Conv}^{d}_{a,b}c$ denotes a convolutional
layer with $c$ filters of size $a\times b$ and stride $d$. To find a
performant set of parameters $\tau$ and $\alpha$, we performed a grid search
over $\tau=\\{2.5\times 10^{-4},2.5\times 10^{-3},0.025,0.25,2.5\\}$ and
$\alpha=\\{10^{-4},10^{-3},10^{-2},0.1,0.5,0.9\\}$.

Table 3: Parameters used for TEMDQN, log-sparsemax MDQN and TsallisDQN on all MinAtar games.  Parameter | Value
---|---
$T$ (total steps) | $1\times 10^{7}$
$C$ (interaction period) | 1
$|\mathcal{B}|$ (buffer size) | $1\times 10^{5}$
$\mathcal{B}_{t}$ (batch size) | 32
$\gamma$ (discount rate) | 0.99
$I$ (update period) | $1000$
$\epsilon$ (epsilon greedy threshold) | $1.0\rightarrow 0.05|_{10\%}$
$\Delta$ (Munchausen number) | $10^{-8}$
$\tau$ (Tsallis entropy coefficient) | 0.025
$\alpha$ (Munchausen coefficient) | 0.01
Q-network architecture |  $\text{Conv}^{1}_{3,3}16$ \- FC128
activation units | ReLU
optimizer | RMSProp
optimizer learning rate | $2.5\times 10^{-4}$
(RMSProp) squared momentum | 0.95
(RMSProp) minimum momentum | 0.01

###  A.4 Atari Games

Table 4: Parameters used for distributional TEMDQN and log-sparsemax MDQN on Atari games.  Parameter | Value
---|---
$T$ (total steps) | $3\times 10^{7}$
$C$ (interaction period) | 4
$|\mathcal{B}|$ (buffer size) | $1\times 10^{6}$
$\mathcal{B}_{t}$ (batch size) | 32
$\gamma$ (discount rate) | 0.99
$I$ (update period) | $8000$
$\epsilon$ (epsilon greedy threshold) | $1.0\rightarrow 0.01|_{10\%}$
$\Delta$ (Munchausen number) | $10^{-8}$
$\tau$ (Tsallis entropy coefficient) | 0.025
$\alpha$ (Munchausen coefficient) | 0.001
Q-network architecture |
$\text{Conv}^{4}_{8,8}32$ \- $\text{Conv}^{2}_{4,4}64$ \-
$\text{Conv}^{1}_{3,3}64$ \- FC512 - FC
activation units | ReLU
optimizer | Adam
optimizer learning rate | $10^{-4}$

We compared distributional TEMDQN and log-sparsemax MDQN on Atari games which
are more challenging and high dimensional than the optimized MinAtar games. We
leveraged the optimized Stable-Baselines3 architecture [Raffin et al., 2021]
for best performance. The details can be seen from Table 4. The Q-network uses
3 convolutional layers. The epsilon greedy threshold is initialized at 1.0 and
gradually decays to 0.01 at the end of first 10% of learning. To find a
performant set of parameters $\tau$ and $\alpha$, we performed a grid search
over $\tau=\\{2.5\times 10^{-4},2.5\times 10^{-3},0.025,0.25,2.5\\}$ and
$\alpha=\\{10^{-3},10^{-2},0.1,0.9\\}$.


# Theory of Fractional Quantum Hall States of the $\mathcal{A}$-phase in the
Second Landau Level

Sudipto Das  Sahana Das  Sudhansu S. Mandal  Department of Physics, Indian
Institute of Technology, Kharagpur, West Bengal 721302, India

###### Abstract

A proposal of the existence of an Anomalous phase ($\mathcal{A}$-phase) at the
experimental range of moderate Landau-level-mixing strength has recently been
made for $5/2$ state. We here report that the gapped $\mathcal{A}$-phase is
generic to the sequence of spin-polarized fractional quantum Hall states with
filling fractions $\nu=n/(nm-1)$ and $\nu=1-n/(nm-1)$, $(n\geqslant
1,\,m\geqslant 3)$, thereby predicting several other states in addition to the
observed ones in the second Landau level for GaAs systems. Our proposed trial
wave functions for all these states have remarkably high overlaps with the
corresponding exact ground states and can support non-Abelian quasiparticle
excitations with charge $e/[2(nm-1)]$. By analyzing edge modes, we predict
experimentally verifiable thermal Hall conductance $2.5(\pi^{2}k_{B}^{2}T/3h)$
for all the states in these sequences.

††preprint: APS/123-QED

Surprising discovery Willett  _et al._ (1987) of an even-denominator
fractional quantum Hall effect (FQHE) state $5/2$ and subsequent observations
Xia  _et al._ (2004); Pan  _et al._ (2008); Choi  _et al._ (2008); Kumar  _et
al._ (2010); Zhang  _et al._ (2012); Shingla  _et al._ (2018) of other odd-
and even-denominator states in the second Landau level (SLL) fascinate for
rich physics. Following the pioneering proposal of Moore and Read (MR) Moore
and Read (1991) for 5/2 state, even-denominator FQHE states are believed to
host excitations of non-Abelian quasiparticles. This occurs Read and Green
(2000) owing to the Cooper-pairing with inherent $\mathbb{Z}_{2}$ symmetry
amongst emergent composite fermions Jain (1989a, 2007) arising due to the
interaction between electrons in the presence of a high magnetic field. There
have also been proposals in terms of particle-hole conjugate Lee  _et al._
(2007); Levin  _et al._ (2007) of MR Pfaffian state and particle-hole
symmetric Son (2015) Pfaffian state as alternative topological orders of the
5/2 state.

The odd-denominator observed states like 7/3, 8/3, 11/5, and 14/5 are believed
Dolev  _et al._ (2011); Wurstbauer  _et al._ (2015); MacDonald and Girvin
(1986); d’Ambrumenil and Reynolds (1988) to behave as Abelian FQHE states. On
the other hand, a generalization Read and Rezayi (1999) of MR theory for 5/2
state yielding $2+n/(n+2)$ sequence of states supporting quasiparticles as
$\mathbb{Z}_{n}$ parafermions Read and Rezayi (1999) predicts non-Abelian 8/3
state. This sequence, however, does not reproduce all the observed states in
the SLL. Other generalizations Bishara  _et al._ (2008); Jolicoeur (2007);
Bonderson and Slingerland (2008) of MR theory fetch certain odd- and even-
denominator non-Abelian states as well. Both the Abelian and non-Abelian FQHE
states are also proposed in parton model Jain (1989b); Balram  _et al._ (2018,
2019). However, none of the different kinds of topological orders predicted by
these theories have yet been confirmed in any experiment.

The validity of the proposed wave functions based on the above mentioned
theories is mainly tested numerically Balram  _et al._ (2018, 2019); Rezayi
and Simon (2011); Zaletel  _et al._ (2015); Pakrouski  _et al._ (2015); Rezayi
(2017); Simon  _et al._ (2020); Pakrouski  _et al._ (2016) for the pure
Coulomb interaction or at the best very low strength of the Landau-level-
mixing (LLM), $\kappa$, which is quantified as the ratio of the Coulomb energy
scale and the cyclotron energy scale. Because the experiments are typically
performed Willett  _et al._ (1987); Xia  _et al._ (2004); Pan  _et al._
(2008); Choi  _et al._ (2008); Kumar  _et al._ (2010); Zhang  _et al._ (2012);
Shingla  _et al._ (2018); Dean  _et al._ (2008a, b); Zhang  _et al._ (2010);
Pan  _et al._ (2001, 2014); Samkharadze  _et al._ (2017); Banerjee  _et al._
(2018); Dutta  _et al._ (2022a, b) between $1$ through $12$ Tesla magnetic
field for these FQHE states in GaAs systems, the role of filled and empty
Landau levels also become important as $\kappa$ becomes Bishara and Nayak
(2009); Peterson and Nayak (2013); Sodemann and MacDonald (2013); Simon and
Rezayi (2013) moderately higher in the range $0.7$–$2.5$. Due to the LLM
Bishara and Nayak (2009); Peterson and Nayak (2013); Sodemann and MacDonald
(2013); Simon and Rezayi (2013), not only the two-body pseudopotentials get
renormalized, but certain three-body pseudopotentials are also emerged and
consequently the particle-hole symmetry no longer becomes an exact symmetry in
the SLL. Exact diagonalization study with perturbatively obtained two-body and
three-body pseudopotentials Peterson and Nayak (2013) in the spherical
geometry Haldane (1983) suggests Das  _et al._ (2022) a topological phase
transition between the conventional phase and an Anomalous phase
($\mathcal{A}$-phase) at $\kappa\sim 0.7$ for 5/2 state. The ground state at
this $\mathcal{A}$-phase of 5/2 state is well described by a trial
wavefunction Das  _et al._ (2022) which is nearly orthogonal to MR, particle-
hole conjugate of MR, and particle-hole symmetric Pfaffian wave functions.

Here we show by calculating overlaps of the exact ground states of the
Hamiltonian consisting of LLM corrected two-body and three-body
pseudopotentials at the total angular momentum $L=0$ in spherical geometry for
different values of $\kappa$, $\mathcal{A}$-phase is generic Not  to all the
observed Willett  _et al._ (1987); Xia  _et al._ (2004); Pan  _et al._ (2008);
Choi  _et al._ (2008); Kumar  _et al._ (2010); Zhang  _et al._ (2012); Shingla
_et al._ (2018) and some of the predicted FQHE states in the SLL. We propose
trial wave functions for all these FQHE states as a generalization to the
recently proposed trial wavefunction Das  _et al._ (2022) for 5/2 state in the
$\mathcal{A}$-phase. These wavefunctions have remarkably high overlaps with
the corresponding exact ground states and are argued to support non-Abelian
quasiparticle excitations. The $\mathcal{A}$-phase of all these FQHE states is
found to have a positive finite charge gap in the thermodynamic limit. An
analysis of the low-energy effective Lagrangian density Wen and Zee (1992) in
terms of Chern-Simons gauge fields with coupling matrix derived from the
proposed trial wave functions provide quasiparticle charge as $q=e/[2(nm-1)]$
for the sequences of filling factors $\nu=n/(nm-1)$ and $1-\nu$ in the SLL. We
further predict that all these FQHE states will provide a unique thermal Hall
conductance $2.5G_{0}$ ($G_{0}=\pi^{2}k_{B}^{2}T/3h$) governed by two (three)
bosonic downstream edge modes for two (three) filled Landau levels and one
downstream (upstream) Majorana edge mode for FQHE states $\nu\,(1-\nu)$.

The topological order of the $\mathcal{A}$-phase in the SLL for $\nu=1/2$
state is distinct Das  _et al._ (2022) from the competing conventional MR
Pfaffian, its particle-hole conjugate anti-Pfaffian, and particle-hole
symmetric Pfaffian orders. A physical interpretation of the proposed trial
wave function Das  _et al._ (2022) for this phase is that the composite bosons
(CBs)–bound-state of an electron with one unit of flux quantum– divide
themselves into two groups, CBs in each group is noninteracting and condense
while the CBs between two groups repulse each other such that they feel zeros
of order two at each other’s position. In this letter, we generalize this in
terms of dividing each group into a number of sectors and fixing a number of
zeros between a pair of CBs belonging to different groups. If each group is
divided into $n$ fictitious sectors with an equal number of CBs, each CB of a
sector feels $2(m-2)$ zeros at the positions of other CBs of a sector and
$2(m-1)$ zeros at the positions of CBs at all other $(n-1)$ sectors in the
other group, the corresponding FQHE state forms at the filling factor

$\nu=\frac{n}{nm-1}$ (1)

and its particle-hole conjugate state at $\nu=1-n/(nm-1)$ for $m\geqslant 3$
and $n\geqslant 1$. The total angular momentum of the corresponding
wavefunction (ignoring ubiquitous Gaussian factor)

$\displaystyle\Psi_{\mathcal{A}}\,(n,m)=\prod_{i<j}^{N}(z_{i}-z_{j})\,{\cal
S}\left[\prod_{1\leqslant k,l\leqslant N/(2n)}\right.$
$\displaystyle\left(\prod_{\alpha=0}^{n-1}\left(z_{k+\alpha
N/(2n)}-z_{l+N/2+\alpha N/(2n)}\right)^{2(m-2)}\right)$
$\displaystyle\left.\left(\prod_{\alpha\neq\beta,0}^{n-1}\left(z_{k+\alpha
N/(2n)}-z_{l+N/2+\beta N/(2n)}\right)^{2(m-1)}\right)\right]$ (2)

that we propose becomes $M=N(\nu^{-1}N-1)/2$ in the disk-geometry implying the
same ‘flux-shift’ (by $1$) in the spherical geometry for all the FQHE states
in the SLL. Here ${\cal S}$ represents symmetrization with respect to all $N$
particle indices and the complex particle coordinates
$z_{j}=(x_{j}-i\,y_{j})/\ell_{0}$ with $\ell_{0}$ being the magnetic length.
The CBs form due to the attachment of one flux quantum attached to each
electron described by the Jastrow factor $\prod_{i<j}^{N}(z_{i}-z_{j})$. The
indices $\alpha$ and $\beta$ represent $n$ sectors in both the condensates of
CBs. The suffix $\mathcal{A}$ in Eq. (Theory of Fractional Quantum Hall States
of the $\mathcal{A}$-phase in the Second Landau Level) represents that the
proposed wave functions are for the $\mathcal{A}$-phase of the corresponding
$\nu=n/(nm-1)$. The wavefunction $\Psi_{\mathcal{A}}\,(n,m)$ does not vanish
even if up to macroscopic $N/2$ CBs of a group coincide. In analogy to other
known FQHE wavefunctions Moore and Read (1991); Read and Rezayi (1999)
supporting non-Abelian quasiparticles having similar properties (albeit for
finite number of particles), we believe that $\Psi_{\mathcal{A}}\,(n,m)$ too
will support non-Abelian quasiparticles.

The observed Xia  _et al._ (2004); Pan  _et al._ (2008); Choi  _et al._
(2008); Zhang  _et al._ (2012); Shingla  _et al._ (2018) principal sequence
$\nu=1/2,\,2/5,\,3/8,\cdots$ corresponds to $m=3$ and $n=1,\,2,\,3,\cdots$
respectively in Eq. (1). Other observed Xia  _et al._ (2004); Pan  _et al._
(2008); Choi  _et al._ (2008); Zhang  _et al._ (2012); Shingla  _et al._
(2018) FQHE states in the SLL are $\nu=1/3$ (also 2/3) for $m=4$ and $n=1$,
$2/9$ (also 7/9) for $m=5$ and $n=2$, and $1/5$ (also 4/5) for $m=6$ and
$n=1$. Only $\nu=6/13$ (in between two consecutive filling factors, namely
$1/2$ and $2/5$ of the principal sequence) amongst observed Kumar  _et al._
(2010); Shingla  _et al._ (2018) FQHE states does not fit in the sequence (1),
just as $4/11$ state Mukherjee  _et al._ (2014); Das  _et al._ (2021) in the
lowest Landau level is in between two states (1/3 and 2/5) of the principal
Jain sequence Jain (1989a).

Figure 1: (color online) Overlaps (shown as color map) $\mathcal{O}_{ij}$ of
the exact ground states (found at $L=0$ with different values of $\kappa_{i}$)
of the Hamiltonian $\hat{H}_{\text{eff}}$ in Eq. (3) with LLM-corrected
pseudopotentials pse  at different FQHE filling factors $\nu$ observed in the
SLL. The number of particles and the corresponding number of flux quanta for a
specific filling factor are mentioned as $(N,N_{\Phi})$ in each of the panels
with a fixed flux shift: $N_{\Phi}=\nu^{-1}N-1$. Two topologically distinct
phases are generically found as the overlaps of the ground states at different
$\kappa$ values belonging to the same phase are closer to unity and two
different phases are closer to zero. These two phases are intermediated by an
unquantized regime (shown as crossed marks) of $\kappa$ where the ground state
is not found at $L=0$. The gray zones indicate that no overlap has been shown
as one of the ground states corresponds to the unquantized regime.

The effective Hamiltonian for the spin-polarized electrons including the
effect of LLM Bishara and Nayak (2009); Peterson and Nayak (2013) in the SLL
is given by

$\displaystyle\hat{H}_{\text{eff}}(\kappa)$ $\displaystyle=$
$\displaystyle\sum_{m\,\text{odd}}\left[V_{m}^{(2)}+\kappa\,\delta
V_{m}^{(2)}\right]\sum_{i<j}\hat{P}_{ij}(m)$ (3)
$\displaystyle+\sum_{m\geqslant
3}\kappa\,V_{m}^{(3)}\sum_{i<j<k}\hat{P}_{ijk}(m)$

for the electrons in a spherical geometry, where $V_{m}^{(2)}$ represents two-
body Coulomb pseudo-potential in the SLL and $\delta V_{m}^{(2)}$ is its
correction due to the LLM and $V_{m}^{(3)}$ is the emergent three-body pseudo-
potential arising due to the LLM. Here $\hat{P}_{ij}(m)$ and
$\hat{P}_{ijk}(m)$ are two- and three-body projection operators respectively
onto pairs or triplets of electrons with relative angular momentum $m$. We
exactly diagonalize $\hat{H}_{\text{eff}}$ with pseudopotentials (see Ref. pse
) for the observed FQHE states given by the sequences of states as shown in
Eq. (1) for a finite system of $N$ electrons with the number of flux quanta
$N_{\Phi}=\nu^{-1}N-1$. We then determine overlaps of the exact ground states
when found at the total angular momentum $L=0$ for different values of
$\kappa$:
$\mathcal{O}_{ij}=\langle\Psi_{\text{gs}}(\kappa_{i})|\Psi_{\text{gs}}(\kappa_{j})\rangle$.
As found in Ref. Das  _et al._ , 2022 for $\nu=1/2$ state, we show (Fig. 1)
that $\mathcal{A}$-phase in the SLL at the moderate regime of $\kappa$ is
generic to all the observed FQHE states belonging to the sequence $\nu$ and
$1-\nu$ in Eq. (1), viz, $\nu=1/2$, 2/5, 3/8, 1/3, 2/9, 1/5, 2/3, 7/9, and
4/5. The exact ground states in the $\mathcal{A}$-phase is orthogonal to the
corresponding ground states at the low-$\kappa$ phase. These two phases are
intermediated by a regime of unquantized states.

Figure 2: (color online) Charge gap
$\Delta_{c}=(E_{\text{qh}}+E_{\text{qp}})/2$ with quasiparticle energy
$E_{\text{qp}}=E(N,N_{\Phi}-1)-E(N,N_{\Phi})$ and quasihole energy
$E_{\text{qh}}=E(N,N_{\Phi}+1)-E(N,N_{\Phi})$ for an FQHE system given by
$(N,N_{\Phi})$ is scaled with $1/N$ for different $\nu$ and at a chosen value
of $\kappa$ belonging to the corresponding $\mathcal{A}$-phase. Each $E$ is
estimated by subtracting the corresponding background energies. Energies for
finite systems are corrected with the density correction factor $\sqrt{\nu
N_{\Phi}/N}$. Positive finite values of $\Delta_{c}$ in the thermodynamic
limit indicate robust quantized FQHE states.

We calculate charge excitation energy gap $\Delta_{c}$ Rezayi  _et al._ (2021)
for different pairs of $N$ and $N_{\Phi}$ for a given FQHE state by taking the
average of a single quasiparticle and a single quasihole excitation energies.
Fig. 2 shows scaling of $\Delta_{c}$ with $1/N$ for all the observed FQHE
states belonging to the sequences $\nu$ and $1-\nu$. A thermodynamic extension
of this scaling indicates positive and finite excitation energies of a pair of
a quasiparticle and a quasihole. Therefore, all these FQHE states in the
$\mathcal{A}$-phase are incompressible.

Figure 3: (color online) Overlaps of the proposed wavefunctions (Theory of
Fractional Quantum Hall States of the $\mathcal{A}$-phase in the Second Landau
Level) with the exact ground states for noted $\nu$ and $N$ at different
values of $\kappa$ in the corresponding regime of $\mathcal{A}$-phase.

In Fig. 3, we show overlap of the exact ground states of $\nu=1/2$, 2/5, 3/8,
1/3, 2/9, and 1/5 in the $\mathcal{A}$-phase with the proposed trial
wavefunctions (Theory of Fractional Quantum Hall States of the
$\mathcal{A}$-phase in the Second Landau Level). The overlaps of the exact
ground states in the $\mathcal{A}$-phase of 2/3, 7/9, and 4/5 which are
particle-hole conjugate of 1/3, 2/9, and 1/5 states respectively, with the
corresponding particle-hole conjugate form of the wavefunctions given in Eq.
(Theory of Fractional Quantum Hall States of the $\mathcal{A}$-phase in the
Second Landau Level) are also shown in Fig. 3. All of our proposed
wavefunctions in a general footing have remarkably high overlap with the
corresponding exact ground states.

The numerical studies on FQHE states such as 1/3 and 1/5 are tagged as Abelian
in literature Dolev  _et al._ (2011); Wurstbauer  _et al._ (2015); MacDonald
and Girvin (1986); d’Ambrumenil and Reynolds (1988); Faugno  _et al._ (2021).
However, these studies are based on either zero or very low $\kappa$. These
states also appear in the $\mathcal{A}$-phase at the moderate regime of
$\kappa$, which coincides with the experimental regime in GaAs system, and we
predict that these states too are non-Abelian. The predicted flux-shift here
is 1 for all the FQHE states given by $\nu$ in Eq. (1) as well as its
particle-hole conjugate filling factor $1-\nu$. The only previously proposed
wavefunctions that have the same flux-shift that of ours are for 1/2 and 2/5
in Ref. Jolicoeur, 2007 and 3/8 states in Refs. Jolicoeur, 2007; Bonderson and
Slingerland, 2008. However, the wavefunctions in Ref. Jolicoeur, 2007 is not
convenient for implementing in numerical comparison with any other
wavefunctions. The vanishingly small overlap of earlier proposed wavefunction
Bonderson and Slingerland (2008); Hutasoit  _et al._ (2017) for 3/8 with that
of ours suggests they belong to different topological classes, despite having
the same flux-shift.

Figure 4: (color online) Filling factors: $\nu=4/11$, 2/7, and 1/4. (a)–(c)
Ovelaps as described in the caption of Fig. 1. (d) Scaling of $\Delta_{c}$ as
described in the caption of Fig. 2. $\Delta_{c}$ for $\nu=4/11$ is shown only
for $N=8$ electrons as exact diagonalization for next higher particle systems
of $N=16$ is beyond our scope. Nevertheless, we believe that thermodynamic
extrapolation of $\Delta_{c}$ for $\nu=4/11$ will be finite positive as well.
(e) Overlaps of the exact ground states with the proposed trial wavefunctions
Eq. (Theory of Fractional Quantum Hall States of the $\mathcal{A}$-phase in
the Second Landau Level) as described in the caption of Fig. 3.

In Fig. 4, we show the occurrence of incompressible FQHE states 4/11, 2/7, and
1/4 in the $\mathcal{A}$-phase of the SLL. Our proposed wavefunctions (Theory
of Fractional Quantum Hall States of the $\mathcal{A}$-phase in the Second
Landau Level) for these states have excellent overlap with the corresponding
exact ground states. These FQHE states have not yet been observed.
Experimental findings of these states will further confirm our general theory
presented in this paper.

Figure 5: (color online) Schematic of edge modes for the filling factors (a)
$\nu=1/2$, (b) $\nu=2/5$, (c) $\nu=1/3$, (d) $\nu=2/3$, and (e) $\nu=3/8$ in
the SLL. Arrow-headed solid double-line, solid single line, wavy line, and
dashed line, respectively represent electronic mode, charged quasiparticle
mode, charge neutral mode, and Majorana mode. Each panel has two parts: (i) In
upper part, edge modes are shown as per the $\mathbb{K}$ matrix in Eq. (5);
(ii) in the lower part, an equivalent possibilities of modes due to disorder
are shown. In the latter case, the quasihole edge modes are converted Kane
_et al._ (1994); Bid  _et al._ (2010) from the former case into neutral modes
by changing quasiparticle charge in the quasiparticle modes. Arrow-head
indicates the direction of a mode. Symbols $(e^{\ast},\kappa_{{}_{T}})$
represent charge and thermal Hall-conductivity in the unit of $G_{0}$ carried
by different modes respectively. The values of $e^{\ast}$ for
quasiparticle/quasihole modes are shown beside the corresponding modes. The
top two modes (outside the panels) are for two filled Landau levels.

From the $2n$-component structure of the proposed wave function Eq. (Theory of
Fractional Quantum Hall States of the $\mathcal{A}$-phase in the Second Landau
Level) we extract the topological properties of the $\mathcal{A}$-phase
through the low-energy effective Lagrangian density Wen and Zee (1992) as,

$\mathcal{L}=-\frac{1}{4\pi}\epsilon^{\alpha\beta\gamma}\sum_{I,J=1}^{2n}\mathbb{K}_{IJ}a_{\alpha}^{I}\partial_{\beta}a_{\gamma}^{J}-\frac{1}{2\pi}\epsilon^{\alpha\beta\gamma}\sum_{I=1}^{2n}t_{I}A_{\alpha}\partial_{\beta}a_{\gamma}^{I}$
(4)

Here $a_{\alpha}^{I}$ represents $I$-th component of $2n$-component Chern-
Simons gauge fields, $A_{\alpha}$ is the external electromagnetic field, and
$\epsilon^{\alpha\beta\gamma}$ is the antisymmetric Levi-Cevita tensor. The
symmetric $\mathbb{K}$-matrix for the sequence of states given in Eq. (1) can
be read from the proposed wavefunction (Theory of Fractional Quantum Hall
States of the $\mathcal{A}$-phase in the Second Landau Level) as

$\mathbb{K}=\begin{pmatrix}\mathbb{C}&\mathbb{M}\\\
\mathbb{M}&\mathbb{C}\end{pmatrix}\,,$ (5)

where $\mathbb{C}$ and $\mathbb{M}$ are $n\times n$ matrices given
respectively by $\mathbb{C}_{ij}=1$ and
$\mathbb{M}_{ij}=2(m-1-\delta_{ij})+1$. Further introducing charge vector
$t^{T}=(1,1,1,\cdots)_{2n}$ and quasiparticle vector
$l^{T}=(1,0,0,\cdots)_{2n}$ and following Ref. Wen and Zee, 1992, we find
topological properties such as filling factor
$\nu=t^{T}\mathbb{K}^{-1}t=n/(nm-1)$ and quasiparticle charge
$q=e\,l^{T}\mathbb{K}^{-1}t=e/[2(nm-1)]$. Half of the eigenvalues of the
$\mathbb{K}$-matrix are positive and the rest are negative. Therefore, $n$
downstream and $n$ upstream quasiparticle/quasihole charge edge modes will
exist for the FQHE states with $\nu=n/(nm-1)$ as well as its particle-hole
conjugate filling factor $1-\nu$. Owing to the presence of disorder, these
quasihole charge modes may be converted Kane  _et al._ (1994); Bid  _et al._
(2010) into a neutral mode with an addition of an additional quasiparticle in
the downstream mode. Because $\Psi_{\mathcal{A}}(n,m)$ in Eq. (Theory of
Fractional Quantum Hall States of the $\mathcal{A}$-phase in the Second Landau
Level) indicates that every CB has choices of joining two available
condensates which can accommodate $N/2$ of them and remains noninteracting, it
has a hidden $\mathbb{Z}_{2}$ symmetry Cappelli  _et al._ (2001).
Consequently, there will be one neutral downstream (upstream) Majorana edge
mode carrying $0.5G_{0}$ thermal Hall conductance for the sequence of filling
factors $\nu$ ($1-\nu$). Figure 5 illustrates the possible edge modes for
$\nu=1/2$, 2/5, 1/3, 2/3, and 3/8 in the SLL. Considering two (three)
completely filled Landau levels for $\nu$ ($1-\nu$) and thereby two (three)
downstream bosonic edge modes, the total thermal Hall conductance will be
$2.5G_{0}$ irrespective of the FQHE states in the $\mathcal{A}$-phase of the
SLL.

In summary, we find a sequence (1) that exhausts Not  all the observed FQHE
states in the SLL and are shown them as incompressible in the
$\mathcal{A}$-phase at the moderate regime of $\kappa\sim 1$. Our proposed
trial wave functions (Theory of Fractional Quantum Hall States of the
$\mathcal{A}$-phase in the Second Landau Level) for all these states have very
high overlap with the corresponding exact ground states. The characteristics
of these wavefunctions support non-Abelian quasiparticle excitations out of
these states. Based on the proposed wave functions, we determine the possible
edge modes and consequently predict $2.5G_{0}$ thermal Hall conductance for
all these states.

We appreciate the generosity of the developers of the DiagHam package for
keeping it open access. We acknowledge the Param Shakti (IIT Kharagpur) – a
National Supercomputing Mission, Government of India for providing
computational resources. S.S.M. is supported by Science and Engineering
Research Board (Government of India) through Grant No. MTR/2019/000546.

## References

  * Willett  _et al._ (1987) R. Willett, J. P. Eisenstein, H. L. Störmer, D. C. Tsui, A. C. Gossard, and J. H. English, Phys. Rev. Lett. 59, 1776 (1987).
  * Xia  _et al._ (2004) J. S. Xia, W. Pan, C. L. Vicente, E. D. Adams, N. S. Sullivan, H. L. Stormer, D. C. Tsui, L. N. Pfeiffer, K. W. Baldwin, and K. W. West, Phys. Rev. Lett. 93, 176809 (2004).
  * Pan  _et al._ (2008) W. Pan, J. S. Xia, H. L. Stormer, D. C. Tsui, C. Vicente, E. D. Adams, N. S. Sullivan, L. N. Pfeiffer, K. W. Baldwin, and K. W. West, Phys. Rev. B 77, 075307 (2008).
  * Choi  _et al._ (2008) H. C. Choi, W. Kang, S. Das Sarma, L. N. Pfeiffer, and K. W. West, Phys. Rev. B 77, 081301 (2008).
  * Kumar  _et al._ (2010) A. Kumar, G. A. Csáthy, M. J. Manfra, L. N. Pfeiffer, and K. W. West, Phys. Rev. Lett. 105, 246808 (2010).
  * Zhang  _et al._ (2012) C. Zhang, C. Huan, J. S. Xia, N. S. Sullivan, W. Pan, K. W. Baldwin, K. W. West, L. N. Pfeiffer, and D. C. Tsui, Phys. Rev. B 85, 241302 (2012).
  * Shingla  _et al._ (2018) V. Shingla, E. Kleinbaum, A. Kumar, L. N. Pfeiffer, K. W. West, and G. A. Csáthy, Phys. Rev. B 97, 241105 (2018).
  * Moore and Read (1991) G. Moore and N. Read, Nuclear Physics B 360, 362 (1991).
  * Read and Green (2000) N. Read and D. Green, Phys. Rev. B 61, 10267 (2000).
  * Jain (1989a) J. K. Jain, Phys. Rev. Lett. 63, 199 (1989a).
  * Jain (2007) J. K. Jain,  _Composite Fermions_ (Cambridge University Press, New York, 2007).
  * Lee  _et al._ (2007) S.-S. Lee, S. Ryu, C. Nayak, and M. P. A. Fisher, Phys. Rev. Lett. 99, 236807 (2007).
  * Levin  _et al._ (2007) M. Levin, B. I. Halperin, and B. Rosenow, Phys. Rev. Lett. 99, 236806 (2007).
  * Son (2015) D. T. Son, Phys. Rev. X 5, 031027 (2015).
  * Dolev  _et al._ (2011) M. Dolev, Y. Gross, R. Sabo, I. Gurman, M. Heiblum, V. Umansky, and D. Mahalu, Phys. Rev. Lett. 107, 036805 (2011).
  * Wurstbauer  _et al._ (2015) U. Wurstbauer, A. L. Levy, A. Pinczuk, K. W. West, L. N. Pfeiffer, M. J. Manfra, G. C. Gardner, and J. D. Watson, Phys. Rev. B 92, 241407 (2015).
  * MacDonald and Girvin (1986) A. H. MacDonald and S. M. Girvin, Phys. Rev. B 33, 4009 (1986).
  * d’Ambrumenil and Reynolds (1988) N. d’Ambrumenil and A. M. Reynolds, J. Phys. C: Solid State Phys. 21, 119 (1988).
  * Read and Rezayi (1999) N. Read and E. Rezayi, Phys. Rev. B 59, 8084 (1999).
  * Bishara  _et al._ (2008) W. Bishara, G. A. Fiete, and C. Nayak, Phys. Rev. B 77, 241306 (2008).
  * Jolicoeur (2007) T. Jolicoeur, Phys. Rev. Lett. 99, 036805 (2007).
  * Bonderson and Slingerland (2008) P. Bonderson and J. K. Slingerland, Phys. Rev. B 78, 125323 (2008).
  * Jain (1989b) J. Jain, Phys. Rev. B 40, 8079 (1989b).
  * Balram  _et al._ (2018) A. C. Balram, S. Mukherjee, K. Park, M. Barkeshli, M. S. Rudner, and J. K. Jain, Phys. Rev. Lett. 121, 186601 (2018).
  * Balram  _et al._ (2019) A. C. Balram, M. Barkeshli, and M. S. Rudner, Phys. Rev. B 99, 241108 (2019).
  * Rezayi and Simon (2011) E. H. Rezayi and S. H. Simon, Phys. Rev. Lett. 106, 116801 (2011).
  * Zaletel  _et al._ (2015) M. P. Zaletel, R. S. K. Mong, F. Pollmann, and E. H. Rezayi, Phys. Rev. B 91, 045115 (2015).
  * Pakrouski  _et al._ (2015) K. Pakrouski, M. R. Peterson, T. Jolicoeur, V. W. Scarola, C. Nayak, and M. Troyer, Phys. Rev. X 5, 021004 (2015).
  * Rezayi (2017) E. H. Rezayi, Phys. Rev. Lett. 119, 026801 (2017).
  * Simon  _et al._ (2020) S. H. Simon, M. Ippoliti, M. P. Zaletel, and E. H. Rezayi, Phys. Rev. B 101, 041302 (2020).
  * Pakrouski  _et al._ (2016) K. Pakrouski, M. Troyer, Y.-L. Wu, S. Das Sarma, and M. R. Peterson, Phys. Rev. B 94, 075108 (2016).
  * Dean  _et al._ (2008a) C. R. Dean, B. A. Piot, P. Hayden, S. Das Sarma, G. Gervais, L. N. Pfeiffer, and K. W. West, Phys. Rev. Lett. 101, 186806 (2008a).
  * Dean  _et al._ (2008b) C. R. Dean, B. A. Piot, P. Hayden, S. Das Sarma, G. Gervais, L. N. Pfeiffer, and K. W. West, Phys. Rev. Lett. 100, 146803 (2008b).
  * Zhang  _et al._ (2010) C. Zhang, T. Knuuttila, Y. Dai, R. R. Du, L. N. Pfeiffer, and K. W. West, Phys. Rev. Lett. 104, 166801 (2010).
  * Pan  _et al._ (2001) W. Pan, H. L. Stormer, D. C. Tsui, L. N. Pfeiffer, K. W. Baldwin, and K. W. West, Solid State Communications 119, 641 (2001).
  * Pan  _et al._ (2014) W. Pan, A. Serafin, J. S. Xia, L. Yin, N. S. Sullivan, K. W. Baldwin, K. W. West, L. N. Pfeiffer, and D. C. Tsui, Phys. Rev. B 89, 241302 (2014).
  * Samkharadze  _et al._ (2017) N. Samkharadze, D. Ro, L. N. Pfeiffer, K. W. West, and G. A. Csáthy, Phys. Rev. B 96, 085105 (2017).
  * Banerjee  _et al._ (2018) M. Banerjee, M. Heiblum, V. Umansky, D. E. Feldman, Y. Oreg, and A. Stern, Nature 559, 205 (2018).
  * Dutta  _et al._ (2022a) B. Dutta, W. Yang, R. Melcer, H. K. Kundu, M. Heiblum, V. Umansky, Y. Oreg, A. Stern, and D. Mross, Science 375, 193 (2022a).
  * Dutta  _et al._ (2022b) B. Dutta, V. Umansky, M. Banerjee, and M. Heiblum, Science 377, 1198 (2022b).
  * Bishara and Nayak (2009) W. Bishara and C. Nayak, Phys. Rev. B 80, 121302 (2009).
  * Peterson and Nayak (2013) M. R. Peterson and C. Nayak, Phys. Rev. B 87, 245129 (2013).
  * Sodemann and MacDonald (2013) I. Sodemann and A. H. MacDonald, Phys. Rev. B 87, 245425 (2013).
  * Simon and Rezayi (2013) S. H. Simon and E. H. Rezayi, Phys. Rev. B 87, 155426 (2013).
  * Haldane (1983) F. D. M. Haldane, Phys. Rev. Lett. 51, 605 (1983).
  * Das  _et al._ (2022) S. Das, S. Das, and S. S. Mandal, arXiv:2206.04419 (2022).
  * (47) Amongst all the observed FQHE states, the filling factor $6/13$ is not studied here as it does not belong to the sequence in Eq. (1) and the construction of its wavefunction will differ from Eq. (Theory of Fractional Quantum Hall States of the $\mathcal{A}$-phase in the Second Landau Level). A detailed analysis of $6/13$ state in $\mathcal{A}$-phase will be published elsewhere.
  * Wen and Zee (1992) X. G. Wen and A. Zee, Phys. Rev. B 46, 2290 (1992).
  * Mukherjee  _et al._ (2014) S. Mukherjee, S. S. Mandal, Y.-H. Wu, A. Wójs, and J. K. Jain, Phys. Rev. Lett. 112, 016801 (2014).
  * Das  _et al._ (2021) S. Das, S. Das, and S. S. Mandal, Phys. Rev. B 103, 075304 (2021).
  * (51) Pseudopotentials values are obtained from the Ref. Peterson and Nayak (2013). For zero quantum well width and fully polarized electrons, LLM induced the 2-body Pseudo-potentials are $\delta V_{1}^{(2)}=-0.2143$, $\delta V_{3}^{(2)}=-0.1039$, $\delta V_{5}^{(2)}=-0.0353$, $\delta V_{7}^{(2)}=-0.0115$, and $\delta V_{9}^{(2)}=-0.0023$ and the 3-body Pseudo-potentials are $V_{3}^{(3)}=-0.0147$, $V_{5}^{(3)}=-0.0054$, $V_{6}^{(3)}=-0.0099$, $V_{7}^{(3)}=0.0005$, and $V_{8}^{(3)}=-0.0009$ in the unit of $e^{2}/(\epsilon\ell_{0})$. The pseudopotential $V_{9}^{(3)}$ studied in Ref. Rezayi (2017) has not been considered here as it is previously shown that it does not make any qualitative change Das  _et al._ (2022) as far the occurence of $\mathcal{A}$-phase is concerned.
  * Rezayi  _et al._ (2021) E. H. Rezayi, K. Pakrouski, and F. D. M. Haldane, Phys. Rev. B 104, L081407 (2021).
  * Faugno  _et al._ (2021) W. N. Faugno, T. Zhao, A. C. Balram, T. Jolicoeur, and J. K. Jain, Phys. Rev. B 103, 085303 (2021).
  * Hutasoit  _et al._ (2017) J. A. Hutasoit, A. C. Balram, S. Mukherjee, Y.-H. Wu, S. S. Mandal, A. Wójs, V. Cheianov, and J. K. Jain, Phys. Rev. B 95, 125302 (2017).
  * Kane  _et al._ (1994) C. L. Kane, M. P. A. Fisher, and J. Polchinski, Phys. Rev. Lett. 72, 4129 (1994).
  * Bid  _et al._ (2010) A. Bid, N. Ofek, H. Inoue, M. Heiblum, C. L. Kane, V. Umansky, and D. Mahalu, Nature 466, 585 (2010).
  * Cappelli  _et al._ (2001) A. Cappelli, L. S. Georgiev, and I. T. Todorov, Nuclear Physics B 599, 499 (2001).