# Learning to Optimize for Reinforcement Learning

Qingfeng Lan
Department of Computing Science
University of Alberta
<EMAIL_ADDRESS>
&A. Rupam Mahmood
Department of Computing Science
University of Alberta
CIFAR AI Chair, Amii
<EMAIL_ADDRESS>
Shuicheng Yan
Sea AI Lab
<EMAIL_ADDRESS>&Zhongwen Xu
Sea AI Lab
<EMAIL_ADDRESS>
###### Abstract

In recent years, by leveraging more data, computation, and diverse tasks,
learned optimizers have achieved remarkable success in supervised learning,
outperforming classical hand-designed optimizers. Reinforcement learning (RL)
is essentially different from supervised learning and in practice these
learned optimizers do not work well even in simple RL tasks. We investigate
this phenomenon and identity three issues. First, the gradients of an RL agent
vary across a wide range in logarithms while their absolute values are in a
small range, making neural networks hard to obtain accurate parameter updates.
Second, the agent-gradient distribution is non-independent and identically
distributed, leading to inefficient meta-training. Finally, due to highly
stochastic agent-environment interactions, the agent-gradients have high bias
and variance, which increase the difficulty of learning an optimizer for RL.
We propose gradient processing, pipeline training, and a novel optimizer
structure with good inductive bias to address these issues. By applying these
techniques, for the first time, we show that learning an optimizer for RL from
scratch is possible. Although only trained in toy tasks, our learned optimizer
can generalize to unseen complex tasks in Brax.

##  1 Introduction

Deep learning has achieved great success in many areas (LeCun et al., 2015),
which is largely attributed to the automatically learned features that surpass
handcrafted expert features. The use of gradient descent enables automatic
adjustments of parameters within a model, yielding highly effective features.
Despite these advancements, as another important component in deep learning,
optimizers are still largely hand-designed and heavily reliant on expert
knowledge. To reduce the burden of hand-designing optimizers, researchers
propose to learn to optimize with the help of meta-learning (Sutton, 1992;
Andrychowicz et al., 2016; Wichrowska et al., 2017; Maheswaranathan et al.,
2021). Compared to designing optimizers with human expert knowledge, learning
an optimizer is a data-driven approach, reducing the reliance on expert
knowledge. During training, a learned optimizer can be optimized to speed
learning and help achieve better optimization performance.

Despite the significant progress in learning optimizers, previous works only
present learned optimizers for supervised learning (SL). These learned
optimizers usually have complex neural network structures and incorporate
numerous human-designed input features, requiring a large amount of
computation and human effort to design and train them. Moreover, they have
been shown to perform poorly in reinforcement learning (RL) tasks (Metz et
al., 2020b ; Metz et al., 2022b ). Learning to optimize for RL remains an open
and challenging problem.

Classical optimizers are typically designed for optimization in SL tasks and
then applied to RL tasks. However, RL tasks possess unique properties largely
overlooked by classical optimizers. For example, unlike SL, the input
distribution of an RL agent is non-stationary and non-independent and
identically distributed (non-iid) due to locally correlated transition
dynamics (Alt et al., 2019). Additionally, due to policy and value iterations,
the target function and the loss landscapes in RL are constantly changing
throughout the learning process, resulting in a much more unstable and complex
optimization process. In some cases, these properties also make it
inappropriate to apply optimization algorithms designed for SL to RL directly,
such as stale accumulated gradients (Bengio et al., 2020a ) or unique
interference-generalization phenomenon (Bengio et al., 2020b ). We still lack
optimizers specifically designed for RL tasks.

In this work, we aim to learn optimizers for RL. Instead of manually designing
optimizers by studying RL optimization, we apply meta-learning to learn
optimizers from data generated in the agent-environment interactions. We first
investigate the failure of traditional learned optimizers in RL tasks and find
that the complicated agent-gradient distribution impedes the training of
learned optimizers for RL. Specifically, it is difficult to obtain accurate
parameter updates due to the logarithmic variation of agent-gradients across a
wide range, despite their small absolute values. Furthermore, the non-iid
nature of the agent-gradient distribution also hinders meta-training. Lastly,
the highly stochastic agent-environment interactions can lead to agent-
gradients with high bias and variance, exacerbating the difficulty of learning
an optimizer for RL. In response to these challenges, we propose a novel
approach, Optim4RL, a learned optimizer for RL that involves gradient
processing, pipeline training, and a specialized optimizer structure with good
inductive bias. Compared with previous works, Optim4RL is more stable to train
and more effective in optimizing RL tasks, without complex optimizer
structures or numerous human-designed input features. We demonstrate that
Optim4RL can learn to optimize RL tasks from scratch and generalize to unseen
tasks. Our work is the first to propose a learned optimizer for deep RL tasks
that works well in practice.

##  2 Background

###  2.1 Reinforcement Learning

The process of reinforcement learning (RL) can be formalized as a Markov
decision process (MDP). Formally, let
$M=(\mathcal{S},\mathcal{A},\mathrm{P},r,\gamma)$ be an MDP which includes a
state space $\mathcal{S}$, an action space $\mathcal{A}$, a state transition
probability function
$\mathrm{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$,
a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and a
discount factor $\gamma\in[0,1)$. At each time-step $t$, the agent observes a
state $S_{t}\in\mathcal{S}$ and samples an action $A_{t}$ by the policy
$\pi(\cdot|S_{t})$. Then it observes the next state $S_{t+1}\in\mathcal{S}$
according to $\mathrm{P}$ and receives a scalar reward
$R_{t+1}=r(S_{t},A_{t})$. The return is defined as the weighted sum of
rewards, i.e., $G_{t}=\sum_{k=t}^{\infty}\gamma^{k-t}R_{k+1}$. The state-value
function $v_{\pi}(s)$ is defined as the expected return starting from a state
$s$. The agent aims to find an optimal policy $\pi^{*}$ to maximize the
expected return.

Proximal policy optimization (PPO) (Schulman et al., 2017) and advantage
actor-critic (A2C) (Mnih et al., 2016) are two widely used RL algorithms for
continuous control. PPO improves training stability by using a clipped
surrogate objective to prevent the policy from changing too much at each time
step. A2C is a variant of actor-critic method (Sutton and Barto, 2018),
featured with multiple-actor parallel training and synchronized gradient
update. In both algorithms, $v_{\pi}$ is approximated with $v$ which is
usually parameterized as a neural network. In practice, temporal difference
(TD) learning is applied to approximate $v_{\pi}$:

$v(S_{t})\leftarrow v(S_{t})+\alpha(R_{t+1}+\gamma v(S_{t+1})-v(S_{t})),$ (1)

where $\alpha$ is the learning rate, $S_{t}$ and $S_{t+1}$ are two successive
states, and $R_{t+1}+\gamma v(S_{t+1})$ is named the TD target. TD targets are
usually biased, non-stationary, and noisy due to changing state-values,
complex state transitions, and noisy reward signals (Schulman et al., 2016).
They usually induce a changing loss landscape that evolves during training. As
a result, the agent-gradients usually have high bias and variance which can
lead to sub-optimal performance or even a failure of convergence.

###  2.2 Learning to Optimize with Meta-Learning

We aim to learn an optimizer using meta-learning. Let $\theta$ be the agent-
parameters of an RL agent that we aim to optimize. A (learned) optimizer is
defined as an update function $U$ which maps input gradients to parameter
updates, implemented as a meta-network, parameterized by the meta-parameters
$\phi$. Let $z$ be the input of this meta-network which may include gradients
$g$, losses $L$, exponential moving average of gradients, etc. Let $h$ be an
optimizer state which stores historical values. We can then compute agent-
parameters updates $\Delta\theta$ and the updated agent-parameters
$\theta^{\prime}$:

$\Delta\theta,h^{\prime}=U_{\phi}(z,h)\text{ and
}\theta^{\prime}=\theta+\Delta\theta.$

Note that all classical first-order optimizers can be written in this form
with $\phi=\emptyset$. As an illustration, for RMSProp (Tieleman and Hinton,
2012), let $z=g$; $h$ is used to store the average of squared gradients. Then
$U_{\operatorname{RMSProp}}(g,h)=(-\frac{\alpha
g}{\sqrt{h^{\prime}+\epsilon}},h^{\prime})$, where $h^{\prime}=\beta
h+(1-\beta)g^{2}$, $\beta\in[0,1]$, and $\epsilon$ is a tiny positive number
for numerical stability.

Similar to Xu et al., (2020), we apply bilevel optimization to optimize
$\theta$ and $\phi$. First, we collect $M+1$ trajectories
$\mathcal{T}=\\{\tau_{i},\tau_{i+1},\cdots,\tau_{i+M-1},\tau_{i+M}\\}$. For
the inner update, we fix $\phi$ and apply multiple steps of gradient descent
updates to $\theta$ by minimizing an inner loss $L^{\text{inner}}$.
Specifically, for each trajectory $\tau_{i}\in\mathcal{T}$, we have

$\Delta\theta_{i}\propto\nabla_{\theta}L^{\text{inner}}(\tau_{i};\theta_{i},\phi)\text{
and }\theta_{i+1}=\theta_{i}+\Delta\theta_{i},$

where $\nabla_{\theta}L^{\text{inner}}$ are agent-gradients of $\theta$. By
repeating the above process for $M$ times, we get
$\theta_{i}\xrightarrow{\phi}\theta_{i+1}\cdots\xrightarrow{\phi}\theta_{i+M}$.
Here, $\theta_{i+M}$ are functions of $\phi$. For simplicity, we abuse the
notation and still use $\theta_{i+M}$. Next, we use $\tau_{i+M}$ as a
validation trajectory to optimize $\phi$ with an outer loss
$L^{\text{outer}}$:

$\Delta\phi\propto\nabla_{\phi}L^{\text{outer}}(\tau_{i+M};\theta_{i+M},\phi)\text{
and }\phi=\phi+\Delta\phi,$

where $\nabla_{\phi}L^{\text{outer}}$ are meta-gradients of $\phi$. Since
$\theta_{i+M}$ are functions of $\phi$, we can apply the chain rule to compute
meta-gradients $\nabla_{\phi}L^{\text{outer}}$, with the help of automatic
differentiation packages.

##  3 Related Work

Our work is closely related to two areas: optimization in reinforcement
learning (RL) and learning to optimize in supervised learning (SL).

###  3.1 Optimization in Reinforcement Learning

Henderson et al., (2018) tested different optimizers in RL and pointed out
that classical adaptive optimizers may not always consider the complex
interactions between RL algorithms and environments. Sarigül and Avci, (2018)
benchmarked different momentum strategies in deep RL and found that Nesterov
momentum is better at generalization. Bengio et al., 2020a  took one step
further and showed that unlike SL, momentum in temporal difference (TD)
learning becomes doubly stale due to changing parameter updates and
bootstrapping. By correcting momentum in TD learning, the sample efficiency is
improved in policy evaluation. These works together indicate that it may not
always be appropriate to bring optimization methods in SL directly to RL,
without considering the unique properties in RL. Unlike these works which
hand-design new optimizers for RL, we adopt a data-driven approach and apply
meta-learning to learn an RL optimizer from data generated in the agent-
environment interactions.

###  3.2 Learning to Optimize in Supervised Learning

Initially, learning to optimize is only applied to tune the learning rate
(Jacobs, 1988; Sutton, 1992; Mahmood et al., 2012). Recently, researchers
started to learn an optimizer completely from scratch. Andrychowicz et al.,
(2016) implemented learned optimizers with long short-term memory networks
(Hochreiter and Schmidhuber, 1997) and showed that learned optimizers could
generalize to similar and unseen tasks. Li and Malik, (2017) applied a guided
policy search method to find a good optimizer. Metz et al., 2022a  developed
learned optimizers with multi-layer perceptrons which achieve a better balance
among memory, computation, and performance.

Learned optimizers are known to be hard to train. Harrison et al., (2022)
investigated the training stability of optimization algorithms with tools from
dynamical systems and proposed to improve the stability of learned optimizers
by adding adaptive nominal terms from Adam (Kingma and Ba, 2015) and AggMo
(Lucas et al., 2019). Metz et al., 2020a  trained a general-purpose optimizer
by training optimizers on thousands of tasks with a large amount of
computation. Following the same spirit, Metz et al., 2022b  continued to
perform large-scale optimizer training, leveraging more computation ($4,000$
TPU-months) and more diverse SL tasks. The learned optimizer, VeLO, requires
no hyperparameter tuning and works well on a wide range of SL tasks. VeLO is
the precious outcome of long-time research in the area of learning to
optimize, building on the wisdom and effort of many generations. Although
marking a milestone for the success of learned optimizers in SL tasks, VeLO
still performs poorly in RL tasks, as shown in Section 4.4.4 in Metz et al.,
2022b .

The failure of VeLO in RL tasks suggests that designing learned optimizers for
RL is still a challenging problem. Unlike previously mentioned works that
focus on learning optimizers for SL, we aim to learn to optimize for RL. As we
will show next, our method is simple, stable, and effective, without using
complex neural network structures or incorporating numerous human-designed
features. As far as we know, our work is the first to demonstrate the success
of learned optimizers in deep RL.

(a) At the beginning of training (b) In the middle of training (c) At the end
of training

Figure 1: Visualizations of agent-gradient distributions (a) at the beginning
of training, (b) in the middle of training, and (3) at the end of training.
All agent-gradients are collected during training A2C in big_dense_long ,
optimized by RMSProp. We compute $\log(|g|+10^{-16})$ to avoid the error of
applying $\log$ function to non-positive agent-gradients.

##  4 Learning to Optimize for RL is Much Harder

Learned optimizers for SL are infamously hard to train, suffering from high
training instability (Wichrowska et al., 2017; Metz et al., 2019; Metz et al.,
2020a ; Harrison et al., 2022). Learning an optimizer for RL is even harder.
In the following, we identify three issues in learning to optimize for RL.

###  4.1 The Agent-Gradient Distribution is Non-IID

In RL, a learned optimizer takes the agent-gradient $g$ as an input and
outputs the agent-parameter update $\Delta\theta$. To investigate the hardness
of learning an optimizer for RL, we train an A2C agent in a gridworld (i.e.
big_dense_long , see the appendix for details) with RMSProp (Tieleman and
Hinton, 2012) and collect the agent-gradients at different training stages. We
plot these agent-gradients with logarithmic $x$-axis in Figure 1. The $y$-axis
shows the probability mass in each bin. Clearly, the agent-gradient
distribution is non-iid, changing throughout the training process.
Specifically, at the beginning of training, there are two peaks in the agent-
gradient distribution. In the middle of training, most of agent-gradients are
non-zero, concentrated around $10^{-3}$. At the end of the training, a large
portion of the agent-gradients are zeros. It is well-known that a non-iid
input distribution makes training more unstable and reduces learning
performance in many settings (Ma et al., 2022; Wang et al., 2023; Khetarpal et
al., 2022). Similarly, the violation of the iid assumption would also increase
learning instability and decrease efficiency for training learned optimizers.

###  4.2 RNNs Fail to Approximate an Identity Function with Agent-Gradients as
Inputs

In this section, we conduct an experiment to verify if a learned optimizer,
represented by a recurrent neural network (RNN) model, can approximate the
simplest gradient optimization method — SGD with learning rate $1$. In this
setting, essentially the RNN model is required to approximate an identity
function with agent-gradients as inputs. Surprisingly, the RNN model fails to
accomplish this task.

To begin with, similarly to Figure 1, we first train an A2C agent in a
gridworld (i.e. big_dense_long ) and collect all agent-gradients during
training, as shown in Figure 2 (a). We train the RNN model for $1,000$ epochs
by minimizing the mean squared error $(\hat{g}-g)^{2}$, where $g$ the input
agent-gradient and $\hat{g}$ is the predicted value. The model makes a correct
prediction if
$\min((1-\epsilon^{\prime})g,(1+\epsilon^{\prime})g)\leq\hat{g}\leq\max((1-\epsilon^{\prime})g,(1+\epsilon^{\prime})g)$
where $\epsilon^{\prime}=0.1$; otherwise, $\hat{g}$ is a wrong prediction.

We measure the performance by averaged accuracy over $10$ runs, plotted as the
red curve in Figure 2 (b). Surprisingly, the accuracy is only about $75\%$,
indicating that the RNN fails to approximate the identity function well with
agent-gradients as inputs. To verify that the model has enough expressivity
and capacity to represent an identity function, we further train it with
randomly uniformly generated data. The input distribution of this pseudo
dataset is also non-iid and the data size is similar to the data size of the
agent-gradients. As shown in Figure 2 (b), the training accuracy on this
dataset (the orange curve) quickly reaches $100\%$. Together, these results
suggest that it is neither non-iid data distribution nor a lack of model
expressivity that leads to poor performance of the RNN model.

To conclude, this experiment reconfirms the difficulty of learning a good
optimizer for RL: given that it is hard for an RNN model to approximate an
identity function accurately with agent-gradients as inputs, how could we
expect to learn a much more complex parameter update function from scratch?

(a) Agent-gradient distribution (b) Approximate an identity function with RNNs

Figure 2: (a) The agent-gradient distribution of training A2C in
big_dense_long . (b) The accuracies of RNN models approximating an identity
function with original gradients, processed gradients, and random data as
inputs, respectively. Results are averaged over $10$ runs, reported with
standard errors.

###  4.3 A Vicious Spiral of Bilevel Optimization

Learning an optimizer while optimizing parameters of a model is a bilevel
optimization, suffering from high training instability (Wichrowska et al.,
2017; Metz et al., 2020a ; Harrison et al., 2022). In RL, due to highly
stochastic agent-environment interactions, the agent-gradients have high bias
and variance which make the bilevel optimization even more unstable.

Specifically, in SL, it is often assumed that the training set consists of iid
samples. However, the input data distribution in RL is non-iid, which makes
the whole training process much more unstable and complex, especially when
learning to optimize is involved. In most SL settings, true labels are
noiseless and stationary (i.e. time-invariant). For example, the true label of
a written digit $2$ in MNIST (Deng, 2012) is always $y=2$ which does not
change during training. In RL, TD learning (see Equation 1) is widely used and
TD targets play a similar role as labels in SL. Unlike labels in SL, TD
targets are biased, non-stationary, and noisy, due to highly stochastic agent-
environment interactions. This leads to a changing loss landscape that evolves
during training. Together with biased TD targets, the randomness from state
transitions, reward signals, and agent-environment interactions, makes the
bias and variance of agent-gradients relatively high. During learning to
optimize for RL, meta-gradients are afflicted with large noise induced by the
high bias and variance of agent-gradients. With noisy and inaccurate meta-
gradients, the improvement of the learned optimizer is unstable and slow.
Using a poorly performed optimizer, policy improvement is no longer
guaranteed. A poorly performed agent is unlikely to collect “high-quality”
data to boost the performance of the agent and the learned optimizer. In the
end, this bilevel optimization gets stuck in a vicious spiral: a poor
optimizer $\rightarrow$ a poor agent policy $\rightarrow$ collected data of
low-quality $\rightarrow$ a poor optimizer $\rightarrow$ $\cdots$.

Figure 3: (a) An example of pipeline training where the reset interval $m=3$
and the number of units $n=3$. All training units are reset at regular
intervals to diversify training data. (b) The optimizer network structure of
Optim4RL. $g$ is the input agent-gradient, $h_{i}$ and $h^{\prime}_{i}$ are
hidden states, $\alpha$ is the learning rate, $\epsilon$ is a small positive
constant, and $\Delta\theta$ is the parameter update.

##  5 Optim4RL: A Learned Optimizer for Reinforcement Learning

To overcome the three issues mentioned in Section 4, we propose pipeline
training, gradient processing, and a novel optimizer structure. Our optimizer
combines all three techniques, named Optim4RL, which is more robust and
sample-efficient to train in RL than previous methods.

###  5.1 Pipeline Training

As shown in Figure 1, the gradient distribution of a single agent is non-iid
during training. Generally, a good optimizer should be well-functioned under
different agent-gradient distributions in the whole training process. To make
the agent-gradient distribution more iid, we propose pipeline training.

Instead of training only one agent, we train $n$ agents in parallel, each with
its own task and optimizer state. Together the three elements form a training
unit (agent, task, optimizer state); and we have $n$ training units in total.
Let $m$ be a positive integer we call the reset interval. A complete training
interval lasts for $m$ training iterations. In Figure 3 (a), we show an
example of pipeline training with $m=n=3$. To train an optimizer effectively,
the input of the learned optimizer includes agent-gradients from all $n$
training units. Before training, we choose $n$ integers
$\\{r_{1},\cdots,r_{n}\\}$ such that they are evenly spaced over the interval
$[0,m-1]$. Then we assign $r_{i}$ to training unit $i$ for
$i\in\\{1,\cdots,n\\}$. At training iteration $t$, we reset training unit $i$
if $r_{i}\equiv t\pmod{m}$, including the agent-parameters, the task, and the
optimizer state. By resetting each training unit at regular intervals,
pipeline training guarantees that at any iteration $t$, we can access training
data across one training interval. Specifically, in Figure 3 (a), for any
training iteration $t\geq 2$, the input consists of agent-gradients from a
whole training stage. For instance, at $t=3$, the input consists of agent-
gradients from unit $1$ at the beginning of a training interval, agent-
gradients from unit $2$ at the end of a training interval, and agent-gradients
from unit $3$ in the middle of a training interval, indicated by the dashed
line in Figure 3 (a). With pipeline training, the input agent-gradients are
more diverse, spreading across a whole training interval. This makes the input
agent-gradient distribution more iid and less time-dependent. In practice,
pipeline training significantly improves training stability and efficiency.

###  5.2 Gradient Processing

In Section 4.2, we show that an RNN fails to approximate an identity function
with agent-gradients as inputs. Figure 2 show that the agent-gradients vary
across a wide range in logarithms, from $-\infty$ (we truncate it to $-16$ in
the plot) to $0$, while their absolute values are in a small range (i.e.,
$[0,1]$). This property requires a learned optimizer to distinguish small
value differences between various agent-gradients in order to generate
accurate parameter updates.

To deal with the particular agent-gradient distribution, before feeding agent-
gradients into neural networks, we apply gradient processing to generate
better gradient representations. Specifically, for each scalar gradient $g$,
we map it to a pair of scalars:

$g\rightarrow[\operatorname{sgn}(g),\log(|g|+\varepsilon)],$ (2)

where $\varepsilon=10^{-18}$. By transforming $g$ to
$[\operatorname{sgn}(g),\log(|g|+\varepsilon)]$, a neural network can recover
$g$ in theory, since this mapping is a one-to-one function. Furthermore, the
$\log$ function amplifies the value differences among small agent-gradients,
making it easier to distinguish small value differences.

To demonstrate the effectiveness of gradient processing, we apply it and
repeat the experiment in Section 4.2 with the same RNN model and training
setting. In Figure 2, we plot the training accuracy in the blue curve,
averaged over $10$ runs. Compared with the result without gradient processing,
gradient processing improves the training accuracy from $75\%$ to $90\%$! In
practice, gradient processing also significantly improves the training
stability and performance of learned optimizers.

###  5.3 Improving the Inductive Bias of Learned Optimizers

Recently, Harrison et al., (2022) proved that adding adaptive terms to learned
optimizers improves the training stability of optimizing a noisy quadratic
model. Experimentally, Harrison et al., (2022) showed that adding terms from
Adam (Kingma and Ba, 2015) and AggMo (Lucas et al., 2019) improves the
stability of learned optimizers as well. However, including human-designed
features not only makes an optimizer more complex but is also against the
spirit of learning to optimize — ideal learned optimizers should be able to
automatically learn useful features, reducing the reliance on human expert
knowledge as much as possible. Instead of incorporating terms from adaptive
optimizers directly, we design the parameter update function in a similar form
to adaptive optimizers:

$\Delta\theta=-\alpha\frac{m}{\sqrt{v+\epsilon}},$ (3)

where $\alpha$ is the learning rate, $\epsilon$ is a small positive value, and
$m$ and $v$ are the processed outputs of dual-RNNs, as shown in Figure 3 (b).

By parameterizing the parameter update function as Equation 3, we improve the
inductive bias of learned optimizers, by choosing a suitable hypothesis space
for learned optimizers and reducing the burden of approximating square root
and division for neural networks. In general, we want to learn a good
optimizer in a reasonable hypothesis space. It should be large enough to
include as many good known optimizers as possible, such as Adam (Kingma and
Ba, 2015) and RMSProp (Tieleman and Hinton, 2012). Meanwhile, it should also
rule out bad choices so that a suitable candidate can be found efficiently. An
optimizer in the form of Equation 3 meets the two requirements exactly.
Moreover, as we show in Section 4.2, it is generally hard for neural networks
to approximate mathematical operations (e.g., an identity mapping) accurately
(Telgarsky, 2017; Yarotsky, 2017; Boullé et al., 2020; Lu et al., 2021). With
Equation 3, a neural network can spend all its expressivity and capacity
learning $m$ and $v$, reducing the burden of approximating square root and
division.

Finally, we combine all three techniques and propose our method — a learned
optimizer for RL (Optim4RL). Following Andrychowicz et al., (2016), our
optimizer also operates coordinatewisely on agent-parameters so that all
agent-parameters share the same optimizer. Besides gradients, many previously
learned optimizers for SL include human-designed features as inputs, such as
moving average of gradient values at multiple timescales, moving average of
squared gradients, and Adafactor-style accumulators (Shazeer and Stern, 2018).
In theory, these features can be learned and stored in the hidden states of
RNNs in Optim4RL. So for simplicity, we only consider agent-gradients as
inputs. As we will show next, despite its simplicity, our learned optimizer
Optim4RL achieves excellent performance in many RL tasks, outperforming
several state-of-the-art learned optimizers.

##  6 Experiment

In this section, we first design experiments to verify that Optim4RL can learn
to optimize for RL from scratch. Then we investigate the generalization
ability of Optim4RL in different RL tasks. Based on the investigation, we show
how to train a general-purpose learned optimizer for RL.

Following Oh et al., (2020), we design six gridworlds: small_dense_long ,
big_sparse_short , big_sparse_long , big_dense_short , small_dense_short , and
big_dense_long . These gridworlds are designed with various properties,
different in horizons, reward functions, and state-action spaces. More details
are described in the appendix. Besides gridworlds, we also test our method in
several Brax tasks (Freeman et al., 2021). We mainly consider two RL
algorithms — A2C (Mnih et al., 2016) and PPO (Schulman et al., 2017). For all
experiments, we train A2C in gridworlds and train PPO in Brax tasks. In
Optim4RL, we use two GRUs (Cho et al., 2014) with hidden size $8$; both multi-
layer perceptrons (MLPs) have two hidden layers with size $16$. To meta-learn
optimizers, we set $M=4$ in all experiments; that is, for every outer update,
we do $4$ inner updates. We use Adam to optimize learned optimizers. For A2C,
the inner loss is the standard A2C loss, while the outer loss is the actor
loss in A2C loss. For PPO, both the inner loss and outer loss are the standard
PPO loss. More implementation details are included in the appendix.

(a) small_dense_short  (b) big_dense_long  (c) Ant

Figure 4: The optimization performance of different optimizers in three RL
tasks. Note that the performance of VeLO is estimated based on Figure 11 (a)
in Metz et al., 2022b . All other results are averaged over $10$ runs and the
shaded areas represent standard errors. Optim4RL achieves satisfactory
performance in all three tasks which is a significant achievement itself,
given that all other learned optimizers fail in big_dense_long .

###  6.1 Learning an Optimizer for RL from Scratch

We first show that it is feasible to train Optim4RL in RL tasks from scratch
and that learned optimizers for SL do not work well in RL. We consider both
classical optimizers and learned optimizers as baselines. Specifically, Adam
and RMSProp are two selected classical optimizers. For learned optimizers, we
have $\text{L2LGD}^{2}$ (Andrychowicz et al., 2016), a classical learned
optimizer for SL; STAR (Harrison et al., 2022) and VeLO (Metz et al., 2022b ),
two state-of-the-art learned optimizers for SL. Besides, we propose a learned
optimizer LinearOptim which has a “linear” parameter update function:
$\Delta\theta=-\alpha(a*g+b)$, where $\alpha$ is the learning rate, $a$ and
$b$ are the processed outputs of a RNN model. The only difference between
LinearOptim and Optim4RL is the inductive bias — the parameter update function
of LinearOptim is in the form of a linear function but the parameter update
function of Optim4RL is inspired by adaptive optimizers (see Equation 3).

We select two simple gridworlds (small_dense_short and big_dense_long ) and
one complex Brax task (Ant) as test environments. Except for VeLO, we meta-
learn optimizers in one task and then test the fixed learned optimizers in
this specific task. The optimization performance of different optimizers is
measured by returns averaging over $10$ runs, as shown in Figure 4. In
small_dense_short , Optim4RL and STAR perform pretty well, on par with Adam
and RMSProp. However, in big_dense_long , a gridworld with a slightly larger
state-action space than small_dense_short , STAR fails to optimize effectively
— the return increases slowly as training proceeds. In both small_dense_short
and big_dense_long , LinearOptim and $\text{L2LGD}^{2}$ fail to optimize,
proving the advantage of the inductive bias of Optim4RL. In Ant, Optim4RL
performs similarly to Adam and RMSProp while significantly outperforming the
state-of-the-art optimizer — VeLO. While all other learned optimizer fail in
big_dense_long , the success of training Optim4RL in RL tasks from scratch is
a significant accomplishment in its own right, as it demonstrates the efficacy
of our approach and its potential for practical applications.

(a) ur5e (b) Humanoid (c) Ant (d) Grasp

(e) Dist. in ur5e (f) Dist. in Humanoid (g) Dist. in Ant (h) Dist. in Grasp

Figure 5: (a-d) The optimization performance of different optimizers in four
Brax tasks, averaged over $10$ runs. The shaded areas show standard errors.
The performance of VeLO is estimated based on Figure 11 (a) in Metz et al.,
2022b . (e-h) The agent-gradient distributions in four Brax tasks.

###  6.2 Toward a General-Purpose Learned Optimizer for RL

To test the generalization ability of Optim4RL, first we train Optim4RL by
running A2C in big_dense_long and then apply Optim4RL to optimize PPO in four
Brax tasks. As shown in Figure 5 (a-d), Optim4RL generalizes to ur5e and
Humanoid but fails in Ant and Grasp. To investigate, we plot the distribution
of all agent-gradients in Figure 5 (e-h). Looking into Figure 5 (e), we
observe that there is a large overlap between the support of agent-gradient
distribution in ur5e and the one in big_dense_long . Same for Humanoid but not
for Ant or Grasp. This aligns with the intuition: a learned optimizer is
unlikely to perform well given unseen gradients. In other words, a learned
optimizer can only be expected to generalize to tasks with similar agent-
gradient distributions.

(a) ur5e (b) Humanoid (c) Ant (d) Grasp

Figure 6: Optim4RL shows strong generalization ability and achieves good
performance in Brax tasks, although it is only trained in six simple
gridworlds from scratch. For comparison, VeLO (Metz et al., 2022b ) is trained
for $4,000$ TPU-months with thousands of tasks but only achieves sub-optimal
performance in Ant. The above results demonstrate the generalization ability
of Optim4RL in complex unseen tasks which is a significant achievement itself,
proving the effectiveness of our approach.

The insight points out a promising approach to learning a general-purpose
optimizer for RL: meta-train learned optimizers in multiple RL tasks with
various agent-gradient distributions. The six gridworlds are designed such
that the union of all agent-gradients covers a large range. The learned
optimizers are meta-trained in these gridworlds and tested in eight Brax
tasks. We only show the results on four tasks in Figure 6 while other plots
are in the appendix. First, we observe that Optim4RL achieves satisfactory
performance in these tasks which means the generalization ability of Optim4RL
is greatly improved compared with Figure 5. Moreover, Optim4RL surpasses VeLO
(the state-of-the-art learned optimizer) in Ant significantly. This is a great
success since VeLO is trained for $4,000$ TPU-months on thousands of tasks
while Optim4RL is only trained in six toy tasks for a few GPU-hours from
scratch. Training a universally applicable learned optimizer for RL tasks is
an inherently formidable challenge. Our results demonstrate the generalization
ability of Optim4RL in complex unseen tasks which is a significant achievement
itself, proving the effectiveness of our approach.

##  7 Conclusion and Future Work

In this work, we studied the failures of learning optimizers for RL. Our
investigation reveals that agent-gradients in RL are non-iid, vary across a
wide range, and have high bias and variance. To mitigate these problems, we
introduced pipeline training, gradient processing, and a novel optimizer
structure with good inductive bias. Combining these techniques, we proposed a
learned optimizer for RL, Optim4RL, which shows significant training stability
and learning efficiency. Our Optim4RL is the first learned optimizer that can
be meta-learned to optimize RL tasks entirely from scratch. Although only
trained in toy tasks, Optim4RL showed its strong generalization ability to
unseen complex tasks.

Due to memory and computation constraints, the current result is limited since
we can only train Optim4RL in a small number of toy tasks. In addition, we
only considered two on-policy RL algorithms, A2C and PPO. In the future, by
leveraging more computation and memory, we expect to extend our approach to a
larger scale, training Optim4RL with diverse RL agents in more tasks.

## References

  * Alt et al., (2019) Alt, B., Šošić, A., and Koeppl, H. (2019).  Correlation priors for reinforcement learning.  Advances in Neural Information Processing Systems, 32.
  * Andrychowicz et al., (2016) Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., and De Freitas, N. (2016).  Learning to learn by gradient descent by gradient descent.  Advances in Neural Information Processing Systems.
  * (3) Bengio, E., Pineau, J., and Precup, D. (2020a).  Correcting momentum in temporal difference learning.  NeurIPS Workshop on Deep RL.
  * (4) Bengio, E., Pineau, J., and Precup, D. (2020b).  Interference and generalization in temporal difference learning.  In International Conference on Machine Learning.
  * Boullé et al., (2020) Boullé, N., Nakatsukasa, Y., and Townsend, A. (2020).  Rational neural networks.  Advances in Neural Information Processing Systems, 33:14243–14253.
  * Bradbury et al., (2018) Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018).  JAX: composable transformations of Python+NumPy programs.
  * Cho et al., (2014) Cho, K., van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014).  On the properties of neural machine translation: Encoder–decoder approaches.  In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation.
  * Deng, (2012) Deng, L. (2012).  The MNIST database of handwritten digit images for machine learning research.  IEEE Signal Processing Magazine.
  * Freeman et al., (2021) Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. (2021).  Brax - a differentiable physics engine for large scale rigid body simulation.
  * Harrison et al., (2022) Harrison, J., Metz, L., and Sohl-Dickstein, J. (2022).  A closer look at learned optimization: Stability, robustness, and inductive biases.  In Advances in Neural Information Processing Systems.
  * Henderson et al., (2018) Henderson, P., Romoff, J., and Pineau, J. (2018).  Where did my optimum go?: An empirical analysis of gradient descent optimization in policy gradient methods.  In The 14th European Workshop on Reinforcement Learning.
  * Hochreiter and Schmidhuber, (1997) Hochreiter, S. and Schmidhuber, J. (1997).  Long short-term memory.  Neural computation.
  * Jacobs, (1988) Jacobs, R. A. (1988).  Increased rates of convergence through learning rate adaptation.  Neural Networks.
  * Khetarpal et al., (2022) Khetarpal, K., Riemer, M., Rish, I., and Precup, D. (2022).  Towards continual reinforcement learning: A review and perspectives.  Journal of Artificial Intelligence Research.
  * Kingma and Ba, (2015) Kingma, D. P. and Ba, J. (2015).  Adam: A method for stochastic optimization.  In International Conference on Learning Representations.
  * LeCun et al., (2015) LeCun, Y., Bengio, Y., and Hinton, G. (2015).  Deep learning.  Nature.
  * Li and Malik, (2017) Li, K. and Malik, J. (2017).  Learning to optimize.  In International Conference on Learning Representations.
  * Lu et al., (2021) Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E. (2021).  Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators.  Nature Machine Intelligence.
  * Lucas et al., (2019) Lucas, J., Sun, S., Zemel, R., and Grosse, R. (2019).  Aggregated momentum: Stability through passive damping.  In International Conference on Learning Representations.
  * Ma et al., (2022) Ma, X., Zhu, J., Lin, Z., Chen, S., and Qin, Y. (2022).  A state-of-the-art survey on solving non-iid data in federated learning.  Future Generation Computer Systems.
  * Maheswaranathan et al., (2021) Maheswaranathan, N., Sussillo, D., Metz, L., Sun, R., and Sohl-Dickstein, J. (2021).  Reverse engineering learned optimizers reveals known and novel mechanisms.  Advances in Neural Information Processing Systems.
  * Mahmood et al., (2012) Mahmood, A. R., Sutton, R. S., Degris, T., and Pilarski, P. M. (2012).  Tuning-free step-size adaptation.  In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
  * (23) Metz, L., Freeman, C. D., Harrison, J., Maheswaranathan, N., and Sohl-Dickstein, J. (2022a).  Practical tradeoffs between memory, compute, and performance in learned optimizers.  In Conference on Lifelong Learning Agents.
  * (24) Metz, L., Harrison, J., Freeman, C. D., Merchant, A., Beyer, L., Bradbury, J., Agrawal, N., Poole, B., Mordatch, I., Roberts, A., et al. (2022b).  VeLO: Training versatile learned optimizers by scaling up.  arXiv preprint arXiv:2211.09760.
  * (25) Metz, L., Maheswaranathan, N., Freeman, C. D., Poole, B., and Sohl-Dickstein, J. (2020a).  Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves.  arXiv preprint arXiv:2009.11243.
  * Metz et al., (2019) Metz, L., Maheswaranathan, N., Nixon, J., Freeman, D., and Sohl-Dickstein, J. (2019).  Understanding and correcting pathologies in the training of learned optimizers.  In International Conference on Machine Learning.
  * (27) Metz, L., Maheswaranathan, N., Sun, R., Freeman, C. D., Poole, B., and Sohl-Dickstein, J. (2020b).  Using a thousand optimization tasks to learn hyperparameter search strategies.  arXiv preprint arXiv:2002.11887.
  * Mnih et al., (2016) Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016).  Asynchronous methods for deep reinforcement learning.  In International conference on machine learning.
  * Oh et al., (2020) Oh, J., Hessel, M., Czarnecki, W. M., Xu, Z., van Hasselt, H. P., Singh, S., and Silver, D. (2020).  Discovering reinforcement learning algorithms.  Advances in Neural Information Processing Systems.
  * Sarigül and Avci, (2018) Sarigül, M. and Avci, M. (2018).  Performance comparison of different momentum techniques on deep reinforcement learning.  Journal of Information and Telecommunication.
  * Schulman et al., (2016) Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2016).  High-dimensional continuous control using generalized advantage estimation.  In International Conference on Learning Representations.
  * Schulman et al., (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017).  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347.
  * Shazeer and Stern, (2018) Shazeer, N. and Stern, M. (2018).  Adafactor: Adaptive learning rates with sublinear memory cost.  In International Conference on Machine Learning.
  * Sutton, (1992) Sutton, R. S. (1992).  Adapting bias by gradient descent: An incremental version of delta-bar-delta.  In Proceedings of the AAAI Conference on Artificial Intelligence.
  * Sutton and Barto, (2018) Sutton, R. S. and Barto, A. G. (2018).  Reinforcement Learning: An Introduction.  MIT Press, second edition.
  * Telgarsky, (2017) Telgarsky, M. (2017).  Neural networks and rational functions.  In International Conference on Machine Learning.
  * Tieleman and Hinton, (2012) Tieleman, T. and Hinton, G. (2012).  Lecture 6.5-RMSProp: Divide the gradient by a running average of its recent magnitude.  COURSERA Neural Networks Neural Networks for Machine Learning.
  * Wang et al., (2023) Wang, L., Zhang, X., Su, H., and Zhu, J. (2023).  A comprehensive survey of continual learning: Theory, method and application.  arXiv preprint arXiv:2302.00487.
  * Wichrowska et al., (2017) Wichrowska, O., Maheswaranathan, N., Hoffman, M. W., Colmenarejo, S. G., Denil, M., Freitas, N., and Sohl-Dickstein, J. (2017).  Learned optimizers that scale and generalize.  In International Conference on Machine Learning.
  * Xu et al., (2020) Xu, Z., van Hasselt, H. P., Hessel, M., Oh, J., Singh, S., and Silver, D. (2020).  Meta-gradient reinforcement learning with an objective discovered online.  Advances in Neural Information Processing Systems.
  * Yarotsky, (2017) Yarotsky, D. (2017).  Error bounds for approximations with deep ReLU networks.  Neural Networks.

##  Appendix A Pseudocode of Optim4RL

The source code will be released upon acceptance. The pseudocode of Optim4RL
is presented in Algorithm 1. Specifically, in all our experiments, we use GRUs
with hidden size $8$, MLPs with hidden sizes $[16,16]$, and
$\operatorname{ReLU}$ activation functions.

Algorithm 1 Optim4RL: An Optimizer for Reinforcement Learning

$\operatorname{RNN}_{1}$ and $\operatorname{RNN}_{2}$,
$\operatorname{MLP}_{1}$ and $\operatorname{MLP}_{2}$, hidden states $h_{1}$
and $h_{2}$, $\epsilon_{1}=10^{-18}$, $\epsilon_{2}=10^{-18}$, gradient
clipping parameter $c=1$, $b=1$, a learning rate $\alpha$, input gradient $g$.

if $c>0$ then

$g\leftarrow\operatorname{clip}(g,-c,c)$ $\triangleright$ Clip the input
gradient

end if

$g_{sign}\leftarrow\perp[\operatorname{sign}(g)]$ $\triangleright$ $\perp$
denotes the stop-gradient operation

$g_{log}\leftarrow\perp[\operatorname{log}(\left|g\right|+\epsilon_{1})]$

$g_{in}\leftarrow[g_{sign},g_{log}]$ $\triangleright$ Concatenate $g_{sign}$
and $g_{log}$

$h_{1},x_{1}\leftarrow\operatorname{RNN}_{1}(h_{1},g_{in})$ and
$o_{1}=\operatorname{MLP}_{1}(x_{1})$

$m_{sign}=\tanh(o_{1}+b)$ $\triangleright$ Add a small bias so that
$m_{sign}>0$ initially

$m_{sign}=\perp[2(m_{sign}>=0)-1-m_{sign}]+m_{sign}$ $\triangleright$ Use
straight-through s.t. $m_{sign}\in\\{-1,1\\}$

$m=g_{sign}m_{sign}\exp(o_{1})$ $\triangleright$ Compute $m$: 1st pseudo
moment estimate

$h_{2},x_{2}\leftarrow\operatorname{RNN}_{2}(h_{2},2g_{log})$ and
$o_{2}=\operatorname{MLP}_{2}(x_{2})$

$v=\exp(o_{2})$ $\triangleright$ Compute $v$: 2nd pseudo moment estimate

$\Delta\theta\leftarrow-\alpha\frac{m}{\sqrt{v+\epsilon_{2}}}$
$\triangleright$ Compute the parameter update

##  Appendix B Gridworlds

We follow Oh et al., (2020) and design $6$ gridwolds. In each gridworld, there
are $N$ objects. Each object is described as
$[r,\epsilon_{\text{term}},\epsilon_{\text{respawn}}]$. Object locations are
randomly determined at the beginning of each episode, and an object reappears
at a random location after being collected, with a probability of
$\epsilon_{\text{respawn}}$ for each time-step. The observation consists of a
tensor $\\{0,1\\}^{N\times H\times W}$, where $N$ is the number of objects,
and $H\times W$ is the size of the grid. An agent has $9$ movement actions for
adjacent positions, including staying in the same position. When the agent
collects an object, it receives the corresponding reward $r$, and the episode
terminates with a probability of $\epsilon_{\text{term}}$ associated with the
object. In the following tables, we describe each gridworld in detail.

Table 1: small_dense_short
Component | Description
---|---
Size ($H\times W$) | $4\times 6$
Objects |  $[100.0,0.0,0.5]$, $[-100.0,0.5,0.5]$
Horizon | $50$
Table 2: small_dense_long
Component | Description
---|---
Size ($H\times W$) | $6\times 4$
Objects |  $[1000.0,0.0,0.5]$, $[-1000.0,0.5,0.5]$
Horizon | $500$
Table 3: big_sparse_short
Component | Description
---|---
Size ($H\times W$) | $10\times 12$
Objects |  $2\times[100.0,0.0,0.05]$, $2\times[-100.0,0.5,0.05]$
Horizon | $50$
Table 4: big_sparse_long
Component | Description
---|---
Size ($H\times W$) | $12\times 10$
Objects |  $2\times[10.0,0.0,0.05]$, $2\times[-10.0,0.5,0.05]$
Horizon | $500$
Table 5: big_dense_short
Component | Description
---|---
Size ($H\times W$) | $9\times 13$
Objects |  $2\times[10.0,0.0,0.5]$, $2\times[-10.0,0.5,0.5]$
Horizon | $50$
Table 6: big_dense_long
Component | Description
---|---
Size ($H\times W$) | $13\times 9$
Objects |  $2\times[1.0,0.0,0.5]$, $2\times[-1.0,0.5,0.5]$
Horizon | $500$

##  Appendix C Experimental Details

In this work, we apply Jax (Bradbury et al., 2018) to do automatic
differentiation. Unless mentioned explicitly, we use $\operatorname{ReLU}$ as
the activation function. For A2C training in gridworlds, the feature net is an
MLP with hidden size $32$ for the “small” gridworlds. For the “big”
gridworlds, the feature net is a convolution neural network (CNN) with $16$
features and kernel size $2$, followed by an MLP with output size $32$. We set
$\lambda=0.95$ to compute $\lambda$-returns. The discount factor
$\gamma=0.995$. One rollout has $20$ steps. The critic loss weight is $0.5$
and the entropy weight is $0.01$. For PPO training in Brax games, we use the
same settings in Brax examples
111https://github.com/google/brax/blob/main/notebooks/training.ipynb. Other
details are presented in the following.

###  C.1 Computation Resources

All our experiments can be trained with V100 GPUs. For some experiments, we
use $4$ V100 GPUs due to a large GPU memory requirement. The computation to
repeat all experimental results in this work is not high (less $10$ GPU
years). However, the exact number of used computation is hard to estimate.

###  C.2 Implementation Details for Section 4

We collect agent-gradients by training A2C in big_dense_long for $30M$ steps
with learning rate $3e-3$, optimized by RMSProp. To approximate the identity
function, the RNN model is an LSTM with hidden size $8$, followed by an MLP
with hidden sizes $[16,16]$. We train the RNN model for $1,000$ epochs and
pick the best learning rate from $\\{3e-3,1e-3,3e-4,1e-4\\}$.

###  C.3 Implementation Details for Section 6.1

For both LinearOptim and $\text{L2LGD}^{2}$, the model consists a GRU with
hidden size $8$, followed by an MLP with hidden sizes $[16,16]$. For STAR, we
use the official implementation from learned_optimization
222https://github.com/google/learned_optimization/blob/main/learned_optimization/learned_optimizers/adafac_nominal.py.

The agent learning rate in small_dense_short and big_dense_long are set to
$1e-2$ and $3e-3$, respectively. The reset interval is chosen from
$[256,512]$. We use Adam as the meta optimizer and choose the meta learning
rate from $\\{3e-5,1e-4,3e-4,1e-3\\}$. For STAR, we pick step multiplicator
from $\\{3e-3,1e-3,3e-4\\}$, nominal step-size from
$\\{3e-3,1e-3,3e-4,0.0\\}$, and weight decay from $\\{0.0,0.1,0.5\\}$. For
Adam and RMSProp, we pick the best agent learning rate from
$\\{3e-2,1e-2,3e-3,1e-3,3e-4,1e-4\\}$.

For training Optim4RL in Ant from scratch, in order to reduce memory
requirement, we set the number of mini-batches from the default value $32$ to
$8$. The hidden sizes of the value network are changed from
$[256,256,256,256,256]$ to $[64,64,64,64,64]$. The reset interval is chosen
from $\\{512,1024\\}$.

###  C.4 Implementation Details for Section 6.2

Optim4RL is meta-trained in $6$ gridworlds and then tested in $8$ gridworlds.
We use Adam as the meta optimizer and choose the meta learning rate from
$\\{3e-5,1e-4,3e-4,1e-3\\}$. The reset interval is chosen from
$\\{256,512,1024\\}$. In Figure 7, we present results in $8$ Brax games.

(a) HalfCheetah (b) Pusher (c) Fetch (d) Reacher

(e) ur5e (f) Humanoid (g) Ant (h) Grasp

Figure 7: Optim4RL shows strong generalization ability and achieves good
performance in Brax tasks, although it is only trained in six simple
gridworlds from scratch. For comparison, VeLO (Metz et al., 2022b ) is trained
for $4,000$ TPU-months with thousands of tasks but only achieves sub-optimal
performance in Ant. The above results demonstrate the generalization ability
of Optim4RL in complex unseen tasks which is a significant achievement itself,
proving the effectiveness of our approach.


# Strong correlation, Bloch bundle topology, and spinless Haldane-Hubbard
model

Ilya Ivantsov  Bogoliubov Laboratory of Theoretical Physics, Joint Institute
for Nuclear Research, Dubna, Russia  Alvaro Ferraz  International Institute of
Physics - UFRN, Department of Experimental and Theoretical Physics - UFRN,
Natal, Brazil  Evgenii Kochetov  Bogoliubov Laboratory of Theoretical Physics,
Joint Institute for Nuclear Research, Dubna, Russia

###### Abstract

Different realizations of the Hubbard operators in different Hilbert spaces
give rise to various microscopic lattice electron models driven by strong
correlations. In terms of the Gutzwiller projected electron operators, the
most familiar examples are the conventional $t-J$ model and the BCS-Hubbard
model at strong coupling. We focus instead on the spin-dopon representation of
the Hubbard operators. In this case the no double occupancy (NDO) constraint
can be reexpressed as a Kondo interaction. As an explicit example, the
effective low-energy action is derived in terms of itinerant spinless fermions
(dopons) strongly interacting with localized lattice spins. The spontaneous
breaking of the time-reversal symmetry reduces that action to the one
describing a spinless version of the Haldane-Hubbard topological theory. Our
consideration suggests that the topologically nontrivial $U(1)$ Bloch bundle
associated with this model can in principle be realized dynamically due to the
presence of strong correlations even in the absence of any external local or
global fluxes.

##  I Introduction

The key point in constructing lattice electron models that incorporate strong
electron correlations, is the local NDO constraint that reduces the original
$4d$ Hilbert space to a $3d$ on-site space. This constraint is an identity in
terms of the Hubbard operators, $X$:

$X_{i}^{\uparrow\uparrow}+X_{i}^{\downarrow\downarrow}+X_{i}^{00}=1,$ (1)

where $X^{pq}=|p\rangle\langle q|,$ with $p,q=\sigma,0$ and
$\sigma=\uparrow,\downarrow.$ The Hubbard $X$ operators act in the $3d$ on-
site Hilbert space spanned by the spin up and down states $|\sigma\rangle$ and
by the vacuum state, $|0\rangle.$ The $X$ operators provide the faithful (one-
to-one, isomorphic) representation of the $u(2|1$) algebra generators wieg .
In algebraic language, Eq.(1) is just the first Casimir operator in the
fundamental $u(2|1)$ representation which fixes its dimension. Since the $X$
operators fulfil the complicated (anti)commutation relations of the $u(2|1)$
(super)algebra, one usually proceeds, in practical calculations, by first
attempting to replace the Hubbard operators by suitable combinations of
operators with simplest commutation relations. To illustrate this point,
consider the homomorphic mapping

$\phi:X^{\sigma
0}_{i}\to\tilde{c}^{\dagger}_{i\sigma}=c^{\dagger}_{i\sigma}(1-n_{i-\sigma}),$

where $c^{\dagger}_{i\sigma}$ is the creation operator of electron on a site
$i$ with spin projection $\sigma$ and
$n_{i}=\sum_{\sigma}c^{\dagger}_{i\sigma}c_{i\sigma}.$ Although the Gutzwiller
projector operators $\tilde{c}_{\sigma}$ fulfil the commutation relations of
the Hubbard ($u(2|1)$) superalgebra, they are not isomorphic to the Hubbard
operators. The $X^{\prime}s$ and $\tilde{c}^{\prime}s$ operators display
different number of degrees of freedom. As a matter of fact the homomorphic
image of the first Casimir operator in these new variables becomes,

$\phi(X_{i}^{\uparrow\uparrow}+X_{i}^{\downarrow\downarrow}+X_{i}^{00})=\sum_{\sigma}\tilde{c}^{\dagger}_{i\sigma}\tilde{c}_{i\sigma}+\tilde{c}_{i\uparrow}\tilde{c}^{\dagger}_{i\uparrow}=1-n_{i\uparrow}n_{i\downarrow}.$

It is clear that the projected electron operators establish a faithful
representation of the $u(2|1)$ algebra, or in other words, they become
isomorphic to the Hubbard operators, provided the local NDO constraint holds,

$n_{i\uparrow}n_{i\downarrow}=0.$ (2)

Only under this condition one can write
$\tilde{c}^{\dagger}_{i\sigma}=X^{\sigma 0}_{i}.$ From now on we identify the
$\tilde{c}_{i}$ operators with the corresponding Hubbard operators implying
that they both act in a $3d$ on-site Hilbert space singled out by Eq.(2).

The focus of this paper is primarily theoretical. We review both the Hubbard
and the generalized Kondo models which are shown to be equivalent to each
other at the infinitely strong coupling but generate quite different physics
out of this extreme regime. We discuss how this phenomenon might be used to
explain the origin of charge-order effects in high temperature superconductors
and the origin of magnetic ordering at very low doping. We end up by showing
that, quite surprisingly, the topologically non-trivial U(1) Bloch bundle and
the associated spinless Haldane-Hubbard model can be realized dynamically in
the strong-coupling regime even if external fluxes are not present in the
physical system.

##  II Preliminaries

Let us start our analysis with the conventional Hubbard model that describes
lattice electrons hopping in a regular lattice in any dimension subjected to a
on-site Coulomb repulsion:

$H_{U}=-t\sum_{ij\sigma}c_{i\sigma}^{\dagger}c_{j\sigma}+U\sum_{i}n_{i\uparrow}n_{i\downarrow},$
(3)

where $t$ is the hopping amplitude between nearest neighbouring (nn) sites.,
and $U$ is the on-site Coulomb repulsion. Consider now the $U=+\infty$ limit.
Since the on-site operator $n_{i\uparrow}n_{i\downarrow}$ possesses only non-
negative eigenvalues, the infinitely strong $U$-coupling enforces the local
NDO constraint, i.e., $n_{i\uparrow}n_{i\downarrow}=0.$ In this limit, it then
follows that

$H_{U=\infty}=-t\sum_{ij,\sigma}\tilde{c}^{\dagger}_{i\sigma}\tilde{c}_{j\sigma}=-t\sum_{ij,\sigma}X^{\sigma
0}_{i}X^{0\sigma}_{j}.$ (4)

There are many other Hubbard algebra representations, the most popular being
the so-called oscillatory slave-particle representations. The term
”oscillatory” refers to the fact that those representations are constructed in
terms of bilinear boson/fermion combinations. Within a standard slave-boson
representation such as

$\phi(X_{i}^{\sigma 0})=f^{\dagger}_{i\sigma}b_{i},$

$f_{i\sigma}$ stands for a fermion spinful operator and $b_{i}$ denotes a
boson operator that keeps track of the charge degrees of freedom. This
homomorphic mapping becomes again one-to-one, provided the NDO constraint
holds. It takes now the form
$\sum_{\sigma}f^{\dagger}_{i\sigma}f_{\sigma}+b^{\dagger}_{i}b_{i}=1$. However
in contrast with the representation (2) this constraint cannot be lifted to
the action by adding a new term with a global large coupling as suggested by
Eq.(3).

Among several possible Hubbard algebra representations there is one that is
most appropriate to address the physics close to half-filling. In that regime,
there are localized lattice electrons that carry spin degrees of freedom as
well as a small number of conduction fermions – dopons – which are essentially
vacancies hopping along the lattice. Exactly the same degrees of freedom are
relevant for a lattice Kondo model, which suggests that there is a close
relationship between these two model Hamiltonians in this regime. A starting
point is therefore to establish a representation of the Hubbard operators in
terms of these localized and itinerant degrees of freedom. Such a
representation was proposed in wen ,

$\displaystyle\phi(X^{\uparrow 0}_{i},X^{\downarrow
0}_{i})^{T}=\frac{1}{\sqrt{2}}(\frac{1}{2}-2\vec{S}_{i}\cdot\vec{\tau})\tilde{d}_{i},$
(5)

where T means the transpose and $\vec{\tau}$ is the set of Pauli matrices
normalized by the condition $\vec{\tau}^{2}=3/4$. In this framework, the
localized electron is represented by the lattice spin $\vec{S}\in su(2)$
whereas the mobile doped hole (dopon) is described by the projected hole
operator, $\tilde{d}_{i\sigma}=d_{i\sigma}(1-n^{d}_{i-\sigma})$. Here
$\tilde{d}=(\tilde{d}_{\uparrow},\tilde{d}_{\downarrow})^{t}$. The local dopon
number operator is denoted as $n^{d}_{i\sigma}.$ In principle, the ”tilde”
sign over the dopon operators can now be dropped, as that would cause an error
of order $O(n_{d}^{2})$ which is unimportant in the underdoped regime. More
details can be found in ifk2 .

The set of operators (5) becomes identical to the full set of the Hubbard
operators, provided pfk

$\displaystyle\vec{S_{i}}\cdot\vec{s_{i}}+\frac{3}{4}n^{d}_{i}=0.$ (6)

Here $\vec{s}_{i}$ stands for a spin dopon operator that interacts with the
local spin operator through the Kondo-type coupling. It is important that the
operator on the left hand side of Eq. (6) possesses only non-negative
eigenvalues, which implies that the local NDO constraint (6) can be imposed by
adding to the Hamiltonian the term with global coupling constant,
$\lambda\sum_{i}(\vec{S_{i}}\cdot\vec{s_{i}}+\frac{3}{4}n^{d}_{i}),\,\,\lambda\to+\infty.$

In view of that the hopping of the constrained electrons takes on a form of
the modified Kondo model (MKM),

$\displaystyle
H_{\lambda}\,=\,2t\sum_{ij\sigma}{d}_{i\sigma}^{\dagger}{d}_{j\sigma}+\lambda\sum_{i}(\vec{S_{i}}\cdot\vec{s_{i}}+\frac{3}{4}n^{d}_{i}),$
(7)

where $\lambda/t\to\infty$. In contrast to the conventional Kondo model, there
is now an extra term, $\sim n^{d}_{i}$, that comes from the NDO constraint. It
regularizes the ground-state energy and it ensures that the theory remains
finite as $\lambda\to+\infty.$ Indeed, let us consider the canonical Kondo
model with the AF coupling $K>0$,

$H_{K}=-t\sum_{ij\sigma}{d}_{i\sigma}^{\dagger}{d}_{j\sigma}+K\sum_{i}\vec{S_{i}}\cdot\vec{s_{i}}.$
(8)

In the limit $K\gg t$, it takes the form sigrist

$H_{K}=\frac{t}{2}\sum_{ij\sigma}\tilde{c}_{i\sigma}^{\dagger}\tilde{c}_{j\sigma}-\frac{3K}{4}\sum_{i}(1-\tilde{n}_{i})+{\cal
O}(t^{2}/K),$ (9)

where
$\tilde{n}_{i}=\sum_{\sigma}\tilde{c}_{i\sigma}^{\dagger}\tilde{c}_{i\sigma}=1-\tilde{n}^{d}_{i}.$
It is clear that the ground-state energy in this limit $\sim-K$. On the other
hand, comparing this with Eq.(7) gives

$H_{\lambda=\infty}=-t\sum_{ij,\sigma}\tilde{c}^{\dagger}_{i\sigma}\tilde{c}_{j\sigma}=H_{U=\infty}.$
(10)

We thus see that both the standard Hubbard model (3) and the MKM (7) possess
the same strong- coupling limit given by Eq.(10).

##  III Hubbard physics

To uncover the physics behind both the standard Hubbard model (3) and the MKM
(7) one needs to derive both the leading as well as the next-to-leading terms
in the corresponding effective Hamiltonians. In this section, we discuss the
models of strongly correlated electrons that arise from the conventional
Hubbard model at a strong on-site Coulomb repulsion.

###  III.1 $t-J$ model

For the reader’s convenience and as a check against some well known results,
we first perform an expansion of the conventional Hubbard model (3) in inverse
powers of $U$. To go beyond already well-known results, we present here a
derivation in a general case of the so-called mass-imbalanced Hubbard model.
This model implies that the hopping amplitude $t$ is now taken to be spin-
dependent, i.e., $t\to t_{\sigma}.$ We thus write,

$H_{U}^{\uparrow\downarrow}=H_{0}+V,$ (11)

with

$H_{0}=U\sum_{i}n_{i\uparrow}n_{i\downarrow},$ (12)
$V=-\sum_{ij\sigma}t_{\sigma}c_{i\sigma}^{\dagger}c_{j\sigma}.$ (13)

In the large-$U$ limit, we can treat $V$ as a small perturbation. The ground
state of $H_{0}$ contains no doubly occupied sites and it is spanned by the
physical states given by the set:

$|O_{g}\rangle=\\{|\uparrow\rangle_{i},|\downarrow\rangle_{i},|0\rangle_{i}\\}.$

Let us now define the operator $P=\prod_{i}P_{i}$ that projects into the
ground state of $H_{0}:$

$P_{i}=|0\rangle_{i}\langle
0|_{i}+|\uparrow\rangle_{i}\langle\uparrow|_{i}+|\downarrow\rangle_{i}\langle\downarrow|_{i}.$

Making use of convetional perturbation theory, up to second order in $V$, we
can define the effective Hamiltonian:messiah

$H_{eff}=PVP+\sum_{{\phi_{n}}\neq
O_{g}}\frac{PV|\phi_{n}\rangle\langle\phi_{n}|VP}{\epsilon_{0}-\epsilon_{n}},$
(14)

where $\epsilon_{0}=0$ is the ground state energy and $|\phi_{n}\rangle$ is an
eigenstate of $H_{0}$ with eigenvalue $\epsilon_{n}.$ Here we have
$Pc^{\dagger}_{\sigma}P=|\sigma\rangle\langle 0|=X^{\sigma
0}=\tilde{c}^{\dagger}_{\sigma}$ and
$Pc_{\sigma}P=|0\rangle\langle\sigma|=X^{0\sigma}=\tilde{c}_{\sigma},$ so that

$PVP=-t\sum_{ij\sigma}\tilde{c}_{i\sigma}^{\dagger}\tilde{c}_{j\sigma}.$ (15)

For simplicity from now on we will be considering solely nearest neighbour
sites $i$ and $j$ ignoring the $3$-site interactions. In such a case we get
the following eigenstates of $H_{0}$ with eigenvalue $U:$

$|\phi_{1}\rangle=|0\rangle_{i}|\uparrow,\downarrow\rangle_{j}$

and

$|\phi_{2}\rangle=|0\rangle_{j}|\uparrow,\downarrow\rangle_{i}.$

We then have

$PV|\phi_{1}\rangle=|\uparrow\rangle_{i}|\downarrow\rangle_{j}-|\downarrow\rangle_{i}|\uparrow\rangle_{j}.$

The state $PV|\phi_{2}\rangle$ is given by the same expression with the change
$i\leftrightarrow j.$ As a result, we get

$\sum_{\phi_{n}\neq O_{g}}PV|\phi_{n}\rangle\langle\phi_{n}|VP=$
$2(X^{\uparrow\uparrow}_{i}X^{\downarrow\downarrow}_{j}-X^{\uparrow\downarrow}_{i}X^{\downarrow\uparrow}_{j}-X^{\downarrow\uparrow}_{i}X^{\uparrow\downarrow}_{j}+X^{\downarrow\downarrow}_{i}X^{\uparrow\uparrow}_{j}).$

Equation (14) then yields the $t-J$ mass-imbalanced model Hamiltonian at
arbitrary doping,

$\displaystyle H_{t-J}^{\uparrow\downarrow}$ $\displaystyle=$ $\displaystyle-
t_{\uparrow}\sum_{ij}\tilde{c}_{i\uparrow}^{\dagger}\tilde{c}_{j\uparrow}-t_{\downarrow}\sum_{ij}\tilde{c}_{i\downarrow}^{\dagger}\tilde{c}_{j\downarrow}$
(16) $\displaystyle+$
$\displaystyle\sum_{ij}\frac{2(t_{\uparrow}^{2}+t_{\downarrow}^{2})}{U}(Q^{z}_{i}\cdot
Q_{j}^{z}-\frac{\tilde{n}_{i}\tilde{n}_{j}}{4})$ $\displaystyle+$
$\displaystyle\sum_{ij}\frac{4t_{\uparrow}t_{\downarrow}}{U}(Q^{x}_{i}\cdot
Q_{j}^{x}+Q^{y}_{i}\cdot Q_{j}^{y})$

if we neglect three-site interactions. Here
$\vec{Q}_{i}=\sum_{\sigma,\sigma^{\prime}}\tilde{c}_{i\sigma}^{\dagger}\vec{\tau}_{\sigma\sigma^{\prime}}\tilde{c}_{i\sigma^{\prime}}=\sum_{\sigma,\sigma^{\prime}}{c}_{i\sigma}^{\dagger}\vec{\tau}_{\sigma\sigma^{\prime}}{c}_{i\sigma^{\prime}}$
is the local electron spin operator. At $t_{\uparrow}=t_{\downarrow}=:t$ this
model reduces to the standard $SU(2)$ invariant $t-J$ model,

$H_{t-J}=-t\sum_{ij\sigma}\tilde{c}_{i\sigma}^{\dagger}\tilde{c}_{j\sigma}+J\sum_{ij}(\vec{Q}_{i}\cdot\vec{Q}_{j}-\frac{\tilde{n}_{i}\tilde{n}_{j}}{4}),$
(17)

with the spin exchange coupling $J=4t^{2}/U.$ At $U=\infty$ the double
occupancy virtual process is totally prohibited leading to $J=0$. When
$t_{\uparrow}\neq t_{\downarrow}$ the $SU(2)$ and the time reversal symmetries
are both broken. The mass-imbalanced Hubbard model with unequal hopping
amplitudes can be implemented in an optical lattice by loading mixtures of
ultracold fermionic atoms with different masses liu ; jotzu . At half-filling
the model (16) reduces to the spin-1/2 $XXZ$ model studied in liu  by making
use of the quantum Monte Carlo method.

###  III.2 BCS-Hubbard model

As a second example, let us consider the BCS-Hubbard model,

$H_{\Delta-U}=\sum_{ij}\Delta_{ij}(c_{i\uparrow}^{\dagger}c_{j\downarrow}^{\dagger}-c_{i\downarrow}^{\dagger}c_{j\uparrow}^{\dagger}+h.c.)+U\sum_{i}n_{i\uparrow}n_{i\downarrow}.$
(18)

Here $\Delta_{ij}$ denotes a singlet pair amplitude along the nn bond. Away
from the antifferomagnetic (AF) long-range ordered phase, this Hamiltonian
describes a quantum liquid of singlet-spin pairs.

In the limit $U=\infty$, the electrons are subjected to the NDO constraint. At
large but finite $U$ we arrive at the effective Hamiltonian ($\Delta/U\ll 1)$,

$\displaystyle H_{\Delta-J-V}$ $\displaystyle=$
$\displaystyle\sum_{ij}\Delta_{ij}(\tilde{c}^{\dagger}_{i\uparrow}\tilde{c}_{j\downarrow}^{\dagger}-\tilde{c}^{\dagger}_{i\downarrow}\tilde{c}_{j\uparrow}^{\dagger}+h.c.)$
(19) $\displaystyle+$ $\displaystyle
J\sum_{ij}(\vec{Q}_{i}\cdot\vec{Q}_{j}-\frac{\tilde{n}_{i}\tilde{n}_{j}}{4})+V\sum_{ij}\tilde{n}_{i}\tilde{n}_{j},$

where the spin exchange coupling becomes $J=\Delta^{2}/2U$ and the charge
exchange $V=\Delta^{2}/U=2J.$ It is interesting to note that both spin and
charge correlations are driven by a unique parameter, the electron spin
singlet amplitude, $\Delta.$ As is well known high temperature
superconductivity (SC) can arise in a quantum spin liquid and , provided such
a state can be stabilized either by geometrical frustration or by doping. A
quasi long-range SC order was recently discovered in a doped spin liquid using
density-matrix renormalization group studies to treat the $t-J$ model with
small hole doping for long $4$ and $6$ leg cylinderskivelson . This is so far
the strongest indication of SC that has been found on the square lattice of
width $>4$.

In contrast to the approaches that display the insulating parent state as a
quantum spin liquid, the strongly coupled BCS-Hubbard model (19) reduces at
half filling $(n_{i}=1)$ to the Heisenberg AF that exhibits an ordered
magnetic ground state as observed in experiment,

$H_{\Delta-J-V}\to H_{J}=J\sum_{ij}(\vec{S}_{i}\cdot\vec{S}_{j}-\frac{1}{4}).$

The present analytical result fully agrees with what comes out by explicitly
computing the wave-function overlap via exact diagonalization in finite
clusters.park  At finite doping however the strongly coupled BCS-Hubbard model
(19) generates both the spin exchange term $\sim J$ as well as the charge
exchange interaction $\sim V$.

The derivation of the model (19) implies that the dynamics of the pairing
interaction arises from virtual states, whose energies correspond to the Mott
gap $\sim U$. In this case one might refer to an unretarded pairing, i.e. a
pairing without a glue14 , which differs from the traditional picture of a
retarded pairing interaction. For the cuprate materials, the relative weight
of the retarded and nonretarded contributions to the paring interaction still
remains an open question.

##  IV Kondo physics

In this section, we focus on the physics of strongly correlated electrons
driven by the Kondo interaction. In the underdoped cuprates, an enhancement of
the nonlocal charge-charge repulsion indicates that the Kondo physics may play
an essential role in the PG phase. Recent experiments point toward a possibly
dominant role of charge order in describing the physics of high temperature
superconductors, essentially driven by dynamically generated charge
fluctuations (see italian  and references therein).

###  IV.1 $t-V$ model

Let us now consider the mass-imbalanced MKM:

$H_{\lambda}^{\uparrow\downarrow}=H_{0}+V,$ (20)

with

$H_{0}=\lambda\sum_{i}(\vec{S_{i}}\cdot\vec{s_{i}}+\frac{3}{4}n^{d}_{i}),$
(21)

and

$V=2\sum_{ij\sigma}t_{\sigma}{d}_{i\sigma}^{\dagger}{d}_{j\sigma}$ (22)

In the large-$\lambda$ limit, we treat $V$ as a perturbation to $H_{0}$. In
the limit $\lambda\to\infty,$ the unperturbed Hamiltonian $H_{0}$ enforces the
NDO constraint to the system of the hopping dopons.

The ground state of the MKM is spanned by the states

$|O_{g}\rangle=\\{|\Uparrow 0\rangle_{i},|\Downarrow
0\rangle_{i},\frac{|\Uparrow\downarrow\rangle_{i}-|\Downarrow\uparrow\rangle_{i}}{\sqrt{2}}\\},$

making use of the notation $|\sigma a\rangle$ with
$\sigma=\Uparrow,\Downarrow$ labeling the spin projection of the lattice spins
and $a=0,\uparrow,\downarrow$ labeling the dopon states (dopon double
occupancy is not allowed).

The projector into the ground state of $H_{0}$ now reads

$P_{i}=|vac\rangle_{i}\langle vac|_{i}+|\Uparrow 0\rangle_{i}\langle\Uparrow
0|_{i}+|\Downarrow 0\rangle_{i}\langle\Downarrow 0|_{i},$

where the vacancy state is the Kondo singlet

$|vac\rangle_{i}=\frac{|\Uparrow\downarrow\rangle_{i}-|\Downarrow\uparrow\rangle_{i}}{\sqrt{2}}.$

We further get

$Pd_{\uparrow i}P=-\frac{|\Downarrow 0\rangle_{i}\langle
vac|_{i}}{\sqrt{2}},=-\tilde{c}^{\dagger}_{\downarrow i}/\sqrt{2}.$

In the same way,

$Pd_{\downarrow i}P=\frac{\tilde{c}^{\dagger}_{\uparrow
i}}{\sqrt{2}},\,\,Pd^{\dagger}_{\uparrow i}P=-\frac{\tilde{c}_{\downarrow
i}}{\sqrt{2}},\,\,Pd^{\dagger}_{\downarrow i}P=\frac{\tilde{c}_{\uparrow
i}}{\sqrt{2}}.$

As a result,

$PVP=-t\sum_{ij,\sigma}\tilde{c}^{\dagger}_{i\sigma}\tilde{c}_{j\sigma},$ (23)

which again proves (10).

At this stage the following remarks are in order. The proposed MKM differs
from the conventional Kondo model by the addition of the
$\frac{3}{4}n^{d}_{i}$ term in Eq. (21). As emphasized earlier, this extra
term makes the theory finite in the infinitely strong coupling limit and this
results in a three-fold degeneracy of the ground state of the on-site
Hamiltonian in (21). In contrast, the canonical on-site Kondo coupling
exhibits a non-degenerate ground state (the on-site spin-dopon singlet) with
the eigenvalue $-3/4K,\,\,K\gg t,$ which makes the theory unstable as
$K/t\to\infty.$

To evaluate a small departure from the infinite $\lambda$ limit, one needs to
take into account $8$ (virtual) eigenstates of $H_{0}$ with eigenvalue
$\lambda$ that contribute to Eq.(14). These are

$|\phi_{1}\rangle=|\Downarrow
0\rangle_{i}\frac{|\Uparrow\downarrow\rangle_{j}+|\Downarrow\uparrow\rangle_{j}}{\sqrt{2}},$
$|\phi_{2}\rangle=|\Uparrow
0\rangle_{i}\frac{|\Uparrow\downarrow\rangle_{j}+|\Downarrow\uparrow\rangle_{j}}{\sqrt{2}}$
$|\phi_{3}\rangle=|\Downarrow
0\rangle_{i}|\Uparrow\uparrow\rangle_{j},\quad|\phi_{4}\rangle=|\Uparrow
0\rangle_{i}|\Downarrow\downarrow\rangle_{j}.$

The other $4$ states can be obtained through the change $i\leftrightarrow j$.
Equation (14) then reduces to

$\displaystyle H_{t-V}^{\uparrow\downarrow}=$ $\displaystyle=$ $\displaystyle-
t_{\uparrow}\sum_{ij}\tilde{c}_{i\uparrow}^{\dagger}\tilde{c}_{j\uparrow}-t_{\downarrow}\sum_{ij}\tilde{c}_{i\downarrow}^{\dagger}\tilde{c}_{j\downarrow}$
(24) $\displaystyle+$
$\displaystyle\sum_{ij}\frac{t_{\uparrow}^{2}}{\lambda}(\tilde{n}_{i}\tilde{n}_{j}+\tilde{n}_{i}\tilde{n}_{j\uparrow})$
$\displaystyle+$
$\displaystyle\sum_{ij}\frac{t_{\downarrow}^{2}}{\lambda}(\tilde{n}_{i}\tilde{n}_{j}+\tilde{n}_{i}\tilde{n}_{j\downarrow}),$

where $\tilde{n}_{\sigma}=n_{\sigma}(1-n_{-\sigma}).$ This model exhibits a
$Z_{2}$ global invariance $(\uparrow\leftrightarrow\downarrow)$. It is a
useful model to describe a mixture of strongly interacting ultracold fermionic
gases in a regime where charge fluctuations play a dominant role.

At $t_{\uparrow}=t_{\downarrow}=:t$, it reduces to the $t-V$ model that
exhibits the global $SU(2)$ symmetry,

$H_{t-V}=-t\sum_{ij,\sigma}X^{\sigma
0}_{i}X^{0\sigma}_{j}+V\sum_{ij}\tilde{n}_{i}\tilde{n}_{j},$ (25)

where $\tilde{n}_{i}=\sum_{\sigma}X^{\sigma\sigma}_{i}.$ The inter-site
coupling is given by $V=3t^{2}/\lambda$ with $V/t\ll 1.$ Here we have dropped
an irrelevant constant term $\sim-VN$ where $N$ is a total number of the
electrons. Let us once again notice that in deriving (24) the three-site terms
in the $V$ term have been ignored. In the context of the Kondo lattice
physics, the model (25) was proposed earlier in hirsch ; lacroix . We show
here that it actually arises as a strong coupling limit of the modified Kondo
model (7) controlled by the NDO constraint. In contrast, in this limit the
conventional Kondo model (8) becomes unstable against $K\to\infty$. The $t-V$
model allows for the following virtual process: a dopon is transferred from a
Kondo spin-dopon singlet at site $i$ to a spin state to form a virtual triplet
state, and back to the Kondo singlet at a nearest-neighbour site $j$. Since a
spin-dopon singlet carries charge and no spin this process gives rise to an
effective charge-charge exchange interaction– a repulsion between two nearest-
neighbour states. This is in a clear contrast with the $t-J$ model where the
opposite spin singly-occupied configurations are connected by virtual
excitations to and from a doubly occupied state to produce the inter-site AF
spin exchange coupling. Within the $t-V$ model, the inter-site repulsion
emerges even if a direct Coulomb interaction is ignored. It is just the
manifestation of short-range strong electron correlations which give rise to
both spin-spin and charge-charge inter-site couplings.

The Gutzwiller projector operators and the projector operators given by Eq.(5)
are isomorphic to the Hubbard operators and, hence, to each other provided the
NDO constraint is rigorously imposed. Because of this, the Hubbard model (3)
at $U=\infty$ and the Kondo-lattice model (7) at $\lambda=\infty$ are
isomorphic to each other. They merge into one and the same limiting form given
by Eq.(4). However at finite couplings the NDO constraint is relaxed and the
isomorphism between the two sets of operator no longer holds. As a result the
Hubbard model and the MKM display different physics. Those models, at finite
couplings, are not isomorphic to each other and cannot therefore be mapped
onto each other. The spin-spin exchange $\sim J$ enhances the AF spin
correlations, whereas the inter-site repulsion $\sim V$ that arises in the MKM
at strong coupling enhances the formation of charge correlations. Indeed,
guided by the unusual behavior of the electronic entropy revealed by specific
heat measurements, it was recently proposed that the Kondo physics may play an
essential role in the formation of the PG phase in the underdoped
cuprates.cooper

##  V Effective action and topology

Direct treatment of the $t-V$ model (25) written down in terms of the Hubbard
operators poses a severe technical problem due to the complicated
(anti)commutation relations of the underlying $u(2|1)$ superalgebra. However
it is necessary to impose the local NDO constraint rigorously at the very
beginning prior to any approximation, e.g. in a mean field treatment, to avoid
incorrect physical results. Because of this we focus on the effective low-
energy action of the model in terms of the variables that resolve the NDO
constraint explicitly.

###  V.1 Effective action

In this section, we derive a low-energy effective action of the $t-V$ model
(25) in the $su(2|1)$ coherent-state (CS) basis. Such a representation is
formulated in terms of the variables that explicitly resolve the constraint.
The partition function takes the form of the CS phase-space path integral (see
Appendix B):

$Z_{t-V}=\int D\mu(z,\xi)\ e^{S_{t-V}},$ (26)

where

$D\mu(z,\xi)=\prod_{i,t}\frac{d\bar{z}_{i}(t)dz_{i}(t)}{2\pi
i(1+|z_{i}|^{2})^{2}}\,d\bar{\xi}_{i}(t)d\xi_{i}(t).$

Here $z_{i}$ is a complex number that keeps track of the spin degrees of
freedom, while $\xi_{i}$ is a complex Grassmann parameter that describes the
charge degrees of freedom. As $\xi_{i}^{2}=0$, the NDO constraint is resolved
explicitly. In contrast to the slave-particle representations, the $u(2|1)$
dynamical variables are gauge independent: no auxiliary degrees of freedom are
introduced.

The effective action

$\displaystyle S_{t-V}$ $\displaystyle=$
$\displaystyle\sum_{i}\int_{0}^{\beta}ia_{i}^{(0)}(t)-\sum_{i}\int_{0}^{\beta}\bar{\xi}_{i}\left(\partial_{t}+ia_{i}^{(0)}\right)\xi_{i}dt$
(27) $\displaystyle-$ $\displaystyle\int_{0}^{\beta}H_{t-V}dt$

involves the $u(1)$-valued connection one-form of the magnetic monopole bundle
(see Appendix A) that can formally be interpreted as a spin ”kinetic” term,

$ia^{(0)}=-\langle
z|\partial_{t}|z\rangle=\frac{1}{2}\frac{\dot{\bar{z}}z-\bar{z}\dot{z}}{1+|z|^{2}},$

with $|z\rangle$ being the $su(2)$ coherent state. This term is also
frequently referred to as the Berry connection. The dynamical part of the
action takes the form

$\displaystyle
H_{t-V}=-t\sum_{ij}\bar{\xi}_{i}\xi_{j}e^{ia_{ji}}+H.c.+V\sum_{ij}\bar{\xi}_{i}\xi_{i}\bar{\xi}_{j}\xi_{j},$
(28)

where

$a_{ij}=-i\log\langle z_{i}|z_{j}\rangle,\,\,\langle
z_{i}|z_{j}\rangle=\frac{1+\overline{z}_{i}z_{j}}{\sqrt{(1+|z_{j}|^{2})(1+|z_{i}|^{2})}}$

and $z_{i}(t)$ and $\xi_{i}(t)$ are the dynamical fields.

Under a global $SU(2)$ rotation,

$\displaystyle
z_{i}\rightarrow\frac{uz_{i}+v}{-\overline{v}z_{i}+\overline{u}},$ (29)

we get

$a_{i}^{(0)}\to a^{(0)}_{i}-\partial_{t}\theta_{i},\quad a_{ij}\rightarrow
a_{ij}+\theta_{j}-\theta_{i},$ (30)

where

$\displaystyle\theta_{i}=-i\log\sqrt{\frac{-v\overline{z}_{i}+u}{-\overline{v}z_{i}+\overline{u}}},\quad\left(\begin{array}[]{ll}u&v\\\
-\overline{v}&\overline{u}\end{array}\right)\in\mathrm{SU(2)}.$ (33)

Note that the $\theta_{i}$ is an angular variable,

$\theta_{i}=-arg(-v\bar{z}_{i}+u).$

The effective action (27) is invariant under the $SU(2)$ rotation given by
Eqs. (30) accompanied by the $U(1)$ transformation of the fermionic field,

$\xi_{i}\rightarrow e^{i\theta_{i}}\xi_{i}.$ (34)

A ”flux” through a plaquette $\sum_{plaq}a_{ij}$ generated by the $SU(2)$
transformation remains invariant under (30).

For further convenience let us consider separately a real and imaginary parts
of $a_{ij}$:

$a_{ji}=\phi_{ji}+i\chi_{ji}.$

We get

$\phi_{ji}=\frac{i}{2}\log\frac{1+\bar{z}_{i}z_{j}}{1+\bar{z}_{j}z_{i}},$

and

$\chi_{ji}=-\frac{1}{2}\log\frac{(1+\bar{z}_{i}z_{j})(1+\bar{z}_{j}z_{i})}{(1+|z_{i}|^{2})(1+|z_{j}|^{2})}.$

It can be checked that the $\phi_{ji}$ potential transforms under (29) in the
same way as the $a_{ji}$ does, i.e.

$\phi_{ji}\to\phi_{ji}+\theta_{i}-\theta_{j},$ (35)

whereas, in contrast, $\chi_{ji}$ remains intact. This transformation appears
as a gauge fixing by choosing a specific rotationally covariant frame. The
dynamical fluxes do not depend on that choice.

The $\chi_{ji}$ potential results in a renormalization of the electron hopping
amplitude. Its fluctuations are gapped and thus determine high energy physics.
Low-energy physics is governed by the fluctuations of the phase variable,

$\phi_{ji}=-arg(1+\bar{z}_{i}z_{j})=arg(1+\bar{z}_{j}z_{i}),$

defined modulo $(2\pi)$. It is clear that $\phi_{ji}=-\phi_{ij}$ The
potentials $\phi_{i}^{(0)}:=ia^{(0)}_{i}$ and $\phi_{ji}$ formally remind
those gauge fields that define a compact $U(1)$ lattice gauge theory. This is
due to the fact that both theories are formulated as quantum $U(1)$ line
bundles. The gauge potentials (local connections) in both theories transform
in the same way under a change in the local trivilization.nakahara  In our
case, such a change is caused by a rotation of the underlying base space – a
canonical transformation of the phase space.

###  V.2 Topology

In the low-energy limit, the Hamiltonian function (28) takes on the form

$\displaystyle
H_{t-V}=-\sum_{ij}t_{ij}\bar{\xi}_{i}\xi_{j}e^{i\phi_{ji}}+H.c.+V\sum_{ij}\bar{\xi}_{i}\xi_{i}\bar{\xi}_{j}\xi_{j},$
(36)

This Hamiltonian describes the interaction between the spin textures and the
conduction dopons. It may display nontrivial topology – a nonzero Chern number
– only provided that there are at least two filled bands. This is because the
Chern number of a full set of bands is zero – the corresponding vector bundle
is always trivial. We restrict ourselves to the case of two bands with two
opposite Chern numbers. To this end, let us consider a bipartite $2d$ lattice
$L$ which is a direct sum of two sublattices $A$ and $B$, i.e. $L=A\oplus B$.
We then make the following change of variables on the sublattice $B$,

$z_{i}\to-\frac{1}{\bar{z}_{i}},\quad\xi_{i}\to\xi_{i}e^{-i\theta_{i}}\quad
i\in B.$ (37)

Under this transformation, $\phi_{ji}\to\phi_{ij}+\theta_{j}-\theta_{i}$. The
CS image of the on-site electron spin operator changes sign,
$\vec{Q}_{i}\to-\vec{Q}_{i}$ (see Appendix B). The effective action
Hamiltonian function then becomes

$\displaystyle
H_{t-V}=-t_{1}\sum_{<ij>}\bar{\xi}_{i}\xi_{j}e^{i\phi_{ji}}-t_{2}\sum_{<ij>\in
A}\bar{\xi}_{i}\xi_{j}e^{i\phi_{ji}}$ (38) $\displaystyle-$ $\displaystyle
t_{2}\sum_{<ij>\in
B}\bar{\xi}_{i}\xi_{j}e^{-i\phi_{ji}}+H.c.+V\sum_{<ij>}\bar{\xi}_{i}\xi_{i}\bar{\xi}_{j}\xi_{j},$

where the symbol $<ij>$ stands for the nearest neighbour sites. The resulting
total effective action displays rotation, translation and time-reversal
symmetries. Time-reversal invariance implies that the action is invariant
under the change

$z(t)\to-1/\bar{z}(\beta-t).$

It is important that the phase $\phi_{ji}$ enters the hopping terms on the $A$
and $B$ sublattices with opposite signs. This in turn means that the fluxes
through elementary plaquettes in the $A$ and $B$ within a unit cell are
opposite to each other.

The Hall conductance $\sigma_{xy}$ by definition changes sign under the time-
reversal transformation. A nonzero value can only occur if time-reversal
invariance is broken. To realize a finite quantum Hall response, it is
therefore necessary to break time-reversal symmetry. In the Haldane model for
graphene, this is done by inserting local fluxes which sum up to zero over a
unit cell. These fluxes can be described by introducing homogeneous phase
factors in the second neighbour hopping amplitude, $t_{2}\to
t_{2}e^{\pm\phi}.$ In the dynamical model (38) this can be achieved by
enforcing the spontaneous breaking of the time-reversal symmetry:
$<\phi_{ji}>=\phi.$ Here the brackets stands for a ground-state averaging.
This results in the corresponding mean-field operator Hamiltonian:

$\displaystyle H_{t-V}$ $\displaystyle=$ $\displaystyle-t_{2}\sum_{<ij>\in
A}f^{\dagger}_{i}f_{j}e^{i\phi}-t_{2}\sum_{<ij>\in
B}f^{\dagger}_{i}f_{j}e^{-i\phi}$ (39) $\displaystyle+$ $\displaystyle
H.c.+V\sum_{<ij>}f^{\dagger}_{i}f_{i}f^{\dagger}_{j}f_{j},$

where the $f_{i}$ stands for an on-site spinless fermion operator. Since
$H_{t-V}^{*}\neq H_{t-V}$ the time-reversal symmetry is broken: the resulting
Hamiltonian acquires chirality. We may set $t_{1}=0$ since it does not
qualitatively affect topological properties of the model. At $V=0$ this model
reduces to the non-interacting spinless Haldane model considered incayssol .
It was shown that the system remains gapless if $\phi=0,\pi$. In contrast, for
complex hopping there is a gap $\sim t_{2}\sin\phi$ in the single-particle
density of states at the Fermi level which is indicative of a Chern insulator.
The corresponding Chern numbers are $\pm 1$. On the other hand at nonzero $V$
the model is referred to as the spinless Haldane – Hubbard model. It was shown
that at small $V$ the system is a topological Chern insulator whereas at
sufficiently large $V$ a CDW Mott insulator phase sets in instead.yi

A few comments concerning the physical meaning of the angle $\phi$ are in
order at this stage. In the Haldane model, the homogeneous phase factors
$t_{2}\to t_{2}e^{\pm\phi}$ can be accounted for by introducing external local
magnetic dipole moments, ordered ferromagnetically normal to the plane at the
center of each hexagonal cell of the honeycomb lattice.haldane  In contrast,
within our approach, the spontaneous symmetry breaking is supposed to be at
work. Here we propose an alternative route to show that the Haldane-Hubbard
model can at least in principle be realized dynamically even in the absence of
any local or global external fields. All this is effectively driven by strong
electron correlations.

##  VI Conclusion

In conclusion, we propose an unified approach to address distinct aspects of
strongly correlated electron systems in different contexts based on the
different representations of the NDO constraint. To set the stage we first
rederive the spin-imbalanced $t-J$ model Hamiltonian to test our approach
against well known results. To proceed further, we then consider a strong
coupling expansion of the BCS-Hubbard model. In contrast to the approaches
that display the insulating parent state of the high temperature SC as a
quantum spin liquid, the strongly coupled BCS-Hubbard model is shown to reduce
at half filling to the Heisenberg AF that exhibits an ordered magnetic ground
state as observed in experiments in lightly doped cuprates.

Our major concern here is the derivation of the microscopic $t-V$ model that
describes constrained lattice electrons subjected to the resulting dynamically
emergent short-range repulsion. Such a repulsion arises as a direct
manifestation of strong electron correlations. While the conventional $t-J$
model is related to Hubbard-model physics, the $t-V$ model is shown to be
driven by the Kondo physics instead. Only in the limit of infinitely strong
couplings both models coincide with each other but generate quite different
physics out of this extreme regime. We discuss how this phenomenon might be
used to explain the origin of charge-order effects in high temperature
superconductors and the origin of magnetic ordering at very low doping.

The low-energy limit of the $t-V$ model is shown to describe a system of
itinerant spinless fermions strongly coupled to localized lattice $su(2)$
spins. Under the assumption that there is a spontaneous breaking of time-
reversal symmetry the resulting action reduces to the spinless version of the
Haldane-Hubbard topological theory. Quite surprisingly, the topologically non-
trivial $U(1)$ Bloch bundle and the associated spinless Haldane-Hubbard model
can at least in principle be realized dynamically in the strong-coupling
regime even if external fluxes are not present in the physical system.

##  VII Appendix

###  VII.1 $su(2)$ algebra and coherent states

Consider the $su(2)$ algebra in the lowest $s=1/2$ representation:

$[S_{z},S_{\pm}]=\pm S_{\pm},\quad[S_{+},S_{-}]=2S_{z},\quad\vec{S}^{2}=3/4.$
(40)

Acting with the “lowering“ spin operator $S^{-}$ on the “highest weight“ state
$|\uparrow\rangle$ we get the normalized $su(2)$ CS parametrized by a complex
number $z$

$|z\rangle=\frac{1}{\sqrt{1+|z|^{2}}}\exp(zS^{-})|\uparrow\rangle=\frac{1}{\sqrt{1+|z|^{2}}}(|\uparrow\rangle+z|\downarrow\rangle).$
(41)

In the basis spanned by the vectors $|\uparrow\rangle,\,|\downarrow\rangle$ we
have
$S_{+}=|\uparrow\rangle|\langle\downarrow|,\,S_{-}=|\downarrow\rangle|\langle\uparrow|,\,S_{z}=\frac{1}{2}(|\uparrow\rangle|\langle\uparrow|-|\downarrow\rangle|\langle\downarrow|).$
The CS symbols of the $su(2)$ generators are then easily evaluated to be

$\displaystyle S_{+}^{cl}:$ $\displaystyle=$ $\displaystyle\langle
z|S_{+}|z\rangle=\frac{z}{1+|z|^{2}},\quad
S_{-}^{cl}=\frac{\bar{z}}{1+|z|^{2}},$ $\displaystyle S_{z}^{cl}$
$\displaystyle=$
$\displaystyle\frac{1}{2}\frac{1-|z|^{2}}{1+|z|^{2}},\quad\vec{S}_{cl}^{2}=1/4,\quad(\vec{S}^{2})_{cl}=3/4.$
(42)

In the main text, we drop the lowercase index $"cl"$ whenever this causes no
confusion.

There is a one-to-one correspondence between the $su(2)$ generators (40) and
their CS (classical) symbols given by Eqs. (42). These symbols form the
$su(2)$ algebra under the Poisson (classical) brackets,
$[S^{cl}_{z},S^{cl}_{\pm}]_{PB}=\pm S^{cl}_{\pm},$ etc, where

$[A,B]_{PB}:=i(1+|z|^{2})^{2}(\partial_{z}A\partial_{\bar{z}}B-z\bar{z}).$

Given a Hamiltonian $H=H(\vec{S})$, the corresponding imaginary time phase-
space action takes on the form,

${\cal S}_{su(2)}(\bar{z},z)=-\,\int^{\beta}_{0}\langle
z|\frac{d}{dt}+H|z\rangle dt,$ (43)

with the kinetic term being given by

$ia=-\langle
z|\frac{d}{dt}|z\rangle=\frac{1}{2}\frac{\dot{\bar{z}}z-\bar{z}\dot{z}}{1+|z|^{2}}.$

From the geometrical viewpoint, the $su(2)$ coherent states $|z\rangle$ can be
thought of as sections of the magnetic monopole bundle $P(S^{2},U(1))$, with
the U(1) connection one-form, $ia$, frequently refereed to as the Berry
connection. The base space of that bundle, the two-sphere $S^{2}=CP^{1}$,
appears as a classical phase-space of the spin, whereas its covariantly
constant sections, $|z\rangle:\,\,(d+ia)|z\rangle=0$, form a Hilbert space for
quantum spin.

###  VII.2 su$(2|1)$ superalgebra and coherent states

Acting with the “lowering“ superspin operators $X^{\downarrow\uparrow}$ and
$X^{\downarrow 0}$ on the “highest weight“ state $|\uparrow\rangle$ we get the
normalized su$(2|1)$ coherent state in the $3d$ fundamental representation,

$\displaystyle|z,\xi\rangle$ $\displaystyle=$
$\displaystyle(1+\bar{z}z+\bar{\xi}\xi)^{-1/2}\exp\left(zX^{\downarrow\uparrow}+\xi
X^{0\uparrow}\right)|\uparrow\rangle$ (44) $\displaystyle=$
$\displaystyle(1+\bar{z}z+\bar{\xi}\xi)^{-1/2}(|\uparrow\rangle+z|\downarrow\rangle+\xi|0\rangle),$

where $z$ is a complex number, and $\xi$ is a complex Grassmann parameter. The
Grassmann parameter appears here due to the fact that $X^{\downarrow 0}$ is a
fermionic operator in contrast with the operator $X^{\downarrow\uparrow}$. The
product $\xi X^{0\uparrow}$ represents therefore a bosonic quantity as
required.

At $\xi=0$, the su$(2|1)$ CS reduces to the ordinary $su(2)$ CS,
$|z,\xi=0\rangle\equiv|z\rangle$ (41), parametrized by a complex coordinate
$z\in$ CP1. In contrast, at $z=0$, it represents a pure fermionic CS.

The CS symbols of the $X$ operators, $X_{cl}=\langle z,\xi|X|z,\xi\rangle$,
are respectively

$\displaystyle X^{0\downarrow}_{cl}$ $\displaystyle=$
$\displaystyle-\frac{z\bar{\xi}}{1+|z|^{2}},\quad X^{\downarrow
0}_{cl}=-\frac{\bar{z}\xi}{1+|z|^{2}},$ $\displaystyle X^{0\uparrow}_{cl}$
$\displaystyle=$ $\displaystyle-\frac{\bar{\xi}}{1+|z|^{2}},\quad X^{\uparrow
0}_{cl}=-\frac{\xi}{1+|z|^{2}},$ $\displaystyle
Q^{+}_{cl}=X^{\uparrow\downarrow}_{cl}$ $\displaystyle=$
$\displaystyle\frac{z}{1+|z|^{2}}\left(1-\frac{\bar{\xi}\xi}{1+|z|^{2}}\right),$
$\displaystyle Q^{-}_{cl}=X^{\downarrow\uparrow}_{cl}$ $\displaystyle=$
$\displaystyle\frac{\bar{z}}{1+|z|^{2}}\left(1-\frac{\bar{\xi}\xi}{1+|z|^{2}}\right),$
$\displaystyle
Q^{z}_{cl}=\frac{1}{2}(X^{\uparrow\uparrow}_{cl}-X^{\downarrow\downarrow}_{cl})$
$\displaystyle=$
$\displaystyle\frac{1}{2}\frac{1-|z|^{2}}{1+|z|^{2}}\left(1-\frac{\bar{\xi}\xi}{1+|z|^{2}}\right).$
(45)

Given a Hamiltonian as a polinomial function of the Hubbard
operators,$H=H(X)$, the corresponding imaginary time phase-space action takes
on the form,

${\cal S}_{su(2|1)}=-\,\int^{\beta}_{0}\langle
z,\xi|\frac{d}{dt}+H(X)|z,\xi\rangle dt,$ (46)

with the kinetic term given by

$\displaystyle\langle
z,\xi|(-\frac{d}{dt})|z,\xi\rangle=\frac{1}{2}\frac{\dot{\bar{z}}z-\bar{z}\dot{z}+\dot{\bar{\xi}}\xi-\bar{\xi}\dot{\xi}}{1+|z|^{2}+\bar{\xi}\xi}.$
(47)

Substituting $H(X)$ into Eq.(46) and making the change of variables $z_{i}\to
z_{i},\,\,\,\xi_{i}\to\xi_{i}\sqrt{1+|z_{i}|^{2}}$, we are led to the
effective action (26-28).

## References

  * (1) P.B. Wiegmann, Phys. Rev. Lett. 60, 821 (1988).
  * (2) T.C. Ribeiro, and X.-G. Wen, Phys. Rev. Lett. 95, 057001 (2005); T.C. Ribeiro, and X.-G. Wen, Phys. Rev. B 74, 155113 (2006).
  * (3) I. Ivantsov, A. Ferraz, and E. Kochetov, Phys. Rev. B 95, 155115 (2017).
  * (4) A. Ferraz, E. Kochetov, and B. Uchoa, Phys. Rev. Lett. 98, 069701 (2007); R.T. Pepino, A. Ferraz, and E. Kochetov, Phys. Rev. B 77, 035130 (2008).
  * (5) M. Sigrist, H. Tsunetsugu, K. Ueda, and T.M. Rice, Phys. Rev. B 46, 13838 (1992).
  * (6) A. Messiah, Quantum Mechanics (Dover, Mineola, NY, 1999).
  * (7) G. Jotzu, M. Messer, F. Görg, D. Greif, R. Desbuquois, and T. Esslinger, Phys. Rev. Lett. 115, 073002 (2015).
  * (8) Ye-Hua Liu and Lei Wang, Phys. Rev. B 92, 235129 (2015); and references therein.
  * (9) P.W. Anderson, Science 235, 1196 (1987).
  * (10) Hong-Chen Jiang and Steven A. Kivelson, arXiv:2104.01485v1; arXiv:2105.07048v1.
  * (11) K. Park, Phys. Rev. Lett. 95, 027001 (2005).
  * (12) T.A. Maier, D. Poilblanck, and D.J. Scalapino, Phys. Rev. Lett. 100, 237001 (2008).
  * (13) R. Arpaia and G. Ghiringhelli, arXiv:2106.00731v1.
  * (14) J.E. Hirsch, Phys. Rev. B 30, 5383 (1984).
  * (15) C. Bastide and C. Lacroix, Europhys. Lett. 4, 935 (1987).
  * (16) J.R. Cooper, arXiv:2108.02108v1.
  * (17) M. Nakahara, Geometry, Topology and Physics, Second Edition, 2 ed., Taylor and Francis, (2003).
  * (18) J. Cayssol, arXiv:1310.0792v1.
  * (19) T.-C. Yi, S. Hu, E.V. Castro, and R. Mondaini, arXiv:2107.10056v1.
  * (20) F.D.M. Haldane, Phys. Rev. Lett. 61, 2015 (1988).