# The Superconducting and Pseudogap Phase Diagram of High-Tc Cuprates

E. C. Marino1, Reginaldo O. C. Junior1, Lizardo H. C. M. Nunes2,Van Sérgio
Alves3 1Instituto de Física, Universidade Federal do Rio de Janeiro, C.P.
68528, Rio de Janeiro RJ, 21941-972, Brazil
2Departamento de Ciências Naturais, Universidade Federal de São João del Rei,
36301-000 São João del Rei, MG, Brazil
3Faculdade de Física, Universidade Federal do Pará, Av. Augusto Correa 01,
Belém PA, 66075-110, Brazil.

###### Abstract

We derive analytic expressions for the critical temperatures of the
superconducting (SC) and pseudogap (PG) phases of the high-Tc cuprates, which
are in excellent agreement with the experimental data for single-layered
materials such as LSCO, Bi2201 and Hg1201. Our effective Hamiltonian, defined
in the oxygen square sub-lattices formed by the alternate hybridization of
$p_{x}$ and $p_{y}$ orbitals with the $3d$ copper orbitals, provides an
unified explanation for the $d_{x^{2}-y^{2}}$ symmetry of both the SC and PG
order parameters. Attractive and repulsive interactions involve holes of the
two different sublattices and can be derived from the spin-fermion model.
Optimal doping occurs when the chemical potential vanishes. For $N$-layered
cuprates, the growth of the optimal temperature with $N$, as well as the trend
of the SC and AF domes to superimpose, can be simply understood. Our results
for the optimal SC transition temperature are in excellent agreement with the
experiments for $N=2$ materials of the $Bi$ and $Hg$ families. For $N=3$ the
agreement is still satisfactory, while for $N>3$, it becomes poor. The
explanation for these facts allows us to suggest a method for increasing the
critical SC temperature in cuprates.

Introduction

Understanding the mechanism of high-Tc superconductivity in the cuprate
materials is, at the same time, one of the most fascinating and challenging
problems in physics. Thirty years after the experimental discovery of
superconductivity in such materials bm , we still have several fundamental
phenomenological issues of the high-Tc cuprates that cannot be properly
accounted for by an underlying theory, despite the enormous experimental and
theoretical efforts applied htsc1 ; htsc2 ; htsc3 ; htsc4 ; htsc5 . To mention
just a few of these issues, let us recall that so far, the specific analytic
expression for the curves representing the critical transition temperature as
a function of doping, namely, $T_{c}(x)$, which form the characteristic SC
domes in all high-Tc materials, is not known. Also, a theoretical framework
that could provide an accurate analytical expression for $T^{*}(x)$ is not
available.

Concerning multi-layered cuprates, an explanation is still lacking for the
fact that the optimal transition critical temperature increases as a function
of the number of adjacent $CuO_{2}$ planes, up to a point and then stabilizes,
as one can observe, for instance, in the, $Bi$, $Hg$ and $Tl$ families of
cuprates honma ; honma1 . Furthermore, it is not understood why, in multi-
layered cuprates with a number, $N$, of adjacent $CuO_{2}$ planes in the
primitive unit cell, the antiferromagnetic (AF) and superconducting (SC) domes
found in the phase diagram of such materials come closer to each other and
eventually superimpose, as we increase $N$.

In this study, we adress all the above issues and provide an explanation
thereof. For this, we start from an effective Hamiltonian, defined on the
oxygen lattice of the $CuO_{2}$ planes of the cuprates. A crucial feature of
our model is the observation that such lattice breaks down into two
inequivalent sublattices, for which the $p_{x}$ or $p_{y}$ oxygen orbitals,
respectively, overlap with the copper $3d$ orbitals. This Hamiltonian
describes the kinematics and dynamics of the holes doped into the oxygen ions.
Cooper pairs are formed by combining holes belonging to the two different
sublattices of the oxygen square lattice, which contain respectively, $p_{x}$
and $p_{y}$ orbitals. This naturally leads to a d-wave SC order parameter,
which is favored by the attractive interaction sector. A term describing the
repulsion between holes, conversely, favors the onset of a non-vanishing
d-wave PG order parameter, which results from exciton (electron-hole pair,
each belonging to a different sublattice) condensation. Hence, our model
naturally provides a unified explanation for the d-wave charater both of the
SC and PG order parameters, the latter leading to the DDW scenario ddw
proposed to explain the PG phenomena. The phase diagram of the cuprates,
hence, derives from the duality between the formation of Cooper pair and (DDW)
exciton condensates, both with a d-wave symmetry. The two effective
interaction terms contained in our Hamiltonian can be derived from a purely
magnetic interaction, namely, the spin-fermion model ecm1 , which is the
inspiration for the present approach. The doping mechanism is explicitly taken
into account by the introduction of a constraint relating the fermion number
to a function of the stoichiometric doping parameter.

Quantum dynamical effects are brought up by functional integrating out the
fermion degrees of freedom. This allows one to obtain an effective action in
terms of the superconducting order parameter, $\Delta_{0}$, the pseudogap
order parameter $M_{0}$, the chemical potential $\mu_{0}$ and the temperature.
Then, minimizing the effective potential, which corresponds, to this action we
are able to verify that the occurrence of nonzero $\Delta_{0}$ and $M_{0}$
are, in general, mutually excludent, thereby indicating a competition between
the PG and SC phases. The exception occurs when $g=g_{1}$, which is the case
when one derives the Hamiltonian (LABEL:0) from the spin-fermion system ecm1 .

By taking the limits $\Delta_{0}\rightarrow 0$ and $M_{0}\rightarrow 0$,
respectively, we capture the threshold for the SC and PG transition and
thereby arrive at an analytic expression for the critical SC and PG
temperatures as a function of doping, namely $T_{c}(x)$ and $T^{*}(x)$. This
reproduces the familiar SC domes, as well as the PG lines found in the
cuprates and is in excellent agreement with the experimental data for single-
layerd materials such as LSCO, Bi2201 and Hg1201. Our results indicate that
the optimal amount of stoichiometric doping, $x_{0}$, which leads to the
maximal $T_{c}$ occurs when the chemical potential vanishes:
$\mu_{0}(x_{0})=0$, $T_{c}(x)\leq T_{c}(x_{0})$.

The increase of the optimal temperature, as well as the tendency of the SC and
AF domes to superimpose as we increase the number of adjacent planes in the
primitive unit cell of multi-layered cuprates, can be simply understood within
our approach, as we show that the effective coupling parameter $g$ is enhanced
by the number, $N$, of such planes: $g\rightarrow Ng$.

Methods

A) The Mechanism of Doping

An outstanding feature of all High-Tc cuprates is the presence of one or more
$CuO_{2}$ planes, intercepting the primitive unit cell of such compounds. The
$CuO_{2}$ planes have a lattice structure in which $Cu^{++}$ ions occupy the
sites and oxygen ions the links of a square lattice, with a lattice parameter
${\textbf{a}}^{\prime}=3.8$Å. These ions are in a $3d^{9}$ electronic
configuration, which results in one spin 1/2 per site. The system of copper
ions is a Mott-Hubbard insulator, hence, from this point of view, it forms an
array of localized spins interacting with the nearest-neighbors through the
super-exchange mechanism. This structure is ultimately responsible for the
antiferromagnetic properties observed in the high-Tc cuprates. From the point
of view of the oxygen ions, however, the picture is different. Indeed, the
oxygen ions are themselves, placed on the sites of a square lattice, with a
lattice parameter ${\textbf{a}}=\sqrt{2}{\textbf{a}}^{\prime}/2=1.9\sqrt{2}$ Å
which possesses two sublattices, containing, respectively, $p_{x}$ and $p_{y}$
oxygen orbitals, which overlap with the $Cu^{++}$ d-orbitals (see Fig.1),
thereby forming bridges that will allow not only hole hopping along the whole
oxygen lattice, but also the formation of Cooper pairs as well as excitons
along these bridges. As we shall see, this fact naturally explains why both
the SC and PG gaps have a d-wave symmetry.

In the case of the pure parent compounds the oxygen ions are doubly charged,
namely: $O^{--}$. Such ions are in a $2p^{6}$ configuration and the $p_{x}$
and $p_{y}$ orbitals contain two electrons each. The valence band which
corresponds to the above described oxygen structure contains two electrons per
site and, therefore, is completely filled. The electron density is
$N_{e}=\frac{2}{A}$, where $A=a^{2}$. As doping is introduced, through some
stoichiometric process, parametrized by $x$, one of the two electrons, either
from the $p_{x}$ or the $p_{y}$ oxygen sublattices is pulled out of the plane,
thereby creating a hole in such orbital. Expressing the average hole density
per site in the oxygen lattice as $N_{h}=\frac{2}{A}y$, where $y\in[0,1]$, it
follows that the average electron density becomes $N_{e}=\frac{2}{A}(1-y)$.
Now, one must consider that the relation between the stoichiometric doping
parameter, $x$ and the average number of holes per site in the oxygen lattice
of the $CuO_{2}$ planes, associated to the $y$-parameter, is not universally
known, in general; usually exhibiting different forms for each of the cuprate
materials honma0 ; honma ; honma1 . Furthermore, the presence of interactions
should influence such relation. Consequently, we have the hole density
parameter, $y$, given by some non-universal function of the doping parameter:
$y=f(x)$. We typically do not know the function $f(x)$, therefore, we will
describe the doping process through a constraint relating the fermion number
directly to the stoichiometric doping parameter $x$, rather then to the
density of holes in the oxygen lattice, which is parametrized by $y$. As we
increase the doping parameter $x$, the number of holes in the oxygen lattice
will somehow increase as well, eventually reaching an amount where the
critical SC temperature reaches a maximum. We call $x_{0}$ the value of the
doping parameter for which this happens. As we will see, the chemical
potential will vanish precisely at $x=x_{0}$.

B) The Model

The underlying pairing mechanism responsible for the superconductivity in
cuprates, whatever it may be, must generate an effective hole-attractive
interaction Hamiltonian on the oxygen square lattice. A derivation of such a
term from a purely magnetic interaction was provided in ecm1 . There, the
hole-attractive interaction comes accompanied by a hole-repulsive term and the
competition of both effectively governs the electrons and holes. An attentive
analysis must consider the fact that the $Cu^{++}$ ions form bridges between
the two oxygen sublattices by overlaping the corresponding $p_{x}$ and $p_{y}$
orbitals. It follows that both the hopping and the interaction of the
corresponding oxygen holes (see Fig 1), thereby assisted by the $Cu^{++}$
ions, must involve the two different $p_{x}$ and $p_{y}$ oxygen sublattices.
We are going to use a simple Hamiltonian, inspired in ecm1 , consisting of a
hopping term, a hole-attractive and a hole-repulsive terms, that altogether
capture these features, namely,

$\displaystyle H$ $\displaystyle=H_{0}+H_{SC}+H_{PG}$ $\displaystyle H$
$\displaystyle=-t\sum_{\textbf{R},\textbf{d}_{i}}\psi_{B\sigma}^{\dagger}(\textbf{R}+\textbf{d}_{i})\psi_{A\sigma}(\textbf{R})+hc$
$\displaystyle-$ $\displaystyle
g\sum_{\textbf{R},\textbf{d}_{i}}\Big{[}\psi_{B\uparrow}^{\dagger}(\textbf{R}+\textbf{d}_{i})\psi_{A\downarrow}^{\dagger}(\textbf{R})-\psi_{B\downarrow}^{\dagger}(\textbf{R}+\textbf{d}_{i})\psi^{\dagger}_{A\uparrow}(\textbf{R})\Big{]}$
$\displaystyle\times$
$\displaystyle\Big{[}\psi_{A\downarrow}(\textbf{R})\psi_{B\uparrow}(\textbf{R}+\textbf{d}_{i})-\psi_{A\uparrow}(\textbf{R})\psi_{B\downarrow}(\textbf{R}+\textbf{d}_{i})\Big{]}+$
$\displaystyle g_{1}$
$\displaystyle\sum_{\textbf{R},\textbf{d}_{i}}\Big{[}\psi_{B,\sigma}^{\dagger}(\textbf{R}+\textbf{d}_{i})\psi_{A,\sigma^{\prime}}(\textbf{R})\Big{]}\Big{[}\psi_{A,\sigma^{\prime}}^{\dagger}(\textbf{R})\psi_{B,\sigma}(\textbf{R}+\textbf{d}_{i})\Big{]}$

In the above expressions, R denotes the sites of a square lattice and
$\textbf{d}_{i}$, $i=1,...,4$, its nearest neighbors.
$\psi_{A,B\sigma}^{\dagger}$ is the creation operator of a hole, or,
equivalently, the destruction operator of an electron, with spin
$\sigma=\uparrow,\downarrow$ in sublattice $A,B$. Such sublattices are formed
as follows: each oxygen ion possesses one $p_{x}$ and one $p_{y}$ orbitals but
only one of them hybridizes with the copper 3d orbitals, alternatively, either
$p_{x}$ or $p_{y}$. Two inequivalent oxygen sublattices are thereby formed,
one having hybridized $p_{x}$ orbitals and the other having $p_{y}$. $t$ is
the usual hopping parameter, $g$, the hole-attractive interaction coupling
parameter and $g_{1}$, the hole-repulsive coupling parameter. When this model
is derived from the spin-fermion model, one obtains the particular case where
$g=g_{1}$ ecm1 .

This can be written, up to a constant, in trilinear form, in terms of the
Hubbard-Stratonovitch fields $\Phi$ and $\chi$, namely,

$\displaystyle H$
$\displaystyle=-t\sum_{\textbf{R},\textbf{d}_{i}}\psi_{B\sigma}^{\dagger}(\textbf{R}+\textbf{d}_{i})\psi_{A\sigma}(\textbf{R})+hc$
$\displaystyle+$
$\displaystyle\sum_{\textbf{R},\textbf{d}_{i}}\Phi(\textbf{d}_{i})\Big{[}\psi_{B\uparrow}^{\dagger}(\textbf{R}+\textbf{d}_{i})\psi^{\dagger}_{A\downarrow}(\textbf{R})-\psi_{B\downarrow}^{\dagger}(\textbf{R}+\textbf{d}_{i})\psi^{\dagger}_{A\uparrow}(\textbf{R})\Big{]}$
$\displaystyle+$ $\displaystyle hc$ $\displaystyle+i$
$\displaystyle\sum_{\textbf{R},\textbf{d}_{i}}\chi(\textbf{d}_{i})\Big{[}\psi_{B,\sigma}^{\dagger}(\textbf{R}+\textbf{d}_{i})\psi_{A,\sigma^{\prime}}(\textbf{R})$
$\displaystyle-$
$\displaystyle\psi_{A,\sigma^{\prime}}^{\dagger}(\textbf{R})\psi_{B,\sigma}(\textbf{R}+\textbf{d}_{i})\Big{]}$
$\displaystyle+$
$\displaystyle\frac{1}{g}\sum_{\textbf{R},\textbf{d}_{i}}\Phi^{\dagger}(\textbf{R}+\textbf{d}_{i})\Phi(\textbf{R}+\textbf{d}_{i})+\frac{1}{2g_{1}}\sum_{\textbf{R},\textbf{d}_{i}}\chi^{2}(\textbf{R}+\textbf{d}_{i}),$

Varying with respect to $\Phi$ and $\chi$, we obtain, respectively,

$\displaystyle\Phi^{\dagger}=g\Big{[}\psi_{B\uparrow}^{\dagger}\psi^{\dagger}_{A\downarrow}-\psi_{B\downarrow}^{\dagger}\psi^{\dagger}_{A\uparrow}\Big{]}$
(3)

and

$\displaystyle\chi=ig_{1}\Big{[}\psi_{B,\sigma}^{\dagger}\psi_{A,\sigma^{\prime}}-\psi_{A,\sigma^{\prime}}^{\dagger}\psi_{B,\sigma}\Big{]}$
(4)

$\Phi^{\dagger}$ is a Cooper pair creation operator, the vacuum expectation
value of which, namely, $\Delta=\langle\Phi\rangle$, is a SC order parameter.
The PG order parameter, conversely, is $M=\langle\chi\rangle$, where $\chi$ is
an exciton creation operator. Cooper pair, as well as exciton formation
occurs, respectively, for holes-holes or electron-holes, belonging to
different sublattices.

An XY-asymmetry is naturally in-built, produced by the different signs of
overlaping $p_{x}$ and $p_{y}$ orbitals along the oxygen lattice x and y
directions (see Fig. 1). This imposes the relations
$\Delta(\textbf{d}_{x})=-\Delta(\textbf{d}_{y})$ and
$M(\textbf{d}_{x})=-M(\textbf{d}_{y})$, which, as we shall see, will yield SC
and PG order parameters with a d-wave symmetry.

Figure 1: The two sublattices of oxygen ions, formed, respectively, by $p_{x}$
and $p_{y}$ orbitals that overlap the $Cu^{++}$ d-orbital. Notice that the
$Cu^{++}$-assited overlap of $p_{x}$ and $p_{y}$ orbitals along the
$x$-direction (of the oxygen lattice) gives a positive sign, whereas the one
along the $y$-direction gives a negative sign. Black dots are the $Cu^{++}$
ions.

In momentum space, we have the corresponding Hamiltonian

$\displaystyle H$
$\displaystyle=\sum_{\textbf{k},\sigma}\epsilon(\textbf{k})\Big{[}\psi_{B\sigma}^{\dagger}(\textbf{k})\psi_{A\sigma}(\textbf{k})+hc\Big{]}$
$\displaystyle+$
$\displaystyle\sum_{\textbf{k}}\Phi(\textbf{k})\Big{[}\psi_{B\uparrow}^{\dagger}(-\textbf{k})\psi^{\dagger}_{A\downarrow}(\textbf{k})-\psi_{B\downarrow}^{\dagger}(-\textbf{k})\psi^{\dagger}_{A\uparrow}(\textbf{k})\Big{]}+hc$
$\displaystyle+i$
$\displaystyle\sum_{\textbf{k}}\chi(\textbf{k})\Big{[}\psi_{A\sigma}^{\dagger}(-\textbf{k})\psi_{B\sigma}(\textbf{k})-\psi_{B\sigma}^{\dagger}(-\textbf{k})\psi_{A\sigma}(\textbf{k})\Big{]}$
$\displaystyle+$
$\displaystyle\frac{1}{g}\sum_{\textbf{k}}\Phi^{\dagger}(-\textbf{k})\Phi(\textbf{k})+\frac{1}{2g_{1}}\sum_{\textbf{k}}\chi(-\textbf{k})\chi(\textbf{k})$

where $\epsilon(\textbf{k})=2t[\cos k_{x}a+\cos k_{y}a]$ is the usual tight-
binding energy.

Using the xy-asymmetry of $\Delta(\textbf{d}_{i})$, which is caused by the
asymmetric overlap of $p_{x}$ and $p_{y}$ orbitals described above, we obtain
accordingly

$\displaystyle\Delta(\textbf{k})=\sum^{4}_{\textbf{d}_{i}=1}\Delta(\textbf{d}_{i})e^{i\textbf{k}\cdot\textbf{d}_{i}}=2\Delta_{0}[\cos
k_{x}a-\cos k_{y}a]$ (6)

and

$\displaystyle
M(\textbf{k})=\sum^{4}_{\textbf{d}_{i}=1}M(\textbf{d}_{i})e^{i\textbf{k}\cdot\textbf{d}_{i}}=2M_{0}[\cos
k_{x}a-\cos k_{y}a],$ (7)

both of which have d-wave symmetry.

Using a mean-field approach for the Cooper pair and exciton fields, $\Phi$ and
$\chi$, we obtain the energy eigenvalues
$E(\textbf{k}))=\pm\sqrt{\epsilon^{2}(\textbf{k})+|M|^{2}(\textbf{k})+|\Delta|^{2}(\textbf{k})}$.
The eigenvalues of $H-\mu\mathcal{N}$, where $\mu$ is the chemical potential
and $\mathcal{N}$ is the number operator, conversely, are
$\mathcal{E}(\textbf{k}))=\pm\sqrt{(\sqrt{\epsilon^{2}(\textbf{k})+|M|^{2}(\textbf{k})}\pm\mu)^{2}+|\Delta|^{2}(\textbf{k})}$.

When we start to dope and, consequently introduce holes in the system, it is
natural to expect hole pockets to form around the points where
$E^{2}_{\pm}(\textbf{K})=0$, namely,
$\textbf{K}=(\pm\frac{\pi}{2a},\pm\frac{\pi}{2a})$. By making a Taylor
expansion of the energy E(k)) around the points K, it is easy to show that the
constant energy curves are the ellipses (or arcs thereof) depicted in Fig. 2.

Figure 2: Ellipses or arcs thereof with semi-axes proportional to the total
energy are the constant energy curves. For vanishing energy these reduce to
the pockets located at the four points K.

This is precisely what is observed in ARPES experiments arpes , thus puting
our model in a solid experimental basis.

Notice that all the pseudogap phenomenology, which is explained by the d-wave
gap ddw  including the time-reversal symmetry breakdown and the Nernst effect
ddw1  are accounted for by our model as well.

As we mentioned above, the specific Hamiltonian interaction we use here, can
be derived from a spin-fermion system, which describes the multiple magnetic
interactions of a system of localized and itinerant spins, ecm1 ; ecm2 . A
similar Hamiltonian is described in vv . Our elliptic constant energy curves
would coincide with the ones obtained from an asymmetric kinetic Dirac
lagrangean cms , whereas the corresponding curves obtained from an usual Dirac
lagrangean would correspond to circles. All of these must be in the same class
of universality, therefore leading to the same phase diagram.

We shall now integrate over the fermions. For this purpose, we arrange the
electron and hole fermion operators of oxygen, with spin
$\sigma=\uparrow,\downarrow$ in the form of a four-component Nambu fermion
field:

$\displaystyle\Psi_{a}=\left(\begin{array}[]{c}\psi_{A,\uparrow,a}\\\
\psi_{B,\uparrow,a}\\\ \psi^{\dagger}_{A,\downarrow,a}\\\
\psi^{\dagger}_{B,\downarrow,a}\end{array}\right)$ (12)

where the indices, A and B, respectively, denote each of the two $p_{x}$ and
$p_{y}$ sublattices.

The cuprates can be classified according to the number of adjacent $CuO_{2}$
planes contained in their primitive unit cell. The index $a$ indicates to
which of the adjacent $CuO_{2}$ planes the electrons and holes belong. This
index runs from $1$ to $N$, where $N=1,2,3,...$, according to the number of
planes intersecting the primitive unit cell of the material. In this approach,
we shall neglect interplane interactions. We introduce the $x$-dependence
through the constraint

$\displaystyle\lambda\Big{[}\sum_{a=1}^{N}\sum_{C=A,B}\psi^{\dagger}_{C,\sigma,a}\psi_{C,\sigma,a}-Nd(x)\Big{]}$
(13)

which is enforced by integrating over the Lagrange multiplier field $\lambda$,
whose vacuum expectation value is the chemical potential:
$\langle\lambda\rangle=\mu_{0}$. Here $d(x)$ is a function of the
stoichiometric doping parameter, to be determined. For consistency we must
have $d(0)=\frac{2}{A}$, where $A=a^{2}$ is the unit cell area of the oxygen
lattice: $A=2\ (1.9)^{2}$ Å2.

Conveniently integrating on the complex fields, $\Phi$ and $\chi$, which act
as Hubbard-Stratonovitch fields, the Hamiltonian then exhibits two quartic
interactions in the fermions, one attractive and another repulsive,
respectively with couplings $g$ and $g_{1}$. Conversely including the doping
constraint and performing the quadratic functional integral over the fermion
fields we obtain the effective Euclidean action $S[\Delta,M,\lambda]$,

$\displaystyle S[\Delta,M,\lambda]=\int
d^{2}\textbf{r}d\tau\Big{[}\frac{\Delta^{\dagger}\Delta}{g}$ $\displaystyle+$
$\displaystyle\frac{M^{2}}{2g_{1}}+N\lambda d(x)\Big{]}+$ (14) $\displaystyle
N$ $\displaystyle tr\ln\mathcal{A}[\Delta,M,\lambda]$

where $\mathcal{A}=i\partial_{0}-\mathcal{H}+\lambda I\otimes(-I)$. (Here $I$
is the $2\times 2$ identity matrix and
$H=\Psi^{\dagger}_{a}\mathcal{H}\Psi_{a}$).

Minimizing this, namely

$\displaystyle\left[\frac{\delta S}{\delta\Delta}\right]=\left[\frac{\delta
S}{\delta M}\right]=\left[\frac{\delta
S}{\delta\lambda}\right]_{\Delta=\Delta_{0};M=M_{0};\lambda=\mu_{0}}=0\,,$
(15)

we can determine $\Delta_{0},M_{0},\mu_{0}$.

Corresponding to (15), we find three equations, namely

$\displaystyle
2\Delta_{0}\Big{[}\frac{2T}{\alpha}F(\Delta_{0},M_{0},\mu_{0})-\Big{(}\frac{\Lambda}{\alpha}-\frac{1}{Ng}\Big{)}\Big{]}=0$
(16) $\displaystyle
2M_{0}\Big{[}\frac{2T}{\alpha}F(\Delta_{0},M_{0},\mu_{0})-\Big{(}\frac{\Lambda}{\alpha}-\frac{1}{Ng_{1}}\Big{)}\Big{]}=0$
(17)

and

$\displaystyle d(x)=\mu_{0}\frac{4T}{\alpha}F(\Delta_{0},M_{0},\mu_{0})$ (18)

where $F(\Delta_{0},M_{0},\mu_{0})$ is a function, which, in the regime where
$\Delta_{0}\sim 0,M_{0}\sim 0$ is given by

$\displaystyle F(\Delta_{0},M_{0},\mu_{0})\Huge|_{\Delta_{0}\sim 0,M_{0}\sim
0}=\ln 2$ $\displaystyle+$
$\displaystyle\frac{1}{2}\ln\cosh\Big{[}\frac{\sqrt{\Delta_{0}^{2}+(M_{0}+\mu_{0}(x))^{2}}}{2T}\Big{]}$
$\displaystyle+$
$\displaystyle\frac{1}{2}\ln\cosh\Big{[}\frac{\sqrt{\Delta_{0}^{2}+(M_{0}-\mu_{0}(x))^{2}}}{2T}\Big{]}$

In the expressions above, $\alpha=2\pi v_{eff}^{2}$, and $v_{eff}\simeq 2ta$
is the characteristic velocity and $\Lambda$ is a momentum (energy)
characteristic scale, which appears em  in connection to the characteristic
length of the system, namely, the coherence length $\xi$, which essentially
measures the range of the pairing interaction (or the Cooper pair size). In
cuprates we have $\xi\geq\xi_{0}\simeq 10$Å, whereas in conventional
superconductors $\xi\geq\xi_{0}\simeq 500$Å. The momentum (energy) cutoff is
then $\Lambda\simeq hv_{eff}/\xi_{0}=\sqrt{2\pi\alpha}/\xi_{0}$. It determines
the energy scale below which we may consider Cooper pairs as quasiparticles,
hence it must be of the order of $T_{c}$.

We see that (for $g\neq g_{1}$) it is, in general, impossible to satisfy (16)
and (17) simultaneously with both $\Delta_{0}\neq 0$ and $M_{0}\neq 0$, so we
must have either $\Delta_{0}\neq 0$ and $M_{0}=0$ or $\Delta_{0}=0$ and
$M_{0}\neq 0$. The first is the SC phase, while the second is the PG phase.
The only possibility of having both the SC and PG different from zero would be
for $g=g_{1}$, for which case, $\Delta_{0}=M_{0}\neq 0$ .

C) The SC Order Parameters and the Critical SC Temperature: $T_{c}(x)$

Let us consider firstly the case $\Delta_{0}\neq 0$ and $M_{0}=0$.

Then (16) and (18) imply

$\displaystyle\mu_{0}(x)=d(x)\frac{g_{c}}{2\eta(gN)}\ \ \ ;\ \ \
\eta(gN)=\frac{Ng-g_{c}}{Ng}$ (20)

where $g_{c}=\alpha/\Lambda$.

In order to find the critical temperature $T_{c}(x)$, we impose on (21) the
condition $\Delta_{0}=0$ and $M_{0}=0$, which express the fact that the system
is in one of the points belonging to the critical curve which separates the SC
and PG phases. Indeed, from (16), we obtain

$\displaystyle T_{c}(x)=\lim_{\Delta_{0}\rightarrow
0}\frac{\frac{\alpha\eta(gN)}{2g_{c}}}{F(\Delta_{0},M_{0}=0,\mu_{0}(x))}.$
(21)

From (21), we see that, for $\Delta_{0}=0$ and $M_{0}=0$, the upper bound of
$T_{c}(x)$ occurs at a point $x=x_{0}$, where $\mu_{0}(x_{0})=0$ and
$T_{max}=T_{c}(x_{0})$. Optimal doping occurs when the chemical potential
vanishes. According to (20), this implies $d(x_{0})=0$. The simplest
parametrization satisfying this and $d(0)=\frac{2}{A}$ is
$d(x)=\frac{2}{Ax_{0}}(x_{0}-x)$, such that

$\mu_{0}=2\gamma(gN)(x_{0}-x),$ (22)

with

$\gamma(gN)=\frac{g_{c}}{2Ax_{0}\eta(gN)}.$ (23)

This combined with (21) allows us to express the optimal temperature as

$\displaystyle T_{max}=\frac{\alpha}{2\ln 2}\frac{\eta(gN)}{g_{c}}$ (24)

The critical curve delimiting the boundary of the SC phase, consequently, is
obtained from (21) and yields

$\displaystyle\frac{T_{c}(x)}{\ln 2T_{max}}$ $\displaystyle=$
$\displaystyle\lim_{M_{0}\rightarrow 0}\Big{\\{}\ln
2+\frac{1}{2}\ln\cosh\Big{[}\frac{|M_{0}+\mu_{0}(x)|}{2T_{c}(x)}\Big{]}$ (25)
$\displaystyle+$
$\displaystyle\frac{1}{2}\ln\cosh\Big{[}\frac{|M_{0}-\mu_{0}(x)|}{2T_{c}(x)}\Big{]}\Big{\\}}^{-1}.$

We see that $T_{max}$ depends linearly on the couplings: $Ng-g_{c}$, whereas
in conventional SC, there is an exponential dependence. This kind of behavior
has been extensively studied before em ; ecm2 . By using the experimental
values of $T_{max}$ for the many different compounds studied here, we find
$\Lambda\simeq 0.018eV$. This is compatible with values of $hv_{eff}$ and
$\xi_{0}$, found in previous studies emsc .

Inside the SC phase, we have $M_{0}=0$. Inserting this condition in (21), we
can derive an expression for the SC gap as a function of the temperature and
doping, which is valid for $T\simeq Tc$ (note that both $\mu_{0}$ and $T_{c}$
depend on $x$)

$\displaystyle\Delta_{0}^{2}(T,x)=\Big{[}2T\cosh^{-1}\Big{[}\cosh\Big{(}\frac{\mu_{0}}{2T_{c}}\Big{)}[r]^{\frac{T_{c}}{T}-1}\Big{]}\Big{]}^{2}-\mu_{0}^{2},$
(26)

where

$\displaystyle r=\exp\Big{[}\ln 2\frac{T_{max}}{T_{c}(x)}\Big{]}>1.$ (27)

Notice that $\Delta_{0}(T_{c},x)=0$ and, since $r>1$ and $\cosh$ is a
monotonically increasing function, we must have $\Delta_{0}(T>T_{c},x)=0$,
while $\Delta_{0}(T<T_{c},x)\neq 0$.

D) The PG Order Parameters and the Critical PG Temperature: $T^{*}(x)$

We consider now the case where $\Delta_{0}=0$ and $M_{0}\neq 0$. In order to
find the critical temperature $T^{*}(x)$, we take (17) in the limit
$M_{0}\rightarrow 0$, which leads to

$\displaystyle T^{*}(x)=\lim_{M_{0}\rightarrow
0}\frac{\frac{\alpha\eta(g_{1}N)}{2g_{c}}}{F(\Delta_{0}=0,M_{0},\tilde{\mu}_{0}(x))}.$
(28)

Now (18) yields the following expression for the chemical potential

$\tilde{\mu}_{0}=2\tilde{\gamma}(g_{1}N)(\tilde{x}_{0}-x).$ (29)

Observe that, because $M_{0}\neq 0$ in the PG phase, the chemical potential
$\tilde{\mu}_{0}(x)$, no longer vanishes at the optimal doping $x_{0}$.

Results

We describe now the results obtained by applying our approach firstly to three
single-layered cuprates ($N=1$), namely LSCO, Bi2201 and Hg1201, for which,
$\gamma\equiv\gamma(gN)$ and $\tilde{\gamma}\equiv\gamma(g_{1}N)$, for $N=1$.
Thereafter we present our results concerning multi-layered cuprates.

A) SC Gap

A.1) LSCO

Starting from (21) we can write (see Suplementary Material)

$\displaystyle T_{c}(x)=\frac{\ln 2\ \ T_{max}}{\ln
2+\frac{\gamma|x_{0}-x|}{T_{c}(x)}+\frac{1}{2}\Big{\\{}\exp\left[-\frac{2\gamma|x_{0}-x|}{T_{c}(x)}\right]-1\Big{\\}}}.$
(30)

The solution $T_{c}(x)$ of this implicit equation for the critical temperature
of the SC transition, obtained with MAPLE, is depicted in Fig. 3.

Figure 3: Solution of Eq. (30) for the SC dome of LSCO. Experimental data from
1 ; 2 ; 3 ; 4 .

For obtaining this result, we used $\gamma=0.020\ eV$, $x_{0}=0.16$ and
$T_{max}=0.0031\ eV$ (Notice that$\ T(K)=11604\ T(eV)$). Observe also, that
with the choice of $\gamma$, we adjust only one parameter, namely $g$, of our
analytic expression for $T_{c}$. Entering the experimental values of $T_{max}$
and $x_{0}$ and using the value of $\Lambda$, we find $g=0.40226\ eV^{-1}$,
$g_{c}=0.30766\ eV^{-1}$ and $\eta(1g)=0.23517$.

Taking the limit $T_{c}\rightarrow 0$ in (30), we find the two quantum
critical points where the SC dome starts at $T=0$. These are given by

$\displaystyle x^{\pm}_{SC}=x_{0}\pm\frac{T_{max}}{\gamma}\ln 2=x_{0}\pm\alpha
Ax_{0}\frac{\eta^{2}(gN)}{g^{2}_{c}}.$ (31)

Inserting the above numerical values, for $N=1$, we find: $x^{-}_{SC}=0.053$
and $x^{+}_{SC}=0.2669$.

It is instructive to compare our result with the empirical curve, obtained by
fitting the data for the LSCO dome, by the parabola honma ; honma1 ; emp

$T_{c}(x)=T_{max}\left[1-82.616(x_{0}-x)^{2}\right].$

In Fig. 4 we superimpose it with our solution of (30).

Figure 4: The empirical parabolic fit for the SC dome of LSCO, dotted line,
superimposed with our solution for Eq. (30), solid line. Experimental data
from 1 ; 2 ; 3 ; 4 ; 7 .

A.2) Bi2201 and Hg1201

Now, for Bi2201 and Hg1201, we have different equations for $T_{c}(x)$ in the
underdoped, $x<x_{0}$ and overdoped, $x>x_{0}$ regions. The reason is the
order in which we take the limit $M\rightarrow 0$, starting from (21),
considering that $\mu_{0}(x)$ has different signs for $x_{0}-x>0$ and
$x_{0}-x<0$, thus leading to different equations in each region. Indeed, we
obtain (see Suplementary Material)

$\displaystyle T_{c}(x)=\frac{\ln 2\ \ T_{max}}{\ln
2+\frac{\gamma|x_{0}-x|}{T_{c}(x)}+\frac{1}{2}\Big{\\{}\exp\left[-\frac{2\gamma(x_{0}-x)}{T_{c}(x)}\right]-1\Big{\\}}}$
(32)

for $x<x_{0}$ and

$\displaystyle T_{c}(x)=\frac{\ln 2\ \
T_{max}}{\ln\Big{[}1+\exp\left[-\frac{2\gamma(x_{0}-x)}{T_{c}(x)}\right]\Big{]}}$
(33)

for $x>x_{0}$.

Let us consider Bi2201 first. The solution $T_{c}(x)$ of these implicit
equations for the critical temperature of the SC transition of Bi2201,
obtained by MAPLE, is depicted in Fig. 5.

Figure 5: Solution of Eqs. (32) and (33) for the SC dome of Bi2201.
Experimental data from 5 ; 6 . Figure 6: Solution of Eqs. (32) and (33) for
the SC dome of Hg1201. Experimental data from 7 ; 8 .

For obtaining this result, we used $N=1$, $\gamma=0.0120\ eV$, $x_{0}=0.29$
and $T_{max}=0.0030\ eV$, which imply $g=0.40800\ eV^{-1}$ and $g_{c}=0.31552\
eV^{-1}$ and $\eta(N=1)=0.22666$.

Taking the limit $T_{c}\rightarrow 0$ in (30), we find the two quantum
critical points where the SC dome starts at $T=0$. These are given by

$\displaystyle x^{-}_{SC}=x_{0}-\frac{T_{max}}{\gamma}\ln 2\ \ ;\ \
x^{+}_{SC}=x_{0}+\frac{T_{max}}{2\gamma}\ln 2.$ (34)

Inserting the above numerical values, we find: $x^{-}_{SC}=0.1175$ and
$x^{+}_{SC}=0.3766$.

Now consider Hg1201. Then, using Eqs. (32) and (33) with parameters
$\gamma=0.0310\ eV$, $x_{0}=0.25$ and $T_{max}=0.00835\ eV$, we obtain the
solution $T_{c}(x)$ of this implicit equation for the critical temperature of
the SC transition, which is depicted in Fig. 6. The above values imply
$g=2.14781\ eV^{-1}$ and $g_{c}=0.78612\ eV^{-1}$ and $\eta(1)=0.63398$.

Inserting the above numerical values in (34), we now find: $x^{-}_{SC}=0.064$
and $x^{+}_{SC}=0.3433$.

B) Pseudogap

We are now going to obtain the upper critical line delimiting the PG phase,
namely $T^{*}(x)$. For this purpose, we start from (28) and taking the limit
$M_{0}\rightarrow 0$, obtain

$\displaystyle
T^{*}(x)=\frac{\frac{\alpha\eta(g_{1}N)}{2g_{c}}}{\ln\Big{[}1+\exp\left[-\frac{2\tilde{\gamma}(\tilde{x}_{0}-x)}{T^{*}(x)}\right]\Big{]}}.$
(35)

B.1) LSCO, Bi2201 and Hg1201

We now show in Fig. 7, Fig. 8 and Fig. 9 the solution of (35) for the PG
temperature, $T^{*}(x)$, for LSCO, Bi2201 and Hg1201, respectively. The SC
dome is displayed in the same figure.

Figure 7: Solution of Eq. (30) for the SC dome of LSCO, together with the
solution of (35) for the pseudogap temperature $T^{*}(x)$. Experimental data
for $T_{c}(x)$ from 1 ; 2 ; 3 ; 4  and for $T^{*}(x)$ from 77 .

The PG temperature for LSCO was obtained with $\tilde{\gamma}=0.20\ eV$ and
$\tilde{x}_{0}=0.255$, which implies $\eta(g_{1})=0.01475$ and $g_{1}=0.31226\
eV^{-1}$.

Figure 8: Solution of Eqs. (32) and (33) for the SC dome of Bi2201, together
with the solution of (35) for the pseudogap temperature $T^{*}(x)$.
Experimental data for $T_{c}(x)$ from 1 ; 2 ; 3 ; 4  and for $T^{*}(x)$ from 7
.

The PG temperature for Bi2201 was obtained with $\tilde{\gamma}=0.372\ eV$ and
$\tilde{x}_{0}=0.315$, wich implies $\eta(g_{1})=0.00731$ and $g_{1}=0.31769\
eV^{-1}$.

Figure 9: Solution of Eqs. (32) and (33) for the SC dome of Hg1201, together
with the solution of (35) for the pseudogap temperature $T^{*}(x)$.
Experimental data for $T_{c}(x)$ and for $T^{*}(x)$ both from7 ; 8 .

The PG temperature for Hg1201 was obtained with $\tilde{\gamma}=0.186\ eV$ and
$\tilde{x}_{0}=0.345$, which implies $\eta(g_{1})=0.53$ and $g_{1}=1.66\
eV^{-1}$.

Notice that the ratio between the two couplings, $g$ and $g_{1}$ remains the
same for any of the three compounds, LSCO, Bi2201 and Hg1201: for the three of
them we have $g/g_{1}\simeq 1.28$. This fact suggests the existence of a
universal relation between the couplings responsible for the formation of the
SC gap and the Pseudogap, which should originate in the underlying interaction
leading to (LABEL:0).

C) Increase of $T_{max}$ with The Number of Adjacent Planes

It is an evident experimental fact that the optimal transition temperature
becomes higher as one increases the number of $CuO_{2}$ planes per primitive
unit cell. Bi2201 and Hg1201, for instance, are single-layered materials,
which have multi-layered relatives with a higher optimal temperature.

The mercury family, for instance, consists of 7 ; 8 ; mer1 ; mer2 : Hg1201
(single-layered) ($T_{max}=97\ K$), Hg1212 (double-layered) ($T_{max}=125\
K$), Hg1223 (triple-layered) ($T_{max}=134\ K$), Hg1234 (four-
layered)($T_{max}=127\ K$) and Hg1245 (five-layered)($T_{max}=120\ K$). It
shows an increase of the optimal temperatures as the number of adjacent layers
is increased from $N=1$ to $N=3$. Then for $N=4,5$, $T_{max}$ stabilizes at a
temperature approximately corresponding to $N=2$.

For the bismuth family, conversely, we have honma ; honma1 ; bis1 ; bis2
Bi2201 (single-layered) (Tmax = 34 K), Bi2212 (double-layered) (Tmax = 92 K),
Bi2223 (triple-layered) (Tmax = 108 K).

From (24), we see that, assuming that $g$ and $g_{c}$ are the same for all
members of a family, we may express the optimal temperature of a multi-layered
cuprate with $N$ adjacent $CuO_{2}$ planes in terms of the same temperature of
the single-layered one, as

$\displaystyle T_{max}(N)=\frac{\eta(N)}{\eta(1)}T_{max}(1).$ (36)

Observing that $\eta(N)$ is a monotonically increasing function of $N$, the
obvious effect of increasing the number of adjacent planes is to increase
$T_{max}$. This follows directly from the enhancement of the coupling
parameter, namely: $g\rightarrow Ng$.

For the bismuth family: $\eta(1)=0.22666$, $\eta(2)=0.61333,\eta(3)=0.74222$,
which according to (36) gives

$T_{max}(2)=91.99\ K,\ \ \ T_{max}^{exp}(2)=92\ K$ $T_{max}(3)=111.33\ K.\ \ \
T_{max}^{exp}(3)=108\ K$

The first values in each line above correspond to our theoretical expression
(36). These should be compared with the experimental values honma ; honma1 ;
bis1 ; bis2 , appearing on the right of each line.

Accordingly, for the mercury family we have: $\eta(1)=0.63398$,
$\eta(2)=0.81699,\eta(3)=0.87799,\eta(4)=0.90849,\eta(5)=0.92679$, which
implies

$T_{max}(2)=125.0\ K,\ \ \ T_{max}^{exp}(2)=125.0\ K$ $T_{max}(3)=134.33\ K,\
\ \ T_{max}^{exp}(3)=134.0\ K$ $T_{max}(4)=139.00\ K,\ \ \
T_{max}^{exp}(1)=127.0\ K$ $T_{max}(5)=141.80\ K,\ \ \ T_{max}^{exp}(1)=120.0\
K$

The first values in each line above correspond to our theoretical expression
(36), while the latter values are the experimental results honma ; honma1 ;
mer1 ; mer2 . We see that our theoretical values for the optimal temperature
of the multi-layered members of the Bi and Hg families, are in excellent
agreement with the experimental values for $N=2$. Then, for $N=3$, the
agreement is fairly good, whereas for $N>3$, there is no agreement. The
discrepancy, which starts to show at $N=3$ and increases for larger $N$’s can
be ascribed to another effect that evidently must be taken into account as we
increase the number of planes. This is the distance of such planes to the
charge absorving atoms doped into the system, which becomes progressively
larger as the number of planes increases. Indeed, for $N=1$ we have the two
“charge reservoir” regions adjacent to the unique $CuO_{2}$ plane. For $N=2$
still each of the two planes is adjacent to a charge reservoir. Then, for
$N=3$ one of the planes is no longer adjacent to any charge reservoir, while
for $N=4$ and $N=5$ the innermost planes are located far away from the charge
reservoirs. It happens that while the outer planes are optimally doped the
inner planes are poorly doped and, consquently, remain, to a large extent,
underdoped ml . The number of active $CuO_{2}$ planes, namely, the ones that
are adjacent to a charge reservoir, in this case, is equivalent to the one we
have for $N=2$, hence the temperature stabilizes at values similar to the ones
we had for $N=2$.

D) Superposition of the SC and AF Domes

An evident effect in the phenomenology of multi-layered cuprates is the
observation that, as we increase the number of $CuO_{2}$ planes per primitive
unit cell, the SC and AF domes come closer to each other and eventually
superimpose ml . From (34), we clearly see that the quantum critical point
$x^{\pm}_{SC}$ decreases as we increase $N$, the number of adjacent planes, as
a result of the enhancement of the effective coupling constant $g$. Hence it
will eventually come inside the AF dome that will consequently be superimposed
to the SC dome. Indeed, using (31), (34) and considering that $\eta(N)$ is a
monotonically increasing function of $N$ we can understand why $x^{-}_{SC}(N)$
decreases as $N$ increases, therefore producing a superposition of the SC and
AF domes. Furthermore, since the innermost planes are far from the charge
reservoirs, they remain underdoped, and consequently in the AF phase while the
outermost planes are efficiently doped and go into the SC phase.

Discussion

Our results indicate that the mechanism of Cooper pair formation in the
cuprates must produce an effective theory for the holes doped into the system,
whose Hamiltonian is defined on the oxygen lattice of the $CuO_{2}$ planes.
Because of the alternate overlap between $p_{x}$ and $p_{y}$ orbitals with the
$Cu^{++}$ ion orbitals, which forms bridges for the doped holes, the oxygen
lattice splits into two inequivalent sublattices. Cooper pairs are formed by
holes belonging to different sublattices, naturally yielding a d-wave SC order
parameter. Pseudogap phenomena, conversely, can be ascribed to a hole-
repulsive term, favoring exciton formation, which condense in a d-wave
symmetric gap (DDW) in a PG phase which competes with the Cooper pair
formation of the SC phase. The d-wave character of both order parameters
originates from the splitting of the oxygen lattice. For the case when
$g=g_{1}$, this model provides a SC pairing interaction derived from the spin-
fermion model ecm1 , the two sub-lattices, $p_{x}$ and $p_{y}$ of the oxygen
lattice being a realization of the two fermion species contained in that
model. The experimental observation of co-existence of the SC and PG d-wave
order parameters would be a clear sign of that.

By integrating over the fermions and minimizing the resulting effective
action, we derive implicit equations, both for the critical SC temperature
$T_{c}(x)$ and for the PG critical temperature $T^{*}(x)$, as a function of
doping. The solution of such equations is, then compared with the experimental
data for different compounds, showing an excellent agreement. The increase of
$T_{max}$ with the number of adjacent planes as well as the superposition of
SC and AF domes in multi-layered cuprates can be understood as a consequence
of the enhancement of the coupling $g\rightarrow Ng$ produced by the presence
of these planes. As $N$ increases, however, the inner planes progressively
recede from the charge reservoirs, an effect that counteracts the enhancement
o the coupling parameter, thus leading to a stabilization (or even decrease)
of $T_{max}$ as we increase $N$. Based on our results one can devise a way to
increase $T_{c}$ in cuprates: this would be achieved by effectively doping the
innermost planes in multilayered cuprates. For that purpose, one should design
materials with a unit cell containing as much layers as possible but with
charge reservoirs intercalating no more than two layers. This would neutralize
the above effect, thereby increasing $T_{c}$.

It would be quite interesting to investigate the possible relation of our
expression for $T_{max}$, with the formula derived in R1 , on the basis of a
Coulomb interaction.

Our results open a new avenue of investigation of the physical properties of
High-Tc cuprates, with outstanding possibilities. Among these, how to describe
the pseudogap transition and the charge ordering phases within this framework,
how to include the antiferromagnetic phase in the picture, how to describe the
interplay of the AF and SC phases, how to describe the resistivity above
$T_{c}$. The crucial issue in high-Tc superconductivity, of course, remains
the underlying mechanism of pair formation. The results reported here could be
a concrete step forward towards the complete understanding of the nature of
this mechanism.

Acknowledgments

We thank A. V. Balatsky and C. Morais Smith for stimulating conversations. We
would also like to thank E. Fradkin, S. Kivelson, J. Tranquada and P. A.
Marchetti, for very useful comments. E. C. Marino was supported in part by
CNPq and by FAPERJ. V. S. Alves acknowledges CNPq for support. Reginaldo de
Oliveira Jr acknowledges CAPES and FAPERJ for support.

Competing Interests

The authors declare that they have no competing financial or non-financial
interests.

Data Availability Statement

The authors declare that all data supporting the findings of this study are
available within the paper and its supplementary information files. The MAPLE
codes leading to the curves displayed in the text are available, upon request,
from the corresponding author.

Participation of the Authors

E.C.M. devised and proposed the problem. E.C.M. and R.O.C.J. (under the
supervision of E.C.M.) made the calculations. The Supplementary Material was
prepared by R.O.C.J., L.H.C.M.N. and V.S.A. The manuscript was written by
E.C.M. with input from all the authors. All the authors discussed every detail
of the work.

Corresponding author: ECM<EMAIL_ADDRESS>
## References

  * (1) E. C. Marino and L. H. C. M. Nunes, Competing effective interactions of Dirac electrons in the Spin-Fermion system, Ann. of Phys. 340, 13 (2014).
  * (2) J. G. Bednorz and K. A. Müller, Possible high-Tc superconductivity in the Ba-La-Cu-O system, Z. Phys. B64, 189 (1986).
  * (3) A. Damascelli, Z.-X. Shen and Z. Hussain, Angle resolved photoemmission studies of the cuprate superconductors, Rev. Mod. Phys. 75, 473 (2003)
  * (4) S. Hüfner, M. A. Hossain, A. Damascelli and G. A. Sawatzky Two gaps make a high temperature superconductor ?, Rep. Prog. Phys. 71, 062501 (2008)
  * (5) P.A. Lee, N. Nagaosa, and X.-G. Wen. Doping a Mott insulator: physics of high-temperature superconductivity. Rev. Mod. Phys. 78, 17 (2007)
  * (6) C. M. Varma, Theory of the pseudogap state of the cuprates, Phys. Rev. B 73, 155113 (2006)
  * (7) M. A. Kastner, R. J. Birgeneau, G. Shirane, and Y. Endoh. Magnetic, transport and optical properties of monolayer copper oxides. Rev. Mod. Phys. 70, 897 (1998)
  * (8) T. Honma and P. H. Hor, Universal optimal hole-doping concentration in single-layer high-temperature cuprate superconductors, Supercond. Sci. Technol. 19, 907 (2006).
  * (9) T. Honma and P. H. Hor, Unified electronic phase diagram for hole-doped high-Tc cuprates, Phys. Rev. B 77, 184520 (2008).
  * (10) T. Honma, P. H. Hor, H. H. Hsieh and M. Tanimoto Universal intrinsic scale of the hole concentration in high-Tc cuprates, Phys. Rev. B 70, 214517 (2004)
  * (11) S. Chakravarty, R. B. Laughlin, D. K. Morr and C. Nayak, Hidden Order in Cuprates, Phys. Rev. B 63, 094503 (2001)
  * (12) S. Chakravarty, C. Nayak and S. Tewari and , Angular-resolved photoemmission spectra in cuprates from d-density wave theory, Phys. Rev. B 68, 100504 (R) (2003)
  * (13) C. Zhang, S. Tewari and S. Chakravarty, Quasiparticle Nernst effect in the cuprate superconductors from the d-density-wave theory of the pseudogap phase, Phys. Rev. B 81, 104517 (2010)
  * (14) E. Razzoli, Y Sassa, G Drachuck, M Mansson, A Keren, M Shay, M H Berntsen, O Tjernberg, M Radovic, J Chang S Pailhès, N Momono, M Oda, M Ido, O J Lipscombe, S M Hayden, L Patthey, J Mesot and M Sh, The Fermi surface and band folding in LSCO, probed by angle-resolved photoemission, New J. Phys 12, 125003 (2010).
  * (15) E. C. Marino Quantum Field Theory Approach to Condensed Matter Physics, Cambridge University Press, Cambridge, UK (2017).
  * (16) O. Vafek and A. Vishwanath, Dirac Fermions in Solids: From High-Tc Cuprates and Graphene to Topological Insulators and Weyl Semimetals, Ann. Rev. Cond. Mat. Phys. 5, 83 (2014).
  * (17) L. K. Lim, A. Lazarides, A. Hemmerich and C. M. Smith, Strongly interacting two-dimensional Dirac fermions, Europhys. Lett. 88, 36001 (2009)
  * (18) E. C. Marino and L. H. C. M. Nunes, Quantum criticality and superconductivity in quasi-two-dimensional Dirac electronic systems, Nucl. Phys. B741, 404 (2006).
  * (19) E. C. Marino and M. B. S. Neto, Magnetic-texture-driven charge pairing in the spin-fermion Hubbard model and superconductivity in the high-Tc cuprates, Phys. Rev. B 66, 224512 (2002).
  * (20) M. R. Presland et. al., General trends in oxygen stoichiometry effects on Tc in Bi and Tl superconductors Physica C 176, 95 (1991).
  * (21) T. Nagano, Y. Tomioka, Y. Nakayama, K. Kishio and K. Kitazawa, Bulk superconductivity in both tetragonal and orthorhombic solid solutions of $(La_{1-x}Sr_{x})_{2}CuO_{4-\delta}$, Phys. Rev. B 48, 9689 (1993).
  * (22) P. G. Radaelli, D. G. Hinks, A. W. Mitchell, B. A. Hunter, J. L. Wagner, B. Dabrowski, K. G. Vandervoort, H. K. Viswanathan and J. D. Jorgensen, Structural and superconducting properties of $La_{2-x}Sr_{x}CuO_{4}$ as a function of Sr content, Phys. Rev. B 49, 4163 (1994).
  * (23) N. Kakinuma, Y. Ono and Y. Koike, Anomalies of Tc, resistivity, and thermoelectric power in the overdoped region of $La_{2-x}Sr_{x}Cu_{1-y}Zn_{y}O4$, Phys. Rev. B 59, 1491 (1999).
  * (24) S. Komiya, H. D. Chen, S. C. Zhang and Y. Ando, Magic Doping Fractions for High-Temperature Superconductors, Phys. Rev. Lett. 94, 207004 (2005);
  * (25) Y. Ando, Y. Hanaki, S. Ono, T. Murayama, K. Segawa, N. Miyamoto, and S. Komiya, Carrier concentrations in $Bi_{2}Sr_{2-z}La_{z}CuO_{6+\delta}$ single crystals and their relation to the Hall coefficient and thermopower, Phys. Rev. B 61, R14956 (2000).
  * (26) Y. Okada and H. Ikuta, R dependence of superconducti-vity and thermopower in the $Bi_{2}Sr_{2-x}R_{x}CuO_{y}$ (R $=$ La, Sm and Eu) system Physica C 445-448, 84 (2006).
  * (27) O. Cyr-Choinière, R. Daou, F. Laliberté, C. Collignon, S. Badoux, D. LeBoeuf, J. Chang, B. J. Ramshaw, D. A. Bonn, W. N. Hardy, R. Liang, J.-Q. Yan, J.-G. Cheng, J.-S. Zhou, J. B. Goodenough, S. Pyon, T. Takayama, H. Takagi, N. Doiron-Leyraud, and Louis Taillefer Pseudogap temperature $T^{*}$ of cuprate superconductors from the Nernst effect, Phys. Rev. B 97, 064502 (2018).
  * (28) A. Yamamoto, W. Z. Hu and S. Tajima, Thermoelectric power and resistivity of $HgBa_{2}CuO_{4+\delta}$ over a wide doping range, Phys. Rev. B 63, 024504 (2000).
  * (29) J. A. Wilson and M. Farbod, A view of pseudogap formation and the HTSC mechanism from the perspective of Seebeck results on $HgBa_{2}CuO_{4+\delta}$ Supercond. Sci. Tech. 13, 307 (2000).
  * (30) Y. Ando and T. Murayama, Nonuniversal power law of the Hall scattering rate in a single-layer cuprate $Bi_{2}Sr_{2-x}La_{x}CuO_{6}$ Phys. Rev. B 60, R6991 (1999)
  * (31) M. Akoshima, T. Noji, Y. Ono and Y. Koike, Anomalous suppression of superconductivity in Zn-substituted $Bi_{2}Sr_{2}Ca_{1-x}Y_{x}(Cu_{1-y}Zn_{y})_{2}O_{8+\delta}$, Phys. Rev. B 57, 7491 (1998).
  * (32) T. Fujii, I. Terasaki, T. Watanabe and A. Matsuda, Doping dependence of anisotropic resistivities in the trilayered superconductor $Bi_{2}Sr_{2}Ca_{2}Cu_{3}O_{10+\delta}$, Phys. Rev. B 66, 024507 (2002).
  * (33) A. Fukuoka, A. Tokiwa-Yamamoto, M. Itoh, R. Usami, S. Adachi and K. Tanabe, Dependence of Tc and transport properties on the Cu valence in $HgBa_{2}Ca_{n-1}Cu_{n}O_{2(n+1)+\delta}$ (n$=$2,3) superconductors, Phys. Rev. B 55, 6612 (1997).
  * (34) J. L. Cohn, C. P. Popoviciu, Q. M. Lin and C. W. Chu, Hole localization in underdoped superconducting cuprates near $\frac{1}{8}$ doping, Phys. Rev. B 59, 3823 (1999).
  * (35) H. Mukuda, N. Shiki, N. Kimoto, M. Yashima, Y. Kitaoka, K. Tokiwa and A. Iyo, Novel Interplay between High-Tc Superconductivity and Antiferromagnetism in Tl-based Six-CuO2-Layered Cuprates: 205Tl- and 63Cu-NMR Probes, J. Phys. Soc. Japan 85, 083701 (2017)
  * (36) D. R. Harshman, A. T. Fiory and J. D. Dow, Theory of high-TC superconductivity: transition temperature J. Phys.: Condens. Matter 23, 349501 (2011) ; D. R. Harshman and A. T. Fiory, High-T C superconductivity in Cs3C60 compounds governed by local CsC60 Coulomb interactions J. Phys.: Condens. Matter 29, 145602 (2017); D. R. Harshman and A. T. Fiory, Compressed H3S: inter-sublattice Coulomb coupling in a high-Tc superconductor, J. Phys.: Condens. Matter 29, 445702 (2017)


# Remembering Transformer For Continual Learning

Yuwei Sun1,2, Ippei Fujisawa1, Arthur Juliani3, Jun Sakuma2,4,, Ryota
Kanai1,††footnotemark:
1Araya, 2RIKEN AIP, 3Microsoft Research, 4Tokyo Institute of Technology
Equal advising; Corresponding author<EMAIL_ADDRESS>
###### Abstract

Neural networks encounter the challenge of Catastrophic Forgetting (CF) in
continual learning, where new task learning interferes with previously learned
knowledge. Existing data fine-tuning and regularization methods necessitate
task identity information during inference and cannot eliminate interference
among different tasks, while soft parameter sharing approaches encounter the
problem of an increasing model parameter size. To tackle these challenges, we
propose the Remembering Transformer, inspired by the brain’s Complementary
Learning Systems (CLS). Remembering Transformer employs a mixture-of-adapters
architecture and a generative model-based novelty detection mechanism in a
pretrained Transformer to alleviate CF. Remembering Transformer dynamically
routes task data to the most relevant adapter with enhanced parameter
efficiency based on knowledge distillation. We conducted extensive
experiments, including ablation studies on the novelty detection mechanism and
model capacity of the mixture-of-adapters, in a broad range of class-
incremental split tasks and permutation tasks. Our approach demonstrated SOTA
performance surpassing the second-best method by 15.90% in the split tasks,
reducing the memory footprint from 11.18M to 0.22M in the five splits CIFAR10
task.

##  1 Introduction

Figure 1: Remembering Transformer leverages the mixture-of-adapters that are
sparsely activated with a novelty detection mechanism.

The Catastrophic Forgetting (CF) problem arises in the sequential learning of
neural networks, wherein the new task knowledge tends to interfere with old
knowledge, with previously learned tasks forgotten. The ability to learn a new
task without interfering with previously learned ones is of great importance
to achieve continual learning. An intuitive approach to CF is data fine-tuning
[28]. A model alleviates the forgetting of previous tasks by replaying and
retraining on old task samples during learning a new task. Nevertheless, as
the number of encountered tasks increases, it necessitates the storage of a
large number of old task samples. In addition, the data fine-tuning based on
memory replay usually cannot eliminate the CF problem leading to suboptimal
performance.

Biological neural networks exhibit apparent advantages in continual learning
via the Complementary Learning Systems (CLS) [20, 16]. In CLS, the Hippocampus
rapidly encodes task data and then consolidates the task knowledge into the
Cortex by forming new neural connections. The hippocampus has developed a
novelty detection mechanism to facilitate consolidation by switching among
neural modules in the Cortex for various tasks [12, 7]. To this end, we
propose a novel Remembering Transformer inspired by the CLS in the brain. Our
approach consolidates a mixture-of-adapters architecture with generative
routing to tackle the challenges in conventional continual learning methods
(see Figure 1). Notably, the mixture-of-adapters architecture incorporates and
jointly trains low-rank adapters with a pretrained Transformer. Then, these
adapters are selectively activated for parameter-efficient fine-tuning through
a novelty detection mechanism, where each task’s samples are encoded in
specific autoencoders for effective routing. During inference, the
autoencoders, each encoding different task knowledge, predict routing weights
for the various adapters, thus allocating an input sample to the most relevant
adapter.

Moreover, we leverage low-rank autoencoders to facilitate novelty detection
based on the magnitudes of reconstruction losses of various autoencoders, with
the current task data as input. The reconstruction losses represent the
similarity between the current task and the old task knowledge encoded in
these autoencoders. In a more challenging setting of limited model parameter
capacity, we assume there is a limit for the number of adapters added to the
Transformer. To tackle this challenge, we further propose the adapter fusion
approach based on knowledge distillation [8] by distilling knowledge from the
most relevant old adapter into a newly learned adapter. The knowledge
distillation leverages a small set of old task replay memory and trains the
new adapter on the probability distribution of the replay memory samples.
Consequently, we demonstrate the superiority of Remembering Transformer in
terms of enhanced task accuracy and parameter efficiency compared to a wide
range of conventional continual learning methods. The results indicate that
the proposed Remembering Transformer can achieve competitive performance even
with the limited model capacity.

Overall, our main contributions are three-fold:

1) We propose the Remembering Transformer inspired by the Complementary
Learning Systems to tackle the catastrophic forgetting problem in split tasks
and permutation tasks. We investigate two challenging real-world scenarios in
continual learning: class-incremental learning without task identity
information and learning with limited model parameter capacity (Section 3.3).

2) We propose the adapter fusion to distill old task knowledge from existing
adapters and aggregate relevant adapters, enhancing parameter efficiency when
learning with limited model parameter capacity (Section 3.4).

3) The empirical experiment results demonstrate the superiority of Remembering
Transformer compared to conventional methods including Feature Translation and
Representation Expansion, in terms of task accuracy and parameter efficiency
in various continual learning tasks (Section 4).

The remainder of this paper is structured as follows. Section 2 reviews the
most relevant work on continual learning. Section 3 demonstrates the essential
definitions, assumptions, and technical underpinnings of the Remembering
Transformer. Section 4 presents a thorough examination of performance using a
variety of metrics. Section 5 concludes findings and gives out future
directions.

##  2 Related work

This section provides a summary and comparison of relevant research on
continual learning. To tackle the catastrophic forgetting (CF) problem, there
are approaches including fine-tuning, generative replay, regularization, and
soft parameter sharing. The fine-tuning method retrains the model with data
from previous tasks when training on new task data [28]. Generative replay
aims to train a generative model to reconstruct previous task data for the
retraining [10, 2]. However, with the increasing number of tasks, it becomes
less and less feasible to either store or learn a generative model for the
retraining on the previous task data. A recent study [10] showed that the
learned generative model was encountered with the CF problem as well, leading
to the degraded quality of reconstructed task data. Additionally, fine-tuning
on each observed task is computationally expensive for continual learning. To
tackle these problems, regularization and soft parameter sharing methods
selectively update model parameters without disturbing parameters important
for old tasks [14, 1]. The soft parameter sharing method leverages a modular
architecture to learn task-specific neural module parameters [11, 6, 2]. The
isolation of parameters alleviates negative interference among various tasks.
However, the soft parameter sharing method is inefficient in terms of model
parameters and typically requires the task identity information for selecting
relevant parameters.

In contrast, the Remembering Transformer leverages a novelty detection method
to accurately infer relevant neural modules without requiring any task
identity information. It significantly reduces the model parameter size of
each module based on a mixture-of-adapters architecture, wherein a set of
task-specific linear transformation matrices is employed in each attention
layer of a pre-trained Transformer model. By adaptively learning, activating,
and fusing these neural modules, it significantly reduces the memory cost for
continual learning.

##  3 Methodology

In this section, we delve into a comprehensive exploration of the assumptions
and the proposed Remembering Transformer’s technical underpinnings. These
include the incorporation of mixture-of-adapters in Transformer models, a
generative model-based novelty detection for expert routing, and adapter
fusion based on knowledge distillation (see Figure 2). The adapter fusion
enables efficient knowledge retrieval from relevant previously learned
adapters while learning new tasks, preventing catastrophic forgetting of
previous tasks.

Figure 2: Adapter fusion based on knowledge distillation with a limited
capacity $E$. We update the $E+1$-th adapter
$\\{B^{l}_{E+1},A^{l}_{E+1}\\}_{l=1}^{L}$ using new task data and the soft
probability output of the old task replay.

###  3.1 Continual learning task setting

We aim to tackle continual learning on a sequence of tasks
$\\{T^{(1)},T^{(2)},\dots,T^{(N)}\\}$, where
$T^{(t)}=\\{x_{i}^{t},y_{i}^{t}\\}_{i=1}^{K_{t}}\subset T$ is a labeled
dataset including $K_{t}$ pairs of instances $x^{t}$ and their corresponding
labels $y^{t}$. We investigate the most challenging class-incremental learning
setting in continual learning, where each task $T^{(t)}$ is defined by a non-
overlapping subset of a dataset $T^{(t)}\subset\\{x_{i},y_{i}\\}_{i=1}^{K}$
where $K=\sum_{t=1}^{N}K_{t}$. $Y^{t}\subset Y$ is changing over time, i.e.,
$Y^{i}\cap Y^{j}=\emptyset\,$ $\forall i\neq j$. The posterior $P(X|Y)$ is
consistent across tasks $T^{(t)}\subset T$. During inference, we evaluate on
$P(X)\sim\frac{1}{K}\sum_{t=1}^{N}P(X|Y)P_{t}(Y)$ without the task identity
information $t$.

###  3.2 Vision Transformers with mixture-of-adapters

Our method is inspired by the Mixture-of-Experts (MoE) [24, 5, 3], with
diverse neural modules constituting a comprehensive neural network. Different
parts of the neural network learn and adapt independently without disrupting
the knowledge that other parts have acquired, thus mitigating interference
between tasks. The intuition is to learn a diverse collection of neural
modules based on a pretrained Vision Transformer (ViT) and adaptively select
the most relevant neural module for adapting to various continual learning
tasks.

ViT partitions an image into a sequence of non-overlapping patches. Let
$x\in\mathbb{R}^{H\times W\times C}$ be an image input, where $(H,W)$ is the
resolution of the image and $C$ is the number of channels. $x$ is separated
into a sequence of 2D patches $x_{p}\in\mathbb{R}^{N\times(P^{2}\cdot C)}$,
where $(P,P)$ is the resolution of each image patch and $\frac{HW}{P^{2}}$ is
the number of patches. These patches are mapped to tokens
$v_{p}\in\mathbb{R}^{\frac{HW}{P^{2}}\times D}$ with a learnable linear
projection. A learnable 1D position embedding $v_{0}\in\mathbb{R}^{D}$ is
prepended to the tokens to retain positional information, resulting in
$v\in\mathbb{R}^{(\frac{HW}{P^{2}}+1)\times D}$.

Moreover, we employ the soft parameter sharing method to enhance ViT’s ability
for continual learning, alleviating catastrophic forgetting. Notably, for each
task $T^{(t)}\subseteq T$, we utilize the Low-Rank Adaptation (LoRA) method
[9] for efficient fine-tuning using low-rank decomposition matrices. We employ
the trainable decomposition matrices to the attention weights of ViT’s
different layers. For each attention layer $l$, linear transformations are
applied to the query, key, value, and output weights
($W_{l}^{Q},W_{l}^{K},W_{l}^{V},W_{l}^{O}$). For $W_{l}\in\mathbb{R}^{D\times
D}$ $\in$ ($W_{l}^{Q},W_{l}^{K},W_{l}^{V},W_{l}^{O}$), the parameter update
$\Delta W_{l}$ is then computed by $W_{l}+\Delta W_{l}=W_{l}+B^{l}A^{l}$,
where $A^{l}\in\mathbb{R}^{r\times D},B^{l}\in\mathbb{R}^{D\times r}$, and $r$
is a low rank $r\ll D$. For an input token $v^{t}$, the output of the $l$th
attention layer is $\hat{W_{l}}v^{t}+\Delta
W_{l}v^{t}=\hat{W_{l}}v^{t}+B^{l}A^{l}v^{t}$ where $\hat{\cdot}$ indicates
untrainable parameters. For each task $T^{(t)}$,
$\theta^{t}_{\text{adapter}}=\\{A_{t}^{l},B_{t}^{l}\\}_{l=1}^{L}$ is added to
the pretrained ViT model
$\\{\hat{\theta}_{\text{ViT}},\theta^{t}_{\text{adapter}}\\}$ where $L$
represents the total number of attention layers. We formulate the optimization
of the adapter based on task $t$’s dataset
$\\{v_{i}^{t},y_{i}^{t}\\}_{i=1}^{K_{t}}$ as follows:
$\hat{\theta}_{\text{adapter}}^{t}\leftarrow\underset{\theta_{\text{adapter}}^{t}}{\text{arg
min
}}-\sum_{i=1}^{K_{t}}f(v_{i}^{t};\\{\hat{\theta}_{\text{ViT}},\theta_{\text{adapter}}^{t}\\})\text{log}(y_{i}^{t}).$
Then, $\hat{\theta}_{\text{adapter}}^{t}$ is added to the list of adapters
$\\{{\theta}_{\text{adapter}}^{1},{\theta}_{\text{adapter}}^{2},...,{\theta}_{\text{adapter}}^{t-1}\\}$
to form the mixture-of-adapters architecture.

###  3.3 Generative model-based novelty detection for expert routing

Utilizing the input tokens, our goal is to adaptively activate the most
relevant adapters within the mixture-of-adapters framework. We propose a
novelty detection mechanism based on generative models for effective adapter
routing. In particular, a routing neural network outputs gating weights
$W_{g}\in\mathbb{R}^{E}$ that indicate the probability of employing a specific
adapter for a given task, where $E$ represents the number of existing
adapters. Conventional routing functions in Mixture-of-Experts (MoE) usually
struggle with the more complex continual learning tasks, where sequential task
inputs necessitate a continual update of the gating weights leading to
forgetting of the routing neural networks.

A generative model encodes tokens from a specific task and assesses the
familiarity of a new task in relation to the encoded knowledge of the old
task. Notably, to facilitate efficient routing, we employ a low-rank
autoencoder $\theta_{\text{AE}}$ that consists of an encoder and decoder, each
of which leverages one linear transformation layer $F\in\mathbb{R}^{s\times
D}$ and $G\in\mathbb{R}^{D\times s}$ where $s$ is a low rank $s\ll D$. For
each task $t$, we train the AE to encode the embedding layer output $V^{t}$,
which are flattened and passed through a Sigmoid activation $\sigma(\cdot)$,
$\hat{V^{t}}=G_{t}F_{t}\sigma(V^{t})$. The Sigmoid activation constraints the
reconstructed tokens from the AE between the range of 0 and 1 to facilitate
learning. Then, the AE is updated based on the mean squared error loss in Eq.
(1).

$\displaystyle\ell_{t}(\theta_{\text{AE}}^{t})=(\sigma(V^{t})-\hat{V^{t}})^{2},\,\hat{\theta}_{\text{AE}}=\underset{\theta_{\text{AE}}^{t}}{\text{arg
min }}\ell_{t}.$ (1)

During inference, the collection of AEs
$\hat{\theta}_{\text{AE}}^{e}\in\\{\hat{G}_{e},\hat{F}_{e}\\}_{e=1}^{E}$
trained on various tasks provide estimates of the input tokens $v$’s novelty
in relation to the previously learned tasks based on the computed
reconstruction loss $\ell_{e}(v)$. In particular, when the input closely
resembles an old task learned by an existing AE, the reconstruction loss for
the input task by this AE tends to be low. Consequently, the old task $e^{*}$
corresponding to the AE with the minimum reconstruction loss is the most
relevant to the input task. Then, the adapter
$\theta^{e^{*}}_{\text{adapter}}$ is leveraged to adapt the pretrained ViT for
tackling the input task. We devise the proposed novelty detection mechanism as
follows:

$\displaystyle\ell_{e}=(\sigma(v)-\hat{G}_{e}\hat{F}_{e}\sigma(v))^{2},\,e^{*}=\underset{e}{\text{arg
min }}\ell_{e},$ (2) $\displaystyle W_{g}(e)=\begin{cases}1&\text{if
}e=e^{*}\\\ 0&\text{otherwise}.\end{cases}$

The output $v_{l}$ of the $l$th attention layer is then computed by
$\hat{W_{l}}v_{l-1}+\sum_{e=1}^{E}W_{g}(e)B^{l}_{e}A^{l}_{e}v_{l-1}$, where
$v_{l-1}$ is the output of the previous $l-1$th attention layer. Additionally,
for enhanced parameter efficiency, the routing model
$\\{\theta_{\text{AE}}^{t}\\}_{t=1}^{N}$ is globally shared across different
attention layers.

###  3.4 Adapter fusion based on knowledge distillation

Remembering Transformer leverages the mixture-of-adapters architecture and the
low-rank generative routing for enhanced parameter efficiency in continual
learning. However, with the increasing number of tasks, it is still
encountered with increasing parameters. To further enhance its parameter
efficiency, we explore a scenario where the number of adapters is constrained.
We then propose a novel adapter fusion approach within the mixture-of-adapters
framework to identify and aggregate resembling adapters based on knowledge
distillation [8]. This involves transferring the knowledge of a selected old
adapter $\theta_{\text{adapter}}^{e^{*}}$ to the new adapter
$\theta_{\text{adapter}}^{t}$ by replaying old task samples when the model
capacity $E$ is reached, i.e., when $t\leq E+1$ (see Figure 2).

Furthermore, to reduce the computational cost of knowledge distillation, we
leverage the Herding method [29, 22] to obtain a small set of the most
representative replay samples of each task during training. In particular,
given tokens from a specific class $y\in Y^{t}$ of task $t$, i.e.,
$V^{t}_{y}=\\{v^{t}_{y,1},v^{t}_{y,2},\dots,v^{t}_{y,K_{t}^{y}}\\}$ where
$K_{t}^{y}$ is the total number of samples from class $y$, we compute the
class center $\mu_{y}$ based on the latent representations learned by the
autoencoder $\hat{\theta}_{\text{AE}}^{t}:=\\{\hat{G}_{t},\hat{F}_{t}\\}$. The
class center in the latent space is
$\mu_{y}\leftarrow\frac{1}{K_{t}^{y}}\sum_{i=1}^{K_{t}^{y}}\hat{F}_{t}\sigma(v^{t}_{y,i})$.
Then, the distance of each token’s latent representation to the class center
$||\mu_{y}-\hat{F}_{t}\sigma(v^{t}_{y,i})||$ is ranked in the ascending order.
For each class $y\in Y^{t}$, we select and add the top-$\frac{M}{U}$ tokens
($U$ is the number of classes and $M$ is the task replay memory size) with the
least distance to the class center in the latent space to the replay memory
$\Xi_{t}$ of task $t$, i.e.,
$\Xi_{t}\leftarrow\\{\text{top-}\frac{M}{U}(v^{t}_{y,i})\\}_{y\in Y^{t}}$.

Algorithm 1 Remembering Transformer with Adapter Fusion

1: for each task $t=1,2,\dots,N$ do

2: Initialize a new autoencoder $\theta_{\text{AE}}^{t}:\\{G_{t},F_{t}\\}$.

3:
$\ell_{t}=\frac{1}{K_{t}}\sum_{i=1}^{K_{t}}(\sigma(v^{t}_{i})-GF\sigma(v^{t}_{i}))^{2}.$

4: $\hat{\theta}_{\text{AE}}^{t}=\underset{\theta_{\text{AE}}^{t}}{\text{arg
min }}\ell_{t}.$

5: Add $\hat{\theta}_{\text{AE}}^{t}$ to the list of AE
$\\{{\theta}_{\text{AE}}^{1},{\theta}_{\text{AE}}^{2},...,{\theta}_{\text{AE}}^{t-1}\\}$.

6: For $y\in Y^{t}$,
$\mu_{y}\leftarrow\frac{1}{K_{t}^{y}}\sum_{i=1}^{K_{t}^{y}}\hat{F}_{t}\sigma(v^{t}_{y,i})$.

7: $\Xi_{t}\leftarrow\\{\text{top-}\frac{M}{U}(v^{t}_{y,i})\\}_{y\in Y^{t}}$
by ranking $||\mu_{y}-\hat{F}_{t}\sigma(v^{t}_{y,i})||$ in the ascending
order.

8: Add $\Xi_{t}$ to the replay memory $\\{\Xi_{1},\Xi_{2},...,\Xi_{t-1}\\}$.

9: for each AE $e=1,2,...,t-1$ in the AE list do

10:
$\ell_{e}=(\sigma(v_{i}^{t})-\hat{G_{e}}\hat{F_{e}}\sigma(v_{i}^{t}))^{2}.$

11: end for

12: Obtain the index of the most relevant AE: $e^{*}=\underset{e}{\text{arg
min }}\ell_{e}$.

13: if $t>E$ then

14: Replay old task $e^{*}$ samples from the memory:
$\\{v^{e^{*}}_{i}\\}_{i=1}^{M}\leftarrow\Xi_{e^{*}}$.

15:
$\ell_{\text{L2}}(\theta_{\text{adapter}}^{t})=\sum_{i=1}^{M}\left(f(v^{e^{*}}_{i};\\{\hat{\theta}_{\text{ViT}},\hat{\theta}_{\text{adapter}}^{e^{*}}\\})-f(v^{e^{*}}_{i};\\{\hat{\theta}_{\text{ViT}},\theta_{\text{adapter}}^{t}\\})\right)^{2}$.

16:
$\ell_{\text{CE}}(\theta_{\text{adapter}}^{t})=-\sum_{i=1}^{K_{t}}f(v_{i}^{t};\\{\hat{\theta}_{\text{ViT}},\theta_{\text{adapter}}^{t}\\})\log(y^{t}_{i})$.

17:
$\hat{\theta}_{\text{adapter}}^{t}\leftarrow\underset{\theta_{\text{adapter}}^{t}}{\text{arg
min }}\alpha\cdot\ell_{\text{CE}}+(1-\alpha)\cdot\ell_{\text{L2}}.$

18: The old adapter is removed and the samples of task $e^{*}$ are distributed
to the newly learned adapter $t$: Gate$(e^{*})\leftarrow t$.

19: else

20: Initialize a new adapter
$\theta_{\text{adapter}}^{t}=\\{A_{t}^{l},B_{t}^{l}\\}_{l=1}^{L}$.

21:
$\hat{\theta}_{\text{adapter}}^{t}\leftarrow\underset{\theta_{\text{adapter}}^{t}}{\text{arg
min
}}-\sum_{i=1}^{K_{t}}f(v_{i}^{t};\\{\hat{\theta}_{\text{ViT}},\theta_{\text{adapter}}^{t}\\})\text{log}(y_{i}^{t}).$

22: Add $\hat{\theta}_{\text{adapter}}^{t}$ to the list of adapters
$\\{{\theta}_{\text{adapter}}^{1},{\theta}_{\text{adapter}}^{2},...,{\theta}_{\text{adapter}}^{t-1}\\}$.

23: end if

24: end for

The old adapter $e^{*}$ for knowledge distillation is selected based on the
novelty detection mechanism in Eq. (2). We determine the most relevant old
adapter for the new task by identifying the minimum reconstruction loss. We
compute the soft probability output
$f(\Xi_{e^{*}};\\{\hat{\theta}_{\text{ViT}},\hat{\theta}_{\text{adapter}}^{e^{*}}\\})$
of the old adapter by replaying samples $\Xi_{e^{*}}$ from the memory. Then,
the new adapter $\theta_{\text{adapter}}^{E+1}$ is trained using its task data
$\\{V^{E+1},Y^{E+1}\\}$ and the soft probability output of the old task
$\\{\Xi_{e^{*}},f(\Xi_{e^{*}};\\{\hat{\theta}_{\text{ViT}},\hat{\theta}_{\text{adapter}}^{e^{*}}\\})\\}$.
Notably, we employ the cross-entropy loss $\ell_{\text{CE}}(\cdot)$ and the L2
loss $\ell_{\text{L2}}(\cdot)$ for training on the new and old tasks,
respectively. We devise the optimization process for the adapter fusion as
follows:

$\displaystyle\ell_{\text{CE}}(\theta_{\text{adapter}}^{E+1})=-\sum_{i=1}^{K_{E+1}}f(v_{i}^{E+1};\\{\hat{\theta}_{\text{ViT}},\theta_{\text{adapter}}^{E+1}\\})\log(y^{E+1}_{i}),$
(3)
$\displaystyle\ell_{\text{L2}}(\theta_{\text{adapter}}^{E+1})=\sum_{i=1}^{M}\left(f(v^{e^{*}}_{i};\\{\hat{\theta}_{\text{ViT}},\hat{\theta}_{\text{adapter}}^{e^{*}}\\})-f(v^{e^{*}}_{i};\\{\hat{\theta}_{\text{ViT}},\theta_{\text{adapter}}^{E+1}\\})\right)^{2},$
$\displaystyle\hat{\theta}_{\text{adapter}}^{E+1}=\underset{\theta_{\text{adapter}}^{E+1}}{\text{arg
min }}\alpha\cdot\ell_{\text{CE}}+(1-\alpha)\cdot\ell_{\text{L2}},$

where $\alpha$ is a coefficient to balance the two loss items.

Additionally, after adapter fusion, the old adapter is removed, and the
samples of task $e^{*}$ are distributed to the newly learned adapter $E+1$ for
inference, i.e., Gate$(e^{*})\leftarrow E+1$ where Gate($\cdot$) is a function
to project the result of novelty detection to the defined new gate. Note that
Gate($\cdot$) is initialized with an identity projection
$\text{Gate}_{\text{init}}(e)=e$. We can formulate the routing weights as
follows: $W_{g}(e)=\begin{cases}1&\text{if }e=\text{Gate}(e^{*})\\\
0&\text{otherwise}.\end{cases}$

We devise the algorithm for the Remembering Transformer in Algorithm 1.

##  4 Experiments

In this section, we provide a detailed description of the datasets, model
architectures, and metrics used in the experiments. An extensive empirical
evaluation including an ablation study is performed based on a wide range of
split and permutation tasks in continual learning. Moreover, we investigate
the Remembering Transformer’s performance when its capacity of adapters is
limited. The results demonstrate that the Remembering Transformer achieves
SOTA performance while retaining great parameter efficiency compared with
conventional methods.

###  4.1 Settings

#### Datasets

We evaluated the model performance based on continual learning datasets
adapted from CIFAR10, CIFAR100 [15], and Permuted MNIST [26] tasks. We
investigated two types of continual learning datasets: (1) the split task
where each dataset is split into several subsets of equal numbers of classes,
with each subset as a task. For example, we divide the CIFAR-10 dataset into
five tasks. The first task consists of digits (classes) 0 - 1 and the second
task consists of digits (classes) 2 - 3 and so on. We denote the task of
CIFAR-10 in five splits as CIFAR10/5. (2) the permutation task where the input
pixels of an image in the training and test data are shuffled with a random
permutation, with a different permutation for each task.

#### Metrics

For the split tasks, we measure the average task accuracy over all tasks after
training, i.e., $\text{Acc}=\frac{1}{N}\sum_{i=1}^{N}\text{Acc}_{i}$ where
$\text{Acc}_{i}$ is the accuracy on the $i$th task after learning the final
task $T^{(N)}$. Moreover, we evaluate the memory footprint of a method based
on its trainable model parameter size. For the permutation tasks, we evaluate
the average task accuracy each time the model learns a new permutation. We
then report the mean and standard deviation of three individual experiments
with random seeds, with each seed resampling the splits to avoid any favorable
class split and resampling the permutation patterns to avoid any favorable
permutation.

#### Hyperparameters

We employed the base-size Vision Transformer pretrained on ImageNet-21k, with
a resolution of 224x224 pixels and a patch size of $16\times 16$. For the
detailed architecture, we followed the default author-suggested settings [5].
We employed a two-layer autoencoder with a latent dimension of one for all
split tasks (5-20 splits) and a four-layer autoencoder with a hidden dimension
of 32 and a latent dimension of one for the permutation task (50
permutations). Moreover, the hyperparameters were chosen based on a grid
search. Batch sizes of 128 and 1 were employed for training and test,
respectively. We utilized the AdamW optimizer with $\beta_{1}=0.9$,
$\beta_{2}=0.999$, and a weight decay of 0.01. We employed learning rates of
0.001 and 0.005 for updating the adapters and autoencoders, respectively. We
employed a coefficient of 0.5 in the loss function of the knowledge
distillation. We trained the model for 50 epochs in the CIFAR10 split task,
200 epochs in the CIFAR100 split tasks, 2000 epochs in the permuted-MNIST
task. Additionally, we employed the average along the latent dimension of each
token of the embedding layer output to train the autoencoders for enhanced
parameter-efficiency. The autoencoders were trained once for 10 epochs at the
beginning of each task. All experiments were based on PyTorch and four A100
GPUs. The code would be made publicly available.

###  4.2 Class-incremental split tasks

#### Average task accuracy

We train all tasks for $\frac{\text{total epochs}}{\text{\\# of tasks}}$
epochs before the model training on a different task. We conducted a
comprehensive evaluation in the CIFAR10/5, CIFAR100/10, and CIFAR100/20 split
tasks based on the average task accuracy, comparing to a wide range of
conventional methods. Table 1 demonstrated that Remembering Transformer
significantly enhanced model performance in the challenging class-incremental
learning without the task identity information. Remembering Transformer
surpassed the second-best method, FeTrIL [21], by 15.90%, establishing a SOTA
performance.

Table 1: Average task accuracy in the split tasks. Method | CIFAR10/5 (%) | CIFAR100/10 (%) | CIFAR100/20 (%) | Average (%)
---|---|---|---|---
SupSup [30] |  26.2 $\pm$ 0.46 |  33.1 $\pm$ 0.47 |  12.3 $\pm$ 0.30 | 23.87
BI-R+SI [26] |  41.7 $\pm$ 0.25 |  22.7 $\pm$ 0.81 |  19.1 $\pm$ 0.04 | 27.83
HyperNet [27] |  47.4 $\pm$ 5.78 |  29.7 $\pm$ 2.19 |  19.4 $\pm$ 1.44 | 32.17
MUC [19] |  53.6 $\pm$ 0.95 |  30.0 $\pm$ 1.37 |  14.4 $\pm$ 0.93 | 32.67
OWM [31] |  51.7 $\pm$ 0.06 |  29.0 $\pm$ 0.72 |  24.2 $\pm$ 0.11 | 35.63
PASS [33] |  47.3 $\pm$ 0.97 |  36.8 $\pm$ 1.64 |  25.3 $\pm$ 0.81 | 36.47
LwF [18] |  54.7 $\pm$ 1.18 |  45.3 $\pm$ 0.75 |  44.3 $\pm$ 0.46 | 48.10
iCaRL [23] |  63.4 $\pm$ 1.11 |  51.4 $\pm$ 0.99 |  47.8 $\pm$ 0.48 | 54.20
Mnemonicsy [18] |  64.1 $\pm$ 1.47 |  51.0 $\pm$ 0.34 |  47.6 $\pm$ 0.74 | 54.23
DER++ [4] |  66.0 $\pm$ 1.27 |  55.3 $\pm$ 0.10 |  46.6 $\pm$ 1.44 | 55.97
IL2A[32] |  92.0 $\pm$ 0.23 |  60.3 $\pm$ 0.14 |  57.9 $\pm$ 0.31 | 70.07
CLOM [13] |  88.0 $\pm$ 0.48 |  65.2 $\pm$ 0.71 |  58.0 $\pm$ 0.45 | 70.40
SSRE [34] |  90.5 $\pm$ 0.61 |  65.0 $\pm$ 0.27 |  61.7 $\pm$ 0.15 | 72.40
FeTrIL [21] |  90.9 $\pm$ 0.38 |  65.2 $\pm$ 0.16 |  61.5 $\pm$ 0.73 | 72.53
Ours | 99.3 $\pm$ 0.24 | 86.1 $\pm$ 2.09 | 79.9 $\pm$ 3.56 | 88.43

#### Accuracy curves

Figure 3: Test accuracy curves in the CIFAR100 tasks.

Figure 3 demonstrated the accuracy curves of the CIFAR100/10 and CIFAR100/20
split tasks. For comparison, we visualized the performance of FeTrIL [21] and
SSRE [34]. We added an oracle model based on an Independent and Identically
Distributed (IID) training setting where tasks are defined as identical
instances of the same dataset with all classes. This is equivalent to
finetuning the pretrained model on the entire dataset based on a single
adapter. The Remembering Transformer demonstrated greatly enhanced continual
learning ability. In contrast, the existing methods encountered the forgetting
problem when the task changed. The Remembering Transformer even outperforms
the oracle model in the CIFAR100/10 task leveraging the proposed modular
architecture for reduced interference among classes.

#### Memory footprint

We evaluated the memory footprint by measuring the trainable model parameter
size. Figure 4 demonstrated that the Remembering Transformer is parameter-
efficient for continual learning compared to conventional methods, reducing
the memory footprint in the CIFAR10 split task from 11.18M (FeTrIL [21]) to
0.22M. Moreover, although OWM and HyperNet utilized smaller model sizes, they
achieved much worse model performance. The Remembering Transformer achieved
SOTA performance while retaining a small memory footprint.

Figure 4: Memory footprint of the comparison models.

###  4.3 Ablation studies

We studied the efficacy of the Remembering Transformer by conducting ablations
on the generative routing mechanism and varying the model capacity of
adapters.

#### Ablating the generative routing

Remembering Transformer leverages the generative routing to facilitate sparse
adapter activation. We investigated whether the Remembering Transformer with
the generative routing ablated could tackle the CIFAR100 split tasks (see
Figure 3). The empirical results demonstrated that the mixture-of-adapters
architecture alone could not adapt to the continual learning task. It
highlighted the importance of the generative routing mechanism for a model
continually learning new tasks without forgetting old knowledge.

#### Varying the model capacity

The adapter fusion method leverages knowledge distillation to transfer old
task knowledge to a newly learned adapter for reducing the memory footprint.
To assess the model efficacy under a constrained adapter capacity, we varied
the capacity by reducing the adapters to $E={3,2}$ in the CIFAR10/5 split task
while employing a replay memory size of $M=512$. Table 3 demonstrated that the
Remembering Transformer could learn a number of tasks that were more than the
adapters, retaining competitive performance. Moreover, we varied the replay
memory size to $M=\\{128,256,512\\}$ when employing a model capacity of $E=3$
(see Table 3). Consequently, compared to FeTrIL, which obtained a performance
of 90.9% with a memory footprint of 11.18M (see Table 1), the Remembering
Transformer achieved a performance of 93.2% with a much smaller memory
footprint of 0.22M, when $E=3$ and $M=512$.

Table 2: Model capacity vs. test accuracy. #Adapters (memory footprint) | Test accuracy (%)
---|---
5 (0.37M) | 99.3 $\pm$ 0.24
3 (0.22M) |  93.2 $\pm$ 0.72
2 (0.15M) |  87.1 $\pm$ 0.85

Table 3: Replay memory size vs. test accuracy. #Replay samples | Test accuracy (%)
---|---
512 | 93.2 $\pm$ 0.72
256 |  90.5 $\pm$ 0.41
128 |  86.6 $\pm$ 0.26

###  4.4 Permutation tasks

Figure 5: Test accuracy curves of the Remembering Transformer compared to the
conventional methods for the permuted-MNIST task. Figure 6: Heatmap of the
reconstruction losses of autoencoders with respect to different tasks during
inference time.

We evaluated the Remembering Transformer in the permuted-MNIST dataset. Each
task consists of all 10 digits, with a specific permutation shift employed to
the image pixels. During inference, the model is given samples featuring the
same set of permutation patterns but lacks identity information on the various
permutations. Notably, we employed a total of 50 different random permutation
patterns and trained the model for 2000 epochs, with 40 epochs for each
permutation. We evaluated the average task accuracy each time the model
learned a new permutation. We demonstrated the Remembering Transformer’s
performance, comparing to a broad range of conventional methods for this
dataset, including Learning without Forgetting [17], Elastic Weight
Consolidation (EWC) [14], Online EWC [25], and Brain-Inspired Replay [26].
Figure 5 demonstrated that the Remembering Transformer retained its
performance while learning new permutations. While Brain-Inspired Replay also
demonstrated competitive performance, it relies on model retraining using
replay memory samples from all old tasks. In contrast, our method utilizes
only the most relevant old task’s replay samples during the adapter fusion
when there is a constraint on the model capacity. Furthermore, we visualized
the reconstruction losses between pairwise tasks and autoencoders during
inference. A small magnitude of the reconstruction loss indicates a high
degree of relevance of a task to the knowledge encoded in the specific
autoencoder. For visualization, the raw losses were normalized within a range
of 0 and 1. Figure 6 demonstrated that the proposed generative routing method
facilitated the adaptive selection of the most relevant adapter (along the
diagonal) for tackling a specific task.

##  5 Limitations and conclusions

The Remembering Transformer leverages the mixture-of-adapters architecture
that is seamlessly integrated into conventional vision Transformers for
parameter-efficient continual learning. The novelty detection method
identifies task similarities and routes samples to the most relevant adapters.
We demonstrated the superiority of the Remembering Transformer in various
class-incremental split tasks and permutation tasks, resulting in SOTA
performance with a small memory footprint. Remembering Transformer learns a
set of adapters to efficiently tackle various tasks, reducing the memory
footprint through the adapter fusion. However, addressing longer task
sequences necessitates learning hierarchical representations within these
adapters. This would facilitate more flexible and efficient knowledge reuse
for continual learning.

## Acknowledgements

We would like to thank Steve Lin and Zhirong Wu for their insightful
discussions and valuable contributions to this work.

## References

  * [1] Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon.  Uncertainty-based continual learning with adaptive regularization.  Advances in neural information processing systems, 32, 2019.
  * [2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars.  Expert gate: Lifelong learning with a network of experts.  In CVPR, 2017.
  * [3] James Urquhart Allingham, Florian Wenzel, Zelda E. Mariet, and et al.  Sparse moes meet efficient ensembles.  arXiv:2110.03360, 2021.
  * [4] Pietro Buzzega, Matteo Boschini, Angelo Porrello, and et al.  Dark experience for general continual learning: a strong, simple baseline.  In NeurIPS, 2020.
  * [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, and et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [6] Beyza Ermis, Giovanni Zappella, Martin Wistuba, and et al.  Continual learning with transformers for image classification.  In CVPR Workshops, 2022.
  * [7] Ruy Gómez-Ocádiz, Massimiliano Trippa, Chun-Lei Zhang, Lorenzo Posani, Simona Cocco, Rémi Monasson, and Christoph Schmidt-Hieber.  A synaptic signal for novelty processing in the hippocampus.  Nature Communications, 13(1):4122, 2022.
  * [8] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.  Distilling the knowledge in a neural network.  arXiv:1503.02531, 2015.
  * [9] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [10] Wenpeng Hu, Zhou Lin, Bing Liu, and et al.  Overcoming catastrophic forgetting for continual learning via model adaptation.  In ICLR, 2019.
  * [11] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton.  Adaptive mixtures of local experts.  Neural Comput., 3(1):79–87, 1991.
  * [12] Alex Kafkas and Daniela Montaldi.  How do memory systems detect and respond to novelty?  Neuroscience letters, 680:60–68, 2018.
  * [13] Gyuhak Kim, Sepideh Esmaeilpour, Changnan Xiao, and Bing Liu.  Continual learning based on OOD detection and task masking.  In CVPR Workshops, pages 3855–3865. IEEE, 2022.
  * [14] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [15] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  2009\.
  * [16] Dharshan Kumaran, Demis Hassabis, and James L McClelland.  What learning systems do intelligent agents need? complementary learning systems theory updated.  Trends in cognitive sciences, 20(7):512–534, 2016.
  * [17] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  * [18] Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun.  Mnemonics training: Multi-class incremental learning without forgetting.  In CVPR, pages 12242–12251. Computer Vision Foundation / IEEE, 2020.
  * [19] Yu Liu, Sarah Parisot, Gregory G. Slabaugh, and et al.  More classifiers, less forgetting: A generic multi-classifier paradigm for incremental learning.  In ECCV (26), volume 12371 of Lecture Notes in Computer Science, pages 699–716. Springer, 2020.
  * [20] James L McGaugh.  Memory–a century of consolidation.  Science, 287(5451):248–251, 2000.
  * [21] Grégoire Petit, Adrian Popescu, Hugo Schindler, and et al.  Fetril: Feature translation for exemplar-free class-incremental learning.  In WACV, pages 3900–3909. IEEE, 2023.
  * [22] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert.  icarl: Incremental classifier and representation learning.  In CVPR, pages 5533–5542. IEEE Computer Society, 2017.
  * [23] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert.  icarl: Incremental classifier and representation learning.  In CVPR, pages 5533–5542. IEEE Computer Society, 2017.
  * [24] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, and et al.  Scaling vision with sparse mixture of experts.  In NeurIPS, 2021.
  * [25] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.  Progress & compress: A scalable framework for continual learning.  In International conference on machine learning, pages 4528–4537. PMLR, 2018.
  * [26] Gido M Van de Ven, Hava T Siegelmann, and Andreas S Tolias.  Brain-inspired replay for continual learning with artificial neural networks.  Nature communications, 11(1):4069, 2020.
  * [27] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe.  Continual learning with hypernetworks.  In ICLR, 2020.
  * [28] Tianchun Wang, Wei Cheng, Dongsheng Luo, and et al.  Personalized federated learning via heterogeneous modular networks.  In ICDM, 2022.
  * [29] Max Welling.  Herding dynamical weights to learn.  In ICML, volume 382 of ACM International Conference Proceeding Series, pages 1121–1128. ACM, 2009.
  * [30] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, and et al.  Supermasks in superposition.  In NeurIPS, 2020.
  * [31] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu.  Continuous learning of context-dependent processing in neural networks.  Nature Machine Intelligence.
  * [32] Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Chenglin Liu.  Class-incremental learning via dual augmentation.  In NeurIPS, pages 14306–14318, 2021.
  * [33] Fei Zhu, Xu-Yao Zhang, Chuang Wang, and et al.  Prototype augmentation and self-supervision for incremental learning.  In CVPR, pages 5871–5880. Computer Vision Foundation / IEEE, 2021.
  * [34] Kai Zhu, Wei Zhai, Yang Cao, and et al.  Self-sustaining representation expansion for non-exemplar class-incremental learning.  In CVPR, pages 9286–9295. IEEE, 2022.