Snowmass White Paper: Hamiltonian Truncation

A. Liam Fitzpatrick1, Emanuel Katz1

1 Department of Physics, Boston University, Boston, MA 02215, USA

Abstract
Strongly-coupled Quantum Field Theories (QFTs) are ubiquitous in high energy
physics and many-body physics, yet our ability to do precise computations in
such systems remains limited. Hamiltonian Truncation is a method for doing
nonperturbative computations of real-time evolution in strongly coupled QFT in
the continuum limit, and works by numerically solving the Schrodinger equation
in a truncated subspace of the full Hilbert space. Recent advances in
understanding this method have opened the door to progress in a range of
applications, from gauge theories in $d\geq 2$ dimensions to relativistic
nonequilibrium physics.

##  1 Motivation

###  1.1 Limitations of the most common approaches to QFT

Quantum Field Theory (QFT) has been very effective as a framework both in
providing the language for the Standard Model (SM) as well as helping us
parametrize possible Beyond-the-Standard-Model (BSM) scenarios. For the most
part, by QFT one usually means either the Perturbation Theory (PT)
formulation, or the Lattice Field Theory (LFT) formulation. Both techniques
have been enormously successful, partly because Nature has been kind to us. In
particular, we have been very lucky that the SM is weakly coupled in the
ultraviolet (UV) and that the only sector that is strongly coupled in the
infrared (IR), Quantum Chromodynamics (QCD), has a lattice formulation (which
for instance requires the empirical fact that the $\theta_{\rm QCD}$ parameter
is negligible).

Nevertheless, it is important to note that both PT and LFT have limitations
and inefficiencies. PT is an incomplete formulation, and clearly fails to
describe physics when the coupling becomes strong. However, it is also of note
that PT fails even for so-called “weakly coupled” theories in processes with a
large multiplicity of particles, and so even at weak coupling some important
phenomena are beyond its reach. For instance, at high energies most states of
weakly coupled QFTs are expected to exhibit chaotic properties. In contrast to
PT, LFT is a complete nonperturbative formulation of QFT, but as a universal
approach to QFT it faces its own challenges. Perhaps the most direct, is that
a lattice is in fact a dynamical system in its own right, with potentially
undesired physical consequences. Indeed, LFT is rather like a medium, with QFT
degrees of freedom only emergent in the IR, typically when lattice parameters
are tuned appropriately. Phrased in this way, obtaining a particular desired
QFT in the IR is not necessarily guaranteed, and can involve subtle lattice
dynamics. This appears to be the case for chiral gauge theories, like the SM.
It is noteworthy that despite the fact that LFT is a rather mature field with
beautiful constructions (such as domain wall fermions for maintaining chiral
symmetry) and decades of development, it has thus far been unable to
convincingly yield chiral gauge theories. This is true even in the much
simpler setting of 2d chiral gauge theories. Other theories with non-trivial
space-time symmetries, like supersymmetry, have also been difficult to realize
using LFT. Moreover, as is well known, even if in principle LFT dynamics
manifest a certain QFT in the IR, practical considerations of Monte Carlo
methods further require that the weight associated with field configuration on
the lattice must be positive definite (i.e. cannot fluctuate in sign or have a
phase). Again, though lattices have been used extensively both in the High
Energy Theory (HET) and the Condensed Matter Theory (CMT) communities for
decades, a robust solution to this “sign-problem” has not yet emerged. It thus
seems entirely plausible that the SM as well as many other interesting QFTs
useful for BSM physics will not have practical LFT formulations anytime soon.

In addition, even for cases where an LFT formulation of a QFT is known, there
are certain dynamical quantities that are difficult to extract directly due to
the fact that the LFT formulation is typically Euclidean. For example, a QCD
observable like $e^{+}e^{-}\rightarrow hadrons$ is most directly related to
the Wightman current two-point correlator as a function of real time, i.e. in
Lorentzian signature. Similarly, obtaining information about bound-state
wavefunctions, like determining the PDF of the proton is challenging to do
directly. Finally, there is no currently available LFT approach in
circumstances when one is interested in real-time dynamics in an environment
with a large multiplicity of particles, such as at finite temperature or
chemical potential, (e.g. for heavy ion collisions, or in the early universe,
or with high particle multiplicity events in high energy scattering of
scalars).

##  2 Recent Developments and Future Goals

Hamiltonian Truncation is a framework for doing nonperturbative computations
in quantum systems that works rather differently from PT or LFT. In
Hamiltonian truncation one attempts to approximate the eigenstates of the
Hamiltonian by truncating the full Hilbert space of the theory to a well-
chosen finite-dimensional subspace. The resulting Hamiltonian in the truncated
space is then a finite size matrix which can be numerically diagonalized.
Therefore, this approach uses only the degrees of freedom of the QFT itself,
and can in many cases preserve the symmetries enjoyed by the untruncated
theory. Typically, the Hamiltonian is taken to be $H=H_{0}+V$, where $H_{0}$
is integrable, with known eigenstates, while $V$ consists of a relevant local
operator. In the future, however, one could envision studying cases where the
Hamiltonian, $H_{0}$, for the undeformed theory is not integrable, but is a
CFT that can be numerically determined using the conformal bootstrap. Unlike
LFT, the QFT is realized directly, without describing it as the IR
fluctuations of some other quantum system.

QFT Hamiltonian truncation was begun by Brooks and Frautschi in 1984 [1], but
reached a wider audience in 1990 with seminal papers by Yurov and
Zamolodchikov [2, 3, 4]. Their approach is known as the Truncated Conformal
Space Approach (TCSA). Here $H_{0}$ is the CFT Hamiltonian and the truncation
is a cutoff on the energy of CFT states on the sphere (in the context of
radial quantization). This approach was quickly applied to many other RG flows
in 2d relativistic field theories, including all deformations of the 2d Ising
model and other minimal models, as well as the Sine-Gordon and Wess-Zumino-
Witten models; for a recent thorough review, see [5]. It was also later
generalized to $d>2$ [6]. An appealing aspect of TCSA is that it was
formulated in terms of CFT states, and therefore did not require a theory to
have a Lagrangian description. A complementary approach also emerged in the
1990s, named Discrete Light-cone Quantization (DLCQ) [7]. This was instead a
Fock space method formulated in the infinite momentum or light-cone (LC) limit
of a 2d theory. Here $H_{0}$ is the free particle lightcone Hamiltonian and
the truncation is on the total units of lightcone momentum carried by the
particles. DLCQ was mainly applied to 2d gauge theories and supersymmetric
(SUSY) theories, both of which benefit from LC quantization’s ability to
preserve gauge invariance and part of the supersymmetry. A more recent
truncation scheme is Lightcone Conformal Truncation (LCT) [8, 9, 10, 11]. Here
$H_{0}$ is the lightcone Hamiltonian of the CFT in Minkowski space and the
truncation is on the scaling dimension of primary operators in the CFT basis.
In early work, the above methods mostly focused on obtaining the energy
spectrum for different theories in 2d. However, the full set of observables
and theories that can be studied with Hamiltonian Truncation is much richer.
In the following, we describe some of the main developments of this method in
recent years, as well as possible major applications in the future, and key
goals for improving its efficiency and breadth.

###  2.1 Gauge Theories

One of the most important goals for Hamiltonian Truncation is to be a
versatile tool for studying gauge theories at strong coupling. The major
challenge with studying gauge theories is that an energy or momentum cutoff in
the Fock space framework violates gauge invariance. In $d=2$, where gauge
fields have no degrees of freedom, this problem can be solved by using
lightcone quantization methods. Consequently, there have been a large number
of nonperturbative studies of 2d nonabelian gauge theories coupled to matter
fields. These 2d methods are understood quite well at this point and it is
known how to handle an arbitrary gauge group with scalar and matter fields in
arbitrary representations [12, 13, 14, 11, 15]; for instance, they have
produced important results on the phase structure of 2d QCD with adjoint
matter. Ideas for preserving SUSY are also well studied [16, 17, 18, 19, 20,
21]. The space of Lagrangian theories containing scalars, fermions, and gauge
fields is vast and fascinating, and provides an extremely rich set of toy
models with which to explore strong dynamics and non-trivial RG-flows. The
chiral limit of non-abelian theories, in addition, is dual in the IR to
deformations of many WZW and minimal models. It was challenging to reach this
chiral limit with earlier truncation methods; however, with new LCT technology
the chiral frontier is now accessible, yielding precise results (see for
example Fig. 2, right) [22]. Another case which can be attempted with existing
technology is that of 2d QED, coupled to scalars or fermions, with a nonzero
$\theta$ parameter. This example is expected to have an interesting phase
structure, but is difficult to analyze using Monte Carlo due to a sign problem
when $\theta\neq 0$.

#### Future Goals

For $d>2$, however, new ideas must be implemented to maintain gauge
invariance. In principle, in $d=4$ the issue with gauge invariance can be
resolved by making use of ideas from Conformal Field Theory. In particular,
many interesting asymptotically free gauge theories can be embedded into a
weakly coupled CFT by adding vector-like matter. For instance, by adding more
flavors, QCD can be made conformal and also weakly coupled (e.g. at
$N_{f}=15$, $\alpha_{s}^{*}\approx 0.15$, while at $N_{f}=16$,
$\alpha_{s}^{*}\approx 0.04$), known as a Banks-Zaks-Caswell fixed point. One
can then get back regular confining QCD simply by taking the weakly coupled
CFT and deforming it by a relevant deformation – namely, a mass for all the
extra flavors that were added, $V=\int
d^{3}x~{}m\sum_{i}\bar{\psi}_{i}\psi_{i}$. In this context, $H_{0}$ is just
the CFT Hamiltonian for the fixed point, and the matrix elements of the
deformation, $\langle{\cal O}_{i}|V|{\cal O}_{j}\rangle$ are in essence
completely determined by the CFT data (i.e. the dimensions and OPE
coefficients of the mass operator with all other operators) and by conformal
kinematic functions which determine CFT three-point functions. This data can
be computed in perturbation theory or possibly by the conformal bootstrap in
the future. The CFT states are all gauge invariant (being related to local
gauge invariant operators), and the relevant mass operator in $V$ is also
local and gauge invariant. Thus, the entire construction only uses gauge
invariant data, and there is no danger that a UV-cutoff will spoil gauge
invariance. In principle, TCSA and the LCT method, both of which are CFT
deformations approaches, can make use of this construction. Note, that this
scheme can realize asymptotically free chiral gauge theories as well, by
adding vector-like matter to make such theories conformal.

Implementing the above construction of 4d QCD would be a major undertaking,
however. The practical issue currently is that neither the CFT data nor the
kinematic CFT functions in 4d (needed for LCT) have yet been computed. It is
thus more prudent to start with a UV fixed point theory about which more CFT
data is known. ${\cal N}=4$ SYM at large $N$ is a natural choice because of
integrability. It can be deformed by mass terms that preserve some but not all
of the supersymmetry, providing additional control over UV divergences. Still,
much work is needed even here, involving a collection of expertise. The
kinematic CFT functions need to be parameterized properly and the specific CFT
data needs to be extracted efficiently from integrability technology. Both are
significant projects that can be attempted in the immediate future, but must
be completed before truncation work can begin.

Finally, gauge theories in $d=3$ are likely to be intermediate in difficulty
between the $d=2$ and $d=4$ cases. Because of their super-renormalizability,
the breaking of gauge invariance by an energy cutoff may still be a problem
that can be resolved by adding appropriate counterterms. Recent advances [23,
24] in understanding how to handle counterterms in $d=3$ provide a concrete
set of strategies that may be applied to 3d gauge theories. A natural target
is 3d QED, whose phase structure as a function of the number of flavors,
$N_{f}$, remains an open question.

###  2.2 Correlators and Scattering Amplitudes from Eigenstates

One of the main advantages of Hamiltonian Truncation is that by diagonalizing
the Hamiltonian, one obtains the eigenvalue spectrum as well as the
eigenvectors of the theory at strong coupling. Together, these represent an
enormous amount of information about the dynamics of the theory, allowing one
to compute real-time evolution of any initial state. Actually, one can compute
both Lorentzian and Euclidean correlators, and Euclidean correlators of local
operators in the vacuum, which are the correlators naturally computed in LFT,
are straightforward to obtain from the eigenvector data. For instance, the
two-point function of a local operator ${\cal O}$ in the vacuum can be
obtained through the evaluation of the spectral function $\rho_{\cal O}$ for
${\cal O}$, as a sum over states $\rho_{\cal O}(s)\sim\sum_{j}|\langle{\cal
O}(0)|j\rangle|^{2}\delta(s-s_{j})$. The Euclidean two-point function is then
the Euclidean free propagator weighted by the spectral function. But now one
can also use the spectral function to compute correlators in the Lorentzian
regime as well, or indeed under any complex rotation of the momenta or
positions! This procedure works much better than one might expect. In
particular, truncation discretizes the spectrum, so the spectral function
becomes a sum over unphysical delta functions even when it should be a smooth
function. Nevertheless, methods that take advantage of unitarity and
analyticity have been developed to resolve this issue [11, 25] and accurately
restore the smooth underlying spectral functions. In Fig. 2 (right), we show
these smoothing methods applied to the Zamolodchikov $C$-function (the
integral of the $T_{--}$ spectral function) in 2d QCD with $N_{c}=3$ and a
very light quark in the fundamental. This theory is strongly coupled in the
IR, where it is expected to be dual to the Sine-Gordon model, allowing a
precise test of the smoothed $C$-function from truncation. Remarkably, even in
this Lorentzian regime, Hamiltonian Truncation achieves better than 1%
accuracy using a basis of only 77 states. These methods can likely be
understood more thoroughly and improved.

#### Future Goals

Scattering amplitudes and multi-particle form factors are fundamental in QFT.
The difficulty in extracting them from the eigenstate data is that
multiparticle asymptotic states are not eigenstates of the Hamiltonian. The
standard approach to extracting scattering amplitudes is the LSZ procedure,
and developing LSZ applied to Hamiltonian truncation is likely to be a crucial
area for future development. There is also the potential for new, more
indirect techniques for obtaining scattering amplitudes and form factors.
Indeed, in $d=2$, methods have been used to obtain scattering amplitudes from
truncation data using analyticity and crossing symmetry rather than LSZ
reduction [5, 25, 26]. In Fig. 1 (right), we show a result from [26] for the
elastic S-matrix of the nonintegrable 2d Ising model with both a spin $\sigma$
and energy $\epsilon$ deformation, extracted indirectly from the eigenvector
data of Hamiltonian Truncation calculations using Luscher’s method [27, 28];
the presence of an unstable resonance shows up as a zero since the region
plotted represents the physical (first) sheet in Mandelstam $s$. We also show
(Fig. 1,left) a result from [25] for the elastic S-matrix in 2d $\phi^{4}$
theory for couplings ranging from the free theory up to the fixed point,
extracted from Hamiltonian Truncation calculations; at the fixed point, the
result matches the prediction of the free 2d Ising model. In both of these
results, Hamiltonian Truncation data was combined with S-matrix bootstrap
methods using analyticity and unitarity assumptions.

Figure 1: Examples of Scattering Amplitudes in 2d theories calculated using
Hamiltonian Truncation. Left: (from [25]) the elastic scattering amplitude in
2d $\phi^{4}$ theory, for couplings $\bar{\lambda}$ ranging from the free
theory up to near the fixed point, where the amplitude matches the prediction
of the free 2d Ising model. Right: A plot (from [26]) of $\log|S(z)|$, where
$S(z)$ is the elastic scattering amplitude in the 2d Ising model with both a
spin $\sigma$ and energy $\epsilon$ deformation, analytically continued to the
complex plane in the variable $z$, where
$(\frac{1+z}{1-z})^{2}=\frac{s(s-4m^{2})}{3m^{4}}$, which maps the first sheet
in Mandelstam $s$ into the unit disk.

There is also an important connection between scattering amplitudes and
spectral densities, when a strongly coupled theory is perturbatively coupled
to an external probe: the spectral densities themselves directly provide the
scattering amplitudes of the weakly coupled probe particles. This is the case
for instance with using electromagnetic scattering as a probe of QCD. In this
way, the 2d QCD spectral density shown in Fig. 2 (left) can be directly
interpreted as a scattering cross-section similar to that of
$e^{+}e^{-}\rightarrow$ hadrons in 4d QCD. So, while much work remains to be
done, in principle there is a path towards computing
$\sigma(e^{+}e^{-}\rightarrow$ hadrons) in 4d QCD with Hamiltonian truncation,
making it an important target for the future.

Figure 2: Left: A plot of a Hamiltonian Truncation computation of the spectral
density for the stress tensor $T_{\mu\nu}$ as a function of center-of-mass
energy $\sqrt{s}$, in 2d QCD with $N_{c}=3$ colors and a single massive
fundamental quark. The theory contains a light scalar ‘pion’ meson with mass
$m_{\pi}$, and a stable bound state of two ‘pions’ at $\sqrt{s}\lesssim
2m_{\pi}$. An unstable resonance is also visible at around $\sqrt{s}\sim
4.4m_{\pi}$. Right: A plot (from [22]) of the Zamolodchikov $C$-function
(which is the integral of the stress tensor spectral density), for a very
small value of the quark mass in which case the theory is expected to be dual
to the Sine-Gordon model. Black, dashed: the raw $C$-function from truncation,
which contains unphysical steps due to the discreteness of the truncated
spectrum. Red, solid: the improved $C$-function, after smoothing. Blue,
dashed: the $C$-function in the Sine-Gordon model computed using integrability
methods. The relative error between the Sine-Gordon $C$-function and the
improved $C$-function from truncation is shown in the inset.

###  2.3 Improved Truncation and Renormalization Procedures

A generic property of truncation schemes is the rapid growth in the size of
the space of states as the truncation parameter is increased. Consequently, a
concern is that obtaining results with the desired high level of precision
could require an intractably large subspace in Hamiltonian Truncation.

One strategy for overcoming this problem is to find more efficient truncation
subspaces that lead to faster convergence. For instance, in [29], the authors
studied 2d $\phi^{4}$ theory using DLCQ, and with improved algorithms and
computational resources the authors were able to take the truncation parameter
$K=96$, corresponding to about 60 million states. Such a large truncation made
it possible to reliably establish that the truncation error in the critical
coupling $\lambda_{c}$ converged to zero like $O(\frac{1}{\sqrt{K}})$ as
$K\rightarrow\infty$. By contrast, in the LCT basis, the truncation error
converges like $O(\frac{1}{K})$; in fact, to good approximation, the critical
coupling in LCT and DLCQ as a function of $K$ are related by
$\lambda_{c}^{(\rm LCT)}(K)=\lambda_{c}^{(\rm DLCQ)}(K^{2})$.111In 2d
$\phi^{4}$ theory in both DLCQ and LCT, the number of states as a function of
$K$ is the number $P(K)$ of integer partitions of $K$, which grows rapidly
with $K$ like $P(K)\approx\frac{1}{4K\sqrt{3}}e^{\pi\sqrt{\frac{2}{3}K}}$; for
instance, $P(20)=627$, while $P(400)=6.7\times 10^{18}$.  So, different
choices can lead to vastly different rates of convergence as the size of the
subspace is increased. Additional strategies can be employed to identify more
efficient subspaces that keep only the most important states in the basis; see
section VII of [5] for examples.

A second strategy is to systematically approximate the contributions of states
above the truncation. Here, significant improvements have come from recent
progress on how to formulate low-energy effective descriptions for cutoffs in
Hamiltonian frameworks [30, 31, 32]. Such progress not only allows one to
improve the rate of convergence by explicitly computing the leading
corrections in the truncation parameter, but moreover provides a better
understanding of the expected form and truncation-dependence of such
corrections. For example, the improvements of [31] yielded a very precise
value of the critical coupling for 2d $\phi^{4}$, as well as various other
data near criticality (see Fig. 3). Also, quite importantly, understanding the
effect of truncation cutoffs is crucial for handling UV divergences in a
truncation framework.

In fact, the difficulty of renormalization with an energy cutoff is one of the
most conceptually challenging issues facing the implementation of Hamiltonian
Truncation in a wider range of theories. A hard cutoff in the total energy of
the system is a nonlocal condition, and relates distant regions of spacetime
to each other. Consequently, the counterterms that are required to cancel
divergences often cannot be represented as local interactions, and are much
more difficult to understand [33]. However, there have been a number of
significant breakthroughs in the past few years [24, 23, 34, 6, 35] in the
treatment of such counterterms. As a result, it has become possible to use
Hamiltonian truncation in many super-renormalizable theories with UV
divergences in $d>2$. In particular, [24, 23] recently completed Hamiltonian
truncation studies of strongly coupled 3d $\phi^{4}$ theory. A highly
nontrivial check that the method employed in [23] is working is that near the
critical point, correlators of operators exhibit IR universality over a range
of scales, seen by different UV operators, with the expected scaling exponents
(shown in Fig. 4, right). Likewise, the method employed in [24] passes the
nontrivial test that the spectrum satisfies a strong-weak duality of the model
(Fig. 4, left).

Figure 3: Comparison of energy levels at the critical coupling for 2d
$\phi^{4}$ with CFT predictions (from [31]).

Figure 4: Hamiltonian Truncation results for 3d $\phi^{4}$ theory at strong
coupling. Left: The mass spectrum from [24] is explicitly seen to satisfy a
strong-weak duality. Right: the correlators of various operators from [23]
near the critical point obey universal IR scaling behavior, consistently with
the known critical exponents of the 3d Ising model.

#### Future Goals

Improving the truncation procedure will require developing more efficient
methods for obtaining the Hamiltonian matrix elements themselves. Finding
these matrix elements is often a highly nontrivial task. In TCSA and LCT, they
are equivalent to OPE coefficients of primary operators in the CFT of the
ultraviolet fixed point. Even in the case where the UV fixed point is a free
theory, these OPE coefficients are computationally expensive at very large
truncations, due to the complexity of high-dimension primary operators. In
fact, already, taking advantage of the conformal structure of the UV theory
operators has been crucial to being able to compute these OPE coefficients
efficiently, allowing for work with very large bases [11, 36]. Moreover, in
the LCT framework the Hamiltonian matrix elements involve kinematic factors,
which are essentially the Fourier transforms of CFT three-point functions.
While these are known in closed form in 2d, in $d>2$ the required Fourier
transforms are still not known in general.

In addition, there is more work to be done on understanding counterterms in
Hamiltonian Truncation frameworks. Even with recent improvements, the
available procedures for dealing with counterterms can be ad hoc at times, and
further work is needed. Moreover, they should be employed and tested in a
wider range of theories, as at present most $d>2$ work has been limited to
scalar field theories. A natural extension is to a 3d Yukawa theory, as
possibly the simplest theory with both scalars and fermions.

###  2.4 Non-Equilibrium Physics

Because Hamiltonian Truncation produces the nonperturbative eigenvalues and
eigenvectors of a theory, it provides one of the only known methods for
calculating real-time non-equilibrium dynamics in strongly coupled,
relativistic systems. Various measures of quantum chaos can be calculated and
compared with the predictions of Random Matrix Theory (RMT). Eigenstates can
be used to directly construct canonical and microcanonical ensembles, or to
test eigenstate thermalization, and nonequilibrium fluctuations around such
backgrounds can then be computed from correlators of local operators [37, 38,
39].

Most quantities of this nature can already be computed in practice in simple
2d theories. For instance, using the distribution of eigenvalue spacings, one
can observe the chaotic nature of 2d $\phi^{4}$ theory at both weak and strong
couplings (see Fig. 5, left plot), and more generally in deformations of
rational theories [40, 41]; one can also study the emergence of chaos through
the statistics of eigenvector components. From the density of eigenvalues as a
function of energy, one can extract the entropy and other thermodynamic
quantities. This includes quantities which characterize hydrodynamics in 2d,
like the speed of sound, and the KPZ diffusion constant [42]. Thus, truncation
is unique in its ability to capture both thermalization and approaches to
equilibrium (as expected of UV relativistic systems), followed by late time
behavior indicative of chaos. Indeed, other numerical approaches to
nonequilbrium physics use lattices, which violate space-time symmetries and
whose typical excited states are rather different from relativistic QFTs.
Calculating the Spectral Form Factor (SFF) exhibits the flexibility enjoyed by
truncation methods (see Fig. 5, right plot): It is a real-time observable
whose early time behavior is determined by the physical density of states of
the QFT, but whose late time behavior is controlled by universal chaotic
dynamics as dictated by RMT.

Figure 5: Left: Distribution of eigenvalue spacings in 2d $\phi^{4}$ theory
from Hamiltonian Truncation. The distribution closely follows the Wigner-Dyson
distribution predicted by RMT, rather than a Poisson distribution which is
characteristic of integrable theories. Similar results have been obtained in
deformations of rational 2d CFTs in [40]. Right: The magnitude of the
integrated Spectral Form Factor, SFF$(t)\equiv\sum_{i,j}e^{i(E_{i}-E_{j})t}$
(show as data points) in strongly coupled 2d $\phi^{4}$ theory, for various
values of the truncation parameter $\Delta_{\rm max}$, compared against the
late-time RMT predictions (solid lines) and the early-time prediction from the
density of states (dashed lines). Plots from [43].

#### Future Goals

Extending this approach holds the potential to address a wide range of
questions about nonequilibrium physics in strongly coupled theories. The study
of relativistic hydrodynamics in particular presents an opportunity for
Hamiltonian Truncation in continuum QFT, since the presence of a lattice in
most other strongly coupled numeric approaches breaks the translation
invariance that is essential for the long-lived hydrodynamic modes. The
inclusion of theories with conserved global symmetries would also allow one to
study diffusion and the effect of chemical potentials. In particular, an
important goal is to apply Hamiltonian truncation to study transport in the 3d
$O(2)$ model, which describes superfluids and many other systems at low
energies. The wealth of data provided by the knowledge of eigenvectors in
Hamiltonian Truncation offers many exciting opportunities for progress in the
future.

### Acknowledgments

We thank Joan Elias-Miró, Robert Konik, and Slava Rychkov for comments on an
earlier draft. ALF and EK were supported in part by the US Department of
Energy Office of Science under Award Number DE-SC0015845.

## References

  * [1] E. Brooks and S. C. Frautschi, _Scalars coupled to fermions in 1+ 1 dimensions_ , _Zeitschrift für Physik C Particles and Fields_ 23 (1984) 263–273.
  * [2] V. Yurov and A. B. Zamolodchikov, _Truncated comformal space approach to scaling lee-yang model_ , _International Journal of Modern Physics A_ 5 (1990) 3221–3245.
  * [3] V. P. Yurov and A. B. Zamolodchikov, _Correlation functions of integrable 2-D models of relativistic field theory. Ising model_ , _Int. J. Mod. Phys. A_ 6 (1991) 3419–3440.
  * [4] M. Lassig, G. Mussardo and J. L. Cardy, _The scaling region of the tricritical Ising model in two-dimensions_ , _Nucl. Phys. B_ 348 (1991) 591–618.
  * [5] A. J. James, R. M. Konik, P. Lecheminant, N. J. Robinson and A. M. Tsvelik, _Non-perturbative methodologies for low-dimensional strongly-correlated systems: From non-abelian bosonization to truncated spectrum methods_ , _Reports on Progress in Physics_ 81 (2018) 046002.
  * [6] M. Hogervorst, S. Rychkov and B. C. van Rees, _Truncated conformal space approach in d dimensions: A cheap alternative to lattice field theory?_ , _Phys. Rev. D_ 91 (2015) 025005, [1409.1581].
  * [7] H. C. Pauli and S. J. Brodsky, _Discretized Light Cone Quantization: Solution to a Field Theory in One Space One Time Dimensions_ , _Phys. Rev. D_ 32 (1985) 2001.
  * [8] E. Katz, G. Marques Tavares and Y. Xu, _Solving 2D QCD with an adjoint fermion analytically_ , _JHEP_ 05 (2014) 143, [1308.4980].
  * [9] S. S. Chabysheva and J. R. Hiller, _Basis of symmetric polynomials for many-boson light-front wave functions_ , _Phys. Rev. E_ 90 (2014) 063310, [1409.6333].
  * [10] E. Katz, Z. U. Khandker and M. T. Walters, _A Conformal Truncation Framework for Infinite-Volume Dynamics_ , _JHEP_ 07 (2016) 140, [1604.01766].
  * [11] N. Anand, A. L. Fitzpatrick, E. Katz, Z. U. Khandker, M. T. Walters and Y. Xin, _Introduction to Lightcone Conformal Truncation: QFT Dynamics from CFT Data_ , 2005.13544.
  * [12] G. Bhanot, K. Demeterfi and I. R. Klebanov, _(1+1)-dimensional large N QCD coupled to adjoint fermions_ , _Phys. Rev. D_ 48 (1993) 4980–4990, [hep-th/9307111].
  * [13] K. Demeterfi, I. R. Klebanov and G. Bhanot, _Glueball spectrum in a (1+1)-dimensional model for QCD_ , _Nucl. Phys. B_ 418 (1994) 15–29, [hep-th/9311015].
  * [14] R. Gopakumar, A. Hashimoto, I. R. Klebanov, S. Sachdev and K. Schoutens, _Strange Metals in One Spatial Dimension_ , _Phys. Rev. D_ 86 (2012) 066003, [1206.4719].
  * [15] R. Dempsey, I. R. Klebanov and S. S. Pufu, _Exact symmetries and threshold states in two-dimensional models for QCD_ , _JHEP_ 10 (2021) 096, [2101.05432].
  * [16] Y. Matsumura, N. Sakai and T. Sakai, _Mass spectra of supersymmetric Yang-Mills theories in (1+1)-dimensions_ , _Phys. Rev. D_ 52 (1995) 2446–2461, [hep-th/9504150].
  * [17] A. Hashimoto and I. R. Klebanov, _Matrix model approach to d $>$ 2 noncritical superstrings_, _Mod. Phys. Lett. A_ 10 (1995) 2639–2649, [hep-th/9507062].
  * [18] F. Antonuccio, O. Lunin and S. Pinsky, _Nonperturbative spectrum of two-dimensional (1,1) superYang-Mills at finite and large N_ , _Phys. Rev. D_ 58 (1998) 085009, [hep-th/9803170].
  * [19] O. Lunin and S. Pinsky, _SDLCQ: Supersymmetric discrete light cone quantization_ , _AIP Conf. Proc._ 494 (1999) 140–218, [hep-th/9910222].
  * [20] J. R. Hiller, O. Lunin, S. Pinsky and U. Trittmann, _Towards a SDLCQ test of the Maldacena conjecture_ , _Phys. Lett. B_ 482 (2000) 409–416, [hep-th/0003249].
  * [21] A. L. Fitzpatrick, E. Katz, M. T. Walters and Y. Xin, _Solving the 2D SUSY Gross-Neveu-Yukawa model with conformal truncation_ , _JHEP_ 01 (2021) 182, [1911.10220].
  * [22] N. Anand, A. L. Fitzpatrick, E. Katz and Y. Xin, _Chiral Limit of 2d QCD Revisited with Lightcone Conformal Truncation_ , 2111.00021.
  * [23] N. Anand, E. Katz, Z. U. Khandker and M. T. Walters, _Nonperturbative dynamics of (2+1)d $\phi^{4}$-theory from Hamiltonian truncation_, _JHEP_ 05 (2021) 190, [2010.09730].
  * [24] J. Elias-Miró and E. Hardy, _Exploring Hamiltonian Truncation in $\bf{d=2+1}$_, _Phys. Rev. D_ 102 (2020) 065001, [2003.08405].
  * [25] H. Chen, A. L. Fitzpatrick and D. Karateev, _Form Factors and Spectral Densities from Lightcone Conformal Truncation_ , 2107.10285.
  * [26] B. Gabai and X. Yin, _On The S-Matrix of Ising Field Theory in Two Dimensions_ , 1905.00710.
  * [27] M. Luscher, _Volume Dependence of the Energy Spectrum in Massive Quantum Field Theories. 1. Stable Particle States_ , _Commun. Math. Phys._ 104 (1986) 177.
  * [28] M. Luscher, _Volume Dependence of the Energy Spectrum in Massive Quantum Field Theories. 2. Scattering States_ , _Commun. Math. Phys._ 105 (1986) 153–188.
  * [29] J. P. Vary, M. Huang, S. Jawadekar, M. Sharaf, A. Harindranath and D. Chakrabarti, _Critical Coupling for Two-dimensional $\phi^{4}$ Theory in Discretized Light-Cone Quantization_, 2109.13372.
  * [30] S. Rychkov and L. G. Vitale, _Hamiltonian truncation study of the $\phi^{4}$ theory in two dimensions_, _Phys. Rev. D_ 91 (2015) 085011, [1412.3460].
  * [31] J. Elias-Miro, S. Rychkov and L. G. Vitale, _NLO Renormalization in the Hamiltonian Truncation_ , _Phys. Rev. D_ 96 (2017) 065024, [1706.09929].
  * [32] T. Cohen, K. Farnsworth, R. Houtz and M. A. Luty, _Hamiltonian Truncation Effective Theory_ , 2110.08273.
  * [33] R. J. Perry, A. Harindranath and K. G. Wilson, _Light-front tamm-dancoff field theory_ , _Physical Review Letters_ 65 (1990) 2959.
  * [34] D. Rutter and B. C. van Rees, _Counterterms in Truncated Conformal Perturbation Theory_ , 1803.05798.
  * [35] J. Elias Miro and J. Ingoldby, _Hamiltonian Truncation with Larger Dimensions_ , 2112.09049.
  * [36] N. Anand, Z. U. Khandker and M. T. Walters, _Momentum space CFT correlators for Hamiltonian truncation_ , _JHEP_ 10 (2020) 095, [1911.02573].
  * [37] A. J. A. James, R. M. Konik and N. J. Robinson, _Nonthermal states arising from confinement in one and two dimensions_ , _Phys. Rev. Lett._ 122 (2019) 130603, [1804.09990].
  * [38] N. J. Robinson, A. J. A. James and R. M. Konik, _Signatures of rare states and thermalization in a theory with confinement_ , _Phys. Rev. B_ 99 (2019) 195108, [1808.10782].
  * [39] T. Rakovszky, M. Mestyán, M. Collura, M. Kormos and G. Takács, _Hamiltonian truncation approach to quenches in the ising field theory_ , _Nuclear Physics B_ 911 (2016) 805–845.
  * [40] G. P. Brandino, R. M. Konik and G. Mussardo, _Energy Level Distribution of Perturbed Conformal Field Theories_ , _J. Stat. Mech._ 1007 (2010) P07013, [1004.4844].
  * [41] M. Srdinšek, T. Prosen and S. Sotiriadis, _Signatures of Chaos in Nonintegrable Models of Quantum Field Theories_ , _Phys. Rev. Lett._ 126 (2021) 121602, [2012.08505].
  * [42] L. V. Delacretaz, A. L. Fitzpatrick, E. Katz and M. T. Walters, _Thermalization and Hydrodynamics of Two-Dimensional Quantum Field Theories_ , 2105.02229.
  * [43] L. V. Delacretaz, A. L. Fitzpatrick, E. Katz and M. T. Walters. Private communication.


# CITB: A Benchmark for Continual Instruction Tuning

Zihan Zhang1, Meng Fang2, Ling Chen1, Mohammad-Reza Namazi-Rad3
1University of Technology Sydney 2University of Liverpool
3University of Wollongong
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Continual learning (CL) is a paradigm that aims to replicate the human ability
to learn and accumulate knowledge continually without forgetting previous
knowledge and transferring it to new tasks. Recent instruction tuning (IT)
involves fine-tuning models to make them more adaptable to solving NLP tasks
in general. However, it is still uncertain how instruction tuning works in the
context of CL tasks. This challenging yet practical problem is formulated as
Continual Instruction Tuning (cit). In this work, we establish a cit benchmark
consisting of learning and evaluation protocols. We curate two long dialogue
task streams of different types, InstrDialog and InstrDialog++, to study
various CL methods systematically. Our experiments show that existing CL
methods do not effectively leverage the rich natural language instructions,
and fine-tuning an instruction-tuned model sequentially can yield similar or
better results. We further explore different aspects that might affect the
learning of cit. We hope this benchmark will facilitate more research in this
direction111Code and data are available at https://github.com/hyintell/CITB..

##  1 Introduction

[width=]instruction_example_cropped.pdf

Figure 1:  Illustration of proposed continual instruction tuning (cit). Unlike
previous works, we evaluate the instruction-tuned model on the initial
training, unseen, and newly learned tasks.

Recent studies have shown that multi-task instruction tuning (IT) makes
language models better zero-shot learners (Wei et al., 2022; Sanh et al.,
2022; Wang et al., 2022; Chung et al., 2022; Longpre et al., 2023). IT fine-
tunes pre-trained language models (PLMs) on various tasks with natural
language instructions (Fig.1) and can achieve remarkably well generalization
to unseen tasks.

Despite their impressive performance, these instruction-tuned PLMs still fall
short on domain-specific tasks due to the limited exposure to relevant
knowledge and vocabulary from the training corpus (Luo et al., 2023).
Moreover, PLMs are static after deployment, and there is no mechanism to
update themselves or adapt to a changing environment (Zhang et al., 2023;
Bubeck et al., 2023).

Continual learning (CL) aims to enable information systems to learn from a
continuous data stream across time (Biesialska et al., 2020). Therefore, it is
promising to leverage CL for instruction-tuned PLMs to continually adapt to
new domains and tasks without costly re-training. Despite its importance, it
is non-trivial to alleviate catastrophic forgetting, a phenomenon in which
previously learned knowledge or abilities are degraded due to overwritten
parameters (McCloskey and Cohen, 1989). Moreover, enabling knowledge transfer
is also essential since many tasks are similar and have common knowledge (Ke
et al., 2021).

Unfortunately, there is little work on applying CL for IT and has only been
explored in rather specific settings. Scialom et al. (2022) continually fine-
tune a T0 (Sanh et al., 2022) on eight new tasks with memory replay to avoid
forgetting. Despite effectiveness, they need to store large number of
instances per task in memory, which is too costly when scaling to a larger
number of tasks. In addition, they do not study knowledge transfer between
tasks. Yin et al. (2022) propose to use history task instructions to reduce
forgetting and enable knowledge transfer. However, they do not compare with
commonly adopted CL methods, which makes the effectiveness of other CL methods
unknown. Moreover, they only evaluate the model on the newly learned tasks
while largely ignoring previously learned tasks during the multi-task training
stage (Fig.1). They also overlook the intrinsic ability of the instruction-
tuned model on unseen tasks. Lastly, both of them use different evaluation
metrics and setups, which creates an obstacle to comparing different
techniques and hinders the development of this field.

To this end, we first formulate this practical yet under-explored problem as
Continual Instruction Tuning (cit). Then, we propose a first-ever benchmark
suite to study cit systematically. Our benchmark, citB, consists of both
learning and evaluation protocol and is built on top of the recently proposed
SuperNI dataset (Wang et al., 2022). We create two cit task streams:
InstrDialog stream, which consists of 19 dialogue-related tasks spanning three
categories; InstrDialog++ stream, which includes all the tasks in InstrDialog
stream and 19 additional tasks selected from broad categories and domains.
Using the two long task streams, we implement various CL methods to study
forgetting and knowledge transfer under the setup of cit. We find that
directly fine-tuning an instruction-tuned model sequentially yields
competitive performance with existing CL methods. With further investigation,
we find that rich natural language instructions enable knowledge transfer and
reduce forgetting, which is barely fully leveraged by current CL methods. We
conduct comprehensive experiments to explore what effects the learning of cit.
We hope our citB benchmark will serve as a helpful starting point and
encourage substantial progress and future work by the community in this
practical setting. To summarize, our main contributions are:

  * •

We formulate the problem of cit and establish a benchmark suite consisting of
learning and evaluation protocols.

  * •

We curate two long task streams of various types based on the SuperNI dataset
to study different setups of cit.

  * •

We implement various CL methods of different categories, conduct extensive
experiments and ablation studies to analyze the lack of current practices, and
propose a future direction.

##  2 Related Work

#### Instruction Tuning.

Much effort has been made recently to use natural language instructions to
solve multiple tasks concurrently or to align with human preferences (Touvron
et al., 2023; Zhou et al., 2023; OpenAI, 2023). Unlike simple and short
prompts (Liu et al., 2021), natural language instructions (Fig.2) can be more
comprehensive, including components such as task definition, in-context
examples (Brown et al., 2020), and explanations. Through IT, PLMs learn to
complete tasks by following instructions, which enables them to solve new
tasks by following instructions without learning (i.e., generalization
ability). Ideally, we expect the instruction-tuned model to understand any
given task instruction so that an end user can directly leverage the model to
solve the task without the need to annotate a large dataset and train it.
Unfortunately, despite the instruction-tuned models such as FLAN (Wei et al.,
2022; Longpre et al., 2023), T0 (Sanh et al., 2022), and Tk-Instruct (Wang et
al., 2022) showing strong generalization performance to their evaluation
tasks, there is still a sizeable gap compared with supervised training, which
limits the usage of the models. From a practical point of view, a desirable
instruction-tuned model should be able to extend its ability by continually
learning those under-performed tasks or any new task, while not forgetting the
old ones.

[width=]dialogue_example_final_cropped.pdf

Figure 2: An example of natural language instruction that consists of a
descriptive task definition, one positive and one negative in-context example
with explanation (Wang et al., 2022). Given a task instruction and the input
of a test instance, a model needs to produce the desired output.

#### Continual Learning.

In contrast to multi-task learning, continually fine-tuning a model on tasks
might lead to catastrophic forgetting (McCloskey and Cohen, 1989), where the
model forgets previously acquired knowledge after learning new tasks. In CL
literature, approaches to overcoming catastrophic forgetting can be grouped
into three categories (Biesialska et al., 2020; Ke and Liu, 2023).
Regularization-based methods use an additional loss to prevent important
parameters of previous tasks from being updated (Kirkpatrick et al., 2017; De
Lange et al., 2019). Replay-based methods store and replay a small subset of
training data from previous tasks to prevent forgetting (Rebuffi et al., 2017;
Scialom et al., 2022); Architecture-based methods introduce task-specific
components for new tasks and isolate parameters of old tasks (Madotto et al.,
2021; Zhu et al., 2022). However, the effectiveness of these CL methods for
cit remains unknown. cit differs from traditional CL in heavily relying on
comprehensive instructions. Can previous methods fully leverage the rich
instructions to avoid forgetting while facilitating knowledge transfer between
tasks? Moreover, many proposed CL methods target tasks of specific types
(e.g., text classification, relation extraction) (Huang et al., 2021; Qin and
Joty, 2022; Zhu et al., 2022), while cit can learn broad tasks of different
categories because of the natural language instructions222All tasks can be
filled into a natural language instruction template and transformed into a
text-to-text format (Wei et al., 2022).. Yin et al. (2022); Scialom et al.
(2022); Mok et al. (2023) study a similar problem using IT in the CL setting,
but no benchmarks are built for different categories of CL methods. To tackle
cit, it is essential to establish an unified benchmark to compare existing
approaches and promote the development of this field. However, to our best
knowledge, cit is still immature and no public benchmark is available.

##  3 Preliminaries

#### Instruction Tuning (IT).

Following previous studies (Wei et al., 2022; Sanh et al., 2022; Wang et al.,
2022), each task $t\in\mathcal{T}$ consists of its natural language
instruction $I^{t}$ and a set of $N$ input-output instances
$\mathcal{D}^{t}=\left\\{\left(x_{i}^{t},y_{i}^{t}\right)\in\mathcal{X}^{t}\times\mathcal{Y}^{t}\right\\}_{i=1}^{N}$,
which can be split into the training $\mathcal{D}^{t}_{\text{train}}$,
validation $\mathcal{D}^{t}_{\text{dev}}$ and test sets
$\mathcal{D}^{t}_{\text{test}}$. Each instance is filled into an instruction
template such that different tasks can be transformed into a unified text-to-
text format (Fig.2). IT aims to learn a model
$f:I^{t}\times\mathcal{X}^{t}\rightarrow\mathcal{Y}^{t}$ that can predict the
output $y_{i}^{t}$ given the task instruction $I^{t}$ and an input
$x_{i}^{t}$. In general, the model is first trained on a mixture of tasks (
$\mathcal{T}_{\text{seen}}$) and then evaluated for its zero-shot
generalization ability on held-out tasks ($\mathcal{T}_{\text{unseen}}$),
where $\mathcal{T}_{\text{seen}}\cap\mathcal{T}_{\text{unseen}}=\emptyset$.
The model is expected to learn to follow instructions via the training tasks
and then solve new tasks with only the help of task instructions.

##  4 Continual Instruction Tuning Benchmark

In this section, we first formalize the cit problem (Fig.1). Then, we present
the learning and evaluation protocol of our framework citB. Lastly, we
describe the data for creating the benchmark.

###  4.1 Continual Instruction Tuning

In contrast to static IT, which only learns a fixed set of tasks
($\mathcal{T}_{\text{seen}}$), the model should be able to keep learning new
tasks without catastrophically forgetting previously learned knowledge and
facilitate knowledge transfer if possible. Let us expand the definition such
that we have a set of $T$ tasks
$\mathcal{T}_{\text{seq}}=\\{t_{1},\cdots,t_{T}\\}$ that arrives sequentially.
Note that the tasks in the stream can be any type and are not restricted to
specific categories or domains. Similarly, each task
$t_{j}\in\mathcal{T}_{\text{seq}}$ has a natural language instruction
$I^{t_{j}}$, training $\mathcal{D}^{t_{j}}_{\text{train}}$, validation
$\mathcal{D}^{t_{j}}_{\text{dev}}$ and test sets
$\mathcal{D}^{t_{j}}_{\text{test}}$. Likewise traditional CL, the goal of cit
is to learn a single model $f$ from $\mathcal{T}_{\text{seq}}$ sequentially.

####  cit vs. Traditional CL.

While sharing similar desiderata with traditional CL, cit differs in that: (1)
it pays more attention to effectively leveraging the rich natural language
instructions to prevent catastrophic forgetting and encourage knowledge
transfer; (2) because of the multi-task nature of instructions, all tasks can
be formatted in the unified text-to-text format, therefore cit can learn any
task or domain instead of a few specific tasks or domains; (3) after learning
a few tasks, the model should have learned how to follow instructions to
complete tasks, therefore, we expect fewer training instances required and
higher knowledge transfer for future tasks.

###  4.2 Learning Protocol of CIT Benchmark

A non-instruction-tuned model (e.g., T5; Raffel et al. 2020) may struggle to
understand instructions if trained only on a task sequentially. It is also
against our motivation to extend a model’s ability that is already
instruction-tuned. Therefore, we separate the learning process into two
stages.

#### Stage 1: Initial Multi-task Fine-tuning.

To teach a model a better understanding of task instructions, we first fine-
tune the model on instruction data. Suppose we have another group of $M$ tasks
$\mathcal{T}_{\text{init}}=\\{t_{1},\cdots,t_{M}\\}$ that also equips with
natural language instructions, where
$\mathcal{T}_{\text{init}}\cap\mathcal{T}_{\text{seq}}=\emptyset$. We fine-
tune a base pre-trained model on the training set (i.e.,
$\mathcal{D}^{\text{init}}_{\text{train}}=\bigcup_{i=1}^{M}\mathcal{D}^{t_{i}}_{\text{train}}$)
of the mixed $M$ tasks to get an instruction-tuned model, denoted as
$f_{\text{init}}$. After training, most of the training data
$\mathcal{D}^{\text{init}}_{\text{train}}$ is unavailable for subsequent
sequential learning, but a memory $\mathcal{M}_{\text{init}}$
($\mathcal{M}_{\text{init}}\ll|\mathcal{D}^{\text{init}}_{\text{train}}|$)
that stores a small portion of training instances is accessible. We use this
model as the starting point to conduct the subsequent learning.

#### Stage 2: Sequential Single Task Fine-tuning.

To keep extending knowledge of the instruction-tuned $f_{\text{init}}$, we
fine-tune it on the training set $\mathcal{D}^{t_{j}}_{\text{train}}$ of each
task $t_{j}$ in the stream $\mathcal{T}_{\text{seq}}$. Similarly, when
learning the task $t_{j}$, the training data of previous tasks in the stream
(i.e.,
$\mathcal{D}^{\text{seq}}_{\text{train}}=\bigcup_{i=1}^{j-1}\mathcal{D}^{t_{i}}_{\text{train}}$,
$i<j<T$) is unavailable, but a small memory $\mathcal{M}_{\text{seq}}$ can be
used for training.

###  4.3 Evaluation Protocol of CIT Benchmark

#### Evaluation Process.

After learning each task $t_{j}$ ($1<j<T$) in stream
$\mathcal{T_{\text{seq}}}$, we consider three datasets to measure the model’s
performance. (1) Similar to the standard CL, we evaluate the model on the test
sets of all previously learned tasks in the stream, the test set of the
current task, and the test set of the next task, denoted as
$\mathcal{D}^{\text{seq}}_{\text{test}}$. This helps us measure whether the
model forgets previous knowledge and whether it is helpful to learn future
tasks; (2) We evaluate the model on the test sets of the $M$ tasks that are
used in stage 1 to teach the model how to follow instructions, denoted as
$\mathcal{D}^{\text{init}}_{\text{test}}=\bigcup_{i=1}^{M}\mathcal{D}^{t_{i}}_{\text{test}}$.
This is where different from the conventional CL. In CL, previous works only
evaluate downstream tasks in the stream but not the tasks during the pre-
training phase because such data is generally not accessible to end-users (Ke
et al., 2022). (3) Since multi-task instruction-tuned models have shown strong
zero-shot generalization to unseen tasks (Wang et al., 2022), our initial
model trained in stage 1 might also have zero-shot generalization to some
unseen tasks $\mathcal{T}_{\text{unseen}}$, where
$\mathcal{T}_{\text{init}}\cap\mathcal{T}_{\text{seq}}\cap\mathcal{T}_{\text{unseen}}=\emptyset$.
Let $\mathcal{D}^{\text{unseen}}_{\text{test}}$ be the test sets of all tasks
in $\mathcal{T}_{\text{unseen}}$. We evaluate the model on
$\mathcal{D}^{\text{unseen}}_{\text{test}}$ if it is available. To sum up,
once a new task is learned, the model will be evaluated on:

$D=\mathcal{D}^{\text{seq}}_{\text{test}}+\mathcal{D}^{\text{init}}_{\text{test}}+\mathcal{D}^{\text{unseen}}_{\text{test}}$
(1)

In cit, it is more critical for the instruction-tuned model to maintain its
existing abilities than learn new ones because it can solve multiple tasks by
following instructions. Otherwise, if it forgets many tasks, there is no point
in using such a model than a task-specific one. Therefore, it is essential to
evaluate on $\mathcal{D}^{\text{init}}_{\text{test}}$ and
$\mathcal{D}^{\text{unseen}}_{\text{test}}$.

#### Evaluation Metrics.

Due to the diversity of the tasks in cit and the open-ended generation nature
of the text-to-text format, we follow Wang et al. (2022) to use ROUGE-L (Lin,
2004) to measure the aggregated performance of each task. They have shown that
ROUGE-L generally works well for both generation and classification tasks.

Following Lopez-Paz and Ranzato (2017) and Biesialska et al. (2020), we also
use CL-related metrics to measure the learning procedure. Let $a_{j,i}$ be the
ROUGE-L score of the model on the test set of task $t_{i}$ right after
training on task $t_{j}$, we define the following:

Average ROUGE-L (AR), which measures the average performance of the model on
all tasks after the final task $t_{T}$ is learned:

$\textbf{\text{AR}}_{T}=\frac{1}{T}\sum_{i=1}^{T}a_{T,i}$ (2)

We use Final ROUGE-L (FR) to measure the performance of the model on
$\mathcal{D}^{\text{init}}_{\text{test}}$ and
$\mathcal{D}^{\text{unseen}}_{\text{test}}$, respectively, after the final
task $t_{T}$ is learned.

Forward Transfer (FWT), which measures how much the model can help to learn
the new task. FWT also tests the model’s zero-shot generalization to new
tasks:

$\textbf{\text{FWT}}_{T}=\frac{1}{T-1}\sum_{i=2}^{T}a_{i-1,i}$ (3)

Backward Transfer (BWT), which measures the impact that continually learning
on subsequent tasks has on previous tasks:

$\textbf{\text{BWT}}_{T}=\frac{1}{T-1}\sum_{i=1}^{T-1}\left(a_{T,i}-a_{i,i}\right)$
(4)

Notably, positive BWT indicates that subsequent tasks can improve the
performance of previous tasks, while negative value implies knowledge
forgetting.

###  4.4 Data Curation

In this work, we adopt the recently proposed SuperNI (Wang et al., 2022)
dataset to establish the benchmark. SuperNI consists of more than 1,600 NLP
tasks, spanning a diverse variety of 76 broad task types, such as language
generation, classification, question answering, and translation. Moreover,
each task is equipped with an instruction and a set of instances, and all the
instances can be transformed into the text-to-text format. Therefore, the
dataset is suitable for studying cit. The official training set of
SuperNI333https://github.com/allenai/natural-instructions consists of 756
English tasks spanning 60 broad NLP categories, while 119 tasks from 12
categories are used for zero-shot evaluation. We keep the official 119
evaluation tasks untouched and create two cit task streams from the 756
training tasks.

#### InstrDialog Stream.

Dialogue is an important field to study continual learning because new tasks,
domains or intents are continuously emerging (Madotto et al., 2021). To
investigate how a model learns new dialogue data under the setup of cit, we
carefully curate all dialogue-related tasks from the training set of SuperNI
to form the cit task stream. Specifically, we use 4 tasks from dialogue state
tracking, 11 tasks from dialogue generation, and 4 tasks from intent
identification, resulting in a total of 19 dialogue tasks, i.e.,
$|\mathcal{T_{\text{seq}}}|=19$. We remove tasks that are excluded by the
official task splits444https://github.com/allenai/natural-
instructions/tree/master/splits.

#### InstrDialog++ Stream.

Because of the multi-task nature of instructions, an instruction-tuned model
can learn any new task with different types (section 4.1). To study how
different types of tasks and how a long-task curriculum affects cit, we first
include all 19 dialogue tasks from the InstrDialog stream, then we manually
select the other 19 tasks from the remaining training task set. We
intentionally select tasks from broad categories, including sentence ordering,
style transfer, toxic language detection, and others. In total, we have 38
tasks of 18 categories (3 categories from InstrDialog and 15 categories from
the new 19 tasks), i.e., $|\mathcal{T_{\text{seq}}}|=38$.

The remaining training tasks can be used for stage 1 initial multi-task fine-
tuning (section 4.2). In summary, the number of initial fine-tuning tasks
available is $M=|\mathcal{T}_{\text{init}}|=718$, and we use the official 119
test task sets as $\mathcal{T}_{\text{unseen}}$ to evaluate whether the
performance deteriorates for unseen tasks after learning new tasks. For all
tasks, we fill instances in a natural language instruction template and
transform them into a unified text-to-text format (section 3). Unless
otherwise specified, we use the instruction template consisting of the task
definition and two positive examples for all tasks because it generally yields
the best performance (Wang et al., 2022). See an example of natural language
instructions in Fig.2, and the selected tasks in Table A. We study the effect
of the instruction template in section 6.3.

1.0! l|ccc|cc|ccc InstrDialog $\mathcal{T}_{\text{init}}$
$\mathcal{T}_{\text{unseen}}$
Method AR FWT BWT FR FR Mem. +P (Tun) Time
FT-no-init 29.62.1 8.00.2 $-$10.82.3 17.30.5† 16.51.3† 0 0 (1) 0.50.02

AdapterCL 8.10.1 9.40.7 $-$21.90.9 \- - 0 T*0.02 (0.02) 0.40.03

Init 22.5† \- - 43.5 36.5† \- - -

FT-init 35.70.2 18.50.7 $-$4.60.2 38.60.3 32.30.6† 0 0 (1) 0.60.01

L2 35.60.1 17.50.5 $-$3.81.2 39.40.4 34.91.2† 0 1 (1) 0.60.1

EWC 34.50.6 16.80.4 $-$6.81.5 37.00.1 32.50.5† 0 2 (1) 1.10.2

AGEM (10) 33.20.4 19.10.1 $-$7.31.0 38.61.1 32.40.0† (T+M)*10 0 (1) 1.10.1

AGEM (50) 34.90.9 18.11.0 $-$6.00.9 37.70.1 32.61.0† (T+M)*50 0 (1) 1.30.1

Replay (10) 38.40.7 23.70.0 $-$1.30.5 42.70.7 32.40.4† (T+M)*10 0 (1) 1.40.04

Replay (50) 40.40.0 22.90.1 1.61.2 47.10.5 31.81.0† (T+M)*50 0 (1) 3.20.5

Multi 42.10.6 \- - 44.71.3 32.80.9† 0 0 (1) 1.10.2

Table 1: Performance of different methods on the InstrDialog stream. Means and
standard deviations are reported. $\dagger$ means zero-shot performance.
"Mem." means the number of instances stored in the memory for each task; $T$
is the total number of tasks in the stream and $M$ is the number of tasks used
for initial training. "+P" means the percentage of additional parameters added
for each task, measured by the total parameters of the base model; "Tun" is
the portion of tunable parameters during training. "Time" is the average hours
for each method to complete the task stream. Best numbers are in bold.

##  5 Experiments

Using our citB benchmark, we conduct experiments on various popular CL methods
of different kinds. We describe our experiment setups and compared methods in
this section.

###  5.1 Setup

#### Model.

We use the LM-adapted version of T5-small (Raffel et al., 2020), which is
further trained with a language modeling objective. We initialize a T5 model
from HuggingFace555https://huggingface.co/google/t5-small-lm-adapt. Since it
is costly to fine-tune on all 718 tasks, we randomly select 100 tasks from
$\mathcal{T}_{\text{init}}$ and fine-tune T5 to obtain an instruction-tuned
model $f_{\text{init}}$ (section 4.2), which has learned to understand some
instructions and can act as a good starting point to conduct subsequent
learning. Note that the 100 randomly selected training tasks do not overlap
with InstrDialog and InstrDialog++, but their task categories might overlap
with the categories in InstrDialog++.

#### Train/Dev/Test Splits.

Since the number of instances in each task is imbalanced and a large number of
training instances do not help generalization in IT (Wang et al., 2022), we
follow Wang et al. (2022) use a fixed size of 500/50/100 instances per task as
the train/dev/test set for the InstrDialog stream. For InstrDialog++, since it
has a longer task sequence, to save computational cost, we use 100/50/100
instances per task instead. For $\mathcal{T}_{\text{init}}$, we use 100/50/100
instances per task and 100 instances per task for
$\mathcal{T}_{\text{unseen}}$. We study the effect of different numbers of
training instances in section 6.3.

###  5.2 Baselines and Compared Methods

We implement commonly used CL methods from three categories (Biesialska et
al., 2020) to benchmark cit.

Regularization-based methods rely on a fixed model capacity with an additional
loss term to consolidate previously gained knowledge while learning subsequent
tasks. We use L2 and EWC (Kirkpatrick et al., 2017), which uses a fisher
information matrix to reduce forgetting by regularizing the loss to penalize
the changes made to important parameters of previous tasks.

Replay-based methods store a small subset of training instances from previous
tasks in a memory. The data are replayed later to reduce forgetting. We adopt
Replay, which saves random instances from each task in a memory and then
jointly trains the model on new task data and the old data in the memory; AGEM
(Chaudhry et al., 2019), which adds constraint to prevent parameter update
from increasing the loss of each previous task. The loss of previous tasks is
calculated using the instances stored in the memory.

Architectural-based methods introduce task-specific parameters to the base
model to prevent subsequent tasks from interfering with previously learned
parameters. We adopt AdapterCL (Madotto et al., 2021), which freezes the pre-
trained model and trains a residual Adapter (Houlsby et al., 2019) for each
task independently.

Apart from the three categories of CL, we also implement instruction-based
baselines because all previous CL methods are not designed for cit. No prior
work had tried to fine-tune an instruction-tuned model sequentially without
any mechanism for preventing forgetting or encouraging knowledge transfer.
However, it is commonly considered a performance lower bound in CL literature.
To this end, we propose to continually fine-tune the initial instruction-tuned
model (section 5.1) on subsequent tasks, named as FT-init. As a direct
comparison, we initialize a new T5 model, which is not tuned on any
instruction data, and we continually fine-tune it (FT-no-init). In addition,
we also report the performance of the initial instruction-tuned model (Init),
which is the starting point before subsequent learning. Lastly, we jointly
fine-tune a T5 model using all the data, including the training data used in
stage 1 and the training data of all subsequent tasks in the stream (Multi).
This is often regarded as the performance upper bound in CL and does not have
catastrophic forgetting and knowledge transfer.

l|ccc|cc InstrDialog++ $\mathcal{T}_{\text{init}}$
$\mathcal{T}_{\text{unseen}}$
Method AR FWT BWT FR FR
FT-no-init 31.32.4 26.80.3 $-$5.21.4 28.90.6† 31.22.1†

AdapterCL 20.70.2 14.30.9 $-$7.31.1 \- -

Init 30.5† \- - 43.5 36.5†

FT-init 34.80.3 29.80.5 $-$2.80.6 40.80.2 35.90.5†

L2 36.10.2 27.90.3 $-$4.01.2 41.10.4 34.91.1†

EWC 38.60.4 28.60.5 $-$4.01.8 41.30.3 34.00.7†

AGEM (10) 39.30.6 28.90.2 $-$3.81.1 40.30.9 34.10.3†

Replay (10) 43.10.7 29.00.4 $-$3.60.8 44.00.5 30.10.6†

Multi 44.90.5 \- - 44.60.9 33.90.5

Table 2: Performance of different methods on the InstrDialog++ stream.
$\dagger$ means zero-shot performance.

###  5.3 Implementation Details

For both InstrDialog stream and InstrDialog++ stream, we conduct continual
instruction tuning using the same initial instruction-tuned model for all
methods except FT-no-init, AdapterCL, and Multi. For these methods, we
initialize a new T5 model. For AGEM and Replay, we experiment on a memory size
of 10 and 50, i.e., we set the memory size $\mathcal{M}_{\text{init}}$ to 10
and 50, same as $\mathcal{M}_{\text{seq}}$. We jointly train the data in
$\mathcal{M}_{\text{init}}$, $\mathcal{M}_{\text{seq}}$, and the new task data
for these two methods. For AdapterCL, we use a bottleneck of 100. Due to
limited computing resource, we randomly permute the task streams and run all
experiments using three random seeds. We refer this as task order 1. We study
the effect of task orders in section 6.3. The selected tasks for task stream
InstrDialog and InstrDialog++ are listed in Table A. The task orders are
listed in Table 6. More details are in Appendix A.

##  6 Results and Analysis

In this section, we report the performance of various baselines discussed in
section 5.2 on our benchmark.

###  6.1 Results on InstrDialog Stream

Table 4.4 shows each method’s overall performance and resource requirement
after continually learning the InstrDialog stream. We have the following
observations:

First, all methods except AdapterCL have improved AR, compared to the zero-
shot performance (22.5) of the starting point model Init. This shows cit can
extend a model’s knowledge. In contrast, although AdapterCL is parameter-
efficient and does not rely on memory, it performs even worse than Init. We
conjecture that AdapterCL fails to learn instructions effectively because it
is initialized from a non-instruction-tuned model (T5) and the few tunable
parameters restrict it from learning complex instructions.

Second, among all baselines, Replay generally have the best performance. All
methods except Replay (50) have negative BWT, meaning that they all suffer
from catastrophic forgetting. Furthermore, forgetting on
$\mathcal{T_{\text{init}}}$ and $\mathcal{T_{\text{unseen}}}$ is even worse,
which demonstrates that the ability of the initial instruction-tuned model
Init has deteriorated after learning new dialogue tasks. We also find storing
more examples in memory improves Replay but does not significantly help AGEM.
It might be because the constraints added to the loss are roughly the same, no
matter how many instances are stored. Despite used additional parameters,
regularization-based L2 and EWC perform similar to other baselines. Multi
overall performs well, with the highest AR and improved FR on
$\mathcal{T_{\text{init}}}$, however, it also forgets tasks in
$\mathcal{T_{\text{unseen}}}$. Replay (50) has a higher FR on
$\mathcal{T_{\text{init}}}$ than Multi because the 5,000 instances stored in
$\mathcal{M}_{\text{init}}$ are jointly trained multiple times when learning
subsequent tasks (section 5.3), leading to better data fitting.

Third, FT-init performs surprisingly well on all metrics and is competitive to
L2, EWC, and AGEM. This finding contradicts the common sense in CL that simply
fine-tuning a model sequentially would lead to catastrophic forgetting because
all parameters can be freely updated in learning new tasks (Kirkpatrick et
al., 2017). Even for FT-no-init, which is not tuned on any instruction data,
shows increased AR after learning the 19 dialogue tasks. This raises a
question: do various CL methods truly mitigate forgetting and promote
knowledge transfer in cit? We hypothesize that the rich natural language
instructions lead to the remarkable performance of the baselines (section
6.3).

###  6.2 Results on InstrDialog++ Stream

The performance of all methods after learning the InstrDialog++ steam is shown
in Table 5.2. We observe most of the same findings as in section 6.1, except
that:

First, Init has a higher (30.5 vs. 22.5) zero-shot performance on this long
stream than on InstrDialog, as in Table 4.4. We analyze that the categories of
the selected 100 training tasks (section 5.1) overlap with the categories in
the stream, which enables more knowledge transfer of tasks between the same
category because of the similar natural language instructions. For example,
both sets have tasks from sentiment analysis and toxic language detection. In
contrast, Init did not learn dialogue tasks, thus showing lower generalization
on InstrDialog.

Second, we can see improved performance for almost all methods compared to
Table 4.4, especially on $\mathcal{T_{\text{init}}}$ and
$\mathcal{T_{\text{unseen}}}$. For FT-init and FT-no-init, the improvements of
FWT and BWT are particularly significant, reaching the best among all CL
methods.

Combining the results on the two streams from Table 4.4 and 5.2, we find that
catastrophic forgetting exists in cit. However, learning a longer task stream
and diverse tasks of different types leads to better knowledge transfer and
lower forgetting.

!  FT-init FT-no-init Instr. AR FWT BWT AR FWT BWT id 13.9 10.0 -30.3 11.5 8.0
-27.7 def 38.0 21.7 -6.9 33.8 16.1 -7.3 def+2p 34.8 29.8 -2.8 31.3 26.8 -5.2
def+2p +2n 40.4 28.1 -7.1 33.8 22.3 -7.3

Table 3: Effect of instruction templates on InstrDialog++. "id": only use the short task id; "def": only use descriptive task definitions; "def+2p": use task definition and two positive examples; "def+2p+2n": use additional two negative examples.  | FT-init | FT-no-init
---|---|---
Instr. | $\mathcal{T}_{\text{init}}$ | $\mathcal{T}_{\text{unseen}}$ | $\mathcal{T}_{\text{init}}$ | $\mathcal{T}_{\text{unseen}}$
id | 3.6 | 2.8 | 1.0 | 1.7
def | 25.8 | 23.6 | 20.3 | 22.4
def+2p | 40.8 | 35.9 | 28.9 | 31.2
def+2p+2n | 37.7 | 35.7 | 27.2 | 33.1
Table 4: Effect of instruction templates on $\mathcal{T_{\text{init}}}$ and
$\mathcal{T_{\text{unseen}}}$.

[width=]instr_learning_trend.pdf

Figure 3: AR of each method during learning the InstrDialog stream (task order
1).

###  6.3 Ablation Studies

In this section, we investigate the reason why instruction-based baselines
(FT-init and FT-no-init) perform as well as or even better than conventional
CL methods (section 6.1 & section 6.2). We also explore different aspects that
might affect cit.

#### Rich instructions enable knowledge transfer and reduce forgetting in cit.

We use the same setup as in Table 5.2, except for using different instruction
templates. Results in Table 3 confirms our hypothesis that the remarkable
performance of FT-init and FT-no-init comes from the rich natural language
instructions. When only using the task Id (e.g.,
task565_circa_answer_generation) as instruction without descriptive task
definition or in-context examples, simply fine-tuning the model sequentially
yields low AR, FWT, and BWT, which aligns with the findings in CL literature.
Additionally, providing in-context examples (2p or 2p+2n) generally improves
performance. However, although task performance (AR) is improved with two
additional negative examples, we witness decreased knowledge transfer (FWT and
BWT). For the model that is not fine-tuned on any instruction data (FT-no-
init), we find it worse than FT-init, showing the benefits of the initial
multi-task training on instruction data.

Similar observations are found on $\mathcal{T_{\text{init}}}$ and
$\mathcal{T_{\text{unseen}}}$ in Table 4, where the model catastrophically
forgets its initial abilities after learning a long stream of tasks. Providing
descriptive task definitions significantly boosts task performance as well as
facilitates knowledge transfer. Moreover, it also maintains the model’s
generalization ability on unseen tasks. Combining the results in Table 4.4,
5.2, 3, and 4, we find that those conventional CL methods do not fully
leverage the instructions to reduce forgetting and facilitate knowledge
transfer while learning continuously because the naive FT-init and FT-no-init
can also achieve the same. This calls for novel CL methods designed for cit.

[width=]instr_learning_trend_order2.pdf

Figure 4: AR of each method during learning the InstrDialog stream (task order
2).

[width=]instr_learning_trend_order3.pdf

Figure 5: AR of each method during learning the InstrDialog stream (task order
3).

#### Task types and learning order matter to cit.

To explore how task types and orders in a stream affects cit, we randomly
permute the InstrDialog stream to get two new task orders and conduct the same
learning as Table 4.4. We present the intermediate learning trends of all 19
tasks in Fig.3, Fig.4 and Fig.5. One can see from the plots that all baselines
are highly affected by task orders, fluctuating dramatically when different
tasks are learned first. We argue that it is because the task difficulties and
similarities vary a lot. Learned tasks transfer knowledge through the
instructions to new tasks of the same type, therefore facilitating its
learning. For example, the last task in order 1 (Fig.3) is a type of dialogue
generation, which is the dominant task type in the stream (11/19, section
4.4), therefore all baselines are improved. However, all baselines reach below
Multi after learning all 19 tasks, demonstrating knowledge forgetting will
eventually appear if learning longer enough tasks.

[width=]ablation_train_num_fwt.pdf

Figure 6: Effect of training instances per task on FWT.

[width=]ablation_train_num_bwt.pdf

Figure 7: Effect of training instances per task on BWT.

#### A large number of training instances do not help knowledge transfer.

We vary the number of instances per task used for learning the InstrDialog
stream from $[10,20,100,200,500]$. As shown in Fig.6 and Fig.7, FWT and BWT
gradually decrease when the number of training instances is scaled to large
values. It aligns with the findings by Wang et al. (2022) in standard IT,
where large number of instances do not help generalization to unseen tasks, we
also find it is true in cit. Additionally, we find instruction-tuned models
(FT-init) have better generalization to new tasks (FWT) than the model not
fine-tuned on any instruction data (FT-no-init). This shows that, after
learning a few tasks, the model have learned how to follow instructions to
complete tasks, thus fewer training instances are required for new tasks.

##  7 Conclusion

In this work, we establish a benchmark for continual instruction tuning, with
two 19 and 38 long task streams to be learned sequentially. We implement and
compare various continual learning methods of different types using the
benchmark to study their effectiveness under this new domain. We conduct
extensive ablation studies to analyze the lack of current practices, and
propose a future direction.

## Limitations

We identify our limitations as follows. First, due to limited resources, all
experiments in this work use T5-small (LM-adapted) as the backbone, which
might not entirely reflect continual instruction tuning in general. As Wang et
al. (2022) points out, there is a sizable gap between the smaller models and
the 11B or 3B models in generalizing to new tasks. Second, when creating the
two cit task streams, we only use English tasks from the SuperNI dataset (Wang
et al., 2022). In future, it can be extended to multilingual task streams to
study cross-language continual instruction tuning. Third, we follow the
SuperNI dataset to use ROUGE-L as an aggregated metric to evaluate all tasks.
Although it acts as a good proxy for the model’s overall performance, it might
not serve as an effective measurement for some specific tasks. Fourth, while
we selected diverse tasks to form the InstrDialog and InstrDialog++ task
streams, we did not analyse the characteristics of these tasks (Kim et al.,
2023). In future, we consider to select better source tasks and study how
source tasks affect cit.

## Acknowledgements

This work is supported by TPG Telecom. We would like to thank anonymous
reviewers for their valuable comments.

## References

  * Biesialska et al. (2020) Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020.  Continual lifelong learning in natural language processing: A survey.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6523–6541, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.
  * Chaudhry et al. (2019) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019.  Efficient lifelong learning with a-GEM.  In _International Conference on Learning Representations_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * De Lange et al. (2019) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2019.  Continual learning: A comparative study on how to defy forgetting in classification tasks.  _arXiv preprint arXiv:1909.08383_ , 2(6):2.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Huang et al. (2021) Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021.  Continual learning for text classification with information disentanglement based regularization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2736–2746, Online. Association for Computational Linguistics.
  * Ke et al. (2022) Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, and Bing Liu. 2022.  Continual training of language models for few-shot learning.  _arXiv preprint arXiv:2210.05549_.
  * Ke and Liu (2023) Zixuan Ke and Bing Liu. 2023.  Continual learning of natural language processing tasks: A survey.
  * Ke et al. (2021) Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. 2021.  Achieving forgetting prevention and knowledge transfer in continual learning.  In _Advances in Neural Information Processing Systems_.
  * Kim et al. (2023) Joongwon Kim, Akari Asai, Gabriel Ilharco, and Hannaneh Hajishirzi. 2023.  Taskweb: Selecting better source tasks for multi-task nlp.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114(13):3521–3526.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _CoRR_ , abs/2107.13586.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023.  The flan collection: Designing data and methods for effective instruction tuning.
  * Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc' Aurelio Ranzato. 2017.  Gradient episodic memory for continual learning.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023.  Augmented large language models with parametric knowledge guiding.
  * Madotto et al. (2021) Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Pascale Fung, and Zhiguang Wang. 2021.  Continual learning in task-oriented dialogue systems.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7452–7467, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pages 109–165. Elsevier.
  * Mok et al. (2023) Jisoo Mok, Jaeyoung Do, Sungjin Lee, Tara Taghavi, Seunghak Yu, and Sungroh Yoon. 2023.  Large-scale lifelong learning of in-context instructions and how to tackle it.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12573–12589, Toronto, Canada. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Qin and Joty (2022) Chengwei Qin and Shafiq Joty. 2022.  Continual few-shot relation learning via embedding space regularization and data augmentation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2776–2789, Dublin, Ireland. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Scialom et al. (2022) Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022.  Fine-tuned language models are continual learners.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6107–6122, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Yin et al. (2022) Wenpeng Yin, Jia Li, and Caiming Xiong. 2022.  ConTinTin: Continual learning from task instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3062–3072, Dublin, Ireland. Association for Computational Linguistics.
  * Zhang et al. (2023) Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad, and Jun Wang. 2023.  How do large language models capture the ever-changing world knowledge? a review of recent advances.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023.  Lima: Less is more for alignment.
  * Zhu et al. (2022) Qi Zhu, Bing Li, Fei Mi, Xiaoyan Zhu, and Minlie Huang. 2022.  Continual prompt tuning for dialog state tracking.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1124–1137, Dublin, Ireland. Association for Computational Linguistics.

##  Appendix A Implementation Details

For both InstrDialog stream and InstrDialog++ stream, we conduct continual
instruction tuning using the same initial instruction-tuned model for all
methods except FT-no-init, AdapterCL, and Multi. For these methods, we
initialize a new T5 model. Since AdapterCL needs to train a task-specific
adapter for each task, it is too costly to train the 100 adapters for it,
therefore we initialize it from a T5 model. For AdapterCL, we use a bottleneck
of 100.

The initial instruction-tuned model (section 5.1) is trained on 100 tasks with
100 instances per task (in total 10,000). We use a maximum of 15 epochs with
learning rate 1e-05. We perform checkpoint selection using the development
set.

For AGEM and Replay, we experiment on a memory size of 10 and 50, i.e., we set
the memory size $\mathcal{M}_{\text{init}}$ to 10 and 50, same as
$\mathcal{M}_{\text{seq}}$. We jointly train the data in
$\mathcal{M}_{\text{init}}$, $\mathcal{M}_{\text{seq}}$, and the new task data
for these two methods.

For L2 and EWC, we tune the regularization term from $[0.001,0.01,0.1]$, and
use 0.01. For AdapterCL, we use learning rate of 1e-3. For other methods, we
use 1e-5. For the InstrDialog stream, we use a batch size of 4 for EWC and 8
for all other methods; for the InstrDialog++ stream, we use a batch size of
16. For all experiments except AdapterCL, we train a maximum epoch of 15 and
perform early stopping with 3 patience to avoid overfitting; for AdapterCL, we
use a larger epoch of 20 with patience of 5.

The selected tasks for task stream InstrDialog and InstrDialog++ are listed in
Table A. The task orders are listed in Table 6. All experiments are done in an
RTX3090 Ti with 24GB VRAM.

cccc

Task Category Number Task Name Domain

1-1Dialogue Generation 1 task565_circa_answer_generation Dialogue
2 task574_air_dialogue_sentence_generation Dialogue
3 task576_curiosity_dialogs_answer_generation Dialogue, Commonsense
4 task611_mutual_multi_turn_dialogue Dialogue
5 task639_multi_woz_user_utterance_generation Dialogue
6 task1590_diplomacy_text_generation Dialogue, Game
7 task1600_smcalflow_sentence_generation Dialogue, Commonsense
8 task1603_smcalflow_sentence_generation Dialogue
9 task1714_convai3_sentence_generation Dialogue
10 task1729_personachat_generate_next Dialogue
11 task1730_personachat_choose_next Dialogue

1-1Intent Identification 12  task294_storycommonsense _motiv_text_generation
Story
13 task573_air_dialogue_classification Dialogue
14 task848_pubmedqa_classification Medicine
15 task1713_convai3_sentence_generation Dialogue

1-1Dialogue State Tracking 16 task766_craigslist_bargains_classification
Dialogue
17 task1384_deal_or_no_dialog_classification Dialogue, Commonsense
18 task1500_dstc3_classification Dialogue, Public Places
19 task1501_dstc3_answer_generation Dialogue, Public Places

1-1Style Transfer 20 task927_yelp_negative_to_positive_style_transfer Reviews

1-1Sentence Ordering 21 task1549_wiqa_answer_generation_missing_step Natural
Science

1-1Word Semantics 22 task459_matres_static_classification News

1-1Text Categorization 23 task379_agnews_topic_classification News

1-1Pos Tagging 24 task347_hybridqa_incorrect_answer_generation Wikipedia

1-1Fill in The Blank 25 task1360_numer_sense_multiple_choice_qa_generation
Commonsense

2-1Program Execution 26 task1151_swap_max_min Mathematics
27 task636_extract_and_sort_unique_alphabets_in_a_list Mathematics

1-1Question Generation 28 task301_record_question_generation News
29  task082_babi_t1_single_supporting _fact_question_generation Commonsense

1-1Misc. 30 task306_jeopardy_answer_generation_double Knowledge Base
31 task1427_country_region_in_world Countries

1-1Coherence Classification 32 task298_storycloze_correct_end_classification
Story

1-1Question Answering 33 task864_asdiv_singleop_question_answering Mathematics
34 task598_cuad_answer_generation Law

1-1Summarization 35 task1553_cnn_dailymail_summarization News

1-1Commonsense Classification 36 task1203_atomic_classification_xreact
Commonsense

1-1Wrong Candidate Generation 37  task967_ruletaker_incorrect_fact
_generation_based_on_given_paragraph Commonsense

1-1Toxic Language Detection 38 task1607_ethos_text_classification Social

Table 5:  List of tasks selected from SuperNI (Wang et al., 2022)

Task Order | Task’s IDs in order
---|---
Order1 | 848 611 565 1714 574 1590 1730 294 576 1600 1500 639 1729 1501 1713 766 1603 1384 573
Order2 | 848 1603 1714 565 611 1590 1600 639 294 1500 1384 1713 1501 576 574 1729 766 573 1730
Order3 | 1713 576 1384 294 573 611 1729 1600 574 1590 848 639 766 1501 565 1603 1730 1500 1714
Table 6: Task orders for three runs of the InstrDialog Stream.