# Multi-Stability in Cavity QED with Spin-Orbit Coupled Bose-Einstein
Condensate

Kashif Ammar Yasir<EMAIL_ADDRESS>Department of Physics, Zhejiang Normal
University, Jinhua 321004, China.  Gao Xianlong<EMAIL_ADDRESS>Department
of Physics, Zhejiang Normal University, Jinhua 321004, China.

###### Abstract

We investigate the occurrence of steady-state multi-stability in a cavity
system containing spin-orbit coupled Bose-Einstein condensate and driven by a
strong pump laser. The applied magnetic field splits the Bose-Einstein
condensate into pseudo-spin states, which then became momentum sensitive with
two counter propagating Raman lasers directly interacting with ultra-cold
atoms. After governing the steady-state dynamics for all associated
subsystems, we show the emergence of multi-stable behavior of cavity photon
number, which is unlike with previous investigation on cavity-atom systems.
However, this multi-stability can be tuned with associated system parameters.
Further, we illustrate the occurrence of mixed-stability behavior for atomic
population of the pseudo spin-$\uparrow$ amd spin-$\downarrow$ states, which
are appearing in so-called bi-unstable form. The collective behavior of these
atomic number states interestingly possesses a transitional interface among
the population of both spin states, which can be enhance and controlled by
spin-orbit coupling and Zeeman field effects. Furthermore, we illustrate the
emergence of secondary interface mediated by increasing the mechanical
dissipation rate of the pseudo-spin states. These interfaces could be cause by
the non-trivial behavior of synthetic spin state mediated by cavity. Our
findings are not only crucial for the subject of optical switching, but also
could provide foundation for future studies on mechanical aspect of synthetic
atomic states with cavity quantum electrodynamics.

Cavity Quatnum Electrodynamics, Spin-Orbit Coupling, Bose-Einstein Condensate,
Multi-stability, Optical Switching

##  I Introduction

Cavity quantum electrodynamics (QED) – strong photonic modes confined in a
highly reflective oppositely facing mirrors – is appeared to be the
fascinating tool to generate coherent photonic interaction Ref1 ; Ref2 ; Ref3
; Ref4 . The interaction of these confined and coherent optical modes with
other physical objects, like atoms and ultra-cold atoms, further enhances the
advantages of cavity QED Ref5 ; Ref6 ; Ref7 . This combination of coherent
atom-light interaction has the ability to even practically test the foundation
of quantum computation Ref8 ; Ref9 . Further, the inclusion of cold and ultra-
cold atoms in the cavity QED (specially in cavity-optomechanics where the
mechanical effects of light manipulates the motion of one end mirror(s)),
provided another direction of multi-species (hybrid) cavity QED Esslinger .
The hybrid cavity QED led to the study of Hamiltonian chaos Meystre2010 ,
chaos induced quantum mechanical localization kashif1 ; kashif2 , atom induced
optomechanical cooling peter2 , multi-particle quantum entanglement Vitali2012
; Vitali2014 ; Sete2014 ; Hofer  and high-fidelity state transfer YingPRL2012
; SinghPRL2012 . Quantum nonlinear optics with hybrid cavity QED further
results in the concept of multiple electromagnetically induced transparencies
agarwal2010 ; Stefan2010 ; Peng2014 ; SafaviNaeini2011 ; kashif11 ; kashif111
. The manipulation of synthetic inter-atomic states of ultra-cold atoms in
hybrid optomechanical systems lead to the further enhancements of hybrid
cavity QED operations kashif3 ; kashif33 ; kashif4 .

Meanwhile, the spin-orbit (SO-) coupling, a crucial relation between the
momentum of a quantum particle and its own spin Ref32 ; Ref33 ; Ref34 , is
proven to be the key tool to study spin-Hall effect Ref35 ; Ref36  and
topological insulators Ref37 ; Ref38 ; Ref39 ; Ref40 , and is the subject of
increasing investigations Ref41 ; Ref42 ; Ref43 ; Ref44 . The recent
demonstrations of SO-coupling Bose-Einstein condensate (BEC) in optical
cavities Ref45 ; Ref46 ; Ref47  and optomechanical systems, where the SO-
coupling has been used to cool the mechanical mirror Ref48 , have provided
basis to manipulate photonic interactions at pseudo-spin state level. The SO-
coupling mediated dressed states interactions in an optical cavity further led
to the discovery of topological states in the transmitting probe light
kashif55 . However, in spite of these investigation, still a study on steady-
state stability behavior is desirable because it will lead to the dynamical
aspect of such hybrid systems.

In this paper, we demonstrated the opportunity to achieved multi-stability in
the steady-state behavior of a cavity with SO-coupled BEC and driven by a
single-mode external pump laser. The externally interacting magnetic field
excites atomic pseudo-spin states and two counter propagating Raman beams
directly interact with confined atomic modes in cavity resulting in SO-
coupling between magnetically engineered pseudo-spin states. We compute
steady-state behavior for all associated degrees of freedom from quantum
Langevin equations driven from total Hamiltonian of the system. From governed
steady-state equations, show that the intra-cavity photon number not only
possesses multiple stable states (or multi-stable behavior) unlike previous
studies, but it is also tunable with other associated system parameters.
Further, we illustrate a unique and mixed stable and unstable behavior of
steady-state atomic population in both spin states because of the Raman
process mediated SO-coupling and Raman coupling. We also demonstrate the
dependence of both populations on each other by observing their collective
behavior versus external laser power. We found an intersection between these
atomic numbers that could be because of the phase transition occurring in
dress states with SO-coupling. This feature can be enhanced and controlled
with associated parameters, especially with atomic mechanical damping.

The manuscript is organized as: Section II contains the system modeling and
details of mathematical calculations. Section III illustrates the results of
steady-state behavior of intra-cavity photon number. While the section IV
contains the results of multi-stability occurring in the steady-state behavior
of pseudo-spin states. Finally, section V contains the conclusion of the
study.

##  II System Description and Hamiltonian

Figure 1: (a) The schematic diagram of a high-$Q$ Fabry-Pérot cavity
containing spin-orbit coupled Bose-Einstein condensate. A pump laser $\eta$
(with frequency $\omega_{E}$) drives the cavity exciting strong cavity mode,
which then generates coupling between cavity and trapped spin-orbit coupled
Bose-Einstein condensate. A bias magnetic field with strength $B_{0}$
interacts with the atomic states trapped along $\hat{y}$-axis causing Zeeman
splitting $\hbar\omega_{z}$ between atomic pseudo states. While two counter
propagating Raman lasers interact transversely along $\hat{x}$-axis with atoms
to generate spin-orbit coupling. (b) The energy level excitation diagram
illustrating the excitations induced by the bias magnetic field $B_{0}$ and
counter propagating Raman lasers.

The $N\approx 1.8\times 10^{5}$ ${}^{87}Rb$ bosonic particles are trapped in a
high-$Q$ Fabry-Pérot cavity having length $L\approx 12.5\times 10^{-3}$m, see
Fig.1(a) Refnew0 ; Refnew2 ; Refnew1 ; Ref49 . External pump laser with
frequency $\omega_{E}$ and strength
$|\eta|=\sqrt{P\times\kappa/\hbar\omega_{E}}$, where $P$ is the power, drives
the cavity generating a strong cavity mode $\omega_{c}\approx 1.9\times 2\pi
GHz$ having detuning $\Delta_{c}=\omega_{E}-\omega_{c}$ and decay rate
$\kappa$. The $10G$ of a bias magnetic field $B_{0}$ interacts with ultra-cold
atoms trapped inside cavity and produces a Zeeman splitting $\hbar\omega_{z}$
under condition $|\omega_{z}/\kappa|>>1$ Ref45 . After that, in order to
generate SO-coupling, two counter propagating Raman lasers ($\omega_{R}$ and
$\omega_{R}+\delta\omega_{R}$), having wavelength $\lambda=804.1nm$ and
detuning $\delta=1.6E_{R}$ Ref34 , transversely excite atoms along
$\hat{x}$-axis. It yields in the coupling between two internal pseudo-spin
states, within the spectrum of $5S_{1/2}$, of ultra-cold atoms
($|\uparrow\rangle=|F=2,m_{F}=0\rangle$ and
$|\downarrow\rangle=|F=2,m_{F}=-1\rangle$) at electronic manifold $F=2$, see
Fig. 1(b). We selected these particular parametric values from the recent and
available experimental studies Refnew0 ; Refnew2 ; Refnew1  in literature to
make our findings experimentally feasible. But our findings could be valid for
a range of selected parametric values crucially depending upon the coupling
strengths.

The total Hamiltonian of the considered system, with adiabatic and rotating-
wave approximation Ref34 ; Ref46 ; Ref47 ; Ref48 , reads as,

$\displaystyle\hat{\mathcal{H}}$ $\displaystyle=$ $\displaystyle\int
d\boldsymbol{r}\boldsymbol{\hat{\psi}^{{\dagger}}}(\boldsymbol{r})\bigg{(}\hat{\mathcal{H}}_{0}+\mathcal{V}_{Lat}\bigg{)}\boldsymbol{\hat{\psi}}(\boldsymbol{r})$
(1) $\displaystyle+$ $\displaystyle\frac{1}{2}\int
d\boldsymbol{r}\sum_{\sigma,\acute{\sigma}}\mathcal{U}_{\sigma,\acute{\sigma}}\hat{\psi}^{{\dagger}}_{\sigma}(\boldsymbol{r})\hat{\psi}^{{\dagger}}_{\acute{\sigma}}(\boldsymbol{r})\hat{\psi}_{\acute{\sigma}}(\boldsymbol{r})\hat{\psi}_{\sigma}(\boldsymbol{r})$
$\displaystyle+$
$\displaystyle\hbar\Delta_{c}\hat{c}^{{\dagger}}\hat{c}-i\hbar\eta(\hat{c}-\hat{c}^{{\dagger}}).$

Here
$\boldsymbol{\hat{\psi}}=[\hat{\psi}_{\uparrow},\hat{\psi}_{\downarrow}]^{T}$
represents the bosonic field operators corresponding to atomic pseudo-spin
states $|\uparrow\rangle$ and $|\downarrow\rangle$.
$\hat{\mathcal{H}}_{0}=\hbar^{2}\boldsymbol{k}^{2}\sigma_{0}/2m_{a}+\tilde{\alpha}k_{x}\sigma_{y}+\frac{\delta}{2}\sigma_{y}+\frac{\Omega_{z}}{2}\sigma_{z}$
is the single particle Hamiltonian accommodating spin-orbit interactions with
strength $\tilde{\alpha}=E_{R}/k_{L}$. Here $\delta=-g\mu_{B}B_{z}$ is the
Raman detuning and $\Omega_{z}=-g\mu_{B}B_{y}$ is the Raman coupling
corresponding to the magnetic field effects along the $\hat{z}$ and
$\hat{y}$-axis, respectively Ref34 ; Ref48 . The quasi-momentum is in one-
dimension $\boldsymbol{k}=[k_{x},0,0]$ because of the one dimensional SO-
coupling occurring along $\hat{x}$-axis. $\sigma_{x,y,z}$ represents Pauli
matrices with unit matrix $\sigma_{0}$. The two-dimensional lattice potential
induced by the longitudinal (cavity mode driven by pump laser) and transverse
(counter propagating Raman beans) fields reads as
$\mathcal{V}_{Lat}=\hbar\hat{c}^{{\dagger}}\hat{c}U_{0}[cos^{2}(kx)+cos^{2}(ky)]$
Ref48 , where $\hat{c}$ and $\hat{c}^{\dagger}$ correspond to the annihilation
and creation operator of cavity mode with wave number $k$, respectively.
$U_{0}=g_{0}^{2}/\Delta_{a}$ corresponds to the potential depth defined by the
Rabi oscillations $g_{0}$ and cavity-atomic detuning $\Delta_{a}$ Ref48 . The
atom-atom interaction can be defined by
$\mathcal{U}_{\sigma,\acute{\sigma}}=4\pi
a_{\sigma,\acute{\sigma}}^{2}\hbar^{2}/m_{a}$, where
$a_{\sigma,\acute{\sigma}}$ is the s-wave scattering. Last two terms
correspond to the intra-cavity optical mode and its relation with pump drive,
respectively, as mentioned before.

We assume that the intra-species and inter-species interactions of pseudo-spin
state as $U_{\uparrow,\uparrow}=U_{\downarrow,\downarrow}=U$ and
$U_{\uparrow,\downarrow}=U_{\downarrow,\uparrow}=\varepsilon U$, respectively,
with laser configuration parameter $\varepsilon$ Ref34 ; Ref48 . After that,
we insert plane-wave ansatz
$\boldsymbol{\hat{\psi}}(r)=e^{ikr}\boldsymbol{\hat{\varphi}}$, with
$\boldsymbol{\hat{\varphi}}=[\hat{\varphi}_{\uparrow},\hat{\varphi}_{\downarrow}]^{T}$,
to the bosonic wave-function and evaluate the corresponding integral. By using
normalization condition
$|\hat{\varphi}_{\uparrow}|^{2}+|\hat{\varphi}_{\downarrow}|^{2}=N$, we derive
the quantum Langevin equations Ref48 ; Ref49  to incorporate associated
damping and noises with the cavity-atom system.

$\displaystyle\frac{d\hat{c}}{dt}$ $\displaystyle=$
$\displaystyle\dot{\hat{c}}=(i\Delta-
iG\boldsymbol{\hat{\varphi}^{\dagger}\hat{\varphi}}-\kappa)\hat{c}+\eta+\sqrt{2\kappa}\hat{c}_{in},$
(2) $\displaystyle\frac{d\boldsymbol{\hat{\varphi}}}{dt}$ $\displaystyle=$
$\displaystyle\dot{\boldsymbol{\hat{\varphi}}}=(\frac{\Omega\sigma_{0}}{2}+\tilde{\alpha}\boldsymbol{k_{x}}\sigma_{y}+\frac{\delta}{2}\sigma_{y}+\frac{\Omega_{z}}{2}\sigma_{z}-\gamma+G\hat{c}^{{\dagger}}\hat{c})\boldsymbol{\hat{\varphi}}$
(3)
$\displaystyle+\frac{1}{2}U\boldsymbol{\hat{\varphi}}^{\dagger}\boldsymbol{\hat{\varphi}}\boldsymbol{\hat{\varphi}}+\frac{1}{2}\varepsilon
U\hat{\varphi}_{\sigma}^{\dagger}\hat{\varphi}_{\acute{\sigma}}\hat{\varphi}_{\sigma}+\sqrt{\gamma}f_{a}.$

Here $\sigma,\acute{\sigma}\in\\{\uparrow,\downarrow\\}$ and $\kappa$
corresponds to the intra-cavity photonic decay rate.
$\tilde{\Delta}=\Delta_{c}-NU_{0}/2$ is the effective atom-cavity detuning and
$\hat{c}_{\mathrm{in}}$ represents associated Markovian input noise associated
with cavity mode, with zero-average $\langle\hat{c}_{in}(t)\rangle=0$ over
delta-correlation
$\langle\hat{c}_{in}(t)\hat{c}_{in}^{\dagger}(\acute{t})\rangle=\delta(t-\acute{t})$
defined over the condition $\hbar\omega_{c}>>k_{B}T$ Dalibard2011 . The
external harmonic trapping potential for atomic mode , which we ignored
previously as it is assumed to be spin independent, cause the atomic motional
damping. $\gamma$ corresponds to pseudo states damping and $\hat{f}_{a}$ is
the associated Markovian noise operators with delta-correlation
$\langle\hat{f}_{a}(t)\hat{f}_{a}^{\dagger}(\acute{t})\rangle=\delta(t-\acute{t})$
with condition $\hbar\Omega>>k_{B}T$.

Further, $g_{a}=\frac{\omega_{c}}{L}\sqrt{\hbar/m_{bec}4\omega_{r}}$ is the
atom cavity coupling, with effective recoil frequency $\Omega=4\omega_{r}$ and
mass $m_{bec}=\hslash\omega_{c}^{2}/(L^{2}U^{2}_{0}\omega_{r})$ Ref48 ; Ref49
. By using the definitions of Pauli matrices, the quantum Langevin equations
will read as,

$\displaystyle\frac{d\hat{c}}{dt}$ $\displaystyle=$
$\displaystyle\dot{\hat{c}}=(i\Delta-
iG(\hat{\varphi}_{\uparrow}^{\dagger}\hat{\varphi}_{\uparrow}+\hat{\varphi}_{\downarrow}^{\dagger}\hat{\varphi}_{\downarrow})-\kappa)\hat{c}+\eta+\sqrt{2\kappa}\hat{c}_{in}$
(4)
$\displaystyle\frac{d}{dt}\binom{\hat{\varphi}_{\uparrow}}{\hat{\varphi}_{\downarrow}}$
$\displaystyle=$
$\displaystyle\begin{pmatrix}\frac{\Omega}{2}+\frac{\Omega_{z}}{2}+G\hat{c}^{{\dagger}}\hat{c}+\frac{1}{2}UN-\gamma+\sqrt{\gamma}f_{a}&-i(\alpha+\frac{\delta}{2})+\frac{1}{2}U(\varepsilon-1)\hat{\varphi}_{\downarrow}^{\dagger}\hat{\varphi}_{\uparrow}\\\
i(\alpha+\frac{\delta}{2})+\frac{1}{2}U(\varepsilon-1)\hat{\varphi}_{\uparrow}^{\dagger}\hat{\varphi}_{\downarrow}&\frac{\Omega}{2}-\frac{\Omega_{z}}{2}+G\hat{c}^{{\dagger}}\hat{c}+\frac{1}{2}UN-\gamma+\sqrt{\gamma}f_{a}\end{pmatrix}\binom{\hat{\varphi}_{\uparrow}}{\hat{\varphi}_{\downarrow}},$
(5)

where, $\alpha=\tilde{\alpha}\boldsymbol{k_{x}}$. The quantum Langevin
equations mention above are responsible for governing the dynamics of
collective system and are crucially important.

Figure 2: The steady-state behavior of cavity photon number $n_{s}$ as a
function of normalized external pump laser intensity $\eta/\kappa$ for
different cavity-atom couplings $G$ (a) and spin-orbit coupling strengths
$\alpha$ (b). In (a), the black, blue, and red curves correspond
$G=0.5\Omega$, $0.7\Omega$ and $1.0\Omega$, respectively, at
$\alpha=2.0\gamma$. While in (b), the black, blue, and red curves correspond
$\alpha=0.0\gamma$, $1.5\gamma$ and $2.0\gamma$, respectively, at
$G=0.7\Omega$. The pseudo-spin states are considered with frequency
$\Omega\approx 19\times 2\pi$kHz. While the other parametric are considered as
$\Omega_{z}=0.5\Omega$, $\delta=0$, $U/\Omega=5.5$, $\Delta=2.5\kappa$, and
$\kappa\approx 1.3\times 2\pi$MHz Refnew2 ; Refnew1 . Figure 3: The steady-
state cavity photon number $n_{s}$ as a function of $\eta/\kappa$ at constant
$G=0.7\Omega$ and $\alpha=2.0\gamma$. Here (a) contains the effects of Raman
coupling $\Omega_{z}$, while (b), (c) and (d) illustrate the influences of
Raman detuning $\delta$, inter-species interaction strength $\epsilon$, and
cavity decay rate $\kappa$, respectively. The other parameters used in
calculations are same as in Fig.2.

Steady-state behavior of any complex and hybrid system is crucially important
in order to develop the whole picture. In this study, we are going to govern
and discuss the steady-state dynamics of each associated sub-system in our
setup. The steady-state behavior is obtained by assuming subsystems as
classical variables (that is why we omit the motion of hat from variables) and
by putting time-derivative equals to zero in Quantum Langevin equations, which
will then read as,

$\displaystyle c_{s}$ $\displaystyle=$
$\displaystyle\frac{\eta}{\kappa+i\big{(}\Delta+G(\varphi_{\uparrow}^{\dagger}\varphi_{\uparrow}+\varphi_{\downarrow}^{\dagger}\varphi_{\downarrow})\big{)}},$
(6) $\displaystyle\varphi_{\uparrow,\downarrow}$ $\displaystyle=$
$\displaystyle\frac{(\pm\alpha\mp\frac{\delta}{2})\varphi_{\downarrow,\uparrow}}{\frac{\Omega\pm
i\Omega_{z}}{2}+Gc_{s}^{{\dagger}}c_{s}+\frac{1}{4}UN(\varepsilon+1)-\gamma},$
(7) $\displaystyle\varphi_{\uparrow,\downarrow}^{\dagger}$ $\displaystyle=$
$\displaystyle\frac{(\pm\alpha\mp\frac{\delta}{2})\varphi_{\downarrow,\uparrow}^{\dagger}}{\frac{\Omega\mp
i\Omega_{z}}{2}+Gc_{s}^{{\dagger}}c_{s}+\frac{1}{4}UN(\varepsilon+1)-\gamma}.$
(8)

The total number of steady-state photons inside the cavity can be defined as
$|c_{s}^{{\dagger}}c_{s}|=n_{s}$. Further, by assuming the total number
bosonic particles in atomic pseudo-spin state $\uparrow$ and $\downarrow$ as
$|\varphi_{\uparrow}^{\dagger}\varphi_{\uparrow}|=N_{\uparrow}$ and
$|\varphi_{\downarrow}^{\dagger}\varphi_{\downarrow}|=N_{\downarrow}$,
respectively, one can rewrite the steady-state equations as,

$\displaystyle n_{s}$ $\displaystyle=$
$\displaystyle\frac{\eta^{2}}{\kappa^{2}+\big{(}\Delta+G(N_{\uparrow}+N_{\downarrow})\big{)}^{2}},$
(9) $\displaystyle N_{\uparrow}$ $\displaystyle=$
$\displaystyle\frac{(\alpha-\frac{\delta}{2})^{2}N_{\downarrow}}{\big{(}\frac{\Omega}{2}+Gn_{s}+\frac{1}{4}UN(\varepsilon+1)-\gamma\big{)}^{2}+\big{(}\frac{\Omega_{z}}{2}\big{)}^{2}},$
(10) $\displaystyle N_{\downarrow}$ $\displaystyle=$
$\displaystyle\frac{(\alpha-\frac{\delta}{2})^{2}N_{\uparrow}}{\big{(}\frac{\Omega}{2}+Gn_{s}+\frac{1}{4}UN(\varepsilon+1)-\gamma\big{)}^{2}+\big{(}\frac{\Omega_{z}}{2}\big{)}^{2}}.$
(11)

Here, one can note that if we solve Eqs. 10 and 11, then we will get a similar
steady-state response for atomic number in both pseudo-spin states. The reason
is the same coupling parameter between atoms and cavity because both atomic
states are interacting with same cavity mode. If we consider different
strengths of cavity-atom coupling, then obviously response will be different.

##  III Multi-Stablity of Intra-Cavity Photon Number

In absence of SO-coupling $\alpha$ and Zeeman field effects $\Omega_{z}$ and
$\delta$, the steady-state response of the system will be almost same as in
cavity atomic-optomechanical system Meystre2010 ; kashif3 , which will be in
the form of bistable states. The bistable behavior of such systems have
extensively studied in literature and that is the reason of not discussing
such behavior here. But in presence of SO-coupling and Zeeman field effects,
the steady-state dynamics will dramatically change because now system does not
have only cavity-atom coupling $G$ but its pseudo-spin states are also SO-
coupled with Zeeman field effects. As the Zeeman field induced pseudo-spin
states are acting like two different species whose angular momentum is coupled
with each other in presence of cavity-atom coupling, therefore, the steady-
state dynamics will illustrate different behavior than previous
investigations, where it is simply bistable. Here, in our case, it is appeared
to be multi-stable (having more than two stable states) as illustrated in
Fig.2, where intra-cavity photon number is plotted versus normalized external
pump field coupling $\eta/\kappa$.

One can note in Fig.2(a) that the steady-state cavity photon number $n_{s}$
possesses multiple stable states at fixed SO-coupling $\alpha=2.0\gamma$. More
precisely saying, it contains three stable and two unstable states. The two
unstable states correspond to two saddle points representing the occurrence of
higher order nonlinearities or chaos in the system. One can some-how say that
at these unstable states, cavity photon number $n_{s}$ following unpredictable
and chaotic behavior. The three stable states determine where the $n_{s}$ is
predictable and provide us the parametric opportunity to operate system in a
stable domain. If we increase the strength of cavity-atom coupling, then these
multi-stable states are appeared to be moving towards low photon number domain
and enlarging towards higher intensities of external pump drive, as can be
seen by blue and red curves in Fig.2(a). It means that the increase in $G$
shrinks the total number photon in the steady-state, which is obvious as in
strong coupling domain more photons are used in tightly binding the cavity
mode. Similarly, if we keep the cavity-atom coupling constant and change the
SO-coupling, then one can observe the transformation of steady-state cavity
photon number from bistable dynamics to multi-stable behavior, as can be seen
in Fig.2(b), where $G=0.7\Omega$ is kept constant and $\alpha$ is being
changed. It can be seen that in absence of SO-coupling, photon number follows
bistable curvature (see black curve in Fig.2(b)). But when we apply and
increase SO-coupling $\alpha$, the cavity photon number shifts from bistable
to multi-stable behavior, as illustrated by blue and red curves in Fig.2(b),
where $\alpha=1.5\gamma$ and $\alpha=2.0\gamma$, respectively. The reason, as
mentioned previously, is that the pseudo-spin states are now acting as two
different entities not only coupled with each other via SO-interactions but
also coupled with cavity mode. Thus, these results provide evidence for the
occurrence of multi-stability in $n_{s}$ in presence of SO-coupled atomic
states.

Figure 4: The steady-state atomic number of pseudo spin-$\uparrow$ state
$N_{\uparrow}$ as a function of $\eta/\kappa$ under influence of cavity-atom
couplings $G$ (a) and spin-orbit coupling strengths $\alpha$ (b). In (a), the
black, blue, and red curves correspond $G=1.5\Omega$, $2.0\Omega$ and
$2.5\Omega$, respectively, at $\alpha=2.0\gamma$. While in (b), the black,
blue, and red curves represent $\alpha=2.5\gamma$, $3.0\gamma$ and
$3.5\gamma$, respectively, at $G=2.0\Omega$. The other parameters used in
calculations are same as in Fig.2.

However, these multi-stable dynamics crucially depends on the other system
parameters, as illustrated in Fig.3. The presence and increase in Raman
coupling $\Omega_{z}$ is appeared to be first minimizing domain of multi-
stability over the external pump intensity $\eta$ and then, at larger values,
suppressing the effects of multi-stability, as illustrated by the blue and red
curves of Fig.3(a). It is because, at higher values of Raman coupling
$\Omega_{z}$, the atomic pseudo states become more separated with each other
yielding in more strong Zeeman field interaction Ref34 , which likewise to
cavity-atom coupling results in the suppression of multi-stable photonic
modes. This feature could be crucial for performing parametric optical
switching in with our system. However, it also critically depends on the
strength of SO-coupling. In case of Raman detuning $\delta$, the difference
between photon number in different stable states is being increased with
decreased in Raman detuning $\delta$, as can be seen in Fig.3(b). The black,
blue, and red curves represent Raman detuning $\delta=+1\kappa$,
$\delta=0\kappa$ and $\delta=-1\kappa$, respectively. The decreasing in
$\delta$ minimizes the gap-splitting between energy levels resulting in
freeing more photons to add-up to the photonic stable states. It can also be
related with asymmetric behavior of eigen energy spectrum of SO-coupled BEC
Ref48 .

Further, similarly like $\Omega_{z}$ and cavity-atom coupling $G$, the
strength of inter-species (interaction between different pseudo-spin states)
also suppresses the multi-stability photon number, as shown in Fig.3(c). The
reason is again the same because with higher inter-species interaction values,
both pseudo-spin states more interactive with each other suppressing the
number of photons in the stable states. In Fig.3(d), one can observe the
influence of the ratio between cavity decay rate and dissipation associated
with atomic degrees fo freedom $\kappa/\gamma$. The decrease in
$\kappa/\gamma$ means that the atomic dissipation is being increase as
compared to cavity decay rate, as illustrated by black, blue, and red curves
where $\kappa/\gamma>1$, $\kappa=1\gamma$ and $\kappa<1\gamma$, respectively.
It is obvious because the atomic mechanical dissipation will directly
contribute to the photonic number $n_{s}$ inside cavity, which will result in
stable states with higher photon number. The dependence of multi-stability
over all these parameters is not only crucial but also provides the
controlability over steady-state dynamics kashif3 .

##  IV Multi-Stablity Analysis of Pseudo-Spin States

So-far we discussed the multi-stability of cavity photon number, but it is
also very significant to investigate the steady-state behavior of atomic
number (or bosonic particle number) of pseudo-spin states $N_{\uparrow}$ and
$N_{\downarrow}$. In this section, we will perform the stability analysis of
atomic number in pseudo-spin states. The steady-state behavior of
$N_{\uparrow}$ and $N_{\downarrow}$ is appeared to different from the behavior
of steady-state photon number. But, interestingly, it is also quite different
from conventional atom-optomechanical systems, as illustrated in Fig.4. Here,
it should be noted that we just illustrated the results of pseudo
spin-$\uparrow$ state because behavior spin-$\downarrow$ state will exactly be
likewise. However, later we will also discuss the combined steady-state
spectrum of the pseudo spin-$\uparrow$ and spin-$\downarrow$.

Figure 5: The steady-state population $N_{\uparrow}$ of atomic pseudo
spin-$\uparrow$ state versus normalized $\eta/\kappa$ under the effects of
cavity-decay rate $\kappa$ (a), Raman detuning $\delta$ (b) and inter-species
interactions $\epsilon$ (c). In (a), the black, blue, and red curves represent
$\kappa/\gamma=11$, $\kappa/\gamma=1$ and $\kappa/\gamma=0.5$, respectively.
In (b), the black, blue, and red curves correspond to $\delta=+1\kappa$,
$\delta=0\kappa$ and $\delta=-1\kappa$, respectively, while in (c) the black,
blue, and red curves represent $\epsilon=0.0U$, $\epsilon=0.1U$ and
$\epsilon=0.2U$, respectively. The remaining parameters used are same as in
Fig.2.

By analyzing Fig.4, one can note that the steady-state $N_{\uparrow}$
possesses one stable while two unstable states, which is quite unusual as
compared to the previous studies. The reason behind this is the SO-interaction
of pseudo-spin states of atomic degree of freedom. On one-side, these pseudo
states are coupled with the cavity mode, which one the other-side they are
mutually coupled with each other via SO-coupling. Because of the cavity-atom
coupling $G$, the both pseudo states possess multi-stable behavior, but the
stability behavior is also depending on the SO-coupling, which is generating
coupling between pseudo states. It can be easily imagined by considering three
balls which are mutually coupled with other. Whenever the state of one ball
will be changed, it will affect the state of all balls. In previous cases, if
there are multiple degrees of freedom in a cavity then they were coupled with
each other via cavity field, not by directly with each other. However, we will
further analyze this behavior later in this manuscript. Here, we will see the
effects of different system parameter on steady-state behavior of
$N_{\uparrow}$.

The increase in both cavity-atom coupling $G$ and SO-coupling $\alpha$ appear
to be enhancing population in spin-$\uparrow$ state towards the intensity of
external pump laser, as illustrated by the black, blue, and red curves of
Figs.4(a) and 4(b). However, total population $N_{\uparrow}$ is remained same.
It is because both $G$ and $\alpha$ are strengthening the coupling of pseudo
states with cavity mode, which makes them more dependent on the intensity of
external drive. It eventually increases the mechanical effects of light on
pseudo-spin states causing stable and unstable state to move further with
external drive intensity, while the total population remains the same. These
results could be referred as bi-unstable states because of the so-called two
unstable and one stable state. Similarly, other system parameters will also
alter the steady-state behavior of atomic population in spin-$\uparrow$ state,
as illustrated in Fig.5. The decrease in the ratio between cavity decay rate
and atomic damping $\kappa/\gamma$ is yielding in strengthening the bi-
unstable states towards external pump laser intensity similar to the photon
number case, as illustrated in Fig.5(a). The reason behind this is same that
the increase in atomic damping is appearing as contribution to the atomic
population. But interestingly here, the atomic population $N_{\uparrow}$ is
initially low, with low $\kappa/\gamma$, but appeared to be increasing with
increase in $\eta$.

Figure 6: The combined steady-state population $N_{\uparrow}$ and
$N_{\downarrow}$, atomic pseudo spin-$\uparrow$ and spin-$\downarrow$ states,
respectively, versus normalized $\eta/\kappa$ with respect to the cavity-atom
coupling $G$. In (a), $G=1.5\Omega$, while in (b) $G=8.5\Omega$ at
$\alpha=2.0\gamma$. Here the reddish curvature indicates the stability
dynamics of $N_{\uparrow}$, while yellowish curvature corresponds to the
population $N_{\downarrow}$. The remaining parameters used in numerical
calculation are same as in Fig.2.

However, the change in Raman detuning will dramatically alter the steady-state
behavior. When the $\delta=0\kappa$, the steady-state atomic population
$N_{\uparrow}$ possesses minimum values but when we move $\delta$ from
symmetric to asymmetric case, then the population $N_{\uparrow}$ is
significantly enhancing in bi-unstable states, as can be seen by the blue and
red curves of Fig.5(b), where $\delta=+\kappa$ and $\delta=-1\kappa$,
respectively. The reason of it the occurrence of asymmetric shift in phase
transition of atomic pseudo-spin states releasing more particle to bi-unstable
states and these effects are appeared to be more prominent in negative phase
shift case $\delta=-1\kappa$. However, inter-species interaction of pseudo
states is showing similar effects on bi-stable states of $N_{\uparrow}$ as it
is showing on intra-cavity photon number state $n_{s}$, as illustrated in
Fig.5(c). The reason is also the same as more interactive pseudo-states
results in less unstable particles in bi-stable atomic states. These results
not only provide the evidence for the occurrence of bi-unstability but also
illustrate the parametric controllability of the steady-state behavior.

Figure 7: The collective dynamics of population $N_{\uparrow}$ and
$N_{\downarrow}$ versus normalized $\eta/\kappa$ with respect to the SO-
coupling $\alpha$ (a-c) and Raman coupling $\Omega_{z}$ (d-f).
$\alpha=0.0\gamma$, $\alpha=2.0\gamma$ and $\alpha=4.0\gamma$ are using in
(a), (b) and (c), respectively. While $\Omega_{z}=0.5\Omega$,
$\Omega_{z}=1.5\Omega$ and $\Omega_{z}=2.0\Omega$ are using in (d), (e) and
(f), respectively. The other parameters are same as in Fig.2.

In order to further explore the steady-state dynamics of atomic population in
different pseudo-spin states, we plot $N_{\uparrow}$ and $N_{\downarrow}$
together against normalized $\eta/\kappa$, as illustrated in Fig.6. One can
note the emergence of transaction between steady-state $N_{\uparrow}$ and
$N_{\downarrow}$. It clearly indicates that the atomic modes are possessing
phase transition between spin-$\uparrow$ and spin-$\downarrow$ states. It
could also be reason of having one state and two unstable states in previous
results because another stable state is appeared to be coupled with other
population of other pseudo-spin state. The reason behind this is of course the
SO-coupling between these sub-atomic states that is inducing another unstable
state (instead of a stable state) in the pseudo state with opposite spin. In
other words, the mechanical effects of light produce by the cavity mode are
initially induced bistable states for atomic population but later the SO-
coupling between pseudo states is converting one stable state to unstable.
This type of cavity mediated SO-interaction in ultra-cold atoms are very
crucial in order to bringing our new physics of multi-species hybrid
interaction and has subject of various recent investigation kashif55 . If we
increase the cavity-atom coupling, the transitional interface between
$N_{\uparrow}$ and $N_{\downarrow}$ is appeared to be moving up at higher
external pump laser intensities, as illustrated in Figs.6(a) and 6(b), where
$G=1.5\Omega$ is moved to $G=8.5\Omega$. It is obvious because the strength of
cavity mode, causing the coupling between cavity and atoms, is directly
proportional to the intensity of external pump. However, these all features
crucially depends on other associated system parameter, which we are going
discuss next in the manuscript.

As the strength of SO-coupling $\alpha$ and Zeeman field effect induced Raman
coupling $\Omega_{z}$ are both responsible for the splitting and creation of
atomic pseudo-spin states, which are main reason for the alterations appearing
in steady-state dynamics of both cavity mode and atomic pseudo states.
Therefore, it is important to see the collective behavior of these pseudo spin
spin-$\uparrow$ and spin-$\downarrow$ states under SO-coupling $\alpha$ and
Raman coupling $\Omega_{z}$, as illustrated in Figs.7(a-c) and 7(d-f),
respectively. One can easily observe that, in the absence of SO-coupling
$\alpha=0$, the population in both state $N_{\uparrow}$ and $N_{\downarrow}$
illustrate similar behavior and remain zero with respect to external pump
intensity, see Fig.7(a). It is obvious because in this configuration, there
are no pseudo-spin states and BEC is acting as a single entity. However, when
we start applying and increasing the SO-coupling $\alpha$, the intersection
between $N_{\uparrow}$ and $N_{\downarrow}$ versus $\eta$ starts appearing,
similarly as illustrated in Fig.6. At lower strengths of $\alpha$, that
transitional connection is occurring at lower population values. However, when
we increase $\alpha$, it is shifted to higher values of $N_{\uparrow}$ and
$N_{\downarrow}$, as illustrated in Figs.7(b) and 7(c), where
$\alpha=2.0\gamma$ and $\alpha=4.0\gamma$, respectively. The justification is
same as given in one-dimensional case that at higher values of $\alpha$, the
stable states saturate to higher population because now more atoms are
interacting with not only $\alpha$ but also with $G$. In other words, at
higher $\alpha$ more atoms are vulnerable to the mechanical effects of light.
Similar to the one-dimensional case, the Raman coupling $\Omega_{z}$ here is
appeared to be suppressing the number of atoms in pseudo-spin states, as
illustrated in Figs.7(d-f), where $\Omega_{z}=0.5\Omega$,
$\Omega_{z}=1.5\Omega$ and $\Omega_{z}=2.0\Omega$, respectively. Not only
$N_{\uparrow}$ and $N_{\downarrow}$ are appearing to be saturating at lower
values but the intersection between $N_{\uparrow}$ and $N_{\downarrow}$ is
also being transferred to low atomic number domain. The reason, as mentioned
earlier, is the higher order splitting between pseudo spin-$\uparrow$ and
spin-$\downarrow$ state with higher Zeeman field effects resulting in less
atomic bound among pseudo states. In other words, at higher Zeeman field
effects, the SO-interaction dominates the mechanical effects of light and that
is why less atomic population will be influenced by the external pump laser
intensity.

Figure 8: The collective dynamics of $N_{\uparrow}$ and $N_{\downarrow}$ with
respect to normalized $\eta/\kappa$ under influence of the ratio between
cavity decay rate and atomic damping $\kappa/\gamma$. (a-c) correspond to the
ratio of cavity decay rate and atomic damping $\kappa/\gamma=5.0$,
$\kappa/\gamma=1.0$ and $\kappa/\gamma=0.5$, respectively. The remaining
parameters are same as in Fig.2.

Moving further, the ratio between cavity decay rate and atomic damping
$\kappa/\gamma$ also dramatically alters the steady-state atomic number pseudo
spin-$\uparrow$ and spin-$\downarrow$ states, as shown in Figs.8(a-c), where
$\kappa/\gamma=5.0$, $\kappa/\gamma=1.0$ and $\kappa/\gamma=0.5$,
respectively. One can note that when we increase the value of atomic
dissipation $\gamma$ while keeping the cavity decay rate $\kappa$ constant,
then $\gamma$ is appeared to be contributing to interactions between atomic
spin-$\uparrow$ and spin-$\downarrow$ states. It results in the appears of
another transitional interface between atomic population $N_{\uparrow}$ and
$N_{\downarrow}$, which is indeed very interesting. It is similar to the
appearance of amplification of probe light transparency from SO-coupled BEC
case, see reference kashif55 , where atomic dissipation is appearing as gain
to probe light. This phenomenon also led to the creation of topological edge
state in transmitted bulk mode of probe light, as discussed in same study
kashif55 . Thus, the appearance of secondary transitional interface between
the atomic population $N_{\uparrow}$ and $N_{\downarrow}$ could be because (or
cause) of the occurrence of non-trivial phase transitions of pseudo atomic
spin states, and it could provide the foundation for multiple studies in this
direction.

##  V Conclusion

In conclusion, we demonstrate multi-stable steady-state behavior of a cavity
with spin-orbit coupled Bose-Einstein condensate. After governing steady-state
dynamics, we illustrate multi-stability, with multiple stable and unstable
states, of cavity photon number controllable with system parameters. Such
controllable optical switching between stable and unstable states is highly
crucial for modern optical devices. Further, we discuss the occurrence of
similar multi-stability for the atomic population in each pseudo-spin state
and show that atomic population possesses two unstable (so-called bi-unstable)
states corresponding to external pump drive. The population of spin states is
appeared to be also depending on each other because of the emerging
transitional interface between both number when we observe them collectively.
That unique feature becomes further crucial when another interface appears
between spin state population by increasing atomic mechanical damping rate
higher than the cavity decay rate. This could correspond to the non-trivial
optical interactions of SO-coupled BEC inside cavity kashif55 . The study not
only provides another direction to talk about optical switching between
multiple stable and unstable states, but it is also crucial for steady-states
interaction between different atomic synthetic states.

###### Acknowledgements.

K.A.Y. acknowledges the support of Research Fund for International Young
Scientists by NSFC under grant No. KYZ04Y22050, Zhejiang Normal University
research funding under grant No. ZC304021914 and Zhejiang province
postdoctoral research project under grant number ZC304021952. G.X.L.
acknowledges the support of National Natural Science Foundation of China under
Grant Nos. 11835011 and 11774316.

## References

  * (1) H. Walther, B. T. H. Varcoe, B. G. Englert, and T. Becker, Rep. Prog. Phys. 69 1325–1382 (2006).
  * (2) S. Haroche and J.-M. Raimond, Exploring the Quantum: Atoms, Cavities and Photons, Oxford University Press (2006).
  * (3) S. Haroche, and J. M. Raimond, Scientific American 268 54-60 (1993).
  * (4) S. Haroche, Rev. Mod. Phys. 85 1083 (2013).
  * (5) A. Blais, Arne L. Grimsmo, S. M. Girvin, and A. Wallraff, Rev. Mod. Phys. 93, 025005 (2021).
  * (6) R. Miller et al., J. Phys. B: At. Mol. Opt. Phys. 38 S551 (2005).
  * (7) P. Goy, J. M. Raimond, M. Gross, and S. Haroche, Phys. Rev. Lett. 50 1903 (1983).
  * (8) C. Monroe, Nature 416 238-246 (2002).
  * (9) J. Stajic, Science 339 1163 (2013).
  * (10) F. Brennecke, S. Ritter, T. Donner, and T. Esslinger, Science 322, 235 (2008).
  * (11) K. Zhang, W. Chen, M. Bhattacharya, and P. Meystre, Phys. Rev. A 81, 013802 (2010).
  * (12) K. A. Yasir, M. Ayub, and F. Saif, J. Mod. Opt. 61, 1318 (2014).
  * (13) M. Ayub, K. A. Yasir, and F. Saif, Laser Phys. 24, 115503 (2014).
  * (14) M. Paternostro, G. D. Chiara, and G. M. Palma, Phys. Rev. Lett. 104, 243602 (2010).
  * (15) M. Abdi, et al., Phys. Rev. Lett. 109, 143601 (2012).
  * (16) M. Abdi, et al., Phys. Rev. A 89, 022331 (2014).
  * (17) E. A. Sete, et al., J. Opt. Soc. Am. B 31, 2821 (2014).
  * (18) S. G. Hofer, et al., Phys. Rev. A 84, 052327 (2011).
  * (19) Ying-Dan Wang and A. A. Clerk, Phys. Rev. Lett. 108, 153603 (2012).
  * (20) S. Singh, et al., Phys. Rev. A 86, 021801 (2012).
  * (21) G. S. Agarwal, and S. Huang, Phys. Rev. A 81, 041803(R) (2010).
  * (22) S. Weis, et al., Science 330, 1520 (2010).
  * (23) B. Peng, et al., Nat. Commun. 10, 1038 (2014).
  * (24) A. H. Safavi-Naeini, et al., Nature 472, 69 (2011).
  * (25) K. A. Yasir, Zhaoxin Liang, G. Xianlong, and W. M. Liu, Eur. Phys. J. Plus 138, 29 (2023).
  * (26) A. Munir, K. A. Yasir, W. M. Liu, and G. Xianlong, Eur. Phys. J. Plus 137, 1143 (2022).
  * (27) K. A. Yasir, and W. M. Liu, Sci. Rep. 5, 10612 (2015).
  * (28) K. A. Yasir, Opt. Commun. 488, 126820 (2021).
  * (29) K. A. Yasir, and W. M. Liu, Sci. Rep. 6, 22651 (2016).
  * (30) V. Galitski, and I. B. Spielman, Nature 494, 49 (2013).
  * (31) X. J. Liu et al., Phys. Rev. Lett. 102, 046402 (2009).
  * (32) Y. J. Lin et al., Nature 462, 628 (2009).
  * (33) Y. K. Kato et al., Science 306, 1910-1913 (2004).
  * (34) M. Konig et al., Science 318, 766-770 (2007).
  * (35) B. A. Bernevig, T. L. Hughes, and S. C. Zhang, Science 314, 1757-1761 (2006).
  * (36) D. Hsieh, et al., Nature 452, 970-974 (2008).
  * (37) X. Li, E. Zhao, and W. V. Liu, Nat. Commun. 4, 4023 (2013).
  * (38) K. Sun et al., Nat. Phys. 8, 67-70 (2012).
  * (39) J. D. Koralek et al., Nature 458, 610-613 (2009).
  * (40) C. Hamner, Phys. Rev. Lett. 114, 070401 (2015).
  * (41) Y. J. Lin, K. Jim$\acute{e}$nez-Garc$\acute{\iota}$a, and I. B. Spielman, Nature 471, 83 (2011).
  * (42) H. Hu et al., Phys. Rev. Lett. 108, 010402 (2012).
  * (43) Y. Deng et al., Phys. Rev. Lett. 112, 143007 (2014).
  * (44) B. Padhi, and S. Ghosh, Phys. Rev. A 90, 023627 (2014).
  * (45) L. Dong et al., Phys. Rev. A 89, 011602(R) (2014).
  * (46) K. A. Yasir, L. Zhuang, and W. M. Liu, Phys. Rev. A 95, 013810 (2017).
  * (47) K. A. Yasir, L. Zhuang, and W. M. Liu, npj Quantum Inf. 8, 109 (2022).
  * (48) S. Zibrov, et al., Phys. Rev. A 72, 011801(R) (2005).
  * (49) F. Brennecke, et al., Nature 450, 268-271 (2007).
  * (50) F. Brennecke, S. Ritter, T. Donner, and T. Esslinger, Science 322, 235-238 (2008).
  * (51) M. Paternostro, G. D. Chiara, and G. M. Palma, Phys. Rev. Lett. 104, 243602 (2010).
  * (52) J. Dalibard et al., Rev. Mod. Phys. 83, 1523 (2011).


# Enhancing Continual Learning in Visual Question Answering
with Modality-Aware Feature Distillation

Malvina Nikandrou1 Georgios Pantazopoulos1,2 Ioannis Konstas1,2 Alessandro
Suglia1,2 1Heriot-Watt University; 2Alana AI
{mn2002, gmp2000, i.konstas<EMAIL_ADDRESS>

###### Abstract

Continual learning focuses on incrementally training a model on a sequence of
tasks with the aim of learning new tasks while minimizing performance drop on
previous tasks. Existing approaches at the intersection of Continual Learning
and Visual Question Answering (VQA) do not study how the multimodal nature of
the input affects the learning dynamics of a model. In this paper, we
demonstrate that each modality evolves at different rates across a continuum
of tasks and that this behavior occurs in established encoder-only models as
well as modern recipes for developing Vision & Language (VL) models. Motivated
by this observation, we propose a modality-aware feature distillation (MAFED)
approach which outperforms existing baselines across models of varying scale
in three multimodal continual learning settings. Furthermore, we provide
ablations showcasing that modality-aware distillation complements experience
replay. Overall, our results emphasize the importance of addressing modality-
specific dynamics to prevent forgetting in multimodal continual learning.

Enhancing Continual Learning in Visual Question Answering
with Modality-Aware Feature Distillation

Malvina Nikandrou1 Georgios Pantazopoulos1,2 Ioannis Konstas1,2 Alessandro
Suglia1,2

1Heriot-Watt University; 2Alana AI

{mn2002, gmp2000, i.konstas<EMAIL_ADDRESS>

##  1 Introduction

Large Language Models (LLMs) Touvron et al. (2023); Jiang et al. (2023) and
Visual Language Models (VLMs) (Bai et al., 2023b; Liu et al., 2024), have
achieved unprecedented performance and have become the go-to option for most
NLP and Vision & Language (VL) tasks. However, once they have been trained, it
is not straightforward how to update them to accommodate for novel concepts or
concepts with reworked meanings. As a result, over time, the knowledge of
these models may be obsolete or needs to be refined periodically to maintain
their relevance. Consider two examples: 1) As of July 2023, Twitter has been
re-branded to X with a new logo. 2) According to the Oxford English Dictionary
(OED), as of March 2024, more than 1000 English words or phrases have been
either been revised or included as novel entries111 OED March 2024 update
lists entries appearing for the first time, or entries with updated meanings..
In these cases, models trained with data from a preceding period will
inevitably show performance deterioration Lazaridou et al. (2021).
Commercialized LLMs circumvent this limitation OpenAI (2022); Gemini Team et
al. (2023); Anthropic (2024) with statements regarding the knowledge cutoff of
these models. On the other hand, humans continuously update their knowledge
and acquire new skills over time. Continual learning is a paradigm that aims
to simulate this behavior, focusing on models that learn incrementally from a
sequence of tasks with minimal catastrophic forgetting McCloskey and Cohen
(1989); Ratcliff (1990).

Figure 1: Overview of MAFED. Along with training on the data from the current
task and a memory buffer, we apply feature distillation using the previous
checkpoint as the teacher. The distillation losses applied to the
representations from question and visual tokens are weighted separately to
compensate for modality-specific training dynamics.

Continual learning has started being explored more widely in VL settings Greco
et al. (2019); Srinivasan et al. (2022); Nikandrou et al. (2022); Zhang et al.
(2023); Lei et al. (2023); Cui et al. (2024). However, existing approaches do
not explicitly account for the dissimilarities in the representation space of
multimodal inputs and their effect on the learning dynamics, which we argue is
necessary for effective continual learning of VL tasks. Previous work on the
optimization dynamics of multimodal learning has demonstrated that different
modalities are learned at different speeds Wang et al. (2020); Wu et al.
(2022). Using encoder and decoder-only VLMs, we empirically showcase a similar
forgetting discrepancy (Section 4.1), which indicates that representations
from each modality evolve at different rates across a sequence of tasks.

Motivated by this observation, we propose MAFED, a Modality-Aware FEature
Distillation approach summarized in Figure 1. We explore different strategies
for weighting the distillation losses derived from the tokens of each
modality, using either fixed balanced weights or adaptive weights derived from
the loss gradients computed with respect to the inputs. We combine both
variants with experience replay and show promising results across all VLM
families compared to established continual learning methods. Additionally, we
conduct experiments with decoder-only VLMs ranging from 100M to 1B parameters,
showing that although scale alleviates forgetting, certain settings remain
challenging. Our ablations comparing experience replay and feature
distillation approach showcase that these methods are complementary and yield
greater performance when combined. Overall, our results emphasize the need to
address modality-specific dynamics to effectively mitigate forgetting in
multimodal continual learning.

Figure 2: Illustration of tasks in each of the three continual learning
settings for VQA. Each of these settings consists of five tasks. The first two
settings are defined based on the visual categories. In Diverse Content, the
objects present in each task are grouped randomly, while in Taxonomy Content,
the objects are grouped based on their supercategory. Finally, in Question
Types, the tasks are defined according to the type of the questions.

##  2 Related Work

###  2.1 Continual Learning

##### Continual Learning Approaches

Continual learning approaches can be categorized as regularization, replay,
and architecture-based Delange et al. (2021). Regularization-based approaches
introduce auxiliary losses that aim to constrain the model weights Kirkpatrick
et al. (2017); Zenke et al. (2017); Aljundi et al. (2018), outputs Li and
Hoiem (2018); Rebuffi et al. (2017), or internal representations Hou et al.
(2019). Replay-based approaches rely on storing Chaudhry et al. (2019);
Buzzega et al. (2020); Bagus and Gepperth (2021) or generating samples Van de
Ven and Tolias (2018); Sun et al. (2019) from past tasks so that they can be
sampled along with new samples during training. Finally, architecture-based
approaches introduce task-specific parameters, either by masking the model
parameters Yoon et al. (2020); Serrà et al. (2018) or adding new ones for each
task Fernando et al. (2017); Madotto et al. (2021).

Most recent work tends to combine techniques from multiple categories in order
to maximize performance. Similarly, our work utilizes replay and
regularization through feature distillation. Distillation has been used in
various continual learning approaches. Some focus on knowledge distillation on
the output level, using the logits Li and Hoiem (2018) or pseudo-labels from a
past checkpoint Wang et al. (2022); Karim et al. (2022). Other work applies
distillation on the internal model representations Dhar et al. (2019);
Douillard et al. (2020); Kang et al. (2022). MAFED expands this line of work
by introducing different weighting schemes to balance the distillation loss
from visual and textual representations.

##### VL Continual Learning

Previous work has studied varying instantiations of VL continual learning
problems, including image captioning Del Chiaro et al. (2020); Nguyen et al.
(2019), compositional phrase generalization Jin et al. (2020), and task-
incremental learning Srinivasan et al. (2022). Within VQA, prior work has
investigated continual learning based on question types Greco et al. (2019),
across varying domains Zhang et al. (2022); Lao et al. (2023), and from a
compositionality perspective Zhang et al. (2023). Lei et al. (2023); Nikandrou
et al. (2022) further study how VQA models evolve in different settings,
including novel visual scenes or different question types. However, these
works have not investigated the effect of modality-aware methods with the
exception of Qian et al. (2023) that focus on multimodal prompt learning for
vision, text and fusion modules. In contrast, feature distillation does not
assume separate modality-specific and multimodal parameters and can be applied
to more varied VLM architectures.

###  2.2 VLMs

Progress in representation learning has led to models that achieve impressive
performance across multimodal benchmarks Goyal et al. (2017); Hudson and
Manning (2019); Li et al. (2023). Early approaches relied on complicated
architectures Tan and Bansal (2019); Lu et al. (2019), and multiple learning
objectives Chen et al. (2020); Li et al. (2021); Jia et al. (2021). More
recently, given the rapid development of increasingly capable LLMs Touvron et
al. (2023); Jiang et al. (2023); Bai et al. (2023a), these approaches have
been superseded by a new paradigm where representations from visual experts
Radford et al. (2021); Oquab et al. (2024) are treated as input tokens for the
LLM. This shift has led the development of modern VLMs Liu et al. (2024); Dai
et al. (2024); Laurençon et al. (2024a) that are based on the same underlying
principles with deviations regarding the choice of the experts, or how the
patch tokens are integrated into the language model. Our experiments
demonstrate the effectiveness of our approach in both encoder-only Chen et al.
(2020); Kim et al. (2021) as well as decoder-only models.

Figure 3: Ratio of text-to-image representation similarity across layers and
tasks, for UNITER (first row), ViLT (second-row), and VL-Pythia (third-row).
We consistently observe that in the earlier layers, the ratio is close to one,
indicating that representations from both modalities change at a similar rate.
However, in intermediate or deeper layers, text representations seem to retain
larger similarities.

##  3 Preliminaries

###  3.1 Data

We leverage an existing evaluation suite for Continual Learning in VQA
Nikandrou et al. (2022) comprised of three settings based on the visual and
the language input. In particular, each of these settings consists of five
tasks and is designed to test the model’s performance on learning varying
concepts or question types. Figure 2 illustrates exemplary images and
questions from different tasks in each of the settings. Below, we provide a
brief summary for each of these settings.

##### Diverse Content

corresponds to a real-world use case as well as a common standard procedure
within continual learning Lomonaco and Maltoni (2017); Rebuffi et al. (2017);
Zenke et al. (2017); Lin et al. (2021), where a model is trained on new sets
of concepts progressively that do not necessarily comply to a taxonomy. Each
task in this setting covers 10 distinct object categories from the COCO
dateset Lin et al. (2014).

##### Taxonomy Content

In this setting, each task consists of questions regarding objects based on
the same super-category. This setting contains questions from the following
categories: Animals, Food, Interior, Sports, and Transport, and simulates a
more progressive approach, where a model learns about a fixed (and similar)
pool of concepts before being applied to a different domain. Importantly, we
note that in both Diverse Content and Taxonomy Content, images containing
objects shared between tasks are discarded to create clean task splits,
preventing contamination between them. In total, there are 181K train, 45K
validation, and 110K test samples for both settings.

##### Question Types

The final setting resembles a scenario where the model learns to answer
different categories of questions. In this setting, the model is tasked with
learning from a sequence of five tasks: Count, Color, Scene-level,
Subcategory, and Action recognition. Question Types have a total of 140K
train, 35K validation, and 84K test samples.

###  3.2 Models

Throughout our experiments, we use two families of models, including encoder-
and decoder-only pretrained models. Specifically, we use the encoder-only
models, UNITER-base Chen et al. (2020) and ViLT-base Kim et al. (2021), that
differ in terms of how the visual input is encoded. UNITER uses region
features extracted from the Faster R-CNN object detector Anderson et al.
(2018). On the other hand, ViLT is a patch-based model that does not use an
additional vision encoder.

However, recent trends in the development of VLMs have moved towards decoder-
only architectures that combine an expert vision encoder with an LLM using a
connector module that learns a mapping between them Liu et al. (2024). We
employ a similar recipe to combine EVA02 Fang et al. (2023) as the visual
encoder and Pythia (Biderman et al., 2023) as our LLM. Regarding the
connection module, we follow the LLaVA-1.5 Liu et al. (2023) approach with a
two-layer MLP that matches the dimensionality of the visual and the language
embeddings. We refer to this model as VL-Pythia.

We refrain from using other existing VLMs for two reasons. First, we aim to
match the parameters and pretraining data between the encoder-only and the
generative models. Pythia models provide a collection of checkpoints covering
a wide range of sizes that have been pretrained following a state-of-art
transformer recipe Su et al. (2024); Dao et al. (2022). For a controlled
comparison with encoder-only models, we train VL-Pythia using the same data
used to train UNITER and ViLT (see Section A.1 for additional details
regarding the pretraining of the model). In particular, we use the checkpoints
with 160M, 410M, and 1B parameters. The smaller model is on par with the
encoder-only ones, while the larger models allow us to explore the role of
model capacity. Secondly, existing VLMs Liu et al. (2024); Dai et al. (2024);
Bai et al. (2023a); Laurençon et al. (2024a, b) are typically instruction-
tuned on datasets that include VQA-v2 Goyal et al. (2017) on which the
continual learning settings are based. This overlap is undesirable since we
want to prevent any form of data contamination between tasks.

During continual learning, we keep the vision encoders of UNITER and VL-Pythia
frozen. In encoder-only models, VQA is treated as a classification task. The
classification (CLS) token is passed to a classification head that gets
expanded with the new answers from each task. On the contrary, VL-Pythia is
fine-tuned to generate answers autoregressively. In our experiments, we follow
a greedy decoding strategy during inference.

##  4 Method

###  4.1 Motivation

In this work, we argue that modality-specific learning dynamics, and more
specifically, the different speeds at which each modality is learned Wu et al.
(2022) or forgotten, should be accounted for in multimodal continual learning
settings. We demonstrate this behavior by measuring the similarities of the
question $Q$ and image $V$ representations from sequential model checkpoints
using Centered Kernel Alignment (CKA) Kornblith et al. (2019). In particular,
we extract text $Q_{t}$ and image $V_{t}$ representations for data of the
first task after training on the first $t=1$ and each subsequent task
$t=2\cdots T$, and we compute the CKA similarity across time. Finally, we
visualize the ratio $R_{t}$ of the text over image similarities:

$R_{t}=\frac{CKA(Q_{1},Q_{t})}{CKA(V_{1},V_{t})}\ \quad\forall t=2\cdots T$
(1)

Figure 3 shows the Text-to-Image CKA ratio per layer. We observe that there
are differences in how the modalities evolve within the models and across
settings. First, we note that the ratio is greater or equal to one in all
cases, meaning that visual tokens exhibit decreasing similarity throughout the
learning process. In UNITER, the ratio remains close to one for earlier layers
and increases only for the last three layers. ViLT and Pythia exhibit a
different trend, where the ratio peaks for intermediate layers. As a result,
we hypothesize that incorporating the variability of each modality in a
regularization technique can benefit continual learning strategies. Due to
parameter sharing between the two modalities, we materialize this in a
modality-aware feature distillation strategy, which we elaborate on below.

###  4.2 Modality-Aware Feature Distillation

Feature Distillation (FD) is an established continual learning technique Hou
et al. (2019); Douillard et al. (2020); Kang et al. (2022) that adds a
regularization loss term to prevent the drift of model representations. Given
two model checkpoints $f_{t-1}$ and $f_{t}$ from consecutive tasks, we extract
representations $H$ from an intermediate layer and compute the feature
distillation loss $l_{i}$ using the representations of each token $h_{i}$:

$\mathrm{L_{FD}}=\frac{1}{N}\sum_{i=1}^{N}l_{i}=\frac{1}{N}\sum_{i=1}^{N}\parallel
h_{i,t}-h_{i,t-1}\parallel^{2}_{2}$ (2)

Assuming an example has Q text tokens and V visual tokens, we can rewrite
Equation 2 in terms of the average loss contributed by the language and the
vision tokens, $\mathrm{L_{FD,Q}}$ and $\mathrm{L_{FD,V}}$ respectively:

$\mathrm{L_{FD,weighted}}=\alpha\cdot\mathrm{L_{FD,Q}}+(1-\alpha)\cdot\mathrm{L_{FD,V}}$
(3)

In the simplest case, $\alpha$ is proportional to the number of tokens
available from each input modality. In practice, this might be suboptimal
since it depends on the input tokenization strategy. In fact, across the
examined settings, the average tokenized inputs have approximately 9 question
tokens and 33 or 199 visual tokens for region and patch-based image features,
respectively. Consequently, visual tokens will dominate the distillation loss
of Equation 3 potentially leading to inferior performance.

Therefore, we experiment with two modifications: i) MAFED-B which balances the
losses from each modality by fixing $\alpha$ to 0.5, and ii) MAFED-A which
uses an adaptive weighting approach based on modality importance inspired by
Kang et al. (2022). Modality importances $I_{Q}$ and $I_{V}$ are estimated
using the gradient of the VQA classification loss222Or the language model head
in the case of Pythia., with respect to the intermediate model representations
$H_{m}$ from each modality $m$. $I_{Q}$ and $I_{V}$ are updated at the
beginning of each training task using the available memory data $M_{t}$ as
follows:

$I_{m}=\mathbb{E}_{(x,y)\sim
M_{t}}\left[\parallel\nabla_{H_{m}}L_{cls}(f_{t}(x),y)\parallel^{2}_{F}\right]$
(4)

where $\parallel\cdot\parallel$ corresponds to the Frobenius norm. Finally,
the weight $\alpha$ is computed by normalizing the importance of the question
tokens:

$\alpha=\frac{I_{Q}}{I_{Q}+I_{V}}$ (5)

We apply feature distillation to all layers except the last since only the
representation of CLS token in encoder-only and the final question token in
decoder-only models is propagated to the model output’s head. Furthermore, in
Section 4.1, we showcased that the representations of deeper layers are
affected more during continual learning. Therefore, we introduce a discount
factor $w_{d}$ that is used to weigh the contribution of the loss from each
layer proportionally to its distance $d$ from the model’s head:

$w_{d}=\frac{\gamma^{d}}{\sum_{d=0}^{D}\gamma^{d}}\ $ (6)

|  | Diverse Content | Taxonomy Content | Question Types
---|---|---|---|---
Model | Method | Accuracy | SBWT | Accuracy | SBWT | Accuracy | SBWT
UNITER | FT∗ | 64.59 $\pm$ 0.56 | -1.93 $\pm$ 0.39 | 63.65 $\pm$ 0.63 | -3.89 $\pm$ 0.53 | 48.81 $\pm$ 5.56 | -22.43 $\pm$ 7.02
EWC∗ | 66.26 $\pm$ 0.55 | -0.67 $\pm$ 0.29 |  67.70 $\pm$ 0.29 | -0.62 $\pm$ 0.19 | 66.77 $\pm$ 3.54 | -2.62 $\pm$ 2.28
ER∗ | 66.47 $\pm$ 0.51 | -0.29 $\pm$ 0.18 | 66.76 $\pm$ 0.16 | -1.22 $\pm$ 0.10 | 69.01 $\pm$ 0.76 | -1.42 $\pm$ 0.31
FD | 66.67 $\pm$ 0.38 | -0.17 $\pm$ 0.19 | 66.94 $\pm$ 0.23 | -0.68 $\pm$ 0.17 | 69.53 $\pm$ 0.12 | -1.22 $\pm$ 0.51
MAFED-B |  66.77 $\pm$ 0.24 |  -0.12 $\pm$ 0.17 | 67.05 $\pm$ 0.23 |  -0.57 $\pm$ 0.08 |  69.58 $\pm$ 0.55 | -1.17 $\pm$ 0.31
MAFED-A | 66.52 $\pm$ 0.26 | -0.23 $\pm$ 0.12 | 66.84 $\pm$ 0.25 | -0.70 $\pm$ 0.45 | 69.34 $\pm$ 0.43 |  -0.94 $\pm$ 0.48
Multitask∗ | 69.76 $\pm$ 0.18 | - | 70.08 $\pm$ 0.18 | - | 72.54 $\pm$ 0.15 | -
ViLT | FT∗ | 61.07 $\pm$ 0.41 | -2.80 $\pm$ 0.41 | 61.25 $\pm$ 0.50 | -4.09 $\pm$ 0.50 | 36.95$\pm$ 11.09 | -32.86 $\pm$ 11.09
EWC∗ | 61.80 $\pm$ 0.96 | -1.14 $\pm$ 0.96 | 63.69 $\pm$ 0.46 | -0.92 $\pm$ 0.46 | 60.25 $\pm$ 2.86 | -8.19 $\pm$ 2.86
ER∗ | 64.22 $\pm$ 0.10 | -0.25 $\pm$ 0.10 | 63.52 $\pm$ 0.20 | -1.46 $\pm$ 0.20 | 65.61 $\pm$ 0.76 | -2.86 $\pm$ 0.76
FD | 64.57 $\pm$ 0.57 | -0.51 $\pm$ 0.12 | 64.24 $\pm$ 0.73 | -1.07 $\pm$ 0.46 | 67.70 $\pm$ 0.54 | -1.98 $\pm$ 0.70
MAFED-B | 64.78 $\pm$ 0.55 | -0.34 $\pm$ 0.27 | 64.51 $\pm$ 0.36 | -1.02 $\pm$ 0.17 |  67.76 $\pm$ 0.27 |  -1.85 $\pm$ 0.61
MAFED-A |  65.00 $\pm$ 0.41 |  -0.28 $\pm$ 0.19 |  64.63 $\pm$ 0.37 |  -0.89 $\pm$ 0.21 | 67.67 $\pm$ 0.46 | -2.01 $\pm$ 0.84
Multitask∗ | 67.51 $\pm$ 1.94 | - | 67.84 $\pm$ 3.92 | - | 72.41 $\pm$ 3.75 | -
Table 1: UNITER and ViLT average accuracy and semantic backward transfer over
five task orders. ∗ results reported in Nikandrou et al. (2022).

where $\gamma\in(0,1]$ is a hyperparameter such that lower $\gamma$ values
assign more weight to deeper layers, and $\gamma=1$ weighs the losses from all
layers equally. Unless stated otherwise, we combine feature distillation with
replay since it requires no computational overhead and helps mitigate the
miscalibration of the output layer, which can negatively impact performance on
past tasks Wu et al. (2019).

##  5 Experiments

###  5.1 Baselines

We compare our methods against naive Fine Tuning (FT), where a model is
trained sequentially on each task. Our continual learning baselines include
Elastic Weight Consolidation (EWC) Kirkpatrick et al. (2017) and Experience
Replay (ER) Chaudhry et al. (2019), which have been shown to perform
competitively. Finally, we report the upper bound performance of Multitask
learning, where the model is trained on all tasks simultaneously. Note that
this is the de facto standard for instruction-tuning in VLMs Dai et al.
(2024); Liu et al. (2023).

###  5.2 Evaluation Metrics

We measure the performance of a model using three metrics. First, we report
the macro-average accuracy at the end of a training sequence,
$\mathrm{A}=\frac{1}{T}\sum_{i=1}^{T}A_{T,i}$, where $A_{T,i}$ depicts the
performance of the model on the data from task $i$ after training on final
task $T$. Additionally, we report the Semantic Backward Transfer (SBWT)
Nikandrou et al. (2022), which captures the impact of catastrophic forgetting,
weighted by the semantic similarity of the prediction and the target:

$\mathrm{SBWT}=\frac{1}{T-1}\sum_{i=1}^{T-1}S_{T,i}$ (7)

where $S_{T,i}$ is the average weighted accuracy difference for task $i$.

###  5.3 Implementation Details

We train all models using the Adam optimizer Kingma and Ba (2014) and a
learning rate schedule that follows linear decay after a warmup for 10% of the
training steps. The maximum learning rate is optimized through grid search
separately for each setting based on the performance of the finetuning method.
For UNITER and ViLT, we set the number of epochs per task to 60 with a
patience of 5. We found that VL-Pythia models reach their peak accuracy for
fewer updates, possibly because they do not use a randomly initialized
classification head. As a result, we set the maximum number of epochs to 15.
For all replay and feature distillation runs, we keep a memory of 1000
randomly selected samples per task, ensuring that the same samples are stored
across methods for the same task order. Further details about the selected
hyperparameters are listed in Section A.2.

###  5.4 Results

|  | VL-Pythia 160M | VL-Pythia 410M | VL-Pythia 1B
---|---|---|---|---
Setting | Method | Accuracy | SBWT | Accuracy | SBWT | Accuracy | SBWT
Question Types | FT | 25.98 $\pm$ 8.23 | -31.35 $\pm$ 7.27 | 63.20 $\pm$ 2.10 | -7.22 $\pm$ 1.98 | 65.52 $\pm$ 5.60 | -6.16 $\pm$ 5.42
EWC | 41.55 $\pm$ 8.31 | -8.58 $\pm$ 6.96 | 66.83 $\pm$ 2.45 | -3.85 $\pm$ 2.48 | 66.78 $\pm$ 2.98 | -5.30 $\pm$ 2.93
ER | 53.56 $\pm$ 0.72 | -5.20 $\pm$ 1.06 | 70.25 $\pm$ 1.00 | -0.52 $\pm$ 0.67 | 69.66 $\pm$ 3.34 | -2.27 $\pm$ 1.88
FD | 56.19 $\pm$ 1.59 | -3.04 $\pm$ 1.21 | 70.76 $\pm$ 0.50 | -0.22 $\pm$ 0.25 | 71.85 $\pm$ 1.07 | -0.62 $\pm$ 0.54
MAFED-B | 57.53 $\pm$ 0.76 | -2.83 $\pm$ 0.42 | 70.82$\pm$ 0.38 | -0.17 $\pm$ 0.10 | 72.19 $\pm$ 1.40 | -0.55 $\pm$ 0.80
MAFED-A |  57.65 $\pm$ 0.24 |  -2.46 $\pm$ 0.29 |  71.06 $\pm$ 0.30 |  -0.19 $\pm$ 0.28 |  72.69 $\pm$ 0.12 |  -0.10 $\pm$ 0.07
Multitask | 65.65 $\pm$0.14 | - | 71.96 $\pm$ 0.15 | - | 73.44 $\pm$ 0.18 | -
Table 2: Performance of different VL-Pythia model sizes across three task
orders.

####  5.4.1 Encoder-only Models

Table 1 reports the results across the three settings using the encoder-only
models UNITER and ViLT. Adding the feature distillation loss improves upon the
ER baseline in all settings. Although the benefits with UNITER are moderate,
as ER already achieves low forgetting, when using ViLT, FD offers substantial
accuracy gains of up to 2.6 in Question Types. Comparing the feature
distillation variants, modality-aware weighting (MAFED-A or MAFED-B)
consistently boosts performance. For UNITER and ViLT in Question Types,
equally balancing the modality losses with MAFED-B shows the best performance.
In the image-based settings with ViLT, adaptive weighting performs better.
Given the similarity ratios shown in Figure 3, these results suggest that
MAFED-B is more effective when the relative change of text and vision
representation is small, while MAFED-A is more appropriate in cases of larger
discrepancies.

####  5.4.2 Scaling to larger decoder-only VLMs

Setting | Accuracy | BWT
---|---|---
Diverse Content | 70.11 | 1.58
Taxonomy Content | 69.03 | 0.44
Question Types | 66.01 | -9.70

Table 3: Average accuracy and backward transfer for finetuning VL-Pythia 1B
across settings. We report the accuracy of three task orders on the validation
set.

As decoder-only architectures have become more widely used, we also experiment
with three model sizes of VL-Pythia (160M, 410M, 1B parameters). In our
initial results shown in Table 3, we find that larger models exhibit no
forgetting in the image-based settings of Diverse and Taxonomy Content. As a
result, we focus on more challenging Question Types setting.

Table 2 provides the results for different continual learning strategies using
the VL-Pythia variants. We observe that scaling leads to higher final accuracy
and less catastrophic forgetting similar to previous work Mirzadeh et al.
(2022); Ramasesh et al. (2022). Nevertheless, even the largest explored model
has a gap of almost 8% between naive finetuning and multitask learning.
Compared to experience replay, we find that the benefit of EWC diminishes for
larger models. As in encoder-only models, the inclusion of feature
distillation further improves performance, and modality-aware weighting of the
distillation losses is beneficial in all cases. For VL-Pythia, MAFED-A leads
to the best performance, improving the accuracy on average by +2.6 compared to
ER and +0.87 compared to FD. As mentioned in Section 5.4.1, we hypothesize
that adaptive modality weighting can be particularly effective where the text
and visual representations change more unequally. Overall, our results
indicate that stabilizing the representations from vision and language tokens
separately can improve multimodal continual learning.

##  6 Analysis

###  6.1 Distillation without Replay

Method | Accuracy | SBWT
---|---|---
FT | 62.77 | -5.27
ER | 66.18 | -3.42
FD | 72.05 | -1.00
w/o Replay | 63.53 | -4.87
MAFED-B | 72.91 | -0.16
w/o Replay | 67.66 | -3.67
MAFED-A | 72.56 | -0.24
w/o Replay | 64.14 | -4.09

Table 4: Ablation of feature distillation methods without replay using VL-
Pythia (1B) on Question Types.

In the previous section, we applied feature distillation in conjunction with
experience replay. Table 4 shows the performance when applying feature
distillation with and without experience replay on VL-Pythia (1B) for one task
order on the Question Types setting. Feature distillation improves the model
performance, while standalone modality-aware methods are competitive with
experience replay. The combination of the two approaches yields the greatest
performance with no additional cost over standalone feature distillation. Both
methods require maintaining a memory of past samples that are passed through
the current model. Therefore, applying replay puts no additional overhead on
training time and memory.

###  6.2 Distillation Layer Ablation

Figure 4: Ablation of feature distillation from a single or cumulative (+)
model layers.

Next, we explore the effectiveness of applying modality-aware feature
distillation from a single layer, as well as a subset of layers. Figure 4
illustrates the performance of VL-Pythia (1B) after applying MAFED-B and
MAFED-A every three layers. As expected, applying both methods on a deeper
layer of the model but also distilling from all previous layers yields greater
performance. However, the performance of all four variants does not increase
monotonically with the layer depth. More specifically, we observe that
distilling from layer 6 leads to performance degradation. This behavior
correlates with the results in Figure 3, where the per-modality similarities
diverge the most at layer 6 and gradually align throughout the deeper layers
of the model.

###  6.3 Modality Weights in MAFED-A

Figure 5: Language weight during MAFED-A. Note that language tokens in the
encoder family (UNITER and ViLT) are weighted similarly across the layers of
the models. For the causal VL-Pythia model, the language tokens have higher
weights.

Figure 5 shows the weight $\alpha$ placed on the distillation loss of the
language tokens using the MAFED-A method. In all models, MAFED-A assigns more
weight to language tokens. Interestingly, we observe that in encoder-only
models, the language weight progressively increases up to layer 8 and then
drops. On the other hand, in VL-Pythia (1B), more than 90% of the weight is
assigned to language tokens for all layers. We hypothesize that this is
because the model is causal, and the last token before the answer is a text
token.

##  7 Conclusion

In this paper, we argued that applying approaches that were developed with
unimodal models in mind is suboptimal for continual learning in VQA since this
ignores modality-specific learning dynamics. We empirically showcased that the
visual and the textual representations evolve at different rates – a
phenomenon that occurs in both encoder-only and decoder-only VLMs. Given this
observation, we proposed two modality-aware feature distillation approaches
that equally weigh the distillation loss from each modality or adaptively
estimate the importance of a modality based on the gradients with respect to
the inputs. We believe this is a promising direction towards closing the gap
with multitask training in multimodal continual learning.

###  7.1 Limitations & Future Work

Despite the promising results, our method has certain limitations. First,
distillation is more computationally expensive than replay, as it requires
accessing the representations from the previous model. However, compared to
established distillation methods, MAFED-B improves performance with no
overhead, while MAFED-A requires computing importance weights, which are only
updated between tasks. Furthermore, our work does not investigate the
potential effectiveness of architecture-based approaches, which could offer
greater control over the learning of each modality through novel parameter-
isolation approaches. Finally, we show that larger models exhibit less or even
no forgetting depending on the setting. Future work should explore whether
increasing the model size or the VL pretraining data Ostapenko et al. (2022)
can further decrease forgetting in VQA settings.

## Acknowledgements

This work was supported by the Edinburgh International Data Facility (EIDF)
and the Data-Driven Innovation Programme at the University of Edinburgh.

## References

  * Aljundi et al. (2018) Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018.  Memory aware synapses: Learning what (not) to forget.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 139–154.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Anthropic (2024) Anthropic. 2024.  Introducing the next generation of claude.
  * Bagus and Gepperth (2021) Benedikt Bagus and Alexander Gepperth. 2021.  An investigation of replay-based approaches for continual learning.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–9. IEEE.
  * Bai et al. (2023a) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023a.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Bai et al. (2023b) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pages 2397–2430. PMLR.
  * Buzzega et al. (2020) Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. 2020.  Dark experience for general continual learning: a strong, simple baseline.  In _NeurIPS_.
  * Chaudhry et al. (2019) Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato. 2019.  Continual learning with tiny episodic memories. _CoRR_ , abs/1902.10486.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _Computer Vision – ECCV 2020_ , pages 104–120, Cham. Springer International Publishing.
  * Cui et al. (2024) Zhenyu Cui, Yuxin Peng, Xun Wang, Manyu Zhu, and Jiahuan Zhou. 2024.  Continual vision-language retrieval via dynamic knowledge rectification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 11704–11712.
  * Dai et al. (2024) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _Advances in Neural Information Processing Systems_ , 36.
  * Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  _Advances in Neural Information Processing Systems_ , 35:16344–16359.
  * Del Chiaro et al. (2020) Riccardo Del Chiaro, Bartł omiej Twardowski, Andrew Bagdanov, and Joost van de Weijer. 2020.  Ratt: Recurrent attention to transient tasks for continual image captioning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 16736–16748. Curran Associates, Inc.
  * Delange et al. (2021) Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. 2021.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pages 1–1.
  * Dhar et al. (2019) Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. 2019.  Learning without memorizing.  In _Conference on Computer Vision and Pattern Recognition_.
  * Douillard et al. (2020) Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. 2020.  Podnet: Pooled outputs distillation for small-tasks incremental learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16_ , pages 86–102. Springer.
  * Fang et al. (2023) Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023.  Eva-02: A visual representation for neon genesis.  _arXiv preprint arXiv:2303.11331_.
  * Fernando et al. (2017) Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. 2017.  Pathnet: Evolution channels gradient descent in super neural networks. _CoRR_ , abs/1701.08734.
  * Gemini Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, et al. 2023.  Gemini: A Family of Highly Capable Multimodal Models.  _Preprint_ , arxiv:2312.11805.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Greco et al. (2019) Claudio Greco, Barbara Plank, Raquel Fernández, and Raffaella Bernardi. 2019.  Psycholinguistics meets continual learning: Measuring catastrophic forgetting in visual question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3601–3605, Florence, Italy. Association for Computational Linguistics.
  * Hou et al. (2019) Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. 2019.  Learning a unified classifier incrementally via rebalancing.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Hudson and Manning (2019) Drew A Hudson and Christopher D Manning. 2019.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6700–6709.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Jin et al. (2020) Xisen Jin, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. 2020.  Visually grounded continual learning of compositional phrases.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2018–2029, Online. Association for Computational Linguistics.
  * Kang et al. (2022) Minsoo Kang, Jaeyoo Park, and Bohyung Han. 2022.  Class-incremental learning by knowledge distillation with adaptive feature consolidation.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16050–16059. IEEE.
  * Karamcheti et al. (2024) Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. 2024.  Prismatic vlms: Investigating the design space of visually-conditioned language models.  _arXiv preprint arXiv:2402.07865_.
  * Karim et al. (2022) Nazmul Karim, Umar Khalid, Ashkan Esmaeili, and Nazanin Rahnavard. 2022.  Cnll: A semi-supervised approach for continual noisy label learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 3878–3888.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 5583–5594. PMLR.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526.
  * Kornblith et al. (2019) Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019.  Similarity of neural network representations revisited.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 3519–3529. PMLR.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123:32–73.
  * Lao et al. (2023) Mingrui Lao, Nan Pu, Yu Liu, Zhun Zhong, Erwin M. Bakker, Nicu Sebe, and Michael S. Lew. 2023.  Multi-domain lifelong visual question answering via self-critical distillation.  In _Proceedings of the 31st ACM International Conference on Multimedia_ , MM ’23, page 4747–4758, New York, NY, USA. Association for Computing Machinery.
  * Laurençon et al. (2024a) Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. 2024a.  Obelics: An open web-scale filtered dataset of interleaved image-text documents.  _Advances in Neural Information Processing Systems_ , 36.
  * Laurençon et al. (2024b) Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024b.  What matters when building vision-language models? _arXiv preprint arXiv:2405.02246_.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomáš Kočiský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  In _Advances in Neural Information Processing Systems_.
  * Lei et al. (2023) Stan Weixian Lei, Difei Gao, Jay Zhangjie Wu, Yuxuan Wang, Wei Liu, Mengmi Zhang, and Mike Zheng Shou. 2023.  Symbolic replay: Scene graph as prompt for continual learning on vqa task.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 1250–1259.
  * Li et al. (2021) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ , 34:9694–9705.
  * Li et al. (2023) Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023.  Evaluating object hallucination in large vision-language models.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
  * Li and Hoiem (2018) Zhizhong Li and Derek Hoiem. 2018.  Learning without forgetting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(12):2935–2947.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision – ECCV 2014_ , pages 740–755, Cham. Springer International Publishing.
  * Lin et al. (2021) Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. 2021.  The CLEAR benchmark: Continual LEArning on real-world imagery.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_.
  * Liu et al. (2024) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36.
  * Lomonaco and Maltoni (2017) Vincenzo Lomonaco and Davide Maltoni. 2017.  Core50: a new dataset and benchmark for continuous object recognition.  In _Proceedings of the 1st Annual Conference on Robot Learning_ , volume 78 of _Proceedings of Machine Learning Research_ , pages 17–26. PMLR.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _arXiv preprint arXiv:1908.02265_.
  * Madotto et al. (2021) Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Pascale Fung, and Zhiguang Wang. 2021.  Continual learning in task-oriented dialogue systems.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7452–7467, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pages 109–165. Elsevier.
  * Mirzadeh et al. (2022) Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. 2022.  Wide neural networks forget less catastrophically.  In _International conference on machine learning_ , pages 15699–15717. PMLR.
  * Nguyen et al. (2019) Giang Nguyen, Tae Joon Jun, Trung Tran, Tolcha Yalew, and Daeyoung Kim. 2019.  Contcap: A scalable framework for continual image captioning.  _arXiv preprint arXiv:1909.08745_.
  * Nikandrou et al. (2022) Mavina Nikandrou, Lu Yu, Alessandro Suglia, Ioannis Konstas, and Verena Rieser. 2022.  Task formulation matters when learning continually: A case study in visual question answering.  _arXiv preprint arXiv:2210.00044_.
  * OpenAI (2022) OpenAI. 2022.  Introducing ChatGPT.
  * Oquab et al. (2024) Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. 2024.  DINOv2: Learning robust visual features without supervision.  _Transactions on Machine Learning Research_.
  * Ordonez et al. (2011) Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011.  Im2text: Describing images using 1 million captioned photographs.  In _Advances in Neural Information Processing Systems_ , volume 24. Curran Associates, Inc.
  * Ostapenko et al. (2022) Oleksiy Ostapenko, Timothee Lesort, Pau Rodríguez, Md Rifat Arefin, Arthur Douillard, Irina Rish, and Laurent Charlin. 2022.  Continual learning with foundation models: An empirical study of latent replay.  In _Conference on lifelong learning agents_ , pages 60–91. PMLR.
  * Qian et al. (2023) Zi Qian, Xin Wang, Xuguang Duan, Pengda Qin, Yuhong Li, and Wenwu Zhu. 2023.  Decouple before interact: Multi-modal prompt learning for continual visual question answering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2953–2962.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Ramasesh et al. (2022) Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. 2022.  Effect of scale on catastrophic forgetting in neural networks.  In _International Conference on Learning Representations_.
  * Ratcliff (1990) Roger Ratcliff. 1990.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. _Psychological review_ , 97(2):285.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 2001–2010.
  * Serrà et al. (2018) Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. 2018.  Overcoming catastrophic forgetting with hard attention to the task.  In _ICML_ , pages 4555–4564.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565.
  * Srinivasan et al. (2022) Tejas Srinivasan, Ting-Yun Chang, Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. 2022.  Climb: A continual learning benchmark for vision-and-language tasks.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 29440–29453. Curran Associates, Inc.
  * Su et al. (2024) Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024.  Roformer: Enhanced transformer with rotary position embedding.  _Neurocomputing_ , 568:127063.
  * Sun et al. (2019) Fan-Keng Sun, Cheng-Hao Ho, and Hung yi Lee. 2019.  Lamol: Language modeling for lifelong language learning.  In _International Conference on Learning Representations_.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  _arXiv preprint arXiv:1908.07490_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Van de Ven and Tolias (2018) Gido M Van de Ven and Andreas S Tolias. 2018.  Generative replay with feedback connections as a general strategy for continual learning.  _arXiv preprint arXiv:1809.10635_.
  * Wang et al. (2020) Weiyao Wang, Du Tran, and Matt Feiszli. 2020.  What makes training multi-modal classification networks hard? In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12695–12705.
  * Wang et al. (2022) Zhen Wang, Liu Liu, Yiqun Duan, and Dacheng Tao. 2022.  Continual learning through retrieval and imagination.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 8594–8602.
  * Wu et al. (2022) Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof J Geras. 2022.  Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks.  In _International Conference on Machine Learning_ , pages 24043–24055. PMLR.
  * Wu et al. (2019) Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. 2019.  Large scale incremental learning.  In _Conference on Computer Vision and Pattern Recognition_.
  * Yoon et al. (2020) Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. 2020.  Scalable and order-robust continual learning with additive parameter decomposition.  In _International Conference on Learning Representations_.
  * Zenke et al. (2017) Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017.  Continual learning through synaptic intelligence.  In _International Conference on Machine Learning_ , pages 3987–3995. PMLR.
  * Zhang et al. (2023) Xi Zhang, Feifei Zhang, and Changsheng Xu. 2023.  Vqacl: A novel visual question answering continual learning setting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19102–19112.
  * Zhang et al. (2022) Yao Zhang, Haokun Chen, Ahmed Frikha, Yezi Yang, Denis Krompass, Gengyuan Zhang, Jindong Gu, and Volker Tresp. 2022.  Cl-crossvqa: A continual learning benchmark for cross-domain visual question answering.  _arXiv preprint arXiv:2211.10567_.

##  Appendix A Experiments

###  A.1 Pretraining VL-Pythia

We pretrain all VL-Pythia models following the LLaVA-1.5 recipe Liu et al.
(2023). The only difference is that we skip the first stage for multimodal
alignment as recent work Karamcheti et al. (2024) has shown that the two-stage
training can be redundant and the same performance can achieved when omitting
the first stage of training. Throughout VL pretraining, the vision encoder
remains frozen, while the LLM and connector parameters are trained using the
Adam optimizer Kingma and Ba (2014) with a batch size of 256 and a learning
rate of 1e-3. For all models, we used the same data to train UNITER and ViLT -
COCO Lin et al. (2014), SBU captions Ordonez et al. (2011), Visual Genome
captions Krishna et al. (2017) and Conceptual Captions 3M Sharma et al.
(2018). We perform one epoch of pretraining and keep the final checkpoint.

###  A.2 Hyperparameters

Model | Setting | Batch Size | LR | EWC $\lambda$ | FD $\gamma$
---|---|---|---|---|---
| Diverse Content | 1024 | 8e-5 | 500 | 0.8
UNITER | Taxonomy Content | 1024 | 5e-5 | 500 | 0.8
| Question Types | 512 | 5e-5 | 20K | 0.6
| Diverse Content | 1024 | 1e-5 | 500 | 1
ViLT | Taxonomy Content | 1024 | 1e-5 | 700 | 1
| Question Types | 512 | 8e-5 | 10K | 0.5
| Diverse Content | 128 | 5e-5 | - | 0.5
VL-Pythia | Taxonomy Content | 128 | 5e-5 | - | 0.5
| Question Types | 128 | 5e-5 | 10K | 0.5

Table 5: Selected hyperparameters.

We tune the hyperparameters using grid search based on the validation accuracy
of a single task order. For VL-Pythia variants, we use the same
hyperparameters for all model sizes, as we find them to perform reasonably
well. For UNITER and ViLT, we keep the batch size, learning rate (LR), and EWC
loss weight $\lambda$ reported in prior work Nikandrou et al. (2022). For the
remaining values, we perform the following grid search: lr
$\in\\{1e-5,5.e-5,8e-5,1e-4\\}$, EWC $\lambda\in\\{500,1K,5K,10K\\}$, FD
discount factor $\gamma\in[0.3,1.0]$ with a step of 0.1.