11institutetext: Koondanibha Mitra 22institutetext: Mathematics & Computer
Science Department, Eindhoven University of Technology, 5600 MB Eindhoven, The
Netherlands, 22email<EMAIL_ADDRESS>33institutetext: Qiyao Peng
44institutetext: Mathematical Institute, Faculty of Science, Leiden
University, Neils Borhweg 1, 2333 CA, Leiden, The Netherlands. 44email:
<EMAIL_ADDRESS>55institutetext: Cordula Reisch 66institutetext:
Institute for Partial Differential Equations, TU Braunschweig,
Universitaetsplatz 2, 38106 Braunschweig, Germany 66email: c.reisch@tu-
braunschweig.de

# Studying wildfire fronts using advection–diffusion–reaction models

Koondanibha Mitra0000-0002-8264-5982  Qiyao Peng0000-0002-7077-0727 and
Cordula Reisch0000-0003-1442-1474

###### Abstract

In this work, we study the propagation of wildfires using an
advection–diffusion–reaction model which also includes convective and
radiative heat loss. An existing model is discussed asensio_2002  and a
physically consistent modification of the model is proposed. Using this, the
existence of travelling waves (TWs) in the one-dimensional case is
investigated. Prior numerical studies reveal the existence of TWs reisch_2023
. Under the travelling wave ansatz and certain approximation, the model is
reduced to a semi-autonomous dynamical system with three unknowns which can be
analyzed by a shooting algorithm. It is hypothesized that under mild wind
speeds, TWs in both directions exist, and under strong tailwinds only TWs in
the direction of wind are possible. The theoretical implications are
investigated using both solvers for the PDE models and the shooting algorithm.
The results match, and unveil the dependence of the fronts on the parameters
consistent with the predictions.

##  1 Introduction

Wildfire as a natural phenomenon poses a palpable threat to our society which
has become evident in recent years. Due to climatic change, the frequency and
severity of wildfires will only increase over the next decades senande_2022 .
This makes it important to develop accurate mathematical models and analytical
tools that predict how wildfires spread. For instance, wildfires are known to
possess a front that moves with a given speed which depends heavily on the
wind speed and flammability of the vegetation. Naturally, the question arises
if its spread can be approximated by a _travelling wave_ , abbreviated as TW
henceforth. To investigate this question, in this work, we focus on continuous
advection–diffusion–reaction (ADR) models for wildfires which were studied for
example in asensio_2002 ; reisch_2023 ; burger_2020b . The ADR models describe
the fire dynamics by modeling the variation of temperature and biomass. There
exists a two-way and nonlinear coupling between these two primary variables.
In addition, the reaction term is discontinuous only playing a role in the
domain that is burning. This leads to challenges in analyzing the model and
predicting the existence of travelling waves. Having estimates about TW
solutions in terms of parameters such as wind speed will also help us to
predict the areas at risk for planning interventions and potentially necessary
evacuations.

Advection–diffusion–reaction (ADR) models for wildfires have been intensively
studied: In weber1991  TW solutions for a diffusive model for the temperature
in solid and gaseous phases are analyzed. VolpertTW1994  gives an overview of
TWs for static problems and for problems where both variables diffuse.
weber1997  analyses the travelling wave speed for a fully diffusive combustion
model with a continuous reaction function including the Arrhenius law. A major
observation of the analytical and numerical investigations is that the
spreading speed of the fire depends mostly on the fire switching ‘on’ and
‘off’ and not on the cooling down of the temperature. asensio_2002  proves the
existence and uniqueness of weak solutions for the model including an unsteady
switching function for the combustion process. Based on the existence results,
a mixed finite element method is proposed. In mandel2008  the discontinuous
combustion function is approximated by a continuous function including the
Arrhenius law and linear cooling processes. The approximation function is
chosen such that TW solutions can be observed numerically. Data assimilation
is discussed for estimating the model parameters. burger_2020b ; burger_2020
propose implicit-explicit numerical methods for simulating the ADR model. The
spread of fire is numerically studied in heterogeneous environments and under
the influence of wind.

Besides the numerical studies and the analytical proof of the existence and
uniqueness of weak solutions, results also indicate the existence of TWs for
models with one non-diffusive variable. In babak_2009 , the existence of TW
solutions was proven for a simplified model where ambient cooling effects were
not regarded. In this simplified model, there is no mechanism by which the
fire can die out and hence the temperature does not decrease. This leads to an
unphysical situation. Instead, in the current work, inspired by MitraTW2023
where TWs having similar profiles were analyzed for the spreading of biofilms,
we make several important steps toward showing the existence of TWs in a much
more general setting. In Section 2, we discuss an existing ADR model and
propose its modification. Then we approximate the nonlinear combustion term
and non-dimensionalize the system. In Section 3, a TW transformation is
proposed for the one-dimensional case leading to an ordinary differential
equation system in the wave variables. In Section 4, a shooting algorithm is
proposed to find the TW solutions directly for the approximate model.
Additionally, the dynamics of the partial and the ordinary differential
equation systems are numerically compared to verify the transformation and our
predictions.

##  2 Advection–diffusion–reaction models for wildfire

###  2.1 The existing ADR model

The model proposed in asensio_2002  and studied in reisch_2023  reads

$\left\\{\begin{aligned} \rho_{0}C\left(\frac{\partial T}{\partial
t}+\mathbf{w}\cdot\nabla T\right)&=\nabla\cdot\left[K(T)\nabla
T\right]-h(T-T_{\infty})+\Psi(t)\rho_{0}HY,\\\ \frac{\partial Y}{\partial
t}&=-\Psi(t)Y,\end{aligned}\right.$ (1)

with the temperature $T$ and the biomass $Y>0$. The model parameters are the
bulk density $\rho_{0}$, the specific heat $C$, the convection coefficient
$h$, the ambient air temperature $T_{\infty}$, the heating value $H$, and the
wind speed $\mathbf{w}$. The first term on the right denotes the combined
effect of heat diffusion and radiation, with the resulting diffusivity being

$K(T)=k+\epsilon\sigma T^{3},k:\text{heat diffusivity},\,\sigma:\text{Stefan-
Boltzman constant,}\,\epsilon:\text{scaling factor.}$

See asensio_2002  for the details. The second term is convective heat loss
with the convection constant $h>0$, and the last term denotes the heat
generated by combustion. The function $\Psi$ describes this nonlinear process
(with $\mathbbm{1}(B)$ denoting characteristic function of a measurable set
$B$)

$\Psi(t)=\Psi_{0}(T(t)):=A\mathbbm{1}(\\{T(t)\geqslant\bar{T}\\})\exp\left(-\frac{T_{\rm
ac}}{T}\right)$ (2)

with the activation temperature $T_{\rm ac}$, the ignition temperature
$\bar{T}$ and a pre-exponential factor $A>0$ arising from the Arrhenius
formula. Model (2) will henceforth be called the _extinguishing fire_ model or
_EF_ since after ignition, the fire stops if $T<\bar{T}$. Suitable boundary
conditions and initial conditions (with maximum biomass $Y_{\rm ref}>0$)
complete the problem (1)–(2).

###  2.2 Model variations

In this work, we propose a slight variation of the expression (2) for $\Psi$
which we believe to be more realistic, and which helps in the travelling wave
analysis later on:

$\displaystyle\Psi(t)=\Psi_{2}(\\{T(\tau):\tau\in[0,t]\\}):=A\mathbbm{1}\left(\bigcup_{0\leqslant\tau\leqslant
t}\\{T(\tau)\geqslant\bar{T}\\}\right)\exp\left(-\frac{T_{\rm ac}}{T}\right).$
(3)

This new expression assumes that the fire remains turned ‘on’ once it has
started at a location (after the temperature exceeds $\bar{T}$) which is in
contrast to (2) which assumes that it immediately gets extinguished if
$T<\bar{T}$. Hence, it will be called the _persistent fire_ model or _PF_
moving forward. This is more physical in our opinion and is supported by
aerial observations of wildfires. Furthermore, weber1997  assesses the higher
importance of the on-setting of combustion to the travelling wave speed
compared to the cooling processes at the end of the burning (tail region).

Purely for the sake of analysis, we would further assume

$\exp\left(-\frac{T_{\rm ac}}{T}\right)\approx\Lambda(T-T_{\infty})$ (4)

for a constant $\Lambda>0$ that can be obtained by least-square fitting in the
physically relevant temperature range, see Figure 1. This linearizes the
exponential term, making algebraic manipulation possible. Moreover, it
introduces a relatively small error in the intermediate temperature range as
$\exp(-T_{\rm ac}/T_{\infty})$ is small which implies that the zeroth order
term can be ignored.

Figure 1: The exponential function $\exp(-T_{\rm ac}/T)$ and its least-square
fitted linear approximation in the interval $300\leq T\leq 1500$. Here $T_{\rm
ac}=400$.

###  2.3 Non-dimensionalization and effective model

Observing that the diffusion, convection, and heat production terms have to be
roughly in balance for the formation and propagation of the wildfire, we
propose the following non-dimensionalization. Define the scaled variables

$\displaystyle\tilde{t}:=\frac{t}{t_{\rm
ref}},\quad\tilde{\mathbf{x}}:=\frac{\mathbf{x}}{L_{\rm ref}},\quad
u:=\frac{T-T_{\infty}}{\bar{T}-T_{\infty}},\quad v:=\frac{Y}{Y_{\rm ref}},$
(5)

with reference time $t_{\rm ref}:=\frac{C}{A\Lambda H}$ and length $L_{\rm
ref}:=\sqrt{\tfrac{k}{\rho_{0}A\Lambda H}}$ resulting in the dimensionless
quantities

$\displaystyle\tilde{h}:=\frac{h}{\rho_{0}A\Lambda
H},\quad\tilde{\mathbf{w}}:=\frac{t_{\rm ref}\,\mathbf{w}}{L_{\rm
ref}},\quad\gamma:=\frac{C(\bar{T}-T_{\infty})}{H},\quad\tilde{K}(u(T)):=\frac{K(T)}{k}.$
(6)

With the abuse of notation, dropping the $\tilde{}$ in the quantities above,
we get from (1) with (3)–(4), the ADR-PF model with linear approximation:

$\displaystyle\left\\{\begin{aligned} \partial_{t}u+\mathbf{w}\cdot\nabla
u&=\nabla\cdot[K(u)\nabla
u]-h\,u+u\,v\,\mathbbm{1}(\cup_{0\leqslant\tau\leqslant t}\\{u(\tau)\geqslant
1\\})\\\
\partial_{t}v&=-\gamma\,u\,v\,\mathbbm{1}(\cup_{0\leqslant\tau\leqslant
t}\\{u(\tau)\geqslant 1\\})\end{aligned}\right.$ (7)

##  3 One-dimensional fronts and travelling waves

In this section, we restrict our discussion to the one-dimensional case.
Hence, $\nabla$ in (7) is replaced by $\partial_{x}$. Without loss of
generality, assume wind speed $w\geq 0$. For analyzing TWs, the natural
boundary condition is to impose that the temperature reaches the atmospheric
temperature as $x\to\pm\infty$, and as a consequence, the diffusive/radiative
heat fluxes vanish. On the other hand, ahead of the fire, the biomass remains
constant at $Y_{\rm ref}$. In dimensionless terms, the boundary and initial
conditions satisfy

$\displaystyle u(\pm\infty,t)=\partial_{x}u(\pm\infty,t)=0$
$\displaystyle\forall\,t>0,$ (8a) $\displaystyle u(x,0)<1,\quad v(x,0)=1\quad$
$\displaystyle\forall\,x>0$ (8b) with $u(0,0)=1$ used to fix the fire-front.

The travelling wave ansatz in this case is the assumption that solutions of
the form

$\displaystyle u(x,t)=u(\xi(x,t)),\;\;v(x,t)=v(\xi(x,t))$ (9)

exists for the travelling wave coordinate $\xi=x-ct$ with an unknown speed
$c\not=0$. The TW ansatz transforms System (7) into

$\displaystyle\left\\{\begin{aligned}
(w-c)\frac{\mathrm{d}u}{\mathrm{d}\xi}&=\frac{\mathrm{d}}{\mathrm{d}\xi}\left(K(u)\frac{\mathrm{d}u}{\mathrm{d}\xi}\right)-h\,u+u\,v\,\mathbbm{1}(\\{\xi\leq
0\\}),\\\
c\frac{\mathrm{d}v}{\mathrm{d}\xi}&=\gamma\,u\,v\,\mathbbm{1}(\\{\xi\leq
0\\}).\end{aligned}\right.$ (10)

with the transformed boundary conditions consistent with (8)

$\displaystyle
u(\pm\infty)=\frac{\mathrm{d}u}{\mathrm{d}\xi}(\pm\infty)=0,\quad
u(\xi)<1,\quad v(\xi)=1\quad\forall\,\xi>0,\quad\text{ and }\;u(0)=1.$ (11)

Since the above conditions imply that the fire has been on only in $\xi\leq
0$, the characteristic function in (7) simply reduces to
$\mathbbm{1}(\\{\xi\leq 0\\})$ in this case.

To reduce (10) further, the total thermal energy is introduced:

$\displaystyle
z=\frac{1}{c}\int^{\infty}_{\xi}u(\tilde{\xi})\,\mathrm{d}\tilde{\xi},\quad\Rightarrow\quad
c\frac{\mathrm{d}z}{\mathrm{d}\xi}=-u.$ (12)

Inserting $z$ and the expression of $\frac{\mathrm{d}v}{\mathrm{d}\xi}$ in
(10) and integrating in $\xi$, one finds the first integral (conserved
quantity)

$\displaystyle
K(u)\frac{\mathrm{d}u}{\mathrm{d}\xi}+(c-w)u+c\,h\,z+\frac{c}{\gamma}v=D\text{
(constant independent of $\xi$)}.$ (13)

Passing the limit $\xi\to\pm\infty$, and using the boundary conditions (11)
one obtains

$\displaystyle v(-\infty)=:r,\quad z(-\infty)=\frac{1}{\gamma
h}(1-r)\quad\text{ for some }r\in(0,1).$ (14)

The $\xi\to\infty$ limit further yields $D=\frac{c}{\gamma}$. Using the above,
rearrangement of (13) yields the dynamical system for the triplet $(u,v,z)$
with unknown wave-speed $c\not=0$,

$\left\\{\begin{aligned}
K(u)\frac{du}{d\xi}&=(w-c)u-c\,h\,z+\frac{c}{\gamma}(1-v),\\\
c\frac{dv}{d\xi}&=\gamma\,u\,v\mathbbm{1}(\\{\xi\leq 0\\}),\\\
c\frac{dz}{d\xi}&=-u.\end{aligned}\right.$ (15)

The conditions (11) demand that for an unknown $r\in(0,1)$,

$\displaystyle(u,v,z)\to\begin{cases}(0,1,0)&\text{ as }\;\xi\to+\infty,\\\
\left(0,r,\tfrac{1}{\gamma h}(1-r)\right)&\text{ as
}\;\xi\to-\infty,\end{cases}$ (16)

with the $\xi$-cordinates being shifted so that the supremum of the set
$\\{\tilde{\xi}:u(\tilde{\xi})>1\\}$ is 0. The points on the right of (16) are
the equilibrium points for the semi-autonomous system (15).

The above system can be investigated numerically as a shooting problem, i.e.
solving (15) backward starting from $(0,1,0)$ for different values of $c$ to
find when the condition at $\xi\to-\infty$ is satisfied. This is precisely
what is done in the next section, see Algorithm 1. Moreover, in a future work,
we also conclude the existence of TWs by analyzing (15)–(16). Here, we
summarize our claim without proof:

######  Claim 1 (Existence of travelling waves for the ADR-PF model).

Let $h<1$ (necessary condition). Then under certain parametric constraints
involving $h$ and $\gamma$, there exists $\bar{w}>0$ such that for all
$w\in[0,\bar{w})$, at least two travelling wave solutions exist of the System
(15)–(16) with $c=c_{+}>0$ and $c=c_{-}<0$ respectively (with
$c_{+}+c_{-}>0$), whereas for $w>\bar{w}$, there exist only travelling waves
with $c=c_{+}>0$.

The duplicity of the TWs results from the directional symmetry of the system.
For example for $w=0$, the problem is completely symmetric with respect to the
transformation $x\mapsto-x$, see also the numerical results in reisch_2023 .
Moreover, we stress that no TWs exist if $h>1$, see (g)-(h) plots of Figure 2.

##  4 Numerical results

In this study, the finite-element method (FEM) together with the Strang
splitting is utilized for numerical simulation of the EF and the PF models.
The dimensional values of the parameters used in the simulation are taken from
Table 1 of reisch_2023 , i.e., $\rho_{0}=40kg/m^{3}$, $C=1kJ/(kg\cdot K)$,
$k=2kW/(m\cdot K)$, $h=4kW/(m^{3}\cdot K)$, $A=0.05s^{-1}$, $H=4000kJ/kg$, and
$T_{\rm ac}=400K$. The computational domain is $(-250,250)m$ and the time step
size is $\Delta t=0.4s$. For the initial condition, the temperature in
$(-25,25)m$ is $470K$, which is higher than the ignition temperature
$\displaystyle\bar{T}=400K$, while the rest of the domain is in ambient
temperature $T_{\infty}=300K$. In the beginning, the fuel concentration $Y$ is
$1$ over the entire domain.

Three main test cases are considered in this work for the TWs: (i) no wind
$w=0m/s$, (ii) mild wind $w=0.03m/s$, and (iii) strong wind $w=0.3m/s$. The
evolution of temperature $T$ and the fuel concentration $Y$ profiles are shown
in Figure 2 for these three test-cases. They show that the profiles
immediately develop into TWs. For the no wind case, there are two symmetric
TWs as expected. For the mild wind case, they are asymmetric with
$c_{+}+c_{-}>0$; and for strong wind, there is only one TW with $c_{+}>0$ due
to strong advection. The TWs speeds are calculated by linear regression on
points $x(t)$ ($t>0$) for which $T(x(t),t)=\bar{T}$, see Table 1. The results
support our claim in Section 3. Furthermore, a fourth case is considered where
no TWs exist even though there is no wind. It also agrees with our
predictions, since the scaled convection constant $h>1$ in this case.

(a) No wind and TWs exist

(b) No wind and TWs exist

(c) Mild wind and TWs exist

(d) Mild wind and TWs exist

(e) Strong wind and TW exists

(f) Strong wind and TW exists

(g) No wind and no TW

(h) No wind and no TW

Figure 2: The evolution of temperature $T$ (left column) and fuel $Y$ (right
column) profiles are shown over (part of) the computational domain. The
transparency of the profiles decreases as the time increases
($t\in\\{0,40,120,280,400\\}$). Here, (a)-(b) represents the no wind
($w=0\,m/s$) case; (c)-(d) mild wind ($w=0.03m/s$); (e)-(f) strong wind
($w=0.3m/s$); and (g)-(h) no wind case where TWs do not exist (dimensional
$h=19.85kW/(m^{3}\cdot K)$, dimensionless $h=1.5$).

Next, Figure 3 compares three models mentioned in Section 2, i.e. ADR-EF, ADR-
PF, and ADR-PF with approximation (4), for the three test-cases. It is
observed that the profiles for ADR-EF and ADR-PF models coincide until the
tail region, where $T<\bar{T}$, is reached. Hence the existence of TWs and
their speeds agree in both cases confirming the observations in weber1997
that the tail-region does not play a crucial role in this regard. However, the
residual fuel $Y$ differs significantly in both cases which exemplifies why
proper modeling is necessary for these cases. On the other hand, the
approximate model results in much higher peak temperatures. This is because
the exponential heat generation factor in (3) plateaus at higher temperatures,
whereas, the linear one in (4) does not. However, this approximate model still
manages to predict the existence and speed of the TWs in all three cases
revealing its usefulness as an analytical tool, see Table 1.

(a) No wind

(b) No wind

(c) Mild wind $(w=0.03)$

(d) Mild wind $(w=0.03)$

(e) Strong wind $(w=0.3)$

(f) Strong wind $(w=0.3)$

Figure 3: The comparison between profiles of ADR-EF (black solid), ADR-PF
(blue dashed), and the approximated (red dashed) models shown at various times
($t\in\\{0,40,400\\}$).

Next, we compare the PDE results of the approximated ADR-PF model (1), (3)–(4)
with solutions of the dynamical system (15) with conditions (16). To find
these ODE solutions, we use

######  Algorithm 1 (A shooting ODE solver).

For a given $c>0$, let $(u_{c},v_{c},z_{c})$ denote the solution of (15)
connecting to $(0,1,0)$ as $\xi\to\infty$ with $u_{c}(0)=1$. Define,
$d_{c}:=\inf\\{u_{c}(\xi)^{2}+(hz_{c}(\xi)-\gamma^{-1}(1-v_{c}(\xi)))^{2}:\xi<0\\}$
as the closest approach of $(u_{c},v_{c},z_{c})$ to another equilibrium point
of System (15). Then assuming that the unique minimizer of $d_{c}$ is in an
interval $[\check{c},\hat{c}]$ for some $0<\check{c}<\hat{c}$, we employ a
line-search algorithm to find $c=c_{+}$ such that $d_{c}\approx 0$.

For finding $c=c_{-}<0$, the same process is repeated taking $w^{\prime}=-w$
due to symmetry. For the actual ODE simulations, a fourth-order Runge-Kutta
scheme has been used to compute backward from $(10^{-3},1,0)$ with
$\Delta\xi=10^{-3}$, and $(\check{c},\hat{c})=(0.1,1.0)$. Figure 4 shows the
comparison of temperature profiles resulting from the PDE (approximated ADR-
PF) and the ODE solutions for the three test cases. The profiles are a close
match to each other until the orbits of the ODE solutions are near the second
equilibrium point. Table 1 shows a comparison between TW speeds for the
different solvers which differ from each other by at most $4\%$. Overall, we
find the matching to be excellent.

(a) no wind $w=0$ (b) mild wind $w=0.03m/s$ (c) strong wind $w=0.3m/s$

Figure 4: Comparison of temperature profiles between the PDE and the ODE simulations of the approximated ADR-PF model. Case | Wind speed $w$ | ADR-PF $(c_{-},c_{+})$ | Lin. Approx. $(c_{-},c_{+})$ | ODE $(c_{-},c_{+})$
---|---|---|---|---
No wind | $0$ | $(-0.1421,0.1421)$ | $(-0.1585,0.1585)$ | $(-0.15196,0.15196)$
Mild wind | $0.03$ | $(-0.1112,0.1781)$ | $(-0.1079,0.1843)$ | $(-0.116358,0.18585)$
Strong wind | $0.3$ | $({\rm NA},0.4686)$ | $({\rm NA},0.5264)$ | $({\rm NA},0.4702196)$
Table 1: Comparison of wave-speeds of the numerical solutions of the ADR-PF
model (Eqs. (1), (3), the approximated ADR-PF model (Eqs. (1), (3)–(4)), and
Algorithm 1 (ODE solver). All three test cases (no, mild, and strong wind) are
considered.

##  5 Discussion and future directions

We studied an existing advection–diffusion–reaction model for wildfire
propagation and proposed an alteration in which fire persists behind the
front. A one-dimensional version of the model was considered with an
approximation used for estimating the Arrhenius factor. Using travelling wave
ansatz, we reduced the system to a dynamical system with boundary values
prescribed. This system is easier to analyze and solve rapidly using a
shooting algorithm (Algorithm 1). It was also hypothesized that the travelling
waves emanate in both directions under mild wind conditions, but only in the
direction of wind provided the wind speed is high enough. Our predictions were
corroborated by numerical results.

The algorithm can be used to quickly predict the wildfire propagation speed
under different parametric conditions to prevent disasters. The actual proof
of the existence of the travelling waves will be elaborated in a future work.

###### Acknowledgements.

CR acknowledges funding by the Seed Funding Programme of Technische
Universität Braunschweig, 2022 Interdisciplinary Collaboration: Strengthening
Interdisciplinarity – Expanding Research Collaboration. KM was supported by
Fonds Wetenschappelijk Onderzoek (FWO) through the Junior Postdoctoral
Fellowship.

## References

  * [1] Asensio, M. I., Ferragut, L.: On a wildland fire model with radiation. International Journal for Numerical Methods in Engineering 54, 137–157 (2002). doi: 10.1002/nme.420.
  * [2] Reisch, C., Navas-Montilla, A., Özgen-Xian, I.: Analytical and numerical insights into wildfire dynamics: Exploring the advection-diffusion-reaction model. arXiv:2307.16174 (2023)
  * [3] Senande-Rivera, M., Insua-Costa, D., Miguez-Macho, G.: Spatial and temporal expansion of global wildland fire activity in response to climate change. Nat Commun 13, 1208 (2022). doi:10.1038/s41467-022-28835-2
  * [4] Bürger, R., Gavián, E., Inzunza, D., Mulet, P., Villada, L. M.: Implicit-Explicit Methods for a Convection-Diffusion-Reaction Model of the Propagation of Forest Fires. Mathematics 8, 1034 (2020) doi: 10.3390/math8061034
  * [5] Weber, R.O.: Toward a Comprehensive Wildfire Spread Model. Int. J. Wildland Fire 1, 245-248 (1991). doi: 10.1071/WF9910245
  * [6] Volpert, A.I., Volpert, V.A., Volpert, V.A.: Traveling Wave Solutions of Parabolic Systems. Translations of Mathematical Monographs 140, AMS (1994). doi: 10.1090/mmono/140
  * [7] Weber R.O., Mercer G.N., Sidhu H.S., Gray B.F.: Combustion waves for gases ($Le=1$) and solids ($Le\to\infty$). Proc. R. Soc. Lond. A. 453, 1105–1118 (1997). doi: 10.1098/rspa.1997.0062
  * [8] Mandel, J., Bennethum, L.S., Beezley, J.D., Coen, J.L., Douglas, C.C., Kim, M., Vodacek, A.: A wildland fire model with data assimilation. Mathematics and Computers in Simulation 79(3), 584-606 (2008). doi: 10.1016/j.matcom.2008.03.015.
  * [9] Bürger, R., Gavián, E., Inzunza, D., Mulet, P., Villada, L. M.:Exploring a convection-diffusion-reaction model of the propagation of forest fires: Computation of risk maps for heterogeneous environments. Mathematics 8, 1674 (2020) doi: 10.3390/math8101674
  * [10] Babak, P., Bourlioux, A., Hillen, T.: The Effect of Wind on the Propagation of an Idealized Forest Fire. SIAM Journal on Applied Mathematics 70(4) 1364–1388 (2009) doi: 10.1137/080727166.
  * [11] Mitra, K., Hughes, J.M., Sonner, S., Eberl, H.J., Dockery, J.D.: Travelling Waves in a PDE–ODE Coupled Model of Cellulolytic Biofilms with Nonlinear Diffusion. J Dyn Diff Equat (2023). https://doi.org/10.1007/s10884-022-10240-4


# Advancing Cross-domain Discriminability in Continual Learning of Vison-
Language Models

Yicheng Xu1 Yuxin Chen211footnotemark: 1 Jiahao Nie3
Yusong Wang1 Huiping Zhuang4 Manabu Okumura1
1Tokyo Institute of Technology 2University of California Berkeley
3Nanyang Technological University 4South China University of Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal contribution

###### Abstract

Continual learning (CL) with Vision-Language Models (VLMs) has overcome the
constraints of traditional CL, which only focuses on previously encountered
classes. During the CL of VLMs, we need not only to prevent the catastrophic
forgetting on incrementally learned knowledge but also to preserve the zero-
shot ability of VLMs. However, existing methods require additional reference
datasets to maintain such zero-shot ability and rely on domain-identity hints
to classify images across different domains. In this study, we propose
Regression-based Analytic Incremental Learning (RAIL), which utilizes a
recursive ridge regression-based adapter to learn from a sequence of domains
in a non-forgetting manner and decouple the cross-domain correlations by
projecting features to a higher-dimensional space. Cooperating with a
training-free fusion module, RAIL absolutely preserves the VLM’s zero-shot
ability on unseen domains without any reference data. Additionally, we
introduce Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting. In
this setting, a CL learner is required to incrementally learn from multiple
domains and classify test images from both seen and unseen domains without any
domain-identity hint. We theoretically prove RAIL’s absolute memorization on
incrementally learned domains. Experiment results affirm RAIL’s state-of-the-
art performance in both X-TAIL and existing Multi-domain Task-Incremental
Learning settings. The code will be released upon acceptance.

##  1 Introduction

Continual learning (CL) hadsell2020embracing ; de2021continual ;
wang2024comprehensive  is a crucial area in machine learning, which requires a
learner to incrementally learn new data instead of training from scratch. The
main challenge in CL is known as catastrophic forgetting
mccloskey1989catastrophic , where learning new knowledge results in the
forgetting of the old one. To this end, various CL approaches
kirkpatrick2017overcoming ; li2017learning ; rebuffi2017icarl ;
buzzega2020dark ; wang2024comprehensive  have been proposed to solve the
forgetting issue. As a typical CL setting, Class-Incremental Learning (CIL)
(Fig. 1 (a)) aims to achieve robust discriminability on all seen classes.
Despite the advancements, existing approaches mainly focus on classifying
images only from seen classes, thereby limiting the model’s generalizability.

Consequently, Zheng et al. Zheng_2023_ICCV  proposed Multi-domain Task-
Incremental Learning (MTIL), which cooperates CL with the zero-shot ability of
Vision-Language Models (VLMs) li2021align ; kim2021vilt ; radford2021learning
; li2022blip  such as CLIP radford2021learning . This integration equips
models with the ability to classify domains they have not yet encountered,
enhancing their generalizability across multiple domains (Fig. 1 (b)). Several
methods Zheng_2023_ICCV ; yu2024boosting  have been specifically designed for
MTIL, in which the model is required to retain both the incrementally learned
knowledge during CL and the zero-shot ability of VLMs. However, these methods
require a domain-identity hint to indicate the specific domain of the test
image, which is often not applicable in real-world scenarios rebuffi2017icarl
. Additionally, the use of reference datasets during training is necessary to
maintain pre-trained VLMs’ zero-shot performance.

To address the aforementioned limitations, we introduce Regression-based
Analytic Incremental Learning (RAIL), a novel approach that incrementally
learns new knowledge and performs effectively on both seen and unseen domains.
Specifically, we leverage non-linear projection functions from both primal and
dual perspectives to enhance the expressiveness of features extracted by the
pre-trained CLIP. It endows the learner with the ability to classify images in
a cross-domain label set without any domain-identity hint. In the incremental
learning process, RAIL utilizes a ridge regression-based adapter that updates
the parameters recursively. This is identical to learning on all encountered
domains at once, achieving absolute memorization on learned domains.
Additionally, we freeze the pre-trained CLIP and design a training-free fusion
module to determine whether the test data belongs to seen or unseen domains.
This strategy absolutely preserves CLIP’s zero-shot ability on unseen domains,
meeting practical requirements for models deployed in dynamic environments.

To demonstrate the effectiveness of our method, we propose Cross-domain Task-
Agnostic Incremental Learning (X-TAIL) setting as illustrated in Fig. 1 (c).
Particularly, X-TAIL requires CL methods to incrementally transfer a pre-
trained VLM to multiple domains while evaluating the model’s performance on
both seen and unseen domains. Moreover, domain hints are forbidden in X-TAIL,
making it more realistic and challenging wang2024comprehensive . As a result,
effective CL methods must classify a test image into the correct domain and
class simultaneously.

Figure 1: Comparison of different CL settings. (a) In CIL, models classify
images within all previously encountered classes. (b) In MTIL, models classify
images from both seen and unseen domains based on the given domain-identities.
(c) In X-TAIL, models classify images from both seen and unseen domains
without any domain-identity hint.

Our contributions are summarized as follows:

  * •

We propose a new CL method RAIL to incrementally adapt the pre-trained VLM to
multiple domains without forgetting both pre-trained and incrementally learned
knowledge.

  * •

To meet the practical scenario where CL methods need to sequentially learn
data from new domains and classify images across these domains, we propose a
new setting X-TAIL to evaluate the preservation of VLM’s zero-shot ability and
the adaptability to new domains.

  * •

We theoretically prove the RAIL’s absolute memorization on incrementally
learned domains and demonstrate that the zero-shot ability of the pre-trained
VLM on unseen domains is absolutely preserved.

  * •

We empirically show that the proposed method achieves state-of-the-art
performances on both existing MTIL and the novel X-TAIL settings.

##  2 Related work

Early CL methods focused on Task-Incremental Learning (TIL) hsu2018re , where
a task-id is given during testing. Subsequently, a more practical and
challenging setting of Class-Incremental Learning (CIL) rebuffi2017icarl  was
proposed, where the access to the task-id is forbidden at inference time.
Methods for CIL must therefore distinguish between all classes encountered in
learned tasks. More recently, Zheng et al. Zheng_2023_ICCV  proposed Multi-
Domain Task-Incremental Learning (MTIL), which is especially designed to
evaluate CL methods with pre-trained VLMs. In MTIL, a pre-trained VLM
continually adapts to multi-domain tasks. The performance on both seen and
unseen tasks measure the retention of both incrementally acquired and pre-
trained knowledge. However, it still requires the task-id to create specific
domain label space at inference time. Apart from them, X-TAIL combines the
challenges of both CIL and MTIL, in which the model learns new classes from
various incoming domains and distinguishes between both seen and unseen
classes without any domain-identity.

Prevailing continual learning methods include replay-based, distillation-
based, regularization-based, and architecture-based approaches
wang2024comprehensive . Replay-based methods such as iCaRL rebuffi2017icarl
typically store a small portion of the previous task data as exemplars. The
model is then trained jointly on new task data and the saved exemplars to
preserve the previous knowledge. Distillation-based methods such as LwF
li2017learning  use either weight or function regularization to transfer
knowledge from the previous model to the current model for knowledge
distillation. Regularization-based methods such as ZSCL Zheng_2023_ICCV
penalize the shift of either model parameter or feature space by adding a
regularization term to the cross-entropy loss function. To preserve the
robustness of the strong pre-trained model without access to the pre-trained
dataset, ZSCL utilizes large-scale reference datasets to regularize the
parameter space. Architecture-based methods mallya2018piggyback ; wang2022s ;
wang2024hierarchical  expand the model by constructing task-specific
parameters to avoid inter-task interference. For example, MoE-Adapters
yu2024boosting  cooperates the pre-trained CLIP with mixture of experts (MoE)
jacobs1991adaptive  to learn from different domains. By leveraging a reference
dataset to initialize a task-id indicator, it enables the model to distinguish
unseen tasks from seen ones.

The aforementioned methods either neglect the forgetting issue of pre-trained
knowledge or require multiple iterations and large-scale reference datasets
for training, making it challenging to efficiently adapt to new data in
continual learning scenarios. By contrast, RAIL employs an analytical solution
that achieves the optimum in a single epoch without additional reference data,
ensuring its efficiency.

##  3 Cross-domain task-agnostic incremental learning

###  3.1 Problem setting

We define Cross-domain Task-Agnostic Incremental Learning (X-TAIL) as follows.
Given a pre-trained VLM, the learner is required to incrementally transfer it
to $N$ different domains
$\\{D^{\left(1\right)},D^{\left(2\right)},...,D^{\left(N\right)}\\}$. Each
domain
$D^{\left(n\right)}=\\{(\mathbf{x}_{j}^{\left(n\right)},y_{j}^{\left(n\right)})\\}_{j=1}^{|D^{\left(n\right)}|}$
is available only during the $n$-th learning step. The class labels
$y_{j}^{\left(n\right)}\in C_{\text{label}}^{\left(n\right)}$ from the
incrementally learned domain $D^{\left(n\right)}$ are added to the set of seen
class labels. During inference at all steps, the learner attempts to classify
input images from any domain without the domain-identity hint. In other words,
the ground-truth label of the test image belongs to $C_{N}=C_{L}\cup C_{U}$,
where $C_{L}=\bigcup_{i=0}^{n}C_{\text{label}}^{\left(i\right)}$ is the union
of seen class labels from all previous learning steps and $C_{U}$ is the set
of unseen class labels.

Similar to the task-id in TIL, the domain-identity hint allows the learner to
classify input data within the label space of a specific domain during
evaluation. Essentially, the learner knows the domain of test images, which is
far from real-world application scenarios. For instance, the learner is
supposed to predict an image as the class of husky from all possible classes
$C_{N}=${car, bus, …, churros, donuts, …, husky, beagle, bulldog, …}. However,
if the domain-identity hint is given, the learner only needs to predict from a
limited subset $C_{\text{dog}}=${husky, beagle, bulldog, …}, which is simpler
but less realistic compared to practical applications. Therefore, we extend
our setting to a task-agnostic scenario. Specifically, the learner predicts
images from $C_{N}$, the union of any potential class labels, without any
domain hint.

###  3.2 Datasets

In X-TAIL’s cross-domain setting, the learner should encompass as extensive
data distributions as possible. Following previous work Zheng_2023_ICCV ;
yu2024boosting ; gao2024clip ; radford2021learning ; Zhou_2022 , we select 10
different image-classification datasets from different domains for our
setting: Aircraft maji2013finegrained , Caltech101 xu2016simple , DTD
Saad_2020 , EuroSAT helber2019eurosat , Flowers flowers_dataset , Food
bossard14 , MNIST 6296535 , OxfordPet oxfordpet , StanfordCars dehghan2017view
, and SUN397 xiao2010sun . CL methods under X-TAIL should discriminate images
from a total of $1,101$ classes across all domains.

###  3.3 Evaluation metrics

Figure 2: Metrics for X-TAIL setting.

We adopt the evaluation metrics from Zheng_2023_ICCV  for our setting. As
illustrated in Fig. 2, each column represents the performance on a specific
domain after each learning step, while the rows correspond to the learning
sequence. In traditional CL settings, only the results in the lower diagonal
where the learner has been exposed to the exemplars from the test domains are
measured. Nevertheless, X-TAIL extends this evaluation to cover the entire
matrix, recording performances across both learned and unlearned domains. The
_“Average”_ metric indicates the average accuracy of all learning steps across
all domains. The _“Last”_ metric, which is the average of performance across
all domains after the final learning step, reflects the learner’s adaptability
to new domains. Additionally, the average of the upper diagonal blocks,
referred to as the _“Transfer”_ metric, measures the extent to which the zero-
shot ability is preserved throughout incremental learning.

##  4 Approach

###  4.1 Motivation

While CLIP demonstrates the generalizability of zero-shot classification, it
still struggles in certain unfamiliar domains gao2024clip . Leveraging CLIP’s
robust feature extraction capabilities, linear probe offers a straightforward
approach to transfer the CLIP to these domains radford2021learning . Among
various linear solutions, Ridge Regression (RR) provides an effective
classification strategy by mapping the image features onto one-hot-label
targets toh2008between . Given a pre-trained CLIP image encoder $f_{I}$ and a
dataset $D=\\{(\mathbf{X},\mathbf{Y})\\}$, where $\mathbf{X}$ is the tensor of
training images and $\mathbf{Y}$ is the matrix of corresponding one-hot
labels, the predicted logits and the optimization problem are defined as:

$\hat{\mathbf{y}}=\mathbf{X}_{e}\mathbf{W},\quad\operatorname*{arg\,min}_{\mathbf{W}}\left\|\mathbf{Y}-\mathbf{X}_{e}\right\|_{F}^{2}+\lambda\left\|\mathbf{W}\right\|_{F}^{2},$
(1)

where $\mathbf{X}_{e}=f_{I}\left(\mathbf{X}\right)$ denotes the CLIP extracted
features, $\mathbf{W}$ is the classifier parameter, and $\lambda$ is the
regularization parameter.

In the context of X-TAIL, the classifier needs to distinguish a wide range of
classes from different domains. However, the extracted CLIP features of images
from different domains suffer from certain cross-domain correlations, leading
to limited domain discriminability. Based on Cover’s theorem
cover1965geometrical , one promising approach huang2006extreme ;
zhuang2022acil ; mcdonnell2023ranpac  to enhance the linear separability of
features is to project the features into a higher-dimensional space via some
non-linear projection function. We explore this non-linear projection function
from two perspectives:

Primal form ridge regression. Following zhuang2022acil , we use a Randomly-
initialized Hidden Layer (RHL) to project raw features to a higher dimensional
space. By explicitly defining the projection function as $\phi(\cdot)$, the
classifier parameter is determined as follows:

$\mathbf{W}=\left(\mathbf{\Phi}^{\top}\mathbf{\Phi}+\lambda\mathbf{I}\right)^{-1}\mathbf{\Phi}^{\top}\mathbf{Y},$
(2)

where $\mathbf{\Phi}=\phi\left(\mathbf{X}_{e}\right)$. In this way,
$\phi(\cdot)$ is fixed throughout the training process.

Dual form ridge regression saunders1998ridge . Instead of manually designing
the projection function, we utilize the Kernel method hofmann2008kernel  to
implicitly define $\phi(\cdot)$ based on the inner-product nature of dual form
ridge regression. Depending on the choice of kernel function, this approach
allows for an infinite projection dimension, which is unachievable through any
explicit definition. The dual form solution is defined as:

$\mathbf{\alpha}=\left(\mathbf{K}+\lambda\mathbf{I}\right)^{-1}\mathbf{Y},$
(3)

where $\mathbf{K}=\mathcal{K}\left(\mathbf{X},\mathbf{X}\right)$ denotes the
covariance kernel matrix, and $\mathcal{K}\left(\cdot,\cdot\right)$ can be any
positive-definite kernel function. The classification logits are derived as
$\mathbf{\hat{y}}=\mathcal{K}\left(f_{I}\left(\mathbf{x}_{\text{test}}\right),\mathbf{X}\right)\mathbf{\alpha}$.
Throughout the paper, we use the Radial Basis Function (RBF) kernel
han2012parameter  by default. The choice between primal and dual form ridge
regression depends on whether the system is over-determined (more equations
than unknowns) or under-determined (more unknowns than equations)
gentle2012numerical . Details on the relationships between primal and dual
ridge regression can be found in Appendix B.

Figure 3: Pearson correlation coefficients (CCs) for 10 pairs of domain-
prototypes.

Figure 4: Comparison of in-domain accuracy (%) on each domain with three
classifiers.

To empirically verify whether these non-linear projections enhances the
separability of CLIP features of images from different domains, we trained
three types of classifiers (denoted as Linear, Primal and Dual, respectively)
on 10 domains introduced in Sec. 3.2 jointly. To compare the standard linear
regression form with aforementioned two approaches, we take the averaged
weight vectors as the domain-prototypes and then calculate the inter-domain
Pearson correlation coefficients (CCs) between 10 pairs of domain-prototypes.
As shown in Fig. 3, the linear regression classifier exhibits high cross-
domain correlations. By contrast, the RHL in the primal form significantly
reduces these correlations. The implicit projection provided by the kernel
trick in the dual form enables better disentangling of different domains. We
further evaluate the in-domain accuracy, which represents the rate of
correctly classifying images into the appropriate domains. Fig. 4 shows that
the in-domain accuracy is negatively correlated to cross-domain correlations.
Both primal and dual forms demonstrate certain improvements through the
projection designs, allowing for accurate classification of images into their
respective domains without domain identity hint.

###  4.2 Regression-based analytic incremental learning

Based on the projection approaches introduced above, we propose the
Regression-based Analytic Incremental Learning (RAIL) method, which
incorporates a ridge regression-based adapter and a training-free fusion
module. The adapter progressively adapts the pre-trained CLIP to new domains,
while the training-free fusion module preserves CLIP’s zero-shot ability on
unseen domains. An overview of RAIL is illustrated in Fig. 5. The pseudo-codes
of both training and testing algorithms are provided in Appendix A.

Figure 5: RAIL Overview. (a) During inference, the fusion module utilizes the
Zero-shot logits to identify whether a test image is aligned with seen or
unseen classes. If classified as a seen class, the Fusion logits combine the
RAIL-Adapter logits and the Zero-shot logits; otherwise solely rely on the
Zero-shot logits. (b) Primal: at the $n$-th learning step, features
$\mathbf{X}_{e}$ extracted by CLIP’s image encoder are projected to higher
dimensional $\mathbf{\Phi}$ via RHL and then update the parameter $\mathbf{W}$
and memory $\mathbf{M}_{p}$ by Theorem 1. (c) Dual: features extracted by
CLIP’s image encoder update the kernel $\mathbf{K}$, parameter
$\mathbf{\alpha}$, and memory $\mathbf{M}_{d}$ by Theorem 2.

####  4.2.1 RAIL-Adapter

In the context of CL, in which data arrives progressively, we extend both
primal and dual ridge regression solutions to an incremental learning manner.
Our solutions are identical to that obtained by joint training, which achieves
absolute non-forgetting of learned knowledge. Let
$D^{(n)}=\\{\mathbf{X}^{(n)},\mathbf{Y}^{(n)}\\}$ represent the $n$-th
training set and $D^{(1:n)}=\\{\mathbf{X}^{(1:n)},\mathbf{Y}^{(1:n)}\\}$
represent the union of the training sets from the first $n$ domains. At the
$n$-th learning step, the optimization target for the joint training is
expressed as

$\operatorname*{arg\,min}_{\mathbf{W}^{\left(n\right)}}\left\|\mathbf{Y}^{(1:n)}-\mathbf{\Phi}^{\left(1:n\right)}\mathbf{W}^{\left(n\right)}\right\|_{F}^{2}+\lambda\left\|\mathbf{W}^{\left(n\right)}\right\|_{F}^{2},$
(4)

where
$\mathbf{\Phi}^{\left(1:n\right)}=\phi\left(f_{I}\left(\mathbf{X}^{\left(1:n\right)}\right)\right)$.
The objective is to obtain $\mathbf{W}^{\left(n\right)}$ that satisfies Eqn. 4
without accessing data from the previous $n-1$ domains. For primal form, we
propose to solve $\mathbf{W}^{\left(n\right)}$ recursively using
$\mathbf{W}^{\left(n-1\right)}$ and a memory matrix
$\mathbf{M}_{p}^{\left(n\right)}$. The solution is summarized as in Theorem 1.

###### Theorem 1

The parameter calculated by

$\mathbf{W}^{\left(n\right)}=\begin{bmatrix}{\mathbf{W}^{\left(n-1\right)}-\mathbf{M}_{p}^{\left(n\right)}\mathbf{\Phi}^{\left(n\right)\top}\mathbf{\Phi}^{\left(n\right)}\mathbf{W}^{\left(n-1\right)}}&\mathbf{M}_{p}^{\left(n\right)}\mathbf{\Phi}^{\left(n\right)\top}\mathbf{Y}^{\left(n\right)}\\\
\end{bmatrix}$ (5)

is an optimal solution to the optimization problem of joint training on all
$n$ domains in Eqn. 4, where $\mathbf{M}_{p}^{\left(n\right)}$ is obtained by

$\mathbf{M}_{p}^{\left(n\right)}=\mathbf{M}_{p}^{\left(n-1\right)}-\mathbf{M}_{p}^{\left(n-1\right)}\mathbf{\Phi}^{\left(n\right)\top}\left(\mathbf{I}+\mathbf{\Phi}^{\left(n\right)}\mathbf{M}_{p}^{\left(n-1\right)}\mathbf{\Phi}^{\left(n\right)\top}\right)^{-1}\mathbf{\Phi}^{\left(n\right)}\mathbf{M}_{p}^{\left(n-1\right)}.$
(6)

Similarly, the dual parameter $\mathbf{\alpha}^{\left(n\right)}$ satisfying
Eqn. 4 can be obtained based on $\mathbf{\alpha}^{\left(n-1\right)}$, an
updating kernel $\mathbf{K}^{\left(n\right)}$, and a memory matrix
$\mathbf{M}_{d}^{\left(n\right)}$. We denote the matrix
$\mathbf{C}^{\left(n\right)}$ as the concatenated one-hot label matrices of
all $n$ domains. The solution is defined in Theorem 2.

###### Theorem 2

The parameter calculated by

$\mathbf{\alpha}^{\left(n\right)}=\left(\mathbf{K}^{\left(n\right)}+\lambda\mathbf{I}\right)^{-1}\mathbf{C}^{\left(n\right)}$
(7)

is an optimal solution to the optimization problem of joint training on all
$n$ domains in Eqn. 4, where

$\mathbf{K}^{\left(n\right)}=\begin{bmatrix}\mathbf{K}^{\left(n-1\right)}&\mathcal{K}\left(\mathbf{X}_{e}^{\left(n\right)},\mathbf{M}_{d}^{\left(n-1\right)}\right)^{\top}\\\
\mathcal{K}\left(\mathbf{X}_{e}^{\left(n\right)},\mathbf{M}_{d}^{\left(n-1\right)}\right)&\mathcal{K}\left(\mathbf{X}_{e}^{\left(n\right)},\mathbf{X}_{e}^{\left(n\right)}\right)\end{bmatrix},\quad\mathbf{C}^{\left(n\right)}=\begin{bmatrix}\mathbf{C}^{\left(n-1\right)}&\mathbf{0}\\\
\mathbf{0}&\mathbf{Y}^{\left(n\right)}\end{bmatrix},$ (8)

and the memory matrix is given by
$\mathbf{M}_{d}^{\left(n\right)}=\begin{bmatrix}\mathbf{M}_{d}^{\left(n-1\right)\top}&\mathbf{X}_{e}^{\left(n\right)\top}\end{bmatrix}^{\top}$.

At each incremental learning step, the kernel matrix $\mathbf{K}$ updates
recursively along the main diagonal, preserving the correlations among class-
prototypes from all domains. During testing, the kernel covariance between the
feature of test image extracted by CLIP and memory matrix $\mathbf{M}_{d}$ is
calculated to obtain the classification logits
$\hat{\mathbf{y}}=\mathcal{K}\left(f_{I}\left(\mathbf{x}_{\text{test}}\right),\mathbf{M}_{d}\right)\mathbf{\alpha}$.
Specifically, $\mathbf{M}_{d}$ dynamically updates according to the data
stream via concatenated class-prototypes. These class-prototypes can be the
feature embeddings $\mathbf{X}_{e}$ extracted by CLIP, K-means centroids, or
Gaussian Mixture Model means. We use raw feature embeddings $\mathbf{X}_{e}$
by default, which is sufficient to validate our method. The complete proofs
for both theorems are provided in Appendix C.

####  4.2.2 RAIL-Fusion

Next, we introduce the fusion strategy, which leverages the refined knowledge
on seen domains from the RAIL-Adapter while preserving CLIP’s pre-trained
knowledge on unseen domains. To distinguish data from different domains
without any domain-identity hint, a common approach wang2022s  is to compute
domain centers from class-prototypes. The test image is first assigned to a
specific domain based on the distances to these domain centers and then
classified by a domain-specific classifier. However, this method fails when
statistics for unseen domains are unavailable.

An alternative solution is to leverage CLIP’s zero-shot ability to indicate
the domain of the test image. Despite CLIP’s strong generalization ability
across domains, certain cross-domain errors (i.e., misclassification into an
incorrect domain) persist and do not diminish during incremental learning
process. Therefore, the task of classifying images across seen domains is
delegated to the RAIL-Adapter, which significantly reduces these errors, as
discussed in Sec. 4.1. Consequently, CLIP’s zero-shot ability is only
leveraged to distinguish classes in unseen domains (i.e., Out-Of-Distribution
or OOD) from those in seen ones (i.e., In-Distribution or ID) to maintain its
performance on unseen domains. We summarize this approach as RAIL-Fusion,
which combines both CLIP’s zero-shot logits and RAIL-Adapter logits for
prediction, regardless of whether the domain of the test image is seen or
unseen.

Specifically, CLIP first makes a rough prediction based on its zero-shot
logits, i.e., the similarity scores between image embeddings and language
embeddings from the cross-domain label set $C_{N}$:

$\hat{\mathbf{y}}_{\text{zs}}=\text{Softmax}\left(f_{I}\left(\mathbf{x}_{\text{test}}\right)f_{T}\left(\text{Tokenizer}\left([P,C_{N}]\right)\right)^{\top}\right),$
(9)

where $\hat{\mathbf{y}}_{\text{zs}}$ represents the zero-shot logits, $P$
denotes the pre-defined prompt template, and $f_{T}$ and $f_{I}$ are the CLIP
text encoder and image encoder, respectively. The result determines whether
the test image aligns with the seen classes (ID) that have been encountered
during the incremental learning or with the unseen classes (OOD). If
classified as ID, the RAIL-adapter refines the rough prediction using its
incrementally learned knowledge. If classified as OOD, the rough prediction is
taken as the final prediction, fully relying on CLIP’s zero-shot ability.
Notably, our fusion strategy guarantees that OOD images correctly classified
by CLIP’s zero-shot prediction will never be misclassified as ID, thereby
absolutely preserving CLIP’s zero-shot ability on unseen domains. In addition,
to prevent the forgetting of the pre-trained knowledge of CLIP on domains with
good zero-shot performance, we combine the zero-shot logits and RAIL-Adapter
logits with a weighted sum:

$\hat{\mathbf{y}}_{\text{fs}}=\left(1-\beta\right)\hat{\mathbf{y}}_{\text{ad}}+\beta\hat{\mathbf{y}}_{\text{zs}},$
(10)

where $\hat{\mathbf{y}}_{\text{ad}}$ denotes the RAIL-Adapter logits and
$\beta$ is the fusion ratio that adjusts the influence of zero-shot prediction
on seen domains. The ablation study on $\beta$ is presented in Appendix E.2.

##  5 Experiments

We evaluate RAIL method under both X-TAIL and MTIL settings, as mentioned in
Sec. 3.2. The learning order is set alphabetically: Aircraft, Caltech101, DTD,
EuroSAT, Flowers, Food, MNIST, OxfordPet, StanfordCars, and SUN397. Additional
experiments with a random order are provided in Tab. 3 in Appendix H. To
ensure compatibility with different domains, we follow the common practice of
sampling a 16-shot training set for each domain, while using the original test
set for evaluation gao2024clip ; zhang2021tip ; zhou2022conditional ;
Zhou_2022 ; khattak2023maple . The implementation details are provided in
Appendix D.

Figure 6: Accuracy (%) on five domains changes over all learning steps.

###  5.1 Comparison results

##### Cross-domain task-agnostic incremental learning.

The performances averaged on 10 domains of RAIL and other baseline methods in
the X-TAIL setting are presented in the _Average_ column of Tab. 1. Specific
performances on each domain are provided in the columns named after each
respective domain. We evaluate RAIL in both primal and dual forms. _Zero-shot_
indicates the zero-shot performance of the pre-trained CLIP model on each
domain. _Fine-tune_ denotes the performance of fine-tuning both CLIP image and
text encoders with a joint dataset of all 10 domains, serving as a strong
baseline for comparison.

Specifically, the primal-RAIL outperforms the previous best one with a $6.4\%$
improvement in _“Transfer”_ accuracy, achieves an additional $7.7\%$ in
_“Average”_ accuracy, and gains an $8.6\%$ improvement in _“Last”_ accuracy.
The dual-RAIL further surpasses the primal one by $1.2\%$ in _“Average”_
accuracy and $3.3\%$ in _“Last”_ accuracy, while maintaining consistent
_“Transfer”_ accuracy due to the same fusion strategy. These results indicate
that RAIL has more stable transfer performance and is more robust to
catastrophic forgetting, effectively preserving both knowledge from new
domains and pre-trained knowledge. We repeat the experiments with a random
order and present the results in Tab. 3. RAIL consistently outperforms the
baselines, reaffirming the previous conclusions.

We illustrate how accuracy changes on several example domains in Fig. 6. We
observe that the accuracy of RAIL remains consistent with the zero-shot
results before learning the corresponding domain. Furthermore, RAIL exhibits
strong cross-domain discriminative capabilities. For example, once DTD is
learned, learning further new domains does not affect the accuracy on DTD.
Accuracy on certain domains, like Caltech101, even improves due to the fusion
module’s ability to reduce OOD errors by learning from more domains.

Table 1: Comparison of different CL methods on X-TAIL for each domain in terms of "Transfer", "Average", and "Last" scores (%). The best results are highlighted with bold style. Method |  Aircraft |  Caltech101 |  DTD |  EuroSAT |  Flowers |  Food101 |  MNIST |  Pets |  Cars |  Sun397 |  Average
---|---|---|---|---|---|---|---|---|---|---|---
Zero-shot |  23.5 |  76.8 |  37.3 |  36.7 |  63.6 |  84.0 |  46.7 |  86.7 |  66.1 |  63.7 |  58.5
Fine-tune |  39.6 |  93.3 |  68.2 |  89.2 |  95.4 |  85.5 |  95.1 |  84.4 |  77.4 |  72.4 |  80.1
Transfer |
LwF li2017learning  |  – |  66.6 |  26.9 |  19.5 |  51.0 |  78.4 |  26.6 |  68.9 |  35.5 |  56.1 |  47.7
WiSE-FT wortsman2022robust  |  – |  70.1 |  31.9 |  25.3 |  56.3 |  79.8 |  29.9 |  74.9 |  45.6 |  56.8 |  52.3
iCaRL rebuffi2017icarl  |  – |  71.7 |  35.0 |  43.0 |  63.4 |  86.9 |  43.9 |  87.8 |  63.7 |  60.0 |  61.7
ZSCL Zheng_2023_ICCV  |  – |  73.3 |  32.6 |  36.8 |  62.1 |  83.8 |  42.1 |  83.6 |  56.5 |  60.2 |  59.0
MoE-Adapter yu2024boosting  |  – |  71.0 |  34.9 |  19.2 |  63.0 |  86.6 |  20.0 |  87.2 |  63.7 |  58.6 |  56.0
Primal-RAIL |  – |  76.8 |  37.3 |  36.7 |  63.6 |  84.0 |  46.7 |  86.7 |  66.1 |  63.7 |  62.4
Dual-RAIL |  – |  76.8 |  37.3 |  36.7 |  63.6 |  84.0 |  46.7 |  86.7 |  66.1 |  63.7 |  62.4
Average |
LwF |  24.7 |  79.7 |  38.3 |  36.9 |  63.9 |  81.0 |  36.5 |  71.9 |  42.7 |  56.7 |  53.2
WiSE-FT |  27.1 |  76.5 |  40.9 |  31.3 |  68.7 |  81.6 |  31.4 |  74.7 |  51.7 |  58.4 |  54.2
iCaRL |  25.4 |  72.1 |  37.5 |  51.6 |  65.1 |  87.1 |  59.1 |  88.0 |  63.7 |  60.1 |  61.0
ZSCL |  36.0 |  75.0 |  40.7 |  40.5 |  71.0 |  85.3 |  46.3 |  83.3 |  60.7 |  61.5 |  60.0
MoE-Adapter |  43.6 |  77.9 |  52.1 |  34.7 |  75.9 |  86.3 |  45.2 |  87.4 |  66.6 |  60.2 |  63.0
Primal-RAIL |  42.4 |  89.8 |  55.7 |  68.5 |  84.0 |  83.3 |  65.3 |  85.8 |  67.9 |  64.5 |  70.7
Dual-RAIL |  45.3 |  89.9 |  57.6 |  68.7 |  83.9 |  85.5 |  65.2 |  88.4 |  69.4 |  65.0 |  71.9
Last |
LwF |  25.5 |  72.1 |  38.9 |  55.4 |  65.5 |  87.3 |  81.9 |  88.6 |  63.6 |  61.5 |  64.0
WiSE-FT |  21.8 |  76.8 |  42.9 |  20.8 |  77.5 |  84.9 |  30.7 |  76.6 |  75.8 |  72.5 |  58.0
iCaRL |  25.5 |  72.1 |  38.9 |  55.4 |  65.5 |  87.3 |  81.9 |  88.6 |  63.6 |  61.5 |  64.0
ZSCL |  33.1 |  75.3 |  43.5 |  35.2 |  74.6 |  87.4 |  50.4 |  84.2 |  77.3 |  73.4 |  63.4
MoE-Adapter |  43.2 |  78.7 |  57.6 |  32.8 |  79.4 |  86.0 |  86.7 |  87.8 |  78.2 |  74.2 |  70.5
Primal-RAIL |  41.7 |  94.0 |  66.0 |  86.4 |  97.2 |  82.4 |  93.1 |  83.6 |  75.0 |  71.3 |  79.1
Dual-RAIL |  45.3 |  94.2 |  69.0 |  87.0 |  97.2 |  87.2 |  93.0 |  92.4 |  82.5 |  76.3 |  82.4

##### Multi-domain task-incremental learning.

We follow the setting in yu2024boosting  to evaluate our methods on the few-
shot MTIL. In this context, the RAIL-Adapter reduces to domain-specific multi-
head classifiers with provided domain-identity during testing. The comparison
results are shown in Appendix G. RAIL consistently outperforms previous state-
of-the-art approaches on all three metrics. These results demonstrate that the
proposed non-linear projections significantly improve the separability of
features extracted by CLIP. Consequently, the ridge regression-based
classifier can effectively adapt the pre-trained model to new domains.

###  5.2 Discussion

##### Regression targets.

For VLMs, aside from using one-hot labels as the regression targets
$\mathbf{Y}$ in Eqn. 4, the text embeddings generated from class labels is
also a viable option mcdonnell2023ranpac . We compare the _“Last”_ accuracy of
10 domains using the dual RAIL-Adapter with these two different regression
targets as shown in Fig. 7(a). The results indicate that training with one-hot
labels surpasses its counterpart with text embeddings by an average of
$3.8\%$. We emphasize that using text embeddings as targets is suboptimal
compared to uniformly distributed one-hot labels. This effect is particularly
notable in domains such as Aircraft, where the _“Last”_ accuracy with one-hot
label targets outperforms that with text embedding targets by $7.5\%$. We
argue that insufficiently semantic class names (e.g., “707-320”) result in
text embeddings that are not well-dispersed in the feature space, thus
compromising the classification performance.

(a) Comparison of different regression targets.

(b) Comparison of different fusion strategies.

Figure 7: Each bar indicates the _“Last”_ accuracy (%) on each domain after
the last learning step.

##### Fusion strategies.

The CL setting associated with multiple domains is typically decomposed into
two stages: domain-identity inference and in-domain prediction
wang2024hierarchical . As discussed in Sec. 4.2.2, an intuitive strategy for
distinguishing classes from different domains in X-TAIL is to utilize CLIP’s
zero-shot prediction as a domain indicator, which then cooperates with
multiple domain-specific classifiers to perform classification within each
distinct domain. We evaluate the effectiveness of the RAIL-Fusion against this
strategy by comparing the _“Last”_ accuracy across 10 domains using the dual
RAIL-Adapter. The _“Transfer”_ accuracy remains consistent between these two
strategies.

As shown in Fig. 7(b), the RAIL-Fusion strategy outperforms the multi-
classifier approach in most domains by an average of $7.5\%$. This improvement
is due to RAIL-Fusion’s design, which focuses on distinguishing unseen classes
(OOD domain) from seen classes (ID domain), rather than identifying specific
domains (Sec. 4.2.2). This OOD detection design incrementally reduces errors
as the number of ID domains grows, making it particularly effective in
dynamically adapting to new domains. By contrast, using a domain indicator
maintains a consistent level of cross-domain errors associated for each
domain, leading to error propagation in the final prediction. This issue is
especially critical for domains with low in-domain accuracy, where any
misalignment between the domain indicator and the correct domain can
significantly impact performance. These cross-domain errors are mitigated in
the RAIL-Adapter thanks to its non-linear projection design.

##  6 Conclusion

In this work, we introduce the Cross-domain task-agnostic incremental learning
(X-TAIL) to evaluate the preservation of pre-trained knowledge and cross-
domain discriminative ability in a continual learning context. We introduce a
novel CL approach, Regression-based Analytic Incremental Learning (RAIL), to
improve the performance of pre-trained vision-language models on progressively
incoming domains, while maintaining its zero-shot ability on unseen domains.
We theoretically prove the absolute memorization on learned knowledge and show
that the fusion module inherently avoids the forgetting of VLM’s zero-shot
ability. Comprehensive experiments on both existing and proposed settings
empirically demonstrate the superiority of our method.

## References

  * [1] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu.  Embracing change: Continual learning in deep neural networks.  Trends in cognitive sciences, 24(12):1028–1040, 2020.
  * [2] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021.
  * [3] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu.  A comprehensive survey of continual learning: Theory, method and application.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
  * [4] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [5] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [6] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  * [7] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [8] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline.  Advances in neural information processing systems, 33:15920–15930, 2020.
  * [9] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You.  Preventing zero-shot transfer degradation in continual learning of vision-language models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19125–19136, October 2023.
  * [10] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  Advances in neural information processing systems, 34:9694–9705, 2021.
  * [11] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In International conference on machine learning, pages 5583–5594. PMLR, 2021.
  * [12] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [13] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International conference on machine learning, pages 12888–12900. PMLR, 2022.
  * [14] Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Dong Wang, Huchuan Lu, and You He.  Boosting continual learning of vision-language models via mixture-of-experts adapters.  arXiv preprint arXiv:2403.11549, 2024.
  * [15] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira.  Re-evaluating continual learning scenarios: A categorization and case for strong baselines.  arXiv preprint arXiv:1810.12488, 2018.
  * [16] Arun Mallya, Dillon Davis, and Svetlana Lazebnik.  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  In Proceedings of the European conference on computer vision (ECCV), pages 67–82, 2018.
  * [17] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong.  S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning.  Advances in Neural Information Processing Systems, 35:5682–5695, 2022.
  * [18] Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu.  Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality.  Advances in Neural Information Processing Systems, 36, 2024.
  * [19] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton.  Adaptive mixtures of local experts.  Neural computation, 3(1):79–87, 1991.
  * [20] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  International Journal of Computer Vision, 132(2):581–595, 2024.
  * [21] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision, 130(9):2337–2348, July 2022.
  * [22] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft, 2013.
  * [23] Xinxing Xu, Joey Tianyi Zhou, IvorW. Tsang, Zheng Qin, Rick Siow Mong Goh, and Yong Liu.  Simple and efficient learning using privileged information, 2016.
  * [24] Ahmed Ben Saad, Youssef Tamaazousti, Josselin Kherroubi, and Alexis He.  Where is the fake? patch-wise supervised gans for texture inpainting.  In 2020 IEEE International Conference on Image Processing (ICIP). IEEE, October 2020.
  * [25] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
  * [26] Alexander Mamaev.  Flowers dataset.  https://universe.roboflow.com/joseph-nelson/flowers, apr 2020.  visited on 2024-05-02.
  * [27] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101 – mining discriminative components with random forests.  In European Conference on Computer Vision, 2014.
  * [28] Li Deng.  The mnist database of handwritten digit images for machine learning research [best of the web].  IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  * [29] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012.
  * [30] Afshin Dehghan, Syed Zain Masood, Guang Shu, and Enrique. G. Ortiz.  View independent vehicle make, model and color recognition using convolutional neural network, 2017.
  * [31] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010.
  * [32] Kar-Ann Toh and How-Lung Eng.  Between classification-error approximation and weighted least-squares learning.  IEEE transactions on pattern analysis and machine intelligence, 30(4):658–669, 2008.
  * [33] Thomas M Cover.  Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition.  IEEE transactions on electronic computers, pages 326–334, 1965.
  * [34] Guang-Bin Huang, Qin-Yu Zhu, and Chee-Kheong Siew.  Extreme learning machine: theory and applications.  Neurocomputing, 70(1-3):489–501, 2006.
  * [35] Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, and Zhiping Lin.  Acil: Analytic class-incremental learning with absolute memorization and privacy protection.  Advances in Neural Information Processing Systems, 35:11602–11614, 2022.
  * [36] Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel.  Ranpac: Random projections and pre-trained models for continual learning.  Advances in Neural Information Processing Systems, 36, 2023.
  * [37] Craig Saunders, Alexander Gammerman, and Volodya Vovk.  Ridge regression learning algorithm in dual variables.  In Proceedings of the Fifteenth International Conference on Machine Learning, ICML ’98, page 515–521, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc.
  * [38] Thomas Hofmann, Bernhard Schölkopf, and Alexander J. Smola.  Kernel methods in machine learning.  The Annals of Statistics, 36(3):1171 – 1220, 2008.
  * [39] Shunjie Han, Cao Qubo, and Han Meng.  Parameter selection in svm with rbf kernel function.  In World Automation Congress 2012, pages 1–4. IEEE, 2012.
  * [40] James E Gentle.  Numerical linear algebra for applications in statistics.  Springer Science & Business Media, 2012.
  * [41] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Tip-adapter: Training-free clip-adapter for better vision-language modeling.  arXiv preprint arXiv:2111.03930, 2021.
  * [42] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816–16825, 2022.
  * [43] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Maple: Multi-modal prompt learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023.
  * [44] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7959–7971, 2022.
  * [45] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.

## Appendix

##  Appendix A Algorithm details

In this section, we summarize the training and testing procedures of RAIL in
Algorithm 1 and 2, respectively.

Algorithm 1 RAIL training

$N$ domains $\\{D^{\left(1\right)},...,D^{\left(N\right)}\\}$, pre-trained
CLIP model $\\{f_{I},f_{T}\\}$

Ridge regression parameter $\lambda$, random projection function
$\phi\left(\cdot\right)$, kernel function
$\mathcal{K}\left(\cdot,\cdot\right)$

Dataset 1 learning

for training batches in $D^{\left(1\right)}$ do

Extract features
$\mathbf{X}_{e}^{\left(1\right)}=f_{I}\left(\mathbf{X}^{\left(1\right)}\right)$

if Primal: then

Initialize the memory matrix $\mathbf{M}_{p}^{\left(1\right)}$ using Eqn. 15

Obtain the primal parameter $\mathbf{W}^{\left(1\right)}$ using Eqn. 2

end if

if Dual: then

Initialize the memory matrix $\mathbf{M}_{d}^{\left(1\right)}$, label matrix
$\mathbf{C}^{\left(1\right)}$ & kernel $\mathbf{K}^{\left(1\right)}$ by
Theorem 2

Obtain the dual parameter $\mathbf{\alpha}^{\left(1\right)}$ using Eqn. 7

end if

end for

Incremental learning

for $D^{\left(n\right)}$ in $\\{D^{\left(2\right)},...,D^{\left(N\right)}\\}$
do

for training batches in $D^{\left(1\right)}$ do

Extract features
$\mathbf{X}_{e}^{\left(n\right)}=f_{I}\left(\mathbf{X}^{\left(n\right)}\right)$

if Primal: then

Update the memory matrix $\mathbf{M}_{p}^{\left(n\right)}$ using Eqn.6

Update the primal parameter $\mathbf{W}^{\left(n\right)}$ using Eqn. 5

end if

if Dual: then

Update the memory matrix $\mathbf{M}_{d}^{\left(n\right)}$, label matrix
$\mathbf{C}^{\left(n\right)}$ & kernel $\mathbf{K}^{\left(n\right)}$ by
Theorem 2

Update the dual parameter $\mathbf{\alpha}^{\left(n\right)}$ using Eqn. 7

end if

end for

end for

Algorithm 2 RAIL testing

Test dataset $D_{\text{test}}$, CLIP model $\\{f_{I},f_{T}\\}$, trained RAIL-
Adapter $R_{ad}\left(\cdot\right)$, cross-domain label set $C_{N}$

for $\mathbf{x}_{\text{test}}\in D_{\text{test}}$ do

Obtain the rough prediction
$y^{*}_{\text{zs}}={\operatorname{argmax}}\,\hat{\mathbf{y}}_{zs}$ by Eqn. 9

if $y^{*}_{\text{zs}}\in C_{L}$ then

Obtain RAIL-adapter logits
$\hat{\mathbf{y}}_{ad}=R_{ad}\left(f_{I}\left(\mathbf{x}_{\text{test}}\right)\right)$

Obtain predicted class based on fusion logits
$y^{*}={\operatorname{argmax}}\,\hat{\mathbf{y}}_{fs}$ by Eqn. 10

else

Regard $y^{*}_{\text{zs}}$ as the final prediction

end if

end for

##  Appendix B Connection between primal & dual ridge regression

In this section, we introduce the connection between primal and dual forms of
ridge regression.

Based on the identity
$(\mathbf{P}^{-1}+\mathbf{B}^{\top}\mathbf{R}^{-1}\mathbf{B})^{-1}\mathbf{B}^{\top}\mathbf{R}^{-1}=\mathbf{P}\mathbf{B}^{\top}(\mathbf{B}\mathbf{P}\mathbf{B}^{\top}+\mathbf{R})^{-1}$,
the solution of ridge regression is given by:

$\mathbf{W}=(\mathbf{\Phi}^{\top}\mathbf{\Phi}+\lambda\mathbf{I}_{d})^{-1}\mathbf{\Phi}^{\top}\mathbf{Y}=\mathbf{\Phi}^{\top}(\mathbf{\Phi}\mathbf{\Phi}^{\top}+\lambda\mathbf{I}_{n})^{-1}\mathbf{Y},$
(11)

where the former solution is based on the outer-product of data and the latter
one is based on the inner-product of data. Parameter $\mathbf{W}$ can thus be
rewritten as: $\mathbf{W}=\mathbf{\Phi}^{\top}\mathbf{\alpha}$ with
$\mathbf{\alpha}=(\mathbf{\Phi}\mathbf{\Phi}^{\top}+\lambda\mathbf{I}_{n})^{-1}\mathbf{Y}$.
In this way, the solution $\mathbf{W}$ is interpreted to lie in the span of
the sample-cases, even if the dimensionality of the projected features
$\mathbf{\Phi}$ is larger than the number of samples.

Utilizing the kernel method, we never actually require access to the explicit
features $\mathbf{\Phi}$, which could be of indefinite dimensions. We obtain
the prediction with given data $\mathbf{x}$ by projecting it onto the solution
$\mathbf{W}$,

$\mathbf{\hat{y}}=\phi\left(\mathbf{x}\right)\mathbf{W}=\phi\left(\mathbf{x}\right)\mathbf{\Phi}^{\top}(\mathbf{\Phi}\mathbf{\Phi}^{\top}+\lambda\mathbf{I}_{n})^{-1}\mathbf{Y}=\mathcal{K}\left(\mathbf{x},\mathbf{X}\right)\mathbf{\alpha},$
(12)

where
$\mathcal{K}\left(\mathbf{x}_{i},\mathbf{x}_{j}\right)=\phi\left(\mathbf{x}_{i}\right)^{\top}\phi\left(\mathbf{x}_{j}\right)$.
What we require here is the choice of kernel function
$\mathcal{K}\left(\cdot,\cdot\right)$ instead of explicitly defining the
projection function.

##  Appendix C Proof of Theorems

In this section, we provide two mathematical proofs for both Theorem 1 and
Theorem 2.

First, we prove the Theorem 1 from the solution of joint training on $n$
datasets with primal ridge regression:

$\mathbf{W}^{\left(n\right)}=\left(\mathbf{\Phi}^{\left(1:n\right)\top}\mathbf{\Phi}^{\left(1:n\right)}+\lambda\mathbf{I}\right)^{-1}\mathbf{\Phi}^{\left(1:n\right)\top}\mathbf{Y}^{\left(1:n\right)}.$
(13)

By decoupling the $n$-th data from previous datasets, the
$\mathbf{W}^{\left(n\right)}$ can be written as:

$\displaystyle\mathbf{W}^{\left(n\right)}$
$\displaystyle=\left(\begin{bmatrix}{\mathbf{\Phi}^{\left(1:n-1\right)\top}}&{\mathbf{\Phi}^{\left(n\right)\top}}\end{bmatrix}\begin{bmatrix}{\left(\mathbf{\Phi}^{\left(1:n-1\right)}\right)}\\\
{\left(\mathbf{\Phi}^{\left(n\right)}\right)}\end{bmatrix}+\lambda\mathbf{I}\right)^{-1}\begin{bmatrix}{\mathbf{\Phi}^{\left(1:n-1\right)\top}}&{\mathbf{\Phi}^{\left(n\right)\top}}\end{bmatrix}\begin{bmatrix}\mathbf{Y}^{\left(1:n-1\right)}&\mathbf{0}\\\
\mathbf{0}&\mathbf{Y}^{\left(n\right)}\end{bmatrix}$ (14)
$\displaystyle=\left({\mathbf{\Phi}^{\left(1:n-1\right)\top}}{\mathbf{\Phi}^{\left(1:n-1\right)}}+\lambda\mathbf{I}+{\mathbf{\Phi}^{\left(n\right)\top}}{\mathbf{\Phi}^{\left(n\right)}}\right)^{-1}\begin{bmatrix}{\mathbf{\Phi}^{\left(1:n-1\right)\top}}\mathbf{Y}^{\left(1:n-1\right)}&{\mathbf{\Phi}^{\left(n\right)\top}}\mathbf{Y}^{\left(n\right)}\\\
\end{bmatrix}.$

We introduce the memory matrix as in the following definition:

$\mathbf{M}_{p}^{\left(n\right)}=\left(\mathbf{\Phi}^{\left(1:n\right)\top}\mathbf{\Phi}^{\left(1:n\right)}+\lambda\mathbf{I}\right)^{-1},$
(15)

which is the matrix inversion term of Eqn 13.
Noticing that
$\mathbf{M}_{p}^{\left(n-1\right)}=\left({\mathbf{\Phi}^{\left(1:n-1\right)\top}}{\mathbf{\Phi}^{\left(1:n-1\right)}}+\lambda\mathbf{I}\right)^{-1}$,
by Woodbury matrix identity where
$\left(\mathbf{A}+\mathbf{UCV}\right)^{-1}=\mathbf{A}^{-1}-\mathbf{A}^{-1}\mathbf{U}\left(\mathbf{C}^{-1}+\mathbf{V}\mathbf{A}^{-1}\mathbf{U}\right)^{-1}\mathbf{V}\mathbf{A}^{-1}$
and treating $\mathbf{M}_{p}^{\left(n-1\right)}$ as $\mathbf{A}^{-1}$, the
memory at $n$-th step can be further defined as a recursive solution:

$\mathbf{M}_{p}^{\left(n\right)}=\mathbf{M}_{p}^{\left(n-1\right)}-\mathbf{M}_{p}^{\left(n-1\right)}\mathbf{\Phi}^{\left(n\right)\top}\left(\mathbf{I}+\mathbf{\Phi}^{\left(n\right)}\mathbf{M}_{p}^{\left(n-1\right)}\mathbf{\Phi}^{\left(n\right)\top}\right)^{-1}\mathbf{\Phi}^{\left(n\right)}\mathbf{M}_{p}^{\left(n-1\right)}.$
(16)

Thus, the parameter $\mathbf{W}^{\left(n\right)}$ is derived as

$\mathbf{W}^{\left(n\right)}=\begin{bmatrix}\mathbf{M}_{p}^{\left(n\right)}{\mathbf{\Phi}^{\left(1:n-1\right)\top}}\mathbf{Y}^{\left(1:n-1\right)}&\mathbf{M}_{p}^{\left(n\right)}{\mathbf{\Phi}^{\left(n\right)\top}}\mathbf{Y}^{\left(n\right)}\end{bmatrix}.$
(17)

Denotes the left submatrix
$\mathbf{M}_{p}^{(n)}\mathbf{\Phi}^{(1:n-1)\top}\mathbf{Y}^{(1:n-1)}$ as
$\mathbf{H}$. By substituting Eqn 16 into 17,

$\mathbf{H}=\mathbf{W}^{\left(n-1\right)}-\mathbf{M}_{p}^{(n-1)}\mathbf{\Phi}^{(n)\top}\left(\mathbf{I}+\mathbf{\Phi}^{(n)}\mathbf{M}_{p}^{(n-1)}\mathbf{\Phi}^{(n)\top}\right)^{-1}\mathbf{\Phi}^{(n)}\mathbf{W}^{\left(n-1\right)}.$
(18)

Based on the identity of
$\left(\mathbf{I}+\mathbf{P}\right)^{-1}=\mathbf{I}-\left(\mathbf{I}+\mathbf{P}\right)^{-1}\mathbf{P}$,
it is further derived as:

$\mathbf{H}=\mathbf{W}^{\left(n-1\right)}-\mathbf{M}_{p}^{(n)}\mathbf{\Phi}^{(n)\top}\mathbf{\Phi}^{(n)}\mathbf{W}^{\left(n-1\right)}.$
(19)

Thus,

$\mathbf{W}^{\left(n\right)}=\begin{bmatrix}{\mathbf{W}^{\left(n-1\right)}-\mathbf{M}_{p}^{\left(n\right)}\mathbf{\Phi}^{\left(n\right)\top}\mathbf{\Phi}^{\left(n\right)}\mathbf{W}^{\left(n-1\right)}}&\mathbf{M}_{p}^{\left(n\right)}\mathbf{\Phi}^{\left(n\right)\top}\mathbf{Y}^{\left(n\right)}\\\
\end{bmatrix}.$ (20)

The Theorem 1 is proved.

Next, we prove the Theorem 2 as follows. We use the few-shot features
$\mathbf{X}_{e}$ as the class-prototypes as default. The solution of joint
training on $n$ datasets with dual form ridge regression is shown as:

$\mathbf{\alpha}^{\left(n\right)}=\left(\mathcal{K}\left(\mathbf{X}_{e}^{\left(1:n\right)},\mathbf{X}_{e}^{\left(1:n\right)}\right)+\lambda\mathbf{I}\right)^{-1}\mathbf{Y}^{\left(1:n\right)}.$
(21)

Define the memory matrix $\mathbf{M}_{d}$ as the concatenation of class-
prototypes from learned domains, the memory matrix at $n$-th step is obtained
by:

$\mathbf{M}_{d}^{\left(n\right)}=\begin{bmatrix}\mathbf{M}_{d}^{\left(n-1\right)}&\mathbf{X}_{e}^{\left(n\right)}\end{bmatrix}.$
(22)

The kernel matrix at $n$-th step for $\mathbf{\alpha}^{\left(n\right)}$ can be
partitioned as:

$\displaystyle\mathbf{K}^{\left(n\right)}$
$\displaystyle=\mathcal{K}\left(\mathbf{X}_{e}^{\left(1:n\right)},\mathbf{X}_{e}^{\left(1:n\right)}\right)$
(23)
$\displaystyle=\begin{bmatrix}\mathcal{K}\left(\mathbf{X}_{e}^{\left(1:n-1\right)},\mathbf{X}_{e}^{\left(1:n-1\right)}\right)&\mathcal{K}\left(\mathbf{X}_{e}^{\left(n\right)},\mathbf{M}_{d}^{\left(n-1\right)}\right)^{\top}\\\
\mathcal{K}\left(\mathbf{X}_{e}^{\left(n\right)},\mathbf{M}_{d}^{\left(n-1\right)}\right)&\mathcal{K}\left(\mathbf{X}_{e}^{\left(n\right)},\mathbf{X}_{e}^{\left(n\right)}\right)\end{bmatrix}$
$\displaystyle=\begin{bmatrix}\mathbf{K}^{\left(n-1\right)}&\mathcal{K}\left(\mathbf{X}_{e}^{\left(n\right)},\mathbf{M}_{d}^{\left(n-1\right)}\right)^{\top}\\\
\mathcal{K}\left(\mathbf{X}_{e}^{\left(n\right)},\mathbf{M}_{d}^{\left(n-1\right)}\right)&\mathcal{K}\left(\mathbf{X}_{e}^{\left(n\right)},\mathbf{X}_{e}^{\left(n\right)}\right)\end{bmatrix}.$

It indicates that the kernel matrix updates recursively along the diagonal by
kernel matrices of the intra-domain covariance within
$\mathbf{X}_{e}^{\left(n\right)}$ and the inter-domain covariance between
$\mathbf{X}_{e}^{\left(n\right)}$ and the memory
$\mathbf{M}_{d}^{\left(n-1\right)}$. The kernel matrix $\mathbf{K}$ therefore
memorizes all the covariance information of learned domains. We further denote
$\mathbf{Y}^{\left(1:n\right)}$ as $\mathbf{C}^{\left(n\right)}$, which is
updated by:

$\mathbf{C}^{\left(n\right)}=\begin{bmatrix}\mathbf{C}^{\left(n-1\right)}&\mathbf{0}\\\
\mathbf{0}&\mathbf{Y}^{\left(n\right)}\end{bmatrix},$ (24)

where the matrix $\mathbf{Y}^{\left(n\right)}$ consists of one-hot labels that
are disjoint with those in previous $n-1$ domains. In this way, the parameter
$\mathbf{\alpha}^{\left(n\right)}$ is solved by updating
$\mathbf{K}^{\left(n\right)}$ and $\mathbf{C}^{\left(n\right)}$, resulting in
an identical solution to the one of joint learning in Eqn. 21. Thus, the
Theorem 2 is proved.

##  Appendix D Implementation details

In this section, we introduce the implementation details, including model
configuration, hardware setup, and hyperparamter selection. We use the pre-
trained CLIP [12] model of the ViT-B/16 image encoder [45]. All the results
are conducted on Ubuntu 20.04 with Intel Core i9-13900K CPU with a single RTX
4090Ti GPU by the average of 3 runs. We conduct a grid search for the
regularization parameter $\lambda$ over the range ${10^{-6},10^{-5},...,1}$
and the RBF kernel bandwidth over the range ${10^{-6},10^{-5},...,10}$. The
optimal values are determined by minimizing the regression error on the
validation set of the first domain, without access to future domains. These
parameters are then fixed for all subsequent learning steps. We use the
simplest prompt template _“A photo of a{}.”_ for generalization across
different domains.

##  Appendix E Ablation studies

In this section, we conduct two ablation studies to observe the average
performance on 10 domains w.r.t. the hidden dimension of RHL and the fusion
ratio, respectively.

###  E.1 RHL dimension

We first ablate the hidden dimension of RHL in primal RAIL over the _“Last”_
accuracy in Fig. 8. It is evident that an increase in the RHL dimension
correlates with improved adapter’s accuracy. Specifically, increasing the
dimension from 1k to 10k leads to notable improvements (from 73.6% to 79.0%).
However, beyond the 10k threshold, the gain in accuracy becomes saturated. The
dimensions of 15k and 20k result the same performance of 79.1%. Considering
the computational cost associated with higher dimensions, we set the RHL
dimension to 15k as default in the experiments.

Figure 8: RHL dimension vs. _“Last”_ accuracy (%) averaged on 10 domains.

###  E.2 Fusion ratio

Figure 9: Fusion ratio vs. _“Average”_ and _“Last”_ accuracy (%) averaged on
10 domains.

Additionally, we conduct an ablation study of fusion ratio $\beta$ in terms of
_“Average”_ and _“Last”_ accuracy. The _“Transfer”_ accuracy is not considered
here since $\beta$ does not influence it. From Fig. 9, we observe that the
best ratio is $0.8$ for both _“Average”_ and _“Last”_ scores. Note that when
$\beta$ equals to $1$, the performance on seen domains fully relies on the
RAIL-Adapter. The _“Last”_ accuracy with $\beta=0.8$ surpasses the one of pure
RAIL-Adapter performance ($\beta=1$) by 1.9% and 1.3% in the primal and dual
RAIL, respectively. This result verifies our claim that the cooperation with
CLIP’s generalization ability can preserve the performance on its confident
domains (already having good zero-shot performance) and avoid overfitting on
limited domain exemplars.

##  Appendix F Detailed performance of RAIL with order-I

(a) Primal-RAIL

(b) Dual-RAIL

Figure 10: Accuracy (%) of Primal-RAIL and Dual-RAIL in the X-TAIL setting
with order-I. Each row denotes the performance on every domain after learning
on one domain.

In this section, we visualize the performance on each domain after every
learning step in Fig. 10. The upper-diagonal represents the performance before
learning on the corresponding domain, which remains consistent with the zero-
shot performance thanks to RAIL’s absolute memorization of the zero-shot
ability on unseen domains. On the other hand, the performance after learning a
specific domain (lower-diagonal) does not degrades. Performance on some
domains even improves with the learning step (e.g., Caltech-101), benefiting
from our RAIL fusion module design, which reduces OOD errors as more domains
are learned, thereby enhancing overall accuracy.

##  Appendix G Comparison of different methods on few-shot MTIL setting

In this section, we evaluate the performance of RAIL in 5-shot MTIL setting,
comparing it against the performance of baselines reported in [14]. In the
context of MTIL, where the domain-identity is provided during testing, RAIL is
reduced to the structure of multiple domain-specific classifiers trained on
each domain separately. The domain-identity guides the test image to the
corresponding classifier for within-domain prediction. As shown in 2, RAIL
still surpasses all baseline methods in both primal and dual forms. This
superior performance can be attributed to the non-linear projection, which
significantly enhances the separability of features extracted by CLIP.

Table 2: Comparison with state-of-the-art methods on 5-shot MTIL setting in terms of "Transfer", "Average", and "Last" scores (%). The best results are highlighted with bold style. Method |  Aircraft |  Caltech101 |  CIFAR100 |  DTD |  EuroSAT |  Flower |  Food101 |  MNIST |  Pets |  Cars |  Sun397 |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---
Zero-shot |  24.3 |  88.4 |  68.2 |  44.6 |  54.9 |  71.0 |  88.5 |  59.6 |  89.0 |  64.7 |  65.2 |  65.3
Fine-tune |  30.6 |  93.5 |  76.8 |  65.1 |  91.7 |  92.9 |  83.3 |  96.6 |  84.9 |  65.4 |  71.3 |  77.5
Transfer |  |
LwF |  – |  72.1 |  49.2 |  35.9 |  44.5 |  41.1 |  66.6 |  50.5 |  69.0 |  19.0 |  51.7 |  50.0
LwF-VR |  – |  82.2 |  62.5 |  40.1 |  40.1 |  56.3 |  80.0 |  60.9 |  77.6 |  40.5 |  60.8 |  60.1
WiSE-FT |  – |  77.6 |  60.0 |  41.3 |  39.4 |  53.0 |  76.6 |  58.1 |  75.5 |  37.3 |  58.2 |  57.7
ZSCL |  – |  84.0 |  68.1 |  44.8 |  46.8 |  63.6 |  84.9 |  61.4 |  81.4 |  55.5 |  62.2 |  65.3
MoE-Adapter |  – |  87.9 |  68.2 |  44.1 |  48.1 |  64.7 |  88.8 |  69.0 |  89.1 |  64.5 |  65.1 |  68.9
Primal-RAIL |  – |  88.4 |  68.2 |  44.6 |  54.9 |  71.0 |  88.5 |  59.6 |  89.0 |  64.7 |  65.2 |  69.4
Dual-RAIL |  – |  88.4 |  68.2 |  44.6 |  54.9 |  71.0 |  88.5 |  59.6 |  89.0 |  64.7 |  65.2 |  69.4
Average |  |
LwF |  23.5 |  77.4 |  43.5 |  41.7 |  43.5 |  52.2 |  54.6 |  63.4 |  68.0 |  21.3 |  52.6 |  49.2
LwF-VR |  24.9 |  89.1 |  64.2 |  53.4 |  54.3 |  70.8 |  79.2 |  66.5 |  79.2 |  44.1 |  61.6 |  62.5
WiSE-FT |  32.0 |  87.7 |  61.0 |  55.8 |  68.1 |  69.3 |  76.8 |  71.5 |  77.6 |  42.0 |  59.3 |  63.7
ZSCL |  28.2 |  88.6 |  66.5 |  53.5 |  56.3 |  73.4 |  83.1 |  56.4 |  82.4 |  57.5 |  62.9 |  64.4
MoE-Adapter |  30.0 |  89.6 |  73.9 |  58.7 |  69.3 |  79.3 |  88.1 |  76.5 |  89.1 |  65.3 |  65.8 |  71.4
Primal-RAIL |  32.9 |  94.5 |  69.9 |  58.1 |  71.8 |  84.4 |  88.5 |  70.4 |  89.0 |  66.1 |  65.7 |  71.9
Dual-RAIL |  36.0 |  94.2 |  70.9 |  58.8 |  70.6 |  84.3 |  88.5 |  70.3 |  89.7 |  66.5 |  65.8 |  72.3
Last |  |
LwF |  22.1 |  58.2 |  17.9 |  32.1 |  28.1 |  66.7 |  46.0 |  84.3 |  64.1 |  31.5 |  60.1 |  46.5
LwF-VR |  22.9 |  89.9 |  59.3 |  57.1 |  57.6 |  79.2 |  78.3 |  77.7 |  83.6 |  60.1 |  69.8 |  66.9
WiSE-FT |  30.8 |  88.9 |  59.6 |  60.3 |  80.9 |  81.7 |  77.1 |  94.9 |  83.2 |  62.8 |  70.0 |  71.9
ZSCL |  26.8 |  88.5 |  63.7 |  55.7 |  60.2 |  82.1 |  82.6 |  58.6 |  85.9 |  66.7 |  70.4 |  67.4
MoE-Adapter |  30.1 |  89.3 |  74.9 |  64.0 |  82.3 |  89.4 |  87.1 |  89.0 |  89.1 |  69.5 |  72.5 |  76.1
Primal-RAIL |  32.9 |  95.1 |  70.3 |  63.2 |  81.5 |  95.6 |  88.5 |  89.7 |  89.0 |  72.5 |  71.0 |  77.2
Dual-RAIL |  36.0 |  94.8 |  71.5 |  64.1 |  79.5 |  95.3 |  88.5 |  89.4 |  91.5 |  74.6 |  71.3 |  77.9

##  Appendix H Comparison of different methods on X-TAIL with order II.

In this section, we compare different methods in X-TAIL setting with a random
order: StanfordCars, Aircraft, OxfordPet, Food, SUN397, MNIST, Flowers, DTD,
Caltech101, EuroSAT. As shown in Tab. 3, our method again outperforms previous
methods on all metrics, reinforcing the conclusions presented in Sec. 5.1.

Table 3: Comparison of different CL methods on X-TAIL for each domain with order II in terms of "Transfer", "Average", and "Last" scores (%). The best results are highlighted with bold style. Method |  Cars |  Aircraft |  Pets |  Food |  SUN397 |  MNIST |  Flower |  DTD |  Caltech101 |  EuroSAT |  Average
---|---|---|---|---|---|---|---|---|---|---|---
Zero-shot |  66.1 |  23.5 |  86.7 |  84.0 |  63.7 |  46.7 |  63.6 |  37.3 |  76.8 |  36.7 |  58.5
Fine-tune |  77.4 |  39.6 |  84.4 |  85.5 |  72.4 |  95.1 |  95.4 |  68.2 |  93.3 |  89.2 |  80.1
Transfer |
LwF |  – |  20.0 |  74.1 |  79.6 |  58.1 |  34.1 |  48.9 |  27.7 |  64.4 |  15.1 |  46.9
WiSE-FT |  – |  21.3 |  79.5 |  83.3 |  61.0 |  39.9 |  56.5 |  29.6 |  68.0 |  20.8 |  51.1
ZSCL |  – |  23.0 |  84.3 |  87.2 |  63.0 |  42.1 |  65.2 |  34.6 |  71.4 |  40.9 |  56.9
MoE-Adapter |  – |  17.1 |  87.2 |  87.5 |  58.4 |  12.6 |  65.5 |  35.9 |  70.0 |  17.9 |  50.2
Primal-RAIL |  – |  23.5 |  86.7 |  84.0 |  63.7 |  46.7 |  63.5 |  37.3 |  76.8 |  36.7 |  57.7
Dual-RAIL |  – |  23.5 |  86.7 |  84.0 |  63.7 |  46.7 |  63.5 |  37.3 |  76.8 |  36.7 |  57.7
Average |
LwF |  49.0 |  27.4 |  69.7 |  83.0 |  65.7 |  42.2 |  63.5 |  33.1 |  68.5 |  17.5 |  52.0
WiSE-FT |  57.9 |  29.6 |  77.8 |  85.4 |  68.0 |  51.6 |  69.3 |  35.5 |  71.0 |  23.0 |  56.9
ZSCL |  74.4 |  36.4 |  86.7 |  88.7 |  68.9 |  50.0 |  75.1 |  40.1 |  72.5 |  43.7 |  63.6
MoE-Adapter |  74.4 |  38.6 |  87.7 |  87.3 |  67.9 |  50.6 |  76.5 |  43.7 |  72.3 |  18.8 |  61.8
Primal-RAIL |  77.9 |  40.4 |  85.6 |  83.3 |  68.3 |  62.2 |  76.6 |  45.8 |  80.4 |  41.7 |  66.2
Dual-RAIL |  82.3 |  43.4 |  90.4 |  86.0 |  71.0 |  62.8 |  76.7 |  46.7 |  80.3 |  41.7 |  68.1
Last |
LwF |  29.6 |  17.5 |  63.0 |  83.8 |  67.7 |  44.9 |  79.3 |  44.8 |  84.6 |  39.0 |  55.4
WiSE-FT |  46.1 |  23.5 |  71.3 |  85.7 |  70.2 |  59.1 |  85.5 |  47.9 |  82.4 |  42.8 |  61.5
ZSCL |  71.7 |  35.3 |  86.5 |  89.2 |  71.8 |  52.3 |  89.8 |  52.0 |  77.1 |  68.4 |  69.4
MoE-Adapter |  75.1 |  41.1 |  87.9 |  87.1 |  74.1 |  89.7 |  92.6 |  61.2 |  81.0 |  27.4 |  71.7
Primal-RAIL |  77.7 |  41.9 |  86.1 |  83.3 |  71.8 |  91.6 |  97.3 |  66.4 |  94.8 |  86.9 |  79.8
Dual-RAIL |  82.3 |  45.8 |  92.0 |  87.1 |  76.3 |  93.1 |  97.4 |  69.1 |  94.4 |  87.1 |  82.5

##  Appendix I Limitation

RAIL exhibits its superior performance and efficiency for transfering pre-
trained VLMs to various domains. A limitation here is that the pre-trained VLM
remains frozen, with its feature extraction ability unenhanced during the
incremental learning process. A promising direction for future work is to
adjust the pre-trained encoder according to new data with low computational
cost, thus further boosting RAIL’s performance while maintain its efficiency.
Beside, extending RAIL to encompass additional downstream tasks of VLMs, such
as image segmentation, can broaden its applicability and enhance its utility
in more complex visual understanding scenarios.