#  An explicit algebraic closure for passive scalar-flux: Applications in
heated channel flows subjected to system rotation iiiThis research did not
receive any specific grant from funding agencies in the public, commercial, or
not-for-profit sectors.

C. F. Panagiotou  F. S. Stylianou  E. Gravanis  E. Akylas  S. C. Kassinos
Nireas International Water Research Center,
Department of Civil & Environmental Engineering,
University of Cyprus, 75 Kallipoleos, Nicosia 1678, Cyprus  Computational
Sciences Laboratory, UCY-CompSci,
Department of Mechanical & Manufacturing Engineering,
University of Cyprus, 75 Kallipoleos, Nicosia 1678, Cyprus  Department of
Civil Engineering & Geomatics,
Cyprus University of Technology, PO Box 50329, Limassol 3603, Cyprus
Eratosthenes Centre of Excellence,
Cyprus University of Technology, Limassol, Cyprus

###### Abstract

We present an algebraic model for turbulent scalar-flux vector that stems from
tensor representation theory. The resulting closure contains direct dependence
on mean velocity gradients and on frame rotation tensor that accounts for
Coriolis effects. Model coefficients are determined from Direct Numerical
Simulations (DNS) data of homogeneous shear flows subjected to arbitrary mean
scalar gradient orientations. This type of tuning process renders the proposed
model to be objective towards inhomogeneous applications. Model performance is
evaluated in several heated channel flows in both stationary and rotating
frames, showing good results. To place the performance of the proposed model
into context, we compare with Younis algebraic model [1], which is known to
provide reasonable predictions for several engineering flows.

##  1 INTRODUCTION

Turbulent scalar-flux vector has a notable role in the scalar transport for a
wide range of practical applications. It appears in the Reynolds-Averaged
scalar transport equations as a term that needs to be modelled so that closure
is achieved. An elegant choice is the development of engineering models with
aim to provide estimations of this quantity at low computational complexity.
Engineering models for the scalar-flux are classified into two categories:
differential and algebraic. Differential transport models (DTM) are proven to
be beneficial tools, being capable of handling rotational and curvature
effects. However, robustness issues and performance inconsistencies, combined
with computational overheads associated with the solution of a differential
transport equation for each scalar-flux component, prevent this class of
models from penetrating further into the mainstream of engineering practice.
On the other hand, algebraic approaches are based on assumptions that lead to
constitutive equations between turbulent statistics and mean deformation. The
simplest algebraic model is based on the gradient-diffusion hypothesis (GDH),
which assumes that turbulent scalar-flux is aligned to the mean scalar
gradient. Despite its implementational and computational elegance, this
assumption is an important reason why this model fails to capture important
flow features, such as turbulence anisotropy or the effects of mean and system
rotation. As a result, Batchelor [2] proposed a generalization of GDH (GGDH)
from which several algebraic closures have emerged. For example, Daly $\&$
Harlow [3] adopted scale functions to express the turbulent scalar-flux vector
as a product between Reynolds stress and mean scalar-gradient, allowing the
mis-alignment of the scalar-fluxes with the mean scalar deformation. However,
this expression is found to perform poorly on predicting the correct
magnitudes of scalar-flux components in the directions normal to the mean
scalar gradient. Suga $\&$ Abe [4] improved the performance of the GGDH model
by adding a non-linear term that contains quadratic products of the Reynolds
stress tensor, which correctly captures the anisotropy levels of the scalar-
flux components in the vicinity of wall boundaries. This addition resulted in
the construction of Higher-Order GGDH models (HO-GGDH), which were
successfully applied in a wide range of heated channel flows. Another
interesting approach is to derive algebraic expressions directly from the
exact transport equation of turbulent scalar-flux through equilibrium
assumptions. A common choice is to apply the weak-equilibrium assumption (WEA)
[5], which states that the transient variations of turbulent anisotropies are
negligible compared to the variation of turbulent scales. To some extent,
these models are considered to be a good alternative to DTM, as they have been
successfully applied in different flow configurations while requiring less
computational capacity [6, 7, 8].

An alternative approach for estimating the turbulent scalar-fluxes was
proposed by Younis et al. [9]. Using as guidance the exact scalar-flux
transport equation, Younis and coworkers expressed this quantity as a function
of several tensor quantities. This approach is elegant, since it avoids the
reduction of transport equations through the WEA with all the modelling
uncertainties that entails. It also provides a general framework, from which
different algebraic expressions can be obtained. They proposed a multi-linear
closure that exhibited distinct improvements over other algebraic scalar-flux
closures in benchmark two-dimensional free shear flows, while it was
successfully used in flow configurations involving Coriolis and curvature
effects [10, 11]. However, the linear nature of this specific closure is the
main reason why it fails to capture the proper near-wall anisotropy levels of
scalar-flux vector, thus revealing the importance of incorporating non-linear
information regarding turbulence anisotropy. Hence, in this study we propose
an algebraic model that stems directly from the Younis formulation and
involves products of the Reynolds stress tensor, a dependence missing from the
multi-linear model. Special attention is given so that model complexity is
kept minimal for ease implementation in existing industrial codes, while its
performance ability is tested on several heated Couette and Poiseuille flows
in stationary and rotating frames.

A description of the exact transport equations of the turbulent passive-scalar
fluxes along with an extensive summary of the Younis approach are given in
Section 2 with the motivation behind the use of a non-linear term involving
products of Reynolds stress given in Section 3. The proposed formulation is
discussed in Section 4 and the performance of this closure is evaluated in
Section 5, yielding good results. Summary and conclusions are given in Section
6.

##  2  YOUNIS’ FORMULATION

The motivation behind the work reported by Younis et al. [9] arose from the
need to provide a better alternative to the existing gradient-transport
closures for the turbulent scalar-fluxes. As starting point, they considered
the exact transport equation of these fluxes in a frame rotating at a constant
angular velocity rate $\Omega^{f}$

$\begin{split}\frac{\partial\overline{u^{\prime}_{i}\phi^{\prime}}}{\partial
t}+\overline{u}_{j}\frac{\partial\overline{u^{\prime}_{i}\phi^{\prime}}}{\partial
x_{j}}=&-\overline{u^{\prime}_{i}u^{\prime}_{j}}\frac{\partial\overline{\phi}}{\partial
x_{j}}-\overline{u^{\prime}_{j}\phi^{\prime}}\frac{\partial\overline{u}_{i}}{\partial
x_{j}}-2\epsilon_{ijk}\Omega^{f}_{j}\overline{u^{\prime}_{k}\phi^{\prime}}\\\
&-\frac{1}{\rho}\overline{p^{\prime}\frac{\partial\phi^{\prime}}{\partial
x_{i}}}-(\gamma+\nu)\overline{\frac{\partial\phi^{\prime}}{\partial
x_{j}}\frac{\partial u^{\prime}_{i}}{\partial x_{j}}}\\\
&-\frac{\partial}{\partial
x_{j}}\bigg{(}\overline{u^{\prime}_{i}u^{\prime}_{j}\phi^{\prime}}+\frac{1}{\rho}\overline{p^{\prime}\phi^{\prime}}\delta_{ij}-\gamma\overline{u^{\prime}_{i}\frac{\partial\phi^{\prime}}{\partial
x_{j}}}-\nu\overline{\phi^{\prime}\frac{\partial u^{\prime}_{i}}{\partial
x_{j}}}\bigg{)}\,,\end{split}$ (1)

where the instantaneous flow variables are decomposed into a mean part,
denoted by overbar, and a fluctuating part, denoted by prime symbol.
Hereafter, we are using index notation whereby repeated indexes imply
summation. The coefficient of fluid viscosity and the coefficient of scalar
diffusivity are denoted by $\nu$ and $\gamma$ respectively, $\rho$ is the
density of the fluid and $u_{i}$, $\phi$ are the instantaneous fluid velocity
and passive scalar fields respectively. The fluctuating pressure is denoted
with $p^{\prime}$. The first and second terms on the RHS of equation (1)
represent the generation of scalar-flux as a consequence of turbulence-mean
deformation interactions. The third term arises when flow is subjected to
system rotation around an arbitrary axis, while the fourth term refers to the
fluctuating pressure-scalar correlations and is responsible for the
redistribution of the flux among the different components. The fifth term
refers to the rate at which the scalar-flux is destructed, while the last term
is interpreted as the turbulent-transport term. Younis et al. [9] used
equation (1) as a guide to provide a rationally assumed relationship between
the scalar-flux vector and various tensor quantities,

$\overline{u^{\prime}_{i}\phi^{\prime}}=f_{i}\bigg{(}R_{ij},S_{ij},W_{ij},\Omega^{f}_{ij},\Lambda_{i},\epsilon,\epsilon_{\phi},\overline{\phi^{{}^{\prime}2}}\bigg{)}\,,$
(2)

where $R_{ij}=\overline{u^{\prime}_{i}u^{\prime}_{j}}$ is the Reynolds stress
tensor, $\epsilon$ is the energy dissipation rate, $\epsilon_{\phi}$ is half
the scalar dissipation rate and $\overline{\phi^{{}^{\prime}2}}$ is the
scalar-variance. $S_{ij}$, $W_{ij}$, $\Omega^{f}_{ij}$ and $\Lambda_{i}$
denote the mean strain-rate tensor, the mean vorticity tensor, the frame-
rotation rate tensor and the mean scalar gradient vector respectively, defined
as

$\begin{split}S_{ij}&=\frac{1}{2}\bigg{(}\frac{\partial\overline{u}_{i}}{\partial
x_{j}}+\frac{\partial\overline{u}_{j}}{\partial x_{i}}\bigg{)}\,,\qquad
W_{ij}=\frac{1}{2}\bigg{(}\frac{\partial\overline{u}_{i}}{\partial
x_{j}}-\frac{\partial\overline{u}_{j}}{\partial
x_{i}}\bigg{)}\,,\qquad\Omega^{f}_{ij}=\epsilon_{ikj}\,\Omega^{f}_{k}\,,\qquad\Lambda_{i}=\frac{\partial\overline{\phi}}{\partial
x_{i}}\,.\end{split}$ (3)

With the aid of tensor representation theory, Younis et al. constructed an
explicit model for $\overline{u^{\prime}_{i}\phi^{\prime}}$ in accordance with
equation (2). Under this approach, $\overline{u^{\prime}_{i}\phi^{\prime}}$
can be expressed as a series of basis vectors $\Upsilon_{i}$,

$\overline{u^{\prime}_{i}\phi^{\prime}}=\sum^{M}_{n=1}\alpha_{n}\Upsilon^{n}_{i}\,,$
(4)

where $\alpha_{n}$ can depend on all the tensor variables appearing in
equation (2). The basis vectors are formed from the products of the symmetric
($S_{ij},R_{ij}$), the skew-symmetric ($W_{ij}$) tensors and the vector
($\Lambda_{i}$), leading to the following algebraic expression

$\begin{split}\overline{u^{\prime}_{i}\phi^{\prime}}&=\alpha_{1}\Lambda_{i}+\alpha_{2}R_{ij}\Lambda_{j}+\alpha_{3}S_{ij}\Lambda_{j}+\alpha_{4}R_{ik}R_{kj}\Lambda_{j}+\alpha_{5}S_{ik}S_{kj}\Lambda_{j}+\alpha_{6}W_{ij}\Lambda_{j}\\\
&+\alpha_{7}W_{ik}W_{kj}\Lambda_{j}+\alpha_{8}\bigg{(}S_{ik}W_{kj}+S_{jk}W_{ki}\bigg{)}\Lambda_{j}+\alpha_{9}\bigg{(}R_{ik}S_{kj}+R_{jk}S_{ki}\bigg{)}\Lambda_{j}\\\
&+\alpha_{10}\bigg{(}R_{ik}W_{kj}+R_{jk}W_{ki}\bigg{)}\Lambda_{j}\,,\end{split}$
(5)

where for simplicity we neglected the presence of Coriolis effects in the
above expression. To bring the above expression into a more compact form,
Younis et al. [9] assumed sufficiently small anisotropies and turbulent time
scales. Further assuming that the effects of $S_{ij}$ and $W_{ij}$ are in
balance, results to the following multilinear expression:

$\overline{u^{\prime}_{i}\phi^{\prime}}=C_{1}\frac{\kappa^{2}}{\epsilon}\Lambda_{i}+C_{2}\frac{\kappa}{\epsilon}R_{ij}\Lambda_{j}+C_{3}\frac{\kappa^{3}}{\epsilon^{2}}G^{T}_{ij}\Lambda_{j}+C_{4}\frac{\kappa^{2}}{\epsilon^{2}}\bigg{(}R_{ik}G^{T}_{jk}+R_{jk}G^{T}_{ik}\bigg{)}\Lambda_{j}\,,\qquad
G^{T}_{ij}=G_{ij}+\Omega^{f}_{ij}\,,$ (6)

where $\kappa=R_{kk}/2$ is the turbulent kinetic energy, while the
$\alpha_{i}$ coefficients were replaced based on dimensional arguments.
Coriolis effects are explicitly incorporated in the closure by adding the
frame-rotation tensor $\Omega^{f}_{ij}$ into the mean velocity gradient tensor
$G_{ij}=S_{ij}+W_{ij}$, an approach also followed in previous studies [12].
The first term on the RHS of equation (6) represents the GDH, while the second
term coincides with the production term that appears in equation (1) and is
associated with the mean scalar deformation, also known as the Generalized
Gradient Diffusion Hypothesis (GGDH) model proposed by Daly $\&$ Harlow [3].
The remaining two terms involve products between the gradients of mean scalar
and mean velocity; a dependence proposed by the analyses of Dakos $\&$ Gibson
[13] and Yoshizawa [14]. The values of $C_{i}$ coefficients were determined
based on the LES results of Kaltenbach [15] for homogeneous flows subjected to
a uniform shear with uniform scalar gradients

$C_{1}=0.0455\,,\qquad C_{2}=-0.373\,,\qquad C_{3}=0.00373\,,\qquad
C_{4}=0.0235\,.$ (7)

Regarding inhomogeneous flows, detailed analysis of the model performance in
relation to data from a large number of studies on wall-bounded flows showed a
serious error in the normal flux component [10]. As a result, the value of
$C_{1}$ coefficient was modified in the near-wall region by applying the
following damping function

$C_{1}=0.0455\,f_{C_{1}},\qquad f_{C1}=1-\exp(-A\,\beta\,Pe^{\alpha}),\qquad
Pe=Pr\,Re_{\tau}\,,$ (8)

where $\alpha=-0.02$ and $\beta=1.9$ are the values proposed by Younis et al.
[1], $Pr=\nu/\gamma$ is the Prandtl number,
$Re_{t}=\frac{\kappa^{2}}{\nu\epsilon}$ is the turbulence Reynolds number,
while $A$ is the stress-flatness parameter, defined as

$\begin{split}A_{ij}&=\frac{R_{ij}}{\kappa}-\frac{2}{3}\delta_{ij}\,,\qquad
A_{2}=A_{ij}A_{ji}\,,\qquad A_{3}=A_{ij}A_{jk}A_{ki}\,,\qquad
A=1-\frac{9}{8}(A_{2}-A_{3})\,,\end{split}$ (9)

where $A_{2}$ and $A_{3}$ are the second and third invariants of the
normalized Reynolds-stress anisotropic tensor $A_{ij}$. More details regarding
the mathematical formulation can be found in Younis et al. [10].

As already mentioned in the introduction, the linear form of Younis closure is
an important reason why this closure exhibits certain limitations. For
example, it fails to capture the near-wall anisotropy levels of the scalar-
flux vector. In the following section we discuss the importance of
incorporating a term that contains non-linear information regarding the
turbulent anisotropy, vital to predict the proper near-wall behaviour.

##  3 QUADRATIC DEPENDENCE ON REYNOLDS STRESS

It is well known that models involving only the second term of equation (6)
cannot predict the streamwise heat-flux component reasonably well [16]. In an
attempt to extend their applicability, Abe $\&$ Suga [17] performed a series
of LES simulations for fully-developed turbulent channel flows under different
boundary conditions and for a wide range of Prandtl numbers. They relied on
the findings of Kim $\&$ Moin [18], who pointed out that scalar fluctuations
are correlated more strongly with streamwise than transverse velocity
fluctuations in the near-wall region, to propose an algebraic relation between
the turbulent scalar-flux vector and quadratic products of Reynolds stress
tensor

$\overline{u^{\prime}_{i}\phi^{\prime}}=-C_{\phi}\tau\bigg{(}\frac{R_{ik}R_{kj}}{\kappa}\bigg{)}\Lambda_{j}\,,$
(10)

where $\tau$ is a turbulent time scale and $C_{\phi}$ is a model coefficient.
In wall-bounded channel flows where statistical quantities are functions only
of the wall-distance, the above expression approaches the following near-wall
limit

$R_{u^{\prime}\phi^{\prime}}\equiv\frac{\overline{u^{\prime}_{2}\phi^{\prime}}}{\overline{u^{\prime}_{1}\phi^{\prime}}}=\frac{R^{2}_{12}+R^{2}_{22}}{(R_{11}+R_{22})R_{12}}\rightarrow\frac{R_{12}}{R_{11}}\,,$
(11)

where subscripts 1, 2 and 3 are respectively the streamwise, wall-normal and
spanwise directions. Abe $\&$ Suga [17] also observed that the correlation
between $\phi^{\prime}$ and $u^{\prime}_{2}$ relatively increases with the
decrease of $Pr$, reaching almost the same level as that between
$\phi^{\prime}$ and $u^{\prime}_{1}$ for $Pr=0.025$. As a result, they
recommended that an effective algebraic scalar-flux model should depend both
on linear and quadratic forms of Reynolds stress, written as

$\overline{u^{\prime}_{i}\phi^{\prime}}=-\kappa\tau\bigg{(}C_{\phi,1}\frac{R_{ij}}{\kappa}+C_{\phi,2}\frac{R_{ik}R_{kj}}{\kappa^{2}}\bigg{)}\Lambda_{j}\,,$
(12)

where $C_{\phi,1}$ and $C_{\phi,2}$ are model coefficients. These coefficients
should be determined so that the non-linear term in equation (12) becomes
dominant in regions of high deformation rates, while the linear term becomes
significant in the presence of weak strain rates, as well as in low Prandtl
numbers. Hence, the present study aims to propose an algebraic model based on
Younis general form (5) that involves the non-linear term, and validate its
performance in different heated channel flows.

##  4 PROPOSED FORMULATION

In this section we introduce the proposed model, which is essentially a
combination between the multi-linear model of Younis (6) and the functional
form proposed by Abe $\&$ Suga (12), thus incorporating non-linear information
regarding turbulence anisotropy. Special attention is given to keep the model
as simple as possible so that it can be easily implemented in existing
industrial codes. This is achieved by investigating the contribution of each
term appearing in the above equations in an attempt to propose a minimal
combination of these terms that is able to provide reasonable predictions. We
neglect the $C_{1}$-related term, since this term is known to provide
erroneous predictions for the streamwise flux component. In addition, this
choice can be partially justified if we assume that information regarding this
term is already included in the isotropic part of the second term of equation
(6). We have also excluded the term associated with $C_{4}$ for the following
reasons: Firstly, this term cannot induce the proper near-wall anisotropy
levels of the scalar-flux vector, as given in equation (11). Secondly, this
term involves products between the mean velocity gradients and the Reynolds
stress, thus being more complex than the $C_{3}$-related term, which also
contains gradients of the mean velocity field. Thirdly, its absence simplifies
the process of including wall-damping corrections through the use of simple
exponential functions, as will be shown in this Section. Consequently, the
proposed closure takes the following compact form in a rotating frame of
reference:

$\overline{u^{\prime}_{i}\phi^{\prime}}=C_{2}\frac{\kappa}{\epsilon}R_{ij}\Lambda_{j}+C_{3}\frac{\kappa^{3}}{\epsilon^{2}}G^{T}_{ij}\Lambda_{j}+\frac{C_{5}}{\epsilon}R_{ik}R_{kj}\Lambda_{j}\,,$
(13)

where we keep the same indexing for the model coefficients as in equation (6).
Alternatively, the above equation essentially differs from equation (12) in
involving the $C_{3}$-related term, which accounts for the products between
mean scalar and mean velocity gradients. Model coefficients are determined
through a two-step process. As a starting point, model constants
($C_{2},C_{3},C_{5}$) are determined through a linear regression fitting using
data derived from the LES non-buoyant results of Kaltenbach [15]

$C_{2}=-0.0848\,,\qquad C_{3}=0.00496\,,\qquad C_{5}=-0.2942\,,$ (14)

who considered homogeneous shear flows under different orientations of the
mean scalar gradient. The complete dataset used to determine model
coefficients is given in the Appendix. Next, we seek to modify the above
coefficients to account for inhomogeneous effects, particularly in the
vicinity of wall-boundaries. As a result, we considered a fully-developed
channel flow at friction Reynolds number
$Re_{\tau}=\frac{u_{\tau}\delta}{\nu}=150$, where $u_{\tau}$ is the friction
velocity and $\delta$ is the half-channel height. The flow is driven by a mean
pressure gradient, with uniform heat flux applied at both walls. The Prandtl
number is set equal to 0.71, whereas the presented data is expressed in wall-
units. Figure LABEL:fig:CALIBRATION_CASE shows model predictions for heat
fluxes, obtained by using DNS results of Tomita et al. [19] in a stationary
frame. For this case, only a streamwise component of the mean velocity field
exists that varies along the normal component $x_{2}$. We observe that the
proposed model overestimates the streamwise component, especially the near-
wall peak magnitude, while reasonable predictions are provided for the normal
component. For simplicity, we focus on improving only the streamwise
component. A simple way to do that is by modifying $C_{3}$, since the term
associated with this coefficient does not contribute to the normal component,
as indicated in equation (13). Hence, we apply the following damping function
to this coefficient to reduce its value close to the wall boundary,

$C_{3}=0.00496\,f_{C_{3}}\,,\qquad
f_{C_{3}}=1.0-\exp\bigg{(}-A\,\beta^{*}\,Pe^{\alpha^{*}}\bigg{)}\,,\qquad\alpha^{*}=-0.1\,,\qquad\beta^{*}=30.0\,.$
(15)

For $y^{+}<15$, the uncorrected $C_{3}$-related term plays a dominant role,
with its maximum occurring at about $y^{+}=10$, a location at which mean
strain rate is also maximized (not shown here). The $C_{5}$-related term
contributes the most for $y^{+}>15$, while the impact of $C_{2}$-related term
becomes non-trivial in regions far away from the wall that are characterized
by weak deformations. Applying the damping function on $C_{3}$ coefficient
yields a reduction of the associated term, which remains important in the
high-shear region, while resulting in the quadratic term being dominant in the
entire region outside the viscous sublayer ($y^{+}>5$). Contrary, the
$C_{3}$-related term does not contribute to the normal component, which is
prevailed by the non-linear term for $y^{+}>15$. Consequently, the near-wall
behaviour of the model is driven by two mechanisms: one that involves products
between mean velocity and scalar gradients, and a mechanism arising from the
turbulence-turbulence interactions. Combining these two terms yields an
alternative form for the proposed model

$\overline{u^{\prime}_{i}\phi^{\prime}}=C_{2}\frac{\kappa}{\epsilon}R_{ij}\Lambda_{j}+C_{3}\frac{\kappa^{3}}{\epsilon^{2}}G^{e}_{ij}\Lambda_{j}\,.$
(16)

The above equation suggests that the non-linear interactions provide a
gradient, in addition to the actual mean gradient, thus yielding an effective
gradient $G^{e}_{ij}$

$G^{e}_{ij}=G^{T}_{ij}+\frac{4C_{5}}{C_{3}\tau}r_{ik}r_{kj}\,.$ (17)

This idea, called the “effective-gradients” hypothesis, has been originally
proposed by Kassinos & Reynolds [20] to construct a Reynolds-stress transport
closure, and has been recently extended by Panagiotou & Kassinos [21, 22, 23]
for passive scalar transport. Thus, equation (16) can be thought as an
extension of Abe & Suga proposal (10), since it contains a linear term that
becomes important in regions of weak deformation rate, while replacing the
quadratic term by a term involving the effective mean gradient that dominates
the region of high and moderate deformations rates.

##  5 MODEL ASSESSMENT

In this section we investigate the estimation ability of the proposed closure
for heated channel flows under different boundary conditions and Reynolds
numbers, in both stationary and rotating frames. For all cases considered, the
Prandtl number equals to 0.71 and the flow variables are expressed in wall-
units. In order to reduce uncertainties associated with the numerical solution
of model equations, we import results from DNS into the proposed algebraic
expression for the scalar-fluxes. To facilitate the discussion during the
validation procedure, we have performed additional computations to account for
the performance of Younis’ model (6). For all cases considered, the mean flow
varies only along the wall-normal direction $x_{2}$. Under these conditions,
and for a frame subjected to rotation around an arbitrary axis, the total mean
gradient $G^{T}_{ij}$ and the expressions for the flux components according to
equation (13) become

$G^{T}_{ij}=\begin{pmatrix}0&G_{12}-\Omega^{f}_{3}&\Omega^{f}_{2}\\\
\Omega^{f}_{3}&0&-\Omega^{f}_{1}\\\ -\Omega^{f}_{2}&G_{32}+\Omega^{f}_{1}&0\\\
\end{pmatrix},$

and

$\displaystyle\overline{u^{\prime}_{1}\phi^{\prime}}$
$\displaystyle=\bigg{\\{}C_{2}\frac{\kappa}{\epsilon}R_{12}+C_{3}\frac{\kappa^{3}}{\epsilon^{2}}(G_{12}-\Omega^{f}_{3})+\frac{C_{5}}{\epsilon}\bigg{(}R_{12}(R_{11}+R_{22})+R_{13}R_{23}\bigg{)}\bigg{\\}}\Lambda_{2}\,,$
(18a) $\displaystyle\overline{u^{\prime}_{2}\phi^{\prime}}$
$\displaystyle=\bigg{\\{}C_{2}\frac{\kappa}{\epsilon}R_{22}+\frac{C_{5}}{\epsilon}\bigg{(}R^{2}_{12}+R^{2}_{22}+R^{2}_{23}\bigg{)}\bigg{\\}}\Lambda_{2}\,,$
(18b)

respectively. The flow geometry and coordinate system are shown in figure 2.

Figure 2:  Flow configuration and coordinate system.

###  5.1 Heated Couette flows

Initially, we evaluate the model performance in a fully-developed Couette flow
with the top wall moving at constant speed $u_{w}$ relative to the bottom
wall. As a result, the mean flow is driven by the shear stress due to the
relative movement between top and bottom walls, while the temperature
difference between the heated top wall and the cooled bottom wall is kept
constant. We consider two cases at different Reynolds numbers, particularly
$Re_{w}=\frac{2\,u_{w}\,\delta}{\nu}=8600$ and 12800, for which detailed DNS
data are provided by Kawamura et al. [24]. Figures LABEL:fig:COUETTE_ScF_8600
and LABEL:fig:COUETTE_ScF_12800 show a comparison between the two models
(proposed and Younis) for the scalar-flux components with the corresponding
DNS results for the two cases. Regarding the streamwise component, both models
perform well near the wall boundary, being able to capture both the magnitude
and the location of the near-wall peak, while both closures predict faster
reduction of the shape profile with respect to the DNS results while moving
away from the wall. The proposed model agrees well with the DNS data for the
wall-normal component, especially at the wall region and close to the channel
center, while Younis’ model underestimates the wall-normal component outside
the viscous sublayer ($y^{+}>10$).

###  5.2 Heated Poiseuille flows

Here, we consider the case of a heated channel flow under fully-developed
conditions at two different Reynolds numbers, particularly $Re_{\tau}=395$ and
640. The non-slip boundary condition is adopted in the wall-normal direction
(i.e. at the top and bottom walls) while the thermal boundary condition is
uniform heat-flux on both walls. The quality of the predictions is compared
with the corresponding DNS results [17, 25]. Figure
LABEL:fig:POISEUILLE_ScF_395_640 compares the turbulent scalar-fluxes as
obtained from the present model and the Younis model for both cases. We
observe that both models provide similar predictions for the streamwise
component, achieving good agreement with the DNS data in both the inner and
outer region of the channel. For the high $Re$ case though, both models tend
to overpredict the magnitude of the peak value, thus showing a mild Reynolds
dependence. Looking into the balance between the terms appearing in equation
(18a) (not shown here) revealed that this tendency is mainly attributed to the
$C_{3}$-related term, since this term exhibits the highest sensitivity to the
Reynolds number. Contrary to algebraic closures, DNS predict non-zero values
for the streamwise component at the channel centerline. This happens because
the turbulent scalar field is being transported by the mean flow or the
turbulence. These transport processes constitute a non-local mechanism that
cannot be captured by a rational algebraic closure [9]. For the low $Re$ case,
the proposed model achieves a considerably better agreement than Younis’ model
regarding the normal flux, being able to quickly adjust to the slope’s sign
change that occurs in the inner region. As expected, both models tend to
overestimate the magnitude of the flux for $Re_{\tau}=640$, with the proposed
model still being able to produce satisfactory results.

Next, we further investigate the near-wall performance of the proposed
closure. A useful parameter for that purpose is the scalar-flux ratio
$R_{u^{\prime}\phi^{\prime}}$, defined in equation (11). Figure
LABEL:fig:POISEUILLE_ScR shows a comparison of the scalar-flux ratio
predictions of both models (proposed and Younis) with the corresponding DNS
data, for which this ratio varies as $R_{12}/R_{11}$ in the near-wall region
under high shear strain, as already mentioned in Section 3. The proposed model
achieves better agreement with the DNS results, being able to capture the
near-wall limit up to $y^{+}=20$. This is attributed to the dominant role that
$C_{5}$-related term plays in the buffer layer for both flux-components, as
already shown in figure LABEL:fig:CALIBRATION_CASE, which exhibits the proper
near-wall physical behaviour (see discussion in Section 3). In contrast,
Younis’ model fails to capture this limiting behaviour for $y^{+}>10$, since
no term appearing in its model equation (6) captures the correct turbulent
anisotropy.

###  5.3 Heated channel flow subjected to streamwise rotation

The first considered case involving Coriolis effects is that of a flow in a
plane channel that rotates about its streamwise axis. This scenario is more
complex than the stationary one, since it induces a non-trivial spanwise mean
velocity component, thus activating additional Reynolds stress elements. The
walls are assumed to be kept at different, but constant temperatures without
fluctuations, with the upper wall cooled and the lower wall heated. Comparison
is made with DNS studies that differ substantially in the relative strength of
the imposed rotation. The first study corresponds to the DNS results of Wu &
Kasagi [26] for Rossby and Reynolds numbers
$Ro_{\tau}=\frac{2\Omega^{f}_{1}\delta}{u^{*}_{\tau}}=2.5$ and
$Re_{\tau^{*}}=\frac{u^{*}_{\tau}\delta}{\nu}=300$ respectively, where
$u^{*}_{\tau}$ is the friction velocity calculated from the wall shear stress
averaged on the two walls. Figure LABEL:fig:RE_300_Ro25_ST shows results for
the scalar-flux components. For the streamwise component, both models
accurately capture the location and magnitude of the peak value. However,
similar to the stationary cases, their predictions drop much faster than the
DNS results while approaching the channel’s centerline. The present closure
provides reasonable estimations for the normal component, being able to
capture the near-wall anisotropy, while it mildly underestimates the level of
this quantity while moving away from the wall. This happens possibly due to
the fact that the effects of streamwise rotation on the normal component are
explicitly absent, as indicated in equation (18b). The opposite is true for
Younis’ formulation, for which this dependence enters via the $C_{4}$-related
term, as implied by equation (6).

The second case study considered is by El-Samni & Kasagi [27] for
$Re_{\tau^{*}}=163$ in the presence of a much stronger rotation rate,
corresponding to $Ro_{\tau}=15$, whereas the thermal boundary conditions are
the same as in the low-$Ro_{\tau}$ case. Figure LABEL:fig:RE_163_Ro15_ST shows
profiles of the scalar-flux components, showing large discrepancies between
DNS and models. Both models significantly overpredict the maximum value of the
streamwise component. Regarding the proposed closure, this is partly
associated with the presence of non-trivial contribution from the product
$R_{12}\,R_{23}$ that appears in the $C_{5}$-related term of the model
equation (18a). As a result, the value of this term is greatly increased
compared to the stationary case. Significant increase (compared to the non-
rotating case) also occurs for $C_{3}$-related term, suggesting that
modifications to the associated model coefficient might be required so that
the model attains the proper anisotropy levels. In accordance with the
previous case study, the proposed model predicts smaller values for the normal
component than Younis’ formulation, achieving better agreement with the DNS in
the near wall region as well as close to channel’s centerline.

###  5.4 Heated channel flow subjected to wall-normal rotation

Here, the simulated flow field is rotated at a specified angular velocity
around the wall-normal axis, which induces a strong spanwise mean velocity
that makes the absolute mean flow tilt towards to the spanwise direction. The
DNS results used for this case are those of El-Samni & Kasagi [27] at
$Re_{\tau^{*}}=160.1$ and $Ro_{\tau}=0.04$. The profiles of the streamwise
scalar-flux component are shown in figure LABEL:fig:RE_160_Ro004_WN_a. The
models provide similar predictions, being able to capture accurately the near-
wall peak while underestimating the value of this quantity in the remaining
region. As shown in figure LABEL:fig:RE_160_Ro004_WN_b, Younis’ model achieves
very accurate predictions for the normal component, while the present model
underestimates the level of this quantity outside the proximity of the wall
boundary ($y^{+}>20$).

###  5.5 Heated channel flow subjected to spanwise rotation

Next, we consider the presence of Coriolis forces emerging from the spanwise
rotation. For this configuration, existing studies [28, 29] have shown that at
moderate $Ro$ numbers, turbulence and especially the wall-normal fluctuations
are augmented on the channel side where the system rotation is anti-cyclonic
(called the unstable side), while they are damped on the channel side where
the rotation is cyclonic (called the stable side). The scalar is kept at
constant but different values at each wall, in accordance with the previous
rotating cases. Model predictions are compared with the DNS results of Wu &
Kasagi [26] for $Re_{\tau^{*}}=295.5$ and $Ro_{\tau}=2.5$. Figure
LABEL:fig:RE_295_Ro25_SP_a shows that both models severely overestimate the
level of the peak magnitude in the near-wall region. Focusing on the present
model, the strength of the individual terms that contribute to this quantity
(18a) is shown in figure LABEL:fig:RE_295_Ro25_SP_b. We observe the failure of
the model in the near-wall region mainly because the $C_{3}$-related term
becomes extremely high in the high-strain rate region, while it becomes
trivial for $y^{+}>50$, yielding reasonable predictions of the model outside
the buffer layer ($y^{+}>30$). Consequently, modifications on the damping
function $f_{C_{3}}$ should be applied to improve model performance in this
region. Figure 11 reveals that both models are sensitized to the rotation-
induced asymmetry, with the proposed model being in closer agreement with the
DNS results than Younis’ model. Note that equation (18b) indicates that the
$C_{3}$-related term does not contribute to the estimation, thus supporting
the notion that modifications on the $f_{C_{3}}$ can improve model performance
for this case.

Figure 11:  As in Figure LABEL:fig:RE_295_Ro25_SP_a, but for the wall-normal
component.

##  6  SUMMMARY AND CONCLUSIONS

In this study we have proposed an explicit algebraic closure for the turbulent
scalar-flux vector based on the general formulation of Younis, which is
essentially an extended version of the model proposed by Abe & Suga (12)
through the inclusion of an extra term involving products between the
gradients of the mean velocity and scalar field. This closure consists of
three terms, making it simpler than Younis’ model and an elegant choice for
use in general-purpose computational codes. It is also motivated by the
“effective-gradients” hypothesis, which postulates that turbulence-turbulence
interactions provide a gradient that acts supplementary to the mean shear,
thus giving an alternative physical interpretation of the proposal compared to
other closures. The resulting model explicitly depends on Coriolis effects in
a similar manner to other models, which ensures its consistency with the
principle of coordinate invariance. To minimize model bias to inhomogeneous
applications, the values of the model coefficients are determined based on
existing LES predictions of homogeneous shear flows in the presence of
arbitrary mean scalar gradients, while a simple damping function was applied
to account for the near-wall effects. In order to test the quality of the
proposed model, we have considered different types of heated channel flows,
particularly Poiseuille and Couette flows, in both stationary and rotating
frames. The proposed model performs well in Couette flows at different
Reynolds numbers, showing its sensitivity to the near-wall turbulent
anisotropies. Regarding Poiseuille flows, good predictions are obtained for
both flux components, with the model showing a mild tendency to overestimate
the near-wall peak magnitude as the Reynolds number increases. Furthermore,
the proposed model captured the proper near-wall limit for the scalar ratio
$R_{u^{\prime}\phi^{\prime}}$ for wall distances up to $y^{+}\approx 20$,
significantly further than Younis’ model ($y^{+}\approx 10$). In all cases,
the proposed model achieved substantially better agreement with the DNS
results than Younis’ model for the normal component, while similar predictions
between the models have been obtained for the streamwise component. The above
non-rotating cases served as a guidance regarding the role of the different
terms appearing in the algebraic expressions, showing the ability of the
proposal to capture the anisotropy between the different flux components. The
performance of the proposed model was further challenged in heated channel
flows subjected to different modes of rotation, resulting in more complex flow
configurations due to the emergence of secondary flows. Generally, the
presented algebraic expression provides reasonable predictions for the normal
component. Under spanwise rotation though, the model fails in capturing the
proper near-wall behavior (the same is also evident with Younis model) of the
streamwise component, suggesting that further modifications on the damping
functions should be investigated. Future work will focus on testing the
proposed closure in additional cases involving frame-rotation, in an attempt
to further improve the estimation performance in the presence of Coriolis
effects. For that purpose, we intend to use DNS data in heated channel flows
to quantify the response of passive scalar transport for a wider range of
Rossby numbers.

##  Appendix A Details regarding the calibration process.

The model coefficients of the proposed algebraic closures are determined based
on the LES data of Kaltenbach et al. [15], who investigated the turbulent
transport of three passive species which have uniform gradients in either the
normal, streamwise or spanwise direction. All cases under consideration start
from the same shear parameter ratio
$S^{*}_{o}=\bigg{(}\frac{2S\kappa}{\epsilon}\bigg{)}_{o}=5.04$ and the same
Prandtl number $Pr=1.0$, in the initial absent of scalar fluctuations. The
mean flow configuration is expressed by

$G_{ij}=S\delta_{i1}\delta_{j3}\,,\qquad\Lambda_{i}=\Lambda\delta_{i\alpha}\,,$
(19)

where greek index $\alpha$ denotes the direction of the mean scalar gradient.

$St$ | $R_{11}$ | $R_{13}$ | $R_{22}$ | $R_{33}$ | $\epsilon$ | $\Lambda_{i}$ | $\overline{u^{\prime}_{1}\phi^{\prime}}$ | $\overline{u^{\prime}_{2}\phi^{\prime}}$ | $\overline{u^{\prime}_{3}\phi^{\prime}}$
---|---|---|---|---|---|---|---|---|---
8 | 4.98 | -1.42 | 2.82 | 1.60 | 0.911 | $(1,0,0)$ | -10.9 |  | 2.22
10 | 6.32 | -1.84 | 3.63 | 2.20 | 1.190 | $(1,0,0)$ | -13.9 |  | 3.01
12 | 7.67 | -2.27 | 4.71 | 3.12 | 1.600 | $(1,0,0)$ | -16.3 |  | 4.08
8 | 4.98 | -1.42 | 2.82 | 1.60 | 0.911 | $(0,1,0)$ |  | -3.84 |
10 | 6.32 | -1.84 | 3.63 | 2.20 | 1.190 | $(0,1,0)$ |  | -4.62 |
12 | 7.67 | -2.27 | 4.71 | 3.12 | 1.600 | $(0,1,0)$ |  | -5.78 |
8 | 4.98 | -1.42 | 2.82 | 1.60 | 0.911 | $(0,0,1)$ | 4.32 |  | -1.83
10 | 6.32 | -1.84 | 3.63 | 2.20 | 1.190 | $(0,0,1)$ | 5.42 |  | -2.47
12 | 7.67 | -2.27 | 4.71 | 3.12 | 1.600 | $(0,0,1)$ | 6.33 |  | -3.42
Table 1:  Summary of the LES data used to determine model coefficients. Data
are extracted at different total shear instants $St$ (where $t$ is time).

## References

  * [1] B.A. Younis, B. Weigand, F. Mohr, and M. Schmidt.  Modeling the effects of system rotation on the turbulent scalar fluxes.  J. Heat Transf., 132(5):051703, 2010.  doi: 10.1115/1.4000446.
  * [2] G.K. Batchelor.  Diffusion in a field of homogeneous turbulence.  Aust. J. Sci. Res. Ser. A, 2:437–450, 1949.  doi: 10.1071/PH490437.
  * [3] B.J. Daly and F.H. Harlow.  Transport equations of turbulence.  Phys. Fluids, 13 (11):2634–2649, 1970.  doi: 10.1063/1.1692845.
  * [4] K. Suga and K. Abe.  Nonlinear eddy viscosity modelling for turbulence and heat transfer near wall and shear-free boundaries.  Int. J. Heat Fluid Fl., 21(1):37–48, 2000.  doi: 10.1016/S0142-727X(99)00060-0.
  * [5] W. Rodi.  The prediction of free turbulent boundary layers by use of a two-equation model of turbulence.  PhD Thesis, Department of Heat Transfer, Imperial College, London, 1972.
  * [6] S.S. Girimaji and S. Balachandar.  Analysis and modeling of buoyancy-generated turbulence using numerical data.  Int. J. Heat Mass Tran., 41 (6-7):915–929, 1998.  doi: 10.1016/S0017-9310(97)00166-X.
  * [7] P.M. Wikstrom, S. Wallin, and A.V. Johansson.  Derivation and investigation of a new explicit algebraic model for the passive scalar flux.  Phys. Fluids, 12:688–702, 2000.  doi: 10.1063/1.870274.
  * [8] W.M.J. Lazeroms, G. Brethouwer, S. Wallin, and A.V. Johansson.  An explicit algebraic Reynolds-stress and scalar-flux model for stably stratified flows.  J. Fluid Mech., 723:91–125, 2013.  doi: 10.1017/jfm.2013.116.
  * [9] B.A. Younis, C.G. Speziale, and T.T. Clark.  A rational model for the turbulent scalar fluxes.  Proc. R. Soc. A, 461:575–594, 2005.  doi: 10.1098/rspa.2004.1380.
  * [10] B.A. Younis, B. Weigand, and S. Spring.  An explicit algebraic model for turbulent heat transfer in wall-bounded flow with streamline curvature.  J. Heat Transf., 129(4):425–433, 2007.  doi: 10.1115/1.2709960.
  * [11] H. Müller, B.A. Younis, and Weigand B.  Development of a compact explicit algebraic model for the turbulent heat fluxes and its application in heated rotating flows.  Int. J. Heat Mass Tran., 86:880–889, 2015.  doi: 10.1016/j.ijheatmasstransfer.2015.03.059.
  * [12] B.E. Launder, D.P. Tselepidakis, and B.A. Younis.  A second-moment closure study of rotating channel flow.  J. Fluid Mech., 183:63–75, 1987.  doi: 10.1017/S0022112087002520.
  * [13] T. Dakos and M.M. Gibson.  On modelling the pressure terms of the scalar flux equations.  pages 7–18. Turbulent Shear Flows 5. Springer, Berlin, Heidelberg, 1987\.  doi: 10.1007/978-3-642-71435-1_2.
  * [14] A. Yoshizawa.  Statistical analysis of the anisotropy of scalar diffusion in turbulent shear flows.  Phys. Fluids, 28(11):3226–3231, 1985.  doi: 10.1063/1.865371.
  * [15] H.J. Kaltenbach, T. Gerz, and U. Schumann.  Large-eddy simulation of homogeneous turbulence and diffusion in stably stratified shear flow.  J. Fluid Mech., 280:1–40, 1994.  doi: 10.1017/S0022112094002831.
  * [16] B.E. Launder.  On the computation of convective heat transfer in complex turbulent flows.  J. Heat Transf., 110(4b):1112–1128, 1988.  doi: 10.1115/1.3250614.
  * [17] K. Abe and K. Suga.  Towards the development of a Reynolds-averaged algebraic turbulent scalar-flux model.  Int. J. Heat Fluid Fl., 22(1):19–29, 2001.  doi: 10.1016/S0142-727X(00)00062-X.
  * [18] J. Kim and P. Moin.  Transport of passive scalars in a turbulent channel flow.  In Andre JC., J. Cousteix, F. Durst, B.E. Launder, F.W. Schmidt, and J.H. Whitelaw, editors, Turbulent Shear Flows, volume 6. Springer, Berlin, Heidelberg, 1989.
  * [19] Y. Tomita, N. Kasagi, and A. Kuroda.  Establishment of the Direct Numerical Simulation databases of turbulent transport phenomena.  http://thtlab.jp/DNS/CH122_PG.WL1, 1993.
  * [20] S.C. Kassinos and W.C. Reynolds.  A structure-based model with stropholysis effects.  Annual Research Briefs, pages 197–209, California, USA, 1998. Center for Turbulence Research, NASA Ames/Stanford University.
  * [21] C.F. Panagiotou and S.C. Kassinos.  A structure-based model for the transport of passive scalars in homogeneous turbulent flows.  Int. J. Heat Fluid Fl., 57:109–129, 2016.  doi: 10.1016/j.ijheatfluidflow.2015.11.008.
  * [22] C.F. Panagiotou and S.C. Kassinos.  A structure-based model for transport in stably stratified homogeneous turbulent flows.  Int. J. Heat Fluid Fl., 65:309–322, 2017.  doi: 10.1016/j.ijheatfluidflow.2016.12.005.
  * [23] C.F. Panagiotou, F.S. Stylianou, and S.C. Kassinos.  Structure-based transient models for scalar dissipation rate in homogeneous turbulence.  Int. J. Heat Fluid Fl., XX:XX, 2020.
  * [24] H. Kawamura, H. Abe, and K. Shingai.  DNS of turbulence and heat transport in a channel flow with different Reynolds and Prandtl numbers and boundary conditions.  In Y. Nagano, K. Hanjalic, and T. Tsuji, editors, 3rd International Symposium on Turbulence, Heat and Mass Transfer, volume 6. Aichi Shuppan, 2000.
  * [25] H. Abe, H. Kawamura, and Y. Matsuo.  Surface heat-flux fluctuations in a turbulent channel flow up to ${R}e_{\tau}$=1020 with Pr=0.025 and 0.71.  Int. J. Heat Fluid Fl., 25(3):404–419, 2004.  doi: 101016/jijheatfluidflow200402010.
  * [26] H. Wu and N. Kasagi.  Effects of arbitrary directional system rotation on turbulent channel flow.  Phys. Fluids, 16(4):979–990, 2004.  doi: 10.1063/1.1649337.
  * [27] O. El-Samni and N. Kasagi.  Heat and momentum transfer in rotating turbulent channel flow.  In 4th JSME-KSME Thermal Engi. Conf., volume 3. Kobe, 2000.
  * [28] O. Grundestam, S. Wallin, and A.V Johansson.  Direct numerical simulations of rotating turbulent channel flow.  J. Fluid Mech., 598:177–199, 2008.  doi: 10.1017/S0022112007000122.
  * [29] Z. Xia, Y. Shi, and S. Chen.  Direct numerical simulation of turbulent channel flow with spanwise rotation.  J. Fluid Mech., 788:42–56, 2016.  doi: 10.1017/jfm.2015.717.


# Revisiting Catastrophic Forgetting in Large Language Model Tuning

Hongyu Li
Wuhan University
<EMAIL_ADDRESS>
&Liang Ding
The University of Sydney
<EMAIL_ADDRESS>
Meng Fang
University of Liverpool
<EMAIL_ADDRESS>
&Dacheng Tao
Nanyang Technological University
<EMAIL_ADDRESS>Corresponding author.

###### Abstract

Catastrophic Forgetting (CF) means models forgetting previously acquired
knowledge when learning new data. It compromises the effectiveness of large
language models (LLMs) during fine-tuning, yet the underlying causes have not
been thoroughly investigated. This paper takes the first step to reveal the
direct link between the flatness of the model loss landscape and the extent of
CF in the field of LLMs. Based on this, we introduce the sharpness-aware
minimization to mitigate CF by flattening the loss landscape. Experiments on
three widely-used fine-tuning datasets, spanning different model scales,
demonstrate the effectiveness of our method in alleviating CF. Analyses show
that we nicely complement the existing anti-forgetting strategies, further
enhancing the resistance of LLMs to CF.

Revisiting Catastrophic Forgetting in Large Language Model Tuning

Hongyu Li Wuhan University<EMAIL_ADDRESS>Liang Ding††thanks:
Corresponding author. The University of Sydney<EMAIL_ADDRESS>

Meng Fang University of Liverpool<EMAIL_ADDRESS>Dacheng Tao Nanyang
Technological University<EMAIL_ADDRESS>

##  1 Introduction

Instruction fine-tuning is key to improving the capabilities and
controllability of large language models (LLMs) Brown et al. (2020); Fedus et
al. (2022), which have already demonstrated strong performance in various
tasks Zhong et al. (2023); Peng et al. (2023); Lu et al. (2023); Ren et al.
(2024). One major obstacle to tuning LLMs is catastrophic forgetting (CF,
Kirkpatrick et al., 2017), which means LLMs forget prior knowledge when
learning new data. Recent works have provided substantial evidence confirming
the negative impact of CF on LLMs, e.g., Bi et al. (2024) empirically show
that the fine-tuned model is even worse than their foundation counterpart on
several tasks, and Zeng et al. (2023) reveal the dropped general performance
of LLMs after developing their agent capabilities.

Understanding the effectiveness of LLMs during fine-tuning is important for
downstream tasks, however, the underlying causes of CF remain largely
unexplored. There are two types of work addressing CF, from data and model
perspectives, respectively. Scialom et al. (2022) propose continual learning
with rehearsal in instruction tuning, though its effectiveness varies with
task selection. Lin et al. (2023) indicate that both continual learning and
weight averaging (Wise-FT) effectively preserve generality.

While the above techniques are somewhat successful, they require expensive
extra data-constructing and training costs, and are even sometimes impractical
in LLMs, because a) the data cards of many pretrained models are unclear Shi
et al. (2023), making rehearsal unfeasible, and b) anti-forgetting training
brings an unstable and expensive training process Datta et al. (2023).

In this work, we turn to finding a cheap, stable, and orthogonal solution to
alleviate the CF in tuning LLMs. In particular, we ❶ reveal the high
correlation between the extent of CF and the flatness of the loss landscape
(LLS, Keskar et al., 2017; Dziugaite and Roy, 2017), ❷ mitigate the CF in LLMs
by flattening the LLS from optimization perspective, ❸ show the
complementarity between our method and existing anti-forgetting works.
Specifically, we designed three probing analyses to achieve ❶, and found that
a flatter LLS could reduce the severity of CF. Based on our observation, we
introduced “Sharpness-Aware Minimization” (SAM, Foret et al., 2021) to flatten
the model LLS to approach ❷. For ❸, we found that our introduced optimizer
nicely complements a series of anti-forgetting methods, including rehearsal
Scialom et al. (2022) and Wise-FT Lin et al. (2023). Our contributions are as
follows:

(a) Alpaca.

(b) Alpaca $\rightarrow$ Open-Platypus

(c) Alpaca $\rightarrow$ Auto-Wiki

Figure 1: Visualization of different models’ loss landscapes (LLS) with
contour lines. We can see that with the data/task gaps of (a), (b), (c)
gradually increase, the disturbance of their contours becomes more obvious.

  * •

To the best of our knowledge, we are the first to empirically reveal a direct
correlation between the flatness of the model LLS and CF.

  * •

We present the first work that mitigates CF of LLMs from the perspective of
optimization.

  * •

The proposed method can be synergistically combined with existing methods to
enhance the resilience of LLMs against forgetting.

##  2 Unveiling the Hidden Nexus: CF and the Model LLS

We investigate the relationship between CF and the flatness of LLS through
three analyses: i) loss landscape visualization, ii) flatness degree of LLS,
and iii) general task performance, where the visualized comparison of i) and
results of ii) & iii) are discussed in § 2.1 and § 2.2, respectively, to
intuitively and quantitatively reveal their correlation.

We simulate continue-learning scenarios with three models: Llama2-7b Touvron
et al. (2023) tuned on 1) “Alpaca Taori et al. (2023)”, 2)
“Alpaca$\rightarrow$Open-Platypus Lee et al. (2023)”, and 3)
“Alpaca$\rightarrow$Auto-Wiki Jiang et al. (2020); Alva-Manchego et al.
(2020)”, where “$\rightarrow$” means continue-train on the following dataset
and the data gaps are gradually increased. For the experimental details,
please refer to the Appendix A.1.1.

###  2.1 Visualized LLS comparison of different CF settings

Following Hao et al. (2019); Zan et al. (2022), we visualize the 2D loss
surface to explore the differences in the LLS of models under different
settings. We represent the weight of a well-trained model as $\theta_{0}$ and
define the loss surface with the function:
$f(\alpha,\beta)=\mathcal{L}\left(\bm{\theta}_{0}+\alpha\bm{\delta}_{1}+\beta\bm{\delta}_{2}\right),$
where $\mathcal{L}$ is the loss function, $\alpha$ and $\beta$ are scalar
values representing the current coordinate, with ${\delta}_{1}$ and
${\delta}_{2}$ as two direction vectors. We choose Gaussian noise as the basis
vectors for exploring the parameter space, and the axes correspond to $\alpha$
and $\beta$ values.

| Flatness Degree | General Perf.
---|---|---
| SC | AG | MAG | MMLU
(a)Alpaca | 52.87 | 105.37 | 65.53 | 40.53
(b)$\rightarrow$Open-Platypus | 52.98 | 106.41 | 68.91 |  $\text{33.46}^{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\textbf{-7.1}}}$
(c)$\rightarrow$Auto-Wiki | 53.77 | 111.03 | 70.71 |  $\text{23.31}^{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\textbf{-17.2}}}$
Table 1: Quantitative results of the LLS flatness (“Flatness Degree”) of LLMs
and their general task performance (“General Perf.”).

From the LLS visualization in Fig. 1, we observe that with the continue-
learned task gap increases, i.e., (a)$\leq$(b)$\leq$(c), their landscape and
contour lines become sharper and more disturbed, intuitively demonstrating the
high correlation between the flatness degree (of model loss landscape) and the
forgetting degree (of continually tuning LLMs).

###  2.2 Quantitative flatness degree and general performance of different CF
settings

Besides intuitive visualization, we quantitatively measure the flatness degree
of loss landscape (with three metrics: ‘SC’ (Surface Curvature, %), ‘AG’
(Average Gradient,%), ‘MAG’ (Mean Absolute Gradient, %)), and the model
performance on general tasks (in terms of MMLU benchmark Hendrycks et al.
(2021a)). Details of our proposed metrics can see Appendix A.3.

As seen in Tab. 1, for different CF settings (difficulty rank:
(a)$\leq$(b)$\leq$(c)), their performance on downstream tasks, i.e., MMLU
tasks, is significantly dropped (up to -17.2 on average), confirming our
hypothesis that continually training on harder tasks (with larger data gaps)
will exacerbate CF problem. Also, we find that the flatness degree rank is as
follows: (a)$\geq$(b)$\geq$(c), where their average values are 74.59, 76.1,
and 78.50, respectively, implying that there is a highly positive correlation
between the severity of CF and the sharpness of the loss landscape.

##  3 Improving CF with Sharpness-Aware Optimization

Motivated by the above findings, we naturally consider introducing the
Sharpness-Aware Minimization (SAM, Foret et al., 2021) method to flatten the
optimization landscape, thereby mitigating the CF problem. Given model weights
as $w\in\mathbb{R}^{d}$, loss function as $f$, and a training dataset
$S=\\{(x_{i},y_{i})\\}_{i=1}^{n}$ $\bm{i.i.d.}$ sampled from the distribution
$D$, the working mechanism of SAM can be formulated as follows:

$\min_{\bm{w}}\max_{\|\bm{\epsilon}\|_{2}\leq\rho}f(\bm{w}+\bm{\epsilon}),$
(1)

where $\epsilon$ is the perturbation in a neighbourhood ball area with a
radius denoted as $\rho$. The optimization objective is to ensure that the
loss $f$ does not increase substantially with $\epsilon$ constrained by
$\rho$. Further utilizing Taylor series expansion, $\epsilon$ can be
approximate as
$\rho\bm{\nabla_{w}}f(\bm{w})/{\|\bm{\nabla_{w}}f(\bm{w})\|_{2}}$. In case,
Eq. 1 can be simplified as:

$\min_{\bm{w}}f\left(\bm{w}+\rho\frac{\bm{\nabla_{w}}f(\bm{w})}{\|\bm{\nabla_{w}}f(\bm{w})\|_{2}}\right).$
(2)

Eq. 2 could be solved through a two-step gradient descent process. Firstly,
the perturbation $\epsilon$ is computed by
$\rho\bm{\nabla_{w}}f(\bm{w})/{\|\bm{\nabla_{w}}f(\bm{w})\|_{2}}$. Following
this, the second phase of gradient descent performs the actual update of the
weights.

##  4 Experimental Setting

##### Dataset

To validate the capability of SAM on diverse datasets, we conduct experiments
on four widely-used instruction fine-tuning datasets: Alpaca Taori et al.
(2023), ShareGPT52K Chiang et al. (2023), MetaMathQA Yu et al. (2023), and
Open-Platypus Lee et al. (2023). For detailed introduction, please refer to
the Appendix A.1.1.

##### Model Selection

To further demonstrate the compatibility of SAM with models of different
sizes, we conducted experiments on models including TinyLlama-1.1B Zhang et
al. (2024), Llama2-7B, and Llama2-13B Touvron et al. (2023).

Dataset | DK | Understanding | Reasoning | Exams | AVG | $\bm{\Delta}$
---|---|---|---|---|---|---
Alpaca | 40.53 | 58.74 | 63.33 | 45.08 | 51.92 | —
$\rightarrow$ShareGPT(w/o) | 26.08 | 52.84 | 58.68 | 45.76 | 45.84 | -6.08
$\rightarrow$ShareGPT(w/) | 40.08 | 57.91 | 63.78 | 44.41 | 51.55 | +5.71
$\rightarrow$Open-Platypus(w/o) | 31.13 | 50.70 | 61.09 | 37.97 | 45.22 | -6.70
$\rightarrow$Open-Platypus(w/) | 41.07 | 58.28 | 64.50 | 45.08 | 52.23 | +7.01
$\rightarrow$Meta-Math(w/o) | 33.13 | 52.61 | 58.46 | 36.27 | 45.12 | -6.80
$\rightarrow$Meta-Math(w/) | 34.79 | 55.77 | 62.38 | 42.71 | 48.91 | +3.79
(a) General Performance of Evaluation Tasks on SAM (w/o) & (w/) across
Different Datasets.

Model | Dataset | DK | Understanding | Reasoning | Exams | AVG | $\bm{\Delta}$
---|---|---|---|---|---|---|---
TinyLlama | Alpaca | 23.16 | 30.62 | 46.16 | 26.78 | 31.68 | —
$\rightarrow$Open-Platypus(w/o) | 23.04 | 31.06 | 46.23 | 27.12 | 31.86 | +0.18
$\rightarrow$Open-Platypus(w/) | 23.14 | 30.39 | 46.91 | 26.10 | 31.64 | -0.22
Llama2-7B | Alpaca | 40.53 | 58.74 | 63.33 | 45.08 | 51.92 | —
$\rightarrow$Open-Platypus(w/o) | 31.13 | 50.70 | 61.09 | 37.97 | 45.22 | -6.70
$\rightarrow$Open-Platypus(w/) | 41.07 | 58.28 | 64.50 | 45.08 | 52.23 | +7.01
Llama2-13B | Alpaca | 48.15 | 69.90 | 64.37 | 63.73 | 61.54 | —
$\rightarrow$Open-Platypus(w/o) | 28.23 | 61.92 | 64.12 | 54.58 | 52.21 | -9.33
$\rightarrow$Open-Platypus(w/) | 49.38 | 69.80 | 65.72 | 63.05 | 61.99 | +9.78
(b) General Performance of Evaluation Tasks on SAM (w/o) & (w/) on Different
Model Sizes.

Method | DK | Understanding | Reasoning | Exams | AVG | $\bm{\Delta}$
---|---|---|---|---|---|---
Alpaca | 40.53 | 58.74 | 63.33 | 45.08 | 51.92 | —
$\rightarrow$Open-Platypus(w/o) | 31.13 | 50.70 | 61.09 | 37.97 | 45.22 | -6.70
$\rightarrow$Open-Platypus(w/) | 41.07 | 58.28 | 64.50 | 45.08 | 52.23 | +7.01
Wise-FT (w/o) | 37.75 | 56.64 | 62.65 | 47.12 | 51.04 | -0.88
Wise-FT (w/) | 40.59 | 58.14 | 64.56 | 44.75 | 52.01 | +0.97
Rehearsal (w/o) | 33.38 | 54.69 | 61.25 | 43.05 | 48.09 | -3.83
Rehearsal (w/) | 40.35 | 57.09 | 63.27 | 43.73 | 51.11 | +3.02
(c) General Performance of Evaluation Tasks on SAM Comparing/Combining with
Wise-FT and Rehearsal.

Table 2: General Performance of Evaluation Tasks on Four Analyses of SAM.

##### Implementation Details

For a fair comparison across the same pre-trained language model, our initial
phase follows Taori et al. (2023) to train an instruction fine-tuning model on
the Alpaca dataset. This baseline model underpins our subsequent fine-tuning
experiments on various datasets. Besides, all hyper-parameters and settings
are maintained uniformly in the following fine-tuning. We opt for the AdamW
Loshchilov and Hutter (2019) optimizer in scenarios without SAM. All
experiments are executed on 16 NVIDIA A800 GPUs.

##### Evaluation Tasks

To effectively assess the extent of the general knowledge retained in LLMs and
to quantify the degree of CF, we have implemented a collection of general
assessment tasks. These tasks are categorized as follows: (1) Domain knowledge
benchmark (DK): MMLU Hendrycks et al. (2021a), including STEM, Humanities,
Social Sciences, and Other domains. (2) Reasoning: SuperGlue Wang et al.
(2019) include $AX_{b}$, $AX_{g}$, RTE, COPA, and datasets like Hellaswag
Zellers et al. (2019), Boolq Clark et al. (2019), and Siqa Sap et al. (2019).
(3) Understanding: RACE (middle & high) Lai et al. (2017), Openbookqa-fact
Mihaylov et al. (2018), and Csl-dev Li et al. (2022). (4) Exams: ARC-c Clark
et al. (2018). Besides, we also conduct experiments on domain-specific
datasets like TruthfulQA Lin et al. (2022) to assess the degree of retention
of specific abilities. Refer to Appendix A.1.2 for more details.

##  5 Experimental Results

We demonstrate the superiority of our proposed method through four
experimental analyses:

  1. 1.

Mitigating CF on different datasets

  2. 2.

Mitigating CF across different model sizes

  3. 3.

Comparison with other advanced methods

  4. 4.

Complementary to existing works

The average results for 4 evaluation categories are presented in Tab. 2
(values in %), where “(w/o)” and “(w/)” refers to train without and with SAM,
respectively. The four experiments are as follows, and the total experimental
results can be refereed to the Appendix A.6.

##### Performance of SAM in Mitigating CF on different datasets

As presented in Tab. 2(a), for different CF settings (Alpaca as the baseline,
difficulty rank: ShareGPT52K$\leq$Open-Platypus$\leq$MetaMathQA), their
performance on evaluation tasks experiences a significant decline ($\Delta$
CF: ShareGPT: 11.71%; Open-Platypus: 12.90%; Meta-Math: 13.10%), confirming
that different degrees of CF occur in the fine-tuning phase. With the
introduction of SAM, the CF is markedly mitigated and the performance is
comparable to baseline. Notably, the Tab. 2 shows the results in balanced
training cost, i.e., (w/o) epoch=2 and (w/) epoch=1. For complete epoch
experiments and experiment of capability preseving in domain, refer to the
Appendix A.4 and Appendix A.5, respectively.

##### Performance of SAM on LLMs of Different Sizes

For the results in Tab. 2(b), we conduct Alpaca$\rightarrow$Open-Platypus
across models of different sizes, and they exhibited different degrees of
performance dropping (e.g., $\Delta$ DK: TinyLlama-1.1B: 0.52%; Llama2-7B:
23.19%; Llama2-13B: 41.37%), indicating that the severity of the forgetting
increasing with model size. Compared to the performance of SAM in mitigating
CF across models, SAM could significantly mitigate the extent of CF with model
size increasing.

##### In Comparison with Other Advanced Anti-Forgetting Methods

For the comparative experimental results represents in Tab.2(c), SAM
significantly outperforms Wise-FT and Rehearsal, although the other two also
achieve decent performance ($\Delta$ Mitigating CF: SAM: 7.01%; Wise-FT:
5.82%; Rehearsal: 3.79%).

##### Complementary to Existing Advanced Anti-Forgetting Methods

For the results in Table 2(c), SAM could effectively improve the performance
of CF along with the two methods ($\Delta$ Mitigating CF: Wise-FT(w/): 0.97%;
Rehearsal(w/): 3.02%), which illustrates that SAM can be orthogonally combined
with other methods, providing incremental benefits to mitigate CF.

##  6 Conclusion

We reexamine the phenomenon of Catastrophic Forgetting (CF) and reveal the
direct link between the flatness of the model loss landscape (LLS) and the
extent of CF in LLMs. Based on our findings, we introduce sharpness-aware
optimization to flatten the LLS, thereby mitigating CF. Experiments show its
effectiveness, especially in larger models, suggesting our method may become a
standard strategy to mitigate CF during LLMs tuning.

In future work, we would like to explore more strategies to orthogonally
address the forgetting problem in LLM tuning, such as advanced optimizers Sun
et al. (2024); Zhong et al. (2022); Yang et al. (2023), proper anti-forgetting
learning curricula Ding et al. (2021); Zhou et al. (2021); Faber et al.
(2024), and model merging techniques Li et al. (2023); He et al. (2023).

## Limitations

Despite the progress we made, there still exist limitations in our work. On
the one hand, our experiments are concentrated on a specific aspect of the
Catastrophic Forgetting (CF) — the direct link between the flatness of the
model’s loss landscape and CF. While this finding makes us introduce the
sharpness-aware minimization technique to flatten the loss landscape and
thereby mitigate CF, it also means that other potentially influential factors
in CF have not been comprehensively explored. On the other hand, our research
primarily addresses CF during the fine-tuning phase of LLMs, but it does not
directly tackle CF that may occur during other stages. For instance, CF can
also be a significant challenge when models are updated with new data post-
deployment. Therefore, the scope of our findings and the applicability of our
solution may be limited to specific phases of the LLMs lifecycle.

## Ethics Statement

We take ethical considerations seriously and strictly adhere to the ACL Ethics
Policy. This paper revisits a major obstacle of fine-tuning the large language
models – Catastrophic Forgetting, and provides an optimization-level solution
to alleviate it. All employed models and datasets in this paper are publicly
available and have been widely adopted by researchers. All experimental
results upon these open models and datasets are reported accurately and
objectively. Thus, we believe that this research will not pose any ethical
issues.

##### Reproducibility.

We provide the detailed experimental setup in the Appendix, such as hyper-
parameters and statistic descriptions. Also, we release code to facilitate
reproducing the experimental results: https://github.com/Li-
Hyn/LLM_CatastrophicForgetting.

## References

  * Alva-Manchego et al. (2020) Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoît Sagot, and Lucia Specia. 2020.  Asset: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations.  _arXiv preprint_.
  * Bi et al. (2024) Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024.  Deepseek llm: Scaling open-source language models with longtermism.  _arXiv preprint_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  In _NeurIPS_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _blog_.
  * Clark et al. (2019) Clark, Christopher, Lee, Kenton, Chang, Ming-Wei, Kwiatkowski, Tom, Collins, Michael, and Kristina Toutanova. 2019.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In _NAACL_.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.  _arXiv preprint_.
  * Datta et al. (2023) Arghya Datta, Subhrangshu Nandi, Jingcheng Xu, Greg Ver Steeg, He Xie, Anoop Kumar, and Aram Galstyan. 2023.  Measuring and mitigating local instability in deep neural networks.  _arXiv preprint_.
  * Ding et al. (2021) Liang Ding, Longyue Wang, Xuebo Liu, Derek F Wong, Dacheng Tao, and Zhaopeng Tu. 2021.  Progressive multi-granularity training for non-autoregressive translation.  In _Findings of ACL_.
  * Dziugaite and Roy (2017) Gintare Karolina Dziugaite and Daniel M. Roy. 2017.  Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data.  In _UAI_.
  * Faber et al. (2024) Kamil Faber, Dominik Zurek, Marcin Pietron, Nathalie Japkowicz, Antonio Vergari, and Roberto Corizzo. 2024.  From mnist to imagenet and back: benchmarking continual curriculum learning.  _Machine Learning_.
  * Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer. 2022.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _JMLR_.
  * Foret et al. (2021) Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2021.  Sharpness-aware minimization for efficiently improving generalization.  In _ICLR_.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007.  The third pascal recognizing textual entailment challenge.  In _ACL-PASCAL_.
  * Hao et al. (2019) Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019.  Visualizing and understanding the effectiveness of bert.  In _EMNLP-IJCNLP_.
  * He et al. (2023) Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. 2023.  Merging experts into one: Improving computational efficiency of mixture of experts.  In _EMNLP_.
  * Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a.  Measuring massive multitask language understanding.  In _ICLR_.
  * Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b.  Measuring mathematical problem solving with the MATH dataset.  In _NeurIPS_.
  * Jiang et al. (2020) Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020.  Neural crf model for sentence alignment in text simplification.  _arXiv preprint_.
  * Keskar et al. (2017) Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2017.  On large-batch training for deep learning: Generalization gap and sharp minima.  In _ICLR_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _PNAS_.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017.  RACE: large-scale reading comprehension dataset from examinations.  In _EMNLP_.
  * Lee et al. (2023) Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. 2023.  Platypus: Quick, cheap, and powerful refinement of llms.  _arXiv preprint_.
  * Li et al. (2023) Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. 2023.  Deep model fusion: A survey.  _arXiv preprint_.
  * Li et al. (2022) Yudong Li, Yuqing Zhang, Zhe Zhao, Linlin Shen, Weijie Liu, Weiquan Mao, and Hui Zhang. 2022.  Csl: A large-scale chinese scientific literature dataset.  _arXiv preprint_.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Truthfulqa: Measuring how models mimic human falsehoods.  In _ACL_.
  * Lin et al. (2023) Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. 2023.  Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models.  _arXiv preprint_.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _ICLR_.
  * Lu et al. (2023) Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2023.  Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt.  _arXiv preprint_.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _EMNLP_.
  * Peng et al. (2023) Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023.  Towards making the most of chatgpt for machine translation.  In _Findings of EMNLP_.
  * Ren et al. (2024) Zhiyao Ren, Yibing Zhan, Baosheng Yu, Liang Ding, and Dacheng Tao. 2024.  Healthcare copilot: Eliciting the power of general llms for medical consultation.  _arXiv preprint_.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019.  Socialiqa: Commonsense reasoning about social interactions.  _arXiv preprint_.
  * Scialom et al. (2022) Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022.  Fine-tuned language models are continual learners.  In _EMNLP_.
  * Shi et al. (2023) Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023.  Detecting pretraining data from large language models.  In _NeurIPS_.
  * Sun et al. (2024) Hao Sun, Li Shen, Qihuang Zhong, Liang Ding, Shixiang Chen, Jingwei Sun, Jing Li, Guangzhong Sun, and Dacheng Tao. 2024.  Adasam: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks.  _Neural Networks_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  _Stanford blog_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _NeurIPS_.
  * Yang et al. (2023) Enneng Yang, Li Shen, Zhenyi Wang, Shiwei Liu, Guibing Guo, and Xingwei Wang. 2023.  Data augmented flatness-aware gradient projection for continual learning.  In _ICCV_.
  * Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023.  Metamath: Bootstrap your own mathematical questions for large language models.  _arXiv preprint_.
  * Zan et al. (2022) Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022.  On the complementarity between pre-training and random-initialization for resource-rich machine translation.  In _COLING_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence? In _ACL_.
  * Zeng et al. (2023) Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023.  Agenttuning: Enabling generalized agent abilities for llms.  _arXiv preprint_.
  * Zhang et al. (2024) Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024.  Tinyllama: An open-source small language model.  _arXiv preprint_.
  * Zhong et al. (2023) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023.  Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert.  _arXiv preprint_.
  * Zhong et al. (2022) Qihuang Zhong, Liang Ding, Li Shen, Peng Mi, Juhua Liu, Bo Du, and Dacheng Tao. 2022.  Improving sharpness-aware minimization with fisher mask for better generalization on language models.  In _Findings of EMNLP_.
  * Zhou et al. (2021) Lei Zhou, Liang Ding, Kevin Duh, Shinji Watanabe, Ryohei Sasano, and Koichi Takeda. 2021.  Self-guided curriculum learning for neural machine translation.  In _IWSLT_.

##  Appendix A Appendix

###  A.1 Details of Training and Evaluation Datasets

As outlined in §4 and §2, our study involves extensive experiments across five
widely-utilized datasets. Additionally, we assess our model’s performance on
selected tasks from the SuperGLUE benchmark along with other distinct
datasets. In this section, we provide detailed descriptions of the datasets
and tasks employed. The description of each task includes:

####  A.1.1 Training Datasets

  1. 1.

Alpaca: Alpaca Taori et al. (2023) is a dataset of 52,000 instructions and
demonstrations generated by OpenAI’s text-davinci-003 engine. This instruction
data can be used to conduct instruction-tuning for language models and make
the language model follow instruction better.

  2. 2.

ShareGPT52K: ShareGPT52K Chiang et al. (2023) is a collection of approximately
52,000 conversations scraped via the ShareGPT API. These conversations include
both user prompts and responses from OpenAI’s ChatGPT.

  3. 3.

MetaMathQA: MetaMathQA Yu et al. (2023) is a dataset of 395,000 instructions
which augmented from the training sets of GSM8K Cobbe et al. (2021) and MATH
Hendrycks et al. (2021b).

  4. 4.

Open-Platypus: Open-Platypus Lee et al. (2023) is focused on improving LLM
logical reasoning skills and has the item amount to 24,900.

  5. 5.

Auto-Wiki: Auto-Wiki Jiang et al. (2020); Alva-Manchego et al. (2020) provides
a set of aligned sentences from English Wikipedia and Simple English Wikipedia
as a resource to train sentence simplification systems.

Epoch | Dataset | DK | Understanding | Reasoning | Exams | AVG
---|---|---|---|---|---|---
3 | Alpaca | 40.53 | 58.74 | 63.33 | 45.08 | 51.92
2 |  $\rightarrow$ShareGPT(w/o) | 26.08 | 52.84 | 58.68 | 45.76 | 45.84
1 |  $\rightarrow$ShareGPT(w/o) | 24.98 | 52.40 | 59.12 | 45.08 | 45.39
1 |  $\rightarrow$ShareGPT(w/) | 40.08 | 57.91 | 63.78 | 44.41 | 51.55
2 |  $\rightarrow$Open-Platypus(w/o) | 31.13 | 50.70 | 61.09 | 37.97 | 45.22
1 |  $\rightarrow$Open-Platypus(w/o) | 29.89 | 49.37 | 60.60 | 40.34 | 45.05
1 |  $\rightarrow$Open-Platypus(w/) | 41.07 | 58.28 | 64.50 | 45.08 | 52.23
2 |  $\rightarrow$Meta-Math(w/o) | 33.13 | 52.61 | 58.46 | 36.27 | 45.12
1 |  $\rightarrow$Meta-Math(w/o) | 32.84 | 52.69 | 58.43 | 38.64 | 45.65
1 |  $\rightarrow$Meta-Math(w/) | 34.79 | 55.77 | 62.38 | 42.71 | 48.91
Table 3: General Performance of Different Epochs on Llama2-7b. Dataset | Hellaswag | TruthfulQA | AVG
---|---|---|---
Alpaca | 65.00 | 22.64 | 43.82
$\rightarrow$Open-Platypus(w/o) | 73.16 | 23.87 |  $\text{48.52}^{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\uparrow\textbf{+4.7}}}$
$\rightarrow$Open-Platypus(w/) | 73.49 | 23.99 |  $\text{48.74}^{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\uparrow\textbf{+4.9}}}$
Table 4: Performance in Fine-tuning Domain.

####  A.1.2 Evaluation Tasks

  1. 1.

MMLU: Massive Multitask Language Understanding benchmark (MMLU) Hendrycks et
al. (2021a) to evaluate the knowledge stored in the LLMs, which can be divided
into STEM, Human, Social, and Other.

  2. 2.

$AX_{b}$: AX-b Wang et al. (2019) is a broad-coverage diagnostic task, which
requires to determine the logical relation between the given sentence pair,
with three relations: entailment, contradiction and neutral.

  3. 3.

$AX_{g}$: AX-g Wang et al. (2019) is a Winogender diagnostic task, which
requires to determine which noun the pronoun refers to according to the given
sentence and pronoun.

  4. 4.

RTE: Recognizing Textual Entailment Giampiccolo et al. (2007), given a premise
and a hypothesis, is a task to predict whether the premise entails the
hypothesis.

  5. 5.

COPA: COPA Wang et al. (2019) is a causal inference task, which requires to
select the correct causal relation based on the given premise.

  6. 6.

Hellaswag: HellaSwag Zellers et al. (2019) is a challenge dataset for
evaluating commonsense natural language inference, which is specially hard for
state-of-the-art models, though its questions are trivial for humans (>95%
accuracy).

  7. 7.

Boolq: Boolean Question Clark et al. (2019) is a question-answering task where
each sample consists of a short passage and a yes/no question about the
passage.

  8. 8.

Siqa: Siqa Sap et al. (2019) is a social interaction question-answering task,
which requires selecting the most reasonable behaviour based on the given
scenario and three possible subsequent behaviours.

  9. 9.

RACE: RACE Lai et al. (2017) is a large-scale reading comprehension dataset
with more than 28,000 passages and nearly 100,000 questions.

  10. 10.

Openbookqa-fact: OpenBookQA Mihaylov et al. (2018) contains questions that
require multi-step reasoning, application of common-sense knowledge, and in-
depth comprehension of text.

  11. 11.

Csl-dev: CSL Li et al. (2022) is a large-scale Chinese Scientific Literature
dataset, which contains the titles, abstracts, keywords and academic fields of
396k papers

  12. 12.

ARC-c: The AI2’s Reasoning Challenge (ARC) Clark et al. (2018) is a multiple-
choice question-answering dataset, containing questions from science exams
from grade 3 to grade 9.

  13. 13.

TruthfulQA: TruthfulQA Lin et al. (2022) is a benchmark to measure whether a
language model is truthful in generating answers to questions.

###  A.2 Experimental Setup

In this paper, we trained from the base model (TinyLlama, Llama2-7b, and
Llama2-13B). For a fair comparison, we follow the setup in Taori et al. (2023)
in all first phases of experiments, that is, the number of epochs is 3, and
the learning rate (lr) is set as 2e-5, with the batch size of 128. We
confirmed that the Alpaca model, trained under this configuration, exhibits
performance roughly equivalent to that of the original model as reported in
the Taori et al. (2023).

For the subsequent training procedures, we aligned our fine-tuning
configuration with that used in real-world downstream tasks. Taking into
account the characteristic of SAM performing two forward passes, to facilitate
a fair comparison, we established our experimental parameters as follows:
without SAM: epochs = 2, lr = 5e-6, batch size = 128, and the optimizer is
AdamW; with SAM: epochs = 1, lr = 5e-6, batch size = 128. For the selection of
the $\rho$ parameter within the SAM training process, we suggest a reference
value of $\rho=2$.

Epoch | Dataset | STEM | Humanities | Social Sciences | Other
---|---|---|---|---|---
3 | Alpaca | 31.60 | 43.03 | 47.23 | 43.94
2 |  $\rightarrow$ShareGPT(w/o) | 22.69 | 29.72 | 23.57 | 29.20
1 |  $\rightarrow$ShareGPT(w/o) | 22.55 | 30.35 | 22.96 | 24.83
1 |  $\rightarrow$ShareGPT(w/) | 31.08 | 42.82 | 47.24 | 42.96
2 |  $\rightarrow$Open-Platypus(w/o) | 25.44 | 34.70 | 34.20 | 32.50
1 |  $\rightarrow$Open-Platypus(w/o) | 24.95 | 33.77 | 31.58 | 31.21
1 |  $\rightarrow$Open-Platypus(w/) | 32.18 | 43.56 | 48.11 | 44.15
2 |  $\rightarrow$Meta-Math(w/o) | 26.57 | 34.77 | 39.93 | 34.19
1 |  $\rightarrow$Meta-Math(w/o) | 28.22 | 33.45 | 37.74 | 34.03
1 |  $\rightarrow$Meta-Math(w/) | 28.48 | 34.84 | 39.50 | 38.82
(a) DK Performance of Different Epochs on Llama2-7b.

|  | Understanding | Exams
---|---|---|---
Epoch | Dataset | RACE (middle) | RACE (high) | Openbookqa-fact | Csl-dev | ARC-c
3 | Alpaca | 55.22 | 55.22 | 66.4 | 58.13 | 45.08
2 |  $\rightarrow$ShareGPT(w/o) | 50.7 | 47.4 | 57 | 56.25 | 45.76
1 |  $\rightarrow$ShareGPT(w/o) | 49.86 | 47.74 | 57 | 55 | 45.08
1 |  $\rightarrow$ShareGPT(w/) | 55.43 | 51.49 | 66.6 | 58.13 | 44.41
2 |  $\rightarrow$Open-Platypus(w/o) | 46.31 | 44.54 | 53.8 | 58.13 | 37.97
1 |  $\rightarrow$Open-Platypus(w/o) | 46.52 | 44.17 | 51.8 | 55 | 40.34
1 |  $\rightarrow$Open-Platypus(w/) | 56.34 | 51.69 | 67.6 | 57.5 | 45.08
2 |  $\rightarrow$Meta-Math(w/o) | 56.55 | 51.8 | 45.2 | 56.88 | 36.27
1 |  $\rightarrow$Meta-Math(w/o) | 54.67 | 52.03 | 47.8 | 56.25 | 38.64
1 |  $\rightarrow$Meta-Math(w/) | 53.2 | 49.69 | 65.2 | 55 | 42.71
(b) Performance of Understanding and Exams Different Epochs on Llama2-7b.

Epoch | Dataset | BoolQ | AX_b | AX_g | RTE | COPA | Hellaswag | Siqa
---|---|---|---|---|---|---|---|---
3 | Alpaca | 78.29 | 59.42 | 60.67 | 61.37 | 65.00 | 65.00 | 53.53
2 |  $\rightarrow$ShareGPT(w/o) | 75.20 | 57.70 | 50.00 | 47.65 | 60.00 | 73.81 | 46.42
1 |  $\rightarrow$ShareGPT(w/o) | 73.52 | 57.88 | 50.00 | 49.46 | 64.00 | 72.48 | 46.52
1 |  $\rightarrow$ShareGPT(w/) | 77.92 | 59.33 | 56.46 | 61.37 | 65.00 | 73.18 | 53.22
2 |  $\rightarrow$Open-Platypus(w/o) | 78.20 | 59.24 | 50.00 | 50.18 | 69.00 | 73.16 | 47.85
1 |  $\rightarrow$Open-Platypus(w/o) | 75.72 | 58.70 | 50.00 | 49.10 | 70.00 | 72.29 | 48.36
1 |  $\rightarrow$Open-Platypus(w/) | 79.11 | 59.06 | 63.76 | 58.48 | 65.00 | 73.49 | 52.61
2 |  $\rightarrow$Meta-Math(w/o) | 69.60 | 58.42 | 50.00 | 47.65 | 64.00 | 72.09 | 47.49
1 |  $\rightarrow$Meta-Math(w/o) | 70.61 | 58.33 | 50.00 | 47.65 | 62.00 | 71.90 | 48.52
1 |  $\rightarrow$Meta-Math(w/) | 77.61 | 58.88 | 53.65 | 53.43 | 70.00 | 71.70 | 51.43
(c) Reasoning Performance of Different Epochs on Llama2-7b.

Table 5:  Comprehensive Experimental Results of Different Epochs on Llama2-7b.

###  A.3 Flatness Metric

Consider a function $f(x_{i},y_{j})$, where $x_{i}$ and $y_{j}$ represent two
arbitrary parameters $param_{1}$ and $param_{2}$, and the N represents the
total number of points in this area.

Surface Curvature
Surface curvature (SC) typically refers to a measure that describes the degree
of curvature of a surface at a certain point or area. A lower average
curvature usually indicates a flatter loss landscape. To quantify the average
curvature of a loss surface defined by two parameters, the approximate
quantification of the curvature of the function $f(x_{i},y_{j})$ within its
parameter space is as follows:

$K\left(x_{i},y_{j}\right)=f_{xx}\left(x_{i},y_{j}\right)+f_{yy}\left(x_{i},y_{j}\right).$
(3)

We define the overall curvature as the average curvature, which is computed by
taking the mean of the absolute values of the curvature across the entire
parameter space, formulated as:

$SC(x,y)=\frac{1}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{N}\left|K\left(x_{i},y_{j}\right)\right|.$
(4)

Average Gradient
We take average gradient (AG) serve as an indicator of the flatness of loss
landscape, with smaller AG typically corresponding to flatter regions of the
loss landscape. The AG refers to the average of the gradient vectors
calculated within a certain region of the function, expressed as follows:

$\text{ AG }=\frac{1}{N}\sum_{i=1}^{N}\nabla f\left(x_{i},y_{i}\right).$ (5)

Mean Absolute Gradient
To more comprehensively evaluate the flatness of LLS, we introduce the metric
Mean Absolute Gradient (MAG). With lower MAG, the loss landscape tends to be
flatter. MAG quantifies the average rate of change of the loss function in its
parameter space. The function could be expressed as follows:

$\text{ MAG }=\frac{1}{N-1}\sum_{i=2}^{N}\left|t_{i}-t_{i-1}\right|,$ (6)

where $t_{i}$ represents any data point in LLS.

###  A.4 Experiment on different Epochs

Given the need for a fair comparison, the number of epochs for training
differs between SAM without (w/o) and with (w/) settings. Consequently, it is
imperative to fully present the results for SAM (w/o) at epoch = 1, to
substantiate the rigor of our experimental methodology.

The results in Tab. 3 clearly show that in most cases the results at epoch=2
are superior to those at epoch=1. Furthermore, across all three settings of
the same dataset, SAM (w/) consistently achieved the best results. These
experimental results further demonstrate the superiority of our method. Also,
to further present the result, we give the complete results of all datasets in
Tab. 5.

Models | Dataset | STEM | Humanities | Social Sciences | Other
---|---|---|---|---|---
TinyLlama | Alpaca | 22.18 | 24.03 | 22.88 | 23.84
$\rightarrow$Open-platypus(w/o) | 22.29 | 23.55 | 22.59 | 23.90
$\rightarrow$Open-platypus(w/) | 21.96 | 24.16 | 22.53 | 23.92
Llama2-13b | Alpaca | 39.58 | 52.29 | 55.83 | 48.75
$\rightarrow$Open-platypus(w/o) | 24.58 | 33.23 | 27.42 | 28.98
$\rightarrow$Open-platypus(w/) | 39.51 | 55.59 | 56.92 | 49.84
(a) Dk Performance of Different sizes of LLMs.

Models | Dataset | Understanding | Exams
---|---|---|---
RACE (middle) | RACE (high) | Openbookqa-fact | Csl-dev | ARC-c
TinyLlama | Alpaca | 22.91 | 21.90 | 25.80 | 51.88 | 26.78
$\rightarrow$Open-Platypus(w/o) | 24.09 | 22.27 | 26.00 | 51.88 | 27.12
$\rightarrow$Open-Platypus(w/) | 22.63 | 21.84 | 25.2 | 51.88 | 26.10
Llama2-13b | Alpaca | 72.84 | 68.21 | 80.40 | 58.13 | 63.73
$\rightarrow$Open-Platypus(w/o) | 71.10 | 68.67 | 50.40 | 57.5 | 54.58
$\rightarrow$Open-Platypus(w/) | 72.49 | 68.15 | 79.8 | 58.75 | 63.05
(b) Performance of Understanding and Exams of Different sizes of LLMs.

Models | Dataset | BoolQ | AX_b | AX_g | RTE | COPA | Hellaswag | Siqa
---|---|---|---|---|---|---|---|---
TinyLlama | Alpaca | 43.98 | 45.56 | 50.28 | 52.71 | 53.00 | 43.91 | 33.67
$\rightarrow$ Open-Platypus(w/o) | 45.47 | 49.28 | 50.00 | 50.90 | 53.00 | 42.86 | 32.09
$\rightarrow$Open-Platypus(w/) | 44.50 | 52.99 | 50.28 | 53.79 | 52.00 | 42.75 | 32.09
Llama2-13b | Alpaca | 85.41 | 52.54 | 55.34 | 49.46 | 68.00 | 77.61 | 62.23
$\rightarrow$Open-Platypus(w/o) | 83.27 | 56.07 | 55.90 | 48.38 | 72.00 | 77.17 | 56.04
$\rightarrow$Open-Platypus(w/) | 85.57 | 52.63 | 55.62 | 50.18 | 68.00 | 85.53 | 62.54
(c) Reasoning Performance of Different sizes of LLMs.

Table 6:  Comprehensive Experimental Results across Different sizes of LLMs.

Method | Dataset | STEM | Humanities | Social Sciences | Other
---|---|---|---|---|---
Wife-FT(w/o) |  Alpaca$\rightarrow$Open-Platypus | 29.95 | 41.04 | 42.77 | 40.42
Rehearsal(w/o) | 26.98 | 35.00 | 38.32 | 35.90
(a) DK Performance of Comparison Methods.

Models | Dataset | Understanding | Exams
---|---|---|---
RACE (middle) | RACE (high) | Openbookqa-fact | Csl-dev | ARC-c
Wife-FT(w/o) | Alpaca$\rightarrow$Open-Platypus | 53.48 | 50.97 | 64.60 | 57.50 | 47.12
Rehearsal(w/o) | 53.13 | 50.74 | 58.00 | 56.88 | 43.05
(b) Performance of Understanding and Exams of Comparison Methods.

Method | Dataset | BoolQ | AX_b | AX_g | RTE | COPA | Hellaswag | Siqa
---|---|---|---|---|---|---|---|---
Wife-FT(w/o) | Alpaca$\rightarrow$Open-Platypus | 77.77 | 59.24 | 51.97 | 57.76 | 64.00 | 73.96 | 53.84
Rehearsal(w/o) | 78.01 | 60.78 | 50.00 | 51.99 | 66.00 | 73.43 | 48.52
(c) Reasoning Performance of Comparison Methods.

Table 7:  Comprehensive Experimental Results of Comparison with Other Methods.

Method | Dataset | STEM | Humanities | Social Sciences | Other
---|---|---|---|---|---
Wife-FT(w/) |  Alpaca$\rightarrow$Open-Platypus | 31.81 | 43.1 | 47.3 | 43.8
Rehearsal(w/) | 32.13 | 42.53 | 46.97 | 43.19
(a) DK Performance of Methods in Combination with SAM.

Models | Dataset | Understanding | Exams
---|---|---|---
RACE (middle) | RACE (high) | Openbookqa-fact | Csl-dev | ARC-c
Wife-FT(w/) | Alpaca$\rightarrow$Open-Platypus | 56.06 | 51.6 | 67.40 | 57.50 | 44.75
Rehearsal(w/) | 54.25 | 50.40 | 66.20 | 57.50 | 43.73
(b) Performance of Understanding and Exams in Combination with SAM.

Method | Dataset | BoolQ | AX_b | AX_g | RTE | COPA | Hellaswag | Siqa
---|---|---|---|---|---|---|---|---
Wife-FT(w/) | Alpaca$\rightarrow$Open-Platypus | 78.69 | 58.97 | 60.96 | 61.73 | 65.00 | 73.67 | 52.87
Rehearsal(w/) | 76.70 | 57.97 | 55.62 | 60.65 | 67.00 | 72.75 | 52.20
(c) Reasoning Performance in Combination with SAM.

Table 8:  Comprehensive Experimental Results of Combination with Other
Methods.

###  A.5 Performance in fine-tuning domain

Results resented in in Tab. 4 further validate the capability preservation in
domain, which illustrates SAM effectively mitigates the CF with a limited
impact on fine-tuning performance in specific domains.

###  A.6 Comprehensive Experimental Results Across Evaluation Datasets

####  A.6.1 Comprehensive Experimental Results on different datasets

Based on the Llama2-7B model, our comprehensive experimental results on
different datasets are presented in Tab. 5, which presents the performance of
the model across different evaluation tasks during the fine-tuning stage.

####  A.6.2 Comprehensive Experimental Results across different model sizes

On the Open-Platypus dataset, we conduct experiments across models of
different sizes, with comprehensive experimental results displayed in Tab. 6,
which demonstrates the generality of the SAM across models of different sizes.
As the comprehensive results of llama2-7B have already been presented in Tab.
5, they will not be reiterated in Tab. 6.

####  A.6.3 Comprehensive Experimental Results of comparison with other
advanced methods

We conduct experiments to compare SAM with Wise-FT and Rehearsal on Llama-7B
model, with comprehensive experimental results displayed in Tab. 7, which
indicates SAM significantly outperforms Wise-FT and Rehearsal. Since the
performance of SAM has been comprehensively demonstrated in Tab. 5, here we
only additionally present the results of Wise-FT(w/o) and Rehearsal(w/o).

####  A.6.4 Comprehensive Experimental Results of complementary to existing
works

To further demonstrate the orthogonality of SAM, we combine SAM with existing
anti-forgetting methods. The results, as shown in Tab. 8, indicate that our
method provides incremental benefits to mitigate CF. Similarly, we only
present the results of Wise-FT(w/) and Rehearsal(w/) combined with SAM.