# NTK-Guided Few-Shot Class Incremental Learning

Jingren Liu, Zhong Ji, Senior Member, IEEE, Yanwei Pang, Senior Member, IEEE,
YunLong Yu This work was supported by the National Key Research and
Development Program of China (Grant No. 2022ZD0160403), and the National
Natural Science Foundation of China (NSFC) under Grant 62176178 (Corresponding
author: Zhong Ji).Zhong Ji and Yanwei Pang are with the School of Electrical
and Information Engineering, Tianjin Key Laboratory of Brain-Inspired
Intelligence Technology, Tianjin University, Tianjin 300072, China, and also
with the Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China
(e-mail<EMAIL_ADDRESS>pyw@tju.edu.cn).Jingren Liu is with the School of
Electrical and Information Engineering, Tianjin Key Laboratory of Brain-
Inspired Intelligence Technology, Tianjin University, Tianjin 300072, China
(e-mail: jrl0219@tju.edu.cn).YunLong Yu is with the College of Information
Science and Electronic Engineering, Zhejiang University, Hangzhou, 310027,
China. (e-mail: yuyunlong@zju.edu.cn).

###### Abstract

While anti-amnesia FSCIL learners often excel in incremental sessions, they
tend to prioritize mitigating knowledge attrition over harnessing the model’s
potential for knowledge acquisition. In this paper, we delve into the
foundations of model generalization in FSCIL through the lens of the Neural
Tangent Kernel (NTK). Our primary design focus revolves around ensuring
optimal NTK convergence and NTK-related generalization error, serving as the
theoretical bedrock for exceptional generalization. To attain globally optimal
NTK convergence, we employ a meta-learning mechanism grounded in mathematical
principles to guide the optimization process within an expanded network.
Furthermore, to reduce the NTK-related generalization error, we commence from
the foundational level, optimizing the relevant factors constituting its
generalization loss. Specifically, we initiate self-supervised pre-training on
the base session to shape the initial network weights. Then they are carefully
refined through curricular alignment, followed by the application of dual NTK
regularization tailored specifically for both convolutional and linear layers.
Through the combined effects of these measures, our network acquires robust
NTK properties, significantly enhancing its foundational generalization. On
popular FSCIL benchmark datasets, our NTK-FSCIL surpasses contemporary state-
of-the-art approaches, elevating end-session accuracy by 2.9% to 8.7%.

###### Index Terms:

Few-shot Class-Incremental Learning, Neural Tangent Kernel, Generalization,
Self-supervised Learning.

##  I Introduction

Few-shot class-incremental learning (FSCIL) extends class-incremental learning
(CIL) by facilitating the continuous learning of incremental classes with only
a few data samples. Drawing inspiration from established CIL approaches [1, 2,
3], contemporary FSCIL methods [4, 5, 6, 7] primarily emphasize safeguarding
the pre-established knowledge in every session, specifically addressing the
catastrophic forgetting problem. In contrast to CIL, which undergoes training
in every session, the FSCIL learner is exclusively trained in the base
session. Parameter updates for the FSCIL learner do not occur during
subsequent incremental sessions. Consequently, the effectiveness of the
subsequent incremental sessions relies heavily on the generalization
capabilities developed during the training of the base session. Despite its
significance, the emphasis on model generalization has been somewhat
overlooked in existing FSCIL literature. To address this gap, we introduce the
concept of the “Neural Tangent Kernel (NTK)” and highlight its pivotal role in
understanding the generalization properties of neural networks in the FSCIL
context.

######  Definition 1 (Neural Tangent Kernel)

For a typical ConvNet with randomly initialized parameters $\theta_{0}$, NTK
is given by $\Phi_{0}(x,x^{\prime})=\left\langle\frac{\partial
f(\theta_{0},x)}{\partial\theta_{0}},\frac{\partial
f(\theta_{0},x^{\prime})}{\partial\theta_{0}}\right\rangle$, and it converges
to a fixed kernel as the width $l$ approaches infinity and the different
inputs $x$ and $x^{\prime}$ undergo constant alteration,

$\displaystyle\Phi(\cdot)=\lim_{l\rightarrow\infty}\Phi_{0}(\cdot),$ (1)

where $f$ denotes the neural network, $\frac{\partial
f(\theta,\cdot)}{\partial\theta}$ is the neural network jacobian [8],
$\Phi(\cdot)$ signifies the fully converged NTK matrix and $l$ indicates the
model’s width.

A fundamental insight from the NTK corpus posits that _increasing the
network’s width tends to facilitate the NTK’s convergence to a stable matrix
during optimization, thereby bolstering generalization capabilities_. This
phenomenon suggests that post-optimization, the outputs of a network with
infinite width become robust to variations in parameters and inputs,
effectively capturing the inherent structures of the input data to produce
reliable and consistent outputs for both familiar and novel data. Such a
mechanism significantly mitigates the risk of overfitting, enhancing the
model’s aptitude for generalizing to unseen data and advancing it towards an
ideal state of generalization.

As emphasized in Definition 1, it’s noted that incrementally broadening the
network width simplifies the path to NTK convergence, diminishing the reliance
on manual tuning during the NTK optimization process. However, it’s imperative
to acknowledge that even before achieving a theoretically infinite width,
manual interventions may be necessary, often incurring considerable
computational demands as delineated in preceding research. For instance, the
work by [9] underscores the computational intensity of computing the NTK for
randomized data through ConvNets, requiring up to 200 gigabytes of GPU memory,
thus posing a substantial challenge for a wide array of applications.
Moreover, while the concept of an infinitely wide network is conceptually
appealing, its practical application remains constrained by real-world
limitations.

Therefore, incorporating NTK into FSCIL presents the challenge of ensuring
that a finite-width network exhibits NTK properties akin to those of an
infinitely wide network, even with limited optimization resources. Ideally, an
infinitely wide network would demonstrate both excellent NTK convergence [10,
11, 12] and a generalization loss approaching zero after optimization [13, 11,
14, 15]. Our goal is to replicate these properties within the constraints of a
finite-width network.

Guided by these insights, our work focuses on addressing two critical
issues—NTK Convergence and NTK Generalization—and seamlessly integrating them
within the FSCIL framework. These methods are approached through the lens of
the equivalence between meta-learning optimization and NTK convergence from
Theorem 1 and Theorem 2, as well as through the NTK-related generalization
loss in Theorem 3.

To achieve NTK convergence in our expanded network during optimization, we
theoretically align NTK convergence with meta-learning optimization. This
alignment is based on the understanding that both meta-loss and meta-outputs
are intrinsically linked to the NTK matrix and labels. Ensuring effective
meta-learning performance in FSCIL is pivotal for attaining optimal NTK
convergence. Consequently, we propose a bespoke meta-learning strategy for
FSCIL task, aiming to ensure that initiating meta-training from scratch leads
to foundational FSCIL performance.

Simultaneously, to ensure optimal NTK dynamics and minimal generalization
loss, we incorporate self-supervised pre-training weights, curricular
alignment, and dual NTK regularization, in line with [16] and the FSCIL
guidelines. These strategies are chosen for their effectiveness in Eq. 3 and
Theorem 3. In the regularization segment, to reduce computational overhead, we
decompose the ConvNet and apply spectral regularization in convolutional
layers [16]. For the additional linear layers, we directly compute the NTK
matrix and impose a spectral constraint to its eigenvalues, compressing their
distribution to render it more consolidated.

In a nutshell, our contributions are threefold:

  * •

To the best of our knowledge, our work is the first to integrate NTK with
FSCIL, advancing model generalization through robust theoretical foundations.

  * •

To establish foundational NTK convergence during the FSCIL optimization, we
theoretically align NTK convergence with meta-learning optimization and
propose a bespoke meta-learning schema specifically tailored for the FSCIL
task, initializing meta-training from scratch.

  * •

Furthermore, to harvest optimal NTK dynamic outputs and the smallest NTK-
related generalization loss, drawing inspiration from [17, 16, 18, 19], we
employ self-supervised pre-training, curricular alignment, and dual NTK
regularization to achieve state-of-the-art results both theoretically and
empirically.

##  II Related Works

Neural Tangent Kernel. It is initially proposed in [20], which reveals the
relationship between the convergence of infinitely wide neural networks and
model generalization. Continuing this pioneering work, subsequent studies [21,
22] further extend these findings to the case of shallow neural networks.
Furthermore, recent works [13, 11, 14, 15] on highly over-parameterized neural
networks show that gradient descent can achieve zero training error even for
finite but sufficiently wide CNNs, revealing promising generalization
potential. For example, [13, 23] elucidate the correlation between network
width and generalization, demonstrating that over-parameterization and random
initialization enable models to converge linearly to the global minimum. Zhu
et al. [15] show that over-parameterized wide networks can fit samples in
polynomial time. Moreover, Bombari et al. [18] discuss model stability,
feature alignment and their relation to generalization error. Diverging from
these typical NTK approaches, our work seeks to further integrate FSCIL with
NTK dynamics and improve generalization capabilities.

Few-shot Class-Incremental Learning. It is proposed to address CIL in few-shot
context. In general, the current FSCIL approaches [24, 25, 26] principally are
alleviating catastrophic forgetting in incremental sessions. For example,
TOPIC [24] is the pioneer, which leverages neural gas structure to solidify
knowledge. CEC [4] employs episodic training via a context-propagated graph
model to enhance adaptability. Moreover, MCNet [27] introduces ensemble
learning to complement different memorized knowledge. Building upon these
foundations, subsequent methods [6, 7, 28, 29] have advanced via the class
extension techniques, which is to preemptively prevent forgetting in future
classes. For example, considered from forward compatibility, Zhou et al. [28]
assign virtual prototypes to preserve space for incremental classes. ALICE
[29] extends the virtual classes to limit and facilitates well-clustered
features via sophisticate-designed loss. Compared to them, our NTK-FSCIL
focuses more on the model generalization and alleviates catastrophic
forgetting from the side.

##  III NTK Theoretical Foundation in FSCIL

###  III-A FSCIL Problem Formulation

In a dynamic training environment featuring $N$ supervised tasks,
$\Upsilon_{t=1}^{N}$, the essence of FSCIL is the incremental learning of
these tasks without overlap. Specifically, the class sets $\Upsilon_{i}$ and
$\Upsilon_{j}$ are disjoint for all $i\neq j$. The base session (session 0)
provides an extensive dataset, while subsequent incremental sessions offer
only a limited number of samples. Each task $\Upsilon_{t}$ is configured in
the n-way m-shot k-query setting for meta-training, represented as:

$\Upsilon_{t}=(\mathbf{X}_{i},\mathbf{X}_{j},\mathbf{Y}_{i},\mathbf{Y}_{j}),$

The pair $[\mathbf{X}_{i},\mathbf{Y}_{i}]\in\mathbb{R}^{m\times h\times
w\times c},\mathbb{R}^{m\times p}$ signifies $m$ support samples along with
their corresponding labels. Here, $h$, $w$ and $c$ denote the image’s
dimensional indices, while $p$ is the dimension of the one-hot encoded labels.
Similarly, the pair $[\mathbf{X}_{j},\mathbf{Y}_{j}]\in\mathbb{R}^{k\times
h\times w\times c},\mathbb{R}^{k\times p}$ represents $k$ query samples
associated with their labels. Moreover, $[X^{t}_{i},Y^{t}_{i}]$ and
$[X^{t}_{j},Y^{t}_{j}]$ are all support and query samples in task $t$.

###  III-B NTK Dynamics in FSCIL

Within the realm of NTK theory, foundational works by [10, 30] have
illuminated the operational dynamics of fully connected neural networks,
structured as delineated below:

$u_{l}=\frac{\sigma_{w}W_{l}h_{l-1}}{\sqrt{Z_{l-1}}}+\sigma_{b}b_{l},\quad
h_{l}=Relu(u_{l})\quad(l=1,\ldots,L),$ (2)

with $W_{l}\in\mathbb{R}^{Z_{l}\times Z_{l-1}}$ and
$b_{l}\in\mathbb{R}^{Z_{l}}$ denoting weight matrices and bias vectors
initialized via $\mathcal{N}(0,1)$.

In the infinite-width limit ($Z_{l}\rightarrow\infty$), with finite samples
and depth, gradient descent targets a global minimum, encapsulating the NTK
regime. This regime in FSCIL yields the model for any input $x$ as:

$\displaystyle f(x)=f_{0}^{*}(x)+\Phi(x,X)(\Phi(X,X)+\lambda
I)^{-1}(Y-f_{0}^{*}(X)).$ (3)

Here, $f_{0}^{*}(x)$ represents the model’s initial weights. $\Phi$ represents
the NTK matrix for the base session, with $x$ denoting an individual sample
and $X$ embodying the aggregated dataset specific to the base session.

Remark: The NTK is applicable across various architectures, including ResNets
and Transformers [31], with the primary distinctions manifesting in the NTK
matrix structure.

###  III-C Generalization and NTK Analysis

To mathematically capture the essence of model generalization in FSCIL, we
adapt Theorem 1 and Theorem 2 from [11], alongside the Theorem 3 from [32].

######  Theorem 1 (NTK-related Meta-Learning Output)

Assuming negligible learning rates $\eta$ and $\lambda$, and given a network
width $l$ approaching infinity, the meta-outputs $F_{t}$ for inputs
$\mathbf{X}_{j}\in\mathbf{X}_{j}^{t}$, in relation to the training pairs
$(\mathbf{X}_{i},\mathbf{Y}_{i})\in(\mathbf{X}_{i}^{t},\mathbf{Y}_{i}^{t})$,
are highly likely to converge to a state describable by the NTK after
optimization:

$\begin{split}&F_{t}(\mathbf{X}_{j},\mathbf{X}_{i},\mathbf{Y}_{i})=G_{\Phi}^{\tau}(\mathbf{X}_{j},\mathbf{X}_{i},\mathbf{Y}_{i})+\Phi((\mathbf{X}_{j},\mathbf{X}_{i}),\\\
&\quad\quad(\mathbf{X}_{j}^{t},\mathbf{X}_{i}^{t}))T^{\eta}_{\Phi}(t)(\mathbf{Y}_{j}^{t}-G^{\tau}_{\Phi}(\mathbf{X}_{j}^{t},\mathbf{X}_{i}^{t},\mathbf{Y}_{i}^{t})),\end{split}$
(4)
$\widetilde{T}^{\lambda}_{\Phi}(\cdot,\tau)=\Phi(\cdot,\cdot)^{-1}(I-e^{-\lambda\Phi(\cdot,\cdot)\tau}),$
(5)
$G_{\Phi}^{\tau}(\mathbf{X}_{j},\mathbf{X}_{i},\mathbf{Y}_{i})=\Phi(\mathbf{X}_{j},\mathbf{X}_{i})\widetilde{T}^{\lambda}_{\Phi}(\mathbf{X}_{i},\tau)\mathbf{Y}_{i},$
(6)

where
$\Phi((\mathbf{X}_{j},\mathbf{X}_{i}),(\mathbf{X}_{j}^{t},\mathbf{X}_{i}^{t}))$
is a kernel function.

######  Theorem 2 (NTK-related Meta-Learning Convergence)

Define NTK matrix
$\Phi=\lim_{l\rightarrow\infty}\frac{1}{l}J(\theta_{0})J(\theta_{0})^{T}$ and
learning rate $\eta_{0}=\frac{2}{\sigma_{min}(\Phi)+\sigma_{max}(\Phi)}$. In a
randomly initialized ConvNet of progressive width $l$, for any infinitesimal
positive $\delta>0$, there exist positive real numbers R and $\lambda_{0}$
such that an upper bound on the training loss is assuredly maintained with a
probability exceeding $1-\delta$.

$\displaystyle\mathcal{L}(\theta_{t})$
$\displaystyle=\frac{1}{2}\|F_{\theta_{t}}(\mathbf{X}_{i},\mathbf{X}_{j},\mathbf{Y}_{i})-\mathbf{Y}_{j}\|_{2}^{2}$
$\displaystyle\leq\left(1-\frac{\eta_{0}\sigma_{min}(\Phi)}{3}\right)^{2t}\frac{R^{2}}{2}.$
(7)

######  Theorem 3 (NTK-related Generalization)

In the realm of FSCIL, for the base session, there exists an optimal function
$f^{*}(x)$, fundamentally influenced by the spectral characteristics of the
NTK. The expected generalization loss $L_{D}(f^{*})$ is dictated by these
spectral properties and can be quantified as:

$L_{D}(f^{*})=\frac{1}{1-\varepsilon}\sum_{i=0}^{\infty}\lambda_{i}w_{i}^{*2}\left(\frac{\beta}{\beta+N\lambda_{i}}\right)^{2}+\frac{\varepsilon}{1-\varepsilon}\sigma^{2},$
(8)

where $\beta$ represents a numerically determined parameter, $\varepsilon$ is
a function of the eigenvalues $\lambda_{i}$ of the NTK as well as the sample
size $N$, and $w_{i}^{*}$ symbolize the optimal weights corresponding to the
deconstructed NTK in RKHS. The $\beta$ and $\varepsilon$ satisfy the relations
outlined in the subsequent equations:

$\displaystyle\sum_{i=0}^{\infty}\frac{\lambda_{i}}{\beta+N\lambda_{i}}=1,\qquad\sum_{i=0}^{\infty}\frac{N\lambda_{i}^{2}}{(\beta+N\lambda_{i})^{2}}=\varepsilon.$
(9)

From the insights provided in Eq. 3, it becomes clear that the pivotal factors
influencing the optimization process during the base session are the initial
weights $f_{0}^{*}(x)$ and the alignment of logits $Y-f_{0}^{*}(X)$. To
address these critical elements, we adopt self-supervised pre-training
alongside curricular alignment as strategies for augmentation.

Ensuring optimal NTK convergence during the dynamic optimization phase
necessitates establishing a mathematical equivalence between NTK convergence
and meta-learning optimization within the context of FSCIL. Drawing upon
Theorem 2, we ascertain that the meta-outputs and meta-loss are significantly
influenced by the eigenvalue distribution of the NTK. This realization
underscores the importance of fostering meta-learning optimization to
facilitate NTK convergence, as highlighted in seminal works by [10, 11, 12].
Consequently, Subsec. IV is dedicated to elucidating strategies through which
the FSCIL learner can surmount prevailing obstacles and achieve convergence
within the meta-learning paradigm.

To minimize the generalization loss towards zero within the Reproducing Kernel
Hilbert Space (RKHS), it is recognized that an optimal generalization loss
depends on both the sample quantity and a stable eigenvalue spectrum [33, 34].
The formula
$\sum_{i=0}^{\infty}\lambda_{i}w_{i}^{*2}\left(\frac{\beta}{\beta+N\lambda_{i}}\right)^{2}$
underscores that an eigenvalue’s $\lambda_{i}$ impact on generalization loss
is adjusted by its multiplication with the corresponding optimal weight
$w_{i}^{*}$. This setup implies that larger eigenvalues, when coupled with
larger weights, can significantly contribute to increased loss, whereas the
influence of smaller eigenvalues is comparably minimal. Achieving uniformity
in the NTK’s eigenvalues — thereby avoiding extremes — ensures a more
equitable contribution across all eigenvalues, mitigating any disproportionate
effects on the total generalization loss. This balance is crucial for overall
loss reduction.

In general, enhancing the number of base class samples and carefully
moderating NTK eigenvalues are pivotal strategies for lowering generalization
loss. We adopt the strategy of enlarging virtual class samples alongside dual
NTK regularization to achieve this goal. These strategic measures collectively
fortify the FSCIL learner’s generalization efficiency, securing top-tier
performance both theoretically and practically during the base session
optimization.

##  IV Meta-Learning Convergence in FSCIL

As delineated in Theorem 1 and Theorem 2, it is evident that the NTK governs
the trajectory of meta-outputs during meta-training, with the NTK matrix’s
eigenvalue distribution shaping the meta-loss for backpropagation
optimization. Thus, when optimizing meta-loss through gradient descent, the
NTK is bound to achieve global convergence. At this point, NTK convergence is
equated to meta-learning optimization.

Grounded in the established equivalence between NTK convergence and meta-
learning optimization, we endeavor to steer FSCIL’s meta-learning towards
global optimal convergence, thereby ensuring the NTK’s convergence to its
optimal region. However, many existing FSCIL methods [4, 28, 5, 35] argue that
initiating meta-learning from scratch often fails to converge to the global
optimum, leading to suboptimal FSCIL performance. To find the reasons of this
performance degradation, we conduct a comprehensive study of the engineering
implementations of existing methods. From this study, we identify two primary
causes: (1) the inflexibility in logit combinations, and (2) the oversight of
scenarios similar to incremental sessions during training.

In response to the aforementioned two reasons, we reformulate the
classification loss, dividing it into two distinct facets as depicted in Eq.
10. The first component integrates the mixup mechanism with a superior margin-
based loss (elaborated in Subsec. V-B), addressing the rigid class-level
combinations during meta-training. Simultaneously, the second component
introduces an unsupervised few-shot loss to simulate testing operations and
enhance adaptability to scenarios similar to incremental sessions during
training.

$\mathcal{L}_{cls}=\leavevmode\hbox to24.99pt{\vbox
to14.67pt{\pgfpicture\makeatletter\hbox{\hskip 12.497pt\lower-7.33331pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\pgfsys@color@rgb@stroke{1}{0}{0}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{1,0.6,0.6}\pgfsys@color@rgb@fill{1}{0.6}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\pgfsys@color@rgb@stroke{1}{0}{0}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{1,0.6,0.6}\pgfsys@color@rgb@fill{1}{0.6}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-12.097pt}{-6.93332pt}{24.194pt}{13.86664pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-10.097pt}{-1.9pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\mathcal{L}_{logits}$}} }}\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}+\gamma\,\leavevmode\hbox
to40.2pt{\vbox to14.67pt{\pgfpicture\makeatletter\hbox{\hskip
20.10109pt\lower-7.33331pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@rgb@stroke{0}{0}{1}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0.6,0.6,1}\pgfsys@color@rgb@fill{0.6}{0.6}{1}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@rgb@stroke{0}{0}{1}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0.6,0.6,1}\pgfsys@color@rgb@fill{0.6}{0.6}{1}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-19.7011pt}{-6.93332pt}{39.40219pt}{13.86664pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-17.7011pt}{-1.9pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\mathcal{L}_{embeddings}$}} }}\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$ (10) Margin-based
LossAdaptability Loss

To mitigate the constraints of rigid class-level combinations during meta-
training, inspired by [28, 29], we deploy a mixup strategy111See Sec. VI-E for
an in-depth examination of various mixup strategies and their impact on FSCIL
performance.. This mixup strategy, by virtually expanding class boundaries,
facilitates the generation of highly diverse logits for virtual and real
classes. Such an expansion is crucial for overcoming the rigid optimization
inherent in scenarios with limited logit variability.

In prior meta-learning paradigms [4, 35], supervised pre-training and meta-
learning refinement are typically performed across the entire class domain,
generating associated logits. However, these methods often overlook the
embedding-based classification method specific to each meta-task subspace,
designed to simulate testing operations in incremental sessions. To address
this, we deploy an unsupervised few-shot loss to better simulate and adapt to
incremental session scenarios. Diverging from traditional methods, we
reconfigure class indices in each batch to define sub-domain labels
$\mathbf{y^{*}}$, and utilize embeddings-derived prototypes to generate
pseudo-labels $\mathbf{pl}$.

$\mathbf{pl}=\frac{{\mathbf{X}_{j}\cdot\Omega_{n}(\frac{1}{m}\sum_{i=1}^{m}\mathbf{X}_{i})}}{{\|\mathbf{X}_{j}\|_{2}\cdot\|\Omega_{n}(\frac{1}{m}\sum_{i=1}^{m}\mathbf{X}_{i})\|_{2}}},$
(11)

where $n$ and $m$ denote their quantities in the $n$-way, $m$-shot scenario,
and $\Omega_{n}$ is the operation converging $n$ prototypes.

After generating pseudo-labels in the sub-domain, we employ the classic cross-
entropy loss to formulate the adaptability loss, denoted as
$\mathcal{L}_{embeddings}=CE(\mathbf{pl},\mathbf{y}^{*})$.

Ultimately, by employing the mixup strategy to diversify class-level logit
combinations and incorporating an adaptability loss to simulate incremental
scenarios, our bespoke meta-learning strategy can directly start meta-training
from scratch and omit the pre-training process, outperforming the conventional
multi-step paradigm in performance and efficiency. Additionally, assisted by
meta-learning, our NTK matrix theoretically reaches the global optimal
convergence.

##  V Reducing Generalization Loss in FSCIL

Drawing inspiration from Theorem 3 and [36, 16, 18, 19], we consider NTK-
related dynamics and generalization loss. In this literature, a more stable
eigenvalue distribution of NTK matrix indicates a better generalization state,
emphasizing the importance of real-time monitoring and constraining of the
NTK’s eigenvalue distribution. Moreover, following the ideas from Eq. 3 and
[16, 18], we understand that utilizing pre-trained models to initialize the
model weights, combined with better logit-label alignment, will effectively
optimize the NTK dynamic outputs, enhancing the model generalization
capabilities. Thus, in this section, our exploration will focus on three
facets: initialization, alignment and regularization.

Figure 1: The difference in FSCIL performance amongst various self-supervised
learners, utilizing ResNet-18$\times$2 on CIFAR100.

###  V-A Initialize Weights: Self-Supervised Pre-Training

As highlighted in Eq. 3 and [16], pre-trained models significantly enhance
model generalization and stabilize eigenvalue decay in NTK matrix. Yet, in
FSCIL, using larger datasets for pre-training is often discouraged due to the
risk of excessively enhancing the FSCIL learner’s generalization capabilities,
which could potentially overshadow the challenges inherent in continual and
few-shot scenarios. Adhering to FSCIL guidelines, we opt for self-supervised
pre-training on the base session for effective initialization. To further
investigate the impact of various self-supervised methods on FSCIL, we conduct
experiments using two mainstream frameworks, CEC [4] and ALICE [29], and apply
weights trained on the base session from self-supervised methods such as SwAV
[37], Dino [38], SimCLR [39], SimSiam [40], MAE [41], MoCo-v3 [42], BYOL [43]
and SparK [44]. The details and results are meticulously presented in Sec.
VI-C and Fig. 1.

As evidenced in Fig. 1, the generative SparK framework [44] outperforms its
contrastive learning-based counterparts. When the FSCIL learner is initialized
with SparK’s weights, we observe notable improvements in incremental session
accuracies, alongside maintained base session accuracy, which is indicative of
enhanced generalization potential. Furthermore, as detailed in Sec. VI-H, the
NTK’s eigenvalue distribution with SparK initialization demonstrates greater
stability. Therefore, due to its excellent performance in FSCIL task and NTK
properties, SparK emerges as our preferred choice.

###  V-B Better Alignment: Margin-based Loss

In Sec. IV, our mixup strategy addresses the rigidity in class-level
combinations, fostering enriched interactions for the FSCIL learner across
various virtual classes, thereby increasing the number of base class samples
$N$ and reducing the generalization loss $L_{D}(f^{*})$ as described in
Theorem 3. Yet, this strategy also introduces certain side effects, namely,
making traditional cross-entropy loss less effective in scenarios with a large
number of classes [28, 29]. However, as highlighted in Eq. 3, alignment plays
a crucial role in NTK dynamics and model generalization. Hence, to achieve
more effective logit-label alignment, we replace cross-entropy loss with a
margin-based loss [29, 45]. The fundamental structure of our margin-based
softmax loss is outlined as follows:

${\mathcal{L}_{logits}=-\log\frac{e^{sP(\cos\theta_{y_{i}})}}{e^{s{P(\cos\theta_{y_{i}})}}+\sum^{n}_{j\neq
y_{i}}{e^{sN(t,\cos\theta_{j})}}},}$ (12)

Here, $\theta_{y_{i}}$ and $\theta_{j}$ represent the cosine similarities for
the $i$-th and $j$-th class prototypes. $P(\cdot)$ and $N(\cdot)$ adjust
positive and negative cosine similarities, respectively.

Furthermore, since the mixup mechanism’s inherent randomness in creating
virtual classes, it can easily lead to an imbalance between virtual and real
classes, which presents a significant challenge for margin-based loss. To
counteract this, we adopt curriculum learning concepts [46], which involve
adaptive alterations to $P(\cdot)$ and $N(\cdot)$, as follows:

$N(\ast)=\begin{cases}\cos\theta_{j},&\mbox{$\mathbf{\varepsilon}\geq 0$}\\\
\cos\theta_{j}(t+\cos\theta_{j}),&\mbox{$\mathbf{\varepsilon}<0$},\end{cases}$
(13) $P(\ast)=\begin{cases}\cos\theta_{y_{i}}\cos m-\cos^{2}\theta_{y_{i}}\sin
m,&\mbox{$\mathbf{\varepsilon}\geq 0$}\\\ \cos(\theta_{y_{i}}+m)+m\sin
m,&\mbox{$\mathbf{\varepsilon}<0$},\end{cases}$ (14)

where $\mathbf{\varepsilon}=\cos(\theta_{i}+m)-\cos\theta_{j}$ embodies the
threshold, $t$ is the mean logits updated via EMA mechanism over each batch,
and $m$ designates a predetermined, constant margin.

Through this refined design, training begins by deemphasizing negative
samples, setting the initial value of $t$ to zero, while simultaneously
accentuating positive samples through the incorporation of $m$. As training
progresses and reaches the desired performance, indicated by
$\mathbf{\varepsilon}\geq 0$, $t$ is incrementally increased to shift the
model’s focus towards hard samples in positive classes. When certain classes
achieve the target performance, we activate a suppression mechanism for highly
accurate positive samples, thereby redirecting focus to less accurate ones.
This recalibration ensures balanced performance across all class domains.

Ultimately, this adaptive optimization not only addresses the class weight
imbalance but also significantly optimizes NTK dynamic outputs, as detailed in
Sec. VI-H and VI-D.

###  V-C Refine NTK eigenvalues: Dual NTK Regularization

After ensuring the NTK dynamics reach an optimal state, according to Theorem
3, we need to further refine the eigenvalue distribution to avoid excessively
large or small NTK eigenvalues, ensuring a stable NTK matrix is presented.
However, as mentioned earlier, the computational burden associated with the
NTK approximation in ConvNets challenges the limits of contemporary hardware.
To address this, we adopt a compromise strategy, implementing dual NTK
regularization that is specifically tailored to convolutional and linear
layers. This strategy delicately refines the NTK matrix, both indirectly and
directly, by intervening in and tightening the NTK’s eigenvalue distribution.
The details of the associated composite losses are provided below:

$\mathcal{L}_{reg}=\leavevmode\hbox to36.63pt{\vbox
to16.03pt{\pgfpicture\makeatletter\hbox{\hskip 18.31302pt\lower-8.01665pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.88,0}\pgfsys@color@cmyk@stroke{0.91}{0}{0.88}{0.12}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0.588,0.952,0.6}\pgfsys@color@cmyk@fill{0.364}{0}{0.352}{0.048}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.88,0}\pgfsys@color@cmyk@stroke{0.91}{0}{0.88}{0.12}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0.588,0.952,0.6}\pgfsys@color@cmyk@fill{0.364}{0}{0.352}{0.048}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-17.91302pt}{-7.61665pt}{35.82605pt}{15.2333pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-15.91302pt}{-1.21666pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\mathcal{L}_{conv\\_reg}$}} }}\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}+\leavevmode\hbox
to32.28pt{\vbox to16.03pt{\pgfpicture\makeatletter\hbox{\hskip
16.14159pt\lower-8.01665pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.49,0}\pgfsys@color@cmyk@stroke{0}{0.51}{1}{0}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{1,0.796,0.6}\pgfsys@color@cmyk@fill{0}{0.204}{0.4}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.49,0}\pgfsys@color@cmyk@stroke{0}{0.51}{1}{0}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{1,0.796,0.6}\pgfsys@color@cmyk@fill{0}{0.204}{0.4}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-15.7416pt}{-7.61665pt}{31.48318pt}{15.2333pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-13.7416pt}{-1.21666pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\mathcal{L}_{lin\\_reg}$}} }}\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$ (15) Convolutional
Spectral RegularizationLinear NTK Regularization

To ensure a slower eigenvalue decay in convolutional layers, we implement
spectral regularization on their weights [16, 36]. Furthermore, to reduce the
computational overhead associated with large-scale weight calculations, we
employ Singular Value Decomposition (SVD) to decompose the weights. The
detailed formulation is as follows:

$\mathcal{L}_{conv\\_reg}=\alpha\cdot\sum_{c\in
f}\frac{\varphi_{\text{{max}}}^{c}}{\varphi_{\text{{min}}}^{c}},$ (16)

where $\alpha$ is a hyperparameter, $\varphi$ denotes the SVD function, and
$c$ is each convolutional weight in the ConvNet.

Conversely, for the additional linear layers, simplification is not necessary.
Inspired by self-supervised linear evaluation studies [38, 41, 42, 44], we
directly compute their NTK matrix and impose constraints on the eigenvalue
distribution. Moreover, acknowledging the NTK’s inherent property— the optimal
NTK stabilizes into a constant matrix irrespective of inputs after
optimization—we concentrate on refining the stability of eigenvalue
fluctuations in each batch, aiming to maintain relative consistency during
optimization. Consequently, we devise a regularization loss to control the
variance of these NTK eigenvalues:

$\displaystyle\mathcal{L}_{lin\\_reg}$
$\displaystyle=\beta\cdot\frac{\sigma_{max}(\Phi_{linear})-\sigma_{min}(\Phi_{linear})}{\sigma^{{}^{\prime}}_{max}(\Phi_{linear})-\sigma^{{}^{\prime}}_{min}(\Phi_{linear})},$
(17)

where $\Phi_{linear}$ denotes the linear layer’s NTK matrix, $\beta$
symbolizes the hyperparameter governing the NTK eigenvalue fluctuation, while
$\sigma(\cdot)$ and $\sigma(\cdot)^{{}^{\prime}}$ refer to the eigenvalues of
the current and preceding batches respectively.

Optimization: Overall, our NTK-FSCIL adeptly navigates NTK convergence and
NTK-related generalization loss through a three-stage process: SparK-based
self-supervised pre-training on the base session, fully supervised FSCIL
optimization on the base session and NCM testing on incremental sessions.
During the crucial FSCIL optimization phase, the FSCIL learner is optimized
through meta-learning, utilizing the sum of Eq. 10 and Eq. 15, ensuring
optimal NTK convergence, NTK dynamics and minimal generalization loss.

##  VI Experiments

In the experimental section, we commence with a detailed overview of the
utilized datasets, as delineated in Sec. VI-A. This foundational step sets the
stage for a series of experiments conducted on the most prominent FSCIL
methods, meticulously validating the rationale and efficacy of each component.
These components include the impact of convolutional network width on FSCIL
performance as explored in Sec. VI-B, the influence of self-supervised pre-
training (i.e., the initialization $f_{0}^{*}(x)$) on FSCIL performance
detailed in Sec. VI-C, the effect of logit-label alignment (i.e.,
$Y-f_{0}^{*}(X)$) on FSCIL performance as discussed in Sec. VI-D, and the
impact of expanding virtual classes, corresponding to $N$ in Theorem 3, on
outcomes as examined in Sec. VI-E.

Subsequent to these foundational experiments, we conduct comparative and
analytical evaluations between our comprehensive approach and the latest
methodologies in the field, supplemented by detailed ablation studies. This
analytical journey culminates in a quantitative assessment of the eigenvalue
distribution within the NTK matrix, alongside the visualization of embeddings
through t-SNE plots, vividly illustrating NTK characteristics and model
generalization capabilities.

###  VI-A Experimental Setup

Datasets and Splits: Our method is evaluated on multiple datasets: CIFAR100
[47] (consisting of 60,000 images across 100 classes), CUB200-2011 [48]
(featuring 200 bird species classes), miniImageNet [49] (a subset of
ImageNet-1K with 100 classes at reduced resolution) and ImageNet100 [50]
(comprising 100 classes selected from ImageNet-1K). For CIFAR100,
miniImageNet, and ImageNet100, we partition the classes into two sets: 60 base
classes and 40 incremental classes. The latter are further divided into eight
5-way 5-shot tasks. In the case of CUB200, we have 100 base classes and 100
incremental classes, organized into ten 10-way 5-shot tasks. Our training
splits align with those used in previous work [24], except for ImageNet100,
which follows the splits introduced in [28]. More detailed content is shown in
Tab. I.

Dataset | $\mathcal{C}^{base}$ | $\mathcal{N}_{base}$ | $\mathcal{C}^{inc}$ | #Inc. | Shots | Resolution
---|---|---|---|---|---|---
CIFAR100 [47] | 60 | 30000 | 40 | 8 | 5 | 32×32
_mini_ ImageNet [49] | 60 | 30000 | 40 | 8 | 5 | 84×84
CUB200 [48] | 100 | 3000 | 100 | 10 | 5 | 224×224
ImageNet100 [50] | 60 | 77352 | 40 | 8 | 5 | 112×112

TABLE I: Statistics of benchmarks datasets. $\mathcal{C}^{base}$: number of
classes in base session. $\mathcal{C}^{inc}$: total number of classes in
incremental sessions. #Inc.: number of incremental sessions. Shots: training
shots for incremental sessions. $\mathcal{N}_{base}$: number of samples in
base session.

###  VI-B Impacts of Network Width in FSCIL

In this subsection, we assess how network width influences FSCIL performance.
Specifically, we select two methods, i.e., CEC [4] and ALICE [29], for
evaluation by gradually widening the backbone. As shown in Fig. 2, an
expansion in network width leads to a gradual enhancement in accuracy across
sessions, providing additional evidence that widening the network improves its
generalization. However, excessively increasing the width of the network poses
significant challenges to optimization. Hence, unless stated otherwise, our
default backbone is ResNet-18$\times$2 (‘width128’ in Fig. 2), with doubled
convolutional block widths.

Figure 2: The FSCIL performance on CIFAR100 across different widths in
ResNet-18, employing the CEC [4] and ALICE [29].

Next, we aim to clearly demonstrate the impact of varying the width of
individual convolutional layer on FSCIL performance. Using a ConvNet with
doubled width as our baseline, we experimentally modify the width of specific
layers and observe the resulting changes in FSCIL performance. These
experiments are conducted using various ResNet architectures on the CIFAR100
dataset to ensure efficient and informative outcomes. Table II provides a
detailed analysis of the effects of varying the width of each convolutional
layer on FSCIL performance. The data suggests a direct correlation: as the
width of these layers increases, FSCIL efficacy consistently improves. This
enhancement is observed not only when the entire ConvNet is widened but also
when individual convolutional layer is selectively widened.

Subsequently, we extend this width modification approach to other members of
the ResNet family, including ResNet-12, ResNet-20, ResNet-34 and ConvNext-
tiny, conducting similar experiments on CIFAR100 as done with ResNet-18. The
detailed outcomes of these experiments are presented in Fig. 3. These results
consistently demonstrate that, irrespective of the specific CNN architecture,
the generalization capabilities in incremental sessions are positively
influenced by the width of the convolutional layers. The wider these layers,
the better the performance in terms of generalization and FSCIL. These
findings corroborate the NTK theory’s proposition regarding the relationship
between a network’s width and its generalization ability, even within
different CNN architectures.

ResNet-18 | CEC | ALICE | ResNet-18 | CEC | ALICE
---|---|---|---|---|---
32$\Rightarrow$256$\Rightarrow$512$\Rightarrow$1024 | 81.2$\rightarrow$53.9 | 79.4$\rightarrow$58.0 | 128$\Rightarrow$64$\Rightarrow$512$\Rightarrow$1024 | 81.1$\rightarrow$53.1 | 78.8$\rightarrow$57.3
64$\Rightarrow$256$\Rightarrow$512$\Rightarrow$1024 | 81.6$\rightarrow$53.6 | 80.0$\rightarrow$58.0 | 128$\Rightarrow$128$\Rightarrow$512$\Rightarrow$1024 | 81.8$\rightarrow$54.2 | 79.6$\rightarrow$58.2
128$\Rightarrow$256$\Rightarrow$512$\Rightarrow$1024 | 82.1$\rightarrow$54.4 | 80.6$\rightarrow$58.2 | 128$\Rightarrow$256$\Rightarrow$512$\Rightarrow$1024 | 82.1$\rightarrow$54.4 | 80.6$\rightarrow$58.2
256$\Rightarrow$256$\Rightarrow$512$\Rightarrow$1024 | 82.3$\rightarrow$54.7 | 80.7$\rightarrow$58.2 | 128$\Rightarrow$512$\Rightarrow$512$\Rightarrow$1024 | 83.0$\rightarrow$54.5 | 80.6$\rightarrow$58.6
ResNet-18 | CEC | ALICE | ResNet-18 | CEC | ALICE
128$\Rightarrow$256$\Rightarrow$128$\Rightarrow$1024 | 81.3$\rightarrow$52.7 | 78.8$\rightarrow$57.2 | 128$\Rightarrow$256$\Rightarrow$512$\Rightarrow$256 | 84.7$\rightarrow$54.9 | 79.7$\rightarrow$57.4
128$\Rightarrow$256$\Rightarrow$256$\Rightarrow$1024 | 81.6$\rightarrow$53.4 | 79.4$\rightarrow$57.7 | 128$\Rightarrow$256$\Rightarrow$512$\Rightarrow$512 | 83.8$\rightarrow$54.7 | 80.6$\rightarrow$58.1
128$\Rightarrow$256$\Rightarrow$512$\Rightarrow$1024 | 82.1$\rightarrow$54.4 | 80.6$\rightarrow$58.2 | 128$\Rightarrow$256$\Rightarrow$512$\Rightarrow$1024 | 82.1$\rightarrow$54.4 | 80.6$\rightarrow$58.2
128$\Rightarrow$256$\Rightarrow$1024$\Rightarrow$1024 | 83.0$\rightarrow$54.9 | 80.4$\rightarrow$58.9 | 128$\Rightarrow$256$\Rightarrow$512$\Rightarrow$2048 | 80.6$\rightarrow$52.3 | 80.3$\rightarrow$58.7

TABLE II: The FSCIL performance on CIFAR100 using ResNet-18 with varying
widths of convolutional layers is presented. The baseline is indicated in
bold, and the performance is measured solely by the accuracies in the base and
final sessions.

(a) ResNet-12

(b) ResNet-20

(c) ResNet-34

(d) ConvNext-tiny

Figure 3: This figure elucidates the FSCIL results of four different ConvNets
(a.k.a. ResNet-12, ResNet-20, ResNet-34 and ConvNext), grounded on CEC and
ALICE, following different width expansions on CIFAR100.

###  VI-C Exploring Self-supervised Pre-training

In this subsection, we explicate and scrutinize the related experimental
specifics and outcomes associated with the self-supervised pre-training
component.

CEC: In assessing CEC’s performance, we exclude components such as the
continually evolved classifier and the fine-tuning post pre-training, opting
instead for the ‘autoaugment’ image transformation. The performance analysis,
as detailed in Table III, involves six distinct self-supervised methodologies,
excluding SimSiam [40] and SwAV [37] due to non-convergence on CIFAR100. The
results indicate that self-supervised pre-training on base classes does not
consistently enhance FSCIL performance for ConvNets in CEC. Only DINO [38],
SparK [44] and MoCo-v3 [42] demonstrate positive results, whereas simCLR [39]
and BYOL [43] show negative effects. This disparity may stem from the reliance
of some self-supervised algorithms on contrastive learning and hand-crafted
view transformations, which can lead to representation collapse in ConvNets,
especially given CIFAR100’s small image size. Such issues seem less prevalent
in generative methods like masked image modeling.

ALICE: ALICE undergoes similar adaptations to CEC, including the removal of
the 128-dimensional fully-connected layer from the backbone, reduction in mix-
up volume, abandonment of the dual-branch training model, and discontinuation
of balanced testing. Furthermore, ALICE’s outcomes, as detailed in Table IV,
align with those observed in CEC. The performance of contrastive learning
methodologies in FSCIL is notably inconsistent, with only BYOL showing slight
improvements. In contrast, generative strategies based on masked image
modeling, particularly SparK, consistently perform well. This supports our
earlier hypothesis and suggests the reliability of these strategies from a
different perspective.

In the spirit of rigorous scientific investigation, we extend our experiments
to equivalent-sized Transformer-based models. Traditionally, these models are
perceived as less effective than ConvNets in small datasets and few-shot
tasks. However, our results show that with self-supervised pre-training,
Transformers significantly close the performance gap, mirroring the results of
the original CEC. This intriguing finding suggests that Transformer-based
models may approach or even match ConvNets’ efficiency in few-shot scenarios,
presenting an exciting avenue for future research.

Method | Acc. in each session (%) ↑ | PD $\downarrow$
---|---|---
Backbone | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
CEC [4] | ResNet-20 | 73.1 | 68.9 | 65.3 | 61.2 | 58.1 | 55.6 | 53.2 | 51.3 | 49.1 | 24.0
$\textbf{CEC}^{*}$ [4] | ResNet-20 | 79.0 | 73.1 | 68.8 | 64.8 | 61.4 | 58.3 | 56.0 | 53.8 | 51.7 | 27.3
$\textbf{CEC}^{*}$ [4] | ResNet-18$\times$2 | 85.3 | 79.7 | 74.9 | 70.6 | 66.9 | 63.8 | 61.3 | 58.7 | 55.2 | 30.1
$\textbf{CEC}^{*}$ [4] | ViT-tiny | 54.9 | 47.6 | 44.7 | 42.2 | 39.8 | 37.7 | 36.1 | 35.0 | 33.5 | 21.4
$\textbf{CEC}^{*}$ [4] | ViT-small | 58.0 | 50.1 | 47.4 | 44.5 | 42.2 | 40.2 | 38.3 | 37.0 | 35.5 | 22.5
DINO [38] | ResNet-18$\times$2 | 85.8 | 80.1 | 75.1 | 70.1 | 67.0 | 63.7 | 61.0 | 58.6 | 56.0 | 29.8
DINO [38] | ViT-tiny | 75.0 | 69.8 | 65.2 | 61.1 | 57.8 | 54.6 | 51.8 | 49.6 | 47.4 | 22.4
DINO [38] | ViT-small | 67.8 | 63.0 | 58.8 | 55.1 | 51.9 | 49.1 | 46.6 | 44.5 | 42.6 | 25.2
MAE [41] | ViT-tiny | 78.6 | 73.8 | 69.1 | 64.9 | 61.3 | 58.3 | 55.4 | 53.0 | 50.4 | 28.2
MAE [41] | ViT-small | 78.6 | 73.2 | 68.6 | 64.5 | 61.0 | 57.9 | 55.2 | 52.7 | 50.3 | 28.3
SparK [44] | ResNet-18$\times$2 | 85.5 | 80.0 | 75.1 | 70.8 | 67.1 | 63.8 | 60.9 | 58.6 | 56.2 | 29.3
MoCo-v3 [42] | ResNet-18$\times$2 | 85.5 | 79.8 | 74.7 | 70.7 | 67.0 | 63.9 | 61.1 | 58.9 | 56.2 | 29.3
MoCo-v3 [42] | ViT-tiny | 69.4 | 64.8 | 60.9 | 57.3 | 54.0 | 51.3 | 48.8 | 46.5 | 51.7 | 28.1
MoCo-v3 [42] | ViT-small | 74.1 | 69.6 | 65.9 | 62.0 | 58.8 | 55.9 | 53.4 | 51.3 | 49.2 | 24.9
simCLR [39] | ResNet-18$\times$2 | 81.3 | 76.0 | 71.4 | 67.3 | 63.6 | 60.4 | 57.9 | 55.5 | 53.0 | 25.8
simCLR [39] | ViT-tiny | 78.5 | 73.5 | 68.7 | 64.8 | 61.1 | 57.9 | 55.0 | 52.6 | 50.2 | 28.3
simCLR [39] | ViT-small | 78.4 | 72.8 | 68.2 | 63.9 | 60.3 | 57.2 | 54.5 | 52.1 | 49.8 | 28.6
BYOL [43] | ResNet-18$\times$2 | 82.0 | 76.9 | 72.1 | 68.2 | 64.5 | 61.5 | 59.0 | 56.6 | 54.1 | 27.9
BYOL [43] | ViT-tiny | 66.2 | 59.6 | 56.1 | 52.7 | 50.1 | 47.8 | 45.6 | 44.0 | 42.3 | 23.9
BYOL [43] | ViT-small | 66.1 | 60.0 | 56.5 | 53.1 | 50.3 | 47.8 | 45.5 | 43.4 | 41.7 | 24.4

TABLE III: The performance gleaned from various self-supervised learners on
CIFAR100, anchored on CEC and the modified $\textbf{CEC}^{*}$. The red-bolded
segments represent the optimal results, while the blue-bolded segments
represent suboptimal results.

Method | Acc. in each session (%) ↑ | PD $\downarrow$
---|---|---
Backbone | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
ALICE [29] | ResNet-18 | 79.0 | 70.5 | 67.1 | 63.4 | 61.2 | 59.2 | 58.1 | 56.3 | 54.1 | 24.9
$\textbf{ALICE}^{*}$ [29] | ResNet-18 | 78.8 | 75.2 | 72.3 | 68.7 | 65.5 | 62.1 | 60.7 | 58.3 | 56.6 | 22.2
$\textbf{ALICE}^{*}$ [4] | ResNet-18$\times$2 | 80.9 | 76.9 | 73.6 | 69.9 | 66.9 | 63.9 | 62.5 | 60.4 | 58.3 | 22.6
$\textbf{ALICE}^{*}$ [4] | ViT-tiny | 58.7 | 55.4 | 52.3 | 49.0 | 46.8 | 44.4 | 42.4 | 40.8 | 39.1 | 19.6
$\textbf{ALICE}^{*}$ [4] | ViT-small | 62.4 | 58.5 | 55.2 | 51.8 | 49.5 | 46.8 | 44.8 | 43.1 | 41.4 | 21.0
DINO [38] | ResNet-18$\times$2 | 84.7 | 79.1 | 74.9 | 70.3 | 67.2 | 64.1 | 62.0 | 59.9 | 57.8 | 26.9
DINO [38] | ViT-tiny | 82.7 | 76.9 | 71.9 | 67.4 | 63.9 | 60.7 | 57.7 | 55.1 | 52.8 | 29.9
DINO [38] | ViT-small | 71.8 | 67.0 | 62.8 | 58.8 | 55.6 | 52.5 | 50.1 | 48.0 | 45.9 | 25.9
MAE [41] | ViT-tiny | 81.9 | 76.7 | 71.7 | 67.5 | 64.2 | 60.9 | 58.1 | 55.5 | 53.2 | 28.7
MAE [41] | ViT-small | 81.5 | 76.4 | 71.3 | 67.1 | 63.5 | 60.3 | 57.4 | 54.9 | 52.7 | 28.8
SparK [44] | ResNet-18$\times$2 | 83.6 | 77.7 | 73.7 | 69.8 | 67.4 | 64.6 | 63.0 | 61.0 | 58.8 | 24.8
MoCo-v3 [42] | ResNet-18$\times$2 | 82.4 | 77.5 | 73.3 | 68.9 | 66.2 | 63.1 | 60.9 | 58.8 | 56.8 | 25.6
MoCo-v3 [42] | ViT-tiny | 78.7 | 73.4 | 68.7 | 64.7 | 61.1 | 57.8 | 55.2 | 52.8 | 50.5 | 28.2
MoCo-v3 [42] | ViT-small | 78.4 | 73.1 | 68.4 | 64.0 | 60.3 | 57.3 | 54.5 | 52.1 | 49.8 | 28.6
simCLR [39] | ResNet-18$\times$2 | 74.1 | 70.4 | 67.3 | 63.8 | 61.2 | 58.2 | 56.7 | 55.2 | 53.1 | 21.0
simCLR [39] | ViT-tiny | 80.7 | 75.7 | 70.9 | 66.4 | 62.8 | 59.5 | 56.5 | 53.9 | 51.7 | 29.0
simCLR [39] | ViT-small | 79.8 | 74.8 | 69.9 | 65.6 | 62.1 | 58.8 | 56.0 | 53.5 | 51.3 | 28.5
BYOL [43] | ResNet-18$\times$2 | 82.9 | 77.7 | 73.9 | 69.8 | 67.3 | 64.5 | 63.0 | 61.1 | 58.8 | 24.1
BYOL [43] | ViT-tiny | 56.9 | 53.5 | 50.4 | 47.3 | 45.0 | 42.7 | 40.7 | 39.0 | 37.2 | 19.7
BYOL [43] | ViT-small | 55.5 | 52.0 | 48.8 | 45.4 | 43.2 | 40.8 | 38.9 | 37.6 | 35.9 | 19.6

TABLE IV: The performance from various self-supervised learners on CIFAR100,
anchored on ALICE and the modified $\textbf{ALICE}^{*}$. The red-bolded
segments represent the optimal results, while the blue-bolded segments
represent suboptimal results.

###  VI-D Better Logit-Label Alignment

Drawing from the findings of [29], we delve into the performance of various
loss functions, placing particular emphasis on those rooted in margin-based
losses. More detailed results are presented in Table V. Our empirical analysis
uncovers significant performance limitations when solely relying on the
traditional cross-entropy loss. Delving deeper, it becomes apparent that
margin-based losses possess inherent advantages in handling a vast number of
classes in the class extension mechanism. Among these margin-based losses,
curricular alignment stands out significantly. This is largely attributed to
its incorporation of the curriculum learning philosophy, mirroring the human
education process of transitioning from simple to complex. It effectively
balances the relationship between easy and hard samples and further alleviates
the class imbalance issues introduced by the mixup mechanism. Furthermore,
compared to other margin-based losses, our curricular alignment, despite
lagging in base session accuracy, excels in incremental session accuracies.
This indicates that it effectively learns from virtual classes, potentially
reducing performance on real classes to some extent, but in turn, achieving
better model generalization.

Classification Loss | Acc. in each session (%) ↑
---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
CE | 24.5 | 23.1 | 21.4 | 19.9 | 19.1 | 18.1 | 17.3 | 16.7 | 15.9
ArcFace | 76.5 | 71.2 | 67.4 | 63.5 | 60.6 | 57.7 | 56.1 | 54.2 | 52.1
CosFace | 77.0 | 71.9 | 67.1 | 63.2 | 60.8 | 57.9 | 56.2 | 54.3 | 52.2
Curricular | 75.5 | 71.7 | 68.2 | 64.5 | 61.9 | 58.9 | 57.5 | 55.6 | 53.2

TABLE V: The notations CE, ArcFace, CosFace and Curricular correspond to Cross
Entropy Loss, ArcFace Loss in [51], CosFace Loss in [52] and Curricular Loss
respectively. All experiments are implemented on the ALICE [29] framework.

For hyperparameters $s$ and $m$, our settings are closely aligned with those
used in ALICE [29]. Specifically, for CIFAR100 and _mini_ ImageNet, we set $s$
to 15.0 and $m$ to 0.1. For datasets with larger image size and more complex
information, such as CUB200 and ImageNet100, we adjust $s$ to 25.0 while
keeping $m$ at 0.1, to accommodate the need for greater separation between
positive and negative classes.

Methods | Acc. in each session (%) ↑ | Base Acc. | Incremental Acc. | Harmonic Mean
---|---|---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
Mix-up [53] | 84.2 | 79.7 | 76.1 | 71.8 | 69.0 | 66.4 | 64.8 | 62.9 | 60.6 | 84.2 | 37.8 | 50.5
CutMix [54] | 85.1 | 80.0 | 76.0 | 71.4 | 68.7 | 65.7 | 64.2 | 62.3 | 60.2 | 85.1 | 36.2 | 49.1
AugMix [55] | 80.8 | 76.8 | 73.2 | 69.2 | 66.3 | 63.1 | 61.7 | 59.9 | 57.8 | 80.8 | 33.3 | 46.0
PuzzleMix [56] | 84.4 | 79.9 | 76.6 | 72.3 | 69.6 | 66.7 | 65.0 | 63.1 | 61.0 | 84.4 | 37.2 | 50.1

TABLE VI: Average accuracy across base and incremental sessions using
different mix-up methods, with their harmonic mean.

###  VI-E Exploring Logits Diversity in Meta-Learning

In FSCIL, the prevailing approaches often leverage the mix-up mechanism for
class extension, directly merging two images [29, 7]. In this subsection, we
delve into the nuanced variations of this mechanism. To investigate the impact
of virtual classes generated by different mixup mechanisms on logits
diversity, and further explore their effect on model generalization, we employ
the harmonic mean for an intuitive representation.

$\displaystyle H=\frac{2\times Base\ Acc\times Incremental\ Acc}{Base\
Acc+Incremental\ Acc}.$ (18)

We conduct experiments with various mixup mechanisms on CIFAR100, and the
results are displayed in Table VI. As shown in Table VI, methods that perform
mixup operations globally on images produce better virtual samples and more
flexible logit combinations, resulting in improved base session accuracy and
harmonic mean. However, mixup methods involving image transformations fail to
alleviate the rigid logit combinations in meta-learning. This could be due to
the small image size of CIFAR100, where overly complex image transformations
might irreversibly damage the original information and generate unrealistic
virtual samples.

###  VI-F Comparison with State-of-the-Art

In this subsection, we report the primary results for CIFAR100 in Tab. VII,
_mini_ ImageNet in Tab. VIII, CUB200 in Tab. IX, and ImageNet100 in Tab. X.
From the results, we observe that our NTK-FSCIL consistently outperforms its
competitors. It exhibits superior end-session accuracy and an admirable PD,
which represents the difference between initial and final session accuracies.
A reduced PD value suggests enhanced anti-amnesia capability, while an
elevated end-session accuracy indicates advanced generalization across
incremental sessions. For instance, our method achieves a significant 8.2%
improvement in end-session accuracy and 11.4% improvement in PD over NC-FSCIL
on CIFAR100, underscoring its exceptional generalization and anti-amnesia
capabilities. Furthermore, similar outcomes are also observed on other
datasets. For the end-session accuracy on _mini_ ImageNet and ImageNet100, our
NTK-FSCIL surpasses the nearest competitors, NC-FSCIL [7] and CEC [4], by 3%
and 6.2%.

For CUB200, considering that most methodologies, such as those proposed by [4,
29, 7], leverage the pre-trained weights of ResNet-18, it becomes necessary to
bypass both the widening and self-supervised modules to maintain a fair
comparison. Despite this limitation, our partial implementation of NTK-FSCIL
still demonstrates remarkable efficacy. Additionally, we have adopted two more
efficient pre-trained weights, namely Swin Transformer [57] and ConvNeXt [58],
which have achieved far superior FSCIL performance.

At its core, our NTK-FSCIL, grounded in the principles of the NTK to enhance
model generalization, primarily evaluates its success through performance in
incremental learning sessions. Here, it showcases superior results, placing it
at the forefront of the field. This adeptness in managing incremental sessions
further enables our NTK-FSCIL model to achieve prominent positions in the PD
metric, illustrating its comprehensive and balanced performance across various
measures.

Method | Acc. in each session (%) ↑ | PD $\downarrow$
---|---|---
Backbone | Params | FLOPs | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
iCaRL [1] | ResNet-32 | 0.47M | 0.07G | 64.1 | 53.3 | 41.7 | 34.1 | 27.9 | 25.1 | 20.4 | 15.5 | 13.7 | 50.4
TOPIC [24] | ResNet-18 | 11.17M | 0.56G | 64.1 | 55.9 | 47.1 | 45.2 | 40.1 | 36.4 | 34.0 | 31.6 | 29.4 | 34.7
CEC [4] | ResNet-20 | 0.27M | 0.04G | 73.1 | 68.9 | 65.3 | 61.2 | 58.1 | 55.6 | 53.2 | 51.3 | 49.1 | 24.0
LIMIT [5] | ResNet-20 | 0.27M | 0.04G | 73.0 | 70.8 | 67.5 | 63.4 | 60.0 | 56.9 | 54.8 | 52.2 | 49.9 | 23.1
FACT [28] | ResNet-20 | 0.27M | 0.04G | 74.6 | 72.1 | 67.6 | 63.5 | 61.4 | 58.4 | 56.3 | 54.2 | 52.1 | 22.5
ALICE [29] | ResNet-18 | 11.17M | 0.56G | 79.0 | 70.5 | 67.1 | 63.4 | 61.2 | 59.2 | 58.1 | 56.3 | 54.1 | 24.9
GKEAL [59] | ResNet-20 | 0.27M | 0.04G | 74.0 | 70.5 | 67.0 | 63.1 | 60.0 | 57.3 | 55.5 | 53.4 | 51.4 | 22.6
WaRP [60] | ResNet-20 | 0.27M | 0.04G | 80.3 | 75.9 | 71.9 | 67.6 | 64.4 | 61.3 | 59.2 | 57.1 | 54.7 | 25.6
SoftNet [6] | ResNet-18 | 11.17M | 0.56G | 79.9 | 75.5 | 71.6 | 67.5 | 64.5 | 61.1 | 59.1 | 57.3 | 55.3 | 24.6
NC-FSCIL [7] | ResNet-12 | 12.42M | 0.53G | 82.5 | 76.8 | 73.3 | 69.7 | 66.2 | 62.9 | 61.0 | 59.0 | 56.1 | 26.4
NTK-FSCIL | ResNet-18 | 11.17M | 0.56G | 81.8 | 77.3 | 74.3 | 69.8 | 67.1 | 64.2 | 62.9 | 61.1 | 58.7 | 23.1
NTK-FSCIL | ResNet-18$\times$2 | 44.65M | 2.22G | 84.4 | 79.9 | 76.6 | 72.3 | 69.6 | 66.7 | 65.0 | 63.1 | 61.0 | 23.4

TABLE VII: Performance evaluation on CIFAR100 dataset within the context of a
5-way 5-shot class-incremental learning paradigm. The red-bolded segments
represent the optimal results, while the blue-bolded segments represent
suboptimal results.

Method | Acc. in each session (%) ↑ | PD $\downarrow$
---|---|---
Backbone | Params | FLOPs | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
iCaRL [1] | ResNet-18 | 11.17M | 3.94G | 61.3 | 46.3 | 42.9 | 37.6 | 30.5 | 24.0 | 20.9 | 18.8 | 17.2 | 44.1
TOPIC [24] | ResNet-18 | 11.17M | 3.94G | 61.3 | 50.1 | 45.2 | 41.2 | 37.5 | 35.5 | 32.2 | 29.5 | 24.4 | 36.9
CEC [4] | ResNet-18 | 11.17M | 3.94G | 72.0 | 66.8 | 63.0 | 59.4 | 56.7 | 53.7 | 51.2 | 49.2 | 47.6 | 24.4
LIMIT [5] | ResNet-18 | 11.17M | 3.94G | 71.9 | 67.9 | 63.6 | 60.2 | 57.3 | 54.3 | 52.0 | 50.0 | 48.4 | 23.5
FACT [28] | ResNet-18 | 11.17M | 3.94G | 72.6 | 69.6 | 66.4 | 62.8 | 60.6 | 57.3 | 54.3 | 52.2 | 50.5 | 22.1
ALICE [29] | ResNet-18 | 11.17M | 3.94G | 80.6 | 70.6 | 67.4 | 64.5 | 62.5 | 60.0 | 57.8 | 56.8 | 55.7 | 24.9
GKEAL [59] | ResNet-18 | 11.17M | 3.94G | 73.6 | 68.9 | 65.3 | 62.3 | 59.4 | 56.7 | 54.2 | 52.6 | 51.3 | 22.3
WaRP [60] | ResNet-18 | 11.17M | 3.94G | 73.0 | 68.1 | 64.3 | 61.3 | 58.6 | 56.1 | 53.4 | 51.7 | 50.7 | 22.3
SoftNet[6] | ResNet-18 | 11.17M | 3.94G | 79.4 | 74.3 | 69.9 | 66.2 | 63.4 | 60.8 | 57.6 | 55.7 | 54.3 | 25.1
NC-FSCIL [7] | ResNet-12 | 12.42M | 3.53G | 84.0 | 76.8 | 72.0 | 67.8 | 66.4 | 64.0 | 61.5 | 59.5 | 58.3 | 25.7
NTK-FSCIL | ResNet-18 | 11.17M | 3.94G | 80.0 | 75.5 | 71.6 | 68.6 | 66.4 | 64.0 | 61.3 | 59.9 | 58.8 | 21.2
NTK-FSCIL | ResNet-18$\times$2 | 44.65M | 15.69G | 82.4 | 77.2 | 73.3 | 70.2 | 67.8 | 65.3 | 62.7 | 60.9 | 60.0 | 22.4

TABLE VIII: Performance evaluation on _mini_ ImageNet dataset within the
context of a 5-way 5-shot class-incremental learning paradigm. The red-bolded
segments represent the optimal results, while the blue-bolded segments
represent suboptimal results.

Method | Acc. in each session (%) ↑ | PD $\downarrow$
---|---|---
Backbone | Params | FLOPs | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
iCaRL [1] | ResNet-18 | 11.17M | 27.37G | 68.7 | 52.7 | 48.6 | 44.2 | 36.6 | 29.5 | 27.8 | 26.3 | 24.0 | 23.9 | 21.2 | 47.5
TOPIC [24] | ResNet-18 | 11.17M | 27.37G | 68.7 | 62.5 | 54.8 | 50.0 | 45.3 | 41.4 | 38.4 | 35.4 | 32.2 | 28.3 | 26.3 | 42.4
CEC [4] | ResNet-18 | 11.17M | 27.37G | 75.9 | 71.9 | 68.5 | 63.5 | 62.4 | 58.3 | 57.7 | 55.8 | 54.8 | 53.5 | 52.3 | 23.6
LIMIT [5] | ResNet-18 | 11.17M | 27.37G | 76.3 | 74.2 | 72.7 | 69.2 | 68.8 | 65.6 | 63.6 | 62.7 | 61.5 | 60.4 | 58.4 | 17.9
FACT [28] | ResNet-18 | 11.17M | 27.37G | 75.9 | 73.2 | 70.8 | 66.1 | 65.6 | 62.2 | 61.7 | 59.8 | 58.4 | 57.9 | 56.9 | 19.0
ALICE [29] | ResNet-18 | 11.17M | 27.37G | 77.4 | 72.7 | 70.6 | 67.2 | 65.9 | 63.4 | 62.9 | 61.9 | 60.5 | 60.6 | 60.1 | 17.3
GKEAL [59] | ResNet-18 | 11.17M | 27.37G | 78.9 | 75.6 | 72.3 | 68.6 | 67.2 | 64.3 | 63.0 | 61.9 | 60.2 | 59.2 | 58.7 | 20.2
WaRP [60] | ResNet-18 | 11.17M | 27.37G | 77.7 | 74.2 | 70.8 | 66.9 | 65.0 | 62.6 | 61.4 | 59.9 | 58.0 | 57.8 | 57.0 | 20.7
SoftNet[6] | ResNet-18 | 11.17M | 27.37G | 78.1 | 74.6 | 71.3 | 67.5 | 65.1 | 62.4 | 60.8 | 59.2 | 57.4 | 57.1 | 56.6 | 21.5
NC-FSCIL [7] | ResNet-18 | 11.17M | 27.37G | 80.5 | 76.0 | 72.3 | 70.3 | 68.2 | 65.2 | 64.4 | 63.3 | 60.7 | 60.0 | 59.4 | 21.1
NTK-FSCIL | ResNet-18 | 11.17M | 27.37G | 78.1 | 75.0 | 72.0 | 68.6 | 67.1 | 64.6 | 63.6 | 62.7 | 61.1 | 61.0 | 60.4 | 17.9
NTK-FSCIL | Swin-Small | 50.0M | 8.7G | 84.0 | 81.8 | 80.1 | 77.4 | 76.6 | 74.6 | 74.6 | 74.4 | 74.0 | 73.9 | 73.6 | 10.4
NTK-FSCIL | ConvNeXt-Base | 88.6M | 15.4G | 83.3 | 81.3 | 80.1 | 78.0 | 77.2 | 74.9 | 74.6 | 74.5 | 74.3 | 74.2 | 74.0 | 9.3

TABLE IX: Performance evaluation on CUB-200-2011 dataset within the context of
a 10-way 5-shot class-incremental learning paradigm. The red-bolded segments
represent the optimal results, while the blue-bolded segments represent
suboptimal results.

Method | Acc. in each session (%) ↑ | PD $\downarrow$
---|---|---
Backbone | Params | FLOPs | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
CEC* [4] | ResNet-18 | 11.17M | 27.37G | 72.9 | 66.0 | 62.8 | 60.3 | 56.8 | 54.0 | 52.8 | 52.3 | 50.0 | 22.9
ALICE* [29] | ResNet-18 | 11.17M | 27.37G | 68.8 | 63.2 | 60.0 | 57.1 | 53.4 | 50.3 | 48.5 | 47.7 | 45.5 | 23.3
NTK-FSCIL | ResNet-18 | 11.17M | 27.37G | 72.8 | 68.9 | 65.5 | 63.0 | 58.9 | 56.2 | 54.4 | 53.4 | 51.1 | 21.7
NTK-FSCIL | ResNet-18$\times$2 | 44.65M | 108.99G | 74.9 | 69.5 | 66.4 | 63.5 | 60.0 | 57.7 | 56.0 | 55.4 | 53.1 | 21.8

TABLE X: Performance evaluation on ImageNet100 dataset within the context of a
5-way 5-shot class-incremental learning paradigm. The red-bolded segments
represent the optimal results, while the blue-bolded segments represent
suboptimal results.

ML | WC | SSI | LNR | CSR | Acc. in each session (%) ↑ | $\triangle_{last}\uparrow$
---|---|---|---|---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
✓ |  |  |  |  | 75.3 | 71.9 | 68.5 | 64.2 | 61.7 | 59.1 | 57.7 | 56.2 | 54.4 | +0.0
✓ | ✓ |  |  |  | 79.2 | 75.4 | 72.0 | 67.9 | 65.4 | 62.7 | 61.2 | 59.6 | 57.4 | +3.0
✓ | ✓ | ✓ |  |  | 82.9 | 78.7 | 74.5 | 70.5 | 67.8 | 65.1 | 63.7 | 61.8 | 59.0 | +4.6
✓ | ✓ | ✓ | ✓ |  | 83.3 | 78.7 | 75.2 | 71.0 | 67.7 | 65.1 | 63.5 | 61.6 | 59.5 | +5.1
✓ | ✓ | ✓ | ✓ | ✓ | 84.4 | 79.9 | 76.6 | 72.3 | 69.6 | 66.7 | 65.0 | 63.1 | 61.0 | +6.6

TABLE XI: Ablation studies (%) on CIFAR100 benchmark. Specifically, the
symbols ML, WC, SSI, LNR and CSR represent Meta-Learning in Subsec. IV, Wider
ConvNets in in Subsec. VI-B, Self-Supervised Initialization in Subsec. V-A,
Linear NTK Regularization and Convolutional Spectral Regularization in Subsec.
V-C, respectively. The $\triangle_{last}\uparrow$ denotes the improvement over
the first row’s scenario.

###  VI-G Ablation Study

To demonstrate the effectiveness of NTK-FSCIL’s components, we perform an
ablation study, detailed in Tab. XI. Each experiment utilizes curricular
alignment, the details of which are outlined in Sec. V-B. For additional
insights into alignment strategies, refer to Sec. VI-D.

(a) Linear NTK Regularization

(b) Convolutional NTK Regularization

(c) Curricular Alignment

(d) Self-supervised Pre-training

(e) Meta-Learning

(f) Wider ConvNet

Figure 4: Investigate the NTK dynamics post-elimination of Linear NTK
Regularization, Convolutional NTK Regularization, Curricular Alignment, Meta-
Learning and Wider ConvNet strategies.

The evidence from Table XI and Sec. VI-B, Sec. VI-C, Sec. VI-D and Sec. VI-E
clearly show that each component of our NTK-FSCIL substantially improves the
FSCIL learner’s generalization in incremental sessions. Our meta-learning
strategy, targeting NTK convergence, either matches or exceeds the current
pre-training and fine-tuning paradigm in both efficiency and convergence.
Furthermore, our findings reveal that network expansion, as per Definition 1,
notably enhances the FSCIL learner’s generalization, yielding an approximate
5.5% improvement in end-session accuracy. Moreover, initializing the NTK with
self-supervised pre-training on the base session significantly boosts its
potential, establishing pre-training as a key strategy for enhancing NTK
stability and overall model generalization. Compared to the baseline of width
expansion and meta-learning, it results in a significant 2.8% increase in end-
session accuracy.

Alongside these components, implementing linear and convolutional NTK
regularization also contributes to a significant uptick in end-session
accuracy, by around 3.4%. Despite this performance enhancement, it’s important
to acknowledge the resultant uptick in computational demands, predominantly
from the increased gradient computations per batch. Nevertheless, the overall
computational burden of our NTK-FSCIL approach remains comparatively modest,
especially when juxtaposed with conventional NTK fitting methodologies,
striking a balance between computational efficiency and enhanced performance
outcomes.

Figure 5: NTK-related metrics and accuracy on the base session.

###  VI-H NTK Contributions

To measure model generalization during optimization, we randomly sample images
from both base and incremental sessions. For GPU efficiency, we only compute
the NTK matrix solely for the final nested linear layers. This allows us to
scrutinize the linear operations, thereby providing insight into how the
ultimate linear layers contribute to the model generalization. Following the
construction of the NTK matrix, we monitor them in real time by evaluating the
evolution of eigenvalues and the accuracy of the base session. The condition
number of the NTK matrix is then computed as an indicator of its stability, in
a manner analogous to Eq. 16, defined as the maximum-to-minimum eigenvalue
ratio.

The detailed evolutionary trajectories, including changes in the minimum and
maximum eigenvalues, condition number, and accuracy, are illustrated in Fig.
5. Interestingly, we observe a strong positive correlation between the
eigenvalues of the NTK matrix and the accuracy. The NTK’s eigenvalues
gradually converge, maintaining a state of equilibrium, mirroring the trend of
ascending and then stabilizing accuracy. Moreover, the condition number
remains relatively stable, illustrating the gradual convergence and stability
of the NTK matrix, thereby facilitating improved generalization.

To probe the impact of widening ConvNet, meta-learning, self-supervised pre-
training, curricular alignment and dual NTK regularization on NTK properties,
we conduct a comprehensive analysis on CIFAR100. We systematically omit each
module one by one to observe the resulting shifts in the NTK’s eigenvalue
distribution, as depicted in Fig. 4.

From Fig. 4LABEL:sub@NTK_without_linear, Fig. 4LABEL:sub@NTK_without_conv,
Fig. 4LABEL:sub@NTK_with_arcface and Fig. 4LABEL:sub@NTK_without_SSI, it’s
evident that Linear NTK Regularization, Convolutional NTK Regularization,
Curricular Alignment and Self-supervised Pre-training play vital roles in
ensuring NTK stability during optimization. Omitting or altering any of these
components leads to a markedly erratic NTK’s eigenvalue distribution,
characterized by significant variations between the highest and lowest
eigenvalues. Such instability negatively impacts the base session accuracy and
hampers model generalization. Similarly, as shown in Fig.
4LABEL:sub@NTK_without_ml and Fig. 4LABEL:sub@NTK_without_widernet, Meta-
Learning and a Wider ConvNet are essential for NTK convergence. Their absence
results in unpredictable fluctuations in the NTK eigenvalue distribution, with
the maximum and minimum eigenvalues lacking a consistent path during
optimization, further affecting model generalization.

In summary, each component within our NTK-FSCIL plays a critical role in
ensuring the NTK’s convergence and stability during optimization. Their
combined efforts enable our FSCIL learner to exhibit strong generalization
capabilities at the NTK theoretical level, leading to impressive performance
in incremental sessions. Consequently, it not only improves overall FSCIL
performance but also effectively mitigates the catastrophic forgetting
problem.

###  VI-I Visualization

To compellingly showcase the advantages of our NTK-FSCIL methodology, we
employ t-SNE visualization to depict the embeddings generated by our optimized
model, as illustrated in Fig. 6. Initially, we select five categories from the
base session, with each category comprising 20 samples, as displayed in Fig.
6LABEL:sub@fig:old_tsne. Following this, we enrich this dataset by adding four
additional categories from subsequent incremental sessions, with each new
category contributing 5 samples. This augmentation leads to a more detailed
and insightful t-SNE visualization, as exhibited in Fig.
6LABEL:sub@fig:new_tsne, which offers a clearer perspective on the optimized
embedding space.

(a) Five classes in base session

(b) Nine classes in both sessions

Figure 6: t-SNE visualization of embeddings from randomly sampled instances in
the base session and incremental sessions.

As demonstrated in Fig. 6, our NTK-FSCIL method distinctively segregates
samples from both base and incremental sessions, showcasing enhanced
separability and a clear testament to the model’s generalization capabilities.
This visualization not only highlights the effectiveness of our approach in
creating distinct representations for different categories but also
underscores its ability to adapt and maintain performance across incremental
learning phases.

##  VII Conclusion

Our research applies the Neural Tangents Kernel (NTK) theory to the FSCIL
paradigm, establishing a robust correlation between model generalization
capabilities and NTK convergence and NTK-related generalization loss. Through
both theoretical and empirical analyses, we validate the significance of NTK
in FSCIL generalization. Our NTK-FSCIL approach, emphasizing NTK matrix
optimization, attains exceptional results on well-established FSCIL
benchmarks.

## References

  * [1] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2001–2010.
  * [2] S. Yan, J. Xie, and X. He, “Der: Dynamically expandable representation for class incremental learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3014–3023.
  * [3] A. Douillard, A. Ramé, G. Couairon, and M. Cord, “Dytox: Transformers for continual learning with dynamic token expansion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9285–9295.
  * [4] C. Zhang, N. Song, G. Lin, Y. Zheng, P. Pan, and Y. Xu, “Few-shot incremental learning with continually evolved classifiers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 12 455–12 464.
  * [5] D.-W. Zhou, H.-J. Ye, L. Ma, D. Xie, S. Pu, and D.-C. Zhan, “Few-shot class-incremental learning by sampling multi-phase tasks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [6] J. Yoon, S. Madjid, S. J. Hwang, C.-D. Yoo _et al._ , “On the soft-subnetwork for few-shot class incremental learning,” in _International Conference on Learning Representations_. International Conference on Learning Representations, 2023.
  * [7] Y. Yang, H. Yuan, X. Li, Z. Lin, P. Torr, and D. Tao, “Neural collapse inspired feature-classifier alignment for few-shot class-incremental learning,” in _International Conference on Learning Representations_ , 2023.
  * [8] R. Novak, J. Sohl-Dickstein, and S. S. Schoenholz, “Fast finite width neural tangent kernel,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 17 018–17 044.
  * [9] F. Cagnetta, A. Favero, and M. Wyart, “What can be learnt with wide convolutional neural networks?” in _International Conference on Machine Learning_. PMLR, 2022.
  * [10] A. Jacot, F. Gabriel, and C. Hongler, “Neural tangent kernel: Convergence and generalization in neural networks,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [11] H. Wang, Y. Wang, R. Sun, and B. Li, “Global convergence of maml and theory-inspired neural architecture search for few-shot learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9797–9808.
  * [12] F. Caron, F. Ayed, P. Jung, H. Lee, J. Lee, and H. Yang, “Over-parameterised shallow neural networks with asymmetrical node scaling: Global convergence guarantees and feature learning,” _arXiv preprint arXiv:2302.01002_ , 2023.
  * [13] S. S. Du, X. Zhai, B. Poczos, and A. Singh, “Gradient descent provably optimizes over-parameterized neural networks,” _arXiv preprint arXiv:1810.02054_ , 2018.
  * [14] D. Zou, Y. Cao, D. Zhou, and Q. Gu, “Stochastic gradient descent optimizes over-parameterized deep relu networks,” _arXiv preprint arXiv:1811.08888_ , 2018.
  * [15] Z. Allen-Zhu, Y. Li, and Z. Song, “A convergence theory for deep learning via over-parameterization,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 242–252.
  * [16] A. Wei, W. Hu, and J. Steinhardt, “More than a toy: Random matrix models predict how real-world neural representations generalize,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 23 549–23 588.
  * [17] D. Richards and I. Kuzborskij, “Stability & generalisation of gradient descent for shallow neural networks without the neural tangent kernel,” in _Advances in Neural Information Processing Systems_ , A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021.
  * [18] S. Bombari and M. Mondelli, “Stability, generalization and privacy: Precise analysis for random and ntk features,” _arXiv preprint arXiv:2305.12100_ , 2023.
  * [19] H. Taheri and C. Thrampoulidis, “Generalization and stability of interpolating neural networks with minimal width,” _arXiv preprint arXiv:2302.09235_ , 2023.
  * [20] R. M. Neal and R. M. Neal, “Priors for infinite networks,” _Bayesian Learning for Neural Networks_ , pp. 29–53, 1996.
  * [21] N. Le Roux and Y. Bengio, “Continuous neural networks,” in _Artificial Intelligence and Statistics_. PMLR, 2007, pp. 404–411.
  * [22] T. Hazan and T. Jaakkola, “Steps toward deep kernel methods from infinite neural networks,” _arXiv preprint arXiv:1508.05133_ , 2015.
  * [23] S. Du and W. Hu, “Width provably matters in optimization for deep linear neural networks,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 1655–1664.
  * [24] X. Tao, X. Hong, X. Chang, S. Dong, X. Wei, and Y. Gong, “Few-shot class-incremental learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 183–12 192.
  * [25] G. Shi, J. Chen, W. Zhang, L.-M. Zhan, and X.-M. Wu, “Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 6747–6761, 2021.
  * [26] M. Hersche, G. Karunaratne, G. Cherubini, L. Benini, A. Sebastian, and A. Rahimi, “Constrained few-shot class-incremental learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9057–9067.
  * [27] Z. Ji, Z. Hou, X. Liu, Y. Pang, and X. Li, “Memorizing complementation network for few-shot class-incremental learning,” in _IEEE Transactions on Image Processing_ , 2023, pp. 937–948.
  * [28] D.-W. Zhou, F.-Y. Wang, H.-J. Ye, L. Ma, S. Pu, and D.-C. Zhan, “Forward compatible few-shot class-incremental learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9046–9056.
  * [29] C. Peng, K. Zhao, T. Wang, M. Li, and B. C. Lovell, “Few-shot class-incremental learning from an open-set perspective,” in _European Conference on Computer Vision_. Springer, 2022, pp. 382–397.
  * [30] J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington, “Wide neural networks of any depth evolve as linear models under gradient descent,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [31] G. Yang, “Tensor programs ii: Neural tangent kernel for any architecture,” _arXiv preprint arXiv:2006.14548_ , 2020.
  * [32] A. Canatar, B. Bordelon, and C. Pehlevan, “Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks,” _Nature communications_ , vol. 12, no. 1, p. 2914, 2021.
  * [33] Q. Nguyen, M. Mondelli, and G. F. Montufar, “Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8119–8129.
  * [34] M. Murray, H. Jin, B. Bowman, and G. Montufar, “Characterizing the spectrum of the ntk via a power series expansion,” _arXiv preprint arXiv:2211.07844_ , 2022.
  * [35] Z. Chi, L. Gu, H. Liu, Y. Wang, Y. Yu, and J. Tang, “Metafscil: a meta-learning approach for few-shot class incremental learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 14 166–14 175.
  * [36] C. Wei, J. D. Lee, Q. Liu, and T. Ma, “Regularization matters: Generalization and optimization of neural nets vs their induced kernel,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [37] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 9912–9924, 2020.
  * [38] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9650–9660.
  * [39] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 1597–1607.
  * [40] X. Chen and K. He, “Exploring simple siamese representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 750–15 758.
  * [41] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 000–16 009.
  * [42] X. Chen*, S. Xie*, and K. He, “An empirical study of training self-supervised vision transformers,” _arXiv preprint arXiv:2104.02057_ , 2021.
  * [43] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [44] K. Tian, Y. Jiang, qishuai diao, C. Lin, L. Wang, and Z. Yuan, “Designing BERT for convolutional networks: Sparse and hierarchical masked modeling,” in _International Conference on Learning Representations_ , 2023. [Online]. Available: https://openreview.net/forum?id=NRxydtWup1S
  * [45] Y. Zou, S. Zhang, Y. Li, and R. Li, “Margin-based few-shot class-incremental learning with class-level overfitting mitigation,” in _Advances in Neural Information Processing Systems_ , A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022.
  * [46] Y. Huang, Y. Wang, Y. Tai, X. Liu, P. Shen, S. Li, J. Li, and F. Huang, “Curricularface: adaptive curriculum learning loss for deep face recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5901–5910.
  * [47] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [48] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The caltech-ucsd birds-200-2011 dataset,” 2011.
  * [49] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” _International Journal of Computer Vision (IJCV)_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [50] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu, “Large scale incremental learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 374–382.
  * [51] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4690–4699.
  * [52] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu, “Cosface: Large margin cosine loss for deep face recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5265–5274.
  * [53] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/forum?id=r1Ddp1-Rb
  * [54] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 6023–6032.
  * [55] D. Hendrycks*, N. Mu*, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, “Augmix: A simple method to improve robustness and uncertainty under data shift,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=S1gmrxHFvB
  * [56] J.-H. Kim, W. Choo, and H. O. Song, “Puzzle mix: Exploiting saliency and local statistics for optimal mixup,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5275–5285.
  * [57] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 10 012–10 022.
  * [58] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 11 976–11 986.
  * [59] H. Zhuang, Z. Weng, R. He, Z. Lin, and Z. Zeng, “Gkeal: Gaussian kernel embedded analytic learning for few-shot class incremental task,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 7746–7755.
  * [60] D.-Y. Kim, D.-J. Han, J. Seo, and J. Moon, “Warping the space: Weight space rotation for class-incremental few-shot learning,” in _International Conference on Learning Representations_ , 2023. [Online]. Available: https://openreview.net/forum?id=kPLzOfPfA2l

|  Jingren Liu received the B.S. degree in Computer Science and Technology
from Nanjing University of Finance and Economy, Nanjing, China, in 2019, and
is currently working toward the PhD degree in the School of Electrical and
Information Engineering, Tianjin University, Tianjin, China. His current
research interests include continual learning, few shot leanring, and prompt
learning.
---|---
|  Zhong Ji received the Ph.D. degree in signal and information processing
from Tianjin University, Tianjin, China, in 2008. He is currently a Professor
with the School of Electrical and Information Engineering, Tianjin University.
He has authored over 100 technical articles in refereed journals and
proceedings. His current research interests include continual learning, few
shot leanring, and cross-modal analysis.
---|---
|  YanWei Pang received the Ph.D. degree in electronic engineering from the
University of Science and Technology of China, Hefei, China, in 2004. He is
currently a Professor with the School of Electrical and Information
Engineering, Tianjin University, Tianjin, China. He has authored over 200
scientific papers. His current research interests include object detection and
recognition, vision in bad weather, and computer vision.
---|---
|  YunLong Yu received the Ph.D. degree in information and communication
engineering from Tianjin University, Tianjin, China, in 2019. He is currently
a Distinguished Associate Researcher with the College of Information Science
and Electronic Engineering, Zhejiang University, Hangzhou, China. His current
research interests include machine learning and computer vision.
---|---