# MASIL: Towards Maximum Separable Class Representation for Few Shot Class
Incremental Learning

Anant Khandelwal
Applied Scientist, Amazon

###### Abstract

Few Shot Class Incremental Learning (FSCIL) with few examples per class for
each incremental session is the realistic setting of continual learning since
obtaining large number of annotated samples is not feasible and cost
effective. We present the framework MASIL as a step towards learning the
maximal separable classifier. It addresses the common problem i.e forgetting
of old classes and over-fitting to novel classes by learning the classifier
weights to be maximally separable between classes forming a simplex
Equiangular Tight Frame. We propose the idea of concept factorization
explaining the collapsed features for base session classes in terms of concept
basis and use these to induce classifier simplex for few shot classes. We
further adds fine tuning to reduce any error occurred during factorization and
train the classifier jointly on base and novel classes without retaining any
base class samples in memory. Experimental results on miniImageNet, CIFAR-100
and CUB-200 demonstrate that MASIL outperforms all the benchmarks.

##  1 Introduction

The success of Convolutional Neural Networks (CNN) in wide range of computer
vision tasks [40, 27, 60, 48, 51, 43] relies on the fact that the training
requires large scale image datasets [14] and the train and test distributions
are almost identical [41]. However, when deploying them in real world
environments it requires that these models to quickly adapt to changing
streams of data and hence can recognize the novel classes emerged over a
period of time. But the underlying bottleneck for this adaptation is that CNN
requires large amount of data to be collected for each of the novel classes,
this takes lot of human effort to annotate them which is infeasible. However,
annotating only a few samples seems reasonable, we term this ability to adapt
to novel classes (with only few examples) without forgetting the old classes
as the few shot class incremental learning (FSCIL). Fine-tuning the pre-
trained network with limited number of training examples of only novel classes
cause the model to forget old classes (catastrophic forgetting) and
overfitting on recent novel classes [22, 31, 68, 17]. Large amount of studies
has been conducted to solve the problem of catastrophic forgetting [25]. This
includes approaches based on: constraining the weight changes [37, 78, 16, 2,
44], retaining the samples from previous data in a memory [54, 13, 6, 63, 2]
data augmentation [77, 84, 86, 72], dynamic expansion based architectures
(DEA) which expands the network for each new incoming task id while the
weights of the base network are frozen for learning keeping both old and new
information [20, 24, 33, 73, 18, 45]. All these approaches are broadly
categorised into two main themes i.e. multi-task and multi-class. Multi-task
approaches like DEA requires resolving the task id during inference, which is
typically unavailable. Multi-class scenario refers to learning a single
classifier with a aim to recognize the base and novel classes in a single
task. In this paper we study the FSCIL problem under multi-class scenario,
since it is more realistic and practical. Recent approaches [80, 28, 1] have
proposed to learn the backbone network as feature extractor using data of base
classes, and then use this frozen feature extractor to learn the classifier
prototypes for novel classes incrementally. But this does not guarantee the
maximum separability between the classifier prototypes for base and novel
classes and hence can lead to the confusion between the old and new classes
resulting in limited performance. Other approaches [9, 1, 28], which uses
custom loss functions and regularizers to learn the classifier prototypes for
novel classes along with preventing forgetting on base classes, are also
limited by performance because of misalignment between fixed features of base
classes and classifier. Recent work NC-FSCIL[75] proposed the use of neural
collapse to learn the maximally separable classifier. They proposed to learn
the two layer classifier with pre-fixed simplex weights for base and each
incremental sessions. The two layer classifier training with pre-fixed weights
for few shot novel classes will not be able to generalize well and hence
results in overfitted class representation. Opposed to them we proposed to
learn itself the maximum separable class representation using Neural Collapse
properties, but we used concept factorization on backbone network to be able
to represent any class in general and hence obtain the generalized classifier
for novel few shot classes.

In this work, we address this problem of misalignment between fixed features
of backbone network (feature extractor) and classifier prototypes to prevent
forgetting of base class. Towards that we attempt to learn the maximal
separable classifier to avoid confusion between base and novel classes in each
incremental session. Our work is inspired from two main studies: 1) Neural
Collapse (NC) for imbalanced data [12, 53] and, 2) Concept Factorization [19,
35]. Neural collapse is the phenomenon where the network when trained beyond
zero error towards zero loss, results in collapsing the last layer features of
backbone network to form an Equiangular Tight Frame (ETF). The vertices of
this frame denotes the feature vector representing the class and aligned with
classifier prototype of the corresponding class [53]. This guarantees a
maximal separable classifier since ETF is a geometric structure forming a
simplex where the within class variance is minimized (because of collapse to a
single vector) and between class variance is maximized lying at equal angles
from each other. However, with pre-fixed simplex for base classes, the feature
extractor is easy to train since the sufficient data is available for each of
the class in base session ($t=0$) resulting in collapse but for any
incremental session $t\geq 1$ with few labelled samples learning the collapsed
features for novel classes is challenging since with few samples (as much as 5
samples for a class) the fixed feature extractor is not able to align well
with the novel class prototype. To resolve that we introduce the mechanism of
concept factorization, where we dissect the collapsed feature extractor on
base session to identify the concept basis in the input images. Once the
concept basis (”concept bank”) is identified from base session, we recognize
them as the building block from which the incremental session classifier
simplex is induced and hence the new set of coefficients can be learnt for
inducing the simplex with novel classes. This is additionally fine-tuned along
with base class simplex to further align this with few shot instances to
reduce any irreducible error occurred during calculating optimal coefficients
for the ”concept bank”. This has been illustrated in Fig.1. To summarize, our
main contributions are as follows:

  * •

We introduce a novel framework MASIL as an attempt to learn the maximal
separable classifier for FSCIL.

  * •

We identified the mechanism where the base session collapsed features
(obtained as per Neural Collapse properties) can further be dissected in terms
of ”concept bank”, which forms the basis for building classifier prototype of
novel classes encountered during incremental session.

  * •

Evaluation on three popular FSCIL benchmarks datasets demonstrating state-of-
the-art performance. Extensive ablation study has been done to analyze the
importance of loss function introduced using Neural collapse properties and
the advantage of simplex fine-tuning to reduce the irreducible error.

##  2 Related Work

###  2.1 Few Shot Learning

The idea of few shot learning (FSL) is to adapt the model on novel classes
(with only few labelled instances) without caring for the performance on base
classes. Most of the works uses meta-learning [66, 21, 65, 64] or metric
learning [64, 69, 66]. Recently, the approaches [23, 59] have demonstrated the
use of meta learning to recognize the base and novel classes both, by sampling
”fake” few shot classification task from base classes to learn a classifier
for novel classes. Finally, the learned classifier weights are combined to
jointly recognize the base and novel classes. Some of the works [59] regard
this as sort of incremental learning. Contrastively, FSCIL setting is much
more realistic where the base dataset is not accessible during the incremental
stage and we have to adapt the model for novel classes without catastrophic
forgetting [68, 17]. Metric learning approaches focus on learning a strong
backbone network for learning transferable features across the tasks, on top
which the similarity function (like k-nearest neighbours in [69], non linear
distance metric in [66] ) is learnt to demonstrate the ability to classify the
novel classes with transferable features. However, this requires to train the
as much similarity function as the number of incremental sessions in FSCIL but
the aim of FSCIL is to train one unified classifier for the base and novel
classes. We will discuss in the next section how existing works have dealt the
problem of FSCIL different from FSL.

Figure 1: MASIL: Illustrating the concept basis obtained from concept
factorization of features after feature extractor and their relation with
input images. After base session training, classifier simplex for novel
classes are induced from these concept basis. Implicit memory (not shown)
storing the feature mean of classes seen till current session. This is used to
jointly updating the weights of classifier for both base and novel classes
during Simplex Finetuning.

###  2.2 Few Shot Class Incremental Learning

Class Incremental Learning (CIL): We start by first discussing the idea of
Class Incremental Learning (CIL), it aims to learn a classifier that manages
to continuously update itself to recognize all the novel classes without
forgetting the base classes [58, 7, 44]. To overcome this forgetting mechanism
CIL studies has been categorized into three broad categories: regularization
based [37, 44, 16], rehearsal based [54, 13, 2, 6], and knowledge distillation
[58, 30, 71]. Regularization based methods constraint the weight changes on
the novel classes thereby keeping the information previously learnt for base
classes, this causes these methods to suffer for generalization on novel
classes because of small allowed change in weights. Rehearsal based methods in
which the model is continually be refreshed using old data reserve so that it
maintains to learn the novel classes along with old classes. This is limited
by the amount of old data it can retain in memory, and the how the instances
from old data to be selected for maximal information with minimum memory
requirements. These methods are not scalable to large number of classes
because of the limited memory. For example, iCaRL[58] learns the nearest
neighbour classifier for novel classes while maintaining the memory of
exemplars from base session. Knowledge Distillation based methods requires the
use of large teacher model to guide the learning of small student model [74,
29]. It works by distilling the previously learned information to new model
with novel classes, so as to avoid forgetting of base classes. Recent studies
[16, 18, 30] conducted the distillation on feature level rather than on output
logit level at the classifier. However, these solutions suffer from a problem
of distinguishing between base and novel classes leading to limitation in the
performance.

Few Shot Class Incremental Learning (FSCIL): Compared to CIL setting, FSCIL
aims to learns the novel classes (along with base classes) with few labelled
instances [68, 17], which is much more realistic and hard, since learning from
few instances of novel classes causes over-fitting on novel classes [64, 66].
In order to do this, some studies have focused to align base and incremental
session using augmentation [56], searching for flat minima [62]. However, for
avoiding over-fitting to novel classes it is required that the classifier
prototypes for novel classes should be maximally separable from base classes.
Adjusting prototypes for base classes is not feasible since that requires the
use of base session data. However, these studies [80, 85] have focused on
evolving prototypes for novel classes. Large number of existing works have
focused on building the custom loss and regularizer [59, 30, 67, 34, 50, 28,
1, 74]. However, the same disadvantages we discussed in CIL for regularization
and custom loss functions applies in FSCIL as well. In this work we focused on
the optimal evolution of prototypes for novel classes which is derived from
the same function using which base class prototypes have been developed and
ensuring the maximal separability between old and novel classes.

##  3 Problem Statement and Context

In this section we will introduce the problem definition of Few Shot Class
Incremental Learning in Section 3.1 and context in subsequent sections.

###  3.1 Few Shot Class Incremental Learning

Formally, we define Few Shot Class Incremental Learning (FSCIL) as the stream
of labelled data in time sequence as $\mathcal{D}_{0}$, $\mathcal{D}_{1}$,
….., where
$\mathcal{D}_{t}=\\{(\mathbf{x}_{j}^{t},y_{j}^{t})\\}_{j=1}^{j=|\mathcal{D}_{t}|}$.
$\mathcal{C}_{t}$ be the number of classes in training set $\mathcal{D}_{t}$,
where $\forall(i,j)\textrm{ }\mathcal{C}_{i}\cap\mathcal{C}_{j}=\emptyset$.
Specifically, we consider $\mathcal{D}_{0}$ as the base session with large
label space $\mathcal{C}_{0}$ with each class $c\in\mathcal{C}_{0}$ have
sufficient training images. For $t>0$ each of the incremental session
$\mathcal{D}_{t}$ have only few labelled images for each novel classes. FSCIL
is defined as the time step incremental training of model $\Theta$ on
$\mathcal{D}_{t}\textrm{ }\forall\textrm{ }t>0$ with no access to any of the
previous labelled set from $\mathcal{D}_{0}$ to $\mathcal{D}_{t-1}$. For
$t>0\textrm{ }\mathcal{D}_{t}$ we denote the setting as C classes with K
training examples per class as C-way K-Shot FSCIL where
$\mathcal{C}_{t}\cap\mathcal{C}_{t}^{\prime}=\emptyset\textrm{ }\forall t\neq
t^{\prime}$. After each incremental session training with $\mathcal{D}_{t}$,
model $\Theta$ is evaluated to recognize all the training classes encountered
so far i.e. $\cup_{i=0}^{i=t}\textrm{ }\mathcal{C}^{i}$. Hence, FSCIL not only
aims to recognize novel classes but to avoid forgetting of the old classes and
the setting of learning of novel classes is highly imbalanced and suffers from
data scarcity problem as well. This makes FSCIL setting more suited for real
world applications.

Initialization: Assuming $\mathcal{C}_{0}$ as the number of base classes and
we have total $T$ incremental session and each session has $k$ classes, so
there will be total $K=\mathcal{C}_{0}+Tk$ classes. To be able to perform
FSCIL, we denote the model trained on base session consists of backbone
feature extractor $f(;\theta_{f})$ and classifier parameters
$\mathbf{W}\in\mathbb{R}^{K\times d}$, where $\mathbf{W}$ is a MLP classifier
consisting of L layers denoted as
$\mathbf{W}=\mathbf{W}_{1}\mathbf{W}_{2}.....\mathbf{W}_{L}$. For input
$X\in\mathbb{R}^{n}$ we denote the features obtained from feature extractor as
$\mathbf{H}=f(X,\theta_{f})\in\mathbb{R}^{d\times N}$, where $N$ is the total
number of training instances. Similar to [12] we also consider last layer
features $\mathbf{H}$ as freely optimization variables. The optimization
objective is then defined as follows:

$\underset{\mathbf{W},\mathbf{H}}{min}\textrm{
}\mathcal{L}(\mathbf{W},\mathbf{H})=\frac{1}{2N}||\mathbf{W}\mathbf{H}-\mathbf{Y}||_{F}^{2}+\frac{\lambda_{\mathbf{W}}}{2}||\mathbf{W}||_{F}^{2}+\\\
\frac{\lambda_{\mathbf{H}}}{2}||\mathbf{H}||_{F}^{2}$ (1)

where $\mathbf{Y}\in\mathbb{R}^{K\times N}$, is the class label for each of
the training instance in $N$ instances and $\lambda_{W}$, $\lambda_{H}$ are
regularization hyperparameters.

###  3.2 Neural Collapse

In recent works [53, 12] which have studied the practice of training the DNN
beyond zero error towards zero loss. This reveals the geometric structure in
the form of simplex equiangular tight frame, formed by the last layer features
along with classifier weights. This has been demonstrated on the balanced data
and models with various popular architectures. Neural Collapse as defined in
[53] consists of the following four properties:

  * •

($\mathcal{NC}1$) Variability Collapse: Last layer features of the backbone
network for a particular class collapse to within-class mean.

  * •

($\mathcal{NC}2$) Convergence: results in optimal class-means which are
equally and maximally pairwise separated forming a simplex Equiangular Tight
Frame (ETF).

  * •

($\mathcal{NC}3$) Classifier Convergence: Optimal class means forming ETF are
aligned to the corresponding classifier weights uptio rescaling.

  * •

($\mathcal{NC}4$) Simplification to nearest class center When
($\mathcal{NC}1$)-($\mathcal{NC}3$) holds, the model prediction using logits
respects nearest class centers.

In addition to the balanced data, this [12] have derived the geometrical
analysis for the imbalanced data given as:

######  Definition 1.

Let ($\mathbf{W}$*, $\mathbf{H}$*) be the global optimizer of equation 1,
$r=min(K,d)$ and $\mathbf{W}=U_{W}S_{W}V_{W}^{T}$ be the SVD factorization of
$\mathbf{W}$. Then the following holds for the imbalanced data:

  * •

($\mathcal{NC}$1) leads to collapse of features within the same class
$\mathbf{H}^{*}=\overline{\mathbf{H}}^{*}\mathbf{Y}$, where
$\overline{\mathbf{H}}^{*}=[\mathbf{h}^{*}_{1},\mathbf{h}^{*}_{2},....\mathbf{h}^{*}_{K}]\in\mathbb{R}^{d\times
K}$

  * •

($\mathcal{NC}$3) leads to alignment between classifier weights and
corresponding class mean as
$\mathbf{w}_{k}^{*}=\sqrt{\frac{n_{k}\lambda_{H}}{\lambda_{W}}}\mathbf{h}_{k}^{*}\textrm{
}\forall k\in[K]$, where $n_{k}$ is the number of instances of class $k$.

  * •

($\mathcal{NC}$2) leads to optimal class means equally and maximally separated
forming simplex Equiangular Tight Frame (ETF)
$\mathbf{W}^{*}\mathbf{W}^{*\top}=\textrm{ diag }\\{s_{k}^{2}\\}_{k=1}^{k=K}$,
where $s_{k}$ are the singular values of $\mathbf{W}^{*}$

Another approach i.e. Deep Simplex Classifier[8], proposed the optimization
problem as minimization of features obtained from feature extractor to the
vertices of simplex as:

$\underset{\mathbf{h}_{i}\in\mathbf{H}^{T}}{\textrm{min
}}\frac{1}{n}\sum_{i=1}^{n}||\mathbf{h}_{i}-{s_{y}}_{i}||^{2}$ (2)

where ${s_{y}}_{i}$ is vertex of simplex and is treated as the class center
for class $y_{i}$.

##  4 MASIL

Overall framework of our proposed method is illustrated in Fig. 1. FSCIL aims
to learn the classifier weights $W^{K\times d}$ which works for all classes
irrespective of whether they belongs to the base classes during $t=0$ or few
shot classes during $t>0$. Traditionally, this has been achieved by first
learning the classifier weights for $\mathcal{C}_{0}$ base classes and then
learn the weights for novel classes $W^{(t)}\in\mathbb{R}^{k\times d}$ with
the regularized constraint in the loss function that the old weights
$W\in\mathbb{R}^{C_{0}+(t-1)k}$ be preserved with little or no updates.
However, this leads to misalignment between the classifier prototypes of old
and novel classes causing old new confusion(ONC) [31] and catastrophic
forgetting [25]. This causes drop in performance of FSCIL classifier as the
number of incremental session grows resulting in poor generalizability even in
recognizing the base classes. To mitigate this, in this work we adopted the
properties obtained from Neural Collapse to learn maximally separable
classifier along with concept factorization to learn classifier weights
(organized as simplex) for novel classes with few samples. We restricted the
feature extractor from updates during incremental session training and rely on
concept factorization of the activations obtained for base classes to obtain
the basis of concepts called ”concept bank”, using which we can represent
maximally separable classifier weights i.e. simplex for few shot classes. To
represent the classifier simplex using ”concept bank” it requires to solve
only for the coefficient matrix which can be done by just solving the Non
Negative Least Squares (NNLS).

###  4.1 Concept Factorization

The idea of concept factorization relates to the phenomena of neural collapse,
where it learns to maximally separate the classes by forming the simplex at
class level on both levels of class features and classifier weights. In order
to achieve this it merge the activations (during forward pass) of the same
class until they all converge to the one hot class vector at the logits layer
as depicted in equation 2. This allows the class wise feature vectors which
are concentrated at higher layers to be recursively broken into multiple
concepts moving from highest layer to lower layers tracing back to the input
images where it can be explained with regions as concepts, combination of
which makes it possible to be able to classify it to particular class. We
adopted NMF (Non Negative Matrix Factorization) as in [19] of activations
obtained at the output of feature extractor given as:

$\underset{\mathbf{P}\geq 0,\mathbf{Q}\geq 0}{\textrm{min
}}\frac{1}{2}||\mathbf{A}-\mathbf{P}\mathbf{Q}^{T}||_{F}^{2}$ (3)

where, $||.||_{F}$ is the Frobenius norm, the activations
$\mathbf{A}\in\mathbb{R}^{n\times d}$ obtained from crop of images
$X_{i}={\tau}(x_{i})\textrm{ }X_{i}\in X^{n\times p}$ with $\tau$ is a crop
function. We take random crops (governed by $\tau$) of images, this results in
unique concepts across the categories to be able to build the bank of unique
concept vectors called ”concept bank”. Activations at the last layer of
feature extractor after global pooling for these random crops is given as
$\textbf{A}=f(X,\theta_{f})\in\mathbb{R}^{n\times d}$. NMF is simply the
factorization of concept activations $\mathbf{A}$ into the ”concept bank”
$\mathbf{Q}\in\mathbb{R}^{v\times d}$ (where it follows low rank factorization
$v\ll\mathbf{min}(n,p)$) and coefficients $\mathbf{P}\in\mathbb{R}^{n\times
v}$ denote the importance of each of the concepts in explaining the
activations $\mathbf{A}$. Once the ”concept bank” is precomputed, we can
obtain the coefficients $P(x)$ for any input $x$ using NNLS (Non-Negative
Least Squares) i.e. $\underset{P\geq 0}{min}\textrm{
}\frac{1}{2}||f(x;\theta_{f})-P(x)\mathbf{Q}^{T}||_{F}^{2}$. Relating
activation factorization in equation 3 and neural collapse in equation 2,
implies that the activations when collapses to the mean features vector for
each class forming the class simplex vector which is composed of concept basis
vectors and the corresponding coefficients, combining for all classes which
gives the overall basis called ”concept bank”.

###  4.2 NMF Layer

During NMF factorization of equation 3, we keep the feature extractor
$f(.;\theta_{f})$ frozen. We approached the NMF problem solution using ADMM
(Alternating Direction Method of Multipliers) [5] since NMF is non-convex, but
however it can be made convex by fixing the value of either of the two factors
$(\mathbf{P,Q})$ which requires alternating update of either of two factors
fixing one at a time, which is equivalent to solving a Non-Negative Least
Squares (NNLS) problem making it convex. This alternating update mechanism
called as ADMM, formulated as:

$\mathbf{P}_{t+1}=\textrm{arg }\underset{\mathbf{P}\geq 0}{\textrm{min
}}\frac{1}{2}||\mathbf{A-PQ_{t}}^{T}||_{F}^{2}$ (4)
$\mathbf{Q}_{t+1}=\textrm{arg }\underset{\mathbf{Q}\geq 0}{\textrm{min
}}\frac{1}{2}||\mathbf{A-P_{t}Q}^{T}||_{F}^{2}$ (5)

It ensures global or local minimum since each of the NNLS problem obeys
Karush–Kuhn–Tucker (KKT) optimality conditions [36, 42]. Using these
conditions forming the implicit function [26] makes the implicit
differentiation [26, 38, 3] allows to compute the gradients $(\frac{\partial
P}{\partial A},\frac{\partial Q}{\partial A})$, but however we have to relate
the concepts with the input image regions we require to compute
$(\frac{\partial P}{\partial X},\frac{\partial Q}{\partial X})$. This can be
calculated as:

$\frac{\partial P}{\partial X}=\frac{\partial A}{\partial X}\frac{\partial
P}{\partial A},\textrm{ }\frac{\partial Q}{\partial X}=\frac{\partial
A}{\partial X}\frac{\partial Q}{\partial A}$ (6)

Computation of $\frac{\partial A}{\partial X}$ is fairly straight word using
Pytorch. More details on implementation of combining gradients from implicit
differentiation in Jax [32, 4] and gradient from Pytorch computation is
detailed in Section A. Once we precompute the ”concept bank” for base classes
using equation 5 and 6, we fixed the $\mathbf{Q}$ and allows only to compute
optimal coefficients $P(x)$ for any input $x$ using NNLS

$\underset{\mathbf{P}\geq 0}{\textrm{min }}\textrm{
}\frac{1}{2}||f(x;\theta_{f})-P(x)\mathbf{Q}^{T}||_{F}^{2}$ (7)

which give the optimal representation of activation for any input $x$ in terms
concept basis vectors.

###  4.3 Classifier Simplex Representation

Equation 2 is optimizing the feature representation for each class resulting
in collapsed representation for class $y_{i}$ as $s_{y_{i}}$. Similarly,
equation 1 results in simplex representation for each class i.e.
$\mathbf{w}_{y_{i}}\in\mathbf{W}$. So if we consider the normalized simplex
representation on a unit hypersphere [8] of each class then:

$\mathbf{w}_{y_{i}}^{T}s_{y_{i}}=1\textrm{ }\forall\textrm{
}y_{i}\in\cup_{j=0}^{j=t}\mathcal{C}_{j}$ (8)

which results in the modified loss function of equation 2 to:

$\underset{\mathbf{h}_{i}}{\textrm{min
}}\frac{1}{|\mathcal{D}_{j}|}\sum_{(x_{i},y_{i})\in\mathcal{D}_{j}}||\mathbf{w}_{y_{i}}^{T}\mathbf{h}_{i}-1||_{F}^{2}$
(9)

s.t. $\mathbf{w}_{y_{i}}^{T}s_{y_{i}}=1$ which is same as in equation 1 and
hence follow the neural collapse properties. Moreover, optimizing equation 2,
results in the collapsed feature representation for all instances belonging to
that class. Additionally, equation 3 computes the best approximation of
collapsed feature representation $\mathbf{H}\approx\mathbf{PQ}^{T}$. For any
input $(x_{i},y_{i})$ belongs to $\mathcal{D}_{j},\textrm{ }j>0$, then optimal
$\mathbf{h}_{i}$ obtained from equation 7 is given as:

$\mathbf{h}_{i}=\mathbf{P}(x_{i})\mathbf{Q}^{T}$ (10)

From ($\mathcal{NC}1$), the collapsed feature representation of each class
converge to a unique vector e.g. for class $y_{i}$ the feature representation
of all instances is denoted as $\mathbf{H}_{y_{i}}\in\mathbf{H}$,
($\mathcal{NC}1$) implies covariance
$\sum_{\mathbf{H}_{y_{i}}}\xrightarrow{}0$. i.e. the features collapse to
their corresponding class means i.e.
$\mathbf{h}_{y_{i}}^{*}=\sum_{i=1}^{n_{y_{i}}}\mathbf{h}_{i}$, where
$n_{y_{i}}$ is the number of instances for class $y_{i}$, and as per loss in
equation 2, this is minimum when $s_{y_{i}}=\mathbf{h}_{y_{i}}^{*}$, then from
equation 8 and 10:

$\displaystyle\mathbf{\hat{w}}_{y_{i}}=\frac{1}{|\mathcal{D}_{j}|}\left(\sum_{(x_{i},y_{i})\in\mathcal{D}_{j}}\mathbf{P}(x_{i})\right)\mathbf{Q}^{T}\textrm{
}\forall j>0$ (11)

where the coefficients $\mathbf{P}(x_{i})$ for each instance of class $y_{i}$
are calculated using NNLS as per equation 7, additionally,
$y_{i}\in\mathcal{C}_{j},j>0$ are the few shot classes and classifier weights
are the optimal simplex representation for few shot classes. For base session
classes ($j=0$) the classifier simplex representation is simply
$\mathbf{\hat{w}}_{y_{i}}=s_{y_{i}}^{T}$. Since we implemented the classifier
using MLP with $L=2$ layers, for each layer the simplex representation is
$\mathbf{\hat{w}}_{l,y_{i}}=(\mathbf{\hat{w}}_{y_{i}})^{1/L}$.

###  4.4 Simplex Finetuning

In Section 4.3 we described the optimal simplex representation for each class
belongs to the few shot class $y_{i}\in\mathcal{C}_{j},j>0$. But, however due
to the the inherent irreducible error to NNLS, we approach the optimal
representation of simplex for few shot class by further fine-tuning the
classifier weights (keeping the feature extractor frozen) initialized using
simplex representation as obtained in equation 11. To avoid deviating the
weights to much from optimal simplex representation we add a constraint to the
loss in equation 9 as:

$\underset{\mathbf{w}_{y_{i}}}{\textrm{min}}\mathcal{L}(\mathbf{w}_{y_{i}})=\frac{1}{|\mathcal{D}_{j}|}\sum_{(x_{i},y_{i})\in\mathcal{D}_{j}}||\mathbf{w}_{y_{i}}^{T}\mathbf{h}_{i}-1||_{F}^{2}+\\\
\alpha||\mathbf{w}_{y_{i}}-\mathbf{\hat{w}}_{y_{i}}||_{F}^{2},\alpha\in[0,1]$
(12)

where $y_{i}\in\mathcal{C}_{j},j>0$ and feature extractor is frozen and hence
optimizing for the best $\mathbf{w}_{y_{i}}$. Since base session training
(i.e. $j=0\textrm{ and dataset }\mathcal{D}_{0}$) is governed using the loss
function of equation 2 and hence results in collapsed representation of
features at the terminal layer for each class $y_{i}\in\mathcal{C}_{0}$. For
the simplex representation for each class in $\mathcal{C}_{0}$ to remain
maximally separable with the ones obtained for few shot class we utilized the
collapsed representation of features for each class to further fine tune the
simplex representation, but without keeping the image instances in memory we
memorized the collapsed representation (which is the mean representation of
instance features for each class) in $\mathcal{M}$ given as:

$\mathcal{M}_{y_{i}}=\frac{1}{n_{y_{i}}}\sum_{i=1}^{n_{y_{i}}}\mathbf{h}_{i},\forall\mathcal{M}_{y_{i}}\in\mathcal{M}$
(13)

where, $n_{y_{i}}$ is the number of instances of class $y_{i}$. The updated
loss function during fine tuning stage include the base session classes and
few shot class is given as:

$\underset{\mathbf{w}_{y_{i}}}{\textrm{min}}\mathcal{L}(\mathbf{w}_{y_{i}})=\frac{1}{|\mathcal{D}_{j}|}\sum_{(x_{i},y_{i})\in\mathcal{D}_{j}}||\mathbf{w}_{y_{i}}^{T}\mathbf{h}_{i}-1||_{F}^{2}+\\\
\frac{1}{|\mathcal{M}|}\sum_{(\mathcal{M}_{y_{i}},y_{i})\in\mathcal{M}}||\mathbf{w}_{y_{i}}^{T}\mathcal{M}_{y_{i}}-1||_{F}^{2}+\\\
\alpha||\mathbf{w}_{y_{i}}-\mathbf{\hat{w}}_{y_{i}}||_{F}^{2},\alpha\in[0,1]$
(14)

Methods | Session Accuracy (%) ($\uparrow$) |  Average Acc. ($\uparrow$) |  Relative Improvement
---|---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
iCaRL [58] | 61.31 | 46.32 | 42.94 | 37.63 | 30.49 | 24.00 | 20.89 | 18.80 | 17.21 | 33.29 | +41.65
NCM [30] | 61.31 | 47.80 | 39.30 | 31.90 | 25.70 | 21.40 | 18.70 | 17.20 | 14.17 | 30.83 | +44.69
D-Cosine [69] | 70.37 | 65.45 | 61.41 | 58.00 | 54.81 | 51.89 | 49.10 | 47.27 | 45.63 | 55.99 | +13.23
TOPIC [68] | 61.31 | 50.09 | 45.17 | 41.16 | 37.48 | 35.52 | 32.19 | 29.46 | 24.42 | 39.64 | +34.44
IDLVQ [9] | 64.77 | 59.87 | 55.93 | 52.62 | 49.88 | 47.55 | 44.83 | 43.14 | 41.84 | 51.16 | +17.02
Self-promoted [85] | 61.45 | 63.80 | 59.53 | 55.53 | 52.50 | 52.50 | 46.69 | 43.79 | 41.92 | 52.76 | +16.94
CEC [80] | 72.00 | 66.83 | 62.97 | 59.43 | 56.70 | 53.73 | 51.19 | 49.24 | 47.63 | 57.75 | +11.23
LIMIT [83] | 72.32 | 68.47 | 64.30 | 60.78 | 57.95 | 55.07 | 52.70 | 50.72 | 49.19 | 59.06 | +9.67
Regularizer [1] | 80.37 | 74.68 | 69.39 | 65.51 | 62.38 | 59.03 | 56.36 | 53.95 | 51.73 | 63.71 | +7.13
MetaFSCIL [11] | 72.04 | 67.94 | 63.77 | 60.29 | 57.58 | 55.16 | 52.90 | 50.79 | 49.19 | 58.85 | +9.67
C-FSCIL [28] | 76.40 | 71.14 | 66.46 | 63.29 | 60.42 | 57.46 | 54.78 | 53.11 | 51.41 | 61.61 | +7.45
Data-free Replay [47] | 71.84 | 67.12 | 63.21 | 59.77 | 57.01 | 53.95 | 51.55 | 49.52 | 48.21 | 58.02 | +10.65
ALICE [56] | 80.60 | 70.60 | 67.40 | 64.50 | 62.50 | 60.00 | 57.80 | 56.80 | 55.70 | 63.99 | +3.16
SSFE-Net [52] | 72.06 | 66.17 | 62.25 | 59.74 | 56.36 | 53.85 | 51.96 | 49.55 | 47.73 | 57.74 | +11.13
NC-FSCIL [75] | 84.02 | 76.80 | 72.00 | 67.83 | 66.35 | 64.04 | 61.46 | 59.54 | 58.31 | 67.82 | +0.55
MASIL(Ours) | 85.15 | 77.00 | 72.20 | 67.92 | 66.60 | 64.2 | 61.50 | 59.60 | 58.86 | 68.11 |
Table 1: Performance comparison on miniImageNet with ResNet-18 as backbone architecture under 5-way 5-shot FSCIL setting. Table denotes the accuracy in each session, average accuracy across sessions and ”Relative Improvement” denotes the improvement of our method in the last session. Methods above separating line are CIL methods for FSCIL as in [68] and [80] Methods | Session Accuracy (%) ($\uparrow$) |  Average Acc. ($\uparrow$) |  Relative Improvement
---|---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
iCaRL [58] | 64.10 | 53.28 | 41.69 | 34.13 | 27.93 | 25.06 | 20.41 | 15.48 | 13.73 | 32.87 | +42.42
NCM [30] | 64.10 | 53.05 | 43.96 | 36.97 | 31.61 | 26.73 | 21.23 | 16.78 | 13.54 | 34.22 | +42.61
D-Cosine [69] | 74.55 | 67.43 | 63.63 | 59.55 | 56.11 | 53.80 | 51.68 | 49.67 | 47.68 | 58.23 | +8.47
TOPIC [68] | 64.10 | 55.88 | 47.07 | 45.16 | 40.11 | 36.38 | 33.96 | 31.55 | 29.37 | 42.62 | 26.78
Self-promoted [9] | 64.10 | 65.86 | 61.36 | 57.45 | 53.69 | 50.75 | 48.58 | 45.66 | 43.25 | 54.52 | +12.9
CEC [80] | 73.07 | 68.88 | 65.26 | 61.19 | 58.09 | 55.57 | 53.22 | 51.34 | 49.14 | 59.53 | +7.01
DSN [11] | 73.00 | 68.83 | 64.82 | 62.64 | 59.36 | 56.96 | 54.04 | 51.57 | 50.00 | 60.14 | +6.15
LIMIT [83] | 73.81 | 72.09 | 67.87 | 63.89 | 60.70 | 57.77 | 55.67 | 53.52 | 51.23 | 61.84 | +4.92
MetaFSCIL [1] | 74.50 | 70.10 | 66.84 | 62.77 | 59.48 | 56.52 | 54.36 | 52.56 | 49.97 | 60.79 | +6.18
C-FSCIL [28] | 77.47 | 72.40 | 67.47 | 63.25 | 59.84 | 56.95 | 54.42 | 52.47 | 50.47 | 61.64 | +5.68
Data-free Replay [47] | 74.40 | 70.20 | 66.54 | 62.51 | 59.71 | 56.58 | 54.52 | 52.39 | 50.14 | 60.78 | +6.01
ALICE [56] | 79.00 | 70.50 | 67.10 | 63.40 | 61.20 | 59.20 | 58.10 | 56.30 | 54.10 | 63.21 | +2.05
NC-FSCIL [75] | 82.52 | 76.82 | 73.34 | 69.68 | 66.19 | 62.85 | 60.96 | 59.02 | 56.11 | 67.50 | +1.12
MASIL(Ours) | 82.55 | 76.98 | 73.44 | 69.75 | 66.48 | 62.98 | 61.4 | 59.81 | 57.23 | 67.84 |
Table 2: Performance comparison on CIFAR-100 with ResNet-18 as backbone
architecture under 5-way 5-shot FSCIL setting. Table denotes the accuracy in
each session, average accuracy across sessions and ”Relative Improvement”
denotes the improvement of our method in the last session. Methods above
separating line are CIL methods for FSCIL as in [68] and [80]

where the constraint is now valid for base session classes as well along with
few shot classes with the fact that simplex representation for each class
should not deviate much (depends on the contributing factor $\alpha$) from the
optimal simplex representation. In each incremental session we train our
classifier network using this loss function after deriving the simplex
representation for each few shot classes from equation 11.

##  5 Experiments

We prove the effectiveness of MASIL on three well known FSCIL benchmark
datasets (as in ALICE [56]) described in Section 5.1 along with FSCIL setting
and compared its performance with the state-of-the-art methods (Section 5.2).
Training details and hyper parameters are discussed in Appendix A.

###  5.1 Dataset Details

  * •

CIFAR-100 [39] consists of 100 classes in total with color images of size
$32\times 32$. Each class consists of 500 images for training and 100 images
for testing. The base session ($t=0$) consists of 60 classes and the rest 40
classes contributed for 8 incremental session with 5-way 5-shot setting (i.e.
5 images for each of the 5 classes) for $1\leq t\leq 8$.

  * •

miniImageNet [61] is a variant of ImageNet[15] with color images of size
$84\times 84$. It also consits of same number of classes as CIFAR-100 and same
number of images in train and test, resulting in the same configuration for
base and incremental sessions.

  * •

CUB-200 [70] consists of 11,788 images (size $224\times 224$) in total
spanning across 200 classes. There are 5,994 images in train and 5,794 images
in test. Base session ($t=0$) consists of 100 classes and rest 100 classes
contributed towards 10 incremental session ($1\leq t\leq 10$) with 10-way
5-shot setting (5 images for 10 classes each).

###  5.2 Benchmark Evaluation

Performance comparison on miniImageNet, CIFAR-100 and CUB-200 is demonstrated
in Table 1, 2 and 5 (given in Appendix B due to space limitation)
respectively. Our method MASIL outperforms in all the methods in the last
session with relative improvement of +3.16%, +2.05% and +0.14% on
miniImageNet, CIFAR-100, CUB-200 respectively as compared to strongest
baseline ALICE [56]. Additionally, our method outperforms all the methods in
all the sessions (except on CUB-200 session 2). Moreover, on average accuracy
our method outperforms atleast by +1.79% as compared to strongest baseline,
collectively is an indicator that our model helps in mitigating the forgetting
issue in a realistic setting of continual learning namely FSCIL.

Models | miniImageNet | CIFAR-100 | CUB-200
---|---|---|---
Final ($\uparrow$) | Average ($\uparrow$) | Final ($\uparrow$) | Average ($\uparrow$) | Final ($\uparrow$) | Average ($\uparrow$)
Learnable + CE | 50.04 | 61.30 | 52.13 | 62.68 | 50.38 | 59.58
$\mathcal{NC}$ \+ CE | 56.66 | 68.23 | 54.42 | 64.00 | 56.83 | 65.51
$\mathcal{NC}$ \+ ETF Loss | 58.31 | 67.82 | 56.11 | 67.50 | 59.44 | 67.28
$\mathcal{NC}$ \+ ETF Loss + CF | 58.72 | 68.04 | 56.13 | 67.51 | 59.72 | 67.45
MASIL(Ours) | 58.86 | 68.11 | 57.23 | 67.84 | 60.24 | 67.54
Table 3: Ablation Studies on three datasets investigating the effects of
Simplex based loss, Concept Factorization and Simplex Fine Tuning

Figure 2: Average cosine similarities between different classes at each
session for Train (Left) and Test (Right) on miniImageNet. Calculation of
cosine similarity is done for all the classes encountered so far after the
model gets trained in current session.

###  5.3 Ablation Studies

We consider variations to base model (backbone network i.e. ResNet-18 with
classifier and memory as introduced in equation 13) to validate the 1) effects
of loss introduced in equation 9 (ETF) as compared to cross-entropy (CE) loss
with and without neural collapse induced simplex classifier prototypes, 2)
effects of few shot simplex induced with concept factorization and 3) effects
of simplex fine tuning. To validate the first effect there are two models. The
first model (Learnable + CE) uses a classifier with learning weights from CE
loss, which is the most common practice. Second model ($\mathcal{NC}$ \+ CE)
uses the CE loss with classifier weights as per the neural collapse properties
but uses the CE loss instead of the loss in equation 9. To validate the second
effect (third model i.e. $\mathcal{NC}$ \+ ETF Loss) we did not initialize the
classifier weights for few shot classes and train them as per the loss in
equation 9 with memory of base classes as in equation 13. For third effect
i.e. fourth model ($\mathcal{NC}$ \+ ETF Loss + CF) we reported the
performance without fine tuning and just use the classifier weights calculated
from concept factorization (CF) as in equation 11. Finally, we reported
performance of MASIL to compare among all of them. As shown in Table 3,
adopting the loss function in equation 9 is definitely helps in mitigating
performance drop as compared to CE loss even with classifier weights is
assumed to be forming simplex, and it further mitigates using the weight
initialized with CF and further with fine tuning. It indicates the success of
CF along with neural collapse towards optimal solution for FSCIL.

###  5.4 Analysing Classifier Weights

We further analysed the classifier weights alignment with respect to the mean
feature (collapsed feature) of each class. We used the classifier weights and
mean feature from each of the models described in ablation studies to validate
the effect of MASIL in learning the maximal separable classifier, where the
separable property between classes is measured by cosine similarity.
Specifically, we plotted the average cosine similarities between mean feature
and the classifier weights of different classes i.e. $\textrm{Avg}_{k\neq
k^{\prime}}\\{h_{k}\cdot w_{k}^{\prime}\\}$ for both train and test datasets.
We have illustrated this for miniImageNet in Fig. 2. Clearly, on both the
train and test the similarity between different classes goes on increasing for
the ”Learnable + CE” model. While using the loss in equation 9 (as per Neural
Collapse) have no increasing trend. Incorporating concept factorization and
simplex fine tuning (in MASIL) further reduces the similarities as the session
grows on and hence mitigate the effect of forgetting and confirming the
maximum separability with MASIL.

##  6 Conclusion

In this paper we propose the novel framework MASIL as an step towards learning
the maximum separable classifier in a competitive setting of continual
learning i.e. FSCIL. We propose to induce the simplex from concept
factorization helps in few shot cases. We introduced novel loss function where
the base and novel classes can be learnt together during fine tuning to
further mitigate forgetting and overfitting. In experiments MASIL outperforms
all the benchmarks with sufficient margin on three datasets proving its
efficiency.

## References

  * [1] Afra Feyza Akyürek, Ekin Akyürek, Derry Wijaya, and Jacob Andreas.  Subspace regularizers for few-shot class incremental learning.  arXiv preprint arXiv:2110.07059, 2021.
  * [2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 139–154, 2018.
  * [3] Bradley M Bell and James V Burke.  Algorithmic differentiation of implicit functions and optimal values.  In Advances in Automatic Differentiation, pages 67–77. Springer, 2008.
  * [4] Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares-López, Fabian Pedregosa, and Jean-Philippe Vert.  Efficient and modular implicit differentiation.  arXiv preprint arXiv:2105.15183, 2021.
  * [5] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al.  Distributed optimization and statistical learning via the alternating direction method of multipliers.  Foundations and Trends® in Machine learning, 3(1):1–122, 2011.
  * [6] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari.  End-to-end incremental learning.  In Proceedings of the European conference on computer vision (ECCV), pages 233–248, 2018.
  * [7] Gert Cauwenberghs and Tomaso Poggio.  Incremental and decremental support vector machine learning.  Advances in neural information processing systems, 13, 2000.
  * [8] Hakan Cevikalp and Hasan Saribas.  Deep simplex classifier for maximizing the margin in both euclidean and angular spaces.  arXiv preprint arXiv:2212.11747, 2022.
  * [9] Kuilin Chen and Chi-Guhn Lee.  Incremental few-shot learning via vector quantization in deep embedded space.  In International Conference on Learning Representations, 2020.
  * [10] Ali Cheraghian, Shafin Rahman, Sameera Ramasinghe, Pengfei Fang, Christian Simon, Lars Petersson, and Mehrtash Harandi.  Synthesized feature based few-shot class-incremental learning on a mixture of subspaces.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8661–8670, 2021.
  * [11] Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang.  Metafscil: A meta-learning approach for few-shot class incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14166–14175, 2022.
  * [12] Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho.  Neural collapse in deep linear network: From balanced to imbalanced data.  arXiv preprint arXiv:2301.00437, 2023.
  * [13] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  Continual learning: A comparative study on how to defy forgetting in classification tasks.  arXiv preprint arXiv:1909.08383, 2(6):2, 2019.
  * [14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [16] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa.  Learning without memorizing.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5138–5146, 2019.
  * [17] Songlin Dong, Xiaopeng Hong, Xiaoyu Tao, Xinyuan Chang, Xing Wei, and Yihong Gong.  Few-shot class-incremental learning via relation knowledge distillation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1255–1263, 2021.
  * [18] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle.  Podnet: Pooled outputs distillation for small-tasks incremental learning.  In European Conference on Computer Vision, pages 86–102. Springer, 2020.
  * [19] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre.  Craft: Concept recursive activation factorization for explainability.  arXiv preprint arXiv:2211.10154, 2022.
  * [20] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra.  Pathnet: Evolution channels gradient descent in super neural networks.  arXiv preprint arXiv:1701.08734, 2017.
  * [21] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International conference on machine learning, pages 1126–1135. PMLR, 2017.
  * [22] Robert M French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3(4):128–135, 1999.
  * [23] Spyros Gidaris and Nikos Komodakis.  Dynamic few-shot visual learning without forgetting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4367–4375, 2018.
  * [24] Siavash Golkar, Michael Kagan, and Kyunghyun Cho.  Continual learning via neural pruning.  arXiv preprint arXiv:1903.04476, 2019.
  * [25] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  arXiv preprint arXiv:1312.6211, 2013.
  * [26] Andreas Griewank and Andrea Walther.  Evaluating derivatives: principles and techniques of algorithmic differentiation.  SIAM, 2008.
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [28] Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi.  Constrained few-shot class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9057–9067, 2022.
  * [29] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2(7), 2015.
  * [30] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 831–839, 2019.
  * [31] Bingchen Huang, Zhineng Chen, Peng Zhou, Jiayin Chen, and Zuxuan Wu.  Resolving task confusion in dynamic expansion architectures for class incremental learning.  arXiv preprint arXiv:2212.14284, 2022.
  * [32] Kejun Huang, Nicholas D Sidiropoulos, and Athanasios P Liavas.  A flexible and efficient algorithmic framework for constrained matrix and tensor factorization.  IEEE Transactions on Signal Processing, 64(19):5052–5065, 2016\.
  * [33] Ching-Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen.  Compacting, picking and growing for unforgetting continual learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [34] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Vineeth N Balasubramanian.  Energy-based latent aligner for incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7452–7461, 2022.
  * [35] Mahdi M Kalayeh, Haroon Idrees, and Mubarak Shah.  Nmf-knn: Image annotation using weighted multi-view non-negative matrix factorization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 184–191, 2014.
  * [36] William Karush.  Minima of functions of several variables with inequalities as side conditions.  In Traces and Emergence of Nonlinear Programming, pages 217–245. Springer, 2014.
  * [37] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [38] Steven George Krantz and Harold R Parks.  The implicit function theorem: history, theory, and applications.  Springer Science & Business Media, 2002.
  * [39] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [40] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
  * [41] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [42] Harold W Kuhn and Albert W Tucker.  Nonlinear programming.  In Traces and emergence of nonlinear programming, pages 247–258. Springer, 2014.
  * [43] Diangang Li, Xing Wei, Xiaopeng Hong, and Yihong Gong.  Infrared-visible cross-modal person re-identification with an x modality.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4610–4617, 2020.
  * [44] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  * [45] Zhuoyun Li, Changhong Zhong, Sijia Liu, Ruixuan Wang, and Wei-Shi Zheng.  Preserving earlier knowledge in continual learning with the help of all previous feature extractors.  arXiv preprint arXiv:2104.13614, 2021.
  * [46] Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu.  Negative margin matters: Understanding margin in few-shot classification.  In European conference on computer vision, pages 438–455. Springer, 2020.
  * [47] Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang.  Few-shot class-incremental learning via entropy-regularized data-free replay.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV, pages 146–162. Springer, 2022.
  * [48] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song.  Sphereface: Deep hypersphere embedding for face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212–220, 2017.
  * [49] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [50] Bin Lu, Xiaoying Gan, Lina Yang, Weinan Zhang, Luoyi Fu, and Xinbing Wang.  Geometer: Graph few-shot class-incremental learning via prototype representation.  arXiv preprint arXiv:2205.13954, 2022.
  * [51] Zhiheng Ma, Xing Wei, Xiaopeng Hong, and Yihong Gong.  Bayesian loss for crowd count estimation with point supervision.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6142–6151, 2019.
  * [52] Zicheng Pan, Xiaohan Yu, Miaohua Zhang, and Yongsheng Gao.  Ssfe-net: Self-supervised feature enhancement for ultra-fine-grained few-shot class incremental learning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6275–6284, 2023.
  * [53] Vardan Papyan, XY Han, and David L Donoho.  Prevalence of neural collapse during the terminal phase of deep learning training.  Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.
  * [54] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  Neural Networks, 113:54–71, 2019.
  * [55] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al.  Scikit-learn: Machine learning in python.  the Journal of machine Learning research, 12:2825–2830, 2011.
  * [56] Can Peng, Kun Zhao, Tianren Wang, Meng Li, and Brian C Lovell.  Few-shot class-incremental learning from an open-set perspective.  In European Conference on Computer Vision, pages 382–397. Springer, 2022.
  * [57] F. Pernici, M. Bruni, C. Baecchi, F. Turchini, and A. Del Bimbo.  Class-incremental learning with pre-allocated fixed classifiers.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 6259–6266, Los Alamitos, CA, USA, jan 2021. IEEE Computer Society.
  * [58] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [59] Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard Zemel.  Incremental few-shot learning with attention attractor networks.  Advances in Neural Information Processing Systems, 32, 2019.
  * [60] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [61] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [62] Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu.  Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima.  Advances in Neural Information Processing Systems, 34:6747–6761, 2021.
  * [63] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  Advances in neural information processing systems, 30, 2017.
  * [64] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  Advances in neural information processing systems, 30, 2017.
  * [65] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele.  Meta-transfer learning for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 403–412, 2019.
  * [66] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199–1208, 2018.
  * [67] Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong.  Topology-preserving class-incremental learning.  In European Conference on Computer Vision, pages 254–270. Springer, 2020.
  * [68] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong.  Few-shot class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12183–12192, 2020.
  * [69] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  Advances in neural information processing systems, 29, 2016.
  * [70] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [71] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 374–382, 2019.
  * [72] Ye Xiang, Ying Fu, Pan Ji, and Hua Huang.  Incremental learning using conditional adversarial networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6619–6628, 2019.
  * [73] Shipeng Yan, Jiangwei Xie, and Xuming He.  Der: Dynamically expandable representation for class incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014–3023, 2021.
  * [74] Dongbao Yang, Yu Zhou, Wei Shi, Dayan Wu, and Weiping Wang.  Rd-iod: Two-level residual-distillation-based triple-network for incremental object detection.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 18(1):1–23, 2022.
  * [75] Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao.  Neural collapse inspired feature-classifier alignment for few-shot class incremental learning.  arXiv preprint arXiv:2302.03004, 2023.
  * [76] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz.  Dreaming to distill: Data-free knowledge transfer via deepinversion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715–8724, 2020.
  * [77] Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer.  Semantic drift compensation for class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6982–6991, 2020.
  * [78] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In International Conference on Machine Learning, pages 3987–3995. PMLR, 2017.
  * [79] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen.  Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12203–12213, 2020.
  * [80] Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu.  Few-shot incremental learning with continually evolved classifiers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12455–12464, 2021.
  * [81] Hanbin Zhao, Yongjian Fu, Mintong Kang, Qi Tian, Fei Wu, and Xi Li.  Mgsvf: Multi-grained slow vs. fast framework for few-shot class-incremental learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [82] Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan.  Forward compatible few-shot class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9046–9056, 2022.
  * [83] Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan.  Few-shot class-incremental learning by sampling multi-phase tasks.  arXiv preprint arXiv:2203.17030, 2022.
  * [84] Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu.  Prototype augmentation and self-supervision for incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5871–5880, 2021.
  * [85] Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jun Zha.  Self-promoted prototype refinement for few-shot class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6801–6810, 2021.
  * [86] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha.  Self-sustaining representation expansion for non-exemplar class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9296–9305, 2022.

##  Appendix A Implementation Details

Backbone Architecture: Existing works in FSCIL have leveraged ResNet-18,
ResNet-12, ResNet-20 [27] as the backbone network for feature extractor.
Following ALICE [56], we use ResNet-18 as the backbone network for feature
extractor on top which the two layer MLP for projecting the features as the
classification layer is trained for base and incremental sessions.

Concept Factorization: We used low rank factorization variable $v=64$ for
CIFAR and miniImageNet, and $v=72$ for CUB-200. For $\tau$ function, it
corresponds to randomly choosing 10 cropped patches of size $18\times 18$ on
CIFAR-100 and miniImageNet and patch size of $64\times 64$ for CUB-200. We
didn’t use the scikit-learn implementation [55] of NMF, we leverage the work
of [19, 32], which uses Jax [4, 55] implementation of ADMM[5] using Jaxopt
library. We convert the Jax array to tensor array to be able to combine with
the tensor array obtained from Pytorch on the gradient $\frac{\partial
A}{\partial X}$ and compute the gradient with respect to input images i.e.
$\frac{\partial P}{\partial X},\frac{\partial Q}{\partial X}$.

Dataset | Base Session | Incremental Session
---|---|---
Epcohs |  | Learning
---
Rate
Iterations |  | Learning
---
Rate
CIFAR-100 | 200 | 0.25 | 20-80 | 0.25
miniImageNet | 500 | 0.25 | 120-190 | 0.025
CUB-200 | 80 | 0.025 | 80-150 | 0.05

Table 4: Training Details for Base and Incremental Session

Training Details: Data augmentation strategies like random crop, horizontal
flip, rotation, brightness variation, cutout, resizing, flipping and color
jittering were all applied following the recent works [56, 68, 80].
Additionally, we adopted the standard data pre-processing as in [56]. With a
variation of number of epochs and iterations for base and incremental session
across three datasets we keep the batch size of 512 for base session and 64
for incremental session during simplex finetuning as explained in Section 4.4.
Dataset wise epochs/iterations and learning rates for base and incremental
session is given in Table 4. Additionally, we use SGDR [49] with momentum as
optimizer which uses cosine annealing strategy to reduce learning rate. Our
code will be publicly available upon acceptance.

Methods | Session Accuracy (%) ($\uparrow$) |  Average Acc. ($\uparrow$) |  Relative Improvement
---|---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
iCaRL [58] | 68.68 | 52.65 | 48.61 | 44.16 | 36.62 | 29.52 | 27.83 | 26.26 | 24.01 | 23.89 | 21.16 | 36.67 | +39.08
EEIL [6] | 68.68 | 53.63 | 47.91 | 44.20 | 36.30 | 27.46 | 25.93 | 24.70 | 23.95 | 24.13 | 22.11 | 36.27 | +38.13
NCM [30] | 68.68 | 57.12 | 44.21 | 28.78 | 26.71 | 25.66 | 24.62 | 21.52 | 20.12 | 20.06 | 19.87 | 32.49 | +40.37
Fixed classifier [57] | 68.47 | 51.00 | 45.42 | 40.76 | 35.90 | 33.18 | 27.23 | 24.24 | 21.18 | 17.34 | 16.20 | 34.63 | +44.04
D-NegCosine [46] | 74.96 | 70.57 | 66.62 | 61.32 | 60.09 | 56.06 | 55.03 | 52.78 | 51.50 | 50.08 | 48.47 | 58.86 | +11.77
D-DeepEMD [79] | 75.35 | 70.69 | 66.68 | 62.34 | 59.76 | 56.54 | 54.61 | 52.52 | 50.73 | 49.20 | 47.60 | 58.73 | +12.64
D-Cosine [69] | 75.52 | 70.95 | 66.46 | 61.20 | 60.86 | 56.88 | 55.40 | 53.49 | 51.94 | 50.93 | 49.31 | 59.36 | +10.93
DeepInv [76] | 75.90 | 70.21 | 65.36 | 60.14 | 58.79 | 55.88 | 53.21 | 51.27 | 49.38 | 47.11 | 45.67 | 57.54 | +14.57
TOPIC [68] | 68.68 | 62.49 | 54.81 | 49.99 | 45.25 | 41.40 | 38.35 | 35.36 | 32.22 | 28.31 | 26.28 | 43.92 | +33.96
IDLVQ [9] | 77.37 | 74.72 | 70.28 | 67.13 | 65.34 | 63.52 | 62.10 | 61.54 | 59.04 | 58.68 | 57.81 | 65.23 | +2.43
SPPR [85] | 68.68 | 61.85 | 57.43 | 52.68 | 50.19 | 46.88 | 44.65 | 43.07 | 40.17 | 39.63 | 37.33 | 49.32 | +22.91
[10] | 68.78 | 59.37 | 59.32 | 54.96 | 52.58 | 49.81 | 48.09 | 46.32 | 44.33 | 43.43 | 43.23 | 51.84 | +17.01
CEC [80] | 75.85 | 71.94 | 68.50 | 63.50 | 62.43 | 58.27 | 57.73 | 55.81 | 54.83 | 53.52 | 52.28 | 61.33 | +7.96
LIMIT [83] | 76.32 | 74.18 | 72.68 | 69.19 | 68.79 | 65.64 | 63.57 | 62.69 | 61.47 | 60.44 | 58.45 | 66.67 | +1.79
MgSvF [81] | 72.29 | 70.53 | 67.00 | 64.92 | 62.67 | 61.89 | 59.63 | 59.15 | 57.73 | 55.92 | 54.33 | 62.37 | +5.91
MetaFSCIL [11] | 75.9 | 72.41 | 68.78 | 64.78 | 62.96 | 59.99 | 58.3 | 56.85 | 54.78 | 53.82 | 52.64 | 61.93 | +7.6
FACT [82] | 75.90 | 73.23 | 70.84 | 66.13 | 65.56 | 62.15 | 61.74 | 59.83 | 58.41 | 57.89 | 56.94 | 64.42 | +3.3
Data-free replay[47] | 75.90 | 72.14 | 68.64 | 63.76 | 62.58 | 59.11 | 57.82 | 55.89 | 54.92 | 53.58 | 52.39 | 61.52 | +7.85
ALICE [56] | 77.40 | 72.70 | 70.60 | 67.20 | 65.90 | 63.40 | 62.90 | 61.90 | 60.50 | 60.60 | 60.10 | 65.75 | +0.14
SSFE-Net [52] | 76.38 | 72.11 | 68.82 | 64.77 | 63.59 | 60.56 | 59.84 | 58.93 | 57.33 | 56.23 | 54.28 | 62.98 | +5.96
NC-FSCIL [75] | 80.45 | 75.98 | 72.30 | 70.28 | 68.17 | 65.16 | 64.43 | 63.25 | 60.66 | 60.01 | 59.44 | 67.28 | +0.8
MASIL(Ours) | 80.50 | 76.02 | 72.25 | 70.30 | 68.85 | 65.72 | 64.45 | 63.28 | 60.80 | 60.60 | 60.24 | 67.54 |
Table 5: Performance comparison on CUB-200 with ResNet-18 as backbone
architecture under 10-way 5-shot FSCIL setting. Table denotes the accuracy in
each session, average accuracy across sessions and ”Relative Improvement”
denotes the improvement of our method in the last session. Methods above
separating line are CIL methods for FSCIL as in [68] and [80]

##  Appendix B Additional Results

We continued summarising results for CUB-200 comparing various methods with
MASIL. Although improvement in last session is very small +0.14% as compared
to strongest baseline ALICE [56]. But we are consistently better in average
accuracy and session wise accuracy (except only two sessions 2 and 8). On
average accuracy we outperform ALICE [56] by +1.79% as shown in the Table 5.
To further analyze the underlying reason for performance improvement because
of fine tuning of classifier weights obtained from concept basis, we
calculated the average cosine similarity of the concept basis $c_{i}$ with all
$c_{j}$, where $j\neq i$ for all the three datasets as given in the Table 6.
Formally it is calculated as:

$\frac{1}{K(K-1)}\sum_{i=1}^{K}\sum_{j=1,j\neq i}^{K}c_{i}\cdot c_{j}$ (15)

Dataset | Average Cosine Similarity
---|---
miniImageNet | -5.22e-4
CIFAR-100 | -8.78e-3
CUB-200 | -4.54e-4
Table 6: Calculated cosine similarity among concept basis for each of the
three benchmark datasets

These entries are almost close to zeros resulting in the concept basis which
are non-overlapping and non-repetitive and hence can induce the unique
classifier weights correspond to novel classes, that can be represented in
terms of their combination.