# PointCLIMB: An Exemplar-Free Point Cloud Class Incremental Benchmark

${\text{Shivanand Kundargi}}^{*}$, ${\text{Tejas Anvekar}}^{*}$, Ramesh Ashok
Tabib, Uma Mudenagudi
Center of Excellence in Visual Intelligence (CEVI), KLE Technological
University.
Vidya Nagar, Hubballi, Karnakata, India.
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Point clouds offer comprehensive and precise data regarding the contour and
configuration of objects. Employing such geometric and topological 3D
information of objects in class incremental learning can aid endless
application in 3D-computer vision. Well known 3D-point cloud class incremental
learning methods for addressing catastrophic forgetting generally entail the
usage of previously encountered data, which can present difficulties in
situations where there are restrictions on memory or when there are concerns
about the legality of the data. Towards this we pioneer to leverage exemplar
free class incremental learning on Point Clouds. In this paper we propose
PointCLIMB: An exemplar Free Class Incremental Learning Benchmark. We focus on
a pragmatic perspective to consider novel classes for class incremental
learning on 3D point clouds. We setup a benchmark for 3D Exemplar free class
incremental learning. We investigate performance of various backbones on
3D-Exemplar Free Class Incremental Learning framework. We demonstrate our
results on ModelNet40 dataset.

.

##  1 Introduction

Point cloud analysis with pioneering works exploring global [16] and local
[25, 17, 12, 3] geometries has become an increasingly popular approach for
understanding 3D objects and environments, with many potential
applications[24] in real-time settings. Considering the real-time applications
of point cloud analysis, there is always a new incoming stream of data
available to the learner. Comparing this realistic scenario to human
cognition, humans leverage their existing knowledge and build upon it when
learning new things, rather than starting from scratch every time. In real-
world, for the tasks such as autonomous driving and robotic applications, the
data is not static but rather accumulates over time. Therefore, mimicking the
human cognition there is a dire need for models built upon such real world
applications to learn incrementally as new data is added, rather than
retraining the entire model from scratch each time. This approach is known as
class incremental learning(a subset of continual or life long learning)[13],
which allows the model to adapt to changes in the data distribution over time,
while retaining knowledge learned from previous data[15].

Class-incremental learning has been investigated to a certain extent in
2D(image data) realm[19, 10, 20], its exploration on 3D data(point cloud data)
has been relatively limited. There have been initial works towards mitigating
catastrophic forgetting in point cloud class incremental learning 3D-FSCIL[4],
I3DOL[6]. these methods have an unfortunate lacunae: They require extensive
memory bank for replay of point cloud data from previous task. The problem of
memory constraints in class-incremental learning is a matter of great concern
in computer vision for two primary reasons. Firstly, numerous applications
with point cloud analysis operate on devices with limited memory capacity,
rendering the issue of memory consumption becomes a crucial consideration [11,
21]. Secondly, many 3D-computer vision applications acquire data that is
subject to legal restrictions[28, 5, 1], making storage a difficult and often
infeasible task. These caveats lead us to ask How can 3D-computer vision
systems incrementally incorporate new geometric information without storing
previous data.

Towards modeling 3D-EFCIL(3D Exemplar Free Class incremental learning), we
propose PointCLIMB: An Exemplar-Free Point Cloud Class Incremental Benchmark.
We summarize our contributions as follows

  * •

We are the first to model 3D-EFCIL and benchmark the results on Modelnet40
dataset[26].

  * •

We are the first to investigate the importance of backbone(feature extractor)
in 3D-EFCIL.

  * •

We propose to employ a weighted knowledge distillation loss towards mitigating
catastrophic forgetting.

  * •

We propose to model a pragmatic approach to benchmark 3D-EFCIL, contrast to
other works in point cloud class-incremental learning which never focus on
pragmatic approach considering novel data arrival.

Figure 1: The process of network optimization in PointCLIMB can be illustrated
through two tasks. The first, base task involves training a teacher model
using a feature extractor $f$ with parameters $\phi_{O}$ and a linear
classifier $g$ with parameters $V_{O}$. The operation $\mathcal{A}$, which can
be mean, max or sum, is applied to make the output symmetric. In the second,
class incremental novel task N; a student model is introduced when a novel
task N arrives. The weights of the student model are initialized by copying
the weights of the teacher model. Specifically, $g(;V_{S})=g(;V_{O})\cup
g(;V_{\zeta})$, where $g(;V_{\zeta})$ represents the weights associated with
the novel class. The teacher model is kept frozen during this process. To
mitigate the issue of catastrophic forgetting, we use census knowledge
distillation loss, which compares the logits of the teacher and student
models.

##  2 Related Works

Point Cloud Analysis With the recent emergence of Lidar sensors, numerous
studies have been conducted on directly classifying 3D point cloud objects.
voxel and multiview based methods [22, 27] were the works carried out
initially towards point cloud analysis. PointNet[16], was the first to employ
multi-layer perceptron (MLP) networks to interpret 3D point clouds. Addressing
the limitation of considering only global topological information,
PointNet++[17] was proposed which considers the deep hierarchical features for
pointcloud processing. DGCNN[25] proposed Edge-Conv a graph based dynamic
convolution for exploiting local geometry of a point cloud. PointMLP[12], a
purely residual MLP network that achieves high performance without the need
for complex local geometric extractors. Despite its simplicity, PointMLP
performs remarkably well and is highly competitive with more sophisticated
models. These local and global topology aware methodologies never explore
learning similarities between region of point cloud.Towards addressing the
aforementioned challenge Point Relation-Aware Network (PRA-Net)[3] was
proposed, which is composed of an Intra-region Structure Learning (ISL) module
and an Inter-region Relation Learning (IRL) module. The ISL module has the
ability to dynamically integrate local structural information into point
features, while the IRL module can effectively capture inter-region
interactions using a differentiable region division method and a
representative point-based technique, which is both adaptable and efficient.
Hence methods that explore both inter-region Relation and Intra-region
Structure Learning which extract superior geometric features might strengthen
the backbone for Class Incremental Learning on point clouds

Incremental Learning Task Incremental, Domain Incremental and Class
Incremental are the three main categories in continual learning. The issue of
catastrophic forgetting has been widely recognized for many years, with
evidence dating back to the 1980s when [14] demonstrated that algorithms
trained with backpropagation were prone to this phenomenon. Subsequent
research by [18] corroborated these findings and extended them to a broader
range of tasks trained using backpropagation. A comprehensive review of early
attempts to address catastrophic forgetting is provided by [7].There have been
many subsequent works in 2D continual learning whereas in 3D realm, the
continual learning framework has been underexplored. 3D-FSCIL[4] explored the
few shot aspect of class incremental learning on point clouds. They propose
microshapes to mitigate catastrophic forgetting. The existing 3D class-
incremental learning methods store exemplars or microshapes data securities
and memory.I3DOL[6] proposes to mitigate the issue of catastrophic forgetting
that can result from the presence of redundant geometric information,towards
this an attention mechanism that is sensitive to geometric properties has been
introduced. This mechanism quantifies the significance of local geometric
structures and identifies distinctive 3D geometric attributes that make
substantial contributions to incremental learning of classes.Though
aforementioned 3D Class-Incremental learning methods are well established they
make extensive replay of previous data for training incrementally.

Exemplar-Free Class-Incremental learning According to recent reviews of class
incremental learning (CIL), the majority of methods aimed at mitigating
catastrophic forgetting incorporate techniques that involve replaying samples
from past classes[2].which have data privacy restrictions or storage
limitations[23]. Towards addressing the aforementioned challange Learning
Without Forgetting[10] was proposed as knowledge distillation method for
class-incremental learning set-up. [8] weight and data regularisation based
methods were proposed to mitigate catastrophic forgetting without storing
exemplars.

##  3 PointCLIMB

In this section, we present a practical scenario-based assessment of the
3D-EFCIL model, dubbed as PointCLIMB. Additionally, we propose robust
benchmark models, conduct comprehensive experiments, and elucidate the
rationale behind the selection of backbone architectures.

Exemplar free class incremental learning (3D-EFCIL) is quintessential yet an
under-explored paradigm in the realm of point cloud analysis. To facilitate
benchmarking in realistic scenarios for 3D-EFCIL, this paper introduces
PointCLIMB: An Exemplar Free Point Cloud CLass IncreMental Benchmark.
PointCLIMB investigates the need for point cloud backbones that extracts
superior geometric features capable of mitigating the challenge of
catastrophic forgetting in realistic 3D-EFCIL settings. We propose to employ
Census; a weighted knowledge distillation loss between old and new class
logits of point cloud backbones. Census is dynamic to incremental classes that
enhances the performance of point cloud backbones in 3D-EFCIL. We employ
veristic task sampler which mimicks the natural way of sampling the tasks to
be learnt incrementally in 3D-EFCIL setting.

Incremental learning problem $\mathcal{T}$ consists of sequence of $m$ tasks:

$\mathcal{T}=[(C^{1},D^{1}),(C^{2},D^{2})\text{, ... ,}(C^{m},D^{m})]$ (1)

###  3.1 PointCLIMB Settings

1

2

3

4

Input : tc, low, and high.

/* tc is total number of classes, low represents least no of categories
present in a task, and high represents maximum no of categories present in a
task. */

Output : The lists $\mathcal{T}$.

// $\mathcal{T}$ is task list for continual learning.

5

6classes $\leftarrow$ arrange(0,tc-1); shuffle(classes)

7

8TL $\leftarrow$ []; $\mathcal{T}\leftarrow$ []

9 condition $\leftarrow 0$

10while  _tc $\neq$ 0 and condition $\geq$ 0_ do

11 base $\leftarrow$ RANDOMINT(low, high)

12 condition $:=$ tc\- base

13 if  _condition $\leq$ 0_ then

14 TL.append (tc)

15 break

16

17 else

18 TL.append (base)

19

20 tc $:=$ condition

21

22

23s $\leftarrow$ 0

24 for  _i $\leftarrow$ 0 to length(TL)_ do

25 $\mathcal{T}$.append $\Big{(}$classes $\big{[}$TL [s]:TL [s]+TL
[i]$\big{]}$$\Big{)}$

26 s $:=$ TL [i]

27

28

29return _ $\mathcal{T}$_;

Algorithm 1 Veristic Task Sampler: A pragmatic sampler mimicking the paradigm
of class incremental learning

The veristic task sampler is used to model $\mathcal{T}$ by selecting tasks
based on a naturalistic setting that imitates the paradigm of novel data
arrival in Class Incremental learning best explained in Algorithm 1. Where
each task $t$ is represented by a set of classes
$C^{t}=\\{c^{t}_{1},c^{t}_{2}\text{, ... ,}c^{t}_{m^{t}}\\}$ and the training
data $D^{t}$. We use $M^{t}$ to represent the total number of classes in all
tasks up to and including task $t:M^{t}=\sum_{i=1}^{t}C^{i}$. The 3D class-
incremental problem in which $D^{t}=\\{(p_{1},y_{1}),(p_{2},y_{2})\text{, ...
,}(p_{l^{t}},y_{l^{t}})\\}$, where $p$ is point cloud with $n$ points such
that $p\in\mathbb{R}^{n\times 3}$. During training for task $t$, the learner
only has access to $C^{t},D^{t}$ where as during inference the evaluation is
done for the union of all previous tasks $\bigcup_{i=1}^{t}C^{i},D^{i}$. For
instance if we encounter task $t=2$, the learner has access to $(C^{2},D^{2})$
where as evaluation is done for $\\{(C^{1},D^{1}),(C^{2},D^{2})\\}$. Towards
modeling 3D-EFCIL problem setting; 1: we don not allow class overlaps between
tasks (i.e, $C^{i}\cap C^{j}=$Øif $i\neq j$), 2: we do not maintain any
coreset(exemplars) of previous task $t-1$ for training the current task $t$.

We consider incremental learners, the teacher model $O(p,\theta_{O})$
parameterized by weights $\theta_{O}$ and the student model $S(p,\theta_{S})$
parameterized by weights $\theta_{S}$ to indicate the output logits of the
network on input $p$. We further split the neural network in a feature
extractor $f$ with weights $\phi$ and linear classifier $g$ with weights $V$
according to $O(p,\theta_{O})=g\big{(}f(p;\phi_{O});V_{O}\big{)}$ and
$S(p,\theta_{S})=g\big{(}f(p;\phi_{O});V_{S}\big{)}$. The student model’s
weights are designed such that $V_{S}=V_{O}\cup V_{\zeta}$ where $V_{\zeta}$
is novel task specific parameters as shown in Figure 1. We use
$({\hat{y}_{O};\tau})=\sigma(O(p;\theta_{O}),\tau)$ and
$({\hat{y}_{S};\tau})=\sigma(S(p;\theta_{S}),\tau)$ to identify teacher and
student network predictions, where $\sigma(,\tau)$ indicates the softmax
functions with temperature $\tau$.

###  3.2 Influence of Backbones on PointCLIMB

To investigate the importance of superior and robust feature extraction in
PointCLIMB, we train point cloud class-incremental classifiers as per
PointCLIMB Settings. For our case study we select three types of open-source
state-of-the-art backbones.

  * •

Global per-point based PointNet[16].

  * •

Local neighbourhood (Intra-region) based PoinetNet++[17], DGCNN[25],
PointMLP[12].

  * •

Intra-region structure aware and Inter-region relation aware PRA-Net[3].

Backbones act as a feature extractor whose features are used for knowledge
distillation in exemplar free class-incremental learning Methods. None of the
previous works on 3D point cloud class incremental learning focuses on the
importance of feature extraction. We investigate the significance of feature
extractor by evaluating different state of the art classification networks as
feature extractor on PointCLIMB settings. Our findings infer that, backbones
that incorporate both Intra-region structure and Inter-region relation aware
properties perform significantly better in PointCLIMB settings compared to
other types as depicted in Table 1. The reason for this may be the extracted
features of PRA-Net[3] best describe the topology, similarity, proximity, and
symmetry of a point cloud when compared with graph based semantic features of
DGCNN. Another reason is due to the incorporation of gated units in IRL blocks
of PRA-Net which resembles highway-connection classifier networks (HCNs)[9].
We conclude that networks with gated unit incorporates high stability as
explained by [9] in 2D realm.

###  3.3 Knowledge Distillation

The process of Knowledge Distillation in 3D-EFCIL involves the transfer of
geometric and topological knowledge from a previously trained model (known as
the teacher model: $O(p,\theta_{O})$) to a new model (referred to as the
student model: $S(p,\theta_{S})$). This transfer of knowledge aims to enable
the student model to accurately classify new point cloud categories, while
still retaining its ability to classify the categories it was previously
trained on.

Learning Without Forgetting (LwF) enables the student model to learn new tasks
without forgetting the knowledge it has already acquired, by leveraging the
distillation of knowledge from the teacher model and can be expressed as:

$\mathcal{L}_{LwF}=\lambda\mathcal{L}_{distill}+\mathcal{L}_{class}$ (2)

where $\mathcal{L}_{distill}$ is the distillation loss term, which measures
the difference between the output probabilities of the teacher model and the
student model, and is given by:

$\mathcal{L}_{distill}=-\frac{1}{N}\sum_{i=1}^{n}(\hat{y}^{i}_{O};\tau)\log{\big{(}(\hat{y}^{i}_{S};\tau)\big{)}}$
(3)

Here, $\hat{y}^{i}_{O}$ represents the output probability of the teacher model
for the i-th input sample, and $\hat{y}^{i}_{S}$ represents the output
probability of the student model for the same input sample with temperature
$\tau$. The second term, $\mathcal{L}_{class}$, is the classification loss
term, which measures the deviation of the student model’s output probabilities
from the true labels of the current task, and is given by:

$\mathcal{L}_{class}=-\frac{1}{N}\sum_{i=1}^{n}y^{i}\log{\hat{y}^{i}_{S}}$ (4)

Here, $y^{i}$ represents the true label of the i-th input sample. Finally,
$\lambda$ is a hyperparameter that balances the relative importance of the
distillation and classification loss terms.

Census Knowledge Distillation is an improved variant of LwF that dynamically
adjusts the weight of the distillation loss term for every new increment of
task $t$, based on the current count of classes in task $\eta$ and the number
of tasks elapsed $T$. This dynamic weight helps to ensure robustness towards
catastrophic forgetting and can be expressed as

$\mathcal{L}_{census}=(\eta*T)\mathcal{L}_{distill}$ (5)

where $\mathcal{L}_{distill}$ is briefed in Eq. 3 Allocating dynamic weights
to the knowledge distillation loss based on the arrival of new tasks enhances
the importance of geometric features associated with the newly arrived
classes. This approach can help mitigate the issue of task recency bias,
ultimately leading to improved performance. To investigate the significance of
employed Census knowledge distilation loss, we train PointNet [16], PointNet++
[17], DGCNN [25], PointMLP [12], and PRA-Net [3] on PointCLIMB settings and
compare with LwF [10] as depicted in Table 1.

##  4 Experiments

Training pipeline of PointCLIMB is based on a fine-tuning strategy, which
involves two steps. Firstly, a base task is trained on a backbone architecture
using the cross-entropy loss given by Eq. 4. We refer to this model as the
teacher model.Subsequently, when an incremental task containing a novel class
is presented, we compute a weighted knowledge distillation loss between the
logits of the weight-frozen teacher model and the weight-shared student model,
as shown in Eq. 5. This loss is combined with the cross-entropy loss to train
the novel set of classes. In other words, the student model learns from the
teacher model while retaining the knowledge of the previous tasks, by jointly
minimizing the cross-entropy loss and the weighted knowledge distillation
loss. This approach ensures that the student model adapts to the new task
while preventing catastrophic forgetting of the previously learned tasks.

Table 1: Performance of different backbone’s with PointCLIMB on ModelNet40
dataset. Joint depicts Upper bound and FT represents Fine-Tuning approach. We
demonstrate supremacy of census over other knowledge distillation loss for
each backbone on 3D-EFCIL. We represent our findings by 1) best by bold
underline and 2) second best by bold accuracy values.

Methods | Loss | 20 | 5 | 5 | 5 | 5
---|---|---|---|---|---|---
| Joint | 94.77 | 92.11 | 86.98 | 85.43 | 84.67
| FT | 94.10 | 02.65 | 01.61 | 00.31 | 00.24
| LwF |  95.25 | 53.37 | 04.92 | 02.62 | 00.32
PointNet [16] | Census | 94.01 |  67.77 |  49.25 |  26.95 |  18.47
| Joint | 93.17 | 92.76 | 87.34 | 85.29 | 85.32
| FT | 92.33 | 01.23 | 01.28 | 00.58 | 00.24
| LwF | 92.51 | 61.34 | 12.63 | 07.32 | 01.86
PointNet++ [17] | Census |  93.39 |  71.24 |  56.26 |  44.09 |  29.09
| Joint | 95.44 | 94.62 | 88.13 | 86.15 | 85.87
| FT | 94.67 | 01.88 | 01.23 | 00.63 | 00.68
| LwF |  95.50 | 61.21 | 06.74 | 01.99 | 00.93
PointMLP [12] | Census | 95.25 |  64.39 |  52.30 |  27.49 |  28.68
| Joint | 96.52 | 93.93 | 88.49 | 87.96 | 86.69
| FT | 96.42 | 05.47 | 01.40 | 00.40 | 00.60
| LwF | 96.38 | 47.98 | 11.67 | 00.36 | 00.76
DGCNN [25] | Census |  96.58 |  72.37 |  52.56 |  36.27 |  27.19
| Joint | 96.84 | 95.41 | 90.94 | 88.01 | 87.23
| FT | 96.67 | 01.16 | 00.35 | 00.40 | 00.32
| LwF | 96.67 | 52.14 | 04.17 | 00.40 | 00.89
PRA-Net [3] | Census |  96.92 |  72.31 |  59.72 |  47.61 |  35.73

Table 2: Performance of different backbones on ModelNet40 considering one of
the scenarios with 7 tasks modeled by PointCLIMB. We represent our findings by
1) best by bold underline and 2) second best by bold accuracy values.

Methods | 10 | 5 | 5 | 5 | 5 | 5 | 5
---|---|---|---|---|---|---|---
Pointnet [16] | 96.11 $\pm$ 1.29 | 40.88 $\pm$ 9.36 | 20.39 $\pm$ 5.63 | 10.26 $\pm$ 2.29 | 4.56 $\pm$ 1.19 | 5.59 $\pm$ 2.29 | 4.47 $\pm$ 1.62
Pointnet++ [17] | 95.84 $\pm$ 2.17 | 47.20 $\pm$ 8.86 | 22.61 $\pm$ 3.49 | 13.09 $\pm$ 5.74 | 6.79 $\pm$ 1.33 | 4.82 $\pm$ 1.67 | 6.73 $\pm$ 4.80
DGCNN [25] |  96.78 $\pm$ 0.73 | 54.80 $\pm$ 13.58 | 11.60 $\pm$ 4.42 | 5.59 $\pm$ 0.97 | 6.22 $\pm$ 0.64 | 4.42 $\pm$ 1.17 | 3.92 $\pm$ 2.25
PointMLP [12] | 95.04 $\pm$ 0.59 | 44.40 $\pm$ 4.81 | 13.36 $\pm$ 3.68 | 5.72 $\pm$ 3.33 | 4.64 $\pm$ 0.72 | 5.39 $\pm$ 0.80 | 3.60 $\pm$ 0.27
Ours (PRA-Net [3] \+ Census) | 97.45 $\pm$ 1.86 |  69.49 $\pm$ 3.99 |  49.92 $\pm$ 6.74 |  36.19 $\pm$ 5.19 |  33.47 $\pm$ 2.28 |  24.04 $\pm$ 3.89 |  21.96 $\pm$ 2.8

Table 3: Performance of different backbones on ModelNet40 considering one of
the scenarios with uniform tasks modeled by PointCLIMB. We represent our
findings by 1) best by bold underline and 2) second best by bold accuracy
values.

Methods | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4
---|---|---|---|---|---|---|---|---|---|---
Pointnet [16] | 96.87 $\pm$ 1.96 | 32.81 $\pm$ 10.55 | 17.18 $\pm$ 8.80 | 9.25 $\pm$ 2.65 | 3.45 $\pm$ 1.28 | 2.18 $\pm$ 0.09 | 3.34 $\pm$ 1.18 | 4.36 $\pm$ 2.29 | 4.01 $\pm$ 1.95 | 3.52 $\pm$ 1.07
Pointnet++ [17] | 97.18 $\pm$ 1.54 | 32.81 $\pm$ 13 | 21.45 $\pm$ 4.47 | 8.33 $\pm$ 4.40 | 5.80 $\pm$ 2.86 | 5.33 $\pm$ 0.99 | 3.09 $\pm$ 0.54 | 4.61 $\pm$ 0.46 | 2.36 $\pm$ 1.14 | 3.76 $\pm$ 1.71
DGCNN [25] |  97.81 $\pm$ 2.15 | 50.12 $\pm$ 12.29 | 20.88 $\pm$ 5.88 | 7.66 $\pm$ 4.28 | 5.06 $\pm$ 2.2 | 2.46 $\pm$ 0.14 | 2.84 $\pm$ 1.09 | 4.46 $\pm$ 0.37 | 3.24 $\pm$ 1.13 | 2.13 $\pm$ 0.42
PointMLP [12] | 96.875 $\pm$ 0.45 | 30.68 $\pm$ 7.69 | 16.66 $\pm$ 6.80 | 9.5 $\pm$ 3.47 | 7.07 $\pm$ 1.19 | 5.65 $\pm$ 1.11 | 2.30 $\pm$ 1.27 | 4.2 $\pm$ 1.15 | 2.11 $\pm$ 0.87 | 4.93 $\pm$ 2.33
Ours (PRA-Net [3] \+ Census) | 97.12 $\pm$ 1.22 |  56.76 $\pm$ 11.47 |  31.4 $\pm$ 5.88 |  25.70 $\pm$ 5.26 |  20.97 $\pm$ 3.88 |  19.53 $\pm$ 5.53 |  16.85 $\pm$ 2.27 |  14.55 $\pm$ 3.73 |  10.34 $\pm$ 3.98 |  8.18 $\pm$ 2.41

Implementation Details To train PointCLIMB, we used 1024 points and trained
each task for 40 epochs using the Adam optimizer with a learning rate of
0.0001 and a batch size of 32. To dynamically vary the logits of the
classification layer based on the number of arrived novel classes, we referred
to the 3D-FSCIL[4] approach. We also utilized their class and label mapper for
arrived novel data and modified the incremental data loader to make it
exemplar-free. Our implementation of PointCLIMB was based on the PyTorch
framework and ran on an NVIDIA Quadro GV100 32GB.

Evaluation of PointCLIMB, we used the ModelNet40 dataset and assessed the
performance of different backbones. Specifically, we considered a scenario
where the base task consists of 20 random classes, and each novel task
contains 5 randomly chosen classes for Class-Incremental learning as depicted
in Table 1. To quantify the performance of different backbones, we compared
naive fine-tuning, LwF [10], and Census. We evaluated PointCLIMB on different
scenarios modeled by a veristic task sampler, which involved three scenarios.
The first scenario had 20 random classes in the base task and 5 random classes
in each novel task, with a total of 5 tasks. The results of this scenario are
reported in Table 4. The second scenario involved 10 random classes in the
base task and 5 random classes in each novel task, with a total of 7 tasks.
The results of this scenario are also reported in Table 4. Lastly, we
considered a scenario consisting of tasks with all uniform classes, where 4
classes were encountered in both the base and novel tasks, with a total of 10
tasks. The results of this scenario are reported in Table 4. The endless
scenarios representing the actual paradigm of Class Incremental learning can
be modeled using PointCLIMB. We benchmarked three possible scenarios of
PointCLIMB for benchmarking, demonstrating the flexibility and effectiveness
of our approach.

Results and Discussion As shown in Table 1, we provide upper (Joint) and lower
bounds (Fine Tuning) on different backbone architectures. To evaluate the
performance of LWF[10] with different backbones, we report task-wise
accuracies. Moreover, we introduce a novel variant of LWF loss, known as
Census knowledge distillation loss, and observe a remarkable improvement in
performance compared to the traditional LWF loss. Among all the backbone
architectures, PRA-Net with Census Knowledge distillation loss outperforms
others due to its capability to extract superior geometric features and use
gated units to maintain stability. Thus, for modeling pragmatic scenarios, we
recommend using PRA-Net and Census Knowledge Distillation 5 as the superior
baseline for 3D-EFCIL towards future research.

To evaluate the performance of the pragmatic scenarios modeled by the veristic
task sampler described in Algorithm 1 on different backbones using LWF loss
versus the suggested approach (PRA-Net + Census Knowledge Distillation), we
conduct extensive ablation experiments with 5, 7, and 10 tasks in Tables 4, 2,
and 3, respectively. We run 3 experiments with different random seeds to
assess the true response of backbones towards realistic settings sampled by
PointCLIMB, and report the mean and standard deviation of all the backbones
for better comprehensibility. Our results demonstrate that the suggested
baseline outperforms all other backbones with all possible pragmatic settings.
Apart from our recommended baseline, PointNet++ [17] performs the second-best
in most scenarios and various incremental tasks. Moreover, we observe that all
other architectures do not produce stable outcomes, unlike our proposed
baseline that remains stable throughout the scenarios and incremental tasks.

Table 4: Performance of different backbones on ModelNet40 considering one of
the scenarios with 5 tasks modeled by PointCLIMB. We represent our findings by
1) best by bold underline and 2) second best by bold accuracy values.

Methods | 20 | 5 | 5 | 5 | 5
---|---|---|---|---|---
Pointnet [16] | 95.25 $\pm$ 2.47 | 53.37 $\pm$ 5.56 | 8.92 $\pm$ 2.50 | 5.31 $\pm$ 1.70 | 6.31 $\pm$ 1.24
Pointnet++ [17] | 96.83 $\pm$ 0.86 | 63.62 $\pm$ 8.56 | 6.49 $\pm$ 2.05 | 7.94 $\pm$ 1.58 | 4.39 $\pm$ 2.75
DGCNN [25] | 96.33 $\pm$ 1.47 | 58.75 $\pm$ 10.78 | 11.67 $\pm$ 5.73 | 7.48 $\pm$ 2.69 | 3.25 $\pm$ 0.96
PointMLP [12] | 95.50 $\pm$ 2.27 | 61.21 $\pm$ 8.62 | 6.74 $\pm$ 1.56 | 3.99 $\pm$ 0.27 | 2.93 $\pm$ 0.19
Ours (PRA-Net [3] \+ Census) |  96.92 $\pm$ 0.74 |  67.30 $\pm$ 5.32 |  55.17 $\pm$ 7.51 |  45.12 $\pm$ 5.61 |  30.18 $\pm$ 7.31

###  4.1 Limitations

Despite its potential, 3D exemplar-free class incremental learning has some
limitations that must be considered. One of the main challenges is the need
for large amounts of high-quality data to train and evaluate the model. This
can be particularly difficult in 3D environments, where obtaining and
processing data can be time-consuming and expensive. Current backbone that we
test are not robust to noise so this is major limitation. Additionally, 3D
exemplar-free class incremental learning may face limitations when dealing
with complex and highly variable objects, where there may be significant
intra-class, inter-class variation with-in and among incremental tasks. This
can lead to difficulty in accurately classifying these objects and may require
more specialized models or additional training data.

Although there are some limitations to 3D exemplar-free class incremental
learning, we are confident that our study provides valuable insights within
point cloud environments. We are optimistic that our research will inspire
further investigation into these limitations, ultimately leading to the
development of more resilient and efficient methods for 3D exemplar-free class
incremental learning.

##  5 Conclusions

In this paper, we pioneered into the uncharted territory of exemplar-free
class incremental learning on point clouds (3D-EFCIL) and presented a
pragmatic experimental setup that mirrors real-world continual learning
scenarios. Our investigation into various Point cloud classification backbones
yielded encouraging results on the ModelNet40 dataset for 3D exemplar-free
class incremental learning. Our analysis revealed that backbones with intra-
inter local neighbourhood relation awareness significantly outperformed global
topology-based and local neighbourhood-based methods on 3D-EFCIL. Furthermore,
we explored how the use of weighted knowledge distillation loss (census) could
alleviate the issue of catastrophic forgetting in 3D-EFCIL. We anticipate that
our work will inspire further research in this field, leading to the emergence
of more effective and robust methods for few-shot learning in point clouds.

##  6 Broader Impact

The main goal of this research is to establish realistic benchmarks for 3D
exemplar-free class incremental learning, and explore the optimal feature
extractors / backbone, and network design choices for knowledge distillation
loss in this context.

3D exemplar-free incremental learning has the potential to generate
significant and far-reaching impacts in various fields. By enabling machines
to learn new object categories without prior examples, exemplar-free
incremental learning can facilitate real-time learning and adaptation in
dynamic scenarios, making it ideal for robotics, computer vision, and
artificial intelligence. This research can improve the efficiency of machine
learning algorithms by reducing computational costs and enable autonomous
vehicles, robots to operate more safely and efficiently. Additionally,
exemplar-free incremental learning can enhance quality control in
manufacturing processes by detecting defects and ensuring product
specifications are met. Overall, this research has the potential to transform
several industries, paving the way for innovative solutions and improving
efficiency, accuracy, and safety.

## References

  * [1] Brett K Beaulieu-Jones, Zhiwei Steven Wu, Chris Williams, Ran Lee, Sanjeev P Bhavnani, James Brian Byrd, and Casey S Greene.  Privacy-preserving generative deep neural networks support clinical data sharing.  Circulation: Cardiovascular Quality and Outcomes, 12(7):e005122, 2019.
  * [2] Francisco M. Castro, Manuel J. Marin-Jimenez, Nicolas Guil, Cordelia Schmid, and Karteek Alahari.  End-to-end incremental learning.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [3] Silin Cheng, Xiwu Chen, Xinwei He, Zhe Liu, and Xiang Bai.  Pra-net: Point relation-aware network for 3d point cloud analysis.  IEEE Transactions on Image Processing, 30:4436–4448, 2021.
  * [4] Townim Chowdhury, Ali Cheraghian, Sameera Ramasinghe, Sahar Ahmadi, Morteza Saberi, and Shafin Rahman.  Few-shot class-incremental learning for 3d point cloud objects.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XX, pages 204–220. Springer, 2022.
  * [5] Anupam Das, Martin Degeling, Xiaoyou Wang, Junjue Wang, Norman Sadeh, and Mahadev Satyanarayanan.  Assisting users in a world full of cameras: A privacy-aware infrastructure for computer vision applications.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1387–1396. IEEE, 2017.
  * [6] Jiahua Dong, Yang Cong, Gan Sun, Bingtao Ma, and Lichen Wang.  I3dol: Incremental 3d object learning without catastrophic forgetting.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6066–6074, 2021.
  * [7] Robert M French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3(4):128–135, 1999.
  * [8] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [9] Nicholas I-Hsien Kuo, Mehrtash Harandi, Nicolas Fourrier, Christian Walder, Gabriela Ferraro, and Hanna Suominen.  Plastic and stable gated classifiers for continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3553–3558, June 2021.
  * [10] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  * [11] Bo Liu, Xuesu Xiao, and Peter Stone.  A lifelong learning approach to mobile robot navigation.  IEEE Robotics and Automation Letters, 6(2):1090–1096, 2021.
  * [12] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu.  Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework.  In International Conference on Learning Representations, 2022.
  * [13] Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer.  Class-incremental learning: survey and performance evaluation on image classification.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [14] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [15] Benedikt Pfülb, Alexander Gepperth, Saad Abdullah, and André Kilian.  Catastrophic forgetting: still a problem for dnns.  In Artificial Neural Networks and Machine Learning–ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part I 27, pages 487–497. Springer, 2018.
  * [16] Charles Ruizhongtai Qi, Hao Su, Mo Kaichun, and Leonidas J. Guibas.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 77–85, 2017.
  * [17] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [18] Roger Ratcliff.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.  Psychological review, 97(2):285, 1990.
  * [19] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [20] James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira.  Always be dreaming: A new approach for data-free class-incremental learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9374–9384, 2021.
  * [21] Prathap Soma, Ravi Kumar Jatoth, and Hathiram Nenavath.  Fast and memory efficient de-hazing technique for real-time computer vision applications.  SN Applied Sciences, 2:1–10, 2020.
  * [22] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller.  Multi-view convolutional neural networks for 3d shape recognition.  In Proc. ICCV, 2015.
  * [23] Ragav Venkatesan, Hemanth Venkateswara, Sethuraman Panchanathan, and Baoxin Li.  A strategy for an uncompromising incremental learner.  arXiv preprint arXiv:1705.00744, 2017.
  * [24] Qian Wang and Min-Koo Kim.  Applications of 3d point cloud data in the construction industry: A fifteen-year review from 2004 to 2018.  Advanced Engineering Informatics, 39:306–319, 2019.
  * [25] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon.  Dynamic Graph CNN for Learning on Point Clouds.  ACM Transactions on Graphics (TOG), 2019.
  * [26] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  * [27] Yusheng Xu, Xiaohua Tong, and Uwe Stilla.  Voxel-based representation of 3d point clouds: Methods, applications, and its potential use in the construction industry.  Automation in Construction, 126:103675, 2021.
  * [28] Yuqing Zhu, Xiang Yu, Manmohan Chandraker, and Yu-Xiang Wang.  Private-knn: Practical differential privacy for computer vision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11854–11862, 2020.


# Riemannian classification of EEG signals with missing values

###### Abstract

This paper proposes a strategy to handle missing data for the classification
of electroencephalograms using covariance matrices. It relies on the observed-
data likelihood within an expectation-maximization algorithm. This approach is
compared to two existing state-of-the-art methods: _(i)_ covariance matrices
computed with imputed data; _(ii)_ Riemannian averages of partially observed
covariance matrix. All approaches are combined with the minimum distance to
Riemannian mean classifier and applied to a classification task of two widely
known paradigms of brain-computer interfaces. In addition to be applicable for
a wider range of missing data scenarios, the proposed strategy generally
performs better than other methods on the considered real EEG data.

###### Index Terms:

Missing data, Electroencephalography, Brain-computer interfaces, Covariance
estimation, Riemannian geometry.

##  I Introduction

Electroencephalography (EEG) is a non-invasive neuroimaging modality pioneered
by Hans Berger [1], which consists in recording the electrical activity of the
brain by placing electrodes on the scalp. The low cost, simplicity and high
temporal resolution (it captures well the brain activity dynamics) of EEG have
made its popularity and allowed for its use in various applications. EEG is
indeed widely used in brain research for instance to study sleep or epilepsy.
It is also central for brain-computer interfaces (BCI), where the subject
interacts with a computer through brain signals. These can for example be
employed to control exoskeleton [2] or help mechanical ventilation [3].

This paper focuses onto BCI and more specifically onto two of the various
possible paradigms: event related potentials (ERP) and Motor Imagery (MI). ERP
are specific time-locked responses of the brain to some stimuli, such as
flashes [4]. Given the brain signals, the goal is to determine if the subject
was exposed to the stimulus, _i.e._ , find out if the specific time-locked
response is in the EEG recording. Concerning MI, the goal is to recognize a
mental task (as hand or foot movement) which is represented in different areas
of the motor cortex. In both cases, BCI boils down to a classification task of
brain signals at hand. Various methods have specifically been designed to
achieve this; see [5] for a recent review. In this work, we consider one of
the most efficient approach, called minimum distance to Riemannian mean (MDRM)
[6], which exploits covariance matrices of the data along with their natural
Riemannian geometry.

BCI classification methods have shown good performance on a number of
classical datasets; see _e.g._ , [7]. However, these datasets have been
recorded in laboratories, which are well controlled environments, and data
have been curated by hand to ensure the quality of the recordings. In real
life scenarios, obtaining such idealistic data seems unrealistic. One can
expect the data at hand to contain faulty measurements due for instance to
impedence change, electrode disconnections [8] or artifacts coming from
movements of the subject, blinks, _etc._ These can lead to missing data in EEG
recordings as one may decide to discard it from the data. When it comes to
classification tasks, missing data impedes the exploitation of complementary
information in non-missing channels. Considering this, it is of primary
importance to develop an appropriate procedure to classify incomplete EEG
signals.

To achieve this, several strategies can be employed. A first possibility is to
exploit some missing data imputation technique in order to retrieve a complete
EEG recording. It has for instance been considered for EEG in [9] with an
algorithm relying on spherical spline interpolation. Imputation can also be
performed through the $K$-nearest neighbors (KNN) algorithm [10]. Another
possibility is to directly work in the covariance space. When a channel is
missing, an imputation of symmetric positive definite (SPD) matrices with
uncorrelated white noise has been proposed in [11]. In a similar vein, the
authors of [12] propose to estimate the Riemannian mean of partially observed
covariance matrices with missing variables (channels) to later perform a
classification task with the MDRM. When the data contains missing samples111To
appreciate the different nature of works handling missing samples in the data
and those treating on missing variables in the data and/or the covariance
matrix, see Table 1 in [12]., it is possible to estimate the covariance matrix
of incomplete data using an expectation-maximization (EM) algorithm [13]. More
recently, a neural network approach was designed to interpolate missing
electrodes in EEG [14].

In this paper, we propose to adapt the state-of-the-art EEG classification
pipeline [6] to handle missing data by exploiting an EM algorithm to compute
covariance matrices. Compared to two state-of-the-art approaches – missing
data imputation using KNN [10] and Riemannian averaging over incomplete
covariance matrices [12] – our method is applicable in more missing data
scenarios. Furthermore, experiments conducted onto real ERP and motor imagery
(MI) EEG data tend to show that our method is competitive in terms of
performance as compared to others.

##  II EEG classification for complete data

To classify EEG data, rather than directly using the raw data, one usually
exploits their covariance matrices as features [6]. The classification of
these covariance matrices is then performed by leveraging their Riemannian
geometry: a well-adapted distance is used to compute centers of mass of the
different classes and assigning the class to unknown recordings.

###  II-A Covariance matrix estimation

A common hypothesis for EEG signals is to assume a multivariate normal
distribution [15]. Let
$\bm{X}=[\bm{x}_{1},\dots,\bm{x}_{n}]\in\mathbb{R}^{p\times n}$ be a recorded
EEG trial containing $p$-dimensional real vectors, where $p$ is the number of
electrodes and $n$ the time samples. $\bm{X}$ follows a zero-mean Gaussian
distribution with covariance matrix $\bm{\Sigma}\in\mathbb{R}^{p\times p}$,
denoted $\bm{X}\sim\mathcal{N}(\bm{0},\bm{\Sigma})$. When the data is
complete, the Maximum Likehood Estimator (MLE) is the well-known Sample
Covariance Matrix (SCM):

$\bm{\Sigma}=\frac{1}{n}\bm{X}\bm{X}^{\top}$ (1)

###  II-B Minimum distance to Riemannian mean

The idea consists in leveraging a non-Euclidean distance based on the geometry
of the space of SPD matrices (thus covariance matrices). In peculiar,
theoretical studies have shown benefits to use the so-called affine invariant
distance [16]. Given two covariances matrices
$\bm{\Sigma}_{1},\bm{\Sigma}_{2}$, its expression is given by:

$\delta_{\mathcal{AF}}(\bm{\Sigma}_{1},\bm{\Sigma}_{2})=\left\|\log\left(\bm{\Sigma}_{1}^{-1/2}\bm{\Sigma}_{2}\bm{\Sigma}_{1}^{-1/2}\right)\right\|_{\mathrm{F}},$
(2)

where $\|\cdot\|_{\mathrm{F}}$ is the Frobenius norm and $\log$ is the matrix
logarithm. Then, given a classification problem of $Z$ classes, with training
data $\\{(\bm{\Sigma}_{\ell},y_{\ell}):1\leq\ell\leq L\\}$, the training phase
consists in computing the mean of each class
$\\{\overline{\bm{\Sigma}}^{(z)}:z\in\\{1,\dots,Z\\}\\}$ according to the
affine-invariant distance by using the iterative algorithm of [17]. A
predicted label $y$ is then obtained on a new sample $\bm{\Sigma}$ by
assigning it to the closest mean:

$y=\underset{z\in\\{1,\dots,Z\\}}{\operatorname*{arg\,min}}\delta_{\mathcal{AF}}(\bm{\Sigma},\overline{\bm{\Sigma}}^{(z)})$
(3)

##  III Existing strategies to handle missing data for EEG classification

###  III-A Data imputation: KNN algorithm

A first possibility to handle missing data is to impute them, which then
allows to compute their covariance matrix according to (1) – and thus apply
the MDRM classification method. We focus here on $K$-nearest neighbors (KNN)
interpolation [10]. If the $j^{\text{th}}$ electrode of $\bm{x}_{i}$ is
missing, KNN finds its $K$-closest neighbors in the training set according to
an Euclidean distance $d\big{(}\bm{x}_{i},\\{\bm{x}_{q}\\}_{q=1,q\neq
i}^{n}\big{)}$ compatible with missing data, called the heterogeneous
Euclidean-overlap metric (HEOM) [18]. If $\\{\bm{v}_{k}\\}_{k=1}^{K}$
represents the set of $K$-selected neighbors, then the imputed value
$\hat{x}_{ij}$ is

$\hat{x}_{ij}=\frac{1}{K}\sum_{k=1}^{K}w_{k}v_{kj},\;\;w_{k}=\frac{1}{d(\bm{x}_{i},\bm{v}_{k})^{2}},\;\;\sum_{k=1}^{K}w_{k}=1$
(4)

where $w_{k}$ denotes the corresponding weight of the $k$-th nearest neighbor
$\bm{v}_{k}$ and $d(\bm{x}_{i},\bm{v}_{k})^{2}$ is the squared HEOM. The
choice of $K$ can be determined by cross-validation on observed data only,
which can be challenging with high missing data ratio.

###  III-B Masked Minimum to Riemannian mean

In [12], authors propose an implementation of the MDM algorithm for incomplete
data. Such design requires to construct a mask to extract222Such matrix is
often called selection matrix. the observed data from an incomplete input
$\bm{X}\in\mathbb{R}^{p\times n}$. A mask $\bm{M}\in\mathbb{R}^{p\times(p-r)}$
is defined as the identity matrix with columns of the $r$ missing variables
removed. If $\bm{X}$ is incomplete, a complete $(p-r)\times n$ submatrix
$\check{\bm{X}}$ can be extracted using $\bm{M}^{\top}\bm{X}$. The covariance
matrix becomes

$\check{\bm{\Sigma}}=\frac{1}{n}\bm{M}^{\top}\bm{\Sigma}\bm{M}$ (5)

where $\bm{\Sigma}$ is the complete covariance matrix as defined in (1). The
next step is to consider the set of $\\{\bm{\Sigma}_{i}\\}_{i=1}^{n}$
incomplete SPD matrices, where each $\bm{\Sigma}_{i}$ has $r_{i}$ missing
variables and an associated mask $\bm{M}_{i}$. For each class
$z\in\\{1,\dots,Z\\}$, the masked Riemannian mean
$\bar{\bm{\Sigma}}\in\mathcal{S}_{++}^{p}$ is defined as:

$\overline{\bm{\Sigma}}^{(z)}=\operatorname*{arg\,min}\limits_{\bm{\Sigma}\in\mathcal{S}_{++}^{p}}\frac{1}{2}\sum_{i=1}^{n}w_{i}\delta^{2}_{\mathcal{AF}}(\bm{M}_{i}^{\top}\bm{\Sigma}_{i}\bm{M}_{i},\bm{M}_{i}^{\top}\bm{\Sigma}\bm{M}_{i})$
(6)

Finally, the label prediction is performed as in (3) with the suited mask
added.

##  IV An EM algorithm for covariance estimation with missing data

A powerful strategy to deal with missing values is to directly estimate the
covariance feature without imputation. In this case, one can rely on maximum
likelihood estimation using the EM algorithm developed in [19]. Here, the
unknown parameter to estimate is simply $\bm{\theta}=\\{\bm{\Sigma}\\}$. In
the following EM algorithm, $\bm{x}_{i}$ is transformed into
$\widetilde{\bm{x}}_{i}$ such that
$\widetilde{\bm{x}}_{i}=\bm{P}_{i}\bm{x}_{i}=\begin{pmatrix}\bm{x}_{i}^{\text{o}}&\bm{x}_{i}^{\text{m}}\end{pmatrix}^{\top}$,
where $\bm{P}_{i}\in\mathbb{R}^{p\times p}$ is a permutation matrix,
$\bm{x}_{i}^{\text{o}}$ and $\bm{x}_{i}^{\text{m}}$ denote the observed and
missing elements of $\bm{x}_{i}$, respectively. The covariance matrix of
$\widetilde{\bm{x}}_{i}$ is

$\widetilde{\bm{\Sigma}}_{i}=\begin{pmatrix}\widetilde{\bm{\Sigma}}_{i,\text{oo}}&\widetilde{\bm{\Sigma}}_{i,\text{mo}}\\\
\widetilde{\bm{\Sigma}}_{i,\text{om}}&\widetilde{\bm{\Sigma}}_{i,\text{mm}}\end{pmatrix}=\bm{P}_{i}\bm{\Sigma}\bm{P}_{i}^{\top}$
(7)

where $\widetilde{\bm{\Sigma}}_{i,\text{mm}}$,
$\widetilde{\bm{\Sigma}}_{i,\text{mo}}$,
$\widetilde{\bm{\Sigma}}_{i,\text{oo}}$ are the block CM of
$\bm{x}_{i}^{\text{m}}$, of $\bm{x}_{i}^{\text{m}}$ and
$\bm{x}_{i}^{\text{o}}$, and of $\bm{x}_{i}^{\text{o}}$. Replacing
$\bm{x}_{i}$ by its permuted version $\widetilde{\bm{x}}_{i}$, the complete
log-likelihood for the Gaussian distribution is expressed as:

$\displaystyle\mathcal{L}_{c}(\bm{\theta}|\bm{x})$
$\displaystyle\propto-n\log|\bm{\Sigma}|-\sum_{i=1}^{n}\begin{pmatrix}\bm{x}_{i}^{\text{o}}\\\
\bm{x}_{i}^{\text{m}}\end{pmatrix}^{\top}\widetilde{\bm{\Sigma}}_{i}^{-1}\begin{pmatrix}\bm{x}_{i}^{\text{o}}\\\
\bm{x}_{i}^{\text{m}}\end{pmatrix}$ (8)

The EM algorithm, which is given in Algorithm 1, can now be formulated.

Initialization. At step $t=0$ of the algorithm, initialize the covariance
matrix with the SCM computed from fully observed $\bm{x}_{i}$ (i.e., no
missing values), denoted $\bm{\Sigma}_{\text{SCM-obs}}$.

E-step. Compute the expectation of the missing data conditioned by the
observed data and the estimated covariance at the $t$-th iteration:

$\displaystyle Q_{i}(\bm{\theta}|\bm{\theta}^{(t)})$
$\displaystyle=\mathbb{E}_{\bm{x}_{i}^{\text{m}}|\bm{x}_{i}^{\text{o}},\bm{\theta}^{(t)}}\big{[}\mathcal{L}_{c}(\bm{\theta}|\bm{x}_{i}^{\text{o}},\bm{x}_{i}^{\text{m}})\big{]}$
$\displaystyle=\text{tr}(\bm{B}_{i}^{(t)}\widetilde{\bm{\Sigma}}_{i}^{-1})$
(9)

with
$\bm{B}_{i}^{(t)}=\begin{pmatrix}\bm{x}_{i}^{\text{o}}\bm{x}_{i}^{\text{o}^{\top}}&\mathbb{E}_{\bm{x}_{i}^{\text{m}}|\bm{x}_{i}^{\text{o}},\bm{\theta}^{(t)}}[\bm{x}^{\text{o}}_{i}\bm{x}^{\text{m}^{\top}}_{i}]\\\
\mathbb{E}_{\bm{x}_{i}^{\text{m}}|\bm{x}_{i}^{\text{o}},\bm{\theta}^{(t)}}[\bm{x}^{\text{m}}_{i}\bm{x}^{\text{o}^{\top}}_{i}]&\mathbb{E}_{\bm{x}_{i}^{\text{m}}|\bm{x}_{i}^{\text{o}},\bm{\theta}^{(t)}}[\bm{x}^{\text{m}}_{i}\bm{x}^{\text{m}^{\top}}_{i}]\end{pmatrix}$.
The only expectation to compute are
$\mathbb{E}_{\bm{x}_{i}^{\text{m}}|\bm{x}_{i}^{\text{o}},\bm{\theta}^{(t)}}[\bm{x}^{\text{m}}_{i}]$
and
$\mathbb{E}_{\bm{x}_{i}^{\text{m}}|\bm{x}_{i}^{\text{o}},\bm{\theta}^{(t)}}[\bm{x}^{\text{m}}_{i}\bm{x}^{\text{m}^{\top}}_{i}]$,
which are the expectation of the sufficient statistics $\bm{x}^{\text{m}}_{i}$
and $\bm{x}^{\text{m}}_{i}\bm{x}^{\text{m}^{\top}}_{i}$ (denoted
$\mathbb{E}\big{[}\bm{x}^{\text{m}}_{i}\big{]}$ and
$\mathbb{E}\big{[}\bm{x}^{\text{m}}_{i}\bm{x}^{\text{m}^{\top}}_{i}\big{]}$
hereafter). A classical result in conditional distributions [20] gives the
following expressions:

$\displaystyle\mathbb{E}\big{[}\bm{x}^{\text{m}}_{i}\big{]}$
$\displaystyle=\widetilde{\bm{\Sigma}}_{i,\text{mo}}\widetilde{\bm{\Sigma}}_{i,\text{oo}}^{-1}\widetilde{\bm{x}}_{i}^{\text{o}}$
(10)
$\displaystyle\mathbb{E}\big{[}\bm{x}^{\text{m}}_{i}\bm{x}^{\text{m}^{\top}}_{i}\big{]}$
$\displaystyle=\widetilde{\bm{\Sigma}}_{i,\text{mm}}-\widetilde{\bm{\Sigma}}_{i,\text{mo}}\widetilde{\bm{\Sigma}}_{i,\text{oo}}^{-1}\widetilde{\bm{\Sigma}}_{i,\text{om}}+\mathbb{E}\big{[}\bm{x}^{\text{m}}_{i}\big{]}\mathbb{E}\big{[}\bm{x}^{\text{m}}_{i}\big{]}^{\top}$
(11)

M-step. Resolve the following maximization problem:

$\bm{\Sigma}^{(t+1)}=\operatorname*{arg\,max}\limits_{\bm{\Sigma}\in\mathcal{S}_{++}^{p}}\sum_{i=1}^{n}Q_{i}(\bm{\Sigma}|\bm{\Sigma}^{(t)})$
(12)

where $\mathcal{S}_{++}^{p}$ is the set of symmetric positive definite (SPD)
matrices. Solving (12) requires to solve $\frac{\partial
Q_{i}}{\partial\bm{\Sigma}}=0$. Fortunately in this case, the following
closed-form expression can be found with simple derivation calculus:

$\bm{\Sigma}^{(t+1)}=\frac{1}{n}\sum_{i=1}^{n}\bm{P}_{i}\bm{B}_{i}^{(t)\top}\bm{P}_{i}^{\top}$
(13)

1:$\\{\widetilde{\bm{x}}_{i}\\}_{i=1}^{n}\sim\mathcal{N}(\bm{0},\bm{\Sigma}),\\{\bm{P}_{i}\\}_{i=1}^{n}$

2:$\widehat{\bm{\Sigma}}$

3:Initialization: $\bm{\Sigma}^{(0)}=\widehat{\bm{\Sigma}}_{\text{SCM-obs}}$

4:repeat

5: Compute
$\bm{B}_{i}^{(t)}=\begin{pmatrix}\bm{x}_{i}^{\text{o}}\bm{x}_{i}^{o\top}&\mathbb{E}[\bm{x}^{\text{o}}_{i}\bm{x}^{\text{m}^{\top}}_{i}]\\\
\mathbb{E}[\bm{x}^{\text{m}}_{i}\bm{x}^{\text{o}^{\top}}_{i}]&\mathbb{E}[\bm{x}_{i}^{\text{m}}\bm{x}_{i}^{\text{m}\top}]\end{pmatrix}$

6: Compute
$\bm{\Sigma}^{(t+1)}=\frac{1}{n}\sum_{i=1}^{n}\bm{P}_{i}\bm{B}_{i}^{(t)\top}\bm{P}_{i}^{\top}$

7: $t\leftarrow t+1$

8:until $||\bm{\Sigma}^{(t+1)}-\bm{\Sigma}^{(t)}||^{2}_{F}$ converges

Algorithm 1 Estimation of $\bm{\Sigma}$ with the EM algorithm

##  V Numerical experiments

###  V-A Data

The proposed procedures are assessed through a detection task of EEG
recordings of two distinct BCI experiments333Both data sets are accessed
through the moabb database [7].:

  * i)

P300 Event Related Potentials triggered from target stimuli consisting of
flashes [4]. This dataset consists in 3 recording sessions of EEG potentials
on 10 subjects using 16 electrodes444Fz, FCz, Cz, CPz, Pz, Oz, F3, F4, C3, C4,
CP3, CP4, P3, P4, PO7, PO8. Please refer to [21] for an extensive description
of the dataset.. The signals are band-pass filtered between 1 and 20Hz,
downsampled to 128Hz and epoched into 1728 trials: one trial out of six
corresponds to the flash stimulus (class 1) while the rest correspond to the
absence of target (class 0). Before computing covariance matrices, raw data
are augmented by concatenating the prototyped ERP response as prescribed in
[22]. This leads for each subjects to a dataset of $L=1728$ covariance
matrices as in (1) with $p=16$ and $n=103$.

  * ii)

Motor Imagery: this dataset [23] consists of EEG data from 9 subjects recorded
using 22 electrodes. This BCI paradigm consists of four different motor
imagery tasks, namely the imagination of movement of the left hand (class 1),
right hand (class 2), both feet (class 3), and tongue (class 4). Two sessions
were recorded for each subject, with a total of 288 trials per session. This
leads for each subjects to a dataset of $L=576$ covariance matrices with
$p=22$ and $n=1001$.

###  V-B Missing data scenarios

$\dotsc$$\displaystyle n$ time samples$\displaystyle p$ electrodes eye
blinking electrode popping Fig. 1: Illustration of electrode popping (red) and
eye blinking (purple).

We consider two realistic scenarios of incomplete EEG signals: electrode
popping (s1) and eye blinking (s2). As illustrated in Fig.1, electrode popping
yields one or multiple missing electrodes during signal acquisition. Eye
blinking creates an undesired artifact over a short time period (typically
$\sim$200 ms) over a group of electrodes. Such artifacts are then removed,
which creates missing data. As shown in Table I, compared to the other methods
considered, our proposed strategy is the only one working in both cases.

| KNN [10] | Masked mean [12] | EM [19]
---|---|---|---
Electrode popping (s1) | $\times$ | ✓ | ✓
Eye blinking (s2) | ✓ | $\times$ | ✓
Table I: Applicability (✓: applicable; $\times$: not applicable) of the
proposed methods for different missingness scenarios.

For scenario s1, two electrodes are entirely removed. For scenario s2, three
groups of electrodes (11 for P300 and 12 for MI) during three time intervals
are removed. In both cases, removals are done on an increasing number of
trials to appreciate the effect on classification accuracy.

###  V-C Results

To compare the accuracy of the classification of P300 and Motor Imagery
datasets, the MRDM classifier as described in section II is used. The
following strategies are considered to estimate the covariance matrices: SCM:
the SCM of complete data, i.e., without missing values; KNN + SCM: KNN imputer
implemented from the scikit-learn package [24] applied to incomplete data with
a number of $k=5$ neighbors. Masked-SCM: MDRM classifier using a masked
version of the Riemannian mean [12]. EM-SCM: hybrid strategy where the
covariance matrix is estimated through Algorithm 1 and fed into the MDRM
classifier.

Performance is evaluated using the classical training-test paradigm with
stratified $K$-folding ($K=5$). Algorithms are calibrated on data recorded
during the training phase, and applied on data recorded during the test phase.

Electrode popping experiment. Accuracy results for this experiment are
presented in Figure 2. The best performance is achieved on complete data
(SCM), which serves as a baseline. EM-based classification achieves equal or
higher scores than the masked MDRM for both datasets. On these data, our
proposed strategy thus appears more advantageous than the masked version of
the Riemannian mean.

10203040$\displaystyle{80}$$\displaystyle{82}$$\displaystyle{84}$$\displaystyle{86}$$\displaystyle{88}$Incomplete
epochs (%)P300SCMEM-SCMMask-
SCM10203040$\displaystyle{42}$$\displaystyle{46}$$\displaystyle{50}$$\displaystyle{54}$$\displaystyle{58}$$\displaystyle{62}$Incomplete
epochs (%)Mean accuracy (%)Motor Imagery Fig. 2: Electrode popping experiment:
classification mean accuracy as function of the incomplete epochs ratio.

Eye blinking experiment. Results for this experiment are presented in Figure
3. For the motor imagery dataset, KNN imputation does not perform as well as
the EM-based classification. For the P300 dataset, KNN imputation shows higher
accuracies as compared to the EM algorithm when the amount of missing data
increases.

10203040$\displaystyle{53}$$\displaystyle{56}$$\displaystyle{59}$$\displaystyle{62}$Incomplete
epochs (%)Mean accuracy (%)Motor
Imagery10203040$\displaystyle{82}$$\displaystyle{84}$$\displaystyle{86}$$\displaystyle{88}$Incomplete
epochs (%)P300SCMEM-SCMk-NN + SCM Fig. 3: Eye blinking experiment:
classification mean accuracy as function of the incomplete epochs ratio.

##  VI Conclusion

In this paper, a new strategy to handle missing data in the context of EEG
classification is proposed. It relies on an EM algorithm to directly estimate
covariance matrices in the presence of missing data. As compared to two other
state-of-the-art strategies – based on KNN imputation and masked Riemannian
means – our proposed method works with a wider range of missing data
scenarios. In addition, experiments conducted on real EEG data tends to show
that our method is also competitve in terms of accuracy results.

## References

  * [1] H. Berger, “Über das elektrenkephalogramm des menschen,” _Archiv für psychiatrie und nervenkrankheiten_ , vol. 87, no. 1, pp. 527–570, 1929\.
  * [2] E. K. Kalunga, S. Chevallier, Q. Barthélemy, K. Djouani, E. Monacelli, and Y. Hamam, “Online ssvep-based bci using riemannian geometry,” _Neurocomputing_ , vol. 191, pp. 55–68, 2016.
  * [3] S. Chevallier, G. Bao, M. Hammami, F. Marlats, L. Mayaud, D. Annane, F. Lofaso, and E. Azabou, “Brain-machine interface for mechanical ventilation using respiratory-related evoked potential,” in _International Conference on Artificial Neural Networks_ , 2018, pp. 662–671.
  * [4] J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, “Brain–computer interfaces for communication and control,” _Clinical Neurophysiology_ , vol. 113, no. 6, pp. 767–791, 2002.
  * [5] F. Lotte, L. Bougrain, A. Cichocki, M. Clerc, M. Congedo, A. Rakotomamonjy, and F. Yger, “A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update,” _Journal of neural engineering_ , vol. 15, no. 3, 2018.
  * [6] A. Barachant, S. Bonnet, M. Congedo, and C. Jutten, “Multiclass brain–computer interface classification by Riemannian geometry,” _IEEE Transactions on Biomedical Engineering_ , vol. 59, no. 4, pp. 920–928, 2011.
  * [7] V. Jayaram and A. Barachant, “MOABB: trustworthy algorithm benchmarking for BCIs,” _Journal of Neural Engineering_ , vol. 15, no. 6, p. 066011, 2018\.
  * [8] N. Bahador, K. Erikson, J. Laurila, J. Koskenkari, T. Ala-Kokko, and J. Kortelainen, “A correlation-driven mapping for deep learning application in detecting artifacts within the EEG,” _Journal of Neural Engineering_ , vol. 17, no. 5, p. 056018, 2020.
  * [9] F. Perrin, J. Pernier, O. Bertrand, and J. Echallier, “Spherical splines for scalp potential and current density mapping,” _Electroencephalography and Clinical Neurophysiology_ , vol. 72, no. 2, pp. 184–187, 1989.
  * [10] O. Troyanskaya, M. Cantor, G. Sherlock, P. Brown, T. Hastie, R. Tibshirani, D. Botstein, and R. B. Altman, “Missing value estimation methods for DNA microarrays,” _Bioinformatics_ , vol. 17, no. 6, pp. 520–525, 2001.
  * [11] P. L. C. Rodrigues, M. Congedo, and C. Jutten, “A data imputation method for matrices in the symmetric positive definite manifold,” in _XXVIIème colloque GRETSI (GRETSI 2019)_ , 2019.
  * [12] F. Yger, S. Chevallier, Q. Barthélemy, and S. Sra, “Geodesically-convex optimization for averaging partially observed covariance matrices,” in _Proceedings of The 12th Asian Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, vol. 129, 2020, pp. 417–432.
  * [13] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” _J. Royal Statistical Society. Series B (Methodological)_ , vol. 39, no. 1, pp. 1–38, 1977.
  * [14] S. Saba-Sadiya, T. Alhanai, T. Liu, and M. M. Ghassemi, “EEG channel interpolation using deep encoder-decoder networks,” in _2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , 2020, pp. 2432–2439.
  * [15] M. Congedo, C. Gouy-Pailler, and C. Jutten, “On the blind source separation of human electroencephalogram by approximate joint diagonalization of second order statistics,” _Clinical Neurophysiology_ , vol. 119, no. 12, pp. 2677–2686, 2008.
  * [16] X. Pennec, “3 - manifold-valued image processing with SPD matrices,” in _Riemannian Geometric Statistics in Medical Image Analysis_. Academic Press, 2020, pp. 75–134.
  * [17] B. Jeuris, R. Vandebril, and B. Vandereycken, “A survey and comparison of contemporary algorithms for computing the matrix geometric mean,” _Electronic Transactions on Numerical Analysis_ , vol. 39, pp. 379–402, 2012\.
  * [18] P. J. García-Laencina, J.-L. Sancho-Gómez, A. R. Figueiras-Vidal, and M. Verleysen, “K nearest neighbours with mutual information for simultaneous classification and missing data imputation,” _Neurocomputing_ , vol. 72, no. 7, pp. 1483–1493, 2009.
  * [19] A. Hippert-Ferrer, M. El Korso, A. Breloy, and G. Ginolhac, “Robust low-rank covariance matrix estimation with a general pattern of missing values,” _Signal Processing_ , vol. 195, p. 108460, 2022.
  * [20] T. Anderson, _An Introduction to Multivariate Statistical Analysis_. New York: Wiley, 1965.
  * [21] P. Aricò, F. Aloise, F. Schettini, S. Salinari, D. Mattia, and F. Cincotti, “Influence of P300 latency jitter on event related potential-based brain–computer interface performance,” _Journal of Neural Engineering_ , vol. 11, no. 3, p. 035008, 2014.
  * [22] A. Barachant and M. Congedo, “A plug&play P300 BCI using information geometry,” _arXiv preprint arXiv:1409.0107_ , 2014.
  * [23] M. Tangermann, K.-R. Müller, A. Aertsen, N. Birbaumer, C. Braun, C. Brunner, R. Leeb, C. Mehring, K. Miller, G. Mueller-Putz, G. Nolte, G. Pfurtscheller, H. Preissl, G. Schalk, A. Schlögl, C. Vidaurre, S. Waldert, and B. Blankertz, “Review of the bci competition iv,” _Frontiers in Neuroscience_ , vol. 6, 2012.
  * [24] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg _et al._ , “Scikit-learn: Machine learning in python,” _the Journal of machine Learning research_ , vol. 12, pp. 2825–2830, 2011.