# Motor Imagery Decoding Using Ensemble Curriculum Learning and Collaborative
Training

Georgios Zoumpourlis and Ioannis Patras The work of Georgios Zoumpourlis was
supported by QMUL Principal’s Studentship.Georgios Zoumpourlis and Ioannis
Patras are with the School of Electronic Engineering and Computer Science,
Queen Mary University of London, London E1 4NS, United Kingdom. E-mail:
{g.zoumpourlis<EMAIL_ADDRESS>
###### Abstract

Objective: In this work, we study the problem of cross-subject motor imagery
(MI) decoding from electroenchephalography (EEG) data. Multi-subject EEG
datasets present several kinds of domain shifts due to various inter-
individual differences (e.g. brain anatomy, personality and cognitive
profile). These domain shifts render multi-subject training a challenging task
and also impede robust cross-subject generalization. Method: We propose a two-
stage model ensemble architecture, built with multiple feature extractors
(first stage) and a shared classifier (second stage), which we train end-to-
end with two loss terms. The first loss applies curriculum learning, forcing
each feature extractor to specialize to a subset of the training subjects and
promoting feature diversity. The second loss is an intra-ensemble distillation
objective that allows collaborative exchange of knowledge between the models
of the ensemble. Results: We compare our method against several state-of-the-
art techniques, conducting subject-independent experiments on two large MI
datasets, namely Physionet and OpenBMI. Our algorithm outperforms all of the
methods in both 5-fold cross-validation and leave-one-subject-out evaluation
settings, using a substantially lower number of trainable parameters.
Conclusion: We demonstrate that our model ensembling approach combining the
powers of curriculum learning and collaborative training, leads to high
learning capacity and robust performance. Significance: Our work addresses the
issue of domain shifts in multi-subject EEG datasets, paving the way for
calibration-free BCI systems.

###### Index Terms:

Brain-Computer Interface (BCI), electroencephalography (EEG), motor imagery,
domain generalization

##  I Introduction

Brain-Computer Interfaces (BCIs) [51] are communication systems that enable
human users to interact with computers, robotic limbs or wheelchairs,
translating brain activity into commands. BCIs have a wide spectrum of
applications, including post-stroke rehabilitation of limb motor impairments
[6], character typing through visual spellers [50] and interactive image
generation [45]. The operation of BCI systems leverages neuroimaging
techniques to collect brain signals, with the most prevalent one being
electroencephalography (EEG) [8, 1]. Advancing EEG-based BCIs towards out-of-
the-lab settings, requires equipping them with machine learning models that
have robust cross-subject generalization.

With the advent of deep learning (DL), significant steps have been made in the
exploration of training methodologies and model architectures [20] that can
accomodate learning from EEG datasets with increasingly large number of
participants. Inter-subject variability is one of the biggest challenges in
the field of EEG-based BCIs, referring to the existence of differences in the
characteristics of EEG signals acquired from different individuals (i.e.,
there are different data distributions for each subject) [35]. In the
literature, often the data of each individual are considered as a separate
domain [28], hence the inter-subject differences are treated as domain shifts.

Figure 1: Overview of our proposed ensemble architecture during inference.
First, an input EEG trial is fed to multiple feature extractors that produce
diverse feature representations. Then, a single shared classifier predicts the
class scores corresponding to each feature representation, and these class
scores are averaged to compute the final prediction.

In this work, we consider the problem of EEG-based motor imagery (MI) decoding
in subject-independent settings. Motor imagery is a well-known paradigm for
BCIs, involving the imagination of motor acts, without overt motor execution
or muscle activation [34]. The usage of the MI paradigm is based on the
phenomenon of sensorimotor rhythms [52], i.e. rhythmic oscillations over the
sensorimotor cortex, that arise during motor imagery. There are several
sources of variation that lead to domain shifts in cross-subject MI decoding
problems, such as personality type, cognitive profile, neurophysiological
predictors, brain anatomy and familiarity with BCI technology [24, 25]. These
variations in turn lead to spatial, spectral and temporal differences in the
manifestation of sensorimotor rhythms across individuals [40]. These
differences are the sources of domain shifts that we aim to overcome in order
to obtain robust performance in subject-independent settings.

Under this perspective, two learning paradigms have been widely explored to
build convolutional neural networks (CNNs) with strong subject-independent
performance: domain adaptation and domain generalization. In domain
adaptation, the goal is to reduce the domain shift using either solely
labelled [21] or unlabelled [33] data from the target domain. Evidently, the
requirement to have available data from the target domain (i.e. a test
subject) during the training phase is a limiting factor. By contrast, the goal
of domain generalization is generalizing to new (unseen) target domains
without requiring any data from them, which is more attractive for subject-
independent EEG-based BCI systems.

A category of domain generalization techniques [47] that has been successfully
applied on EEG-based problems [5, 39] is ensemble learning [55]. A resulting
property of ensembling is the emergence of diverse feature representations,
leading to better generalization. However, existing ensembling works achieve
this diversity at the cost of increased computational complexity, lengthy
model selection processes and less generic feature extractors. In the complex
ensemble architecture of InceptionEEG-Net (IENet) [15], each individual CNN
model has more than seventy convolutional layers. The model ensemble proposed
by Dolzhikova et al. [14] requires multiple hyperparameter tuning runs to
train each base model. In the method of Chadaklar et al. [12], a deep ensemble
model builds on handcrafted features, extracted from the subject-specific
algorithm of Filter Bank CSP [3].

Hence, some disadvantages of existing ensemble learning works are that they
rely on complex architectures, do not support end-to-end training in a single
phase and build on subject-specific models. Differently from works that
require multiple training runs [14, 5, 39], our method involves training a
single model in an end-to-end manner. Moreover, in contrast to Chadaklar et
al. [12], we refrain from combining individual models that are trained on a
single subject each, as this compromises their generalization capabilities.
Compared to IENet [15], our ensemble is composed from base models that have a
compact and simplistic architecture.

We frame our approach as a model ensembling method combined with: (i) a
curriculum learning strategy to promote the diversity across individual models
and (ii) a collaborative training scheme to exchange knowledge between the
models through a distillation loss. We design a training curriculum, such that
each model of the ensemble is trained on all the source domains (i.e. training
subjects), but progressively specializes to a specific subset of subjects.
This leads each model to capture patterns that are mostly specific to the EEG
signal characteristics of a subset of training subjects, rather than the
entire training set. Training our architecture under such a curriculum, equips
it with strong generalization capabilities, by covering a wide range of
patterns through several models that act as diverse feature extractors. To
regulate the trade-off between diversity and generalization [9], we introduce
an intra-ensemble distillation loss that pushes the predictions of each
individual model close to the average of the predictions of all the other
models, thereby controlling the diversity within the ensemble. This is done by
using pseudolabels (obtained from the predictions of all the other models) as
groundtruth targets for each model. In essence, our collaborative training
scheme leads to distillation of knowledge across the models, working
complementary with the curriculum that is designed for each model. The balance
between diversity and generalization is controlled through a hyperparameter
that weighs the contribution of the pseudolabel-based loss to the total loss.

Our contributions are the following:

  * •

We propose a model ensembling architecture which we pair with a novel
curriculum learning scheme. Our curriculum promotes diversity across the
models of the ensemble, driving each model to specialize to a different subset
of training subjects. To our knowledge, curriculum learning has not been
previously explored for cross-subject MI decoding.

  * •

We propose an auxiliary intra-ensemble distillation loss, allowing the
exchange of knowledge between the individual models of the ensemble. This
balances the diversity-generalization trade-off, leading to further
performance improvement. Our work is the first to propose a pseudolabelling
scheme for EEG-based knowledge distillation.

  * •

We conduct our experimental analysis on two large motor imagery datasets
(Physionet [16] and OpenBMI [32]) totalling more than 150 subjects. We compare
our method against four state-of-the-art techniques, namely TIDNet [28],
EEGSym [38], MIN2Net [4] and ATL [54], showing superior results.

  * •

We make our code publicly
available111https://github.com/gzoumpourlis/Ensemble-MI to support
reproducibility.

The rest of the manuscript is organized as follows. In Section II we outline
relevant previous works, while in Section III we describe our proposed method.
In Section IV we present the results of our experimental analyses and ablation
studies. Lastly, in Section V we conclude the manuscript.

##  II Related Work

In this Section, we present an overview of the related work on the topics of
domain generalization, ensemble learning and feature diversity.

Domain generalization: A much larger volume of works has been published on
visual data, compared to the works on EEG data. Regarding learning from visual
data, there is a line of works that claim various benefits, however upon a
rigorous comparison, Gulrajani et al. [17] show that the trivial approach of
Empirical Risk Minimization (ERM) [46] can, if carefully tuned, outperform
several state-of-the-art domain generalization techniques. Several works
building subject-independent models for EEG data (which by nature is a domain
generalization problem), do not explicitly take care of inter-subject
variability [4, 56, 38]. Such methods adopt ERM-based approaches that simply
minimize the training loss over all source domains (i.e. training subjects).
Apart from the effective ERM baseline, other methods that have been
occasionaly used for multi-subject EEG training are MixUp [53, 28] (which was
initially proposed for visual data) and Euclidean/Riemannian Alignment (EA/RA)
[19, 28, 49]. Alignment methods such as EA/RA are powerful domain
generalization baselines for learning domain-invariant representations, which
can be combined with other multi-domain learning techniques [28]. By contrast,
MixUp has been found to have a rather detrimental effect on multi-subject
training [28]. In our work, we leverage the benefits of RA, along with our
proposed curriculum learning and intra-ensemble distillation techniques.

Ensemble learning: Model ensembling [18] is ubiquitous in the applications of
machine learning on EEG data [44]. Combining the predictions of multiple
models has been tried both in intra-subject [37] and subject-independent [41]
settings, using variants of the Common Spatial Pattern (CSP) [27, 10]
algorithm. The fact that CSP methods extract handcrafted features is a
restricting factor, and their cross-subject generalization capabilities have
been shown to be poor. A successful example of model ensembling using CNNs is
the work of Bakas et al. [5], where a $k$-fold cross validation process
results in $k$ trained models, with each model trained on data from all the
available training subjects. Reuben et al. [39] leverage the power of
available crowdsourced algorithms for an EEG-based seizure prediction
competition [29], exploring the possibility of obtaining performance
improvements by combining them through model ensembling.

Feature diversity: One of the key properties of ensemble learning, is the
emergence of diverse feature representations across the individual base models
of ensembles. However, feature diversity can also be obtained through
alternative techniques which do not fall within the category of ensemble
learning, as they explore ways to obtain diverse features through a single
model. Ma et al. [36] propose a multi-branch network architecture where the
input EEG signal is divided in four frequency bands, with a dedicated branch
for each band. Altuwaijri et al. [2] introduce a multi-branch network based on
EEGNet [31], where each branch contains a different number of temporal
filters, as well as a different temporal filter length. A network capable of
processing spectral-spatial representations as inputs is presented by Kwon et
al. [30].

We draw inspiration from Wei et al. [48], where a multi-branch Separate-
Common-Separate Network (SCSN) is proposed to tackle the issue of negative
transfer learning. Negative learning can appear when training subject-agnostic
feature extractors, i.e. when all the layers of a single model are trained on
all the training subjects. As a remedy to this, SCSN has a separate feature
extractor for each training subject. However we claim that such an approach
leads to non-optimal solutions, as training subject-specific layers
compromises their generalization capability. We propose a model ensembling
approach that differs from these two scenarios (i.e. subject-specific or
subject-agnostic layers), yet combines the best of both worlds. In contrast to
SCSN [48], we train multiple feature extraction models on all training
subjects, yet we guide each individual feature extractor to specialize on a
subset of multiple subjects.

##  III Proposed Method

In this section we describe the proposed methodology, which consists of a
model ensemble architecture, a curriculum training scheme and an intra-
ensemble distillation loss. We provide an overview of the training pipeline
for our proposed architecture in Fig. 2 and present its individual components
in the following subsections. Specifically, we begin by explaining our
ensemble architecture in Subsection III-A. Then, we introduce the first loss
term that materializes our curriculum learning scheme in Subsection III-B, as
well as the second loss term that enables collaborative training across the
models of the ensemble, in Subsection III-C.

Figure 2: Our proposed architecture has $\mathrm{K}$ first stage models and a
shared classifier in the second stage. The input trial $\mathbf{x}$ is
separately passed to each one of the first stage models, obtaining the feature
vectors $[\mathbf{f_{1}},\mathbf{f_{2}},\ldots,\mathbf{f_{\mathrm{K}}}]$ (Eq.
3). For the $k$-th model, the class-wise scores $\mathbf{\hat{y}_{k}}$ are
computed by forwarding $\mathbf{f_{k}}$ to the shared classifier of the second
stage (Eq. 4). In an ensembling scenario where the architecture is trained
without curriculum learning, we compute the individual model losses
$\mathcal{L}_{\texttt{CE}}^{\mathrm{k}}$ (Eq. 5) and minimize the loss
$\mathcal{L}_{\texttt{CE}}^{\mathrm{total}}$ (Eq. 6) for all models. In the
ensemble curriculum learning scenario, we compute the individual subject-
weighted losses $\mathcal{L}_{\mathrm{subj}}^{\mathrm{k}}$ (Eq. 9) and
minimize the loss $\mathcal{L}_{\mathrm{subj}}^{\mathrm{total}}$ (Eq. 10) for
all models. When also performing collaborative training, we additionaly
compute the losses $\mathcal{L}_{\mathrm{distill}}^{\mathrm{k}}$ (Eq. 12) and
minimize the total loss $\mathcal{L}_{\mathrm{total}}$ (Eq. 14) for all
models.

###  III-A Architecture

####  III-A1 Single model

In this work, we use the well-established EEGNet [31] architecture as our
strong single-model baseline. The selection of EEGNet is justified from the
fact that it achieves compelling performance, with a reasonably small number
of trainable parameters and a simple network design (e.g. without streams of
varying kernel lengths, or band-wise processing streams). In the task of MI
decoding, the time-series signals
$\mathbf{x}\in\mathbb{R}^{\mathrm{C}\times\mathrm{T}}$ of an EEG trial with
$\mathrm{C}$ electrodes and $\mathrm{T}$ samples in the temporal dimension,
are fed as input to EEGNet. The class-wise scores
$\mathbf{\hat{y}}\in\mathbb{R}^{\mathrm{N_{C}}}$ (where $\mathrm{N_{C}}$ is
the number of classes) are obtained as output, while the groundtruth label
$\mathbf{y}\in\mathbb{R}^{\mathrm{N_{C}}}$ is represented in the form of a
one-hot vector. Thus, in the case of EEGNet the output scores are computed as

$\mathbf{\hat{y}}=\texttt{EEGNet}(\mathbf{x}),$ (1)

and the network is optimized by minimizing the cross-entropy (CE) loss
$\mathcal{L}_{\texttt{CE}}=\texttt{CE}(\mathbf{\hat{y}},\mathbf{y})$, given by

$\texttt{CE}(\mathbf{\hat{y}},\mathbf{y})=-\sum_{\mathrm{i}=1}^{\mathrm{N_{C}}}y_{\mathrm{i}}\log\Bigl{(}\texttt{softmax}(\hat{y}_{\mathrm{i}})\Bigr{)},$
(2)

where $y_{\mathrm{i}}$ and $\hat{y}_{\mathrm{i}}$ are the $i$-th elements of
$\mathbf{y}$ and $\mathbf{\hat{y}}$ respectively.

####  III-A2 Model ensemble

Our model ensemble architecture (shown in Fig. 2) consists of two stages and
uses EEGNet as its elementary component. The first stage contains multiple
models in parallel, with all models having exactly the same architecture
design. These models act as feature extractors on an input sample, with each
model producing a feature vector. We use $\texttt{F}_{\texttt{k}}(\cdot)$ and
$\mathbf{f_{\mathrm{k}}}$ to denote the $k$-th feature extractor and its
output feature vector. The output feature vectors from the first stage, are
computed as

$[\mathbf{f_{\mathrm{1}}},\mathbf{f_{\mathrm{2}}},\ldots,\mathbf{f_{\mathrm{K}}}]=[\texttt{F}_{\texttt{1}}(\mathbf{x}),\texttt{F}_{\texttt{2}}(\mathbf{x}),\ldots,\texttt{F}_{\texttt{K}}(\mathbf{x})].$
(3)

The second stage has a single shared classification head $\texttt{G}(\cdot)$,
that computes the class-wise prediction scores for each feature vector
originating from the first stage. We use $\mathbf{\hat{y}}_{\mathrm{k}}$ to
denote the scores corresponding to the $k$-th feature vector
$\mathbf{f_{\mathrm{k}}}$. The scores are computed as

$[\mathbf{\hat{y}}_{\mathrm{1}},\mathbf{\hat{y}}_{\mathrm{2}},\ldots,\mathbf{\hat{y}}_{\mathrm{K}}]=[\texttt{G}(\mathbf{f_{\mathrm{1}}}),\texttt{G}(\mathbf{f_{\mathrm{2}}}),\ldots,\texttt{G}(\mathbf{f_{\mathrm{K}}})].$
(4)

In the simple scenario where no curriculum learning occurs, this architecture
is trained by minimizing the sum of the individual losses for the predictions
of each model. The loss $\mathcal{L}_{\texttt{CE}}^{\mathrm{k}}$ for the
predictions $\mathbf{\hat{y}}_{\mathrm{k}}$ of the $k$-th model, and the total
loss $\mathcal{L}_{\texttt{CE}}^{\mathrm{total}}$, are computed as

$\mathcal{L}_{\texttt{CE}}^{\mathrm{k}}=\texttt{CE}(\mathbf{\hat{y}}_{\mathrm{k}},\mathbf{y})$
(5)
$\mathcal{L}_{\texttt{CE}}^{\mathrm{total}}=\sum_{\mathrm{k}=1}^{\mathrm{K}}\mathcal{L}_{\texttt{CE}}^{\mathrm{k}}.$
(6)

In the inference phase, to classify an input sample $\mathbf{x}$ we fuse the
model-wise scores through a simple average operation and obtain a final score
vector $\mathbf{\hat{y}}_{\mathrm{ens}}$ as follows:

$\mathbf{\hat{y}}_{\mathrm{ens}}=\frac{1}{\mathrm{K}}\sum_{\mathrm{k}=1}^{\mathrm{K}}\mathbf{\hat{y}}_{\mathrm{k}}.$
(7)

To this end, the described architecture is purely subject-agnostic, having no
subject-specific layers (both in the first and second stage). In the following
subsection we propose an ensemble curriculum learning scheme that is applied
during training and changes the nature of the first stage layers. Our
curriculum provides a strong alternative to the typical subject-agnostic
layers, that can be adopted in ensemble learning.

###  III-B Ensemble curriculum learning

Our goal is to make each feature extractor to specialize on a specific subset
of subjects. That is, we want to induce local (i.e. focused on a subset of the
entire training set) feature extraction power to each model in the first
stage. Let
$\mathbf{\mathcal{D}}=\\{\mathbf{\mathcal{D}}_{1},\mathbf{\mathcal{D}}_{2},\ldots,\mathbf{\mathcal{D}}_{\mathrm{N}}\\}$
be a dataset with the data of $\mathrm{N}$ subjects, where
$\mathbf{\mathcal{D}}_{\mathrm{n}}$ denotes the sub-dataset containing the
trials of the $n$-th subject. For an ensemble with $\mathrm{K}$ models
($\mathrm{K}\geq 2$), we split $\mathbf{\mathcal{D}}$ into $\mathrm{K}$ non-
overlapping subsets $\mathbf{\mathcal{S}}$:
$\mathbf{\mathcal{D}}=\\{\mathbf{\mathcal{S}}_{1},\ldots,\mathbf{\mathcal{S}}_{\mathrm{K}}\\}$.
We do this splitting process by randomly assigning the sub-dataset of each
subject to one of the $\mathrm{K}$ subsets, with a uniform probability for all
subsets. Therefore, we have
$\bigcup_{\mathrm{k}=1}^{\mathrm{K}}\mathbf{\mathcal{S}}_{\mathrm{k}}=\mathbf{\mathcal{D}}$
and
$\mathbf{\mathcal{S}}_{\mathrm{i}}\cap\mathbf{\mathcal{S}}_{\mathrm{j}}=\emptyset$
for $i\neq j$. Each subset $\mathbf{\mathcal{S}}_{\mathrm{k}}$ corresponds to
the $k$-th model and contains the sub-datasets of the subjects on which we
drive the $k$-th model to specialize.

To achieve this specialization, we design a subject-weighted loss function
where we inject subject-specific coefficients to weigh the contribution of
each subject to the loss of each model. Considering the subject-weighted loss
$\mathcal{L}_{\mathrm{subj}}^{\mathrm{k}}$ that is used to train the $k$-th
model, the subject-specific coefficients linearly decay over epochs the loss
contribution of the subjects that do not belong to
$\mathbf{\mathcal{S}}_{\mathrm{k}}$. Effectively, this makes the $k$-th model
to focus more on the subjects of $\mathbf{\mathcal{S}}_{\mathrm{k}}$ that have
a non-decaying loss contribution. We scale the contribution of a training
sample $\mathbf{x}$ to the loss $\mathcal{L}_{\mathrm{subj}}^{\mathrm{k}}$
through the coefficient $\beta(\mathbf{x},\mathrm{k})$. If trial $\mathbf{x}$
corresponds to a subject that belongs in $\mathbf{\mathcal{S}}_{\mathrm{k}}$
(hence $\mathbf{x}\in\mathbf{\mathcal{S}}_{\mathrm{k}}$), then we keep
$\beta(\mathbf{x},\mathrm{k})=1$ throughout the whole training process.
Otherwise ($\mathbf{x}\notin\mathbf{\mathcal{S}}_{\mathrm{k}}$), we decay
$\beta(\mathbf{x},\mathrm{k})$ from 1 to 0 while training progresses, that is:

$\beta(\mathbf{x},\mathrm{k})=\left\\{\begin{array}[]{ll}1&,\mathrm{if}~{}\mathbf{x}\in\mathbf{\mathcal{S}}_{\mathrm{k}}\\\
\alpha&,\mathrm{if}~{}\mathbf{x}\notin\mathbf{\mathcal{S}}_{\mathrm{k}}\\\
\end{array}\right.,$ (8)

where
$\alpha=1-\frac{\mathrm{epoch}}{\mathrm{N}_{\mathrm{epochs}}}\in\left[0,1\right]$
represents the progression of training, as $\mathrm{N}_{\mathrm{epochs}}$ is
the maximum number of training epochs and $\mathrm{epoch}$ is the current
epoch. The loss $\mathcal{L}_{\mathrm{subj}}^{\mathrm{k}}$ of the $k$-th model
and the total subject-weighted loss
$\mathcal{L}_{\mathrm{subj}}^{\mathrm{total}}$ are computed as follows:

$\mathcal{L}_{\mathrm{subj}}^{\mathrm{k}}=\underbrace{\beta(\mathbf{x},\mathrm{k})}_{\text{\parbox{44.10185pt}{\centering
subject-specific \\\
coefficient\@add@centering}}}\cdot\mathcal{L}_{\texttt{CE}}^{\mathrm{k}}$ (9)
$\mathcal{L}_{\mathrm{subj}}^{\mathrm{total}}=\sum_{\mathrm{k}=1}^{\mathrm{K}}\mathcal{L}_{\mathrm{subj}}^{\mathrm{k}}.$
(10)

An indicative illustration of our curriculum learning scheme is shown in the
supplementary material.

###  III-C Intra-ensemble distillation for collaborative training

In this subsection we propose a collaborative training scheme which helps to
regulate the diversity-generalization trade-off in our model ensemble. In
order to classify a sample, we extract its first stage representations, feed
them to the shared classifier of the second stage and average the individual
scores across models. The diversity between the first stage representations of
a sample can make the classifier to compute inconsistent class scores across
models. This, in turn, can negatively affect the final prediction scores, as
they will be the result of fusing multiple contradicting predictions. We
observe that, although feature diversity is a desirable property of our
ensemble, it can also have an adverse effect on the generalization
capabilities.

To overcome this phenomenon, we introduce a loss term that promotes
consistency across the multiple model predictions, in order to improve the
performance of the entire ensemble. We design our proposed intra-ensemble
distillation loss to operate on the predicted scores of the second stage,
instead of operating on the features extracted from the first stage. An
overview of our distillation loss is shown in Fig. 2. Considering each
prediction $\hat{\mathbf{y}}_{\mathrm{k}}$ of the $k$-th model, our loss
pushes it closer to the softmaxed average of the predictions from all the
other models (which is the pseudolabel in our distillation loss).
Specifically, we compute the pseudolabel $\tilde{\mathbf{y}}_{\mathrm{k}}$ for
the $k$-th model as:

$\mathbf{\tilde{y}}_{\mathrm{k}}=\texttt{softmax}\biggl{(}\frac{1}{\mathrm{K}-1}\sum_{\mathrm{i=1,i\neq
k}}^{\mathrm{K}}\mathbf{\hat{y}}_{\mathrm{i}}\biggr{)},$ (11)

and use the cross-entropy loss between the prediction
$\hat{\mathbf{y}}_{\mathrm{k}}$ and the pseudolabel
$\mathbf{\tilde{y}}_{\mathrm{k}}$. We note that we apply a stop-gradient [13]
operation on the pseudolabels, as shown in Fig. 2. We do this in order to
ensure that only the weights of the $k$-th model are updated based on this
loss term, while the other models remain unaffected. For the $k$-th model, we
opt to not apply this loss on the samples of $\mathcal{S}_{\mathrm{k}}$. This
is done through a binary mask that zeroes out the distillation loss of these
samples. We do so, as the $k$-th model is sufficiently trained on the samples
of $\mathcal{S}_{\mathrm{k}}$ through their groundtruth labels $\mathbf{y}$.

We note that it is necessary to scale the contribution of the intra-ensemble
distillation loss to the total loss of the architecture, in accordance with
the progress of training. In the beginning of the training process, the
weights of the architecture are randomly initialized. Hence, penalizing the
distance of model predictions from the derived pseudolabels is not so
meaningful in the early epochs. As training proceeds, each feature extractor
progressively focuses on a subset of subjects and feature diversity increases.
As shown later in the experiments, our distillation loss indirectly controls
this emerging feature diversity by bringing closer the class scores computed
from various first stage features. We linearly increase the contribution of
the distillation loss to the total loss, across training epochs, by
multiplying it with the scalar $1-\alpha$ that quantifies the training
progress.

The distillation loss $\mathcal{L}_{\mathrm{distill}}^{\mathrm{k}}$ of the
$k$-th model, and the total distillation loss
$\mathcal{L}_{\mathrm{distill}}^{\mathrm{total}}$ are computed as follows:

$\mathcal{L}_{\mathrm{distill}}^{\mathrm{k}}=(1-\alpha)\cdot\underbrace{\mathds{1}(\mathbf{x}\notin\mathbf{\mathcal{S}_{\mathrm{k}}})}_{\text{\parbox{44.10185pt}{\centering
subject-specific \\\
mask\@add@centering}}}\cdot\texttt{CE}(\mathbf{\hat{y}}_{\mathrm{k}},\mathbf{\tilde{y}}_{\mathrm{k}})$
(12)
$\mathcal{L}_{\mathrm{distill}}^{\mathrm{total}}=\sum_{\mathrm{k}=1}^{\mathrm{K}}\mathcal{L}_{\mathrm{distill}}^{\mathrm{k}}.$
(13)

We compute the total loss $\mathcal{L}_{\mathrm{total}}$ of our architecture
as:

$\mathcal{L}_{\mathrm{total}}=\lambda_{\mathrm{subj}}\cdot\mathcal{L}_{\mathrm{subj}}^{\mathrm{total}}+\lambda_{\mathrm{distill}}\cdot\mathcal{L}_{\mathrm{distill}}^{\mathrm{total}},$
(14)

where we empirically set $\lambda_{\mathrm{subj}}=\mathrm{K}$ and
$\lambda_{\mathrm{distill}}=0.7$.

##  IV Experimental results

###  IV-A Datasets

We apply our method on the problem of motor imagery decoding and work on two
large datasets: Physionet [16, 42] and OpenBMI [32]. First we provide a brief
description of the datasets and then we describe the common signal
preprocessing pipeline that is followed for both datasets.

Physionet dataset: The dataset contains EEG recordings from $109$
participants, with trials that belong to $4$ classes: left-hand, right-hand
and feet imagery, as well as rest. The data are recorded with $64$ EEG
electrodes at a sampling frequency of $160$Hz. We choose to work on the two-
class problem of classifying left-hand versus right-hand imaginary movements,
discarding the data from the other classes. Similarly to other works [28, 7],
we also discard data from $6$ participants (specifically S088, S090, S092,
S100, S104 and S106) that have inconsistent sampling frequencies or trial
lengths. In our experiments we use the signals from all $64$ electrodes.

OpenBMI dataset: The data of OpenBMI correspond to trials of $2$ classes
(left-hand and right-hand imagery) collected from the EEG recordings of $54$
participants, with $62$ electrodes at a sampling frequency of $1000$Hz. Each
participant has data from two sessions and each session has two runs. The
first run of each session is done in an offline manner, i.e. without feedback.
The second run is done in an online manner, providing real-time visual
feedback to the user. When not otherwise stated, we use a subset of $20$
electrodes and we use the data from the two offline runs (i.e. the first run
of the first and second session) for each participant, following the default
settings of the MOABB [22] benchmark.

Preprocessing: Our preprocessing steps are the following: (i) we remove the
$50$Hz component through notch filtering (ii) we perform bandpass filtering
($4$Hz-$38$Hz) (iii) we resample the signals to $100$Hz and (iv) for each
trial, we crop a temporal window of $4$ seconds, starting from its onset
event. Upon obtaining the cropped trials, we use the session-wise covariance
matrices of the EEG signals and perform Riemannian Alignment on the time-
series of each trial, as in [57].

###  IV-B Comparison with other works and baseline

We compare our proposed method with four state-of-the-art techniques that
provide their source code, namely Adaptive Transfer Learning (ATL) [54],
EEGSym [38], TIDNet [28] and MIN2Net [4]. In order to fairly judge the impact
of our proposed methodology, we also implement two additional methods: a
single model baseline and an ensembling technique using the EEGNet
architecture. The baseline method (mentioned as “EEGNet-Single”) is a single
EEGNet model, that serves as a reference for the performance of an EEGNet
architecture without ensembling. We implement the ensembling technique by
training multiple individual EEGNet models. During inference, we fuse their
predictions through a simple averaging operation, to obtain the final
prediction. In essence, this method (mentioned as “EEGNet-Ensemble”)
represents a post-training model ensemble. In the supplementary material, we
provide more details about the implementations of all the aforementioned
methods.

###  IV-C Evaluation settings

We perform evaluation in two ways: (i) in a 5-fold cross-validation (CV)
manner and (ii) in a Leave-One-Subject-Out (LOSO) manner. In the 5-fold CV
scenario, we split the subjects of our dataset into 5 disjoint folds and run 5
experiments. In each experiment, we use a different fold as our test set and
then assign 3 folds to our training set and the 1 remaining fold to our
validation set. In the LOSO scenario for a dataset with $\mathrm{N}$ subjects,
we run $\mathrm{N}$ experiments where in the $n$-th experiment we use the data
of the $n$-th subject as our test set. In each experiment, we split the
remaining $\mathrm{N}-1$ subjects into our training and validation set.
Specifically, we assign $80\%$ of these subjects to the training set and the
rest $20\%$ to the validation set of the experiment. In both CV and LOSO
scenarios, the reported accuracy is the average of the test accuracies across
all experiments.

###  IV-D Training details

We train all models (i.e. our proposed method, the single model baseline and
the model ensembling method) for $120$ epochs with a batch size of $64$. We
use a Stochastic Gradient Descent (SGD) optimizer, setting the momentum to
$0.9$ and weight decay to $0.01$. We initialize the learning rate at $0.01$
for the first $60$ epochs and then decrease it to $0.002$ for the remaining
$60$ epochs.

###  IV-E Results (5-fold cross-validation)

In the first part of our experimental analysis we evaluate against methods
that provide source code, under a 5-fold cross-validation scenario, without
any model adaptation on test data or pretraining on external datasets. We note
that these experiments are performed using exactly the same train, validation
and test splits, the same trial length and the same number of electrodes for
all methods (except for the method of EEGSym [38] that has an architectural
requirement of 16 electrodes). Having the same experimental settings enables
us to fairly judge the performance of all methods.

Table I shows the results of the methods trained on Physionet dataset with
5-fold cross-validation. Concerning our baseline method, we observe that a
single EEGNet model achieves an accuracy of $82.09\%$. The standard model
ensembling technique of EEGNet-Ensemble reaches an accuracy of $84.56\%$ when
fusing eight individual EEGNet models. Our proposed method presents a
substantial boost of $+1.80\%$ over the standard ensemble scenario, reaching
an accuracy of $86.36\%$ when we use seven models at the first stage of our
architecture. The model of EEGSym [38] achieves an accuracy of $83.91\%$,
using $\sim 10\times$ more trainable parameters than the best performing
architecture of our proposed method. EEGSym without pretraining on external
data, performs worse than both our method and the standard model ensemble.
Regarding the method of TIDNet [28], the accuracy of $82.19\%$ is similar to
that of our EEGNet-Single baseline model.

Table II shows the results of our 5-fold cross-validation experiment on
OpenBMI for various methods. The baseline model of EEGNet-Single achieves an
accuracy of $78.31\%$ and the method of EEGNet-Ensemble provides a small boost
of $+0.67\%$ when using eight individual EEGNet models, leading to an accuracy
of $78.98\%$. Our proposed method performs superiorly, yielding an accuracy of
$79.73\%$ when using three first stage networks. The method of MIN2Net [4] has
a low performance, with an accuracy of $69.44\%$. Regarding the method of ATL
[54], the accuracy of $77.52\%$ falls behind the results of both our proposed
method and our baseline, using $\sim 60\times$ more trainable parameters than
our proposed method. Our results show that a simple ensemble architecture
trained with a curriculum learning scheme and an auxiliary loss can achieve
high cross-subject generalization, without any adaptation on test data or
complex model architecture.

Regarding the reported results of our proposed method in Table I and Table II,
we note that the optimal number of first stage feature extractors $\mathrm{K}$
is inferred from the accuracy on the validation set. We provide more details
about the impact of $\mathrm{K}$ on the performance of our architecture in
Subsection IV-G (Table V and Table VI). Similarly, regarding the reported
results of the EEGNet-Ensemble method in Table I and Table II, the optimal
number of individual EEGNet models within an ensemble is chosen based on the
validation accuracy. We elaborate on the results of the EEGNet-Ensemble method
in the supplementary material.

TABLE I: Performance of various methods on Physionet dataset, under 5-fold CV evaluation settings. The best accuracy is highlighted with bold. Method | Parameters |  Accuracy (%)
---|---|---
EEGNet-Single | 2.5K | 82.09
EEGNet-Ensemble, 8 models | 20.0K | 84.56
EEGSym [38] | 147.8K | 83.91
TIDNet [28] | 694.2K | 82.19
Ours, $\mathrm{K}$=7 | 15.7K | $\mathbf{86.36}$
TABLE II: Performance of various methods on OpenBMI dataset, under 5-fold CV evaluation settings. The best accuracy is highlighted with bold. Method | Parameters |  Accuracy (%)
---|---|---
EEGNet-Single | 1.8K | 78.31
EEGNet-Ensemble, 8 models | 14.3K | 78.98
MIN2Net [4] | 37.1K | 69.44
ATL [54] | 278.8K | 77.52
Ours, $\mathrm{K}$=3 | 4.6K | $\mathbf{79.73}$

###  IV-F Results (LOSO)

In this experiment, we compare our method against other state-of-the-art works
that report LOSO results on Physionet and OpenBMI. We note that we mention the
results of these methods as reported in their original works, ensuring that
they do not utilise labelled data from the test subjects.

The results on Physionet dataset are shown in Table III. The method of EEGSym
achieves state-of-the-art performance reaching an accuracy of $88.56\%$.
EEGSym performs transfer learning by pretraining on four external datasets,
which proves to be highly valuable. Our proposed method is the best performing
model among the works that do not train on external data. We outperform the
method of [7] that trains separate convolutional layers for each training
subject. This further validates the existence of more efficient and accurate
alternatives to complex deep architectures and the incorporation of subject-
specific components.

The results on OpenBMI dataset are shown in Table IV. Our method presents
state-of-the-art performance, scoring an accuracy of $85.07\%$ when using all
62 electrodes of OpenBMI and having $\mathrm{K}=4$ first stage models. We
outperform all other techniques, including the method of EEGSym that employs
pretraining on external data. The geometric deep learning technique of TSMNet
[26] and the algorithm of [30] that trains a convolutional architecture on
spectral-spatial inputs, present an accuracy gap of more than $\sim 10\%$ from
the methods of ATL, EEGSym and our technique. This indicates that deep
architectures operating on covariance matrices of EEG time-series (e.g. [26]
and [30]), are generally less suitable for cross-subject MI decoding, compared
to architectures that operate on raw EEG time-series.

TABLE III: Comparison with other state-of-the-art methods on Physionet dataset
with LOSO evaluation settings. The best accuracy is highlighted with bold.
*: pretrained on external data Method | Parameters |  Accuracy (%)
---|---|---
OPS [49] \- Human Neurosc. 2020 | N/A | 67.00
Causal Viewpoint [7] \- ICLRW 2022 | N/A | 83.90
EEGSym* [38] \- TNSRE 2022 | 147K | $\mathbf{88.56}$
Ours, $\mathrm{K}$=7 | 15.7K | 85.82
TABLE IV: Comparison with other state-of-the-art methods on OpenBMI dataset
with LOSO evaluation settings. The best accuracy is highlighted with bold.
*: pretrained on external data Method | Parameters |  Accuracy (%)
---|---|---
MIN2Net [4] \- TBME 2022 | 37.1K | 72.03
Mutual Inf. [23] \- TNNLS 2021 | N/A | 73.32
Spectral-Spatial [30] \- TNNLS 2019 | 77M | 74.15
TSMNet [26] \- NeurIPS 2022 | 4.5K | 74.60
ATL [54] \- Neural Networks 2021 | 305K | 84.19
EEGSym* [38] \- TNSRE 2022 | 147K | 84.72
Ours, $\mathrm{K}$=4 | 8.7K | $\mathbf{85.07}$

###  IV-G Ablation studies

In our ablation studies we investigate the impact of three components on the
performance of our ensemble architecture. The first component is the number of
first stage models $\mathrm{K}$ in the architecture. The second component is
the loss $\mathcal{L}_{\mathrm{subj}}^{\mathrm{total}}$, that materializes our
curriculum learning scheme. The third component is the distillation loss
$\mathcal{L}_{\mathrm{distill}}^{\mathrm{total}}$ that enables collaborative
training. We concurrently explore the effects of all these component choices,
performing a sweep over the hyperparameter $\mathrm{K}$ and trying
combinations of our loss terms.

Our first set of experiments (denoted as “$\mathcal{L}_{\texttt{CE}}$”)
corresponds to the scenario of training a model ensemble architecture (as
described in Sec. III-A2), i.e. without curriculum learning and without our
distillation loss. In our second set of experiments (denoted as
“$\mathcal{L}_{\mathrm{subj}}$”) we train our architecture with ensemble
curriculum learning, as described in Sec. III-B, i.e. without our distillation
loss. In the third experimental run (denoted as
“$\mathcal{L}_{\mathrm{total}}$”) we apply our entire method (i.e. using both
subject-weighted loss and intra-ensemble distillation loss), training our
architecture with the loss of Eq. 14. All experiments are performed with a
5-fold cross-validation setting.

The results of our ablation study on Physionet dataset are shown in Table V.
We observe a general trend of increasing accuracy for all our experimental
sets, as $\mathrm{K}$ increases up to the value of $7$ (further increasing
$\mathrm{K}$ does not yield performance improvements). The only exception is
the case where we train our architecture without curriculum learning (i.e.
first row in Table V), where the accuracy saturates at $\mathrm{K}=6$. This
indicates that training multiple feature extractors by equally fitting them to
the entire training set, is a suboptimal approach of training on multiple
source domains. Thus, applying our curriculum learning scheme through
$\mathcal{L}_{\mathrm{subj}}$ to induce diversity in the feature extractors,
is a straightforward step. The results of the second row in Table V verify the
positive impact of curriculum learning in our ensemble architecture. In some
cases (i.e. when $\mathrm{K}=3$, $\mathrm{K}=4$ and $\mathrm{K}=6$) curriculum
learning provides negligible accuracy gains. When further incorporating our
distillation loss in the total optimization objective of our architecture
(i.e. third row in Table V), we get additional accuracy boosts, except for the
cases of $\mathrm{K}=2$ and $\mathrm{K}=4$. The beneficial effect of
regulating the balance between feature diversity and model generalization
through our distillation loss, is higher in the cases of $\mathrm{K}=6$ and
$\mathrm{K}=7$ where the accuracy boosts are $+0.42\%$ and $+0.68\%$
respectively. This finding is particularly interesting, showing that the
combination of our two loss terms can increase the performance of model
ensembles, even when using many feature extraction models. On the contrary, an
ensemble architecture trained solely with the standard cross-entropy loss, is
more prone to performance saturation.

The results of our ablation study on OpenBMI dataset are shown in Table VI.
The standard ensemble architecture trained without curriculum learning (i.e.
first row in Table VI) achieves a maximum accuracy of $79.24\%$ when
$\mathrm{K}=5$. By using our curriculum learning scheme, we improve the
accuracy of our architecture in four out of six cases, achieving a maximum
accuracy of $79.58\%$ when $\mathrm{K}=3$. The incorporation of our
distillation loss term in the total loss of our architecture (i.e. third row
in Table VI) provides consistent improvements in all cases. Our best model has
an accuracy of $79.73\%$ when $\mathrm{K}=3$, with a boost of $0.65\%$ over
its corresponding standard ensemble model.

TABLE V: Ablation study on Physionet dataset with 5-fold CV evaluation settings. Rows correspond to experiment sets done with different optimization objectives. Columns correspond to the number of first stage models ($\mathrm{K}$) in our architecture. The best accuracy of each row is highlighted with bold. Loss terms | Accuracy (%)
---|---
K=2 | K=3 | K=4 | K=5 | K=6 | K=7
$\mathcal{L}_{\texttt{CE}}$ | 83.34 | 84.70 | 84.97 | 84.93 | $\mathbf{85.53}$ | 85.34
$\mathcal{L}_{\mathrm{subj}}$ | 84.38 | 84.72 | 85.10 | 85.40 | 85.62 | $\mathbf{85.68}$
$\mathcal{L}_{\mathrm{total}}$ | 83.76 | 84.78 | 85.02 | 85.48 | 86.04 | $\mathbf{86.36}$
TABLE VI: Ablation study on OpenBMI dataset with 5-fold CV evaluation settings. Rows correspond to experiment sets done with different optimization objectives. Columns correspond to the number of first stage models ($\mathrm{K}$) in our architecture. The best accuracy of each row is highlighted with bold. Loss terms | Accuracy (%)
---|---
K=2 | K=3 | K=4 | K=5 | K=6 | K=7
$\mathcal{L}_{\texttt{CE}}$ | 79.15 | 79.08 | 78.96 | $\mathbf{79.24}$ | 78.94 | 79.20
$\mathcal{L}_{\mathrm{subj}}$ | 79.02 | $\mathbf{79.58}$ | 79.13 | 79.15 | 79.01 | 79.31
$\mathcal{L}_{\mathrm{total}}$ | 79.25 | $\mathbf{79.73}$ | 79.53 | 79.46 | 79.10 | 79.66

##  V Conclusion

In this work, we propose a method for cross-subject motor imagery decoding
that leverages the combined strengths of model ensembling, curriculum learning
and collaborative training. We design an ensemble architecture that is trained
end-to-end in a single phase. We show that our curriculum training scheme can
induce diversity to the feature extraction models of our architecture,
improving its performance over standard ensembling. Our method also benefits
from the exchange of knowledge between the models of our ensemble, that occurs
through our auxiliary distillation loss. We conduct experiments on the
datasets of Physionet [16] and OpenBMI [32], totalling more than 150
participants, and demonstrate state-of-the-art results. Our proposed method
outperforms other approaches that try to tackle MI decoding using complex
networks [54, 28], multi-task learning [4], geometric deep learning [26],
subject-specific layers [7] or pretraining on multiple external datasets [38].
Our work highlights the importance of feature diversity as a property of model
ensembles, paving the way for robust EEG-based domain generalization
techniques.

## References

  * [1] R. Abiri, S. Borhani, E. W. Sellers, Y. Jiang, and X. Zhao.  A comprehensive review of eeg-based brain–computer interface paradigms.  Journal of neural engineering, 16(1):011001, 2019.
  * [2] G. A. Altuwaijri, G. Muhammad, H. Altaheri, and M. Alsulaiman.  A multi-branch convolutional neural network with squeeze-and-excitation attention blocks for eeg-based motor imagery signals classification.  Diagnostics, 12(4):995, 2022.
  * [3] K. K. Ang, Z. Y. Chin, H. Zhang, and C. Guan.  Filter bank common spatial pattern (fbcsp) in brain-computer interface.  In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence), pages 2390–2397. IEEE, 2008\.
  * [4] P. Autthasan, R. Chaisaen, T. Sudhawiyangkul, P. Rangpong, S. Kiatthaveephong, N. Dilokthanakul, G. Bhakdisongkhram, H. Phan, C. Guan, and T. Wilaiprasitporn.  Min2net: End-to-end multi-task learning for subject-independent motor imagery eeg classification.  IEEE Transactions on Biomedical Engineering, 69(6):2105–2118, 2021\.
  * [5] S. Bakas, S. Ludwig, K. Barmpas, M. Bahri, Y. Panagakis, N. Laskaris, D. A. Adamos, and S. Zafeiriou.  Team cogitat at neurips 2021: Benchmarks for eeg transfer learning competition.  arXiv preprint arXiv:2202.03267, 2022.
  * [6] P. D. E. Baniqued, E. C. Stanyer, M. Awais, A. Alazmani, A. E. Jackson, M. A. Mon-Williams, F. Mushtaq, and R. J. Holt.  Brain–computer interface robotics for hand rehabilitation after stroke: A systematic review.  Journal of NeuroEngineering and Rehabilitation, 18(1):1–25, 2021\.
  * [7] K. Barmpas, Y. Panagakis, D. Adamos, N. Laskaris, and S. Zafeiriou.  A causal viewpoint on motor-imagery brainwave decoding.  In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
  * [8] H. Berger.  Über das elektroenkephalogramm des menschen.  Archiv für psychiatrie und nervenkrankheiten, 87(1):527–570, 1929.
  * [9] Y. Bian and H. Chen.  When does diversity help generalization in classification ensembles?  IEEE Transactions on Cybernetics, 2021.
  * [10] B. Blankertz, R. Tomioka, S. Lemm, M. Kawanabe, and K.-R. Muller.  Optimizing spatial filters for robust eeg single-trial analysis.  IEEE Signal processing magazine, 25(1):41–56, 2007.
  * [11] L. Breiman.  Bagging predictors.  Machine learning, 24(2):123–140, 1996.
  * [12] D. D. Chakladar, S. Dey, P. P. Roy, and M. Iwamura.  Eeg-based cognitive state assessment using deep ensemble model and filter bank common spatial pattern.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 4107–4114. IEEE, 2021.
  * [13] X. Chen and K. He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [14] I. Dolzhikova, B. Abibullaev, R. Sameni, and A. Zollanvari.  An ensemble cnn for subject-independent classification of motor imagery-based eeg.  In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 319–324. IEEE, 2021\.
  * [15] Y. Du and J. Liu.  Ienet: a robust convolutional neural network for eeg based brain-computer interfaces.  Journal of Neural Engineering, 2022.
  * [16] A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley.  Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals.  circulation, 101(23):e215–e220, 2000.
  * [17] I. Gulrajani and D. Lopez-Paz.  In search of lost domain generalization.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [18] L. K. Hansen and P. Salamon.  Neural network ensembles.  IEEE transactions on pattern analysis and machine intelligence, 12(10):993–1001, 1990.
  * [19] H. He and D. Wu.  Transfer learning for brain–computer interfaces: A euclidean space data alignment approach.  IEEE Transactions on Biomedical Engineering, 67(2):399–410, 2019\.
  * [20] F. A. Heilmeyer, R. T. Schirrmeister, L. D. Fiederer, M. Volker, J. Behncke, and T. Ball.  A large-scale evaluation framework for eeg deep learning architectures.  In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 1039–1045. IEEE, 2018.
  * [21] E. R. M. Heremans, H. Phan, P. Borzée, B. Buyse, D. Testelmans, and M. De Vos.  From unsupervised to semi-supervised adversarial domain adaptation in eeg-based sleep staging.  Journal of Neural Engineering, 2022.
  * [22] V. Jayaram and A. Barachant.  Moabb: trustworthy algorithm benchmarking for bcis.  Journal of neural engineering, 15(6):066011, 2018.
  * [23] E. Jeon, W. Ko, J. S. Yoon, and H.-I. Suk.  Mutual information-driven subject-invariant and class-relevant deep representation learning in bci.  IEEE Transactions on Neural Networks and Learning Systems, 2021\.
  * [24] C. Jeunet, F. Lotte, M. Hachet, and B. N’Kaoua.  Impact of cognitive and personality profiles on motor-imagery based brain-computer interface-controlling performance.  In 17th World Congress of Psychophysiology (IOP2014), 2014.
  * [25] C. Jeunet, B. N’Kaoua, S. Subramanian, M. Hachet, and F. Lotte.  Predicting mental imagery-based bci performance from personality, cognitive profile and neurophysiological patterns.  PloS one, 10(12):e0143962, 2015.
  * [26] R. J. Kobler, J.-i. Hirayama, Q. Zhao, and M. Kawanabe.  Spd domain-specific batch normalization to crack interpretable unsupervised domain adaptation in eeg.  arXiv preprint arXiv:2206.01323, 2022.
  * [27] Z. J. Koles.  The quantitative extraction and topographic mapping of the abnormal components in the clinical eeg.  Electroencephalography and clinical Neurophysiology, 79(6):440–447, 1991.
  * [28] D. Kostas and F. Rudzicz.  Thinker invariance: enabling deep neural networks for bci across more people.  Journal of Neural Engineering, 17(5):056008, 2020.
  * [29] L. Kuhlmann, P. Karoly, D. R. Freestone, B. H. Brinkmann, A. Temko, A. Barachant, F. Li, G. Titericz Jr, B. W. Lang, D. Lavery, et al.  Epilepsyecosystem. org: crowd-sourcing reproducible seizure prediction with long-term human intracranial eeg.  Brain, 141(9):2619–2630, 2018.
  * [30] O.-Y. Kwon, M.-H. Lee, C. Guan, and S.-W. Lee.  Subject-independent brain–computer interfaces based on deep convolutional neural networks.  IEEE transactions on neural networks and learning systems, 31(10):3839–3852, 2019.
  * [31] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance.  Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces.  Journal of neural engineering, 15(5):056013, 2018.
  * [32] M.-H. Lee, O.-Y. Kwon, Y.-J. Kim, H.-K. Kim, Y.-E. Lee, J. Williamson, S. Fazli, and S.-W. Lee.  Eeg dataset and openbmi toolbox for three bci paradigms: An investigation into bci illiteracy.  GigaScience, 8(5):giz002, 2019.
  * [33] J. Li, S. Qiu, C. Du, Y. Wang, and H. He.  Domain adaptation for eeg emotion recognition based on latent representation similarity.  IEEE Transactions on Cognitive and Developmental Systems, 12(2):344–353, 2019.
  * [34] M. Lotze and L. G. Cohen.  Volition and imagery in neurorehabilitation.  Cognitive and behavioral neurology, 19(3):135–140, 2006.
  * [35] B.-Q. Ma, H. Li, W.-L. Zheng, and B.-L. Lu.  Reducing the subject variability of eeg signals with adversarial domain generalization.  In International Conference on Neural Information Processing, pages 30–42. Springer, 2019.
  * [36] W. Ma, H. Xue, X. Sun, S. Mao, L. Wang, Y. Liu, Y. Wang, and X. Lin.  A novel multi-branch hybrid neural network for motor imagery eeg signal classification.  Biomedical Signal Processing and Control, 77:103718, 2022.
  * [37] S.-H. Park, D. Lee, and S.-G. Lee.  Filter bank regularized common spatial pattern ensemble for small sample motor imagery classification.  IEEE Transactions on Neural Systems and Rehabilitation Engineering, 26(2):498–505, 2017.
  * [38] S. Pérez-Velasco, E. Santamaría-Vázquez, V. Martínez-Cagigal, D. Marcos-Martínez, and R. Hornero.  Eegsym: Overcoming inter-subject variability in motor imagery based bcis with deep learning.  IEEE Transactions on Neural Systems and Rehabilitation Engineering, 30:1766–1775, 2022.
  * [39] C. Reuben, P. Karoly, D. R. Freestone, A. Temko, A. Barachant, F. Li, G. Titericz Jr, B. W. Lang, D. Lavery, K. Roman, et al.  Ensembling crowdsourced seizure prediction algorithms using long-term human intracranial eeg.  Epilepsia, 61(2):e7–e12, 2020.
  * [40] S. Rimbert, D. Trocellier, and F. Lotte.  Is event-related desynchronization variability correlated with bci performance?  In 2022 IEEE International Conference on Metrology for eXtended Reality, Artificial Intelligence, and Neural Engineering, 2022.
  * [41] C. Sannelli, C. Vidaurre, K.-R. Müller, and B. Blankertz.  Csp patches: an ensemble of optimized spatial filters. an evaluation study.  Journal of Neural Engineering, 8(2):025012, 2011.
  * [42] G. Schalk, D. J. McFarland, T. Hinterberger, N. Birbaumer, and J. R. Wolpaw.  Bci2000: a general-purpose brain-computer interface (bci) system.  IEEE Transactions on biomedical engineering, 51(6):1034–1043, 2004\.
  * [43] R. T. Schirrmeister, J. T. Springenberg, L. D. J. Fiederer, M. Glasstetter, K. Eggensperger, M. Tangermann, F. Hutter, W. Burgard, and T. Ball.  Deep learning with convolutional neural networks for eeg decoding and visualization.  Human brain mapping, 38(11):5391–5420, 2017.
  * [44] A. Soria-Frisch.  A critical review on the usage of ensembles for bci.  Towards Practical Brain-Computer Interfaces, pages 41–65, 2012\.
  * [45] M. Spape, K. Davis, L. Kangassalo, N. Ravaja, Z. Sovijarvi-Spape, and T. Ruotsalo.  Brain-computer interface for generating personally attractive images.  IEEE Transactions on Affective Computing, 1(1), 2021.
  * [46] V. Vapnik.  Statistical learning theory.  Wiley, 1998.
  * [47] J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu.  Generalizing to unseen domains: A survey on domain generalization.  IEEE Transactions on Knowledge and Data Engineering, 2022.
  * [48] X. Wei, P. Ortega, and A. A. Faisal.  Inter-subject deep transfer learning for motor imagery eeg decoding.  In 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER), pages 21–24. IEEE, 2021.
  * [49] L. Xu, M. Xu, Y. Ke, X. An, S. Liu, and D. Ming.  Cross-dataset variability problem in eeg decoding with deep learning.  Frontiers in human neuroscience, 14:103, 2020.
  * [50] M. Xu, J. Han, Y. Wang, T.-P. Jung, and D. Ming.  Implementing over 100 command codes for a high-speed hybrid brain-computer interface using concurrent p300 and ssvep features.  IEEE Transactions on Biomedical Engineering, 67(11):3073–3082, 2020\.
  * [51] D. Yadav, S. Yadav, and K. Veer.  A comprehensive assessment of brain computer interfaces: Recent trends and challenges.  Journal of Neuroscience Methods, 346:108918, 2020.
  * [52] H. Yuan and B. He.  Brain–computer interfaces using sensorimotor rhythms: current state and future perspectives.  IEEE Transactions on Biomedical Engineering, 61(5):1425–1435, 2014\.
  * [53] H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz.  mixup: Beyond empirical risk minimization.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018.
  * [54] K. Zhang, N. Robinson, S.-W. Lee, and C. Guan.  Adaptive transfer learning for eeg motor imagery classification with deep convolutional neural network.  Neural Networks, 136:1–10, 2021.
  * [55] K. Zhou, Y. Yang, Y. Qiao, and T. Xiang.  Domain adaptive ensemble learning.  IEEE Transactions on Image Processing, 30:8008–8018, 2021.
  * [56] H. Zhu, D. Forenzo, and B. He.  On the deep learning models for eeg-based brain-computer interface using motor imagery.  IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022.
  * [57] G. Zoumpourlis and I. Patras.  Covmix: Covariance mixing regularization for motor imagery decoding.  In 2022 10th International Winter Conference on Brain-Computer Interface (BCI), pages 1–7. IEEE, 2022.

##  VI Supplementary Material

In the supplementary material, we first provide details about our proposed
curriculum learning scheme in Section VI-A. Then, in the following sections we
describe the architecture and training process of various methods, namely: i)
a baseline method of a single EEGNet [31] model (Section VI-B), ii) two model
ensembling methods (Section VI-C) and iii) our proposed model ensemble
architecture (Section VI-D). Moreover, in Section VI-E we present experimental
results of the two model ensembling methods. These results can serve as
additional references against which the results of our proposed method can be
compared. Finally, in Section VI-F we provide details on the training process
that we follow in order to reproduce four state-of-the-art techniques for
motor imagery decoding, namely Adaptive Transfer Learning (ATL) [54], EEGSym
[38], TIDNet [28] and MIN2Net [4].

###  VI-A Ensemble curriculum learning

Figure 3: Indicative illustration of our curriculum learning scheme. In this
example, we are provided with a dataset $\mathbf{\mathcal{D}}$ containing EEG
data from 10 human subjects and our proposed model ensemble architecture
consists of $\mathrm{K}=3$ models.

In Fig. 2 of our main paper we present the training pipeline of our
architecture that employs an ensemble curriculum learning scheme and a
distillation loss. We also describe the subject-weighted loss that
materializes our curriculum learning scheme in Subsection III-B of our main
paper and present its formula in Eq. 9 of our main paper. To allow a better
understanding of the coefficient $\beta(\mathbf{x},\mathrm{k})$ that is
involved in Eq. 9 of our main paper, we show an overview of our curriculum
learning scheme in Fig. 3. Specifically, we consider an example where we are
provided with a dataset $\mathbf{\mathcal{D}}$ containing EEG data from 10
human subjects and our proposed model ensemble architecture consists of
$\mathrm{K}=3$ models. The dataset
$\mathbf{\mathcal{D}}=\\{\mathbf{\mathcal{D}}_{\mathrm{1}},\mathbf{\mathcal{D}}_{\mathrm{2}},\ldots,\mathbf{\mathcal{D}}_{\mathrm{10}}\\}$
containing the sub-datasets of 10 subjects is split into $\mathrm{K}=3$ non-
overlapping subsets: $\mathbf{\mathcal{S}}_{\mathrm{1}}$,
$\mathbf{\mathcal{S}}_{\mathrm{2}}$ and $\mathbf{\mathcal{S}}_{\mathrm{3}}$.
Our curriculum learning scheme aims to make the $k$-th model to specialize on
the subjects belonging to subset $\mathbf{\mathcal{S}}_{\mathrm{k}}$ of
$\mathbf{\mathcal{D}}$, while still training on the whole dataset
$\mathbf{\mathcal{D}}$. This is done using the coefficient
$\beta(\mathbf{x},\mathrm{k})$ that controls the loss contribution of a
training sample $\mathbf{x}$ on the weight updating process for the $k$-th
model. To achieve specialization on the samples of
$\mathbf{\mathcal{S}_{\mathrm{k}}}$, when
$~{}\mathbf{x}\in\mathbf{\mathcal{S}_{\mathrm{k}}}$, we set
$\beta(\mathbf{x},\mathrm{k})=1$ throughout the training process. We also want
to train the $k$-th model on the rest of the subjects of
$\mathbf{\mathcal{D}}$ (i.e. those that do not belong to
$\mathbf{\mathcal{S}_{\mathrm{k}}}$), albeit with a progressively decreasing
loss contribution over time. For this reason, when
$~{}\mathbf{x}\notin\mathbf{\mathcal{S}_{\mathrm{k}}}$ we set
$\beta(\mathbf{x},\mathrm{k})=\alpha$, with $\alpha$ decaying from 1 to 0
while training progresses.

###  VI-B Single model baseline

EEGNet-Single: Our baseline method (mentioned as “EEGNet-Single”) is a single
EEGNet model, trained in the entire training set as described in Subsection
III-A1 of our main paper. The detailed architecture of EEGNet is shown in
Table VII.

###  VI-C Model ensembling methods

EEGNet-Ensemble: We implement the first ensembling method by training multiple
individual EEGNet models in the entire training set. During inference, we fuse
their predictions through a simple averaging operation to obtain the final
prediction. In essence, this ensembling method (mentioned as “EEGNet-
Ensemble”) represents a post-training model ensemble.

EEGNet-Bagging: We implement the second ensembling method by training multiple
individual EEGNet models in random subsets of the training set. Specifically,
we train each individual EEGNet model on $85\%$ of all the available training
subjects. We choose the subjects to be kept for training in each experiment,
by simply performing random subsampling. During inference, we fuse the
predictions of all models through an averaging operation to obtain the final
prediction. This method (mentioned as “EEGNet-Bagging”) represents the well-
known ensembling technique of bootstrap aggregating [11]. We note that we
report results using this ensembling method in this supplementary material,
for the sake of a more complete evaluation, albeit not including this method
in the main manuscript of our work.

###  VI-D Proposed model

Our proposed model ensemble architecture, described in Subsection III-A2 of
our main paper, consists of two stages and uses EEGNet as its elementary
component. The first stage of our proposed architecture contains multiple
feature extractors in parallel, with each first stage network producing a
feature vector. Considering the EEGNet architecture that is presented in Table
VII, each first stage network of our architecture contains all the layers up
to (and including) the feature flattening layer of EEGNet. The second stage
has a single shared classification head that computes the class-wise
prediction scores for each feature vector originating from the first stage.
Based on the EEGNet layers that are presented in Table VII, the second stage
of our architecture corresponds to the last layer of EEGNet, i.e. a single
fully connected layer that performs classification.

TABLE VII: Architecture of a single EEGNet model. The input of the model has a shape of $\mathrm{B}\times 1\times\mathrm{C}\times\mathrm{T}$, where $\mathrm{B}$ is the batch size, $\mathrm{C}$ is the number of EEG electrodes and $\mathrm{T}$ is the number of samples in the temporal dimension. The output of the model has a shape of $\mathrm{B}\times 2$, in the case of two output classes. Layer | Input shape | Output shape
---|---|---
Dropout (p=0.4) | $\mathrm{B}\times\mathrm{1}\times\mathrm{C}\times\mathrm{T}$ | $\mathrm{B}\times\mathrm{1}\times\mathrm{C}\times\mathrm{T}$
| Temporal Convolution, 8 filters
---
kernel=(1, 64), stride=(1, 1), pad=(0, 32)
$\mathrm{B}\times\mathrm{1}\times\mathrm{C}\times\mathrm{T}$ | $\mathrm{B}\times\mathrm{8}\times\mathrm{C}\times\mathrm{T}$
| Spatial Convolution, 16 filters
---
kernel=(1, C), stride=(1, 1), pad=(0, 0)
max. weight norm=1.0
$\mathrm{B}\times\mathrm{8}\times\mathrm{C}\times\mathrm{T}$ | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T}$
| Temporal Pooling
---
kernel=(1, 4), stride=(1, 4), pad=(0, 0)
$\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T}$ | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T}$
Batch Normalization 2D | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$ | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$
ELU activation | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$ | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$
Dropout (p=0.1) | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$ | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$
| Separable Convolution Depthwise, 16 filters, 16 groups
---
kernel=(1, 16), stride=(1, 1), pad=(0, 8)
$\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$ | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$
| Separable Convolution Pointwise, 16 filters, 16 groups
---
kernel=(1, 1), stride=(1, 1), pad=(0, 0)
$\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$ | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$
Batch Normalization 2D | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$ | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$
ReLU activation | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$ | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$
| Temporal Pooling
---
kernel=(1, 8), stride=(1, 8), pad=(0, 0)
$\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/4}$ | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/32}$
Flatten | $\mathrm{B}\times\mathrm{16}\times\mathrm{1}\times\mathrm{T/32}$ | $\mathrm{B}\times\mathrm{T/2}$
Fully Connected | $\mathrm{B}\times\mathrm{T/2}$ | $\mathrm{B}\times\mathrm{2}$

###  VI-E Experimental results (5-fold cross-validation)

In the first part of the experimental analysis presented in our main paper, we
compare our proposed method against our single model baseline as well as the
standard model ensembling method, under a 5-fold cross-validation scenario. We
note that these experiments are performed using exactly the same train,
validation and test splits. Here we provide additional results, presenting the
performance of the EEGNet-Ensemble and EEGNet-Bagging methods as the number
$\mathrm{M}$ of individual EEGNet models varies.

EEGNet-Ensemble: In Table VIII we show the cross-subject performance of
EEGNet-Ensemble on Physionet dataset under a 5-fold cross-validation scenario,
when using from 2 to 9 EEGNet models. The best accuracy ($84.56\%$) is
achieved when fusing the predictions from 8 EEGNet models. In Table IX we show
the cross-subject performance of EEGNet-Ensemble on OpenBMI dataset under a
5-fold cross-validation scenario, when using from 2 to 9 EEGNet models. The
best accuracy ($78.98\%$) is achieved when fusing the predictions from 8
EEGNet models.

EEGNet-Bagging: In Table X we show the cross-subject performance of EEGNet-
Bagging on Physionet dataset under a 5-fold cross-validation scenario, when
using from 2 to 9 EEGNet models. The best accuracy ($84.81\%$) is achieved
when using the predictions from 8 EEGNet models. In Table XI we show the
cross-subject performance of EEGNet-Bagging on OpenBMI dataset under a 5-fold
cross-validation scenario, when using from 2 to 9 EEGNet models. The best
accuracy ($79.28\%$) is achieved when using the predictions from 7 EEGNet
models.

TABLE VIII: Performance of EEGNet-Ensemble on Physionet dataset with 5-fold
cross-validation evaluation settings. Columns correspond to the number of
individual EEGNet models ($\mathrm{M}$) that we use. The best accuracy is
highlighted with bold. Accuracy (%)
---
M=2 | M=3 | M=4 | M=5 | M=6 | M=7 | M=8 | M=9
82.99 | 84.01 | 84.10 | 84.21 | 84.04 | 84.26 | $\mathbf{84.56}$ | 84.47
TABLE IX: Performance of EEGNet-Ensemble on OpenBMI dataset with 5-fold cross-
validation evaluation settings. Columns correspond to the number of individual
EEGNet models ($\mathrm{M}$) that we use. The best accuracy is highlighted
with bold. Accuracy (%)
---
M=2 | M=3 | M=4 | M=5 | M=6 | M=7 | M=8 | M=9
78.95 | 78.91 | 78.86 | 78.93 | 78.91 | 78.95 | $\mathbf{78.98}$ | 78.97
TABLE X: Performance of EEGNet-Bagging on Physionet dataset with 5-fold cross-
validation evaluation settings. Columns correspond to the number of individual
EEGNet models ($\mathrm{M}$) that we use. The best accuracy is highlighted
with bold. Accuracy (%)
---
M=2 | M=3 | M=4 | M=5 | M=6 | M=7 | M=8 | M=9
83.05 | 84.22 | 84.17 | 84.49 | 84.68 | 84.73 | $\mathbf{84.81}$ | 84.74
TABLE XI: Performance of EEGNet-Bagging on OpenBMI dataset with 5-fold cross-
validation evaluation settings. Columns correspond to the number of individual
EEGNet models ($\mathrm{M}$) that we use. The best accuracy is highlighted
with bold. Accuracy (%)
---
M=2 | M=3 | M=4 | M=5 | M=6 | M=7 | M=8 | M=9
78.99 | 79.00 | 79.18 | 79.26 | 79.20 | $\mathbf{79.28}$ | 79.07 | 79.11

###  VI-F Training details of state-of-the-art methods

In our main paper, we compare our proposed method with four state-of-the-art
techniques that provide their source code, namely Adaptive Transfer Learning
(ATL) [54], EEGSym [38], TIDNet [28] and MIN2Net [4]. In the following
paragraphs, we briefly refer to the details of the training settings for each
technique, regarding our 5-fold cross-validation experiments. In general, we
keep the same hyperparameter choices with the original works, except from the
temporal duration of the trials, which we set equal to $4.0$ seconds for all
methods for fair comparison.

ATL: We use the official implementation222https://github.com/zhangks98/eeg-
adapt of ATL [54]. The method employs the Deep4Net [43] model architecture, as
defined in braindecode333https://github.com/braindecode/braindecode toolbox.
We perform subject-independent training, i.e. no data from the test subjects
are used during training. We train all models for 200 epochs with a batch size
of 16. We use an AdamW optimizer, with a learning rate of 0.01 and a weight
decay of 0.0005. We also use a temporal length of 4 seconds for all trials.

EEGSym: We use the official implementation444https://github.com/Serpeve/EEGSym
of EEGSym [38]. The method employs the custom model architecture that is
described in the original publication [38]. The model architecture does not
support an arbitrary number of input EEG electrode channels, restricting the
options to 8 or 16 electrodes. We choose to use 16 electrodes, to retain more
information in the input EEG signals. To ensure fair comparison with other
methods and our models, we do not use the model weights that are provided, as
they are obtained by pretraining on four external datasets. We perform
training for 500 epochs with a batch size of 32, and set the patience for
early stopping to 25 epochs. We use Cross-Entropy as the loss function and
employ an Adam optimizer with a learning rate of 0.001. We set the dropout
rate to 0.4 and use 24 filters per branch in the model architecture. We also
use a temporal length of 4 seconds for all trials.

TIDNet: We use the official implementation555https://github.com/SPOClab-
ca/ThinkerInvariance of TIDNet [28]. We train the TIDNet architecture for 30
epochs using a batch size of 16. We perform MixUp setting its hyperparameter
$\beta$ equal to 8.0, and employ Euclidean Alignment on the EEG time-series.
We use a temporal length of 4 seconds for all trials. We use the
Kullback–Leibler (KL) divergence as the loss function. In each epoch, we
perform evaluation using the Exponentially Weighted Moving Average (EWMA) of
the model weights from the last 5 epochs.

MIN2Net: We use the official implementation666https://github.com/IoBT-
VISTEC/MIN2Net of MIN2Net [4]. We train the MIN2Net model architecture for 200
epochs with a batch size of 100. We set the initial learning rate to 0.001 and
decay the learning rate by a factor of 0.5, with a patience of 20 epochs. The
minimum learning rate is set equal to 0.0001. We set the hyperparameters of
MIN2Net as follows: $\alpha=1.0$, $\beta_{1}=0.5$, $\beta_{2}=0.5$ and
$\beta_{3}=1.0$. We use a temporal length of 4 seconds for all trials.