# Deep comparisons of Neural Networks from the EEGNet family

Csaba Márton Köllőd  0000-0003-3817-6709 András Adolf  0000-0001-7855-1492
Gergely Márton  0000-0003-4359-0111 István Ulbert  0000-0001-9941-9159

###### Abstract

Most of the Brain-Computer Interface (BCI) publications, which propose
artificial neural networks for Motor Imagery (MI) Electroencephalography (EEG)
signal classification, are presented using one of the BCI Competition
datasets. However, these databases contain MI EEG data from less than or equal
to 10 subjects . In addition, these algorithms usually include only bandpass
filtering to reduce noise and increase signal quality. In this article, we
compared 5 well-known neural networks (Shallow ConvNet, Deep ConvNet, EEGNet,
EEGNet Fusion, MI-EEGNet) using open-access databases with many subjects next
to the BCI Competition 4 2a dataset to acquire statistically significant
results. We removed artifacts from the EEG using the FASTER algorithm as a
signal processing step. Moreover, we investigated whether transfer learning
can further improve the classification results on artifact filtered data. We
aimed to rank the neural networks; therefore, next to the classification
accuracy, we introduced two additional metrics: the accuracy improvement from
chance level and the effect of transfer learning. The former can be used with
different class-numbered databases, while the latter can highlight neural
networks with sufficient generalization abilities. Our metrics showed that the
researchers should not avoid Shallow ConvNet and Deep ConvNet because they can
perform better than the later published ones from the EEGNet family.

###### keywords:

BCI , EEG , Neural Networks , EEGNet

††journal: Biomedical Signal Processing and Control

[inst1]organization=Roska Tamás Doctoral School of Sciences and
Technology,city=Budapest, country=Hungary

[inst2]organization=Faculty of Information Technology and Bionics, Pázmány
Péter Catholic University,city=Budapest, country=Hungary

[inst3]organization=Cognitive Neuroscience and Psychology, Research Centre for
Natural Sciences,city=Budapest, country=Hungary

##  1 Introduction

Artificial Neural Networks made one of the earliest significant impacts in the
field of Brain-Computer Interfaces (BCI) when Schirrmeister et al. introduced
Deep ConvNet and Shallow ConvNet in 2017 schirrmeister_deep_2017 (1) for
electroencephalographic (EEG) signal classification. Since then, neural
networks have become one of the hottest topics in BCI literature.

BCIs are integrated systems that include software and hardware components. As
it is presented by Wolpaw et al. wolpaw_braincomputer_2002 (2), these systems
record bioelectrical signals from the brain, extract useful information from
the EEG-noise mixture, and convert them to computer commands. EEG is defined
as the postsynaptic membrane potential fluctuation of the neurons, recorded
from the surface of the head. Figure 1 presents the components of a BCI
System.

Figure 1: Components of a Brain-Computer Interface system

If a new system is developed for motor imagery (MI) signal classification, it
is often tested and compared with the previously published ones on one of the
BCI Competition databases: blankertz_bci_2004 (3, 4, 5, 6). However, these
datasets contain records from less than or equal to 10 subjects. Other open-
access databases have EEG records from more than 50 subjects but are avoided
mainly by the researchers. One is the MI EEG dataset on PhysioNet
goldberger_ary_l_physiobank_2000 (7) recorded by the BCI2000 software
schalk_bci2000_2004 (8), which includes EEG records from 109 subjects. The
other one was recorded utilizing the OpenBMI toolbox lee_eeg_2019 (9) and
contained data from 52 subjects. Each subject in this dataset participated in
two experimental days. In addition, we have recorded our dataset, which
included 25 experiments from 9 subjects kollod_ttk_2022 (10). We hypothesize
that databases with more than 20 experimental days are sufficient for BCI
system comparison.

Next to the offline comparisons, the Cybathlon competition
riener_cybathlon_2014 (11) was introduced. It aimed to investigate the
reliability of BCI systems working in real-time, out of the lab situation. 11
teams participated successfully in the BCI discipline of Cybathlon 2016
novak_benchmarking_2018 (12), and two published their concepts, training
protocols, and BCI Systems after the competition. perdikis_cybathlon_2018 (13,
14) As a continuation of this competition, the 2019 Cybathlon BCI Series and
the 2020 Cybathlon Global Edition were organized, from which multiple teams
shared their preparation and results. benaroch_long-term_2021 (15, 16, 17, 18,
19, 20)

Before neural networks, scientists intended to investigate and develop hand-
crafted feature extraction methods combined with simple classification
algorithms. Blankertz et al. blankertz_non-invasive_2007 (21) successfully
used the Common Spatial Patterns (CSP) algorithm with Linear Discriminant
Analysis (LDA) classifier to control the cursor in one dimension. Barachant et
al. barachant_riemannian_2010 (22) introduced Riemannian geometry for BCI with
an LDA classifier to successfully classify EEG covariance matrices. Lotte and
Guan lotte_regularizing_2011 (23) introduced a unifying theoretical framework
for regularizing the CSP and compared it with 10 other regularized versions of
the CSP algorithm. Another feature extraction algorithm, considering the CSP,
is the Filter Bank Common Spatial Pattern (FBCSP) with a Naïve Bayesian Parzen
Window classifier ang_filter_2012 (24), which was compared with the ConvNets
lawhern_eegnet_2018 (25, 1) on the BCI Competition IV 2a database. The winner
of the BCI discipline of the Cybathlon competitions used the power spectral
density of the EEG signals as a feature perdikis_cybathlon_2018 (13, 19) with
a Gaussian classifier.

With the introduction of the Deep and Shallow ConvNets, a new trend started in
BCI development. The focus had been shifted from hand-crafted features to
creating neural networks which not just classify the signal but also include
the feature extraction step. Lawhern et al. lawhern_eegnet_2018 (25)
introduced the EEGNet, which was inspired by former neural networks that were
designed for EEG signal processing, including MI-based BCIs
sakhavi_parallel_2015 (26, 1, 27, 28). It was demonstrated that EEGNet makes a
similar feature extraction as the FBCSP. This neural network inspired many
scientists, which resulted in many improved versions of the EEGNet
altaheri_physics-inform_2022 (29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
41, 42), creating a whole family of the neural network. Other publications
outside the EEGNet family dokur_classification_2021 (43, 44, 45, 46, 47, 48,
49) highlight the importance of neural network-based BCI research.

Along with the development of neural networks, scientists started
investigating the effect of transfer learning weiss_survey_2016 (50). This
method aims to transfer knowledge between two domains and increase the
classification’s accuracy. Khademi et al. khademi_transfer_2022 (51) used a
CNN-LSTM hybrid model, which was pretrained on the ILSVRC subset of the
ImageNet dataset to classify MI EEG signals. This way, they aimed to transfer
the knowledge of image classification and use it on spatial EEG images
generated with continuous wavelet transformation method using complex Morlet
mother wavelet. Another strategy is to utilize the entire EEG dataset and
combine cross-subject and within-subject training, as presented in
mattioli_1d_2022 (52, 49, 53). In this case, the knowledge is granted from
subjects not used in the test set of the neural network. The network is
pretrained on data from all but one subject, as it would in a cross-subject
training procedure. But then the data of the test subject is also split to
train and test set, as in the within-subject, and the training part is used
for fine-tuning the pretrained neural network. We selected the later version
of transfer learning because it is architecture independent and aimed to use
it after artifact filtering.

In this article, all the experiments were conducted on data purified from
artifacts because eye and muscle movement activity can distort the EEG signal
nolan_faster_2010 (54). Moreover, it was demonstrated that artifacts could
successfully be used for BCI purposes noboa_development_2021 (55); however, in
our perspective, an actual BCI does not depend on artifacts, only on pure
brain waves.

To reduce the computational time of the experiments, we have arbitrarily
selected Shallow and Deep ConvNet schirrmeister_deep_2017 (1) as predecessors
of EEGNet, the EEGNet itself lawhern_eegnet_2018 (25), the EEGNet Fusion
roots_fusion_2020 (41), and the MI-EEGNet riyad_mi-eegnet_2021 (39) from the
EEGNet family.

##  2 Materials and Methods

The databases and neural networks are presented in this section, with the
experimental setups and concepts. The code used in this study is available
under: https://github.com/kolcs/bionic_apps

###  2.1 Databases

In the following, we present the datasets which were used for the EEGNet
family comparisons. The databases were processed in an “independent days”
configuration, meaning that if a subject participated in an experiment
multiple times on different experimental days, the data were handled as if it
would have been recorded from various subjects. According to our knowledge,
EEG data can greatly be varied by many factors, such as recording setup,
period of the day, and mental state of the subjects. The latter could lead to
poorer performance if the data is merged concerning the subjects. With the
independent days configuration, we aimed to overcome this problem and extend
the number of subjects to strengthen the results of the statistical analyses.

####  2.1.1 Physionet

The open-access database PhysioNet (Goldberger Ary L. et al., 2000) is a
valuable resource for numerous physiological datasets. Among these datasets is
the EEG Motor Movement/Imagery Dataset, which was captured by Schalk et al.
schalk_bci2000_2004 (8) utilizing the BCI2000 paradigm control program. For
convenience, we will refer to this specific dataset as the Physionet database.
It encompasses four MI EEG signals obtained from 109 individuals: Left Hand,
Right Hand, Both Hands, and Both Legs. The MI periods are of 4-second duration
and are interspersed with 4-second long Rest periods. The recordings were
sampled at 160 Hz, over 64 channels, without using hardware filters.

Four subjects out of the 109 were dropped from the database before the
experiments. For subject 89 the labels were incorrect. In the case of subjects
88, 92, and 100 the timing was incorrect. The execution of the MI tasks and
the resting phases were 5.125 and 1.375 seconds, respectively. Moreover, the
sampling frequency was changed from 160 Hz to 128 Hz. Other articles, which
use the Physionet database fan_bilinear_2020 (56, 52, 41), also reported these
problems

####  2.1.2 Giga

Lee et al. lee_eeg_2019 (9) published an EEG dataset that included three
paradigms: MI, event-related potential, and steady-state visually evoked
potential. The experimental paradigms were conducted with the OpenMBI toolbox,
custom written in MATLAB. We selected the files corresponding to the MI EEG
paradigm from these three paradigms, which contains a 2-class classification
problem, where the tasks are Left Hand and Right Hand movement imagination.
The EEG signals were recorded with a 62-channeled BrainAmp amplifier system
with a sampling rate of 1000 Hz. 54 subjects participated in the experiments,
and each subject was present on two different experimental days. Therefore,
concerning our independent days configuration, this dataset contains 108
subjects. To shrink the size of the raw EEG files, we resampled the data to a
500 Hz sampling frequency.

####  2.1.3 BCI Competition IV 2a

Tangermann et al. tangermann_review_2012 (6) published the well-known and
highly used BCI competition IV database, which includes 5 sub-datasets with
different paradigms and challenges. This popular dataset is used as a standard
in the BCI literature to compare the developed methods and algorithms. This
article uses only the 2a sub-dataset, an MI dataset with Left Hand, Right
Hand, Both Feet, and Tongue tasks. The EEG signals were recorded with a 250 Hz
sampling frequency on 22 electrodes. The amplifier included a hardware
bandpass filter between 0.5 and 100 Hz and a notch filter at 50 Hz to remove
the powerline noise.

This dataset was recorded with the help of 9 experimental subjects, and each
subject participated in two different experimental days. Therefore, concerning
the independent days configuration, this dataset contains 18 subjects.

####  2.1.4 TTK

The TTK database kollod_ttk_2022 (10) was recorded in Research Centre for
Natural Sciences (TTK, as a Hungarian abbreviation). A 64-channeled ActiChamp
amplifier system (Brain Products GmbH, Gliching, Germany) was used to capture
the EEG signals, which were operated with a 500 Hz sampling frequency.

The EEG signals were recorded with a custom designed MATLAB based, paradigm
leader code called General Offline Paradigm (GoPar). GoPar was presented in
the Supplementary Materials of kollod_closed_2023 (57) and available at
https://github.com/kolcs/GoPar. The paradigm of the Physionet database
inspired this code. GoPar was designed to conduct multiple different MI
paradigms with 4 tasks. In the case of the TTK dataset, these tasks were Left
Hand, Right Hand, Left Foot, and Right Foot. The paradigm started with a one-
minute-long eye-open session, followed by a one-minute-long eye-closed one as
the initial task. These tasks aimed to get the subjects’ full attention,
preparing them for the core part of the experiment, and served as a baseline.
The paradigm continued with 2 warmup sessions where two out of the four MI
tasks were selected and practiced overtly and covertly. The warmup sessions
aimed to lead the subjects on how to execute MI tasks. After the warmup
sessions, the pure MI sessions were followed with a randomized order of the
four MI tasks.

In total 25 experiments were conducted with 9 subjects. No hardware or
software filters were applied.

###  2.2 Signal processing

As a first step, the EEG signals were filtered with a 5th-order Butterworth
bandpass filter between 1 and 45 Hz. Then a customized FASTER algorithm
nolan_faster_2010 (54) was applied, as presented in kollod_closed_2023 (57),
to remove artifacts corresponding to eye movement or muscle activity. The
first step removed EEG channels which were assumed to be constantly noisy
throughout an entire experiment, concerning the variance, the correlation, and
the Hurst exponent. Secondly, epochs containing motions (chewing, yawning)
were discarded by measuring the deviation from the channel average, the
amplitude range, and variance parameters. In the third step eye related
artifacts are filtered out utilizing independent component analysis. The final
step filtered out EEG channels from epochs individually, which were still
considered noisy, concerning the variance, median gradient, amplitude range,
and channel deviation parameters. The fifth step of the original FASTER
algorithm, which detected artifacts through subjects, was omitted, because we
aimed our signal processing algorithm to be subject-specific.

The purified 4-second-long epochs were split into 2-second-long windows with
0.1-second shifts to enhance the number of samples. Then the signals were
normalized using the standard scaling, where the mean of the data is set to
zero and the standard deviation to one. These processed EEG windows were used
to train and test the classifiers of the BCI System.

In the case of within-subject classification, 5-fold cross-validation was
conducted subject-wise, where the database was split on the epoch level to
ensure that windows originating from the same epoch are exclusively used in
either the train or test set. Approximately 10 % of the data from the training
set were used as a validation set, splitting it on the epoch level.

###  2.3 Nerual Networks

This section presents the used neural networks, our methods, and modifications
concerning the original ones.

####  2.3.1 Callbacks

Under the training of neural networks, a modified early stopping and model-
saving strategy were applied. The conventional early stopping approach
prechelt_early_2012 (58) focuses on monitoring the validation loss and halting
the learning process when it increases to prevent the network from
overfitting. Additionally, a patience parameter can be defined to determine
the number of training epochs that should be waited for before the monitored
value shows improvement again. We extended this strategy with an additional
patience-like value called “give up”. This strategy aims to handle training
situations where the validation loss increases above the initial training
loss, and after a particular time, the neural network starts to learn;
therefore, the validation loss decreases. The give up value defines how many
training epochs should be waited until the validation loss reaches the initial
amount of validation loss. The original patience value is activated if the
initial loss has been reached under the give up limit. Otherwise, the training
is finished.

Our model saving strategy was designed to reflect on the modified early-
stopping strategy. Until the initial validation loss was reached, model
weights with the highest validation accuracy were saved. After reaching the
initial validation loss, model weights were only saved if improvements were
detected with respect to both the validation loss and validation accuracy.
Before the test phase, the best model weights were restored.

We conducted our experiments by setting the training epochs to 500, the give
up value to 100, and the patience to 20.

####  2.3.2 ConvNets

For implementing the Deep and Shallow ConvNets the source code in
lawhern_eegnet_2018 (25) was used, which uses a few modified parameters
concerning the originally published ones in schirrmeister_deep_2017 (1). No
additional modifications were made concerning the architecture of the
networks.

####  2.3.3 EEGNets

The networks of the EEGNet family, the EEGNet lawhern_eegnet_2018 (25), the
EEGNet Fusion roots_fusion_2020 (41), and the MI-EEGNet riyad_mi-eegnet_2021
(39) were all modified in such a way that they could be automatically used for
databases with different sampling frequencies instead of setting up the input
parameters manually. In the article of the EEGNet lawhern_eegnet_2018 (25),
the authors explicitly said that the filter size of the first convolutional
block should be half of the sampling frequency rate. Therefore, in our
implementation, instead of directly defining the size of the kernel, it is
calculated from the used signals sampling frequency. We followed this strategy
in the case of the other two networks.

###  2.4 Transfer learning

Next to the subject-wise learning, we also investigated the effect of transfer
learning. First, test subjects were selected as distinct groups of 10. The
remaining subjects, called pre-train subjects, were used to set the initial
optimal weights for the neural networks. A validation set was separated from
the pre-train data to use it with our modified early stopping and model-saving
strategy. When the pretraining phase converged, either reaching the maximum
training epoch number or stopping it by the early stopping strategy, the best
weights of the network were stored. For each test subject, 5-fold within-
subject cross-validation was conducted as described in the third paragraph of
2.2. Before each cross-validation step, the saved model’s weights were loaded,
and the selected training set of the selected test subject was used as a fine-
tuning data for the neural networks. Under the fine-tuning step, validation
sets were again used with our early-stopping and model-saving strategies.

###  2.5 EEGNet family comparison

Many computational experiments were conducted on each database (Physionet,
Giga, TTK, and BCI Competition IV 2a) to compare the neural networks from the
EEGNet family (Shallow ConvNet, Deep ConvNet, EEGNet, EEGNet Fusion, MI-
EEGNet). In the case of an experimental subject who participated in multiple
experiments on different days, the data was handled as if multiple subjects
would have participated instead, called independent days configuration.
However, on the BCI Competition IV 2a dataset, we also conducted experiments
where the data of a subject is combined, overseeing the date of the records
because this way, the results are more comparable with previous BCI studies.
We highlighted this experiment with the “merged subject data” words.

A within-subject and a transfer learning phase were conducted on each neural
network database. The results of the cross-validations were collected, and the
normality test was used to select a proper statistical test: the t-test or
Wilcoxon for normally or not normally distributed accuracy levels,
respectively. The received p-values were adjusted with Bonferroni correction.
The significance level was preset to 0.05.

We aimed to rank the neural networks; therefore, two additional metrics were
introduced next to the pure accuracy comparison. These metrics were
investigated on the independent days configured databases. The first metric is
the accuracy improvement of the EEGNet family with respect to the chance
level. One advantage of this metric is that it can be used on databases with
different class numbers. This metric was calculated and averaged for both
within-subject and transfer learning.

The second metric focused on the effect of transfer learning, which was
investigated by comparing the results of within-subject classification with
the transfer learning classification. The difference between the two methods
was calculated for each database concerning independent days.

###  2.6 Significance investigation of databases

The number and quality of significant differences were investigated to
quantitatively measure our hypothesis that databases with higher than 20
experimental days are sufficient only for BCI system comparison. For each
database configuration, we calculated two numbers. The sum of significance
levels, where the significance levels were categorized as it, is presented in
Table 1. and the count of significant differences. We correlated these numbers
with the number of subjects in the databases.

Table 1: Level of a significance tests Level | p-value range
---|---
1 | 1e-2 < p <= 5e-2
2 | 1e-3 < p <= 1e-2
3 | 1e-4 < p <= 1e-3
4 | p <= 1e-4

##  3 Results

After receiving the 5-fold cross-validated accuracy levels for all the
combinations of the 4 databases, 5 neural networks, and 2 learning methods
(within-subject and transfer learning), the normality tests showed a not
normal distribution. Therefore, the Wilcoxon statistical test was used with
Bonferroni correction for significance analysis. The results are presented in
Figure 2. In general, it can be concluded that transfer learning significantly
improved the results at all the databases except the BCI Competition 2a.

Figure 2: EEGNet family comparison on 4 databases handling the datasets in
independent days configuration.
The p-value annotation legend is the following: ns:5e-2 < p; *: 1e-2 < p <=
5e-2; **: 1e-3 < p <= 1e-2; ***: 1e-4 < p <= 1e-3; ****: p <= 1e-4. The mean
of the data is presented with the ‘+’ symbol.

On the Physionet (Figure 2A), in the case of within-subject classification,
the MI-EEGNet significantly reached the highest accuracy (0.4646) compared to
the other methods, while in the case of transfer learning, the Deep ConvNet
showed the significantly highest performance (0.5377).

On the Giga database (Figure 2B), the MI-EEGNet reached the highest
accuracies, 0.725 and 0.7724, in the case of within-subject and transfer
learning, respectively. This network significantly outperformed the others,
except compared to the Shallow ConvNet in transfer learning mode.

Analyzing the results from the TTK dataset (Figure 2C), it can be concluded
that the EEGNet reached the highest accuracies: 0.4437 and 0.4724 in the case
of within-subject and transfer learning, respectively. These results were
significantly higher than the other networks except for the MI-EEGNet.

Figure 3: EEGNet family comparison on BCI Competition IV 2a. The p-value
annotation legend is the following: ns:5e-2 < p; *: 1e-2 < p <= 5e-2; **: 1e-3
< p <= 1e-2; ***: 1e-4 < p <= 1e-3; ****: p <= 1e-4. The mean of the data is
presented with the ‘+’ symbol.

In the case of the BCI Competition IV 2a dataset, handled as independent days
(Figure 2D), the Shallow ConvNet reached 0.719 and 0.733 accuracies in the
case of within-subject and transfer learning, respectively. In the case of
transfer learning, it was significant compared to the other neural networks.
However, concerning the within-subject classification, the results were
comparable with the EEGNet and the MI-EEGNet. On the other hand, when the data
corresponding to one subject was merged, ignoring the experimental days, the
Shallow ConvNet reached again the highest accuracies, 0.749 and 0.7533 in the
case of within-subject and transfer learning, respectively; however, the
difference between the neural network was insignificant.

To establish a hierarchy among the neural networks, we analyzed the accuracy
improvement of the EEGNet family relative to the chance level. Table 3
displays the ranking of these networks based on their training modes. Among
all the databases in an independent days setup, MI-EEGNet displayed the most
significant average improvement in within-subject classification. On the other
hand, the Shallow ConvNet outperformed the other networks for transfer
learning.

Table 2: Classification Improvements by transfer learning on databases with
independetn day configuration

Rank Neural Networks Physionet Giga TTK BCI Comp IV 2a Avg. impr. 1 Deep
ConvNet 0.1557 0.1418 0.0708 0.0614 0.1075 2 Shallow ConvNet 0.0928 0.0497
0.0509 0.0141 0.0519 3 EEGNet 0.0716 0.0487 0.0288 -0.0065 0.0357 4 EEGNet
Fusion 0.0381 0.0586 0.0379 0.0007 0.0338 5 MI-EEGNet -0.0058 0.0475 0.0564
-0.0015 0.0241

Table 3: Ranking the performance of neural networks on databases concerning independent days. | Rank | Classifier |  | Avg. Acc. impr.
---
from chance level
Within subject | 1 | MI-EEGNet | 0.2306
2 | Shallow ConvNet | 0.2071
3 | EEGNet | 0.1997
4 | EEGNet Fusion | 0.1871
5 | Deep ConvNet | 0.1249
Transfer learning | 1 | Shallow ConvNet | 0.2721
2 | Deep ConvNet | 0.2598
3 | MI-EEGNet | 0.2537
4 | EEGNet | 0.2521
5 | EEGNet Fusion | 0.2312

Another factor we considered was the enhancement of neural networks through
transfer learning, which we have presented in Table 2. Deep ConvNet showed the
most substantial improvement, achieving results 0.1 higher than within-subject
classification mode on average. Whereas Shallow ConvNet, ranked first in
transfer learning performance, could only improve by 0.05 compared to within-
subject classification.

Finally, the databases were ranked concerning the number of significant
differences. Table 4 presents the sum of significance ranges (corresponding to
the number of stars on the figures) and the count of significant differences
with the number of subjects in the databases. The sum of the significance
ranges and the number of subjects in the databases were found to be strongly
correlated, r(3) = 0.7709, but insignificant (p-value 0.127014 > 0.05).

Table 4: Significance investigation | Significance level |
---|---|---
Database | Sum | Count | Subjects
Physionet | 63 | 18 | 105
Giga | 49 | 15 | 108
TTK | 45 | 16 | 25
BCI Comp IV 2a | 31 | 15 | 18
| BCI Comp IV 2a –
---
merged subject data
0 | 0 | 9

##  4 Discussion

Most of the articles presenting MI EEG signal classification with artificial
neural networks from the EEGNet family are presenting and comparing their
results on one of the BCI competition databases. This article aimed to
highlight that datasets with many subjects are required for statistically
significant comparisons. Therefore, we compared 5 neural networks from the
EEGNet family on 4 databases which include various subjects. Concerning the
datasets, we introduced the independent day configuration, where the data of a
subject who participated in multiple experimental days counted as if multiple
subjects participated instead. With this configuration, we aimed to extend the
number of experiments and increase the comparisons’ significance level. All
four databases, namely the BCI competition IV 2a database
tangermann_review_2012 (6), the Physionet goldberger_ary_l_physiobank_2000 (7,
8), the Giga lee_eeg_2019 (9), and our TTK dataset kollod_ttk_2022 (10), were
used in this configuration. In the case of Physionet, the authors claim that
the experiments were conducted with 109 volunteers; therefore, the independent
subject configuration is irrelevant here. Concerning the BCI competition IV 2a
database, we conducted another experiment where the data of a subject was
combined into one (“merged subject data”). Width this configuration, we aimed
to conduct classifications comparable to other reports. In addition, we used
these results to test our hypothesis about the correlation between the number
of subjects in the database and the number of significant comparisons (Table
4). The correlation between the number of subjects and our significance metric
was strong but insignificant. However, Table 4 presents that a database with 9
subjects is insufficient for significance testing. Therefore, we suggest that
databases with many subjects, such as the Physionet or the Giga datasets,
should be used to compare BCI systems. To further investigate our hypothesis,
additional open-access MI EEG databases are required.

We also would like to highlight that in our experiments artifact filtered EEG
data were used, unlike the articles about the investigated neural networks
lawhern_eegnet_2018 (25, 39, 41, 1). They included bandpass filtering and
standardization before the classification. In our signal processing step, next
to the 5th ordered bandpass Butterworth filter from 1 to 45 Hz, we utilized
the FASTER nolan_faster_2010 (54) artifact removal algorithm to detect and
remove artifacts corresponding to eye movement and muscle activity. This is
highly important; otherwise, it may happen that instead of pure EEG signals,
artifacts are classified. It was demonstrated in noboa_development_2021 (55)
that electromyography could be successfully used for BCI purposes.

Most articles that investigate the effect of transfer learning are presented
using datasets without artifact filtering kant_cwt_2020 (59, 51, 52, 49, 53).
We showed that on databases with many subjects (Physionet and Giga), even
after artifact filtering, the transfer learning significantly improves the
accuracy of the neural networks compared to the within-subject classification
(Figure 2 A & B). We also showed that Deep ConvNet could gain the most from
the transfer learning considering all the databases (Table 2). On the other
hand, the Shallow ConvNet reached the highest results concerning our
“improvement from the chance level” metric on all transfer learning trained
neural networks (Table 3). Nevertheless, the difference between the ConvNets
is insignificant concerning the Physionet and the Giga database (Figure 2 A &
B).

Our results highlight that various aspects should be considered for a good
ranking between neural networks. Using unfiltered datasets with few numbers of
subjects and considering only the accuracy differences between networks may
lead to a vague conclusion.

In future work, it would be interesting to conduct transfer learning by
utilizing data from multiple databases. However, one must overcome the problem
that different datasets are recorded with different EEG amplifiers; therefore,
the position, number of electrodes, and sampling frequency could differ.

##  5 Conclusion

In this paper, we critically compared neural networks from the EEGNet family,
namely the Shallow ConvNet, the Deep ConvNet, the EEGNet, the EEGNet Fusion.
and the MI-EEGNet, on MI EEG signal classification tasks. The comparisons were
conducted utilizing the BCI competition IV 2a database and the Giga and the
Physionet, which include many subjects. In addition, we also used our TTK
dataset. Within-subject and transfer learning classification were conducted on
each database configuration and neural network combination. All the received
results were 5-fold cross-validated. The classification results were acquired
after cleaning the raw signals from artifacts with the FASTER algorithm.

To the best of our knowledge, there were no articles published before that
compare neural networks from the EEGNet family on artifact filtered databases,
which include high numbers of subjects (>20), and in addition, the results are
also cross-validated. We also demonstrated that transfer learning could
improve the classification results, even on artifact-filtered MI EEG data. To
rank the neural networks, we introduced two metrics. The first considered the
neural networks’ accuracy improvement from the chance level, while the second
investigated the classification improvements by transfer learning. These
metrics showed that Shallow ConvNet and Deep ConvNet could perform better than
the later published networks from the EEGNet family. Finally, we showed that
databases with few numbers of subjects ($\leq$10) are not sufficient for
comparing BCI systems statistically.

## CRediT Authorship contribution statement

Csaba Köllőd: Conceptualization, Methodology, Experiments, Software,
Statistics, Visualization, Writing – original draft
András Adolf: Software - FASTER algorithm implementation, Experiments, Writing
- review & editing
Gergely Márton: Experiments, Writing - review & editing
István Ulbert: Supervision, Project administration, Writing - review & editing

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgement

We thank our Pilots for their weekly availability and participation in the
experiments. We are also grateful for their regular feedback about the system.

Prepared with the professional support of the Doctoral Student Scholarship
Program of the Co-operative Doctoral Program of the Ministry of Innovation and
Technology financed from the National Research, Development and Innovation
Fund.

## Data availability statement

Databases and source codes are available under the following links.

Source codes:

  * 1.

Signal processing and classification framework –
https://github.com/kolcs/bionic_apps

  * 2.

Paradigm Handler – https://github.com/kolcs/GoPar

Datasets:

  * 1.

Physionet – https://physionet.org/content/eegmmidb/1.0.0/

  * 2.

Giga – http://gigadb.org/dataset/100542

  * 3.

BCI Competition IV – https://www.bbci.de/competition/iv/

  * 4.

TTK – https://hdl.handle.net/21.15109/CONCORDA/UOQQVK

## Supplementary Materials

Excel tables about database accuracies: Supplementary Materials.xlsx

## References

  * (1) Robin Tibor Schirrmeister et al.  “Deep learning with convolutional neural networks for EEG decoding and visualization” 1112 citations (Crossref) [2023-02-09] 00960 667 citations (Semantic Scholar/DOI) [2021-07-02] _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/hbm.23730  In _Human Brain Mapping_ 38.11, 2017, pp. 5391–5420  DOI: 10/gfwjqj
  * (2) Jonathan R Wolpaw et al.  “Brain–computer interfaces for communication and control” 4778 citations (Crossref) [2022-01-21] ZSCC: 0007964  In _Clinical Neurophysiology_ 113.6, 2002, pp. 767–791  DOI: 10/ctbv3h
  * (3) B. Blankertz et al.  “The BCI competition 2003: progress and perspectives in detection and discrimination of EEG single trials” 641 citations (Semantic Scholar/DOI) [2022-02-08] Conference Name: IEEE Transactions on Biomedical Engineering  In _IEEE Transactions on Biomedical Engineering_ 51.6, 2004, pp. 1044–1051  DOI: 10/dvwfp8
  * (4) B. Blankertz et al.  “The BCI competition III: validating alternative approaches to actual BCI problems” 713 citations (Semantic Scholar/DOI) [2022-02-08] Conference Name: IEEE Transactions on Neural Systems and Rehabilitation Engineering  In _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ 14.2, 2006, pp. 153–159  DOI: 10/fms95g
  * (5) P. Sajda et al.  “A data analysis competition to evaluate machine learning algorithms for use in brain-computer interfaces” 122 citations (Semantic Scholar/DOI) [2022-02-08] Conference Name: IEEE Transactions on Neural Systems and Rehabilitation Engineering  In _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ 11.2, 2003, pp. 184–185  DOI: 10/frfxks
  * (6) Michael Tangermann et al.  “Review of the BCI Competition IV”  In _Frontiers in Neuroscience_ 6, 2012  DOI: 10/ghhvkk
  * (7) Goldberger Ary L. et al.  “PhysioBank, PhysioToolkit, and PhysioNet” 6290 citations (Crossref) [2022-01-10] 00000 ZSCC: 0009754 Publisher: American Heart Association  In _Circulation_ 101.23, 2000, pp. e215–e220  DOI: 10/ggwwsx
  * (8) Gerwin Schalk et al.  “BCI2000: a general-purpose brain-computer interface (BCI) system” 1539 citations (Crossref) [2022-01-10] 00000 ZSCC: 0002514  In _IEEE transactions on bio-medical engineering_ 51.6, 2004, pp. 1034–1043  DOI: 10/b7bs63
  * (9) Min-Ho Lee et al.  “EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy” 128 citations (Semantic Scholar/DOI) [2022-09-30]  In _GigaScience_ 8.5, 2019, pp. giz002  DOI: 10/gjx5fr
  * (10) Csaba Köllőd et al.  “TTK dataset - 4 class Motor-Imagery EEG” Publisher: CONCORDA Type: dataset, 2022  URL: https://hdl.handle.net/21.15109/CONCORDA/UOQQVK
  * (11) R. Riener and L.. Seward  “Cybathlon 2016” 7 citations (Crossref) [2022-11-07] 00000 ZSCC: 0000012 ISSN: 1062-922X  In _2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , 2014, pp. 2792–2794  DOI: 10/ghvj8w
  * (12) Domen Novak et al.  “Benchmarking Brain-Computer Interfaces Outside the Laboratory: The Cybathlon 2016” 29 citations (Semantic Scholar/DOI) [2022-11-02]  In _Frontiers in Neuroscience_ 11, 2018  DOI: 10/gcwc8r
  * (13) Serafeim Perdikis et al.  “The Cybathlon BCI race: Successful longitudinal mutual learning with two tetraplegic users” 74 citations (Crossref) [2022-11-07] ZSCC: 0000061  In _PLOS Biology_ 16.5, 2018, pp. e2003787  DOI: 10/gdhqkq
  * (14) Karina Statthaler et al.  “Cybathlon experiences of the Graz BCI racing team Mirage91 in the brain-computer interface discipline” 14 citations (Crossref) [2022-11-07] 00000 ZSCC: 0000012 ZSCC: 0000011  In _Journal of NeuroEngineering and Rehabilitation_ 14.1, 2017, pp. 129  DOI: 10/ghtnkr
  * (15) Camille Benaroch et al.  “Long-Term BCI Training of a Tetraplegic User: Adaptive Riemannian Classifiers and User Training” 10 citations (Semantic Scholar/DOI) [2022-11-02]  In _Frontiers in Human Neuroscience_ 15, 2021  DOI: 10/gjm6wm
  * (16) Lea Hehenberger et al.  “Long-Term Mutual Training for the CYBATHLON BCI Race With a Tetraplegic Pilot: A Case Study on Inter-Session Transfer and Intra-Session Adaptation” 5 citations (Crossref) [2022-11-07]  In _Frontiers in Human Neuroscience_ 15, 2021  DOI: 10/gnqjf4
  * (17) Attila Korik et al.  “Competing at the Cybathlon championship for people with disabilities: long-term motor imagery brain–computer interface training of a cybathlete who has tetraplegia” 0 citations (Semantic Scholar/DOI) [2022-11-02]  In _Journal of NeuroEngineering and Rehabilitation_ 19.1, 2022, pp. 95  DOI: 10/gq568m
  * (18) Neethu Robinson et al.  “Design Considerations for Long Term Non-invasive Brain Computer Interface Training With Tetraplegic CYBATHLON Pilot” 5 citations (Semantic Scholar/DOI) [2022-11-02]  In _Frontiers in Human Neuroscience_ 15, 2021  DOI: 10/gq568q
  * (19) Stefano Tortora et al.  “Neural correlates of user learning during long-term BCI training for the Cybathlon competition” 0 citations (Semantic Scholar/DOI) [2022-11-02] 0 citations (Crossref) [2022-10-07]  In _Journal of NeuroEngineering and Rehabilitation_ 19.1, 2022, pp. 69  DOI: 10/gq568p
  * (20) Federica Turi, Maureen Clerc and Théodore Papadopoulo  “Long Multi-Stage Training for a Motor-Impaired User in a BCI Competition” 5 citations (Semantic Scholar/DOI) [2022-11-02]  In _Frontiers in Human Neuroscience_ 15, 2021  DOI: 10/gq568n
  * (21) Benjamin Blankertz et al.  “The non-invasive Berlin Brain–Computer Interface: Fast acquisition of effective performance in untrained subjects” 803 citations (Semantic Scholar/DOI) [2022-11-08]  In _NeuroImage_ 37.2, 2007, pp. 539–550  DOI: 10.1016/j.neuroimage.2007.01.051
  * (22) Alexandre Barachant, Stéphane Bonnet, Marco Congedo and Christian Jutten  “Riemannian Geometry Applied to BCI Classification” 149 citations (Semantic Scholar/DOI) [2022-11-02]  In _Latent Variable Analysis and Signal Separation_ , Lecture Notes in Computer Science  Berlin, Heidelberg: Springer, 2010, pp. 629–636  DOI: 10/cvgc25
  * (23) F. Lotte and C. Guan  “Regularizing Common Spatial Patterns to Improve BCI Designs: Unified Theory and New Algorithms” 680 citations (Crossref) [2022-11-07] ZSCC: 0000525[s0]  In _IEEE Transactions on Biomedical Engineering_ 58.2, 2011, pp. 355–362  DOI: 10/bxncfg
  * (24) Kai Keng Ang et al.  “Filter Bank Common Spatial Pattern Algorithm on BCI Competition IV Datasets 2a and 2b” 553 citations (Crossref) [2022-11-07] 00000 537 citations (Semantic Scholar/DOI) [2021-05-26] Publisher: Frontiers  In _Frontiers in Neuroscience_ 6, 2012  DOI: 10/ghmkc7
  * (25) Vernon J. Lawhern et al.  “EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces” 964 citations (Crossref) [2023-02-09]  In _Journal of Neural Engineering_ 15.5, 2018, pp. 056013  DOI: 10/ggkjnq
  * (26) Siavash Sakhavi, Cuntai Guan and Shuicheng Yan  “Parallel convolutional-linear neural network for motor imagery classification” 83 citations (Semantic Scholar/DOI) [2022-11-08] ISSN: 2076-1465  In _2015 23rd European Signal Processing Conference (EUSIPCO)_ , 2015, pp. 2736–2740  DOI: 10.1109/EUSIPCO.2015.7362882
  * (27) Irene Sturm, Sebastian Lapuschkin, Wojciech Samek and Klaus-Robert Müller  “Interpretable deep neural networks for single-trial EEG classification” 283 citations (Semantic Scholar/DOI) [2022-12-14]  In _Journal of Neuroscience Methods_ 274, 2016, pp. 141–145  DOI: 10.1016/j.jneumeth.2016.10.008
  * (28) Yousef Rezaei Tabar and Ugur Halici  “A novel deep learning approach for classification of EEG motor imagery signals” 448 citations (Crossref) [2022-11-08] ZSCC: 0000102[s0]  In _Journal of Neural Engineering_ 14.1, 2017, pp. 016003  DOI: 10/gdvjb4
  * (29) Hamdi Altaheri, Ghulam Muhammad and Mansour Alsulaiman  “Physics-inform attention temporal convolutional network for EEG-based motor imagery classification” 0 citations (Semantic Scholar/DOI) [2022-11-07] Conference Name: IEEE Transactions on Industrial Informatics  In _IEEE Transactions on Industrial Informatics_ , 2022, pp. 1–10  DOI: 10.1109/TII.2022.3197419
  * (30) Alessandro Bria, Claudio Marrocco and Francesco Tortorella  “Sinc-based convolutional neural networks for EEG-BCI-based motor imagery classification” 2 citations (Semantic Scholar/arXiv) [2022-11-07] 00000 ZSCC: 0000000 arXiv: 2101.10846  In _arXiv:2101.10846 [eess]_ , 2021  URL: http://arxiv.org/abs/2101.10846
  * (31) X. Deng et al.  “Advanced TSGL-EEGNet for Motor Imagery EEG-Based Brain-Computer Interfaces” 24 citations (Crossref) [2023-02-09] 00000 ZSCC: 0000000 Conference Name: IEEE Access  In _IEEE Access_ 9, 2021, pp. 25118–25130  DOI: 10/gh2hw3
  * (32) Chang Gao, Wenchao Liu and Xian Yang  “Convolutional neural network and riemannian geometry hybrid approach for motor imagery classification” 0 citations (Semantic Scholar/DOI) [2023-02-09]  In _Neurocomputing_ 507, 2022, pp. 180–190  DOI: 10.1016/j.neucom.2022.08.024
  * (33) Wenkai Huang, Yihao Xue, Lingkai Hu and Hantang Liuli  “S-EEGNet: Electroencephalogram Signal Classification Based on a Separable Convolution Neural Network With Bilinear Interpolation” 16 citations (Semantic Scholar/DOI) [2022-12-14] Conference Name: IEEE Access  In _IEEE Access_ 8, 2020, pp. 131636–131646  DOI: 10.1109/ACCESS.2020.3009665
  * (34) Hongli Li, Man Ding, Ronghua Zhang and Chunbo Xiu  “Motor imagery EEG classification algorithm based on CNN-LSTM feature fusion network” 16 citations (Crossref) [2023-02-09] 0 citations (Semantic Scholar/DOI) [2022-01-03]  In _Biomedical Signal Processing and Control_ 72, 2022, pp. 103342  DOI: 10/gnx6sd
  * (35) Hongli Li et al.  “A parallel multi-scale time-frequency block convolutional neural network based on channel attention module for motor imagery classification” 0 citations (Crossref) [2023-02-09] 0 citations (Semantic Scholar/DOI) [2022-08-30]  In _Biomedical Signal Processing and Control_ 79, 2022, pp. 104066  DOI: 10/gq569b
  * (36) Xiaolin Liu et al.  “TCACNet: Temporal and channel attention convolutional network for motor imagery classification of EEG-based BCI” 4 citations (Crossref) [2023-02-09] 0 citations (Semantic Scholar/DOI) [2022-08-08]  In _Information Processing & Management_ 59.5, 2022, pp. 103001  DOI: 10/gq5685
  * (37) Weifeng Ma et al.  “A channel-mixing convolutional neural network for motor imagery EEG decoding and feature visualization” 5 citations (Crossref) [2023-02-09] 00000  In _Biomedical Signal Processing and Control_ 70, 2021, pp. 103021  DOI: 10/gmhbp5
  * (38) Yazeed K. Musallam et al.  “Electroencephalography-based motor imagery classification using temporal convolutional network fusion” 31 citations (Crossref) [2023-02-09]  In _Biomedical Signal Processing and Control_ 69, 2021, pp. 102826  DOI: 10/gkc2s2
  * (39) Mouad Riyad, Mohammed Khalil and Abdellah Adib  “MI-EEGNET: A novel convolutional neural network for motor imagery classification” 15 citations (Crossref) [2023-02-09] 00000 ZSCC: 0000000  In _Journal of Neuroscience Methods_ 353, 2021, pp. 109037  DOI: 10/gjqj84
  * (40) Mouad Riyad, Mohammed Khalil and Abdellah Adib  “A novel multi-scale convolutional neural network for motor imagery classification” 6 citations (Crossref) [2023-02-09] 00000  In _Biomedical Signal Processing and Control_ 68, 2021, pp. 102747  DOI: 10/gj5hmt
  * (41) Karel Roots, Yar Muhammad and Naveed Muhammad  “Fusion Convolutional Neural Network for Cross-Subject EEG Motor Imagery Classification” 21 citations (Crossref) [2023-02-09] 2 citations (Semantic Scholar/DOI) [2021-07-02] Number: 3 Publisher: Multidisciplinary Digital Publishing Institute  In _Computers_ 9.3, 2020, pp. 72  DOI: 10/gk3p86
  * (42) Huiming Yao et al.  “FB-EEGNet: A fusion neural network across multi-stimulus for SSVEP target detection” 3 citations (Semantic Scholar/DOI) [2023-02-09]  In _Journal of Neuroscience Methods_ 379, 2022, pp. 109674  DOI: 10.1016/j.jneumeth.2022.109674
  * (43) Zümray Dokur and Tamer Olmez  “Classification of motor imagery electroencephalogram signals by using a divergence based convolutional neural network” 2 citations (Crossref) [2022-11-07] ZSCC: 0000000  In _Applied Soft Computing_ 113, 2021, pp. 107881  DOI: 10/gmvjfx
  * (44) Ward Fadel et al.  “Multi-Class Classification of Motor Imagery EEG Signals Using Image-Based Deep Recurrent Convolutional Neural Network” 16 citations (Crossref) [2022-11-08] 4 citations (Semantic Scholar/DOI) [2022-02-09] ISSN: 2572-7672  In _2020 8th International Winter Conference on Brain-Computer Interface (BCI)_ , 2020, pp. 1–4  DOI: 10/gpdz72
  * (45) Ward Fadel et al.  “Chessboard EEG Images Classification for BCI Systems Using Deep Neural Network” 0 citations (Crossref) [2022-11-08] 2 citations (Semantic Scholar/DOI) [2022-02-09]  In _Bio-inspired Information and Communication Technologies_ , Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering  Cham: Springer International Publishing, 2020, pp. 97–104  DOI: 10/gpdz7t
  * (46) Yuexing Han et al.  “A classification method for EEG motor imagery signals based on parallel convolutional neural network” 4 citations (Crossref) [2022-11-07] 00000  In _Biomedical Signal Processing and Control_ 71, 2022, pp. 103190  DOI: 10/gm7xvr
  * (47) Xueyu Jia, Yonghao Song and Longhan Xie  “Excellent fine-tuning: From specific-subject classification to cross-task classification for motor imagery” 0 citations (Crossref) [2022-11-07] 0 citations (Semantic Scholar/DOI) [2022-08-30]  In _Biomedical Signal Processing and Control_ 79, 2023, pp. 104051  DOI: 10/gq5689
  * (48) Xueyu Jia, Yonghao Song, Lie Yang and Longhan Xie  “Joint spatial and temporal features extraction for multi-classification of motor imagery EEG” 3 citations (Crossref) [2022-11-07] 00000  In _Biomedical Signal Processing and Control_ 71, 2022, pp. 103247  DOI: 10/gm7xvn
  * (49) Arunabha M. Roy  “Adaptive transfer learning-based multiscale feature fused deep convolutional neural network for EEG MI multiclassification in brain–computer interface” 18 citations (Semantic Scholar/DOI) [2023-02-08]  In _Engineering Applications of Artificial Intelligence_ 116, 2022, pp. 105347  DOI: 10.1016/j.engappai.2022.105347
  * (50) Karl Weiss, Taghi M. Khoshgoftaar and DingDing Wang  “A survey of transfer learning” 1315 citations (Semantic Scholar/DOI) [2023-02-09]  In _Journal of Big Data_ 3.1, 2016, pp. 9  DOI: 10.1186/s40537-016-0043-6
  * (51) Zahra Khademi, Farideh Ebrahimi and Hussain Montazery Kordy  “A transfer learning-based CNN and LSTM hybrid deep learning model to classify motor imagery EEG signals” 16 citations (Semantic Scholar/DOI) [2023-02-08]  In _Computers in Biology and Medicine_ 143, 2022, pp. 105288  DOI: 10.1016/j.compbiomed.2022.105288
  * (52) F. Mattioli, C. Porcaro and G. Baldassarre  “A 1D CNN for high accuracy classification and transfer learning in motor imagery EEG-based brain-computer interface” 13 citations (Semantic Scholar/DOI) [2023-02-08] Publisher: IOP Publishing  In _Journal of Neural Engineering_ 18.6, 2022, pp. 066053  DOI: 10.1088/1741-2552/ac4430
  * (53) Ruilong Zhang et al.  “Hybrid deep neural network using transfer learning for EEG motor imagery decoding” 34 citations (Semantic Scholar/DOI) [2023-02-09]  In _Biomedical Signal Processing and Control_ 63, 2021, pp. 102144  DOI: 10.1016/j.bspc.2020.102144
  * (54) H. Nolan, R. Whelan and R.. Reilly  “FASTER: Fully Automated Statistical Thresholding for EEG artifact Rejection” 653 citations (Crossref) [2022-11-07] 00000 604 citations (Semantic Scholar/DOI) [2021-06-28]  In _Journal of Neuroscience Methods_ 192.1, 2010, pp. 152–162  DOI: 10/fdmpw9
  * (55) Erick Noboa et al.  “Development of an EMG based SVM supported control solution for the PlatypOUs education mobile robot using MindRove headset” 1 citations (Semantic Scholar/DOI) [2022-11-02] 0 citations (Crossref) [2022-05-27]  In _IFAC-PapersOnLine_ 54.15, 11th IFAC Symposium on Biological and Medical Systems BMS 2021, 2021, pp. 304–309  DOI: 10/gq569x
  * (56) Chen-Chen Fan et al.  “Bilinear neural network with 3-D attention for brain decoding of motor imagery movements from the human EEG” 3 citations (Crossref) [2022-05-19] 00000 ZSCC: 0000000 ZSCC: 0000000  In _Cognitive Neurodynamics_ , 2020  DOI: 10/ghsn4r
  * (57) Csaba Köllőd et al.  “Closed loop BCI System for Cybathlon 2020” 0 citations (Semantic Scholar/arXiv) [2023-02-09] 0 citations (Semantic Scholar/DOI) [2023-02-09] arXiv:2212.04172 [cs, eess]  arXiv, 2023  DOI: 10.48550/arXiv.2212.04172
  * (58) Lutz Prechelt  “Early Stopping — But When?”  In _Neural Networks: Tricks of the Trade: Second Edition_ , Lecture Notes in Computer Science  Berlin, Heidelberg: Springer, 2012, pp. 53–67  DOI: 10.1007/978-3-642-35289-8_5
  * (59) Piyush Kant, Shahedul Haque Laskar, Jupitara Hazarika and Rupesh Mahamune  “CWT Based Transfer Learning for Motor Imagery Classification for Brain computer Interfaces” 13 citations (Crossref) [2022-01-21] 00000 ZSCC: 0000002  In _Journal of Neuroscience Methods_ 345, 2020, pp. 108886  DOI: 10/ghk43n