11institutetext: University of Cassino and Southern Latium, Cassino, FR 03043
Italy 22institutetext: University of Salerno, Fisciano, SA 84084 Italy

# Sinc-based convolutional neural networks for EEG-BCI-based motor imagery
classification††thanks: This work was supported by MIUR (Minister for
Education, University and Research, Law 232/216, Department of Excellence).

Alessandro Bria 11 0000-0002-2895-6544 Claudio Marrocco 11 0000-0003-0840-7350
Francesco Tortorella 22 0000-0002-5033-9323

###### Abstract

Brain-Computer Interfaces (BCI) based on motor imagery translate mental motor
images recognized from the electroencephalogram (EEG) to control commands. EEG
patterns of different imagination tasks, e.g. hand and foot movements, are
effectively classified with machine learning techniques using band power
features. Recently, also Convolutional Neural Networks (CNNs) that learn both
effective features and classifiers simultaneously from raw EEG data have been
applied. However, CNNs have two major drawbacks: (i) they have a very large
number of parameters, which thus requires a very large number of training
examples; and (ii) they are not designed to explicitly learn features in the
frequency domain. To overcome these limitations, in this work we introduce
Sinc-EEGNet, a lightweight CNN architecture that combines learnable band-pass
and depthwise convolutional filters. Experimental results obtained on the
publicly available BCI Competition IV Dataset 2a show that our approach
outperforms reference methods in terms of classification accuracy.

###### Keywords:

Motor imagery Brain computer interface Convolutional neural networks.

##  1 Introduction

A Brain-Computer Interface (BCI) translates brain signals into messages or
commands for an interactive task. This enables a wide range of applications
from clinic to industry for both patients and healthy users, such as
rehabilitation devices for stroke patients [22], controllable wheelchairs and
prostheses [35], new gaming input devices [8], to name a few. Among different
brain activity monitoring modalities, noninvasive approaches based on
electroencephalography (EEG) use multiple electrodes placed on the skull
surface to record the activity of cerebral cortical neurons [5] and are widely
used in many BCI studies thanks to their ease of implementation, reduced costs
and high availability [20]. The most popular EEG signals used to control BCI
systems are P300 evoked potentials, steady-state visual evoked potentials
(SSVEP) and motor imagery (MI) which is the focus of our work. Specifically,
MI refers to the imagination of moving certain body parts without actual
movement [28]. Different MI tasks result into discriminable patterns observed
from the oscillatory activities in the sensorimotor cortex region of the brain
[21]. Imagination of left hand, right hand, foot and tongue movements are the
most investigated MI tasks in the BCI literature [15].

Handcrafted feature extraction methods coupled with conventional classifiers
like Linear Discriminant Analysis (LDA), Support Vector Machines (SVM),
Bayesian classifiers, and Nearest Neighbor classifiers have been used in a
number of studies for MI task recognition [15]. A widely used approach is to
extract and combine band power features from different channel(electrode)
signals to capture connectivity patterns among different regions of the
sensorimotor cortex and, ultimately, their interaction and engagement with
each other. This is thought to play a fundamental role in accomplishing
movement imaginations [14]. Common spatial patterns (_CSP_) were introduced to
this end in [23] and received a large share of research in the field [4, 16,
25, 26, 34], but their effectiveness depended on subject-specific frequency
bands. This problem was alleviated by the popular filter bank CSP (_FBCSP_)
[1] that decomposes the EEG into multiple frequency pass bands prior to
spatial filtering, feature selection and classification. This method also won
the BCI Competition IV [33] for 4-class motor imagery recognition (Dataset 2a)
and was since used as a reference method for comparison.

Given their effectiveness in other fields [9, 29], deep learning methods, and
in particular Convolutional Neural Networks (CNNs)[13], have the potential to
learn both effective features and classifiers simultaneously from raw EEG
data. Several studies have recently explored deep learning for MI
classification [17, 27, 31, 32, 12]. Notably, [27] showed that their _Shallow
ConvNet_ (one temporal convolution, one spatial convolution, squaring and mean
pooling) could outperform their _Deep ConvNet_ (temporal convolution, spatial
convolution, then three layers of standard convolution) as well as _FBCSP_. A
similar result was achieved by [12] with _EEGNet_ , a compact lightweight
network (one temporal convolution, one depthwise convolution, one separable
convolution, and a fully connected layer) that compared favorably with _Deep
ConvNet_ and performed on par with _Shallow ConvNet_. These results indicate
that shallow networks having a small number of parameters are beneficial for
MI applications that are characterized by very small numbers of training
examples because of the difficulty in performing millions or even thousands of
mental commands during training sessions.

In this paper we propose _Sinc-EEGNet_ , a 4-layer CNN architecture that
combines the benefits of both EEG frequency band decomposition of classical
methods, such as _FBCSP_ , and automatic feature learning and extraction of
lightweight CNN models, such as _EEGNet_. In particular, the first
convolutional layer of our network is restricted to use parameterized sinc
functions that implement band pass filters. The subsequent depthwise and
separable convolution layers learn a spatial filter and combine the features
from the different frequency bands previously selected, which are then
inputted to the final classification layer. An overview of the proposed
architecture is shown in Fig. 1.

Figure 1: An overview of the proposed _Sinc-EEGNet_ architecture.

##  2 Sinc layer

A standard CNN convolution layer applied on a one-dimensional discrete time-
domain signal $s[t]$ performs convolutions with $F$ one-dimensional filters
$h_{1},...,h_{F}$ each having $K$ learnable weights. Conversely, the Sinc
layer performs convolutions with $F$ predefined functions $g_{1},...,g_{F}$
each implementing a learnable bandpass filter $G$ as the difference between
two low-pass filters in the frequency domain:

$G[f]=rect\left(\frac{f}{2f_{2}}\right)-rect\left(\frac{f}{2f_{1}}\right)$ (1)

where $f_{1}$ and $f_{2}>f_{1}$ are the learnable low and high cutoff
frequencies. Using the inverse Fourier transform, the time-domain filter $g$
is obtained as:

$g[t]=2f_{2}sinc(2\pi f_{2}t)-2f_{1}sinc(2\pi f_{1}t)$ (2)

where the sinc function is defined as $sinc(x)=sin(x)/x$. The cutoff
frequencies are initialized by sampling from a Gaussian distribution with mean
and variance equal to $f_{s}/4$, where $f_{s}$ represents the sampling
frequency of the input signal. The constraint $f_{2}>f_{1}$ is implemented by
using in Eq. 2 the following cutoff frequencies $f_{1}^{abs}$ and
$f_{2}^{abs}$:

$f_{1}^{abs}=|f_{1}|$ (3) $f_{2}^{abs}=f_{1}+|f_{2}-f_{1}|.$ (4)

Because of the discrete approximation of $g$, the resulting bandpass filter is
nonideal and may present ripples in the passband and limited attenuation in
the stopband. To alleviate this problem, we multiply $g$ with the popular
Hamming window $w$ [18] defined as:

$w[t]=0.54-0.46\cdot\cos\left(\frac{2\pi t}{L}\right)$ (5)

where $L$ is the number of discrete samples used to approximate $g$. The sinc
convolutional layer transforming the input signal $s[t]$ into the band-
decomposed output signal $o_{1},...,o_{F}$ is then defined by:

$o_{i}[t]=s[t]*\left(g_{i}[t]\cdot w[t]\right).$ (6)

##  3 The Sinc-EEGNet architecture

The proposed Sinc-EEGNet is a combination and adaptation of the Sinc
convolution layer originally proposed by [24] for speech recognition with
_SincNet_ , and _EEGNet_ [12] for what concerns the spatial filtering
implemented with depthwise convolution. Specifically, the architecture of
Sinc-EEGNet (see Fig. 1 and Table LABEL:tab:architecture) consists of four
blocks described as follows:

  1. 1.

_Sinc Convolution_. The first block takes in input a signal having $C$
channels and $T$ time samples, and performs convolution with $F_{1}$ sinc
filters having $L$ time samples. Compared to the first standard convolution
layer used in other CNN architectures such as _EEGNet_ , here the sinc filters
are explicitly designed to learn the optimal band decomposition for the MI
classification task and, when the CNN is trained with data from a single BCI
user, this will reflect the peculiarities of the EEG oscillatory activity of
that user. Another advantage is the reduced number of parameters, from
$K\times F_{1}$ of the standard convolution to $2\times F_{1}$ of the sinc
convolution. This also implies faster convergence and better generalization
capabilities especially when using small training sets as in the case of MI
applications. Computational efficiency also is improved since the filters are
symmetric, thus the convolution can be performed on one side of the filter and
inheriting the result for the other half.

  2. 2.

_Depthwise Convolution_. Similarly to _EEGNet_ [12], we use a Depthwise
Convolution layer [6] of size $(C,1)$ to learn $D$ spatial filters for each of
the $F_{1}$ inputted feature maps across the channel dimension, for a total of
$F_{2}=D\times F_{1}$ filters. Combined with the first layer that performs
optimal band decomposition, this two-step sequence can be considered a
‘learnable’ version of the well known _FBCSP_ [1] approach.

  3. 3.

_Separable Convolution_. Similarly to _EEGNet_ , we summarize each feature map
individually using a Depthwise Convolution of size $(1,16)$, and then merge
the outputs using $F_{2}$ $(1,1)$ Pointwise Convolutions. This allows optimal
combination of the information within and across feature maps.

  4. 4.

_Classification_. The last layer is a fully connected layer that receives the
flattened features from the previous layer and maps them to 4 decision classes
(left hand, right hand, foot, tongue).

At the end of blocks 1-3 we apply Average Pooling of size $(1,4)$ for
dimensionality reduction, Layer Normalization [2], Dropout regularization
[30], and CELU activation [3]. Layer Normalization, as opposed to Batch
Normalization [10] used in other architectures (_EEGNet_ , _Deep ConvNet_ ,
_Shallow ConvNet_), calculates the mean and variance across channels instead
than batches. This is especially useful for BCI datasets characterized by a
high number of channels(electrodes) and small batch sizes resulting from the
scarcity of training data. As to the CELU activation, it is an improvement
over the ELU activation [7] used in other architectures (_EEGNet_ , _Deep
ConvNet_ , _Shallow ConvNet_) since its derivative does not diverge and it
contains both the linear transfer function and ReLU [19] activation as special
cases.

[ caption=Sinc-EEGNet architecture, where $C$ = number of channels, $T$ =
number of time points, $L$ = number of sinc samples, $F_{1}$ = number of
temporal filters, $D$ = number of spatial filters, $F_{2}$ = number of
pointwise filters, and $N$ = number of classes., label = tab:architecture,
width = pos = !t, doinside=]m1.0cmm3.1cmm1.2cmm1.0cmXXm1.3cm Block Layer
filters size params Output Activation
1 Input $(C,T)$
Reshape $(1,C,T)$
Sinc Convolution $F_{1}$ $(1,L)$ $2\times F_{1}$ $(F_{1},C,T)$
Average Pooling $(1,4)$ $(F_{1},C,\frac{T}{4})$
Layer Normalization $2\times F_{1}$ $(F_{1},C,\frac{T}{4})$ CELU
Dropout $(F_{1},C,\frac{T}{4})$
2 Depthwise Convolution $D\times F_{1}$ $(C,1)$ $C\times D\times F_{1}$
$(D\times F_{1},1,\frac{T}{4})$
Average Pooling $(1,4)$ $(D\times F_{1},1,\frac{T}{16})$
Layer Normalization $2\times D\times F_{1}$ $(D\times F_{1},1,\frac{T}{16})$
CELU
Dropout $(D\times F_{1},1,\frac{T}{16})$
Depthwise Convolution $D\times F_{1}$ $(1,16)$ $16\times D\times F_{1}$
$(D\times F_{1},1,\frac{T}{16})$
Layer Normalization $2\times D\times F_{1}$ $(D\times F_{1},1,\frac{T}{16})$
CELU
Dropout $(D\times F_{1},1,\frac{T}{16})$
3 Pointwise Convolution $F_{2}$ $(1,1)$ $F_{2}\times(D\times F_{1})$
$(F_{2},1,\frac{T}{16})$
Average Pooling $(1,4)$ $(F_{2},1,\frac{T}{64})$
Layer Normalization $2\times F_{2}$ $(F_{2},1,\frac{T}{64})$ CELU
Dropout $(F_{2},1,\frac{T}{64})$
4 Flatten $F_{2}\times\frac{T}{64}$
Fully Connected $N\times F_{2}\times\frac{T}{64}$ $N$ Softmax

##  4 Experiments

The EEG data used in this study comes from the BCI Competition IV Dataset 2A
[33]. The data consists of four classes of imagined movements of left and
right hands, feet and tongue recorded from 9 subjects during two separate
sessions, each composed by 288 trials. The EEG data were originally recorded
using $C=22$ Ag/AgCl electrodes(channels), sampled at 250 Hz and bandpass
filtered between 0.5 and 100 Hz. We applied a further bandpass filtering to
suppress frequencies above 64 Hz and resampled the timeseries to 128 Hz as in
[12]. Z-score standardization was used to normalize the signals within each
trial.

EEG data were splitted for training and testing according to three different
paradigms:

  1. 1.

_Competition-based_. The training and test sets were the same as indicated in
the BCI Competition. This allowed to compare our method with reference methods
from the literature that reported their results using the same data split,
namely _FBSCP_ [1], _Deep ConvNet_ [27], and _Shallow ConvNet_ [27] as well as
all other participants to the original challenge.

  2. 2.

_Within-subject_. For each subject, a dedicated experiment was performed using
only data from that subject from the BCI Competition training and test sets.

  3. 3.

_Cross-subject_. For each subject, a dedicated experiment was performed using
only data from other subjects from the BCI Competition training set, and only
data from that subject from the BCI Competition test set.

In all the experiments, we performed a four-class classification using
accuracy as the summary measure. In the within- and cross-subject experiments,
we also trained and tested an _EEGNet_ with $F_{1}=8$ and $D=2$, which was the
best performing CNN reported in [12]. As to our _Sinc-EEGNet_ , we chose $D=2$
for a fair comparison with _EEGNet_ , but we set $F_{1}=32$ since our Sinc
layer is specifically designed for frequency band decomposition and thus can
benefit from learning a wide variety of bandpass filters. This can be seen in
Fig. 2 that shows 32 distinct filters learnt by _Sinc-EEGNet_ in the
competition-based experiment. The number of samples $L$ used to discretize the
sinc functions was set to $64$ that resulted from a trade-off between
approximation precision and computational complexity.

All the CNNs were trained using backpropagation and Adam optimizer [11] with
weight updates that proceeded in batches of $20$ samples for $100$ epochs. The
base learning rate was set to $10^{-3}$. Momentum and weight decay were set
respectively to $0.9$ and $2\times 10^{-2}$. Following [12], for the Dropout
layers we chose $p=0.5$ for within-subject experiments, and $p=0.25$ for
competition-based and cross-subject experiments that used more training data
and thus required less regularization. The loss function was categorical
cross-entropy.

[ caption=Comparison of classification accuracies between our method and
reference methods on the BCI Competition IV-2A., label = tab:results, pos =
!t, doinside=]m4.0cmm2.0cm Method Accuracy
_FBCSP_ $68.0\%$
_Deep ConvNet_ $70.9\%$
_Shallow ConvNet_ $73.7\%$
_Sinc-EEGNet_ $75.39\%$

##  5 Results

The comparison between _Sinc-EEGNet_ and the reference methods from the
literature on the competition-based data split are reported in Table
LABEL:tab:results. Remarkably, _Sinc-EEGNet_ outperforms all other methods in
terms of accuracy and sets a new state-of-the-art on the BCI Competition IV-2A
with an accuracy of $75.39\%$ that improves _FBCSP_ by $17.39\%$. As to the
within- and cross-subject experiments, _EEGNet_ yielded an average accuracy of
$60.99\%$ and $58.75\%$, respectively, and _Sinc-EEGNet_ of $70.56\%$ and
$58.98\%$, respectively. Also in this case, our method exhibited superior
performance, with an improvement of almost $10\%$ accuracy in the more
practically adopted within-subject classification.

##  6 Conclusions

In this work we proposed _Sinc-EEGNet_ , a lightweight convolutional neural
network for EEG-BCI-based motor imagery classification that learns optimal
band decomposition and spatial filtering, mimicking the behavior of the well-
known _FBCSP_ but learning the filters directly from the raw EEG data. Our
method outperformed reference methods from the literature, including _FBCSP_
and _EEGNet_ , on the publicly available BCI Competition IV-2A dataset. To the
best of our knowledge, this is the first work that validated the use of
learnable bandpass filters in the first layer of a CNN for EEG signal
classification. Future work will investigate alternative frequency filters,
such as Difference of Gaussian (DoG) filter, that are less subject to discrete
approximation issues, and architecture variants that explore different spatial
filtering and feature map combination approaches.

Figure 2: The 32 sinc filters learnt by _Sinc-EEGNet_ on the BCI Competition
IV Dataset 2A.

## References

  * [1] Ang, K.K., Chin, Z.Y., Wang, C., Guan, C., Zhang, H.: Filter bank common spatial pattern algorithm on bci competition iv datasets 2a and 2b. Frontiers in neuroscience 6, 39 (2012)
  * [2] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
  * [3] Barron, J.T.: Continuously differentiable exponential linear units. arXiv pp. arXiv–1704 (2017)
  * [4] Blankertz, B., Tomioka, R., Lemm, S., Kawanabe, M., Muller, K.R.: Optimizing spatial filters for robust eeg single-trial analysis. IEEE Signal processing magazine 25(1), 41–56 (2007)
  * [5] Britton, J.W., Frey, L.C., Hopp, J.L., Korb, P., Koubeissi, M.Z., Lievens, W.E., Pestana-Knight, E.M., St, E.L.: Electroencephalography (EEG): An introductory text and atlas of normal and abnormal findings in adults, children, and infants. American Epilepsy Society, Chicago (2016)
  * [6] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1251–1258 (2017)
  * [7] Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015)
  * [8] Coyle, D., Principe, J., Lotte, F., Nijholt, A.: Guest editorial: Brain/neuronal-computer game interfaces and interaction. IEEE Transactions on Computational Intelligence and AI in games 5(2), 77–81 (2013)
  * [9] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1026–1034 (2015)
  * [10] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
  * [11] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [12] Lawhern, V.J., Solon, A.J., Waytowich, N.R., Gordon, S.M., Hung, C.P., Lance, B.J.: Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces. Journal of neural engineering 15(5), 056013 (2018)
  * [13] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
  * [14] Liu, T., Li, F., Jiang, Y., Zhang, T., Wang, F., Gong, D., Li, P., Ma, T., Qiu, K., Li, H., et al.: Cortical dynamic causality network for auditory-motor tasks. IEEE Transactions on Neural Systems and Rehabilitation Engineering 25(8), 1092–1099 (2016)
  * [15] Lotte, F., Bougrain, L., Cichocki, A., Clerc, M., Congedo, M., Rakotomamonjy, A., Yger, F.: A review of classification algorithms for eeg-based brain–computer interfaces: a 10 year update. Journal of neural engineering 15(3), 031005 (2018)
  * [16] Lotte, F., Guan, C.: Regularizing common spatial patterns to improve bci designs: unified theory and new algorithms. IEEE Transactions on biomedical Engineering 58(2), 355–362 (2010)
  * [17] Lu, N., Li, T., Ren, X., Miao, H.: A deep learning scheme for motor imagery classification based on restricted boltzmann machines. IEEE transactions on neural systems and rehabilitation engineering 25(6), 566–576 (2016)
  * [18] Mitra, S.K.: Digital Signal Processing. McGraw-Hill Science/Engineering/Math (2005)
  * [19] Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: ICML (2010)
  * [20] Nicolas-Alonso, L.F., Gomez-Gil, J.: Brain computer interfaces, a review. sensors 12(2), 1211–1279 (2012)
  * [21] Pfurtscheller, G., Da Silva, F.L.: Event-related eeg/meg synchronization and desynchronization: basic principles. Clinical neurophysiology 110(11), 1842–1857 (1999)
  * [22] Pichiorri, F., Morone, G., Petti, M., Toppi, J., Pisotta, I., Molinari, M., Paolucci, S., Inghilleri, M., Astolfi, L., Cincotti, F., et al.: Brain–computer interface boosts motor imagery practice during stroke recovery. Annals of neurology 77(5), 851–865 (2015)
  * [23] Ramoser, H., Muller-Gerking, J., Pfurtscheller, G.: Optimal spatial filtering of single trial eeg during imagined hand movement. IEEE transactions on rehabilitation engineering 8(4), 441–446 (2000)
  * [24] Ravanelli, M., Bengio, Y.: Interpretable convolutional filters with sincnet. arXiv preprint arXiv:1811.09725 (2018)
  * [25] Rivet, B., Cecotti, H., Phlypo, R., Bertrand, O., Maby, E., Mattout, J.: Eeg sensor selection by sparse spatial filtering in p300 speller brain-computer interface. In: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology. pp. 5379–5382. IEEE (2010)
  * [26] Samek, W., Kawanabe, M., Müller, K.R.: Divergence-based framework for common spatial patterns algorithms. IEEE Reviews in Biomedical Engineering 7, 50–72 (2013)
  * [27] Schirrmeister, R.T., Springenberg, J.T., Fiederer, L.D.J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., Ball, T.: Deep learning with convolutional neural networks for eeg decoding and visualization. Human brain mapping 38(11), 5391–5420 (2017)
  * [28] Schuster, C., Hilfiker, R., Amft, O., Scheidhauer, A., Andrews, B., Butler, J., Kischka, U., Ettlin, T.: Best practice for motor imagery: a systematic literature review on motor imagery training elements in five different disciplines. BMC medicine 9(1), 75 (2011)
  * [29] Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Mastering the game of Go with deep neural networks and tree search. nature 529(7587), 484 (2016)
  * [30] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15(1), 1929–1958 (2014)
  * [31] Sturm, I., Lapuschkin, S., Samek, W., Müller, K.R.: Interpretable deep neural networks for single-trial eeg classification. Journal of neuroscience methods 274, 141–145 (2016)
  * [32] Tabar, Y.R., Halici, U.: A novel deep learning approach for classification of eeg motor imagery signals. Journal of neural engineering 14(1), 016003 (2016)
  * [33] Tangermann, M., Müller, K.R., Aertsen, A., Birbaumer, N., Braun, C., Brunner, C., Leeb, R., Mehring, C., Miller, K.J., Mueller-Putz, G., et al.: Review of the bci competition iv. Frontiers in neuroscience 6, 55 (2012)
  * [34] Yger, F., Lotte, F., Sugiyama, M.: Averaging covariance matrices for eeg signal classification based on the csp: an empirical study. In: 2015 23rd European Signal Processing Conference (EUSIPCO). pp. 2721–2725. IEEE (2015)
  * [35] Zhang, R., Li, Y., Yan, Y., Zhang, H., Wu, S., Yu, T., Gu, Z.: Control of a wheelchair in an indoor environment based on a brain–computer interface and automated navigation. IEEE transactions on neural systems and rehabilitation engineering 24(1), 128–139 (2015)