# Subject-Independent Classification of Brain Signals using Skip Connections
111††thanks: This work was partly supported by Institute for Information &
Communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (No. 2017-0-00451, Development of BCI based Brain and
Cognitive Computing Technology for Recognizing User’s Intentions using Deep
Learning; No.2021-0-02068, Artificial Intelligence Innovation Hub).

Soowon Kim
Young-Eun Lee
Dept. Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Dept. Brain and Cognitive Engineering
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Ji-Won Lee
Seo-Hyun Lee
Dept. Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Dept. Brain and Cognitive Engineering
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

Untapped potential for new forms of human-to-human communication can be found
in the active research field of studies on the decoding of brain signals of
human speech. A brain-computer interface system can be implemented using
electroencephalogram signals because it poses more less clinical risk and can
be acquired using portable instruments. One of the most interesting tasks for
the brain-computer interface system is decoding words from the raw
electroencephalogram signals. Before a brain-computer interface may be used by
a new user, current electroencephalogram-based brain-computer interface
research typically necessitates a subject-specific adaption stage. In
contrast, the subject-independent situation is one that is highly desired
since it allows a well-trained model to be applied to new users with little or
no precalibration. The emphasis is on creating an efficient decoder that may
be employed adaptively in subject-independent circumstances in light of this
crucial characteristic. Our proposal is to explicitly apply skip connections
between convolutional layers to enable the flow of mutual information between
layers. To do this, we add skip connections between layers, allowing the
mutual information to flow throughout the layers. The output of the encoder is
then passed through the fully-connected layer to finally represent the
probabilities of the 13 classes. In this study, overt speech was used to
record the electroencephalogram data of 16 participants. The results show that
when the skip connection is present, the classification performance improves
notably.

††publicationid: pubid: 978-1-6654-6444-4/23/$31.00 ©2023 IEEE

Keywords–brain–computer interface, deep learning, electroencephalography,
speech processing

##  I INTRODUCTION

Brain signals carry information about human behavior, imagery or mental and
physical states [1], which is useful for deciphering intents. By analyzing a
user’s brain activity, brain-computer interface (BCI) technology can generate
external commands that can be utilized to control the surroundings[2, 3, 4].
Through the use of their intents, which are translated from brain signals,
users of BCI can control prostheses, offering an additional means of
communication between people and external machines[5]. A recent area of BCI
research, brain-to-speech (BTS), aims to produce spoken words from brain
signals. The goal of BTS systems is to decipher the speech-related intents
from brain activity and then provide orders for communication, in contrast to
other communication techniques like event-related potential spellers. Brain-
to-brain systems may be a means of intuitive communication as speech is the
most common mode of communication[6]. We hypothesize that there must be a
meaningful brain activation that might encode a substantial aspect of the
speech because it has been demonstrated that it is possible to reconstruct
speech from brain signals of spoken speech[7]. EEG signal analysis methods
have seen a great deal of development, with promising results [8, 9, 10, 11,
12, 13]. A brief calibration session is required before a BCI system may be
utilized by a new user because the majority of recent studies concentrate on
the subject-dependent scenario in which training and test data are from the
same subject [14, 15, 7]. This labor- and time-intensive calibration procedure
must be carried out for every new subject and usage. The subject-independent
scenario, in contrast, is little studied but is greatly sought to enhance the
user-experience. In this scenario, a BCI system is trained using data from
seen individuals while being applied straight to new users without pre-
adaptation. Exploring the subject-independent scenario, however, is quite
challenging. EEG signals change significantly even between recording sessions
of the same user under the same experimental paradigm and show strong subject-
to-subject variability [16]. The majority of techniques rely on conventional
machine learning techniques and manually created features, making them one of
the few subject-independent investigations. [17] extracts and categorizes
information from each subject’s EEG data using a pair of Linear Discriminant
Analysis (LDA) and Common Spatial Patterns (CSP) algorithms. An ensemble
classifier is created by combining many classifiers using l1 regularized
regression. Due to the handcrafted features’ and conventional learning
algorithms’ limited capabilities, these methods are unable to deliver
performance that is sufficient. Deep learning techniques have recently made
impressive strides, showing promise for addressing complex cross-subject
scenarios [18, 19]. However, only a small number of successful deep learning
studies have shown strong generalization capabilities from known subjects to
new ones [20, 21, 22, 23, 24]. Therefore, deep learning based models for EEG
signal decoding still remains challenging tasks [25].

Skip connection is a common method for enhancing the performance and
convergence of deep neural networks. It works by propagating a linear
component through the layers of the neural network, which is thought to ease
the difficulties in optimization caused by non-linearity [26, 27]. As the name
implies, Skip Connections (or Shortcut Connections) omit some neural network
layers and provide the result of one layer as the input to subsequent levels.
Skip Connections were developed to address various issues in various
architectures. Skip connections allowed us to address the degradation issue in
the case of ResNets, while it ensured feature reusability in the case of
DenseNets [28]. In the sections below, we’ll go into further detail on each
one.

In this article, we introduce an unique subject-independent EEG data analysis
method based on convolutional neural network that is taking advantage of skip
connection. Through multiple skip connections the flow of mutual information
is preserved. The encoding of EEG signals is accomplished using convolutional
neural networks.

Figure 1: Experimental paradigm for recording EEG signals from overt speech
Figure 2: Overall architecture of the model used in this work. Waveform of the
overt speech EEG signals is used as an input to the model with 64 channels.
Each block performas 1-dimensional convolution, 1-dimensional batch
normalization, and activation (ELU). To enable skip connection throughout the
intermediate layers, maxpooling is applied to match the dimensions. After
extracting the latent vector from the CNN, fully-connected layer is applied to
output the final classification probabilities.

##  II Materials and methods

###  II-A Subjects

The experimental protocol was designed to record EEG signals while performing
overt speech following the perception stage with auditory stimuli in between.
Sixteen subjects participated in the study. The study was approved by the
Korea University Institutional Review Board [KUIRB-2019-0143-01] and was
conducted in accordance with the Declaration of Helsinki. Informed consent was
obtained from all subjects. All subjects are asked to fill out the
questionnaire before and after the experiment to check their physiological and
mental conditions and evaluate the experimental paradigms.

###  II-B Experimental Protocol and Paradigm

We recorded EEG signals from scalp during overt speech through 64 channels.
First, an auditory stimuli is given to the subject which is a beeping sound.
Then, the perception is performed which is a hearing of the given word with
visual perception as well through the monitor. With another auditory stimuli,
the corresponding overt speech is performed. EEG signals from both stages are
collected. In total, more than 1,400 trials are performed in each subject The
experimental paradigm is described in Fig. 1. Thirteen classes are introduced,
with labels ranging from 0 to 12. These classes include “ambulance,” “clock,”
“hello,” “help me,” “light,” “pain,” “stop,” “thank you,” “toilet,” “TV,”
“water,” and “yes,” and a silent phase.

TABLE I: Classification performance of deep convolutional neural networks with
and without skip connections

Subject | Accuracy (%) | F1-score (%) | Precision (%) | Recall (%)
---|---|---|---|---
Proposed method | 98.69 | 98.70 | 98.74 | 98.69
Without skip connection | 80.24 | 80.64 | 83.38 | 80.25

###  II-C Preprocessing

From the beginning of each trial, the EEG signal was segmented into 1.5
seconds and sampled at 1,000 Hz. A fifth Butterworth filter was used to
preprocess the EEG data in the high-gamma region of 0.5-120 Hz, and baseline
was corrected by deducting the average of 500 ms before to the start of each
trial. We chose the channels (AF3, F3, F5, FC3, FC5, T7, C5, TP7, CP5, and P5)
that are in the Broca and Wernicke’s areas. Using independent component
analysis and references from EOG and EMG, we performed artifact reduction
techniques for the muscle activity around the mouth. Using OpenBMI Toolbox
[25], BBCI Toolbox [29], and EEGLAB [30], all data processing operations were
carried out in Python and Matlab.

###  II-D Architecture

The proposed classification framework consists of convolution layers and skip
connection to extract time-spectral-spatial information, as shown in Fig. 2.
The raw EEG signals as a waveform were used as an input to the model. To
capture various EEG features such as spectral, spatial, and temporal
information, we used deep convolutional network. In each encoding block,
1-dimensional convolutional layer is followed by 1-dimensional batch
normalization, activation and dropout. The probability of the dropout was 0.5.
In total, 5 encoding blocks are used to produce the final latent
representation of the raw EEG signals. We used the exponential linear unit
(ELU) as the activation function. 64 channels are maintained throughout each
encoding block.

The classifier (Fully-connected layer) is then applied with 3 linear, ELU
activation and dropout combination. 128 hidden units are used. The output for
classification is set to 13 classes with the input as raw signals (C × T). For
training, we applied the mean squared error loss. For each condition, 2000
epoch training and 5-fold cross-validation were used to conduct the
evaluation. The likelihood of chance level for this experiment was $5\%$ due
to the 13 classes present.

Figure 3: The confusion matrix for the model comparing the proposed method
with no skip connection. The true labels are on the vertical axis, while the
predicted labels are on the horizontal axis. The upper matrix represent the
results from the model without skip connection and the lower matrix represents
the model with skip connection.

###  II-E Model training

The training session consists of more than 1,000 epochs. For every 10 epochs,
we validated the model with a validation set, which is $10\%$ randomly
selected portion of the entire dataset. When the model’s performance does not
increase for more than 10 validation, we stop the training.

Adam was used for the training optimizer. The learning rate was $0.0001$ which
was fixed throughout the training without weight decay with betas
$(b_{1},b_{2})$ with 0.9 and 0.999. A batch size of 128 was used for all
experiments.

##  III Results and Discussion

For 13 classes of overt speech, we created frameworks for deciphering speech-
related EEG data. For overt speech, the performance of the model with and
without skip connection was examined. As table I shown above, for 13 classes
with skip connections, the average accuracy of overt speech was 98.69%,
compared to 80.24% for the model without skip connections. F1-score, as well
as precision and recall showed clear advantage, as shown in table I,
indicating a better performance of the model using the skip connection
compared to the model without skip connection. The confusion matrix for the
model with skip connection is depicted in Fig. 3.

##  IV Conclusion

In this study, we proposed a deep convolutional neural network for EEG
decoding that applies skip connections in between convolutional layers to
enable the flow of mutual information between layers. To do this, we add a
skip connection between first, second and third convolutional layer, allowing
the mutual information to flow from the input layer to the output layer. The
output of the encoder is then passed through the fully-connected layer to
finally represent the probabilities of the 13 classes. According to the
findings, the performance was significantly enhanced when the subject-
independent classification was carried out on seven subjects utilizing the
skip connection. As a result, the method that was proposed for decoding brain
activity with skip connections had the potential to be used with reliable BCI
devices for any subject.

## References

  * [1] Y. Zhang, H. Zhang, X. Chen, S.-W. Lee, and D. Shen, “Hybrid high-order functional connectivity networks using resting-state functional mri for mild cognitive impairment diagnosis,” _Scientific reports_ , vol. 7, no. 1, pp. 1–15, 2017.
  * [2] T. Castermans, M. Duvinage, M. Petieau, T. Hoellinger, C. De Saedeleer, K. Seetharaman, A. Bengoetxea, G. Cheron, and T. Dutoit, “Optimizing the performances of a p300-based brain–computer interface in ambulatory conditions,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , vol. 1, no. 4, pp. 566–577, 2011.
  * [3] K.-H. Thung, P.-T. Yap, E. Adeli, S.-W. Lee, D. Shen, A. D. N. Initiative _et al._ , “Conversion and time-to-conversion predictions of mild cognitive impairment using low-rank affinity pursuit denoising and matrix completion,” _Medical image analysis_ , vol. 45, pp. 68–82, 2018.
  * [4] J.-H. Jeong, N.-S. Kwak, C. Guan, and S.-W. Lee, “Decoding movement-related cortical potentials based on subject-dependent and section-wise spectral filtering,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 28, no. 3, pp. 687–698, 2020.
  * [5] D.-O. Won, H.-J. Hwang, D.-M. Kim, K.-R. Müller, and S.-W. Lee, “Motion-based rapid serial visual presentation for gaze-independent brain-computer interfaces,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 26, no. 2, pp. 334–343, 2017.
  * [6] A. G. Huth, W. A. De Heer, T. L. Griffiths, F. E. Theunissen, and J. L. Gallant, “Natural speech reveals the semantic maps that tile human cerebral cortex,” _Nature_ , vol. 532, no. 7600, pp. 453–458, 2016.
  * [7] J.-H. Jeong, K.-H. Shim, D.-J. Kim, and S.-W. Lee, “Brain-controlled robotic arm system based on multi-directional cnn-bilstm network using eeg signals,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 28, no. 5, pp. 1226–1238, 2020.
  * [8] M. Yadava, P. Kumar, R. Saini, P. P. Roy, and D. Prosad Dogra, “Analysis of eeg signals and its application to neuromarketing,” _Multimedia Tools and Applications_ , vol. 76, no. 18, pp. 19 087–19 111, 2017.
  * [9] Y. Roy, H. Banville, I. Albuquerque, A. Gramfort, T. H. Falk, and J. Faubert, “Deep learning-based electroencephalography analysis: a systematic review,” _Journal of neural engineering_ , vol. 16, no. 5, p. 051001, 2019.
  * [10] J. B. Ewen, J. A. Sweeney, and W. Z. Potter, “Conceptual, regulatory and strategic imperatives in the early days of eeg-based biomarker validation for neurodevelopmental disabilities,” _Frontiers in integrative neuroscience_ , vol. 13, p. 45, 2019.
  * [11] M. Lee, B. Baird, O. Gosseries, J. O. Nieminen, M. Boly, B. R. Postle, G. Tononi, and S.-W. Lee, “Connectivity differences between consciousness and unconsciousness in non-rapid eye movement sleep: a tms–eeg study,” _Scientific reports_ , vol. 9, no. 1, pp. 1–9, 2019.
  * [12] H.-I. Suk, S. Fazli, J. Mehnert, K.-R. Müller, and S.-W. Lee, “Predicting bci subject performance using probabilistic spatio-temporal filters,” _PloS one_ , vol. 9, no. 2, p. e87056, 2014.
  * [13] M.-H. Lee, J. Williamson, D.-O. Won, S. Fazli, and S.-W. Lee, “A high performance spelling system based on eeg-eog signals with visual feedback,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 26, no. 7, pp. 1443–1459, 2018.
  * [14] Y. Liu and O. Sourina, “Real-time subject-dependent eeg-based emotion recognition algorithm,” in _Transactions on Computational Science XXIII_. Springer, 2014, pp. 199–223.
  * [15] M. T. Sadiq, X. Yu, Z. Yuan, and M. Z. Aziz, “Identification of motor and mental imagery eeg in two and multiclass subject-dependent tasks using successive decomposition index,” _Sensors_ , vol. 20, no. 18, p. 5283, 2020\.
  * [16] N.-S. Kwak, K.-R. Müller, and S.-W. Lee, “A lower limb exoskeleton control system based on steady state visual evoked potentials,” _Journal of neural engineering_ , vol. 12, no. 5, p. 056009, 2015.
  * [17] M. I. Khalid, T. Alotaiby, S. A. Aldosari, S. A. Alshebeili, M. H. Al-Hameed, F. S. Y. Almohammed, and T. S. Alotaibi, “Epileptic meg spikes detection using common spatial patterns and linear discriminant analysis,” _IEEE Access_ , vol. 4, pp. 4629–4634, 2016.
  * [18] M. Lee, R. D. Sanders, S.-K. Yeom, D.-O. Won, K.-S. Seo, H. J. Kim, G. Tononi, and S.-W. Lee, “Network properties in transitions of consciousness during propofol-induced sedation,” _Scientific reports_ , vol. 7, no. 1, pp. 1–13, 2017.
  * [19] Y.-E. Lee and M. Lee, “Decoding visual responses based on deep neural networks with ear-eeg signals,” in _2020 8th International Winter Conference on Brain-Computer Interface (BCI)_. IEEE, 2020, pp. 1–6.
  * [20] N. Waytowich, V. J. Lawhern, J. O. Garcia, J. Cummings, J. Faller, P. Sajda, and J. M. Vettel, “Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials,” _Journal of neural engineering_ , vol. 15, no. 6, p. 066031, 2018.
  * [21] M. H. Bhatti, J. Khan, M. U. G. Khan, R. Iqbal, M. Aloqaily, Y. Jararweh, and B. Gupta, “Soft computing-based eeg classification by optimal feature selection and neural networks,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 10, pp. 5747–5754, 2019.
  * [22] S.-H. Lee, M. Lee, and S.-W. Lee, “Neural decoding of imagined speech and visual imagery as intuitive paradigms for bci communication,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 28, no. 12, pp. 2647–2659, 2020.
  * [23] K.-T. Kim, C. Guan, and S.-W. Lee, “A subject-transfer framework based on single-trial emg analysis using convolutional neural networks,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 28, no. 1, pp. 94–103, 2019.
  * [24] O.-Y. Kwon, M.-H. Lee, C. Guan, and S.-W. Lee, “Subject-independent brain–computer interfaces based on deep convolutional neural networks,” _IEEE transactions on neural networks and learning systems_ , vol. 31, no. 10, pp. 3839–3852, 2019.
  * [25] S.-H. Lee, M. Lee, and S.-W. Lee, “Eeg representations of spatial and temporal features in imagined speech and overt speech,” in _Asian Conference on Pattern Recognition_. Springer, 2019, pp. 387–400.
  * [26] F. Liu, X. Ren, Z. Zhang, X. Sun, and Y. Zou, “Rethinking skip connection with layer normalization,” in _Proceedings of the 28th International Conference on Computational Linguistics_ , 2020, pp. 3586–3598.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [28] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [29] R. Krepki, B. Blankertz, G. Curio, and K.-R. Müller, “The berlin brain-computer interface (bbci)–towards a new communication channel for online control in gaming applications,” _Multimedia Tools and Applications_ , vol. 33, no. 1, pp. 73–90, 2007.
  * [30] A. Delorme and S. Makeig, “Eeglab: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis,” _Journal of neuroscience methods_ , vol. 134, no. 1, pp. 9–21, 2004.