# Design and Implementation of EEG-Mechatronic System Interface for
Computational Intelligence

Cameron Aume  Shantanu Pal<EMAIL_ADDRESS>Subhas Mukhopadhyay
School of Engineering, Macquarie University, Sydney, NSW 2109, Australia
School of Computer Science, Queensland University of Technology, Brisbane, QLD
4000, Australia

###### Abstract

The devices that can read Electroencephalography (EEG) signals have been
widely used for Brain-Computer Interfaces (BCIs). Popularity in the field of
BCIs has increased in recent years with the development of several consumer-
grade EEG devices that can detect human cognitive states in real-time and
deliver feedback to enhance human performance. Several studies are conducted
to understand the fundamentals and essential aspects of EEG in BCIs. However,
the significant issue of how can consumer-grade EEG devices be used to control
mechatronic systems effectively has been given less attention. In this paper,
we have designed and implemented an EEG BCI system using the OpenBCI Cyton
headset and a user interface running a game. We employ real-world participants
to play a game to gather training data that was later put into multiple
machine learning models, including a linear discriminant analysis (LDA),
k-nearest neighbours (KNN), and a convolutional neural network (CNN). After
training the machine learning models, a validation phase of the experiment
took place where participants tried to play the same game but without direct
control, utilising the outputs of the machine learning models to determine how
the game moved. We find that a CNN trained to the specific user playing the
game performed with the highest activation accuracy from the machine learning
models tested, allowing for future implementation with a mechatronic system.

###### keywords:

Electroencephalography (EEG), Brain-Computer Interfaces (BCIs), Machine
learning, Consumer-grade EEG devices, Mechatronic systems.

##  1 Introduction

The interface between humans and computers has long been a challenging problem
to solve [1]. With the development of Electroencephalographic (EEG) headsets,
there has been a particular interest in reading brainwaves to develop a Brain-
Computer Interface (BCI) [2]. In 1890, Adolf Beck discovered that changing
voltage potentials could be measured on the heads of some animals in response
to sensory stimulation [3]. This was one of the first experiments which
displayed changing electrical voltages on the head. Over the years, there have
been many developments to find ways to detect the electrical activity of the
human brain. Towards this, EEG is a safe, painless, and efficient method for
humans. EEG reads voltage potentials on the human’s scalp. The first EEG
experiment was conducted in 1924 when electrical activity of the human brain
was measured using non-invasive techniques [4]. Since the first developments
of EEG, it has been found useful in many diverse areas, including diagnosis of
several medical conditions, e.g., sleep disorders [5], brain tumours [6] and
epilepsy [7] [8]. With the advancement of human-machine cooperation, smart
sensors, Internet of Things (IoT) applications and services [9] [10] [11], and
intelligent wearable devices [12], industries try to synergise the
intelligence of the brain and the ability of cutting-edge machine learning
methods for BCI designs [13] [14]. BCIs involve the brain to sense, think, and
act against signals by decoding information from neural activity [15] [16].

There is significant research in the area of EEG, exploring how the human
brain works [17] [18] [19] [20]. It has been generalised to four different
frequency bands of signals that occur in the brain: (i) delta, (ii) theta,
(iii) alpha, and (iv) beta. Delta waves are most commonly associated with deep
sleep. Theta waves often signify meditation and/or drowsiness. Alpha waves are
correlated with relaxation and eyes closed. Finally, beta waves generally show
high concentration and thinking [21]. Due to the difference between the
varying frequency bands, a frequency representation of the brainwaves can be
useful for BCI applications.

BCIs can be developed by analysing different representations and parts of EEG
signals. For example, by looking at the blinking pattern of a user, and
analysing the alpha wave band, a computer can be controlled. In order to solve
more complex forms of BCIs, machine learning models are often built to give
specific outputs for BCI tasks [22]. BCI is a powerful communication tool
between users and systems. In other words, BCI bridges the human brain and the
external world that typically do not depend upon information delivery methods.
The function of BCI within a mechatronic system plays a fundamental role in
its design, including its accuracy, speed (e.g., for real-time applications),
and usability (e.g., learnability, and ease of use) [23]. Mechatronic systems
include but are not limited to quadcopter drones, aged-care assistant,
autonomous vehicles, healthcare, etc. For instance, the EEG-mechatronic system
interface can particularly be useful in medical applications, where a user may
not be able to move their muscles to control a robot, but is able to control
it with their mind. Fundamentally, mechatronic systems are beneficial to
assisting people, but often with the help of some human interaction. Utilising
BCIs can enable a more seamless control methodology.

Despite its various successes, BCI needs to overcome several challenges to
gain more user acceptance and seamless integration with the newly discovered
technologies. One of those challenges is how consumer-grade EEG devices can be
used to control mechatronic systems. The motivation of this research is to
develop a system with computational intelligence that can be utilised to
create an interface between the human brain and a mechatronic system. We aim
at utilising a consumer-grade EEG headset to develop a BCI, specifically with
the interest of integrating it with a mechatronic system, e.g., a quad-copter
drone or a robotic arm. Furthermore, unlike the present proposals, we aim to
develop a system that enables online EEG data classification, allowing for
live system control using the BCI. The major contributions of the paper can be
summarised as follows:

  * •

We provide a detailed discussion on consumer-grade EEG headsets to develop a
BCI. We utilise a dry EEG electrode headset with a computer feedback system
for a motor-imagery BCI.

  * •

We provide a detailed design of the system. We determine optimal machine
learning models and their subsequent architecture for a mechatronic system
interface.

  * •

We provide complete proof of concept prototype implementation of the system.
Using real-world participants, we enable a command-based BCI control
methodology using the headset and a video game.

  * •

We present detailed evaluation results based on various machine learning
models and compare their performance analysis.

The rest of the paper is organised as follows. In Section 2, we discuss the
background and preliminary work in the area of EEG technology. In Section 3,
we discuss the design and development of a system used for the
experimentation. In Section 4, the experimental procedures are discussed. In
Section 5, the results from the experiment are detailed. In Section 6, the
results are analysed and discussed. Section 7 presents some related work.
Finally, in Section 8, we conclude the paper and discuss potential future
work. In Table 1, we provide a list of abbreviations and their full forms.

Table 1: List of abbreviations used in the paper and their full name.

Abbreviations |  Full name
---|---
ADC |  Analog to Digital Converter
ANN |  Artificial Neural Network
BCI |  Brain-Computer Interface
CAD |  Computer-Aided Design
CNN |  Convolutional Neural Network
DNN |  Deep Neural Network
EEG |  Electroencephalography
EI |  Experiment Investigator
FFT |  Fast-Fourier Transformation
KNN |  K-Nearest Neighbours
LDA |  Linear Discriminant Analysis
RNN |  Recurrent Neural Network
VEP |  Visually Evoked Potential

##  2 Background and Preliminaries

In this section, we provide a background discussion related to EEG BCI systems
that are significant to our research, including the hardware, software, and
BCI paradigms. In Fig. 1, we illustrate the components of an EEG BCI system.

Figure 1: Components of an EEG BCI system.

###  2.1 Hardware

Hardware is one of the most fundamental parts of an EEG system. Important
subcomponents of the hardware include the selected electrode used to read
voltage potentials on the scalp, the electrical signal processing components,
signal transmission, and the accessible data format of the data available to
the researcher [24].

####  2.1.1 Electrodes

EEG electrodes act as the interface between the scalp of an experimental
participant and the electronic system of the controller/microprocessor which
gathers the data. These electrodes are typically placed in positions with
accordance to the 10-20 international system of electrode placement [25]. The
10-20 international system can be seen in Fig. 2 alongside the expanded 10-10
system.

Figure 2: 10-20 International System of Electrode Placement (Black) and 10-10
expanded system (Grey) [25].

Many types of electrodes exist, including but not limited to wet/saline based
electrodes, pin-based dry electrodes, micro-spike dry electrodes, and flat-
contact dry electrodes. Each of these electrodes have their own benefits and
drawbacks. Wet electrodes are more common in the medical industry due to their
data consistency [26]. Conversely, they require a longer setup time compared
to dry electrodes and sometimes require abrasive cleaning of the scalp [27].
Wet electrodes also have data degradation issues due to the gel drying over
time [28]. Dry electrodes generally consist of a conductive material
contacting the scalp. These electrodes consist of varying types, from micro-
spike electrodes [29], flat-contact electrodes [30], and pin-based electrodes
[31]. The dry pin-based electrode will be used for this experiment as it is
able to contact the scalp through hair, unlike the flat-contact [32, 26] and
micro-spike electrodes [33]. These electrodes also have a consistent signal
quality over time and require minimal setup [34], unlike the micro-spike
electrodes [35]. This electrode can be seen in Fig. 3.

Figure 3: Dry pin-based EEG electrodes [36].

####  2.1.2 Consumer-grade EEG headsets

Since the invention of EEG technology, it has been primarily a tool used by
the medical industry, but in 2009, NeuroSky released the MindFlex, an EEG
controlled game designed for children. Since the release of the MindFlex,
there has been an increased interest in consumer-grade EEG devices [37]. Many
new consumer-grade EEG devices have since been released, aiming to bring EEG
technology outside the medical industry.

An important component of an EEG system is the headset. This is the component
which converts the analog signals from the electrodes into digital figures
which can be processed by a micro-controller. It also contains any analog
filtering circuits or data transmission components [38]. OpenBCI is a company
which developed open-source hardware consumer EEG devices. OpenBCI has two
main boards they have designed for EEG applications: the Ganglion and the
Cyton. The Ganglion has 4 channels with a 24-bit resolution[39]. Research
using this board is limited, but simple BCI tasks have been demonstrated with
this board [40]. The Cyton board is an 8-channel (configurable to 16-channels
with a daisy chain board) board with a 24-bit resolution[39]. This board has
been used for BCI applications including biometrics [41], motor imagery [42]
[43], auditory responses [44], and need reporting for disable persons [45].
Further, the Ultracortex Mark IV Headset is a headset designed to be 3D
printed using a hobby-level 3D printer. In Table 2, we provide a comparison of
the available EEG headsets in the market.

Table 2: Feature comparison of different consumer-grade EEG headsets.

Headset |  | Electrode
---
Count
| Electrode
---
Type
| ADC
---
Res.
| Sample
---
Rate
| Transfer
---
Protocol
| Cost
---
($AU)
| NeuroSky
---
MindFlex
1 |  | Dry
---
(flat)
12 | 512 |  | Proprietary
---
2.4 GHz
102
| NeuroSky
---
MindWave
1 |  | Dry
---
(flat)
12 | 512 | BLE | 228
| Emotiv
---
EPOC
14 |  | Saline
---
(wet)
| 14 or
---
16
| 128 or
---
256
BLE | 1098
| Emotiv
---
Insight
5 |  | Dry
---
(flat)
14 | 128 | BLE | 387
| Emotiv
---
EPOC Flex
32 |  | Gel/Saline
---
(wet)
14 | 128 | BLE | 2197
| Emotiv
---
MN8
4 |  | Dry
---
(flat)
14 | 128 | BLE | —
| OpenBCI
---
Ganglion
4 | Dry/Wet | 24 | 200 |  | Proprietary
---
2.4 GHz
750
| OpenBCI
---
Cyton
8 | Dry/Wet | 24 | 250 | BLE | 1355
| OpenBCI
---
Cyton + Daisy
16 | Dry/Wet | 24 | 250 | BLE | 2155
NextMind | 9 |  | Dry
---
(fingers)
— | — | BLE | 685
| InteraXon
---
Muse
2 |  | Dry
---
(flat)
| 12-
---
16
| 220 or
---
500
BLE | 325

###  2.2 Brain computer interfaces

There are many different forms of BCIs which can be implemented with EEG
headsets. Some of these paradigms required external stimulation, e.g., with
visually evoked potentials (VEPs), while other paradigms only required actions
taken by the user, such as with eyes-closed paradigms and motor imagery.

It is well know that different regions of the brain perform different actions
and respond differently to stimuli and thoughts [46]. For example, the motor
cortex, located near the top of the head, is responsible for controlling
muscle movements, while the visual cortex, near the back of the head, reacts
to visual stimulus. A colour-coded cortical map can be seen in Figs. 4(a) and
4(b). The relevant region and corresponding electrode placements for a BCI
will depend heavily on the type of BCI paradigm used.

(a) Brodmann’s cortical map (lateral).

(b) Brodmann’s cortical map (medial).

Figure 4: Brodmann’s cortical map [47]. Different colours represent different
functions of the brain. For example, the light purple area 4 is responsible
for motor movement, while the dark blue areas 17 and 18 are responsible for
vision processing.

Motor imagery is the process of analysing EEG signals to determine how the
user is moving their muscles [48]. Motor imagery is popular for BCI tasks and
has been used to control and play games [49]. It has also been show to help
improve the motor functions of those with limited motor function, e.g.,
paralysed people or those who have experienced a stroke [50, 51]. The motor
cortex is located toward the top of the head, as shown in Fig. 4. Because of
this, the ideal location for electrodes is around FCz, Cz, CPz, C3, C34, C5,
and C6 [52].

###  2.3 Data processing

Proper data analysis and extraction is required for BCIs. While looking at EEG
wave-forms and manually extracting features can sometimes be done, such as
with the P300 or N170 paradigms, it can be complex to automate. There are many
ways to automate this data analysis process, the most common of which is
machine learning [53].

####  2.3.1 Pre-processing/feature selection

Before analysing data from an EEG, it must first be preprocessed into a useful
format. Preprocessing includes filtering, normalising, and formatting the
data. Filtering is important to remove electrical noise from power lines,
along with other environmental noise, and can be done with an Infinite Impulse
Response (IIR) Filter [54], followed by a simple normalisation of data which
should be formatted. This formatting can be of the form of a Fast-Fourier
Transform (FFT), or an Independent Component Analysis (ICA).

##### Independent component analysis

ICA is a method of separating separate independent samples from a given
signal. An example of a problem ICA solves is “cocktail party problem” in
which a microphone signal is analysed to isolate one person’s voice from a
room full of people speaking. This method of preprocessing is often used in
offline EEG signal processing, such as in [55] where EEG artefacts, such as
eye-blinks, are omitted and SSVEPs are classified, and also in [56] where
noise and other artefacts are removed, and motor imagery is analysed. ICA is a
computationally and time intensive algorithm and is often used for offline
data preprocessing, but is not limited to such.

##### Wavelet transform

A wavelet transform is another form of signal decomposition in which small
wavelets are stretched and shifted to compose the full signal. The
coefficients of these wavelets can be used for EEG BCI purposes, such as in
one article [57] which discusses the use of wavelets for pre-processing EEG
signals and compares them to ICA, finding better results with the wavelet
transform.

##### Fast-fourier transform

FFT is an transformation algorithm which transforms a repeating signal from
the time domain into the frequency domain, using a composition of sine and
cosine waves. This allows for the analysing of signals with regards to how
they are repeating rather than how they are changing with time. This
transformation allows for the specific bands, alpha, beta, delta, and theta,
to be analysed [21] [58].

####  2.3.2 Machine learning

The specific type of Machine Learning relevant to this EEG BCIs is
classification, in which a set of inputs is given and the model tells you what
type of data has been given to it. The models discussed here are K-Nearest
Neighbours, Linear Discriminant Analysis, and Artificial Neural Networks.

##### K-nearest neighbours

K-Nearest Neighbours (KNN) is a classification algorithm which compares the
input data with the training data and predicts the classification dependent on
the closest k neighbours to the input. This algorithm requires all training
data be loaded into memory, and hence can be quite resource intensive with
large sets of training data. This algorithm has been used for BCI tasks,
including recognition of emotion [59, 60] and motor imagery [61].

##### Linear discriminant analysis

Linear Discriminant Analysis (LDA) is another classification machine learning
algorithm which utilises the linear discriminant to split data into groups by
reducing the system dimensions. In order to do this, a hyper-plane is created
to project data onto in a way which maximises the distance between the centre
of each classification while minimising the spread. LDA is often utilised for
motor imagery [62], but has also been shown to work well for emotion
classification [63].

##### Artificial neural networks

Artificial Neural Networks (ANNs) are a machine learning model which tries to
replicate the neural function of a brain [64]. ANNs are often used for
classification and regression. The generic layout of an ANN can be seen in
Fig. 5. With this architecture, an input layer is given the input data, which
is multiplied by some weight with a bias added to it, and run through some
activation function. This value is then sent to the nodes in the next layer,
and so on.

Figure 5: Artificial Neural Network (ANN) architecture [65].

Within ANNs, there is a subcategory called deep learning, in which an ANN has
many layers, called Deep Neural Networks (DNNs) [66]. Typically, DNNs require
much more training data in order to get sufficient results, but are able to
understand more complex systems.

There are many types of deep learning, including but not limited to
convolutional neural networks (CNNs) and recurrent neural networks (RNNs).
Many papers discuss the use of different types of DNNs for classifying EEG
data [67, 68]. DNNs also allow for transfer learning, where a large dataset
can be used to train the model, and a smaller dataset can be used to shape the
model for the specific use case. This is done by locking some of the weights
and biases in the nodes of the network and retraining on the new data. One
article [69] discusses the use of DNNs for general purpose EEG data transfer
learning, allowing for EEG modelling between differing hardware, tasks, and
subjects with a single model.

One article [70] discusses the use of neural network transfer learning for
emotion detection of users, helping get a deeper understanding of how transfer
learning can be utilised in different studies for a better emotional
understanding of experimental subjects. Another article [71] discusses the
successful use of transfer learning for motor imagery BCI applications,
referencing EEG optical flow and image processing using common image
classification CNNs such as VGG16 [72] and AlexNet [73].

CNNs are a type of neural network in which has some convolutional layers.
These convolutional layers are able to extract features from the input data,
allowing for more optimised classification, most typically in the context of
image processing. Several papers [74, 75] discuss the use of CNNs to analyse
EEG signals for emotional classification and epileptic seizure detection, both
of which could properly classify their targets with over an 80% accuracy. One
paper [76] also discusses the use of a CNN for mental command purposes to
control a robotic arm. This study discusses the ability to control the robotic
arm with 8 separate commands with an accuracy of around 92%.

##  3 System design and development

In this section, we discuss the design and development of the EEG-mechatronic
system interface. The following are discussed: the selection of the consumer-
grade EEG headset, the placement of the EEG electrodes, data pre-processing
methodology, and machine learning model development.

###  3.1 Headset selection

Many factors must be considered in order to determine which headset should be
used in our experiment, e.g., the type of electrode, number of electrodes,
sample rate, ADC resolution, transmission protocol, and cost. The OpenBCI
Cyton board (without the Daisy Extension board) has been chosen as the EEG
headset to be utilised for this research project. OpenBCI’s available finger-
based dry electrode is ideal for this application does not require a messy gel
and is consistent over long periods of time. The relatively high ADC
resolution in this headset is ideal to avoid a low signal to noise ratio. This
headset has demonstrated success with BCI applications, suggesting its ease of
use and reliability.

###  3.2 Electrode placement

The electrodes on the headset were be placed at 10-20 positions at the
following locations: Cz, C1, C2, C3, C4, Cp1, Cp2, and Fpz. Because the
Ultracortex Mark IV does not have electrode positions at C1 and C2, a 3D model
was designed in the Fusion 360 Computer Aided Design (CAD) program to go
between other electrodes on the headset. The model was created by creating a
frame which could reach to the beams around positions C1 and C2. After this
frame was designed, the “MECHINSERT” model provided by OpenBCI was merged with
the frame to create the screw-in threads. The model was 3D printed, allowing
for electrodes to be placed at C1 and C2, using cable ties to secure the parts
to the headset. A render of this model can be seen in Fig. 6(a). This part can
be seen mounted on the headset in Fig. 6(b).

(a) Electrode adapter CAD render.

(b) Electrode adapter on headset C1 position.

Figure 6: Electrodes C1 and C2 adapter as shown in (a) and (b). Figure 7:
Participant wearing the headset and acquiring training data in our experiment.

###  3.3 Data preprocessing

After acquiring the EEG data, preprocessing should be done in order to clean
the data and extract features. The process of filtering raw data and data
balancing are discussed as follows:

####  3.3.1 Digital data filtering

EEG signals are very prone to environmental noise from varying electromagnetic
sources, so several filters were applied to the signals. In order to develop a
digital filter for the EEG data, multiple filters are combined then
quanitised. The filter used in this experiment is a combination of a low-pass
filter to remove noise above the frequencies shown in EEG signals, a high-pass
filter to remove low-frequency/DC noise, and a notch filter to remove noise
from mains electricity. The resulting Infinite-Impulse Response (IIR) filter
is of the form seen in Equation 1, where $x(t)$ is the input signal voltage at
time $t$, $a-n$ and $b-n$ are the denominator and numerator coefficients
respectively for the filter’s discrete transfer function, $\Delta t$ is the
time-step between subsequent signal acquisition, and $y(t)$ is the output
signal voltage at time $t$. A sample of EEG data filtered in this IIR filter
can be seen in Fig. 8.

$y(t)=b-0x(t)+b-1x(t-\Delta t)+...-a-1y(t-\Delta t)-...-a-4y(t-4\Delta t)$ (1)

Figure 8: Filtered EEG data compared to unfiltered data.

####  3.3.2 Data balancing

In order to have the best results, classification machine learning models
generally work best with balanced data, meaning their training data has close
to an equal number of samples for each classification. To ensure the data was
balanced for the machine learning training, the count of each classification
in the training samples was considered. The count of the classifier with the
least number of samples was saved and that number of samples is randomly taken
from each classifier to ensure the training data was balanced.

###  3.4 Selection of machine learning models

Different machine learning models (e.g., KNN, LDA, CNN) were tested throughout
the development of this system, each with its own merits. In order to validate
each machine learning model, EEG data, in an FFT format, was provided as the
input for the model alongside the button pressed from the user, which was the
desired model output. This data was split at a 70/30 ratio, where 70% of the
data was used to train the model and 30% was utilised to test the model.

####  3.4.1 CNN architecture

In order to find the most effective CNN architecture, many different hyper-
parameters were tested. The architectures which were tested all started with a
convolutional layer (50 filters, kernel size of 4, and sigmoid activation
function), followed by a batch normalisation, and a max pooling (size of 2).
These 3 layers were then added again for each desired number of convolutions
$n$. After the convolutional layers were added, a flatten layer was added,
followed by a dense layer with length $l$ and a sigmoid activation. Finally, a
dense layer with 4 nodes was added for the outputs, with a softmax activation.
This means there were two hyper-parameters that were tuned: the number of
convolutions $n$ along with the dense layer length $l$.

Figure 9: Architecture diagram of CNN. The layers surrounded by a blue box can
be optionally repeated to create more convolutional layers, batch
normalisation layers, and max pooling layers, which would also impact the
flatten layer’s length. The length of the blue dense layer can also be
adjusted with the hyper-parameters.

This machine learning model can be trained with 10% of the training data, 100%
of the training data, and a specific participant’s training data and the
models can be saved. This is done with 1 convectional layer up to 4
convolutional layers, and with a dense layer length of 100, 200, 400, 800,
1600, and 3200, resulting in 24 unique CNN models for each training dataset,
resulting in a total of 72 unique CNN models.

##  4 Experimental procedures

In this section, we present the experimental procedures for the experiments
involved in this paper. The following are discussed: participant selection and
demographics, experimental methodology, headset configuration, training data
collection, machine learning training, and post-machine learning training
procedures.

###  4.1 Participants selection

We aim to enable multiple participants to control a mechatronic system in a
multi-week experiment. The training data collection portion of the experiment
had a total of 19 participants, 6 female and 13 male. The participants ranged
from 18 to 67 years old with a median of 23 years and a standard deviation of
12 years. Two of these participants had experience with EEG usage, while the
rest were new to this area of research. The final validation portion of the
experiment had 2 participants who both completed the training data portion of
the experiment, who were both males aged 21 and 22 with experience with EEG
usage.

###  4.2 Experimental methodology

We note that motor imagery is the selected BCI methodology for this
experiment. The experiment consisted of two main portions: (i) the training
data collection, and (ii) model validation, with a machine learning model
training step in between the two. The training portion of the experiment
consisted of a computer setup connected to the Cyton board via a bluetooth
connection. The goal of this portion of the experiment was to collect motor-
imagery training data to later consolidate and feed into a machine learning
model. The machine learning model training portion of the experiment had the
goal of combining all of the training data collected from all participants and
training models to later determine which model best suits the given problem.
The model validation portion of the experiment again consisted of a computer
setup and the headset. The model validation read new data from the user
wearing the headset and used transfer learning to re-train the machine
learning models, which was followed by determining which model best suits the
problem.

####  4.2.1 Headset configuration

In each 5-minute experimental session the participant had the headset placed
on their head by the Experiment Investigator (EI). The electrode units were
then be adjusted until each electrode was seated on the scalp of the
participant. This process was done by running the “OpenBCI GUI” program to
validate each channel is showing active brainwaves. Once the headset was
properly seated, the EI followed the steps required to begin the data
acquisition program. A comparison of poor electrode contact and good electrode
contact can be seen in Fig. 10, where poor contact is defined as when any of
the 8 channels are “Railed 100%”.

(a) Poor electrode contact. Channels 2, 3, 7, and 8 are “Railed 100%”.

(b) Good electrode contact. No channels are “Railed 100%”.

Figure 10: OpenBCI GUI electrode contact.

####  4.2.2 Training data collection

In order to gather training data for the machine learning models, an
experiment was made to provide information on the user’s motor cortex when
specific motor movements are performed. To get this data, the users were
instructed to play a computer game which requires them to press keyboard keys
with their hands. While the user was playing the video game, the EEG data was
streamed to the computer, processed, and saved for future training sessions.

Data was acquired while the participant was playing a 2-dimensional video game
in which the user will move left or right by pressing the left and right
‘control’ keys on a keyboard with fingers on their left and right hands
respectively. A screenshot of this program can be seen in Fig. 11. In this
game, the participant was able to move a bar left and right, and was trying to
catch a falling box. Once the box hit the bar, or the box hit the bottom of
the screen, a new box appeared at a random point at the top of the screen.
This game and data acquisition was run for 5 minutes for each participant.

Figure 11: Screenshot of the 2D game which the user plays. The blue square box
falls downward continuously, and the user tries to move the red bar to catch
the box. Once the blue box collides with the red bar or reaches the bottom of
the screen, a new box appears randomly at the top of the screen.

After the first portion of the training data gathering, the training data was
then fit into a machine learning model, with a training-test split of 70-30
respectively. The 70% training data was used to train the machine learning
model, and the remaining 30% was used to provide a model accuracy to validate
the model. This process generally only took 10-90 seconds, depending on the
model used, thus the user would remain seated for the next portion of the
experiment while this was occurring.

Once the machine learning model was built, a 1-minute demonstration of the
machine learning model was conducted. During this period, the participant
continued to press the buttons on the keyboard, but the output of the machine
learning model controlled the game rather than the button presses. This
demonstrated how well the model trained to the user with their current headset
configuration.

####  4.2.3 Data consolidation and machine learning training

Once a total of 24 5-minute sessions were recorded, the data was be
consolidated into one dataset for machine learning training. Two forms of
consolidation take place: one which consolidated 100% of each session of data,
and another where only 10% of each historical session was taken. Many
variations of the machine learning models were trained in order to determine
the most optimal architecture.

####  4.2.4 Post machine learning training procedures

Once the training-data collection portion of the experiment was complete and
the machine learning models were built, the models were validated against live
data. This was be done with the model validation portion of the experiment.

In this portion of the experiment, the headset first needed to be configured
on a participant, as in Section 4.2.1. After configuring the headset, the
participant played the game described in Section 4.2.2 for 30 seconds while
the data was gathered. After gathering data, machine learning models with
transfer learning capabilities were trained with the newly acquired data,
while models without transfer learning capabilities were trained only with
this 30 seconds of training. After training the model, a 30-second validation
period of the game began, where the EEG data being fed into the machine
learning model controlled the game. After each model was tested a form
discussing model responsiveness was completed by the participant.

The model accuracy, number of boxes caught, number of boxes caught in a row,
and user responsiveness were compared between each model to determine which
best suits this EEG-mechatronic system interface.

##  5 Results and performance evaluation

In this section, we present the achieved results and discuss the performance
of our system. As stated in Section 3.4, many forms of machine learning
classification algorithms were compared throughout this experiment, including
KNN, LDA, and CNN.

###  5.1 Results from KNN model

This model was utilised in the beginnings of the development of this
experiment. During testing, it was noticed that this model had overfitting
issues. Whenever training data would be fit to the model, a very high testing
accuracy would be seen (greater than 95%), but when new data were provided,
such as with live EEG data, the model would perform with significantly lower
accuracy (around 40-45%).

This overfitting was likely due to the data being relatively similar from
sample to sample, as an FFT representation of data does not change
significantly in a single time-step. In order to fix this overfitting issue,
less of each data sample should be used (e.g. 1% of the recorded data) and
more sample data should be provided.

The KNN classification algorithm also requires all training data be loaded
into the computer’s memory for predicting values from new data. This means the
more training data is loaded, the more system memory is required. With
gigabytes of training data gathered from this experiment, it is a challenge to
create a model with all of the training data.

####  5.1.1 KNN model validation

During the machine learning validation portion of the experiment, only the
data recorded in the data collection portion of the experiment (the 30 seconds
prior to training) were used to train the model. Even with this limitation,
the model appears to respond some amount to the EEG signals, see Table 3. Of
the 10 trials of KNN validation between two participants, an average of 1.5
boxes were caught per session, with an average user rating of 2.2/5.

Table 3: KNN training accuracy, number of boxes caught during testing, and the user rating from the model. Trial |  | Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
| P2 | P6 | P2 | P6 | P2 | P6
1 | 0.97 | 0.86 | 1 | 2 | 2 | 3
2 | 0.95 | 0.88 | 0 | 0 | 1 | 1
3 | 0.89 | 0.89 | 0 | 2 | 2 | 3
4 | 0.88 | 0.88 | 2 | 3 | 2 | 3
5 | 0.89 | 0.91 | 0 | 5 | 1 | 4
Average | 0.90 | 1.5 | 2.2

###  5.2 Results from LDA model

An LDA was also utilised in the beginnings of the development of this
experiment. This classification algorithm resulted in a lower training
accuracy compared to the KNN algorithm, but displayed higher validation
accuracy with new data.

####  5.2.1 LDA model validation

Similarly to the KNN model validation, only the 30 seconds of training data
were used for the LDA model validation because of the lack of transfer
learning. During the machine learning validation portion of the experiment,
participants caught an average of 1.3 boxes per session, and gave the
responsiveness an average of 2.4/5, see Table 4.

Table 4: LDA training accuracy, number of boxes caught during testing, and the user rating from the model. Trial |  | Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
| P2 | P6 | P2 | P6 | P2 | P6
1 | 0.56 | 0.39 | 0 | 0 | 1 | 2
2 | 0.39 | 0.43 | 2 | 3 | 2 | 4
3 | 0.44 | 0.39 | 2 | 1 | 3 | 3
4 | 0.34 | 0.47 | 0 | 1 | 2 | 2
5 | 0.53 | 0.53 | 2 | 2 | 2 | 3
Average | 0.45 | 1.3 | 2.4

###  5.3 Results from CNN model

A CNN was also used as a classification algorithm in this experiment. In order
to train the various CNN architectures, a high specification server would be
required for various reasons discussed in Section 6.3.

####  5.3.1 CNN model with 10% of training data

24 different CNN models were trained with 10% of the training data. A plot of
model accuracy vs number of dense layer nodes can be seen in Fig. 12. It can
be seen that as the number of dense layer nodes increases from 100 to around
1600 the accuracy increases for 1, 2, and 3 convolutional layers, while the
accuracy for 4 convolutions remained around a 70% accuracy for all dense layer
nodes. It can also be seen that for all dense layer lengths, 1 or 2
convolutional layers had the highest accuracy. Additionally 2 convolutional
layers have the highest average accuracy and the lowest average service units,
suggesting that 2 convolutional layers may be the best out of these CNN
models.

Figure 12: Computational service units and accuracy vs number of dense layer
nodes and convolutional layers with 10% training dataset.

####  5.3.2 CNN model with full training data

The CNN architectures were also trained with the full training dataset. The
accuracy of each model can be seen in Fig. 13. In this, it can be seen that 1
and 2 convolutional layers again have the highest accuracy. A clear upward
trajectory can be also seen for all number of convolutional layers up until
around 800 dense layer nodes. It can also be seen that the processing time
increases with the number of convolutional layers.

Figure 13: Computational service units and accuracy vs number of dense layer
nodes and convolutional layers with full training dataset.

####  5.3.3 CNN model with specific user training data

In addition to training the CNN architectures with training data from all
users, the models were also trained specifically using only the data from the
participant with the most training data. The accuracy of each model can be
seen in Fig. 14. In this, it can be seen that 2 and 3 convolutional layers
again have the highest accuracy. No obvious trend is visible in this figure,
other than a possible increase in 3 convolutional layers accuracy with an
increase in number of dense layer nodes.

Figure 14: Computational service units and accuracy vs number of dense layer
nodes and convolutional layers with participant 2 training dataset.

####  5.3.4 CNN model validation

Because there were 48 different CNN models, 48 validation experiments were
required. The 10% models were tested on two participants, while the full model
was only tested on one participant.

The results for 10% model can be seen in Table 5(d). It can be seen that 2
convolutional layers had the highest average user rating between the two
subjects (2.6/5), and also had the highest average number of boxes caught (1.9
boxes). Within the models with 2 convolutional layers, the model with the
highest average user rating was from the model with 100 dense layer nodes
(4/5), while the highest average number of boxes caught was from the model
with 3200 dense layer nodes (4 boxes).

Table 5: CNN model validation for 1-4 convolutional layers and dense node
numbers from 100-3200 with 10% of training data.

(a) 1 convolutional layer

| Dense
---
Number
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
| P2 | P6 | P2 | P6 | P2 | P6
100 | 0.94 | 0.96 | 0 | 1 | 2 | 1
200 | 0.81 | 0.87 | 4 | 2 | 4 | 3
400 | 0.85 | 0.83 | 1 | 1 | 2 | 2
800 | 0.49 | 0.76 | 1 | 0 | 1 | 1
1600 | 0.88 | 0.92 | 1 | 0 | 2 | 2
3200 | 0.73 | 0.89 | 4 | 1 | 4 | 2
Average | 0.83 | 1.3 | 2.2

(b) 2 convolutional layers

| Dense
---
Number
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
| P2 | P6 | P2 | P6 | P2 | P6
100 | 0.81 | 0.85 | 4 | 0 | 5 | 3
200 | 0.86 | 0.84 | 5 | 0 | 3 | 2
400 | 0.88 | 0.96 | 2 | 0 | 2 | 2
800 | 0.78 | 0.84 | 1 | 0 | 1 | 2
1600 | 0.88 | 0.81 | 0 | 3 | 1 | 4
3200 | 0.74 | 0.85 | 6 | 2 | 3 | 3
Average | 0.84 | 1.9 | 2.6

(c) 3 convolutional layers

| Dense
---
Number
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
| P2 | P6 | P2 | P6 | P2 | P6
100 | 0.88 | 0.95 | 0 | 1 | 1 | 2
200 | 0.65 | 0.89 | 3 | 3 | 2 | 3
400 | 0.80 | 0.95 | 0 | 1 | 1 | 4
800 | 0.89 | 0.89 | 1 | 0 | 1 | 2
1600 | 0.85 | 0.87 | 0 | 1 | 2 | 3
3200 | 0.87 | 0.94 | 5 | 0 | 4 | 3
Average | 0.87 | 1.3 | 2.3

(d) 4 convolutional layers

| Dense
---
Number
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
| P2 | P6 | P2 | P6 | P2 | P6
100 | 0.93 | 0.93 | 0 | 4 | 1 | 4
200 | 0.97 | 0.94 | 0 | 1 | 2 | 2
400 | 0.80 | 0.93 | 2 | 0 | 2 | 3
800 | 0.89 | 0.81 | 3 | 2 | 3 | 3
1600 | 0.88 | 0.88 | 1 | 4 | 2 | 4
3200 | 0.87 | 0.93 | 1 | 1 | 1 | 2
Average | 0.89 | 1.2 | 1.8

The results for full training-data model can be seen in Table 6(d). It should
also be noted these models took significantly longer to train than the 10%
models. Because of this longer training time, and because the accuracy of each
model and the number of boxes caught was lower than the 10% models, these
models were only tested on one subject. We see that 2 convolutional layers had
the highest average user rating between the two subjects (2.6/5), and also had
the highest average number of boxes caught (1.9 boxes). Within the models with
2 convolutional layers, the model with the highest average user rating was
from the model with 100 dense layer nodes (4/5), while the highest average
number of boxes caught was from the model with 3200 dense layer nodes (4
boxes).

Table 6: CNN model validation for 1-4 convolutional layers and dense node
numbers from 100-3200 for full training data.

(a) 1 convolutional layer

| Dense
---
Number
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
100 | 0.56 | 0 | 1
200 | 0.63 | 2 | 3
400 | 0.67 | 2 | 2
800 | 0.67 | 1 | 2
1600 | 0.70 | 3 | 3
3200 | 0.82 | 0 | 1
Average | 0.68 | 1.33 | 2.00

(b) 2 convolutional layers

| Dense
---
Number
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
100 | 0.26 | 1 | 1
200 | 0.76 | 0 | 1
400 | 0.75 | 0 | 1
800 | 0.77 | 1 | 1
1600 | 0.67 | 0 | 1
3200 | 0.74 | 4 | 4
Average | 0.66 | 1.00 | 1.50

(c) 3 convolutional layers

| Dense
---
Number
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
100 | 0.26 | 2 | 2
200 | 0.71 | 0 | 2
400 | 0.78 | 4 | 2
800 | 0.68 | 3 | 3
1600 | 0.88 | 0 | 1
3200 | 0.70 | 1 | 2
Average | 0.67 | 1.67 | 2.00

(d) 4 convolutional layers

| Dense
---
Number
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
100 | 0.88 | 2 | 4
200 | 0.82 | 0 | 2
400 | 0.80 | 3 | 2
800 | 0.73 | 0 | 2
1600 | 0.84 | 0 | 2
3200 | 0.78 | 2 | 2
Average | 0.81 | 1.17 | 2.33

The results for the user specific training-data model can be seen in Table
7(d). It can be seen that 3 convolutional layers had the highest average user
rating by far with 3.3/5 along with the highest average number of boxes caught
at 2.8 per session.

Table 7: CNN model validation for 1-4 convolutional layers and dense node
numbers from 100-3200 for participant 2’s training data.

(a) 1 convolutional layer

| Dense
---
Length
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
100 | 0.57 | 2 | 2
200 | 0.51 | 1 | 1
400 | 0.62 | 0 | 1
800 | 0.57 | 0 | 1
1600 | 0.62 | 0 | 1
3200 | 0.80 | 1 | 1
Average | 0.61 | 0.67 | 0.67

(b) 2 convolutional layers

| Dense
---
Length
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
100 | 0.56 | 3 | 2
200 | 0.66 | 1 | 3
400 | 0.53 | 1 | 2
800 | 0.81 | 2 | 2
1600 | 0.66 | 2 | 3
3200 | 0.69 | 1 | 1
Average | 0.65 | 1.67 | 2.17

(c) 3 convolutional layers

| Dense
---
Length
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
100 | 0.60 | 2 | 4
200 | 0.65 | 5 | 5
400 | 0.69 | 1 | 2
800 | 0.78 | 5 | 3
1600 | 0.75 | 2 | 3
3200 | 0.73 | 2 | 3
Average | 0.70 | 2.83 | 3.33

(d) 4 convolutional layers

| Dense
---
Length
| Training
---
Accuracy
| Boxes
---
Caught
| User
---
Rating
100 | 0.70 | 2 | 5
200 | 0.63 | 0 | 1
400 | 0.72 | 1 | 1
800 | 0.81 | 2 | 2
1600 | 0.78 | 1 | 2
3200 | 0.73 | 1 | 2
Average | 0.73 | 1.17 | 2.17

##  6 Discussion and lessons learned

In this section, we analyse the results from Section 5. We also provide the
major findings of the results, and the limitations of our study.

###  6.1 Machine learning model analysis

After performing the experiments, the results of KNN, LDA, and all
configurations of the CNN should be compared and discussed. Each of these
models will be analysed individually, with the final results compared between
each configuration.

####  6.1.1 Analysis of KNN model

As seen in Table 3, the KNN machine learning model had an average training
accuracy of 90%, with an average number of boxes caught by the user of 1.5 per
session and an average user rating of responsiveness of 2.2/5. It should also
be noted the two participants had significantly different results when
comparing the user ratings and number of boxes caught per session. During
model validation, participant 2 was only able to catch an average of 0.6 boxes
per session and gave the model an average responsiveness rating of 1.6/5,
while participant 6 was able to catch an average of 2.4 boxes per session and
gave an average rating of 2.8/5. Relative to the results of the other machine
learning models, this model is respectable but not the best. One apparent
advantage of this model is the lack of significant training data required to
get some result, as only 30 seconds of training data were required to get the
results discussed above.

####  6.1.2 Analysis of LDA model

As seen in Table 4, the LDA machine learning model had an average training
accuracy of 45%, with an average number of boxes caught by the user of 1.3 per
session and an average user rating of responsiveness of 2.4/5. There was only
a slight difference between the two participants’ number of boxes caught with
the LDA compared to the KNN, as participant 2 caught an average of 1.2 boxes
per session and gave an average rating of 2/5, while participant 6 caught an
average of 1.4 boxes per session and gave an average rating of 2.8. Similarly
to the KNN model, the LDA is able to give some results with minimal training
data, but is not able to utilise large amounts of training data.

####  6.1.3 Analysis of CNN models

From Table 5(d), it can be seen there is a larger difference between
performance of the model between the two participants for most of the model
configurations. Between all of the CNN architectures, participant 2 caught an
average of 1.9 boxes and gave an average rating of 2.2, while participant 6
caught an average of 1.2 boxes but gave an average user rating of 2.6. Because
participant 6 gave a higher average rating but caught fewer boxes on average,
it is likely the system was responding reasonably well but was nearly missing
catching the boxes. Conversely, for participant 2, the higher number of boxes
caught but the lower average user rating suggest the system may have responded
well some of the time but not for others, allowing for box catches while the
system was responding well, but missing the boxes during the rest of the time.

It can also be seen that the CNN which was trained specifically for
participant 2 had some of the highest number of boxes caught and user ratings
from the models with 3 convolutional layers. This suggests CNNs which are
specially trained for each user would perform better compared to the models
which are trained with every user’s data. This would be expected, as each
individual’s brain works differently and a one-size-fits-all solution would
likely be more difficult than a tailored solution. Further studies are
required to further validate this theory. It should also be noted that there
does not appear to be a significant difference between the results dependent
on the number of nodes in the parameterised dense layer.

####  6.1.4 Summary of models

From the machine learning results seen in Tables 3, 4, 5(d), 6(d), 7(d), it
can be seen that overall, CNNs had the highest average number of boxes caught
per session, suggesting it is best for this application. More specifically, a
CNN with 3 convolutional layers which is specially trained for each
participant of the experiment is ideal.

###  6.2 Pre-processing

This experiment was designed with an FFT in mind for the main preprocessing
component. While FFT is common for BCI paradigms, other forms of preprocessing
have also been shown to be viable for BCI applications, see Section 2.3.1.

Particularly, ICA and wavelet transform preprocessing methods should be
considered. Because ICA can separate components from the signals, it could be
utilised to determine when specific actions occur. Utilising the ICA could
help separate and remove artefacts from the waveforms, such as from eye
blinks, face twitches, or other unrelated movements. Separately from the ICA,
a wavelet transform could also be used to extract features from EEG signals,
as it has also been widely used within EEG BCI tasks.

In order to reduce overfitting of training data, another form of train-test
split could be used. Within the scope of this experiment, a randomised train-
test split was done. This means that after the training data was collected, a
random 70% was utilised for training and the remaining 30% was used to test
the model. This means that two consecutive datapoints could be utilised in the
training and testing of the model, causing overfitting. If a temporal train-
test split was done, where the first 70% of the time-sorted data were used to
train the model and the last 30% of the data were used to test the model,
overfitting could be reduced.

###  6.3 Limitations

Due to the high density of the EEG data, several gigabytes of data were
recorded. Because certain machine learning models require all training data to
be loaded into the computer memory even after training, it is difficult to
utilise the complete training data for specific models on a standard-
specification laptop. Because of this limitation, training of the CNN models
was performed on a supercomputer.

##  7 Related Work

In this section, we provide a discussion of related works. Specifically, we
intend to contrast similar papers with our own and discuss relevant
similarities and our novel implementations. For example, proposal [77]
discusses the use of a 118-channel EEG headset to implement a motor-imagery
BCI system which can control a lower-limb exoskeleton. The system discussed
was able to determine desired motions of the exoskeleton based on historical
EEG data from the BCI competition III [78]. Because this data was from a
competition and not from a live system, the results are likely to differ from
a system in the real, live world.

In [79], authors discuss the use of steady-state visually evoked potentials
(SSVEPs) to control a prosthetic arm using the EMOTIV EPOC +, a 16-channel wet
EEG headset. The described system is able to utilise the SSVEP to detect
commands from the EEG data using offline testing. This system is limited
similarly to [77] because it was not implemented with online/live EEG data.

Proposal [80] presents an analysis of EEG data to determine the cognitive
state of airplane pilots. This would then be used to perform safety manoeuvres
in the event of decreased cognitive activity or loss of consciousness. While
this system is able to determine the cognitive state with high accuracy and
could be used in conjunction with the research done in this article, it does
not discuss an active control methodology for the controlled system.

Other proposals, e.g., [81], [82], and [83] discuss the use of an EEG motor
imagery BCI system for a motor imagery BCI classification. These proposals
discuss high machine learning classification accuracy, but do not discuss
performance with online EEG experimentation or the corresponding activation
accuracy, whereas the research done in our paper focuses in this specific
area.

The use of transfer learning for EEG motor imagery tasks is discussed in
additional proposals, e.g., [84] and [85]. These proposals highlight the
differences between participants and difficulties with transferring learning
between them. This issue is addressed within our research as participants have
machine learning models built with all participants data, along with models
for individual participants as well.

In summary, unlike the existing proposals, in our paper, we utilise online EEG
data classification for motor imagery BCI. This allows for a true
understanding of the performance of any machine learning models built for our
system. Additionally in our paper, transfer learning is applied by using
historical data of individuals, avoiding any issues with differences between
participants.

##  8 Conclusion and future work

In this paper, to answer the question ‘how can consumer-grade EEG devices be
used to control mechatronic systems’, an EEG BCI system was developed with the
usage of the OpenBCI Cyton board and a user-interface running a game. Training
data was gathered from real-world participants, and several machine learning
models were built. After machine learning models were built, participants went
through a model validation experiment where they played the game, gathered
smaller amounts of training data, and applied transfer learning with this data
to reconfigure the model to their specific head and placement of the headset.
With this configuration, KNN, LDA, and CNN models all were able to control the
system, with the CNN performing best out of the three in terms of user rated
responsiveness. The outputs of the CNN model could further be used to control
a mechatronic system. One way to improve design is to predict the intention to
move muscles rather than their movements. Further, developing an intent-based
motor imagery BCI would allow the system to be run while performing other
motor actions without false positives, giving the system further versatility.
However, these are separate research directions, and we leave these for future
works.

## References

## References

  * [1] X. Gu, Z. Cao, A. Jolfaei, P. Xu, D. Wu, T. Jung, C. Lin, Eeg-based brain-computer interfaces (bcis): A survey of recent studies on signal sensing technologies and computational intelligence approaches and their applications, IEEE/ACM transactions on computational biology and bioinformatics (2021) 1165–1176. https://doi.org/10.1109/TCBB.2021.3052811
  * [2] D. Pal, S. Palit, A. Dey, Brain computer interface: A review, Computational Advancement in Communication, Circuits and Systems (2022) 25–35. https://doi.org/10.1007/978-981-16-4035-3_3
  * [3] A. Coenen, E. Fine, O. Zayachkivska, Adolf beck: A forgotten pioneer in electroencephalography, Journal of the History of the Neurosciences 23 (3) (2014) 276–286. https://doi.org/10.1080/0964704X.2013.867600
  * [4] R. Ince, S. Adanir, F. Sevmez, The inventor of electroencephalography (eeg): Hans berger (1873-1941), Child’s nervous system 37 (9) (2021) 2723–2724. https://doi.org/10.1007/s00381-020-04564-z
  * [5] H. Korkalainen, J. Aakko, S. Nikkonen, S. Kainulainen, A. Leino, B. Duce, I. Afara, S. Myllymaa, J. Töyräs, T. Leppänen, Accurate deep learning-based sleep staging in a clinical population with suspected obstructive sleep apnea, IEEE Journal of Biomedical and Health Informatics 24 (7) (2020) 2073–2081. http://dx.doi.org/10.1109/JBHI.2019.2951346
  * [6] M. Murugesan, R. Sukanesh, Automated detection of brain tumor in eeg signals using artificial neural networks, 2009 International Conference on Advances in Computing, Control, and Telecommunication Technologies (2009) 284–288. http://dx.doi.org/10.1109/ACT.2009.77
  * [7] Z. Chen, G. Lu, Z. Xie, W. Shang, A unified framework and method for eeg-based early epileptic seizure detection and epilepsy diagnosis, IEEE Access, 8 (2020) 20080–20092. http://dx.doi.org/10.1109/ACCESS.2020.2969055
  * [8] U. Orhan, M. Hekim, M. Ozer, I. Provaznik, Epilepsy diagnosis using probability density functions of eeg signals, 2011 International Symposium on Innovations in Intelligent Systems and Applications (2011) 626–630. http://dx.doi.org/10.1109/INISTA.2011.5946171
  * [9] S. Pal, Internet of Things and Access Control: Sensing, Monitoring and Controlling Access in IoT-Enabled Healthcare Systems, Springer, Springer Nature 37 (2021) 205. http://doi.org/10.1007/978-3-030-64998-2
  * [10] S. Pal, M. Hitchens, V. Varadharajan, T. Rabehaja, Fine-grained access control for smart healthcare systems in the internet of things, EAI Endorsed Transactions on Industrial Networks and Intelligent Systems 4 (13) (2018). http://dx.doi.org/10.4108/eai.20-3-2018.154370
  * [11] A. Sreedevi, T. Harshitha, V. Sugumaran, P. Shankar, Application of cognitive computing in healthcare, cybersecurity, big data and iot: A literature review, Information Processing & Management 59 (2) (2022) 102888. https://doi.org/10.1016/j.ipm.2022.102888
  * [12] S. Pal, M. Hitchens, V. Varadharajan, Access control for internet of things—enabled assistive technologies: An architecture, challenges and requirements, Assistive technology for the elderly, Elsevier, (2020) 1–43. https://doi.org/10.1016/B978-0-12-818546-9.00001-4
  * [13] M. Hosseini, D. Pompili, K. Elisevich, H. Soltanian-Zadeh, Optimized deep learning for eeg big data and seizure prediction bci via internet of things, IEEE Transactions on Big Data 3 (4) (2017) 392–404. http://doi.org/10.1109/TBDATA.2017.2769670
  * [14] K. Chicaiza, M. Benalcázar, A brain-computer interface for controlling iot devices using eeg signals, 2021 IEEE Fifth Ecuador Technical Chapters Meeting (ETCM), IEEE, (2021) 1–6. http://doi.org/10.1109/ETCM53643.2021.9590711
  * [15] M. Siegel, The sense-think-act paradigm revisited 1st International Workshop on Robotic Sensing, 2003. ROSE’03., IEEE, (2003) 5. http://doi.org/10.1109/ROSE.2003.1218700
  * [16] A. Yudhana, S. Mukhopadhyay, I. Karas, A. Azhari, M. Mardhia, S. Akbar, A. Muslim, F. Ammatulloh, Recognizing human emotion patterns by applying fast fourier transform based on brainwave features, 2019 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS), IEEE, (2019), 249–254. http://doi.org/10.1109/ICIMCIS48181.2019.8985227
  * [17] S. Aggarwal, N. Chugh, Review of machine learning techniques for eeg based brain computer interface, Archives of Computational Methods in Engineering (2022) 1–20. https://doi.org/10.1007/s11831-021-09684-6
  * [18] J. Zhang, M. Wang, A survey on robots controlled by motor imagery brain-computer interfaces, Cognitive Robotics 1 (2021) 12–24. https://doi.org/10.1016/j.cogr.2021.02.001
  * [19] J. Mahendra Kumar, M. Rashid, R. Musa, M. Mohd Razman, N. Sulaiman, R. Jailani, A. PP Abdul Majeed, The classification of wink-based eeg signals: The identification of significant time-domain features, Advances in Mechatronics, Manufacturing, and Mechanical Engineering, Springer, (2021) 283–291. https://doi.org/10.1007/978-981-15-7309-5_28
  * [20] S. Pal, S. Mukhopadhyay, N. Suryadevara, Development and progress in sensors and technologies for human emotion recognition, Sensors 21 (16) (2021) 5554. https://doi.org/10.3390/s21165554
  * [21] M. Sunny, N. Afroze, E. Hossain, Eeg band separation using multilayer perceptron for efficient feature extraction and perfect bci paradigm, 2020 Emerging Technology in Computing, Communication and Electronics (ETCCE), (2020) 1–6. http://dx.doi.org/10.1109/ETCCE51779.2020.9350883
  * [22] S. Rasheed, A review of the role of machine learning techniques towards brain–computer interface applications, Machine Learning and Knowledge Extraction 3 (4) (2021) 835–862. https://doi.org/10.3390/make3040042
  * [23] S. Abdulkader, A. Atia, M. Mostafa, Brain computer interfacing: Applications and challenges, Egyptian Informatics Journal 16 (2) (2015) 213–230. https://doi.org/10.1016/j.eij.2015.06.002
  * [24] H. Gonzalez, R. George, S. Muzaffar, J. Acevedo, S. Hoeppner, C. Mayr, J. Yoo, F. Fitzek, I. Elfadel, Hardware acceleration of eeg-based emotion classification systems: A comprehensive survey, IEEE Transactions on Biomedical Circuits and Systems 15 (3) (2021) 412–442. http://doi.org/10.1109/TBCAS.2021.3089132
  * [25] R. Oostenveld, P. Praamstra, The five percent electrode system for high-resolution eeg and erp measurements, Clinical Neurophysiology 112 (4) (2001) 713–719. http://dx.doi.org/https://doi.org/10.1016/S1388-2457(00)00527-7
  * [26] G. Flumeri, P. Arico, G. Borghini, N. Sciaraffa, A. Florio, F. Babiloni, The dry revolution: Evaluation of three different eeg dry electrode types in terms of signal spectral features, mental states classification and usability, PubMed 19 (2019) 1365. http://dx.doi.org/10.3390/s19061365
  * [27] R. Matthews, N. McDonald, H. Anumula, J. Woodward, P. Turner, M. Steindorf, K. Chang, J. Pendleton, Novel hybrid bioelectrodes for ambulatory zero-prep eeg measurements using multi-channel wireless eeg system, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 4565, (2007), 137–146. https://doi.org/10.1007/978-3-540-73216-7_16
  * [28] Y. Chi, Y.-Wang, Y. Wang, C. Maier, T. Jung, G. Cauwenberghs, Dry and noncontact eeg sensors for mobile brain–computer interfaces, IEEE Transactions on Neural Systems and Rehabilitation Engineering 20 (2) (2012) 228–235. http://dx.doi.org/10.1109/TNSRE.2011.2174652
  * [29] M. Lopez-Gordo, D. Sanchez-Morillo, F. Valle, Dry EEG electrodes, Sensors 14 (7) (2014) 12847–12870. http://dx.doi.org/10.3390/s140712847
  * [30] Insight Brainwear 5 Channel Wireless EEG Headset — EMOTIV, (n.d.). https://www.emotiv.com/insight/
  * [31] S. Krachunov, A. Casson, 3d printed dry EEG electrodes, Sensors 16 (10) (2016) 1635. https://doi.org/10.3390/s16101635
  * [32] C. Fonseca, J. Cunha, R. Martins, V. Ferreira, J. de Sa, M. Barbosa, A. Martins da Silva, A novel dry active electrode for eeg recording, IEEE Transactions on Biomedical Engineering 54 (1) (2007) 162–165. http://dx.doi.org/10.1109/TBME.2006.884649
  * [33] J. Lozano, B. Stoeber, Microspike array electrode with flexible backing for biosignal monitoring, 2019 IEEE SENSORS, (2019), 1–4. http://dx.doi.org/10.1109/SENSORS43011.2019.8956869
  * [34] K. Gao, H. Yang, L. Liao, C. Jiang, N. Zhao, X. Wang, X. Li, X. Chen, B. Yang, J. Liu, A novel bristle-shaped semi-dry electrode with low contact impedance and ease of use features for eeg signal measurements, IEEE Transactions on Biomedical Engineering 67 (3) (2020) 750–761. http://dx.doi.org/10.1109/TBME.2019.2920711
  * [35] J. Chiou, L. Ko, C. Lin, C. Hong, T. Jung, S. Liang, J. Jeng, Using novel mems eeg sensors in detecting drowsiness application, 2006 IEEE Biomedical Circuits and Systems Conference, (2006), 33–36. http://dx.doi.org/10.1109/BIOCAS.2006.4600301
  * [36] B. Grundlehner, V. Mihajlović, Ambulatory eeg monitoring, in: R. Narayan (Ed.), Encyclopedia of Biomedical Engineering, Elsevier, Oxford, (2019), 223–239. http://dx.doi.org/https://doi.org/10.1016/B978-0-12-801238-3.10885-2
  * [37] J. Katona, I. Farkas, T. Ujbanyi, P. Dukan, A. Kovari, Evaluation of the neurosky mindflex eeg headset brain waves data, 2014 IEEE 12th international symposium on applied machine intelligence and informatics (SAMI), IEEE, (2014), 91–94. http://doi.org/10.1109/SAMI.2014.6822382
  * [38] A. Khosla, P. Khandnor, T. Chand, A comparative analysis of signal processing and classification methods for different applications based on eeg signals, Biocybernetics and Biomedical Engineering 40 (2) (2020) 649–690. https://doi.org/10.1016/j.bbe.2020.02.002
  * [39] OpenBCI Featured Products — OpenBCI Online Store, (2021). https://shop.openbci.com/collections/frontpage
  * [40] Q. Wu, W. Zhao, T. Jacopo, Towards objective assessment of movie trailer quality using human electroencephalogram and facial recognition, in: 2018 IEEE International Conference on Electro/Information Technology (EIT), (2018), 0449–0452. http://dx.doi.org/10.1109/EIT.2018.8500283
  * [41] H. Do, V. Truong, K. George, B. Shirke, Eeg-based biometrics utilizing image recognition for patient identification, 2019 IEEE 10th Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON), (2019), 0591–0595. http://dx.doi.org/10.1109/UEMCON47517.2019.8992962
  * [42] A. K, S. Kuruvila, S. Pramode, N. Krupa, Brain computer interface for neurorehabilitation with kinesthetic feedback, 2020 5th International Conference on Robotics and Automation Engineering (ICRAE), (2020), 153–157. http://dx.doi.org/10.1109/ICRAE50850.2020.9310801
  * [43] T. Lal, M. Schroder, T. Hinterberger, J. Weston, M. Bogdan, N. Birbaumer, B. Scholkopf, Support vector channel selection in bci, IEEE Transactions on Biomedical Engineering 51 (6) (2004) 1003–1010. http://dx.doi.org/10.1109/TBME.2004.827827
  * [44] B. Luu, B. Hansberger, M. Chiu, V. Karigar Shivappa, K. George, Scalable smart home interface using occipitalis semg detection and classification, 2018 9th IEEE Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON), (2018), 1002–1008. http://dx.doi.org/10.1109/UEMCON.2018.8796615
  * [45] B. Meneses-Claudio, A. Roman-Gonzalez, Study of the brain waves for the identification of the basic needs of patients with amyotrophic lateral sclerosis, 2018 Congreso Argentino de Ciencias de la Informática y Desarrollos de Investigación (CACIDI), (2018), 1–6. http://dx.doi.org/10.1109/CACIDI.2018.8584193
  * [46] M. Strotzer, One century of brain mapping using brodmann areas*, Clinical Neuroradiology 19 (3) (2009) 179–186. http://dx.doi.org/10.1007/s00062-009-9002-3
  * [47] N. Gage, B. Baars, Chapter 9 - decisions, goals, and actions, in: N. M. Gage, B. J. Baars (Eds.), Fundamentals of Cognitive Neuroscience (Second Edition), second edition Edition, Academic Press, San Diego, (2018), 279–319. http://dx.doi.org/https://doi.org/10.1016/B978-0-12-803813-0.00009-X
  * [48] P. Ghimire, J. Lavrador, A. Baig Mirza, N. Pereira, H. Keeble, M. Borri, L. Furlanetti, C. Brogna, J. Jarosz, R. Gullan, F. Vergani, R. Bhangoo, K. Ashkan, Intraoperative mapping of pre-central motor cortex and subcortex: a proposal for supplemental cortical and novel subcortical maps to penfield’s motor homunculus, Brain Structure and Function 226 (5) (2021) 1601–1611. http://dx.doi.org/10.1007/s00429-021-02274-z
  * [49] J. Doležal, V. Černý, J. Štastný, Online motor-imagery based bci, 2012 International Conference on Applied Electronics, (2012), 65–68. https://ieeexplore.ieee.org/document/6328868
  * [50] F. Pichiorri, G. Morone, M. Petti, J. Toppi, I. Pisotta, M. Molinari, S. Paolucci, M. Inghilleri, L. Astolfi, F. Cincotti, D. Mattia, Brain–computer interface boosts motor imagery practice during stroke recovery, Annals of Neurology 77 (5) (2015) 851–865. http://arxiv.org/abs/https://onlinelibrary.wiley.com/doi/pdf/10.1002/ana.24390
  * [51] A. Ramos-Murguialday, D. Broetz, M. Rea, L. Läer, O. Yilmaz, F. Brasil, G. Liberati, M. Curado, E. Garcia-Cossio, A. Vyziotis, W. Cho, M. Agostini, E. Soares, S. Soekadar, A. Caria, L. Cohen, N. Birbaumer, Brain–machine interface in chronic stroke rehabilitation: A controlled study, Annals of Neurology 74 (1) (2013) 100–108. http://dx.doi.org/https://doi.org/10.1002/ana.23879
  * [52] S. Sr, J. Rabha, D. Samanta, P. Mitra, M. Sarma, Classification of motor imagery based eeg signals using sparsity approach, International Conference on Intelligent Human Computer Interaction, (2017), 47–59. http://dx.doi.org/10.1007/978-3-319-72038-8_5
  * [53] M. Besserve, K. Jerbi, F. Laurent, S. Baillet, J. Martinerie, L. Garnero, Classification methods for ongoing EEG and MEG signals, Biological Research 40 (2007) 415–437. http://dx.doi.org/10.4067/S0716-97602007000500005
  * [54] M. Tibdewal, M. Mahadevappa, A. Ray, M. Malokar, H. Dey, Power line and ocular artifact denoising from eeg using notch filter and wavelet transform, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom), (2016), 1654–1659. https://ieeexplore.ieee.org/document/7724548
  * [55] M. Samadi, N. Cooke, Vog-enhanced ica for ssvep response detection from consumer-grade eeg, 2014 22nd European Signal Processing Conference (EUSIPCO), (2014), 2025–2029. https://ieeexplore.ieee.org/document/6952745
  * [56] H. Qi, Y. Zhu, D. Ming, B. Wan, Independent component analysis using clustering on motor imagery eeg, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, (2009), 4735–4738. http://dx.doi.org/10.1109/IEMBS.2009.5334189
  * [57] N. Bajaj, Wavelets for eeg analysis, Wavelet Theory (2021). http://doi.org/10.5772/intechopen.94398
  * [58] R. Veldhuizen, E. Jonkman, D. Poortvliet, Sex differences in age regression parameters of healthy adults-normative data and practical implications, Electroencephalography and Clinical Neurophysiology 86 (6) (1993) 377–384. http://dx.doi.org/https://doi.org/10.1016/0013-4694(93)90133-G
  * [59] R. Majid Mehmood, R. Du, H. Lee, Optimal feature selection and deep learning ensembles method for emotion recognition from human brain eeg sensors, IEEE Access 5 (2017) 14797–14806. http://dx.doi.org/10.1109/ACCESS.2017.2724555
  * [60] R. Mehmood, H. Lee, Emotion classification of eeg brain signal using svm and knn, 2015 IEEE International Conference on Multimedia Expo Workshops (ICMEW), (2015), 1–5. http://dx.doi.org/10.1109/ICMEW.2015.7169786
  * [61] P. Kant, J. Hazarika, S. Laskar, Wavelet transform based approach for eeg feature selection of motor imagery data for braincomputer interfaces, 2019 Third International Conference on Inventive Systems and Control (ICISC), (2019), 101–105. http://dx.doi.org/10.1109/ICISC44355.2019.9036445
  * [62] S. Muthong, P. Vateekul, M. Sriyudthsak, Stacked probabilistic regularized lda on partitioning non-stationary eeg data for left/right hand imagery classification, 2016 IEEE EMBS Conference on Biomedical Engineering and Sciences (IECBES), (2016), 301–306. http://dx.doi.org/10.1109/IECBES.2016.7843462
  * [63] A. Bhardwaj, A. Gupta, P. Jain, A. Rani, J. Yadav, Classification of human emotions from eeg signals using svm and lda classifiers, 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN), (2015), 180–185. http://dx.doi.org/10.1109/SPIN.2015.7095376
  * [64] R. Sánchez-Reolid, A. García, M. Vicente-Querol, L. Fernández-Aguilar, M. López, A. Fernández-Caballero, P. González, Artificial neural networks to assess emotional states from brain-computer interface, Electronics 7 (12) (2018) 384. https://doi.org/10.3390/electronics7120384
  * [65] N. Sairamya, L. Susmitha, S. George, M. Subathra, Chapter 12 - hybrid approach for classification of electroencephalographic signals using time–frequency images with wavelets and texture features, in: D. J. Hemanth, D. Gupta, V. Emilia Balas (Eds.), Intelligent Data Analysis for Biomedical Applications, Intelligent Data-Centric Systems, Academic Press, (2019), 253–273. http://dx.doi.org/https://doi.org/10.1016/B978-0-12-815553-0.00013-6
  * [66] M. Dehghani, A. Mobaien, R. Boostani, A deep neural network-based transfer learning to enhance the performance and learning speed of bci systems, Brain-Computer Interfaces 8 (1-2) (2021) 14–25. https://doi.org/10.1080/2326263X.2021.1943955
  * [67] M. Ahmadi, A. Erfanian, An on-line bci system for hand movement control using real-time recurrent probabilistic neural network, 2009 4th International IEEE/EMBS Conference on Neural Engineering, (2009), 367–370. http://dx.doi.org/10.1109/NER.2009.5109309
  * [68] W. Ko, J. Yoon, E. Kang, E. Jun, J. Choi, H. Suk, Deep recurrent spatio-temporal neural network for motor imagery based bci, 2018 6th International Conference on Brain-Computer Interface (BCI), (2018), 1–3. http://dx.doi.org/10.1109/IWW-BCI.2018.8311535
  * [69] D. Kostas, S. Aroca-Ouellette, F. Rudzicz, Bendr: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data, Frontiers in Human Neuroscience 15 (2021) 653659. http://dx.doi.org/10.3389/fnhum.2021.653659
  * [70] W. Li, W. Huan, B. Hou, Y. Tian, Z. Zhang, A. Song, Can emotion be transferred? – a review on transfer learning for eeg-based emotion recognition, IEEE Transactions on Cognitive and Developmental Systems (2021) 1–1. http://dx.doi.org/10.1109/TCDS.2021.3098842
  * [71] C. Tan, F. Sun, W. Zhang, Deep transfer learning for eeg-based brain computer interface, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (2018), 916–920. http://dx.doi.org/10.1109/ICASSP.2018.8462115
  * [72] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, (2015). https://doi.org/10.48550/arXiv.1409.1556
  * [73] A. Krizhevsky, I. Sutskever, G. Hinton, Imagenet classification with deep convolutional neural networks, F. Pereira, C. J. C. Burges, L. Bottou, K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems, Vol. 25, Curran Associates, Inc., (2012). https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
  * [74] P. Boonyakitanont, A. Lek-uthai, J. Songsiri, Automatic epileptic seizure onset-offset detection based on cnn in scalp eeg, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (2020), 1225–1229. http://dx.doi.org/10.1109/ICASSP40776.2020.9053143
  * [75] H. Donmez, N. Ozkurt, Emotion classification from eeg signals in convolutional neural networks, 2019 Innovations in Intelligent Systems and Applications Conference (ASYU), (2019), 1–6. http://dx.doi.org/10.1109/ASYU48272.2019.8946364
  * [76] Z. Lim, N. Yong Quan, Convolutional neural network based electroencephalogram controlled robotic arm, 2021 IEEE International Conference on Automatic Control Intelligent Systems (I2CACIS), (2021), 26–31. http://dx.doi.org/10.1109/I2CACIS52118.2021.9495879
  * [77] G. Roy, A. Bhoi, S. Das, S. Bhaumik, Cross-correlated spectral entropy-based classification of eeg motor imagery signal for triggering lower limb exoskeleton, Signal, Image and Video Processing http://dx.doi.org/10.1007/s11760-022-02142-1
  * [78] BCI Competition III, (n.d.). https://www.bbci.de/competition/iii/
  * [79] A. Chaudhry, U. Khan, M. Palla, S. Singh, S. Deshmukh, A prosthetic arm based on electroencephalography by signal acquisition and processing on matlab, International Journal of Research in Engineering, Science and Management 5 (1) (2022) 119–-124.
URL http://www.journals.resaim.com/ijresm/article/view/1691

  * [80] E. Wu, Z. Cao, P. Xiong, A. Song, L. Zhu, M. Yu, Brain-computer interface using brain power map and cognition detection network during flight, IEEE/ASME Transactions on Mechatronics (2022) 1–11. http://dx.doi.org/10.1109/TMECH.2022.3148141
  * [81] M. Behri, A. Subasi, S. Qaisar, Comparison of machine learning methods for two class motor imagery tasks using eeg in brain-computer interface, 2018 Advances in Science and Engineering Technology International Conferences (ASET), (2018), 1–5. http://dx.doi.org/10.1109/ICASET.2018.8376886
  * [82] T. Mwata-Velu, J. Ruiz-Pinales, H. Rostro-Gonzalez, M. Ibarra-Manzano, J. Cruz-Duarte, J. Avina-Cervantes, Motor imagery classification based on a recurrent-convolutional architecture to control a hexapod robot, Mathematics 9 (6). http://dx.doi.org/10.3390/math9060606
  * [83] D. Ramdhani, E. Djamal, F. Nugraha, Brain-computer interface based on motor imagery and emotion using convolutional neural networks, 2020 FORTEI-International Conference on Electrical Engineering (FORTEI-ICEE), (2020), 108–112. http://dx.doi.org/10.1109/FORTEI-ICEE50915.2020.9249937
  * [84] Z. Khademi, F. Ebrahimi, H. Kordy, A transfer learning-based cnn and lstm hybrid deep learning model to classify motor imagery eeg signals, Computers in Biology and Medicine 143 (2022) 105288\. http://dx.doi.org/https://doi.org/10.1016/j.compbiomed.2022.105288
  * [85] R. Zhang, Q. Zong, L. Dou, X. Zhao, Y. Tang, Z. Li, Hybrid deep neural network using transfer learning for eeg motor imagery decoding, Biomedical Signal Processing and Control 63 (2021) 102144. http://dx.doi.org/https://doi.org/10.1016/j.bspc.2020.102144