# A Human-Machine Joint Learning Framework to Boost Endogenous BCI Training

Hanwen Wang, Yu Qi*, Lin Yao, Yueming Wang,
Dario Farina, Gang Pan Hanwen Wang is with the College of Computer Science and
Technology, Zhejiang University, Hangzhou, China.Yu Qi is with the Affiliated
Mental Health Center & Hangzhou Seventh People’s Hospital, and the MOE
Frontier Science Center for Brain Science and Brain-machine Integration,
Zhejiang University School of Medicine, the State Key Lab of Brain-Machine
Intelligence, Hangzhou, China.Lin Yao is with the Department of Neurobiology,
Affiliated Mental Health Center & Hangzhou Seventh People’s Hospital, Zhejiang
University School of Medicine, MOE Frontiers Science Center for Brain and
Brain-Machine Integration, Zhejiang University, Department of Biomedical
Engineering, Zhejiang University, and College of Computer Science and
Technology, Zhejiang University, Hangzhou, ChinaYueming Wang is with the
Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou,
China.Dario Farina is with the Department of Bioengineering, Imperial College
London, London, UKGang Pan is with the State Key Lab of Brain-Machine
Intelligence, the College of Computer Science and Technology, Zhejiang
University, Hangzhou, China, and the First Affiliated Hospital, Zhejiang
University, Hangzhou, China.* Corresponding authors: Yu Qi<EMAIL_ADDRESS>and Gang Pan (gpan@zju.edu.cn).

###### Abstract

Brain-computer interfaces (BCIs) provide a direct pathway from the brain to
external devices and have demonstrated great potential for assistive and
rehabilitation technologies. Endogenous BCIs based on electroencephalogram
(EEG) signals, such as motor imagery (MI) BCIs, can provide some level of
control. However, mastering spontaneous BCI control requires the users to
generate discriminative and stable brain signal patterns by imagery, which is
challenging and is usually achieved over a very long training time
(weeks/months). Here, we propose a human-machine joint learning framework to
boost the learning process in endogenous BCIs, by guiding the user to generate
brain signals towards an optimal distribution estimated by the decoder, given
the historical brain signals of the user. To this end, we firstly model the
human-machine joint learning process in a uniform formulation. Then a human-
machine joint learning framework is proposed: 1) for the human side, we model
the learning process in a sequential trial-and-error scenario and propose a
novel “copy/new” feedback paradigm to help shape the signal generation of the
subject toward the optimal distribution; 2) for the machine side, we propose a
novel adaptive learning algorithm to learn an optimal signal distribution
along with the subject’s learning process. Specifically, the decoder reweighs
the brain signals generated by the subject to focus more on “good” samples to
cope with the learning process of the subject. Online and psuedo-online BCI
experiments with 18 healthy subjects demonstrated the advantages of the
proposed joint learning process over co-adaptive approaches in both learning
efficiency and effectiveness.

###### Index Terms:

Electroencephalogram (EEG), Brain-Computer Interface (BCI), Motor Imagery
(MI), Neural Decoding.

##  I Introduction

Brain-computer interfaces (BCIs) act as an intermediary between the brain and
external devices, translating brain signals to control signals [1, 2]. BCIs
employing electroencephalogram (EEG) signals have been developed successfully
for many applications, including neural rehabilitation, prosthesis control and
emotion recognition [3, 4, 5, 6, 7, 8, 9, 10]. Most existing BCI systems can
be divided into endogenous and exogenous BCIs [11, 12]. Exogenous BCIs such as
the P300 and SSVEP systems, rely on brain signals evoked by external stimuli
[13, 14], such as visual flashes or audio events. Conversely, endogenous BCIs,
such as based on motor imagery (MI), rely on self-regulated brain signals
often induced by users’ covert attention or mental tasks [15, 16]. Thus,
endogenous BCIs can provide more flexible and natural brain control, resulting
in high potential for a wide range of applications [17, 18, 19, 20]. In the
remainder of this paper, we will only focus on endogenous BCIs.

Effective BCI control relies on the close collaboration of the brain (the
subject) and the machine (the decoder)[21]. The user should generate
discriminative and sufficient brain signals as different control commands,
while the decoder should identify different brain signal patterns and robustly
interpret/map them into control commands. For endogenous BCIs, the main
challenge lies in the difficulty for the user in generating effective and
stable brain signals. To achieve robust online control, the BCI user should be
able to accurately replicate at least two discriminative and stable patterns
for basic control, such as going left and right. A typical endogenous BCI is
based on MI, where the user is asked to imagine movements, for example of the
left/right hand. However, generating MI is challenging for most users, and a
typical user usually requires a long and difficult training process to learn
the MI control. Commonly, training for MI lasts several weeks or even months
[22, 23]. Furthermore, there are at least 15% to 30% of subjects across the
population who cannot control BCI systems through learning, a case called BCI
illiteracy [24]. For these reasons, the facilitation of the learning process
in endogenous BCIs is a very relevant problem.

The learning process in BCIs includes two parts: the learning of the decoder
(machine learning), and the learning of the user (human learning). Here we
briefly review the existing approaches.

  * •

Machine learning. Traditional BCI training processes start with a static
subject learning process where the subject tries to generate brain signals
without feedback. Then a decoder is trained with the initial data to learn
discriminative features from different brain signals (as shown in Fig. 1A).
The discrimination of different brain signal patterns can be regarded as a
typical classification problem. Thus, linear and nonlinear classifiers have
been proposed for the decoding problem.  For linear methods, decoding in motor
imagery (MI) can be separated into feature extraction and classification.
Common Spatial Pattern (CSP) is a widely-used linear approach for extracting
discriminative features from different brain signal patterns. Filter Bank
Common Spatial Pattern (FBCSP) [25] extends CSP with a set of band-pass
filters to enhance features, and it can also be applied in neural networks
[26]. However, traditional CSP is not suitable for multiclass problems. Joint
Approximate Diagonalization (JAD)-based methods are used to improve CSP [27].
Moreover, modifications have been made to enhance CSPs to deal with the
nonstationarity in EEG signals. Fuzzy covariance matrices are deployed in
DivCSP-WS to extract steady features [28]. Channel selection methods could
also improve stability between sessions, improving the classification accuracy
[29].

In terms of classification methods, linear discriminant analysis (LDA) is
commonly used for extracted features [30]. To enhance robustness on non-
stationary EEG signals, variations of LDA such as rLDA and BLDA have been
introduced [31, 32, 33]. Support Vector Machines (SVMs) with flexible kernel
design are employed with feature extraction methods in some studies [34, 35].
Recently, deep learning approaches including Long Short Term Memory (LSTM)
[36, 37, 38] and convolutional neural networks (CNNs) [39, 40, 41, 42, 43],
such as EEGnet [44], have demonstrated effectiveness in MI classification.
Combining the feature extraction methods and the deep learning approach,
neural structured learning (NSL) has been proposed to train deep neural
networks with feature inputs and the structured signals, which enhances the
robustness [45].

  * •

Human learning. After the initial decoder is trained, the subject can learn to
control the BCI with the feedback of the decoder, where the design of the
feedback plays an important part (Fig. 1B). Traditional feedbacks usually
directly reflect the decoding result, using a moving cursor or a falling ball
[46, 47]. To improve the efficiency of subject learning, new paradigms with
different types of feedback have been introduced. For example, Hwang et al.
proposed an intuitive feedback that uses a surface topography of EEG signal
[23], where the real-time spectrogram is presented to enhance the MI [48].
Wang et al. proposed a neurofeedback training paradigm, in which the
performance of MI was transferred into the distance between a needle and hands
on the screen [49].  Alternatively, auditory feedbacks of different sounds
have been presented to subject depending on their MI performance [50].
Nevertheless, Yao et al. proposed tactile stimulation to help subjects improve
the MI activation [51, 52]. Also, Ono et al. adopted exoskeleton attached to
subjects, making feedbacks more sensible [53].

  * •

Co-adaptive learning. The aforementioned approaches mostly focus on one side
of the learning, either the signal generation by the user or the algorithm
training by the machine. However, co-adaptive methods have also been
introduced [54, 55]. Instead of using a fixed decoder after the initial
training process, co-adaptive approaches retrain both the decoder and the
subject iteratively in sequential learning sessions to cope with each other
[56, 54] (Fig. 1C). To this end, different decoder recalibration algorithms
have been proposed using machine learning approaches such as LDA [57], SVM
[58], probabilistic graphical models [59] to update the decoder with newly
collected data to cope with changes in brain signals during the subject
learning process. By considering the interaction between the subject and the
decoder, co-adaptive approaches demonstrated superior performance in BCI
learning than separate learning approaches [60, 61]. To provide a mathematical
basis of co-adaptive learning processes, previous work has modeled the process
with mathematical formulations. For example, in [62], an encoder-decoder model
was proposed, in which the co-adaptation was described by the optimization of
the user’s control. With the assumption of two linear learning systems, a
theoretical formulation was derived by the stochastic gradient descent method
in [63].

Figure 1: Illustration of BCI training with different strategies. (A) The
process of decoder learning gives a set of subject-generated data (indicated
by the red circle). (B) The subject learning process with feedback from a
fixed decoder. (C) The co-adaptation learning process is where the subject and
the decoder learn in an alternate manner. (D) The proposed joint learning
process is where the subject and the decoder share the same loss function
during learning.

In Fig. 1, we illustrate and compare the aforementioned learning process in an
intuitive way. Suppose there is a global optimal point for the human-machine
BCI learning process. With the machine learning model alone (Fig. 1A), where
the human model is static, the optimal performance is highly limited by the
boundary of brain signal patterns. With the human learning model alone (Fig.
1B), where the decoder is fixed, the subject can learn with the guidance of
the decoder’s feedback. The performance highly relies on the effectiveness of
the decoder. Since the decoder is usually trained with the initially imperfect
brain signals of the subject, it can be suboptimal and can degrade the human
learning process. The co-adaptation model (Fig. 1C) considers both machine
learning and human learning, where the decoder is iteratively updated along
with the human learning process. Specifically, it contains an iterative
process where the machine and the human learn alternately, which improves the
theoretical overall performance. However, it assumes that the subject can
effectively learn at every human learning stage, which can be difficult,
especially at the beginning rounds of the training process.

This study proposes a novel BCI learning scheme for efficient and effective
BCI training. As shown in Fig. 1D, we aim to construct a joint learning model
between humans and machines, where human and machine processes can be
optimized simultaneously. To this end, we propose a novel human-machine joint
learning framework for effective BCI training. Specifically, we assume the
human learning behavior in sequential training is a trial-and-error process,
and let the decoder determine whether a brain signal is “good” or “poor” by
its discrimination ability according to the distribution of the feature space.
The contribution of this study is summarized in three folds.

  1. 1.

We formally formulate the human-machine joint learning model and propose a
human-machine joint loss function, where the subject is encouraged to generate
more discriminative brain signals, and the decoder optimizes the classifier to
separate the different brain signal modes.

  2. 2.

From the human side, we formulate human behavior in a sequential trial-and-
error learning process. A novel paradigm is proposed to guide the subject to
optimize brain signals. Specifically, if a “good” signal is generated, the
system encourages the subject to “copy” the state; otherwise, if a “poor”
signal is generated, the subject is encouraged to change the brain signal.

  3. 3.

From the machine side, we propose a novel adaptive learning algorithm to learn
an optimal signal distribution along with the subject’s learning process.
Specifically, the decoder reweighs the brain signals generated by the subject
to focus more on “good” samples to cope with the learning process of the
subject.

Online BCI experiments with 18 healthy subjects demonstrated that the joint
learning framework can efficiently guide the subject learning discriminate
brain signals for effective BCI control. Compared with traditional co-adaptive
approaches, our method significantly improved the average control accuracy
from 69.1% to 74.5%.

##  II The Human-machine joint learning framework

We first formulate the BCI learning process in a uniform model and propose a
human-machine joint loss function for learning. Then we propose the human
learning model and propose a loss function for the human learning process. We
then introduce the machine learning process where the decoder learns to select
optimal brain signals that the subject generated to guide and cope with the
learning process of the subject. Finally, we propose a novel BCI learning
paradigm and framework for human-machine joint learning.

###  II-A Modeling of the human-machine joint learning

Considering BCI learning as a human-machine learning problem, the subject and
the decoder share the same goal of accurate intention decoding from brain
signals. To achieve this goal, the subject tries to generate more
discriminative brain signals, and the decoder optimizes the classifier to best
separate different brain signal patterns. To this end, the human learning
model can be described as:

$x=g(\theta_{H},y),$ (1)

where $x$ stands for the signal generated according to the given label $y$.
And the parameter of generation and the function are denoted as $\theta_{H}$
and $g(.)$ respectively. For the machine part, the model is defined by:

$y^{\prime}=h(\theta_{M},x),$ (2)

in which the predict label $y^{\prime}$ is decoded by function $h(.)$ and
parameter $\theta_{M}$.

Thus, the human-machine mutual goal of BCI learning is to minimize the
distance between the true label $y$ and the label estimated by the decoder
given brain signals $y^{\prime}$:

$\begin{split}L_{H-M}&=||y^{\prime}-y||.\end{split}$ (3)

Then the joint optimization of humans and machines can be achieved by
minimizing the joint loss function $L_{H-M}$:

$\begin{split}\mathop{\arg\min_{\theta_{H},\theta_{M}}}L_{H-M}&=\mathop{\arg\min_{\theta_{H},\theta_{M}}}||h(\theta_{M},g(\theta_{H},y))-y||.\end{split}$
(4)

Figure 2: The diagram of the human-machine joint learning process. From the
subject’s view, the subject tries to generate brain signals according to the
instructions (such as left or right). At each trial, the system gives feedback
(left or right) during the process, and evaluates the subject’s brain signals
according to the discriminative ability. For a trial with high signal quality,
a “copy” command is given such that the subject should maintain the brain
signal patterns; otherwise, a “new” command is given and the subject should
change the way of thinking to improve the signal quality.

The optimization of the joint loss function relies on both the subject and the
machine. On the one hand, the subject learns to maximize the discrimination
ability between the two brain signal modes according to the decoder by
optimizing $\theta_{H}$; on the other hand, the decoder tries to optimize a
classification plane with $\theta_{M}$, where the brain signals from different
modes can be maximumly classified. To solve the joint learning problem, we
should re-model the learning for both humans and machines to cope with the
jointly learning process.

####  II-A1 Modeling of human learning

Here we model the learning process of the user. In a feedback learning
process, the user adjusts the brain signals according to the feedback. In the
human learning model, we simplify the problem by assuming that the subject
generates new signals depending only on the last feedback.  Then we can regard
the process as a Markov process, which has been widely applied to modeling the
human attention, decision-making or control system [64, 65, 66, 67].

Given the instruction $y$, the subject aims to generate brain signals in a
certain mode. We divide the brain signals into two groups of “good” and
“bad/poor” according to their discriminative ability with the decoder, as
shown in Fig. 2. We denote the “good” and “bad/poor” samples by $x_{G}$ and
$x_{B}$. Thus, the objective of the subject is to increase the probability of
generating $x_{G}$, and to decrease the probability of generating $x_{B}$. The
learning objective is therefore given by:

$\mathop{\arg\max_{\theta_{H}}}\frac{P(x_{G})}{P(x_{B})},$ (5)

where $P(x_{G})$ and $P(x_{B})$ represent the probability of good and bad
samples of the subject. In the Markov sequential learning process, we can
define a transition matrix of $x_{G}$ and $x_{B}$ by:

$\begin{bmatrix}P_{GG}&P_{GB}\\\
P_{BG}&P_{BB}\end{bmatrix}=\begin{bmatrix}P_{GG}&1-P_{GG}\\\
1-P_{BB}&P_{BB}\end{bmatrix},$ (6)

where $P_{ab}$ stands for the probability of transition from condition a to
condition b. The steady-state vector could be derived as:

$\begin{bmatrix}P(x_{G})&P(x_{B})\end{bmatrix}=\begin{bmatrix}\dfrac{1-P_{BB}}{2-P_{GG}-P_{BB}}&\dfrac{1-P_{GG}}{2-P_{GG}-P_{BB}}\end{bmatrix}.$
(7)

Substituting the variables into the loss function, we get

$\mathop{\arg\max_{\theta_{H}}}\frac{P(x_{G})}{P(x_{B})}=\mathop{\arg\max_{\theta_{H}}}\frac{1-P_{BB}}{1-P_{GG}}.$
(8)

According to the model, a high $P_{GG}$ and a low $P_{BB}$ are required for
effective BCI learning.

####  II-A2 Modeling of machine learning

Towards human-machine joint learning, we propose a novel classification
algorithm based on a sample evaluation and adaptive learning of the decoder.
The process of decoder learning has two folds. First, we evaluate the brain
signals generated by the subject and determine whether a sample is “good” or
“bad” for feedback in the sequential learning process. Second, the decoder
should be dynamically tuned to cope with the new samples generated by the
subject, where a novel training algorithm based on sample reweighting is
proposed.

  * •

Sample evaluation. The determination of “good” and “bad” brain signals plays a
key role in the process of human learning, and the evaluation of brain signal
quality is the bond between the human and machine learning process.

Intuitively, “good” samples should contain high discriminative ability in the
decoder’s feature space. Thus, the determination of sample quality is given by
the decoder, and the computation of sample quality can be diverse with
different decoders. Here we take the SVM classifier as an example. With SVM,
the determination of different classes is according to a sample’s distance to
the hyperplane constituted by the support vectors. In this manner, a sample
far away from the hyperplane indicates low confusion in classification, thus
it is considered a “good” sample, and samples with a small distance to the
hyperplane are considered of low quality. Generally, the quality of a sample
$i$ should be negatively correlated to its classification loss $L_{i}$.
Therefore, we evaluate the quality of the sample $q_{i}$ as

${q_{i}}\propto\frac{1}{L_{i}}.$ (9)

  * •

Adaptive decoder learning with sample reweighting.

Given the quality of brain signal samples, the decoder assigns weights to the
samples according to their quality to construct a better discriminative sample
distribution, which serves as guidance for the subject to learn.

In typical machine learning approaches such as the SVM and Adaboost, samples
with low quality (difficult samples) are more focused to improve the
classification performance. While in the human-machine joint learning problem,
we take the opposite strategy that focuses more on the “good” samples rather
than the “bad” ones. It is because the human-machine joint learning process
forms a new problem that is quite different from typical ones. Instead of
using fixed data, the sample generator (human) is intelligent and can
adaptively cope with the decoder by learning to generate more “good” samples
and fewer “bad” ones. By focusing on samples with high quality, the feedback
of the decoder leads the subject to generate “good” samples to facilitate the
learning process. Thus, the weight $v_{i}$ of the sample $i$ is assigned by

${v_{i}}={q_{i}}.$ (10)

Here we develop a novel adaptive learning approach by focusing on samples with
high quality via a sample reweighting algorithm, which satisfies Eqs. 9 and
10. Specifically, we design a novel loss function for the decoder to connect
the loss and weight by the self-paced learning [68] algorithm, where the
learning starts with an easy sample and gradually to difficult ones. The loss
function of the decoder learning process can be described as:

$\displaystyle\mathop{\min_{w,v}}$ $\displaystyle E(w,v;\lambda)=$ (11)
$\displaystyle\sum_{i=1}^{n}v_{i}L(x_{i},y_{i},w)+(1-\lambda)v_{i}-\frac{(1-\lambda)^{v_{i}}}{log(1-\lambda)},$

where $\sum_{i=1}^{n}v_{i}L(x_{i},y_{i},w)$ stands for the traditional loss
function with weights $v$ and

$\sum_{i=1}^{n}(1-\lambda)v_{i}-\frac{(1-\lambda)^{v_{i}}}{log(1-\lambda)}$

represents the penalty function $f(v;\lambda)$, which controls the pace of the
self-paced learning. In this way, the weights are included in the loss
function.

Since the loss could be taken as a biconvex function, an Alternative Convex
Search [69] is used to optimize the loss function, which alternately finds the
optimal solution to one group of variables by fixing the other variables.

    * –

Optimization of $v$: We first fix parameter $w$ and optimize $v$. The partial
gradient of Eq. 11 is:

$\frac{\partial E(w,v;\lambda)}{\partial
v_{i}}=L(y_{i},x_{i},w)+((1-\lambda)-(1-\lambda^{v_{i}})).$ (12)

Let Eq. 12 $=$ 0, and we can easily deduce:

$log(L(y_{i},x_{i},w)+(1-\lambda))=v_{i}log(1-\lambda).$ (13)

So the solution for $E(w,v;\lambda)$ could be given by:

$v^{\prime}_{i}=\left\\{\begin{aligned}
&\frac{1}{log(1-\lambda)}log(L_{i}+(1-\lambda)),\quad L_{i}<\lambda,\\\
&0,\quad{\rm otherwise},\end{aligned}\right.$ (14)

where $L_{i}$ stands for $L(y_{i},x_{i},w)$. As a result, we get the close-
form solution for $v$. And in Eq. 14, there is $v_{i}\propto\frac{1}{L_{i}}$,
which means samples with higher loss will be given smaller weights, satisfying
the requirement.

    * –

Optimization of $w$: Given $v$, we fix $v$ and optimize $w$ as follows:

$\frac{\partial E(w,v;\lambda)}{\partial
w}=\frac{\partial\sum_{i=1}^{n}v_{i}L(x_{i},y_{i},w)}{\partial w}.$ (15)

As shown in Eq. 15, the gradient is reduced to the partial derivative of a
weighted loss problem. Close-form solutions can be derived if the loss
function is hinge loss such as with the SVM. And for the loss of deep learning
models, it also can be optimized through gradient descent.

####  II-A3 Experimental paradigm for human-machine joint learning

Given the modeling of human-machine joint learning, the problem is how to
effectively feedback on the sample quality to the user. Here a “copy/new”
feedback strategy is proposed together with the paradigm of the joint learning
process.

  * •

The “copy/new” feedback strategy. We propose a novel “copy/new” feedback
strategy to indicate whether the previous brain signal is “good” or “poor”. In
our paradigm, the trials are split into pairs and the instruction for the next
trial depends on the quality of the previous trial. If the previous trial is
of good quality, there would be a “copy” for the next trial. With the “copy”
feedback, the subject is encouraged to generate brain signals similar to the
previous one, which increases the $P_{GG}$. Otherwise, there would be a “new”
instruction where the subject is asked to change the way of thinking and
explore better signals, which helps decrease $P_{BB}$ in Section II-A1.  In
other words, the “copy” signals help subjects to maintain the previous “good”
signal, and the “new” instruction will guide subjects to try other
possibilities. The feedback lets the subject learn in a trial-and-error
process, where the subject learns to shape the distribution of brain signals
to a more discriminative condition with the guidance of the decoder.

  * •

The paradigm of the joint learning process. The diagram of the human-machine
joint learning process is illustrated in Fig. 2 and Algorithm 1. The subject
tries to generate brain signals according to the instructions (such as left or
right). Then the system gives the decoding results as feedback, meanwhile
evaluating the quality of brain signals according to the discriminative
ability. For a trial with high signal quality, a “copy” command is given such
that the subject should maintain the brain signal patterns; otherwise, a “new”
command is given and the subject should change the way of thinking to improve
the signal quality. The aforementioned process constitutes a training block,
and the process repeats several times in a training session. The decoder
updates after each training session.

Algorithm 1 Pseudo code of the human-machine joint learning process

Input: Number of training sessions $n$, proposed algorithm $A$ in in Section
II-A2 and proposed paradigm $P$ in in Section II-A3
Output: Fine-tuned classifier $C$ and trained subject $S$

1: Let $k=1$.

2: Calibration session of the paradigm without feedback.

3: Update the classifier $C$ by $A$.

4: $k=k+1$.

5: while $k\leq n$ do

6: Online training with $P$.

7: Update the classifier $C$ by $A$.

8: $k=k+1$.

9: end while

Figure 3: The joint learning experiment paradigm with the MI task. (A) The
trials are designed in a sequential manner as the First Trials and the Next
Trials. For each trial, there are a 2-second-long preparation where a cross
sign (+) is presented. Then an instruction arrow of “left/right” is presented
for two seconds. With the Next trials, there are “copy/new” instructions along
with the arrows in this stage. After that, the subject starts to perform MI as
instructed for five seconds, during which feedback are given (as shown in B).
(B) The feedbacks during the MI process include two squares indicating the
decoding results for left and right respectively with the color of the
squares. Lighter color indicates more discriminative classification
performance.  (C) The whole scheme of experiment including human side and
machine side, which illustrates the joint learning process.

###  II-B The joint learning paradigm for motor imagery training

Here we specify the process of the joint learning paradigm with the MI task.

####  II-B1 Experiment paradigm with the MI task

The paradigm is presented in Fig. 3. Subjects are asked to focus on the cross
on the screen and perform left or right MI tasks as indicated by instruction
arrows. The cross is shown on the screen for 2 seconds, turning from white to
gray. After that, there is an arrow that points left or right, indicating the
left or right MI task. The subject should perform MI for 5 seconds. In the
calibration session, the MI is performed without feedback. Subjects need to
stay focused on the cross and keep their MI mental task without feedback.
After the calibration session, the parameters of the algorithm are updated and
feedback with “copy/new” instructions are provided. To reduce eye movement, we
set the feedback as the brightness of the target. There are two squares
spreading horizontally in the middle of the screen, in the left and right
parts respectively. The subjects’ task is to keep the indicated square as
bright as possible in 5 seconds through MI. These feedbacks are denoted as
$C^{n}_{left}$ and $C^{n}_{right}$, in which $n$ stands for the number of
updates. $C^{0}_{left}$ and $C^{0}_{right}$ are set to 0.5 initially and
updated as follows:

$\begin{split}C^{n+1}_{left}&=C^{n}_{left}+\alpha((P_{left}-0.5)*2),\\\
C^{n+1}_{right}&=1-C^{n+1}_{left},\end{split}$ (16)

where $P_{left}\in[0,1]$ is the posterior probability of left calculated by
the online discriminator and $\alpha$ is a parameter to adjust the pace.

With the “copy/new” paradigm, the trials are split into pairs, namely the
“First trials” and the “Next trials” (Fig. 3). If the accuracy of the first
trial achieves a threshold of $T$, there would be a “copy” along with the
instruction arrow in the next trial, which informs subjects to keep the way of
MI. Otherwise, these would be “new” in the screen above the instruction arrow,
indicating that subjects should change the way of signal generation. The
accuracy is calculated by the average accuracy of each slice during the online
trial.

####  II-B2 MI decoding with sample reweighting

The decoding of MI brain signals consists of two stages. The first one is to
extract effective features, where methods such as the common spatial patterns
(CSP) and filter bank common spatial pattern (FBCSP) are used. And the second
stage is to find a discriminator to classify the features. In order to deploy
the proposed self-paced learning-based sample reweighting method, both stages
should be taken into consideration.

For the feature extraction, we used the weighted CSP by adding weights to the
traditional CSP. Different from CSP, the normalized covariance
$\overline{R_{1}}$ and $\overline{R_{2}}$ are calculated by weighted averaged
over samples of each group:

$R=\overline{R_{1}}+\overline{R_{2}}=\sum_{i}v_{1}^{i}r_{1}^{i}+\sum_{j}v_{2}^{j}r_{2}^{j},$
(17)

where $v_{1}^{i},v_{2}^{j}$ are the corresponding weight of samples in each
group and

$r_{1}^{i}=\frac{X_{1}^{i}X_{1}^{iT}}{trace(X_{1}^{i}X_{1}^{iT})},\quad
r_{2}^{j}=\frac{X_{2}^{j}X_{2}^{iT}}{trace(X_{2}^{j}X_{2}^{jT})},$ (18)

in which $X_{1}^{i},X_{2}^{j}\in R^{N\times T}$ stand for the signals matrices
of two conditions (left or right), $N$ denotes the number of channels, and $T$
is the number of samples per channel.

For the discriminator, we chose the SVM to cooperate with weights. Details of
weighted-SVM could be found in [70], and the loss function is as follows:

$L=\sum_{i=1}^{N}v_{i}(1-y_{i}f(x_{i})).$ (19)

Then the optimization problem becomes:

$\begin{split}&\min_{\mathbf{w},b,\xi}\quad\frac{1}{2}||\mathbf{w}||^{2}+\sum_{i=1}^{N}v_{i}\xi_{i}\\\
\mathrm{s.t.}\quad&y_{i}f(x_{i})\geq 1-\xi_{i},\quad\xi_{i}\geq 0.\end{split}$
(20)

This is the traditional weighted SVM problem, which could be solved easily.

For the self-paced learning algorithm, we introduce new parameters $\Lambda$
and $\Delta\Lambda$ to control the learning pace more precisely, where
$\Lambda$ stands for the proportion of samples we want to recruit for the
training initially and $\Delta\Lambda$ is the incremental ratio of samples for
training in every iteration. And the parameter $\lambda$ is tuned by $\Lambda$
and $\Delta\Lambda$ in terms of the rank. The number of samples to be included
in each iteration is specified by $\Lambda$ and $\Delta\Lambda$, and then
$\lambda$ is calculated accordingly as [71], which can be described as the
$1/\Lambda$ quantiles of loss $v$.

Then we can design the workflow of our self-paced learning-based sample
reweighting algorithm. The details of the algorithm are shown in Algorithm 2
for MI decoding. Assuming we have a set of labeled data, we first randomly
split the data into a training set $\left\\{X,Y\right\\}$ of $N$ samples and a
validation set $\left\\{X^{\prime},Y^{\prime}\right\\}$. At first,
$\Lambda\times N$ samples are randomly selected from the training set. These
data are used to train the initial weighted CSP and weighted SVM. After that,
all data in the training set could get a weight, and the accuracy of the
validation set is calculated. If $\Lambda<1$, which means not all the training
set has been used, more data would be included in the update of weighted CSP
and weighted SVM according to their weights. The iteration will continue until
all the training set has been included. And the round with the highest
accuracy on the validation set would be selected. The weight and parameters of
weighted CSP and weighted SVM are the output.

Also, the joint learning method could be employed in other machine learning
methods, including deep learning. Under that condition, random selected
$\lambda\times N$ samples will be fed to the neural network for
initialization. After that, more and more samples would be assigned with
different weights according to their loss to update the network by iteration.
And the best model will be selected depending on the accuracy on
$\left\\{X^{\prime},Y^{\prime}\right\\}$, which is similar to the Algrithm 2.

Algorithm 2 The joint learning decoding algorithm for MI

Input: Training set $\left\\{X,Y\right\\}$ with $N$ samples, validation set
$\left\\{X^{\prime},Y^{\prime}\right\\}$.
Parameter: $\Lambda$ and $\Delta\Lambda$, controlling the training pace
Output: Weights: $v$, trained weighted-CSP and weighted-SVM

1: Initialize $k=0$.

2: Random select $\lambda\times N$ samples $\left\\{X_{k},Y_{k}\right\\}$ from
$\left\\{X,Y\right\\}$.

3: Train weighted-CSP on $\left\\{X_{k},Y_{k}\right\\}$.

4: Train weighted-SVM on $\left\\{X_{k},Y_{k}\right\\}$.

5: Update $v_{k}$ by Eq. 14.

6: while $\Lambda<1$ do

7: $\Lambda=\Lambda+\Delta\Lambda$.

8: $k=k+1$.

9: $\lambda=1/\Lambda$ quantiles of $L$.

10: Select data $\left\\{X_{k},Y_{k}\right\\}$ by $L_{i}<\lambda$.

11: Update weighted-CSP on $\left\\{X_{k},Y_{k}\right\\}$.

12: Update weighted-SVM on $\left\\{X_{k},Y_{k}\right\\}$.

13: Test weighted-CSP and weighted-SVM on
$\left\\{X^{\prime},Y^{\prime}\right\\}$.

14: Update $v_{k}$ by Eq. 14.

15: end while

16: Select best iteration $K$ by accuracy on
$\left\\{X^{\prime},Y^{\prime}\right\\}$.

17: return Weights at $K$ iteration: $v_{K}$, weighted-CSP and weighted-SVM at
$K$ iteration

Figure 4: Comparison of BCI control accuracy with and without joint learning.
(A) The online accuracy of subjects with and without the joint learning
process. $\ast$ indicates the significance with $t$-test (one-sided,
$p\textless 0.005$). (B) The proportion of “good” trials across the sessions.
(C) The trial-wise BCI control accuracy across trials.

##  III Experiments and results

###  III-A Experimental settings

We evaluated the proposed human-machine joint learning model with online BCI
experiments in comparison with co-adaptive BCIs without joint learning.  A
total of 18 subjects participated in the BCI experiments (eight females and
ten males, age ranging from 23 to 27). 14 subjects took part in the experiment
with CSP and SVM. And 4 subjects attended the experiment of EEGnet. Each
participant performed two experiments: a joint learning experiment and a co-
adaptive learning experiment (baseline). To mitigate the effects of session
order, the two experiments were arranged on two different days, at least two
days apart. The order of the experiments was also randomly selected, where  9
subjects conducted joint learning first the rest performed co-adaptive
learning experiments first. After each experiment, a simple questionnaire was
filled out by the subject. The settings of the experiments were as follows:

  * •

Joint learning experiment. There were a total of five sessions in the joint
learning experiment. The first one was an initial calibration session without
feedback, which had 60 trials including 30 left and 30 right trials in random
order. After the initial calibration session, there were four joint learning
sessions with feedback, consisting of 20 trials for each direction, and each
trial was 9-second-long. The trials were presented in Fig. 3 and Section
II-B1. For preprocessing, a four-order Butterworth Filter was adopted to
filter the data from 8Hz to 30Hz. The classifier was updated after every
session. The threshold $T$ of “copy/new” was set to 70%. $\Lambda$ was set to
0.2 for initialization and $\Delta\Lambda$ was set to 0.05 for updating the
sample reweighting algorithm  for SVM.  While for EEGnet, more samples were
needed to initialize the training process, so $\Lambda$ was set to 0.5. And
$\Delta\Lambda$ was set to 0.1 to reduce training iterations for saving time.

  * •

Co-adaptive learning experiment without joint learning (baseline). We used the
classical co-adaptive learning paradigm as the baseline for comparison.
Specifically, the co-adaptive learning method in the paper is adopted from
[60], named as RETRAIN, which achieved the best performance. The same method
was also used in [61]. The number of sessions in the co-adaptive learning
experiment (control group) was the same as in the joint learning experiment.
After the calibration session, the trials in the rest training session were
provided with feedback from an  SVM/EEGnet decoder, which was also updated
after every session.

For both the joint learning and the co-adaptive learning experiments, the
software system was set up in Matlab, with the Psychtoolbox [72].  And the
computer was equipped with an Intel i7-10700k CPU, 64G RAM and an NVIDIA
GeForce RTX 3080 GPU. In CSP feature computing, three pairs of features were
selected for classification. In the SVM classifier, a linear kernel was
adopted.  The time for inference was approximately 0.003 seconds. And the
training of the algorithm cost 5.55$\pm$2.87 seconds. For EEGnet, we chose
ADAM as the optimizer which is same to [44]. The inference time was
approximately 0.004 seconds, while the training time was 153$\pm$23 seconds.

The feedback were calculated by the window of one second and updated every
frame (1/60s). $\alpha$ was set to 0.2 for the feedback. For the offline
analysis, we cut the five-second trial into four trial slices for one second,
starting from 0.5 seconds and ending at 4.5 seconds after the instruction.

The EEG signals were recorded using a wireless EEG system (NSW24, Neuracle)
with a sampling rate of 1000Hz. A total of 20 electrodes were placed on FC5,
FC3, FC1, FCz, FC2, FC4, FC6, C5, C3, C1, Cz, C2, C4, C6, CP5, CP3, CP1, CPz,
CP2, CP4 and CP6, according to the standard international 10-20 system. The
ground and reference were FPz and CPz, respectively. The impedances of all
electrodes were below the recommended value (10k$\Omega$) in the experiments.

###  III-B Comparison of BCI control performance

Here we compared the performance of the joint learning and the co-adaptive
learning method both on SVM and EEGnet. The performance was evaluated by the
accuracy of online BCI control. We chose the highest accuracy among the last
two sessions as the online BCI control accuracy, where the accuracy of
sessions was averaged by the accuracy of all trials in the sessions.

####  III-B1 Analysis on SVM with joint learning

Fig. 4A illustrated the online BCI control accuracy for both joint learning
and co-adaptive learning with the fourteen subjects. Overall, the BCI control
accuracy of joint learning sessions was significantly higher than co-adaptive
learning sessions without joint learning (paired $t$-test, $p\textless
0.005$). The average accuracies of methods with joint learning and without
joint learning were 74.75% and 69.06%. For 11 out of 14 subjects, joint
learning achieved a superior BCI control accuracy compared with the baseline
co-adaptive experiments. The performance increases were most obvious with
subjects with lower BCI control performance, indicating the joint learning
process effectively facilitated the BCI training process.

In Fig. 4B, we examined the discrete performance by the proportion of
successful trials. Specifically, we defined a trial with accuracy above the
threshold $T$ (70%) as a successful trial, which can be used as the discrete
control signal. Overall, the joint learning sessions obtained a higher
proportion of successful trials compared with the baseline. With the joint
learning system, the proportion of successful trials rose sustainingly over
the four sessions, indicating the improvement of the subject’s BCI control
ability. Specifically, the proportion of successful trials increased to 60.0%,
64.2%, 65.5% and at the 2nd, 3rd, and 4th respectively. While without joint
learning, the proportion rose with the first two sessions, and the increase
became trivial afterward. With session 4, the joint learning system
outperformed the co-adaptive system significantly by 9.83% (paired $t$-test,
$p\textless 0.05$).

Figure 5: Comparison of BCI control accuracy with and without joint learning
on EEGnet. (A) The offline accuracy of subjects with and without the joint
learning process. $\ast$ indicates the significance with $t$-test (one-sided,
$p\textless 0.005$). (B) Comparison between CSP SVM method and EEGnet. $\ast$
indicates the significance with $t$-test (one-sided, $p\textless 0.005$) (C)
The online accuracy of EEGnet both with and without joint learning. Figure 6:
Analysis of the joint learning process. With the joint learning framework,
subjects rapidly learn to generate well-separable signals (solid lines) with
the guidance of the algorithm (dashed lines). Results on all subjects are in
the Supplementary Material Fig.1.

We further compared the trial-wise accuracy of BCI control. The trial-wise
accuracy was computed by the mean accuracy of online accuracy during the
trials, which was smoothed by the window of 10 points. Fig. 4C illustrates the
average accuracy of every trial over all the subjects. For both joint and co-
adaptive systems, the initial accuracies were around 60%. With the learning
process, the accuracy of joint learning rose rapidly to 76.58%, while the
accuracy of the baseline co-adaptive learning increased slower to 70.79%. The
results indicated that the subject learned more effectively with the joint
learning system.

####  III-B2 Analysis on EEGnet with joint learning

We conducted both online and pseudo-online experiments on EEGnet with the
proposed joint learning framework. The results are presented in Fig. 5.
Specifically, we conducted pseudo-online experiments on the EEG data of 14
subjects, comparing the performance of the EEGnet algorithm with and without
joint learning. As depicted in Fig. 5A, 11 out of 14 subjects demonstrated a
significant increase in performance with the joint learning algorithm.
Moreover, the average classification accuracy improved from 62.59% to 69.03%
($p\textless 0.005$), indicating that the proposed framework is effective and
robust for both SVM and EEGnet algorithms. These findings suggested that joint
learning framework help subjects to generate better signals and improved
performance in MI-based BCI systems.

Fig. 5B demonstrated the performance comparison between SVM and EEGnet
methods. The results indicate a significant advantage of the CSP SVM method
over the deep learning EEGnet method, with accuracies of 74.75% and 69.03%,
respectively. This performance difference may be attributed to the fact that
our framework starts the discriminator with a small number of samples, which
can lead to overfitting in the neural network.

We further conducted online experiments on 4 subjects with the EEGnet and the
results were shown in Fig. 5C. Both lines represent the online accuracy in the
feedback sessions. Without the joint learning framework, the average accuracy
increased slowly, from 63.24% to 65.62%. However, with the joint learning
paradigm, subjects demonstrated a faster learning process, starting at 61.40%,
rising to 64.88% and 67.85%, ending at 69.49%. These results provided evidence
of the effectiveness of our proposed joint learning method and its flexibility
in working with different algorithms.

Figure 7:  Changes of subject’s brain signal distributions during the joint
learning process. Results on all subjects are in the Supplementary Material
Fig.2.

###  III-C Effectiveness of the joint learning method

####  III-C1 Analysis of the joint learning process

The major feature of human-machine joint learning is that the machine can
guide the learning process of the subject for more efficient BCI training. In
the MI task, the joint goal of the subject and the decoder is to maximize the
discriminative ability of the “left” and “right” brain signal patterns on the
decoder’s space.

Thus, we illustrated the discriminative ability of the “left” and “right”
signals during the training process in Fig. 6. The discriminative ability was
evaluated by the SVM score, indicating the distance to the classification
plane in the hyper-space. The blue and red lines stood for the scores of left
and right respectively. With the joint learning process, the blue and red
lines became distant rapidly, indicating the improvement of discriminative
ability and the effectiveness of the joint learning process. Specifically, for
subject1 in Fig. 6A, the initial SVM scores for both “left” and “right” were
around 0.5, which were difficult to discriminative by the decoder, indicating
bad signal quality. In the training process, the “left” and “right” became
more separated during training. After about 160 training trials (at the second
training session), the SVM scores of brain signals for “left” and “right” were
-1.59 and 0.42 respectively, which were well discriminative with the decoder.
At the end of the training process, the brain signals for “left” and “right”
were discriminative with SVM scores of -0.97 and 1.39 respectively. With the
traditional co-adaptive learning process in Fig. 6B, the learning process was
inefficient compared with the joint learning process. Specifically, the brain
signals were difficult to classify until the 250th trials, with SVM scores of
-1.04 and 0.33. And the SVM scores were -0.88 and 0.65 at the end of training.
With the subject 13, similar results were observed in Fig. 6C and Fig. 6D. It
is worth noting that, for a fair comparison, the two chosen subjects had the
experiments in a different order. Subject 1 did the baseline experiment first
and subject 13 did the joint learning first. The results strongly suggested
that the joint learning process improved the efficiency of BCI learning
effectively.

Figure 8:  Signal changes of subjects. (A) Sample distance between the left
and right signals across sessions. (B) Analysis of CSP features before and
after joint learning. Results on all subjects are in the Supplementary
Material Fig.3.

####  III-C2 Analysis of the decoder’s guidance to the subject

Here we further investigated the effectiveness of the joint learning process
by examining how the decoder guided the learning of the subject. In Fig. 6,
the dashed lines represented the mean of SVM scores accounting for the sample
weights, which indicated the expected scores of subjects’ signals by the
decoder. In Fig. 6A and Fig. 6C, it was observed that the decoder’s
expectations by sample reweighting were with higher absolute values for both
“left” and “right” brain signals, and the gap between the expected scores
between the two classes became larger during the training process. It
indicated that the brain signal quality improved with the training process,
and the decoder adaptively tuned itself to cope with the learning of the
subject. Also, the expectations from the decoder suggested the joint optimal
point for both humans and machines. With the training process, the SVM scores
of brain signals became closer to the optimal point, indicating the joint
learning process drove both the subject and the decoder to learn toward the
“efficient” direction effectively. While with the traditional co-adaptive
approach, the decoder’s expectations showed lower absolute values compared
with the sample scores, thus the learning was mostly driven by the subject
learning process, which degraded the efficiency of learning.

In order to illustrate the change of scores in detail, distributions of scores
in different sessions across the experiments were shown in Fig. 7. The red and
blue solid curves stood for the distribution of left and right scores. And the
dashed curves were weighted scores from feedback. All the distributions were
fitted by Gaussian distributions. Generally, the red and blue solid curves
followed the separation of the dashed curves. With the dash curves moving more
and more aside, the gap between solid curves also became more and more
significant. This not only demonstrated the learning process of subjects but
also presented the details of the guidance of the sample reweighting
algorithm. Quantitatively, the differences of $\mu$ (peaks of the fitted
distribution) for left and right distributions were computed to show the
separation. For the subject1, the difference was 0.44 at the first session for
the generated signal, but the difference for feedback was around 3.2. With the
training process, the difference for generated signal gently rose to 2.1 in
the last session. And the differences for feedback were always larger than it
for generated signals, leading the way. The change was almost the same for
subject13. The difference for generated signals grew from 0.1 to 4.7. While
the difference in feedback increased from 5.1 to 6.6. These changes meant
subjects learned to generate better signals with more separable distributions
with the guidance of feedback, explaining the process of joint learning.

####  III-C3 Analysis of brain signals with the learning process

Here we analyzed how the brain signals of the subjects change during the
learning process. We firstly illustrated the distance between “left” and
“right” signals along with sessions in Fig 8A. In this experiment, we re-
calculated the CSP features offline with brain signals from the last two
sessions, and the filters were applied to all 5 sessions for CSP feature
computation. The distance was computed by:

$distance=\dfrac{D_{inter}}{D_{intra}}$ (21)

where $D_{inter}$ stood for the inter-group distance, which was calculated by
the Euclidean Distance between the mean points of two groups of CSP features.
And $D_{intra}$ represented the intra-group distance, which was expressed by
the Euclidean Norm of the standard deviation of two groups. We illustrated the
average distance over 14 subjects in Fig. 8A.

With the training process, the distance between “left” and “right” brain
features became more separable for both joint learning and the baseline
approach, indicating the learning process of the subject. With the joint
learning approach, the distance between “left” and “right” increased more
rapidly from 0.229 to 0.263. The increase in joint learning was 0.034, which
was 0.020 higher than the baseline approach without joint learning. In the
last two sessions, the average distance between “left” and “right” brain
features was 0.261 for the joint learning approach, which was 0.011 higher
than the baseline method. Further, we illustrated the CSP features before and
after the joint learning experiments of a subject in Fig. 8B. For both before
and after learning conditions, we selected the most discriminative CSP
features for comparison. After the joint learning process, the subject learned
the typical MI pattern with the CSP feature peaking around the C3 channel.

The results demonstrated that the joint learning approach could help the
subject learn brain signal patterns both effectively and rapidly.

Figure 9: Analysis of the “copy/new” paradigm. (A) The accuracy of different
trials. (B) Average sample weights of different trials.

###  III-D Effectiveness of the “copy/new” paradigm

In this section, we evaluated the effectiveness of our “copy/new” paradigm
from the aspects of accuracy and weight. We compared the averaged accuracy
before and after a “copy/new” instruction. Specifically, we labeled the trials
with four groups: 1) the first trial in a “copy” pair (before a “copy”
instruction); 2) the next trial in a “copy” pair (after a “copy” instruction);
3) the first trial in a “new” pair (before a “new” instruction); 4) the next
trial in a “new” pair (after a “new” instruction).

Overall, the “copy” instruction guided the subjects to maintain a good brain
signal pattern effectively, while the next trials after a “new” instruction
demonstrated a significant performance increase. As shown in Fig. 9A, the
first trials in a “copy” pair demonstrated a high average accuracy of 88%;
after the “copy” instruction, the performance of the next trial slightly
decreased while still keeping a high average accuracy of to 81%. The results
suggested that the “copy” instruction could guide the subject to repeat and
keep a good brain signal pattern. With the “new” instruction, the first trials
were with low accuracies of about 43%. After the “new” instruction, the
average accuracy of the next trial rose to 60%. The results showed that the
subjects tried to generate a new brain signal pattern different from the
previous ones to seek a better performance, which was in line with
expectations.

Furthermore, we analyzed the weights in different types of trials to show the
cooperation between our paradigm and algorithm. Fig. 9B presented the weight
in different types of trials. The weights were calculated by averaging the
weights on trials of the selected type. Overall, although the weights varied
in a big range, the average weight on the next trials was slightly higher than
the weight on the first trials. Specifically, the mean weight on the first
trials was 0.58, and the mean weight on the next trials was 0.62 ($p\textless
0.05$ with one-sided t-test), which explained the better separability of the
next trials. This indicated that the proposed “copy/new” instructions worked
well with the sample reweighting algorithm, which also demonstrated the
validity of the proposed learning paradigm.

Figure 10: Analysis of the questionnaires. (A) Scores before and after the
experiments. (B) Scores of the interaction manner. (C) Scores of fatigue
level.

###  III-E Analysis of the questionnaires

The results of the questionnaire were presented to analyze from a subjective
point of view. The questionnaire consisted of six simple questions, measured
by scores from one to seven, which were shown in the supplementary material in
detail. The first four questions were about the ability of MI of the left and
right hand respectively. One pair was answered before the experiments and the
other pair should be scored after the experiments (How do you think about your
motor imagery ability of left/right hand before/after the experiment? one
stands for very poor, and seven stands for very good). The result was plotted
in Fig. 10A, where the score change was calculated by the sum of two questions
after the experiments minus the sum before the experiments. It could be seen
from the picture that red lines often had a higher value than blue lines,
which meant subjects thought that they had better MI ability after our
experiment with joint learning. In detail, the mean value of score change for
the co-adaptive paradigm was -0.14, while for our joint learning paradigm, the
mean change was 1.64. This meant subjects thought the co-adaptive training did
not have significant effects, while our proposed joint learning method had a
better training effect. This result was self-reported by subjects, which might
be because the better control accuracy made subjects feel more confident.

Question 5 was about the degree of interaction, and level of knowing the
feedback (Do you think you can understand the feedback? one stands for no, and
seven stands for very good understanding). The results were in Fig. 10B. Our
proposed method turned out to be slightly better than the control group,
achieving an average score of around 6.29 (5.5 in the control group). This
might be due to the “copy/new” instructions in our paradigm, giving subjects a
stronger sense of participation. Also with the instructions, the subject can
know about the separability or quality of generated signals more clearly. This
also helped with the interaction during the experiments.

The last question was about fatigue (How much fatigue do you feel during the
experiment? one stands for none, and seven stands for too much and
intolerable). Fig. 10C showed the averaged score, in which scores of both
experiments were almost the same (3.29 in the control group and 3.35 in our
experiments). So, there was no significant difference between the efforts in
the two experiments. Our experiments just helped subjects to change or hold
their thoughts. This did not increase the workload of subjects.

Figure 11:  (A) Accuracy with different $\Lambda$ and $\Delta\Lambda$ (B)
Accuracy with different number of electrodes. (C) Layout of electrodes.

##  IV Discussion

###  IV-A Novelty and contributions

Spontaneous BCI training is often hindered by the training burden it poses.
Existing co-adaptive learning methods, which adopt an alternate learning
process, have shown suboptimal performance. In such methods, either the
decoder or the user learns at a time [60, 61]. However, human and computer
algorithms are heterogeneous in information representation and learning
mechanisms. Therefore, it is difficult to let humans and computers learn
together in a uniform framework. In this study, we use brain signal samples
generated by subjects as the intermedia, to enable the joint learning process
between both sides. In the human learning process, the subject tries to
generate “better” brain samples with the guidance of the computer, which tells
the subject what are “good” samples. At the same time, the computer
continuously updates itself adaptively to the human learning process. Thus,
the learning process of both humans and computers can be optimized
simultaneously.

In this paper, we first modeled the aforementioned joint learning process in a
uniform model, with the Markov-based human learning process and the self-paced
learning-based computer learning process. And then, we implement the framework
by proposing a novel copy/new paradigm, to achieve the joint learning process.
Online experiments demonstrated the effectiveness of the joint learning
process.

###  IV-B Sensitivity of parameters

In this section, we conducted experiments to investigate the sensitivity of
the algorithm’s parameters. Specifically, we focused on two parameters,
$\Lambda$ and $\Delta\Lambda$, which control the initialization and training
pace of the algorithm. To evaluate their influence, we performed a grid search
on the data from 14 subjects. We varied $\Lambda$ in the range of [0.2, 0.4,
0.6, 0.8] and $\Delta\Lambda$ in the range of [0.05, 0.10, 0.15, 0.20]. The
results are presented in Fig. 11A. We observed that the accuracy ranged from
75.81% to 78.11%, with no significant difference found among the different
parameter settings. The highest accuracy was achieved with $\Lambda=0.2$ and
$\Delta\Lambda=0.05$, which is consistent with the parameter settings used in
the online joint learning experiment. Conversely, the lowest accuracy was
obtained with $\Lambda=0.2$ and $\Delta\Lambda=0.20$. Moreover, we found that
the accuracy increased with the increment of $\Delta\Lambda$ when
$\Lambda=0.2$. This suggests that when the number of initial samples is
relatively small, more rounds are needed to reduce the effect of random
initialization. In contrast, when more samples are included in the
initialization, the results tend to be more stable.

###  IV-C Electrode selection

Electrode selection is a critical issue for motor imagery (MI) performance, in
addition to parameters such as $\Lambda$ and $\Delta\Lambda$. The primary area
for MI in the brain is the primary motor cortex (M1), which is located in the
precentral gyrus of the frontal lobe [47]. Therefore, electrodes of interest
are typically C3 and C4, which are located over the left and right primary
motor cortex, respectively [73]. In the online experiments, we recorded data
from 20 electrodes, along with CPz as a reference. As shown in Fig. 11C, the
selected electrode set was centered at C3 and C4. To investigate the impact of
electrode selection further, we reduced the number of electrodes from the
outer side towards the center. We reduced the original 20 electrodes (FC5,
FC3, FC1, FCz, FC2, FC4, FC6, C5, C3, C1, Cz, C2, C4, C6, CP5, CP3, CP1, CPz,
CP2, CP4,CP6) to 14 electrodes (FC3, FC1, FCz, FC2, FC4, C3, C1, Cz, C2, C4,
CP3, CP1, CP2, CP4), then 8 electrodes (FC3, FCz, FC4, C3, Cz, C4, CP3, CP4),
and finally to 3 electrodes (C3, Cz, C4). The accuracy change is plotted in
Fig. 11B. As expected, the accuracy decreased with the reduction of electrodes
from 74.75% to 63.97%. The average accuracy of our proposed methods was always
higher than the method without joint learning. However, the advantage
diminished from 5.69% to 1.64%. This could be attributed to the decreasing
number of electrodes resulting in less useful information. This result also
helps to demonstrate the effectiveness of our proposed method.

###  IV-D Limitations and future works

####  IV-D1 Multi-classification problem

In this paper, we proposed novel a joint learning framework for effective BCI
training, which took an MI system of the binary classification as an example.
With the framework, different base algorithms could be deployed to deal with
the training problem. For the multi-classification problem, our framework
could easily fit to the problem. Because from the algorithm aspect, the key
part is to evaluate samples according to their loss, to guide the human
learning process. From the aspect of paradigm, the “copy/new” instructions
would remain the same. As a result, the challenge lies in the base algorithm.
In the experiments, we adopted the CSP SVM and the EEGnet as the base
algorithms. For the EEGnet, it was introduced to deal with all kinds of EEG
problems. So it would be rather convenient to deal with multi-classification
problem. For the CSP SVM, it was designed for the binary classification
problem [25]. However, there are some improvements to enhance CSP for
multiclass MI [27]. With these new methods, our proposed framework promises to
achieve good results on multi-classification problems.

####  IV-D2 Markov process and endogenous attention

As discussed in Section II-A1, the Markov process is a widely used framework
for modeling human decision-making processes. Recently, it has been applied to
the study of endogenous attention, as it can accurately reproduce reaction
times in GO-GO experiments [64]. Moreover, Markov models have been employed in
the field of visual decision-making [65], and hidden Markov models have been
used to reveal endogenous neural activities that trigger perceptual changes by
analyzing dynamic neural patterns [74]. In this paper, we model the learning
process, including decision-making about MI strategies, using a Markov
process. In future work, we aim to expand the model to include EEG patterns
and features. This will provide further insights into human learning and
decision-making processes in MI training.

####  IV-D3 Applications on various EEG devices

The emergence of advanced sensor technology has facilitated the creation of
compact and intelligent wearable electroencephalography (EEG) devices tailored
for personal use [75]. Devices with novel dry EEG sensors, which is convenient
to wear and remove, have enhanced the usability of BCI systems [76]. Wireless
devices with dry and noncontact EEG electrode sensor have also been introduced
[77]. These devices are intended for real-life usage scenarios, which are
significantly more complex than controlled laboratory environments.
Consequently, ensuring the stability of BCI systems poses a considerable
challenge. Our proposed human-machine joint learning framework exhibits vast
prospects for application and further work in this field.

##  V Conclusion

Towards efficient and effective BCI training, the proposed human-machine joint
learning framework enables the simultaneous learning of both the user and the
decoder. We model the human-machine joint learning process in a uniform
formulation and propose a novel joint learning framework for efficient BCI
training. Online and psuedo-online experiments demonstrate the effectiveness
of the proposed joint learning framework in rapid BCI learning. The proposed
framework can be extended to MI tasks with more control degrees and also has
the potential to guide the subject to generate new brain patterns out of the
MI manner, to broaden the usability of BCI systems.

##  VI ACKNOWLEDGMENT

This work was partly supported by the grants from STI 2030 Major Projects
(2021ZD0200400), and Natural Science Foundation of China (U1909202 and
61925603), and the Key Research and Development Program of Zhejiang Province
in China (2020C03004).

## References

  * [1] J. J. Shih, D. J. Krusienski, and J. R. Wolpaw, “Brain-computer interfaces in medicine,” in _Mayo Clinic Proceedings_ , vol. 87, no. 3. Elsevier, 2012, pp. 268–279.
  * [2] G. Pan, J.-J. Li, Y. Qi, H. Yu, J.-M. Zhu, X.-X. Zheng, Y.-M. Wang, and S.-M. Zhang, “Rapid decoding of hand gestures in electrocorticography using recurrent neural networks,” _Frontiers in neuroscience_ , vol. 12, p. 555, 2018.
  * [3] K. K. Ang and C. Guan, “Eeg-based strategies to detect motor imagery for control and rehabilitation,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 25, no. 4, pp. 392–401, 2016.
  * [4] I. Lazarou, S. Nikolopoulos, P. C. Petrantonakis, I. Kompatsiaris, and M. Tsolaki, “Eeg-based brain–computer interfaces for communication and rehabilitation of people with motor impairment: a novel approach of the 21st century,” _Frontiers in human neuroscience_ , vol. 12, p. 14, 2018.
  * [5] F. Cincotti, F. Pichiorri, P. Aricò, F. Aloise, F. Leotta, F. de Vico Fallani, J. d. R. Millán, M. Molinari, and D. Mattia, “Eeg-based brain-computer interface to support post-stroke motor rehabilitation of the upper limb,” in _2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society_. IEEE, 2012, pp. 4112–4115.
  * [6] J. Li, S. Qiu, Y.-Y. Shen, C.-L. Liu, and H. He, “Multisource transfer learning for cross-subject eeg emotion recognition,” _IEEE transactions on cybernetics_ , vol. 50, no. 7, pp. 3281–3293, 2019.
  * [7] S. Pancholi, A. Giri, A. Jain, L. Kumar, and S. Roy, “Source aware deep learning framework for hand kinematic reconstruction using eeg signal,” _IEEE Transactions on Cybernetics_ , 2022.
  * [8] R. Xu, N. Jiang, C. Lin, N. Mrachacz-Kersting, K. Dremstrup, and D. Farina, “Enhanced low-latency detection of motor intention from eeg for closed-loop brain-computer interface applications,” _IEEE Transactions on Biomedical Engineering_ , vol. 61, no. 2, pp. 288–296, 2013.
  * [9] Y. Qi, B. Liu, Y. Wang, and G. Pan, “Dynamic ensemble modeling approach to nonstationary neural decoding in brain-computer interfaces,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [10] T. Gu, Z. Wang, X. Xu, D. Li, H. Yang, and W. Du, “Frame-level teacher-student learning with data privacy for eeg emotion recognition,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [11] J. Bhattacharya _et al._ , “Complexity analysis of spontaneous eeg,” _Acta neurobiologiae experimentalis_ , vol. 60, no. 4, pp. 495–502, 2000\.
  * [12] F. Di Russo, A. Martínez, M. I. Sereno, S. Pitzalis, and S. A. Hillyard, “Cortical sources of the early components of the visual evoked potential,” _Human brain mapping_ , vol. 15, no. 2, pp. 95–111, 2002.
  * [13] W. Yijun, W. Ruiping, G. Xiaorong, and G. Shangkai, “Brain-computer interface based on the high-frequency steady-state visual evoked potential,” in _Proceedings. 2005 First International Conference on Neural Interface and Control, 2005._ IEEE, 2005, pp. 37–39.
  * [14] H. Wang, Y. Qi, H. Yu, Y. Wang, C. Liu, G. Hu, and G. Pan, “Rcit: An rsvp-based concealed information test framework using eeg signals,” _IEEE Transactions on Cognitive and Developmental Systems_ , 2021.
  * [15] S. Hu, H. Wang, J. Zhang, W. Kong, Y. Cao, and R. Kozma, “Comparison analysis: Granger causality and new causality and their applications to motor imagery,” _IEEE transactions on neural networks and learning systems_ , vol. 27, no. 7, pp. 1429–1444, 2015.
  * [16] S. Gao, Y. Wang, X. Gao, and B. Hong, “Visual and auditory brain–computer interfaces,” _IEEE Transactions on Biomedical Engineering_ , vol. 61, no. 5, pp. 1436–1447, 2014.
  * [17] K. Choi and A. Cichocki, “Control of a wheelchair by motor imagery in real time,” in _International conference on intelligent data engineering and automated learning_. Springer, 2008, pp. 330–337.
  * [18] A. J. Doud, J. P. Lucas, M. T. Pisansky, and B. He, “Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain-computer interface,” _PloS one_ , vol. 6, no. 10, p. e26322, 2011.
  * [19] N. Jiang, L. Gizzi, N. Mrachacz-Kersting, K. Dremstrup, and D. Farina, “A brain–computer interface for single-trial detection of gait initiation from movement related cortical potentials,” _Clinical Neurophysiology_ , vol. 126, no. 1, pp. 154–159, 2015.
  * [20] Y. Qi, L. Ding, Y. Wang, and G. Pan, “Learning robust features from nonstationary brain signals by multi-scale domain adaptation networks for seizure prediction,” _IEEE Transactions on Cognitive and Developmental Systems_ , 2021.
  * [21] Z. Wu, R. Reddy, G. Pan, N. Zheng, P. F. Verschure, Q. Zhang, X. Zheng, J. C. Principe, A. Kreilinger, M. Rohm _et al._ , “The convergence of machine and biological intelligence,” _IEEE Intelligent Systems_ , vol. 28, no. 5, pp. 28–43, 2013.
  * [22] B. J. Edelman, J. Meng, D. Suma, C. Zurn, E. Nagarajan, B. Baxter, C. C. Cline, and B. He, “Noninvasive neuroimaging enhances continuous neural tracking for robotic device control,” _Science robotics_ , vol. 4, no. 31, 2019.
  * [23] H.-J. Hwang, K. Kwon, and C.-H. Im, “Neurofeedback-based motor imagery training for brain–computer interface (bci),” _Journal of neuroscience methods_ , vol. 179, no. 1, pp. 150–156, 2009.
  * [24] C. Vidaurre and B. Blankertz, “Towards a cure for bci illiteracy,” _Brain topography_ , vol. 23, no. 2, pp. 194–198, 2010.
  * [25] K. K. Ang, Z. Y. Chin, H. Zhang, and C. Guan, “Filter bank common spatial pattern (fbcsp) in brain-computer interface,” in _2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence)_. IEEE, 2008, pp. 2390–2397.
  * [26] C. Ju and C. Guan, “Tensor-cspnet: A novel geometric deep learning framework for motor imagery classification,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–15, 2022.
  * [27] S. Kumar, T. K. Reddy, V. Arora, and L. Behera, “Formulating divergence framework for multiclass motor imagery eeg brain computer interface,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 1344–1348.
  * [28] T. K. Reddy and L. Behera, “Driver drowsiness detection: An approach based on intelligent brain–computer interfaces,” _IEEE Systems, Man, and Cybernetics Magazine_ , vol. 8, no. 1, pp. 16–28, 2022.
  * [29] K. Sadatnejad and F. Lotte, “Riemannian channel selection for bci with between-session non-stationarity reduction capabilities,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 30, pp. 1158–1171, 2022.
  * [30] R. Zhang, P. Xu, L. Guo, Y. Zhang, P. Li, and D. Yao, “Z-score linear discriminant analysis for eeg based brain-computer interfaces,” _PloS one_ , vol. 8, no. 9, p. e74433, 2013.
  * [31] J. H. Friedman, “Regularized discriminant analysis,” _Journal of the American statistical association_ , vol. 84, no. 405, pp. 165–175, 1989.
  * [32] X. Lei, P. Yang, and D. Yao, “An empirical bayesian framework for brain–computer interfaces,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 17, no. 6, pp. 521–529, 2009.
  * [33] B. Blankertz, S. Lemm, M. Treder, S. Haufe, and K.-R. Müller, “Single-trial analysis and classification of erp components—a tutorial,” _NeuroImage_ , vol. 56, no. 2, pp. 814–825, 2011.
  * [34] A. Subasi and M. I. Gursoy, “Eeg signal classification using pca, ica, lda and support vector machines,” _Expert systems with applications_ , vol. 37, no. 12, pp. 8659–8666, 2010.
  * [35] A. Bhardwaj, A. Gupta, P. Jain, A. Rani, and J. Yadav, “Classification of human emotions from eeg signals using svm and lda classifiers,” in _2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN)_. IEEE, 2015, pp. 180–185.
  * [36] P. Wang, A. Jiang, X. Liu, J. Shang, and L. Zhang, “Lstm-based eeg classification in motor imagery tasks,” _IEEE transactions on neural systems and rehabilitation engineering_ , vol. 26, no. 11, pp. 2086–2095, 2018\.
  * [37] J. Zhou, M. Meng, Y. Gao, Y. Ma, and Q. Zhang, “Classification of motor imagery eeg using wavelet envelope analysis and lstm networks,” in _2018 Chinese Control And Decision Conference (CCDC)_. IEEE, 2018, pp. 5600–5605.
  * [38] S. Tortora, S. Ghidoni, C. Chisari, S. Micera, and F. Artoni, “Deep learning-based bci for gait decoding from eeg with lstm recurrent neural network,” _Journal of neural engineering_ , vol. 17, no. 4, p. 046011, 2020\.
  * [39] X. Du, C. Ma, G. Zhang, J. Li, Y.-K. Lai, G. Zhao, X. Deng, Y.-J. Liu, and H. Wang, “An efficient lstm network for emotion recognition from multichannel eeg signals,” _IEEE Transactions on Affective Computing_ , 2020\.
  * [40] G. Dai, J. Zhou, J. Huang, and N. Wang, “Hs-cnn: a cnn with hybrid convolution scale for eeg motor imagery classification,” _Journal of neural engineering_ , vol. 17, no. 1, p. 016025, 2020.
  * [41] D. Zhang, L. Yao, K. Chen, S. Wang, X. Chang, and Y. Liu, “Making sense of spatio-temporal preserving representations for eeg-based human intention recognition,” _IEEE transactions on cybernetics_ , vol. 50, no. 7, pp. 3033–3044, 2019.
  * [42] J.-S. Bang, M.-H. Lee, S. Fazli, C. Guan, and S.-W. Lee, “Spatio-spectral feature representation for motor imagery classification using convolutional neural networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.
  * [43] Y. Hou, S. Jia, X. Lun, Z. Hao, Y. Shi, Y. Li, R. Zeng, and J. Lv, “Gcns-net: a graph convolutional neural network approach for decoding time-resolved eeg motor imagery signals,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [44] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, “Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces,” _Journal of neural engineering_ , vol. 15, no. 5, p. 056013, 2018.
  * [45] V. Gupta, J. Meenakshinathan, T. K. Reddy, and L. Behera, “Performance study of neural structured learning using riemannian features for bci classification,” in _2022 National Conference on Communications (NCC)_. IEEE, 2022, pp. 297–301.
  * [46] B. Blankertz, G. Dornhege, M. Krauledat, K.-R. Müller, and G. Curio, “The non-invasive berlin brain–computer interface: fast acquisition of effective performance in untrained subjects,” _NeuroImage_ , vol. 37, no. 2, pp. 539–550, 2007.
  * [47] G. Pfurtscheller and C. Neuper, “Motor imagery and direct brain-computer communication,” _Proceedings of the IEEE_ , vol. 89, no. 7, pp. 1123–1134, 2001.
  * [48] M. Mihara, I. Miyai, N. Hattori, M. Hatakenaka, H. Yagura, T. Kawano, M. Okibayashi, N. Danjo, A. Ishikawa, Y. Inoue _et al._ , “Neurofeedback using real-time near-infrared spectroscopy enhances motor imagery related cortical activation,” _PloS one_ , vol. 7, no. 3, p. e32234, 2012.
  * [49] Z. Wang, Y. Zhou, L. Chen, B. Gu, S. Liu, M. Xu, H. Qi, F. He, and D. Ming, “A bci based visual-haptic neurofeedback training improves cortical activations and classification performance during motor imagery,” _Journal of neural engineering_ , vol. 16, no. 6, p. 066012, 2019.
  * [50] F. Nijboer, A. Furdea, I. Gunst, J. Mellinger, D. J. McFarland, N. Birbaumer, and A. Kübler, “An auditory brain–computer interface (bci),” _Journal of neuroscience methods_ , vol. 167, no. 1, pp. 43–50, 2008.
  * [51] X. Shu, L. Yao, X. Sheng, D. Zhang, and X. Zhu, “Enhanced motor imagery-based bci performance via tactile stimulation on unilateral hand,” _Frontiers in human neuroscience_ , vol. 11, p. 585, 2017.
  * [52] L. Yao, N. Jiang, N. Mrachacz-Kersting, X. Zhu, D. Farina, and Y. Wang, “Reducing the calibration time in somatosensory bci by using tactile erd,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 30, pp. 1870–1876, 2022.
  * [53] Y. Ono, K. Wada, M. Kurata, and N. Seki, “Enhancement of motor-imagery ability via combined action observation and motor-imagery training with proprioceptive neurofeedback,” _Neuropsychologia_ , vol. 114, pp. 134–142, 2018.
  * [54] S. Perdikis and J. d. R. Millan, “Brain-machine interfaces: a tale of two learners,” _IEEE Systems, Man, and Cybernetics Magazine_ , vol. 6, no. 3, pp. 12–19, 2020.
  * [55] J. d. R. Millán, “Brain-machine interfaces: the perception-action closed loop: a two-learner system,” _IEEE Systems, Man, and Cybernetics Magazine_ , vol. 1, no. 1, pp. 6–8, 2015.
  * [56] J. R. Millan, “On the need for on-line learning in brain-computer interfaces,” in _2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No. 04CH37541)_ , vol. 4. IEEE, 2004, pp. 2877–2882.
  * [57] C. Vidaurre, C. Sannelli, K.-R. Müller, and B. Blankertz, “Co-adaptive calibration to improve bci efficiency,” _Journal of neural engineering_ , vol. 8, no. 2, p. 025009, 2011.
  * [58] Y. Li, C. Guan, H. Li, and Z. Chin, “A self-training semi-supervised svm algorithm and its application in an eeg-based brain computer interface speller system,” _Pattern Recognition Letters_ , vol. 29, no. 9, pp. 1285–1294, 2008.
  * [59] A. Llera, V. Gómez, and H. J. Kappen, “Adaptive classification on brain-computer interfaces using reinforcement signals,” _Neural Computation_ , vol. 24, no. 11, pp. 2900–2923, 2012.
  * [60] P. Shenoy, M. Krauledat, B. Blankertz, R. P. Rao, and K.-R. Müller, “Towards adaptive classification for bci,” _Journal of neural engineering_ , vol. 3, no. 1, p. R13, 2006.
  * [61] A. Abu-Rmileh, E. Zakkay, L. Shmuelof, and O. Shriki, “Co-adaptive training improves efficacy of a multi-day eeg-based motor imagery bci training,” _Frontiers in Human Neuroscience_ , p. 362, 2019.
  * [62] J. Merel, D. M. Pianto, J. P. Cunningham, and L. Paninski, “Encoder-decoder optimization for brain-computer interfaces,” _PLoS computational biology_ , vol. 11, no. 6, p. e1004288, 2015.
  * [63] J. S. Müller, C. Vidaurre, M. Schreuder, F. C. Meinecke, P. Von Bünau, and K.-R. Müller, “A mathematical model for the two-learners problem,” _Journal of neural engineering_ , vol. 14, no. 3, p. 036005, 2017.
  * [64] C. A. Mugruza-Vassallo, J. Granados, V. Flores-Benites, and L. Córdoba-Berríos, “Different markov chains modulate visual stimuli processing in a go-go experiment in 2d, 3d and augmented reality,” 2022\.
  * [65] A. Ghaderi-Kangavari, J. A. Rad, K. Parand, and M. D. Nunez, “Neuro-cognitive models of single-trial eeg measures describe latent effects of spatial attention during perceptual decision making,” _Journal of Mathematical Psychology_ , vol. 111, p. 102725, 2022.
  * [66] J. R. Busemeyer, P. D. Kvam, and T. J. Pleskac, “Comparison of markov versus quantum dynamical models of human decision making,” _Wiley Interdisciplinary Reviews: Cognitive Science_ , vol. 11, no. 4, p. e1526, 2020\.
  * [67] E. A. Feinberg and A. Shwartz, _Handbook of Markov decision processes: methods and applications_. Springer Science & Business Media, 2012, vol. 40.
  * [68] L. Jiang, D. Meng, Q. Zhao, S. Shan, and A. G. Hauptmann, “Self-paced curriculum learning,” in _Twenty-Ninth AAAI Conference on Artificial Intelligence_ , 2015.
  * [69] M. Kumar, B. Packer, and D. Koller, “Self-paced learning for latent variable models,” _Advances in neural information processing systems_ , vol. 23, 2010\.
  * [70] M. Lapin, M. Hein, and B. Schiele, “Learning using privileged information: Svm+ and weighted svm,” _Neural Networks_ , vol. 53, pp. 95–108, 2014.
  * [71] L. Jiang, D. Meng, S.-I. Yu, Z. Lan, S. Shan, and A. Hauptmann, “Self-paced learning with diversity,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [72] M. Kleiner, D. Brainard, and D. Pelli, “What’s new in psychtoolbox-3?” 2007.
  * [73] C. Neuper, R. Scherer, S. Wriessnegger, and G. Pfurtscheller, “Motor imagery and action observation: modulation of sensorimotor brain rhythms during mental control of a brain–computer interface,” _Clinical neurophysiology_ , vol. 120, no. 2, pp. 239–247, 2009.
  * [74] D. Lyu, S. Naik, D. K. Menon, and E. A. Stamatakis, “Intrinsic brain dynamics in the default mode network predict involuntary fluctuations of visual awareness,” _Nature Communications_ , vol. 13, no. 1, p. 6923, 2022.
  * [75] X. Gu, Z. Cao, A. Jolfaei, P. Xu, D. Wu, T.-P. Jung, and C.-T. Lin, “Eeg-based brain-computer interfaces (bcis): A survey of recent studies on signal sensing technologies and computational intelligence approaches and their applications,” _IEEE/ACM transactions on computational biology and bioinformatics_ , vol. 18, no. 5, pp. 1645–1666, 2021.
  * [76] L.-D. Liao, C.-T. Lin, K. McDowell, A. E. Wickenden, K. Gramann, T.-P. Jung, L.-W. Ko, and J.-Y. Chang, “Biosensor technologies for augmented brain–computer interfaces in the next decades,” _Proceedings of the IEEE_ , vol. 100, no. Special Centennial Issue, pp. 1553–1566, 2012.
  * [77] Y. M. Chi and G. Cauwenberghs, “Wireless non-contact eeg/ecg electrodes for body sensor networks,” in _2010 International Conference on Body Sensor Networks_. IEEE, 2010, pp. 297–301.