# A Transformer-based deep neural network model for SSVEP classification

Jianbo Chen  Yangsong Zhang  Yudong Pan  Peng Xu  Cuntai Guan  Laboratory for
Brain Science and Medical Artificial Intelligence, School of Computer Science
and Technology, Southwest University of Science and Technology, Mianyang,
China  MOE Key Laboratory for NeuroInformation, Clinical Hospital of Chengdu
Brain Science Institute, and Center for Information in BioMedicine, School of
Life Science and Technology, University of Electronic Science and Technology
of China, Chengdu, China  School of Computer Science and Engineering, Nanyang
Technological University, Singapore

###### Abstract

Steady-state visual evoked potential (SSVEP) is one of the most commonly used
control signal in the brain-computer interface (BCI) systems. However, the
conventional spatial filtering methods for SSVEP classification highly depend
on the subject-specific calibration data. The need for the methods that can
alleviate the demand for the calibration data become urgent. In recent years,
developing the methods that can work in inter-subject classification scenario
has become a promising new direction. As the popular deep learning model
nowadays, Transformer has excellent performance and has been used in EEG
signal classification tasks. Therefore, in this study, we propose a deep
learning model for SSVEP classification based on Transformer structure in
inter-subject classification scenario, termed as SSVEPformer, which is the
first application of the transformer to the classification of SSVEP. Inspired
by previous studies, the model adopts the frequency spectrum of SSVEP data as
input, and explores the spectral and spatial domain information for
classification. Furthermore, to fully utilize the harmonic information, an
extended SSVEPformer based on the filter bank technology (FB-SSVEPformer) is
proposed to further improve the classification performance. Experiments were
conducted using two open datasets (Dataset 1: 10 subjects, 12-class task;
Dataset 2: 35 subjects, 40-class task) in the inter-subject classification
scenario. The experimental results show that the proposed models could achieve
better results in terms of classification accuracy and information transfer
rate, compared with other baseline methods. The proposed model validates the
feasibility of deep learning models based on Transformer structure for SSVEP
classification task, and could serve as a potential model to alleviate the
calibration procedure in the practical application of SSVEP-based BCI systems.

###### keywords:

Brain-computer interface , Steady-state visual evoked potential , Transformer
, Deep learning , Filter bank

††journal: Neural Networks

##  1 Introduction

Brain-computer interface (BCI) has become a popular research direction in
human-computer interaction and medical rehabilitation, which can directly
connect the brain to external devices without going through the peripheral
nervous system, enabling bidirectional information transmission and feedback
[42, 47]. Electroencephalogram (EEG)-based BCIs obtain the intentions of the
brain through EEG signals, and have attracted attention due to the advantages
of convenience, low cost, and non-invasiveness [1]. Among the various EEG
paradigms, the high signal-to-noise ratio and low training time of steady-
state visual evoked potential (SSVEP) make it one of the most popular
paradigms.

SSVEP refers to the EEG in the visual cortex when the subject gazes at a
flickering visual stimulus modulated by a constant frequency [56]. The
frequencies of SSVEP are the same as the coding frequency of received visual
stimuli as well as its harmnoics[33]. By virtue of this characteristic of
SSVEP, it is possible to design SSVEP-based BCI system, such as SSVEP-based
speller [27], in which different targets are encoded by different stimulus
frequencies. When the subjects need to select a command, they can gaze at the
corresponding flickering target stimulus that coding the command on the
interface. The generated SSVEP can be identified by a specially designed
decoder to obtain the intention of the subject.

In the SSVEP-based BCI system, the robust classification of the SSVEPs is very
important [28]. As the SSVEP frequency is the same as the stimulus frequency,
some researches developed the algorithms based on the prior frequency
information, such as power spectral density analysis (PSDA) [40] and canonical
correlation analysis (CCA) [22], etc. In addition to the fundamental frequency
component, SSVEP also contains harmonic components whose frequencies are
multiples of the fundamental frequency [25]. Based on this characteristic,
filter bank technology was introduced to extend the original CCA (FBCCA) [5].
FBCCA uses CCA in multiple subbands of SSVEP data, and finally weights the
correlation coefficients calculated from these subbands. The FBCCA improves
the classification performance by distinguishing the fundamental frequency and
harmonics, demonstrating the effectiveness of the filter bank technique on
SSVEP classification. Nowadays, filter bank technology has been widely used in
various methods [57, 31].

Figure 1: The diagram of two classification scenarios. (a) intra-subject
classification; (b) inter-subject classification.

However, due to the complexity of EEG, SSVEP data always contains noises, such
as spontaneous EEG and electromagnetic interference, seriously polluting the
signal [17]. Traditional training-free methods (such as PSDA, CCA) have better
results only when the data length is long. To address the noise interference
in SSVEP, a series of recognition algorithms based on machine learning have
been proposed. Such methods perform under the intra-subject classification
condition, in which the training and testing data are from the same subjects,
as shown in Fig. 1(a). In this condition, the model can obtain parameters that
are more suitable for a specific subject, thereby reducing noise interference
[50]. For example, individual template based CCA (IT-CCA) calculates the
average of the subject’s existing SSVEP signals at each stimulation frequency
and uses it as the reference signal for CCA [4]. This method can add subject-
specific patterns to the reference signal, and is widely used in subsequent
algorithms. Task-related component analysis (TRCA) method obtains spatial
filters by maximizing the reconstitution between SSVEPs of different trials to
reduce the noise of SSVEPs and reference signals [26]. Correlated component
analysis (CORCA) learns spatial filters by maximizing the correlation between
data to reduce background noise [55]. Task-discriminant component analysis
(TDCA) uses multi-class linear discriminant analysis to learn spatiotemporal
filters and classify in a discriminant manner [23].

The above method has significant effect in the intra-subject classification
experiment, in which the training data and the testing data belong to the same
subject [32]. However, the collection of SSVEP data is a time-consuming and
laborious work. Hence, a potential and challenging direction is to transfer
the data from existing subjects to new subjects in the inter-subject
classification scenario, under which a classifier can be obtained with the
data from already existing subjects and then used the classifier to test the
data from new subjects, as shown in Fig. 1(b). Although many works have
improved traditional state-of-the-art methods to adapt them to inter-subject
scenario, the results may be not optimal [46]. Because the brain processes
natural sensory stimuli in a dynamic, non-fixed, and nonlinear manner, SSVEP
is non-stationary and varies widely among individuals [18]. Even the data
collected by the same subject, the data acquired at different times may also
have different distribution. These situations pose great challenges for inter-
subject experiments, and the performance of traditional machine learning-based
algorithms under inter-subject condition degrades greatly, which is far from
its performance under intra-subject condition.

In recent years, deep learning has been developed significantly and has made
milestone progress in areas such as computer vision and natural language
processing [19, 8]. Deep learning models have powerful feature extraction
capabilities and can directly be applied on the raw data[9, 34]. Deep learning
models have been used on many EEG classification tasks, including
convolutional neural networks (CNN) [60], recurrent neural networks (RNN)
[12], graph neural networks (GNN) [59], etc. Several studies have used deep
learning to process SSVEP data, achieving outstanding performance on
classification tasks, especially inter-subject classification. For instances,
EEGNet is a compact convolutional neural network (CNN) that uses CNNs to
implement the spatial-temporal filtering and feature extraction, achieving
significantly better results than traditional methods under inter-subject
conditions [41]. The idea of using temporal and spatial convolutions has
achieved promising results, which has also influenced many later algorithms.
The subject invariant SSVEP generative adversarial network (SIS-GAN) uses
generative adversarial networks to generate artificial SSVEP data to expand
the training dataset [2]. Complex convolutional neural network (CCNN) uses the
complex spectrum of SSVEP signal as the input of CNN for classification,
demonstrating the effectiveness of complex spectral features on SSVEP
classification [32]. InceptionEEG-Net (IENet) combines Inception with residual
connections and uses multi-scale convolution kernels to extract features from
receptive fields of different sizes [13]. In addition, filter bank technology
is also applied in deep learning models to extend the existing models, such as
FB-EEGNet and FBCNN [48, 58].

Although the deep learning-based SSVEP recognition model has made great
progress compared with the traditional machine learning algorithm in the
inter-subject environment, it still has large improvement space to meet the
actual needs for SSVEP-based BCI. Under the inter-subject condition, the model
should have good generalization performance for unseen subjects. In addition,
due to the black-box nature of deep learning, the interpretability of existing
deep learning-based models still need to investigate for explaining the
classification mechanisms as the traditional methods. In recent years,
Transformer becomes one of the most promising model structures, which was
first used in machine translation and quickly took natural language processing
by storm with its excellent performance [10, 37]. It was then applied to the
field of computer vision and achieved brilliant results [11]. So far,
Transformer based models become very powerful in many fields with wide
applicability, and are more interpretable compared with other neural
networks[38]. Transformer has excellent feature extraction ability, and the
extracted features have better performance on downstream tasks. In BCI
systems, some studies have applied Transformers to the EEG classification
tasks, and achieved good results [44, 14, 30]. To the best of our knowledge,
the Transformer has not been leveraged for SSVEP classification.

In this paper, a deep learning model for SSVEP classification based on
Transformer structure, termed as SSVEPformer, is proposed. Inspired by
previous study [32], the complex spectra of the SSVEP data were adopted as the
input of the SSVEPformer model, allowing the model to focus on the frequency
domain property of SSVEP data. In addition, we presented a extended
SSVEPformer based on the filter bank technology, termed as FB-SSVEPformer, to
further improve the classification performance by fully utilizing the harmonic
information. To validate the performance of the proposed and compared models,
we utilized two public SSVEP datasets. Dataset 1 has 12 categories from ten
subject [27], and Dataset 2 has 40 categories from 35 subjects [39]. Using 1 s
time window, FB-SSVEPformer achieves 88.37$\%$ accuracy and 112.45 bits/min
information transfer rate (ITR) on Dataset 1, and 83.19$\%$ accuracy and
157.65 bits/min ITR on Dataset 2, under inter-subject classification scenario.

The rest of the paper is structured as follows: Section 2 introduces the
datasets and baseline methods, and presents the proposed SSVEPformer and FB-
SSVEPformer. The performance of the baseline methods and the proposed method
on the two datasets is presented in the Section 3. Section 4 discusses the
performance and limitations of the model. Finally, Section 5 presents the
conclusion for this study.

##  2 Materials and methods

###  2.1 Datasets

Two public datasets were adopted to evaluate the performance of all the
methods.

Dataset 1 [27]: This dataset was acquired with 12 visual target stimuli on a
27-inch LCD monitor, which were modulated by the frequencies ranged from 9.25
Hz to 14.75 Hz with 0.5 Hz steps. Ten subjects with normal or corrected-to-
normal vision participated in the experiment, sitting in chairs 60 cm from the
monitor in a dimly lit room. The BioSemi ActiveTwo EEG system (Biosemi, Inc.)
was used to acquire EEG data from eight electrodes in the occipital region.The
whole experiment consisted of 15 blocks. In each block, there were 12 trials
corresponding to the 12 flickering target stimuli, in each of which the
subjects were required to gaze at one of the 12 target stimuli. The gazed
target stimulus in each trial was random selected, and each trial lasted for 4
seconds. The EEG signal was sampled at a sampling rate of 2048 Hz, and then
downsampled to 256 Hz.

Dataset 2 [39]: The dataset consisted of 40 visual target stimuli displayed on
an LCD monitor. The 40 targets were encoded using the joint frequency and
phase modulation (JFPM) method, with target stimulation frequencies ranged
from 8 Hz to 15.8 Hz with 0.2 Hz step, and phases starting at 0 with 0.5 $\pi$
steps. Thirty-five subjects with normal or corrected-to-normal vision
participated in the experiment, eight of whom had experience using SSVEP-based
spell. For each subject, the experiment consisted of 6 blocks. In each block,
the subjects stared at 40 targets in random order according to the prompts,
resulting in a total of 40 trials. Each trial lasted for 6 seconds. For the
first 0.5 seconds, the subjects were asked to move the realization to the
target stimulus position according to the prompt, then fixated on the target
stimulus for 5 seconds, and finally the monitor was blank for 0.5 seconds. EEG
data were acquired using a Synamps2 EEG system (Neuroscan, Inc.) at a sampling
rate of 1000 Hz and down-sampled to 250 Hz. Finally, a 50 Hz notch filter was
used to eliminate power frequency interference.

###  2.2 Data preprocessing

As in the previous study, all eight channels (O1, Oz, O2, PO3, POZ, PO4, PO7,
PO8) in Dataset 1 and nine channels (O1, Oz, O2, PO3, POZ, PO4, PZ, PO5 and
PO6) in Dataset 2 were adopted for classification [32, 26]. For the models
that need time-domain data as input, fourth-order Butterworth filter with 8-64
Hz bandpass range are used to filter the data and take the filtered data as
input. Suppose the time window length for classification be $d~{}s$,
considering the visual delay, the data segments in Dataset 1 were extract in
the time window $[0.135~{}s,d+0.135~{}s]$ after the stimulus onset [27]; the
data segments were extracted in the time window $[0.64~{}s,d+0.64~{}s]$ in the
Dataset 2 [39].

For models that require frequency domain data as input, FFT was used to
convert EEG data in the time domain into the frequency domain. The result of
the FFT can be expressed as:

$FFT(x)=Re[FFT(x)]+iIm[FFT(x)]$ (1)

where $x$ represents the preprocessed EEG data in the time domain, $i$ is the
imaginary unit, $Re$ and $Im$ represent the real and imaginary parts of
complex spectrum, respectively. For frequency domain data, there are two ways
to transform it into model input, namely magnitude spectrum $X_{mag}$ and
complex spectrum $X_{comp}$ [32]:

$\displaystyle X_{mag}=\sqrt{\\{Re[FFT(x)]\\}^{2}+\\{Im[FFT(x)]\\}^{2}}$ (2)
$\displaystyle X_{comp}=Re[FFT(x)]||Im[FFT(x)]$ (3)

The symbol $||$ denotes the concatenation operation. The magnitude spectrum
calculates the sum of the squares of the real and imaginary parts at each
frequency point, discarding the phase information of the data and only
containing the magnitude information. The complex spectrum concatenates the
real and imaginary parts of the complex Fourier spectrum, and both magnitude
and phase information are preserved. Previous studies have shown that phase
information has a role in SSVEP classification [29, 7], and complex spectrum
input also outperforms magnitude spectrum in comparative experiments [32].
Therefore, in this study, we used the complex spectrum $X_{comp}$ of the data
as the input of the proposed model. The complex spectra input denoted as
$I_{comp}$, is defined as:

$I_{comp}=\begin{bmatrix}X_{comp}(CH_{1})\\\ X_{comp}(CH_{2})\\\ \vdots\\\
X_{comp}(CH_{n})\par\end{bmatrix}$ (4)

where $CH_{1},CH_{2},\cdots,CH_{n}$ represent different EEG channels.
Specifically, for dataset 1, the time-domain input data is padded with zeros
so that the resolution after FFT is 0.25 Hz; for dataset 2, zero padding is
still used so that the frequency resolution after FFT is 0.2 Hz. After that,
we selected the complex spectrum in the range of 8 to 64 Hz, and concatenated
the real part and the imaginary part as the input of SSVEPformer and FB-
SSVEPformer.

###  2.3 The baseline methods

####  2.3.1 TRCA

TRCA is a spatial filtering method that learns the spatial filters by
maximizing the reproducibility between existing data of the same class, which
can extract task-related components in the data [26]. The average of existing
data of the same class is then used as the reference signal for that class.
With the spatial filters obtained by TRCA based on the calibration data, the
correlation coefficients between the projected features of the test sample and
various reference signals can be computed, and then obtained the
classification result. In the experiment of this study, owing to the inter-
subject classification scenario, the data of one test subject will be selected
as the test sampels, and the data of all other subjects were pooled together
as the calibration data to calculate the spatial filter and the reference
signals of each stimulus frequency.

####  2.3.2 EEGNet

EEGNet is a deep learning model specially designed for EEG signal processing,
and has been widely used in various EEG classification tasks since it was
proposed, such as motor imagery, P300, SSVEP, etc [41, 20]. Waytowich et al.
applied EEGNet to SSVEP classification and achieved excellent results in the
inter-subject classification task [41]. EEGNet consists of four layers. The
first layer is a convolutional layer that simulates filtering operation on the
EEG data in each channel. The second layer is a depth-wise convolutional
layer, which is equivalent to a spatial filter that weights all the channels.
The third layer is a separable convolutional layer for extracting categorical
features. The fourth layer is a fully connected layer, which outputs the
classification result.

####  2.3.3 CCNN

CCNN transforms the SSVEP to the frequency domain using FFT, using the complex
spectrum of the signal as the input of the model in order to extract frequency
and phase information [32]. The model consists of convolutional layers and
fully connected layers. Convolutional layers are responsible for spatial
filtering, temporal filtering, and feature extraction. The fully connected
layer summarizes the extracted features to get the final result. CCNN achieves
excellent results using complex spectrum input, proving that complex spectrum
representation can be beneficial for SSVEP classification.

###  2.4 The proposed SSVEPformer

Figure 2: The architecture of SSVEPformer model. SSVEPformer consists of six
blocks: the input, complex spectrum representation, channel combination,
SSVEPformer encoder, MLP head and the output. The “$\times 2$” means that two
identical and successive operation sub-encoders shown in the red rectangle box
constitute the SSVEPformer encoder.

The model adapts to the unique characteristics of SSVEP as much as possible on
the basis of the Transformer structure, and can be considered as a
Transformer-esque SSVEP recognition model. Fig. 2 illustrates the model
architecture diagram. The SSVEPformer consists of six blocks: the input, the
complex spectrum representation, channel combination, SSVEPformer encoder,
multilayer perceptron (MLP) head and the output. In the complex spectrum
representation block, the input EEG was transformed into complex spectrum
$I_{comp}$ as defined in formula (4). As the core components of SSVEPformer,
the detailed description of channel combination, SSVEPformer encoder and MLP
head is as follows, and the detailed structure is shown in Table 1. In the
following description, $N$ is denoted as the total number of classes, $C$ is
the number of SSVEP channels, and $F$ is the length of the complex spectrum in
each channel.

Table 1: Detailed architecture and parameters in the core components of
SSVEPformer.

Block | Module | Layer | Output size | Explanation
---|---|---|---|---
Channel combination |  | Input | (C, F) |
Conv1d | (2 $\times$ C, F) | filters = 2 $\times$ C, kernalsize = 1, padding = ’same’
LayerNorm | (2 $\times$ C, F) |
Activation | (2 $\times$ C, F) | GELU
Dropout | (2 $\times$ C, F) | dropoutrate = 0.5
SSVEPformer encoder | Sub-encoders ($\times$ 2) | CNN module | LayerNorm | (2 $\times$ C, F) |
Conv1d | (2 $\times$ C, F) | filters = 2 $\times$ C, kernalsize = 31, padding = ’same’
LayerNorm | (2 $\times$ C, F) |
Activation | (2 $\times$ C, F) | GELU
Dropout | (2 $\times$ C, F) | dropoutrate = 0.5
Residual1 | (2 $\times$ C, F) | adding the input of this module
Channel MLP module | LayerNorm | (2 $\times$ C, F) |
Linear | (2 $\times$ C, F) |
Activation | (2 $\times$ C, F) | GELU
Dropout | (2 $\times$ C, F) | dropoutrate = 0.5
Residual2 | (2 $\times$ C, F) | adding the input of this module
MLP head |  | Flatten | (2 $\times$ C $\times$ F) |
Dropout | (2 $\times$ C $\times$ F) | dropoutrate = 0.5
Linear | (6 $\times$ N) |
LayerNorm | (6 $\times$ N) |
Activation | (6 $\times$ N) | GELU
Dropout | (6 $\times$ N) | dropoutrate = 0.5
Linear | (N) |

####  2.4.1 Channel combination block

The SSVEP data were usually acquired from multiple channels, and these data
not only contain valuable SSVEP classification information, but also include
various artifacts that interfere with the classification. Since the channels
are distributed at different locations on the scalp of brain, the background
components may be different. Using the channel combination to calculate the
weighted combination of all the channels can suppress the noise and enhance
the SSVEP component that helps the classification. In addition, the channel
combination will be used multiple times to obtain multiple channel weighted
combinations. This is because different combination methods focus on different
classification information or suppress different noises, and multiple
combination operations can improve the robustness and performance of the
model. The channel combination block uses convolutional layers to perform
weighted combination between channels. The convolutional layer convolves the
number of data channels from $C$ to $2\times C$ using a convolution kernel of
length 1 with Conv1d function in Pytorch framework. This process is equivalent
to learning $2\times C$ spatial filters to weight the data of each channel.

####  2.4.2 SSVEPformer encoder block

The original Transformer encoder generally consists of two components, an
attention module and a channel MLP module [37]. The former is used to mix
information between tokens, and the latter includes channel MLP for feature
extraction. Although the attention mechanism was widely used in the original
Transformer, many subsequent studies have found that the attention mechanism
is not necessary for the Transformer [36]. Some studies have used convolution,
MLP or even pooling to replace the attention module and achieved similar
results [49]. Similarly, SSVEPformer keep the channel MLP module unchanged and
replace the attention module with a CNN module.

The SSVEPformer encoder consists of two identical and successive sub-encoders
as shown in Fig. 2. Each sub-encoder consists of two modules, the first is the
CNN module and the second is the channel MLP module. Residual connection is
used in each module, the input of each module is also added to the output of
this module. In the CNN module, the Conv1d layer extracts features in the
channel dimension using a convolution kernel of length 31, and fuses the
features of each channel. In the channel MLP module, features are extracted on
each channel using a linear transformation. This linear transformation applies
the same operation to each channel, mapping from $F$ elements to new $F$
elements, using high computational cost to obtain global fine-grained
features.

####  2.4.3 MLP head block

The features extracted by the upstream blocks are finally input into the MLP
head block, which is mainly composed of two fully connected layers. The input
data is first flattened to facilitate subsequent operations. Followed by two
fully connected layers, the data length is gradually mapped to the number of
categories of the dataset. Considering that the input data is already the
features extracted by the previous blocks, in order to avoid losing useful
information, the MLP head uses two linear layers to continuously refine the
input features to obtain the final result.

The LayerNorm layer, GELU layer, and Dropout layer used in the network are all
common components of deep neural networks. The LayerNorm layer regularizes the
data in the channel dimension to ensure the stability of the data feature
distribution, making training faster and more stable [45]. The GELU layer adds
nonlinear operations to the network, making the model capable of nonlinear
fitting [16]. Meanwhile, GELU has been proven to be a high-performance
activation function and one of the current state-of-the-art activation
functions [53]. Dropout can avoid overfitting during model training and
improve model generalization [3].

###  2.5 FB-SSVEPformer

In the SSVEP data, there are harmonic components whose frequencies are
multiple times of the fundamental frequency, which can also be used as
classification features. For example, FBCCA filters the data into several
subbands and actively guides the model to pay attention to the information of
harmonics, and achieves better results than original CCA [5]. Here, we
presented a extended SSVEPformer based on the filter bank technology (FB-
SSVEPformer) to further improve the classification performance by fully
utilizing the harmonic information. The model structure is shown in Fig. 3.
The original EEG data are filtered into $S$ frequency subbands, which are then
fed to the corresponding subnetworks as input. FB-SSVEPformer uses SSVEPformer
as a subnetwork, and each subnetwork handles data of different subbands. The
results of each subnetwork($\rho_{s,f},s=1,2,\cdots,S,f=1,2,\cdots,N$) are
finally weighted and fused to obtain the final result $\mathop{Y_{f}}$ as
follows:

Figure 3: The architecture of FB-SSVEPformer, where $S$ is the number of
subbands and the subnetwork is SSVEPformer. The input data is transformed into
the data of $S$ subbands through the filter bank, and then transformed into
the frequency domain through FFT. Each subnetwork uses the data of
corresponding subband to obtain the result, and then obtains the final result
through the convolutional fusion operation.

$\sideset{}{}{\mathop{Y_{f}}}=\sideset{}{}{\mathop{argmax}}_{f={1,...,N}}\sum_{s=1}^{S}w_{s}\cdot\rho_{s,f}$
(5)

where $N$ is the total number of categories, $w_{s}(s=1,2,\cdots,S)$ is the
weighted parameters of the convolution kernel, and $S$ is the number of
subbands. The fusion operation refers to the method of FBCCA fusing the
results of each subband, which gives special weights to the results of each
subband, and then weights them as the final result. However, the weights in
FBCCA are constants obtained by grid search [6]. FB-SSVEPformer uses
convolutional layers to implement weighting operation, and the parameters move
in the direction of the best classification result during training and
eventually stabilize.

For the selection of subbands, the lower and upper cutoff frequencies of the
$m-th$ subband were set to $m\times 8$ Hz and 80 Hz on Dataset 1, and $m\times
9$ Hz and 80 Hz on Dataset 2. The effect of the subband division method has
been verified [5, 55, 26]. During the implementation of band-pass filtering,
the lower limit of each subband was subtracted by an additional 2 Hz bandwidth
[5]. In order to verify the relationship between the number of subbands and
the classification effect, FB-SSVEPformer with different numbers of subbands
uses data with a data length of 1 s to conduct experiments on Dataset 1, and
the results are shown in Fig. 4. When the number of subbands is 4, the
accuracy of the model is slightly improved, but the number of parameters is
greatly increased compared to when the number of subbands is 3. When the
number of subbands is 2, the accuracy of the model is lower than that when the
number of subbands is 3. Considering the balance between the performance and
model parameters, the number of subbands in the FB-SSVEPformer model is set to
3 in following analysis.

Figure 4: The relationship between the number of subbands of the FB-
SSVEPformer and the accuracy and parameters of the model. The experiment uses
Dataset 1 with the data length of 1 s. The bars represent the average
precision, the lines represent the number of parameters, and the ’M’ means the
magnitude of millions.

###  2.6 Experimental settings and performance evaluation

All the deep learning models were implemented with the Pytorch framework. For
the SSVEPformer and FB-SSVEPformer models, the convolutional and linear layer
parameters were initialized with a normal distribution with mean 0 and
variance 0.01. During training, cross-entropy was used to calculate the loss,
and backpropagation was used to update the parameters. Both models used a
stochastic gradient descent(SGD) algorithm to compute parameter updates with
the learning rate of 0.001, the momentum of 0.9, and an L2 regularization
penalty of 0.001. The batch size was set to 128, and the dropout rate is set
to 0.5. Notably, for SSVEPformer, the number of training epochs is 100. For
FB-SSVEPformer, the model first uses data from different frequency bands to
train each subnetwork for 100 epochs to stabilize their results, and then
trains the entire model for 20 epochs.

Three models of EEGNet, CCNN, TRCA were used as baseline methods to compare
with the proposed two models. For the fair comparison, the preprocessing
procedures of the input data are the same as the operation described above.
For CCNN method, the EEG data need further transform into complex spectrum
[32].

The classification accuracy and ITR were used as metrics to evaluate the
performance of each method. The accuracy is the ratio of the number of the
correctly classified samples to the number of the total test samples. ITR
(bits/minute) is calculated as follows [43]:

$ITR=\frac{60}{T}\times\left[log_{2}N+Plog_{2}P+(1-P)log_{2}\frac{1-P}{N-1}\right]$
(6)

where $N$ is the total number of categories and $T$ is the average time
(seconds) for selection. During calculating the ITR, 0.5 s gaze movement time
was added into the parameter $T$ as previous studies [26, 6]. For example,
when the data length is 1 s, then the T is set to 1.5 s during the ITR
calculation with the formula (6).

In current study, we focused on the inter-subject classification experiment,
the leave-one-subject-out(LOSO) strategy was adopted. Specifically, when
evaluating on each of the two datasets, the data of one subject was used as
the test set, and the data of all other subjects were used for training set.
The procedure repeated until all the subjects were served as the test subject
once.

##  3 Result

In order to evaluate all the methods, the experiments were conducted on
Dataset 1 and Dataset 2. The TRCA, EEGNet and CCNN were used as the baseline
methods, which have achieve excellent performance in previous studies [26, 32,
41].

###  3.1 Dataset 1

Figure 5: The average classification results across subjects of the five
methods with different data length on Dataset 1. (a) average accuracies; (b)
average ITRs. The error bars indicate the standard errors of each results.

Fig. 5(a) and (b) show the average classification accuracies and ITRs of the
five methods on Dataset 1, respectively. The data length ranges from 0.5 s to
1 s with an interval of 0.1 s. The results show that the proposed SSVEPformer
and FB-SSVEPformer outperform other compared baseline methods. FB-SSVEPformer
achieves the best results in both average classification accuracies and ITRs,
and SSVEPformer achieves better results than the three baseline methods,
especially when the data length is shorter than 1 s. From 0.5 s to 1 s, the
average accuracies of SSVEPformer are 64.93$\%$, 68.61$\%$, 73.72$\%$,
78.61$\%$, 82.33$\%$ and 84.16$\%$, and the ITRs are 94.17 bits/min, 95.35
bits/min, 99.90 bits/min, 103.96 bits/min, 105.74 bits/min and 102.25
bits/min, respectively. From 0.5 s to 1 s, the average accuracies of FB-
SSVEPformer are 68.00$\%$, 71.72$\%$, 78.22$\%$, 82.44$\%$, 85.00$\%$, and
88.37$\%$, and the ITRs are 103.86 bits/min, 103.91 bits/min, 112.10 bits/min,
113.14 bits/min, 111.73 bits/min and 112.45 bits/min, respectively.

###  3.2 Dataset 2

Figure 6: The average classification results across subjects of the five
methods with different data length on Dataset 2. (a) average accuracies; (b)
average ITRs. The error bars indicate the standard errors of each results.

Fig. 6(a) and (b) show the average classification accuracies and ITRs of the
five methods on Dataset 2, respectively. As the results on Dataset 1, the
experimental results show that the SSVEPformer and FB-SSVEPformer outperform
other three methods. FB-SSVEPformer achieves the best results in both average
classification accuracies and ITRs, and SSVEPformer is also better than the
baseline methods. From 0.5 s to 1 s, the average accuracies of SSVEPformer are
51.08$\%$, 58.71$\%$, 66.06$\%$, 73.20$\%$, 78.10 $\%$, and 80.40$\%$, and the
ITRs are 113.31 bits/min, 126.83 bits/min, 139.14 bits/min, 149.41 bits/min,
153.68 bits/min and 149.95 bits/min, respectively. From 0.5 s to 1 s, the
average accuracies of FB-SSVEPformer are 54.42$\%$, 62.63$\%$, 71.05$\%$,
76.90$\%$, 81.18$\%$ and 83.19$\%$ and the ITRs are 125.13 bits/min, 140.10
bits/min, 154.36 bits/min, 160.76 bits/min, 162.28 bits/min and 157.65
bits/min, respectively.

Based on the results on the two widely used datasets for the method
evaluation, we could find that the SSVEPformer and FB-SSVEPformer promote the
SSVEP classification performance in the inter-subject scenario.

##  4 Discussion

###  4.1 A feasible method in inter-subject scenario for SSVEP classification

The SSVEP-based BCI could provide enough number of targets to code the
commands or characters for the application. Although lots of methods have been
proposed [51], it is still challenging to recognize the target with the EEG
signals for the users, especially when dealing with the large number of
targets. In recent years, the spatial filtering methods based on the
calibration data become a popular solution to achieve high classification
accuracy, such as the TRCA method and its variant. Whereas, the collection of
calibration data is time-consuming and laborious. Therefore, a potential
strategy is to utilize the data from existing subjects to train the method for
the new subjects in a inter-subject classification scenario that can realize
the plug-and-play application without new data collection and calibration
procedure. Unfortunately, the data distribution may vary largely among
different subjects, which result in the performance of traditional algorithms
under inter-subject scenario degrades greatly. These situations pose great
challenges for inter-subject classification with the traditional method, such
as TRCA [32].

In recent years, deep learning methods have achieved great success for brain
signals analysis [54]. Deep learning-based solutions for alleviating the
calibration data under different BCI paradigms have gradually increased in
recent years. For the SSVEP paradigm, the CCNN model provide a option for
inter-subject classification without the calibration data from a new subject
[32]. Even so, it still has large improvement space to meet the practical
needs. In recent years, the Transformer and the variants have achieved state-
of-the-art performance in various fields [15, 21]. In this study, we attempted
to design a deep learning model with Transformer structure for SSVEP
classification in calibration-free data condition. We proposed SSVEPformer and
a variant FB-SSVEPformer with filter bank technology, which is the first
application of the Transformer to the SSVEP classification. Evaluated on two
public datasets with different numbers of targets, with data length of 1 s,
the SSVEPformer obtains 84.16 $\%$ and 80.40 $\%$ accuracies, and the FB-
SSVEPformer obtains 88.37 $\%$ and 83.19 $\%$ accuracies, on 12-class and
40-class classification task, respectively. The experimental results show that
the proposed models could achieve excellent performance in inter-subject
classification task. The proposed model validates the feasibility of deep
learning models based on Transformer structure for calibration-free SSVEP
classification task, and could serve as a potential model to alleviate the
calibration procedure in the practical application of SSVEP-based BCI systems.

###  4.2 Enhancing the performance with subject-adaptive strategy

Figure 7: Subject-adaptive experimental results of SSVEPformer and FB-
SSVEPformer. The experiments were coducted on Dataset 2 with the duration of 1
second. The horizontal axis is the amount of data used for subject-adaptive
training. When the number of blocks is 0, it means that no subject-adaptive
training was performed.

In recent years, deep learning-based methods have been widely used in SSVEP
classification, showing no less performance than traditional methods in both
classification performance and generalization. In the SSVEP-based BCI system,
when the user uses it for a period of time, some of the user’s own data could
be collected. These data can be used to update the model for adapting to the
user’s data and improve the performance of the system. This method of adding
part of the data from the test subject in inter-subject classification
scenario is subject-adaptive strategy [52]. Here, in order to investigate the
performance of the proposed method under the subject-adaptive strategy, we use
the data with 1 second length in dataset 2 for experiment. Specifically, for
SSVEPformer, after the model is trained under the inter-subject rule, it will
continue to train for 30 epochs using part of the data from the test subject,
and finally validate on the remaining data from that test subject. For FB-
SSVEPformer, after the model is trained under the inter-subject rule, it will
first use part of the data from the test subject to train each subnetwork for
20 epochs, then train the entire model for 10 epochs, and finally verify it on
the remaining data from that test subject. Furthermore, we checked the
influence of the different numbers of data blocks at each frequency from the
test subject on the classification result for subject-adpative training. The
result is shown in the Fig. 7. It can be seen that using the data from test
subject for subject-adaptive can significantly improve the performance of the
model, and the accuracy of both models has increased above 90$\%$.
Surprisingly, after subject-adaptive with only one block data at each
frequency, the performance was improved by 11.80$\%$ and 10.33$\%$ for
SSVEPformer and FB-SSVEPformer, respectively. Besides, when the number of the
data blocks increases from one to five, the accuracies incease by 2.72$\%$ and
2.36$\%$ for the two methods, respectively. These results show that the
proposed models can achieve satisfactory performance when only a small amount
of data are available for subject-adaptive classification.

###  4.3 Model visualization and interpretation

Figure 8: The t-SNE visualization results of a representative subject with the
four models on Dataset 1. (a) EEGNet; (b) CCNN; (c) SSVEPformer; (d) FB-
SSVEPformer. Each dot represents a sample data, and different colors indicate
the category to which the data belongs.

To further display the possible reasons that the proposed methods achieve
better results than the baseline methods, we adopted the t-Stochastic
Neighborhood Embedding (t-SNE) to visualize the learned embedding features of
the four deep learning methods that yield the top 4 accuracies, i.e, EEGNet,
CCNN, SSVEPformer, FB-SSVEPformer. Owing to the large number of categories on
Dataset 2 that is hard for plotting the results, we only present the
experiment results on Dataset 1 in this section. Fig. 8 shows the
visualization results of a representative subject(subject No.3) with the four
models on Dataset 1. It can be seen that the features extracted by the
proposed SSVEPformer and FB-SSVEPformer have smaller intra-category distance
and larger inter-category distance than those of the baseline methods, which
can result in better classification results.

Figure 9: Visualization results of SSVEPformer using grad-CAM, which represent
the correlation of each position in the input data with the output of the
SSVEPformer encoder. (a) and (b) are grad-CAM visualization results for 11.75
Hz and all 12.75 Hz, respectively, for the a representative subject in Dataset
1. (c) and (d) are grad-CAM visualization results for 8.4 Hz and 11.6 Hz,
respectively, for the a representative subject in Dataset 2. The upper and
lower subgraphs in (a), (b), (c) and (d) are the results of the real and
imaginary parts of the input data, respectively. The yellow line represents
the mean of real or imaginary amplitude in the input data of channel Oz. The
blue line denotes the weights from the grad-CAM. The vertical red dotted lines
indicate the positions fundamental frequency and harmonics.

The classification mechanism of the deep learning model is still not as
intuitive as the traditional machine learning method. So, the model
interpretability is an important property for the deep learning method. In
current study, we try to adopt the gradient-weighted Class Activation Mapping
(grad-CAM) to investigate the interpretability of the SSVEPformer model [35].
Grad-CAM can visually demonstrate how the deep learning model makes the
decision based on the input data. The higher the weights, the greater the
contribution of the corresponding data of the input data to the result.
Concretely, we calculated the heatmaps(weights) with the grad-CAM to represent
the relevance of each position in the input data for the output of SSVEPformer
encoder. Fig. 9 shows the grad-CAM visualization results of 11.75 Hz and 12.75
Hz for the representative subject (subject No.4) on Dataset 1, and 8.4 Hz and
all 11.6 Hz for the representative subject (subject No.32) on Dataset 2. It
can be observed that for both the real part and the imaginary part of the
input data, the weights at the stimulation frequency and harmonics points are
obviously higher than those of other frequency points. The results of grad-CAM
prove that the SSVEPformer can find the classification features in the input
data, and use the basic frequency information of the SSVEP data for
classification decisions.

###  4.4 Limitation

Even though both SSVEPformer and FB-SSVEPformer achieve promising performance,
some limitations still exist in this study. First, we only used two public
datasets to test the model, and more datasets such as BETA should be adopted
in future experiments [24]. Second, although SSVEPformer and FB-SSVEPformer
were implemented in inter-subject scenarios, they still require a large amount
of data from existing subjects for training. When only limited data are
available, how to further compress the amount of training data while
maintaining the model performance is a problem worthy of study. Besides, all
experiments in this study are performed under offline conditions, and future
work can further investigate the efficiency and effectiveness in the online
SSVEP-based BCI system.

##  5 Conclusion

In the case of time-consuming and laborious data collection, designing a model
that can yield excellent classification result under inter-subject conditions
is a realistic requirement for SSVEP-based BCI systems. According to the
structure of Transformer and the characteristics of SSVEP data, we proposed a
SSVEPformer model and its variant FB-SSVEPformer with the filter bank
techonolgy. To validate the model performance, we conducted extensive
experiments on two public datasets under inter-subject conditions with data
lengths ranging from 0.5 s to 1 s. Experimental results show that the proposed
model outperforms three popular baseline methods on both datasets. The FB-
SSVEPformer model can achieve the best results. Furthermore, we used grad-CAM
to visualize the impact of different locations of the input data on the output
results, demonstrating the interpretability of the model. The proposed model
has high interpretability, and holds promising potential to promote the
practical applications of SSVEP-based BCI systems.

## Acknowledgments

This work was supported in part by the National Natural Science Foundation of
China under Grant No.62076209 and No.61871423.

## References

  * Abiri et al. [2019] Abiri, R., Borhani, S., Sellers, E.W., Jiang, Y., Zhao, X., 2019.  A comprehensive review of EEG-based brain–computer interface paradigms.  Journal of neural engineering 16, 011001.
  * Aznan et al. [2021] Aznan, N.K.N., Atapour-Abarghouei, A., Bonner, S., Connolly, J.D., Breckon, T.P., 2021.  Leveraging Synthetic Subject Invariant EEG Signals for Zero Calibration BCI, in: 2020 25th International Conference on Pattern Recognition (ICPR), IEEE. pp. 10418–10425.
  * Baldi and Sadowski [2013] Baldi, P., Sadowski, P.J., 2013\.  Understanding dropout.  Advances in neural information processing systems 26\.
  * Bin et al. [2011] Bin, G., Gao, X., Wang, Y., Li, Y., Hong, B., Gao, S., 2011.  A high-speed BCI based on code modulation VEP.  Journal of neural engineering 8, 025015\.
  * Chen et al. [2015a] Chen, X., Wang, Y., Gao, S., Jung, T.P., Gao, X., 2015a.  Filter bank canonical correlation analysis for implementing a high-speed ssvep-based brain-computer interface.  Journal of neural engineering 12, 046008.
  * Chen et al. [2015b] Chen, X., Wang, Y., Nakanishi, M., Gao, X., Jung, T.P., Gao, S., 2015b.  High-speed spelling with a noninvasive brain–computer interface.  Proceedings of the national academy of sciences 112, E6058–E6067.
  * Chen et al. [2014] Chen, X., Wang, Y., Nakanishi, M., Jung, T.P., Gao, X., 2014.  Hybrid frequency and phase coding for a high-speed SSVEP-based BCI speller, in: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, IEEE. pp. 3993–3996.
  * Collobert et al. [2011] Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P., 2011\.  Natural language processing (almost) from scratch.  Journal of machine learning research 12, 2493–2537.
  * Craik et al. [2019] Craik, A., He, Y., Contreras-Vidal, J.L., 2019.  Deep learning for electroencephalogram (EEG) classification tasks: a review.  Journal of neural engineering 16, 031001.
  * Devlin et al. [2018] Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805 .
  * Dosovitskiy et al. [2020] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al., 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929 .
  * Du et al. [2020] Du, X., Ma, C., Zhang, G., Li, J., Lai, Y.K., Zhao, G., Deng, X., Liu, Y.J., Wang, H., 2020.  An efficient LSTM network for emotion recognition from multichannel EEG signals.  IEEE Transactions on Affective Computing .
  * Du and Liu [2022] Du, Y., Liu, J., 2022.  IENet: a robust convolutional neural network for EEG based brain-computer interfaces.  Journal of Neural Engineering .
  * Guo et al. [2022] Guo, J.Y., Cai, Q., An, J.P., Chen, P.Y., Ma, C., Wan, J.H., Gao, Z.K., 2022\.  A Transformer based neural network for emotion recognition and visualizations of crucial EEG channels.  Physica A: Statistical Mechanics and its Applications 603, 127700.
  * Han et al. [2022] Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., Yang, Z., Zhang, Y., Tao, D., 2022.  A survey on vision transformer.  IEEE Transactions on Pattern Analysis and Machine Intelligence , 1–1doi:10.1109/TPAMI.2022.3152247.
  * Hendrycks and Gimpel [2016] Hendrycks, D., Gimpel, K., 2016\.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415 .
  * Hsu et al. [2015] Hsu, H.T., Lee, I.H., Tsai, H.T., Chang, H.C., Shyu, K.K., Hsu, C.C., Chang, H.H., Yeh, T.K., Chang, C.Y., Lee, P.L., 2015\.  Evaluate the feasibility of using frontal SSVEP to implement an SSVEP-based BCI in young, elderly and ALS groups.  IEEE Transactions on Neural Systems and Rehabilitation Engineering 24, 603–615.
  * Ibanez-Soria et al. [2019] Ibanez-Soria, D., Soria-Frisch, A., Garcia-Ojalvo, J., Ruffini, G., 2019\.  Characterization of the non-stationary nature of steady-state visual evoked potentials using echo state networks.  PloS one 14, e0218771.
  * Krizhevsky et al. [2017] Krizhevsky, A., Sutskever, I., Hinton, G.E., 2017.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM 60, 84–90.
  * Lawhern et al. [2018] Lawhern, V.J., Solon, A.J., Waytowich, N.R., Gordon, S.M., Hung, C.P., Lance, B.J., 2018\.  EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces.  Journal of neural engineering 15, 056013.
  * Lin et al. [2021] Lin, T., Wang, Y., Liu, X., Qiu, X., 2021.  A survey of transformers.  arXiv preprint arXiv:2106.04554 .
  * Lin et al. [2006] Lin, Z., Zhang, C., Wu, W., Gao, X., 2006.  Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs.  IEEE transactions on biomedical engineering 53, 2610–2614.
  * Liu et al. [2021] Liu, B., Chen, X., Shi, N., Wang, Y., Gao, S., Gao, X., 2021.  Improving the performance of individually calibrated SSVEP-BCI by task-discriminant component analysis.  IEEE Transactions on Neural Systems and Rehabilitation Engineering 29, 1998–2007.
  * Liu et al. [2020] Liu, B., Huang, X., Wang, Y., Chen, X., Gao, X., 2020\.  BETA: A large benchmark database toward SSVEP-BCI application.  Frontiers in neuroscience 14, 627\.
  * Müller-Putz et al. [2005] Müller-Putz, G.R., Scherer, R., Brauneis, C., Pfurtscheller, G., 2005\.  Steady-state visual evoked potential (SSVEP)-based communication: impact of harmonic frequency components.  Journal of neural engineering 2, 123\.
  * Nakanishi et al. [2017] Nakanishi, M., Wang, Y., Chen, X., Wang, Y.T., Gao, X., Jung, T.P., 2017\.  Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis.  IEEE Transactions on Biomedical Engineering 65, 104–112.
  * Nakanishi et al. [2015] Nakanishi, M., Wang, Y., Wang, Y.T., Jung, T.P., 2015\.  A comparison study of canonical correlation analysis based methods for detecting steady-state visual evoked potentials.  PloS one 10, e0140703.
  * Nakanishi et al. [2014] Nakanishi, M., Wang, Y., Wang, Y.T., Mitsukura, Y., Jung, T.P., 2014.  A high-speed brain speller using steady-state visual evoked potentials.  International journal of neural systems 24, 1450019.
  * Pan et al. [2011] Pan, J., Gao, X., Duan, F., Yan, Z., Gao, S., 2011\.  Enhancing the classification accuracy of steady-state visual evoked potential-based brain–computer interfaces using phase constrained canonical correlation analysis.  Journal of neural engineering 8, 036027\.
  * Phan et al. [2022] Phan, H., Mikkelsen, K., Chén, O.Y., Koch, P., Mertins, A., De Vos, M., 2022\.  SleepTransformer: Automatic sleep staging with interpretability and uncertainty quantification.  IEEE Transactions on Biomedical Engineering 69, 2456–2467.
  * Qin et al. [2021] Qin, K., Wang, R., Zhang, Y., 2021.  Filter Bank-Driven Multivariate Synchronization Index for Training-Free SSVEP BCI.  IEEE Transactions on Neural Systems and Rehabilitation Engineering 29, 934–943.  doi:10.1109/TNSRE.2021.3073165.
  * Ravi et al. [2020] Ravi, A., Beni, N.H., Manuel, J., Jiang, N., 2020\.  Comparing user-dependent and user-independent training of CNN for SSVEP BCI.  Journal of neural engineering 17, 026028.
  * Regan [1989] Regan, D., 1989.  Evoked potentials and evoked magnetic fields in science and medicine.  Human brain electrophysiology , 59–61.
  * Schmidhuber [2015] Schmidhuber, J., 2015.  Deep learning in neural networks: An overview.  Neural networks 61, 85–117.
  * Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D., 2017\.  Grad-cam: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE international conference on computer vision, pp. 618–626.
  * Tay et al. [2021] Tay, Y., Bahri, D., Metzler, D., Juan, D.C., Zhao, Z., Zheng, C., 2021\.  Synthesizer:rethinking self-attention for transformer models, in: International conference on machine learning, PMLR. pp. 10183–10192.
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., 2017\.  Attention is all you need, in: Advances in neural information processing systems, pp. 6000–6010.
  * Vig [2019] Vig, J., 2019.  A multiscale visualization of attention in the transformer model.  arXiv preprint arXiv:1906.05714 .
  * Wang et al. [2016] Wang, Y., Chen, X., Gao, X., Gao, S., 2016.  A benchmark dataset for SSVEP-based brain-computer interfaces.  IEEE Transactions on Neural Systems and Rehabilitation Engineering 25, 1746–1752.
  * Wang et al. [2006] Wang, Y., Wang, R., Gao, X., Hong, B., Gao, S., 2006\.  A practical VEP-based brain-computer interface.  IEEE Transactions on neural systems and rehabilitation engineering 14, 234–240.
  * Waytowich et al. [2018] Waytowich, N., Lawhern, V.J., Garcia, J.O., Cummings, J., Faller, J., Sajda, P., Vettel, J.M., 2018.  Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials.  Journal of neural engineering 15, 066031.
  * Wolpaw [2007] Wolpaw, J.R., 2007.  Brain-computer interfaces (BCIs) for communication and control, in: Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility, pp. 1–2.
  * Wolpaw et al. [2002] Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M., 2002.  Brain-computer interfaces for communication and control.  Clinical neurophysiology 113, 767–791.
  * Xie et al. [2022] Xie, J., Zhang, J., Sun, J., Ma, Z., Qin, L., Li, G., Zhou, H., Zhan, Y., 2022.  A Transformer-Based Approach Combining Deep Learning Network and Spatial-Temporal Information for Raw EEG Classification.  IEEE Transactions on Neural Systems and Rehabilitation Engineering 30, 2126–2136.
  * Xu et al. [2019] Xu, J., Sun, X., Zhang, Z., Zhao, G., Lin, J., 2019\.  Understanding and improving layer normalization.  Advances in Neural Information Processing Systems 32\.
  * Yan et al. [2022] Yan, W., Wu, Y., Du, C., Xu, G., 2022.  Cross-subject spatial filter transfer method for SSVEP-EEG feature recognition.  Journal of Neural Engineering 19, 036008.
  * Yao et al. [2020] Yao, D., Zhang, Y., Liu, T., Xu, P., Gong, D., Lu, J., Xia, Y., Luo, C., Guo, D., Dong, L., et al., 2020\.  Bacomics: a comprehensive cross area originating in the studies of various brain–apparatus conversations.  Cognitive Neurodynamics 14, 425–442.
  * Yao et al. [2022] Yao, H., Liu, K., Deng, X., Tang, X., Yu, H., 2022\.  FB-EEGNet: A fusion neural network across multi-stimulus for SSVEP target detection.  Journal of Neuroscience Methods 379, 109674.
  * Yu et al. [2022] Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., Yan, S., 2022.  Metaformer is actually what you need for vision, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10819–10829.
  * Zerafa et al. [2018a] Zerafa, R., Camilleri, T., Falzon, O., Camilleri, K.P., 2018a.  To train or not to train? A survey on training of feature extraction methods for SSVEP-based BCIs.  Journal of Neural Engineering 15, 051001.
  * Zerafa et al. [2018b] Zerafa, R., Camilleri, T., Falzon, O., Camilleri, K.P., 2018b.  To train or not to train? A survey on training of feature extraction methods for SSVEP-based BCIs.  Journal of Neural Engineering 15, 051001.
  * Zhang et al. [2021a] Zhang, K., Robinson, N., Lee, S.W., Guan, C., 2021a.  Adaptive transfer learning for eeg motor imagery classification with deep convolutional neural network.  Neural Networks 136, 1–10.
  * Zhang et al. [2021b] Zhang, X., Chang, D., Qi, W., Zhan, Z., 2021b.  A study on different functionalities and performances among different activation functions across different ANNs for image classification.  Journal of Physics: Conference Series 1732, 012026.
  * Zhang et al. [2021c] Zhang, X., Yao, L., Wang, X., Monaghan, J., Mcalpine, D., Zhang, Y., 2021c.  A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers.  Journal of Neural Engineering 18, 031002.
  * Zhang et al. [2018] Zhang, Y., Guo, D., Li, F., Yin, E., Zhang, Y., Li, P., Zhao, Q., Tanaka, T., Yao, D., Xu, P., 2018.  Correlated component analysis for enhancing the performance of SSVEP-based brain-computer interface.  IEEE Transactions on Neural Systems and Rehabilitation Engineering 26, 948–956.
  * Zhang et al. [2012] Zhang, Y., Xu, P., Liu, T., Hu, J., Zhang, R., Yao, D., 2012.  Multiple frequencies sequential coding for SSVEP-based brain-computer interface.  PLoS ONE 7, e29519.
  * Zhang et al. [2019] Zhang, Y., Yin, E., Li, F., Zhang, Y., Guo, D., Yao, D., Xu, P., 2019.  Hierarchical feature fusion framework for frequency recognition in SSVEP-based BCIs.  Neural Networks 119, 1–9.
  * Zhao et al. [2021] Zhao, D., Wang, T., Tian, Y., Jiang, X., 2021.  Filter Bank Convolutional Neural Network for SSVEP Classification.  IEEE Access 9, 147129–147141.
  * Zhong et al. [2020] Zhong, P., Wang, D., Miao, C., 2020.  EEG-based emotion recognition using regularized graph neural networks.  IEEE Transactions on Affective Computing .
  * Zhou et al. [2018] Zhou, M., Tian, C., Cao, R., Wang, B., Niu, Y., Hu, T., Guo, H., Xiang, J., 2018.  Epileptic seizure detection based on EEG signals and CNN.  Frontiers in neuroinformatics 12, 95.


* Lyu et al. (2020) Zhaoyang Lyu, Ching-Yun Ko, Zhifeng Kong, Ngai Wong, Dahua Lin, and Luca Daniel. 2020\.  Fastened CROWN: Tightened Neural Network Robustness Certificates. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_. AAAI Press, 5037–5044.  https://doi.org/10.1609/aaai.v34i04.5944
  * Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018\.  Towards Deep Learning Models Resistant to Adversarial Attacks. In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.  https://openreview.net/forum?id=rJzIBfZAb
  * Maréchal and Périn (2017) Alexandre Maréchal and Michaël Périn. 2017.  Efficient elimination of redundancies in polyhedra using raytracing.
  * Mirman et al. (2018) Matthew Mirman, Timon Gehr, and Martin T. Vechev. 2018\.  Differentiable Abstract Interpretation for Provably Robust Neural Networks. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ _(Proceedings of Machine Learning Research)_ , Jennifer G. Dy and Andreas Krause (Eds.), Vol. 80. PMLR, 3575–3583.  http://proceedings.mlr.press/v80/mirman18b.html
  * Morrison et al. (2016) David R Morrison, Sheldon H Jacobson, Jason J Sauppe, and Edward C Sewell. 2016.  Branch-and-bound algorithms: A survey of recent advances in searching, branching, and pruning.  _Discrete Optimization_ 19 (2016), 79–102.  https://doi.org/10.1016/j.disopt.2016.01.005
  * Motzkin et al. (1953) Theodore S Motzkin, Howard Raiffa, Gerald L Thompson, and Robert M Thrall. 1953.  The double description method.  _Contributions to the Theory of Games_ 2, 28 (1953), 51–73.  https://doi.org/10.1515/9781400881970-004
  * Müller et al. (2021) Christoph Müller, Francois Serre, Gagandeep Singh, Markus Püschel, and Martin Vechev. 2021\.  Scaling Polyhedral Neural Network Verification on GPUs.  _Proc. Machine Learning and Systems (MLSys)_ (2021).
  * Müller et al. (2020) Christoph Müller, Gagandeep Singh, Markus Püschel, and Martin Vechev. 2020.  Neural Network Robustness Verification on GPUs.  arXiv:cs.LG/2007.10868
  * Palma et al. (2021) Alessandro De Palma, Harkirat S. Behl, Rudy R. Bunel, Philip H. S. Torr, and M. Pawan Kumar. 2021\.  Scaling the Convex Barrier with Active Sets. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.  https://openreview.net/forum?id=uQfOy7LrlTR
  * Raghunathan et al. (2018) Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018\.  Semidefinite relaxations for certifying robustness to adversarial examples. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 10900–10910.  https://proceedings.neurips.cc/paper/2018/hash/29c0605a3bab4229e46723f89cf59d83-Abstract.html
  * Ruoss et al. (2020a) Anian Ruoss, Maximilian Baader, Mislav Balunović, and Martin Vechev. 2020a.  Efficient Certification of Spatial Robustness.  _ArXiv preprint_ abs/2009.09318 (2020).  https://arxiv.org/abs/2009.09318
  * Ruoss et al. (2020b) Anian Ruoss, Mislav Balunovic, Marc Fischer, and Martin T. Vechev. 2020b.  Learning Certified Individually Fair Representations. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).  https://proceedings.neurips.cc/paper/2020/hash/55d491cf951b1b920900684d71419282-Abstract.html
  * Ryou et al. (2020) Wonryong Ryou, Jiayu Chen, Mislav Balunovic, Gagandeep Singh, Andrei Dan, and Martin Vechev. 2020\.  Fast and effective robustness certification for recurrent neural networks.  _ArXiv preprint_ abs/2005.13300 (2020).  https://arxiv.org/abs/2005.13300
  * Salman et al. (2019a) Hadi Salman, Jerry Li, Ilya P. Razenshteyn, Pengchuan Zhang, Huan Zhang, Sébastien Bubeck, and Greg Yang. 2019a.  Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 11289–11300.  https://proceedings.neurips.cc/paper/2019/hash/3a24b25a7b092a252166a1641ae953e7-Abstract.html
  * Salman et al. (2019b) Hadi Salman, Greg Yang, Huan Zhang, Cho-Jui Hsieh, and Pengchuan Zhang. 2019b.  A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 9832–9842.  https://proceedings.neurips.cc/paper/2019/hash/246a3c5544feb054f3ea718f61adfa16-Abstract.html
  * Sartipizadeh and Vincent (2016) Hossein Sartipizadeh and Tyrone L Vincent. 2016.  Computing the approximate convex hull in high dimensions.  _ArXiv preprint_ abs/1603.04422 (2016).  https://arxiv.org/abs/1603.04422
  * Seidel (1995) Raimund Seidel. 1995\.  The upper bound theorem for polytopes: an easy proof of its asymptotic version.  _Computational Geometry_ 5, 2 (1995), 115–116.  https://doi.org/10.1016/0925-7721(95)00013-Y
  * Singh et al. (2019a) Gagandeep Singh, Rupanshu Ganvir, Markus Püschel, and Martin T. Vechev. 2019a.  Beyond the Single Neuron Convex Barrier for Neural Network Certification. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 15072–15083.  https://proceedings.neurips.cc/paper/2019/hash/0a9fdbb17feb6ccb7ec405cfb85222c4-Abstract.html
  * Singh et al. (2018) Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin T. Vechev. 2018\.  Fast and Effective Robustness Certification. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 10825–10836.  https://proceedings.neurips.cc/paper/2018/hash/f2f446980d8e971ef3da97af089481c3-Abstract.html
  * Singh et al. (2019b) Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. 2019b.  An abstract domain for certifying neural networks.  _Proceedings of the ACM on Programming Languages_ 3, POPL (2019), 1–30.  https://doi.org/10.1145/3290354
  * Singh et al. (2019c) Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin T. Vechev. 2019c.  Boosting Robustness Certification of Neural Networks. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.  https://openreview.net/forum?id=HJgeEh09KQ
  * Singh et al. (2017) Gagandeep Singh, Markus Püschel, and Martin Vechev. 2017\.  Fast Polyhedra Abstract Domain. In _Proc. Principles of Programming Languages (POPL)_. 46–59.  https://doi.org/10.1145/3009837.3009885
  * Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014\.  Intriguing properties of neural networks. In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).  http://arxiv.org/abs/1312.6199
  * Tjandraatmadja et al. (2020) Christian Tjandraatmadja, Ross Anderson, Joey Huchette, Will Ma, Krunal Patel, and Juan Pablo Vielma. 2020.  The Convex Relaxation Barrier, Revisited: Tightened Single-Neuron Relaxations for Neural Network Verification. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).  https://proceedings.neurips.cc/paper/2020/hash/f6c2a0c4b566bc99d596e58638e342b0-Abstract.html
  * Tjeng et al. (2019) Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. 2019\.  Evaluating Robustness of Neural Networks with Mixed Integer Programming. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.  https://openreview.net/forum?id=HyGIdiRqtm
  * Udacity (2016) Udacity. 2016.  Using Deep Learning to Predict Steering Angles.  https://github.com/udacity/self-driving-car.
  * Urban and Miné (2021) Caterina Urban and Antoine Miné. 2021.  A Review of Formal Methods applied to Machine Learning.  _ArXiv preprint_ abs/2104.02466 (2021).  https://arxiv.org/abs/2104.02466
  * Wang et al. (2018) Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018.  Efficient Formal Safety Analysis of Neural Networks. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 6369–6379.  https://proceedings.neurips.cc/paper/2018/hash/2ecd2bd94734e5dd392d8678bc64cdab-Abstract.html
  * Wang et al. (2021) Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. 2021.  Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Verification.  _ArXiv preprint_ abs/2103.06624 (2021).  https://arxiv.org/abs/2103.06624
  * Weng et al. (2018) Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane S. Boning, and Inderjit S. Dhillon. 2018.  Towards Fast Computation of Certified Robustness for ReLU Networks. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ _(Proceedings of Machine Learning Research)_ , Jennifer G. Dy and Andreas Krause (Eds.), Vol. 80. PMLR, 5273–5282.  http://proceedings.mlr.press/v80/weng18a.html
  * Wong et al. (2018) Eric Wong, Frank R. Schmidt, Jan Hendrik Metzen, and J. Zico Kolter. 2018.  Scaling provable adversarial defenses. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 8410–8419.  https://proceedings.neurips.cc/paper/2018/hash/358f9e7be09177c17d0d17ff73584307-Abstract.html
  * Xiang et al. (2018) Weiming Xiang, Hoang-Dung Tran, and Taylor T Johnson. 2018\.  Output reachable set estimation and verification for multilayer neural networks.  _IEEE transactions on neural networks and learning systems_ 29, 11 (2018), 5777–5783.  https://doi.org/10.1109/TNNLS.2018.2808470
  * Xu et al. (2020) Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. 2020.  Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).  https://proceedings.neurips.cc/paper/2020/hash/0cbc5671ae26f67871cb914d81ef8fc1-Abstract.html
  * Xu et al. (2021) Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, and Cho-Jui Hsieh. 2021.  Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.  https://openreview.net/forum?id=nVZtXBI6LNn
  * Xu et al. (1998) Zong-Ben Xu, Jiang-She Zhang, and Yiu-Wing Leung. 1998\.  An approximate algorithm for computing multidimensional convex hulls.  _Applied mathematics and computation_ 94, 2-3 (1998), 193–226.  https://doi.org/10.1016/S0096-3003(97)10043-1
  * Zhang et al. (2018) Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. 2018.  Efficient Neural Network Robustness Certification with General Activation Functions. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 4944–4953.  https://proceedings.neurips.cc/paper/2018/hash/d04863f100d59b3eb688a11f95b0ae60-Abstract.html
  * Zhong et al. (2014) Jinhong Zhong, Ke Tang, and A Kai Qin. 2014.  Finding convex hull vertices in metric space. In _2014 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1587–1592.  https://doi.org/10.1109/IJCNN.2014.6889699