# EEGEyeNet: a Simultaneous Electroencephalography and Eye-tracking Dataset
and Benchmark for Eye Movement Prediction

Ard Kastrati1 Martyna Beata Płomecka211footnotemark: 1 Damián Pascual1 Lukas
Wolf1
Victor Gillioz1 Roger Wattenhofer1 Nicolas Langer2
1ETH Zurich, Switzerland
2University of Zurich, Switzerland
<EMAIL_ADDRESS><EMAIL_ADDRESS>Both authors contributed equally

###### Abstract

We present a new dataset and benchmark with the goal of advancing research in
the intersection of brain activities and eye movements. Our dataset,
EEGEyeNet, consists of simultaneous Electroencephalography (EEG) and Eye-
tracking (ET) recordings from 356 different subjects collected from three
different experimental paradigms. Using this dataset, we also propose a
benchmark to evaluate gaze prediction from EEG measurements. The benchmark
consists of three tasks with an increasing level of difficulty: left-right,
angle-amplitude and absolute position. We run extensive experiments on this
benchmark in order to provide solid baselines, both based on classical machine
learning models and on large neural networks. We release our complete code and
data and provide a simple and easy-to-use interface to evaluate new methods.

##  1 Introduction

Tracking eye position is a subject of active research due to its multiple
applications across different fields such as behavioural science [1],
assistive technology [2], or user experience [3]. Eye tracking in combination
with conventional research methods, like behavioral measures, can help assess
and potentially diagnose neurological diseases such as Autism Spectrum
Disorder [4], Obsessive Compulsive Disorder [5], Schizophrenia [6],
Parkinson‘s [7], and Alzheimer‘s disease [8]. Additionally, eye tracking
technology can be used to detect states of drowsiness [9], to support
communication for locked-in patients [10], and to measure attention in
marketing [11]. In the last decade, technological advances have allowed
complementing eye-tracking technology with EEG — a non-invasive, minimally
restrictive, and, relatively low-cost measure of mesoscale brain dynamics with
high temporal resolution. Combining behavioral information gained from eye
tracking with the neurophysiological markers provided by EEG enables
researchers to study perceptual, attentional or cognitive processes in
naturalistic situations [12]. Notably, estimating gaze position using EEG
would make available gaze position information in a wide variety of studies
that cannot acquire eye tracking data otherwise, either because of the
unavailability of ET hardware or the lack of in-house expertise. This, in
turn, could accelerate scientific discovery on human behavior and neurological
and psychiatric diseases, particularly during free viewing of complex stimuli
(i.e. naturalistic paradigms) and in clinical settings, in which the
installation of an eye-tracker is impractical (e.g. hospital bed).

Advancing cognitive research at the intersection of brain dynamics and eye
movement requires synchronized data from EEG and eye-tracking. Such data
contains gaze pattern of eye movements recorded with eye- tracking as well as
the neurophysiological markers provided by EEG, allowing researchers to study
attention and reaction time [13], or to improve brain-computer interfaces
[14]. In the same line, estimating gaze position from EEG signals is typically
approached with machine learning and deep learning models [15, 16], which need
significant amounts of annotated data for training. However, collecting and
annotating simultaneous EEG and eye position data is time-consuming and
expensive since it requires equipment and expertise for both EEG acquisition
and eye-tracking. Hence, the access to concurrently recorded EEG-ET data is
highly restricted, which significantly slows down progress in this field. To
help bridging this gap, we release EEGEyeNet, a large dataset of EEG data
synchronized with precise eye-tracking recordings. Furthermore, given the
multiple benefits that EEG-based eye tracking can bring to different domains,
in conjunction with our dataset, we present a benchmark for evaluating gaze
estimation from EEG. This benchmark comprises three tasks of increasing
difficulty and is conceived as a tool to facilitate comparable and
reproducible research on gaze estimation from EEG data. We run extensive
experiments on the proposed benchmark to establish baseline performance. In
order to foster further research in this field we make all of our code and
infrastructure available in the following site: http://www.eegeye.net.

To conclude, our key contributions can be summarized as:

  * •

A dataset of high-density 128-channel EEG data synchronized with video-
infrared eye tracking from 356 healthy adults that amounts to a total of more
than 47 hours of recording. Along with the raw dataset we provide preprocessed
data and code for preprocessing and feature extraction.

  * •

A benchmark for gaze estimation from EEG signals that consists of three
evaluation tasks with increasing difficulty. This benchmark is built on a
subset of the EEGEyeNet dataset.

  * •

An extensive experimentation that establishes baseline performance on the
proposed benchmark.

##  2 Related Work

Gaze prediction is an active research topic with applications in human
behaviour analysis [17], advertisement [18], and human-computer interaction
[19], to name a few. Previous research found evidence suggesting that action
selection is facilitated by attention [1]. Furthermore, [20] demonstrated the
possibility of performing activity recognition from eye movements. To predict
gaze location, some models use saliency maps [21, 22], while others leverage
machine learning techniques to estimate gaze position from indirect data:
Krafka et al. [23] use webcam images, and Son et al. [24] and LaConte et al.
[25] use functional magnetic resonance imaging (fMRI). Similarly, O’Connell
and Chun [26] reconstructed fixations maps, which can predict eye movement
patterns, directly from fMRI data. Interestingly, our recent work indicates
that it might be possible to infer gaze direction directly from EEG data [16].
Although to our knowledge no previous work has employed deep learning to
estimate eye position from EEG, there are studies demonstrating that combining
EEG and ET can improve performance in various tasks, as compared with a single
modality. In particular, this has been reported in the vigilance estimation
[27], information extraction and sentiment analysis [28], or analysing users’
behaviour when performing a web search [29].

#### Related Datasets.

There exist some openly available datasets that combine EEG and ET data.
However, in comparison to EEGEyeNet, they are acquired from a smaller sample
of individuals [28, 30] or using a less advanced EEG-ET setup [31, 32]. A
prominent example of a multimodal neurophysiological dataset, including EEG
and ET data collected from a significant sample of participants (126
individuals), was proposed by Langer et al. [33]. However, it is devised as a
resource for assessing information processing in the developing brain and
contains data from young participants only. This way, EEGEyeNet dataset is the
first large-scale and precisely annotated EEG-ET dataset containing recording
from participant across the adult lifespan (18-80 years old). In Table 1 we
compare the EEGEyeNet dataset with the cited datasets in terms of the number
of participants, their age, and recording/session length.

Paper/Dataset | $Participants$ | $Female/Male$ | $Age$ |  $Recording$ $Length$ |  $Session$ $Length$
---|---|---|---|---|---
ZuCo 1.0 | 12 | 5/7 | 25-51 | x | 48-72h
ZuCo 2.0 | 18 | 8/10 | 23-52 | x | 30h-54h
Notaro et al | 22 | 11/11 | 18-40 | 7h | x
MAMEM | 36 | 9/27 | 25-71 | x | 52h
Langer et al | 126 | 56/70 | 6-44 | x | 378h
EEGEyeNet | 356 | 190/166 | 18-80 | 47h | 415h

Table 1: Metadata comparison of the cited datasets. Since [28, 30, 32, 33]
reported only the duration of the entire experimental session (including
participant’s preparation, practice trials that were aimed to acquaint the
participant with the experimental procedures, breaks between the subsequent
experiments, etc.), we decided to distinguish the recording’s length from the
whole session length.

##  3 EEGEyeNet Dataset

In this section, we provide a detailed description of the EEGEyeNet dataset.
Together with the raw data, we release two sets of preprocessed data:
minimally and maximally preprocessed; as well as the preprocessing code. This
way, we give users the freedom to manipulate raw data while easing the
experimentation barrier by additionally providing ready-to-use clean data.

###  3.1 Data Acquisition

#### Participants.

Data were recorded from 356 healthy adults. The study included 190 female and
166 male participants, of ages between 18 and 80 years. All participants gave
their written informed consent before participation in the experiment and
received a monetary compensation (the local currency equivalent of 50 US
Dollars). The data was collected according to the principles expressed in the
Declaration of Helsinki [34].

Figure 1: Recording Setup

#### Recording setup.

High-density EEG data was recorded at a sampling rate of 500 Hz, with midline
central recording reference, using a 128-channel EEG Geodesic Hydrocel system.
The impedance of each electrode was checked prior to each recording session
and kept below 40 kOhm. Simultaneously, eye position was recorded with an
infrared video-based ET EyeLink 1000 Plus from SR Research at a sampling rate
of 500 Hz and an instrument spatial resolution of less than 0.01$\degree$ root
mean square (RMS) of the distances between successive samples. The ET was
calibrated with a 9-point grid before each recording. In a validation step,
the ET calibration was repeated until the error between two measurements at
any point was less than 0.5$\degree$, or the average error for all points was
less than 1$\degree$. Participants were seated at a distance of 68 cm from a
24-inch monitor with a resolution of $800\times 600$ pixels. A stable head
position was ensured with a chin rest. The illustration of the recording setup
can be seen in Figure 1.

###  3.2 Preprocessing

EEG data is often contaminated by artifacts produced by environmental factors,
e.g., temperature, air humidity, as well as other sources of electromagnetic
noise, such as line noise [35]. These artifacts interact in a complex manner
with participant-related artifacts, typically reflecting unwanted
physiological signals such as eye movements, eye blinks, muscular noise, heart
signals or sweating, which differ from participant to participant. The
resulting artifacts in the EEG data are typically more prominent than the
signal of interest (i.e. brain activity). Therefore, EEG data requires
preprocessing in the form of artifact cleaning or artifact correction [36]. We
preprocessed the entire EEGEyeNet dataset using the openly available toolbox
from Pedroni et al. [37] in two ways: minimally and maximally. Minimal
preprocessing includes the detection and interpolation of bad electrodes, and
filtering the data with 40 Hz high-pass filter and 0.5 Hz low-pass filter (cf.
Appendix A). The difference between these two types of preprocessing is that
maximal preprocessing removes a much larger number of artifacts (muscles,
heart, eyes, line noise, channel noise). To do this, independent component
analysis (ICA) is applied in combination with _IClabel_ [38], a pre-trained
classifier that estimates the probability of a component reflecting
artifactual activity. If a component receives a probability estimation larger
than $0.8$ for any class of artifact we remove it from the data. Minimally
preprocessed data includes ocular artifacts, which is expected to make the
estimation of gaze position easier. On the other hand, maximal preprocessing
is a state-of-the-art technique for neuroscientific applications [37] that
aims to keep only neurophysiological information in the data.

After preprocessing (both minimally and maximally), the EEG and eye-tracking
data were synchronized using “EYE EEG” [39] to enable EEG analyses time-locked
to the onsets of relevant events depending on the experimental paradigm.
Synchronization quality was ensured by comparing the trigger latencies
recorded in the EEG and eye-tracker data. All synchronization errors did not
exceed 2 ms.

###  3.3 Data Annotation

Existing literature studying eye movement generally distinguishes between
three different events [40]: saccades, fixations, and blinks. Saccades are
rapid, ballistic eye movements that instantly change the gaze position.
Fixations are defined as time periods without saccades, and blinks are
considered a special case of fixation, where the pupil diameter is zero. For
each of the experimental paradigms described in Section 3.4, we provide
annotations in the form of start and end time of each event, as well as the
start and end position of saccades and the average position of fixations. See
Appendix B for further details.

###  3.4 Experimental Paradigms

Figure 2:  Pro- and Antisaccade. Schematic view of the experimental setup and
gaze behavior during a prosaccade (left) and antisaccade (right) trial.

#### Pro- and Antisaccade.

The pro- and antisaccade paradigm was designed according to the
internationally standardized protocol for antisaccade testing developed by
Antoniades et al. [41] and is described in detail in [42]. Each trial starts
with a central fixation square. The participants are asked to focus on the
center of the screen for a randomized time-period between 1 and 3.5 seconds.
Subsequently, the cue (i.e. a dot) appears horizontally on the left or the
right hand-side of the central fixation square. In the prosaccade trials, the
participants are asked to focus their gaze on the cue as fast as possible,
while in the antisaccade trials the participants are instructed to perform a
saccade towards the opposite side of the cue. In both cases, the cue is shown
for one second. As soon as the cue disappears, the participants shift their
focus back to the center of the screen; this is illustrated in Figure 2. Data
recorded following this paradigm may be used for different research purposes,
such as estimating gaze direction or examining responses to inhibition.

Figure 3: Large Grid. Schematic view of the experimental setup and stimuli
location on the screen.

#### Large Grid.

Participants are asked to fixate on a series of dots that are sequentially
presented, each at one of 25 different screen positions. Unlike the others,
the dot at the center of the screen appears three times, resulting in 27
trials (displayed dots) per block, each dot is displayed for $1.5$ to $1.8$
seconds. The positions of the dots were selected to ensure coverage of all
corners of the screen as well as the center (see Figure 3). The shape of the
grid and its use for eye gaze estimation follows the work from Son et al.
[24]. Given that Son et al. [24] used the Large Grid paradigm for functional
Magnetic Resonance Imaging (fMRI), we have adapted the length of the stimulus
and the number of repetitions to our setup. To record a larger number of
trials and reduce the predictability of the subsequent positions in the
primary sequence of the stimulus, we use different pseudo-randomized orderings
of the dots presentation, distributed in five experimental blocks, as shown in
Figure 3. The entire procedure is repeated 6 times during the measurement,
resulting in 810 stimuli for each participant.

Figure 4: Visual Symbol Search. Three representative trials of the VSS task.
For each row, the first two symbols are target symbols, the next five
correspond to the search group, and the response check boxes are labeled as
response.

#### Visual Symbol Search.

Visual Symbol Search (VSS) is a computerized version of a clinical assessment
to measure processing speed (Symbol Search Subtest of the Wechsler
Intelligence Scale for Children IV (WISC-IV) [43]) and the Wechsler Adult
Intelligence Scale (WAIS-III) [44]). Participants are shown 15 rows at a time,
where each row consists of two _target_ symbols, five _search_ symbols and two
additional symbols that contain respectively the words “YES” and “NO”. For
each row, participants need to indicate by clicking with the mouse button on
the “YES” or “NO” symbol, whether or not one of the two target symbols appears
among the five search symbols. Each recording of the VSS paradigm takes 120
seconds with a maximum of 60 trials, where one trial corresponds to one row;
in $50\%$ of the trials one of the target symbols does appear in the search
symbols and in the remaining $50\%$ none does. Once participants finish a set
of 15 rows, they press a “next page” button which displays a new set of 15
rows. Participants are instructed to solve as many rows, or trials, as
possible within the given 120 seconds. Before beginning the actual recording,
participants perform a training of four trials, for which they receive
feedback to ensure they understand the task. No feedback is provided
throughout the actual recording. Data collected according to this paradigm may
be used for investigating behavioral and neurophysiological correlates of
processing speed.

The EEGEyeNet dataset contains data recorded following all three different
experimental paradigms mentioned above. These paradigms cover typical
cognitive tasks and provide a wealth of eye movement patterns.

Finally, in Table 2 we report for each paradigm the number of appearances of
the three events that we extract: fixations, saccades and blinks; we report
these after both minimal (min) and maximal (max) preprocessing. In Appendix C
we give further details about the characteristics of each of these events.
These numbers make apparent the large size of the EEGEyeNet dataset, with a
total of over 47 hours of recorded events.

Paradigm | Preproc. | # Fixations | # Saccades | # Blinks | Total time
---|---|---|---|---|---
_Pro- Antisac._ | min | 357115 | 358384 | 56179 | 38 h
max | 358587 | 359856 | 57991 | 38 h 6 mins
_Large Grid_ | min | 68075 | 68245 | 11108 | 7 h 52min
max | 69013 | 69185 | 11237 | 7 h 58 min
_VSS_ | min | 43384 | 43443 | 971 | 1 h 29 min
max | 43279 | 43339 | 945 | 1 h 28 min
Total | min | 468574 | 470072 | 68258 | 47 h 21 min
max | 470879 | 472380 | 70173 | 47 h 33 min
Table 2:  Overview of EEGEyeNet Dataset. Number of eye-tracking events
(fixations, saccades, blinks) for the EEGEyeNet experimental paradigms. The
difference in the number of eye events between the two preprocessing versions
is due to fact that in the minimal preprocessing more events are identified as
artifacts and removed from the sample.

##  4 Benchmark

Figure 5:  Each sample of EEG data has a shape of $500\times 128$, i.e., $500$
time points for each of the $128$ EEG channels. On the left side (A) we can
see gaze data (along _XY_ -axes) of the one-second sample. On the right side
(B) we can see a subset of the preprocessed EEG data (electrodes matching the
10–20 systems were chosen).

Based on the EEGEyeNet dataset we propose a benchmark to assess EEG-based eye
tracking methods. This benchmark consists of three different tasks with
increasing difficulty. For each of the tasks, we provide data preparation
modules that cut the data into samples of one second with a temporal
resolution of $2ms$ from all 128 EEG channels; This way all samples have a
shape of $500\times 128$, i.e., $500$ time points for each of the $128$ EEG
channels. This is also illustrated in Figure 5. We use exclusively minimally
preprocessed EEG data as it produces better performance as compared to maximal
preprocessed data. In Appendix E, we report our results for the maximally
preprocessed data, as well as the results of the experiments ran on the Zuco
2.0 dataset [30]. For comparability, we also provide a stable split of the
data across participants, with 70% of the participants for training, 15% for
validation and 15% for test. Note that each participant’s data is contained
only in one of the three sets, i.e., the same participant does not appear
during training and validation or test. In Table 3 we include a summary of how
the data is split in each task of the benchmark.

| # Participants | # Samples
---|---|---
Dataset | Total | Train | Validation | Test | Total | Train | Validation | Test
Left-Right | 329 | 229 | 50 | 50 | 30842 | 21042 | 4980 | 4820
Angle/Amplitude | 27 | 19 | 4 | 4 | 17830 | 12275 | 2836 | 2719
Abs. Position | 27 | 19 | 4 | 4 | 21464 | 14706 | 3277 | 3481

Table 3: Benchmark Data. Detail of the data split for each task of the
proposed benchmark.

###  4.1 Task 1: Left-right

The first and simplest task of our benchmark consists of determining the
direction of the subject’s gaze along the horizontal axis. This task is
performed exclusively on the first experimental paradigm (_Pro- and
Antisaccade_). We make sure that in all the samples provided for this task,
after the cue, the participant performs a saccade towards the correct
direction and then fixates on the cue; otherwise, the sample is not included
in the data. Given that the antisaccade task has been viewed as an indicator
of behavioral inhibition abilities [45] and that here we are interested in
saccade direction, for the sake of simplicity, we use only data from the
prosaccade trials of the antisaccade experimental paradigm. From the $30,842$
samples used in this task, $14,827$ correspond to the “left” label and
$16,015$ to “right”.

Each sample starts at the instant when a cue is shown on the screen and
contains one saccade: the goal is to determine whether it is a left or a right
saccade. This is a binary classification task, and therefore, performance is
measured in terms of accuracy with a random baseline of $52.3\%$, given by the
majority class. Although seemingly simple, determining the horizontal
direction of the human gaze is of paramount importance in multiple real-world
applications, e.g., gaze-based writing systems for people with disabilities
[46].

###  4.2 Task 2: Angle/Amplitude

The second task of our benchmark consists of determining the angle and
amplitude of a saccade and is performed on data from the _Large Grid_
paradigm. The one-second samples in this task contain a saccade onset in the
middle of the sample. This way, each sample contains a complete saccade as
well as a part of the preceding and succeeding fixations. During the data
preparation for this task, we remove samples with fixations that are shorter
than $150ms$ and with saccades with an amplitude of less than $1\degree$ [47].

Given a sample, the task of the model is to regress the two target values,
i.e., angle and amplitude of the relative change of the gaze position during
the saccade. The evaluation metric for this task is the square root of the
mean squared error (RMSE) for the angle (in radians along the shortest path of
the unit circle) and amplitude (in millimeters) separately. The naive baseline
is given by the mean angle and amplitude in the training set and amounts to
$1.90$ RMSE radians for the angle and $74.7$ RMSE mm for the amplitude. This
task is significantly harder than the Left-Right task and aims at serving as
an intermediate step for the development of a purely EEG-based ET. Despite its
difficulty, there is evidence that EEG recordings contain angle information
[48].

###  4.3 Task 3: Absolute Position

Finally, we propose the task of determining the absolute position of the
subject’s gaze in the screen, described in terms of _XY_ -coordinates. Again,
this task is performed on the _Large Grid_ paradigm. We provide samples of one
second during which the participant is performing only one fixation. The data
preparation module ensures that in this time window there is no other event
happening, i.e., the participant performs only a fixation. However, we note
that in order to estimate the current gaze position we expect past
information, e.g., the previous gaze position, to be helpful. For this reason,
we also encourage experimentation on different ways of processing and cutting
the full EEG recordings provided in the dataset.

Performance is measured as the euclidean distance in millimeters between the
actual and the estimated gaze position in the _XY_ -coordinate plane. Random
performance is again calculated as the mean position across the training set
and corresponds to a distance of $123.3$ mm. This is the hardest task in the
proposed benchmark and aims at simulating a purely EEG-based eye-tracker. We
expect this task to help in the development of general methods to further
improve gaze estimation systems as well as in setting an upper bound on the
reach of EEG-based eye-tracking.

##  5 Baselines

We run extensive experiments on the proposed benchmark in order to provide
baselines of different complexity. In our repository we provide an intuitive
interface to reproduce our results and to use the methods presented here as a
starting point for further research. We consider both models based on
classical machine learning as well as large neural networks.

###  5.1 Models

#### Machine Learning.

These models operate on features extracted from the data rather than on the
raw data. Therefore, in a feature extraction step, we apply a band-pass filter
in the alpha band [$8-13$ Hz] on the continuous EEG signals across the entire
trial. The choice of the alpha band is motivated by growing evidence
suggesting that alpha activity is integral to spatial attention, and therefore
plays a central role in covert orienting in a range of paradigms [49]. After
band-passing the signal, the Hilbert transform was applied, resulting in a
complex time series from which we extract phase and amplitude. Using the
resulting features, we experiment with different models. _Left-Right_ is a
classification task while _Angle/Amplitude_ and _Absolute Position_ are
regression tasks and thus, some of the considered models can be applied only
to either of those two types of tasks. In particular, the classification-only
models that we study are Gaussian Naive Bayes (NB) Linear Support Vector
Classification (SVC) and Radial Basis Function (RBF) kernel SVC, whereas the
regression-only models are Linear, Ridge and Lasso Regression as well as
Elastic Net and RBF Support Vector Machine for regression (SVR). Furthermore,
we use K-Nearest Neighbours (KNN) and four tree-based models, Random Forest,
Gradient Boost, AdaBoost and XGBoost, which can be used for both,
classification and regression. In all cases we use the implementation from the
Sklearn library [50], for detailed model hyperparameters see Appendix D.

#### Deep Learning.

We evaluate five state-of-the-art architectures for time series on the
proposed benchmark: a standard one-dimensional convolutional neural network
(CNN), a CNN with pyramidal shape, the EEGNet model by Lawhern et al. [51], an
InceptionTime model [52], and an Xception model [53]. All of these models use
convolutions as the primary operation (see Appendix D for architectural
details). We tune the learning rate and other hyperparameters on the
validation set of the _Left-Right_ task and use those values in all the
reported results (cf. Appendix D). We use binary cross entropy loss to train
the models for _Left-Right_ , and mean square error (MSE) loss for the other
two tasks; in all cases we use the Adam optimizer [54] and early stopping on
the validation sets.

###  5.2 Results

In Table 4 we provide the results of our evaluation for each of the models
considered and for each of the three tasks. We tune the learning rate of the
different deep learning models on the validation set of the _Left-Right_ task
and use the same rate in the other two tasks, i.e., $1e^{-4}$. We run 5 times
each experiment and report mean performance and standard deviation.

| Left-Right | Angle/Amplitude | Abs. Position
---|---|---|---
Model | Accuracy | Angle RMSE | Amp. RMSE | RMSE
KNN | 90.7 $\scriptstyle\pm$0 | 1.26 $\scriptstyle\pm$0 | 59.3 $\scriptstyle\pm$0 | 119.7 $\scriptstyle\pm$0
GaussianNB | 87.7 $\scriptstyle\pm$0 | - | - | -
LinearSVC | 92.0 $\scriptstyle\pm$0 | - | - | -
RBF SVC/SVR | 89.4 $\scriptstyle\pm$0 | 1.88 $\scriptstyle\pm$0 | 75.9 $\scriptstyle\pm$0 | 123 $\scriptstyle\pm$0
Linear Regression | - | 1.39 $\scriptstyle\pm$0 | 64.6 $\scriptstyle\pm$0 | 118.3 $\scriptstyle\pm$0
Ridge Regression | - | 1.39 $\scriptstyle\pm$0 | 64.2 $\scriptstyle\pm$0 | 118.2 $\scriptstyle\pm$0
Lasso Regression | - | 1.38 $\scriptstyle\pm$0 | 63.9 $\scriptstyle\pm$0 | 118 $\scriptstyle\pm$0
Elastic Net | - | 1.38 $\scriptstyle\pm$0 | 63.6 $\scriptstyle\pm$0 | 118.1 $\scriptstyle\pm$0
Random Forest | 96.5 $\scriptstyle\pm$0 | 1.09 $\scriptstyle\pm$0.01 | 59.8 $\scriptstyle\pm$0.1 | 116.7 $\scriptstyle\pm$0.1
Gradient Boost | 97.3 $\scriptstyle\pm$0.1 | 1.11 $\scriptstyle\pm$0.01 | 60 $\scriptstyle\pm$0.1 | 117 $\scriptstyle\pm$0.1
AdaBoost | 96.3 $\scriptstyle\pm$0 | 1.43 $\scriptstyle\pm$0.01 | 65 $\scriptstyle\pm$0.1 | 119.4 $\scriptstyle\pm$0.1
XGBoost | 97.9 $\scriptstyle\pm$0 | 1.11 $\scriptstyle\pm$0 | 61.3 $\scriptstyle\pm$0 | 118 $\scriptstyle\pm$0
CNN | 98.3 $\scriptstyle\pm$0.5 | 0.33 $\scriptstyle\pm$0.05 | 32 $\scriptstyle\pm$3.6 | 70.2 $\scriptstyle\pm$1.1
PyramidalCNN | 98.5 $\scriptstyle\pm$0.2 | 0.34 $\scriptstyle\pm$0.04 | 30.7 $\scriptstyle\pm$1 | 73.6 $\scriptstyle\pm$1.9
EEGNet | 98.6 $\scriptstyle\pm$0.1 | 0.70 $\scriptstyle\pm$0.08 | 46 $\scriptstyle\pm$5.2 | 81.7 $\scriptstyle\pm$1.0
InceptionTime | 97.9 $\scriptstyle\pm$1.1 | 0.44 $\scriptstyle\pm$0.16 | 43.6 $\scriptstyle\pm$21.85 | 70.8 $\scriptstyle\pm$0.8
Xception | 98.8 $\scriptstyle\pm$0.1 | 0.47 $\scriptstyle\pm$0.28 | 32.2 $\scriptstyle\pm$1.9 | 78.7 $\scriptstyle\pm$1.6
Naive Baseline | 52.3 | 1.90 | 74.7 | 123.3
Table 4: Results. Mean and standard deviation of 5 runs of the considered
models on the three benchmark tasks. _Angle_ is measured in radians,
_Amplitude_ and _Abs. Position_ in mm.

#### Left-Right.

We see that classical machine learning models achieve high performance in this
task, much above the naive baseline of $52.26\%$. In particular, tree-based
models reach a performance of over $96\%$, which confirms Left-Right as the
easiest task in the proposed benchmark. Notably, although classical models
obtain high scores, all the deep learning models reach a performance
consistently higher, with an accuracy of over $98\%$ in all cases (except for
InceptionTime, $97.9\%)$. This shows that despite the high performance,
differences in performance can still be noticeable.

#### Angle/Amplitude.

The results in Table 4 clearly show that this task is harder than _Left-
Right_. Except for RBF SVC/SVR, which performs close to random, all classical
machine learning models reach a similar performance for the estimation of both
amplitude and angle. This result is above the naive baseline although not by a
big margin. Tree-based models perform the best among classical statistical
models, however, they are clearly inferior to deep learning models in this
task. Somewhat surprisingly, among the deep learning models the simplest ones,
i.e., CNN and PyramidalCNN perform the best, with an RMSE of $0.33$ and $0.34$
radians in angle estimation, and $32$ and $30.7$ mm in amplitude estimation,
respectively. Overall, we see that there is still a considerable gap between
the best results reported here and ideal performance. Our results constitute a
baseline that should orient future work on estimating angle and amplitude of
saccades from EEG data.

#### Absolute Position.

Finally, we see in the last column of Table 4 the results of our
experimentation on absolute position estimation. We see that classical models
generally fail in this task, with performances very close to the naive
baseline. On the other hand, deep learning models reach an euclidean distance
with respect to the true location in the range of $70$ to $80$ mm. Although
these results are far from ideal performance, the considerable gap with the
naive baseline shows that EEG-based eye tracking can potentially be achieved
to an acceptable degree of accuracy. Our results, set the baseline for this
task in $70.2$ mm, as reached by CNN, again one of the simpler deep learning
models. However, as explained in Section 4.3, exploiting previous information
is likely to improve performance, and thus, it would be interesting to see how
sequence models, such as Recurrent Neural Networks or Transformers would
perform in this task. There is a lot of room for improvement and we hope that
this task will help future work in advancing EEG-based eye tracking.

In summary, our evaluation reveals that deep learning models are superior to
other statistical techniques in estimating gaze position from EEG data.
Although this is not surprising, given the complexity of the task and the
larger expressive capacity of neural networks, it confirms that EEGEyeNet is a
valuable resource for developing large neural models. We expect that future
work will surpass our scores advancing EEG-based eye tracking.

##  6 Discussion and Future Work

The dataset and benchmark presented in this work provide an approachable
framework to conduct research in the intersection of brain activity and eye
movement. We will actively maintain and continuously extend both the dataset
and the benchmark with further measurements from more participants and new
experimental paradigms. We acknowledge as current limitations the relatively
small number of participants recorded for the _Large Grid_ paradigm and the
fact that the test sets of our benchmark are publicly available. We will
address both points by (1) recording more data and (2) building an automatic
evaluation service that keeps the test set hidden from the users and includes
a leaderboard.

To facilitate extensive use of EEGEyeNet for various research purposes, we
provide in our repository data preparation tools with an easy-to-use interface
where users can define their own benchmarking tasks or extract other
information from the dataset. In particular, the user can specify whether some
recording blocks from the experimental paradigm should be ignored, which
events to extract from the data and how the data should be preprocessed.
Additionally, the user can also decide whether feature extraction should be
performed or not. As with the other resources presented in this work, the data
preparation module is in continuous development. We plan to adapt this
software tool in the future according to the users’ needs. Overall, we expect
that EEGEyeNet will become a central resource for a broad range of EEG and
Eye-Tracking related research, specifically:

#### Research in Cognitive Area.

EEGEyeNet’s rich structure and high-density coverage of EEG and Eye-Tracking
data may help advance other areas that study the combination of gaze position
and brain activity to identify variations in attention, arousal and
participant’s compliance with the task demands. Moreover, the behavioral
information gained from eye tracking with the high temporal resolution and
neurophysiological markers provided by EEG enables research of the perceptual,
attentional, cognitive processes.

#### Benchmarking.

We expect the high quality, diversity and large scale of the EEGEyeNet dataset
to be leveraged, in order to include new tasks in the proposed benchmark, as
well as to build benchmarks for related domains. In particular, we plan to
include segmentation tasks that evaluate the ability of a given model to
detect and distinguish events such as fixations or saccades.

##  7 Conclusion

Recording eye-tracking data is a complex and expensive process that requires
specialized hardware, trained operators and participants’ consent. Collecting
such data in combination with EEG adds an additional level of complexity to
the data acquisition process. Therefore, behavioral research studying the
combination of brain activity and eye movements is typically restricted by the
lack of appropriate data. In this work, we have introduced EEGEyeNet, a large
dataset of EEG and eye tracking data, with the view of making basic cognitive
neuroscience research more approachable. Furthermore, given the potential
benefits that EEG-based eye-tracking can bring in different domains, we have
proposed a benchmark to facilitate the development of new methods tackling
this challenge. Our experiments on this benchmark show that deep learning
models perform better than classical statistical models. This confirms that
the amount of data contained in EEGEyeNet is large enough to reliably train
large neural models, which is a promising direction for further developing
EEG-based eye tracking.

## References

  * Eckstein et al. [2017] Maria K Eckstein, Belén Guerra-Carrillo, Alison T Miller Singley, and Silvia A Bunge.  Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development?  _Developmental cognitive neuroscience_ , 25:69–91, 2017\.
  * Rivu et al. [2020] Radiah Rivu, Yasmeen Abdrabou, Ken Pfeuffer, Augusto Esteves, Stefanie Meitner, and Florian Alt.  Stare: Gaze-assisted face-to-face communication in augmented reality.  ETRA ’20 Adjunct. Association for Computing Machinery, 2020.
  * Bergstrom and Schall [2014] Jennifer Romano Bergstrom and Andrew Schall.  _Eye tracking in user experience design_.  Elsevier, 2014.
  * Falck-Ytter et al. [2013] Terje Falck-Ytter, Sven Bölte, and Gustaf Gredebäck.  Eye tracking in early autism research.  _Journal of neurodevelopmental disorders_ , 5(1):1–13, 2013.
  * Bradley et al. [2016] Maria C Bradley, Donncha Hanna, Paul Wilson, Gareth Scott, Paul Quinn, and Kevin FW Dyer.  Obsessive–compulsive symptoms and attentional bias: An eye-tracking methodology.  _Journal of behavior therapy and experimental psychiatry_ , 50:303–308, 2016.
  * Holzman et al. [1973] Philip S Holzman, Leonard R Proctor, and Dominic W Hughes.  Eye-tracking patterns in schizophrenia.  _Science_ , 181(4095):179–181, 1973.
  * Shibasaki et al. [1979] Hiroshi Shibasaki, Sadatoshi Tsuji, and Yoshigoro Kuroiwa.  Oculomotor abnormalities in parkinson’s disease.  _Archives of Neurology_ , 36(6):360–364, 1979\.
  * Hutton et al. [1984] J Thomas Hutton, JA Nagel, and Ruth B Loewenson.  Eye tracking dysfunction in alzheimer-type dementia.  _Neurology_ , 34(1):99–99, 1984.
  * Devi and Bajaj [2008] Mandalapu Sarada Devi and Preeti R Bajaj.  Driver fatigue detection based on eye tracking.  In _2008 First International Conference on Emerging Trends in Engineering and Technology_ , pages 649–652. IEEE, 2008.
  * Abu-Faraj et al. [2006] Ziad O Abu-Faraj, Maya J Mashaalany, Habib C Bou Sleiman, Jean-Louis D Heneine, and Waleed M Al Katergi.  Design and development of a low-cost eye tracking system for the rehabilitation of the completely locked-in patient.  In _2006 International Conference of the IEEE Engineering in Medicine and Biology Society_ , pages 4905–4908. IEEE, 2006.
  * Wedel and Pieters [2008] Michel Wedel and Rik Pieters.  _Eye tracking for visual marketing_.  Now Publishers Inc, 2008.
  * König et al. [2012] Peter König, Michael Plöchl, and Jose Pablo Ossandón.  Combining eeg and eye tracking: Identification, characterization and correction of eye movement artifacts in electroencephalographic data.  _Biomedical Engineering/Biomedizinische Technik_ , 57(SI-1-Track-F):392–392, 2012.
  * Plöchl et al. [2012] Michael Plöchl, José Pablo Ossandón, and Peter König.  Combining eeg and eye tracking: identification, characterization, and correction of eye movement artifacts in electroencephalographic data.  _Frontiers in human neuroscience_ , 6:278, 2012.
  * Ma et al. [2018] Xinyao Ma, Zhaolin Yao, Yijun Wang, Weihua Pei, and Hongda Chen.  Combining brain-computer interface and eye tracking for high-speed text entry in virtual reality.  In _23rd International Conference on Intelligent User Interfaces_ , pages 263–267, 2018.
  * Shishkin et al. [2016] Sergei L Shishkin, Yuri O Nuzhdin, Evgeny P Svirin, Alexander G Trofimov, Anastasia A Fedorova, Bogdan L Kozyrskiy, and Boris M Velichkovsky.  Eeg negativity in fixations used for gaze-based control: Toward converting intentions into actions with an eye-brain-computer interface.  _Frontiers in neuroscience_ , 10:528, 2016.
  * Kastrati et al. [2021] Ard Kastrati, Martyna Beata Plomecka, Roger Wattenhofer, and Nicolas Langer.  Using deep learning to classify saccade direction from brain activity.  In _ACM Symposium on Eye Tracking Research and Applications_ , ETRA ’21 Short Papers. Association for Computing Machinery, 2021.
  * Chaaraoui et al. [2012] Alexandros André Chaaraoui, Pau Climent-Pérez, and Francisco Flórez-Revuelta.  A review on vision techniques applied to human behaviour analysis for ambient-assisted living.  _Expert Systems with Applications_ , 39(12):10873–10888, 2012.
  * Okada et al. [2018] Genki Okada, Kenta Masui, and Norimichi Tsumura.  Advertisement effectiveness estimation based on crowdsourced multimodal affective responses.  In _Proceedings of the IEEE Conference on computer vision and pattern recognition workshops_ , pages 1263–1271, 2018.
  * Majaranta and Bulling [2014] Päivi Majaranta and Andreas Bulling.  Eye tracking and eye-based human–computer interaction.  In _Advances in physiological computing_ , pages 39–65. Springer, 2014.
  * Bulling et al. [2010] Andreas Bulling, Jamie A Ward, Hans Gellersen, and Gerhard Tröster.  Eye movement analysis for activity recognition using electrooculography.  _IEEE transactions on pattern analysis and machine intelligence_ , 33(4):741–753, 2010.
  * Nakashima et al. [2015] Ryoichi Nakashima, Yu Fang, Yasuhiro Hatori, Akinori Hiratani, Kazumichi Matsumiya, Ichiro Kuriki, and Satoshi Shioiri.  Saliency-based gaze prediction based on head direction.  _Vision research_ , 117:59–66, 2015.
  * Koch and Ullman [1987] Christof Koch and Shimon Ullman.  Shifts in selective visual attention: towards the underlying neural circuitry.  In _Matters of intelligence_ , pages 115–141. Springer, 1987.
  * Krafka et al. [2016] Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, and Antonio Torralba.  Eye tracking for everyone.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2176–2184, 2016.
  * Son et al. [2020] Jake Son, Lei Ai, Ryan Lim, Ting Xu, Stanley Colcombe, Alexandre Rosa Franco, Jessica Cloud, Stephen LaConte, Jonathan Lisinski, Arno Klein, et al.  Evaluating fmri-based estimation of eye gaze during naturalistic viewing.  _Cerebral Cortex_ , 30(3):1171–1184, 2020.
  * LaConte et al. [2006] S LaConte, S Peltier, K Heberlein, and X Hu.  Predictive eye estimation regression (peer) for simultaneous eye tracking and fmri.  In _Proc. Intl. Soc. Magn. Reson. Med_ , volume 2808, page 13, 2006\.
  * O’Connell and Chun [2018] Thomas P O’Connell and Marvin M Chun.  Predicting eye movement patterns from fmri responses to natural scenes.  _Nature communications_ , 9(1):1–15, 2018.
  * Zheng and Lu [2017] Wei-Long Zheng and Bao-Liang Lu.  A multimodal approach to estimating vigilance using eeg and forehead eog.  _Journal of neural engineering_ , 14(2):026017, 2017.
  * Hollenstein et al. [2018] Nora Hollenstein, Jonathan Rotsztejn, Marius Troendle, Andreas Pedroni, Ce Zhang, and Nicolas Langer.  Zuco, a simultaneous eeg and eye-tracking resource for natural sentence reading.  _Scientific data_ , 5(1):1–13, 2018.
  * Slanzi et al. [2017] Gino Slanzi, Jorge A Balazs, and Juan D Velásquez.  Combining eye tracking, pupil dilation and eeg analysis for predicting web users click intention.  _Information Fusion_ , 35:51–57, 2017.
  * Hollenstein et al. [2019] Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer.  Zuco 2.0: A dataset of physiological recordings during natural reading and annotation.  _CoRR_ , abs/1912.00903, 2019.  URL http://arxiv.org/abs/1912.00903.
  * Notaro and Diamond [2018] Gina M Notaro and Solomon G Diamond.  Simultaneous eeg, eye-tracking, behavioral, and screen-capture data during online german language learning.  _Data in brief_ , 21:1937–1943, 2018.
  * Nikolopoulos et al. [2017] Spiros Nikolopoulos, Panagiotis C Petrantonakis, Kostas Georgiadis, Fotis Kalaganis, Georgios Liaros, Ioulietta Lazarou, Katerina Adam, Anastasios Papazoglou-Chalikias, Elisavet Chatzilari, Vangelis P Oikonomou, et al.  A multimodal dataset for authoring and editing multimedia content: The mamem project.  _Data in brief_ , 15:1048–1056, 2017.
  * Langer et al. [2017] Nicolas Langer, Erica J Ho, Lindsay M Alexander, Helen Y Xu, Renee K Jozanovic, Simon Henin, Agustin Petroni, Samantha Cohen, Enitan T Marcelle, Lucas C Parra, et al.  A resource for assessing information processing in the developing brain using eeg and eye tracking.  _Scientific data_ , 4(1):1–20, 2017.
  * Association et al. [2013] World Medical Association et al.  World medical association declaration of helsinki: ethical principles for medical research involving human subjects.  _Jama_ , 310(20):2191–2194, 2013.
  * Kappenman and Luck [2010] Emily S Kappenman and Steven J Luck.  The effects of electrode impedance on data quality and statistical significance in erp recordings.  _Psychophysiology_ , 47(5):888–904, 2010.
  * Keil et al. [2014] Andreas Keil, Stefan Debener, Gabriele Gratton, Markus Junghöfer, Emily S Kappenman, Steven J Luck, Phan Luu, Gregory A Miller, and Cindy M Yee.  Committee report: publication guidelines and recommendations for studies using electroencephalography and magnetoencephalography.  _Psychophysiology_ , 51(1):1–21, 2014.
  * Pedroni et al. [2019] Andreas Pedroni, Amirreza Bahreini, and Nicolas Langer.  Automagic: Standardized preprocessing of big eeg data.  _Neuroimage_ , 200:460–473, 2019.
  * Pion-Tonachini et al. [2019] Luca Pion-Tonachini, Ken Kreutz-Delgado, and Scott Makeig.  Iclabel: An automated electroencephalographic independent component classifier, dataset, and website.  _NeuroImage_ , 198:181–197, 2019.
  * Dimigen et al. [2011] Olaf Dimigen, Werner Sommer, Annette Hohlfeld, Arthur M Jacobs, and Reinhold Kliegl.  Coregistration of eye movements and eeg in natural reading: analyses and review.  _Journal of experimental psychology: General_ , 140(4):552, 2011.
  * Toivanen et al. [2015] Miika Toivanen, Kati Pettersson, and Kristian Lukander.  A probabilistic real-time algorithm for detecting blinks, saccades, and fixations from eog data.  _Journal of Eye Movement Research_ , 8(2), 2015.
  * Antoniades et al. [2013] Chrystalina Antoniades, Ulrich Ettinger, Bertrand Gaymard, Iain Gilchrist, Arni Kristjánsson, Christopher Kennard, R John Leigh, Imran Noorani, Pierre Pouget, Nikolaos Smyrnis, et al.  An internationally standardised antisaccade protocol.  _Vision research_ , 84:1–5, 2013.
  * Płomecka et al. [2020] Martyna Beata Płomecka, Zofia Barańczuk-Turska, Christian Pfeiffer, and Nicolas Langer.  Aging effects and test–retest reliability of inhibitory control for saccadic eye movements.  _Eneuro_ , 7(5), 2020.
  * Wechsler and Kodama [1949] David Wechsler and Habuku Kodama.  _Wechsler intelligence scale for children_ , volume 1.  Psychological corporation New York, 1949.
  * Wechsler [1955] David Wechsler.  Wechsler adult intelligence scale–.  _Archives of Clinical Neuropsychology_ , 1955.
  * Klimesch et al. [2007] Wolfgang Klimesch, Paul Sauseng, and Simon Hanslmayr.  Eeg alpha oscillations: the inhibition–timing hypothesis.  _Brain research reviews_ , 53(1):63–88, 2007\.
  * Bates et al. [2007] Richard Bates, Michael Donegan, Howell O Istance, John Paulin Hansen, and K-J Räihä.  Introducing cogain: communication by gaze interaction.  _Universal Access in the Information Society_ , 6(2):159–166, 2007.
  * Martinez-Conde et al. [2013] Susana Martinez-Conde, Jorge Otero-Millan, and Stephen L Macknik.  The impact of microsaccades on vision: towards a unified theory of saccadic function.  _Nature Reviews Neuroscience_ , 14(2):83–96, 2013\.
  * Hládek et al. [2018] L’uboš Hládek, Bernd Porr, and W Owen Brimijoin.  Real-time estimation of horizontal gaze angle by saccade integration using in-ear electrooculography.  _Plos one_ , 13(1):e0190420, 2018.
  * Foster et al. [2017] Joshua J Foster, David W Sutterer, John T Serences, Edward K Vogel, and Edward Awh.  Alpha-band oscillations enable spatially and temporally resolved tracking of covert spatial attention.  _Psychological science_ , 28(7):929–941, 2017\.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Lawhern et al. [2018] Vernon J Lawhern, Amelia J Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance.  Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces.  _Journal of neural engineering_ , 15(5):056013, 2018.
  * Fawaz et al. [2020a] Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean.  Inceptiontime: Finding alexnet for time series classification.  _Data Mining and Knowledge Discovery_ , 34(6):1936–1962, 2020a.
  * Chollet [2017a] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1251–1258, 2017a.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Delorme and Makeig [2004] Arnaud Delorme and Scott Makeig.  Eeglab: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis.  _Journal of neuroscience methods_ , 134(1):9–21, 2004.
  * de Cheveigné [2020] Alain de Cheveigné.  Zapline: A simple and effective method to remove power line artifacts.  _Neuroimage_ , 207:116356, 2020.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR, 2015.
  * Fawaz et al. [2020b] Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean.  Inceptiontime: Finding alexnet for time series classification.  _Data Mining and Knowledge Discovery_ , 34(6):1936–1962, 2020b.
  * Szegedy et al. [2017] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31, 2017.
  * Chollet [2017b] Francois Chollet.  Xception: Deep learning with depthwise separable convolutions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017b.
  * Gebru et al. [2018] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford.  Datasheets for datasets.  _arXiv preprint arXiv:1803.09010_ , 2018.

## Author statement

Hereby we confirm that we bear all responsibility in case of violation of
rights, etc., and confirmation of the data license. All data are de-identified
and participants gave permission for their data to be openly shared as part of
the informed consent process.

Public data are distributed under the the Creative Commons Attribution 4.0
International Public License (https://creativecommons.org/licenses/by/4.0/).

We release our code used for our experiments, data collection and data
preparation at the following repository :
https://github.com/ardkastrati/EEGEyeNet and our dataset at
https://doi.org/10.17605/OSF.IO/KTV7M.

## Acknowledgments

We thank Tzvetan Popov for useful discussions related to this project and
Marta Marciniak for the language editing, and proofreading. Dataset collection
and maintenance is generously supported by the Velux Foundation and Swiss
National Science Foundation.

We thank the participants for taking the time to be in the study, as well as
their willingness to have their data shared with the scientific community.

## Supplementary Material

##  Appendix A Preprocessing

#### EEG Minimal Preprocessing

In this preprocessing strategy first, we identify bad channels using the
EEGLAB [55] clean rawdata algorithm 111 clean_rawdata() from:
http://sccn.ucsd.edu/wiki/Plugin_list_process. This algorithm consists of
three steps: (1) it detects channels that have no signal variation for more
than 5 seconds and filters the EEG of the remaining channels with a forward-
backwards (non-causal) filter, here we use a high-pass filter with a
transition band of [$0.25$, $0.75$] Hz; (2), it searches for channels with a
lower correlation to its robust estimate than a given threshold that we set to
$0.85$; and (3) it removes EEG channels with excessive line noise that in our
case corresponds to $4$ standard deviations. Subsequently, the high Variance
Criterion (HVC) was applied, with a pre- specified threshold of $100\mu V$ as
the upper limit for identifying bad channels. Later, the EEG data were band-
pass filtered at $0.5$ and $40$ Hz and then, Zapline toolkit [56] was applied
to remove line noise artifacts, discarding $7$ power line components. Finally,
all the channels marked as bad were interpolated using spherical spline
interpolation.

#### EEG Maximal Preprocessing.

In addition to the whole minimal preprocessing pipeline, in the Maximal
Preprocessing Pipeline, the Independent component analysis (ICA) is used to
isolate the various source generator processes underlying those recordings.
Non-Brain artifactual source components are removed based on the automatic
classification result as provided by Independent Component Label (ICLabel)
[38]. ICA included Optimizing the ICA-based removal of ocular EEG artifacts
from free viewing experiments (Dimigen 2020) as implemented in Pedroni et al.
[37].

#### Objective Quality Classification

After the bad channel interpolation, all EEG datasets were rated with the
objective quality criteria of the Automagic Toolbox [37]. Any data file rated
as bad, meaning that the proportion of high-amplitude data points in the
signal (> $30\mu V$) was larger than 0.2, more than 20% of the time points
showed a variance larger than $15\mu V$ across channels, 40% of the channels
showed high variance ($15\mu V$), or the ratio of bad channels was higher than
0.4 was not included in the further data processing.

##  Appendix B Annotations

Saccades are detected by the velocity and acceleration of the eye movements.
We used the default system (SR Research, http://www.sr-research.com/)
parameters to define saccades: an acceleration threshold of 8000° per second,
a velocity threshold of 30° per second, and a deflection threshold of 0.1°.

Fixations were defined as time periods without saccades. Fixation might
include small saccades (i.e., microsaccades), which fall below the threshold
for saccade detection. Furthermore, a blink was be regarded as a special case
of a fixation, where the pupil diameter is either zero or outside a
dynamically computed valid pupil, or the horizontal and vertical gaze
positions are zero.

We extracted the following information about the saccades, fixations and
blinks: start and end time, duration, coordinates of start positions on the
computer screen in pixels, and for saccades additionally: end positions, and
amplitudes.

##  Appendix C Events Characteristics

In this appendix we show further details of each dataset. More precisely, for
each experimental paradigm we present the distribution of the fixation
duration, the distribution of the fixation positions, the distribution of the
saccade amplitude (in pixels, where 1 pixel corresponds to 0.5 mm), and the
distribution of the saccade angle.

(a) Distribution of the fixation duration

(b) Distribution of the fixation positions

(c) Distribution of the saccade amplitude

(d) Distribution of the saccade angle

Figure 6: Antisaccade Paradigm

(a) Distribution of the fixation duration

(b) Distribution of the fixation positions

(c) Distribution of the saccade amplitude

(d) Distribution of the saccade angle

Figure 7: Large Grid Paradigm

(a) Distribution of the fixation duration

(b) Distribution of the fixation positions

(c) Distribution of the saccade amplitude

(d) Distribution of the saccade angle

Figure 8: Visual Symbol Search Paradigm

(a) Distribution of the fixation duration

(b) Distribution of the fixation positions

(c) Distribution of the saccade amplitude

(d) Distribution of the saccade angle

Figure 9: ZuCo 2.0 Paradigm

##  Appendix D Details of the Models

In this appendix we present the details of our models used during our
experimentations.

###  D.1 Deep Learning Models

In the following we present five different deep learning architectures that we
evaluate on the EEGEyeNet benchmark.

#### EEGNet.

[51] is a convolutional neural network tailored to Brain-Computer Interfaces
applications. This model first performs a temporal convolution to learn
frequency filters, then a depthwise convolution to learn frequency-specific
spatial filters followed by a separable convolution which should learn a
temporal summary for each individual feature map. Finally, a pointwise
convolution is applied to mix the feature maps. For details on this
architecture we refer the reader to the original work [51], which we follow in
our implementation.

#### Convolutional Neural Network (CNN).

We experiment with a standard one-dimensional convolutional neural network
with $12$ layers and additive residual connections around blocks of three
layers. Each layer consists of (1D-)convolution, batch normalization [57],
ReLU activation and max pooling. In the convolutions we use $16$ filters of
size size $64$, and for the pooling operation a kernel of size $2$ and stride
$1$. Each residual connection performs a convolution followed by batch
normalization.

#### Pyramidal CNN.

We consider a CNN with an inverted pyramidal shape and $6$ layers. Each layer
consists of the same modules as the CNN, with the difference that the number
of filters is a multiple of $16$ that grows with depth, i.e., $16$ in the
first layer, $32$ in the second and so on. In this case, the kernel size is
$16$ and there are not residual connections.

#### InceptionTime

We re-implement the model from [58], which is an adaptation of the
Inception-v4 architecture [59] to time series classification. We instantiate
this model with $12$ layers and skip connections every three layers. Each
layer consists of an InceptionTime module, which takes as input $64$ channels
and performs a $1\times 1$ bottleneck convolution that produces a feature map
with $16$ channels. Then, this feature map is passed through three different
convolution operators with $16$ filters each and kernel sizes of $64$, $32$,
$16$; and a max-pooling operator with a kernel size of $3$. The four resulting
feature maps are concatenated to form the layer’s output, which has $64$
channels.This model also uses residual connections around each block of three
layers.

#### Xception

Finally, we build a model based on the Xception architecture proposed by
Chollet [60]. This model follows the same general structure as our CNN, with
$12$ layers and residual connections after three layers. Each layer contains a
1D depthwise separable convolution [60], with $64$ filters and kernel size of
$40$, followed by batch normalization and ReLU activation.

Model | $Par1$ | $Par2$ | $Par3$ | $Par4$
---|---|---|---|---
KNN | leaf size 10 | 10LR/25AN/50AP | - | -
GaussianNB | var smooth: 0.0004941713 | - | - | -
LinearSVC | C:0.01 | tol: 1e-05 | max iter: 1200 | -
RBF SVC/SVR | C:1 | tol: 1e-05 | max iter: 1200 | gam:0.01
Linear regression | all default | - | - | -
Lasso regression | alpha: 0.01AN/1 | tol: 1e-05 | max iter: 1200 | -
Elastic Net | alpha: 0.1AN/1 | l1 ratio: 0.9LR/0.3AN/0.6AP, | tol: 1e-05 | gam:0.01
Random Forest | max depth 10LR/10AN/50 | est: 50AN/50AM/250 | - | -
AdaBoost | lr: 0.5LR/0.5AN/0.1AM/0.01AP | est: 100LR/50 | - | -
XGBoost | eta: 0.05AN/0.1 | est: 100AP/250 | max depth: 10AP/5 |  -

Table 5: Hyperparameters of classical machine learning models. LR - Left/Right
task, AN - Angle task, AM - Amplitude task, and AP - Absolute Position task.

Parameter | $CNN$ | $PyramidalCNN$ | $EEGNet$ | $InceptionTime$ | $Xception$
---|---|---|---|---|---
Depth | 12 | 6 | 2 | 12 | 18
Number of filters | 16 | 16 | 16/256 | 16 | 64
Kernel size | 64 | 16 | 256 | 64 | 40
Batch size | 64 | 64 | 64 | 64 | 64
Epochs | 50 | 50 | 50 | 50 | 50
Early stopping patience | 20 | 20 | 20 | 20 | 20
Residual connections | True | False | False | True | True
Bottleneck size | - | - | - | 16 | -
Dropout rate | - | - | 0.5 | - | -

Table 6: Hyperparameters of the deep learning models. EEGNet uses two
different amounts of filters in its two blocks.

###  D.2 Hyperparameter Tuning

In Table 5 we report our tuned hyperparameters for the classical machine
learning models and in Table 6 for the deep learning models.

##  Appendix E Further Experiments

###  E.1 Experiments on the Maximally Preprocessed Data

Besides the minimally preprocessed data presented in the benchmark results at
Section 4, we performed the same experiments on the maximally preprocessed
data. As expected, the models perform much worse compared to the ones with
minimally preprocessed data (see Table 7). However, the following results show
that even after removing eye artifacts during preprocessing, it is still
possible to infer gaze direction based on EEG data.

| Left-Right | Angle/Amplitude | Abs. Position
---|---|---|---
Model | Accuracy | Angle RMSE | Amp. RMSE | RMSE
KNN | 60.9 $\scriptstyle\pm$0 | 1.83 $\scriptstyle\pm$0 | 72.2 $\scriptstyle\pm$0 | 123.1 $\scriptstyle\pm$0
GaussianNB | 58.5 $\scriptstyle\pm$0 | - | - | -
LinearSVC | 63.2 $\scriptstyle\pm$0 | - | - | -
RBF SVC/SVR | 53.9 $\scriptstyle\pm$0 | 1.91 $\scriptstyle\pm$0 | 76.1 $\scriptstyle\pm$0 | 122.6 $\scriptstyle\pm$0
Linear Regression | - | 1.76 $\scriptstyle\pm$0 | 72.2 $\scriptstyle\pm$0 | 123.5 $\scriptstyle\pm$0
Ridge Regression | - | 1.79 $\scriptstyle\pm$0 | 71.6 $\scriptstyle\pm$0 | 122.6 $\scriptstyle\pm$0
Lasso Regression | - | 1.77 $\scriptstyle\pm$0 | 71.4 $\scriptstyle\pm$0 | 122.6 $\scriptstyle\pm$0
Elastic Net | - | 1.80 $\scriptstyle\pm$0 | 71.5 $\scriptstyle\pm$0 | 122.5 $\scriptstyle\pm$0
Random Forest | 66.1 $\scriptstyle\pm$0.2 | 1.77 $\scriptstyle\pm$0.01 | 71.1 $\scriptstyle\pm$0 | 123.5 $\scriptstyle\pm$0.2
Gradient Boost | 66.9 $\scriptstyle\pm$0.2 | 1.78 $\scriptstyle\pm$0.01 | 71 $\scriptstyle\pm$0 | 122.4 $\scriptstyle\pm$0
AdaBoost | 65.9 $\scriptstyle\pm$0 | 1.84 $\scriptstyle\pm$0.01 | 72.8 $\scriptstyle\pm$0 | 122.7 $\scriptstyle\pm$0.1
XGBoost | 68.3 $\scriptstyle\pm$0 | 1.75 $\scriptstyle\pm$0 | 70.5 $\scriptstyle\pm$0 | 123.2 $\scriptstyle\pm$0
CNN | 75.5 $\scriptstyle\pm$1.7 | 1.07 $\scriptstyle\pm$0.05 | 76.8 $\scriptstyle\pm$2.3 | 134.6 $\scriptstyle\pm$2.8
PyramidalCNN | 86.6 $\scriptstyle\pm$1.1 | 1.38 $\scriptstyle\pm$0.35 | 76.6 $\scriptstyle\pm$2.2 | 142.1 $\scriptstyle\pm$3.1
EEGNet | 83.6 $\scriptstyle\pm$1.1 | 1.31 $\scriptstyle\pm$0.03 | 67.8$\scriptstyle\pm$2.1 | 119.9 $\scriptstyle\pm$0.8
InceptionTime | 80.4 $\scriptstyle\pm$2.9 | 1.65 $\scriptstyle\pm$0.33 | 70.8 $\scriptstyle\pm$1.7 | 137.2 $\scriptstyle\pm$1.7
Xception | 75.7 $\scriptstyle\pm$1.8 | 1.57 $\scriptstyle\pm$0.29 | 76.3 $\scriptstyle\pm$2.1 | 141.1 $\scriptstyle\pm$1.3
Naive Baseline | 52.3 | 1.90 | 74.7 | 123.3
Table 7: Data maximally preprocessed, 5 runs per DL model, Adam optimizer with
learning rate 1e-4, early stopping patience 20. _Angle_ is measured in
radians, _Amplitude_ and _Abs. Position_ in mm.

#### Left-Right.

We see that classical machine learning models achieve a better performance
than the naive baseline of $52.26\%$. In particular, tree-based models
(RandomForest, GradientBoosting, AdaBoost and XGBoost) reach a performance of
over $68\%$. Although it is not a high performance, it shows that the eye
movement information between left and right can also be inferred from “pure”
brain activity. Furthermore, the deep learning models achieve an even higher
performance of over $86\%$, showing that one can achieve satisfactory results
for this task also from the maximally preprocessed data.

#### Angle/Amplitude.

As in the case for minimally preprocessed data (see Table 4), the results in
Table 7 show that this task is more demanding than Left-Right. All classical
models perform close to random. The deep learning models perform slightly
better, and, interestingly, the simple CNN architecture performs best. This
result is above the naive baseline but not by a big margin. Nonetheless, it
shows that information about the saccade angle can be extracted from maximally
preprocessed data as well. In comparison to the angle, we see that the
amplitude task is strictly more difficult for maximally preprocessed data in
the third column. All models except EEGNet fail at this task, with EEGNet
performing slightly better than the naive baseline. This leaves the open
question to what extent the angle and amplitude of saccades can be estimated
from maximally preprocessed EEG data.

#### Absolute Position.

In the last column of Table 7 we see that all models fail in this task, with
performances very close to the naive baseline. Deep learning models (except
EEGNet) perform even worse than the naive baseline. At this point, it is not
clear whether the absolute position of fixations can be inferred from the
maximally preprocessed EEG data.

###  E.2 Experiments on ZuCo 2.0 Dataset

For comparison, we performed the same experiments on one of these openly
available datasets, namely the ZuCo 2.0. We have used the same pipeline from
our infrastructure. First, we have synchronized the EEG-ET data and extracted
the relevant events for annotation (fixation and saccades). Then we also
performed feature extraction. Afterwards, we used our data preparation tool to
extract the samples for our benchmarking tasks. Finally, we ran our benchmark
with this dataset. Note that we didn’t use the LR-task for this dataset due to
the nature of the ZuCo 2.0 experimental paradigm. Although during reading,
most of the eye movements are towards the right, the dataset contains eye
movements in all other directions (see Figure 9). Other referenced datasets
don’t provide an interface for benchmarking purposes. Again, this is because
there are differences in the recording setup, protocol and paradigms.

| Angle/Amplitude | Abs. Position
---|---|---
Model | Angle RMSE | Amp. RMSE | RMSE
KNN | 1.42 $\scriptstyle\pm$0 | 59.9$\scriptstyle\pm$0 | 72.2 $\scriptstyle\pm$0
RBF SVC/SVR | 1.47 $\scriptstyle\pm$0 | 91.4 $\scriptstyle\pm$0 | 77.5 $\scriptstyle\pm$0
Linear Regression | 1.40 $\scriptstyle\pm$0 | 67.6 $\scriptstyle\pm$0 | 72.2 $\scriptstyle\pm$0
Ridge Regression | 1.41 $\scriptstyle\pm$0 | 67.4 $\scriptstyle\pm$0 | 72 $\scriptstyle\pm$0
Lasso Regression | 1.41 $\scriptstyle\pm$0 | 67.3 $\scriptstyle\pm$0 | 72 $\scriptstyle\pm$0
Elastic Net | 1.42 $\scriptstyle\pm$0 | 67.3 $\scriptstyle\pm$0 | 72 $\scriptstyle\pm$0
Random Forest | 1.42 $\scriptstyle\pm$0.01 | 70.6 $\scriptstyle\pm$0 | 72.9 $\scriptstyle\pm$0.1
Gradient Boost | 1.42 $\scriptstyle\pm$0.01 | 67.4 $\scriptstyle\pm$0 | 61.9 $\scriptstyle\pm$0
AdaBoost | 1.41 $\scriptstyle\pm$0.01 | 69 $\scriptstyle\pm$0 | 75.3 $\scriptstyle\pm$0.2
XGBoost | 1.42 $\scriptstyle\pm$0 | 67.5 $\scriptstyle\pm$0 | 72.1 $\scriptstyle\pm$0
CNN | 1.54 $\scriptstyle\pm$0.4 | 67.1 $\scriptstyle\pm$1.1 | 75.8 $\scriptstyle\pm$0.3
PyramidalCNN | 1.70 $\scriptstyle\pm$0.4 | 61.9 $\scriptstyle\pm$0.6 | 73.8 $\scriptstyle\pm$1.6
EEGNet | 1.35 $\scriptstyle\pm$0.2 | 62.1 $\scriptstyle\pm$1.5 | 72.7 $\scriptstyle\pm$0.1
InceptionTime | 1.81 $\scriptstyle\pm$0.02 | 67.3 $\scriptstyle\pm$1.1 | 80 $\scriptstyle\pm$1.1
Xception | 1.67 $\scriptstyle\pm$0.4 | 65.9 $\scriptstyle\pm$1.9 | 76.2 $\scriptstyle\pm$2.4
Naive Baseline | 1.41 | 67 | 72
Table 8: Results of Angle/Amplitude and Absolute Position tasks on the ZuCo
dataset. Same hyperparameters as for experiments on our dataset. _Angle_ is
measured in radians, _Amplitude_ and _Abs. Position_ in mm.

As expected, the models perform worse compared to the ones with minimally
preprocessed data (see Table 4).

#### Angle/Amplitude.

We can see a clear difference between the results in our dataset (see Table 4)
and the results obtained in ZuCo 2.0 dataset. The models performs worse in the
ZuCo 2.0 dataset. For the angle task, the best model is the EEGNet with RMSE
1.35 radians. It performs better then the naive baseline (RMSE 1.90 radians),
but not by a big margin. In comparison, in our dataset, EEGNet performs much
better with an RMSE of 0.70 radians. In addition, the best performing model
for our dataset is the simple CNN, which has an RMSE of 0.33 radians. We can
also see a clear difference for the amplitude task as well. The best
performing model for ZuCo 2.0 dataset is PyramidalCNN with an RMSE of $61.9$
mm, while in our dataset PyramidalCNN has a better performance with an RMSE of
$30.7$ mm.

#### Absolute Position.

In this task, the results are not directly comparable to our dataset, since
they depend on the distribution of the fixation positions. The difference of
the distributions can be seen in Figure 7 and Figure 9. Furthermore, in Table
4 we can see that the naive baseline for our dataset is 123.3, whereas for the
ZuCo 2.0 dataset (Table 8), it is 72. In other words, in ZuCo 2.0 simply
predicting the mean position leads to a better performance compared to our
dataset, because the fixations are not scattered across all possible positions
on the screen. In contrast, our dataset is specifically tailored for eye
movement prediction. However, we can still see a clear difference in the
performance. In our dataset, there is a considerable gap between the deep
learning models and the naive baseline, whereas in the ZuCo dataset all models
perform close to the baseline. One possible explanation for this (besides the
size and the type of dataset) is that ZuCo dataset uses a different
preprocessing method, whereas our dataset uses minimally preprocessing data
which is more suitable for eye movement prediction.

##  Appendix F Datasheets for Datasets

For the dataset documentation we used the recommended documentation framework
"Datasheets for Datasets" [61].

###  F.1 Motivation

#### For what purpose was the dataset created?

The primary purpose of the project is to advance research that studies the
combination of brain activities and gaze position.

#### Who created the dataset

The dataset was collected by the Methods of Plasticity Research Lab at the
University of Zurich. All persons participating in the data collection are
acknowledged on our website: www.eegeye.net

#### Who funded the creation of the dataset? If there is an associated grant,
please provide the name of the grantor and the grant name and number

This work was supported by the Velux Stiftung Project No. 1126 and by the
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
(SNF) Grant $100014_{1}75875$.

#### Any other comments?

[N/A]

###  F.2 Composition

#### What do the instances that comprise the dataset represent (e.g.,
documents, photos, people, countries)? Are there multiple types of instances
(e.g., movies, users, and ratings; people and interactions between them; nodes
and edges)?

the EEGEyeNet dataset. contains electroencephalography and eye-tracking
recordings following three different experimental paradigms. Together with the
raw data, we release two sets of preprocessed data: minimally and maximally
preprocessed; as well as the preprocessing code.

#### How many instances are there in total (of each type, if appropriate)?

The number of instances is presented in the Table 3 (Benchmark statistic).

#### Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set?

The dataset used in this project was recorded in our laboratory in the context
of a larger project that aims to quantify age effects on eye movement
behaviour and electroencephalography (EEG) recordings of resting-state and
task-related activity. Therefore, for the EEGEyeNet dataset, we have used
three paradigms that can help in the gaze position prediction.

#### What data does each instance consist of? “Raw” data (e.g., unprocessed
text or images) or features?

For each experimental paradigm, as described in the section 3.4, we release
raw data, two sets of preprocessed data: minimally and maximally preprocessed;
as well as the preprocessing code.

#### Is there a label or target associated with each instance? If so, please
provide a description

[Yes] All labels and targets are described in the Chapter 4 .

#### Is any information missing from individual instances? If so, please
provide a description, explaining why this information is missing (e.g.,
because it was unavailable). This does not include intentionally removed
information, but might include, e.g., redacted text.

[N/A]

#### Are relationships between individual instances made explicit (e.g.,
users’ movie ratings, social network links)?

[N/A]

#### Are there recommended data splits (e.g., training,
development/validation, testing)?

[Yes] All recommended splits with rationales behind them are reported in the
chapter 4.

#### Are there any errors, sources of noise, or redundancies in the dataset?

Raw data is often contaminated by artifacts influenced by technical and
environmental factors and the recorded participant’s specific.

####  Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)?

The dataset is self contained, there are no restrictions associated with any
of the external resources that might apply to a future user.

#### Does the dataset contain data that might be considered confidential

[No] The dataset was anonymized.

#### Does the dataset contain data that, if viewed directly, might be
offensive, insulting, threatening, or might otherwise cause anxiety?

[N/A] .

#### Does the dataset relate to people?

[Yes]

#### Does the dataset identify any subpopulations (e.g., by age, gender)?

[Yes] Subpopulations are described in the Subsection 3.1.

#### Is it possible to identify individuals (i.e., one or more natural
persons), either directly or indirectly (i.e., in combination with other data)
from the dataset?

[No] .

#### Does the dataset contain data that might be considered sensitive in any
way

[No]

#### Any other comments?

[N/A]

###  F.3 Collection Process

#### How was the data associated with each instance acquired? Was the data
directly observable (e.g., raw text, movie ratings), reported by subjects
(e.g., survey responses), or indirectly inferred/derived from other data
(e.g., part-of-speech tags, model-based guesses for age or language)?

We recorded ET and EEG brain activity, all procedures and mechanims are
described in the Section 3.

#### What mechanisms or procedures were used to collect the data (e.g.,
hardware apparatus or sensor, manual human curation, software program,
software API)? How were these mechanisms or procedures validated?

All procedures and mechanims are described in the Section 3.

#### If the dataset is a sample from a larger set, what was the sampling
strategy (e.g., deterministic, probabilistic with specific sampling
probabilities)?

[N/A]

#### Who was involved in the data collection process (e.g., students,
crowdworkers, contractors) and how were they compensated (e.g., how much were
crowdworkers paid)?

Research Assistants and PhD Students were involved in the data collection
process. All obtained fixed salary for Research Assistants/PhD Students.

#### Over what timeframe was the data collected?

The collection of data used in the project began in the first quarter of 2018
and lasted until May 15, 2021.

#### Were any ethical review processes conducted (e.g., by an institutional
review board)?

This study was conducted according to the principles expressed in the
Declaration of Helsinki. The study was approved by the Institutional Review
Board of Canton Zurich (BASEC-Nr. 2017-00226).

#### Does the dataset relate to people?

[Yes]

####  Did you collect the data from the individuals in question directly, or
obtain it via third parties or other sources (e.g., websites)?

We collected the data from the individuals in question directly.

#### Were the individuals in question notified about the data collection?

[Yes] All participants gave their written informed consent before
participation in the study, as provided in the Appendix "Formal consent of
Participants".

#### Did the individuals in question consent to the collection and use of
their data?

[Yes]  The template of the formal consent can be found in the Appendix J.

#### If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses?

[Yes] The Participants signed the following agreement:"I decide voluntarily
and can withdraw this decision at any time. If I no longer want to
participate, my data will be irrevocably deleted. I only inform the project
management and do not have to justify this decision". The German version of
the agreement can be found in the Appendix J.

#### Has an analysis of the potential impact of the dataset and its use on
data subjects (e.g., a data protection impact analysis)been conducted?

[No]

#### Any other comments?

[Yes]  Prior to visiting the laboratory, participants completed a 10 min. pre-
screening interview over the phone with a research assistant to confirm their
eligibility and safety to participate in the study. This brief interview
obtains information regarding an individual’s psychiatric history, including
past or present diagnoses and/or treatment, as well as current medications and
any neurological disorders (see H). If a participant demonstrates no
contraindications for EEG (e.g., history of seizures or epilepsy), he or she
is then scheduled for a research study appointment.

###  F.4 Preprocessing/cleaning/labeling

#### Was any preprocessing/cleaning/labeling of the data done (e.g.,
discretization or bucketing, tokenization, part-of-speech tagging, SIFT
feature extraction, removal of instances, processing of missing values)?

[Yes] All preprocessing steps are described in the Dataset section remainder
of the questions in this section.

#### Was the “raw” data saved in addition to the preprocessed/cleaned/labeled
data (e.g., to support unanticipated future uses)

[Yes] Together with the raw data, we release two sets of preprocessed data:
minimally and maximally preprocessed; as well as the preprocessing code. This
way, we give a user the freedom to manipulate raw data while easing the
experimentation barrier by additionally providing ready-to-use clean data.

#### Is the software used to preprocess/clean/label the instances available?
If so, please provide a link or other access point.

[Yes]

#### Any other comments?

[N/A]

###  F.5 Uses

#### Has the dataset been used for any tasks already?

[Yes] A subset of the Antisaccade Paradigm was already used in the publication
[42].

#### Is there a repository that links to any or all papers or systems that use
the dataset?

[No] This repository will be created and maintained in the future.

#### What (other) tasks could the dataset be used for?

The dataset includes experimental paradigms assessing key cognitive functions,
like inhibitory control and processing speed. Additionally, the dataset can be
used for a development of new segmentation algorithms for fixations, saccades,
and blinks.

#### Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses?

[Yes] As described in the preprocessing section of the manuscript, depending
on how the data is processed and whether the eye movements component is
retained, we may receive different results.

#### Are there tasks for which the dataset should not be used?

[No]

#### Any other comments?

[N/A]

###  F.6 Distribution

####  Will the dataset be distributed to third parties outside of the entity
(e.g., company, institution, organization) on behalf of which the dataset was
created?

[No]

#### How will the dataset will be distributed (e.g., tarball on website, API,
GitHub)? Does the dataset have a digital object identifier (DOI)?

The dataset will be made available along with the manuscript on our website.
Raw and preprocessed EEG and eye-tracking data are available online and have
the digital object identifier: DOI 10.17605/OSF.IO/KTV7M

#### When will the dataset be distributed?

The dataset will be distributed along with the manuscript submission.

#### Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)?

The dataset will distributed under the terms of the Creative Commons CC BY
license, which permits unrestricted use, distribution, and reproduction in any
medium, provided the original work is properly cited.

#### Have any third parties imposed IP-based or other restrictions on the data
associated with the instances?

[No]

#### Do any export controls or other regulatory restrictions apply to the
dataset or to individual instances?

[No]

#### Any other comments

[N/A]

###  F.7 Maintenance

#### Who is supporting/hosting/maintaining the dataset?

The dataset will be maintanted by the EEGEyeNet team.

#### How can the owner/curator/manager of the dataset be contacted (e.g.,
email address)?

Email addres<EMAIL_ADDRESS><EMAIL_ADDRESS>
#### Is there an erratum? If so, please provide a link or other access point

[No]

#### Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)?

[Yes] The dataset will be regularly updated, information about each update
will be published in our repository, available on the website www.eegeye.net.

#### If the dataset relates to people, are there applicable limits on the
retention of the data associated with the instances (e.g., were individuals in
question told that their data would be retained for a fixed period of time and
then deleted)?

[No] .

#### Will older versions of the dataset continue to be
supported/hosted/maintained?

[Yes] Yes, all published versions of the dataset will be on our webpage
www.eegeye.net.

#### If others want to extend/augment/build on/contribute to the dataset, is
there a mechanism for them to do so?

[Yes] We release our complete infrastructure and provide a simple and easy-to-
use interface to evaluate new methods.

#### Any other comments?

[N/A]

##  Appendix G Risk categorisation

Our study was classified as "Risk category A" by the Institutional Review
Board of Canton Zurich (BASEC-Nr. 2017-00226). Applied methods of EEG and eye-
tracking are non-invasive and do not pose risk to participants. The involved
tasks do not involve deception and completion of these do not pose any harm to
participants.

##  Appendix H Inclusion and Exclusion criteria

Inclusion criteria for participation in the study were left and right-
handedness, healthy male and female participants, with age between 20 and 80
years old. Before each recording, we assured that the participant did not
suffer from psychiatric symptoms, had no severe neurologic disorders, prior
head injuries, a stroke, a transient circulatory disorder of the brain,
diagnosis of dementia Huntington’s disease, Parkinson’s disease, sensory
and/or motor problems that interfere with computer tasks. Moreover, exclusion
criteria were current use of psychotropic drugs, intake of recreational
synthetic or natural drugs.

##  Appendix I Potential negative societal impacts of the work

In the specific case of our dataset, one risk of EEG-based eye-tracking models
is that they could allow the tracking of the gaze of a subject without their
consent, i.e., if the subject agrees to get their EEG data analyzed but not
their gaze patterns. Furthermore, a major ethical concern is data privacy. A
large-scale dataset such as the one we release in this work could be used in
the future to develop methods capable of inferring sensitive information from
EEG recordings, such as medical conditions. For this reason, we do not release
any privacy-sensitive data. Especially, the anonymity of the participants will
be guaranteed when presenting the data at scientific meetings or publishing
them in scientific journals. Interested third parties can access the data (but
under no circumstances the personal data) via the research repository, solely
for scientific purposes (e.g. replication or further analysis). Individual
participant medical information obtained from this research project is
considered confidential, and disclosure to third parties is prohibited.
Participant confidentiality will be further ensured by utilizing
identification code numbers to correspond to medical information in the
computer files.

##  Appendix J Formal consent of Participants

In the following pages we present the template of the formal consent given to
each participant. Every participant signed the formal consent before taking
part in this study. See EP_ETwithEEG1.pdf See EP_ETwithEEG2pdf.pdf See
EP_ETwithEEG7.pdf