[1]Deborah N. Jakobi

[1,2]Lena A. Jäger

1]Department of Computational Linguistics, University of Zurich,
Andreasstrasse 15, Zurich, 8050, Switzerland

2]Department of Computer Science, University of Potsdam, An der Bahn 2,
Potsdam, 14476, Germany

# PoTeC: A German Naturalistic Eye-tracking-while-reading Corpus

<EMAIL_ADDRESS>Thomas Kern  David R. Reich<EMAIL_ADDRESS>Patrick Haller<EMAIL_ADDRESS><EMAIL_ADDRESS>[  [

###### Abstract

The Potsdam Textbook Corpus (PoTeC) is a naturalistic eye-tracking-while-
reading corpus containing data from 75 participants reading 12 scientific
texts. PoTeC is the first naturalistic eye-tracking-while-reading corpus that
contains eye-movements from domain-experts as well as novices in a within-
participant manipulation: It is based on a 2$\times$2$\times$2 fully-crossed
factorial design which includes the participants’ level of study and the
participants’ discipline of study as between-subject factors and the text
domain as a within-subject factor. The participants’ reading comprehension was
assessed by a series of text comprehension questions and their domain
knowledge was tested by text-independent background questions for each of the
texts. The materials are annotated for a variety of linguistic features at
different levels. We envision PoTeC to be used for a wide range of studies
including but not limited to analyses of expert and non-expert reading
strategies. The corpus and all the accompanying data at all stages of the
preprocessing pipeline and all code used to preprocess the data are made
available via GitHub: https://github.com/DiLi-Lab/PoTeC.

###### keywords:

eye-tracking, reading, German, corpus

##  1 Introduction

Eye-tracking-while-reading data is widely used in different areas of research
including linguistics, cognitive psychology and computer science. In
pyscholinguistic reading research it is considered a gold standard dependent
variable for investigating the cognitive processes involved in language
comprehension [70, 71]. An overwhelming majority of eye-tracking studies in
psycholinguistic research are based on controlled (also referred to as
planned) experiments with hand-crafted stimuli that constitute minimal pairs.
However, as recent research has shown, it is crucial to (also) study language
processing in ecologically valid settings using naturalistic, real-world
stimuli [16, 15, 63].

The stimulus materials of planned experiments, comprising minimal pairs (and
relevant filler items), are specifically designed to test a small number of
predefined, typically theoretically motivated, hypotheses in a controlled
linguistic environment, and thus cover only a small range of linguistic
constructions. For example, the hypothesis that one syntactic construction is
more difficult to process than another one can be investigated by designing
minimal pair stimuli that contain the experimental manipulation and certain
words or constituents, where the difference in processing difficulty is
expected. This experimental approach is crucial to make statements about
specific phenomena which are often highly infrequent in naturally occurring
text. Studying infrequent, complex phenomena can be theoretically relevant as
it often allows for teasing apart competing accounts that in many cases make
very similar predictions for frequently occurring, simple constructions. Only
such a controlled experimental approach allows for drawing conclusions about
the causality of observed effects — although one needs to keep in mind that
linguistic manipulations are almost always only quasi-experimental in nature.

Conversely, naturalistic reading corpora, where participants read naturally
occurring text (e.g., newspaper articles) rather than minimal pair stimuli,
cover a wider range of linguistic constructions. This allows for studying a
broad and representative range of phenomena without the need to design new
experiments and collect new data. The phenomena can not only be studied within
a single dataset but also within-subject, which is not possible in planned
experiments where typically one study is limited to a single phenomenon. The
coverage of existing theories can be evaluated on these datasets and the
theories can be modified accordingly. Additionally, exploratory data analyses
can lead to new theories or inspire new research questions [28]. While planned
experiments using minimal pair stimuli are essential for closely examining and
comparing different theories and testing hypotheses, naturalistic corpora
permit observational studies in ecologically valid settings, enhancing
generalizability, and inspiring new theories based on exploratory data
analyses.

Besides the broad categorization into planned experiments with minimal pair
stimuli and naturalistic reading corpora, there are more fine-grained
differences within naturalistic eye movement corpora. One crucial dimension
for characterizing naturalistic eye movement data revolves around the nature
of the stimuli which spans from entirely naturalistic to partially
constructed, where, for instance, specific target words within the stimulus
sentences are hand-picked based on their lexical frequency. Seminal work on
eye-tracking-while-reading corpora, such as the work by Kliegl . [44], has
adopted this approach by using partially constructed stimuli which enables the
inclusion of a broad range of linguistic phenomena that are highly infrequent
in natural text (but relevant test cases for linguistic theories) while still
utilizing sentences that maintain a higher degree of naturalness than typical
planned experiments. Other naturalistic reading corpora leverage naturally
occurring text as stimuli which are not specifically designed for the
experiment but simply being reused from an existing source. Often, these texts
span multiple sentences which allows for analyses that go beyond the sentence-
level. In the most stringent cases, alterations of the texts are entirely
avoided. However, in many cases, texts need to be edited to, for example,
compensate for missing context or exclude figures and tables. Leveraging texts
that closely resemble or are identical to real-world reading material enables
to study cognitive processes involved in everyday reading at both sentence and
text-level, while, as a natural consequence, specific phenomenon (e.g., long-
distance dependencies) might occur rarely or never within an entire dataset.

In addition to psycholinguistic hypothesis testing, eye-tracking-while-reading
data has become increasingly relevant for other areas of research such as
Natural Language Processing (NLP). NLP research has been leveraging eye
movements in reading for a wide range of tasks such as sentiment analysis [52,
61], named entity recognition [35], part-of-speech tagging [4] or generating
image captions [79] among other tasks. In recent NLP research, eye-tracking-
while-reading data has been used to analyze computational language models, for
example by investigating their cognitive plausibility [6, 38]. Sood . [75]
analyzed attention weights learned by transformer language models and compared
those to human attention implicitly encoded in human gaze data. More than
that, cognitive signals can be used to improve language models [30].
Naturalistic reading data can be used to cognitively enhance and augment
language models with human gaze data [90, 20, 68]. For all of these research
objectives, it is crucial to have large and diverse amounts of cognitive data
available. Many NLP tasks rely on long text passages and thus require
cognitive data for entire paragraphs of natural texts rather than just single
sentences.

In this work, we present the Potsdam Textbook Corpus (PoTeC), a naturalistic
eye-tracking-while-reading corpus containing eye-tracking data of German
native speakers reading German textbook passages from two different domains.
PoTeC is the first corpus to include the level of expertise of each
participant as a within-subject variable and thus allows for analyzing reading
strategies used by experts and non-experts. The naturalistic experimental
setting of PoTeC encourages various kinds of analyses which are not restricted
to test one specific hypothesis and thus has valuable properties that
complement other existing corpora.

##### New standard for data publication

In addition to the publication of the eye-tracking data, our aim is to foster
transparency and re-usability of the data and facilitate leveraging PoTeC for
a variety of different use cases (e.g., psycholinguistic research, NLP
research, development of new preprocessing algorithms, eye-movement-based
biometrics, and many more). In order to achieve this goal, we are presenting a
new standard to publish eye-tracking datasets. Following the FAIR principles
(Findable, Accessible, Interoperable, Reusable), our data is shared in the
following way:

  * •

The data is shared via two channels suitable for the respective data type
(large data files and code; see Section 3.6).

  * •

The eye-tracking data is made available at all stages (e.g., raw sample data,
reading measures data, etc.; see Table 16).

  * •

All the code that was used for preprocessing and analyzing the data is made
publicly available in reproducible formats.

  * •

All data and code is accompanied by extensive documentation that not only
makes the process transparent and reproducible, but also makes it maximally
easy for users with different backgrounds to reuse the scripts for their own
purposes.

In addition to following the general FAIR principles, there are two very
specific implementations of these principles: As a unique feature of PoTeC,
the eye-tracking fixation data is made available in both the original version
as it was collected and as a version where the horizontal calibration drift
was manually corrected post-hoc. The availability of this data not only makes
the process transparent but also allows to study and compare the original data
with the manually corrected data which enables the development of new tools,
including machine-learning-based methods, to automatically detect and/or
correct horizontal drift. The data is furthermore integrated into the open-
source Python package
pymovements111https://pymovements.readthedocs.io/en/stable/. The package
enables to build easy machine learning and psycholinguistic pipelines for the
processing of eye movement data and can be used in Python and R, which
increases re-usability [46].

##  2 Related Work

###  2.1 Naturalistic text passage corpora for German

Reading completely naturalistic text passages consisting of multiple sentences
shown at the same time as in PoTeC exist for several languages, however, only
very few for German (see Table 1). A very recently created corpus is the
Multilingual Eye-Movements Corpus (MECO-L1) which is a 13-language reading
corpus. The German subset contains data from 45 participants reading 12
encyclopedic texts on various topics specifically chosen not to require an
academic background [74]. The PopSci Corpus includes 17 participants reading
16 popular science texts from different sources in German [86]. WebQAmGaze is
another natural reading dataset which includes a German subset [72]. However,
the data was recorded using a webcam which results in substantially lower data
quality compared to a high-precision eye tracker typically used in eye-
tracking-while-reading experiments, and is therefore limited to a small subset
of use cases compared to other reading corpora.

Table 1: Naturalistic eye-tracking-while-reading corpora for German text passages Stimuli |  Participants |  Additional characteristics
---|---|---
PoTeC (present study)
12 texts from undergraduate textbooks on physics or biology ($\scriptstyle\\#\textstyle w$: 1,896, $\overline{\scriptstyle\\#\textstyle w}$: 158) |  75 graduate or undergraduate biology or physics students (native speakers)
($\bar{a}$: $24.2$ ($4.2$)) |  The corpus is designed to specifically enable comparison of expert and non-expert reading.
Eye-tracker: EyeLink 1000
Sampl. Freq.: 1,000 Hz
MECO-L1 [74]
12 Wikipedia-style texts on various topics not requiring an academic
background
($\scriptstyle\\#\textstyle w$: 2,028, $\overline{\scriptstyle\\#\textstyle w}$: 169) |  45 native speakers
($\bar{a}$: 23.76) |  Data is available for 12 other languages (nl, en, el, de, he, it, ru, es, tr, ko, no, et and fi) for a total of 535 participants reading text in their native language.
Eye-tracker: EyeLink Portable Duo, 1000 or 1000+
Sampl. Freq.: 1,000 Hz
PopSci Corpus [86]
16 popular science texts on natural and applied science ($\scriptstyle\\#\textstyle w$: 20,000, $\overline{\scriptstyle\\#\textstyle w}$: 1,250) |  17 participants |  Eye-tracker: EyeLink 1000
Sampl. Freq.: 1,000 Hz
WebQAmGaze [72]
38 Wikipedia-style texts: 2 long texts ($\scriptstyle\\#\textstyle w$: 370, $\overline{\scriptstyle\\#\textstyle w}$: 185) and 36 shorter texts ($\scriptstyle\\#\textstyle w$: 3,010, $\overline{\scriptstyle\\#\textstyle w}$: 83.6) with questions and annotated answer spans |  19 native speakers |  The data can be used to study reading during question answering, it was collected using a Webcam, and is available in two other languages (en, es).
Eye-tracker: Webcam
Sampl. Freq. (mean): 24.93 Hz
|  |

###  2.2 Naturalistic text passage corpora for languages other than German

The vast majority of eye-tracking-while-reading datasets exists for English
stimulus texts (see Tables 2 and 3). The Provo Corpus contains data collected
from 84 participants reading 55 short texts from various sources and includes
human predictability norms [53]. Yaneva [89] collected a dataset containing
data from 108 participants of which some are diagnosed with Autism Spectrum
Disorder (ASD) and some are part of a control group, and each participant is
reading a subset of in total 27 text. For the GazeBase reading task, a total
of 322 participants read up to 18 passages of a poem over a three year period
[27]. Malmaud . [56] collected eye-tracking data for the OneStopQA dataset
which is designed to study reading comprehension. They collected data from 296
participants each reading 10 different articles out of 30 articles with the
questions presented either before or after reading each text. A corpus
constructed to study parafoveal viewing by placing target words inside or
outside of the parafoveal view contains 48 participants reading 40 text
passages [67].

There exist datasets that were specifically designed to automatically assess
reader or text properties or use the data for NLP purposes. SB-SAT is a
95-participant dataset for which participants were asked to judge the
subjective difficulty of 4 passages taken from practice tests for the
Scholastic Assessment Test (SAT) [3]. The MAQ-RC corpus contains 23
participants reading 32 movie plots and answering comprehension questions
which were also answered by computational models and thus allows for comparing
human and machine reading comprehension [75]. The CFILT coreference dataset
contains 14 participants (including 2 linguistic experts) reading 22 texts and
at same time the participants were asked to annotate coreferences in the texts
[11]. A similar dataset is the CFILT scanpaths dataset that contains data from
16 participants (including 3 linguistic experts) reading 32 paragraphs from
(simple) Wikipedia articles. The participants annotated the texts for the
effort to read them such that scanpath complexity can be studied, as more
effort to read presumably results is more complex scanpaths [62]. The CFILT
text quality dataset includes data from 20 fluent English speakers reading 30
texts from different sources which were asked to rate the text quality given
three properties (organization, coherence and cohesion) [57]. The CFILT essay
grading dataset includes 8 fluent English speakers reading and grading 48
essays while their eye movements were being tracked [58].

Table 2: Naturalistic eye-tracking-while-reading corpora for text passages in English (part I) Stimuli |  Participants |  Additional characteristics
---|---|---
SB-SAT [3]
4 SAT passages taken from practice tests for reading comprehension |  95 undergraduate students |  Subjective difficulty ratings
Eye-tracker: EyeLink 1000
Sampl. Freq.: 1,000 Hz
Passage reading [67]
40 text passages constructed around target words which will either appear in parafovea or not |  48 native speakers
($\bar{a}$: 26.48 (14.83)) |  Constructed specifically to study parafoveal viewing.
Eye-tracker: EyeLink 1000
Sampl. Freq.: 1,000 Hz
Provo Corpus [53]
55 short texts from various different sources ($\scriptstyle\\#\textstyle w$: 2,750, $\overline{\scriptstyle\\#\textstyle w}$: 50) |  84 native speakers |  Includes human predictability norms
Eye-tracker: EyeLink 1000+
Sampl. Freq.: 1,000 Hz
MQA-RC [75]
32 movie plots with comprehension questions including results from computational models answering the questions (200–250 words) |  23 native speakers |  The participants had to answer comprehension questions in different settings (e.g. with/without plot).
Eye-tracker: Tobii
Sampl. Freq.: 600 Hz
ASD Data [89]
27 texts from various sources ($\scriptstyle\\#\textstyle w$: 4,212, $\overline{\scriptstyle\\#\textstyle w}$: 156 (49.94)) |  108 native speakers, 56 diagnosed with ASD and a control group ($\bar{a}$: 33.73 (8.36)) |  Groups of participants read disjoint subsets of the texts.
Eye-tracker: Gazepoint GP3
Sampl. Freq.: 60 Hz
GazeBase - Reading Task [27]
18 passages from the same poem with a maximum of 60s to read two texts |  322 participants, not all of them read all passages |  Reading was spread over 3 years
Eye-tracker: EyeLink 1000
Sampl. Freq.: 1,000 Hz
|  |
Table 3: Naturalistic eye-tracking-while-reading corpora for text passages in English (part II) Stimuli |  Participants |  Additional characteristics
---|---|---
OneStopQA Eye-Tracking [56]
30 Guardian articles, each in an advanced difficulty version ($\scriptstyle\\#\textstyle w$: 3,858, $\overline{\scriptstyle\\#\textstyle w}$: 128.6) and an elementary version ($\scriptstyle\\#\textstyle w$: 3,369, $\overline{\scriptstyle\\#\textstyle w}$: 112.3) |  269 participants, each participant read 10 articles |  One question per text was shown prior to reading or after reading the text
Eye-tracker: EyeLink 1000+
Sampl. Freq.: 1,000 Hz
CFILT - Coreference [11]
22 texts (less than 10 sentences each text) |  14 non-native participants:
2 are expert linguists (age: 47–50), 12 (post-)graduates (age: 20–30) |  Participants annotated coreferences in the text.
Eye-tracker: EyeLink 1000+
Sampl. Freq.: 500 Hz
CFILT - Scanpath [62]
32 paragraphs from (simple) Wikipedia (50–200 words each) |  16 non-native participants: 3 are expert linguists (age: 47–50), 13 (post-)graduates (age: 20–30) |  Participants annotated texts for the effort to read them.
Eye-tracker: EyeLink 1000+
Sampl. Freq.: –
CFILT - Essay Grading [58]
48 essays (max. words per essay: 250) |  8 fluent English speakers ($\bar{a}$: 25) |  Participants graded the essays after reading.
Eye-tracker: EyeLink 1000
Sampl. Freq.: 500 Hz
CFILT - Text Quality [57]
30 texts from different sources (ca. 200 words in each text) |  20 fluent English speakers (age: 20–25) |  Participants annotated the text quality based on three given properties (organization, coherence and cohesion).
Eye-tracker: EyeLink 1000
Sampl. Freq.: 500 Hz
MECO-L2 [47]
12 encyclopedic texts originally designed for English language testing ($\scriptstyle\\#\textstyle w$: 1,653, $\overline{\scriptstyle\\#\textstyle w}$: 137.8 (25.3)) |  543 non-native participants ($\bar{a}$: 23.4),
L1: nl, en, el, de, he, it, ru, es, tr, no, et, fi (ISO-639-1) |  Eye-tracker: EyeLink Portable Duo, 1000 or 1000+
Sampl. Freq.: 1,000 Hz
|  |

Probably the earliest eye-tracking-while-reading corpus for text passages in
English and French is the Dundee corpus containing data from 10 English and 10
French native speakers reading newspaper extracts in their native language
[39, 40]. MECO-L2 includes 543 non-native speakers of 12 different L1
backgrounds (nl, en, el, de, he, it, ru, es, tr, no, et and fi) reading 12
English texts which is the largest existing corpus of this type [47]. Another
corpus containing L2 data is GECO which contains data from 14 English
monolinguals reading an entire novel in English, and 19 bilinguals (Dutch: L1,
English: L2) reading half of of the novel in Dutch and the other half in
English [13]. For the same stimulus text, data was also collected from 30
Chinese native speakers reading half of the novel in Chinese and the other
half in English [78, GECO-CN]. To study gaze behaviour for summarization
tasks, 50 participants’ eye movements were tracked while reading 100 Chinese
articles with the specific task to summarize the articles after reading [91].
For another corpus in Chinese, 29 participants were instructed to judge the
relevance of a given document for a given query for a total of 60 document-
query pairs [51]. The first corpus to study eye movements while reading text
passages in Danish is CopCo [31]. The dataset contains data from 22 native, 19
dyslexic and 10 Danish L2 speakers reading 20 speech manuscripts in Danish.
RastrOS contains eye movement data from 37 participants reading 50 paragraphs
from different sources in Portugese and includes human predictability norms
[49]. The Mental Simulation Corpus includes data from 102 participants reading
3 literary short stories in Dutch and were subsequently asked to answer
questions that allow for studying their mental simulation during reading [54].

Table 4: Naturalistic eye-tracking-while-reading corpora on text passages in different languages (except German and English), or on multilingual text passage corpora L |  Stimuli |  Participants |  Additional characteristics
---|---|---|---
GECO [13]
en, nl |  Complete novel which is easy to read in English and Dutch ($\scriptstyle\\#\textstyle w$ en: 54,364, $\scriptstyle\\#\textstyle w$ nl: 59,716) |  33 participants: 19 bilinguals (L1: nl, L2: en) ($\bar{a}$: 21.2 (2.2)) and 14 monolinguals (en) ($\bar{a}$: 21.8 (5.6)) |  Bilinguals read one half in Dutch, the other in English.
Eye-tracker: EyeLink 1000
Sampl. Freq.: 1,000 Hz
GECO-CN [78]
zh, en |  Complete novel in Chinese and English ($\scriptstyle\\#\textstyle w$ en: 54,364, $\scriptstyle\\#\textstyle w$ zh: 59,403) |  30 bilinguals (L1: zh, L2: en) ($\bar{a}$: 25.3 (2.6)) |  Participants read one half in Chinese, the other in English.
Eye-tracker: EyeLink 1000+
Sampl. Freq.: 1,000 Hz
Gaze Behavior in Text Summarization [91]
zh |  100 articles from public news websites each belonging to one out of ten categories ($\overline{\scriptstyle\\#\textstyle c}$: 502) |  50 participants ($\bar{a}$: 23.1 (1.1)) |  Participants were asked to summarize each text after reading.
Eye-tracker: Tobii EyeTracking 4C
Sampl. Freq.: 100 Hz
CopCo [31]
da |  20 speech manuscripts ($\scriptstyle\\#\textstyle w$: 34,897, $\overline{\scriptstyle\\#\textstyle w}$: 1,745) |  51 participants: 22 native speakers, 19 dislexic native speakers and 10 L2 speakers (age: 21–62) |  Eye-tracker: EyeLink 1000+
Sampl. Freq.: 1,000 Hz
Reading Attention [51]
zh |  60 documents, each belongs to one out of 15 search queries |  29 native speakers (age: 17–28)
each participant read 15 documents (one for each query) |  The participants were asked to judge the relevance of the document for the query.
Eye-tracker: Tobii X2-30
Sampl. Freq.: – Hz
Mental Simulation Corpus [54]
nl |  3 literary short stories ($\scriptstyle\\#\textstyle w$: 10,800, $\overline{\scriptstyle\\#\textstyle w}$: 2600) |  102 participants ($\bar{a}$: 23) |  Participants answered questions that allow for studying their mental simulation during reading.
Eye-tracker: EyeLink 1000+
Sampl. Freq.: 500 Hz
Dundee [39, 40]
en, fr |  Extracts of newspaper articles in French and English |  20 participants, 10 each language |  Participants read the texts written in their native language.
Eye-tracker: Dr Bouis Oculometer Eyetracker
Sampl. Freq.: 1,000 Hz
RastrOS [49, 83]
pt |  50 paragraphs from journalistic, literary and popular science texts ($\overline{\scriptstyle\\#\textstyle w}$: 2,494, $\overline{\scriptstyle\\#\textstyle w}$: 49 (7.47)) |  37 native speakers ($\bar{a}$: 22.2 (4.7)) |  Includes human predictability norms.
Eye-tracker: EyeLink 1000
Sampl. Freq.: 1,000 Hz
|  |  |

###  2.3 Single-sentence corpora with partially constructed stimuli for
different languages

Naturalistic single-sentence-experiments have the advantage that the resulting
data is easier to analyze as, for example, it does not necessarily involve
multiple lines or even multiple pages per stimulus item. On the other hand,
their use cases are limited to analyses of reading patterns at the sentence
level. Many single-sentence naturalistic reading corpora leverage partially or
entirely constructed sentences, however, the stimuli are often specifically
constructed to contain a wide range of constructions. The Potsdam Sentence
Corpus (PSC) is such a corpus which includes 33 young and 32 older adults
reading 144 individual German sentences [44, 43]. The sentences were
constructed to cover a wide range of predefined linguistic constructions by
selecting specific target words. There are more such corpora in different
languages that follow the same or a similar procedure to create the stimuli
based on a target word: The Russian Sentence Corpus (RSC) contains data from
96 participants reading 144 sentences [48]. [92] collected a Chinese dataset
used to study Chinese word length effects based on data from 30 participants
reading 90 sentences. The Potsdam-Allahabad Hindi Eyetracking Corpus contains
data from 30 participants reading 153 sentences in Hindi and Urdu [36]. TURead
is a Turkish reading corpus including data from 196 participants reading 192
stimulus texts each consisting of 1 to 3 sentences [2]. See Table 6 for an
overview.

###  2.4 Naturalistic single-sentence corpora for different languages

Single-sentence corpora with more naturalistic stimuli exist for several
languages (see Table 5 and Table 6). For the UCL Corpus, 205 sentences were
sampled from English novels, and eye-tracking data is available for 43
participants. A very recent corpus is RaCCooNS [22] which includes data from
37 participants reading 200 narrative sentences. The largest (in terms of
participants) single-sentence corpus with completely naturalistic stimuli is
the Corpus of Eye Movements in L1 and L2 English Reading (CELER) which is a
365-participant corpus with data from 69 native and 296 non-native speakers
reading 156 English sentences from the Wall Street Journal. The non-native
speakers have five different native languages (Japanese, Arabic, Spanish,
Portuguese and Chinese) and half of the sentences in the stimulus corpus are
uniquely read by one participant. The Hong Kong Corpus of Chinese Sentence and
Passage Reading includes 96 participants reading 300 single-line sentences and
7 multi-line passages in Chinese [88]. Zhang . [93] collected data from 1,718
participants reading in total 7,577 different Chinese sentences. The Beijing
Sentence Corpus (BSC) is a 60-participant Chinese reading corpus with 150
stimulus sentences chosen from the People’s Daily where strong political tones
had been removed [66]. For the CFILT sarcasm dataset, 7 non-native speakers of
English read 1,000 sentences of which 350 had been previously labeled as
sarcastic and 650 as non-sarcastic [60]. Participants had to rate the
sentences as either positive or negative while their eye movements were
tracked. A similar dataset is the CFILT sentiment complexity dataset where 5
participants were asked to rate the sentiment (positive, negative or
objective) of 1,059 English sentences [37].

Table 5: Naturalistic eye-tracking-while-reading corpora for single sentences in English Stimuli |  Participants |  Additional characteristics
---|---|---
CELER [7]
156 sentences from the Wall Street Journal ($\scriptstyle\\#\textstyle w$: 900, $\overline{\scriptstyle\\#\textstyle w}$: 11.3) |  365 participants: 69 English L1 and 296 English L2 ($\bar{a}$: 27.3 (6.8)) |  The L2 readers have five different L2 backgrounds and half of the stimuli are uniquely read by one participant.
Eye-tracker: EyeLink 1000+
Sampl. Freq.: 1,000 Hz
CFILT Sarcasm [60]
1000 sentences, 350 are labeled as sarcastic and 650 as non-sarcastic |  7 non-native English speakers |  Participants were asked to label the sentences as either positive or negative.
Eye-tracker: EyeLink 1000
Sampl. Freq.: 500 Hz
CFILT Sentiment [37]
1059 movie reviews or twitter posts |  5 participants |  Participants were asked to label the sentiment of the sentences as either positive, negative or objective.
Eye-tracker: Tobii TX 300
Sampl. Freq.: 300 Hz
UCL Corpus [23]
205 sentences selected from free online novels ($\scriptstyle\\#\textstyle w$: 2,399, $\overline{\scriptstyle\\#\textstyle w}$: 13.7 (6.36)) |  43 native speakers ($\bar{a}$: 25.8) |  The dataset contains self-paced-reading data as well.
Eye-tracker: EyeLink II
Sampl. Freq.: 500 Hz
ZuCo 1 [33]
1107 Wikipedia sentences and movie reviews ($\overline{\scriptstyle\\#\textstyle w}$: 19.54 (9.72)) |  12 native speakers
($\bar{a}$: 38 (9.8)) |  Co-registration of EEG data and each sentence was part of a block with a specific task.
Eye-tracker: EyeLink 1000+
Sampl. Freq.: 500 Hz
ZuCo 2 [34]
739 Wikipedia sentences ($\scriptstyle\\#\textstyle w$: 15,138, $\overline{\scriptstyle\\#\textstyle w}$: 20.5 (9.2)) |  18 native speakers
($\bar{a}$: 34 (8.3)) |  Co-registration of EEG data and each sentence was part of a block with a specific task.
Eye-tracker: EyeLink 1000+
Sampl. Freq.: 500 Hz
|  |
Table 6: Naturalistic eye-tracking-while-reading corpora for single sentences in different languages other than English L |  Stimuli |  Participants |  Additional characteristics
---|---|---|---
Beijing Sentence Corpus [66]
zh |  150 from the People’s Daily newspaper ($\overline{\scriptstyle\\#\textstyle w}$: 11.2 (1.6)) |  60 native speakers ($\bar{a}$: 22.0 (2.6)) |  The dataset includes human predictability norms.
Eye-tracker: EyeLink II
Sampl. Freq.: 500 Hz
Potsdam Sentence Corpus [43, 44]
de |  144 sentences constructed around target words ($\overline{\scriptstyle\\#\textstyle w}$: 7.9) |  222 native speakers (age: 16–84) |  The dataset includes human predictability norms.
Eye-tracker: EyeLink I / EyeLink II
Sampl. Freq.: 250 / 500 Hz
Potsdam-Allahabad Hindi Eyetracking Corpus [36]
hi, ur |  153 sentences from the Hindi-Urdu treebank ($\scriptstyle\\#\textstyle w$: 2,610, $\overline{\scriptstyle\\#\textstyle w}$: 17.0) |  30 participants |  The same sentences were read in Hindi and Urdu script in two separate sessions.
Eye-tracker: SMI iView X HED
Sampl. Freq.: 500 Hz
Chinese Word Length Effect [92]
zh |  90 constructed sentences ($\overline{\scriptstyle\\#\textstyle c}$: 19.0 (2.0)) |  30 native speakers ($\bar{a}$: 24.0 (2.0)) |  Sentences are rated for their naturalness; includes human predictability norms
Eye-tracker: EyeLink 1000
Sampl. Freq.: – Hz
Russian Sentence Corpus [48]
ru |  144 selected sentences based on target words from the Russian National Corpus ($\scriptstyle\\#\textstyle w$: 1,362) |  96 native speakers ($\bar{a}$: 24.0) |  The dataset includes human predictability norms.
Eye-tracker: EyeLink 1000+
Sampl. Freq.: 1,000 Hz
RaCCooNS [22]
nl |  200 narrative sentences ($\scriptstyle\\#\textstyle w$: 2,783) |  37 native speakers ($\bar{a}$: 26.2) |  Co-registration of EEG data.
Eye-tracker: EyeLink 1000+
Sampl. Freq.: 1,000 Hz
Hong Kong Corpus of Chinese Sentence and Passage Reading [88]
zh |  300 single-line sentences ($\scriptstyle\\#\textstyle w$: 5,250) and 7 multi-line passages ($\scriptstyle\\#\textstyle w$: 4,967) from newspaper articles |  96 native speakers ($\bar{a}$: 26 (3.64)) |  Eye-tracker: EyeLink 1000
Sampl. Freq.: 1,000 Hz
Eye-movement Measures on Words in Chinese Reading [93]
zh |  7,577 sentences ($\scriptstyle\\#\textstyle w$: 8,551, $\overline{\scriptstyle\\#\textstyle c}$: 22.48) |  1,718 native speakers |  Eye-tracker: EyeLink 1000
Sampl. Freq.: 1,000 Hz
TURead [2]
tr |  192 sentences selected from existing Turkish corpora based on target words ($\overline{\scriptstyle\\#\textstyle w}$: 15.3 (2.9)); 37 stimuli consist of 2 or 3 sentences |  196 native speakers ($\bar{a}$: 22.7 (2.6)) |  Includes human predictability norms; both silent and aloud reading.
Eye-tracker: EyeLink 1000
Sampl. Freq.: 1,000 Hz
|  |  |

###  2.5 Naturalistic self-paced reading corpora for different languages

Self-paced reading (SPR) is another method to study natural reading where
participants read a text word by word and typically control the reading speed
by proceeding to the next word by, e.g., pressing a button. There exist a few
such corpora in multiple languages (see Table 7). The UCL corpus contains SPR
data in addition to eye-tracking data [23]. 117 psychology students read 361
sentences from English novels. The Natural Stories Corpus [24] consists of SPR
data from 181 participants reading (a subset of) 10 English stories that have
been edited to contain hard-to-process constructions.

###  2.6 Variations of naturalistic eye-tracking-while-reading corpora

There exist variations of naturalistic reading corpora that study reading in
other settings such as the Multimodal Duolingo Bio-Signal Dataset, which
contains data from participants navigating a German language learning website
[64]. Another area where eye-tracking data can provide interesting insights is
source code reading [65]. For example, the EMIP data set contains data of 216
participants reading source code in different programming languages [5].

Table 7: Self-paced naturalistic reading corpora for different languages for both single sentences and text passages L |  Stimuli |  Participants |  Additional characteristics
---|---|---|---
UCL [23]
en |  361 sentences selected from free online novels ($\scriptstyle\\#\textstyle w$: 4,946, $\overline{\scriptstyle\\#\textstyle w}$: 13.7 (6.36)) |  117 psychology students ($\bar{a}$: 18.9) |  The dataset contains eye-tracking data as well.
Natural Stories Corpus [24]
en |  10 stories with a total of 485 sentences ($\overline{\scriptstyle\\#\textstyle w}$ per sentence: 22.38), the texts have been edited to contain many hard-to-process constructions |  181 participants, not all of them read all 10 stories |
|  |  |

###  2.7 Differences of PoTeC to existing corpora

In sum, the overwhelming majority of naturalistic eye-tracking-while-reading
datasets uses English stimulus items with either native or non-native
speakers. Moreover, the stimulus materials of the majority of the datasets (be
it English or another language) are single (i.e., isolated) sentences rather
than paragraphs or texts. Apart from these differences in language and
stimulus length, PoTeC differs from already existing datasets in various ways.
First, PoTeC is larger than any other existing dataset for naturalistic
reading in German in terms of number of participants (for other languages, a
few larger datasets do exist). Second, the stimulus texts of PoTeC are
relatively demanding and presumably require more cognitive effort to process
than, for example, Wikipedia excerpts as used in previous work. This higher
difficulty level of the texts is presumably reflected in more complex eye
movement patterns. Third, the readers’ domain expertise about the topic
presented in the stimulus texts is experimentally controlled and assessed by
asking specific background questions on the topics covered in the texts that
are not directly answered in the texts and thus cannot be derived from the
read text.

##  3 Methods

PoTeC is an eye-tracking-while-reading dataset using stimulus materials
adapted from German university-level textbooks on either physics or biology
(see Section 3.1). The data collection follows a 2$\times$2$\times$2 fully-
crossed factorial design. The three (quasi-experimental) factors are: 1) The
reader’s discipline of studies with the levels physics and biology, which is
manipulated between-subjects and within-items: The selection criteria specify
that each participant studies exactly one of the two disciplines; and each
item (i.e. each text from one of the domains) is read by the participants from
both disciplines. 2) The text domain (i.e. the domain of the stimulus texts)
with the levels physics and biology which is manipulated within-subjects and
between-items: all readers read both the physics and the biology texts, and
each text belongs unambiguously to exactly one domain. 3) The reader’s level
of studies with levels graduate and undergraduate which is manipulated
between-subjects and within-items: each reader belongs to exactly one of the
groups and each item is read by each of the groups. The level of studies is
defined by the semester or program the students are enrolled in at the time of
the experiment: undergraduate is defined as first semester BSc, while graduate
is defined as being enrolled in an MSc or PhD program of the respective
discipline of studies. See Figure 1 for an overview on the study design.

Figure 1: The 2$\times$2$\times$2 fully-crossed factorial study design of
PoTeC. The red cubes denote expert reading, that is, participants having the
level of studies graduate who are reading a text, whose text domain is equal
to the reader’s discipline of studies.

In addition to the eye-tracking data, the reader’s text comprehension and
text-independent background knowledge was assessed via comprehension and
background questions (see Section 3.1.2), demographic information was
collected from each of the participants (see Section 3.3.2), and comprehensive
linguistic annotations of the stimulus texts are provided (see Section 3.1).
The eye-tracking data is pre-processed to obtain fixations and reading
measures (among other data formats) as described in more detail in Section
3.4.

###  3.1 Materials

The present section provides an overview of the materials used and how the
materials were annotated. Table 8 summarizes the different types of
annotations and the tools that were used for the different steps of the
stimulus annotation process.

Table 8: Overview of the tools used to annotate the materials

Annotation |  Data Description |  Tools
---|---|---
Manual annotation |  Word features (manual): stored in one file for each text with one row for each word together with all word level tags (see Section 3.1.3). |  Manual
STTS tag set [73]
Corpus-based annotation |  Word features (extracted): stored together with the other word level tags (see Section 3.1.4). |  dlexDB online interface111Abbr.: $\scriptstyle\\#\textstyle w$, $\overline{\scriptstyle\\#\textstyle w}$ = (mean) number of words of stimuli; $\bar{a}$ = mean age of participants (SD)Abbr.: $\scriptstyle\\#\textstyle w$, $\overline{\scriptstyle\\#\textstyle w}$ = (mean) number of words of stimuli (SD); $\bar{a}$ = mean age of participants (SD)Abbr.: $\scriptstyle\\#\textstyle w$, $\overline{\scriptstyle\\#\textstyle w}$ = (mean) number of words of stimuli (SD); $\bar{a}$ = mean age of participants (SD)Abbr.: L = language of stimuli (ISO-639-1); $\scriptstyle\\#\textstyle w$, $\overline{\scriptstyle\\#\textstyle w}$ = (mean) number of words of stimuli (SD); $\overline{\scriptstyle\\#\textstyle c}$ = mean number of characters per stimulus text; $\bar{a}$ = mean age of participants (SD)Abbr.: $\scriptstyle\\#\textstyle w$, $\overline{\scriptstyle\\#\textstyle w}$ = (mean) number of words of stimuli (SD); $\bar{a}$ = mean age of participants (SD)Abbr.: L = language of stimuli (ISO-639-1); $\scriptstyle\\#\textstyle w$, $\overline{\scriptstyle\\#\textstyle w}$ = (mean) number of words of stimuli (SD); $\overline{\scriptstyle\\#\textstyle c}$ = mean number of characters of stimuli (SD); $\bar{a}$ = mean age of participants (SD)Abbr.: L = language of stimuli (ISO-639-1); $\scriptstyle\\#\textstyle w$, $\overline{\scriptstyle\\#\textstyle w}$ = (mean) number of words of stimuli (SD); $\bar{a}$ = mean age of participants (SD)
dlexDB [29]
Language-model-based annotation |  Surprisal: The estimated surprisal values are directly added to the files containing the other word level features (see Section 3.1.5). |  Pre-trained language models
Python scripts: surprisal.py and get_surprisal.py
Semi-automatic annotation |  Dependency & constituency trees: The trees are created and stored in separate files per tree type and text each containing all trees for one text (see Section 3.1.6). |  Python script: add_syntax_trees.py
|  |

11footnotetext: http://www.dlexdb.de/query/kern/typposlem/

####  3.1.1 Stimulus texts

We selected a total of twelve texts from various German university-level
physics (six texts) and biology textbooks (six texts) [18, 17, 19, 1, 82, 26,
9]. The original texts were chosen in a way that each text was approximately
150 words long (min: 126 words; max: 180 words; mean: 158 words). If
necessary, the text’s content was adjusted to account for the deletion of
mathematical formulas, figures and tables. The resulting texts were self-
contained, and consisted only of plain text. The texts enable cross-domain
comparisons as they are very similar in terms of text features such as lexical
frequency (see Figure 2). Their main difference lies in their content and not
in, for example, texts in one domain simply containing less frequent words.

Figure 2: Domain-specific and text-specific summary of the word length in
characters, the log-lexical lemma frequency, and surprisal (as estimated by
GPT-2 large).

####  3.1.2 Comprehension & background questions

For each text, three text comprehension questions and three background
questions were created. The text comprehension questions required a thorough
understanding of the text, but did not require any additional background
knowledge. The background questions, in contrast, tested the general knowledge
in the topic presented in the text and hence required background knowledge.
The background questions could not be answered with knowledge only acquired
through the respective text. All questions had been designed by experts in the
respective field.

####  3.1.3 Manual word level annotation

Each stimulus text was manually part-of-speech (PoS) tagged according to the
Stuttgart-Tübingen-Tagset (STTS) [73]. In addition to the PoS-tag of the word
itself, we provide the PoS-tag of punctuation marks that directly (i.e.,
without a white space) precede or follow the word, as well as hand-crafted
tags to indicate whether a word was contained in a constituent that is in
quotes or parentheses. Furthermore, the words were manually tagged for other
lexical and orthographic features that arguably affect eye movement behavior
in reading, namely whether the word is an expert or general technical term,
with expert technical term being a term that is typically only understood by
experts in that area (e.g., biology: “homolog”, physics: “phasenrichtig”) and
general technical term being a term that is generally understood (e.g.,
biology: “Proteine”, physics: “Kristalle”). Other tags are whether the word is
(e.g., “DNA”) or contains an abbreviation (e.g., “DNA-Fragment”), contains a
non-Latin character or symbol (e.g., “$\beta$-D-Glucose”), or contains a
hyphen (e.g., “z-Richtung”). Finally, we added various ordinal or binary tags
to encode positional information for each word (ordinal position of the word
in the text and in the sentence, and whether it is the first or last word of a
clause or sentence). See Table 9 for the precise definitions of the different
hand-crafted word tags.

Table 9: Definitions of the manually created lexical and sentence-level
features

Feature |  Definition |  Example
---|---|---
Lexical and orthographic features
expert technical term |  The word is a technical term typically only understood by experts in the field |  homolog (biology), phasenrichtig (physics)
general technical term |  The word is a technical term but generally understood without specific expertise required |  Proteine (biology), Kristalle (physics)
is abbreviation |  The entire word is an abbreviation |  DNA
contains abbreviation |  The word contains an abbreviation |  DNA-Fragment
contains symbol |  Word contains a symbol |  $+$Ende; $\beta$-D-Glucose
contains hyphen |  Word contains at least one hyphen that is not STTS-tagged as TRUNC |  DNA-Fragment; z-Richtung; $\beta$-D-Glucose
Linear position information
word index in text |  Position of the word within the current text, irrespective of sentences coded as integer |
word index in sentence |  Position of the word within the current sentence coded as integer |
sentence index |  Position of the sentence to which the word belongs within the current text coded as integer |
Punctuation
STTS punctuation before |  STTS-tag of the punctuation that precedes the word if applicable [73] |
STTS punctuation after |  STTS-tag of the punctuation that follows the word if applicable [73] |
quote |  Word is (part of an expression that is) in quotes |  “Gleisen”
parentheses |  Word is (part of an expression that is) in parentheses |  (z.B. dielektrische … )
Syntactic features
clause begin/end |  Marks the first/last word of a new clause |
sentence begin/end |  Marks the first/last word of a new sentence |
dependency & constituency trees |  Dependency trees for each sentence created semi-automatically |
|  |

####  3.1.4 Corpus-based word level annotation

Moreover, for each word, several word length measures, lexical frequency
measures, and lexical neighborhood measures commonly used in reading research
were extracted from the lexical database dlexDB [29], which is based on the
reference corpus underlying the Digital Dictionary of the German Language
(DWDS) [14]. All extracted values were manually corrected by a linguistic
expert labeller. In particular, the type-to-lemma mapping was disambiguated
and incorrect database entries (e.g., incorrect lemmatization) were corrected.
If the lemma has been manually added, the lemma frequency was coded as missing
value. Overviews of all features extracted from dLexDB are provided in Tables
10, 11, and 12.

Table 10: Definition of the corpus-based features extracted from dlexDB: Linguistic representations & word length features Feature |  Definition
---|---
Type |  Orthographic representation of a word as found in the stimulus text (case sensitive)
Lemma |  Headword, i.e., an uninflected form that may or may not occur in the stimulus corpus itself
Syllables |  Syllables of which the word consists
Type length char. |  Number of characters of a type
Type length syll. |  Number of syllables of a type
Lemma length |  Number of characters of a lemma
|
Table 11: Definition of the corpus-based features extracted from dlexDB: Frequency measures Feature |  Definition
---|---
Annotated type frequency |  Number of occurrences of a unique combination of a type, its STTS tag and its lemma in the corpus (per mio tokens)
Type frequency |  Number of occurrences of a type in the corpus (per mio tokens)
Lemma frequency |  Total number of occurrences of types associated with this lemma in the corpus (per mio tokens)
Document frequency |  The number of documents with at least one occurrence of this type (per 10.000 documents)
Sentence frequency |  Number of sentences with at least one occurrence of this type (per 100.000 sentences)
Cumulative syllable corpus frequency |  Cumulative frequency of the individual syllables of the word in the corpus (per mio tokens)
Cumulative syllable lexicon frequency |  Cumulative frequency of the individual syllables of the word as listed in the lexicon (per mio types)
Cumulative character corpus frequency |  Cumulative corpus frequency of all characters contained in this type (per mio tokens)
Cumulative character lexicon frequency |  Cumulative lexicon frequency of all characters contained in this type (per mio types)
Cumulative character bigram corpus frequency |  Cumulative corpus frequency of all character bigrams contained in this type (per mio tokens)
Cumulative character bigram lexicon frequency |  Cumulative lexicon frequency of all character bigrams contained in this type (per mio types)
Cumulative character trigram corpus frequency |  Cumulative corpus frequency of all character trigrams contained in this type (per mio tokens)
Cumulative character trigram lexicon frequency |  Cumulative lexicon frequency of all character trigrams contained in this type (per mio types)
Initial letter frequency |  Cumulative frequency of all types sharing the same initial letter (per mio tokens)
Initial bigram frequency |  Cumulative frequency of all types sharing the same initial character bigram (per mio tokens)
Initial trigram frequency |  Cumulative frequency of all types sharing the same initial character trigram (per mio tokens)
Average conditional probability in bigrams |  Conditional probability of the respective word being the second component in word bigrams, given the occurrence of its first component, averaged across all possible bigrams with that word as second component (computed on the basis of the annotated type information)
Average conditional probability in trigrams |  Conditional probability of the respective word being the third component in word trigrams, given the occurrence of its first and second component, averaged across all possible trigrams with that word as third component (computed on the basis of the annotated type information)
Familiarity |  Cumulative frequency of all types of the same length sharing the same initial trigram
Regularity |  The number of types of the same length sharing the same initial trigram
|
Table 12: Definition of the corpus-based features extracted from dlexDB: Neighborhood measures Feature |  Definition
---|---
Cumulative frequency of higher frequency neighbors (Coltheart) |  Cumulative frequency of all higher frequency orthographic neighbors according to the definition of Coltheart . [12]: Neighbors need to be of the same length and differ at one character position from each other. E.g., “Hans” has higher frequency Coltheart neighbor “Haus”.
|
Count of of higher frequency neighbors |  Number of higher frequency orthographic neighbors according to the definition of Coltheart . [12]
|
Cumulative frequency of all neighbors (Coltheart) |  Cumulative frequency of all orthographic neighbors according to the definition of Coltheart . [12]
|
Count of all neighbors (Coltheart) |  Number of orthographic neighbors according to the definition of Coltheart . [12]
|
Cumulative frequency of higher frequency neighbors (Levenshtein) |  Cumulative frequency of all higher frequency orthographic neighbors according to the definition of Levenshtein [50]: Words are neighbors if they differ by one change operation (inserting, deleting or exchanging a character).
|
Count of of higher frequency neighbors (Levenshtein) |  Number of higher frequency orthographic neighbors according to the definition of Levenshtein [50]
|
Cumulative frequency of all neighbors (Levenshtein) |  Cumulative frequency of all orthographic neighbors according to the definition of Levenshtein [50]
|
Count of all neighbors (Levenshtein) |  Number of orthographic neighbors according to the definition of Levenshtein [50]
|
|

####  3.1.5 Language-model-based word level annotation

In addition to the manual tagging and the dlexDB tags, all words in the texts
were annotated with surprisal values obtained from different language models.
As has been shown by previous research, surprisal values and their predictive
power differ depending on the language model that was used for their
estimation [25, 85, 84]. Therefore, the surprisal values were estimated by
three models differing in model architectures and size. Two models had been
trained in an auto-regressive manner which takes only the left-hand side
context of a word into account to predict the next word, which mimics the
incremental nature of human language processing. The first is GerPT2 [59], a
language model that was trained on German data but initialized from the
English GPT-2 model [69]. Both GerPT2
Large222https://huggingface.co/benjamin/gerpt2-large (876M parameters) and
Base333https://huggingface.co/benjamin/gerpt2 (176M parameters) were used. The
second model is LeoLM, a German language model built on Llama 2 [81] and is
fine-tuned on German data. We used the 7b444https://huggingface.co/LeoLM/leo-
hessianai-7b and 13b555https://huggingface.co/LeoLM/leo-hessianai-13b versions
of LeoLM. The third model used is a German
variant666https://huggingface.co/bert-base-german-cased of BERT [21] with 109M
parameters. BERT is not an auto-regressive model as it takes both the left-
and right-hand context into account.

In order to obtain the surprisal values, all texts were sub-word-tokenized for
the respective model and the tokenized input sequence was processed by the
model to obtain the log-probabilities of the individual sub-word-tokens which
were then added up to get the surprisal for each word. All surprisal values
were estimated once with the respective (left-hand) sentence as context and
once with the entire (left-hand) text as context as the participants had also
seen the entire text on one page during reading.

Table 13: Language-model-based word level features estimated by different
model types

Feature |  Model type |  Size
---|---|---
| Auto-regressive
sentence context surprisal
&
text context surprisal |  GerPT2 base (GPT-based) |  176M parameters
|  GerPT2 large (GPT-based) |  876M parameters
|  LeoLM 7b (Llama-2-based) |  7B parameters
|  LeoLM 13b (Llama-2-based) |  13B parameters
| Non-auto-regressive
|  BERT |  190M parameters
|  |

66footnotetext: For all model types, surprisal has been estimated once with
the sentence and once with the entire text as context. As the surprisal is
estimated on sub-word-token-level, the resulting surprisal values have been
added up to obtain the surprisal for one word.

####  3.1.6 Semi-automatically created dependency and constituency trees

To facilitate future analysis of the data, dependency and constituency trees
were added to all sentences. In order to create the dependency trees, the
Python library spaCy777https://spacy.io/ was used. The tool to create the
dependency trees was trained on the TIGER corpus and it therefore uses the
TIGER annotation scheme [10, 80]. The tool automatically parses and annotates
each word with its respective dependencies. The dependency trees were manually
corrected by a German linguistic expert. The constituency trees were created
using the open-source tool benepar [41, 42]. The tool receives each text
separately as input, splits it into sentences, tokenizes the sentences,
annotates the words with PoS-tags and groups the resulting annotated words
into sentence constituents and annotates them accordingly. As the entire
pipeline relies on pre-trained models for the different tasks, it might result
in different PoS-tag annotation compared to our manual tags. The resulting
constituency trees were consequently manually corrected such that the PoS-tags
correspond with our manual tags and to account for any other errors.

###  3.2 Participants

75 students888Data for 76 participants was collected, however, for one
participant only the data for 5 out of 12 trials was available which is why
this participant’s data was eventually excluded. of the University of Potsdam
all of whom were native speakers of German with normal or corrected-to-normal
vision participated in the experiment. They were either students of biology or
of physics in either their first semester of the BSc program (undergraduate)
or graduate students currently attending an MSc or PhD program (graduate). In
total there were 12 undergraduate physics students, 20 graduate physics
students, 16 undergraduate biology students and 27 graduate biology students.
Participants were requested to not have consumed any alcohol the day of the
experiment and come to the experiment well rested. Participants received a
compensation of 20 EUR. A short overview of the participants mean age and a
selection of other characteristics is found in Table 14.

Table 14: Overview over the participants mean age and a selection of other characteristics Discipline of studies |  Level of studies |  Number of participants |  Mean age (SD) |  Mean hours of sleep (SD) |  Glasses
---|---|---|---|---|---
Biology |  undergrad. |  16 |  21.5 (3.2) |  7.3 (0.9) |  no: 14, yes: 1, N/A: 1
Biology |  graduate |  27 |  26.2 (4.1) |  7.3 (1.0) |  no: 17, yes: 10
Physics |  undergrad. |  12 |  20.5 (3.3) |  6.6 (2.3) |  no: 8, yes: 4
Physics |  graduate |  20 |  25.7 (2.8) |  7.4 (1.2) |  no: 15, yes: 5
Overall |  75 |  24.2 (4.2) |  7.2 (1.3) |  no: 54, yes: 20, N/A: 1

###  3.3 Experiment Procedure and Technical Set-up

The data collection was carried out in accordance with the Helsinki
Declaration [87]. Informed consent was obtained from all participants prior to
starting the experiment. Participants were instructed about the procedure of
the experiment in written form and clarified their questions with the
experimenter orally. The experiment started with the recording of the eye
movements (see Section 3.3.1) which was followed by a short demographic
questionnaire (see Section 3.3.2). The total duration of the experiment
including instructions, camera-setup, breaks, calibrations and questionnaire
was approximately 90 minutes.

####  3.3.1 Eye-tracking-while-reading

Participants’ eye movements (right eye monocular tracking) were recorded at a
sampling rate of 1,000 Hz using an Eyelink 1000 eye tracker manufactured by SR
Research with a desktop mounted camera system with a 35 mm lens. A Cedrus
button box was used as a response pad. The experimental presentation and the
communication between the presentation computer and the eye tracker was
implemented using the Experiment Builder software provided by SR Research.

The participant was seated at a height-adjustable table to ensure a constant
eye-to-screen distance across participants. The participant’s head was
stabilized using a chin- and forehead rest which should increase the quality
of the recorded data. The texts were presented on a 22 inch monitor with a
resolution of 1680$\times$1050 pixels and a screen size of 47.5$\times$30 cm.
The eye-to-screen distance measured 61 cm and the eye-to-camera distance was
65 cm. Both distance measures were measured as specified in the EyeLink
Installation Guide [76, p. 15, p. 70].

The texts were presented in a mono-spaced white font (Courier, font size 18)
on a black background. The reason for choosing a black background was the
rather long duration of the experiment. A bright background color would strain
the participants’ eyes and potentially lead to wet eyes which has a negative
impact on calibration accuracy.

After the set-up and initial calibration (9-point calibration) of the camera
and the validation, the participant first read one practice text followed by
six practice questions to get familiar with the experimental procedure. The
twelve experimental texts were presented in randomized order (separate
randomization for each participant). Each experimental trial began with the
presentation of the header of the following text on an individual screen. The
participant had to press a button to continue to the text which was shown on a
new screen. Each text fit onto a single screen. There were no restrictions
regarding the time spent on reading each text. After having finished reading
the text, the participant had to look at a green sticker that was placed on
the bottom right corner of the monitor and at the same time press a button to
continue to the questions. This procedure helps to avoid random fixations on
the text after the participant has finished reading as they are fixating a
specified target away from the areas of interest. Each question was presented
on a separate screen together with four answer options of which the
participant had to select one by pressing the respective button on the
response pad. It was not possible to go back to the text or previous questions
nor was it possible to undo an answer. The order of answer options was
randomly shuffled for each participant. The three text comprehension questions
always preceded the three background questions in order to minimize memory
effects on the response accuracy; the order of the three questions within each
type was randomized for each participant. Participants were informed that some
of the questions required background knowledge, however they were not informed
which ones.

If necessary, re-calibrations were performed before the beginning of a new
trial followed by another validation. For all of the participants, re-
calibrations were performed throughout the experiment (see Table 15 for an
overview on the calibrations and validations performed). Appendix B presents a
table where all the average validation scores for each session are listed
together with the number of validations and calibrations performed in that
session. Participants were allowed to take a break before the beginning of a
new trial.

Table 15:  Statistics over the number of calibrations and validations
performed in the experiment and the validation scores.

Type | Validation score (avg) | Validation score (max) | # per session
---|---|---|---
|  mean |  std |  max |  min |  mean |  std |  max |  min |  mean |  std |  max |  min
cal |  |  |  |  |  |  |  |  |  14.418 |  4.73 |  27 |  2
val |  0.421 |  0.253 |  2.37 |  0.188 |  0.84 |  0.567 |  5.045 |  0.422 |  8.899 |  3.07 |  15 |  2
|  |  |  |  |  |  |  |  |  |  |  |

88footnotetext: Note that the validation scores (avg and max) are both
averages over all the validations performed in one session. See Appendix B for
a session overview. SR research recommends an average validation score below
$0.5$.

####  3.3.2 Demographic questionnaire

After the eye-tracking experiment was concluded, participants had to fill in a
short demographic questionnaire. The following information was collected: the
field of studies (including area of specialization if applicable) and the
current semester of studies in order to verify the expert status, gender, age,
handedness, whether the participant was wearing (soft or hard) contact lenses
or glasses, hours of sleep the night before the experiment, alcohol
consumption within 24h hours before the experiment, whether or not the
participant had grown up bilingually, and the state (“Bundesland”) where the
German language was acquired.

###  3.4 Data Preprocessing

The eye-tracking data for each participant and text were pre-processed and are
made available in different formats. Whenever the tools and scripts are
created by ourselves and not protected by a license or copyright, the tools
used to complete the different steps are made publicly available. Table 16
provides an overview of the different preprocessing steps and the tools used.

Table 16: Data processing pipeline including the tools that were used Pre-processing Step |  Data Description |  Tools
---|---|---
Pre-processing of eye movement data
Collect:
raw data |  Raw data (as-is): Raw data including metadata for each session in one file. At experiment-time non-human-readable edf files are written which are directly converted to human-readable asc files. |  EyeLink 1000, Experiment Builder software (SR Research), SR Research edf2asc
Pre-process:
raw data |  Raw data (pre-processed): Raw data as collected is pre-processed to only contain the relevant samples (right eye $x$ and $y$ coordinates, pupil diameter, timestamp) for each trial. The files contains one data sample per line (tsv format). |  asc_to_csv.py,
SR Research asc2csv
Compute:
raw data $\rightarrow$ fixation data |  Uncorrected fixation data: From the pre-processed raw data, fixations are computed. Contains the character-level area of interest for each fixation and fixation and saccade features. |  Eyelink Data Viewer [77], default parameter settings
Correct:
fixation data $\rightarrow$ corrected fixation data |  Fixations: Manually corrected fixation data. Contains information about whether the fixation was corrected or not and if so, information on the original fixation (see Section 3.4.3). |  Python script: correct_fixations.py and split_fixation_ report.py
Pre-processing of stimulus material
Compute:
character index $\rightarrow$ word index |  Character to word mapping: The indices of the character-level areas of interest are mapped to the word indices in the respective text. |  Python script: char_index_to_ word_index.py
Compute:
text $\rightarrow$ word/sentence limits |  Word limits and sentence limits: Contains information about the first and the last character index (i.e., aois) of each word and each sentence in each text. |  Python script: create_word_ aoi_limits.py
Pre-processing of stimulus material and eye movement data combined
Compute:
corrected fixation data $\rightarrow$ word level reading measures |  Reading measures: For each trial, word level reading measures are computed that are written to a separate file for each reader and text (see Section 3.4.4). |  Python script: compute_reading_ measures.py
Merge:
corrected fixation data with character and words |  Scanpaths: Each character-level fixation in all scanpaths was merged with the fixated character and word, and information on trial-level like the text identifier is added (see Section 3.5). |  Python script: generate_scanpaths.py
Merge:
reading measures with stimulus corpus data |  Reading measures merged: Merge reading measures with all features at all levels that have not been included yet (see Table 18 for an overview) |  Python script: merge_reading_ measures.py
Merge:
scanpaths with stimulus corpus data |  Scanpaths merged: Merge scanpaths with all features at all levels that have not been included yet (see Table 18 for an overview). |  Python script: merge_scanpaths.py
|  |

####  3.4.1 Pre-processing of raw data

The data files originally written by the eye-tracking device are non-human
readable edf files which were directly converted to a human-readable asc
format using the SR Research edf2asc tool. The asc files contain the data for
one session including metadata and need to be parsed to extract the relevant
samples for each trial and store it in a separate tsv file which then contains
one sample per line for one trial. One sample consists of the $x$ and $y$
coordinates for the tracked eye and the timestamp.

####  3.4.2 Computation of fixations

Fixations and saccades were computed from the pre-processed raw data using the
EyeLink Data Viewer software provided by SR Research with the default
parameter settings [77]. Subsequently, each fixation was mapped to the
character index in the text that was fixated and annotated with the line index
and the character index in that line. Fixations on the white space between two
words were mapped to the closest character.

####  3.4.3 Manual correction of the fixation data

Visual inspection of the fixation data revealed that in certain fixation
sequences, vertical calibration error gradually increased over time. This
measurement error was manually corrected by adjusting the fixation-to-
character mapping (i.e., re-mapping a fixation to the character in the line
above or below the currently mapped character). Horizontal calibration drift
could not be corrected as there is no way to infer the magnitude of the
measurement error from the data. An exception are fixations on the first or
last word of a line. If the fixation was just next to the line and not on the
word, it was corrected to be on the first, respectively last word of the line.
In any case, horizontal measurement error is less dramatic for the purpose of
this study (and reading experiments in general) as fixations are eventually
mapped to words, thus a horizontal measurement error will only result in a
wrong fixation-to-word mapping when the measurement error is larger than the
distance of the real fixation location to the word boundary. In contrast, even
small measurement error on the vertical axis can lead to incorrectly mapping
the fixation to be on a word in the line above or below. In some trials, the
participant did not only read the text but also scanned the screen without
reading. Such sequences of fixations can be easily distinguished from eye
movements reflecting reading by visual inspection (automatized approaches
exist, but are less accurate, see Biedert . [8]). Whenever such sequences of
non-reading fixations occurred at the beginning or the end of a trial, they
were deleted within the process of correcting the fixation locations. In
addition, very short fixations that looked like optical artifacts were
deleted. All three stages of the data up to this point, the pre-processed raw
data, the originally computed uncorrected fixations and the manually corrected
fixation data (corrected fixation location and deleted non-reading sequences)
are made available in the data repository.

####  3.4.4 Computation of word-level reading measures

From the fixation data, various reading measures commonly used in reading
research were computed for each text and reader. Each word (defined by the
surrounding white spaces) was considered one area of interest for all measures
except for landing position which was based on the characters within a word.
Fixations on the punctuation marks were considered to belong to the preceding
word by default and to the following word in case of opening parentheses or
opening quotation marks. Definitions of the various measures are provided in
Table 17. Note that several subsets of these measures are linearly dependent.

Table 17: Definition of the word-level eye movement measures. Measure |  Abbr. |  Definition
---|---|---
Continuous measures (in ms)
first-fixation duration |  FFD |  duration of the first fixation on a word if this word is fixated in first-pass reading, otherwise 0
first duration |  FD |  duration of the first fixation on a word (identical to FFD if not skipped in the first-pass)
first-pass reading time |  FPRT |  sum of the durations of all first-pass fixations on a word (0 if the word was skipped in the first-pass)
single-fixation duration |  SFD |  duration of the only first-pass fixation on a word, 0 if the word was skipped or more than one fixations occurred in the first-pass (equals FFD in case of a single first-pass fixation)
first-reading time |  FRT |  sum of the duration of all fixations from first fixating the word (independent if the first fixations occurs in first-pass reading) until leaving the word for the first time (equals FPRT in case the word was fixated in the first-pass)
total-fixation time |  TFT |  sum of all fixations on a word (FPRT$+$RRT)
re-reading time |  RRT |  sum of the durations of all fixations on a word that do not belong to the first-pass (TFT$-$FPRT)
inclusive regression-path duration |  RPD_inc |  sum of all fixation durations starting from the first first-pass fixation on a word until fixating a word to the right of this word (including all regressive fixations on previous words), 0 if the word was not fixated in the first-pass (RPD_exc$+$RBRT)
exclusive regression-path duration |  RPD_exc |  sum of all fixation durations after initiating a first-pass regression from a word until fixating a word to the right of this word, without counting fixations on the word itself (RPD_inc$-$RBRT)
right-bounded reading time |  RBRT |  sum of all fixation durations on a word until a word to the right of this word is fixated (RPD_inc$-$RPD_exc).
Binary measures
fixation |  Fix |  1 if the word was fixated, otherwise 0 (FPF or RR)
first-pass fixation |  FPF |  1 if the word was fixated in the first-pass, otherwise 0
first-pass regression |  FPReg |  1 if a regression was initiated in the first-pass reading of the word, otherwise 0 (sign(RPD_exc))
re-reading |  RR |  1 if the word was fixated after the first-pass reading, otherwise 0 (sign(RRT))
Ordinal measures
total fixation count |  TFC |  number of all fixations on a word
landing position |  LP |  position of the first saccade on the word expressed by ordinal position of the fixated character
incoming saccade length |  SL_in |  length of the saccade that leads to first fixation on a word in number of words; positive sign if the saccade is a progressive one, negative sign if it is a regression
outgoing saccade length |  SL_out |  length of the first saccade that leaves the word in number of words; positive sign if the saccade is a progressive one, negative sign if it is a regression; 0 if the word is never fixated
total count of outgoing regressions |  TRC_out |  total number of regressive saccades initiated from this word
total count of incoming regressions |  TRC_in |  total number of regressive saccades landing on this word
|  |

###  3.5 Summary of the Available Features

After all the data was annotated and pre-processed, PoTeC has different
features available at different levels (word-, text-, reader-, fixation- and
trial-level)999For some textual features like the the discipline of studies or
the text domain, a numerical encoding was added for easy computational
processing while still keeping an explicitly understandable text-encoded label
(e.g., level of studies physics). As both features encode the same
information, it won’t be explicitly stated in the following overview but is
defined in the data repository.. 1) Word-level features: All word-level
features, which include manual and (semi)-automatically computed or estimated
tags, are defined in Tables 9–13. 2) Text-level features: Each text was given
a unique identifier encoded as textual feature explicitly encoding the text
domain. In addition to the text itself, the text’s domain, the comprehension
questions, all the answer options, and the encoding of the correct answers to
all questions are text-level features. 3) Reader-level features: The data
collected through the demographic questionnaire and a unique reader identifier
are reader-level features as well as the level of studies and the discipline
of studies. The order of the stimuli texts and the order of the answer options
for each of the questions are reader-level features and the average response
accuracy over all questions for each question type. 4) Fixation-level
features: A chronological fixation index, fixation duration, previous saccade
duration, next saccade duration and the area of interest are available
fixation-level features. In addition, whether or not the fixation has been
manually corrected, the original chronological fixation index, the uncorrected
fixation location and the uncorrected area of interest are added as features.
5) Trial-level features: For each text read by a participant, the response
accuracy for all questions and the mean accuracy for all text and all
background questions are provided. In addition, a feature was added to each
trial that encodes whether this specific trial is expert reading or not (see
Figure 1). In addition, trial-level features include the word-level reading
measures that have been computed for each text and reader as described in
Section 3.4.4.

Table 18: Available features of PoTeC at different levels

Feature level |  Features
---|---
Reader-level |  Unique reader identifier, age, handedness, gender, how many hours of sleep the reader had the night before the experiment, whether of not the reader wears glasses, whether or not the participant consumed alcohol within 24 hours before the experiment start, level of studies, discipline of studies, and domain expert level (e.g., physics-undergraduate), the order of texts in this session, and the order of answer options for all question.
Text-level |  Unique text identifier, text, questions, answer options, the correct answer for all questions, and text domain.
Trial-level |  Answer accuracy for all questions for the respective text, mean accuracies for text and background questions for the text, and whether or not the trial is an example of expert reading (i.e., the text is read by an expert in the text domain) and the word level reading measures defined in Table 17.
Word-level |  All features defined in Tables 9–13.
Fixation-level |  Chronological fixation index, fixation duration, previous saccade duration, next saccade duration, whether or not the fixation has been manually corrected, the original chronological fixation index, fixated character and word, line index of the fixated character and the character index in the line.
|

99footnotetext: Note: The features marked in bold are either the experimental
factors or features based on the experimental factors.

The fixation data and reading measures are processed further and merged with
different features of the corpus material on different levels to facilitate a
simple use of the data. Note that these steps do not add any additional
information but combine and merge the existing corpus data in such a way that
common analyses of eye movement data for both psycholinguistic as well as NLP
use cases are simplified and therefore encouraged. As the fixation data is
originally only associated with the index of the fixated character, it is
further processed to explicitly include the fixated character and word. Also
included are word-level features of the fixated word as well as most trial-,
session-, reader- and text-level features in addition to the fixation-level
features. The resulting data representation is then referred to as the
scanpath for this reader and text which represents the chronological order of
the eye movements on the text. The reading measures are merged with all the
remaining reader-, text-, trial-, and session-level features that had not been
added before as well as the remaining word-level features (i.e., the various
word-level tags). Both the reading measures data and the scanpaths are made
available in a format including only the most important features and in an
extended format that includes all available features merged with the eye
movement data (compare Table 16). Note that those two data representations
represent the same data in two very different formats with the scanpath data
representing one fixation per line in chronological fixation order and the
reading measures data representing the eye movement data at the word-level
with one line representing one word in the order of the text.

###  3.6 Accessing the Data

PoTeC consists of various different types of data which include on the one
hand many large data files and on the other hand different Python scripts. To
account for the differences in those data formats two channels have been
chosen to share the data. The data files are stored in an Open Science
Framework (OSF) repository: https://osf.io/dn5hp/. The code is stored in a
GitHub repository: https://github.com/DiLi-Lab/PoTeC. The GitHub repository
can be cloned locally and the data files can be automatically downloaded using
a Python script101010download_data_files.py, please consult the README files
in the repository for further instructions.. Alternatively, the data files can
be manually downloaded from OSF.

Additionally, PoTeC has been integrated into the Python package
pymovements111111https://pymovements.readthedocs.io/en/stable/reference/index.html
[46]. The package allows for downloading the raw data directly within a Python
or R script and can then be further processed using the package. For example,
events can be detected or plots can be created to visualize the data121212More
instructions are found in our GitHub repository..

##  4 Usage of the Data

The data has already been used for different research purposes. For example,
Krakowczyk, Prasse . [45] used PoTeC to develop Explainable Artificial
Intelligence (XAI) methods for analysing deep neural networks processing eye-
tracking data. Hollenstein . [32] fine-tuned large language models on
different eye-tracking corpora including PoTeC to understand better to what
extent the representations learned by large language models are comparable to
human reading behavior. Makowski . [55] used PoTeC for biometric
identification and the prediction of reading comprehension.

##  5 Analyses of the Data

We have already illustrated a few past use cases for PoTeC in Section 4. In
the following, we present a series of analyses on a range of reading measures
(first-pass reading time, total fixation time, re-reading time, first-pass
regressions) to test the effects of the word- and sentence-level features as
well as the impact of expertise on eye-movements. More specifically, we deploy
hierarchical (generalized) linear-mixed models to:

  1. 1.

explore the reading behavior of experts and non-experts (see Figure 1 for a
visual representation of what we refer to as expert reading),

  2. 2.

test the effect of different word- and sentence-level features such as word
length, surprisal, and lexical frequency, as well as whether the word
represents an expert term, whether it was read in the expert reading condition
(see Table 18), and the reader’s discipline of studies.

We model each word-level reading measure $y$ using expert reading (binary;
1=stimulus text is read by a graduate student of the discipline that
constitutes the text domain, 0=else; see Figure 1), expert technical
term131313Note that in the data, there are two types of technical terms.
Please refer to Section 3.1.3 and Table 9 for an explanation of those terms
and all the features used in this present section. (binary; 1=expert technical
term that is not generally understandable, 0=else), reader discipline (binary;
1=physics, 0=biology; reader’s discipline of studies), word length, log-lemma
frequency and lexicalized surprisal141414We used sentence-based surprisal
values extracted from GPT-2 large. as predictors, formalized as follows:

$\begin{split}y_{ij}=&g(\beta_{0}+\beta_{0i}+\beta_{1}~{}\texttt{expert
reading}_{i}+\beta_{2}~{}\texttt{reader discipline}_{i}+\\\
&\quad\beta_{3}~{}\texttt{reader discipline}_{i}*\texttt{expert
reading}_{i}+\\\ &\quad\beta_{4}~{}\texttt{expert technical
term}_{i}+\beta_{5}~{}\texttt{word length}_{j}+\\\
&\quad\beta_{6}~{}\texttt{expert reading}_{i}*\texttt{word
length}_{j}+\beta_{7}~{}\texttt{log-lemma frequency}_{j}+\\\
&\quad\beta_{8}~{}\texttt{expert reading}_{i}*\texttt{log-lemma
frequency}_{j}+\beta_{9}~{}\texttt{surprisal}_{j}+\\\
&\quad\beta_{10}~{}\texttt{expert
reading}_{i}*\texttt{surprisal}_{j})\end{split}$

where $y_{ij}$ refers to the eye-tracking reading measure of subject $i$ for
the $j$th word in the stimulus corpus across all texts. $\beta_{0}$ represents
the global intercept, and $\beta_{0i}$ the random intercept for subject $i$.
$g(\cdot)$ denotes the linking function: $g(z)=\ln\frac{z}{1-z}$ for the
binary measures (first-pass regression) with $y_{ij}$ following a Bernoulli
distribution (i.e., logistic regression); and the identity function for the
remaining continuous measures with $y_{ij}$ following a log-normal
distribution (i.e., linear regression on a log-transformed dependent
variable). We run each model for $6000$ iterations using standard priors. We
provide the Stan code including priors in the repository.

Figure 3: Posterior effect estimates for predictors _expert reading_ , _reader
discipline_ , the interaction of _expert reading_ with _reader discipline_ ,
_expert technical term_ as well as _log-lexical frequency_ , _surprisal_ and
_word length_ and the interactions of _log-lexical frequency_ , _surprisal_ ,
and _word length_ with _expert reading_.

With respect to the effects of word features on reading times, we find
different expected effects. Higher lemma-frequency facilitates processing
(shorter reading times, fewer regressions) while increased word length is
associated with increased processing effort (longer reading times, more
regressions). Similarly, high surprisal was associated with longer reading
times. Moreover, participants showed increased reading times on expert
technical terms. We find that expert reading is characterized by shorter
reading times, most strongly observed in total fixation times, and fewer
regressions. Lastly, while we did not find any effects of reader discipline
itself on any of the reading measures, the positive interaction between expert
reading and reader discipline suggests that physics graduates (i.e., experts)
make more regressions and exhibit higher total-fixation and re-reading times
during expert reading.

##  6 Summary & Conclusion

We presented PoTeC, a naturalistic German eye-tracking-while-reading corpus.
PoTeC is the first eye-tracking dataset using a novel experiment design where
the readers’ expertise has been manipulated within-subject. That is, experts
and beginners from two different disciplines read texts from both disciplines.
The stimulus corpus has been comprehensively linguistically annotated with a
wide range of features on different linguistic levels obtained through manual
annotation, extracted from databases or computed by state-of-the-art
computational models. The eye movement data is made available at all pre-
processing stages with maximal flexibility for the user to chose which version
of the data is useful for their use case (e.g., areas of interest at
character- and word-level, uncorrected and corrected fixation data, scanpath
data, raw data samples, etc.). The data is released with all the code that has
been written for annotating or pre-processing the data which means that the
entire pipeline can be fully reproduced and the code can be adapted to other
use cases. Extensive demographic data has been collected for the participants
and reading comprehension scores have been collected through comprehension
questions and, in addition, the readers’ background knowledge has been
assessed through background questions on the different topics of the stimulus
texts. The data and code is made available in a way that creates maximal
transparency, maximally increases the re-usability of the data and the code,
and makes the data accessible in a user-friendly way using easy to understand
data formats. In addition, the data has been integrated into an existing open-
source
software151515https://pymovements.readthedocs.io/en/stable/reference/index.html
for the processing of eye-movements that can be used in Python and R.

Given the different features described above, we envision different use cases
of the data: 1) PoTeC can be used to study within-subject expert and non-
expert reading patterns. As our exploratory analyses have shown, expert
reading behavior can be characterized by shorter reading times when compared
with non-expert reading. PoTeC is the first eye-tracking-while-reading dataset
to allow such analyses. 2) All of these features can be used to train
computational models to learn how to distinguish expert readers from non-
expert readers. That is, models could be trained, for example, to infer the
reader’s domain expertise when presented with a scanpath on a particular text.
Such models could be used to conduct tests in e-learning scenarios to assess
the learners progress. 3) The naturalistic reading data can be leveraged for
NLP purposes without specifically studying expert reading. 4) The aim of the
corpus is not only to make all the data available but also to publish any
scripts and tools that were used to preprocess the data. This should increase
transparency and foster high-quality data as it allows for analyzing not only
the data but also the data collection and preprocessing pipeline which can
help to continuously improve future data collections and algorithms to
preprocess eye-tracking data. 5) Making the uncorrected and the manually
corrected fixation data available allows for investigating how fixation data
can be corrected automatically. PoTeC is the first corpus to make this kind of
data available. In particular, the data can be used to train computational
models to automatically correct fixation data as the manual correction is a
very time-consuming endeavor but in practice often necessary and often part of
preprocessing pipelines.

There exists many more use cases such as: creating models based on raw data,
analyses of high and low reading comprehension, inference of the reader’s
reading comprehension, inference of whether a text is difficult for a reader
or whether it is a text from their domain of expertise. Our vision is that the
corpus can be used for all of the above mentioned use cases and many more that
go beyond psycholinguistics, cognitive reading research or NLP. Finally, the
work we described aims to encourage further eye-tracking data collections that
can complement the existing data.

Acknowledgments We thank Shravan Vasishth for generously granting access to
his eye-tracking lab and for providing the necessary financial support to
compensate the participants, enabling the successful execution of the
experiment. We further thank Marius Huber for his assistance with setting up
the data repository.

This work was partially funded by the Swiss National Science Foundation under
grant 212276 and the German Federal Ministry of Education and Research under
grant 01$|$ S20043.

Open Practices Statement Materials and code are available in a GitHub
repository at https://github.com/DiLi-Lab/PoTeC. The large data files are
stored at https://osf.io/dn5hp/, but can be downloaded via the GitHub
repository. None of the reported studies were preregistered.

##  Appendix A Text Characteristics

Figure 4 presents a more extensive overview over different text
characteristics.

Figure 4: Domain-specific and text-specific summary of word length (both
character and syllable counts), log-lexical frequency (lemma and type) and
surprisal (GPT-2 large estimated with left-hand sentence context) across
texts.

##  Appendix B Calibration and Validation Overview

Table 19:  Number of 9-point calibrations and validations per session and the averages of the average and the maximal validation score for each validation in each session. ID |  Type |  Eye |  $\overline{avg}$ |  $\overline{max}$ |  # |  ID |  Type |  Eye |  $\overline{avg}$ |  $\overline{max}$ |  # |  ID |  Type |  Eye |  $\overline{avg}$ |  $\overline{max}$ |  #
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
0 |  cal |  R |  |  |  16 |  34 |  cal |  R |  |  |  9 |  78 |  cal |  R |  |  |  14
|  val |  R |  0.364 |  0.737 |  9 |  |  val |  R |  0.422 |  0.679 |  8 |  |  val |  R |  0.307 |  0.554 |  11
1 |  cal |  R |  |  |  20 |  35 |  cal |  R |  |  |  12 |  79 |  cal |  R |  |  |  14
|  val |  R |  0.52 |  1.536 |  7 |  |  val |  R |  0.312 |  0.538 |  8 |  |  val |  R |  0.268 |  0.601 |  8
2 |  cal |  R |  |  |  25 |  36 |  cal |  R |  |  |  15 |  80 |  cal |  R |  |  |  12
|  val |  R |  0.597 |  1.068 |  6 |  |  val |  R |  0.283 |  0.431 |  15 |  |  val |  R |  0.341 |  0.655 |  11
3 |  cal |  R |  |  |  13 |  37 |  cal |  R |  |  |  15 |  81 |  cal |  R |  |  |  13
|  val |  R |  0.306 |  0.65 |  7 |  |  val |  R |  0.445 |  0.828 |  10 |  |  val |  R |  0.437 |  0.78 |  13
4 |  cal |  R |  |  |  6 |  38 |  cal |  R |  |  |  15 |  82 |  cal |  R |  |  |  14
|  val |  R |  0.228 |  0.45 |  5 |  |  val |  R |  0.347 |  0.611 |  7 |  |  val |  R |  0.569 |  0.962 |  9
5 |  cal |  R |  |  |  10 |  39 |  cal |  R |  |  |  15 |  83 |  cal |  L |  |  |  19
|  val |  R |  0.293 |  0.677 |  7 |  |  val |  R |  0.355 |  0.681 |  11 |  |  |  R |  |  |  3
6 |  cal |  R |  |  |  12 |  40 |  cal |  R |  |  |  16 |  |  val |  L |  0.882 |  1.773 |  9
|  val |  R |  0.289 |  0.489 |  9 |  |  val |  R |  0.463 |  0.88 |  10 |  |  |  R |  2.37 |  5.045 |  2
7 |  cal |  R |  |  |  10 |  41 |  cal |  R |  |  |  14 |  84 |  cal |  R |  |  |  12
|  val |  R |  0.251 |  0.566 |  8 |  |  val |  R |  0.343 |  0.744 |  7 |  |  val |  R |  0.334 |  0.876 |  7
8 |  cal |  R |  |  |  7 |  60 |  cal |  R |  |  |  9 |  85 |  cal |  R |  |  |  15
|  val |  R |  0.287 |  0.528 |  6 |  |  val |  R |  0.475 |  1.135 |  4 |  |  val |  R |  0.364 |  0.701 |  14
9 |  cal |  R |  |  |  17 |  61 |  cal |  R |  |  |  27 |  87 |  cal |  R |  |  |  11
|  val |  R |  0.579 |  1.739 |  7 |  |  val |  R |  0.518 |  0.922 |  8 |  |  val |  R |  0.274 |  0.572 |  8
10 |  cal |  R |  |  |  9 |  62 |  cal |  R |  |  |  22 |  90 |  cal |  R |  |  |  19
|  val |  R |  0.294 |  0.508 |  5 |  |  val |  R |  0.34 |  0.58 |  13 |  |  val |  R |  0.688 |  1.693 |  6
12 |  cal |  R |  |  |  12 |  63 |  cal |  R |  |  |  17 |  91 |  cal |  R |  |  |  18
|  val |  R |  0.384 |  0.754 |  5 |  |  val |  R |  0.5 |  0.817 |  9 |  |  val |  R |  0.378 |  0.637 |  6
13 |  cal |  R |  |  |  21 |  64 |  cal |  R |  |  |  21 |  92 |  cal |  R |  |  |  14
|  val |  R |  0.375 |  0.664 |  8 |  |  val |  R |  0.372 |  0.6 |  14 |  |  val |  R |  0.261 |  0.486 |  12
14 |  cal |  R |  |  |  16 |  65 |  cal |  R |  |  |  16 |  93 |  cal |  R |  |  |  14
|  val |  R |  0.416 |  0.789 |  10 |  |  val |  R |  0.499 |  0.967 |  14 |  |  val |  R |  0.304 |  0.549 |  8
15 |  cal |  R |  |  |  17 |  66 |  cal |  R |  |  |  21 |  94 |  cal |  R |  |  |  10
|  val |  R |  0.407 |  0.981 |  14 |  |  val |  R |  0.266 |  0.566 |  13 |  |  val |  R |  0.443 |  0.813 |  7
16 |  cal |  L |  |  |  2 |  67 |  cal |  R |  |  |  22 |  95 |  cal |  R |  |  |  13
|  |  R |  |  |  20 |  |  val |  R |  0.441 |  1.301 |  9 |  |  val |  R |  0.188 |  0.422 |  5
|  val |  L |  0.515 |  0.92 |  2 |  68 |  cal |  R |  |  |  11 |  96 |  cal |  R |  |  |  21
|  |  R |  0.743 |  1.425 |  13 |  |  val |  R |  0.285 |  0.493 |  6 |  |  val |  R |  0.278 |  0.482 |  14
17 |  cal |  R |  |  |  14 |  69 |  cal |  R |  |  |  17 |  97 |  cal |  R |  |  |  16
|  val |  R |  0.315 |  0.61 |  13 |  |  val |  R |  0.508 |  1.021 |  9 |  |  val |  R |  0.282 |  0.515 |  12
18 |  cal |  R |  |  |  13 |  70 |  cal |  R |  |  |  15 |  98 |  cal |  R |  |  |  16
|  val |  R |  0.409 |  0.774 |  10 |  |  val |  R |  0.305 |  0.528 |  12 |  |  val |  R |  0.454 |  0.906 |  7
19 |  cal |  R |  |  |  14 |  71 |  cal |  R |  |  |  16 |  99 |  cal |  R |  |  |  15
|  val |  R |  0.409 |  0.756 |  10 |  |  val |  R |  0.291 |  0.52 |  12 |  |  val |  R |  0.355 |  0.725 |  6
20 |  cal |  R |  |  |  14 |  72 |  cal |  R |  |  |  8 |  100 |  cal |  R |  |  |  9
|  val |  R |  0.369 |  0.67 |  10 |  |  val |  R |  0.267 |  0.532 |  6 |  |  val |  R |  0.412 |  0.799 |  9
22 |  cal |  R |  |  |  16 |  73 |  cal |  R |  |  |  8 |  101 |  cal |  R |  |  |  17
|  val |  R |  0.574 |  1.351 |  9 |  |  val |  R |  0.316 |  0.616 |  7 |  |  val |  R |  0.53 |  1.069 |  14
23 |  cal |  R |  |  |  15 |  74 |  cal |  R |  |  |  14 |  102 |  cal |  R |  |  |  18
|  val |  R |  0.426 |  0.784 |  8 |  |  val |  R |  0.353 |  0.581 |  7 |  |  val |  R |  0.522 |  1.163 |  15
30 |  cal |  R |  |  |  7 |  75 |  cal |  R |  |  |  10 |  103 |  cal |  R |  |  |  15
|  val |  R |  0.34 |  0.613 |  3 |  |  val |  R |  0.318 |  0.48 |  6 |  |  val |  R |  0.5 |  0.993 |  10
31 |  cal |  R |  |  |  21 |  76 |  cal |  R |  |  |  19 |  104 |  cal |  R |  |  |  13
|  val |  R |  0.532 |  1.002 |  9 |  |  val |  R |  0.342 |  0.793 |  10 |  |  val |  R |  0.395 |  0.982 |  10
32 |  cal |  R |  |  |  9 |  77 |  cal |  R |  |  |  8 |  105 |  cal |  R |  |  |  14
|  val |  R |  0.267 |  0.503 |  6 |  |  val |  R |  0.453 |  0.82 |  7 |  |  val |  R |  0.438 |  0.818 |  9

## References

  *   * Ableitner [2014] MolBioAbleitner, O.  2014\.  Einführung in die Molekularbiologie. Basiswissen für das Arbeiten im Labor Einführung in die Molekularbiologie. Basiswissen für das Arbeiten im Labor.  WiesbadenSpringer.
  * Acartürk . [2023] Acartrk2023Acartürk, C., Özkan, A., Pekçetin, T.N., Ormanoğlu, Z. Kırkıcı, B.  2023\.  TURead: An eye movement dataset of Turkish reading TURead: An eye movement dataset of Turkish reading.  Behavior Research Methods1-14,  https://doi.org/10.3758/s13428-023-02120-6
  * Ahn . [2020] ahn2020SATAhn, S., Kelton, C., Balasubramanian, A. Zelinsky, G.  2020\.  Towards Predicting Reading Comprehension From Gaze Behavior Towards predicting reading comprehension from gaze behavior.  ACM Symposium on Eye Tracking Research and Applications. ACM Symposium on Eye Tracking Research and Applications.  Association for Computing Machinery.
  * Barrett . [2016] barrett-etal-2016-pos-tagBarrett, M., Bingel, J., Keller, F. Søgaard, A.  2016\.  Weakly Supervised Part-of-speech Tagging Using Eye-tracking Data Weakly supervised part-of-speech tagging using eye-tracking data.  Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) ( 579–584).  Association for Computational Linguistics.
  * Bednarik . [2020] emip2020Bednarik, R., Busjahn, T., Gibaldi, A., Ahadi, A., Bielikova, M., Crosby, M.van der Linde, I.  2020\.  EMIP: The eye movements in programming dataset EMIP: The eye movements in programming dataset.  Science of Computer Programming1981–11,  https://doi.org/10.1016/j.scico.2020.102520
  * Beinborn Hollenstein [2023] beinborn2023cognitiveBeinborn, L. Hollenstein, N.  2023\.  Cognitive Plausibility in Natural Language Processing Cognitive Plausibility in Natural Language Processing.  Springer Nature.
  * Berzak . [2022] celer2022Berzak, Y., Nakamura, C., Smith, A., Weng, E., Katz, B., Flynn, S. Levy, R.  2022\.  CELER: A 365-Participant Corpus of Eye Movements in L1 and L2 English Reading CELER: A 365-Participant Corpus of Eye Movements in L1 and L2 English Reading.  Open Mind1-10,  https://doi.org/10.1162/opmi_a_00054
  * Biedert . [2012] Biedert2012Biedert, R., Hees, J., Dengel, A. Buscher, G.  2012\.  A robust realtime reading-skimming classifier A robust realtime reading-skimming classifier.  Proceedings of ETRA ’12 (2012) Proceedings of ETRA ’12 (2012) ( 123–130).
  * Boujard . [2014] ZellMolbioBoujard, D., Anselme, B., Cullin, C. Raguénès-Nicol, C.  2014\.  Zell- und Molekularbiologie im Überblick Zell- und Molekularbiologie im Überblick (S. Lechowski, ).  BerlinSpringer.
  * Brants . [2004] tiger2004corpusBrants, S., Dipper, S., Eisenberg, P., Hansen, S., König, E., Lezius, W.Uszkoreit, H.  2004\.  TIGER: Linguistic Interpretation of a German Corpus TIGER: Linguistic interpretation of a German corpus.  Journal of Language and Computation597-620,  https://doi.org/10.1007/s11168-004-7431-3
  * Cheri . [2016] cheri-etal-2016-coreferenceCheri, J., Mishra, A. Bhattacharyya, P.  2016\.  Leveraging Annotators’ Gaze Behaviour for Coreference Resolution Leveraging annotators’ gaze behaviour for coreference resolution.  Proceedings of the 7th Workshop on Cognitive Aspects of Computational Language Learning Proceedings of the 7th Workshop on Cognitive Aspects of Computational Language Learning ( 22–26).  Association for Computational Linguistics.
  * Coltheart . [1977] Coltheart1977Coltheart, M., Davelaar, E.J., Jonasson, J.T. Besner, D.  1977\.  Access to the Internal Lexicon Access to the internal lexicon.  S. Dornic (), Attention and Performance Attention and Performance ( VI, 535–555).  HillsdaleLawrence Erlbaum Associates.
  * Cop . [2016] Cop2016gecoCop, U., Dirix, N., Drieghe, D. Duyck, W.  2016\.  Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading.  Behavior Research Methods492602–615,  https://doi.org/10.3758/s13428-016-0734-0
  * Das Digitale Wörterbuch der deutschen Sprache [2016] dwdsDas Digitale Wörterbuch der deutschen Sprache (DWDS) Das digitale Wörterbuch der deutschen Sprache (DWDS).  2016\.  Berlin-Brandenburg Academy of Science.  http://www.dwds.de.
  * Demberg Keller [2008] DembergKeller2008Demberg, V. Keller, F.  2008\.  Data from eyetracking corpora as evidence for theories of syntactic processing complexity Data from eyetracking corpora as evidence for theories of syntactic processing complexity.  Cognition109193–210,  https://doi.org/10.1016/j.cognition.2008.07.008
  * Demberg Keller [2019] demberg2019cognitiveDemberg, V. Keller, F.  2019\.  Cognitive models of syntax and sentence processing Cognitive models of syntax and sentence processing.  Human Language: From Genes and Brains to Behavior293–312,  https://doi.org/10.7551/mitpress/10841.003.0027
  * Demtröder [2010] Demtroeder3Demtröder, W.  2010\.  Experimentalphysik 3: Atome, Moleküle und Festkörper Experimentalphysik 3: Atome, Moleküle und Festkörper (4th ).  BerlinSpringer.
  * Demtröder [20141] Demtroeder2Demtröder, W.  20141\.  Experimentalphysik 2: Elektrizität und Optik Experimentalphysik 2: Elektrizität und Optik (6th ).  BerlinSpringer.
  * Demtröder [20142] Demtroeder4Demtröder, W.  20142\.  Experimentalphysik 4: Kern-, Teilchen- und Astrophysik Experimentalphysik 4: Kern-, Teilchen- und Astrophysik (4th ).  BerlinSpringer.
  * Deng . [2023] deng-etal-2023-preDeng, S., Prasse, P., Reich, D., Scheffer, T. Jäger, L.  2023\.  Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding Pre-trained language models augmented with synthetic scanpaths for natural language understanding.  Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing ( 6500–6507).  Association for Computational Linguistics.
  * Devlin . [2019] devlin-etal-2019-bertDevlin, J., Chang, MW., Lee, K. Toutanova, K.  2019\.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding BERT: Pre-training of deep bidirectional transformers for language understanding.  Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) ( 4171–4186).  Association for Computational Linguistics.
  * Frank Aumeistere [2023] Frank2023Frank, S.L. Aumeistere, A.  2023\.  An eye-tracking-with-EEG coregistration corpus of narrative sentences An eye-tracking-with-EEG coregistration corpus of narrative sentences.  Language Resources and Evaluation1–17,  https://doi.org/10.1007/s10579-023-09684-x
  * Frank . [2013] frank2013readingFrank, S.L., Fernandez Monsalve, I., Thompson, R.L. Vigliocco, G.  2013\.  Reading time data for evaluating broad-coverage models of English sentence processing Reading time data for evaluating broad-coverage models of English sentence processing.  Behavior research methods451182–1190,  https://doi.org/10.3758/s13428-012-0313-y
  * Futrell . [2017] futrell2017natural-storiesFutrell, R., Gibson, E., Tily, H., Blank, I., Vishnevetsky, A., Piantadosi, S.T. Fedorenko, E.  2017\.  The Natural Stories Corpus The natural stories corpus.  CoRRabs/1708.05763,  https://doi.org/10.48550/arXiv.1708.05763
  * Goodkind Bicknell [2018] goodkind-bicknell-2018-predictiveGoodkind, A. Bicknell, K.  2018\.  Predictive power of word surprisal for reading times is a linear function of language model quality Predictive power of word surprisal for reading times is a linear function of language model quality.  Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018) Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018) ( 10–18).  Association for Computational Linguistics.
  * Graw [2015] GenetikGraw, J.  2015\.  Genetik Genetik (6th ).  BerlinSpringer.
  * Griffith . [2021] Griffith2021gazebaseGriffith, H., Lohr, D., Abdulin, E. Komogortsev, O.  2021\.  GazeBase, a large-scale, multi-stimulus, longitudinal eye movement dataset GazeBase, a large-scale, multi-stimulus, longitudinal eye movement dataset.  Scientific Data811–9,  https://doi.org/10.1038/s41597-021-00959-y
  * Hamilton Huth [2020] hamilton2020REvolutionHamilton, L.S. Huth, A.G.  2020\.  The revolution will not be controlled: natural stimuli in speech neuroscience The revolution will not be controlled: natural stimuli in speech neuroscience.  Language, Cognition and Neuroscience355573-582,  https://doi.org/10.1080/23273798.2018.1499946
  * Heister . [2011] Heister2011Heister, J., Würzner, KM., Bubenzer, J., Pohl, E., Hanneforth, T., Geyken, A. Kliegl, R.  2011\.  dlexDB – eine lexikalische Datenbank für die psychologische und linguistische Forschung dlexDB – eine lexikalische Datenbank für die psychologische und linguistische Forschung.  Psychologische Rundschau62110–20,  https://doi.org/10.1026/0033-3042/a000029 http://www.dlexdb.de/
  * Hollenstein, Barrett Beinborn [2020] hollenstein-etal-2020-practicesHollenstein, N., Barrett, M. Beinborn, L.  2020\.  Towards Best Practices for Leveraging Human Language Processing Signals for Natural Language Processing Towards best practices for leveraging human language processing signals for natural language processing.  Proceedings of the Second Workshop on Linguistic and Neurocognitive Resources Proceedings of the Second Workshop on Linguistic and Neurocognitive Resources ( 15–27).  European Language Resources Association.
  * Hollenstein . [2022] hollenstein-etal-2022-copenhagenHollenstein, N., Barrett, M. Björnsdóttir, M.  2022\.  The Copenhagen Corpus of Eye Tracking Recordings from Natural Reading of Danish Texts The copenhagen corpus of eye tracking recordings from natural reading of Danish texts.  Proceedings of the Thirteenth Language Resources and Evaluation Conference Proceedings of the Thirteenth Language Resources and Evaluation Conference ( 1712–1720).  European Language Resources Association.
  * Hollenstein . [2021] hollenstein2021Hollenstein, N., Pirovano, F., Zhang, C., Jäger, L.A. Beinborn, L.  2021\.  Multilingual Language Models Predict Human Reading Behavior Multilingual language models predict human reading behavior.  Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( 106–123).  Association for Computational Linguistics.
  * Hollenstein . [2018] Hollenstein2018zuco1Hollenstein, N., Rotsztejn, J., Troendle, M., Pedroni, A., Zhang, C. Langer, N.  2018\.  ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading.  Scientific Data511–13,  https://doi.org/10.1038/sdata.2018.291
  * Hollenstein, Troendle . [2020] hollenstein-etal-2020-zucoHollenstein, N., Troendle, M., Zhang, C. Langer, N.  2020\.  ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation ZuCo 2.0: A dataset of physiological recordings during natural reading and annotation.  Proceedings of the Twelfth Language Resources and Evaluation Conference Proceedings of the Twelfth Language Resources and Evaluation Conference ( 138–146).  European Language Resources Association.
  * Hollenstein Zhang [2019] hollenstein-zhang-2019-entityHollenstein, N. Zhang, C.  2019\.  Entity Recognition at First Sight: Improving NER with Eye Movement Information Entity recognition at first sight: Improving NER with eye movement information.  Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) ( 1–10).  Association for Computational Linguistics.
  * Husain . [2014] Husain-Vasishth-Srinivasan-2014-hindiHusain, S., Vasishth, S. Srinivasan, N.  2014\.  Integration and prediction difficulty in Hindi sentence comprehension: Evidence from an eye-tracking corpus Integration and prediction difficulty in hindi sentence comprehension: Evidence from an eye-tracking corpus.  Journal of Eye Movement Research821–12,  https://doi.org/10.16910/jemr.8.2.3
  * Joshi . [2014] joshi-etal-2014-measuringJoshi, A., Mishra, A., Senthamilselvan, N. Bhattacharyya, P.  2014\.  Measuring Sentiment Annotation Complexity of Text Measuring sentiment annotation complexity of text.  Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) ( 36–41).  Association for Computational Linguistics.
  * Keller [2010] keller-2010-cogn-plausibilityKeller, F.  2010\.  Cognitively Plausible Models of Human Language Processing Cognitively plausible models of human language processing.  Proceedings of the ACL 2010 Conference Short Papers Proceedings of the ACL 2010 Conference Short Papers ( 60–67).  Association for Computational Linguistics.
  * Kennedy . [2003] dundeeKennedy, A., Hill, R. Pynte, J.  2003\.  The Dundee Corpus. The Dundee Corpus.  Proceedings of the 12th European conference on eye movement.
  * Kennedy . [2013] kennedy2013frequencyKennedy, A., Pynte, J., Murray, W.S. Paul, SA.  2013\.  Frequency and predictability effects in the Dundee Corpus: An eye movement analysis Frequency and predictability effects in the Dundee Corpus: An eye movement analysis.  Quarterly Journal of Experimental Psychology663601–618,  https://doi.org/10.1080/17470218.2012.676054
  * Kitaev . [2019] kitaev-etal-2019-multilingualKitaev, N., Cao, S. Klein, D.  2019\.  Multilingual Constituency Parsing with Self-Attention and Pre-Training Multilingual constituency parsing with self-attention and pre-training.  Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics ( 3499–3505).  Association for Computational Linguistics.
  * Kitaev Klein [2018] kitaev-klein-2018-constituencyKitaev, N. Klein, D.  2018\.  Constituency Parsing with a Self-Attentive Encoder Constituency parsing with a self-attentive encoder.  Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) ( 2676–2686).  Association for Computational Linguistics.
  * Kliegl . [2004] kliegl2004pscKliegl, R., Grabner, E., Rolfs, M. Engbert, R.  2004\.  Length, frequency, and predictability effects of words on eye movements in reading Length, frequency, and predictability effects of words on eye movements in reading.  European Journal of Cognitive Psychology161-2262-284,  https://doi.org/10.1080/09541440340000213
  * Kliegl . [2006] PSCKliegl, R., Nuthmann, A. Engbert, R.  2006\.  Tracking the mind during reading: The influence of past, present, and future words on fixation durations Tracking the mind during reading: The influence of past, present, and future words on fixation durations.  Journal of Experimental Psychology: General135112–35,  https://doi.org/10.1037/0096-3445.135.1.12
  * Krakowczyk, Prasse . [2023] Krakowczyk2023Krakowczyk, D.G., Prasse, P., Reich, D.R., Lapuschkin, S., Scheffer, T. Jäger, L.A.  2023\.  Bridging the Gap: Gaze Events as Interpretable Concepts to Explain Deep Neural Sequence Models Bridging the gap: Gaze events as interpretable concepts to explain deep neural sequence models.  Proceedings of the 2023 Symposium on Eye Tracking Research and Applications. Proceedings of the 2023 Symposium on Eye Tracking Research and Applications.  Association for Computing Machinery.
  * Krakowczyk, Reich . [2023] pymovementsKrakowczyk, D.G., Reich, D.R., Chwastek, J., Jakobi, D.N., Prasse, P., Süss, A.Jäger, L.A.  2023\.  pymovements: A Python Package for Processing Eye Movement Data pymovements: A python package for processing eye movement data.  2023 Symposium on Eye Tracking Research and Applications. 2023 Symposium on Eye Tracking Research and Applications.  Association for Computing Machinery.
  * Kuperman . [2023] kuperman-2023-mecol2Kuperman, V., Siegelman, N., Schroeder, S., Acartürk, C., Alexeeva, S., Amenta, S.et al.  2023\.  Text reading in English as a second language: Evidence from the Multilingual Eye-Movements Corpus Text reading in english as a second language: Evidence from the multilingual eye-movements corpus.  Studies in Second Language Acquisition4513–37,  https://doi.org/10.1017/S0272263121000954
  * Laurinavichyute . [2018] Laurinavichyute2018RSCLaurinavichyute, A.K., Sekerina, I.A., Alexeeva, S., Bagdasaryan, K. Kliegl, R.  2018\.  Russian Sentence Corpus: Benchmark measures of eye movements in reading in Russian Russian sentence corpus: Benchmark measures of eye movements in reading in russian.  Behavior Research Methods5131161–1178,  https://doi.org/10.3758/s13428-018-1051-6
  * Leal . [2022] Leal2022rastrosLeal, S.E., Lukasova, K., Carthery-Goulart, M.T. Aluísio, S.M.  2022\.  RastrOS Project: Natural Language Processing contributions to the development of an eye-tracking corpus with predictability norms for Brazilian Portuguese RastrOS project: Natural language processing contributions to the development of an eye-tracking corpus with predictability norms for Brazilian Portuguese.  Language Resources and Evaluation5641333–1372,  https://doi.org/10.1007/s10579-022-09609-0
  * Levenshtein [1966] Levenshtein1966Levenshtein, V.I.  1966\.  Binary Codes Capable of Correcting Deletions, Insertions, and Reversals Binary codes capable of correcting deletions, insertions, and reversals.  Soviet Physics-Doklady108845–848,
  * Li . [2018] li2018reading-attentionLi, X., Liu, Y., Mao, J., He, Z., Zhang, M. Ma, S.  2018\.  Understanding Reading Attention Distribution during Relevance Judgement Understanding reading attention distribution during relevance judgement.  Proceedings of the 27th ACM International Conference on Information and Knowledge Management Proceedings of the 27th ACM International Conference on Information and Knowledge Management ( 733–742).  Association for Computing Machinery.
  * Long . [2017] long-etal-2017-sentimentLong, Y., Lu, Q., Xiang, R., Li, M. Huang, CR.  2017\.  A Cognition Based Attention Model for Sentiment Analysis A cognition based attention model for sentiment analysis.  Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing ( 462–471).  Association for Computational Linguistics.
  * Luke Christianson [2017] Luke2017provoLuke, S.G. Christianson, K.  2017\.  The Provo Corpus: A large eye-tracking corpus with predictability norms The Provo corpus: A large eye-tracking corpus with predictability norms.  Behavior Research Methods502826–833,  https://doi.org/10.3758/s13428-017-0908-4
  * Mak Willems [2019] mak2019mentalsimulationMak, M. Willems, R.M.  2019\.  Mental simulation during literary reading: Individual differences revealed with eye-tracking Mental simulation during literary reading: Individual differences revealed with eye-tracking.  Language, Cognition and Neuroscience344511-535,  https://doi.org/10.1080/23273798.2018.1552007
  * Makowski . [2019] MakowskiECML2018Makowski, S., Jäger, L.A., Abdelwahab, A., Landwehr, N. Scheffer, T.  2019\.  A discriminative model for identifying readers and assessing text comprehension from eye movements A discriminative model for identifying readers and assessing text comprehension from eye movements.  Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2018 Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2018 ( 11051, 209–225).  Springer Nature.
  * Malmaud . [2020] malmaud-etal-2020-bridgingMalmaud, J., Levy, R. Berzak, Y.  2020\.  Bridging Information-Seeking Human Gaze and Machine Reading Comprehension Bridging information-seeking human gaze and machine reading comprehension.  Proceedings of the 24th Conference on Computational Natural Language Learning Proceedings of the 24th Conference on Computational Natural Language Learning ( 142–152).  Association for Computational Linguistics.
  * Mathias . [2018] mathias-etal-2018-eyesMathias, S., Kanojia, D., Patel, K., Agrawal, S., Mishra, A. Bhattacharyya, P.  2018\.  Eyes are the Windows to the Soul: Predicting the Rating of Text Quality Using Gaze Behaviour Eyes are the windows to the soul: Predicting the rating of text quality using gaze behaviour.  Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) ( 2352–2362).  Association for Computational Linguistics.
  * Mathias . [2020] mathias-etal-2020-happyMathias, S., Murthy, R., Kanojia, D., Mishra, A. Bhattacharyya, P.  2020\.  Happy Are Those Who Grade without Seeing: A Multi-Task Learning Approach to Grade Essays Using Gaze Behaviour Happy are those who grade without seeing: A multi-task learning approach to grade essays using gaze behaviour.  Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing ( 858–872).  Association for Computational Linguistics.
  * Minixhofer [2020] Minixhofer_GerPT2_German_large_2020Minixhofer, B.  2020\.  GerPT2: German large and small versions of GPT2. GerPT2: German large and small versions of GPT2.  https://github.com/bminixhofer/gerpt2
  * Mishra . [2016] Mishra-Kanojia-Bhattacharyya-2016Mishra, A., Kanojia, D. Bhattacharyya, P.  2016\.  Predicting Readers’ Sarcasm Understandability by Modeling Gaze Behavior Predicting readers’ sarcasm understandability by modeling gaze behavior.  Proceedings of the AAAI Conference on Artificial Intelligence3011–7,  https://doi.org/10.1609/aaai.v30i1.9884
  * Mishra . [20171] mishra-2017-sentimentMishra, A., Kanojia, D., Nagar, S., Dey, K. Bhattacharyya, P.  20171\.  Leveraging Cognitive Features for Sentiment Analysis Leveraging cognitive features for sentiment analysis.  CoRRabs/1701.05581156–166,  https://doi.org/10.18653/v1/K16-1016 1701.05581
  * Mishra . [20172] mishra2017scanpathMishra, A., Kanojia, D., Nagar, S., Dey, K. Bhattacharyya, P.  20172\.  Scanpath Complexity: Modeling Reading Effort Using Gaze Information Scanpath complexity: Modeling reading effort using gaze information.  Proceedings of the AAAI Conference on Artificial Intelligence3111–8,  https://doi.org/10.1609/aaai.v31i1.11159
  * Nastase . [2020] NASTASE2020117254Nastase, S.A., Goldstein, A. Hasson, U.  2020\.  Keep it real: rethinking the primacy of experimental control in cognitive neuroscience Keep it real: rethinking the primacy of experimental control in cognitive neuroscience.  NeuroImage222117254,  https://doi.org/10.1016/j.neuroimage.2020.117254
  * Notaro Diamond [2018] notaro2018duolingoNotaro, G.M. Diamond, S.G.  2018\.  Simultaneous EEG, eye-tracking, behavioral, and screen-capture data during online German language learning Simultaneous EEG, eye-tracking, behavioral, and screen-capture data during online German language learning.  Data in Brief211937-1943,  https://doi.org/10.1016/j.dib.2018.11.044
  * Obaidellah . [2018] Obaidellah-2019-code-readingObaidellah, U., Al Haek, M. Cheng, P.CH.  2018\.  A Survey on the Usage of Eye-Tracking in Computer Programming A survey on the usage of eye-tracking in computer programming.  ACM Computing Surveys5111–58,  https://doi.org/10.1145/3145904
  * Pan . [2021] Pan2021BSCPan, J., Yan, M., Richter, E.M., Shu, H. Kliegl, R.  2021\.  The Beijing Sentence Corpus: A Chinese sentence corpus with eye movement data and predictability norms The Beijing Sentence Corpus: A Chinese sentence corpus with eye movement data and predictability norms.  Behavior Research Methods5441989–2000,  https://doi.org/10.3758/s13428-021-01730-2
  * Parker . [2017] parker2017passage-readingParker, A.J., Kirkby, J.A. Slattery, T.J.  2017\.  Predictability effects during reading in the absence of parafoveal preview Predictability effects during reading in the absence of parafoveal preview.  Journal of Cognitive Psychology298902-911,  https://doi.org/10.1080/20445911.2017.1340303
  * Prasse . [2023] prasse2023spygan-reviewPrasse, P., Reich, D.R., Makowski, S., Scheffer, T. Jäger, L.A.  2023\.  Evaluating the utility of synthetic eye movement data for pre-training neural networks. Evaluating the utility of synthetic eye movement data for pre-training neural networks.  Under review
  * Radford . [2019] radford2019languageRadford, A., Wu, J., Child, R., Luan, D., Amodei, D. Sutskever, I.  2019\.  Language Models are Unsupervised Multitask Learners. Language models are unsupervised multitask learners.  https://api.semanticscholar.org/CorpusID:160025533
  * Rayner [1998] Rayner1998Rayner, K.  1998\.  Eye movements in reading and information processing: 20 years of research Eye movements in reading and information processing: 20 years of research.  Psychological Bulletin1243372–422,  https://doi.org/10.1037/0033-2909.124.3.372
  * Rayner Carroll [2018] raynerCarroll2018Rayner, K. Carroll, P.J.  2018\.  Eye movements and reading comprehension Eye movements and reading comprehension.  New Methods in Reading Comprehension Research New Methods in Reading Comprehension Research ( 129–150).  Routledge.
  * Ribeiro . [2023] ribeiro2023webqamgazeRibeiro, T., Brandl, S., Søgaard, A. Hollenstein, N.  2023\.  WebQAmGaze: A Multilingual Webcam Eye-Tracking-While-Reading Dataset. WebQAmGaze: A multilingual webcam eye-tracking-while-reading dataset.  arXiv preprint
  * Schiller . [1999] STTSguidelinesSchiller, A., Teufel, S. Stöckert, C.  1999\.  Guidelines für das Tagging deutscher Textcorpora mit STTS (Kleines und großes Tagset). Guidelines für das Tagging deutscher Textcorpora mit STTS (Kleines und großes Tagset).  www.sfs.uni-tuebingen.de/resources/stts-1999.pdf.
  * Siegelman . [2022] Siegelman2022Siegelman, N., Schroeder, S., Acartürk, C., Ahn, HD., Alexeeva, S., Amenta, S.Kuperman, V.  2022\.  Expanding horizons of cross-linguistic research on reading: The Multilingual Eye-movement Corpus (MECO) Expanding horizons of cross-linguistic research on reading: The Multilingual Eye-movement Corpus (MECO).  Behavior Research Methods5462843–2863,  https://doi.org/10.3758/s13428-021-01772-6
  * Sood . [2020] sood2020interpretingSood, E., Tannert, S., Frassinelli, D., Bulling, A. Vu, N.T.  2020\.  Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension Interpreting attention models with human visual attention in machine reading comprehension.  Proceedings of the 24th Conference on Computational Natural Language Learning Proceedings of the 24th conference on computational natural language learning ( 12–25).
  * SR Research Ltd [2010] eyelink-install-guideSR Research Ltd  2010\.  EyeLink 1000 Installation Guide EyeLink 1000 Installation Guide [].  Mississauga, Canada.  http://sr-research.jp/support/files/EyeLinkCLsetupguide.pdf Document version 1.5.2
  * SR Research Ltd. [2011] dataviewerSR Research Ltd.  2011\.  EyeLink Data Viewer user’s manual EyeLink Data Viewer user’s manual [].  Mississauga, Canada.  Document version 1.11.1
  * Sui . [2022] Sui2022gecocnSui, L., Dirix, N., Woumans, E. Duyck, W.  2022\.  GECO-CN: Ghent Eye-tracking COrpus of sentence reading for Chinese-English bilinguals GECO-CN: Ghent eye-tracking COrpus of sentence reading for Chinese-English bilinguals.  Behavior Research Methods5562743–2763,  https://doi.org/10.3758/s13428-022-01931-3
  * Takmaz . [2020] takmaz-etal-2020-img-captionTakmaz, E., Pezzelle, S., Beinborn, L. Fernández, R.  2020\.  Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze Generating image descriptions via sequential cross-modal alignment guided by human gaze.  Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) ( 4664–4677).  Association for Computational Linguistics.
  * TIGER Project [2003] tiger2003schemaTIGER Project  2003\.  TIGER Annotationsschema TIGER annotationsschema [].  Universität des Saarlands, Universität Stuttgart, Universität Potsdam.
  * Touvron . [2023] touvron2023llamaTouvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y.Scialom, T.  2023\.  Llama 2: Open Foundation and Fine-Tuned Chat Models. Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288
  * Townsend . [2003] OekologieTownsend, C.R., Begon, M. Harper, J.L.  2003\.  Ökologie Ökologie (J. Steidle, F. Thomas, B. Stadler, U. Hoffmeister T. Hoffmeister, ).  BerlinSpringer.
  * Vieira [2020] vieira2020rastroVieira, J.M.M.  2020\.  The Brazilian Portuguese eye tracking corpus with a predictability study focusing on lexical and partial prediction The Brazilian Portuguese eye tracking corpus with a predictability study focusing on lexical and partial prediction Universidade Federal do Ceará.  http://repositorio.ufc.br/handle/riufc/55798
  * E. Wilcox . [2023] wilcox-etal-2023-languageWilcox, E., Meister, C., Cotterell, R. Pimentel, T.  2023\.  Language Model Quality Correlates with Psychometric Predictive Power in Multiple Languages Language model quality correlates with psychometric predictive power in multiple languages.  Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing ( 7503–7511).  Association for Computational Linguistics.
  * E.G. Wilcox . [2020] wilcox-etal:2020-on-the-predictive-powerWilcox, E.G., Gauthier, J., Hu, J., Qian, P. Levy, R.P.  2020\.  On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior On the predictive power of neural language models for human real-time comprehension behavior.  Proceedings of the 42nd Annual Meeting of the Cognitive Science Society Proceedings of the 42nd Annual Meeting of the Cognitive Science Society ( 1707–1713).