[affiliation=1,2]Sri HarshaDumpala [affiliation=3]KaterinaDikaios
[affiliation=1]AbrahamNunes [affiliation=1,2]FrankRudzicz
[affiliation=1,4]RudolfUher [affiliation=1,2]SageevOore

# Self-Supervised Embeddings for Detecting Individual Symptoms of Depression

###### Abstract

Depression, a prevalent mental health disorder impacting millions globally,
demands reliable assessment systems. Unlike previous studies that focus solely
on either detecting depression or predicting its severity, our work identifies
individual symptoms of depression while also predicting its severity using
speech input. We leverage self-supervised learning (SSL)-based speech models
to better utilize the small-sized datasets that are frequently encountered in
this task. Our study demonstrates notable performance improvements by
utilizing SSL embeddings compared to conventional speech features. We compare
various types of SSL pretrained models to elucidate the type of speech
information (semantic, speaker, or prosodic) that contributes the most in
identifying different symptoms. Additionally, we evaluate the impact of
combining multiple SSL embeddings on performance. Furthermore, we show the
significance of multi-task learning for identifying depressive symptoms
effectively.

###### keywords:

Self-supervised learning, depression, depressive symptoms, multi-task
learning, semantic, speaker, prosody

##  1 Introduction

Major depressive disorder, commonly known as depression, is the most prevalent
mental health disorder, and a leading cause of global disability [1]. Yet, it
remains largely under-detected and under-treated [2]. Therefore, accurately
identifying and treating depression is a significant public health priority
aimed at reducing its prevalence. However, scarcity of trained clinicians and
other resources hinder this objective. While deep learning-based mental health
assessment systems have shown promise in tasks including patient screening and
monitoring, further research is necessary to develop more informative and
interpretable deep learning models for widespread deployment in clinical
settings.

Recent research has demonstrated the efficacy of utilizing natural speech as a
potentially inexpensive and easily scalable indicator for assessing depression
[3, 4, 5, 6, 7]. Previous studies primarily focused on either detecting
depression [8, 9, 10] or predicting its severity [11, 12, 13, 14]. Potential
vocal biomarkers for depression identified in these studies encompass a
spectrum of traditional acoustic features, including prosodic elements (such
as pitch and speech rate), spectral characteristics (like Mel-frequency
cepstral coefficients and formant frequencies), and glottal attributes
(related to vocal fold behaviour) [4, 9, 6], as well as neural embeddings from
pretrained models like $x$-vectors, wav2vec 2.0, and HuBERT [14, 15, 16].
However, these studies often operate under the assumption that of all the
measurable aspects of speech, depression severity is the most significant and
influential [17]. Yet, merely identifying or estimating depression severity
may prove inadequate compared to the clinical setting of monitoring the core
subset of depressive symptoms for effective treatment outcomes [18]. In this
study, we adopt a clinically oriented approach to depression assessment –
aiming to detect depressive symptoms while also predicting overall depression
severity. This method also offers an implicit explanation for the decisions
made by the automated models.

Only a few studies have explored the task of identifying individual symptoms
of depression using speech as input [19, 20, 17]. Fara et al. [19, 17] used
acoustic features such as eGeMAPS [21] and prosodic features extracted from
spontaneous speech (in English) to train random forest classifiers for
detecting individual symptoms of depression based on the self-reported Patient
Health Questionnaire (PHQ-8) [22]. Similarly, Takano et al. [20] used
spectral, prosodic, and glottal features extracted from read speech (in
Japanese) to train decision tree models for detecting individual symptoms of
depression based on the clinician-rated HAM-D scale [23]. These works used
conventional speech features to train simple machine learning models for
detecting individual symptoms of depression. In this work, we use neural
embeddings extracted from deep learning models to detect individual symptoms
of depression based on the Montgomery and Åsberg Depression Rating Scale
(MADRS) [24], which is considered a gold standard for clinical assessment of
depression severity.

One of the major hindrances in applying deep learning techniques to the
medical domain, including depression assessment, is insufficient training data
[25]. To address the challenge of low-resourced data, we leverage SSL-based
training to exploit the availability of large amounts of unlabeled speech
data. Previous studies have employed SSL-based speech models to detect
depression [26, 27, 15]. We extend these approaches by leveraging SSL-based
speech models to both (A) identify individual symptoms of depression and (B)
predict depression severity. Specifically, we analyze several SSL-based speech
models pretrained with different objective functions such as masked language
modeling (MLM), contrastive learning, and speaker-invariant training [28, 29,
30]. The choice of the pre-training objective function defines the information
encoded by the SSL speech models, such as semantic, speaker, and prosodic
information. Using these models, we assess the significance of each
information type in identifying individual symptoms of depression.
Additionally, we examine the impact of combining embeddings obtained from
different SSL models in identifying individual symptoms of depression.

(a) Single SSL model

(b) Combining multiple SSL models

Figure 1: Schematic diagram of the symptom detection model using (a) single
and (b) multiple SSL models (N models). In this work, N = 2 or 3. Multi-task
setting will have 10 symptom detection heads in the output layer ($S_{i}$ for
$1\leq i\leq 10$) along with one regression (Reg) head. In single-task
setting, there will be only one head (specific-symptom detection or
regression).

##  2 SSL-models for Analysis

In this paper, we analyze several SSL-based speech models pretrained using
different objective functions. The type of speech information, whether
semantic, speaker, or prosodic, encoded by the SSL model depends on the pre-
training objective function. Different pre-training objectives used to train
SSL-based speech models include masked language modeling (MLM), contrastive
pre-training, speaker-invariant training, and student-teacher training with
cross-entropy loss [28, 29, 30, 31]. Below, we explain the SSL models that we
use in this paper.

  * •

HuBERT [28] was pretrained using MLM which uses an offline clustering step to
align target labels. This model predominantly encodes semantic information in
speech.

  * •

WavLM [32] was pretrained using MLM and denoising objectives. WavLM
predominantly encodes semantics with some speaker-specific information. Here,
we use the WavLM base model.

  * •

BEATS [33] uses an iterative framework to optimize an acoustic tokenizer and
an audio SSL model. The audio SSL model is optimized using MLM where the
discrete labels are generated by the acoustic tokenizer. This objective allows
BEATS to learn semantic-rich embeddings.

  * •

ContentVec [30] uses a speaker-invariant training objective to suppress
speaker information contained in the pretrained HuBERT model. ContentVec
encodes semantic information while discarding speaker-specific information.

  * •

RDINO [31] uses a regularized distillation with no labels (DINO) framework to
pre-train models to encode speaker-specific information.

  * •

AudioMAE [34] is an encoder-decoder model pretrained using MLM with a very
high masking ratio – around 80% of frames masked. The pre-training task of
reconstruction makes the model encode semantic, speaker, and prosodic
information.

  * •

BYOL-Audio [29] was pretrained by maximizing the cosine similarity between
multiple augmented versions of the same speech signal. This model encodes the
global information contained in speech, i.e., mainly speaker and prosodic.

Note that WavLM, BEATS, HuBERT, ContentVec, and RDINO take raw speech as input
whereas BYOL-Audio and AudioMAE take spectrograms. As shown in Figure 1, we
freeze all the weights of these pretrained models, and train only the feed-
forward and output layers to detect the individual symptoms of depression. We
analyze SSL models using two different architectures: (1) single SSL model –
here we use the embeddings of a single SSL model (Figure 1(a)), and (2)
Multiple SSL models – here we combine embeddings of multiple SSL models
(Figure 1(b)). Moreover, we evaluate each of this architecture in single-task
and multi-task settings. For the multi-task setting, the output layer consists
of $10$ classification heads, each with two softmax units to detect each
symptom, and a single linear unit for predicting the overall depression
severity. For the single-task setting, we train a separate model for each
symptom which consists of a single classification head as the output layer.

We also use conventional speech features such as spectrograms, COVAREP [35],
and the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [21] as
baselines. We extract 80-dimensional spectrograms, 88-dimensional eGeMAPS
features using OpenSMILE [36], and 74-dimensional COVAREP features using
COVAREP [35] to train convolutional neural network (CNN) models. The CNNs
trained using conventional speech features consist of two convolutional
layers, each with 100 channels. The kernels have sizes of 3 and 5 for the
first and second layers, respectively. The outputs of the second layer are
then passed through a fully connected layer with $100$ sigmoid linear units
(SiLU) [37]. Subsequently, a symptom-specific classification head with two
softmax units follows the fully connected layer. Separate CNN models are
trained for detecting each symptom. The output layer of the CNN model trained
to predict overall depression severity consists of a linear unit.

Table 1: Individual symptoms of depression as defined in MADRS and their
abbreviations, Abbr.

Symptom | Abbr. | Symptom | Abbr.
---|---|---|---
1) Apparent Sadness | ASad | 6) Concentration Difficulties | ConD
2) Reported Sadness | RSad | 7) Lassitude | Lass
3) Inner Tension | InTen | 8) Inability to Feel | IFeel
4) Reduced Sleep | RSlp | 9) Pessimistic Thoughts | PesT
5) Reduced Appetite | RApp | 10) Suicidal Thoughts | SuiT

Figure 2: Distribution of samples between class-0 (symptom absent) and class-1
(symptom present) for each symptom in the MADRS. Symptom abbreviations are
provided in Table 1. Table 2: Performance of different self-supervised
learning speech models in single task setting: Comparison of the model
performance (for each individual symptom of MADRS). Each cell contains
F-scores in the form: macro F-score $F_{M}$ (Symptom absent $F_{A}$, Symptom
present $F_{P}$). RMSE shown for the regression task in predicting the overall
depression severity score according to MADRS. Best performance (in terms of
$F_{M}$) for each symptom is presented in bold. a-priori refers to the system
that always predicts the output as the majority class i.e., symptom absent.

Symptoms | a-priori | WavLM | HuBERT | BEATS | ContentVec | RDINO | BYOL-Audio | AudioMAE
---|---|---|---|---|---|---|---|---
(1) ASad | 43 (86, 0) | 63 (84, 42) | 60 (79, 41) | 59 (81, 36) | 57 (73, 41) | 62 (81, 43) | 57 (77, 37) | 62 (83, 41)
(2) RSad | 40 (79, 0) | 72 (79, 65) | 71 (74, 68) | 68 (75, 61) | 61 (64, 57) | 69 (72, 66) | 68 (74, 61) | 68 (72, 64)
(3) InTen | 36 (72, 0) | 68 (71, 65) | 68 (70, 66) | 67 (69, 64) | 58 (57, 59) | 69 (70, 68) | 64 (65, 63) | 66 (64, 67)
(4) RSlp | 38 (76, 0) | 66 (75, 57) | 67 (73, 61) | 64 (73, 55) | 58 (61, 55) | 66 (73, 59) | 62 (64, 59) | 65 (73, 56)
(5) RApp | 45 (90, 0) | 47 (84, 10) | 48 (87, 08) | 52 (87, 18) | 41 (75, 07) | 47 (88, 06) | 42 (79, 05) | 53 (88, 17)
(6) ConD | 40 (79, 0) | 70 (80, 59) | 72 (79, 65) | 72 (82, 61) | 65 (70, 61) | 67 (74, 59) | 69 (75, 63) | 69 (77, 61)
(7) Lass | 41 (82, 0) | 68 (77, 58) | 66 (78, 54) | 61 (75, 47) | 57 (64, 51) | 64 (76, 52) | 59 (73, 45) | 65 (78, 52)
(8) IFeel | 43 (86, 0) | 68 (76, 59) | 68 (77, 60) | 63 (79, 46) | 60 (63, 57) | 73 (81, 64) | 61 (77, 45) | 66 (77, 55)
(9) PesT | 41 (82, 0) | 72 (80, 64) | 72 (75, 70) | 69 (73, 64) | 65 (68, 62) | 73 (70, 75) | 74 (81, 66) | 71 (73, 68)
(10) SuiT | 47 (93, 0) | 59 (82, 36) | 60 (83, 36) | 54 (79, 29) | 55 (79, 31) | 63 (82, 43) | 60 (83, 37) | 61 (82, 39)
MADRS(RMSE) | – | 8.53 | 8.44 | 8.92 | 9.46 | 8.76 | 8.85 | 9.05

##  3 Dataset Details

We use an internal speech-based depression dataset which we call the
Clinically-Labelled with INdividual symptoms of Depression (CLIND) dataset.
CLIND consists of speech samples in English language collected from $505$
participants ($351$ female and $154$ male). Each participant was asked to
speak about their experiences from the past few weeks with different prompts.
The three prompts were designed to evoke neutral, positive, and negative
contexts, respectively. The interviewer let the participant speak
uninterrupted and only used a standardized prompt if the participant was
silent for 30 seconds. At least $4$ minutes of speech, including pauses, were
recorded following each prompt, providing a total of approximately 10 minutes
of speech per participant. Trained clinicians used structured interviews to
score each participant's current depression severity on MADRS, which consists
of 10 individual symptoms (see Table 1), with each symptom scored in the range
of $0-6$, and a total score in the range of $0-60$. The range of total MADRS
scores in our dataset range from $0$ to $47$. In this study, we consider a
symptom absent (Class-0) if the score for a symptom ($S_{s}$) is $0$ $\leq$
$S_{s}$ $\leq$ $1$, and a symptom to be present (Class-1) if $2$ $\leq$
$S_{s}$ $\leq$ $6$. The distribution of class-level samples for each symptom
is illustrated in Figure 2. To train and test the models, we segment the
speech samples into $10$-second segments, resulting in a total of $25,471$
segments (we segment after segregating the speech samples into train and test
sets to avoid speaker overlap). We use the sample-level labels, encompassing
both symptoms and overall depression severity scores, for all segments. Test
performance is reported by majority voting across all segments of the test
samples.

It is essential to note that the majority of publicly available depression
datasets were labeled using self-reported scores like the Patient Health
Questionnaire (PHQ-8) [22]. In contrast, our dataset is rated by trained
clinicians using the MADRS, which is regarded as a gold standard for clinical
assessment of depression severity.

Table 3: Comparing performance (in terms of $F_{M}$ ($F_{A},F_{P}$)) of
conventional speech features to HuBERT (SSL-based model). `Spectro' refers to
spectrograms.

Symptoms | Spectro | COVAREP | eGeMAPS | HuBERT
---|---|---|---|---
(1) ASad | 47(51, 43) | 52(61, 43) | 53(63, 43) | 60(79, 41)
(2) RSad | 60(63, 56) | 58(64, 52) | 59(65, 52) | 71(74, 68)
(3) InTen | 54(60, 48) | 55(64, 46) | 58(61, 54) | 68(70, 66)
(4) RSlp | 52(54, 50) | 57(65, 49) | 55(58, 52) | 67(73, 61)
(5) RApp | 42(72, 11) | 41(77, 5) | 42(74, 10) | 48(87, 08)
(6) ConD | 48(55, 41) | 53(62, 44) | 61(72, 50) | 72(79, 65)
(7) Lass | 46(53, 39) | 48(59, 37) | 50(66, 34) | 66(78, 54)
(8) IFeel | 56(66, 46) | 53(70, 36) | 54(72, 37) | 68(77, 60)
(9) PesT | 61(67, 54) | 60(68, 52) | 62(66, 59) | 72(75, 70)
(10) SuiT | 52(72, 32) | 49(77, 21) | 53(73, 32) | 60(83, 36)
MADRS | 10.81 | 10.18 | 9.82 | 8.44
(RMSE)

Table 4: Comparing single-task (ST) and multi-task (MT) learning: Performance
of the model (in terms of macro F-score) for each individual item of MADRS.
MADRS (RMSE) refers to the RMSE error in estimating total depression severity
based on MADRS score.

Symptoms | WavLM | HuBERT | BEATS | ContentVec | RDINO | BYOL-Audio | AudioMAE | {CV,RD,BY} | {HB,RD,BY}
---|---|---|---|---|---|---|---|---|---
| ST | MT | ST | MT | ST | MT | ST | MT | ST | MT | ST | MT | ST | MT | ST | MT | ST | MT
(1) ASad | 63 | 65($\uparrow$) | 60 | 59($\downarrow$) | 59 | 58($\downarrow$) | 57 | 57 | 62 | 64($\uparrow$) | 57 | 57 | 62 | 63($\uparrow$) | 62 | 63($\uparrow$) | 65 |  66($\uparrow$)
(2) RSad | 72 | 72 | 71 | 71 | 68 | 70($\uparrow$) | 61 | 62($\uparrow$) | 69 | 70($\uparrow$) | 68 | 70($\uparrow$) | 68 | 69($\uparrow$) | 69 | 71($\uparrow$) | 73 |  74($\uparrow$)
(3) InTen | 68 | 68 | 68 | 67($\downarrow$) | 67 | 66($\downarrow$) | 58 | 56($\downarrow$) | 69 |  71($\uparrow$) | 64 | 65($\uparrow$) | 66 | 64($\downarrow$) | 68 | 67($\downarrow$) | 70 | 70
(4) RSlp | 66 | 66 | 67 | 67 | 64 | 65($\uparrow$) | 58 | 59($\uparrow$) | 66 | 66 | 62 | 62 | 65 | 66($\uparrow$) | 67 | 67 | 70 | 69($\downarrow$)
(5) RApp | 47 | 45($\downarrow$) | 48 | 47($\downarrow$) | 52 |  53($\uparrow$) | 41 | 42($\uparrow$) | 47 | 47 | 42 | 43($\uparrow$) | 53 | 50($\downarrow$) | 49 | 48($\downarrow$) | 53 | 52($\downarrow$)
(6) ConD | 70 | 71($\uparrow$) | 72 | 73($\uparrow$) | 72 | 72 | 65 | 62($\downarrow$) | 67 | 66($\downarrow$) | 69 | 70($\uparrow$) | 69 | 71($\uparrow$) | 72 | 72 | 74 |  75($\uparrow$)
(7) Lass | 68 | 69($\uparrow$) | 66 | 67($\uparrow$) | 61 | 63($\uparrow$) | 57 | 60($\uparrow$) | 64 | 67($\uparrow$) | 59 | 61($\uparrow$) | 65 | 66($\uparrow$) | 64 | 65($\uparrow$) | 68 |  70($\uparrow$)
(8) IFeel | 68 | 69($\uparrow$) | 68 | 70($\uparrow$) | 63 | 61($\downarrow$) | 60 | 62($\uparrow$) | 73 |  75($\uparrow$) | 61 | 61 | 66 | 67($\uparrow$) | 73 | 72($\downarrow$) | 73 |  75($\uparrow$)
(9) PesT | 72 | 74($\uparrow$) | 72 | 75($\uparrow$) | 69 | 71($\uparrow$) | 65 | 63($\downarrow$) | 73 | 75($\uparrow$) | 74 |  75($\uparrow$) | 71 | 69($\downarrow$) | 77 | 77 | 79 | 79
(10) SuiT | 59 | 62($\uparrow$) | 60 | 63($\uparrow$) | 54 | 59($\uparrow$) | 55 | 57($\uparrow$) | 63 | 65($\uparrow$) | 60 | 64($\uparrow$) | 61 | 63($\uparrow$) | 67 | 68($\uparrow$) | 70 |  71($\uparrow$)
MADRS | 8.5 | 7.7($\downarrow$) | 8.4 | 7.9($\downarrow$) | 8.9 | 8.2($\downarrow$) | 9.5 | 8.9($\downarrow$) | 8.8 | 8.1($\downarrow$) | 8.9 | 8.2($\downarrow$) | 9.1 | 8.5($\downarrow$) | 8.1 | 7.8($\downarrow$) | 7.9 | 7.5($\downarrow$)
(RMSE)

##  4 Experiments

###  4.1 Model training

For each pretrained SSL model (see Figure 1), we freeze all its weights and
extract embeddings. We perform average pooling on the embeddings across the
temporal dimension before passing as input to a feed-forward layer comprising
$100$ SiLU. In the scenario of combining multiple SSL models, we pass the
output embedding of each model through a $100$-dimensional feed-forward layer
with SiLU units before concatenating the embeddings. Subsequently, the
concatenated output is passed through a $100$-dimensional feed-forward layer
with SiLU activation, followed by an output layer. We train both the feed-
forward and output layers using the Adam optimizer with a learning rate of
$1e-3$, a weight decay of $1e-5$, and a batch size of $32$. Negative log-
likelihood (NLL) and mean squared error (MSE) loss functions are utilized for
training models on classification and regression tasks, respectively. All
experiments are conducted using a single A40 GPU. Each model is trained for 5
epochs, and the results are reported using 5-fold cross-validation, where the
folds are speaker independent (no overlap of speakers between folds). A
randomly selected subset ($10\%$) of the training set is allocated as the
validation set for hyperparameter tuning.

We train CNNs on the conventional speech features using Adam optimizer with
$\beta_{1}=0.9$, $\beta_{2}=0.99$, an initial learning rate of $0.001$, weight
decay of $1e-5$, and a batch size of $32$. Dropout rates of $0.3$ and $0.4$
were used for the convolutional and fully connected layers, respectively to
avoid model overfitting. NLL and MSE loss functions were used to train models
on classification and regression tasks, respectively.

###  4.2 Results

The performance of the models is evaluated in terms of F-scores. We provide
three F-scores: F-score of the symptom being absent ($F_{A}$), F-score of the
symptom being present ($F_{P}$), and the macro F-score ($F_{M}$), computed as
$F_{M}$ = ($F_{A}$ \+ $F_{P}$)/2. We perform 5-fold cross validation, and
provide the performance metrics as the mean across the 5-folds.

Table 2 provides the performance of different SSL models, in single-task
setting, for each of the symptoms as defined in the MADRS. Models such as
WavLM and HuBERT which encode semantic information along with some speaker and
prosodic information achieved better performance on ASad, RSad, RSlp, ConD,
and Lass symptoms. By contrast, RDINO, which encodes speaker information,
achieved better performance on InTen, IFeel, and SuiT symptoms. Model BYOL-
Audio, which encodes prosodic information, achieved better performance on
PesT. ContentVec, which encodes semantic information and suppresses speaker
information, achieved the lowest performance across all the models. These
results suggest that semantic information alone, without speaker or prosodic
information (as encoded by ContentVec), may not be sufficient to detect
different symptoms of depression.

Table 3 compares the performance of conventional speech features (Spectrogram,
COVAREP and eGeMAP) with the embeddings extracted from SSL-based speech
models. SSL-based models outperformed the conventional speech features. Among
the conventional features, eGeMAPS perform better than Spectrogram and COVAREP
features for most symptoms. Tables 2 and 3 show that most of the SSL-based
models outperform the conventional speech features across all symptoms. For
the regression task, SSL models perform better than the conventional speech
features with HuBERT achieving the best performance.

Table 4 compares performance between single-task and multi-task learning. The
systems trained with multi-task learning perform comparably or better than the
single task learning systems. Moreover, the computational load of multi-task
learning is much lower than the single-task learning system. For all the
models, the performance on the RApp (Reduced Appetite) symptom is very low.
This could be due to the least correlation between the symptom RApp and speech
[38].

Table 5 provides the performance when different SSL-based speech embeddings
are combined. These model combinations are selected such that different types
of information (semantic, speaker and prosodic) can be comnbined. The
performances are provided in terms of macro F-scores ($F_{M}$), and absolute
improvement in performance when compared to the best performing single model
in the set is provided in parenthesis. Combining semantic (ContentVec) with
speaker (RDINO) or prosodic information (BYOL-Audio), i.e., {CV,RD} or
{CV,BY}, improves performance. But combining semantic with speaker and
prosodic information ({CV,RD,BY} and {HB,RD,BY}) further improves performance
for most of the symptoms, with {HB,RD,BY} achieving the best performance on
most of the symptoms. This shows that the semantic, speaker, and prosodic
information might all be essential for most of the tasks.

Table 5: Performance (in terms of macro F-score) obtained by combining
different SSL-based speech models. CV, RD, BY and HB refer to ContentVec,
RDINO, BYOL-A, HuBERT, respectively. The absolute improvement in performance
(in terms of macro F-score) compared to the best performing single model in
the set is given in parenthesis.

Symptoms | {CV,RD} | {CV,BY} | {RD,BY} | {CV,RD,BY} | {HB,RD,BY}
---|---|---|---|---|---
(1) ASad | 63 (1$\uparrow$) | 61 (4$\uparrow$) | 64 (2$\uparrow$) |  62 |  65 (3$\uparrow$)
(2) RSad | 70 (1$\uparrow$) | 68 | 70 (1$\uparrow$) |  69 |  73 (2$\uparrow$)
(3) InTen | 67 (2$\downarrow$) | 66 (2$\uparrow$) |  70 (1$\uparrow$) |  68 (1$\downarrow$) |  70 (1$\uparrow$)
(4) RSlp | 67 (1$\uparrow$) | 63 (1$\uparrow$) | 69 (3$\uparrow$) |  67 (1$\uparrow$) |  70 (3$\uparrow$)
(5) RApp | 48 (1$\uparrow$) | 44 (2$\uparrow$) | 52 (5$\uparrow$) |  49 (2$\uparrow$) |  53 (5$\uparrow$)
(6) ConD | 68 (1$\uparrow$) | 72 (3$\uparrow$) | 70 (1$\uparrow$) |  72 (3$\uparrow$) |  74 (2$\uparrow$)
(7) Lass | 63 (1$\downarrow$) | 61 (2$\uparrow$) | 64 |  64 |  68 (2$\uparrow$)
(8) IFeel | 72 (1$\downarrow$) | 63 (2$\uparrow$) | 72 (1$\downarrow$) |  73 |  73
(9) PesT | 76 (3$\uparrow$) | 75 (1$\uparrow$) | 78 (4$\uparrow$) |  77 (3$\uparrow$) |  79 (5$\uparrow$)
(10) SuiT | 66 (3$\uparrow$) | 62 (2$\uparrow$) | 68 (5$\uparrow$) |  67 (4$\uparrow$) |  70 (7$\uparrow$)
MADRS |  8.29 |  8.35 |  8.20 |  8.12 |  7.92
Score (RMSE) | (0.47$\downarrow$) | (0.50$\downarrow$) | (0.56$\downarrow$) |  (0.64$\downarrow$) |  (0.52$\downarrow$)

We employ bootstrapping approach [39] to compute confidence intervals for
comparing SSL-based models with those trained using conventional features for
each symptom. The 95% confidence interval for the difference between group
means do not contain zero, implying a significantly better performance for
SSL-based models than conventional features. We observe this behaviour for all
the symptoms except for the reduced appetite for which the performance of all
the models is very low.

##  5 Conclusions

In this study, we explore the application of self-supervised learning
(SSL)-based speech models for detecting individual symptoms of depression. We
demonstrate that SSL embeddings yield huge performance improvements compared
to conventional speech features such as spectrograms, eGeMAPS, and COVAREP.
Upon comparing different SSL-based speech models, we find that models encoding
predominant semantic information, combined with other information, demonstrate
improved performance in symptoms such as apparent and reported sadness,
concentration difficulties. Meanwhile, models encoding speaker and prosodic
information excel in symptoms such as inability to feel, pessimistic thoughts,
and suicidal tendencies. Importantly, combining multiple SSL-based embeddings
that encode various speech aspects, enhances detection performance across most
depressive symptoms, signifying the importance of semantic, speaker and
prosodic information. Additionally, we show that multi-task learning, while
being efficient, performs in-par or better than single-task learning.

##  6 Acknowledgements

This work has been supported by the Canada Research Chairs Program (file
number 950 - 233141) and the Canadian Institutes of Health Research. We thank
the Canadian Institute for Advanced Research (CIFAR) for their support.
Resources used in preparing this research were provided, in part, by NSERC,
the Province of Ontario, the Government of Canada through CIFAR, and companies
sponsoring the Vector Institute www.vectorinstitute.ai/#partners.

## References

  * [1] J. Rehm and K. D. Shield, ``Global burden of disease and the impact of mental and addictive disorders,'' _Current psychiatry reports_ , vol. 21, no. 2, p. 10, 2019.
  * [2] H. Herrman, V. Patel, C. Kieling, M. Berk, C. Buchweitz, P. Cuijpers, T. A. Furukawa _et al._ , ``Time for united action on depression: a lancet–world psychiatric association commission,'' _The Lancet_ , vol. 399, no. 10328, pp. 957–1022, 2022.
  * [3] N. Cummins, S. Scherer, J. Krajewski, S. Schnieder, J. Epps, and T. F. Quatieri, ``A review of depression and suicide risk assessment using speech analysis,'' _Speech communication_ , vol. 71, 2015.
  * [4] F. Ringeval, B. Schuller, M. Valstar, N. Cummins _et al._ , ``Avec 2019 workshop and challenge: state-of-mind, detecting depression with ai, and cross-cultural affect recognition,'' in _Proceedings Audio/visual Emotion Challenge and Workshop_ , 2019, pp. 3–12.
  * [5] M. Yamamoto, A. Takamiya, K. Sawada, M. Yoshimura, M. Kitazawa, K.-c. Liang, T. Fujita, M. Mimura, and T. Kishimoto, ``Using speech recognition technology to investigate the association between timing-related speech features and depression severity,'' _PloS one_ , vol. 15, no. 9, 2020.
  * [6] D. M. Low, K. H. Bentley, and S. S. Ghosh, ``Automated assessment of psychiatric disorders using speech: A systematic review,'' _Laryngoscope investigative otolaryngology_ , vol. 5, no. 1, 2020.
  * [7] K. Dikaios, S. Rempel, S. H. Dumpala, S. Oore, M. Kiefte, and R. Uher, ``Applications of speech analysis in psychiatry,'' _Harvard Review of Psychiatry_ , vol. 31, no. 1, pp. 1–13, 2023.
  * [8] A. Harati, E. Shriberg, T. Rutowski, P. Chlebek, Y. Lu, and R. Oliveira, ``Speech-based depression prediction using encoder-weight-only transfer learning and a large corpus,'' in _ICASSP_. IEEE, 2021, pp. 7273–7277.
  * [9] S. P. Dubagunta, B. Vlasenko, and M. Magimai.-Doss, ``Learning voice source related information for depression detection,'' in _IEEE ICASSP_ , 2019, pp. 6525–6529.
  * [10] V. Ravi, J. Wang, J. Flint, and A. Alwan, ``Enhancing accuracy and privacy in speech-based depression detection through speaker disentanglement,'' _Computer Speech & Language_, vol. 86, 2024.
  * [11] S. H. Dumpala, S. Rempel, K. Dikaios, M. Sajjadian, R. Uher, and S. Oore, ``Estimating severity of depression from acoustic features and embeddings of natural speech,'' in _ICASSP_. IEEE, 2021.
  * [12] N. Seneviratne and C. Espy-Wilson, ``Multimodal depression severity score prediction using articulatory coordination features and hierarchical attention text embeddings,'' _Interspeech_ , 2022.
  * [13] S. H. Dumpala, R. Uher, S. Matwin, M. Kiefte, and S. Oore, ``Sine-wave speech and privacy-preserving depression detection,'' in _Proc. SMM21, Workshop on Speech, Music and Mind_ , vol. 2021, 2021, pp. 11–15.
  * [14] S. H. Dumpala, K. Dikaios, S. Rodriguez, R. Langley, S. Rempel, R. Uher, and S. Oore, ``Manifestation of depression in speech overlaps with characteristics used to represent and recognize speaker identity,'' _Scientific Reports_ , vol. 13, no. 1, p. 11155, 2023.
  * [15] W. Wu, C. Zhang, and P. C. Woodland, ``Self-supervised representations in speech-based depression detection,'' in _ICASSP_. IEEE, 2023, pp. 1–5.
  * [16] S. H. Dumpala, C. S. Sastry, R. Uher, and S. Oore, ``On combining global and localized self-supervised models of speech,'' in _INTERSPEECH_ , 2022.
  * [17] S. Fara, O. Hickey, A. Georgescu, S. Goria, E. Molimpakis, and N. Cummins, ``Bayesian networks for the robust and unbiased prediction of depression and its symptoms utilizing speech and multimodal data,'' in _Proc. INTERSPEECH_ , 2023, pp. 1728–1732.
  * [18] S. Yadav, J. Chauhan, J. P. Sain, K. Thirunarayan, A. P. Sheth, and J. Schumm, ``Identifying depressive symptoms from tweets: Figurative language enabled multitask learning framework,'' in _Proceedings of COLING_ , 2020, pp. 696–709.
  * [19] S. Fara, S. Goria, E. Molimpakis, and N. Cummins, ``Speech and the n-back task as a lens into depression. how combining both may allow us to isolate different core symptoms of depression,'' in _Interspeech_. ISCA, 2022, pp. 1911–1915.
  * [20] T. Takano, D. Mizuguchi, Y. Omiya, M. Higuchi, M. Nakamura, S. Shinohara, S. Mitsuyoshi, T. Saito, A. Yoshino, H. Toda _et al._ , ``Estimating depressive symptom class from voice,'' _International Journal of Environmental Research and Public Health_ , vol. 20, no. 5, p. 3965, 2023.
  * [21] F. Eyben, K. Scherer, B. Schuller _et al._ , ``The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,'' _IEEE Transactions on Affective Computing_ , vol. 7, pp. 1–1, 01 2015.
  * [22] K. Kroenke, T. W. Strine, R. L. Spitzer, J. B. Williams, J. T. Berry, and A. H. Mokdad, ``The phq-8 as a measure of current depression in the general population,'' _Journal of affective disorders_ , vol. 114, no. 1-3, pp. 163–173, 2009.
  * [23] J. B. Williams, K. A. Kobak, P. Bech, N. Engelhardt, K. Evans, J. Lipsitz, J. Olin, J. Pearson, and A. Kalali, ``The grid-hamd: standardization of the hamilton depression rating scale,'' _International clinical psychopharmacology_ , vol. 23, no. 3, pp. 120–129, 2008.
  * [24] S. A. Montgomery and M. Åsberg, ``A new depression scale designed to be sensitive to change.'' _The British Journal of Psychiatry_ , 1979.
  * [25] T. Rutowski, A. Harati, E. Shriberg, Y. Lu, P. Chlebek, and R. Oliveira, ``Toward corpus size requirements for training and evaluating depression risk models using spoken language.'' in _INTERSPEECH_ , 2022, pp. 3343–3347.
  * [26] P. Zhang, M. Wu, H. Dinkel, and K. Yu, ``Depa: Self-supervised audio embedding for depression detection,'' in _Proceedings of ACM international conference on multimedia_ , 2021, pp. 135–143.
  * [27] E. L. Campbell, J. Dineley, P. Conde, F. Matcham, K. M. White, C. Oetzmann, S. Simblett, S. Bruce, A. A. Folarin, T. Wykes _et al._ , ``Classifying depression symptom severity: Assessment of speech representations in personalized and generalized machine learning models,'' in _INTERSPEECH_ , 2023, pp. 1738–1742.
  * [28] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [29] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino, ``BYOL for Audio: Exploring pre-trained general-purpose audio representations,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, p. 137–151, 2023.
  * [30] K. Qian, Y. Zhang, H. Gao, J. Ni, C.-I. Lai, D. Cox, M. Hasegawa-Johnson, and S. Chang, ``Contentvec: An improved self-supervised speech representation by disentangling speakers,'' in _ICML_. PMLR, 2022, pp. 18 003–18 017.
  * [31] Y. Chen, S. Zheng, H. Wang, L. Cheng, and Q. Chen, ``Pushing the limits of self-supervised speaker verification using regularized distillation framework,'' in _ICASSP_. IEEE, 2023, pp. 1–5.
  * [32] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , ``Wavlm: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1505–1518, 2022.
  * [33] S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, ``Beats: Audio pre-training with acoustic tokenizers,'' in _ICML_ , 2023.
  * [34] P.-Y. Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, ``Masked autoencoders that listen,'' _NeurIPS_ , vol. 35, pp. 28 708–28 720, 2022.
  * [35] G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer, ``Covarep— a collaborative voice analysis repository for speech technologies,'' in _IEEE ICASSP_ , 2014, pp. 960–964.
  * [36] F. Eyben, M. Wöllmer, and B. Schuller, ``Opensmile: the munich versatile and fast open-source audio feature extractor,'' in _Proc. ACM conference on Multimedia_ , 2010, pp. 1459–1462.
  * [37] S. Elfwing, E. Uchibe, and K. Doya, ``Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,'' _Neural networks_ , vol. 107, pp. 3–11, 2018.
  * [38] R. Uher, A. Farmer, W. Maier _et al._ , ``Measuring depression: comparison and integration of three scales in the gendep study,'' _Psychological medicine_ , vol. 38, pp. 289–300, 2008.
  * [39] L. Ferrer and P. Riera, ``Confidence intervals for evaluation in machine learning [computer software],'' Github, 2022. [Online]. Available: https://github.com/luferrer/ConfidenceIntervals


# Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human
Processing Difficulty From Syntactic Ambiguities

Suhas Arehalli
Johns Hopkins University
<EMAIL_ADDRESS>
&Brian Dillon
University of Massachusetts, Amherst
<EMAIL_ADDRESS>
Tal Linzen
New York University
<EMAIL_ADDRESS>

Humans exhibit garden path effects: When reading sentences that are
temporarily structurally ambiguous, they slow down when the structure is
disambiguated in favor of the less preferred alternative. Surprisal theory
Hale (2001); Levy (2008), a prominent explanation of this finding, proposes
that these slowdowns are due to the unpredictability of each of the words that
occur in these sentences. Challenging this hypothesis, van Schijndel and
Linzen (2021) find that estimates of the cost of word predictability derived
from language models severely underestimate the magnitude of human garden path
effects. In this work, we consider whether this underestimation is due to the
fact that humans weight syntactic factors in their predictions more highly
than language models do. We propose a method for estimating syntactic
predictability from a language model, allowing us to weigh the cost of lexical
and syntactic predictability independently. We find that treating syntactic
predictability independently from lexical predictability indeed results in
larger estimates of garden path. At the same time, even when syntactic
predictability is independently weighted, surprisal still greatly
underestimate the magnitude of human garden path effects. Our results support
the hypothesis that predictability is not the only factor responsible for the
processing cost associated with garden path sentences.

##  1 Introduction

Readers exhibit garden path effects: When reading a temporarily syntactically
ambiguous sentence, they tend to slow down when the sentence is disambiguated
in favor of the less preferred parse. For example, a participant who reads the
sentence fragment The suspect sent the file … …to the lawyer. …deserved
further investigation. can construct a partial parse in at least two distinct
ways: In one reading, the verb sent acts as the main verb of the sentence, and
the continuation of the sentence as an additional argument to sent (as in 1).
In another, less likely, reading, sent the file acts as a modifier in a
complex subject, which then requires an additional verb phrase to form a
complete sentence (as in 1). Prior work has demonstrated that regions like
deserved further investigation, which disambiguate these temporarily ambiguous
sentences in favor of the modifier parse (1), are read slower than those same
words would be in an unambiguous version of sentence, such as the following:
The suspect who was sent the file deserved further investigation. In (1), the
presence of who was signals to the reader that sent the file acts as a
modifier (Frazier and Fodor, 1978).

One account of this phenomenon, surprisal theory (Hale, 2001; Levy, 2008),
suggests that readers maintain a probabilistic representation of all possible
parses of the input as they process the sentence incrementally. Processing
difficulty in garden path sentences is the cost associated with updating this
representation; this cost is proportional to the negative log probability, or
surprisal, of the newly observed material under the reader’s model of upcoming
words. This theory predicts that the slowdown associated with garden path
sentences can be entirely captured by the differences in surprisal between the
disambiguating region in ambiguous garden path sentences and that same region
in a matched unambiguous sentence.

van Schijndel and Linzen (2021) tested this hypothesis. They estimated the
surprisals associated with garden path sentences using LSTM language models
(LMs) trained over large natural language corpora. Based on the core
assumption of surprisal theory—that processing difficulty on a word, when all
lexical factors are kept constant, stands in a constant proportion to the
word’s surprisal, regardless of its syntactic context—they estimated a
conversion factor between surprisal and reading times from non-garden path
sentences. Applying this conversion factor to the critical words in garden
path sentences, van Schijndel and Linzen found that surprisal theory, when
paired with the surprisals estimated by their models, severely underestimated
the magnitude of the garden path effect for three garden path constructions,
consistent with attempts to estimate the magnitude of other syntactically-
modulated effects Wilcox et al. (2021). Moreover, the predicted reading times
did not correctly predict differences across the difference garden path
constructions, suggesting that no single conversion factor between surprisal
and reading times could predict the magnitude of the garden path effect in all
three constructions.

The underestimation documented by van Schijndel and Linzen can be interpreted
in one of two ways: Either (1) surprisal theory cannot, on its own, account
for garden path effects; or (2) predictability estimates derived from LSTM LMs
fail to capture some aspect of human prediction that is crucial to explaining
the processing of garden path sentences. This work investigates the latter
possibility. We ask if the gap between the magnitude of human garden path
effects in humans and the magnitude that surprisal theory predicts from LMs is
due to a mismatch between how humans and LMs weigh two contributors to word-
level surprisal: syntactic and lexical predictability. We hypothesize that the
LM next-word prediction objective does not sufficiently emphasize the
importance that syntactic structure carries for human readers, who may be more
actively concerned with interpreting the sentence. In this scenario, since
garden paths are the product of unpredictable syntactic structure—as opposed
to an unpredictable lexical item—using a LM predictability estimate for the
next word could lead to underestimation of garden path effects.

Figure 1: A depiction of the relationship between syntactic and lexical
surprisal. Some word tokens, such as are in the context of owls are more
flexible, are highly predictable in all respects. Others are unpredictable due
to the syntactic structures they imply (trying in girls trying to save up),
and are expected to be assigned high syntactic and lexical surprisal. Tokens
such as microbes in the context the newfound microbes were, on the other hand,
appear in a predictable syntactic environment, but are unpredictable due to
their low lexical frequency; such words should be assigned low syntactic
surprisal but high lexical surprisal. Since words that appear in unpredictable
syntactic environments are themselves unpredictable, we do not expect to find
words with high syntactic surprisal but low lexical surprisal.

We test the hypothesis that the gap between model and human effects can be
bridged by teasing apart the overall predictability of a word from the
surprisal associated with the syntactic structure implied by the word (see
Figure 1) and weighting the two factors independently, possibly assigning a
higher weight to syntactic surprisal. In this reasoning, we follow prior work
on syntactic or unlexicalized surprisal carried out in the context of symbolic
parsers, where the probability of a structure and particular lexical item can
be explicitly disentangled Demberg and Keller (2008); Roark et al. (2009). But
while past work has demonstrated that that unlexicalized suprisal from
symbolic parsers correlates with measures of human processing difficulty
Demberg and Keller (2008), simple recurrent neural networks trained to predict
sequences of part-of-speech tags have been shown to track processing
difficulty even more strongly Frank (2009), suggesting that even fairly
limited syntactic representations like part-of-speech tags can act as a
reasonable proxy of syntactic structure when modeling human behavior.

To compute LSTM-based syntactic surprisal, we train the LM with an auxiliary
objective—estimating the likelihood of the next word’s supertag under the
Combinatory Categorial Grammar (CCG) framework (Steedman, 1987)—following
Enguehard et al. (2017). Such supertags can be viewed as enriched part-of-
speech tags that encode syntactic information about how a particular word can
be combined with its local environment. We then define syntactic surprisal in
terms of the likelihood of the next word’s CCG supertag, and propose a method
of estimating that likelihood using our modified LMs. We validate our
formulation of syntactic surprisal by demonstrating that it captures syntactic
processing difficulty in garden path sentences, while, crucially, not tracking
unpredictability that is due to low frequency lexical items. Following van
Schijndel and Linzen (2021), we then use the syntactic and lexical surprisal
values derived from those models to predict reading times for three types of
garden path sentences. We find that adding syntactic surprisal as a separate
predictor does lead to larger estimates of garden path effects, but those
estimates are still an order of magnitude lower than empirical garden path
effects. Finally, we discuss the implications of these findings for surprisal
theory and single-stage models of syntactic processing.

##  2 Computing Syntactic Surprisal

Each incoming word can cause an adjustment in the reader’s beliefs about the
syntactic structure of the sentence; when a syntactic structure that was
assigned a low probability prior to reading the word now has high probability,
the word can be said to have high syntactic surprisal. We will operationalize
this intuition as the predictability of next word’s supertag under the
Combinatory Categorial Grammar (CCG) formalism (Steedman, 1987):

$\text{surp}_{\text{syn}}=-\log(P(c_{n}\mid w_{1},...,w_{n-1})),$ (1)

where $c_{n}$ is the CCG supertag of the $n$-th word. A CCG supertag encodes
how a word combines syntactically with adjacent constituents. For example, a
token with the tag S\NP combines with an NP to its left to form an S
constituent, and a token with the tag (S\NP)/NP combines with an NP to its
right to form an S\NP constituent. Since the sequence of supertags associated
with all of the words of a sentence often allows only one valid parse,
accurately predicting a sentence’s supertags has been described as “almost
parsing" (Bangalore and Joshi, 1999); consequently, incremental CCG
supertagging can be seen as almost incremental parsing.

We contrast this syntactic surprisal measure with the standard token surprisal
measure, which we refer to as lexical surprisal:

$\text{surp}_{\text{lex}}=-\log(P(w_{n}\mid w_{1},...,w_{n-1})).$ (2)

Note that what we call lexical surprisal captures all factors that contribute
to a token’s predictability, including syntactic ones.

In order to compute syntactic and lexical surprisal for a given word, we need
models that predict, given a left context, not only the next token, as a
standard LM does, but also the next token’s supertag. To do this, we train
models with both a language modeling and CCG supertagging objective, and
estimate the distribution over the next word’s tag by marginalizing over the
distribution over the next word that is defined by the LM. Formally, for a
sequence of words $w_{1},...,w_{n}\in W$ with supertags $c_{1},...,c_{n}\in
C$, our model estimates the probability of the next word given all observed
words, $p_{w_{n+1}}=P(w_{n+1}\mid w_{1},...,w_{n})$, and the probability of
the most recent word’s supertag given all currently observed words,
$p_{c_{n}\mid w_{n}}=P(c_{n}\mid w_{1},...,w_{n})$. We then infer the
distribution over the next word’s supertag as

$\displaystyle P(c_{n+1}|w_{1},...,w_{n})=\sum_{w^{*}_{n+1}\in
W}p_{c_{n+1}\mid w^{*}_{n+1}}p_{w^{*}_{n+1}}$ (3)

If we knew the supertag of the next word $c_{n+1}$, we could simply compute
the surprisal of that supertag, $-\log P(c_{n+1}\mid w_{1},...,w_{n})$. By
contrast with lexical surprisal, however—where there is no uncertainty about
the identity of $w_{n+1}$ once that word has been read—a word’s supertag is
often ambiguous during incremental processing. Consider the verb gathered in
the following sentences, for example: The squirrels gathered near the tree.
The squirrels gathered a few acorns. In (2), gathered would eventually be
assigned the supertag S\NP, indicating that gathered is used in its
intransitive frame—a number of squirrels assembled together as a group—and
takes no direct object. In (2), on the other hand, the appropriate supertag
would be (S\NP)/NP, which indicates that in this sentence gathered is used in
a transitive frame and takes the noun phrase a few acorns as a direct object.
When processing this sentence incrementally, a reader must maintain this
uncertainty over the appropriate supertag for a word past the point at which
they have read that word. A measure of syntactic surprisal that aims to model
processing difficulty at a particular word should similarly take into account
uncertainty over the supertag of a word even after the word itself has been
processed. We take this uncertainty into account by using the distribution
$p_{c_{n}\mid w_{n}}$ that our models define over supertags, and computing
syntactic surprisal by marginalizing over this distribution:

$\displaystyle p_{c_{n+1}\mid w_{n}}$ $\displaystyle=P(c_{n+1}\mid
w_{1},...,w_{n})$ (4) $\displaystyle surp_{syn}$
$\displaystyle=-\log\sum_{c_{n+1}^{*}\in C}p_{c_{n+1}^{*}\mid
w_{n}}p_{c_{n+1}\mid w_{n+1}}$ (5)

###  2.1 Model Architecture and Training

We trained four models, differing only in their random seed, on both a
language modeling and CCG supertagging objective. The models consisted of an
LSTM shared across the two objectives, which we refer to here as the encoder,
and two classifiers, one for language modeling and another for CCG
supertagging, which we refer to as the decoders.

Following Gulordava et al. (2018), the encoder was a two-layer LSTM with 650
units per layer. Each decoder consists of a single linear layer followed by
the softmax operation. For the supertagging objective, we trained using
CCGBank Hockenmaier and Steedman (2007), a set of CCG annotations for the Wall
Street Journal section of the Penn Treebank Marcus et al. (1993). The corpus
we used for language modeling was a concatenation of the Wall Street Journal
portion of the Penn Treebank and the 80 million words from the English
Wikipedia used by Gulordava et al. (2018). Language modeling and supertagging
losses were weighted equally during training.

Models achieved language modeling perplexities ranging from 74.76 to 75.70 on
the test portion of the Gulordava et al. (2018) corpus, while Gulordava et al.
(2018)’s best language model achieved a perplexity of 52.0. Models assigned
the highest likelihood to the correct CCG supertag in the CCGBank test set
between 84.1% and 84.5% of the time, compared to bi-LSTM supertaggers which
can achieve an accuracy of 94.1% Vaswani et al. (2016). Note that these
supertagging numbers are not directly comparable, as our models use
unidirectional LSTMs and thus have no access to a word’s right context when
supertagging.

###  2.2 Experimental data

We evaluated our model on a subset of the Syntactic Ambiguity Processing (SAP)
Benchmark Huang et al. (2022), a dataset containing self-paced reading times
from 2000 native English speakers who read a variety of syntactically complex
constructions as well as comparatively simple filler sentences. The large size
of the dataset allows us to get precise estimates of the magnitude of the
garden path effect for each of the three types of garden path sentences it
contains. We describe each of these constructions in what follows.

#### Main Verb/Reduced Relative (MVRR)

This ambiguity is illustrated in (2.2):

The suspect sent the file deserved further investigation given the new
evidence. The suspect who was sent the file deserved further investigation
given the new evidence.

In (2.2), before reading the word deserved, the reader can interpret sent the
file either as a main verb and direct object (where the subject has sent the
file) or as a reduced relative clause (where the subject has had the file sent
to them). This is disambiguated in favor of the reduced relative clause
reading by the next word, deserved, which is the true main verb of the
complete sentence. We can measure the processing difficulty incurred by this
disambiguation by comparing the reading times at deserved in (2.2) with the
reading times at deserved in (2.2), where the relative clause who was sent the
file is unreduced and thus unambiguous.

#### Noun Phrase/Sentence (NPS)

The NPS ambiguity is illustrated in (2.2): The suspect showed the file
deserved further investigation during the murder trial. The suspect showed
that the file deserved further investigation during the murder trial. Before
reading deserved in (2.2), the file can be interpreted as either a direct
object, where the suspect is presenting a file to someone, or as the beginning
of a sentential complement, where the suspect is making a point. The word
deserved disambiguates the sentence in favor of the less frequent sentential
complement reading. As before, the matched control sentence (2.2) avoids the
ambiguity, here by using the explicit complementizer that before the file;
this control makes it possible to measure the slowdown associated with
disambiguation.

Figure 2: Differences in surprisal estimates between ambiguous and unambiguous
garden path sentences at and around the disambiguating verb. Bars indicate 95%
confidence intervals.

#### Noun Phrase/Zero (NPZ):

Finally, in (2.2), before reading deserved, changed can be interpreted in two
ways: as a transitive verb taking the file as a noun phrase direct object
(where the file was changed by the suspect); or as an intransitive verb, with
the file as the subject of a separate clause (where the suspect was changed):

Because the suspect changed the file deserved further investigation during the
jury discussions. Because the suspect changed, the file deserved further
investigation during the jury discussions.

The word deserved disambiguates the sentence in favor of the less frequent
intransitive reading. Introducing a comma between the clauses in the matched
sentence (2.2) removes the ambiguity.

##  3 Validating Syntactic Surprisal

We first validate that our syntactic surprisal measure successfully isolates
syntactic predictability from word predictability. To be satisfied that that
is the case, we will require two things be true: first, we expect syntactic
surprisal to capture processing difficulty that is the result of syntactic
unpredictability; and second, we expect that syntactic surprisal is not
redundant with lexical predictability. We will evaluate each of these
desiderata in turn.

###  3.1 Syntactic Surprisal Captures Syntactic Processing Difficulty

To verify that syntactic surprisal can capture syntactic unpredictability, we
investigate differences in syntactic surprisal between the ambiguous and
unambiguous garden path sentences in Huang et al. (2022). Since garden path
effects are the result of ambiguity about the syntactic structure of a
sentence, a difference in surprisal at the point of disambiguation indicates
sensitivity to differences in syntactic predictability.

We found differences in the expected direction for all three types of garden
sentences. This was the case both for lexical surprisal—consistent with prior
work (Hale, 2001; van Schijndel and Linzen, 2021)—and for syntactic surprisal
(Figure 2). We did not find differences in the same direction before the point
of disambiguation, indicating that the differences we observe after
disambiguation are not a consequence differences in surprisal earlier in the
sentence that the LM has not fully recovered from.

(a) Lexical vs. syntactic surprisal.

(b) Log unigram frequency vs. syntactic surprisal.

Figure 3: Correlations between syntactic surprisal, lexical surprisal, and
unigram frequency for each word in the filler items of Huang et al. (2022).
Since these results are fairly consistent across model instances, we present
results from a single instance. Two words — one with high syntactic surprisal
and high lexical surprisal and one with high lexical surprisal but low
syntactic surprisal — are labeled with their context.

###  3.2 Syntactic Surprisal Captures Only Syntactic Predictability

To verify that syntactic surprisal successfully isolates syntactic factors on
predictability, we make two comparisons: first, to lexical surprisal, to
verify that syntactic surprisal does not capture all of the variance captured
by lexical surprisal; and second, to unigram frequency, to verify that
syntactic surprisal is not driven by the frequency of specific lexical items.

#### Syntactical surprisal does not capture all of the variance captured by
lexical surprisal

If syntactic surprisal captures a strict subset of the variance captured by
lexical surprisal, we expect to see a subset of words with high lexical
surprisal and low syntactic surprisal (in addition, perhaps, to words with
highly correlated syntactic and lexical surprisals). This subset should
represent words that are unpredictable for reasons that are independent of the
syntactic structures they imply. By contrast, words that introduce infrequent
syntactic structures should have both high syntactic surprisal and high
lexical surprisal, as the unpredictability of the syntactic structure means
that a word that implies that structure is necessarily unpredictable. This
matches what we see in Figure 3(a): The relatively frequent verb trying
introducing a reduced relative clause has high syntactic and lexical
surprisal, while infrequent nouns like microbe have low syntactic surprisal
but high lexical surprisal.

#### High syntactic surprisal does not reflect low unigram frequency

In Figure 3(b), we plot the syntactic surprisals of words from the filler
items with their log-frequency in the Corpus of Contemporary American English
(COCA; Davies 2008–). We find a significant but small positive correlation
between the two ($r=0.064$, $t=3.18$, $p<0.005$), indicating that more
frequent words have a higher syntactic surprisal — the opposite of what we
would expect if lexical frequency were driving syntactic surprisal effects.
This may be due to the fact that function words, which are generally high-
frequency, typically introduce additional syntactic structure and thus have
higher-than-average syntactic surprisal.

These three results — that syntactic surprisal captures garden path effects,
that we find a subset of words with low syntactic surprisal and high lexical
surprisal, and that we find no evidence of low lexical frequency driving
syntactic surprisal — suggest that syntactic surprisal captures only the
syntactic contributions to a word’s unpredictability. We will now use
syntactic surprisal in concert with lexical surprisal to directly predict the
magnitude of garden path effects.

##  4 Evaluating Against Human Reading Times

Recall that surprisal theory assumes a linear relationship between surprisal
and measures of processing difficulty such as reading times. We follow van
Schijndel and Linzen (2021) and estimate a mapping between our surprisal
measures and reading times by fitting linear mixed-effects models to the
filler (i.e., non-garden path) materials from Huang et al. (2022). In order to
compare syntactic and lexical surprisal, we fit four conversion models: one
with syntactic surprisal as a predictor, one with lexical surprisal, one with
both types of surprisal, and one that does not include either surprisal
measure. All four models included baseline predictors other than surprisal —
unigram frequency, word position, and word length — which on their own are not
expected to capture garden path effects. To account for spillover effects,
where processing difficulty from a word spills over to affect reading times at
future words, we included all of the aforementioned factors (except word
position) not only for the current word but also for the two prior words (a
simplification of the technique of van Schijndel and Linzen 2021). This
process is repeated with each of the four sets of surprisals extracted from
our four language model/supertagger instances. Further details about the
surprisal-to-RT conversion process are presented in Appendix A.1. After all
four of our models have been fit to the filler items, we use the estimated
coefficients to predict reading times for the each of the critical items.

##  5 Results

(a) Model predictions and human results.

(b) Model predictions only.

Figure 4: Empirical and model-predicted readings times for the three garden
path constructions. Bars indicate the difference between the mean readings
times for the ambiguous and unambiguous sentences across participants for each
condition. Error bars indicate bootstrapped 95% confidence intervals.

Predicted RT differences from our conversion models, as well as the RT
differences observed in humans, are presented in Figure 4(a). Regardless of
the predictors used in the mixed-effects model—lexical surprisal, syntactic
surprisal, neither, or both—predicted reading time differences greatly
underestimate the reading time differences observed in humans. This is
unlikely to be an issue with our surprisal-to-reading-times conversion method
more broadly, as at the pre-disambiguation word, RTs and predicted RTs match
much more closely than in post-disambiguation regions (particularly in
capturing effects in the pre-disambiguation region in NPS sentences),
indicating that the difference in magnitudes is due specifically to an
underestimation of the garden path effect.

Disambig | MVRR | NPS | NPZ
---|---|---|---
Both vs. Syntactic Only |  $\beta=0.37$, $p<0.001$ |  $\beta=1.19$, $p<0.001$ |  $\beta=0.15$, $p=0.056$
Syntactic Only vs. Lexical Only |  $\beta=3.26$, $p<0.001$ |  $\beta=0.89$, $p<0.001$ |  $\beta=1.89$, $p<0.001$
Syntactic Only vs. Neither |  $\beta=6.77$, $p<0.001$ |  $\beta=2.85$, $p<0.001$ |  $\beta=1.77$, $p<0.001$
Spillover 1 | MVRR | NPS | NPZ
Both vs. Syntactic Only |  $\beta=1.80$, $p<0.001$ |  $\beta=1.25$, $p<0.001$ |  $\beta=2.09$, $p<0.001$
Syntactic Only vs. Lexical Only |  $\beta=-1.02$, $p<0.001$ |  $\beta=-0.53$, $p<0.001$ |  $\beta=-1.62$, $p<0.001$
Syntactic Only vs. Neither |  $\beta=5.10$, $p<0.001$ |  $\beta=3.27$, $p<0.001$ |  $\beta=2.36$, $p<0.001$
Spillover 2 | MVRR | NPS | NPZ
Both vs. Syntactic Only |  $\beta=1.68$, $p<0.001$ |  $\beta=0.92$, $p<0.001$ |  $\beta=2.10$, $p<0.001$
Syntactic Only vs. Lexical Only |  $\beta=-4.59$, $p<0.001$ |  $\beta=-3.86$, $p<0.001$ |  $\beta=-5.03$, $p<0.001$
Syntactic Only vs. Neither |  $\beta=-0.30$, $p<0.001$ |  $\beta=-0.92$, $p<0.001$ |  $\beta=-3.61$, $p<0.001$

Table 1: Results of a Linear Mixed Effects analysis over our model-predicted
reading times for our effect of interest: the interaction between ambiguity
and the conversion model. A significant result with a positive coefficient
indicates that the conversion model on the left side of the contrast label
predicted a significantly larger garden path effect than the model on the
right. See Appendix A.2 for further details.

While the inclusion of syntactic surprisal does not close the gap between
model predictions and the empirical reading times, it does typically lead to a
larger predicted garden path effect. To see this difference more clearly, in
Figure 4(b) we exclude the human reading times and zoom in on the garden path
effects predicted by the models. To determine whether adding syntactic
surprisal as a predictor affected the magnitude of the garden path effects we
predicted, we fit a Linear Mixed Effects Model over all of our conversion
models’ predicted reading times for each garden path construction at each word
in the critical region. We present the results of this analysis for the effect
of interest (the interaction between the conversion model and the garden path
effect) in Table 1. We find that (1) models containing both surprisals
predicted the largest garden path effects at the disambiguating word and first
spillover word, (2) models with only syntactic surprisal predicted greater
garden path effects than models with only lexical or no surprisal at the
disambiguating word, and (3) models with only lexical surprisal predicted
larger garden path effects than models with only syntactic or no surprisal in
the spillover regions. Note that while models with only lexical surprisal did
predict larger effects than other conversion models at the second spillover
word, the fact that this only takes place long after the disambiguating word
suggests that this difference is due to differing spillover profiles amongst
our surprisal measures. Since this work focuses on the estimation of the
magnitude of garden path effects, we leave an investigation of this to future
work.

##  6 Discussion

What is the source of the discrepancy between the magnitude of garden path
effects in humans and surprisal-based estimates of those magnitudes from
neural network language models? In this paper, we have evaluated one possible
answer to this question: that word predictability estimates from LMs
underweight the importance of syntax to the predictions made by humans. We
have proposed a method of estimating syntactic predictability from LSTM LMs
augmented with a CCG supertagging auxiliary objective; confirmed that this
measure matches our intuitive desiderata from a syntactic surprisal measure;
and compared garden path effect magnitude predictions derived from standard,
lexical surprisal and syntactic surprisal. Our main finding is that while the
syntactic surprisal measure we propose does typically lead to larger predicted
garden path effects, model-predicted garden path effects still vastly
underestimate the magnitude of garden path effects found in humans.

We defined syntactic surprisal in terms of the predictability of the next
word’s CCG supertag. This choice is motivated by the relative simplicity of
computing this measure—a straightforward auxiliary objective that can be added
to any conceivable neural language model—as well as two substantive
desiderata: First, we would like the measure to capture processing difficulty
due to syntactic unpredictability. Since a word’s CCG supertags captures how
the word combines with the local syntactic structure, we hypothesize that the
surprisal of that supertag—which indicates the extent to which that syntactic
combination is unexpected—is a good predictor of syntactic unpredictability.
This was borne out in our analysis that showed that syntactic surprisal
predicts differences in the correct direction in three garden path
constructions.

Second, since syntactic surprisal is designed to isolate syntactic
predictability from other forms of predictability, it should not be perfectly
correlated with lexical factors. The comparisons to lexical surprisal and word
frequency showed that this desideratum was met: We were able to identify in
our materials words that were lexically surprising but had low syntactic
surprisal, and we found a positive correlation between frequency and syntactic
surprisal — the opposite of what would be predicted if high syntactic
surprisal was driven by low word frequency.

The increase in model-predicted garden path magnitudes when we use syntactic
surprisal, compared to using just standard lexical surprisal, suggests that
predictability estimates from LSTM LMs indeed understate the role that
syntactic factors play in human prediction. To see why that is, recall that
syntactic surprisal captures a subset of the variance that lexical surprisal
does. The fact that adding syntactic surprisal produces a better fit to human
reading times than lexical surprisal, then, suggests that syntactic factors
affect lexical surprisal less than they would need to in order to capture
variation in human reading times. One potential explanation for this
discrepancy is the difference in the tasks humans and LMs perform: While LMs
need only predict words in corpora, humans must to comprehend what they read.
While both tasks demand some sensitivity to syntactic structure, the need to
interpret sentences may place greater importance on predicting structure,
leading to a higher sensitivity to syntactic unpredictability.

While models with syntactic surprisal provided a better fit to the human data
than those with just lexical surprisal, there remained a very large
discrepancy between model-predicted and human garden path effect sizes. It may
be possible to further close this gap within the surprisal framework using
different approaches to estimating syntactic predictability; one such approach
could rely on Recurrent Neural Network Grammars Dyer et al. (2016), which
derive word-level predictability estimates from explicit syntactic parsing
mechanisms.

Another possibility is that the discrepancy is not due to flaws in our
estimates of human predictability: perhaps surprisal, even based on a perfect
simulation of human predictions, is simply not the correct account of the
magnitude of the garden path effect observed in humans van Schijndel and
Linzen (2021). One family of alternative accounts consists of two-stage,
serial models of processing Frazier and Fodor (1978); Fodor and Inoue (1994);
Lewis (1998); Bader (1998); Sturt et al. (1999). In such a model, when readers
first read through the ambiguous fragment of the sentence, they commit to a
small set of preferred parses. When they reach a disambiguating region where
all of the parses they have committed to are no longer consistent with the
input, a reader would engage a separate, costly reanalysis process in order to
construct a new partial parse consistent with the all of the currently
available input. The processing cost associated with this reanalysis
procedures incurs a slowdown in reading times that does not occur in an
unambiguous sentence where the incorrect initial parse is not available,
resulting the garden path effects that we observe. Unlike surprisal-based
accounts, however, it is often unclear how to derive broad-coverage,
quantitative predictions for the size of garden path effects from existing
two-stage accounts. As a result, it is difficult to know whether the
quantitative mismatches between surprisal-accounts and human reading times
that we observed should be taken as evidence for an explicit reanalysis
process. This further highlights the need for precise implementations of two-
stage serial models that we can quantitatively evaluate against surprisal
accounts.

## Acknowledgements

This work was supported by the National Science Foundation, grant nos.
BCS-2020945 and BCS-2020914, by the United States–Israel Binational Science
Foundation (award no. 2018284), and in part through the NYU IT High
Performance Computing resources, services, and staff expertise. We would also
like to thank members of the NYU Computation and Psycholinguistics lab, as
well as members of the Society for Human Sentence Processing community, for
insightful discussion around this work.

## References

  * Bader (1998) Markus Bader. 1998.  Prosodic influences on reading syntactically ambiguous sentences.  In Janet Dean Fodor and Fernanda Ferreira, editors, _Reanalysis in Sentence Processing_ , pages 1–46. Springer Netherlands, Dordrecht.
  * Bangalore and Joshi (1999) Srinivas Bangalore and Aravind K. Joshi. 1999.  Supertagging: An approach to almost parsing.  _Computational Linguistics_ , 25(2):237–265.
  * Davies (2008–) Mark Davies. 2008–.  The Corpus of Contemporary American English (COCA).
  * Demberg and Keller (2008) Vera Demberg and Frank Keller. 2008.  Data from eye-tracking corpora as evidence for theories of syntactic processing complexity.  _Cognition_ , 109(2):193–210.
  * Dyer et al. (2016) Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016.  Recurrent neural network grammars.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 199–209, San Diego, California. Association for Computational Linguistics.
  * Enguehard et al. (2017) Émile Enguehard, Yoav Goldberg, and Tal Linzen. 2017.  Exploring the syntactic abilities of RNNs with multi-task learning.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 3–14, Vancouver, Canada. Association for Computational Linguistics.
  * Fodor and Inoue (1994) Janet Dean Fodor and Atsu Inoue. 1994.  The diagnosis and cure of garden paths.  _Journal of Psycholinguistic Research_ , 23(5):407–434.
  * Frank (2009) Stefan Frank. 2009.  Surprisal-based Comparison between a Symbolic and a Connectionist Model of Sentence Processing.  In _Proceedings of the 31st Annual Meeting of the Cognitive Science Society_ , pages 1139––1144. Cognitive Science Society.
  * Frazier and Fodor (1978) Lyn Frazier and Janet Dean Fodor. 1978.  The sausage machine: A new two-stage parsing model.  _Cognition_ , 6(4):291–325.
  * Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018.  Colorless green recurrent networks dream hierarchically.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1195–1205, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hale (2001) John Hale. 2001.  A probabilistic Earley parser as a psycholinguistic model.  In _Second Meeting of the North American Chapter of the Association for Computational Linguistics_.
  * Hockenmaier and Steedman (2007) Julia Hockenmaier and Mark Steedman. 2007.  CCGbank: A corpus of CCG derivations and dependency structures extracted from the Penn Treebank.  _Computational Linguistics_ , 33(3):355–396.
  * Huang et al. (2022) Kuan-Jung Huang, Suhas Arehalli, Mari Kugemoto, Grusha Prasad Christian Muxica, Brian Dillon, and Tal Linzen. 2022.  SPR mega-benchmark shows surprisal tracks construction- but not item-level difficulty.  In _35th Annual Conference on Human Sentence Processing_ , Santa Cruz, California. Society for Human Sentence Processing.
  * Levy (2008) Roger Levy. 2008.  Expectation-based syntactic comprehension.  _Cognition_ , 106(3):1126–1177.
  * Lewis (1998) Richard L Lewis. 1998.  Reanalysis and Limited Repair Parsing: Leaping off the Garden Path.  In Janet Dean Fodor and Fernanda Ferreira, editors, _Reanalysis in Sentence Processing_ , pages 247–285. Springer Netherlands, Dordrecht.
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19(2):313–330.
  * Roark et al. (2009) Brian Roark, Asaf Bachrach, Carlos Cardenas, and Christophe Pallier. 2009.  Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing.  In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_ , pages 324–333, Singapore. Association for Computational Linguistics.
  * Steedman (1987) Mark Steedman. 1987.  Combinatory grammars and parasitic gaps.  _Natural Language & Linguistic Theory_, 5(3):403–439.
  * Sturt et al. (1999) Patrick Sturt, Martin J Pickering, and Matthew W Crocker. 1999.  Structural Change and Reanalysis Difficulty in Language Comprehension.  _Journal of Memory and Language_ , 40(1):136–150.
  * van Schijndel and Linzen (2021) Marten van Schijndel and Tal Linzen. 2021.  Single-stage prediction models do not explain the magnitude of syntactic disambiguation difficulty.  _Cognitive Science_ , 45(6):e12988.
  * Vaswani et al. (2016) Ashish Vaswani, Yonatan Bisk, Kenji Sagae, and Ryan Musa. 2016.  Supertagging with LSTMs.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 232–237, San Diego, California. Association for Computational Linguistics.
  * Wilcox et al. (2021) Ethan Wilcox, Pranali Vani, and Roger Levy. 2021.  A targeted assessment of incremental processing in neural language models and humans.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 939–952, Online. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Converting Surprisals to Reading Times

In order to gauge the impact of syntactic surprisal on the predicted reading
time at word $n$, $rt_{n}$, we fit four mixed effects models over the filler
data: one containing only lexical surprisal ($s^{lex}_{n}$), one containing
only syntactic surprisal ($s^{syn}_{n}$), one containing both, and one
containing neither. As reading times are sensitive to other features of the
word being read like unigram frequency ($f_{n}$), position in sentence $p$,
and length in characters ($c_{n}$), we include those variables as additional
factors in the regression. In order to account for spillover effects, where
processing difficulty from a word often surfaces in the reading times of
subsequent words, we include all of the aforementioned factors for the prior
two words. We additionally include random intercepts by item and by
participant, as well as random slopes by item for all of the surprisal fixed
effects. This gives us the following linear mixed effects model formulas:

$\displaystyle\begin{split}rt_{n}&\sim f_{n}*c_{n}+f_{n-1}*c_{n-1}\\\
&+f_{n-2}*c_{n-2}+p\\\
&+(1\mid\text{item})+(1\mid\text{participant})\end{split}$ (neither)
$\displaystyle\begin{split}rt_{n}&\sim
s^{lex}_{n}+s^{lex}_{n-1}+s^{lex}_{n-2}\\\ &+f_{n}*c_{n}+f_{n-1}*c_{n-1}\\\
&+f_{n-2}*c_{n-2}+p\\\
&+(1+s^{lex}_{n}+s^{lex}_{n-1}+s^{lex}_{n-2}\mid\text{item})\\\
&+(1\mid\text{participant})\end{split}$ (lexical)
$\displaystyle\begin{split}rt_{n}&\sim
s^{syn}_{n}+s^{syn}_{n-1}+s^{syn}_{n-2}+\\\ &+f_{n}*c_{n}+f_{n-1}*c_{n-1}\\\
&+f_{n-2}*c_{n-2}+p\\\
&+(1+s^{syn}_{n}+s^{syn}_{n-1}+s^{syn}_{n-2}\mid\text{item})\\\
&+(1\mid\text{participant})\end{split}$ (syntactic)
$\displaystyle\begin{split}rt_{n}&\sim
s^{lex}_{n}+s^{lex}_{n-1}+s^{lex}_{n-2}\\\
&+s^{syn}_{n}+s^{syn}_{n-1}+s^{syn}_{n-2}\\\ &+f_{n}*c_{n}+f_{n-1}*c_{n-1}\\\
&+f_{n-2}*c_{n-2}+p\\\ &+(1+s^{lex}_{n}+s^{lex}_{n-1}+s^{lex}_{n-2}\\\
&+s^{syn}_{n}+s^{syn}_{n-1}+s^{syn}_{n-2}\mid\text{item})\\\
&+(1\mid\text{participant})\end{split}$ (both)

These models were fit using filler data from Huang et al. (2022), and the
coefficients from each model were used to predict reading times for all of the
critical, garden path items from the corresponding surprisals, frequencies,
lengths, and positions.

###  A.2 Statistical Analysis of Predicted RTs

To analyze the predicted reading times that come from our four models of
surprisal-to-reading time conversion, we fit three separate linear mixed
effects models: one over MVRR garden paths, one over NPS garden paths, and one
over NPZ garden paths. Each model includes fixed effects of ambiguity and the
types of surprisals used in predicting reading times: syntactic surprisal
only, lexical surprisal only, both surprisals, or neither. Crucially, we
include the interaction between these two factors, representing how our choice
of surprisal-to-RT conversion model affects the size of the predicted garden
path effect. We additionally include random intercepts by item and by
participant. This results in the following mixed effects model formula:

$\displaystyle pred\\_rt$ $\displaystyle\sim ambiguity*model$
$\displaystyle+(1\mid\text{item})+(1\mid\text{participant}).$

Since we have four different models converting between surprisals and RTs, we
estimate three contrasts for the interaction term: the model with both
surprisals vs. the model with only syntactic surprisals, the model with only
syntactic surprisals vs. the model with only lexical surprisals, and the model
with only lexical surprisals vs. the model with neither surprisal. The
estimated magnitude (represented by the $\beta$ coefficient) as well as
significance of the difference for each of these contrasts is reported in the
main text in Table 1.

###  A.3 Variability in Conversion Analysis Results Across Model Instances

In order to assess the robustness of our results with respect to the
randomness in the training of our neural network models, we repeated our
analysis using surprisals generated from four instances of our LM/supertagging
model. These models differed only in the random seed used during the
initialization and training procedure. In Figure 4(b) in the main text, we
presented predicted reading times averaged across these analyses. In Figure 5
we present the same results broken out across each model instance.

Figure 5: Empirical and model-predicted readings times for the three garden
path constructions, broken out by the LM/Supertagger models used to generate
the surprisals. Bars indicate the difference between the mean readings times
for the ambiguous and unambiguous sentences across participants for each
condition. Error bars indicate bootstrapped 95% confidence intervals.