© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

# Reliability of Smartphone-Based Vibration Threshold Measurements ††thanks:
1R. A. G. Adenekan, K. T. Yoshida, and A. M. Okamura are with the Mechanical
Engineering Department and A. Gonzalez Reyes is with the Computer Science
Department at Stanford University, Stanford, CA 94305. (email: {adenekan;
kyle3<EMAIL_ADDRESS>alegre@cs.stanford.edu).††thanks: 2A. Benyoucef
is with the Mechanical Engineering Department at the University of Colorado
Denver, Denver, CO 14850. (email: anis.benyoucef@ucdenver.edu).††thanks: 3A.
E. Adenekan, Houston, TX 77083. (email: aadenekan@yahoo.com).††thanks: 4C. M.
Nunez is with the Sibley School of Mechanical and Aerospace Engineering at
Cornell University, Ithaca, NY 14853. (email: cmn97@cornell.edu).††thanks:
*This work was supported in part by pilot grants from the Precision Health and
Integrated Diagnostics Center at Stanford, the Stanford Center for Digital
Health, and the Stanford Diabetes Research Center (pilot grant #P30DK116074).
The work was also supported by the National Science Foundation Graduate
Research Fellowship Program, the Stanford Graduate Fellowship Program, and the
Stanford Summer Undergraduate Research Fellowship Program (SURF).

Rachel A. G. Adenekan1, , Kyle T. Yoshida1, ,
Anis Benyoucef2, Alejandrina Gonzalez Reyes1, Adeyinka E. Adenekan3,
Allison M. Okamura1, and Cara M. Nunez4

###### Abstract

Smartphone-based measurement platforms can collect data on human sensory
function in an accessible manner. We developed a smartphone app that measures
vibration perception thresholds by commanding vibrations with varying
amplitudes and recording user responses via (1) a staircase method that
adjusts a variable stimulus, and (2) a decay method that measures the time a
user feels a decaying stimulus. We conducted two studies with healthy adults
to assess the reliability and usability of the app when the smartphone was
applied to the hand and foot. The staircase mode had good reliability for
repeated measurements, both with and without the support of an in-person
experimenter. The app has the potential to be used at home in unguided
scenarios.

##  I Introduction

Figure 1: Smartphone setup pertaining to the studies described in Section II
and Section III for both the hand (A and C) and foot (B and D). The phone is
placed on a pillow, and users wear noise cancelling headphones playing white
noise. Detailed placement of the hand (C) and foot (D) placements are also
displayed.

Vibrotactile perception enables humans to perform sensorimotor tasks important
for activities of daily living, such as object manipulation [1] and walking
[2]. Vibrotactile perception diminishes with age, and peripheral neuropathy,
caused by diabetes, autoimmune disorders, chemotherapy, or neurological
impairments, can also result in substantial alterations [3, 4]. Diminished
vibrotactile perception poses a significant risk to individuals, increasing
the likelihood of infections, ulcers, and falls [5]. Thus, it is important to
measure and track vibrotactile perception, especially for individuals with or
at risk of developing peripheral neuropathy.

A 128 Hz clinical tuning fork (CTF) exam is the standard method used to
measure vibrotactile perception thresholds (VPTs) [6]. To perform the exam, a
clinician strikes the tines of the tuning fork, places the end of the fork on
the patient, and either counts the seconds until the patient verbally states
that they can no longer feel a vibration or, more commonly, simply records
whether or not the patient feels any vibrations. This exam often results in
inconsistently applied vibrations caused by differences in how the clinician
strikes the fork from one use to the next [7]. Therefore, this method does not
provide accurate longitudinal measurements. A measure used to more precisely
measure VPTs is a 64 Hz Rydel-Seiffer tuning fork (RSTF) exam [8]. To perform
the exam, the clinician presses the tines of the fork together, releases them,
and places the end of the fork on the individual. When the fork is vibrating,
markings on a tine create the optical illusion of a triangle, the length of
which indicates the amplitude of the vibration. The experimenter reads the
length of the illusory triangle on a scale between 0 and 8 at the time the
patient states they can no longer feel the vibration. While this semi-
quantitative approach yields more insight than the CTF, the RSTF has not been
widely adopted in clinical practice because clinicians state that its benefit
often does not justify its cost. It has mostly been used in neurology research
[9]. A precise, high-resolution, and accessible measurement tool is required
to effectively monitor changes in vibrotactile perception over time scales
that are conducive to guiding treatment and management plans.

Due to the ubiquity of smartphones and recent advancements in smartphone
haptic hardware and controls, researchers have expressed interest in using
smartphones to conduct mobile haptics experiments [10, 11, 12], and
specifically to measure VPTs [13, 14, 15]. In our recent work, we present a
smartphone app platform that utilizes psychophysics, specifically the
staircase method, to measure VPTs [16]. The results from this prior work
indicate that we can measure VPTs with our smartphone app platform which are
correlated to current clinical methods. However, to realize the full potential
of this platform to track tactile perception and possibly even measure disease
progression and regression outside of a clinic, it is imperative that the
measurements from our tool are also consistent, reliable, and able to be
conducted by individuals without guidance from an experimenter or clinician.
In this work, we expand upon our smartphone app platform and explore this
topic by conducting two human subject user studies. In Section II, we present
our smartphone app which can calculate VPTs using the previously developed
smartphone staircase method and a new smartphone decay method that mimics the
applied vibrations from a CTF or RSTF. We also present the first human subject
study we conducted in which we evaluated the reliability of our methods
compared to current clinical methods by collecting repeated measurements.
Next, in Section III, we discuss the second human subject user study we
conducted which explores if our platform can collect similar VPT measurements
when participants do not receive guidance from an experimenter. Last, we
discuss the conclusions from our work and future directions in Section IV.

##  II Study 1: Reliability of Repeated Measurements

The purpose of this study was to quantify how reliably our platform measures
VPTs. To that end, we measured how VPTs change between three repeated trials
measured via our platform (smartphone staircase and smartphone decay) and via
clinical benchmarks (RSTF and CTF) at two body locations (the hand and the
foot) in 11 participants and compared VPTs of repeated measurements for each
condition. We describe the smartphone and tuning fork measurement methods,
user study protocol, and statistical analysis, followed by results and
discussion.

###  II-A Methods

####  II-A1 Smartphone Measurement Methods

Figure 2: Repeated threshold measurements for each tool on Day 1. Individual
participants ($n=10$) are shown in shades of pink, and group level data is
shown in black. For the smartphone methods (A and B), mean and standard
deviation are shown. For the tuning fork methods (C and D), median and
quantiles (25th and 75th) are shown. Unlike the other tools, the smartphone
decay method (B) had significant variability between each of the trials on a
single day for both the hand and foot positions. Only Day 1 data is plotted in
this figure given space limitations and lack of major differences between Days
1 and 2. Statistical results for both Days 1 and 2 are provided in Table I.
Although we tested 11 participants, we only present data from 10 participants.
One participant’s results were not used in the analyses as explained in
Section II-A5

We conducted all smartphone VPT tests on an Apple iPhone XS Max, in which the
amplitude of the vibration is modified using both the staircase method
described in prior work [16] and a newly designed decay method. The decay
method is based on the exponentially decaying vibration pattern created from a
clinical tuning fork. The frequency of the phone vibrations was held constant
at 230 Hz (hapticSharpness = 1), while the variable controlling the amplitude
of the phone’s vibration, hapticIntensity, was autonomously and continuously
linearly lowered from an initial value of hapticIntensity = 1 at time = 0 to
hapticIntensity = 0 at time = 20 seconds. This continuous decay happened over
20 seconds (which is similar to the amount of time it takes a 128 Hz clinical
tuning fork’s vibrations to decay after being struck. Both hapticIntensity and
hapticSharpness are variables within Apple’s Core Haptics Framework that can
be set to any number between 0 and 1. We chose hapticSharpness = 1 (230 Hz)
and hapticIntensity = 1 (0.65g) because those are the maximum frequency and
vibration acceleration amplitude output by the smartphone we used (iPhone XS
Max) [16]. 230 Hz is also within the stimulation range of the Pacinian
corpuscles which play a key role in vibrotactile perception at high
frequencies [17]. As described in [16], linear decays in hapticIntensity
result in exponential decays in vibration amplitude (in units g) so by
linearly decaying the hapticIntensity, we are exponentially decaying the
vibration amplitude.

In the decay exam, users were instructed to say “start” when they were ready
to begin the exam. The app uses Apple’s Speech Framework, which can recognize
spoken word from recorded or live audio, to identify this command and trigger
the start of the vibrations. Users were instructed to say “stop” when they
could no longer feel the vibrations. Users who did not immediately feel a
vibration after saying “start” were instructed to say “nothing”. Upon
recognizing the “stop” command, the app recorded the current value of the
hapticIntensity and ended the trial. Upon recognizing the “nothing” response,
the app recorded that the user could not feel the vibrations and ended the
trial. After completion of all the trials, a .csv file containing the trial
data was automatically saved via Google Firebase. Trials were repeated three
times in a row at both body locations of interest (pad of the index finger and
pad of the big toe on the dominant hand/foot). The smartphone VPT ranges from
hapticIntensitity of 0.05 (smallest vibration detected) to 1 (largest possible
vibration output via Apple’s framework). As such, lower scores indicate better
perception. If the largest vibrations output by the phone (vibrations
generated by setting hapticIntensity to 1) were not felt, the threshold for
that trial is recorded as NaN because their threshold was outside the range of
the smartphone-generated vibration stimuli.

Figure 1 shows the setup for both the hand and foot positions. In both cases,
the phone was placed on a pillow because placing the phone directly on the
table or floor results in dampened vibrations and thus reduces the range of
the tool. As shown in Fig. 1C, the first crease (distal interphalangeal joint)
of the index finger was aligned with the bottom-center edge of the phone and
the rest of the fingers were curled into a fist and rested on the table.
Similarly, the crease (interphalangeal joint) of the big toe was aligned with
the bottom-center edge of the phone and the heel was in contact with the floor
(Fig. 1D). Participants wore noise-cancelling headphones playing white noise
to enforce reliance on feeling instead of hearing the vibrations.

####  II-A2 Tuning Fork Measurement Methods

We conducted tuning fork measurements using two techniques: (1) the 128 Hz CTF
exam and (2) the semi-quantitative 64 Hz RSTF exam. We tested the same two
body parts as in the smartphone setup as shown in Fig. 1. The physical setup
for both tuning forks was the same as that for the tuning fork exam described
in [16].

Following the methods described in [16], the experimenter conducted the 128 Hz
CTF exam by striking the tines of the tuning fork (Cynamed), placing the end
of the fork on the individual, and counting the seconds until the individual
verbally states that they can no longer feel a vibration. A time of 0 seconds
indicates that a person cannot feel the tuning fork’s vibrations. As such,
longer times indicate better perception.

For the RSTF exam, the experimenter pressed the tines of a 64 Hz RSTF (US
Neurologicals) together, released them, and then placed the base of the tuning
fork on the body of the participant. Participants were instructed to say
“stop” when they no longer felt any vibrations. Each participant’s RSTF score
was recorded as the number between 0 and 8 that was closest to the tip of the
illusory triangle generated by the vibration and markings on the tines, at the
time the participant said “stop”. This follows the methods used in clinical
research settings; figures that depict an RSTF in use are shown in [8]. The
RSTF score ranges from 0 (largest vibration undetected) to 8 (smallest
vibration detected). As such, higher scores indicate better perception. Prior
to collection, participants were touched with a vibrating tuning fork so they
could get a sense of what it felt like and so that they understood that the
sensation was not painful. Participants wore a blindfold so that they could
not see when the tuning fork made contact with their body, but did not wear
headphones as the sound of the tuning fork vibrations are not easily
discernible. At each body location, the RSTF score was measured three times in
a row before moving to the next body part.

####  II-A3 User Study Participants

Eleven adult participants with no known history of diabetes or other disorders
linked to peripheral neuropathy completed this study. Participant demographics
were as follows: 6 reported female and 5 reported male for their sex assigned
at birth; aged 21 to 27 (mean: 24.7, std: 1.9); all right-handed; 11 right-
and 1 left-footed; 4 were Black/of African descent, 3 were Asian, 4 were
White, 1 was Hispanic or Latino, and 1 preferred not to answer (some
participants listed multiple identities). This study was approved by the
Stanford University Institutional Review Board under Protocol 22514 and
written consent was provided by all participants. Prior to completing the
study, participants completed a pre-survey that inquired about demographic
information as well as hobbies or injuries that may impact touch sensitivity
at the hands or feet.

####  II-A4 User Study Protocol

Participants completed a two-day protocol with a one-hour session each day.
The same time block was used for each day. All four exams (smartphone
staircase, smartphone decay, RSTF, and CTF) were conducted on both days. The
ordering of the exams within each session was randomized. For each given
modality, the ordering of the body parts tested was also randomized. All
perception measurements were collected on the participant’s dominant side body
parts.

####  II-A5 Statistical Methods

Smartphone (staircase and decay) and tuning fork (CTF and RSTF) thresholds for
one participant were removed from further analysis due to a file-save error
during the staircase portion of the experiment. This resulted in ten
participants with usable data. We aimed to assess the precision of our
smartphone methods by understanding how repeatable the measurements are within
a single day by collecting 3 repeated measurements of each method. We repeated
the protocol and corresponding statistics on a second day and present
statistics from both days separately in Section II-B.

One-way repeated measures ANOVAs were used to analyze whether the VPT for the
smartphone methods is reproducible between each consecutive trials. Friedman’s
Tests (the non-parametric equivalent of the ANOVA) were used to compare the
effect of each trial on VPT for the tuning fork methods, due to lack of
normalcy and their more discrete characteristics. The repeated measures ANOVA
and Friedman’s Tests were conducted in R (Version 4.2.1) using R Studio
(Version 2022.07.2).

The quantile calculations for the tuning fork data displayed in Fig. 2 were
performed using R allowing for quantiles to be calculated for discrete and
non-normally distributed data that are unsuitable for linear interpolation. We
used R’s type 1 quantile calculation algorithm.

Intraclass correlation coefficients (ICC) with two-way mixed effects, absolute
agreement, single rater models are used to compare the precision (how similar
each of the three trials within a day are to each other) of the two smartphone
methods to each other and understand how this relates to the currently
accepted reliability of the tuning forks. The ICC calculations were performed
using IBM SPSS Statistics (Version 29.0.0.0).

###  II-B Results

TABLE I: Reliability of Repeated Measurements Statistical Results |  | Day 1 | Day 2
---|---|---|---
Location | Method | P-value | $\eta_{\text{p}}^{2}$ | W |  | ICC (Confidence Interval)
---
P-value | $\eta_{\text{p}}^{2}$ | W | ICC (Confidence Interval)
Hand | Staircase | 0.574 | 0.06 | - | 0.924 (0.802, 0.979) | 0.335 | 0.11 | - | 0.951 (0.870, 0.986)
Decay |  $<$0.001 | 0.56 | - | 0.555 (0.121, 0.851) |  $<$0.001 | 0.54 | - | 0.775 (0.363, 0.938)
CTF | 0.368 | - | 0.1 | 0.880 (0.704, 0.965) | 0.048 | - | 0.3 | 0.773 (0.456, 0.932)
RSTF | 0.368 | - | 0.1 | 0.908 (0.765, 0.974) | 0.264 | - | 0.1 | 0.905 (0.759, 0.973)
Foot | Staircase | 0.540 | 0.07 | - | 0.797 (0.535, 0.939) | 0.115 | 0.25 | - | 0.733 (0.424, 0.917)
Decay | 0.006 | 0.43 | - | 0.703 (0.323, 0.909) | 0.121 | 0.21 | - | 0.785 (0.514, 0.934)
CTF | 0.368 | - | 0.1 | 0.302 (-0.060, 0.710) | 0.196 | - | 0.2 | 0.759 (0.474, 0.925)
RSTF | 0.247 | - | 0.1 | 0.881 (0.704, 0.965) | 0.819 | - | 0.0 | 0.912 (0.769, 0.975)

Fig. 2 shows VPTs measured for three trials using each tool (Smartphone-
staircase, Smartphone-decay, RSTF, and CTF). Table I shows the results from
the described statistical analyses examining changes in VPT between trials and
calculating ICCs. $\eta_{\text{p}}^{2}$ shows the partial effect sizes from
the ANOVAs, while Kendall’s W shows the effect sizes from the Friedman’s
Tests. The ICCs and their corresponding confidence intervals show how similar
the VPT is between trials.

####  II-B1 Between-Trial Comparisons

ANOVAs and Friedman’s Tests were used to determine if there was a significant
impact of trial on the measured VPT. Significant differences between trials
would indicate that the VPT fluctuates between different measurements.

_Hand_ : The VPT was not statistically significantly different between trials
for the smartphone staircase nor the RSTF method for either Day 1 or 2. The
VPT was not statistically significantly different between trials for the CTF
method on Day 1, but was different on Day 2 (${\chi}^{2}$$(2)=6.07$,
$p=0.048$, $W=0.303$). However, post hoc pairwise comparisons using Wilcoxon
signed-rank tests revealed no statistically significant differences between
any groups. The VPT was statistically significantly different between trials
on both Day 1 ($F(2,18)=11.64$, $p<.001$, $\eta_{\text{p}}^{2}$$=0.56$) and
Day 2 ($F(2,18)=10.54$, $p<.001$, $\eta_{\text{p}}^{2}$$=0.54$) for the
smartphone decay method. Post hoc pairwise comparisons for Day 1 showed
statistically significant differences between trial 1 and trial 2 ($p=0.012$)
and between trial 1 and trial 3 ($p=0.004$). Post hoc pairwise comparisons for
Day 2 showed statistically significant differences between trial 1 and trial 3
($p=0.005$).

_Foot_ : The VPT was not statistically significantly different between trials
for the smartphone staircase, CTF, nor RSTF methods for neither Day 1 nor 2.
The VPT was statistically significantly different between trials for the
smartphone decay method on Day 1 ($F(2,18)=6.80$, $p=.006$,
$\eta_{\text{p}}^{2}$$=0.43$), but was not on Day 2. Post hoc pairwise
comparisons showed statistically significant differences between trial 1 and
trial 2 on Day 1 ($p=0.011$).

####  II-B2 Intraclass Correlation Coefficients

ICCs are statistical measures that are commonly used to assess the reliability
of clustered data. ICC values can be evaluated as follows: $>0.9$ indicates
excellent reliability, $0.75-0.9$ indicates good reliability, $0.5-0.75$
indicates moderate reliability, and values $<0.5$ indicate poor reliability
[18]. ICC values and their confidence intervals for each tool and body
location on both days are present in Table I. Below we highlight some key
results.

_Hand_ : The smartphone staircase and RSTF methods had excellent reliability
at the hand on both days, with ICC values consistently above $0.9$. The decay
method showed the lowest ICC value on Day 1 ($0.555$), though this was higher
on Day 2 ($0.755$).

_Foot_ : At the foot, the RSTF method had the highest ICC on both Days 1 and
2. The CTF method showed the lowest ICC overall at $0.302$ on Day 1. Notably,
the smartphone staircase method still showed good reliability at the foot.

###  II-C Discussion

From this user study, we aimed to evaluate the reliability of VPTs measured
from a smartphone device and to understand how the reliability of smartphone
VPTs compares to those typically used in clinical settings. Our results
indicate that although CTFs are the current standard in clinical practice,
their reliability is questionable due to variability in striking, and thus in
resulting vibration amplitudes experienced by patients (which has been alluded
to in prior work [7]). The CTF ICC values were in the good reliability range
for the hand on both days, but only good reliability on the foot on Day 2; the
ICC value for the CTF on Day 1 was the lowest across all conditions at $0.302$
indicating poor reliability. The smartphone decay method, which was designed
based on the vibrations provided by CTF and RSTF, did not produce the most
reliable measurements and significantly varied across trials within a given
session for the hand on both days and for the foot on Day 1 (although not Day
2). The ICC values for the smartphone decay method typically fell in the
moderate range, with the highest ICC value of $0.785$ occurring at the foot on
Day 2 and the lowest ICC of $0.555$ occurring at the hand on Day 1.

In contrast, VPT measurements using the RSTF and smartphone staircase methods
were never significantly different within a session and resulted in similar
ICC values mainly in the good or excellent reliability range. The measured ICC
values for the RSTF were in range with measurements obtained in prior work
[19]. The smartphone staircase method resulted in the highest ICC values on
the hand ($0.924$ for Day 1 and $0.951$ for Day 2) all in the excellent
reliability range indicating that it was the best-performing VPT measurement
method. Although the ICC values on the foot for the smartphone staircase
method are a little bit lower ($0.797$ for Day 1 and $0.733$ for Day 2), they
still indicate that the smartphone staircase method can serve as an effective
and reliable tool for measuring VPTs. One reason for the better performance of
the staircase exam compared to the decay exam is that the recorded threshold
is more likely to be affected by speech processing delays during the decay
exam (the app relies on the user saying “stop” to trigger the recording of
vibration amplitude detected). In contrast, the staircase app provides users
with a dedicated response window of $2.5$ seconds before providing the next
vibration stimulus. Furthermore, thresholds measured via increasing stimuli
are known to perform better than those measured via decaying stimuli, and the
staircase exam essentially averages increasing and decreasing stimuli [20].

To conclude, our results indicate that measuring VPTs using our smartphone
staircase method is just as reliable as one of the tools used in clinical
research (RSTF) and more reliable than the clinical practice standard (CTF).
Each VPT measurement using our smartphone staircase method takes $2$ to $3$
minutes to acquire, depending on how well the participant is able to perceive
the vibrations. The smartphone-based exam thus takes longer to complete than
the CTF or RSTF exams ($<20$ second completion time). Next, we further
evaluate the potential of the staircase method as a reliable tool when used
without the support of an in-person experimenter.

##  III Study 2: Guided vs. Unguided

The purpose of this study was to better understand how users would interact
with our tool when they do not have verbal clarifications and corrections from
the experimenter (simulating how one may use the tool during an eventual at-
home study). To that end, we measured VPTs in two usage conditions (unguided
and guided) at two body locations (the hand and the foot) in 15 participants
and compared VPTs for each condition. Below, we provide details on the
smartphone measurement methods, user study protocol, and our statistical
analysis followed by our results and discussion.

###  III-A Methods

####  III-A1 Smartphone Measurement Methods

We conducted a user study measuring the VPTs at two body positions (hand and
foot) using the smartphone staircase method from the previous study comparing
two usage conditions: (1) guided and (2) unguided. For this study, we made a
few modifications to the vibration stimuli. Namely, we increased the length of
the vibration (the Core Haptics variable “hapticDuration”) to $1.0$ s from the
previous duration of 0.1 s. To accommodate for this change in our state
machine, the interval between vibrations was randomly adjusted to a range of
$4.5-7.5$ s instead of $3-6$ s. The period of time given to respond to each
vibration was also increased to $3$ s from an original value of $2.5$ s. These
changes were based on input from our collaborating neurologists who suggested
that the original vibration duration ($0.1$ s) was quite short, and that a
value of $1.0$ s would be better for assessing vibration perception.

####  III-A2 User Study Participants

Fifteen adult participants with no known history of diabetes or other
disorders linked to peripheral neuropathy completed this study. Self-reported
demographics of the participants are as follows: 8 reported female and 7 male
for their sex assigned at birth; 5 were between the ages of 18 to 23 (mean:
20.2, std: 0.84) and 10 between the ages of 40 to 58 (mean: 48.5, std: 4.38);
all right-handed; 12 right-footed and 3 ambipedal; 2 were Black/of African
descent, 2 were Asian, 7 were White, 5 were Hispanic or Latino, 1 was Hawaiian
or other Pacific Islander, and 3 preferred not to answer (some participants
listed multiple identities). The separate age groupings above are explicitly
stated to provide granular information regarding the participant demographics
since the range of ages was quite large. However, we do not make any claims
regarding differences between the performance of the two age groups in this
current study. This study was approved by the Stanford University
Institutional Review Board under Protocol 22514, and written consent was
provided by all participants. Prior to completing the study, participants
completed a pre-survey that inquired about demographic information as well as
hobbies or injuries that may impact touch sensitivity at the hands or feet.

####  III-A3 User Study Protocol

Participants were provided an iPhone XS Max, with our app pre-loaded onto the
phone. All participants completed the unguided condition first, followed by
the guided condition. The order of hand or foot in each condition was
randomized, and measurements at each location were repeated three times with
one exception. The first location in the unguided condition was repeated four
times so that users could familiarize themselves with the basic usage of the
platform prior to beginning the experimental trials.

In the unguided portion, participants were provided a printed set of
instructions with the following step-by-step instructions:

  1. 1.

Use the provided noise canceling headphones that play white noise with volume
set at 75 percent.

  2. 2.

Place the smartphone on the provided pillow and open the app.

  3. 3.

Position your dominant index finger or big toe (experiment randomly assigned
which location to do first) on the bottom center of the smartphone with the
joint line located closest to the nail aligned with the bottom-center edge of
the phone (Participants were provided with Fig. 1).

  4. 4.

Click “Next” in the app to begin the trial with toes and fingers held in
place.

  5. 5.

Say “yes” each time a vibration is felt until the screen says that the trial
is complete.

  6. 6.

Repeat the above process three times ensuring that the finger or toe remains
in the designated position.

During this unguided phase, the experimenter took notes documenting how the
user interacted with the phone and various points of confusion or error for
the participants. The experimenter did not answer any questions to simulate
how a participant might use the app during an in-home setting. We also
recorded how the participant interacted with the phone (via video camera) so
that it could be analyzed as necessary after the experiment.

After completing the unguided condition, the experimenter answered any
questions that the participants had. The experimenter also demonstrated how to
place their finger and big toe on the phone. Then the participant completed 3
more trials at each location under the guidance of the experimenter who
corrected any errors as they arose.

Figure 3: Guided vs. unguided VPTs at the hand (left) and foot (right).
Experimenter guidance did not significantly impact measured VPTs at the hand
(p = 0.327) nor at the foot (p = 0.166). Individual subjects are shown in
shades of pink ($n=15$ for hand, $n=13$ for foot) and mean and standard
deviation are shown in black. Figure 4: Flowchart depicting the errors made by
participants during the unguided condition. Participant errors predominately
consisted of incorrect toe or finger placement, either placing only a portion
of the finger or toe on the phone or placing the finger or toe too high on the
phone (noted in the darker-colored squares). Some participants may have made
multiple mistakes (for instance, having only the toe pad, while also placing
it too high on the phone), while some participants may have made mistakes only
for a specific trial. Figure 5: The two most common errors made by
participants in the unguided condition were (a) placing only the finger pad on
the phone (instead of the finger pad and distal interphalangeal (DIP) joint)
and (b) placing the toe too high on the phone (interphalangeal (IP) joint not
aligned with bottom-center edge of phone). As a reminder, the instructed
placements are found in Figure 1.

####  III-A4 Statistical Methods

Paired t-tests were used to compare the difference between the guided and
unguided conditions at the hand and the foot. All analysis was conducted in R
(Version 4.2.1) using R Studio (Version 2022.07.2). Notes taken by the
experimenter during the trials as well as videos were evaluated to identify
the most common errors in the unguided conditions.

Although data from a total of 15 participants was collected, two were removed
from the quantitative analysis for the foot portion. One subject was removed
from the foot portions because this subject couldn’t feel any of the phone
vibrations at the foot. The other subject was removed from the foot portion
because, during the unguided portion of the foot trials, this subject placed
the phone directly on the ground instead of on the pillow, which resulted in
vibrations that were dampened to the point of not being able to be perceived
by the participant. This participant felt the foot vibrations during the
guided portion of the experiment when the experimenter was able to correct the
user’s mistake of placing the phone directly on the ground. As explained in
Section II-A1, when a participant is unable to feel the maximum amplitude
vibrations that the phone outputs (hapticIntensity = $1$), their VPT for that
trial is undefined and is thus recorded as NaN. As such, we cannot include
these subjects in quantitative analyses (Fig. 3), although we do include them
in the qualitative analysis described in Fig. 4.

###  III-B Statistical Results

Figure 3 shows the individual subject data and mean and standard deviation of
the measured VPTs for each body location when guided and unguided. Paired
t-tests using the mean VPT of the three trials for each condition yielded a
non-statistically significant difference between the guided and unguided
conditions at the hand ($t(14)=-1.01$, $p=0.327$, $d=-0.262$) or at the foot
($t(12)=-1.48$, $p=0.166$, $d=-0.409$). The mean VPT for the unguided
conditions for both the hand and foot were slightly lower (as indicated by the
negative Cohen’s d values), though not statistically significant.

###  III-C Observational Results

Figure 4 summarizes the errors that participants made during the study via a
flowchart. Figure 5 illustrates the most common errors that occurred when
using the phone. The most common error for the foot was that the toe was
placed too high on the phone (7 errors), and the most common error for the
hand was that only the finger pad (and not the pad and distal interphalangeal
(DIP) joint) were placed on the phone (5 errors). Similarly, the second most
common error for the foot was that only the toe pad (and not the pad and
interphalangeal (IP) joint) were placed on the phone (3 errors), and the
second most common error for the hand was that the finger was placed too high
on the phone (4 participants). Most of the mistakes in following the
directions occurred in terms of finger or toe placement (9 hand, 10 foot),
while only 2 errors total resulted from a procedural issue (such as forgetting
to use headphones or placing the phone on the wrong surface).

###  III-D Discussion

This lack of difference in VPT measurements between unguided and guided
conditions at both the hand and the foot shows that guidance has minimal
impact on VPT measured using the smartphone. This is promising, as it suggests
that our smartphone app could eventually be distributed in unguided settings
and yield reliable results. Although statistically insignificant, it is
interesting that the measured VPTs from the unguided portion of the study
resulted in slightly lower averages. This could be connected to the fact that
many of the errors resulted from toe or finger placement being too high on the
phone (which results in the limb having more contact with the phone). It is
also known that accelerations vary throughout the phone [11], which may have
resulted in different acceleration amplitudes that may have been more
recognizable. To ensure that data is collected via consistent body placements
even in unguided settings, we will modify the provided instructions to include
an additional top view of the foot placement on the phone. We also plan to add
an on-screen guide for foot placement (i.e. a green oval indicating where to
place the toe). The foot location is of particular importance for diagnostic
purposes such as tracking the progression of diabetic peripheral neuropathy
[21]. Hence any efforts we can make to improve data collected at the foot is
of particular interest.

##  IV Conclusion

In this work, we conducted two separate user studies to investigate the
reliability of smartphone-based vibration perception threshold measurements.
From the first study, we found that our smartphone staircase method yielded
repeatable measurements that did not significantly differ from one another and
resulted in intraclass correlation coefficients indicating good or excellent
reliability, on par with measurements from the Rydel-Seiffer tuning fork used
in neurological research and better than the standard clinical tuning fork. We
then conducted a second study using this smartphone staircase method to
evaluate if the measured values would change if the participants were not
given real-time guidance on how to use the tool, simulating how they might use
the tool to conduct measurements on themselves at home. Excitingly, we
discovered that the vibration perception threshold measurements collected when
the participants received no guidance from an experimenter did not
significantly differ from the measurements collected with assistance from an
experimenter. We also classified common errors made by participants when
conducting the measurements on themselves and presented modifications that can
be made to the provided instructions and the smartphone interface to minimize
these errors moving forward. The results from the work presented in this paper
indicate that our smartphone staircase app could eventually be distributed for
use in at-home, unguided settings for a variety of applications. Prior to
realizing our long-term goal of distributing the staircase app to be used as a
diagnostic tool, we will build upon this work by conducting more extensive
testing in diverse groups of participants including both healthy and patient
populations. We will also conduct tests in lab (enabling us to compare to
neurology clinic gold standards such as electromyography-nerve conduction
exams) and at home (allowing us to test the usability of the tool outside the
lab or clinic). We are particularly excited by the potential of this tool to
aid in measuring the progression and regression of sensory damage caused by
various diseases such as diabetic peripheral neuropathy and chemotherapy-
induced peripheral neuropathy.

## ACKNOWLEDGMENTS

We thank the participants for their time and effort. We also thank our
clinical collaborators Dr. Kenneth K. Leung and Dr. Srikanth Muppidi for their
contributions to selection of the vibration stimuli.

## References

  * [1] A. J. Brisben, S. S. Hsiao, and K. O. Johnson, “Detection of vibration transmitted through an object grasped in the hand,” _Journal of Neurophysiology_ , vol. 81, no. 4, pp. 1548–1558, 1999.
  * [2] P. Era, M. Schroll, H. Ytting, I. Gause-Nilsson, E. Heikkinen, and B. Steen, “Postural Balance and Its Sensory-Motor Correlates in 75-Year-Old Men and Women: A Cross-National Comparative Study,” _Journal of Gerontology: Medical Sciences_ , vol. 51A, no. 2, pp. M53–M63, 1996.
  * [3] P. Halonen, “Quantitative vibration perception thresholds in healthy subjects of working age.” _European Journal of Applied Physiology and Occupational Physiology_ , vol. 54, pp. 647–655, 1986.
  * [4] R. Hanewinckel, M. Ikram, and P. Van Doorn, “Peripheral neuropathies,” _Handbook of clinical neurology_ , vol. 138, pp. 263–282, 2016.
  * [5] J. L. O’Loughlin, Y. Robitaille, J.-F. Boivin, and S. Suissa, “Incidence of and risk factors for falls and injurious falls among the community-dwelling elderly,” _American Journal of Epidemiology_ , vol. 137, no. 3, pp. 342–354, 1993.
  * [6] R. Pop-Busui, A. J. Boulton, E. L. Feldman, V. Bril, R. Freeman, R. A. Malik, J. M. Sosenko, and D. Ziegler, “Diabetic Neuropathy: A Position Statement by the American Diabetes Association,” _Diabetes Care_ , vol. 40, pp. 136–154, 2017.
  * [7] T. D. Rossing, D. A. Russell, and D. E. Brown, “On the acoustics of tuning forks,” _American Journal of Physics_ , vol. 60, no. 7, pp. 620–626, 1992.
  * [8] F. B. Panosyan, J. M. Mountain, M. M. Reilly, M. E. Shy, and D. N. Herrmann, “Rydel-seiffer fork revisited: Beyond a simple case of black and white,” _Neurology_ , vol. 87, no. 7, pp. 738–740, 2016.
  * [9] M. H. Alanazy, N. A. Alfurayh, S. N. Almweisheer, B. N. Aljafen, and T. Muayqil, “The conventional tuning fork as a quantitative tool for vibration threshold,” _Muscle & Nerve_, vol. 57, no. 1, pp. 49–53, 2018.
  * [10] J. R. Blum, P. E. Fortin, F. Al Taha, P. Alirezaee, M. Demers, A. Weill-Duflos, and J. R. Cooperstock, “Getting Your Hands Dirty Outside the Lab: A Practical Primer for Conducting Wearable Vibrotactile Haptics Research,” _IEEE Transactions on Haptics_ , vol. 12, no. 3, pp. 232–246, 2019.
  * [11] K. T. Yoshida, J. X. Kiernan, R. A. G. Adenekan, S. H. Trinh, A. J. Lowber, A. M. Okamura, and C. M. Nunez, “Cognitive and Physical Activities Impair Perception of Smartphone Vibrations,” _IEEE Transactions on Haptics_ , vol. 16, no. 4, pp. 672–679, 2023.
  * [12] K. T. Yoshida, J. X. Kiernan, A. M. Okamura, and C. M. Nunez, “Exploring Human Response Times to Combinations of Audio, Haptic, and Visual Stimuli from a Mobile Device,” in _Proc. IEEE World Haptics_ , 2023. [Online]. Available: https://arxiv.org/pdf/2305.17180.pdf
  * [13] R. A. G. Adenekan, A. J. Lowber, B. N. Huerta, A. M. Okamura, K. T. Yoshida, and C. M. Nunez, “Feasibility of Smartphone Vibrations as a Sensory Diagnostic Tool,” _Haptics: Science, Technology, Applications_ , pp. 337–339, 2022.
  * [14] W. O. Torres, M. E. Abbott, Y. Wang, and H. S. Stuart, “Skin sensitivity assessment using smartphone haptic feedback,” _IEEE Open Journal of Engineering in Medicine and Biology_ , vol. 4, pp. 216–221, 2023.
  * [15] O. Lindsay, H. Hammad, J. Baysic, A. Young, N. Osman, R. Ferber, N. Culos-Reed, and R. Peters, “Healthcare monitoring at your fingertips: validation of a smartphone alternative to routine measures of skin sensibility,” manuscript submitted for publication. [Online]. Available: https://www.researchsquare.com/article/rs-3694234/latest
  * [16] R. A. G. Adenekan, A. G. Reyes, K. T. Yoshida, S. Kodali, A. M. Okamura, and C. M. Nunez, “A comparative analysis of smartphone and standard tools for touch perception assessment across multiple body sites,” _IEEE Transactions on Haptics, Accepted for publication_ , 2024. [Online]. Available: https://arxiv.org/abs/2401.07211
  * [17] R. T. Verrillo, “Vibration sensation in humans,” _Music Perception_ , vol. 9, no. 3, pp. 281–302, 1992.
  * [18] T. K. Koo and M. Y. Li, “A guideline of selecting and reporting intraclass correlation coefficients for reliability research,” _Journal of chiropractic medicine_ , vol. 15, no. 2, pp. 155–163, 2016.
  * [19] A. Marcuzzi, A. C. Wainwright, D. S. Costa, and P. J. Wrigley, “Vibration testing: optimizing methods to improve reliability,” _Muscle & Nerve_, vol. 59, no. 2, pp. 229–235, 2019.
  * [20] J. Goldberg and U. Lindblom, “Standardised method of determining vibratory perception thresholds for diagnosis and screening in neurological investigation.” _Journal of Neurology, Neurosurgery & Psychiatry_, vol. 42, no. 9, pp. 793–803, 1979.
  * [21] N. A. ElSayed, G. Aleppo, V. R. Aroda, R. R. Bannuru, F. M. Brown, D. Bruemmer, B. S. Collins, C. H. Gibbons, J. M. Giurini, M. E. Hilliard _et al._ , “12. retinopathy, neuropathy, and foot care: Standards of care in diabetes—2023,” pp. S203–S215, 2023.


# Revisiting Supertagging for HPSG

Olga Zamaraeva, Carlos Gómez-Rodríguez
Universidade da Coruña, CITIC
Departamento de Ciencias de la Computación y Tecnologías de la Información
Campus de Elviña s/n, 15071, A Coruña, Spain
{olga.zamaraeva<EMAIL_ADDRESS>
###### Abstract

We present new supertaggers trained on HPSG-based treebanks. These treebanks
feature high-quality annotation based on a well-developed linguistic theory
and include diverse and challenging test datasets, beyond the usual WSJ
section 23 and Wikipedia data. HPSG supertagging has previously relied on
MaxEnt-based models. We use SVM and neural CRF- and BERT-based methods and
show that both SVM and neural supertaggers achieve considerably higher
accuracy compared to the baseline. Our fine-tuned BERT-based tagger achieves
97.26% accuracy on 1000 sentences from WSJ23 and 93.88% on the completely out-
of-domain _The Cathedral and the Bazaar_ (_cb_)). We conclude that it
therefore makes sense to integrate these new supertaggers into modern HPSG
parsers, and we also hope that the diverse and difficult datasets we used here
will gain more popularity in the field. We contribute the complete dataset
reformatted for token classification.

##  1 Introduction

We present new supertaggers for English which can be used in particular to
improve parsing efficiency for HPSG grammars. Head-Driven Phrase Structure
Grammar (Pollard and Sag, 1994, HPSG) is a theory of syntax that has been
applied in computational linguistic research (see Bender and Emerson 2021
§3-§4 for a recent overview). At the core of such research are precision
grammars which encode a strict notion of grammaticality — their purpose is to
only cover and generate grammatical structures. They include a relatively
small set of phrase-structure rules and a large lexicon where lexical entries
contain information about the word’s syntactic behavior. HPSG treebanks (and
the grammars that produce them) encode not only constituency but also
dependency and semantic relations and have proven useful in natural language
processing, e.g. in grammar coaching (Flickinger and Yu, 2013; da Costa et
al., 2016; Da Costa et al., 2020), natural language generation (Hajdik et al.,
2019), and creating training data for parsers (Buys and Blunsom, 2017; Chen et
al., 2018).

HPSG parsing is relatively slow, sometimes prohibitively so for long sentences
(Oepen and Carroll, 2002; Marimon et al., 2014). Thus the true potential of
HPSG parsing in NLP remains not fully realized. Approaches to speed up HPSG
parsing include local ambiguity packing (Tomita, 1985; Malouf et al., 2000;
Oepen and Carroll, 2002), on the one hand, and forgoing exact search and
reducing the parser search space, on the other (Dridan et al., 2008; Dridan,
2009, 2013). Here we contribute to the second line of research, aka
supertagging, a technique to discard unlikely interpretations of tokens.
Dridan et al. (2008); Dridan (2009, 2013) used maximum entropy-based models
trained on a combination of gold and automatically labeled data from English,
and they report an efficiency improvement of a factor of 3 for the parser they
worked with (Callmeier, 2000).

We present new models for HPSG supertagging, an SVM-based one, a neural CRF-
based one, and a fine-tuned-BERT one, and compare their tagging accuracy with
a MaxEnt baseline. We now have more English gold training data thanks to the
HPSG grammar engineering consortium’s treebanking efforts (Flickinger, 2000;
Oepen et al., 2004; Flickinger, 2011; Flickinger et al., 2012).111The data is
available as part of the 2020 release of the English Resource Grammar (the
ERG): https://github.com/delph-in/docs/wiki/RedwoodsTop. It makes sense to
train modern models on this wealth of gold data. Once more accurate taggers
such as the one we present here are integrated into HPSG parsers, the parsers
can be used to create even larger HPSG treebanks and also more widely in NLP,
leading to higher precision and interpretability and reducing computational
resources spent in training of purely statistical and neural parsers.

We trained the neural models with NVIDIA GeForce RTX 2080 GPU, CUDA version
11.2. The SVN model and the MaxEnt baseline were trained using Intel Core
i7-9700K 3,60Hz CPU. Further details can be found in the Appendix. The code
and configurations for the reported results are on
GitHub.222https://github.com/olzama/neural-supertagging The data we used is
publicly available.1 The details about model training and reproducibility are
in the Appendix.

##  2 Background

Below we explain HPSG lexical types (§2.1), and in §2.2, we give the
background on the English treebanks which served as our training and
evaluation data and the quality of which we think is key to our results. §2.3
is a summary for HPSG parsing.

###  2.1 Lexical types

Any HPSG grammar consists of a hierarchy of types, including phrasal and
lexical types, and of a large lexicon which can be used to map surface tokens
to lexical types. Each token in the text is recognized by the parser as
belonging to one or more of the lexical entries in the lexicon (assuming such
an orthographic form is present at all). Lexical entries, in turn, belong to
lexical types. Lexical types are similar to part-of-speech (POS) tags but are
more fine grained (e.g. a precision grammar may distinguish between multiple
types of proper nouns or multiple types of _wh_ -words, etc). After the
lexical analysis stage, the bottom-up parser runs a constraint unification-
based algorithm (Carpenter, 1992) to return a (possibly empty) set of possible
parses for the sentence.

###  2.2 The ERG treebanks

The English Resource Grammar (ERG; Flickinger, 2000, 2011) is a broad-coverage
precision grammar of English implemented in the HPSG formalism. The latest
release is from 2023. Its intrinsic evaluation relies on a set of English text
corpora. Each release of the ERG includes a treebank of those texts parsed by
the current version. The parses are created automatically and treebanked
manually. Treebanking in the ERG context is the process of choosing
linguistically (semantically) correct structures from the multiple trees
corresponding to one string that the grammar may produce. In the case of the
ERG, most of the treebanking was done by its main developer, Dan Flickinger.
Fast treebanking is made possible by automatically comparing parse forests and
by discriminant-based bulk elimination of unwanted trees (Oepen, 1999;
Packard, 2015). The treebanks are stored as databases that require specialized
software for processing e.g. Pydelphin333https://pydelphin.readthedocs.io/.

The 2020 ERG release comes with 30 treebanked corpora containing over 1.5
million tokens and 105,155 sentences. In principle, there are 43,505 different
lexical types in the ERG (cf. 48 tags in the Penn Treebank POS tagset (PTB;
Marcus et al., 1993)) however only 1299 of them are found in the training
portion of the treebank. The genres include well-edited text (news, Wikipedia
articles, fiction, travel brochures, and technical essays) as well as customer
service emails and transcribed phone conversations. It also includes
constructed test suites illustrating linguistic phenomena such as raising and
control.444https://github.com/delph-in/docs/wiki/RedwoodsTop As such, these
treebanks present more challenging test data compared to the conventional
WSJ23 (which is also included). The ERG 2020’s average treebanked coverage
over all the corpora is 92.95% (raw coverage before treebanking is 96.14%).
The ERG uses PTB-style punctuation tokens and includes PTB POS tags in all
tokens, along with a lexical type (§2.1).

###  2.3 HPSG parsing

Several parsers for different variations of the HPSG formalism exist. We work
with the DELPH-IN formalism (Copestake, 2002) which is deliberately
restricted; it only encodes the unification operation natively. The DELPH-IN
restrictions were motivated by efficiency considerations, among other things,
though its worst-case complexity is intractable Oepen and Carroll (2002).
Carroll (1993, §3.2.3) (cited in Bender and Emerson 2021, p.1109) states that
the worst-case parsing time for HPSG feature structures is proportional to
$C^{2}n^{\rho+1}$ where $\rho$ is the maximum number of children in a phrase
structure rule and C is the (potentially large) maximum number of feature
structures. The unification operator takes two feature structures as input and
outputs one feature structure which satisfies the constraints encoded in both
inputs. Given the complex nature of such structures, implementing a fast
unification parser is a hard problem. As it is, the existing parsers may take
prohibitively long to parse a long sentence (see e.g. Marimon et al., 2014).

##  3 Methodology

Supertagging (Bangalore and Joshi, 1999) reduces the parser search space by
discarding the less likely interpretations of an orthography. For example, the
word _bark_ in English can be a verb or a noun, and in _the dog barks_ it is a
lot less likely to be a noun than a verb. In HPSG, there are fine-grained
lexical types within the POS class (e.g. subtypes of common nouns or _wh_
-words), so the search space can be reduced further compared to POS tagging.

model grammar training tok tagset size speed-up factor N-gram (Prins and van
Noord, 2004) Alpino (Dutch) 24 mln 1365 2 HMM (Blunsom, 2007, p. 167) ERG
(English) 113K 615 8.5 MEMM (Dridan, 2009, p. 169) ERG (English) 158K 676 12

Table 1: Supertagging effects on HPSG parsing speed.

dataset description sent tok train tok MaxEnt SVM NCRF++ BERT D2009 cb
technical essay 713 17,244 0 88.96 89.53 91.94 93.88 74.61 ecpr e-commerce
1088 11,550 24,934 91.80 91.99 95.09 96.09 jh*,tg*,ps*, ron* travel brochures
2116 34,098 147,166 90.45 91.21 95.44 96.11 91.47 petet textual entailment 581
7135 1578 92.88 95.31 96.93 97.71 vm32 phone conv. 1000 8730 86,630 93.57
94.29 95.62 96.64 ws213-214 Wikipedia 1470 29,697 161,623 91.31 92.02 93.66
95.59 wsj23 Wall Street J. 950 22,987 959,709 94.27 94.72 96.05 97.26 all all
test sets as one 7,918 131,441 1,381,645 91.57 92.28 94.46 96.02 all average
7,918 131,441 1,381,645 91.89 92.72 94.96 96.18 speed (sen/sec) average 7,918
131,441 1,381,645 1024 7414 125 346

Table 2: Baseline (MaxEnt) and experimental supertaggers’ accuracy and speed
on test data; tagset size is 1299.

In precision grammars, this methodology comes at a cost to coverage; selecting
a wrong lexical type means the entire sentence will not be parsed correctly.
Thus accuracy is important.

###  3.1 Previous and related work

Bangalore and Joshi (1999) introduced the concept of supertagging. Clark and
Curran (2003) showed mathematically that supertagging improves parsing
efficiency for a lexicalized formalism (CCG). They used a maximum entropy
model; Xu et al. (2015) introduced a neural supertagger for CCG. Vaswani et
al. (2016) and Tian et al. (2020) further improved the accuracy of neural-
based CCG supertagging achieving an accuracy of 96.25% on WSJ23. Liu et al.
(2021) use finer categories within the CCG tagset and report 95.5% accuracy on
in-domain test data and 81% and 92.4% accuracy on two out-of-domain datasets
(Bioinfer and Wikipedia).555These works do not report experiments on parsing
speed.

Supertagging experiments with HPSG parsing speed using hand-engineered
grammars (Prins and van Noord, 2004) are summarized in Table 1. In addition,
there were experiments on the use of supertagging for parse ranking with
statistically derived HPSG-like grammars (Ninomiya et al., 2007; Matsuzaki et
al., 2007; Miyao and Tsujii, 2008; Zhang et al., 2009, 2010; Zhang and
Krieger, 2011; Zhang et al., 2012). These statistically derived systems are
principally different from the ERG as they do not represent HPSG theory as
understood by syntacticians. In the context of the ERG, Dridan et al. 2008
represents our baseline SOTA.666Dridan 2013 is a related work on
“ubertagging”, which includes multi-word expressions.

###  3.2 Data

We train and evaluate our taggers, both for the baseline (§3.3) and for the
experiment (§3.4), on gold lexical types from the ERG 2020 release (§2.2). We
use the train-dev-test split recommended in the release.777Download
redwoods.xls from the ERG repository for details and see
https://github.com/delph-in/docs/wiki/RedwoodsTop. This split is different
than in Dridan 2009. There are 84,894 sentences in the training data (83,607
after removing one-word sentences), 2,045 in dev, and 7,918 in test. WSJ
section 23 is used as test data, as is traditional, but so are a number of
other corpora, notably _The Cathedral and the Bazaar_ (Raymond, 1999), a
technical essay which serves as the out-of-domain test data. See Table 2 for
the details about the data.

###  3.3 Baseline

For our baseline, we use a MaxEnt model similar to Dridan 2009. While Dridan
(2009) used off-the-shelf TnT (Brants, 2000) and C&C (Clark and Curran, 2003)
taggers, we use the off-the-shelf logistic regression library from scikit-
learn (Pedregosa et al., 2011) which is a popular and relatively recent off-
the-shelf tool for classic machine learning algorithms. The baseline tagger
accuracy is included in Table 2 in §4. The details how the best baseline model
was chosen are in Appendix A. Here, we note that we tried autoregressive and
non-autoregressive models and, since non-autoregressive were not inferior in
terms of accuracy and were much faster in decoding, we use a non-
autoregressive MaxEnt model as our baseline.

###  3.4 SVM, LSTM+CRF, and fine-tuned BERT

We train a liblinear SVM model with default parameters (L2 Squared Hinge loss,
C=1, one-v-rest, up to 1000 training iterations) using scikit-learn library.
To train an lstm sequence labeling model, we use the NCRF++ library (Yang and
Zhang, 2018). We choose the model by training and validating 31 models up to
100 iterations with the starting learning rate of 0.009 and the batch size of
3 (the latter parameters are the largest that are feasible for the combination
of our data and the library code). The best NCRF++ model is described in the
Appendix in Table 3. To fine-tune BERT, we use the Huggingface transformers
library (Wolf et al., 2019) and Pytorch (Paszke et al., 2017). We try both
‘base-bert-cased’ and ‘base-bert-uncased’ pretrained models which we fine-tune
for up to 50 epochs (stopping once there is no improvement for 5 epochs) with
weight decay=0.01. The ‘cased’ model with learning rate 2e-5 achieves the best
accuracy on the dev set (see Table 7 in the Appendix).

##  4 Results: Tagger accuracy and speed

The results are presented in Table 2. The names of the datasets are given as
in the ERG2020 release. The number of sentences is the number of sentences for
which there is a gold parse. The number of tokens is given from similar
datasets used for training — e.g. for WSJ23 the ‘training tok’ number
corresponds to WSJ 1–22.

Table 2 shows that the baseline models achieve similar performance to Dridan
2009 (D2009 in Table 2) on in-domain data and are better on out-of-domain
data. This may indicate that these models are close to their maximum
performance on in-domain data on this task but adding more training data still
helps for out-of-domain data. dridan2009using models were trained on a subset
of our data. In particular, the test data did not include the ron* corpus
which was instead included in the training data. Dridan (2009, p.84) reports
getting 91.47% accuracy on the in-domain data (which loosely corresponds to
row ‘jh*, tg*, ps*, ron*’) using the TnT off-the-shelf tagger (Brants,
2000).888As a sanity check, we obtain 91.94% using an autoregressive one-
versus-rest L1 SAGA MaxEnt model trained with the scikit-learn library
(Pedregosa et al., 2011) on training and test datasets very similar to the
ones used by Dridan (2009) (this experiment is not included in Table 2). On
_The Cathedral and the Bazaar_ with the same setup, we obtain 73.85% compared
to dridan2009using 74.61%. We attribute it to the differences between TnT and
scikit-learn.

The SVM and the neural models are better than the baseline models on all test
datasets, and fine-tuned BERT is the best overall. On the portion of WSJ23 for
which we have gold data, fine-tuned BERT achieves 97.26%. The neural models
are slower than the baseline models (using GPU for decoding); on the other
hand, SVM is remarkably fast (at over 7000 sen/sec).

##  5 Conclusion and future work

We used the advancements in HPSG treebanking to train more accurate
supertaggers, which we tested on diverse data well beyond the usual WSJ23. The
ERG is a major project in syntactic theory and an important resource for
semantic parsing and grammar coaching. It has the potential to contribute to
NLP tasks that require high precision and/or interpretability, and thus making
HPSG parsing faster is strategic for NLP. The next step is integrating the new
supertagging models into state-of-the-art HPSG parsers and achieving
improvements in HPSG parsing speed allowing us to create even larger
treebanks. As the methodologies discussed here are applicable to any grammar
for which there is sufficient treebank data, with further advancements in
multilingual grammar engineering we will see higher precision in NLP for many
languages. In the meantime, we contribute the complete ERG dataset converted
to huggingface transformers format intended for token classification, along
with the code which can be adapted for other means.

##  6 Limitations

Our paper is concerned with training supertagging models on an English HPSG
treebank. The limitations therefore are associated mainly with the training of
the models including neural networks, and with the building of broad-coverage
grammars such as the English Resource Grammar. Crucially, while our method
does not require industry-scale computational resources, training a neural
classifier such as ours still requires a certain amount of training data, and
this means that our method assumes that a large HPSG treebank is available for
training. The availability of such a treebank, in turn, depends directly on
the availability of a broad-coverage grammar. While choosing the gold trees
for the treebank can be done relatively fast using treebanking tools once the
grammar parsed the corpus, building a broad-coverage grammar itself requires
an investment of years of expert work. At the moment, such an investment was
made only for a few languages (English, Spanish, Japanese, Chinese), English
being the largest one. Furthermore, the coverage of a precision grammar is
never perfect and regular grammar updates are needed. A limitation related to
using neural networks is that while the NCRF++ library can in principle be
very efficient on some tasks (e.g. POS tagging), with our data and large label
set it proved relatively slow, and so ideally a more efficient neural
architecture may be required for future work in this direction.

## Acknowledgments

We acknowledge the European Research Council (ERC), which has funded this
research under the Horizon Europe research and innovation programme (SALSA,
grant agreement No 101100615, and GAUSS, grant agreement No 101063104),
ERDF/MICINN-AEI (SCANNER-UDC, PID2020-113230RB-C21), Xunta de Galicia (ED431C
2020/11), and Centro de Investigación de Galicia “CITIC”, funded by the Xunta
de Galicia through the collaboration agreement between the Consellería de
Cultura, Educación, Formación Profesional e Universidades and the Galician
universities for the reinforcement of the research centres of the Galician
University System (CIGUS).

## References

  * Anthony et al. (2020) Lasse F. Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. 2020.  Carbontracker: Tracking and predicting the carbon footprint of training deep learning models.  ICML Workshop on Challenges in Deploying and monitoring Machine Learning Systems.  ArXiv:2007.03051.
  * Bangalore and Joshi (1999) Srinivas Bangalore and Aravind Joshi. 1999.  Supertagging: An approach to almost parsing.  _Computational linguistics_ , 25(2):237–265.
  * Bender and Emerson (2021) Emily M Bender and Guy Emerson. 2021.  Computational linguistics and grammar engineering.  In Stephan Müller, Anne Abeillé, Robert D. Borsley, and Jean-Pierre Koenig, editors, _Head-Driven Phrase Structure Grammar: The handbook_.
  * Blunsom (2007) Philip Blunsom. 2007.  _Structured classification for multilingual natural language processing_.  Ph.D. thesis, University of Melbourne.
  * Brants (2000) Thorsten Brants. 2000.  TnT-a statistical part-of-speech tagger.  _arXiv preprint cs/0003055_.
  * Buys and Blunsom (2017) Jan Buys and Phil Blunsom. 2017.  Robust incremental neural semantic graph parsing.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1215–1226.
  * Callmeier (2000) Ulrich Callmeier. 2000.  PET–a platform for experimentation with efficient hpsg processing techniques.  _Natural Language Engineering_ , 6(1):99–107.
  * Carpenter (1992) Robert Carpenter. 1992.  _The logic of typed feature structures: with applications to unification grammars, logic programs and constraint resolution_ , volume 32.  Cambridge University Press.
  * Carroll (1993) John Carroll. 1993.  _Practical unification-based parsing of natural language_.  Ph.D. thesis, University of Cambridge.
  * Chen et al. (2018) Yufei Chen, Weiwei Sun, and Xiaojun Wan. 2018.  Accurate SHRG-based semantic parsing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 408–418, Melbourne, Australia. Association for Computational Linguistics.
  * Clark and Curran (2003) Stephen Clark and James R Curran. 2003.  Log-linear models for wide-coverage CCG parsing.  In _Proceedings of the 2003 conference on Empirical methods in natural language processing_ , pages 97–104.
  * Copestake (2002) Ann Copestake. 2002.  Definitions of typed feature structures.  In Stephan Oepen, Dan Flickinger, Jun-ichi Tsujii, and Hans Uszkoreit, editors, _Collaborative Language Engineering_ , pages 227–230. CSLI Publications, Stanford, CA.
  * da Costa et al. (2016) Luis Morgado da Costa, Francis Bond, and Xiaoling He. 2016.  Syntactic well-formedness diagnosis and error-based coaching in computer assisted language learning using machine translation.  In _Proceedings of the 3rd Workshop on Natural Language Processing Techniques for Educational Applications (NLPTEA2016)_ , pages 107–116.
  * Da Costa et al. (2020) Luis Morgado Da Costa, Roger VP Winder, Shu Yun Li, Benedict Christopher Lin Tzer Liang, Joseph Mackinnon, and Francis Bond. 2020.  Automated writing support using deep linguistic parsers.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 369–377.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Dridan (2009) Rebecca Dridan. 2009.  _Using lexical statistics to improve HPSG parsing_.  Ph.D. thesis, University of Saarland.
  * Dridan (2013) Rebecca Dridan. 2013.  Ubertagging: Joint segmentation and supertagging for english.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1201–1212.
  * Dridan et al. (2008) Rebecca Dridan, Valia Kordoni, and Jeremy Nicholson. 2008.  Enhancing performance of lexicalised grammars.  In _Proceedings of ACL-08: HLT_ , pages 613–621.
  * Flickinger (2000) Dan Flickinger. 2000.  On building a more efficient grammar by exploiting types.  _Natural Language Engineering_ , 6(01):15–28.
  * Flickinger (2011) Dan Flickinger. 2011.  Accuracy v. robustness in grammar engineering.  In Emily M. Bender and Jennifer E. Arnold, editors, _Language from a Cognitive Perspective: Grammar, Usage and Processing_ , pages 31–50. CSLI Publications, Stanford, CA.
  * Flickinger and Yu (2013) Dan Flickinger and Jiye Yu. 2013.  Toward more precision in correction of grammatical errors.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_ , pages 68–73.
  * Flickinger et al. (2012) Dan Flickinger, Yi Zhang, and Valia Kordoni. 2012.  Deepbank. a dynamically annotated treebank of the wall street journal.  In _Proceedings of the 11th International Workshop on Treebanks and Linguistic Theories_ , pages 85–96.
  * Hajdik et al. (2019) Valerie Hajdik, Jan Buys, Michael W Goodman, and Emily M Bender. 2019.  Neural text generation from rich semantic representations.  In _Proceedings of NAACL-HLT_ , pages 2259–2266.
  * Liu et al. (2021) Yufang Liu, Tao Ji, Yuanbin Wu, and Man Lan. 2021.  Generating CCG categories.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 13443–13451.
  * Malouf et al. (2000) Robert Malouf, John Carroll, and Ann Copestake. 2000.  Efficient feature structure operations without compilation.  _Natural Language Engineering_ , 6(1):29–46.
  * Marcus et al. (1993) Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of English: The Penn Treebank.  University of Pennsylvania Department of Computer and Information Science Technical Report No. MS-CIS-93-87.
  * Marimon et al. (2014) Montserrat Marimon, Núria Bel, and Lluís Padró. 2014.  Automatic selection of hpsg-parsed sentences for treebank construction.  _Computational Linguistics_ , 40(3):523–531.
  * Matsuzaki et al. (2007) Takuya Matsuzaki, Yusuke Miyao, and Jun’ichi Tsujii. 2007.  Efficient hpsg parsing with supertagging and cfg-filtering.  In _Proceedings of the 20th international joint conference on Artifical intelligence_ , pages 1671–1676.
  * Miyao and Tsujii (2008) Yusuke Miyao and Jun’ichi Tsujii. 2008.  Feature forest models for probabilistic HPSG parsing.  _Computational linguistics_ , 34(1):35–80.
  * Ninomiya et al. (2007) Takashi Ninomiya, Takuya Matsuzaki, Yusuke Miyao, and Jun’ichi Tsujii. 2007.  A log-linear model with an n-gram reference distribution for accurate hpsg parsing.  In _Proceedings of the 10th International Conference on Parsing Technologies_ , pages 60–68. Citeseer.
  * Oepen (1999) Stephan Oepen. 1999.  [incr tsdb ()] competence and performance laboratory. User and reference manual.
  * Oepen and Carroll (2002) Stephan Oepen and John Carroll. 2002.  Efficient parsing for unification-based grammars.  In Stephan Oepen, Dan Flickinger, Jun-ichi Tsujii, and Hans Uszkoreit, editors, _Collaborative Language Engineering_. CSLI Press.
  * Oepen et al. (2004) Stephan Oepen, Dan Flickinger, Kristina Toutanova, and Christopher D Manning. 2004.  LinGO Redwoods.  _Research on Language and Computation_ , 2(4):575–596.
  * Packard (2015) Woodley Packard. 2015.  Full-forest treebanking.  Master’s thesis, University of Washington.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017.  Automatic differentiation in pytorch.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pollard and Sag (1994) Carl Pollard and Ivan A. Sag. 1994.  _Head-Driven Phrase Structure Grammar_.  Studies in Contemporary Linguistics. The University of Chicago Press and CSLI Publications, Chicago, IL and Stanford, CA.
  * Prins and van Noord (2004) RP Prins and GJM van Noord. 2004.  Reinforcing parser preferences through tagging.  _Traitement Automatique des Langues_ , 3:121–139.
  * Ratnaparkhi et al. (1996) Adwait Ratnaparkhi et al. 1996.  A maximum entropy model for part-of-speech tagging.  In _EMNLP_ , volume 1, pages 133–142. Citeseer.
  * Raymond (1999) Eric Raymond. 1999.  The cathedral and the bazaar.  _Knowledge, Technology & Policy_, 12(3):23–49.
  * Tian et al. (2020) Yuanhe Tian, Yan Song, and Fei Xia. 2020.  Supertagging combinatory categorial grammar with attentive graph convolutional networks.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6037–6044.
  * Tomita (1985) Masaru Tomita. 1985.  An efficient context-free parsing algorithm for natural languages.  In _IJCAI_ , volume 2, pages 756–764.
  * Vaswani et al. (2016) Ashish Vaswani, Yonatan Bisk, Kenji Sagae, and Ryan Musa. 2016.  Supertagging with lstms.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 232–237.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Xu et al. (2015) Wenduan Xu, Michael Auli, and Stephen Clark. 2015.  CCG supertagging with a recurrent neural network.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 250–255.
  * Yang and Zhang (2018) Jie Yang and Yue Zhang. 2018.  NCRF++: An open-source neural sequence labeling toolkit.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_.
  * Zhang et al. (2012) Yao-Zhong Zhang, Takuya Matsuzaki, and Jun’ichi Tsujii. 2012.  Structure-guided supertagger learning.  _Natural Language Engineering_ , 18(2):205–234.
  * Zhang et al. (2009) Yao-Zhong Zhang, Takuya Matsuzaki, and Jun’ichi Tsujii. 2009.  HPSG supertagging: A sequence labeling view.  In _Proceedings of the 11th International Conference on Parsing Technologies (IWPT’09)_ , pages 210–213.
  * Zhang et al. (2010) Yao-Zhong Zhang, Takuya Matsuzaki, and Jun’ichi Tsujii. 2010.  A simple approach for HPSG supertagging using dependency information.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 645–648.
  * Zhang and Krieger (2011) Yi Zhang and Hans-Ulrich Krieger. 2011.  Large-scale corpus-driven PCFG approximation of an HPSG.  In _Proceedings of the 12th international conference on parsing technologies_ , pages 198–208.

##  Appendix A Appendix A

###  A.1 Tuning ranges

Parameter value default/tuned range lstm layers 2 tuned 1–4 hidden dim. 800
tuned 100–1200 word embeddings glove840B pretrained word emb. dim. 300 N/A
char emb. dim. 50 tuned 30–50 momentum 0 default dropout 0.5 default l2 1-8
default

Table 3: NCRF++ model parameters

BERT (Devlin et al., 2019) was fine-tuned using transformers (Wolf et al.,
2019) and pytorch (Paszke et al., 2017) using 4 learning rates: 1e-5, 2e-5,
3e-5, and 5e-6. Cased and uncased pretrained BERT models were tried.

###  A.2 Computational resources

We trained the neural models with a single NVIDIA GeForce RTX 2080 GPU, CUDA
version 11.2. The SVM model and the MaxEnt baseline were trained using Intel
Core i7-9700K 3,60Hz CPU (using single core processing for each model). We
have experimented with Stochastic Gradient Descent (SGD) optimizer along with
AdaGrad, Adam, and AdaDelta. The ranges for parameter values can be found in
Table 3. The decoding time (sentence per second) for the models can be found
in Table 2. The training times are presented in this Appendix in Table 4. The
energy costs as estimated by the Python library carbontracker (Anthony et al.,
2020)999https://pypi.org/project/carbontracker/ are in Table 5.

Model type | models trained for tuning | total time for all models in this row (sec)
---|---|---
SVM Scikit-learn | 1 | 3664
MaxEnt Scikit-learn | 14 | 106,922
NCRF++ | 31 | 955,500 (approx.)
BERT | 5 | 100,000 (approx.)
Table 4: Training times for models used to choose the best baseline and best experimental models Measurement | Value NCRF++ | Value BERT
---|---|---
Process used | 5.55 kWh | 3.5 kWh
Carbon emissions | 1.63 kg CO2 | 5.5 kg CO2
Equivalent km driven | 13 km | 5.5 km
Table 5: Energy cost estimate for training the final NCRF++ model in 38 epochs
(31 were trained in total, number of epochs varied) and for BERT 50 epochs

###  A.3 Development accuracies

The development (validation set) accuracies are presented in Tables 6, 7, and
8. The best models are bolded. NCRF++ has nondeterministic components, and the
average dev accuracy of the best (bolded) model in Table 7; the average
accuracy is 95.15%; standard deviation 0.07088.

Model | dev accuracy(%)
---|---
multinomial L2 SAG | 91.59
multinomial L2 SAG autoreg | 91.41
OVR L2 SAG | 91.18
OVR L2 SAG autoreg | 91.27
multinomial L2 SAGA | 91.53
multinomial L2 SAGA autoreg | 88.56
OVR L2 SAGA | 91.17
OVR L2 SAGA autoreg | 91.26
OVR L1 SAGA | 92.17
OVR L1 SAGA autoreg | 92.12
multinomial L1 SAGA | 92.12
multinomial L1 SAGA autoreg | 91.17
liblinear SVM | 92.88
Table 6: Development (validation) set accuracies for MaxEnt and SVM

optimizer lstm layers char embed hid dim embed epochs (up to 100) dev
accuracy(%) SGD 1 30 200 pretrained (NCRF++) 17 92.14 SGD 4 30 200 pretrained
(NCRF++) 23 93.20 SGD 1 30 200 random 19 92.82 SGD 1 30 200 glove840 17 93.69
SGD 4 30 200 glove840 23 94.27 SGD 1 30 250 glove840 20 93.86 SGD 4 30 400
glove840 18 94.68 SGD 4 30 100 glove840 19 93.07 SGD 2 30 200 glove840 20
94.04 SGD 4 30 1200 glove840 18 94.74 SGD 4 30 600 glove840 19 94.62 SGD 2 30
400 glove840 13 94.56 SGD 4 30 1400 glove840 16 94.68 SGD 5 30 1200 glove840
12 94.35 SGD 4 50 1200 glove840 21 94.66 SGD 3 30 1200 glove840 21 94.92 SGD 2
30 1200 glove840 19 95.01 SGD 1 30 1200 glove840 16 94.60 SGD 3 30 1400
glove840 14 94.65 SGD 3 30 600 glove840 15 94.86 SGD 2 50 1200 glove840 19
95.00 SGD 2 30 1400 glove840 8 94.57 SGD+0.3 momentum 3 30 1200 glove840 21
94.67 SGD 2 30 1200 glove840 12 94.69 SGD 2 30 1000 glove840 17 94.95 SGD 2 30
800 glove840 17 95.12 adagrad 1 30 200 glove840 17 92.00 adagrad 4 30 200
glove840 42 92.42 adam 1 30 200 glove840 1 86.85 adadelta 4 30 200 glove840
stuck on 73 did not finish

Table 7: Development (validation) set accuracies for neural models Model | dev accuracy(%)
---|---
BERT cased LR 2e-5 | 96.46
BERT cased LR 1e-5 | 96.37
BERT cased LR 3e-5 | 96.31
BERT cased LR 5e-6 | 96.34
BERT uncased LR 2e-5 | 95.97
Table 8: Development (validation) set accuracies fine-tuned BERT models

###  A.4 MaxEnt model selection

Below we describe in detail how we trained the baseline MaxEnt models. Rather
than comparing our experimental numbers with numbers obtained by Dridan
(2009), we create our own baseline because we want to be able to compare
classic models with neural models with the same amount of training data.

We experimented with autoregressive and non-autoregressive MaxEnt models and
in the end chose one MaxEnt as the baseline.

###  A.5 Feature vectors for baseline models

We construct feature vectors similarly to what is described in Dridan 2009 and
ultimately in Ratnaparkhi et al. 1996. The training vector consists of the
word orthography itself, the two previous and the two subsequent words, the
word’s POS tag, and, for autoregressive models, the two gold lexical type
labels for the two previous words. Non-autoregressive models simply do not
have the previous tag features. The test vector is the same except, for
autoregressive models, instead of the gold labels for the two previous tokens,
it has labels assigned to the two previous tokens by the model itself in the
previous evaluation steps (an autoregressive model). The word orthographic
forms come from the treebank derivation terminals obtained using the Pydelphin
library.101010https://pydelphin.readthedocs.io/ The PTB-style POS tags come
from the treebanks and they were automatically assigned by an HMM-based tagger
that is part of the ACE parser code. The POS tags provided by the parser are
per token, not per terminal, so for terminals which consist of more than one
token, we map the combination of more than one tag to a single PTB-style tag
using a mapping constructed manually by the first author for the training
data. Any combination of tags not in the training data are at test time mapped
to the first tag based on that being the most frequently correct prediction in
the training data.111111The first tag is the correct tag in about 1/3 of the
cases. In the end, we only encountered 15 unknown combinations of tags in the
entire dev and test data.

###  A.6 MaxEnt classifiers

We use scikit learn Python library (Pedregosa et al., 2011) to train the
baseline MaxEnt classifiers.121212https://scikit-
learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
The scikit learn classifiers are optimized for processing a large number of
observations. For that reason, we organized our evaluation data (dev and test)
so as to maximize the number of observations passed to the classifier at each
step. dridan2009using models were autoregressive; we also implemented
autoregressive baseline models, and in order to make them faster at test time,
we organized the evaluation data by the word’s position in the sentence. So
the classifier would first process all the first words in all sentences, then
all the second words, etc. For non-autoregressive models, which we also tried
in order to find the best-performing baseline model, we just pass the
classifier the entire list of observations in their original order.

We choose the single baseline MaxEnt model from the following types of models,
by validation on the dev set: (1) MaxEnt autoregressive models which at test
time, if more than one sentence is passed to the classifier, first classify
all first tokens in all sentences, then all second tokens, etc; (2) MaxEnt
non-autoregressive models where the observation tokens are organized in the
same way as in (1); (3) MaxEnt non-autoregressive models where tokens are not
reordered in any way and are stored consecutively. The best model happens to
be of type (3).

All models achieve above 91% accuracy on the dev set. The validation (dev)
data consists of one Wikipedia section and one e-commerce corpus (2267
sentences and 25,076 tokens total), with both domains represented also in the
training data. Our best performing MaxEnt baseline model is a non-
autoregressive ‘One-versus-rest’ (OVR) model with L1 regularization and SAGA
optimizer (92.21% accuracy on the dev set).131313We tried multinomial and OVR
models, L1 and L2 regularization, and SAG and SAGA solvers on the dev set. The
autoregressive models were not strictly better than the non-autoregressive
ones, achieving very similar accuracy on the dev set. They were also much
slower, being able to process only 2 sentences per second.