# “Are you telling me to put glasses on the dog?” Content-Grounded Annotation
of Instruction Clarification Requests in the CoDraw Dataset

Brielen Madureira1 David Schlangen1,2
1Computational Linguistics, Department of Linguistics
University of Potsdam, Germany
2German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
<EMAIL_ADDRESS>
###### Abstract

Instruction Clarification Requests are a mechanism to solve communication
problems, which is very functional in instruction-following interactions.
Recent work has argued that the CoDraw dataset is a valuable source of
naturally occurring iCRs. Beyond identifying when iCRs should be made,
dialogue models should also be able to generate them with suitable form and
content. In this work, we introduce CoDraw-iCR (v2), which extends the
existing iCR identifiers fine-grained information grounded in the underlying
dialogue game items and possible actions. Our annotation can serve to model
and evaluate repair capabilities of dialogue agents.

##  1 Introduction

If someone requests you to put glasses on a dog, you may doubt yourself: Is
that really what I am supposed to do? Before attempting that, you’d likely
seek confirmation, for instance, by posing a clarification request. In real
life, dogs do fine without glasses, but, as we see in Figure 1, that is indeed
a correct action in the context of a scene construction dialogue game.

In instruction following settings, ambiguous or underspecified instructions
may elicit clarification requests when the instruction follower realises they
cannot act properly without further information. These are Instruction
Clarification Requests (iCRs), namely CRs that occur in Clark’s 4th level of
communication (Clark, 1996), when an utterance (here, an instruction) is
understood generally, but not at the level of uptake (Schlöder and Fernández,
2014).

Figure 1: A communication problem occurring and being resolved with the aid of
clarification requests, in an instruction following interaction (CoDraw, ID
9429, CC BY-NC 4.0, scene from Zitnick and Parikh (2013)). When an instruction
is not clear enough, the instruction follower asks for clarification, in order
to act accordingly (here, placing cliparts in the scene).

Madureira and Schlangen (2023) have recently argued that the CoDraw dataset
(Kim et al., 2019) is a rich and large source of spontaneous iCRs. They
identified iCRs among all instruction follower utterances and proposed using
the annotation to model the tasks of knowing when to ask and to reply to an
iCR. However, knowing what and how to ask are also topical devices for a
competent instruction follower dialogue model. To account for that, continue
their initiative by adding information about the content and form of iCRs, in
order to allow modelling and evaluating the subsequent task of generating
iCRs, not yet explored in this corpus.

In this work, we describe our annotation procedure and present a corpus
analysis of CoDraw-iCR (v2). Our annotation complements CoDraw-iCR (v1) by
adding mood categories and by mapping each utterance to its corresponding
objects and action-related attributes. We find that iCRs are mostly posed
right after an unclear instruction and typically trigger a response at the
next turn, so dialogue models should also know how to react timely. We show
that this sample is an appealing ensemble of mostly unique surface forms
through which interesting relations in co-occurring objects and attributes
emerge, making it a handy resource for further CR research. The data is
available for the community at https://osf.io/gcjhz/.

| Category |  Description |  Values
---|---|---|---
Form | Mood |  the iCR surface form |  declarative, polar question, alternative question, wh- question, imperative, other
Locale | Source |  relation to preceding IG utterance |  bool, 1 if iCR is about instruction in immediately preceding utterance
| Response |  relation to following IG utterance |  bool, 1 if iCR gets response in immediately following utterance
Content | Quantity |  how many objects are mentioned |  one, two, many, unknown
| Objects |  which objects are mentioned |  clipart identifiers (up to five)
| Attributes |  position |  bool, 1 if iCR is about an object’s position
|  |  size |  bool, 1 if iCR is about an object’s size
|  |  orientation |  bool, 1 if iCR is about an object’s orientation
|  |  relation |  bool, 1 if iCR is about relation between objects
|  |  object disambiguation |  bool, 1 if iCR is disambiguating objects
|  |  person disambiguation |  bool, 1 if iCR is disambiguating person’s pose or facial expression
Table 1: Content-grounded schema used to annotate iCRs in the CoDraw dataset.

##  2 Related Work

Clarification Requests are a multifaceted phenomenon in dialogue, with vast
literature on categorising, documenting and modelling their various
realisations as well as their relations to other utterances and to the
context. Annotation efforts have been conducted to identify their causes
(Gabsdil, 2003; Rodríguez and Schlangen, 2004; Bohus and Rudnicky, 2005;
Benotti, 2009; Koulouri and Lauria, 2009) forms (Purver et al., 2003;
Rodríguez and Schlangen, 2004; Rieser and Moore, 2005; Deits et al., 2013;
Khalid et al., 2020; Gervits et al., 2021) and readings (Purver et al., 2003;
Gabsdil, 2003; Rodríguez and Schlangen, 2004; Bohus and Rudnicky, 2005; Rieser
and Moore, 2005; Rieser et al., 2005; Kato et al., 2013; Liu et al., 2014;
Braslavski et al., 2017; Benotti and Blackburn, 2021; Shi et al., 2022).

Still, it remains an open research area; in particular, we cannot delineate
yet to what extent CR mechanisms can be learnt via data-driven methods
(Benotti and Blackburn, 2021), and dealing with underspecifications is still
hard for pretrained language models (Li et al., 2022).

Benotti and Blackburn (2021) have recently raised awareness to the different
world modalities upon which clarifications can be grounded, like vision,
movement or physical objects. Still, few works exist that systematically map
the content of CRs to elements related to the context where they occur
(Gervits et al., 2021). Some examples are Benotti and Blackburn (2017), who
use a methodology to classify CRs according to why they make implicated
premises explicit (e.g. wrong plan, not explainable plan or ambiguous plan in
instruction giving), in a corpus that is further analysed in Benotti and
Blackburn (2021) with a recipe to detecting grounded clarifications. Gervits
et al. (2021) propose a fine-grained annotation schema for CR types related to
the environment (object location, feature, action, description, etc). The
small size of these corpora, however, does not meet the needs of current data-
driven methods.

CoDraw (Kim et al., 2019) is a dialogue game where an instruction giver, who
sees a clipart scene (Zitnick and Parikh, 2013), provides written, turn-based
instructions to a drawer, who needs to reconstruct the scene. The drawer sees
a gallery of objects (a subset of 58 cliparts) and can place, remove, resize
or flip the objects and can also ask questions when they wish. Madureira and
Schlangen (2023) have put forward a desiderata for Instruction Clarification
Requests datasets suitable for data-driven research and demonstrated that the
CoDraw-iCR (v1) dataset meets most requirements: Naturalness, specificity,
frequency, diversity and relevance (regularity, according to the authors,
requires further investigation). Its size (9.9k dialogues with more than 8k
iCRs) also makes it more suitable for neural network-based models.

##  3 Fine-Grained Annotation of iCRs

Relying on the existing iCR identification in CoDraw-iCR (v1), we hereby
introduce CoDraw-iCR (v2). It contains a fine-grained annotation of the subset
of iCR utterances in CoDraw, using categories that are expected to be directly
relevant for generation (form and content), as summarised in Table 1.

#### Motivation

In CoDraw-iCR (v1), the annotators identified an emblematic dialogue act,
whose cause and function are demarcated well: iCRs that happen on Clark’s 4th
level of communication, their source utterances (i.e. the utterance where the
communication problem originates) occur during instruction giving, and their
purpose is getting an appropriate response that enables them to decide how to
follow the instruction by making actions. Their realisation, however, evinces
many other degrees of variation. An adequate instruction follower model
playing the CoDraw dialogue game has a series of decision-making steps to
perform when it comes to iCRs. It must detect the time to ask, decide what to
ask about in terms of game objects and actions, and define the surface form to
realise the iCR. The existing annotation can directly inform the training
process for the first, but the other two are left at the mercy of the
capabilities of end-to-end models to learn them implicitly from the data.
Alternatively, our more detailed annotation can guide the learning process
more explicitly and also enrich the evaluation of generated iCRs.

#### Procedure

The annotation was performed by a fluent non-native English speaker working as
a student assistant at our lab and paid according to the national regulations.
The person is a female linguistics bachelor student who went through a
learning phase to familiarise herself with the CR literature and with CoDraw
objects and rules. The annotation instructions were given as shown in Figure
2. Utterances were then presented in an internally developed graphical user
interface where all the predefined values were available (see Figure 3, Step
Two). The immediately previous and next turns by the instruction giver were
shown as context. Some less evident cases were discussed with the author; the
main decisions are documented in the data repository. The annotation was
performed in a period of around 4 months.

#### Locale

Given that iCRs are typically a local phenomenon (Rodríguez and Schlangen,
2004; Schlangen and Fernández, 2007; Rieser and Moore, 2005), and also the
incremental nature of the task and the turn-based setting, only one utterance
before and one utterance after the iCR were presented as context. To validate
that assumption, the annotator decided whether the previous utterance is the
source and whether the next utterance is a response.

Figure 2: Instructions provided to the annotator. Figure 3: The GUI with the
iCR annotation schema.

#### Form

We follow the schema proposed by Rodríguez and Schlangen (2004) and annotate
the mood of the iCRs. For each iCR utterance, the annotator could select among
declarative, polar question, alternative question, wh- question, imperative
and other. Utterances expressing more than one mood were annotated with all
suitable categories.111Due to a limitation of the GUI script, the exact order
could not be preserved.

(a) Distribution of length

(b) Relative frequency of attributes

(c) Quantity of mentioned cliparts

(d) Ten most common moods

(e) Frequency of co-occurring attributes.

(f) Co-occurrence of thematic entity categories, distribution by row in %.

Figure 4: Overview of the distributions of annotated categories in CoDraw iCR
utterances.

#### Content

The main goal of our fine-grained annotation is to have categories grounded in
the underlying dialogue game. It is similar to the types in Gervits et al.
(2021), but adapted to CoDraw, where two aspects are relevant: Entities and
actions. Entities are the cliparts, and actions refer to the possible
manipulations of their attributes: Size, position, orientation and their
relation to the scene and to other entities. Upon a close qualitative
examination of some dialogues, we also identified iCRs being used to
disambiguate between related entities, either because multiple types exist
(e.g. trees and hats) or, in case of the boy and girl, their pose and facial
expression. Therefore, the content annotation was twofold. First, the
annotator decided whether it is possible to identify the cliparts being
mentioned in the utterance. They were grouped in an aggregate category of
quantity (one, two, many or unknown) and then listed (up to five) in the order
they occur. Besides the game cliparts, we also added a category for the
background (which is the same for all scenes). Following a similar approach in
Rodríguez and Schlangen (2004), we introduced six ambiguity classes to account
for cases when it was not possible to precisely detect the clipart, five for
specific groups of related objects (tree, ball, cloud, hat, glasses) and one
for other ambiguities. Then, non-mutually exclusive binary labels were
assigned if the iCR was about an object’s position, size, orientation,
disambiguation of object, disambiguation of person and relation to other
objects.

##  4 Corpus Analysis

In this section, we conduct a detailed examination of the annotated
utterances, among which 8,765 utterances (7,710 types) were identified as
iCRs.222The absolute numbers differ slightly from Madureira and Schlangen
(2023) due to the inclusion of the in-context annotation of duplicate types.
We present a descriptive analysis enriched with examples of each annotation
category, anchoring them in the linguistic components of the game.

###  4.1 Locale

The immediately preceding instruction giver utterance is the source utterance
(i.e. the utterance where the communication problem manifests) for 80.26% of
the iCR utterances, similar to what is reported by Purver et al. (2003) but
around 15% less than in the corpus study by Rodríguez and Schlangen (2004).
78.49% of the iCR utterances get a response from the instruction giver in the
immediately following turn. For 63.85% of them, both conditions are true. This
corroborates the assumption that CRs are usually a local phenomenon in CoDraw,
thus the context we use is enough for our purposes in the majority of the
utterances.

polar |  is girl angry?
---|---
|  large cloud is left?
|  so half sun is visible?
wh |  what are they doing?
|  which tree and what size?
|  how are the boys arms?
alternative |  is she large or small
|  is the balloon in the air or sitting on the ground?
|  right or left or center
declarative |  i don’t see a baseball available.
|  you said the tree is on the right side
|  i don’t understand where the basketball is supposed to be
imperative |  describe the boy please.
|  confirm the boy is about a half inch from the left of the scene.
|  please clarify
Table 2: Examples of each type of mood.

###  4.2 Form

The average number of tokens in iCR utterances is 8.39, around four times the
average length of all instruction follower’s utterances, which contained a
large portion of very short acknowledgements like ok. The distribution is
illustrated in Figure 4(a). They are realised in manifold surface forms,
ranging from short and generic (sorry?), to very specific (owl is med?), to
long and verbose (is the girl sitting or standing i need to know as there are
multiple options and her expression as well). Figure 4(d) shows the relative
frequency of the ten most common moods. Polar questions are the most common,
followed by wh-questions and alternative questions. Declarative and imperative
moods are much less frequent. Examples for each mood are shown in Table 2. In
11.5% of the cases, the instruction follower uses more than one mood, when
asking more than one CR in a utterance, or even integrating them in the same
sentence. Examples of utterances containing multiple moods are which way is
the bolt and is it touching the ground or above it? (asking about two
different attributes with a wh-question and an alternative question) and which
tree? apple or bushy tree? (refining the first wh-question to make que CR more
specific with an alternative).

###  4.3 Content

We now turn to analyse the two aspects of iCR content, namely entities and
manipulable attributes, which directly map to the game objects and actions. As
we will see, the utterances cover all available objects and are well
distributed among actions.

position |  campfire on right or left edge?
---|---
|  tree top and left edge cut?
|  so the girl is in the middle of the scene?
relation |  left handle touching sun?
|  where is the soda located on the table?
|  which hands are they holding things?
direction |  facing left or right?
|  which direction is rocket facing?
|  cat facing left or right
size |  ok and girl size
|  sun big?
|  are you sure the tree is large?
amb. person |  does he look happy or surprised?
|  happy or sad girl?
|  is she standing?
amb. object |  is the tree pointed at the top?
|  what is the color of hat?
|  does the cloud have rain or lightning
Table 3: Examples of each type of attribute.

#### Attributes

In Figure 4(b), we see the relative frequency of each attribute in CoDraw
iCRs. While object disambiguation is somewhat less frequent, all other types
occur more evenly, each in around 20% of the iCRs. Attributes are not mutually
exclusive: While 82.87% of the iCR utterances refer to only one attribute,
14.15% mentions two. Three and four occur together in less than 2% and are
interesting examples of very detailed iCRs. For example, is she sitting, and
is she in the sandbox or on the right of it outside? is about position,
relation and ambiguous person and how close to the bottom are their feet? i
have the crown and the boy’s hands over grass line. are they smaller? is about
position, size and relation. Their co-occurrence is depicted in Figure 4(e).
Position and direction occur very often with relation. Disambiguations of
objects occur more rarely with other attributes. Examples for each attribute
are shown in Table 3. We observe some patterns in the ways iCRs are realised
for each attribute. The ten most common initial bigrams are shown in Figure 6.
Some bigrams, especially the first ones, look very predictable given the
attribute (for instance, keywords like where, close and far are relevant for
positioning and way, facing and direction are indicative of direction). On the
other hand, is the is a versatile initial bigram which is frequent for all
attributes. This is pertinent information to be integrated during generation.

Figure 5: The ten most common tokens (excluding stopwords) associated with
each game object in iCRs. The percentages are the relative frequency over all
iCR utterances. Clipart images from Zitnick and Parikh (2013). Figure 6: Ten
most common initial bigrams for each iCR attribute category. Initial ok,
tokens are excluded. Figure 7: Dialogues with at least one iCR about each
clipart, normalised by their frequency across scenes.

#### Entities

When it comes to the game objects, the relevant aspects are quantity (how many
objects are mentioned in an iCR), frequency (which objects lead to more need
to clarify) and co-occurrences (how the relations between specific objects
need to be clarified). In terms of quantity, in Figure 4(c) we see that 58.28%
of the iCRs refer to only one object, and a considerable portion refers to two
(32.29%). The relative frequency of each object in iCRs and the most common
vocabulary associated to each of them is shown in Figure 5. The person
cliparts are the most common.333For illustration purposes, only one is shown
in Figure 5. Next comes the background, whose horizon is commonly used as a
point of reference when positioning objects. The three types of trees are also
a common subject. Multiplicity is, however, not an explaining factor on its
own, because the different hats and glasses appear much less often. To explore
that it in more detail, Figure 7 shows, for each entity, the percentage of
times they triggered at least one iCR in a dialogue among all scenes in which
they appeared. Higher percentages mean that the object is frequently involved
in communication problems. The boy and the girl, each of them occurring in
more than 90% of the scenes, very often require clarification. This is likely
due to their various poses and facial expressions that require disambiguation
and also to their relation to other cliparts (for instance, hats on their
heads or their hands holding objects). The thematic category of large objects
and some objects in multiple forms (cloud and tree) also brought on many iCRs.
Figure 9 shows the co-occurrences of cliparts, with distributions by row. We
can see clusters in ambiguous objects like balls and clouds, but semantically
related objects like shovel and sandbox, racket and baseball bat, and table
and pizza also tend to co-occur. Persons and the background occur often with
almost all other objects. In Figure 4(f), entities are grouped into thematic
categories. Besides the evident ubiquity of person and background, some less
obvious relations become clear: hats and balls, sky objects and glasses, sky
objects and trees, balls and clouds.

###  4.4 Interrelations

The distribution of attribute for each object category and mood is shown in 8.
Polar questions are often used to clarify relations, while alternative and
imperative are common to disambiguate persons. Wh-questions and declarative
are used more uniformly for all attributes. Objects frequency in iCRs is
positively correlated with their frequency in the scenes (Spearman
$\rho=0.82$). When grouped by thematic category, we see that relation is a
predominant topic for almost all groups (note, however, that relations always
include more than one clipart in the count). Glasses and hats very often
require object disambiguation. For sky objects and trees, position is also a
usual topic.

Figure 8: Co-occurrence of clipart categories and attributes with mood,
distribution by row. Figure 9: Co-occurrence of entities, distribution by row.

##  5 Conclusion

We have analysed the manifestation of Instruction Clarification Requests in
the CoDraw dataset in terms of form (mood and length) and content (entities,
attributes and their co-occurrences) and some of their interrelations. Our
findings further support the argument that the CoDraw data collection setting
was an effective means to elicit iCRs. Even in its controlled environment with
a limited number of actions and objects, the resulting iCR utterances are very
diverse in their surface form and very fertile in their substance. With the
release of the annotated data, the community gains a larger resource with
sequential, spontaneous iCRs in turn-based dialogues. We aim to encourage more
research on modelling clarification requests on instruction following
interactions, and also to enable more detailed evaluation of iCR generation.

#### Limitations

Due to the availability or resources in this project, the annotation was
performed by only one annotator, thus inter-annotator agreement could not be
measured. Still, the quality of a sample was verified by the author to during
the initial annotation phase. The scenes were not available during annotation,
so clipart annotation can contain misunderstandings. The macro categories for
ambiguous cases help alleviate that, but we still observed that
misclassifications occur in some cases. The annotator worked on utterance
level but, as we showed, some utterances contain non-iCR content or multiple
categories. A possible enhancement is to further segment utterances in order
to allow each category to be mapped to sentences or phrases (for mood and
attribute) and to tokens or phrases (for entities and their referring
expressions), and also to identify the tokens that are unrelated to the
current iCR. For the portion of the iCRs whose source utterances occur further
back in the dialogue context, or whose responses are not given in the next
turn, we currently lack annotation that would allow full examination of their
context.

## Acknowledgements

We are very thankful to our student assistants, Sebastiano Gigliobianco, for
performing the initial identification and implementing the GUI an Sophia Rauh,
for doing the fine-grained annotation.

## References

  * Benotti (2009) Luciana Benotti. 2009.  Clarification potential of instructions.  In _Proceedings of the SIGDIAL 2009 Conference_ , pages 196–205, London, UK. Association for Computational Linguistics.
  * Benotti and Blackburn (2017) Luciana Benotti and Patrick Blackburn. 2017.  Modeling the clarification potential of instructions: Predicting clarification requests and other reactions.  _Computer Speech & Language_, 45:536–551.
  * Benotti and Blackburn (2021) Luciana Benotti and Patrick Blackburn. 2021.  A recipe for annotating grounded clarifications.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4065–4077, Online. Association for Computational Linguistics.
  * Bohus and Rudnicky (2005) Dan Bohus and Alexander I. Rudnicky. 2005.  Sorry and I didn’t catch that! - an investigation of non-understanding errors and recovery strategies.  In _Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue_ , pages 128–143, Lisbon, Portugal. Special Interest Group on Discourse and Dialogue (SIGdial).
  * Braslavski et al. (2017) Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017.  What do you mean exactly? analyzing clarification questions in cqa.  In _Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval_ , CHIIR ’17, page 345–348, New York, NY, USA. Association for Computing Machinery.
  * Clark (1996) Herbert H Clark. 1996.  _Using language_.  Cambridge university press.
  * Deits et al. (2013) Robin Deits, Stefanie Tellex, Pratiksha Thaker, Dimitar Simeonov, Thomas Kollar, and Nicholas Roy. 2013.  Clarifying commands with information-theoretic human-robot dialog.  _J. Hum.-Robot Interact._ , 2(2):58–79.
  * Gabsdil (2003) Malte Gabsdil. 2003.  Clarification in spoken dialogue systems.  In _Proceedings of the 2003 AAAI Spring Symposium. Workshop on Natural Language Generation in Spoken and Written Dialogue_ , pages 28–35.
  * Gervits et al. (2021) Felix Gervits, Antonio Roque, Gordon Briggs, Matthias Scheutz, and Matthew Marge. 2021.  How should agents ask questions for situated learning? an annotated dialogue corpus.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 353–359, Singapore and Online. Association for Computational Linguistics.
  * Kato et al. (2013) Makoto P Kato, Ryen W White, Jaime Teevan, and Susan T Dumais. 2013.  Clarifications and question specificity in synchronous social q&a.  In _CHI’13 Extended Abstracts on Human Factors in Computing Systems_ , pages 913–918.
  * Khalid et al. (2020) Baber Khalid, Malihe Alikhani, and Matthew Stone. 2020.  Combining cognitive modeling and reinforcement learning for clarification in dialogue.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4417–4428, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Kim et al. (2019) Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019.  CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6495–6513, Florence, Italy. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Koulouri and Lauria (2009) Theodora Koulouri and Stanislao Lauria. 2009.  Exploring miscommunication and collaborative behaviour in human-robot interaction.  In _Proceedings of the SIGDIAL 2009 Conference_ , pages 111–119, London, UK. Association for Computational Linguistics.
  * Li et al. (2022) Haau-Sing Li, Mohsen Mesgar, André FT Martins, and Iryna Gurevych. 2022.  Asking clarification questions for code generation in general-purpose programming language.  _arXiv preprint arXiv:2212.09885_.
  * Liu et al. (2014) Alex Liu, Rose Sloan, Mei-Vern Then, Svetlana Stoyanchev, Julia Hirschberg, and Elizabeth Shriberg. 2014.  Detecting inappropriate clarification requests in spoken dialogue systems.  In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_ , pages 238–242, Philadelphia, PA, U.S.A. Association for Computational Linguistics.
  * Madureira and Schlangen (2023) Brielen Madureira and David Schlangen. 2023.  Instruction clarification requests in multimodal collaborative dialogue games: Tasks, and an analysis of the CoDraw dataset.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2303–2319, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Purver et al. (2003) Matthew Purver, Jonathan Ginzburg, and Patrick Healey. 2003.  On the means for clarification in dialogue.  In _Current and new directions in discourse and dialogue_ , pages 235–255. Springer.
  * Rieser et al. (2005) Verena Rieser, Ivana Kruijff-Korbayová, and Oliver Lemon. 2005.  A corpus collection and annotation framework for learning multimodal clarification strategies.  In _Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue_ , pages 97–106, Lisbon, Portugal. Special Interest Group on Discourse and Dialogue (SIGdial).
  * Rieser and Moore (2005) Verena Rieser and Johanna Moore. 2005.  Implications for generating clarification requests in task-oriented dialogues.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 239–246, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Rodríguez and Schlangen (2004) Kepa Joseba Rodríguez and David Schlangen. 2004.  Form, intonation and function of clarification requests in german task-oriented spoken dialogues.  In _Proceedings of Catalog (the 8th workshop on the semantics and pragmatics of dialogue; SemDial04)_.
  * Schlangen and Fernández (2007) David Schlangen and Raquel Fernández. 2007.  Beyond repair – testing the limits of the conversational repair system.  In _Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue_ , pages 51–54, Antwerp, Belgium. Association for Computational Linguistics.
  * Schlöder and Fernández (2014) Julian Schlöder and Raquel Fernández. 2014.  Clarification requests at the level of uptake.  In _Proceedings of the 18th Workshop on the Semantics and Pragmatics of Dialogue - Poster Abstracts_ , Edinburgh, United Kingdom. SEMDIAL.
  * Shi et al. (2022) Zhengxiang Shi, Yue Feng, and Aldo Lipani. 2022.  Learning to execute actions or ask clarification questions.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2060–2070, Seattle, United States. Association for Computational Linguistics.
  * Zitnick and Parikh (2013) C Lawrence Zitnick and Devi Parikh. 2013.  Bringing semantics into focus using visual abstraction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3009–3016.


# Discourse Context Predictability Effects in Hindi Word Order

Sidharth Ranjan
IIT Delhi
<EMAIL_ADDRESS>
&Marten van Schijndel
Cornell University
<EMAIL_ADDRESS>
Sumeet Agarwal
IIT Delhi
<EMAIL_ADDRESS>
&Rajakrishnan Rajkumar
IISER Bhopal
<EMAIL_ADDRESS>

###### Abstract

We test the hypothesis that discourse predictability influences Hindi
syntactic choice. While prior work has shown that a number of factors (e.g.,
information status, dependency length, and syntactic surprisal) influence
Hindi word order preferences, the role of discourse predictability is
underexplored in the literature. Inspired by prior work on syntactic priming,
we investigate how the words and syntactic structures in a sentence influence
the word order of the following sentences. Specifically, we extract sentences
from the Hindi-Urdu Treebank corpus (HUTB), permute the preverbal constituents
of those sentences, and build a classifier to predict which sentences actually
occurred in the corpus against artificially generated distractors. The
classifier uses a number of discourse-based features and cognitive features to
make its predictions, including dependency length, surprisal, and information
status. We find that information status and LSTM-based discourse
predictability influence word order choices, especially for non-canonical
object-fronted orders. We conclude by situating our results within the broader
syntactic priming literature.

##  1 Introduction

Grammars of natural languages have evolved over time to factor in cognitive
pressures related to production (Hawkins, 1994, 2000) and comprehension
(Hawkins, 2004, 2014), learnability (Christiansen and Chater, 2008) and
communicative efficiency (Jaeger and Tily, 2011; Gibson et al., 2019). In this
work, we test the hypothesis that maximization of discourse predictability
(quantified using lexical repetition surprisal and adaptive LSTM surprisal) is
a significant predictor of Hindi syntactic choice, when controlling for
information status, dependency length, and surprisal measures estimated from
$n$-gram, LSTM and incremental constituency parsing models.

Our hypothesis is inspired by a solid body of evidence from studies based on
dependency treebanks of typologically diverse languages which show that
grammars of languages tend to order words by minimizing dependency length
(Liu, 2008; Futrell et al., 2015) and maximizing their trigram predictability
(Gildea and Jaeger, 2015). Parallel to this line of work on sentence-level
word order, another strand of work has focused on discourse-level estimates of
entropy starting from the Constant Entropy Rate hypothesis (CER; Genzel and
Charniak, 2002). To overcome the major difficulty of estimating sentence
probabilities conditioned on the previous discourse context, Qian and Jaeger
(2012) approximated discourse-level entropy using lexical cues from the
previous context. In contrast, we leverage modern computational
psycholinguistic neural techniques to obtain word and sentence-level estimates
of inter-sentential discourse predictability and study the impact of these
measures on Hindi word order choices. We conclude that discourse-level priming
influences Hindi word order decisions and interpret our findings in the light
of the factors outlined by Reitter et al. (2011).

Hindi (Indo-Aryan language; Indo-European language family) has a rich case-
marking system and flexible word order, though it mainly follows SOV word
order (Kachru, 2006) as exemplified below.

[] amar ujala-ko yah _sukravar_ -ko daak-se prapt hua
Amar Ujala-acc it _friday_ -on post-inst receive be.pst.sg
Amar Ujala received it by post on _Friday_. [] yah amar ujala-ko _sukravar_
-ko daak-se prapt hua [] _sukravar_ -ko yah amar ujala-ko daak-se prapt hua

To test ordering preferences, we generated meaning-equivalent grammatical
variants (Examples 1 and 1 above) of reference sentences (Example 1) from the
Hindi-Urdu Treebank corpus of written text (HUTB; Bhatt et al., 2009) by
permuting their preverbal constituent ordering. Subsequently, we used a
logistic regression model to distinguish the original reference sentences from
the plausible variants based on a set of cognitive predictors. We test whether
fine-tuning a neural language model on preceding sentences improves
predictions of preverbal Hindi constituent order in later sentences over other
cognitive control measures. The motivation for our fine-tuning method is that,
during reading, encountering a syntactic structure eases the comprehension of
subsequent sentences with similar syntactic structures as attested in a wide
variety of languages (Arai et al., 2007; Tooley and Traxler, 2010) including
Hindi (Husain and Yadav, 2020). Our cognitive control factors are motivated by
recent works which show that Hindi optimizes processing efficiency by
minimizing lexical and syntactic surprisal (Ranjan et al., 2019) and
dependency length (Ranjan et al., 2022a) at the sentence level.

Our results indicate that discourse predictability is maximized by reference
sentences compared with alternative orderings, indicating that discourse
predictability influences Hindi word-order preferences. This finding
corroborates previous findings of adaptation/priming in comprehension (Fine et
al., 2013; Fine and Jaeger, 2016) and production (Gries, 2005; Bock, 1986).
Generally, this effect is influenced by lexical priming, but we also find that
certain object-fronted constructions prime subsequent object-fronting,
providing evidence for self-priming of larger syntactic configurations. With
the introduction of neural model surprisal scores, dependency length
minimization effects reported to influence Hindi word order choices in
previous work (Ranjan et al., 2022a) disappear except in the case of direct
object fronting, which we interpret as evidence for the Information Locality
Hypothesis (Futrell et al., 2020). Finally, we discuss the implications of our
findings for syntactic priming in both comprehension and production.

Our main contribution is that we show the impact of discourse predictability
on word order choices using modern computational methods and naturally
occurring data (as opposed to carefully controlled stimuli in behavioural
experiments). Cross-linguistic evidence is imperative to validate theories of
language processing (Jaeger and Norcliffe, 2009), and in this work we extend
existing theories of how humans prioritize word order decisions to Hindi.

##  2 Background

###  2.1 Surprisal Theory

Surprisal Theory (Hale, 2001; Levy, 2008) posits that comprehenders construct
probabilistic interpretations of sentences based on previously encountered
structures. Mathematically, the surprisal of the ${k}^{th}$ word, $w_{k}$, is
defined as the negative log probability of $w_{k}$ given the preceding
context:

${}S_{k}=-\log
P(w_{k}|w_{1...k-1})=\log\frac{P(w_{1}...w_{k-1})}{P(w_{1}...w_{k})}$ (1)

These probabilities can be computed either over word sequences or syntactic
configurations and reflect the information load (or predictability) of
$w_{k}$. High surprisal is correlated with longer reading times (Levy, 2008;
Demberg and Keller, 2008; Staub, 2015) as well as longer spontaneous spoken
word durations (Demberg et al., 2012; Dammalapati et al., 2021). Lexical
predictability estimated using _n_ -gram language models is one of the
strongest determinants of word-order preferences in both English (Rajkumar et
al., 2016) and Hindi (Ranjan et al., 2022a, 2019; Jain et al., 2018).

###  2.2 Dependency Locality Theory

Dependency locality theory (Gibson, 2000) has been shown to be effective at
predicting the comprehension difficulty of a sequence, with shorter
dependencies generally being easier to process than longer ones (Temperley,
2007; Futrell et al., 2015; Liu et al., 2017, cf. Demberg and Keller, 2008).

##  3 Data and Models

Our dataset comprises 1996 reference sentences containing well-defined subject
and object constituents from the HUTB111https://verbs.colorado.edu/hindiurdu/
corpus of dependency trees (Bhatt et al., 2009). The HUTB corpus, which
belongs to the newswire domain and contains written text in a natural
discourse context, is a human-annotated, multi-representational, and multi-
layered treebank. The dependency trees here assumes Panini’s grammatical model
where each sentence is represented as a series of modifier-modified elements
(Bharati et al., 2002; Sangal et al., 1995). Each tree in the HUTB corpus
denotes words in the sentence with nodes such that head words (modified) are
linked to the dependent words (modifier) via labelled links denoting the
grammatical relationship between word pairs.

For each reference sentence in the HUTB corpus, we created artificial variants
by permuting the preverbal constituents whose heads were linked to the root
node in the dependency tree. Inspired by grammar rules proposed in the NLG
literature (Rajkumar and White, 2014), ungrammatical variants were
automatically filtered out by detecting dependency relation sequences not
attested in the original HUTB corpus. After filtering, we had 72833 variant
sentences for our classification task. Figure 1 in Appendix A displays the
dependency tree for Example sentence 1 and explains our variant generation
procedure in more detail.

To determine whether the original word order (i.e. the reference sentence) is
preferred to the permuted word orders (i.e. the variant sentences), we
conducted a targeted human evaluation via forced-choice rating task and
collected sentence judgments from 12 Hindi native speakers for 167 randomly
selected reference-variant pairs in our data set. Participants were first
shown the preceding sentence, and then they were asked to select the best
continuation between either the reference or the variant. We found that 89.92%
of the reference sentences which originally appeared in the HUTB corpus were
also preferred by native speakers compared to the artificially generated
grammatical variants expressing similar meaning (Further details are provided
in Appendix G). Therefore, in our analyses we treat the HUTB reference
sentences as human-preferred gold orderings compared with other possible
automatically-generated constituent orderings.

###  3.1 Models

We set up a binary classification task to separate the original HUTB reference
sentences from the variants using the cognitive metrics described in Section
2. To alleviate the data imbalance between the two classes (1996 references vs
72833 variants), we transformed our data set using the approach described in
Joachims (2002). This technique converts a binary classification problem into
a pair-wise ranking task by training the classifier on the difference of the
feature vectors of each reference and its corresponding variants (see
Equations 2 and 3). Equation 2 displays the objective of a standard binary
classifier, where the classifier must learn a feature weight ($w$) such that
the dot product of $w$ with the reference feature vector ($\phi(reference)$)
is greater than the dot product of $w$ with the variant feature vector
($\phi(variant)$). This objective can be rewritten as equation 3 such that the
dot product of $w$ with the difference of the feature vectors is greater than
zero.

$w~{}\cdot~{}\phi(reference)>w~{}\cdot~{}\phi(variant)$ (2)

$w~{}\cdot~{}(\phi(reference)~{}-~{}\phi(variant))>0$ (3)

Every variant sentence in our dataset was paired with its corresponding
reference sentence with order balanced across these pairings (e.g., Example 1
would yield (1,1) and (1,1)). Thereafter, their feature vectors were
subtracted (e.g., 1-1 and 1-1), and binary labels were assigned to each
transformed data point. Reference-Variant pairs were coded as “1" and Variant-
Reference pairs were coded as “0". The alternate pair ordering thus re-
balanced our previously severely imbalanced classification task. Table 5 in
Appendix D illustrates the original and transformed values of the independent
variables.

For each reference sentence, our objective was to model the possible syntactic
choices entertained by the speaker. In each instance, the author chose to
generate the reference order over the variant, implicitly demonstrating an
order preference. If the cognitive factors in our study influenced that
decision, a logistic regression model should be able to use those factors to
predict which syntactic choice was ultimately chosen by the author. Using the
transformed features dataset labelled with 1 (denoting a preference for the
reference order) and 0 (denoting a preference for the variant order), we
trained a logistic regression model to predict each reference sentence (see
Equation 4). We report our classification results using 10-fold cross-
validation. The regression results are reported on the entire transformed test
data for the respective experiments. All experiments were done with the
Generalized Linear Model (GLM) package in $R$.

$choice\sim\begin{cases}&\text{$\delta$ dependency length +}\\\
&\text{$\delta$ trigram surp + $\delta$ pcfg surp +}\\\ &\text{$\delta$ IS
score + $\delta$ lexical repetition surp +}\\\ &\text{$\delta$ lstm surp +
$\delta$ adaptive lstm surp}\end{cases}$ (4)

Here choice is encoded by the binary dependent variable as discussed above
($1$: reference preference and $0$: variant preference). To obtain sentence-
level surprisal measures, we summed word-level surprisal of all the words in
each sentence. The values for independent variables were calculated as
follows.

  1. 1.

Dependency length: We computed a sentence-level dependency length measure by
summing the head-dependent distances (measured as the number of intervening
words) in the HUTB reference and variant dependency trees.

  2. 2.

Trigram surprisal: For each word in a sentence, we estimated its local
predictability using a 3-gram language model (LM) trained on the written
section of the EMILLE Hindi Corpus (Baker et al., 2002), which consists of 1
million mixed genre sentences, using the SRILM toolkit (Stolcke, 2002) with
Good-Turing discounting.

  3. 3.

PCFG surprisal: The syntactic predictability of each word in a sentence was
estimated using the Berkeley latent-variable PCFG parser2225-fold cross-
validated parser training and testing F1-score metrics were 90.82% and 84.95%,
respectively. (Petrov et al., 2006). 12000 phrase structure trees were created
to train the parser by converting Bhatt et al.’s HUTB dependency trees into
constituency trees using the approach described in Yadav et al. (2017).
Sentence level log-likelihood of each test sentence was estimated by training
a PCFG LM on four folds of the phrase structure trees and then testing on a
fifth held-out fold.

  4. 4.

Information status (IS) score: We automatically annotated whether each
sentence exhibited given-new ordering. The subject and object constituents in
a sentence were assigned a Given tag if its head was a pronoun or any content
word within it was mentioned in the preceding sentence. All other phrases were
tagged as New. For each sentence, IS score was computed as follows: a) Given-
New order = +1 b) New-Given order = -1 c) Given-Given and New-New = 0. For an
illustration of givenness coding, see Example A in Appendix A and the
description in Appendix B.

  5. 5.

Lexical repetition surprisal: For each word in a sentence, we accounted for
lexical priming by interpolating a 3-gram language model with a unigram cache
LM based on the history of words ($H=100$) containing only the preceding
sentence. We used the original implementation provided in the SRILM toolkit
with a default interpolation weight parameter ($\mu=0.05$; see Equations 5 and
6) based on the approach described by Kuhn and De Mori (1990). The idea is to
keep a count of recently occurring words in the sentence history and then
boost their probability within the trigram language model. Words that have
occurred recently in the text are likely to re-occur333Out of 13274 sentences
present in HUTB, 71.20% sentences contained at least one content word
previously mentioned in the preceding sentence Jain et al. (2018). in
subsequent sentences (Kuhn and De Mori, 1990; Clarkson and Robinson, 1997).

$\displaystyle
P(w_{k}|w_{1},w_{2},....w_{k-1})=\mu~{}P_{cache}(w_{k}|w_{1},w_{2},....w_{k-1})$
(5) $\displaystyle+(1-\mu)~{}{P_{trigram}(w_{k}|w_{k-2},w_{k-1})}$
$\displaystyle
P_{cache}(w_{k}|w_{k-H},w_{k-H+1},..w_{k-1})=\frac{w_{k}~{}counts_{(cache)}}{H}$
(6)

  6. 6.

LSTM surprisal: We estimated the predictability for each word according to the
entire sentence prefix using a long short-term memory language model (LSTM;
Hochreiter and Schmidhuber, 1997) trained on the 1 million written sentences
from the EMILLE Hindi corpus (Baker et al., 2002). We used the LSTM
implementation provided in the Neural Complexity toolkit (van Schijndel and
Linzen, 2018) with default hyper-parameter settings to estimate surprisal
using the neural context within each sentence. The exact parameters are as
follows: 2 LSTM layers with 200 hidden units each, 200-dimensional word
embeddings, 20 units each of learning rate, and training epoch with early
stopping. Rest other parameters were set to default
setting.444https://github.com/vansky/neural-complexity#model-parameters

  7. 7.

Discourse LSTM surprisal: We estimated the discourse predictability of each
word in the sentence using the adapt function of the neural complexity
toolkit. van Schijndel and Linzen (2018) proposed a simple way to continuously
adapt a neural LM to each successive test sentence, and found that adaptive
surprisal predicts human reading times significantly better than non-adaptive
surprisal. Their method takes a pre-trained LSTM LM, and, after generating
surprisals for a test sentence, the parameters of the LM get updated based on
the cross-entropy loss for that sentence. After that, the revised LM weights
are used to predict the next test sentence. This continuous fine-tuning
approach effectively modulates a sentence-level LSTM through discourse
priming. In our work, for each test sentence, we used our base LSTM LM and
adapted it to the immediately preceding context sentence and then used it to
generate (discourse-sensitive) surprisal values for the desired sentence. We
used an adaptive learning rate of 2 as it minimized the perplexity of the
validation data set (see Table 1).555Interestingly, van Schijndel and Linzen
(2018) found that an adaptive learning rate of 2 minimized validation
perplexity in English as well, though we leave further investigation of this
to future work.

Learning Rate | 0 | 0.002 | 0.02 | 0.2 | 2 | 20 | 200
---|---|---|---|---|---|---|---
Perplexity | 103.29 | 98.79 | 87.78 | 66.64 | 56.86 | 117.91 | $\sim 10^{9}$

Table 1: Learning rate influence on adaptive LSTM validation perplexity
($N=13274$ sentences; the initial non-adaptive model uses a learning rate of
0)

##  4 Experiments and Results

We tested the hypothesis that discourse predictability (estimated from
adaptive LSTM and lexical repetition surprisal) influences constituent
ordering in Hindi over other baseline cognitive controls, including dependency
length, information status and trigram and non-adaptive surprisal. The
adaptive LSTM surprisal had a high correlation with all other surprisal
features and a low correlation with dependency length and information status
score (see Figure 2 in Appendix C). We report the results of regression and
prediction experiments on the full data set as well as on subsets of the data
consisting of two types of non-canonical constructions.

###  4.1 Regression Analysis

Our regression results over the entire data set (Table 2) show that all of our
measures are significant predictors for the task of classifying reference and
variant sentences. The negative regression coefficients for our surprisal
metrics (including adaptive LSTM surprisal) indicate that surprisal is
consistently lower in the reference sentences compared with the competing
variants. And adding adaptive discourse LSTM surprisal into a model containing
all other predictors significantly improved the fit of our regression model
($\chi^{2}$ = 66.81; p $<$ 0.001). Thus these results support our core
hypothesis that word order choices seem to maximize discourse predictability
compared with possible alternative productions. The positive regression
coefficient for information status (IS) score indicates that reference
sentences adhere to given-new ordering. Similarly, adding IS score into a
model containing all other predictors significantly improved the fit of our
regression model ($\chi^{2}$ = 127.94; p $<$ 0.001). However, the positive
regression coefficient of dependency length suggests that reference sentences
exhibit _longer_ dependency lengths compared to their variant counterparts,
violating locality considerations. Thus dependency length might be in conflict
with (and/or overridden by) other factors like discourse priming or
information locality (see Section 6 for more discussion of this idea).

Predictor | $\hat{\beta}$ | $\hat{\sigma}$ | t
---|---|---|---
intercept | 1.50 | 0.001 | 1496.47
trigram surprisal | -0.08 | 0.005 | -14.53
dependency length | 0.02 | 0.001 | 15.55
pcfg surprisal | -0.07 | 0.002 | -39.46
IS score | 0.01 | 0.001 | 11.32
lex-rept surprisal | -0.03 | 0.005 | -5.31
lstm surprisal | -0.14 | 0.016 | -9.26
adaptive lstm surprisal | -0.13 | 0.016 | -8.18

Table 2: Regression model on full data set ($N=72833$; all significant
predictors denoted by $|$t$|$>2)

We also examined the contribution of each predictor on two non-canonical
syntactic configurations, _direct object (DO) fronted_ and _indirect object
(IO) fronted_ constructions, which have been studied extensively in the
sentence comprehension literature. Prior work has shown that salient objects
tend to occur early in the sentence, thus leading to fronting (Wierzba and
Fanselow, 2020; Kaiser and Trueswell, 2004). In the specific context of Hindi,
Vasishth (2004) examined the role of locality effects in processing these non-
canonical word orders in salient as well as non-salient contexts. He showed
that the increased distance to the verb in DO-fronted sentences leads to high
self-paced reading times at the inner-most verb as compared to its canonical
counterpart in both salient and non-salient conditions. However, in IO-fronted
constructions, he found that salient contexts alleviated the processing
difficulty which was caused by increased distance. Based on these findings, we
predict that adaptive surprisal should be more effective in IO-fronted than
DO-fronted constructions.

To test this hypothesis, we isolated reference sentences where the direct
object precedes the subject (for a DO-fronted test set) and reference
sentences where the indirect object precedes the subject (for an IO-fronted
test set) along with their context sentences. We compared both sets to paired
variants that exhibited canonical order (i.e. where the subject preceded both
objects). Tables 3a and 3b present regression results for DO- and IO-fronted
constructions respectively. These subsets constitute a very small fraction of
our dataset due to the infrequency of these constructions in Hindi. The
regression coefficient for adaptive LSTM surprisal was significantly negative
for both subsets, indicating that the non-canonical structures are more common
in the context of similarly non-canonical structures. This pattern is more
robust for IO-fronted reference sentences ($\chi^{2}$ = 90.90; p $<$ 0.001)
than for DO-fronted reference sentences ($\chi^{2}$ = 4.03; p $=$ 0.04),
validating our proposed prediction about these constructions. Coming to the
efficacy of IS scores over these two non-canonical constructions, givenness is
effective in case of DO-fronted reference sentences only ($\chi^{2}$ = 49.06;
p $<$ 0.001). Furthermore, in contrast to the IO-fronted subset, the
regression coefficient for dependency length in DO-fronted items is
significantly negative suggesting that locality considerations are limited to
constructions involving a high dependency length difference between reference
and variants,666The average dependency length difference for the DO-subset is
13.92 and for the IO-subset is 7.77 words. a similar finding to that reported
in Ranjan et al. (2022a) on the same task.

Predictor | $\hat{\beta}$ | $\hat{\sigma}$ | t
---|---|---|---
intercept | 1.49 | 0.008 | 171.18
trigram surp | -0.28 | 0.049 | -5.84
dep length | -0.05 | 0.008 | -6.22
pcfg surp | 0.001 | 0.014 | 0.12
IS score | 0.04 | 0.006 | 7.04
lex repetition surp | 0.07 | 0.044 | 1.67
lstm surp | 0.03 | 0.114 | 0.23
adaptive lstm surp | -0.23 | 0.113 | -2.00
(a) Subtable 1 list of tables text

Predictor | $\hat{\beta}$ | $\hat{\sigma}$ | t
---|---|---|---
intercept | 1.51 | 0.008 | 188.49
trigram surp | -0.18 | 0.039 | -4.54
dep length | 0.02 | 0.012 | 1.77
pcfg surp | -0.13 | 0.015 | -8.34
IS score | -0.01 | 0.005 | -1.87
lex repetition surp | 0.03 | 0.036 | 0.92
lstm surp | 1.21 | 0.154 | 7.87
adaptive lstm surp | -1.50 | 0.155 | -9.67
(b) Subtable 2 list of tables text

Table 3: Discourse adaptation regression model on DO/IO fronted cases (all
significant predictors denoted by $|$t$|$>2)

###  4.2 Prediction Accuracy

Predictors |  | Full
---
Accuracy %
DO | IO |  | Predictors |  | Full
---
Accuracy %
DO | IO
a = IS score | 51.84 | 53.88 | 50.92 | Collective: with repetition effects
b = dep length | 62.31*** | 68.49*** | 58.91*** | base1 = a+b+c+d+e+f | 95.05 | 80.99 | 89.06
c = pcfg surp | 86.86*** | 65.90 | 78.86*** | base1 + g | 95.06 | 81.06 | 89.65*
d = lex repetition surp | 90.07*** | 77.33*** | 85.07*** | Collective: without repetition effects
e = 3-gram surp | 91.18*** | 78.95* | 87.29**
f = lstm surp | 94.01*** | 79.55 | 87.28 | base2 = a+b+c+e+f | 95.06 | 81.24 | 89.65
g = adaptive lstm surp | 94.06 | 79.97 | 88.32*** | base2 + g | 95.09* | 81.42 | 89.80

Table 4: Prediction performances (Full data set (72833 points), Direct objects
(DO; 1663 points) and indirect object (IO; 1353 points) fronted cases; each
row refers to a distinct model; *** McNemar’s two-tailed significance compared
to model on previous row)

While the previous section explored how predictors contribute to Hindi
ordering preferences across all of the data in aggregate, in this section we
frame our model as a classification task on held-out data to determine how
many sentences are affected by each predictor. This enables us to examine the
relative performance of different predictors in identifying Hindi reference
sentences amidst artificially generated grammatical variants and to conduct
more detailed error analysis of our results. We used 10-fold cross-validation
to evaluate model classification accuracy, i.e. the percentage of data points
where a model correctly predicted the referent sentence over a paired variant,
for different subsets of predictors (see Table 4).

Non-adaptive LSTM surprisal (94.01% accuracy) and adaptive LSTM surprisal
(94.06%) yielded the best classification accuracies when no other predictors
were included. Over a baseline model comprised of every other feature except
lexical repetition surprisal (see _base2_ in Table 4), adaptive LSTM surprisal
induced a small but significant increase of 0.03% in accuracy (p = 0.04 using
McNemar’s two-tailed test). When we included lexical repetition surprisal in
the baseline model (see _base1_ in Table 4), adaptive LSTM surprisal ceased to
be a significant predictor. This suggests that, in the general case, the
maximization of discourse predictability is driven by localized lexical
priming captured by our trigram cache model. Apart from the content words,
adaptive LSTM surprisal accounts for the re-occurrence of function words
(e.g., case markers) which have been shown to modulate syntactic priming and
drive parsing processes (Husain and Yadav, 2020).

To study prediction accuracy on non-canonical constructions, we restricted our
analyses to IO- and DO-fronted items in the test partition (still training the
classifier on the full training partition for each fold). In contrast to the
DO-fronted subset, adaptive surprisal was a significant predictor of IO-
fronted syntactic choice, even in the presence of lexical repetition
surprisal, as is evident from the significant increase of 0.6% in accuracy (p
= 0.02 using McNemar’s two-tailed test; see the rightmost IO column in Table
4). This result indicates that discourse predictability is effective in
predicting IO-fronting in sentences that follow other IO-fronted sentences,
suggesting the presence of syntactic priming effects. We consider adaptive
LSTM LM surprisal (i.e., updating LM weights on successive sentences at test
time) as an indicative of syntactic priming in this work but not the vanilla
LSTM LM surprisal. We present a more nuanced discussion on this theme in
Section 6.

Both our regression and classification results demonstrate that discourse
adaptation is more effective in IO-fronted than DO-fronted constructions,
mirroring the findings in Hindi sentence comprehension, where Vasishth (2004)
showed that discourse context could compensate for the processing difficulty
induced by indirect object fronting. The findings of our computational
modelling reported in Table 4 are further validated by the agreement accuracy
of our human evaluation study described in Section 3. Participants were more
prone to prefer IO-fronted construction (80%) compared to DO-fronted
construction (65%) as shown in Table 9 of Appendix G.

###  4.3 Qualitative Analysis: Success of Adaptive LSTM Surprisal

Further linguistic analyses in IO-fronted constructions revealed that LSTM
adaptation also captured the priming of given-given items, potentially
modeling the preferred ordering of multiple _given_ items, a case not captured
by IS score or lexical repetition surprisal. Reference sentence 1 is correctly
predicted by the model containing adaptive LSTM surprisal and all other
features (i.e., base1+g in Table 4) but a model without adaptive LSTM
surprisal (i.e., base1) predicts the variant Example 1. Appendix E Table 6
presents the exact scores of different predictors for the referent-variant
pairs (1 and 1). All predictors but LSTM and adaptive LSTM surprisal assign
high score for the reference sentence with respect to its paired variant.
Adaptive LSTM surprisal assigns a low per-word surprisal for the phrase amar
ujala when it comes at the first position in the reference sentence (1) with
respect to when it comes at the second position in the variant (1),
potentially modeling givenness as this word occurred in the previous sentence
(Example A in Appendix E) as well. See Figure 3 in Appendix E for the
information profile of the reference-variant pairs.

###  4.4 What causes priming?

In the priming literature, there is debate as to whether priming is driven by
residual neural activation (short-lived effects) or by humans learning and
updating their language expectations (long-lived effects). Bock and Griffin
(2000) showed that syntactic priming in humans persisted even when prime and
target sentences were separated by 10 intervening sentences, supporting the
implicit learning (long-lived) hypothesis of syntactic priming. In order to
test this effect on constituent ordering choice, we repeated our adaptation
experiment by adapting to additional context sentences from the preceding
discourse. Adaptive LSTM surprisal and lexical repetition surprisal were
estimated by adapting the base LSTM LM and trigram LM, respectively, to five
preceding context sentences, rather than the single context sentence we used
for our other analyses. We found that for non-canonical IO/DO-fronted
constructions, additional context sentences do not improve the adaptive LSTM
LM’s word order predictions, suggesting that priming may be driven by short-
term residual activation (see Table 8 in the Appendix F).

##  5 Variance Inflation Factor

In this section, we evaluate our regression models for multicollinearity in
terms of variance inflation factor (VIF) score. As Figure 2 in Appendix C
denotes, the adaptive LSTM surprisal measure has a high correlation with all
other surprisal predictors, which raises some suspicion that estimates of
effects of the variables in our regression model might be unreliable. Table 10
in Appendix H presents the VIF scores for different regression models
containing different set of predictors on full dataset, DO- and IO-fronted
subsets. The outcomes indicate that all the surprisal predictors except PCFG
surprisal in our original regression models have very high VIF scores (see
Table 10a in Appendix H). Nevertheless, removing the more correlated measures,
such as trigram surprisal and base LSTM surprisal does not alter any of our
old results (see Table 11 for new regression results and Table 10b for
corresponding VIF scores in Appendix H). All the surprisal measures including
adaptive LSTM and lexical repetitions surprisal have negative regression
coefficients suggesting that Hindi tends to optimize for discourse
predictability.

##  6 Discussion

Our main findings suggest that in written Hindi, people choose word orders
that maximize discourse predictability. The actual psychological mechanisms
are conceivably lexical and structural priming. Our results indicate that
lexical priming is most influential in canonical sentence contexts, but
syntactic priming does influence ordering preferences in non-canonical
contexts. Below, we discuss the implications of our findings in terms of the 4
factors affecting syntactic priming discussed in detail by Reitter et al.
(2011): inverse frequency interaction, decay, lexical boost, and cumulativity.
The IO-fronted construction is very rare (0.76% of our data) compared to DO-
fronted non-canonical sentences (1% of our data) in the HUTB corpus of 13274
sentences. We find strong priming effects in IO-fronted constructions but weak
priming in the case of DO-fronted constructions, providing evidence for an
inverse frequency interaction (Scheepers, 2003; Jaeger and Snider, 2007).

Our finding that priming is not aided by long-term contexts indicates a decay
effect in priming, which supports the residual activation (short-lived)
hypothesis of priming in comprehension (Pickering and Branigan, 1998).
Nevertheless, there has been evidence for implicit learning effects in
comprehension as well (Luka and Barsalou, 2005; Wells et al., 2009). More
recently, Ranjan et al. (2022b) using a similar setup as our current work
argued for the existence of both the accounts _viz.,_ residual activation and
implicit learning, and demonstrated the role of dual mechanism priming effects
(Tooley and Traxler, 2010) in Hindi word order.

Previous work suggests that lexical overlap between prime and target sentences
enhances syntactic priming (Pickering and Branigan, 1998; Gries, 2005). The
repeated lexical items become cues during sentence planning and bias the
speaker to produce similar structures that those repeated lexical items tend
to occur in. Overall, we find that lexical repetition influences Hindi
syntactic choice; however, syntactic priming is observed over and above
lexical repetition in non-canonical constructions. It’s interesting to note
that comparable results have also been reported for English dialogue corpora
(Healey et al., 2014; Green and Sun, 2021). We plan to conduct a systematic
investigation on Hindi spoken data as a part of future work.

Finally, with regards to the cumulativity of priming, Jaeger and Snider (2007)
showed in their corpus study of production of passives and that-
insertion/omission that the effect of priming increases with the number of
primes preceding it. Our work does not investigate this specifically, and more
controlled experiments would be required.

The success of LSTM-based surprisal estimates over and above dependency length
can also be interpreted in light of Futrell’s (2019) point about the
limitation of Surprisal Theory with respect to word order. Futrell modified
Surprisal Theory by positing that the per-word processing difficulty is
proportional to its surprisal given a lossy memory representation of the
preceding context. Moreover, Futrell et al. (2020) proposed the Information
Locality Hypothesis (ILH) which states that all pairs of words with high
mutual information (not merely syntactically related words) tend to be located
close to one another. The long window offered by LSTM surprisal thus models
relationships between words at varying distances (over and above conventional
trigram models). The success of these surprisal estimates for the task of
reference sentence prediction provides some preliminary evidence for ILH in
the case of word order.

Future work needs to tease apart priming effects of both vanilla LSTM and
adaptive LSTM surprisal in the light of recent works. In this work, sentences
are treated as independent while estimating their surprisal using vanilla LSTM
LM, so there is no way vanilla LSTM can exhibit syntactic priming given the
independent sentences. However, Misra et al. (2020) demonstrated that BERT
exhibits “priming effect”. The BERT LM was able to predict a word with greater
probability when the context included a related word than an unrelated word.
However, the effect decreased as the amount of information provided by the
context increases. In other words, the related prime under high contextual
constraint started acting as distractor—actively demoting the target word in
the probability distribution; thus exhibiting “mispriming effect” Kassner and
Schütze (2020). This could be due to stylistic avoidance of repeated
structures/words in the adjacent sentences. Future work also needs to
investigate whether word-order preferences can be jointly optimized using
multiple factors (Gildea and Jaeger, 2015). In particular, the relationship
between the drive to minimize surprisal (as found in this work) and the
tendency to make information profiles uniform (Jaeger, 2010) needs to be
explored more thoroughly in the light of recent findings (Meister et al.,
2021).

Overall, our results demonstrate that Hindi word order preferences are
influenced by discourse predictability maximization considerations. The actual
mechanisms of discourse effects are plausibly lexical and syntactic priming.

##  7 Limitations

The ‘levels’ problem discussed in Levy (2018) which posits 2 levels of
linguistic optimisation is germane while evaluating our work. Our results are
restricted to the level of syntactic choices made by individual speakers or
users of a given language over a lifetime (and not at the level of entire
grammars and evolutionary timescales). Our experiments conducted on written
text need to be performed on spoken data in order to make claims about priming
in language production.

## Acknowledgements

We thank the first author’s dissertation committee members, Drs. Mausam and
Samar Husain, as well as the Cornell C.Psyd members for their insightful
comments and suggestions on this work. We thank Rupesh Pandey’s logistical
assistance in gathering the human judgment data for this work. Additionally,
we are grateful for the thorough feedback provided by the anonymous reviewers
of EMNLP 2022, ACL ARR 2021, and COMCO 2021. The first author acknowledges the
extramural travel grant from Microsoft Research India to present the work at
EMNLP 2022. Finally, the last two authors also thank extramural funding from
the Department of Science and Technology of India through the Cognitive
Science Research Initiative (project no. DST/CSRI/2018/263).

## References

  * Arai et al. (2007) Manabu Arai, Roger PG Van Gompel, and Christoph Scheepers. 2007.  Priming ditransitive structures in comprehension.  _Cognitive psychology_ , 54(3):218–250.
  * Baker et al. (2002) Paul Baker, Andrew Hardie, Tony McEnery, Hamish Cunningham, and Robert Gaizauskas. 2002.  Emille: a 67-million word corpus of indic languages: data collection, mark-up and harmonization.  In _Proceedings of LREC 2002_ , pages 819–827. Lancaster University.
  * Bharati et al. (2002) Akshar Bharati, Rajeev Sangal, Vineet Chaitanya, Amba Kulkarni, Dipti Misra Sharma, and KV Ramakrishnamacharyulu. 2002.  Anncorra: building tree-banks in indian languages.  In _COLING-02: The 3rd Workshop on Asian Language Resources and International Standardization_.
  * Bhatt et al. (2009) Rajesh Bhatt, Bhuvana Narasimhan, Martha Palmer, Owen Rambow, Dipti Misra Sharma, and Fei Xia. 2009.  A multi-representational and multi-layered treebank for Hindi/urdu.  In _Proceedings of the Third Linguistic Annotation Workshop_ , ACL-IJCNLP ’09, pages 186–189, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Bock and Griffin (2000) Bock and Z. Griffin. 2000.  The persistence of structural priming: Transient activation or implicit learning.  _Journal of Experimental Psychology_ , 2(120):177–192.
  * Bock (1986) J.Kathryn Bock. 1986.  Syntactic persistence in language production.  _Cognitive Psychology_ , 18(3):355 – 387.
  * Christiansen and Chater (2008) Morten H. Christiansen and Nick Chater. 2008.  Language as shaped by the brain.  _Behavioral and Brain Sciences_ , 31(5):489–509.
  * Clarkson and Robinson (1997) P.R. Clarkson and A. J. Robinson. 1997.  Language model adaptation using mixtures and an exponentially decaying cache.  In _In Proceedings of ICASSP-97_ , pages 799–802.
  * Dammalapati et al. (2021) Samvit Dammalapati, Rajakrishnan Rajkumar, and Sumeet Agarwal. 2021.  Effects of Duration, Locality, and Surprisal in Speech Disfluency Prediction in English Spontaneous Speech.  In _Proceedings of the Society for Computation in Linguistics_ , volume 4, page 10.
  * Demberg and Keller (2008) Vera Demberg and Frank Keller. 2008.  Data from eye-tracking corpora as evidence for theories of syntactic processing complexity.  _Cognition_ , 109(2):193–210.
  * Demberg et al. (2012) Vera Demberg, Asad B. Sayeed, Philip J. Gorinski, and Nikolaos Engonopoulos. 2012\.  Syntactic surprisal affects spoken word duration in conversational contexts.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , EMNLP-CoNLL ’12, pages 356–367, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Fine and Jaeger (2016) Alex B Fine and T Florian Jaeger. 2016.  The role of verb repetition in cumulative structural priming in comprehension.  _Journal of Experimental Psychology: Learning, Memory, and Cognition_ , 42(9):1362.
  * Fine et al. (2013) Alex B Fine, T Florian Jaeger, Thomas A Farmer, and Ting Qian. 2013.  Rapid expectation adaptation during syntactic comprehension.  _PloS one_ , 8(10):e77661.
  * Futrell (2019) Richard Futrell. 2019.  Information-theoretic locality properties of natural language.  In _Proceedings of the First Workshop on Quantitative Syntax (Quasy, SyntaxFest 2019)_ , pages 2–15, Paris, France. Association for Computational Linguistics.
  * Futrell et al. (2020) Richard Futrell, Edward Gibson, and Roger P. Levy. 2020.  Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing.  _Cognitive Science_ , 44(3):e12814.
  * Futrell et al. (2015) Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015.  Large-scale evidence of dependency length minimization in 37 languages.  _Proceedings of the National Academy of Sciences_ , 112(33):10336–10341.
  * Genzel and Charniak (2002) Dmitriy Genzel and Eugene Charniak. 2002.  Entropy rate constancy in text.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, pages 199–206, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Gibson (2000) Edward Gibson. 2000.  Dependency locality theory: A distance-based theory of linguistic complexity.  In Alec Marantz, Yasushi Miyashita, and Wayne O’Neil, editors, _Image, Language, brain: Papers from the First Mind Articulation Project Symposium_. MIT Press, Cambridge, MA.
  * Gibson et al. (2019) Edward Gibson, Richard Futrell, Steven T. Piandadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019.  How efficiency shapes human language.  _Trends in Cognitive Sciences_ , 23(5):389–407.
  * Gildea and Jaeger (2015) Daniel Gildea and T. Florian Jaeger. 2015.  Human languages order information efficiently.  _CoRR_ , abs/1510.02823.
  * Green and Sun (2021) Clarence Green and He Sun. 2021.  Global estimates of syntactic alignment in adult and child utterances during interaction: Nlp estimates based on multiple corpora.  _Language Sciences_ , 85:101353.
  * Gries (2005) Stefan Th. Gries. 2005.  Syntactic priming: A corpus-based approach.  _Journal of Psycholinguistic Research_ , 34(4):365–399.
  * Hale (2001) John Hale. 2001.  A probabilistic Earley parser as a psycholinguistic model.  In _Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies_ , NAACL ’01, pages 1–8, Pittsburgh, Pennsylvania. Association for Computational Linguistics.
  * Hawkins (1994) John A. Hawkins. 1994.  _A Performance Theory of Order and Constituency_.  Cambridge University Press, New York.
  * Hawkins (2000) John A. Hawkins. 2000.  The relative order of prepositional phrases in english: Going beyond manner-place-time.  _Language Variation and Change_ , 11(03):231–266.
  * Hawkins (2004) John A. Hawkins. 2004.  _Efficiency and Complexity in Grammars_.  Oxford University Press.
  * Hawkins (2014) John A. Hawkins. 2014.  _Cross-Linguistic Variation and Efficiency_.  Oxford University Press.
  * Healey et al. (2014) Patrick GT Healey, Matthew Purver, and Christine Howes. 2014.  Divergence in dialogue.  _PloS one_ , 9(6):e98598.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Husain and Yadav (2020) Samar Husain and Himanshu Yadav. 2020.  Target complexity modulates syntactic priming during comprehension.  _Frontiers in Psychology_ , 11:454.
  * Jaeger (2010) T. Florian Jaeger. 2010.  Redundancy and reduction: Speakers manage information density.  _Cognitive Psychology_ , 61(1):23–62.
  * Jaeger and Norcliffe (2009) T. Florian Jaeger and Elizabeth Norcliffe. 2009.  The cross-linguistic study of sentence production: State of the art and a call for action.  _Language and Linguistic Compass_ , 3(4):866–887.
  * Jaeger and Snider (2007) T. Florian Jaeger and Neal Snider. 2007.  Implicit learning and syntactic persistence: Surprisal and cumulativity.  _University of Rochester Working Papers in the Language Sciences_ , 3:26–44.
  * Jaeger and Tily (2011) T. Florian Jaeger and Harold Tily. 2011.  Language processing complexity and communicative efficiency.  _WIRE: Cognitive Science_ , 2(3):323–335.
  * Jain et al. (2018) Ayush Jain, Vishal Singh, Sidharth Ranjan, Rajakrishnan Rajkumar, and Sumeet Agarwal. 2018.  Uniform Information Density Effects on Syntactic Choice in Hindi.  In _Proceedings of the Workshop on Linguistic Complexity and Natural Language Processing_ , pages 38–48, Santa Fe, New-Mexico. Association for Computational Linguistics.
  * Joachims (2002) Thorsten Joachims. 2002.  Optimizing search engines using clickthrough data.  In _Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’02, pages 133–142, New York, NY, USA. ACM.
  * Kachru (2006) Y. Kachru. 2006.  _Hindi_.  London Oriental and African language library. John Benjamins Publishing Company.
  * Kaiser and Trueswell (2004) Elsi Kaiser and John C Trueswell. 2004.  The role of discourse context in the processing of a flexible word-order language.  _Cognition_ , 94(2):113–147.
  * Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020.  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7811–7818, Online. Association for Computational Linguistics.
  * Kuhn and De Mori (1990) Roland Kuhn and Renato De Mori. 1990.  A cache-based natural language model for speech recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 12(6):570–583.
  * Levy (2008) Roger Levy. 2008.  Expectation-based syntactic comprehension.  _Cognition_ , 106(3):1126 – 1177.
  * Levy (2018) Roger P Levy. 2018.  Communicative efficiency, uniform information density, and the rational speech act theory.
  * Liu (2008) Haitao Liu. 2008.  Dependency distance as a metric of language comprehension difficulty.  _Journal of Cognitive Science_ , 9(2):159–191.
  * Liu et al. (2017) Haitao Liu, Chunshan Xu, and Junying Liang. 2017.  Dependency distance: A new perspective on syntactic patterns in natural languages.  _Physics of Life Reviews_ , 21:171 – 193.
  * Luka and Barsalou (2005) Barbara J Luka and Lawrence W Barsalou. 2005.  Structural facilitation: Mere exposure effects for grammatical acceptability as evidence for syntactic priming in comprehension.  _Journal of Memory and Language_ , 52(3):436–459.
  * Meister et al. (2021) Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021.  Revisiting the Uniform Information Density hypothesis.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 963–980, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Misra et al. (2020) Kanishka Misra, Allyson Ettinger, and Julia Rayz. 2020.  Exploring bert’s sensitivity to lexical cues using tests from semantic priming.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4625–4635.
  * Petrov et al. (2006) Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006.  Learning accurate, compact, and interpretable tree annotation.  In _Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics_ , ACL-44, pages 433–440, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Pickering and Branigan (1998) Martin J. Pickering and Holly P. Branigan. 1998.  The Representation of Verbs: Evidence from Syntactic Priming in Language Production.  _Journal of Memory and Language_ , 39(4):633–651.
  * Qian and Jaeger (2012) Ting Qian and T. Florian Jaeger. 2012.  Cue effectiveness in communicatively efficient discourse production.  _Cognitive Science_ , 36(7):1312–1336.
  * Rajkumar et al. (2016) Rajakrishnan Rajkumar, Marten van Schijndel, Michael White, and William Schuler. 2016.  Investigating locality effects and surprisal in written english syntactic choice phenomena.  _Cognition_ , 155:204–232.
  * Rajkumar and White (2014) Rajakrishnan Rajkumar and Michael White. 2014.  Better surface realization through psycholinguistics.  _Language and Linguistics Compass_ , 8(10):428–448.  ISSN: 1749-818X.
  * Ranjan et al. (2019) Sidharth Ranjan, Sumeet Agarwal, and Rajakrishnan Rajkumar. 2019.  Surprisal and Interference Effects of Case Markers in Hindi Word Order.  In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_ , pages 30–42, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ranjan et al. (2022a) Sidharth Ranjan, Rajakrishnan Rajkumar, and Sumeet Agarwal. 2022a.  Locality and expectation effects in hindi preverbal constituent ordering.  _Cognition_ , 223:104959.
  * Ranjan et al. (2022b) Sidharth Ranjan, Marten van Schijndel, Sumeet Agarwal, and Rajakrishnan Rajkumar. 2022b.  Dual mechanism priming effects in hindi word order.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, AACL/IJCNLP_.
  * Reitter et al. (2011) David Reitter, Frank Keller, and Johanna D. Moore. 2011.  A computational cognitive model of syntactic priming.  _Cognitive Science_ , 35(4):587–637.
  * Sangal et al. (1995) Rajeev Sangal, Vineet Chaitanya, and Akshar Bharati. 1995.  _Natural language processing: a Paninian perspective_.  PHI Learning Pvt. Ltd.
  * Scheepers (2003) C Scheepers. 2003.  Syntactic priming of relative clause attachments: persistence of structural configuration in sentence production.  _Cognition_ , 89:179–205.
  * Staub (2015) Adrian Staub. 2015.  The effect of lexical predictability on eye movements in reading: Critical review and theoretical interpretation.  _Language and Linguistics Compass_ , 9(8):311–327.
  * Stolcke (2002) Andreas Stolcke. 2002.  SRILM — An extensible language modeling toolkit.  In _Proc. ICSLP-02_.
  * Temperley (2007) David Temperley. 2007.  Minimization of dependency length in written English.  _Cognition_ , 105(2):300–333.
  * Tooley and Traxler (2010) Kristen M Tooley and Matthew J Traxler. 2010.  Syntactic priming effects in comprehension: A critical review.  _Language and Linguistics Compass_ , 4(10):925–937.
  * van Schijndel and Linzen (2018) Marten van Schijndel and Tal Linzen. 2018.  A neural model of adaptation in reading.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4704–4710.
  * Vasishth (2004) S. Vasishth. 2004.  Discourse context and word order preferences in Hindi.  _Yearbook of South Asian Languages_ , pages 113–127.
  * Wells et al. (2009) Justine B Wells, Morten H Christiansen, David S Race, Daniel J Acheson, and Maryellen C MacDonald. 2009.  Experience and sentence processing: Statistical learning and relative clause comprehension.  _Cognitive psychology_ , 58(2):250–271.
  * Wierzba and Fanselow (2020) Marta Wierzba and Gisbert Fanselow. 2020.  Factors influencing the acceptability of object fronting in german.  _The Journal of Comparative Germanic Linguistics_ , 23(1):77–124.
  * Yadav et al. (2017) Himanshu Yadav, Ashwini Vaidya, and Samar Husain. 2017.  Keeping it simple: Generating phrase structure trees from a Hindi dependency treebank.  In _TLT_.

## Appendix

##  Appendix A Variant Generation

(a) Figure

Label | Dependency
---|---
| relation
Invariant syntactic relations |
k1 | subject/agent
k2 | object/patient
k3 | instrument
k4 | object/recipient
k7t | location in time
Complex predicate relation |
pof | parts of
| conjunct verb
pof_cn | parts of
| compound noun
Local word group (lwg) |
lwg_psp | postposition
lwg_vaux | auxilliary verb
Symbols |
rsym | symbol relation
(b) Table

Figure 1: Example HUTB dependency tree and relation labels

Context sentence amar ujala-ki bhumika nispaksh rehti hai
Amar Ujala-gen role unbiased remain be.prs.sg
Amar Ujala’s role remains unbiased.

[] amar ujala-ko yah _sukravar_ -ko daak-se prapt hua [Given-Given = 0]
(Reference)
Amar Ujala-acc it _friday_ -on post-inst receive be.pst.sg
Amar Ujala received it by post on _Friday_. [] yah amar ujala-ko _sukravar_
-ko daak-se prapt hua [Given-Given = 0] (Variant 1) [] _sukravar_ -ko yah amar
ujala-ko daak-se prapt hua [New-Given = -1] (Variant 2)

This work uses sentences from the Hindi-Urdu Treebank (HUTB) corpus of
dependency trees (Bhatt et al., 2009) containing well-defined subject and
object constituents. Figure 1 displays the dependency tree (and a glossary of
relation labels) for reference sentence 1. The grammatical variants were
created using an algorithm that took as input the dependency tree
corresponding to each HUTB reference sentence. The re-ordering algorithm
permuted the preverbal777Hindi is not a strictly verb-final language but the
majority of the constituents in the HUTB corpus are preverbal. Our corpus
analysis of 13274 sentences present in HUTB suggests 20,750 pairs of preverbal
constituents and 2599 pairs of postverbal constituents. Therefore, our variant
generation (via reordering of constituents) and subsequent experiments focus
on word-order variation in the preverbal domain, considering preverbal domain
to be the locus of word order variation. Only preverbal constituents are
permuted to generate grammatical variants and leave the postverbal
constituents in the reference-variants sentences as it is. dependents of the
root verb and linearized the resulting tree to obtain variant sentences. For
example, corresponding to the reference sentence 1 and its root verb “hai”
(see figure 1a), the preverbal constituents with parents as “ujala”, “yah”,
“suravar”, “daak”, and “prapt” were permuted to generate the artificial
variants (1 and 1). The ungrammatical variants were automatically filtered out
using dependency relation sequences (denoting grammar rules) attested in the
gold standard corpus of HUTB trees. In the dependency tree 1a, “k4-k1”,
“k7t-k1”, “k3-k7t”, and “pof-k3” are dependency relation sequences. In cases
where the total number of variants exceeded 100 (a random cutoff),888Higher
and lower cutoffs do not affect our results. we chose 99 non-reference
variants randomly along with the reference sentence.

##  Appendix B Information Status Annotation

The subject and object constituents in a sentence were assigned a Given tag if
any content word within them was mentioned in the preceding sentence or if the
head of the phrase was a pronoun. All other phrases were tagged as New. The
sentence example A illustrates the proposed annotation scheme.

  * •

Example 1 follows Given-Given ordering — The object “Amar Ujala” in the
sentence is mentioned in the preceding context sentence A, it would be
annotated as _Given_. In contrast, the subject “yah” is a pronoun so it would
also be tagged as _Given_ following the annotation scheme.

  * •

Example 1 follows New-Given ordering — The object “sukravar" in the sentence
should be tagged as _New_ as it is not mentioned in the preceding context
sentence A. In contrast, the subsequent pronoun “yah” which acts as the
subject of the sentence should be tagged as _Given_ following the annotation
scheme.

##  Appendix C Correlation Plot

The Pearson’s correlation coefficients between different predictors are
displayed in Figure 2. The adaptive LSTM surprisal has a high correlation with
all other surprisal features and a low correlation with dependency length and
information status score.

Figure 2: Pearson’s coefficient of correlation between different pairs of
predictors

##  Appendix D Joachims Transformation

This technique converts a binary classification problem into a pair-wise
ranking task involving the feature vectors of a reference sentence and each of
its variants. Table 5 displays the Joachims’s transformation. The delta
($\delta$) refers to the difference between feature vectors of the reference
sentence and its paired variant. The overall goal is to model two-alternative
choices for each reference sentence such that the speaker generated the
reference sentence after rejecting a potential grammatical variant. Moreover,
the reference sentence that appeared originally in the corpus must have been
present due to its properties (_viz.,_ dependency length, discourse context,
accessibility, or surprisal), and variant sentences are less likely to be
produced due to the same reasons.

Condition | Label | Dependency |  $n$-gram | pcfg
---|---|---|---|---
|  | length | surprisal | surprisal
Reference | 1 | 18 | 24.69 | 61.13
Variant1 | 0 | 20 | 23.80 | 60.67
Variant2 | 0 | 18 | 23.02 | 60.02

(a) Subtable 1 list of tables text

Condition | Label |  $\delta$ Dependency |  $\delta$ $n$-gram |  $\delta$ pcfg
---|---|---|---|---
|  | length | surprisal | surprisal
Variant1 | 0 | 2 | -0.89 | -0.46
-Reference |  |  |  |
Reference | 1 | 0 | 1.67 | 1.11
-Variant2 |  |  |  |

(b) Subtable 2 list of tables text

Table 5: Joachims transformation

##  Appendix E Information Profile for IO-fronted Example

The LSTM LM when adapted to the previous sentence (A) in the discourse assigns
lower surprisal score to the given item when it occurs in the first position
(“amar ujala" in sentence 1) than when it appears in the second position
(“amar ujala" in sentence 1) in the subsequent sentence.

Figure 3: Information profile for the reference-variant pair 1 and 1

Type | 3g surp | Deplen | PCFG surp | IS score | LSTM surp | Adaptive LSTM surp | Lex rept surp
---|---|---|---|---|---|---|---
Example 1 | Reference | 24.69 | 18 | 61.13 | 0 | 91.80 | 89.52 | 23.80
Example 1 | Variant | 23.80 | 20 | 60.67 | 0 | 93.78 | 93.17 | 22.19

Table 6: Predictor scores for reference-variant pairs (1, 1)

##  Appendix F Contextual Adaptation on One Vs. Multiple Sentences for DO/IO
Constructions

| Non-canonical
---
HUTB Sentences
| Frequency
---
Count (%)
| Baseline
---
Perplexity
| Adapted
---
Perplexity (Prev1)
| Perplexity
---
Dip (Prev1)
| Adapted
---
Perplexity (Prev5)
| Perplexity
---
Dip (Prev5)
DO | 133 (1%) | 183.92 | 103.40 | -80.52 | 77.33 | -106.59
IO | 101 (0.76%) | 138.78 | 88.26 | -50.52 | 68.45 | -70.33

Table 7: Effect of adaptation on discourse sentences (Prev1: Preceding one
sentence in discourse, Prev5: Preceding five sentences in discourse)

We investigated if adapting the LSTM LM to the preceding five contextual
sentences instead of one contextual sentence will help predict word-ordering
patterns better for IO/DO constructions. Table 7 showcases perplexity dip on
test sentences during 1 vs. 5 contextual sentence adaptation. Table 8
highlights classification accuracy of different models containing combination
of features. Our results indicate that adding Prev5-adaptive LSTM surprisal in
the machine learning model above and beyond every other features including
Prev1-adaptive surprisal (_i.e.,_ Baseline) does not significantly boost
prediction accuracy for both IO- and DO-fronted subset. A similar finding was
observed when Prev5-lexical-repetition surprisal (Prev5-Lex-Rept LM: base
trigram LM interpolated with unigram cache LM containing 5 preceding sentences
history) was included in the classifier model above and beyond every other
features (_i.e.,_ Baseline) including Prev1-lexical-repetition surprisal
(Prev1-Lex-Rept LM: base trigram LM interpolated with unigram cache LM
containing only 1 preceding sentence history).

Type | Baseline | \+ Prev5 Adaptive | \+ Prev5 Lex-Rept
---|---|---|---
|  | LSTM Surp | Surp
DO | 81.06 | 81.12 | 80.94
IO | 89.65 | 89.73 | 89.51
Table 8: Prediction performance (Direct objects (DO: 1663 points), Indirect
Objects (IO: 1353 points)); Baseline denotes base1+g shown in Table 4; bold
denotes McNemar’s two-tailed significance compared to baseline model in the
same row

##  Appendix G Human Evaluation

To determine whether the permuted word order (variant) is dispreferred to the
original word order (reference), we conducted a targeted human evaluation via
forced-choice rating task and collected sentence ratings from 12 Hindi native
speakers for 167 randomly selected reference-variant pairs in our data set.
Participants were first shown the preceding sentence, and then they were asked
to judge the subsequent most likely sentence as the best choice between the
reference-variant pair. Each sentence was assigned a human label of “1” if
more than 50% participants voted it as best choice else human label of “0”.
The stimuli belonged to two different constructions, viz., the reference
sentence (Ref) has canonical ordering whereas, the variant (Var) has non-
canonical ordering (DO-fronted or IO-fronted) and vice versa.

Table 9 presents the results. On the entire dataset containing 167 reference-
variant pairs, 89.92% (agreement accuracy) of the reference sentences which
originally appeared in the HUTB corpus were also preferred by native speakers
compared to the artificially generated grammatical variants expressing similar
meaning. Overall, the full model containing all the features, including
adaptive LSTM surprisal predicted human preference (76.65%) much better than
corpus choice labels (74.85%). Furthermore, the Pearson’s correlation
coefficient of the classifier’s prediction with the human judgement is 0.534
and with corpus labels is 0.497. Interestingly, as initially hypothesized, the
Hindi participants were more prone to prefer IO-fronted construction (80%)
compared to DO-fronted construction (65%) as captured by the agreement
accuracy validating the findings reported in Table 4.

Construction | Agreement (%) | Model (%) | Model (%)
---|---|---|---
Type (item count) | human:corpus | corpus labels | human labels
Ref: Canonical | Var: DO-fronted (20) | 95 | 90 | 85
Ref: DO-fronted | Var: Canonical (20) | 65 | 25 | 50
Ref: Canonical | Var: IO-fronted (20) | 100 | 85 | 85
Ref: IO-fronted | Var: Canonical (20) | 80 | 65 | 65
Ref: Non-canonical | Var: Canonical (80) | 85.00 | 66.25 | 71.25
Ref: Canonical | Var: Canonical (87) | 94.25 | 82.76 | 81.61
Total (167) | 89.92 | 74.85 | 76.65

Table 9: Targeted human evaluation — Agreement human/corpus: Percentages of
times human judgement matches with corpus reference choice; Model corpus:
Percentages of corpus choice correctly predicted by the classifier containing
all the predictors (_base1 + g_ as per Table 4); Model human: Percentages of
human label correctly predicted by the classifier containing all the
predictors (_base1 + g_ as per Table 4)

##  Appendix H Variance Inflation Factor Analysis

Table 10a displays the VIF scores for each predictors in the different
regression models. The VIF scores for the regression models without containing
the correlated features, such as trigram surprisal and vanilla LSTM surprsial
is documented in Table 10b. Correspondingly, Table 11 reports the results of
regression experiment when the model did not containing these highly
correlated features.

Predictors | Full (72833) | DO (1663) | IO (1353)
---|---|---|---
trigram surp | 27.61 | 18.87 | 18.87
dependency length | 1.18 | 1.46 | 1.46
pcfg surp | 3.08 | 1.86 | 1.86
IS score | 1.01 | 1.03 | 1.03
lex-rept surp | 24 | 16.47 | 16.47
lstm surp | 241.62 | 109.04 | 109.04
adaptive lstm surp | 244.37 | 106.97 | 106.97
performance |  |  |
Residual standard err | 0.271 | 0.294 | 0.354
Multiple R-squared | 0.707 | 0.657 | 0.502
Adjusted R-squared | 0.707 | 0.655 | 0.500

(a) Subtable 1 list of tables text

Predictors | Full (72833) | DO (1663) | IO (1353)
---|---|---|---
dependency len | 1.18 | 1.42 | 1.18
pcfg surp | 2.95 | 1.73 | 2.08
IS score | 1.01 | 1.01 | 1.06
lex-rept surp | 4.99 | 4.45 | 4.26
adaptive lstm surp | 5.77 | 4.39 | 4.29
performance |  |  |
Residual std. err | 0.271 | 0.357 | 0.304
Multiple R-sqrd | 0.706 | 0.492 | 0.633
Adjusted R-sqrd | 0.706 | 0.490 | 0.631

(b) Subtable 2 list of tables text

Table 10: Variance inflation factor analysis on different regression models
containing: (a) all predictors b) all predictors but correlated features; Each
column denotes individual models on a given dataset with different set of
predictors; VIF larger than 5 or 10 indicates that the model has problems
estimating the coefficient of variables

Predictor | $\hat{\beta}$ | $\hat{\sigma}$ | t
---|---|---|---
Intercept | 1.5 | 0.001 | 1493.53
dependency len | 0.02 | 0.0011 | 15.84
pcfg surp | -0.08 | 0.0017 | -43.49
IS score | 0.01 | 0.001 | 11.92
lex-rept surp | -0.09 | 0.0022 | -39.97
adaptive lstm surp | -0.28 | 0.0024 | -116.06

Table 11: Regression model on full data set after removing the correlated
features ($N=72833$; all significant predictors denoted by $|$t$|$>2)