# A Novel Computational and Modeling Foundation
for Automatic Coherence Assessment

Aviya Maimon  Reut Tsarfaty
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Department of Computer Science, Bar Ilan University

###### Abstract

Coherence is an essential property of well-written texts, that refers to the
way textual units relate to one another. In the era of generative AI,
coherence assessment is essential for many NLP tasks; summarization,
generation, long-form question-answering, and more. However, in NLP coherence
is an ill-defined notion, not having a formal definition or evaluation
metrics, that would allow for large-scale automatic and systematic coherence
assessment. To bridge this gap, in this work we employ the formal linguistic
definition of Reinhart (1980) of what makes a discourse coherent, consisting
of three conditions — cohesion, consistency and relevance — and formalize
these conditions as respective computational tasks. We hypothesize that (i) a
model trained on all of these tasks will learn the features required for
coherence detection, and that (ii) a joint model for all tasks will exceed the
performance of models trained on each task individually. On two benchmarks for
coherence scoring rated by humans, one containing 500 automatically-generated
short stories and another containing 4k real-world texts, our experiments
confirm that jointly training on the proposed tasks leads to better
performance on each task compared with task-specific models, and to better
performance on assessing coherence overall, compared with strong baselines. We
conclude that the formal and computational setup of coherence as proposed here
provides a solid foundation for advanced methods of large-scale automatic
assessment of coherence.

##  1 Introduction

Coherence refers to the quality of a text, which stems from the ways its
various elements – such as sentences, ideas, and paragraphs – flow smoothly
and are logically connected. In a coherent text, each part follows logically
from the preceding one, creating a clear and understandable progression of
ideas. Coherence detection is crucial for NLP tasks involving text quality
measurements such as essay scoring or quality measurements Somasundaran et al.
(2014); Feng et al. (2014); Lai and Tetreault (2018). On top of that,
coherence detection is gaining increased attention due to the growing
awareness of the impact of artificially generated texts. As large language
models (LLMs) become prevalent in applications such as text generation,
summarization, and question answering Guan et al. (2021); Xu et al. (2018); Yi
et al. (2019), ensuring coherent output has become a priority, in order to
make sure that the generated texts are meaningful and understandable.

Detecting coherence in texts is challenging due to the elusive and difficult-
to-define nature of the term coherence. Many linguistic theories have been put
forth Halliday and Hasan (1976); Joshi and Weinstein (1981); Givon (1995);
Hobbs (1979); Dijk (1979); Mann and Thompson (1988), leading to the
development of several modeling techniques that follow up on them Lapata
(2003); Miltsakaki et al. (2000). However, in NLP the current approaches for
capturing coherence relied mostly on proxy tasks that were presumed to reflect
coherence, most noticeable the sentence reordering task Lapata (2003).

Relying on a proxy task, such as sentences reordering, oversimplifies the
notion of coherence, and does not completely reflect the multifaceted nature
of coherence in real-world texts, potentially resulting in the development of
models that are incomplete in their capacity to detect and assess
(in)coherence in a holistic fashion. Furthermore, coherence may vary across
different genres, contexts, and writing styles, making it challenging to
create proxy tasks that adequately represent all possible aspects of
coherence. This can lead to models that are optimized for the proxy tasks but
struggle to generalize their coherence assessment capabilities in diverse
real-world scenarios.

Therefore, models that have been used so far for modeling coherence in NLP,
such as sentences reordering Barzilay and Lapata (2008), have either not been
further evaluated for coherence and were taken at face value, or they were
evaluated by checking the impact of the model as another feature in downstream
application like readability assessment Guinaudeau and Strube (2013); Mesgar
and Strube (2016, 2018) and essay scoring Mesgar and Strube (2018);
Somasundaran et al. (2014); Tay et al. (2017). However, this approach not only
entails significant costs but also runs the risk of evaluating the model
solely on downstream task-related features, potentially undermining the
general coherence features in and of themselves.

Our goal in this paper is to provide a computationally workable definition of
coherence that will in turn allow to assess coherence automatically,
effectively, and at scale. Specifically, in this paper, we adopt the
linguistic theory of coherence provided by Reinhart (1980) that defines three
conditions that are essential for coherence to hold: (i) Cohesion, (ii)
consistency and (iii) relevance. Our proposal instantiates these conditions as
computational NLP tasks and trains those tasks jointly to subsequently produce
a model that will reflect the associated features. Our hypothesis is two-fold.
First, we hypothesize that such a model will function effectively as a model
for automatically assessing coherence. Secondly, we expect that the
performances of each of these individual tasks will be enhanced as a result of
the shared information within the model.

To test our hypotheses, we implement a model trained on the selected tasks
jointly, each of which captures different aspects of coherence. The unified
model incorporates 5 tasks: sentence reordering Lapata (2003), irrelevant
sentence detection, discourse relations detection Miltsakaki et al. (2004),
natural language inference Dagan et al. (2005)) and NP enrichment Elazar et
al. (2022).

In order to confirm our first hypothesis, that such a model is suitable for
assessing coherence, we evaluate this model on the coherence scoring task
using two coherence scoring benchmarks, both comprise of human-annotated
coherence scores for texts: (_GCDC_) Lai and Tetreault (2018) contains real-
world text across four domains, and (_CoheSentia_) anonymous (underreview)
contains artificially-generated text.

Our findings provide empirical evidence confirming our hypotheses,
demonstrating 22% improvement in accuracy for _GCDC_ benchmark and significant
improvements for _CoheSentia_ producing new SOTA results on those benchmarks,
and enhanced performances of the joint model over compared with the standalone
models for the individual tasks. The two coherence-scoring human-annotated
benchmarks encompass the two types of texts used in NLP, natural and
artificially-generated, and we conclude that the significant increase in
performance on both benchmarks further demonstrates the suitability of the
model for automatic assessment of coherence. We subsequently argue that the
formal and computational foundations we provide will enable future models to
automatically identify incoherent regions within texts, and analyze the
underlying causes for this incoherence. Furthermore, we suggest that the
integration of this methodology into the process of text generation will yield
texts of superior quality.

The remainder of this paper is organized as follows. In Section 2, we provide
an overview of the linguistic foundation of our proposal. Later on in Section
3 we use those conditions as our coherence detection engine, where we map
those conditions into a set of NLP tasks. Subsequently, in Section 4, we delve
into the details of our implementation of the coherence model using proxy
tasks. We then provide the experimental setup for evaluating the coherence
model in Section 5. Section 6 is dedicated to presenting the results of our
experiments, for both the standalone proxy tasks as well as the unified model,
and for evaluating the resulting model on the coherence scoring tasks. In
Section 7, we highlight the existing body of related work, and lastly, in
Section 8 we provide concluding remarks and directions for future endeavors.

##  2 The Proposal: Coherence à la Reinhart

This work addresses the lack of a concrete and computationally-workable
definition of coherence in NLP, by adopting its formalization as proposed by
Reinhart (1980). In her work, coherence is equated with a set of three
conditions that are verifiable. They don’t only describe the set of actual
coherent texts but also determine which text is coherent. In this section, we
present these conditions.

Reinhart (1980) defines three conditions for a well-formed text. She termed
these conditions Cohesion, Consistency and Relevance, which reflect syntactic,
logical, and pragmatic aspects of the text, respectively. She claims that, for
a text to be coherent, it has to meet those three conditions. We hereby
describe each condition in turn.

#### Cohesion

The cohesion condition comprises of two parts, both of which refer to
syntactic aspects of sentence composition, that is, what formal elements are
used to connect the sentences together. This condition subsumes formal
syntactic elements such as anaphoric pronouns, prepositions, discourse
markers, etc. Reinhart categorizes the methods for linking sentences together
into two distinct groups, and deems a text to be cohesive “iff for every pair
of sentences, at least one of the following conditions is met”:

  1. 1.

_Referentially linked:_ a pair of sentences $\langle S_{1},S_{2}\rangle$ is
referentially linked when the topic or a main expression in $S_{2}$ is
determined by a specific reference to an entity mentioned in $S_{1}$. An
example is a simple pronominal anaphor:

> “Dan is a nice fellow. Even Rosa likes him.”

The two sentences are cohesive as the underlined entities co-refer. Other
types of referential links may be prepositional NP links Elazar et al. (2022)
or bridging anaphora Hou (2021).

  2. 2.

Linked by a _semantic sentence connector_. A pair of sentences $\langle
S_{1},S_{2}\rangle$ are said to be connected if there exists a discourse
relation connecting them. These connectors encompass markers that indicate
various semantic relations, such as cause and effect, comparison, contrast,
temporal relations, exemplification, and more Prasad et al. (2008). Semantic
connectors also allow the text to present new topics into the discourse. An
example of a paragraph which is linked by semantic sentence connector:

> “Dan is a tough guy. But even he cried at that movie”.

The two sentences are cohesive because of the underlined connector. Note that
these connectors may be explicit, or they may be implicit (as in, e.g., Pitler
et al. (2009)).

#### Consistency

The consistency condition refers to the formal semantic aspects of the text,
that is, its logical coherence, and it plays an important role in how we
interpret texts to derive meanings.

Formally, this condition requires that for a set of sentences
$\\{S_{i}\\}_{i=0}^{n}$, the meaning of each sentence $S_{i}$ will be
consistent with the meanings of all previous sentences
$\\{S_{j}\\}_{j=0}^{i-1}$. This means that all sentences can be true within a
single scene, given the scene’s assumptions and restrictions. An example of a
violation of this condition is presented below:

> "My father is dead now. That’s why he has decided to smoke a pipe.” Freeman
> and Gathercole (1966)

The first and second sentences demonstrate the use of not one but two cohesive
devices (the use of anaphor, and the use of a connector “that’s why”).
However, they cannot be seamlessly integrated with our underlying assumption
about the world, namely that a deceased individual cannot make a decision or
smoke a pipe. Consequently, the text cannot be deemed coherent.111The
consistency condition was further presented and embraced in the work of
Honovich et al. (2021), as a means for more safely relying on automatically
generated texts.

#### Relevance

The relevance condition encompasses reflects pragmatic aspects of the
utterance, imposing constraints not only on the relationships among all
sentences in the discourse $\\{S_{i}\\}_{i=0}^{N}$ but also on their relation
to the underlying discourse topic and the contextual elements of the
utterance.

While the concept of relevance lacks a precise definition, it is typically
intuitively clear to a human. This condition subscribes to the Grice maxims of
relevance and manner and assumes a collaborative speaker. Grice (1975)
exemplifies relevance violation as follows: "A philosophy professor is asked
to evaluate the research ability of a student, John Smith. His reply is:"

> “John Smith has clear handwriting and he has never been late to class.”

While the professor’s answer is cohesive and consistent with the question, it
lacks relevance. This, in turn, deems the letter non-collaborative and thus
implies a negative assessment. In this work, we assume a collaborative speaker
and thus require relevance of all sentences to the topic at hand.

All in all, Reinhart’s theory presents a set of conditions that encapsulate
the fundamental aspects of coherence, aiming to determine the coherence of a
given text. Here we propose, to design NLP tasks that act as proxies for these
conditions.

##  3 Research Hypotheses and Computational Tasks

At the heart of our approach is the proposal to use the set of conditions
proposed by Reinhart and realize them computationally using a minimal set of
NLP tasks, which capture the specific coherence features in the three
conditions. We suggest that this set of tasks jointly captures coherence
overall.

We hypothesize that a model trained jointly on all of the tasks will learn the
coherence features. We will verify this hypothesis using a coherence scoring
task, which gives an overall coherence score for a paragraph in two types of
benchmarks, one contains real-world text and the other generated text. We
further hypothesize that the performance of a model jointly trained on all
tasks will be better than ones trained on each task separately.

The remainder of this section elaborates on the set of proxy tasks we propose
and motivates them by the coherence conditions they reflect. Here we define
the tasks formally, and in the next section, we will elaborate on the
experimental setup (data, model, metrics) for each of them.

#### The Sentence Reordering Task:

This is a self-supervised task proposed by Lapata (2003), which involves
reordering shuffled sentences to their original coherent form. The model’s
purpose is to reorganize them to the original order.

Input: a list of $N$ sentences ${\\{S_{i}\\}}_{i=0}^{N}$ that were randomly
shuffled.

Output: The original order

An example of the task is presented in Fig 1.

Figure 1: An example to illustrate the sentence ordering task. The set of
sentences is confusing in its current order. We aim to reorganize them in a
more coherent order.

Given that a naturally-ordered discourse exhibits greater coherence than
random permutations of the sentences Lin et al. (2011a), the sentence
reordering task was introduced. The premise is that a model capable of
reconstructing a paragraph into its most coherent form would necessitate
features related to the flow of information from one sentence to another
within the text, and to the semantically consistent relation among sentences.
Hence, this task encompasses elements of both cohesion and consistency
conditions.

#### The Irrelevant-Sentence Detection Task:

We propose a new self-supervised task where the model identifies irrelevant
sentence inserted in a coherent paragraph. The model should detect which
sentence is the irrelevant one.

Input: a paragraph consisting of a list of $N$ sentences
${\\{S_{i}\\}}_{i=0}^{N}$. One of them is irrelevant to the other sentences in
the paragraph.

Output: the irrelevant one - $S_{i}$.

Here is an example of the task with the input:

> (1) Milton has three children.
> (2) He was excited to go to the World Series game.
> (3) Milton isn’t happy with life.
> (4) He makes a choice.
> (5) He leaves his kids.
> (6) He moves away from them never to be seen again.

The output is sentence (2) - which is irrelevant to the topic of the
paragraph.

Training a model on this task involves acquiring the ability to differentiate
a sentence’s relevance to other sentences and to the overarching discourse
topic and context. Therefore, this task serves as a proxy for the relevance
condition.

#### The Discourse-Relation Recognition Task:

Given a pair of sentences, which are considered our discourse units, we aim to
recognize the discourse relation between them, reflecting notions such as
cause and effect, comparison, contrast, temporal relations, exemplification,
and so on.

The task purpose is to predict the discourse relation $R$ that holds between
them, such that $R(DU_{1},DU_{2})$.222Specifically, we focus on recognizing
_implicit_ discourse relation because preliminary experiments have shown that
implicit relation capture semantic representation better since they capture
subtler, contextually rich, and more nuanced semantic information

Input: a pair of discourse units (DUs) (sentences or clauses): $DU_{1}$ and
$DU_{2}$.

Output: the discourse relation between them $R$

Here is an example of this task with the input:

> “John worked all night. He slept all day today.”

The sentences are connected with the implicit relation marker “so” reflecting
a CONTINGENCY output.

Training a model on this task involves identifying the discourse relation
between sentence pairs. Incorporating this task enhances the model’s
capability to establish a discourse connection between the sentences, thereby
responding to the requirement of using a sentence-connector as (sub)condition
of cohesion.

#### The Noun Phrase (NP) Enrichment Task:

Proposed by Elazar et al. (2022), this task involves identifying prepositions
that are linking between noun entities in a text. Given a pair of noun phrases
$\langle NP_{1},NP_{2}\rangle$, the purpose of the task is to determine if
there exists a preposition-mediated relation between a pair of nouns, and if
there is one, determine the preposition $p$ that best describes their relation
$p(NP_{1}NP_{2})$. The task purpose is to produce the relation between each
pair of entities.

Input: a paragraph containing different NP entities $\\{{NP_{i}}\\}_{i=0}^{k}$
($k$ is the number of NP entities in the text).

Output: preposition relation between all NP pairs:
$R_{ij}=p_{ij}(NP_{i},NP_{j})$.

For example, in the paragraph:

> “Crown Princess Mary of Denmark gives birth to male child.”

there are 4 NP and therefore 12 NP pairs. The output for all pairs is:

> of(crown Princess Mary, Denmark), by(birth, crown Princess Mary), of(male
> child, crown Princess Mary), in(birth, Denmark), in(male child, Denmark),
> of(birth, male child)

A model trained on this task contains the features required for capturing the
interrelation between nouns. Therefore, serves as a proxy for the referential
linking (sub)condition of cohesion.

#### The Natural Language Inference Task:

The task introduced by Bowman et al. (2015) aims to determine the truth value
of a hypothesis based on a provided premise.

Input: a premise $P$ and hypothesis $H$.

Output: whether the hypothesis entails, contradicts, or is neutral based on
the information in the premise.

For example: for the input:

> Premise: ‘A man inspects the uniform of a figure in some East Asian
> country.’
>
> Hypothesis: ‘The man is sleeping.’

The output will be “contradiction” as the hypothesis here contradicts the
premise.

NLI serves as a task for evaluating NLP models’ ability to comprehend
language, reason, and capture logical relationships between sentences. Hence,
it acts as a proxy for the consistency condition.

Overall, we propose to train a model jointly on the aforementioned five tasks:
sentence reordering (SRO), irrelevance sentence recognition (ISR), discourse
relation recognition (DRR), NP enrichment (NPE), and natural language
inference (NLI). Those tasks, taken together, cover all the conditions for
coherence based on Reinhart’s definition of coherence, and therefore, we
hypothesize, that this model will be an effective coherence detection model.
Due to the information sharing among the tasks during training, we also expect
that the performance of this model on the specific tasks will surpass the
performance of a model trained on each task individually.

#### The Coherence Scoring Task

To confirm our hypothesis that the proposed model captures coherence, we
evaluate its performances on the coherence scoring task where we aim to
predict the coherence score for a given text as would be assessed by a human
reader. The model aims to produce an overall coherence score.

Input: a text $P$.

Output: coherence score $C$.

In what follows we first present our experimental setup for each individual
task (Section 4), we then elaborate the experimental setup for assessing
coherence overall (Section 5), and finally detail our results and analysis
(Section 6).

##  4 Task-Specific Experimental Setup

For each task (Section 3) we elaborate on its dataset, evaluation metrics, and
models we use to solve it. For each task, we experiment with two kinds of
models: classification-based which is finetuned on BERT model, and generation-
based which is finetuned on T5 model, and where relevant we elaborate on their
different settings. Table 1 summarizes the datasets we use for the various
tasks and main statistics.

Task | Dataset | Split | Per Instance
---|---|---|---
|  | Train | Validation | Test | Max #tokens | Avg #tokens | Max #sentences | Avg #sentences
SRO | RocStories | 68k | 14k | 14k | 135 | 57 | 5 | 5
ISR | RocStories | 68k | 14k | 14k | 152 | 77 | 6 | 6
DRR | PDTB3 | 17.5k | 1.7k | 1.5k | 556 | 30 | 2 | 2
NPE | TNE | 3.5k | 500 | 500 | 284 | 163 | 15 | 6.9
NLI | MNLI | 393k | 7.5k | 2.5k | (194,70) | (20,10) | (8,8) | (2,2)

Table 1: The dataset used for each task and the train/dev/test split size with
the max and average number of tokens and sentences. For the NLI task (x,y)
refer to the numbers of (premise, hypothesis) respectively.

###  4.1 The SRO task

#### Dataset:

We used the ROCStories Mostafazadeh et al. (2016) dataset which contains
5-sentence stories. To train the model on this task we shuffled the sentences
and aimed to predict the right order.

#### Models:

We use two kinds of models, classification- and generation-based.

Classification-Based Modeling: We adopt the topological sort architecture of
Shrimai Prabhumoye (2020). Both in training and inference, for each paragraph,
all sentence pairs are phrased as triplets $\langle S_{i},C_{k},S_{j}\rangle$
where $C_{k}$ is the pair relation indicating whether $S_{i}$ comes before or
after $S_{j}$. The model architecture for the task has two components. The
first is a binary classification head that predicts the pair relations. In the
second stage, the predicted paragraph order $O$ is produced using the
topological sort algorithm Tarjan (1976) (see Appendix D).

Generation-Based Modeling: The model’s input is comprised of the shuffled
sentences as well as a prompt: "reorder: what is the order of the sentences so
that the paragraph is coherent? sentence 1: $\langle S_{1}\rangle$ sentence 2:
$\langle S_{2}\rangle$ … $\langle S_{N}\rangle$". The target is a list of
position markers $Y={Y_{1},Y_{2},...,Y_{N}}$ where $Y_{i}$ denotes the
position of the $i_{th}$ sentence of the corresponding ordered sequence
$S_{i}$ in the shuffled input. The ordered paragraph can be reconstructed as
$O=\\{S_{y_{i}}\\}_{i=0}^{N}$.

#### Evaluation:

We employ two commonly used evaluation metrics for the task:

  * •

Perfect Match Ratio (PMR): Chen et al. (2016) calculates the percentage of
samples for which the entire sequence was correctly predicted.

$PMR=\frac{1}{N}\sum_{i=1}^{N}1\\{O^{i}=P^{i}\\}$

  * •

Sentence Accuracy (ACC): Logeswaran et al. (2017a) calculates the percentage
of sentences for which their absolute position was correctly predicted.

$Acc=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{v_{i}}\sum_{j=1}^{v_{i}}1\\{O_{j}^{i}=P_{j}^{i}\\}$

###  4.2 The ISR task

#### Dataset:

We again used ROCStories as in the previous task. For each story, a sentence
for the rest of the ROCStories dataset was inserted at a random position. The
inserted sentences were produced in an unsupervised method. The inserted
sentence was randomly chosen among all other stories in the dataset. Since
both this task and the SRO task use the same benchmark, the split into
train/dev/test was done for both tasks is the same.

#### Models:

Here too we use two kinds of models, classification- and generation-based.

Classification-Based Modeling: The task architecture consists of two stages.
First, all sentence pairs are comprised into triplets of $\langle S_{i}$,
$S_{j},C_{k}\rangle$ where $C_{k}$ is the pair relation, i.e. whether $S_{i}$
and $S_{j}$ are relevant to one another. The relation is produced using a
binary classification head. In the second stage, the sentence with the lowest
relation score with the other sentences is the one deemed irrelevant.

Generation-Based Modeling: The input contains all sentences with the following
prompt: "relevance: what is the irrelevant sentence in the text? sentence1:
$\langle S_{1}\rangle$ sentence2: $\langle S_{2}\rangle$ sentence3: …$\langle
S_{N}\rangle$". The target is the index of the irrelevant sentence in the
paragraph.

#### Evaluation:

Our metrics for evaluating relevance is the accuracy metric on the percentage
of paragraphs where the model correctly detected what is the irrelevant
sentence:

$Accuracy=\frac{\\#correct\\_detection\\_irrelevant\\_S_{i}}{\\#stories}$

###  4.3 The DRR task

#### Dataset:

We used the Penn Discourse TreeBank3 (PDTB3) Level 2 dataset Miltsakaki et al.
(2004); Prasad et al. (2008, 2019). The PDTB is a resource for extracting
discourse relations between pairs of sentences. We used only the labels with
more than 100 instances, which leaves us with 14 senses from $L_{2}$. The
variability of data splits used in the literature is substantial, therefore,
we follow earlier work by Ji and Eisenstein (2015); Bai and Zhao (2018); Liu
et al. (2020); Xiang et al. (2022) using Sections 2-20, 0-1 and 21-22 for
training, validation and testing respectively. When multiple annotated labels
are present, we adopt the approach described by Qin et al. (2016) and consider
them as distinct instances during the training phase. During testing, if a
prediction matches any of the reference labels, it is considered correct. Read
more on the PDTB dataset and possible discourse relations in App A.

#### Models:

Again we use two kinds of models, classification- and generation-based.

Classification-Based Models: The model’s input is a pair of DUs: $\langle
DU_{1}\rangle$, $\langle DU_{2}\rangle$, and the model uses a classification
head to predict the discourse relation between them.

Generation-Based Models: The model’s input consists of an argument pair, where
the target for the model is not simply the final L2 discourse relation but in
a chain-of-thought method Wei et al. (2023). 333Preliminary experiments showed
that using chain-of-thought method for detecting the discourse relation
outperforms a simple $l_{2}$ discourse relation target It aims to mimic the
way humans choose the discourse relation, starting with predicting the
connector itself and then mapping it into $l_{2}$ and $l_{1}$ discourse
relation.

The model is required to navigate through the path and determine the connector
and relations and therefore the prompt for the input is: "discourse
relation:$\langle DU_{1}\rangle\langle DU_{2}\rangle$". and the target is:
"$\langle$l1 relation$\rangle\rightarrow\langle$l2
relation$\rangle\rightarrow\langle$connector$\rangle$".

#### Evaluation:

Our metrics for evaluating this task is the accuracy metric on the number of
sentence pairs the model correctly predicted the $L_{2}$ discourse relation:

$Accuracy=\frac{\\#correct\\_discourse\\_relation}{\\#discourse\\_relation\\_pairs}$

###  4.4 The NPE task

#### Dataset:

We used the TNE dataset Elazar et al. (2022) which contains 4.5k documents and
relation between every noun pair in it (with a total number of nouns is 190k
and a total number of NP relations of 1M). There are 28 possible prepositions
(including no relation). More about the TNE dataset and the possible NP-
relations used is in Appendix B.

#### Models:

We use two kinds of models, classification- and generation-based.

Classification-Based Modeling: The architecture for this task uses a new
classification head which aims to classify, for each NP anchor-complement pair
$\langle{NP_{i}},{NP_{j}}\rangle$ the preposition connecting the NP pair (no-
relation is an option).

Specifically, in order to capture complex syntactic relationships, we use an
extension of the Bi-Affine architecture Dozat and Manning (2017) for
predicting the preposition relation of each pair of NPs. The embedding for
each NP is obtained through a pooling operation applied to all tokens that
represent the respective NP. The head finally predicts the preposition between
the pair using the NP’s anchor and complement representations. Fig 2
illustrates the token head.

Figure 2: Illustration of the token head which contains several stages:
starting with (1) embedding for each token in the text, (2) creating an
embedding for each NP when it acts as the complement and the anchor
separately, (3) a representation for each NP pair and finally (4) a
classification layer.

Generation-Based Modeling: For this task, each document has several
instantiations as the number of NPs in it. In each instantiation, the
preposition relation between a specific $NP_{i}$ and another specific $NP_{j}$
is generated by the model. The model’s input is a modified version of the
document text, such that for $NP_{i}$ "*" appears before and after it, and
"**" appears before and after $NP_{j}$. In addition, a prompt is added before
the text: "coreference text: what is the preposition relations between
<$NP_{i}$> and <$NP_{j}$>? text: <$P$>". The target is the preposition. While
splitting the data into train/dev/test based on documents and not NPs in order
to avoid data leakage.

#### Evaluation:

Our metrics for evaluating this task are the precision, recall, and F1 on the
NP pairs with the relation between them.

###  4.5 The NLI task

#### Dataset:

We used MNLI dataset Williams et al. (2018).

#### Models:

Classification-Based Modeling: The model’s input is a pair of hypothesis and
premise: $\langle P,H\rangle$. The model uses a 3-way classification head to
predict the relation label between them.

Generation-Based Modeling: The input is the premise and hypothesis pair with a
prompt: "mnli: Does this hypothesis contradict, entail or neutral with the
premise? hypothesis: $\langle H\rangle$ premise: $\langle P\rangle$". The
target is the relation between the sentences.

#### Evaluation:

Our metric for evaluating the task is the accuracy metric on the amount of
hypothesis-premise pairs that the model correctly predicts their relation:

$Accuracy=\frac{\\#correctly\\_labeled\\_P\\_H\\_pairs}{\\#H\\_P\\_pairs}$

###  4.6 Putting It All Together

To test our hypotheses, we implement for both the classification-based and
generation-based variants a model that is trained to solve all tasks jointly.

For the classification-based model, we use multi-task learning (MTL). That is,
we finetuned a BERT Devlin et al. (2019)-based encoder that is shared among
the different tasks, which a dedicated unique classification head created for
each task.

For every task $j$, a shared encoder is employed, which takes a text as input.
Each task is associated with an exclusive classification head that predicts
the same target as elaborated in the preceding sections. The classification-
based architecture is presented in Fig 3.

Figure 3: Illustration of the encoder-only based model architecture where the
input to the encoder is a pair of sentences for all tasks but NPE and for it
the input is a single document text. Additionally, for NPE the classification
head receives a list of the tokens IDs of the different NPs in it.

To overcome the known issue of forgetting in MTL, an interleaved training
approach was used. This method alternately trains the tasks, with each batch
containing training samples from a specific task, different from the batch
preceding and following task. By employing this approach, the problem of
forgetting during MTL training is effectively mitigated. Further information
on MTL can be found in App C.

For the generated-based model we use T5 Raffel et al. (2020) encoder-decoder
model as our backbone model. A main advantage of using T5 model is its
flexibility in fine-tuning several tasks simultaneously by employing different
prompts for different tasks. The prompt for each task is identical as was when
we fine-tuned T5 for each task individually. The main difference is training
all examples simultaneously when each batch contains samples from the specific
task which differs from the batch beforehand.

##  5 Coherence Assessment Experimental Setup

This section elaborates on the datasets and metrics for the task of coherence
scoring task. Here again, we experiment with two kinds of models —
classification-based and generation-based — and we elaborate on each model
setting. The datasets sizes and splits are given in Tab 2.

Dataset | Split | Per Instance
---|---|---
| Train | Validation | Test | Max #tokens | Avg #tokens | Max #sentences | Avg #sentences
GCDC | 3.6k | 800 | 800 | 333 | 156 | 10 | 32
CoheSentia | 350 | 75 | 75 | 226 | 150 | 15 | 6.5

Table 2: Datasets for Coherence Scoring.

#### Datasets:

We use two datasets, one is GCDC Lai and Tetreault (2018) and the other is
CoheSentia dataset anonymous (underreview). The first consists of real-world
text from 4 domains: Clinton emails, Enron emails, Yahoo answers, and Yelp
reviews. Every document is assigned a discrete coherence score (1 - not
coherent to 3 - highly coherent). The second consists of GPT-3 generated
stories. Each story is assigned a discrete coherence score (1 - not coherent
to 5 - highly coherent). We used the incremental final score for each story.

#### Models:

As for the tasks presented above two neural models are used, an encoder-only
model and a generative encoder-decoder model. Both produce the coherence score
of the paragraph.

Classification-Based Models: The model’s input is the text $P$ and the model
uses a 3-way or 5-way classification head based on the dataset to predict the
coherence score $C$ of the text.

Generation-Based Models: The models’ input is the text with prompts based on
the dataset:

  * •

For GCDC: "GCDC coherence: what is the coherence score of the text (3 - high,
1 - low)? text: $\langle P\rangle$"

  * •

For CoheSentia: "CoheSentia coherence: what is the coherence score of the text
(5 - high, 1 - low)? title: $\langle T\rangle$ text: $\langle P\rangle$"

The target is the coherence score of the text.

#### Evaluation:

To continue the work of Lai and Tetreault (2018) our metrics for evaluating
this task is the accuracy metric on the final coherence score of the text:

$Accuracy=\frac{\\#correct\\_coherence\\_score}{\\#paragraphs}$

##  6 Results

In 6.1 we show that our hypothesis that training a unified model on all tasks
jointly indeed produces better performances rather than training a model on
each task individually. We also get new SOTA results for most tasks. Regarding
our hypothesis that this unified model acts as a coherence model, we show a
substantial performance improvement on coherence scoring for both benchmarks
in 6.2.

###  6.1 Task-Specific Results

#### Comparing Results for the Different Tasks

We assess task performance for models trained separately on each task and all
tasks jointly. We report outcomes for classification-based and generation-
based models. Furthermore, we compare task performance with respective SOTA
benchmarks.

In Table 3 we present the performances of all tasks compared to each one SOTA
results. For all tasks, the generation-model exhibits higher performance
compared to classification-models. This difference can be due to the fact that
we employed relatively simple classification models to maintain a unified
architecture across all tasks, while the T5-based model was more fine-tuned
for each specific task. It can also be because T5 models are more suitable for
MTL architecture.

Furthermore, the findings illustrate that for all tasks, exclusive fine-tuning
for each individual task yielded inferior performance compared to a model that
was jointly fine-tuned.

For SRO, ISR, and DRR tasks, the joint fine-tuning model gains a notable
enhancement in performance and outperforms SOTA results. For the NPE task, we
see a substantial performance gain of recall while for precision we don’t
surpass SOTA results. The SOTA results perform well on precision and low on
recall in contradiction to our model which performs similarly for precision
and recall.

However, for the NLI task, in contrast with the other tasks, our model does
not outperform SOTA results. We hypothesize that the reason for this is
attributed to the substantial disparity in the number of parameters between
our fine-tuned backbone T5 model and the SOTA model, with the latter being
based on a significantly larger 11 billion parameters.

Model | SRO | ISR | DRR | NPE | NLI
---|---|---|---|---|---
| PMR | ACC | Accuracy | Accuracy | F1 | P | R | Accuracy
SOTA | 81.9 | 90.8 | - | 64.7 | 64.0 | 80.5 | 53.1 | 92.0
Ours-Individual (bert-large) | 51.8 | 69.5 | 60.4 | 60.0 | 53.1 | 67.1 | 44.0 | 87.4
Ours-ALL (bert-large) | 67.1 | 83.2 | 78.6 | 65.7 | 64.4 | 79.8 | 54.2 | 90.2
Ours-Individual (t5-large) | 75.7 | 87.8 | 80.4 | 64.8 | 59.8 | 68.5 | 53.1 | 89.9
Ours-ALL (t5-large) | 83.8 | 92.1 | 82.2 | 67.3 | 76.7 | 76.7 | 76.7 | 91.5

Table 3: Results for all tasks compared to SOTA performances. The SOTA model
for SRO is ReBARTBasu Roy Chowdhury et al. (2021), for DRR is Contrastive
LearningLong and Webber (2023), for NPE is TNE Elazar et al. (2022) and for
NLI T5-11B Raffel et al. (2020).

In summary, the overall performance across all tasks surpasses that of a model
trained individually for each task. Furthermore, in the case of all tasks but
NLI, the joint model outperforms the SOTA results, showcasing the advantage of
training a unified model on all tasks simultaneously.

This highlights the substantial contribution of information gathered from
other tasks to enhance the performance of each specific task, despite the
relatively simple model architecture we used.

#### The Effect of Different Tasks on One Another

In Fig. 4 we show the performances of the different tasks when finetuned on
different amounts and types of other tasks, for BERT-base and BERT-large
models.

The Figure demonstrates that as the number of tasks on which the base model
underwent finetuning increased, there is a corresponding rise in the
performance for each individual task. This finding supports our hypothesis
that the shared information among tasks enhances feature capturing and
utilization, leading to improved performance.

By analyzing the graphs, we can discern the relative impact of different tasks
on each other. For example, surprisingly, DRR has a relatively minimal effect
on the SRO task. This could potentially be attributed to the limited number of
examples available for the DRR task in comparison to the other tasks. It is
also evident that the performance of the various tasks is significantly
influenced by the NLI task.

An important finding is the significant improvement achieved through the use
of the ISR task on other tasks’ performances. To the best of our knowledge, we
are the first to employ this method, and it offers the advantage of being a
self-supervised task. We believe that this task effectively captures relevance
errors and recommend further exploration of its potential in future research.

By observing the similar patterns in trends for both BERT-base and BERT-large,
we conclude that although the ultimate performance diverges due to variations
in the number of parameters in the base models, the impact of a specific task
on the remaining tasks exhibits similarity.

(a) SRO

(b) ISR

(c) DRR

(d) NPE

(e) NLI

Figure 4: Results for all tasks, for different permutations of tasks finetuned
upon. The labels is the number of tasks and in curly brackets which tasks (1 -
SRO, 2 - ISR, 3 - DRR, 4 - NPE, 5 - NLI).

###  6.2 Assessment of the Coherence Model

#### Comparing Results for Different Benchmarks

In Table 4, the results of the coherence scoring task on the GCDC and
CoheSentia datasets are showcased subsequent to finetuning our model, which
was created based on the coherence proxy tasks that we detailed previously.

When comparing the results to SOTA models, we observe an improvement of 30%
accuracy for CoheSentia and 22% accuracy for GCDC in the model’s ability to
capture coherence by utilizing the models that were previously finetuned on
the proposed set of tasks. This finding further highlights the fact that these
tasks effectively capture the fundamental aspects of coherence as we propose.

The CoheSentia dataset also demonstrates high performances, although it is
less pronounced compared to GCDC. This may be attributed to the fact that the
proxy tasks were trained on real-world data, and the generated text used in
CoheSentia may exhibit fewer shared features with real-world coherence scores.
Additionally, scoring coherence for automatically generated text can be more
challenging due to the presence of coherence errors that are less likely to
occur in human writing. It can also be because the scoring span for CoheSentia
is larger than for GCDC.

The tasks being related to specific conditions provide an opportunity for
further exploration of the reasons behind incoherence. However, we leave
investigating these aspects for future work.

Model | GCDC | CoheSentia
---|---|---
SOTA | 57.5 | 35.3
Ours-ALL (bert-large) | 72.5 | 55.7
Ours-ALL (t5-large) | 79.8 | 66.7

Table 4: Accuracy for coherence scoring task on both datasets GCDC and
CoheSentia. The SOTA model for GCDC is ParSeq by Lai and Tetreault (2018) and
for CoheSentia is by anonymous (underreview).

#### Comparing a Different Number of Finetuned tasks

Fig 5 demonstrates a clear trend: as the number of finetuned tasks increases,
the coherence score for both datasets improves. Especially, when more than 3
tasks are finetuned, the increase is more noticeable. Moreover, we can
identify the tasks that have a greater impact on the final coherence score
compared to others, specifically, NLI.

(a) GCDC dataset

(b) CoheSentia dataset

Figure 5: Results for coherence scoring for each dataset, for different
permutations of tasks finetuned upon. The labels is the number of tasks and in
curly brackets which tasks (1 - SRO, 2 - ISR, 3 - DRR, 4 - NPE, 5 - NLI).

##  7 Related Work

###  7.1 Linguistic and NLP Approaches for Coherence Assessment

In linguistics, various theories have been proposed to explicate what makes a
discourse coherent and study the relations between discourse elements,
including lexico-grammatical Halliday and Hasan (1976); Webber (1988); Hoey
(2005), entity-based Joshi and Weinstein (1981, 1993); Grosz et al. (1995),
psychological Kintsch and Dijk (1978); Graesser et al. (1994); Givon (1995),
semantic Hobbs (1979); Redeker (1990); Sanders et al. (1992), pragmatic
Widdowson (1978); Dijk (1979); Lascarides and Asher (1991)) and structural
Danes (1974); Grosz and Sidner (1986); Mann and Thompson (1988) theories.

These investigations have laid the groundwork for the development of coherence
modeling, a field that seeks to computationally assess the coherence of texts.
The entity-based local model for representing and evaluating text coherence,
put forth by Barzilay and Lee (2004); Barzilay and Lapata (2008), has been
particularly influential. Their approach involves portraying a text through an
entity grid, i.e., a 2D array, which captures the local transitions of
discourse entities across sentences to discern patterns that dictate
coherence. This model incorporates the concept of entity salience,
distinguishing transitions involving significant entities from those involving
less significant ones by quantifying entity occurrence frequency. Extending
the fundamental entity grid model, Elsner and Charniak (2011) incorporated
non-head nouns as entities within the grid. They introduced entity-specific
attributes like named entities, noun classes, and modifiers to differentiate
between entities of varying types. In an alternative perspective, Lin et al.
(2011a) focused on modeling the transitions of discourse roles for entities.

Guinaudeau and Strube (2013) proposed an unsupervised graph-based method for
coherence modeling. Their approach operates under the assumption that
sentences within a coherent discourse should exhibit shared structural
syntactic patterns. Similarly, Louis and Nenkova (2012) introduced a coherence
model based on syntactic patterns in text. Their method comprises both local
and global coherence models, with the former capturing the co-occurrence of
structural features in adjacent sentences and the latter capturing global
structures through clusters of sentences exhibiting similar syntax.

More recently, we have seen the adoption of neural networks for coherence
modeling, outperforming the aforementioned traditional statistical models.
These neural approaches leverage the ability to automatically learn relevant
features from unstructured text. A few approaches incorporate entity-grid
representations of text as input to a neural model Nguyen and Joty (2017);
Mohiuddin et al. (2018). Other approaches are end-to-end producing models
evaluated on SRO task with some focusing on capturing global context Li and
Jurafsky (2017); Logeswaran et al. (2017b); Cui et al. (2018); Bohn et al.
(2019); Kumar et al. (2019) and others focusing on capturing local coherence
Li and Hovy (2014); Cui et al. (2017); Mesgar and Strube (2018); Xu et al.
(2019); Chen et al. (2019); Moon et al. (2019); Mesgar et al. (2020). These
approaches, while effective, do not capture the coherence of a text fully,
whether they capture only local coherence, capture some of the features
related to coherence, or have trouble dealing with long transitions.

In our approach, we combine the use of several tasks, one of them is the SRO
task, each designed to encapsulate distinct facets of coherence. As a result,
our model doesn’t contain only SRO related features as previously was achieved
but also incorporates additional coherence indicators that are crucial for
detecting and assessing coherence.

###  7.2 Evaluation of Models for Coherence Assessment

Numerous approaches for assessing models for coherence detection exist. One
direct method involves obtaining expert ratings on document coherence and
training a classifier to predict these human assessments, which can be
categorical or continuous. This method is suitable for end tasks like essay
grading that demand expert judgment for definitive labels. Nevertheless,
acquiring human labels can be costly, and a need arises for a coherence
ranking applicable to general text without specific tasks or data in mind.

Henceforth, the prevailing trend in coherence evaluation methods has relied on
employing proxy tasks. Among these, the widely utilized unsupervised approach
involves sentences reordering Barzilay and Lapata (2008), based on the
assumption that naturally-ordered discourses exhibit higher coherence than
randomly permuted ones, thereby capturing coherence implicitly Lin et al.
(2011b). Other methods include readability assessment Guinaudeau and Strube
(2013); Mesgar and Strube (2016, 2018) and essay scoring Mesgar and Strube
(2018); Somasundaran et al. (2014); Tay et al. (2017). However, these
approaches not only entail significant costs but also risk evaluating the
model solely on downstream task-related features, potentially undermining
coherence assessment.

Given the widespread adoption of LLM and generative models, there has been a
growing interest in coherence evaluation. As a result, two benchmarks for
coherence assessment have been established. The first is GCDC by Lai and
Tetreault (2018), which comprises annotated coherence scores for real-world
text across four domains. The second is CoheSentia by anonymous (underreview),
which contains annotated text for coherence evaluation specifically focused on
GPT-3 generated text. In our work, we measured our models’ ability in
capturing coherence using both of those benchmarks.

##  8 Conclusion

In this paper, we propose a new coherence modeling method that is based on a
linguistic theory set of conditions for coherence: cohesion, consistency, and
relevance. Those conditions are said to capture coherence fully and
accurately. Based on those conditions, we proposed using 5 mostly known and
important NLP tasks, as proxy for those conditions and train an MTL model. We
show that the unified coherence model further achieves SOTA results on those
tasks. In addition, this model performs better on coherence scoring task
evaluated on both real-world text and generated text. We hope that using this
modeling method and proposed tasks in the future will further improve our
ability to quantify and evaluate the quality of text, whether generated or
human written.

## References

  * anonymous (underreview) anonymous. underreview.  Cohesentia: A novel benchmark of incremental versus holistic assessment of coherence in generated texts.
  * Bai and Zhao (2018) Hongxiao Bai and Hai Zhao. 2018.  Deep enhanced representation for implicit discourse relation recognition.
  * Barzilay and Lapata (2008) Regina Barzilay and Mirella Lapata. 2008.  Modeling local coherence: An entity-based approach.  _Computational Linguistics_ , 34(1):1–34.
  * Barzilay and Lee (2004) Regina Barzilay and Lillian Lee. 2004.  Catching the drift: Probabilistic content models, with applications to generation and summarization.  _HLT- NAACL 2004: Main Proceedings_ , 1:113–120.
  * Basu Roy Chowdhury et al. (2021) Somnath Basu Roy Chowdhury, Faeze Brahman, and Snigdha Chaturvedi. 2021.  Is everything in order? a simple way to order sentences.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10769–10779, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Bohn et al. (2019) Tanner Bohn, Yining Hu, Jinhang Zhang, and Charles X. Ling. 2019.  Learning sentence embeddings for coherence modelling and beyond.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.
  * Caruana (1997) Rich Caruana. 1997.  Multitask learning.  _Machine learning_ , 28:41–75.
  * Chen et al. (2019) Mingda Chen, Zewei Chu, and Kevin Gimpel. 2019.  Evaluation benchmarks and learning criteria for discourse-aware sentence representations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 649–662, Hong Kong, China. Association for Computational Linguistics.
  * Chen et al. (2016) Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. 2016.  Neural sentence ordering.
  * Cui et al. (2018) Baiyun Cui, Yingming Li, Ming Chen, and Zhongfei Zhang. 2018.  Deep attentive sentence ordering network.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4340–4349, Brussels, Belgium. Association for Computational Linguistics.
  * Cui et al. (2017) Baiyun Cui, Yingming Li, Yaqing Zhang, and Zhongfei Zhang. 2017.  Text coherence analysis based on deep neural network.  In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. ACM.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The pascal recognising textual entailment challenge.  pages 177–190.
  * Danes (1974) Frantisek Danes. 1974.  Functional sentence perspective and the organization of the text.  _functional sentence perspective_ , pages 106–128.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dijk (1979) Teun A Van Dijk. 1979.  Pragmatic connectives.  _Discourse processes_ , 3:447–456.
  * Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017.  Deep biaffine attention for neural dependency parsing.
  * Duong et al. (2015) Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015.  Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 845–850, Beijing, China. Association for Computational Linguistics.
  * Elazar et al. (2022) Yanai Elazar, Victoria Basmov, Yoav Goldberg, and Reut Tsarfaty. 2022.  Text-based np enrichment.
  * Elsner and Charniak (2011) Micha Elsner and Eugene Charniak. 2011.  Extending the entity grid with entity-specific features.  _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers_ , 2(11):125–129.
  * Feng et al. (2014) Vanessa Wei Feng, Ziheng Lin, and Graeme Hirst. 2014.  The impact of deep hierarchical discourse structures in the evaluation of text coherence.  _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , 1:940–949.
  * Freeman and Gathercole (1966) Thomas Freeman and CE Gathercole. 1966.  Perseveration—the clinical symptoms—in chronic schizophrenia and organic dementia.  _The British Journal of Psychiatry_ , 112(482):27–32.
  * Givon (1995) T Givon. 1995.  Coherence in text vs. coherence in mind.  _Coherence in spontaneous text_ , pages 31–59.
  * Graesser et al. (1994) Arthur C Graesser, Murray Singer, and Tom Trabasso. 1994.  Constructing inferences during narrative text comprehension.  _Psychological review_ , 101(3):371.
  * Grice (1975) Herbert P Grice. 1975.  Logic and conversation.  _In Speech acts_ , pages 41–58.
  * Grosz and Sidner (1986) Barbara J. Grosz and Candace L. Sidner. 1986.  Attention, intentions, and the structure of discourse.  _Computational Linguistics_ , 12:175–204.
  * Grosz et al. (1995) Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. 1995.  Centering: A framework for modeling the local coherence of discourse.  _Computational Linguistics_ , 21(2):16–18,27–28,82.
  * Guan et al. (2021) Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wenbiao Ding, and Minlie Huang. 2021.  Long text generation by modeling sentence-level and discourse-level coherence.
  * Guinaudeau and Strube (2013) Camille Guinaudeau and Michael Strube. 2013.  Graph-based local coherence modeling.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 93–103, Sofia, Bulgaria. Association for Computational Linguistics.
  * Halliday and Hasan (1976) M.A.K. Halliday and Ruqaiya Hasan. 1976.  _Cohesion in English_.  Longman, Dallas, Texas.
  * Hashimoto et al. (2017) Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017.  A joint many-task model: Growing a neural network for multiple NLP tasks.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1923–1933, Copenhagen, Denmark. Association for Computational Linguistics.
  * Hobbs (1979) Jerry R Hobbs. 1979.  Coherence and coreference.  _Cognitive science_ , 3(1):67–90.
  * Hoey (2005) Michael Hoey. 2005.  _Lexical priming: A new theory of words and language_.  Psychology Press.
  * Honovich et al. (2021) Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.  $q^{2}$: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering.
  * Hou (2021) Yufang Hou. 2021.  End-to-end neural information status classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1377–1388, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ji and Eisenstein (2015) Yangfeng Ji and Jacob Eisenstein. 2015.  One vector is not enough: Entity-augmented distributed semantics for discourse relations.  _Transactions of the Association for Computational Linguistics_ , 3:329–344.
  * Joshi and Weinstein (1981) Aravind K Joshi and Scott Weinstein. 1981.  Control of inference: Role of some aspects of discourse structure-centering.  _IJCAI_ , pages 385–387.
  * Joshi and Weinstein (1993) Aravind K Joshi and Scott Weinstein. 1993.  Pronouns, names, and the centering of attention in discourse.  _Cognitive science_ , 17(3):311–347.
  * Kintsch and Dijk (1978) Walter Kintsch and Teun A Van Dijk. 1978.  Toward a model of text comprehension and production.  _Psychological review_ , 85(5):363.
  * Kumar et al. (2019) Pawan Kumar, Dhanajit Brahma, Harish Karnick, and Piyush Rai. 2019.  Deep attentive ranking networks for learning to order sentences.
  * Lai and Tetreault (2018) Alice Lai and Joel Tetreault. 2018.  Discourse coherence in the wild: A dataset, evaluation and methods.  _In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue. Association for Computational Linguistics_ , 1:214–223.
  * Lapata (2003) Mirella Lapata. 2003.  Probabilistic text structuring: Experiments with sentence ordering.  _EMNLPIJCNLP_ , pages 2273–2283.
  * Lascarides and Asher (1991) Alex Lascarides and Nicholas Asher. 1991.  Discourse relations and defeasible knowledge.  _Computational Linguistics_ , 1:55–62.
  * Li and Hovy (2014) Jiwei Li and Eduard Hovy. 2014.  A model of coherence based on distributed sentence representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2039–2048, Doha, Qatar. Association for Computational Linguistics.
  * Li and Jurafsky (2017) Jiwei Li and Dan Jurafsky. 2017.  Neural net models for open-domain discourse coherence.
  * Lin et al. (2011a) Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011a.  Automatically evaluating text coherence using discourse relations.  _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics_ , 1:997–1006.
  * Lin et al. (2011b) Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011b.  Automatically evaluating text coherence using discourse relations.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 997–1006, Portland, Oregon, USA. Association for Computational Linguistics.
  * Liu et al. (2020) Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2020.  On the importance of word and sentence representation learning in implicit discourse relation classification.
  * Logeswaran et al. (2017a) Lajanugen Logeswaran, Honglak Lee, and Dragomir Radev. 2017a.  Sentence ordering and coherence modeling using recurrent neural networks.
  * Logeswaran et al. (2017b) Lajanugen Logeswaran, Honglak Lee, and Dragomir Radev. 2017b.  Sentence ordering and coherence modeling using recurrent neural networks.
  * Long and Webber (2023) Wanqiu Long and Bonnie Webber. 2023.  Facilitating contrastive learning of discourse relational senses by exploiting the hierarchy of sense relations.
  * Louis and Nenkova (2012) Annie Louis and Ani Nenkova. 2012.  A coherence model based on syntactic patterns.  _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL, Stroudsburg, PA, USA. Association for Computational Linguistics_ , 1:1157–1168.
  * Mann and Thompson (1988) William C Mann and Sandra A Thompson. 1988.  Rhetorical structure theory: Toward a functional theory of text organization.  _Text Interdisciplinary Journal for the Study of Discourse_ , 1:243–281.
  * Mesgar et al. (2020) Mohsen Mesgar, Sebastian Bücker, and Iryna Gurevych. 2020.  Dialogue coherence assessment without explicit dialogue act labels.
  * Mesgar and Strube (2016) Mohsen Mesgar and Michael Strube. 2016.  Lexical coherence graph modeling using word embeddings.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1414–1423, San Diego, California. Association for Computational Linguistics.
  * Mesgar and Strube (2018) Mohsen Mesgar and Michael Strube. 2018.  A neural local coherence model for text quality assessment.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4328–4339, Brussels, Belgium. Association for Computational Linguistics.
  * Miltsakaki et al. (2000) Miltsakaki, Eleni, Kukich, and Karen. 2000.  Automated evaluation of coherence in student essays.
  * Miltsakaki et al. (2004) Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004.  The Penn Discourse Treebank.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Mohiuddin et al. (2018) Tasnim Mohiuddin, Shafiq Joty, and Dat Tien Nguyen. 2018.  Coherence modeling of asynchronous conversations: A neural entity grid approach.
  * Moon et al. (2019) Han Cheol Moon, Tasnim Mohiuddin, Shafiq Joty, and Xu Chi. 2019.  A unified neural coherence model.
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016.  A corpus and evaluation framework for deeper understanding of commonsense stories.
  * Nguyen and Joty (2017) Tien Nguyen and Joty. 2017.  A neural local coherence model.  _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , 1:1320–1330.
  * Pitler et al. (2009) Emily Pitler, Annie Louis, and Ani Nenkova. 2009.  Automatic sense prediction for implicit discourse relations in text.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 683–691, Suntec, Singapore. Association for Computational Linguistics.
  * Prasad et al. (2019) Prasad, Rashmi, Webber, Bonnie, Lee, Alan, and Aravind Joshi. 2019.  Penn discourse treebank version 3.0.
  * Prasad et al. (2008) Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008.  The Penn Discourse TreeBank 2.0. In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08)_ , Marrakech, Morocco. European Language Resources Association (ELRA).
  * Qin et al. (2016) Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016.  Shallow discourse parsing using convolutional neural network.  In _Proceedings of the CoNLL-16 shared task_ , pages 70–77, Berlin, Germany. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Redeker (1990) Gisela Redeker. 1990.  Ideational and pragmatic markers of discourse structure.  _Journal of Pragmatics_ , 14:367–381.
  * Reinhart (1980) Tanya Reinhart. 1980.  _Conditions for text coherence. Poetics Today 1(4): 16t-180_ , volume 1.
  * Ruder et al. (2018) Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2018.  Latent multi-task architecture learning.
  * Sanders et al. (1992) Ted JM Sanders, Wilbert PM Spooren, and Leo GM Noordman. 1992.  Toward a taxonomy of coherence relations.  _Discourse processes_ , 15:1–35.
  * Shrimai Prabhumoye (2020) Alan W Black Shrimai Prabhumoye, Ruslan Salakhutdinov. 2020.  Topological sort for sentence ordering.
  * Søgaard and Goldberg (2016) Anders Søgaard and Yoav Goldberg. 2016.  Deep multi-task learning with low level tasks supervised at lower layers.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 231–235, Berlin, Germany. Association for Computational Linguistics.
  * Somasundaran et al. (2014) Swapna Somasundaran, Jill Burstein, and Martin Chodorow. 2014.  Lexical chaining for measuring discourse coherence quality in test-taker essays.  _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , 1:950–961.
  * Tarjan (1976) Robert Endre Tarjan. 1976.  Edge-disjoint spanning trees and depth-first search.  _Acta Informatica_ , 6(2):171–185.
  * Tay et al. (2017) Yi Tay, Minh C. Phan, Luu Anh Tuan, and Siu Cheung Hui. 2017.  Skipflow: Incorporating neural coherence features for end-to-end automatic text scoring.
  * Webber (1988) Bonnie Lynn Webber. 1988.  Tense as discourse anaphor.  _Computational Linguistics_ , 14(2):61–73.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-thought prompting elicits reasoning in large language models.
  * Widdowson (1978) Henry George Widdowson. 1978.  _Teaching language as communication_.  Oxford University Press.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Xiang et al. (2022) Wei Xiang, Bang Wang, Lu Dai, and Yijun Mo. 2022.  Encoding and fusing semantic connection and linguistic evidence for implicit discourse relation recognition.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3247–3257, Dublin, Ireland. Association for Computational Linguistics.
  * Xu et al. (2018) Jingjing Xu, Xuancheng Ren, Yi Zhang, Qi Zeng, Xiaoyan Cai, and Xu Sun. 2018.  A skeleton-based model for promoting coherence among sentences in narrative story generation.
  * Xu et al. (2019) Peng Xu, Hamidreza Saghir, Jin Sung Kang, Teng Long, Avishek Joey Bose, Yanshuai Cao, and Jackie Chi Kit Cheung. 2019.  A cross-domain transferable neural coherence model.
  * Yang and Hospedales (2017) Yongxin Yang and Timothy M. Hospedales. 2017.  Trace norm regularised deep multi-task learning.
  * Yi et al. (2019) Sanghyun Yi, Rahul Goel, Chandra Khatri, Alessandra Cervone, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur. 2019.  Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators.

##  Appendix A PDTB dataset

The Penn Discourse TreeBank (PDTB) is the second commonly used dataset for
discourse relations detection Miltsakaki et al. (2004); Prasad et al. (2008).
The dataset consists of paragraphs where each paragraph is split into several
small units and a relation between a pair of consecutive units was annotated.
PDTB labeling is lexically grounded. Instead of asking annotators to directly
tag the discourse relation between text spans, they were given a list of
discourse connectives, words that signal discourse relations, like because,
although, when, since, or as a result. In a part of a text where these words
marked a discourse relation between two text spans, the connective and the
spans were then annotated.

In the example below: the phrase as a result signals a causal relationship
between what PDTB calls Arg1 and Arg2 (the relation is underlined in bold).

> “Jewelry displays in department stores were often cluttered and uninspired.
> And the merchandise was, well, fake. As a result, marketers of faux gems
> steadily lost space in department stores to more fashionable
> rivals—cosmetics makers.”

Not all coherence relations are marked by an explicit discourse connective as
in the example above, and so the PDTB also annotates pairs of neighboring
sentences with no explicit signal. The annotator first chooses the word or
phrase that could have been its signal, and then labels its sense. An example
of this is shown below:

> In July, the Environmental Protection Agency imposed a gradual ban on
> virtually all uses of asbestos. (implicit=as a result) By 1997, almost all
> remaining uses of cancer-causing asbestos will be outlawed.

after adding the discourse marker “as a result” it is mapped into the final
discourse relation (Contingency.Cause sense).

As we can probably imagine predicting the discourse relations is a lot easier
when it is an explicit relation rather than an implicit one. For PDTB, the
possible relations between sentences are split into several levels: level1,
level2, and level3. We will use the level2 relations which consist of 14
unevenly distributed possible relations: manner, asynchronous, cause,
cause+belief, condition, level-of-detail, synchronous, substitution,
equivalence, concession, instantiation, purpose, conjunction, contrast.

##  Appendix B TNE dataset

The Text-Based NP Enrichment Task Dataset (TNE) is an NLU task, which focuses
on relations between noun phrases (NPs) that can be mediated via prepositions.
The dataset was annotated exhaustively with all possible links between the NPs
in each document. The main data comes from WikiNews. It also contains an
additional set of 509 documents to serve as out-of-domain, from Books, IMDB
reviews, and Reddit.

This dataset advantage is that it not only contains real and sometimes long
paragraphs but also that it was annotated into several possible labels. Among
them, there is the title for each paragraph, as well as what are the noun
phrases and their coreference clusters and their relation to other noun
phrases.

There are 28 unevenly distributed possible preposition relations between pair
of nouns.

In Fig 6 you can see an example of a text with all noun entities marked.

Figure 6: An example of NP-enriched document from the TNE dataset. In the
upper image there is the original paragraph, in the middle - the document
after enrichment (in green are the NP-enrichments explicitly mentioned and in
red are that are not explicitly mentioned), in the bottom are the clusters for
some of the NP’s.

##  Appendix C Multi-Task Learning (MTL)

Machine learning models typically focus on performing a single task and
accordingly tune their parameters to optimize a particular loss function
(e.g., mean square error or binary cross entropy). This Single-Task Learning
(STL) setup ignores useful information that could be learned from other
related tasks. In contrast, MTL is “an approach to inductive transfer that
improves generalization by using the domain information contained in the
training signals of related tasks as an inductive bias” Caruana (1997). MTL is
a type of transfer learning where two or more tasks are learned
simultaneously, the tasks are either of the same importance, or there is a
main task that the model focuses on and one or more auxiliary tasks to help
improve performance on the main one. The model optimizes a cost function that
combines the loss functions of the different tasks. One simple approach is to
calculate a weighted sum over the individual functions:

$Loss=\sum_{n=1}^{T}\lambda_{i}L_{i}$

where T is the number of tasks, $\lambda_{i}$ is the weight of the i-th task
and $L_{i}$ is its loss function. All tasks are either assigned equal weights
(e.g., $\lambda_{i}$ = 1) or different ones to give more attention to
particular tasks.

MTL is achieved by parameter sharing between tasks. There are several methods
to share parameters between the tasks:

  * •

Hard Sharing Caruana (1997): the model hidden layers or encodings are shared
between tasks while keeping the output layers task-specific. The main benefit
of this technique is that it mitigates the risk of overfitting. By learning
multiple tasks simultaneously, the model needs to discover representations
that can handle various tasks, thereby reducing overfitting on any single
task. However, this method’s effectiveness diminishes when the tasks are not
closely related or demand reasoning at different levels.

  * •

Soft Sharing Duong et al. (2015); Yang and Hospedales (2017): each task has
its own model with its own parameters and the distance between the parameters
of the networks is regularised to be similar Ruder et al. (2018).

  * •

Hierarchical sharing Søgaard and Goldberg (2016): tasks that are considered
more basic will be at lower layers when used as auxiliary tasks. (Hierarchy of
tasks where in the bottom will be basic tasks as auxiliary tasks and later
more fine-tuning tasks). This is similar to taking LM as a pre-trained model
and keeps changing its parameters when fine-tuning specific tasks.

In MTL, each task is associated with a dataset that can be shared between, or
different from, other tasks. In the latter case, the model alternates between
the tasks, during training, using various methods, including randomly
selecting a task and randomly sampling a batch from its corresponding dataset
at each training epoch Søgaard and Goldberg (2016), or iterating over all the
tasks and processing their full training datasets in each epoch Hashimoto et
al. (2017).

##  Appendix D Topological Sort

Topological sort Tarjan (1976) is a standard algorithm for linear ordering of
the vertices of a directed graph. The sort produces an ordering $O$ of the
vertices such that for every directed edge u → v from vertex u to vertex v, u
comes before v in the ordering $O$. We use the depth-first search based
algorithm which loops through each node of the graph, in an arbitrary order.
The algorithm visits each node n and pretends it to the output ordering $O$
only after recursively calling the topological sort on all descendants of n in
the graph. The algorithm terminates when it hits a node that has been visited
or has no outgoing edges (i.e. a leaf node). Hence, we are guaranteed that all
nodes that depend on n are already in the output ordering $O$ when the
algorithm adds node n to $O$.

We use topological sort to find the correct ordering $O$ of the sentences in a
document. The sentences can represent the nodes of a directed graph and the
directed edges are represented by the ordering between the two sentences. The
direction of the edges are the constraints predicted by the classifier. For
example, if the classifier predicts the constraint that sentence $s_{1}$
precedes $s_{2}$, then the edge $s_{1}\rightarrow s_{2}$ would be from the
node of $s_{1}$ to $s_{2}$.

Topological algorithm is in Algo 1.

Algorithm 1 Topological Sort Algorithm

Input: A digraph G with n vertices

Output: A topological ordering v1,v2…vn of G.

L $\leftarrow$ Empty list that will contain the sorted nodes

S $\leftarrow$ Set of all nodes with no incoming edge

while S is not empty do

remove a node n from S

add n to L

for each node m with an edge e from n to m do

remove edge e from the graph

if m has no other incoming edges then

insert m into S

end if

end for

end while

if graph has edges then

return error (graph has at least one cycle)

else

return L (a topologically sorted order)

end if