# Topic-driven Distant Supervision Framework for Macro-level
Discourse Parsing

Feng Jiang1, Longwang He2, Peifeng Li2, Qiaoming Zhu2, Haizhou Li1
1 School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
2 School of Computer Science and Technology, Soochow University, Suzhou, China
{jeffreyjiang<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Discourse parsing, the task of analyzing the internal rhetorical structure of
texts, is a challenging problem in natural language processing. Despite the
recent advances in neural models, the lack of large-scale, high-quality
corpora for training remains a major obstacle. Recent studies have attempted
to overcome this limitation by using distant supervision, which utilizes
results from other NLP tasks (e.g., sentiment polarity, attention matrix, and
segmentation probability) to parse discourse trees. However, these methods do
not take into account the differences between in-domain and out-of-domain
tasks, resulting in lower performance and inability to leverage the high-
quality in-domain data for further improvement. To address these issues, we
propose a distant supervision framework that leverages the relations between
topic structure and rhetorical structure. Specifically, we propose two
distantly supervised methods, based on transfer learning and the teacher-
student model, that narrow the gap between in-domain and out-of-domain tasks
through label mapping and oracle annotation. Experimental results on the MCDTB
and RST-DT datasets show that our methods achieve the best performance in both
distant-supervised and supervised scenarios.

##  1 Introduction

Every discourse unit (e.g., a clause, sentence, or paragraph) is semantically
closely connected in a coherent document. Discourse parsing is discovering the
internal structure of the whole document formed by these discourse units,
which is beneficial to many NLP applications, such as automatic summarization
Cohan and Goharian (2018), reading comprehension Mihaylov and Frank (2019),
and machine translation Tan et al. (2022).

As one of the most influential theories of discourse analysis, Rhetorical
Structure Theory (RST) Mann and Thompson (1987) represents a document as a
hierarchical Discourse Tree (DT). In general, a discourse tree can be split
into micro and macro levels Van Dijk and Kintsch (1983). Compared with the
micro-level structure focusing on the relationship between clauses and
sentences, the macro-level structure focuses on that between paragraphs or
chapters to understand the full document at a higher level, which is more
important for downstream tasks Kobayashi et al. (2021).

Since the existing manual annotated corpora Carlson et al. (2003); Subba and
Di Eugenio (2009); Jiang et al. (2018) only contain several hundred documents,
their small size constraints further improve the performance of the supervised
deep neural network Zhang et al. (2021); Jiang et al. (2021); Yu et al.
(2022), especially at the macro level. Due to the large annotation granularity
and the complex annotation object, it is time-consuming and expensive to
annotate a large-scale, high-quality discourse corpus manually.

Therefore, mainstream research attempts to use other NLP tasks to distantly
supervise parsing discourse trees with fewer domain-dependent data. Earlier
work used sentiment analysis Huber and Carenini (2019, 2020) and summarization
Xiao et al. (2021) tasks to build discourse trees with distant supervision.
Huber et al. (2022) proposed a topic-driven distantly supervised method,
achieving better performance. They trained a topic segmentation model and
converted the results into discourse trees from top to down according to the
order of topic segmentation probability.

Figure 1: The topic-driven distant supervision by result converting. The
models and corpora are inconsistent between in- and out-of-domain tasks.
Appendix A shows the details process of the result converting.

However, they do not take into account the inconsistency of the model and
corpus between in- and out-of-domain, as shown in Figure 1. It brings two
challenges to break the bottleneck of discourse parsing due to the different
learning objectives and annotation forms between in- and out-of-domain tasks.
The first challenge is that there is still a large gap between the performance
of distant supervised and supervised methods. The second challenge is that the
distant supervised models cannot leverage in-domain high-quality annotations
for further improvement.

To solve the above issues, we propose a topic-driven distant supervision
framework, as shown in Figure 2. Based on the previous work Huber et al.
(2022), we further propose two novel distant supervised methods in the
framework to narrow the distance between in-domain and out-of-domain tasks,
which utilizes two internal relations between topic structure and rhetorical
structure Jiang et al. (2021): 1) Locally, if two adjacent discourse units
have a rhetorical relation, they are likely to belong to the same topic. 2)
Globally, the topic structure is the skeleton of the rhetorical structure
tree, and each topic contains a discourse sub-tree.

Unlike the result converting method Huber et al. (2022), we first propose a
transfer learning method following the first relation. It uses label mapping
to keep the consistency of the model. Furthermore, we propose a teacher-
student model following the second relation. It uses the teacher model to
construct the silver rhetorical corpus by oracle annotation to ensure the
consistency of the corpus. Moreover, since our proposed methods introduce
models in the in-domain task, we can use high-quality in-domain data for
retraining the model to make further improvements. Experimental results on the
Chinese MCDTB and English RST-DT corpora show that our method achieves the
best performance in both distant and supervised scenarios, demonstrating the
effectiveness of our proposed framework.

##  2 Related Work

###  2.1 Topic Segmentation

Topic segmentation Hearst (1997) aims to mine topic maintenance and shift in
text, and is generally formalized as determining whether each part is the
boundary of a topic given a text sequence. With the large-scale topic of
corpora construction (e.g., WIKI727K Koshorek et al. (2018)), supervised
methods based on deep learning, especially based on the pre-trained models,
are more popular.

One research line used sequential labeling to predict topic boundaries with
different double-layer neural models, such as CNN+LSTM Badjatiya et al.
(2018), LSTM+LSTM Koshorek et al. (2018) and transformer+transformer Glavaš
and Somasundaran (2020). Thanks to the pre-trained language model, Lukasik et
al. (2020) proposed three models using BERT as sentence encoder (i.e., cross
BERT, BERT+bi-LSTM and hierarchy BERT) to improve the performance of topic
segmentation, while Jiang et al. (2021) provided a TM-BERT as a local model
with slide window to predict topic boundaries.

Another research line on topic segmentation used point network to predict
topic boundaries. For example, Li et al. (2018) proposed a segbot model, which
encodes text through the gated recurrent unit (GRU) module and used a pointer
network to obtain topic segmentation points. Xing et al. (2020) combined
sequential labeling of topic segmentation with local coherence modeling to
achieve better performance.

Figure 2: The Topic-driven distant supervision framework for discourse
parsing. It contains three methods according to the distance between in- and
out-of-domain tasks: (a) Result Converting, (b) Transfer Learning and (c)
Teacher-Student Model.

###  2.2 Distantly Supervised Discourse Parser

Compared with the flat topic structure, the hierarchy rhetorical structure is
more complex. Due to the lack of large-scale manually annotated corpora,
existing studies attempt to construct discourse trees that are distantly
supervised by other tasks.

Huber and Carenini (2019, 2020) used distant supervision to generate discourse
trees from sentiment analysis. They leveraged the relation between the
sentiment polarity of children and parents through multiple-case learning to
construct discourse rhetorical structure trees. Xiao et al. (2021) constructed
discourse trees that are distantly supervised by summarization. They obtained
the association between each Elementary Discourse Unit (EDU) by attention
matrix in the transformer-based summary model and parsed the discourse tree
through the CYK and CLE algorithms. Huber et al. (2022) built discourse trees
using distant supervision based on topic segmentation. They greedily
constructed a discourse tree from top to down by order of topic segmentation
probabilities.

##  3 Topic-driven Distant Supervision Framework for Macro-level Discourse
Parsing

We propose a topic-driven distant supervision framework for macro-level
discourse parsing based on the internal relations between topic structure and
rhetorical structure, as shown in Figure 2. It includes three methods: result
converting, transfer learning, and teacher-student model.

The result converting method (Figure 2 (a)) was proposed by Huber et al.
(2022). Although this is a successful attempt at topic-driven distant
supervision for discourse parsing, it does not consider the two internal
relations between topic structure and rhetorical structure.

Following the first internal relation, we propose a distantly supervised
method based on transfer learning, as shown in Figure 2 (b). To ensure the
consistency of the learning objective, it unifies the rhetorical structure and
topic structure prediction of the two adjacent discourse units into discourse
coherence prediction. Thus, the topic segmentation model is transformed into a
rhetorical structure tree construction model, which can be used to construct
intra-domain discourse trees via the shift-reduce algorithm.

Furthermore, we propose a distantly supervised method based on the teacher-
student model following the second internal relation, as shown in Figure 2
(c). The teacher model (topic segmentation model) first uses the oracle
annotation to construct a silver rhetorical structure corpus by the golden
topic structure. Then, the student model (rhetorical structure tree
construction model) can learn from it to build the discourse tree, ensuring
the consistency of the annotated information between in- and out-of-domain
tasks.

###  3.1 Distantly Supervised Discourse Parser on Transfer Learning

To address the inconsistency between in- and out-of-domain learning
objectives, we propose a distantly supervised discourse parser based on
transfer learning. Instead of converting results, it converts models from out-
of-domain to in-domain using the discourse coherence consistency of labels
between the topic segmentation model and the rhetorical tree construction
model, as shown in Figure 3.

Figure 3: The architecture of distantly supervised discourse parser based on
transfer learning.

Figure 4: The oracle annotation in teacher model. The EDUs in the same topic
have the same color. Seg Prob. indicates the segmentation probability of each
EDU predicted by the topic segmentation model, Golden Prob. indicates the
topic boundary probability of each EDU (last one ignored), and Final Prob.
indicates the split probability in the oracle annotation to build the
discourse tree from top to down.

In the out-of-domain task, we adopt the sequential labeling model Jiang et al.
(2021), which uses a local TM-BERT model to segment topics through sliding
windows. For each discourse unit, the model needs to predict whether it is the
boundary of the topic according to the context. The predicted results are
labeled as combine or split.

In the in-domain task, we adopt the transition-based parser Jiang et al.
(2021), which views the discourse tree construction into a sequence of actions
containing the shift and reduce. According to the first relation, we map the
labels of the topic segmentation model to the transition-based parser through
the coherence between two adjacent discourse units to maintain the consistency
of the learning objective, as shown in Table 1.

Learning Objective | Topic Model | Rhetorical Model | Label
---|---|---|---
coherent | combine | reduce | 0
incoherent | split | shift | 1

Table 1: The label mapping in transfer learning method.

Figure 5: The architecture of the teacher-student model.

###  3.2 Distantly Supervised Discourse Parser on Teacher-Student Model

To address the inconsistency of the annotation form, we further propose a
distantly supervised discourse parser based on the teacher-student model
according to the second relation, as shown in Figure 5.

In the out-of-domain task, we first use a topic segmentation model as the
teacher model trained on the topic structure corpus. We then propose the
oracle annotation to build rhetorical trees with golden topic boundaries for
obtaining a silver rhetorical structure corpus.

In the in-domain task, we bring this silver rhetorical structure corpus and
propose a bi-directional pointer network BLINK as the student model to learn
rhetorical structures from it. Therefore, we unify the in-domain and out-of-
domain annotation forms through the oracle-annotated silver rhetorical
structure corpus and build the exclusive student and teacher model in the in-
and out-of-domain tasks, narrowing the distance between them.

####  3.2.1 Teacher Model

The result converting method Huber et al. (2022) offers the possibility of
using the topic segmentation model to construct rhetorical structure trees but
its improvement is limited, while Jiang et al. (2021) used golden topic
structures to assist discourse parsing and achieved a higher accuracy (about
83%). Inspired by the above two methods, we propose an oracle annotation
method to build a silver rhetorical structure corpus with higher quality based
on the golden topic structure, as shown in Figure 4.

First, we use a topic segmentation model111Here, we use TM-BERT Jiang et al.
(2021). Although we have tried other models (e.g., BERT+Bi LSTM and Hier.
BERT), TM-BERT achieves the highest performance. to predict the probability of
each topic segment point, following previous work Huber et al. (2022). Then,
different from it, we use the golden topic boundary as the constraint
condition according to the second relation, greedily building the silver
discourse rhetorical structure tree from top to down by the final probability
(Final Prob.). It can ensure that the constructed discourse rhetorical tree is
better. For example, EDU6 in Figure 4 should be the first split point with the
highest segmentation probability (Seg Porb.) to generate a wrong tree if we do
not consider the golden topic boundary, while the silver rhetorical discourse
tree constructed by oracle annotation is better due to using the golden
probability (Golden Prob.).

We conduct a ten-fold cross-validation of the out-of-domain topic structure
corpus to get the silver discourse trees by the teacher model with the oracle
annotation. In each fold, we use nine parts as training sets and the rest one
part as the test set. More details about the silver rhetorical corpus are
shown in Appendix B.

####  3.2.2 Student Model

Inspired by previous work Kobayashi et al. (2019), we propose a bi-directional
pointer network (BLINK) as the student model. The model consists of two
popular pointer networks: a top-down split network (PT (Down)) and a bottom-up
merge network (PT (Up)). When building a discourse rhetorical tree, the final
operation of each step is determined by the maximum probability of the
prediction of two networks, as shown in Figure 6.

Figure 6: The architecture of student model (BLINK).

PT (Down) and PT (Up) have the same architecture. In the encoder, we first use
the pre-trained model XLNet to encode all paragraphs of the document. Then, we
use XLNet to obtain the vector representation of each word in the input
$W=\\{w_{1},w_{2},...,w_{m}\\}$, where $m$ represents the number of words
input in the document. After that, we use the Bi-GRU module to encode $W$ to
obtain the overall semantic representation of the document
$E=\\{e_{1},e_{2},...,e_{m}\\}$, as shown in Eq. 1. Then, at each step $t$, we
obtain the vector of each <SEP> token as the representation of paragraphs
$P=\\{p_{<t,1>},p_{<t,2>},...,p_{<t,n>}\\}$, where $n$ is the number of
paragraphs included in a document.

$E,h_{f}=f_{Bi-GRU}(W,h_{0})$ (1)

At the decoding step $t$, we feed the vector of the last paragraph
($p_{<t,l>}$) and the hidden layer vector $h_{t-1}$ of the decoder in the
previous time step into the decoder (GRU) to obtain the decoding
representation ($d_{t}$) of the current discourse units sequence, as shown in
Eq. 2.

$d_{t},h_{t}=f_{GRU}(p_{<t,l>},h_{t-1})$ (2)

Finally, we calculate the attention score ($score$) between $d_{t}$ and each
paragraph through dot product ($\delta$) to obtain the probability
distribution of the split point or merge point at the current time step (t),
as shown in Eqs. 3, 4 and 5.

$score^{m}_{<t,i>}=\delta(d_{t}^{m},p_{<t,i>}^{m})\quad m\in{c,s}$ (3)
$C_{p}=argmax_{p}(Softmax(score^{c}_{t}))$ (4)
$S_{p}=argmax_{p}(Softmax(score^{s}_{t}))$ (5)

where $C_{p}=\\{c_{1},c_{2},...,c_{n}\\}$ represents the probability
distribution of each paragraph as the combination point,
$S_{p}=\\{s_{1},s_{2},...,s_{n}$} indicates the probability distribution of
each paragraph as the split point. We select the highest probability value
from Cp and Sp as the final action. For example, as shown in Figure 6, at this
step, the BLINK model finally selects the maximum probability value ($C_{3}$),
which means that paragraph 3 ($P_{3}$) and paragraph 4 ($P_{4}$) should be
combined.

##  4 Experimentation

###  4.1 Datasets and Matrics

We first verify the effectiveness of the proposed method on Chinese MCDTB,
which contains 720 documents annotated with macro discourse rhetorical
structure. Following previous work Jiang et al. (2021), we split the dataset
into a train set (80%) and a testing set (20%).

To obtain the out-of-domain topic structure corpus, we collected 14393 Xinhua
news documents from the Gigaword
corpus222https://catalog.ldc.upenn.edu/LDC2009T2. Each document has
subheadings as topic boundaries. Then we use oracle annotation on it to create
the silver rhetorical structure corpus (MCDTB_dist), as mentioned in Section
3.2.1. It is worth noting that our methods only use the train data from the
MCDTB_dist in the distantly supervised scenario, while our methods use both
the train data of the in-domain (MCDTB) and out-of-domain corpus (MCDTB_dist)
in the supervised scenario.

The evaluation method consists of previous work Morey et al. (2017); Jiang et
al. (2021), which is a more rigorous approach by evaluating the span accuracy.
The details of the experimental setup are shown in Appendix C.

###  4.2 Baselines

Distantly Supervised Method.

Dist (Paragraph) method and Dist (Topic) method Huber et al. (2022). Since
there are no existing distantly supervised methods in Chinese, we choose these
two English distant supervised methods based on result converting. The former
uses the paragraph boundaries in the in-domain corpus as the learning
objective, while the latter uses the topic boundaries in the out-of-domain
corpus as the learning objective. We reproduce them with TM-BERT in Chinese
for a fair comparison.

Supervised Model.

BERT method Devlin et al. (2019). Bert is a popular model in various NLP
tasks, and we take it as the simple classification local model in the parser.

PDParser (w/o TS) model and PDParser (w/ auto TS) model Jiang et al. (2021).
They are two SOTA models in Chinese discourse parsing, which are based on a
triple semantic matching BERT model (TM-BERT). Their difference is that
PDParser (w/ auto TS) model has the predicted topic boundaries to help build
discourse trees while PDParser (w/o TS) did not have that.

###  4.3 Results

The experimental results are shown in Table 2. In distant supervised methods,
our transfer learning model and the teacher-student model achieve 56.41% and
61.51%, which are 1.08% and 6.18% higher than the best baseline Dist (Topic).
Moreover, the gradual increase of performance (55.33/56.41/61.51) among the
results converting, transfer learning, and the teacher-student model also
proves our proposed methods can narrow the distance between in- and out-of-
domain tasks.

Scenario | Method | Span
---|---|---
Distantly Supervised | Dist (Paragraph) | 50.23
Distantly Supervised | Dist (Topic) | 55.33
Distantly Supervised | Transfer Learning (ours) | 56.41
Distantly Supervised | Teacher-Student (ours) | 61.51
Supervised | BERT | 57.19
Supervised | PDParser (w/o TS) | 63.06
Supervised | PDParser (w/ auto TS) | 66.31
Supervised | Transfer Learning (ours) | 66.15
Supervised | Teacher-Student (ours) | 68.01

Table 2: The performance on MCDTB.

In addition, in supervised methods, our transfer learning model and teacher-
student model, which re-train in- and out-of-domain training data, have
achieved 66.15% and 68.01% performance, respectively. They are 3.09% and 4.95%
higher than the baseline model PDParser (w/o TS), and the teacher-student
model even gets 1.7% higher than the SOTA baseline PDParser (w/ auto TS). It
shows that the proposed method can fully exploit both in- and out-of-domain
annotations, breaking the bottleneck of discourse parsing.

##  5 Analysis

###  5.1 Ablation Study

We perform an ablation study of our proposed teacher-student model to
demonstrate its effectiveness, as shown in Table 3. The first three lines are
teacher-student models with different student models. The fourth and fifth
lines are two models (Transfer Learning and Dist(Topic)) both using TM-BERT as
the local model and topic boundary as the learning objective. The last two
lines are two supervised baselines trained only on in-domain data.

Method | Model | Dist. | Sup.
---|---|---|---
Teacher-Student | BLINK | 61.51 | 68.01
Teacher-Student | PT (up) | 57.96 | 66.00
Teacher-Student | PT (down) | 58.89 | 64.14
Transfer Learning | TM-BERT + Topic | 56.41 | -
Dist (Topic) | TM-BERT + Topic | 55.33 | -
- | BLINK | - | 63.37
PDParser (w/o TS) | TM-BERT | - | 63.06

Table 3: The ablation study of our proposed methods.

First, the significant improvement between our BLINK and the two single-
direction models in both distantly supervised and supervised scenarios shows
the effectiveness of our proposed model as a student model. Since it can
select the most reliable result from different direction models to build a
better discourse structure tree, BLINK achieves the best performance
(61.51/68.01) both in distantly supervised and supervised scenarios.

Moreover, whether in supervised or distantly supervised scenarios, even the
single-direction models (PT(up) and PT(down)) can outperform the baseline
models, proving our proposed framework’s effectiveness.

Lastly, since our teacher-student model can leverage both the in- and out-of-
domain data to do a re-train, it gets further improvement and is 4.64% higher
than BLINK that only trains on in-domain data. It demonstrates that our
created out-of-domain silver rhetorical structure corpus is beneficial for re-
training the supervised method.

###  5.2 The Effect of Re-training in Different Layers of the Discourse Tree

Since our model unifies the learning objective and annotation form, and
further improves by re-training in supervised scenarios using the silver
corpus, we further analyze the effect of re-training in different layers of
the discourse tree, as shown in Figure 7.

Figure 7: The performance of various models in different layers.

First of all, the distant supervised models based on Transfer Learning (Tran)
and Teacher-Student (Teac) are comparable to the two supervised learning
models (BERT and PDParser (w/o TS) (PDPa)) on the bottom two layers and the
middle layers, while it is slightly weaker than the supervised learning model
on the top two layers. For example, compared with BERT and PDPa, the Teac
model decreases by 1.05% and 4.90%, respectively. The main reason is that
topic structure only guarantees the correctness of middle-level boundaries of
discourse rhetorical structure, but not that of boundaries higher than topics.

Secondly, the two-stage re-training model (Tra2 and Tea2) can fully use in-
domain high-quality annotation information to make up for this defect,
achieving better performance. The re-training model (Tra2) based on transfer
learning makes further improvement in the middle layer and the top two layers,
with an increase of 13.64% and 14.29%, respectively, while the teacher-student
model (Tea2) makes further improvement in the middle layer and the bottom two
layers, with an increase of 7.14% and 7.79% respectively.

###  5.3 The Effect of Out-of-Domain Corpus in Supervision

Since our proposed methods also gain a significant improvement with the out-
of-domain corpus under the supervised scenario, we further analyze the effect
of out-of-domain corpus on constructing structure trees of different length
documents, as shown in Figure 8.

The transfer learning model achieves 81.38%, 67.99%, and 56.08% in documents
with 2-10 paragraphs. Compared with the supervised baseline model (PDParser
(w/o TS)), it has improved significantly in shorter documents with 2-4
paragraphs, reaching 5.52%. We believe that the transfer learning model learns
the topic structure better through label mapping due to the relatively simple
topic structure of short documents, outperforming the baseline model on these
documents.

Figure 8: The performance on different length documents without in-domain
topic information.

In addition, the teacher-student model reaches 69.67%, 68.09%, and 52.99%,
respectively, in documents with more than five paragraphs and increases
significantly in 5-10 paragraph documents by 5.28% and 14.30% than the
baseline model. Moreover, our teacher-student model is more robust: the
PDParser (w/o TS) model whose performance decreases rapidly (22.07%) with the
increase of the number of paragraphs, while that of our teacher-student model
only decreases by 7.42% (from 74.51% to 68.09%).

One reason for this significant improvement is the large-scale silver
rhetorical structure corpus (MCDTB_dist) oracle annotated by golden topic
structure can better cover complex discourse rhetorical structures. In the
MCDTB corpus, there are 27-37 types of discourse rhetorical trees annotated in
paragraphs 6-10, which do not increase with the number of paragraphs. However,
in MCDTB_dist corpus, the types of discourse rhetorical structure trees in
6-10 paragraphs have increased from 35 to 437, covering complex discourse
rhetorical structure trees better. More details are shown in Figure 10 in
Appendix D.

###  5.4 Performance on English RST-DT

We also conduct experiments on RST-DT to demonstrate the generalization of our
framework. Following previous works Sporleder and Lascarides (2004); Jiang et
al. (2021); Huber et al. (2022), we prune and modify the original discourse
tree in RST-DT to the macro level to evaluate discourse parsing at the macro
level.

Similar to previous work Huber et al. (2022), we select 5500 documents from
the WIKI727K as the topic structure corpus and use the teacher model to
generate the silver rhetorical structure corpus (WIKI_dist) by oracle
annotation. We then divide it into 5000 documents as the training set and 500
documents as the validation dataset for the student model. To focus more on
the macro-level structure, we use first- and second-level section names as
topic boundaries and lower-level section names as paragraph boundaries.

For the components of the teacher-student model, we use the TM-BERT model
Jiang et al. (2021) as the teacher model and the latest SOTA model Kobayashi
et al. (2022) as the student model, which is a top-down discourse parser based
on DeBERTa He et al. (2020) 333All hyper-parameters are default in the
published paper..

####  5.4.1 Baselines

Distantly Supervised Methods.

Parsersenti. Huber and Carenini (2020) and Parsersumm. Xiao et al. (2021) are
two distantly supervised methods using document-level sentiment information
and the attention matrix of the summarization model to build the discourse
tree. TSwiki Huber et al. (2022) is the SOTA distantly supervised method
parsing discourse rhetorical trees by converting results from topic
segmentation.

Supervised Methods.

SL04 Sporleder and Lascarides (2004) is the first greedy bottom-up method to
build macro-level discourse trees on the RST-DT. WL17 Wang et al. (2017) is a
discourse parser based on the traditional SVM model and builds the discourse
tree with the shift-reduce algorithm. PDParser (w/ auto TS) Jiang et al.
(2021) is a discourse parser using the synthetic topic structure to build the
discourse tree. SpanBERT Guz and Carenini (2020) is one SOTA method based on
the pre-trained language model (SpanBERT). It also uses the shift-reduce
algorithm to build the discourse tree. DeBERTa Kobayashi et al. (2022) is the
latest SOTA model, which uses the DeBERTa as the local model to build the
discourse tree from top to down.

####  5.4.2 Results

Table 4 shows the performance of our proposed methods and baselines on English
RST-DT. Similar to that in Chinese MCDTB, our proposed teacher-student model
achieves the best performance (44.42%) among a variety of distant supervised
models. Moreover, by unifying the in- and out-of-domain annotation form and
learning objective, the teacher-student model with oracle annotation also
achieves the best performance among supervised models, especially 3.37% higher
than the latest SOTA model (DeBERTa). It demonstrates the effectiveness of our
proposed method.

Scenario | Method | Span
---|---|---
Distantly Supervised | Parsersenti. | 31.62
Distantly Supervised | Parsersumm. | 32.09
Distantly Supervised | TSWiki | 41.90
Distantly Supervised | Teacher-Student (ours) | 44.42
Supervised | SL04 | 34.29
Supervised | WL17 | 37.40
Supervised | PDParser (w/ auto TS) | 40.52
Supervised | SpanBERT | 52.75
Supervised | DeBERTa | 54.81
Supervised | Teacher-Student (ours) | 58.18

Table 4: The performance on RST-DT.

##  6 Conclusion

In this paper, we propose a topic-driven distant supervision framework for
macro-level discourse parsing by exploring the relation between topic
structure and rhetorical structure444https://github.com/fjiangAI/TDSF_DP. In
the framework, we propose two novel methods based on transfer learning and
teacher-student models that narrow the distance between in- and out-of-domain
tasks from the consistency of the model and the corpus and can fully leverage
both in- and out-of-domain samples for retraining. Experimental results show
that the proposed method achieves the best performance in both Chinese MCDTB
and English RST-DT datasets, demonstrating the effectiveness of our framework.
In future research, we will jointly learn the rhetorical and topic structure
and analyze the discourse structure of the text more comprehensively.

## Limitations

In this paper, we are concerned about the completeness of the silver
rhetorical structure corpus we constructed. Despite being annotated with both
topic and rhetorical structure, the MCDTB_dist and WIKI_dist corpus is not
entirely correct, as its rhetorical structure was constructed through oracle
annotation. We aim to improve its quality and incorporate human input in
future work. Furthermore, we plan to expand its unannotated attributes, such
as nuclearity and the rhetorical relationship between discourse units, to
better represent the discourse structure of the text.

## References

  * Badjatiya et al. (2018) Pinkesh Badjatiya, Litton J Kurisinkel, Manish Gupta, and Vasudeva Varma. 2018.  Attention-based neural text segmentation.  In _European Conference on Information Retrieval (ECIR)_ , pages 180–193.
  * Carlson et al. (2003) Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003.  Building a discourse-tagged corpus in the framework of rhetorical structure theory.  _Current and New Directions in Discourse and Dialogue_ , pages 85–112.
  * Cohan and Goharian (2018) Arman Cohan and Nazli Goharian. 2018.  Scientific document summarization via citation contextualization and scientific discourse.  _International Journal on Digital Libraries_ , 19(2):287–303.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Glavaš and Somasundaran (2020) Goran Glavaš and Swapna Somasundaran. 2020.  Two-level transformer and auxiliary coherence modeling for improved text segmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , pages 7797–7804.
  * Guz and Carenini (2020) Grigorii Guz and Giuseppe Carenini. 2020.  Coreference for discourse parsing: A neural approach.  In _Proceedings of the First Workshop on Computational Approaches to Discourse_ , pages 160–167, Online. Association for Computational Linguistics.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_.
  * Hearst (1997) Marti A. Hearst. 1997.  Text tiling: Segmenting text into multi-paragraph subtopic passages.  _Computational Linguistics_ , 23(1):33–64.
  * Huber and Carenini (2019) Patrick Huber and Giuseppe Carenini. 2019.  Predicting discourse structure using distant supervision from sentiment.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2306–2316, Hong Kong, China. Association for Computational Linguistics.
  * Huber and Carenini (2020) Patrick Huber and Giuseppe Carenini. 2020.  MEGA RST discourse treebanks with structure and nuclearity from scalable distant sentiment supervision.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7442–7457, Online. Association for Computational Linguistics.
  * Huber et al. (2022) Patrick Huber, Linzi Xing, and Giuseppe Carenini. 2022.  Predicting above-sentence discourse structure using distant supervision from topic segmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_.
  * Jiang et al. (2021) Feng Jiang, Yaxin Fan, Xiaomin Chu, Peifeng Li, Qiaoming Zhu, and Fang Kong. 2021\.  Hierarchical macro discourse parsing based on topic segmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , pages 13152–13160.
  * Jiang et al. (2018) Feng Jiang, Sheng Xu, Xiaomin Chu, Peifeng Li, Qiaoming Zhu, and Guodong Zhou. 2018\.  MCDTB: A macro-level Chinese discourse TreeBank.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3493–3504, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Kobayashi et al. (2021) Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2021.  Improving neural RST parsing model with silver agreement subtrees.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1600–1612, Online. Association for Computational Linguistics.
  * Kobayashi et al. (2022) Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2022.  A simple and strong baseline for end-to-end neural rst-style discourse parsing.  _arXiv preprint arXiv:2210.08355_.
  * Kobayashi et al. (2019) Naoki Kobayashi, Tsutomu Hirao, Kengo Nakamura, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2019.  Split or merge: Which is better for unsupervised RST parsing? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5797–5802, Hong Kong, China. Association for Computational Linguistics.
  * Koshorek et al. (2018) Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018.  Text segmentation as a supervised learning task.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 469–473, New Orleans, Louisiana. Association for Computational Linguistics.
  * Li et al. (2018) Jing Li, Aixin Sun, and Shafiq Joty. 2018.  Segbot: A generic neural text segmentation model with pointer network.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI)_ , pages 4166–4172.
  * Lukasik et al. (2020) Michal Lukasik, Boris Dadachev, Kishore Papineni, and Gonçalo Simões. 2020\.  Text segmentation by cross segment attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4707–4716, Online. Association for Computational Linguistics.
  * Mann and Thompson (1987) William C Mann and Sandra A Thompson. 1987.  _Rhetorical structure theory: A theory of text organization_.  University of Southern California, Information Sciences Institute.
  * Mihaylov and Frank (2019) Todor Mihaylov and Anette Frank. 2019.  Discourse-aware semantic self-attention for narrative reading comprehension.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2541–2552, Hong Kong, China. Association for Computational Linguistics.
  * Morey et al. (2017) Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017.  How much progress have we made on RST discourse parsing? a replication study of recent results on the RST-DT.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1319–1324, Copenhagen, Denmark. Association for Computational Linguistics.
  * Sporleder and Lascarides (2004) Caroline Sporleder and Alex Lascarides. 2004.  Combining hierarchical clustering and machine learning to predict high-level discourse structure.  In _COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics_ , pages 43–49, Geneva, Switzerland. COLING.
  * Subba and Di Eugenio (2009) Rajen Subba and Barbara Di Eugenio. 2009.  An effective discourse parser that uses rich linguistic information.  In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 566–574, Boulder, Colorado. Association for Computational Linguistics.
  * Tan et al. (2022) Xin Tan, Longyin Zhang, Fang Kong, and Guodong Zhou. 2022.  Towards discourse-aware document-level neural machine translation.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , pages 4383–4389.  Main Track.
  * Van Dijk and Kintsch (1983) Teun A Van Dijk and Walter Kintsch. 1983.  _Strategies of discourse comprehension_.  Acadamic Press.
  * Wang et al. (2017) Yizhong Wang, Sujian Li, and Houfeng Wang. 2017.  A two-stage parsing method for text-level discourse analysis.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 184–188, Vancouver, Canada. Association for Computational Linguistics.
  * Xiao et al. (2021) Wen Xiao, Patrick Huber, and Giuseppe Carenini. 2021.  Predicting discourse trees from transformer-based neural summarizers.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4139–4152, Online. Association for Computational Linguistics.
  * Xing et al. (2020) Linzi Xing, Brad Hackinen, Giuseppe Carenini, and Francesco Trebbi. 2020.  Improving context modeling in neural topic segmentation.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 626–636, Suzhou, China. Association for Computational Linguistics.
  * Yu et al. (2022) Nan Yu, Meishan Zhang, Guohong Fu, and Min Zhang. 2022.  RST discourse parsing with second-stage EDU-level pre-training.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4269–4280, Dublin, Ireland. Association for Computational Linguistics.
  * Zhang et al. (2021) Longyin Zhang, Fang Kong, and Guodong Zhou. 2021.  Adversarial learning for discourse rhetorical structure parsing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3946–3957, Online. Association for Computational Linguistics.

Figure 9: An example of the topic-driven distant supervision by result
converting.

##  Appendix A The Process of the Topic-driven Distant Supervision by Result
Converting

Figure 9 shows an example of topic-driven distant supervision by result
converting. The topic segmentation model could predict the sequence of EDU to
get the segmentation probability (Seg Prob.). Then the result converting
method will split the sequence according to the order of the probability of
segment points. For example, sentence 2 ($Sent_{2}$) is the highest
probability (0.7) that is split first. Then is sentence 4 ($Sent_{4}$) and
sentence 3 ($Sent_{3}$). Therefore, it uses the top-down parsing method to
convert the topic segmentation result into a rhetorical structure tree.

(a)  MCDTB_dist

(b) MDCTB

Figure 10: The main distribution of discourse tree types in MDCTB_dist and
MCDTB.

##  Appendix B The Details of Silver Rhetorical Structure Corpus

We construct our macro topic structure corpus for the out-of-domain task using
the Gigaword 555https://catalog.ldc.upenn.edu/LDC2009T2 and WIKI727K Koshorek
et al. (2018) corpora as our data sources. In Chinese, we select 14,393 news
documents with subheadings from the Xinhua News Agency news in the Gigaword
corpus and use the subheadings as topic boundaries. In English, we use 5,500
wiki documents with section names, following previous work Huber et al.
(2022), and use the first- and second-level section names as topic boundaries
and lower-level section names as paragraph boundaries.

In our transfer learning method, we train an out-of-domain topic segmentation
model using the topic structure corpus and then map the labels to convert it
into a rhetorical tree construction model.

In the teacher-student model, we use a ten-fold cross-validation method to
oracle annotate the topic structure corpus into a silver rhetorical structure
corpus (MCDTB_dist and WIKI_dist). It means that we split the dataset into 10
folds, and the silver rhetoric structure on each fold is obtained by the topic
segmentation model trained by the remaining nine datasets through the oracle
annotation method.

##  Appendix C Experimental Settings

###  C.1 MCDTB

The hyper-parameters of the topic segmentation model used are following the
previous work Jiang et al. (2021): batch-size=2, epoch=10, max-length=512, and
learning rate=1e-5. The pre-trained language model is the bert-base model
(https://huggingface.co/bert-base-chinese).

In the teacher-student model we proposed, the main hyper-parameters of the
student model (BLINK) are the following: the batch-size=2, epoch=50, the
hidden size of GRU is 64, the layer number of GRU is 4, and the learning
rate=1e-6. The pre-trained language model is the chinese-xlnet-mid model
(https://huggingface.co/hfl/chinese-xlnet-mid).

We use an NVIDIA Tesla V100 GPU with 32GB to conduct the experiment.

###  C.2 RST-DT

The hyper-parameter of the topic segmentation model used is the same as the
model in MCDTB, except that the pre-trained language model is an English bert-
base-uncased model (https://huggingface.co/bert-base-uncased).

In the teacher-student model we proposed, the main hyper-parameters of the
student model (Deberta) are the same as previous work Kobayashi et al. (2022).

We use an NVIDIA RTX 3090 GPU with 24GB to conduct the experiment.

##  Appendix D The Main Distribution of Discourse Tree Types in MCDTB_dist and
MCDTB

Figure 10 shows the main distribution of discourse tree types in MCDTB_dist
and MCDTB. In MCDTB_dist corpus, the discourse tree types increase with the
number of paragraphs when the document has less than 13 paragraphs. Utilizing
various types of discourse rhetorical structure trees can lead to a more
robust structure tree construction model and improved performance.
Additionally, even though there may be a decline in diversity in longer
documents (#paragraphs > 13), it is still significantly more than the types in
manually annotated MCDTB. For instance, documents with 25 paragraphs still
contain over 200 different types of discourse structure trees in MCDTB_dist,
while MCDTB is basically not covered that.