# Causal Intervention for Abstractive Related Work Generation

First Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain
&Second Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain

###### Abstract

Abstractive related work generation has attracted increasing attention in
generating coherent related work that better helps readers grasp the
background in the current research. However, most existing abstractive models
ignore the inherent causality of related work generation, leading to
tendentiously acquiring spurious correlations and impairing the models’
generalizability. In this study, we argue that causal intervention can address
this limitation and improve the quality and coherence of the generated related
works. To this end, we propose a novel Causal Intervention Module for Related
Work Generation (CaM) to effectively capture causalities in the generation
process and improve the quality and coherence of the generated related works.
Specifically, we first model the relations among the relevant factors in
related work generation using a causal graph. Then, to implement the causal
intervention and mitigate the negative impact of spurious correlations, we use
do-calculus to derive ordinary conditional probabilities and identify causal
effects through CaM. Finally, we subtly fuse CaM with Transformer to obtain an
end-to-end generation model. Extensive experiments on two real-world datasets
show that causal interventions in CaM can effectively promote the model to
learn causal relations and produce related work of higher quality and
coherence.

##  1 Introduction

A comprehensive related work necessarily covers abundant reference papers,
which costs authors plenty of time in reading and summarization and even
forces authors to pursue ever-updating advanced work. Fortunately, the task of
related work generation emerged and attracted increasing attention from the
community of text summarization and content analysis in recent years. Related
work generation can be considered as a variant of the multi-document
summarization task [li2022automatic-meta-review]. Distinct from multi-document
summarization, related work generation entails comparison after the
summarization of a set of references and needs to sort out the similarities
and differences between these references [agarwal-etal-2011-scisumm].

Recently, various abstractive text generation methods have been proposed to
generate related work based on the abstracts of references. For example, xing-
etal-2020-automatic used the context of citation and the abstract of each
cited paper as the input to generate related work. ge-etal-2021-baco encoded
the citation network and used it as external knowledge for generating related
work. target-aware-2022 proposed a target-aware related work generator that
captures the relations between reference papers and the target paper through a
target-centered attention mechanism. Equipped with well-designed encoding
strategies, external knowledge, or novel training techniques, these studies
have made promising progress in generating coherent related works.

However, those models tendentiously explore and exploit spurious correlations
such as high-frequency word/phrase patterns, writing habits, or presentation
skills, building superficial shortcuts between reference papers and the
related work of the target paper. Such spurious correlations may astrict or
even harm the quality of the generated related work, especially under the
distribution shift between the testing set and training set. This is
attributed that spurious correlations different from genuine causal relations
may not intrinsically contribute to the related work generation and easily
cause the robustness problem and impair the models’ generalizability.

Figure 1: An illustration of the effect difference between causality (solid
arrows) and spurious correlation (dashed arrows) in related work generation.

Figure 1 illustrates the difference between causality and spurious
correlation. Figure 1 illustrates the difference between causality and
spurious correlation. The phrases "for example" and "later" are often used to
bridge two sentences in related work. Their usage may be attributed to
writers’ presentation habits about organizing sentence orders (spurious
correlation), or the reference document relationships corresponding to the
sentences (causal correlation). Ideally, a related work generation model is
expected to learn the reference relation and distinguish it from the writing
habits. However, the generation model easily captures the superficial habitual
sentence organization instead of learning complex semantic reference
relations, especially when the habitual patterns frequently exist in the
training set. In this case, the conjunctives generated mainly based on writing
habits are very likely to be unsuitable and subsequently affect the content
generation of related work during testing when the training and testing sets
are not distributed uniformly. To address the issues, causal intervention can
effectively remove spurious correlations and focus on the causal correlations
by intervening the learning process. It not only observes the impact of the
sentence order on the conjunctive generation but probe the impact of each
possible conjunctives on the whole generation of related work, thereby
removing the spurious correlations [pearl-intervention]. Accordingly, causal
intervention serving as an effective solution allows causal relations to exert
a greater impact and instruct the model to produce the correct content.

Causal intervention avoids the constraints of conditioning by using the do-
calculus, serving as an effective solution. It not only observes the impact of
the current sentence order on the content but weighs the impact of each
possible value on the whole population, thereby removing the spurious
correlations [pearl-intervention]. While the spurious correlations are
weakened, the model is able to re-measure the importance of document relations
for content generation, and thereby better captures the causalities involved.

the learned correlation is actually spurious that that is likely to impair the
model’s generalizability during testing, and could be even more noticeable
when the training and testing sets are not distributed uniformly. Causal
intervention can effectively remove the spurious correlations by intervening
the learning process, allowing causal relations to exert a greater impact and
instruct the model to produce the correct content.

Ideally, the model should separately learn the causal effects of sentence
order and document relation in guiding the generation of "for example" and
"later", but when such identical text pattern of related work occurs
abundantly in the dataset, the model may be inclined to focus only on the
sentence order since it is easier to capture. In this case, the correlation is
actually spurious that is likely to impair the model’s generalizability during
testing, and could be even more noticeable when the training and testing sets
are not distributed uniformly. Causal intervention can effectively remove the
spurious correlations by intervening the learning process, allowing causal
relations to exert a greater impact and instruct the model to produce the
correct content.

Their usage may be attributed to writing habits that writers get used to
ignore the and or

There are two possible reasons why writers begin their sentences with these
two words. One is that the writer ignores the exact content and is accustomed
to using these words to organize the text. This is when the writer is simply
determining the words based on sentence order. Second, the author does need
these words to clarify the relations between different studies in the related
work, i.e., the author decides which words to use to bridge the sentences
based on the document relations. Ideally, the model should separately learn
the causal effects of sentence order and document relation in guiding the
generation of "for example" and "later", but when such identical text pattern
of related work occurs abundantly in the dataset, the model may be inclined to
focus only on the sentence order since it is easier to capture. In this case,
the correlation is actually spurious that is likely to impair the model’s
generalizability during testing, and could be even more noticeable when the
training and testing sets are not distributed uniformly. Causal intervention
can effectively remove the spurious correlations by intervening the learning
process, allowing causal relations to exert a greater impact and instruct the
model to produce the correct content.

To address the aforementioned gaps in existing work for related work
generation, we propose a Causal Intervention Module for Related Work
Generation (CaM). CaM can effectively remove spurious correlations by
performing the causal intervention and produce related works of high quality.
Specifically, we first model the relations over relevant factors involved in
related work generation and figure out the confounder that raises spurious
correlation (see Figure 2). Then, we implement causal intervention via the
proposed CaM which consists of three components: 1) Primitive Intervention
cuts off the path that induces spurious correlation in the causal graph by
leveraging do-calculus and backdoor criterion [pearl-intervention]; 2)
Context-aware Remapping smoothens the distribution of intervened embeddings
and injects contextual information; and 3) Optimal Intensity Learning learns
the best intensity of overall intervention by controlling the output from
different parts. Finally, we strategically fuse CaM with Transformer
[attentionallyouneed] to deliver an end-to-end causal related work generation
model. Our contributions are as follows:

  * •

To the best of our knowledge, this work is the first attempt to introduce
causality theory into related work generation and multi-document summarization
tasks.

  * •

We propose a novel Causal Intervention Module for Related Work Generation
(CaM) which implements causal intervention to mitigate the impact of spurious
correlations. CaM is subtly fused with Transformer to derive an end-to-end
causal model, enabling the propagation of intervened information.

  * •

Extensive experiments on two related work generation datasets demonstrate that
our model outperforms the state-of-the-art approaches and verifies the
effectiveness and rationality of bringing causality theory into the related
work generation task.

##  2 Problem Formulation

Given a set of reference papers $D=\\{{r}_{1},...,{r}_{|D|}\\}$, we assume the
ground truth related work $Y=(w_{1},w_{2},...,w_{M})$, where
${r}_{i}=(w^{i}_{1},w^{i}_{2},...,w^{i}_{|r_{i}|})$ denotes a single cited
paper, $w^{i}_{j}$ is the $j$-th word in ${r}_{i}$, and $w_{j}$ is the $j$-th
word in related work ${Y}$. Generally, the related work generation task can be
formulated as generating a related work section
$\hat{{Y}}=(\hat{w}_{1},\hat{w}_{2},...,\hat{w}_{\hat{M}})$ based on the
reference input $D$ and minimizing the difference between ${Y}$ and
$\hat{{Y}}$. Considering that the abstract section is usually well-drafted to
provide a concise paper summarization [hu-wan-2014-automatic], we use the
abstract section to represent each reference paper.

Figure 2: Causal graph $G$ for related work generation. By applying
$do$-calculus, path $c\to x$ is cut off and the impact of spurious correlation
$c\to x\to y$ is mitigated. Figure 3: The structure of CaM fused with the
Transformer in the decoder. CaM consists of three parts: Primitive
Intervention, Context-aware Remapping and Optimal Intensity Learning.

##  3 Methodology

We first analyze the causalities in related work generation, identify the
confounder that raises spurious correlations, and use the causal graph to
model these relations. Then, we introduce how CaM is designed to enhance the
quality of related work through causal intervention. Finally, we describe how
CaM, as an intervention module, is integrated with the Transformer to
influence the entire generation process. The overall structure of our model is
shown in Figure 3.

###  3.1 Causal Modeling for Related Work Generation

We believe that three aspects play significant roles in related work
generation for a better depiction of the relations between different
references, namely, sentence order $c$, document relation $x$ and transitional
content $y$ (illustrated in Figure 2). In many cases, sentence order is
independent of the specified content and directly establishes relations with
transitional content. For example, we tend to use "firstly" as the beginning
and "finally" as the end while composing a paragraph, regardless of what
exactly is in between. This relation corresponds to path $c\to y$, and it
should be preserved as an experience and pattern of writing. Meanwhile, there
is a lot of transitional content that portrays the relations between referred
papers based on the actual content, at this time, models need to analyze and
use these relations. The corresponding path is $x\to y$.

Though ideally, sentence order and document relation should instruct the
generation of transitional content respectively and independently, quite
often, the deep learning models are unable to trade off the influence of these
two aspects correctly and develop a preference towards sentence order. This is
attributed to the unbalanced data distribution or the fact that sentence order
information is easily accessible and learnable. In Figure 2, such relation
corresponds to $c\to x\to y$. In this case, sentence order $c$ is the
confounder that raises a spurious correlation with transitional content $y$.
Although performing well on the training set, Once the related work from the
test set focuses more on document relations, discarding the previous writing
patterns, the transitional content instructed by sentence order can be quite
unreliable. In order to mitigate the impact of this spurious correlation, we
need to cut off the path $c\to x$, enabling the model to generate transitional
content based on the correct and reliable causality of both $c\to y$ and $x\to
y$.

###  3.2 Causal Intervention Module for Related Work Generation

The proposed Causal Intervention Module for Related Work Generation (CaM)
contains three parts. Primitive Intervention performs causal intervention and
preliminarily removes the spurious correlations between sentence order and
transitional content.Context-aware Remapping captures and fuses contextual
information, facilitating the smoothing of the intervened embeddings. Optimal
Intensity Learning learns the best intensity of holistic causal intervention.
The overall structure is demonstrated in Figure 3.

####  3.2.1 Primitive Intervention

Based on the causal graph G shown in Figure 2, we first perform the following
derivation using do-calculus and backdoor criterion.

$\displaystyle p(y|do(x))$
$\displaystyle=\begin{matrix}\sum_{\mathbf{c}}p(y|do(x),\mathbf{c})p(\mathbf{c}|do(x))\end{matrix}$
(1)
$\displaystyle=\begin{matrix}\sum_{\mathbf{c}}p(y|x,\mathbf{c})p(\mathbf{c}|do(x))\end{matrix}$
$\displaystyle=\begin{matrix}\sum_{\mathbf{c}}p(y|x,\mathbf{c})p(\mathbf{c})\end{matrix}$

As a result, by taking into consideration the effect of each possible value of
sentence order $c$ on transitional content $y$, $c$ stops affecting document
relation $x$ when using $x$ to estimate $y$, which means path $c\to x$ is cut
off (see the arrow-pointed graph in Figure 2). Next, we will explain how to
estimate separately $p(y|x,\mathbf{c})$ and $p(\mathbf{c})$ using deep
learning models and finally obtain $p(y|do(x))$.

Let $E^{ori}=(e^{ori}_{1},{e}^{ori}_{2},...,{e}^{ori}_{\hat{M}})$ denote the
input embeddings corresponding to $\hat{M}$-sized related work and
${E}^{itv}=({e}^{itv}_{1},{e}^{itv}_{2},...,{e}^{itv}_{\hat{M}})$ denote the
output embeddings of Primitive Intervention. We integrate the sentence order
information into the input embeddings:

${e}^{odr(j)}_{i}=\mathrm{Linear}({e}^{ori}_{i}\oplus{o}_{j})$ (2)

${O}=\\{{o}_{j}\\}_{j=1}^{s}$ denotes the order information for each sentence,
and $s$ is the total number of sentences in the generated related work and can
be considered as a hyper-parameter. ${e}^{odr(j)}_{i}$ denotes the order-
enhanced embedding for the $i$-th word which corresponds to the $j$-th
sentence in related work. We take ${o}_{j}=(\lg{(j+1)},\cdots,\lg{(j+1)})$
with the same dimension as ${e}^{ori}$. The linear layer (i.e.,
$\mathrm{Linear}$) further projects the concatenated embedding to ${e}^{odr}$
with the same dimension as ${e}^{ori}$. Accordingly, we have the estimation of
$p(y|x,\mathbf{c}):={e}^{odr}$. Then, we use a feed-forward network and the
output subsequence $E_{sub}=({e}^{itv}_{1},...,{e}^{itv}_{i-1})$ to predict
the sentence position probability of the current decoding word:

$\displaystyle{h}_{i}=\mathrm{Softmax}(\mathrm{FFN}(\mathrm{ReLU}(\begin{matrix}\sum^{i-1}E_{sub}\end{matrix})))$
(3)

each $h_{i}^{j}\in{h}_{i}$ denotes the probability. Thus, we estimate the
sentence position probability of each decoding word $p(\mathbf{c}):={h}$.

After obtaining the estimation of $p(y|x,\mathbf{c})$ and $p(\mathbf{c})$, the
final embedding with primitive causal intervention can be achieved:

$\begin{matrix}{e}^{itv}_{i}=\sum^{s}_{j=1}{{e}^{odr(j)}_{i}\times
h^{j}_{i}},h^{j}_{i}\in{h}_{i}\end{matrix}$ (4)

where ${e}^{odr(j)}_{i}\times h^{j}_{i}$ multiplying sentence order
probability with order-enhanced embeddings is exactly
$p(y|x,\mathbf{c})p(\mathbf{c})$ in Equation 1. The summation for each
position $j$ completes the last step of Primitive Intervention. Since most
transitions are rendered by start words, our approach CaM intervenes only with
these words, that is part of ${e}^{itv}\in{E}^{itv}$ is equal to
${e}^{ori}\in{E}^{ori}$. For simplicity, we still use ${E}^{itv}$ in the
following.

####  3.2.2 Context-aware Remapping

Two problems may exist in Primitive Intervention: 1) The lack of trainable
parts may lead to the mapping spaces of the intervened embeddings and the
original ones being apart and obstructs the subsequent decoding process. 2)
Intervention on individual words may damage the context along with the order-
enhanced embedding. To solve these two problems, we propose the Context-aware
Remapping mechanism.

First, we scan ${E}^{itv}$ with a context window of fixed size $n_{w}$:

$\displaystyle{B}_{i}$
$\displaystyle=\mathrm{WIN}_{i:i+n_{w}}([{e}^{itv}_{1},{e}^{itv}_{2},...,{e}^{itv}_{\hat{M}}])$
(5)
$\displaystyle=({e}^{itv}_{i},...,{e}^{itv}_{i+n_{w}}),i=1,...,\hat{M}-n_{w}$

where $\mathrm{WIN}(\cdot)$ returns a consecutive subsequence of ${E}^{itv}$
at length $n_{w}$. Then, we follow the process of Multi-head Attention
Mechanism [attentionallyouneed] to update the embeddings in ${B}_{i}$:

$\displaystyle{B}_{i}^{rmp}$
$\displaystyle=\mathrm{MultiHead}({B}_{i},{B}_{i},{B}_{i})$ (6)
$\displaystyle=({e}^{rmp}_{i},...,{e}^{rmp}_{i+n_{w}})$

Even though all embeddings in ${B}_{i}$ are updated, we only keep the renewed
${e}^{rmp}_{i+(n_{w}/2)}\in{B}_{i}^{rmp}$ , and leave the rest unchanged.
Since $\mathrm{WIN}(\cdot)$ scans the entire sequence step by step, every
embedding will have the chance to update. The output is denoted as
${E}^{rmp}=({e}^{rmp}_{1},{e}^{rmp}_{2},...,{e}^{rmp}_{\hat{M}})$.

####  3.2.3 Optimal Intensity Learning

In many cases, there is no guarantee that causal intervention with maximum
(unaltered) intensity will necessarily improve model performance, especially
when combined with pre-trained models [nips-gpt3, lewis-etal-2020-bart], as
the intervention may conflict with the pre-training strategies. To guarantee
performance improvement, we propose Optimal Intensity Learning.

By applying Primitive Intervention and Context-aware Remapping, we have three
types of embeddings, ${E}^{ori}$,${E}^{itv}$, and ${E}^{rmp}$. To figure out
their respective importance to the final output, we derive the output
intensity corresponding to each of them as follows:

$\displaystyle g^{ori}_{i}$
$\displaystyle=\sigma({W}_{ori}\cdot{e}^{ori}_{i})$ (7) $\displaystyle
g^{itv}_{i}$ $\displaystyle=\sigma({W}_{itv}\cdot{e}^{ori}_{i})$ (8)
$\displaystyle g^{rmp}_{i}$
$\displaystyle=\sigma({W}_{rmp}\cdot{e}^{ori}_{i})$ (9) $\displaystyle
c^{ori}_{i},c^{itv}_{i},c^{rmp}_{i}$
$\displaystyle=f_{s}([g^{ori}_{i},g^{itv}_{i},g^{rmp}_{i}])$ (10)

$\sigma(\cdot)$ is the $\mathrm{sigmoid}$ function, $f_{s}(\cdot)$ is the
$\mathrm{softmax}$ function. Combining $c^{ori}_{i},c^{itv}_{i},c^{rmp}_{i}$,
we can obtain the optimal intervention intensity and the final word embedding
set ${E}^{opm}=({e}^{opm}_{1},...,{e}^{opm}_{\hat{M}})$ with causal
intervention:

${e}^{opm}_{i}=c^{ori}_{i}{e}^{ori}_{i}+c^{itv}_{i}{e}^{itv}_{i}+c^{rmp}_{i}{e}^{rmp}_{i}$
(11)

###  3.3 Fusing CaM with Transformer

To derive an end-to-end causal generation model and ensure that the intervened
information can be propagated, we choose to integrate CaM with Transformer
[attentionallyouneed].However, unlike the RNN-based models that generate words
recurrently [rnn-text-summarization], the attention mechanism computes the
embeddings of all words in parallel, while the intervention is performed on
the sentence start words.

To tackle this challenge, we perform vocabulary mapping on word embeddings
before intervention and compare the result with sentence start token
$\mathrm{[CLS]}$ to obtain ${Mask}$:

$\displaystyle I=\mathrm{argmax}[\mathrm{Linear}_{vocab}({E}^{ori})]$ (12)
$\displaystyle{Mask}=\delta(I,ID_{CLS})$ (13)

$I$ contains the vocabulary index of each word. $\delta(\cdot)$ compares the
values of the two parameters, and returns $1$ if the same, $0$ otherwise.
$Mask$ indicates whether the word is a sentence start word. Therefore,
${E}^{opm}$ can be calculated as:

${E}^{opm}={E}^{opm}\odot{Mask}+{E}^{ori}\odot(\sim{Mask})$ (14)

The $\odot$ operation multiplies each embedding with the corresponding
$\\{0,1\\}$ values, and $\sim$ denotes the inverse operation. Note that we
omit ${Mask}$ for conciseness in Section 3.2.3. ${Mask}$ helps restore the
non-sentence-start word embeddings and preserve the intervened sentence-start
ones.

As illustrated in Figure 3, we put CaM between the Transformer layers in the
decoder. The analysis of the amount and location settings will be discussed in
detail in Section 4.6. The model is trained to minimize the cross-entropy loss
between the predicted $\hat{Y}$ and the ground-truth ${Y}$, $c$ is the
vocabulary index for $w_{i}\in Y$:

$\mathcal{L}=-\begin{matrix}\sum_{i}^{\hat{M}}\log
p_{i}^{c}({\hat{Y}})\end{matrix}$ (15)

Statistic | S2ORC | Delve
---|---|---
Pairs # | 126k/5k/5k | 72k/3k/3k
source # | 5.02 | 3.69
words/sent(doc) # | 1079/45 | 626/26
words/sent(sum) # | 148/6.69 | 181/7.88
vocab size # | 377,431 | 190,381
Table 1: Statistics of the datasets Model | S2ORC | Delve
---|---|---
ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-1 | ROUGE-2 | ROUGE-L
Extractive Methods |  |  |  |  |  |
TextRank | 22.36 | 2.65 | 19.73 | 25.25 | 3.04 | 22.14
BertSumEXT | 24.62 | 3.62 | 21.88 | 28.43 | 3.98 | 24.71
MGSum-ext | 24.10 | 3.19 | 20.87 | 27.85 | 3.95 | 24.28
Abstractive Methods |  |  |  |  |  |
TransformerABS | 21.65 | 3.64 | 20.43 | 26.89 | 3.92 | 23.64
BertSumABS | 23.63 | 4.17 | 21.69 | 28.02 | 3.50 | 24.74
MGSum-abs | 23.94 | 4.58 | 21.57 | 28.13 | 4.12 | 24.95
GS | 23.92 | 4.51 | 22.05 | 28.27 | 4.36 | 25.08
RRG | 25.46 | 4.93 | 22.97 | 29.10 | 4.94 | 26.29
CaM (ours) | 26.65 | 6.40 | 24.62 | 29.31 | 7.17 | 26.61
Table 2: ROUGE scores comparison between our CaM and the baselines.

##  4 Experiments

###  4.1 Datasets

Following the settings in chen-etal-2021-capturing,target-aware-2022, we adopt
two publicly available datasets derived from the scholar corpora S2ORC [lo-
wang-2020-s2orc] and Delve [aku-delve] respectively to evaluate our proposed
method in related work generation. S2ORC consists of scientific papers from
multiple domains, and Delve focuses on the computer domain. The datasets are
summarized in Table 1, where the corresponding ratios of the
training/validation/test pairs are detailed.

###  4.2 Settings

We implement our model with PyTorch on NVIDIA 3080Ti GPU. In our model, the
maximum reference paper number is set to 5, i.e., $|D|=5$. We select the first
$440/|D|$ words in each reference paper abstract and concatenate them to
obtain the model input sequence. The total number of sentences in target
related work is set to 6, i.e., $s=6$. We use beam search for decoding, with a
beam size of 4 and a maximum decoding step of 200. When fusing CaM with
Transformer, the dimension of word embedding is set to 768, both attention
heads number and layer number is set to 12, and the intermediate size is set
to 3072. We use Stochastic Gradient Descent(SGD) as the optimizer with a
learning rate 1e-2. To ensure desirable performance and save training costs,
we utilized pretrained BERT [devlin-etal-2019-bert]. We use ROUGE-1, ROUGE-2
and ROUGE-L on F1 as the evaluation metrics [lin-2004-rouge]. Since we adopt
exactly the same datasets (including the dataset settings) as RRG [chen-
etal-2021-capturing] used, we directly use the results in the RRG paper for
baseline comparison.

###  4.3 Compared Methods

We compare our CaM with the following eight state-of-the-art baselines,
including both extractive and abstractive methods.

####  4.3.1 Extractive Methods

(1) TextRank [mihalcea-tarau-2004-textrank]: A graph-based text ranking model
that can be used in multi-document sentence extraction. (2) BertSumEXT [liu-
lapata-2019-text-bertsumext]: An extractive document summarization model that
extends BERT by inserting multiple [CLS] tokens.(3) MGSum-ext [jin-
etal-2020-mgsumext]: A multi-granularity interaction network that jointly
learns different semantic representations.

####  4.3.2 Abstractive Methods

(1) TransformerABS [attentionallyouneed]: An abstractive summarization model
based on Transformer with attention mechanism. (2) BertSumABS [liu-
lapata-2019-text-bertsumext]: An abstractive model based on BERT with a
designed two-stage fine-tuning approach. (3) MGSum-abs [jin-
etal-2020-mgsumext]: A multi-granularity interaction network that can be
utilized for abstractive document summarization.(4) GS [li-
etal-2020-leveraging-graph-gs]: An abstractive summarization model that
utilizes special graphs to encode documents to capture cross-document
relations. (5) RGG [chen-etal-2021-capturing]: An encoder-decoder model
specifically tailored for related work generation, which constructs and
refines the relation graph of reference papers.

###  4.4 Overall Performance

It can be found in Table 2 that abstractive models have attracted more
attention in recent years and usually outperform extractive ones. Though
BertSumEXT and MGSum-ext perform better than some abstractive models on the
ROUGE-1 score on S2ORC, they underperform the latest abstractive models both
on ROUGE-2 and ROUGE-L scores due to the lack of more explicit modeling of
document relations. Both GS and RRG explicitly introduce different graphs
containing relation information, and they outperform other baselines.
Particularly, RRG achieves the best performance among them with its renewal
process for the relation graph.

Our proposed CaM achieves the best performance on both datasets. Due to fusing
CaM with Transformer, its large scale ensures that our model can still
effectively capture document relations without additional modeling.
Accordingly, CaM enables the model to obviate distractions from the confounder
through causal intervention and promotes the model to learn more robust
causalities to achieve the best performance.

###  4.5 Ablation Study

Figure 4: Ablation result on S2ORC. Figure 5: Ablation result on Delve.

To analyze the contribution of the different components of CaM, we separately
control the use of Primitive Intervention (PI), Context-aware Remapping (RMP)
and Optimal Intensity Learning (OPT). Figure 4 and Figure 5 show the
performance comparison between different variants of CaM.

First, it can be observed that the basic Transformer model already guarantees
a desirable base performance. When only PI is used, the model generally shows
a slight performance drop. PI+RMP outperforms RMP, showing the necessity of
the PI and the effectiveness of RMP. PI+RMP+OPT achieves optimal results,
indicating that OPT can effectively exploit the information across different
representations.

###  4.6 Fusing Strategy Comparison

Figure 6: Performance analysis on the number of CaMs fused with Transformer.

In our setting, the base Transformer model consists of $12$ layers, so there
are multiple locations to fuse a different number of CaMs. For each scenario,
CaMs are placed evenly among the Transformer layers, and one will always be
placed at the end of the entire model. The results of all cases are shown in
Figure 6. It can be observed that the model performs best when the number of
CaM is 4 both on S2ORC and Delve. With a small number of CaMs, the model may
underperform the benchmark model and fail to achieve optimal performance due
to the lack of sufficient continuous intervention. If there are too many CaMs,
the distance between different CaMs will be too short, leaving an insufficient
learning process for the Transformer layers, and this might cause the CaMs to
bring the noise.

###  4.7 Robustness Analysis

To investigate the robustness of the learned knowledge brought by causal
intervention, we designed two experiments on CaM and the base Transformer
model (TF).

####  4.7.1 Testing with Reordered Samples

We randomly select 50 samples (15 from S2ORC and 35 from Delve) and manually
rearrange the order of the cited papers in each of them, as well as the order
of their corresponding sentences.Transitional content in related works is also
removed since the reordering damages the original logical relations.

Figure 7: Comparison between Transformer and CaM on original and reordered
samples.

It can be observed from Figure 7 that CaM has better performance regardless of
whether the samples have been reordered or not. By switching from the original
to the reordered samples, the performance of Transformer decreases on all
three metrics, but CaM only decreases on ROUGE-1 and ROUGE-2 at a much lower
rate. Particularly, compared to the Transformer, CaM makes improvement on
ROUGE-L when tested with reordered samples. The result indicates that CaM is
able to tackle the noise disturbance caused by reordering, and the generated
content maintains better coherence.

####  4.7.2 Testing with Migrated Test Set

Figure 8: The result of migrating test set from Delve to S2ORC (trained on
Delve).

We train the models on Delve and test them on S2ORC, which is a challenging
task and significant for robustness analysis. As expected, the performances of
all models drop, but we can still obtain credible conclusions. Since CaM
outperforms Transformer initially, simply comparing the ROUGE scores after
migrating the test set is not informative. To this end, we use Relative
Outperformance Rate (ROR) for evaluation:

$\mathrm{ROR}=(\mathrm{S_{CaM}}-\mathrm{S_{TF}})/\mathrm{S_{TF}}$ (16)

$\mathrm{S_{CaM}}$ and $\mathrm{S_{TF}}$ are the ROUGE scores of CaM and
Transformer, respectively. ROR computes the advantage of CaM over Transformer.

Figure 8 reports that CaM outperforms Transformer regardless of migrating from
Delve to S2ORC for testing. In addition, comparing the change of ROR, we
observe that although migration brings performance drop, CaM not only
maintains its advantage over Transformer but also enlarges it. The above two
experiments demonstrate that the CaM effectively learns causalities to improve
model robustness.

###  4.8 Causality Visualization

Figure 9: Visualization of the generating process within CaM and
Transformer(TF).

To visualize how causal intervention worked in the generation process, we
compare the related works generated by the base Transformer and CaM with a
case study (full results in Table B). Specifically, we map their cross
attention corresponding to "however" and "the" to the input content using
different color shades (see Figure 10) to explore what information of these
two words rely on. More details of the above two experiments can be found in
Appendix B.

We picked out the words that "however" and "the" focused on the most and
analyzed the implications of these words in the context of the input. The
results are shown in Figure 9. It can be found that the words highlighted by
CaM have their respective effects in the cited papers. When generating
"however", the model aggregates this information, comparing the relations
between the documents and producing the correct result. However, there is no
obvious connection between the words focused on by Transformer, hence there is
no clear decision process after combining the information, and the generated
word "the" is simply a result obtained from learned experience and preference.
Through causality visualization, it can be observed very concretely how CaM
improves model performance by conducting causal intervention.

##  5 Conclusions

In this paper, we propose a Causal Intervention Module for Related Work
Generation (CaM) to capture causalities in related work generation. We first
model the relations among the relevant factors in the generation using a
causal graph. The proposed CaM implements causal intervention and enables the
model to capture causality. We subtly fuse CaM with Transformer to obtain an
end-to-end model to integrate the intervened information throughout the
generation process. Extensive experiments show the superiority of CaM over the
latest models and demonstrate our method’s effectiveness.

## Limitations

Although extensive experiments have demonstrated that CaM can effectively
improve the performance of the base model, as mentioned above, since the
intervention occurs on the sentence start words, it is inconclusive that CaM
can bring improvement if the generation of sentence start words is inaccurate.
That is, CaM can improve the effect of large-scale models or pre-trained
models very well, but if it is to be combined with small-scale models and
trained from scratch, then the effectiveness of the model might not be
ensured. This will also be a direction of improvement for our future work.

##  Appendix A Related Work

###  A.1 Related Work Generation

The related work generation task can be viewed as a variant of the multi-
document summarization task, and its methods can be categorized as extractive
or abstractive. Most of the early studies use extractive methods. The work of
hoang-kan-2010-towards-automated-firstwork is one of the first attempts. They
propose a heuristic approach to generate general and specific content
separately given a topic tree. wang-cited-text-span train the model to extract
cited text spans through a specific training set and use a greedy algorithm to
select the most suitable candidate sentences to compose related works. Most
recent studies focus on abstractive approaches. xing-etal-2020-automatic-
citation-context use the citation context and the abstract of the cited papers
together as inputs to generate citation text. chen-etal-2021-capturing
construct a relation graph of the cited papers during the encoding process and
update them iteratively. The relation graph is used as an auxiliary
information for decoding. The most recent work is done by target-aware-2022,
in which they take the paper that related work belongs to as the target and
employ a target-centered attention mechanism to generate informative related
works. Besides the information from reference papers, they also use words that
can represent the target paper as the input, so we did not compare this method
as a baseline.

###  A.2 Causal Intervention

In recent years, causality theory has attracted increasing attention in
various domains. In the field of recommendation system, multi-scen-causalint
use the causal graph to model multi-scenario recommendation and solve the
problem of existing systems that may introduce unnecessary information from
other scenarios.unbiased-recom-latent-confounder propose a framework for
sequential recommendation that can perceive data biases by reweighing training
data and using inverse propensity scores[ips-score]. In the field of natural
language processing, counterfactual-reasoning introduce counterfactual
reasoning into the sentiment analysis task and leverage the knowledge of both
factual and counterfactual samples. wang-culotta-2020-identifying propose a
method for identifying spurious correlations in the text classification task.
The method extracts the words with the highest relevance to the category and
uses an estimator to determine whether the correlation is a spurious
correlation.

##  Appendix B Experiment Result for Causal Visualization

In this section, we will give an extra analysis of the experiments introduced
in Section 4.8.

Ground Truth |  Many dimension reduction techniques are proposed based on the vector forms, which are generally divided into two parts, linear and nonlinear. The classical methods of principal component analysis and multi-dimensional scaling are linear, since the outputs returned by these methods are related to the input patterns by a simple linear transformation. oliveHowever, when the input patterns lie on or near a low dimensional sub of the input space,  that is the structure of the data set may be highly nonlinear, then linear methods are bound to fail. As the research for manifold learning, several graph-based nonlinear methods have been proposed, such as locally linear em.
---|---
Transformer |  Reduction methods have been proposed on the dimensional space, such are divided into two categories: linear and nonlinear. The first method are the component analysis, the dimensional of the methods are linear to the kernel data. The data of the input dimensional space are not linear to the large dimensional space. oliveThe data space dimensional of the data be the nonlinear, and are not used.  The graph-based non-linear methods have been proposed. Including as the linear kernel, and entropy.
CaM |  Reduction methods have been proposed on the kernel space, such are divided into two categories: linear and nonlinear. The first approach component analysis are linear and dimensional analysis are based the kernel of the methods. Data of the input are not represented to the low dimensional space. oliveHowever, the data are not on a low dimensional space. The data space is more nonlinear, and the methods can not be used. The graph-based nonlinear methods have been proposed. Including as the linear entropy.

Table 3: Related works generated by CaM and Transformer. Analysis of the
bolded words is in Section 4.8.

###  B.1 Generated Related Work Comparison

From Table B, we can notice that CaM generates enriched content and its
meaning is closer to ground truth compared to Transformer. Crucially, when
pointing out the problems of previous approaches and presenting the new
ones(sentence marked in green), CaM correctly generates "however" at the
beginning of the sentence and the entire sentence has a more accurate
expression, making the transitions more seamless. But Transformer only
generates a very high-frequency word "the" at the same position. It can be
perceived that in this process Transformer is not making effective decisions,
but simply generating with preference and experience.

Figure 10: Raw visualization result from CaM on the word "however" and
Transformer on the word "the".

###  B.2 Visualization Result Analysis on Full Text

Figure 10 visualizes the cross attention of words "however" and "the" in CaM
and Transformer. Different cited papers are split with vertical lines. The
deeper blue color denotes the higher attention received by the input source
word. Judging from the overall coloring situation, we can find that in CaM,
there is more deep blue text, as well as more light-colored text. This means
the information that "however" focuses on is more targeted and more important,
and CaM is capable to produce correct content by accurately capturing document
relations and avoid distractions from the confounder. In the result of
Transformer, both light and deep blue text become less visible, and the
coverage of normal blue increases greatly, indicating that "the" focuses on a
wider range of information but lacks emphasis. It indicates that the decision
process in Transformer is unclear and ineffective.

Detailed analysis of the exact words they focus on and the decision process of
the models is presented in Section 4.8.