# A Topic-aware Summarization Framework with Different Modal Side Information

Xiuying Chen  CBRC, KAUST
CEMSE, KAUSTSaudi Arabia<EMAIL_ADDRESS>,  Mingzhe Li  Ant
GroupChina<EMAIL_ADDRESS>,  Shen Gao  Shandong UniversityChina
<EMAIL_ADDRESS>,  Xin Cheng  Peking UniversityChina
<EMAIL_ADDRESS>,  Qiang Yang  CBRC, KAUST
CEMSE, KAUSTSaudi Arabia<EMAIL_ADDRESS>,  Qishen Zhang  Ant
GroupChina<EMAIL_ADDRESS>,  Xin Gao† CBRC, KAUST
CEMSE, KAUSTSaudi Arabia<EMAIL_ADDRESS>and  Xiangliang Zhang†
University of Notre Dame
CEMSE, KAUST USA<EMAIL_ADDRESS>
(2023)

###### Abstract.

Automatic summarization plays an important role in the exponential document
growth on the Web. On content websites such as CNN.com and WikiHow.com, there
often exist various kinds of side information along with the main document for
attention attraction and easier understanding, such as videos, images, and
queries. Such information can be used for better summarization, as they often
explicitly or implicitly mention the essence of the article. However, most of
the existing side-aware summarization methods are designed to incorporate
either single-modal or multi-modal side information, and cannot effectively
adapt to each other. In this paper, we propose a general summarization
framework, which can flexibly incorporate various modalities of side
information. The main challenges in designing a flexible summarization model
with side information include: (1) the side information can be in textual or
visual format, and the model needs to align and unify it with the document
into the same semantic space, (2) the side inputs can contain information from
various aspects, and the model should recognize the aspects useful for
summarization. To address these two challenges, we first propose a unified
topic encoder, which jointly discovers latent topics from the document and
various kinds of side information. The learned topics flexibly bridge and
guide the information flow between multiple inputs in a graph encoder through
a topic-aware interaction. We secondly propose a triplet contrastive learning
mechanism to align the single-modal or multi-modal information into a unified
semantic space, where the summary quality is enhanced by better understanding
the document and side information. Results show that our model significantly
surpasses strong baselines on three public single-modal or multi-modal
benchmark summarization datasets.

abstractive summarization, multimodal summarization

* Equal contribution.

$\dagger$ Corresponding authors.

††journalyear: 2023††copyright: rightsretained††conference: Proceedings of the
46th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 23–27, 2023; Taipei, Taiwan††booktitle:
Proceedings of the 46th International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR ’23), July 23–27, 2023, Taipei,
Taiwan††doi: 10.1145/3539618.3591630††isbn: 978-1-4503-9408-6/23/07††ccs:
Information retrieval Summarization

##  1\. Introduction

The rapid growth of the World Wide Web has led to the flood of information
across the Internet (Huang et al., 2019; Shin et al., 2019; Ye et al., 2020).
On content websites such as CNN.com, Twitter.com, and WikiHow.com, there are
often corresponding images, videos, and side text along with the main
document, which can attract readers’ attention and help them understand the
content better (Wang et al., 2018; Chen et al., 2018; Yan et al., 2019; Qian
et al., 2021). Herein, we regard the auxiliary images/videos/text as side
information. Since the side information frequently make reference to the
article’s main content explicitly or implicitly, such information can also be
used to improve summarization quality, as shown by the two examples from CNN
and WikiHow Apps in Figure 1. There is also other side information in real-
world applications such as citation papers, summary templates, and reader
comments, which are helpful for summarization (Gao et al., 2021). It is thus
desired to extend text-based summarization models for taking advantage of the
summarization clues included in such side information.

There are previous works exploring utilizing side information from a specific
domain. For example, Narayan et al. (2017) first proposed to utilize image
captions to enhance summarization performance. Other textual side information
such as citation papers (An et al., 2021), reader comments (Gao et al.,
2019b), user queries (Koupaee and Wang, 2018), prototype templates (Gao et
al., 2019a) are also utilized in summarization tasks. Recently, the benefits
of visual information on summarization have also been explored. To name a few,
Zhu et al. (2018) incorporated multimodal images and Li et al. (2020) utilized
videos to help better summarization. These works are typically designed for
one specific modality of side information, while a more generally useful
summarization framework should be able to process different modalities of
information in a flexible way. Hence, in this paper, we target to address a
general summarization framework that can flexibly unify different modal side
information with the input document to generate better summaries.

Figure 1. Articles with various side information and summary collected from
the CNN and WikiHow APPs. The side information (video and user query) can
enhance the summarization performance.

There are two main challenges in this task. The first challenge comes from the
different modalities of side information. Regardless of the presented _format_
of side information, a summarization model needs to align and unify it with
the document into the same semantic space.  The other challenge lies in the
fact that the side inputs can contain information from various _aspects_ , and
the model should recognize the aspects useful for summarization. In the first
case in Figure 1, only if the summarization model can connect the visual
information “earth” and “launching” to the textual information can it generate
the informative summary. In the second case in Figure 1, the query describes
the question from computer and safety aspects, which should be the focus when
making a summary.

In this work, we propose a Unified-Modal Summarization model with Side
information (USS) to tackle the above challenges. Firstly, we propose to use
topics as the bridge to model the relationship between the main document and
the side information. Topics are a subject or theme of documents or videos,
and traditional works employ topics as cross-document semantic units to bridge
different documents (Cui and Hu, 2021). Moreover, we observe that topics can
also be an information bridge for multi-modal inputs. For instance, in the
first case in Figure 1, we can use topics “aerospace” and “nature” to relate
the videos with the summary text. Hence, in this work, we expand the topic
modeling from single-modal to multi-modal for unifying the main document and
various types of side information. For the second challenge, apart from the
limited side-document pairs, we utilize rich non-paired side and document
inputs in the collected datasets, and propose a cross-modal contrastive
learning module to align the main document and side information into a unified
semantic space. Concretely, in our model, we first introduce a unified topic
model (UTM) to learn the latent topics of the target summary by using the main
document and the side information to predict the topic distributions of the
summary. Since UTM aims to predict the topic distribution of the target
summary, it does not rely on the specific modality attributes of the input.
Based on the learned topics, we construct a graph encoder to model the
relationship between the main document and side inputs. In this topic-aware
graph encoder, we let information from two sources flow through different
channels, i.e., by direct edges and indirect edges through topics. In the
decoding process, we propose a hierarchical decoder that attends to multi-
granularity nodes in the graph guided by the topics. Moreover, the triplet
contrastive learning mechanism pushes the paired document and side
representations closer and unpaired representations far away from each other,
so as to enhance the model’s capability of understanding the main document and
side information.

Our contributions can be summarized as follows:

$\bullet$ We propose a general summarization paradigm that can take advantage
of different types of side information in a flexible way to enhance
summarization performance.

$\bullet$ To model the interaction between various inputs and unify them into
the same semantic space, we propose a unified topic model and a triplet
contrastive learning mechanism.

$\bullet$ Empirical results demonstrate that our proposed approach brings
substantial improvements over strong baselines on benchmark datasets.

##  2\. Related Work

Summarization with Side Information. Simply relying only on the main body of
the document for summarization cues is challenging (Zhao et al., 2019; Zhu et
al., 2021; Mao et al., 2022; Lima and Santos, 2022). In fact, articles in
real-world applications often have side information that is beneficial for
summarization. A series of works utilized textual side information such as
image captions (Narayan et al., 2017), questions (Koupaee and Wang, 2018; Deng
et al., 2019, 2020), prototype summaries (Gao et al., 2019b), citation papers
(Chen123 et al., 2021; An et al., 2021), timeline information (Chen et al.,
2019), and prototype templates (Gao et al., 2019a). Recently, research on
multimodal understanding gets popular, and the benefits of using visual
information on summarization have also been explored. Gao et al. (2021)
provided a survey on side information-aware summarization. Side information-
aware summarization can also be regarded as a kind of multi-document
summarization. Cui and Hu (2021); Zhou et al. (2021) introduced topic and
entity information in the summarization process, respectively. Different from
previous works which either take visual or textual side input, we propose a
general framework that can be flexibly applied with different types of side
inputs.

Topic Modeling  Neural topic modeling (NTM) was first proposed by Miao et al.
(2017), which assumes a Gaussian distribution of the topics in a document. Liu
et al. (2019); Fu et al. (2020); Yang et al. (2020); Xie et al. (2021) further
explored it in the summarization task in the text domain. Specifically, Cui
and Hu (2021) employed NTM to jointly discover latent topics that can act as
cross-document semantic units to bridge different documents and provide global
information to guide the summary generation. Liu et al. (2021) proposed topic-
aware contrastive learning objectives to implicitly model the topic change and
handle information scattering challenges for the dialogue summarization task.
In this work, we come up with a unified topic model to fit in the unified-
modal setting, which requires discovering latent topics beyond single-modal
text input.

Contrastive Learning. Contrastive learning is used to learn representations by
teaching the model which data samples are similar or not. Due to its excellent
performance on self-supervised and semi-supervised learning, it has been
widely used in natural language processing. Lee et al. (2020) generated
positive and negative examples by adding perturbations to the hidden states.
Cai et al. (2020) augmented contrastive dialogue learning with group-wise dual
sampling. It has also been utilized in caption generation (Mao et al., 2016),
summarization (Gao et al., 2019a; Liu et al., 2021; Liu and Liu, 2021; Bui et
al., 2021), dialog generation (He et al., 2022), machine translation (Bonab et
al., 2019; Yang et al., 2019) and so on. In this work, we use contrastive
learning to unify multimodal information in the summarization task.

##  3\. Model

In this section, we first define the task of unified summarization with side
information, then describe our USS model in detail.

###  3.1. Problem Formulation

Given the main document $X^{d}$ and its side information $X^{s}$, we assume
there is a ground truth summary $Y=(y_{1},y_{2},\dots,y_{N_{y}})$. To be
specific, the document $X^{d}$ is represented as a sequence of words
$(x^{d}_{1},x^{d}_{2},\dots,x^{d}_{N_{d}})$. The side information can be in
textual or visual formats. For textual side information, it is represented as
$X_{s}=(x^{s}_{1},x^{s}_{2},\dots,x^{s}_{N_{s}})$, and for visual side
information, we use $X_{s}$ to denote the images. $N_{d}$ and $N_{s}$ are the
word number or the image number in a document or side information. Given
$X^{d}$ and $X^{s}$, our model generates a summary
$\hat{Y}=\\{\hat{y}_{1},\hat{y}_{2},\dots,\hat{y}_{N_{\hat{y}}}\\}$. Finally,
we use the difference between the generated $Y$ and the gold $\hat{Y}$ as the
training signal to optimize the model parameters.

###  3.2. Overview

Our model is illustrated in Figure 2, which follows the Transformer-based
encoder-decoder architecture. We augment the encoder with a unified topic
modeling network (§ 3.3) which learns the latent topic representations from
source inputs and target summary, based on which a topic-aware graph encoder
(§ 3.4) builds graphs for the document and side input, and models their
relationship through the learned topics. Correspondingly, we design a summary
decoder (§ 3.5) which generates the summary with a topic-aware attention
mechanism. To better represent the representations from different spaces, we
also design a triplet contrastive learning module (§ 3.6) to align the paired
multimodal information into the same space.

###  3.3. Unified Topic Modeling

We first use a unified topic model (UTM) to establish the relationship between
the document and side information. The model takes inspiration from the neural
topic model (NTM) (Miao et al., 2017) which only applies for textual inputs.
We first introduce the NTM, and how we adapt NTM to grasp the semantic
meanings of multimodal inputs.

Figure 2. Overview of USS, which consists of four parts: (1) Unified Topic
Modeling (left) jointly learns latent topics from both inputs; (2) Topic-aware
Graph Encoder (bottom) relates the document to the side information; (3)
Summary Decoder (right) with hierarchical topic-aware attention mechanism; and
(4) Triplet Contrastive Learning (top) aligns the multiple inputs and outputs
into a unified semantic space.

Overall, NTM assumes the existence of $K$ underlying topics throughout the
inputs. Concretely, NTM encodes the bag-of-word term vector of the input to a
topic distribution variable, based on which it reconstructs the bag-of-word
representation. In the reconstruction process, the topic representations can
be extracted from a projection matrix. In our UTM, instead of reconstruction,
we aim to predict the bag-of-word vector of the target summary based on the
two inputs. The benefits are threefold. Firstly, we no longer require the
input to be in textual format and can encode various modal semantic meanings
of the inputs into the distribution variable. Secondly, we can preserve the
most salient information from the inputs, instead of keeping them all, which
is consistent with the information filtering attribute of the summarization
task. Lastly, the combination of topic modeling on document and side input can
better fit the target summary topic distribution.

Concretely, we first process the document $X^{d}$ into the bag-of-word
representation $h^{d}\in\mathbb{R}^{|V|}$, where $|V|$ is the vocabulary size.
The same is true for the side information when it is in the textual format,
leading to $h^{s}$. When the side information is images or videos, we use
EfficientNet (Tan and Le, 2019) to obtain the vector representation, also
denoted as $h^{s}$. We then employ an MLP encoder to estimate their exclusive
priors $\sigma^{*}$ and $\mu^{*}$, which are used to generate the topic
variables of the two inputs through a Gaussian softmax:

(1) $\displaystyle\sigma^{*}=f_{\sigma}\left(h^{*}\right),\mu^{*}$
$\displaystyle=f_{\mu}\left(h^{*}\right),$ (2) $\displaystyle
z^{*}\sim\mathcal{N}\left(\sigma^{*},\mu^{*2}\right),$

where $*$ can be $d$ or $s$, $f_{\sigma}(.)$ and $f_{\mu}(.)$ are neural
perceptrons with ReLU activation. $\mathcal{N}(.)$ is a Gaussian distribution.
$z^{*}\in\mathbb{R}^{K}$ are the latent topic variables of the document or the
side information.

Given the topic variables $z^{d}$ and $z^{s}$, UTM predicts the bag-of-word
representation of target summary, i.e., $y_{bow}$:

(3)
$\displaystyle\theta=\operatorname{softmax}\left(z^{d}+z^{s}\right),\hat{y}_{bow}=\operatorname{softmax}\left(\theta\mathrm{W}_{\phi}\right).$

We add the topic variables of the two inputs together to include information
from two sources, as well as to emphasize the salient information that is
shared between both sides. Based on the topic distribution $\theta$ we
construct the bag-or-word of target summary $y_{bow}$. In this process, the
weight matrix of $\mathrm{W}_{\phi}\in\mathbb{R}^{K\times|V|}$ can be regarded
as the topic-word relationship, where $\mathrm{W}_{\phi}^{i,j}$ indicates the
weight of the $j$-th word in the $i$-th topic, and $K$ is the topic number.
$\theta\in\mathbb{R}^{K}$ reflects the proportion of each topic, and higher
$\theta_{i}$ score means the $i$-th topic is more important. We will take
advantage of this distribution to determine the main topics of each case in
the next section.

The objective function is to simultaneously minimize the Wasserstein distance
between $p(z^{*})$ and $q(z^{*}\mid h^{*}_{x})$, and maximize the constructing
probability of $y_{bow}$:

$\displaystyle\mathcal{L}_{UTM}=\textstyle\sum_{*\in\\{d,s\\}}\operatorname{W}(p(z^{*})\|q(z^{*}\mid
h^{*}_{x}))-\mathbb{E}_{q(z)}[\log p(y_{bow}\mid z^{*})],$

where $p(z^{*})$ is the standard Gaussian distribution. We employ the
Wasserstein distance instead of traditional KL-divergence since the former is
proved to be superior to the latter by experiments (Tolstikhin et al., 2017).

###  3.4. Topic-aware Graph Encoder

Graph Construction. Since we have extracted the salient topic distribution of
the two inputs, we can use them as bridges to let two information sources
interact with each other. We thus design a topic-aware graph encoder where we
model the relation between document and side inputs through different
channels, i.e., by direct edges and indirect edges through topics. By direct
edges, we let information flow globally in the graph, while by indirect edges,
the document communicates specific information with side input under different
topics.

Node Initialization. For both inputs, we use the Transformer encoder (Vaswani
et al., 2017) or the EfficientNet model to encode each document or image
independently to capture the contextual information. We first introduce the
Transformer architecture in detail, and we will also propose variations of the
attention mechanism. Generally, Transformer consists of a stack of token-level
layers to obtain contextual word representations in the document or side
information. We take the document to illustrate this process.

For the $l$-th Transformer layer, we first use a fully-connected layer to
project word state $h^{d,l-1}_{x_{i}}$ into the query, i.e.,
$Q^{l-1}_{i}=F_{q}(h^{d,l-1}_{x_{i}})$. For self-attention mechanism, the key
and value are obtained in a similar way: i.e.,
$K^{l-1}_{i}=F_{k}(h^{d,l-1}_{x_{i}})$,
$V^{l-1}_{i}=F_{v}(h^{d,l-1}_{x_{i}})$. Then, the updated representation of
$Q$ is formed by linearly combining the entries of $V$ with the weights:

(4)
$\displaystyle\alpha_{i,j}=\frac{\exp\left(Q^{l-1}_{i}K^{l-1}_{j}\right)}{\sum_{n=1}^{N_{d}}\exp\left(Q^{l-1}_{i}K^{l-1}_{n}\right)},a_{i}=\sum_{j=1}^{N_{d}}\frac{\alpha_{i,j}V^{l-1}_{j}}{\sqrt{d_{e}}},$

where $d_{e}$ stands for hidden dimension. The above process is summaized as
$\text{MHAtt}(h^{d,l-1}_{x_{i}},h^{d,l-1}_{x_{*}})$, where $*$ denotes index
from $1$ to $N_{d}$. Then, a residual layer takes the output of self-attention
sub-layer as the input:

(5) $\displaystyle\hat{h}_{x_{i}}^{d,l-1}$
$\displaystyle=\text{LN}\left(h_{x_{i}}^{d,l-1}+\text{MHAtt}\left(h_{x_{i}}^{d,l-1},h_{x_{*}}^{d,l-1}\right)\right),$
(6) $\displaystyle h_{x_{i}}^{d,l}$ $\displaystyle=\text{LN
}\left(\hat{h}_{x_{i}}^{d,l-1}+\operatorname{FFN}\left(\hat{h}_{x_{i}}^{d,l-1}\right)\right),$

where FFN is a feed-forward network with an activation function, LN is a layer
normalization (Ba et al., 2016).

The final output representations for words in the document and textual side
information are denoted as $\\{h^{d}_{1},...,h^{d}_{N_{d}}\\}$ and
$\\{h^{s}_{1},...,h^{s}_{N_{s}}\\}$. For images in visual side information, we
also denote them by $\\{h^{s}_{1},...,h^{s}_{N_{s}}\\}$. As for the topic
nodes, we use the intermediate parameters $W_{\phi}$ learned from UTM as raw
features to build topic representations $H^{o}=f_{\phi}(W_{\phi})$, where the
$i$-th row of $H^{o}\in\mathbb{R}^{K\times d_{e}}$, denoted as $h^{o}_{i}$, is
a topic vector with predefined dimension $d_{e}$. $f_{\phi}(.)$ is a neural
perceptron with ReLU activation.

Graph Encoding. The document and side graphs communicate with each other
through topic-guided and direct interactions. The topic-guided interaction
starts from the learning of document representations and the side
representations, and then the topic representations. The direct interaction
only updates the document and side nodes. We omit the layer index here for
brevity.

Concretely, in the topic-guided interaction, the document and side information
representations are updated from three sources. Take the document nodes for
example, they are updated by (1) performing self-attention across document
nodes; (2) performing cross-attention to obtain the topic-aware document
representations, as shown in Figure 3(a); and (3) performing our designed
topic-guided attention mechanism, as shown in Figure 3(b). This mechanism
starts with the application of self-attention mechanism on the document nodes:
$\dot{h}^{d}_{i}=\text{MHAtt}\left(h^{d}_{i},h^{d}_{*}\right)$. Then, taking
the topic representation $h^{o}=\textstyle\sum^{K}_{i=1}h^{o}_{i}$ as
condition, the attention score $\beta_{i}$ on each original document
representation $h^{d}_{i}$ is calculated as:

(7) $\displaystyle\beta_{i}^{d}$
$\displaystyle=\text{softmax}\left(\text{FFN}\left(h^{d}_{i}(h^{o})^{T}\right)\right).$

The topic-aware document representation is $\dot{h}^{d}_{i}$ weighted by
$\beta_{i}^{d}$, denoted as $\beta_{i}\dot{h}^{d}_{i}$. In this way, we
highlight the salient part of the two inputs under the guidance of the topics.
Last, a feed-forward network is employed to integrate three information
sources.

The topic representation is updated by performing (1) self-attention and (2)
cross-attention on the adjacent document and side nodes. In the cross-
attention, the topic representation is taken as the query, and the document
and side representations are taken as the keys and values. Lastly, a feed-
forward network integrates two information sources to obtain the updated topic
representation.

Aside from communicating the graphs through topics, we also have a direct
interaction that concatenates all document and side nodes in the graph and
then apply a self-attention mechanism.

The topic-aware and direct interactions are processed iteratively, and we
denote the final updated representations for document, side information, and
topics as $\hat{h}^{d}\in\mathbb{R}^{N_{d}\times d_{e}}$,
$\hat{h}^{s}\in\mathbb{R}^{N_{s}\times d_{e}}$, and
$\hat{h}^{o}\in\mathbb{R}^{K\times d_{e}}$.

Figure 3.  (a) Cross attention mechanism for document and topic nodes. (b)
Topic-guided attention mechanism, which captures semantic information across
the document and side information under the guidance of the topics.

###  3.5. Summary Decoder

Since the decoder needs to incorporate the information from multiple sources
in the graph encoder, we design a hierarchical decoder that firstly focuses on
the topics and then attends to inputs. This topic-guided mechanism indicates
which topics should be discussed in each decoding step. Our hierarchical
decoder follows the style of Transformer, and we omit the layer index next for
brevity.

For each layer, at the $t$-th decoding step, we first apply the masked self-
attention on the summary embeddings (MSAttn), obtaining the decoder state
$\tilde{g}_{t}$. The masking mechanism ensures that the prediction of the
position $t$ depends only on the known output of the position before $t$:

$\displaystyle\tilde{g}_{t}=\text{LN
}\left(g_{t}+\text{MSAttn}\left(g_{t},g_{*}\right)\right).$

Based on $\tilde{g}_{t}$ we compute the cross-attention scores over topics:

(8) $\displaystyle
z_{o,t}=\text{ReLU}([\tilde{g}_{t}W_{a}(\hat{h}^{o}W_{b})^{T}]),$

where $W_{a},W_{b}\in\mathbb{R}^{d_{e}\times d_{e}}$,
$z_{o,t}\in\mathbb{R}^{K}$. We then use the topic attention to guide the
attention on the other two graphs, where the topics can be regarded as an
indicator of saliency. Taking the main document for example, we incorporate
$z_{o,t}$ with similarity weight $e_{d}$ to obtain the document attention
weight $z_{d,t}\in\mathbb{R}^{N_{d}}$:

(9) $\displaystyle e_{d}$
$\displaystyle=\text{FFN}(\hat{h}^{o}(\hat{h}^{d})^{T}),$ (10) $\displaystyle
z_{d,t}$ $\displaystyle=\textstyle z_{o,t}e_{d},$

where $e_{d}\in\mathbb{R}^{K\times N_{d}}$ is the similarity matrix between
the topics and document. In a similar way, we obtain the attention weights
$z_{s,t}\in\mathbb{R}^{N_{s}}$ on the side information.

The attention weights $z_{o,t}$, $z_{d,t}$, and $z_{s,t}$ are then used to
obtain the context vectors $c_{o,t}$, $c_{d,t}$, and $c_{s,t}$, respectively.
Take the topics as an example:

(11) $\displaystyle c_{o,t}=z_{o,t}\hat{h}^{o}.$

These context vectors, treated as salient contents summarized from various
sources, are concatenated with the decoder hidden state $g_{t}$ to produce the
distribution over the target vocabulary:

(12) $\displaystyle P_{t}^{\text{vocab
}}=\text{Softmax}\left(W_{d}\left[g_{t};c_{o,t};c_{d,t};c_{s,t}\right]\right).$

All the learnable parameters are updated by optimizing the negative log
likelihood objective function of predicting the target words:

(13) $\displaystyle\mathcal{L}_{s}=-\textstyle\sum_{t=1}^{N_{y}}\log
P_{t}^{\text{vocab}}\left(\hat{y}_{t}\right).$

Table 1.  Comparison with other baselines when side information is in text.
All our ROUGE scores have a 95% confidence interval of at most $\pm$0.28 as
reported by the official ROUGE script. Numbers in bold mean that the
improvement to the best baseline is statistically significant (a two-tailed
paired t-test with p-value <0.01). ‘-’ indicates unavailability.

|  | CNN |  |  |  | WikiHow |  |  |  | VMSMO |  |
---|---|---|---|---|---|---|---|---|---|---|---|---
R1 | R2 | RL | BS | R1 | R2 | RL | BS | R1 | R2 | RL | BS
Lead3 | 29.1 | 11.1 | 25.9 | 85.4 | 24.4 | 5.5 | 15.6 | 84.7 | 16.2 | 5.3 | 13.9 | 84.5
BERTSumEXT | 31.2 | 12.2 | 27.8 | 86.3 | 30.4 | 8.6 | 28.3 | 85.7 | 27.3 | 9.6 | 23.4 | 85.7
SideNet | 30.7 | 11.7 | 27.2 | 86.2 | 28.4 | 6.2 | 25.9 | 85.4 | - | - | - | -
BERTSumABS | 31.3 | 11.6 | 28.7 | 86.5 | 35.9 | 13.9 | 34.8 | 86.9 | 27.8 | 10.1 | 24.7 | 85.9
BERTSumABS-concat | 31.7 | 11.9 | 29.1 | 86.6 | 35.7 | 14.0 | 34.2 | 86.7 | - | - | - | -
SAGCopy | 31.9 | 11.8 | 28.9 | 86.5 | 36.1 | 13.7 | 33.9 | 86.6 | - | - | - |
EMSum | 31.9 | 12.6 | 29.7 | 86.8 | 35.4 | 13.8 | 34.6 | 86.7 | - | - | - | -
TG-MultiSum | 32.2 | 12.7 | 29.9 | 86.8 | 36.2 | 14.6 | 35.2 | 87.0 | - | - | - | -
OFA | 31.6 | 11.4 | 28.4 | 86.1 | 34.6 | 13.2 | 33.8 | 86.3 | 30.5 | 13.5 | 28.4 | 86.9
MOF | - | - | - |  | - | - | - |  | 30.8 | 13.4 | 28.4 | 86.9
VMSMO | - | - | - |  | - | - | - |  | 31.2 | 13.8 | 28.5 | 87.1
USS | 33.9 | 14.2 | 31.3 | 87.1 | 37.7 | 15.8 | 36.5 | 87.3 | 32.7 | 15.1 | 30.0 | 87.9
USS w/o unified topic modeling | 32.5 | 13.0 | 30.3 | 86.8 | 36.5 | 15.0 | 35.4 | 87.0 | 30.9 | 12.8 | 27.9 | 87.4
USS w/o graph encoder | 32.3 | 12.9 | 30.0 | 86.7 | 36.2 | 14.8 | 35.5 | 87.1 | 29.5 | 11.6 | 26.5 | 87.3
USS w/o contrastive learning | 32.8 | 13.2 | 30.6 | 86.9 | 36.7 | 15.3 | 35.8 | 87.1 | 30.4 | 13.1 | 27.8 | 87.5

###  3.6. Triplet Contrastive Learning

The challenge of unifying different modalities is to align and unify their
representations at different levels. In this section, we propose a triple
contrastive learning mechanism that determines whether the textual and visual
representations match each other. We can utilize the large-scale non-paired
text corpus and image collections to learn more generalizable textual and
visual representations, and improve the capability of vision and language
understanding.

As shown in the fourth part in Figure 2, the main idea is to let the
representations of the paired images or text close to each other in the
semantic space while the non-paired be far away. For the positive sample
construction, we apply mean pooling on the representations of
$\hat{h}^{d}_{*}$ as the overall representation $D$ for the document, and $S$
for side information in the same way. The final decoder state of generator
$g_{N_{\hat{y}}}$ is taken as the overall representation $G$ for the generated
summary, as it stores all the accumulated information. For the negative sample
construction, we randomly sample a negative side input, document, or the
generated summary from the same training batch for each case. Note that
different from the positive pairs, the sampled side and texts are encoded
individually without graph encoders as they mainly carry weak correlations. In
this way, we can create positive examples $\mathcal{X}^{d+}$ consisting of
paired document-side samples $(D,S)$, $\mathcal{X}^{s+}$ consisting of paired
side-generation $(S,G)$, and $\mathcal{X}^{g+}$ consisting of paired document-
generation $(G,D)$. Negative examples are denoted as $\mathcal{X}^{d-}$,
$\mathcal{X}^{s-}$, and $\mathcal{X}^{g-}$, respectively.

Based on these positive and negative pairs, the following contrastive loss
$\mathcal{L}_{triplet}$ is utilized to learn detailed semantic alignments
across vision and language:

$\displaystyle\mathcal{L}_{triplet}=\mathbb{E}_{A,B}\left[-\log\frac{\sum_{\left(A^{+},B^{+}\right)\in\mathcal{X}^{\\{g+,d+,s+\\}}}\exp\left(d\left(A^{+},B^{+}\right)/\tau\right)}{\sum_{\left(A^{*},B^{*}\right)\in\mathcal{X}^{\\{g^{*},d^{*},s^{*}\\}}}\exp\left(d\left(A^{*},B^{*}\right)/\tau\right)}\right],$

where $\tau$ denotes the temperature parameter, $A,B\in\\{D,S,G\\}$, and
$d(A,B)$ denotes cosine similarity between the two representations.

##  4\. Experiments

###  4.1. Dataset

We evaluated our model on three public summarization datasets with side
information: (1) CNN dataset is collected by Narayan et al. (2017) from CNN
webpage, which associates the document-summary pairs with the image captions.
The dataset contains 90,266/1,220/1,093 training/validation/test samples. Note
that this dataset is different from the CNN/DM dataset (Hermann et al., 2015).
(2) WikiHow dataset (Koupaee and Wang, 2018) is an abstractive summarization
dataset collected from a community-based QA website, in which each sample
consists of a query, a document as the answer to the question, and a short
summary of the document. We use the provided script and obtain more than
170,000 pairs, with 168128/6000/6000 training/validation/test samples. (3)
VMSMO dataset (Li et al., 2020) is a multimodal summarization dataset with
videos as side information. The average video duration is one minute and the
frame rate of the video is 25 fps. Overall, there are 184,920 samples in the
dataset, which is split into a training set of 180,000 samples, a validation
set of 2,460 samples, and a test set of 2,460 samples. The average document
length, side information length, and summary length of CNN and WikiHow
datasets are 763, 71, 50 words, 579, 10, 62 words, respectively. The document
and summary length for VMSMO is 97 and 11 words. We chose these datasets to
examine the flexibility of our model on different formats of side information.

###  4.2. Baselines

Our extractive baselines include:

Lead3 produces the three leading sentences of the document as the summary as a
baseline.

SideNet (Narayan et al., 2017) consists of an attention-based extractor with
attention over side information.

BERTSumEXT (Liu and Lapata, 2019b) is an extractive summarization model with
pretrained BERT encoder that is able to express the semantics of a document
and obtain representations for its sentences. It only takes the document as
input.

Abstractive single-document and multi-document summarization baseline models
include:

BERTSumABS (Liu and Lapata, 2019b) is an abstractive summarization system
built on BERT${}_{\text{base}}$ with a new designed fine-tuning schedule. It
only takes the document as input. We also have a BERTSumABS-concat that
concatenates the textual side information with the original document.

SAGCopy (Xu et al., 2020) is an augmented Transformer with a self-attention
guided copy mechanism.

EMSum (Zhou et al., 2021) is an entity-aware model for abstractive multi-
document summarization with BERT encoder.

TG-MultiSum (Cui and Hu, 2021) is a multi-document summarizer with topics act
as cross-document semantic units.

The above two multi-document summarization baselines take the textual side
input as the second document. We also compare our model with multimodal
summarization baselines including:

MOF (Zhu et al., 2020) is a summarization model with a multimodal objective
function with the guidance of multimodal reference to use the loss from the
summary generation and the image selection.

VMSMO (Li et al., 2020) is a dual interaction-based multimodal summarizer with
multiple inputs. The four above models are all equipped with
BERT${}_{\text{base}}$ encoder for fairness.

OFA (Wang et al., 2022) is a recent unified paradigm for multimodal
pretraining. We adapt it for the side-aware summarization setting, where we
directly concatenate the document and side representations encoded by OFA. We
choose $\text{OFA}_{\text{base}}$ version for fairness.

###  4.3. Evaluation Metrics

For both datasets, we evaluated by standard ROUGE-1 (R1), ROUGE-2 (R2), and
ROUGE-L (RL) (Lin, 2004) on full-length F1, which refer to the matches of
unigram, bigrams, and the longest common subsequence, respectively. We then
used BERTScore (BS) (Zhang et al., 2020) to calculate a similarity score
between the summaries based on their BERT embeddings.

Schluter (2017) noted that only using the ROUGE metric to evaluate generation
quality can be misleading. Therefore, we also evaluated our model by human
evaluation. Concretely, we asked three PhD students proficient in English to
rate 100 randomly sampled cases generated by models from the CNN and WikiHow
datasets which cover different domains. The setting follows (Liu et al., 2022)
with four times larger evaluation scale. The evaluated baselines are EMSum,
TG-MultiSum, and OFA which achieve top performances in automatic evaluations.

Our first evaluation quantified the degree to which the models can retain the
key information following a question-answering paradigm (Liu and Lapata,
2019a). We created a set of questions based on the gold-related work and
examined whether participants were able to answer these questions by reading
generated text. The principle for writing a question is that the information
to be answered is about factual description, and is necessary for the
summarization. Two annotators wrote three questions independently for each
sampled case. Then they together selected the common questions as the final
questions that they both consider to be important. Finally, we obtained 147
questions, where correct answers are marked with 1 and 0 otherwise. Our second
evaluation study assessed the overall quality of the related works by asking
participants to score them by taking into account the following criteria:
Informativeness (does the related work convey important facts about the topic
in question?), Coherence (is the related work coherent and grammatical?), and
Succinctness (does the related work avoid repetition?). The rating score
ranges from 1 to 3, with 3 being the best. Both evaluations were conducted by
another three PhD students independently, and a model’s score is the average
of all scores.

###  4.4. Implementation Details

All models were trained for 200,000 steps on NVIDIA A100 GPU. We implemented
our model in Pytorch and OpenNMT (Klein et al., 2017). For neural-based
baselines except OFA and our model, we used the ‘bert-base’ or ‘bert-base-
chinese’ versions of BERT for fair comparison. Both source and target texts
were tokenized with BERT’s subwords tokenizer. Our Transformer decoder has 768
hidden units and the hidden size for all feed-forward layers is 2,048. In all
abstractive models, we applied dropout with probability 0.1 before all linear
layers; label smoothing (Szegedy et al., 2016) with smoothing factor 0.1 was
also used. For CNN dataset, the encoding step is set to 750 for a document and
70 for side information. The minimum decoding step is 30, and the maximum step
is 50. For WikiHow dataset, the four parameters are set to 600, 10, 30, 65.
For the Chinese VMSMO dataset, the parameters are 200, 125, 10, 50, where 125
is the encoded frame number. The video frames are selected every 25 frames to
ensure the continuity of the images, similar to (Li et al., 2020). We used
Adam optimizer as our optimizing algorithm. We also applied gradient clipping
with a range of $[-2,2]$ during training. During decoding, we used beam search
size 5, and tuned the $\alpha$ for the length penalty (Wu et al., 2016)
between 0.6 and 1 on the validation set; we decoded until an end-of-sequence
token is emitted and repeated trigrams are blocked. Our decoder applies
neither a copy nor a coverage mechanism, since we also rarely observe issues
with out-of-vocabulary words in the output; moreover, trigram-blocking
produces diverse summaries managing to reduce repetitions. On VMSMO, the video
frames are selected every 25 frames to ensure the continuity of the images,
similar to (Li et al., 2020). We selected the 5 best checkpoints based on
performance on the validation set and report averaged results on the test set.

###  4.5. Experimental Results

Automatic Evaluation. The performance comparison is shown in Table 1. Firstly,
we can see that the attributes of the three datasets vary. CNN is a news
dataset with a pyramid structure, where Lead3 and extractive methods achieve
higher performance than other datasets. Secondly, combining side information
by simple concatenation cannot make full use of it, as we can see that the
performance of BERTSumABS-concat does not improve significantly compared with
BERTSumABS. Incorporating side information by multi-document summarization
structures is a better way utilize side information, but they cannot be
applied in the multimodal scenario, and their improvements are also limited.
Thirdly, the recent multimodal pretrained baseline OFA achieves relatively
good performance on VMSMO, but has borderline performance on single-modal
datasets CNN and WikiHow. This is consistent with the previous observation
that OFA has better performance on cross-modal tasks (Wang et al., 2022).
Specifically, OFA has trouble when generating long text, which leads to a
performance drop when the target is relatively long. Finally, our USS model
obtains consistently better performance on all three datasets. Specifically,
USS achieves 2.0/1.6/1.7/0.3 improvements on R1, R2, RL, and BERTScore
compared with one of the latest baseline EMSum on the CNN dataset, and obtains
1.4/1.0/1.3/0.7 improvements on the VMSMO dataset compared with TG-MultiSum.

Table 2. QA performance, Informativeness (Info), Coherence (Coh), and Succinctness (Succ) comparison by human evaluation. | QA(%) | Info | Coh | Succ
---|---|---|---|---
OFA | 34.6 | 2.14 | 2.01 | 2.03
EMSum | 37.4 | 2.39 | 2.13 | 2.26
TG-MultiSum | 38.1 | 2.42 | 2.15 | 2.28
USS | 42.8 | 2.54 | 2.27 | 2.41

Human Evaluation. As shown in Table 2, on both evaluations, participants
overwhelmingly prefered our model. The kappa statistics are 0.42, 0.49, and
0.45 for Info, Coh, and Succ respectively, indicating the moderate agreement
between annotators. All pairwise comparisons among systems are statistically
significant using a two-tailed paired t-test for strong significance for
$\alpha$ = 0.01. We also provide examples of the system output in Table 3. We
can see that with the side information showing the figure of the main
character, the lottery result, and the mobile phone, USS successfully captures
the gist information that “a man post a lottery ticket on social media” in the
generated summary. For BERTSumABS-concat and VMSMO, they miss key information
such as “where he post the lottery” and “how quickly the lottery was falsely
claimed”.

Table 3. Examples of the generated summary by baselines and USS on CNN and
VMSMO datasets. Unfaithful and redundant information is highlighted in blue.
In the second case, keywords with the same semantics are highlighted in red
and green.  _Article:_ Recently, a citizen of Nantong, Jiangsu, won a lottery
ticket. He took photos of the entire lottery ticket and uploaded them to
Moments. Unexpectedly, someone else falsely claimed the lottery winnings as
his own based on the information on the lottery. The lottery was redeemed
within only 35 seconds after the start of the redemption day as investigated
by the Sports Lottery Center.
近日，江苏南通，市民张先生彩票中了奖后，将整张彩票拍照上传了朋友圈，不料被人根据彩票上的信息冒领了奖金。经体彩中心调查当天开始兑奖后仅35秒奖金就被兑走。
---
_Reference summary:_ Too excited to win the lottery, post the lottery in
Moments and got falsely claimed immediately 中奖太兴奋，朋友圈晒彩票瞬间被冒领
_OFA:_ 35 seconds after winning, the lottery was falsely claimed 中奖35秒后被冒领彩票
_MOF:_ Man showed the winning lottery and was falsely claimed in 35 seconds
男子晒中奖35秒被冒领
_VMSMO:_ Post lottery in Moments and get falsely claimed 朋友圈晒中奖被冒领
_USS:_ Friends from Moments falsely claimed the lottery, only 35 seconds after
the redemption started 朋友圈冒领彩票，中奖35秒就被兑走
_Highest three topics:_
Topic1: old friend 老朋友, Liang family 梁家, WeChat 微信, phone calls 通电话, Brothers
兄弟俩
Topic2: covet 贪图, steal 偷盗, kidnap 拐骗, holocaust 大屠杀, steal everything 抢光
Topic3: prize 奖金, tens of thousands 好几万, giants 豪门, net flow 净流入, more than
100 million yuan 亿余元
Side information (sampled images from video):

##  5\. ANALYSIS AND DISCUSSION

###  5.1. Ablation Study

We conducted ablation tests to assess the importance of the topic modeling,
graph encoder, and triplet contrastive learning. For USS w/o unified topic
modeling, only the traditional neural topic model (NTM) is applied to the
textual document to obtain the topic representations. For USS w/o graph
encoder, there are no topic-related interactions, and the outputs from the
topic modeling are directly used for decoding. The ROUGE score results are
shown in the last block of Table 1. All ablation models perform worse than USS
in terms of all metrics, which demonstrates the preeminence of USS.
Concretely, graph encoder makes a great contribution to the model, improving
the performance on CNN by 1.3 in terms of the R2 score, and improving the R2
score by 1.0 on the WikiHow dataset. Contrastive learning also contributes to
the model, bringing 0.7 RL improvements on the CNN dataset.

Figure 4. (a) Relationships between the number of topics and $\overline{R}$
(the average of RG-1, RG-2 and RG-L) . Best viewed in color. (b) Relationships
between the number of graph layer and $\overline{R}$.

We further conduct experiments on VMSMO to probe into the impact of two
important parameters, i.e., the topic number $K$ and the graph layer number
$L$. From Figure 4, we can see that for both experiments, the ROUGE scores
increase with the topic and layer number, to begin with. After reaching the
upper limit it begins to drop. Note that with only one graph layer our model
outperforms the best baseline, which demonstrates that our topic-aware graph
module is effective. Hence, we set the default topic number to 100 and the
graph layer number to 4.

Table 4. Coherence score $C_{v}$ and inferred topic words of different topic models. Blue text denotes repetition or non topic words. Models | $C_{v}$ | Sampled words
---|---|---
GSM |  0.392 |  political, growth, economy, europe, according, world, states, better, opportunity
W-LDA |  0.462 |  mcconnell, consensus, electorate, reduction, repeal, partisan, economy, economies, growth
USS | 0.493 |  consumers, billion, growth, economy, global, companies, cost, oil, infrastructure, sector

Figure 5. (a) The multi topic distribution of the document and side
information. (b) UTM loss ($\mathcal{L}_{UTM}$) curve in the training process.

###  5.2. Topic Quality Analysis

In this subsection, we qualitatively and quantitatively investigate the
quality of the selected topics. We compared the learned topics from our model
with baseline topic models trained on the CNN dataset including (1) GSM (Miao
et al., 2017), a classic NTM model with VAE and Gaussian softmax, and (2)
W-LDA (Nan et al., 2019), a novel neural topic model in the Wasserstein
autoencoders framework.

In Table 4, we use the coherence score $C_{v}$ (Röder et al., 2015) to
quantitatively evaluate inferred topics, which has been proved highly
consistent with human evaluation. We also show the inferred words for the
topic “economy”. It can be seen that our USS outperforms other baselines in
terms of the coherence score, and the inferred topic words are more accurate
and concentrated. The possible reasons are twofold. Firstly, our model
incorporates the main and side inputs to predict the topic distribution of the
target summary. The multiple descriptions of the same content bring more topic
clues, and the prediction task that requires reasoning and filtering abilities
makes the topic model strong and robust. Secondly, the assistant summarization
task can boost the performance of topic modeling.

Figure 6. (a) Visualizations of the attention weights on topics. (b)
Contrastive learning loss curve ($\mathcal{L}_{triplet}$) in the triplet
contrastive learning module.

###  5.3. Effect of Unified Topic Modeling

Since we have verified the quality of the topics, we are interested to see the
effect of the learned topics on summarization, i.e., how the unified topic
modeling helps summarization?

We first examine from the encoder side, where we show the learned topic
distributions from two inputs for the case in Table 3 in Figure 5(a). It can
be seen that though the document and side information has different topic
distributions, generally, they focus on the same important topics, which are
related to the ground summary by human evaluation. From the statistic view, we
draw the loss of $\mathcal{L}_{UTM}$ in Figure 5(b). The curve has a steady
downtrend, to begin with, and finally reaches convergence. The above
observations demonstrate that the topic modeling can grasp the gist of the
target summary and the effectiveness of topic modeling.

We next examine the topic effectiveness in the summarization process from the
decoder side. We visualize the attention weights $z_{o,t}$ on topics in Figure
6(a) for the same case. It can be seen that the topic attention first
emphasizes topic 1, and then on topic 2 as well as topic 3. The three topics
are shown in Table 3, which are related to “social media”, “crime”, and
“finance”, respectively. This is consistent with the generated sentence, where
the keyword starts from “Moments”, and then changes to “falsely claimed
redemption”. In this way, we can see that the topics play a guidance role when
generating summaries.

###  5.4. Contrastive Learning Analysis

We lastly examine the performance of the triplet contrastive learning module
by visualizing the contrastive loss curve in Figure 6(b) on VMSMO. It can be
seen that the loss score fluctuates at the beginning of the training and
gradually reaches convergence. This phenomenon demonstrates that the generated
text, the document, and the side information belonging to the same case are
getting closer in the semantic space. On the other hand, the unpaired triplets
are becoming more distant.

##  6\. Conclusion and limitation

In this paper, we proposed a general summarization framework, which can
flexibly incorporate various modalities of side information. We first proposed
a unified topic model to learn latent topic distributions from various modal
inputs. We then employed a topic-aware graph encoder that relates one input to
another by topics. Experiments on three public benchmark datasets show that
our model produces fluent and informative summaries, outperforming strong
systems by a wide margin.

## Acknowledgments

We would like to thank the anonymous reviewers for their constructive
comments. The work was supported by King Abdullah University of Science and
Technology (KAUST) through grant awards FCC/1/1976-44-01, FCC/1/1976-45-01,
REI/1/5234-01-01, and RGC/3/4816-01-01.

## References

  * (1)
  * An et al. (2021) Chenxin An, Ming Zhong, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2021\.  Enhancing scientific papers summarization with citation graph. In _Proc. of AAAI_.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016\.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ (2016).
  * Bonab et al. (2019) Hamed Bonab, James Allan, and Ramesh Sitaraman. 2019.  Simulating CLIR translation resource scarcity using high-resource languages. In _Proc. of SIGIR_.
  * Bui et al. (2021) Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. 2021.  Self-supervised contrastive learning for code retrieval and summarization via semantic-preserving transformations. In _Proc. of SIGIR_.
  * Cai et al. (2020) Hengyi Cai, Hongshen Chen, Yonghao Song, Zhuoye Ding, Yongjun Bao, Weipeng Yan, and Xiaofang Zhao. 2020.  Group-wise Contrastive Learning for Neural Dialogue Generation. In _EMNLP_.
  * Chen et al. (2019) Xiuying Chen, Zhangming Chan, Shen Gao, Meng-Hsuan Yu, Dongyan Zhao, and Rui Yan. 2019\.  Learning towards Abstractive Timeline Summarization. In _Proc. of IJCAI_.
  * Chen et al. (2018) Yangyu Chen, Shuhui Wang, Weigang Zhang, and Qingming Huang. 2018\.  Less is more: Picking informative frames for video captioning. In _Proc. of ECCV_.
  * Chen123 et al. (2021) Xiuying Chen123, Hind Alamro34, Shen Gao, Xiangliang Zhang, Dongyan Zhao125, and Rui Yan. 2021\.  Capturing Relations between Scientific Papers: An Abstractive Model for Related Work Section Generation.  (2021).
  * Cui and Hu (2021) Peng Cui and Le Hu. 2021\.  Topic-Guided Abstractive Multi-Document Summarization. In _Proc. of ACL_.
  * Deng et al. (2019) Yang Deng, Wai Lam, Yuexiang Xie, Daoyuan Chen, Yaliang Li, Min Yang, and Ying Shen. 2019.  Joint Learning of Answer Selection and Answer Summary Generation in Community Question Answering.  _Proc. of AAAI_ (2019).
  * Deng et al. (2020) Yang Deng, Wenxuan Zhang, and Wai Lam. 2020.  Multi-hop Inference for Question-driven Summarization. In _Proc. of EMNLP_.
  * Fu et al. (2020) Xiyan Fu, Jun Wang, Jinghan Zhang, Jinmao Wei, and Zhenglu Yang. 2020.  Document summarization with vhtm: Variational hierarchical topic-aware mechanism. In _Proc. of AAAI_.
  * Gao et al. (2019a) Shen Gao, Xiuying Chen, Piji Li, Zhangming Chan, Dongyan Zhao, and Rui Yan. 2019a.  How to Write Summaries with Patterns? Learning towards Abstractive Summarization through Prototype Editing. In _Proc. of EMNLP_.
  * Gao et al. (2019b) Shen Gao, Xiuying Chen, Piji Li, Zhaochun Ren, Lidong Bing, Dongyan Zhao, and Rui Yan. 2019b.  Abstractive Text Summarization by Incorporating Reader Comments. In _Proc. of AAAI_.
  * Gao et al. (2021) Shen Gao, Xiuying Chen, Zhaochun Ren, Dongyan Zhao, and Rui Yan. 2021.  From standard summarization to new tasks and beyond: summarization with manifold information. In _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_.
  * He et al. (2022) Wanwei He, Yinpei Dai, Min Yang, Jian Sun, Fei Huang, Luo Si, and Yongbin Li. 2022\.  Unified dialog model pre-training for task-oriented dialog understanding and generation. In _Proc. of SIGIR_.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  _Advances in neural information processing systems_ 28 (2015).
  * Huang et al. (2019) Chao Huang, Chuxu Zhang, Jiashu Zhao, Xian Wu, Dawei Yin, and Nitesh Chawla. 2019\.  Mist: A multiview and multimodal spatial-temporal learning framework for citywide abnormal event forecasting. In _Proc. of WWW_.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017.  OpenNMT: Open-Source Toolkit for Neural Machine Translation. In _Proceedings of ACL 2017, System Demonstrations_.
  * Koupaee and Wang (2018) Mahnaz Koupaee and William Yang Wang. 2018.  WikiHow: A Large Scale Text Summarization Dataset.  _ArXiv_ (2018).
  * Lee et al. (2020) Seanie Lee, Dong Bok Lee, and Sung Ju Hwang. 2020.  Contrastive learning with adversarial perturbations for conditional text generation.  _ICLR_ (2020).
  * Li et al. (2020) Mingzhe Li, Xiuying Chen, Shen Gao, Zhangming Chan, Dongyan Zhao, and Rui Yan. 2020\.  VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles. In _Proc. of EMNLP_.
  * Lima and Santos (2022) Rennan C Lima and Rodrygo LT Santos. 2022.  On Extractive Summarization for Profile-centric Neural Expert Search in Academia. In _Proc. of SIGIR_.
  * Lin (2004) Chin-Yew Lin. 2004\.  Rouge: A package for automatic evaluation of summaries.  _Text Summarization Branches Out_ (2004).
  * Liu et al. (2021) Junpeng Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojie Wang. 2021.  Topic-Aware Contrastive Learning for Abstractive Dialogue Summarization. In _Proc. of ACL_.
  * Liu et al. (2019) Luyang Liu, Heyan Huang, Yang Gao, Yongfeng Zhang, and Xiaochi Wei. 2019.  Neural variational correlated topic modeling. In _Proc. of WWW_.
  * Liu and Lapata (2019a) Yang Liu and Mirella Lapata. 2019a.  Hierarchical Transformers for Multi-Document Summarization. In _Proc. of ACL_.
  * Liu and Lapata (2019b) Yang Liu and Mirella Lapata. 2019b.  Text Summarization with Pretrained Encoders. In _Proc. of EMNLP_.
  * Liu and Liu (2021) Yixin Liu and Pengfei Liu. 2021.  SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization.  _arXiv preprint arXiv:2106.01890_ (2021).
  * Liu et al. (2022) Yang Liu, Chenguang Zhu, and Michael Zeng. 2022.  End-to-End Segmentation-based News Summarization. In _Proc. of ACL Findings_.
  * Mao et al. (2016) Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016\.  Generation and comprehension of unambiguous object descriptions. In _Proc. of CVPR_.
  * Mao et al. (2022) Qianren Mao, Hongdong Zhu, Junnan Liu, Cheng Ji, Hao Peng, Jianxin Li, Lihong Wang, and Zheng Wang. 2022\.  MuchSUM: Multi-channel Graph Neural Network for Extractive Summarization. In _Proc. of SIGIR_.
  * Miao et al. (2017) Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017\.  Discovering discrete latent topics with neural variational inference. In _Proc. of ICML_.
  * Nan et al. (2019) Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. 2019\.  Topic Modeling with Wasserstein Autoencoders. In _Proc. of ACL_.
  * Narayan et al. (2017) Shashi Narayan, Nikos Papasarantopoulos, Shay B Cohen, and Mirella Lapata. 2017.  Neural extractive summarization with side information.  _arXiv preprint arXiv:1704.04530_ (2017).
  * Qian et al. (2021) Xin Qian, Eunyee Koh, Fan Du, Sungchul Kim, Joel Chan, Ryan A Rossi, Sana Malik, and Tak Yeon Lee. 2021\.  Generating accurate caption units for figure captioning. In _Proc. of WWW_.
  * Röder et al. (2015) Michael Röder, Andreas Both, and Alexander Hinneburg. 2015\.  Exploring the space of topic coherence measures. In _Proc. of WSDM_.
  * Schluter (2017) Natalie Schluter. 2017\.  The limits of automatic summarisation according to rouge. In _Proc. of EACL_.
  * Shin et al. (2019) Kijung Shin, Amol Ghoting, Myunghwan Kim, and Hema Raghavan. 2019.  Sweg: Lossless and lossy summarization of web-scale graphs. In _Proc. of WWW_.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision. In _Proc. of CVPR_.
  * Tan and Le (2019) Mingxing Tan and Quoc Le. 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks. In _Proc. of ICML_.
  * Tolstikhin et al. (2017) Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. 2017.  Wasserstein auto-encoders.  _arXiv preprint arXiv:1711.01558_ (2017).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention Is All You Need. In _Proc. of NIPS_.
  * Wang et al. (2018) Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. 2018\.  Reconstruction network for video captioning. In _Proc. of CVPR_.
  * Wang et al. (2022) Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022\.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In _Proc. of ICML_.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_ (2016).
  * Xie et al. (2021) Qianqian Xie, Jimin Huang, Pan Du, Min Peng, and Jian-Yun Nie. 2021.  Graph topic neural network for document representation. In _Proc. of WWW_.
  * Xu et al. (2020) Song Xu, Haoran Li, Peng Yuan, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020\.  Self-Attention Guided Copy Mechanism for Abstractive Summarization. In _Proc. of ACL_.
  * Yan et al. (2019) Chenggang Yan, Yunbin Tu, Xingzheng Wang, Yongbing Zhang, Xinhong Hao, Yongdong Zhang, and Qionghai Dai. 2019.  STAT: Spatial-temporal attention mechanism for video captioning.  _IEEE transactions on multimedia_ (2019).
  * Yang et al. (2020) Liang Yang, Fan Wu, Junhua Gu, Chuan Wang, Xiaochun Cao, Di Jin, and Yuanfang Guo. 2020.  Graph attention topic modeling network. In _Proc. of WWW_.
  * Yang et al. (2019) Zonghan Yang, Yong Cheng, Yang Liu, and Maosong Sun. 2019\.  Reducing word omission errors in neural machine translation: A contrastive learning approach.  (2019).
  * Ye et al. (2020) Wei Ye, Rui Xie, Jinglei Zhang, Tianxiang Hu, Xiaoyin Wang, and Shikun Zhang. 2020\.  Leveraging code generation to improve code retrieval and summarization via dual learning. In _Proc. of WWW_.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020.  BERTScore: Evaluating Text Generation with BERT. In _Proc. of ICLR_.
  * Zhao et al. (2019) Zhou Zhao, Haojie Pan, Changjie Fan, Yan Liu, Linlin Li, Min Yang, and Deng Cai. 2019.  Abstractive meeting summarization via hierarchical adaptive segmental network learning. In _Proc. of WWW_.
  * Zhou et al. (2021) Hao Zhou, Weidong Ren, Gongshen Liu, Bo Su, and Wei Lu. 2021.  Entity-Aware Abstractive Multi-Document Summarization. In _Proc. of ACL_.
  * Zhu et al. (2021) Chenguang Zhu, Ziyi Yang, Robert Gmyr, Michael Zeng, and Xuedong Huang. 2021.  Leveraging lead bias for zero-shot abstractive news summarization. In _Proc. of SIGIR_.
  * Zhu et al. (2018) Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2018\.  MSMO: Multimodal summarization with multimodal output. In _Proc. of EMNLP_.
  * Zhu et al. (2020) Junnan Zhu, Yu Zhou, Jiajun Zhang, Haoran Li, Chengqing Zong, and Changliang Li. 2020\.  Multimodal summarization with guidance of multimodal reference. In _Proc. of AAAI_.