# Informative Text Generation from Knowledge Triples

Zihao Fu†, Yijiang River Dong♣, Lidong Bing‡, Wai Lam♠
†Language Technology Lab, TAL, University of Cambridge, ♣University of
Pennsylvania
‡DAMO Academy, Alibaba Group, ♠The Chinese University of Hong Kong
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As the development of the encoder-decoder architecture, researchers are able
to study the text generation tasks with broader types of data. Among them, KB-
to-text aims at converting a set of knowledge triples into human readable
sentences. In the original setting, the task assumes that the input triples
and the text are exactly aligned in the perspective of the embodied
knowledge/information. In this paper, we extend this setting and explore how
to facilitate the trained model to generate more informative text, namely,
containing more information about the triple entities but not conveyed by the
input triples. To solve this problem, we propose a novel memory augmented
generator that employs a memory network to memorize the useful knowledge
learned during the training and utilizes such information together with the
input triples to generate text in the operational or testing phase. We derive
a dataset from WebNLG for our new setting and conduct extensive experiments to
investigate the effectiveness of our model as well as uncover the intrinsic
characteristics of the setting.

##  1 Introduction

Text generation is a longstanding NLP task, and congenitally it focuses on
addressing two questions separately: what to say and how to say Reiter and
Dale (1997); Jurafsky and Martin (2000). Accordingly, earlier systems usually
have three components: content selection, macro/micro-planning, and surface
realization. The recent development of the encoder-decoder architecture allows
researchers to perform text generation from richer input data. The KB-to-text
generation problem, e.g. the WebNLG task Gardent et al. (2017a, b), aims at
directly converting a set of KB triples into human-readable sentences. For
example, given the knowledge triples ( $<$ Bill Gates, Birthplace, Seattle
$>$, $<$ Bill Gates, FounderOf, Microsoft $>$), the goal is to generate a
comprehensible sentence such as “Bill Gates, the founder of Microsoft, was
born in Seattle.”

Figure 1: The proposed task: generating text based on the input triples
together with the information memory.

In the current KB-to-text setting, it assumes that the input data (i.e.
triples) and the output text are exactly aligned. In other words, the
generated text only contains the information conveyed by the input triples.
However, this setting is somewhat rigid and less practical. When human beings
interpret the data via natural language to others, they are prone to adding
extra information according to some common experience to make the sentence
more comprehensive. For example, when humans are explaining the triple $<$Bill
Gates, FounderOf, Microsoft$>$, they may add extra words to tell that the
headquarter of Microsoft is exactly the place where Gates was born. Thus the
whole text is more informative especially for those readers who know less
about Bill Gates or Microsoft. Therefore, assuming that the data and the text
are exactly aligned mismatches with how humans usually write sentences.

In this paper, we propose a new problem named Informative Text Generation
(ITG). It extends the current KB-to-text task to a setting that requires to
generate text from an input triple set together with a information repository.
It should generate more informative text containing more information about the
triple entities that is not conveyed by the input triples. As shown in Fig. 1,
when the input triple is “$<$Bill Gates, FounderOf, Microsoft$>$”, the ITG
task is to generate a sentence as exemplified by “Bill Gates, the founder of
Microsoft, was born on 28 October 1955 in Seattle, the headquarter of
Microsoft.” Note that though the generated text may not be exactly the same as
the example text, it should be a sentence containing more information about
“Bill Gates” and “Microsoft” in addition to the fact that Bill Gates is a
founder of Microsoft. Our new setting requires the trained model to retrieve
relevant knowledge automatically from the memory and generate text containing
some extra narrative that is not conveyed in the input triples. Therefore, the
generated sentences are more informative. Obviously, existing data-to-text
models cannot handle our new problem.

In order to solve the ITG problem, we propose a novel information Memory
Augmented Generator (iMAG). During the training, the model randomly drops some
input triples and store the information that is conveyed in the text but not
corresponding to any remaining triples as a background knowledge into the
information memory. It also learns to retrieve this kind of knowledge
automatically from the memory. Afterwards, the model will be able to generate
text with the presence of the memory providing extra knowledge about the input
entities in the operational or testing stage. Specifically, our model is
composed of two components: the information memory and the sequence-to-
sequence (seq2seq) network. Within the information memory component, part of
the input triples is dropped randomly in the training phase making the target
sentence containing more information than the remaining input triples. Then,
we design a querier that can fetch the related information from the memory
according to the remaining input triples. The seq2seq network generates output
sentences by considering both the fetched information and the remaining input
triples. To avoid the tendency that the fetched information may repeat some
information conveyed by the input triples, we propose several ways to suppress
the repetitions. We derive a dataset from the WebNLG dataset suitable for
conducting experiments for our problem setting . If needed, other datasets can
be derived in a similar way from a source dataset with similar nature. We
compare our proposed model with some existing models adapted to this new
setting. The experimental results show that our model has a better capability
of capturing the extra information encountered in the training data and thus
generates more informative text.

##  2 Related Works

Recently, a few data-to-text tasks have been proposed which aim to generate
text from some formatted data such as tables and knowledge triples. The WebNLG
task Gardent et al. (2017a, b); Tran and Nguyen (2020) generates sentences
corresponding to a set of related triples sampled from DBpedia Auer et al.
(2007); Lehmann et al. (2015). Lebret et al. (2016) propose to generate
people’s short biography from Wikipedia infobox. Novikova et al. (2017)
propose to generate restaurants’s reviews based on corresponding attributes.
Wiseman et al. (2017) and Chen and Mooney (2008) generate match summaries for
ball games, while Liang et al. (2009) generate weather forecast. Our task in
this paper is an extension of this setting, particularly WebNLG (other dataset
can be adopted easily in a similar manner). The difference is that these
existing settings do not generate text containing extra knowledge that is not
mentioned in the input data.

The memory mechanism Weston et al. (2014); Graves et al. (2014); Sukhbaatar et
al. (2015) have been proposed in many Neural Machine Translation (NMT)
systems. Wang et al. (2016); Meng et al. (2018) propose to use memory to
enhance the decoder in the seq2seq framework. Miller et al. (2016) propose
key-value memory networks to read and comprehend documents in a QA framework
while Chen et al. (2019); Yang et al. (2019) propose to utilize specific
external knowledge triples and ConceptNet explicitly to help generate which is
restricted to specific areas. Shen et al. (2019) propose to use a pragmatic
method to ensure that the information is fully conveyed. We propose to utilize
an embedded memory to add more relevant descriptions.

It has been observed in many works Mi et al. (2016); Tu et al. (2016); Wu et
al. (2016); Holtzman et al. (2019) that many generation systems suffer from
the content-repeating problem. Actually, this problem occurs because the
attention focuses on the same region many times. Mi et al. (2016) and Tu et
al. (2016) propose the coverage mechanism to handle this problem while Meng et
al. (2018) propose to use memory-based attention to eliminate the content-
repeating problem. However, in our problem, the content-repeating problem is
mainly caused by the information overlapping between the input source and the
information memory. Therefore, simply averaging the attention cannot handle
our problem.

##  3 The Proposed Model

We use $h$, $r$ and $t$ to denote the head entity, the relation and the tail
entity respectively in a knowledge triple. Given an input triple set with $c$
triples $G=\\{<h_{1},r_{1},t_{1}>,\cdots,<h_{c},r_{c},t_{c}>\\}$, our
information Memory Augmented Generator (iMAG) aims at generating a text
sequence $S=(s_{1},s_{2},\cdots,s_{m})$ that maximizes the conditional
probability given the input triple set $G$ and the information memory $M$:

$\max_{\theta,\psi}p_{\theta}(S|G,M_{\psi}),$ (1)

where $\theta$ represents all learnable parameters of the network and $\psi$
stands for the memory parameters.

As illustrated in Fig. 2, our model contains two components. In the
Information Memory component, a memory matrix $V$ with each column $V_{i}$ as
a memory slot is designed to memorize the extra information during the
training. Given an input triple set $G$, the model firstly removes part of the
input triples by a remaining ratio. Then, it generates a querier matrix $Q$ by
sliding a CNN window over the flattened input triples to get the querying
vector $Q_{j}$ for the $j$th window. We get the relevant score $W_{j}$ of each
memory slot for the $j$th window and get the relevant knowledge vector $C_{j}$
by a weighted sum of the memory slots. The relevant knowledge matrix $C$
composed of all $C_{j}$ as its columns is then concatenated to the embedding
of the original input sequence and fed into the subsequent component. A
sequence-to-sequence network first encodes the sequence into hidden vectors
and then its decoder generates the output text. The attention Bahdanau et al.
(2014) and copy Gu et al. (2016) mechanisms are used to enhance the
performance. Since the information memory may return hidden representations
containing the same information with the input triples and resulting in the
content-repeating problem, we design several variants of iMAG to alleviate the
problem, such as the repeating sentence penalty (i.e. $\mathcal{L}_{rsp}$ at
the top of Fig. 2)

Figure 2: Framework of our model. We omit some components such as attention
and copy for clarity.

###  3.1 Information Memory

Given an input triple set, the information memory firstly removes some triples
according to a preserving ratio $\zeta$ in the training phase (it does not
remove anything when testing). Then, it flattens the remaining triples as a
word sequence: $G^{\prime}=[h_{1}^{(1)},\cdots,$
$h_{1}^{(n_{h_{1}})},r_{1}^{(1)},\cdots,r_{1}^{(n_{r_{1}})},t_{1}^{(1)},\cdots,t_{1}^{(n_{t_{1}})},\cdots,h_{c^{\prime}}^{(1)},\cdots,$
$h_{c^{\prime}}^{(n_{h_{c^{\prime}}})},r_{c^{\prime}}^{(1)},\cdots,r_{c^{\prime}}^{(n_{r_{c^{\prime}}})},t_{c^{\prime}}^{(1)},\cdots,t_{c^{\prime}}^{(n_{t_{c^{\prime}}})}]$,
where $h_{i}^{(j)}$ stands for the $j$th word of the head entity in the $i$th
remaining triple in $G$. $n_{h_{i}}$ stands for the number of words in the
$i$th remaining entity $h_{i}$ and $c^{\prime}$ is the number of the remaining
triples. Hereafter, we denote $G^{\prime}=[x_{1},x_{2},\cdots,x_{n}]$ for
clarity in which $x_{i}$ stands for each word in $G^{\prime}$ and $n$ is the
word count in $G^{\prime}$. Then $G^{\prime}$ is mapped into vectors by an
embedding layer, $E=Emb(G^{\prime})$, where $E\in\mathbb{R}^{e\times n}$ while
$e$ is the embedding dimension.

We extract the feature of each segment of the sequence by applying a CNN
module on the embedding $E$, namely, $Q=CNN(E)$. The kernel size of CNN is set
to $e\times k$ while $k$ is the kernel length. Therefore, the CNN module
encodes each window of $k$ input words into a hidden representation and use
them to retrieve a memory vector related to those words. The $j$th column of
the matrix $Q\in\mathbb{R}^{e\times(n-k+1)}$ represents the query vector of
the $j$th window. $Q$ will be used as a querier to get the relatedness of each
memory slot by a matrix product as $\tilde{W}=K^{T}Q$, where
$K\in\mathbb{R}^{e\times l}$ is a learnable matrix used to calculate the
relatedness of each memory slot and $l$ is the total memory slot number which
is design to be a tunnable hyperparameter.
$\tilde{W}\in\mathbb{R}^{l\times(n-k+1)}$ is the relatedness of $n-k+1$
windows to the $l$ memory slot, it will be further normalized with a softmax
layer as:

$W_{ij}=\frac{\exp{(\tilde{W}_{ij}})}{\sum_{i=1}^{l}\exp{(\tilde{W}_{ij})}},$
(2)

in which $W\in\mathbb{R}^{l\times(n-k+1)}$ has the same size as $\tilde{W}$.
The only difference is that each column of $W$ sums up to $1$ and can be
therefore regarded as the probability distribution of each memory slot.

With the probability of each memory slot, the relevant knowledge matrix can be
calculated by a weighted sum as $C=V^{T}W$, in which $V\in\mathbb{R}^{l\times
e}$ is the memory matrix. $C\in\mathbb{R}^{e\times(n-k+1)}$ is the weighted
sum of the memory values while the $j$th column $C_{i}$ is the relevant
knowledge for the $j$th sliding window. The output of the information memory
is a simple concatenation of the original input embedding $E$ and the memory
information $C$, namely, $X=[E;C]$, in which
$X\in\mathbb{R}^{e\times(2n-k+1)}$ will be fed into the subsequent seq2seq
network. It contains the information of both the input triples and the fetched
memory and thus it can generate sentences with more information than merely
the input triples.

###  3.2 Sequence-to-Sequence Network

The seq2seq takes $X$ as input. We use attention to enhance the network’s
expression capability. In the encoder, $X$ is fed into an LSTM layer to get
the hidden representation: $H=LSTM(X)$, in which
$H\in\mathbb{R}^{e\times(2n-k+1)}$ has the same size with $X$. We denote
$h_{-1}$ as the last hidden vector of $H$. $h_{-1}$ aggregates the information
from the entire input sequence and thus can be regarded as the context vector.

In the decoder, the hidden state $u_{t}=LSTM([u_{t-1};z_{t-1}])$ is calculated
based on the last output $z_{t-1}\in\mathbb{R}^{e}$ and the last hidden state
$u_{t-1}$, where $u_{1}=h_{-1}$. $u_{t}$ is the hidden representation of the
generated sequence, and it will be refined by an attention layer together with
the input hidden representation $H$. The calculation is denoted as
$z_{t}=Attn(u_{t},H)$, where $Attn$ is the attention function Bahdanau et al.
(2014) which considers $u_{t}$ and its relation to each $H_{i}$ to calculate
$z_{t}$. The word probability distribution $y_{t}$ is calculated as
$y_{t}=\text{softmax}(M_{p}z_{t}+b_{p})$, in which $y_{t}\in\mathbb{R}^{|V|}$
and $|V|$ is the vocabulary size. $M_{p}\in\mathbb{R}^{|V|\times e}$ is a
transformation matrix and $b_{p}\in\mathbb{R}^{|V|}$ is the bias vector. In
the training, our model minimizes the following negative log likelihood loss:

$\mathcal{L}_{s2s}=-\sum_{t=1}^{T}\log y_{t}[s_{t}],$ (3)

in which $s_{t}\in S$ is the $t$th gold standard target word and
$y_{t}[s_{t}]$ denotes the $s_{t}$th element of $y_{t}$. We also apply the
copy mechanism He et al. (2017) to enhance the model capability.

###  3.3 Variants for Handling Content-Repeating

As mentioned above, the fetched information from the memory may repeat some
information of the input triples, which causes the content-repeating problem.
We investigate several variants of our core model iMAG to solve this problem.

Repeating Sentence Penalty (RSP). The output token sequence $\\{t_{i}\\}$ is
calculated by taking the maximum probability of $y_{i}$, namely
$t_{i}=\arg\max(y_{i})$. Then, the start and the end of the last Longest
Repeating Non-overlapping Substring (LRNS) are calculated by applying the LRNS
algorithm on $\\{t_{i}\\}$. LRNS is calculated as:

$\begin{split}LRN&S(S)\overset{\Delta}{=}\arg\max_{p,q}(q-p)\\\ s.t.\ \ \
&1\leq p<q\leq m\\\ &k<2p-q\\\ &S[p+r]=S[k+r]\ \ \ 0\leq r\leq
q-p,\end{split}$ (4)

in which $m$ is the length of the output sequence $S$. $p,q\in\mathbb{N}$ are
index variables representing the start and end indexes for the LRNS algorithm.
It can be solved with dynamic programming with the complexity of
$\mathcal{O}(n^{2})$.

Then the repeating sentence loss $\mathcal{L}_{rsp}$ is defined as the sum of
the probability of each repeating token found by the LRNS algorithm:

$\mathcal{L}_{rsp}=\sum_{r=p}^{q}y_{r}[t_{r}],$ (5)

where $y_{r}[t_{r}]$ is the probability value of $t_{r}$ in the distribution
$y_{r}$. Finally, the total loss of this variants is defined as the weighted
sum of $\mathcal{L}_{s2s}$ and $\mathcal{L}_{rsp}$:

$\mathcal{L}=\mathcal{L}_{s2s}+\alpha\mathcal{L}_{rsp},$ (6)

in which $\alpha$ is a weight coefficient.

Repeating Word Penalty (RWP). Instead of penalizing for the repeated substring
as described above, we can also penalize for the repeating words. This is
inspired by the observation that in an ordinary sentence, a word may not
appear too many times. In a generated sequence, each word will have a
probability. We accumulate the probability of each word in the whole sequence
and penalize it if the probability sum of a certain word is greater than a
threshold $\gamma$. The loss can be expressed as:

$\mathcal{L}_{rwp}=\mathbf{1}^{T}\max(\mathbf{0},\sum_{i=1}^{m}y_{i}-\gamma\mathbf{1}),$
(7)

in which $\mathbf{0}$ and $\mathbf{1}\in\mathbb{R}^{|V|}$, $\max(\cdot,\cdot)$
returns a vector that takes the element-wise maximum of the two input vectors.

Reinforcement Learning (RL). RL algorithm can optimize the parameters
according to the model performance, no matter whether it is differentiable. We
can use the LRNS score as the reward function. Similar to Yu et al. (2017);
Narayan et al. (2018), we apply RL to the generated sequence. Specifically,
the REINFORCE algorithm Williams (1992) is used to calculate the gradient of
the parameters based on the probability of each output word and the LRNS
length ratio. The gradient can be expressed as:

$\nabla_{\theta,\psi}\mathcal{L}_{RL}(\theta,\psi)=\mathbb{E}[\nabla_{\theta,\psi}\sum_{i=1}^{m}\log
y_{i}[t_{i}]\frac{m-q+p}{m}],$ (8)

in which $p$ and $q$ are calculated by the LRNS algorithm and we use the
length ratio $\frac{m-q+p}{m}$ as the reward function of the RL algorithm.

Coverage (CVG). We employ the coverage mechanism Tu et al. (2016); Gehrmann et
al. (2018) which is capable of solving the repetition problem in neural
machine translation systems. It averages the attention in the decoder input
and thus suppresses generating repeating content.

###  3.4 Learning and Prediction

The training of our core model iMAG is straightforward, however, there are two
major problems when training its variants. Firstly, it suffers from the
exposure bias problem Ranzato et al. (2016); Paulus et al. (2018). In the
traditional seq2seq framework, the output is generated in a teacher-forced
manner. Precisely, the ground-truth word in the training sequence at each time
step, instead of the predicted one, is fed into the next time step. The
benefit is that it can avoid error propagation and enable parallel training.
However, it hides the content-repeating problem in the training stage.
Consequently, it makes the model unable to deal with this problem in testing
since it has not learned that capability. Secondly, if we train the model
without the teacher-forced manner, the procedure becomes impossible to be
parallelized since each step needs the output of the previous step, which
makes the training extremely slow.

In order to overcome the above dilemma, we train the model in two separate
stages. In the first stage, we train the seq2seq network without the repeating
sentence penalty in a teacher-forced way. After certain epochs, we switch the
training to using the repeating sentence penalty and removing the teacher
forcing.

##  4 Experiments

###  4.1 Dataset

We adapt WebNLG v2 111 https://gitlab.com/shimorina/webnlg-dataset and derive
a new dataset suitable for the new problem setting. It simply resamples the
training/testing partition and can thus be easily extended to other datasets
in a similar manner. For the training data, we draw (triples, text) pairs that
contain at least two triples to make it possible to drop some triples. There
is no gold standard target text for these triples since we hope the model can
decide the most suitable knowledge to retrieve and present. Therefore, the
models will be evaluated by a pseudo target sentence and human evaluation
which is discussed in Sec. 4.3.

| Train | Dev | Test
---|---|---|---
# (triples, text) pairs | 33,608 | 706 | 1,501
Avg. triples per pair | 1.512 | 1.508 | 1.512
Avg. target length | 27.23 | - | -
Table 1: Statistics of our dataset. Length is measured by word count. | RL | PL | FL | R2 | P2 | F2 | RSU4 | PSU4 | FSU4 | LEN | LRNSR$\downarrow$ | DRATE$\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---
S2S | 0.234 | 0.538 | 0.317 | 0.164 | 0.399 | 0.225 | 0.147 | 0.407 | 0.206 | 23.7 | 0.084 | 0.807
S2SF | 0.187 | 0.713 | 0.290 | 0.134 | 0.570 | 0.210 | 0.104 | 0.600 | 0.170 | 13.4 | 0.056 | 0.916
iMAG | 0.281 | 0.428 | 0.335 | 0.192 | 0.299 | 0.230 | 0.186 | 0.301 | 0.226 | 35.4 | 0.120 | 0.670
iMAG+CVG | 0.288 | 0.420 | 0.336 | 0.195 | 0.289 | 0.229 | 0.189 | 0.292 | 0.225 | 37.8 | 0.129 | 0.649
iMAG+RL | 0.269 | 0.436 | 0.327 | 0.182 | 0.301 | 0.223 | 0.177 | 0.309 | 0.220 | 33.3 | 0.111 | 0.693
iMAG+RWP | 0.276 | 0.430 | 0.331 | 0.187 | 0.298 | 0.226 | 0.181 | 0.302 | 0.221 | 34.5 | 0.112 | 0.690
iMAG+RSP | 0.271 | 0.441 | 0.330 | 0.185 | 0.309 | 0.227 | 0.178 | 0.314 | 0.222 | 33.5 | 0.102 | 0.706
iMAG w/o memory | 0.135 | 0.486 | 0.206 | 0.073 | 0.287 | 0.114 | 0.065 | 0.334 | 0.105 | 14.5 | 0.063 | 0.886
iMAG w/o source | 0.168 | 0.452 | 0.240 | 0.075 | 0.210 | 0.108 | 0.078 | 0.262 | 0.116 | 21.9 | 0.110 | 0.729
Table 2: Main results on our ITG dataset. $\downarrow$: smaller is better,
$\uparrow$: larger is better.

###  4.2 Comparison Models and Our Variants

We compare our model with two baselines. S2S uses vanilla seq2seq Sutskever et
al. (2014); Cho et al. (2014) framework with the standard attention and copy
mechanisms. S2SF is the same as S2S, but utilizes the original WebNLG without
removing any triples for training. For testing and development, it uses the
same data as other methods. We also examine different variants of our model.
iMAG is the core model of our framework that contains the information memory
without any component dealing with the content-repeating problem, while
iMAG+CVG, iMAG+RL, iMAG+RWP, and iMAG+RSP are variants that employ the methods
presented in Sec. 3.3 to deal with this problem. We also conduct two ablation
experiments. iMAG w/o memory employs our iMAG model at the training phase, but
for testing, the information memory is deliberately omitted. Therefore, this
model allows us to examine how much information can be generated only with the
input triples. iMAG w/o source also has the same training phase as iMAG, but
for testing, its seq2seq network excludes the input triples and only uses the
information memory (of course, queried with the input triples). Therefore,
this model allows us to examine whether the memory can give correct
information about the input entities.

###  4.3 Experimental Setup

Evaluating the generated text is challenging since there are no gold standard
target sentences. We propose to evaluate the models by comparing with the
pseudo target sentence, reporting several automatic evaluation metrics, and
conducting a human evaluation. The pseudo target sentence is made from those
1-triple data samples in the original WebNLG dataset. Each data sample
contains one triple and one corresponding sentence and all of them have not
been included in the training set. The pseudo target sentence is made by
concatenating all 1-triple sentences describing any of the entities in the
given input triple set. For example, if the entity “Bill Gates” and
“Microsoft” are included in the input triple set, the pseudo target sentence
will concatenate all 1-triple sentences describing them as “Bill Gates was
born in Seattle. Bill Gates is a US citizen. Microsoft was founded in Seattle.
…”. Several metrics are employed for the comparison, namely ROUGEL, ROUGE2,
ROUGESU4, LEN, LRNSR, and DRATE. In the ROUGE metric Lin (2004), P, R and F
(precision, recall and F-value) are all reported. LEN denotes the average
length of the output. LRNSR stands for LRNS Ratio and shows how much
duplicated information is included in each sentence. It is defined as the
length of the LRNS divided by the length of the sentence. DRATE stands for
distinctness rate which is defined as the ratio of the number of distinct
words over the sentence length. The smaller the metric is, the less
information is contained in the sentence. Note that instead of BLEU, we report
ROUGE-P as the precision-oriented metric because the reference is too long and
the brevity penalty is close to 0 making the BLEU scores close to 0. We keep
all the hyperparameters in Klein et al. (2017) unchanged. We use grid search
to tune hyperparameters on the development set. We choose CNN kernel size
$k=3$ from $\\{2,3,5,7\\}$, memory size $l=600$ from
$\\{200,400,600,800,1000\\}$, $\alpha=0.5$ from $\\{0.2,0.5,1.0,2.0\\}$, and
$\zeta=0.5$ from $\\{0.1,0.2,0.5,0.7,0.9\\}$. We iterate 20K batches (50
samples each batch) with teacher forcing mechanism and then remove such
mechanism and train for another 40K batches which is choosed from
$\\{10K/50K,20K/40K,30K/30K,40K/20K,$ $50K/10K\\}$. It takes 3.8 hours to
train the model.

###  4.4 Experimental Results

The experimental results are shown in Table 2. It can be concluded from the
results that our proposed models can generate valid sentences containing
informative knowledge. The F values show that in general, the iMAG model can
generate more favorable text which has a better balance between precision and
recall. Considering the R values and the LEN values together, we can conclude
that the advantage in F values attributes to the contribution of our
information memory. It indeed memorizes useful knowledge in training and helps
generate longer and meaningful text at the cost of affecting some precision.
Another cause of the relatively lower precision is that the pseudo target
sentence is concatenated by 1-triple sentences which may be different from the
generated style. On the other hand, we find that S2SF achieves the best
precision but with the smallest LEN. The reason is that S2SF is trained with
strictly aligned (triples, text) pairs and it will not output additional
information about the entities during testing, thus it has less chance to get
wrong. It can also be observed that the S2S and iMAG model generate longer
sentences than S2SF. This is because, in the training phase, S2S and iMAG
models are trained to retrieve related information from the memory while S2SF
model does not learn such capability. As a result, the S2SF model can only
generate text exactly describing the input triple set when testing.

| Plausibility | Grammaticality | Richness
---|---|---|---
S2S | 6.05 | 6.01 | 5.84
S2SF | 6.11 | 6.05 | 5.32
iMAG + RSP | 5.84 | 6.05 | 7.02
Table 3: Human Evaluation.

LRNSR and DRATE are two metrics that can more intuitively illustrate the
characteristics of the generated text. While iMAG outputs longer and
informative text, it also produces more repetitions. In the second part of
Table 2, we examine the effect of those four variants for alleviating the
content-repeating problem. The results of iMAG+CVG show that the coverage
mechanism performs poorly. Precisely, both LRNSR and DRATE become worse. This
is because the fetched memory information may have the same information as the
input triples, averaging attention to each hidden vector of the encoder cannot
help solve the content-repeating problem caused by the repetitions in the
encoder input sequence. The results of iMAG+RL show that the reinforcement
learning (RL) module here can help solve the problem. However, the improvement
is not significant because the RL module depends on the sampling of the output
sequences. If the probability is low, the gradient is also very small and thus
it is hard to converge.

iMAG+RWP and iMAG+RSP, which are the two variants directly penalizing the
repetitions, are more effective in solving the problem. iMAG+RSP performs
better, and its LRNSR value drops to 0.102 while DRATE increases to 0.706.
iMAG+RWP penalizes nearly all repeating words, and thus it is sometimes too
harsh since high-frequency words can appear multiple times and should not be
penalized. iMAG+RSP, penalizing on the longest repeating common substring, is
a more favorable solution.

The third part of Table 2 gives the results of the two ablations. It shows
that the trained integrated model will not perform well if one type of the
input is removed. The results of “iMAG w/o memory” show that without the
information memory, the model can only generate a short sentence. Meanwhile,
its LRNSR measure drops dramatically to 0.063 and the DRATE increases to
0.886, which confirms that the repeating is mainly caused by the fact that the
fetched memory repeats some information of the input triples.

| CR | PC | CC | AR
---|---|---|---|---
S2S | 0.819 | 0.658 | 0.879 | 1.38
S2SF | 0.838 | 0.528 | 0.852 | 0.73
iMAG + RSP | 0.803 | 0.676 | 0.889 | 1.78
Table 4: Acquisition analysis. | RL | PL | FL | LEN | LRNSR
---|---|---|---|---|---
S2S | 0.148 | 0.479 | 0.223 | 15.2 | 0.074
S2SF | 0.117 | 0.757 | 0.198 | 7.68 | 0.119
iMAG | 0.183 | 0.478 | 0.261 | 21.2 | 0.079
iMAG+RSP | 0.183 | 0.479 | 0.260 | 21.0 | 0.077
Table 5: Results of entity biography generation. Input Entity |  Sentence
---|---
Allen Forrest |  Allen Forrest is a solo singer who plays acoustic music. He was born in Fort Campbell, KY and was raised in Dothan, AL .
109 Felicitas |  109 Felicitas, which has an epoch date of December 31st 2006, has a periapsis of 283326000000.0 and an orbital period of 139705000.0 .
Alhambra |  The Alhambra has a ship beam of 8.3m and is 63800.0 millimetres long. It was launched on 31st May 1855.
Table 6: Case study for entity biography generation.

###  4.5 Human Evaluation

To further evaluate the generation quality, we conduct a human evaluation
focusing on plausibility, grammaticality, and richness. Specifically,
plausibility refers to the correctness of the generated sentence. The
grammaticality refers to the grammatical correctness while the richness refers
to how much new correct information is added. Human raters are asked to score
the generated sentences from 1 to 10 in three different perspectives. The
results are shown in Table 3. It can be concluded that the iMAG+RSP
outperforms S2S and S2SF models significantly in richness showing that our
proposed model is capable of generating more informative text. The
plausibility of the iMAG+RSP model falls a little behind other models since
generating more content makes it easier to make mistakes. However,
quantitatively, the hallucination generation problem Tian et al. (2019) is not
much severe than other models showing the model can retrieve correct
knowledge. The grammaticality scores are almost the same for all models
showing that all of these models can generate grammatically correct sentences.

Model |  Sentence
---|---
S2S |  Aaron Turner played with the bands Twilight and Old Man Gloom .
S2SF |  Aaron Turner played with the band Twilight .
iMAG |  Aaron Turner was born in the United States and started performing in 1995 . He played with the band Twilight and is associated with the group Greymachine . He played with the band Twilight .
iMAG+ RSP |  Aaron Turner is an artist for the band Isis and is associated with the group Greymachine . He played with the band Twilight , the musical genre of which , is black metal .
Table 7: Case study for the input “$<$Aaron Turner, associatedBand, Twilight (
band ) $>$”. Blue font: information not mentioned in the input triple. Orange
font: repeating sentence.

###  4.6 Acquisition Analysis

In order to quantitatively evaluate how much knowledge and which knowledge can
be obtained from the memory, we conduct an acquisition analysis as shown in
Table 4. Given a generated sentence corresponding to an input triple set, we
say it covers a 1-triple data sample if it contains 80% of the words in the
1-triple sentence. We analyze the relevance between the input triple’s
relations and the relations covered by the generated text. We denote
$r_{j}^{i}$ as the $i$th relation in the $j$th input triple set and
$\hat{r}_{j}^{k}$ as the $k$th covered relation for the $j$th generated
sentence. The co-occurrence ratio (CR) is defined as the portion of the
$(r_{j}^{i},\hat{r}_{j}^{k})$ pairs that co-exist in any of the input triple
set of the training set. The pair correlation (PC) is defined as the
correlation coefficient between the count of the pair
$(r_{j}^{i},\hat{r}_{j}^{k})$ and the count of its co-occurrence in the
training input triple set. The count correlation (CC) is defined as the
correlation coefficient between the count of the relation $\hat{r}_{j}^{k}$
and the count of it in the training input triples. All of the CR, PC, CC
metrics give a quantitative analysis of which relation is preferred when
retrieving from the memory. It can be concluded from the results that if two
relations are closely connected in the training set, the model is more likely
to retrieve one relation from the memory if the other is given. The
acquisition ratio (AR) is defined as the count of the covered new relations
which does not contain the relations in the input triple set, over the count
of the input triples. It gives an intuition of how much knowledge on average
the model will retrieve from the memory. From the result, we can observe that
the iMAG+RSP model retrieves more relations than other models.

###  4.7 Entity Biography Generation

To show that our model is capable of storing useful information into the
memory, we experiment on generating an entity’s biography by only inputting
the entity name. The numerical results are shown in Table 5 while some
generated example biographies are shown in Table 6. Both iMAG and iMAG+RSP can
generate longer descriptions than that generated by S2S and S2SF. The reason
is the same as above. Concretely, S2S and S2SF heavily depend on the
information from the input triples while our model is capable of utilizing the
information from the memory. Moreover, our proposed model can generate real
background knowledge about the given entities instead of hallucination
generation Tian et al. (2019).

Compared with Table 2, the PL values of iMAG and iMAG+RSP in Table 5 are
higher. The reason is that for this experiment, we do not have the relation
name as part of the input which may fetch some information about the relation
of other entities, while the reference here is only about the input entity. In
Table 5, all the metrics for iMAG and iMAG+RSP are very similar. The reason is
that we only have the entity name as input and thus the information redundancy
in the input sequence (i.e. containing similar information with the fetched
memory) of the seq2seq network does not exist anymore. Table 6 shows the
generated biographies of three entities. We can see that each biography
contains rich information about the entity. In the original data, such
information exists as multiple small sentences and here they are reasonably
merged into longer and well-formatted sentences.

###  4.8 Case Study of Generated Text for Triples

In order to provide a more intuitive understanding of how our models are
capable of generating sentences with extra useful information, we analyze the
output result of the input triple “$<$Aaron Turner, associatedBand, Twilight (
band ) $>$”. The triple only gives the basic information of the artist and the
models are trying to give more background knowledge about Aaron Turner and
Twilight. Table 7 shows the generations of different models.

It can be observed from Table 7 that instead of hallucination generation Tian
et al. (2019), our proposed models iMAG and iMAG+RSP can successfully add
useful extra information about the entities in the given triple. They infer
from the memory that Aaron Turner is in Isis and associated with the group
Greymachine, the musical genre of Twilight is black metal, etc. On the other
hand, the S2S model can also output some extra information that was encoded in
the network. However, due to the deprivation of using auxiliary modules to
store the information, S2S fails to provide extra messages about Aaron Turner.
The S2SF model fails to introduce extra information for the entities, because
it has neither been trained to add information nor does it have any extra
component to store the extra information. We also observe that the iMAG model
suffers from the content-repeating problem, while with the help of RSP,
iMAG+RSP can learn to overcome the problem.

##  5 Conclusions and Future Work

We studied a new problem setting of KB-to-text, namely informative text
generation, which requires models to generate text that not only conveys the
input triples but also depicts the extra information of the involved entities.
To solve the problem, we propose a novel memory augmented generator which has
a specifically designed information memory to store and utilize the extra
information. The experimental results show that our model can generate
informative sentences from the input triples by fetching extra related
information that is not mentioned by the input triples. Moreover, our model is
capable of generating fluent biographies by only using the memory, and the
proposed variants can effectively alleviate the content-repeating problem.

The following directions remain to be explored: (1) Our model only needs to
make minor changes to the existing dataset and can thus be naturally extended
to other datasets. (2) We only focus on resolving the task in specific domains
and the open domain setting can also be explored in the future.

## References

  * Auer et al. (2007) Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007.  Dbpedia: A nucleus for a web of open data.  In _6th International Semantic Web Conference_ , pages 722–735.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Chen and Mooney (2008) David L Chen and Raymond J Mooney. 2008.  Learning to sportscast: a test of grounded language acquisition.  In _Proceedings of the 25th international conference on Machine learning_ , pages 128–135. ACM.
  * Chen et al. (2019) Shuang Chen, Jinpeng Wang, Xiaocheng Feng, Feng Jiang, Bing Qin, and Chin-Yew Lin. 2019.  Enhancing neural data-to-text generation models with external background knowledge.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3013–3023.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using rnn encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734.
  * Gardent et al. (2017a) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017a.  Creating training corpora for nlg micro-planners.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 179–188.
  * Gardent et al. (2017b) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017b.  The webnlg challenge: Generating text from rdf data.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 124–133.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Falcon Dai, Henry Elder, and Alexander Rush. 2018.  End-to-end content and plan selection for data-to-text generation.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , pages 46–56.
  * Graves et al. (2014) Alex Graves, Greg Wayne, and Ivo Danihelka. 2014.  Neural turing machines.  _arXiv preprint arXiv:1410.5401_.
  * Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016.  Incorporating copying mechanism in sequence-to-sequence learning.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1631–1640.
  * He et al. (2017) Shizhu He, Cao Liu, Kang Liu, and Jun Zhao. 2017.  Generating natural answers by incorporating copying and retrieving mechanisms in sequence-to-sequence learning.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 199–208.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Jurafsky and Martin (2000) Daniel Jurafsky and James H. Martin. 2000.  _Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition_ , 1st edition.  Prentice Hall PTR, Upper Saddle River, NJ, USA.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017\.  OpenNMT: Open-source toolkit for neural machine translation.  _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, System Demonstrations_ , pages 67–72.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016.  Neural text generation from structured data with application to the biography domain.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1203–1213.
  * Lehmann et al. (2015) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, et al. 2015.  Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia.  _Semantic Web_ , 6(2):167–195.
  * Liang et al. (2009) Percy Liang, Michael Jordan, and Dan Klein. 2009.  Learning semantic correspondences with less supervision.  In _Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics_ , pages 91–99.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  _Workshop on Text Summarization Branches Out_.
  * Meng et al. (2018) Fandong Meng, Zhaopeng Tu, Yong Cheng, Haiyang Wu, Junjie Zhai, Yuekui Yang, and Di Wang. 2018.  Neural machine translation with key-value memory-augmented attention.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , pages 2574–2580. AAAI Press.
  * Mi et al. (2016) Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016.  Coverage embedding models for neural machine translation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 955–960.
  * Miller et al. (2016) Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016.  Key-value memory networks for directly reading documents.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1400–1409.
  * Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018.  Ranking sentences for extractive summarization with reinforcement learning.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , volume 1, pages 1747–1759.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017.  The e2e dataset: New challenges for end-to-end generation.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 201–206.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _International Conference on Learning Representations_.
  * Ranzato et al. (2016) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.  Sequence level training with recurrent neural networks.  In _International Conference on Learning Representations_.
  * Reiter and Dale (1997) Ehud Reiter and Robert Dale. 1997.  Building applied natural language generation systems.  _Nat. Lang. Eng._ , 3(1):57–87.
  * Shen et al. (2019) Sheng Shen, Daniel Fried, Jacob Andreas, and Dan Klein. 2019.  Pragmatically informative text generation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4060–4067.
  * Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015.  End-to-end memory networks.  In _Advances in neural information processing systems_ , pages 2440–2448.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems_ , pages 3104–3112.
  * Tian et al. (2019) Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019.  Sticking to the facts: Confident decoding for faithful data-to-text generation.  _arXiv preprint arXiv:1910.08684_.
  * Tran and Nguyen (2020) Trung Tran and Dang Tuan Nguyen. 2020.  Webnlg 2020 challenge: Semantic template mining for generating references from rdf.  In _Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)_ , pages 177–185.
  * Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016.  Modeling coverage for neural machine translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 76–85.
  * Wang et al. (2016) Mingxuan Wang, Zhengdong Lu, Hang Li, and Qun Liu. 2016.  Memory-enhanced decoder for neural machine translation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 278–286.
  * Weston et al. (2014) Jason Weston, Sumit Chopra, and Antoine Bordes. 2014.  Memory networks.  _arXiv preprint arXiv:1410.3916_.
  * Williams (1992) Ronald J Williams. 1992.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ , 8(3-4):229–256.
  * Wiseman et al. (2017) Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017.  Challenges in data-to-document generation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2253–2263.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Yang et al. (2019) Pengcheng Yang, Lei Li, Fuli Luo, Tianyu Liu, and Xu Sun. 2019.  Enhancing topic-to-essay generation with external commonsense knowledge.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2002–2012.
  * Yu et al. (2017) Adams Wei Yu, Hongrae Lee, and Quoc Le. 2017.  Learning to skim text.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1880–1890.