# Identifying Chemicals in Metabolomic Samples Through Dimensionality
Reduction

Emile Anand
Computing and Mathematical Science
California Institute of Technology
Pasadena, CA, 91125
<EMAIL_ADDRESS>This work was funded by a Caltech Summer Undergraduate
Research Fellowship, 2020 Charles Steinhardt
Cosmic Dawn Center, Niels Bohr Institute
University of Copenhagen
Copenhagen, Denmark
<EMAIL_ADDRESS>
###### Abstract

Civilizations have tried to make drinking water safe to consume for thousands
of years. The process of determining water contaminants has evolved with the
complexity of the contaminants due to pesticides and heavy metals. The routine
procedure to determine water safety is to use targeted analysis which searches
for specific substances from some known list; however, we do not explicitly
know which substances should be on this list. Before experimentally
determining which substances are contaminants, how do we answer the sampling
problem of identifying all the substances in the water? Here, we present an
approach that builds on the work of Jaanus Liigand et al., which used non-
targeted analysis that conducts a broader search on the sample to develop a
random-forest regression model, to predict the names of all the substances in
a sample, as well as their respective concentrations[1]. This work utilizes
techniques from dimensionality reduction and linear decompositions to present
a more accurate model using data from the European Massbank Metabolome Library
to produce a global list of chemicals that researchers can then identify and
test for when purifying water.

_Keywords_ Dimensionality Reduction $\cdot$ Identifying Chemicals $\cdot$ Non-
Negative Matrix Factorization $\cdot$ Mass Spectra $\cdot$ Principal Component
Analysis $\cdot$ Convolutional Filters $\cdot$ Spectral Identification

##  1 Introduction

There is an undeniable need to provide safe water across the world. Scientists
currently use targeted analysis to search for water contaminants. A targeted
analysis is a narrow search performed on a sample to identify specific
substances. However, this introduces a statistical sampling problem: while
scientists know how to test for and filter specific substances, how do they _a
priori_ determine which substances these tests should encompass? An emerging
method to analyze samples of water is untargeted analysis[1][4][5], which is
when a broad search is performed on the water sample to detect both known and
unknown chemicals using data from mass spectrometry. Untargeted analysis in
this field necessitates a data-driven approach that utilizes machine learning
to learn the function that maps data from a mass spectrometer to a chemical
formula. Therefore, the onus of this problem of ensuring safe water access and
identifying the composition of a metabolome (the set of chemicals in a
biological sample) is to utilize a data-driven approach to learn this function
and use it to identify and predict the concentration of each chemical in the
metabolomic landscape.

In 2020, Liigand et al.[1] attempted to solve this problem by developing a
hyperparamaterized regularized random forest-regressor. A random forest
regression model is a discrete machine learning model that trains an ensemble
of randomized regression trees on a random subset of the data. The model
estimates values by averaging predictions over each random tree regressor.
Liigand et. al.’s model achieved a strong regularization condition (a means to
prevent over-fitting the model on the data) by enforcing a maximum leaf size
of 120 nodes on each tree[1]. Therefore, upon input of data from a mass-
spectrometer, the random forest regression model[2] discretized the output
space to estimate the decision boundaries; in doing so, it learned some
fragments of the underlying relationships between each chemical and its mass
spectrum. Liigand et. al.’s hyperparameterized model produces a mean error of
8.8%[1], which can potentially be improved since discrete models ignore
possible continuous relations between the mass spectra and the profile of the
corresponding metabolite. This is provable using a t-stochastic neighbor
embedding (t-SNE) argument:

T-SNE [3] is a popular tool for visualizing high-dimensional data. It creates
a probability distribution for each point (using a normalized Gaussian
centered at that point) and minimizes the Kullback-Leibler divergence between
this distribution and the distribution of a set of points in $\mathbb{R}^{2}$
using stochastic gradient descent algorithm. In $\mathbb{R}^{2}$, T-SNE tends
to form clusters of points such that each cluster can be qualitatively
described using a single variable; therefore, the number of clusters provide
clues to the approximate minimum number of variables needed to describe the
data[3]. Figure 1 displays a t-SNE plot that is color-coded with the presence
of Chlorine atoms (here, chlorine-containing compounds are black and non-
chlorine containing compounds are yellow). Figure 1b shows that there exists a
reliable 3-variable clustering of substances in reduced-parameter form due to
the presence of 3 clusters. Observe that the variance in the colored labels is
reasonable since the clusterings cannot distinguish between similar and
dissimilar data points (the t-SNE algorithm simply does not check for this).
Thus, our approach to solving this problem can potentially produce a mean
ground-truth error of 0 since we do not make assumptions about the
distribution[8].

Figure 1: t-SNE outputs for objects with a perplexity of 10, 50, and 100,
trained on 500 iterations. Lower perplexity values form localized and complex
structures, whereas higher perplexity values form globalized and superficial
structures due to more significant clusterings. These graphics were developed
independently through the Python graphics engine and the tSNE and Manifold
libraries.

##  2 Methods

###  2.1 Data Acquisition and Preprocessing

The ideal machine-learning model that reverse-engineers the process of mass
spectrometry takes in a mass spectrum as an input and predicts the name of the
substance corresponding to the spectra. Before constructing such a model, it
is important that we only train it on qualitatively clean data. This
necessitates the need to refine it.

The Human Metabolome Database and the European MassBank Database use a mass-
spectrometer to generate chromatograms and mass spectra of 1755357 peaks
corresponding to 85582 various substances. Each spectra contains numerous
peaks, and each peak represents a fragment of the ionized sample deflected in
the mass spectrometer (as shown in Figure 2). The height of the peak is the
relative intensity of the fragment in the sample, and the location of the peak
is the mass-to-charge ratio of the fragment. To ensure that the data-points we
would choose were of high-quality, we only extracted chemicals with more than
3 peaks in their corresponding spectrum in which each peak exceeded a
predetermined threshold signal-to-noise ratio (SNR). In order to compare
different spectra, we discretized the mass-charge values by binning the data.
While binning leads to a small loss of information, it was essential for this
problem since it is otherwise impossible to compare continuous spectra and it
standardizes the different sub-datasets in the overall database, thus
accounting for differences in how the data was posted.

Figure 2: Visualizations of the spectrum of the compounds $C_{5}H_{10}O_{5}$
and $C_{7}H_{6}O_{2}$. The data was acquired from the European MassBank and
the Human Metabolome Database, and the figures were generated through Python’s
Matplotlib graphics engine.

###  2.2 Chemical Encoding is a Barrier to Developing Continuous Models

The standard method of finding an all-encompassing relationship in a dataset
with hundreds of variables is through fitting the data on a neural network.
Our work reveals that a neural network cannot be used to predict the
relationship between a mass spectrograph and its corresponding chemicals due
to its intractability: specifically, similar chemicals (chemicals that are
structurally similar) have dissimilar spectra; therefore, there is no possible
continuous loss function that a neural network can attempt to minimize to
begin with. An in-depth analysis reveals that in order for a neural network to
be able to predict any existing relationship, it needs to understand the
output, and this necessitates that the output space (the names of the
chemicals) follows a numerical encoding. There are various forms of encoding
which can be used, all of which are problematic. Firstly, the name of the
molecule is encoded into a 16 character ID called the InChIKey through the
SHA-256 hash function. One encoding strategy is to use a one-hot encoding, in
which a Boolean column vector of 0s has a 1 only at one specific position.
However, this yields a space complexity of $O(2^{2^{16n}})\in O(2^{2^{n}})$
which is too large for a neural network. Another encoding strategy is to use
an integer encoding by assigning a natural number to each unique chemical.
However, the integer encoding assumes that numbers close to each other
correspond to chemicals that are similar, which is not true. Finally, the only
other possible means of encoding the chemical is to directly encode the
structure of the chemical through a graph neural network. This is not a
meaningful solution since the output of the graph neural network layer would
be fed in as an input to the artificial neural network, both of which have
incompatible architectural layers.

##  3 Starting from Scratch: Keeping Track of and Destroying Assumptions

###  3.1 Principal Component Analysis (PCA)

We first try to predict the spectral relationship with a simplistic model that
makes known assumptions. We assume that all the spectra are formed from linear
combinations of orthonormal basis vectors, where the physical interpretation
of each basis vector is an unknown atom or a functional group. This assumption
is reasonable since compounds can be viewed as a combination of various atoms
or functional groups. Therefore, we use Principal Component Analysis (PCA),
which is a dimensionality-reduction algorithm that re-derives the basis
vectors for high-dimensional data. Specifically, we feed the spectra to the
PCA algorithm, in the hopes of finding which bases vectors constitute the
chemical spectrum.

Figure 3: The first two dimensions of the 20-dimensional spectral matrix and
its bases vectors generated from the PCA algorithm represented respectively.
The points are generated from the Python Scikit-learn Decomposition-PCA
library, and are scatter-plotted using the Python Matplotlib graphics engine.

However, two problems emerge from this approach:

  1. 1.

Firstly, the basis vectors for any dimension greater than 1 is not unique. A
specific example of this is in $\mathbb{R}^{2}$ where the basis vectors for
any set of vector is just any pair of orthogonal vectors. Therefore, when we
try to reduce the 2660-dimension binned data to any reasonable size (smaller
than 100), we extract orthonormal basis vectors, where none of the basis
vectors correspond to any spectra. If we knew the rotation matrix that would
superimpose the extracted basis vectors into a physically realizable solution,
and we don’t, this would still not solve the problem since the PCA-produced
coefficients for each basis vector are very similar; therefore, the
reconstructed compounds would be too distinct from the original compounds to
be of any meaning. Hence, PCA does not provide a sufficient decomposition for
the data.

  2. 2.

Secondly (and as a consequence of the first point), the coefficients of some
of the basis vectors produced by the algorithm can be negative. Since basis-
vectors correspond to the presence of a specific substance, negative
coefficients correspond to negative intensities which is physically
meaningless.

###  3.2 Non-negative Matrix Factorization (NMFA)

Since PCA fails to decompose the data, we move one assumption down the ladder
and stop assuming that the spectra are formed from linear combinations of any
orthonormal basis vectors, and are instead from linear-combinations of sparse
orthonormal basis vectors. Furthermore, to make the solution physically
meaningful, we assume that none of the coefficients are negative, since all
peaks have to be positive.

Therefore, we develop a non-negative matrix factorization (NMFA) model which
approximately factorizes the spectral matrix $X$ into a matrix of N columns of
weights $W$ and N rows of basis vectors $H$. We apply the NMFA model to the
spectrum to identify the weights and the basis vectors, while experimentally
optimizing N in order to reduce N while maintaining a sufficient degree of
intensity of the data. To optimize N, we need to decide if the reconstructed
spectra is close enough to the original spectra.

Figure 4: Pipeline of Reconstructing Spectra

To obtain a reconstruction of the original matrix using fewer principal
components, we use the method described above. Note here that the product of
the $i$’th column of W and the $i$’th row of H forms a reconstruction of the
$i$’th spectrum. Using this process, we plot the sum of the absolute values of
the differences of the spectral matrix between the original spectra and the
reconstructed spectra (Figure 5).

Figure 5: The sum of the absolute values of the differences of the spectral
matrix between the original spectra and the reconstructed spectra, plotted
against varying number of components.

Figure 5 demonstrates that at 125 components, more components does not cause
the NMFA error to decrease significantly. Therefore, we chose 120 components
and ran the non-negative matrix factorization algorithm to generate the $H$
and $W$ matrices. We then plotted the actual spectrum and reconstructed
spectrum for various substances and observed a variance in the level of
accuracy of reconstructions. Figure 6 displays a relatively high-quality
reconstruction of a sample. Throughout the reconstructions, we observed that
reconstructed spectra which were not consistent with the actual spectra
differed slightly in horizontal position, but vastly in vertical scaling.

Figure 6: The actual spectrum (left) and reconstructed spectrum (right) of
$C_{10}H_{16}N_{5}O_{13}P_{3}$. The reconstructed spectra was produced using
Python’s Scikit-learn NMFA library and plotted using Python’s Matplotlib
graphics engine.

We then visualize the actual and reconstructed spectral matrices produced by
the non-negative matrix factorization (in Figure 7) and note that the
differences in the reconstruction are (at a high-level point of view)
negligible, as desired.

Figure 7: The actual spectral matrix (left) and reconstructed spectral matrix
(right). The visualizations were produced using Python’s Matplotlib graphic
engine and the Python Imshow feature.

Additionally, we plotted the actual and reconstructed spectral matrix on a
logarithmic plot (scaling each plot from the lowest values to the highest
values of the corresponding spectral matrix). In doing so, the differences in
the reconstruction became more distinct.

Figure 8: The actual spectral matrix (left) and reconstructed spectral matrix
(right) plotted on a logarithmic scale.

##  4 Extracting NMF Bases and Developing an Identifier algorithm

The ideal physical interpretation of each basis vector is that it represents
some ’token’, where each token is a monoatomic element or a functional group
that is a building block of all chemical compounds. Therefore, if the NMFA
decomposition correctly produced a set of tokens, they would be represented by
the bases vectors. In this instance, the spectrum of the base would correspond
to one significant peak that represented this smaller token. Since this is
falsifiable, we randomly plotted 2 of our basis vectors (in Figure 9) and
affirmed that the basis vectors indeed contained a significant peak which
confirmed a correspondence to a small building-block-like structure. We
additionally confirmed that this pattern propagated throughout all the basis
vectors.

Figure 9: The first (left) and ninth (right) basis vector extracted from the
weight matrix produced from the NMFA decomposition with 120 components.

##  5 Extraction Algorithm

Previously we showed that the NMFA extracted basis vectors that were
consistent with possible tokens, where we defined tokens to be either
monoatomic elements or small functional groups that could be possible building
blocks of larger chemical compounds. However, we cannot naïvely know whether
this is true without an in-depth analysis.

Therefore, we developed the following algorithm (SEARCHER) that searches for
various tokens in compounds to determine their mass-charge ratio. Since most
compounds in the dataset were organic and were only ionized once, the mass-
charge ratio is effectively the same as the atomic mass. Therefore, we can
determine if the basis vectors are physically meaningful by comparing the
atomic mass that the SEARCHER algorithm outputs with the known atomic mass of
the token. In turn, this provides a better statistical test for checking that
the basis vectors actually represent real physically interpretive tokens.

Algorithm 1 SEARCHER

Token

Tokenize all the compounds in the dataset and look for all the compounds
containing the token.

Find the reconstructed spectrum of each compound that contains the token.

Average all the intensities in each bin across every compound that contains
the token.

Find the bin (mass-charge) that contains the highest-intensity.

Return the mass-charge of the corresponding bin.

##  6 Results of the SEARCHER algorithm

###  6.1 Running SEARCHER on Oxygen

When the searcher algorithm was run on the ’O’ token corresponding to Oxygen,
we obtained surprising results (Figure 10). Firstly, the algorithm converged
on one specific basis-vector. When this basis-vector was examined, it
contained a single peak that was primarily at the mass-charge ratio of
31.14505 which is exactly the atomic mass of Oxygen. Note that this result was
obtained without ever telling the algorithm the atomic mass of any substance
at any point in time, therefore conclusively showing that the basis vector has
physical meaning in representing a token.

Figure 10: The output of the searcher algorithm on a ’O’ token corresponding
to Oxygen. The highest peak here occurs at the mass-charge of 31.145 which is
to 2 s.f. the atomic mass of Oxygen.

###  6.2 Running SEARCHER on Chlorine

We similarly ran the algorithm on the Chlorine token and obtained an atomic
mass of 22.87 (Figure 11), whereas the atomic mass of the average Chlorine ion
is 35.5 amu. To explain this, we notice that 22.87 is (to 2 s.f.) accurate to
the atomic mass of Sodium, and that many of the data collectors in this
dataset dissolved their compounds in a solution of brine (concentrated sodium
chloride solution), where the sodium ions were likely picked up in the mass
spectrometer. Thus, we believe the algorithm is still working correctly and
that this is a fair result.

Figure 11: The output of the searcher algorithm on a ’Cl’ token corresponding
to Chlorine. The highest peak here occurs at 22.87 which is to 2 s.f. the
atomic mass of Sodium. We suspect this is because all solutions in the dataset
are dissolved in brine (concentrated Sodium Chloride) which interferes with
the readings.

###  6.3 Running SEARCHER on Carbon and Hydrogen

When running the algorithm on the Carbon token (Figure 12 - left) and the
Hydrogen token (Figure 12 - right), we obtained the exact same results. In
particular, the algorithm converged the Carbon and Hydrogen token to one basis
vector with a corresponding mass-charge value of 31.42 (to 4 s.f.). We know
this to be an erroneous result, since Carbon’s atomic mass is 12.01 (to 2
s.f.) and Hydrogen’s atomic mass is 1.01 (to 2 s.f.), and this is actually the
same atomic mass produced when the SEARCHER algorithm is run on the Oxygen
token.

We again suspect that the algorithm is working correctly, and we attribute
these results to the kinds of data we are working with: 85% of the compounds
in the dataset are organic (of which, 70% contain Oxygen). Specifically, these
Oxygen-containing compounds dominate the average compound, and the mass-
spectrum of Oxygen dominates those of Carbon and Hydrogen, which are again
reasonable choices for tokens.

Figure 12: The output of the searcher algorithm on a ’C’ token (left)
corresponding to Carbon and a ’H’ token (right) corresponding to Hydrogen.
Note that these spectra are exactly the same, and that they are also congruent
to the spectrum of Oxygen. This is because most compounds in the dataset are
organic and contain Carbon Hydrogen and Oxygen atoms. However, the true Carbon
and Hydrogen basis vectors are not revealed due to Oxygen’s mass-charge ratio
dominating those of Carbon and Hydrogen.

It is non-trivial to rectify this problem (and other variations of it), since
in the naïve approach, after every physical token is found and confirmed to be
correct, we would have to adjust the spectral matrix by subtracting
$c$(spectra of the token), $c\in\mathbb{R}$, to minimize the spectral peak
corresponding to that token. However, finding $c$ is a hard problem since the
spectra for most of these tokens contain smaller peaks which scale
significantly with $c$, and since different spectra contain intensities that
are of different orders of magnitudes, creating a statistical problem in
determining the correct precision (the number of bits needed to encode the
bytes) of $c$.

Additionally, a clear computational bound which exists is that after each time
the spectral matrix is adjusted and a token is determined, the NMF algorithm
must be run again on the modified matrix, despite the extensive time costs of
performing the NMFA. However, there might exist optimal methods to solve this
problem that do not rely on aforementioned naïve approach and its uneconomic
time complexity.

##  7 Discussion

###  7.1 Potential Sources of Error and Noise

The primary source of noise in this approach stems from the data. The European
Massbank is a collection of numerous small datasets from various
organizations. Each of these organizations had varying degrees of accuracy for
their data collection, and each of them posted their data differently. For
instance, Figure 13 displays the spectra for two different chemicals, each
posted by different organizations. The second chemical (Figure 13 - top right)
clearly conveys more information per peak than what is conveyed in the first
chemical (Figure 13 - top left). However, by binning (discretizing the mass-
charge regions of the data and letting each bin-value be the maximum intensity
located within the region), we can eliminate the lack of standardization (as
seen in the more comparable plots shown in Figure 13 - lower layer). While
this does standardize the data sufficiently while allowing us to perform the
essential task of comparing different spectra, binning does introduce a loss
of information and there is no naïve way to overcome this problem.

Figure 13: Unstandardized data (top). Binned data (lower). Clearly, binning
allows comparison between spectra. The mass-charge units are kg/C, and the
intensity units are arbitrary (it is a measure of how hard different fragments
collide with the detector and this measure is dependent on the detectors
sensitivity) so only the relative amount is important.

One possible source of error in the data lies in the problem’s chemistry. When
a mass-spectrometer uses a sample, it vaporizes and bombards it with electrons
to fragment the sample in various ways. These fragments represent the peaks in
the mass-spectrograph. The mass-charge corresponds to the mass of the fragment
by measuring how much the fragment deflects in the mass-spectrometer. It also
measures the force with which the fragments collides with the detector, thus
assigning the fragment an intensity value. The fragmentation within the mass-
spectrometer is not necessarily unique since large compounds can fragment in
many ways. Thus, the data collection groups repeat this process numerous times
to obtain every possible fragmented source/peak, which introduces the
possibility of missing fragments and repeated fragments at different mass-
charge values due to random error/uncertainty[6][7].

###  7.2 Nonnegative Matrix Factorization Accuracies

The non-negative matrix factorization (NMF) algorithm is mathematically
equivalent to a clustering algorithm where the goal is to minimize
$||X-WH||,H\geq 0,W\geq 0$. However, the algorithm (when used on our spectral
matrix, $X$) could not minimize $||X-WH||$ to 0 when a suitable number of
principal components (basis vectors) were used. Note here that ’suitable’ is a
twisted word. When the number of principal components is equal to 85582 (the
number of compounds in the dataset, and therefore the number of rows in the
spectral matrix), the minimization works perfectly since each spectrum gets
assigned to a basis vector. However, in doing so, we learn nothing new and
cannot extract any of the underlying information about the spectral matrix and
its constituent building blocks. Therefore, the goal of the dimensionality
reduction is to reduce the number of principal components as much as possible
while still retaining as much accuracy as possible. Therefore, when we chose
125 principal components, we deliberately introduced some noise into the
model. However, we also combat this by proposing that the token identification
process of the 125 basis vectors be done manually to ensure that the results
still make physical sense. This model has the potential to perform much better
than any discrete model, and the results we have obtained are a testimony to
this. The next section details the future work to be done on this project, as
well as the problems that will need to be solved in order to identify the
basis vectors.

##  8 Future Works

To further solve this problem, future works could focus on understanding how
to find residual bases vectors. In doing so, this would allow researchers to
identify non-dominant basis vectors as physical tokens, thus avoiding the
problem (and other analogs) of Oxgyen’s spectrum dominating those of Carbon
and Hydrogen. Furthermore, it is well-known that the presence of various atoms
or functional groups has a large effect on the characteristics of the entire
molecule. This effect is non-linear, since the presence of such a token
changes the spectral landscape of the entire molecule. This cannot be
accurately modeled by linear decomposition models such as PCA or NMFA, and
might require novel decomposition methods, possibly through an appropriate
modification of the Tucker decomposition process. Once the basis vectors are
obtained, future work could focus on reconstructing all possible corresponding
compounds from the known ’tokens’ contained.

##  9 Acknowledgements

  1. 1.

Caltech SURF and the COSMIC Dawn Center - Niels Bohr Institute

  2. 2.

Professor Charles Steinhardt - COSMIC Dawn Center, Niels Bohr Institute,
University of Copenhagen, Denmark

  3. 3.

Professor Martin Hansen - Aarhus University, Denmark

  4. 4.

Xiaomin Zhu - Aarhus University, Denmark

  5. 5.

Vadim Rusakov - COSMIC Dawn Center, Niels Bohr Institute, University of
Copenhagen, Denmark

  6. 6.

Professor Robert Phillips - California Institute of Technology, USA

  7. 7.

Human Metabolome Database and European Mass Bank Spectral Libraries

##  10 References

[1] Jaanus Liigand, T. Wang, J. Kellogg: Quantification for non-targeted LC/MS
screening without standard substances. Scientific Reports, 2020

[2] Tianqi Chen, Carlos Guestrin (2016): XGBoost: A Scalable Tree Boosting
System, arXiV 2016.

[3]: Laurens van der Maaten and Geoffrey Hinton: Visualizing Data using t-SNE,
Journal of Machine Learning Research, 2008, Volume 9, Page 2579-2605, Page 86,
http://jmlr.org/papers/v9/vandermaaten08a.html

[4]: Alder, Lutz, et al.: Residue Analysis of 500 High Priority Pesticides:
Better by GC–MS or LC– MS/MS? Mass Spectrometry Reviews, 2020,
10.1002/mas.20091

[5]: Tomas Cajka, Oliver Fiehn: Toward Merging Untargeted and Targeted Methods
in Mass Spectrometry-Based Metabolomics and Lipidomics. Analytical Chemistry,
2020, 10.1021/acs.analchem.5b04491

[6]: Djoumbou-Feunang, Yannick, et al. BioTransformer: A Comprehensive
Computational Tool for Small Molecule Metabolism Prediction and Metabolite
Identification. Journal of Cheminformatics, 2020 10.1186/s13321-018-0324-5

[7]: Xu, Raymond Naxing, et al. Recent Advances in High-Throughput
Quantitative Bioanalysis by LC–MS/MS. Journal of Pharmaceutical and Biomedical
Analysis, 2020, 10.1016/j.jpba.2007.02.006

[8]: R. Silipo, M. Widmann: 3 new techniques for Data-Dimensionality reduction
in machine learning: aRxiV 2020


# Reducing Hallucinations in Entity Abstract Summarization with Facts-Template
Decomposition

Fangwei Zhu, Peiyi Wang, Zhifang Sui
National Key Laboratory for Multimedia Information Processing, Peking
University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Entity abstract summarization aims to generate a coherent description of a
given entity based on a set of relevant Internet documents. Pretrained
language models (PLMs) have achieved significant success in this task, but
they may suffer from hallucinations, i.e. generating non-factual information
about the entity. To address this issue, we decompose the summary into two
components: Facts that represent the factual information about the given
entity, which PLMs are prone to fabricate; and Template that comprises generic
content with designated slots for facts, which PLMs can generate competently.
Based on the facts-template decomposition, we propose SlotSum, an explainable
framework for entity abstract summarization. SlotSum first creates the
template and then predicts the fact for each template slot based on the input
documents. Benefiting from our facts-template decomposition, SlotSum can
easily locate errors and further rectify hallucinated predictions with
external knowledge. We construct a new dataset WikiFactSum to evaluate the
performance of SlotSum. Experimental results demonstrate that SlotSum could
generate summaries that are significantly more factual with credible external
knowledge. 111Code & data available at https://github.com/solitaryzero/SlotSum

##  1 Introduction

Entity abstract summarization aims to generate a correct and fluent summary of
an entity based on a collection of relevant documents gathered from the
Internet. The task is typically viewed as a variant of the multi-document
summarization task, as illustrated in Figure 1. Entity abstract summarization
could provide users with condensed information about a given entity in search
engines, and assists the writing process of new Wikipedia entries Sauper and
Barzilay (2009).

Figure 1: A demonstration of the entity abstract summarization task. The
summarization model aims to generate a brief summary $S$ that resembles the
golden summary $S^{*}$ about the person Peter Wichers from source documents
$D$ collected from the Internet. Blue indicates correct facts, and Red
indicates noise and hallucinated predictions.

Early pieces of work Sauper and Barzilay (2009); Banerjee and Mitra (2016) on
entity abstract summarization generally adopt extractive methods that select
valuable sentences from the input documents, which face the problem of poor
fluency in the generated summary. Inspired by the large-scale WikiSum Liu et
al. (2018) dataset, recent work tends to directly generate abstractive
summaries from the given input documents. Various approaches have been
explored to identify necessary information from the vast input documents:
Perez-Beltrachini et al. (2019) classifies documents into different topics to
filter out noisy data, HierSumm Liu and Lapata (2019) ranks the input
documents with a learning-based ranker, and models the dependency across
different paragraphs, and NoisySumm Liu et al. (2021) distills the knowledge
in a noisy teacher model into a student model.

However, these models suffer from hallucinations as they are not able to
ensure the factual correctness of generated summaries. Hallucinations refer to
the nonsensical or unfaithful contents in the generated texts Maynez et al.
(2020). They could hinder performance and introduce false information for
real-world applications Ji et al. (2023). Take Figure 1 as an example, the
summarization model correctly identifies the band name and band genre of the
given entity Peter Wichers, but incorrectly assumes the nationality to be
American rather than Sweden. Hallucinated by the text snippet “from Atlanta
and Baltimore” in the source, the summarization model mistakenly relates the
text snippet to the nationality of America, ignoring the fact that Peter
Wichers comes from Sweden. These factual errors are difficult to detect with
automatic metrics and could degrade the system performance, leading to the
failure in meeting user expectations in real-world scenarios Ji et al. (2023).

Hallucination is an intrinsic drawback of language models and is challenging
even for large language models Bubeck et al. (2023). Hallucinations are
difficult to eliminate under the traditional sequence-to-sequence paradigm,
and various attempts have been taken to introduce external knowledge like
dense retrieval Lewis et al. (2020b) or API calling Schick et al. (2023).
Templates are naturally suitable for utilising external knowledge, as the
slots in templates could provide guidance for retrieving knowledge. While
template-based methods are widely adopted in various text generation problems
like data-to-textLuo et al. (2020) and dialogue system Liang et al. (2021),
there has not yet been template-based solutions for entity abstract
summarization. To face the difficulty of locating hallucinations in plain
text, we augment the language model to generate semi-structured templates
before outputting the final summary. We view the entity abstract as a
combination of Facts and Template, where Facts represents the factual
information about the given entity, and Template represents the remaining
generic components. In this way, we can easily locate the facts in generated
summaries, and correct erroneous predictions with external knowledge like
knowledge base or human assistance.

Figure 2: The overall framework of SlotSum, with the entity George Burroughs
Torrey as an example. In step 1, SlotSum generates a template from the source
documents which contain several slots to be filled like name and birth_date.
In step 2, SlotSum first predicts the slots according to the source documents
and then corrects hallucinated predictions (like 1903 for birth_date) with
credible external knowledge. Finally, The summary is constructed by filling
slots into the template.

Based on the facts-template decomposition, in this paper, we propose SlotSum,
an explainable entity abstract summarization framework that disentangles facts
from generated entity abstracts to reduce hallucination. SlotSum first
generates a template according to the input documents, where the facts in the
template are replaced by slots marked with special tokens. Afterwards, SlotSum
predicts the facts for each slot, and attempts to rectify the predictions with
credible external knowledge. The final summary is constructed by filling facts
into slots. SlotSum could ensure factual correctness by introducing external
knowledge sources, while retaining the ability to generate summaries solely
from the input documents. Moreover, the framework is highly controllable and
explainable, as the facts are explicitly stated and can be easily edited.

By supplementing the WikiSum Liu et al. (2018) dataset with related facts, we
construct WikiFactSum to observe the effectiveness of disentangling facts.
SlotSum, with BART-large Lewis et al. (2020a) as the backbone model, clearly
outperforms other baseline models on ROUGE score when credible external
knowledge is provided. We also evaluate the factual correctness with both
automatic metrics and human evaluation, where SlotSum significantly provides
more correct facts.

In conclusion, the main contributions of our work are as follows:

  * •

We disentangle fact from the generated text in the entity abstract
summarization task, which helps to reduce hallucinations produced by
pretrained language models.

  * •

We propose SlotSum, an entity abstract summarization framework that first
generates templates with slots, then predicts facts and further correct them
with external knowledge to fill in the slots.

  * •

We construct a new dataset WikiFactSum, and conduct a series of experiments on
it. SlotSum demonstrates a clear advantage over other baseline models,
indicating disentangling facts from the text would lead to more factually
correct summaries.

##  2 Method

In this section, we propose SlotSum, an entity abstract summarization model
that disentangles facts from the text to reduce hallucinations in generated
summaries. As illustrated in Figure 2, SlotSum first trains a template
generator module to generate templates from input documents, where the facts
in the templates are replaced with slots to be filled. Each slot is
represented with a feature name that acts as the key of the fact. Afterwards,
SlotSum predicts the slots with a slot predicting module that extracts
relevant information from source documents given the feature name. As the
predicted slots could contain hallucinations, we further correct the
predictions with credible external knowledge sources. In this way, SlotSum is
able to enhance the factual correctness with further external knowledge.

###  2.1 Generating Templates

Entity abstract summarization aims to generate an abstractive description
$\mathcal{S}=(s_{1},s_{2},\ldots,s_{m})$ about a given entity $e$ from a set
of relevant text snippets $\mathcal{D}=(d_{1},d_{2},\ldots,d_{n})$. Given the
dataset $\mathcal{C}=(\mathcal{D},\mathcal{S})$, we first extract templates
$\mathcal{T}$ from the golden summaries $\mathcal{S}$. We denote the facts
related to the entity $e$ with the form of feature pairs $\\{(k,v)\\}$, where
the key of each pair $k$ represents the feature name and the value $v$
represents the fact content. Each fact $v$ about the given entity in the
original text $s$ are replaced with a slot $t$, and each slot $t$ is
represented by its feature name $k$:

$t(v)=\texttt{[SLT]}\ k\ \texttt{[/SLT]}$ (1)

where [SLT] and [/SLT] are special tokens.

To gather the training data for the template generation model, we construct
golden templates by replacing the facts in the golden summary with slots
according to external fact pairs $\\{(k^{*},v^{*})\\}$. Notice that the fact
value $v^{*}$ does not necessarily have an exact match in the original text
$s$, we adopt the fuzzy matching technique to discover more potential slots.
For each fact content $v^{*}$, we search for a substring $s_{v}$ in $s$ which
has the highest score $sim(v^{*},s_{v})$ under a given similarity metric
$sim$. We replace $s_{v}$ with a slot when its similarity score
$sim(v^{*},s_{v})$ surpasses a threshold $\delta$, and keep it unchanged
otherwise:

$s_{v}\leftarrow\left\\{\begin{aligned} &t(v^{*}),\
sim(v^{*},s_{v})\geq\delta\\\ &s_{v},\ \text{otherwise}\end{aligned}\right.$
(2)

where the threshold $\delta$ is a hyperparameter. In this way, we replace the
facts in $\mathcal{S}$ with slots and obtain the knowledge-agnostic template
set $\mathcal{T}$.

Predicting templates can be viewed as a standard sequence-to-sequence task,
where the input is web documents $\mathcal{D}$ and the output is the desired
templates $\mathcal{T}$. As the slots are represented by feature keys $k$
rather than values $v$, the model does not have access to external factual
knowledge in the training process, and therefore will not cause data leakage.

We use BART as the backbone of the template predictor and train the model with
cross-entropy loss. The normalized sorted Indel distance is selected as the
string similarity metric $sim$: given two strings $a$ and $b$, we first sort
the tokens in the strings, and then calculate the Levenshtein distance
$L(a,b)$ between $a$ and $b$ where only insertions and deletions are allowed.
The distance is normalized as the final similarity:

$sim(a,b)=1-\frac{L(a,b)}{Len(a)+Len(b)}$ (3)

###  2.2 Predicting and Correcting Slots

After generating the template, we need to fill facts into the template slots
to obtain the final summary. SlotSum first predicts the facts according to
input documents, then corrects the facts with credible external knowledge.

####  2.2.1 Predict from Input Documents

Firstly, SlotSum tries to extract facts from input documents to fill in the
slots. We use a simple sequence-to-sequence slot predictor to extract desired
facts from the source documents, whose input can be denoted as:

$\texttt{[CLS]}\ n\ k\ \texttt{[SEP]}\ d\ \texttt{[SEP]}$ (4)

where $n$ is the name of the entity to be summarized, $k$ is the slot name,
$d$ refers to the source documents, [CLS] and [SEP] are special tokens. The
slot predictor is trained on the golden fact set with the label smoothed
cross-entropy loss. We denote the predicted results as $\\{(k_{p},v_{p})\\}$.

####  2.2.2 Correct with External Knowledge

By introducing external knowledge sources, SlotSum is able to ensure the
correctness of facts. An external knowledge source, for example, knowledge
bases and human experts, will provide a set of trustworthy facts, which could
fill the slots without producing hallucination. Similar to the slot replacing
scenario in Section 2.1, the slot names do not exactly match the fact feature
names. Therefore, we only correct slots that have a candidate with name
similarity greater than a threshold $\delta$. We denote the corrected results
as $\\{(k_{c},v_{c})\\}$, where the key set $K_{c}$ contains corrected slots.

####  2.2.3 Slot-Filling Strategies

We adopt three distinct slot-filling strategies to get the slot value $v$ for
each slot $k$:

##### Discard

The discard strategy only fills slots corrected with external knowledge, and
fills other slots with empty strings to ensure the precision of facts. This
strategy can be formulated as:

$v=\left\\{\begin{aligned} v_{c}&,k\in K_{c},\\\ \varnothing&,k\notin
K_{c}.\end{aligned}\right.$ (5)

##### Predict

The predict strategy predicts all slots first and then tries to correct the
slots with external knowledge, which maximizes the information recall in the
generated summary. This strategy can be formulated as:

$v=\left\\{\begin{aligned} v_{c}&,k\in K_{c},\\\ v_{p}&,k\notin
K_{c}.\end{aligned}\right.$ (6)

##### All-Predict

The all-predict strategy predicts all slots without utilizing external
knowledge, which is applicable when no additional knowledge is provided. This
strategy can be formulated as:

$v=\left\\{\begin{aligned} v_{p}&,k\in K_{c},\\\ v_{p}&,k\notin
K_{c}.\end{aligned}\right.$ (7)

Feature | Value
---|---
# Examples | 27372
Train/Valid/Test | 21934/2697/2741
# Slots | 83782
Avg. Slots | 3.06
# Keys | 367642
Avg. Keys | 13.43
Avg. Value Len | 4.11
Avg. Src Length | 2000
Avg. Tgt Length | 26.47
Table 1: Statistics of WikiFactSum.

##  3 Experiments

###  3.1 Experimental Setup

##### Dataset Construction

Model | ROUGE | BERTScore | FactCC | Unieval
---|---|---|---|---
1 | 2 | L | P | R | F |  |
BART-large | 45.96 | 26.30 | 43.69 | 0.911 | 0.871 | 0.890 | 0.416 | 0.203
T5-base | 40.44 | 21.29 | 38.67 | 0.893 | 0.870 | 0.881 | 0.218 | 0.117
PEGASUS-xsum | 38.67 | 20.57 | 37.15 | 0.867 | 0.809 | 0.836 | 0.454 | 0.217
NoisySumm | 43.44 | 23.96 | 41.19 | 0.901 | 0.879 | 0.889 | 0.326 | 0.171
SlotSum (Discard) | 59.91 | 38.53 | 56.30 | 0.929 | 0.895 | 0.911 | 0.585 | 0.258
SlotSum (Predict) | 59.53 | 38.17 | 55.96 | 0.927 | 0.901 | 0.914 | 0.559 | 0.222
SlotSum (All-Predict) | 46.14 | 26.12 | 44.35 | 0.915 | 0.884 | 0.899 | 0.207 | 0.095
Table 2: Automatic evaluation results on WikiFactSum. Bold numbers indicate
the best performance on each metric. SlotSum refers to our proposed method,
and Discard, Predict and All-Predict refer to the 3 slot-filling strategies in
Section 2.2.3 respectively.

We conduct our experiments on WikiFactSum, a proposed dataset constructed by
combining WikiSum Liu et al. (2018) and WikiBio Lebret et al. (2016). WikiSum
is a widely-used entity abstract summarization dataset, which aims to generate
the Wikipedia abstract of an entity from websites searched by Google. WikiBio
provides structured data about a specific person and aims to generate a
biographical abstract about that person. WikiSum contains only textual data,
thus introducing WikiBio could complement WikiSum with facts.

In the WikiBio dataset, we select people with two occupations to construct our
WikiFactSum: Artist and Soccer Player, whose features are distinct to each
other. The entries with selected occupations are extracted from the WikiAsp
Hayashi et al. (2021) dataset, which is also derived from WikiSum. As WikiAsp
(and the original WikiSum dataset) does not provide the Wikipedia title of
each example, we try to find the corresponding entries in WikiBio by fuzzy
matching. We compute the bag-of-words similarity $S_{bow}$ of the abstract of
entry $a$ in WikiAsp and the abstract of entry $b$ in WikiBio:

$S_{bow}=\frac{A\cap B}{A\cup B}$ (8)

where $A$ and $B$ are the word set of $a$ and $b$. We assume $a$ and $b$ are
identical if $S_{bow}>0.8$, and map the corresponding entry between WikiAsp
(which is a subset of WikiSum) and WikiBio.

By combining the fact pairs $F=\\{(k,v)\\}$ in WikiBio with the source
document $D$ and the desired summary $S$ in WikiSum, we construct our proposed
dataset WikiFactSum. In this way, each entry in WikiFactSum consists of three
parts: the source document $D$, the desired summary $S$, and external facts
$F=\\{(k,v)\\}$. Table 1 demonstrates the detailed parameters of WikiFactSum.

##### Baselines and Metrics

We compare our model with three pretrained autoregressive language models BART
Lewis et al. (2020a), T5 Raffel et al. (2020), and PEGASUS Zhang et al.
(2020), which are widely used in summarization tasks. We also choose NoisySumm
Liu et al. (2021) as a baseline model, which is an entity abstract
summarization model that filters noise information by distilling the teacher
model into a student model. Four metrics are adopted to evaluate the
performance of models:

  * •

The widely used token-level similarity metric ROUGE Lin (2004);

  * •

BERTScore Zhang et al. (2019) that evaluates text similarity with BERT;

  * •

FactCC Kryściński et al. (2020) that identifies conflicts between source
documents and claims;

  * •

UniEval Zhong et al. (2022) that learns multi-dimensional evaluators from
intermediate tasks.

##### Implementation Details

We choose BART-large Lewis et al. (2020a) as the backbone model of SlotSum,
and set the similarity threshold $\delta=0.8$. The model is trained on a
single NVIDIA RTX 3090 for 4 epochs with a learning rate of 1e-5.
Specifically, we use T5-base and PEGASUS-xsum as baseline models, which match
the scale of BART-large.

Model | Linguistic Quality | QA
---|---|---
C | F | S | Overall
BART-large | 5.00 | 5.00 | 4.87 | 4.96 | 1.458
T5-base | 4.57 | 4.73 | 4.50 | 4.62 | 1.208
PEGASUS-xsum | 4.27 | 4.80 | 4.80 | 4.66 | 1.192
NoisySumm | 4.77 | 4.80 | 4.80 | 4.91 | 1.205
SlotSum (Discard) | 4.33 | 4.93 | 4.90 | 4.79 | 2.048
SlotSum (Predict) | 4.80 | 5.00 | 4.90 | 4.90 | 2.075
SlotSum (All-Predict) | 4.73 | 5.00 | 4.90 | 4.88 | 1.460
Table 3: Human evaluation results on WikiFactSum. Bold numbers indicate the
best performance on each metric. C, F, and S represent completeness, fluency,
and succinctness respectively.

###  3.2 Main Experiment Results

Table 2 demonstrates the automatic evaluation results on WikiFactSum, from
which several insights could be concluded:

##### Credible Knowledge Matters.

After correcting the summaries with external knowledge, it can be observed
that SlotSum shows a clear advantage over other baseline models on both
metrics. We can safely assume that introducing external knowledge will
drastically improve the quality of generated summaries.

##### General Summarization Models Struggles.

T5 and PEGASUS have been specially pretrained on summarization tasks, however,
they perform poorly on entity abstract summarization, even worse than BART.
This phenomenon indicate that the abilities required for entity abstract
summarization are not identical to those required for general summarization
tasks. The input documents of entity abstract summarization contain much more
noise, which would mislead general summarization models to low-quality
summaries.

##### Unchecked Predictions are Dangerous.

Compared with baseline models, SlotSum with the all-predict strategy seems to
be performing slightly better on ROUGE and BERTScore; However, on fact-
oriented metrics FactCC and Unieval, the all-predict strategy obviously falls
behind, performing even worse than the baseline BART model. We infer that
templates tend to guide the model to discover more facts in order to fill in
the slots, whereas the slot predicting module may fabricate facts, in turn
leading to the detriment of factual correctness. In practice, it should be
paid extra caution not to completely trust the predictions. Subsequent manual
checking may be very beneficial to the final summary.

Model | ROUGE | BERTScore | FactCC | Unieval
---|---|---|---|---
1 | 2 | L | P | R | F |  |
BART-large+K | 48.05 | 27.89 | 45.68 | 0.912 | 0.888 | 0.899 | 0.304 | 0.169
T5-base+K | 43.46 | 23.80 | 41.46 | 0.902 | 0.877 | 0.889 | 0.211 | 0.127
PEGASUS-xsum+K | 41.64 | 22.79 | 39.92 | 0.886 | 0.844 | 0.864 | 0.376 | 0.193
SlotSum (K+Discard) | 62.55 | 41.30 | 58.70 | 0.934 | 0.906 | 0.920 | 0.592 | 0.263
SlotSum (K+Predict) | 62.46 | 41.23 | 58.58 | 0.934 | 0.907 | 0.920 | 0.577 | 0.256
SlotSum (K+All-Predict) | 48.03 | 27.87 | 46.08 | 0.920 | 0.886 | 0.903 | 0.213 | 0.108
BART-large+KV | 69.88 | 54.13 | 66.68 | 0.949 | 0.923 | 0.935 | 0.842 | 0.360
T5-base+KV | 67.66 | 51.11 | 64.51 | 0.945 | 0.916 | 0.930 | 0.831 | 0.321
PEGASUS-xsum+KV | 63.10 | 43.06 | 60.30 | 0.936 | 0.900 | 0.917 | 0.753 | 0.185
SlotSum (KV+Discard) | 68.08 | 48.09 | 63.99 | 0.942 | 0.917 | 0.929 | 0.787 | 0.354
SlotSum (KV+Predict) | 68.20 | 48.17 | 64.06 | 0.942 | 0.917 | 0.929 | 0.773 | 0.349
SlotSum (KV+All-Predict) | 53.81 | 34.43 | 51.49 | 0.921 | 0.897 | 0.910 | 0.243 | 0.122
Table 4: Ablation results on WikiFactSum. K indicates taking the keys as part
of the input, while KV indicates taking the full key-value pair list as part
of the input. Bold numbers indicate the best performance on each metric under
certain setting (K or KV). SlotSum refers to our proposed method, and Discard,
Predict and All-Predict refer to the 3 slot-filling strategies in Section
2.2.3 respectively.

###  3.3 Human Evaluation

We conduct human evaluation consisting of two parts on the performance of
different models. Each part is annotated by 3 annotators on 20 examples
randomly sampled from the test set.

The first part evaluates the linguistic quality of generated summaries. We
require the annotators to evaluate the linguistic quality from three aspects:
Completeness (is the summary informative?), Fluency (is the summary
grammatically correct?), and Succinctness (does the summary get rid of
redundant text?). The score of each metric ranges from 1 to 5, and the final
result average is obtained by averaging the score of three annotators.

The second part evaluates the factual correctness of generated summaries,
where we follow the question-answering (QA) scheme Clarke and Lapata (2010)
for examination. We manually create 3 questions on the base of golden
summaries for each example, and ask annotators to answer these questions by
viewing generated summaries as backgrounds. An expert then examines the
annotated answers and assigns scores to different models. A model gets 1 point
for a correct answer and 0 point for a wrong answer. Specially, we give 0.1
point for a model if a question is not answerable on the base of generated
summary, as it does not cause hallucination.

From Table 3, we can observe that all models achieve decent scores on
linguistic quality metrics, while BART-large performs the best. The
speculation why SlotSum gets a lower score on Completeness is that some
information cannot be conveyed through a single slot, which leads to minor
information loss compared with the original BART. On the scope of QA
examination, the factual correctness of generated summaries significantly
increased after calibrating slot values with external knowledge, which proves
the effectiveness of introducing external knowledge from another perspective.

###  3.4 Ablation Study

To further investigate how facts influence the final summary, we evaluate the
models222NoisySumm does not fit the data augmentation, and is thus excluded
from ablation study. under two alternative settings: (1) Taking the keys
$\\{k\\}$ as part of the input, denoted as +K; (2) Taking the full key-value
pair list $\\{(k,v)\\}$ as part of the input, denoted as +KV.

The keys or key-value pairs are serialized by concatenating them with special
tokens [KV1] and [KV2]:

$k_{1}\ \texttt{[KV1]}\ k_{2}\ \ldots$ (9) $k_{1}\ \texttt{[KV1]}\ v_{1}\
\texttt{[KV2]}\ k_{2}\ \texttt{[KV1]}\ v_{2}\ldots$ (10)

where we use the token ’|’ for [KV1] and ’#’ for [KV2] in practice. The
serialized string $f_{s}$ is then concatenated with the source document $d$
with the separation token [SEP] as the final input:

$\texttt{[CLS]}\ f_{s}\ \texttt{[SEP]}\ d\ \texttt{[SEP]}$ (11)

Table 4 demonstrates the results under different settings, from which we draw
three conclusions:

##### SlotSum Maintains Its Competitiveness.

As expected, SlotSum with the predict or discard strategy still maintains a
clear advantage over other models on all metrics under the +K setting,
demonstrating its ability in generating factually correct entity summaries.

##### Guidance from Keys Degrades Baseline Performance.

An interesting discovery is that under the +K setting, baseline models guided
by keys gain a boost on ROUGE and BERTScore, but perform worse on FactCC and
Unieval instead. This tendency is identical with the main experiment
performance of SlotSum under the all-predict strategy, confirming the
hypothesis in section 3.2 that guiding the model with facts needed would in
turn induce hallucination and harm its factual correctness.

##### Providing Golden Facts Wipes the Gap.

We observe that when adding the full key-value pairs to the input, the scores
achieved by SlotSum are slightly lower than those achieved by BART, and the
gaps between baseline models are also narrower. Considering that the entity
abstract summarization problem is essentially degenerated into the data-to-
text problem under the +KV setting, we think it is understandable as direct
filling values into slots inevitably affects the fluency of summaries.

Figure 3: A case study on the entity Marc Muniesa Martínez. Blue texts
indicate golden facts, Green texts indicate correct predictions, and Red texts
indicate hallucinated predictions.

###  3.5 Case Study

Figure 3 demonstrates the summaries generated by different models about the
entity Marc Muniesa Martínez. We found two major observations by comparing
these summaries:

##### External Knowledge is Crucial to Improving Factual Correctness.

Factual errors occur across different models, for example, it can be clearly
observed that every model fails in predicting the exact birth date of the
person. The factual errors are difficult to detect and correct, but can be
easily mitigated by filling external knowledge into slots.

##### Hallucinated Facts Still Lurk in Templates.

Despite efficiently reducing hallucination, SlotSum still fails to disentangle
all facts from the template. The nationality of Marc Muniesa Martínez is not
explicitly stored in the external fact list, and SlotSum generates the
hallucinated nationality Portuguese as a part of the template rather than
disentangling it as a slot.

To sum up, by disentangling facts from templates and introducing external
knowledge, SlotSum improves the factual correctness in an explainable manner,
and provides guidance on which facts are more important. However, we should be
cautious that the templates cannot completely get rid of the hallucinations,
and further check on factual correctness is still necessary.

##  4 Related Work

###  4.1 Entity Abstract Summarization

Entity abstract summarization aims to generate a concise description of a
given entity from relevant texts gathered from the internet, and is generally
viewed as a variant of multi-document summarization problem. Compared with
common multi-document summarization tasks, the input documents of entity
abstract summarization contain more noise, which will hallucinate the result.

Early pieces of work mainly take an extractive approach. Sauper and Barzilay
(2009) first delves into the entity abstract summarization problem, which
selects useful sentences with Integer Linear Programming (ILP). WikiWrite
Banerjee and Mitra (2016) further takes the coherence scores between selected
sentences into consideration to improve the linguistic quality of the final
summary.

With the development of transformer models, WikiSum Liu et al. (2018) proposes
an extractive-then-abstractive framework. The framework first selects salient
paragraphs according to TF-IDF scores, and then summarizes the selected
paragraphs with a transformer decoder. Meanwhile, WikiSum constructs a large-
scale dataset with the same name, which gathers webpages relevant to the given
entity via search engines as the input document, and views the abstract
section of Wikipedia entries as golden outputs.

Inspired by WikiSum, succeeding models improve the quality of generated
summaries from various aspects. WikiCatSum Perez-Beltrachini et al. (2019)
utilizes the Latent Dirichlet Allocation (LDA) model to render the decoder
topic-aware, and HierSumm Liu and Lapata (2019) adopts a learning-based model
for the extractive stage, and computes the attention between paragraphs to
model the inter-paragraph dependencies. Multiple attempts have also been made
to improve the factual correctness: TWAG Zhu et al. (2021) divides input
documents into different topics to capture different aspects of facts, and
NoisySumm Liu et al. (2021) distills the base model into a student model to
make it more robust to noise. However, these models still suffer from
hallucinations and produce non-factual but seemingly plausible summaries.

###  4.2 Template-based Text Generation

In text generation problems, using templates could give extra control to the
generation process, and improve the readability and stability of output text.
Thus, template-based techniques are widely adopted in various fields like
dialog generation, data-to-text, and even summarization. Different tasks
present different challenges, and the template-based methods are motivated to
design specialized template decomposition frameworks for each unique task. For
example, the data-to-text model TS2 Luo et al. (2020) first constructs slots
with provided data, and then stitches the slots with additional text to form a
fluent sentence; the task-oriented dialogue system NTRD Liang et al. (2021)
generates a template with only one special slot for recommending items, which
acts as textual context to improve the precision of item selection; the
summarization model $\text{Re}^{3}\text{Sum}$ Cao et al. (2018) retrieves
similar summaries in the training dataset as soft templates, improving the
readability and stability of seq2seq summarization systems. To the best of our
knowledge, our work is the first to develop a template-based generation method
for entity abstract summarization. Specifically, we design a facts-template
decomposition framework that explicitly integrates external knowledge to
reduce the hallucinations in entity abstract summarization.

##  5 Conclusion

In this paper, we propose SlotSum, a framework for entity abstract
summarization that reduces hallucinations by disentangling facts. SlotSum
views the summary of an entity as a combination of facts and fact-agnostic
template, and generates the template first. To fill the slots in the template,
SlotSum predicts facts from the input documents, and further corrects the
erroneous predictions with external knowledge sources. In this way, the
hallucinations in generated summaries could be reduced in an explainable and
controllable way.

We combine WikiSum Liu et al. (2018) with WikiBio Lebret et al. (2016) to
construct WikiFactSum, an entity abstract summarization dataset which provides
trustworthy knowledge aside from the input documents. A series of experiments
are conducted on WikiFactSum to compare the performance of SlotSum against
various baseline models. Both automatic and human evaluation results prove
that SlotSum effectively reduces hallucinations after correcting slots with
external knowledge in an explainable manner, while retaining comparable
performance without additional knowledge against the baseline models. In the
future, we will further explore the fusion between knowledge and large
language models, and explore a better approach to explicitly incorporating
knowledge.

## Limitations

Our work is still limited in some aspects. First, our proposed dataset is
limited to the domain of biographies. To generalize our model to broader
domains, more training data is needed.

Second, we do not test the performance of large language models like ChatGPT,
as the large language models may memorize the golden summaries (see Appendix
A). A more complete and more novel dataset is required to study the framework
on LLMs.

##### Potential Risks.

Our proposed dataset WikiFactSum uses WikiSum as the data source, which is
built on a frozen version of Wikipedia. The contents in Wikipedia will change
along with time, and WikiFactSum may contain incorrect information.

## Ethics Statement

In this section, we will discuss the ethical considerations of our work.

##### Licenses and terms.

WikiBio is shared under the CC BY-SA 3.0 license, and WikiAsp is shared under
the CC BY-SA 4.0 license. These datasets are used in entity abstract
summarization by multiple pieces of research, and we believe that they have
been anonymized and desensitized.

##### Human Annotation.

We recruited 3 human annotators to annotate the linguistic qualities and
answer the questions, and 1 expert annotator to determine the golden answer
for questions and score the answers of the 3 annotators. These annotators are
employed by commercial data annotation companies. These recruited annotators
have been paid with mutually agreed rewards under the agreed working time and
price. The annotators have been informed about the usage of annotated data,
which has been recorded in the contract.

##### Intended use.

Researchers are intended to use WikiFactSum to investigate the performance of
entity abstract summarization with trustworthy knowledge. WikiBio and WikiSum
are intended for summarization research, which is compatible with our work.

## Acknowledgements

## References

  * Banerjee and Mitra (2016) Siddhartha Banerjee and Prasenjit Mitra. 2016.  Wikiwrite: generating wikipedia articles automatically.  In _Proceedings of the 25th International Joint Conference on Artificial Intelligence_ , pages 2740–2746.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Cao et al. (2018) Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018.  Retrieve, rerank and rewrite: Soft template based neural summarization.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 152–161.
  * Clarke and Lapata (2010) James Clarke and Mirella Lapata. 2010.  Discourse constraints for document compression.  _Computational Linguistics_ , 36(3):411–441.
  * Hayashi et al. (2021) Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2021.  Wikiasp: A dataset for multi-domain aspect-based summarization.  _Transactions of the Association for Computational Linguistics_ , 9:211–225.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Kryściński et al. (2020) Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016.  Neural text generation from structured data with application to the biography domain.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1203–1213.
  * Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Lewis et al. (2020b) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Liang et al. (2021) Zujie Liang, Huang Hu, Can Xu, Jian Miao, Yingying He, Yining Chen, Xiubo Geng, Fan Liang, and Daxin Jiang. 2021.  Learning neural templates for recommender dialogue system.  _arXiv preprint arXiv:2109.12302_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2018) Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018.  Generating wikipedia by summarizing long sequences.  In _International Conference on Learning Representations_.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Hierarchical transformers for multi-document summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5070–5081.
  * Liu et al. (2021) Yang Liu, Sheng Shen, and Mirella Lapata. 2021.  Noisy self-knowledge distillation for text summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 692–703.
  * Luo et al. (2020) Bingfeng Luo, Zuo Bai, Kunfeng Lai, and Jianping Shen. 2020.  Make templates smarter: A template based data2text system powered by text stitch model.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1057–1062.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919.
  * Perez-Beltrachini et al. (2019) Laura Perez-Beltrachini, Yang Liu, and Mirella Lapata. 2019.  Generating summaries with topic templates and structured convolutional decoders.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5107–5116.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Sauper and Barzilay (2009) Christina Sauper and Regina Barzilay. 2009.  Automatically generating wikipedia articles: a structure-aware approach.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 208–216.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022.  Towards a unified multi-dimensional evaluator for text generation.  _arXiv preprint arXiv:2210.07197_.
  * Zhu et al. (2021) Fangwei Zhu, Shangqing Tu, Jiaxin Shi, Juanzi Li, Lei Hou, and Tong Cui. 2021.  Twag: A topic-guided wikipedia abstract generator.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4623–4635.

Slot | Frequency | Popularity
---|---|---
name | 21743 | 79.44%
birth_date | 21595 | 78.89%
position | 5041 | 18.42%
currentclub | 4817 | 17.6%
occupation | 4595 | 16.79%
fullname | 3712 | 13.56%
death_date | 3286 | 12.0%
birth_place | 3089 | 11.29%
nationality | 2696 | 9.85%
genre | 2338 | 8.54%
Table 5: The 10 most frequent slots in WikiFactSum. Figure 4: Data leakage in
ChatGPT, with the entity Peter Wichers and Marc Muniesa Martinez as examples.
Figure 5: The frequency of different slots in WikiFactSum.

##  Appendix A Data Leakage in LLMs

We do not take large language models like ChatGPT as baseline models, as we
notice obvious data leakage in large language models. Take Figure 4 as an
example, by prompting ChatGPT with simple instructions, the model could
generate summaries similar to the golden summary even without input documents.
A possible explanation is that LLMs remember the Wikipedia summaries, which
act as the golden answers in WikiSum, in the pretraining process. As a result,
LLMs could simply “recall” these memories when queried. We think it is unfair
to compare other models with LLMs due to data leakage, and we do not take LLMs
as baseline models.

##  Appendix B Slot Distribution In WikiFactSum

Figure 5 shows the frequency of slots in WikiFactSum, and Table 5 lists 10
most frequent slot names in WikiFactSum. It can be observed that the frequency
of slots has a long-tailed distribution.