# mFACE: Multilingual Summarization with Factual Consistency Evaluation

Roee Aharoni Shashi Narayan Joshua Maynez
Jonathan Herzig Elizabeth Clark Mirella Lapata
Google Research
<EMAIL_ADDRESS>

###### Abstract

Abstractive summarization has enjoyed renewed interest in recent years, thanks
to pre-trained language models and the availability of large-scale datasets.
Despite promising results, current models still suffer from generating
factually inconsistent summaries, reducing their utility for real-world
application. Several recent efforts attempt to address this by devising models
that automatically detect factual inconsistencies in machine generated
summaries. However, they focus exclusively on English, a language with
abundant resources. In this work, we leverage factual consistency evaluation
models to improve _multilingual_ summarization. We explore two intuitive
approaches to mitigate hallucinations based on the signal provided by a
multilingual NLI model, namely data filtering and controlled generation.
Experimental results in the 45 languages from the XLSum dataset show gains
over strong baselines in both automatic and human evaluation.

mFACE: Multilingual Summarization with Factual Consistency Evaluation

Roee Aharoni Shashi Narayan Joshua Maynez Jonathan Herzig Elizabeth Clark
Mirella Lapata Google Research
<EMAIL_ADDRESS>

##  1 Introduction

The past few years have witnessed a huge leap forward in abstractive
summarization thanks to large-scale pretraining Devlin et al. (2019); Lewis et
al. (2020) and the availability of benchmark datasets. A well-known issue
limiting the wider adoption of abstractive summarization models is their
tendency to generate factually inconsistent summaries, a.k.a “hallucinations”
(Maynez et al., 2020; Zhao et al., 2020, inter alia). A recently popular line
of work explores how to best detect hallucinations in machine generated text,
thereby enabling the automatic identification of factually inconsistent
summaries (Eyal et al., 2019; Falke et al., 2019; Kryscinski et al., 2020;
Wang et al., 2020; Goyal and Durrett, 2021; Scialom et al., 2021; Honovich et
al., 2022; Tang et al., 2022, inter alia).

While such approaches may prove useful for automatic evaluation, it remains
unclear how to best leverage them for improving summarization models _in
multiple languages_. While focusing exclusively on English, previous work
suggests different techniques to this effect such as discarding “noisy”
training examples Gehrmann et al. (2021), contrastive learning paradigms Nan
et al. (2021b), controlled generation and planning Narayan et al. (2021);
Rashkin et al. (2021b), or reinforcement-learning approaches that use the
evaluation model score as a reward function Gunasekara et al. (2021). Despite
promising results, no single method has emerged as a clear winner in English,
let alone across languages with varying amounts of data and resources.

In this work, we focus on leveraging factual consistency evaluation models for
improving summarization systems in multiple languages. Specifically, we employ
Textual Entailment models (a.k.a. Natural Language Inference – NLI) Dagan et
al. (2005); Bowman et al. (2015) in order to determine whether a summary is
factually consistent Maynez et al. (2020); Laban et al. (2021). We
opportunistically opt for NLI given the availability of multilingual
benchmarks for model training Conneau et al. (2018); Nie et al. (2020).
Approaches based on question generation and answering have been also shown to
work well for factuality evaluation (i.e. Scialom et al., 2021; Honovich et
al., 2021; Deutsch et al., 2021), however, they are not easily portable to the
multilingual scenario due to the scarcity of respective resources in languages
other than English.

We first analyze the quality of the training data for summarization models
using a strong multilingual NLI model. In particular, we train our
multilingual NLI model (as evaluated on the XNLI, Conneau et al.; 2018),
following the guidelines from the TRUE survey Honovich et al. (2022) for the
assessment of factual consistency. Focusing on the XLSum multilingual
summarization dataset Hasan et al. (2021), we find that for some languages up
to 70% of training examples are not factually consistent according to the NLI
model, while such examples are commonly used for training. We explore two main
approaches to improve summary quality based on the NLI signal: (1) data
filtering, where we only train on examples where the summaries are predicted
to be entailed by the input, and (2) controlled generation, where we also
leverage “negative” training examples by conditioning the summarization model
on the NLI signal. We evaluate the proposed approaches using automatic metrics
and human-elicited judgments in 45 languages, and observe significant gains in
the faithfulness of the generated summaries over strong baselines.

To summarize, the contributions of this work are three-fold: (1) we analyze
the quality of the XLSum dataset Hasan et al. (2021) using strong multilingual
NLI models and reveal severe issues with faithfulness in the training data
across languages; (2) we explore methods for improving downstream
summarization models trained on this data using a multilingual NLI signal, and
show large gains in both automatic and human evaluation; and (3) we will
release our NLI models, summarization models, and human judgments of summaries
in multiple languages to foster future work on this task. To the best of our
knowledge, our work is the first to examine the faithfulness of summarization
systems in multilingual settings, and we hope it will encourage the
development of better metrics and models in multilingual text generation.

##  2 Related Work

Despite significant improvements in recent years Liu and Lapata (2019); Lewis
et al. (2020); Roberts et al. (2020), abstractive summarization models are
still prone to hallucination, i.e., the inclusion of factual errors in the
generated summaries Song et al. (2018); Maynez et al. (2020); Kryscinski et
al. (2020); Gabriel et al. (2021).

A plethora of approaches have been proposed for the automatic detection of
factual inconsistencies in machine generated text (see Honovich et al. 2022
and Tang et al. 2022 for overviews) with varying degrees of success. There is
growing consensus that techniques based on off-the-shelf textual entailment
Kryscinski et al. (2020); Maynez et al. (2020); Goyal et al. (2021); Goyal and
Durrett (2021) and question generation and answering models Durmus et al.
(2020); Wang et al. (2020); Deutsch et al. (2021); Fabbri et al. (2021);
Scialom et al. (2021); Honovich et al. (2021) achieve strong performance
across tasks and datasets Laban et al. (2021); Honovich et al. (2022). Another
line of recent work uses synthetically generated data to train models on the
factuality detection task Kryscinski et al. (2020); Zhao et al. (2020); Goyal
and Durrett (2020).

Aside from assessing system output, several studies have proposed novel model
architectures which enforce factuality during training or inference. These
include extracting facts from the source and incorporating them as additional
input to the model Cao et al. (2018); Aralikatte et al. (2021); Zhu et al.
(2021), planning using entity chains and avoiding entities that are not in the
input Narayan et al. (2021, 2022), using reinforcement learning to optimize
model training with factual corectnesss as a reward Zhang et al. (2020);
Arumae and Liu (2019); Pasunuru and Bansal (2018); Nan et al. (2021b),
reranking candidate summaries within a beam using entailment predictions Falke
et al. (2019) or quantity verification scores Zhao et al. (2020), using
contrastive learning Cao and Wang (2021); Wan and Bansal (2022), modifying the
training objective to only maximize the likelihood of factual words Goyal and
Durrett (2021), incorporating factuality into the pretraining objective of
models tailored to text summarization tasks Wan and Bansal (2022), and
adaptively removing examples with high log loss Kang and Hashimoto (2020).
Other work simply removes noisy training samples Nan et al. (2021a); Goyal and
Durrett (2021) in the hope that factuality will improve by training on better
examples.

Despite promising results, it is unclear whether previous techniques transfer
to languages beyond English. Our own work aims to improve the factuality of
abstractive summarization across languages. Leveraging recent progress on
multilingual pretrained models Xue et al. (2021), we show that entailment-
based metrics can be trained to detect non-factual instances in multiple
languages, and that this signal can be leveraged to improve summarization
systems in those languages.

##  3 Multilingual Factual Consistency Evaluation

We cast factual consistency evaluation as a Natural Language Inference (NLI)
task. The input forms the premise and the summary forms the hypothesis Maynez
et al. (2020); Laban et al. (2021); Honovich et al. (2022) and the NLI model
is used to predict whether the summary is entailed by the input. More
formally, given input document $d$ and summary $d$, we define an NLI model
$\mathcal{M}$ as a binary classifier, where $\mathcal{M}(d,s)\approx
p(s~{}\textrm{is entailed by}~{}d)$.

Recent studies on evaluating factual consistency in summarization and other
related tasks in English Honovich et al. (2022) have obtained promising
results when finetuning large pretrained models on NLI datasets. Specifically,
they finetune the T5 pretrained encoder-decoder models Raffel et al. (2020)
for binary classification where the entailment relation translates to a
positive label and contradiction/neutral relations are merged to a negative
label. Their model encodes the concatenation of the premise (document) and
hypothesis (summary) and decodes a single token that represents the class
label (entailment or no entailment).111Other works Laban et al. (2021);
Schuster et al. (2022) breaks documents into sentences before running NLI
models, however, we refrained from doing so to avoid loss of context.

Since we are interested in evaluating factual consistency in multiple
languages, we extend the modeling approach of Honovich et al. (2022) to a
multilingual setting. As our pretrained model, we use mT5-XXL Xue et al.
(2021) which was trained on mC4, a dataset drawn from the public Common Crawl
covering 101 languages. We fine-tuned mT5-XXL on the ANLI Nie et al. (2020)
and XNLI Conneau et al. (2018) datasets. ANLI contains 162K English-only
examples, while XNLI has 37K examples222We use the XNLI development set for
training as it was manually curated, unlike the XNLI training set which was
automatically translated. in 15 languages. As mentioned above for the English
case, the multilingual model is trained to generate a binary label when given
the concatenation of a premise and hypothesis, where the positive label
corresponds to an entailment relation, and the negative label stands for a
neutral/contradiction relation. During inference, we score a premise and
hypothesis input by measuring the output probability when force-decoding the
positive label, resulting in a score between 0 (no entailment) and 1
(entailment).

We measured the quality of our multilingual NLI model by evaluating on the
XNLI Conneau et al. (2018) test set and the TRUE benchmark Honovich et al.
(2022). The latter is a standarised collection of datasets representing
various tasks (i.e., summarization, dialog generation, paraphrasing and fact-
checking) with manual annotations for factual consistency. On XNLI, our model
yields an average accuracy of 90.0 over 15 languages in comparison to 87.8
reported in Xue et al. (2021).333The numbers are not fully comparable as Xue
et al. (2021) classify into three classes while we use binary classification.
We present results for individual languages in Appendix A. On the (English-
only) TRUE benchmark, our model’s average ROC AUC is 82.4 in comparison to
83.4 reported in Honovich et al. (2022) for their best performing English-
only, T5-11B model Raffel et al. (2020) trained on ANLI. While our model is
trained on both ANLI (English) and XNLI (15 languages, detailed in Table 1),
we assume it can generalize to additional languages (for which NLI data is not
available) due to the nature of the pretrained model (mT5, trained on 101
languages).

Language | # Train | # Entailed | % Entailed
---|---|---|---
Amharic | 5,761 | 2,458 | 42.67%
Arabic$~{}\clubsuit$ | 37,519 | 16,954 | 45.19%
Azerbaijani | 6,478 | 2,212 | 34.15%
Bengali | 8,102 | 3,936 | 48.58%
Burmese | 4,569 | 1,484 | 32.48%
Chinese (simp.)$~{}\clubsuit$ | 37,362 | 18,984 | 50.81%
Chinese (trad.)$~{}\clubsuit$ | 37,373 | 18,906 | 50.59%
English$~{}\clubsuit$ | 306,522 | 183,918 | 60.00%
French$~{}\clubsuit$ | 8,697 | 3,214 | 36.96%
Gujarati | 9,119 | 2,699 | 29.60%
Hausa | 6,418 | 2,530 | 39.42%
Hindi$~{}\clubsuit$ | 70,778 | 30,913 | 43.68%
Igbo | 4,183 | 1,491 | 35.64%
Indonesian | 38,242 | 20,068 | 52.48%
Japanese | 7,113 | 4,905 | 68.96%
Kirundi | 5,746 | 2,544 | 44.27%
Korean | 4,407 | 2,156 | 48.92%
Kyrgyz | 2,266 | 736 | 32.48%
Marathi | 10,903 | 3,192 | 29.28%
Nepali | 5,808 | 2,996 | 51.58%
Oromo | 6,063 | 2,471 | 40.76%
Pashto | 14,353 | 6,434 | 44.83%
Persian | 47,251 | 23,243 | 49.19%
Pidgin | 9,208 | 35,99 | 39.09%
Portuguese | 57,402 | 2,3069 | 40.19%
Punjabi | 8,215 | 2,324 | 28.29%
Russian$~{}\clubsuit$ | 62,243 | 25,208 | 40.50%
Scottish Gaelic | 1,313 | 557 | 42.42%
Serbian (Cyrillic) | 7,275 | 2,261 | 31.08%
Serbian (Latin) | 7,276 | 2,227 | 30.61%
Sinhala | 3,249 | 1,187 | 36.53%
Somali | 5,962 | 2,295 | 38.49%
Spanish$~{}\clubsuit$ | 38,110 | 13,450 | 35.29%
Swahili$~{}\clubsuit$ | 7,898 | 3,383 | 42.83%
Tamil | 16,222 | 8,213 | 50.63%
Telugu | 10,421 | 3,667 | 35.19%
Thai$~{}\clubsuit$ | 6,616 | 3,099 | 46.84%
Tigrinya | 5,451 | 2,013 | 36.93%
Turkish$~{}\clubsuit$ | 27,176 | 12,041 | 44.31%
Ukrainian | 43,201 | 16,535 | 38.27%
Urdu$~{}\clubsuit$ | 67,665 | 28,536 | 42.17%
Uzbek | 4,728 | 1,721 | 36.40%
Vietnamese$~{}\clubsuit$ | 32,111 | 11,600 | 36.12%
Welsh | 9,732 | 4,806 | 49.38%
Yoruba | 6,350 | 2,359 | 37.15%
Avg. | 24,670 | 11,675 | 41.37%

Table 1: Statistics on the XLSum training data, showing the total number of
examples for each language, and the number and ratio of examples where the
summary was entailed by the input. Languages that appear both in XLSum and
XNLI are denoted with $\clubsuit$, while for other languages NLI
classification is zero-shot.

##  4 Summarization Models

We next describe two summarization approaches which exploit the factual
consistency evaluation signal provided by the multilingual NLI model.

###  4.1 Data Filtering

An intuitive approach to improving the factuality of machine generated
summaries is to enhance the quality of the training data, simply by filtering
noisy training samples Nan et al. (2021a); Goyal and Durrett (2021). More
formally, given a training corpus $\mathbb{D}$ of input document-summary
pairs, we find $\mathbb{D^{+}}\subset\mathbb{D}$ such that for each document-
summary pair $(d,s)\in\mathbb{D^{+}}$, $p(s~{}\textrm{is entailed
by}~{}d)>0.5$.

We used our multilingual NLI model (see §3) to annotate the training data
portion of XLSum Hasan et al. (2021) in all 45 languages. Table 1 shows the
number and ratio of examples where the summary was predicted to be entailed by
the input (using a threshold of 0.5 on the NLI model score). The ratio of
entailed summaries ranges from 68.96% (for Japanese) down to 28.29% (for
Punjabi). For all languages but 3 (English, Japanese, Nepali) less than half
of the training summaries were predicted to be entailed by the input according
to the NLI model. We find these numbers strikingly low; this may be due to the
nature of the dataset, since the relationship between news headlines and their
corresponding article can be somewhat loose (e.g., headlines may include
“clickbate” and additional details that are not mentioned in the article).
Another reason might be errors of the NLI model (while it was shown to work
well on TRUE/XNLI, XLSum may be from a different distribution). Overall, the
results in Table 1 indicate that relying on the NLI signal by filtering
training data can have a large impact on the resulting summarization model. On
one hand, training a summarization model only on the entailed portion of the
training data may result in more entailed summaries. On the other hand, such
model unavoidably sees fewer examples (e.g., 557 instances for Scottish Gaelic
after filtering, while the original training set includes 1,313 examples)
which might in turn have a negative effect on summary quality.

###  4.2 Controlled Generation

Another way to leverage the NLI signal for improving the summarization model
is via controlled generation Keskar et al. (2019); Rashkin et al. (2021b). In
this approach, special tokens are prepended to the model’s input to
indicate/control whether the output should be entailed or not.

Given a training corpus $\mathbb{D}$ of input document-summary pairs, we
annotate each document-summary pair $(d,s)\in\mathbb{D}$ as $(d^{\prime},s)$,
where $d^{\prime}$ is “<entailed> $d$” if $p(s~{}\textrm{is entailed
by}~{}d)>0.5$ or “<not-entailed> $d$”, otherwise. The model trained with the
augmented $\mathbb{D}$ is expected to learn the correlation between entailment
and the special token value, and as a result can be “controlled” to produce
more faithful summaries by prepending the token that corresponds to faithful
(aka entailed) summaries in inference time. This method _implicitly_ teaches
the model to learn from the entailment signal while taking advantage of all
available training data. It may, however, be more sensitive to wrong
predictions by the entailment model as noisy examples are not discarded.

##  5 Experimental Setup

Our study focuses on the recently introduced multilingual summarization
dataset (XLSum, Hasan et al.; 2021) which extends techniques from the English-
only XSum dataset Narayan et al. (2018a) to construct a large-scale
multilingual dataset comprising 1 million professionally annotated article-
summary pairs from BBC, covering 45 languages.

###  5.1 Model Details

We finetuned three models based on mT5 XXL Xue et al. (2021). The first is a
“Vanilla” model which is trained on the XLSum data as-is. As previous work has
shown that multilingual training improves performance for low-resource
languages (Aharoni et al., 2019; Hasan et al., 2021), we also follow this
setting and finetune a single massively multilingual model for all 45
languages in XLSum. The second model (“Filtered”) is finetuned only on the
portion of the data that passed the multilingual NLI filter. The third model
(“Controlled”) is trained on all data, using the controlled generation
approach mentioned above. Specifically, for control tokens “<entailed>” and
“<not-entailed>”, we used two extra spare tokens from the mT5 vocabulary and
prepended them to the input (Keskar et al., 2019; Rashkin et al., 2021b).
During inference, we always prepend the input with “<entailed>” and report on
the whole development and test sets.

###  5.2 System Comparisons

We compare the above approaches to three additional baselines. First, a
“Random Selection” baseline, in which we randomly select the same number of
examples as the number of examples who passed the NLI filter, per language. We
then fine-tune a model similarly to the “Filtered” model above using this
randomly selected data. Second, we introduce a “Self-ROUGE” baseline, in which
we select the examples where the ROUGE of the summary with respect to the
input document is highest. Again, we select the same number of examples as the
number of examples who passed the NLI filter, and fine-tune a model on this
data. Finally, we evaluate the model outputs from the original XLSum paper
Hasan et al. (2021) which finetuned an mT5-base pretrained model.

###  5.3 Automatic Evaluation

We report on ROUGE Lin (2004) which is commonly used to measure the
informativeness and fluency of model generated summaries against gold-standard
references. We also report on reference-free NLI score (Maynez et al., 2020;
Honovich et al., 2022, inter alia) for faithfulness. Specifically, since there
are no tokenizers available for many of the languages in XLSum, we report
ROUGE-L computed using the sentencepiece tokenization of mT5. Regarding NLI
scores, we compute for each summary whether it is entailed by the input, and
report the average over all examples in a partition (test or development set).

Model | Best-NLI | Best-ROUGE
---|---|---
Vanilla | 3,600 | 15,000
Filtered | 2,200 | 12,000
Controlled | 3,400 |  8,800
Table 2: Number of fine-tuning steps for best checkpoint for each model
according to NLI and ROUGE on the XLSum development set.

#### Checkpoint Selection using ROUGE and NLI

Ideally, we would like to evaluate a _single_ model checkpoint for _all_
languages; the best checkpoint is often selected using ROUGE in literature.
However, we also employ the NLI scores to quantify improvements in
faithfulness. For each model we select two checkpoints that are best according
to ROUGE and NLI (on development set), when averaged across all languages.
Table 2 summarizes the number of finetuning steps that led to the best
checkpoints for each model according to ROUGE and NLI. In general, we observe
for all models that best NLI checkpoints are earlier than ROUGE-based ones.

[Quality] Is the summary comprehensible? Incomprehensible: The summary is
difficult to understand. It can have serious grammatical errors, low fluency,
and/or repeated information. Somewhat Comprehensible: The summary generally
makes sense but suffers from grammatical errors, low fluency, and/or repeated
information. Comprehensible: The summary is understandable. It does not
exhibit any grammatical errors, disfluencies, and/or repeated information.
---
[Attribution] Is all the information in the summary fully attributable to the
article? Yes, it is attributable: Select this option if it is accurate to say,
“The provided news article says…” or “According to the news article…” with the
summary following this phrase. No, not fully attributable: Select this option
if only some of the information is supported in the news article, but other
parts of the information are missing from the news article or not an accurate
representation.
[Informativeness] Is the summary a good summary of the article? Bad summary:
The summary does not capture the important information in the article, or the
captured information is not accurate with the article. It can also exhibit
grammatical issues, low fluency, and/or repeated information. Good Summary:
The summary captures the important information in the article and presents it
accurately and concisely. It does not exhibit any grammatical errors,
disfluencies, and/or repeated information.

Figure 1: Instructions that were used in our large scale human elicitation
study.

| Vanilla | Filtered | Controlled
---|---|---|---
| Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI
Language | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI
Amharic | 35.60 | 72.56 | 34.91 | 78.71 | 35.15 | 74.65 | 34.55 | 79.74 | 35.1 | 75.52 | 35.22 | 79.12
Arabic | 32.00 | 55.15 | 33.31 | 64.02 | 33.33 | 72.83 | 33.05 | 74.88 | 31.67 | 71.11 | 33.34 | 71.68
Azerbaijani | 29.95 | 62.15 | 29.07 | 62.2 | 29.28 | 68.03 | 29.26 | 73.41 | 27.99 | 66.75 | 29.0 | 67.03
Bengali | 34.19 | 71.12 | 33.71 | 74.49 | 34.17 | 76.53 | 33.38 | 81.14 | 34.14 | 73.64 | 33.81 | 73.48
Burmese | 41.40 | 68.54 | 39.83 | 69.56 | 40.58 | 73.68 | 39.08 | 76.26 | 40.55 | 70.74 | 40.78 | 74.32
Chinese (simp.) | 33.54 | 62.43 | 33.08 | 69.07 | 35.66 | 76.95 | 34.15 | 81.3 | 32.48 | 78.83 | 34.44 | 77.72
Chinese (trad.) | 34.39 | 63.10 | 34.37 | 69.02 | 36.49 | 78.15 | 34.43 | 79.48 | 32.86 | 77.50 | 34.95 | 77.95
English | 32.51 | 68.31 | 32.93 | 74.23 | 33.23 | 80.32 | 32.4 | 84.4 | 33.07 | 81.75 | 33.14 | 82.99
French | 34.12 | 64.88 | 33.87 | 67.61 | 34.49 | 72.94 | 33.98 | 76.4 | 33.18 | 77.11 | 34.35 | 72.59
Gujarati | 33.21 | 62.52 | 32.19 | 62.12 | 32.59 | 63.54 | 32.46 | 71.64 | 32.23 | 62.38 | 32.74 | 62.85
Hausa | 38.02 | 60.59 | 36.51 | 63.07 | 37.3 | 67.47 | 36.75 | 72.18 | 37.25 | 66.37 | 36.97 | 64.58
Hindi | 36.88 | 66.74 | 36.92 | 69.77 | 37.45 | 74.20 | 36.97 | 77.26 | 36.17 | 73.31 | 37.26 | 74.70
Igbo | 34.17 | 56.01 | 35.45 | 60.22 | 35.44 | 64.36 | 34.19 | 72.00 | 35.3 | 66.77 | 35.32 | 70.01
Indonesian | 34.47 | 67.63 | 34.05 | 72.5 | 35.26 | 79.06 | 33.76 | 82.31 | 34.05 | 78.27 | 34.66 | 78.65
Japanese | 41.19 | 76.81 | 40.86 | 79.35 | 41.30 | 78.60 | 39.52 | 81.29 | 41.0 | 87.84 | 40.65 | 85.60
Kirundi | 33.70 | 65.50 | 32.67 | 66.31 | 33.63 | 70.66 | 31.74 | 75.37 | 33.05 | 72.57 | 32.73 | 67.72
Korean | 40.36 | 72.05 | 39.28 | 74.00 | 39.83 | 78.51 | 38.24 | 82.18 | 39.9 | 80.02 | 39.28 | 78.12
Kyrgyz | 26.48 | 67.57 | 27.02 | 65.58 | 26.54 | 67.99 | 25.41 | 78.02 | 26.82 | 73.01 | 26.70 | 73.39
Marathi | 30.11 | 61.84 | 28.08 | 64.23 | 31.31 | 67.86 | 30.95 | 70.41 | 28.85 | 66.95 | 27.75 | 63.77
Nepali | 39.25 | 76.43 | 38.8 | 78.46 | 38.67 | 80.57 | 37.48 | 85.65 | 38.42 | 80.86 | 37.40 | 77.18
Oromo | 30.67 | 68.92 | 28.76 | 72.55 | 29.61 | 76.35 | 28.76 | 79.62 | 29.53 | 71.34 | 29.62 | 73.94
Pashto | 39.07 | 70.01 | 39.04 | 70.9 | 39.88 | 76.34 | 38.83 | 80.21 | 38.45 | 72.46 | 38.85 | 73.72
Persian | 35.11 | 71.68 | 36.47 | 71.57 | 37.86 | 79.22 | 36.73 | 80.7 | 34.16 | 78.50 | 35.82 | 80.23
Pidgin | 34.33 | 62.07 | 32.95 | 67.91 | 34.41 | 72.03 | 33.07 | 76.32 | 34.07 | 70.35 | 33.82 | 69.85
Portuguese | 33.3 | 52.89 | 32.88 | 57.12 | 33.9 | 64.24 | 33.14 | 64.91 | 32.76 | 68.01 | 33.16 | 68.12
Punjabi | 36.12 | 60.96 | 35.71 | 61.95 | 36.39 | 64.61 | 35.73 | 67.67 | 35.33 | 61.34 | 35.64 | 61.57
Russian | 28.48 | 58.42 | 28.38 | 62.88 | 28.71 | 69.20 | 28.26 | 72.51 | 28.09 | 71.88 | 28.58 | 70.54
Scottish Gaelic | 33.41 | 57.38 | 33.01 | 66.48 | 32.95 | 68.14 | 32.71 | 76.06 | 33.47 | 71.90 | 32.57 | 66.07
Serbian (Cyrillic) | 28.31 | 56.71 | 27.32 | 62.36 | 29.20 | 62.97 | 27.93 | 69.32 | 28.14 | 66.63 | 27.68 | 62.71
Serbian (Latin) | 26.67 | 51.23 | 25.49 | 54.11 | 28.21 | 59.93 | 26.84 | 65.08 | 25.74 | 62.00 | 24.09 | 51.26
Sinhala | 36.33 | 71.25 | 35.72 | 74.55 | 36.04 | 74.76 | 35.67 | 81.58 | 35.53 | 73.61 | 36.03 | 74.28
Somali | 31.77 | 58.23 | 31.04 | 66.32 | 31.35 | 69.42 | 30.59 | 76.42 | 31.36 | 67.22 | 30.99 | 68.14
Spanish | 27.40 | 53.33 | 26.38 | 58.58 | 27.33 | 62.34 | 26.7 | 69.32 | 27.06 | 61.36 | 26.92 | 57.55
Swahili | 35.64 | 64.57 | 35.93 | 69.28 | 36.46 | 73.25 | 35.23 | 78.58 | 35.59 | 73.35 | 36.05 | 68.49
Tamil | 31.94 | 70.37 | 30.31 | 77.9 | 33.26 | 81.91 | 30.99 | 83.11 | 31.49 | 82.14 | 30.93 | 81.76
Telugu | 29.35 | 60.61 | 28.61 | 67.74 | 29.11 | 67.83 | 28.88 | 75.10 | 29.11 | 66.30 | 28.4 | 66.54
Thai | 30.59 | 61.83 | 29.92 | 68.77 | 30.66 | 73.27 | 29.63 | 78.95 | 29.1 | 74.01 | 30.42 | 71.33
Tigrinya | 36.73 | 69.83 | 35.73 | 73.32 | 36.37 | 76.71 | 34.01 | 78.24 | 35.63 | 72.37 | 35.24 | 72.27
Turkish | 30.80 | 62.36 | 32.19 | 64.58 | 33.65 | 74.66 | 32.45 | 75.65 | 30.81 | 74.60 | 32.5 | 72.57
Ukrainian | 28.34 | 58.83 | 28.61 | 63.15 | 28.99 | 70.87 | 27.66 | 73.64 | 28.01 | 72.41 | 28.42 | 72.61
Urdu | 36.93 | 67.84 | 37.12 | 69.5 | 38.18 | 74.81 | 37.05 | 77.31 | 36.14 | 73.41 | 37.45 | 75.64
Uzbek | 28.37 | 67.41 | 29.02 | 69.66 | 28.06 | 70.62 | 28.27 | 78.32 | 27.39 | 70.40 | 27.76 | 67.74
Vietnames | 35.91 | 65.94 | 35.77 | 67.54 | 36.69 | 73.01 | 35.46 | 76.94 | 34.99 | 70.91 | 35.86 | 71.52
Welsh | 35.31 | 65.27 | 33.75 | 65.32 | 33.89 | 76.06 | 32.47 | 76.11 | 34.33 | 71.58 | 33.66 | 71.47
Yoruba | 37.65 | 63.91 | 36.01 | 63.22 | 37.01 | 68.48 | 35.15 | 75.41 | 35.63 | 68.52 | 36.78 | 68.55
Average | 33.65 | 64.30 | 33.18 | 67.82 | 34.00 | 72.17 | 32.98 | 76.49 | 33.00 | 72.17 | 33.28 | 71.38

Table 3: ROUGE (ROUGE-L in particular) and NLI scores per language on the
XLSum test set for Best-ROUGE and Best-NLI checkpoints, with the highest ROUGE
and NLI on the development set, respectively. Highest ROUGE and NLI numbers in
each row are in bold.

###  5.4 Human Evaluation

In addition to automatic evaluation, we conducted a large scale human
elicitation study assessing different dimensions of output quality in all 45
languages. Firstly, we asked participants to read a system summary and assess
its Quality (Is the summary comprehensible?), without looking at the source
article, using a 1–3 rating scale where 3 means fully understandable and 1
indicates the summary has serious fluency errors. After the first assessment,
participants were shown the source article and were asked to rate the summary
according to:  _Attribution_ (Is all the information in the summary fully
attributable to the article?) using the attribution definition provided in
Rashkin et al. (2021a); and  _Informativeness_ (Is the summary a good summary
of the article?). Both assessments used binary judgements and we report on the
percentage of times each system was rated positively. Figure 1 presents a
detailed description of our elicitation study.444We also present our template
in the Appendix (Figure 2) that was presented to our participants.

In order to evaluate summarization output in such a diverse multilingual
setting, we have taken several measures to scale our study to 45 languages
while maintaining high inter-annotator agreement. First, we have written our
instructions in English and invited bilingual participants (native speakers of
the respective language who are also proficient in English) to take part in
our study. Second, each participant had to pass a Screener test consisting of
25 questions with an accuracy of 85% before they can take part in the study.
Finally, we conducted two pilot studies before the final evaluation to provide
participants feedback and improve their agreements. Our final elicitation
study is conducted using 100 instances per language, each randomly sampled
from their respective test sets. We collected ratings from three different
annotators for each data point.

##  6 Results

#### Filtered Model Performs Best on NLI-based Evaluation

Table 3 presents our final results on the test set for all languages, for our
three model variants (Vanilla, Filtered, and Controlled), and for the two
checkpoint selection methods (Best-ROUGE and Best-NLI).555The corresponding
validation results can be found in Appendix, Table 13. We see that the
Filtered model outperforms all other models across languages achieving an
average score of 76.49 on the test set for the Best-NLI checkpoint (last row
in Table 3), obtaining best NLI scores in 43 of the 45 languages. This
suggests that data filtering is a viable approach for improving the factual
consistency of summarization systems across languages. The next best model in
terms of NLI is the Best-ROUGE checkpoint of the Filtered and Controlled
variants, both achieving an average NLI score of 72.17.

The other two model variants (Controlled, Vanilla) perform mostly worse than
the Filtered model in terms of NLI: 71.38 (Best-NLI) and 72.17 (Best-ROUGE)
for the Controlled model checkpoints, and 67.82 (Best-NLI) and 64.30 (Best-
Rouge) for the Vanilla model. Note how the NLI score gap between the Best-
ROUGE Vanilla model and the Best-NLI Filtered model is 12.19 points on
average, which we find significant considering that the Vanilla model is not a
strawman, being based on T5-XXL.

The Best-NLI checkpoint outperforms the Best-ROUGE checkpoint for the Filtered
model (average NLI scores of 76.49 vs 72.17). However, we observe a
degradation of 0.79 NLI points when comparing the Best-NLI and Best-ROUGE
checkpoints for the Controlled model (average NLI scores of 71.38 vs 72.17).

| Vanilla | Filtered | Controlled
---|---|---|---
| Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI
Language | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI
English | 32.51 | 68.31 | 32.93 | 74.23 | 33.23 | 80.32 | 32.4 | 84.4 | 33.07 | 81.75 | 33.14 | 82.99
Varying Number of Training Resource
High | 33.21 | 63.42 | 33.25 | 67.26 | 32.29 | 69.24 | 31.23 | 67.79 | 28.53 | 64.79 | 29.37 | 64.80
Medium | 33.21 | 63.22 | 32.37 | 66.37 | 33.46 | 69.68 | 32.54 | 74.09 | 32.64 | 69.96 | 32.39 | 67.52
Low | 34.38 | 65.89 | 33.69 | 69.00 | 34.03 | 71.98 | 33.03 | 77.61 | 33.65 | 71.88 | 33.69 | 71.40
Language Families
Indo-European | 32.62 | 63.23 | 32.11 | 66.22 | 33.07 | 69.44 | 32.32 | 74.48 | 32.02 | 69.61 | 32.20 | 68.18
Romance | 31.61 | 57.03 | 31.04 | 61.10 | 31.91 | 66.51 | 31.27 | 70.21 | 31.00 | 68.83 | 31.48 | 66.09
Turkic | 28.90 | 64.87 | 29.32 | 65.50 | 29.38 | 70.32 | 28.85 | 76.35 | 28.25 | 71.19 | 28.99 | 70.18
Semitic | 34.78 | 65.85 | 34.65 | 72.02 | 34.95 | 74.73 | 33.87 | 77.62 | 34.13 | 73.00 | 34.60 | 74.36
Afro-Asiatic | 33.49 | 62.58 | 32.10 | 67.31 | 32.75 | 71.08 | 32.03 | 76.07 | 32.71 | 68.31 | 32.53 | 68.89
Indo-Iranian | 36.77 | 67.55 | 37.07 | 68.14 | 38.04 | 73.39 | 37.10 | 76.19 | 35.98 | 70.77 | 36.77 | 71.84
XNLI Candidacy
XNLI Lang | 33.01 | 62.68 | 33.09 | 67.30 | 34.03 | 73.53 | 33.06 | 77.15 | 32.40 | 73.78 | 33.48 | 72.71
Non-XNLI Lang | 33.90 | 64.97 | 33.21 | 68.03 | 33.99 | 71.63 | 32.94 | 76.23 | 33.27 | 71.52 | 33.20 | 70.83

Table 4: ROUGE and NLI scores on the XLSum test set for the best checkpoints
averaged across language groups with 1) varying number of training resources,
2) language families and 3) XNLI candidacy. For training resources, we
consider three groups: High ([70K–10K]), Medium ([10K–8K]) and Low (less than
6K) numbers of training examples. For language families, the Indo-European
language cluster represents Bengali, Gujarati, Hindi, russian, Serbian
(cyrillic and Latin), and Sinhala; the Romance cluster comprises of French,
Protuguese, and Spanish; the Turkic cluster contains Azerbaijani, Kyrgyz,
Turkish, and Uzpek; Semitic languages are Amharic, Arabic, and Tigrinya; the
Afro-Asiatic cluster groups together Hausa, Oromo, and Somali; finally, the
Indo-Iranian cluster represents Pashto, Persian, and Punjabi; we omit clusters
with two members and singletons. Finally, we divide XLSum languages in two
subsets: the XNLI Lang comprises of the XLSum languages that appear in the
XNLI dataset used to train our multilingual NLI model (see §3), and the Non-
XNLI Lang, the XLSum languages that don’t appear in the XNLI dataset. Highest
ROUGE and NLI numbers in each row are in bold.

#### Effect of Entailment Signals on ROUGE

As shown above, NLI scores improve in all languages when training using the
signal from the NLI model, either by filtering data or by using controlled
generation. But what is the effect on ROUGE? Looking at the average ROUGE
scores across all languages in Table 3, we again see that the best average
score is obtained by the Filtered model, using the Best-ROUGE checkpoint
(34.00). Interestingly, this model is trained on much fewer examples, but
obtains better results than the Vanilla and Controlled variants that use all
training examples in XLSum. In addition, this model obtains higher or
comparable NLI scores (72.17 and 76.49, for Best-ROUGE and Best-NLI,
respectively) than the other models, suggesting that it is more accurate with
respect to the reference summaries and more faithful with respect to the
input. In general, all 6 checkpoints obtain very similar ROUGE scores, ranging
between 32.98 and 34.00, while the range of NLI scores is much larger (64.30
to 76.49).

#### ROUGE and NLI across Different Language Groups

Table 4 shows the results clustered by (1) the number of training examples per
language, (2) language families and (3) and whether XNLI training data is
available. Using available training resources, we cluster languages into three
clusters – High (10k–70k examples), Medium (9k–16k examples) and Low (less
than 6k examples). We also report the results on English for comparison, as it
is the language with the largest number of examples (307k). The language
familiy clusters the results based on language families, including Indo-
European, Romance, Turkic, Semitic, Afro-Asiatic, and Indo-Iranian languages.
Finally, we present results clustered in two subsets: the languages that
appear in the XNLI dataset used to train our multilingual NLI model, and the
languages that don’t appear in the XNLI dataset. See Table 10 in Appendix for
how different languages were assigned to different groups.

Again, the conclusions are consistent, with the Filtered model outperforming
the others in most cases, including English. Vanilla scores better on ROUGE
for Low resource and Afro-Asiatic languages, although the difference is less
than 1 ROUGE point from the other models. We further observe that the
Controlled model does not outperform Filtered or Vanilla in any configuration,
irrespectively of how languages are grouped into clusters. In conclusion, we
find that the Filtered model dramatically improves the NLI score, while
maintaining similar ROUGE performance with respect to the other models.

| Vanilla | Filtered | Controlled |
---|---|---|---|---
Language | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Human
English | 0.91 | 0.92 | 0.91 | 0.92 | 0.89 | 0.91 | 0.9
Varying Number of Training Resource
High | 0.84 | 0.84 | 0.86 | 0.87 | 0.83 | 0.87 | 0.89
Medium | 0.85 | 0.85 | 0.84 | 0.86 | 0.85 | 0.86 | 0.86
Low | 0.83 | 0.82 | 0.83 | 0.84 | 0.82 | 0.83 | 0.87
Language Families
Indo-European | 0.81 | 0.82 | 0.81 | 0.84 | 0.82 | 0.83 | 0.86
Romance | 0.93 | 0.94 | 0.94 | 0.95 | 0.9 | 0.96 | 0.96
Turkic | 0.84 | 0.82 | 0.84 | 0.86 | 0.82 | 0.84 | 0.85
Semitic | 0.83 | 0.84 | 0.84 | 0.85 | 0.85 | 0.86 | 0.90
Afro-Asiatic | 0.82 | 0.79 | 0.81 | 0.83 | 0.83 | 0.78 | 0.89
Indo-Iranian | 0.74 | 0.75 | 0.74 | 0.76 | 0.74 | 0.74 | 0.70
XNLI Training Data
XNLI Lang | 0.85 | 0.85 | 0.86 | 0.88 | 0.84 | 0.87 | 0.89
Non-XNLI Lang | 0.84 | 0.84 | 0.84 | 0.86 | 0.84 | 0.85 | 0.87

Table 5: Human evaluation results for Summary Quality for the best checkpoints
averaged across language groups with 1) varying number of training resources,
2) language families and 3) XNLI candidacy. See Table 4 for more details about
different groups. Best results in each row are in bold.

| Vanilla | Filtered | Controlled |
---|---|---|---|---
Language | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Human
English | 0.38 | 0.45 | 0.46 | 0.52 | 0.46 | 0.4 | 0.3
Varying Number of Training Resource
High | 0.41 | 0.46 | 0.46 | 0.52 | 0.47 | 0.47 | 0.29
Medium | 0.47 | 0.49 | 0.50 | 0.54 | 0.53 | 0.5 | 0.34
Low | 0.43 | 0.46 | 0.44 | 0.51 | 0.46 | 0.45 | 0.3
Language Families
Indo-European | 0.48 | 0.5 | 0.47 | 0.55 | 0.53 | 0.5 | 0.36
Romance | 0.42 | 0.47 | 0.47 | 0.56 | 0.5 | 0.48 | 0.28
Turkic | 0.53 | 0.54 | 0.57 | 0.60 | 0.54 | 0.57 | 0.37
Semitic | 0.39 | 0.46 | 0.47 | 0.51 | 0.50 | 0.46 | 0.29
Afro-Asiatic | 0.42 | 0.44 | 0.45 | 0.49 | 0.43 | 0.45 | 0.35
Indo-Iranian | 0.45 | 0.47 | 0.44 | 0.47 | 0.49 | 0.46 | 0.27
XNLI Training Data
XNLI Lang | 0.44 | 0.49 | 0.49 | 0.56 | 0.52 | 0.50 | 0.32
Non-XNLI Lang | 0.44 | 0.46 | 0.45 | 0.51 | 0.48 | 0.46 | 0.30

Table 6: Human evaluation results for Attribution for the best checkpoints
averaged across language groups with 1) varying number of training resources,
2) language families and 3) XNLI candidacy. See Table 4 for more details about
different groups. Best results in each row are in bold.

| Vanilla | Filtered | Controlled |
---|---|---|---|---
Language | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Human
English | 0.33 | 0.39 | 0.41 | 0.45 | 0.38 | 0.33 | 0.27
Varying Number of Training Resource
High | 0.34 | 0.38 | 0.38 | 0.44 | 0.38 | 0.4 | 0.26
Medium | 0.40 | 0.43 | 0.42 | 0.46 | 0.44 | 0.43 | 0.30
Low | 0.36 | 0.37 | 0.37 | 0.43 | 0.39 | 0.38 | 0.27
Language Families
Indo-European | 0.41 | 0.44 | 0.39 | 0.47 | 0.45 | 0.43 | 0.32
Romance | 0.3 | 0.35 | 0.34 | 0.41 | 0.34 | 0.34 | 0.21
Turkic | 0.48 | 0.5 | 0.52 | 0.56 | 0.50 | 0.53 | 0.35
Semitic | 0.32 | 0.36 | 0.39 | 0.41 | 0.41 | 0.39 | 0.26
Afro-Asiatic | 0.34 | 0.37 | 0.37 | 0.42 | 0.39 | 0.35 | 0.31
Indo-Iranian | 0.38 | 0.41 | 0.39 | 0.42 | 0.40 | 0.41 | 0.24
XNLI Training Data
XNLI Lang | 0.36 | 0.40 | 0.39 | 0.45 | 0.41 | 0.41 | 0.28
Non-XNLI Lang | 0.38 | 0.39 | 0.39 | 0.44 | 0.41 | 0.40 | 0.27

Table 7: Human evaluation results for Informativeness for the best checkpoints
averaged across language groups with 1) varying number of training resources,
2) language families and 3) XNLI candidacy. See Table 4 for more details about
different groups. Best results in each row are in bold.

#### Human Assessment for Quality, Attribution and Informativeness

Tables 5, 6, and 7 present our human evaluation results for Quality,
Attribution and Informativeness, respectively. We use the same language groups
as in Table 4. See Tables 15, 16 and 17 in Appendix for per language results.
Finally, Table 8 presents all automatic and human evaluation results averaged
across all languages for a side-by-side comparison.

Unsurprisingly, human written summaries were more understandable than Vanilla,
Filtered or Controlled summaries, with least fluency-related issues. But, the
summaries generated using our Filtered Best-NLI model were most attributed (or
faithful) and informative, to their input documents. Our human evaluation
results strengthen our automatic evaluations that Filtered model are better at
generating faithful and informative summaries, with respect to the other
models.

| Vanilla | Filtered | Controlled |
---|---|---|---|---
Language | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Human
Length Ratio | 0.93 | 0.88 | 0.89 | 0.87 | 1.00 | 0.81 | 1.00
ROUGE | 33.65 | 33.18 | 34.00 | 32.98 | 33.02 | 33.28 | –
NLI | 64.31 | 67.82 | 72.18 | 76.50 | 72.18 | 71.38 | –
Quality | 0.85 | 0.84 | 0.85 | 0.86 | 0.84 | 0.86 | 0.88
Attribution | 0.44 | 0.47 | 0.46 | 0.52 | 0.49 | 0.47 | 0.31
Informativeness | 0.37 | 0.40 | 0.39 | 0.45 | 0.41 | 0.40 | 0.27

Table 8: Final average results on the XLSum test set for a side-by-side
comparison. For Length Ratio, we compute an average of ratios of predicted
length and target length across all test examples. Best results in each row
are in bold.

#### Effect on Summary Length

One may argue that we may be improving faithfulness by favoring shorter
summaries. To study this, we report predicted length and target length ratios
averaged across all test examples, for different models. Results are presented
in Table 8.

Best-NLI checkpoints do yield a reduction in predicted length across different
models compared to their Best-ROUGE checkpoints; the length ratios drop from
0.93 to 0.88 for Vanilla, from 0.89 to 0.87 for Filtered, and from 1.00 to
0.81 for Controlled. However, shorter summaries are not necessarily more
faithful; the worst length ratio (0.81) is for the Controlled Best-NLI model
which performs worse on NLI, Attribution, and Informativeness, compared to the
Filtered Best-NLI model with a higher length ratio (0.87). The Filtered Best-
NLI model only yields a marginal reduction in summary length compared to the
Vanilla Best-Rouge summaries (Length ratio: 0.87 vs 0.93), but improves on NLI
scores (76.50 vs 64.31), Quality (0.86 vs 0.85), Attribution (0.52 vs 0.44),
and Informativeness (0.45 vs 0.37) assessments.

| Filtered | XLSum mT5-Base | Self-ROUGE | Random
---|---|---|---|---
Language | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI
Amharic | 35.15 | 74.65 | 31.14 | 55.07 | 33.76 | 74.96 | 34.33 | 76.97
Arabic | 33.33 | 72.83 | 32.64 | 53.90 | 34.35 | 67.29 | 34.19 | 69.55
Azerbaijani | 29.28 | 68.03 | 27.14 | 48.94 | 28.04 | 61.06 | 26.83 | 66.14
Bengali | 34.17 | 76.53 | 30.90 | 61.05 | 32.75 | 73.34 | 34.04 | 75.61
Burmese | 40.58 | 73.68 | 36.64 | 51.81 | 38.63 | 60.56 | 37.87 | 73.03
Chinese (simp.) | 35.66 | 76.95 | 37.86 | 61.61 | 36.22 | 70.42 | 36.36 | 76.37
Chinese (trad.) | 36.49 | 78.15 | 37.78 | 62.45 | 36.70 | 71.09 | 36.66 | 77.58
English | 33.23 | 80.32 | 34.16 | 62.83 | 37.21 | 75.68 | 37.46 | 77.25
French | 34.49 | 72.94 | 32.48 | 55.52 | 33.84 | 65.92 | 33.86 | 68.38
Gujarati | 32.59 | 63.54 | 29.84 | 51.81 | 32.08 | 64.33 | 31.39 | 65.60
Hausa | 37.30 | 67.47 | 35.16 | 48.26 | 35.64 | 59.22 | 36.38 | 59.69
Hindi | 37.45 | 74.20 | 35.42 | 54.61 | 38.14 | 68.47 | 37.82 | 71.51
Igbo | 35.44 | 64.36 | 32.92 | 42.32 | 33.55 | 62.96 | 34.42 | 56.86
Indonesian | 35.26 | 79.06 | 34.22 | 60.99 | 35.99 | 71.20 | 36.53 | 73.66
Japanese | 41.30 | 78.60 | 38.81 | 55.37 | 40.41 | 78.43 | 41.00 | 78.67
Kirundi | 33.63 | 70.66 | 30.12 | 49.06 | 32.27 | 63.90 | 32.32 | 67.49
Korean | 39.83 | 78.51 | 36.86 | 62.19 | 38.98 | 74.68 | 38.67 | 79.27
Kyrgyz | 26.54 | 67.99 | 24.95 | 54.10 | 24.49 | 69.23 | 25.36 | 75.01
Marathi | 31.31 | 67.86 | 29.28 | 49.31 | 27.30 | 66.20 | 27.06 | 66.32
Nepali | 38.67 | 80.57 | 35.11 | 62.41 | 37.62 | 78.49 | 38.66 | 78.44
Oromo | 29.61 | 76.35 | 26.97 | 58.98 | 28.19 | 78.00 | 28.95 | 71.41
Pashto | 39.88 | 76.34 | 36.91 | 55.39 | 39.35 | 72.90 | 38.89 | 74.32
Persian | 37.86 | 79.22 | 36.77 | 62.97 | 37.97 | 75.01 | 38.50 | 77.54
Pidgin | 34.41 | 72.03 | 33.68 | 54.00 | 33.52 | 69.38 | 33.63 | 65.81
Portuguese | 33.90 | 64.24 | 32.79 | 40.26 | 34.40 | 52.93 | 35.63 | 57.67
Punjabi | 36.39 | 64.61 | 32.56 | 48.98 | 35.16 | 60.76 | 35.88 | 63.86
Russian | 28.71 | 69.20 | 28.28 | 52.04 | 29.28 | 65.32 | 30.42 | 65.05
Scottish Gaelic | 32.95 | 68.14 | 28.55 | 43.12 | 31.83 | 61.42 | 30.92 | 58.67
Serbian (Cyrillic) | 29.20 | 62.97 | 26.72 | 44.89 | 27.10 | 62.19 | 28.33 | 62.36
Serbian (Latin) | 28.21 | 59.93 | 24.85 | 38.66 | 25.63 | 58.28 | 27.14 | 58.69
Sinhala | 36.04 | 74.76 | 31.42 | 57.27 | 31.52 | 68.35 | 34.36 | 76.12
Somali | 31.35 | 69.42 | 29.10 | 51.16 | 30.71 | 62.08 | 30.17 | 63.61
Spanish | 27.33 | 62.34 | 26.90 | 43.45 | 27.44 | 55.76 | 27.53 | 59.26
Swahili | 36.46 | 73.25 | 33.68 | 54.17 | 34.98 | 69.04 | 34.38 | 66.90
Tamil | 33.26 | 81.91 | 31.37 | 63.84 | 31.68 | 77.17 | 31.25 | 80.79
Telugu | 29.11 | 67.83 | 26.39 | 55.12 | 27.54 | 60.45 | 27.82 | 68.07
Thai | 30.66 | 73.27 | 28.43 | 49.35 | 29.62 | 63.97 | 29.80 | 71.96
Tigrinya | 36.37 | 76.71 | 32.24 | 59.20 | 34.69 | 69.19 | 35.23 | 73.26
Turkish | 33.65 | 74.66 | 32.81 | 51.91 | 33.01 | 66.81 | 33.58 | 68.67
Ukrainian | 28.99 | 70.87 | 28.32 | 54.65 | 29.08 | 64.65 | 29.64 | 65.28
Urdu | 38.18 | 74.81 | 36.64 | 54.70 | 38.65 | 70.80 | 38.67 | 72.61
Uzbek | 28.06 | 70.62 | 25.96 | 52.32 | 25.40 | 68.34 | 26.15 | 72.99
Vietnamese | 36.69 | 73.01 | 32.77 | 53.04 | 36.07 | 68.67 | 37.41 | 69.45
Welsh | 33.89 | 76.06 | 31.97 | 49.66 | 34.43 | 66.18 | 33.93 | 67.17
Yoruba | 37.01 | 68.48 | 33.90 | 50.81 | 35.03 | 67.52 | 35.28 | 68.20
Average | 34.00 | 72.17 | 31.85 | 53.41 | 33.12 | 67.39 | 33.44 | 69.62

Table 9: ROUGE and NLI scores per language on the XLSum test set for our
Filtered model vs. the additional baselines. For simplicity, all models are
compared using their Best-ROUGE checkpoints. XLSum mT5-base predictions are
taken from the original XLSum paper Hasan et al. (2021). However, we report on
the recomputed ROUGE-L using the SentencePiece tokenization of mT5 to make it
comparable with others. See §5.2 for more details on Self-Rouge and Random
baselines.

#### Comparison with Other Baselines

Finally, comparison of the Best-ROUGE filtered model to other baselines is in
Table 9.666We did not study the Best-NLI checkpoints for the baselines as the
Best-ROUGE filtered model was already better on both ROUGE and NLI against
their baseline counterparts. The table shows that the filtered model performs
significantly better than the original XLSum paper results (average NLI scores
of 72.17 vs 53.41) that are probably affected by the different model
capacities (XXL for the filtered model and Base for the baseline). The
filtered model also performs better than the Self-ROUGE and Random baselines
that achieve average NLI scores of 67.39 and 69.62 respectively, reiterating
the benefit in filtering training examples specifically according to their
entailment value.

##  7 Conclusions and Future Work

We suggested that automatic factual consistency evaluation can be scaled to
multiple languages via multilingual pretrained language models, fine-tuned on
NLI data. We showed that this works well by experimenting on the XLSum
dataset, where we showed that one can improve the faithfulness of generated
summaries in 45 languages using such multilingual NLI signal. Specifically, we
showed that training models on a subset of the data selected using the NLI
signal improves results in both automatic and human evaluation. We intend to
publish the human judgements we collected in all languages to foster future
work on the subject.

Despite progress, our annotators still found that summaries were (on average)
only 52% of the time fully faithful to their documents; this number is much
worse for some of the languages (see Table 16 in the Appendix). Interesting
avenues for future work may include directly optimizing for high NLI scores
via Reinforcement Learning Narayan et al. (2018b) or Calibrating Sequence
Likelihood (Zhao et al., 2022). Better modeling of the multilingual NLI task
may also result in further gains. We hope our work will inspire future
research on modeling and measuring faithfulness in text generation tasks for
languages other than English.

## Acknowledgements

We thank Ankur Parikh, Sebastian Gehrmann, Dipanjan Das and William Cohen for
their feedback on this work. The human rating process was managed by Muqthar
Mohammad, Kiranmai Chennuru, Aishwarya Gomatam, Raghava Ram Pamidigantam and
Mahesh Maddinala, without them this work would not have been possible. Thanks
for invaluable support from Sheila de Guia and Suneet Dhingra.

## References

  * Aharoni et al. (2019) Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019.  Massively multilingual neural machine translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3874–3884, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Aralikatte et al. (2021) Rahul Aralikatte, Shashi Narayan, Joshua Maynez, Sascha Rothe, and Ryan McDonald. 2021.  Focus attention: Promoting faithfulness and diversity in summarization.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6078–6095, Online. Association for Computational Linguistics.
  * Arumae and Liu (2019) Kristjan Arumae and Fei Liu. 2019.  Guiding extractive summarization with question-answering rewards.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2566–2577, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015\.  A large annotated corpus for learning natural language inference.  _arXiv preprint arXiv:1508.05326_.
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6633–6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Cao et al. (2018) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018.  Faithful to the original: Fact aware neural abstractive summarization.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 4784–4791. AAAI Press.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The pascal recognising textual entailment challenge.  In _Machine learning challenges workshop_ , pages 177–190. Springer.
  * Deutsch et al. (2021) Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021.  Towards question-answering as an automatic metric for evaluating the content quality of a summary.  _Transactions of the Association for Computational Linguistics_ , 9:774–789.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Eyal et al. (2019) Matan Eyal, Tal Baumel, and Michael Elhadad. 2019.  Question answering as an automatic evaluation metric for news article summarization.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3938–3948, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fabbri et al. (2021) Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2021.  Qafacteval: Improved qa-based factual consistency evaluation for summarization.  _CoRR_ , abs/2112.08542.
  * Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019.  Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2214–2220, Florence, Italy. Association for Computational Linguistics.
  * Gabriel et al. (2021) Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021\.  GO FIGURE: A meta evaluation of factuality in summarization.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 478–487, Online. Association for Computational Linguistics.
  * Gehrmann et al. (2021) Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021.  The GEM benchmark: Natural language generation, its evaluation and metrics.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 96–120, Online. Association for Computational Linguistics.
  * Goyal et al. (2021) Navita Goyal, Balaji Vasan Srinivasan, Anandhavelu N, and Abhilasha Sancheti. 2021\.  Multi-style transfer with discriminative feedback on disjoint corpus.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3500–3510, Online. Association for Computational Linguistics.
  * Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020.  Evaluating factuality in generation with dependency-level entailment.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3592–3603, Online. Association for Computational Linguistics.
  * Goyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021.  Annotating and modeling fine-grained factuality in summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1449–1462, Online. Association for Computational Linguistics.
  * Gunasekara et al. (2021) Chulaka Gunasekara, Guy Feigenblat, Benjamin Sznajder, Ranit Aharonov, and Sachindra Joshi. 2021.  Using question answering rewards to improve abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 518–526, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021.  XL-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4693–4703, Online. Association for Computational Linguistics.
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022.  TRUE: Re-evaluating factual consistency evaluation.  In _Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering_ , pages 161–175, Dublin, Ireland. Association for Computational Linguistics.
  * Honovich et al. (2021) Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.  $q^{2}$: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7856–7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kang and Hashimoto (2020) Daniel Kang and Tatsunori B. Hashimoto. 2020.  Improved natural language generation via loss truncation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 718–731, Online. Association for Computational Linguistics.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019.  Ctrl: A conditional transformer language model for controllable generation.  _ArXiv_ , abs/1909.05858.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Laban et al. (2021) Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2021.  Summac: Re-visiting nli-based models for inconsistency detection in summarization.  _arXiv preprint arXiv:2111.09525_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Nan et al. (2021a) Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021a.  Entity-level factual consistency of abstractive text summarization.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2727–2733, Online. Association for Computational Linguistics.
  * Nan et al. (2021b) Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021b.  Improving factual consistency of abstractive summarization via question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6881–6894, Online. Association for Computational Linguistics.
  * Narayan et al. (2018a) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Narayan et al. (2018b) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b.  Ranking sentences for extractive summarization with reinforcement learning.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1747–1759, New Orleans, Louisiana. Association for Computational Linguistics.
  * Narayan et al. (2022) Shashi Narayan, Gonçalo Simões, Yao Zhao, Joshua Maynez, Dipanjan Das, Michael Collins, and Mirella Lapata. 2022.  A well-composed text is half done! composition sampling for diverse conditional generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1319–1339, Dublin, Ireland. Association for Computational Linguistics.
  * Narayan et al. (2021) Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simões, Vitaly Nikolaev, and Ryan McDonald. 2021.  Planning with learned entity prompts for abstractive summarization.  _Transactions of the Association for Computational Linguistics_ , 9:1475–1492.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * Pasunuru and Bansal (2018) Ramakanth Pasunuru and Mohit Bansal. 2018.  Multi-reward reinforced summarization with saliency and entailment.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 646–653, New Orleans, Louisiana. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rashkin et al. (2021a) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021a.  Measuring attribution in natural language generation models.  _CoRR_ , abs/2112.12870.
  * Rashkin et al. (2021b) Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021b.  Increasing faithfulness in knowledge-grounded dialogue with controllable features.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 704–718, Online. Association for Computational Linguistics.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? _CoRR_ , abs/2002.08910.
  * Schuster et al. (2022) Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, and Donald Metzler. 2022.  Stretching sentence-pair nli models to reason over long documents and clusters.  _arXiv preprint arXiv:2204.07447_.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021.  QuestEval: Summarization asks for fact-based evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6594–6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Song et al. (2018) Kaiqiang Song, Lin Zhao, and Fei Liu. 2018.  Structure-infused copy mechanisms for abstractive summarization.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1717–1729, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Tang et al. (2022) Liyan Tang, Tanya Goyal, Alexander R. Fabbri, Philippe Laban, Jiacheng Xu, Semih Yahvuz, Wojciech Kryściński, Justin F. Rousseau, and Greg Durrett. 2022\.  Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors.
  * Wan and Bansal (2022) David Wan and Mohit Bansal. 2022.  FactPEGASUS: Factuality-aware pre-training and fine-tuning for abstractive summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1010–1028, Seattle, United States. Association for Computational Linguistics.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D. Manning, and Curtis Langlotz. 2020.  Optimizing the factual correctness of a summary: A study of summarizing radiology reports.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5108–5120, Online. Association for Computational Linguistics.
  * Zhao et al. (2022) Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J. Liu. 2022.  Calibrating sequence likelihood improves conditional language generation.  _ArXiv_.
  * Zhao et al. (2020) Zheng Zhao, Shay B. Cohen, and Bonnie Webber. 2020.  Reducing quantity hallucinations in abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2237–2249, Online. Association for Computational Linguistics.
  * Zhu et al. (2021) Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2021.  Enhancing factual consistency of abstractive summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 718–733, Online. Association for Computational Linguistics.

##  Appendix A Intrinsic NLI Model Evaluation

To measure the quality of our multilingual NLI model we evaluate it on the
following two datasets where full results are detailed here.

Table 11 shows model accuracy when evaluated on XNLI’s test set for all 15
languages, comparing our results to mT5-XXL fine-tuned on MNLI by Xue et al.
(2021). The table shows our model achieves higher accuracy for all languages,
however results are not fully comparable as we only consider binary labels
(entailment/non-entailment) in our setting in comparison to three classes for
Xue et al. (2021) (entailment/neutral/contradiction).

We additionally evaluate our model on the TRUE factual consistency benchmark
Honovich et al. (2022). TRUE consists of 11 diverse datasets that include
outputs of grounded text generation systems, annotated with binary factual
consistency labels. Although TRUE only includes English examples, we use it
for our evaluation due to its relevance to factual consistency in
summarization. Table 12 shows the area under the ROC curve (ROC AUC) results
for all dataset in TRUE where we compare our multilingual model to T5-11B
trained on ANLI Nie et al. (2020) reported by Honovich et al. (2022). Results
show that our multilingual performs on par with the baseline, where fine-
tuning our model on non-English data might cause the slight decrease in
performance.

Language | Family | XNLI
---|---|---
Lang | Family | XNLI
Amharic | Semitic | non-xnli
Arabic | Semitic | xnli
Azerbaijani | Turkic | non-xnli
Bengali | Indo-European | non-xnli
Burmese | Sino-Tibetan | non-xnli
Chinese Simplified | Sino-Tibetan | xnli
Chinese Traditional | Sino-Tibetan | xnli
English | Indo-European | xnli
French | Romance | xnli
Gujarati | Indo-European | non-xnli
Hausa | Afro-Asiatic | non-xnli
Hindi | Indo-European | xnli
Igbo | Niger-Congo | non-xnli
Indonesian | Austronesian | non-xnli
Japanese | Japonic | non-xnli
Kirundi | Bantu | non-xnli
Korean | Koreanic | non-xnli
Kyrgyz | Turkic | non-xnli
Marathi | Indo-Aryan | non-xnli
Nepali | Indo-Aryan | non-xnli
Oromo | Afro-Asiatic | non-xnli
Pashto | Indo-Iranian | non-xnli
Persian | Indo-Iranian | non-xnli
Pidgin | Unknown | non-xnli
Portuguese | Romance | non-xnli
Punjabi | Indo-Iranian | non-xnli
Russian | Indo-European | xnli
Scottish Gaelic | Celtic | non-xnli
Serbian Cyrillic | Indo-European | non-xnli
Serbian Latin | Indo-European | non-xnli
Sinhala | Indo-European | non-xnli
Somali | Afro-Asiatic | non-xnli
Spanish | Romance | xnli
Swahili | Bantu | xnli
Tamil | Dravidian | non-xnli
Telugu | Dravidian | non-xnli
Thai | Kra-Dai Languages | xnli
Tigrinya | Semitic | non-xnli
Turkish | Turkic | xnli
Ukrainian | Slavic | non-xnli
Urdu | Indo-european | xnli
Uzbek | Turkic | non-xnli
Vietnamese | Austroasiatic | xnli
Welsh | Celtic | non-xnli
Yoruba | Niger-Congo | non-xnli

Table 10: XLSum language groups: Language families and XNLI candidacy.

Model | ar | bg | de | el | en | es | fr | hi | ru | sw | th | tr | ur | vi | zh | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Xue et al. (2021) | 84.5 | 87.7 | 87.3 | 87.3 | 91.6 | 87.8 | 86.9 | 83.2 | 85.1 | 80.3 | 81.7 | 83.8 | 79.8 | 84.6 | 83.6 | 85.0
Ours | 90.0 | 91.6 | 91.2 | 91.4 | 93.5 | 91.7 | 91.4 | 88.7 | 90.3 | 87.2 | 88.0 | 89.1 | 86.3 | 89.6 | 89.8 | 90.0

Table 11: Accuracy results on XNLI’s test set.

Model | FRANK | MNBM | QAGS-C | QAGS-X | BEGIN | $Q^{2}$ | DialFact | PAWS | FEVER | VitaminC | SummEval | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---
Honovich et al. (2022) | 89.4 | 77.9 | 82.1 | 83.8 | 82.6 | 72.7 | 77.7 | 86.4 | 93.2 | 88.3 | 80.5 | 83.1
Ours | 88.1 | 77.2 | 82.8 | 84.5 | 80.5 | 72.8 | 80.1 | 82.6 | 93.7 | 88.2 | 75.6 | 82.4

Table 12: ROC AUC results on TRUE.

| Vanilla | Filtered | Controlled
---|---|---|---
| Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI
Language | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI
Amharic | 33.54 | 60.4 | 33.02 | 65.43 | 33.17 | 66.47 | 32.72 | 73.38 | 33.57 | 62.12 | 32.94 | 66.09
Arabic | 29.94 | 38.2 | 31.94 | 53.01 | 32.49 | 61.29 | 32.11 | 65.29 | 30.58 | 52.63 | 31.91 | 58.97
Azerbaijani | 29.05 | 53.14 | 28.76 | 51.56 | 28.79 | 61.14 | 28.39 | 67.39 | 27.84 | 53.96 | 28.25 | 55.29
Bengali | 33.96 | 62.85 | 33.22 | 61.61 | 33.36 | 70.6 | 33.61 | 77.53 | 33.68 | 59.77 | 32.88 | 67.1
Burmese | 40.88 | 57.62 | 39.6 | 60.48 | 39.19 | 70.19 | 38.91 | 76.96 | 39.41 | 62.03 | 39.4 | 66.49
Chinese (simp.) | 32.44 | 47.32 | 32.76 | 58.2 | 34.93 | 71.09 | 34.31 | 72.99 | 32.61 | 62.88 | 33.64 | 67.46
Chinese (trad.) | 33.57 | 52.99 | 34.11 | 58.7 | 36.3 | 72.42 | 34.98 | 71.09 | 33.28 | 62.3 | 34.35 | 66.06
English | 31.68 | 55.52 | 32.26 | 70.7 | 32.37 | 72.83 | 32.3 | 80.48 | 32.61 | 69.61 | 32.44 | 74.58
French | 33.62 | 51.19 | 33.04 | 51.02 | 33.13 | 61.24 | 32.94 | 64.92 | 32.6 | 60.07 | 32.82 | 58.26
Gujarati | 32.83 | 50.1 | 31.93 | 48.79 | 32.04 | 55.54 | 32.25 | 61.14 | 32.45 | 53.92 | 31.59 | 52.64
Hausa | 37.34 | 48.85 | 35.64 | 51.52 | 36.74 | 56.83 | 35.96 | 62.81 | 36.93 | 54.71 | 35.89 | 57.34
Hindi | 35.53 | 56.8 | 35.67 | 62.27 | 36.39 | 68.92 | 36.1 | 68.31 | 35.62 | 62.78 | 36.15 | 62.02
Igbo | 35.09 | 44.85 | 35.23 | 48.13 | 35.36 | 57.84 | 35.14 | 59.15 | 34.98 | 48.25 | 35.6 | 52.89
Indonesian | 33.06 | 57.62 | 33.32 | 60.29 | 34.28 | 69.7 | 33.32 | 70.54 | 33.3 | 63.68 | 33.38 | 68.13
Japanese | 40.3 | 68.48 | 39.82 | 72.57 | 40.9 | 76.52 | 39.7 | 77.08 | 39.92 | 81.95 | 39.54 | 81.07
Kirundi | 32.37 | 47.9 | 31.32 | 46.54 | 32.44 | 52.65 | 31.2 | 60.33 | 33.15 | 49.56 | 31.74 | 51.04
Korean | 39.83 | 66.88 | 37.21 | 67.36 | 38.96 | 70.23 | 37.75 | 79.42 | 39.89 | 70.31 | 36.29 | 74.05
Kyrgyz | 25.8 | 53.27 | 26.37 | 50.89 | 24.78 | 56.52 | 25.29 | 68.1 | 25.29 | 58.87 | 25.62 | 60.43
Marathi | 28.62 | 47.15 | 27.2 | 52.72 | 30.09 | 63.59 | 30.84 | 63.35 | 29.32 | 49.29 | 26.81 | 52.68
Nepali | 37.96 | 67.53 | 37.48 | 67.11 | 38.14 | 72.3 | 37.06 | 74.48 | 37.4 | 67.72 | 36.32 | 71.51
Oromo | 29.63 | 59.69 | 27.37 | 63.12 | 28.83 | 68.41 | 28.07 | 75.19 | 28.61 | 59.08 | 28.19 | 61.31
Pashto | 38.96 | 58.37 | 38.57 | 57.27 | 39.55 | 70.66 | 38.73 | 73.06 | 38.19 | 60.74 | 38.17 | 65.73
Persian | 34.91 | 60.31 | 36.35 | 64.62 | 37.37 | 71.47 | 36.83 | 72.66 | 35.55 | 63.05 | 35.99 | 68.45
Pidgin | 34.27 | 51.04 | 33.78 | 60.66 | 34.5 | 67.76 | 33.27 | 67.96 | 34.43 | 57.5 | 34.1 | 61.76
Portuguese | 32.33 | 39.16 | 32.21 | 48.18 | 33.43 | 57.06 | 32.66 | 57.37 | 32.4 | 48.76 | 32.67 | 57.22
Punjabi | 34.88 | 48.49 | 33.82 | 48.97 | 34.41 | 50.05 | 34.53 | 55.12 | 34.46 | 45.76 | 33.87 | 50.55
Russian | 27.33 | 47.09 | 28.02 | 52.99 | 28.06 | 56.07 | 27.94 | 61.11 | 27.89 | 59.66 | 27.95 | 61.59
Scottish Gaelic | 33.7 | 50.55 | 33.36 | 61.03 | 32.38 | 56.82 | 32.63 | 72.31 | 33.24 | 63.18 | 32.71 | 62.77
Serbian (Cyrillic) | 27.76 | 47.21 | 27.14 | 48.95 | 27.69 | 55.19 | 26.78 | 57.38 | 27.34 | 50.49 | 26.55 | 51.76
Serbian (Latin) | 27.15 | 38.59 | 26.84 | 47.07 | 27.95 | 50.49 | 26.25 | 55.74 | 27.14 | 44.18 | 25.19 | 41.25
Sinhala | 36.16 | 57.9 | 34.36 | 60.04 | 34.67 | 61.27 | 34.24 | 76.16 | 35.3 | 66.45 | 34.61 | 69.81
Somali | 30.93 | 51.75 | 30.46 | 60.48 | 31.57 | 62.42 | 30.52 | 69.28 | 30.9 | 57.59 | 30.76 | 57.92
Spanish | 26.63 | 38.53 | 26.25 | 42.13 | 26.88 | 48.84 | 26.48 | 61.87 | 26.67 | 46.36 | 26.65 | 44.34
Swahili | 34.95 | 56.8 | 34.83 | 60.1 | 34.91 | 66.87 | 33.66 | 71.64 | 34.74 | 62.74 | 33.61 | 61.51
Tamil | 30.82 | 61.6 | 29.5 | 70.35 | 31.98 | 78.94 | 30.46 | 81.49 | 30.22 | 75.1 | 29.81 | 75.82
Telugu | 27.9 | 54.06 | 27.94 | 56.94 | 28.01 | 61.29 | 28.29 | 69.97 | 28.47 | 54.84 | 27.53 | 58.79
Thai | 29.46 | 54.3 | 30.22 | 61.89 | 29.2 | 68.32 | 29.19 | 72.93 | 29.08 | 60.84 | 29.78 | 65.0
Tigrinya | 35.25 | 57.08 | 33.9 | 63.66 | 35.11 | 67.61 | 32.9 | 73.36 | 33.41 | 52.28 | 34.69 | 60.57
Turkish | 29.94 | 50.76 | 30.75 | 54.47 | 32.18 | 66.95 | 31.39 | 68.55 | 30.64 | 57.64 | 31.09 | 60.72
Ukrainian | 28.16 | 44.18 | 28.6 | 52.46 | 28.75 | 61.42 | 27.98 | 63.55 | 28.43 | 56.66 | 28.49 | 60.08
Urdu | 36.03 | 52.54 | 35.79 | 52.69 | 37.34 | 66.87 | 36.75 | 70.13 | 36.02 | 59.9 | 36.15 | 62.75
Uzbek | 28.1 | 60.46 | 27.94 | 59.44 | 28.12 | 61.3 | 27.75 | 70.78 | 27.7 | 59.99 | 27.46 | 59.42
Vietnames | 34.71 | 53.8 | 34.65 | 59.88 | 35.48 | 67.14 | 34.95 | 68.21 | 34.38 | 59.73 | 35.1 | 64.89
Welsh | 34.56 | 55.44 | 32.7 | 52.02 | 33.64 | 66.7 | 32.21 | 62.06 | 33.5 | 57.04 | 32.46 | 60.19
Yoruba | 37.25 | 54.79 | 35.76 | 54.26 | 36.66 | 58.31 | 36.04 | 66.82 | 36.82 | 50.83 | 36.03 | 55.87
Average | 32.87 | 53.18 | 32.47 | 57.16 | 33.17 | 63.91 | 32.56 | 68.65 | 32.67 | 58.59 | 32.38 | 61.43

Table 13: ROUGE and NLI scores per language on the XLSum development set for
the best checkpoints (chosen by averaging across all languages). Highest ROUGE
and NLI numbers in each row are in bold.

| Vanilla | Filtered | Controlled
---|---|---|---
| Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI
Language | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI | ROUGE | NLI
English | 27.94 | 55.52 | 28.52 | 70.70 | 28.69 | 72.83 | 28.42 | 80.48 | 28.79 | 69.61 | 28.49 | 74.58
Varying Number of Training Resource
High | 30.33 | 48.86 | 30.53 | 55.10 | 31.50 | 63.90 | 30.87 | 66.72 | 30.54 | 57.81 | 30.64 | 61.39
Medium | 31.51 | 53.67 | 30.75 | 56.36 | 31.67 | 63.96 | 31.03 | 67.03 | 31.31 | 58.10 | 30.29 | 59.94
Low | 33.22 | 56.09 | 32.20 | 58.64 | 32.84 | 63.86 | 32.04 | 70.16 | 32.70 | 57.80 | 32.09 | 61.06
Language Families
Indo-European | 33.13 | 46.37 | 34.92 | 47.77 | 36.67 | 51.84 | 38.00 | 56.02 | 39.20 | 50.31 | 36.13 | 51.36
Romance | 29.02 | 42.96 | 28.47 | 47.11 | 29.36 | 55.71 | 28.79 | 61.39 | 28.69 | 51.73 | 28.61 | 53.27
Turkic | 26.27 | 54.41 | 26.32 | 54.09 | 26.28 | 61.48 | 26.09 | 68.71 | 25.79 | 57.62 | 25.85 | 58.97
Semitic | 32.78 | 51.89 | 32.52 | 60.70 | 33.38 | 65.12 | 32.21 | 70.68 | 32.36 | 55.68 | 32.79 | 61.88
Afro-Asiatic | 31.63 | 53.43 | 30.05 | 58.37 | 31.17 | 62.55 | 30.30 | 69.09 | 31.15 | 57.13 | 30.43 | 58.86

Table 14:  ROUGE & NLI scores for the best checkpoints averaged across
language groups on XLSum development set: 1) varying number of training
resources and 2) language families See Table 4 for more details about
different groups. Highest ROUGE and NLI numbers in each row are in bold.
---
Q1: Is the summary comprehensible? Incomprehensible: The summary is difficult
to understand. It can have serious grammatical errors, low fluency, and/or
repeated information. Somewhat Comprehensible: The summary generally makes
sense but suffers from grammatical errors, low fluency, and/or repeated
information. Comprehensible: The summary is understandable. It does not
exhibit any grammatical errors, disfluencies, and/or repeated information.
Q2: Is all the information in the summary fully attributable to the article?
Attribution: We follow the attribution definition provided in Rashkin et al.
(2021a). A fully attributable (or supported) system-generated summary contains
an accurate representation of information in the source news article. No
information in the summary is unattested when compared against the source news
article. Yes, it is attributable: Select this option if it is accurate to say,
“The provided news article says…” or “According to the news article…” with the
summary following this phrase. No, not fully attributable: Select this option
if only some of the information is supported in the news article, but other
parts of the information are missing from the news article or not an accurate
representation.
Q3: Is the summary a good summary of the article? Bad summary: The summary
does not capture the important information in the article, or the captured
information is not accurate with the article. It can also exhibit grammatical
issues, low fluency, and/or repeated information. Good Summary: The summary
captures the important information in the article and presents it accurately
and concisely. It does not exhibit any grammatical errors, disfluencies,
and/or repeated information.
Figure 2: The template and the accompanied instructions that were used in our human evaluations. | Vanilla | Filtered | Controlled |
---|---|---|---|---
Language | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Human
amharic | 0.74 | 0.77 | 0.75 | 0.7 | 0.75 | 0.78 | 0.75
arabic | 0.89 | 0.91 | 0.92 | 0.96 | 0.93 | 0.95 | 0.97
azerbaijani | 0.69 | 0.67 | 0.68 | 0.7 | 0.64 | 0.68 | 0.68
bengali | 0.92 | 0.92 | 0.93 | 0.94 | 0.94 | 0.94 | 0.94
burmese | 0.94 | 0.91 | 0.93 | 0.93 | 0.91 | 0.91 | 0.94
chinese (simp.) | 0.85 | 0.78 | 0.92 | 0.88 | 0.81 | 0.87 | 0.95
chinese (trad.) | 0.83 | 0.81 | 0.86 | 0.83 | 0.82 | 0.86 | 0.94
english | 0.91 | 0.92 | 0.91 | 0.92 | 0.89 | 0.91 | 0.9
french | 0.92 | 0.91 | 0.91 | 0.92 | 0.88 | 0.94 | 0.93
gujarati | 0.88 | 0.89 | 0.85 | 0.9 | 0.86 | 0.9 | 0.87
hausa | 0.98 | 0.97 | 0.97 | 0.97 | 0.96 | 0.95 | 0.99
hindi | 0.78 | 0.79 | 0.82 | 0.83 | 0.74 | 0.79 | 0.87
igbo | 0.86 | 0.88 | 0.85 | 0.89 | 0.87 | 0.88 | 0.88
indonesian | 0.87 | 0.84 | 0.88 | 0.9 | 0.88 | 0.92 | 0.9
japanese | 0.85 | 0.9 | 0.88 | 0.85 | 0.91 | 0.88 | 0.94
kirundi | 0.79 | 0.81 | 0.78 | 0.82 | 0.8 | 0.8 | 0.88
korean | 0.88 | 0.83 | 0.9 | 0.87 | 0.84 | 0.88 | 0.94
kyrgyz | 0.93 | 0.94 | 0.94 | 0.96 | 0.94 | 0.94 | 0.96
marathi | 0.78 | 0.76 | 0.81 | 0.81 | 0.81 | 0.75 | 0.79
nepali | 0.87 | 0.95 | 0.92 | 0.95 | 0.85 | 0.94 | 0.87
oromo | 0.61 | 0.56 | 0.62 | 0.61 | 0.66 | 0.55 | 0.78
pashto | 0.71 | 0.7 | 0.67 | 0.7 | 0.7 | 0.68 | 0.6
persian | 0.81 | 0.83 | 0.86 | 0.85 | 0.78 | 0.84 | 0.84
pidgin | 0.87 | 0.84 | 0.84 | 0.84 | 0.87 | 0.84 | 0.84
portuguese | 0.92 | 0.94 | 0.97 | 0.95 | 0.94 | 0.96 | 0.97
punjabi | 0.71 | 0.71 | 0.69 | 0.74 | 0.75 | 0.71 | 0.67
russian | 0.63 | 0.66 | 0.64 | 0.7 | 0.68 | 0.68 | 0.69
scottish gaelic | 0.84 | 0.83 | 0.83 | 0.84 | 0.81 | 0.84 | 0.85
serbian (cyrillic) | 0.84 | 0.91 | 0.81 | 0.91 | 0.86 | 0.88 | 0.95
serbian (latin) | 0.85 | 0.82 | 0.8 | 0.81 | 0.88 | 0.84 | 0.94
sinhala | 0.95 | 0.93 | 0.97 | 0.96 | 0.93 | 0.94 | 0.99
somali | 0.86 | 0.84 | 0.83 | 0.91 | 0.86 | 0.86 | 0.91
spanish | 0.94 | 0.96 | 0.95 | 0.98 | 0.9 | 0.98 | 0.98
swahili | 0.93 | 0.95 | 0.95 | 0.95 | 0.94 | 0.97 | 0.92
tamil | 0.86 | 0.84 | 0.85 | 0.86 | 0.84 | 0.84 | 0.82
telugu | 0.87 | 0.89 | 0.87 | 0.9 | 0.85 | 0.9 | 0.91
thai | 0.88 | 0.87 | 0.89 | 0.89 | 0.87 | 0.91 | 0.97
tigrinya | 0.88 | 0.84 | 0.87 | 0.88 | 0.86 | 0.84 | 0.96
turkish | 0.86 | 0.85 | 0.88 | 0.91 | 0.86 | 0.86 | 0.95
ukrainian | 0.9 | 0.93 | 0.91 | 0.94 | 0.87 | 0.94 | 0.97
urdu | 0.64 | 0.67 | 0.63 | 0.67 | 0.68 | 0.67 | 0.59
uzbek | 0.89 | 0.83 | 0.85 | 0.88 | 0.83 | 0.87 | 0.81
vietnamese | 0.97 | 0.96 | 0.95 | 0.95 | 0.92 | 0.96 | 0\. 96
welsh | 0.88 | 0.86 | 0.91 | 0.87 | 0.89 | 0.9 | 0.96
yoruba | 0.83 | 0.78 | 0.78 | 0.78 | 0.75 | 0.75 | 0.85
Avgerage | 0.85 | 0.84 | 0.85 | 0.86 | 0.84 | 0.86 | 0.88
Table 15: Human evaluation results for Summary Quality per language on the XLSum test set for Best-ROUGE and Best-NLI checkpoints. We also include the Human written summaries. | Vanilla | Filtered | Controlled |
---|---|---|---|---
Language | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Human
amharic | 0.47 | 0.53 | 0.53 | 0.5 | 0.55 | 0.54 | 0.39
arabic | 0.2 | 0.29 | 0.36 | 0.37 | 0.34 | 0.34 | 0.14
azerbaijani | 0.39 | 0.44 | 0.46 | 0.46 | 0.41 | 0.41 | 0.28
bengali | 0.79 | 0.8 | 0.8 | 0.84 | 0.79 | 0.79 | 0.61
burmese | 0.32 | 0.39 | 0.3 | 0.4 | 0.33 | 0.34 | 0.26
chinese (simp.) | 0.47 | 0.5 | 0.49 | 0.62 | 0.47 | 0.47 | 0.31
chinese (trad.) | 0.46 | 0.53 | 0.54 | 0.53 | 0.5 | 0.5 | 0.33
english | 0.38 | 0.45 | 0.46 | 0.52 | 0.46 | 0.4 | 0.3
french | 0.55 | 0.53 | 0.58 | 0.65 | 0.64 | 0.57 | 0.3
gujarati | 0.48 | 0.49 | 0.51 | 0.5 | 0.51 | 0.47 | 0.45
hausa | 0.29 | 0.32 | 0.32 | 0.35 | 0.31 | 0.35 | 0.15
hindi | 0.44 | 0.5 | 0.49 | 0.54 | 0.48 | 0.49 | 0.37
igbo | 0.41 | 0.47 | 0.4 | 0.47 | 0.36 | 0.44 | 0.22
indonesian | 0.38 | 0.42 | 0.43 | 0.49 | 0.46 | 0.43 | 0.14
japanese | 0.11 | 0.16 | 0.13 | 0.19 | 0.16 | 0.16 | 0.07
kirundi | 0.31 | 0.28 | 0.31 | 0.41 | 0.36 | 0.33 | 0.23
korean | 0.43 | 0.51 | 0.48 | 0.58 | 0.49 | 0.44 | 0.2
kyrgyz | 0.59 | 0.63 | 0.63 | 0.67 | 0.62 | 0.67 | 0.34
marathi | 0.61 | 0.65 | 0.7 | 0.72 | 0.68 | 0.64 | 0.52
nepali | 0.51 | 0.5 | 0.45 | 0.59 | 0.55 | 0.48 | 0.24
oromo | 0.46 | 0.45 | 0.46 | 0.5 | 0.48 | 0.44 | 0.48
pashto | 0.59 | 0.6 | 0.56 | 0.56 | 0.61 | 0.55 | 0.37
persian | 0.26 | 0.29 | 0.25 | 0.26 | 0.31 | 0.29 | 0.12
pidgin | 0.29 | 0.32 | 0.32 | 0.41 | 0.38 | 0.31 | 0.16
portuguese | 0.29 | 0.35 | 0.3 | 0.42 | 0.3 | 0.35 | 0.17
punjabi | 0.49 | 0.52 | 0.5 | 0.58 | 0.55 | 0.53 | 0.31
russian | 0.36 | 0.4 | 0.38 | 0.47 | 0.47 | 0.44 | 0.28
scottish gaelic | 0.49 | 0.55 | 0.52 | 0.56 | 0.57 | 0.56 | 0.53
serbian (cyrillic) | 0.41 | 0.47 | 0.43 | 0.48 | 0.52 | 0.45 | 0.3
serbian (latin) | 0.36 | 0.33 | 0.33 | 0.44 | 0.46 | 0.36 | 0.28
sinhala | 0.39 | 0.43 | 0.31 | 0.51 | 0.41 | 0.38 | 0.17
somali | 0.5 | 0.56 | 0.58 | 0.63 | 0.52 | 0.56 | 0.41
spanish | 0.42 | 0.54 | 0.52 | 0.61 | 0.56 | 0.51 | 0.35
swahili | 0.52 | 0.6 | 0.58 | 0.66 | 0.64 | 0.59 | 0.39
tamil | 0.55 | 0.55 | 0.54 | 0.61 | 0.57 | 0.55 | 0.24
telugu | 0.33 | 0.36 | 0.35 | 0.37 | 0.38 | 0.36 | 0.26
thai | 0.43 | 0.46 | 0.37 | 0.55 | 0.5 | 0.52 | 0.3
tigrinya | 0.5 | 0.56 | 0.53 | 0.65 | 0.62 | 0.5 | 0.34
turkish | 0.52 | 0.55 | 0.61 | 0.62 | 0.57 | 0.6 | 0.47
ukrainian | 0.54 | 0.56 | 0.58 | 0.67 | 0.6 | 0.57 | 0.39
urdu | 0.56 | 0.61 | 0.52 | 0.61 | 0.6 | 0.6 | 0.41
uzbek | 0.63 | 0.55 | 0.58 | 0.64 | 0.57 | 0.6 | 0.4
vietnamese | 0.44 | 0.43 | 0.44 | 0.54 | 0.47 | 0.45 | 0.21
welsh | 0.36 | 0.3 | 0.41 | 0.44 | 0.37 | 0.41 | 0.25
yoruba | 0.42 | 0.38 | 0.42 | 0.38 | 0.41 | 0.39 | 0.33
Average | 0.44 | 0.47 | 0.46 | 0.52 | 0.49 | 0.47 | 0.31
Table 16: Human evaluation results for Attribution per language on the XLSum test set for Best-ROUGE and Best-NLI checkpoints. We also include the Human written summaries. | Vanilla | Filtered | Controlled |
---|---|---|---|---
Language | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Best-ROUGE | Best-NLI | Human
amharic | 0.37 | 0.38 | 0.4 | 0.36 | 0.43 | 0.44 | 0.32
arabic | 0.16 | 0.24 | 0.29 | 0.33 | 0.28 | 0.3 | 0.14
azerbaijani | 0.36 | 0.43 | 0.42 | 0.45 | 0.39 | 0.38 | 0.28
bengali | 0.77 | 0.76 | 0.78 | 0.82 | 0.76 | 0.77 | 0.61
burmese | 0.29 | 0.35 | 0.27 | 0.35 | 0.28 | 0.32 | 0.23
chinese (simp.) | 0.25 | 0.24 | 0.27 | 0.34 | 0.24 | 0.27 | 0.19
chinese (trad.) | 0.4 | 0.44 | 0.48 | 0.43 | 0.45 | 0.41 | 0.32
english | 0.33 | 0.39 | 0.41 | 0.45 | 0.38 | 0.33 | 0.27
french | 0.35 | 0.38 | 0.39 | 0.43 | 0.43 | 0.38 | 0.23
gujarati | 0.32 | 0.34 | 0.33 | 0.33 | 0.35 | 0.36 | 0.28
hausa | 0.28 | 0.32 | 0.29 | 0.32 | 0.3 | 0.32 | 0.15
hindi | 0.39 | 0.45 | 0.42 | 0.51 | 0.42 | 0.44 | 0.34
igbo | 0.35 | 0.4 | 0.34 | 0.42 | 0.32 | 0.38 | 0.19
indonesian | 0.33 | 0.37 | 0.4 | 0.45 | 0.39 | 0.4 | 0.13
japanese | 0.11 | 0.13 | 0.11 | 0.13 | 0.12 | 0.11 | 0.07
kirundi | 0.27 | 0.25 | 0.26 | 0.35 | 0.34 | 0.29 | 0.21
korean | 0.34 | 0.36 | 0.39 | 0.42 | 0.36 | 0.33 | 0.17
kyrgyz | 0.53 | 0.59 | 0.6 | 0.63 | 0.59 | 0.64 | 0.31
marathi | 0.57 | 0.59 | 0.65 | 0.66 | 0.64 | 0.57 | 0.5
nepali | 0.35 | 0.37 | 0.35 | 0.47 | 0.39 | 0.37 | 0.17
oromo | 0.28 | 0.26 | 0.27 | 0.33 | 0.35 | 0.23 | 0.36
pashto | 0.39 | 0.41 | 0.38 | 0.4 | 0.37 | 0.39 | 0.26
persian | 0.29 | 0.31 | 0.29 | 0.29 | 0.29 | 0.33 | 0.15
pidgin | 0.26 | 0.29 | 0.29 | 0.37 | 0.35 | 0.28 | 0.16
portuguese | 0.28 | 0.33 | 0.3 | 0.4 | 0.26 | 0.33 | 0.17
punjabi | 0.48 | 0.51 | 0.49 | 0.56 | 0.52 | 0.52 | 0.31
russian | 0.32 | 0.38 | 0.34 | 0.43 | 0.39 | 0.4 | 0.27
scottish gaelic | 0.41 | 0.46 | 0.4 | 0.42 | 0.46 | 0.42 | 0.4
serbian (cyrillic) | 0.34 | 0.43 | 0.34 | 0.41 | 0.43 | 0.38 | 0.28
serbian (latin) | 0.31 | 0.28 | 0.23 | 0.36 | 0.41 | 0.3 | 0.26
sinhala | 0.35 | 0.37 | 0.28 | 0.47 | 0.37 | 0.35 | 0.16
somali | 0.46 | 0.52 | 0.54 | 0.61 | 0.51 | 0.51 | 0.42
spanish | 0.25 | 0.34 | 0.32 | 0.39 | 0.33 | 0.32 | 0.23
swahili | 0.46 | 0.56 | 0.53 | 0.61 | 0.58 | 0.53 | 0.35
tamil | 0.54 | 0.55 | 0.54 | 0.61 | 0.55 | 0.53 | 0.24
telugu | 0.32 | 0.33 | 0.35 | 0.37 | 0.36 | 0.35 | 0.29
thai | 0.38 | 0.39 | 0.34 | 0.49 | 0.46 | 0.47 | 0.29
tigrinya | 0.44 | 0.46 | 0.46 | 0.56 | 0.54 | 0.43 | 0.32
turkish | 0.49 | 0.52 | 0.55 | 0.57 | 0.54 | 0.57 | 0.46
ukrainian | 0.47 | 0.49 | 0.52 | 0.63 | 0.49 | 0.54 | 0.37
urdu | 0.46 | 0.48 | 0.4 | 0.46 | 0.51 | 0.48 | 0.38
uzbek | 0.56 | 0.48 | 0.49 | 0.58 | 0.47 | 0.53 | 0.34
vietnamese | 0.38 | 0.34 | 0.38 | 0.45 | 0.38 | 0.37 | 0.19
welsh | 0.29 | 0.24 | 0.36 | 0.34 | 0.29 | 0.37 | 0.23
yoruba | 0.36 | 0.28 | 0.35 | 0.29 | 0.33 | 0.31 | 0.27
Average | 0.37 | 0.40 | 0.39 | 0.45 | 0.41 | 0.40 | 0.27
Table 17: Human evaluation results for Informativeness per language on the
XLSum test set for Best-ROUGE and Best-NLI checkpoints. We also include the
Human written summaries.