# Multilingual Simplification of Medical Texts

Sebastian Joseph1 Kathryn Kazanas1 Keziah Reina1 Vishnesh J. Ramanathan2
Wei Xu2 Byron C. Wallace3 Junyi Jessy Li1
1The University of Texas at Austin
2Georgia Institute of Technology, 3Northeastern University
{sebaj, kreina<EMAIL_ADDRESS><EMAIL_ADDRESS>
{vishnesh<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Automated text simplification aims to produce simple versions of complex
texts. This task is especially useful in the medical domain, where the latest
medical findings are typically communicated via complex and technical
articles. This creates barriers for laypeople seeking access to up-to-date
medical findings, consequently impeding progress on health literacy. Most
existing work on medical text simplification has focused on monolingual
settings, with the result that such evidence would be available only in just
one language (most often, English). This work addresses this limitation via
_multilingual_ simplification, i.e., directly simplifying complex texts into
simplified texts in multiple languages. We introduce MultiCochrane, the first
sentence-aligned multilingual text simplification dataset for the medical
domain in four languages: English, Spanish, French, and Farsi. We evaluate
fine-tuned and zero-shot models across these languages, with extensive human
assessments and analyses. Although models can now generate viable simplified
texts, we identify outstanding challenges that this dataset might be used to
address.

Multilingual Simplification of Medical Texts

Sebastian Joseph1 Kathryn Kazanas1 Keziah Reina1 Vishnesh J. Ramanathan2 Wei
Xu2 Byron C. Wallace3 Junyi Jessy Li1 1The University of Texas at Austin
2Georgia Institute of Technology, 3Northeastern University {sebaj, kreina,
<EMAIL_ADDRESS><EMAIL_ADDRESS>{vishnesh<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Figure 1: Examples of our dataset and system outputs for multilingual medical
text simplification. The dataset has simplifications available in four
languages (English, Spanish, French, Farsi), and system outputs were analyzed
for factual errors (red), fluency errors (blue), and simplicity ( green) among
other criteria.

Important findings in medicine are typically presented in technical, jargon-
laden language in journal articles or reviews, which is difficult for a
general audience to understand. This impedes transparent and fair access to
critical medical information and ultimately hinders health literacy, which is
“one of the most promising and cost-effective approaches to overcome the Non-
Communicable Disease challenge” Liu et al. (2020).

Text simplification models which automatically transform complex texts into
simpler versions understandable by lay readers Siddharthan (2014); Alva-
Manchego et al. (2020) have emerged as a promising means of providing wider
access to published medical evidence. Recent work on simplification has fine-
tuned large pre-trained models Van et al. (2020); Cardon and Grabar (2020);
Devaraj et al. (2021); Guo et al. (2022); Trienes et al. (2022); Basu et al.
(2023), explored reinforcement learning Phatak et al. (2022), and evaluated
zero-shot performance via prompting August et al. (2022).

However, this work has so far exclusively considered monolingual text
simplification. Consequently, _we do not know how well models perform when the
simplified text is not in the same language as the original, complex text_.
This limits the (potential) availability of simplified information to a few
high-resource languages—especially for the medical domain—and leads to equity
issues such that individuals who do not speak English will still face a high
barrier to information access.111While machine translation, to a certain
extent, may be able to overcome this limitation, it is far from ideal due to
variability in translation quality across languages Ruder et al. (2021) and
other issues including cost and interpretability as discussed in Vu et al.
(2022). This paper evaluates a simplify-then-translate pipeline. Work in this
direction is impeded by a lack of data. The largest resources in text
simplification are not in the medical domain, and parallel corpora for medical
simplification only exist in single languages, namely English Devaraj et al.
(2021); Guo et al. (2022); Basu et al. (2023), French Grabar and Cardon
(2018); Cardon and Grabar (2020), and German Trienes et al. (2022).

This paper advances the state of multilingual medical simplification, in which
complex texts are to be directly simplified into target languages. Our
contributions are as follows. First, we introduce MultiCochrane (Figure 1),
the first parallel dataset for medical text simplification across multiple
languages: English, Spanish, French, and Farsi. MultiCochrane contains aligned
sentence pairs sourced from the Cochrane Library of Systematic
Reviews,222https://www.cochranelibrary.com which is a library of meta-analyses
of treatment effectiveness. These review articles include both a technical
abstract and a plain-language summary (PLS), from which we derive the two
subsets of MultiCochrane: (1) MC-clean, 101 technical abstracts with expert-
annotated manual alignments first derived in English, then semi-automatically
aligned to other languages (partially verified by bilingual speakers). MC-
clean contains $\sim$5k sentence pairs across all 4 languages. (2) MC-noisy, a
larger but noisier subset created with an automatic sentence alignment model
that was trained on MC-clean. MC-noisy is sourced from around 7.8K medical
reviews, with $\sim$100K sentence pairs across languages.

MultiCochrane enables systematic evaluations of medical text simplification
models. Here we evaluate a range of large pre-trained language models, in both
zero-shot and fine-tuned settings, on the task of text simplification across
four languages. In addition to automatic evaluations, we report human
assessments which cover simplicity, fluency and factuality Devaraj et al.
(2022). We also report the correlation between the automatic metrics and these
human assessments, which we find to be mostly weak. Our results show that
while pre-trained models are effective at simplification in English, their
abilities degrade significantly on other languages, where they tend to
introduce factuality errors. GPT-3 (zero-shot) yields outputs that are
comparatively factually accurate, but which tend to be extractive and so not
adequately simplified. Flan-T5 (zero-shot) degrade significantly in quality
when simplifying to other languages, producing many factual and fluency
errors. We also analyze the approach of translating English simplifications to
other languages, which in many instances is able bypass these issues.

We publicly release MultiCochrane, model outputs, and all human judgments
collected for this work.333https://github.com/SebaJoe/MultiCochrane Our hope
is that this dataset and our initial results motivate essential future work on
multilingual medical text simplification to advance model performance across
languages.

##  2 Related Work

The largest resources used to train automatic simplification models are two
general domain corpora: the Wikipedia-Simple Wikipedia corpus (Zhu et al.,
2010; Woodsend and Lapata, 2011; Coster and Kauchak, 2011; Jiang et al.,
2020), and the Newsela corpus (Xu et al., 2015; Jiang et al., 2020). This
paper focuses on simplification in the medical domain, which is important if
we are to bridge the information gap exemplified by low medical literacy
levels worldwide (Kickbusch et al., 2013).

##### Medical Simplification Datasets.

While there have recently been increasing efforts to create parallel corpora
for medical text simplification in English Van den Bercken et al. (2019); Cao
et al. (2020); Devaraj et al. (2021); Guo et al. (2022); Basu et al. (2023),
data in other languages remain scarce. Grabar and Cardon (2018) constructed
the CLEAR dataset in French, part of which is derived from 13 Cochrane
articles; Trienes et al. (2022) introduced a dataset on German consisting of
clinical notes. Other prior work in non-English medical text simplification
has focused on lexical simplification Abrahamsson et al. (2014); Kloehn et al.
(2018); Alfano et al. (2020), where the primary focus was on substituting
synonyms and semantically related terms. In terms of data sourcing, our work
is most similar to Devaraj et al. (2021) (which is in English only). However,
their work derived roughly-aligned paragraphs consisting of specific sections
(methods and conclusions only) of the Cochrane abstracts and PLS; by contrast,
we derive manual and automatically aligned sentences for full abstracts.

##### Multilingual Simplification Datasets.

Multilingual datasets have been introduced for the task of summarization. The
MLSUM dataset Scialom et al. (2020) is a large summarization corpus containing
article/summary pairs in five languages, sourced from newspaper articles.
General-domain non-English datasets for simplification also exist as
extensively surveyed in Ryan et al. (2023). Notably, MUSS Martin et al. (2022)
used Common Crawl to mine millions of sequence pairs in English, Spanish, and
French. The significant difference between MUSS and our dataset is MUSS’s lack
of cross-lingual alignment, i.e., alignment of sequence pairs from one
language to another. Agrawal and Carpuat (2019) derived noisily aligned
English-Spanish data from the Newsela corpus Xu et al. (2015). However, to
date, there is no aligned dataset for text simplification where one source
sentence is paired with target sentences in multiple languages simultaneously
as in our work.

##  3 The MultiCochrane Corpus

We source MultiCochrane from the Cochrane Library of Systematic Reviews’s
publicly available technical abstracts and plain-language summaries; these are
comparable texts, though not parallel Devaraj et al. (2021). Crucially, from a
multilingual perspective these abstracts and PLS exist in several languages
(detailed in Sections 3.1.2 and 3.2.2), aligned almost one-to-one with their
English counterparts. This allows annotations and alignments to be easily
adapted to create a multilingual dataset. In total, we use 7,655 abstract-PLS
pairs from the Cochrane Library.

We first create MC-clean (Section 3.1, 101 abstracts) using a mixture of
manual alignments (for English) and semi-automatic alignments with partial
verification (for other languages). The rest of the abstracts were used for
the creation of the large-scale noisy MC-noisy (Section 3.2). The total number
of sentences, along with the average number of tokens in all sentences, are
shown in Table 1.

###  3.1 Clean Alignments: MC-clean

For MC-clean, we first create a manually aligned subset of 101 abstracts for
English. We then automatically align for other languages by exploiting
Cochrane’s natural 1-1 sentence alignment from English to other languages;
this alignment was verified by annotators on a sizable sample of the data.

Our annotation team consists of 5 trained linguistics students with native
English proficiency and native or advanced mastery over one or more of the
other languages covered in this work.

| Dataset | en$\rightarrow$en | en$\rightarrow$es | en$\rightarrow$fr | en$\rightarrow$fa
---|---|---|---|---|---
S | MC-noisy | 61,194 | 59,319 | 35,402 | 31,246
\+ filtering | 30,032 | 29,351 | 17,260 | 15,514
MC-clean | 1,632 | 1,602 | 1,111 | 646
T | MC-noisy | 36.55 | 49.97 | 56.07 | 55.24
\+ filtering | 36.39 | 54.42 | 61.12 | 59.54
MC-clean | 39.11 | 55.53 | 64.87 | 61.34
Table 1: Number of sentence pairs (S) and the average number of tokens per
sentence (T) in the training set for English (en), Spanish (es), French (fr),
and Farsi (fa). MC-noisy contains a full set as well as a filtered set
(Section 3.2.1) that handles misalignments at the cost of higher
extractiveness.

####  3.1.1 English Annotation

Annotators were provided abstract-PLS pairs and were tasked with aligning
sentences in the abstract with sentences in PLS according to their similarity
in content. This is a challenging task: annotators had to be able to
understand complex medical texts, and align sentences by repeatedly searching
through text on both the simple and complex sides. To assist with their
annotation, we built the anno-viewer tool inspired by Jiang et al. (2020) to
enable annotators to efficiently align sentences (see screenshots in Appendix
F).444We release the anno-viewer annotation tool publicly together with our
datasets. This annotation tool also has functionality allowing the annotator
to use existing similarity measures (listed in Section 3.2.1) to help find
sentence alignments faster. When a sentence is selected for alignment, the
tool can sort the other document (either abstract or PLS) by which sentences
are most similar to the selected sentence. After aggregating all alignments
for a document into a bipartite graph for analysis, we processed the data into
individual sentence pairs (see Appendix D). The breakdown of different
alignment types in the dataset is presented in Table 2.

Alignment Type | % of Dataset
---|---
1 - 1 | 50.3%
N - 1 | 36.6%
1 - N | 10.1%
N - M | 2.9%
Table 2: Distribution of alignment types in MC-clean.

Due to the high cognitive load imposed by aligning medical documents, we first
selected a set of 25 documents aligned by at least 2 annotators (15 of which
aligned by at least 3). Once we were confident of their inter-annotator
agreement (see below), the rest of 75 documents were then single-annotated.

Aligned Complex Sentences | 1,325
---|---
Aligned Simple Sentences | 1,165
Unaligned Complex Sentences (Deletion) | 1,966
Unaligned Simple Sentences (Elaboration) | 591
Average Deletion Ratio | 60.1%
Average Elaboration Ratio | 28.2%
Table 3: Statistics on unaligned and aligned sentences in both complex and
simple texts for the English side of MC-clean.

##### Inter-annotator Agreement.

We used the aforementioned 25 articles to calculate inter-annotator agreement
based on the F1 score, following prior research on word and sentence alignment
Jiang et al. (2020); Lan et al. (2021).555Alignment requires $N$ (sentences)
by $M$ (sentences) comparisons, while only a small amount of which being
positive (i.e., aligned), thus F1 is more commonly used than correlation
measurements for calculating agreement for alignment. For each annotator,
their alignments were evaluated against the set of alignments from other
annotators (in the case of multiple annotators, we use aggregated alignments
where a majority of the other annotators agree).

Overall, the macro-average F1 score across annotators was 0.89, indicating
high agreement for the alignment task. Non-overlapping alignments are rare; a
major source of disagreement between annotators was related to multiple
alignments made to the same simple sentence (multi-alignments) that were
either captured by some annotators but excluded by others. Since the content
in the abstracts is highly technical (and in some cases contrasted
significantly in style and vocabulary with the simplified summary), annotators
had difficulty determining whether some sentences conveyed similar meaning. In
general, annotators were conservative, only aligning sentences when they were
confident in the alignments. Therefore, there is a low prevalence of incorrect
alignments (i.e., alignments are high-precision). Annotators typically used
anno-viewer’s sentence similarity sorting to verify their alignments or to
identify additional multi-alignments.

##### Dataset Summary.

Table 3 presents additional statistics on the aligned and unaligned sentences
in both complex and simple texts in the English portion of MC-noisy. For
complex texts (i.e., technical abstracts), an unaligned sentence indicates
that its information is not present in (_deleted_ from) the simplified text.
On average, just over 60% of sentences in complex texts is deleted, indicating
that most of the core information from complex texts is included in simple
texts. In PLS, an unaligned sentence indicates added information not present
in the complex version. We observe that an overwhelming majority of this added
information _elaborates_ on concepts or defines terms Srikanth and Li (2021).
The average elaboration ratio—ratio of unaligned sentences to total sentences
in a PLS—is less than half of the average deletion ratio (same ratio for a
technical abstract), indicating that simplified texts tend to be mostly
concise with this core information.

####  3.1.2 Multilingual Data

As mentioned above, the Cochrane Library provides abstracts and PLS in
multiple languages. These translations have been produced through a
combination of volunteer and professional work, as well as human-verified
machine translation.666More information about the translation procedure can be
found at:
https://documentation.cochrane.org/display/TH/About+translation+at+Cochrane.

To create the multilingual portion of our dataset, i.e., pairs of semantically
equivalent (source, target) sentences where source $=$ English abstract,
target $\in$ PLS in {en, es, fr, …}. The PLS versions of other languages
mostly correspond 1-1 sentence-wise to the English PLS. Thus we use the
English PLS as a bridge to align the English abstract with PLS in other
languages.

We use a sentence aligner that incorporates LASER multilingual sentence
embeddings Artetxe and Schwenk (2019) combined with a dynamic programming-
based alignment algorithm (Thompson and Koehn, 2019) to align sentences in the
English versions to those in other languages. We did this for the entire set
of 7,655 abstract/PLS pairs. Multilingual sentence pairs in MC-clean consist
of these alignments that belong to the 101 articles manually aligned for
English.

##### Human Verification.

To verify that the multi-lingual sentence alignments across different
languages of the PLS are valid, we asked 3 expert bilingual annotators
(students in our institutions) to evaluate a random sample of 400 sentence
pairs to verify that each alignment is a direct translation. Each annotator
was highly proficient in both the languages of the alignments being verified.
Annotators found zero instances of misalignment. We also analyzed the number
of sentences of the English abstract/PLS and their multilingual counterparts,
and found that in almost every instance they were practically identical. These
assessments, combined with information about the Cochrane Library’s
methodology in making these translations, instills high confidence that the
derived multilingual sentence alignments are valid.

##### Dataset Summary.

While Cochrane provides a trove of diverse multilingual data, these
multilingual versions of abstracts and PLS do not consistently exist for all
abstracts. Consequently, we were unable to derive a clean set in which all
articles are mapped to all languages. This is shown in Figure 2. Almost all of
the articles annotated had a Spanish version. There is also a sizable amount
of French and Farsi versions of the annotated articles.

For this work, we selected the top three most frequent non-English
languages—Spanish, French, and Farsi—to use for training and evaluation. We
report the resulting number of paired sentences in each language in Table 1.

###  3.2 Fully Automated Alignments: MC-noisy

While MC-clean provides clean and accurate alignments across multiple
languages, human alignment of medical documents is challenging to scale.
Fortunately, MC-clean enables us to accurately evaluate a variety of
_automatic_ alignment methods; a good automatic alignment method will help
create silver alignments expanding over the entirety of the Cochrane dataset.
This is especially important for multilingual alignments, where the number of
annotated alignments is significantly lower for some languages. While
automatic sentence alignment is standard in text simplification Zhang and
Lapata (2017); Xu et al. (2015), we are not aware of evaluation work for
sentence alignment algorithms on medical texts. The construction of MC-noisy
allows us to create a tiered version of MultiCochrane that is large enough for
training state-of-the-art neural models.

Method | Precision | Recall | F1
---|---|---|---
Jaccard | 0.737 | 0.238 | 0.360
Sentence-BERT | 0.437 | 0.416 | 0.426
TFIDF | 0.372 | 0.354 | 0.363
BERTScore | 0.468 | 0.446 | 0.457
Neural-CRF${}_{\text{Newsela}}$ | 0.454 | 0.592 | 0.514
Neural-CRF${}_{\text{Cochrane}}$ | 0.781 | 0.466 | 0.584
Table 4: Precision, Recall, F1 for sentence alignment methods evaluated on the
manually aligned English portion of MC-clean.

####  3.2.1 Automatic Alignment on English Data

We evaluate a series of automatic sentence alignment methods to determine the
approach most compatible with the medical data at hand to derive the English
side of MC-noisy:

  * •

Jaccard Similarity. The Jaccard Similarity score of every possible sentence
pair was calculated. We consider pairs that score over a threshold of 0.3
(selected on the validation set) to be “aligned”; this is the alignment method
used in Xu et al. (2015).

  * •

TF-IDF Similarity. We generated TF-IDF vectors for every possible pair. Pairs
were scored based on the cosine similarity of their embeddings. We aligned
each sentence in the PLS to the most similar sentence in the corresponding
complex version. This is the alignment method used in Paetzold et al. (2017).

  * •

Sentence-BERT Similarity. Similar to TF-IDF, but here we use Sentence-BERT
embeddings (Reimers and Gurevych, 2019) instead.777For both TF-IDF and
Sentence-BERT, we tested a version where a similarity threshold is selected;
this led to much worse F1 scores, likely due to similarity being used as an
absolute measure as opposed to a relative one.

  * •

BERTScore. The BERTScore Zhang et al. (2020) between the complex and
simplified sentences was calculated for every possible pair, and like for TF-
IDF and Sentence-BERT, we considered the highest-scoring pair for each
sentence in the PLS to be aligned.

  * •

Neural CRF Alignment. We also evaluate a BERT-based CRF alignment model (Jiang
et al., 2020). We test both the original model trained on the Newsela dataset,
and a model fine-tuned on the training set of MC-clean. The hyperparameters
used for fine-tuning are the same those used for the original model.

For experiments, we used 395 alignments from 22 MC-clean English articles as
the test set, and 84 aligned sentence pairs from 5 articles as the validation
set. The rest of the 1149 alignments from 74 articles were used for fine-
tuning the neural CRF model. Table 4 reports the precision, recall, and F1
scores of these methods against reference annotated alignments in MC-clean. We
observe that the CRF alignment model trained on the Cochrane dataset achieved
the best F1 measure; this is unsurprising, as it is the state-of-the-art
supervised method. We therefore adopt this Neural-CRF${}_{\text{Cochrane}}$
alignment model on all abstract-PLS pairs, other than those in MC-clean
already, to create MC-noisy.

##### Filtering.

Although the CRF model handles unaligned sentences by default, the resulting
data remains noisy. We therefore adapt the method described in Kim et al.
(2021), initially used in the context of sentence-splitting, to further filter
misalignments in MC-noisy. This method can be described as follows: For each
sentence pairing identified by the CRF model, we derive the set of lemmatized
tokens in the complex and simple sentence denoted by $\mathcal{L}_{c}$ and
$\mathcal{L}_{s}$ respectively. If the proportion of the overlapping tokens
from the simplified sentence exceeds a threshold $r$, it is included in the
filtered dataset:

$r=\frac{|\mathcal{L}_{c}\cap\mathcal{L}_{s}|}{|\mathcal{L}_{s}|}$ (1)

We used $r=0.5$ to strike a balance between the number of potential
misalignments in the dataset and the extractiveness of the data. Overall, as
evident in Table 1, this filtering process reduced the dataset size by about
half.

####  3.2.2 Multilinguality

Figure 2: For English abstracts included in MC-clean and MC-noisy, the number
of corresponding PLS across languages. From left to right: English (en),
Spanish (es), French (fr), Farsi (fa), Chinese (zh), Japanese (ja), Portuguese
(pt), Korean (ko), Thai (th), German (de), Russian (ru), Malay (ms), and
Croatian (hr).

We use the same 1-1 alignment property between English and other languages in
Cochrane abstracts and PLS described above (Section 3.1.2) to derive the
multilingual portion of MC-noisy from the English data described in Section
3.2.1.

As shown in Figure 2, the distribution of the articles and sentence pairs
across language in the noisy dataset is similar to that of the human-annotated
dataset. There is significantly more data present for English, Spanish,
French, and Farsi as compared to other languages.

##  4 Evaluation of Simplification Models

MultiCochrane enables extensive evaluation of multilingual sentence
simplification systems simultaneously across 4 languages. In this section we
present the first study of this kind (to our knowledge), assessing zero-shot
and fine-tuned models (Section 4.1) with both automatic metrics (Section 4.2)
and human evaluation (Section 4.3) for medical text simplification.

###  4.1 Models

We experiment with zero-shot and fine-tuned models, as well as a simplify-
then-translate pipeline.

##### Zero-shot models:

  * •

GPT-3 zero-shot. we evaluate zero-shot simplifications generated by the
GPT-3-davinci-003 model using a prompt similar to that used in August et al.
(2022), adapted for multilinguality:

> My fifth grader asked me what this sentence means: [TEXT TO SIMPLIFY] I
> rephrased it for him in [LANG], in plain language a fifth grader can
> understand:

  * •

GPT-3 simplify-then-translate. GPT-3 has been shown to have strong performance
on English simplification for medical texts Shaib et al. (2023). Therefore, we
evaluate a pipeline model first using GPT-3 for English simplification using
the above prompt, and then translating the simplified text to other languages
with a strong translation model (Google’s Cloud Translation API).

  * •

Flan-T5 zero-shot. We also evaluate zero-shot Flan-T5 Chung et al. (2022)
performance. We used a simple prompt, specifically prepending _“Simplify this
sentence:”_ to the input (complex) sentence.888Preliminary experiments showed
that August et al. (2022)’s prompt, though worked well on GPT-3, did not yield
good results with Flan-T5. When simplifying to languages other than English,
the prompt is changed to _“Simplify this sentence in [LANG]:”_. Unfortunately,
we were unable to generate simplifications in Farsi using Flan-T5, so we only
evaluated this system on English, Spanish, and French.

##### Fine-tuned models:

  * •

mT5: we fine-tune separate mT5${}_{\text{base}}$ models Xue et al. (2021) on
different (source, target) language pairs: (English, English), (English,
Spanish), (English, French), and (English, Farsi).

  * •

Flan-T5 fine-tuned: we further evaluate fine-tuned versions of Flan-T5 for
each language pair, following the setup of its zero-shot counterpart. This
system also failed to generate Farsi outputs.

##### Training Setup.

We evaluate models on train/test/validation splits shown in Table 5. The train
split is composed entirely of data from MC-noisy while test and validation
splits are subsets of MC-clean. Fine-tuned models (mT5, Flan-T5) were trained
over a single epoch with a learning rate of 5e-5 and used the AdamW optimizer.
For both mT5 and Flan-T5 (zero-shot and fine-tuned), nucleus sampling with a
top-p of 0.95 was used as the decoding strategy. GPT-3 was evaluated using a
temperature of 0.7.

| train | test | validation
---|---|---|---
en | 7655 (61194/30032) | 22 (395) | 5 (84)
es | 7600 (59319/29351) | 22 (395) | 5 (84)
fr | 4702 (35402/17260) | 11 (224) | 4 (66)
fa | 3105 (31246/15514) | 8 (149) | 3 (57)
Table 5: Train/test/validation splits used for evaluation. Both the number of
articles and the total number of alignments are displayed (for training:
unfiltered ($r=0$)/filtered ($r=0.5$)).

###  4.2 Automatic Evaluation

| System | BLEU (o, s) | BLEU (o, c) | BS (no bl) | BS (bl) | SARI | LCS
---|---|---|---|---|---|---|---
English | GPT3 (0-shot) | 2.38 | 8.63 | 0.8774 | 0.2738 | 42.111 | 0.569
Flan-T5 (0-shot) | 8.12 | 54.07 | 0.8810 | 0.2949 | 39.057 | 0.340
mT5 (r=0) | 7.66 | 29.95 | 0.8785 | 0.2801 | 40.219 | 0.466
mT5 (r=0.5) | 8.82 | 35.45 | 0.8843 | 0.3158 | 39.579 | 0.379
Flan-T5-ft (r=0.5) | 8.70 | 44.09 | 0.8875 | 0.3332 | 39.526 | 0.319
Spanish | GPT3 (0-shot) | 5.70 | 27.76 | 0.7412 | 0.2924 | 37.972 | 0.361
GPT3-simp-trans | 3.11 | 8.50 | 0.7188 | 0.2313 | 41.123 | 0.561
Flan-T5 (0-shot) | 3.04 | 16.14 | 0.7100 | 0.2070 | 37.890 | 0.367
mT5 (r=0) | 4.97 | 10.51 | 0.7337 | 0.272 | 41.224 | 0.564
mT5 (r=0.5) | 5.18 | 12.75 | 0.7381 | 0.284 | 39.522 | 0.508
Flan-T5-ft (r=0.5) | 5.21 | 13.42 | 0.7376 | 0.2825 | 40.064 | 0.486
French | GPT3 (0-shot) | 4.42 | 28.30 | 0.7325 | 0.2861 | 38.096 | 0.380
GPT3-simp-trans | 2.82 | 8.34 | 0.7134 | 0.2352 | 40.941 | 0.579
Flan-T5 (0-shot) | 1.63 | 11.93 | 0.6989 | 0.1965 | 39.452 | 0.449
mT5 (r=0) | 2.35 | 6.65 | 0.7146 | 0.2385 | 40.412 | 0.602
mT5 (r=0.5) | 2.96 | 9.01 | 0.7157 | 0.2414 | 39.096 | 0.553
Flan-T5-ft (r=0.5) | 3.89 | 9.06 | 0.7284 | 0.2752 | 40.989 | 0.528
Farsi | GPT3 (0-shot) | 1.36 | 8.91 | 0.7103 | - | 41.080 | 0.497
GPT3-simp-trans | 1.16 | 4.20 | 0.7094 | - | 43.738 | 0.577
mT5 (r=0) | 2.21 | 4.27 | 0.7111 | - | 43.280 | 0.622
mT5 (r=0.5) | 2.67 | 3.87 | 0.7139 | - | 43.154 | 0.631
Table 6: Automatic metrics for all generations.

##### Metrics.

Model outputs were evaluated through five automatic evaluation metrics:

  * •

BLEU (Post, 2018). We report both using the aligned simplified sentence as
reference (o, s), and using the original, complex sentence as reference (o,
c).

  * •

BERTScore (Zhang et al., 2020). We report both with and without language-
specific scaling; we use RoBERTa-Large as the underlying multilingual
embedding.

  * •

SARI (Xu et al., 2016). In contrast to BLEU and BERTScore, SARI takes into
account both the original and simplified references in a single score focusing
on edit operations.

  * •

Least-Common Subsequence (LCS). We calculate the longest common subsequence, a
distance metric, and has an inverse relationship with the actual least common
subsequence. This metric estimates the extractiveness of outputs (lower means
more extractive).999Exact implementation details found here:
https://github.com/luozhouyang/python-string-similarity#metric-longest-common-
subsequence.

We do not include Flesch-Kincaid as a metric, for reasons stated in Tanprasert
and Kauchak (2021), which provides a compelling analysis on how easily it be
manipulated, and instead opt for human evaluation of simplicity in Section
4.3.

Language | BLEU | LCS
---|---|---
English | 11.1 | 0.6082
Spanish | 13.02 | 0.5900
French | 10.88 | 0.6022
Farsi | 8.48 | 0.6155
Table 7: BLEU and LCS scores comparing the gold complex to gold simple
sentences in the test dataset for each language.

##### Results.

Results of these evaluations are presented in Table 6. The data in this table
show a common trend in that filtering (Section 3.2.1) largely improves
performance, indicating that filters improve data quality. A very visible
trend in this data, is that across the board, with the exception of the GPT
results, English simplifications vastly outperform those of other languages.
This probably due to the vast resource gap between English and the other
languages, as well a possible bias towards English in the pre-trained mT5-base
model.

We observe exceedingly low BLEU scores when the simplified version is used as
reference, compared to the complex version. Given that BLEU is a very lexical-
centric metric, this reveals that models tend to translate rather than
simplify. Upon inspection, we find that simplified sentences in the Cochrane
dataset frequently featured insertions and elaborations, creating at times an
information mismatch between the model outputs and the corresponding reference
simplifications. Moreover, the style in which plain language summaries were
written varied greatly.

Another attribute that is of importance when judging simplification is the
level of extractiveness, or how much the output copied the input. Based on the
LCS metric, it seems that while GPT-3 produces less extractive outputs than
English mT5, for the other languages the opposite is true. Table 7 reports
BLEU and LCS scores between the reference complex and simple sentences in the
test set., providing a “reference” level of extractiveness in the data. With
the exception of English, the mT5 outputs trained on MC-noisy were much closer
to these values than their GPT counterparts.

Extractiveness and poor multilingual simplifications are also evident in the
results for zero-shot Flan-T5 generations. For multilingual simplifications,
in particular, fine-tuning significantly increased the LCS distance while
improving BLEU and BERTScore metrics, indicating higher quality and less
extractive simplifications. Interestingly, English simplifications generated
by both settings have a significantly lower LCS distance compared those
generated by other systems. Why this occurs is unclear to us and requires
further analysis.

The simplify-translate approach to simplification in different languages
elicited similar results as its English counterpart and clearly produces less
extractive generations compared to zero-shot multilingual simplification. In
terms of LCS and SARI, this approach resulted in similar scores as those
produced by fine-tuned models. While it is appears that the simplify-translate
approach improved upon the zero-shot multilingual performance for GPT, it is
difficult to draw conclusions about its performance relative to fine-tuned
models using automatic metrics alone.

###  4.3 Human Evaluation

| System | Fact. | Fluency | Simpl.
---|---|---|---|---
English | mT5 (r=0) | 0.87 | 0.50 | 0.24
mT5 (r=0.5) | 0.59 | 0.58 | 0.31
Flan-T5-ft (r=0.5) | 0.50 | 0.25 | 0.20
GPT3 (0-shot) | 0.33 | 0.25 | 0.02
Flan-T5 (0-shot) | 0.39 | 0.18 | 0.18
Reference | 0.25 | 0.07 | 0.02
Spanish | mT5 (r=0) | 1.05 | 0.44 | 0.50
mT5 (r=0.5) | 0.89 | 0.46 | 0.41
Flan-T5-ft (r=0.5) | 0.76 | 0.83 | -0.02
GPT3 (0-shot) | 0.02 | 0.00 | 0.90
GPT3-simp-trans | 0.44 | 0.02 | 0.51
Flan-T5 (0-shot) | 1.22 | 1.05 | 0.78
| Reference | 0.50 | 0.02 | 0.17
French | mT5 (r=0) | 1.25 | 0.67 | -0.40
mT5 (r=0.5) | 1.3 | 0.70 | -0.03
Flan-T5-ft (r=0.5) | 0.97 | 0.63 | 0.04
GPT3 (0-shot) | 0.43 | 0.27 | 0.51
GPT3-simp-trans | 0.38 | 0.44 | 0.01
Flan-T5 (0-shot) | 1.11 | 1.32 | 0.49
| Reference | 0.09 | 0.04 | -0.03
Farsi | mT5 (r=0) | 0.65 | 0.49 | -0.41
mT5 (r=0.5) | 0.65 | 0.59 | -0.53
GPT3 (0-shot) | 0.40 | 0.14 | -0.28
Reference | 0.87 | 0.82 | -0.51
Table 8: Summary of human evaluation results across all evaluated systems as
well as the reference simplification, showing average factuality (0-2,
lower=better), fluency (0-2, lower=better) and simplicity (-2
(oversimplify)–2(too hard)) ratings. Best system performance bolded.

Finally, we perform human evaluation of the factuality, linguistic quality,
and simplicity of model outputs. We also evaluate the reference
simplifications. One hundred outputs were evaluated for each system. While the
alignment may be sound, the references may still contain excess deletions or
inconsistencies with the original text; prior work Devaraj et al. (2022) found
similar phenomena with the Newsela and Wiki datasets. Annotators were blinded
to the system ID (or whether they are evaluating the manually aligned pairs
from the texts themselves).

##### Metrics.

We manually evaluate the following aspects: (1) Factual faithfulness: We
measure overall factuality using a 0-2 scale, where 0 indicates no factual
issues and 2 indicates severe factual issues. (2) Fluency: We rate linguistic
fluency of outputs using a scale of 0-2, where 0 indicates fluent output and 2
indicates severe linguistic issues. (3) Simplicity: Simplicity is assessed on
a scale of -2 to 2, where -2 indicates heavy oversimplification, 0 indicates
ideal simplification, and 2 indicates heavy under-simplification.

These attributes as well as the specific criteria are further described in
Appendix A.

##### Results.

The human evaluation results presented in Table 8 agree with many of the
automatic metrics. First, annotators found English outputs from the mT5 models
to be more factual and fluent than those from the other languages. Similarly,
factuality and fluency also improved with the filtered dataset for the most
part (as was also indicated by the automatic metrics). This is probably due to
filtering removing less extractive sentence pairs from the training set.
However, for French and Farsi, filtering slightly worsened the factuality and
fluency, perhaps due to the fewer number of examples in the data. As for the
effect of filtering on simplicity, with the exception of English outputs,
filtering also seems make outputs simpler.

There is a stark difference in all three characteristics between the GPT-3 and
the mT5 outputs. GPT-3 produced significantly more factually consistent and
more fluent text. With respect to simplicity, mT5 outputs were deemed simpler
with the exception of English. However, qualitative feedback from annotators
suggests that non-English GPT-3 outputs are skewed due to a severe degree of
extractiveness; in some cases, the output is simply a word-for-word
translation of the input. This inflates the factuality and fluency scores, but
this level of extractiveness is manifest in the simplicity scores. The only
area where GPT-3 does proper simplification is in the typical English to
English case. It seems GPT-3 may be limited in its ability to perform
multilingual simplification.

The trends in the results for Flan-T5 mirror those for GPT-3. When comparing
zero-shot generations to fine-tuned generations, it is clear that Flan-T5 is
limited in its ability to perform multilingual simplification. Not only are
generations less simplified, but they frequently have factual and fluency
errors.

The simplify-translate approach proved to be fairly strong compared to other
systems which directly performed multilingual simplification. The current
state of direct multilingual simplification in these large language models
seems to be not yet surpassing monolingual simplification combined with
machine translation. However, as discussed in Section 1, a pipeline system
does come with its drawbacks in terms of cost and robustness.

Annotators also evaluated the reference labels of each complex input from the
dataset. Overall, with the exception of Farsi, these references were deemed
relatively fluent and simplified. However, due to insertions and deletions of
information that naturally occur in the data, factuality errors were detected
by annotators; these findings echo those in Devaraj et al. (2022).

##### Correlation with automatic metrics.

Table 9 displays the Pearson correlation between automatic metrics and the
human evaluation scores. Overall the correlation between them is mostly weak.
BLEU and BERTScore, not surprisingly, does have a weak inverse correlation
with factuality and fluency measures. The metric with the strongest
correlation to human scores is LCS, showing a moderate positive correlation
with factuality (more abstractive sentences tend to contain more factual
errors), as well as a moderate inverse correlation with the simplicity score
(more abstractive sentences tend to be simpler). The other simplification
metric, SARI, had varied results depending on the language.

| Auto Metric | Fact. | Fluency | Simpl.
---|---|---|---|---
English | BLEU | -0.124 | -0.032 | 0.037
BERTScore | -0.174 | -0.174 | -0.030
SARI | -0.078 | 0.033 | 0.014
LCS | 0.319 | 0.144 | -0.254
Spanish | BLEU | -0.190 | -0.100 | 0.017
BERTScore | -0.105 | -0.083 | 0.010
SARI | -0.037 | 0.091 | -0.367
LCS | 0.196 | 0.095 | -0.395
French | BLEU | -0.197 | -0.108 | 0.042
BERTScore | -0.219 | -0.086 | 0.047
SARI | 0.029 | 0.035 | -0.186
LCS | 0.392 | 0.188 | -0.253
Farsi | BLEU | -0.120 | -0.059 | 0.055
BERTScore | -0.281 | -0.074 | 0.227
SARI | -0.243 | 0.031 | 0.286
LCS | 0.319 | 0.257 | -0.347
Table 9: Pearson correlation between automatic metrics and human evaluation
scores. Human evaluation metrics: closer to 0 $=$ better; BLEU/BERTScore/SARI:
higher $=$ better; LCS: higher $=$ more abstractive.

##### Examples of common errors.

In addition to these quantitative results, annotators provided examples of
interesting errors they found while evaluating. One common error that was
found in outputs from all evaluated systems were mistranslated acronyms.
Medical acronyms often differ from one language to another. However, many
systems are limited in their ability to properly translate this technical
knowledge. Usually, the acronym is either not translated, or translated
incorrectly, often randomly. The following example in French was generated by
mT5 fine-tuned on the unfiltered version of MC-noisy.

> Input complex sentence: Stress and a sedentary lifestyle are major
> determinants of cardiovascular disease (CVD).
> Output: Le stress et le règlement inactif sont des causes importantes de
> maladies cardiovasculaires (AVC) les plus courants.
> Gloss: The stress and inactive regulation are the most common important
> causes of cardiovascular disease (AVC)

The correct translation of CVD, or cardiovascular disease, in French is MCV,
not AVC. However, interestingly, AVC is the French acronym for stroke,
indicating there is some relevance in the model output to the original
context.

Fluency issues were especially common in outputs generated by mT5 and Flan-T5.
In the case of Flan-T5, in particular, misspellings and incorrect applications
of gendered language are quite prevalent. Consider the following example in
Spanish generated by Flan-T5:

> Input complex sentence: The rates were 24.5 for the sponge and 10.9 for the
> diaphragm in the UK trial.
> Output: Las tasas de asistencia respiratoria fueron de un 2,4 para la
> secreción del vejiga y 10,9 para el diafragmata.
> Gloss: The rates of respiratory assistance were 2.4 for the secretion of the
> bladder and 10.9 for the diaphragm.

This example has a number of errors. There are many hallucinations that
include the insertion and substitution of irrelevant information. However,
with respect to fluency, the main issue is the incorrect gender used for the
word “vejiga” as well as the misspelling of the word “diafragma” as
“diafragmata”. The important thing to note here is that most of these errors
are localized to technical words or phrases. More work needs to be done to
adapt large language models to these niche vocabularies.

More notable examples can be found in Appendix E.

##  5 Conclusions

This paper introduces the first human-aligned multilingual text simplification
dataset, MC-clean, for the medical domain for the languages of English,
Spanish, French, and Farsi. We further expanded this dataset using automatic
alignments to create a much more vast but noisier dataset, MC-noisy, that
could be used for training language models. We also performed an evaluation of
multilingual simplification for various systems, testing some of their zero-
shot capabilities as well their performance when fine-tuned on MC-noisy.

## Acknowledgements

We acknowledge Pouya Nekouei for his help with the Farsi language. This
research was partially supported by National Science Foundation (NSF) grants
IIS-2145479, IIS-2144493 and IIS-2112633, and by the National Institutes of
Health (NIH) under the National Library of Medicine (NLM) grant 2R01LM012086.
We acknowledge the Texas Advanced Computing Center
(TACC)101010https://www.tacc.utexas.edu at UT Austin for many of the results
within this paper.

## References

  * Abrahamsson et al. (2014) Emil Abrahamsson, Timothy Forni, Maria Skeppstedt, and Maria Kvist. 2014.  Medical text simplification using synonym replacement: Adapting assessment of word difficulty to a compounding language.  In _Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR)_ , pages 57–65.
  * Agrawal and Carpuat (2019) Sweta Agrawal and Marine Carpuat. 2019.  Controlling text complexity in neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1549–1564, Hong Kong, China. Association for Computational Linguistics.
  * Alfano et al. (2020) Marco Alfano, Biagio Lenzitti, Giosuè Lo Bosco, Cinzia Muriana, Tommaso Piazza, and Giovanni Vizzini. 2020.  Design, development and validation of a system for automatic help to medical text understanding.  _International journal of medical informatics_ , 138:104109.
  * Alva-Manchego et al. (2020) Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2020.  Data-driven sentence simplification: Survey and benchmark.  _Computational Linguistics_ , 46(1):135–187.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _Transactions of the Association for Computational Linguistics_ , 7:597–610.
  * August et al. (2022) Tal August, Lucy Lu Wang, Jonathan Bragg, Marti A. Hearst, Andrew Head, and Kyle Lo. 2022.  Paper plain: Making medical research papers approachable to healthcare consumers with natural language processing.
  * Basu et al. (2023) Chandrayee Basu, Rosni Vasu, Michihiro Yasunaga, and Qian Yang. 2023.  Med-easi: Finely annotated dataset and models for controllable simplification of medical texts.  In _AAAI Conference on Artificial Intelligence_.
  * Cao et al. (2020) Yixin Cao, Ruihao Shui, Liangming Pan, Min-Yen Kan, Zhiyuan Liu, and Tat-Seng Chua. 2020.  Expertise style transfer: A new task towards better communication between experts and laymen.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1061–1071, Online. Association for Computational Linguistics.
  * Cardon and Grabar (2020) Rémi Cardon and Natalia Grabar. 2020.  French biomedical text simplification: When small and precise helps.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 710–716.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Coster and Kauchak (2011) William Coster and David Kauchak. 2011.  Simple English Wikipedia: A new text simplification task.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 665–669, Portland, Oregon, USA. Association for Computational Linguistics.
  * Devaraj et al. (2021) Ashwin Devaraj, Iain Marshall, Byron Wallace, and Junyi Jessy Li. 2021.  Paragraph-level simplification of medical texts.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4972–4984, Online. Association for Computational Linguistics.
  * Devaraj et al. (2022) Ashwin Devaraj, William Sheffield, Byron Wallace, and Junyi Jessy Li. 2022.  Evaluating factuality in text simplification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7331–7345, Dublin, Ireland. Association for Computational Linguistics.
  * Grabar and Cardon (2018) Natalia Grabar and Rémi Cardon. 2018.  Clear–simple corpus for medical french.  In _Proceedings of the 1st Workshop on Automatic Text Adaptation (ATA)_ , pages 3–9.
  * Guo et al. (2022) Yue Guo, Wei Qiu, Gondy Leroy, Sheng Wang, and Trevor Cohen. 2022.  Cells: A parallel corpus for biomedical lay language generation.  _arXiv preprint arXiv:2211.03818_.
  * Jiang et al. (2020) Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020.  Neural CRF model for sentence alignment in text simplification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7943–7960, Online. Association for Computational Linguistics.
  * Kickbusch et al. (2013) Ilona Kickbusch, Jürgen M Pelikan, Franklin Apfel, and Agis D Tsouros. 2013.  _Health literacy: the solid facts_.  World Health Organization. Regional Office for Europe.
  * Kim et al. (2021) Joongwon Kim, Mounica Maddela, Reno Kriz, Wei Xu, and Chris Callison-Burch. 2021\.  BiSECT: Learning to split and rephrase sentences with bitexts.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6193–6209, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kloehn et al. (2018) Nicholas Kloehn, Gondy Leroy, David Kauchak, Yang Gu, Sonia Colina, Nicole P Yuan, and Debra Revere. 2018.  Improving consumer understanding of medical text: Development and validation of a new subsimplify algorithm to automatically generate term explanations in english and spanish.  _Journal of medical Internet research_ , 20(8):e10779.
  * Lan et al. (2021) Wuwei Lan, Chao Jiang, and Wei Xu. 2021.  Neural semi-Markov CRF for monolingual word alignment.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6815–6828, Online. Association for Computational Linguistics.
  * Liu et al. (2020) Chenxi Liu, Dan Wang, Chaojie Liu, Junnan Jiang, Xuemei Wang, Haihong Chen, Xin Ju, and Xinping Zhang. 2020.  What is the meaning of health literacy? a systematic review and qualitative synthesis.  _Family medicine and community health_ , 8(2).
  * Martin et al. (2022) Louis Martin, Angela Fan, Éric Villemonte De La Clergerie, Antoine Bordes, and Benoît Sagot. 2022.  Muss: Multilingual unsupervised sentence simplification by mining paraphrases.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 1651–1664.
  * Paetzold et al. (2017) Gustavo Paetzold, Fernando Alva-Manchego, and Lucia Specia. 2017.  MASSAlign: Alignment and annotation of comparable documents.  In _Proceedings of the IJCNLP 2017, System Demonstrations_ , pages 1–4, Tapei, Taiwan. Association for Computational Linguistics.
  * Phatak et al. (2022) Atharva Phatak, David W Savage, Robert Ohle, Jonathan Smith, and Vijay Mago. 2022\.  Medical text simplification using reinforcement learning (teslea): Deep learning–based text simplification approach.  _JMIR Medical Informatics_ , 10(11):e38095.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Ruder et al. (2021) Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, et al. 2021.  Xtreme-r: Towards more challenging and nuanced multilingual evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10215–10245.
  * Ryan et al. (2023) Michael Joseph Ryan, Tarek Naous, and Wei Xu. 2023.  Revisiting non-english text simplification: A unified multilingual benchmark.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Scialom et al. (2020) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020.  Mlsum: The multilingual summarization corpus.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8051–8067.
  * Shaib et al. (2023) Chantal Shaib, Millicent L Li, Sebastian Joseph, Iain J Marshall, Junyi Jessy Li, and Byron C Wallace. 2023.  Summarizing, simplifying, and synthesizing medical evidence using gpt-3 (with varying success).  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_.  ArXiv preprint arXiv:2305.06299.
  * Siddharthan (2014) Advaith Siddharthan. 2014.  A survey of research on text simplification.  _ITL-International Journal of Applied Linguistics_ , 165(2):259–298.
  * Srikanth and Li (2021) Neha Srikanth and Junyi Jessy Li. 2021.  Elaborative simplification: Content addition and explanation generation in text simplification.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 5123–5137, Online. Association for Computational Linguistics.
  * Tanprasert and Kauchak (2021) Teerapaun Tanprasert and David Kauchak. 2021.  Flesch-kincaid is not a text simplification evaluation metric.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 1–14, Online. Association for Computational Linguistics.
  * Thompson and Koehn (2019) Brian Thompson and Philipp Koehn. 2019.  Vecalign: Improved sentence alignment in linear time and space.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1342–1348, Hong Kong, China. Association for Computational Linguistics.
  * Trienes et al. (2022) Jan Trienes, Jörg Schlötterer, Hans-Ulrich Schildhaus, and Christin Seifert. 2022.  Patient-friendly clinical notes: towards a new text simplification dataset.  In _Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022)_ , pages 19–27.
  * Van et al. (2020) Hoang Van, David Kauchak, and Gondy Leroy. 2020.  Automets: The autocomplete for medical text simplification.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1424–1434.
  * Van den Bercken et al. (2019) Laurens Van den Bercken, Robert-Jan Sips, and Christoph Lofi. 2019.  Evaluating neural text simplification in the medical domain.  In _The World Wide Web Conference_ , pages 3286–3292.
  * Vu et al. (2022) Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, and Noah Constant. 2022\.  Overcoming catastrophic forgetting in zero-shot cross-lingual generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9279–9300.
  * Woodsend and Lapata (2011) Kristian Woodsend and Mirella Lapata. 2011.  Learning to simplify sentences with quasi-synchronous grammar and integer programming.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 409–420, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Xu et al. (2015) Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015.  Problems in current text simplification research: New data can help.  _Transactions of the Association for Computational Linguistics_ , 3:283–297.
  * Xu et al. (2016) Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016\.  Optimizing statistical machine translation for text simplification.  _Transactions of the Association for Computational Linguistics_ , 4:401–415.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  BERTScore: Evaluating text generation with BERT.  In _International Conference on Learning Representations_.
  * Zhang and Lapata (2017) Xingxing Zhang and Mirella Lapata. 2017.  Sentence simplification with deep reinforcement learning.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 584–594.
  * Zhu et al. (2010) Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010.  A monolingual tree-based translation model for sentence simplification.  In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_ , pages 1353–1361, Beijing, China. Coling 2010 Organizing Committee.

##  Appendix A Human Evaluation Framework

The following is the exact framework that annotators used for evaluating
simplification outputs.

Factual (faithful to input) |  Choices - 0-2
Give an overall rating on how factually consistent the model-generated output
is on a scale from 0 to 2, where 0 is completely factual, 1 indicates
inconsequential factual errors, and 2 indicates severe factual errors.

Example:
Rating 0:
Original: We discovered a high percentage of study participants experienced
adverse reactions while being treated for ischemic heart disease.
Output: We found that many study participants had bad side effects from heart
disease treatments.

Rating 1:
Original: We discovered a high percentage of study participants experienced
adverse reactions while being treated for ischemic heart disease.
Output: We discovered some participants had bad side effects from heart
disease treatments.

Rating 2:
Original: We discovered a high percentage of study participants experienced
adverse reactions while being treated for ischemic heart disease.
Output: We discovered participants had no side effects from heart disease
treatments.
---|---
Fluency |  Choices - 0-2
Give an overall rating on how well the model-generated output follows
grammatical rules and appears as a natural sentence, where 0 indicates no
fluency issues, 1 indicates superficial issues, and 2 indicates severe fluency
issues that obfuscate the meaning of the sentence.

Example:
Rating 1:
Original: We discovered a high percentage of study participants experienced
adverse reactions while being treated for ischemic heart disease.
Output: We found that many study participants had badly side effects from
heart disease treatments.

Rating 2:
Original: We discovered a high percentage of study participants experienced
adverse reactions while being treated for ischemic heart disease.
Output: In heart disease treatments, many participants had effects bad on the
side.
Simplicity |  Choices - (-2 - 2)
Rate the level of simplification that occurred from the original English
sentence to the model-generated output. Note: since the input readability is
varied, please make judgements independent of the input.
-2: Output is severely oversimplified to the point where the original intent of the original sentence has been lost.
-1: Output is oversimplified, missing some important details. However, the intent of the original sentence is preserved.
0: Output is ideally simplified. It is readable to the average layman and does
not omit important details.
1: Output should be simplified more and include uncommon technical terms
without any elaboration/explanation.
2: Output should be simplified MUCH more. There is little to no change in the
style of the sentence from the original sentence, and the output is difficult
for the average layman to understand.

Example:
Rating -2:
Original: We discovered a high percentage of study participants experienced
adverse reactions while being treated for ischemic heart disease.
Output: Study participants had some effects while being treated for some
disease.

Rating -1:
Original: We discovered a high percentage of study participants experienced
adverse reactions while being treated for ischemic heart disease.
Output: Study participants had bad effects while being treated for heart
disease.

Rating 0:
Original: We discovered a high percentage of study participants experienced
adverse reactions while being treated for ischemic heart disease.
Output: We found that many study participants had bad side effects from heart
disease treatments.

Rating 1:
Original: We discovered a high percentage of study participants experienced
adverse reactions while being treated for ischemic heart disease.
Output: We found that many study participants had experienced adverse effects
from ischemic heart disease treatments.

Rating 2:
Original: We discovered a high percentage of study participants experienced
adverse reactions while being treated for ischemic heart disease.
Output: We discovered a high percentage of study participants experienced
antagonistic reactions while being remedied for ischemic heart disease.

##  Appendix B Human Evaluation Score Distributions

Figure 3 show the distributions of the human evaluation scores for all
evaluated systems.

Figure 3: These graphs show the distribution of human evaluation scores for
each evaluated system in English, Spanish, French, and Farsi.

##  Appendix C Human Evaluation Agreement

| mT5 (r=0) | mT5 (r=0.5) | GPT-3 | Gold
---|---|---|---|---
Fact. | 44% | 56% | 59% | 50%
Fluency | 52% | 52% | 70% | 92%
Simpl. | 37% | 33% | 49% | 49%
Table 11: % of examples that all 3 annotators agreed on a label. | mT5 (r=0) | mT5 (r=0.5) | GPT-3 | Gold
---|---|---|---|---
Fact. | 94% | 93% | 99% | 94%
Fluency | 98% | 95% | 100% | 100%
Simpl. | 97% | 92% | 93% | 92%
Table 12: % of examples that at least 2 annotators agreed on a label.

In addition to evaluating outputs in their respective languages, 3 out of the
4 human evaluators (for English, French, and Spanish) also evaluated the four
sets of English outputs (unfiltered mT5, filtered mT5, GPT-3, and gold
reference labels). These were collected to estimate how similarly annotators
would evaluate outputs.

Table 11 and Table 12 show the percent agreement for each human evaluation
measure for the evaluated outputs. Overall, there is moderate agreement among
annotators for most systems.

##  Appendix D Alignment Analysis

Alignment Type | Method A | Method B
---|---|---
1 - 1 | 36.4% | 50.3%
N - 1 | 20.3% | 36.6%
1 - N | 9.4% | 10.1%
N - M | 33.9% | 2.9%
Table 13: Distribution of alignment types in MC-clean for each method used.

Since annotators were given total freedom to align sentences, many different
alignment relationships are present in MC-clean. To quantify the proportion of
alignments that belong to these alignment relationships, two methods were used
for analysis. For future reference, 1-1, N-1, 1-N, and N-M alignments refer to
number of complex sentence aligned to the number of simple sentences in an
alignment group.

##### Method A.

This method treats alignments in a document as edges in a bipartite graph,
with complex and simple sentences as vertices. Relationships are found by
tracking the connected components in this graph through a depth-first search.
The type of relationship is determined by a 2-tuple of the number of complex
sentences and the number of simple sentences in a connected component.

##### Method B.

This method is an extension of Method A that specifically targets on breaking
up N-M alignment relationships. While Method A counts any connected component
as an alignment group, this method requires that each component must be fully
connected (every complex sentence is aligned to every simple sentence in the
group) as well. As such, an N-M alignment group that is not fully connected is
broken up into smaller fully connected components.

The results of this analysis is presented in Table 13. The results from Method
A can be viewed as a lower bound for 1-1, N-1, and 1-N, alignment types.
Method B could produce different results depending on how the N-M alignment
groups are broken up. It is difficult to know exactly how to break up an
alignment group in Method B without knowing the annotator’s intentions, so
heuristic methods were used. For these results, the alignment groups were
broken down such that simple sentences that were aligned to all complex
sentences or vice versa were first grouped together. Then, the rest of the
alignments were split into 1-N or N-1 groups.

##  Appendix E Notable Examples

###  E.1 Good Simplifications

Below are some examples of good simplifications made by the systems. GPT3
systems in particular were good at explaining medical terms and information
present in parentheses. Meanwhile, other systems like Flan-T5-ft (r=0.5) and
Flan-T5 (0-shot) had fluency issues and hallucinations even in the best
simplifications and had more difficulty with longer input sentences. mT5
outputs had issues with extractiveness and retained complex medical
vocabulary.

Table 14:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
English | GPT3 (0-shot) |  We also found improved success rates within two attempts (RR 1.78, 95% CI 1.25 to 2.51, 134 catheters, two RCTs, moderate-quality evidence) with ultrasound guidance compared with other types of guidance. |  This sentence is talking about a study that was done to compare two different types of guidance when using a medical tool called a catheter. The study found that if ultrasound guidance was used, it was more successful than other types of guidance. Out of 134 catheters that were used, the success rate was 1.78 times higher when ultrasound guidance was used. This result was reliable because two different tests were done, and the evidence was of moderate quality. |  -
Spanish | GPT3 (0-shot) |  Four RCTs compared ultrasound with palpation, and one compared ultrasound with Doppler auditory assistance. |  Cuatro estudios de investigación evaluaron la eficacia de usar ultrasonido para detectar problemas médicos, comparándolo con usar la palpación (tocar con las manos) y uno de los estudios comparó el uso del ultrasonido con ayuda de un aparato llamado Doppler para escuchar los sonidos del cuerpo. |  Four investigative studies evaluated the efficacy of using ultrasound to detect medical problems, comparing it to using the palpation (touching with the hands) and one of the studies compared the use of the ultrasound with the help of an apparatus called Doppler to listen to the sounds of the body.
English | mT5 (r=0.5) |  There were no serious adverse events related to either nitrous oxide or carbon dioxide pneumoperitoneum (three studies; 196 participants; very low quality of evidence). |  There were no serious side effects related to either nitrous oxide or carbon dioxide pneumoperitoneum. |  -
French | mT5 (r=0.5) |  Thiospot was more effective than placebo (SMD -2.61, 95% CI -3.76 to -1.47). |  La thiospots était plus efficace que le placebo. |  Thiospot was more effective than the placebo.
Spanish | GPT3-simp-trans |  There were no serious adverse events related to either nitrous oxide or carbon dioxide pneumoperitoneum (three studies; 196 participants; very low quality of evidence). |  Los tres estudios realizados con 196 personas mostraron que no hubo efectos negativos al respirar gas nitrógeno o dióxido de carbono durante un procedimiento médico. |  The three studies carried out with 196 people showed that there were no negative effects in breathing nitrogen gas or carbon dioxide during a medical procedure.
French | GPT3-simp-trans |  To assess interventions used in the management of all types of melasma: epidermal, dermal, and mixed. |  Pour déterminer quels traitements fonctionnent le mieux pour les différents types de mélasma (une affection cutanée), comme le type qui affecte la couche supérieure de la peau, le type qui affecte la couche sous la couche supérieure et le type qui affecte les deux couches. |  In order to determine which treatments work the best for the different types of melasma (a skin affliction), like the type which affects the top layer of the skin, the type which affects the layer under the top layer and the type which affects the two layers.
Spanish | mT5 (r=0) |  For smokeless tobacco use, no difference was found between the skills-community arm and the control group at 42 weeks (RR 0.93, 95% CI 0.67 to 1.30), though a significant difference was observed between the skills-only arm and the control group (RR 0.57, 95% CI 0.39 to 0.85). |  En cuanto a los fumadores sin cigarrillos, se identificaron pocos casos y no hubo diferencias entre el grupo de ejercicios y el grupo control a las 42 semanas, pero se observaron efectos secundarios entre el grupo de ejercicios y el grupo de control. |  As for smokers without cigarettes, few cases were identified and there were no differences between the exercise group and the control group at 42 weeks, but side effects were observed between the exercise group and the control group.
French | mT5 (r=0) |  We searched the Cochrane Stroke Group Trials Register (16 January 2018), the Cochrane Central Register of Controlled Trials (CENTRAL; 2017, Issue 12) in the Cochrane Library (January 2018), MEDLINE Ovid (1946 to January 2018), Embase Ovid (1974 to January 2018), Web of Science Core Collection (1950 to January 2018) and three Chinese databases (January 2018). |  Nous avons recherché des études complémentaires publiées jusqu’au 16 janvier 2018. |  We researched additional published studies until the 16 of January 2018.
Spanish | Flan-T5-ft (r=0.5) |  We included seven trials, involving 735 participants, in this review. |  Se incluyeron siete ensayos, que incluyeron 735 participantes. |  Seven trials, involving 735 participants, were included.
French | Flan-T5-ft (r=0.5) |  The quality of the evidence was rated as high. |  La qualité des preuves était élevée. |  The quality of the evidence was high.
Spanish | Flan-T5 (0-shot) |  We searched until January 2016. |  También buscamos hasta enero 2016. |  We also searched until January 2016.
English | Flan-T5 (0-shot) |  We searched the Cochrane Kidney and Transplant Specialised Register up to 17 August 2017 through contact with the Information Specialist using search terms relevant to this review. |  Cochrane Kidney and Transplant Specialised Register was searched for up to 17 August 2017. |

###  E.2 Fluency

Fluency errors across systems include repetition and grammatical errors, as
well as the use of foreign words in the Flan outputs.

##### Repetition.

This category includes examples of fluency errors involving the repetition of
a single phrase or word that confuses the meaning of the output and, in some
cases, makes it completely meaningless. This error was present in all systems
except for the GPT3 (0-shot) and GPT3-simp-trans systems.

Table 15:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
English | mT5 (r=0) |  In people with symptomatic intracerebral artery stenosis, recurrent stroke was significantly reduced by RIC (risk ratio (RR) 0.32, 95% confidence interval (CI) 0.12 to 0.83; 2 trials, 182 participants, low-quality evidence). |  Compared to other antipsychotic drugs, recurrent stroke was significantly reduced by recurrent stroke. |  -
English | mT5 (r=0.5) |  Similarly, our results suggest, but do not confirm, the possibility of a positive influence of expertise in the use of ultrasound on the first attempt success rate. |  A significantly significantly significant difference in the use of ultrasound on the first attempt success rate. |  -
Spanish | Flan-T5-ft (r=0.5) |  We located 10 RCTs involving 2961 participating surgeons performing an operation in which the use of blunt needles was compared to the use of sharp needles. |  Se identificaron 10 ECA con 2961 médicos que realizaron una operación que fue realizada con agujas y cuando se comparó agujas tópicas con agujas tópicas y con agujas agujas. |  We identified 10 RCTs with 2961 doctors who performed an operation that was performed with needles and when topical needles were compared with topical needles and with needles needles.
Spanish | mT5 (r=0) |  Results of the ongoing trials will add to the evidence base but additional longer-term, high-quality trials are needed. |  Los resultados de los ensayos en curso adicionales a corto plazo, adicionales, de alta calidad, se necesitan más estudios de alta calidad. |  The results of the ongoing, additional short-term trials, additional [trials], of high quality, more high-quality studies are needed.
French | mT5 (r=0) |  Smoking prevalence in Indigenous youth is twice that of the non-Indigenous population, with tobacco experimentation commencing at an early age. |  La prévalence du tabagisme chez les jeunes jeunes jeunes développent des problèmes d’éstomac à l’avenir et présentent une amélioration des symptômes. |  The prevalence of smoking in the youth youth youth developed stomach problems in the future and presents an improvement of the symptoms.
French | mT5 (r=0.5) |  The rates were 24.5 for the sponge and 10.9 for the diaphragm in the UK trial. |  Les taux étaient étaient 25 pour le tube de sponge et 10,9 pour le diaphragme au Royaume-Uni. |  The rate was was 25 for the sponge tube and 10.9 for the diaphragm in the UK.

##### Grammar.

This category includes examples of grammatical issues in the output sentences,
such as the overuse of prepositions, spelling errors, and disagreement between
articles and nouns. Multiple prepositions appeared sequentially in a way that
was incoherent, and there were instances where articles did not agree with
nouns in gender or number.

Table 16:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
French | mT5 (r=0) |  There appears to be no significant differences in the improvement of technical skills between different methods of box model training. |  En ce qui concerne les modifications de la technique manuelle des méthodes de sélection de modèles bovés, la majorité des participants dans leur étude n’ont montré pas de différence significative entre les méthodes d’entrainement au modèle d’étudiant de modèle d’évaluation en termes d’amélioration des compétences informatiques pour différentes méthodes d’entraînement au modèle bouchon. |  Concerning the modifications of the manual technique of the selection methods of bové(?) models, the majority of participants in their study didn’t show a significant difference between the training methods in the student model of the evaluation model in terms of improvement of informatic skills for the different training methods in the blockage model.
Spanish | Flan-T5-ft (r=0.5) |  carbon dioxide, helium, argon, nitrogen, nitrous oxide, and room air) used for establishing pneumoperitoneum in participants undergoing laparoscopic general abdominal or gynaecological pelvic surgery. |  El uso de los dióxidos carbonosas, el alto carbono, la argón, la náusea y el fármaco nitroso (es decir, los flujos náuseos y el fármaco nitroso). |  The use of the carbon dioxides, the high carbon, the argon, the nausea and the nitrous drug (which is to say, the nausea flows and the nitrous drug).
Spanish | Flan-T5-ft (r=0.5) |  Indigenous youth in particular commence smoking at an early age, and a disproportionate burden of substance-related morbidity and mortality exists as a result. |  Es probable que la morbilidad de los nios de ingesta a un término temprano o un pas a tener un empeoramiento de salud y muerte debido a una pérdida del tratamiento. |  It is likely that the morbidity of the chiren of intake to an early term or a ste to have a worsening of health and death due to a loss of the treatment.

##### Foreign words (Flan-T5).

This category includes examples of fluency errors involving the use of foreign
words in the output sentences. For example, Flan-T5 (0-shot) produced a French
output that started with the English phrase ‘in four studies’, and Flan-T5-ft
(r=0.5) produced an English output that started with the Spanish word ‘otros’.
This type of error was mostly found in the Flan-T5-ft (r=0.5) and Flan-T5
(0-shot) systems.

Table 17:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
French | Flan-T5-ft (r=0.5) |  It usually involves restricting blood flow to limbs and then releasing the ischaemic blood to promote a neuroprotective effect. |  L’exercice consiste généralement à la restriction du flux sanguin de la membres des limbs et ensuite à libérer les bloods ischémies pour promouvoir un effet neuroprotecteur. |  The exercise generally involves the restriction of the blood flow to the members of the limbs and then to free the ischemic blood to promote a neuroprotective effect.
French | Flan-T5 (0-shot) |  In four studies, the use of blunt needles reduced the number of self-reported needle stick injuries with a RR of 0.31 (95% CI 0.14 to 0.68). |  In four studies, l’utilisation des nejéros strru00f7u a réduit l’injurage autoremprésenté aux nejéros d’une rRR de 0,31, en moyenne de 0,14 à 0,68. |  In four studies, the use of nejéros strru00f7u (?) to reduce the autoremprésenté (?) injury to the nejéros (?) of a risk ration of .31, on average from .14 to .68.
English | Flan-T5-ft (r=0.5) |  Therefore, other gases have been introduced as alternatives to carbon dioxide for establishing pneumoperitoneum. |  Otros such as carbon dioxide are used as alternative gases for establishing pneumoperitoneum. |  -
Spanish | Flan-T5 (0-shot) |  Remote ischaemic conditioning (RIC) has been developed as a neuroprotective strategy to prevent and treat ischaemic stroke. |  El afecto izquierdo de condicionamiento remoto (RIC) ha sido desarrollado como una estrategia neuropromotiva para prevenir y tratar strokes izquierdo. |  The remote conditioning left effect (RIC) has been developed as a neutropromotive to help prevent and treat left strokes.

###  E.3 Factuality

Factuality errors displayed by the systems could be broadly categorized into 3
types - minor hallucinations, extreme hallucinations, and contradictions.

##### Minor hallucinations.

This category includes examples where the output contained minor details that
were not present in the input (hallucination). The systems would hallucinate
details like numbers, dates, medical terms, incorrect expansions of acronyms,
etc. GPT3 systems would also misinterpret the context of words present in the
input. In one of the examples, the word ‘rates’ in the input is referring to
‘pregnancy rates’ but GPT3 misinterpreted the word to mean ‘cost’.

Table 18:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
French | mT5 (r=0) |  To assess interventions used in the management of all types of melasma: epidermal, dermal, and mixed. |  Cette revue a examiné différents type de mélanome, de la peau, d’œil et des organes, et s’est améliorée dans des années futures. |  This review has examined different types of melanoma, of the skin, of the eye and of the organs, and improved in the future years.
Spanish | GPT3 (0-shot) |  It usually involves restricting blood flow to limbs and then releasing the ischaemic blood to promote a neuroprotective effect. |  Esta oración está hablando de un procedimiento médico llamado Preacondicionamiento Isquémico (IPC). El procedimiento consiste en cortar temporalmente el flujo de sangre a ciertas partes del cuerpo y luego devolver la sangre al cuerpo. El objetivo de este procedimiento es proteger el cerebro y el sistema nervioso del daño. |  This sentence is talking about a medical procedure called Ischemic Preconditioning (IPC). The procedure consists of temporarily cutting off the flow of blood to certain parts of the body and then returning the blood to the body. The objective of this procedure is to protect the brain and the nervous system from damage.
Spanish | mT5 (r=0.5) |  We included five RCTs reporting 444 arterial cannulations in paediatric participants. |  Se incluyeron cinco ensayos controlados aleatorios que informaron 556 cannulaciones arteriales en los participantes niños. |  Five randomized controlled trials that reported 556 arterial cannulations in pediatric participants were included.
French | mT5 (r=0) |  The rates were 24.5 for the sponge and 10.9 for the diaphragm in the UK trial. |  Les taux étaient de 18,3 pour le diaphragme et de 10,9 pour le diaphragme avec un diaphragme UK dans l’essai. |  The rates were 18.3 for the diaphragm and 10.9 for the diaphragm with a UK diaphragm in the trial.
English | mT5 (r=0) |  We included three trials (involving 371 participants) in the analysis of the effects of RIC on ischaemic stroke prevention. |  We included three trials with 373 participants and provided one trial with seven trials involving 373 participants. |  -
French | mT5 (r=0) |  The adverse events most commonly reported were mild and transient such as skin irritation, itching, burning, and stinging. |  Les effets secondaires les plus fréquemment signalés étaient les plus graves et transitoires : la irritation cutanée et le fuite d’eau, les brûlures, les vertiges et les vertiges. |  The secondary effects most frequently reported were the most serious and transitory: skin irritation and the water leak, burns, dizziness and dizziness.
English | mT5 (r=0) |  We searched the Cochrane Kidney and Transplant Specialised Register up to 17 August 2017 through contact with the Information Specialist using search terms relevant to this review. |  We searched scientific databases for studies on 27 March 2017. |  -
French | mT5 (r=0.5) |  We included seven trials, involving 735 participants, in this review. |  Nous n’avons trouvé aucun essai, avec un total de 535 participants. |  We haven’t found a trial, with a total of 535 participants.
English | Flan-T5-ft (r=0.5) |  Our results suggest, but do not confirm, that a possible advantage of ultrasound guidance for the first attempt success rate over other techniques is more pronounced in infants and small children than in older children. |  Furthermore, ultrasound guidance is more accurate and safer than alternative techniques for treating first tentative baby ear infection. |  -
English | GPT3 (0-shot) |  The rates were 24.5 for the sponge and 10.9 for the diaphragm in the UK trial. |  In the UK trial, it cost 24.5 pounds to buy a sponge and 10.9 pounds to buy a diaphragm. |  -

##### Extreme hallucinations.

This category includes examples of hallucinations that were extremely
irrelevant. Information hallucinated in the output sentences has no connection
to the information present in the input. This degree of hallucinations was
much more prevalent and extreme in the Flan systems as compared to other
systems.

Table 19:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
English | Flan-T5 (0-shot) |  Randomized controlled trials based on uniform and standard methods are needed. |  The aim of the experiments is to give patients a chance to perform the experiment they want to do. |  -
French | Flan-T5-ft (r=0.5) |  In acute ischaemic stroke, for people receiving intravenous thrombolysis, the rate of death or dependency was significantly increased by RIC treatment compared with non-RIC treatment (RR 2.34; 95% 1.19 to 4.61; 1 trial, 285 participants, low-quality evidence). |  Pour les personnes victimes d’un AVC aigu, les femmes admissées de la greffe avait une probabilité accrue d’être victimes ou de la maladie grâce à un traitement remmanant par injection d’une coagulation récurrente. |  For the victims of acute stroke, the women administered (?) the transplant had an increased probability of being victims or of the disease thanks to a remmanant (?) treatment by an injection of a recurrent coagulation.
Spanish | mT5 (r=0.5) |  In people with carotid stenosis undergoing carotid stenting, there was no significant difference in the incidence of ischaemic stroke between participants treated with RIC and non-RIC (RR 0.22, 95% CI 0.01 to 4.03; 1 trial, 189 participants, low-quality evidence); however the stroke severity (assessed by infarct volume) was significantly lower in participants treated with RIC (mean difference (MD) -0.17 mL, 95% CI -0.23 to -0.11; 1 trial, 189 participants, low-quality evidence). |  En los pacientes con enfermedad isquémica cuando se acortan los brazos en los pacientes tratados con ENIC y ningún riesgo de caídas. |  In the patients with ischemic disease, when arms are shorted in the patients treated with ENIC and no risk of falls.

##### Contradictions.

This category includes examples where the output directly contradicts ideas
stated in the input. This error was infrequent in the GPT3 systems but was
equally prevalent in other systems.

Table 20:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
French | Flan-T5 (0-shot) |  There was considerable heterogeneity between the studies included in this review and studies were small and at some risk of bias |  Il y a lieu d’échantillonner des études incluant dans cette étude et les études étaient grandes et en hausse une hausse de la biase. |  There is a reason to sample the studies included in this study and the studies were big and increasing an increase of bias.
English | mT5 (r=0) |  There is qualitative evidence that oral morphine has much the same efficacy as other available opioids. |  It is not clear if oral morphine is as effective as other available opioids. |  -
French | mT5 (r=0.5) |  In people with acute ischaemic stroke, there was no significant difference between RIC and non-RIC for reducing stroke severity as assessed by the National Institutes of Health Stroke Scale score and the final infarct volume (standardised mean difference (SMD) -0.24 mL, 95% CI -1.02 to 0.54; 2 trials, 175 participants, very low quality evidence). |  La SEP réduisait la gravité du AVC ischémique, mais les essais confondus avaient des preuves de très faible qualité. |  The SEP reduced the severity of ischemic stroke, but the included trials have evidence of very weak quality.
Spanish | mT5 (r=0.5) |  Our results suggest, but do not confirm, that a possible advantage of ultrasound guidance for the first attempt success rate over other techniques is more pronounced in infants and small children than in older children. |  Los resultados indican que la guía ecográfica redujo la tasa de éxito del ensayo más amplia en los lactantes y los niños pequeños en comparación con otras técnicas. |  The results indicate that the ultrasound guidance reduced the success rate of the broader trial in the infants and the young children in comparison with other techniques.
English | mT5 (r=0.5) |  There is qualitative evidence that oral morphine has much the same efficacy as other available opioids. |  The evidence for this topic is current to December 2018 that oral morphine has a serious toxicity as a different opioid. |  -

###  E.4 Simplification

Simplification errors included instances of the system being overly
extractive, oversimplification of terms and ideas, and excessive deletion of
entire parts of the original text.

##### Extractive.

GPT3 (0-shot) was highly extractive and failed to explain complex medical
language in everyday terms that a general audience could understand. The
system also left in statistics from the input sentence that are not
interpretable to the general public. mT5 (r=0) exhibited similar patterns with
regards to extractiveness.

Table 21:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
Spanish | GPT3 (0-shot) |  Interventions included: thrombolytic therapy versus placebo (1 study); low versus high dose thrombolytic therapy (1); alteplase versus urokinase (1); short versus long thrombolytic dwell (1); thrombolytic therapy versus percutaneous fibrin sheath stripping (1); fibrin sheath stripping versus over-the-wire catheter exchange (1); and over-the-wire catheter exchange versus exchange with and without angioplasty sheath disruption (1). |  Esta frase describe un estudio que se hizo para comparar diferentes tratamientos para ayudar a mejorar la circulación sanguínea. Estos tratamientos incluyeron: terapia con trombolíticos en comparación con un placebo (1 estudio); dosis baja en comparación con dosis alta de trombolíticos (1); alteplasa en comparación con uroquinasa (1); tiempo corto de trombolíticos en comparación con tiempo largo (1); terapia con trombolíticos en comparación con despeje de fibrina con una aguja (1); despeje de fibrina en comparación con cambio de catéter con alambre (1); y cambio de catéter con alambre en comparación con cambio sin despeje de fibrina con angioplastia (1). |  This sentence describes a study that was conducted to compare different treatments to help improve the blood circulation. These treatments included: therapy with thrombolytics in comparison with a placebo (1 study); low dose in comparison with high dose of thrombolytics (1); alteplase compared to urokinase (1); short time of thrombolytics compared with long time (1); therapy with thrombolytics compared to clearance of fibrin with a needle (1); clearance of fibrin in comparison with wire catheter change (1); and wire catheter change in comparison with change without clearance of fibrin with angioplasty (1).
Spanish | mT5 (r=0) |  On current data there is no evidence to support physical intervention over the use of pharmaceutical agents in the acute setting. |  Esta evidencia no apoya la intervención física sobre la administración de agentes farmacológicos en la práctica aguda. |  This evidence does not support the physical intervention over the administration of pharmaceutical agents in the acute practice.
French | GPT3 (0-shot) |  In acute ischaemic stroke, for people receiving intravenous thrombolysis, the rate of death or dependency was significantly increased by RIC treatment compared with non-RIC treatment (RR 2.34; 95% 1.19 to 4.61; 1 trial, 285 participants, low-quality evidence). |  En cas de crise d’ischémie aiguë, pour les personnes qui ont reçu un traitement par thrombolyse par voie intraveineuse, le taux de décès ou de dépendance a été significativement augmenté par le traitement RIC par rapport au traitement non RIC (RR 2,34; 95% 1,19 à 4,61; 1 essai, 285 participants, preuves de faible qualité). |  In the case of an acute ischemic crisis, for the people who have received a treatment by intravenous thrombolysis, the rate of death or dependence was significantly increased by the RIC treatment in comparison to the non-RIC treatment (RR 2.34; 95% 1.19 to 4.61; 1 trial, 285 participants, evidence of weak quality).

##### Oversimplification

Flan-T5-ft (r=0.5) had issues with excessive deletions, resulting in
fragmented sentences that failed to communicate the meaning of the input
sentence. Excessive deletion in mT5 (r=0) and mT5 (r=0.5) manifested as a loss
of information that was necessary to preserve the meaning of the input
sentence. The resulting outputs were missing critical information, causing
them to be generic and vague, such as in the Spanish mT5 (r=0.5) example that
discusses the use of RIC for ischemic stroke prevention.

Table 22:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
French | Flan-T5-ft (r=0.5) |  We rated the quality of evidence for all outcomes as moderate owing to imprecision due to wide confidence intervals, modest sample sizes and limited numbers of events. |  La qualité des preuves |  The quality of the evidence
French | mT5 (r=0) |  Four RCTs compared ultrasound with palpation, and one compared ultrasound with Doppler auditory assistance. |  Cinq ECR ont comparé l’échographie avec une palpation. |  Five RCTs have compared ultrasounds with palpation.
French | mT5 (r=0.5) |  We included three trials (involving 371 participants) in the analysis of the effects of RIC on ischaemic stroke prevention. |  Les auteurs de la revue ont inclus un total de 371 participants. |  The authors of the review included a total of 371 participants.
Spanish | mT5 (r=0.5) |  The quality of the evidence was rated as high. |  Calidad de la evidencia |  Quality of the evidence
Spanish | mT5 (r=0.5) |  We included three trials (involving 371 participants) in the analysis of the effects of RIC on ischaemic stroke prevention. |  Se incluyeron tres ensayos que incluyeron a 372 participantes. |  Three trials with 372 participants were included.
English | mT5 (r=0) |  There were no serious adverse events related to either nitrous oxide or carbon dioxide pneumoperitoneum (three studies; 196 participants; very low quality of evidence). |  There were no serious adverse events. |  -
English | mT5 (r=0.5) |  We reran the searches for all databases in March 2016, and when we update the review, we will deal with the two studies of interest found through this search that are awaiting classification. |  The search for most scientific databases in March 2016. |  -

###  E.5 Other Errors

##### Repetition of output (GPT3 (0-shot)).

The GPT3 (0-shot) system had a unique error that exclusively appeared in the
French outputs where a separate paraphrase was generated in addition to the
main output sentence. The system sometimes gave a generated output, then
repeated the same information in different words using the phrase ‘in other
words’ or simply generating two different versions of a simplified output.

Table 23:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
French | GPT3 (0-shot) |  All but one trial were at high risk of bias. |  Tous les essais sauf un étaient à haut risque de biais. En d’autres mots, la plupart des essais présentaient un risque élevé d’être biaisés. |  All the trials except one were at high risk of bias. In other words, the majority of the trials presented an elevated risk of being biased.
French | GPT3 (0-shot) |  We rated the quality of evidence for all outcomes as moderate owing to imprecision due to wide confidence intervals, modest sample sizes and limited numbers of events. |  Nous avons estimé que la qualité de la preuve pour tous les résultats était modérée en raison d’une imprécision due à des intervalles de confiance larges, des tailles d’échantillon modestes et à un nombre limité d’événements. Nous avons estimé que la preuve était pas très précise parce que les intervalles de confiance étaient grands, les échantillons pas très grands et il y avait peu d’événements. |  We estimated that the quality of the evidence for all the results was moderate on account of an imprecision due to large confidence intervals, the size of the modest samples and to a limited number of events. We estimated that the evidence wasn’t very precise because the confidence intervals were big, the not very big samples and there were few events.

##### Acronyms.

Across all systems, a frequent issue that arose when simplifying medical text
was acronyms. These issues can be further broken down into failure to
translate into the target output language, hallucinated acronym generations,
and factually incorrect generations.

##### Acronym translation.

Systems failed to translate the input sentence’s acronym from English into
Spanish or French. GPT3 (0-shot) would explicitly state that the acronym was
in English, rather than translating, in some cases.

Table 24:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
Spanish | GPT3-simp-trans |  To assess the benefits and harms of RIC for preventing ischaemic stroke and for treating people with ischaemic stroke and those at risk for ischaemic stroke. |  Esta oración nos pide que averigüemos si cierto tipo de tratamiento médico (RIC, por sus siglas en inglés) es útil para prevenir y tratar los accidentes cerebrovasculares. Queremos saber si los beneficios de usar RIC son mayores que los riesgos. |  This sentence is asking us to find out if a certain medical treatment (RIC, by its acronym in English) is useful to prevent and treat strokes. We want to know if the benefits of using RIC are greater than the risks.
French | GPT3 (0-shot) |  We included randomised controlled trials (RCTs) comparing RIC with sham RIC or medical management in people with ischaemic stroke or at risk of ischaemic stroke. |  Nous avons inclus des essais contrôlés randomisés (RCT) pour comparer le RIC avec un faux RIC ou une gestion médicale chez les personnes ayant subi un accident vasculaire cérébral ou à risque d’accident vasculaire cérébral. |  We have included randomly controlled trials (RCT) to compare the RIC with a fake RIC or medical management in people that suffered a stroke or at risk of a stroke.
French | Flan-T5 (0-shot) |  To assess the benefits and harms of RIC for preventing ischaemic stroke and for treating people with ischaemic stroke and those at risk for ischaemic stroke. |  évaluer les avantages et les éléments de la RIC en vue de supprimer le stroke insécutif et en vue de réduire les risques liées à l’insécutif. |  To evaluate the advantages and the elements of RIC in order to remove the insécutif (?) stroke and in order to reduce the risks associated with the insécutif (?).

##### Acronym hallucination.

Flan-T5-ft (r=0.5) hallucinated acronyms in the output sentence that do not
exist in the target language. This occurred across all languages and in both
Flan and mT5 systems.

Table 25:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
English | Flan-T5-ft (r=0.5) |  In acute ischaemic stroke, for people receiving intravenous thrombolysis, the rate of death or dependency was significantly increased by RIC treatment compared with non-RIC treatment (RR 2.34; 95% 1.19 to 4.61; 1 trial, 285 participants, low-quality evidence). |  For acute ischaemic stroke, for people receiving intravenous thrombolysis, the rate of death or dependency was significant increased by RAC treatment compared with non-RAC treatment (low-quality evidence). |
French | Flan-T5-ft (r=0.5) |  Preclinical studies have suggested that RIC may have beneficial effects in ischaemic stroke patients and those at risk of ischaemic stroke. |  Les résultats de cette revue ont suggéré que les BRA pourrait avoir des effets bénéfiques dans les patients souffrant d’une PIO et chez les personnes ayant un accident cérébral ischémique. |  The results of this review suggested that the BRA could have beneficial effects in patients suffering a PIO and in people who had an ischemic stroke.
French | mT5 (r=0.5) |  In people with acute ischaemic stroke, there was no significant difference between RIC and non-RIC for reducing stroke severity as assessed by the National Institutes of Health Stroke Scale score and the final infarct volume (standardised mean difference (SMD) -0.24 mL, 95% CI -1.02 to 0.54; 2 trials, 175 participants, very low quality evidence). |  La SEP réduisait la gravité du AVC ischémique, mais les essais confondus avaient des preuves de très faible qualité. |  The SEP reduced the severity of ischemic stroke, but the included trials have evidence of very weak quality.

##### Acronym factuality.

Flan-T5 (0-shot) and mT5 (r=0) produced factuality issues in the output
sentences. In the Spanish output example, Flan generated the acronym for AIDS
(SIDA), though the input was referring to randomized controlled trials (RCTs).
In the French example, mT5 (r=0) generated the acronym for a stroke (AVC), but
the input sentence refers to cardiovascular disease (CVD).

Table 26:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
Spanish | Flan-T5 (0-shot) |  Eight RCTs with 733 women in total that compared brief co-incubation and the standard insemination protocol were included. |  Ocho estudios compararon la co-incubación breve con un protocolo de inserción estándar en pacientes con SIDA en total. |  Eight studies compared the brief co-incubation with a protocol of standard insertion in patients with AIDS in total.
French | mT5 (r=0) |  Stress and a sedentary lifestyle are major determinants of cardiovascular disease (CVD). |  Le stress et le règlement inactif sont des causes importantes de maladies cardiovasculaires (AVC) les plus courants. |  Stress and inactive regulation are the most common important causes of cardiovascular disease (AVC).

##### Input hallucination (GPT3).

The systems GPT3-simp-trans and GPT3 (0-shot) show some minor hallucination by
referencing the input sentence in the output. Some output generations would
start with ‘this phrase’ or ‘this sentence’ in talking about the input. This
error was not found in any of the other systems.

Table 27:  Language | System |  Input |  Output |  Gloss
---|---|---|---|---
English | GPT3 (0-shot) |  There was no significant difference between RIC and non-RIC for improving the psychological impairment (SMD -0.37 points, 95% CI -1.15 to 0.41; 1 trial, 26 participants, very low quality evidence) and the cognitive impairment (SMD -0.26 points; 95% CI -0.72 to 0.21; 3 trials, 79 participants, low-quality evidence) in people with acute ischaemic stroke and cerebral small vessel disease. |  This sentence is talking about two different groups of people who had strokes: those who received a type of treatment called RIC (RIC stands for Rehabilitation Intervention in Cerebrovascular Disease) and those who did not. It is talking about how the treatment affected their psychological and cognitive health. The results showed that there was not much difference between the two groups in terms of improving the psychological and cognitive health of the people with strokes. |  -
French | GPT3 (0-shot) |  In people with carotid stenosis undergoing carotid stenting, there was no significant difference in the incidence of ischaemic stroke between participants treated with RIC and non-RIC (RR 0.22, 95% CI 0.01 to 4.03; 1 trial, 189 participants, low-quality evidence); however the stroke severity (assessed by infarct volume) was significantly lower in participants treated with RIC (mean difference (MD) -0.17 mL, 95% CI -0.23 to -0.11; 1 trial, 189 participants, low-quality evidence). |  Cette phrase parle d’une étude qui a été faite sur des personnes souffrant de sténose carotide et qui ont été traitées par stent carotidien. L’étude montre qu’il n’y a pas de différence significative entre les personnes traitées par RIC (Revascularisation Intracarotidienne) et celles qui ne l’ont pas été, au niveau du risque d’accident vasculaire cérébral. Cependant, les personnes traitées par RIC ont présenté une sévérité moindre des accidents vasculaires cérébraux (mesurée par le volume d’infarctus). |  This phrase talks about a study that was done on people suffering from carotid stenosis and who were treated by carotid stenting. The study shows that there isn’t a significant difference between people treated by RIC (Revascularization Intracarotid) and those who weren’t, at the level of risk of stroke. However, the people treated by RIC presented a lessened severity of strokes (measured by the volume of infarction).
Spanish | GPT3-simp-trans |  Three trials randomised participants to nitrous oxide pneumoperitoneum (100 participants) or carbon dioxide pneumoperitoneum (96 participants). |  Esta oración describe un experimento científico que probó dos tipos diferentes de tratamientos médicos en dos grupos de personas. El primer grupo tenía 100 personas y se les dio un tratamiento con óxido nitroso. El segundo grupo tenía 96 personas y se les dio un tratamiento con dióxido de carbono. |  This sentence describes a scientific experiment that tested two different types of medical treatment in two groups of people. The first group had 100 people and they were given a treatment with nitrous oxide. The second group had 96 people and they were given a treatment with carbon dioxide.

##  Appendix F anno-viewer

anno-viewer is inspired by a similar annotation tool that was used in Jiang et
al. (2020) for correcting crowdsourced alignment labels used for training and
evaluating a neural CRF model for aligning sentences.

The primary function of this tool is to enable annotators to make alignments
efficiently. anno-viewer also enables annotators to make factuality
annotations for alignments. These annotations are largely based from the
factuality annotations used in Devaraj et al. (2022), and also includes an
additional field to annotate for elaborations in a similar manner.

This annotation tool also has the additional functionality of allowing
annotators to used existing similarity measures to help find sentence
alignments faster. When a sentence is selected for alignment, the tool can
sort the other document by which sentences are most similar (determined by the
automatic similarity measure) to the selected sentence.

Figure 4: Screenshots of anno-viewer. The sentence sorting tool is featured in
the bottom image.