# GreekBART: The First Pretrained Greek Sequence-to-Sequence Model

Iakovos Evdaimon1 Hadi Abdine1 Christos Xypolopoulos1,2 Stamatis Outsios3
Michalis Vazirgiannis1,4 Giorgos Stamou2
1École Polytechnique, 2National Technical University of Athens,
3Athens University of Economics and Business, 4KTH Royal Institute of
Technology

###### ?abstractname?

The era of transfer learning has revolutionized the fields of Computer Vision
and Natural Language Processing, bringing powerful pretrained models with
exceptional performance across a variety of tasks. Specifically, Natural
Language Processing tasks have been dominated by transformer-based language
models. In Natural Language Inference and Natural Language Generation tasks,
the BERT model and its variants, as well as the GPT model and its successors,
demonstrated exemplary performance. However, the majority of these models are
pretrained and assessed primarily for the English language or on a
multilingual corpus. In this paper, we introduce GreekBART, the first Seq2Seq
model based on BART-base architecture and pretrained on a large-scale Greek
corpus. We evaluate and compare GreekBART against BART-random, Greek-BERT, and
XLM-R on a variety of discriminative tasks. In addition, we examine its
performance on two NLG tasks from GreekSUM, a newly introduced summarization
dataset for the Greek language. The model, the code, and the new summarization
dataset will be publicly available.

##  1 Introduction and Related Work

The field of machine learning has entered a new era with the establishment of
transfer learning, providing new possibilities, especially in the areas of
Computer Vision Krizhevsky et al. (2017) and Natural Language Processing.
Transfer learning has become a new trend that is so uncommon to train a model
for computer vision or natural language processing tasks from scratch, dealing
with the issue of insufficient training data for real-world machine learning
applications. Tasks are solved by reusing pretrained models which are trained
on enormous amounts of data, and the resulting models have reached state-of-
the-art performance. Transformer Vaswani et al. (2017) based pretrained
models, as BERT Devlin et al. (2019) and its variants, are broadly used in
Natural Language Processing, as have been shown to be effective in many tasks.

BART Lewis et al. (2020) is a denoising auto-encoder for pretraining sequence-
to-sequence models. It is trained by corrupting text with an arbitrary noising
function and learning a model to reconstruct the original text. It uses a
standard Transformer-based neural machine translation architecture and a
standard seq2seq architecture with a bidirectional encoder (like BERT) and a
left-to-right decoder (like GPT Radford et al. (2018)). This means the
encoder’s attention mask is fully visible, like BERT, and the decoder’s
attention mask is causal, like GPT2 Radford et al. (2019). The unsupervised
pretrained BART learns a language model, giving us the possibility to adapt it
to a particular NLP task. So, large-scale labeled datasets are not required
for fine-tuning. This type of model is suitable for machine translation,
question-answering, and especially, text summarization tasks, but that does
not mean that BART is insufficient in sequence classification tasks, on the
contrary, it is also quite effective in that type of tasks.

In the last few years, a lot of research has been conducted on other
languages, except for the English language. For instance, CamemBERT Martin et
al. (2020) and BARThez Kamal Eddine et al. (2021) for French language,
CAMeLBERT Inoue et al. (2021) and AraBART Eddine et al. (2022) for Arabic
language, BART for Japanese language Kim and Komachi (2021), BETO Cañete et
al. (2020) and NASes Ahuir et al. (2021) for Spanish and Catalan languages,
and BARTpho Tran et al. (2021) for Vietnamese language. Recently, a variety of
multilingual language models have been presented, covering multiple languages
by being pretrained on a large-scale corpus of different languages, trying to
learn the language model of multiple languages at once. Notably, M-BERT Devlin
et al. (2019) is a case of a multilingual pretrained language model, which
consists of the multilingual version of BERT, pretrained in the top 100
languages with the largest Wikipedias. Another case of a popular multilingual
model is the XLM Conneau and Lample (2019) which is a transformer-based
multilingual language model pretrained on Wikipedias of 15 languages. This
model was trained in two auxiliary tasks, Masked Language Modeling, and the
Translation Language Modeling task. Training a cross-lingual language model
can be very beneficial for low-resource languages, as all languages are
processed with the same shared vocabulary. Conneau et al. 2020 introduced
XLM-R, an improved version of XLM based on the RoBERTa model. The model was
trained with a cross-lingual masked language modeling objective on 2.5TB data
in 100 languages from Common Crawl Wenzek et al. (2020); Conneau et al.
(2020), increasing the amount of training available data for low-resource
languages by two orders of magnitude on average. Finally, mBART Liu et al.
(2020) is the multilingual version of BART and it is pretrained on a subset of
25 languages from the same dataset as XLM-R. In mBART, we use its 250K
sentencepiece Kudo and Richardson (2018) model which was trained using
monolingual data for 100 languages from XLM-R, supporting languages beyond the
original 25 mBART was trained on. The parameters of mBART25 are roughly 610M.
Later, an extension of mBART in additional 25 languages (_e.g._ total 50
languages) was proposed, mBART50 Tang et al. (2020), increasing the number of
parameters to approximately 680M. Except for mBART and mBART50, all other
aforementioned multilingual models support the Greek language. mBART25 and
mBART50 are not pretrained on modern Greek, but it is included in their
vocabulary. Nevertheless, multilingual models cannot compete with the
performance of monolingual models in most NLP tasks. In the last months,
another related model to BART that is in the spotlight of the NLP research
area is ChatGPT 111https://openai.com/blog/chatgpt. ChatGPT is built on top of
GPT-3 architectureBrown et al. (2020), so it is a transformer-based language
model that has been pretrained on massive amounts of text data and fine-tuned
for conversational AI applications. Like BART, ChatGPT is capable of
generating high-quality sequences of text, making it suitable for tasks such
as text summarization and question answering. However, unlike BART, ChatGPT is
specifically designed for conversational applications, making it well-suited
for chatbots and other dialogue systems. In addition, ChatGPT’s architecture
is unidirectional, which means that it can generate text in a left-to-right
sequence, making it more suitable for tasks such as language generation and
dialogue.

Compared to languages that are widely spoken, Greek has fewer linguistic
resources available. Especially, the available research in deep learning
models for Greek is still very undeveloped. However, there are some efforts to
develop datasets, models, knowledge bases, and frameworks for Greek NLP.
Outsios et al. 2018 presented the production of Greek word embeddings, where a
large corpus of about 50GB (contains 120 million sentences), crawled from
about 20 million URLs, was used for their work. Later, Lioudakis et al. 2020
presented an ensemble method, Continuous Bag-of-Skip-grams, for extracting
word representations for Greek. Recently, Koutsikakis et al. 2020 employed
Greek-BERT, the first transformer-based language model, based on BERT, for the
Greek language. The model was pretrained on a dataset of 29GB, achieving
state-of-the-art performance in several NLP tasks in Greek. It is worth noting
that Papantoniou and Tzitzikas 2020 have provided a throughout survey of the
work that has been conducted in NLP for the Greek language.

In this contribution, we try to handle the issue that the multilingual models
are not sufficient to compete with the monolingual ones and the limited
available deep learning models for the Greek language. Thus, we propose the
first pretrained Seq2Seq monolingual model for the Greek language. The model
is called GreekBART, as we pretrained the BART-base architecture on a large
monolingual Greek corpus. Despite the existence of the Greek-BERT Koutsikakis
et al. (2020), our model exceeds the possibilities of Greek-BERT, focusing on
generative tasks. GreekBART is evaluated on two different generative tasks and
on four discriminative tasks. Our main contributions are:

  * •

We introduce the pretrained Seq2Seq model for the Greek language, based on
BART-base architecture Lewis et al. (2020), and pretrained on a large corpus
of 87.6 GB. We examine the performance of our model in four discriminative
tasks (_i.e._ two classification tasks, one sentimental analysis task, and one
Natural Language Inference task) and in two generative tasks.

  * •

We present the first summarization dataset in Greek, GreekSUM, introducing two
generative tasks and a classification task by processing this dataset.

  * •

We compare GreekBART against popular language models, already pretrained or
not on Greek. In the case of the discriminative tasks we collate our model, a
BART-random model, Greek-BERT Koutsikakis et al. (2020) and XLM-R Conneau et
al. (2020). We also inspect the differences, in terms of performance, between
the GreekBART (_i.e._ our model), BART-random model, mBART25 Liu et al. (2020)
and mBART50 Tang et al. (2020) on two novel generative tasks.

  * •

We will publish our code and
models222https://github.com/iakovosevdaimon/GreekBART, providing access to
everyone, who wants to further extend the applications of our work or take
advantage of our contributions in favor of his/her work.

##  2 GreekBART

Our proposed model is based on BART Lewis et al. (2020) a denoising auto-
encoder. We use the _BASE_ architecture, with 6 encoder and 6 decoder layers.
Also, it is used 768 hidden dimensions, 12 attention heads in both the encoder
and the decoder, and a normalization layer on top of both the encoder and the
decoder Liu et al. (2020) is added. The purpose of these additional layers is
to stabilize the training when FP16 precision Micikevicius et al. (2017) is
applied. The use of FP16 precision speeds up the pretraining of the model. In
total, our model has roughly 181M parameters. Generally, we followed a similar
methodology as Kamal Eddine et al. 2021, in which a monolingual model in a
different language than English is pretrained, following BART Lewis et al.
(2020) and mBART Liu et al. (2020) methodologies.

###  2.1 Pretraining corpus

The pretrained corpus is produced by the following corpora: (a) the Greek part
of Wikipedia333https://dumps.wikimedia.org/elwiki/; (b) the Greek part of the
European Parliament Proceedings Parallel Corpus
(EuroParl)444https://www.statmt.org/europarl/ Koehn (2005); (c) the Greek part
of OSCAR555https://oscar-corpus.com/ Abadji et al. (2022), a clean version of
CommonCrawl666https://commoncrawl.org/; (d) the Greek Web Corpus, crawled from
about 20 million Greek-language URLs777http://nlp.polytechnique.fr/resources-
greek Outsios et al. (2018). In particular, we use the same datasets as the
Greek-BERT Koutsikakis et al. (2020) model, including also the dataset of
Outsios et al. 2018 in order to have a larger corpus that will be well suited
for the pretraining of BART model. Moreover, by choosing these datasets we
cover a wide variety of Greek language areas, which includes formal and
informal text, news articles, encyclopedic information, and political
conversations. This diverse range of text types helps to ensure that the
pretraining of the BART model is robust and able to handle different styles
and registers of Greek language use. Overall, the choice of datasets helps to
ensure that the Greek BART model is well-equipped to handle a wide range of
natural language processing tasks in the Greek language.

We preprocessed each of the aforementioned corpora by removing URLs, emojis,
tags, and hashtags. Also, we erase comments, and some observed noisy sentences
which do not provide any additional contextual meaning. The noisy sentences
differ from dataset to dataset, so we had to detect them ”manually”.
Furthermore, for all corpora except Wikipedia’s dataset, we got rid of
documents that contained less than one thousand characters. In the case of
Wikipedia, we removed documents with less than thirty characters. Generally,
we did not remove non-Greek characters, because we supposed that it will not
prevent the GreekBART from understanding the language model, as their amount
is insignificant. We deduplicated each corpora and then, we concatenated all
of them in one corpus. Again, we deduplicated the merged dataset for a final
time. The deduplication process was done using the runiq
package888https://github.com/whitfin/runiq. To generate our vocabulary, we
used SentencePiece999https://github.com/google/sentencepiece Kudo and
Richardson (2018) that implements byte-pair-encoding (BPE) Sennrich et al.
(2016). So, any type of pre-tokenization was not necessary. We fixed the size
of the vocabulary to 50K sub-words and the SentencePiece model was trained on
a 20GB random sample of the pretraining corpus. We set the character coverage
to $99.95\%$. The total corpus size was 76.9/87.6GB before/after SentencePiece
tokenization.

Corpus | Size before | Size after
---|---|---
deduplication | deduplication
OSCAR | $51.7$ | $44.6$
Greek Web Corpus | $38.4$ | $30.9$
Wikipedia | $0.9$ | $0.9$
EuroParl | $0.5$ | $0.5$
Total | $91.5$ | $76.9$
?tablename? 1: Datasets which consists of the GreekBART pretraining corpus
(sizes in GB, before and after cleaning and deduplication).

###  2.2 Training details

We adhere to the same pretraining process as BART. Thus, GreekBART tries to
reconstruct the corrupted input by minimizing the cross-entropy loss between
the decoder’s output and the original input. Two types of noise are applied in
the input text. First, we employ the text infilling technique, where a number
of text spans are replaced by a special token, called [MASK], masking $30\%$
of text. A Poisson distribution with $(lambda=3.5)$ is used to determine the
spans’ length. Sentence permutation is the second perturbation method, where
the sentences of the input document are shuffled randomly. We pretrained
GreekBART on Jean Zay, using a batch size equal to 768000 tokens per GPU, as
we set the update frequency to 128. We used the Adam optimizer Kingma and Ba
(2015) with $\epsilon=10^{-6},\beta_{1}=0.9$, and $\beta_{2}=0.999$, with a
learning rate starting from $6.10^{-4}$ and decreasing linearly as a function
of the training step. We used a warm-up of $6\%$ of the total number of
training steps. In the first 12 epochs, we fixed the dropout to 0.1, for
epochs 12 to 16 we decreased it to 0.05, and finally, we set it to zero for
epochs 16 to 20. All experiments were carried out using the Fairseq
library101010https://github.com/facebookresearch/fairseq Ott et al. (2019).

##  3 GreekSUM

Dataset | train/val/test | avg. doc length | avg. summary length | vocabulary size
---|---|---|---|---
words | sentences | words | sentences | docs | summaries
CNN | $90.3/1.22/1.09$ | $760.50$ | $33.98$ | $45.70$ | $3.58$ | $34$ | $89$
DailyMail | $197/12.15/10.40$ | $653.33$ | $29.33$ | $54.65$ | $3.86$ | $564$ | $180$
NY Times | $590/32.73/32.73$ | $800.04$ | $35.55$ | $45.54$ | $2.44$ | $1233$ | $293$
XSum | $204/11.33/11.33$ | $431.07$ | $19.77$ | $23.26$ | $1.00$ | $399$ | $81$
OrangeSum Title | $30.6/1.5/1.5$ | $315.31$ | $10.87$ | $11.42$ | $1.00$ | $483$ | $43$
OrangeSum Abstract | $21.4/1.5/1.5$ | $350$ | $12.06$ | $32.12$ | $1.43$ | $420$ | $71$
GreekSUM Title | $146.046/10/10$ | $355.49$ | $14.26$ | $9.95$ | $1.05$ | $663$ | $91$
GreekSUM Abstract | $129.159/10/10$ | $368.97$ | $14.76$ | $24.55$ | $1.46$ | $629$ | $127$
?tablename? 2: Sizes (column 2) are given in thousands of documents. Document
and summary lengths are in words, while vocabulary sizes are in thousands of
tokens

Dataset | % of novel n-grams in gold summary | LEAD | EXT-ORACLE
---|---|---|---
unigrams | bigrams | trigrams | 4-grams | R-1 | R-2 | R-L | R-1 | R-2 | R-L
CNN | $16.75$ | $54.33$ | $72.42$ | $80.37$ | $29.15$ | $11.13$ | $25.95$ | $50.38$ | $28.55$ | $46.58$
DailyMail | $17.03$ | $53.78$ | $72.14$ | $80.28$ | $40.68$ | $18.36$ | $37.25$ | $55.12$ | $30.55$ | $51.24$
NY Times | $22.64$ | $55.59$ | $71.93$ | $80.16$ | $31.85$ | $15.86$ | $23.75$ | $52.08$ | $31.59$ | $46.72$
XSum | $35.76$ | $83.45$ | $95.50$ | $98.49$ | $16.30$ | $1.61$ | $11.95$ | $29.79$ | $8.81$ | $22.65$
OrangeSum Title | $26.54$ | $66.70$ | $84.18$ | $91.12$ | $19.84$ | $08.11$ | $16.13$ | $31.62$ | $17.06$ | $28.26$
OrangeSum Abstract | $30.03$ | $67.15$ | $81.94$ | $88.3$ | $22.21$ | $07.00$ | $15.48$ | $38.36$ | $20.87$ | $31.08$
GreekSUM Title | $26.7$ | $67.9$ | $84.5$ | $91.4$ | $14.68$ | $04.46$ | $14.37$ | $23.36$ | $07.39$ | $23.12$
GreekSUM Abstract | $20.6$ | $50.8$ | $65.3$ | $73.0$ | $17.11$ | $06.17$ | $16.69$ | $34.18$ | $14.17$ | $33.93$

?tablename? 3: Degree of abstractivity of GreekSUM compared with that of other
datasets. It depicts that GreekSUM follows XSum, and OrangeSum, being more
abstractive than traditional summarization datasets.

Transformer-based Seq2Seq models, including BART, can perform not only
extractive but abstractive summarization, as well. This type of summarization
is one of the most central and challenging evaluation tasks in NLP. However,
there is not any available summarization dataset for the Greek language.
Therefore, we created the first dataset in the Greek language, well-suited to
the abstractive summarization task.

###  3.1 Motivation

Our main goal was to create a Greek version equivalent of the OrangeSum
dataset111111https://github.com/Tixierae/OrangeSum Kamal Eddine et al. (2021)
and XSum dataset Narayan et al. (2018). OrangeSum was produced by scraping
articles, their single-sentence title, and their brief abstract from the
”Orange Actu” website121212https://actu.orange.fr/. The title and the abstract
of each article are written by the author of the article. Well-performed
models on OrangeSum, as well as XSum, require a high degree of abstractivity.

###  3.2 Data collection

We followed a similar approach, scraping the ”News24/7”
website131313https://www.news247.gr/. News24/7 is one of the leading news
websites in Greece, part of the 24 MEDIA digital publishing
group141414https://www.24media.gr/. We collected data from web pages that span
from October 2007 to June 2022, covering five major categories: politics,
society, economy, culture, and world. Each article had a one-sentence title
and a succinct abstract, features which were extracted, yielding two
summarization tasks: GreekSUM Title and GreekSUM Abstract. The average length
of these two novel tasks’ gold summaries is 9.95 and 24.55 words respectively
(see Table 2).

###  3.3 Post-processing

Initially, we filtered the scrapped pages, removing all empty articles and
articles whose titles were shorter than 2 words or whose abstracts were less
than 5 words. Secondly, we filtered the duplicated articles (_i.e._ articles
with the same body, or with the same title, or with the same abstract), as an
article can belong to more than one category, and thus be crawled multiple
times. Finally, we noticed that several abstracts looked more like
introductions rather than actual summaries of the article. Therefore, we
eliminated 10% of the articles with the highest proportion of novel unigrams
in the abstracts. This corresponded to a threshold of 46.7% novel unigrams.
For both proposed summarization tasks, we reserved 10k pairs for testing, 10k
for validation, and all the remaining pairs for training. The released
GreekSUM dataset can be reproduced by using our
code151515https://github.com/iakovosevdaimon/GreekSUM.

###  3.4 Analysis

In Table 2 is compared the GreekSUM with OrangeSum, XSum, and the well-known
CNN, DailyMail, and NY Times datasets Hermann et al. (2015). We can observe
that GreekSUM and OrangeSum datasets are very equivalent in terms of average
documents and summaries length. Also, GreekSUM has a similar scale to XSum.
Inspecting the Table 3, it is noticeable that extractive methods (_i.e._ LEAD
and EXT-ORACLE) do not perform so well on GreekSUM, thus our dataset is less
biased towards extractive models. Because of the poor performance of the two
extractive methods, it seems that GreekSUM is more abstractive than the
traditional summarization datasets (_i.e._ CNN, DailyMail, NY Times). However,
the summaries and the titles of GreekSUM do not display such a high degree of
novelty as the ones of OrangeSum and XSum. In the GreekSUM dataset, there are
20.6% novel unigrams in the abstracts and 26.7% novel unigrams in the titles
compared with 30% in the OrangeSum Abstract, 26.5% in the OrangeSum Title, and
35.7% in XSum. Therefore, we can conclude that the summaries of GreekSUM are
not as abstractive as we would like them to be.

##  4 Experiments

In this section, we present the results of all experiments. Basically, we have
two types of downstream tasks, discriminative tasks, and summarization tasks.
In the case of discriminative tasks, we compare GreekBART with BART-random,
Greek-BERT Koutsikakis et al. (2020), and XLM-R model Conneau et al. (2020).
Except for BART-random, the other models are already pretrained on the Greek
language. So, we evaluate the performance of our model against the current
state-of-the-art monolingual model pretrained only on the Greek language as
well as against a widely used multilingual model. We fine-tuned all the above-
mentioned models on the downstream tasks.

For the summarization task, we set side by side the GreekBART, the BART-random
and the two versions of mBART Liu et al. (2020); Tang et al. (2020). mBART25
and mBART50 are built upon the _LARGE_ architecture of BART, and they are
pretrained on 25 and 50 languages respectively, excluding the Greek language.
Therefore, we performed zero-shot learning for the summarization task. On the
other hand, the BART-random model uses the same architecture and vocabulary as
GreekBART, however, it is trained from scratch on the downstream tasks.

###  4.1 Discriminative tasks

Except for generative tasks, the BART model achieves remarkable results also
in discriminative tasks Lewis et al. (2020). In the case of sequence
classification, a classification head is added on top of the model and the
input is fed into both the encoder and the decoder. The representation of the
final decoder token is used by the newly introduced multi-class linear
classifier. We examine the performance of the models (_i.e._ Greek-BERT,
XLM-R, BART-random, GreekBART) on four discriminative tasks. More precisely,
we evaluate our model on two classification tasks, one task of sentimental
analysis and a Natural Language Inference (NLI) task.

####  4.1.1 Training details

In all experiments, we fine-tuned the models with a learning rate chosen from
{$10^{-4},5.10^{-5},10^{-5}$}, based on the best validation score. We repeat
each experiment 3 times with different seeds and we record the mean and
standard deviation of their accuracy on the test set of each aforementioned
task.

####  4.1.2 NCC task (News Category Classification task)

For the first classification task, we used the novel summarization dataset
(GreekSum, see section 3) which we scraped from the news website News24/7
161616https://www.news247.gr/. We considered the five distinct subjects that
an article may fall into politics, society, economy, culture, and world. These
categories serve as labels for the classification task that our model is being
trained to perform. Essentially, the model is fed with the content of an
article and learns to predict which category it belongs to (_i.e._ subject).
We fine-tuned all examined models for 5 epochs, using a batch size equal to
32. For XLM-R model we set the learning rate equal to $5.10^{-5}$ while for
the rest of the models, the learning rate is equal to $10^{-4}$. The training
set consists of 146,046 samples, whereas both the validation and the test set
have 10,000 instances exactly like the two summarization datasets (_i.e._
GreekSUM Abstract and GreekSUM Title).

In the second classification task, we used the proposed Greek classification
dataset of Lioudakis et al. 2020, which was created from articles from
Makedonia newspaper. The dataset contains 8005 articles from 18 different
categories: Sports, Reportage, Economy, Politics, International, Television,
Arts-Culture, Letters, Opinions, Interviews, Weather, Society, Advertisements,
Biographies, Others, Articles, Police, and Zodiacs. We reserved 70% of the
dataset for train and the remaining 30% for both validation and test. So, the
train set consists of 5610 samples, whereas the test set and the validation
set consist of 1191 and 1204 instances, respectively. All the models are fine-
tuned for 20 epochs, with a batch size of 16 and a learning rate equal to
$5.10^{-5}$. Due to the small size of the dataset, we trained the models for
more epochs and smaller batch sizes.

####  4.1.3 Natural Language Inference

Cross-lingual Natural Language Inference Corpus (XNLI) Conneau et al. (2018)
contains pairs of sentences. The objective of this task is to determine
whether the first sentence, also known as the premise, entails, contradicts,
or is neutral in relation to the second sentence, referred to as the
hypothesis. The XNLI corpus contains 5,000 test and 2,500 validation pairs,
and 340k training pairs from the MultiNLI corpus Williams et al. (2018). The
dataset has been translated from English to 14 languages, including Greek.
Unfortunately, a large number of the training pairs are of extremely poor
quality, as they are produced by machine translation. This condition may
affect the performance of models. We fine-tuned for 5 epochs, using 32
batches, and a learning rate equal to $5.10^{-5}$.

####  4.1.4 Sentimental Analysis task

We used a publicly available sentimental analysis
dataset171717https://www.kaggle.com/datasets/nikosfragkis/greek-movies-dataset
about movies’ reviews in Greek. We preprocessed the dataset by mainly removing
emojis and hashtags. Each instance consists of a review and a rating. To
distinguish between positive and negative reviews, we established a threshold
of 3 out of 5. Ratings above this threshold were categorized as positive
reviews, while those at or below 3 out of 5 were classified as negative
reviews. In an effort to create a balanced dataset, we aimed to include a
similar number of positive and negative reviews. For the purpose of our task,
we only retained the reviews and the ratings, discarding any additional
information. We split the dataset into the train, validation, and test set.
The train set consists of 104,157 samples, while the validation and test
contain 22,320 and 22,318 instances respectively. We set the learning rate and
the batch size equal to $5.10^{-5}$ and 16 respectively. We fine-tuned the
models for 5 epochs.

Model | NCC | Sentimental Analysis | XNLI
---|---|---|---
News24/7 (ours) | Makedonia Lioudakis et al. (2020)
Greek-BERT | $92.61^{\pm 0.19}$ | $89.45^{\pm 0.84}$ | $\textbf{86.39}^{\pm 0.06}$ | $78.6^{\pm 0.62}$
XLM-R | $93.1^{\pm 0.51}$ | $89.6^{\pm 0.29}$ | $85.43^{\pm 0.05}$ | $78.2^{\pm 0.59}$
BART-random | $91.33^{\pm 0.17}$ | $80.17^{\pm 0.09}$ | $80.87^{\pm 0.12}$ | $60.1^{\pm 0.43}$
GreekBART (ours) | $\textbf{93.2}^{\pm 0.29}$ | $\textbf{91.1}^{\pm 0.43}$ | $85.43^{\pm 0.19}$ | $\textbf{78.67}^{\pm 0.25}$

?tablename? 4: Results on discriminative tasks. We present the mean accuracy
as well as the standard deviation.

####  4.1.5 Results

Table 4 reports the test set accuracy on the four different tasks. We compare
our model with Greek-BERT Koutsikakis et al. (2020), XLM-R Conneau et al.
(2020), and BART-random. For all models, their corresponding _BASE_
architecture is used. Among the models, we observe that GreekBART is the best
in almost all discriminative tasks, except for the sentimental analysis task,
where Greek-BERT achieved the best performance. Generally, it is common for
BERT models to perform better than BART models in that kind of tasks. The
performance of our model (_i.e._ GreekBART) verifies the results of BART paper
Lewis et al. (2020) that models based on that architecture perform well on
both generative and discriminative tasks.

|  | GreekSUM Abstract | GreekSUM Title
---|---|---|---
R-1 | R-2 | R-L | BertScore | R-1 | R-2 | R-L | BertScore
| LEAD | $17.11$ | $06.17$ | $16.69$ | $72.61/63.56$ | $14.68$ | $04.46$ | $14.37$ | $70/57.13$
| EXT-ORACLE | $34.18$ | $14.17$ | $33.93$ | $73.89/65.43$ | $23.36$ | $07.39$ | $23.12$ | $70.02/57.33$
BASE | BART-random | $13.85$ | $04.47$ | $13.65$ | $72.44/63.27$ | $11.55$ | $03.27$ | $11.42$ | $74.47/62.22$
GreekBART (ours) | 16.5 | 06.13 | 16.21 | $73.03/\textbf{64.46}$ | $15.35$ | $05.02$ | $15.18$ | $75.78/63.98$
LARGE | mBART25 | $15.07$ | $05.8$ | $14.82$ | $72.75/64.08$ | $16.09$ | $05.58$ | $15.93$ | 76.81/65.38
mBART50 | $15.53$ | $06.$ | $15.31$ | $\textbf{73.07}/64.43$ | 16.1 | 05.59 | 15.96 | 76.81/65.38

?tablename? 5: Results on GreekSUM. Except for ROUGE, we provide also the
BertScore. The left-hand BERTScore has calculated using the M-BERT model
Devlin et al. (2019), while the right-hand uses the Greek-BERT Koutsikakis et
al. (2020)

|  | GreekSUM Abstract | GreekSUM Title
---|---|---|---
unigrams | bigrams | trigrams | 4-grams | length | unigrams | bigrams | trigrams | 4-grams | length
| Gold | $20.6$ | $50.8$ | $65.3$ | $73.0$ | $24.55$ | $26.7$ | $67.9$ | $84.5$ | $91.4$ | $9.95$
BASE | BART-random | $9.6$ | $43.0$ | $64.5$ | $76.8$ | $20.27$ | $21.6$ | $69.4$ | $89.1$ | $95.8$ | $9.37$
GreekBART (ours) | 7.4 | 23.5 | 34.5 | 42.2 | $23.63$ | 14.9 | 50.1 | 69.3 | 79.9 | 9.78
LARGE | mBART25 | $6.2$ | $20.0$ | $29.4$ | $36.0$ | $26.22$ | $12.8$ | $46.6$ | $65.6$ | $76.2$ | $10.67$
mBART50 | $6.5$ | $21.8$ | $32.3$ | $39.7$ | $23.95$ | $12.8$ | $46.6$ | $65.6$ | $76.2$ | $10.67$

?tablename? 6: Proportion of novel n-grams in the generated summaries. Also,
it is given the length (number of words) of the generated summaries

|  | Repetitions (%)
---|---|---
Abstract | Gold | $7.77$
BART-random | $28.12$
GreekBART (ours) | $12.19$
mBART25 | $12.7$
mBART50 | $10.03$

|  | Repetitions (%)
---|---|---
Title | Gold | $0.91$
BART-random | $8.76$
GreekBART (ours) | $3.62$
mBART25 | $2.52$
mBART50 | $2.52$

?tablename? 7: The percentage of repeated words on the summaries

###  4.2 Summarization

We evaluate our model in two distinct summarization tasks, in which the model
learns to predict the title and the abstract of an article based on its
corresponding content. In both generative tasks, the GreekBART was fine-tuned
for 30 epochs with a learning rate equal to $5.10^{-5}$ that was warmed up for
6% of the training steps and then decreased linearly to 0. We used the same
set of hyper-parameters as those of GreekBART to train mBART25 and mBART50.
While for BART-random, we trained the model for 60 epochs. To produce the
summaries for the test set, we used ROUGE-L Lin (2004) to select the
checkpoint that was associated with the best validation score. In addition, we
incorporated two extractive techniques as baselines: EXT-ORACLE and LEAD
Narayan et al. (2018). The LEAD technique generates a summary by extracting
the first $N$ sentences from the document, with $N$ set to 1 in our case. On
the other hand, EXT-ORACLE selects the set of sentences from the document that
maximizes a specific score, with ROUGE-L being the score used in our
implementation. In particular, we extracted the one sentence of the document
with the highest ROUGE-L score. In Table 5, we report the ROUGE-1, ROUGE-2,
ROUGE-L scores Lin (2004) and two different BERTScores Zhang et al. (2019),
using the M-BERT Devlin et al. (2019) model and the Greek-BERT model in order
to calculate the contextual embeddings. BERTScore is a recently proposed
metric that makes use of the contextual representations of the predicted and
gold sentences. BERTScore focuses on semantic similarity between tokens of
reference and hypothesis, trying to understand the meaning of what you have
generated and what was supposed to be generated. We report BERTScore because
ROUGE can mainly capture n-gram overlap, which is inadequate for the
abstractive summarization setting. Some examples of the generated
summarizations are available in the appendix section A, B.

####  4.2.1 Quantitative results

In Table 5 we compare the performance of our models fine-tuned on the
summarization task. Despite that GreekBART is a BART-_BASE_ model and it is
compared with BART-_LARGE_ models, it is able to achieve better performance
than all other models in the task of GreekSUM abstract. Only mBART50 achieves
a slightly higher BERTScore than GreekBART when evaluated using the M-BERT
model. On the other hand, both mBART models surpass our model in the GreekSUM
title task. Although, even in that task the performance of GreekBART is
comparable to one of the two mBART models, both in terms of ROUGE and
BERTScore. Our evaluation indicates that mBART50 and GreekBART are the most
promising models for the two summarization tasks. Specifically, mBART50
performs better overall in both generative tasks, being the top-performing
model in the GreekSUM title task and second-best in the GreekSUM Abstract
task, according to its ROUGE and BERTScores. On the other hand, GreekBART
excels in the GreekSUM abstract task, but ranks third-best in the GreekSUM
title task. Generally, it is remarkable the fact that both mBART models, which
are not pretrained on the Greek language, are capable to achieve a good
performance due to the size of GreekSUM dataset, which contains more than 100k
training samples. It is clear that BART-random has the poorest performance by
a significant margin. Finally, it is interesting that mBART50 has a better
performance than mBART25 in terms of both ROUGE and BERTScore, while their
only difference is the number of languages on which they are pretrained. This
situation warrants further investigation, as it is possible that some of the
additional 25 languages supported by mBART50 have roots in the Greek language,
potentially contributing to a better understanding of the language model.

####  4.2.2 Qualitative results

As shown in Table 6, GreekBART is more abstractive than the two mBART models,
as its generated summaries display a higher degree of novel n-grams. In
general, none of the models surpass the LEAD method in terms of ROUGE scores.
Furthermore, the ROUGE scores of the models suggest that the machine-generated
summaries tend to be extractive, as the gold summaries are also predominantly
extractive in nature. This situation is confirmed by the proportion of novel
n-grams that are introduced (Table 6), where few new words are introduced in
the gold summaries of GreekSUM, influencing, therefore, the training of the
examined models, forcing them to generate more extractive summaries. Moreover,
Table 6 depicts that the length of all generated summaries is pretty close to
the length of ground truth summaries. According to Table 7 the generated
summaries of mBART50 contain the smallest percentage of repetitions, with
GreekBART following. The rate of repeated words on mBART50 summaries is close
to the one of ground truth summaries. Finally, we notice that BART-random
introduces many new words, however, they are irrelevant.

| System | Score
---|---|---
| Gold | $45.24$
BASE | BART-random | $-72.62$
GreekBART (ours) | $10.71$
LARGE | mBART25 | $-03.57$
mBART50 | 20.24
?tablename? 8: The results of human evaluation study

####  4.2.3 Human Evaluation

In order to further understand and validate the quantitative results, we
conducted a human evaluation study, using Best-Worst Scaling Louviere et al.
(2015). We chose 11 native Greek speakers from diverse age groups, ranging
from 18 to 60 years old, with varying educational backgrounds and levels.
Following Narayan et al. 2018 method, we randomly selected 14 documents from
the test set of GreekSUM abstract and for each document we generated all
possible pairs of human-authored (Gold), GreekBART, BART-random, mBART25, and
mBART50 summaries, resulting in a total of 140 pairs for all documents. Thus,
each pair of summaries consists of two summaries generated by two different
models. Volunteers were presented with a document and a pair of summaries and
they should decide which one is the best summary and which was the worst,
based on the accuracy (does the summary contain accurate facts?), the
informativeness (is important information captured?) and the fluency (is the
summary written in well-formed Greek?). Each summary pair was assigned
randomly to three participants, and a system’s score was determined by
calculating the percentage of times it was selected as the best summary, minus
the percentage of times it was selected as the worst summary. Thus, the
maximum score that a model can achieve is $100$, whereas the minimum score can
be $-100$. The results of the human evaluation study are presented in Table 8.
Gold reaches first place, followed by mBART50 and GreekBART. According to the
evaluators, Gold is by far the most preferred summary, while the score of
mBART50 is remarkably higher than that of GreekBART, verifying our assumptions
based on the quantitative results. Finally, the high negative score of BART-
random indicates that its summaries were considered to be worse in the
majority of cases.

##  5 Conclusion

We implemented GreekBART, the first pretrained Seq2Seq model for the Greek
language specifically. Also, we created the first summarization dataset for
the Greek language. Our model showed to outperform former state-of-the-art
models on 3 out of 4 discriminative tasks and to be on par with BART-_LARGE_
models on summarization tasks. Moreover, we presented the capabilities of
zero-shot learning, training from scratch a multilingual BART model on
summarization tasks, even though it was not pretrained on the Greek language.
As a future work, we can consider the creation of a more abstractive
summarization dataset, and the investigation of any correlation between the
Greek language and one or more of the 25 extra languages of mBART50. Finally,
it would be interesting to try to boost the performance of mBART50 on
summarization tasks by applying an affordable language-adaptive phase in order
to further pretrain it on the Greek language for a logical number of epochs.

## Ethics Statement

The collection of the GreekSUM dataset was performed using a Python crawler
that respected the robots.txt of http://www.news247.gr. As the dataset is used
only for evaluation purposes the content follows the legal instructions listed
on the webpage.

For the training of GreekBART we used a cluster of GPUs consisting of 2 NVIDIA
V100 GPUs for 20 days. As the majority of language models that are based on
BART architecture the energy resources required for pretraining models
currently are very high and need to be tackled soon Strubell et al. (2019).

## Limitations

The proposed GreekSUM dataset that we used for the evaluation of our model is
limited to news articles from one webpage only. Thus, the capability of
abstractive summarization of GreekBART is only assessed on one domain only.
This is due to the fact that there is a lack of non-English benchmarks and
tasks. This is also applicable in the discriminative tasks, where the only
available ones for Greek are either sentence classification or natural
language inference. While other evaluation datasets are not existing for the
Greek language (i.e. Word Sense Disambiguation) or are not available to the
public (i.e. Named Entity Recognition dataset).

On the other hand, GreekBART is only compared with extractive summarization
methods or with large multi-lingual language models for the summarization
task. Since it is the first base model for this language and since the base
mBART model does not exist publicly, a fair in-depth comparison of GreekBART
with other summarization systems could not be conducted.

## Acknowledgements

This research was supported by the ANR chair AML/HELAS (ANR-CHIA-0020-01).

This work was granted access to the HPC resources of IDRIS under the
allocation 2022-AD011013750 made by GENCI.

We would like to express our sincere gratitude to all the participants who
took part in this human evaluation study. Your time and effort in completing
the questionnaires and participating in the study have been invaluable in
helping us gather meaningful data.

Your willingness to share your experiences, insights, and opinions has been
instrumental in informing our research, and we appreciate the trust you have
placed in us. Your contributions have helped us improve our understanding of
the topic under investigation and have the potential to make a significant
impact on future research and practice.

We would also like to acknowledge the importance of obtaining informed consent
from all participants before their involvement in the study. Your
participation was entirely voluntary, and we appreciate your willingness to
take part in the study.

Once again, we extend our sincere thanks to all the participants for their
valuable contributions to this study.

##  ?refname?

  * Abadji et al. (2022) Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022.  Towards a cleaner document-oriented multilingual crawled corpus.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4344–4355, Marseille, France. European Language Resources Association.
  * Ahuir et al. (2021) Vicent Ahuir, Lluís-F. Hurtado, José Ángel González, and Encarna Segarra. 2021\.  Nasca and nases: Two monolingual pre-trained models for abstractive summarization in catalan and spanish.  _Applied Sciences_ , 11(21).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cañete et al. (2020) José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. 2020.  Spanish pre-trained bert model and evaluation data.  In _PML4DC at ICLR 2020_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  _Advances in neural information processing systems_ , 32.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Eddine et al. (2022) Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022.  Arabart: a pretrained arabic sequence-to-sequence model for abstractive summarization.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Inoue et al. (2021) Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021\.  The interplay of variant, size, and task type in Arabic pre-trained language models.  In _Proceedings of the Sixth Arabic Natural Language Processing Workshop_ , pages 92–104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  * Kamal Eddine et al. (2021) Moussa Kamal Eddine, Antoine Tixier, and Michalis Vazirgiannis. 2021.  BARThez: a skilled pretrained French sequence-to-sequence model.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9369–9390, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kim and Komachi (2021) Hwichan Kim and Mamoru Komachi. 2021.  TMU NMT system with Japanese BART for the patent task of WAT 2021.  In _Proceedings of the 8th Workshop on Asian Translation (WAT2021)_ , pages 133–137, Online. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _Proceedings of Machine Translation Summit X: Papers_ , pages 79–86, Phuket, Thailand.
  * Koutsikakis et al. (2020) John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2020.  Greek-bert: The greeks visiting sesame street.  In _11th Hellenic Conference on Artificial Intelligence_ , SETN 2020, page 110–117, New York, NY, USA. Association for Computing Machinery.
  * Krizhevsky et al. (2017) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017.  Imagenet classification with deep convolutional neural networks.  _Commun. ACM_ , 60(6):84–90.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lioudakis et al. (2020) Michalis Lioudakis, Stamatis Outsios, and Michalis Vazirgiannis. 2020.  An ensemble method for producing word representations focusing on the Greek language.  In _Proceedings of the 3rd Workshop on Technologies for MT of Low Resource Languages_ , pages 99–107, Suzhou, China. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Louviere et al. (2015) Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015.  _Best-Worst Scaling: Theory, Methods and Applications_.  Cambridge University Press.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suá rez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty french language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Micikevicius et al. (2017) Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2017.  Mixed precision training.  _CoRR_ , abs/1710.03740.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Outsios et al. (2018) Stamatis Outsios, Konstantinos Skianis, Polykarpos Meladianos, Christos Xypolopoulos, and Michalis Vazirgiannis. 2018.  Word embeddings from large-scale greek web content.  _arXiv preprint arXiv:1810.06694_.
  * Papantoniou and Tzitzikas (2020) Katerina Papantoniou and Yannis Tzitzikas. 2020.  Nlp for the greek language: A brief survey.  In _11th Hellenic Conference on Artificial Intelligence_ , SETN 2020, page 101–109, New York, NY, USA. Association for Computing Machinery.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Tang et al. (2020) Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020.  Multilingual translation with extensible multilingual pretraining and finetuning.
  * Tran et al. (2021) Nguyen Luong Tran, Duong Minh Le, and Dat Quoc Nguyen. 2021.  Bartpho: Pre-trained sequence-to-sequence models for vietnamese.  _CoRR_ , abs/2109.09701.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4003–4012, Marseille, France. European Language Resources Association.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with BERT.  _CoRR_ , abs/1904.09675.

## Appendices

##  ?appendixname? A Appendix-GreekSUM Abstract

In this appendix section, we present the reference and model summaries of 5
randomly selected documents from the test set of the GreekSUM Abstract.

| Document |  «Ο ϰυβος ερριφϑη. Ο ´Αρµιν Λαςετ ϑα ειναι ο υποψηφιος Καγϰελαριος για την Χριςτιανιϰη Ενωςη», δηλωςε πριν απο λιγο ο Αρχηγος της Χριςτιανοϰοινωνιϰης Ενωςης (CSU) ϰαι Πρωϑυπουργος της Βαυαριας Μαρϰους Ζεντερ, αναγνωριζοντας το αποτελεςµα της ψηφοφοριας του προεδρειου του Χριςτιανοδηµοϰρατιϰου Κοµµατος (CDU), το οποιο ςε ποςοςτο 77,5% ταχϑηϰε υπερ της υποψηφιοτητας του ϰ. Λαςετ. Πριν απο την ςυνεδριαςη του προεδρειου του CDU, ο ϰ. Ζεντερ ειχε δηλωςει οτι παραχωρει ςτο  CDU το προβαδιςµα ςτην επιλογη του υποψηφιου Καγϰελαριου της Χριςτιανιϰης Ενωςης (CDU/CSU) ϰαι ςηµερα επανελαβε οτι δεχεται το αποτελεςµα «χωρις µηνιςιϰαϰια» ϰαι οτι ταςςεται υπερ της ενοτητας της Χριςτιανιϰης Ενωςης.
---|---|---
Abstract | Gold |  Ο Αρµιν Λαςετ ϑα ειναι ο υποψηφιος των CDU ϰαι CSU για την ϰαγϰελαρια της Γερµανιας ςτις εϰλογες του Σεπτεµβριου.
BART-random |  Ο Αρµιν Λαςετ ϑα ειναι ο υποψηφιος πρωϑυπουργος της Χριςτιανιϰης Ενωςης, µετα απο ςχετιϰη ςυνεδριαςη.
mBART25 |  Ο πρωϑυπουργος της Βαυαριας δεχϑηϰε το αποτελεςµα της ψηφοφοριας του προεδρειου του CDU, το οποιο ςε ποςοςτο 77,5% ταχϑηϰε υπερ της υποψηφιοτητας του Αρµιν Λαςετ.
mBART50 |  Σε ποςοςτο 77,5% ταχϑηϰε υπερ της υποψηφιοτητας του Αρµιν Λαςετ ςτο προεδρειο του CDU, ο Πρωϑυπουργος της Βαυαριας Μαρϰους Ζεντερ.
GreekBART |  ϒπερ του Αρµιν Λαςετ ταςςεται ο Μαρϰους Ζεντερ, αναγνωριζοντας το αποτελεςµα της ψηφοφοριας του προεδρειου του CDU.
?tablename? 9: Example 1-GreekSUM Abstract

| Document |  Κλειςτη ειναι η λεωφορος Βας. Κωνςταντινου ςτο υψος του Παναϑηναιϰου Σταδιου, µε αποτελεςµα να εχει δηµιουργηϑει ϰυϰλοφοριαϰο προβληµα, ϰαϑως εχει χυϑει µεγαλη ποςοτητα λαδιου απο φορτηγο, ςτην ςυµβολη µε την λεωφορο Βας. Ολγας ϰαι ειναι επιϰινδυνη η διελευςη των οχηµατων. Η Τροχαια εχει διαϰοψει την ϰυϰλοφορια ςτο ϰατερχοµενο ρευµα ςτο υψος της οδου Ριζαρη ϰαι ςτο ανερχοµενο απο την αρχη της Αρδητου ϰαι ϰανει εϰτροπη, αλλα εχει δηµιουργηϑει µποτιλιαριςµα. Στο ςηµειο που εχουν χυϑει τα λαδια βριςϰονται ςυνεργεια του Δηµου, που ριχνουν πριονιδι ϰαι αλλα υλιϰα για να αντιµετωπιςουν την ολοςϑηροτητα του οδοςτρωµατος ϰαι να αποϰαταςταϑει η ϰυϰλοφορια.
---|---|---
Abstract | Gold |  Η Τροχαια εχει διαϰοψει την ϰυϰλοφορια ςτο ϰατερχοµενο ρευµα ςτο υψος της οδου Ριζαρη ϰαι ςτο ανερχοµενο απο την αρχη της Αρδητου ϰαι ϰανει εϰτροπη - Μποτιλιαριςµα ςτο ςηµειο.
BART-random |  Η Τροχαια, που εχει δηµιουργηϑει ςτο υψος του Παναϑηναιϰου, εχει διαϰοπει την ϰυϰλοφορια των οχηµατων ςτην λεωφορο Βας.Α.
mBART25 |  Κλειςτη ειναι η λεωφορος Βας. Κωνςταντινου ςτο υψος του Παναϑηναιϰου Σταδιου, µε αποτελεςµα να εχει δηµιουργηϑει ϰυϰλοφοριαϰο προβληµα.
mBART50 |  Κυϰλοφοριαϰο προβληµα ςτο υψος του Παναϑηναιϰου Σταδιου, ϰαϑως εχει χυϑει µεγαλη ποςοτητα λαδιου απο φορτηγο ςε λεωφορο Βας. Κωνςταντινου.
GreekBART |  Κυϰλοφοριαϰο προβληµα εχει δηµιουργηϑει ςτην Λεωφορο Βας. Κωνςταντινου ςτο υψος του Παναϑηναιϰου Σταδιου, µε αποτελεςµα να εχει δηµιουργηϑει µποτιλιαριςµα.
?tablename? 10: Example 2-GreekSUM Abstract

| Document |  Η Καγϰελαριος Ανγϰελα Μερϰελ δεν ϑα παραςτει ςτην επιςηµη δεξιωςη που ϑα παραϑεςει την Παραςϰευη ο Οµοςπονδιαϰος Προεδρος Φρανϰ-Βαλτερ Σταινµαιερ προς τιµην του Προεδρου της Τουρϰιας Ρετζεπ Ταγιπ Ερντογαν, ςυµφωνα µε ϰυβερνητιϰες πηγες τις οποιες επιϰαλειται το περιοδιϰο «Der Spiegel». Η δεξιωςη αλλα ϰαι οι ςτρατιωτιϰες τιµες µε τις οποιες ϑα υποδεχϑει τον προςϰεϰληµενο του ο Γερµανος Προεδρος προϰαλουν ςοβαρες αντιδραςεις ςτον πολιτιϰο ϰοςµο της χωρας. Η Μερϰελ ειναι παντα προςϰεϰληµενη του Οµοςπονδιαϰου Προεδρου ςε δεξιωςεις η επιςηµα δειπνα που παρατιϑενται προς τιµην υψηλων προςϰεϰληµενων. Η ιδια ωςτοςο ςυνηϑιζει να παρευριςϰεται µονο ςε εξαιρετιϰες περιπτωςεις. Η τελευταια φορα που παρεςτη ςε ϰατι αναλογο ηταν το επιςηµο δειπνο που ειχε παρατεϑει το 2015 προς τιµην της Βαςιλιςςας Ελιςαβετ, ενω την προηγουµενη χρονια ειχε παρευρεϑει ςτο δειπνο µε τον Εµιρη του Καταρ. Αντιϑετως, δεν ειχε παρευρεϑει ςτην δεξιωςη προς τιµη του Κινεζου Προεδρου Σι Τζινπινγϰ το 2017. Η Καγϰελαριος οµως δεν ϑα ειναι η µονη που ϑα απορριψει την προςϰληςη του Σταινµαιερ. Ο Προεδρος των Φιλελευϑερων (FDP) Κριςτιαν Λιντντερ αναϰοινωςε οτι δεν ςϰοπευει να παραςτει, ϰαϑως δεν επιϑυµει «να ςυµµεταςχει ςτην προπαγανδα του Ερντογαν». Την ιδια ςταςη ϑα τηρηςει ϰαι η εϰπροςωπος του ϰοµµατος για την εξωτεριϰη πολιτιϰη, Μπιτζαν Ντζιρ-Σαραι, ενω ςυςςωµη η ηγετιϰη οµαδα των Πραςινων, οι ςυµπροεδροι Αναλενα Μπερµποϰ ϰαι Ροµπερτ Χαµπεϰ ϰαι οι επιϰεφαλης της Κοινοβουλευτιϰης Οµαδας Κατριν Γϰερινγϰ-Εϰαρτ ϰαι ´Αντον Χοφραιτερ, δηλωςαν οτι ϑα απεχουν απο την δεξιωςη. Το ιδιο ιςχυει ϰαι για τους επιϰεφαλης της Εναλλαϰτιϰης για την Γερµανια (AfD) ´Αλεξαντερ Γϰαουλαντ ϰαι Αλις Βαιντελ ϰαι για την επιϰεφαλης της Κ. Ο. της Αριςτερας Σεβιµ Νταγϰντελεν. Αντιϑετως, την προϑεςη του να παραςτει ςτην δεξιωςη ςτο Προεδριϰο Αναϰτορο Bellevue εξεφραςε ο πρωην Προεδρος των Πραςινων Τζεµ Εζντεµιρ, διευϰρινιζοντας ταυτοχρονα οτι ο Τουρϰος Προεδρος «δεν ειναι ϰανονιϰος Προεδρος ϰαι δεν αξιζει» να παρατεϑει δεξιωςη προς τιµην του. Με την παρουςια του, δηλωςε ο ϰ. Εζντεµιρ ςτην «Tagesspiegel», ελπιζει να ςτειλει ενα µηνυµα τοςο προς την Τουρϰια οςο ϰαι προς την τουρϰογερµανιϰη ϰοινοτητα: «Η αντιπολιτευςη ςτην Γερµανια ειναι µερος της πολιτιϰης αυτης της χωρας, ειµαςτε ενα ςταϑερο ϰαι απαραιτητο ςυςτατιϰο ςτοιχειο της δηµοϰρατιας µας. Ο ϰ. Ερντογαν ϑα πρεπει να µε ανεχϑει».
---|---|---
Abstract | Gold |  Η ϰαγϰελαριος ειναι παντα προςϰεϰληµενη του οµοςπονδιαϰου προεδρου ςε δεξιωςεις η δειπνα προς τιµην υψηλων προςϰεϰληµενων, ωςτοςο δινει το παρων µονο ςε εξαιρετιϰες περιπτωςεις.
BART-random |  Δεν ϑα παραςτει ςτην επιςηµη δεξιωςη που ϑα παραϑεςει την Τουρϰια προς τιµην του Ρετζεπ Ταγιπ Ερντογαν ο εϰπροςωπος της Γερµανιας Ανγϰελα Μερϰελ.
mBART25 |  Αντιδραςεις απο τον πολιτιϰο ϰοςµο της χωρας προϰαλουν η δεξιωςη που ϑα παραϑεςει ο Φρανϰ-Βαλτερ Σταινµαιερ προς τιµην του Προεδρου της Τουρϰιας - Δεν ϑα ειναι η µονη που ϑα απορριψει την προςϰληςη του Σταινµαιερ.
mBART50 |  Η Μερϰελ ειναι παντα προςϰεϰληµενη του Οµοςπονδιαϰου Προεδρου ςε δεξιωςεις η επιςηµα δειπνα που παρατιϑενται προς τιµην υψηλων προςϰεϰληµενων. Η ιδια ωςτοςο ςυνηϑιζει να παρευριςϰεται µονο ςε εξαιρετιϰες περιπτωςεις.
GreekBART |  Απο τον πολιτιϰο ϰοςµο της Γερµανιας. Η Ανγϰελα Μερϰελ δεν ϑα παραςτει ςτην επιςηµη δεξιωςη προς τιµην του Γερµανου Προεδρου Φρανϰ-Βαλτερ Σταινµαιερ.
?tablename? 11: Example 3-GreekSUM Abstract

| Document |  Απο το 2011 ϰαι µετα αρϰετοι εϰατονταδες ανϑρωποι εχουν πεϑανει ςτην προςπαϑεια τους να βγαλουν την τελεια selfie. Οι περιςςοτεροι ϑανατοι εχουν λαβει χωρα ςτην Ινδια. Αϰολουϑει η Ρωςια, οι Ηνωµενες Πολιτειες ϰαι υςτερα το Παϰιςταν µε τους νεϰρους ςυνολιϰα να φτανουν τους 259. Βεβαια υπαρχουν ϰαποια ςηµεια, τα οποια ςυµφωνα µε ερευνες, παρουςιαζουν µεγαλυτερη επιϰινδυνοτητα, οπως το νερο ϰαι οι ψηλες ϰυλιοµενες ςϰαλες. Οι πιο «ςυνηϑιςµενες» αιτιες ϑανατου απο selfie ςυµπεριλαµβανουν τον πνιγµο, την πτωςη, τη ςυγϰρουςη µε ϰινουµενο οχηµα ϰαι τις φωτιες. Οςον αφορα τα ςτατιςτιϰα ςτοιχεια τα 3/4 των ϑυµατων ειναι ανδρες ϰαι ϰατω απο την ηλιϰια των 30. Αν ϰαι οι γυναιϰες βγαζουν περιςςοτερες selfie ςυµφωνα µε τις µελετες, οι ανδρες ειναι πιο επιρρεπεις ςτον ϰινδυνο. Αϰοµα, οι τουριςτες ειναι αυτοι που πληττονται πιο ςυχνα ςτην προςπαϑεια να βγαλουν µια φωτογραφια που ϑα εντυπωςιαςει τους αϰολουϑους τους. Οι αρχες ψαχνουν τροπους προϰειµενου να αποτρεψουν τους ϑανατους. Για παραδειγµα η ρωςιϰη αςτυνοµια µοιραςε φυλλαδια, τα οποια εµπεριειχαν προειδοποιηςεις ςχετιϰα µε τους ϰινδυνους που «ϰαραδοϰουν» πιςω απο µια selfie. Στις Ηνωµενες Πολιτειες, τα εϑνιϰα παρϰα εχουν εϰδωςει οδηγους για το πως να βγαζεις «αςφαλεις» selfies, ενω ςτην Ινδια υπαρχουν επιςηµα ςχεδιαςµενες πιναϰιδες που προειδοποιουν για υψηλου ϰινδυνου περιοχες η αλλιως “No selfie zones”.Αν ϰαι η εµµονη µε τις selfie δεν φαινεται να περναει οι αρχες ϰανουν οτι µπορουν για να περιοριςουν την επιϰινδυνοτητα ϰαι τους ϑανατους.
---|---|---
Abstract | Gold |  Οι πιο «ςυνηϑιςµενες» αιτιες ϑανατου απο ςελφιε ςυµπεριλαµβανουν πνιγµο, πτωςη, ϰαι τη ςυγϰρουςη µε ϰινουµενο οχηµα - Ανδρες ϰατω των 30 τα περιςςοτερα ϑυµατα.
BART-random |  Οι Ηνωµενες Πολιτειες, Ινδια, Αν. ϰαι Παϰιςταν ϰαι Αν. Ινδια αναζητουν αναζητουν ςτοιχεια για να βγαλουν την τελεια selfie τους ςτην προςπαϑεια τους.
mBART25 |  Η Ινδια µετραει τους 259 ϑανατους απο selfie,τα οποια ςυµπεριλαµβανουν τον πνιγµο, την πτωςη, τη ςυγϰρουςη µε ϰινουµενο οχηµα ϰαι τις φωτιες. Οι αρχες ψαχνουν τροπους προϰειµενου να αποτρεψουν τους ϑανατους.
mBART50 |  Στην Ινδια, τα εϑνιϰα παρϰα εχουν εϰδωςει οδηγους για το πως να βγαζεις «αςφαλεις» selfies, ενω ςτην Ινδια υπαρχουν επιςηµα ςχεδιαςµενες πιναϰιδες που προειδοποιουν για υψηλου ϰινδυνου περιοχες.
GreekBART |  Πολλοι ανϑρωποι εχουν πεϑανει ςτην προςπαϑεια τους να βγαλουν µια selfie, µε τις «ςυνηϑιςµενες» αιτιες να ςυµπεριλαµβανουν τον πνιγµο, την πτωςη, τη ςυγϰρουςη µε ϰινουµενο οχηµα ϰαι τις φωτιες.
?tablename? 12: Example 4-GreekSUM Abstract

| Document |  Στην απωλεια του Μιϰη Θεοδωραϰη αναφερϑηϰε ο πρωϑυπουργος Κυριαϰος Μητςοταϰης ςτην εναρξη της ςυνεδριαςης του ϒπουργιϰου Συµβουλιου, ϰηρυςςοντας τριηµερο εϑνιϰο πενϑος. Ο πρωϑυπουργος ειδιϰοτερα δηλωςε: “Τη ςηµερινη µας ςυνεδριαςη ςϰιαζει δυςτυχως µια πολυ ϑλιβερη ειδηςη: Ο Μιϰης Θεοδωραϰης περνα πια ςτην αιωνιοτητα. Η φωνη του ςιγηςε ϰαι µαζι του ςιγηςε ϰαι ολοϰληρος ο Ελληνιςµος. Οπως ειχε γραφτει ϰαι για τον Παλαµα, «ολοι ειχαµε ξεχαςει πως ειναι ϑνητος». Οµως, µας αφηνει παραϰαταϑηϰη τα τραγουδια του, την πολιτιϰη του δραςη, αλλα ϰαι την εϑνιϰη του προςφορα ςε ϰριςιµες ςτιγµες. Η Ρωµιοςυνη ςηµερα ϰλαιει. Και γι’ αυτο ϰαι µε αποφαςη της ϰυβερνηςης απο ςηµερα ϰηρυςςεται τριηµερο εϑνιϰο πενϑος. Οπως ξερετε, ειχα την τιµη να τον γνωριζω για πολλα χρονια ϰαι ςχετιϰα προςφατα µαλιςτα τον ειχα επιςϰεφτει. Οι ςυµβουλες του ηταν παντα πολυτιµες για µενα, ϰυριως αυτες που αφορουςαν ςτην ενοτητα του λαου µας ϰαι ςτην υπερβαςη των διαχωριςτιϰων γραµµων. Πιςτευω πως η ϰαλυτερη τιµη προς αυτον τον παγϰοςµιο Ελληνα ϑα ειναι εµεις, µε το ϰαϑηµερινο µας εργο, να ϰανουµε πραξη αυτο αϰριβως το µηνυµα του. Ο Μιϰης ειναι η Ιςτορια µας ϰαι πρεπει να τη ςυνεχιςουµε οπως ϑα ηϑελε ϰαι εϰεινος.” Πεϑανε ο Μιϰης Θεοδωραϰης - Ορφανεψε η Ρωµιοςυνη Ελενα Αϰριτα - Ο ‘Ηλιος (που ϰρυφτηϰε) ϰαι ο Χρονος (που χαϑηϰε), Μιϰη Μιϰης Θεοδωραϰης: Τα 5 τραγουδια του ςπουδαιου µουςιϰου που «µιλουν» ςτην ψυχη της Ελλαδας Ο πολιτιϰος Μιϰης Θεοδωραϰης: Παντα ςτο πλευρο των απλων ανϑρωπων.
---|---|---
Abstract | Gold |  Η Ρωµιοςυνη ςηµερα ϰλαιει δηλωςε ο πρωϑυπουργος ςτην εναρξη της ςυνεδριαςης του υπουργιϰου ςυµβουλιου αναφεροµενος ςτο ϑανατο του Μιϰη Θεοδωραϰη.
BART-random |  Ο πρωϑυπουργος ϰατα την εναρξη της ςυνεδριαςης του ϒπουργιϰου Συµβουλιου ϰηρυςςοντας την απωλεια του Μιϰη Θεοδωραϰη.
mBART25 |  Ο πρωϑυπουργος Κυριαϰος Μητςοταϰης απο το υπουργιϰο ςυµβουλιο για τον ϑανατο του Μιϰη Θεοδωραϰη.
mBART50 |  Τριηµερο εϑνιϰο πενϑος ϰηρυχϑηϰε ςτη ςυνεδριαςη του υπουργιϰου ςυµβουλιου, µε τον πρωϑυπουργο να ςηµειωνει οτι ο Μιϰης Θεοδωραϰης περνα πια ςτην αιωνιοτητα.
GreekBART |  Το διϰο του µηνυµα για την απωλεια του Μιϰη Θεοδωραϰη εςτειλε ο πρωϑυπουργος Κυριαϰος Μητςοταϰης ϰατα τη ςυνεδριαςη του ϒπουργιϰου Συµβουλιου.
?tablename? 13: Example 5-GreekSUM Abstract

##  ?appendixname? B Appendix- GreekSUM Title

In the second section of the appendices, we present the reference and model
titles of 5 randomly selected documents from the test set of the GreekSUM
Title.

| Document |  Ενας 33χρονος εχαςε τη ζωη του, υςτερα απο ςυγϰρουςη δυο αυτοϰινητων, εξω απο τη Θεςςαλονιϰη. Οπως εγινε γνωςτο, το ϑανατηφορο τροχαιο ςυνεβη ςτις 2.15 µετα τα µεςανυχτα ςε παραδροµο της Εγνατιας Οδου, ςτο υψος του Ωραιοϰαςτρου. Συµφωνα µε την Αςτυνοµια, ο 33χρονος, οδηγος του ενος οχηµατος, διαϰοµιςτηϰε ςτο νοςοϰοµειο Παπαγεωργιου, οπου οµως λιγη αργοτερα υπεϰυψε ςτα τραυµατα του, ενω η οδηγος του αλλου οχηµατος υπεςτη ελαφρα τραυµατα. Οι αϰριβεις ςυνϑηϰες υπο τις οποιες προϰληϑηϰε η ςυγϰρουςη ερευνωνται απο το αρµοδιο τµηµα τροχαιας.
---|---|---
Title | Gold |  Τροχαιο δυςτυχηµα ςτη Θεςςαλονιϰη µε εναν νεϰρο
BART-random |  Τροχαιο εξω απο τη Θεςςαλονιϰη - Δυο τραυµατιες
mBART25 |  Θεςςαλονιϰη: Νεϰρος 33χρονος υςτερα απο ςυγϰρουςη δυο αυτοϰινητων
mBART50 |  Θεςςαλονιϰη: Νεϰρος 33χρονος υςτερα απο ςυγϰρουςη δυο αυτοϰινητων
GreekBART |  Τροχαιο ςτη Θεςςαλονιϰη: Νεϰρος 33χρονος ςε παραδροµο
?tablename? 14: Example 1-GreekSUM Title

| Document |  Ολες οι χωρες της Ευρωπαιϰης Ενωςης ςυµφωνουν οτι δεν ϑα πληρωςουν τη Ρωςια απευϑειας ςε ρουβλια για τις ειςαγωγες ρωςιϰου φυςιϰου αεριου, δηλωςαν υψηλοβαϑµοι ευρωπαιοι αξιωµατουχοι, ςηµειωνοντας οτι οι εποµενες πληρωµες ειναι προγραµµατιςµενες για τις 20 Μαιου. «Αυτο που γνωριζουµε, ϰαι υπαρχει ςυναινεςη επ αυτου µεταξυ ολων των ϰρατων µελων, ειναι οτι ϰανεις δεν ειναι προϑυµος να πληρωςει ςε ρουβλια», δηλωςε ο ενας αξιωµατουχος ϰατα την διαρϰεια ενηµερωςης των δηµοςιογραφων ϰαι προςϑετοντας οτι η Ευρωπαιϰη Επιτροπη δεν γνωριζει ποςοι αγοραςτες εχουν ανοιξει λογαριαςµους για πληρωµες προµηϑειας φυςιϰου αεριου µεςω της Gazprombank. Στο µεταξυ, ανωτερος αξιωµατουχος της Ευρωπαιϰης Ενωςης δηλωςε πως ϰαι µονο το ανοιγµα τραπεζιϰου λογαριαςµου ςε ρουβλια ςτην Gazprombank ενδεχεται να αποτελει παραβιαςη των ϰυρωςεων που εχει επιβαλει η ΕΕ ςε βαρος της Ρωςιας, οµως η ΕΕ δεν εχει ενδειξη πως ϰαποια εταιρεια φυςιϰου αεριου της ΕΕ εχει ϰανει ϰατι τετοιο. Ο αξιωµατουχος δηλωςε πως «εϰ πρωτης οψεως» το ανοιγµα τραπεζιϰων λογαριαςµων ςε ρουβλια απο ειςαγωγεις φυςιϰου αεριου φαινεται οτι παραβιαζει τις ϰυρωςεις. Ο αξιωµατουχος προςϑεςε πως η Ευρωπαιϰη Επιτροπη δεν εχει ϰαποια επιςηµη ενδειξη οτι εταιρειες της ΕΕ εχουν δηµιουργηςει ςτηνGazprombank λογαριαςµους ςε ρουβλια για την πληρωµη του φυςιϰου αεριου. Επιςης διευϰρινιςε πως η Πολωνια ϰαι η Βουλγαρια χρηςιµοποιηςαν τις υφιςταµενες µεϑοδους πληρωµης για το ρωςιϰο αεριο, πριν η Μοςχα αναςτειλει χϑες, Τεταρτη, τις προµηϑειες των χωρων αυτων µε αεριο, ϰαι πως δεν χρηςιµοποιηςαν τον µηχανιςµο που προτεινει η Μοςχα για να πληρωςουν ςε ρουβλια. «Συµφωνα µε τις πληροφοριες µας, αµφοτερες οι χωρες επεµειναν ςτην αρχιϰη µορφη πληρωµης», δηλωςε ο αξιωµατουχος ςε δηµοςιογραφους. Ωςτοςο δυο πηγες ειπαν ςηµερα ςτο Ροιτερς οτι λιγες ευρωπαιϰες εταιρειες εχουν αρχιςει να πληρωνουν ςε ρουβλια τη Ρωςια για το φυςιϰο αεριο, αν ϰαι µεγαλοι πελατες της δεν το εχουν ϰανει αϰοµη. «Μεριϰες εµποριϰες εταιρειες, ιςως περιςςοτερες απο πεντε, εχουν αρχιςει τις πληρωµες», ειπε µια πηγη, ζητωντας να µην ϰατονοµαςτει, επειδη δεν ειχε εξουςιοδοτηϑει να µιληςει ςτα µεςα ενηµερωςης.
---|---|---
Title | Gold |  Φυςιϰο αεριο: Ολες οι χωρες της ΕΕ ςυµφωνουν οτι δεν ϑα πληρωςουν τη Ρωςια ςε ρουβλια
BART-random |  Ε.Ε.: «Δεν ϑα πληρωςουν» οι χωρες της ΕΕ για το φυςιϰο αεριο ςε ρουβλια
mBART25 |  ΕΕ: Οι χωρες δεν πληρωνουν ςε ρουβλια τη Ρωςια για το φυςιϰο αεριο
mBART50 |  ΕΕ: Οι χωρες δεν πληρωνουν ςε ρουβλια τη Ρωςια για το φυςιϰο αεριο
GreekBART |  ΕΕ: Δεν ϑα πληρωςουµε τη Ρωςια ςε ρουβλια για το φυςιϰο αεριο
?tablename? 15: Example 2-GreekSUM Title

| Document |  Στις ηµερες του Παςχα εχει προςαρµοςτει το προγραµµα λειτουργιας λεωφορειων, τρολει, ηλεϰτριϰου ϰαι µετρο. Ειδιϰοτερα, τα λεωφορεια ϰαι τα τρολει ςηµερα, Μεγαλη Παραςϰευη, ϑα ϰινουνται µε προγραµµα Σαββατου. Οι ςυρµοι ςτο µετρο ϑα διερχονται απο τους ςταϑµους ανα 7 λεπτα απο τις 09.00 εως τις 17.00 ϰαι ανα 10 λεπτα τις υπολοιπες ωρες. ϒπενϑυµιζεται πως δεν ϑα ιςχυςει η διωρη παραταςη λειτουργιας που εφαρµοζεται τις Παραςϰευες. Στον ηλεϰτριϰο οι ςυρµοι ϑα διερχονται απο τους ςταϑµους ανα 10,5 λεπτα. Τα λεωφορεια ϰαι τα τρολει ϑα ϰινηϑουν µε προγραµµα Κυριαϰης, ενω ϑα αποςυρϑουν νωριτερα, ωςτε να βριςϰονται ςτα αµαξοςταςια ςτις 23.00. Τα λεωφορεια ϑα ϰινηϑουν µε προγραµµα Κυριαϰης ϰαι τα τρολει µε ειδιϰο προγραµµα Κυριαϰης. Τοςο ςτα δροµολογια των λεωφορειων οςο ϰαι ς´ αυτα των τρολει ϑα εφαρµοςτει ειδιϰο προγραµµα Σαββατου. Αϰινητοποιηµενοι ϑα µεινουν την Τεταρτη 1η Μαιου οι ςυρµοι του ηλεϰτριϰου (πρωην ΗΣΑΠ), τα λεωφορεια, τα τρολει, αλλα ϰαι ο ςιδηροδροµος, λογω 24ωρης απεργιας των εργαζοµενων, που ϑα ςυµµεταςχουν ςτις απεργιαϰες ςυγϰεντρωςεις για την Πρωτοµαγια. Οπως αναφερουν ςε αναϰοινωςη τους οι εργαζοµενοι ςτον πρωην ΗΣΑΠ, «ειναι µερα αγωνα, τιµης ϰαι µνηµης. Θυµοµαςτε ϰαι τιµαµε τους πρωτοπορους αγωνιςτες ϰαι τα ϑυµατα των εργατιϰων αγωνων για βελτιωςη των ςυνϑηϰων δουλειας για αξιοπρεπεις αµοιβες ϰαι την ϰατοχυρωςη των διϰαιωµατων µας. Αναςυγϰροτουµαςτε, ϑετουµε τους ςτοχους µας ϰαι προχωραµε ςε νεους αγωνες. Διεϰδιϰουµε ϰαι παλευουµε για την αναπληρωςη απωλειων απο τις µνηµονιαϰες πολιτιϰες λιτοτητας, για πραγµατιϰες αυξηςεις ςτους µιςϑους ϰαι ςτις ϰοινωνιϰες παροχες». Και προςϑετουν «υπεραςπιζοµαςτε τον δηµοςιο χαραϰτηρα των ςυγϰοινωνιων. Διεϰδιϰουµε την υπογραφη νεας Συλλογιϰης Συµβαςης Εργαςιας. Αγωνιζοµαςτε για αςφαλεις, φϑηνες ςυγϰοινωνιες. Με αγωνες ϰαταϰταµε τα διϰαιωµατα µας».
---|---|---
Title | Gold |  Παςχα 2019: Πως ϑα ϰινηϑουν λεωφορεια, τρολει, ηλεϰτριϰος ϰαι µετρο
BART-random |  Μεςα Μαζιϰης Μεταφορας: Πως ϑα ϰινηϑουν ςηµερα τα Μεςα Μεταφορας
mBART25 |  Μεςα Παςχα: Πως ϑα ϰινηϑουν ςηµερα λεωφορεια, τρολει, ηλεϰτριϰο ϰαι µετρο
mBART50 |  Μεςα Παςχα: Πως ϑα ϰινηϑουν ςηµερα λεωφορεια, τρολει, ηλεϰτριϰο ϰαι µετρο
GreekBART |  Παςχα: Πως ϑα ϰινηϑουν ςηµερα λεωφορεια, τρολει, ηλεϰτριϰου ϰαι µετρο
?tablename? 16: Example 3-GreekSUM Title

| Document |  Συναντηςη µε οιϰονοµιϰους παραγοντες απο το Σιτι του Λονδινου εχει αυτη την ωρα ο Αλεξης Τςιπρας ςτο ϰεντρο της βρετανιϰης πρωτευουςας. Τον Ελληνα πρωϑυπουργο υποδεχϑηϰε ο αντιπροεδρος της Επιτροπης Πολιτιϰης του Σιτι, Τοµ Σλει (Tom Sleigh). Επιςηµαινεται οτι η Επιτροπη υπεχει ϑεςη Διοιϰηςης του Σιτι του Λονδινου. Απο την αιϑουςα της «Παλιας Βιβλιοϑηϰης», ο πρωϑυπουργος ϑα απευϑυνϑει ςε εναν ϰυϰλο περιςςοτερων απο εϰατο ςηµαινοντων ςτελεχων της επενδυτιϰης/χρηµατοπιςτωτιϰης ϰοινοτητας του Σιτι ϰαι, ςυµφωνα µε πληροφοριες, ςτη ςυνεχεια ϑα αϰολουϑηςει ςυναντηςη ςε πιο ςτενο ϰυϰλο ςυµµετεχοντων. Στον αποηχο της αποφαςης του Eurogroup για την ελαφρυνςη του χρεους, οι επαφες του Αλεξη Τςιπρα µε ςηµαντιϰους εϰπροςωπους της επενδυτιϰης/χρηµατοπιςτωτιϰης της ϰοινοτητας του οιϰονοµιϰου ϰεντρου της Ευρωπης, ςηµατοδοτουν ενα ευϰρινες διεϑνες µηνυµα για τις προοπτιϰες της ελληνιϰης οιϰονοµιας ϰαι της «εποµενης µερας», ςτην περιοδο µετα την ολοϰληρωςη των µνηµονιων. Οπως ανεφερε ϰυβερνητιϰος αξιωµατουχος, οι ςηµερινες ςυναντηςεις ειναι ενας ςηµαντιϰος ςταϑµος ςε µια «αλυςιδα» επαφων ϰαι ςυνοµιλιων που ϑα ςυνεχιςτουν ςτο αµεςως εποµενο διαςτηµα των ϰαλοϰαιρινων µηνων ϰαι το φϑινοπωρο. Ενδειϰτιϰη της ευνοιϰης ςυγϰυριας για την ελληνιϰη οιϰονοµια ϰαι το ςτοιχηµα της αναϰαµψης, η χϑεςινοβραδινη αναβαϑµιςη, απο τον αµεριϰανιϰο οιϰο αξιολογηςης Standard & Poor’s της µαϰροπροϑεςµης πιςτοληπτιϰης ιϰανοτητας της χωρας ςε B+, χαιρετιζοντας την αποφαςη του Eurogroup . Στις 18:00 το απογευµα ωρα Ελλαδας, ο πρωϑυπουργος ϑα περαςει το ϰατωφλι της Downing Street 10 προϰειµενου να ςυναντηϑει µε την πρωϑυπουργο της Βρετανιας, Τερεζα Μει. Στη ςυνεχεια ϑα εχει ςυναντηςη µε τον αρχηγο του Εργατιϰου Κοµµατος, Τζερεµι Κορµπιν.
---|---|---
Title | Gold |  Συναντηςη µε οιϰονοµιϰους παραγοντες απο το Σιτι του Λονδινου εχει ο Αλεξης Τςιπρας
BART-random |  Μηνυµα Τςιπρα ςτο Λονδινο για το χρεος
mBART25 |  Συναντηςη Τςιπρα µε οιϰονοµιϰους παραγοντες ςτο Σιτι
mBART50 |  Συναντηςη Τςιπρα µε οιϰονοµιϰους παραγοντες ςτο Σιτι
GreekBART |  Βλεµµατα ςτο Λονδινο για την ελληνιϰη οιϰονοµια
?tablename? 17: Example 4-GreekSUM Title

| Document |  Επιβατιϰο τρενο εϰτροχιαςτηϰε ςηµερα περιπου 20 χλµ. βορεια της Ραµπατ, µε αποτελεςµα να ςϰοτωϑουν εξι ανϑρωποι ϰαι αλλοι 86 να τραυµατιςτουν, ςυµφωνα µε επιςηµο απολογιςµο που αναϰοινωϑηϰε ςτον τοπο του δυςτυχηµατος. «Ο εϰτροχιαςµος προϰαλεςε εξι ϑανατους, ςυµφωνα µε τον τρεχοντα απολογιςµο, ϰαι 86 τραυµατιες ςε ςοβαρη ϰαταςταςη», δηλωςε ο Μοχαµεντ Ραµπι Ραχιλ, γενιϰος διευϑυντης της εταιριας ςιδηροδροµων ONCF, ο οποιος µετεβη επι τοπου. «Ξεϰινηςε ερευνα για τον προςδιοριςµο των αιτιων του δυςτυχηµατος», προςϑεςε, ςε βιντεο που αναρτηϑηϰε ςτα µεςα ϰοινωνιϰης διϰτυωςης. Θεαµατιϰες ειϰονες του δυςτυχηµατος, που ςηµειωϑηϰε γυρω ςτις 13:00 ωρα Ελλαδας, περιπου 20 χλµ. βορεια της πρωτευουςας Ραµπατ, ςτο υψος της ϰοινοτητας Σιντι Μπουϰναντελ, ϰανουν τον γυρο των µεςων ϰοινωνιϰης διϰτυωςης, που ειναι πολυ επιϰριτιϰα εναντιον της ONCF. Οι ειϰονες δειχνουν πολλα βαγονια εϰτροχιαςµενα ϰοντα ςε µια γεφυρα ςτους αγρους, ενω η µηχανη ειναι πληρως ϰατεςτραµµενη. Ο οδηγος της αµαξοςτοιχιας ειναι νεϰρος, ςυµφωνα µε πολλα τοπιϰα ΜΜΕ. Ο βαςιλιας αποφαςιςε να αναλαβει τα εξοδα της ϰηδειας των ϑυµατων ϰαι οι τραυµατιες ϑα διαϰοµιςτουν ςτο ςτρατιωτιϰο νοςοϰοµειο της Ραµπατ µε βαςιλιϰες οδηγιες, αναφερεται ςε αναϰοινωςη του γραφειου του βαςιλια.
---|---|---
Title | Gold |  Εϰτροχιαςµος τρενου ςτο Μαροϰο: Στους 6 οι νεϰροι - 86 τραυµατιες
BART-random |  Ραµπατ: 20 νεϰροι απο εϰτροχιαςµο τρενου
mBART25 |  ΗΠΑ: Επιβατιϰο τρενο εϰτροχιαςτηϰε - Εξι νεϰροι ϰαι 86 τραυµατιες
mBART50 |  ΗΠΑ: Επιβατιϰο τρενο εϰτροχιαςτηϰε - Εξι νεϰροι ϰαι 86 τραυµατιες
GreekBART |  Εϰτροχιαςµος τρενου ςτη Ραµπατ: Εξι νεϰροι ϰαι 86 τραυµατιες
?tablename? 18: Example 5-GreekSUM Title


# Performance of near-optimal protocols in weak processes

Pierre Nazé<EMAIL_ADDRESS>Departamento de Física, Instituto de
Geociências e Ciências Exatas, Universidade Estadual Paulista “Júlio de
Mesquita Filho”, 13506-900, Rio Claro, SP, Brazil

###### Abstract

A natural criticism of the universal optimal protocol of the irreversible work
found in the context of weak processes is its experimental difficulty to be
implementable due to its singular part. In this work, I propose as a partial
solution to this problem its continuous linear part as an acceptable near-
optimal protocol. This is based on the analysis of several examples of the
error committed to approximating the solution extended until its second order
in its continuous linear part. The result seems to be universal: depending
mainly on the ratio between switching time and waiting time $\tau/\tau_{w}$,
the error for sudden and slowly-varying processes is less than $1\%$, while
for $\tau\approx\tau_{w}$ it has a peak with an upper bound around $8\%$.
Although implementing Dirac deltas could be an experimental challenge, I
present also the error including those functions, where the results of these
new near-optimal protocols become slightly better.

##  I Introduction

Optimization problems are very subjected to the space of admissible functions
of their solutions [1]. In some cases, computational research may not be able
to visualize completely some regions due to its proper limitations. The
optimal protocols found in this research are nothing more than near-optimal
protocols when related to a bigger space of admissible functions.

Another situation when this happens is when the admissible functions include
generalized functions, such as Dirac deltas and their derivatives [2]. In
particular, the experimental implementation of these cases can be a challenge,
although a proposition of how one can implement them has been proposed [2]. In
this case, accessible near-optimal protocols, tangible to experimentalists,
are of utmost importance.

In Ref. [2], a universal optimal protocol for the irreversible work and its
variance for isothermal and weak processes has been proposed, where the
solution is split into a continuous linear part and a singular part, composed
of Dirac deltas and their derivatives. In the face of the problem reported
above, I propose as an acceptable near-optimal protocol for this case the
continuous linear part. This proposition is based on a universal behavior
verified in the analyses of several examples of the error committed by not
adding the Dirac deltas and first derivatives of the singular part: for sudden
and slowly-varying processes the error is less than 1%, while for other cases
it is less than 8%. Finally, I present the case where the Dirac deltas are
included in the near-optimal protocol, whose result becomes slightly better.

##  II Preliminaries

I start defining notations and developing the main concepts to be used in this
work.

Consider a classical system with a Hamiltonian
$\mathcal{H}(\mbox{\boldmath$z$}(\mbox{\boldmath$z_{0}$},t)),\lambda(t))$,
where $\mbox{\boldmath$z$}(\mbox{\boldmath$z_{0}$},t)$ is a point in the phase
space $\Gamma$ evolved from the initial point $z_{0}$ until time $t$, with
$\lambda(t)$ being a time-dependent external parameter. During a switching
time $\tau$, the external parameter is changed from $\lambda_{0}$ to
$\lambda_{0}+\delta\lambda$, with the system being in contact with a heat bath
of temperature $\beta\equiv{(k_{B}T)}^{-1}$, where $k_{B}$ is Boltzmann’s
constant. The average work performed on the system during this interval of
time is

$\overline{W}\equiv\int_{0}^{\tau}\left\langle\overline{\partial_{\lambda}\mathcal{H}}(t)\right\rangle_{0}\dot{\lambda}(t)dt,$
(1)

where $\partial_{\lambda}$ is the partial derivative in respect to $\lambda$
and the superscripted dot the total time derivative. The generalized force
$\left\langle\overline{\partial_{\lambda}\mathcal{H}}\right\rangle_{0}$ is
calculated using the averaging $\overline{\cdot}$ over the stochastic path and
the averaging $\langle\cdot\rangle_{0}$ over the initial canonical ensemble.
The external parameter can be expressed as

$\lambda(t)=\lambda_{0}+g(t)\delta\lambda,$ (2)

where, to satisfy the initial conditions of the external parameter, the
protocol $g(t)$ must satisfy the following boundary conditions

$g(0)=0,\quad g(\tau)=1.$ (3)

We consider as well that $g(t)\equiv g(t/\tau)$, which means that the
intervals of time are measured according to the switching time unit.

Linear-response theory aims to express average quantities until the first-
order of some perturbation parameter considering how this perturbation affects
the observable to be averaged and the process of average [3]. In our case, we
consider that the parameter does not considerably changes during the process,
$|g(t)\delta\lambda/\lambda_{0}|\ll 1$, for all $t\in[0,\tau]$. In that
manner, using such framework, the generalized force can be approximated until
first-order as

$\begin{split}\left\langle\overline{\partial_{\lambda}\mathcal{H}}(t)\right\rangle_{0}=&\,\left\langle\partial_{\lambda}\mathcal{H}\right\rangle_{0}+\delta\lambda\left\langle\partial_{\lambda\lambda}^{2}\mathcal{H}\right\rangle_{0}g(t)\\\
&-\delta\lambda\int_{0}^{t}\phi_{0}(t-t^{\prime})g(t^{\prime})dt^{\prime}.\end{split}$
(4)

The quantity $\phi_{0}(t)$ is the so-called response function [3], which can
be conveniently expressed as the derivative of the relaxation function
$\Psi_{0}(t)$ [3]

$\phi_{0}(t)=-\frac{d\Psi_{0}}{dt}.$ (5)

In our particular case, the relaxation function is calculated as

$\Psi_{0}(t)=\beta\left\langle\partial_{\lambda}\mathcal{H}(0)\overline{\partial_{\lambda}\mathcal{H}}(t)\right\rangle_{0}-\mathcal{C},$
(6)

where the constant $\mathcal{C}$ is calculated to vanish the relaxation
function for long times [3]. We define as the relaxation timescale of the
system the quantity

$\tau_{R}=\int_{0}^{\infty}\frac{\Psi(t)}{\Psi(0)}dt.$ (7)

The generalized force, written in terms of the relaxation function, can be
expressed as

$\begin{split}\left\langle\overline{\partial_{\lambda}\mathcal{H}}(t)\right\rangle_{0}=&\,\left\langle\partial_{\lambda}\mathcal{H}\right\rangle_{0}-\delta\lambda\widetilde{\Psi}_{0}g(t)\\\
&+\delta\lambda\int_{0}^{t}\Psi_{0}(t-t^{\prime})\dot{g}(t^{\prime})dt^{\prime},\end{split}$
(8)

where
$\widetilde{\Psi}_{0}\equiv\Psi_{0}(0)-\left\langle\partial_{\lambda\lambda}^{2}\mathcal{H}\right\rangle_{0}$.
Finally, combining Eqs. (1) and (8), the average work performed at the linear
response of the generalized force is

$\begin{split}\overline{W}=&\,\delta\lambda\left\langle\partial_{\lambda}\mathcal{H}\right\rangle_{0}-\frac{\delta\lambda^{2}}{2}\widetilde{\Psi}_{0}\\\
&+\delta\lambda^{2}\int_{0}^{\tau}\int_{0}^{t}\Psi_{0}(t-t^{\prime})\dot{g}(t^{\prime})\dot{g}(t)dt^{\prime}dt.\end{split}$
(9)

We observe that the double integral on Eq. (9) vanishes for long switching
times [4]. Therefore the other terms are part of the contribution of the
difference of free energy, since this quantity is exactly the average work
performed for quasistatic processes in isothermal drivings. Thus, we can split
the average work into the difference of free energy $\Delta F$ and
irreversible work $W_{\rm irr}$

$\Delta
F=\delta\lambda\left\langle\partial_{\lambda}\mathcal{H}\right\rangle_{0}-\frac{\delta\lambda^{2}}{2}\widetilde{\Psi}_{0},$
(10)
$\begin{split}W_{\text{irr}}=\delta\lambda^{2}\int_{0}^{\tau}\int_{0}^{t}\Psi_{0}(t-t^{\prime})\dot{\lambda}(t^{\prime})\dot{\lambda}(t)dt^{\prime}dt.\end{split}$
(11)

In particular, the irreversible work can be rewritten using the symmetric
property of the relaxation function [3]

$\begin{split}W_{\rm
irr}=\frac{\delta\lambda^{2}}{2}\int_{0}^{\tau}\int_{0}^{\tau}\Psi_{0}(t-t^{\prime})\dot{\lambda}(t^{\prime})\dot{\lambda}(t)dt^{\prime}dt.\end{split}$
(12)

I establish at this point the regimes where linear-response theory is able to
describe thermodynamic processes. Those regimes are determined by the relative
strength of the driving with respect to the initial value of the protocol,
$\delta\lambda/\lambda_{0}$, and the rate by which the process occurs with
respect to the relaxation time of the system, $\tau_{R}/\tau$. See Fig. 1 for
a diagram depicting the regimes. In region 1, the so-called slowly-varying
processes, the ratio $\delta\lambda/\lambda_{0}$ is arbitrary, while
$\tau_{R}/\tau\ll 1$. By contrast, in region 2, the so-called finite-time and
weak processes, the ratio $\delta\lambda/\lambda_{0}\ll 1$, while
$\tau_{R}/\tau$ is arbitrary. In region 3, the so-called arbitrarily far-from-
equilibrium processes, both ratios are arbitrary. Linear-response theory can
only describe regions 1 and 2 [4]. In this work, we are going to focus on
region 2 only.

Figure 1: (Color online) Diagram of nonequilibrium regions. Region 1: slowly-
varying processes, Region 2: finite-time but weak processes and Region 3:
arbitrarily far-from-equilibrium processes. Linear response theorem can
describe regions 1 and 2.

Consider the irreversible work rewritten in terms of the protocols $g(t)$
instead of its derivative

$\displaystyle W_{\rm irr}=$
$\displaystyle\frac{\delta\lambda^{2}}{2}\Psi(0)+\delta\lambda^{2}\int_{0}^{\tau}\dot{\Psi}_{0}(\tau-t)g(t)dt$
(13)
$\displaystyle-\frac{\delta\lambda^{2}}{2}\int_{0}^{\tau}\int_{0}^{\tau}\ddot{\Psi}(t-t^{\prime})g(t)g(t^{\prime})dtdt^{\prime}.$
(14)

Using calculus of variations, we can derive the Euler-Lagrange equation that
furnishes the optimal protocol $g^{*}(t)$ of the system that will minimize the
irreversible work [5]

$\int_{0}^{\tau}\ddot{\Psi}_{0}(t-t^{\prime})g^{*}(t^{\prime})dt^{\prime}=\dot{\Psi}_{0}(\tau-t).$
(15)

In particular, the optimal irreversible work will be [5]

$W_{\rm
irr}^{*}=\frac{\delta\lambda^{2}}{2}\Psi(0)+\frac{\delta\lambda^{2}}{2}\int_{0}^{\tau}\dot{\Psi}_{0}(\tau-t)g^{*}(t)dt.$
(16)

Also, the Euler-Lagrange equation (15) furnishes also the optimal protocol
that minimizes the variance of the work [6]. In this case, the optimal
variance of work is

$\sigma_{\rm
W}^{2^{*}}=\frac{\beta\delta\lambda^{2}}{4}\Psi(0)+\frac{\beta\delta\lambda^{2}}{4}\int_{0}^{\tau}\dot{\Psi}_{0}(\tau-t)g^{*}(t)dt.$
(17)

In Ref. [2], the following universal solution was found

$g^{*}(t)=\frac{t+\tau_{w}}{\tau+2\tau_{w}}+\sum_{n=0}^{\infty}\frac{a_{n}(\delta^{(n)}(t)-\delta^{(n)}(\tau-t))}{\tau+2\tau_{w}},$
(18)

where the waiting time $\tau_{w}$ is defined as

$\tau_{w}=\mathcal{L}_{t}[\Psi(t)](0).$ (19)

The objective of this work is to respond to a natural criticism of such a
universal solution: due to its singular part, its experimental implementation
can be a real challenge, so how can one overcome this problem? I present here
an analysis of the error committed in the irreversible work in two
approximations: considering only the continuous linear part and this part
added to the Dirac deltas. As we are going to see, these near-optimal
protocols are quite reasonable.

##  III Approximation errors

How much do the addition of deltas peaks and their derivatives affect the
optimal irreversible work? Let us define the following quantity

$\Delta
W_{n}=\frac{\delta\lambda^{2}}{2}\int_{0}^{\tau}\dot{\Psi}(\tau-t)g_{n}^{*}(t)dt,$
(20)

where

$g_{-1}^{*}(t)=\frac{t+\tau_{w}}{\tau+2\tau_{w}},$ (21)

and

$g_{n}^{*}(t)=\frac{a_{n}(\delta^{(n)}(t)-\delta^{(n)}(\tau-t))}{\tau-2\tau_{R}},$
(22)

for $n\geq 0$. Two errors will be analyzed

$\epsilon_{-1}(\tau/\tau_{w})=\left|\frac{\sum_{n=0}^{1}\Delta
W_{n}}{\frac{\delta\lambda^{2}}{2}\Psi(0)+\sum_{n=-1}^{1}\Delta
W_{n}}\right|,$ (23)

and

$\epsilon_{0}(\tau/\tau_{w})=\left|\frac{\Delta
W_{1}}{\frac{\delta\lambda^{2}}{2}\Psi(0)+\sum_{n=-1}^{1}\Delta W_{n}}\right|$
(24)

where the first does not consider the addition of terms of the singular part
and the second one do not consider any derivative. Observe that the errors do
not depend on $\Psi(0)$. I considered also without loss of generality
$\delta\lambda=1$.

##  IV Examples

I present now the examples that will be analyzed in the next section.

###  IV.1 Overdamped Brownian motion

We consider in this example a white noise overdamped Brownian motion subjected
to a time-dependent harmonic potential, with the mass of the system equal to
one, $\gamma$ as a damping coefficient and $\omega_{0}$ as the natural
frequency of the potential. The relaxation function for both moving laser and
stiffening traps [5] are given by

$\Psi_{1}(t)=\Psi_{0}\exp{\left(-\frac{|t|}{\tau_{R}}\right)},$ (25)

where $\tau_{R}$ is the relaxation timescale of each case.

###  IV.2 Underdamped Brownian motion:
moving laser trap

We consider in this example a white noise overdamped Brownian motion subjected
to a time-dependent harmonic potential, with $m$ as the mass of the particle,
$\gamma$ as a damping coefficient, and $\omega_{0}$ as the natural frequency
of the potential. The relaxation function for moving laser trap [5] is given
by

$\Psi_{2}(t)=\Psi_{0}\exp{\left(-\frac{\gamma}{2}|t|\right)}\left(\cos{\omega
t}+\frac{\gamma}{2\omega}\sin{\omega|t|}\right),$ (26)

where $\omega=\sqrt{\omega_{0}^{2}-\gamma^{2}/4}$.

###  IV.3 Underdamped Brownian motion:
stiffening laser trap

For the same system, but in the stiffening trap case, the relaxation function
is [4]

$\Psi_{3}(t)=\Psi_{0}\exp{\left(-\gamma|t|\right)}\left[\frac{2\omega_{0}^{2}}{\omega^{2}}\right.\\\
\left.+\left(\frac{\omega^{2}-2\omega_{0}^{2}}{\omega^{2}}\right)\cos{\omega
t}+\frac{\gamma}{\omega}\sin{\omega|t|}\right],$ (27)

where $\omega=\sqrt{4\omega_{0}^{2}-\gamma^{2}}$.

###  IV.4 Sinc relaxation function

In Ref. [7], when we apply the method of time average in a thermally isolated
system performing an adiabatic process, we produce a new one performing an
isothermal one with a typical relaxation time. In particular, for thermally
isolated systems that have a relaxation function equal to

$\Psi(t)=\Psi_{0}\cos{(\omega t)},$ (28)

will have for time-averaged relaxation function

$\Psi_{4}(t)=\Psi_{0}\,\text{sinc}\left(\frac{\pi}{2}\frac{t}{\tau_{R}}\right),$
(29)

where $\tau_{R}$ is the relaxation timescale of the system.

###  IV.5 Gaussian relaxation function

A relaxation function that satisfies the criteria of compatibility with the
Second Law of Thermodynamics [4] is the Gaussian relaxation function

$\Psi_{5}(t)=\Psi_{0}\exp\left(-\frac{\pi}{4}\left(\frac{t}{\tau_{R}}\right)^{2}\right),$
(30)

where $\tau_{R}$ is the relaxation timescale of the system.

###  IV.6 Bessel relaxation function

The Bessel relaxation function is given by

$\Psi_{6}(t)=\Psi_{0}J_{0}\left(\frac{t}{\tau_{R}}\right),$ (31)

where $J_{0}$ is the Bessel function of the first kind with $\nu=0$ and
$\tau_{R}$ is its relaxation timescale. It satisfies the criteria for
compatibility with the Second Law of Thermodynamics. Such relaxation function
can model the Ising chain subjected to a time-dependent magnetic field and
evolving in time at equilibrium accordingly to Glauber-Ising dynamics [8].

##  V Error analysis

In Fig. 2 are depicted graphics for each one of those examples presented in
the previous section illustrating the errors $\epsilon_{-1}$ and
$\epsilon_{0}$ for ratios $\tau/\tau_{w}$. For $\Psi_{1}(t)$, the errors
$\epsilon_{-1}$ and $\epsilon_{0}$ are null since the optimal protocol does
not depend on the deltas. For $\Psi_{2}(t)$, the error $\epsilon_{0}$ is null
since the optimal protocol does not depend on derivatives of Dirac delta. For
the other cases, a general feature for error $\epsilon_{-1}$ appears: for
sudden and slowly-varying processes, where respectively $\tau/\tau_{w}\ll 1$
and $\tau/\tau_{w}\gg 1$, the error becomes less than $1\%$. This is easy to
understand since there is almost no participation of the Dirac delta and
derivatives in this regimes [2]. However, when $\tau\approx\tau_{w}$ the error
achieves a peak with an upper bound around $8\%$, which indicates a more
participation of the terms of the singular part. For the error $\epsilon_{0}$,
the general feature persists, but the results are better, with a peak with an
upper bound of $4\%$. Therefore, it is quite reasonable to consider such
linear solution or linear solution plus Dirac deltas as acceptable near-
optimal protocols. Finally, it is important to remark that such numbers can in
principle increase as high orders are added. However, for being of such
nature, these increments should be negligible.

##  VI Final remarks

In this work, I proposed the continuous linear part approximation of the
universal optimal protocol for weak processes as a reasonable near-optimal
protocol. The proposition is based on the analyses of several examples where
the errors of adding the terms of the singular parts were considered. The
error $\epsilon_{-1}$, where no term was added, for sudden and slowly-varying
processes is less than 1%, while it has a peak with an upper bound of 8%. The
error $\epsilon_{0}$ this universal feature persists, but the result is
better: the upper bound reduce to 4%. In this way, it is reasonable to
consider such approximations as near-optimal protocols.

Figure 2: Errors for relaxation functions
$\Psi_{1}(t),\Psi_{2}(t),\Psi_{3}(t),\Psi_{4}(t),\Psi_{5}(t),\Psi_{6}(t)$. For
error $\epsilon_{-1}$ (black line) a general feature appears: for sudden
($\tau/\tau_{w}\ll 1$) and slowly-varying ($\tau/\tau_{w}\gg 1$) processes,
error less than 1%. For $\tau\approx\tau_{w}$ error presents a peak with upper
bound around 8%. For error $\epsilon_{0}$ (red line) the feature persist, but
the result is slightly better: the upper bound reduce to only 4%.

## References

  * Kirk [2004] D. E. Kirk,  _Optimal control theory: an introduction_ (Courier Corporation, 2004).
  * Nazé [2023a] P. Nazé, arXiv preprint arXiv:2305.08597 (2023a).
  * Kubo  _et al._ [2012] R. Kubo, M. Toda, and N. Hashitsume,  _Statistical physics II: nonequilibrium statistical mechanics_, Vol. 31 (Springer Science & Business Media, 2012).
  * Nazé and Bonança [2020] P. Nazé and M. V. S. Bonança, Journal of Statistical Mechanics: Theory and Experiment 2020, 013206 (2020).
  * Nazé  _et al._ [2022] P. Nazé, S. Deffner, and M. V. Bonança, Journal of Physics Communications 6, 083001 (2022).
  * Nazé [2023b] P. Nazé, arXiv preprint arXiv:2304.11965 (2023b).
  * Nazé [2022] P. Nazé, arXiv preprint arXiv:2210.16116 (2022).
  * Glauber [1963] R. J. Glauber, Journal of mathematical physics 4, 294 (1963).