# Can we do that simpler?
Simple, Efficient, High-Quality Evaluation Metrics for NLG

Jens Grünwald1, Christoph Leiter2, Steffen Eger2
1 Department of Computer Science, 2 Faculty of Technology
1 Technical University of Darmstadt, 2 Bielefeld University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

We explore efficient evaluation metrics for Natural Language Generation (NLG).
To implement efficient metrics, we replace (i) computation-heavy transformers
in metrics such as BERTScore, MoverScore, BARTScore, XMoverScore, etc. with
lighter versions (such as distilled ones) and (ii) cubic inference time
alignment algorithms such as Word Mover Distance with linear and quadratic
approximations. We consider six evaluation metrics (both monolingual and
multilingual), assessed on three different machine translation datasets, and
16 light-weight transformers as replacement. We find, among others, that (a)
TinyBERT shows best quality-efficiency tradeoff for semantic similarity
metrics of the BERTScore family, retaining 97% quality and being 5x faster at
inference time on average, (b) there is a large difference in speed-ups on CPU
vs. GPU (much higher speed-ups on CPU), and (c) WMD approximations yield no
efficiency gains but lead to a substantial drop in quality on 2 out of 3
datasets we examine.

##  1 Introduction

Evaluation is key to progress in fields such as NLP and machine learning, as
inadequate evaluation may give a false impression of the state-of-the-art and
cause millions of dollars to be spent on inferior technology. Evaluation is
particularly challenging for Natural Language Generation (NLG) systems as text
generation is open-ended and multiple outputs, with very different surface-
level realizations, can be equally correct (Celikyilmaz et al., 2020). This
insight makes classical lexical overlap metrics such as BLEU or ROUGE
unsuitable as high-quality evaluation metrics. As a consequence, there has
been a recent surge of interest in developing evaluation metrics based on
large-scale language models, which can much better cope with lexical
variation, thus yielding metrics that correlate much better with human
assessments of quality (Zhao et al., 2019; Zhang et al., 2020; Yuan et al.,
2021).

However, basing evaluation metrics on large transformers (and thus boosting
quality of the metrics) also has downsides: for example, the associated
computational burden (i) may promote inequality among researchers, hindering
diversity, as not everyone has access to expensive compute resources111E.g.,
Kamal Eddine et al. (2022) mention that they cannot run variants of BERTScore
on a 12GB GPU and the situation would even be worse for more disadvantaged
scholars around the world. and (ii) incurs high environmental costs, one of
the most critical issues of our time (Strubell et al., 2019). (iii)
Inefficient metrics – which are in addition non-transparent (Leiter et al.,
2022) – may also prevent high-quality metrics from being deployed by the
community, a potential reason why older, lower-quality but faster metrics such
as BLEU still dominate (Marie et al., 2021).

As a consequence, it is imperative to develop _light-weight high-quality_
evaluation metrics for NLG, which we explore in depth in this work.

Our main contributions are:

  * •

We provide a comprehensive analysis of inducing efficient high-quality
evaluation metrics, based on two principles: (i) replacing a computation heavy
transformer in the metrics by a much smaller one, obtained e.g. via pruning or
distillation; (ii) replacing costly alignment techniques (Word Mover Distance;
WMD) on top of transformers with cheaper approximations.

  * •

Our analysis comprises three MT datasets, six evaluation metrics (including
monolingual reference-based and multilingual reference-free ones), and 16
light-weight transformers as replacements of the original transformers.

  * •

On the basis of our large-scale analysis, we find that: (a) for each metric,
there is often at least one efficient transformer which leads to higher
quality and higher efficiency at the same time, but on average, there is a
drop in quality when employing more efficient models; (b) for example, for the
“semantic similarity” metrics similar to BERTScore, we find that the distilled
transformer TinyBERT has best performance-quality tradeoff—on average, it
retains 97% of the original quality while being 5x faster at inference time;
(c) speedups differ substantially on CPU vs. GPU; (d) WMD approximations yield
no efficiency gains in our experiments (as WMD itself is less costly than
embedding computation), but have adverse effects on quality on 2 out of 3
datasets (but leading to surprising quality gains on the last).

Our code is available from https://github.com/jbgruenwald/efficient-nlg-
metrics.

##  2 Related work

Our work connects to (1) transformer-based evaluation metrics and to (2)
efficiency.

##### Evaluation metrics

Recent years have seen a surge of interest in transformer based evaluation
metrics as these promise higher quality (as measured in correlation with human
assessments). Among the first such models are BERTScore (Zhang et al., 2020)
and MoverScore (Zhao et al., 2019) which model evaluation as a semantic
similarity task using contextualized BERT representations. Extensions include
BARTScore (Yuan et al., 2021), which reads off probability estimates as metric
scores directly from text generation systems. A distinction of evaluation
metrics is whether they use human references (in reference-based models) or do
not use them (in reference-free models), where the latter are less costly. In
this work, we consider XMoverScore (Zhao et al., 2020) and SentSim (Song et
al., 2021) as reference-free metrics and BARTScore, BERTScore, MoverScore and
BaryScore (Colombo et al., 2021) as reference-based metrics. In MT, the
challenge for reference-free metrics are cross-lingual representation spaces,
for which there are also different efficient transformer variants.

##### Efficiency

is a core issue of modern deep learning systems, which have become bigger and
bigger in a quest for better performance, leading to environmental concerns
and increasing inequality/exclusion. For example, the recent PaLM language
model from Google (Chowdhery et al., 2022) trains on more than 6000 TPUs,
which is beyond the capabilities of most non-industrial organizations. There
are many approaches for obtaining more efficient models, especially in the
context of computation heavy transformers.

Pruning means reducing the number of parameters in a model. Structured pruning
removes complete architectural blocks – e.g., removes encoder layers, shrinks
embedding size or uses less attention heads. Unstructured pruning means
cancelling random parameters inside an architectural block, like applying
dropout (Ganesh et al., 2021). Pruning can be applied to the pre-trained
model, where it benefits from the quality of the originally bigger model that
it was during training (Ganesh et al., 2021), or to the untrained structure of
a network, which afterwards gets trained in the traditional way – e.g., using
Masked Language Modeling (MLM).

Knowledge Distillation involves (i) a student with pruned layers, embedding
size or attention heads or even with layers replaced by alternative simpler
network architectures and (ii) a teacher usually of the same architecture, but
larger. Then the student is trained with outputs from the teacher (Hinton et
al., 2015; Ganesh et al., 2021). While data for traditional training often is
scarce, in knowledge distillation the teacher generates new data for training
the student (Hinton et al., 2015). Knowledge distillation thus usually leads
to a better quality for the student model than traditional training because it
benefits from the knowledge of the larger model. For example, DistilBERT (Sanh
et al., 2019) consists of only 6 layers, which is half the amount of BERTBASE,
but with reported small reductions of model quality. TinyBERT (Jiao et al.,
2020) introduces an improved method for knowledge distillation in which the
student is not only trained with outputs (as DistilBERT does) but also
intermediate results.

Dynamic Inference Acceleration: Sun et al. (2019); Xin et al. (2020); Zhu
(2021) compare intermediate results after each encoder layer, after which they
add an early exit ramp. If the change of the representation from one layer to
the next is lower than a threshold, they assume later layers will improve even
less and skip them by taking the early exit ramp. The final output for that
sample will then be the representation from that layer.

The closest related work to ours is FrugalScore (Kamal Eddine et al., 2022).
They propose a three-step process to induce a distilled metric from MoverScore
and BERTScore as teachers, considering three BERT variants (Tiny, Small,
Medium). They show that they can obtain metrics that retain 97% of performance
while allegedly running 24 times faster (which seems to result from one model
being more than 100 times faster, while most efficiency improvements seem to
be speedups of factors 2-3). Our analysis is much more comprehensive than
theirs, in terms of metrics investigated and efficient transformers explored.
Our results also show that simply replacing the transformer with an efficient
one may already yield comparable or better results than inducing a metric by
distillation and training on synthetic data.

In the context of trained metrics, i.e., metrics that are supervised on human
signals such as direct assessment scores, Rei et al. (2022); Gajbhiye et al.
(2021) also explore knowledge distillation, like FrugalScore, in combination
with other techniques. Belouadi and Eger (2022) go the opposite way and
explore efficiency of metrics in terms of reducing the amount of supervision
required (i.e., no references, no human supervision signals, no parallel data
for inducing metrics).

##  3 Approach

For optimizing the metrics, we explore two approaches: (i) We replace
transformers with smaller and more efficient variants in §3.1 and (ii) we
accelerate token matching by calculating WCD and RWMD instead of WMD for
MoverScore and XMoverScore in §3.2.

###  3.1 Replacing transformer models

We replace standard BERT encoders in evaluation metrics with pruned,
distilled, and dynamically accelerated encoders as described below.

Pruning: We build BERTScore, MoverScore and BaryScore variants using a pruned
BERTTINY from Turc et al. (2020). Distillation: We use DistilBERT (Sanh et
al., 2019) and TinyBERT4 (Jiao et al., 2020) for BERTScore, MoverScore and
BaryScore; dBART (Shleifer and Rush, 2020) for BARTScore; DistilMBERT (Sanh et
al., 2019) and XtremeDistil (Mukherjee and Hassan Awadallah, 2020) for
XMoverScore and SentSim; DistilGPT-2 (von Platen, 2022) for XMoverScore and
mMiniLM (Wang et al., 2021) for XMoverScore, SentSim, COMET and TransQuest.
Dynamic Inference Acceleration: For BERTScore, MoverScore and BaryScore, we
build a version using DeeBERT’s early exiting (Xin et al., 2020).

###  3.2 Replacing WMD

Our second approach for building more efficient metrics is speeding up the
token matching in metrics. Our focus is on Word Mover’s Distance (WMD) in
MoverScore, as it has been a popular approach for token matching in evaluation
metrics (Zhao et al., 2019, 2020; Colombo et al., 2021). WMD was proposed by
Kusner et al. (2015) as a special version of Earth Mover’s Distance (EMD)
(Rubner et al., 1998) applied to word embeddings. WMD calculates the cheapest
solution of moving words in one document to the words in another. To do so, it
solves a constrained optimization problem with two constraints. WMD is costly
in computation speed, as it has a cubic inference complexity.

##### WCD

Rubner et al. (1998) proposed a linear complexity loose lower bound of WMD
which Kusner et al. (2015) call Word Centroid’s Distance (WCD). To calculate
the distance between documents, WCD first calculates their centroids, i.e. the
center or average of their word vectors. Then the Euclidean distance between
the centroids of these documents is calculated:

$\textit{WCD}(x,y)=\sqrt{\frac{1}{|x|}\sum_{i=1}^{|x|}E(x_{i})-\frac{1}{|y|}\sum_{j=1}^{|y|}E(y_{j})}$
(1)

In Eq. (1), $x$ and $y$ are two documents compared and $E$ is an embedding
function.

##### RWMD

Kusner et al. (2015) also propose the much tighter Relaxed Word Mover’s
Distance (RWMD), which removes one of the two constraints in the optimization
problem. Given the distance of every word in the first document to every word
in the other document, RWMD can be calculated with a quadratic complexity.

##  4 Experimental Setup

Following related work, we measure the success of our optimized metrics based
on the time needed to calculate them, the memory used and the storage needed
to save the program or related data.

###  4.1 Evaluation Protocol (disk space, inference time, quality)

We measure runtime, memory usage and model size as efficiency measures and
contrast them with Pearson’s $r$ as a quality measure of the metric
(correlation with human assessments).

##### Runtime

To calculate the execution/inference time, we measure the timestamp
immediately before starting inference and immediately after ending, then
report the difference. Since the computing speed of a system depends on
factors such as hardware and especially scheduling of tasks by the operating
system, runtimes vary from one run to the next, even when all internal
variables stay the same. To reduce this variation, we run every experiment at
least three times and average the measured runtimes. For comparability between
different metrics, we set the batch size of each metric to 1. For
comparability between datasets, we divide the total runtime of a metric by the
number of segments.

##### Memory usage, parameters and disk-space

During inference, we measure the peak memory usage of the program. Since the
metrics use the transformers library built on pytorch, we use pytorch’s
memory_stats array to get the peak usage. We further list the number of
parameters in a model and the size needed to save it on a hard drive.

###  4.2 Data

We use WMT15, WMT16, and WMT21, all published by the annual Machine
Translation conference WMT. The organizers provide source texts, machine
translations and human references. The texts are from various categories, but
we only use the ones from newstest. The organizers also publish human
assessments, which we correlate with our metrics’ output as a quality measure.
For WMT15 and WMT16, the human scores are Direct Assessment (DA) (Stanojević
et al., 2015; Bojar et al., 2016), while for WMT21, they follow the
Multidimensional Quality Metrics (MQM) framework (Freitag et al., 2021).

pair | WMT15 | WMT16 | WMT21
---|---|---|---
cs-en | 500 | 560 |
de-en | 500 | 560 |
fi-en | 500 | 560 |
ro-en |  | 560 |
ru-en | 500 | 560 |
tr-en |  | 560 |
zh-en |  |  | 3250
total | 2000 | 3360 | 3250
Table 1: Overview of amount of segments per language pair in each dataset.

WMT15 provides scored data in 5 language pairs (4 of which are to-English)
with 500 segments each, in total 2000 segments. WMT16 provides scored data in
7 language pairs (6 of which are to-English) with 560 segments each, in total
3360 segments. For WMT21, scored data is available for three language pairs
(en-de, en-ru and zh-en). For the only to-English language pair of those, WMT
provides scores for 650 segments (the same segments for each system). To speed
up analyses, we reduce the data to the outputs of only five MT systems (DIDI-
NLP, Facebook-AI, MiSS, NiuTrans and SMU), in total 3250 segments. An overview
of language pairs and amount of segments available can be found in Table 1.

###  4.3 Hardware

To get results that are less dependent on our hardware, we consider two
different setups: (i) Virtual Machine on a personal computer: The first setup
is a virtual machine on a personal computer. The VM has a Intel Core i5-10310U
CPUs (4 cores, 1.70GHz), a main memory of 10GB, a swap memory of 8GB and runs
Xubuntu 20.04.2 LTS (64bit) as OS. This setup does not have a GPU that can be
used for calculations. (ii) Compute Cluster: The second setup is the Compute
Cluster of a computer science department. For our experiments, we chose an 4
Intel Xeon Gold 5218R CPUs (20 cores each, 2.10GHz), with a main memory of
64GiB, 8 Nvidia A100 GPUs with 40GB memory each and runs CentOS Linux 7 as OS.
Using the Slurm Workload Manager (version 20.02.2) we limited the usable CPU
cores to 4 and GPUs to 1 during our experiments.

For each experiment, we report the runtime on CPU (as an average of setup 1
and setup 2) and on GPU (only from setup 2).

##  5 Experiments

We structure our experiments as follows: We report results for (1) reference-
based metrics (§5.1), (2) reference-free multilingual metrics (§5.2) and then
(3) consider WMD approximations for one of the metrics, MoverScore (§5.3).

###  5.1 Reference-Based Metrics

We first consider the “semantic similarity” metrics MoverScore, BERTScore and
BaryScore, then consider BARTScore which is based on text generation and
therefore uses different transformer types and different corresponding
efficient variants.

####  5.1.1 BERTScore, MoverScore, BaryScore

BERTScore, MoverScore and BaryScore use monolingual BERT based transformers
for embedding reference and hypothesis. We replace the embedding model with a
pruned model, two distilled models and an early exiting model.

RoBERTaLARGE. Although BERTScore was built using various models, for comparing
sentences in English (as both our references and hypotheses are), the authors
suggest using RoBERTaLARGE (Zhang et al., 2020). We only use it on BERTScore,
since it was too slow with MoverScore and BaryScore. Its configuration is:
$L=24,H=1024,A=16$ (Liu et al., 2020), where $L$ is the number of layers, $H$
the size of each layer and $A$ stands for the attention
heads.222https://huggingface.co/roberta-large BERTBASE is the original model
of MoverScore and BaryScore (Zhao et al., 2019; Colombo et al., 2021) and also
a possible optimization for BERTScore. It has $L=12$, $H=768$ and $A=12$
(Devlin et al., 2019).333https://huggingface.co/bert-base-uncased BERTTINY is
the smallest variant of BERT (Turc et al., 2020) and was trained in the
traditional way (directly on data, no Knowledge Distillation). It has $L=2$,
$H=128$, $A=2$.444https://huggingface.co/google/bert_uncased_L-2_H-128_A-2
DistilBERT is a distillation of BERTBASE. It has the same hidden dimensions of
768 and 12 attention heads, but only $L=6$. Also, they optimized the final
output layers (Sanh et al., 2019).555https://huggingface.co/distilbert-base-
uncased TinyBERT4 is another distillation of BERTBASE with a a more robust
training method. It has $L=4$, $H=312$, and $A=12$ (Jiao et al.,
2020).666https://huggingface.co/huawei-noah/TinyBERT_General_4L_312D
DeeBERTMNLI is an early exiting version of BERTBASE. It has the same
structure, but after each encoder layer there is one added classification
layer, that can be used as an off-ramp, to stop inference at intermediate
states (Xin et al., 2020).777https://huggingface.co/ji-xin/bert_base-MNLI-
two_stage

##### Results

We visualize the runtime of the inference on a CPU and the quality achieved by
the metrics in Figure 1. We observe a speedup of the runtime that coarsely
correlates with the size of the model. The figure shows that the fastest model
on each of the metrics is BERTTINY, which is up to 41x faster than the
baseline (on BERTScore), but its quality also decreases by over 10 points
correlation for BERTScore. A better speedup-quality ratio achieves TinyBERT4:
The quality decreases by less than 6 points across all metrics (improves even
for MoverScore). Furthermore, it is still up to 27x faster than the baseline
(on BERTScore). Memory measurements show similar behavior to runtimes, they
coarsely correlate with model size. BERTTINY uses 61x and TinyBERT4 18x less
memory than the baseline on BERTScore. Compared to TinyBERT4, DistilBERT shows
lower speedup and memory saving, but also a lower quality decrease (on all
three metrics approximately half the decrease of TinyBERT4). As expected,
BERTBASE in BERTScore is faster and more memory-efficient than RoBERTaLARGE,
but not as efficient as above mentioned models. In our experiments, DeeBERT
behaves very similar to BERTBASE, both in quality and efficiency.

Further, we observe that speedups – no matter on which metric, model or
dataset – are usually not as big on GPU as on CPU (more details on that in
§6).

##### BARTScore

originally uses BARTLARGE fine-tuned on CNNDM and Parabank2 (Yuan et al.,
2021). Optimizations of BART were researched by Shleifer and Rush (2020). The
models with the best results from them use a Shrink and Fine-Tune approach and
were also trained on CNNDM. Details follow now.

$50$$100$$150$$200$$250$$300$$350$$400$$450$$500$$550$$600$$650$$700$$750$$0.48$$0.5$$0.52$$0.54$$0.56$$0.58$RoBERTaLARGEBERTBASEBERTTINYDistilBERTTinyBERT4DeeBERTMNLIBERTBASEBERTTINYDistilBERTTinyBERT4DeeBERTMNLIBERTBASEBERTTINYDistilBERTTinyBERT4DeeBERTMNLIruntime
in ms/segmentPearson’s r correlation with human
evaluationBERTScoreMoverScoreBaryScore Figure 1: CPU runtime / correlation
plot of BERTScore, MoverScore, BaryScore

BARTLARGE Para is the original model used in BARTScore, fine-tuned on the
Parabank2 dataset. It consists of 12 encoder and 12 decoder layers and has a
hidden size of $H=1024$ (Lewis et al., 2020). BARTLARGE CNN uses the same
architecture, but after pre-training on CNNDM, it was not fine-tuned on
Parabank2. Since the smaller BART models could not be fine-tuned due to
resource limitations, we run experiments on BARTLARGE CNN and used these
results as a baseline for fair comparisons to the other models. BARTBASE was
proposed by Lewis et al. (2020) and differs from BARTLARGE in having 6 encoder
and decoder layers instead of 12. It has a hidden size of 768. dBART-6-6 is a
distilled BART version with 6 encoder layers and 6 decoder layers. It was
shrunk from BARTLARGE and therefore has a hidden size of 1024 (Shleifer and
Rush, 2020).888https://huggingface.co/sshleifer/distilbart-cnn-6-6 dBART-12-3
is a distilled BART version with 12 encoder layers and 3 decoder layers. It
was shrunk from BARTLARGE and also has a hidden size of 1024 (Shleifer and
Rush, 2020).999https://huggingface.co/sshleifer/distilbart-cnn-12-3
dBART-12-6-t is a model from the Hugginface Model
Hub.101010https://huggingface.co/shahrukhx01/distilbart-cnn-12-6-text2sql It
has 12 encoder layers and 6 decoder layers and was trained on WikiSQL dataset.
dBART-12-9-m is also a model from the Huggingface Model
Hub.111111https://huggingface.co/valhalla/distilbart-mnli-12-9 It has 12
encoder layers and 9 decoder layers.

##### Results

We observe differences among the baselines: BARTLARGE Para uses almost twice
the memory of BARTLARGE CNN. The fastest model is again the smallest model –
BARTBASE – with 213ms on CPU and 20.2ms on GPU, which are speedups of 2.7x and
1.8x compared to BARTLARGE CNN. BARTBASE is also the most memory-efficient
model: with a usage of 663MB, it needs 2.6x less than BARTLARGE CNN. Despite
being very efficient, BARTBASE’s quality declines too much, with a correlation
coefficient 12 points lower than the baseline (-25%). Concerning quality,
dBART-6-6 even gets a higher correlation than BARTLARGE CNN by 0.02 (+3.6%).
It brings a speedup of 1.8x (CPU) and 1.7x (GPU) and is 1.7x more memory
efficient. dBART-12-3, dBART-12-6-t and dBART-12-9-m do not achieve
competitive quality, also their acceleration over BARTLARGE CNN is only
moderate.

###  5.2 Reference-Free Metrics

XMoverScore (Zhao et al., 2020) and SentSim (Song et al., 2021) are reference-
less metrics,121212When running SentSim we use the implementation from
Belouadi and Eger (2022), since it is better structured and integrates better
with our evaluation framework. i.e., they compare hypotheses directly to
source texts. Therefore, both of them use a multilingual embedding model,
since source and hypothesis are in different languages in MT. Furthermore both
of them use a second model, which we will optimize in §5.2 and §5.2.

Zhao et al. (2020) realign multilingual embedding spaces on parallel sentences
using fast-align (Dyer et al., 2013). We use WikiMatrix, instead, to train
remappings because it contains more languages. For the baseline, we use the
original remappings from Zhao et al. (2020). We report Pearson’s r with
(remap) and without remapping (direct). To implement efficient reference-less
metrics, we explore the following multilingual embedding models:

mBERT is the original model of XMoverScore used by Zhao et al.
(2020).131313https://huggingface.co/bert-base-multilingual-cased It has
$L=12$, $H=768$, and $A=12$. It was trained on 104 languages (Devlin et al.,
2019). XLM-RBASE is the original model of SentSim used by Song et al. (2021).
It has $L=12$, $H=768$ and was trained on data in 100 languages (Conneau et
al., ).141414https://huggingface.co/xlm-roberta-base DistilMBERT is a
distillation of mBERT (Sanh et al.,
2019).151515https://huggingface.co/distilbert-base-multilingual-cased It has
the same dimensions and attention heads, but only $L=6$ layers and the final
output layers were stripped. These missing output layers are what makes this
model incompatible to XMoverScore. Thus, we only use this model for SentSim.
XtremeDistil is another distillation of mBERT (Mukherjee and Hassan Awadallah,
2020).161616https://huggingface.co/microsoft/xtremedistil-l6-h256-uncased The
model, called TinyMBERT in the first version of the paper, has $L=6$, $H=256$
and $A=12$. mMiniLM6 is a distillation of XLM-RLARGE with $L=6$, $H=384$, and
$A=12$. mMiniLM12 is a distillation of XLM-RLARGE with $L=12$ (Wang et al.,
2021).

$50$$100$$150$$200$$250$$300$$350$$400$$0.36$$0.38$$0.4$$0.42$$0.44$$0.46$mBERT
XLM-RBASEXtremeDistilmMiniLM6mMiniLM12XLM-RBASE
mBERTDistilMBERTXtremeDistilmMiniLM6mMiniLM12runtime in ms/segmentPearson’s r
correlation with human evaluationXMoverScoreSentSim Figure 2: CPU runtime /
correlation plot of XMoverScore and SentSim

##### Results

We present the results in Figure 2. Both mMiniLMs outperform their baselines
in quality and efficiency. The 6 layer version is up to 2.1x faster (on CPU)
than the baseline and has a 0.024 higher correlation for XMoverScore (+5.8%).
Even higher is the quality improvement with mMiniLM12: it achieves a 0.032
higher correlation than the baseline on XMoverScore (+8.4%). Both models also
show quality improvements on SentSim. The space they occupy on a disk is $1/5$
of XLM-RBASE and $1/3$ of mBERT. Although the needed memory is a lot higher
than the disk-space, with up to 1,594MB (mMiniLM12 on SentSim), the models
still need 1.2x (SentSim) and 1.4x (XMoverScore) less inference time than the
baselines. DistilMBERT on SentSim also shows speedups of 1.2x on both CPU and
GPU, but has a 0.034 lower Pearson’s r than XLM-RBASE. XtremeDistil, despite
bringing some memory-efficiency and big saving on disk-space, has a quality
that is too bad on both metrics.

The remappings on XMoverScore do not bring any difference for mBERT and XLM-
RBASE. For the mMiniLMs, we observe a quality increase of 1.6 points (6 layer)
and 1.3 points (12 layer) correlation compared to using no remapping. Only for
XtremeDistil do we see a real difference: using the remappings improves
correlation by approximately 0.122 (+110%) compared to using the model
directly.

##### XMoverScore’s language model

XMoverScore also uses a language model for calculating the perplexity of the
hypothesis. We also replace this model with a lighter one and measure the
efficiency and quality changes. The two compared models are: GPT-2 (Radford et
al., 2019) is the original model used by XMoverScore (Zhao et al.,
2020).171717https://huggingface.co/gpt2 DistilGPT-2 is a distillation of GPT-2
(von Platen, 2022).

##### Results:

We can not see any change in runtime on a CPU, but observe a speedup of 1.3x
on a GPU. We also see a lower memory usage of 317MB (20%) and a lower disk-
space of a third. The quality drops by 3.5 points.

##### SentSim’s sentence embedding model

Besides a word level model, SentSim calculates the cosine distance of sentence
embeddings (Reimers and Gurevych, 2019, 2020) of the source and hypothesis
sentence. We replace this sentence embedding model with a lighter one. We try
3 other models from the SBERT framework. For reasons of clarity, we abbreviate
the names to a letter from the SBERT site: A: xlm-r-bert-base-nli-stsb-mean-
tokens, the original model used by Song et al. (2021). B: distiluse-base-
multilingual-cased-v2. C: paraphrase-multilingual-MiniLM-L12-v2. D:
paraphrase-multilingual-mpnet-base-v2.

##### Results:

We observe a slight speedup of Model B of 1.2x (CPU) and 1.3x (GPU), but also
a decrease of quality of 5.5 points. Model C shows only small decrease of
quality of 1.7 points, but is very efficient: it runs in 1.5x faster speed on
CPU and 1.1x faster on GPU, saves memory (1.4x less) and also only occupies
43% of the disk-space. Model D achieves a slightly higher correlation (+1.2
points) and has the same size as the baseline (comparable speed and memory).

###  5.3 WMD

We replace WMD in MoverScore and XMoverScore with more efficient variants to
speed up runtime. We implement the two variants WCD and RWMD, which have
linear (WCD) and quadratic (RWMD) complexity.

We see a drop in quality from a correlation of 0.54 on average over all three
datasets to 0.43 and 0.39, but, surprisingly, we see a substantial increase of
quality when using RWMD on WMT21. Concerning quality of XMoverScore, we also
see a decline in correlation with human scores. WCD achieves correlations
approximately 10 percent lower than WMD. As on MoverScore, RWMD’s correlations
drop approximately 30 percent for WMT15 and WMT16. For WMT21, we again observe
a very high correlation with RWMD.

##### Runtime

For neither MoverScore nor XMoverScore do we observe a substantial speedup
while using WCD or RWMD instead of WMD. Thus, we investigate the time
consumption of each calculation step in more detail.

Step |  WMD |  WCD |  RWMD
---|---|---|---
get BERT embeddings |  285.499 |  287.915 |  291.122
calculate distance matrix |  0.829 |  0.005 |  0.782
calculate distance |  5.602 |  0.616 |  0.449
Table 2: Runtime (in ms) of each step of MoverScore for various distance
functions using its original model BERTBASE.

In Table 2, we can see that the calculation of WCD and RWMD is significantly
faster than WMD– 9x and 12x on MoverScore and 20x and 27x on XMoverScore. But
it also shows that the calculation of the embeddings (and of the perplexity)
takes a significantly longer time than the calculation of the distance. No
matter how fast the distance can be calculated, the speedup will be eaten up
by the variance of the calculation time for the embeddings (and perplexity).

##  6 Analysis & Discussion

We conduct a deeper analysis on reference-based “semantic similarity” metrics
BERTScore, MoverScore and BaryScore, all of which use the same efficient
encoder architectures (we remove Roberta-large from the analysis, as it has
only been used with BERTScore). We compute results individually across the
three metrics and the three WMT datasets (then report individual results or
averages).

##### Which speedups are obtained?

Table 3 shows absolute speedups of encoders and their relative performance
deterioration relative to BERT. On a CPU, TinyBERT is 5.4x faster at
interference while retaining 97% of the quality of the original metric, which
yields the best tradeoff of quality and inference time. BERT${}_{\text{TINY}}$
yields almost 9x faster inference but at a 12 points performance
deterioration.

Model | Quality | Runtime
---|---|---
|  | (CPU)
BERT${}_{\text{BASE}}$ | 1.00 | 1.00
BERT${}_{\text{TINY}}$ | 0.88 | 8.95
DistilBERT | 0.99 | 1.63
TinyBERT | 0.97 | 5.42
DeeBERT${}_{\text{MNLI}}$ | 0.96 | 1.06
Table 3: Quality-inference values.

##### How stable are the results?

We correlate stability of (normalized) results using Pearson correlation over
vectors corresponding to (a) metrics and (b) datasets. Concerning metrics, we
observe high Pearson correlations from 0.72 (MoverScore-BaryScore, Quality) to
0.99 (MoverScore-BERTScore, inference time). This means that the the encoders
perform similarly (in terms of quality, inference time or both) for each
metric. Across datasets, the correlation is high for WMT15-WMT16 (0.91-0.99
for quality, runtime, and average) but considerably lower between WMT21 and
the other two (0.20-0.96). This is mainly because quality is not stable: for
example, DeeBERT performs badly for WMT21, but quite good for WMT15 and WMT16.
We note however, that the score ranges for WMT21 are low anyway, and
absolutely DeeBERT does not perform much lower than the other encoders here.

##### Correlations between measured values

For calculating the correlation matrices between our 6 measured values
(runtime on CPU and GPU, memory consumption, number of parameters, disk-space,
quality of model), we calculate separate correlation matrices for each
individual evaluation metric and then average those correlations, see Table 4
for the results. (We note that we correlate vectors of size 4 to 12).

|  CPU runtime |  GPU runtime |  Mem. |  Params |  Disk-space
---|---|---|---|---|---
GPU runtime |  0.831 |  |  |  |
Memory |  0.913 |  0.786 |  |  |
Params |  0.946 |  0.788 |  0.977 |  |
Disk-space |  0.927 |  0.772 |  0.984 |  0.963 |
Quality (r) |  0.620 |  0.590 |  0.713 |  0.694 |  0.638
Table 4: Pearson correlation matrix of the measured values.

We find that CPU runtime highly correlates with the amount of parameters
(0.95) and the space on disk (0.93). Runtime on GPU also correlates with the
model size (0.79 and 0.77), but not as much as on a CPU. We have deeper look
into the differences between CPU and GPU runtimes below. The memory needed
shows a very strong correlation with the size of the model on disk (0.98). The
quality of a metric does not very strongly correlate with efficiency measures
(only 0.6-0.7). This is because sometimes smaller and more efficient models
achieve better results than bigger ones.

##### Difference between CPU and GPU runtime

(a) CPU
runtime$50$$100$$150$$200$$250$$300$$350$$400$$0.49$$0.5$$0.51$$0.52$$0.53$$0.54$$0.55$$0.56$$0.57$$0.58$$0.59$RoBERTaLARGEBERTBASEBERTTINYDistilBERTTinyBERT4BERTBASEBERTTINYDistilBERTTinyBERT4ABCfs-
tiny-rfs-small-rfs-medium-druntime in ms/segmentPearson’s r correlation with
human evaluationBERTScoreMoverScoreEnsemblesFrugalScore

(b) GPU
runtime$5$$10$$15$$20$$25$$30$$35$$40$$45$$0.49$$0.5$$0.51$$0.52$$0.53$$0.54$$0.55$$0.56$$0.57$$0.58$$0.59$RoBERTaLARGEBERTBASEBERTTINYDistilBERTTinyBERT4BERTBASEBERTTINYDistilBERTTinyBERT4fs-
tiny-rfs-small-rfs-medium-druntime in ms/segmentPearson’s r correlation with
human evaluation

Figure 3: Selected metrics in runtime / correlation plots on a CPU and GPU.

We notice several differences in runtimes on a CPU and GPU, usually the CPU
speedup is higher than the one on GPU. For example, when comparing BERTScore
using TinyBERT4 to its baseline model (RoBERTaLARGE), the runtime on CPU gets
faster 27 times, but on GPU only 2.2 times.

In their survey on compressing transformers, Ganesh et al. (2021) argue that
this is because of GPUs being optimized for accelerating linear calculations
such as those in the feed-forward layers in a transformer. The self-attention
heads are implemented in several matrix transformations and multiplications
and hence costly in runtime. Therefore, CPUs calculate them in linear runtime
(correlation with parameters = 0.95), but GPUs struggle with those complex
structures when the model is compressed. From another angle, the bigger a
model is, the higher is the relative acceleration by a GPU over a CPU.

##### Quality of efficient metrics

The reasons for some compressed models being better than the larger ones
presumably lies in the training method: Hoffmann et al. (2022) argue that most
recently published large language models are undertrained because they only
focus on scaling the size of the model to achieve better performance,
following scaling laws (Kaplan et al., 2020). However, Hoffmann et al. (2022)
argue that training is equally important as size, but data often is scarce
when training a large model. That is why Knowledge Distillation achieves such
good results, because it creates training data with the larger teacher model.

##### Comparison to FrugalScore

FrugalScore’s best pre-trained model achieves a Pearson’s r of 0.580 on
average and is calculated very fast. In Figure 3(b), we see that their models
are faster than our optimized metrics on a GPU; the best models are fs-small-r
and fs-medium-d. However, Figure 3(a) shows that on a CPU their runtimes is a
lot slower. There, all our optimized metrics are faster than fs-small-r and
fs-medium-r, the best speed-quality ratios achieve MoverScore with TinyBERT4.

##  7 Conclusion

We have investigated efficient evaluation metrics for natural language
generation, particularly MT, in monolingual reference-based and multilingual
reference-free approaches via two approaches: (i) replacing transformers in
metrics by efficient variants, (ii) replacing alignment models in metrics
(precisely: Word Mover Distance) with efficient approximations. We have
explored multiple types of efficient transformers, finding that TinyBERT shows
the best quality-efficiency tradeoff for semantic similarity based metrics: on
average, it retains 97% quality while being more than 5x faster at inference
time and having considerable fewer parameters and lower memory consumption. In
several cases, we have also identified faster models that yield higher quality
at the same time. Finally, we found that efficient alignments on top of
transformers do not result in efficiency gains but have adverse effects on
quality in 2 out of 3 datasets we examined.

In future work, we want to combine efficiency with other highly desirable
properties of evaluation metrics such as _robustness_ (Vu et al., 2022; Chen
and Eger, 2022; Rony et al., 2022) and _explainability_ (Kaster et al., 2021;
Sai et al., 2021; Fomicheva et al., 2021; Leiter et al., 2022) to induce
metrics that jointly satisfy these criteria as well as examine ways to make
_training_ in trained metrics more efficient, besides addressing inference
time.

## References

  * Belouadi and Eger (2022) Jonas Belouadi and Steffen Eger. 2022.  Uscore: An effective approach to fully unsupervised evaluation metrics for machine translation.  _ArXiv_ , abs/2202.10062.
  * Bojar et al. (2016) Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016\.  Results of the WMT16 metrics shared task.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 199–231, Berlin, Germany. Association for Computational Linguistics.
  * Celikyilmaz et al. (2020) Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020.  Evaluation of text generation: A survey.  _ArXiv_ , abs/2006.14799.
  * Chen and Eger (2022) Yanran Chen and Steffen Eger. 2022.  Menli: Robust evaluation metrics from natural language inference.  _ArXiv_ , abs/2208.07316.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.
  * Colombo et al. (2021) Pierre Colombo, Guillaume Staerman, Chloé Clavel, and Pablo Piantanida. 2021\.  Automatic text evaluation through the lens of Wasserstein barycenters.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10450–10466, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * (7) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013.  A simple, fast, and effective reparameterization of IBM model 2.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 644–648, Atlanta, Georgia. Association for Computational Linguistics.
  * Fomicheva et al. (2021) Marina Fomicheva, Piyawat Lertvittayakumjorn, Wei Zhao, Steffen Eger, and Yang Gao. 2021.  The Eval4NLP shared task on explainable quality estimation: Overview and results.  In _Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems_ , pages 165–178, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Freitag et al. (2021) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021.  Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 733–774. Association for Computational Linguistics.
  * Gajbhiye et al. (2021) Amit Gajbhiye, Marina Fomicheva, Fernando Alva-Manchego, Frédéric Blain, Abiola Obamuyide, Nikolaos Aletras, and Lucia Specia. 2021.  Knowledge distillation for quality estimation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 5091–5099, Online. Association for Computational Linguistics.
  * Ganesh et al. (2021) Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, and Marianne Winslett. 2021.  Compressing large-scale transformer-based models: A case study on BERT.  _Transactions of the Association for Computational Linguistics_ , 9:1061–1080.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022.  Training compute-optimal large language models.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  TinyBERT: Distilling BERT for natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4163–4174, Online. Association for Computational Linguistics.
  * Kamal Eddine et al. (2022) Moussa Kamal Eddine, Guokan Shang, Antoine Tixier, and Michalis Vazirgiannis. 2022\.  FrugalScore: Learning cheaper, lighter and faster evaluation metrics for automatic text generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1305–1318, Dublin, Ireland. Association for Computational Linguistics.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.
  * Kaster et al. (2021) Marvin Kaster, Wei Zhao, and Steffen Eger. 2021.  Global explainability of BERT-based evaluation metrics by disentangling along linguistic factors.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8912–8925, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kusner et al. (2015) Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015.  From word embeddings to document distances.  In _Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37_ , ICML’15, page 957–966. JMLR.org.
  * Leiter et al. (2022) Christoph Leiter, Piyawat Lertvittayakumjorn, M. Fomicheva, Wei Zhao, Yang Gao, and Steffen Eger. 2022.  Towards explainable evaluation metrics for natural language generation.  _ArXiv_ , abs/2203.11131.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Ro{bert}a: A robustly optimized {bert} pretraining approach.
  * Marie et al. (2021) Benjamin Marie, Atsushi Fujita, and Raphael Rubino. 2021.  Scientific credibility of machine translation research: A meta-evaluation of 769 papers.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7297–7306, Online. Association for Computational Linguistics.
  * Mukherjee and Hassan Awadallah (2020) Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020.  XtremeDistil: Multi-stage distillation for massive multilingual models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2221–2234, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Rei et al. (2022) Ricardo Rei, Ana C Farinha, José G.C. de Souza, Pedro G. Ramos, André F.T. Martins, Luisa Coheur, and Alon Lavie. 2022.  Searching for COMETINHO: The little metric that could.  In _Proceedings of the 23rd Annual Conference of the European Association for Machine Translation_ , pages 61–70, Ghent, Belgium. European Association for Machine Translation.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4512–4525, Online. Association for Computational Linguistics.
  * Rony et al. (2022) Md Rashad Al Hasan Rony, Liubov Kovriguina, Debanjan Chaudhuri, Ricardo Usbeck, and Jens Lehmann. 2022.  RoMe: A robust metric for evaluating natural language generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5645–5657, Dublin, Ireland. Association for Computational Linguistics.
  * Rubner et al. (1998) Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. 1998.  A metric for distributions with applications to image databases.  _Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271)_ , pages 59–66.
  * Sai et al. (2021) Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. 2021.  Perturbation CheckLists for evaluating NLG evaluation metrics.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7219–7234, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * Shleifer and Rush (2020) Sam Shleifer and Alexander M. Rush. 2020.  Pre-trained summarization distillation.  _ArXiv_ , abs/2010.13002.
  * Song et al. (2021) Yurun Song, Junchen Zhao, and Lucia Specia. 2021.  SentSim: Crosslingual semantic evaluation of machine translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3143–3156, Online. Association for Computational Linguistics.
  * Stanojević et al. (2015) Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015.  Results of the WMT15 metrics shared task.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 256–273. Association for Computational Linguistics.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019.  Patient knowledge distillation for BERT model compression.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4323–4332, Hong Kong, China. Association for Computational Linguistics.
  * Turc et al. (2020) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2020.  Well-read students learn better: On the importance of pre-training compact models.
  * von Platen (2022) Patrick von Platen. 2022.  DistilGPT2.
  * Vu et al. (2022) Doan Nam Long Vu, Nafise Sadat Moosavi, and Steffen Eger. 2022.  Layer or representation space: What makes bert-based evaluation metrics robust?
  * Wang et al. (2021) Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021.  MiniLMv2: Multi-head self-attention relation distillation for compressing pretrained transformers.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2140–2151, Online. Association for Computational Linguistics.
  * Xin et al. (2020) Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020.  DeeBERT: Dynamic early exiting for accelerating BERT inference.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2246–2251, Online. Association for Computational Linguistics.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 27263–27277. Curran Associates, Inc.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao et al. (2020) Wei Zhao, Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020.  On the limitations of cross-lingual encoders as exposed by reference-free machine translation evaluation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1656–1671, Online. Association for Computational Linguistics.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Hong Kong, China. Association for Computational Linguistics.
  * Zhu (2021) Wei Zhu. 2021.  LeeBERT: Learned early exit for BERT with cross-level optimization.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2968–2980, Online. Association for Computational Linguistics.


# First evidence of diffuse ultra-steep-spectrum radio emission surrounding
the cool core of a cluster

F. Savini1, A. Bonafede1,2, M. Brüggen1, R. van Weeren3, G. Brunetti2, H.
Intema3, A. Botteon2,4 T. Shimwell3,5, A. Wilber1, D. Rafferty1, S.
Giacintucci6, R. Cassano2, V. Cuciti2, F. de Gasperin4, H. Röttgering3, M.
Hoeft7, G. White8,9
1 Hamburger Sternwarte, Universität Hamburg, Gojenbergsweg 112, 21029,
Hamburg, Germany.
2 INAF IRA, via P. Gobetti 101, 40129 Bologna, Italy.
3 Leiden Observatory, Leiden University, PO Box 9513, 2300 RA Leiden, The
Netherlands.
4 Dipartimento di Fisica e Astronomia, Universita’ di Bologna, via P. Gobetti
93/2, 40129 Bologna, Italy.
5 ASTRON, the Netherlands Institute for Radio Astronomy, Postbus 2, 7990 AA,
Dwingeloo, The Netherlands.
6 Naval Research Laboratory, 4555 Overlook Avenue SW, Code 7213, Washington,
DC 20375, USA.
7 Thüringer Landessternwarte, Sternwarte 5, 07778 Tautenburg, Germany.
8 Department of Physics & Astronomy, The Open University, Walton Hall, Milton
Keynes MK7 6AA, UK.
9 Space Science & Technology Department, CCLRC Rutherford Appleton Laboratory,
Chilton, Didcot, Oxfordshire OX11 0QX, UK E-mail<EMAIL_ADDRESS>hamburg.de

(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

Diffuse synchrotron radio emission from cosmic-ray electrons is observed at
the center of a number of galaxy clusters. These sources can be classified
either as giant radio halos, which occur in merging clusters, or as mini
halos, which are found only in cool-core clusters. In this paper, we present
the first discovery of a cool-core cluster with an associated mini halo that
also shows ultra-steep-spectrum emission extending well beyond the core that
resembles radio halo emission. The large-scale component is discovered thanks
to LOFAR observations at 144 MHz. We also analyse GMRT observations at 610 MHz
to characterise the spectrum of the radio emission. An X-ray analysis reveals
that the cluster is slightly disturbed, and we suggest that the steep-spectrum
radio emission outside the core could be produced by a minor merger that
powers electron re-acceleration without disrupting the cool core. This
discovery suggests that, under particular circumstances, both a mini and giant
halo could co-exist in a single cluster, opening new perspectives for particle
acceleration mechanisms in galaxy clusters.

###### keywords:

Galaxy clusters; non-thermal emission; particle acceleration; radio emission.
Galaxy clusters: individual: PSZ1G139.61+24.20

††pubyear: 2018††pagerange: First evidence of diffuse ultra-steep-spectrum
radio emission surrounding the cool core of a cluster–First evidence of
diffuse ultra-steep-spectrum radio emission surrounding the cool core of a
cluster

##  1 Introduction

An increasing number of diffuse radio sources associated with galaxy clusters
are being detected with the advent of new facilities and techniques at low
radio frequencies. Not only has the number of sources increased, but the
quality of imaging in terms of resolution and noise has also improved,
revealing various source morphologies and properties that might require a
broadening of the taxonomy of radio emission in galaxy clusters (e.g. de
Gasperin et al., 2017). Diffuse emission in the form of giant radio halos or
mini halos is found in the central regions of some galaxy clusters. These
sources have low surface brightnesses and steep radio spectra111The radio
spectrum follows a power law $S(\nu)\propto\nu^{\alpha}$, where $S$ is the
flux density, and $\nu$ the observing frequency. Steep spectrum radio sources
spectra have spectral indices $\alpha<-1$. that make them much brighter at
lower frequencies. The Low Frequency Array (LOFAR; van Haarlem et al., 2013)
operating between 30 and 240 MHz can discover steep-spectrum sources that are
impossible to detect with other radio telescopes.

Giant radio halos have typical sizes of 1 to 2 Mpc, and are predominantly
found in massive, merging clusters (e.g. Buote, 2001, Cassano et al., 2010,
Cuciti et al., 2015), suggesting that merger-driven turbulence re-accelerates
primary or secondary electrons in the ICM (Brunetti et al., 2007, Brunetti &
Lazarian, 2011, Pinzke et al., 2017). Mini halos have typical sizes of a few
hundred kpc and a higher emissivity than giant halos (Cassano et al., 2008,
Murgia et al., 2009). Mini halos are found exclusively in cool-core clusters,
and are confined within the inner regions of the cluster (Govoni et al., 2009,
Kale et al., 2015, Giacintucci et al., 2017). Cool-core clusters display a
peaked X-ray surface brightness and a significant drop in temperature
($<10^{7}-10^{8}$ K) at the centre. Signatures of minor mergers have been
detected in some cool-core clusters which host mini halos (Mazzotta &
Giacintucci, 2008). When the low-entropy central gas at the bottom of the dark
matter potential well is perturbed by a minor merger and meets the higher-
entropy ICM, a discontinuity in the X-ray emissivity, called a cold front, is
formed. The gas then falls back into the dark matter potential well and
“sloshes", possibly generating the turbulence that re-accelerates weakly
relativistic electrons within the core (ZuHone et al., 2013). Particle
acceleration by turbulence is an inefficient mechanism and according to
theoretical models only major mergers between massive clusters can dissipate
enough energy to power radio emission on Mpc scales up to GHz frequencies. So
far, it is unknown what happens when a cool-core cluster hosting a mini halo
undergoes a minor merger that does not disrupt the core. This scenario is
particularly interesting when observed at low radio frequencies, as it may
provide new insights on the connections between mini halos, giant radio halos,
and the cluster dynamics.

In this paper we report on the results of a LOFAR radio observation of the
galaxy cluster PSZ139139.61+24.20. We assume a flat, $\Lambda$CDM cosmology
with matter density $\Omega_{M}=0.3$ and Hubble constant $H_{0}=67.8$ km s-1
Mpc-1 (Planck Collaboration et al., 2016). The angular to physical scale
conversion at z = 0.267 is 4.137 kpc/′′. All our images are in the J2000
coordinate system.

Table 1: Properties of the galaxy cluster PSZ1G139.61+24.20 (Giacintucci et al., 2017). (1),(2),(3),(4): Target coordinates; (5): Redshift; (6): Global temperature computed within the radius enclosing a mean density of 2500 times the critical density at the cluster redshift. Note that the central region with a radius of 70 kpc was excised; (7): Core entropy; (8): Mass within the radius enclosing a mean density of 500 times the critical density Planck Collaboration et al., 2014; (9): Radius that encloses a mean overdensity of 500 with respect to the critical density at the cluster redshift. 1: RA (h:m:s) | 06:22:13.9
---|---
2: DEC (∘:′:′′) | +74:41:39.0
3: $l$ (∘) | 95.57
4: $b$ (∘) | 74.69
5: $z$ | $0.267$
6: kT (keV) | $7.5\pm 0.4$
7: K0 (keV cm2) | $<20$
8: M500 ($\times 10^{14}$ M⊙) | $7.1\pm 0.6$
9: R500 (Mpc) | 1.24

###  1.1 The cluster

PSZ1G139.61+24 (z = 0.267, RA = 06:22:13.9, DEC = +74:41:39.0; hereafter
PSZ139) has been classified as a galaxy cluster through detection of the
Sunyaev-Zel’dovoch effect with the Planck satellite (Planck Collaboration et
al., 2014). Using GMRT observations at 610 MHz, Giacintucci et al. (2017)
report the detection of a tentative mini halo with an overall source size of
$\sim 100$ kpc located at the cluster centre. They also present density and
temperature profiles derived from a Chandra X-ray observation of PSZ139.
Details can be found in Tab. 1. The cluster core is characterized by low
values of temperature, and the temperature profile inverts and starts
decreasing approaching the cluster centre within a radius of $\sim 100$ kpc
(see Fig.2 in Giacintucci et al. (2017)). The specific entropy (Cavagnolo et
al., 2009) at the cluster centre, $K_{0}$, is used to distinguish between
cool-core or non cool-core clusters (Giacintucci et al., 2017): clusters with
low central entropies ($K_{0}<30-50$ keV cm2) are expected to host a cool
core. The specific entropy of PSZ139 is $K_{0}<20$ keV cm2 (Giacintucci et
al., 2017), indicating that this cluster has a cool core.

##  2 Data reduction

Table 2: Col. 1: Telescope/Survey; Col. 2: Central frequency; Col. 3: Minimum baseline; Col. 4: Largest angular scale; Col. 5: Resolution; Col. 6: rms noise level; Col. 7: Parameters used for LOFAR and GMRT imaging, such as gaussian taper (T) and weighting scheme; when Briggs weighting scheme is used, the robust value is specified (Briggs, 1995). 1: Telescope | 2: Freq. | 3: $B_{min}$ | 4: $LAS$ | 5: Res. | 6: rms | 7: Imaging
---|---|---|---|---|---|---
| (MHz) | ($\lambda$) | (′) |  | ($\mu$Jy/beam) |
LOFAR | 144 | 80 | 43 | 5${}^{\prime\prime}\times$ 5′′ | 140 | Briggs -0.25
|  | 80 | 43 | 11${}^{\prime\prime}\times$ 8′′ | 150 | Briggs -0.25
|  | 80 | 43 | 35${}^{\prime\prime}\times$ 35′′ | 500 | Briggs 0, 20′′ T
|  | 200 | 17 | $18^{\prime\prime}\times 18^{\prime\prime}$ | 240 | uniform, 20′′ T
GMRT | 610 | 150 | 23 | $8.0^{\prime\prime}\times 4.7^{\prime\prime}$ | 27 | Briggs -0.1
|  | 150 | 23 | 32${}^{\prime\prime}\times 32^{\prime\prime}$ | 180 | Briggs 0, 20′′ T
|  | 200 | 17 | $20^{\prime\prime}\times 20^{\prime\prime}$ | 130 | uniform, 20′′ T

###  2.1 LOFAR radio observation

The cluster PSZ139 was observed as part of the LOFAR Two-Metre Sky Survey
(LoTSS; Shimwell et al., 2017) at High Band Antenna (HBA) frequencies (120 -
168 MHz). The observation was carried out on July 27, 2017 (ID LC8_022) with a
total on-source time of 8 h preceded and followed by a flux calibrator (3C295)
observation of 10 min. The calibration and imaging procedure is based on the
facet calibration scheme presented in van Weeren et al. (2016). A complete
outline of the procedure can be found in Savini et al. (2018); here we will
only briefly summarize the main steps:

  * •

Preliminary pre-processing was performed by the Radio Observatory (ASTRON) and
has been applied to the data;

  * •

Initial calibration was performed using the standard LOFAR direction
independent calibration pipeline222https://github.com/lofar-astron/prefactor;

  * •

Flagging was performed after inspecting the dataset; bad data were found and
flagged for a total of 30 min;

  * •

To refine the calibration, a pipelined version of the direction dependent
facet calibration procedure was used333https://github.com/lofar-astron/factor.

In facet calibration the field of view is divided up into a discrete number of
directions (facets) that are separately calibrated through the selection of a
calibrator (with a minimum flux of 0.5 Jy) for each facet. The coordinates of
PSZ139 and a 15′ radius around it were also specified to include the source in
one single facet. We processed 13 facets, i.e. the brightest sources in the
field and those bordering the facet containing PSZ139, which was then
processed at last, so that it could benefit from the calibration of the
preceding facets. All the images were corrected for the station primary beam.

Due to inaccuracies in the LOFAR beam model the images can require rescaling
(e.g. Harwood et al., 2016). In line with other LOFAR studies (van Weeren et
al., 2014), we have cross-checked the 144 MHz LOFAR flux scale against the 150
MHz TIFR GMRT Sky Survey (TGSS; Intema et al., 2017) using 50 compact sources.
We found and applied a scaling factor of $0.75$ with a scatter that we take
into account by assigning a $15\%$ uncertainty in our flux scale. The
corrected integrated surface brightness measurements are reported in Table 3.

Radio imaging was performed using the Common Astronomy Software Applications
(CASA, version 4.5.2; McMullin et al., 2007) tools with different parameters
to obtain different resolutions and increase the sensitivity to diffuse
emission. The imaging details are summarized in Table 2.

###  2.2 GMRT radio observation

GMRT observations at 610 MHz were collected during two distinct observations
on October 25, 2014, and September 4, 2015, under project codes 27_025 and
28_077, respectively. Visibilities were recorded in two polarizations (RR and
LL) over a bandwidth of 33.3 MHz. The total combined on-source time was 9.4 h.
The GMRT data were pre-calibrated, combined and processed using the SPAM
pipeline (see Intema et al., 2017 for details). The primary calibrators used
for flux and bandpass calibrations were 3C147 and 3C48, respectively. We
adopted the same flux standard as for LOFAR (Scaife & Heald, 2012). A $T_{\rm
sys}$ correction of $0.92$ was derived using the all-sky map at 408 MHz by
Haslam et al. (1995), and applied. Both observations had 4 of the 30 antennas
not working properly. Removing them during pre-calibration resulted in a 25%
data loss. Furthermore, the pipeline removed another 15% of the data due to
RFI and various telescope issues. The pipeline output visibilities were
imported into CASA for final imaging, using the multi-scale option of the
clean task. Our highest-fidelity images reach a sensitivity of
$27\,\mu$Jy/beam with a $8.0^{\prime\prime}\times 4.7^{\prime\prime}$ beam. We
adopted a 10% scale error on all flux density measurements (Chandra et al.,
2004).

Figure 1:  Optical Pan-STARRS g,r,i mosaic image with the 144 MHz LOFAR
smoothed image overlaid. The contour levels are at
$(-1,1,2,4,8)\,\times\,3\sigma$ where $\sigma$ = 500 $\mu$Jy/beam. The beam is
35${}^{\prime\prime}\times$ 35′′, and is shown at the bottom left of the
image.

Figure 2: Optical Pan-STARRS g,r,i mosaic image with the 610 MHz GMRT smoothed
image overlaid. The contour levels are at $(-1,1,2,4)\,\times\,3\sigma$ where
$\sigma$ = 180 $\mu$Jy/beam. The beam is 35${}^{\prime\prime}\times$ 35′′, and
is shown at the bottom left of the image.

###  2.3 Chandra X-ray observation

We reprocessed Chandra X-ray observations merging two Chandra ACIS-I
observations of PSZ139 in VFAINT mode with a total exposure time of 28 ks
(ObsID: 15139, 15297). Data were reprocessed with CIAO v4.9 and Chandra CALDB
v4.7.3 using the level=1 event file, following the standard Chandra reduction
threads. Soft proton flares were removed by inspecting the light curves
extracted in the S2 chip using the deflare script. The resulting exposure time
after this procedure is 23.1 ks. A single point spread function map at 1.5 keV
was obtained combining the corresponding exposure maps for each ObsID, then
point sources were detected with wavdetect, confirmed by eye and removed in
the further analysis. Spectra were fitted in the 0.5-11.0 keV band with XSPEC
v12.9.0o adopting an absorbed thermal model with metallicity fixed at 0.3
$Z_{\odot}$ for the ICM emission and with a fixed column density (Kalberla et
al., 2005) $N_{\rm H}=8.1\times 10^{20}$ cm-2 accounting for the Galactic
absorption in the direction of the cluster. The background was treated as
follows: the astrophysical background was assumed to be composed of a Galactic
component, modeled with a two temperature-plasma (with $kT_{1}=0.14$ keV and
$kT_{2}=0.25$ keV), and a cosmic X-ray background component, modeled with an
absorbed power-law (with $\Gamma=1.4$); the instrumental background was
modeled following the analytical approach proposed in Bartalucci et al.
(2014). The X-ray analysis follows the procedure described in Botteon et al.
(2018), to which we refer the reader for more details.

##  3 Results

###  3.1 Radio analysis

Using LOFAR observations at 144 MHz, we have discovered previously undetected
cluster-scale diffuse emission in PSZ139, as visible in Fig. 2. We have re-
analysed two archival 610 MHz GMRT observations, and combined their
visibilities to achieve better sensitivity and uv-coverage. In Fig. 2 and 2,
we present the low-resolution LOFAR and GMRT radio images. The images were
smoothed to enhance the diffuse emission, and are overlaid onto Pan-STARRS
g,r,i optical images (Chambers et al., 2016). In Fig. 4, we present the high-
resolution radio contours in the core region of the cluster. This consists of
two radio components: one at the cluster centre where also the X-ray peak is,
and one towards the N. The central source is likely to be related to the
Brightest Central Galaxy (BCG) that is visible in the optical image, whilst
there is no obvious optical counterpart for the northern radio brightness
source that might be a foreground radio galaxy. For our analysis, we have
considered the X-ray centre as the cluster centre.
Although both GMRT and LOFAR detect emission in the inner 200 kpc, i.e. within
the core, LOFAR reveals a more extended component which is not detected in the
GMRT image. To confirm the detection of radio diffuse emission from PSZ139, we
re-imaged the GMRT residual visibilities (after subtracting the full-
resolution model image) with a gaussian taper of 25′′ while enhancing the
contribution of the short baselines with a Briggs weighting scheme (robust=0).
We detected emission above $3\sigma$ on a scale marginally larger than the
cluster core ($\sim 350$ kpc, having a beam size equivalent to 75 kpc). Since
we do not detect emission that corresponds to the halo observed with LOFAR,
and since instrumental differences of the two observations must be taken into
account, we did not use this image to derive spectral index information, but
proceeded with a more conservative approach, as explained in Section 3.1.1.

From the LOFAR image, we measure an overall source size of $D_{\rm radio}\sim$
550 kpc. This value has been estimated as $D_{\rm radio}=\sqrt{D_{\rm
min}\times D_{\rm max}}$, where $D_{\rm min}$ and $D_{\rm max}$ are the
minimum and maximum diameter of the 3$\sigma$ surface brightness isocontours.
Since this value may depend on the sensitivity of the observation, we have
also estimated the e-folding radius, $r_{\rm e}$, which is defined as the
radius at which the brightness drops to $I_{\rm 0}/e$, where $I_{\rm 0}$ is
the central brightness of the source. Following Murgia et al. (2009), we have
obtained the radio brightness average in concentric circular annuli centred on
the X-ray centre with widths of 12′′ ($\sim$ 50 kpc) that is 1/2 FWHM of the
synthesized beam, and assumed a profile that follows the simple exponential
law $I(r)=I_{\rm 0}\,e^{-r/r_{e}}$. We convolved the exponential profile with
a gaussian with FWHM equal to the beam, obtaining the convolved profile that
we used to fit the data points shown in Fig. 7. The error of each annulus is
equal to $\sqrt{\sigma_{\rm flux}^{2}+\sigma^{2}\times N}$ where the first
contribution is the calibration error on the surface brightness and the second
one is the noise level of the radio image weighted by the number of beam in
the annulus. The best-fit values are $r_{e}=94\pm 10$ kpc and $I_{0}$ = $48\pm
2\,\mu$Jy/arcsec2. The fit shows a relatively compact ($r_{\rm e}\sim 94$ kpc)
emission in the core region. Larger-scale emission, although faint, can be
seen beyond the core region, especially extending towards the SE.

Figure 3: Plots of the inner uv-plane coverage of the LOFAR observation (three
time chunks are plotted; top panel) and GMRT observation (bottom panel). The
red box indicates the region within 200$\lambda$ that was excluded in the
imaging process for the spectral analysis. To minimise the difference between
the two observations we have also used a uniform weighting parameter.

Figure 4: Optical Pan-STARRS image in gray scale with the high-resolution 610 MHz GMRT (top) and 144 MHz LOFAR (bottom) blue contours overlaid. The GMRT and LOFAR beam is $8.0^{\prime\prime}\times 4.7^{\prime\prime}$ and $8.0^{\prime\prime}\times 6.5^{\prime\prime}$ respectively, and the GMRT and LOFAR levels are $(2,4,8)\,\times\,3\sigma$ where $\sigma$ = 27 $\mu$Jy/beam and $\sigma$ = 140 $\mu$Jy/beam respectively. The central radio components are indicated with labels: BCG is the brightest central galaxy that corresponds also to the X-ray centre; S1 is a source that has no obvious optical counterpart, and is likely to be a background galaxy; S2 is a faint source that is not detected in the LOFAR image. The dashed red region indicated the core region of the cluster, which has a size of $\sim 200$ kpc. Table 3: Integrated surface brightness and estimated radio power of the radio emission of PSZ139. The first two coloumns refer to values measured from the LOFAR and GMRT datasets. The last two coloumns refer to the value estimated for the surface brightness rescaled to 1.4 GHz, and the spectral index value used for rescaling. The core component is defined as the emission from the inner region with a size of $\sim 200$ kpc; the diffuse component corresponds to the emission on larger scales. The total value refers to the emission as a whole.  | LOFAR | GMRT | 1.4 GHz | $\alpha$
---|---|---|---|---
$S_{\rm core}$ (mJy) | 12 | 2.3 | - |
$P_{\rm core}$ (W/Hz) | $3\times 10^{24}$ | - | $1.5\times 10^{23}$ | -1.3
$S_{\rm diff}$ (mJy) | 18 | - |  |
$P_{\rm diff}$ (W/Hz) | $4\times 10^{24}$ | - | $2.2\times 10^{23}$ | -1.7 (UL)
$S_{\rm tot}$ (mJy) | 30 | - |  |
$P_{\rm tot}$ (W/Hz) | $7\times 10^{24}$ | - | $3.7\times 10^{23}$ | -1.3

####  3.1.1 Spectral analysis

We have re-imaged the LOFAR and GMRT datasets with a guassian taper of 20′′,
same pixel size, baseline range (200 - 40000 $\lambda$) and uniform weighting
scheme to minimize the differences in the uv-coverage of the two
interferometers.

Since only the central region of the cluster can be seen both at 610 MHz and
144 MHz, we have measured the value of the average spectral index of the inner
$D_{\rm radio}\sim 200$ kpc. To ensure that no contamination from AGN or
background sources was included in our estimate of the spectral index of the
core region, we measured the integrated surface brightness of the compact
sources detected in the high resolution image within the 6$\sigma$ GMRT and
LOFAR contours, as shown in Fig. 4; we then measured the integrated surface
brightness of the entire core region (inner $\sim$200 kpc, indicated by a
dashed red circle), and finally we subtracted the contribution of the compact
sources444The best strategy would be subtracting the compact sources from the
visibilities of the LOFAR and GMRT observations. However, in this case this
procedure is uncertain, since these compact sources may have extended
components (e.g. lobes) that are not easily separable from the surrounding
emission.. We obtained $\alpha_{144}^{610}=-1.3\pm 0.1$. The error takes into
account the flux calibration error. To confirm this value, we also calculated
the spectral index, completely masking the sources at $6\sigma$, obtaining a
consistent value.

To constrain the spectral properties of the diffuse emission (i.e. emission
outside the inner $\sim 200$ kpc) that was detected in the LOFAR image only,
we have also used the LOFAR and GMRT dataset re-imaged with uniform weighting
and same uv-range mentioned above. We first considered the LOFAR mean surface
brightness of the diffuse emission ($3.5$ mJy/beam) and the GMRT rms noise
($0.17$ mJy/beam), deriving $\alpha_{144}^{610}\leq-1.9$. The inner uv-
coverage of the GMRT and LOFAR datasets are different (see Fig. 3). Although
the radio emission extends on a scale of 2′, that is well sampled by both
observations, we have injected a mock radio halo in the GMRT visibilities.
Using this procedure, we can image the diffuse emission and place an upper
limit on the spectral index given the specific uv-coverage of that
observation. The mock source was modeled with the exponential law and the
parameters obtained from the best-fit of the radio surface brightness profile.
The model was Fourier transformed into the visibilities of the GMRT dataset
taking into account the w-projection effect, which is necessary due to the
large field of view and low frequency. We added the mock sources to the
original visibilities in a region close to the cluster but without bright
sources and clear noise structures, such as negative holes, and then re-imaged
the dataset with uniform weighting and measured the properties of the
recovered simulated emission. We created a set of mock sources assuming
different spectral indices, i.e. with different integrated flux densities. We
started with $\alpha_{144}^{610}=-1$ and then we lowered the value, until the
recovered flux of the mock source could not be considered detected anymore,
i.e. when the emission was $<2\sigma$ and the extension $<3r_{e}$. Using this
procedure, we put an upper limit of $\alpha<-1.7$.

On the basis of the spectral information and surface brightness we derived in
our radio analysis, we can distinguish two components of the radio emission:

  * •

a flatter-spectrum ($\alpha_{144}^{610}=-1.3\pm 0.1$), higher-brightness
($\sim$ 8 - 9 mJy/beam at 144 MHz) component within the cluster core with a
size of $D_{\rm radio}\sim 200$ kpc, classified as mini halo by Giacintucci et
al. (2017);

  * •

a steeper-spectrum ($\alpha_{144}^{610}<-1.7$), lower-brightness ($\sim$ 1 - 2
mJy/beam at 144 MHz) component visible on larger scales at low frequencies.

###  3.2 X-ray analysis

Merger activity in clusters leaves a clear imprint on the X-ray brightness
distribution, hence the morphology of the ICM provides a way to discriminate
between merging and non-merging clusters. Here, we focus on two morphological
indicators that can be derived from the X-ray surface brightness distribution
(Cuciti et al., 2015): the emission centroid shift $w$ and the concentration
parameter $c$. The former is defined as the standard deviation of the
projected separation between the peak and centroid of the X-ray surface
brightness distribution when the aperture used to compute it decreases from a
maximum radius of 500 kpc to smaller radii. The latter is defined as the ratio
of the X-ray surface brightness within a radius of 100 kpc over X-ray surface
brightness within a radius of 500 kpc. High values of $w$ indicate a
dynamically disturbed system, whilst high values of $c$ indicate a peaked
core, typical of non-merging systems. The position of PSZ139 in the $w-c$
diagram, based on Chandra X-ray observations, is reported in Fig. 5, where we
see that despite the high value of the concentration parameter
($c=0.362_{-0.004}^{+0.007}$), the value of the emission centroid shift
($w=1.35_{-0.17}^{+0.18}\times 10^{-2}$) is intermediate between merging and
non-merging systems (Cassano et al., 2016). These values strengthen the
argument that the cluster hosts a cool core (as also suggested by the low
central entropy of the ICM) but is not fully virialised (relaxed).

Figure 5: Diagram of the X-ray morphological indicators based on Chandra
observations for the galaxy clusters, $w$ and $c$, of the mass-selected
cluster sample in Cuciti et al. (2015) and Cassano et al. (2016). The red
square represents PSZ139. Following Cassano et al. (2010), we have adopted the
values $w\leq 0.012$ and $c\geq 0.2$ to separate merging from non-merging
clusters. Merging clusters lie in the lower right region of the plot, whilst
non-merging clusters in the upper left region.

Figure 6:  Top left panel: Chandra X-ray image smoothed on a scale of 6′′ with
the overlay of the LOFAR contours and the sector used for the profile
extraction. The inner grey arc indicates the position ($\sim 100$ kpc from the
centre) of the cold front that we have discovered. Radio emission extends far
beyond the cold front. Top right panel: X-ray surface brightness profile of
the discontinuity detected in the Chandra image. The data were rebinned to
reach a minimum signal-to-noise ratio of 7, and fitted with three models:
broken-power law in solid blue ($\chi^{2}/dof$ = 26.5/29), power-law in dashed
red ($\chi^{2}/dof$ = 152.1/31), and beta-model in dashed green
($\chi^{2}/dof$ = 67.8/31). The residuals at the bottom of the plot refer to
the broken-power law model. The two colored boxes indicate the temperature in
keV in the upstream and downstream regions, and their sizes indicate the
radial extension of the spectral region. Bottom panel: Projected temperature
map of the cluster with the contour levels at $(1,2,4,8)\,\times\,3\sigma$
where $\sigma$ = 500 $\mu$Jy/beam of the 144 MHz LOFAR image with a beam of
35${}^{\prime\prime}\times$ 35′′ overlaid. The removed background sources (in
white) are indicated. The presence of a cool core can be clearly seen.

We reprocessed Chandra observations of the cluster in the 0.5-2.0 keV band to
search for possible surface brightness and temperature jumps. We obtained a
projected temperature map of the ICM using CONTBIN v1.4 (Sanders, 2006) shown
in the bottom panel of Fig. 6. Given the short exposure of the Chandra
observations, we required 900 background-subtracted counts per bin. This
resulted in 8 spectral regions that were then extracted and fitted. The
temperature map clearly shows the presence of the cool core, and indicates a
temperature jump from the central regions outwards in the NW direction. We
then obtained the surface brightness profile and found a discontinuity towards
the NW direction. The profile extraction and fitting were performed with
PROFFIT (Eckert et al., 2011) on the exposure-corrected image of the cluster.
The spectral profile was extracted in the sector indicated in grey in the top
left panel of Fig.6 and fitted with three models: broken-power law, power-law,
and beta-model. The profile is best described by a broken power-law with
compression factor $C=1.7\pm 0.1$. The spectral analysis provides temperatures
of $6.8_{-0.8}^{+1.0}$ keV and $10.2_{-2.2}^{+4.0}$ keV in the upstream and
downstream regions, respectively. In general, both shocks and cold fronts are
sharp surface brightness discontinuities (Markevitch & Vikhlinin, 2007),
however shocks mark pressure discontinuities where the gas is heated in the
downstream region, with higher temperature values with respect to the upstream
region, whilst cold fronts do not show a pressure jump and the downstream
temperature is lower than the upstream temperature. We have combined the
temperature jump and compression ratio to estimate the pressure ratio across
the edge and checked that the pressure is continuous across the front, as
expected in the case of a cold front.

Given the discontinuity found both in the emissivity and temperature profiles,
the morphology of the X-ray emission (elongated in the NW-SE direction) and
the presence of a cool core, we argue that the cold front scenario is the most
likely interpretation of the discontinuity.

Figure 7: The azimuthally-averaged brightness profiles of the radio emission
in PSZ139. The best-fit line is indicated in red, and the 3$\sigma$ noise
level of the radio image is indicated with a horizontal dashed-dotted black
line. The best-fit values are $r_{e}=94\pm 10$ kpc and $I_{0}$ = $48\pm
2\,\mu$Jy/arcsec2.

##  4 Discussion and summary

The information from the literature and the X-ray analysis that we have
performed indicate that PSZ139 shows typical features of a non-merging
cluster, i.e. a cool core with low central entropy. The X-ray morphology of
the cluster on larger scales, though, is not spherically symmetric, suggesting
the occurrence of a merger that has left a clear imprint on the gas
distribution towards the SE of the cluster core. The morphological indicators
$w$ and $c$ also suggest a cluster with a dense core and an elongated shape.
Moreover, the radio mini halo discovered by Giacintucci et al. (2017) suggests
that the core has not been disrupted. These features suggest a scenario where
the cluster is undergoing a merger that is not powerful enough to disrupt the
cool core. In fact, simulations have shown that it is quite hard for mergers
to destroy cool cores in galaxy clusters (Burns et al., 2008).

Based on the correlation in Cassano et al. (2013) for giant halos, which
compares cluster mass, halo radio power, and halo size, a cluster with the
mass of PSZ139 is expected to host a giant halo with a power of $P_{\rm
tot,1.4}\sim 2\times 10^{24}$ W Hz-1 and a total size of $D_{\rm radio}\sim
950$ kpc. The total estimated radio power at 144 MHz is $P_{\rm
tot,144}=(7.2\pm 1.1)\times 10^{24}$ W/Hz, which corresponds to $P_{\rm
tot,1.4}=(3.7\pm 0.5)\times 10^{23}$ W/Hz at 1.4 GHz, assuming a conservative
spectral index of $\alpha=-1.3$ for the whole source. Even considering a large
scatter around this correlation, the diffuse emission of PSZ139 is more than
an order of magnitude underluminous and almost a factor of 2 smaller in size.

We argue that PSZ139 is the first example of a cluster that hosts ultra-steep-
spectrum radio halo emission outside of its cool core. This radio emission is
detected only at low frequencies and extends out to smaller radii than in
typical giant radio halos. The radio analysis suggests that the more compact
emission, which is coincident with the cool core and has a spectrum typical of
mini halos ($\alpha_{144}^{610}\sim-1.3$), is surrounded by ultra-steep
diffuse emission ($\alpha_{144}^{610}<-1.7$) that is correlated with the X-ray
morphology on larger scales555We note that ultra-steep spectrum emission can
also originate from the old lobes of a radio galaxy but we consider such a
scenario unlikely, since we observe a strong spatial correlation between the
radio and X-ray emission, as is typically seen in radio halos.. The X-ray
analysis indicates the presence of a cool core with traces of dynamical
activity, especially towards the SE, in line with the direction where most of
the larger-scale diffuse emission is detected. The presence of a cold front
suggests that the core is likely to be sloshing. This observational evidence
motivates us to identify a new scenario that links cool-core clusters and
particle re-acceleration on cluster scales. We argue that the radio emission
found in PSZ139 is caused by a minor merger that has dissipated enough energy
in the ICM to accelerate particles, but leaves the core intact, as indicated
by the low central entropy and temperature. Through the same mechanism that
generates giant radio halos in merging clusters, less energetic mergers are
predicted to form halos with a steeper spectrum and lower power than more
energetic (major) mergers (Cassano et al., 2006, Brunetti et al., 2008). Up
until now, this population of minor-merger, cool-core clusters remains largely
undetected, and PSZ139 may in fact be the first example. The radio emission of
PSZ139 suggests that the energy dissipated during a minor merger can drive
turbulent motions outside the cluster core, but on scales smaller than that of
giant halos, whilst still preserving the mini halo emission. We exclude the
scenario where the source in PSZ139 is a transition object: either that it is
a giant halo decaying into a mini halo, or a mini halo growing into a giant
halo. For a giant radio halo to fade from the edges and shrink to its present
size, the radiative losses, and hence the magnetic field strength, would have
to be stronger at the edges than in the core of the cluster. This is
considered unlikely. In the other case, we can exclude that the relativistic
plasma has been transported from the core out to larger scales since this
would require unrealistically high transport coefficients.

The presence of diffuse emission on scales larger than the core has rarely
been seen in non-merging clusters. Among the few known cases666Few cases of
Mpc-scale halos have been found in massive clusters with no evidence of major
mergers (CL1821+643, Bonafede et al., 2014; A2261, A2390, Sommer et al., 2017;
A2142, Venturi et al., 2017). The former two sources substantially differ from
PSZ139: CL1821+643 for being a flat-spectrum giant halo, and A2261 for not
hosting a cool core. The latter two sources also differ from PSZ139, but
deserve a more accurate comparison, and are therefore discussed in the main
text., A2142 is the cluster that shares some similarities with PSZ139: it
shows a two-component radio halo with flatter-spectrum emission in the core
and a slightly steeper spectrum emission on Mpc scales (Venturi et al., 2017).
However, A2142 does not host a cool core, its halo has a size typical of giant
radio halos, and the difference in the spectral index of the radio emission in
the two components is only marginally significant. A case of a cool-core
cluster hosting Mpc-scale emission is A2390 (Sommer et al., 2017), however the
dynamic of the cluster is not clear, and the large errors on the spectral
indices do not permit an assessment whether the emission on cluster scales has
a steep spectrum.

PSZG139 is the first cool-core cluster to host steep-spectrum emission on
larger scales. This example indicates that the connection between the
evolution of radio emission on different scales and the dynamical status of
the cluster is more complex than previously thought, and that particle
acceleration mechanisms at different scales can be observed simultaneously in
the same cluster. As the energy dissipated by minor mergers is primarily
observed at low radio frequencies, we expect that radio halos with steep
spectra will be found in these types of clusters by forthcoming high-
sensitivity, low-frequency radio observations.

## Acknowledgements

LOFAR, the Low Frequency Array designed and constructed by ASTRON, has
facilities owned by various parties (each with their own funding sources), and
that are collectively operated by the International LOFAR Telescope (ILT)
foundation under a joint scientific policy. The LOFAR software and dedicated
reduction packages on https://github.com/apmechev/GRID_LRT were deployed on
the e-infrastructure by the LOFAR e-infragroup, consisting of J.B.R.O. (ASTRON
& Leiden Observatory), A.P.M. (Leiden Observatory) and T.S. (Leiden
Observatory) with support from N. Danezi (SURFsara) and C. Schrijvers
(SURFsara). This research had made use of the LOFAR Solution Tool (LoSoTo),
developed by F.dG., and of the NASA/IPAC Extragalactic Database (NED), which
is operated by the Jet Propulsion Laboratory, California Institute of
Technology, under contract with the National Aeronautics and Space
Administration. We thank the staff of the GMRT that made the observation
possible. GMRT is run by the National Centre for Radio Tata Institute of
Fundamental Research.
A.B. acknowledges support from the ERC-Stg 714245 DRANOEL. H.R. and R.vW.
acknowledge support from the ERC Advanced Investigator programme
NewClusters321271. R.vW. acknowledges support of the VIDI research programme
with project number 639.042.729, which is financed by the Netherlands
Organisation for Scientific Research (NWO). Basic research in radio astronomy
at the Naval Research Laboratory is supported by 6.1 Base funding. G.W.
gratefully thanks the Leverhulme Trust for funding.

## References

  * Bartalucci et al. (2014) Bartalucci I., Mazzotta P., Bourdin H., Vikhlinin A., 2014, A&A, 566, A25
  * Bonafede et al. (2014) Bonafede A., et al., 2014, MNRAS, 444, L44
  * Botteon et al. (2018) Botteon A., Gastaldello F., Brunetti G., 2018, MNRAS, 476, 5591
  * Briggs (1995) Briggs D. S., 1995, New Mexico Institute of Mining Technology, Socorro, New Mexico, USA,
  * Brunetti & Lazarian (2011) Brunetti G., Lazarian A., 2011, MNRAS, 410, 127
  * Brunetti et al. (2007) Brunetti G., Venturi T., Dallacasa D., Cassano R., Dolag K., Giacintucci S., Setti G., 2007, ApJ, 670, L5
  * Brunetti et al. (2008) Brunetti G., et al., 2008, Nat, 455, 944
  * Buote (2001) Buote D. A., 2001, ApJ, 553, L15
  * Burns et al. (2008) Burns J. O., Hallman E. J., Gantner B., Motl P. M., Norman M. L., 2008, ApJ, 675, 1125
  * Cassano et al. (2006) Cassano R., Brunetti G., Setti G., 2006, MNRAS, 369, 1577
  * Cassano et al. (2008) Cassano R., Gitti M., Brunetti G., 2008, A&A, 486, L31
  * Cassano et al. (2010) Cassano R., Ettori S., Giacintucci S., Brunetti G., Markevitch M., Venturi T., Gitti M., 2010, ApJ, 721, L82
  * Cassano et al. (2013) Cassano R., et al., 2013, ApJ, 777, 141
  * Cassano et al. (2016) Cassano R., Brunetti G., Giocoli C., Ettori S., 2016, A&A, 593, A81
  * Cavagnolo et al. (2009) Cavagnolo K. W., Donahue M., Voit G. M., Sun M., 2009, ApJS, 182, 12
  * Chambers et al. (2016) Chambers K. C., et al., 2016, preprint,  (arXiv:1612.05560)
  * Chandra et al. (2004) Chandra P., Ray A., Bhatnagar S., 2004, ApJ, 612, 974
  * Cuciti et al. (2015) Cuciti V., Cassano R., Brunetti G., Dallacasa D., Kale R., Ettori S., Venturi T., 2015, A&A, 580, A97
  * Eckert et al. (2011) Eckert D., Molendi S., Paltani S., 2011, A&A, 526, A79
  * Giacintucci et al. (2017) Giacintucci S., Markevitch M., Cassano R., Venturi T., Clarke T. E., Brunetti G., 2017, ApJ, 841, 71
  * Govoni et al. (2009) Govoni F., Murgia M., Markevitch M., Feretti L., Giovannini G., Taylor G. B., Carretti E., 2009, A&A, 499, 371
  * Harwood et al. (2016) Harwood J. J., et al., 2016, MNRAS, 458, 4443
  * Haslam et al. (1995) Haslam C. G. T., Salter C. J., Stoffel H., Wilson W. E., 1995, Astronomy Data Image Library,
  * Intema et al. (2017) Intema H. T., Jagannathan P., Mooley K. P., Frail D. A., 2017, A&A, 598, A78
  * Kalberla et al. (2005) Kalberla P. M. W., Burton W. B., Hartmann D., Arnal E. M., Bajaja E., Morras R., Pöppel W. G. L., 2005, A&A, 440, 775
  * Kale et al. (2015) Kale R., et al., 2015, A&A, 579, A92
  * Markevitch & Vikhlinin (2007) Markevitch M., Vikhlinin A., 2007, Phys. Rep., 443, 1
  * Mazzotta & Giacintucci (2008) Mazzotta P., Giacintucci S., 2008, ApJ, 675, L9
  * McMullin et al. (2007) McMullin J. P., Waters B., Schiebel D., Young W., Golap K., 2007, in Shaw R. A., Hill F., Bell D. J., eds, Astronomical Society of the Pacific Conference Series Vol. 376, Astronomical Data Analysis Software and Systems XVI. p. 127
  * Murgia et al. (2009) Murgia M., Govoni F., Markevitch M., Feretti L., Giovannini G., Taylor G. B., Carretti E., 2009, A&A, 499, 679
  * Pinzke et al. (2017) Pinzke A., Oh S. P., Pfrommer C., 2017, MNRAS, 465, 4800
  * Planck Collaboration et al. (2014) Planck Collaboration et al., 2014, A&A, 571, A29
  * Planck Collaboration et al. (2016) Planck Collaboration et al., 2016, A&A, 594, A13
  * Sanders (2006) Sanders J. S., 2006, MNRAS, 371, 829
  * Savini et al. (2018) Savini F., et al., 2018, MNRAS, 474, 5023
  * Scaife & Heald (2012) Scaife A. M. M., Heald G. H., 2012, MNRAS, 423, L30
  * Shimwell et al. (2017) Shimwell T. W., et al., 2017, A&A, 598, A104
  * Sommer et al. (2017) Sommer M. W., Basu K., Intema H., Pacaud F., Bonafede A., Babul A., Bertoldi F., 2017, MNRAS, 466, 996
  * Venturi et al. (2017) Venturi T., et al., 2017, A&A, 603, A125
  * ZuHone et al. (2013) ZuHone J. A., Markevitch M., Brunetti G., Giacintucci S., 2013, ApJ, 762, 78
  * de Gasperin et al. (2017) de Gasperin F., et al., 2017, Science Advances, 3, e1701634
  * van Haarlem et al. (2013) van Haarlem M. P., et al., 2013, A&A, 556, A2
  * van Weeren et al. (2014) van Weeren R. J., et al., 2014, ApJ, 786, L17
  * van Weeren et al. (2016) van Weeren R. J., et al., 2016, ApJS, 223, 2