#  InfoLM: A New Metric to Evaluate Summarization
& Data2Text Generation

Pierre Colombo†, Chloé Clavel⋆,Pablo Piantanida† Work done while Pierre was
funded by IBM & Telecom Paris

###### Abstract

Assessing the quality of natural language generation systems through human
annotation is very expensive. Additionally, human annotation campaigns are
time-consuming and include non-reusable human labour. In practice, researchers
rely on automatic metrics as a proxy of quality. In the last decade, many
string-based metrics (e.g., BLEU) have been introduced. However, such metrics
usually rely on exact matches and thus, do not robustly handle synonyms. In
this paper, we introduce InfoLM a family of untrained metrics that can be
viewed as a string-based metric that addresses the aforementioned flaws thanks
to a pre-trained masked language model. This family of metrics also makes use
of information measures allowing the adaptation of InfoLM to various
evaluation criteria. Using direct assessment, we demonstrate that InfoLM
achieves statistically significant improvement and over $10$ points of
correlation gains in many configurations on both summarization and data2text
generation.

##  1 Introduction

A plethora of applications of natural language processing (NLP) performs text-
to-text transformation (Mellish and Dale 1998; Belz and Reiter 2006; Specia,
Scarton, and Paetzold 2018). Given an input, these systems are required to
produce an output text that is coherent, readable and informative. Due to both
high annotation costs and time, researchers tend to rely on automatic
evaluation to compare the outputs of such systems. Reference-based automatic
evaluation relies on comparing a candidate text produced by the NLG system and
one or multiple reference texts (‘gold standard’) created by a human
annotator. Generic automatic evaluation of NLG is a huge challenge as it
requires building a metric that evaluates the similarity between a candidate
and one or several gold-standard reference texts. However, the definition of
success criteria is task-specific: as an example, evaluation of text
summarization focuses on content, coherence, grammatically, conciseness, and
readability (Mani 2001), whereas machine translation focuses on fidelity,
fluency and adequacy of the translation (Hovy 1999) and data2text generation
(Gardent et al. 2017) consider criteria such as data coverage, correctness and
text structure.

Automatic text evaluation metrics fall into two categories: metrics that are
trained to maximise their correlations using human annotation (e.g., RUSE
(Shimanaka, Kajiwara, and Komachi 2018), BEER (Stanojević and Sima’an 2014),
BLEND (Ma et al. 2017)) and untrained metrics (e.g., BLEU (Papineni et al.
2002), ROUGE (Lin 2004), BERTSCORE (Zhang et al. 2019), DepthScore (Staerman
et al. 2021b), BaryScore (Colombo et al. 2021b), MOVERSCORE (Zhao et al. 2019)
Word Mover Distance (Kusner et al. 2015)). In this work, we focus on untrained
metrics as trained metrics may not generalize well to new data (existing
labelled corpora are of small size). Two categories of untrained metrics can
be distinguished: word or character based-metrics that compute a score based
on string representation and embedding-based metrics that rely on a continuous
representation. String-based metrics (e.g., BLEU) often fail to robustly match
paraphrases (Reiter and Belz 2009) as they mainly focus on the surface form as
opposed to embedding-based metrics relying on continuous representations.

In this paper, we introduce InfoLM a family of new untrained metrics to
evaluate text summarization and data2text generation. At the highest level
InfoLM key components include: (1) a pre-trained masked language model (PMLM)
that is used to compute two discrete probability distributions over the
vocabulary. They represent the probability of observing each token of the
vocabulary given the candidate and the reference sentence, respectively. (2) A
contrast function $\mathcal{I}$ that is used to measure the dissimilarity
between aforementioned probability distributions. InfoLM differs from existing
BERT-based metrics (e.g. BERTSCORE, MOVERSCORE) as it directly relies on the
PMLM which outputs discrete probability distributions. Thus InfoLM does
neither require to arbitrarily select one or several specific layers ( e.g.
BERTSCORE relies on the 9th layer for bert-base-uncased), nor involves
selecting arbitrary aggregations technics (e.g. Power Mean (Rücklé et al.
2018) for MOVERSCORE). As InfoLM relies on statistics on tokens it can also be
seen as a string-based metric. However, it does not suffer from common
pitfalls of string-based metrics (e.g. synonyms, need of an exact string-
match) as the PMLM also allows ones to assign a high score to paraphrases and
to capture distant dependencies.
Contributions Our contributions are summarized below:
(1) A set of novel metrics to automatically evaluate summarization and
data2text generation. In this work, we introduce InfoLM which overcomes the
common pitfall of string matching metrics and does not require to select a
layer, nor to rely on a arbitrary aggregation function. InfoLM combines a pre-
trained model and a contrast function denoted by $\mathcal{I}$ between two
discrete probability distributions. We explore the use of different choices of
contrast functions such as $f$-divergences, $\mathcal{L}_{p}$ distances or
Fisher-Rao distances.
(2) Tasks. First, we demonstrate on both summarization and data2text that
InfoLM is better suited than concurrent metrics. A comparison is conducted,
using multiple correlation measures with human judgment both at the text and
system level. Second, we dissect InfoLM to better understand the relative
importance of each component (e.g. calibration, sensibility to the change of
information measures).

##  2 Problem statement and related work

In this section, we start by introducing notations and formulate the problem
of both evaluating text generation and metrics. Then, we identify and present
the most relevant related work and the existing approaches for the studied
tasks.

###  2.1 Problem statement

NLG evaluation. Given a dataset
$\mathcal{D}=\\{\boldsymbol{x_{i}},\\{\boldsymbol{y_{i}^{s}},h(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{s}})\\}_{s=1}^{S}\\}_{i=1}^{N}$
where $\boldsymbol{x}_{i}$ is the $i$-th reference text;
$\boldsymbol{y}_{i}^{s}$ is the $i$-th candidate text generated by the $s$-th
NLG system; $N$ is the number of texts in the dataset and $S$ the number of
systems available. The vector $\boldsymbol{x_{i}}=({x_{1},\cdots,x_{M}})$ is
composed of M tokens (e.g., words or subwords) and
$\boldsymbol{y_{i}^{s}}=({y_{1}^{s},\cdots,y_{L}^{s}})$ is composed of L
tokens111The reference and candidate text can be composed of several sentences
as it is the case in summarization.. The set of tokens (vocabulary) is denoted
as $\Omega$, $\mathbf{T}$ denotes the set of possible texts.
$h(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{s}})\in\mathcal{R}^{+}$ is the score
associated by a human annotator to the candidate text $\boldsymbol{y}_{i}^{s}$
when comparing it with the reference text $\boldsymbol{x_{i}}$. We aim at
building an evaluation metric $f$ such that
$f(\boldsymbol{x_{i}},\boldsymbol{y_{i}})\in\mathbb{R}^{+}$.

Evaluating evaluation metrics. To assess the relevance of an evaluation metric
$f$, correlation with human judgment is considered to be one of the most
important criteria (Koehn 2009; Specia, Raj, and Turchi 2010; Chatzikoumi
2020). Debate on the relative merits of different correlations for the
evaluation of automatic metrics is ongoing but classical correlation measures
are Pearson (Leusch et al. 2003), Spearman (Melamed, Green, and Turian 2003)
or Kendall (Kendall 1938) tests. Two meta-evaluation strategies are commonly
used: (1) text-level correlation or (2) system-level correlation. Formally,
the text-level correlation $C_{t,f}$ is computed as follows:

$\begin{split}C_{t,f}&\triangleq\frac{1}{N}\sum_{i=1}^{N}K(\mathbf{F}^{t}_{i},\mathbf{H}^{t}_{i}),\end{split}$
(1)

where
$\mathbf{F}_{i}=\big{[}f(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{1}}),\cdots,f(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{S}})\big{]}$
and
$\mathbf{H}_{i}=\big{[}h(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{1}}),\cdots,h(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{S}})\big{]}$
are the vectors composed of scores assigned by the automatic metric $f$ and
the human respectively. and
$K:\mathbb{R}^{N}\times\mathbb{R}^{N}\rightarrow[-1,1]$ is the chosen
correlation measure (e.g., Pearson, Kendall or Spearman). Similarly, the
system level correlation $C_{sy,f}$ is obtained by

$\displaystyle C_{sy,f}$ $\displaystyle\triangleq
K(\mathbf{F}^{sy},\mathbf{H}^{sy}),$ (2) $\displaystyle\mathbf{F}^{sy}$
$\displaystyle=\left[\frac{1}{N}\sum\limits_{i=1}^{N}f(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{1}}),\dots,\frac{1}{N}\sum\limits_{i=1}^{N}f(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{S}})\right]$
$\displaystyle\mathbf{H}^{sy}$
$\displaystyle=\left[\frac{1}{N}\sum\limits_{i=1}^{N}h(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{1}}),\dots,\frac{1}{N}\sum\limits_{i=1}^{N}h(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{S}})\right],$

where the latter are the vectors composed of the averaged scores assigned by
$f$ and the human, respectively. For the significance analysis, we follow
Graham and Baldwin (2014) and use a William test to validate a significant
improvement for dependent correlations (Steiger 1980).

###  2.2 Existing metrics

#### String-based metrics

Two types of string-based metrics exist: _N-Grams matching_ and _Edit
distance-based_ metrics. _N-Grams matching metrics_ count the number of
N-grams in common between the candidate text and the reference text. The three
most-used metrics are BLEU, ROUGE and METEOR (Banerjee and Lavie 2005). If no
N-gram is in common between the input text candidate and the reference, these
metrics fail to produce meaningful scores. The second category of metrics
gathers _edit distance-based metrics_. They measure the number of basic
operations such as edit/delete/insert to measure semantic equivalence.
Variants include TER (Snover et al. 2006), CDER (Leusch, Ueffing, and Ney
2006), EED (Stanchev, Wang, and Ney 2019), CHARACTER (Wang et al. 2016). Edit
distance-based metrics do not handle synonyms and focus on surface form.
InfoLM can be seen as string-based but do not suffer from the aforementioned
matching problem and can handle synonyms as it relies on a PMLM.

#### Embedding-based metrics

Another class of metrics relies on word embeddings. These metrics either use
static words embeddings such as word2vec (Mikolov et al. 2013) or
contextualized embeddings such as ELMO (Peters et al. 2018), BERT (Devlin et
al. 2018) and its variants (Sanh et al. 2019; Liu et al. 2019). Among the most
popular metrics, we can mention MOVERSCORE, BERTSCORE, WMD (Kusner et al.
2015) , WMDO (Chow, Specia, and Madhyastha 2019). Different from these
approaches InfoLM relies on a language model and work with discrete
probability distributions instead of continuous representations.

#### Learning-based metrics

Various trained metrics have been proposed such as BEER, BEND, RUSE, CIDER
(Vedantam, Lawrence Zitnick, and Parikh 2015). These methods rely on
train/dev/test sets composed of human evaluations. InfoLM does not require any
training step and relies on a frozen PMLM.

####  PMLM as a metric.

To the best of our knowledge, using a PMLM (i.e, without further training) as
a reference-based automatic metric remains overlooked. The closest use we
found was to rely on autoregressive models, such as GPT-2 (Radford et al.
2019), to compute the generated sentence perplexity and assess its fluency.
Researchers mainly focused on the use of the learnt embedding of the PMLM.
However, it remains an open question to find a reliable layer aggregation
mechanism (e.g, BERTSCORE arbitrary selects a layer based on a chosen dataset,
MOVERSCORE uses the 5 last layers). InfoLM addresses this aggregation issue by
relying on the PMLM.

###  2.3 Masked language modelling

Language models based on masked language pre-training objectives Devlin et al.
(2018); Liu et al. (2019) aim at reconstructing a corrupt version
$[\boldsymbol{x}]$ of an input text $\boldsymbol{x}$ by minimizing a cross-
entropy loss. This corrupted context corresponds to a ”local view” view of the
sentence. To ensure fair comparison, we do not use existing alternatives (e.g.
GPT-2 based models (Radford et al. 2019)) as concurrent works (Zhang et al.
2019; Zhao et al. 2019) rely on PMLM.

##  3 InfoLM

In this section, we first introduce a novel family of metrics called InfoLM
and then detail the different components of these novel metrics.
Notations We denote by $\theta\in\Theta$, the parameter of the PMLM,
$\mathcal{T}$ its temperature and
$\mathcal{I}:[0,1]^{|\mathbf{\Omega}|}\times[0,1]^{|\mathbf{\Omega}|}$ an
information measure (see Section 3.3) which quantifies the similarity between
two discrete distributions.

###  3.1 Motivations & Definitions

A PMLM has learnt the empirical distribution of a large text corpus. Given a
text $\boldsymbol{x}$, the corrupted context with a mask at position j is
denoted $[\boldsymbol{x}]^{j}$, the LM predicts a distribution
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|[\boldsymbol{x}]^{j};\theta;\mathcal{T})$
over the vocabulary $\mathbf{\Omega}$ given the masked context. As an example,
for a masked input sentence $[\boldsymbol{x}]^{1}=\text{“The [MASK] was
delicious”}$, a pretrained model could place high probabilities on tokens
“food”, “meal” and low probability on ”the”. It is worth noting that
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|[\boldsymbol{x}]^{1};\theta;\mathcal{T})$
represents the probability of observing each token of the vocabulary given the
masked input $[\boldsymbol{x}]^{1}$.

######  Definition 3.1 (Equivalence for masked contexts).

Given $\mathcal{I}$, two masked contexts $[\boldsymbol{x}]^{j}$,
$[\boldsymbol{y}]^{k}$ from input texts $\boldsymbol{x}$, $\boldsymbol{y}$,
with masks at positions j and k respectively, are equivalent (denoted
$[\boldsymbol{x}]^{j}\overset{\mathcal{I}}{\sim}[\boldsymbol{y}]^{k}$) if the
two predicted discrete distributions given by the PMLM, namely
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|[\boldsymbol{x}]^{j};\theta;\mathcal{T})$
and
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|[\boldsymbol{y}]^{k};\theta;\mathcal{T})$,
are similar. Formally,
$[\boldsymbol{x}]^{j}\overset{\mathcal{I}}{\sim}[\boldsymbol{y}]^{k}$ if
$\mathcal{I}\left[p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|[\boldsymbol{x}]^{j};\theta;\mathcal{T}),p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|[\boldsymbol{y}]^{k};\theta;\mathcal{T})\right]\approx
0$.

######  Remark.

We have the intuition that two similar sentences will share several pairs of
equivalent masked contexts. At this point, we make no claim on the
relationship between equivalence and the masked context similarity.

In this work, we make the hypothesis that two similar sentences
$\boldsymbol{x},\boldsymbol{y}$ will share multiple equivalent masked
contexts. However, pairwise comparisons of all the pairs of individual masked
contexts are prohibitively expensive ($\mathcal{O}(L\times M)$ comparisons)
when considering long texts. Motivated by efficiency, we instead propose to
work with two ”global views” of the sentences that are well-formed probability
distributions and are obtained through the aggregation of individual PMLM
predictions. Aggregated probabilities for $\boldsymbol{x}$ and
$\boldsymbol{y}$ are denoted
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x};\theta;\mathcal{T})$ and
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y};\theta;\mathcal{T})$
respectively.

######  Definition 3.2 (Similarity for texts).

Given $\mathcal{I}$, two texts $\boldsymbol{x},\boldsymbol{y}$ are said to be
similar (denoted $\boldsymbol{x}\overset{\mathcal{I}}{\approx}\boldsymbol{y}$)
if
$\mathcal{I}\left[p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x};\theta;\mathcal{T}),p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y};\theta;\mathcal{T})\right]\approx
0$ where
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x};\theta;\mathcal{T})$ and
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y};\theta;\mathcal{T})$
denotes the aggregated individual masked context predictions.

###  3.2 InfoLM

#### Overview

InfoLM uses the notion of similarity given in Definition 3.2. Given a
reference text $\boldsymbol{x_{i}}$ together with a candidate text
$\boldsymbol{y_{i}^{s}}$, InfoLM recursively masks each token position of both
$\boldsymbol{x_{i}}$ and $\boldsymbol{y_{i}^{s}}$ to obtain individual masked
contexts. By relying on a PMLM, InfoLM predicts one distribution for each
individual masked contexts. The resulting distributions are then averaged (we
refer to this operation ”bag of distributions”) to obtain
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}};\theta;\mathcal{T})$
and
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x_{i}};\theta;\mathcal{T})$.
The final step involves comparing two well formed discrete probability
distributions
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}};\theta;\mathcal{T})$
and
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x_{i}};\theta;\mathcal{T})$
through a measure of information $\mathcal{I}$. InfoLM writes as:

$\scalebox{1.2}[1.0]{
{InfoLM}}\left(\boldsymbol{x_{i}},\boldsymbol{y_{i}^{s}}\right)\triangleq\mathcal{I}\big{[}p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x_{i}};\theta;\mathcal{T}),p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}};\theta;\mathcal{T})\big{]}.$
(3)

######  Remark.

It is worth to emphasize that
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x_{i}};\theta;\mathcal{T})$
and
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}};\theta;\mathcal{T})$
are two well formed discrete probability distributions. They represent the
probability of observing each token of the vocabulary given the candidate and
the reference sentence, respectively.

#### Aggregation Procedure

Rare tokens can be more indicative of text similarity than common tokens
(Banerjee and Lavie 2005). Thus, for the aggregation of the individual masked
contexts, we propose to compute a weighted ”bag of distributions” where the
weights are normalized measures of the importance of each token. In practice,
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x};\theta;\mathcal{T})$ and
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}};\theta;\mathcal{T})$
write as:

$\begin{split}p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x};\theta;\mathcal{T})&\triangleq\sum_{k=1}^{M}\gamma_{k}\times
p_{\theta}(\cdot|[\boldsymbol{x}]^{k};\theta;\mathcal{T}),\\\
p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}};\theta;\mathcal{T})&\triangleq\sum_{k=1}^{N}\tilde{\gamma}_{k}\times
p_{\theta}(\cdot|[\boldsymbol{y_{i}^{s}}]^{k};\theta;\mathcal{T}),\end{split}$

where $\tilde{\gamma}_{k}$ and $\gamma_{k}$ are measures of the importance of
the k-th token in the candidate and reference text, respectively, i.e.,
satisfying $\sum_{j=1}^{M}\gamma_{j}=\sum_{j=1}^{N}\tilde{\gamma}_{j}=1$.
These are computed using the inverse document frequency scores determined at
the corpus level (Zhao et al. 2019; Kusner et al. 2015).

#### LM Calibration.

Modern deep neural networks are overconfident (Guo et al. 2017). To re-
calibrate language models several techniques have been proposed (e.g entropy
rates, (Braverman et al. 2020) temperature scaling (Platt et al. 1999),
contextual calibration (Zhao et al. 2021)). Here, we choose to study how
calibration affects InfoLM by relying on temperature222When
$\mathcal{T}\rightarrow 0$ one token receive all the probability mass, and
when $\mathcal{T}\rightarrow\infty$, the probability becomes uniform. scaling
motivated by simplicity and speed.

###  3.3 Information measures

In this work, we focus on comparing a pair of discrete probability
distributions through information measures (see Basseville (2013) for an
exhaustive study). We rely on two types of information measures: _divergences_
and _distances_. The divergence is a measure of dissimilarity that is always
positive or equal to zero if (and only if) the two considered distributions
are strictly identical. We call distance, a function that is symmetric,
positive, respects the triangle inequality and is equal to zero if (and only
if) the two considered distributions are strictly identical. We will use
information measures that belong to either Csiszar $f$-divergences or that are
distances.

Name | Notation | Domain | Expression
---|---|---|---
$\alpha$-divergence (Csiszár 1967) | $\mathcal{D}_{\alpha}$ | $\alpha\not\in\\{0,1\\}$ | $\frac{1}{\alpha(\alpha-1)}(1-\sum q_{i}^{1-\alpha}p_{i}^{\alpha})$
$\gamma$ divergence (Fujisawa and Eguchi 2008) | $\mathcal{D}_{\gamma}^{\beta}$ | $\beta\not\in\\{0,-1\\}$ | $\frac{1}{\beta(\beta+1)}\log\sum p_{i}^{\beta+1}+\frac{1}{\beta+1}\log\sum q_{i}^{\beta+1}-\frac{1}{\beta}\log\sum p_{i}q_{i}^{\beta}$
AB Divergence (Cichocki, Cruces, and Amari 2011) | $\mathcal{D}_{sAB}^{\alpha,\beta}$ |  $(\alpha,\beta)\in(\mathbb{R}^{*})^{2}$ $\beta+\alpha\neq 0$ | $\frac{1}{\beta(\beta+\alpha)}\log\sum p_{i}^{\beta+\alpha}+\frac{1}{\beta+\alpha}\log\sum q_{i}^{\beta+\alpha}-\frac{1}{\beta}\log\sum p_{i}^{\alpha}q_{i}^{\beta}$
$\mathcal{L}_{1}$ distance | $\mathcal{L}_{1}$ |  | $\sum|p_{i}-q_{i}|$
$\mathcal{L}_{2}$ distance | $\mathcal{L}_{2}$ |  | $\sqrt{\sum(p_{i}-q_{i})^{2}}$
$\mathcal{L}_{\infty}$ distance | $\mathcal{L}_{\infty}$ |  | $\max_{i}|p_{i}-q_{i}|$
Fisher-Rao distance | $R$ |  | $\frac{2}{\pi}\arccos{\sum\sqrt{p_{i}\times q_{i}}}$

Table 1: Expression of the divergences (upper group) and distance between two
positives measures $\mathbf{p}=(p_{1},\cdots,p_{N})$ and
$\mathbf{q}=(q_{1},\cdots,q_{N})$ as well as the definition domain (see (Regli
and Silva 2018)). We omit the index in the summations.

#### Divergence measures

Various divergence measures have been proposed for a large variety of
applications (Basseville 2013). The full expression of the studied divergences
can be found in Table 1. We focus here on three families of divergences
$\alpha$ Divergences, $\gamma$ Divergences and $AB$ Divergences. Note that
there exist other families of divergences such as Bregman divergence (Bregman
1967), $\beta$ divergences (Basu et al. 1998), Chernoff divergence (Chernoff
et al. 1952).
$\alpha$-Divergences.  This divergence was introduced by Rényi et al. (1961)
and are a special case of the $f$-divergences (Csiszár 1967). They are widely
used in variational inference (Li and Turner 2016) and closely related to
Rényi divergences but are not a special case. From Table 1 we note special
cases of $\alpha$-Divergences: (i) Kullback-Leiber (KL) is recovered by
letting $\alpha\rightarrow 1$, (ii) Hellinger distance (Hellinger 1909)
follows by choosing $\alpha=0.5$. For this family, $\alpha$ weights the
influence of $\frac{p}{q}$.
$\gamma$-Divergences. This divergence has been introduced by (Fujisawa and
Eguchi 2008) as a scale-invariant modification of the robust
$\beta$-divergences.333In our setting we work we normalised distributions,
thus scale invariance is not a mandatory property. It is worth mentioning as
it could cause practical issues when optimising our metric. For the $\gamma$
divergences the parameter $\beta$ is used to control the importance of the
element of small probabilities (e.g., outliers in some scenarios, tokens with
low probability in our case). If $\beta>1$, the importance of large $q_{i}$ is
reduced which gives more weights to the outliers. Special cases include the
${L}_{2}$ distance (i.e., $\beta=2$) and KL divergence (i.e.,
$\beta\rightarrow 1$).
$AB$-Divergences. The family of $AB$-divergences is flexible and allows to
respectively control the mass coverage or the robustness. Cichocki, Cruces,
and Amari (2011) propose to use $AB$ divergences. As can be seen in Table 1
these divergences have two parameters $\alpha,\beta$. It allows to tune the
mass coverage and the robustness independently. The $\beta$-divergence is
obtained by choosing $\alpha=1,\beta\in\mathbb{R}$.
From information divergences to discrimination. For our application, we would
like to produce a metric between two texts regardless of the source (system or
human). Thus we are interested in symmetric divergence: such divergences are
called discrimination. To obtain discrimination two tricks are commonly
applied either the Jeffrey’s symmetrization, which averages
$\textrm{KL}(p\|q)$ and $\textrm{KL}(q\|p)$), or the Jensen’s symmetrization,
which averages $\textrm{KL}(p\|\frac{p+q}{2})$ and
$\textrm{KL}(q\|\frac{p+q}{2})$. We choose to use Jeffreys symmetrization as
it does not require computing $\frac{p+q}{2}$. The symmetric $KL$ with
Jeffrey’s symmetrization is denoted $JS$.

#### Distances

$\mathcal{L}_{p}$ distances. The $\mathcal{L}_{p}$ distances
$p\in\mathbb{R}_{+}$ can be used to measure the similarity between two
distributions and we restrict ourselves to $p\in\\{1,2,+\infty\\}$.
Fisher-Rao distance. The Fisher-Rao distance ($R$) represents the Geodesic
Distance (Rao 1987) between two distributions. Interestingly, this distance
remains overlooked in the ML community but has been recently used to achieve
robustness against adversarial attacks (Picot et al. 2021).

#### Connection with string matching metrics.

We adopt the following notations
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x_{i}};\theta;\mathcal{T})=[p_{0},\cdots,p_{|\Omega|}]$
and
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}};\theta;\mathcal{T})=[q_{0},\cdots,q_{|\Omega|}]$.
First, let us consider two texts $\boldsymbol{x_{i}},\boldsymbol{y_{i}^{s}}$
such that
$\boldsymbol{x_{i}}\overset{\mathcal{I}}{\approx}\boldsymbol{y_{i}^{s}}$.
InfoLM with $\mathcal{L}_{\infty}$ is closed to $0$ if $\forall
i\in[1,|\Omega|]\quad p_{i}\approx q_{i}$. It means that all likely tokens
(according to the PMLM) when considering $\boldsymbol{x_{i}}$ are also likely
when considering $\boldsymbol{y_{i}^{s}}$. For string matching metrics, it
corresponds to a _perfect match_ between $\boldsymbol{x_{i}}$ and
$\boldsymbol{y_{i}^{s}}$. Second, let us consider
$\boldsymbol{x_{i}},\boldsymbol{y_{i}^{s}}$ such that
$\mathcal{I}\left[p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x_{i}};\theta;\mathcal{T}),p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}};\theta;\mathcal{T})\right]\approx
1$ (dissimilar texts) and a measure of information that relies on product of
$p_{i}$ and $q_{i}$ (e.g Fisher-Rao). In this case, $\forall
i\in[1,|\Omega|]\quad p_{i}\times q_{i}\approx 0$ thus all likely tokens when
considering $\boldsymbol{x_{i}}$ are unlikely when considering
$\boldsymbol{y_{i}^{s}}$ (the converse it true as well). For string matching
metrics this corresponds to _no match_ among the sub-strings of
$\boldsymbol{x_{i}}$ and $\boldsymbol{y_{i}^{s}}$.

##  4 Experimental Frameworks

In this section, we describe our experimental setting. We present the tasks
and the baselines metrics use for each task.

###  4.1 Text summarization

Text summarization aims at compressing long texts into fluent, short sentences
that preserve the salient information.
Datasets. To compare the different metrics previous work (Bhandari et al.
2020; Fabbri et al. 2020) either relies on the TAC datasets (Dang and
Owczarzak 2008; McNamee and Dang 2009) or on new summarization datasets
extracted from CNN/DailyMail (Nallapati et al. 2016). As pointed out by
Peyrard (2019); Bhandari et al. (2020), TAC datasets are old and contain flaws
(e.g systems used to generate summaries were of poor quality), we choose to
work with the newly assemble dataset from CNN/Daily News proposed in Bhandari
et al. (2020). This dataset gathers 11,490 summaries and annotations are
carried using the pyramid method (Nenkova and Passonneau 2004).
Metrics. For text summarization, perhaps the most known metrics are ROUGE and
its extensions (Ng and Abrecht 2015), or METEOR and its variants (Denkowski
and Lavie 2014). Recently, a new set of metrics (e.g BERTSCORE, MOVERSCORE)
have been applied to text summarization.

###  4.2 Data2Text generation

Prior works mainly rely on two task-oriented dialogue datasets (i.e., BAGEL
(Mairesse et al. 2010), SFHOTEL (Wen et al. 2015)). As sentence generated in
these data-sets are unlikely to be representative of the progress of recent
NLG systems we instead rely on a different dataset coming from the WebNLG2020
challenge (Gardent et al. 2017). Given the following example of triple:
(John_Blaha birthDate 1942_08_26) (John_Blaha birthPlace San_Antonio)
(John_E_Blaha job Pilot) the goal is to generate John Blaha, born in San
Antonio on 1942-08-26, worked as a pilot.
Annotations. The WebNLG task is evaluated by human annotators along four
different axes: (1) Data Coverage: Are all the descriptions presented in the
data included in the text? (2) Relevance: Does the text contains only
predicates found in the data? (3) Correctness: Are predicates found in the
data correctly mentioned and adequately introduced? (4) Text structure: Is the
produced text well-structured, grammatically correct and written in acceptable
English?, (5) Fluency: Does the text progress naturally? Is it easy to
understand? Is it a coherent whole?
Metrics. For this task, organisers rely on untrained metrics (e.g. BLEU,
METEOR, TER, BERTSCORE) to compare the performance of the candidate systems.
Thus, we will focus on system-level correlation.

##  5 Numerical Results

|  |  |
---|---|---|---
|  |  |

(a) Abs - Text (b) Ext - Text

(c) Abs - Sys (d) Ext - Sys

(e) Abs - Text (f) Ext - Text

(g) Abs - Sys (h) Ext - Sys

Figure 1: Results of the correlation between metrics and human judgments on
the CNN dataset. First raw reports correlation as measured by the Person ($r$)
and second raw focus on Kendall ($\tau$) coefficient. In this experiment,
parameter are optimized for each criterion. The results on with Spearman
($\rho$) can be found in Section B.5).

In this section, we study the performance of InfoLM on both text summarization
and data2text generation.

###  5.1 Results on Text Summarization

General Analysis. Figure 8 gathers the results of the correlation study
between scores produced by different metrics and human judgement (i.e. pyramid
score). We can reproduce results from Bhandari et al. (2020). We observe a
different behavior depending on the type of systems to be evaluated (e.g.,
abstractive or extractive) and the chosen correlation coefficient. We observe
that InfoLM with $\mathcal{D}_{AB}$ or with $\mathcal{R}$ outperforms other
BERT-based metrics such as MOVERSCORE or BERTSCORE (e.g., it is worth noting
that both metrics perform poorly at the text or system-level when considering
outputs from extractive systems). $\mathcal{D}_{AB}$ largely outperforms
n-gram matching metrics (e.g., ROUGE metrics) on all datasets when measuring
correlation with the Kendall $\tau$ and in almost all configurations (except
when considering abstractive outputs at the system level) when using the
Pearson $r$. It is worth noting the overall good performance of the parameter-
free Fisher-Rao distance.
Choice of information geometric measure for InfoLM. In Figure 8, we can
observe two different types of groups depending on the global behaviour. First
we notice that using $\mathcal{L}_{p}$, $p\in\\{1,2,\dots,+\infty\\}$ leads to
poor performances in many configurations. Good performance of
$\mathcal{L}_{\infty}$ in some configurations is surprising as
$\mathcal{L}_{\infty}$ is extremely selective (i.e. $\mathcal{L}_{\infty}$
computes $\max_{i}|p_{i}-q_{i}|$). As output produced by the PMLM is sparse,
$\max_{i}|p_{i}-q_{i}|$ correspond to one likely word in one sentence and not
likely at all in the other. The second group gathers $JS$, $\mathcal{R}$,
$\mathcal{D}_{\alpha}$, $\mathcal{D}_{\beta}$ and $\mathcal{D}_{AB}$ and
achieves the best performance overall. $\mathcal{D}_{\alpha}$ and
$\mathcal{D}_{AB}$ achieve similar performance suggesting that the flexibility
(e.g., robustness to outliers) introduced by the $\beta$ parameter in
$\mathcal{D}_{AB}$ is not useful in our task. This observation is strengthened
by the lower performance of $\mathcal{D}_{\beta}$. The difference of results
between the two measures is due to the flexibility introduced by $\alpha$
(i.e., $\alpha$ controls the relative importance of the ration
$\frac{p_{i}}{q_{i}}$) which can be interpreted in our case as the ability to
control the importance attributed to less likely words (Jalalzai et al. ).
Takeaways. The best performing metric is obtained with $\mathcal{D}_{AB}$. The
Fisher-Rao distance, denoted by $\mathcal{R}$, achieves good performance in
many scenarios and has the advantage to be parameter-free.

###  5.2 Results on Data2Text

Global Analysis: Table 3 gathers results of the correlation analysis of the
metrics with human judgements following the five different axes. We observe
that the five considered criteria of annotations are not independent: text
structure and fluency achieve a strong correlation coefficient ($>98$).
Additionally, all metrics achieve similar results when the correlation is
computed on these two criteria. We observe that the best performing group of
metric is based on InfoLM followed by metrics based on continuous
representation from BERT (i.e., MOVERSCORE and BERTSCORE) followed by N-gram
matching metrics. Regarding correctness, data coverage and relevance, we
observe that both $\mathcal{D}_{AB}$ and $\mathcal{D}_{\alpha}$ achieve the
best results on almost all correlation coefficients. On data coverage, InfoLM
achieves improvement up to $17$ points in correlation compared to both BERT
based or N-gram matching metrics. Regarding fluency and text structure,
Fisher-Rao distance works better and slightly outperforms the second-best
performing metric, namely BERTSCORE.
Takeaways. Similar to summarisation, we observe very low correlation for
${L}_{p}$, $p\in\\{1,2,\dots,+\infty\\}$. We also observe that
$\beta$-divergences achieve lower results than both $\alpha$ and $AB$
divergences suggesting that, as noticed for summarisation, robustness to
unlikely words (i.e., outliers) is less relevant for our task.

| Correctness | Data Coverage | Fluency | Relevance | Text Structure
---|---|---|---|---|---
Metric | $r$ | $\rho$ | $\tau$ | $r$ | $\rho$ | $\tau$ | $r$ | $\rho$ | $\tau$ | $r$ | $\rho$ | $\tau$ | $r$ | $\rho$ | $\tau$
Correct | 100.0 | 100.0 | 100.0 | 97.6 | 85.2 | 73.3 | 80.0 | 81.1 | 61.6 | 99.1 | 89.7 | 75.0 | 80.1 | 80.8 | 60.0
DataC | 85.2 | 97.6 | 73.3 | 100.0 | 100.0 | 100.0 | 71.8 | 51.7 | 38.3 | 96.0 | 93.8 | 81.6 | 71.6 | 51.4 | 36.6
Fluency | 81.1 | 80.0 | 61.6 | 71.8 | 51.7 | 38.3 | 100.0 | 100.0 | 100.0 | 77.0 | 61.4 | 46.6 | 99.5 | 99.7 | 98.3
Relev | 89.7 | 99.1 | 75.0 | 96.0 | 93.8 | 81.6 | 77.0 | 61.4 | 46.6 | 100.0 | 100.0 | 100.0 | 77.2 | 61.1 | 45.0
TextS | 80.8 | 80.1 | 60.0 | 71.6 | 51.4 | 36.6 | 99.5 | 99.7 | 98.3 | 77.2 | 61.1 | 45.0 | 100.0 | 100.0 | 100.0
$\mathcal{D}_{AB}$ | 88.8 | 89.3 | 76.6 | 81.8 | 82.6 | 70.0 | 86.6 | 92.0 | 76.6 | 89.8 | 87.9 | 73.3 | 86.6 | 91.4 | 75.0
$\mathcal{D}_{\alpha}$ | 88.8 | 89.3 | 76.6 | 81.8 | 82.6 | 70.0 | 86.6 | 92.0 | 76.6 | 89.8 | 87.9 | 73.3 | 86.6 | 91.4 | 75.0
$\mathcal{D}_{\beta}$ | 81.4 | 50.0 | 71.6 | 48.4 | 79.7 | 65.0 | 44.8 | 84.7 | 76.6 | 49.3 | 72.3 | 60.0 | 48.0 | 83.8 | 75.0
$\mathcal{L}_{1}$ | 75.2 | 33.8 | 61.6 | 32.4 | 53.8 | 40.0 | 22.7 | 83.5 | 73.3 | 32.2 | 57.9 | 45.0 | 25.6 | 83.2 | 71.6
$\mathcal{R}$ | 89.7 | 86.0 | 75.0 | 78.7 | 70.5 | 51.6 | 93.3 | 95.7 | 85.3 | 87.6 | 84.4 | 70.0 | 92.4 | 93.8 | 81.6
JS | 79.4 | 81.1 | 70.0 | 69.3 | 75.5 | 60.0 | 89.4 | 91.4 | 75.0 | 81.7 | 70.5 | 60.0 | 91.9 | 91.1 | 73.3
BertS | 85.5 | 83.4 | 73.3 | 74.7 | 68.2 | 53.3 | 92.3 | 95.5 | 85.0 | 83.3 | 79.4 | 65.0 | 91.9 | 95.0 | 83.3
MoverS | 84.1 | 84.1 | 73.3 | 78.7 | 66.2 | 53.3 | 91.2 | 92.1 | 78.3 | 82.1 | 77.4 | 65.0 | 90.1 | 91.4 | 76.3
BLEU | 77.6 | 66.3 | 60.0 | 55.7 | 50.2 | 36.6 | 89.4 | 90.5 | 78.3 | 63.0 | 65.2 | 51.6 | 88.5 | 89.1 | 76.6
R-1 | 80.6 | 65.0 | 65.0 | 61.1 | 59.6 | 48.3 | 76.5 | 76.3 | 60.3 | 64.3 | 69.2 | 56.7 | 75.9 | 77.5 | 58.3
METEOR | 86.5 | 66.3 | 70.0 | 77.3 | 50.2 | 46.6 | 86.7 | 90.5 | 78.3 | 82.1 | 65.2 | 58.6 | 86.2 | 89.1 | 76.6
TER | 79.6 | 78.3 | 58.0 | 69.7 | 58.2 | 38.0 | 89.1 | 93.5 | 80.0 | 75.0 | 70.2 | 77.6 | 89.5 | 91.1 | 78.6

Table 2: Correlation at the system level with human judgement along five
different axis: correctness, data coverage, fluency, relevance and text
structure for the WebNLG task. Best results by group are underlined, overall
best results are bolted.

###  5.3 Further Analysis

#### Correlation between metrics

In this experiment, we complete our global analysis by comparing the scores
obtained by the different metrics with each other. We want to gain an
understanding of how different our metric is from other metrics and how the
choice of information geometric measures affects the predictions. Figure 2
gathers the results of the experiment. We observe a high correlation ($r>88$)
between $\mathcal{D}_{\alpha}$, $\mathcal{D}_{\beta}$, $\mathcal{D}_{AB}$ and
$\mathcal{R}$444Note that these metrics consider the product of $p_{i}$ and
$q_{i}$.. Interestingly, we observe a lower correlation ($r\approx 70$) with
BERTSCORE and N-gram matching metrics, e.g., ROOGE) whereas BERTSCORE achieves
a stronger correlation with ROUGE ($r\approx 80$).
Takeaways. Through the correlation analysis in Figure 2, we observe the impact
of different geometry on InfoLM predictions. The correlation analysis shows
that the prediction of InfoLM when using $\mathcal{D}_{\alpha}$,
$\mathcal{D}_{\beta}$, $\mathcal{D}_{AB}$ and $\mathcal{R}$ are highly
correlated and as illustrated by previous experience achieve high correlation
scores which we believe validate our approach. It is worth noting that
$\mathcal{R}$ requires no tuning as it is parameter free.

Figure 2: Pearson correlation at the system level between metrics when
considering abstractive system outputs.

#### Score Distributions

In Figure 3, we study the text score distribution of different metrics on
abstractive summary. The ideal metric would mimic the human score distribution
(i.e. $Pyr.$) and be able to distinguish between good and bad quality
summaries. The results show that ROUGE and BLEU struggle to distinguish
between between good quality ($Pyr.\geq 0.5$) low quality summaries ($Pyr.\leq
0.5$) which has been reported in Peyrard (2019). We observe that
$\mathcal{D}_{AB}$, $\mathcal{R}$ and $JS$ metrics are able to make the
distinction. Interestingly, as $p$ increases and the $\mathcal{L}_{p}$
distances become more selective (i.e. focus on one word solely), the
$\mathcal{L}_{p}$ distances struggle to distinguish low from high scoring
summaries.

Takeaways. InfoLM when combined with $\mathcal{D}_{AB}$, $\mathcal{R}$ and
$JS$ is able to distinguish high-scoring from low scoring summaries.

Figure 3: Score distribution of text score when considering abstractive system
outputs. $Pyr.$ stands for pyramide score.

#### Temperature Calibration

To study the impact of calibration, we choose to work on system-level
correlation and report in Figure 4 the achieved the correlation measured by
the different coefficients. We limit our study to the Fisher-Rao distance as
it is a parameter-free metric and is among the best-performing metrics of
InfoLM. Due to space constraints, we report the result on extractive systems
only (extractive systems can be found in Figure 5).
Takeaways. Fisher-Rao only considers product $p_{i}\times q_{i}$ thus as
$\mathcal{T}$ increases and the predicted probability of the PMLM becomes more
uniform more words are considered and the aggregated distributions become
richer in terms of considered words. It is worth noting that when changing the
temperature we observe a smooth change in correlation and observe an optimal
temperature $T$ which is reached for $T\in[1,2]$. It suggests that InfoLM
benefits from a PMLM that is not too selective (case $T\ll 1$). For a specific
application, the temperature of InfoLM can be tuned to improve correlation and
InfoLM will likely benefit from well-calibrated PMLM.

Figure 4: Impact of Calibration on system-level correlation for summarization.

##  6 Summary and Concluding Remarks

In this work, we presented InfoLM that does not require training and it is
among the first metrics computing the similarity between two discrete
probability distributions over the vocabulary (which is similar to string-
based metrics) but also leverages the recent advance in language modeling
thanks to a PMLM. Our experiments on both summarization and data2text
generation demonstrate the validity of our approach. Among available contrast
measures, the Fisher-Rao distance is parameter-free and thus, it is easy to
use in practice while the $AB$-Divergence achieves better results but requires
to select $\alpha$ and $\beta$. Future work includes extending our metrics to
new tasks such as translation, image captioning and paraphrase detection, as
well as investigating others information geometric measures (e.g., Wasserstein
distance (Staerman et al. 2021a), or other types of divergences and Mutual
Information (Colombo, Clavel, and Piantanida 2021)) to obtain better
correlation with different attributes.

##  7 Acknowledgments

This work was also granted access to the HPC resources of IDRIS under the
allocation 2021-AP010611665 as well as under the project 2021-101838 made by
GENCI.

## References

  * Auer et al. (2007) Auer, S.; Bizer, C.; Kobilarov, G.; Lehmann, J.; Cyganiak, R.; and Ives, Z. 2007\.  Dbpedia: A nucleus for a web of open data.  In _The semantic web_ , 722–735. Springer.
  * Banerjee and Lavie (2005) Banerjee, S.; and Lavie, A. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of ACL workshop_.
  * Basseville (2013) Basseville, M. 2013.  Divergence measures for statistical data processing—An annotated bibliography.  _Signal Processing_ , 93(4): 621–633.
  * Basu et al. (1998) Basu, A.; Harris, I. R.; Hjort, N. L.; and Jones, M. 1998.  Robust and efficient estimation by minimising a density power divergence.  _Biometrika_ , 85(3): 549–559.
  * Beck et al. (1991) Beck, I. L.; McKeown, M. G.; Sinatra, G. M.; and Loxterman, J. A. 1991.  Revising social studies text from a text-processing perspective: Evidence of improved comprehensibility.  _Reading research quarterly_ , 251–276.
  * Belz and Reiter (2006) Belz, A.; and Reiter, E. 2006.  Comparing automatic and human evaluation of NLG systems.  In _11th Conference of the EACL_.
  * Bhandari et al. (2020) Bhandari, M.; Gour, P.; Ashfaq, A.; Liu, P.; and Neubig, G. 2020.  Re-evaluating evaluation in text summarization.
  * Braverman et al. (2020) Braverman, M.; Chen, X.; Kakade, S.; Narasimhan, K.; Zhang, C.; and Zhang, Y. 2020\.  Calibration, Entropy Rates, and Memory in Language Models.  In III, H. D.; and Singh, A., eds., _Proceedings of the 37th ICML_ , volume 119 of _PMLR_ , 1089–1099. PMLR.
  * Bregman (1967) Bregman, L. M. 1967.  The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming.  _USSR computational mathematics and mathematical physics_ , 7(3).
  * Chapuis et al. (2021) Chapuis, E.; Colombo, P.; Labeau, M.; and Clavel, C. 2021.  Code-switched inspired losses for generic spoken dialog representations.  _arXiv preprint arXiv:2108.12465_.
  * Chapuis et al. (2020) Chapuis, E.; Colombo, P.; Manica, M.; Labeau, M.; and Clavel, C. 2020.  Hierarchical pre-training for sequence labelling in spoken dialog.  _arXiv preprint arXiv:2009.11152_.
  * Chatzikoumi (2020) Chatzikoumi, E. 2020.  How to evaluate machine translation: A review of automated and human metrics.  _Natural Language Engineering_ , 26(2): 137–161.
  * Chen and Bansal (2018) Chen, Y.-C.; and Bansal, M. 2018.  Fast abstractive summarization with reinforce-selected sentence rewriting.  _arXiv preprint arXiv:1805.11080_.
  * Chernoff et al. (1952) Chernoff, H.; et al. 1952.  A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations.  _The Annals of Mathematical Statistics_ , 23(4).
  * Chow, Specia, and Madhyastha (2019) Chow, J.; Specia, L.; and Madhyastha, P. 2019.  WMDO: Fluency-based Word Mover’s Distance for Machine Translation Evaluation.  In _Proceedings of the Fourth WMT (Volume 2: Shared Task Papers, Day 1)_ , 494–500. Florence, Italy: ACL.
  * Cichocki, Cruces, and Amari (2011) Cichocki, A.; Cruces, S.; and Amari, S.-i. 2011.  Generalized alpha-beta divergences and their application to robust nonnegative matrix factorization.  _Entropy_ , 13(1): 134–170.
  * Clark, Celikyilmaz, and Smith (2019) Clark, E.; Celikyilmaz, A.; and Smith, N. A. 2019.  Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts.  In _Proceedings of the 57th ACL_ , 2748–2760. Florence, Italy: ACL.
  * Colombo et al. (2021a) Colombo, P.; Chapuis, E.; Labeau, M.; and Clavel, C. 2021a.  Improving multimodal fusion via mutual dependency maximisation.  _arXiv preprint arXiv:2109.00922_.
  * Colombo et al. (2020) Colombo, P.; Chapuis, E.; Manica, M.; Vignon, E.; Varni, G.; and Clavel, C. 2020\.  Guiding attention in Sequence-to-sequence models for Dialogue Act prediction.  _arXiv preprint arXiv:2002.08801_.
  * Colombo, Clavel, and Piantanida (2021) Colombo, P.; Clavel, C.; and Piantanida, P. 2021.  A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations.
  * Colombo et al. (2021b) Colombo, P.; Staerman, G.; Clavel, C.; and Piantanida, P. 2021b.  Automatic text evaluation through the lens of wasserstein barycenters.  _arXiv preprint arXiv:2108.12463_.
  * Colombo et al. (2019) Colombo, P.; Witon, W.; Modi, A.; Kennedy, J.; and Kapadia, M. 2019.  Affect-driven dialog generation.
  * Colombo et al. (2021c) Colombo, P.; Yang, C.; Varni, G.; and Clavel, C. 2021c.  Beam Search with Bidirectional Strategies for Neural Response Generation.
  * Csiszár (1967) Csiszár, I. 1967.  Information-type measures of difference of probability distributions and indirect observation.  _studia scientiarum Mathematicarum Hungarica_ , 2: 229–318.
  * Dang and Owczarzak (2008) Dang, H. T.; and Owczarzak, K. 2008.  Overview of the TAC 2008 Update Summarization Task.  In _TAC_.
  * Denkowski and Lavie (2014) Denkowski, M.; and Lavie, A. 2014.  Meteor universal: Language specific translation evaluation for any target language.  In _Proceedings of the ninth workshop on statistical machine translation_ , 376–380.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dinkar et al. (2020) Dinkar, T.; Colombo, P.; Labeau, M.; and Clavel, C. 2020.  The importance of fillers for text representations of speech transcripts.  _arXiv preprint arXiv:2009.11340_.
  * Doddington (2002) Doddington, G. 2002.  Automatic evaluation of machine translation quality using n-gram co-occurrence statistics.  In _Proceedings of the second international conference on Human Language Technology Research_ , 138–145.
  * Dong et al. (2019) Dong, L.; Yang, N.; Wang, W.; Wei, F.; Liu, X.; Wang, Y.; Gao, J.; Zhou, M.; and Hon, H.-W. 2019.  Unified language model pre-training for natural language understanding and generation.  _arXiv preprint arXiv:1905.03197_.
  * Fabbri et al. (2020) Fabbri, A. R.; Kryściński, W.; McCann, B.; Xiong, C.; Socher, R.; and Radev, D. 2020.  Summeval: Re-evaluating summarization evaluation.
  * Ferreira et al. (2020) Ferreira, T.; Gardent, C.; Ilinykh, N.; van der Lee, C.; Mille, S.; Moussallem, D.; and Shimorina, A. 2020.  The 2020 bilingual, bi-directional webnlg+ shared task overview and evaluation results (webnlg+ 2020).  In _Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)_.
  * Ferreira et al. (2018) Ferreira, T. C.; Moussallem, D.; Krahmer, E.; and Wubben, S. 2018.  Enriching the WebNLG corpus.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , 171–176.
  * Fujisawa and Eguchi (2008) Fujisawa, H.; and Eguchi, S. 2008.  Robust parameter estimation with a small bias against heavy contamination.  _Journal of Multivariate Analysis_ , 99(9): 2053–2081.
  * Galley et al. (2015) Galley, M.; Brockett, C.; Sordoni, A.; Ji, Y.; Auli, M.; Quirk, C.; Mitchell, M.; Gao, J.; and Dolan, B. 2015.  deltableu: A discriminative metric for generation tasks with intrinsically diverse targets.  _arXiv preprint arXiv:1506.06863_.
  * Ganesan (2018) Ganesan, K. 2018.  Rouge 2.0: Updated and improved measures for evaluation of summarization tasks.  _arXiv preprint arXiv:1803.01937_.
  * Garcia et al. (2019) Garcia, A.; Colombo, P.; Essid, S.; d’Alché Buc, F.; and Clavel, C. 2019.  From the token to the review: A hierarchical multimodal approach to opinion mining.
  * Gardent et al. (2017) Gardent, C.; Shimorina, A.; Narayan, S.; and Perez-Beltrachini, L. 2017.  Creating training corpora for nlg micro-planning.  In _55th ACL (ACL)_.
  * Gehrmann, Deng, and Rush (2018) Gehrmann, S.; Deng, Y.; and Rush, A. 2018.  Bottom-Up Abstractive Summarization.  In _Proceedings of the 2018 EMNLP_ , 4098–4109. Brussels, Belgium: ACL.
  * Graham and Baldwin (2014) Graham, Y.; and Baldwin, T. 2014.  Testing for significance of increased correlation with human judgment.  In _Proceedings of the 2014 EMNLP (EMNLP)_ , 172–176.
  * Guo et al. (2017) Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q. 2017.  On calibration of modern neural networks.  In _ICML_.
  * Guo and Hu (2019) Guo, Y.; and Hu, J. 2019.  Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation.  In _Proceedings of the Fourth WMT (Volume 2: Shared Task Papers, Day 1)_ , 501–506. Florence, Italy: ACL.
  * Hellinger (1909) Hellinger, E. 1909.  Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen.  _Journal für die reine und angewandte Mathematik_ , 1909(136).
  * Hovy (1999) Hovy, E. H. 1999.  Toward finely differentiated evaluation metrics for machine translation.  In _Proceedings of the EAGLES Workshop on Standards and Evaluation_.
  * (45) Jalalzai, H.; Colombo, P.; Clavel, C.; Gaussier, É.; Varni, G.; Vignon, E.; and Sabourin, A. ????  Heavy-tailed representations, text polarity classification & data augmentation.
  * Kedzie, McKeown, and Daume III (2018) Kedzie, C.; McKeown, K.; and Daume III, H. 2018.  Content selection in deep learning models of summarization.  _arXiv preprint arXiv:1810.12343_.
  * Kendall (1938) Kendall, M. G. 1938.  A new measure of rank correlation.  _Biometrika_ , 30(1/2): 81–93.
  * Koehn (2009) Koehn, P. 2009.  _Statistical machine translation_.  Cambridge University Press.
  * Kusner et al. (2015) Kusner, M.; Sun, Y.; Kolkin, N.; and Weinberger, K. 2015.  From word embeddings to document distances.  In _ICML_.
  * Leusch, Ueffing, and Ney (2006) Leusch, G.; Ueffing, N.; and Ney, H. 2006.  CDER: Efficient MT Evaluation Using Block Movements.  In _11th Conference of the EACL_. Trento, Italy: ACL.
  * Leusch et al. (2003) Leusch, G.; Ueffing, N.; Ney, H.; et al. 2003.  A novel string-to-string distance measure with applications to machine translation evaluation.  In _Proceedings of Mt Summit IX_ , 240–247.
  * Lewis et al. (2019) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li and Turner (2016) Li, Y.; and Turner, R. E. 2016.  R$\backslash$’enyi Divergence Variational Inference.  _arXiv preprint arXiv:1602.02311_.
  * Lin (2004) Lin, C.-Y. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text Summarization Branches Out_ , 74–81. Barcelona, Spain: ACL.
  * Liu and Lapata (2019) Liu, Y.; and Lapata, M. 2019.  Text summarization with pretrained encoders.  _arXiv preprint arXiv:1908.08345_.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ma et al. (2017) Ma, Q.; Graham, Y.; Wang, S.; and Liu, Q. 2017.  Blend: a novel combined MT metric based on direct assessment—casict-dcu submission to WMT17 metrics task.  In _Proceedings of the second WMT_ , 598–603.
  * Mairesse et al. (2010) Mairesse, F.; Gasic, M.; Jurcicek, F.; Keizer, S.; Thomson, B.; Yu, K.; and Young, S. 2010.  Phrase-based statistical language generation using graphical models and active learning.  In _Proceedings of the 48th ACL_ , 1552–1561.
  * Mani (2001) Mani, I. 2001.  _Automatic summarization_ , volume 3.  John Benjamins Publishing.
  * McNamee and Dang (2009) McNamee, P.; and Dang, H. T. 2009.  Overview of the TAC 2009 knowledge base population track.  In _Text Analysis Conference (TAC)_ , volume 17, 111–113.
  * Mehri, Eric, and Hakkani-Tur (2020) Mehri, S.; Eric, M.; and Hakkani-Tur, D. 2020.  Example-Driven Intent Prediction with Observers.  _arXiv preprint arXiv:2010.08684_.
  * Melamed, Green, and Turian (2003) Melamed, I. D.; Green, R.; and Turian, J. 2003.  Precision and recall of machine translation.  In _Companion Volume of the Proceedings of HLT-NAACL 2003-Short Papers_ , 61–63.
  * Mellish and Dale (1998) Mellish, C.; and Dale, R. 1998.  Evaluation in the context of natural language generation.  _Computer Speech & Language_, 12(4): 349–373.
  * Mikolov et al. (2013) Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013.  Efficient Estimation of Word Representations in Vector Space.  _arXiv preprint arXiv:1301.3781_.
  * Nallapati et al. (2016) Nallapati, R.; Zhou, B.; Gulcehre, C.; Xiang, B.; et al. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_.
  * Narayan, Cohen, and Lapata (2018) Narayan, S.; Cohen, S. B.; and Lapata, M. 2018.  Ranking sentences for extractive summarization with reinforcement learning.  _arXiv preprint arXiv:1802.08636_.
  * Nenkova and Passonneau (2004) Nenkova, A.; and Passonneau, R. J. 2004.  Evaluating content selection in summarization: The pyramid method.  In _Proceedings of HLT-NACCL 2004_ , 145–152.
  * Ng and Abrecht (2015) Ng, J.-P.; and Abrecht, V. 2015.  Better summarization evaluation with word embeddings for rouge.  _arXiv preprint arXiv:1508.06034_.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002.  Bleu: a Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th ACL_ , 311–318. Philadelphia, Pennsylvania, USA: ACL.
  * Paszke et al. (2017) Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017.  Automatic differentiation in pytorch.
  * Perez-Beltrachini, Sayed, and Gardent (2016) Perez-Beltrachini, L.; Sayed, R.; and Gardent, C. 2016.  Building rdf content for data-to-text generation.  In _The 26th International Conference on Computational Linguistics (COLING 2016)_.
  * Peters et al. (2018) Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018.  Deep contextualized word representations.  In _Proc. of NAACL_.
  * Peyrard (2019) Peyrard, M. 2019.  Studying summarization evaluation metrics in the appropriate scoring range.  In _Proceedings of the 57th ACL_ , 5093–5100.
  * Picot et al. (2021) Picot, M.; Messina, F.; Boudiaf, M.; Labeau, F.; Ben Ayed, I.; and Piantanida, P. 2021.  Adversarial Robustness via Fisher-Rao Regularization.  _arXiv preprint arXiv:_.
  * Platt et al. (1999) Platt, J.; et al. 1999.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in large margin classifiers_ , 10(3): 61–74.
  * Post (2018) Post, M. 2018.  A call for clarity in reporting BLEU scores.  _arXiv preprint arXiv:1804.08771_.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019.  Language Models are Unsupervised Multitask Learners.  _OpenAI Blog_ , 1(8).
  * Raffel et al. (2019) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rao (1987) Rao, C. R. 1987.  Differential metrics in probability spaces.  _Differential geometry in statistical inference_ , 10: 217–240.
  * Regli and Silva (2018) Regli, J.-B.; and Silva, R. 2018.  Alpha-beta divergence for variational inference.  _arXiv preprint arXiv:1805.01045_.
  * Reiter and Belz (2009) Reiter, E.; and Belz, A. 2009.  An investigation into the validity of some metrics for automatically evaluating natural language generation systems.  _Computational Linguistics_ , 35(4): 529–558.
  * Rényi et al. (1961) Rényi, A.; et al. 1961.  On measures of entropy and information.  In _Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics_. The Regents of the University of California.
  * Rücklé et al. (2018) Rücklé, A.; Eger, S.; Peyrard, M.; and Gurevych, I. 2018.  Concatenated power mean word embeddings as universal cross-lingual sentence representations.
  * Sanh et al. (2019) Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * See, Liu, and Manning (2017) See, A.; Liu, P. J.; and Manning, C. D. 2017.  Get to the point: Summarization with pointer-generator networks.  _arXiv preprint arXiv:1704.04368_.
  * ShafieiBavani et al. (2018) ShafieiBavani, E.; Ebrahimi, M.; Wong, R.; and Chen, F. 2018.  A graph-theoretic summary evaluation for rouge.  In _Proceedings of the 2018 EMNLP_ , 762–767.
  * Shimanaka, Kajiwara, and Komachi (2018) Shimanaka, H.; Kajiwara, T.; and Komachi, M. 2018.  Ruse: Regressor using sentence embeddings for automatic machine translation evaluation.  In _Proceedings of the Third WMT: Shared Task Papers_ , 751–758.
  * Snover et al. (2006) Snover, M.; Dorr, B.; Schwartz, R.; Micciulla, L.; and Makhoul, J. 2006.  A study of translation edit rate with targeted human annotation.  In _Proceedings of association for machine translation in the Americas_ , volume 200. Cambridge, MA.
  * Specia, Raj, and Turchi (2010) Specia, L.; Raj, D.; and Turchi, M. 2010.  Machine translation evaluation versus quality estimation.  _Machine translation_ , 24(1): 39–50.
  * Specia, Scarton, and Paetzold (2018) Specia, L.; Scarton, C.; and Paetzold, G. H. 2018.  Quality estimation for machine translation.  _Synthesis Lectures on Human Language Technologies_ , 11(1): 1–162.
  * Staerman et al. (2021a) Staerman, G.; Laforgue, P.; Mozharovskyi, P.; and d’Alché Buc, F. 2021a.  When ot meets mom: Robust estimation of wasserstein distance.  In _International Conference on Artificial Intelligence and Statistics_ , 136–144. PMLR.
  * Staerman et al. (2021b) Staerman, G.; Mozharovskyi, P.; Clémençon, S.; and d’Alché Buc, F. 2021b.  Depth-based pseudo-metrics between probability distributions.
  * Stanchev, Wang, and Ney (2019) Stanchev, P.; Wang, W.; and Ney, H. 2019.  EED: Extended edit distance measure for machine translation.  In _Proceedings of the Fourth WMT (Volume 2: Shared Task Papers, Day 1)_ , 514–520.
  * Stanojević and Sima’an (2014) Stanojević, M.; and Sima’an, K. 2014.  BEER: BEtter Evaluation as Ranking.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , 414–419. Baltimore, Maryland, USA: ACL.
  * Steiger (1980) Steiger, J. H. 1980.  Tests for comparing elements of a correlation matrix.  _Psychological bulletin_ , 87(2): 245.
  * Vedantam, Lawrence Zitnick, and Parikh (2015) Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4566–4575.
  * Wang et al. (2020) Wang, D.; Liu, P.; Zheng, Y.; Qiu, X.; and Huang, X. 2020.  Heterogeneous graph neural networks for extractive document summarization.  _arXiv preprint arXiv:2004.12393_.
  * Wang et al. (2016) Wang, W.; Peter, J.-T.; Rosendahl, H.; and Ney, H. 2016.  Character: Translation edit rate on character level.  In _Proceedings of the First WMT: Volume 2, Shared Task Papers_ , 505–510.
  * Wen et al. (2015) Wen, T.-H.; Gasic, M.; Mrksic, N.; Su, P.-H.; Vandyke, D.; and Young, S. 2015.  Semantically conditioned lstm-based natural language generation for spoken dialogue systems.
  * Witon et al. (2018) Witon, W.; Colombo, P.; Modi, A.; and Kapadia, M. 2018.  Disney at iest 2018: Predicting emotions using an ensemble.  In _Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , 248–253.
  * Wolf et al. (2019) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; and Brew, J. 2019.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  _ArXiv_ , abs/1910.03771.
  * Yoon et al. (2020) Yoon, W.; Yeo, Y. S.; Jeong, M.; Yi, B.-J.; and Kang, J. 2020.  Learning by semantic similarity makes abstractive summarization better.  _arXiv preprint arXiv:2002.07767_.
  * Zhang et al. (2019) Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2019.  Bertscore: Evaluating text generation with bert.
  * Zhao et al. (2021) Zhao, T. Z.; Wallace, E.; Feng, S.; Klein, D.; and Singh, S. 2021.  Calibrate Before Use: Improving Few-Shot Performance of Language Models.
  * Zhao et al. (2019) Zhao, W.; Peyrard, M.; Liu, F.; Gao, Y.; Meyer, C. M.; and Eger, S. 2019.  Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance.
  * Zhong et al. (2020) Zhong, M.; Liu, P.; Chen, Y.; Wang, D.; Qiu, X.; and Huang, X. 2020.  Extractive summarization as text matching.  _arXiv preprint arXiv:2004.08795_.
  * Zhong et al. (2019) Zhong, M.; Liu, P.; Wang, D.; Qiu, X.; and Huang, X. 2019.  Searching for Effective Neural Extractive Summarization: What Works and What’s Next.  _arXiv preprint arXiv:1907.03491_.
  * Zhou et al. (2018) Zhou, Q.; Yang, N.; Wei, F.; Huang, S.; Zhou, M.; and Zhao, T. 2018.  Neural document summarization by jointly learning to score and select sentences.  _arXiv preprint arXiv:1807.02305_.

##  Appendix A Additional Experimental Results

In this section, we report the additional experiments we conduct. Because of
space constraints, we did not report the sensibility analysis (see Section
A.2) in the main paper.

###  A.1 Role of calibration

The complete results on the role of calibration can be found in Figure 5. We
notice a similar behaviour for InfoLM on both extractive and abstractive
systems. There is a smooth variation in terms of performance and a single
maximum for $\mathcal{T}\in[1,2]$.

(a) Abstractive (b) Extractive

Figure 5: Impact of Calibration on system-level correlation (with Pearson
($r$), Spearman ($\rho$) or Kendall ($\tau$)) for CNN. The chosen measure is
Rao as it is parameter-free. Calibration is changed using temperature scaling
using a single temperature $\mathcal{T}$.

###  A.2 Choice of $\alpha$ and $\beta$

In this experiment, we aim at quantifying the sensitivity of
$\mathcal{D}_{AB}$ to the choice of $\alpha$ and $\beta$. Figure 6 gathers the
results of the analysis. We observe that a change in $\beta$ induces a
stronger change in the metric. Additionally the lower $\beta$ the better
result we obtain. We can also note that the variation of both parameters is
smooth. Interestingly for abstractive systems, a low value of $\alpha$ should
be chosen where for extractive higher is better. It suggests that for
evaluating abstractive systems the metric should focus on low values of
$p_{i}/q_{i}$ (words that are probable for both candidate and reference text)
where for extractive systems the attention should be focused on high values of
$p_{i}/q_{i}$ (words that are likely only in one text).
Takeaways. Low values of $\beta$ leads to better results, optimal value of
$\alpha$ is $1.25$ for abstractive and $3$ for extractive. Good parameter
combinations achieve consistently high performance when using different
correlation coefficients.

Figure 6: Impact of change in $\alpha$ and $\beta$ for $\mathcal{D}_{AB}$ .
System level correlation, as measured by Pearson ($r$) or Spearman ($\rho$),
is presented on abstractive (first column) and extractive system (second
column).

###  A.3 Statistical analysis

Automatic metrics are used in the WebNLG challenge to compare the different
systems. To evaluate whether the observed improvement in correlation is
significant, we report the results of William’s Significance test in Figure 7.
Takeaways: (i) Regarding correctness and relevance $\mathcal{D}_{AB}$, is a
suitable choice that is significantly better than other metrics; (ii)
Regarding text structure, $\mathcal{R}$ is significantly better and compare
favourably against all metrics except MOVERSCORE for automatic fluency
evaluation; (iii) Regarding data coverage, METEOR achieves good result however
significance difference is only observed with BERTSCORE.

(a) Correct (b) Data C (c) Fluency (d) Relev (e) Text S

Figure 7: Results of William’s Significance Test: the tested hypothesis is:
“is the increase of correlation significant”. For clarity and due to space
constraints the p-values are truncated and multiply par 100. Only p-values
that are lower than $5.00$ are displayed.

###  A.4 Complete results on Summarization

We gather in Figure 8, the complete results on summarization. Due to space
constraints, we did not report all the Spearman Correlation coefficients in
the main paper. It is worth noting that our observations hold and the best
performing metric is obtained with $\mathcal{D}_{AB}$. The Fisher-Rao
distance, denoted by $R$, achieves good performance in many scenarios and has
the advantage to be parameter-free.

###  A.5 Complete results on Data2text Generation

We gather in Table 3, the complete results on data2text generation. Due to
space constraints, we did not report all the baselines in the main paper.

| Correctness | Data Coverage | Fluency | Relevance | Text Structure
---|---|---|---|---|---
Metric | $r$ | $\rho$ | $\tau$ | $r$ | $\rho$ | $\tau$ | $r$ | $\rho$ | $\tau$ | $r$ | $\rho$ | $\tau$ | $r$ | $\rho$ | $\tau$
Correct | 100.0 | 100.0 | 100.0 | 97.6 | 85.2 | 73.3 | 80.0 | 81.1 | 61.6 | 99.1 | 89.7 | 75.0 | 80.1 | 80.8 | 60.0
DataC | 85.2 | 97.6 | 73.3 | 100.0 | 100.0 | 100.0 | 71.8 | 51.7 | 38.3 | 96.0 | 93.8 | 81.6 | 71.6 | 51.4 | 36.6
Fluency | 81.1 | 80.0 | 61.6 | 71.8 | 51.7 | 38.3 | 100.0 | 100.0 | 100.0 | 77.0 | 61.4 | 46.6 | 99.5 | 99.7 | 98.3
Relev | 89.7 | 99.1 | 75.0 | 96.0 | 93.8 | 81.6 | 77.0 | 61.4 | 46.6 | 100.0 | 100.0 | 100.0 | 77.2 | 61.1 | 45.0
TextS | 80.8 | 80.1 | 60.0 | 71.6 | 51.4 | 36.6 | 99.5 | 99.7 | 98.3 | 77.2 | 61.1 | 45.0 | 100.0 | 100.0 | 100.0
$\mathcal{D}_{AB}$ | 88.8 | 89.3 | 76.6 | 81.8 | 82.6 | 70.0 | 86.6 | 92.0 | 76.6 | 89.8 | 87.9 | 73.3 | 86.6 | 91.4 | 75.0
$\mathcal{D}_{\alpha}$ | 88.8 | 89.3 | 76.6 | 81.8 | 82.6 | 70.0 | 86.6 | 92.0 | 76.6 | 89.8 | 87.9 | 73.3 | 86.6 | 91.4 | 75.0
$\mathcal{D}_{\beta}$ | 81.4 | 50.0 | 71.6 | 48.4 | 79.7 | 65.0 | 44.8 | 84.7 | 76.6 | 49.3 | 72.3 | 60.0 | 48.0 | 83.8 | 75.0
$\mathcal{L}_{1}$ | 75.2 | 33.8 | 61.6 | 32.4 | 53.8 | 40.0 | 22.7 | 83.5 | 73.3 | 32.2 | 57.9 | 45.0 | 25.6 | 83.2 | 71.6
$\mathcal{L}_{2}$ | 67.0 | 21.9 | 56.6 | 21.6 | 37.9 | 33.3 | 11.9 | 75.2 | 58.3 | 20.1 | 43.8 | 38.3 | 14.8 | 75.5 | 60.0
$\mathcal{L}_{\infty}$ | 63.2 | 33.0 | 46.6 | 30.4 | 36.4 | 26.6 | 67.6 | 65.0 | 46.6 | 29.1 | 49.1 | 35.0 | 67.2 | 65.2 | 46.6
$\mathcal{R}$ | 89.7 | 86.0 | 75.0 | 78.7 | 70.5 | 51.6 | 93.3 | 95.7 | 85.3 | 87.6 | 84.4 | 70.0 | 92.4 | 93.8 | 81.6
JS | 79.4 | 81.1 | 70.0 | 69.3 | 75.5 | 60.0 | 89.4 | 91.4 | 75.0 | 81.7 | 70.5 | 60.0 | 91.9 | 91.1 | 73.3
BertS | 85.5 | 83.4 | 73.3 | 74.7 | 68.2 | 53.3 | 92.3 | 95.5 | 85.0 | 83.3 | 79.4 | 65.0 | 91.9 | 95.0 | 83.3
MoverS | 84.1 | 84.1 | 73.3 | 78.7 | 66.2 | 53.3 | 91.2 | 92.1 | 78.3 | 82.1 | 77.4 | 65.0 | 90.1 | 91.4 | 76.3
BLEU | 77.6 | 66.3 | 60.0 | 55.7 | 50.2 | 36.6 | 89.4 | 90.5 | 78.3 | 63.0 | 65.2 | 51.6 | 88.5 | 89.1 | 76.6
R-1 | 80.6 | 65.0 | 65.0 | 61.1 | 59.6 | 48.3 | 76.5 | 76.3 | 60.3 | 64.3 | 69.2 | 56.7 | 75.9 | 77.5 | 58.3
R-2 | 73.6 | 63.3 | 58.3 | 54.7 | 43.1 | 35.0 | 86.4 | 81.9 | 63.4 | 62.0 | 60.8 | 46.7 | 86.5 | 80.5 | 61.7
R-WE | 60.9 | 73.4 | 60.0 | 40.2 | 58.2 | 40.1 | 61.4 | 84.7 | 61.3 | 49.9 | 64.1 | 48.3 | 60.2 | 85.9 | 60.0
METEOR | 86.5 | 66.3 | 70.0 | 77.3 | 50.2 | 46.6 | 86.7 | 90.5 | 78.3 | 82.1 | 65.2 | 58.6 | 86.2 | 89.1 | 76.6
TER | 79.6 | 78.3 | 58.0 | 69.7 | 58.2 | 38.0 | 89.1 | 93.5 | 80.0 | 75.0 | 70.2 | 77.6 | 89.5 | 91.1 | 78.6

Table 3: Correlation at the system level with human judgement along five
different axis: correctness, data coverage, fluency, relevance and text
structure for the WebNLG task. Best results by group are underlined, overall
best results are bolted.

###  A.6 Complete Results on Score Distribution

We gather in Figure 9 the score distribution on the summarization dataset. It
is worth noting that for these experiments we have scaled the divergences
between 0 and 1 and we have considered 1 - InfoLM.

##  Appendix B Additional details on the datasets

###  B.1 Summarization

The summarization dataset CNN can be found in
https://github.com/neulab/REALSumm and no preprocessing has been applied.
Specifically, the summaries have been generated using 14 abstractive systems
(Zhong et al. 2020; Wang et al. 2020; Zhong et al. 2019; Liu and Lapata 2019;
Zhou et al. 2018; Narayan, Cohen, and Lapata 2018; Dong et al. 2019; Kedzie,
McKeown, and Daume III 2018; Zhou et al. 2018) and 11 extractive systems (See,
Liu, and Manning 2017; Chen and Bansal 2018; Raffel et al. 2019; Gehrmann,
Deng, and Rush 2018; Dong et al. 2019; Liu and Lapata 2019; Lewis et al. 2019;
Yoon et al. 2020).
Pyramide Score. The Pyramide score, which was inspired from work in reading
comprehension (see Beck et al. (1991)) is a scoring procedure to assess the
semantic content in the scope of summarization. The manual methods are based
on the concept of Summary Content Units (SCUs) which groups sentences from
different summaries. grouping if they carry the same content.

###  B.2 Data2text

The goal of the WebNLG challenge is to develop efficient Knowledge Base
Verbalizers (Gardent et al. 2017; Perez-Beltrachini, Sayed, and Gardent
2016)(i.e generation algorithms that can verbalise knowledge base fragments)
and thus handle complex interaction that can occur during the micro-planning
phase when generating sentences (Ferreira et al. 2018). Details on the
WebNLG2020 task are given in Ferreira et al. (2020). The dataset is composed
of generated sentences coming from 15 different systems using various
approaches such as symbolic approaches or neural-based systems. All data are
freely available on GitHub https://gitlab.com/shimorina/webnlg-
dataset/-/tree/master/release˙v3.0 and no preprocessing has been applied. For
the dataset, they use the RDF format which is a widely used format for many
datasets such as LinkedGeoData http://linkedgeodata.org/About, FOAF
http://www.foaf-project.org/ or MusicBrainz https://musicbrainz.org/.
extracted from DBpedia (Auer et al. 2007).

###  B.3 Limitations

We have evaluate our metrics on text only datasets. Future work will also
investigate robustness of the metric to type of texts (e.g spoken text (Dinkar
et al. 2020; Chapuis et al. 2021, 2020)) and the extension to multimodal
setting (Colombo et al. 2021a; Garcia et al. 2019) and other type of task (e.g
related to dialog (Colombo et al. 2021c) and affect driven text generation
(Colombo et al. 2019; Witon et al. 2018) and intend generation (Mehri, Eric,
and Hakkani-Tur 2020) with dialog acts (Colombo et al. 2020)).

###  B.4 Metric Choices

Several revised versions of BLEU (Doddington 2002; Galley et al. 2015) and
METEOR (Denkowski and Lavie 2014; Guo and Hu 2019) have been proposed in
recent years. For our implementation we choose to use SACREBLEU (Post 2018).
Similarly, a plethora of ROUGE extension have been proposed ((Ganesan 2018;
ShafieiBavani et al. 2018)) but in our work we choose to work with ROUGE-1,
ROUGE-2 and ROUGE-WE. For the BERT based metrics, we choose to rely on the
most popular although several alternatives exists (e.g. SENTENCE-MOVER (Clark,
Celikyilmaz, and Smith 2019))

|  |  |
---|---|---|---
|  |  |
|  |  |

(a) Abs - Text (b) Ext - Text

(c) Abs - Sys (d) Ext - Sys

(e) Abs - Text (f) Ext - Text

(g) Abs - Sys (h) Ext - Sys

(i) Abs - Text (j) Ext - Text

(k) Abs - Sys (l) Ext - Sys

Figure 8: Results of the correlation between metrics and human judgement on
the CNN dataset.

(a) Score distribution Abstractive Summarization Systems

(b) Score distribution Extractive Summarization Systems

Figure 9: Score distribution considering a larger number of metrics than in
Figure 3.

###  B.5 Parameters Choices

For summarization (see Figure 8) we choose to work with the following
parameters:

  * •

The temperature is set $1$.

  * •

For $\mathcal{D}_{\alpha}$, we choose $\alpha=0.75$ for extractive and
$\alpha=3$ for abstractive.

  * •

For $\mathcal{D}_{\gamma}$, we choose $\beta=0.5$.

  * •

For $\mathcal{D}_{AB}$, we choose $\beta=0.25$ and $\alpha=0.5$ for extractive
and $\beta=0.25$ and $\alpha=3$ for abstractive.

For data2text generation (see Section A.5) we choose to work with the
following parameters:

  * •

The temperature is set $1$.

  * •

For $\mathcal{D}_{\alpha}$, we choose $\alpha=0.75$

  * •

For $\mathcal{D}_{\gamma}$, we choose $\beta=3$

  * •

For $\mathcal{D}_{AB}$, we choose $\beta=0.25,\alpha=3$

Takeaways Although, parameters can be tuned to achieve better results. It is
worth noting that the parameter-free Fisher-Rao distance (i.e. $\mathcal{R}$)
achieve good results when considered for InfoLM.

##  Appendix C Implementation details

###  C.1 Algorithm details

A complete algorithm for InfoLM is given in Algorithm 1.

Algorithm 1 InfoLM

1:Input Candidate text $\boldsymbol{y}_{i}^{s}$ of length L, Reference text
$\boldsymbol{x_{i}}$ of length M, measure of information $\mathcal{I}$

2:$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y}_{i}^{s}),p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x})=0,0$

3:for $k\in[1,L]$ do $\triangleright$ Compute
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y}_{i}^{s})$

4:
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}})=p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}})+\gamma_{k}\times
p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|[\boldsymbol{y}_{i}^{s}]^{k})$

5:end for

6:for $j\in[1,M]$ do $\triangleright$ Compute
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x_{i}})$

7:
$p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x_{i}})=p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x_{i}})+\bar{\gamma_{k}}\times
p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|[\boldsymbol{x_{i}}]^{j})$

8:end for

9:Output
$\mathcal{I}\big{[}p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{y_{i}^{s}}),p_{\mathbf{\Omega}|\mathbf{T}}(\cdot|\boldsymbol{x_{i}})\big{]}$

###  C.2 Computational resources

For all the experiments we use Tesla NVIDIA P100 to compute the BERT based
metrics. Running time is less than an hour. For metrics based on string
matching, we use a single CPU and the running time is less than an hour.

###  C.3 Libraries

For this project among the library we used we can cite:

  * •

Transformers from Wolf et al. (2019).

  * •

SummEval can be found here https://github.com/Yale-LILY/SummEval and has been
proposed in Fabbri et al. (2020) for the dataset CNN.

  * •

RealSumm which can be found here https://github.com/neulab/REALSumm and has
been proposed in Bhandari et al. (2020) for some metrics.

  * •

Pytorch (Paszke et al. 2017) for the GPU support.

  * •

Implementation of MOVERSCORE can be found here
https://github.com/AIPHES/emnlp19-moverscore.

  * •

Implementation of BERTSCORE can be found here
https://github.com/Tiiiger/bert˙score.

  * •

SacreBLEU for BLEU implementation (Post 2018).

  * •

The William test is taken from the author’s code and is available at
https://github.com/ygraham/nlp-williams

We thank the contributor for open-sourcing their libraries.

###  C.4 Negative Results

We tried to remove the stop words and do pre-processing technics. Little
improvement when cleaning the candidate and golden reference text might be
attributed to BERT.