# The exceptional simple Lie group $F_{4(-20)}$,
after J. Tits

Alain VALETTE

###### Abstract

This is a semi-survey paper, where we start by advertising Tits’ synthetic
construction from [Ti53], of the hyperbolic plane $H^{2}(Cay)$ over the Cayley
numbers $Cay$, and of its automorphism group which is the exceptional simple
Lie group $G=F_{4(-20)}$. Let $G=KAN$ be the Iwasawa decomposition. Our
contributions are:

  * •

Writing down explicitly the action of $N$ on $H^{2}(Cay)$ in Tits’model,
facing the lack of associativity of $Cay$.

  * •

If $MAN$ denotes the minimal parabolic subgroup of $G$, characterizing $M$
geometrically.

##  1 Introduction

Let $G$ be a simple Lie group with trivial center, with Iwasawa decomposition
$G=KAN$. The real rank of $G$, i.e. the dimension of $A$, can be viewed as a
measure of the complexity of $G$, with the real rank of $G$ being 0 if and
only if $G$ is compact. It is known that in higher real rank, i.e. $\dim A\geq
2$, the group $G$ satisfies various forms of rigidity (e,g. property (T),
arithmeticity of lattices, Margulis’ super-rigidity: see [Zi84] for all that).
The situation in real rank one is more subtle, with some forms of rigidity
being satisfied and some other not, the situation depending very much on the
group under consideration. Recall that, from Cartan’s classification (see e.g.
[Ti67]), simple real Lie groups with real rank 1 fall into 3 infinite series
and one exceptional group:

  * •

Adjoint group of $SO_{0}(n,1),\;(n\geq 2)$.

  * •

Adjoint group of $SU(n,1),\;(n\geq 2)$.

  * •

Adjoint group of $Sp(n,1),\;(n\geq 2)$.

  * •

$F_{4(-20)}$.

The present paper deals with the exceptional group $F_{4(-20)}$. Several
papers in harmonic analysis on real rank 1 simple Lie groups deal exclusively
with the classical cases, i.e. the three infinite series above: see [Li74] for
a sample. A notable exception is Takahashi’s paper [Ta79], which was inspiring
for us. It is therefore interesting to have explicit constructions of
$F_{4(-20)}$, both algebraic and geometric.

Algebraically, $F_{4(-20)}$ was constructed by Freudenthal [Fr53] as the
automorphism group of the real Albert algebra $J_{1,2}$, which is one out of
the three 27-dimensional exceptional Jordan algebras over the reals (see also
chapter 19 in [Mo73], or more recently [Al98]). Geometrically, it was
constructed by Tits [Ti53] as the automorphism group of the hyperbolic plane
over the octonions $Cay$. This provides a unified construction of all simple
Lie groups of real rank 1 and with trivial centre: let $\mathbf{K}$ be one of
the four finite-dimensional division algebras over the reals, namely
$\mathbf{R},\mathbf{C},\mathbf{H}$ (the Hamilton quaternions), or $Cay$ (the
Cayley numbers). For $n\geq 2$, let $H^{n}(\mathbf{K})$ be $n$-dimensional
hyperbolic space over $\mathbf{K}$ (with $n=2$ when $\mathbf{K}=Cay$). Then
any real rank 1 simple Lie group with trivial center is isomorphic to the
(connected component of identity of the) automorphism group
$Aut(H^{n}(\mathbf{K}))$ for some $n\geq 2$ and some
$\mathbf{K}\in\\{\mathbf{R},\mathbf{C},\mathbf{H},Cay\\}$.

The present, mostly survey, paper aims to advertise Tits’ synthetic
construction of $H^{2}(\mathbf{K})$ from [Ti53]. Tits first defines the
projective plane $P^{2}(\mathbf{K})$ as the affine plane $\mathbf{K}^{2}$
together with a line at infinity, which provides a very suitable
coordinatization of $P^{2}(\mathbf{K})$. The hyperbolic plane then appears as
the set of interior points of a specific hyperbolic polarity, and is realized
as the unit ball $\\{(x_{1},x_{2}):|x_{1}|^{2}+|x_{2}|^{2}<1\\}$ in
$\mathbf{K}^{2}$.

The paper is structured as follows: in section 2, we recall Cayley numbers and
the triality principle; in section 3, we survey Tits’ constructions and
results from [Ti53]. Only the final section 4 has some novelty. Let $KAN$ be
the Iwasawa decomposition of $F_{4(-20)}$; the maximal compact subgroup $K$
was determined by Tits in [Ti53], the one-parameter subgroup $A$ appears in
[Ta79]. The nilpotent subgroup $N$ was determined by Takahashi [Ta79] in
Freudenthal’s algebraic model, it is a central extension of the additive group
of $Cay$ by the additive group $Im(Cay)$ of imaginary Cayley numbers. Our main
contribution consists in writing down explicitly the action of $N$ on
$H^{2}(Cay)$: doing so we pay the price for working over a non-associative
division algebra. Indeed, here is for example the action of the element
$u(y,0)$ in the centre $Z(N)$ on $(x_{1},x_{2})\in H^{2}(Cay)$ (see Corollary
4.3 below): we have $u(y,0)(x_{1},x_{2})=(x^{\prime}_{1},x^{\prime}_{2})$,
where:

$x^{\prime}_{1}=((1-y)x_{1}+y)(-yx_{1}+1+y)^{-1}$

and

$x^{\prime}_{2}=(x_{2}(1-\overline{x_{1}}))[(1-\overline{x_{1}})^{-1}(-yx_{1}+1+y)^{-1}]$

While the formula for $x^{\prime}_{1}$ is what is expected for an action by
Möbius transformations, the formula for $x^{\prime}_{2}$ reveals the problems
that arise with the lack of associativity.

Finally, let $MAN$ be the minimal parabolic subgroup of $F_{4(-20)}$: we
characterize $M$ as the stabilizer in $K$ of the point $(1,0)$ on the boundary
of $H^{2}(Cay)$.

##  2 Octonions and triality

Recall that $Cay$, the algebra of Cayley numbers or octonions, denotes the
8-dimensional non-associative division algebra over $\mathbf{R}$ with basis
$e_{0},e_{1},e_{2},...,e_{7}$ and relations

  * •

$e_{0}.e_{i}=e_{i}.e_{0}=e_{i}$ for $i=0,1,...,7$.

  * •

$e_{i}^{2}=-e_{0}$ for $i>0$.

  * •

$e_{i}.e_{j}=-e_{j}.e_{i}$ for $i,j>0,\;i\neq j$.

  * •

$e_{2}.e_{6}=e_{3}.e_{4}=e_{5}.e_{7}=e_{1}$, plus all relations deduced from
this one by cyclic permutations of the indices $1,2,...,7$.

We will replace $e_{0}$ by 1 for simplicity and identify the real octonion
$a.e_{0}$ with the real number $a$. Any octonion
$x=a+\sum_{i=1}^{7}b_{i}e_{i}$ has a conjugate
$\overline{x}=a-\sum_{i=1}^{7}b_{i}e_{i}$ and a norm
$N(x)=x\overline{x}=\overline{x}x=a^{2}+\sum_{i=1}^{7}b_{i}^{2}$. The modulus
of $x$ is $|x|=:\sqrt{N(x)}$. The norm satisfies $N(xy)=N(x)N(y)$ for every
$x,y\in Cay$.

The imaginary part of an octonion $x$ is $Im(x)=\frac{x-\overline{x}}{2}$: the
octonion $x$ is said to be it imaginary if $x=Im(x)$; we denote by $Im(Cay)$
the set of imaginary octonions.

The algebra $Cay$ is a division algebra as the inverse of a non-zero octonion
$x$ is $x^{-1}=\frac{\overline{x}}{N(x)}$. The algebra $Cay$ is not
associative but alternative, namely for every $x,y\in Cay$:

  * •

$(xy)x=x(yx)$;

  * •

$x(xy)=x^{2}y$;

  * •

$(xy)y=xy^{2}$.

For all this, see either the nice paper [Ba02] or Chapter 1 of the book
[SV00].

The norm allows us to identify $Cay$ linearly isometrically with the
8-dimensional Euclidean space $\mathbf{E}^{8}$. So the rotation group $SO(8)$
acts linearly on $Cay$ and the have the triality principle (see Theorem 3.2.1
in [SV00]) that we recall:

######  Theorem 2.1.

For every $R\in SO(8)$, there exists $R^{\prime},R^{\prime\prime}\in SO(8)$
such that, for every $x,y\in Cay$:

$R(xy)=R^{\prime}(x)R^{\prime\prime}(y)$

Moreover $R^{\prime},R^{\prime\prime}$ are determined up to sign by this
relation. $\blacksquare$

We will need some special features of the triple
$(R,R^{\prime},R^{\prime\prime})$ as above.

######  Lemma 2.2.

Fix $R\in SO(8)$, let $R^{\prime},R^{\prime\prime}\in SO(8)$ be associated
with $R$ as in Theorem 2.1.

  1. 1.

We have $R^{\prime\prime}(1)=1$ if and only if $R=R^{\prime}$.

  2. 2.

Assume $R^{\prime\prime}(1)=1$. Then:

    1. a)

For every non-zero octonion $x$, we have
$R^{\prime\prime}(x^{-1})=R^{\prime\prime}(x)^{-1}$.

    2. b)

$R^{\prime\prime}$ is a Jordan homomorphism, i.e

$R^{\prime\prime}(xy+yx)=R^{\prime\prime}(x)R^{\prime\prime}(y)+R^{\prime\prime}(y)R^{\prime\prime}(x)$

for every $x,y\in Cay$.

    3. c)

For every $x\in Cay$, the map $R^{\prime\prime}$ is a homomorphism in
restriction to the subalgebra generated by $1$ and $x$.

Proof :

  1. 1.

If $R^{\prime\prime}(1)=1$, then for every $x\in Cay$ we get:

$R(x)=R(x.1)=R^{\prime}(x).R^{\prime\prime}(1)=R^{\prime}(x).1=R^{\prime}(x)$

hence $R=R^{\prime}$. Conversely, assuming $R=R^{\prime}$, we get for every
$x\in Cay$:

$R(x)=R(x.1)=R^{\prime}(x)R^{\prime\prime}(1)=R(x)R^{\prime\prime}(1),$

which forces $R^{\prime\prime}(1)=1$.

  2. 2.

    1. a)

As $R^{\prime\prime}(1)=1$, we have
$R^{\prime\prime}(\overline{x})=\overline{R^{\prime\prime}(x)}$, and the
result follows from the formula for the inverse of $x$.

    2. b)

Denoting by $\langle.|.\rangle$ the standard scalar product on $Cay$
associated with the norm $N$, we have by formula (1.8) in [SV00]:

$xy+yx=\langle x|1\rangle y+\langle y|1\rangle x-\langle x|y\rangle 1.$

Applying $R^{\prime\prime}$ and using that it is a linear isometry, we get:

$R^{\prime\prime}(xy+yx)=\langle
R^{\prime\prime}(x)|R^{\prime\prime}(1)\rangle R^{\prime\prime}(y)+\langle
R^{\prime\prime}(y)|R^{\prime\prime}(1)\rangle R^{\prime\prime}(x)-\langle
R^{\prime\prime}(x)|R^{\prime\prime}(y)\rangle R^{\prime\prime}(1)$ $=\langle
R^{\prime\prime}(x)|1\rangle R^{\prime\prime}(y)+\langle
R^{\prime\prime}(y)|1\rangle R^{\prime\prime}(x)-\langle
R^{\prime\prime}(x)|R^{\prime\prime}(y)\rangle
1=R^{\prime\prime}(x)R^{\prime\prime}(y)+R^{\prime\prime}(y)R^{\prime\prime}(x),$

where the second equality follows from the assumption $R^{\prime\prime}(1)=1$.

    3. c)

By formula (1.7) in [SV00], the subalgebra generated by $1$ and $x$ is at most
2-dimensional, so it is commutative and the result follows from the preceding
point. $\blacksquare$

##  3 The hyperbolic plane over $Cay$

In this section we follow rather closely Tits’paper [Ti53].

###  3.1 The projective plane and its collineation group

Let $\mathbf{K}$ denote one of the following division algebras over the reals:
$\mathbf{R},\mathbf{C},\mathbf{H}$(the Hamilton quaternions), or $Cay$. The
projective plane $P^{2}(\mathbf{K})$ is the disjoint union

$P^{2}(\mathbf{K})=(\mathbf{K}\times\mathbf{K})\coprod\mathbf{K}\coprod{(\infty)}$

where $\mathbf{K}\times\mathbf{K}$ is the set of points at finite distance and
$\mathbf{K}\coprod{(\infty)}=\\{(u):u\in\mathbf{K}\coprod\\{\infty\\}\\}$ is
the set of point at infinity. A line in $P^{2}(\mathbf{K})$ is a subset of one
of the following three types:

  * •

$[u,v]=\\{(x,y)\in\mathbf{K}\times\mathbf{K}:y=ux+v\\}\cup\\{(u)\\}$;

  * •

$[u]=\\{(x,y)\in\mathbf{K}\times\mathbf{K}:x=u\\}\cup\\{(\infty)\\}$;

  * •

$[\infty]=\mathbf{K}\coprod{(\infty)}$.

Lines of the first type are oblique, lines of the second type are vertical,
and $[\infty]$ is the line at infinity. It is readily seen that the incidence
axioms of projective geometry are satisfied: two lines intersect in a unique
point, two distinct points determine a unique line through them.

A collineation of $P^{2}(\mathbf{K})$ is a bijection of $P^{2}(\mathbf{K})$
onto itself, that preserves the set of lines.

######  Theorem 3.1.

(Théorème 4.9 in [Ti53]) The group of collineations of $P^{2}(Cay)$ is
isomorphic to the real form $E_{6(-26)}$ of the exceptional Lie group $E_{6}$:
it is therefore a simple Lie group of dimension 78, which is simply connected
with trivial centre. $\blacksquare$

###  3.2 The hyperbolic plane and its automorphism group

A polarity of $P^{2}(\mathbf{K})$ is a bijection $\Pi$ between the set of
points and the set of lines of $P^{2}(\mathbf{K})$, satisfying two conditions:

  * •

if $a$ runs through a line $L$, then $\Pi(a)$ runs in the sheaf of lines
through a given point;

  * •

for any two points $a,b\in P^{2}(\mathbf{K})$, we have
$a\in\Pi(b)\Leftrightarrow b\in\Pi(a)$.

We will need just one example of polarity, given on points by:

$\Pi(x,y)=[-\overline{xy^{-1}},\overline{y^{-1}}]\;\;\mbox{if}\,y\neq 0;$
$\Pi(x,0)=[\overline{x^{-1}}]\;\;\mbox{if}\;x\neq 0;$ $\Pi(0,0)=[\infty].$

This polarity is hyperbolic, meaning that the set of points $P\in
P^{2}(\mathbf{K})$ such that $P\in\Pi(P)$ is non-empty. Actually this set is
the sphere of equation $|x|^{2}+|y|^{2}=1$ in $\mathbf{K}\times\mathbf{K}$.
The hyperbolic plane $H^{2}(\mathbf{K})$ is then the ball $|x|^{2}+|y|^{2}<1$
in $\mathbf{K}\times\mathbf{K}$. Lines in $H^{2}(\mathbf{K})$ are traces on
$H^{2}(\mathbf{K})$ of lines in $P^{2}(\mathbf{K})$ that intersect it (“Klein
model”). An automorphism of $H^{2}(\mathbf{K})$ is a collineation of
$P^{2}(\mathbf{K})$ that commutes with $\Pi$.

######  Theorem 3.2.

Let $G$ be the automorphism group of $H^{2}(Cay)$.

  1. 1.

(6.5 in [Ti53]) The stabilizer $K$ of $(0,0)$ in $G$ is isomorphic to
$Spin(9)$ (the universal cover of $SO(9)$); it acts transitively on the sphere
$|x|^{2}+|y|^{2}=1$ and it is a maximal compact subgroup of $G$.

  2. 2.

(Théorème 6.8 in [Ti53]) $G$ is isomorphic to the real form $F_{4(-20)}$ of
the exceptional Lie group $F_{4}$: it is therefore a simple Lie group of
dimension 52, which is simply connected with trivial centre. $\blacksquare$

Let us single out the following for future use:

######  Lemma 3.3.

(see (4.6) in [Ti53]) An element $k\in K$ fixes the points at infinity $(0)$
and $(\infty)$ if and only if there exists $R\in SO(8)$ such that (in the
notation of the triality principle Theorem 2.1):

$k(x,y)=(R^{\prime\prime}(x),R(y))$

for every $(x,y)\in Cay\times Cay$. $\blacksquare$

##  4 The Iwasawa decomposition and the minimal parabolic subgroup of $G$

Let $G$ and $K$ be as in Theorem 3.2. Let $\mathfrak{g,k}$ denote the Lie
algebras of $G,\,K$ respectively. Using the algebraic realization of $G$ as
the automorphism group of the Albert algebra $J_{1,2}$, Takahashi ([Ta79],
formula (10) in section 3) determines the Cartan decomposition
$\mathfrak{g}=\mathfrak{k}\oplus\mathfrak{p}$, chooses a natural maximal
abelian subalgebra $\mathfrak{a}\subset\mathfrak{p}$, where
$\mathfrak{a}=\mathbf{R}\cdot A$ (as $\dim\mathfrak{a}=1$) and identifies
restricted root subspaces

$\mathfrak{g}_{\alpha}=\\{Z(z):z\in Cay\\}$

and

$\mathfrak{g}_{2\alpha}=\\{Y(y):y\in Im(Cay)\\}.$

Setting $\mathfrak{n}=\mathfrak{g}_{\alpha}\oplus\mathfrak{g}_{2\alpha}$ he
establishes the Iwasawa decomposition at the level of $\mathfrak{g}$ ([Ta79],
(13) in section 3):

$\mathfrak{g}=\mathfrak{k}\oplus\mathfrak{a}\oplus\mathfrak{n}.$

We then set $a_{t}=\exp(tA)$ and $A=\\{a_{t}:t\in\mathbf{R}\\}$, the split
maximal torus $A$ in the Iwasawa decomposition of $G$. Similarly we set

$u(y,z)=\exp(Y(y))\exp(Z(z))=\exp(Z(z))\exp(Y(y))=\exp(Y(y)+Z(z))$

and $N=\\{u(y,z):y\in Im(Cay),z\in Cay\\}$, the nilpotent part in the Iwasawa
decomposition of $G$; so $N$ is a 15-dimensional, 2-step nilpotent group with
centre $Z(N)=\\{u(y,0):y\in Im(Cay)\\}$.

In order to write down the action of $A$ and $N$ on $H^{2}(Cay)$, Takahashi
defines in section 5 of [Ta79] a $G$-equivariant embedding of $H^{2}(Cay)$ in
$J_{1,2}$, which allows him to write down the action of $A$ in full
generality, but the action of $N$ only at the origin $(0,0)\in H^{2}(Cay)$:

######  Theorem 4.1.

(formulae (14) and (20) in section 5 of [Ta79])

  1. 1.

For $(x_{1},x_{2})\in H^{2}(Cay),t\in\mathbf{R}$, we have

$a_{t}(x_{1},x_{2})=((\cosh(t)x_{1}+\sinh(t))(\sinh(t)x_{1}+\cosh(t))^{-1},x_{2}(\sinh(t)x_{1}+\cosh(t))^{-1}).$
(1)

  2. 2.

For $t\in\mathbf{R},y\in Im(Cay),z\in Cay$, we have
$a_{t}u(y,z)(0,0)=(x_{1},x_{2})$, where:

$x_{1}=(\sinh(t)+e^{t}(\frac{|z|^{2}}{2}+y))(\cosh(t)+e^{t}(\frac{|z|^{2}}{2}+y))^{-1}$
(2) $x_{2}=z(\cosh(t)+e^{t}(\frac{|z|^{2}}{2}+y))^{-1}$ (3)

$\blacksquare$

Observe that formula (1) is the expected formula for a hyperbolic Möbius
transformation. Its validity is due to the fact that coefficients of $a_{t}$
are real, hence belong to the centre of $Cay$. Our aim now is to jack up
formulae (2) and (3) to get the action of $N$ on any point $(x_{1},x_{2})$ in
$H^{2}(Cay)$.

######  Theorem 4.2.

For $(x_{1},x_{2})\in H^{2}(Cay),y\in Im(Cay),z\in Cay$, we have
$u(y,z)(x_{1},x_{2})=(x^{\prime}_{1},x^{\prime}_{2})$ where:

$x^{\prime}_{1}=[((1-\frac{|z|^{2}}{2}-y)x_{1}+(\overline{z}(x_{2}(1-\overline{x_{1}})))(1-\overline{x_{1}})^{-1}+\frac{|z|^{2}}{2}+y)(1-\overline{x_{1}})].D^{-1},$
(4) $x^{\prime}_{2}=[(-zx_{1}+x_{2}+z)(1-\overline{x_{1}})].D^{-1},$ (5)

and

$D=[-(\frac{|z|^{2}}{2}+y)x_{1}+(\overline{z}(x_{2}(1-\overline{x_{1}})))(1-\overline{x_{1}})^{-1}+1+\frac{|z|^{2}}{2}+y](1-\overline{x_{1}})$
(6)

Proof We will need two rules for calculation in the group $AN$. First, by
definition of $u(y,z)$ we have:

$a_{t}u(y,z)a_{-t}=u(e^{2t}y,e^{t}{z}).$ (7)

Second, for $y_{1},y_{2}\in Im(Cay),\,z_{1},z_{2}\in Cay$, by lemma 4.5 in
[Ni11]:

$u(y_{1},z_{1})u(y_{2},z_{2})=u(y_{1}+y_{2}+Im(\overline{z_{1}}z_{2}),z_{1}+z_{2}).$
(8)

Takahashi proves (see lemme 2 in section 5 of [Ta79]) that the group $AN$ acts
simply transitively on $H^{2}(Cay)$. So, for $(x_{1},x_{2})\in H^{2}(Cay)$,
there exist uniquely determined $t_{0}\in\mathbf{R},y_{0}\in Im(Cay),z_{0}\in
Cay$ such that

$(x_{1},x_{2})=a_{t_{0}}u(y_{0},z_{0})(0,0),$ (9)

where, setting $r=(1-|x_{1}|^{2}-|x_{2}|^{2})^{1/2}$, the objects
$t_{0},y_{0},z_{0}$ are explicitly given (see formula (21) in section 5 of
[Ta79]):

$e^{-t_{0}}=\frac{|1-x_{1}|}{r}$ (10)
$y_{0}=\frac{x_{1}-\overline{x_{1}}}{2r^{2}}$ (11)
$z_{0}=\frac{x_{2}(1-\overline{x_{1}})}{|1-x_{1}|r}$ (12)

Now we have by formulae (9), (7) and (8):

$(x^{\prime}_{1},x^{\prime}_{2})=u(y,z)(x_{1},x_{2})=u(y,z)a_{t_{0}}u(y_{0},z_{0})(0,0)=a_{t_{0}}u(e^{-2t_{0}}y,e^{-t_{0}}z)u(y_{0},z_{0})(0,0)$
$=a_{t_{0}}u(e^{-2t_{0}}y+y_{0}+e^{-t_{0}}Im(\overline{z}z_{0}),e^{-t_{0}}z+z_{0})(0,0)$

So by formulae (2) and (3):

$x^{\prime}_{1}=(\sinh(t_{0})+E)(\cosh(t_{0})+E)^{-1}$ (13)
$x^{\prime}_{2}=(e^{-t_{0}}z+z_{0})(\cosh(t_{0})+E)^{-1}$ (14)

where

$E=e^{t_{0}}(\frac{|e^{-t_{0}}z+z_{0}|^{2}}{2}+e^{-2t_{0}}y+y_{0}+e^{-t_{0}}Im(\overline{z}z_{0}))$
(15)

It remains to express $x^{\prime}_{1},x^{\prime}_{2}$ in terms of
$x_{1},x_{2}$. In the following computations, we will frequently use the
following weak form of associativity in $Cay$: for every $a,b\in Cay$, we
have, by lemma 1.3.3 in [SV00]:

$(ab)\overline{b}=N(b)a.$

We start by checking formula (5). By formulae (10) and (12):

$e^{-t_{0}}z+z_{0}=\frac{|1-x_{1}|z}{r}+\frac{x_{2}(1-\overline{x_{1}})}{r|1-x_{1}|}=\frac{|1-x_{1}|^{2}z+x_{2}(1-\overline{x_{1}})}{r|1-x_{1}|}$
$=\frac{(z(1-x_{1})+x_{2})(1-\overline{x_{1}})}{r|1-x_{1}|}$

Hence

$e^{-t_{0}}z+z_{0}=\frac{(-zx_{1}+x_{2}+z)(1-\overline{x_{1}})}{r|1-x_{1}|}$
(16)

On the other hand by (15):

$E=e^{t_{0}}[\frac{e^{-2t_{0}}|z|^{2}+e^{-t_{0}}\overline{z}z_{0}+e^{-t_{0}}\overline{z_{0}}z+|z_{0}|^{2}}{2}+e^{-2t_{0}}y+y_{0}+e^{-t_{0}}(\frac{\overline{z}z_{0}-\overline{z_{0}}z}{2})]$
$=\frac{e^{-t_{0}}|z|^{2}+2\overline{z}z_{0}+e^{t_{0}}|z_{0}|^{2}+2e^{-t_{0}}y+2e^{t_{0}}y_{0}}{2};$

hence by (10), (11), (12):

$E=\frac{|1-x_{1}|^{2}|z|^{2}+2\overline{z}(x_{2}(1-\overline{x_{1}}))+|x_{2}|^{2}+2y|1-x_{1}|^{2}+x_{1}-\overline{x_{1}}}{2r|1-x_{1}|}.$
(17)

Now by (10) we have, since $r^{2}=1-|x_{1}]^{2}-|x_{2}|^{2}$:

$\cosh(t_{0})=\frac{1}{2}(\frac{r}{|1-x_{1}|}+\frac{|1-x_{1}|}{r})=\frac{r^{2}+|1-x_{1}|^{2}}{2r|1-x_{1}|}=\frac{2-x_{1}-\overline{x_{1}}-|x_{2}|^{2}}{2r|1-x_{1}|}.$

Hence by (17):

$\cosh(t_{0})+E=\frac{|1-x_{1}|^{2}|z|^{2}+2\overline{z}(x_{2}(1-\overline{x_{1}}))+2y|1-x_{1}|^{2}+2-2\overline{x_{1}}}{2r|1-x_{1}|}$

Factoring out $2(1-\overline{x_{1})}$ on the right in the numerator, we get:

$\cosh(t_{0})+E=\frac{[\frac{|z|^{2}}{2}(1-x_{1})+(\overline{z}(x_{2}(1-\overline{x_{1}})))(1-\overline{x_{1}})^{-1}+y(1-x_{1})+1](1-\overline{x_{1}})}{r|1-x_{1}|}$

hence

$\cosh(t_{0})+E=\frac{[-(\frac{|z|^{2}}{2}+y)x_{1}+(\overline{z}(x_{2}(1-\overline{x_{1}})))(1-\overline{x_{1}})^{-1}+1+\frac{|z|^{2}}{2}+y](1-\overline{x_{1}})}{r|1-x_{1}|}.$

Comparing with (6), we get

$\cosh(t_{0})+E=\frac{D}{r|1-x_{1}|}.$ (18)

In view of (14), formula (5) follows immediately by combining (16) and (18).

We now turn to verifying formula (4). Again from (10), we have:

$\sinh(t_{0})=\frac{1}{2}(\frac{r}{|1-x_{1}|}-\frac{|1-x_{1}|}{r})=\frac{r^{2}-|1-x_{1}|^{2}}{2r|1-x_{1}|}=\frac{x_{1}+\overline{x_{1}}-2|x_{1}|^{2}-|x_{2}|^{2}}{2r|1-x_{1}|},$

hence by (17):

$\sinh(t_{0})+E=\frac{|1-x_{1}|^{2}|z|^{2}+2\overline{z}(x_{2}(1-\overline{x_{1}}))+2y|1-x_{1}|^{2}+2x_{1}-2|x_{1}|^{2}}{2r|1-x_{1}|}.$

Factoring out $2(1-\overline{x_{1})}$ on the right in the numerator, we get:

$\sinh(t_{0})+E=\frac{[\frac{|z|^{2}}{2}(1-x_{1})+(\overline{z}(x_{2}(1-\overline{x_{1}})))(1-\overline{x_{1}})^{-1}+y(1-x_{1})+x_{1}](1-\overline{x_{1}})}{r|1-x_{1}|}$

hence

$\sinh(t_{0})+E=\frac{[(1-\frac{|z|^{2}}{2}-y)x_{1}+(\overline{z}(x_{2}(1-\overline{x_{1}})))(1-\overline{x_{1}})^{-1}+\frac{|z|^{2}}{2}+y](1-\overline{x_{1}})}{r|1-x_{1}|}$
(19)

In view of (13), formula (4) follows immediately by combining (18) and (19).
$\blacksquare$

In restriction to the center $Z(N)$, we get:

######  Corollary 4.3.

For $(x_{1},x_{2})\in H^{2}(Cay),y\in Im(Cay)$, we have
$u(y,0)(x_{1},x_{2})=(x^{\prime}_{1},x^{\prime}_{2})$ where:

$x^{\prime}_{1}=((1-y)x_{1}+y)(-yx_{1}+1+y)^{-1}$ (20)
$x^{\prime}_{2}=(x_{2}(1-\overline{x_{1}}))[(1-\overline{x_{1}})^{-1}(-yx_{1}+1+y)^{-1}]$
(21)

Proof Formula (21) follows immediately from (5) and (6) by setting $z=0$. To
obtain formula (20) in the same way, we must justify why we may use
associativity to cancel out $1-\overline{x_{1}}$. The reason is that
subalgebras of $Cay$ generated by 2 elements are associative (see Theorem
1.4.3 in [SV00]), and $\overline{x_{1}}$ belongs to the subalgebra generated
by $x_{1}$ since $\overline{x_{1}}=|x_{1}|^{2}x_{1}^{-1}$. $\blacksquare$

######  Remark 4.4.

Let us observe that, if we replace the Cayley numbers by the Hamilton
quaternions $\mathbf{H}$, formulae (4) and (5) simplify drastically thanks to
associativity:

$x^{\prime}_{1}=((1-\frac{|z|^{2}}{2}-y)x_{1}+\overline{z}x_{2}+\frac{|z|^{2}}{2}+y)(-(\frac{|z|^{2}}{2}+y)x_{1}+\overline{z}x_{2}+1+\frac{|z|^{2}}{2}+y)^{-1},$
$x^{\prime}_{2}=(-zx_{1}+x_{2}+z)(-(\frac{|z|^{2}}{2}+y)x_{1}+\overline{z}x_{2}+1+\frac{|z|^{2}}{2}+y)^{-1}$

for $y\in Im(\mathbf{H}),z\in\mathbf{H}$. This is nothing but the classical
action of the nilpotent part of $Sp(2,1)$ on $H^{2}(\mathbf{H})$ by Möbius
tranformations (see e.g. page 21 in [Li74]).

We now turn to the minimal parabolic subgroup of $G$ i.e. the subgroup
$P=MAN$, where $M$ denotes the commutant of $A$ in $K$.

######  Proposition 4.5.

We have $M=Stab_{K}(1,0)$, the stabilizer of $(1,0)$ in $K$.

Proof : Observe that $(1,0)$ is the unique attracting point of $A$, i.e.

$(1,0)=\lim_{t\rightarrow+\infty}a_{t}(x,y)$

for every $(x,y)\in H^{2}(Cay)$. So for $m\in M$ we have

$m(1,0)=\lim_{t\rightarrow+\infty}ma_{t}(x,y)=\lim_{t\rightarrow+\infty}a_{t}m(x,y)=(1,0),$

so that $M\subset Stab_{K}(1,0)$. To prove the converse inclusion, take $k\in
Stab_{K}(1,0)$. As $k$ stabilizes the line $[0,0]$ through $(0,0)$ and
$(1,0)$, it fixes its point $(0)$ at infinity. It also fixes
$\Pi[0,0]=(\infty)$. So lemma 3.3 applies, and there exists $R\in SO(8)$ such
that $k(x,y)=(R^{\prime\prime}(x),R(y))$. Since $k(1,0)=(1,0)$, we have
$R^{\prime\prime}(1)=1$. We now check that $k$ commutes with $a_{t}$: indeed,
for every $(x,y)\in H^{2}(Cay)$:

$a_{t}k(x,y)=((\cosh(t)R^{\prime\prime}(x)+\sinh(t))(\sinh(t)R^{\prime\prime}(x)+\cosh(t))^{-1},R(y)(\sinh(t)R^{\prime\prime}(x)+\cosh(t))^{-1})$
$=(R^{\prime\prime}((\cosh(t)x+\sinh(t))(\sinh(t)x+\cosh(t))^{-1}),R^{\prime}(y)R^{\prime\prime}((\sinh(t)x+\cosh(t))^{-1})\;\mbox{by
lemma \ref{monlemme};}$
$=(R^{\prime\prime}((\cosh(t)x+\sinh(t))(\sinh(t)x+\cosh(t))^{-1}),R(y(\sinh(t)x+\cosh(t))^{-1}))\;\mbox{by
Theorem \ref{trial};}$ $=ka_{t}(x,y).$

This concludes the proof. $\blacksquare$

It was proved by Borel [Bo50] and reproved by Tits (see (6.9) in [Ti53]) that
$Stab_{K}(1,0)$ is isomorphic to $Spin(7)$, the universal cover of $SO(7)$.

## References

  * [Al98] D. Allcock,  Reflection groups on the octave hyperbolic plane,  Journal of Algebra 213 (1998), 467-498.
  * [Ba02] J.C. Baez.  The octonions,  Bull. Amer. Math. Soc. 39 (2002), 145-205.
  * [Bo50] A. Borel,  Le plan projectif des octaves et les sphères comme espaces homogènes,  C. R. Acad. Sci. (Ser. A) 230 (1950), 1378-1380.
  * [Fr53] H. Freudenthal,  Zur ebene Oktavengeometrie,  Proc. Kon. Nederl. Akad. Wetensch. Ser. A, 56 (1953), 195-200.
  * [Li74] R.L. Lipsman,  Group representations,  Lect. Notes Math. 388, Springer (1974).
  * [Mo73] G.D. Mostow,  Strong rigidity of locally symmetric spaces,  Annals of math. Studies 78, Princeton Univ. Press, 1973.
  * [Ni11] A. Nishio,  The orbit decomposition and orbit type of the automorphism group of certain exceptional Jordan algebra and its applications,  RIMS Kôkyûroku 1770 (2011), Problems in Representation Theory and Harmonic Analysis (H. Oda ed.), 1-19.
  * [SV00] T.A. Springer and F.D. Veldkamp,  Octonions, Jordan algebras, and exceptional groups,  Springer Monographs in Mathematics, 2000.
  * [Ta79] R. Takahashi,  Quelques résultats sur l’analyse harmonique dans l’espace symétrique non compact de rang 1 du type exceptionnel,  Lect. Notes Math. 739, Springer (1979), 511-567.
  * [Ti53] J. Tits,  Le plan projectif des octaves et les groupes de Lie exceptionnels,  Acad. Roy. Belg. Bull. Cl. Sci. 39 (1953), 309-329.
  * [Ti67] J. Tits.  Tabellen zu den einfachen Lie Gruppen und ihren Darstellungen,  Lect. Notes Math. 40, Springer, 1967.
  * [Zi84] R.J. Zimmer,  Ergodic theory and semisimple Lie group,  Monographs in Math. 81, Birkhäuser, 1984.

Author’s address:
Institut de Mathématiques
Université de Neuchâtel
Unimail
11 Rue Emile Argand
CH-2000 Neuchâtel - SWITZERLAND

<EMAIL_ADDRESS>


# Evaluating the Evaluation Metrics for Style Transfer:
A Case Study in Multilingual Formality Transfer

Eleftheria Briakou, Sweta Agrawal, Joel Tetreault, Marine Carpuat
University of Maryland, Dataminr, Inc.
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

While the field of style transfer (st) has been growing rapidly, it has been
hampered by a lack of standardized practices for automatic evaluation. In this
paper, we evaluate leading st automatic metrics on the oft-researched task of
formality style transfer. Unlike previous evaluations, which focus solely on
English, we expand our focus to Brazilian-Portuguese, French, and Italian,
making this work the first multilingual evaluation of metrics in st. We
outline best practices for automatic evaluation in (formality) style transfer
and identify several models that correlate well with human judgments and are
robust across languages. We hope that this work will help accelerate
development in st, where human evaluation is often challenging to collect.

##  1 Introduction

Textual style transfer (st) is defined as a generation task where a text
sequence is paraphrased while controlling one aspect of its style Jin et al.
(2020). For instance, the informal sentence in Italian “in bocca al lupo!”
(i.e., “good luck”) is rewritten to the formal version “Ti rivolgo un sincero
augurio!” (i.e., “I send you a sincere wish!”). Despite the growing attention
on st in the nlp literature (Jin et al., 2020), progress is hampered by a lack
of standardized and reliable automatic evaluation metrics. Standardizing the
latter would allow for quicker development of new methods and comparison to
prior art without relying on time and cost-intensive human evaluation that is
currently employed by more than $70\%$ of st papers Briakou et al. (2021a).

st is usually evaluated across three dimensions: style transfer (i.e., has the
style of the generated output changed as intended?), meaning preservation
(i.e., are the semantics of the input preserved?), and fluency (i.e., is the
output well-formed?). As we will see, a wide range of automatic evaluation
metrics and models has been used to quantify each of these dimensions. For
example, prior work has employed as many as nine different automatic systems
to rate formality alone (see Table 1). However, it is not clear how different
automatic metrics compare to each other and how well they agree with human
judgments. Furthermore, previous studies of automatic evaluation have
exclusively focused on the English language Yamshchikov et al. (2021); Pang
(2019); Pang and Gimpel (2019); Tikhonov et al. (2019); Mir et al. (2019);
yet, st requires evaluation methods that generalize reliably beyond English.

We address these limitations by conducting a controlled empirical comparison
of commonly used automatic evaluation metrics. Concretely, for all three
evaluation dimensions, we compile a list of different automatic evaluation
approaches used in prior st work and study how well they correlate with human
judgments. We choose to build on available resources as collecting human
judgments across the evaluation dimensions is a costly process that requires
recruiting fluent speakers in each language addressed in evaluation. While
there are many stylistic transformations in st, we conduct our study through
the lens of formality style transfer (fost), which is one of the most popular
style dimensions considered by past st work (Jin et al., 2020; Briakou et al.,
2021a) and for which reference outputs and human judgments are available for
four languages: English, Brazilian-Portuguese, French, and Italian.

  * •

We contribute a meta-evaluation study that is not only the first large-scale
comparison of automatic metrics for st but is also the first work to
investigate the robustness of these metrics in multilingual settings.

  * •

We show that automatic evaluation approaches based on a formality regression
model fine-tuned on xlm-r and the chrf metric correlate well with human
judgments for style transfer and meaning preservation, respectively, and
propose that the field adopts their usage. These metrics are shown to work
well across languages, and not just in English.

  * •

We show that framing style transfer evaluation as a binary classification task
is problematic and propose that the field treats it as a regression task to
better mirror human evaluation.

  * •

Our analysis code and meta-evaluation files with system outputs are made
public to facilitate further work in developing better automatic metrics for
st: https://github.com/Elbria/xformal-FoST-meta.

##  2 Background

paper id | style | meaning | fluency | overall
---|---|---|---|---
| metric | arch. |  | metric | arch. |  | metric | arch. |  | metric |
[1] | reg | Linear reg. |  ✗ | cls | cnn |  ✓ | reg | Linear reg. |  ✗ | r-bleu | ✓
[2] |  |  |  |  |  |  |  |  |  | r-bleu | $-$
[3] | cls | cnn |  $-$ | r-bleu |  |  $-$ |  |  |  | gm(s,m) | $-$
[4] | cls | cnn |  ✗ | r-bleu |  |  ✓ |  |  |  | gm(s,m) | ✓
[5] | cls | cnn |  ✗ | r-bleu |  |  ✓ |  |  |  |  |
[6] | cls | lstm |  $-$ | cls | bert |  $-$ |  |  |  | r-bleu | $-$
[7] |  |  |  |  |  |  |  |  |  | r-bleu | $-$
[8] | cls | cnn |  $-$ |  |  |  |  |  |  |  |
[9] | cls | lstm |  $-$ | emb-sim |  |  $-$ | ppl | lm (rnn) |  $-$ | f$1$(s,m) | $-$
[10] | cls | Roberta |  ✗ | emb-sim |  |  ✓ | ppl |  lm (Roberta) |  $-$ | j(s,m,f) | ✓
[11] | cls | cnn |  ✗ | r-bleu |  |  ✗ |  |  |  | f$1$(s,m) | $-$
[12] | cls | gru |  ✗ |  |  |  | cls | Linear reg. |  ✗ | r-bleu | ✗
[13] | cls | bert |  ✓ | r-bleu |  |  ✓ | ppl |  lm (Kenlm) |  ✗ | gm(s,m,f) | $-$
[14] |  |  |  |  |  |  |  |  |  | r-bleu | $-$
[15] | cls | fastText |  ✓ | r-bleu |  |  ✓ | ppl |  lm (gpt) |  ✓ |  |
[16] | cls | cnn |  $-$ | r-bleu |  |  $-$ | ppl |  lm (lstm) |  $-$ |  |
[17] | cls | cnn |  ✓ | r-bleu |  |  ✓ |  |  |  |  |
[18] | cls | cnn |  ✓ | r-bleu |  |  ✓ |  |  |  | gm|hm(s,m) | ✓
[19] | cls | gru |  $-$ | cls | bert |  $-$ |  |  |  | r-bleu | ✓
[20] | cls | Roberta |  $-$ | r-bleu |  |  $-$ | ppl |  lm (gpt) |  $-$ | gm|hm(s,m) | $-$
[21] | cls | cnn |  $-$ | r-bleu |  |  ✓ | ppl |  lm (gpt) |  ✓ |  |
[22] |  |  |  |  |  |  |  |  |  | r-bleu | $-$
[23] | reg | bert |  ✗ | s-bleu |  |  ✓ | ppl |  lm (Kenlm) |  ✗ | r-bleu | ✗

Table 1: Details on automatic evaluation practices of prior work for fost. For
each dimension, ✓ and ✗ denote whether the best ranking system based on the
automatic evaluation agrees or disagrees with the one pointed by the human
evaluation; $-$ denotes that no human evaluation is conducted. reg stands for
regression, cls for classification, sim for similarity, emb for embedding-
based, lm for language model, gm or hm for geometric or harmonic mean, j for
corpus-level product, and arch. for architecture. The mappings between each
paper id and its corresponding citation is included in Table 2.

paper id |  citation | language
---|---|---
[1] |  Rao and Tetreault (2018) | en
[2] |  Kajiwara (2019) | en
[3] |  Li et al. (2019) | en
[4] |  Luo et al. (2019) | en
[5] |  Shang et al. (2019) | en
[6] |  Wang et al. (2019) | en
[7] |  Xu et al. (2019) | en
[8] |  Korotkova et al. (2019) |  en, et, lv
[9] |  Gong et al. (2019) | en
[10] |  Krishna et al. (2020) | en
[11] |  Sancheti et al. (2020) | en
[12] |  Wu et al. (2020) | en
[13] |  Yi et al. (2020) | en
[14] |  Zhang et al. (2020b) | en
[15] |  Goyal et al. (2020) | en
[16] |  He et al. (2019) | en
[17] |  Cheng et al. (2020) | en
[18] |  Zhou et al. (2020) | en
[19] |  Wang et al. (2020) | en
[20] |  Li et al. (2020) | en
[21] |  Liu et al. (2021) | en
[22] |  Lyu et al. (2021) | en
[23] |  Briakou et al. (2021b) |  br-pt, fr, it

Table 2: List of prior works on fost along with languages (i.e., iso codes)
addressed in each of them.

###  2.1 Limitations of Automatic Evaluation

Recent work highlights the need for research to improve evaluation practices
for st along multiple directions. Not only does st lack standardized
evaluation practices Yamshchikov et al. (2021), but commonly used methods have
major drawbacks which hamper progress in this field. Pang (2019) and Pang and
Gimpel (2019) show that the most widely adopted automatic metric, bleu, can be
gamed. They observe that untransferred text achieves the highest bleu score
for the task of sentiment transfer, questioning complex models’ ability to
surpass this trivial baseline. Mir et al. (2019) discuss the inherent trade-
off between st evaluation aspects and propose that models are evaluated at
specific points of their trade-off plots. Tikhonov et al. (2019) argue that,
despite their cost, human-written references are needed for future experiments
with style transfer. They also show that comparing models without reporting
error margins can lead to incorrect conclusions as state-of-the-art models
sometimes end up within error margins from one another.

###  2.2 Structured Review of st Evaluation

We systematically review automatic evaluation practices in st with formality
as a case study. We select fost for this work since it is one of the most
frequently studied styles Jin et al. (2020) and there is human annotated data
including human references available for these evaluations Rao and Tetreault
(2018); Briakou et al. (2021b). Tables 1 and 2 summarize evaluation details
for all fost methods in papers from the st survey by Jin et al. (2020).111The
complete list is hosted at: https://github.com/fuzhenxin/Style-Transfer-in-
Text Most works employ automatic evaluation for style ($87\%$) and meaning
preservation ($83\%$). Fluency is the least frequently evaluated dimension
($43\%$), while $74\%$ of papers employ automatic metrics to assess the
overall quality of system outputs that captures all desirable aspects.

Across dimensions, papers also frequently rely on human evaluation ($55\%$,
$58\%$, $60\%$, and $40\%$ for style, meaning, fluency, and overall). However,
human judgments and automatic metrics do not always agree on the best-
performing system. In $60\%$ of evaluations, the top-ranked system is the same
according to human and automatic evaluation (marked as ✓ in Table 1), and
their ranking disagrees in $40\%$ of evaluations (marked as ✗ in Table 1).
When there is a disagreement, human evaluation is trusted more and viewed as
the standard. This highlights the need for a systematic evaluation of
automatic evaluation metrics.

Finally, almost all papers ($91\%$) consider fost for English (en), as
summarized in Table 2. There are only two exceptions: Korotkova et al. (2019)
study fost for Latvian (lv) and Estonian (et) in addition to en, while Briakou
et al. (2021b) study fost for $3$ Romance languages: Brazilian Portuguese (br-
pt), French (fr), and Italian (it). The former provides system output samples
as a means of evaluation, and the latter employs human evaluations,
highlighting the challenges of automatic evaluation in multilingual settings.

Next, we review the automatic metrics used for each dimension of evaluation in
fost papers. As we will see, a wide range of approaches is used. Yet, it
remains unclear how they compare to each other, what their respective
strengths and weaknesses are, and how they might generalize to languages other
than English.

###  2.3 Automatic Metrics for fost

##### Formality

Style transfer is often evaluated using model-based approaches. The most
frequent method consists of training a binary classifier on human written
formal vs. informal pairs. The classifier is later used to predict the
percentage of generated outputs that match the desired attribute per evaluated
system—the system with the highest percentage is considered the best
performing with respect to style. Across methods, the corpus used to train the
classifier is the gyafc parallel-corpus Rao and Tetreault (2018) consisting of
$105$K parallel informal-formal human-generated excerpts. This corpus is
curated for fost in en, while similar resources are not available for other
languages. Different model architectures have been used by prior work (e.g.,
cnn, lstm, gru, fine-tuning on pre-trained language models such as Roberta and
bert; Table 1). In most papers, the resulting classifier is evaluated on the
test side of the gyafc corpus, reporting accuracy scores in the range of
$80-90$%. Despite the high accuracy scores, the best ranking system under the
classifier is very often in disagreement with human evaluations (marked as ✗
under the third subcolumn of style of Table 1). A few works train regression-
based models instead, using the training data of Pavlick and Tetreault (2016)
that are human-annotated for formality on a $7$-point scale—while, again, this
resource is only available for en.

##### Meaning Preservation

Evaluation of this dimension is performed using a wider spectrum of
approaches, as presented in the third column of Table 1. The most frequently
used metric is reference-bleu (r-bleu), which is based on the $n$-gram
precision of the system output compared to human rewrites of the desired
formality. Other approaches include self-bleu (s-bleu), where the system
output is compared to its input, measuring the semantic similarity between the
system input and its output, or regression models (e.g., cnn, bert) trained on
data annotated for similarity-based tasks, such as the Semantic Textual
Similarity task (sts) Agirre et al. (2016).

##### Fluency

Fluency is typically evaluated with model-based approaches (see fourth column
of Table 1). Among those, the most frequent method is that of computing
perplexity (ppl) under a language model. The latter is either trained from
scratch on the same corpus used to train the fost models (i.e., gyafc) using
different underlying architectures (e.g., Kenlm, lstm), or employ large pre-
trained language models (e.g., gpt). A few other works train models on en data
annotated for grammaticality (Heilman et al., 2014) or linguistic
acceptability (Warstadt et al., 2019) instead.

##### Overall

Systems’ overall quality (see fifth column of Table 2) is mostly evaluated
using r-bleu or by combining independently computed metrics into a single
score (e.g., geometric mean - gm(.), harmonic mean - hm(.), f$1$(.)).
Moreover, $6$ out of $8$ approaches that rely on combined scores do not
include fluency scores in their overall evaluation.

##### English Focus

Since most of the current work on fost and st is in en, prior work relies
heavily on en resources for designing automatic evaluation methods. For
instance, resources for training stylistic classifiers or regression models
are not available for other languages. For the same reason, it is unclear
whether model-based approaches for measuring meaning preservation and fluency
can be ported to multilingual settings. Furthermore, reference-based
evaluations (e.g., r-bleu) require human rewrites that are only available for
en, br-pt, it, and fr. Finally, even though perplexity does not rely on
annotated data, without standardizing the data language models are trained on,
we cannot make meaningful cross-system comparisons.

###  2.4 Summary

Reviewing the literature shows the lack of standardized metrics for st
evaluation, which hampers comparisons across papers, the lack of agreement
between human judgments and automatic metrics, which hampers system
development, and the lack of portability to languages other than English which
severely limits the impact of the work. These issues motivate the controlled
multilingual evaluation of evaluation metrics in our paper.

##  3 Evaluating Evaluation Metrics

We evaluate evaluation metrics (described in §3.2) for multilingual fost, in
four languages for which human evaluation judgments (described in §3.1) on
fost system outputs are available.

###  3.1 Human Judgments

We use human judgments collected by prior work of Rao and Tetreault (2018) for
en and Briakou et al. (2021b) for br-pt, fr, and it. We include details on
their annotation frameworks, the quality of human judges, and the evaluated
systems below.

##### Human Annotations

We briefly describe the annotation frameworks employed by Rao and Tetreault
(2018) and Briakou et al. (2021b) to collect human judgments for each
evaluation aspect:  1. formalityratings are collected—for each system
output—on a $7$-point discrete scale, ranging from $-3$ to $+3$, as per Lahiri
(2015) (Very informal, Informal, Somewhat Informal, Neutral, Somewhat Formal,
Formal. Very Formal);  2. meaning preservationjudgments adopt the Semantic
Textual Similarity annotation scheme of Agirre et al. (2016), where an
informal input and its corresponding formal system output are rated on a scale
from $1$ to $6$ based on their similarity  (Completely dissimilar, Not
equivalent but on same topic, Not equivalent but share some details, Roughly
equivalent, Mostly equivalent, Completely equivalent);  3. fluencyjudgments
are collected for each system output on a discrete scale of $1$ to $5$, as per
Heilman et al. (2014) (Other, Incomprehensible, Somewhat Comprehensible,
Comprehensible, Perfect);  4. overalljudgments are collected following a
relative ranking approach: all system outputs are ranked in order of their
formality, taking into account both meaning preservation and fluency.

##### Human Annotators

Both studies recruited workers from the Amazon Mechanical Turk platform after
employing quality control methods to exclude poor quality workers (i.e.,
manual checks for en, and qualification tests for br-pt, fr, and it). For all
human evaluations and languages Briakou et al. (2021b) report at least
moderate inter-annotator agreement.

##### Evaluated Systems

The evaluated system outputs were sampled from $5$ fost models for each
language, spanning a range of simple baselines to neural architectures Rao and
Tetreault (2018); Briakou et al. (2021b). We also include detailed
descriptions of them in Appendix B. For each evaluation dimension $500$
outputs are evaluated for en and $100$ outputs per system for br-pt, fr, and
it.

###  3.2 Evaluation Metrics

For the fost evaluation aspects described below, we cover a broad spectrum of
approaches that range from dedicated models for the tasks at hand to more
lightweight methods relying on unsupervised approaches and automated metrics.

##### Formality

We benchmark model-based approaches that fine-tune multilingual pre-trained
language models (i.e., xlm-r, mbert), where the task of formality detection is
modeled either as a binary classification task (i.e., formal vs. informal), or
as a regression task that predicts different formality levels on an ordinal
scale.

##### Meaning Preservation

We evaluate the bleu score Papineni et al. (2002) of the system output
compared to the reference rewrite (r-bleu) since it is the dominant metric in
prior work. Prior reviews of meaning preservation metrics for paraphrase and
sentiment st tasks in en (Yamshchikov et al., 2021) cover $n$-gram metrics and
embedding-based approaches. We consider three additional metric classes to
compare system outputs with inputs, as human annotators do:

  1. 1.

n-gram based metrics include: s-bleu (self-bleu that compares system outputs
with their inputs as opposed to references, i.e., r-bleu), meteor Banerjee and
Lavie (2005) based on the harmonic mean of unigram precision and recall while
accounting for synonym matches, and chrF Popović (2015) based on the character
$n$-gram F-score;

  2. 2.

embedding-based methods fall under the category of unsupervised evaluation
approaches that rely on either contextual word representations extracted from
pre-trained language models or non-contextual pre-trained word embeddings
(e.g., word2vec Mikolov et al. (2013); Glove Pennington et al. (2014)). For
the former, we use bert-score Zhang et al. (2020a) which computes the
similarity between each output token and each reference token based on bert
contextual embeddings. For the latter, we experiment with two similarity
metrics: the first is the cosine distance between the sentence-level feature
representations of the compared texts extracted via averaging their word
embeddings; the second is the Word Mover’s Distance (wmd) metric of Kusner et
al. (2015) that measures the dissimilarity between two texts as the minimum
amount of distance that the embedded words of one text need to “travel" to
reach the word embeddings of the other;

  3. 3.

semantic textual similarity (sts) models constitute supervised methods that we
model via fine-tuning multilingual pre-trained language models (i.e., xlm-r,
mbert) to predict a semantic similarity score for a pair of texts on an
ordinal scale.

##### Fluency

We experiment with perplexity (ppl) and likelihood (ll) scores based on
probability scores of language models trained from scratch (e.g., Kenlm
Heafield (2011)), as well as pseudo-likelihood scores (pseudo-ll) extracted
from pre-trained masked language models similarly to Salazar et al. (2020), by
masking sentence tokens one by one.

##  4 Experiment Settings

dimension | language code | dataset | lineage | labels | size
---|---|---|---|---|---
Formality | en | gyafc | Rao and Tetreault (2018) | informal vs. formal |  $105$K
Formality ratings | Pavlick and Tetreault (2016) |  $[$ $-3$, $-2$, $-1$, $0$, $1$, $2$, $3$ $]$ |  $5$K
Meaning | en | sts | Cer et al. (2017) |  $[$ $1$, $2$, $3$, $4$, $5$ $]$ |  $5$K
Fluency |  en, br-pt, it, fr | OpenSubtitles | Lison and Tiedemann (2016) | none |  $1$M

Table 3: Details on training data used for model-based metrics across the
three st evaluation aspects.

##### Supervised Metrics

For all supervised model-based approaches, we experiment with fine-tuning two
multilingual pre-trained language models:  1. multilingual bert, dubbed mbert
Devlin et al. (2019)—a transformer-based model pre-trained with a masked
language model objective on the concatenation of monolingual Wikipedia corpora
from the $104$ languages with the largest Wikipedias.  2. xlm-rConneau et al.
(2020)—a transformer-based masked language model trained on $100$ languages
using monolingual CommonCrawl data.  All models are based on the HuggingFace
Transformers Wolf et al. (2020)222https://github.com/huggingface/transformers
library. We fine-tune with the Adam optimizer Kingma and Ba (2015), a batch
size of $32$, and a learning rate of $5\mathrm{e}{-5}$ for $3$ and $5$ epochs
for classification and regression tasks, respectively. We perform a grid
search on held-out validation sets over learning rate with values:
$2\mathrm{e}{-3}$, $2\mathrm{e}{-4}$, $2\mathrm{e}{-5}$, and $5\mathrm{e}{-5}$
and over number of epochs with values: $3$, $5$, and $8$.

##### Cross-lingual Transfer

For supervised model-based methods that rely on the availability of human-
annotated instances to train dedicated models for specific tasks, we
experiment with three standard cross-lingual transfer approaches (e.g., Hu et
al. (2020)):  1. zero-shottrains a single model on the en training data and
evaluates it on the original test data for each language;  2. translate-
trainuses machine translation (mt) to obtain training data in each language
through translating the en training set—and trains independent systems for
each language;  3. translate-testtrains a single model on the en training data
and evaluates it on the test data that are translated into en using mt.

##### Unsupervised Metrics

For meaning preservation metrics, we use the open-sourced implementations of:
Post (2018) for bleu Papineni et al. (2002); Banerjee and Lavie (2005) for
meteor; Popović (2015) for
chrf.333https://github.com/mjpost/sacrebleu,444https://www.cs.cmu.edu/~alavie/METEOR/,555https://github.com/m-popovic/chrF
For bert-score we use the implementation of Zhang et al.
(2020a);666https://github.com/Tiiiger/bert_score non-contextualized
embeddings-based approaches are based on fastText pre-trained
embeddings.777https://fasttext.cc For fluency metrics, we use the
implementation of Salazar et al. (2020) for computing pseudo-
likelihood.888https://github.com/awslabs/mlm-scoring ppl and ll scores are
extracted from a $5$-gram Kenlm model (Heafield,
2011).999https://github.com/kpu/kenlm

##### Training Data

Table 3 presents statistics on the training data used for supervised and
unsupervised models across the $3$ st evaluation aspects. For datasets that
are only available for en, we use the already available machine translated
resources for sts 101010https://github.com/PhilipMay/stsb-multi-mt and
formality datasets Briakou et al. (2021b). The former employs the DeepL
service (no information of translation quality is available) while the latter
uses the aws translation service111111https://aws.amazon.com/translate (with
reported bleu scores of $37.16$ (br-pt), $33.79$ (fr), and $32.67$
(it)).121212bleu scores were computed on $5{,}000$ randomly sampled data from
OpenSubtitles. The Kenlm models for all the languages are trained on $1$M
randomly sampled sentences from the OpenSubtitles dataset Lison and Tiedemann
(2016).

##  5 Experimental Results

We analyze the results of comparing the outputs from the several automatic
metrics to their human-generated counterparts for formality style transfer
(§5.1), meaning preservation (§5.2), fluency (§5.3) via conducting segment-
level analysis—and then, turn into analyzing system-level rankings to
evaluation overall task success (§5.4).

method | mbert | xlm-r |  $\mathbf{\delta}$(xlm-r, mbert)
---|---|---|---
| en | br-pt | fr | it | en | br-pt | fr | it | en | br-pt | fr | it
zero-shot | $90$ | $87$ | $84$ | $88$ | $90$ | $90$ | $86$ | $89$ | $0$ | $+3$ | $+2$ | $+1$
translate-train |  |  $85$ ($\mathbf{-2}$) |  $82$ ($\mathbf{-2}$) |  $83$ ($\mathbf{-5}$) |  |  $87$ ($\mathbf{-3}$) |  $84$ ($\mathbf{-2}$) |  $84$ ($\mathbf{-5}$) |  | $+2$ | $+2$ | $+1$
translate-test |  |  $82$ ($\mathbf{-5}$) |  $73$ ($\mathbf{-11}$) |  $79$ ($\mathbf{-11}$) |  |  $82$ ($\mathbf{-8}$) |  $74$ ($\mathbf{-12}$) |  $80$ ($\mathbf{-9}$) |  | $0$ | $+2$ | $+1$

Table 4: F1 scores of binary formality classifiers under different cross-
lingual transfer settings. Numbers in parentheses indicate performance drops
over zero-shot. zero-shot yields the highest scores across languages and pre-
trained language models. xlm-r yields improvements over mbert across most
setting ($\delta$(xlm-r, mbert)).

(a) en

(b) br-pt

(c) fr

(d) it

Figure 1: Number of formal vs. informal predictions of binary classifiers for
each formality bin. Binary classifiers are biased towards outputing formal
predictions for br-pt, fr, and it, while their performance degrades when
moving closer to more neutral formality levels.

method | mbert | xlm-r |  $\mathbf{\delta}$(xlm-r, mbert)
---|---|---|---
| en | br-pt | fr | it | en | br-pt | fr | it | en | br-pt | fr | it
zero-shot | $66$ | $64$ | $51$ | $51$ | $67$ | $72$ | $59$ | $61$ | $+1$ | $+8$ | $+8$ | $+10$
translate-train |  |  $61$ ($\mathbf{-3}$) |  $48$ ($\mathbf{-3}$) |  $49$ ($\mathbf{-2}$) |  |  $63$ ($\mathbf{-9}$) |  $52$ ($\mathbf{-7}$) |  $55$ ($\mathbf{-6}$) |  | $+2$ | $+4$ | $+6$
translate-test |  |  $60$ ($\mathbf{-4}$) |  $39$ ($\mathbf{-12}$) |  $45$ ($\mathbf{-10}$) |  |  $61$ ($\mathbf{-11}$) |  $37$ ($\mathbf{-15}$) |  $51$ ($\mathbf{-10}$) |  | $+1$ | $-2$ | $+6$

Table 5: Spearman’s $\rho$ correlation (%) of formality regression models.
Numbers in parentheses indicate performance drops over zero-shot. zero-shot
yields the highest scores across languages and pre-trained language models.
xlm-r yields improvements over mbert across most settings ($\delta$(xlm-r,
mbert)).

###  5.1 Formality Transfer Metrics

The field is divided on the best way to evaluate the style dimension –
formality in our case. Practitioners use either a binary approach (is the new
sentence formal or informal?) or a regression approach (how formal is the new
sentence?). We discuss the first approach and its limitations in § 5.1.1,
before moving to regression in § 5.1.2.

####  5.1.1 Evaluating Binary Classifiers

As discussed in §2, the vast majority of fost works evaluate style transfer
based on the accuracy of a binary classifier trained to predict whether human-
written segments are formal or informal. Yet, as Table 1 indicates, this
approach fails to identify the best system in this dimension 59% of the time.
To better understand this issue, we evaluate these classifiers on human-
written texts versus st system outputs.

##### Human Written Texts

Table 4 presents F$1$ scores when testing the binary formality classifiers on
the task they are trained on: predicting whether human-written sentences from
gyafc and xformal are formal or informal. First, the last column (i.e.,
$\delta$(xlm-r, mbert)) shows that xlm-r is a better model than mbert for this
task, across languages, with the largest improvements in the zero-shot setting
where xlm-r beats mbert by $+3$, $+2$, $+1$ for br-pt, fr, and it
respectively.

Second, zero-shot is surprisingly the best strategy to port en models to other
languages. translate-train and translate-test hurt F1 by $3$ and $9$ points on
average compared to zero-shot, despite exploiting more resources in the form
of machine translation systems and their training data. However, transfer
accuracy is likely affected by regular translation errors (as suggested by
larger F1 drops for languages with lower mt bleu scores) and by formality-
specific errors. Machine translation has been found to produce outputs that
are more formal than its inputs (Briakou et al., 2021b), which yields noisy
training signals for translate-train and alters the formality of test samples
for translate-test.

##### System Outputs

We now evaluate the best performing binary classifier (i.e., xlm-r in zero-
shot setting) on real system outputs—a setup in line with automatic evaluation
frameworks. Figure 1 presents a breakdown of the number of formal vs. informal
predictions of the classifiers binned by human-rated formality levels. Across
languages, the performance of the classifier deteriorates as we move away from
extreme formality ratings (i.e., very informal ($-3$) and very formal ($+3$)).
This lack of sensitivity to different formality levels is problematic since
system outputs across languages are concentrated around neutral formality
values. In addition, when testing on br-pt, fr, and it (zero-shot settings),
the classifier is more biased towards the formal class, which leads one to
question its ability to correctly evaluate more formal outputs in multilingual
settings. Taken together, these results suggest that validating the
classifiers against human rewrites rather than system outputs is unrealistic
and potentially misleading.

####  5.1.2 Regression Models

Table 5 presents Spearman’s $\rho$ correlation of regression models’
predictions with human judgments. Again, xlm-r with zero-shot transfer yields
the highest correlation across languages. More specifically, the trends across
different transfer approaches and different pre-trained language models are
similar to the ones observed on evaluation of binary classifiers: xlm-r
outperforms mbert for almost all settings, while zero-shot is the most
successful transfer approach, followed by translate-train, with translate-test
yielding the lowest correlations across languages. Interestingly, regression
models highlight the differences between the generalization abilities of xlm-r
and mbert more clearly than the previous analysis on binary predictions: zero-
shot transfer on xlm-r yields $8\%$, $8\%$, and $10\%$ higher correlations
than mbert for br-pt, fr, and it—while both models yield similar correlations
for en.

###  5.2 Meaning Preservation Metrics

| metric | en | it | br-pt | fr
---|---|---|---|---|---
| r-bleu | $0.31$ | $0.00$ | $0.05$ | $0.12$
| s-bleu | $0.48$ | $0.59$ | $0.67$ | $0.66$
| meteor | $0.54$ | $0.58$ | $0.64$ | $0.60$
| chrf | $\mathbf{0.59}$ | $\mathbf{0.67}$ | $\mathbf{0.70}$ | $\mathbf{0.75}$
| wmd | $0.51$ | $0.52$ | $0.57$ | $0.55$
unsupervised | Cosine | $0.49$ | $0.50$ | $0.54$ | $0.52$
|  bert-score | $0.59$ | $0.62$ | $0.65$ | $0.68$
| translate-train (sts)
| m-bert | $0.53$ | $0.58$ | $0.51$ | $0.63$
| xlm-r | $0.55$ | $0.64$ | $0.57$ | $0.60$
| zero-shot (sts)
supervised | m-bert | - | $0.61$ | $0.60$ | $0.66$
| xlm-r | - | $\mathbf{0.67}$ | $0.68$ | $0.65$

Table 6: Spearman’s $\rho$ correlation of meaning preservation metrics with
human judgments.

Table 6 presents Spearman’s $\rho$ correlation of meaning preservation metrics
with human judgments. chrF consistently yields the highest correlations across
languages—this result is in line with prior observations on evaluating meaning
preservation metrics for en st tasks Yamshchikov et al. (2021) and is now
confirmed in a multilingual setting. This trend might be explained by chrf’s
ability to match spelling errors within words via character $n$-grams. xlm-r
trained on sts with zero-shot transfer is a close second to chrf, consistent
with this model’s top-ranking behavior as a formality transfer metric.
However, chrf outperforms the remaining more complex and expensive metrics,
including bert-score and mbert models. In contrast to Yamshchikov et al.
(2021), embedding-based methods (i.e., cosine, wmd) show no advantage over
$n$-gram metrics, perhaps due to differences in word embedding quality across
languages. Finally, it should be noted that r-bleu is the worst performing
metric across languages, and its correlation with human scores is particularly
poor for languages other than English. This is remarkable because it has been
used in $75\%$ of automatic evaluations for fost meaning preservation
evaluation (as seen in Table 1). We, therefore, recommend discontinuing its
use.

###  5.3 Fluency Metrics

Table 7 presents Spearman’s $\rho$ correlation of various fluency metrics with
human judgments. Pseudo-likelihood (pseudo-ll) scores obtained from xlm-r
correlate with human fluency ratings best across languages. Their correlations
are strong across languages, while other methods only yield weak (i.e., Kenlm,
mbert) to moderate correlations (i.e, Kenlm-ppl) for it. We, therefore,
recommend evaluating fluency using Pseudo-likelihood scores derived from xlm-r
to help standardize fluency evaluation across languages.

method | en | it | pt | fr
---|---|---|---|---
Kenlm (ll) | $0.33$ | $0.27$ | $0.43$ | $0.39$
Kenlm (ppl) | $0.40$ | $0.35$ | $0.45$ | $0.41$
mbert (pseudo-ll) | $0.42$ | $0.28$ | $0.43$ | $0.41$
xlm-r (pseudo-ll) | $\mathbf{0.50}$ | $\mathbf{0.46}$ | $\mathbf{0.55}$ | $\mathbf{0.61}$

Table 7: Spearman’s $\rho$ correlation of fluency metrics with human
judgments.

###  5.4 System-level Rankings

Finally, we turn to predict the overall ranking of systems by focusing on how
many correct pairwise system comparisons each metric gets correct. For each
language, there are $5$ systems, which means there are $10$ pairwise
comparisons, for a total of $40$ given the $4$ languages. We analyze corpus-
level r-bleu, commonly used for this dimension, along with leading metrics
from the other dimensions: xlm-r formality regression models, chrf and xlm-r
pseudo-likelihood. r-bleu gets $30$ out of $40$ comparisons correct while the
other metrics get $25$, $22$, and $19$ respectively. This indicates that
r-bleu correlates with human judgments better at the corpus-level than at the
sentence-level, as in machine translation evaluation (Mathur et al., 2020). We
caution that these results are not definitive but rather suggestive of the
best performing metric, given the ideal evaluation would be a larger number of
systems with which to perform a rank correlation. The complete analysis for
each language is in Appendix A.

##  6 Conclusions

Automatic (and human) evaluation processes are well-known problems for the
field of Natural Language Generation Howcroft et al. (2020); Clinciu et al.
(2021) and the burgeoning subfield of st is not immune. st, in particular, has
suffered from a lack of standardization of automatic metrics, a lack of
agreement between human judgments and automatics metrics, as well as a
blindspot to developing metrics for languages other than English. We address
these issues by conducting the first controlled multilingual evaluation for
leading st metrics with a focus on formality, covering metrics for $3$
evaluation dimensions and overall ranking for $4$ languages. Given our
findings, we recommend the formality style transfer community adopt the
following best practices:

##### 1\. Formality

xlm-r formality regression models in the zero-shot cross-lingual transfer
setting yields the clear best metrics across all four languages as it
correlates very well with human judgments. However, the commonly used binary
classifiers do not generalize across languages (due to misleadingly over-
predicting formal labels). We propose that the field use regression models
instead since they are designed to capture a wide spectrum of formality rates.

##### 2\. Meaning Preservation

We recommend using chrf as it exhibits strong correlations with human
judgments for all four languages. We caution against using bleu for this
dimension, despite its overwhelming use in prior work as both its reference
and self variants do not correlate as strongly as other more recent metrics.

##### 3\. Fluency

xlm-r is again the best metric (in particular for French). However, it does
not correlate well with human judgments as compared to the other two
dimensions.

##### 4\. System-level Ranking

chrf and xlm-r are the best metrics using a pairwise comparison evaluation.
However, an ideal evaluation would be to have a large number of systems with
which to draw reliable correlations.

##### 5\. Cross-lingual Transfer

Our results support using zero-shot transfer instead of machine translation to
port metrics from English to other languages for formality transfer tasks.

We view this work as a strong point of departure for future investigations of
st evaluation. Our work first calls for exploring how these evaluation metrics
generalize to other styles and languages. Across the different ways of
defining style evaluation (either automatic or human), prior work has mostly
focused on the three main dimensions covered in our study. As a result,
although our meta-evaluation on st metrics focuses on formality as a case
study, it can inform the evaluation of other style definitions (e.g.,
politeness, sentiment, gender, etc.). However, more empirical evidence is
needed to test the applicability of the best performing metrics for evaluating
style transfer beyond formality. Our work suggests that the top metrics based
on xlm-r and chrf are robust across $4$ Romance languages; yet, our
conclusions and recommendations are currently limited to this set of
languages. We hope that future work in multilingual style transfer will allow
for testing their generalization to a broader spectrum of languages and style
definitions. Furthermore, our study highlights that more research is needed on
automatically ranking systems. For example, one could build a metric that
combines metrics’ outputs for the three dimensions, or one could develop a
singular metric. In line with Briakou et al. (2021a), our study also calls for
releasing more human evaluations and more system outputs to enable robust
evaluation. Finally, there is still room for improvement in assessing how
fluent a rewrite is. Our study provides a framework to address these questions
systematically and calls for st papers to standardize and release data to
support larger-scale evaluations.

## Acknowledgements

We thank Sudha Rao for providing references and materials of the gyafc
dataset, Jordan Boyd-Graber, Pedro Rodriguez, the clip lab at umd, and the
emnlp reviewers for their helpful and constructive comments.

## References

  * Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016.  SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 497–511, San Diego, California. Association for Computational Linguistics.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Briakou et al. (2021a) Eleftheria Briakou, Sweta Agrawal, Ke Zhang, Joel Tetreault, and Marine Carpuat. 2021a.  A review of human evaluation for style transfer.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 58–67, Online. Association for Computational Linguistics.
  * Briakou et al. (2021b) Eleftheria Briakou, Di Lu, Ke Zhang, and Joel Tetreault. 2021b.  Olá, bonjour, salve! XFORMAL: A benchmark for multilingual formality style transfer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3199–3216, Online. Association for Computational Linguistics.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Cheng et al. (2020) Yu Cheng, Zhe Gan, Yizhe Zhang, Oussama Elachqar, Dianqi Li, and Jingjing Liu. 2020\.  Contextual text style transfer.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2915–2924, Online. Association for Computational Linguistics.
  * Clinciu et al. (2021) Miruna-Adriana Clinciu, Arash Eshghi, and Helen Hastie. 2021.  A study of automatic metrics for the evaluation of natural language explanations.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2376–2387, Online. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gong et al. (2019) Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. 2019.  Reinforcement learning based text style transfer without parallel training corpus.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3168–3180, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Goyal et al. (2020) Navita Goyal, Balaji Vasan Srinivasan, Anandhavelu Natarajan, and Abhilasha Sancheti. 2020.  Multi-dimensional style transfer for partially annotated data using language models as discriminators.  _CoRR_ , abs/2010.11578.
  * He et al. (2019) Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019.  A probabilistic formulation of unsupervised text style transfer.  In _International Conference on Learning Representations_.
  * Heafield (2011) Kenneth Heafield. 2011.  KenLM: Faster and smaller language model queries.  In _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics.
  * Heilman et al. (2014) Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreault. 2014.  Predicting grammaticality on an ordinal scale.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 174–180, Baltimore, Maryland. Association for Computational Linguistics.
  * Howcroft et al. (2020) David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4411–4421. PMLR.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2020.  Deep learning for text style transfer: A survey.  _CoRR_ , abs/2011.00416.
  * Kajiwara (2019) Tomoyuki Kajiwara. 2019.  Negative lexically constrained decoding for paraphrase generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6047–6052, Florence, Italy. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Korotkova et al. (2019) Elizaveta Korotkova, Agnes Luhtaru, Maksym Del, Krista Liin, Daiga Deksne, and Mark Fishel. 2019.  Grammatical error correction and style transfer via zero-shot monolingual translation.  _CoRR_ , abs/1903.11283.
  * Krishna et al. (2020) Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020.  Reformulating unsupervised style transfer as paraphrase generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 737–762, Online. Association for Computational Linguistics.
  * Kusner et al. (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015.  From word embeddings to document distances.  In _Proceedings of the 32nd International Conference on Machine Learning_ , volume 37 of _Proceedings of Machine Learning Research_ , pages 957–966, Lille, France. PMLR.
  * Lahiri (2015) Shibamouli Lahiri. 2015.  Squinky! A corpus of sentence-level formality, informativeness, and implicature.  _CoRR_ , abs/1506.02306.
  * Li et al. (2019) Dianqi Li, Yizhe Zhang, Zhe Gan, Yu Cheng, Chris Brockett, Bill Dolan, and Ming-Ting Sun. 2019.  Domain adaptive text style transfer.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3304–3313, Hong Kong, China. Association for Computational Linguistics.
  * Li et al. (2020) Jingjing Li, Zichao Li, Lili Mou, Xin Jiang, Michael Lyu, and Irwin King. 2020.  Unsupervised text generation by learning from search.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 10820–10831. Curran Associates, Inc.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 923–929, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Liu et al. (2021) Yixin Liu, Graham Neubig, and John Wieting. 2021.  On learning text style transfer with direct rewards.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4262–4273, Online. Association for Computational Linguistics.
  * Luo et al. (2019) Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Xu Sun, and Zhifang Sui. 2019.  A dual reinforcement learning framework for unsupervised text style transfer.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 5116–5122. International Joint Conferences on Artificial Intelligence Organization.
  * Lyu et al. (2021) Yiwei Lyu, Paul Pu Liang, Hai Pham, Eduard Hovy, Barnabás Póczos, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2021.  StylePTB: A compositional benchmark for fine-grained controllable text style transfer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2116–2138, Online. Association for Computational Linguistics.
  * Mathur et al. (2020) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020\.  Results of the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 688–725, Online. Association for Computational Linguistics.
  * Mikolov et al. (2013) Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  In _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_.
  * Mir et al. (2019) Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019.  Evaluating style transfer for text.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 495–504, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Pang (2019) Richard Yuanzhe Pang. 2019.  Towards actual (not operational) textual style transfer auto-evaluation.  In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_ , pages 444–445, Hong Kong, China. Association for Computational Linguistics.
  * Pang and Gimpel (2019) Richard Yuanzhe Pang and Kevin Gimpel. 2019.  Unsupervised evaluation metrics and learning criteria for non-parallel textual transfer.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 138–147, Hong Kong. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Pavlick and Tetreault (2016) Ellie Pavlick and Joel Tetreault. 2016.  An empirical analysis of formality in online communication.  _Transactions of the Association for Computational Linguistics_ , 4:61–74.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Belgium, Brussels. Association for Computational Linguistics.
  * Rao and Tetreault (2018) Sudha Rao and Joel Tetreault. 2018.  Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 129–140, New Orleans, Louisiana. Association for Computational Linguistics.
  * Salazar et al. (2020) Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020.  Masked language model scoring.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2699–2712, Online. Association for Computational Linguistics.
  * Sancheti et al. (2020) Abhilasha Sancheti, Kundan Krishna, Balaji Vasan Srinivasan, and Anandhavelu Natarajan. 2020.  Reinforced rewards framework for text style transfer.  In _Advances in Information Retrieval_ , pages 545–560, Cham. Springer International Publishing.
  * Shang et al. (2019) Mingyue Shang, Piji Li, Zhenxin Fu, Lidong Bing, Dongyan Zhao, Shuming Shi, and Rui Yan. 2019.  Semi-supervised text style transfer: Cross projection in latent space.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4937–4946, Hong Kong, China. Association for Computational Linguistics.
  * Tikhonov et al. (2019) Alexey Tikhonov, Viacheslav Shibaev, Aleksander Nagaev, Aigul Nugmanova, and Ivan P. Yamshchikov. 2019.  Style transfer for texts: Retrain, report errors, compare with rewrites.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3936–3945, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2019) Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and Wenhan Chao. 2019.  Harnessing pre-trained neural networks with rules for formality style transfer.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3573–3578, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2020) Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and WenHan Chao. 2020.  Formality style transfer with shared latent space.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2236–2249, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2020) Yu Wu, Yunli Wang, and Shujie Liu. 2020.  A dataset for low-resource stylized sequence-to-sequence generation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):9290–9297.
  * Xu et al. (2019) Ruochen Xu, Tao Ge, and Furu Wei. 2019.  Formality style transfer with hybrid textual annotations.  _CoRR_ , abs/1903.06353.
  * Yamshchikov et al. (2021) Ivan P. Yamshchikov, Viacheslav Shibaev, Nikolay Khlebnikov, and Alexey Tik honov. 2021.  Style-transfer and paraphrase: Looking for a sensible semantic similarity metric.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(16):14213–14220.
  * Yi et al. (2020) Xiaoyuan Yi, Zhenghao Liu, Wenhao Li, and Maosong Sun. 2020.  Text style transfer via learning style instance supported latent space.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 3801–3807. International Joint Conferences on Artificial Intelligence Organization.  Main track.
  * Zhang et al. (2020a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang et al. (2020b) Yi Zhang, Tao Ge, and Xu Sun. 2020b.  Parallel data augmentation for formality style transfer.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3221–3228, Online. Association for Computational Linguistics.
  * Zhou et al. (2020) Chulun Zhou, Liangyu Chen, Jiachen Liu, Xinyan Xiao, Jinsong Su, Sheng Guo, and Hua Wu. 2020.  Exploring contextual word-level style relevance for unsupervised style transfer.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7135–7144, Online. Association for Computational Linguistics.

##  Appendix A System-level Analysis

Table 8 presents the number of correct system-level pair-wise comparisons of
automatic metrics based on human judgments. For sts, chrf, f.reg*, f.class*,
and pseudo-lkl*, system-level scores are extracted via averaging sentence-
level scores. For s-bleu and r-bleu the system scores are extracted at the
corpus-level. The total number of pairwise comparisons for each language is
$10$ (given access to $5$ systems). Among the meaning preservation metrics
(i.e., sts, s-bleu, and chrf), chrf yields the highest number of correct
comparisons (i.e., $37$ out of $40$ for all languages). The formality
regression models (i.e., f.reg*) result in correct rankings more frequently
than the formality classifiers (i.e., f.class*) yielding $35$ out of $40$
correct comparisons. Reference-bleu (i.e., r-bleu) is compared with overall
ranking judemnts. It ranks $8$ out of $10$ systems correctly for en, fr, and
br-pt and only $6$ for it. Finally, perplexity (i.e., ppl) results in the
fewest correct rankings at system-level (i.e., $22$ out of $40$), despite
correlating well with human judgments at the segment-level.

Additionally, in Figure 2 we visualize the differences between relative
rankings induced by human judgments and the best segment-level correlated
metrics for each dimension, averaged per system.

Figure 2: Difference in relative ranking between human judgments and automatic
metrics across systems (i.e, represented by different markers) for different
evaluation dimensions. sts, s-bleu and chrf are compared with meaning
rankings, r-bleu (reference-bleu) with overall, xlm-r classifiers (*f.class)
and regression (*f.ref) models with formality, and xlm-r pseudo-perplexity
(*ppl) with fluency.

| meaning | formality | fluency | overall
---|---|---|---|---
Language | sts | s-bleu | r-bleu | chrf | f.reg* | f.class* | ppl | r-bleu | f.reg* | chrf | pseudo-ll
en | 6 | 7 | 8 | 7 | 9 | 9 | 7 | 8 | 6 | 6 | 3
br-pt | 10 | 9 | 7 | 9 | 7 | 7 | 5 | 8 | 8 | 4 | 5
fr | 9 | 7 | 10 | 10 | 10 | 9 | 5 | 8 | 6 | 3 | 7
it | 10 | 10 | 8 | 10 | 9 | 8 | 5 | 6 | 5 | 9 | 4
all | 35 | 33 | 35 | 36 | 35 | 33 | 22 | 30 | 25 | 22 | 19

Table 8: Number of correct system level pair-wise comparisons between $5$
systems for each language.

##  Appendix B Evaluated Systems Details

For each of br-pt, it, and fr, outputs are sampled from:

  1. 1.

Rule-based systems consisting of hand-crafted transformations (e.g., fixing
casing, normalizing punctuation, expanding contractions, etc.);

  2. 2.

Round-trip translation models that pivot to en and backtranslate to the
original language;

  3. 3.

Bi-directional neural machine translation (mt) models that employ side
constraints to perform style transfer for both directions of formality (i.e.,
informal$\leftrightarrow$formal)—trained on (machine) translated informal-
formal pairs of an English parallel corpus (i.e., gyafc);

  4. 4.

Bi-directional nmt models that augment the training data of 3. via
backtranslation of informal sentences;

  5. 5.

A multi-task variant of 3. that augments the training data with parallel-
sentences from bilingual resources (i.e., OpenSubtitles) and learns to
translate jointly between and across languages.

For en, the outputs were sampled from:

  1. 1.

A rule-based system of similar transformations to ones for br-pt, fr, and it;

  2. 2.

A phrase-based machine translation model trained on informal-formal pairs of
gyafc;

  3. 3.

An nmt model trained on gyafc to perform style transfer uni-directionaly;

  4. 4.

A variant of 3. that incorporates a copy-enriched mechanism that enables
direct copying of words from input;

  5. 5.

A variant of 4. trained on additional back-translated data of target style
sentences using 2.

In general, neural models performed best for all languages according to
overall human judgments, while the simpler baselines perform closer to the
more advanced neural models for br-pt, fr, and it. For each evaluation
dimension $500$ outputs are evaluated for en and $100$ outputs per system for
br-pt, fr, and it.

##  Appendix C Meaning Preservation Metrics (reference-based)

Table 9 presents supplemental results on meaning preservation metrics for
reference-based settings.

metric | en | it | pt | fr
---|---|---|---|---
r-bleu | 0.306 | 0.004 | 0.047 | 0.122
meteor | 0.279 | -0.005 | 0.061 | 0.124
chrf | 0.319 | 0.065 | 0.044 | 0.174
wmd | 0.316 | 0.039 | 0.098 | 0.198
Cosine | 0.218 | 0.027 | 0.048 | 0.161
bert-score | 0.359 | -0.023 | 0.054 | 0.112
mbert (translate-train) | 0.369 | 0.077 | 0.167 | 0.165
mbert (zero-shot) | - | 0.124 | 0.197 | 0.179
xlm-r (translate-train) | 0.385 | 0.183 | 0.136 | 0.259
xlm-r (zero-shot) | - | 0.179 | 0.153 | 0.258

Table 9: Spearman’s $\rho$ correlation of meaning preservation metrics for
reference-based meaning.