# Geometric Conditions for Twisted $O$-sphericity

Arieh Zimmerman  Department of Mathematics and Computer Science, Weizmann
Institute of Science, Rehovot, IL<EMAIL_ADDRESS>
(Date: February 2023)

###### Abstract.

The geometric condition defining a spherical variety for a reductive algebraic
group was generalized in [1], with applications to representation theory. We
twist by a character to generalize this definition, and show its equivalence
to a property of group actions that generalizes Theorem B of [1]. We also
present an example to demonstrate the necessity of this generalization.

##  1\. The Twisted Condition

Let $G$ be an algebraic group over an algebraically closed field $F$. Recall
that, for an algebraic $G$-action on a smooth variety $X$ defined over $F$, we
get a map $\nu:\mathfrak{g}\rightarrow\Gamma(X)$ to global vector fields on
$X$. This in turn gives a map $\mu:T^{*}X\rightarrow\mathfrak{g}^{*}$ defined
by $\mu(s_{(p,v)})(g)=s_{(p,v)}(\nu(g))$. It is then a classical theorem (see
1 below) that there are finitely many orbits if and only if
$\dim\mu^{-1}(0)\leq\dim X$.

Classically, there is a notion of $X$ being spherical, meaning a Borel
subgroup $B$ of $G$ has finitely many orbits on $X$. Recasting the notion with
this theorem, sphericity means that the moment map coming from the action of
all of $G$ satisfies an inequality:
$\dim\mu^{-1}(\mathfrak{b}^{\perp})\leq\dim X$, where $\mathfrak{b}$ is the
Lie algebra of $B$.

In [1], the authors generalize this to larger parabolic subgroups $P\subseteq
G$. This uses its Richardson orbit $O_{P}$, which, if it has Lie algebra
$\mathfrak{p}$ with unipotent radictal $\mathfrak{u}$, is the unique orbit
whose intersection with $\mathfrak{u}$ is open and dense.

######  Definition.

Such a $G$-variety $X$ is called $\overline{O_{P}}$-spherical if
$\dim\mu^{-1}(O\cap\mathfrak{p}^{\perp})\leq\dim X+\frac{1}{2}\dim O$ for all
$O\subseteq\overline{O_{P}}$. When $X$ is a torsor $G/H$, we call the closed
subgroup $H$ an $\overline{O_{P}}$-spherical subgroup.

Note that, in this case where $X$ is this torsor, $\mu^{-1}(O)$ is isomorphic
to $G\times_{H}(\mathfrak{h}^{\perp}\cap O)$, and the inequality can be
rewritten $\dim(O\cap\mathfrak{h}^{\perp})\leq\frac{1}{2}\dim O$.

######  Theorem A ([1], Theorem B).

$P$ has finitely many orbits on $X$ if and only if $X$ is
$\overline{O_{P}}-spherical$.

This theorem provides an accessible way to determine
$\overline{O_{P}}$-sphericity, the importance of which is demonstrated by the
main result in [1]. It continues, for the case $F=\mathbb{C}$, a string of
results on finite multiplicities resulting from sphericity, and partially
generalizes [6], where smooth functions on $G/H$ were shown to have bounded
multiplicities as a $G(\mathbb{R})$-representation if $H$ is spherical.

######  Theorem B ([1], Theorem C).

Assume $G,H$ are defined over $\mathbb{R}$, and let $H$ be
$\overline{O_{P}}$-spherical. For any finitely generated smooth admissible
Frechet representation $\pi$ of moderate growth with wavefront set contained
in $\overline{O_{P}}$, we have

$\dim\textrm{Hom}_{G(\mathbb{R})}(\mathcal{S}(G/H(\mathbb{R})),\pi)<\infty.$

Here $\mathcal{S}(G/H(\mathbb{R}))$ is the space of Schwartz functions on
$\mathbb{R}$-points of $G/H$.

The goal of this note is a generalization of A, with the hope that such a
finite multiplicity result can hold with weaker hypotheses. After
experimentation yielded an example where finite multiplicities are likely to
hold (see Example below), but $\overline{O_{P}}$-sphericity did not, the
author’s advisor took the essential characteristics of this example and
proposed the following definition. Let $H\subseteq G$ be a connected closed
subgroup of an algebraic group over $\mathbb{C}$, $P\subseteq G$ be parabolic,
and $O_{P}\subset\mathfrak{g}^{*}$ be its Richardson orbit. We will also
choose a character $\chi\in\mathfrak{h}^{*}$. We are going to pull back this
character under the restriction
$r:\mathfrak{g}^{*}\rightarrow\mathfrak{h}^{*}$.

######  Definition.

The pair $(H,\chi)$ will be called $O$-spherical if

$\dim(O\cap r^{-1}(\chi))\leq\dim O/2.$

We call it $\overline{O_{P}}$-spherical if it is $O$-spherical for every
$O\subseteq\overline{O_{P}}$.

Note that for $\chi=0$, by Lemma 2.2.2 of [1], this reduces to the
$O_{P}$-sphericity of $H$. In that case $H$ has finitely many orbits on $G/P$.
Along these lines, our goal is to extend the methods of [1] to prove the
following generalization of A:

######  Theorem 1.

Let $Y=\\{x\in G/P:\chi|_{\textrm{stab}_{\mathfrak{h}}(x)}=1\\}$. Then $H$ has
finitely many orbits on $Y$ if and only if $(H,\chi)$ is
$\overline{O_{P}}$-spherical.

Using the theorem, we found examples of potential interest that satisfy
twisted sphericity but require the twist.

######  Example.

Define a pair

$GL_{6}(\mathbb{C})=:G\supseteq H:=\left\\{\left(\begin{array}[]{ccc}B&X&Y\\\
0&B&Z\\\ 0&0&B\end{array}\right)\Bigg{|}\
B=\left(\begin{array}[]{cc}b_{1}&b_{2}\\\ 0&b_{3}\end{array}\right)\in
GL_{2}\right\\}$

acting on the almost-complete flag variety $G/P:=GL_{6}/P_{2,1,1,1,1}$. Then
put $\chi(\log h)=X+Y$ for $h\in H$ written as above (sufficiently close to
Id). Then $(H,\chi)$ is $\overline{O_{P}}$-spherical.

That this example satisfies the hypotheses of the main theorem is carried out
in [8].

##  2\. A Proof via Symplectic Manifolds

We will need some conventions. Take some $y\in Y$, and let’s say its preimage
under the quotient $G\rightarrow G/P$ is the parabolic $Q$, conjugate to $P$.
Since the algebra of the stabilizer of $Q$ in $H$ is
$\mathfrak{q}\cap\mathfrak{h}$, the definition of $Y$ means that
$\lambda\in(\mathfrak{q}\cap\mathfrak{h})^{\perp}=\mathfrak{h}^{\perp}+\mathfrak{q}^{\perp}$
for any $\lambda$ restricting to $\chi$ on $\mathfrak{h}$. When the choice of
parabolic conjugate is clear, we will accordingly write
$\lambda=\lambda^{\prime}+\lambda^{\prime\prime}$.

Next we reformulate the condition that $H$ has finitely many orbits on
$Y\subset G/P$. The $H$ orbits on $G/P$ are in bijection with the $G$ orbits
on $G/H\times G/P$ by the map $G\cdot(g_{1}H,g_{2}P)\mapsto H\cdot
g_{1}^{-1}g_{2}P$. To restrict to those orbits in the $H$-subspace $Y$ is to
consider only the $G$-subspace defined as

$G/H\times G/P\supseteq W=\\{(g_{1}H,g_{2}P)\mid\chi\mid_{\mathfrak{h}\cap
g_{1}^{-1}g_{2}\mathfrak{p}g_{2}^{-1}g_{1}}\equiv 0\\}.$

So by this translation we are dealing with the finiteness of $G$-orbits in
$W$. One advantage to using orbits under $G$ is the following generalization
of a classical proposition:

######  Lemma 1.

For a smooth $G$-space $S$, let $f:T^{*}S\rightarrow S$ be the natural
projection and $W\subseteq S$ be a locally closed $G$-subspace. Then $G$ has
finitely many orbits on $W$ if and only if $\dim
f^{-1}(W)\cap\mu_{G}^{-1}(0)\leq\dim S$.

###### Proof.

Each orbit lying in $W$ has a conormal bundle that is irreducible and
Lagrangian, hence has dimension $\frac{1}{2}\dim T^{*}S=\dim S$. The union of
these is precisely $f^{-1}(W)\cap\mu_{G}^{-1}(0)$.

If $G$ has finitely many orbits on $W$, then the union is finite, and its
dimension is also $\dim S$.

Assume instead that there are infinitely many orbits. By Proposition 1.26 in
[3], there is a stratification $W=U_{1}\sqcup\dots\sqcup U_{k}$ by
$G$-invariant subsets admitting a geometric quotient (here $U_{i+1}$ is the
set of stable points in $U_{i}$). Some $U_{i}$ has infinitely many orbits
under $G$, so the dimension of $f^{-1}(U_{i})\cap\mu_{G}^{-1}(0)$ is strictly
larger than the dimension of the conormal bundle of a single orbit. This means
$\dim f^{-1}(W)\cap\mu_{G}^{-1}(0)>\dim S$.∎

From here, one direction of our desired theorem can be accomplished with only
the use of a lemma.

######  Lemma 2.

For two surjective morphisms $f_{i}:X_{i}\rightarrow Y$ of algebraic sets with
$\dim f_{1}^{-1}(y)=\dim f_{2}^{-1}(y)$ at every $y\in Y$, we get $\dim
X_{1}=\dim X_{2}$.

###### Proof.

See e.g. Lemma 2.1 in [2].∎

This lemma is used to compare subsets of the cotangent bundle to get the
reverse direction in 1.

######  Proposition 1.

If $(H,\chi)$ is $\overline{O_{P}}$-spherical, then $H$ has finitely many
orbits on $Y$.

###### Proof.

We will soon show that $W\subseteq S=G/H\times G/P$ is constructible.
Therefore it is a union of locally closed sets, each of which is
$G$-invariant. By 1, it suffices to show $\dim
f^{-1}(W)\cap\mu_{G}^{-1}(0)\leq\dim S$. We begin by decomposing the fiber
according to coadjoint orbits.For $O\subset\overline{O_{P}}$, we can define
$T_{O}$ as

$\\{((g_{1},\xi),(g_{2},\eta))\in T^{*}G/H\times T^{*}G/P\mid
g_{1}\cdot\xi=-g_{2}\cdot\eta\in O\\}.$

The notation here regards the moment map $T^{*}G/H\rightarrow\mathfrak{g}^{*}$
equivalently as the function on $G\times_{H}\mathfrak{h}^{\perp}$ sending
$(g_{1},\xi)\mapsto g_{1}\cdot\xi$, and likewise with $P$. In this way we get
a decomposition

$\mu^{-1}(0)=\bigcup_{O\in\overline{O_{P}}}T_{O},$

and there are finitely many such $T_{O}$ since there are finitely many
nilpotent orbits [5]. So it will suffice to demonstrate from $O$-sphericity
that $\dim f^{-1}(W)\cap T_{O}\leq\dim S$.

Rather than find the dimension of $T_{O}$, we do so for a twisted version of
$T_{O}$ that we call $T_{O,\lambda}$, defined similarly as

$\\{((g_{1},\xi),(g_{2},\eta))\in T^{*}W\subseteq T^{*}G/H\times T^{*}G/P\mid
g_{1}\cdot(\xi-\lambda)=-g_{2}\cdot\eta\in O\\}.$

Note first that $f(T_{\lambda,O})=W$. This is because

$\displaystyle\chi(\mathfrak{h}\cap
g_{1}^{-1}g_{2}\mathfrak{p}g_{2}^{-1}g_{1})$
$\displaystyle=\xi(\mathfrak{h}\cap
g_{1}^{-1}g_{2}\mathfrak{p}g_{2}^{-1}g_{1})$
$\displaystyle=-g_{1}^{-1}g_{2}\cdot\eta(\mathfrak{h}\cap
g_{1}^{-1}g_{2}\mathfrak{p}g_{2}^{-1}g_{1})=\eta(\mathfrak{p})=0.$

Since $f$ is finitely presented and $T_{O,\lambda}\subset S$ is constructible,
this fulfills the promise that $W$ is constructible [7]. Our plan is to show
that the dimension of $T_{O}$ is the same as that of $T_{O,\lambda}$. Here we
use 2, applied to the projections $F:f^{-1}(W)\cap T_{O}\rightarrow W$ and
$f:T_{O,\lambda}\rightarrow W$ onto our base space $W$. To apply it, we need
only show that the fibers of a point in the image - the intersections with a
cotangent space at a fixed $(g_{1}H,g_{2}P)$ \- have equal dimensions. At this
fixed base point, we write
$\lambda=\lambda^{\prime}+\lambda^{\prime\prime}\in\mathfrak{h}^{\perp}+(g_{1}^{-1}g_{2}\mathfrak{p}g_{2}^{-1}g_{1})^{\perp}$
as in the conventions set earlier.

We claim that the addition of $\lambda$ from $\xi$ is an isomorphism of these
fibers. Indeed,$((g_{1},\xi),(g_{2},\eta))\in T_{O}$ means that
$\xi=-g_{1}^{-1}g_{2}\cdot\eta\in O$, and the space of such pairs is
$\mathfrak{h}^{\perp}\cap-
g_{1}^{-1}g_{2}\mathfrak{p}^{\perp}g_{2}^{-1}g_{1}\cap O$. Adding
$\lambda^{\prime\prime}$ to this, we get the space
$(\mathfrak{h}^{\perp}+\lambda^{\prime\prime})\cap-
g_{1}^{-1}g_{2}\mathfrak{p}^{\perp}g_{2}^{-1}g_{1}\cap(O+\lambda^{\prime\prime})$,
and this is precisely the space of $\xi-\lambda^{\prime}$ for which

$\mathfrak{h}^{\perp}\ni(\xi-\lambda^{\prime})-\lambda^{\prime\prime}=\xi-\lambda=-g_{1}^{-1}g_{2}\cdot\eta\in
O.$

So what is left is a careful counting argument to find the dimension of
$T_{O,\lambda}$. Because the action of $G$ is diagonal, projection to the
second coordinate of $T_{O,\lambda}$ is $G$-equivariant. It is also well-known
that the moment map for $G/P$ is $G$-equivariant. Thus we can find dimensions
by first fixing a point in $O$ and calculating the dimension of its fiber.

Let’s say that $\alpha\in O$ is fixed. Then the space of
$\eta\in\mathfrak{p}^{\perp}$ for which we can present $\alpha$ as some
$g_{2}\cdot\eta$ is $O\cap\mathfrak{p}^{\perp}$, since $G/P\cdot\alpha=O$. The
arbitrariness of our choice of $g_{2}\in G/P$ adds another dimension
$\dim\textrm{Stab}_{G/P}O=\dim G/P-\dim O$.

The calculation for the first coordinate is similar. We have
$O\cap(\mathfrak{h}^{\perp}+\lambda)$ ways to choose $\xi$ that allow us to
write $\xi-\lambda=g_{1}^{-1}\cdot\alpha$, and $\dim G/H-\dim O$ degrees of
freedom in stabilizing this $\xi-\lambda$. Adding this together, the dimension
of our space fibered over $O$ is

$\displaystyle\dim T_{O,\lambda}$ $\displaystyle=$ $\displaystyle\dim O+\dim
O\cap\mathfrak{p}^{\perp}$ $\displaystyle+\dim G/P-\dim O+\dim
O\cap(\mathfrak{h}^{\perp}+\lambda)+\dim G/H-\dim O$
$\displaystyle\leq\frac{1}{2}\dim O+\dim G/P+\frac{1}{2}\dim O+\dim G/H-\dim
O$ $\displaystyle=\dim G/H+\dim G/P=\dim S.$

Here we invoked $O$-sphericity, as well as the fact that $\dim
O\cap\mathfrak{p}^{\perp}$ is an isotropic subspace of $O$ [4] and so has at
most half the dimension.∎

Now we would like the analogue of the cotangent bundles used in [1]. Being
cotangent bundles as such was not as important as the existence of a
symplectic structure, and something akin to the moment map.

######  Proposition 2 ([4], Proposition 1.4.14).

For any
$\lambda\in[\mathfrak{h},\mathfrak{h}]^{\perp}\subset\mathfrak{g}^{*}$, the
space $\mathfrak{h}^{\perp}+\lambda$ is $H$-invariant. Moreover, there is a
natural $G$-invariant symplectic form $\omega$ on
$G\times_{H}\left(\mathfrak{h}^{\perp}+\lambda\right)$ given by

  1. (1)

$\omega(\alpha,\beta)=0$ for $\alpha,\beta\in\mathfrak{h}^{\perp}$ tangent to
the fibers of the projection
$G\times_{H}(\mathfrak{h}^{\perp}+\lambda)\rightarrow G/H$.

  2. (2)

$\omega(v_{x},v_{y})\mid_{\lambda}=\lambda(g[x,y]g^{-1}])$ for any point
$(g,\zeta)$ and vector fields $v$ coming from $x,y\in\mathfrak{g}$.

  3. (3)

$\omega(\alpha,v_{x})=\alpha(gxg^{-1})$ at all $\alpha\in\mathfrak{h}^{\perp}$
tangent to the fiber at $gH\times(\mathfrak{h}^{\perp}+\lambda)$.

Let us now complete the proof of 1.

######  Proposition 3.

If $H$ has finitely many orbits on $Y$, then $(H,\chi)$ is
$\overline{O_{P}}$-spherical.

###### Proof.

Let $O\subseteq\overline{O_{P}}$. In light of 2, we give a different
presentation of $T_{O,\lambda}$ to take advantage of a symplectic structure:
it will now be considered as the submanifold

$\left\\{((g_{1},\zeta),(g_{2},\eta))\in(G\times_{H}(\mathfrak{h}^{\perp}+\lambda))\times
T^{*}G/P\mid g_{1}\cdot\zeta=-g_{2}\cdot\eta\in O\right\\}.$

When we restrict the map $(g_{1},\zeta)\mapsto g_{1}\cdot\zeta$ to have its
range in $O$, the graph will be

$\Gamma_{O,\lambda}=\\{((g_{1},\zeta),g_{1}\cdot\zeta)\\}\subset(G\times_{H}(\mathfrak{h}^{\perp}+\lambda))\times
O.$

This graph has an equivariant projection to $G/H$ with fiber
$O\cap(\mathfrak{h}^{\perp}+\lambda)$ at $1$, making its dimension $\dim
O\cap(\mathfrak{h}^{\perp}+\lambda)$.

Consider the map $(I,\nu):T_{O,\lambda}\rightarrow\Gamma_{O,\lambda}$ , where
$\nu$ is the moment map $T^{*}G/P\rightarrow\mathfrak{g}^{*}$. Note that $\nu$
is $G$-equivariant, so because its image on $\nu^{-1}(O)$ is a single orbit,
it is surjective onto $O$. Any point in $\Gamma_{O,\lambda}$ can therefore be
found in the image of some point in $T_{O,\lambda}$ under $(I,\nu)$. So
$(I,\nu)$ is surjective and equivariant, hence a submersion, and moreover its
codifferentials preserve the symplectic form on
$(G\times_{H}(\mathfrak{h}^{\perp}+\lambda))\times G/P$ by Proposition 2.2.3
of [1]. So our strategy will be to show that the $G$-space $T_{O,\lambda}$ is
isotropic, for this would then prove that the graph $\Gamma_{O,\lambda}$ is
also isotropic and give the dimension formula

$\dim O\cap(\mathfrak{h}^{\perp}+\lambda)+\dim
G/H=\dim\Gamma_{O,\lambda}\leq\frac{1}{2}\dim(G\times_{H}(\mathfrak{h}^{\perp}+\lambda))\times
O\\\ =\frac{1}{2}(\dim G-\dim H+\dim\mathfrak{h}^{\perp}+\dim O)=\dim
G/H+\frac{1}{2}\dim O.$

This is $O$-sphericity.

To show that $T_{O,\lambda}$ is isotropic, define, for a $G$-orbit $C\subseteq
W$, the subspace $T_{C}=f^{-1}(C)$ (here again $f$ is projection to the base).
Since there are finitely many orbits, $T_{O,\lambda}$ would then be a finite
union of such $T_{C}$’s, and it therefore suffices to show that such a $T_{C}$
is isotropic.

Let $((g_{1},\zeta),(g_{2},\eta))\in T_{C}$. Then the tangent space to the
homogeneous space $C$ is spanned by the vectors that come from the diagonal
infinitesimal action of $G$. At a covector with basepoint $c$, if this action
yields the vector $v_{x}$, then the tangent vectors of $T_{C}$ are of the form
$(v_{x}+\alpha,v_{x}+\beta)$ with $\alpha,\beta$ vertical i.e. tangent solely
to $f^{-1}(c)$. The condition $g_{1}\cdot\zeta=-g_{2}\cdot\eta$ differentiates
with respect to $\zeta,\eta$ to give $g_{1}\cdot\alpha=-g_{2}\cdot\beta\
\forall\alpha,\beta$. Now taking the symplectic form $\omega_{1}$ of
$G\times_{H}(\mathfrak{h}^{\perp}+\lambda)$ together with the symplectic form
$\omega_{2}$ of $T^{*}G/P$ for two vectors tangent to our submanifold, we see

$\displaystyle\omega_{1}(v_{x_{1}}+\alpha_{1},v_{x_{2}}+\alpha_{2})+\omega_{2}(v_{x_{1}}+\beta_{1},v_{x_{2}}+\beta_{2})$
$\displaystyle=\zeta(g_{1}[x_{1},x_{2}]g_{1}^{-1})+\eta(g_{2}[x_{1},x_{2}]g_{2}^{-1})$
$\displaystyle+$
$\displaystyle\alpha_{2}(g_{1}x_{1}g_{1}^{-1})+\beta_{2}(g_{2}x_{2}g_{2}^{-1})-\alpha_{1}(g_{1}x_{2}g_{1}^{-1})$
$\displaystyle-\beta_{1}(g_{2}x_{2}g_{2}^{-1})$
$\displaystyle=(g_{1}\cdot\zeta)([x_{1},x_{2}])+(g_{2}\cdot\eta)([x_{1},x_{2}])$
$\displaystyle+$
$\displaystyle(g_{1}\cdot\alpha_{2})(x_{1})+(g_{2}\cdot\beta_{2})(x_{2})$
$\displaystyle-$
$\displaystyle\left((g_{1}\cdot\alpha_{1})(x_{2})+(g_{2}\cdot\beta_{1})(x_{2})\right)$
$\displaystyle=0-0+0.$

∎

## Acknowledgments

This is an abridgement of a thesis submitted for the degree of Master of
Science at the Weizmann Institute of Science. Having the good fortune to be
advised by Dima Gourevitch has been heartening and inspiring. His guidance was
uniquely well-mannered for patience, support, respect, and openness.

## References

  * [1] Avraham Aizenbud and Dmitry Gourevitch.  Finite multiplicities beyond spherical pairs.  arXiv preprint arXiv:2109.00204v3, 2021.
  * [2] Avraham Aizenbud, Dmitry Gourevitch, and Andrei Minchenko.  Holonomicity of relative characters and applications to multiplicity bounds for spherical pairs.  Selecta Mathematica, 22(4):2325–2345, 2016.
  * [3] Michel Brion.  Introduction to actions of algebraic groups.  Les cours du CIRM, 1(1):1–22, 2010.
  * [4] Neil Chriss and Victor Ginzburg.  Representation theory and complex geometry.  Modern Birkhäuser Classics. Birkhäuser Boston, Ltd., Boston, MA, 2010.  Reprint of the 1997 edition.
  * [5] David H Collingwood and William M McGovern.  Nilpotent orbits in semisimple Lie algebras.  Routledge, 2017.
  * [6] Toshiyuki Kobayashi and Toshio Oshima.  Finite multiplicity theorems for induction and restriction.  Advances in Mathematics, 248:921–944, 2013.
  * [7] David Mumford.  The red book of varieties and schemes: includes the Michigan lectures (1974) on curves and their Jacobians, volume 1358.  Springer Science & Business Media, 1999.
  * [8] Arieh Zimmerman.  Geometric conditions for twisted o-sphericity.  Master’s thesis, 2023.


# Machine Translation Evaluation with BERT Regressor

Hiroki Shimanaka†
&Tomoyuki Kajiwara†‡
†Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo,
Japan
<EMAIL_ADDRESS><EMAIL_ADDRESS>
‡Institute for Datability Science, Osaka University, Osaka, Japan
<EMAIL_ADDRESS>
&Mamoru Komachi†

###### Abstract

We introduce the metric that using BERT (Bidirectional Encoder Representations
from Transformers) Devlin et al. (2019) for automatic machine translation
evaluation. The experimental results of the WMT-2017 Metrics Shared Task
dataset show that our metric achieves a state-of-the-art performance in
segment-level metrics task for all to-English language pairs.

##  1 Introduction

This study describes a segment-level metric for automatic machine translation
evaluation (MTE). The MTE metrics with a high correlation with human
evaluation enable the continuous integration and deployment of a machine
translation (MT) system.

In our previous study Shimanaka et al. (2018), we proposed
RUSE111https://github.com/Shi-ma/RUSE (Regressor Using Sentence Embeddings)
that is a segment-level MTE metric using pre-trained sentence embeddings
capable of capturing global information that cannot be captured by local
features based on character or word N-grams. In WMT-2018 Metrics Shared Task
Ma et al. (2018), RUSE was the best metric on segment-level for all to-English
language pairs. This result indicates that pre-trained sentence embeddings are
effective feature for automatic evaluation of machine translation.

Research related to applying pre-trained language representations to
downstream tasks has been rapidly developing in recent years. In particular,
BERT (Bidirectional Encoder Representations from Transformers) Devlin et al.
(2019) has achieved the best performance in many downstream tasks and is
attracting attention. BERT is designed to pre-train using “masked language
model” (MLM) and “next sentence prediction” (NSP) on large amounts of raw text
and fine-tune for a supervised downstream task. For example, in the case of
solving single sentence classification tasks such as sentiment analysis and in
the case of solving sentence-pair classification tasks such as natural
language inference task, fine-tuning is performed in different ways. As a
result, BERT also performs well in the task of estimating the similarity
between sentence pairs, which is considered to be a similar task of automatic
machine translation evaluation.

Therefore, we propose the MTE metric that using BERT. The experimental results
in segment-level metrics task conducted using the datasets for all to-English
language pairs on WMT17 indicated that the proposed metric shows higher
correlation with human evaluations than RUSE, and achieves the best
performance. As a result of detailed analysis, it is clarified that the three
main points of difference with RUSE, the pre-training method, the sentence-
pair encoding, and the fine-tuning of the pre-trained encoder, contribute to
the performance improvement of BERT.

(a) MTE with RUSE.

(b) MTE with BERT.

Figure 1: Outline of each metric. Blue is training but red is fixed.

##  2 Related Work

In this section, we describe the MTE metric that achieves the best performance
in WMT-2017 Bojar et al. (2017) and -2018 Ma et al. (2018) Metrics Shared
Task. In this task, we use direct assessment (DA) datasets of human evaluation
data. DA datasets provides the absolute quality scores of hypotheses by
measuring to what extent a hypothesis adequately expresses the meaning of the
reference translation. Each metric estimates the quality score with the
translation and reference sentence pair as input, and is evaluated by Pearson
correlation with human evaluation. In this paper, we discuss the metrics task
in segment-level for to-English language pairs.

###  2.1 Blend: the metric based on local features

Blend which achieved the best performance in WMT-2017 is an ensemble metric
that incorporates 25 lexical metrics provided by the Asiya MT evaluation
toolkit, as well as four other metrics. Blend is a metric that uses many
features, but relies only on local information that can not simultaneously
consider the whole sentence simultaneously, such as character-based editing
distances and features based on word N-grams.

###  2.2 RUSE: the metric based on sentence embeddings

RUSE Shimanaka et al. (2018) which achieved the best performance in WMT-2018
is a metric using sentence embeddings pre-trained on large amounts of text.
Unlike previous metrics such as Blend, RUSE has the advantage of
simultaneously considering the information of the whole sentence as a
distributed representation.

ReVal222https://github.com/rohitguptacs/ReVal Gupta et al. (2015) is also a
metric using sentence embeddings. ReVal trains sentence embeddings from
labeled data in WMT Metrics Shared Task and semantic similarity estimation
tasks, but can not achieve sufficient performance because it uses only small
data. RUSE trains only regression models from labeled data using sentence
embeddings pre-trained on large data such as Quick Thought Logeswaran and Lee
(2018).

As shown in Figure 1(a), RUSE encodes an MT hypothesis and an reference
translation by a sentence encoder, respectively. Then, following InferSent
Conneau et al. (2017), a features are extracted by combining sentence
embeddings of the two sentences, and the evaluation score is estimated by the
regression model based on multi-layer perceptron (MLP).

Figure 2: BERT sentence-pair encoding.

##  3 BERT for MTE

In this study, we use BERT Devlin et al. (2019) for MTE. Like RUSE, BERT for
MTE uses pre-trained sentence embeddings and estimates the evaluation score
using the regression model based on MLP. However, as shown in the figure 1(b),
in BERT for MTE, both an MT hypothesis and an reference translation are
encoded simultaneously by the sentence-pair encoder. Then, the sentence-pair
embedding is input to the regression model based on MLP. Unlike RUSE, the pre-
trained encoder is also fine-tuning with MLP. In the following, we explain the
three differences between RUSE and BERT in detail which are the pre-training
method, the sentence-pair encoding, and the fine-tuning of the pre-trained
encoder.

###  3.1 Pre-training Method

BERT is designed to pre-train using two types of unsupervised task
simultaneously on large amounts of raw text.

#### Masked Language Model (MLM)

After replacing some tokens in the raw corpus with [MASK] tokens, we estimate
the original tokens by a bidirectional language model. By this unsupervised
pre-training, BERT encoder learns the relation between tokens in the sentence.

#### Next Sentence Prediction (NSP)

Some sentences in the raw corpus are randomly replaced with other sentences,
and then binary classification is performed to determine whether two
consecutive sentences are adjacent or not. By this unsupervised pre-training,
BERT encoder learns the relationship between two consecutive sentences.

###  3.2 Sentence-pair Encoding

In BERT, instead of encoding each sentence independently, it encodes a
sentence-pairs simultaneously for task of dealing with sentence pairs such as
NSP and Natural Language Inference. The first token of every sequence is
always a special classification token ([CLS]) and each sentence is separated
with a special end-of-sentence token ([SEP]) (Figure 2). Finally, the final
hidden state corresponding to a special [CLS] token is used as the aggregate
sequence representation for classification tasks.

###  3.3 Fine-tuning of the Pre-trained Encoder

In BERT, after obtaining a sentence embedding or a sentence-pair embedding
using an encoder, it is used as an input of MLP to solve applied tasks such as
classification and regression. When training an MLP with labeled data of the
applied task, we also fine-tune the pre-trained encoder.

| cs-en | de-en | fi-en | lv-en | ro-en | ru-en | tr-en | zh-en
---|---|---|---|---|---|---|---|---
WMT-2015 | 500 | 500 | 500 | - | - | 500 | - | -
WMT-2016 | 560 | 560 | 560 | - | 560 | 560 | 560 | -
WMT-2017 | 560 | 560 | 560 | 560 | - | 560 | 560 | 560
Table 1: Number of segment-level DA human evaluation datasets for to-English language pairs in WMT-2015 Stanojević et al. (2015), WMT-2016 Bojar et al. (2016), and WMT-2017 Metrics Shared Task Bojar et al. (2017). | cs-en | de-en | fi-en | lv-en | ru-en | tr-en | zh-en | avg.
---|---|---|---|---|---|---|---|---
SentBLEU Bojar et al. (2017) | 0.435 | 0.432 | 0.571 | 0.393 | 0.484 | 0.538 | 0.512 | 0.481
Blend Bojar et al. (2017) | 0.594 | 0.571 | 0.733 | 0.577 | 0.622 | 0.671 | 0.661 | 0.633
RUSE Shimanaka et al. (2018) | 0.614 | 0.637 | 0.756 | 0.705 | 0.680 | 0.704 | 0.677 | 0.682
BERT${}_{\text{BASE}}$ | 0.720 | 0.761 | 0.857 | 0.828 | 0.788 | 0.798 | 0.763 | 0.788
Table 2: Segment-level Pearson correlation of metric scores and DA human
evaluation scores for to-English language pairs in WMT-2017 Metrics Shared
Task.

##  4 Experiments

We performed experiments using the WMT-2017 Metrics Shared Task dataset to
verify the performance of BERT for MTE.

###  4.1 Settings

Table 1 shows the number of instances in WMT Metrics Shared Task dataset
(segment-level) for to-English language pairs333en: English, cs: Czech, de:
German, fi: Finnish, ro: Romanian, ru: Russian, tr: Turkish, lv: Latvian, zh:
Chinese used in this study. A total of 5,360 instances in WMT-2015 and
WMT-2016 Metrics Shared Task datasets will be divided randomly, and 90% is
used for training and 10% for development. A total of 3,920 instances (560
instances for each language pair) in WMT-2017 Metrics Shared Task dataset is
used for evaluation.

As a comparison method, we use SentBLEU444https://github.com/moses-
smt/mosesdecoder/blob/master/scripts/generic/mteval-v13a.pl which is the
baseline of WMT Metrics Shared Task, Blend Ma et al. (2017) which achieved the
best performance in WMT-2017 Metrics Shared Task, and RUSE Shimanaka et al.
(2018) which achieved the best performance in WMT-2018 Metrics Shared Task. We
evaluated each metric using the Pearson correlation coefficient between the
metric scores and the DA human scores.

Among the trained models published by the authors, BERT${}_{\text{BASE}}$
(uncased)555https://github.com/google-research/bert is used for MTE with BERT.
The Hyper-parameters for fine-tuning BERT are determined through grid search
in the following parameters using the development data.

  * •

$\text{Batch~{}size}\in\\{16,32\\}$
---

  * •

$\text{Learning~{}rate}(\text{Adam})\in\\{\text{5e-5},\text{3e-5},\text{2e-5}\\}$
---

  * •

$\text{Number~{}of~{}epochs}\in\\{3,4\\}$
---

  * •

$\text{Dropout~{}rate (MLP)}\in\\{0.1\\}$
---

  * •

$\text{Number~{}of~{}hidden~{}layers (MLP)}\in\\{0\\}$
---

  * •

$\text{Number~{}of~{}hidden~{}units (MLP)}\in\\{768\\}$
---

| cs-en | de-en | fi-en | lv-en | ru-en | tr-en | zh-en | avg.
---|---|---|---|---|---|---|---|---
RUSE with GloVe-BoW | 0.475 | 0.479 | 0.645 | 0.532 | 0.537 | 0.547 | 0.480 | 0.527
RUSE with Quick Thought | 0.599 | 0.588 | 0.736 | 0.690 | 0.655 | 0.710 | 0.645 | 0.660
RUSE with BERT | 0.622 | 0.626 | 0.765 | 0.708 | 0.609 | 0.706 | 0.647 | 0.669
BERT (w/o fine-tuning) | 0.645 | 0.607 | 0.780 | 0.727 | 0.644 | 0.704 | 0.705 | 0.687
BERT | 0.720 | 0.761 | 0.857 | 0.828 | 0.788 | 0.798 | 0.763 | 0.788
Table 3: Comparison of RUSE and BERT in WMT-2017 Metrics Shared Task (segment-
level, to-English language pairs).

###  4.2 Results

Table 2 presents the experimental results of the WMT-2017 Metrics Shared Task
dataset. BERT for MTE achieved the best per-formance in all to-English
language pairs. In Section 5, we compare RUSE and BERT and do a detailed
analysis.

##  5 Analysis: Comparison of RUSE and BERT

In order to analyze the three main points of difference between RUSE and BERT,
the pre-training method, the sentence-pair encoding, and the fine-tuning of
the pre-trained encoder, we conduct an experiment with the following settings.

#### RUSE with GloVe-BoW:

The mean vector of word embeddings of GloVe Pennington et al.
(2014)(glove.840B.300d666https://nlp.stanford.edu/projects/glove) (300
dimension) in each sentence is used as the sentence embeddings in Figure 1(a).

#### RUSE with Quick Thought:

Quick Thought Logeswaran and Lee (2018) pre-trained on both 45 million
sentences in the BookCorpus Zhu et al. (2015) and about 130 million sentences
in UMBC WebBase coupus Han et al. (2013) is used as the sentence encoder in
Figure 1(a).

#### RUSE with BERT:

A concatenation of the last four hidden layers (3,072 dimention) corresponding
to the [CLS] token of BERT that takes a single sentence as input is used as
the sentence embeddings in Figure 1(a).

#### BERT (w/o fine-tuning):

A concatenation of the last four hidden layers (3,072 dimension) corresponding
to the [CLS] token of BERT that takes a sentence-pair as the input sequence is
used as the input of the MLP Regressor in Figure 1(b). In this case, the part
of the BERT encoder is not fine-tuned.

#### BERT:

The last hidden layer (768 dimension) corresponding to the [CLS] token of BERT
that takes a sentence-pair as the input sequence is used as the input of the
MLP Regressor in Figure 1(b). In this case, the part of the BERT encoder is
fine-tuned.

The Hyper-parameters for RUSE and BERT (w/o fine-tuning) are determined
through grid search in the following parameters using the development data.

  * •

$\text{Batch~{}size}\in\\{64,128,256,512,1024\\}$
---

  * •

$\text{Learning~{}rate}(\text{Adam})\in\\{\text{1e-3}\\}$
---

  * •

$\text{Number~{}of~{}epochs}\in\\{1,2,...,30\\}$
---

  * •

$\text{Dropout~{}rate (MLP)}\in\\{0.1,0.3,0.5\\}$
---

  * •

$\text{Number~{}of~{}hidden~{}layers (MLP)}\in\\{1,2,3\\}$
---

  * •

$\text{Number~{}of~{}hidden~{}units (MLP)}\in\\{512,1024,2048,4096\\}$
---

Table 3 presents these experimental results of the WMT-2017 Metrics Shared
Task dataset.

#### Pre-training Method

The top three rows of Table 3 show the performance impact of the method of
pre-learning in the sentence encoder. First, Quick Thought based on sentence
embeddings has better performance consistently than GloVe-BoW based on word
embeddings. Second, BERT pret-rained by both MLM and NSP perform better on
many language pairs than Quick Thought pre-trained only by NSP. In other
words, the pre-training method using Masked Language Model (MLM), which is one
of the major features of BERT, is also useful for MTE.

#### Sentence-pair Encoding

Comparing RUSE with BERT and BERT (w/o fine-tuning) shows the impact of the
sentence-pair encoding on the performance of MTE. In the case of many language
pairs, the latter, which simultaneously encodes an MT hypothesis and a
reference translation, has higher performance than the former, which encodes
them independently. Although RUSE performs feature extraction that combines
sentence embeddings of two sentences in the same way as InferSent, this is not
necessarily the method of feature extraction suitable for MTE. On the other
hand, the sentence-pair encoding of BERT obtains sentence embeddings
considering the relation of sentence-pair without explicitly extracting the
feature. In BERT, there is a possibility that the relation of sentence-pair
can be trained well at the time of pre-training by NSP.

#### Fine-tuning of the Pre-trained Encoder

The bottom two rows of Table 3 show the performance impact of the fine-tuning
of the pre-trained encoder. In the case of all language pairs, BERT, which
fine-tune the pre-trained encoder with MLP, performs much better than RUSE,
which only trains MLP. In other words, the fine-tuning of the pre-trained
encoder, which is one of the major features of BERT, is also useful for
machine translation evaluation.

##  6 Conclusion

In this study, we proposed the metric for automatic machine translation
evaluation with BERT. Our segment-level MTE metric with BERT achieved the best
performance in segment-level metrics tasks on the WMT17 dataset for all to-
English language pairs. In addition, as a result of analysis based on
comparison with RUSE which is our previous work, it is shown that three points
of the pre-training method, the sentence-pair encoding, and the fine-tuning of
the pre-trained encoder contributed to the performance improvement of BERT
respectively.

#### Acknowledgement

Part of this research was funded by JSPS Grant-in-Aid for Scientific Research
(Grant-in-Aid for Research Activity start-up, task number: 18H06465).

## References

  * Bojar et al. (2017) Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017.  Results of the WMT17 Metrics Shared Task.  In _Proceedings of the Second Conference on Machine Translation_ , pages 489–513.
  * Bojar et al. (2016) Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016\.  Results of the WMT16 Metrics Shared Task.  In _Proceedings of the First Conference on Machine Translation_ , pages 199–231.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017.  Supervised Learning of Universal Sentence Representations from Natural Language Inference Data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 670–680.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Gupta et al. (2015) Rohit Gupta, Constantin Orasan, and Josef van Genabith. 2015.  ReVal: A Simple and Effective Machine Translation Evaluation Metric Based on Recurrent Neural Networks.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1066–1072.
  * Han et al. (2013) Lushan Han, Abhay L. Kashyap, Tim Finin, James Mayfield, and Jonathan Weese. 2013\.  UMBC_EBIQUITY-CORE: Semantic Textual Similarity Systems.  In _Second Joint Conference on Lexical and Computational Semantics, Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity_ , pages 44–52.
  * Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018.  An Efficient Framework for Learning Sentence Representations.  In _International Conference on Learning Representations_ , pages 1–16.
  * Ma et al. (2018) Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018.  Results of the WMT18 Metrics Shared Task: Both Characters and Embeddings Achieve Good Performance.  In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_ , pages 682–701.
  * Ma et al. (2017) Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017.  Blend: a Novel Combined MT Metric Based on Direct Assessment - CASICT-DCU submission to WMT17 Metrics Task.  In _Proceedings of the Second Conference on Machine Translation_ , pages 598–603.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global Vectors for Word Representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , pages 1532–1543.
  * Shimanaka et al. (2018) Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018.  RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation.  In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_ , pages 764–771.
  * Stanojević et al. (2015) Miloš Stanojević, Philipp Koehn, and Ondřej Bojar. 2015.  Results of the WMT15 Metrics Shared Task.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 256–273.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books.  _2015 IEEE International Conference on Computer Vision_ , pages 19–27.