# On Nakayama’s theorem

Osamu Fujino  Dedicated to Professor Noboru Nakayama on the occasion of his
sixtieth birthday  Department of Mathematics, Graduate School of Science,
Kyoto University, Kyoto 606-8502, Japan<EMAIL_ADDRESS>
(Date: 2021/7/16, version 0.11)

###### Abstract.

The main purpose of this paper is to make Nakayama’s theorem more accessible.
We give a proof of Nakayama’s theorem based on the negative definiteness of
intersection matrices of exceptional curves. In this paper, we treat
Nakayama’s theorem on algebraic varieties over any algebraically closed field
of arbitrary characteristic although Nakayama’s original statement is
formulated for complex analytic spaces.

###### Key words and phrases:

negativity lemma, exceptional divisors, reflexive sheaves

###### 2010 Mathematics Subject Classification:

Primary 14C20; Secondary 14E30

##  1\. Introduction

In this paper, a variety means an integral separated scheme of finite type
over an algebraically closed field $k$ of any characteristic. The following
theorem is very well known and plays a crucial role in the theory of higher-
dimensional minimal models.

######  Theorem 1.1.

Let $f\colon X\to Y$ be a projective birational morphism from a smooth surface
$X$ to a normal surface $Y$. Then the intersection matrix of the
$f$-exceptional curves is negative definite.

The main purpose of this paper is to make the following theorem by Noboru
Nakayama more accessible. Here we treat varieties over any algebraically
closed field $k$ of arbitrary characteristic although the original statement
is formulated for complex analytic spaces.

######  Theorem 1.2 (Nakayama’s theorem, see [N, Chapter III, 5.10. Lemma
(3)]).

Let $f\colon X\to Y$ be a projective surjective morphism from a smooth variety
$X$ onto a normal variety $Y$. Let $D$ be an $\mathbb{R}$-divisor on $X$. Then
there exists an effective $f$-exceptional divisor $E$ on $X$ such that

$\left(f_{*}\mathcal{O}_{X}(\lfloor
tD\rfloor)\right)^{**}=f_{*}\mathcal{O}_{X}(\lfloor t(D+E)\rfloor)$

holds for every positive real number $t$.

Theorem 1.2 has already played a fundamental role in [T], [PT], [CP], and so
on. Nakayama’s original proof in [N] uses his theory of relative
$\sigma$-decompositions and relative $\nu$-decompositions developed in [N,
Chapter III, §1, §3, and §4]. Hence we had to study $N_{\sigma}$ and $N_{\nu}$
to understand Theorem 1.2. Our argument in this paper clarifies that Theorem
1.2 is an easy consequence of Theorem 1.1. Roughly speaking, Theorem 1.2 is a
variant of the negativity lemma (see, for example, [F, Lemma 2.3.26]). We do
not need $N_{\sigma}$ and $N_{\nu}$.

######  Acknowledgments.

The author was partially supported by JSPS KAKENHI Grant Numbers JP16H03925,
JP16H06337.

##  2\. Preliminaries

Let us start with the definition of exceptional divisors.

######  Definition 2.1 (Exceptional divisors).

Let $f\colon X\to Y$ be a proper surjective morphism between normal varieties.
Let $E$ be a Weil divisor on $X$. We say that $E$ is $f$-exceptional if
$\operatorname{codim}_{Y}\\!f(\operatorname{Supp}E)\geq 2$. We note that $f$
is not always assumed to be birational.

In order to understand Theorem 1.2, we need the following definitions.

######  Definition 2.2.

Let $D=\sum_{i}a_{i}D_{i}$ be an $\mathbb{R}$-divisor on a normal variety $X$
such that $D_{i}$ is a prime divisor on $X$ for every $i$ and that $D_{i}\neq
D_{j}$ for $i\neq j$. We put

$D^{+}=\sum_{a_{i}>0}a_{i}D_{i}\quad\text{and}\quad
D^{-}=-\sum_{a_{i}<0}a_{i}D_{i}\geq 0.$

Note that

$D=D^{+}-D^{-}$

obviously holds. For every real number $x$, $\lfloor x\rfloor$ is the integer
defined by $x-1<\lfloor x\rfloor\leq x$. We put

$\lfloor D\rfloor=\sum_{i}\lfloor a_{i}\rfloor D_{i}$

and call it the round-down of $D$.

######  Definition 2.3.

Let $\mathcal{F}$ be a coherent sheaf on a normal variety $X$. We put

$\mathcal{F}^{*}=\mathcal{H}om_{\mathcal{O}_{X}}(\mathcal{F},\mathcal{O}_{X})$

and

$\mathcal{F}^{**}=(\mathcal{F}^{*})^{*}.$

Then there exists a natural map $\mathcal{F}\to\mathcal{F}^{**}$. If this map
$\mathcal{F}\to\mathcal{F}^{**}$ is an isomorphism, then $\mathcal{F}$ is
called a reflexive sheaf.

For the basic properties of reflexive sheaves, see [H, Section 1]. We prepare
an easy lemma for the reader’s convenience.

######  Lemma 2.4.

Let $V$ be a smooth surface and let $C_{1},\ldots,C_{m}$ be effective Cartier
divisors on $V$ such that the intersection matrix $\left(C_{i}\cdot
C_{j}\right)$ is negative definite and that $C_{i}\cdot C_{j}\geq 0$ for
$i\neq j$. Let

$D=B+\sum_{i=1}^{m}a_{i}C_{i}$

be an $\mathbb{R}$-divisor on $V$. Assume that

  * (a)

$D\cdot C_{i}\leq 0$ (resp. $D\cdot C_{i}<0$) for every $i$, and

  * (b)

$B\cdot C_{i}\geq 0$ for every $i$.

Then $a_{i}\geq 0$ (resp. $a_{i}>0$) for every $i$.

######  Remark 2.5.

In Lemma 2.4, $C_{i}$ may be reducible and disconnected. It may happen that
$C_{i}$ and $C_{j}$ have some common irreducible components for $i\neq j$.

###### Proof of Lemma 2.4.

By (b), $B\cdot C_{j}\geq 0$ for every $j$. Hence
$\left(\sum_{i=1}^{m}a_{i}C_{i}\right)\cdot C_{j}\leq 0$ (resp. $<0$) for
every $j$ by (a). Since $\left(C_{i}\cdot C_{j}\right)$ is negative definite
and $C_{i}\cdot C_{j}\geq 0$ for $i\neq j$, $a_{i}\geq 0$ (resp. $>0$) holds
for every $i$. ∎

##  3\. Proof of Theorem 1.2

In this section, we prove Theorem 1.2.

######  Definition 3.1.

Let $f\colon X\to Y$ be a projective surjective morphism from a smooth
$n$-dimensional quasi-projective variety $X$ onto a normal quasi-projective
variety $Y$. Let $H$ be a very ample Cartier divisor on $Y$ and let $A$ be a
very ample Cartier divisor on $X$. Let $E$ be an $f$-exceptional prime divisor
on $X$ with $\dim f(E)=e$. We put

$C:=E\cap f^{*}H_{1}\cap\cdots\cap f^{*}H_{e}\cap A_{1}\cap\cdots\cap
A_{n-e-2},$

where $H_{i}$ is a general member of $|H|$ for every $i$ and $A_{j}$ is a
general member of $|A|$ for every $j$, and call $C$ a general curve associated
to $E$, $f\colon X\to Y$, $H$, and $A$. We sometimes simply say that $C$ is a
general curve associated to $E$. By construction, $E\cdot C<0$ and $P\cdot
C\geq 0$ for every prime divisor $P$ on $X$ with $P\neq E$. We note that $C$
may be reducible and disconnected. We also note that

$f^{*}H_{1}\cap\cdots\cap f^{*}H_{e}\cap A_{1}\cap\cdots\cap A_{n-e-2}$

is a smooth surface when the characteristic of the base field $k$ is zero by
Bertini’s theorem. Unfortunately, however, it may be singular in general.

For the proof of Theorem 1.2, we prepare several lemmas, which are easy
applications of Theorem 1.1.

######  Lemma 3.2.

Let $f\colon X\to Y$, $H$, $A$ be as in Definition 3.1. Let
$E_{1}\ldots,E_{m}$ be $f$-exceptional prime divisors on $X$ such that
$E_{i}\neq E_{j}$ for $i\neq j$ and $\dim f(E_{i})=e$ for every $i$. Let
$C_{i}$ be a general curve associated to $E_{i}$, $f\colon X\to Y$, $H$, and
$A$ for every $i$. Then the matrix $\left(E_{i}\cdot C_{j}\right)$ is negative
definite and that $E_{i}\cdot C_{j}\geq 0$ for $i\neq j$. Hence there exists
an effective divisor $E$ on $X$ such that
$\operatorname{Supp}E=\sum_{i=1}^{m}E_{i}$ and that $E\cdot C_{i}<0$ for every
$i$.

###### Proof.

We use the same notation as in Definition 3.1. By restricting everything to

$f^{*}H_{1}\cap\cdots\cap f^{*}H_{e}\cap A_{1}\cap\cdots\cap A_{n-e-2},$

we may assume that $X$ is a pure two-dimensional quasi-projective scheme over
$k$, $E_{i}=C_{i}$ for every $i$, and $E_{i}\cdot C_{j}\geq 0$ for $i\neq j$
(see Definition 3.1). We first treat the case where the characteristic of the
base field $k$ is zero. In this case, $X$ is a smooth surface by Bertini’s
theorem. Then it is easy to see that $\left(E_{i}\cdot C_{j}\right)$ is
negative definite (see Theorem 1.1). When the characteristic of the base field
$k$ is positive, $X$ may be singular. In this case, by pulling everything back
to a desingularization of

$\left(f^{*}H_{1}\cap\cdots\cap f^{*}H_{e}\cap A_{1}\cap\cdots\cap
A_{n-e-2}\right)_{\mathrm{red}},$

we can check the negative definiteness of $\left(E_{i}\cdot C_{j}\right)$ (see
Theorem 1.1). Hence we can always take an effective divisor $E$ with the
desired properties (see Lemma 2.4). ∎

######  Lemma 3.3.

Let $f\colon X\to Y$, $H$, and $A$ be as in Definition 3.1. Let
$E_{1},\ldots,E_{m}$ be $f$-exceptional prime divisors on $X$ such that
$E_{i}\neq E_{j}$ for $i\neq j$. Let $C_{i}$ be a general curve associated to
$E_{i}$, $f\colon X\to Y$, $H$, and $A$ for every $i$. Then there is an
effective divisor $E$ on $X$ such that
$\operatorname{Supp}E=\sum_{i=1}^{m}E_{i}$ and that $E\cdot C_{i}<0$ for every
$i$.

###### Proof.

By Lemma 3.2, we can find an effective divisor $E^{j}$ on $X$ such that

$\operatorname{Supp}E^{j}=\sum_{\dim f(E_{i})=j}E_{i}$

and that $E^{j}\cdot C_{i}<0$ where $C_{i}$ is a general curve associated to
$E_{i}$ with $\dim f(E_{i})=j$. We note that $E^{j}\cdot C_{i}=0$ (resp. $\geq
0$) when $C_{i}$ is a general curve associated to $E_{i}$ with $\dim
f(E_{i})>j$ (resp. $<j$) by construction. We put

$E=\sum_{j=0}^{n-2}m_{j}E^{j}$

with

$m_{0}\gg m_{1}\gg\cdots\gg m_{n-2}>0.$

Then $E$ is an effective divisor on $X$ with the desired properties. ∎

######  Lemma 3.4.

Let $f\colon X\to Y$, $H$, and $A$ be as in Definition 3.1. Let $D$ be an
$\mathbb{R}$-divisor on $X$ such that $\operatorname{Supp}D^{-}$ is
$f$-exceptional and that $D\cdot C\leq 0$ for any general curve $C$ associated
to any $f$-exceptional divisor on $X$. Then $D$ is effective.

###### Proof.

Let $\mathcal{E}=\\{E_{1},\ldots,E_{m}\\}$ be the set of all $f$-exceptional
divisors on $X$. By pulling everything back to a desingularization $V$ of

$\left(f^{*}H_{1}\cap\cdots\cap f^{*}H_{n-2}\right)_{\mathrm{red}}$

and using Lemma 2.4 on $V$, we obtain that the pull-back of $D$ to $V$ is
effective. This means that the coefficient of $E_{i}$ in $D$ is nonnegative
when $\dim f(E_{i})=n-2$. Assume that the coefficient of $E_{i}$ in $D$ is
nonnegative when $\dim f(E_{i})\geq e+1$. Then we pull everything back to a
desingularization of

$\left(f^{*}H_{1}\cap\cdots\cap f^{*}H_{e}\cap A_{1}\cap\cdots\cap
A_{n-e-2}\right)_{\mathrm{red}}$

and use Lemma 2.4 again. Then we obtain that the coefficient of $E_{i}$ in $D$
is nonnegative when $\dim f(E_{i})\geq e$. We repeat this process finitely
many times. Then we finally obtain that $D$ is effective. ∎

######  Lemma 3.5.

Let $f\colon X\to Y$ be a projective surjective morphism from a smooth variety
$X$ onto a normal variety $Y$. Let $D$ be an $\mathbb{R}$-divisor on $X$. Then
there exists an effective $f$-exceptional divisor $E$ on $X$ such that if $G$
is any $\mathbb{R}$-divisor on $X$ and $U$ is any Zariski open subset of $Y$
with the following properties:

  * (a)

$G|_{f^{-1}(U)}\equiv_{U}D|_{f^{-1}(U)}$, that is, $G|_{f^{-1}(U)}$ is
relatively numerically equivalent to $D|_{f^{-1}(U)}$ over $U$, and

  * (b)

the support of $G^{-}|_{f^{-1}(U)}$ is $f$-exceptional,

then $(G+E)|_{f^{-1}(U)}$ is effective, equivalently, $G^{-}|_{f^{-1}(U)}\leq
E|_{f^{-1}(U)}$ holds.

###### Proof.

In Step 1, we will treat the case where $Y$ is affine. In Step 2, we will
treat the general case.

######  Step 1.

In this step, we will construct a desired divisor $E$ under the extra
assumption that $Y$ is affine.

When $Y$ is affine, we can take an effective $f$-exceptional divisor $E$ on
$X$ such that $E\cdot C<0$ for any general curve associated to any
$f$-exceptional divisor on $X$ by Lemma 3.3. By replacing $E$ with $mE$ for
some positive integer $m$, we may further assume that $(D+E)\cdot C\leq 0$ for
any general curve $C$ associated to any $f$-exceptional divisor on $X$. Let
$U$ be any Zariski open subset of $Y$ and let $G$ be any $\mathbb{R}$-divisor
on $X$ satisfying (a) and (b). Then,

$(G+E)|_{f^{-1}(U)}\cdot C=(D+E)|_{f^{-1}(U)}\cdot C\leq 0$

holds for any general curve $C$ associated to any $f$-exceptional divisor on
$X$ by (a) and the construction of $E$, and the support of
$(G+E)^{-}|_{f^{-1}(U)}$ is $f$-exceptional by (b). Hence, by Lemma 3.4,
$(G+E)|_{f^{-1}(U)}$ is effective.

######  Step 2.

In this step, we will treat the general case.

We take a finite affine Zariski open cover

$Y=\bigcup_{\alpha}U_{\alpha}$

of $Y$. We consider $f\colon f^{-1}(U_{\alpha})\to U_{\alpha}$ for every
$\alpha$. By Step 1, we have an effective $f$-exceptional divisor $E_{\alpha}$
on $f^{-1}(U_{\alpha})$ with the desired property for every $\alpha$. Let $E$
be an effective $f$-exceptional divisor on $X$ such that $E_{\alpha}\leq
E|_{f^{-1}(U_{\alpha})}$ holds for every $\alpha$. Then $E$ obviously
satisfies the desired property.

We complete the proof. ∎

Let us prove Theorem 1.2.

###### Proof of Theorem 1.2.

We prove this theorem in the following two steps.

######  Step 1.

Let $E$ be any $f$-exceptional divisor on $X$. Then

$\left(f_{*}\mathcal{O}_{X}(\lfloor
tD\rfloor)\right)^{**}=\left(f_{*}\mathcal{O}_{X}(\lfloor
t(D+E)\rfloor)\right)^{**}$

holds. Therefore, the inclusion

$f_{*}\mathcal{O}_{X}(\lfloor
t(D+E)\rfloor)\subset\left(f_{*}\mathcal{O}_{X}(\lfloor
tD\rfloor)\right)^{**}$

always holds.

######  Step 2.

Let $E$ be an effective $f$-exceptional divisor on $X$ satisfying the property
in Lemma 3.5. Let $\Sigma$ denote the smallest Zariski closed subset of $Y$
such that $f$ is equidimensional over $Y\setminus\Sigma$. Note that
$\operatorname{codim}_{Y}\\!\Sigma\geq 2$. By [H, Corollary 1.7],
$f_{*}\mathcal{O}_{X}(\lfloor tD\rfloor)|_{Y\setminus\Sigma}$ is a reflexive
sheaf on $Y\setminus\Sigma$. Let $\mathcal{E}=\\{E_{1},\ldots,E_{m}\\}$ be the
set of all $f$-exceptional divisors on $X$. Hence
$\operatorname{codim}_{X}\left(f^{-1}(\Sigma)\setminus\sum_{i=1}^{m}E_{i}\right)\geq
2$ holds by definition. We take any affine Zariski open subset $U$ of $Y$.
Then we have the following natural inclusion and equalities

(3.1) $\begin{split}\Gamma\left(U,\left(f_{*}\mathcal{O}_{X}(\lfloor
tD\rfloor)\right)^{**}\right)&\subset\Gamma\left(U\setminus\Sigma,\left(f_{*}\mathcal{O}_{X}(\lfloor
tD\rfloor)\right)^{**}\right)\\\
&=\Gamma\left(U\setminus\Sigma,f_{*}\mathcal{O}_{X}(\lfloor
tD\rfloor)\right)\\\ &=\Gamma\left(f^{-1}(U)\setminus
f^{-1}(\Sigma),\mathcal{O}_{X}(\lfloor tD\rfloor)\right)\\\
&=\Gamma\left(f^{-1}(U)\setminus\sum_{i=1}^{m}E_{i},\mathcal{O}_{X}(\lfloor
tD\rfloor)\right),\end{split}$

where the inclusion follows from the torsion-freeness of
$\left(f_{*}\mathcal{O}_{X}(\lfloor tD\rfloor)\right)^{**}$, the first
equality is due to the fact that $f_{*}\mathcal{O}_{X}(\lfloor
tD\rfloor)|_{Y\setminus\Sigma}$ is a reflexive sheaf on $Y\setminus\Sigma$,
the second equality is obvious, and the final equality follows from
$\operatorname{codim}_{X}\left(f^{-1}(\Sigma)\setminus\sum_{i=1}^{m}E_{i}\right)\geq
2$. We note that

$\begin{split}&\Gamma\left(f^{-1}(U)\setminus\sum_{i=1}^{m}E_{i},\mathcal{O}_{X}(\lfloor
tD\rfloor)\right)=\\{\phi\in k(X)\,|\,\left((\phi)+\lfloor
tD\rfloor\right)|_{f^{-1}(U)\setminus\sum E_{i}}\geq
0\\}\cup\\{0\\},\end{split}$

where $k(X)$ stands for the rational function field of $X$ and $(\phi)$ is the
divisor associated to $\phi\in k(X)$. By the definition of $E$, if

$\left((\phi)+\lfloor tD\rfloor\right)|_{f^{-1}(U)\setminus\sum E_{i}}\geq 0$

holds, then

$\left((\phi)+t(D+E)\right)|_{f^{-1}(U)}\geq 0$

holds. Therefore, by taking the round-down, we obtain that

$\left((\phi)+\lfloor t(D+E)\rfloor\right)|_{f^{-1}(U)}\geq 0.$

This implies that

(3.2)
$\begin{split}&\Gamma\left(f^{-1}(U)\setminus\sum_{i=1}^{m}E_{i},\mathcal{O}_{X}(\lfloor
tD\rfloor)\right)\\\ &\subset\\{\phi\in k(X)\,|\,\left((\phi)+\lfloor
t(D+E)\rfloor\right)|_{f^{-1}(U)}\geq 0\\}\cup\\{0\\}\\\
&=\Gamma\left(f^{-1}(U),\mathcal{O}_{X}(\lfloor t(D+E)\rfloor)\right)\\\
&=\Gamma\left(U,f_{*}\mathcal{O}_{X}(\lfloor
t(D+E)\rfloor)\right).\end{split}$

Hence we get the following inclusion

$\Gamma\left(U,\left(f_{*}\mathcal{O}_{X}(\lfloor
tD\rfloor)\right)^{**}\right)\subset\Gamma\left(U,f_{*}\mathcal{O}_{X}(\lfloor
t(D+E)\rfloor)\right)$

by (3.1) and (3.2). This means that the opposite inclusion

$\left(f_{*}\mathcal{O}_{X}(\lfloor tD\rfloor)\right)^{**}\subset
f_{*}\mathcal{O}_{X}(\lfloor t(D+E)\rfloor)$

holds.

By combining Step 1 with Step 2, we see that the effective $f$-exceptional
divisor $E$ on $X$ with the property in Lemma 3.5 is a desired one. ∎

Finally, we note the following statement, which is similar to [N, Chapter III,
5.10. Lemma (4)].

######  Proposition 3.6.

Let $f\colon X\to Y$ be a projective surjective morphism from a smooth quasi-
projective variety $X$ onto a normal quasi-projective variety $Y$. Let $D$ be
an $\mathbb{R}$-divisor on $X$. Assume that $D\cdot C\leq 0$ for any general
curve $C$ associated to any $f$-exceptional divisor on $X$. Then
$f_{*}\mathcal{O}_{X}(\lfloor D\rfloor)$ is reflexive.

###### Proof.

If we further assume that $Y$ is quasi-projective and $D\cdot C\leq 0$ for any
general curve $C$ associated to any $f$-exceptional divisor on $X$ in Lemma
3.5, then we see that $G|_{f^{-1}(U)}$ is effective by Lemma 3.4 and the proof
of Lemma 3.5. Hence, the argument in Step 2 in the proof of Theorem 1.2 works
with $E=0$ and $t=1$. Therefore, we obtain

$f_{*}\mathcal{O}_{X}(\lfloor D\rfloor)=\left(f_{*}\mathcal{O}_{X}(\lfloor
D\rfloor)\right)^{**}.$

This means that $f_{*}\mathcal{O}_{X}(\lfloor D\rfloor)$ is reflexive. ∎

## References

  * [CP] F. Campana, M. Păun, Foliations with positive slopes and birational stability of orbifold cotangent bundles, Publ. Math. Inst. Hautes Études Sci. 129 (2019), 1–49.
  * [F] O. Fujino, Foundations of the minimal model program, MSJ Memoirs, 35. Mathematical Society of Japan, Tokyo, 2017.
  * [H] R. Hartshorne, Stable reflexive sheaves, Math. Ann. 254 (1980), no. 2, 121–176.
  * [N] N. Nakayama, Zariski-decomposition and abundance, MSJ Memoirs, 14. Mathematical Society of Japan, Tokyo, 2004.
  * [PT] M. Păun, S. Takayama, Positivity of twisted relative pluricanonical bundles and their direct images, J. Algebraic Geom. 27 (2018), no. 2, 211–272.
  * [T] S. Takayama, A local ampleness criterion of torsion free sheaves, Bull. Sci. Math. 137 (2013), no. 5, 659–670.


# The gradient flow at higher orders in perturbation theory

Robert Harlander

###### Abstract

Various results for higher-order perturbative calculations in the gradient-
flow formalism are reviewed, including the gradient-flow beta function and the
small-flow-time expansion of the hadronic vacuum polarization and the energy-
momentum tensor. In addition, the strategy of regions is outlined in order to
obtain systematic expansions of gradient-flow integrals, for example at large
and small flow times.

##  1 Introduction

Quantum Chromodynamics (QCD) is a theory with many different facets. So far,
quantitative phenomenological results have been obtained mostly in either the
strong-coupling regime using lattice regularizations or in the weak-coupling
regime where perturbation theory is applicable. Both calculational approaches
are highly evolved in themselves. Cross-fertilization is often hindered by the
inherently different treatment of ultraviolet divergences in these two
calculational approaches.

The gradient-flow formalism (GFF) may provide an excellent opportunity to
change this situation. It represents a UV regularization scheme which can be
implemented both on the lattice and in perturbation theory. This contribution
reviews a number of concrete examples where such a cross-fertilization could
be achieved, and where the perturbative calculations have been performed
beyond next-to-leading order in perturbation theory. Furthermore, the
application of the strategy of regions to gradient-flow integrals is
presented, which allows to obtain systematic expansions in dimensionless
parameters.

##  2 The perturbative gradient flow

In the GFF, one defines flowed fields $B^{a}_{\mu}=B^{a}_{\mu}(t)$ and
$\chi=\chi(t)$ as solutions of the equations [1, 2] (see also [3, 4])

$\begin{split}\partial_{t}B^{a}_{\mu}&=\mathcal{D}^{ab}_{\nu}G^{b}_{\nu\mu}+\kappa\mathcal{D}^{ab}_{\mu}\partial_{\nu}B^{b}_{\nu}\,,\\\
\partial_{t}\chi&=\Delta\chi-\kappa\partial_{\mu}B^{a}_{\mu}T^{a}\chi\,,\\\
\partial_{t}\bar{\chi}&=\bar{\chi}\overleftarrow{\Delta}+\kappa\bar{\chi}\partial_{\mu}B^{a}_{\mu}T^{a}\,.\end{split}$
(1)

The initial conditions supplementing these differential equations establish
the contact to regular QCD:

$\begin{split}B^{a}_{\mu}(t=0)=A^{a}_{\mu}\,,\qquad\chi(t=0)=\psi\,,\end{split}$
(2)

where $A^{a}_{\mu}$ and $\psi$ are the regular gluon and quark fields,
respectively, and

$\begin{split}\mathcal{D}^{ab}_{\mu}&=\delta^{ab}\partial_{\mu}-f^{abc}B_{\mu}^{c}\,,\qquad\Delta=(\partial_{\mu}+B^{a}_{\mu}T^{a})(\partial_{\mu}+B^{b}_{\mu}T^{b})\,,\\\
G_{\mu\nu}^{a}&=\partial_{\mu}B_{\nu}^{a}-\partial_{\nu}B_{\mu}^{a}+f^{abc}B_{\mu}^{b}B_{\nu}^{c}\,.\end{split}$
(3)

The arbitrary parameter $\kappa$ will be set equal to one in the following.

Our practical implementation of the GFF in perturbation theory follows the
strategy developed in Ref. [5]. It leads to Feynman rules resembling those of
regular QCD, but supplemented by flow-time dependent exponentials in the
propagators. In addition, the flow equations are reflected through so-called
flow lines which couple to the flowed quarks and gluons via flowed vertices.
The latter involve integrations over finite intervals of the flow-time
variables.

A systematic method how to handle the corresponding Feynman diagrams and
integrals through three-loop level has been introduced in Ref. [6]. It is
based on qgraf [7] for the generation of Feynman diagrams, FORM [8, 9, 10] for
the algebraic manipulation of the resulting amplitudes, Kira+FireFly [11, 12,
13, 14, 15] for the reduction of the Feynman integrals to master integrals,
and q2e/exp [16] for interfacing all of these programs. The master integrals
can be calculated by following the method outline in Ref. [17], for example.

##  3 Gluon condensate, quark condensates, and gradient-flow beta function

###  3.1 Gluon condensate and gradient-flow beta function

The first quantity considered at finite flow time was the gluon condensate in
massless QCD [1]. Its perturbative expansion reads

$\begin{split}&\langle
G_{\mu\nu}^{a}(t)G_{\mu\nu}^{a}(t)\rangle=3\frac{\alpha_{s}(\mu)}{\pi
t^{2}}\bigg{[}1+\frac{\alpha_{s}(\mu)}{4\pi}\left(e_{10}+4\beta_{0}l_{\mu
t}\right)\\\
&+\left(\frac{\alpha_{s}(\mu)}{4\pi}\right)^{2}\left(e_{20}+8\left(e_{10}\,\beta_{0}+2\,\beta_{1}\right)l_{\mu
t}+16\,\beta_{0}^{2}\,l_{\mu t}^{2}\right)\bigg{]}+\cdots\quad\equiv\quad
3\frac{\bar{\alpha}_{s}(t)}{\pi t^{2}}\,,\end{split}$ (4)

where $l_{\mu t}\equiv\ln 2\mu^{2}t+\gamma_{\text{E}}$ with Euler’s constant
$\gamma_{\text{E}}=0.577\ldots$, and $\alpha_{s}(\mu)$ is the strong coupling
in the $\overline{\text{{MS}}}$ scheme which obeys

$\begin{split}\mu^{2}\frac{\text{d}}{\text{d}\mu^{2}}\frac{\alpha_{s}(\mu)}{\pi}=\beta(\alpha_{s}(\mu))\,,\quad\text{with}\quad\beta(\alpha_{s})=-\sum_{n=0}^{\infty}\left(\frac{\alpha_{s}}{\pi}\right)^{2+n}\beta_{n}\,.\end{split}$
(5)

In QCD, the first three coefficients of the $\beta$ function in the
$\overline{\text{{MS}}}$ scheme read

$\begin{split}\beta_{0}&=\frac{11}{4}-\frac{n_{f}}{6}\,,\qquad\beta_{1}=\frac{51}{8}-\frac{19}{24}\,n_{f}\,,\\\
\beta_{2}&=\frac{2857}{128}-\frac{5033}{1152}\,n_{f}+\frac{325}{3456}\,n_{f}^{2}\approx
22.3-4.37\,n_{f}+0.094\,n_{f}^{2}\,.\end{split}$ (6)

Setting $t$ to some multiple of $\mu^{2}$, i.e. $t=\mu^{2}/\rho$, acting with
$\mu^{2}\text{d}/\text{d}\mu^{2}$ on Eq. 4, and iteratively replacing
$\alpha_{s}$ by $\bar{\alpha}_{s}$ according to Eq. 4, one finds the evolution
of the gradient-flow coupling:

$\begin{split}\mu^{2}\frac{\text{d}}{\text{d}\mu^{2}}\frac{\bar{\alpha}_{s}(\mu)}{\pi}=\bar{\beta}(\bar{\alpha}_{s}(\mu))\,,\quad\text{with}\quad\bar{\beta}(\bar{\alpha}_{s})=-\sum_{n=0}^{\infty}\left(\frac{\bar{\alpha}_{s}}{\pi}\right)^{2+n}\bar{\beta}_{n}\,.\end{split}$
(7)

The first two coefficients are universal, i.e. $\bar{\beta}_{0}=\beta_{0}$ and
$\bar{\beta}_{1}=\beta_{1}$, while

$\begin{split}\bar{\beta}_{2}=\beta_{2}-\frac{1}{4}e_{10}\beta_{1}+\frac{1}{16}\left(e_{20}-e_{10}^{2}\right)\beta_{0}=-59.1-0.536\,n_{f}+0.304\,n_{f}^{2}-0.0030\,n_{f}^{3}\,.\end{split}$
(8)

Note that the $\rho$-dependence drops out of the $\bar{\beta}$ function. The
difference to the $\overline{\text{{MS}}}$ value of Eq. 6 is remarkable. It is
illustrated in dependence of $n_{f}$ in Fig. 1 (a). The impact on the QCD
$\beta$ function is shown in Fig. 1 (b). One immediately notices that the
perturbative convergence is significantly worse in the gradient-flow scheme
than in the $\overline{\text{{MS}}}$ scheme (see also Refs. [18, 19, 20, 21]).
It would be interesting to understand the source of this behavior in order to
allow for a precise independent lattice determination of $\alpha_{s}(M_{Z})$
through the GFF.

|
---|---
(a) | (b)

Figure 1: (a) Three-loop coefficient of the $\beta$ function in the $\overline{\text{{MS}}}$ (solid-red) and the gradient-flow scheme (dashed-blue). (b) $\beta$ function in the $\overline{\text{{MS}}}$ scheme and the gradient-flow scheme for $n_{f}=0$ ($\mathrm{a}_{s}\equiv\alpha_{s}/\pi$).  |
---|---
(a) | (b)

Figure 2: Leading-order contributions for the quark mass effects to the gluon
and the quark condensate. Diagrams produced with FeynGame [22].

###  3.2 Quark mass effects

So far, we have considered the case of massless quarks. For the gluon
condensate, quark mass effects occur only at next-to-leading order (NLO)
through the single Feynman diagram shown in Fig. 2 (a) . They can be taken
into account quite easily by using the well-known one-loop expression for the
two-point function with external gluons. The result has been expressed in
terms of a one-dimensional integral [17]. At higher orders in perturbation
theory, approximate results of the mass effects could be obtained using the
so-called strategy of regions [23]. To illustrate its application in the GFF,
let us consider the simpler case of the quark condensate, where mass effects
occur already at leading order (LO), see Fig. 2 (b). The exact mass dependence
leads to an incomplete $\Gamma$ function in this case [2]:

$\begin{split}S(t)\equiv\langle\bar{\chi}(t)\chi(t)\rangle=-\frac{3m}{8\pi^{2}t^{2}}f(m^{2},t)\,,\end{split}$
(9)

where

$\begin{split}f(m^{2},t)\equiv
t\int_{k}\frac{e^{-tk^{2}}}{k^{2}+m^{2}}=1-m^{2}te^{m^{2}t}\Gamma(0,m^{2}t)\,,\qquad\Gamma(s,x)=\int_{x}^{\infty}\text{d}u\,u^{s-1}e^{-u}\,.\end{split}$
(10)

Assume that we would like to solve the momentum integral in Eq. 10 as an
expansion around $m^{2}\ll 1/t$. Obviously, simply interchanging the expansion
with the integration, which corresponds to assuming $k^{2}\gg m^{2}$ in the
integrand, leads to IR-divergent integrals:

$\begin{split}f^{(i)}(m^{2},t)&=t\sum_{n=1}^{\infty}(-m^{2})^{n-1}\int_{k}\frac{e^{-tk^{2}}}{k^{2n}}=\sum_{n=1}^{\infty}(-m^{2})^{n-1}t^{n-1+\epsilon}\frac{\Gamma(D/2-n)}{\Gamma(D/2)}\\\
&=1+m^{2}t\left(\frac{1}{\epsilon}+\ln
t+1\right)e^{m^{2}t}-(m^{2}t)^{2}-\frac{3}{4}(m^{2}t)^{2}+\ldots\,.\end{split}$
(11)

On the other hand, considering the region $k^{2}\sim m^{2}$, it follows that
$tk^{2}\ll 1$, so we can expand the exponential:

$\begin{split}f^{(ii)}(m^{2},t)&=t\sum_{n=0}^{\infty}\frac{(-t)^{n}}{n!}\int_{k}\frac{k^{2n}}{k^{2}+m^{2}}=m^{2}t\left[-\frac{1}{\epsilon}-1+\gamma_{\text{E}}+\ln
m^{2}\right]e^{m^{2}t}\,.\end{split}$ (12)

We recall that, despite the fact that the expansion of the integrand is
justified only in the respective region, the momentum integral can be taken
over all values of $k$, because all complementary regions will combine to
scale-less integrals which are discarded in dimensional regularization.
Combining the two regions, the $1/\epsilon$ pole cancels and one finds

$\begin{split}f(m^{2},t)&\stackrel{{\scriptstyle m^{2}\ll
1/t}}{{\rightsquigarrow}}f^{(i)}(m^{2},t)+f^{(ii)}(m^{2},t)=\\\
&=1+m^{2}t\,e^{m^{2}t}\,(\ln
m^{2}t+\gamma_{\text{E}})-(m^{2}t)^{2}-\frac{3}{4}(m^{2}t)^{3}+{\cal
O}((m^{2}t)^{4})\,,\end{split}$ (13)

which agrees with the asymptotic expansion of the explicit expression given in
Eq. 10.

Now let us consider the opposite case: $m^{2}\gg 1/t$. We have again two
regions, the first one leading to

$\begin{split}f(m^{2},t)&\stackrel{{\scriptstyle m^{2}\gg
k^{2}}}{{\rightsquigarrow}}\hat{f}^{(i)}(m^{2},t)=t\sum_{n=0}^{\infty}\frac{(-1)^{n}}{m^{2(n+1)}}\int_{k}\,k^{2n}\,e^{-k^{2}t}=\sum_{n=1}^{\infty}(-1)^{n+1}n!\,(m^{2}t)^{-n}\,.\end{split}$
(14)

The second region is given again by $k^{2}\sim m^{2}$, which means that
$k^{2}t\gg 1$. Its contribution vanishes, because the Taylor series of
$e^{-1/x}$ around $x=0$ is identical to zero. Therefore,

$\begin{split}f(m^{2},t)\stackrel{{\scriptstyle m^{2}\gg
1/t}}{{\rightsquigarrow}}=\hat{f}^{(i)}(m^{2},t)=\frac{1}{m^{2}t}-\frac{2}{(m^{2}t)^{2}}+\frac{6}{(m^{2}t)^{3}}+\ldots\end{split}$
(15)

which again agrees with the Taylor series of Eq. 10 around $1/(m^{2}t)=0$.

Of course, our presentation here is only a sketch of the general idea. At
higher orders, one needs to take into account integrations over flow-time
parameters. In the small-$t$ limit, all flow-time integration variables are
bound to be small as well, and the extension to higher loop order is
straightforward. In the large-$t$ limit, however, integration over flow-time
parameters extends over “large” and “small” regions, and the expansion of the
integrand becomes non-trivial. A general treatment of the strategy of regions
for flow-time integrals at higher orders is thus ongoing work and will be
presented elsewhere.

##  4 Hadronic vacuum polarization

Consider the operator product expansion of the correlator of two vector
currents $j_{\mu}(x)$ in $n_{f}$-flavor QCD with a single massive quark flavor
[24]:

$\begin{split}\Pi_{\mu\nu}(Q)&\equiv\int\text{d}^{4}x\,e^{iQx}\langle
Tj_{\mu}(x)j_{\nu}(0)\rangle=(-\delta_{\mu\nu}+Q_{\mu}Q_{\nu}/Q^{2})\Pi(Q^{2})\\\
&\Pi(Q^{2})\stackrel{{\scriptstyle
Q^{2}\to\infty}}{{\sim}}C^{(0),\text{{B}}}(Q)+m_{\text{{B}}}^{2}C^{(2),\text{{B}}}(Q)+\sum_{n}C^{\text{{B}}}_{n}(Q)\langle\mathcal{O}_{n}(x=0)\rangle\,,\end{split}$
(16)

This form reflects the fact that, up to mass dimension two, only the trivial
operators $\mathds{1}$ and $m_{\text{{B}}}^{2}\mathds{1}$ contribute, where
$m_{\text{{B}}}$ is the bare quark mass. At mass dimension four, one has the
following set of physical operators (the space-time argument is suppressed in
most of what follows):

$\begin{split}\mathcal{O}_{1}&=\frac{1}{g_{\text{{B}}}^{2}}F^{a}_{\mu\nu}F^{a}_{\mu\nu}\,,\qquad\mathcal{O}_{2}=\sum_{q=1}^{n_{\text{f}}}\bar{\psi}_{q}\overleftrightarrow{\not{D}}\psi_{q}\,,\qquad\mathcal{O}_{3}=m_{\text{{B}}}^{4}\mathds{1}\,,\end{split}$
(17) $\begin{split}\mbox{where}\qquad
F^{a}_{\mu\nu}&=\partial_{\mu}A^{a}_{\nu}-\partial_{\nu}A^{a}_{\mu}+f^{abc}A_{\mu}^{b}A_{\nu}^{c}\,,\qquad\overleftrightarrow{D}\\!_{\mu}=\partial_{\mu}-\overleftarrow{\partial}_{\mu}+2A^{a}_{\mu}T^{a}\,.\end{split}$
(18)

After renormalization of $m_{\text{{B}}}$ and the bare coupling
$g_{\text{{B}}}$, the operator matrix elements on the r.h.s. of Eq. 16 are
still divergent. The divergences can be absorbed into the bare coefficient
functions with the help of operator renormalization:

$\begin{split}\sum_{n}C_{n}^{\text{{B}}}\mathcal{O}_{n}\equiv\sum_{n}C_{n}Z_{nm}\mathcal{O}_{m}\equiv\sum_{n}C_{n}\mathcal{O}^{\text{{R}}}_{m}\,.\end{split}$
(19)

The fact that the classical mass dimension of the operators is the same as
that of the Lagrangian allows one to express the renormalization matrix $Z$ in
terms of the QCD $\beta$ function, the quark mass anomalous dimension, and the
anomalous dimension of the vacuum energy to all orders [25].

The operator product expansion of Eq. 16 represents a factorization into long-
and short-distance effects. The former are contained in the matrix elements
$\langle\mathcal{O}_{n}\rangle$ and their evaluation requires non-perturbative
methods such as lattice QCD. The latter are in the coefficient function whose
perturbative expressions are known through next-to-next-to-leading order
(NNLO) and beyond (see Refs. [26, 27, 28], for example). A precise prediction
of $\Pi_{\mu\nu}(Q)$ requires full control of the matching between the two,
which is notoriously difficult due to the different regularization and
renormalization schemes.

The small-flow-time expansion (SFTX) provides a potential solution to this
problem by unifying the renormalization scheme for both the coefficient
functions and the operators [5]. The spectrum of possible applications is
enormous (see Refs. [29, 30, 31, 32, 33, 34, 35, 36], for example). The idea
is to define flowed operators $\tilde{\mathcal{O}}_{n}(t)$ by replacing the
regular by flowed fields in Eq. 17, and then expressing them in terms of
regular operators in the limit $t\to 0$:

$\begin{split}\tilde{\mathcal{O}}_{n}(t)&\asymp\zeta_{n}^{(0),\text{{B}}}(t)+\zeta_{n}^{(2),\text{{B}}}(t)m_{\text{{B}}}^{2}+\sum_{m}\zeta^{\text{{B}}}_{nm}(t)\mathcal{O}_{m}=\zeta_{n}^{(0)}(t)+\zeta_{n}^{(2)}(t)m^{2}+\sum_{m,k}\zeta_{nk}(t)\mathcal{O}^{\text{{R}}}_{k}\,.\end{split}$
(20)

Here, the symbol $\asymp$ denotes that the relation holds only asymptotically
for $t\to 0$. The coefficients $\zeta_{n}^{(0)}(t)$, $\zeta_{n}^{(2)}(t)$, and
$\zeta_{nm}(t)$ are UV finite. They have been calculated in Ref. [33] through
NNLO QCD. While the $\zeta_{nm}(t)$ depend only logarithmically on $t$,
$\zeta_{n}^{(0)}(t)$ and $\zeta_{n}^{(2)}(t)$ behave as $1/t^{2}$ and $1/t$ as
$t\to 0$. In fact, they simply correspond to the first two terms in the Taylor
expansion of the vevs around $m=0$:

$\begin{split}\zeta_{n}^{(0)}(t)+\zeta_{n}^{(2)}(t)m^{2}=\langle\mathcal{O}_{n}(t)\rangle\Big{|}_{m=0}+m^{2}\frac{d}{dm^{2}}\langle\mathcal{O}_{n}(t)\rangle\Big{|}_{m=0}\,.\end{split}$
(21)

Inverting Eq. 20 and inserting it into Eq. 16 leads to

$\begin{split}\Pi(Q)\stackrel{{\scriptstyle
Q^{2}\to\infty}}{{\sim}}C^{(0),\text{{B}}}(Q)+m^{2}C^{(2)}(Q)+\sum_{n}\tilde{C}_{n}(t)\bar{\mathcal{O}}_{n}(t)\,,\end{split}$
(22)
$\begin{split}\mbox{with}\qquad\tilde{C}_{n}(t)=\sum_{m}C_{m}\zeta^{-1}(t)_{mn}\,,\qquad\bar{\mathcal{O}}_{n}(t)=\tilde{\mathcal{O}}_{n}(t)-\zeta_{n}^{(0)}(t)-m^{2}\zeta_{n}^{(2)}(t)\,.\end{split}$
(23)

Note that power divergences in the limit $t\to 0$ cancel in the combination
$\bar{O}_{n}(t)$. A precise lattice determination of the
$\langle\bar{\mathcal{O}}(t)$ could thus open the way towards a novel
calculation of the vacuum polarization, and thus independent input for the
lattice determination of hadronic contributions to low-energy observables such
as the muon anomalous magnetic moment.

##  5 Energy-momentum tensor

Dropping terms that vanish either under a BRST transformation or by equations
of motion, the energy-momentum tensor of QCD takes the form of an operator
product expansion similar to Eq. 16:

$\begin{split}T_{\mu\nu}=\sum_{n=1}^{4}C_{n}\mathcal{O}_{n,\mu\nu}\,,\end{split}$
(24)
$\begin{split}\mbox{where}\qquad\mathcal{O}_{1,\mu\nu}&=\frac{1}{g_{\text{{B}}}^{2}}G_{\mu\rho}G_{\rho\nu}\,,\qquad\mathcal{O}_{2,\mu\nu}=\frac{\delta_{\mu\nu}}{g_{\text{{B}}}^{2}}G_{\rho\sigma}G_{\rho\sigma}\,,\\\
\mathcal{O}_{3}&=\bar{\psi}\left(\gamma_{\mu}\overleftrightarrow{D}_{\nu}+\gamma_{\nu}\overleftrightarrow{D}_{\mu}\right)\psi\,,\qquad\mathcal{O}_{4}=\delta_{\mu\nu}m_{\text{{B}}}\bar{\psi}\psi\,.\end{split}$
(25)

However, as opposed to Eq. 16, the “Wilson coefficients” $C_{n}$ in this case
are given by simple numerical constants to all orders in perturbation theory:

$\begin{split}C_{1}=1\,,\qquad C_{2}=-\frac{1}{4}\,,\qquad
C_{3}=\frac{1}{4}\,,\qquad C_{4}=0\,.\end{split}$ (26)

Furthermore, due to the Ward-Takahashi identities among the $Z_{nm}$, the
energy-momentum tensor is finite, in the sense that the coefficients $C_{n}$
are not renormalized, i.e. $C_{n}=C_{n}^{\text{{B}}}$.

Using the SFTX, we write the operators as

$\begin{split}\mathcal{O}_{n}\asymp\sum_{m=1}^{4}\zeta^{\text{{B}},-1}(t)_{nm}\tilde{\mathcal{O}}_{m}(t)\end{split}$
(27)

and insert this into Eq. 24 to obtain [29, 30]

$\begin{split}T_{\mu\nu}\asymp\sum_{n=1}^{4}c_{n}(t)\tilde{\mathcal{O}}_{n,\mu\nu}(t)\,,\qquad
c_{n}(t)=\sum_{m=1}^{4}C_{m}\zeta^{\text{{B}},-1}_{mn}(t)=\sum_{m=1}^{4}C_{m}\zeta^{-1}_{mn}(t)\,.\end{split}$
(28)

The coefficients $c_{n}(t)$ are finite even without operator renormalization.
They have been evaluated through NNLO [29, 30, 37] and used to study
thermodynamics of QCD [32, 38, 39, 40, 41].

#### Acknowledgments.

I would like to thank Aiman El Asad, Janosch Borgulat, Benedikt Gurdon,
Stefano Palmisano, Fabian Lange, Tobias Neumann, Paul Mork, Antonio Rago,
Joshua Schophoven, and Andrea Shindler for useful and inspiring conversations.

## References

  * Lüscher [2010] M. Lüscher, JHEP 08, 071 (2010), [Erratum: JHEP 03, 092 (2014)], 1006.4518.
  * Lüscher [2013] M. Lüscher, JHEP 04, 123 (2013), 1302.5246.
  * Narayanan and Neuberger [2006] R. Narayanan and H. Neuberger, JHEP 03, 064 (2006), hep-th/0601210.
  * Lüscher [2010] M. Lüscher, Commun. Math. Phys. 293, 899 (2010), 0907.5491.
  * Lüscher and Weisz [2011] M. Lüscher and P. Weisz, JHEP 02, 051 (2011), 1101.0963.
  * Artz et al. [2019] J. Artz, R. V. Harlander, F. Lange, T. Neumann, and M. Prausa, JHEP 06, 121 (2019), [Erratum: JHEP 10, 032 (2019)], 1905.00882.
  * Nogueira [2006] P. Nogueira, Nucl. Instrum. Meth. A 559, 220 (2006).
  * van Ritbergen et al. [1999] T. van Ritbergen, A. Schellekens, and J. Vermaseren, Int. J. Mod. Phys. A 14, 41 (1999), hep-ph/9802376.
  * Vermaseren [2000] J. Vermaseren (2000), math-ph/0010025.
  * Kuipers et al. [2013] J. Kuipers, T. Ueda, J. Vermaseren, and J. Vollinga, Comput. Phys. Commun. 184, 1453 (2013), 1203.6543.
  * Maierhöfer et al. [2018] P. Maierhöfer, J. Usovitsch, and P. Uwer, Comput. Phys. Commun. 230, 99 (2018), 1705.05610.
  * Maierhöfer and Usovitsch [2018] P. Maierhöfer and J. Usovitsch (2018), 1812.01491.
  * Klappert and Lange [2020] J. Klappert and F. Lange, Comput. Phys. Commun. 247, 106951 (2020), 1904.00009.
  * Klappert et al. [2021a] J. Klappert, S. Y. Klein, and F. Lange, Comput. Phys. Commun. 264, 107968 (2021a), 2004.01463.
  * Klappert et al. [2021b] J. Klappert, F. Lange, P. Maierhöfer, and J. Usovitsch, Comput. Phys. Commun. 266, 108024 (2021b), 2008.06494.
  * Harlander et al. [1998] R. Harlander, T. Seidensticker, and M. Steinhauser, Phys. Lett. B 426, 125 (1998), hep-ph/9712228.
  * Harlander and Neumann [2016] R. V. Harlander and T. Neumann, JHEP 06, 161 (2016), 1606.03756.
  * Dalla Brida and Lüscher [2016] M. Dalla Brida and M. Lüscher, PoS LATTICE2016, 332 (2016), 1612.04955.
  * Fodor et al. [2018] Z. Fodor, K. Holland, J. Kuti, D. Nogradi, and C. H. Wong, EPJ Web Conf. 175, 08027 (2018), 1711.04833.
  * Dalla Brida and Ramos [2019] M. Dalla Brida and A. Ramos, Eur. Phys. J. C 79, 720 (2019), 1905.05147.
  * Peterson et al. [2021] C. T. Peterson, A. Hasenfratz, J. van Sickle, and O. Witzel, in _38th International Symposium on Lattice Field Theory_ (2021), 2109.09720.
  * Harlander et al. [2020a] R. Harlander, S. Klein, and M. Lipp, Comput. Phys. Commun. 256, 107465 (2020a), 2003.00896.
  * Beneke and Smirnov [1998] M. Beneke and V. A. Smirnov, Nucl. Phys. B 522, 321 (1998), hep-ph/9711391.
  * Dominguez [2014] C. Dominguez, Int. J. Mod. Phys. A 29, 1430069 (2014), 1411.3462.
  * Spiridonov and Chetyrkin [1988] V. Spiridonov and K. Chetyrkin, Sov. J. Nucl. Phys. 47, 522 (1988).
  * Chetyrkin et al. [1985] K. Chetyrkin, V. Spiridonov, and S. Gorishnii, Phys. Lett. B 160, 149 (1985).
  * Chetyrkin et al. [1997] K. Chetyrkin, R. Harlander, J. H. Kühn, and M. Steinhauser, Nucl. Phys. B 503, 339 (1997), hep-ph/9704222.
  * Harlander and Steinhauser [1997] R. Harlander and M. Steinhauser, Phys. Rev. D 56, 3980 (1997), hep-ph/9704436.
  * Suzuki [2013] H. Suzuki, PTEP 2013, 083B03 (2013), [Erratum: PTEP 2015, 079201 (2015)], 1304.0533.
  * Makino and Suzuki [2014] H. Makino and H. Suzuki, PTEP 2014, 063B02 (2014), [Erratum: PTEP 2015, 079202 (2015)], 1403.4772.
  * Hieda and Suzuki [2016] K. Hieda and H. Suzuki, Mod. Phys. Lett. A 31, 1650214 (2016), 1606.04193.
  * Iritani et al. [2019] T. Iritani, M. Kitazawa, H. Suzuki, and H. Takaura, PTEP 2019, 023B02 (2019), 1812.06444.
  * Harlander et al. [2020b] R. V. Harlander, F. Lange, and T. Neumann, JHEP 08, 109 (2020b), 2007.01057.
  * Suzuki et al. [2020] A. Suzuki, Y. Taniguchi, H. Suzuki, and K. Kanaya, Phys. Rev. D 102, 034508 (2020), 2006.06999.
  * Rizik et al. [2020] M. D. Rizik, C. J. Monahan, and A. Shindler (SymLat), Phys. Rev. D 102, 034509 (2020), 2005.04199.
  * Mereghetti et al. [2021] E. Mereghetti, C. J. Monahan, M. D. Rizik, A. Shindler, and P. Stoffer (2021), 2111.11449.
  * Harlander et al. [2018] R. V. Harlander, Y. Kluth, and F. Lange, Eur. Phys. J. C 78, 944 (2018), [Erratum: Eur.Phys.J.C 79, 858 (2019)], 1808.09837.
  * Kitazawa et al. [2019] M. Kitazawa, S. Mogliacci, I. Kolbé, and W. A. Horowitz, Phys. Rev. D 99, 094507 (2019), 1904.00241.
  * Shirogane et al. [2021] M. Shirogane, S. Ejiri, R. Iwami, K. Kanaya, M. Kitazawa, H. Suzuki, Y. Taniguchi, and T. Umeda (WHOT-QCD), PTEP 2021, 013B08 (2021), 2011.10292.
  * Yanagihara et al. [2020] R. Yanagihara, M. Kitazawa, M. Asakawa, and T. Hatsuda, Phys. Rev. D 102, 114522 (2020), 2010.13465.
  * Taniguchi et al. [2020] Y. Taniguchi, S. Ejiri, K. Kanaya, M. Kitazawa, H. Suzuki, and T. Umeda (WHOT-QCD), Phys. Rev. D 102, 014510 (2020), [Erratum: Phys.Rev.D 102, 059903 (2020)], 2005.00251.

  *[QCD]: Quantum Chromodynamics
  *[GFF]: gradient-flow formalism
  *[NLO]: next-to-leading order
  *[LO]: leading order
  *[NNLO]: next-to-next-to-leading order
  *[SFTX]: small-flow-time expansion


# Rejuvenating Low-Frequency Words:
Making the Most of Parallel Data in Non-Autoregressive Translation

Liang Ding
The University of Sydney
<EMAIL_ADDRESS>
&Longyue Wang∗
Tencent AI Lab
<EMAIL_ADDRESS>
&Xuebo Liu
University of Macau
<EMAIL_ADDRESS>
Derek F. Wong
University of Macau
<EMAIL_ADDRESS>
&Dacheng Tao
JD Explore Academy, JD.com
<EMAIL_ADDRESS>
&Zhaopeng Tu
Tencent AI Lab
<EMAIL_ADDRESS>
Liang Ding and Longyue Wang contributed equally to this work. Work was done
when Liang Ding and Xuebo Liu were interning at Tencent AI Lab.

###### Abstract

Knowledge distillation (KD) is commonly used to construct synthetic data for
training non-autoregressive translation (NAT) models. However, there exists a
discrepancy on low-frequency words between the distilled and the original
data, leading to more errors on predicting low-frequency words. To alleviate
the problem, we directly expose the raw data into NAT by leveraging
pretraining. By analyzing directed alignments, we found that KD makes low-
frequency source words aligned with targets more deterministically but fails
to align sufficient low-frequency words from target to source. Accordingly, we
propose reverse KD to rejuvenate more alignments for low-frequency target
words. To make the most of authentic and synthetic data, we combine these
complementary approaches as a new training strategy for further boosting NAT
performance. We conduct experiments on five translation benchmarks over two
advanced architectures. Results demonstrate that the proposed approach can
significantly and universally improve translation quality by reducing
translation errors on low-frequency words. Encouragingly, our approach
achieves 28.2 and 33.9 BLEU points on the WMT14 English-German and WMT16
Romanian-English datasets, respectively. Our code, data, and trained models
are available at https://github.com/alphadl/RLFW-NAT.

##  1 Introduction

Recent years have seen a surge of interest in non-autoregressive translation
(Gu et al., 2018, NAT,), which can improve the decoding efficiency by
predicting all tokens independently and simultaneously. The non-autoregressive
factorization breaks conditional dependencies among output tokens, which
prevents a model from properly capturing the highly multimodal distribution of
target translations. As a result, the translation quality of NAT models often
lags behind that of autoregressive translation (Vaswani et al., 2017, AT,)
models. To balance the trade-off between decoding speed and translation
quality, knowledge distillation (KD) is widely used to construct a new
training data for NAT models Gu et al. (2018). Specifically, target sentences
in the distilled training data are generated by an AT teacher, which makes NAT
easily acquire more deterministic knowledge and achieve significant
improvement Zhou et al. (2020).

Previous studies have shown that distillation may lose some important
information in the original training data, leading to more errors on
predicting low-frequency words. To alleviate this problem, Ding et al. (2021b)
proposed to augment NAT models the ability to learn lost knowledge from the
original data. However, their approach relies on external resources (e.g. word
alignment) and human-crafted priors, which limits the applicability of the
method to a broader range of tasks and languages. Accordingly, we turn to
directly expose the raw data into NAT by leveraging pretraining without
intensive modification to model architectures (§2.2). Furthermore, we analyze
bilingual links in the distilled data from two alignment directions (i.e.
source-to-target and target-to-source). We found that KD makes low-frequency
source words aligned with targets more deterministically but fails to align
low-frequency words from target to source due to information loss. Inspired by
this finding, we propose reverse KD to recall more alignments for low-
frequency target words (§2.3). We then concatenate two kinds of distilled data
to maintain advantages of deterministic knowledge and low-frequency
information. To make the most of authentic and synthetic data, we combine
three complementary approaches (i.e. raw pretraining, bidirectional
distillation training and KD finetuning) as a new training strategy for
further boosting NAT performance (§2.4).

(a) Traditional Training

(b) Raw Pretraining

(c) Bidirectional Distillation Training

Figure 1: An illustration of different strategies for training NAT models.
“distill” and “reverse distill” indicate sequence-level knowledge distillation
with forward and backward AT teachers, respectively. The data block in
transparent color means source- or target-side data are synthetically
generated. Best view in color.

We validated our approach on five translation benchmarks (WMT14 En-De, WMT16
Ro-En, WMT17 Zh-En, WAT17 Ja-En and WMT19 En-De) over two advanced
architectures (Ghazvininejad et al., 2019, Mask Predict,; Gu et al., 2019,
Levenshtein Transformer,). Experimental results show that the proposed method
consistently improve translation performance over the standard NAT models
across languages and advanced NAT architectures. Extensive analyses confirm
that the performance improvement indeed comes from the better lexical
translation accuracy especially on low-frequency tokens.

#### Contributions

Our main contributions are:

  * •

We show the effectiveness of rejuvenating low-frequency information by
pretraining NAT models from raw data.

  * •

We provide a quantitative analysis of bilingual links to demonstrate the
necessity to improve low-frequency alignment by leveraging both KD and reverse
KD.

  * •

We introduce a simple and effective training recipe to accomplish this goal,
which is robustly applicable to several model structures and language pairs.

##  2 Rejuvenating Low-Frequency Words

###  2.1 Preliminaries

#### Non-Autoregressive Translation

Given a source sentence $\bf x$, an AT model generates each target word ${\bf
y}_{t}$ conditioned on previously generated ones ${\bf y}_{<t}$, leading to
high latency on the decoding stage. In contrast, NAT models break this
autoregressive factorization by producing target words in parallel.
Accordingly, the probability of generating $\bf y$ is computed as:

$p({\bf y}|{\bf x})=\prod_{t=1}^{T}p({\bf y}_{t}|{\bf x};\theta)$ (1)

where $T$ is the length of the target sequence, and it is usually predicted by
a separate conditional distribution. The parameters $\theta$ are trained to
maximize the likelihood of a set of training examples according to
$\mathcal{L}(\theta)=\operatorname*{arg\,max}_{\theta}\log p({\bf y}|{\bf
x};\theta)$. Typically, most NAT models are implemented upon the framework of
Transformer Vaswani et al. (2017).

#### Knowledge Distillation

Gu et al. (2018) pointed out that NAT models suffer from the multimodality
problem, where the conditional independence assumption prevents a model from
properly capturing the highly multimodal distribution of target translations.
Thus, the sequence-level knowledge distillation is introduced to reduce the
modes of training data by replacing their original target-side samples with
sentences generated by an AT teacher Gu et al. (2018); Zhou et al. (2020); Ren
et al. (2020). Formally, the original parallel data Raw and the distilled data
$\overrightarrow{\text{KD}}$ can be defined as follows:

Raw $\displaystyle=\\{(\mathbf{x}_{i},\mathbf{y}_{i})\\}^{N}_{i=1}$ (2)
$\displaystyle\overrightarrow{\text{KD}}$
$\displaystyle=\\{(\mathbf{x}_{i},f_{s\mapsto
t}(\mathbf{x}_{i}))|\mathbf{x}_{i}\in\text{Raw}_{\text{s}}\\}^{N}_{i=1}$ (3)

where $f_{s\mapsto t}$ represents an AT-based translation model trained on Raw
data for translating text from the source to the target language. $N$ is the
total number of sentence pairs in training data. As shown in Figure 1 (a),
well-performed NAT models are generally trained on
$\overrightarrow{\text{KD}}$ data instead of Raw.

###  2.2 Pretraining with Raw Data

#### Motivation

Gao et al. (2018) showed that more than 90% of words are lower than 10e-4
frequency in WMT14 En-De dataset. This token imbalance problem biases
translation models towards over-fitting to frequent observations while
neglecting those low-frequency observations Gong et al. (2018); Nguyen and
Chiang (2018); Gu et al. (2020). Thus, the AT teacher $f_{s\mapsto t}$ tends
to generate more high-frequency tokens and less low-frequency tokens during
constructing distilled data $\overrightarrow{\text{KD}}$.

On the one hand, KD can reduce the modes in training data (i.e. multiple
lexical choices for a source word), which lowers the intrinsic uncertainty
(Ott et al., 2018) and learning difficulty for NAT (Zhou et al., 2020; Ren et
al., 2020), making it easily acquire more deterministic knowledge. On the
other hand, KD aggravates the imbalance of high-frequency and low-frequency
words in training data and lost some important information originated in raw
data. Ding et al. (2021b) revealed the side effect of distilled training data,
which cause lexical choice errors for low-frequency words in NAT models.
Accordingly, they introduced an extra bilingual data-dependent prior objective
to augments NAT models the ability to learn the lost knowledge from raw data.
We use their findings as our departure point, but rejuvenate low-frequency
words in a more simple and direct way: directly exposing raw data into NAT via
pretraining.

Data |  $\mathbf{s}\mapsto\mathbf{t}$ LFW Links |  $\mathbf{t}\mapsto\mathbf{s}$ LFW Links
---|---|---
R | P | F1 | R | P | F1
Raw | 66.4 | 81.9 | 73.3 | 72.3 | 80.6 | 76.2
$\overrightarrow{\text{KD}}$ | 73.4 | 89.2 | 80.5 | 69.9 | 79.1 | 74.2
$\overleftarrow{\text{KD}}$ | 61.2 | 79.4 | 69.1 | 82.9 | 83.1 | 83.0
Table 1: Evaluation on aligned links between source- and target-side low-
frequency words (LFW). A directed line indicates aligning bilingual words from
the source to the target side ($\mathbf{s}\mapsto\mathbf{t}$) or in an
opposite way ($\mathbf{t}\mapsto\mathbf{s}$). R, P and F1 are recall,
precision and F1-score.

#### Our Approach

Many studies have shown that pretraining could transfer the knowledge and data
distribution, especially for rare categories, hence improving the model
robustness Hendrycks et al. (2019); Mathis et al. (2021). Here we want to
transfer the distribution of lost information, e.g. low-frequency words. As
illustrated in Figure 1(b), we propose to first pretrain NAT models on Raw
data and then continuously train them on $\overrightarrow{\text{KD}}$ data.
The raw data maintain the original distribution especially on low-frequency
words. Although it is difficult for NAT to learn high-mode data, the
pretraining can acquire general knowledge from authentic data, which may help
better and faster learning further tasks. Thus, we early stop pretraining when
the model can achieve 90% of the best performance of raw data in terms of BLEU
score Platanios et al. (2019)111In preliminary experiments, we tried another
simple strategy: early-stop at fixed step according to the size of training
data (e.g. training 70K En-De and early stop at 20K / 30K / 40K,
respectively). We found that both strategies achieve similar performance.. In
order to keep the merits of low-modes, we further train the pretrained model
on distilled data $\overrightarrow{\text{KD}}$. As it is easy for NAT to learn
deterministic knowledge, we finetune the model for the rest steps. For fair
comparison, the total training steps of the proposed method are same as the
traditional one. In general, we expect that this training recipe can provide a
good trade-off between raw and distilled data (i.e. high-modes and complete
vs. low-modes and incomplete).

Data | Sentence
---|---
Raw${}_{\textbf{S}}$ |  海克曼 和 奥德海姆 提出 … 模型
Raw${}_{\textbf{T}}$ |  Hackman and Oldham propose … model
$\overrightarrow{\text{KD}}_{\textbf{T}}$ |  Heckman and Oddheim propose … model
$\overleftarrow{\text{KD}}_{\textbf{S}}$ |  哈克曼 和 奥尔德姆 提出 … 模式
Table 2: An example in different kinds of data. “Raw” means the original data
while “$\overrightarrow{\text{KD}}$” and “$\overleftarrow{\text{KD}}$”
indicate syntactic data distilled by KD and reverse KD, respectively. The
subscript “S” or “T” is short for source- or target-side. The low-frequency
words are highlighted with colors and italics are incorrect translations.

###  2.3 Bidirectional Distillation Training

#### Analyzing Bilingual Links in Data

KD simplifies the training data by replacing low-frequency target words with
high-frequency ones Zhou et al. (2020). This is able to facilitate easier
aligning source words to target ones, resulting in high bilingual coverage
Jiao et al. (2020). Due to the information loss, we argue that KD makes low-
frequency target words have fewer opportunities to align with source ones. To
verify this, we propose a method to quantitatively analyze bilingual links
from two directions, where low-frequency words are aligned from source to
target ($\text{s}\mapsto\text{t}$) or in an opposite direction
($\text{t}\mapsto\text{s}$).

The method can be applied to different types of data. Here we take
$\text{s}\mapsto\text{t}$ links in Raw data as an example to illustrate the
algorithm. Given the WMT14 En-De parallel corpus, we employ an unsupervised
word alignment method222The FastAlign (Dyer et al., 2013) was employed to
build word alignments for the training datasets. Och and Ney (2003) to produce
a word alignment, and then we extract aligned links whose source words are
low-frequency (called $\text{s}\mapsto\text{t}$ LFW Links). Second, we
randomly select a number of samples from the parallel corpus. For better
comparison, the subset should contains the same $i$ in Equation (2) as that of
other type of datasets (e.g. $i$ in Equation (3) for
$\overrightarrow{\text{KD}}$). Finally, we calculate recall, precision, F1
scores based on low-frequency bilingual links for the subset. Recall (R)
represents how many low-frequency source words can be aligned to targets.
Precision (P) means how many aligned low-frequency links are correct according
to human evaluation. F1 is the harmonic mean between precision and recall.
Similarly, we can analyze $\text{t}\mapsto\text{s}$ LFW Links by considering
low-frequency targets.

Table 1 shows the results on low-frequency links. Compared with Raw,
$\overrightarrow{\text{KD}}$ can recall more $\text{s}\mapsto\text{t}$ LFW
links (73.4 vs. 66.4) with more accurate alignment (89.2 vs. 73.3). This
demonstrates the effectiveness of KD for NAT models from the bilingual
alignment perspective. However, in the $\text{t}\mapsto\text{s}$ direction,
there are fewer LFW links (69.9 vs. 72.3) with worse alignment quality (79.1
vs. 80.6) in $\overrightarrow{\text{KD}}$ than those in Raw. This confirms our
claim that KD harms NAT models due to the loss of low-frequency target words.
Inspired by these findings, it is natural to assume that reverse KD exhibits
complementary properties. Accordingly, we conduct the same analysis method on
$\overleftarrow{\text{KD}}$ data, and found better $\text{t}\mapsto\text{s}$
links but worse $\text{s}\mapsto\text{t}$ links compared with Raw. Take the
Zh-En sentence pair in Table 2 for example, $\overrightarrow{\text{KD}}$
retains the source side low-frequency Chinese words “海克曼” (Raw${}_{\text{S}}$)
but generates the high-frequency English words “Heckman” instead of the golden
“Hackman” ($\overrightarrow{\text{KD}}_{\textbf{T}}$). On the other hand,
$\overleftarrow{\text{KD}}$ preserves the low-frequency English words
“Hackman” (Raw${}_{\text{T}}$) but produces the high-frequency Chinese words
“哈克曼” ($\overleftarrow{\text{KD}}_{\textbf{S}}$).

#### Our Approach

Based on analysis results, we propose to train NAT models on bidirectional
distillation by concatenating two kinds of distilled data. The reverse
distillation is to replace the source sentences in the original training data
with synthetic ones generated by a backward AT teacher.333This is different
from back-translation Edunov et al. (2018), which is an alternative to
leverage monolingual data. According to Equation 3,
$\overleftarrow{\text{KD}}$ can be formulated as:

$\displaystyle\overleftarrow{\text{KD}}$
$\displaystyle=\\{(\mathbf{y}_{i},f_{t\mapsto
s}(\mathbf{y}_{i}))|\mathbf{y}_{i}\in\text{Raw}_{\text{t}}\\}^{N}_{i=1}$ (4)

where $f_{t\mapsto s}$ represents an AT-based translation model trained on Raw
data for translating text from the target to the source language.

Figure 1(c) illustrates the training strategy. First, we employ both
$f_{s\mapsto t}$ and $f_{t\mapsto s}$ AT models to generate
$\overrightarrow{\text{KD}}$ and $\overleftarrow{\text{KD}}$ data,
respectively. Considering complementarity of two distilled data, we combine
$\overrightarrow{\text{KD}}$ and $\overleftarrow{\text{KD}}$ as a new training
data for training NAT models. We expect that 1) distilled data can maintain
advantages of low-modes; 2) bidirectinoal distillation can recall more LFW
links on two directions with better alignment quality, leading to the overall
improvements. Besides, Nguyen et al. (2020) claimed that combining different
distilled data (generated by various models trained with different seeds)
improves data diversification for NMT, and we leave this for future work.

###  2.4 Combining Both of Them: Low-Frequency Rejuvenation (LFR)

We have proposed two parallel approaches to rejuvenate low-frequency knowledge
from authentic (§2.2) and synthetic (§2.3) data, respectively. Intuitively, we
combine both of them to further improve the model performance.

From data view, two presented training strategies are:
$\text{Raw}\rightarrow\overrightarrow{\text{KD}}$ (Raw Pretraining) and
$\overrightarrow{\text{KD}}+\overleftarrow{\text{KD}}$ (Bidirectional
Distillation Training). Considering the effectiveness of pretraining Mathis et
al. (2021) and clean finetuning Wu et al. (2019), we introduce a combined
pipeline:
$\text{Raw}\rightarrow\overrightarrow{\text{KD}}+\overleftarrow{\text{KD}}\rightarrow\overrightarrow{\text{KD}}$
as out best training strategy. There are many possible ways to implement the
general idea of combining two approaches. The aim of this paper is not to
explore the whole space but simply to show that one fairly straightforward
implementation works well and the idea is reasonable. Nonetheless, we compare
possible strategies of combination two approaches as well as demonstrate their
complementarity in §3.3. While in main experiments (in §3.2), we valid the
combination strategy, namely Low-Frequency Rejuvenation (LFR).

Model | Iteration | Speed | En-De | Ro-En
---|---|---|---|---
BLEU | ALF | BLEU | ALF
AT Models
Transformer-Base (Ro-En Teacher) | n/a | 1.0$\times$ | 27.3 | 70.5 | 34.1 | 73.6
Transformer-Big (En-De Teacher) | n/a | 0.8$\times$ | 29.2 | 73.0 | n/a | n/a
Existing NAT Models
NAT (Gu et al., 2018) | 1.0 | 2.4$\times$ | 19.2 | n/a | 31.4 | n/a
Iterative NAT (Lee et al., 2018) | 10.0 | 2.0$\times$ | 21.6 | 30.2
DisCo (Kasai et al., 2020) | 4.8 | 3.2$\times$ | 26.8 | 33.3
Mask-Predict (Ghazvininejad et al., 2019) | 10.0 | 1.5$\times$ | 27.0 | 33.3
Levenshtein (Gu et al., 2019) | 2.5 | 3.5$\times$ | 27.3 | 33.3
Our NAT Models
Mask-Predict (Ghazvininejad et al., 2019) | 10.0 | 1.5$\times$ | 27.0 | 68.4 | 33.3 | 70.9
+Low-Frequency Rejuvenation | 27.8† | 72.3 | 33.9† | 72.4
|  |  |  |  |  |
Levenshtein (Gu et al., 2019) | 2.5 | 3.5$\times$ | 27.4 | 69.2 | 33.2 | 71.1
+Low-Frequency Rejuvenation | 28.2† | 72.8 | 33.8† | 72.7
Table 3: Comparison with previous work on WMT14 En-De and WMT16 Ro-En.
“Iteration” indicates the number of iterative refinement while “Speed” shows
the speed-up ratio of decoding. “ALF” is the translation accuracy on low-
frequency words. “†” indicates statistically significant difference ($p<0.05$)
from corresponding baselines.

##  3 Experiment

###  3.1 Setup

Data Main experiments are conducted on four widely-used translation datasets:
WMT14 English-German (En-De, Vaswani et al. 2017), WMT16 Romanian-English (Ro-
En, Gu et al. 2018), WMT17 Chinese-English (Zh-En, Hassan et al. 2018), and
WAT17 Japanese-English (Ja-En, Morishita et al. 2017), which consist of 4.5M,
0.6M, 20M, and 2M sentence pairs, respectively. We use the same validation and
test datasets with previous works for fair comparison. To prove the
universality of our approach, we further experiment on different data volumes,
which are sampled from WMT19 En-De.444http://www.statmt.org/wmt19/translation-
task.html The Small and Medium corpora respectively consist of 1.0M and 4.5M
sentence pairs, and Large one is the whole dataset which contains 36M sentence
pairs. We preprocess all data via BPE (Sennrich et al., 2016) with 32K merge
operations. We use tokenized BLEU (Papineni et al., 2002) as the evaluation
metric, and sign-test Collins et al. (2005) for statistical significance test.
The translation accuracy of low-frequency words is measured by AoLC Ding et
al. (2021b), where word alignments are established based on the widely-used
automatic alignment tool GIZA++ Och and Ney (2003).

#### Models

We validated our research hypotheses on two state-of-the-art NAT models:

  * •

Mask-Predict (MaskT, Ghazvininejad et al. 2019) that uses the conditional mask
LM (Devlin et al., 2019) to iteratively generate the target sequence from the
masked input. We followed its optimal settings to keep the iteration number as
10 and length beam as 5.

  * •

Levenshtein Transformer (LevT, Gu et al. 2019) that introduces three steps:
deletion, placeholder and token prediction. The decoding iterations adaptively
depends on certain conditions.

We closely followed previous works to apply sequence-level knowledge
distillation to NAT (Kim and Rush, 2016). Specifically, we train both Base and
Big Transformer as the AT teachers. For Big model, we adopt large batch
strategy (i.e. 458K tokens/batch) to optimize the performance. Most NAT tasks
employ Transformer-Big as their strong teacher except for Ro-En and Small En-
De, which are distilled by Transformer-Base.

#### Training

Traditionally, NAT models are usually trained for 300K steps on regular batch
size (i.e. 128K tokens/batch). In this work, we empirically adopt large batch
strategy (i.e. 480K tokens/batch) to reduce the training steps for NAT (i.e.
70K). Accordingly, the learning rate warms up to $1\times 10^{-7}$ for 10K
steps, and then decays for 60k steps with the cosine schedule (Ro-En models
only need 4K and 21K, respectively). For regularization, we tune the dropout
rate from [0.1, 0.2, 0.3] based on validation performance in each direction,
and apply weight decay with 0.01 and label smoothing with $\epsilon$ = 0.1. We
use Adam optimizer (Kingma and Ba, 2015) to train our models. We followed the
common practices (Ghazvininejad et al., 2019; Kasai et al., 2020) to evaluate
the performance on an ensemble of top 5 checkpoints to avoid stochasticity.

Note that the total training steps of the proposed approach (in §2.2$\sim$2.4)
are identical with those of the standard training (in §2.1). Taking the best
training strategy
($\text{Raw}\rightarrow\overrightarrow{\text{KD}}+\overleftarrow{\text{KD}}\rightarrow\overrightarrow{\text{KD}}$)
for example, we empirically set the training step for each stage is 20K, 20K
and 30K, respectively. And Ro-En models respectively need 8K, 8K and 9K steps
in corresponding training stage.

Model | Zh-En | Ja-En
---|---|---
BLEU | ALF | BLEU | ALF
AT | 25.3 | 66.2 | 29.8 | 70.8
MaskT | 24.2 | 61.5 | 28.9 | 66.9
+LFR | 25.1† | 64.8 | 29.6† | 68.9
|  |  |  |
LevT | 24.4 | 62.7 | 29.1 | 66.8
+LFR | 25.1† | 65.3 | 29.7 | 69.2
Table 4: Performance on other language pairs, including WMT17 Zh-En and WAT17
Ja-En. “†” indicates statistically significant difference ($p<0.05$) from
corresponding baselines.

###  3.2 Results

#### Comparison with Previous Work

Table 3 lists the results of previous competitive NAT models Gu et al. (2018);
Lee et al. (2018); Kasai et al. (2020); Gu et al. (2019); Ghazvininejad et al.
(2019) on the WMT16 Ro-En and WMT14 En-De benchmark. We implemented our
approach on top of two advanced NAT models (i.e. Mask-Predict and Levenshtein
Transformer). Compared with standard NAT models, our training strategy
significantly and consistently improves translation performance
(BLEU$\uparrow$) across different language pairs and NAT models. Besides, the
improvements on translation performance are mainly due to a increase of
translation accuracy on low-frequency words (ALF$\uparrow$), which reconfirms
our claims. For instance, our method significantly improves the standard Mask-
Predict model by +0.8 BLEU score with a substantial +3.6 increase in ALF
score. Encouragingly, our approach push the existing NAT models to achieve new
SOTA performances (i.e. 28.2 and 33.9 BLEU on En-De and Ro-En, respectively).

It is worth noting that our data-level approaches neither modify model
architecture nor add extra training loss, thus do not increase any latency
(“Speed”), maintaining the intrinsic advantages of non-autoregressive
generation. We must admit that our strategy indeed increase the amount of
computing resources due to that we should train $f_{t\mapsto s}$ AT teachers
for building $\overleftarrow{\text{KD}}$ data.

#### Results on Other Language Pairs

Table 4 lists the results of NAT models on Zh-En and Ja-En language pairs,
which belong to different language families (i.e. Indo-European, Sino-Tibetan
and Japonic). Compared with baselines, our method significantly and
incrementally improves the translation quality in all cases. For Zh-En, LFR
achieves on average +0.8 BLEU improvement over the traditional training, along
with increasing on average +3.0% accuracy on low-frequency word translation.
For long-distance language pair Ja-En, our method still improves the NAT model
by on average +0.7 BLEU point with on average +2.2 ALF. Furthermore, NAT
models with the proposed training strategy perform closely to their AT
teachers (i.e. 0.2 $\Delta$BLEU). This shows the effectiveness and
universality of our method across language pairs.

Model | Law | Med. | IT | Kor. | Sub.
---|---|---|---|---|---
AT | 41.5 | 30.8 | 27.5 | 8.6 | 15.4
MaskT | 37.3 | 28.2 | 24.6 | 7.3 | 11.2
+LFR | 38.1† | 28.8 | 25.4† | 8.9† | 14.3†
|  |  |  |  |
LevT | 37.5 | 28.4 | 24.7 | 7.5 | 12.4
+LFR | 38.5† | 29.4† | 25.9† | 8.4† | 14.5†
Table 5: Performance on domain shift setting. Models are trained on WMT14 En-
De news domain but evaluated on out-of-domain test sets, including law,
medicine, IT, koran and subtitle. “†” indicates statistically significant
difference ($p<0.05$) from corresponding baselines.

#### Results on Domain Shift Scenario

The lexical choice must be informed by linguistic knowledge of how the
translation model’s input data maps onto words in the target domain. Since
low-frequency words get lost in traditional NAT models, the problem of lexical
choice is more severe under domain shift scenario (i.e. models are trained on
one domain but tested on other domains). Thus, we conduct evaluation on WMT14
En-De models over five out-of-domain test sets Müller et al. (2020), including
law, medicine, IT, Koran and movie subtitle domains. As shown in Table 5,
standard NAT models suffer large performance drops in terms of BLEU score
(i.e. on average -2.9 BLEU over AT model). By observing these outputs, we
found a large amount of translation errors on low-frequency words, most of
which are domain-specific terminologies. In contrast, our approach improves
translation quality (i.e. on average -1.4 BLEU over AT model) by rejuvenating
low-frequency words to a certain extent, showing that LFR increases the domain
robustness of NAT models.

#### Results on Different Data Scales

Model | BLEU
---|---
1.0M | 4.5M | 36.0M
AT | 25.5 | 37.6 | 40.2
MaskT | 23.7 | 35.4 | 36.8
+LFR | 24.3† | 36.2† | 37.7†
Table 6: Performance on different scale of training data. The small and medium
datasets are sampled from the large WMT19 En-De dataset, and evaluations are
conducted on the same testset. “†” indicates statistically significant
difference ($p<0.05$) from corresponding baselines.

To confirm the effectiveness of our method across different data sizes, we
further experiment on three En-De datasets at different scale. The small- and
medium-scale training data are randomly sampled from WM19 En-De corpus,
containing about 1.0M and 4.5M sentence pairs, respectively. The large-scale
one is collected from WMT19, which consists of 36M sentence pairs. We report
the BLEU scores on same testset newstest2019 for fair comparison. We employs
base model to train the small-scale AT teacher, and big model with large batch
strategy (i.e. 458K tokens/batch) to build the AT teachers for medium- and
large-scale. As seen in Table 6, our simple training recipe boost performances
for NAT models across different size of datasets, especially on large scale
(+0.9), showing the robustness and effectiveness of our approach.

#### Complementary to Related Work

Ding et al. (2021b) is relevant to our work, which introduced an extra
bilingual data-dependent prior objective to augment NAT models the ability to
learn low-frequency words in raw data. Our method is complementary to theirs
due to that we only change data and training strategies (model-agnostic). As
shown in Table 7, two approaches yield comparable performance in terms of BLEU
and ALF. Besides, combination can further improve BLEU as well as ALF scores
(i.e. +0.3 and +0.6). This illustrates the complementarity of model-level and
data-level approaches on rejuvenating low-frequency knowldege for NAT models.

###  3.3 Analysis

We conducted extensive analyses to better understand our approach. All results
are reported on the Mask-Predict models.

#### Accuracy of Lexical Choice

To understand where the performance gains come from, we conduct fine-grained
analysis on lexical choice. We divide “All” tokens into three categories based
on their frequency, including “High”, “Medium” and “Low”. Following Ding et
al. (2021b), we measure the accuracy of lexical choice on different frequency
of words. Table 8 shows the results. Takeaway: The majority of improvements on
translation accuracy is from the low-frequency words, confirming our
hypothesis.

Model | BLEU | ALF
---|---|---
Mask-Predict | 27.0 | 68.4
+Raw Data Prior | 27.8 | 72.4
+Low-Frequency | 27.8 | 72.3
+Combination | 28.1 | 72.9
Table 7: Complementary to other work. “Combination” indicates combining “+Raw Data Prior” proposed by Ding et al. (2021b) with our “+Low-Frequency”. Experiments are conducted on WMT14 En-De. Model | En-De | Zh-En | Ja-En
---|---|---|---
All | High | Med. | Low | All | High | Med. | Low | All | High | Med. | Low
MaskT (Raw) | 74.3 | 75.9 | 74.6 | 72.5 | 68.5 | 71.5 | 68.3 | 65.1 | 73.1 | 75.5 | 74.7 | 69.1
MaskT (KD) | 76.3 | 82.4 | 78.3 | 68.4 | 72.7 | 81.4 | 75.2 | 61.5 | 75.3 | 82.8 | 76.3 | 66.9
+Raw-Pretrain | 77.7 | 83.1 | 78.4 | 71.9 | 73.4 | 81.6 | 75.3 | 64.1 | 76.1 | 83.4 | 76.7 | 68.3
+Bi-Distillation | 77.9 | 83.1 | 78.5 | 72.3 | 73.7 | 81.7 | 75.3 | 64.8 | 76.5 | 83.5 | 76.7 | 68.9
Table 8: Analysis on different frequency words in terms of accuracy of lexical
choice. We split “All” words into “High”, “Medium” and “Low” categories.
Shades of cell color represent differences between ours and KD.

#### Low-Frequency Words in Output

We expect to recall more low-frequency words in translation output. As shown
in Table 9, we calculate the ratio of low-frequency words in generated
sentences. As seen, KD biases the NAT model towards generating high-frequency
tokens (Low freq.$\downarrow$) while our method can not only correct this bias
(on average +18% and +26% relative changes for +raw-pretrain and +Bi-
distillation), but also enhance translation (BLEU$\uparrow$ in Table 4).
Takeaway: Our method generates translations that contain more low-frequency
words.

Model | En-De | Zh-En | Ja-En
---|---|---|---
MaskT (Raw) | 10.3% | 6.7% | 9.4%
MaskT (KD) | 7.6% | 4.2% | 6.9%
+Raw-Pretrain | 9.3% | 5.6% | 8.4%
+Bi-Distillation |  9.7% |  6.8% |  8.7%
Table 9: Ratio of low-frequency target words in output. # | Strategy | BLEU | ALF
---|---|---|---
1 | Raw | 24.1 | 69.3
2 | $\overrightarrow{\text{KD}}$ | 25.4 | 66.4
3 | Raw+$\overrightarrow{\text{KD}}$ | 25.6 | 67.7
4 | Raw$\rightarrow$$\overrightarrow{\text{KD}}$ | 25.9 | 68.2
|  |  |
5 | Raw+$\overleftarrow{\text{KD}}$+$\overrightarrow{\text{KD}}$ | 25.7 | 67.9
6 | Raw$\rightarrow$$\overleftarrow{\text{KD}}$+$\overrightarrow{\text{KD}}$ | 25.7 | 68.3
7 | Raw$\rightarrow$$\overleftarrow{\text{KD}}$+$\overrightarrow{\text{KD}}$$\rightarrow$$\overrightarrow{\text{KD}}$ | 26.3 | 69.5
Table 10: Performances of different strategies. The models are trained and
tested on WMT14 En-De. “A+B” means concatenate A and B while “A$\rightarrow$B”
indicates pretraining on A and then finetuning on B.

#### Effects of Variant Training Strategies

As discussed in §2.4, we carefully investigate alternative training approaches
in Table 10. We make the total training step identical to that of vanilla NAT
models, and report both BLEU and ALF scores. As seen, all variant strategies
perform better than the standard KD method in terms both BLEU and ALF scores,
confirming the necessity of our work. Takeaway: 1) Pretraining is more
effective than combination on utilizing data manipulation strategies; 2) raw
data and bidirectional distilled data are complementary to each other; 3) it
is indispensable to finetune models on $\overrightarrow{\text{KD}}$ in the
last stage.

Model | All | High | Med. | Low
---|---|---|---|---
Training on Raw Data
AT-Teacher | 79.3 | 84.7 | 80.2 | 73.0
AT-Student | 76.8 | 80.2 | 77.4 | 72.8
Training on Distilled Data
AT-Student | 77.3 | 82.5 | 78.6 | 70.9
+LFT | 78.1 | 83.2 | 78.7 | 72.5
Table 11: Analysis on AT models in term of the accuracy of lexical choice on
WMT14 En-De. We split “All” words into “High”, “Medium” and “Low” categories.

#### Our Approach Works for AT Models

Although our work is designed for NAT models, we also investigated whether our
LFT method works for general cases, e.g. autoregressive models. We used
Transformer-Big as the teacher model. For fair comparison, we leverage the
Transformer-Base as the student model, which shares the same model capacity
with NAT student (i.e. MaskT). The result lists in Table 11. As seen, AT
models also suffer from the problem of low-frequency words when using
knowledge distillation, and our approach also works for them. Takeaway: Our
method works well for general cases through rejuvenating more low-frequency
words.

##  4 Related Work

#### Low-Frequency Words

Benefiting from continuous representation learned from the training data, NMT
models have shown the promising performance. However, Koehn and Knowles (2017)
point that low-frequency words translation is still one of the key challenges
for NMT according to the Zipf’s law Zipf (1949). For AT models, Arthur et al.
(2016) address this problem by integrating a count-based lexicon, and Nguyen
and Chiang (2018) propose an additional lexical model, which is jointly
trained with the AT model. Recently, Gu et al. (2020) adaptively re-weight the
rare words during training. The lexical choice problem is more serious for NAT
models, since 1) the lexical choice errors (low-resource words in particular)
of AT distillation will propagate to NAT models; and 2) NAT lacks target-side
dependencies thus misses necessary target-side context. In this work, we
alleviate this problem by solving the first challenge.

#### Data Manipulation

Our work is related to previous studies on manipulating training data for NMT.
Bogoychev and Sennrich (2019) show that forward- and backward-translations
(FT/ BT) could both boost the model performances, where FT plays the role of
domain adaptation and BT makes the translation fluent. Fadaee and Monz (2018)
sample the monolingual data with more difficult words (e.g. rare words) to
perform BT, achieving significant improvements compared with randomly sampled
BT. Nguyen et al. (2020) diversify the data by applying FT and BT multiply
times. However, different from AT, the prerequisite of training a well-
performed NAT model is to perform KD. We compared with related works in Table
10 and found that our approach consistently outperforms them. Note that all
the ablation studies focus on exploiting the parallel data without augmenting
additional data.

#### Non-Autoregressive Translation

A variety of approaches have been exploited to bridge the performance gap
between NAT and AT models. Some researchers proposed new model architectures
(Lee et al., 2018; Ghazvininejad et al., 2019; Gu et al., 2019; Kasai et al.,
2020), aided with additional signals (Wang et al., 2019; Ran et al., 2019;
Ding et al., 2020), introduced sequential information (Wei et al., 2019; Shao
et al., 2019; Guo et al., 2020; Hao et al., 2021), and explored advanced
training objectives (Ghazvininejad et al., 2020; Du et al., 2021). Our work is
close to the research line on training methods. Ding et al. (2021b) revealed
the low-frequency word problem in distilled training data, and introduced an
extra Kullback-Leibler divergence term derived by comparing the lexical choice
of NAT model and that embedded in the raw data. Ding et al. (2021a) propose a
simple and effective training strategy, which progressively feeds different
granularity of data into NAT models by leveraging curriculum learning.

##  5 Conclusion

In this study, we propose simple and effective training strategies to
rejuvenate the low-frequency information in the raw data. Experiments show
that our approach consistently and significantly improves translation
performance across language pairs and model architectures. Notably, domain
shift is an extreme scenario to diagnose low-frequency translation, and our
method significant improves them. Extensive analyses reveal that our method
improves the accuracy of lexical choices for low-frequency source words,
recalling more low-frequency words in translations as well, which confirms our
claim.

## Acknowledgments

We are grateful to the anonymous reviewers and the area chair for their
insightful comments and suggestions. Xuebo Liu and Derek F. Wong were
supported in part by the Science and Technology Development Fund, Macau SAR
(Grant No. 0101/2019/A2), and the Multi-year Research Grant from the
University of Macau (Grant No. MYRG2020-00054-FST).

## References

  * Arthur et al. (2016) Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016.  Incorporating discrete translation lexicons into neural machine translation.  In _EMNLP_.
  * Bogoychev and Sennrich (2019) Nikolay Bogoychev and Rico Sennrich. 2019.  Domain, translationese and noise in synthetic data for neural machine translation.  _ArXiv_.
  * Collins et al. (2005) Michael Collins, Philipp Koehn, and Ivona Kučerová. 2005.  Clause restructuring for statistical machine translation.  In _ACL_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_.
  * Ding et al. (2021a) Liang Ding, Longyue Wang, Xuebo Liu, Derek F Wong, Dacheng Tao, and Zhaopeng Tu. 2021a.  Progressive multi-granularity training for non-autoregressive translation.  In _ACL_.
  * Ding et al. (2021b) Liang Ding, Longyue Wang, Xuebo Liu, Derek F. Wong, Dacheng Tao, and Zhaopeng Tu. 2021b.  Understanding and improving lexical choice in non-autoregressive translation.  In _ICLR_.
  * Ding et al. (2020) Liang Ding, Longyue Wang, Di Wu, Dacheng Tao, and Zhaopeng Tu. 2020.  Context-aware cross-attention for non-autoregressive translation.  In _COLING_.
  * Du et al. (2021) Cunxiao Du, Zhaopeng Tu, and Jing Jiang. 2021.  Order-agnostic cross entropy for non-autoregressive machine translation.  In _ICML_.
  * Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013.  A simple, fast, and effective reparameterization of ibm model 2.  In _NAACL_.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  In _EMNLP_.
  * Fadaee and Monz (2018) Marzieh Fadaee and Christof Monz. 2018.  Back-translation sampling by targeting difficult words in neural machine translation.  In _EMNLP_.
  * Gao et al. (2018) Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2018.  Representation degeneration problem in training natural language generation models.  In _ICLR_.
  * Ghazvininejad et al. (2020) Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020\.  Aligned cross entropy for non-autoregressive machine translation.  In _ICML_.
  * Ghazvininejad et al. (2019) Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019.  Mask-predict: Parallel decoding of conditional masked language models.  In _EMNLP_.
  * Gong et al. (2018) Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2018.  Frage: Frequency-agnostic word representation.  _NeurIPS_.
  * Gu et al. (2018) Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018\.  Non-autoregressive neural machine translation.  In _ICLR_.
  * Gu et al. (2019) Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019.  Levenshtein transformer.  In _NeurIPS_.
  * Gu et al. (2020) Shuhao Gu, Jinchao Zhang, Fandong Meng, Yang Feng, Wanying Xie, Jie Zhou, and Dong Yu. 2020.  Token-level adaptive training for neural machine translation.  In _EMNLP_.
  * Guo et al. (2020) Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2020.  Fine-tuning by curriculum learning for non-autoregressive neural machine translation.  In _AAAI_.
  * Hao et al. (2021) Yongchang Hao, Shilin He, Wenxiang Jiao, Zhaopeng Tu, Michael Lyu, and Xing Wang. 2021.  Multi-task learning with shared encoder for non-autoregressive machine translation.  In _NAACL_.
  * Hassan et al. (2018) Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018.  Achieving human parity on automatic chinese to english news translation.  _arXiv_.
  * Hendrycks et al. (2019) Dan Hendrycks, Kimin Lee, and Mantas Mazeika. 2019.  Using pre-training can improve model robustness and uncertainty.  In _ICML_.
  * Jiao et al. (2020) Wenxiang Jiao, Xing Wang, Shilin He, Irwin King, Michael R. Lyu, and Zhaopeng Tu. 2020.  Data rejuvenation: Exploiting inactive training examples for neural machine translation.  In _EMNLP_.
  * Kasai et al. (2020) Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. 2020.  Parallel machine translation with disentangled context transformer.  In _arXiv_.
  * Kim and Rush (2016) Yoon Kim and Alexander M Rush. 2016.  Sequence-level knowledge distillation.  In _EMNLP_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six challenges for neural machine translation.  In _WMT_.
  * Lee et al. (2018) Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018.  Deterministic non-autoregressive neural sequence modeling by iterative refinement.  In _EMNLP_.
  * Mathis et al. (2021) Alexander Mathis, Thomas Biasi, Steffen Schneider, Mert Yuksekgonul, Byron Rogers, Matthias Bethge, and Mackenzie W Mathis. 2021.  Pretraining boosts out-of-domain robustness for pose estimation.  In _WACV_.
  * Morishita et al. (2017) Makoto Morishita, Jun Suzuki, and Masaaki Nagata. 2017.  Ntt neural machine translation systems at wat 2017.  In _IJCNLP_.
  * Müller et al. (2020) Mathias Müller, Annette Rios, and Rico Sennrich. 2020.  Domain Robustness in Neural Machine Translation.  In _AMTA_.
  * Nguyen and Chiang (2018) Toan Nguyen and David Chiang. 2018.  Improving lexical choice in neural machine translation.  In _NAACL_.
  * Nguyen et al. (2020) Xuan-Phi Nguyen, Joty Shafiq, Kui Wu, and Ai Ti Aw. 2020.  Data diversification: A simple strategy for neural machine translation.  In _NeurIPS_.
  * Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003.  A systematic comparison of various statistical alignment models.  _Computational linguistics_.
  * Ott et al. (2018) Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018.  Analyzing uncertainty in neural machine translation.  In _ICML_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _ACL_.
  * Platanios et al. (2019) Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019.  Competence-based curriculum learning for neural machine translation.  In _NAACL_.
  * Ran et al. (2019) Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. 2019.  Guiding non-autoregressive neural machine translation decoding with reordering information.  _arXiv_.
  * Ren et al. (2020) Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, and Tie-Yan Liu. 2020.  A study of non-autoregressive model for sequence generation.  In _ACL_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _ACL_.
  * Shao et al. (2019) Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. 2019.  Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation.  In _AAAI_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NeurIPS_.
  * Wang et al. (2019) Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019.  Non-autoregressive machine translation with auxiliary regularization.  In _AAAI_.
  * Wei et al. (2019) Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019.  Imitation learning for non-autoregressive neural machine translation.  In _ACL_.
  * Wu et al. (2019) Lijun Wu, Yiren Wang, Yingce Xia, QIN Tao, Jianhuang Lai, and Tie-Yan Liu. 2019\.  Exploiting monolingual data at scale for neural machine translation.  In _EMNLP_.
  * Zhou et al. (2020) Chunting Zhou, Graham Neubig, and Jiatao Gu. 2020.  Understanding knowledge distillation in non-autoregressive machine translation.  In _ICLR_.
  * Zipf (1949) George K. Zipf. 1949.  Human behavior and the principle of least effort.