# Structural Dynamics of Contractile Injection Systems

Noah Toyonaga1, L. Mahadevan1,2 1Department of Physics, Harvard University,
Cambridge, MA 02138, USA
2 Department of Organismic and Evolutionary Biology, and School of Engineering
and Applied Sciences, Harvard University, Cambridge, MA 02138, USA

###### Abstract

The dynamics of many macromolecular machines is characterized by chemically-
mediated structural changes that achieve large scale functional deployment
through local rearrangements of constitutive protein sub-units. Motivated by
recent high resolution structural microscopy of a particular class of such
machines, contractile injection systems (CIS), we construct a coarse grained
semi-analytical model that recapitulates the geometry and bistable mechanics
of CIS in terms of a minimal set of measurable physical parameters. We use
this model to predict the size, shape and speed of a dynamical actuation front
that underlies contraction. Scaling laws for the velocity and physical
extension of the contraction front are consistent with our numerical
simulations, and may be generally applicable to related systems.

Macromolecular assemblies of biopolymers made of simple repeating subunits are
ubiquitious in molecular biology, serving as the building blocks of all
biological matter from the cytoskeleton to multi-cellular tissues. Such
assemblies function as reservoirs of information, pathways for transport of
matter, and machines that can transform chemical energy into mechanical work
[1]. In this last role, conformational changes at the level of a constitutive
subunit can be used to both regulate (slow) assembly and (fast) deployment in
which bending and twisting at small scales is amplified geometrically to
shrinkage and extension of the global structure. Prominent examples of these
biological engines include polymorphic bacterial flagella, which transition
between multiple shapes reversibly [2, 3, 4], viral capsids, where self
assembly is regulate by a conformational change in the monomer catalyzed by
the appropriate growth site [5], acrosomal reactions in marine invertebrates
such as the horse-shoe crab wherein an initially twisted actin bundle
straightens in the presence of $\text{Ca}^{2+}$ [6], and polymorphic
transitions in a variety of cytoskeletal assemblies [7, 8] etc.

Recently, there has been much interest in a particular class of macromolecular
machines known as Contractile Injection Systems which includes Myoviridae
bacteriophage tails (T4), R-type Pyocins, Type VI Secretion Systems (T6SS),
and phage-like protein-translocation-structures (PLTS) [9, 10, 11, 12, 13, 14,
15]. These assemblies function as spring-loaded needle mechanisms that can be
deployed by their host organism to puncture membranes, and serve as the
physical pathway for infection.

All CIS consist of three components—a bistable sheath made of interlaced
protein strands, a hollow tube, and a baseplate. Prior to deployment the
hollow tube is contained within the sheath and the combined assembly is capped
by the baseplate. In-vivo the baseplate mediates interactions with target
substrates (e.g. the membrane of a target cell): upon binding to the target,
it dramatically changes conformation [16, 12], thereby initiating the global
contraction of the sheath. In turn, the sheath pushes the tube out through a
hole in the baseplate with sufficient force to puncture the target membrane,
as shown schematically in Fig. 1(a-b). In this work, we are concerned with the
dynamics of the sheath contraction process; that is, how a local
reconfiguration of the sheath proteins (initiated by baseplate reconfiguration
on adhesion with a host) propagates through the sheath system.

High resolution structural microscopy [14, 13] reveals the nature of chemical
interactions between the strands which constitute the sheath, and which are
critical for both the assembly and controlled deployment of the CIS. These
structural studies indicate that variations in the non-covalent bonding
between strands (on the order of 10 kcal/mol per strand subunit) can serve to
both stabilize the CIS in a high energy metastable state and also provides the
energy to drive contraction [10, 17]. Thus, the reconfiguration of strands
during contraction gives rise to a surplus free energy due to changes in
surface bonding which in turn drives deployment via a self-propagating
process[18]. These computational results are also consistent with earlier
measurements of the free energy released during deployment in in-vitro
experiments [19].

It is useful to note that contraction of the sheath can also be triggered in-
vitro by treatment of CIS sheaths with urea [19, 20], elevated temperature
[19], acid [16], and cationic degergents [21], all of which are hypothesized
to disturb the aforementioned strand-strand bonding [17, 22].

The Angstrom-level atomic resolution structures of CIS in the extended and
contracted states naturally raises the question about the dynamical principles
underlying their deployment. Recent works have sought to understand these
dynamics using large-scale molecular computational methods [23] but have been
hampered by the large physical scale of the structure (with molecular weights
of several megadaltons), and the concomitant global nature of the deformation.
Coarse-grained dynamical models [24] represent the CIS sheath as a set of
elastic strands interacting with an ambient fluid but do not account for the
metastability of the CIS and therefore only describe the over-damped
relaxation of the system [25]. (For a survey of related prior work see SI)
Thus, even basic questions such as whether the contraction proceeds uniformly
everywhere or propagates like a wave through the structure remain unanswered
[25, 24].

Here we develop a coarse-grained dynamical model for the extrusion of CIS
based on structural observations by 1) parameterizing the geometry of the CIS
using a Chebyshev net (discussed in detail below), which allows us to project
the dynamics of the full three dimensional structure onto a scalar order
parameter which measures the local deformation, and 2) accounting for the
(surface-bonding induced) meta-stability of the CIS using a bistable potential
whose free parameters are fixed by experimental measurements. The resulting
framework provides scaling predictions for the speed and nature of the
dynamics of the CIS contraction process while elucidating general principles
for the dynamics of similar macromolecular assemblies.

Figure 1: CIS atomic structure and coarse-grained model: (a) The tail of a T4
bacteriophage is an example of a contractile injection system which is used to
infect target cells. (b) The protein sub-units which constitute a CIS are
interlinked to form a lattice. This motivates us to model the geometry of an
arbitrary CIS sheaths as an axisymmetric surface $\Phi(u,v)$ whose coordinate
curves ({$\partial_{u}\Phi$, $\partial_{v}\Phi$}, highlighted in green and
blue) are parallel to the protein strands of the modeled structure. The
opening angle between strands is the dynamical order parameter in our model
which we denote by $\phi(s,t)$. The extended and contracted configurations of
the proposed continuum model correspond to constant values of the order
parameter $\phi$: $\phi(s,t)=\phi_{\text{e}}$ and $\phi(s,t)=\phi_{\text{c}}$,
respectively. (c) A representative contraction front represents a transitional
region between two domains of stable states, $\phi_{\text{e}}$ above and
$\phi_{\text{c}}$ below. (d) The 3D geometry of the sheath depends on the
angle $\phi$ between the strand axes $\\{{u},{v}\\}$. We work in a coordinate
system $\\{{s},{p}\\}$ defined relative to the strand axes as shown in the
zoom-in and described in the SI. (e) Plot of the order parameter
$\phi(s,t=t^{\prime})$ corresponding to the CIS geometry shown in (c). The
dotted vertical lines at $\phi_{\text{e}}$, $\phi_{\text{c}}$ correspond to
the initial and final states of the CIS (shown in (b)). Note the horizontal
axis here is in terms of the material coordinate $s$ rather than $z$, the
vertical coordinate of the embedded geometry (shown in (c)).

The conserved sheath structure of the CIS consists of identical protein sub-
units which are linked in a mesh of helical strands which are connected to one
another, akin to the warp and weft fibers in a woven fabric [17, 10, 14, 13,
5]. This motivates our coarse-grained model for the CIS sheath geometry as a
two dimensional surface $\Phi(u,v)$ whose local tangent vectors
($\partial_{u}\Phi,\partial_{v}\Phi$) are everywhere parallel to the strands
of the corresponding CIS structure. Consistent with structural information
[10], we assume that the relative distance between sub-units are fixed by
inextensible linkages, but remain free to rotate about one another, allowing
us to parameterize the kinematics of deformation by the angle between the
coordinate vectors in the embedded configuration. This allows us to describe
the local conformation of the CIS in terms of a single quantitiy, the shearing
angle $\phi\equiv\phi(s,p,t)$ between strands as a function of the axial
material coordinate $s$ and an azimuthal material coordinate $p$ and time $t$
(see Fig. 1.d) and serves as a scalar order parameter for the transition
between the contracted and extended states.

If we assume that the surface is axisymmetric then the angle $\phi=\phi(s,t)$
is independent of $p$, and at any instant of time $t$ we can construct the
following explicit embedding for $\Phi$ parameterized by $\phi$ (see SI for
details):

$\Phi(s,p,t)=\begin{bmatrix}r(s,t)\cos(p/r_{0}))\\\ r(s,t)\sin(p/r_{0}))\\\
z(s)\end{bmatrix}$ (1a) $r(s,t)=r_{0}\sin(\phi(s,t)/2)$ (1b)
$z(s,t)=\int_{0}^{s}\sqrt{\cos^{2}\left(\frac{\phi\left(s^{\prime},t\right)}{2}\right)-\left(\frac{dr}{ds}\big{|}_{s^{\prime}}\right)^{2}}\,ds^{\prime}$
(1c)

Here $r_{0}$ in Eq. 1b is fixed by the radius of the CIS. When the CIS is its
initial and final cylindrical states the shearing order parameter $\phi(s,t)$
takes the constant value $\phi(s,t)=\phi_{\text{e}},\phi_{\text{c}}$
respectively and the corresponding surface $\Phi$ is a cylinder of radii
$r=r_{0}\sin(\phi_{\text{e}}/2),r=r_{0}\sin(\phi_{\text{c}}/2)$ with helical
coordinate curves as shown in Fig. 1.b. Structural data allows us to
explicitly assign the initial and final geometric states in our model: e.g.
given the radius $r_{\text{c}}$ and shearing angle $\phi_{\text{c}}$ of a CIS
in the contracted state, it follows that
$r_{0}\equiv\frac{r_{\text{c}}}{\sin(\phi_{\text{c}}/2)}$ [13, 14].

To move towards the energetics and dynamics of the CIS parameterized by the
geometry of the shearing order parameter $\phi(s,t)$, we must account for both
1) mechanical stresses in the strands induced in the sub-unit strands as they
are deformed from the initial to final configurations and 2) changes in the
chemical potential due to the formation and breakage of inter-molecular bonds
between sub-units as they change conformation. We model the strands of the
sheath as anisotropic elastic filaments that can bend only in one direction,
with a local energy density $EI\kappa(\phi)^{2}/2$ where $E$ is the Young’s
modulus, $I$ is the second moment of area, and $\kappa(\phi)$ is the filament
curvature. Modeling the chemical energy associated with the changes in the
shear angle via potential $U(\phi)$ (discussed below), leads to an expression
for the total energy of the CIS given by:

$F[\phi]=\int
ds\bigg{[}\underbrace{EI\kappa(\phi)^{2}}_{\text{Bending}}+\underbrace{U(\phi)}_{\text{Shear}}\bigg{]}$
(2)

To characterize the terms in Eq. 2), we first note that the strands of the CIS
lie along the material curves $u$ and $v$ of the map $\Phi(u,v)$, and
primarily bend out of plane but are stiff in-plane (i.e. mechanically they are
ribbon-like). Then, the curvature of each rod is equivalent to the normal
curvature of the surface $\Phi$ parallel to these material curves given by
(see §LABEL:SI:curvature for details). If the shear order parameter
$\phi(s,t)$ is slowly varying (i.e.
$\left(\frac{d\phi}{ds}\right)^{2}<\left(\frac{1}{2r_{0}}\right)^{2}$), we can
drop higher order terms in $\phi^{\prime}\equiv\partial\phi(s,t)/\partial s$,
leading to a simplified expression for the curvature

$\kappa(\phi)=\underbrace{-\frac{\sin\left(\frac{\phi(s,t)}{2}\right)}{r_{0}}}_{\text{helix
curvature}}+\underbrace{\frac{1}{2}r_{0}\phi^{\prime\prime}(s,t)\cos\left(\frac{\phi(s,t)}{2}\right)}_{\text{gradient
contribution}}$ (3)

which is the form we use 111As previously discussed the initial and final
stable configurations of the CIS sheath corresponds to helices (characterized
by the opening angle $\phi_{i}$, $\phi_{f}$); thus, the gradient contribution
will only be active in the region of the shear wave front. Given the radius of
the CIS $\approx\qty{100}{}$, our assumption of a slowly-varying shear angle
$\phi(s,t)$ corresponds to a contraction wave of length $\approx\qty{50}{}$,
or $1/20$ the length of the structure ($\approx\qty{1000}{}$), and is
consistent with experimental observations of the assembly in its contracted
and expanded states[26]., and which can be interpreted as a sum of two terms,
associated with the curvature of a helix of constant radius and a higher order
gradient correction.

We now turn to the chemical free energy in Eq. 2 associated with the changes
in the non-covalent bonding between strands in the CIS sheath that gives rise
to a bistability of the structure. We model this with a simple double well
potential with units of energy density () illustrated in Fig. 2.a, and given
by

$\begin{split}U(\phi)=&b\left(a\left(c\phi-\phi_{0}\right)-\frac{1}{2}\left(c\phi-\phi_{0}\right)^{2}-\right.\\\
&\left.\frac{a}{3}\left(c\phi-\phi_{0}\right)^{3}+\frac{1}{4}\left(c\phi-\phi_{0}\right)^{4}\right)\end{split}$
(4)

We note that Eq. 4 is the simplest form which captures the essential chemical-
mechanics of CIS and is uniquely determined from structural and calorimetric
experiments of CIS [10, 26, 23] with no free parameters 222 A brief summary of
our procedure for fitting the parameters of $U(\phi)$ to experimental data is
provided here, while a longer discussion is in
§LABEL:SI:model_free_parameters. We note that the parameters of Eq. 4
($a,b,c,\phi_{0}$) set the location of the two minima of $U$
($\phi_{\text{e}}$, $\phi_{\text{c}}$), the potential difference between the
minima ($U_{h}$), and the height of the potential barrier dividing them
($U_{a}$) as shown in Fig. 2.a. We assign the minima of $U$ to
$\phi_{\text{e}}$ and $\phi_{\text{c}}$ so that the dynamics of $\phi(s,t)$
correspond to flow of the system from the extended, high energy metastable
state ($\phi_{\text{e}}$) to the contracted, low energy state
($\phi_{\text{c}}$) (see Fig. 2.a, inset). Similarly, we assign $U_{h}$ to be
the enthalpy density and $U_{a}$ corresponds to the free energy density
necessary to initiate contraction, both of which can be measured from
calorimetry or estimated from structural models. Here, $U_{a}$ and $U_{h}$
have units of , since the total energy of the system is integrated over the
length of the CIS (see Eq. 2). For the simulations presented in this work we
take $\phi_{\text{e}}=\qty{1}{}$, $\phi_{\text{c}}=\qty{2.4}{}$ and vary
$\qty{1e-11}{\per}<U_{h}<\qty{4e-11}{\per}$ and
$\qty{1e-11}{\per}<U_{a}<\qty{4e-11}{\per}$, corresponding to the geometry and
energetics of the R2-type Pyocin [10, 26, 23]. .

Figure 2: Propagating front of conformational change: (a) The quartic
potential $U(\phi)$ (defined by Eq. 4) drives the evolution of the shearing
order parameter $\phi(s,t)$. The 2D plot shows the geometric
($\phi_{\text{e}}$, $\phi_{\text{c}}$) and energetic quantities ($U_{h}$,
$U_{a}$) we use to fix the free parameters of $U$ (Eq. 4) based on
experimental measurements (see footnote 2 for description). The inset 3D plot
shows a static snapshot of the shearing order parameter $\phi(t=t^{*},s)$
(drawn in orange) superimposed on the potential, which is constant along the
axial direction, $s$. (b) The shearing order parameter $\phi(s,t)$ from the
simulation described in the main text and indicated by a star in Fig. 3
plotted on the $t,s$ plane. Color corresponds to the value of $\phi$ in
radians. The diagonal band corresponds to the front moving at a constant speed
speed V and with a characteristic width $W$. (c) Plot of $\phi(t=t^{*},s)$
taken at time steps
$t^{*}\in\\{\qty{2}{\micro},\qty{15}{\micro},\qty{28}{\micro}\\}$
(corresponding to vertical highlights in (b)). Again, we observe the geometry
of the front is constant during propagation (i.e. the different lines are
identical up to translation). We characterize the shape of the front by
measuring the width $W$, and leading and trailing ridge widths ($w_{l}$,
$w_{t}$). (d) The three dimensional geometry of the front (as a segment of the
CIS) corresponding to $\phi(s,t)$ at $t=2,15,28\unit{us}$ (these snapshots
correspond to the same solutions highlighted in (b) and (c)).

To derive equations of motion for the CIS from the free energy given in Eq. 2,
we note that the small-scale nature of the CIS (10s of ) suggests that its
dynamical evolution follows from a balance between the internally stored
energy and viscous dissipation. Balancing local torque densities implies that
$\eta\frac{\partial\phi(s,t)}{\partial t}=-\frac{\delta F}{\delta\phi}$, where
$\eta$ is the effective friction factor with dimensions of , leading to the
following equation of motion for the evolution of the shear order parameter
$\phi(s,t)$ (details in SI):

$\begin{split}\eta\frac{\partial\phi}{\partial
t}=&-EI\bigg{(}\frac{\sin(\phi)}{2r_{0}^{2}}+\left(\frac{1}{2}\sin\left(\phi\right)\left(\phi^{\prime}\right)^{2}-\cos\left(\phi\right)\phi^{\prime\prime}\right)+\\\
&\frac{r_{0}^{2}}{8}\left(-2\cos(\phi)\left(\phi^{\prime}\right)^{2}\phi^{\prime\prime}-3\sin(\phi)\left(\phi^{\prime\prime}\right)^{2}\right.\\\
&\left.-4\sin(\phi)\phi^{\prime}\phi^{\prime\prime\prime}+2\cos(\phi)\phi^{\prime\prime\prime\prime}+2\phi^{\prime\prime\prime\prime}\right)\bigg{)}-f(\phi)\end{split}$
(5)

where $f(\phi)\equiv\frac{\partial U}{\partial\phi}$.

We observe that our derivation of Eq. 5 follows from the assumption that local
shear rates $\phi_{t}$ are more important than their spatial derivatives
$\propto\frac{\partial^{2}\phi}{\partial t\partial s}$. The latter choice
would not give rise to a constant velocity front solution (physically the CIS
would, in general, fail to fully actuate). We also note out that the
formulation here is equivalent to that used to explain the dynamics of other
macromolecular assemblies such as the acrosome reaction [6], where the
dynamical equations take the form $\frac{\partial^{2}\phi}{\partial t\partial
s}=-\frac{\partial}{\partial s}(\frac{\delta F}{\delta\phi})$, but for a
different form of the free energy (as a function of the twist strain), which
on integration yields the same form as here.

To follow the dynamics of the contraction process, we search for solutions of
the high-order nonlinear PDE (Eq. 5) that connect the stationary solutions
(i.e. solutions in which $\phi$ transitions from the extended state
$\phi(-\infty,t)-\phi_{\text{e}}=\phi^{\prime}(\infty,t)=0$ to the contracted
state $\phi(\infty,t)-\phi_{\text{c}}=\phi^{\prime}(\infty,t)=0$). We use a
Chebyshev spectral method [27, 28] to solve (Eq. 5), with an initial condition
corresponding to a step function convolved with a Gaussian with width $\sigma$
of the same order as $r_{b}$. This initial condition physically corresponds to
a state where one end of the CIS has been forced into its collapsed, low
energy state; in vivo, this maps to the initial reconfiguration of the lower
sheath sub-units by the base plate upon binding to a target substrate [26,
15].

In Fig. 2.b, we show the results of a representative simulation and observe
that the order parameter $\phi(s,t)$ settles into a steadily propagating front
with a constant velocity V and width W (defined as the distance between the
minimum and maximum values of $\phi$ at a fixed time). We also observe that
the front is preceded/succeeded by asymmetric oscillatory ridges when
$\phi<\phi_{\text{e}}$ and $\phi>\phi_{\text{c}}$, as expected from the
asymptotic behavior of these high-order PDEs. We define the trailing width
$w_{l}$ as the full width half maximum (FWHM) between $\min{\phi}$ and
$\phi_{\text{c}}$ (and analogously define $w_{t}$ as the FWHM between
$\max\phi$ and $\phi_{\text{e}}$) as illustrated in Fig. 2.c.

Figure 3: Scaling laws for speed and width of front: We simulate 946 CIS
systems using parameters given in the SI by numerically solving Eq. 5. From
each simulation we extract the (a) velocity ($V$) and (b) width ($W$) of the
propagating front (see Fig. 2) and plot each against $\xi$ and $\lambda$ (see
Eq. 6, 7), observing they collapse onto a line consistent with our physical
derivation of these scaling parameters. The orange star on each plot indicates
the simulation illustrated in Fig. 2. The blue line in (a) indicates the lower
bound (2.0e5) for the velocity consistent with experimental data (see SI).

To understand the nature of the propagating front solution arising from Eq. 5
we consider a balance between the viscous drag ($\eta\partial\phi/\partial
t$), chemical forcing ($\sim\tilde{f}\equiv f(\phi\approx\pi/2))$, and
generalized mechanical forces dominated by the highest order term
($\sim{EI}{r_{0}^{2}}\phi^{\prime\prime\prime\prime}$) at the front 333This is
equivalent to having substantial curvature gradients (see Eq. (3)) in the
neighborhood of the front.. Balancing the second and third, i.e. the chemical
forcing and mechanical strain yields a characteristic width over which $\phi$
varies at the shearing front:

$\lambda\sim\left(\frac{EIr_{0}^{2}}{\tilde{f}}\right)^{1/4}$ (6)

Balancing the first and second leads to a time scale $\tau\sim\eta/\tilde{f}$,
which combined with the previous length scale (or equivalently stating that
all three terms must balance each other) yields a scaling law for the front
velocity:

$\xi\sim\frac{\lambda}{\tau}\sim\frac{\left(\tilde{f}^{3}EIr_{0}^{2}\right)^{1/4}}{\eta}$
(7)

In Fig. 3 we show that numerical simulations of Eq. 5 yields the front width
and velocity that follow the above scaling laws, consistent with our
assumption that the geometry and dynamics of the CIS traveling front are
controlled by a balance between viscous friction, and chemo-mechanical forces.

Using experimentally available parameter values (see Table
LABEL:tab:physical_parameters) for the geometry of the CIS strands
($r_{0}\approx\qty{100}{}$, $r_{b}\approx\qty{10}{}$), elastic stiffness of
biological filaments ($E\approx\qty{1E9}{}$), activation and enthalpy
densities ($U_{a}\approx\qty{2E-12}{\per}$, $U_{H}\approx\qty{2E-11}{\per}$),
and a dynamic shearing friction coefficient $\eta\approx\qty{1E-20}{N.s}$ (see
SI for a discussion for how this was estimated), we recover a front width of
$l\approx\qty{60}{}$ and a velocity of $v\approx\qty{1e-3}{\per}$. We note
that this front width is consistent with the assumption of a slowly varying
shear used in deriving the simplified bending energy density, while the
velocity is larger than the lower bound ($v_{\text{min}}=\qty{2E-4}{\per}$)
estimated from experiments (see §S.IV for details). Importantly, we note that
the friction coefficient used in our simulations corresponds to an effective
viscosity of the surrounding fluid of $\eta\approx\qty{1e-2}{}$, which is well
within the range of effective viscosities observed in nanoscale systems.

Our model has focused on the physico-chemical mechanisms associated with the
CIS. However, the interplay between a chemically-induced bistability of sub-
unit conformation, the mechanical stiffness of protein strands, and viscous
stresses from the ambient surroundings are features generic to a larger class
of macromolecular machines. It is plausible that in many such machines, the
formation of a critical nucleus (the binding of baseplate to target in the
case of CIS) will be followed by a front of geometric rearrangement which
propagates at constant velocity. Although the evolutionary history and
biological function of such structures may differ, their deployment is likely
to be controlled by scaling relations similar to those of $\lambda$, $\xi$
(Eq. 6, 7), a perhaps general principle worth pondering further.

## References

  * [1] L Mahadevan and P Matsudaira.  Motility powered by supramolecular springs and ratchets.  Science, 288(5463):95–100, April 2000.
  * [2] R Kamiya, S Asakura, K Wakabayashi, and K Namba.  Transition of bacterial flagella from helical to straight forms with different subunit arrangements.  J. Mol. Biol., 131(4):725–742, July 1979.
  * [3] C R Calladine, B F Luisi, and J V Pratap.  A “mechanistic” explanation of the multiple helical forms adopted by bacterial flagellar filaments.  J. Mol. Biol., 425(5):914–928, March 2013.
  * [4] Srikanth V Srigiriraju and Thomas R Powers.  Model for polymorphic transitions in bacterial flagella.  Physical Review E, 73(1):011902, 2006.
  * [5] D L Caspar.  Movement and self-control in protein assemblies. quasi-equivalence revisited.  Biophys. J., 32(1):103–138, October 1980.
  * [6] L Mahadevan, C S Riera, and Jennifer H Shin.  Structural dynamics of an actin spring.  Biophys. J., 100(4):839–844, February 2011.
  * [7] Wanxi Kan and Charles W Wolgemuth.  The shape and dynamics of the leptospiraceae.  Biophysical journal, 93(1):54–61, 2007.
  * [8] Paul M Ryan, Joshua W Shaevitz, and Charles W Wolgemuth.  Bend or twist? what plectonemes reveal about the mysterious motility of spiroplasma.  Physical review letters, 131(17):178401, 2023.
  * [9] D L Clemens, P Ge, B-Y Lee, M A Horwitz, and Z H Zhou.  Atomic structure of T6SS reveals interlaced array essential to function.  Cell, 160(5):940–951, February 2015.
  * [10] P Ge, D Scholl, P G Leiman, X Yu, JF Miller, and Z H Zhou.  Atomic structures of a bactericidal contractile nanotube in its pre- and postcontraction states.  Nat. Struct. Mol. Biol., 22(5):377–382, May 2015.
  * [11] M Kudryashev, R Wang, M Brackmann, S Scherer, T Maier, D Baker, F DiMaio, H Stahlberg, E H Egelman, and M Basler.  Structure of the type VI secretion system contractile sheath.  Cell, 160(5):952–962, February 2015.
  * [12] V A Kostyuchenko, P R Chipman, P G Leiman, F Arisaka, V V Mesyanzhinov, and M G Rossmann.  The tail structure of bacteriophage T4 and its mechanism of contraction.  Nat. Struct. Mol. Biol., 12(9):810–813, August 2005.
  * [13] A Desfosses, H Venugopal, T Joshi, J Felix, M Jessop, H Jeong, J Hyun, J B Heymann, M R H Hurst, I Gutsche, and A K Mitra.  Atomic structures of an entire contractile injection system in both the extended and contracted states.  Nat Microbiol, 4(11):1885–1894, November 2019.
  * [14] J Wang, M Brackmann, D Castaño-Díez, M Kudryashev, K N Goldie, T Maier, H Stahlberg, and M Basler.  Cryo-EM structure of the extended type VI secretion system sheath-tube complex.  Nat Microbiol, 2(11):1507–1512, November 2017.
  * [15] R C Guerrero-Ferreira, M Hupfeld, S Nazarov, N M Taylor, M M Shneider, J M Obbineni, M J Loessner, T Ishikawa, J Klumpp, and P G Leiman.  Structure and transformation of bacteriophage A511 baseplate and tail upon infection of listeria cells.  EMBO J., 38(3), February 2019.
  * [16] D H Coombs and F Arisaka.  T4 tail structure and function.  Molecular biology of bacteriophage, 1994.
  * [17] A A Aksyuk, P G Leiman, L P Kurochkina, M M Shneider, V A Kostyuchenko, V V Mesyanzhinov, and M G Rossmann.  The tail sheath structure of bacteriophage t4: a molecular machine for infecting bacteria.  EMBO J., 28(7):821–829, April 2009.
  * [18] N M I Taylor, M J van Raaij, and P G Leiman.  Contractile injection systems of bacteriophages and related systems.  Mol. Microbiol., 108(1):6–15, April 2018.
  * [19] F Arisaka, J Engel, and H Klump.  Contraction and dissociation of the bacteriophage T4 tail sheath induced by heat and urea.  Prog. Clin. Biol. Res., 64:365–379, 1981.
  * [20] Petr G Leiman, Paul R Chipman, Victor A Kostyuchenko, Vadim V Mesyanzhinov, and Michael G Rossmann.  Three-dimensional rearrangement of proteins in the tail of bacteriophage T4 on infection of its host.  118:419–429.
  * [21] C M To, E Kellenberger, and A Eisenstark.  Disassembly of T-even bacteriophage into structural parts and subunits.  46:493–511.
  * [22] P G Leiman and M M Shneider.  Contractile tail machines of bacteriophages.  Adv. Exp. Med. Biol., 726:93–114, 2012.
  * [23] A Fraser, N S Prokhorov, F Jiao, B M Pettitt, S Scheuring, and P G Leiman.  Quantitative description of a contractile macromolecular machine.  Sci Adv, 7(24), June 2021.
  * [24] A Maghsoodi, A Chatterjee, I Andricioaei, and NC Perkins.  Dynamic model exposes the energetics and dynamics of the injection machinery for bacteriophage T4.  Biophys. J., 113(1):195–205, July 2017.
  * [25] A Maghsoodi, A Chatterjee, I Andricioaei, and N C Perkins.  How the phage T4 injection machinery works including energetics, forces, and dynamic pathway.  Proc. Natl. Acad. Sci. U. S. A., 116(50):25097–25105, December 2019\.
  * [26] P Ge, D Scholl, N S Prokhorov, J Avaylon, M M Shneider, C Browning, S A Buth, M Plattner, U Chakraborty, K Ding, P G Leiman, J F Miller, and Z H Zhou.  Action of a minimal contractile bactericidal nanomachine.  Nature, 580(7805):658–662, April 2020.
  * [27] L N Trefethen.  Spectral methods in MATLAB.  Software, environments, tools. Society for Industrial and Applied Mathematics, Philadelphia, Pa., 2000.
  * [28] Keaton J Burns, Geoffrey M Vasil, Jeffrey S Oishi, Daniel Lecoanet, and Benjamin P Brown.  Dedalus: A flexible framework for numerical simulations with spectral methods.  Physical Review Research, 2(2):023068, 2020.


# The statistical advantage of automatic NLG metrics at the system level

Johnny Tian-Zheng Wei Robin Jia
Department of Computer Science, University of Southern California
{jtwei<EMAIL_ADDRESS>
(6/1/2021)

###### Abstract

Estimating the expected output quality of generation systems is central to
NLG. This paper qualifies the notion that automatic metrics are not as good as
humans in estimating system-level quality. Statistically, humans are unbiased,
high variance estimators, while metrics are biased, low variance estimators.
We compare these estimators by their error in pairwise prediction (which
generation system is better?) using the bootstrap. Measuring this error is
complicated: predictions are evaluated against noisy, human predicted labels
instead of the ground truth, and metric predictions fluctuate based on the
test sets they were calculated on. By applying a bias-variance-noise
decomposition, we adjust this error to a noise-free, infinite test set
setting. Our analysis compares the adjusted error of metrics to humans and a
derived, perfect segment-level annotator, both of which are unbiased
estimators dependent on the number of judgments collected. In MT, we identify
two settings where metrics outperform humans due to a statistical advantage in
variance: when the number of human judgments used is small, and when the
quality difference between compared systems is small.111The data and code to
reproduce our analyses are available at https://github.com/johntzwei/metric-
statistical-advantage.

Bias$\delta^{H}_{S,S^{\prime}}$(True difference)$0$Dist. of estimatorMetric
($\widehat{\delta^{M}_{S,S^{\prime}}}$)Human
($\widehat{\delta^{H}_{S,S^{\prime}}}$)  Figure 1: Distribution of estimators
for the true difference in system quality $\delta^{H}_{S,S^{\prime}}$ between
two generation systems (for illustrative purposes). Notation is defined in
§2.3. An estimate incurs prediction error if its sign is opposite to the true
difference. While humans provide an unbiased estimator of the difference, a
biased estimator derived from a metric can have a smaller error probability
(shaded areas) due to its lower variance. Evidence supporting the illustration
can be found in §5.

##  1 Introduction

Automatic metrics are involved in many developmental settings for natural
language generation (NLG) systems. In machine translation (MT), metrics like
BLEU Papineni et al. (2002) enable settings where the amount of human effort
required would be infeasible, such as architecture or hyperparameter search
Britz et al. (2017). As objective, reproducible quantities, BLEU scores
facilitate cross-paper comparisons Post (2018). Historically, progress in MT
has been attributed to its use Callison-Burch et al. (2006). Metrics are an
active research area in many NLG subfields, including summarization Lin
(2004), dialogue Tao et al. (2018), and image captioning Anderson et al.
(2016), which seek to realize the goal of quick and reliable automatic
evaluation.

In all these subfields, the primary goal when conducting evaluation is
typically to compare NLG systems. Both human annotators and automatic metrics
produce _segment-level_ scores, i.e., scores for individual examples, so
comparing systems requires aggregating segment-level scores into an overall
_system-level_ score for each system. Ideally, we would compare systems by
their expected human annotator score (an average over infinite human
judgments), which we term the true system quality. In practice, we can only
estimate this expectation with a sample mean over a finite number of human
judgments. Metrics offer a cheaper alternative: we can instead compare systems
by their aggregate metric scores on a number of system outputs. When comparing
systems, we care primarily about how well we estimate the _difference_ of
their true system qualities, and in particular the sign of this difference
(i.e., which system is better), which we term the true pairwise label.

There is a gap in our understanding of _system-level_ metrics. To recount a
perplexing anecdote, in the most recent edition of the WMT metrics shared task
Mathur et al. (2020b), initial human evaluation disagreed with most metrics on
a pairwise prediction of two translation systems. In a manual re-evaluation,
the second round results favored the metrics. Our paper offers a statistical
explanation for how humans could go “wrong”: even if human estimation for the
difference in system quality is unbiased, it has high variance. On the other
hand, while estimators based on metrics are biased, they have low variance. It
is therefore possible for metrics to give a more accurate pairwise prediction
than humans when the bias is small (see illustration in Figure 1). Our paper
explores this distinction in the following three questions:

(1) _How can we evaluate system-level metrics?_ When observing estimator error
in terms of pairwise predictions, predictions are evaluated against noisy,
human predicted labels rather than the ground truth. In addition, metric
predictions fluctuate based on the sample of outputs from the generation
system. To disentangle these properties, we examine observed estimator error
under a bias-variance-noise decomposition. Under simulation, we find that the
label noise and metric variance account for a small fraction of observed error
in both MT and summarization.

(2) _How good are these metrics?_ We compare the errors of metric estimators
computed on an infinite number of system outputs, against human estimators
with varying amounts of human judgment. We also derive the error of a perfect
segment-level annotator (i.e. they provide noiseless/expected human scores for
each output), which is also unbiased and judgment dependent. Empirically, some
MT metrics exceed the performance of unbiased estimators with a small number
of judgments.

(3) _What are the limits of system-level evaluation?_ The perfect segment-
level annotator, as the noiseless human, provides an optimistic estimate for
the number of human judgments necessary to achieve a fixed performance. With a
power analysis, we can analytically calculate the number of judgments
necessary to detect differences between systems of varying sizes. When
differences in system quality are small, a prohibitively large number of
perfect annotator judgments are required to give a correct pairwise
prediction.

##  2 Formalization

###  2.1 System-level scores

We will now formalize scoring at the system level, adopting notation from
Chaganty et al. (2018). Let $\mathcal{X}$ be a distribution over inputs (e.g.
source sentences), and $\mathcal{S}$ be a set of systems (e.g. all translation
systems in WMT). Each system $S\in\mathcal{S}$ takes input $x\sim\mathcal{X}$
and returns output $z=S(x)$ (e.g. $z$ is a translation). Let $H(z)$ be a
random variable representing a human judgment according to some evaluation
prompt (e.g. translation adequacy, from 0-100). A central quantity of interest
is the quality of system $S$, defined as

$\mu^{H}_{S}=\mathbb{E}_{x\sim\mathcal{X}}[H(S(x))]$ (1)

and is not directly observable as it requires infinite human judgment. We can
estimate (1) with a finite test set of $n$ examples. Let
$x^{(1)},\dots,x^{(n)}\overset{\text{i.i.d.}}{\sim}\mathcal{X}$ be a sampled
test set and $z^{(1)},\dots,z^{(n)}$ be the set of outputs where each
$z^{(i)}=S(x^{(i)})$. Human judgments are sampled independently as
$y^{(i)}\sim H(z^{(i)})$. The sample mean

$\widehat{\mu^{H}_{S}}=\frac{1}{n}\sum_{i=1}^{n}y^{(i)}$ (2)

is an unbiased estimator of (1). Only (2) is observable, which is a noisy
approximation of (1).

A cheaper alternative to estimating the true quality scores is with an
estimator based on an automatic metric. Let $M$ (e.g. BERTscore) be an
automatic metric that takes as input any number of outputs from a system $S$
and produces score

$\widehat{\mu^{M}_{S}}=M(z^{(1)},\dots,z^{(n)})$ (3)

where $\widehat{\mu_{S}^{M}}$ is a biased estimator of $\mu^{H}_{S}$. As the
test set is sampled, the metric score has non-zero variance. Note that while
we use the greek letter $\mu$, only some system-level metrics (e.g. ROUGE) are
averages of their segment-level counterparts (their score decomposes to
$\widehat{\mu^{M}_{S}}=\frac{1}{n}\sum_{i=1}^{n}M(z^{(i)})$). Empirically, we
find that metrics using other aggregation strategies have convergent
properties similar to an average (see Appendix B). We sidestep this by
defining the “true” metric score as

$\mu^{M}_{S}=M(z^{(1)},\dots,z^{(m)})$ (4)

for test sets of size $m$ sufficiently large so that this true score is nearly
constant.

###  2.2 Problems in evaluating with correlation

Research in system-level metrics have a tradition of evaluating metric
correlation to human judgment with the Pearson correlation coefficient Reiter
(2018). Formally, these evaluations compare
$\widehat{r_{M}}=\text{Corr}_{\mathcal{S}}(\widehat{\mu^{H}_{S}},\widehat{\mu^{M}_{S}})$
for different metrics $M$.

Recently, Mathur et al. (2020a) highlights two issues with the use of
correlation: First, Pearson’s $r$ is neither interpretable nor reflective of
system-level metric use in practice. Second, outlier systems (systems with
very high/low human/metric scores) can arbitrarily inflate Pearson’s $r$, and
outlier systems often exist. Mathur et al. (2020a) propose evaluating metric
accuracy in pairwise prediction (can the metric differentiate which generation
system is better?) as an alternative that mitigates the issues mentioned
above.

We add two points that apply to any measure of metric performance, correlation
or pairwise predictions: First, metrics cannot be perfect due to noise in
human labels. For instance, while $r$ ranges from $[-1,1]$, even for the
metric that predicts $\mu^{H}_{S}$ it has
$\text{Corr}_{\mathcal{S}}(\widehat{\mu^{H}_{S}},\mu^{H}_{S})<1$ due to noise
in $\widehat{\mu^{H}_{S}}$. It is unclear what is the true upper bound of
performance we can expect to achieve. Second, direct measurement of any
performance measure on our datasets introduces sample bias Engstrom et al.
(2020). For correlation, $\widehat{r_{M}}$ could be high because
$\widehat{\mu^{H}_{S}}$ and $\widehat{\mu^{M}_{S}}$ happened to align for this
data collection, but a repeat experiment could yield different results. A more
holistic view is to give an estimate of average case performance.222Pearson’s
$r$ was not formulated for individual distributions $\widehat{\mu^{H}_{S}}$
and $\widehat{\mu^{M}_{S}}$ for each datapoint, so applying the William’s test
Graham and Baldwin (2014) also falls short here.

The evaluation methodology we derive in §4 addresses the latter points we
raise for pairwise predictions and mean squared error (which has direct
relationship to the correlation). However, we also believe that pairwise
predictions is a step in the right direction, and our discussion continues
with pairwise predictions.

###  2.3 Pairwise predictions

We will now formalize pairwise predictions. For systems
$S,S^{\prime}\in\mathcal{S}$, define the true difference in their system
scores as

$\delta^{H}_{S,S^{\prime}}=\mu^{H}_{S}-\mu^{H}_{S^{\prime}}$ (5)

and the observed difference as

$\widehat{\delta^{H}_{S,S^{\prime}}}=\widehat{\mu^{H}_{S}}-\widehat{\mu^{H}_{S^{\prime}}}$
(6)

and likewise for the differences $\delta^{M}_{S,S^{\prime}}$ and
$\widehat{\delta^{M}_{S,S^{\prime}}}$ w.r.t. to a metric $M$. In practice, we
are interested in the pairwise prediction of $S$ and $S^{\prime}$ i.e. whether
$\delta^{H}_{S,S^{\prime}}\stackrel{{\scriptstyle?}}{{>}}0$, given that we
have collected human judgments (we observe
$\widehat{\delta^{H}_{S,S^{\prime}}}\lessgtr 0$), or computed metric scores
(we observe $\widehat{\delta^{M}_{S,S^{\prime}}}\lessgtr 0$). Refer to Figure
1 for an illustration.

To operationalize the pairwise prediction of $S$ and $S^{\prime}$, let the
true pairwise label

$\Delta^{H}_{S,S^{\prime}}=\text{sign}(\delta^{H}_{S,S^{\prime}})$ (7)

be defined as the central quantity of interest. Define the human predicted
pairwise label as

$\widehat{\Delta^{H}_{S,S^{\prime}}}=\text{sign}(\widehat{\delta^{H}_{S,S^{\prime}}})$
(8)

and likewise for the true and estimated predictions
$\Delta^{M}_{S,S^{\prime}}$ and $\widehat{\Delta^{M}_{S,S^{\prime}}}$ w.r.t.
to a metric M. The 0-1 classification loss for metric $M$ on this example is

$L(\Delta^{H}_{S,S^{\prime}},\widehat{\Delta^{M}_{S,S^{\prime}}})=\mathbb{I}[\Delta^{H}_{S,S^{\prime}}\neq\widehat{\Delta^{M}_{S,S^{\prime}}}]$
(9)

and the pairwise error of an estimator is the loss incurred averaged over all
pairwise examples. Ideally, we could calculate the true error of $M$

$\text{Err}_{\text{true}}(M)=\mathbb{E}_{\mathcal{S}}[L(\Delta^{H}_{S,S^{\prime}},\Delta^{M}_{S,S^{\prime}})]$
(10)

but we can only compute an error of $M$ with noisy human labels and metric
scores estimated from finite sized test sets

$\text{Err}_{\text{obs}}(M)=\mathbb{E}_{\mathcal{X},\mathcal{S}}[L(\widehat{\Delta^{H}_{S,S^{\prime}}},\widehat{\Delta^{M}_{S,S^{\prime}})}]$
(11)

which is typically estimated when we calculate metric pairwise accuracy from
our datasets.

|  | Error components
---|---|---
| $\text{Err}_{\text{obs}}(\cdot)$ | $c_{0}\text{Noise}$ | Bias | $c_{1}\text{Var}$
Optimal ($\Delta^{H*}_{S,S^{\prime}}$) | 0.047 | 0.000 | 0.000 | 0.047
Human ($\widehat{\Delta^{H}_{S,S^{\prime}}}$) | 0.065 | 0.019 | 0.000 | 0.047
BERTscore | 0.102 | 0.003 | 0.086 | 0.013
chrF | 0.124 | 0.010 | 0.105 | 0.009
BLEURT** | 0.128 | 0.005 | 0.108 | 0.016
BLEU | 0.141 | 0.008 | 0.127 | 0.007
TER | 0.184 | 0.002 | 0.173 | 0.009

|  | Error components
---|---|---
| $\text{Err}_{\text{obs}}(\cdot)$ | $c_{0}\text{Noise}$ | Bias | $c_{1}\text{Var}$
Optimal ($\Delta^{H*}_{S,S^{\prime}}$) | 0.045 | 0.000 | 0.000 | 0.045
Human ($\widehat{\Delta^{H}_{S,S^{\prime}}}$) | 0.067 | 0.022 | 0.000 | 0.046
ROUGE | 0.296 | -0.006 | 0.294 | 0.008
METEOR | 0.296 | 0.004 | 0.287 | 0.005
ROUGE-WE | 0.317 | 0.007 | 0.301 | 0.008
BERTscore | 0.330 | -0.004 | 0.338 | -0.004
SUPERT*** | 0.390 | 0.000 | 0.382 | 0.008

Table 1: Decomposition of the pairwise error of different metrics (left: WMT,
right: SummEval). Highlighted in bold is the largest error component. 10K
boostrap trials are conducted for estimation of the expectations (estimation
error $<10^{-3}$). *Denotes an estimator assumed to be unbiased in the
simulation. **BLEURT is evaluated only on WMT2019. ***SUPERT is a reference-
less metric.

##  3 Datasets

###  3.1 WMT16-19 metrics shared task

Data. We use the past 4 years of to-English translation data from the WMT
metrics shared task Bojar et al. (2016b, 2017); Ma et al. (2018, 2019).333The
WMT20 metrics shared task data was not publicly available at the time of
submission. Across all years and language pairs, there are 261 MT systems.
Pairs of MT systems are extracted within each year, within each language pair,
resulting in 1324 pairwise examples. For each output of an MT system, there
are one or more humans judgements and one reference for metric scoring.
1306-5117 outputs were collected for each MT system totaling about 1312-5612
judgments, depending on the year and language pair. For ease of
interpretation, we always use raw direct assessment judgments which range from
0-100.

Metrics. We evaluate the performance of the three metrics included in
SacreBleu (BLEU, TER, chrF; Post, 2018; Koehn et al., 2007). These three have
also participated in every year of the metrics task as baselines. In addition,
we include two recently developed metrics: BERTscore Zhang et al. (2020) and
BLEURT Sellam et al. (2020). Both metrics are found to effectively utilize
contextual embeddings Devlin et al. (2019), and BLEURT is a learned metric
(tuned on data outside of WMT2019). For all metrics, we use the default
settings for scoring. Since BLEURT is trained on WMT15-18, we test it only on
WMT2019 pairs.

###  3.2 SummEval

Data. The SummEval dataset Fabbri et al. (2020) contains 100 outputs from 17
summarization systems. This results in 136 pairwise examples. For each system
output, 3 expert judgments, and 11 references for metric scoring. Each
summarization is judged in four categories from 0-5: coherence, consistency,
fluency, and relevance. To compute system-level human scores for a system, we
first average over categories for an aggregate expert score, and then average
the aggregated expert scores per system. Metric scores for system outputs were
computed against as many references as possible.

Metrics. We evaluate the performance of several metrics that were found to be
effective at the system-level in Fabbri et al. (2020). This includes the
traditional ROUGE-4 Lin (2004) summarization metric, its extension ROUGE-WE Ng
and Abrecht (2015), and METEOR Lavie and Agarwal (2007). In addition, we
include two metrics based on BERT Devlin et al. (2019). BertScore Zhang et al.
(2020), also present in the WMT analysis, and SUPERT Gao et al. (2020), which
is a reference-less metric for summarization.

##  4 Decomposing observed metric error

Two sources of variation distinguish the observed pairwise error (11) from the
true error in (10) — the noise in the human predicted labels due to finite
judgements, and the variance in the metric due to finite test sets.
Approximating (11) is straightforward with the bootstrap, but disentangling
the error from these two sources of variation requires more care. With the
bias-variance-noise decomposition, we can adjust our observed error estimates
to the noise-free, infinite test set setting of the true error.

###  4.1 The bias-variance-noise decomposition

The bias-variance-noise decomposition due to Domingos (2000) decomposes the
observed pairwise error in (11) w.r.t. two constant labels for any pairwise
example on systems $S,S^{\prime}\in\mathcal{S}$:

  * •

The _true pairwise label_ for this example is

$\displaystyle\Delta_{S,S^{\prime}}^{H*}:=\arg\min_{y\in\\{-1,1\\}}\mathbb{E}_{\mathcal{X}}[L(\widehat{\Delta_{S,S^{\prime}}^{H}},y)]$
(12)

and the estimator that produces these true labels has, by definition, the
lowest observed error. In the decomposition, the human predicted label noise
and metric bias is defined relative to the true labels. Assuming the central
limit theorem (proof in Appendix A), we actually have
$\Delta_{S,S^{\prime}}^{H*}=\Delta_{S,S^{\prime}}^{H}$ as defined in eq. (5).

  * •

The _main prediction_ of a metric for this ex. is

$\displaystyle\Delta_{S,S^{\prime}}^{M*}=\arg\min_{y\in\\{-1,1\\}}\mathbb{E}_{\mathcal{X}}[L(\widehat{\Delta_{S,S^{\prime}}^{M}},y)]$
(13)

and we assume that the metric prediction converges onto the main prediction as
the test data increases for $S$ and $S^{\prime}$ (empirically validated in
Appendix B). In the decomposition, the metric variance is defined relative to
the main prediction.

Starting from the loss incurred by $M$ on this pairwise example, the
decomposition gives us

$\displaystyle\mathbb{E}_{\mathcal{X}}[L(\widehat{\Delta^{H}_{S,S^{\prime}}},\widehat{\Delta^{M}_{S,S^{\prime}}})]$
$\displaystyle=c_{0}\text{Noise}(\widehat{\Delta^{H}_{S,S^{\prime}}})$ (14)
$\displaystyle+\text{Bias}(\widehat{\Delta^{M}_{S,S^{\prime}}})+c_{1}\text{Var}(\widehat{\Delta^{M}_{S,S^{\prime}}})$

where

  * •

$\text{Noise}(\widehat{\Delta^{H}_{S,S^{\prime}}})=\mathbb{E}[L(\widehat{\Delta^{H}_{S,S^{\prime}}},\Delta_{S,S^{\prime}}^{H*})]$
where the noise is an irreducible loss incurred by computing pairwise accuracy
to the human predicted labels instead of the true labels. Note that this noise
term also exactly corresponds to the lowest achievable observable error (see
§4.2).

  * •

$\text{Bias}(\widehat{\Delta^{M}_{S,S^{\prime}}})=L(\Delta^{H*}_{S,S^{\prime}},\Delta^{M*}_{S,S^{\prime}})$
where the bias is 0 if the main prediction is correct (w.r.t. to the true
label), and 1 otherwise. Note that this term is also the true error of a
metric estimator in a noise-free, infinite test set setting. For unbiased
estimators this term is zero, as their main prediction matches the true label.

  * •

$\text{Var}(\widehat{\Delta^{M}_{S,S^{\prime}}})=\mathbb{E}[L(\widehat{\Delta^{M}_{S,S^{\prime}}},\Delta^{M*}_{S,S^{\prime}})]$
where the variance is a likelihood that the estimator deviates from its main
prediction under random sampling.

  * •

$c_{0}=2P_{\mathcal{X}}(\widehat{\Delta^{M}_{S,S^{\prime}}}=\Delta^{H*}_{S,S^{\prime}})-1$
which means that the influence of label noise on the error becomes small if
the estimator prediction are close to random chance. When the estimator gives
constant predictions, the sign of $c_{0}$ is dependent on the estimator’s
correctness.

  * •

$c_{1}=1$ if $\widehat{\Delta^{M}_{S,S^{\prime}}}=\Delta^{H*}_{S,S^{\prime}}$
and $c_{1}=-1$ otherwise. Variance can both increase and decrease the observed
error. If the estimator is unbiased, the variance causes the prediction to
from the correct main prediction. On the other hand, for a biased estimator,
deviation from its incorrect main prediction occasionally decreases the error.

Unlike the decomposition for mean squared error, the interaction between the
$c_{0}$ and Var terms only allows the error of two hypothetical settings to be
read off directly from the table: when $\text{Noise}\xrightarrow[]{}0$,
corresponding to estimator error when computed against the ground truth; or
when $\text{Noise}+\text{Var}\xrightarrow[]{}0$, when the ground truth is used
and metrics have access to an infinite test set for scoring.

###  4.2 A lower bound for the observed error

By definition the constant estimator that produces the true pairwise labels
$\Delta^{H*}_{S,S^{\prime}}$ (defined in (12)) for each pairwise example has
the lowest possible observable error. The observable error of this optimal
estimator is exactly
$\mathbb{E}[L(\widehat{\Delta^{H}_{S,S^{\prime}}},\Delta^{H*}_{S,S^{\prime}})]=\text{Noise}(\widehat{\Delta^{H}_{S,S^{\prime}}})$.
Since this estimator is constant it has no variance, and since it is
instantiated by definition it has no bias. Analytically, the observed error of
any estimator is lower bounded by
$\text{Noise}(\widehat{\Delta^{H}_{S,S^{\prime}}})$ and is the agreement of
our human predicted labels with the ground truth.

###  4.3 Best-faith estimation with the bootstrap

Assuming the bootstrap Efron and Tibshirani (1993) which is a common procedure
in NLP Dror et al. (2018), we can estimate the expectation quantities in the
decomposition. By assuming that sampling with replacement from our datasets
approximates real sampling, we can repeatedly simulate the quantity in an
expectation. Taking the mean over trials gives the bootstrap estimate of the
expectation. We emphasize that this is a regular application of a widely
accepted technique—the bootstrap assumption allows us to study problems that
would be impossible due to the cost of repeat experiments.

###  4.4 Results

The following analyses refer to the error components (averaged over all
examples) from the simulated decomposition presented in Table 1.

The noise component almost always accounts for a small fraction of the total
error. We found this to be counterintuitive—while the lowest observable error
(optimal predictions, see §4.2) incur about $5\%$ error on both datasets, the
influence of the noise is much smaller than those errors suggest. For the
constant $c_{0}$ scaling the noise, $c_{0}=0$ if the metric prediction is near
random. Since the $c_{0}\text{Noise}$ term on average is small two cases hold
true: when humans are uncertain about the example (noise term large) metrics
are as well ($c_{0}$ term small), and when metrics are certain about the
examples ($c_{0}$ term large) humans are as well (noise term small). The
second case empirically shows studying the sampling distribution of metrics
Koehn (2004); Berg-Kirkpatrick et al. (2012) is effective, as metric certainty
in the difference of system quality often implies human certainty.

Metric variance introduces little to the pairwise error, because it is low.
Alternatively, metrics stand to gain little from using more test set examples.
In MT, dropping both the noise and variance components for the error results
in at most a 1 or 2 percent reduction in the observed error (see §9 for the
implications in metrics research). Metrics generally have low variance, so at
the test set sizes of WMT and SummEval, they are likely to converge to their
main predictions.

Figure 2: Comparison of metrics to human and perfect annotator estimators with
varying number of judgments in WMT. Errors are adjusted to an idealized
setting where true predictions are used for evaluation and metrics are
computed on infinite test sets; here metric predictions become constant, so
their errors are constant. Shaded in grey is the region where BERTscore is
superhuman. Results for SummEval are in Appendix D.

##  5 Comparing to the human estimator

In §4, several MT metrics approach the error of the WMT human evaluation. The
WMT human evaluation is expensive, using thousands of judgments per
translation system. While each human judgment has associated monetary cost,
once a large test set is collected, running metrics only incurs computational
cost. This section explores this asymmetry, and seeks to understand how much
metric predictions are worth, in terms of human judgments.

###  5.1 Noise-free, variance-free error estimates

We wish to give our best comparison between metrics and unbiased estimators
(humans or the perfect annotator). Ideally, metrics would be given their best
chance to perform, by using an infinite test set. With the decomposition, we
can adjust metric errors estimates to a noise-free and infinite test set
setting by taking only their bias component. For human and perfect annotator
estimators, we can adjust their errors to a noise-free setting by taking only
the variance component. The following sections compare these adjusted errors.

###  5.2 Simulating the perfect annotator

While we can estimate the lower bound to the pairwise error for a given
dataset (in §4.2), it is achieved by a constant estimator using system-level
ground truth. Comparing segment-level metrics against the unbiased “perfect
annotator”, or the best scorer at the segment-level, is more informative. At
the high-level, we can simulate scoring with the perfect annotator at $n$
judgments using the human estimator at $n^{\prime}>n$ judgments to match the
variance of the perfect annotator estimator.

Let’s start from the unbiased human estimator $\widehat{\mu_{S}^{H}}$ (2).
Recall that the estimator is a sample mean, so its variance is
$\text{Var}(\widehat{\mu_{S}^{H}})=\text{Var}(H(x))/n$. An insight from
Chaganty et al. (2018) gives us the decomposition of the variance of $H(x)$

$\displaystyle\text{Var}(H(x))$ $\displaystyle=\text{Var}(\mathbb{E}[H(x)|x])$
(15) $\displaystyle+\mathbb{E}[\text{Var}(H(x)|x)]$

with the law of total variance. In words, the _variance_ term can be thought
of as the variance of each output sentence’s true quality score (some
translations produced by $S$ are better than others) and the _expectation_
term is the noise introduced by the humans when estimating the quality of a
sentence (human scores have mean 0 noise around an output’s true quality
score).

One intuition is that even if a perfect annotator gives the correct score for
each sentence, every time, there is still some unavoidable variance in the
estimator due to the variance of the hypothetical quality scores for each
output. To formalize this notion, let $P(x)=\mathbb{E}[H(x)|x]$ be the human
scoring function of a “perfect annotator”, and the estimator
$\widehat{\mu_{S}^{P}}$ be an empirical mean of $n$ independent samples from
$P(x)$ similar to eq. (1). As a sample mean,
$\text{Var}(\widehat{\mu_{S}^{P}})=\text{Var}(P(x))/n$. Relating this to (15)

$\text{Var}(H(x))=\mathbb{E}[\text{Var}(H(x)|x)]+\text{Var}(P(x))$ (16)

and while $\text{Var}(P(x))$ is not directly observable, we can calculate
$\text{Var}(H(x))$ with the sample variance on all the human judgments, and
$\mathbb{E}[\text{Var}(H(x)|x)]$ with a pooled variance over variances from
repeat human judgments on the same output sentence.

Our final step considers the efficiency ratio
$r=\text{Var}(H(x))/\text{Var}(P(x))$. If we are interested in the perfect
annotator estimator at $n$ judgments, the human estimator at $n^{\prime}=rn$
judgments has variance

$\displaystyle\text{Var}(\widehat{\mu_{S}^{H}})$
$\displaystyle=\frac{\text{Var}(H(x))}{rn}$ (17)
$\displaystyle=\frac{\text{Var}(P(x))}{n}=\text{Var}(\widehat{\mu_{S}^{P}})$
(18)

and we invoke the central limit theorem to claim both $\widehat{\mu_{S}^{P}}$
and $\widehat{\mu_{S}^{H}}$ are normal. This completes our reasoning that for
scoring on the system-level, sampling $n^{\prime}=nr$ human judgments is
nearly equivalent to sampling $n$ perfect annotator judgments. See Appendix C
for step-by-step derivations for the perfect annotator variance in our
datasets.

Table 2: Power analysis for the number of judgments needed to give a pairwise
prediction between two systems at $.9$ accuracy ($\alpha=0.05,\beta=0.95$)
under ttest assumptions (normality, equal variance) in WMT. WMT ratings are on
a 0-100 scale, and the perfect annotator variance in WMT19 was $19.27$. Darker
cells indicate less feasible experiments, and the colors are set on a log
scale. Results for SummEval are in Appendix D.

###  5.3 Results

The following analyses refer to the comparison of metric estimators to
unbiased estimators at varying number of judgments for WMT in Figure 2.

Judgments from the perfect annotator have low variance, like those of
professional linguists. While we do not have data from professional linguists,
we can qualitatively compare them to the perfect annotator. A growing body of
MT literature focuses on professional linguists Freitag et al. (2020); Mathur
et al. (2020b), and there are at least two known properties of their
judgments: their judgments have better interannotator agreement (contain less
noise), and they are more sensitive to linguistic phenomena. The perfect
annotator has no noise, as they assign a constant score to each segment.
However, the perfect annotator in WMT is better described as a noiseless
crowdworker. With the biases of crowdworkers, the perfect annotator may not
share the sensitivity property, and our use of crowdworkers may be biased
w.r.t. professional linguists.

In terms of average pairwise error, MT metrics have an equivalence to a high
number of human judgments. Since the error of the human estimator
monotonically increases as the number of judgments decrease, each MT metric
has a breakeven point. Metrics outperform human estimators using judgments
below this threshold. BERTscore is as accurate as using a human estimator with
600 judgments per system, or the perfect annotator estimator with 300
judgments, across the WMT dataset. We highlight the statistical advantage in
variance many metrics share, and that this advantage offers a _possibility_
that metrics can outperform humans, determined by which human estimator the
metric is compared against. This is a consequence of the general fact that
humans are unbiased, high-variance estimators, and metrics are biased, low-
variance estimators, as depicted in Figure 1. For metrics such as BERTscore or
chrF, the bias is low as well, which gives it remarkably good error
properties.

##  6 The limits of human evaluation

The perfect annotator provides optimistic figures for human annotation,
providing the best performance for a fixed number of judgments, and requiring
the least judgments for a fixed performance. In §5, we saw that the perfect
annotator is weak at low number of judgments, due to its non-zero variance. In
this section we identify another consequence of the perfect annotator’s
variance, where estimating small differences in system quality is hard.

###  6.1 Power analysis of the perfect annotator

The performance of an unbiased estimator is dependent on their variance and
the effect size it is trying to detect. This section performs a power analysis
to determine how much annotator effort is needed to reliably detect the
correct pairwise judgment between two systems Card et al. (2020). To make an
optimistic estimate, we assume our annotator variance is close to that of a
perfect annotator. We make two assumptions to apply a basic power analysis for
the estimation of the difference of system quality between two systems:
normality and equal variance across groups. For parameters $\alpha=0.05$
(false positive rate) and $\beta=0.95$ (false negative rate), we can
analytically compute the number of judgments needed to ensure our pairwise
judgment is at least $\beta(1-\alpha)\approx 90\%$ accurate. Table 2 contains
power analyses for different instantiations of annotator variance and effect
size.

In WMT, detecting a difference of 1 point requires at least 10K perfect
annotator judgments, for different instantiations of its variance. To put this
in perspective, the top 5 zh-en translation systems in WMT19 differed by less
than 3 points Barrault et al. (2019). Depending on how much is paid per
judgment, this cost can quickly become infeasible. Here, the merit of such a
task may be argued, as knowing a small difference exists between two systems
may not always be productive. From a scientific perspective, many NLG
techniques will yield small improvements, and not being able to detect small
differences means we will not know whether these techniques are useful.

###  6.2 Metrics more easily achieve significance

Since metrics tend to have lower variance, metrics often achieve significance
in estimating the difference of system qualities, when humans cannot. For
instance, BERTscore achieves significance in estimating quality differences
over half of the pairwise examples where humans do not (see Appendix §E). In
extreme cases, human evaluation is nearly as bad as flipping a coin, but the
metric can still offer a consistent prediction between two systems. When
comparing systems similar in quality, practitioners must accept that the
number of possible analyses are limited. In ablation studies where similar
systems are often compared, metrics may be our only insight into system
performance. With white-box metrics such as BLEU, value can be derived from
qualitative insight (e.g. systems with high BLEU score have high $n$-gram
overlap with the reference set). In addition, we may qualitatively analyze
output statistics not intended to correlate with humans judgment at all Neubig
et al. (2019).

##  7 Caveats to the analysis

Our analysis assumes that the human judgments are unbiased. In WMT16-19,
direct assessment Graham et al. (2013) was used to elicit judgments from a
combination of crowdworkers and researchers. Direct assessment (DA) uses an
adequacy evaluation prompt (“Rate how much you agree that the output
translation adequately expresses the meaning of the reference translation”)
and asks contributors to rate on a 0-100 scale.

The unbiased ground truth is not a fixed goalpost. A number of factors are
known to change the eventual ranking of translation systems with human
scoring. Employing a different collection methodology, such as human
translation edit rate (HTER) of instead of DA, can result in divergent system
rankings Graham et al. (2016). In an earlier edition of WMT, DA judgments were
collected with both a grammaticality prompt and an adequacy prompt,
corresponding to different system rankings by the respective attribute Bojar
et al. (2016a). Several studies have shown scoring differences between
professional linguists and crowdworkers which are due in part to the fact that
linguists are more sensitive to linguistic phenomena Fabbri et al. (2019);
Freitag et al. (2019).

The goals of an evaluation should be decided by the practitioner. We do not
give suggestions on any particular goals, and practitioners should understand
what their application is, and which evaluation is the best approximation
(refer to Gatt and Krahmer, 2018). Unfortunately, since the existing data in
this domain is limited, our analyses are limited as well. However, the
statistical techniques apply to any empirical method. We hope that our
analysis inspires others to think about statistical limits in this domain.

##  8 Pushing the limits of evaluation

To push the limits of what can be evaluated, we need to improve on fundamental
aspects of human evaluation. On the human side, we may focus on creating
larger effect sizes or reducing noise by adopting new annotation schemes
Läubli et al. (2018); Shapira et al. (2019) or employing professional
linguists Fabbri et al. (2020); Toral et al. (2018). To make the human
estimator more efficient, we may consider adaptive data collection techniques
to stop data collection early when significance is achieved, in a
statistically sound manner Johari et al. (2017).

Strategies combining human and metric evaluation are also shown to have
potential. Variance reduction techniques can be applied to the human estimator
by taking advantage of strong metrics Chaganty et al. (2018). Another
bottleneck in human evaluation is in the random sampling of the test set.
Metrics could form the basis of an importance sampling procedure to choose
test sets that would best differentiate two systems, as a form of robust
evaluation Chaganty et al. (2017).

On the metric side, if we can reliably estimate metric bias, we can skip human
evaluation altogether when the metric is known to be good. Probabilistic
reinterpretations of current metrics could be a useful technique for
confidence estimation Keith and O’Connor (2018). Optimistically, metrics could
have provable guarantees, ensuring the correctness of metric decisions Jia et
al. (2019).

##  9 Best practices for metrics research

We reinterpret problems in evaluating metrics with correlation (§2.2) as a set
of guidelines for metrics research. To next year’s organizers of the WMT
metrics shared task and the broader metrics community we suggest the
following: (1) Pairwise accuracy has desirable properties as an evaluation
measure for metrics. Our bias-variance-noise decomposition shows that the
observed pairwise accuracy is very close to the true pairwise accuracy from a
noise-free, infinite test set setting (§4.4). We suggest the use of pairwise
accuracy as it reflects metric performance well (which may be verified using
this analysis). As a normalized form of pairwise accuracy, Kendall’s $\tau$ is
also a suitable measure. (2) Since pairwise accuracy is computed against noisy
human predictions, on average, it should be impossible for metrics to achieve
a perfect accuracy. We suggest providing an upper bound of metric performance
(§4.2) to clarify how much improvement is possible for metrics on the dataset.

##  10 Related work

The fact that a manual evaluation can be weak, and an automatic one can be
better is gaining attention in the metrics community. Mathur et al. (2020b)
studied a disagreement between crowdworkers and metrics, and a reevaluation
favored the metrics over the human prediction. Recently, Freitag et al. (2021)
shows that metrics can achieve higher agreement with professional linguists
than crowdworkers in judging translation systems. Their results fit into our
formalization: if we assume professional linguists are unbiased, the bias and
variance properties of metrics combined are superior to those of crowdworkers.
Our analysis assumes that crowdworkers are unbiased, where they assume
professional linguists are instead.

We wish to highlight several works which inspired the elements of ours:
Chaganty et al. (2018) and Hashimoto et al. (2019) formalize metrics as
statistical estimators and provide understanding of their statistical
properties and limits. In the replication of ImageNet, Engstrom et al. (2020)
found that dataset bias accounted for classifier performance differences
between the original and the replicated dataset, and provide a decomposition
for the sources of error. In automated essay scoring, scorers are often
evaluated against noisy human judgment, and Loukina et al. (2020) developed
the PRMSE to calculate the MSE between scorer prediction and the true
judgment, rather than noisy judgment. Finally, in bioinformatics, Li et al.
(2020) derive an upper bound of the $R^{2}$ coefficient due to experimental
noise when regressing on experiment-derived results.

##  11 Conclusion

Through rigorous comparison between metrics, humans, and the perfect segment-
level annotator, we identify the settings where metrics outperform humans due
to a statistical advantage in variance. These results challenge the notion
that metrics are always secondary to human evaluation. Instead, we encourage
practitioners to understand when human evaluation is weak, and when metrics
are necessary. Finally, we hope to provide tools for analysis and future
directions for evaluation.

## Acknowledgments

Discussions with Nitika Mathur, Markus Freitag, and Thibault Sellam led to
several insights. Nelson Liu and Tianyi Zhang provided feedback on our first
draft, and anonymous reviewers provided feedback on the submitted draft.
Nanyun Peng advised the first author, and on this work. Alex Fabbri provided a
scored version of the SummEval dataset. We thank all who have made our work
possible.

## References

  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  SPICE: semantic propositional image caption evaluation.  In _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V_ , volume 9909 of _Lecture Notes in Computer Science_ , pages 382–398. Springer.
  * Barrault et al. (2019) Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019.  Findings of the 2019 conference on machine translation (WMT19).  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 1–61, Florence, Italy. Association for Computational Linguistics.
  * Berg-Kirkpatrick et al. (2012) Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012.  An empirical investigation of statistical significance in NLP.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 995–1005, Jeju Island, Korea. Association for Computational Linguistics.
  * Bojar et al. (2016a) Ondrej Bojar, C. Federmann, B. Haddow, Philipp Koehn, Matt Post, and Lucia Specia. 2016a.  Ten years of wmt evaluation campaigns: Lessons learnt.  In _Proceedings of the LREC 2016 Workshop “Translation Evaluation – From Fragmented Tools and Data Sets to an Integrated Ecosystem”_.
  * Bojar et al. (2017) Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017.  Results of the WMT17 metrics shared task.  In _Proceedings of the Second Conference on Machine Translation_ , pages 489–513, Copenhagen, Denmark. Association for Computational Linguistics.
  * Bojar et al. (2016b) Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016b.  Results of the WMT16 metrics shared task.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 199–231, Berlin, Germany. Association for Computational Linguistics.
  * Britz et al. (2017) Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le. 2017.  Massive exploration of neural machine translation architectures.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1442–1451, Copenhagen, Denmark. Association for Computational Linguistics.
  * Callison-Burch et al. (2006) Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006.  Re-evaluating the role of Bleu in machine translation research.  In _11th Conference of the European Chapter of the Association for Computational Linguistics_ , Trento, Italy. Association for Computational Linguistics.
  * Card et al. (2020) Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020.  With little power comes great responsibility.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9263–9274, Online. Association for Computational Linguistics.
  * Chaganty et al. (2018) Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018.  The price of debiasing automatic metrics in natural language evalaution.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 643–653, Melbourne, Australia. Association for Computational Linguistics.
  * Chaganty et al. (2017) Arun Chaganty, Ashwin Paranjape, Percy Liang, and Christopher D. Manning. 2017.  Importance sampling for unbiased on-demand evaluation of knowledge base population.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1038–1048, Copenhagen, Denmark. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Domingos (2000) Pedro M. Domingos. 2000.  A unified bias-variance decomposition and its applications.  In _Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 - July 2, 2000_ , pages 231–238. Morgan Kaufmann.
  * Dror et al. (2018) Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018.  The hitchhiker’s guide to testing statistical significance in natural language processing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1383–1392, Melbourne, Australia. Association for Computational Linguistics.
  * Efron and Tibshirani (1993) Bradley Efron and Robert Tibshirani. 1993.  _An Introduction to the Bootstrap_.  Springer.
  * Engstrom et al. (2020) Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Jacob Steinhardt, and Aleksander Madry. 2020.  Identifying statistical bias in dataset replication.  _CoRR_ , abs/2005.09619.
  * Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1074–1084, Florence, Italy. Association for Computational Linguistics.
  * Fabbri et al. (2020) Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020.  Summeval: Re-evaluating summarization evaluation.  _arXiv preprint arXiv:2007.12626_.
  * Freitag et al. (2019) Markus Freitag, Isaac Caswell, and Scott Roy. 2019.  APE at scale and its implications on MT evaluation biases.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_ , pages 34–44, Florence, Italy. Association for Computational Linguistics.
  * Freitag et al. (2021) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.
  * Freitag et al. (2020) Markus Freitag, David Grangier, and Isaac Caswell. 2020.  BLEU might be guilty but references are not innocent.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 61–71, Online. Association for Computational Linguistics.
  * Gao et al. (2020) Yang Gao, Wei Zhao, and Steffen Eger. 2020.  SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1347–1354, Online. Association for Computational Linguistics.
  * Gatt and Krahmer (2018) Albert Gatt and Emiel Krahmer. 2018.  Survey of the state of the art in natural language generation: Core tasks, applications and evaluation.  _J. Artif. Intell. Res._ , 61:65–170.
  * Graham and Baldwin (2014) Yvette Graham and Timothy Baldwin. 2014.  Testing for significance of increased correlation with human judgment.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 172–176, Doha, Qatar. Association for Computational Linguistics.
  * Graham et al. (2016) Yvette Graham, Timothy Baldwin, Meghan Dowling, Maria Eskevich, Teresa Lynn, and Lamia Tounsi. 2016.  Is all that glitters in machine translation quality estimation really gold? In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 3124–3134, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Graham et al. (2013) Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013.  Continuous measurement scales in human evaluation of machine translation.  In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_ , pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics.
  * Hashimoto et al. (2019) Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. 2019.  Unifying human and statistical evaluation for natural language generation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1689–1701, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Jia et al. (2019) Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019.  Certified robustness to adversarial word substitutions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4129–4142, Hong Kong, China. Association for Computational Linguistics.
  * Johari et al. (2017) Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. 2017.  Peeking at A/B tests: Why it matters, and what to do about it.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017_ , pages 1517–1525. ACM.
  * Keith and O’Connor (2018) Katherine Keith and Brendan O’Connor. 2018.  Uncertainty-aware generative models for inferring document class prevalence.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4575–4585, Brussels, Belgium. Association for Computational Linguistics.
  * Koehn (2004) Philipp Koehn. 2004.  Statistical significance tests for machine translation evaluation.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 388–395, Barcelona, Spain. Association for Computational Linguistics.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_ , pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.
  * Läubli et al. (2018) Samuel Läubli, Rico Sennrich, and Martin Volk. 2018.  Has machine translation achieved human parity? a case for document-level evaluation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4791–4796, Brussels, Belgium. Association for Computational Linguistics.
  * Lavie and Agarwal (2007) Alon Lavie and Abhaya Agarwal. 2007.  METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments.  In _Proceedings of the Second Workshop on Statistical Machine Translation_ , pages 228–231, Prague, Czech Republic. Association for Computational Linguistics.
  * Li et al. (2020) Gang Li, Jan Zrimec, Boyang Ji, Jun Geng, Johan Larsbrink, Aleksej Zelezniak, Jens Nielsen, and Martin KM Engqvist. 2020.  Performance of regression models as a function of experiment noise.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Loukina et al. (2020) Anastassia Loukina, Nitin Madnani, Aoife Cahill, Lili Yao, Matthew S. Johnson, Brian Riordan, and Daniel F. McCaffrey. 2020.  Using PRMSE to evaluate automated scoring systems in the presence of label noise.  In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 18–29, Seattle, WA, USA → Online. Association for Computational Linguistics.
  * Ma et al. (2018) Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018.  Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 671–688, Belgium, Brussels. Association for Computational Linguistics.
  * Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019.  Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 62–90, Florence, Italy. Association for Computational Linguistics.
  * Mathur et al. (2020a) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a.  Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4984–4997, Online. Association for Computational Linguistics.
  * Mathur et al. (2020b) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and OndÅ™ej Bojar. 2020b.  Results of the wmt20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 688–725, Online. Association for Computational Linguistics.
  * Neubig et al. (2019) Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019.  compare-mt: A tool for holistic comparison of language generation systems.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 35–41, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ng and Abrecht (2015) Jun-Ping Ng and Viktoria Abrecht. 2015.  Better summarization evaluation with word embeddings for ROUGE.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Reiter (2018) Ehud Reiter. 2018.  A structured review of the validity of BLEU.  _Computational Linguistics_ , 44(3):393–401.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Shapira et al. (2019) Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019.  Crowdsourcing lightweight pyramids for manual summary evaluation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 682–687, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Tao et al. (2018) Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018.  RUBER: an unsupervised method for automatic evaluation of open-domain dialog systems.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 722–729. AAAI Press.
  * Toral et al. (2018) Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018.  Attaining the unattainable? reassessing claims of human parity in neural machine translation.  In _Proceedings of the Third Conference on Machine Translation, Volume 1: Research Papers_ , pages 113–123, Belgium, Brussels. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.

##  Appendix A Equivalence between optimal prediction and true system
differences

There is a slight difference between the definition of the true difference in
(5) which we can alternatively define as

$\Delta^{H}_{S,S^{\prime}}=\text{sign}(\mathbb{E}[\widehat{\mu^{H}_{S}}-\widehat{\mu^{H}_{S^{\prime}}}])$
(19)

and the definition of the optimal prediction $\Delta^{H*}_{S,S^{\prime}}$ in
(12), which is positive when

$P_{\mathcal{X}}(\widehat{\mu^{H}_{S}}-\widehat{\mu^{H}_{S^{\prime}}}>0)>\frac{1}{2}$
(20)

and the two are not immediately equivalent. However, if we assume that the
central limit theorem applies (which can be reasonable as our sample means
always have $n>100$) and
$X=\widehat{\mu^{H}_{S}}-\widehat{\mu^{H}_{S^{\prime}}}$ is normal, the CDF of
$X$ is

$F(x)=\phi((x-\mathbb{E}[X])/\text{Var}(X))$ (21)

where $\phi$ is the CDF of the standard normal distribution. Since the
standard normal is centered and symmetric, $\phi(x)>1/2\iff x>0$. Together we
have

$F(x)>\frac{1}{2}\iff\mathbb{E}[X]>x$ (22)

where for $x=0$ the left and right hand sides are equivalent to (20) and (19),
respectively.

##  Appendix B Convergence of metric predictions to the main prediction

A key assumption in interpreting the results from the bias-variance-noise
decomposition in §4 is that as system-level metrics have access to more
outputs for evaluation, metric predictions converges onto the main prediction.

For many metrics, their system-level score is the mean of their segment-level
scores (e.g. BLEURT, BERTscore, ROUGE etc.). This is true for all
summarization metrics. For these metrics, assuming the central limit theorem
allows us to prove that metrics converge to the main prediction, similar to
the proof in Appendix A. However, some MT metrics (BLEU, TER, and chrF) are
not simple averages of their segment-level scores, making them harder to
analyze.

For system-level metrics that are not simple averages, we analytically observe
that their aggregation method is similar to a mean (e.g. BLEU is a macro-
average). We empirically verify that as the system-level metric evaluates on
more test set outputs, their pairwise predictions converge to the main
predictions. Refer to Figures 3 and 4.

Figure 3: Average agreement of the main prediction to metric predictions
computed from varying test set sizes in WMT. The main predictions were derived
from all of our data. Each point was an estimated with 10K bootstrap trials.
As the size of the test set increases, we see that the agreement monotonically
increases. Note that only BLEURT and BERTscore  are means of their segment-
level scores.  Figure 4: Average agreement of the main prediction to metric
predictions evaluated on varying test set sizes in SummEval. The main
predictions were derived from all of our data. Each point was an estimated
with 10K bootstrap trials. As the size of the test set increases, we see that
the agreement monotonically increases. Note that all metrics are means of
their segment-level scores.

##  Appendix C Efficiency ratios for the perfect annotator

With repeat human judgments for a given output example, we can estimate the
variance of the perfect annotator (or true segment-level score variance) in
WMT and SummEval. For WMT, we use only valid judgments (SYSTEM and REPEAT
judgments), and discard all attention check judgments (BAD_REF judgments). For
SummEval, we use the dataset as is.

In WMT, we analyze all the to-English data grouped by year. We believe this
grouping is appropriate because the to-English evaluation is batched together
every year. Direct assessment, which WMT uses to collect human judgments
Graham et al. (2013), is a score assigned by crowdworkers to an English
translation while referring to an English reference, requiring only
monolingual knowledge.

| 2016 | 2017 | 2018 | 2019
---|---|---|---|---
$\sqrt{\text{Var}(H(x))}$ | 30.01 | 29.65 | 28.21 | 28.81
$\sqrt{\mathbb{E}[\text{Var}(H(x)|x)]}$ | 17.53 | 22.96 | 19.57 | 21.42
$\sqrt{\text{Var}(P(x))}$ | 24.36 | 18.76 | 20.33 | 19.27
$\text{Var}(H(x))/\text{Var}(P(x))$ | 1.52 | 2.50 | 1.93 | 2.24
Table 3: Step-by-step derivation for the efficiency ratio $r$ (fourth row) of the perfect annotator estimator for WMT16-19 as defined in §4.1. Square roots are taken so that values are in terms of the original units (standard deviations, judgments range from 0-100). These were calculated on to-English data only.  | Expert | Turker
---|---|---
$\sqrt{\text{Var}(H(x))}$ | 0.717 | 0.745
$\sqrt{\mathbb{E}[\text{Var}(H(x)|x)]}$ | 0.293 | 0.475
$\sqrt{\text{Var}(P(x))}$ | 0.655 | 0.574
$\text{Var}(H(x))/\text{Var}(P(x))$ | 1.201 | 1.686
Table 4: Step-by-step derivation for the efficiency ratio $r$ (fourth row) of
the perfect annotator estimator for SummEval as defined in §4.1. Square roots
are taken so that values are in terms of the original units (standard
deviations, judgments range from 1-5). Note that there is little agreement
between experts and turkers at the system level.

##  Appendix D SummEval analysis results

The main analyses in §5 and §6 are presented for SummEval here. When comparing
expert humans to metrics, no metric comes close to expert performance at any
number of expert judgments. For the power analysis, small differences are also
hard to detect, similar to the findings in WMT. Note that while the perfect
expert requires relatively less judgments compared to the perfect crowdworker
in WMT, judgments from experts are likely to be much more expensive.

Figure 5: Comparison of metrics to human and perfect annotator estimators with
varying number of judgments in SummEval. Errors are adjusted to an idealized
setting where true predictions are used for evaluation and metrics are
computed on infinite test sets; here metric predictions become constant, so
their errors are constant. No metric comes close to expert performance at any
number of judgments (ROUGE, the best performing summarization metric, has
error $0.221$).  Table 5: Power analysis for the number of judgments needed
from the perfect expert to give a pairwise judgment between two systems at
$.9$ accuracy ($\alpha=0.05,\beta=0.95$) under ttest assumptions (normality,
equal variance) in SummEval. SummEval ratings are on a 1-5 scale, and the true
segment quality variance was $0.655$. Darker cells indicate less feasible
experiments, and the colors are set on a log scale.

##  Appendix E Metric and human significance breakdown

For the pairwise examples in WMT, we provide the co-occurrence of significance
for metric and human estimators. Refer to Figures 6 and 7 for analyses on
BERTscore and BLEURT, respectively.

Figure 6: Co-occurrence of BERTscore and human significance on pairs in
WMT16-19. Pairs are ordered so that the human difference in system quality is
always positive. Significance is tested with a one-sided bootstrap resampling
test, in the direction of the difference for both humans and metrics with 1K
trials at $\alpha=0.05$. BERTscore achieves significance more than half of the
time when humans cannot. Figure 7: Co-occurrence of BLEURT and human
significance on pairs in WMT19. Pairs are ordered so that the human difference
in system quality is always positive. Significance is tested with a one-sided
bootstrap resampling test, in the direction of the difference for both humans
and metrics with 1K trials at $\alpha=0.05$. BLEURT achieves significance more
than half of the time when humans cannot.