# Resonant tunneling in graphene-ferroelectric-graphene junctions

David Koprivica  Raymond and Beverly Sackler School of Physics and Astronomy,
Tel-Aviv University, IL-69978 Tel Aviv, Israel  Eran Sela  Raymond and Beverly
Sackler School of Physics and Astronomy, Tel-Aviv University, IL-69978 Tel
Aviv, Israel

###### Abstract

We study tunnel junctions consisting of a two-dimensional ferroelectric
material sandwiched between graphene electrodes. We formulate a theory for the
interplay of the polarization and induced free charges in such devices, taking
into account quantum capacitance effects. We predict a gate sensitive voltage
difference across the polar domains, which can be measured using electrostatic
force microscopy. Incorporating this electrostatic theory in the tunneling
current-voltage characteristics, we identify a resonance peak associated with
aligned Dirac cones as a highly sensitive probe of the polarization. This
opens the way for device applications with few atom-thick polar layers acting
as readable ultra-high-density memory.

##  I Introduction

Recent breakthroughs in two-dimensional (2D) van der Waals materials led to
the experimental realization of a new form of ferroelectricity (FE) Li and Wu
(2017); Yasuda  _et al._ (2021); Stern Vizner  _et al._ (2021); Weston  _et
al._ (2022); de la Barrera  _et al._ (2021); Deb  _et al._ (2022). This newly
discovered interfacial ferroelectricity results from polar stacking
configurations that break inversion symmetry, such as in AB or BA stacked
hexagonal boron nitride (h-BN) bilayer Yasuda  _et al._ (2021); Stern Vizner
_et al._ (2021) or other 2D materials Weston  _et al._ (2022); de la Barrera
_et al._ (2021); Deb  _et al._ (2022). Remarkably, the resulting polarization
can be flipped Stern Vizner  _et al._ (2021) by a relative sliding of the
layers by a single atomic distance.

Understanding the response to electric fields and the nature of
ferroelectricity Wu and Li (2021); Zhong  _et al._ (2021) in these experiments
requires further study and a possible distinction of these various systems
Bennett (2022). In the bulk of twisted interfaces there is a Moiré-triangular
lattice of alternating AB and BA regions which expand or shrink at finite
field de la Barrera  _et al._ (2021). Yet, _in average_ over many Moiré unit
cells there is no remnant polarization at zero field Bennett and Remez (2022).
On the other hand, in Refs. Yasuda  _et al._ , 2021; Stern Vizner  _et al._ ,
2021 hysteresis was seen, and specifically in Ref. Stern Vizner  _et al._ ,
2021, the flipping of a polar domain was attributed to the sliding of a domain
wall near the edge of the bilayer, rather than in the bulk Bennett (2022).
Leaving the precise sliding FE mechanism aside, here we shall be concerned
only with uniform AB and BA polar interfaces.

Ferroelectric tunnel junctions, consisting of a thin FE material sandwiched
between two electrodes, permit reading the polarization via tunnelling
electroresistance (TER) Pantel and Alexe (2010); Garcia and Bibes (2014); Tao
and Wang (2016); Kang  _et al._ (2020); Yan  _et al._ (2022). What is the fate
of interfacial polarization within a tunnel junction?

In this work, we theoretically study 2D tunnel junctions consisting of uniform
AB or BA polar interfaces sandwiched by graphene electrodes, as shown in Fig.
1(a). Assuming a given value for the bare polarization $V_{P}^{(0)}$, which,
in principle, can be determined by ab initio methods, we focus on the
interplay of the polarization and screening charges forming on the graphene
electrodes. Related questions were recently addressed within density
functional theory (DFT) Yang  _et al._ (2021), where it was found that whereas
metallic electrodes significantly affect the polarization of a bilayer h-BN,
adding graphene spacers between the FE and metals restores the polarization
and results in a significant TER. Here, we provide a phenomenological
electrostatic model. We find that when one of the electrode’s Fermi level is
tuned to the Dirac point, the electronic equilibration is dominated by quantum
capacitance, and then a polar domain associated local voltage can be measured
across the device.

We focus on basic mechanisms allowing to read out the polarization direction
and magnitude from the current-voltage characteristics. One TER mechanism
results from the dependence of the electrostatic tunneling barrier on the
polarization orientation. We discuss a model allowing to estimate this barrier
modulation in the 2D limit.

Moreover, a graphene-FE-graphene junction allows for a more sensitive finite
voltage TER mechanism due to 2D momentum conservation. Resonance tunneling
peaks in the $I(V)$ characteristics were identified both in planar 2D
junctions of semiconductor heterostructures Eisenstein  _et al._ (1991, 1992),
and more recently in 2D materials Britnell  _et al._ (2013); Mishchenko  _et
al._ (2014); Chen (2021), specifically for graphene-h-BN-graphene Britnell
_et al._ (2013). There, the h-BN material simply acts as a barrier Yan  _et
al._ (2021, 2022, 2020), whereas the resonance peaks emerge due to momentum
conserving tunneling between the graphene electrodes. The device we consider
is almost the same as in Ref. Britnell  _et al._ , 2013, where now the barrier
consists of a parallel stacked h-BN bilayer supporting the polar AB or BA
interfaces. Such a device is expected to be a perfect candidate to employ
resonance tunneling peaks as a sensitive probe of interfacial polarization,
which essentially acts as an internal voltage that shifts the resonance. We
indeed find using our self-consistent model a sizable shift of the resonance
peak Britnell  _et al._ (2013); Mishchenko  _et al._ (2014); Chen (2021) for
the two polarization directions.

Figure 1: (a) FE tunnel junction: a pair of tunnel coupled graphene sheets and
a bottom gate are modeled by a pair of plate capacitors $C_{0},C_{g}$. The
polar bilayer interface is stacked inside the graphene-graphene capacitor and
leads to an internal voltage denoted $V_{P}$. (b) Schematic free energy of the
polarization of the interface, $V_{P}$. The system consists of two degenerate
minima corresponding to stacking configurations AB and BA. In our model Eq.
(1) the polarization, $V_{P}$, which takes the value $V_{P}^{(0)}$ for the
bare bilayer, interacts with the electric field due to the electrons in the
graphene sheets and gate.

The paper is organized as follows. After introducing the model in Sec. II, we
study the Kelvin probe voltage in Sec. III. In Sec. IV we estimate the ratio
of the tunneling coefficients in our device. In Sec. V we consider momentum
conserving tunneling and focus on resonance peaks as a means to detect the
polarization. We summarize in Sec. VI.

##  II Model

As shown in Fig. 1(a), we consider a tunneling junction consisting of a 2D
polar interface encapsulated between two graphene sheets. Our theory is not
restricted to a specific material, but for definiteness we consider parallel
stacked bi-layer h-BN, having two polar stacking configurations denoted AB and
BA exhibiting a finite out of plane polarization $P$ which is accounted for by
an internal voltage denoted $V_{P}$, see Fig. 1(b). A tunneling current is
enabled by a bias voltage $V_{b}$. The bottom graphene sheet is gated by a
voltage $V_{g}$. The energy describing the system is given by

$\displaystyle E(Q,Q_{g},V_{P})$ $\displaystyle=$
$\displaystyle\frac{Q^{2}}{2C_{0}}-Q(V_{b}+V_{P})$ (1) $\displaystyle+$
$\displaystyle\frac{Q_{g}^{2}}{2C_{g}}-Q_{g}V_{g}$ $\displaystyle+$
$\displaystyle\frac{C_{0}}{2}\frac{\epsilon_{r}}{\epsilon_{r}-1}(V_{P}-pV_{P}^{(0)})^{2}$
$\displaystyle+$ $\displaystyle E_{G}(Q)+E_{G}(-Q+Q_{g}).$

Here $C_{0}$ is the capacitance between the two graphene sheets, and $C_{g}$
is the capacitance between the bottom graphene sheet and a gate. We measure
energy, charge and capacitance per unit area.

The third line encapsulates a quadratic expansion of the free energy Stern
Vizner  _et al._ (2021) around the minima for either one of the two stacking
configurations denoted by $p=\pm 1$. By construction, the bare KP voltage
(i.e. without the electrodes) is given by $\pm V_{P}^{(0)}$ for AB or BA
stacking, see Fig. 1(b). Its value $V_{P}^{(0)}\approx 120{\rm{mV}}$ was
measured directly using Kelvin probe force microscopy Stern Vizner  _et al._
(2021). In addition, this term renormalizes the capacitance $C_{0}\to
C=C_{0}\varepsilon_{r}$ by the dielectric constant of the material.

The last line represents the quantum capacitance of the graphene sheets, with

$E_{G}(Q)=\frac{2}{3}\beta|Q|^{\frac{3}{2}},$ (2)

and $\beta=\sqrt{\frac{\pi}{e^{3}}}\hbar v_{F}$. $E_{G}(Q)$ is the total
kinetic energy density measured from the Dirac point, valid in the vicinity of
the Dirac point, and $Q>0(Q<0)$ refers to holes (electrons).

The goal of this electrostatic model is to allow variations of the
polarization of the AB or BA interface due to electric fields produced by the
graphene electrodes.

The equilibrium configuration is obtained by minimizing the energy with
respect to the charge induced on the top sheet $Q$ and on the gate $Q_{g}$,
and the polarization of the interface $V_{P}$ . The equation $\frac{\partial
E}{\partial V_{P}}=0$ gives

$V_{P}=pV_{P}^{(0)}+\frac{(\epsilon_{r}-1)Q}{\epsilon_{r}C_{0}},$ (3)

showing that the internal polarization is affected by the electrodes. Defining
the Fermi energy

$E_{F}(Q)=-e\frac{\partial E_{G}(Q)}{\partial
Q}=-e~{}{\rm{sign}}(Q)\beta|Q|^{1/2},~{}~{}~{}(e>0),$ (4)

the equations $\frac{\partial E}{\partial Q}=\frac{\partial E}{\partial
Q_{g}}=0$ yield

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\frac{Q}{C}-V_{b}-pV_{P}^{(0)}-E_{F}(Q)/e+E_{F}(Q_{g}-Q)/e,$ (5)
$\displaystyle 0$ $\displaystyle=$
$\displaystyle\frac{Q_{g}}{C_{g}}-V_{g}-E_{F}(Q_{g}-Q)/e.$ (6)

Eqs. (5) and (6) are subsequently solved numerically for $Q$ and $Q_{g}$ as
function of $V_{g}$ and $V_{b}$ for either sign of the polarization $p=\pm 1$.

##  III Kelvin probe voltage

Figure 2: (a) Schematic view of the Kelvin probe measurement device. We define
the Kelvin probe sensitivity (KPS) $f=\delta V_{KP}/(2V_{P}^{(0)})$ [see Eq.
(8)] as the relative difference in the voltage measured for the two
polarization orientations of the two stacking configurations $AB$ and $BA$.
(b) KPS versus gate and bias voltage. Maximal sensitivity occurs when one of
the graphene sheets is at charge neutrality. (c) Cut along a fixed gate
voltage showing a double peak. A fit (dashed line) to Eq. (10) shows that the
KPS decays at large bias as $V_{b}^{-\frac{1}{2}}$. In this and all subsequent
calculations we use $d=5{\rm{nm}}$, $d_{g}=90{\rm{nm}}$ (width of gate
capacitor) and $\epsilon_{r}=3.8$.

In this section we calculate the KP voltage, which can be measured via
scanning atomic force microscopy (AFM) as function of gate and bias voltages,
in a setup as in Fig. 2(a). Such measurements were performed for the bare
ferroelectric material Yasuda  _et al._ (2021); Stern Vizner  _et al._ (2021);
Weston  _et al._ (2022). Our theory addresses the renormalization of the
internal polarization $V_{P}$ and the resulting total KP voltage $V_{KP}$
across the electrodes.

The total voltage drop measured in a Kelvin probe setup as in Fig. 2(a) is
given by

$V_{KP}\equiv V_{P}-\frac{Q}{C_{0}}=pV_{P}^{(0)}-\frac{Q}{C},$ (7)

where we used Eq. (3). We subsequently focus on the difference of $V_{KP}$ for
the two polarization orientations, $\Delta V_{KP}=V_{KP}^{(+)}-V_{KP}^{(-)}$
which can be measured by scanning the AFM tip across a stacking domain wall,
see Fig. 2(a).

It is convenient to measure $\Delta V_{KP}$ in units of the bare polar domains
voltage by defining the Kelvin probe sensitivity (KPS)

$f\equiv\frac{\Delta
V_{KP}}{2V_{P}^{(0)}}=1-\frac{Q^{+}-Q^{-}}{2CV_{P}^{(0)}}.$ (8)

The second equality, obtained from Eq. (7), shows that $f\to 1$ in the absence
of charges on the electrodes.

The numerical solution for $\Delta V_{KP}$ and hence the KPS is shown in Fig.
2(b) as function of bias and gate voltages. A cut with fixed $V_{b}$ in shown
in Fig. 2(c). We can see that $\Delta V_{KP}$ peaks when either one of the two
graphene sheets is at charge neutrality and takes a value which is of the
order of $V_{P}^{(0)}$ (and hence $f\lesssim 1$). Along these peaks the charge
transfer between the electrodes is small and primarily determined by quantum
capacitance, and hence it very inefficiently screens electrostatically the
internal polarization.

We now discuss the physics describing the tail and peaks of $\Delta V_{KP}$
(or equivalently $f$ in Fig. 2(c)). The tails can be analysed by considering
the quantum capacitance as a small perturbation. One can solve Eqs. (5) and
(6) up to a given order in $\beta$. In the absence of quantum capacitance
($\beta\to 0$) we have

$Q\to Q^{(0)}\equiv C(V_{b}\pm V_{P}^{(0)}),~{}~{}~{}Q_{g}\to
Q_{g}^{(0)}\equiv C_{g}V_{g}.$ (9)

Then $V_{KP}^{\pm}=V_{b}$, $\Delta V_{KP}=0$, and $f\to 0$. The vanishing KPS
is expected since the potential on the electrodes adjusts such as to
completely screen the internal polarization. Considering a small quantum
capacitance correction, $Q=Q^{(0)}+\delta Q$, $Q_{g}=Q_{g}^{(0)}+\delta
Q_{g}$, and expanding Eq. (5) to linear order in these deviations, we obtain
for $V_{b}\gg V_{P}^{(0)}$

$f\to\beta\sqrt{\frac{C}{V_{b}}}.$ (10)

This is confirmed as a dashed line in Fig. 2(c).

The height of the two peaks in Fig. 2(c) can be obtained by considering the
quantum capacitance of the corresponding neutralized graphene sheet as the
dominant term. Consider for example the $Q=0$ peak [nearly vertical peak ridge
in Fig. 2(b)]. We can then decouple Eqs. (5) and (6) by replacing the Fermi
energy of the bottom layer $E_{F}(Q_{g}-Q)$ by $E_{F}(Q_{g})$. We thus obtain
$0=\frac{Q}{C}-V_{b}-pV_{P}^{(0)}-E_{F}(Q)/e+E_{F}(Q_{g})/e$, yielding a peak
at $V_{b}=E_{F}(Q_{g})/e$ where for large $V_{g}$ we have $Q_{g}=C_{g}V_{g}$.
Solving the quadratic equation yields
$Q=p\left(-C\beta+\sqrt{(C\beta)^{2}+4CV_{P}^{(0)}}\right)^{2}/4$.
Substituting in Eq. (8), we have $f=1-\frac{Q}{CV_{P}^{(0)}}$. Expanding in
small $V_{P}^{(0)}$, yields

$\displaystyle
f=1-\frac{V_{P}^{(0)}}{C\beta^{2}}+\mathcal{O}((V_{P}^{(0)})^{2}).$ (11)

We can see that $C\beta^{2}$ sets a voltage scale below which quantum
capacitance effects set in. In our system

$\mathcal{V}\equiv C\beta^{2}=\frac{e}{16\pi
d\varepsilon_{0}}\left(\frac{4\pi\varepsilon_{0}\hbar
v_{F}}{e^{2}}\right)^{2}\sim 0.3V.$ (12)

Thus, the reason that the peaks in KPS approach nearly unity is that the
material property $V_{P}^{(0)}$ is small, but of the order of $\mathcal{V}$.

##  IV Polarization dependent tunneling coefficient

In this section we discuss the influence of the polarization on the tunneling
coefficient $|\mathcal{T}|^{2}$. This effect can be understood from a
polarization-dependent distortion of the electrostatic tunnel barrier
Zhuravlev  _et al._ (2005); Gerra  _et al._ (2007), modifying the tunneling
coefficient due to its exponential sensitivity $|\mathcal{T}|^{2}\propto
e^{-2\kappa d}$.

Let us denote the tunneling amplitudes for the two polarization orientations
by $\mathcal{T}^{\pm}$ and define the relative barrier modulation

$\eta=\frac{|\mathcal{T}^{+}|^{2}-|\mathcal{T}^{-}|^{2}}{|\mathcal{T}^{+}|^{2}+|\mathcal{T}^{-}|^{2}}.$
(13)

As a simple model providing an order of magnitude estimate for $\eta$,
consider the bottom and top graphene sheets to be located at the $z=\pm d/2$
planes, and the two h-BN layers at the $z=\pm d/6$ planes. The tunneling
amplitude from bottom to top is given via 3-rd order perturbation theory by
Brey (2014) $\mathcal{T}\propto\frac{t^{3}}{E(z=-d/6)E(z=d/6)}$, where $t$ are
nearest layer hopping amplitudes, and $E(z=\pm d/6)$ is the energy in the h-BN
layers. We denote by $E_{g}$ the energy gap of the bare h-BN layer at $Q=0$.
Adding the linear potential due to the charged electrodes,
$E(z)=E_{g}+\frac{eQz}{dC}$. The tunneling amplitude becomes
$\mathcal{T}^{\pm}\propto\left(E_{g}^{2}-\left(\frac{eQ^{\pm}}{6C}\right)^{2}\right)^{-1}$.
If the two directions of polarization lead to exactly opposite charge
transfer, $Q^{+}=-Q^{-}$, then $\mathcal{T}^{+}=\mathcal{T}^{-}$. In general
$\eta=0$ when inversion symmetry holds, namely when $V_{b}=0$ and $V_{g}=0$.
Finite barrier modulation results from $Q^{+}\neq-Q^{-}$,

$\eta\approx\frac{\left(\frac{eQ^{-}}{6C}\right)^{2}-\left(\frac{eQ^{+}}{6C}\right)^{2}}{2E^{2}_{g}}.$
(14)

For sufficiently large $V_{b}$, using Eq. (9) which ignores quantum
capacitance effects, we have $\eta=-e^{2}V_{b}V_{P}^{(0)}/(18E_{g}^{2})$
independently of $V_{g}$. Assuming $E_{g}\sim 1eV$, this result leads to a few
percent relative barrier modulation for a bias $V_{b}$ of few volts. The
smallness of the effect derives from the small ratio between the voltage
$V_{P}^{(0)}$ and the gap of h-BN, however this can be made larger in other
materials. Higher $\eta$ may also be obtained in an asymmetric device when one
electrode is weakly coupled while the other one is strongly coupled Rogée  _et
al._ (2022).

We note that the TER is defined [see Eq. (18] below and Refs. Tao and Wang,
2016; Kang  _et al._ , 2020; Yan  _et al._ , 2022) in terms of the currents at
finite bias voltage. Our definition of the relative barrier modulation in Eq.
(13) allows to separate the effect of the modulation of the barrier itself,
which leads to a relatively week contribution to TER, from the effects
associated with momentum conserving tunneling between two graphene sheets, to
be considered in the next section.

##  V Momentum conserving Tunneling

In this section we consider 2D momentum conserving tunneling through the polar
interface. In conventional tunnel junctions 2D momentum conservation leads to
a resonance peak in the $I(V)$ characteristics corresponding to two aligned
Dirac cones Britnell  _et al._ (2013); Mishchenko  _et al._ (2014); Chen
(2021). Our goal is to incorporate the interfacial polarization into the
resonant condition, yielding a sensitive probe of the polarization orientation
and magnitude.

Following the model outlined in Ref. Britnell  _et al._ , 2013, we start with
a Fermi golden rule expression for the tunneling current

$\begin{split}I^{(\pm)}=|\mathcal{T}^{\pm}|^{2}\sum_{\nu_{B},\nu_{T}}&\int
d^{2}k_{T}\int
d^{2}k_{B}\left(f_{B}(\varepsilon_{\vec{k}_{B},\nu_{B}})-f_{T}(\varepsilon_{\vec{k}_{T},\nu_{T}})\right)\\\
&(V_{q}(k_{B},k_{T}))^{2}\delta\left(\varepsilon_{\vec{k}_{B},\nu_{B}}-\varepsilon_{\vec{k}_{T},\nu_{T}}-eV_{KP}\right).\end{split}$
(15)

In this section, in order to disentangle the TER effects of the previous
chapter with effects of momentum conservation, we assume for simplicity
$\mathcal{T}^{+}=\mathcal{T}^{-}$. Here,
$f_{i}(\varepsilon)=(\exp[(\varepsilon-E_{F,i})/T]+1)^{-1}$ are Fermi
functions and $\varepsilon_{\vec{k}_{i},\nu_{i}}=\hbar
v_{F}\nu_{i}|\vec{k}_{i}|$ are the electron ($\nu=1$) and hole ($\nu=-1$)
bands in the bottom or top sheet $(i=B,T)$, respectively (see Fig. 3). This
process describes tunneling of an electron from an occupied momentum state
$\vec{k}_{B}$ in the bottom layer to an unoccupied state with momentum
$\vec{k}_{T}$ at the top layer, measured from the respective Dirac point in a
given valley. The scattering potential contains a phenomenological momentum
dependence
$V_{q}(k_{B},k_{T})=\frac{1}{q_{c}^{2}+(\vec{k}_{B}-\vec{k}_{T}-\vec{Q})^{2}}$.
Here $|\vec{Q}|=K\theta$ is a momentum shift between the top and bottom Dirac
points (at momentum $K$ in the Brillouin zone) due to a relative twist of the
layers by angle $\theta$. The limit $q_{c}\to 0$ corresponds to momentum
conserving tunneling. A finite $q_{c}$ phenomenologically describes non-
momentum-conserving tunneling processes e.g. due to short-range disorder or
the Moiré pattern of either the h-BN or its interface with graphene. We take
$q_{c}^{-1}=12nm$ Britnell  _et al._ (2013). The typical energy band diagram
in the presence of a bias voltage is shown in Fig. 3(a). Since
$E_{F,T}=E_{F}(Q)$ and $E_{F,B}=E_{F}(Q_{g}-Q)$ mark the distance of these
Fermi levels from the corresponding Dirac point; from Eq. (5) it follows that
the energy difference of the two Dirac points is given precisely by $eV_{KP}$,
as marked in Fig. 3(a).

Figure 3: (a) Typical energy band diagram (momentum shift for clarity). The
Fermi levels of the bottom and top layers $E_{F,i}$ ($i=B,T$) are determined
from the charge densities $Q,Q_{g}$. The $KP$ voltage $V_{KP}$ sets the energy
misalignment of the Dirac points. (b) At zero relative twist angle equi-energy
contours are concentric circles. At $V_{KP}=0$ these rings exactly overlap for
all energies (see marked energy diagram in Fig. 4), leading to a resonant
momentum conserving tunneling in the entire voltage window between the two
Fermi energies. (c) For finite twist the equi-energy circles are non-
concentric, and resonance peaks occur when these circles are tangential for
all energies (see marked energy diagrams in Fig. 5), when $eV_{KP}=\pm\Delta$
is satisfied (see Eq. (19)).

###  V.1 Zero twist angle

When the two graphene sheets are perfectly aligned we have $\vec{Q}=0$. For
any energy within the voltage window, the two momenta $\vec{k}_{B}$ and
$\vec{k}_{T}$ belong to two concentric circles in momentum space, as denoted
in Fig. 3(b). The energy displacement of the Dirac cones, $eV_{KP}$, is
controlled by the bias voltage. When $V_{KP}=0$ these two circles in momentum
space overlap, for all energies, leading to a resonant current peak.

The resonance peak can be obtained by performing the angular integration,
leading to

$\begin{split}I^{(\pm)}\propto\int_{-\infty}^{\infty}dk_{T}dk_{B}&\frac{(f_{B}-f_{T})|k_{B}||k_{T}|(q_{c}^{2}+k_{B}^{2}+k_{T}^{2})}{\left(q_{c}^{2}+(k_{B}-k_{T})^{2}\right)^{\frac{3}{2}}\left(q_{c}^{2}+(k_{B}+k_{T})^{2}\right)^{\frac{3}{2}}}\\\
&\delta\left(\hbar v_{F}(k_{B}-k_{T})-eV_{KP}\right).\end{split}$ (16)

This yields a single integral that we evaluate numerically.

Figure 4: Color plot showcasing the calculated current $I^{(+)}$ for one
orientation ($p=+1$) of the polarization, as a function of the gate and bias
voltage, for the untwisted case $\theta=0$. On top of it are overlayed the
curves for which the top and/or the bottom graphene electrodes are at the
Dirac point (green $Q_{t}^{(p)}=0$ and yellow $Q_{b}^{(p)}=0$ curves,
respectively), as well as the curves where the graphene sheets are
electrostatically aligned (red curve, $V_{KP}^{(p)}=0$). The right inset shows
a cut of the $I-V$ curve at $V_{g}=20V$ for both $p=\pm 1$.The two resonance
peaks match with the band alignment condition $V^{(\pm)}_{KP}=0$ and their
line shape is given by Eq. (16). The top inset displays the Dirac bands
(horizontally shifted for clarity) for $p=1$, for points 1-6 corresponding to
$V_{g}=20V$ and $V_{b}=-1,-0.35,-0.17,0,0.34,1.0$V. The resonance case
corresponds to point 3 (blue outline). The green dashed lines represent the
energy midway between the two Dirac points, at which the tunneling process
conserves momentum. Tunneling through this energy state begins at point 5, at
which the plateau in the right inset ends.

Fig. 4 shows a color map of the current $I$ versus $V_{b}$ and $V_{g}$ for a
specific polarization direction. As a guide to the eye, this figure displays
the charge neutrality curves of either bottom or top graphene sheets. We also
plot the curve where $V_{KP}=0$. A cut of the $I(V_{b})$ curve for either
direction of the polarization for fixed $V_{g}$ is shown in the right-top
inset of Fig. 4. We can see a pronounced resonance peak positioned precisely
at $V_{b}$ at which $V_{KP}=0$ where the Dirac cones overlap. This occurs for
different resonant voltages for the two polarization orientations.

The line shape of the peak versus $V_{b}$ stems from the implicit dependence
of $V_{KP}$ on the latter, and takes the form

$I(V_{b})|_{V_{KP}\approx 0}\propto((\hbar
v_{F}q_{c})^{2}+(eV_{KP})^{2})^{-3/2}.$ (17)

We can also observe a plateau in the $I(V_{b})$ curve in Fig. 4 which
terminates at $V_{b}\approx 0.4V$ (at point 5). As shown in the energy
diagrams in Fig. 4, this corresponds to a threshold for transport through the
midway energy between the two Dirac points, see green dashed lines. At the
threshold $V_{b}\approx 0.4V$ this specific energy enters into the voltage
window Feenstra  _et al._ (2012).

We note that the shifted resonance peaks lead to a significant TER Wu  _et
al._ (2020); Yan  _et al._ (2021, 2022), defined as

${\rm{TER}}=\frac{|I^{(+)}-I^{(-)}|}{\min(|I^{(+)}|,|I^{(-)}|)},$ (18)

as shown in Fig. 4 (right-bottom inset). The role of our definition of the
relative barrier modulation $\eta$ in Sec. IV meant to separate this effect
from the modification of the tunneling amplitudes for the two polarization
directions.

###  V.2 Finite twist angle

Tunneling between twisted graphene sheets has been discussed in numerous
theory Feenstra  _et al._ (2012); de la Barrera  _et al._ (2014) and
experimental Mishchenko  _et al._ (2014) works. Whereas at zero twist, there
is a single resonance condition $V_{KP}=0$ at which the two Dirac cones
completely overlap, at a finite twist angle there is a momentum shift between
the Dirac points, given by $|\vec{Q}|=K\theta$ for $\theta\ll 2\pi$. Thus
equi-energy contours of the bottom and top layers are non-concentric circles,
see Fig. 3(c). Upon tuning their relative energy $eV_{KP}$ there are two
situations where the Dirac cones are tangential,

$eV_{KP}=\pm\hbar v_{F}\theta K\equiv\pm\Delta.$ (19)

The calculated current with a finite twist angle of $\theta=3.5^{o}$ is shown
in the color plot of Fig. 5. We can clearly see that the pair of resonance
peaks, also shown along a cut of fixed gate, overlaps with the condition Eq.
(19). The corresponding dispersion relations are shown in the top insets of
Fig. 5. The sensitivity of the resonant peaks to different orientation of the
polarization can be enhanced by tuning the gate such that the resonance peaks
will occur near the charge neutrality condition of one graphene sheet. This is
indeed seen for the cut of the $I(V_{b})$ curve at $V_{g}=9.5V$.

Figure 5: Color plot showcasing the calculated current as a function of gate
and bias voltage for specific orientation of the polarization $(p=+1)$, with
twist angle $\theta=3.5^{o}$ between the graphene sheets. The resonance peak
splitting to two peaks at $eV_{KP}=\pm\Delta$ is observed on the color plot.
The right insets show two cuts at fixed gate voltages, plotting both $I^{(+)}$
and $I^{(-)}$. We can see that each peak of either $eV_{KP}=\Delta$ or
$eV_{KP}=-\Delta$ further splits according to the direction of the
polarization $(p=\pm 1)$. The top inset shows the calculated band structure
alignment (for points 1-4 corresponding to $V_{g}=20V$ and
$V_{b}=-1.14,-0.70,-0.35,0.70$V).

###  V.3 Dependence on G-G twist angle

We now discuss the twist-angle dependence of the resonance peaks. At
$\theta=0$ we observed a resonance peak whose position
$V_{b,{\rm{peak}}}^{(p)}$ depends on the polarization orientation $p=\pm$. The
peak separation $\Delta
V_{b}=|V_{b,{\rm{peak}}}^{(+)}-V_{b,{\rm{peak}}}^{(-)}|$ is a sensitive probe
of the polarization, see Figs. 4 and 5.

The peak separation is plotted in Fig. 6 as a function of $V_{g}$ and
$\theta$. For zero twist angle we have seen in the inset of Fig. 4 that
$\Delta V_{b}\sim 0.15V$ for $V_{g}=20V$. We can see in Fig. 6 that at
$\theta=0$ this significant peak separation persists for any value of $V_{g}$,
and peaks near $V_{g}=0$. But at finite twist angle, the peak separation
becomes visible only for specific values of $V_{g}$. For example for
$\theta=3.5^{o}$, as we have seen in the inset of Fig. 5, the peak separation
is resolved only near $V_{g}\sim\pm 10V$. As discussed, this maximal
sensitivity occurs when both Eq. (19) holds and one graphene sheet is at
charge neutrality $Q_{b}=0$. Therefore, in order to achieve good sensitivity
in terms of resonance peak separation, one should operate such a device at
specific twist angle dependent value of the gate voltage.

The slope of the high sensitivity spokes in the $(V_{g},\theta)$ plane can be
obtained analytically. By solving Eqs. (5) and (6), with the added condition
of Eq. (19), we obtain the expression for the bias voltage at which the peak
occurs as a function of $V_{g}$ and $\theta$. For a given twist angle, the
difference of such bias voltage for the two orientations of the polarization,
representing the peak separation, is extremized analytically with respect to
$V_{g}$. For large enough gate voltages (outside the range where both graphene
sheets are near the charge neutrality), we find

$V_{g}\simeq\pm\frac{C}{C_{g}}\frac{\hbar v_{F}}{e}K\theta,$ (20)

which perfectly fits the spokes in Fig. 6 (not shown).

Figure 6: Peak separation $\Delta V_{b}$ as function of gate voltage and twist
angle $\theta$. The peak separation is defined from the $I(V)$ characteristics
as in the bottom inset of Fig. 5. For each $V_{g}$ and $\theta$, $\Delta
V_{b}$ is the largest among the two peak separations.

##  VI Summary

To summarize, in this work we studied graphene-FE-graphene junctions. We
focused on the self-consistent interplay of the polarization with the
electrostatics and quantum capacitance of the graphene electrodes. We first
asked whether experiments will detect a polar domain voltage across the
conducting graphene electrodes, and discovered that a finite signal appears
only upon gate tuning of one of the Dirac electrodes to charge neutrality.

Moving to the tunneling current, we studied two mechanisms for its dependence
on the polarization direction. First, we discussed a dependence of the
tunneling coefficient on the polarization. This barrier sensitivity becomes
sizable upon increasing the bias voltage which leads to a device asymmetry,
but is relatively small at zero bias for available values of gate voltages. We
then considered 2D momentum conserving tunneling, and showed that one can
detect the actual value of the polarization via a shift in the voltage axis of
a resonant peak occurring in the $I(V)$ characteristics due to the alignment
of the Dirac cones. This resulted in a sizable TER.

In order to switch from one state to the other, one would need, in practice,
to slide a domain wall separating AB and BA stacking through the tunneling
region. This could be achieved by tuning the bias $V_{b}$ above the switching
point. This process goes beyond the presented model.

Our results apply to Moiré superlattice materials such as transition metal
dichalcogenides. We hope that our theory will serve as a basis for analysis of
near-future FE tunnneling experiments.

##  VII Acknowledgments

We thank discussions with Moshe Ben Shalom, Igor Rozhansky, Simon Sallah Atri
and Hadar Steinberg. This project received funding from the European Research
Council (ERC) under the European Union’s Horizon 2020 research and innovation
program under grant agreement No 951541.

## References

  * Li and Wu (2017) Lei Li and Menghao Wu, “Binary compound bilayer and multilayer with vertical polarizations: two-dimensional ferroelectrics, multiferroics, and nanogenerators,” ACS nano 11, 6382–6388 (2017).
  * Yasuda  _et al._ (2021) Kenji Yasuda, Xirui Wang, Kenji Watanabe, Takashi Taniguchi, and Pablo Jarillo-Herrero, “Stacking-engineered ferroelectricity in bilayer boron nitride,” Science 372, 1458–1462 (2021).
  * Stern Vizner  _et al._ (2021) Maayan Stern Vizner, Yuval Waschitz, Wei Cao, Iftach Nevo, Kenji Watanabe, Takashi Taniguchi, Eran Sela, Michael Urbakh, Oded Hod, and Moshe Ben Shalom, “Interfacial ferroelectricity by van der waals sliding,” Science 372, 142–1466 (2021).
  * Weston  _et al._ (2022) Astrid Weston, Eli G Castanon, Vladimir Enaldiev, Fabio Ferreira, Shubhadeep Bhattacharjee, Shuigang Xu, Héctor Corte-León, Zefei Wu, Nicholas Clark, Alex Summerfield, _et al._ , “Interfacial ferroelectricity in marginally twisted 2d semiconductors,” Nat. Nanotechnol. , 1–6 (2022).
  * de la Barrera  _et al._ (2021) Sergio C de la Barrera, Qingrui Cao, Yang Gao, Yuan Gao, Vineetha S Bheemarasetty, Jiaqiang Yan, David G Mandrus, Wenguang Zhu, Di Xiao, and Benjamin M Hunt, “Direct measurement of ferroelectric polarization in a tunable semimetal,” Nat. Commun. 12, 1–9 (2021).
  * Deb  _et al._ (2022) Swarup Deb, Wei Cao, Noam Raab, Kenji Watanabe, Takashi Taniguchi, Moshe Goldstein, Leeor Kronik, Michael Urbakh, Oded Hod, and Moshe Ben Shalom, “Cumulative polarization coexisting with conductivity at interfacial ferroelectrics,”  (2022), 10.48550/ARXIV.2206.12215.
  * Wu and Li (2021) Menghao Wu and Ju Li, “Sliding ferroelectricity in 2d van der waals materials: Related physics and future opportunities,” Proc. Natl. Acad. Sci. U.S.A. 118, e2115703118 (2021).
  * Zhong  _et al._ (2021) Tingting Zhong, Yangyang Ren, Zhuhua Zhang, Jinhua Gao, and Menghao Wu, “Sliding ferroelectricity in two-dimensional moa 2 n 4 (a= si or ge) bilayers: high polarizations and moiré potentials,” J. Mater. Chem A 9, 19659–19663 (2021).
  * Bennett (2022) Daniel Bennett, “Theory of polar domains in moiré heterostructures,” Phys. Rev. B 105, 235445 (2022).
  * Bennett and Remez (2022) Daniel Bennett and Benjamin Remez, “On electrically tunable stacking domains and ferroelectricity in moiré superlattices,” NPJ 2D Mater. Appl. 6 (2022), 10.1038/s41699-021-00281-6.
  * Pantel and Alexe (2010) Daniel Pantel and Marin Alexe, “Electroresistance effects in ferroelectric tunnel barriers,” Phys. Rev. B 82, 134105 (2010).
  * Garcia and Bibes (2014) Vincent Garcia and Manuel Bibes, “Ferroelectric tunnel junctions for information storage and processing,” Nat. Commun. 5, 1–12 (2014).
  * Tao and Wang (2016) LL Tao and J Wang, “Ferroelectricity and tunneling electroresistance effect driven by asymmetric polar interfaces in all-oxide ferroelectric tunnel junctions,” Appl. Phys. Lett. 108, 062903 (2016).
  * Kang  _et al._ (2020) Lili Kang, Peng Jiang, Hua Hao, Yanhong Zhou, Xiaohong Zheng, Lei Zhang, and Zhi Zeng, “Giant tunneling electroresistance in two-dimensional ferroelectric tunnel junctions with out-of-plane ferroelectric polarization,” Phys. Rev. B. 101, 014105 (2020).
  * Yan  _et al._ (2022) Zhi Yan, Zeyu Li, Yulei Han, Zhenhua Qiao, and Xiaohong Xu, “Giant tunneling magnetoresistance and electroresistance in $\alpha$\- in 2 se 3-based van der waals multiferroic tunnel junctions,” Phys. Rev. B. 105, 075423 (2022).
  * Yang  _et al._ (2021) Jie Yang, Jun Zhou, Jing Lu, Zhaochu Luo, Jinbo Yang, and Lei Shen, “Sliding ferroelectric tunnel junctions,” arXiv:2112.02886 (2021).
  * Eisenstein  _et al._ (1991) James P. Eisenstein, Thomas. J. Gramila, Loren N. Pfeiffer, and K. W. West, “Probing a two-dimensional fermi surface by tunneling,” Phys. Rev. B 44, 6511–6514 (1991).
  * Eisenstein  _et al._ (1992) James P. Eisenstein, Loren N Pfeiffer, and KW West, “Coulomb barrier to tunneling between parallel two-dimensional electron systems,” Phys. Rev. Lett. 69, 3804 (1992).
  * Britnell  _et al._ (2013) L. Britnell, R. V. Gorbachev, A. K. Geim, L. A. Ponomarenko, A. Mishchenko, M. T. Greenaway, T. M. Fromhold, K. S. Novoselov, and L. Eaves, “Resonant tunnelling and negative differential conductance in graphene transistors,” Nat. Commun. 4 (2013), 10.1038/ncomms2817.
  * Mishchenko  _et al._ (2014) Artem Mishchenko, JS Tu, Y Cao, Roman V Gorbachev, JR Wallbank, MT Greenaway, VE Morozov, SV Morozov, MJ Zhu, SL Wong, _et al._ , “Twist-controlled resonant tunnelling in graphene/boron nitride/graphene heterostructures,” Nat. Nanotechnol. 9, 808–813 (2014).
  * Chen (2021) Julian Chen,  _Introduction to Scanning Tunneling Microscopy_ , 3rd ed. (Oxford University Press, 2021).
  * Yan  _et al._ (2021) Zhi Yan, Xin Jia, Xiaowen Shi, Xinlong Dong, and Xiaohong Xu, “Barrier-dependent electronic transport properties in two-dimensional mnbi2te4-based van der waals magnetic tunnel junctions,” Appl. Phys. Lett. 118, 223503 (2021).
  * Yan  _et al._ (2020) Zhi Yan, Ruiqiang Zhang, Xinlong Dong, Shifei Qi, and Xiaohong Xu, “Significant tunneling magnetoresistance and excellent spin filtering effect in cri 3-based van der waals magnetic tunnel junctions,” Phys. Chem. Chem. Phys. 22, 14773–14780 (2020).
  * Zhuravlev  _et al._ (2005) M Ye Zhuravlev, Renat F Sabirianov, SS Jaswal, and Evgeny Y Tsymbal, “Giant electroresistance in ferroelectric tunnel junctions,” Phys. Rev. Lett. 94, 246802 (2005).
  * Gerra  _et al._ (2007) G Gerra, AK Tagantsev, and N Setter, “Ferroelectricity in asymmetric metal-ferroelectric-metal heterostructures: a combined first-principles–phenomenological approach,” Phys. Rev. Lett. 98, 207601 (2007).
  * Brey (2014) Luis Brey, “Coherent tunneling and negative differential conductivity in a graphene/$h$-bn/graphene heterostructure,” Phys. Rev. Appl. 2, 014003 (2014).
  * Rogée  _et al._ (2022) Lukas Rogée, Lvjin Wang, Yi Zhang, Songhua Cai, Peng Wang, Manish Chhowalla, Wei Ji, and Shu Ping Lau, “Ferroelectricity in untwisted heterobilayers of transition metal dichalcogenides,” Science 376, 973–978 (2022).
  * Feenstra  _et al._ (2012) Randall M Feenstra, Debdeep Jena, and Gong Gu, “Single-particle tunneling in doped graphene-insulator-graphene junctions,” J. Appl. Phys. 111, 043711 (2012).
  * Wu  _et al._ (2020) Jiangbin Wu, Hung-Yu Chen, Ning Yang, Jun Cao, Xiaodong Yan, Fanxin Liu, Qibin Sun, Xi Ling, Jing Guo, and Han Wang, “High tunnelling electroresistance in a ferroelectric van der waals heterojunction via giant barrier height modulation,” Nat. Electron. 3, 466–472 (2020).
  * de la Barrera  _et al._ (2014) Sergio C de la Barrera, Qin Gao, and Randall M Feenstra, “Theory of graphene–insulator–graphene tunnel junctions,” J. Vac. Sci. Technol B 32, 04E101 (2014).


From the definitions it immediately follows that
$m_{1}^{\mathfrak{u}_{\star}(\bar{N})}=m_{1}^{\tau^{\mathfrak{u}_{\star}(\bar{N})}}$
and
$m_{2}^{\mathfrak{u}_{\star}(\bar{N})}=m_{2}^{\tau^{\mathfrak{u}_{\star}(\bar{N})}}$,
and hence the cyclic strict $C_{\infty}$-algebra
$\mathfrak{C}_{\mathfrak{u}_{\star}(\bar{N})}$ factorizes as

$\displaystyle\mathfrak{C}_{\mathfrak{u}_{\star}(\bar{N})}=\mathfrak{u}(\bar{N})\otimes_{\tau^{\mathfrak{u}_{\star}(\bar{N})}}\mathfrak{Scal}\
.$ (3.38)

Altogether, we have shown that the cyclic strict $L_{\infty}$-algebra of the
noncommutative biadjoint scalar theory on $\mathbbm{R}^{1,d-1}$ admits the
factorization

$\displaystyle\mathfrak{BAS}_{\star}=\mathfrak{g}\otimes\big{(}\mathfrak{u}(\bar{N})\otimes_{\tau^{\mathfrak{u}_{\star}(\bar{N})}}\mathfrak{Scal}\big{)}\
.$ (3.39)

This is one of the main messages of this paper: colour-stripping is achievable
in noncommutative field theories through a suitable notion of twisted
factorisation, which enables one to disentangle colour and kinematical degrees
of freedom. From this perspective noncommutativity is completely absorbed into
the twist maps $\tau^{\mathfrak{u}_{\star}(\bar{N})}$. This will also be
applicable to the noncommutative theories with local gauge symmetry that we
consider later on.

###  3.4 Adjoint scalar theory

The limiting rank one case $\bar{N}=1$ of the noncommutative biadjoint scalar
theory is an interesting noncommutative scalar field theory that does not have
a non-trivial commutative counterpart, in the sense that it becomes non-
interacting in the commutative limit. We therefore refrain from using the
adjective ‘noncommutative’ and simply refer to it as the adjoint scalar theory
on $\mathbbm{R}^{1,d-1}$. We begin by briefly summarising its main features,
which follow by setting $\bar{N}=1$ everywhere in §3.2.

The cyclic strict $L_{\infty}$-algebra $\mathfrak{AS}$ of this theory has
cochain complex

$\displaystyle\mathsf{Ch}(\mathfrak{AS})=\Big{(}\Omega^{0}(\mathbbm{R}^{1,d-1},\mathfrak{g})[-1]\xrightarrow{\
\square\ }\Omega^{0}(\mathbbm{R}^{1,d-1},\mathfrak{g})[-2]\Big{)}\ ,$ (3.40)

and the single non-vanishing higher bracket

$\displaystyle\mu_{2}^{\textrm{\tiny
AS}}(\phi_{1},\phi_{2})=-g\,[\\![T^{a}\otimes\phi_{1}^{a},T^{b}\otimes\phi_{2}^{b}]\\!]_{\textrm{\tiny
AS}}=-\mathrm{i}\,g\,[T^{a},T^{b}]_{\mathfrak{g}}\otimes[\phi_{1}^{a}\stackrel{{\scriptstyle\star}}{{,}}\phi_{2}^{b}]_{\mathfrak{u}(1)}\
,$ (3.41)

where we abbreviate $\llbracket-,-\rrbracket_{\textrm{\tiny
AS}}:=\llbracket-,-\rrbracket_{\mathfrak{g}\otimes\mathfrak{u}_{\star}(1)}$.
Note that this binary operation is not the antisymmetric Lie bracket
$[\phi_{1}\stackrel{{\scriptstyle\star}}{{,}}\phi_{2}]_{\mathfrak{g}}$; in
particular, it vanishes in the commutative limit $t=0$. The cyclic structure
is given by

$\displaystyle\langle\phi,\phi^{+}\rangle_{\textrm{\tiny
AS}}=\int\,\mathrm{d}^{d}x\
\mathrm{Tr}_{\mathfrak{g}}(\phi\star\phi^{+})=\int\,\mathrm{d}^{d}x\
\phi^{a}\star\phi_{a}^{+}\ .$ (3.42)

The equations of motion

$\displaystyle\square\,\phi^{a}-\mathrm{i}\,g\,f^{abc}\,\phi^{b}\star\phi^{c}=0$
(3.43)

of the adjoint scalar theory are the variational equations of the action
functional

$\displaystyle\begin{split}S_{\textrm{\tiny AS}}[\phi]&=\int\,\mathrm{d}^{d}x\
\frac{1}{2}\,\phi^{a}\star\square\,\phi_{a}-\frac{\mathrm{i}\,g}{3}\,f^{abc}\,\phi^{a}\star\phi^{b}\star\phi^{c}\
.\end{split}$ (3.44)

The three-point Feynman vertex defined by eq. 3.44 is

$\begin{split}\leavevmode\hbox to208.82pt{\vbox
to75.2pt{\pgfpicture\makeatletter\hbox{\hskip 36.76437pt\lower-38.78566pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}} {\small{}{{}}{}{{}}
{}{}{}\pgfsys@moveto{-31.38602pt}{26.33624pt}\pgfsys@lineto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-33.76466pt}{29.53595pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$a$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} {}{{}}{}{{}}
{}{}{}\pgfsys@moveto{-31.38602pt}{-26.33624pt}\pgfsys@lineto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-33.31726pt}{-35.78595pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$b$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} {}{{}}{}{{}}
{}{}{}\pgfsys@moveto{40.97218pt}{0.0pt}\pgfsys@lineto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{44.17189pt}{-1.93748pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$c$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} {{}}{}{{}}{}{{}}
{}{}{}{}{}{}{{}}\pgfsys@moveto{-13.27057pt}{17.55864pt}\pgfsys@lineto{-4.85878pt}{10.50006pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{0.76604}{-0.6428}{0.6428}{0.76604}{-4.70558pt}{10.37152pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} {
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-16.79463pt}{22.50835pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p_{1}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} {{}}{}{{}}{}{{}}
{}{}{}{}{}{}{{}}\pgfsys@moveto{-13.27057pt}{-17.55864pt}\pgfsys@lineto{-4.85878pt}{-10.50006pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{0.76604}{0.6428}{-0.6428}{0.76604}{-4.70558pt}{-10.37152pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} {
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-16.79463pt}{-24.63332pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p_{2}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} {{}}{}{{}}{}{{}}
{}{}{}{}{}{}{{}}\pgfsys@moveto{15.93352pt}{4.55238pt}\pgfsys@lineto{4.95238pt}{4.55238pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{4.75238pt}{4.55238pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} {
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{12.40945pt}{9.50209pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$p_{3}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
}\normalsize{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{
{}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} {
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{68.00664pt}{-3.19444pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$=\
-\mathrm{i}\,g\,f^{abc}\,\bar{F}(p_{1},p_{2},p_{3})\ .$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\end{split}$ (3.45)

This is completely antisymmetric due to symmetry of the structure constants of
$\mathfrak{u}_{\star}(1)$ from eq. 3.7 under cyclic permutations of the
momenta, which follows from momentum conservation.


# Large Language Models Are State-of-the-Art
Evaluators of Translation Quality

Tom Kocmi  Christian Federmann
Microsoft, One Microsoft Way, Redmond, WA-98052, USA
<EMAIL_ADDRESS>
###### Abstract

We describe GEMBA, a GPT-based metric for assessment of translation quality,
which works both with a reference translation and without. In our evaluation,
we focus on zero-shot prompting, comparing four prompt variants in two modes,
based on the availability of the reference. We investigate nine versions of
GPT models, including ChatGPT and GPT-4. We show that our method for
translation quality assessment only works with GPT 3.5 and larger models.
Comparing to results from WMT22’s Metrics shared task, our method achieves
state-of-the-art accuracy in both modes when compared to MQM-based human
labels. Our results are valid on the system level for all three WMT22 Metrics
shared task language pairs, namely English into German, English into Russian,
and Chinese into English. This provides a first glimpse into the usefulness of
pre-trained, generative large language models for quality assessment of
translations. We publicly release all our code and prompt templates used for
the experiments described in this work, as well as all corresponding scoring
results, to allow for external validation and
reproducibility.111https://github.com/MicrosoftTranslator/GEMBA

Large Language Models Are State-of-the-Art
Evaluators of Translation Quality

Tom Kocmi and Christian Federmann Microsoft, One Microsoft Way, Redmond,
WA-98052, USA<EMAIL_ADDRESS>

##  1 Introduction

One of the interesting properties of large language models (LLMs) such as GPT
Brown et al. (2020b) is their (implicit) support for multilingual Q&A.
Prompting the model in the right way allows us to translate text between
languages (Vilar et al., 2022). This is surprising as GPT has not been fine-
tuned for the translation task.

Hendy et al. (2023) show that GPT-enabled translation achieves high quality
when applied for the translation of high-resource languages, but still lacks
in terms of translation quality for under-represented languages. Building on
this finding— _if the model can translate, it may be able to differentiate
good from bad translations_ —we apply GPT for the task of translation quality
assessment.

In the remainder of this paper, inspired by recent progress on generative,
pre-trained large language models (LLMs), we explore how these models can be
applied for automated assessment of translation quality. The primary query for
this study centers around the question: _Can LLMs be used for effective
quality assessment of translations?_

We propose GEMBA, which stands for _GPT Estimation Metric Based Assessment_.
The metric evaluates each segment translation in isolation and then averages
across all obtained scores for the final, system-level score.

We define and evaluate several prompt variants for zero-shot assessment of
translation quality in two modes, either with a human reference translation,
as a quality metric, or without one, as a quality estimation task.

We design the main prompts based on the DA+SQM template used for human
assessment of translation quality as implemented in the Appraise framework
Federmann (2018) for WMT22 Kocmi et al. (2022), building on previous work
conducted by Freitag et al. (2021a).

To the best of our knowledge, our research represents the pioneering effort in
exploring the utilization of large language models (LLMs) for the purpose of
quality assessment. Subsequent to the publishing of our findings, Lu et al.
(2023) independently published a related report, corroborating the high
performance of LLMs.

The main contributions of this paper are:

  * -

We demonstrate state-of-the-art capabilities of GPT-based translation quality
assessment on the latest WMT22 metrics evaluation data (on the system level);

  * -

We experiment with four prompt templates, showing that the least constrained
template achieves the best performance;

  * -

We evaluate nine different models of GPT, showing that only GPT 3.5 and larger
models are capable of translation quality assessment;

  * -

We show that GEMBA with GPT-4 model is only slightly behind on segment-level
scores to the best-performing metrics.

Score the following translation from {source_lang} to {target_lang} with
respectto the human reference on a continuous scale from 0 to 100, where score
of zero means"no meaning preserved" and score of one hundred means "perfect
meaning and grammar".{source_lang} source: "{source_seg}"{target_lang} human
reference: {reference_seg}{target_lang} translation: "{target_seg}"Score:

Figure 1: The best-performing prompt based on Direct Assessment expecting a
score between 0–100. Template portions in bold face are used only when a human
reference translation is available.

##  2 The GEMBA Metric

To assess translation quality via prompting an LLM, the following parameters
are needed:

  * -

prompt variant (from a pre-defined set)

  * -

source language name, e.g., “Chinese”

  * -

target language name, e.g., “English”

  * -

source segments $src_{1..N}$

  * -

candidate translations $hyp_{1..N}$

  * -

optionally, reference translations $ref_{1..N}$

We generate a GPT request for every segment, querying as individual zero-shot
problems, and then aggregate results. For this initial proof of concept, we
leave improvements such as few-shot queries or document-level context to
future work.

###  2.1 Prompt variants

We experiment with four distinct prompt types: modeling two scoring and two
classification tasks. For the scoring tasks, first, one based on direct
assessment (_GEMBA-DA_), second, another based on recent research efforts on
scalar quality metrics (_GEMBA-SQM_).222Although names are based on existing
techniques for human assessment, they do not match perfectly. As scoring
translation quality may be an unnatural task for an LLM, we also design two
classification tasks. The first is based on one-to-five stars ranking (_GEMBA-
stars_), which is a style often used when users are asked to review various
services or products. The second prompt asks the LLM to label translation
quality as one of five discrete quality classes (_GEMBA-classes_).

For each of these four prompt types, we experiment with two modes that differ
with respect to the wording of the corresponding query templates which either
have access to a human reference or not. As an example, we show the _GEMBA-DA_
prompt in Figure 1. Based on token count, this is the least constrained prompt
template that we experiment with. The complete set of prompt templates is
available in Appendix A. For naming convention, we mark quality estimation
metrics (without reference) with the suffix ”[noref]”.

###  2.2 Scoring process

The expected scores are in $[0,100]$ for _GEMBA-DA_ and _GEMBA-SQM_ prompts,
same as for human assessment (Graham et al., 2013); for _GEMBA-stars_ the
output ranges from $[1,5]$ and _GEMBA-classes_ assigns one of five class
labels.

We average segment-level scores to obtain system-level scores. For the _GEMBA-
classes_ metric variant, we assign classes a numerical value $[0-4]$, based on
the label, before averaging.

Depending on the GPT model we query, sometimes answers are returned outside
these ranges, as text. When we observe such an _invalid_ answer, we add
randomness and sample more responses, selecting the first answer matching the
output range as the final result.

###  2.3 GPT models

We experiment with seven GPT models— _ranging from GPT 2 up to the latest
GPT-4 model_ —that are described in Table 1.333https://learn.microsoft.com/en-
us/azure/cognitive-services/openai/concepts/models and
https://platform.openai.com/docs/model-index-for-researchers We use the GPT-4
model as the default model for most experiments and compare the performance of
other models in Section 4.3. Specifically, we use these models with brief
description:

GPT 2

We use models provided by Radford et al. (2019), assessing if GPT 2 may be
useful for quality assessment— _we find that it is not_ ;

Ada

GPT 3. Max request size of 2,048 tokens and training data up to October 2019
(Brown et al., 2020a);

Babbage

GPT 3. More capable than Ada (Brown et al., 2020a);

Curie

GPT 3. More capable than Babbage (Brown et al., 2020a);

Davinci-002

GPT 3.5. Max request size of 4,000 tokens and training data up to June 2021.
Uses FeedME training;

ChatGPT

Improved GPT 3.5 model, fine-tuned using Reinforcement Learning from Human
Feedback (RLHF);

Davinci-003

GPT 3.5.1. Uses PPO training;

GPT-3.5-turbo

Davinci-003 model optimized for speed;

GPT-4

there is only limited information about GPT-4, see OpenAI (2023).

GPT 3 models are based on Ouyang et al. (2022). The models are sorted based on
their estimated power or date of release. We acknowledge that OpenAI has not
released detailed information about the architecture and training data behind
given models. Most importantly, OpenAI claims that models have been trained
with data up until September 2021. It is important as we use testsets prepared
and released by December 2022.

Model name | Abbrev. | Model used
---|---|---
GPT-2 | — | Radford et al. (2019)
Ada | — | text-ada-001
Babbage | Bab | text-babbage-001
Curie | Curie | text-curie-001
Davinci-002 | Dav2 | text-davinci-002
ChatGPT | Chat | text-chat-davinci-002
Davinci-003 | Dav3 | text-davinci-003
GPT-3.5-turbo | Turbo | gpt-3.5-turbo
GPT-4 | GPT4 | gpt-4
Table 1: Details of all models used in this work. Models are sorted from
oldest to newest which also reflects their respective power. GPT 2 and Ada do
not work.

##  3 Experiments

To measure the performance of the proposed GEMBA metric, we follow the
methodology and use test data provided by the WMT22 Metrics shared task
(Freitag et al., 2022b) which hosts an annual evaluation of automatic metrics,
benchmarking them against human gold labels. Effectively, we compare GEMBA
against the best-performing automatic metrics: COMET (Rei et al., 2020, 2022),
BLEURT (Sellam et al., 2020), or the non-public winner MetricX XXL.

###  3.1 Test set

We use the MQM 2022 test set which contains human judgments for the following
three translation directions: English into German, English into Russian, and
Chinese into English. The test set contains a total of 54 machine translation
system outputs or human translations. It contains a total of 106k segments.
Translation systems are mainly from participants of the WMT22 General MT
shared task (Kocmi et al., 2022).

The source segments and human reference translations for each language pair
contain around 2,000 sentences from four different texts domains: news,
social, conversational, and e-commerce. The gold standard for scoring
translation quality is based on human MQM ratings, annotated by professionals
who mark individual errors in each translation, as described in Freitag et al.
(2021a).

###  3.2 Evaluation methods

To determine how well automatic metrics correlate with humans, we measure
system-level, pairwise accuracy (_accuracy_ , Kocmi et al., 2021). For
segment-level evaluation, we use Kendall’s Tau ($\tau$, Freitag et al.,
2022a).

Here, accuracy is defined as the number of system pairs ranked correctly by
the metric with respect to the human ranking divided by the total number of
system pair comparisons.

Formally:

$\mbox{Accuracy}=\frac{|\mbox{sign}(\mbox{metric}\Delta)==\mbox{sign}(\mbox{human}\Delta)|}{|\mbox{all
system pairs}|}$

The variant of Kendall’s Tau used for metric evaluation changed over the
years. Initially, Callison-Burch et al. (2011) proposed to use Kendall’s Tau-a
ignoring human rankings that tied, while penalising ties in automatic metrics.

$\tau=\frac{|\mbox{Concordant}|-|\mbox{Discordant}|}{|\mbox{Concordant}|+|\mbox{Discordant}|}$

where Concordant is the set of all human segment comparisons for which a given
metric suggests the same order of systems and Discordant is the set of all
human comparisons for which a given metric disagrees.

|  | Metric
---|---|---
|  | $s_{1}<s_{2}$ | $s_{1}=s_{2}$ | $s_{1}>s_{2}$
Human | $s_{1}<s_{2}$ | Conc | Disc | Disc
$s_{1}=s_{2}$ | – | – | –
$s_{1}>s_{2}$ | Disc | Disc | Conc

This definition was later updated by Macháček and Bojar (2014), who handle
ties as a separate group in contrast to Concordant and Discordant. Metrics
shared tasks Mathur et al. (2020) and Freitag et al. (2021b) changed this back
to the 2011 version. Last year, Freitag et al. (2022a) changed it to Kendall’s
Tau-b, which makes adjustments for ties, we use the latest definition in our
experiments. Overall, ties in automatic metrics are rare for non-identical
translations but are an issue when a method outputs only a discrete set of
scores (as in our case). Additionally, Kendall’s Tau is susceptible to noise
in gold pairwise rankings Freitag et al. (2022a).

We reproduced all scores reported in the WMT22 Metrics shared task findings
paper with the official WMT22 script.444https://github.com/google-research/mt-
metrics-eval Reported scores match Table 11 of the WMT22 metrics findings
paper (Freitag et al., 2022b).

##  4 Results

We investigate GEMBA’s performance for two modes: with a reference translation
and without reference translation (in a quality estimation setting). Table 2
reports pairwise accuracy on the system level, comparing _GEMBA-DA_ against
the best-performing metrics from the WMT22 Metrics shared task (Freitag et
al., 2022b). We use GPT-4 as the main model and _GEMBA-DA_ as the main style
for some experiments.

Metric | Accuracy
---|---
GEMBA-GPT4-DA | 89.8%
GEMBA-GPT4-DA[noref] | 87.6%
MetricX XXL | 85.0%
BLEURT-20 | 84.7%
COMET-22 | 83.9%
COMET-20 | 83.6%
UniTE | 82.8%
MS-COMET-22 | 82.8%
MATESE | 81.0%
YiSi-1 | 79.2%
COMETKiwi[noref] | 78.8%
COMET-QE[noref] | 78.1%
BERTScore | 77.4%
UniTE-src[noref] | 75.9%
MS-COMET-QE-22[noref] | 75.5%
MATESE-QE[noref] | 74.8%
f200spBLEU | 74.1%
chrF | 73.4%
BLEU | 70.8%
Table 2: Results for the system-level pairwise accuracy compared to the
current automatic metric. Metrics marked as “[noref]” do not use a reference
translation.

###  4.1 Reference-based

The results in Table 2 show that our reference-based GEMBA-GPT4-DA metric sets
a new state of the art. It outperforms all of the other reference-based
metrics from the WMT22 Metrics shared task. The observed level of metric
performance is unexpected, especially considering that human labels used as a
gold standard in itself are noisy and therefore an accuracy of 100% is
impossible to obtain for an automatic metric.

###  4.2 Quality estimation

Table 2 shows that our reference-less metric GEMBA-GPT4-DA[noref] achieves the
highest performance for the quality estimation mode, and strongly outperforms
all of the other reference-less metrics. Moreover, it also outperforms all of
the other reference-based metrics, performing only slightly worse than GEMBA-
GPT4-DA. Again, the observed level of assessment quality is unexpectedly high,
highlighting the potential of using LLMs for translation quality assessment
tasks.

| Bab | Curie | Dav2 | Chat | Dav3 | Turbo | GPT4
---|---|---|---|---|---|---|---
DA | 39.1% | 54.4% | 85.8% | 81.0% | 88.0% | 86.5% | 89.8%
DA[noref] | 55.8% | 51.8% | 83.9% | 82.1% | 86.1% | 86.9% | 87.6%
SQM | 51.8% | 40.5% | 85.8% | 85.0% | 85.4% | 87.2% | 88.7%
SQM[noref] | 51.1% | 41.6% | 82.8% | 81.0% | 82.5% | 87.6% | 89.1%
Stars | 48.2% | 37.2% | 88.3% | 85.0% | 85.8% | 89.4% | 91.2%
Stars[noref] | 58.4% | 54.7% | 79.6% | 83.6% | 83.2% | 84.3% | 89.1%
Classes | 47.4% | 43.4% | 79.6% | 87.2% | 85.4% | 82.5% | 89.1%
Classes[noref] | 35.0% | 61.7% | 78.1% | 83.6% | 78.8% | 62.0% | 91.2%
Table 3: Accuracy of the system-level pairwise accuracy for quality estimation
methods for most combinations of prompts and different GPT models. The
evaluation is based on three language pairs and MQM human labels. All results
higher than the WMT22 winner of Metrics shared task MetricX XXL are bolded.

###  4.3 Comparison of GPT models

We compare the performance of various GPT versions as an automatic metric.
Table 3 shows results for all models we have experimented with and all prompt
variants tested.

We do not show results for GPT-2 or Ada models. Neither of those have produced
replies in the specific scoring range and neither seemed to be producing any
meaningful replies. We list a couple of their answers in Appendix C. Based on
our experiments, we conclude that they are not powerful enough to understand
the zero-shot prompts.

By contrast, Babbage and Curie models appear to understand what type of answer
they should produce, but the quality of their scores seems to be close to
random guessing. Thus, both Babbage and Curie are useless for translation
quality assessment.

The main performance jump occurs for GPT 3.5 and larger models, i.e.,
Davinci-002, ChatGPT, Davinci-003, Turbo, and GPT-4. Each of them achieves
highly competitive results for all of the prompt variants we have tested.
Interestingly, ChatGPT in DA style appears to have the lowest quality among
those models. In addition, ChatGPT and Turbo frequently reply with a score
followed by an explanation of why it has assigned that score. One possible
reason may be in the form of the prompt, which wasn’t modified to instruct
ChatGPT not to generate an explanation.

Unsurprisingly, the best performance is obtained by the most powerful LLM,
GPT-4. Moreover, we can see that over time, each generation of models is
slightly better. This confirms the findings of Hendy et al. (2023) who
demonstrated superior translation capabilities with Davinci-003 over all other
previous GPT variants.

###  4.4 Segment-level performance

All previous results are reported on the system level. We also investigate how
well the GEMBA metric performs on the segment level, with respect to the human
gold annotations. We present Kendall’s Tau results for each language pair
separately in Table 4 for GPT-4 and Davinci-003 (results for all metrics are
in Table 7).

GPT-4 models are slightly behind the top-performing metrics but continue to
have a high correlation with human judgment. On the other hand, quality
estimation GEMBA-Dav3-DA [noref] has significantly lower segment-level
performance in contrast to other top-performing metrics.

The lower performance of a segment-level correlation could be attributed to
Kendall’s Tau, which penalizes ties. Our metric in contrast to other automatic
metrics returns a discrete value between 0–100. There is a high probability
that two translations will obtain an equal score.

Metric | Acc | en-de | en-ru | zh-en
---|---|---|---|---
GEMBA-GPT4-DA | 89.8% | 0.36 | 0.36 | 0.38
GEMBA-Dav3-DA | 88.0% | 0.31 | 0.33 | 0.37
GEMBA-GPT4-DA[noref] | 87.6% | 0.31 | 0.40 | 0.41
GEMBA-Dav3-DA[noref] | 86.1% | 0.18 | 0.26 | 0.29
MetricX XXL | 85.0% | 0.36 | 0.42 | 0.43
BLEURT-20 | 84.7% | 0.34 | 0.36 | 0.36
COMET-22 | 83.9% | 0.37 | 0.40 | 0.43
UniTE | 82.8% | 0.37 | 0.38 | 0.36
COMETKiwi[noref] | 78.8% | 0.29 | 0.36 | 0.36
COMET-QE[noref] | 78.1% | 0.28 | 0.34 | 0.36
chrF | 73.4% | 0.21 | 0.17 | 0.15
BLEU | 70.8% | 0.17 | 0.14 | 0.14
Table 4: Kendall’s Tau ($\tau$) segment-level evaluation. Full results are in
Table 7.

In order to investigate this further, we collect all answers across all
systems and all three language pairs and then calculate the frequency of each
distinct answer value.

We can notice several interesting observations in Table 5. The DA reference-
based prompt generates mostly multiples of five. Over three-quarters of all
scores are either score 80, 95, or 100. This could reflect the actual quality
of the system translations as the underlying systems are provably high-
quality. This is also a finding of Freitag et al. (2022b) that many metrics
fall into the same significance cluster.

When we investigate the “DA[noref]”, we notice that 60.5% of all scores are of
value ”95”. Despite this fact, the metric still manages to differentiate the
systems from each other and outperform all other quality estimation metrics on
the system level. This is contributed to the fact that better-performing
systems obtain more segments with a score 95 than worse-performing systems,
therefore getting a lower average score. We should note, that there are no
system-level ties.

We conjecture that frequent segment-level ties and the discrete scale thus may
contribute to the lower Kendall’s Tau segment-level performance.

Answers | DA | DA[noref] | SQM | SQM[noref]
---|---|---|---|---
0 | 0.1% | 0.1% | 0.1% | 0.1%
5 | 0.0% | 0.0% | 0.0% | 0.0%
10 | 0.0% | 0.0% | 0.0% | 0.1%
15 | — | — | 0.0% | 0.0%
20 | 0.2% | 0.3% | 0.2% | 0.3%
25 | — | — | 0.0% | —
30 | 0.1% | 0.2% | 0.1% | 0.1%
35 | — | — | 0.0% | —
40 | 0.5% | 0.6% | 0.5% | 0.6%
45 | 0.0% | 0.0% | 0.0% | 0.0%
50 | 0.0% | 0.1% | 0.1% | 0.0%
55 | — | — | 0.0% | —
60 | 2.1% | 2.3% | 2.0% | 2.1%
65 | — | 0.0% | 0.0% | 0.0%
70 | 1.3% | 0.4% | 1.9% | 0.6%
75 | 0.5% | 1.0% | 0.7% | 0.7%
80 | 6.3% | 4.5% | 7.0% | 5.7%
85 | 4.4% | 2.7% | 6.0% | 2.9%
87 | — | — | 0.0% | —
88 | — | — | 0.0% | —
90 | 21.3% | 13.0% | 27.6% | 14.5%
92 | — | — | 0.0% | —
93 | — | — | 0.0% | —
94 | — | — | 0.0% | —
95 | 53.3% | 60.6% | 44.6% | 49.4%
98 | 0.8% | 0.0% | 0.4% | 0.0%
99 | 0.4% | — | 0.2% | —
100 | 8.6% | 14.1% | 8.5% | 22.8%
Table 5: Distribution of all distinct segment-level score values for MQM 2022
for model GPT-4.

###  4.5 Failure rate

As we described earlier, LLMs may answer with an invalid answer, for example
with a textual answer instead of a score, mostly explaining its decision. When
such a situation happens, we iteratively increase the temperature— _adding
randomness to the model_ —and take the first answer matching the expected
score output range.

This adds non-determinism to our evaluation, therefore we investigate how
frequently this phenomenon happens. Table 6 shows the number of invalid
answers. For almost all combinations of models and prompts, except of SQM-
style, LLMs understand the prompt and provide answers in a valid range with
less than 1% of the answers being invalid.555Roughly 1,000 answers equal to 1%
of the total volume. This has a minimal effect on the final system-level score
and therefore, we conclude that the metric is mostly deterministic.

Additionally, we confirm that a temperature equal to zero always returns the
same answer, which we evaluated by re-running GEMBA-Dav2-DA[noref].

| Bab | Curie | Dav2 | Chat | Dav3 | GPT4
---|---|---|---|---|---|---
DA | 750 | 8,048 | 7 | 565 | 0 | 0
DA[noref] | 146 | 862 | 0 | 935 | 53 | 0
SQM | 89,599 | 129 | 4,827 | 45 | 1,279 | —
SQM[noref] | 15,577 | 95,131 | 1,763 | 59 | 1 | 0
Stars | 18,074 | — | 135 | 1,064 | 58 | —
Stars[noref] | — | 86,593 | 135 | 1,924 | 1 | 0
Classes | 74 | 12 | 0 | 10 | 0 | —
Classes[noref] | 115 | 15 | 0 | 12 | 0 | —
Table 6: Number of invalid answers (full set size 106,758) that needed to be
re-prompted with added randomness. The evaluation of ChatGPT and parts of
GPT-4 were excluded due to their late integration and changes in our codebase.

Processing answers is straightforward as it is usually a stand-alone number.
In some occasions, LLMs give a numerical score and continue with a textual
explanation, for such cases, we parse only the first number. A more complex
approach needs to be taken for _GEMBA-stars_ prompts where the model provides
different answers which we parse separately. Here are some examples of two-
star answers: ”2”, ”two”, ”**”, ”★★”, ”two stars”, or ”2 stars”. For non-
English target languages the answer may be produced in the target language,
e.g., ”一颗星”, or ”五”. We have not observed attempts to translate output for
other prompts.

##  5 Conclusion

We have presented our work on GEMBA, a GPT-based estimation metric-based
assessment method. Comparing our metrics to other automated metrics from the
WMT22 Metrics shared task we report state-of-the-art performance on the MQM
2022 test set across three language pairs: English to German, English to
Russian, and Chinese to English.

We intend to continue research on the application of GPT models for quality
assessment. Further research will focus on the switch to few-shot (as opposed
to our current zero-shot methodology) as well as model fine-tuning. Both of
which promise to increase GEMBA accuracy. Furthermore, modifying prompts to
support MQM error-based evaluation or post-editing efforts may lead to further
improvements.

GPT-enhanced evaluation metrics may allow us to make progress with respect to
document-level evaluation (due to their ability to use much larger context
windows). This could be beneficial as there is little research into document-
level metrics (Vernikos et al., 2022).

## Limitations

While preliminary results indicate that the GEMBA metric performs very well
when compared to other automated metrics evaluated as part of the WMT22
Metrics shared task, it is important to note that these results are based on
human labels for _only three language pairs_. We expect that the metrics
performance may suffer for other language pairs, mainly under-resourced
languages similar to Hendy et al. (2023) who show low translation quality for
such languages. In addition, GEMBA’s state-of-the-art performance only holds
for the system level, while segment-level scores still have room for
improvement. Reported results are indicative of the potential performance LLMs
could achieve for the translation quality assessment task in the long run.
However, more analysis is needed before using it as the main tool for deciding
translation quality.

An additional limitation to consider in this study is the inability to
definitively ascertain that the evaluation data have not been included in
OpenAI’s training dataset. Nevertheless, the available evidence strongly
indicates that this is unlikely. OpenAI claims that their data compilation
only extends up to September 2021, while the test set employed in this
research was generated during the second half of 2022 and made publicly
available in December 2022. Our initial positive results using the Davinci-002
model were obtained in early February, which presents a narrow timeframe for
OpenAI to incorporate and process the evaluation data. Furthermore, the test
set is not readily accessible in plaintext format, necessitating pre-
processing prior to utilization in training.

## Acknowledgments

This work would not have been possible without the help and support from our
friend and colleague, Olivier Nano, who provided access to GPT models via
Microsoft Azure – _Merci beaucoup, Olivier!_ The authors are also grateful to
Matt Post, Vikas Raunak, Shabnam Sadegharmaki, and the Microsoft Translator
research team for fruitful discussions and helpful feedback.

## References

  * Brown et al. (2020a) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020a.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Brown et al. (2020b) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020b.  Language models are few-shot learners.
  * Callison-Burch et al. (2011) Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011.  Findings of the 2011 workshop on statistical machine translation.  In _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , pages 22–64, Edinburgh, Scotland. Association for Computational Linguistics.
  * Federmann (2018) Christian Federmann. 2018.  Appraise evaluation framework for machine translation.  In _Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations_ , pages 86–88, Santa Fe, New Mexico. Association for Computational Linguistics.
  * Freitag et al. (2021a) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.  _Transactions of the Association for Computational Linguistics_ , 9:1460–1474.
  * Freitag et al. (2022a) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022a.  Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Freitag et al. (2022b) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andre F. T. Martins. 2022b.  Results of wmt22 metrics shared task: Stop using bleu neural metrics are better and more robust.  In _Proceedings of the Seventh Conference on Machine Translation_ , pages 46–68, Abu Dhabi. Association for Computational Linguistics.
  * Freitag et al. (2021b) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b.  Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 733–774, Online. Association for Computational Linguistics.
  * Graham et al. (2013) Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013.  Continuous measurement scales in human evaluation of machine translation.  In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_ , pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics.
  * Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023.  How good are gpt models at machine translation? a comprehensive evaluation.  _arXiv preprint arXiv:2302.09210_.
  * Kocmi et al. (2022) Tom Kocmi, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novak, Martin Popel, Maja Popovic, and Mariya Shmatova. 2022.  Findings of the 2022 conference on machine translation (wmt22).  In _Proceedings of the Seventh Conference on Machine Translation_ , pages 1–45, Abu Dhabi. Association for Computational Linguistics.
  * Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021.  To ship or not to ship: An extensive evaluation of automatic metrics for machine translation.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 478–494, Online. Association for Computational Linguistics.
  * Lu et al. (2023) Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023.  Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt.  _arXiv preprint_.
  * Macháček and Bojar (2014) Matouš Macháček and Ondřej Bojar. 2014.  Results of the WMT14 metrics shared task.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 293–301, Baltimore, Maryland, USA. Association for Computational Linguistics.
  * Mathur et al. (2020) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020\.  Results of the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 688–725, Online. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Rei et al. (2022) Ricardo Rei, Jose G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andre F. T. Martins. 2022.  Comet-22: Unbabel-ist 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation_ , pages 578–585, Abu Dhabi. Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Vernikos et al. (2022) Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022.  Embarrassingly easy document-level mt metrics: How to convert any pretrained metric into a document-level metric.  In _Proceedings of the Seventh Conference on Machine Translation_ , pages 118–128, Abu Dhabi. Association for Computational Linguistics.
  * Vilar et al. (2022) David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022.  Prompting palm for translation: Assessing strategies and performance.  _arXiv preprint arXiv:2211.09102_.

##  Appendix A Appendix: Prompt Templates

Below we provide our prompt templates which we use for the experiments
described in this paper. Template portions in bold face are used only when a
human reference translation is available.

###  A.1 DA: Direct Assessment

Output scores range from $0-100$.

Score the following translation from {source_lang} to {target_lang} with
respect to the human reference on a continuous scale from 0 to 100, where a
score of zero means "no meaning preserved" and score of one hundred means
"perfect meaning and grammar". {source_lang} source: "{source_seg}"
{target_lang} human reference: {reference_seg} {target_lang} translation:
"{target_seg}" Score:

###  A.2 SQM: Scalar Quality Metrics

Output scores range from $0-100$.

Score the following translation from {source_lang} to {target_lang} with
respect to the human reference on a continuous scale from 0 to 100 that starts
with "No meaning preserved", goes through "Some meaning preserved", then "Most
meaning preserved and few grammar mistakes", up to "Perfect meaning and
grammar". {source_lang} source: "{source_seg}" {target_lang} human reference:
"{reference_seg}" {target_lang} translation: "{target_seg}" Score (0-100):

###  A.3 Stars: One to Five Stars Ranking

Output scores range from $1-5$. Special care is taken for answers containing
non-numerical answers, such as ”Three stars”, ”****”, or ”1 star”.

Score the following translation from {source_lang} to {target_lang} with
respect to the human reference with one to five stars. Where one star means
"Nonsense/No meaning preserved", two stars mean "Some meaning preserved, but
not understandable", three stars mean "Some meaning preserved and
understandable", four stars mean "Most meaning preserved with possibly few
grammar mistakes", and five stars mean "Perfect meaning and grammar".
{source_lang} source: "{source_seg}" {target_lang} human reference:
"{reference_seg}" {target_lang} translation: "{target_seg}" Stars:

###  A.4 Classes: Quality Class Labels

Output label one of "No meaning preserved", "Some meaning preserved, but not
understandable", "Some meaning preserved and understandable", "Most meaning
preserved, minor issues", "Perfect translation".

Classify the quality of translation from {source_lang} to {target_lang} with
respect to the human reference into one of following classes: "No meaning
preserved", "Some meaning preserved, but not understandable", "Some meaning
preserved and understandable", "Most meaning preserved, minor issues",
"Perfect translation". {source_lang} source: "{source_seg}" {target_lang}
human reference: "{reference_seg}" {target_lang} translation: "{target_seg}"
Class:

Templates available from:
https://github.com/MicrosoftTranslator/GEMBA/gemba/prompt.py

##  Appendix B Appendix: Full Results

Below table lists all GEMBA results we have obtained for this work. Any
missing segment-level scores are due to a subset of segments for which we
could not obtain a score even after adding randomness.

Metric | Accuracy | en-de | en-ru | zh-en
---|---|---|---|---
GEMBA-GPT4-Classes[noref] | 91.2% | 0.304 | 0.390 | 0.313
GEMBA-GPT4-Stars | 91.2% | 0.326 | 0.351 | 0.382
GEMBA-GPT4-DA | 89.8% | 0.357 | 0.358 | 0.382
GEMBA-Turbo-Stars | 89.4% | 0.259 | 0.223 | 0.265
GEMBA-GPT4-Classes | 89.1% | 0.222 | 0.267 | 0.273
GEMBA-GPT4-Stars[noref] | 89.1% | 0.308 | 0.366 | 0.404
GEMBA-GPT4-SQM[noref] | 89.1% | 0.359 | 0.432 | 0.416
GEMBA-GPT4-SQM | 88.7% | 0.380 | 0.388 | 0.398
GEMBA-Dav2-Stars | 88.3% | 0.225 | 0.282 | 0.183
GEMBA-Dav3-DA | 88.0% | 0.306 | 0.332 | 0.371
GEMBA-GPT4-DA[noref] | 87.6% | 0.311 | 0.405 | 0.407
GEMBA-Turbo-SQM[noref] | 87.6% | 0.259 | 0.309 | 0.291
GEMBA-Chat-Classes | 87.2% | 0.220 | 0.270 | 0.259
GEMBA-Turbo-SQM | 87.2% | 0.298 | 0.277 | 0.313
GEMBA-Turbo-DA[noref] | 86.9% | 0.255 | 0.294 | 0.264
GEMBA-Turbo-DA | 86.5% | 0.250 | 0.234 | 0.255
GEMBA-Dav3-DA[noref] | 86.1% | 0.180 | 0.258 | 0.289
GEMBA-Dav3-Stars | 85.8% | 0.294 | 0.294 | 0.297
GEMBA-Dav2-SQM | 85.8% | 0.279 | 0.325 | 0.344
GEMBA-Dav2-DA | 85.8% | 0.231 | 0.302 | 0.303
GEMBA-Dav3-Classes | 85.4% | 0.235 | 0.289 | 0.251
GEMBA-Dav3-SQM | 85.4% | 0.283 | 0.308 | 0.346
MetricX XXL | 85.0% | 0.360 | 0.420 | 0.427
GEMBA-Chat-Stars | 85.0% | 0.292 | 0.248 | 0.343
GEMBA-Chat-SQM | 85.0% | 0.250 | 0.293 | 0.310
BLEURT-20 | 84.7% | 0.344 | 0.359 | 0.361
GEMBA-Turbo-Stars[noref] | 84.3% | 0.255 | 0.279 | 0.261
COMET-22 | 83.9% | 0.368 | 0.400 | 0.428
GEMBA-Dav2-DA[noref] | 83.9% | 0.209 | 0.285 | 0.280
COMET-20 | 83.6% | 0.319 | 0.330 | 0.332
GEMBA-Chat-Classes[noref] | 83.6% | 0.193 | 0.306 | 0.256
GEMBA-Chat-Stars[noref] | 83.6% | 0.209 | 0.323 | 0.356
GEMBA-Dav3-Stars[noref] | 83.2% | 0.198 | 0.310 | 0.235
UniTE | 82.8% | 0.369 | 0.378 | 0.357
MS-COMET-22 | 82.8% | 0.283 | 0.351 | 0.341
GEMBA-Dav2-SQM[noref] | 82.8% | 0.216 | 0.306 | 0.310
GEMBA-Dav3-SQM[noref] | 82.5% | 0.218 | 0.328 | 0.268
GEMBA-Turbo-Classes | 82.5% | 0.170 | 0.167 | 0.178
GEMBA-Chat-DA[noref] | 82.1% | 0.231 | 0.332 | 0.359
MATESE | 81.0% | 0.323 | 0.279 | 0.389
GEMBA-Chat-SQM[noref] | 81.0% | 0.224 | 0.320 | 0.284
GEMBA-Chat-DA | 81.0% | 0.307 | 0.328 | 0.361
GEMBA-Dav2-Classes | 79.6% | 0.173 | 0.260 | 0.184
GEMBA-Dav2-Stars[noref] | 79.6% | 0.142 | 0.203 | 0.193
YiSi-1 | 79.2% | 0.235 | 0.227 | 0.296
COMETKiwi[noref] | 78.8% | 0.290 | 0.359 | 0.364
GEMBA-Dav3-Classes[noref] | 78.8% | 0.176 | 0.271 | 0.172
COMET-QE[noref] | 78.1% | 0.281 | 0.341 | 0.365
GEMBA-Dav2-Classes[noref] | 78.1% | 0.105 | 0.172 | 0.128
BERTScore | 77.4% | 0.232 | 0.192 | 0.316
UniTE-src[noref] | 75.9% | 0.287 | 0.342 | 0.343
MS-COMET-QE-22[noref] | 75.5% | 0.233 | 0.305 | 0.287
MATESE-QE[noref] | 74.8% | 0.244 | 0.229 | 0.337
f200spBLEU | 74.1% | 0.180 | 0.153 | 0.140
chrF | 73.4% | 0.214 | 0.168 | 0.147
BLEU | 70.8% | 0.169 | 0.140 | 0.145
GEMBA-Turbo-Classes[noref] | 62.0% | -0.010 | 0.027 | 0.029
GEMBA-Curie-Classes[noref] | 61.7% | 0.001 | -0.007 | -0.053
GEMBA-Bab-Stars[noref] | 58.4% | — | — | —
GEMBA-Bab-DA[noref] | 55.8% | -0.119 | -0.011 | —
GEMBA-Curie-Stars[noref] | 54.7% | — | — | —
GEMBA-Curie-DA | 54.4% | — | — | —
GEMBA-Curie-DA[noref] | 51.8% | — | 0.054 | —
GEMBA-Bab-SQM | 51.8% | — | — | —
GEMBA-Bab-SQM[noref] | 51.1% | -0.010 | 0.006 | —
GEMBA-Bab-Stars | 48.2% | — | — | —
GEMBA-Bab-Classes | 47.4% | -0.086 | -0.089 | -0.066
GEMBA-Curie-Classes | 43.4% | -0.011 | -0.035 | -0.035
GEMBA-Curie-SQM[noref] | 41.6% | — | — | —
GEMBA-Curie-SQM | 40.5% | 0.026 | 0.013 | 0.007
GEMBA-Bab-DA | 39.1% | -0.110 | — | —
GEMBA-Curie-Stars | 37.2% | — | — | —
GEMBA-Bab-Classes[noref] | 35.0% | -0.010 | -0.033 | 0.067
Table 7: System-level pairwise accuracy and segment-level Kendall Tau scores.

##  Appendix C Appendix: Ada Answers

We present some of the replies from the Ada model on various prompts in Table
8 to illustrate the frequent, invalid answers of this model. Similar behaviour
was observed when querying GPT 2. Both models failed with respect to our
translation quality assessment experiment.

DA |  | 0 = no meaning preserved
---
1 = perfect meaning and grammar
2 = excellent meaning
3 = very good meaning
4 = excellent
5 = very good
6 = excellent
7 = excellent
8 = excellent
9 = excellent
10 = perfect
DA |  | No meaning preserved: 0
---
Perfect meaning and grammar: 100
DA |  | 0 – It does not exist in my language
---
1 – It does not exist in my text
2 – It does not exist in my text
3 – It exists in my text
4 – It exists in my text
5 – It exists in my text
6 – It exists in my text
7 – It exists in my text
8 – It exists in my text
9 – It exists in my text
10 – It exists in my text
DA | This sentence does not have a score.
DA[noref] |  | The source for this translation is English. The translation is 1 out of 100.
---
DA[noref] |  | 0 = no meaning preserved
---
1 = perfect meaning and grammar
2 = perfect meaning
SQM |  | Some meaning preserved
---
Most meaning preserved and few grammar mistakes
SQM[noref] |  | No meaning preserved
---
Some meaning preserved
Most meaning preserved and few grammar mistakes
Table 8: Answers by the Ada model for various prompts. We observe that SQM
prompts are closer to expected outputs than answers to the corresponding DA
prompts. Similar behaviour was observed when querying GPT 2.