# Enhanced quantum tunneling in quantum Zeno dynamics freezing momentum
direction

Miguel A. Porras  Grupo de Sistemas Complejos, ETSIME, Universidad Politécnica
de Madrid, Rios Rosas 21, 28003 Madrid, Spain  Nilo Mata  Departamento de
Energía y Combustibles, ETSIME, Universidad Politécnica de Madrid, Rios Rosas
21, 28003 Madrid, Spain  Isabel Gonzalo  Departamento de Óptica, Facultad de
Ciencias Físicas, Universidad Complutense de Madrid, Ciudad Universitaria s/n,
28040 Madrid, Spain

###### Abstract

Quantum tunneling is a fundamental quantum mechanical effect involved in
plenty of physical phenomena. Its control would impact these phenomena and the
technologies based on them. We show that the quantum tunneling probability
through a potential barrier can be increased to approach unity in a quantum
Zeno dynamics undergone by the tunneling particle in which the direction of
the momentum is frequently monitored. We first model the measurements of the
momentum direction as selective von Neumann projections, and then as
nonselective, direction-sensitive interactions of the particle with probe
particles. Nonselective measurements are more efficient than selective
measurements in enhancing the quantum tunneling probability.

##  I Introduction

To explain the quantum Zeno effect (QZE) it is often said that “a watched pot
never boils”. Frequent measurements that ascertain whether a quantum system
remains in its initial state slow down its evolution. The QZE has been known
for almost a century NEUMANN , was given its name after MISRA , and has been
observed in diverse physical systems ITANO ; FISCHER . The simplest
manifestations take place in quantum two-level systems; for example, the
inhibition of decay of an unstable state ITANO , or inhibition of quantum
tunneling (QT) in a double-well potential ALTENMULLER ; LERNER . If the
repeated measurements are faster than the short-term parabolic transition
probability characteristic of Schrödinger evolution, decay or QT become less
probable, and impossible with a continuous of measurements. Still, the QZE or
“paradox” is not exempt of controversies, also in the double-well with
different models of measurement PASCAZIO ; FEARN ; HOME ; RUSECKAS ; GODBEER .

A generalization of the QZE is quantum Zeno dynamics (QZD) FACCHI2 . The
frequent measurements of an observable of the system ascertain here whether
the state of the system is in a multidimensional subspace of the system’s
Hilbert space FACCHI1 . In QZD, the remaining or concomitant QZE (in the sense
of freezing the evolution) is more subtle: what tends to be frozen is the
transition from the multidimensional subspace defined by the measurements to
the complementary subspace, but the system continues to evolve coherently
within each subspace FACCHI3 . QZD has recently been observed in a rubidium
condensate, where a superselection rule between two- and three-dimensional
subspaces arises SCHAFER .

QZD becomes more involved in an infinite-dimensional Hilbert space FACCHI1 ,
as that of a simple moving particle. In all previous analysis the measurements
involve position measurements, which, being necessarily imprecise, are aimed
to ascertain the location in a region of space, say $\Delta x$. Ref. FACCHI4
provides a theoretical demonstration that the QZD in the limit of continuous,
selective measurements of position, modeled as von Neumann projections,
becomes a unitary evolution confined in the Hilbert subspace defined by
$\Delta x$ with hard, Dirichlet boundary conditions. With a discrete number
$N$ of measurements, a free particle —the Zeno arrow— is seen to tend to stop
in $\Delta x$ when $N\rightarrow\infty$, forming a quantum-cat-state of the
particle moving forward and backward at the same time PORRAS . In WALLACE ,
computer simulations of QZD (although it is called QZE) with measurements of
position, modeled either as nonselective projections or pointer measurements,
are also be seen to freeze the particle. The dynamics of a particle subjected
to position measurements and undergoing reflection, localization or exclusion
from the measurement region has been studied in detail in Refs. MACKRORY ;
GORDON .

To our knowledge, the complementary QZD with momentum measurements in place of
position measurements has not been addressed. Except for the harmonic
oscillator, there is no a space-momentum symmetry in the dynamics of a
particle, so that QZD with position and with momentum measurements are
different dynamical problems. For a free particle, a QZD with projective
measurements of position is of interest because each measurement changes the
position probability density at the time of the next measurement, but a QZD
with measurements of momentum is trivial because the measurements do not
change the momentum probability density. Expressed in the terms of modern
quantum information and technologies BRAGINSKY ; RALPH ; SHOMRONI , momentum
is a quantum nondemolition variable for a free particle whose measurement back
action effects can be evaded, and position is not. For a particle subjected to
forces, however, momentum is no longer a quantum nondemolition variable, which
triggers the rich QZD studied here.

In this paper we show that repeated measurements that would ascertain whether
the momentum is in a region of momentum space tend to freeze the momentum in
that region. We focus on the effect of this QZD on QT through a potential
barrier. To the purpose of tunneling the barrier, it turns out that
measurements that simply determine the direction of momentum (positive or
negative in our one-dimensional geometry) suffice for the direction to freeze,
and therefore for the particle to pass the barrier (see an animation in the
supplementary material SUPPLEMENTAL ).

We first adopt an operational approach to measurements as selective von
Neumann projections. For a particle launched towards positive $x$, only the
possibility that the momentum is positive in all measurements is considered.
Its simplicity allows us to simulate the effect of high enough number of
measurements with which the QT probability approaches unity. Next we move to a
more physical model of measurement as a nonselective interaction with a probe
particle to which information on the direction of the particle is transferred,
and reduces the state of the particle to a mixture of states with opposite
momenta SIMONIUS ; ZUREK ; MENSKY . In this picture, all possibilities that
the momentum is positive at the last measurement are considered. The
measurements as interactions also cause the direction to freeze, and the
particle to tunnel, with a probability higher than with the selective
measurements, and therefore also approaching unity as their frequency
increases, pointing to the emergence of a superselection rule between
Hilbert’s subspaces of different momenta.

Contrary to QZE-induced QT inhibition in a double-well by measuring well
occupation, QZD with momentum measurements promotes QT. As such, relevant
related phenomena would be those where enhancement of QT will have an impact,
such as facilitating nucleus-nucleus collisions in colliders SALGADO , or
controlling of reaction kinetics SCHREINER . In optics, promoting electron
tunneling can have an impact in photoelectron ionization generating high
harmonics. Vice versa, high harmonics are being employed to probe with
attosecond resolution proton dynamics PROTON  and electrons tunneling a
barrier INTERFEROMETRY ; CLOCK . These breakthroughs would bring closer to
hand an implementation of this QZD.

##  II Preliminaries

We consider a particle of mass $m$, initial state $|\psi_{0}\rangle$ of wave
function $\langle x|\psi_{0}\rangle=\psi_{0}(x)=\psi(x)\exp(ik_{0}x)$ well-
within the half-space $x<0$. The mean momentum $p_{0}=\hbar k_{0}$ is positive
and such that its kinetic energy $E_{c}=p_{0}^{2}/2m$ is smaller than the peak
potential energy $V_{0}$ of a barrier $V(x)$ localized about $x=0$. For
concreteness we will use $V(x)=V_{0}/(1+|x/b|^{\alpha})$, where $b$ measures
the half-width of the barrier, and $\alpha>0$ ($\alpha\rightarrow\infty$ is a
square barrier). Of course the details of the tunneling dynamics for different
barrier shapes, but the results concerning QT probability under QZD are the
same. For a well-directed particle translation dominates over wave packet
spreading, implying that $p_{0}$ is larger than its uncertainty, $\Delta p$.
We rewrite Schrödinger equation,

$i\hbar\frac{\partial\psi}{\partial
t}=-\frac{\hbar^{2}}{2m}\frac{\partial^{2}\psi}{\partial x^{2}}+V\psi\,,$ (1)

using the dimensionless spatial coordinate $\xi=x/(\hbar/\Delta p)$ ($x$ in
units of the wave function width $\Delta x\sim\hbar/\Delta p$) and
dimensionless time $\tau=t/(2m\Delta x/p_{0})=t/(2\hbar m/\Delta pp_{0})$
(time in units of the time taken for the particle to traverse its own wave
function), as

$i\frac{\partial\psi}{\partial\tau}=-\frac{1}{\kappa_{0}}\frac{\partial^{2}\psi}{\partial\xi^{2}}+\kappa_{0}v_{0}v\psi\,,$
(2)

where $v=1/(1+|\xi/\xi_{b}|^{\alpha})$, and $\xi_{b}$ is the potential width
in units of the wave function width. The parameters at work in the QT dynamics
are then $v_{0}=V_{0}/E_{c}$ and $\kappa_{0}=p_{0}/\Delta p$. The situation of
interest is that with $v_{0}>1$ (QT) and $\kappa_{0}>1$ (well-directed
particle).

##  III Quantum Zeno dynamics of a tunneling particle with selective
measurements of momentum direction

In contrast to previous research, the QZD considered here is that experienced
by the particle subjected to frequent measurements of momentum, i.e.,
measurements that ascertain whether the momentum is in a certain interval of
momentum space or not. However, in the context of QT, the QZD turns out to be
qualitatively similar if this interval is $\Delta p$, any other interval about
$p_{0}$, or is simply positive.

In a first approach, the measurements are operationally summarized as
projections of the state onto the subspace of positive momenta. These
measurements are selective, i.e., meaning that we only consider the event that
the momentum is positive in all the measurements. This would be the situation
with a measuring device that eliminates the particle upon a negative outcome.
In the standard Zeno scheme, the particle evolves according to (2) during a
certain total time $\tau_{\rm max}$, while the evolution is $N$ times
interrupted to monitor the direction of movement. The projective measurements
are performed at the equispaced time intervals $\Delta\tau=\tau_{\rm max}/N$.
Formally, the state of the particle at the time $\tau_{\rm max}$ is given by

$|\phi_{N}\rangle=\frac{1}{\sqrt{P_{N}}}\Pi
e^{-iH\Delta\tau}\\!\\!\dots\frac{1}{\sqrt{P_{2}}}\Pi
e^{-iH\Delta\tau}\frac{1}{\sqrt{P_{1}}}\Pi
e^{-iH\Delta\tau}|\psi_{0}\rangle\,,$ (3)

where
$H=(-1/\kappa_{0})\partial^{2}\psi/\partial\xi^{2}+\kappa_{0}v_{0}v(\xi)$ is
the Hamiltonian, $\Pi=\int_{0}^{\infty}|\kappa\rangle\langle\kappa|d\kappa$ is
the projector onto the subspace of positive normalized momenta
$\kappa=p/\Delta p$, and $P_{n}$ is the probability that the momentum is
positive at the $n$ intermediate measurement, so that the factors
$1/\sqrt{P_{n}}$ normalize the state after each projective measurement. After
the $N$ measurement at time $\tau_{\rm max}$, the probability that all
outcomes were positive is the product $P_{N}^{(s)}=P_{1}P_{2}\dots P_{N}$.

To obtain $P_{N}^{(s)}$, however, it is simpler and faster computationally to
evaluate

$|\psi_{N}\rangle=\Pi e^{-iH\Delta\tau}\stackrel{{{}^{\mbox{\small$N$
times}}}}{{\dots\dots}}\Pi e^{-iH\Delta\tau}\Pi
e^{-iH\Delta\tau}|\psi_{0}\rangle\,,$ (4)

which is obviously related to the normalized state $|\phi_{N}\rangle$ in (3)
by $|\psi_{N}\rangle=\sqrt{P_{1}P_{2}\dots P_{N}}\,|\phi_{N}\rangle$, and
whose norm, $\langle\psi_{N}|\psi_{N}\rangle=P_{1}P_{2}\dots
P_{N}=P_{N}^{(s)}$, yields directly the probability that all outcomes were
positive. If desired, the normalized final state can be calculated from
$|\phi_{N}\rangle=|\psi_{N}\rangle/\sqrt{P_{N}^{(s)}}$.

On a computer, each $\Pi e^{-iH\Delta\tau}$ step is performed by solving (2)
for the wave function in space representation using a split-step fast Fourier
transform algorithm, and then eliminating negative momenta by multiplying the
wave function in momentum representation by the Heaviside step function. If
$\langle\kappa|\psi_{N}\rangle=\hat{\psi}_{N}(\kappa)$ is the momentum wave
function at $\tau_{\rm max}$, the probability that all outcomes were positive
can be evaluated as

$P_{N}^{(s)}=\langle\psi_{N}|\psi_{N}\rangle=\int_{-\infty}^{\infty}|\hat{\psi}_{N}(\kappa)|^{2}d\kappa$
(5)

(the lower limit can be set to zero due to the last projection at $\tau_{\rm
max}$).

To analyze QT under this QZD, we choose $\psi_{0}(x)$ such that the barrier
does not yet affect the particle, and the time $\tau_{\rm max}$ such that the
transmission/reflection process is completed (with or without measurements).
For a particle launched from the left, the probability of finding the particle
to the right of the barrier at $\tau_{\rm max}$,
$P_{x>0}^{(s)}=\int_{0}^{\infty}|\psi_{N}(\xi)|^{2}d\xi$, must coincide with
the probability $P_{N}^{(s)}$ that the momentum is positive, since the
reflected wave function at $x<0$ has only negative momenta. With a single
measurement, $P_{1}^{(s)}$ yields the standard QT probability.

Figure 1:  (a) Partially transmitted and reflected probability density (solid
curves) at time $\tau_{\rm max}=6.24$ for the input wave packet
$\psi=(2/\pi)^{1/4}\exp{[-(\xi-\xi_{0})^{2}]}\exp(i\kappa_{0}\xi)$
($\kappa_{0}=4$, $\xi_{0}=-4$) at $\tau=0$ (dashed curve) through the barrier
of width $\xi_{b}=1$, $\alpha=6$ and height $v_{0}=1.5$ (red dotted curve)
without any measurement. Tunneling probability is $P^{(s)}_{1}=0.162$. (b) The
same but with $N=2^{4},2^{6},2^{8}$ and $2^{10}$ measurements between $\tau=0$
and $\tau_{\rm max}$ yielding respective tunneling probabilities
$P_{N}^{(s)}=0.31,0.69,0.90$, and $0.97$. The weak reflected wave is
eliminated by the projections. The vertical lines indicate the turning points,
the horizontal line the energy, and the arrows the movement direction. See
animations of (a) and (b) from $\tau=0$ to $\tau_{\rm max}$ in SUPPLEMENTAL .
(c) Probability that the momentum direction remains positive at the $n$th
measurement, for different values of $N$. (d) Tunneling probability as a
function of $N$ for several values of $\kappa_{0}$.

In the example of Fig. 1(a), a Gaussian wave packet (dashed curve) is launched
against the potential barrier (dotted curve). Its shape is neither too sharp
(high $\alpha$) nor too delocalized (low $\alpha$) to facilitate numerical
calculations (less spatial points and temporal steps are required). Without
measurements the QT probability is $P_{1}=0.162$. Figure 1(b) evidences that
monitoring more and more frequently whether the direction remains positive
increases the QT probability by freezing the momentum direction. See the QT
dynamics in more detail in the videos in the supplementary material
SUPPLEMENTAL . To illustrate how the increasing tunneling probability
$P_{N}^{(s)}$ is furnished from one to the next measurement, Fig. 1(c)
represents the decreasing probability $P_{1}P_{2}\dots P_{n}$ that the
momentum remains positive as the number $n$ of measurements performed
increases up to $N$. The larger $N$ (the more steps down), the higher
$P_{N}^{(s)}$ (the less you go down). The QT probability $P_{N}^{(s)}$ is
represented as a function of $N$ in Fig. 1(d) for different values of
$\kappa_{0}$. As $\kappa_{0}\rightarrow\infty$, the QT probability without
measurements tends to zero, which, given the fixed value of the barrier height
$v_{0}>1$, can be regarded as the classical limit. The QT probability tends
always to unity, but it needs more measurements as $\kappa_{0}$ increases.

The QZD of the tunneling particle, without and with measurements, can be seen
unfolded over time in the animations in the supplementary material
SUPPLEMENTAL  and in Fig. 2. Without measurements (left), the spike that
emerges when the wave function reaches the first turning point is mostly
reflected and spreads subsequently. With measurements of momentum direction
(right), the spike is not reflected, but transforms momentarily into a wide
wave packet of low probability density ($\tau\simeq 2.36$) and high positive
current under the barrier, and quickly revives in a new spike beyond the
second turning point.

The QZD in phase space can be seen in the animations of the corresponding
Wigner distribution functions in the supplemental material SUPPLEMENTAL , and
in the selected snapshots in Fig. 3. In contrast to the unmeasured Wigner
distribution (top), the frequently measured distribution (bottom) is reflected
at zero momentum towards positive momenta with higher probability as $N$
increases, while the potential barrier substantially ceases to act as such. At
tunneling times (e.g. $\tau\simeq 2.36$) the Wigner distribution function of
the state of low probability density under the barrier displays not only small
ripples but large alternate regions of positive and negative values (blue and
red), indicating a highly non-classical behavior.

Figure 2:  For the same example as in Fig. 1, contour plot of the probability
density in space and time (a) without measurements and (b) with $N=2^{11}$
measurements. The horizontal white lines are the classical turning points.

Figure 3:  For the same example as in Fig. 1, Wigner distribution function
$W(\xi,\kappa)=(1/\pi)\int_{-\infty}^{\infty}d\eta\psi^{\star}(\xi+\eta)\psi(\xi-\eta)e^{2i\kappa\eta}$
at the indicated instants of time (a,b) without any measurement, and (c,d)
with $N=2^{11}$ measurements. The initial ($\tau=0$) Wigner distribution
function is in the box in (a) and (c). The white vertical lines localize the
classical turning points. See the evolution in time of the Wigner distribution
function in the animation in SUPPLEMENTAL .

##  IV Quantum Zeno dynamics of a tunneling particle with nonselective
interactions measuring momentum direction

These results motivate us to formulate the same problem using a more physical
model of the measurement process SIMONIUS ; ZUREK ; MENSKY , which in turn
could shed light on how they could be implemented in practice. Each
measurement is modelled as a short interaction with an environmental particle,
or probe, to which information on the direction of the particle is
transferred. Upon tracing over the probe states after the interaction, the
positive and negative parts of the wave function decohere. A possible
implementation of these measurements would be the elastic scattering of
photons by a moving atom or ion, as sketched in Fig. 4. Depending on whether a
photon is backscattered by the forward or backward moving atom, the photon
would be red or blue Doppler shifted, while the atom remains substantially
unaltered in its energy and momentum GUO . These measurements discriminate
between the two directions of movement, but neither direction is selected.

With the same QZD scheme, the initial pure state $|\psi_{0}\rangle$ evolves to
$|\psi_{1}\rangle=e^{-iH\Delta\tau}|\psi_{0}\rangle$ in a first time interval
$\Delta\tau$. Under the action of the potential barrier and wave packet
spreading, $|\psi_{1}\rangle$ may have acquired positive and negative momenta
and hence can always be written as the sum
$|\psi_{1}\rangle=|\psi_{1,+}\rangle+|\psi_{1,-}\rangle$ of the orthogonal
vectors

$|\psi_{1,+}\rangle=\int_{0}^{\infty}\\!\\!\\!\\!\\!d\kappa\hat{\psi}_{1}(\kappa)|\kappa\rangle\,,\quad|\psi_{1,-}\rangle=\int_{-\infty}^{0}\\!\\!\\!\\!\\!d\kappa\hat{\psi}_{1}(\kappa)|\kappa\rangle\,.$
(6)

of only positive or negative momenta. Of course they can be written in terms
of normalized states as
$|\psi_{1,\pm}\rangle=\sqrt{P_{\pm}}|\phi_{1,\pm}\rangle$, but as above for
selective measurements, the probabilities of positive and negative outcome are
stored in $|\psi_{1,\pm}\rangle$ as

$\langle\psi_{1,+}|\psi_{1,+}\rangle=P_{+},\quad\langle\psi_{1,-}|\psi_{1,-}\rangle=P_{-},$

The vectors $|\psi_{1,\pm}\rangle$ are directly accessible in momentum
representation as
$\hat{\psi}_{1,\pm}(\kappa)=\hat{\psi}_{1}(\kappa)\theta(\pm\kappa)$, where
$\theta(\cdot)$ is the Heaviside step function, without computing any
integral, which saves computational time.

Interaction with the probe, whose initial state is $|\chi\rangle$, leads to
the transitions
$|\psi_{1,\pm}\rangle|\chi\rangle\rightarrow|\psi_{1,\pm}\rangle|\chi_{\pm}\rangle$
of the system. Discrimination between positive and negative momenta of the
particle requires the states $|\chi_{+}\rangle$ and $|\chi_{-}\rangle$ of the
probe to be orthonormal. For the state $|\psi_{1}\rangle$ at $\Delta\tau$, the
result of the interaction is the entangled state
$|\psi_{1,+}\rangle|\chi_{+}\rangle+|\psi_{1,-}\rangle|\chi_{-}\rangle$, whose
density matrix is

$\displaystyle\rho_{\rm 1,\rm meas}$ $\displaystyle=$
$\displaystyle\left(|\psi_{1,+}\rangle|\chi_{+}\rangle+|\psi_{1,-}\rangle|\chi_{-}\rangle\right)$
(7) $\displaystyle\times$
$\displaystyle\left(\langle\psi_{1,+}|\langle\chi_{+}|+\langle\psi_{1,-}|\langle\chi_{-}|\right)\,,$

and, if only the particle is considered after separation of the probe, its
reduced density matrix is obtained by tracing over the probe:

$\displaystyle\rho_{1,\rm red}$ $\displaystyle=$
$\displaystyle\langle\chi_{+}|\rho_{\rm
meas}|\chi_{+}\rangle+\langle\chi_{-}|\rho_{\rm meas}|\chi_{-}\rangle$ (8)
$\displaystyle=$
$\displaystyle|\psi_{1,+}\rangle\langle\psi_{1,+}|+|\psi_{1,-}\rangle\langle\psi_{1,-}|\,,$

which is a incoherent mixture of a particle travelling forward or backward
with probabilities equal to the respective norms.

Figure 4:  Nonselective measurement of momentum direction of an atom by
photons. The backscattered photon is red-shifted or blue-shifted depending it
finds the atom moving forwards or backwards.

The particle is next left to evolve another time $\Delta\tau$ to
$\rho_{2}=e^{-iH\Delta\tau}\rho_{1,\rm red}e^{iH\Delta\tau}$, that is,

$\displaystyle\rho_{2}$ $\displaystyle=$ $\displaystyle
e^{-iH\Delta\tau}|\psi_{1,+}\rangle\langle\psi_{1,+}|e^{iH\Delta\tau}$ (9)
$\displaystyle+$ $\displaystyle
e^{-iH\Delta\tau}|\psi_{1,-}\rangle\langle\psi_{1,-}|e^{iH\Delta\tau}\,,$

where, as before,
$e^{-iH\Delta\tau}|\psi_{1,+}\rangle\equiv|\psi_{2,+}\rangle=|\psi_{2,++}\rangle+|\psi_{2,+-}\rangle$
and
$e^{-iH\Delta\tau}|\psi_{1,-}\rangle\equiv|\psi_{2,-}\rangle=|\psi_{2,-+}\rangle+|\psi_{2,--}\rangle$,
since, again, any of these states may acquire opposite momenta. Here, also as
above,

$\displaystyle|\psi_{2,++}\rangle$ $\displaystyle=$
$\displaystyle\\!\\!\int_{0}^{\infty}\\!\\!\\!\\!\\!d\kappa\hat{\psi}_{2,+}(\kappa)|\kappa\rangle,\quad|\psi_{2,+-}\rangle=\\!\\!\int_{-\infty}^{0}\\!\\!\\!\\!\\!d\kappa\hat{\psi}_{2,+}(\kappa)|\kappa\rangle,$
$\displaystyle|\psi_{2,-+}\rangle$ $\displaystyle=$
$\displaystyle\\!\\!\int_{0}^{\infty}\\!\\!\\!\\!\\!d\kappa\hat{\psi}_{2,-}(\kappa)|\kappa\rangle,\quad|\psi_{2,--}\rangle=\\!\\!\int_{-\infty}^{0}\\!\\!\\!\\!\\!d\kappa\hat{\psi}_{2,-}(\kappa)|\kappa\rangle,$

whose norms

$\displaystyle\langle\psi_{2,++}|\psi_{2,++}\rangle=P_{+}P_{++},\quad\langle\psi_{2,+-}|\psi_{2,+-}\rangle=P_{+}P_{+-},$
$\displaystyle\langle\psi_{2,-+}|\psi_{2,-+}\rangle=P_{-}P_{-+},\quad\langle\psi_{2,--}|\psi_{2,--}\rangle=P_{-}P_{--},$

provide the probabilities of realization of the four events. The density
matrix is then written as

$\displaystyle\rho_{2}$ $\displaystyle=$
$\displaystyle(|\psi_{2,++}\rangle+|\psi_{2,+-}\rangle)(\langle\psi_{2,++}|+\langle\psi_{2,+-}|)$
(10) $\displaystyle+$
$\displaystyle(|\psi_{2,-+}\rangle+|\psi_{2,--}\rangle)(\langle\psi_{2,-+}|+\langle\psi_{2,--}|)\,,$

and in a second interaction with a probe
($|\psi_{2,\dots\pm}\rangle|\chi\rangle\rightarrow|\psi_{2,\dots\pm}\rangle|\chi_{\pm}\rangle$)
the density matrix of the particle-probe system is

$\displaystyle\rho_{2,\rm meas}$ $\displaystyle=$
$\displaystyle(|\psi_{2,++}|\chi_{+}\rangle\rangle+|\psi_{2,+-}\rangle|\chi_{-}\rangle)$
(11) $\displaystyle\times$
$\displaystyle(\langle\psi_{2,++}|\langle\chi_{+}|+\langle\psi_{2,+-}|\langle\chi_{-}|)$
$\displaystyle+$
$\displaystyle(|\psi_{2,-+}\rangle|\chi_{+}\rangle+|\psi_{2,--}\rangle|\chi_{-}\rangle)$
$\displaystyle\times$
$\displaystyle(\langle\psi_{2,-+}|\langle\chi_{+}|+\langle\psi_{2,--}|\langle\chi_{-}|)\,.$

Tracing over the probe after the interaction, the reduced density matrix of
the particle is

$\displaystyle\rho_{2,\rm red}$ $\displaystyle=$
$\displaystyle|\psi_{2,++}\rangle\langle\psi_{2,++}|+|\psi_{2,+-}\rangle\langle\psi_{2,+-}|$
(12) $\displaystyle+$
$\displaystyle|\psi_{2,-+}\rangle\langle\psi_{2,-+}|+|\psi_{2,--}\rangle\langle\psi_{2,--}|\,,$

which is a incoherent mixture of four states of four possible outcomes in the
two measurements with norms equal to the respective probabilities of
realization.

Figure 5:  (a) Partially transmitted and reflected probability density at time
$\tau_{\rm max}=3.56$ for the input wave packet
$\psi=(2/\pi)^{1/4}\exp{[-(\xi-\xi_{0})^{2}]}\exp(i\kappa_{0}\xi)$
($\kappa_{0}=4$, $\xi_{0}=-2.67$) at $\tau=0$ through the barrier of width
$\xi_{b}=1$, $\alpha=6$ and height $v_{0}=1.5$ (red dotted curve) without any
measurement (dashed curve), and the same with $N=16$ measurements (solid
curve). To concentrate the measurements, 15 of them are performed between
$\tau_{1}=0.88$ and $\tau_{2}=3$, and the last one at $\tau_{\rm max}$. QT
probability without measurements is $P^{(ns)}_{1}=0.162$, and with $N=16$
measurements is $P^{(ns)}_{1}=0.579$. The vertical lines indicate the turning
points, the horizontal line the energy, and the arrows the movement direction.
(b) QT probability as a function of $N$ when measurements are nonselective
(black curve) and selective (gray curve), for comparison.

This procedure is repeated up to $N$ times, whereby a mixture of $2^{N}$
states, e.g., $|\psi_{N,++-++\dots+-}\rangle$, (the number of plus and minus
signs is $N$) corresponding to the $2^{N}$ possible outcomes in each of the
$N$ measurements is obtained, with probabilities equal to their norms. As with
selective measurements, the $2^{N}$ normalized states,
$|\phi_{N,++-++\dots+-}\rangle$, can be obtained by dividing
$|\psi_{N,++-++\dots+-}\rangle$ by
$\sqrt{\langle\psi_{N,++-++\dots+-}|\psi_{N,++-++\dots+-}\rangle}$, if
desired.

Among these states, there are $2^{N-1}$ states yielding positive direction at
the last measurement. Thus, the probability that the direction of the particle
is positive at the $N$ measurement is the sum of their respective
probabilities:

$P^{(ns)}_{N}=\sum^{2^{N-1}}\langle\psi_{N,\dots+}|\psi_{N,\dots+}\rangle=\sum^{2^{N-1}}\\!\\!\int_{-\infty}^{\infty}\\!\\!|\hat{\psi}_{N,\dots+}(\kappa)|^{2}d\kappa$
(13)

(again the lower limit can be set to zero). Note that the term with $N$
positive signs is the probability that all outcomes were positive and
therefore coincides with $P_{N}^{(s)}$ in (5). We then conclude that these
interactions of the particle with probe particles discriminating the particle
direction also result in freezing the direction, and their efficiency is
higher than with selective measurements.

If, for brevity we call each incoherent state at the $N$th measurement
$|\psi_{N,i}\rangle$, the probability density of finding the particle at $\xi$
is $\langle\xi|\rho_{N,\rm
red}|\xi\rangle=\sum_{i}^{2^{N}}|\langle\xi|\psi_{N,i}\rangle|^{2}=\sum_{i}^{2^{N}}|\psi_{N,i}(\xi)|^{2}$,
verifying $\int_{-\infty}^{\infty}\langle\xi|\rho_{N,\rm red}|\xi\rangle
d\xi=1$ since temporal evolution and measurements are unitary transformations.
For a particle initially located well-before the barrier, and with a
sufficiently long time $\tau_{\rm max}$, the probability that the particle is
to the right of the barrier,

$P_{x>0}^{(ns)}=\int_{0}^{\infty}\langle\xi|\rho_{N,\rm red}|\xi\rangle
d\xi=\sum_{i}^{2^{N}}\int_{0}^{\infty}|\psi_{N,i}(\xi)|^{2}d\xi\,,$ (14)

must coincide with $P_{N}^{(ns)}$, implying also an enhanced QT probability.

We have implemented the above procedure on a computer and relevant results are
depicted in Fig. 5. Note that the number of times Schrödinger equation (2) is
to be solved with $N$ measurements is $2^{0}$ up to the first measurement,
$2^{1}$ up to the second, and so on, i.e., $\sum_{n=0}^{N}2^{n}=2^{N+1}-1$,
which grows exponentially and limits the maximum number of measurements to a
few tens in our computer facility. Given the difficulty of simulating a high
number of measurements, and that they have no effect on the freely moving
particle before and after the barrier, the $N$ measurements between $\tau=0$
and $\tau_{\rm max}$ are concentrated in $N-1$ measurements in a shorter time
interval $[\tau_{1},\tau_{2}]$ in which the wave function significantly
overlaps the potential, plus the $N$ measurement at $\tau_{\rm max}$ for the
reflection/transmission process to be concluded. The interval
$[\tau_{1},\tau_{2}]$ is the same for all $N$, so that increasing $N$ is the
same as increasing the frequency. Under the conditions specified in the
caption, Fig. 5(a) shows the spatial probability densities with $N=1$, with QT
probability $P_{1}^{(ns)}=0.162$, and with $N=16$ measurements, with QT
probability $P_{16}^{(ns)}=0.579$. The plot of $P_{N}^{(ns)}$ versus $N$ in
Fig. 5(b) evidences the increasing probability that the momentum direction is
frozen Zeno effect, and hence the increasing QT probability. Since
$P_{N}^{(s)}$ (gray curve) approaches unity, $P_{N}^{(ns)}$ (black curve) does
faster.

##  V Concluding remarks and conclusion

Before concluding, we clarify that both QT inhibition in QZD with position
measurements (well occupation) ALTENMULLER ; LERNER  and the present QT
enhancement in QZD dynamics with momentum measurements involve Zeno effects:
they slow down or freeze the magnitude being observed. They tend to generate
superselection rules that prohibit transitions between the respective
subspaces and their complementary.

On the other hand, anti-Zeno enhancement of QT by position measurements has
also been described AL , usually associated with less frequent measurements.
Analogously, an anti-Zeno inhibition of QT by measurements of momentum
direction appears to exist as well, as can be appreciated in the initially
decaying gray curve of Fig. 5(b) for low number of measurements. A detailed
investigation of this anti-Zeno effect is deferred to future work.

To conclude, we have unveiled the existence of a quantum mechanical dynamics
of a particle —a QZD involving frequent measurements of momentum direction—
that deeply affects QT up to the point of ensuring transmission. Given the
enormous advances in attosecond science (see for example INTERFEROMETRY ;
CLOCK  and references therein for all-optical imaging of tunneling electrons),
“Zeno-assisted tunneling” could be implemented in practice, which would impact
the diverse phenomena where QT is a key player.

M.A.P. acknowledges support from Project No. PGC2018-093854-B-I00, and M.A.P.
and I.G. from project No. FIS2017-87360-P of the Spanish Ministerio de
Ciencia, Innovación y Universidades. N.M. acknowledges support from grant No.
D480 (Beca de colaboración de formación) of the Universidad Politécnica de
Madrid.

## References

  * (1) J. von Neumann, Die Mathematische Grundlagen der Quantenmechanik (Berlin, Springer 1932).
  * (2) B. Misra and E.C.G. Sudarshan, “The Zeno’s paradox in quantum theory,” J. Math. Phys. 18 756 (1977).
  * (3) W.M. Itano, D.J. Heinzen, J.J. Bollinger, and D.J. Wineland, “Quantum Zeno effect,” Phys. Rev. A 41 2295 (1990).
  * (4) M.C. Fischer, B. Gutiérrez-Medina, and M. G. Raizen, “Observation of the Quantum Zeno and Anti-Zeno Effects in an Unstable System,” Phy. Rev. Lett. 87, 040402 (2001).
  * (5) T.P. Altenmuller and A. Schenzle, “Quantum Zeno effect in a double-well potential: A model of a physical measurement,” Phys. Rev. A 49, 2016 (1994).
  * (6) L Lerner Eur. J. Phys. “The quantum Zeno effect in double well tunnelling,” 39, 035407 (2018).
  * (7) S. Pascazio and M. Namiki, “Dynamical Zeno effect,” Phys. Rev. A 50, 4582-4592 (1994).
  * (8) H. Fearn and W. E. Lamb, Jr. “Computational approach to the quantum Zeno efect: Position measurements,” Phys. Rev. A 46, 1199 (1992).
  * (9) D. Home and M.A.B. Whitaker, “Comment on ’Computational approach to the quantum Zeno effect: Position measurements’,” Phys. Rev. A 48, 2502 (1993).
  * (10) J. Ruseckas and B. Kaulakys, “Real measurements and the quantum Zeno effect,” Phys. Rev. A 63, 062103 (2001).
  * (11) A.D. Godbeer, J.S. Al-Khalili, and P.D. Stevenson, “Environment-induced dephasing versus von Neumann measurements in proton tunneling,” Phys. Rev. A 90, 012102 (2014).
  * (12) P. Facchi, V. Gorini, G. Marmo, S. Pascazio, and E.C.G. Sudarshan, “Quantum Zeno dynamics,” Phys. Lett. A 275, 12-19 (2000).
  * (13) P. Facchi and S. Pascazio J. Phys. A: Math. Theor. 41, 493001 (2008).
  * (14) P. Facchi and S. Pascazio, “Quantum Zeno Subspaces,” Phys. Re. Lett. 89, 080401 (2002).
  * (15) F. Schäfer, I. Herrera, S. Cherukattil, C. Lovecchio, F.S. Cataliotti, F. Caruso, and A. Smerzi, “Exprimental realization of quantum Zeno dynamics,” Nature. Comm. 5, 3194 (2014).
  * (16) P. Facchi, S. Pascazio, A. Scardicchio, and L. S. Schulman, “Zeno dynamics yields ordinary constraints,” Phys. Rev. A 65, 012108 (2001).
  * (17) M.A. Porras, A. Luis, and I. Gonzalo, “Quantum Zeno effect for a free-moving particle,” Phs. Rev. A 90, 062131 (2014).
  * (18) D. Wallace, “Simple computer model for the quantum Zeno effect,” Phys. Rev. A 63, 022109 (2001).
  * (19) J.B. Mackrory, K. Jacobs, and D.A. Steck, “Reflection of a particle from a quantum measurement,” New. J. Phys. 12, 113023.
  * (20) G. Gordon, I.E. Mazts, and G. Kurizki, “Quantum particle localization by frequenct coherent monitoring,” Phys. Rev. A 87, 052141 (2013).
  * (21) V. B. Braginsky, Y.I. Vorontsov, and K.S. Thorne, “Quantum nondemolition measurements,” Science 209, 547-557 (1980).
  * (22) T.C. Ralph, S.D. Bartlett, J.L. O’Brien, G.J. Pryde, and H.M. Wiseman, “Quantum nondemolition measurements for quantum information”, Phys. Rev. A 73, 012113 (2006).
  * (23) I. Shomroni, L. Qui, D. Malz, A. Nunnenkamp, and T.J. Kippenberg, “Optical backaction-evading measurement of a mechanical oscillator,” Nat. Commun. 10, 2086 (2019).
  * (24) For the same example as in Fig. 1, see Supplemental Material at https://drive.upm.es/s/BSG9dmZW8ptcuFq for the temporal evolution of the probability density without measurements, https://drive.upm.es/s/Trqgt2MkzHLTguU for the evolution with $N=256$ measurements. https://drive.upm.es/s/JPQHZU6A8rmZI6h for the temporal evolution of the Wigner distribution function without measurements, https://drive.upm.es/s/hcl8g5tJvWTYI8F for the evolution with $N=2^{11}$ measurements.
  * (25) M. Simonius, “Spontaneous Symmetry Breaking and Blocking of Metastable States,” Phys. Rev. Lett. 40, 980 (1977).
  * (26) W.H. Zurek, “Pointer basis of quantum apparatus: Into what mixture does the wave packet collapse?,” Phys. Rev. D 24, 1516 (1981).
  * (27) M.B. Mensky, Quantum Measurements and Decoherence. Models and Phenomenology, Fundamental Theories of Physics 110, (Springer-Science+Business Media, B.Y., 2000).
  * (28) C.A. Salgado et al, “Proton–nucleus collisions at the LHC: Scientific opportunities and requirements,” J. Phys. G: Nucl. Part. Phys. 39, 015010 (2012).
  * (29) P.R. Schreiner, “Tunneling Control of Chemical Reactions: The Third Reactivity Paradigm,” J. Am. Chem. Soc. 139 (43), 15276 (2017).
  * (30) S. Baker, J.S. Robinson, C.A. Haworth, H. Teng, R.A. Smith, C.C. Chirila, M. Lein, J.W.G. Tisch, and J.P. Marangos, “Probing proton dynamics in molecules on an attosecond time scale,” Science 312, 424 (2006).
  * (31) O. Pedatzur et al, “Attosecond tunnelling interferometry,” Nat. Phys. 11, 815 (2015). https://doi.org/10.1038/nphys3436
  * (32) I. Babushkin et al, “All-optical attoclock for imaging tunnelling wavepackets,” Nat. Phys. (2022). https://doi.org/10.1038/s41567-022-01505-2
  * (33) W. Guo, “Light scattering from a moving atom,” J. Opt. Soc. Am. A 29, 2576-2578 (2012).
  * (34) J.S. Al-Khalili and P.D. Stevenson, “Simulating the Quantum Zeno and Anti-Zeno Effects During Position Measurements,” Advanced Science Letters 1, 140-144 (2008).


#  Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language
Models on Multilingual NLU Tasks

Dan Saattrup Nielsen
The Alexandra Institute
<EMAIL_ADDRESS>
Kenneth Enevoldsen
University of Aarhus
<EMAIL_ADDRESS>
Peter Schneider-Kamp
University of Southern Denmark
<EMAIL_ADDRESS>
###### Abstract

This paper explores the performance of encoder and decoder language models on
multilingual Natural Language Understanding (NLU) tasks, with a broad focus on
Germanic languages. Building upon the ScandEval benchmark, which initially was
restricted to evaluating encoder models, we extend the evaluation framework to
include decoder models. We introduce a method for evaluating decoder models on
NLU tasks and apply it to the languages Danish, Swedish, Norwegian, Icelandic,
Faroese, German, Dutch, and English. Through a series of experiments and
analyses, we address key research questions regarding the comparative
performance of encoder and decoder models, the impact of NLU task types, and
the variation across language resources. Our findings reveal that decoder
models can achieve significantly better NLU performance than encoder models,
with nuances observed across different tasks and languages. Additionally, we
investigate the correlation between decoders and task performance via a UMAP
analysis, shedding light on the unique capabilities of decoder and encoder
models. This study contributes to a deeper understanding of language model
paradigms in NLU tasks and provides valuable insights for model selection and
evaluation in multilingual settings.

Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language
Models on Multilingual NLU Tasks

Dan Saattrup Nielsen The Alexandra Institute<EMAIL_ADDRESS>Kenneth
Enevoldsen University of Aarhus<EMAIL_ADDRESS>Peter Schneider-
Kamp University of Southern Denmark<EMAIL_ADDRESS>

##  1 Introduction

Language models have attained remarkable Natural Language Understanding (NLU)
performance, both with encoder-based architectures like BERT Devlin et al.
(2018) and and decoder-based architectures like GPT-3 Brown, Tom and Mann,
Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and
Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry,
Girish and Askell, Amanda and others (2020). The encoder models have excelled
in capturing contextual information for downstream tasks through masked
language modeling objectives, while decoder models have shown strong
generative capabilities by autoregressively predicting subsequent tokens based
on preceding context.

Since the “ChatGPT boom” in 2023, the research community has been increasingly
focused on decoder models Zhao et al. (2023) for both Natural Language
Generation (NLG) and NLU tasks. However, few studies have systematically
compared the performance of encoder and decoder models across a diverse range
of NLU tasks, and the studies that exist have primarily focused on English.
This leaves a gap in our understanding of how the two language model paradigms
perform in multilingual settings across different languages and tasks.

Nielsen (2023) introduced the ScandEval benchmark and evaluated encoder
language models on four different natural language understanding tasks in
Danish, Swedish, Norwegian (Bokmål and Nynorsk), Icelandic and Faroese. In
this paper, we bridge this gap by extending the ScandEval benchmark to
encompass the evaluation of decoder models on multilingual NLU tasks, as well
as expanding the language resources to include German, Dutch and English.

Our main research question is

> Which language model paradigm is better suited for NLU?

We will answer this question with the languages Danish, Swedish, Norwegian,
Icelandic, Faroese, German, Dutch and English as a case study. To concretise
our main question, we will study the following research questions in this
paper:

  1. (Q1)

Can state-of-the-art decoder models achieve significantly better NLU
performance than state-of-the-art finetuned encoder models?

  2. (Q2)

Does the answer to (Q1) depend on the type of NLU task?

  3. (Q3)

Does the answer to (Q1) vary along the language resource spectrum, from low-
to high-resource?

Our main contributions of this paper are the following:

  1. 1.

We extend the ScandEval benchmarking framework with few-shot evaluation of
decoder models and release this extension open-source.

  2. 2.

We extend the languages supported by the ScandEval benchmarking framework by
German, Dutch and English. Together with Danish, Swedish, Norwegian, Icelandic
and Faroese, ScandEval now provides virtually full coverage of the Germanic
language group.

  3. 3.

We evaluate an extensive suite of both encoder and decoder models on NLU tasks
in all of the supported languages and publish these on public leaderboards.

  4. 4.

We give a positive answer to (Q1), showing that decoder models can achieve
significantly better NLU performance than encoder models. This depends on the
language in question however, giving a partially positive answer to (Q3).

  5. 5.

We also show that the decoder models are heavily biased towards the question
answering task (even models that are not instruction tuned), and a UMAP
analysis shows that the performance distribution of decoder models follow a
different “path” than encoder models, from the worst to best performing
models. This gives a positive answer to (Q2).

##  2 Related Work

###  2.1 Comparing Encoder and Decoder Models

There has been a number of studies in recent years comparing encoder models to
decoder models. Zhong et al. (2023) compared GPT-3.5-turbo (January 2023
version) to (finetuned versions of) the base- and large versions of BERT
Devlin et al. (2018) and RoBERTa Liu et al. (2019) on the English GLUE
benchmark Wang et al. (2018). They found that GPT-3.5-turbo was on average on
par with the base-sized encoder models, but fell short of the large-sized
ones. They also note that despite being on par with the base-sized models,
there was a big discrepancy between the models on individual tasks, with
GPT-3.5-turbo for instance being better on the inference tasks while being
worse on the paraphrase tasks. We note however that they only evaluate the
decoder model in a zero-shot setting, and furthermore they only evaluate the
models on 25 samples for each class in the development split, leading to a
potential lack of robustness in their evaluation.

Wang et al. (2023) compared GPT-3.5-turbo (January 2023 version) to a
finetuned version of the base-sized BERT model on 18 English benchmark
datasets related to sentiment analysis. Like Zhong et al. (2023), they found
that the zero-shot performance of GPT-3.5-turbo was on par with the base-sized
BERT model, and that the few-shot performance of GPT-3.5-turbo (with 27 few-
shot examples) was slightly better than BERT, on average. Their test sets
contained, on average, 538 samples, which is a significant improvement over
Zhong et al. (2023). However, the narrow focus on the evaluation tasks as well
as only benchmarking a single encoder and decoder model makes it hard to
generalise the results to other tasks and models.

Kocoń et al. (2023) built a benchmark suite of 25 tasks, where 21 of these
tasks were classification tasks (binary, multi-class and multi-label), 3 being
question answering tasks and the last one being a token classification task.
Two of the classification tasks were in Polish, the rest being English. They
compared the zero-shot and few-shot performance of GPT-3.5-turbo (January 2023
version) to the state-of-the-art encoder performance on each task.
GPT-3.5-turbo was generally found to be worse than state-of-the-art encoder
models. They also evaluated GPT-4 on five of the tasks (inference, question-
answering and emotion datasets), and only found GPT-4 to be marginally better
than GPT-3.5-turbo, still far off the encoder models.

Qiu and Jin (2024) compared GPT-3.5-turbo (January 2023 version) to a
finetuned version of the base-sized BERT model on three manually curated
English multi-class classification datasets with 19, 12 and 7 test samples,
respectively, where they found that the BERT model performed marginally better
than GPT-3.5-turbo in a few-shot setting (and that the zero-shot performance
was significantly worse). The tiny test sets make it hard to generalise the
results, however.

###  2.2 Benchmarks of Generative Language Models

In recent times, several benchmarks of generative language models have been
introduced. The major ones are EleutherAI’s Evaluation Harness Gao et al.
(2023), Hugging Face’s Open LLM Leaderboard hug  which uses the Evaluation
Harness as evaluation engine, and Stanford University’s HELM Bommasani et al.
(2023). These are firstly all English-only benchmarks, making it hard to
generalise the results to other languages, and they only include point
estimates of the dataset performance, and thus do not necessarily provide a
robust assessment of the models. Further, these benchmarks are exclusively for
decoder models, and thus does not provide a way to compare encoders with
decoders.

There has been several language-specific benchmarks introduced as well.
NorBench Samuel et al. (2023) is a collection of Norwegian evaluation datasets
moreso than a dedicated evaluation framework. Further, several datasets in
this collection (NorQuAD, NoReC and NorNE) are already part of ScandEval.
SuperLim Berdičevskis et al. (2023) falls into the same category for Swedish.
DUMB de Vries et al. (2023) is a Dutch benchmarking framework, which is only
focused on encoder models. Danoliterate Holm (2024) is a Danish benchmarking
framework, which has been based largely on the present ScandEval benchmark,
and solely focused on evaluating decoder models. Aside from language modelling
performance, the Danoliterate benchmark also measures calibration, efficiency,
toxicity and fairness. While the development of language-specific benchmarks
is important, it leads to too little overview of trends across benchmarks and
languages and incentivises model development focused on monolingual models
ignoring a potential broader appeal. ScandEval provides a unified and robust
approach for comparison across model categories and Germanic languages.

Benchmarking is not the only way to evaluate language models. A new “arena
approach” has been popularised by the LMSYS Arena Chiang et al. , where users
can submit a prompt and get two responses from two anonymised models at
random, and have to evaluate the responses. The Arena is predominantly used
for English, but also currently supports six other languages. This approach is
a promising way to evaluate language models, but we fear that it is not as
suitable for low-resource languages due to the need of many volunteers to
evaluate the responses.

Lastly, the Scandinavian Embedding Benchmark Enevoldsen et al. (2024b)
complements ScandEval and focuses on evaluating embedding models on a wide
range of tasks in the Scandinavian languages.

##  3 Datasets

In this section we present the datasets that we are evaluating the models on,
all of which are now included in the ScandEval framework. We should note that
these datasets either (a) already existed prior to this publication or (b) are
small extensions of existing datasets. An overview of all the datasets can be
found in Table 1.

###  3.1 Named Entity Recognition

For Norwegian, Swedish and Icelandic we use the NorNE Jørgensen et al. (2020),
SUC 3.0 Gustafson-Capková and Hartmann (2006), MIM-GOLD-NER Ingólfsdóttir et
al. (2020) datasets, which were already included in the ScandEval framework.
For Faroese we replace the previous WikiANN-fo dataset Rahimi et al. (2019)
with the new human annotated FoNE dataset Snæbjarnarson et al. (2023). We also
replace the previous DaNE dataset Hvingelby et al. (2020) with the new DANSK
dataset Enevoldsen et al. (2024a) covering a wider variety of domains. For
German, Dutch and English we add the established NER datasets GermEval
Benikova et al. (2014), the Dutch part of CoNLL-2002 Sang (2002), and the
English CoNLL-2003 Sang and De Meulder (2003).

###  3.2 Sentiment Classification

We re-use the sentiment classification datasets AngryTweets Pauli et al.
(2021), NoReC Velldal et al. (2018) and SweReC Svensson (2017), for Danish,
Norwegian and Swedish, respectively. For German, Dutch and English we add the
existing datasets SB10k Cieliebak et al. (2017), Dutch Social Gupta (2022) and
SST5 Socher et al. (2013). We convert SST5 to the standardised trinary
(negative, neutral, positive) format by converting the “very negative” and
“very positive” labels to “negative” and “positive”, respectively.

###  3.3 Linguistic Acceptability

For linguistic acceptability we re-use the ScaLA datasets for all the
Scandinavian languages, and extend the ScaLA datasets by applying the ScaLA
method from Nielsen (2023) to German, Dutch and English by using the German
McDonald et al. (2013), Dutch van der Beek et al. (2002) and English Zeldes
(2017) dependency treebanks.

###  3.4 Extractive Question Answering

Here we use the ScandiQA dataset Nielsen (2023) for Danish and Swedish, but
replace the manually translated Norwegian ScandiQA dataset with the new
curated NorQuAD dataset Ivanova et al. (2023). We further add the new Natural
Questions in Icelandic dataset Snæbjarnarson and Einarsson (2022) for
Icelandic. For German and English we add the existing extractive question-
answering datasets GermanQuAD Möller et al. (2021) and SQuAD Rajpurkar et al.
(2016), respectively. For Dutch we add the machine translated version of SQuAD
to Dutch Havinga (2023). We acknowledge that a machine translated dataset is
not ideal, and will return to this point when analysing the evaluation
results.

Dataset | Language | #Train | #Val | #Test | #Shots
---|---|---|---|---|---
NER |  |  |  |  |
DANSK Enevoldsen et al. (2024a) | Danish | 1,024 | 256 | 1,024 | 8
SUC 3.0 Gustafson-Capková and Hartmann (2006) | Swedish | 1,024 | 256 | 2,048 | 8
NorNE-nb Jørgensen et al. (2020) | Norwegian Bokmål | 1,024 | 256 | 2,048 | 8
NorNE-nn Jørgensen et al. (2020) | Norwegian Nynorsk | 1,024 | 256 | 2,048 | 8
MIM-GOLD-NER Ingólfsdóttir et al. (2020) | Icelandic | 1,024 | 256 | 2,048 | 8
FoNE Snæbjarnarson et al. (2023) | Faroese | 1,024 | 256 | 2,048 | 8
GermEval Benikova et al. (2014) | German | 1,024 | 256 | 1,024 | 8
CoNLL-nl Sang (2002) | Dutch | 1,024 | 256 | 1,024 | 8
CoNLL-en Sang and De Meulder (2003) | English | 1,024 | 256 | 2,048 | 8
Sentiment Classification |  |  |  |  |
Angry Tweets Pauli et al. (2021) | Danish | 1,024 | 256 | 2,048 | 3
SweReC Svensson (2017) | Swedish | 1,024 | 256 | 2,048 | 3
NoReC Velldal et al. (2018) | Norwegian | 1,024 | 256 | 2,048 | 3
SB10k Cieliebak et al. (2017) | German | 1,024 | 256 | 1,024 | 3
Dutch Social Gupta (2022) | Dutch | 1,024 | 256 | 1,024 | 3
SST5 Socher et al. (2013) | English | 1,024 | 256 | 2,048 | 3
Linguistic Acceptability |  |  |  |  |
ScaLA-da Nielsen (2023) | Danish | 1,024 | 256 | 2,048 | 3
ScaLA-sv Nielsen (2023) | Swedish | 1,024 | 256 | 2,048 | 3
ScaLA-nb Nielsen (2023) | Norwegian Bokmål | 1,024 | 256 | 2,048 | 3
ScaLA-nn Nielsen (2023) | Norwegian Nynorsk | 1,024 | 256 | 2,048 | 3
ScaLA-is Nielsen (2023) | Icelandic | 1,024 | 256 | 2,048 | 3
ScaLA-fo Nielsen (2023) | Faroese | 1,024 | 256 | 1,024 | 3
ScaLA-de Nielsen (2023) | German | 1,024 | 256 | 2,048 | 3
ScaLA-nl Nielsen (2023) | Dutch | 1,024 | 256 | 2,048 | 3
ScaLA-en Nielsen (2023) | English | 1,024 | 256 | 2,048 | 3
Question Answering |  |  |  |  |
ScandiQA-da Nielsen (2023) | Danish | 1,024 | 256 | 2,048 | 4
ScandiQA-sv Nielsen (2023) | Swedish | 1,024 | 256 | 2,048 | 4
NorQuAD Ivanova et al. (2023) | Norwegian Bokmål | 1,024 | 256 | 2,048 | 2
NQiI Snæbjarnarson and Einarsson (2022) | Icelandic | 1,024 | 256 | 1,024 | 4
GermanQuAD Möller et al. (2021) | German | 1,024 | 256 | 2,048 | 4
SQuAD-nl Havinga (2023) | Dutch | 1,024 | 256 | 2,048 | 4
SQuAD Rajpurkar et al. (2016) | English | 1,024 | 256 | 2,048 | 4
Table 1:  All the datasets used in the NLU evaluation. Note that these have
been re-sized and do not represent the sizes of the original dataset.

##  4 Methodology

###  4.1 Formulating NLU Tasks as Generative Tasks

In this section we describe how we rephrase the NLU tasks as text-to-text
tasks, which makes it possible to evaluate generative models on the tasks. We
formulate all the tasks as few-shot tasks, generally formatted as follows:

> [prefix prompt]
>
>
> [document prefix]: [document]
>
> [label prefix]: [label]
>
>
> [document prefix]: [document]
>
> [label prefix]: [label]
>
>
> (…)
>
>
> [document prefix]: [document]
>
> [label prefix]:

We found that the separation of the few-shot examples with double newlines
makes it easier to know when to stop the generation - for the same reason, we
ensure that there are no double newlines in any of the documents. See the
prompts used for the English datasets in Table 2; a full table of the prompts
used for all the tasks in all the languages can be found in the appendix.

For the sentiment classification task, we simply have the models generate
translations of the three labels (positive, negative and neutral). For the
linguistic acceptability task, also a text classification task, we use the
translations of “yes” and “no” as the two labels, corresponding to whether the
document is grammatically correct or not. For the extractive question
answering task, we have the model output the answer directly. For this task we
found that changing the label prefix from “Answer” to “Answer in max 3 words”
resulted in a drastic improvement, due to many of the answers of instruction
tuned models starting with unnecessary text akin to “The answer is”. Lastly,
for the named entity recognition task, we require the output to be a JSON
dictionary ISO/IEC 21778: (2017), with keys being the translated named entity
tags, and values being lists of named entities of that category. To ensure
that we are not biasing the evaluation toward models knowing the JSON format,
we employ structured generation using the outlines package Louf (2023), which
modifies the logits outputted by the model to ensure that the output is always
a valid JSON dictionary in the aforementioned format.

Task | Prefix Prompt | Example Prompt
---|---|---
Named entity recognition | Below are sentences and JSON dictionaries with the named entities that occur in the given sentence. | Sentence: [text]
Named entities: [label]
Sentiment classification | The following are tweets are their sentiment, which can be ’positive’, ’neutral’ or ’negative’. | Tweet: [text]
Sentiment: [label]
Linguistic acceptability | The following are sentences and whether they are grammatically correct. | Sentence: [text]
Grammatically correct: [label]
Question answering | The following are texts with accompanying questions and answers. | Text: [text]
Question: [question]
Answer in max 3 words: [label]
Table 2: The English prompt templates used for the datasets. See all the
prompt templates in the appendix.

###  4.2 Evaluation Methodology

We keep the evaluation methodology for the generative models to be as close to
the methodology for encoder models in Nielsen (2023). We think of the few-shot
examples as analogous to training examples for encoder models. Indeed, as von
Oswald et al. (2023) shows, this assumption is theoretically grounded. We thus
evaluate the models 10 times, where on each iteration we sample few-shot
examples at random from the training split, and we evaluate the model on a
bootstrapped version of the test split. As with the encoder models, this
allows us to take into account more noise in evaluation process, resulting in
more robust evaluation scores.

###  4.3 Score Aggregation Method

From the raw scores of the 10 evaluations per dataset, we need to aggregate
the model scores into a single score. We want an aggregation method that
satisfies the following criteria:

  1. 1.

Task Fairness: Each task should be weighted equally.

  2. 2.

Comparison: If we evaluate models in multiple languages, then it should be
possible to meaningfully compare the language scores of these models with each
other.

  3. 3.

Robustness: If two models do not have a significantly different score on a
dataset, then the aggregated score should reflect this.

  4. 4.

Magnitude Preservation: The magnitude of the difference between the dataset
score of two models should be reflected in the aggregated score.

  5. 5.

Minimal Change: Adding a new model should minimally affect the aggregated
scores of the other models.

Before we introduce our chosen aggregation method, we will briefly discuss
some common aggregation methods and how they do not satisfy the criteria.

The mean score is the most common aggregation method, which would simply be
the mean of the 10 scores for each dataset, and then the mean of the dataset
scores for each task. This method does not satisfy the Task Fairness
criterion, as it does not take into account that metrics have different ranges
and variances. The Comparison criterion is also not satisfied, as datasets
vary from language to language, with some datasets being more difficult than
others. It does, however, satisfy the Robustness, Magnitude Preservation and
Minimal Change criteria.

The mean rank is another common aggregation method, where we compute the rank
of each model on each dataset, and then take the mean of the ranks. This
method satisfies the Task Fairness criterion, as it re-casts the scores into a
common comparable framework, which therefore weights each task equally. For
the same reason, it also satisfies the Comparison criterion (it is important
here that we evaluate all the models on all the languages for this to be
satisfied). It does not satisfy the Robustness and Magnitude Preservation
criteria, by definition of rank. It partially satisfies the Minimal Change
criterion, since it only affects the scores of the models which are worse than
the new model.

We thus see that the mean score and mean rank methods satisfy a disjoint set
of the criteria, but that they together satisfy all the criteria. Based on
this observation, we introduce the mean rank score method, defined as follows.
For each dataset, we start by sorting the models by their mean score on the
dataset. As with a rank, we assign the best model with rank score 1. For the
next best model, we conduct a one-tailed Welch’s t-test to see if the next
best model is significantly worse than the first model ($p<0.05$). If so, we
compute the absolute difference between the mean score of the two models, and
divide that by the standard deviation of all the mean scores of the models on
the dataset.

We then add this to the rank score of the first model. We continue this
process for all the models to get the rank scores for the dataset, and to
compute the overall score for the model, we take the mean of the rank scores
for the datasets. An overview of this aggregation method can be found in
Algorithm 1.

We note that the mean rank score has an intuitive interpretation: it is the
average number of standard deviations from the best scoring model (+1).

This metric satisfies Task Fairness since we normalise all the scores by
dividing by the standard deviation of the dataset scores. The Robustness
criterion is satisfied due to our use of a one-tailed Welch’s t-test. The
Magnitude Preservation criterion is also satisfied, as the magnitude of the
difference between the dataset score of two models is reflected in the rank
score. It also satisfies Comparison, as we compare the models on a common
scale (same argument as the mean rank method). Finally, the Minimal Change
criterion is partially satisfied, as adding new models only minimally changes
the score of existing models. Concretely, adding new scores will affect the
standard deviation normalising factor (this effect tends to zero as the number
of models grows, however), and if the model beats all the other models then
all the scores will be affected, due to the relative nature of the metric.

Algorithm 1 The mean rank score algorithm.

1:Input: $M$ models, $D$ datasets, $S_{m,d,i}$ the $i$’th raw score of model
$m$ on dataset $d$

2:Output: The mean rank score of each model

3:$R\leftarrow\textrm{matrix of size }M\times D$

4:for $\hat{d}\in D$ do

5:
$\textrm{sortedModels}\leftarrow\textrm{argsort}_{m}\nicefrac{{1}}{{10}}\cdot\Sigma_{i}S_{m,\hat{d},i}$

6: for $\hat{m}\in\textrm{sortedModels}$ do

7: if $\hat{m}$ is the first model then

8: $\rho\leftarrow 1$

9: $\textrm{previousScores}\leftarrow S_{\hat{m},\hat{d}}$

10: else

11: $\textrm{pValue}\leftarrow\textrm{one-tailed Welch's
t-test}(\textrm{previousScores},S_{\hat{m},\hat{d}})$

12: if $\textrm{pValue}<0.05$ then

13:
$\Delta\leftarrow\textrm{mean}(S_{\hat{m},\hat{d}})-\textrm{mean}(\textrm{previousScores})$

14:
$\rho\leftarrow\rho+\nicefrac{{\Delta}}{{\textrm{standardDeviation}_{m}S_{m,\hat{d}}}}$

15: $\textrm{previousScores}\leftarrow S_{\hat{m},\hat{d}}$

16: end if

17: end if

18: $R_{\hat{m},\hat{d}}\leftarrow\rho$

19: end for

20:end for

21:$\textrm{output}\leftarrow\nicefrac{{1}}{{D}}\cdot\Sigma_{d}R_{m,d}$

##  5 Analysis

###  5.1 Comparative Performance Analysis on High- and Low-resource Languages

Excerpts of the English, Danish and Icelandic leaderboards can be found in
Table 3, Table 4 and Table 5, respectively. We found that these three
represent three main categories of languages with respect to the open-closed
source divide. Similar excerpts for the remaining languages (Swedish,
Norwegian, Faroese, German and Dutch) can be found in the appendix. The full
leaderboards for all the languages can be found at https://scandeval.com.

From the English results we see that the state-of-the-art decoder model
GPT-4-0613 Achiam et al. (2023) is still outperformed by the DeBERTa-v3-large
and DeBERTa-v3-base models He et al. (2020) as well as the ELECTRA-base model
Clark et al. (2020). Here GPT-4-0613 is, on average, 0.44 standard deviations
worse than the best model. The same pattern is seen for Norwegian, Dutch,
German and Faroese; see the appendix for the corresponding leaderboard
excerpts.

In contrast, on the Danish leaderboard, the top-3 models are all decoder
models, with GPT-4-0613 and GPT-4-1106-preview OpenAI (2023b) in the lead,
followed by the closed-source DanskGPT-Chat-Llama3-70B model from
Syv.AI111https://www.syv.ai/, being a continuation of the Llama-3-70B model
AI@Meta (2024). The GPT-4-0613 model is, on average, 0.24 standard deviations
from the best model. Similar results were found with Swedish; see the appendix
for the corresponding leaderboard excerpt.

Lastly, for Icelandic, we see that the encoders and decoders are tied in
performance, with the mDeBERTa-v3-base model and the GPT-4-1106-preview model
being the top models. The GPT-4-1106-preview model is, on average, 0.24
standard deviations from the best model. We note that Icelandic is the only
language where the switch from GPT-4 (gpt-4-0613) to GPT-4-turbo
(gpt-4-1106-preview) resulted in a significant increase in performance. We
speculate that this is due to the collaboration between OpenAI and Iceland
OpenAI (2023a).

We can thus give an affirmative answer to research question (Q1), showing that
decoder models can achieve significantly better NLU performance than encoder
models, albeit with an order of magnitude more model parameters. For (Q3), we
see that this varies between languages, but without being correlated to the
language resource spectrum.

Model ID | Decoder | Score ($\downarrow$)
---|---|---
microsoft/deberta-v3-large | ✗ | 1.09
microsoft/deberta-v3-base | ✗ | 1.29
google/electra-base-discriminator | ✗ | 1.39
gpt-4-0613 | ✓ | 1.44
FacebookAI/roberta-large | ✗ | 1.46
FacebookAI/roberta-base | ✗ | 1.51
microsoft/mdeberta-v3-base | ✗ | 1.53
gpt-4-1106-preview | ✓ | 1.54
gpt-4o-2024-05-13 | ✓ | 1.64
AI-Sweden-Models/roberta-large-1160k | ✗ | 1.64
gpt-3.5-turbo-0613 | ✓ | 1.78
mistralai/Mistral-7B-v0.1 | ✓ | 1.91
Table 3: Excerpt of the English ScandEval leaderboard. Model ID | Decoder | Score ($\downarrow$)
---|---|---
gpt-4-0613 | ✓ | 1.24
gpt-4-1106-preview | ✓ | 1.25
syvai/danskgpt-chat-llama3-70b | ✓ | 1.29
AI-Sweden-Models/roberta-large-1160k | ✗ | 1.39
danish-foundation-models/encoder-large-v1 | ✗ | 1.40
meta-llama/Meta-Llama-3-70B | ✓ | 1.40
AI-Sweden-Models/Llama-3-8B-instruct | ✓ | 1.44
gpt-4o-2024-05-13 | ✓ | 1.46
ltg/norbert3-large | ✗ | 1.50
NbAiLab/nb-bert-large | ✗ | 1.54
vesteinn/DanskBERT | ✗ | 1.56
google/rembert | ✗ | 1.61
intfloat/multilingual-e5-large | ✗ | 1.62
gpt-3.5-turbo-0613 | ✓ | 1.68
FacebookAI/xlm-roberta-large | ✗ | 1.71
Table 4: Excerpt of the Danish ScandEval leaderboard. Model ID | Decoder | Score ($\downarrow$)
---|---|---
microsoft/mdeberta-v3-base | ✗ | 1.33
gpt-4-1106-preview | ✓ | 1.34
gpt-4o-2024-05-13 | ✓ | 1.43
vesteinn/ScandiBERT-no-faroese | ✗ | 1.48
google/rembert | ✗ | 1.57
vesteinn/XLMR-ENIS | ✗ | 1.59
gpt-4-0613 | ✓ | 1.79
mideind/IceBERT-large | ✗ | 1.85
vesteinn/FoBERT | ✗ | 1.87
meta-llama/Meta-Llama-3-70B | ✓ | 2.03
FacebookAI/xlm-roberta-large | ✗ | 2.34
gpt-3.5-turbo-0613 | ✓ | 2.51
mistralai/Mistral-7B-v0.1 | ✓ | 2.96
Table 5: Excerpt of the Icelandic ScandEval leaderboard.

###  5.2 Task Analysis

In this section we analyse our research question (Q2), asking whether the NLU
performance results from the previous section is dependent on the type of NLU
task.

Firstly, we analyse whether the score distribution across the four NLU tasks
is different for the encoder and decoder models. This is done by applying a
UMAP McInnes et al. (2018) to the results of a given leaderboard, stratified
by both the mean rank score and whether the model is generative. UMAP plots
for the English, Danish, Swedish, Norwegian, German and Dutch leaderboards can
be found in Figure 1.

We see that the worst and best performing models have similar distributions,
irrespective of whether they are generative or not. However, we also note that
the rest of encoder and decoder models follow different “paths” in the UMAP
space, leading to our hypothesis that they learn different things during their
development.

Figure 1: UMAP plots of the models on the ScandEval leaderboards.

In Figure 2 we show the correlation between a model being generative and its
performance on the four NLU tasks. We see that being generative is a strong
predictor for good question answering performance, as well as poor named
entity recognition and linguistic acceptability performance. The correlation
is weaker for sentiment classification and varies across languages. We also
see that these findings seem to generalise across languages, both high- and
low-resource. The large question answering performance persists for non-
instruction-tuned decoder models (see the leaderboards at
https://scandeval.com), showing a likely side-effect of the pre-training
algorithm or the architecture of decoder models making them better at this
task.

Figure 2: The correlation between a model being generative and its performance
on the four NLU tasks.

##  6 Discussion

Having a good mean rank score is not the only thing that matters when choosing
a model for a given task. Model size, inference speed and whether the model
has publicly available weights are all important factors to consider. For this
reason, we also include these metadata in the leaderboard, and we encourage
the community to consider these factors when choosing a model for a given
task.

##  7 Conclusion

We have extended the ScandEval benchmark to include the evaluation of decoder
models, as well as including three new languages: German, Dutch and English.
From the analysis of the corresponding results we found that decoder models
can achieve significantly better NLU performance than encoder models, but that
this varies between languages. We have also shown that being generative is
strongly correlated with both good question answering performance and poor
performance for named entity recognition and linguistic acceptability. Our
analysis showed that the “path” from the worst to the best-performing models
in the UMAP space is different for encoder and decoder models, indicating that
they learn different things during their development.

## Ethics Statement

We have taken care to ensure that the evaluation is fair and unbiased, both
through our selection of the datasets in the benchmark as well as through our
choice of aggregation method of the scores.

## Acknowledgements

This work has received funding by the European Union’s Horizon 2023 Research
and Innovation Actions, as part of the Artificial Intelligence and Robotics
programme, for the project “TrustLLM” (grant agreement number 101135671).
Furthermore, this work reflects only the authors’ view and the European
Research Executive Agency (REA) is not responsible for any use that may be
made of the information it contains.

## References

  * (1) Open LLM Leaderboard - a Hugging Face Space by open-llm-leaderboard — huggingface.co.  https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard.  [Accessed 12-06-2024].
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Benikova et al. (2014) Darina Benikova, Chris Biemann, and Marc Reznicek. 2014.  Nosta-d named entity annotation for german: Guidelines and dataset.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 2524–2531.
  * Berdičevskis et al. (2023) Aleksandrs Berdičevskis, Gerlof Bouma, Robin Kurtz, Felix Morger, Joey Öhman, Yvonne Adesam, Lars Borin, Dana Dannélls, Markus Forsberg, Tim Isbister, et al. 2023.  Superlim: A swedish language understanding evaluation benchmark.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 8137–8153.
  * Bommasani et al. (2023) Rishi Bommasani, Percy Liang, and Tony Lee. 2023.  Holistic evaluation of language models.  _Annals of the New York Academy of Sciences_ , 1525(1):140–146.
  * Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and others (2020) Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and others. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * (8) Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al.  Chatbot arena: An open platform for evaluating llms by human preference.  In _Forty-first International Conference on Machine Learning_.
  * Cieliebak et al. (2017) Mark Cieliebak, Jan Milan Deriu, Dominic Egger, and Fatih Uzdilli. 2017.  A twitter corpus and benchmark resources for german sentiment analysis.  In _Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media_ , pages 45–51.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * de Vries et al. (2023) Wietse de Vries, Martijn Wieling, and Malvina Nissim. 2023.  Dumb: A benchmark for smart evaluation of dutch models.  _arXiv preprint arXiv:2305.13026_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Enevoldsen et al. (2024a) Kenneth Enevoldsen, Emil Trenckner Jessen, and Rebekah Baglini. 2024a.  Dansk and dacy 2.6. 0: Domain generalization of danish named entity recognition.  _arXiv preprint arXiv:2402.18209_.
  * Enevoldsen et al. (2024b) Kenneth Enevoldsen, Márton Kardos, Niklas Muennighoff, and Kristoffer Laigaard Nielbo. 2024b.  The scandinavian embedding benchmarks: Comprehensive assessment of multilingual and monolingual text embedding.  _arXiv preprint arXiv:2406.02396_.
  * Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023.  A framework for few-shot language model evaluation.
  * Gupta (2022) Aakash Gupta. 2022.  dutchsocial · Datasets at Hugging Face — huggingface.co.  https://huggingface.co/datasets/dutch_social.  [revision: 8b7bc6230ebd78f04aa3661acb912f4567f21c76].
  * Gustafson-Capková and Hartmann (2006) Sofia Gustafson-Capková and Britt Hartmann. 2006.  Manual of the Stockholm Umeå corpus version 2.0.  _Stockholm University_.
  * Havinga (2023) Yeb Havinga. 2023.  squadv2dutch · Datasets at Hugging Face — huggingface.co.  [revision: af494fe1b62762178d37c0b71b4a7160f0534f1a].
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_.
  * Holm (2024) Søren Vejlgaard Holm. 2024.  Are gllms danoliterate? benchmarking generative nlp in danish.
  * Hvingelby et al. (2020) Rasmus Hvingelby, Amalie Brogaard Pauli, Maria Barrett, Christina Rosted, Lasse Malm Lidegaard, and Anders Søgaard. 2020.  Dane: A named entity resource for danish.  In _Proceedings of the 12th language resources and evaluation conference_ , pages 4597–4604.
  * Ingólfsdóttir et al. (2020) Svanhvít L Ingólfsdóttir, Ásmundur A Gudjónsson, and Hrafn Loftsson. 2020.  Named Entity Recognition for Icelandic: Annotated Corpus and Models.  In _International Conference on Statistical Language and Speech Processing_ , pages 46–57. Springer.
  * ISO/IEC 21778: (2017) ISO/IEC 21778:2017. 2017.  The JSON data interchange syntax.  Standard, International Organization for Standardization, Geneva, CH.
  * Ivanova et al. (2023) Sardana Ivanova, Fredrik Andreassen, Matias Jentoft, Sondre Wold, and Lilja Øvrelid. 2023.  Norquad: Norwegian question answering dataset.  In _Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 159–168.
  * Jørgensen et al. (2020) Fredrik Jørgensen, Tobias Aasmoe, Anne-Stine Ruud Husevåg, Lilja Øvrelid, and Erik Velldal. 2020.  NorNE: Annotating named entities for Norwegian.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4547–4556, Marseille, France. European Language Resources Association.
  * Kocoń et al. (2023) Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023.  Chatgpt: Jack of all trades, master of none.  _Information Fusion_ , 99:101861.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Louf (2023) Rémi Louf. 2023.  Outlines.  https://github.com/outlines-dev/outlines.
  * McDonald et al. (2013) Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, et al. 2013.  Universal dependency annotation for multilingual parsing.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 92–97.
  * McInnes et al. (2018) Leland McInnes, John Healy, and James Melville. 2018.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_.
  * Möller et al. (2021) Timo Möller, Julian Risch, and Malte Pietsch. 2021.  Germanquad and germandpr: Improving non-english question answering and passage retrieval.  In _Proceedings of the 3rd Workshop on Machine Reading for Question Answering_ , pages 42–50.
  * Nielsen (2023) Dan Saattrup Nielsen. 2023.  ScandEval: A Benchmark for Scandinavian Natural Language Processing.  In _The 24rd Nordic Conference on Computational Linguistics_.
  * OpenAI (2023a) OpenAI. 2023a.  Government of Iceland: How Iceland is using GPT-4 to preserve its language.  https://openai.com/index/government-of-iceland.  [Accessed 12-06-2024].
  * OpenAI (2023b) OpenAI. 2023b.  New models and developer products announced at DevDay.  https://openai.com/blog/new-models-and-developer-products-announced-at-devday.  [Accessed 12-06-2024].
  * Pauli et al. (2021) Amalie Brogaard Pauli, Maria Barrett, Ophélie Lacroix, and Rasmus Hvingelby. 2021.  Danlp: An open-source toolkit for danish natural language processing.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 460–466.
  * Qiu and Jin (2024) Yunjian Qiu and Yan Jin. 2024.  Chatgpt and finetuned bert: A comparative study for developing intelligent design support systems.  _Intelligent Systems with Applications_ , 21:200308.
  * Rahimi et al. (2019) Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019.  Massively multilingual transfer for ner.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 151–164.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392.
  * Samuel et al. (2023) David Samuel, Andrey Kutuzov, Samia Touileb, Erik Velldal, Lilja Øvrelid, Egil Rønningstad, Elina Sigdel, and Anna Palatkina. 2023.  Norbench–a benchmark for norwegian language models.  _arXiv preprint arXiv:2305.03880_.
  * Sang (2002) Erik Tjong Kim Sang. 2002.  Introduction to the conll-2002 shared task: Language-independent named entity recognition.  In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
  * Sang and De Meulder (2003) Erik Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Snæbjarnarson and Einarsson (2022) Vésteinn Snæbjarnarson and Hafsteinn Einarsson. 2022.  Natural questions in icelandic.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4488–4496.
  * Snæbjarnarson et al. (2023) Vésteinn Snæbjarnarson, Annika Simonsen, Goran Glavaš, and Ivan Vulić. 2023.  Transfer to a low-resource language via close relatives: The case study on faroese.  In _Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 728–737.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Svensson (2017) Kristoffer Svensson. 2017.  Sentiment Analysis With Convolutional Neural Networks: Classifying sentiment in Swedish reviews.  Bachelor’s thesis.
  * van der Beek et al. (2002) L van der Beek, G Bouma, R Malouf, and G van Noord. 2002.  The alpino dependency treebank.  In _12th Meeting on Computational Linguistics in the Netherlands (CLIN)_ , pages 8–22. Rodopi.
  * Velldal et al. (2018) Erik Velldal, Lilja Øvrelid, Eivind Alexander Bergem, Cathrine Stadsnes, Samia Touileb, and Fredrik Jørgensen. 2018.  NoReC: The Norwegian review corpus.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * von Oswald et al. (2023) Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, Razvan Pascanu, et al. 2023.  Uncovering mesa-optimization algorithms in transformers.  _arXiv preprint arXiv:2309.05858_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2023) Zengzhi Wang, Qiming Xie, Yi Feng, Zixiang Ding, Zinong Yang, and Rui Xia. 2023\.  Is chatgpt a good sentiment analyzer? a preliminary study.  _arXiv preprint arXiv:2304.04339_.
  * Zeldes (2017) Amir Zeldes. 2017.  The gum corpus: creating multilayer resources in the classroom.  _Language Resources and Evaluation_ , 51(3):581–612.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.
  * Zhong et al. (2023) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023.  Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert.  _arXiv preprint arXiv:2302.10198_.

##  Appendix A Appendix

Model ID | Decoder | Score ($\downarrow$)
---|---|---
AI-Sweden-Models/Llama-3-8B-instruct | ✓ | 1.07
gpt-4-0613 | ✓ | 1.28
four-two-labs/lynx-micro | ✓ | 1.29
AI-Sweden-Models/roberta-large-1350k | ✗ | 1.33
meta-llama/Meta-Llama-3-70B | ✓ | 1.34
gpt-4o-2024-05-13 | ✓ | 1.35
gpt-4-1106-preview | ✓ | 1.37
KBLab/megatron-bert-large-swedish-cased-165k | ✗ | 1.42
AI-Sweden-Models/bert-large-nordic-pile-1M-steps | ✗ | 1.48
KB/bert-base-swedish-cased | ✗ | 1.48
intfloat/multilingual-e5-large | ✗ | 1.49
ltg/norbert3-large | ✗ | 1.49
gpt-3.5-turbo-0613 | ✓ | 1.57
upstage/SOLAR-10.7B-v1.0 | ✗ | 1.96
timpal0l/Mistral-7B-v0.1-flashback-v2 | ✓ | 2.02
mistralai/Mistral-7B-v0.1 | ✓ | 2.17
Table 6: Excerpt of the Swedish ScandEval leaderboard. Model ID | Decoder | Score ($\downarrow$)
---|---|---
ltg/norbert3-large | ✗ | 1.12
NbAiLab/nb-bert-large | ✗ | 1.19
ltg/norbert3-base | ✗ | 1.32
AI-Sweden-Models/roberta-large-1350k | ✗ | 1.38
gpt-4-0613 | ✓ | 1.49
danish-foundation-models/encoder-large-v1 | ✗ | 1.54
google/rembert | ✗ | 1.57
meta-llama/Meta-Llama-3-70B | ✓ | 1.59
gpt-4-1106-preview | ✓ | 1.62
gpt-4o-2024-05-13 | ✓ | 1.67
microsoft/mdeberta-v3-base | ✗ | 1.67
gpt-3.5-turbo-0613 | ✓ | 2.03
upstage/SOLAR-10.7B-v1.0 | ✗ | 2.38
meta-llama/Meta-Llama-3-8B | ✓ | 2.45
mistralai/Mistral-7B-v0.1 | ✓ | 2.79
AI-Sweden-Models/Llama-3-8B-instruct | ✓ | 2.82
Table 7: Excerpt of the Norwegian ScandEval leaderboard. Model ID | Decoder | Score ($\downarrow$)
---|---|---
vesteinn/FoBERT | ✗ | 1.00
microsoft/mdeberta-v3-base | ✗ | 2.01
vesteinn/ScandiBERT-no-faroese | ✗ | 2.66
gpt-4-1106-preview | ✓ | 2.69
google/rembert | ✗ | 3.35
ltg/norbert3-base | ✗ | 3.36
gpt-4-0613 | ✓ | 3.46
mideind/IceBERT-large | ✗ | 3.96
meta-llama/Meta-Llama-3-70B | ✓ | 4.16
gpt-3.5-turbo-0613 | ✓ | 4.23
gpt-4o-2024-05-13 | ✓ | 4.69
mistralai/Mistral-7B-v0.1 | ✓ | 4.84
Table 8: Excerpt of the Faroese ScandEval leaderboard. Model ID | Decoder | Score ($\downarrow$)
---|---|---
deepset/gbert-large | ✗ | 1.17
FacebookAI/xlm-roberta-large | ✗ | 1.34
google/rembert | ✗ | 1.34
gpt-4-0613 | ✓ | 1.49
intfloat/multilingual-e5-large | ✗ | 1.49
meta-llama/Meta-Llama-3-70B | ✓ | 1.57
gwlms/deberta-base-dewiki-v1 | ✗ | 1.61
gpt-4-1106-preview | ✓ | 1.65
upstage/SOLAR-10.7B-v1.0 | ✗ | 1.67
microsoft/mdeberta-v3-base | ✗ | 1.74
gpt-4o-2024-05-13 | ✓ | 1.82
deepset/gbert-base | ✗ | 1.83
gpt-3.5-turbo-0613 | ✓ | 1.95
occiglot/occiglot-7b-de-en-instruct | ✓ | 2.02
meta-llama/Meta-Llama-3-8B | ✓ | 2.02
mistralai/Mistral-7B-v0.1 | ✓ | 2.15
Table 9: Excerpt of the German ScandEval leaderboard. Model ID | Decoder | Score ($\downarrow$)
---|---|---
intfloat/multilingual-e5-large | ✗ | 1.42
gpt-4-0613 | ✓ | 1.69
DTAI-KULeuven/robbert-2022-dutch-base | ✗ | 1.78
pdelobelle/robbert-v2-dutch-base | ✗ | 1.81
meta-llama/Meta-Llama-3-70B | ✓ | 1.84
gpt-4-1106-preview | ✓ | 1.98
intfloat/multilingual-e5-base | ✗ | 2.00
FacebookAI/xlm-roberta-large | ✗ | 2.10
gpt-4o-2024-05-13 | ✓ | 2.14
microsoft/mdeberta-v3-base | ✗ | 2.25
gpt-3.5-turbo-0613 | ✓ | 2.27
google/rembert | ✗ | 2.29
upstage/SOLAR-10.7B-v1.0 | ✗ | 2.33
meta-llama/Meta-Llama-3-8B | ✓ | 2.54
mistralai/Mistral-7B-v0.1 | ✓ | 2.77
Table 10: Excerpt of the Dutch ScandEval leaderboard. Dataset | Prefix Prompt | Example Prompt
---|---|---
DANSK | Følgende er sætninger og JSON-ordbøger med de navngivne enheder, som forekommer i den givne sætning. | Sætning: [text]
Navngivne enheder: [label]
SUC 3.0 | Följande är meningar och JSON-ordböcker med de namngivna enheter som förekommer i den givna meningen. | Mening: [text]
Namngivna entiteter: [label]
NorNE-nb
NorNE-nn | Følgende er fraser og JSON-ordbøker med de navngitte enhetene som forekommer i den gitte frasen. | Frase: [text]
Navngitte enheter: [label]
MIM-GOLD-NER | Eftirfarandi eru setningar ásamt JSON lyklum með nefndum einingum sem koma fyrir í setningunum. | Setning: [text]
Nefndar einingar: [label]
FoNE | Her eru nakrir setningar og nakrar JSON orðabøkur við nevndar eindir, sum eru í setningunum. | Setningur: [text]
Nevndar eindir: [label]
GermEval | Es folgen Sätze und JSON-Wörterbücher mit den benannten Entitäten, die in der angegebenen Phrase vorkommen. | Satz: [text]
Benannte Entitäten: [label]
CoNLL-nl | Hieronder staan zinnen en JSON woordenboeken met de genoemde entiteiten die voorkomen in de gegeven zin. | Zin: [text]
Genoemde entiteiten: [label]
CoNLL-en | Below are sentences and JSON dictionaries with the named entities that occur in the given sentence. | Sentence: [text]
Named entities: [label]
Table 11: The prompt templates used for the NER datasets. Dataset | Prefix Prompt | Example Prompt
---|---|---
Angry Tweets | Følgende er tweets og deres sentiment, som kan være ’positiv’, ’neutral’ eller ’negativ’. | Tweet: [text]
Sentiment: [label]
SweReC | Följande är recensioner och deras sentiment, som kan vara ’positiv’, ’neutral’ eller ’negativ’. | Recension: [text]
Sentiment: [label]
NoReC | Følgende er anmeldelser og deres sentiment, som kan være ’positiv’, ’nøytral’ eller ’negativ’. | Anmeldelse: [text]
Sentiment: [label]
SB10k | Im Folgenden sind Tweets und ihre Stimmung aufgeführt, die ’positiv’, ’neutral’ oder ’negativ’ sein kann. | Tweet: [text]
Stimmungslage: [label]
Dutch Social | Hieronder staan tweets en hun sentiment, dat ’positief’, ’neutraal’ of ’negatief’ kan zijn. | Tweet: [text]
Sentiment: [label]
SST5 | The following are tweets are their sentiment, which can be ’positive’, ’neutral’ or ’negative’. | Tweet: [text]
Sentiment: [label]
Table 12: The prompt templates used for the sentiment classification datasets. Dataset | Prefix Prompt | Example Prompt
---|---|---
ScaLA-da | Følgende er sætninger og om de er grammatisk korrekte. | Sætning: [text]
Grammatisk korrekt: [label]
ScaLA-sv | Följande är meningar och huruvida de är grammatiskt korrekta. | Mening: [text]
Grammatisk korrekt: [label]
ScaLA-nb
ScaLA-nn | Følgende er setninger og hvorvidt de er grammatisk korrekte. | Setning: [text]
Grammatisk korrekt: [label]
ScaLA-is | Eftirfarandi eru setningar og hvort þær eru málfræðilega réttar. | Setning: [text]
Málfræðilega rétt: [label]
ScaLA-fo | Hetta eru nakrir setningar og um teir eru mállæruliga rættir. | Setningur: [text]
Mállæruliga rættur: [label]
ScaLA-de | Die folgenden Sätze und ob sie grammatikalisch korrekt sind. | Satz: [text]
Grammatikalisch richtig: [label]
ScaLA-nl | Hieronder staan zinnen en of ze grammaticaal correct zijn. | Zin: [text]
Grammaticaal correct: [label]
ScaLA-en | The following are sentences and whether they are grammatically correct. | Sentence: [text]
Grammatically correct: [label]
Table 13: The prompt templates used for the linguistic acceptability datasets. Dataset | Prefix Prompt | Example Prompt
---|---|---
ScandiQA-da | Følgende er tekster med tilhørende spørgsmål og svar. | Tekst: [text]
Spørgsmål: [question]
Svar med maks. 3 ord: [label]
ScandiQA-sv | Her følger tekster med tilhørende spørsmål og svar. | Tekst: [text]
Spørsmål: [question]
Svar på maks 3 ord: [label]
NorQuAD | Nedan följer texter med tillhörande frågor och svar. | Text: [text]
Fråga: [question]
Svar på max 3 ord: [label]
NQiI | Eftirfarandi eru textar með tilheyrandi spurningum og svörum. | Texti: [text]
Spurning: [question]
Svaraðu með að hámarki 3 orðum: [label]
GermanQuAD | Im Folgenden finden Sie Texte mit den dazugehörigen Fragen und Antworten. | Text: [text]
Fragen: [question]
Fragen Antwort in maximal 3 Wörtern: [label]
SQuAD-nl | Hieronder volgen teksten met bijbehorende vragen en antwoorden. | Tekst: [text]
Vraag: [question]
Antwoord in max 3 woorden: [label]
SQuAD | The following are texts with accompanying questions and answers. | Text: [text]
Question: [question]
Answer in max 3 words: [label]
Table 14: The prompt templates used for the question answering datasets. Language | PER | LOC | ORG | Negative | Neutral | Positive | Correct | Incorrect
---|---|---|---|---|---|---|---|---
Danish | Person | Sted | Organisation | Negativ | Neutral | Positiv | Ja | Nej
Swedish | Person | Plats | Organisation | Negativ | Neutral | Positiv | Ja | Nej
Norwegian | Person | Sted | Organisasjon | Negativ | Nøytral | Positiv | Ja | Nei
Icelandic | Einstaklingur | Staðsetning | Stofnun |  |  |  | Já | Nei
Faroese | Persónur | Staður | Felagsskapur |  |  |  | Ja | Nei
German | Person | Ort | Organisation | Positiv | Neutral | Negativ | Ja | Nein
Dutch | Persoon | Locatie | Organisatie | Positief | Neutraal | Negatief | Ja | Nee
English | Person | Location | Organization | Negative | Neutral | Positive | Yes | No
Table 15: The label conversions for the datasets.