# An exact solution of vacuum field equation in Finsler spacetime

Xin Li<EMAIL_ADDRESS>Zhe Chang<EMAIL_ADDRESS>Institute of High Energy
Physics,
Theoretical Physics Center for Science Facilities,
Chinese Academy of Sciences, 100049 Beijing, China

###### Abstract

The vacuum field equation in Finsler spacetime is equivalent to vanishing of
Ricci scalar. We present an exact solution of the Finslerian vacuum field
equation. The solution is similar to the Schwarzschild metric. It reduces to
Schwarzschild metric while the Finslerian parameter vanishes. We get solutions
of geodesic equation in such a Schwarzschild-like spacetime, and show that the
geodesic equation returns to the counterpart in Newton’s gravity at weak field
approximation. It is proved that the Finslerian covariant derivative of the
geometrical part of the gravitational field equation is conserved. The
interior solution is also given.

##  I Introduction

In 1912, Einstein proposed his famous general relativity which gives the
connection between Riemann geometry and gravitation. In general relativity,
the effects of gravitation are ascribed to spacetime curvature instead of a
force. In four dimensional spacetime, two solutions of the Einstein vacuum
field equation are well known Wald . These are the Schwarzschild solution
which preserves spherical symmetry, and the Kerr solution which preserves
axial symmetry. The Schwarzschild solution is of vital importance for general
relativity. The physics of Schwarzschild solution is quite different from
Newton’s gravity. The success of general relativity attribute to the four
classical tests Weinberg . The predictions of the four classical tests
directly come from the Schwarzschild solution. Most celestial bodies can be
approximately treated as a sphere. Thus, the Schwarzschild solution is widely
used in investigating the astronomical phenomena. However, the recent
astronomical observations show that the gravitational field of galaxy clusters
offset from its baryonic matters Shan . It implies that the spherical symmetry
may not preserve in the scale of galaxy clusters.

Finsler geometry Book by Bao  is a new geometry which involves Riemann
geometry as its special case. S. S. Chern pointed out that Finsler geometry is
just Riemann geometry without quadratic restriction, in his Notices of AMS.
The symmetry of spacetime is described by the isometric group. The generators
of isometric group are directly connected with the Killing vectors. It is well
known that the isometric group is a Lie group in Riemannian manifold. This
fact also holds in Finslerian manifold Deng . Generally, Finsler spacetime
admits less Killing vectors than Riemann spacetime does Finsler PF . The
numbers of independent Killing vectors of an $n$ dimensional non-Riemannian
Finsler spacetime should no more than $\frac{n(n-1)}{2}+1$ Wang . The causal
structure of Finsler spacetime is determined by the vanish of Finslerian
length Pfeifer1 . And the speed of light is modified. It has been shown that
the translation symmetry is preserved in flat Finsler spacetime Finsler PF .
Thus, the energy and momentum are well defined in Finsler spacetime. In flat
Finsler spacetime, inertial motion preserves the Finslerian length and admits
a modified dispersion relation.

Gibbons et al. Gibbons  have pointed out that Glashow’s Very Special
Relativity Coleman  is Finsler Geometry. The flat Finsler spacetime breaks the
Lorentz symmetry. Thus, it is a possible mechanism of Lorentz violation
Kostelecky . Stavrinos et al. Stavrinos  used the method of osculating
Riemannian space to study the cosmological anisotropy in Finsler spacetime.
Vacaru et al. studied high dimensional gravity in Finsler spacetime Vacaru ref
.

The counterpart of special relativity has been established in flat Finsler
spacetime. However, up to now, Finslerian gravity is still to be completed.
There are types of gravitational field equation in Finsler spacetime Li
Berwald ; Miron ; Rutz ; Vacaru ; Pfeifer . And these equations are not
equivalent to each other. Here are the reasons. It is well known that there is
only a torsion free connection-the Christoffel connection in Riemann geometry.
However, there are types of connection in Finsler geometry. Therefore, the
covariant derivatives that depend on the connection are different. The
Finslerian length element $F$ is constructed on a tangent bundle Book by Bao .
Thus, the gravitational field equation should be constructed on the tangent
bundle in principle. However, the corresponded energy-momentum tensor, which
should be constructed on the tangent bundle, is rather obscure in physics.

The analogy between geodesic deviation equations in Finsler spacetime and
Riemann spacetime gives the vacuum field equation in Finsler gravity Finsler
Bullet . It is the vanishing of Ricci scalar. The vanishing of the Ricci
scalar implies that the geodesic rays are parallel to each other. The
geometric invariant property of Ricci scalar implies that the vacuum field
equation is insensitive to the connection, which is an essential physical
requirement. In this paper, we present an exact solution of the vacuum field
equation in Finsler spacetime. The interior solution is also shown.

This paper is organized as follows. Sec.II is divided into three subsections.
In subsection A, we briefly introduce some basic geometric objects in Finsler
geometry. In subsection B, we present an exact solution of the vacuum field
equation in Finsler spacetime. In subsection C, we investigate the Newtonian
limit of our solution. In Sec. III, we propose a gravitational field equation
with source. We prove that the Finslerian covariant derivative of the
geometrical part of the gravitational field equation is conserved. Then, an
interior solution of gravitational field equation is shown. In Sec. IV, we
investigate particle motion in the Schwarzschild-like spacetime that is given
in Sec. II. We get three solutions of geodesic equations. In Sec. V, we
present an example of two dimensional Finsler space with constant positive
flag curvature, which is a subspace of the Schwarzschild-like spacetime. Then,
we discuss the geometric properties of the two dimensional Finsler space.
Conclusions and remarks are given in Sec. VI.

##  II The exact solution of vacuum field equation

###  II.1 Formulism of Finsler geometry

Instead of defining an inner product structure over the tangent bundle in
Riemann geometry, Finsler geometry is based on the so called Finsler structure
$F$ with the property $F(x,\lambda y)=\lambda F(x,y)$ for all $\lambda>0$,
where $x\in M$ represents position and $y\equiv\frac{dx}{d\tau}$ represents
velocity. The Finslerian metric is given as Book by Bao

$g_{\mu\nu}\equiv\frac{\partial}{\partial y^{\mu}}\frac{\partial}{\partial
y^{\nu}}\left(\frac{1}{2}F^{2}\right).$ (1)

A Finslerian metric is said to be Riemannian, if $F^{2}$ is quadratic in $y$.
Throughout this paper, the indices are lowered and raised by $g_{\mu\nu}$ and
its inverse matrix $g^{\mu\nu}$.

The geodesic equation for Finsler manifold is given as

$\frac{d^{2}x^{\mu}}{d\tau^{2}}+2G^{\mu}=0,$ (2)

where

$G^{\mu}=\frac{1}{4}g^{\mu\nu}\left(\frac{\partial^{2}F^{2}}{\partial
x^{\lambda}\partial y^{\nu}}y^{\lambda}-\frac{\partial F^{2}}{\partial
x^{\nu}}\right)$ (3)

is called geodesic spray coefficients. It can be proved from the geodesic
equation (2) that the Finslerian structure $F$ is constant along the geodesic.

In Finsler geometry, there is geometrical invariant quantity, i.e., Ricci
scalar. It is of the form

$Ric\equiv R^{\mu}_{\mu}=\frac{1}{F^{2}}\left(2\frac{\partial
G^{\mu}}{\partial x^{\mu}}-y^{\lambda}\frac{\partial^{2}G^{\mu}}{\partial
x^{\lambda}\partial y^{\mu}}+2G^{\lambda}\frac{\partial^{2}G^{\mu}}{\partial
y^{\lambda}\partial y^{\mu}}-\frac{\partial G^{\mu}}{\partial
y^{\lambda}}\frac{\partial G^{\lambda}}{\partial y^{\mu}}\right),$ (4)

where
$R^{\mu}_{~{}\nu}=R^{~{}\mu}_{\lambda~{}\nu\rho}y^{\lambda}y^{\rho}/F^{2}$.
Though $R^{~{}\mu}_{\lambda~{}\nu\rho}$ depends on connections,
$R^{\mu}_{~{}\nu}$ does not Book by Bao . The Ricci scalar only depends on the
Finsler structure $F$ and is insensitive to connections. The analogy between
geodesic deviation equations in Finsler spacetime and Riemann spacetime gives
the vacuum field equation in Finsler gravity Finsler Bullet . It is of the
form $Ric=0$.

###  II.2 Vacuum solution

Here, we propose an ansatz that the Finsler structure is of the form

$F^{2}=B(r)y^{t}y^{t}-A(r)y^{r}y^{r}-r^{2}\bar{F}^{2}(\theta,\varphi,y^{\theta},y^{\varphi}).$
(5)

Then, the Finsler metric can be derived as

$\displaystyle g_{\mu\nu}$ $\displaystyle=$ $\displaystyle{\rm
diag}(B,-A,-r^{2}\bar{g}_{ij}),$ (6) $\displaystyle g^{\mu\nu}$
$\displaystyle=$ $\displaystyle{\rm
diag}(B^{-1},-A^{-1},-r^{-2}\bar{g}^{ij}),$ (7)

where $\bar{g}_{ij}$ and its reverse are the metric that derived from
$\bar{F}$ and the index $i,j$ run over angular coordinate $\theta,\varphi$.

Plugging the Finsler structure (5) into the formula (3), we obtain that

$\displaystyle G^{t}$ $\displaystyle=$
$\displaystyle\frac{B^{\prime}}{2A}y^{t}y^{r},$ (8) $\displaystyle G^{r}$
$\displaystyle=$
$\displaystyle\frac{A^{\prime}}{4A}y^{r}y^{r}+\frac{B^{\prime}}{4A}y^{t}y^{t}-\frac{r}{2A}\bar{F}^{2},$
(9) $\displaystyle G^{\theta}$ $\displaystyle=$
$\displaystyle\frac{1}{r}y^{\theta}y^{r}+\bar{G}^{\theta},$ (10)
$\displaystyle G^{\varphi}$ $\displaystyle=$
$\displaystyle\frac{1}{r}y^{\varphi}y^{r}+\bar{G}^{\varphi},$ (11)

where the prime denotes the derivative with respect to $r$, and the $\bar{G}$
is the geodesic spray coefficients derived by $\bar{F}$. Plugging the geodesic
coefficients (8,9,10,11) into the formula of Ricci scalar (4), we obtain that

$\displaystyle F^{2}Ric=$
$\displaystyle\left[\frac{B^{\prime\prime}}{2A}-\frac{B^{\prime}}{4A}\left(\frac{A^{\prime}}{A}+\frac{B^{\prime}}{B}\right)+\frac{B^{\prime}}{rA}\right]y^{t}y^{t}$
(12) $\displaystyle+$
$\displaystyle\left[-\frac{B^{\prime\prime}}{2B}+\frac{B^{\prime}}{4B}\left(\frac{A^{\prime}}{A}+\frac{B^{\prime}}{B}\right)+\frac{A^{\prime}}{rA}\right]y^{r}y^{r}$
$\displaystyle+$
$\displaystyle\left[\bar{R}ic-\frac{1}{A}+\frac{r}{2A}\left(\frac{A^{\prime}}{A}-\frac{B^{\prime}}{B}\right)\right]\bar{F}^{2}~{},$

where $\bar{R}ic$ denotes the Ricci scalar of Finsler structure $\bar{F}$.
Since $\bar{F}$ is independent of $y^{t}$ and $y^{r}$, the vanish of Ricci
scalar implies that the term in each square bracket of equation (12) should
vanish respectively. These equations are given as

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\frac{B^{\prime\prime}}{2A}-\frac{B^{\prime}}{4A}\left(\frac{A^{\prime}}{A}+\frac{B^{\prime}}{B}\right)+\frac{B^{\prime}}{rA},$
(13) $\displaystyle 0$ $\displaystyle=$
$\displaystyle-\frac{B^{\prime\prime}}{2B}+\frac{B^{\prime}}{4B}\left(\frac{A^{\prime}}{A}+\frac{B^{\prime}}{B}\right)+\frac{A^{\prime}}{rA},$
(14) $\displaystyle 0$ $\displaystyle=$
$\displaystyle\bar{R}ic-\frac{1}{A}+\frac{r}{2A}\left(\frac{A^{\prime}}{A}-\frac{B^{\prime}}{B}\right)~{}.$
(15)

Noticing that $\bar{R}ic$ is independent of $r$, thus the equation (15) is
satisfied if and only if $\bar{R}ic$ equals to constant. It means that the two
dimensional Finsler space $\bar{F}$ is a constant flag curvature space. The
flag curvature is a generalization of sectional curvature in Riemann geometry.
Here, we label the constant flag curvature to be $\lambda$. Therefore,
$\bar{R}ic=\lambda$. The equations (13,14,15) are similar to the Schwarzschild
vacuum field equation in general relativity. The solutions of equations
(13,14,15) are given as

$\displaystyle B$ $\displaystyle=$ $\displaystyle a\lambda+\frac{b}{r},$ (16)
$\displaystyle A$ $\displaystyle=$
$\displaystyle\left(\lambda+\frac{b}{ra}\right)^{-1},$ (17)

where $a$ and $b$ are integral constants.

###  II.3 The Newtonian limit

In the above subsection, we have obtained the vacuum field solution in Finsler
spacetime. The integral constants of solutions (16,17) should be determined by
specific boundary conditions which is given by physical requirement. Here, we
require that the solutions should return to Newton’s gravity in weak field
approximation Weinberg . In order to compare with Newton’s gravity, we only
consider the radial motion of particles. Plugging the solutions (16,17) into
the geodesic coefficients (8,9), and noticing that the velocity of particle
$\frac{dr}{dt}$ is small, we obtain the geodesic equations

$\displaystyle\frac{d^{2}t}{d\tau^{2}}=0,$ (18)
$\displaystyle\frac{d^{2}r}{d\tau^{2}}-\frac{b\lambda}{2r^{2}}\left(\frac{dt}{d\tau}\right)^{2}=0.$
(19)

Combining the geodesic equations (18,19), we obtain that

$\frac{d^{2}r}{dt^{2}}=\frac{b\lambda}{2r^{2}}.$ (20)

Comparing the equation (20) with Newton’s gravity, we conclude that

$b\lambda=-2GM,$ (21)

where $M$ denotes the total mass of gravitational source.

##  III Interior solution

It is well known that the Schwarzschild spacetime has interior solution, which
can deduce the famous Oppenheimer-Volkoff equation. The interior behavior of
Finsler spacetime (5) is worth investigating. However, as we mentioned in the
introduction, there are obstructions in constructing gravitational field
equation in Finsler spacetime. In this section, we will show that there is a
self-consistent gravitational field equation in Finsler spacetime (5).

The notion of Ricci tensor in Finsler geometry was first introduced by Akbar-
ZadehAkbar

$Ric_{\mu\nu}=\frac{\partial^{2}\left(\frac{1}{2}F^{2}Ric\right)}{\partial
y^{\mu}\partial y^{\nu}}.$ (22)

And the scalar curvature in Finsler geometry is given as
$S=g^{\mu\nu}Ric_{\mu\nu}$. Here, we define the modified Einstein tensor in
Finsler spacetime

$G_{\mu\nu}\equiv Ric_{\mu\nu}-\frac{1}{2}g_{\mu\nu}S.$ (23)

Plugging the equation of Ricci scalar (12) into the formula (23), and noticing
that $\bar{F}$ is two dimensional Finsler spacetime with constant flag
curvature $\lambda$, we obtain

$\displaystyle G^{t}_{t}$ $\displaystyle=$
$\displaystyle\frac{A^{\prime}}{rA^{2}}-\frac{1}{r^{2}A}+\frac{\lambda}{r^{2}},$
(24) $\displaystyle G^{r}_{r}$ $\displaystyle=$
$\displaystyle-\frac{B^{\prime}}{rAB}-\frac{1}{r^{2}A}+\frac{\lambda}{r^{2}},$
(25) $\displaystyle G^{\theta}_{\theta}=G^{\varphi}_{\varphi}$
$\displaystyle=$
$\displaystyle-\frac{B^{\prime\prime}}{2AB}-\frac{B^{\prime}}{2rAB}+\frac{A^{\prime}}{2rA^{2}}+\frac{B^{\prime}}{4AB}\left(\frac{A^{\prime}}{A}+\frac{B^{\prime}}{B}\right).$
(26)

Next, we investigate the covariant conserve properties of the tensor
$G^{\mu}_{\nu}$. The covariant derivative of $G^{\mu}_{\nu}$ in Finsler
spacetime is given as Book by Bao

$G^{\mu}_{\nu~{}|\mu}=\frac{\delta}{\delta
x^{\mu}}G^{\mu}_{\nu}+\Gamma^{\mu}_{\mu\rho}G^{\rho}_{\nu}-\Gamma^{\rho}_{\mu\nu}G^{\mu}_{\rho},$
(27)

where

$\frac{\delta}{\delta x^{\mu}}=\frac{\partial}{\partial
x^{\mu}}-\frac{\partial G^{\rho}}{\partial y^{\mu}}\frac{\partial}{\partial
y^{\rho}},$ (28)

and $\Gamma^{\mu}_{\mu\rho}$ is the Chern connection. Here, we have used ‘$|$’
to denote the covariant derivative. The form of covariant derivative (27) and
‘$\delta$’-derivative (28) are well defined such that they transform as tensor
under a coordinate change in Finsler spacetime Book by Bao . The Chern
connection can be expressed in terms of geodesic spray coefficients $G^{\mu}$
and Cartan connection
$A_{\lambda\mu\nu}\equiv\frac{F}{4}\frac{\partial}{\partial
y^{\lambda}}\frac{\partial}{\partial y^{\mu}}\frac{\partial}{\partial
y^{\nu}}(F^{2})$

$\Gamma^{\rho}_{\mu\nu}=\frac{\partial^{2}G^{\rho}}{\partial y^{\mu}\partial
y^{\nu}}-A^{\rho}_{\mu\nu|\kappa}\frac{y^{\kappa}}{F}.$ (29)

Noticing that the modified Einstein tensor $G^{\mu}_{\nu}$ only depend on $r$
and do not have $y$-dependence, and Cartan tensor
$A^{\rho}_{\mu\nu}=A^{i}_{jk}$ (index $i,j,k$ run over $\theta,\varphi$), one
can easily get that
$G^{\mu}_{t~{}|\mu}=G^{\mu}_{\theta~{}|\mu}=G^{\mu}_{\varphi~{}|\mu}=0$. The
proof of $G^{\mu}_{r~{}|\mu}=0$ is somewhat subtle. By making use of the
equations (8,10,11), we find from (29) and $A^{\rho}_{\mu\nu}=A^{i}_{jk}$ that

$\Gamma^{t}_{rt}=\frac{B^{\prime}}{2B},~{}~{}~{}\Gamma^{\theta}_{r\theta}=\Gamma^{\varphi}_{r\varphi}=\frac{1}{r}.$
(30)

Then, after a tedious calculation, one can check that the equation
$G^{\mu}_{r~{}|\mu}=0$ indeed satisfy. Following the sprite of general
relativity, we propose that the gravitational field equation in the given
Finsler spacetime (5) should be of the form

$G^{\mu}_{\nu}=8\pi_{F}GT^{\mu}_{\nu},$ (31)

where $T^{\mu}_{\nu}$ is the energy-momentum tensor. The volume of Finsler
space Book Shen  is generally different with the one of Riemann geometry. We
have used $4\pi_{F}$ to denote the volume of $\bar{F}$ in field equation (31).

For simplicity, we set the energy-momentum tensor to be of the form

$T^{\mu}_{\nu}={\rm diag}(\rho(r),-p(r),-p(r),-p(r)),$ (32)

where $\rho(r)$ and $p(r)$ are the energy density and pressure of the
gravitational source, respectively. Then, by making use of equations
(24,25,26), we reduce the gravitational field equation to three independent
equations

$\displaystyle\frac{2p^{\prime}}{\rho+p}$ $\displaystyle=$
$\displaystyle-\frac{B^{\prime}}{B},$ (33)
$\displaystyle\frac{A^{\prime}}{rA^{2}}-\frac{1}{r^{2}A}+\frac{\lambda}{r^{2}}$
$\displaystyle=$ $\displaystyle 8\pi_{F}G\rho,$ (34)
$\displaystyle\frac{B^{\prime}}{rAB}+\frac{1}{r^{2}A}-\frac{\lambda}{r^{2}}$
$\displaystyle=$ $\displaystyle 8\pi_{F}Gp.$ (35)

The solution of equation (34) is given as

$A^{-1}=\lambda-\frac{2Gm(r)}{r},$ (36)

where $m(r)\equiv\int^{r}_{0}4\pi_{F}x^{2}\rho(x)dx$. By making use of
equation (36), and plugging the equation (35) into (33), we obtain that

$-r^{2}p^{\prime}=(\rho+p)(4\pi_{F}Gpr^{3}+Gm)\left(\lambda-\frac{2Gm}{r}\right)^{-1}.$
(37)

The equation (37) reduces to the famous Oppenheimer-Volkoff equation if
Finsler spacetime $\bar{F}$ reduces to two dimensional Riemann sphere.
Combining the modified Oppenheimer-Volkoff equation (37) with the equation of
state, one can obtain the interior structure of gravitational source.

The interior solution (36) should consistent with the exterior solution (17)
at the boundary of the gravitational source. Therefore, we get

$a\lambda=1.$ (38)

At last, combining the boundary condition (38) with the requirement of
Newtonian limit (21), we get the exterior solution $B(r)$ and $A(r)$ as

$\displaystyle B(r)$ $\displaystyle=$ $\displaystyle 1-\frac{2GM}{\lambda r},$
(39) $\displaystyle A(r)$ $\displaystyle=$
$\displaystyle\left(\lambda-\frac{2GM}{r}\right)^{-1}.$ (40)

##  IV The motion of particles

Plugging the equations of geodesic spray coefficients (8,9,10,11) into the
formula of geodesic equation (2), we obtain the geodesic equation of Finsler
spacetime (5)

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\frac{d^{2}t}{d\tau^{2}}+\frac{B^{\prime}}{B}\frac{dr}{d\tau}\frac{dt}{d\tau},$
(41) $\displaystyle 0$ $\displaystyle=$
$\displaystyle\frac{d^{2}r}{d\tau^{2}}+\frac{B^{\prime}}{2A}\left(\frac{dt}{d\tau}\right)^{2}+\frac{A^{\prime}}{2A}\left(\frac{dr}{d\tau}\right)^{2}-\frac{r}{A}\bar{F}^{2}\left(\frac{d\theta}{d\tau},\frac{d\varphi}{d\tau}\right),$
(42) $\displaystyle 0$ $\displaystyle=$
$\displaystyle\frac{d^{2}\theta}{d\tau^{2}}+\frac{2}{r}\frac{dr}{d\tau}\frac{d\theta}{d\tau}+2\bar{G}^{\theta},$
(43) $\displaystyle 0$ $\displaystyle=$
$\displaystyle\frac{d^{2}\varphi}{d\tau^{2}}+\frac{2}{r}\frac{dr}{d\tau}\frac{d\varphi}{d\tau}+2\bar{G}^{\varphi}.$
(44)

The solution of equation (41) is

$B\frac{dt}{d\tau}=1,$ (45)

where we have set the integral constant to be $1$ by the normalization of
$\tau$. Noticing that $y^{\mu}$ equals to $\frac{dx^{\mu}}{d\tau}$ along the
geodesic, and by making use of the equations (43,44), we find that

$\displaystyle\frac{d\bar{F}}{d\tau}=\frac{\partial\bar{F}}{\partial
x^{i}}\frac{dx^{i}}{d\tau}+\frac{\partial\bar{F}}{\partial
y^{i}}\frac{dy^{i}}{d\tau}=y^{i}\left(\frac{\partial\bar{F}}{\partial
x^{i}}-\frac{2\bar{G}_{i}}{\bar{F}}\right)-2y^{r}\bar{F}=-\bar{F}\frac{d\ln
r^{2}}{d\tau},$ (46)

where $\bar{G}_{i}=\bar{g}_{ij}\bar{G}^{j}$ ($i,j$ run over $\theta,\varphi$),
and we have used the fact that $\bar{F}$ is a homogenous function of $y$ of
degree $1$ to derive the third equation of (46). The solution of equations
(46) is given as

$r^{2}\bar{F}=J,$ (47)

where $J$ is an integral constant. By making use of the equations (45,47), we
find from geodesic equation (42) that

$\frac{d}{d\tau}\left(A\left(\frac{dr}{d\tau}\right)^{2}+\frac{J^{2}}{r^{2}}-\frac{1}{B}\right)=0.$
(48)

The solution of (48) is given as

$A\left(\frac{dr}{d\tau}\right)^{2}+\frac{J^{2}}{r^{2}}-\frac{1}{B}=A\left(\frac{dr}{d\tau}\right)^{2}+r^{2}\bar{F}^{2}-B\left(\frac{dt}{d\tau}\right)^{2}=-F^{2},$
(49)

where we have used the equations (45,47) to derive the second equation of
(49). The equation (49) means that $F$ is constant along the geodesic.

Now, we have three solutions (45,47,49) of the geodesic equations, the fourth
one depends on the explicit form of two dimension Finsler space $\bar{F}$.
However, we can still find some information of particle motion from the
obtained solutions. Consider a particle move along radial direction, combining
the equation (45) with (47), and by making use of the exterior solutions
(39,40) of $B(r)$ and $A(r)$, we obtain that

$\frac{dr}{dt}=\sqrt{\lambda^{-1}-F^{2}\left(1-\frac{2GM}{\lambda
r}\right)}\left(1-\frac{2GM}{\lambda r}\right).$ (50)

It is obvious from (50) that $\frac{dr}{dt}\rightarrow 0$ while $r\rightarrow
2GM/\lambda$. The modified Schwarzschild radius in Finsler spacetime (5) is

$r_{s}=\frac{2GM}{\lambda}.$ (51)

##  V Two dimensional Finsler space with constant flag curvature

In this section, we will present an example of two dimensional Finsler space
with constant positive flag curvature.

Bao et al. Bao  have given a completely classification of Randers-Finsler
space Randers  with constant flag curvature. A two dimensional Randers-Finsler
space with constant positive flag curvature $\lambda=1$ is given as

$\bar{F}=\frac{\sqrt{(1-\epsilon^{2}\sin^{2}\theta)y^{\theta}y^{\theta}+\sin^{2}\theta
y^{\varphi}y^{\varphi}}}{1-\epsilon^{2}\sin^{2}\theta}-\frac{\epsilon\sin^{2}\theta
y^{\varphi}}{1-\epsilon^{2}\sin^{2}\theta},$ (52)

where $0\leq\epsilon<1$. It is obvious that the metric (52) returns to
Riemannian sphere while $\epsilon=0$. And the metric (52) is non-reversible
for $\varphi\rightarrow-\varphi$. The Randers-Finsler space (52) has two close
geometrically distinct closed geodesics Rademacher  if $\epsilon$ is
irrational. The two geodesics locate at $\theta=\frac{\pi}{2}$ with length
$L_{\pm}=2\pi(1\pm\epsilon)^{-1}$. This fact can be proved by plugging the
metric (52) into the formula of geodesic equation (2). Then, one can find that
$\theta=\frac{\pi}{2}$ and $\varphi=u\tau+v$ ($u,v$ are integral constants)
are the solutions of geodesic equation. The Randers-Finsler space (52) is
homotopy equivalent to the two dimensional sphere Rademacher .

In terms of Busemann-Hausdorff volumm form, the volume of a close Randers-
Finsler surface $F=\sqrt{a_{ij}(x)y^{i}y^{j}}+b_{i}(x)y^{i}$ is given as Book
Shen

$Vol_{F}=\int(1-(a^{ij}b_{i}b_{j}))^{\frac{3}{2}}\sqrt{\rm
det(a_{ij})}dx^{1}\wedge dx^{2}.$ (53)

Plugging the Randers metric (52) into the formula (53), we obtain that its
volume is $4\pi$, which is the same with unit Riemannian sphere.

##  VI Conclusions and Remarks

In view of geodesic deviation equation, the vacuum field equation $Ric=0$ in
Finsler spacetime implies that the geodesic rays are parallel to each other.
The geometric invariant property of Ricci scalar implies that the vacuum field
equation is insensitive to the connection, which is an essential physical
requirement. Starting from the ansatz (5), we have found an exact solution of
vacuum field equation (16,17).

A general gravitational field equation in Finsler spacetime is still to be
completed. However, we have found that the gravitational field equation for
the metric (5) should be the form as (31). And, we have proved that the
Finslerian covariant derivative of the geometrical part of the gravitational
field equation is conserved. It is obvious that the gravitational field
equation (31) returns to the vacuum field equation while the energy-momentum
tensor vanishes. We have found an interior solution of the gravitational field
equation (31). The interior solution (36) consistent with the exterior
solution (17) at the boundary of the gravitational source. And we required
that the exterior solution should return to Newton’s gravity. The two boundary
conditions constrain that the exterior solution should be the form as (39,40).
Together with specific form of two dimensional Finsler space (52) with
constant positive flag curvature $\lambda=1$, the exterior metric of vacuum
field solution was given as

$\displaystyle F^{2}=$
$\displaystyle\left(1-\frac{2GM}{r}\right)y^{t}y^{t}-\left(1-\frac{2GM}{r}\right)^{-1}y^{r}y^{r}$
(54) $\displaystyle-$ $\displaystyle
r^{2}\left(\frac{\sqrt{(1-\epsilon^{2}\sin^{2}\theta)y^{\theta}y^{\theta}+\sin^{2}\theta
y^{\varphi}y^{\varphi}}-\epsilon\sin^{2}\theta
y^{\varphi}}{1-\epsilon^{2}\sin^{2}\theta}\right)^{2}.$

The metric (54) is no other than the Schwarzschild metric except for the
change from Riemannian sphere to “Finslerian sphere” (52). We have presented
three solutions (45,47,49) of geodesic equations of the metric (54). The
fourth one depends on the geodesic equation of the “Finslerian sphere” (52).
The geometrical properties of ‘Finslerian sphere” (52) are as follows. It is
non-reversible for $\varphi\rightarrow-\varphi$, it has two close geodesics
locate at $\theta=\frac{\pi}{2}$ with length
$L_{\pm}=2\pi(1\pm\epsilon)^{-1}$, its volume that is the surface volume of
unit “Finslerian sphere” equals to $4\pi$, it only have one independent
Killing vector $V^{\varphi}=\rm constant$.

The Schwarzschild spacetime will return to Minkowski spacetime if there is no
gravitational source. As for the Finslerian vacuum spacetime (54), if there is
no gravitational source, namely, $M=0$, the metric reduces to

$\displaystyle F^{2}=$ $\displaystyle
y^{t}y^{t}-y^{r}y^{r}-r^{2}\left(\frac{\sqrt{(1-\epsilon^{2}\sin^{2}\theta)y^{\theta}y^{\theta}+\sin^{2}\theta
y^{\varphi}y^{\varphi}}-\epsilon\sin^{2}\theta
y^{\varphi}}{1-\epsilon^{2}\sin^{2}\theta}\right)^{2}.$ (55)

According to the formula (12), the Ricci scalar or Ricci tensor of the metric
(55) equal to $0$. This fact holds even for three dimensional subspace of the
metric (55). However, the metric (55) or its spatial part is not a flat
Finsler spacetime. The Finsler spacetime is a flat one Book by Bao  if and
only if $Ric=0$ and the geodesic spray coefficients $G^{\mu}$ is quadratic in
$y$. This fact is quite different with Riemann geometry. It is well known that
three dimensional Riemann space is flat while its Ricci tensor equal to $0$.
Nevertheless, even the space part of the metric (55) is not a flat Finsler
space.

In Riemann spacetime without torsion, at any fixed point, one can erect a
local coordinate system such that the metric is Minkowskian. One necessary
condition of this statement is the Riemann metric is quadric. However, this
necessary condition does not hold in a general Finsler metric. Therefore,
Finsler spacetime is not locally isometric to a Minkowski spacetime. One
deduction of it is the speed of light is not locally isotropic. The
propagation of light obeys $F=0$. One can find from the local metric that the
radial speed of light equals to $1$. And the non-radial speed of light satisfy

$c_{\theta}^{2}+(c_{\varphi}-\epsilon\sin\theta)^{2}=1,$ (56)

where $c_{\theta}\equiv\frac{d\theta}{dt}$ and
$c_{\varphi}\equiv\frac{d\varphi}{dt}\sin\theta$.

The Schwarzschild radius form an event horizon in Schwarzschild spacetime. The
Schwarzschild solution can be maximally extended by Kruskal extension. The
coordinate transformation between Schwarzschild metric and Kruskal metric is
only related to $r$ and $t$. Therefore, one can also get a maximally extended
Finslerian vacuum solution (54) by Kruskal’s method.

###### Acknowledgements.

We would like to thank Prof. C.-G. Huang for useful discussions. Project
11375203 and 11305181 supported by NSFC.

## References

  * (1) R. M. Wald, General Relativity, The University of Chicago Press, Chicago, 1984.
  * (2) S. Weinberg, Gravitation and Cosmology: Principles and Applications of the General Theory of Relativity, John Wiley & Sons, New York, 1972.
  * (3) H. Y. Shan, et al., Mon. Not. R. Astron. Soc. 406, 1134 (2010).
  * (4) D. Bao, S. S. Chern, and Z. Shen, An Introduction to Riemann–Finsler Geometry, Graduate Texts in Mathematics 200, Springer, New York, 2000.
  * (5) S. Deng and Z. Hou, Pac. J. Math. 207, 149 (2002).
  * (6) X. Li and Z. Chang, Differ. Geom. Appl. 30, 737 (2012).
  * (7) H. C. Wang, J. Lond. Math. Soc. s1-22(1), 5 (1947).
  * (8) C. Pfeifer, M. N. R. Wohlfarth, Phys. Rev. D 84, 044039 (2011).
  * (9) G. W. Gibbons, J. Gomis, and C. N. Pope, Phys. Rev. D 76, 081701 (2007).
  * (10) S. R. Coleman and S. L. Glashow, Phys. Lett. B405, 249 (1997); S. R. Coleman and S. L. Glashow, Phys. Rev. D59, 116008 (1999); A. G. Cohen and S. L. Glashow, Phys. Rev. Lett. 97, 021601 (2006).
  * (11) V. A. Kostelecky, Phys. Lett. B 701, 137 (2011); V. A. Kostelecky, N. Russell, and R. Tsoc, Phys. Lett. B 716, 470 (2012).
  * (12) A. P. Kouretsis, M. Stathakopoulos, and P. C. Stavrinos, Phys. Rev. D 79, 104011 (2009); Phys. Rev. D 82, 064035 (2010).
  * (13) S. Vacaru, Nuc. Phys. B 494, 590 (1997); S. Vacaru, D. Singleton, V. A. Botan, and D. A. Dotenco, Phys. Lett. B 519, 249 (2001).
  * (14) X. Li and Z. Chang, Chinese Physics C 34, 28 (2010).
  * (15) R. Miron and M. Anastasiei, The Geometry of Lagrange Spaces: Theory and Applications, Kluwer Acad. Publ. FTPH no. 59, (1994).
  * (16) S. F. Rutz, General Relativity and Gravitation, 25(11), 1139 (1993).
  * (17) S. Vacaru, et al., Clifford and Riemann-Finsler Structures in Geometric Mechanics and Gravity, Selected Works, Differential Geometry C Dynamical Systems, Monograph, vol. 7, Geometry Balkan Press, 2006, http://www.mathem.pub.ro/dgds/mono/va-t.pdf, and arXiv:gr-qc/0508023.
  * (18) C. Pfeifer and M. N. R. Wohlfarth, Phys. Rev. D 85, 064009 (2012).
  * (19) X. Li, M.-H. Li, H.-N. Lin, and Z. Chang, Mon. Not. R. Astron. Soc. 428, 2939 (2013).
  * (20) H. Akbar-Zadeh, Acad. Roy. Belg. Bull. Cl. Sci. (5) 74, 281 (1988).
  * (21) Z. Shen, Lectures on Finsler Geometry, World Scientific, Singapore, 2001.
  * (22) D. Bao, C. Robles, and Z. Shen, J. Diff. Geom. 66, 377 (2004).
  * (23) G. Randers, Phys. Rev. 59, 195 (1941).
  * (24) H. B. Rademacher, Math. Ann. 328, 373 (2004).


# Barrier penetration in a discrete basis formalism

K. Hagino  Department of Physics, Kyoto University, Kyoto 606-8502, Japan

###### Abstract

A common approach to model the dynamics of many-particle systems is to map the
Hamiltonian onto a Schrödinger equation, which is then numerically integrated.
An alternative approach is to solve the Hamiltonian equations directly in a
model space of many-body configurations. Here I compare the numerical
convergence of the Numerov integration method to the Hamiltonian method for
calculating penetrabilities of a one-dimensional potential barrier. The
Hamiltonian method is easier to apply and I find that it has better
convergence properties.

##  I Introduction

In most of physics problems, a Schrödinger equation cannot be solved
analytically but has to be solved numerically. For a bound state problem, one
may expand wave functions on some discrete basis and diagonalize the resultant
Hamiltonian matrix. Alternatively, one may discretize the radial coordinates
and successively obtain a wave function at the mesh points with e.g., the
Runge-Kutta method or the Numerov method koonin .

A yet different method, referred to as a discrete-basis formalism, has been
proposed in Ref. FBA18 . In this method, one first forms a Hamiltonian matrix
based on discretized radial meshes and solve it with a linear algebra with
appropriate boundary conditions. An advantage of this method is that the
method is well compatible with a many-body Hamiltonian, in particular in a
configuration-interaction formulation.

Even though the discrete-basis formalism has been applied to an induced
fission problem FBA18 ; YBF2020 ; YBF2021 ; BH2022 , its applicability has not
yet been clarified. In this paper, I therefore apply the discrete-basis
formalism to a simple one-dimensional barrier penetration problem, and carry
out a comparative study of the numerical accuracy. To this end, I shall
consider a Gaussian barrier and compare the penetrabilities obtained with the
discrete-basis formalism to those with the standard Numerov method.

The paper is organized as follows. In Sec. II, I will detail the discrete-
basis formalism for a one-dimensional problem. In Sec. III, I will apply it to
a barrier penetration of a one-dimensional Gaussian barrier and discuss the
applicability of the discrete-basis formalism. I will then summarize the paper
in Sec. IV.

##  II Discrete-basis formalism for barrier penetration

Consider a one-dimensional system for a particle with mass $m$ under a
potential $V(x)$. The Hamiltonian for this system reads,

$H=-\frac{\hbar^{2}}{2m}\frac{d^{2}}{dx^{2}}+V(x).$ (1)

I discretize the radial coordinate as $x_{i}=x_{\rm min}+(i-1)\Delta x$, and
consider the model space from $x_{1}=x_{\rm min}$ to $x_{N}\equiv x_{\rm
max}$. Using the 3-point formula for the kinetic energy in $H$, the
Hamiltonian (1) is transformed to a matrix form of

$H_{ij}=-t\delta_{i,j+1}+(2t+V_{i})\delta_{i,j}-t\delta_{i,j-1},$ (2)

where $t$ is defined as $t=\frac{\hbar^{2}}{2m(\Delta x)^{2}}$ and
$V_{i}\equiv V(x_{i})$. The wave function $\phi_{i}\equiv\phi(x_{i})$ then
obeys

$-t\phi_{0}\delta_{i,1}+\sum_{j=1}^{N}H_{ij}\phi_{j}-t\phi_{N+1}\delta_{i,N}=E\phi_{i}.$
(3)

In the absence of the potential $V$, the wave function $\phi_{n}^{(0)}$ obeys
the equation

$-t(\phi_{n+1}^{(0)}-2\phi_{n}^{(0)}+\phi_{n-1}^{(0)})=E\phi_{n}^{(0)}.$ (4)

I consider a free-particle solution given by

$\phi_{n}^{(0)}\propto e^{-ikn\Delta x}-e^{ikn\Delta x}.$ (5)

Substituting this into Eq. (4), one finds

$\cos(k\Delta x)=1-\frac{E}{2t}.$ (6)

In the presence of the potential $V$, I consider the case where the particle
is incident from the left hand side of the potential. Assuming that the
potential $V$ almost vanishes at $x_{\rm max}$, the wave function $\phi_{N+1}$
is given by $\phi_{N+1}=e^{ik\Delta x}\phi_{N}$. Substituting this into Eq.
(3), one finds

$\phi_{i}=[(\tilde{H}-E)^{-1}]_{i1}t\phi_{0}\equiv G_{i1}t\phi_{0},$ (7)

where $\tilde{H}$ is defined as $\tilde{H}_{ij}=H_{ij}-te^{ik\Delta
x}\delta_{i,N}\delta_{j,N}$ and the Green’s function $G$ is defined as
$G=(\tilde{H}-E)^{-1}$.

We assume that the wave functions $\phi_{1}$ and $\phi_{2}$ are given by

$\displaystyle\phi_{1}$ $\displaystyle=$ $\displaystyle(Ae^{ik\Delta
x}+Be^{-ik\Delta x})t\phi_{0},$ (8) $\displaystyle\phi_{2}$ $\displaystyle=$
$\displaystyle(Ae^{2ik\Delta x}+Be^{-2ik\Delta x})t\phi_{0},$ (9)

respectively. This is equivalent to assume

$G_{11}=Ae^{ik\Delta x}+Be^{-ik\Delta x}.$ (10)

Substituting Eqs. (8) and (9) into Eq. (3) and using Eq. (6), one finds

$Ae^{2ik\Delta x}+Be^{-2ik\Delta x}=2\cos(k\Delta x)G_{11}-\frac{1}{t}.$ (11)

Combining this with Eq. (10), one finds

$\displaystyle A$ $\displaystyle=$ $\displaystyle\frac{e^{-ik\Delta
x}}{e^{ik\Delta x}-e^{-ik\Delta x}}\,(e^{ik\Delta x}G_{11}-1/t),$ (12)
$\displaystyle B$ $\displaystyle=$ $\displaystyle-\frac{e^{ik\Delta
x}}{e^{ik\Delta x}-e^{-ik\Delta x}}\,(e^{-ik\Delta x}G_{11}-1/t).$ (13)

Writing the wave function $\phi_{N}$ as $\phi_{N}=G_{N1}t\phi_{0}\equiv
Te^{ik\Delta x}t\phi_{0}$, the penetrability $P(E)$ reads,

$P(E)=\left|\frac{T}{A}\right|^{2}=\left|\frac{2\sin(k\Delta
x)G_{N1}}{e^{ik\Delta x}G_{11}-1/t}\right|^{2}.$ (14)

##  III Penetrability of a Gaussian barrier

Figure 1:  The penetraibilities of a Gaussian barrier given by Eq. (15) with
$V_{0}=100$ MeV and $s=$ 2 fm. The mass is set to be $m=29m_{N}$, where
$m_{N}$ is the nucleon mass. The upper panel is obtained with the Numerov
method (the dashed line) and the discrete-basis formalism (the filled circles)
with the mesh size of $\Delta x=0.05$ fm. On the other hand, the lower panel
shows the results of the Numerov method (the dashed line), the modified
Numerov method (the dotted line), and the discrete-basis formalism (the solid
line) with the mesh size of $\Delta x=0.15$ fm.

Let us now numerically evaluate the penetrability for a given potential. For
this purpose, I consider a Gaussian potential,

$V(x)=V_{0}\,e^{-x^{2}/2s^{2}}.$ (15)

Following Refs. dasso83 ; dasso83b ; HB04 , the parameters are chosen to be
$V_{0}=100$ MeV and $s=2$ fm together with $m=29m_{N}$, where $m_{N}$ is the
nucleon mass, to mimic the fusion reaction of 58Ni+58Ni. We set $x_{\rm
min}=-10$ fm and $x_{\rm max}=10$ fm.

The upper panel of Fig. 1 shows the penetrabilities of the Gaussian barrier
obtained with $\Delta x=0.05$ fm. The dashed line and the solid circles denote
the results with the standard Numerov method and the discrete-basis formalism,
respectively. The value of $\Delta x$ is small enough in this case, and both
the method lead to accurate results. The lower panel shows the results with a
larger value of $\Delta x$, that is, $\Delta x=0.15$ fm. In this case, the
numerical error is significantly large with the Numerov method: the
penetrabilities do not reach unity even at energies well above the barrier
(see the dashed line). This is the case also with the modified Numverov method
ccfull , with which the penetrability even exceeds unity at high energies with
a non-monotonic behavior (see the dotted line). In marked contrast, the
results with the discrete-basis formalism is rather robust and the
penetrabilities are almost the same as the one with $\Delta x=0.05$ fm shown
in the upper panel. Notice that the discrete-basis formalism employs the
simple 3-point formula for the kinetic energy, while a more sophisticated
formula is used in the Numerov and the modified Numerov methods. Yet, it is
interesting to notice that the discrete-basis method is numerically more
stable than the Numerov and the modified Numerov methods. We point out that
$\Delta x$ cannot be taken larger than $\sqrt{2\hbar^{2}/Em}$, though. If
$\Delta x$ exceeds this value, the right hand side of Eq. (6) exceeds unity
and the wave number $k$ cannot be defined unless it is extended to a complex
number.

##  IV Summary

I examined the applicability of the discrete-basis method for a reaction
theory. To this end, I considered barrier penetration of a one-dimensional
Gaussian barrier. It was demonstrated that the discrete-basis method provides
a more accurate and stable method than the standard Numerov method. This
property may be helpful in obtaining numerically stable solutions of coupled-
channels equations HOM2022 ; HT2012 .

The discrete-basis formalism has a good connection to a many-body Hamiltonian.
As a matter of fact, there have been several applications of this method to
microscopic descriptions of induced fission. In such applications, absorbing
potentials, or imaginary energies, are introduced to a model Hamiltonian, and
the absorbing probability is computed with the so called Datta formula FBA18 .
Even though the model setup is somewhat different from a barrier problem in
one-dimension, in which there is no absorbing part in the Hamiltonian, the
conclusion in this paper would remain the same in the fission problem as well.

###### Acknowledgements.

The author thanks G.F. Bertsch for helpful discussions and for a careful
reading of the manuscript. This work was supported in part by JSPS KAKENHI
Grant Numbers JP19K03861 and JP23K03414.

## References

  * (1) S.E. Koonin and D.C. Meredith, Computational Physics (Addison-Wesley, Reading, 1990).
  * (2) P. Fanto, G.F. Bertsch, and Y. Alhassid, Phys. Rev. C98, 014604 (2018).
  * (3) Y. Alhassid, G.F. Bertsch, and P. Fanto, Ann. of Phys.413, 168233 (2020).
  * (4) Y. Alhassid, G.F. Bertsch, and P. Fanto, Ann. of Phys.424, 168381 (2021).
  * (5) G.F. Bertsch and K. Hagino, Phys. Rev. C105, 034618 (2022).
  * (6) C.H. Dasso, S. Landowne, and A. Winther, Nucl. Phys. A405, 381 (1983).
  * (7) C.H. Dasso, S. Landowne, and A. Winther, Nucl. Phys. A407, 221 (1983).
  * (8) K. Hagino and A.B. Balantekin, Phys. Rev. A70, 032106 (2004).
  * (9) K. Hagino, N. Rowley, and A.T. Kruppa, Comput. Phys. Comm. 123, 143 (1999).
  * (10) K. Hagino, K. Ogata, and A.M. Moro, Prog. in Part. and Nucl. Phys. 125, 103951 (2022).
  * (11) K. Hagino and N. Takigawa, Prog. Theo. Phys. 128, 1061 (2012).


# Significant Improvements over the State of the Art? A Case Study of the MS
MARCO Document Ranking Leaderboard

Jimmy Lin,${}^{{}_{{}_{{}^{{}^{1,2}}}}}$ Daniel
Campos,${}^{{}_{{}_{{}^{{}^{3}}}}}$ Nick Craswell,${}^{{}_{{}_{{}^{{}^{2}}}}}$
Bhaskar Mitra,${}^{{}_{{}_{{}^{{}^{2,4}}}}}$ and Emine
Yilmaz${}^{{}_{{}_{{}^{{}^{4}}}}}$ 1 University of Waterloo 2 Microsoft AI &
Research
3 University of Illinois Urbana-Champaign 4 University College London

###### Abstract.

Leaderboards are a ubiquitous part of modern research in applied machine
learning. By design, they sort entries into some linear order, where the top-
scoring entry is recognized as the “state of the art” (SOTA). Due to the rapid
progress being made in information retrieval today, particularly with neural
models, the top entry in a leaderboard is replaced with some regularity. These
are touted as improvements in the state of the art. Such pronouncements,
however, are almost never qualified with significance testing. In the context
of the MS MARCO document ranking leaderboard, we pose a specific question: How
do we know if a run is significantly better than the current SOTA? We ask this
question against the backdrop of recent IR debates on scale types: in
particular, whether commonly used significance tests are even mathematically
permissible. Recognizing these potential pitfalls in evaluation methodology,
our study proposes an evaluation framework that explicitly treats certain
outcomes as distinct and avoids aggregating them into a single-point metric.
Empirical analysis of SOTA runs from the MS MARCO document ranking leaderboard
reveals insights about how one run can be “significantly better” than another
that are obscured by the current official evaluation metric (MRR@100).

††copyright: none

##  1\. Introduction

Leaderboard rankings and claims of the “state of the art” (SOTA) pervade
modern research in applied machine learning, particularly in natural language
processing, information retrieval, and computer vision. There has been much
debate in the community on the merits of such activities, compared to
alternative uses of the same researcher energy, attention, and resources.
Without participating in this debate, this work attempts to address what we
view as a technical shortcoming of many, if not most, leaderboards today: the
lack of significance testing.

Specifically, we wish to answer the question: Does a particular run
significantly improve the state of the art? The nature of leaderboards and
rapid progress by researchers mean that the top-scoring run is regularly
overtaken and replaced by another run that reports a higher score. This is
communicated (in papers, blog posts, tweets, etc.) as beating the existing
SOTA and achieving a new SOTA. Such pronouncements, however, are rarely
qualified with significance tests. We hope to take a small step towards
rectifying this.

Our study investigates significance testing among SOTA results in the context
of the MS MARCO document ranking leaderboard, but our findings and lessons
learned can perhaps be generalized to other leaderboards in IR and beyond.
Against the backdrop of recent debates about evaluation methodology in IR
(Ferrante et al., 2017; Sakai, 2020; Ferrante et al., 2021), we discover,
unsurprisingly, that there is no simple answer. Our findings can be summarized
as follows:

  1. (1)

The existing single-point metric for quantifying the “goodness” of a run on
the MS MARCO document ranking leaderboard (MRR@100) conflates important
differences in how one run can be “better” than another. Thus, the naive
approach of running standard significance tests on the existing metric may
lead to questionable results.

  2. (2)

To address this issue, we propose an evaluation framework that explicitly
tracks outcomes separately, which then permits meaningful aggregation and
significance testing. From a qualitative perspective, this framework reveals
many insights about differences that are obscured by the existing official
metric.

  3. (3)

Contributing to recent debates in the IR community on scale types and whether
certain statistical operations are mathematically permissible (Ferrante et
al., 2021), we find that in our framework, analysis in terms of expected
search length (ESL), which is a ratio scale, and mean reciprocal rank (MRR),
which is an ordinal scale, yield largely the same findings.

The contribution of this work is a novel evaluation framework that compares
putative SOTA submissions in a nuanced way that contributes to ongoing debates
in the information retrieval community about evaluation methodologies. We find
that runs can be “better” in different ways, but these “different ways” cannot
be reconciled without appealing to a user model of utility (which is presently
absent in the task definition).

It is worth emphasizing that in this paper, we are asking a very narrow
question about entries on a leaderboard and significance testing with respect
to a clearly defined metric (that determines the ranking on the leaderboard).
There are a number of questions that are outside the scope of inquiry, for
example: Is the new SOTA technique practically deployable (e.g., considering
inference latencies, model size, etc.)? Does the model train efficiently? Is
the improvement in the SOTA meaningful from a user perspective (i.e., does it
improve the user experience)? Might the new SOTA technique encode some bias
that would be a cause for concern? And, no doubt, many more questions. While
these are all important considerations, they raise orthogonal issues that we
do not tackle here. Nevertheless, we show that even for such a narrowly framed
question, there is still quite a bit of nuance that is missing in the current
discourse.

##  2\. Background and Related Work

The MS MARCO dataset was originally released in 2016 with the aim of helping
academic researchers explore information access in the large-data regime,
particularly in the context of models based on neural networks that were known
to be data hungry (Nguyen et al., 2016). Initially, the dataset was designed
to study question answering on web passages, but it was later adapted into
traditional ad hoc ranking tasks. Today, the document ranking and passage
ranking tasks host competitive leaderboards that attract much attention from
researchers around the world.

This paper focuses on the document ranking task, which is a standard ad hoc
retrieval task over a corpus of 3.2M web pages with URL, title, and body text.
The organizers have made available a training set with 367K queries and a
development set with 5193 queries; each query has exactly one relevance
judgment. There are 5793 evaluation (test) queries; relevance judgments for
these queries are withheld from the public. Scores on the evaluation queries
can only be obtained by a submission to the leaderboard. The official metric
is mean reciprocal rank at a cutoff of 100 (MRR@100).

Of the myriad metrics that have been proposed to evaluate retrieval systems,
there are those that make strong claims as to modeling user utility, such as
nDCG (Järvelin and Kekäläinen, 2002) and RBP (Moffat and Zobel, 2008), and
those that do not, say, precision at a fixed cutoff. Specifically in the
context of the MS MARCO document ranking task, reciprocal rank (RR) makes at
least some plausible claims about utility. At a high level, the metric says
that the user only cares about getting a single relevant document (not
unrealistic since MS MARCO models question answering “in the wild”), and that
utility drops off rapidly as a function of increasing ordinal rank. While the
functional form of this dropoff might be a matter of debate (similar
disagreements can be had about the functional form of the discount in nDCG),
there is strong empirical support for the claim in general, dating back well
over a decade. In the web search context, log analysis (e.g., (Agichtein et
al., 2006)) as well as eye-tracking experiments (e.g., (Joachims et al.,
2005)) have shown that user click probabilities and attention fall rapidly
with increasing ordinal rank in the retrieved results.

We believe that at least some of the ongoing controversies about evaluation
methodologies in information retrieval stems from confusion on whether a
metric is being used simply as a useful proxy for effectiveness (to aid in
quantifying model improvements) or is actually making a claim about utility.
Thus, in this paper, we are careful to separate the two, and are explicit when
making a claim about utility (and appealing to some user model).

The proximate motivation of this study is the recent work of Ferrante et al.
(2021), who argued that most IR metrics are not interval-scaled and suggested
that decades of IR research may be methodologically flawed and hence the
conclusions may not be reliable. We are not able to provide sufficient justice
to their detailed arguments due to space limitations, but the crux of the
matter in our context is that for MRR, intervals are not equi-spaced; that is,
a difference of $0.1$ (let’s say) “means” different things at different points
on the scale. As a contrast, the standard example of an interval scale is
temperature measured in Celsius, where “one degree” means exactly the same
thing (i.e., difference in temperature) everywhere. Recognizing that ongoing
debates in IR evaluation methodologies are by no means settled, and without
necessarily agreeing with the entirety of their arguments, we take the most
conservative position with respect to scale types and permissibility of
different operations and statistical analyses.

##  3\. Naive First Attempt

Figure 1. The leaderboard of the MS MARCO document ranking task, showing the
effectiveness of runs (MRR@100) on the held-out evaluation set over time.
“State of the art” (SOTA) runs are shown in red.

A summary of the MS MARCO document ranking leaderboard since its launch in
August 2020 (until mid-February, 2021) is shown in Figure 1, where each point
represents a run: the x-axis plots the date of submission and the y-axis plots
the official metric (MRR@100) reported on the leaderboard for the held-out
evaluation (test) set. Circles in red represent the (current and former)
state-of-the-art (SOTA) runs, i.e., a run that displaced a previous run at the
top of the leaderboard, beginning with the first submission that beat the
baselines provided by the organizers when the leaderboard first launched. In
our analysis, we specifically focused on these SOTA runs. Since the identities
of the runs (and for that matter, the actual techniques they used) are not
germane to our analysis, we simply denote them $R_{1}$ to $R_{10}$, arranged
chronologically. That is, $R_{1}$ is the first SOTA run, and $R_{10}$ is the
most recent.

If we wish to ask if one SOTA run is significantly better than another, an
obvious first attempt would be to run some standard statistical test over per-
query scores of the official metric (MRR@100). Among the myriad tests
available, three stand out:

  1. (1)

Wilcoxon rank sum test (WRS): a non-parametric test that requires samples to
be on an ordinal scale.

  2. (2)

Wilcoxon signed rank test (WSR): a non-parametric test that requires samples
to be on an interval scale.

  3. (3)

Student’s $t$-test: a parametric test that requires samples to be on an
interval scale.

As discussed in Ferrante et al. (2021), there has been quite a bit of
controversy (in IR and beyond) on what tests are permissible for what scale
types. Even taking the most stringent position, there is no doubt that
reciprocal rank is an ordinal scale: obtaining a document at rank one is
preferable to obtaining a document at rank two, rank three, $\ldots$ is more
preferable than not having the document in the ranked list at all. This holds
without making any commitments on the “distance” between any of these possible
outcomes (i.e., equi-spacing). Thus, WRS is unequivocally permissible. With
these caveats rendered explicit, let’s just run all the tests anyway.

Runs |  | WRS | WSR |  $t$-test
---|---|---|---|---
$A$ | $B$ | $\Delta$ |  $p$-value |  $p$-value |  $p$-value
$R_{1}$ | $R_{2}$ | 0.0027 | 0.029393 | 0.519758 | 0.722586
$R_{1}$ | $R_{3}$ | 0.0152 | 0.109672 | 0.053127 | 0.038240
$R_{1}$ | $R_{4}$ | 0.0179 | 0.000494 | 0.008780 | 0.013607
$R_{1}$ | $R_{5}$ | 0.0244 | 2.07E-05 | 0.000461 | 0.000785
$R_{1}$ | $R_{6}$ | 0.0301 | 1.16E-06 | 3.21E-05 | 3.77E-05
$R_{1}$ | $R_{7}$ | 0.0328 | 1.16E-07 | 1.00E-06 | 4.18E-06
$R_{1}$ | $R_{8}$ | 0.0360 | 2.81E-05 | 3.24E-06 | 2.50E-06
$R_{1}$ | $R_{9}$ | 0.0406 | 4.15E-06 | 9.06E-08 | 3.61E-08
$R_{1}$ | $R_{10}$ | 0.0516 | 7.85E-14 | 1.92E-12 | 7.57E-12
$R_{9}$ | $R_{10}$ | 0.0110 | 0.010663 | 0.109114 | 0.153225
Table 1. Results of running significance tests on SOTA runs: Wilcoxon rank sum
test (WRS), Wilcoxon signed rank test (WSR), and the Student’s $t$-test.
$R_{1}\ldots R_{10}$ are the SOTA runs, arranged chronologically.

The results of running all three significance tests on pairwise comparisons
between $R_{1}$ (the earliest SOTA run) and every other subsequent SOTA run
$\\{R_{2}\ldots R_{10}\\}$ on the evaluation set are shown in Table 1.
Additionally, we compare the two current top runs on the leaderboard, $R_{9}$
and $R_{10}$. The table reports the absolute differences in
effectiveness,111There is an important detail here worth mentioning: the
official evaluation script deliberately introduces a metric artifact designed
to thwart (simple) attempts at “reverse-engineering” the evaluation set. Thus,
the scores reported on the official leaderboard (and plotted in Figure 1) are
not accurate. This artifact has no impact on the leaderboard rankings, but
does impact significance testing. All our analyses are based the true MRR@100
scores, after the removal of this artifact. Thus, the absolute score
differences may not line up with public leaderboard results. along with the
raw $p$-values of the different tests, prior to the application of the
Bonferroni correction for multiple hypothesis testing.

We see that based on all three tests, the improvements from successive SOTA
runs $R_{2},R_{3},\ldots$ are not statistically significant until we get to
$R_{5}$; all subsequent runs thereafter appear to be significantly better
(even just focusing on WRS). The absolute difference in MRR@100 between
$R_{1}$ and $R_{5}$ is 2.4 points, which is surprisingly large. Independent of
the particulars of any evaluation, the general expectation would be that with
a large number of queries (over 5K in our case), small significant differences
(i.e., small effect sizes) should be detectable. That doesn’t appear to be the
case here. While this unexpected finding does not in and of itself demonstrate
that something is amiss, it would be natural to suspect Type II errors—that
is, our tests are not powerful enough to confidently reject the null
hypothesis. As it turns out, this is not the issue. We will demonstrate in the
context of our proposed evaluation framework: the fundamental issue is that
MRR@100 forces aggregation across outcomes that are not directly comparable
(with respect to some plausible user model).

##  4\. Outcomes Breakdown

This section presents our evaluation framework specifically tailored to the MS
MARCO document ranking task. We begin with a few (hopefully) uncontroversial
claims and from there build an approach to evaluation that explicitly avoids
conflating distinct outcomes in a single-point metric. Note that since the
official relevance judgments contain only one relevant document per query, the
position of that relevant document on the ranked list (or its absence) alone
determines the score (the metric, to be defined below) for that query; this
nicely sidesteps the challenges with different “recall bases” (Ferrante et
al., 2021), i.e., queries that have different numbers of relevant documents.

Consider two hypothetical submissions to the MS MARCO document ranking
leaderboard, runs $A$ and $B$, comprising ranked lists over a set of queries
$Q$. For each query $q\in Q$, there are logically the following distinct
outcomes that cover all possibilities:

  1. (1)

Neither run $A$ nor run $B$ returns the relevant document in the top $k$
ranking. In this case, both runs are equally “bad”.

  2. (2)

Run $A$ returns the relevant document in the top $k$ ranking, while run $B$
does not. Run $A$ is thus “better”. Vice versa with $B$ and $A$ swapped.

  3. (3)

Both run $A$ and $B$ return the relevant document in the top $k$ ranking, but
the document has a ranking closer to the front of the ranked list in run $B$
(i.e., lower ordinal rank). Run $B$ is thus “better”. Vice versa with $B$ and
$A$ swapped.

We believe that the above assertions hold regardless of how one might choose
to operationalize “bad” and “better”. However, to be more precise, let us
define a metric in terms of expected search length (ESL), which has a long
history in IR research dating back to the 1960s (Cooper, 1968). ESL quantifies
how long a user needs to search (more specifically, read the ranked list)
before obtaining a relevant document: A relevant document appearing at rank 1
gets a score of 1, rank 2 gets a score of 2, etc. all the way up to rank 100
(in our case). Thus, the lower the score, the better.

Consider a straightforward user model, that of a patient user who issues a
query and is willing to read 100 documents per query (at a constant pace) to
find the relevant document, and then gives up (if no relevant document is
found). It would be plausible to make the claim that ESL, with respect to this
user model, captures utility measured in user time.222Recognizing that we are
making a few simplifying assumptions such as constant document length and
fixed reading speed. More realism could be added by, for example, taking into
account a more accurate model of reading speed (Smucker and Clarke, 2012), but
these refinements are unlikely to change our overall analysis.

Runs |  | (1) | (2) |  | (3) |  |  | WSR |  $t$-test |  |  | WSR |  $t$-test
---|---|---|---|---|---|---|---|---|---|---|---|---|---
$A$ | $B$ | $\Delta$ | All |  $A$ wins |  $B$ wins | All |  $A$ ESL |  $B$ ESL |  $p$-value |  $p$-value |  $A$ RR |  $B$ RR |  $p$-value |  $p$-value
$R_{1}$ | $R_{2}$ | 0.0027 | 2% | 9% | 16% | 73% | 6.99 | 9.14 | 3.22E-12 | 1.60E-07 | 0.4812 | 0.4468 | 6.75E-05 | 2.66E-05
$R_{1}$ | $R_{3}$ | 0.0152 | 4% | 15% | 14% | 67% | 7.15 | 5.99 | 4.62E-06 | 1.34E-05 | 0.4857 | 0.5107 | 0.004032 | 0.003871
$R_{1}$ | $R_{4}$ | 0.0179 | 3% | 10% | 15% | 72% | 7.06 | 8.13 | 0.000249 | 0.004918 | 0.4838 | 0.4775 | 0.448624 | 0.368583
$R_{1}$ | $R_{5}$ | 0.0244 | 3% | 11% | 15% | 71% | 7.13 | 7.27 | 0.609428 | 0.813828 | 0.4833 | 0.4861 | 0.730242 | 0.765044
$R_{1}$ | $R_{6}$ | 0.0301 | 3% | 10% | 15% | 71% | 7.15 | 7.19 | 0.886547 | 0.381946 | 0.4829 | 0.4910 | 0.331145 | 0.364266
$R_{1}$ | $R_{7}$ | 0.0328 | 4% | 10% | 15% | 72% | 7.27 | 6.98 | 0.279188 | 0.022938 | 0.4799 | 0.4979 | 0.024974 | 0.020952
$R_{1}$ | $R_{8}$ | 0.0360 | 3% | 15% | 15% | 67% | 7.06 | 5.71 | 7.94E-08 | 8.68E-08 | 0.4829 | 0.5273 | 8.97E-07 | 7.59E-07
$R_{1}$ | $R_{9}$ | 0.0406 | 4% | 15% | 14% | 67% | 7.15 | 5.15 | 2.03E-16 | 4.33E-17 | 0.4857 | 0.5404 | 2.49E-10 | 1.85E-10
$R_{1}$ | $R_{10}$ | 0.0516 | 3% | 12% | 15% | 70% | 7.08 | 5.88 | 4.79E-06 | 1.46E-06 | 0.4829 | 0.5204 | 1.60E-05 | 1.10E-05
$R_{9}$ | $R_{10}$ | 0.0110 | 3% | 12% | 17% | 69% | 5.22 | 5.79 | 0.026583 | 0.006238 | 0.5376 | 0.5222 | 0.082654 | 0.077858
Table 2. Analysis of SOTA runs from the MS MARCO document ranking leaderboard,
broken into distinct outcomes: (1) neither run retrieves the relevant
document, (2) one run retrieves the relevant document but not the other, and
(3) both runs retrieve the relevant document.

It is clear that ESL as articulated here is on a ratio scale (which is by
definition also an interval scale). A difference of 1 “means the same thing”
at 2 ESL as it does at 99 ESL: in both cases, the relevant document appears
one rank closer to the top of the ranked list; from the utility perspective,
in both cases the user has saved the same amount of time.333Once again,
setting aside refinements that better model document length, reading speed,
etc. Furthermore, being a ratio scale, 3 ESL is 3$\times$ worse than 1 ESL,
and 100 ESL is 5$\times$ worse than 20 ESL (the analogy here is that ESL is
comparable measuring temperature in Kelvins). Against our user model, these
are also plausible statements to make with respect to utility (time): a
relevant document at rank 4 (4 ESL) costs the user twice as much utility
(time) as a relevant document at rank 2 (2 ESL).

So, for case (3) in the list of outcomes above when comparing run $A$ and run
$B$ for a specific query $q$, we have a good alignment between ESL and
utility, and we can make a number of meaningful claims. For example, the
following statements are permissible and indeed meaningful:

  * •

For $q_{1}$, run $A$ returns the relevant document at rank 2 and run $B$
returns the relevant document at rank 4. For $q_{2}$, run $A$ returns the
relevant document at rank 50 and run $B$ returns the relevant document at rank
52. In both cases, $A$ is better than $B$ by two “ESL units”, and even more
strongly, two units of utility (time).

  * •

For $q_{3}$, run $A$ returns the relevant document at rank 1 and run $B$
returns the relevant document at rank 4. For $q_{4}$, run $A$ returns the
relevant document at rank 10 and run $B$ returns the relevant document at rank
40. In both cases, $A$ is better than $B$ by 4$\times$, in terms of both “ESL
units” and units of utility.

The upshot here is that we can meaningfully average ESL values across a large
set of queries $Q$. Consider: for $q_{5}$, run $A$ returns the relevant
document at rank 1 and run $B$ returns the relevant document at rank 4, and
for $q_{6}$, run $A$ returns the relevant document at rank 5 and run $B$
returns the relevant document at rank 2. We can say that on the set of queries
comprised of $\\{q_{5},q_{6}\\}$ both runs are equally effective, with an
(arithmetic) mean ESL of 3. The arithmetic mean here encodes the assumption
that each query is equally important, which seems reasonable. Furthermore, it
is plausible to say that on these two queries, a user derives equal utility.

Note, critically, however, that this only applies to case (3) above, when both
runs contain the relevant document in their top $k$ lists. For case (2),
however, it is unclear how similar statements can be made. Consider: for
$q_{7}$, run $A$ returns the relevant document at rank 1 and run $B$ returns
the relevant document at rank 100, and for $q_{8}$, run $A$ doesn’t return the
relevant document at all and $B$ returns the relevant document at rank 99.
There is little that we can meaningfully say about the effectiveness of the
runs on a set of queries comprised of $\\{q_{7},q_{8}\\}$, both from the
perspective of ESL (what ESL would we assign to run $A$ for $q_{8}$?) and
utility. For the latter, we would need to quantify the cost of not finding a
relevant document relative to an ESL or time unit. There’s nothing in the
framework we’ve presented thus far that would shed light on this without a
more articulated user model (absent in the current task definition). Note that
the official metric MRR@100 does encode a specific utility difference between
a retrieved document at rank 100 and not retrieving the relevant document
($0.01$ and $0$, respectively), but justifying these values would require
appeal to user models and data (e.g., query logs) that are beyond the scope of
the leaderboard. We argue, instead, that the best way forward is to maintain
an explicit separation and breakdown of the different outcomes. In other
words, cases (2) and (3) represent apples and oranges, and it would be suspect
to average across them without first establishing some way to compare
different fruits.

##  5\. Application to MS MARCO

Let us now apply the evaluation framework proposed above to analyze the SOTA
runs on the MS MARCO document ranking leaderboard. In our analysis, we
compared each of $R_{2}\ldots R_{10}$ against $R_{1}$, the results of which
are shown in Table 2; we additionally compared the top two runs on the
leaderboard, $R_{10}$ against $R_{9}$. We show the percentage of queries that
fall under a particular outcome—case (1), (2), or (3)—as described in the
previous section. Case (2) is broken down into “$A$ wins”, where $A$ returns
the relevant document in its ranked list and $B$ doesn’t, and “$B$ wins”, the
opposite case. For rhetorical convenience, we will use “answered” and
“unanswered” for these cases. For case (3), we show the overall percentage, as
well as the mean ESL and reciprocal rank (RR) for all queries that fall under
that outcome. For ESL and RR, we show $p$-values from the application of the
Wilcoxon signed-rank test and the paired $t$-test. Note that since ESL is on
an interval (ratio) scale, these two tests are unequivocally permissible. For
RR, the application of the Wilcoxon signed-rank test and the paired $t$-test
is subjected to the potential objections raised by Ferrante et al. (2021)
regarding interval scales. In all cases, we report raw $p$-values, prior to
the application of the Bonferroni correction for multiple hypothesis testing.

This specific case study, and more generally our proposed evaluation
framework, reveals many interesting insights that are completely hidden if we
simply reported the means of per-query reciprocal ranks and ran significance
tests on them, as in Section 3. Focusing specifically on ESL, we highlight a
number of interesting observations below:

  * •

In the first row, comparing $R_{1}$ vs. $R_{2}$, we see the overall MRR@100
scores are quite close, but both runs obtain the scores in very different
ways. Looking at the case (3) breakdowns, we see that $R_{2}$ has a higher ESL
than $R_{1}$, and this difference is (highly) statistically significant. From
this perspective, $R_{2}$ is worse than $R_{1}$ (relevant results appearing
later in the ranked lists). However, we see that $R_{2}$ answered far more
queries that went unanswered in $R_{1}$ than the other way around.

A similar observation can be made in the comparison between $R_{1}$ and
$R_{4}$. When focusing only on ranking, case (3), $R_{4}$ is significantly
worse than $R_{1}$, but $R_{4}$ compensates by answering more queries that
went unanswered in $R_{1}$, leading to a higher score in terms of MRR@100.

  * •

Consider the comparison between $R_{1}$ and $R_{3}$ (the second row): contrary
to the examples discussed above, we see that $R_{3}$ significantly improves
ranking, case (3), but has slightly more unanswered queries compared to the
baseline. This also leads to an overall improvement in terms of MRR@100.

  * •

Consider the comparison between $R_{1}$ and $R_{8}$. Here, looking at the
prevalence of case (2): there are equal percentages of cases where the query
was answered by one run but not the other. However, looking at case (3), we
see that $R_{8}$ obtains a statistically significant reduction in ESL. That
is, $R_{8}$ is better than $R_{1}$ not because it obtained more relevant
documents, but rather that it ranked the relevant documents more highly.

  * •

Another interesting observation relates to $R_{1},R_{5},R_{6},R_{7}$, which
are runs from the same team. From the rows comparing $R_{1}$ to
$\\{R_{5},R_{6},R_{7}\\}$, we see that in terms of case (3), differences in
ESL are not statistically significant. That is, the runs are comparable when
it comes to ranking documents that appear in the top 100. The differences in
MRR@100 come primarily from case (2), where $\\{R_{5},R_{6},R_{7}\\}$ have
fewer unanswered queries on balance.

  * •

Looking at the current top two runs on the leaderboard, $R_{10}$ and $R_{9}$,
we see that the second-best run actually does a better job ranking, i.e., case
(3), than the top run. The latter wins by virtue of it answering more queries.

Examining the differences between ESL and RR, our analysis shows that they
largely agree, with the exception of $R_{1}$ vs. $R_{3}$ and $R_{1}$ vs.
$R_{4}$. It seems that differences in ESL are relatively larger than
differences in RR, which could explain these cases. Note that two runs with
the same ESL can have very different MRRs. For example, consider the case of a
run retrieving relevant documents at ranks 1 and 9, and a second run
retrieving relevant documents at ranks 4 and 6. Both runs would have the same
ESL, but very different MRRs, 0.556 and 0.208, respectively. There may be a
tempting intuitive interpretation of MRR as the reciprocal of the mean
position at which a relevant document appears, but this interpretation is not
accurate.

With respect to the ongoing IR debate on scale types, the conclusion seems to
be that, at least in our proposed framework, it doesn’t matter much. That is,
even adopting the most cautious position advocated by Ferrante et al. (2021),
analysis in terms of ESL (a ratio scale) and MRR (not an interval scale) does
not lead to markedly different conclusions. Given this, there may be reasons
to prefer RR over ESL, since the former captures a more realistic user model;
it is highly unlikely that user attention can be sustained across the
examination of 100 results, as suggested by ESL. Here, the theory of IR
evaluation appears to diverge from its actual practice on large-scale, real-
world datasets.

Back to the original question that we set out to answer: Is “this” SOTA run
better than “that” SOTA run? We might say that run $B$ is better than run $A$
if run $B$ wins in terms of case (2) and has a smaller ESL for case (3), or
alternatively, greater MRR. We might further claim that run $B$ is
significantly better than run $A$ if the improvements in both outcomes are
statistically significant: for case (3), significant testing as we have
performed above, and for case (2), perhaps the binomial test.

Alternatively, we might adopt a less stringent definition, somewhat akin to
the Hippocratic Oath (i.e., “do no harm”): a run can be considered
significantly better if it significantly increases the fraction of answered
questions without significantly increasing the ESL or that it significantly
decreases ESL without significantly increasing the fraction of unanswered
questions. The advantage of this approach is that it provides two concrete
facets of “goodness” that researchers can independently tackle while still
being amenable to a linear sort order for populating a leaderboard.

##  6\. Concluding Thoughts

One attraction of leaderboards is that they define a concrete metric to
optimize, and if that metric is well-defined and meaningful, it enables the
community to make rapid progress on a problem. This work focuses on one
important yet under-explored aspect of intrinsic validity: We show that naive
significance testing applied to MRR@100 obscures many potential insights, and
that our proposed framework provides a more nuanced analysis.

How might we generalize our framework to encompass other retrieval tasks and
possibly beyond? We see at least one challenge that limits the broader
applicability of our approach: it critically depends on having only one
relevant document per query, since this property is necessary to separate the
outcomes. Otherwise, it is unclear how we would compare ranked lists that
retrieve different numbers of relevant documents. While this restriction would
not be unrealistic for some tasks, such as question answering, there clearly
needs to be more work before our approach can be generalized.

## References

  * (1)
  * Agichtein et al. (2006) Eugene Agichtein, Eric Brill, Susan Dumais, and Robert Ragno. 2006.  Learning User Interaction Models for Predicting Web Search Result Preferences. In _Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006)_. Seattle, Washington, 3–10.
  * Cooper (1968) William S. Cooper. 1968\.  Expected Search Length: A Single Measure of Retrieval Effectiveness Based on the Weak Ordering Action of Retrieval Systems.  _Journal of the American Society for Information Science_ 19, 1 (1968), 31–40.
  * Ferrante et al. (2021) Marco Ferrante, Nicola Ferro, and Norbert Fuhr. 2021\.  Towards Meaningful Statements in IR Evaluation. Mapping Evaluation Measures to Interval Scales.  _arXiv preprint arXiv:2101.02668_ (2021).
  * Ferrante et al. (2017) Marco Ferrante, Nicola Ferro, and Silvia Pontarollo. 2017\.  Are IR Evaluation Measures on an Interval Scale?. In _Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval_. 67–74.
  * Järvelin and Kekäläinen (2002) Kalervo Järvelin and Jaana Kekäläinen. 2002\.  Cumulative Gain-Based Evaluation of IR Techniques.  _ACM Transactions on Information Systems_ 20, 4 (2002), 422–446.
  * Joachims et al. (2005) Thorsten Joachims, Laura Granka, Bing Pang, Helene Hembrooke, and Geri Gay. 2005.  Accurately Interpreting Clickthrough Data as Implicit Feedback. In _Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005)_. Salvador, Brazil, 154–161.
  * Moffat and Zobel (2008) Alistair Moffat and Justin Zobel. 2008.  Rank-Biased Precision for Measurement of Retrieval Effectiveness.  _ACM Transactions on Information Systems_ 27, 1 (2008), Article 2.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.  _arXiv:1611.09268v1_ (2016).
  * Sakai (2020) Tetsuya Sakai. 2020\.  On Fuhr’s Guideline for IR Evaluation. In _SIGIR Forum_ , Vol. 54. p14.
  * Smucker and Clarke (2012) Mark D. Smucker and Charles L. A. Clarke. 2012.  Time-Based Calibration of Effectiveness Measures. In _Proceedings of the 35th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2012)_. Portland, Oregon, 95–104.