Axial Bianchi I meets drifting extragalactic sources

Thomas Schücker111 Aix Marseille Univ, Université de Toulon, CNRS, CPT,
Marseille, France
<EMAIL_ADDRESS>
Abstract

We recompute the drift of comoving sources in axial Bianchi I universes and
correct the same calculation by Quercellini et al. from 2009. The correction
is pertinent with respect to the Gaia catalogue of measured quasar drifts,
that beg to be fitted.

Key-Words: cosmological parameters

##  1 Introduction

As early as 1933, Georges Lemaître [1], following a recommandation by Albert
Einstein, considers (tri-axial) Bianchi I universes without using this name.
They generalize flat Robertson-Walker universes by allowing three different
scale factors in three orthogonal directions:

$\displaystyle\hbox{\rm{d}}\tau^{2}=\hbox{\rm{d}}t^{2}-a(t)^{2}\hbox{\rm{d}}x^{2}-b(t)^{2}\hbox{\rm{d}}y^{2}-c(t)^{2}\,\hbox{\rm{d}}z^{2},\quad
a(t),\,b(t),\,c(t)>0$ (1)

These universes are homogeneous and anisotropic with a 3-dimensional Abelian
isometry group. Its Lie algebra is the first in the list of Luigi Bianchi’s
famous classification from 1898 [2] of the real, 3-dimensional Lie algebras.
In the same work he also

  * •

exhibits 3-dimensional Riemannian manifolds on which these Lie algebras act
transitively as Killing vectors,

  * •

he exhibits 3-dimensional Riemannian manifolds on which 4-dimensional Lie
algebras act in the same way and shows that they all contain the 3-dimensional
Lie algebras from his list as ideals,

  * •

and he shows that 5-dimensional Lie algebras cannot act in this way.

For Bianchi I this manifold is simply the 3-dimensional Eulidean space and the
manifolds for the other Lie algebras of type Bianchi II - IX underlie the
corresponding higher Bianchi universes, whose study began in the late sixties
[3]-[5].

The universes with four Killing vectors are special cases, which are called
locally rotationally symmetric (L.R.S.) or axial.

The proper time element of the axial Bianchi I universe is obtained by
choosing two equal scale factors, e.g. $b=a$. Then there is a fourth Killing
vector, $x\partial_{y}-y\partial_{x}$, generating rotations around the
$z$-axis. If all three scale factors are equal, we are back at flat Robertson-
Walker universes with the maximally symmetric Euclidean 3-space and three
translations, three rotations.

The purpose of this work is an ab initio calculation of the drift of a
comoving source as seen by a not necessarily comoving observer, whose peculiar
velocity however is much smaller than the speed of light. We carefully
separate the three parts of the calculation:

kinematics:

position of the source on the celestial sphere as a function of the observer’s
cosmic time and the redshift of the source,

dynamics:

Einstein’s equation with cosmological constant and comoving dust,

approximation of small anisotropy:

$c(t)=a(t)[1+{\textstyle\frac{3}{2}}\eta(t)],\ \eta(t)\ll 1$.

Note that only axial Bianchi I and IX universes fulfill the three conditions:
(i) next-to-maximal symmetry, (ii) compatibility with Einstein’s equation
sourced by comoving dust and (iii) admitting maximal symmetry in the limit
$\eta\rightarrow 0$.

In 2014 Darling [6] used tri-axial Bianchi I universes to fit the drift of 429
extra-galactic radio sources measured by Titov & Lambert [7] using Very Long
Baseline Interferometry.

Today, thanks to the Gaia mission, we have hundreds of thousands of Quasars
with drift and redshift measurements ($z$ up to 4.7) begging to be fitted.

##  2 The path of a light ray

The axial Bianchi I metric can be written

$\displaystyle\hbox{\rm{d}}\tau^{2}=\hbox{\rm{d}}t^{2}-a(t)^{2}\left[\hbox{\rm{d}}x^{2}+\hbox{\rm{d}}y^{2}\right]-c(t)^{2}\,\hbox{\rm{d}}z^{2},\quad
a(t),\,c(t)>0.$ (2)

We follow notations and use results of [8]. Consider a geodesic $x^{\mu}(p)$
and denote the derivation with respect to the affine parameter $p$ by
$\dot{}\vcentcolon=\hbox{\rm{d}}/\hbox{\rm{d}}p$. The four Killing vectors
$\partial x,\,\partial y,\,\partial z$ and $x\partial y-y\partial x$ yield
four conserved quantities by Emmy Noether’s theorem:

$\displaystyle A\vcentcolon=a(t(p))^{2}\dot{x},\ B\vcentcolon=a^{2}\dot{y},\
C\vcentcolon=c^{2}\dot{z}\quad\text{and}\quad Ay-Bx=0.$ (3)

The last equation implies that if $A$ and $B$ both vanish the geodesic is a
straight line in the $\pm\,z$-direction. Otherwise the geodesic remains in the
plane defined by the $z$-direction and the direction of the initial
3-velocity.

Let us suppose the geodesic is light-like. Then from
$\dot{x}^{\mu}g_{\mu\nu}\dot{x}^{\nu}=0$ we obtain

$\displaystyle\dot{t}=\,\frac{1}{W}\,\quad\text{with}\quad
W\vcentcolon=\left(\frac{A^{2}+B^{2}}{a(t)^{2}}\,+\,\frac{C^{2}}{c(t)^{2}}\right)^{-1/2}.$
(4)

Eliminating the affine parameter $p$ in favour of the cosmic time $t$ and
using ${}^{\prime}\vcentcolon=\hbox{\rm{d}}/\hbox{\rm{d}}t$, the light-like
geodesic must satisfy the three first order equations:

$\displaystyle\vec{x}^{\prime}=\,\frac{\dot{\vec{x}}}{\dot{t}}\,=\,W\begin{pmatrix}\,\frac{A}{a^{2}}\,\\\\[1.42262pt]
\,\frac{B}{a^{2}}\,\\\\[1.42262pt] \,\frac{C}{c^{2}}\,\end{pmatrix}.$ (5)

Let us fix the Noether constants $A,\,B$ and $C$ in terms of the initial
velocity of a photon emitted by a comoving source at time $t_{e}$ and position
$\vec{x}_{e}\vcentcolon=\vec{x}(t_{e})$. To simplify notations we put
$\vec{x}_{e}=0$.

Let us say that the photon arrives today $t=t_{0}$ in our telescope at
position $\vec{x}_{0}\vcentcolon=\vec{x}(t_{0})$. Then we have:

$\displaystyle x_{0}=\int_{t_{e}}^{t_{0}}\,\frac{A}{a^{2}}\,W,\quad
y_{0}=\int_{t_{e}}^{t_{0}}\,\frac{B}{a^{2}}\,W,\quad
z_{0}=\int_{t_{e}}^{t_{0}}\,\frac{C}{c^{2}}\,W.$ (6)

In our conventions the speed of light is unity; proper time $\tau$, the
coordinate time $t$ and the affine parameter $p$ have units of seconds, the
comoving coordinates $x,\,y,\,z$ are dimensionless and the scale factors $a$
and $c$ and the Noether constants $A,\,B$ and $C$ carry seconds. Then $W$ is
dimensionless. Equation (5) remains unchanged under a simultaneous rescaling
of the three Noether constants. Therefore we may choose
$N\vcentcolon=\sqrt{A^{2}+B^{2}+C^{2}}$ to be any positive constant. Through a
rescaling of the coordinates $x,\,y,\,z$ we may achieve $a_{0}=c_{0},$ with
$a_{0}\vcentcolon=a(t_{0}),\dots$

We also learn from equations (5) or (6) that in the Bianchi I metric, the
apparent position of comoving sources in the sky changes with time, ‘cosmic
parallax, apparent motion, proper motion or drift’ (unless they lie in the
$xy$-plane, $C=0$, or on the $z$-axis, $A=B=0$).

##  3 A second, infinitesimally close light ray

Consider now a second photon emitted a short time $T$ later from the same
source and arriving in our telescope a time $T_{D}$ after the first. (The
subscript $\cdot_{D}$ stands for Doppler.) The following calculations are
relevant in two physical situations:

  * •

$T$ is an atomic period at emission. Then $T_{D}$ is the Doppler shifted
atomic period observed here and today and we define the redshift today

$\displaystyle\underline{z}_{0}\vcentcolon=\,\frac{T_{D}-T}{T}\,.$ (7)

We underline the redshift $\underline{z}$ to avoid confusion with the
coordinate $z$.

If we pretend to know the metric and it has nice properties, then the redshift
allows us to compute the time of flight of the photon.

  * •

$T_{D}$ is a few years and we observe a change in the direction of the
arriving photons. Of course, a change in direction can only be measured with
respect to another direction that supposedly does not change. Axial Bianchi I
universes have indeed the luxury of such fixed directions: the $z$-axis and
the $xy$-plane.

Let us denote the second geodesic by $\tilde{\vec{x}}(t)$ with $t$ varying
from $t_{\tilde{e}}\vcentcolon=t_{e}+T$ to
$t_{\tilde{0}}\vcentcolon=t_{0}+T_{D}$.

In both situations the second photon is sent out at

$\displaystyle\tilde{\vec{x}}(t_{\tilde{e}})={\vec{x}}(t_{e})=\vcentcolon\vec{x}_{e}=0,$
(8)

because we suppose the source to be comoving. For the observer, on the other
hand, we allow for a small peculiar velocity today

$\displaystyle\vec{V}=\vcentcolon a_{0}\,\vec{v}\,,\quad\quad V\ll 1.$ (9)

Then the photon arrives in her telescope at position
$\vec{x}_{0}+T_{D}\,\vec{v}$:

$\displaystyle\tilde{\vec{x}}(t_{\tilde{0}})=\vcentcolon\tilde{\vec{x}}_{\tilde{0}}={\vec{x}}(t_{0})+T_{D}\,\vec{v}=\vec{x}_{0}+T_{D}\,\vec{v}.$
(10)

We suppose that $T$ is small with respect to the ‘time of flight’, $T\ll
t_{0}-t_{e}$; so small that we may keep only linear terms in $T$ and $T_{D}$.
Also the peculiar velocity is small enough so that we may keep only linear
terms in $V$. In particular, we neglect the dilation of the observer’s proper
time with respect to cosmic time $t$.

We may use a rotation in the $xy$-plane to bring the peculiar velocity into
the form

$\displaystyle\vec{v}=\begin{pmatrix}v_{x}\\\ 0\\\ v_{z}\end{pmatrix}.$ (11)

Our task is to slightly change the initial direction of the second geodesic
$\tilde{\vec{x}}(t)$ coded in the Noether constants $\tilde{A},\,\tilde{B}$
and $\tilde{C}$, in such a manner that
$\tilde{\vec{x}}_{\tilde{0}}=\vec{x}_{0}+T_{D}\,\vec{v}$. To this end we set:

$\displaystyle\tilde{A}=A+\alpha N,\quad\tilde{B}=B+\beta
N,\quad\tilde{C}=C+\gamma N,\quad\alpha,\,\beta,\,\gamma\sim T/(t_{0}-t_{e}).$
(12)

We may choose $\tilde{N}=N$ implying

$\displaystyle A\alpha+B\beta+C\gamma\sim 0.$ (13)

The second trajectory is computed as the first:

$\displaystyle\tilde{x}_{\tilde{0}}=\int_{t_{e}+T}^{t_{0}+T_{D}}\,\frac{\tilde{A}}{a^{2}}\,\tilde{W},\quad\tilde{y}_{\tilde{0}}=\int_{t_{e}+T}^{t_{0}+T_{D}}\,\frac{\tilde{B}}{a^{2}}\,\tilde{W},\quad\tilde{z}_{\tilde{0}}=\int_{t_{e}+T}^{t_{0}+T_{D}}\,\frac{\tilde{C}}{c^{2}}\,\tilde{W},$
(14)

with

$\displaystyle\tilde{W}\\!\\!\\!$ $\displaystyle\vcentcolon=$
$\displaystyle\\!\\!\\!\left(\frac{\tilde{A}^{2}+\tilde{B}^{2}}{a^{2}}\,+\,\frac{\tilde{C}^{2}}{c^{2}}\right)^{-1/2}.$
(15)

We linearize $\tilde{W}$,

$\displaystyle\tilde{W}\\!\\!\\!$ $\displaystyle\sim$
$\displaystyle\\!\\!\\!W\left[1-W^{2}N\left(\frac{A\alpha+B\beta}{a^{2}}\,+\,\frac{C\gamma}{c^{2}}\right)\right]$
(16) $\displaystyle\sim$ $\displaystyle
W\left[1+W^{2}NC\gamma\left(\frac{1}{a^{2}}\,-\,\frac{1}{c^{2}}\right)\right]\sim
W\left[1+\,\frac{N}{C}\,\gamma\left(N^{2}\,\frac{W^{2}}{a^{2}}\,-1\right)\right]$
(17) $\displaystyle\sim$ $\displaystyle
W\\!\left[1-W^{2}N(A\alpha+B\beta)\\!\\!\left(\frac{1}{a^{2}}\,-\,\frac{1}{c^{2}}\right)\\!\right]\sim\\!W\left[1-\,\frac{NC}{A^{2}+B^{2}}\,\gamma\\!\left(N^{2}\,\frac{W^{2}}{c^{2}}\,-1\right)\\!\right],$
(18)

where we have used successively the two identities:

$\displaystyle
W^{2}\left(\frac{1}{a^{2}}\,-\,\frac{1}{c^{2}}\right)=\,\frac{N^{2}}{C^{2}}\,\frac{W^{2}}{a^{2}}\,-\,\frac{1}{C^{2}}\,=-\,\frac{N^{2}}{A^{2}+B^{2}}\,\frac{W^{2}}{c^{2}}\,+\,\frac{1}{A^{2}+B^{2}}\,.$
(19)

Then we have to first order in $T/(t_{0}-t_{e})$:

$\displaystyle\tilde{x}_{\tilde{0}}\\!\\!\\!$ $\displaystyle\sim$
$\displaystyle\\!\\!\\!\int_{t_{e}+T}^{t_{0}+T_{D}}\left(\frac{\tilde{A}\tilde{W}}{a^{2}}\,+\,\frac{NW}{a^{2}}\,\alpha\right)$
(20) $\displaystyle\sim$
$\displaystyle\\!\\!\\!\,\frac{AW_{0}}{a^{2}_{0}}\,T_{D}-\,\frac{AW_{e}}{a^{2}_{e}}\,T+\\!\\!\int_{t_{e}}^{t_{0}}\left\\{\frac{AW}{a^{2}}\,-\frac{AW}{a^{2}}\,\,\frac{NC}{A^{2}+B^{2}}\,\gamma\\!\left(\frac{N^{2}W^{2}}{c^{2}}\,-1\right)+\,\frac{NW}{a^{2}}\,\alpha\right\\}$
(21) $\displaystyle\sim$
$\displaystyle\\!\\!\\!x_{0}+A\,F\,T_{D}+\left(\frac{AC}{A^{2}+B^{2}}\,\gamma+\alpha\right)I-\,\frac{AC}{A^{2}+B^{2}}\,\gamma
J.$ (22)

In (21) we have used the second member of (18) and in (22) we have introduced
the short hands,

$\displaystyle
F\vcentcolon=\,\frac{W_{0}}{a_{0}^{2}}\,-\,\frac{W_{e}}{a_{e}^{2}}\,\frac{T}{T_{D}}\,,\quad
F_{c}\vcentcolon=\,\frac{W_{0}}{c_{0}^{2}}\,-\,\frac{W_{e}}{c_{e}^{2}}\,\frac{T}{T_{D}}\,,\quad
I\vcentcolon=N\int_{t_{e}}^{t_{0}}\,\frac{W}{a^{2}}\,,\quad
J\vcentcolon=N^{3}\int_{t_{e}}^{t_{0}}\,\frac{W^{3}}{a^{2}c^{2}}\,.$ (23)

Since $\tilde{x}_{\tilde{0}}=x_{0}+v_{x}T_{D}$ we have:

$\displaystyle v_{x}T_{D}$ $\displaystyle\sim$ $\displaystyle
A\,F\,T_{D}+\alpha I+\,\frac{AC}{A^{2}+B^{2}}\,\gamma(I-J).$ (24)

Similarily for $\tilde{y}_{\tilde{0}}=y_{0}$ and
$\tilde{z}_{\tilde{0}}=x_{0}+v_{z}T_{D}$:

$\displaystyle 0$ $\displaystyle\sim$ $\displaystyle B\,F\,T_{D}+\beta
I+\,\frac{BC}{A^{2}+B^{2}}\,\gamma(I-J),$ (25) $\displaystyle v_{z}T_{D}$
$\displaystyle\sim$ $\displaystyle C\,F_{c}\,T_{D}+\gamma J.$ (26)

To derive equation (26) we used the second member of (17). Let us take the
following linear combination of the last three equations,
$A$(24)+$B$(25)+$C$(26):

$\displaystyle(Av_{x}+Cv_{z})\,T_{D}\sim(A^{2}+B^{2})\,F\,T_{D}+C^{2}\,F_{c}\,T_{D}=\,\frac{T_{D}}{W_{0}}\,-\,\frac{T}{W_{e}}\,.$
(27)

From it we obtain the direction-dependent redshift today,

$\displaystyle\underline{z}_{0}+1\vcentcolon=\,\frac{T_{D}}{T}\,\sim\,\frac{1}{W_{e}}\,\frac{1}{N/a_{0}-(Av_{x}+Cv_{z})}\,.$
(28)

We will also need the corrections $\alpha,\,\beta,\,\gamma$ to the second
photon’s initial direction. To this end let us consider the linear combination
$B$(24)$-A$(25):

$\displaystyle\alpha\sim\,\frac{A}{B}\,\beta+\,\frac{1}{I}\,v_{x}\,T_{D}.$
(29)

Using equations (13) and (26) we obtain,

$\displaystyle\alpha$ $\displaystyle\sim$
$\displaystyle\frac{1}{A^{2}+B^{2}}\,\frac{1}{J}\left[AC^{2}F_{c}+B^{2}\,\frac{J}{I}\,v_{x}-AC\,v_{z}\right]T_{D},$
(30) $\displaystyle\beta$ $\displaystyle\sim$
$\displaystyle\frac{B}{A^{2}+B^{2}}\,\frac{1}{J}\left[C^{2}F_{c}+A\,\frac{J}{I}\,v_{x}-C\,v_{z}\right]T_{D},$
(31) $\displaystyle\gamma$ $\displaystyle\sim$ $\displaystyle\hskip
48.36967pt\frac{1}{J}\left[-C\,F_{c}+v_{z}\right]T_{D}.$ (32)

##  4 Arriving angles

Our first task is to compute the angle $\theta$ between the $z$-axis and the
velocity $\vec{x}^{\prime}_{0}$ of the first photon. To this end we use the
negative of the spatial part of the metric (2). Note that the velocity vector
$\vec{x}^{\prime}$ is a unit vector with respect to this 3-metric, because we
have set the speed of light to one. If we denote by

$\displaystyle\vec{u}_{z}\vcentcolon=\begin{pmatrix}0\\\ 0\\\
\frac{1}{c}\end{pmatrix}$ (33)

the unit vector in the $z$-direction, we have

$\displaystyle\cos\theta=\vec{x}^{\prime}_{0}\cdot\vec{u}_{z0}=W_{0}\,\frac{C}{c_{0}^{2}}\,\,c_{0}^{2}\,\,\frac{1}{c_{0}}\,=\,\frac{CW_{0}}{c_{0}}\,=\,\frac{C}{N}\,.$
(34)

Our second task is to compute the same angle $\tilde{\theta}$ for the second
photon at its arrival time $t_{\tilde{0}}=t_{0}+T_{D}$:

$\displaystyle\cos\tilde{\theta}={\tilde{\vec{x}}}^{\prime}_{\tilde{0}}\cdot\vec{u}_{z\tilde{0}}=\,\frac{\tilde{C}\tilde{W}_{\tilde{0}}}{c_{\tilde{0}}}\,.$
(35)

From the second expression (18) we obtain,

$\displaystyle\,\frac{\tilde{C}\tilde{W}}{c}\,\sim\,\frac{CW}{c}\left[1+\,\frac{N^{2}}{A^{2}+B^{2}}\,\frac{N}{C}\left(1-\left\\{\frac{CW}{c}\right\\}^{2}\right)\right],$
(36)

and

$\displaystyle\,\frac{\tilde{C}\tilde{W}_{\tilde{0}}}{c_{\tilde{0}}}\,\sim\,\frac{CW_{\tilde{0}}}{c_{\tilde{0}}}\left[1+\,\frac{N^{2}}{A^{2}+B^{2}}\,\frac{N}{C}\left(1-\left\\{\frac{CW_{0}}{c_{0}}\right\\}^{2}\gamma\right)\right]\sim\,\frac{CW_{\tilde{0}}}{c_{\tilde{0}}}\left[1+\,\frac{N}{C}\,\gamma\right].$
(37)

We Taylor develop the leading term $w(t)\vcentcolon=C\,W(t)/c(t)$ around
$t_{0}$ viz
$w(t_{0}+T_{D})\sim w_{0}+T_{D}\,w^{\prime}_{0}$:

$\displaystyle w$ $\displaystyle=$
$\displaystyle\left(\frac{A^{2}+B^{2}}{C^{2}}\,\frac{c^{2}}{a^{2}}+1\right)^{-1/2}\,,$
(38) $\displaystyle w^{\prime}$ $\displaystyle=$
$\displaystyle\,\frac{A^{2}+B^{2}}{C^{2}}\,\frac{c^{2}}{a^{2}}\,(H-H_{c})\left(\frac{A^{2}+B^{2}}{C^{2}}\,\frac{c^{2}}{a^{2}}+1\right)^{-3/2},\quad
H\vcentcolon=\,\frac{a^{\prime}}{a}\,\quad
H_{c}\vcentcolon=\,\frac{c^{\prime}}{c}\,,$ (39) $\displaystyle w_{0}$
$\displaystyle=$ $\displaystyle\cos\theta,\quad\quad w_{0}^{\prime}\ =\
\sin^{2}\theta\,\cos\theta\,(H_{0}-H_{c0}).$ (40)

Therefore

$\displaystyle
W_{\tilde{0}}\,\frac{C}{c_{\tilde{0}}}\sim\cos\theta\,[1+\sin^{2}\theta\,(H_{0}-H_{c0})\,T_{D}],$
(41)

and

$\displaystyle\cos\tilde{\theta}\sim\cos\theta\left[1+\sin^{2}\theta\,(H_{0}-H_{c0})\,T_{D}+\,\frac{\gamma}{\cos\theta}\right].$
(42)

The apparent angular drift of the comoving source
$\delta\theta\vcentcolon=\tilde{\theta}-\theta$ then takes the form

$\displaystyle\delta\theta\sim-\sin\theta\,\cos\theta\,(H_{0}-H_{c0})\,T_{D}-\,\frac{\gamma}{\sin\theta}\,.$
(43)

We need a second angle to localize the direction of the incoming photons on
the celestial sphere. Following standard conventions we choose the angle
$\varphi$ in the $xy$-plane between the orthogonal projection of
$\vec{x}^{\prime}$ into this plane and the positive $x$-axis. Denote this
projection by $\vec{x}^{\prime}_{\perp}$ and the unit vector in the positive
$x$-direction by $\vec{u}_{x}$,

$\displaystyle\vec{x}^{\prime}_{\perp}\vcentcolon=\,W\begin{pmatrix}\,\frac{A}{a^{2}}\,\\\\[1.42262pt]
\,\frac{B}{a^{2}}\,\\\\[1.42262pt]
0\end{pmatrix},\quad\vec{u}_{x}\vcentcolon=\begin{pmatrix}\frac{1}{a}\\\ 0\\\
0\end{pmatrix}.$ (44)

Then the angle $\varphi$ today is given by

$\displaystyle\cos\varphi=\,\frac{\vec{x}^{\prime}_{\perp
0}\cdot\vec{u}_{x0}}{\sqrt{\vec{x}^{\prime}_{\perp
0}\cdot\vec{x}^{\prime}_{\perp 0}}}\,=\,\frac{A}{\sqrt{A^{2}+B^{2}}}\,,$ (45)

and we retrieve the usual relations of spherical coordinates:

$\displaystyle A$ $\displaystyle=$ $\displaystyle N\cos\varphi\,\sin\theta,$
(46) $\displaystyle B$ $\displaystyle=$ $\displaystyle
N\sin\varphi\,\sin\theta,$ (47) $\displaystyle C$ $\displaystyle=$
$\displaystyle N\quad\quad\,\cos\theta.$ (48)

Note that if $A=B=0$ the first photon moves along the $z$-axis,
$\sin\theta=0$. Then the projection of the photon’s velocity onto the
$xy$-plane vanishes and the angle $\varphi$ is not defined.

Our last task is to compute the same angle $\tilde{\varphi}$ for the second
photon at its arrival time $t_{\tilde{0}}=t_{0}+T_{D}$:

$\displaystyle\cos\tilde{\varphi}=\,\frac{\tilde{\vec{x}}^{\prime}_{\perp\tilde{0}}\cdot\vec{u}_{x\tilde{0}}}{\sqrt{\tilde{\vec{x}}^{\prime}_{\perp\tilde{0}}\cdot\tilde{\vec{x}}^{\prime}_{\perp\tilde{0}}}}\,=\,\frac{\tilde{A}}{\sqrt{\tilde{A}^{2}+\tilde{B}^{2}}}\,\sim\,\frac{\,\frac{A}{N}\,+\alpha}{\sqrt{\,\frac{A^{2}+B^{2}}{N^{2}}\,-2\,\frac{C}{N}\,\gamma}}\,.$
(49)

At this point we must resist a further linearization in $\alpha$ and $\gamma$.
We will understand this necessity in the next section.

##  5 Linearizing the scale factors

We known that the anisotropy of the universe cannot be large and we will
concentrate on axial Bianchi I universes close to an underlying maximally
symmetric one with only one scale factor $a_{F}(t)$, $\cdot_{F}$ for Friedman.
It is convenient to parameterize the small deviation $\eta(t)$ in the
following way:

$\displaystyle a=\vcentcolon a_{F}\,[1-{\textstyle\frac{1}{2}}\eta],\quad
c=\vcentcolon a_{F}\,[1+\eta],\quad\quad\quad\eta_{0}=0,\quad H\sim
H_{F}-{\textstyle\frac{1}{2}}\eta^{\prime},\quad H_{c}\sim
H_{F}+\eta^{\prime},$ (50)

where we indicate the linearization in $\eta$ and $V$ (neglecting cross-terms
$\eta V$) also by $\sim$. Then we have

$\displaystyle W$ $\displaystyle\sim$
$\displaystyle\,\frac{a_{F}}{N}\left[1-{\textstyle\frac{1}{2}}\left(1-3\cos^{2}\theta\right)\eta\right],$
(51) $\displaystyle\underline{z}_{0}+1$ $\displaystyle\sim$
$\displaystyle\,\frac{a_{F0}}{a_{Fe}}\left[1+{\textstyle\frac{1}{2}}\left(1-3\cos^{2}\theta\right)\eta_{e}+\cos\varphi\,\sin\theta\,V_{x}+\cos\theta\,V_{z}\right],$
(52) $\displaystyle F_{c}$ $\displaystyle\sim\ $
$\displaystyle\frac{1}{Na_{F0}}\left(3\,\sin^{2}\theta\,\eta_{e}+\cos\varphi\,\sin\theta\,V_{x}+\cos\theta\,V_{z}\right),$
(53) $\displaystyle I$ $\displaystyle\sim$
$\displaystyle\chi_{e0}+O(\eta),\quad\quad J\ \sim\ \chi_{e0}+O(\eta),$ (54)
$\displaystyle\alpha$ $\displaystyle\sim$
$\displaystyle\frac{T_{D}}{a_{F0}\chi_{e0}}\,(3\,\cos\varphi\,\sin\theta\,\cos^{2}\theta\,\eta_{e}+[1-\cos^{2}\varphi\,\sin^{2}\theta]\,V_{x}$
(55) $\displaystyle\hskip
42.67912pt-\cos\varphi\,\sin\theta\,\cos\theta\,V_{z}\,),$
$\displaystyle\gamma$ $\displaystyle\sim$
$\displaystyle-\sin\theta\,\frac{T_{D}}{a_{F0}\chi_{e0}}\left(3\,\sin\theta\,\cos\theta\,\eta_{e}+\cos\varphi\,\cos\theta\,V_{x}-\sin\theta\,V_{z}\right),$
(56)

where $\chi_{e0}$ is the conformal time of flight of the photon = the comoving
geodesic ‘distance’ between $\vec{x}_{e}$ and $\vec{x}_{0}$ in the underlying
Friedman universe,

$\displaystyle\chi_{e0}\vcentcolon=\int_{t_{e}}^{t_{0}}\,\frac{1}{a_{F}}\,\quad\text{and}\quad\vec{V}=a_{F0}\vec{v}.$
(57)

Consider first the drift with respect to $\varphi$, equation (49), and
linearize it in $\eta$ and $V$, but not in
$T_{D}/\\!\\!\left\\{a_{F0}\chi_{e0}\right\\}$:

$\displaystyle\cos\tilde{\varphi}\sim\hskip 409.71968pt$ (58)
$\displaystyle\frac{\cos\varphi+[3\cos\varphi\,\cos^{2}\\!\theta\,\eta_{e}+(1-\cos^{2}\\!\varphi\,\sin^{2}\\!\theta)\\!\\!\left\\{V_{x}/\sin\theta\right\\}-\cos\varphi\,\cos\theta\,V_{z}]\,T_{D}/\\!\\!\left\\{a_{F0}\chi_{e0}\right\\}}{\sqrt{1+2\cos\theta[3\cos\theta\,\eta_{e}+\cos\varphi\,\cos\theta\left\\{V_{x}/\sin\theta\right\\}-V_{z}]\,T_{D}/\\!\\!\left\\{a_{F0}\chi_{e0}\right\\}}}\,.$

For $\sin\theta=0$, the angle $\varphi$ is ill-defined and equation (58) is
singular:

$\displaystyle\lim_{\sin\theta\,\rightarrow\,0}\cos\tilde{\varphi}\
=\lim_{\sin\theta\,\rightarrow\,0}\sqrt{\frac{1}{2\cos\varphi}\left\\{\,\frac{V_{x}}{\sin\theta}\right\\}\frac{T_{D}}{a_{F0}\chi_{e0}}}\,.$
(59)

If not only $V\ll 1$, but also $|\\{V_{x}/\sin\theta\\}|\ll 1$, we may further
[9] linearize $\cos\tilde{\varphi}$:

$\displaystyle\cos\tilde{\varphi}$ $\displaystyle\sim$
$\displaystyle\cos\varphi\left[1+\,\frac{\sin^{2}\varphi}{\cos\varphi}\,\frac{V_{x}}{\sin\theta}\,\frac{T_{D}}{a_{F0}\chi_{e0}}\right].$
(60)

In this case the drift with respect to the angle $\varphi$ is,

$\displaystyle\delta\cos\varphi$ $\displaystyle\vcentcolon=$
$\displaystyle\cos\tilde{\varphi}-\cos\varphi\sim\,{\sin^{2}\varphi}\,\frac{V_{x}}{\sin\theta}\,\frac{T_{D}}{a_{F0}\chi_{e0}}\,,$
(61) $\displaystyle\delta\varphi$ $\displaystyle\sim$
$\displaystyle\frac{-1}{\sin\varphi}\,\,\delta\cos\varphi\sim-\,{\sin\varphi}\,\frac{V_{x}}{\sin\theta}\,\frac{T_{D}}{a_{F0}\chi_{e0}}\,.$
(62)

In particular if $V_{x}$ vanishes, then $\varphi$ is constant and the drift
remains in the plane defined by the $z$-axis and this constant angle
$\varphi$.

The definition of the angle $\theta$ does not involve the projection into the
$xy$-plane and linearizing the $\theta$-drift is straight-forward:

$\displaystyle\cos\tilde{\theta}$ $\displaystyle\sim$
$\displaystyle\cos\theta\left[1-\sin^{2}\theta\left(3\left({\textstyle\frac{1}{2}}a_{F0}\chi_{e0}\,\eta^{\prime}_{0}+\eta_{e}\right)+\cos\varphi\,\frac{V_{x}}{\sin\theta}\,-\,\frac{V_{z}}{\cos\theta}\,\right)\frac{T_{D}}{a_{F0}\chi_{e0}}\right],$
(63) $\displaystyle\delta\cos\theta$ $\displaystyle\sim$
$\displaystyle-\sin^{2}\theta\,\cos\theta\left(3\left({\textstyle\frac{1}{2}}a_{F0}\chi_{e0}\,\eta^{\prime}_{0}+\eta_{e}\right)+\cos\varphi\,\frac{V_{x}}{\sin\theta}\,-\,\frac{V_{z}}{\cos\theta}\right)\frac{T_{D}}{a_{F0}\chi_{e0}},$
(64) $\displaystyle\delta\theta$ $\displaystyle\sim$
$\displaystyle\left(3\sin\theta\,\cos\theta\left({\textstyle\frac{1}{2}}a_{F0}\chi_{e0}\,\eta^{\prime}_{0}+\eta_{e}\right)+\cos\varphi\,\cos\theta\,V_{x}-\sin\theta\,V_{z}\right)\,\frac{T_{D}}{a_{F0}\chi_{e0}}\,.$
(65)

Note that the first term, $\eta^{\prime}_{0}$, in the drift (65) does not
depend on the redshift of the source. For a dominating expansion in the
$z$-direction today,
$H_{c0}-H_{0}\sim{\textstyle\frac{3}{2}}\eta^{\prime}_{0}>0$, this first term
produces a drift away from the $z$-axis.

The second term, $\eta_{e}$, in the drift (65) produces the opposite effect.
Indeed, for a dominating expansion in the $z$-direction from emission until
today, $\eta^{\prime}$ is positive and, since we have set $\eta_{0}$ to zero,
$\eta_{e}$ is negative.

If we adopt Einstein’s equations with cosmological constant and dust, then
$a_{F}$ and $\eta$ are given by,

$\displaystyle
a_{F}(t)=a_{F0}\left(\frac{\cosh\left(\sqrt{3\Lambda}\,t\right)-1}{\cosh\left(\sqrt{3\Lambda}\,t_{0}\right)-1}\right)^{1/3},\hskip
11.38109ptt_{0}={\textstyle\frac{2}{3}}\,\frac{\text{arc}\tanh\,\sqrt{\Omega_{\Lambda
0}}}{\sqrt{\Omega_{\Lambda 0}}}\,\frac{1}{H_{F0}}\,,\hskip
11.38109pt\Omega_{\Lambda 0}\vcentcolon=\,\frac{\Lambda}{3\,H_{F0}^{2}}\,,$
(66)

and

$\displaystyle\eta^{\prime}=\eta^{\prime}_{0}\,\frac{a_{F0}^{3}}{a_{F}^{3}}\quad\text{implying}\quad\eta(t)=-\,{\textstyle\frac{2}{3}}\,\frac{\eta^{\prime}_{0}}{H_{F0}}\,\frac{\sqrt{\Omega_{\Lambda
0}}}{1-\Omega_{\Lambda
0}}\left[\coth\left({\textstyle\frac{1}{2}}\sqrt{3\Lambda}\,t\right)-1/\sqrt{\Omega_{\Lambda
0}}\right].$ (67)

We take $a_{F0}=1$ Gyr, $\Omega_{\Lambda 0}=0.7$ and $H_{0}=72$ km/s/Mpc
implying $1/H_{F0}=13.58$ Gyr, $t_{0}=13.09$ Gyr and $1/\sqrt{\Lambda}=9.37$
Gyr.

Let us compute the ratio
$\eta_{e}/{(\textstyle\frac{1}{2}}a_{F0}\chi_{e0}\,\eta^{\prime}_{0})$ of the
second term in the linearized drift equation (65) by the first term. For the
redshift $\underline{z}_{0}=1$, we find $t_{e}=5.59$ Gyr and by numerical
integration $\chi_{e0}=10.48$. Finally we obtain,

$\displaystyle\,\frac{\eta_{e}}{{\textstyle\frac{1}{2}}a_{F0}\chi_{e0}\,\eta^{\prime}_{0}}\,=-\,4.38.$
(68)

Our result disagrees with the result by Quercellini et al. [10][11] without
peculiar velocity. They find – as the redshift ranges from zero to one – that
the absolute value of the ratio is at most 7 %. They also find the opposite
sign in front of the first term. The sign problem was noticed before by
Marcori et al. [12], who also include peculiar velocities.

##  6 Conclusion

This second term in the drift equation (65), which is proportional to
$\eta_{e}=\eta(t_{e})$ and which we find to be dominant, depends on the
redshift of the source via the inverse of the direction-dependent relation
(52). It is therefore phenomenologically more interesting for a fit of the
Gaia catalogue of quasars than the first term. We look forward to this fit.

Acknowledgements: I thank André Tilquin and Galliano Valent for tens of years
of joint research and fun. It is also a pleasure to thank Jeremy Darling for
his constructive advice.

## References

  * [1] George Lemaître, “L’univers en expansion,” Annales Soc. Sci. Bruxelles A 53 (1933), 51-85 doi:10.1023/A:1018855621348;
English translation by M. A. H. MacCallum,“The expanding universe,” Gen. Rel.
Grav. 29 (1997) 641.

  * [2] Luigi Bianchi, “Sugli spazi a tre dimensioni che ammettono un gruppo continuo di movimenti,” Memorie di Matematica e di Fisica della Societa Italiana delle Scienze 11 (1898) 267352;
English translation by R.T. Jantzen,“On the three-dimensional spaces which
admit a continuous group of motions,” Gen. Rel. Grav. 33 (2001) 2171,
http://www34.homepage.villanova.edu/robert.jantzen/bianchi/

  * [3] G. Ellis and M. A. MacCallum, “A class of homogeneous cosmological models,” Comm. Math. Phys. 12 (1969) 108.
  * [4] S. Hawking, “On the rotation of the Universe,” Mon. Not. Roy. Astron. Soc. 142 (1969) 129 doi:10.1093/mnras/142.2.129
  * [5] C. B. Collins and S. Hawking, “The rotation and distortion of the Universe,” Mon. Not. Roy. Astron. Soc. 162 (1973) 305 doi:10.1093/mnras/162.4.307
  * [6] J. Darling, “The Hubble Expansion is Isotropic in the Epoch of Dark Energy,” Mon. Not. Roy. Astron. Soc. 442 (2014) L66 doi:10.1093/mnrasl/slu057 [arXiv:1404.3735 [astro-ph.CO]].
  * [7] O. Titov and S. Lambert, “On the VLBI measurement of the Solar System acceleration”, Astron. Astrophys. 559 (2013) A95 [arXiv:1310.2723 [astro-ph.IM]].
  * [8] T. Schücker, A. Tilquin and G. Valent, “Bianchi I meets the Hubble diagram,” Mon. Not. Roy. Astron. Soc. 444 (2014) 2820 doi:10.1093/mnras/stu1656 [arXiv:1405.6523 [astro-ph.CO]].
  * [9] John von Neumann: Truth is much too complicated to allow for anything but approximations.
  * [10] C. Quercellini, P. Cabella, L. Amendola, M. Quartin and A. Balbi, “Cosmic Parallax as a probe of late time anisotropic expansion,” Phys. Rev. D 80 (2009), 063527 doi:10.1103/PhysRevD.80.063527 [arXiv:0905.4853 [astro-ph.CO]].
  * [11] C. Quercellini, L. Amendola, A. Balbi, P. Cabella and M. Quartin, “Real-time Cosmology,” Phys. Rept. 521 (2012), 95 doi:10.1016/j.physrep.2012.09.002 [arXiv:1011.2646 [astro-ph.CO]].
  * [12] O. H. Marcori, C. Pitrou, J. P. Uzan and T. S. Pereira, “Direction and redshift drifts for general observers and their applications in cosmology,” Phys. Rev. D 98 (2018) no.2, 023517 doi:10.1103/PhysRevD.98.023517 [arXiv:1805.12121 [astro-ph.CO]].


⁴4 ⁵5 (eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:  ¹German Cancer Research Center, GER ²Helmholtz Imaging, GER
³Heidelberg University, GER ⁴University of Minnesota, USA ⁵National Center for
Tumor Diseases, GER ✉:{tim.raedsch<EMAIL_ADDRESS>
# Quality Assured: Rethinking Annotation Strategies in Imaging AI

Tim Rädsch✉ 112233 0000-0003-3518-0315 Annika Reinke 1122 0000-0003-4363-1876
Vivienn Weru 11 0000-0002-7509-3307 Minu D. Tizabi 11 0000-0003-3687-6381
Nicholas Heller 44 0000-0001-8516-8707 Fabian Isensee 1122 0000-0002-3519-5886
Annette Kopp-Schneider 11 0000-0002-1810-0267 Lena Maier-Hein 11223355
0000-0003-4910-9368

###### Abstract

This paper does not describe a novel method. Instead, it studies an essential
foundation for reliable benchmarking and ultimately real-world application of
AI-based image analysis: generating high-quality reference annotations.
Previous research has focused on crowdsourcing as a means of outsourcing
annotations. However, little attention has so far been given to annotation
companies, specifically regarding their internal quality assurance (QA)
processes. Therefore, our aim is to evaluate the influence of QA employed by
annotation companies on annotation quality and devise methodologies for
maximizing data annotation efficacy. Based on a total of 57,648 instance
segmented images obtained from a total of 924 annotators and 34 QA workers
from four annotation companies and Amazon Mechanical Turk (MTurk), we derived
the following insights: (1) Annotation companies perform better both in terms
of quantity and quality compared to the widely used platform MTurk. (2)
Annotation companies’ internal QA only provides marginal improvements, if any.
However, improving labeling instructions instead of investing in QA can
substantially boost annotation performance. (3) The benefit of internal QA
depends on specific image characteristics. Our work could enable researchers
to derive substantially more value from a fixed annotation budget and change
the way annotation companies conduct internal QA.

Figure 1: Research Questions (RQs) tackled in this work. Based on 57,648
instance segmentation masks annotated by 924 annotators and 34 quality
assurance (QA) workers from five different annotation providers, we (1)
compared the effectiveness of generating high-quality annotations between
annotation companies and Amazon Mechanical Turk and (2) investigated the
effects of annotation companies’ internal QA and (3) real-world image
characteristics on the annotation quality.

###### Keywords:

Data Annotation Quality Assurance Crowdsourcing

##  1 Introduction

Following a long period of extensive focus on pure method development, proper
benchmarking and validation have recently been recognized as a major challenge
in AI-based image analysis [43, 33]. In this context, high-quality annotation
of reference datasets is a fundamental requirement to ensure the meaningful
validation of image analysis algorithms [15]. As annotation tasks may be
cumbersome, crowdsourcing is often used to partially outsource image
annotation. Traditionally, such outsourcing is conducted via crowdsourcing
platforms [32], with Amazon Mechanical Turk (MTurk)111https://www.mturk.com *
shared last authoras the predominant choice [9, 49].

Recent literature has reflected the rise of annotation companies as a new type
of provider with potentially better annotation performance compared to
established crowdsourcing platforms such as MTurk [44, 28, 42, 36]. Annotation
companies differ substantially from crowdsourcing platforms. First, they
employ annotators directly and at shared office spaces [44, 28], enabling
better external control of worker conditions and stricter data security
measures, while crowdsourcing platforms can be accessed by anyone, without
specific knowledge or training being required. Second, annotators in
annotation companies typically work in small teams, with an experienced
quality assurance (QA) worker being responsible for the team’s output. In the
observed internal QA, QA workers make adjustments to the annotations generated
by the team by reviewing each individual annotation and modifying those where
they deem changes necessary. As a result, the annotation requester receives
the quality-assured annotation, not the initial one (see Fig. 2a). In
contrast, crowdsourcing platforms usually do not employ internal QA mechanics
similar to annotation companies.

The importance of annotation providers for real-world applications is evident
at the major computer vision conferences, where over the past two years,
approximately 20% of exhibitors were annotation providers (see Fig. 2b). While
QA as a post-processing step for annotations from platforms such as MTurk has
been researched (e.g. [10, 27, 17, 21]), little is known about the benefit of
internal QA on annotation quality within annotation companies. The only
related work that we are aware of pertains to external QA from the perspective
of an annotation company. Unlike in internal QA, in the case of external QA,
domain experts or experienced annotators working on behalf of the requester
check the annotation quality as a post-processing step of the received
annotations [47, 31]. Our paper, on the other hand, focuses on the internal QA
processes of an annotation company.

Figure 2: a) In contrast to Amazon Mechanical Turk (MTurk), annotation
companies commonly perform quality assurance (QA) before delivering the result
to the annotation requester. b) Annotation providers are an integral part of
the computer vision community. They account for around 20% of exhibitors
present at major computer vision conferences (ECCV 2022: n = 40, CVPR 2022: n
= 97, ICCV 2023: n = 43, CVPR 2023: n = 104) in the last two years.

In light of the vast sums of money to be spent on QA [40], the purpose of this
paper is to shed light on whether annotation companies’ internal QA benefits
the annotation quality. Based on more than 50,000 instance segmentation masks
annotated by more than 900 annotators from five different annotation
providers, we investigate the following research questions (RQs; see Fig. 1):

RQ 1) How does the choice of annotation provider influence the quality of
annotations? As the selection of the annotation provider is among the first
and key decisions in any annotation project after data collection and
generation of labeling instructions, we first investigated companies’
effectiveness in generating annotations, which is particularly relevant in
cases of constrained annotation budgets requiring a cost-effective strategy.
As both quantity and quality of annotated images for a given budget are
relevant in provider selection, we compared the annotation quality and costs
of five different providers, including MTurk and four annotation companies, on
identical images.

RQ 2) Does annotation companies’ quality assurance improve annotation quality?
A major decision in the design of an annotation process concerns the
allocation of resources. A common perception is that internal QA of annotation
companies, following the initial annotation, substantially enhances the
quality. Here, we challenged this assumption by investigating the improvement
in annotation quality from internal QA compared to that achieved through
enhanced labeling instructions.

RQ 3) Should quality assurance focus on specific subsets of images? In theory,
the efficacy of QA may be contingent upon specific image characteristics [24,
41, 20]. For instance, underexposed images might benefit more from a secondary
review compared to those with optimal illumination. Concurrently, images that
present annotation challenges for human observers might be particularly
pertinent for the training and validation of algorithms [41, 20]. Here, we
investigated whether and to what extent the benefit of QA depends on specific
image characteristics.

##  2 Related Work

###  2.1 Comparison of annotation providers

With respect to the quality of annotations obtained from varying annotation
providers, the closest work conducted are comparisons between crowdsourcing
platforms and domain experts. Irshad et al. [23] and Petrović et al. [38],
among others, have shown that crowdsourced annotations can reach the
annotation quality of domain experts in safety-critical applications. In
recent literature, Duggan et al. [13] and Kentley et al. [25] compared domain-
specialized crowds as a new type of annotation provider and found comparable
performance to domain experts. Furthermore, Rädsch et al. [42] provided a
comparison of annotation companies to crowdsourcing, but focused solely on raw
annotation quality without QA and without relating it to annotation budgets.

Interestingly, researchers report concerns about the annotation quality
obtained from MTurk [31], such as obtaining unusable annotations in up to
67.8% of cases [6]. Other research fields, such as psychology, even report a
"substantial decrease in data quality" [7]. However, we are not aware of work
that has systematically compared the performance between annotation companies
and crowdsourcing platforms such as MTurk while considering annotation
budgets.

###  2.2 Quality assurance

The main goal of QA is to enhance and ensure the quality of annotations. A
comprehensive overview on QA in crowdsourcing platforms is provided by Daniel
et al. [10]. However, their research does not include annotation companies.
During the annotation process, quality control techniques generally fall into
three categories [31]: (1) comparing to a gold standard with pre-defined
reference annotations (e.g., [27]), (2) analyzing combined outputs from
multiple annotators (e.g., [21]), and (3) investigating annotator behaviors
(e.g., [17, 19]). Other approaches to enhancing annotation quality on
crowdsourcing platforms target training of crowdworkers [12] and iterative
refinement of the annotation environment [14, 3, 4]. It should be noted that
annotation byproducts, such as mouse movement during annotation generation
[16, 17] or inter-rater agreement [48], can be used to boost model
performance, by incorporating this information in the training.

The closest related work focusing on the impact of QA in annotation companies
was introduced by Wang et al. [47] and Lu et al. [31]. However, their work
focuses on external QA conducted by experienced personnel from the requester
rather than internal QA, and features a substantially smaller number of
annotators from annotation companies. To our knowledge, a systematic
investigation into the effects of internal QA processes within annotation
companies is thus, so far, lacking.

###  2.3 Impact of real-world image characteristics on annotation quality

Related research analyzing the impact of real-world image characteristics
highlights their effect on annotations created by machine learning models and
annotators alike. Michaelis et al. [37] demonstrated that models easily
encounter failures with increasingly challenging real-world image
characteristics, such as added snow or poor lighting characteristics.
Similarly, annotators generate less reliable annotations in the presence of
challenging real-world image characteristics. This effect can be observed both
within [24, 31, 1] and across domains [18, 26]. However, a systematic analysis
of the impact of such characteristics on internal QA employed by annotation
companies is lacking.

##  3 Methods

In the following, we describe the selected data, the labeling instructions,
annotation providers and annotators, the QA process, and the experimental
design of our work.

###  3.1 Data

We conducted our experiments on the instance segmentation data from the
Heidelberg Colorectal (HeiCo) dataset for surgical data science in the sensor
operating room [34], representing a safety-critical application that
crowdsourcing platforms can still handle. The task involved segmenting
surgical instruments from laparoscopic videos, resulting in 57,648 instance
segmentation masks on 4,050 unique frames. Furthermore, we used 57,636
metadata annotations on the unique images to quantify the effect of varying
real-world image characteristics (see RQ3). Characteristics included motion
blur, underexposure, object occlusion, or overlapping objects, among others,
and are illustrated in Suppl. A.

###  3.2 Labeling instructions

To investigate the benefit of QA compared to an investment in detailed
labeling instructions, we created three distinct types of labeling
instructions with varying levels of information density, namely, (1) minimal
text, (2) extended text, and (3) extended text including exemplary pictures.
Each subsequent labeling instruction type was structured to contain more
detailed information than the preceding. To ensure accurate labeling of the
imaging data, we established a collection of design principles applicable to
all labeling instructions:
(1) Labeling information is presented in a slide format to enhance the
processing of information by humans.
(2) Each slide contains a distinct ’chunk’ of information, which is a self-
contained piece of content. This approach of ’chunking’ information lessens
the cognitive load on the working memory.
(3) Related information chunks are placed in close proximity to one another.
(4) A uniform design template is used throughout, featuring specific fonts,
symbols, and color schemes.
The three labeling instructions differ as follows:

Minimal text labeling instructions: These instructions provide a concise
textual outline, offering examples of typical annotations and highlighting the
most frequent annotation types (see Suppl. LABEL:sec_sub:li_min). They reflect
a scenario with minimal effort invested in developing the instructions. The
basic framework includes seven slides totaling 168 words.

Extended text labeling instructions: Building upon the simplified version, the
extended text instructions offer a thorough textual explanation, supplemented
by both positive and negative examples (refer to Suppl. LABEL:sec_sub:li_ext).
They cover both regular and some rare annotation scenarios. The enhanced
version comprises ten slides with a total of 446 words.

Extended text including pictures labeling instructions: These instructions
augment the enhanced textual guidelines with images (detailed in Suppl.
LABEL:sec_sub:li_pic). The visual aids feature descriptions, symbols,
annotations, and color coding to effectively communicate the details on the
slides. They also include illustrations of rare annotation cases, indicating a
deep and well-documented understanding of the labeling process. This set
contains 16 slides with a total of 961 words. We refer the reader to Suppl.
LABEL:sec:app_B_li for full details of each labeling instruction type.

###  3.3 Annotation providers

The study was conducted based on 216 annotators and 34 QA workers from four
annotation companies and 708 crowdworkers from the crowdsourcing platform
MTurk. Our goal was to provide the best representation for both annotation
provider types.

Consequently, we established higher standards for our MTurk crowdworkers
compared to the usual research benchmarks, which generally require a 95%
acceptance rate of Human Intelligence Tasks (HITs) and at least 100 completed
tasks [6, 17, 3]. Our criteria included a 98% acceptance rate with over 5,000
completed HITs. We spread out the HITs over 40 days at all times to obtain a
representative MTurk sample and compensated workers fairly, as recommended by
Litman et al. [30]. The annotation companies we selected have a solid
background in handling large-scale annotation tasks in a variety of domains
and operate internationally. The annotators were allocated by the companies
themselves to replicate the typical workflow of a project carried out by an
annotation company. Each annotator was linked to a specific annotation
provider that had been predetermined. No selection process from our side took
place for either the annotators, QA workers, or the MTurk crowdworkers.

To prevent information leakage, we conducted the labeling process with
increasingly detailed labeling instructions (minimal text instruction,
extended text instruction, and extended text including pictures instruction).
Each annotator worked exclusively with one of the three instruction types as
introduced in Sec. 3.2. To enhance security, we included a mandatory minimum
gap of ten days between consecutive labeling instructions.

###  3.4 Quality assurance process

In the internal QA process, annotators typically report to a small number of
experienced QA workers, who are often responsible for training their team and,
in our case, responsible for the QA of the annotations.

The general workflow for all companies consisted of an annotation stage, in
which images were assigned randomly to each annotator, and a QA stage. During
the annotation stage, annotators received raw images with the task of
annotating them until they considered the quality of annotations satisfactory.
Annotators were aware that their work would be checked by QA. Once submitted,
the QA worker gained access to the generated annotations, allowing them to
make modifications until they were satisfied before submitting the final
annotations for delivery to the requester. Throughout this process, both
annotators and QA workers retained access to the respective labeling
instructions at any given moment. We obtained access to the annotations both
before and after QA through a software company handling a substantial number
of annotation projects annually. The experiments were seamlessly integrated
into the software company’s pipeline of annotation projects, reducing the risk
of exposing our experiments. Pricing was established to be representative of
other projects of similar size.

To maintain fairness and avoid giving any annotation provider a potential
informational advantage that may have affected our statistical analysis, no
specific queries from the annotation companies about the content of the
labeling instructions were addressed.

###  3.5 Experimental design

RQ 1: Each of the five labeling providers annotated the same images with each
set of labeling instructions, employing separate annotators for each task. We
analyzed the number of spam images, images with severe errors, i.e., images
with at least one false positive and/or false negative, and the overall
annotation quality with the Dice Similarity Coefficient (DSC) [11] as the
overlap-based metric and the Normalized Surface Distance (NSD) [39] as the
distance-based metric. We define a spam annotation as a purposefully poor-
quality annotation with the intent to obtain the payment without properly
conducting the annotation task. Annotation duration was used as an auxiliary
measure, where accessible. For instance, if an annotator completed the task in
only 17 seconds, they could not have read the instructions and properly
performed annotations on several images. Of note, a spammer could wait a
relevant number of seconds and then enter a poor-quality annotation to bypass
the annotation duration check. Thus, each image was additionally visually
inspected by an engineer with extensive labeling experience. Spam annotation
examples are displayed in Suppl. D.

RQ 2: The annotation companies conducted their internal QA processes directly
after their annotators completed the annotation job, as explained in Sec. 3.4
and Fig. 2a. The experiments began with only providing the minimal text
labeling instructions, followed by the extended text labeling instructions,
and finally, the extended text instructions with pictures, to prevent
information leakage.

For each annotated image, we calculated the DSC and NSD per instance,
aggregated the scores per annotated image, and aggregated the scores over
annotation company and labeling instruction. We then compared the annotation
quality without and with QA across the three types of labeling instructions
for the four annotation companies by computing both absolute and relative
changes in DSC/NSD aggregates, the percentage of modified images, and the
change in the percentage of severe errors, among other parameters.

RQ 3: In contrast to MTurk, the four annotation companies conducted an
additional annotation round, including QA, with the extended text including
pictures labeling instructions to quantify the impact of different real-world
image characteristics, represented by 57,636 metadata annotations, on the
probability of improvement. The data was analyzed with a logistic mixed model
[35] which included random effects for the image, the annotator, and the
annotation company. Further method details are provided in Suppl. C.

Figure 3: Annotation companies are more efficient at generating high-quality
images than Amazon Mechanical Turk (MTurk). Percentage of images without
severe errors (green), images with severe errors (orange), and spam (red)
obtained for the five different annotation providers investigated. Annotation
costs per image were lower for all companies compared to MTurk (median:
61.1%).

##  4 Results

###  4.1 How does the choice of annotation provider influence the quality and
cost of annotations?

Annotated images generated by the annotation companies were of higher quality,
comprising no spam (MTurk: $\sim$20%) and a much higher proportion of
remaining images without severe annotation errors (factor of $>$ 2 improvement
for all companies compared to MTurk), as shown in Fig. 3. In addition, the
annotation companies were 61.1% cheaper in the median, despite their
additional employment of QA workers. More specifically, the costs ranged from
24.5% to 81.5% of the MTurk costs for the same images and number of
annotations. We have not been given permission to reveal the identities and
exact absolute budgets used for the annotation companies, hence we report
costs relative to the MTurk budget. Examples of spam annotations are provided
in Suppl. D.

Regardless of the annotation scenario, the annotation companies generated
higher quality annotations than MTurk with higher DSC scores including smaller
interquartile ranges (IQRs) (see Fig. 7(a)), higher NSD scores including
smaller IQRs (see Fig. 7(b)), and lower numbers of severe errors (see Fig.
7(c)).

Figure 4: QA of annotation companies only provides marginal improvements, if
any. These depend highly on the company and type of labeling instruction. a:
Absolute change of the median DSC resulting from performing QA. For each
company and labeling instruction, the median was obtained by aggregating the
DSC scores over the images. b: Corresponding absolute values of the DSC.
Positive and negative changes from annotation to QA are colored according to
the color scale. Figure 5: Improving the labeling instructions yields a higher
effect compared to adding QA to the annotation process. The baseline
performance (minimal labeling instruction, no QA) according to the DSC (a) and
the percentage of severe errors (b) is shown individually per company with a
red asterisk (minimal labeling instruction, no QA). While adding QA only
yields marginal improvements (dark blue), major performance boosts are
obtained by improving the labeling instructions provided to the annotators
(green). DSC scores were aggregated for each image and per labeling type. DSC
displayed as dots and box plot (band indicates the median, box indicates the
first (25th percentile) and third (75th percentile) quartiles).
$0\leq\text{DSC}\leq 1$.

###  4.2 Does annotation companies’ quality assurance improve annotation
quality?

As shown in Fig. 4, the internal QA conducted by annotation companies does not
substantially improve annotation quality if the QA workers only use minimal
text instructions. The impact of QA when having access to extended text
instructions or extended text including pictures instructions depended highly
on the annotation company, while the effect was still minimal compared to not
performing QA: Two of the four annotation companies show no change in the
median DSC when comparing QA to non-QA outputs for extended text or extended
text and picture instructions. The remaining two companies only show a very
small relative improvement of below 2%.

We obtained even smaller improvements with the NSD scores, as depicted in
Suppl. E. In certain scenarios, QA was able to remove a small proportion of
severe errors, however, the vast majority remains (see Suppl. F). Notably, the
percentage of modified images by QA workers depended highly on the company, as
depicted in Suppl. G. Fig. 5 further illustrates that providing annotators
with high-quality labeling instructions including pictures (green) leads them
to vastly outperform annotators working with lower-quality labeling
instructions, even if these are followed by additional internal QA (dark
blue).

###  4.3 Should quality assurance focus on specific subsets of images?

As shown in Fig. 6a, QA significantly increases the odds of improving the
annotation quality for images with generalizable image characteristics such as
underexposed objects, intersection of objects, objects occluded by background
in comparison to the absence of this characteristic (vertical black line). The
only exceptions are motion artifacts present in the background and object(s)
overexposure.

In contrast to the general increase of images with generalizable image
characteristics, none of the domain-specific image characteristics yield a
significant change in the odds of improving the annotation quality over a
regular image, see Fig. 6b. The only exception for domain-specific
characteristics is object(s) covered by smoke, which could be interpreted as a
form of object occlusion.

Figure 6: QA of annotation companies can be beneficial to certain images with
difficult real-world generalizable image characteristics, such as
underexposure or intersection of objects. We display the impact of (a) real-
world generalizable and (b) domain-specific image characteristics on the odds
of improvement. Improvement is defined as a positive difference in the QA,
while no improvement refers to 0 or negative difference. The effect of an
image characteristic is shown in the form of the log(odds ratio (OR))
representing the logarithms of the odds of improvement in the presence of the
characteristic, compared to the odds of improvement occurring in the absence
of that characteristic. The vertical black line refers to the point of equal
odds. (Note: ***: p-value $\leq$ 0.001,**: p-value $\leq$ 0.01, *: p-value
$\leq$ 0.05).

##  5 Discussion

The data annotation market is estimated to reach a volume of more than USD 4.1
billion by the end of 2024 [8]. As the annotation process becomes increasingly
automated and foundation models evolve, the market is undergoing drastic
shifts. With lower entry barriers for annotation, more efficient data
annotation tools, and a rising impact of QA, it is thought to result in an
estimated USD 8.2 billion market by 2028 [40]. As a research community, we
need to understand how to ensure high annotation quality throughout this
evolution, as reliable reference annotations are an indispensable foundation
for the benchmarking of image analysis algorithms, and thus their ultimate
translation into real-world practice.

To our knowledge, this study is the first to quantitatively and critically
examine the impact of internal QA conducted by annotation companies in the
context of safety-critical applications. The most important insights can be
summarized as follows:

1\. Annotation companies perform better both in terms of quantity and quality
compared to the widely used platform MTurk (Fig. 3). This implies that
requesters working with crowdsourcing platforms should seriously consider
annotation companies as an alternative. The organizational structure of these
companies, which enables a single point of contact for requesters and
eliminates spam annotations, appears to be a contributing factor to their
efficiency. In light of poor working conditions being a major problem in
annotation generation [36, 22], annotation companies, which can more easily be
inspected than distributed crowdsourcing platforms, are thought to offer
better working contracts, enabling fairer working conditions for annotators.
It should be noted that not every annotation company automatically ensures
adequate working conditions. The annotation companies employed in this study
underwent thorough scrutiny to guarantee compliance with good working
condition guidelines. With annotators working at a shared location instead of
in a distributed crowd, verifying annotators’ working conditions has become
substantially easier. We thus view our work as an initial step towards greater
accountability on the part of annotation requesters.

2\. In contrast to common belief, internal QA mechanisms do not necessarily
improve annotation quality (Fig. 4). Furthermore, we observe no systematic QA
quality improvement difference between annotation companies depending on their
region of origin, size, or price, meaning more expensive annotation companies
do not translate to better QA. However, improving labeling instructions
instead of investing in QA can substantially boost the annotation performance
(Fig. 5). Consequently, in order to obtain high-quality annotations,
annotation requesters should prioritize refining their labeling instructions,
followed by implementing QA or even a different external QA process as a
secondary step. Compared to QA, better labeling instructions can be generated
with relatively few resources.

3\. The benefit of QA depends on the specific image characteristics (Fig. 6).
In fact, QA did improve the quality of images with certain challenging real-
world image characteristics, see Fig. 6a. Consequently, it might be useful for
requesters and annotation companies alike to allocate resources to identifying
challenging image characteristics during a project, train classifiers to
identify images containing these characteristics and focus the internal QA
effort on these images.

One could contend that the recent rise of foundation models marginalizes the
role of annotation providers. We argue, however, that it only changes their
role. In this context, it is necessary to differentiate between the data
quality demands for the training and testing of models. With a recent trend
towards fine-tuning pre-trained models, smaller high-quality datasets can
achieve similar performance with less computation [29, 5], highlighting the
importance of the quality of unlabeled data in unsupervised training and its
role in reducing carbon emissions [45]. In testing, especially for safety-
critical applications such as autonomous driving and clinical medicine,
ensuring high-quality annotated test data is crucial for accurate model
evaluation and real-world applicability [43]. Consequently, while training
data annotation may lose relevance, test data annotation and its high quality
demands are here to stay. Regarding foundation models, it should further be
noted that annotations purely produced by foundation models introduce
artifacts and biases [2] and represent an increasing rising problem for MTurk,
as demonstrated by Veselovsky et al. [46].

Our study is subject to several limitations which deserve further discussion.
Comparing prices across providers is non-trivial as the costs to annotate a
given data set depend on many factors including, among others, image quantity,
type of annotations, location of annotation company and quality requirements.
Our analysis was based on the actual costs incurred during our project.
Confidentiality agreements with the software company which integrated our
experiments into their annotation project pipeline prevent us from disclosing
the identities of the annotation companies. Additionally, we took precautions
to protect the identities of individual annotators and QA workers. This was
particularly important given our findings that QA workers did not
significantly enhance annotation quality, potentially exposing them to
negative repercussions.

Another limitation could be seen in the fact that our data set only represents
one safety-critical domain. While this is a valid concern, we would like to
highlight that we chose a dataset which combined quality, complexity, and
volume. Furthermore, regarding the breadth of annotation providers and the
number of annotators involved, our study is likely the largest of its kind,
thereby incurring substantial costs.

Finally, conducting multiple experiments with identical annotation companies
and varying datasets could have potentially revealed our experimental design.
Had the annotation companies been aware of our research, outcomes could
potentially have been biased to appear more favorable, given companies’
inherent interest in demonstrating their services as reliable. We thus chose
to conduct our experiments such that they could be seamlessly integrated into
the software company’s annotation pipeline, reducing their likelihood of
exposure.

##  6 Conclusion and Future Work

In conclusion, our study underscores the pivotal role of enhancing the initial
annotation quality, for instance via allocating resources to the improvement
of labeling instructions, over QA in obtaining higher reference annotation
quality in high-stakes AI [43]. Future research should delve into the
development and implementation of comprehensive training programs for
annotators from annotation companies to further enhance initial annotation
quality. Exploring the role of annotation companies in the context of
increasingly automated annotation processes driven by foundation models
presents another avenue for future research.

##  7 Acknowledgements

A part of this work was funded by M. Mengler and S. Funke - during their time
at understand.ai -, and Helmholtz Imaging. T. Rädsch was supported by a
scholarship from the Hanns Seidel Foundation with funds from the Federal
Ministry of Education and Research Germany (BMBF). We thank M. Gelz, S.
Strzysch and N. Kraft for their continuous support.

(a) Annotation companies generate higher DSC scores than MTurk.
$0\leq\text{DSC}\leq 1$.

(b) Annotation companies generate higher NSD scores than
MTurk.$0\leq\text{NSD}\leq 1$.

(c) Annotation companies generate a lower number of severe errors.

Figure 7: Annotation companies generate higher quality annotations with their
annotation pipeline than MTurk. This is observed for (a) the DSC, (b) the NSD,
and (c) number of severe errors (lower is better). Annotation companies
include a QA step, in contrast to crowdsourcing on MTurk. (a) and (b)
displayed as dots and box plots (band indicates the median, box indicates the
first (25th percentile) and third (75th percentile) quartiles).

## References

  * [1] Ali, S., Zhou, F., Braden, B., Bailey, A., Yang, S., Cheng, G., Zhang, P., Li, X., Kayser, M., Soberanis-Mukul, R.D., Albarqouni, S., Wang, X., Wang, C., Watanabe, S., Oksuz, I., Ning, Q., Yang, S., Khan, M.A., Gao, X.W., Realdon, S., Loshchenov, M., Schnabel, J.A., East, J.E., Wagnieres, G., Loschenov, V.B., Grisan, E., Daul, C., Blondel, W., Rittscher, J.: An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy. Scientific Reports 10(1), 2748 (Feb 2020). https://doi.org/10.1038/s41598-020-59413-5, https://www.nature.com/articles/s41598-020-59413-5, number: 1 Publisher: Nature Publishing Group
  * [2] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J.Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D.E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P.W., Krass, M., Krishna, R., Kuditipudi, R., Kumar, A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li, X.L., Li, X., Ma, T., Malik, A., Manning, C.D., Mirchandani, S., Mitchell, E., Munyikwa, Z., Nair, S., Narayan, A., Narayanan, D., Newman, B., Nie, A., Niebles, J.C., Nilforoshan, H., Nyarko, J., Ogut, G., Orr, L., Papadimitriou, I., Park, J.S., Piech, C., Portelance, E., Potts, C., Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani, Y., Ruiz, C., Ryan, J., Ré, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K., Tamkin, A., Taori, R., Thomas, A.W., Tramèr, F., Wang, R.E., Wang, W., Wu, B., Wu, J., Wu, Y., Xie, S.M., Yasunaga, M., You, J., Zaharia, M., Zhang, M., Zhang, T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., Liang, P.: On the Opportunities and Risks of Foundation Models (Jul 2022). https://doi.org/10.48550/arXiv.2108.07258, http://arxiv.org/abs/2108.07258
  * [3] Bragg, J., Mausam, Weld, D.S.: Sprout: Crowd-Powered Task Design for Crowdsourcing. Proceedings of the ACM on User Interface Software and Technology (2018). https://doi.org/10.1145/3242587.3242598
  * [4] Chang, J.C., Amershi, S., Kamar, E.: Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. pp. 2334–2346. CHI ’17, Association for Computing Machinery, New York, NY, USA (May 2017). https://doi.org/10.1145/3025453.3026044
  * [5] Chen, L., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V., Tang, Z., Srinivasan, V., Zhou, T., Huang, H., Jin, H.: AlpaGasus: Training A Better Alpaca with Fewer Data (Nov 2023). https://doi.org/10.48550/arXiv.2307.08701, http://arxiv.org/abs/2307.08701
  * [6] Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H.A.W.M., de Bruijne, M.: Early Experiences with Crowdsourcing Airway Annotations in Chest CT. In: Carneiro, G., Mateus, D., Peter, L., Bradley, A., Tavares, J.M.R.S., Belagiannis, V., Papa, J.P., Nascimento, J.C., Loog, M., Lu, Z., Cardoso, J.S., Cornebise, J. (eds.) Deep Learning and Data Labeling for Medical Applications. pp. 209–218. Lecture Notes in Computer Science, Springer International Publishing, Cham (2016). https://doi.org/10.1007/978-3-319-46976-8_22
  * [7] Chmielewski, M., Kucker, S.C.: An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results. Social Psychological and Personality Science 11(4), 464–473 (May 2020). https://doi.org/10.1177/1948550619875149, publisher: SAGE Publications Inc
  * [8] Cognilytica: Data engineering, preparation, and labeling for ai 2020 (2020), https://www.cognilytica.com/2020/01/31/data-preparation-labeling-for-ai-2020/
  * [9] Créquit, P., Mansouri, G., Benchoufi, M., Vivot, A., Ravaud, P.: Mapping of Crowdsourcing in Health: Systematic Review. Journal of Medical Internet Research 20(5), 187 (2018). https://doi.org/10.2196/jmir.9330, https://www.jmir.org/2018/5/e187/
  * [10] Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., Allahbakhsh, M.: Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions. ACM Computing Surveys 51, 1–40 (Jan 2019). https://doi.org/10.1145/3148148
  * [11] Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26(3), 297–302 (1945)
  * [12] Dow, S., Kulkarni, A., Klemmer, S., Hartmann, B.: Shepherding the crowd yields better work. In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. pp. 1013–1022. CSCW ’12, Association for Computing Machinery, New York, NY, USA (Feb 2012). https://doi.org/10.1145/2145204.2145355
  * [13] Duggan, N.M., Jin, M., Mendicuti, M.A.D., Hallisey, S., Bernier, D., Selame, L.A., Asgari-Targhi, A., Fischetti, C.E., Lucassen, R., Samir, A.E., Duhaime+, E., Kapur, T., Goldsmith, A.J.: Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Dataset Labeling (Jun 2023). https://doi.org/10.48550/arXiv.2306.06773, http://arxiv.org/abs/2306.06773
  * [14] Gaikwad, S., Chhibber, N., Sehgal, V., Ballav, A., Mullings, C., Nasser, A., Richmond-Fuller, A., Gilbee, A., Gamage, D., Whiting, M., et al.: Prototype tasks: improving crowdsourcing results through rapid, iterative task design (2017). https://doi.org/10.48550/arXiv.1707.05645, https://arxiv.org/abs/1707.05645
  * [15] Halevy, A., Norvig, P., Pereira, F.: The unreasonable effectiveness of data. IEEE intelligent systems 24(2), 8–12 (2009). https://doi.org/10.1109/MIS.2009.36, iSBN: 1541-1672 Publisher: IEEE
  * [16] Han, D., Choe, J., Chun, S., Chung, J.J.Y., Chang, M., Yun, S., Song, J.Y., Oh, S.J.: Neglected free lunch–learning image classifiers using annotation byproducts (2023). https://doi.org/10.48550/arXiv.2303.17595, https://arxiv.org/abs/2303.17595
  * [17] Heim, E., Roß, T., Seitel, A., März, K., Stieltjes, B., Eisenmann, M., Lebert, J., Metzger, J., Sommer, G., Sauter, A.W.: Large-scale medical image annotation with crowd-powered algorithms. Journal of Medical Imaging 5(3), 034002 (2018). https://doi.org/10.1117/1.JMI.5.3.034002
  * [18] Hettiachchi, D., Schaekermann, M., McKinney, T.J., Lease, M.: The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help. Proceedings of the ACM on Human-Computer Interaction 5, 332:1–332:26 (Oct 2021). https://doi.org/10.1145/3476073
  * [19] Hutson, M., Kanzheleva, O., Taggart, C., Campana, B.J.L., Duong, Q.: Quality Control Challenges in Crowdsourcing Medical Labeling (2019)
  * [20] Idrissi, B.Y., Bouchacourt, D., Balestriero, R., Evtimov, I., Hazirbas, C., Ballas, N., Vincent, P., Drozdzal, M., Lopez-Paz, D., Ibrahim, M.: ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations (Nov 2022). https://doi.org/10.48550/arXiv.2211.01866, http://arxiv.org/abs/2211.01866
  * [21] Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on Amazon Mechanical Turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation. pp. 64–67. ACM, Washington DC (Jul 2010). https://doi.org/10.1145/1837885.1837906, https://dl.acm.org/doi/10.1145/1837885.1837906
  * [22] Irani, L.: The cultural work of microwork. New Media & Society 17(5), 720–739 (May 2015). https://doi.org/10.1177/1461444813511926, publisher: SAGE Publications
  * [23] Irshad, H., Montaser-Kouhsari, L., Waltz, G., Bucur, O., Nowak, J., Dong, F., Knoblauch, N., Beck, A.: Crowdsourcing image annotation for nucleus detection and segmentation in computational pathology: evaluating experts, automated methods, and the crowd. In: Biocomputing 2015. pp. 294–305. WORLD SCIENTIFIC, Kohala Coast, Hawaii, USA (Nov 2014). https://doi.org/10.1142/9789814644730_0029, http://www.worldscientific.com/doi/abs/10.1142/9789814644730_0029
  * [24] Joskowicz, L., Cohen, D., Caplan, N., Sosna, J.: Inter-observer variability of manual contour delineation of structures in CT. European Radiology 29(3), 1391–1399 (Mar 2019). https://doi.org/10.1007/s00330-018-5695-5, http://link.springer.com/10.1007/s00330-018-5695-5
  * [25] Kentley, J., Weber, J., Liopyris, K., Braun, R.P., Marghoob, A.A., Quigley, E.A., Nelson, K., Prentice, K., Duhaime, E., Halpern, A.C., Rotemberg, V.: Agreement Between Experts and an Untrained Crowd for Identifying Dermoscopic Features Using a Gamified App: Reader Feasibility Study. JMIR Medical Informatics 11, e38412 (Jan 2023). https://doi.org/10.2196/38412, https://medinform.jmir.org/2023/1/e38412
  * [26] Kohli, M.D., Summers, R.M., Geis, J.R.: Medical image data and datasets in the era of machine learning—whitepaper from the 2016 C-MIMI meeting dataset session. Journal of digital imaging 30(4), 392–399 (2017). https://doi.org/10.1007/s10278-017-9976-3, publisher: Springer
  * [27] Le, J., Edmonds, A., Hester, V., Biewald, L.: Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In: SIGIR 2010 workshop on crowdsourcing for search evaluation. vol. 2126, pp. 22–32 (2010)
  * [28] Le Ludec, C., Cornet, M., Casilli, A.A.: The problem with annotation. Human labour and outsourcing between France and Madagascar. Big Data & Society 10(2), 20539517231188723 (Jul 2023). https://doi.org/10.1177/20539517231188723, http://journals.sagepub.com/doi/10.1177/20539517231188723
  * [29] Lee, A.N., Hunter, C.J., Ruiz, N.: Platypus: Quick, Cheap, and Powerful Refinement of LLMs (Aug 2023). https://doi.org/10.48550/arXiv.2308.07317, http://arxiv.org/abs/2308.07317
  * [30] Litman, L., Robinson, J., Rosenzweig, C.: The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behavior Research Methods 47(2), 519–528 (Jun 2015). https://doi.org/10.3758/s13428-014-0483-x, http://link.springer.com/10.3758/s13428-014-0483-x
  * [31] Lu, X., Ratcliffe, D., Kao, T.T., Tikhonov, A., Litchfield, L., Rodger, C., Wang, K.: Rethinking Quality Assurance for Crowdsourced Multi-ROI Image Segmentation. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 11(1), 103–114 (Nov 2023). https://doi.org/10.1609/hcomp.v11i1.27552, https://ojs.aaai.org/index.php/HCOMP/article/view/27552, number: 1
  * [32] Maier-Hein, L., Mersmann, S., Kondermann, D., Bodenstedt, S., Sanchez, A., Stock, C., Kenngott, H.G., Eisenmann, M., Speidel, S.: Can masses of non-experts train highly accurate image classifiers? In: International conference on medical image computing and computer-assisted intervention. pp. 438–445. Springer (2014). https://doi.org/10.1007/978-3-319-10470-6_55
  * [33] Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M.D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., et al.: Metrics reloaded: recommendations for image analysis validation. Nature Methods 21, 195–212 (2024). https://doi.org/10.1038/s41592-023-02151-z, https://www.nature.com/articles/s41592-023-02151-z
  * [34] Maier-Hein, L., Wagner, M., Ross, T., Reinke, A., Bodenstedt, S., Full, P.M., Hempe, H., Mindroc-Filimon, D., Scholz, P., Tran, T.N., Bruno, P., Kisilenko, A., Müller, B., Davitashvili, T., Capek, M., Tizabi, M.D., Eisenmann, M., Adler, T.J., Gröhl, J., Schellenberg, M., Seidlitz, S., Lai, T.Y.E., Pekdemir, B., Roethlingshoefer, V., Both, F., Bittel, S., Mengler, M., Mündermann, L., Apitz, M., Kopp-Schneider, A., Speidel, S., Nickel, F., Probst, P., Kenngott, H.G., Müller-Stich, B.P.: Heidelberg colorectal data set for surgical data science in the sensor operating room. Scientific Data 8(1), 101 (Apr 2021). https://doi.org/10.1038/s41597-021-00882-2, https://www.nature.com/articles/s41597-021-00882-2, number: 1 Publisher: Nature Publishing Group
  * [35] McCulloch, C.E., Searle, S.R.: Generalized, linear, and mixed models. John Wiley & Sons (2004). https://doi.org/10.1002/0471722073, https://books.google.com/books?hl=en&lr=&id=bWDPukohugQC&oi=fnd&pg=PR6&dq=Generalized,+Linear,+and+Mixed+Models.&ots=sSy3HM7CGo&sig=Ss04EHKG_xkcZZCYWSEJl6paHos
  * [36] Miceli, M., Posada, J.: The Data-Production Dispositif. Proceedings of the ACM on Human-Computer Interaction 6(CSCW2), 1–37 (Nov 2022). https://doi.org/10.1145/3555561, https://dl.acm.org/doi/10.1145/3555561
  * [37] Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A.S., Bethge, M., Brendel, W.: Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (Mar 2020). https://doi.org/10.48550/arXiv.1907.07484, http://arxiv.org/abs/1907.07484
  * [38] Petrović, N., Moyà-Alcover, G., Varona, J., Jaume-i Capó, A.: Crowdsourcing human-based computation for medical image analysis: A systematic literature review. Health Informatics Journal 26(4), 2446–2469 (Dec 2020). https://doi.org/10.1177/1460458220907435, http://journals.sagepub.com/doi/10.1177/1460458220907435
  * [39] Reinke, A., Tizabi, M.D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A.E., Rädsch, T., Sudre, C.H., Acion, L., Antonelli, M.: Understanding metric-related pitfalls in image analysis validation. Nature methods pp. 1–13 (2024). https://doi.org/doi.org/10.1038/s41592-023-02150-0, https://www.nature.com/articles/s41592-023-02150-0, publisher: Nature Publishing Group
  * [40] Research, G.V.: Data Collection And Labeling Market Size & Share Report 2030 (2022), https://www.grandviewresearch.com/industry-analysis/data-collection-labeling-market
  * [41] Roß, T., Bruno, P., Reinke, A., Wiesenfarth, M., Koeppel, L., Full, P.M., Pekdemir, B., Godau, P., Trofimova, D., Isensee, F.: How can we learn (more) from challenges? A statistical approach to driving future algorithm development. arXiv preprint arXiv:2106.09302 (2021). https://doi.org/10.48550/arXiv.2106.09302, https://arxiv.org/abs/2106.09302
  * [42] Rädsch, T., Reinke, A., Weru, V., Tizabi, M.D., Schreck, N., Kavur, A.E., Pekdemir, B., Roß, T., Kopp-Schneider, A., Maier-Hein, L.: Labelling instructions matter in biomedical image analysis. Nature Machine Intelligence 5(3), 273–283 (Mar 2023). https://doi.org/10.1038/s42256-023-00625-5, https://www.nature.com/articles/s42256-023-00625-5, number: 3 Publisher: Nature Publishing Group
  * [43] Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., Aroyo, L.M.: “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. pp. 1–15. ACM, Yokohama Japan (May 2021). https://doi.org/10.1145/3411764.3445518, https://dl.acm.org/doi/10.1145/3411764.3445518
  * [44] Schmidt, F.A.: The Planetary Stacking Order of Multilayered Crowd-AI Systems. In: Graham, M., Ferrari, F. (eds.) Digital Work in the Planetary Market, pp. 137–156. The MIT Press (Jun 2022). https://doi.org/10.7551/mitpress/13835.003.0012, https://direct.mit.edu/books/book/5319/chapter/3800159/The-Planetary-Stacking-Order-of-Multilayered-Crowd
  * [45] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G.: LLaMA: Open and Efficient Foundation Language Models (Feb 2023). https://doi.org/10.48550/arXiv.2302.13971, http://arxiv.org/abs/2302.13971
  * [46] Veselovsky, V., Ribeiro, M.H., West, R.: Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks (Jun 2023). https://doi.org/10.48550/arXiv.2306.07899, http://arxiv.org/abs/2306.07899
  * [47] Wang, K., Hill, M., Knowles-Barley, S., Tikhonov, A., Litchfield, L., Bare, J.C.: Improving Segmentation of Breast Arterial Calcifications from Digital Mammography: Good Annotation is All You Need. In: Zheng, Y., Keleş, H.Y., Koniusz, P. (eds.) Computer Vision – ACCV 2022 Workshops, vol. 13848, pp. 134–150. Springer Nature Switzerland, Cham (2023). https://doi.org/10.1007/978-3-031-27066-6_10, https://link.springer.com/10.1007/978-3-031-27066-6_10, series Title: Lecture Notes in Computer Science
  * [48] Yu, S., Zhou, H.Y., Ma, K., Bian, C., Chu, C., Liu, H., Zheng, Y.: Difficulty-Aware Glaucoma Classification with Multi-rater Consensus Modeling. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. pp. 741–750. Lecture Notes in Computer Science, Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-59710-8_72
  * [49] Ørting, S.N., Doyle, A., Hilten, A.v., Hirth, M., Inel, O., Madan, C.R., Mavridis, P., Spiers, H., Cheplygina, V.: A Survey of Crowdsourcing in Medical Image Analysis. Human Computation 7, 1–26 (Dec 2020). https://doi.org/10.15346/hc.v7i1.1, https://hcjournal.org/index.php/jhc/article/view/111

Disclaimer: The Supplementary contains bloody surgery images.

##  A Real-world image characteristics examples

The 57,636 metadata annotations represent real-world image characteristics and
were created by three engineers with the help of some crowdworkers. Examples
are displayed in Supplementary Fig. 1. The characteristics can be
generalizable (Supplementary Fig. 1a) or domain-specific (Supplementary Fig.
1b), and occur in the background, on the objects or as an overlay.
Generalizable characteristics include: motion artifacts in the background,
other objects present in the background, object occluded by background,
intersection of objects, motion artifacts on the object, object(s)
overexposure, object(s) underexposure. Domain-specific characteristics
include: background covered by blood, background covered by reflections,
background covered by smoke, image overlaid with text, caption at the bottom
of the image, black box present in the image, trocar present in the image,
object covered by blood, object(s) covered by reflections, object(s) covered
by smoke.

Supplementary Fig. 1: Real-world image characteristics examples. During
recording, a broad range of (a) generalizable and (b) domain-specific image
characteristics occur and can influence humans and models alike.

See pages 1- of appendix_files/instruction_01_minimal_text.pdf

See pages 1-8 of appendix_files/instruction_02_extended_text.pdf See pages 9-
of appendix_files/instruction_02_extended_text.pdf

See pages 1-8 of appendix_files/instruction_03_picture_version.pdf See pages
9- of appendix_files/instruction_03_picture_version.pdf

##  C Additional method details

Analysis of computer vision conference exhibitors
We included the major computer vision conferences from 2022 and 2023, namely:
ECCV 2022, CVPR 2022, ICCV 2023, and CVPR 2023. The list of exhibitors was
extracted from the respective conference websites222Accessed 2024-03-04:
https://hallerickson.ungerboeck.com/prod/app85.cshtml?aat=
{wRAm7m6yFc6aFoOwv9ggPME6PgM3DwEgvwcFfgyN3YY%3d;
Cr9Jl1p0qL33RLclLrfrlBKRmhROPXGkDwcQAVv7zHM%3D;
Zva%2bsPD7WeTWWGsGWvarzixed2IJHWExCd84dKNS6Yg%3d};
https://iccv2023.thecvf.com/exhibitor.list.and.floor.plan-365000-5-44-50.php.
We define an annotation provider as an entity that provides annotations
performed by human workers. Consequently, each exhibitor’s website was
analyzed to determine whether the exhibitor provides human-generated
annotations as a core component of their offering. Companies offering solely
synthetic data were not included. In the last two years, around 20% of the
conference exhibitors were annotation providers, as illustrated in Fig. 2. As
the majority of exhibitors were offering specific tech products, we excluded
exhibitor booths from our analysis that were not clearly related to a specific
offering (e.g., the booth for the ECCV coffee station or the Springer
Publishing Group). This resulted in three excluded exhibitors for ECCV 2022
(total before exclusion: 43), seven excluded exhibitors for CVPR 2022 (total
before exclusion: 104), three excluded exhibitors for ICCV 2023 (total before
exclusion: 47), and eight excluded exhibitors for CVPR 2023 (total before
exclusion: 112).

Annotation tools
MTurk operates on a self-service basis. In this model, the individual or
organization requesting annotations is responsible for supplying all aspects,
including training for annotators and the annotation tools, whereas the
crowdworkers are engaged as independent contractors. In contrast, annotation
companies provide a managed service for handling annotators, who can work on
any annotation tool that the requester gives access to. Since the tool used
for the annotation companies was not available for MTurk, we used a different
annotation tool for MTurk. Both annotation tools were developed with best
design practices to ensure the production of high-quality annotations.
Furthermore, the tools were tested internally, until a sufficient user
experience and annotation result were achieved. As intermediate annotations
without QA are typically not visible to the requester, the software company
added automated backend exports after the annotation step to enable this
study. Professional annotators are aware that their work is internally
verified by QA workers. None of the annotators participating had previous
experience with the annotation tools they were using.

Logistic mixed model
The logistic mixed model was implemented in the lme4 package in R. The
obtained estimates of the covariates are on the log-odds scale and were
exponentiated to obtain the odds ratio for each covariate. Software: R version
4.0.2 (package lme4 version 1.1.33).

$\text{I}\sim(1|\text{Image\\_name})+(1|\text{Company}/\text{Annotator})+\sum_{i=1}^{7}G_{i}+\sum_{i=1}^{10}D_{i}$
(1)

where:

  * •

Improvement (I) is the response variable.

  * •

$(1|\text{Image\\_name})$ represents a random intercept for each Image_name.

  * •

$(1|\text{Company}/\text{Annotator})$ represents a random intercept, with
Annotator nested within Company.

  * •

$\sum_{i=1}^{7}G_{i}$ represents the generalizable fixed effects, with $G_{i}$
being the $i^{th}$ analyzed generalizable covariate.

  * •

$\sum_{i=1}^{10}D_{i}$ represents the domain-specific fixed effects, with
$D_{i}$ being the $i^{th}$ analyzed domain-specific covariate.

List of covariates:

  * •

Generalizable covariates:

    * –

Motion artifacts in the background

    * –

Other objects present in the background

    * –

Object occluded by background

    * –

Intersection of objects

    * –

Motion artifacts on the object

    * –

Object(s) overexposure

    * –

Object(s) underexposure

  * •

Domain-specific covariates:

    * –

Background covered by blood

    * –

Background covered by reflections

    * –

Background covered by smoke

    * –

Image overlaid with text

    * –

Caption at the bottom of the image

    * –

Black box present in the image

    * –

Trocar present in the image

    * –

Object covered by blood

    * –

Object(s) covered by reflections

    * –

Object(s) covered by smoke

Code will be published with acceptance.

Annotation provider analysis
The 57,636 metadata annotations represent different real-world imaging
characteristics metadata. Utilizing these meta-annotations, we sorted the
images of the dataset into nine categories that reflect the complexity of
annotation, which guided the selection process for subsequent image analysis:
(1) Simple category: The image does not contain any artifacts on the
instruments.
(2) Chaos category: The image contains at least three different artifacts on
the instruments, and those containing a greater number of instruments are
preferred.
(3) Trocar category: The image contains at least one trocar.
(4) Intersection category: The image contains at least two medical instruments
intersecting.
(5) Motion blur category: The image contains at least one medical instrument
exhibiting the motion blur artifact.
(6) Underexposure category: The image contains at least one medical instrument
that is underexposed.
(7) Text overlay category: The image contains text overlay that obstructs the
view.
(8) Image overlay category: The image contains an image overlay that obstructs
the view of the image.
(9) Random category: Images are randomly selected from the remaining images in
the test set.

For the comparison between the different annotation provider types, we chose
234 distinct frames in line with the specified categories. We manually picked
15 distinct images from each of the first eight categories, which made up
about half of the total images. The exception was category 8, which only had
11 unique images fit its criteria. The remainder of the images were randomly
selected from category 9. Each frame was labeled four times following each set
of labeling instructions and by each annotation provider, accumulating 60
annotations for each frame. Each image annotation of the companies was checked
by QA workers (culminating in 25,272 labeled images for the provider
comparison). While most annotation companies prioritize sustained work
packages with longer completion times, MTurk focuses on microtasks that are
completed rapidly and in parallel. Thus, every professional annotator
completed annotations for a total of 72 images and each MTurk crowdworker
annotated four images, to accurately replicate the parallelization seen in
crowdsourcing.

Each annotated image from all annotation providers for both analyses was
checked by an engineer in our team to identify spam annotations. The
annotation companies generated no spam annotations, whereas MTurk annotated
images contained a proportion of spam annotations of around 20%. Examples of
spam annotations are displayed in Supplementary Fig. 2.

Real-world image condition analysis
For the impact analysis of real-world image characteristics on QA, we focused
solely on the extended text including pictures labeling instructions and the
annotation companies. All 4,050 unique frames of the HeiCo dataset were
annotated two times by each annotation company, accumulating eight annotations
for each frame. For this analysis we utilized all 57,636 metadata annotations.

##  D Spam annotation examples

Supplementary Fig. 2: Only MTurk crowdworkers generated spam annotations. The
annotation companies generated no spam annotations, whereas around 20% of the
collected MTurk annotations were spam. The spam annotations vary greatly in
their style and seem to display different strategies to game the MTurk
annotation tasks.

##  E Impact of annotation companies’ QA on NSD scores

Table 1: QA of annotation companies only provides marginal improvements of the Normalized Surface Distance (NSD) scores, if any. |  | 25th Percentile | Median | 75th Percentile
---|---|---|---|---
Company | Stage | Annotate | QA | Annotate | QA | Annotate | QA
Company 1 | 1) Minimal text | 0.65 | 0.66 | 0.99 | 0.99 | 1 | 1
Company 1 | 2) Extended text | 0.62 | 0.61 | 0.99 | 0.99 | 1 | 1
Company 1 | 3) Extended text + pictures | 0.97 | 0.97 | 0.99 | 0.99 | 1 | 1
Company 2 | 1) Minimal text | 0.61 | 0.49 | 0.98 | 0.97 | 1 | 1
Company 2 | 2) Extended text | 0.66 | 0.92 | 0.99 | 0.99 | 1 | 1
Company 2 | 3) Extended text + pictures | 0.93 | 0.95 | 0.99 | 0.99 | 1 | 1
Company 3 | 1) Minimal text | 0.63 | 0.65 | 0.99 | 0.99 | 1 | 1
Company 3 | 2) Extended text | 0.5 | 0.67 | 0.98 | 0.99 | 1 | 1
Company 3 | 3) Extended text + pictures | 0.82 | 0.93 | 0.99 | 0.99 | 1 | 1
Company 4 | 1) Minimal text | 0.66 | 0.66 | 0.99 | 0.99 | 1 | 1
Company 4 | 2) Extended text | 0.96 | 0.96 | 0.99 | 0.99 | 1 | 1
Company 4 | 3) Extended text + pictures | 0.98 | 0.98 | 1 | 1 | 1 | 1

##  F Impact of annotation companies’ QA on severe errors

Table 2: QA of annotation companies only provides marginal reductions of severe errors. These depend highly on the company and type of labeling instruction. Company | Stage | Annotate | QA | Difference
---|---|---|---|---
Company 1 | 1) Minimal text | 0.18 | 0.17 | -0.01
Company 1 | 2) Extended text | 0.2 | 0.2 | 0
Company 1 | 3) Extended text + pictures | 0.1 | 0.1 | 0
Company 2 | 1) Minimal text | 0.22 | 0.28 | 0.06
Company 2 | 2) Extended text | 0.19 | 0.13 | -0.06
Company 2 | 3) Extended text + pictures | 0.14 | 0.11 | -0.02
Company 3 | 1) Minimal text | 0.21 | 0.19 | -0.01
Company 3 | 2) Extended text | 0.25 | 0.17 | -0.08
Company 3 | 3) Extended text + pictures | 0.16 | 0.12 | -0.04
Company 4 | 1) Minimal text | 0.18 | 0.17 | 0
Company 4 | 2) Extended text | 0.12 | 0.12 | 0
Company 4 | 3) Extended text + pictures | 0.09 | 0.09 | 0

##  G Modification distribution by QA workers

The number of modified images depended highly on the annotation company, as
displayed in Supplementary Subfig. 3a. Nevertheless, a substantial proportion
of modified images did not inherently imply superior annotation quality
(Supplementary Subfig. 3b). For instance, Company 1 on average made
modifications to every third annotated image, while Company 4 only made
modifications to every 10th image. However, it is evident that Company 4’s
resulting annotations clearly outperformed those of Company 1 when employing
the extended text and extended text including pictures labeling instructions.
The combination of a low modification rate and high annotation quality
achieved by Company 4 could suggest a higher level of annotator expertise.

Overall, the annotations conducted with the extended text including pictures
labeling instructions outperform all other experiments and QA efforts. This is
a clear indication that further research on internal QA processes is needed.

Supplementary Fig. 3: The percentage of modifications by QA workers differed
greatly between the companies. However, this did not necessarily result in
higher annotation quality, when compared to other annotation companies. a, The
percentages of modified and unadapted images by QA workers. An image counts as
modified if the responsible QA worker adapted the received instance
segmentation mask from the annotator. b, The DSC was aggregated for each
resulting annotated image and is displayed aggregated for each pair of
annotation company and labeling instruction as a dots- and boxplot (the band
indicates the median, the box indicates the first and third quartiles and the
whiskers indicate $\pm 1.5\times$ interquartile range), the DSC maximum is 1
and the minimum is 0 for each image.