# Note on agegraphic dark energy inspired by modified Barrow entropy

A. Sheykhi<EMAIL_ADDRESS>Department of Physics, College of Sciences,
Shiraz University, Shiraz 71454, Iran  Biruni Observatory, College of
Sciences, Shiraz University, Shiraz 71454, Iran  S. Ghaffari
<EMAIL_ADDRESS>Research Institute for Astronomy and Astrophysics
of Maragha (RIAAM), University of Maragheh, P.O. Box 55136-553, Maragheh, Iran

###### Abstract

We revisit agegraphic dark energy (ADE) model when the entropy associated with
the apparent horizon is in the form of Barrow entropy, $S\sim A^{1+\delta/2}$,
where $0\leq\delta\leq 1$ indicates the amount of the quantum-gravitational
deformation effects of the horizon. The modification to the entropy
expression, not only change the energy density of ADE, but also modifies the
Friedmann equations due to thermodynamics-gravity conjecture. Based on this,
we investigate the cosmological consequences of ADE through modified Barrow
cosmology and disclose the effects of Barrow exponent $\delta$ on the
evolutions of the cosmological parameters. We observe that, depending on the
values of $\delta$, the transition from early decelerated phase to the late
time accelerated phase occurs, and the equation of state (EoS) parameter
$w_{de}$ varies from quintessence $-1<w_{de}<-1/3$ to the phantom regime
$(w_{de}<-1)$. When $\delta=0$, all results of ADE in standard cosmology are
restored.

Keywords: Barrow entropy; agegraphic; dark energy; acceleration.

##  I Introduction

According to the thermodynamics-gravity conjecture, one can rewrite the
Friedmann equations of Friedmann-Robertson-Walker (FRW) universe in the form
of the first law of thermodynamics on the apparent horizon and vice versa Cai1
; Cai2 ; Cai3 ; Akbar1 ; Akbar2 ; Akbar3 ; Sheykhi1 ; Sheykhi3 ; Sheykhi4 ;
Sheykhi5 ; SheyECFE . For several reasons, this correspondence is quite
interesting. First, it supports the holography idea, since the Friedmann
equations describe the evolution of the universe in the bulk, while the first
law of thermodynamics is written on the boundary. Second, it provides a way to
derive the field equations of gravity using thermodynamic arguments. Third, in
some cases, where the entropy expression is not well established, it can be
useful to derive the entropy expression associated with the horizon Sheykhi3 ;
Sheykhi4 .

On the other side, dark energy (DE) puzzle, as responsible energy for
describing the acceleration of the cosmic expansion, has been one of the
attractive field of research, for both theoretical and experimental
cosmologists. One of the well-known candidates of DE is the so called
holographic DE which is based on the holographic principle developed by
Fischler and Susskind Suss1 ; Suss2 ; Coh . Another interesting model is the
so called ADE model which suggested by Cai in 2007 CaiADE . This model is
based on the uncertainty relation of quantum mechanics together with the
gravitational effect in general relativity. Accordingly, the energy density of
metric fluctuations of the Minkowski spacetime is given by
$\rho_{de}\sim{m^{2}_{p}}/{t^{2}},$ where $m_{p}$ is the reduced Planck mass
Maz . In this model the observed DE originates from the spacetime and matter
field fluctuations in the universe Wei1 ; Wei2 . The ADE models have received
a lot of attentions in the literatures (see e.g. Cui ; Kim ; sheyADE1 ;
sheyADE2 ; Setare  and checked with observations Wei3 ). Other studies on ADE
models can be carried out in JKS ; Majid ; Pan ; Kumar .

According to the new proposal of J.D. Barrow, the horizon of a black hole may
has a fractal structure and the corresponding area of the horizon cloud
increase due to the quantum-gravitational deformation Barrow . In this
scenario, the area law of entropy get modified as $S_{h}\sim A^{1+\delta/2}$
where $\delta$ quantifies the quantum-gravitational deformation. In the
cosmological setup, the effects of Barrow entropy on the cosmic evolution have
been explored from different viewpoints. In this regards, a new holographic DE
model based on Barrow entropy has been proposed Emm1 ; Ana . A cosmological
scenarios based on Barrow entropy was investigated in Emm2 , where it was
shown that new extra terms that constitute an effective energy density sector
are appeared in the Friedmann equations. In another approach, the modified
Friedmann equations through Barrow entropy and its cosmological consequences
were explored in SheB1 ; SheB2 ; SheB3 , where it was shown that the geometry
part of the Friedmann equations get modified due to the corrections to the
entropy expression. It is worth noting that the exponent $\delta$ in Barrow
entropy, cannot reproduce any term which may play the role of DE and one still
needs to take into account the DE (cosmological constant) component in the
Friedmann equations to reproduce the accelerated universe SheB2 ; SheB3 . On
the other hand, it was recently proven that Barrow entropy as well as any
other known entropy (Tsalis, Renyi, Kaniadakis, etc) is just sub-case of
generalized entropy expression introduced in Odin1 ; Odin2 . Other studies on
the cosmological consequences of the Barrow entropy can be carried out in Emm3
; Abr1 ; Mam ; Abr2 ; Bar2 ; Sri ; Das ; Pra ; Odin3 ; Odin4 .

The modified ADE model when the entropy is in the form of Barrow entropy,
$S_{h}\sim A^{1+\delta/2}$, has been explored in BADE . However, the authors
of BADE  only modify the energy density of ADE, while they still use the
standard Friedmann equations. This is indeed inconsistent with the
thermodynamics-gravity conjecture. Indeed, any modification to the entropy
expression, not only change the energy density of ADE, but also modifies the
Friedmann equations describing the background evolution. Our work differs from
BADE  in that we consider modification to both energy density as well as
Friedmann equations due to the Barrow correction to the area law of entropy.
In the light of all mentioned above, it becomes obvious that the investigation
on the ADE models in the context of modified Barrow cosmology is well
motivated. In particular, we would like to disclose the effects of Barrow
exponent $\delta$ on the cosmological parameters in the late time when the ADE
is dominated.

The plan of the work is as follows. In section II, we review constructing the
modified Friedmann equations through Barrow entropy using the thermodynamics-
gravity conjecture. In section III, we study the cosmological consequences of
the ADE in the context of modified Barrow cosmology. In section IV, we
consider the new model of ADE while the time scale is chosen to be the
conformal time instead of the age of the universe. The last section is devoted
to conclusions and discussions.

##  II Modified Friedmann equations through Barrow entropy

Consider a homogeneous and isotropic FRW universe which is described by the
line elements

$ds^{2}={h}_{\mu\nu}dx^{\mu}dx^{\nu}+\tilde{r}^{2}(d\theta^{2}+\sin^{2}\theta
d\phi^{2}),$ (1)

where $\tilde{r}=a(t)r$, $a(t)$ is the scale factor, $x^{0}=t,x^{1}=r$, and
$h_{\mu\nu}$=diag $(-1,a^{2}/(1-kr^{2}))$ represents the two dimensional
metric. Here $k=0,\pm 1$ represent the curvature of the three dimensional
space. We take apparent horizon as the boundary of spacetime, with radius
Akbar2

$\tilde{r}_{A}=\frac{1}{\sqrt{H^{2}+k/a^{2}}},$ (2)

where $H=\dot{a}/a$ stands for the Hubble parameter which measures the rate of
the universe expansion. The temperature of the apparent horizon is given by
Akbar2

$T_{h}=-\frac{1}{2\pi\tilde{r}_{A}}\left(1-\frac{\dot{\tilde{r}}_{A}}{2H\tilde{r}_{A}}\right).$
(3)

We assume, the energy momentum tensor of the universe is in the form of
perfect fluid,$T_{\mu\nu}=(\rho+p)u_{\mu}u_{\nu}+pg_{\mu\nu},$ where $\rho$
and $p$ are, respectively, the energy density and pressure of the matter
inside the Universe. The total energy content of the universe satisfies the
conservation equation, $\nabla_{\mu}T^{\mu\nu}=0$, which yields the continuity
equation,

$\dot{\rho}+3H(\rho+p)=0.$ (4)

Since our universe is expanding thus we have a work term in the first law of
thermodynamics. The work density is given by Hay2

$W=-\frac{1}{2}T^{\mu\nu}h_{\mu\nu},\ \ \Rightarrow\ \ W=\frac{1}{2}(\rho-p).$
(5)

We propose the first law of thermodynamics holds on the apparent horizon

$dE=T_{h}dS_{h}+WdV,$ (6)

where $V=\frac{4\pi}{3}\tilde{r}_{A}^{3}$ is the volume enveloped by a
3-dimensional sphere, and $T_{h}$ and $W$ are given by Eqs. (3) and (5). In
the above expression $S_{h}$ is the entropy associated with the apparent
horizon. Taking differential of total energy inside the apparent horizon,
$E=\rho V$, yields

$dE=4\pi\tilde{r}_{A}^{2}\rho d\tilde{r}_{A}-4\pi
H\tilde{r}_{A}^{3}(\rho+p)dt.$ (7)

where we have also used the continuity equation (4). The Barrow entropy
associated to the apparent horizon is given by Barrow

$\displaystyle S_{h}=\left(\frac{A}{A_{0}}\right)^{1+\delta/2},$ (8)

where $A=4\pi\tilde{r}_{A}^{2}$ is apparent horizon area and $A_{0}$ is the
Planck area. The exponent $\delta$ ranges as $0\leq\delta\leq 1$ and stands
for the amount of the quantum-gravitational deformation effects Barrow . When
$\delta=0$, the area law is restored and $A_{0}\rightarrow 4G$, while
$\delta=1$ represents the most intricate and fractal structure of the horizon.
Using the thermodynamics-gravity conjecture, the differential form of the
Friedmann equation derived from the first law of thermodynamics (6), based on
Barrow entropy, is given by (see SheB1  for details)

$-\frac{2+\delta}{2\pi
A_{0}}\left(\frac{4\pi}{A_{0}}\right)^{\delta/2}\frac{d\tilde{r}_{A}}{\tilde{r}_{A}^{3-\delta}}=\frac{d\rho}{3}.$
(9)

After integration, we find the modified first Friedmann equation in Barrow
cosmology as

$\left(H^{2}+\frac{k}{a^{2}}\right)^{1-\delta/2}=\frac{8\pi G_{\rm
eff}}{3}\rho+\frac{\Lambda}{3},$ (10)

where $\Lambda$ is a constant of integration which can be interpreted as the
cosmological constant, and $G_{\rm eff}$ stands for the effective Newtonian
gravitational constant,

$G_{\rm
eff}\equiv\frac{A_{0}}{4}\left(\frac{2-\delta}{2+\delta}\right)\left(\frac{A_{0}}{4\pi}\right)^{\delta/2}.$
(11)

If we define $\rho_{\Lambda}={\Lambda}/(8\pi G_{\rm eff})$, Eq. (10), can be
rewritten as

$\left(H^{2}+\frac{k}{a^{2}}\right)^{1-\delta/2}=\frac{8\pi G_{\rm
eff}}{3}(\rho+\rho_{\Lambda}).$ (12)

Combining the modified Friedmann equation (12) with the continuity equation
(4), we arrive at

$\displaystyle(2-\delta)\frac{\ddot{a}}{a}\left(H^{2}+\frac{k}{a^{2}}\right)^{-\delta/2}+(1+\delta)\left(H^{2}+\frac{k}{a^{2}}\right)^{1-\delta/2}$
$\displaystyle=-8\pi G_{\rm eff}(p+p_{\Lambda}),$ (13)

where $p_{\Lambda}=-{\Lambda}/(8\pi G_{\rm eff})$. This is the second modified
Friedmann equation governing the evolution of the universe based on Barrow
entropy. In the limiting case where $\delta=0$ ($G_{\rm eff}\rightarrow G$),
Eqs. (12) and (13) reduce to the Friedmann equation in standard cosmology.

As usual, we define the density parameters as

$\displaystyle\Omega_{i}=\frac{\rho_{i}}{\rho_{c}},\ \ \
\rho_{c}=\frac{3H^{2-\delta}}{8\pi G_{\rm eff}}.\ \ $ (14)

Therefore, in terms of the density parameters, the first Friedmann equation
(12) can be written as

$\displaystyle\Omega_{m}+\Omega_{\Lambda}=(1+\Omega_{k})^{1-\delta/2}.$ (15)

where the curvature density parameter is defined as usual,
$\Omega_{k}=k/(a^{2}H^{2})$.

Now we are going to show that the modified Friedmann equations derived in Eq.
(12) can describe the late time accelerated expansion only in the presence of
DE (cosmological constant). For a flat universe filled with pressureless
matter ($p=p_{m}=0$) and cosmological constant, we have
$\Omega_{m}+\Omega_{\Lambda}=1.$ The total EoS parameter can be written

$\displaystyle\omega_{\rm
t}=\frac{p_{\Lambda}}{\rho_{m}+\rho_{\Lambda}}=-\frac{1}{\rho_{m}/\rho_{\Lambda}+1},$
(16) $\displaystyle\Rightarrow\omega_{\rm
t}(z)=-\frac{\Omega_{\Lambda,0}}{(1-\Omega_{\Lambda,0})(1+z)^{3}+\Omega_{\Lambda,0}},$

where $p_{\Lambda}/\rho_{\Lambda}=-1$, $\rho_{\Lambda}=\rho_{\Lambda,0}$, and
$\rho_{m}=\rho_{m,0}(1+z)^{3}$. If we take $\Omega_{\Lambda,0}\simeq 0.7$ and
$\Omega_{m,0}\simeq 0.3$, we have

$\displaystyle\omega_{\rm t}(z)=-\frac{0.7}{0.7+0.3(1+z)^{3}}.$ (17)

At the present time where $z\rightarrow 0$, we have $\omega_{\rm t}=-0.7$,
while at the early universe where $z\rightarrow\infty$, we get $\omega_{\rm
t}=0$. This implies that at the early stages, the universe undergoes a
decelerated phase while at the late time it experiences an accelerated phase.

(a) (b) (c)

Figure 1: The evolution of the $\Omega_{D}(z)$, $q(z)$ and $w_{tot}(z)$
parameters for the original ADE in Barrow cosmology for different values of
$\delta$ parameter.

(a) (b) (c)

Figure 2: The evolution of the $\Omega_{D}(z)$, $q(z)$ and $w_{tot}(z)$
parameters for the original ADE in Barrow cosmology for different values of
$b^{2}$ parameter.

##  III THE ORIGINAL ADE IN Barrow cosmology

If we consider a time dependent EoS parameter for DE, we need to replace
$\rho_{\Lambda}$ with $\rho_{de}$. In this section, we consider ADE as a time
varying candidate of DE. The energy density of the original ADE is given by
Cai1

$\rho_{de}=\frac{3c^{2}m_{p}^{2}}{T^{2}},$ (18)

where $c$ is a constant and $T$ is the age of the universe,

$T=\int{dt}=\int_{0}^{a}{\frac{da}{Ha}}.$ (19)

Consider a flat FRW universe filled with pressureless matter and ADE. The
first Friedmann equation (12) can be written

$H^{2-\delta}=\frac{1}{3M^{2}_{\rm eff}}(\rho_{m}+\rho_{de}),$ (20)

where we have defined $M^{2}_{\rm eff}=(8\pi G_{\rm eff})^{-1}$. When entropy
is changed, the energy density of ADE get modified as well. The modified
energy density of ADE, inspired by Barrow entropy, is given by Emm1  for

$\rho_{de}=\frac{3c^{2}M^{2}_{\rm eff}}{T^{2-\delta}}.$ (21)

For generality, we consider the interaction between DM and ADE. In this case,
they satisfy the semi-conservation equations,

$\displaystyle\dot{\rho}_{m}+3H\rho_{m}=Q,$ (22)
$\displaystyle\dot{\rho}_{D}+3H\rho_{D}(1+w_{de})=-Q,$ (23)

where $w_{de}$ is the EoS parameter of ADE and $Q=3b^{2}H(\rho_{de}+\rho_{m})$
is the interaction term, with $b^{2}$ being a coupling constant.

Using Eq. (21) and $\rho_{c}=3M^{2}_{\rm eff}H^{2-\delta}$, the density
parameter $\Omega_{de}$ can be written

$\displaystyle\Omega_{de}=\frac{\rho_{de}}{\rho_{c}}=\frac{c^{2}}{(HT)^{2-\delta}}.$
(24)

Taking the time derivative of (21), we find

$\displaystyle\dot{\rho}_{de}=\frac{\delta-2}{T}\rho_{de}$ (25)

Substituting (25) into (23) after using relation (24), we find the EoS
parameter of ADE in Barrow cosmology,

$\displaystyle
w_{de}=-1+\frac{2-\delta}{3}\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}-b^{2}\Omega^{-1}_{de}.$
(26)

Taking the time derivative of both side of the Friedman equation (20) and
using Eqs. (23) and (24), it is easy to find that

$\displaystyle\frac{\dot{H}}{H^{2}}=-\frac{3}{2-\delta}(1+\Omega_{de}w_{de}).$
(27)

Using (26), the above relation can be written

$\displaystyle\frac{\dot{H}}{H^{2}}=-\frac{3}{2-\delta}(1-\Omega_{de})+\frac{3b^{2}}{2-\delta}-\Omega_{de}\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}.$

Also, if we take the time derivative of (24), after using (27) and
$\dot{\Omega}_{de}=H{\Omega^{\prime}_{de}}$, we find

$\displaystyle{\Omega^{\prime}_{de}}=\Omega_{de}\Bigg{\\{}3(1+\Omega_{de}w_{de})-(2-\delta)\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}\Bigg{\\}},$

where the prime denotes derivative with respect to $x=\ln{a}$. Substituting
$w_{de}$ from Eq. (26) into the above relation, yields

${\Omega^{\prime}_{de}}=\Omega_{de}(1-\Omega_{de})\Bigg{\\{}3-(2-\delta)\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}\Bigg{\\}}-3b^{2}\Omega_{de}.$
(30)

The total EoS parameter is given by

$\displaystyle
w_{\mathrm{tot}}=\frac{p_{de}}{\rho_{m}+\rho_{de}}=\Omega_{de}w_{de}$
$\displaystyle=\Omega_{de}\left[-1+\frac{2-\delta}{3}\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}\right]-b^{2}.$
(31)

For completeness, we give the deceleration parameter

$\displaystyle q=-\frac{\ddot{a}}{aH^{2}}=-1-\frac{\dot{H}}{H^{2}},$ (32)

which combined with the Hubble parameter and the dimensionless density
parameters form a set of useful parameters for the description of the
astrophysical observations. Substituting Eq. (27) in Eq. (32) we get

$\displaystyle q$ $\displaystyle=$
$\displaystyle-1+\frac{3}{2-\delta}(1-\Omega_{de})+\Omega_{de}\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}$
(33) $\displaystyle-\frac{3b^{2}}{2-\delta}.$

We have plotted the evolutions of $\Omega_{de}$, $q$ and $w_{\mathrm{tot}}$
versus redshift $z$ for interacting original ADE in Barrow cosmology for
different values of $\delta$ and $b^{2}$ in Figs. 1 and 2, respectively. And
in all Figs., the $\delta=0$ case is also considered so that the model can be
compared with the standard model. In all figures, we have taken
$\Omega^{0}_{de}=0.73$ in agreement with observations and $c=3$.

In figs. 1(a) and 2(a), it can be easily seen that at the early universe,
$(z\rightarrow\infty)$, we get $\Omega_{de}\rightarrow 0$, while at the late
time $(z\rightarrow-1)$, we have DE dominated era; namely,
$\Omega_{de}\rightarrow 1$.

We have shown the behavior of the deceleration parameter $q(z)$ vs $z$ in
Figs. 1(b) and 2(b). From these figures we find out that, in both cases, our
Universe has a transition from deceleration to the acceleration phase around
$0.6<z<0.9$ which is compatible with observations Daly ; Komatsu ; Salvatelli
. Moreover, we observe that by decreasing $\delta$ (increasing $b^{2}$), the
transition happens earlier (higher redshift), while the value of the
deceleration parameter at the present time, $q_{0}$, decreases with increasing
$b^{2}$ (decreasing $\delta$). For $\delta=0.3$, the transition occurs around
$z_{\rm trn}=0.63$ which is the best fit with observational data. In general,
as one can see the behavior of the deceleration parameter for $0<\delta<0.5$
is more consistent with the cosmological observation than the $\delta=0$ case
(standard ADE).

In Figs.1(c) and 2(c), we have plotted the behavior the total EoS parameter,
$w_{\mathrm{tot}}(z)$, for the different values of $\delta$ and $b^{2}$,
respectively. It is obvious that the total EoS parameter, according to the
cosmological observations, can explain the evolution of the universe, as at
the early times $(z\rightarrow\infty)$ we have $w_{\mathrm{tot}}\rightarrow
0$, it means that the pressureless DM is dominant, and then it gets the
quintessence regime $(-1<w_{\mathrm{tot}}<-1/3)$ from the present period to
the late time.

Figure 3: The evolution of the $r(z)$ and $r(q)$ for original ADE in Barrow
cosmology for different values of $\delta$ parameter.

Figure 4: The evolution of the $r(z)$ and $r(q)$ for original ADE in Barrow
cosmology for different values of $b^{2}$ parameter.

Furthermore, in order to find more sensitive discriminator of the expansion
rate, we consider the statefinder parameters which contain the third
derivative of the scale factor. The statefinder pair $\\{r,s\\}$ is defined as
GhaffariRevD

$\displaystyle
r=\frac{\dddot{a}}{aH^{3}}=1+3\frac{\dot{H}}{H^{2}}+\frac{\ddot{H}}{H^{3}}$
(34) $\displaystyle s=\frac{r-1}{3(q-1/2)}.$ (35)

Since the statefinder parameters include the higher derivative of the scale
factor, it can be a fine tool to distinguish different DE models. It is worth
noting that the statefinder parameter $\\{r,s\\}=\\{1,0\\}$ corresponds to the
$\Lambda CDM$ model and it can be used to distinguish the difference between a
given DE model from the $\Lambda CDM$ model in the $(r-s)$ plane.

Taking the time derivative of both sides of Eq. (27) and using Eq. (26), we
obtain the statefinder pair $\\{r,s\\}$ for the original ADE in Barrow
cosmology as

$\displaystyle
r=1-\frac{9}{2-\delta}\left(1+\Omega_{de}w_{de}\right)\left(1-\frac{2}{2-\delta}(1+\Omega_{de}w_{de})\right)$
$\displaystyle-\frac{3\Omega^{\prime}_{de}}{2-\delta}\Biggl{\\{}w_{de}+\frac{1}{3}\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}+\frac{b^{2}}{\Omega_{de}}\Biggr{\\}},$
(36)

and

$\displaystyle
s=\Biggl{\\{}\frac{-9}{2-\delta}\left(1+\Omega_{de}w_{de}\right)\left(1-\frac{2}{2-\delta}(1+\Omega_{de}w_{de})\right)$
$\displaystyle-\frac{3\Omega^{\prime}_{de}}{2-\delta}\Biggl{[}w_{de}+\frac{1}{3}\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}+\frac{b^{2}}{\Omega_{de}}\Biggr{]}\Biggr{\\}}\times$
$\displaystyle\Biggl{\\{}-\frac{3}{2}+\frac{3}{2-\delta}(1-\Omega_{de}-b^{2})+\Omega_{de}\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}\Biggr{\\}}^{-1}.$

The evolutionary behavior of the statefinder parameter $r(z)$ and $r(q)$ are
plotted in Figs. 3 and 4 for the original ADE in Barrow cosmology for
different values of the $\delta$ and $b^{2}$ parameters, respectively. It
should be noted that in all figures, the black solid horizontal line denotes
the evolution of the $\Lambda CDM$ mode.

From Figs. 3 and 4 we observe that the behavior of the parameters $r(z)$ and
$r(q)$ distinct from $\Lambda CDM$ model during the evolution of the universe.

##  IV THE NEW MODEL OF ADE IN Barrow cosmology

Another version of ADE has been also investigated in the literatures where
instead of the age of the universe, the conformal time $\eta$ is chosen as IR
cutoff. Taking this into account, the energy density of the new ADE is given
by

$\rho_{de}=\frac{3c^{2}m_{p}^{2}}{\eta^{2}},$ (38)

where the conformal time is given by

$\eta=\int{\frac{dt}{a}}=\int_{0}^{a}{\frac{da}{Ha^{2}}},$ (39)

and hence $\dot{\eta}=1/a$. Based on Barrow entropy and motivated by (38), we
define the energy density of the new ADE in Barrow cosmology as

$\rho_{de}=\frac{3c^{2}M^{2}_{\rm eff}}{\eta^{2-\delta}},$ (40)

In order to study the cosmological implications of this model, we first derive
the EoS parameter. Taking the time derivative of (38) we find

$\displaystyle\dot{\rho}_{de}=\frac{\delta-2}{\eta a}\rho_{de}.$ (41)

The density parameter in this case reads

$\displaystyle\Omega_{de}=\frac{\rho_{de}}{\rho_{c}}=\frac{c^{2}}{(H\eta)^{2-\delta}}.$
(42)

Substituting Eq. (41) into the continuity equation (23), after some
calculations and using relation (42) we reach

$\displaystyle
w_{de}=-1+\frac{2-\delta}{3a}\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}-b^{2}\Omega^{-1}_{de}.$
(43)

In this case, it is a matter of calculations to show that the evolutions of
$\Omega_{de}$ and $q$ are governed by the following equations

${\Omega^{\prime}_{de}}=\Omega_{de}(1-\Omega_{de})\Bigg{\\{}3-\frac{2-\delta}{a}\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}\Bigg{\\}}-3b^{2}\Omega_{de},$
(44) $\displaystyle
q=-1+\frac{3}{2-\delta}(1-\Omega_{de})+\frac{\Omega_{de}}{a}\left(\frac{\Omega_{de}}{c^{2}}\right)^{1/(2-\delta)}-\frac{3b^{2}}{2-\delta}.$

In Figs. 5 and 6, we have plotted the evolutions of $\Omega_{de}$, $q$ and
$w_{\mathrm{tot}}$ versus $z$ for interacting new ADE in Barrow cosmology
where we fix $\Omega_{de}(z=0)=\Omega_{de_{0}}=0.73$ (in agreement with
observations), $c=1.7$.

The evolution of the dimensionless density parameter $\Omega_{de}(z)$ is shown
in Figs. 5(a) and 6(a) for new ADE model. These figures confirm that we have a
DM dominated universe $(\Omega_{de}\rightarrow 0)$ at the early stages of the
universe, and the DE dominated universe $(\Omega_{de}\rightarrow 1)$ at the
late time which is consistent with the cosmological observations.

The behavior of the deceleration parameter $q(z)$ is depicted in Figs. 5(b)
and 6(b). As one can see, the universe undergoes a phase transition from the
deceleration to an acceleration within the interval(around) $0.7<z_{tr}<1$. We
observe that for larger values of $\delta$ parameter (smaller values of
$b^{2}$), the phase transition occurs at the lower redshifts. We also find out
that the deceleration parameter for new ADE model in Barrow cosmology tends to
$-1$ at the late time and this is consistent with a de-Sitter expansion phase.

The behavior of the total EoS parameter are shown in Figs. 5(c) and 6(c), for
the different values of $\delta$ and $b^{2}$, respectively. From these figures
we see that at the early times, we have $w_{\mathrm{tot}}\rightarrow 0$ which
means that the universe is in the pressureless DM dominated era while at the
late time, we have $w_{\mathrm{tot}}\rightarrow-1$ indicates the ending of the
universe is in a Big-Rip singularity.

(a) (b) (c)

Figure 5: The evolution of the $\Omega_{D}(z)$, $q(z)$ and $w_{tot}(z)$ for
the new ADE in Barrow cosmology for different values of $\delta$ parameter.

(a) (b) (c)

Figure 6: The evolution of the $\Omega_{D}(z)$, $q(z)$ and $w_{tot}(z)$ for
the new ADE in Barrow cosmology for different values of $b^{2}$ parameter.

By taking the time derivative of Eq. (27) and using Eq. (43), one can obtain
the statefinder pair $\\{r,s\\}$ for the new ADE model in the Barrow
cosmology. Since this expression is too long, for the economic reason, we
shall not present it here, and only plot its evolution in Figs. 7 and 8.

Figure 7: The evolution of the $r(z)$ $r(q)$ and $r(s)$ for the new ADE in
barrow cosmology for different values of $\delta$ parameter.

Figure 8: The evolution of the $r(z)$ $r(q)$ and $r(s)$ for the new ADE in
barrow cosmology for different values of $b^{2}$ parameter.

We have also studied the behavior of $r(z)$, $r(s)$ and $r(q)$, for the new
ADE in Barrow cosmology and for different values of $\delta$ and $b^{2}$
parameters in Figs. 7 and 8, respectively. It should be noted that in $r(z)$
and $(r-q)$ figures, the black solid horizontal line denotes the evolution of
the $\Lambda CDM$ mode and the fixed point presented by the star in $r-s$
panel is the $\Lambda CDM$ model. From these figures we find that the
evolutionary behavior of the first statefinder parameter $r(z)$ and $(r-q)$,
for new ADE in Barrow cosmology for different values of the model parameters,
is well distinct from $\Lambda CDM$ model in the region $-0.5<z<3$, but it can
not be distinguished from $\Lambda CDM$ model in the region $-1<z<-0.5$. In
Figs. 7 and 8, $(r-q)$ plans show that both $\Lambda CDM$ model and the new
ADE model start evolving from the same point $(r,q)=(1,0.5)$ at the early
time, which indicate the DM dominated area, and they tend to the
$(r,q)=(1,-1)$ in the future which correspond to the de-Sitter expansion.

Finally, from the $r-s$ plans, one can observe that the evolutionary
trajectories of $(r-s)$ start from $(r>1,s<0)$ in the past, then it enters the
quintessence region, i.e. $(r<1,s>0)$ at the present time and finally their
evolution ends at the $\Lambda CDM$ point $(r,s)=(1.0)$ in the future for
different values of the model parameters.

##  V Conclusions and discussions

Inspired by the modified Barrow entropy, we have revisited the ADE model in
the background of Barrow cosmology. We have explored the effects of the Barrow
parameter $\delta$ on the evolution of the cosmological parameters for both
original and new model of ADE. To this end, we derived the differential
equation for the evolution of dimensionless ADE density parameter and used its
numerical solution in order to study the behavior of the corresponding
cosmological parameters. In all graphs we have set initial condition
$\Omega_{de}(z=0)=\Omega_{de_{0}}=0.73$ in agreement with observations.

From the behavior of deceleration parameter $q$, we have found that our
universe, for both original ADE model an new ADE model in the Barrow
cosmology, has a phase transition from early decelerated phase to the present
accelerated phase for all choices of the model parameter $\delta$. For
$\delta=0.3$ this transition happens around $z_{\rm tr}\approx 0.63$ which
shows a good compatibility with recent studies $0.5<z_{\rm tr}<1$. We have
also observed from evolution of the $q(z)$ panel in Figs. 1 and 5 that for
smaller values of $\delta$ parameter, transition from decelerated to
accelerated phases occurs at the earlier universe (higher redshifts). It is
noteworthy that for $0<\delta<0.5$ the behavior of the deceleration parameter,
for both original and new ADE models, is more consistent with the cosmological
observation than $\delta=0$ (standard ADE model).

We observed that the total EoS parameter $w_{\rm tot}$ starts from DE
dominated universe( $w_{\rm tot}=0$) at the early
universe($z\rightarrow\infty$) then enters to the quintessence area
($-1<w_{\rm tot}<-1/3$) around the present time and finally converges to the
cosmological constant( $w_{\rm tot}=-1$ ) at the late time independent of the
value of $\delta$ parameter.

In the end, for the more clear discrimination among these DE models, we used
the statefinder pair, which include a higher-order time derivative of the
scale factor. By study the behavior of the $r(z)$, $r-q$ and $r-s$, we have
found out that both original ADE and new ADE models, are completely
distinguishable from the $\Lambda CDM$ model during the evolution of the
universe. As we have observed in Figs. 7 and 8, for the new ADE in Barrow
cosmology, all trajectories of $r-q$ and $r-s$, start from the DM dominated at
the early universe then enter to the DE dominated universe (quintessence
region) and finally end to the $\Lambda CDM$ at the future time.

## References

  * (1) R.G. Cai and S.P. Kim, First law of thermodynamics and Friedmann equations of Friedmann-Robertson-Walker universe, JHEP 02, 050 (2005) [arXiv:hep-th/0501055].
  * (2) R. G. Cai and N. Ohta, Horizon Thermodynamics and Gravitational Field Equations in Horava-Lifshitz Gravity, Phys. Rev. D 81, 084061 (2010) [arXiv:0910.2307]
  * (3) R. G. Cai, L. M. Cao and Y. P. Hu, Corrected entropy-area relation and modified Friedmann equations, JHEP 08, 090 (2008), [arXiv:0807.1232].
  * (4) M. Akbar and R.G. Cai, Friedmann equations of FRW universe in scalar-tensor gravity, f(R) gravity and first law of thermodynamics, Phys. Lett. B 635(1) , 7 (2006) [arXiv:hep-th/0602156].
  * (5) M. Akbar and R. G. Cai, Thermodynamic Behavior of Friedmann Equations at Apparent Horizon of FRW Universe, Phys. Rev. D 75, 084003 (2007) [arXiv:hep-th/0609128].
  * (6) M. Akbar and R. G. Cai, Friedmann equations of FRW universe in scalar-tensor gravity, f(R) gravity and first law of thermodynamics, Phys. Lett. B 635, 7 (2006) [arXiv:hep-th/0602156].
  * (7) A. Sheykhi, Modified Friedmann equations from Tsallis entropy, Phys. Lett. B 785, 118 (2018) [arXiv:1806.03996] .
  * (8) A. Sheykhi, B. Wang and R. G. Cai, Thermodynamical Properties of Apparent Horizon in Warped DGP Braneworld, Nucl. Phys. B 779, 1 (2007) [arXiv:hep-th/0701198].
  * (9) A. Sheykhi, B. Wang, and R. Cai, Deep connection between thermodynamics and gravity in Gauss-Bonnet braneworlds, Phys. Rev. D 76, 023515 (2007) [arXiv:hep-th/0701261].
  * (10) A. Sheykhi and B. Wang, Generalized second law of thermodynamics in Gauss-Bonnet braneworld, Physics Letters B 678(5) , 434 (2009) [arXiv:0811.4478].
  * (11) A. Sheykhi, Entropic corrections to Friedmann equations, Phys. Rev. D 81, 104011 (2010), [arXiv:1004.0627].
  * (12) G. t Hooft, gr-qc/9310026;
L. Susskind, J. Math. Phys. 36 (1995) 6377\.

  * (13) W. Fischler, L. Susskind,hep-th/9806039.
  * (14) A. Cohen, D. Kaplan, A. Nelson, Effective Field Theory, Black Holes, and the Cosmological Constant, Phys. Rev. Lett. 82, 4971 (1999), [arXiv:hep-th/9803132].
  * (15) R. G. Cai, A Dark Energy Model Characterized by the Age of the Universe, Phys. Lett. B 657, 228 (2007), [arXiv:0707.4049].
  * (16) M. Maziashvili, Space-time in light of Karolyhazy uncertainty relation, Int. J. Mod. Phys. D 16, 1531 (2007), [arXiv:gr-qc/0612110].
  * (17) H. Wei and R. G. Cai, Interacting gegraphic dark energy, Eur. Phys. J. C 59, 99 (2009) [arXiv:0707.4052].
  * (18) H. Wei and R. G. Cai, A new model of agegraphic dark energy, Phys. Lett. B 660, 113 (2008), [arXiv:0708.0884].
  * (19) J. Cui, et al., New agegraphic dark energy as a rolling tachyon, Chin. Phys. B 19, 019802 (2010) [arXiv:0902.0716];
  * (20) Y. W. Kim, et al., New agegraphic dark energy model with generalized uncertainty principle, Mod. Phys. Lett. A 23, 3049 (2008), [arXiv:0803.0574].
  * (21) A. Sheykhi, Interacting agegraphic dark energy models in non-flat universe, Phys. Lett. B 680, 113 (2009) [arXiv:0907.5144].
  * (22) A. Sheykhi, Interacting new agegraphic dark energy in non-flat Brans-Dicke cosmology, Phys. Rev. D 81, 023525 (2010) [arXiv:0908.0606].
  * (23) M. R. Setare, Interacting new agegraphic Phantom model of dark energy in non-flat universe, [arXiv:0907.4910].
  * (24) H. Wei and R. G. Cai, Cosmological constraints on new agegraphic dark energy, Phys. Lett. B 663, 1 (2008) [arXiv:0708.1894].
  * (25) Mubasher Jamil, Kayoomars Karami, Ahmad Sheykhi, Restoring New Agegraphic Dark Energy in RS II Braneworld, Int. J. Theor. Phys. 50, 3069 (2011), [arXiv:1005.0123].
  * (26) M. Abdollahi Zadeh, A. Sheykhi, H. Moradpour, Tsallis Agegraphic Dark Energy Model, Mod. Phys. Lett. A 34, No. 11, 1950086 (2019), [arXiv:1810.12104].
  * (27) Pankaj, Bramha Dutta Pandey, P. Suresh Kumar, Umesh Kumar Sharma, New Tsallis Agegraphic Dark Energy, [arXiv:2205.01095].
  * (28) P. S. Kumar, et. al., Kaniadakis Agegraphic Dark Energy, [arXiv:2205.03272].
  * (29) J. D. Barrow, The area of a rough black hole, Phys. Lett. B 808, 135643 (2020) [arXiv:2004.09444].
  * (30) E. N. Saridakis, Barrow holographic dark energy, Phys. Rev. D 102, 123525 (2020), [arXiv:2005.04115].
  * (31) F. K. Anagnostopoulos, S. Basilakos, E. N. Saridakis, Observational constraints on Barrow holographic dark energy, Eur. Phys. J. C 80, 826 (2020), [arXiv:2005.10302].
  * (32) E. N. Saridakis, Modified cosmology through spacetime thermodynamics and Barrow horizon entropy, JCAP 07, 031 (2020), [arXiv:2006.01105].
  * (33) A. Sheykhi, Barrow entropy corrections to Firedmann equations, Phys. Rev. D 103, 123503 (2021), [arXiv:2102.06550].
  * (34) A. Sheykhi, Modified cosmology through Barrow entropy, Phys. Rev. D 107, 023505 (2023), [arXiv:2210.12525].
  * (35) A. Sheykhi, M. Sahebi Hamedan, Holographic Dark Energy in Modified Barrow Cosmology, Entropy 25, 569 (2023), [arXiv:2211.00088].
  * (36) S. Nojiri, S. D. Odintsov and V. Faraoni, From nonextensive statistics and black hole entropy to the holographic dark universe, Phys. Rev. D 105, 044042 (2022), [arXiv:2201.02424].
  * (37) S. Nojiri, Sergei D. Odintsov, T. Paul, Early and late universe holographic cosmology from a new generalized entropy, Phys. Lett. B 831, 137189 (2022), [arXiv:2205.08876].
  * (38) E. N. Saridakis and S. Basilakos, The generalized second law of thermodynamics with Barrow entropy, Eur. Phys. J. C 81, 644 (2021), [arXiv:2005.08258].
  * (39) E. M. C. Abreu, et. al., Barrow’s black hole entropy and the equipartition theorem, Eur. Phys. Lett. 130, 40005 (2020), [arXiv:2005.11609].
  * (40) A. A. Mamon, et. al., Dynamics of an Interacting Barrow Holographic Dark Energy Model and its Thermodynamic Implications , The European Physical Journal Plus 136, 134 (2021), [arXiv:2007.16020].
  * (41) E. M. C. Abreu, J. A. Neto, Barrow black hole corrected-entropy model and Tsallis nonextensivity, Phys. Lett. B 810, 135805 (2020), [arXiv:2009.10133].
  * (42) J. D. Barrow, S.s Basilakos, E. N. Saridakis, Big Bang Nucleosynthesis constraints on Barrow entropy, [arXiv:2010.00986].
  * (43) S. Srivastava, U. K. Sharma, Barrow holographic dark energy with Hubble horizon as IR cutoff, Int. J. Geom. Methods Mod. Phys. Vol. 18, No. 1, 2150014 (2021), [arXiv:2010.09439].
  * (44) B. Das, B. Pandey, A study of holographic dark energy models using configuration entropy, [arXiv:2011.07337].
  * (45) A. Pradhan, A. Dixit, V. K. Bhardwaj, Barrow HDE model for Statefinder diagnostic in FLRW Universe, Int. J. Mod. Phys. A, Vol. 36, No. 4, 2150030 (2021), [arXiv:2101.00176].
  * (46) S. Nojiri, S. D. Odintsov and T. Paul Different Faces of Generalized Holographic Dark Energy, Symmetry 13, 928 (2021), [arXiv:2105.08438].
  * (47) S. Nojiri, S. D. Odintsov and T. Paul, Barrow entropic dark energy: A member of generalized holographic dark energy family, Phys. Lett. B 825, 136844 (2022), [arXiv:2112.10159].
  * (48) U. K. Sharma, G. Varshney, V. C. Dubey, Barrow agegraphic dark energy , Int. J. Mod. Phys. D 30, No. 03, 2150021 (2021), [arXiv:2012.14291].
  * (49) S. A. Hayward, Unified first law of black-hole dynamics and relativistic thermodynamics, Class. Quant. Grav. 15, 3147 (1998), [arXiv:gr-qc/9710089].
  * (50) S. Ghaffari, A. Sheykhi, M.H. Dehghani, Statefinder diagnosis for holographic dark energy in the DGP braneworld Phys. Rev. D 91 (2015) 023007.
  * (51) R. A. Daly et al., improved Constraints on the Acceleration History of the Universe and the Properties of the Dark Energy, Astrophys. J. 677, 1 (2008),[arXiv:astro-ph/0710.5345].
  * (52) E. Komatsu et al., Seven-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Cosmological Interpretation [WMAP Collaboration], Astrophys. J. Suppl. 192, 18 (2011), [arXiv:astro-ph/1001.4538].
  * (53) V. Salvatelli, A. Marchini, L. L. Honorez and O. Mena, New constraints on coupled dark energy from the Planck satellite experiment, Phys. Rev. D 88, 023531 (2013), [arXiv:astro-ph/1304.7119].


# Pitfalls in Machine Learning Research:
Reexamining the Development Cycle

Stella Biderman
The AI Village
Booz Allen Hamilton
<EMAIL_ADDRESS>Walter J. Scheirer
The AI Village
University of Notre Dame
<EMAIL_ADDRESS>
###### Abstract

Machine learning has the potential to fuel further advances in data science,
but it is greatly hindered by an ad hoc design process, poor data hygiene, and
a lack of statistical rigor in model evaluation. Recently, these issues have
begun to attract more attention as they have caused public and embarrassing
issues in research and development. Drawing from our experience as machine
learning researchers, we follow the machine learning process from algorithm
design to data collection to model evaluation, drawing attention to common
pitfalls and providing practical recommendations for improvements. At each
step, case studies are introduced to highlight how these pitfalls occur in
practice, and where things could be improved.

##  1 Introduction

There is much to be excited about in the field of machine learning these days.
From championship video game AI to advances in autonomous vehicles, it seems
that no matter where we look we see machine learning transforming yet another
domain. But while the field has been taking its victory lap for these
successes, problems have surfaced from the depths of the machine learning
development cycle that potentially jeopardize the entire endeavor by producing
illusory experimental effects. In some cases these problems are systematic,
and extend beyond the existing discussion of problems related to dataset bias.
Without an adequate response to solve them, the utility of machine learning as
a constructive technology is brought into question.

At present, pitfalls can emerge at three critical points in the machine
learning development cycle:

  1. 1.

In the design process of an algorithm, when a team is formed to solve a
problem, assumptions about the problem and solution are formulated, and an
algorithm is developed.

  2. 2.

At the point of data collection for a model that will be trained using the
newly developed algorithm.

  3. 3.

During evaluation, where the model’s performance as a solution to the problem
is assessed.

There are serious technical and ethical ramifications when there is a
breakdown at any critical point in the development cycle. Illusory
experimental effects lead to machine learning papers that not only fail to
replicate, but also create serious and often difficult-to-diagnose problems
for the people who try to implement the research in practice. Recent examples
of this include algorithms for determining criminal tendencies [49, 21] and
detecting sexual orientation [47] from photos of faces, as well as commercial
services that automatically determine suitable job candidates for hiring
managers [11]. In all of these cases, the claimed functionality is not what it
appears to be due to underlying problems in experimental design, data use, and
evaluation [13, 3, 4].

It is important to note that we are not merely claiming that some research is
morally dubious. While some of the research we discuss in this paper is
morally dubious, we see the same problems in research that isn’t, including
neural architecture search, the ImageNet dataset, and route-finding
algorithms. Problems in the machine learning development cycle also appear
when machine learning is applied to other fields of science. For example, Liu
et al. [30] found that out of tens of thousands of papers on deep learning
published in the medical imaging literature, only a tiny fraction were
methodologically sound.

The objective of this paper is to identify and illustrate common pitfalls in
the machine learning development cycle. It is not to shame or embarrass
particular researchers or organizations. For every example we cite in this
paper, there are a dozen other examples that could have been given.
Furthermore, all of the pitfalls raised in this paper are problems that have
come up in our own work. The first author has withdrawn multiple submitted
papers after issues raised in this paper were brought to their attention and
has had to explain to end-users that models they developed could not be put
into practice due to methodological shortcomings. The second author has worked
on data-driven modeling for years, and regrets not always scrutinizing data
sources and delivering evaluations that could have been more rigorous.

While some of the issues are things that individual researchers can address in
their work, many are not. Even when the problems can be addressed
individually, the culture of machine learning research fails to enforce
important behavioral norms. We aim to fix that.

The bulk of the rest of this paper traces the development cycle of machine
learning research. We begin by discussing algorithm design, before moving on
to data collection and finally model evaluation. Within each topic we raise
some of the pressing problems with current machine learning practice,
elucidate these problems with case studies, and then present recommendations
to improve the current practice of the field. In the final section of this
paper we discuss takeaways, dividing up suggestions for improvement into
short-term, medium-term, and long-term goals.

##  2 Designing the Right Algorithm

Before one can answer the question “Am I doing the research right?” one must
first tackle the question “Am I doing the right research?” Although ethical
issues with machine learning research have attracted increasing attention
recently, we contend that many projects suffer from fundamental design flaws
that make them — even on a purely technical level — non-starters from the
beginning.

###  2.1 Problems with Algorithm Design

Not Engaging with Stakeholders. Modern machine learning and data science has
national and even global repercussions, but researchers rarely admit the scope
of the interests involved. While progress has been made in getting researchers
and engineers to identify users as stakeholders, as machine learning and data
science technologies become increasingly popular for public use the importance
of recognizing anyone who may interact with the system as a stakeholder
becomes crucial. For example, the stakeholders in FBI predictive policing
algorithms [17] include every person who will set foot in the U.S. while the
algorithms are in use and the stakeholders in the routing algorithms in Google
Maps and Waze include the people whose neighborhoods and lives are disrupted
by having drivers rerouted by their homes [31, 29]. Existing notions of
participatory design in machine learning are not sufficient to address these
sorts of situations.

Ignoring Assumptions. Algorithms necessarily rely on oversimplified models of
the world, but not enough attention is paid to the assumptions that are
designed into models and the impacts that they have on the results. This is a
perpetual problem in computer science, famously highlighted a decade ago in
the blog post “Falsehoods Programmers Believe About Names” [32]. Assumptions
that are built into algorithms and datasets (often implicitly) can influence
the results of research in ways that fundamentally undermine its credibility
[24, 44] or dissuade people from pursuing certain types of research [5, 9].

Lack of Oversight. When most scientists seek to do a study that impacts
humans, they are required to obtain approval by an Institutional Review Board
(IRB) whose job it is to protect the rights and interests of the subjects of
the study. Unfortunately, most applied machine learning and data science
research is exempt from requiring IRB approval because it analyzes preexisting
data. This often means that machine learning research is not reviewed by a
third party until it is given to peer-reviewers, a point far too late to make
meaningful changes to the design of the experiment or algorithm.

###  2.2 Case study: Gender and Machine Learning

Automated gender recognition (AGR) is a textbook example of how assumptions
made in the design process can invalidate research. One example of this is how
they operationalize gender without attending to the direction of causation.
There are hundreds of papers that claim to be able to accurately determine
people’s gender based on anything from a writing sample [6, 52, 18], to their
gait [53, 14, 16], to the texture of their skin [50, 27, 1]. Regardless of
what one thinks constitutes “gender,” almost nobody believes that one’s gender
is determined by their handwriting, gait, or skin care regimen. Indeed, many
of the study authors explicitly acknowledge this disconnect before proceeding
to ignore it.

The typical justifications of these methodologies uses a V-shaped causation
pattern: something causes gender, that same thing causes the attribute
measured, and therefore we can infer things about gender from the measurement.
This is invalid reasoning, but it is highly prevalent in machine learning
research. When these algorithms are being used to prescriptively assign gender
labels to people, something that all of the mentioned papers cite as an
application of their work, it is not sufficient to invoke correlation instead
of causation as a justification of the effect. None of the referenced papers
employ sufficient controls to make any sort of causal claims111The skin
texture papers do not even control for skin-care regimen..

AGR research also highlights how ontological assumptions can interfere with
research. According to Keyes [24], 94.8% of AGR papers define gender in a
binary fashion with no acknowledgment of the existence of transgender people.
By prescriptively defining the bins that people must fall into, the algorithm
is incapable of understanding humans beyond the scope of the author’s
conceptions. This is a very common way for researchers to imprint their own
biases into their data, and best practices in fields such as medicine [7] and
HCI [42] strongly discourage it.

###  2.3 Best Practices: Recommendations for Algorithm Design

Not every problem needs to be solved via an algorithm. Although as machine
learning researchers we are inherently biased towards trying to solve problems
with machine learning, many issues can be solved by simply not trying to solve
them at all. In a conversation about whether or not Twitter’s automatic
cropping algorithm is racially biased, Amy Zhang [55] pointed out

> The easiest fix for that biased cropping AI? No it’s not to build another
> AI – it’s to give people the power to select crop boundaries when posting a
> photo.

A similar sentiment shared by many others. Twitter took this criticism to
heart, and announced changes to cropping including giving users control over
how their images were cropped [2]

Democratize your notions of stakeholdership. Any person who produces data used
in a model, uses a model, or whose life is impacted by the outputs of a model
is a stakeholder. This especially includes groups who are222It is typical to
say “who are historically” here, but this is misleading and casts
diversification as a solved problem which it is not. undervalued and
underrepresented in the design process.

Integrate stakeholder feedback into the default ML development cycle. While
the recent paper “Participation is not a Design Fix for Machine Learning” [41]
raises important points about the limits of how participatory design is often
done, that does not mean that participatory design is not a crucial component
of the ML development cycle. It is vital that machine learning researchers
take the criticisms of their research by stakeholders seriously, making
meaningful changes to their concepts and approaches rather than
“participation-washing” their research.

Subject research to ethical review. Recent strides towards improving the
ethicality of machine learning research have been made by NeurIPS’s
requirement of a Broader Impacts statement. While this is a good first step,
ethical review needs to be taken more seriously industry-wide, and integrated
earlier in the research process to prevent significant amounts of time and
money being spent on fundamentally invalid or harmful research.

##  3 Collecting the Data

The adage “what is counted counts” has never been more true than it is today.
Troves of data are collected and stored every day, much of it to be analyzed
by machine learning algorithms. How that data is collected and what precisely
it counts is vital to understanding the resulting analysis. Yet far too often
questions of data methodology are ignored by industry and academic researchers
alike.

###  3.1 Problems with Data Collection

Statistical and Social Bias. At this point, it should not be surprising to
anyone working on machine learning to hear that datasets are heavily biased.
However, bias remains a key problem facing all fields pursuing data-driven
modeling. Biases tend to manifest in one of two ways: statistical and social.
Statistical biases can take the form of class imbalance or repeating
irregularities associated with both salient and non-salient information in
data points. While recent discussion makes this seem as though it were a newly
discovered problem, statistical bias in machine learning has been studied for
decades [26]. Social biases can be more subtle and tend to occur as artifacts
of a data collection process or as reflections of societal biases.

Not Testing on Data Collected from the Real World. The machine learning
community has decided that widely-used benchmark datasets are the best way to
evaluate an algorithm’s performance. Unfortunately this can cause assessments
of an algorithm’s reliability in a paper to be wildly different from what
happens in the real world [15]. The only way to evaluate an algorithm’s
performance in the real world is on freshly collected real world data from the
population that the algorithm will actually be applied to. The field of
robotics is one that machine learning can look to for best practices, as it
has always done this and never hesitates to point out the mismatch between
datasets and the real world [43].

The Limitations of the Big Data Paradigm. The dominant strategy in AI product
development has been to collect vast troves of data for high-capacity machine
learning models. There can be no doubt that big data has led to advances in
numerous applications, from photo tagging to autonomous vehicles. But this
approach has its limitations, some of which are only now becoming apparent.
Perhaps obvious in retrospect, “bigness” can be a liability. For many
datasets, there are too many images for manual scrutiny or meaningful
validation. Researchers have no good way to exhaustively examine each and
every datum, which can lead to problematic data samples being provided to an
algorithm during training. Recent work by Prabhu and Birhane [37] highlights
major issues with several commonly used image datasets, including ImageNet.

Noisy Labeling. Because we lean heavily on human annotators for supervised
learning, what is inherently a noisy labeling process is prone to mistakes.
Rater reliability can be assessed [39, 34], but rarely is. Nearly all datasets
in common use have a single label assigned by one person to each sample. This
does not give us any indication about the correctness or difficulty of the
sample, and can lead to problems down the line during training. The labeling
process is also prone to malicious attack. It has been shown that it is
possible to change labels in such a way that provides a favorable outcome to
an attacker (e.g., a backdoor in a trained model) [12]. The problem is
exacerbated by big data: if a small number of labels change in a sea of
millions of samples, does anybody notice? Those changes may be consequential
to model outcomes.

###  3.2 Case Study: The Tiny Images Dataset

A very recent example that reflects almost all of the above problems is the
retraction [44] of the Tiny Images dataset [46], which had been used for
object recognition research. This dataset contains very small images
($32\times 32$ pixels) for over 50,000 different noun categories, and was
meant to develop visual recognition capabilities that match the “remarkable
tolerance of the human visual system.” As of this writing, the paper
describing the dataset has been cited over 1,700 times, and numerous
algorithms have been developed using it as source data. The retraction was
prompted by an investigation by Prabhu and Birhane [37], who noted that Tiny
Images used several categories for images labeled with racial and misogynistic
slurs. Under closer scrutiny, they found that the dataset also contained non-
consensual pornography such as up-skirt photographs and imagery degrading to
various marginalized groups.

The problems that were exposed in Tiny Images map directly to the problems we
have singled out related to bias, big data, and labeling. Tiny Images most
obviously suffered from a case of social bias in its racist and misogynistic
categories. These were unfortunate reflections of Internet culture, which were
unavoidable in the collection strategy used by the creators of the dataset: an
automated data procedure that relied on nouns from WordNet [33]. This saga
also exemplifies the limitations of the big data paradigm. According to the
retraction issued by Torralba et al. “The dataset is too large (80 million
images) and the images are so small (32 x 32 pixels) that it can be difficult
for people to visually recognize its content.” Thus, it is argued that the
very advantage of big data is rendered moot by the presence of even a small
number of problematic data samples. Because it is not possible to find all of
the problematic instances in a dataset, according to Torralba et al., the only
recourse is to take a dataset out of service if problems are found. And when
it comes to labels, Tiny Images contained arbitrary label assignments that
reflected accepted and derogatory racial categories in WordNet. Moreover, the
crawling process relied on tags from the web, not multiple assignments from a
collection of annotators. Without a consensus judgment, this means the
accuracy of the labels remains unclear for many of the images. Finally, there
is a temporal aspect to the labels: assigned labels can evolve over time as
social and cultural norms change.

###  3.3 Best Practices: Recommendations for Data Collections

Hypothesis Driven Data Collection. Instead of simply hoovering up data and
then trying to use it for whatever arbitrary application that comes down the
line, a better approach would be to design a collection with a hypothesis in
mind. There is no reasonable expectation that any question can be answered by
using a generic pool of data, no matter how large. Across the natural
sciences, experimental data collection is tightly coupled with a specific
hypothesis that is formulated before work begins. The same should be true of
experiments in machine learning.

Audit Datasets. While we acknowledge that it is impossible to go exhaustively
through today’s machine learning datasets by hand, there are still some
sensible strategies for auditing that can be used. Given that most dataset
come from sites where users can upload their own content, there are fairly
obvious problems to look for in a targeted way: hate speech, profanity, and
pornography. Less obvious catches can be made by auditing the sources of the
data. Are celebrity news sites overly represented in a dataset for photo
captioning? That may lead to racial bias in operation [40]. Thus better
heterogeneity in sourcing is needed. In software engineering, a set of tests
that is not comprehensive, but still useful to reveal failure modes is known
as smoke testing. This idea transfers nicely to dataset auditing, where
feasible checks for the above items and others can made in a reasonable amount
of time. Machine learning researchers can look to other fields that face
similar problems, such as archivists [22] to develop best practices.

Quantify Annotator Uncertainty. When it comes to the quantification of
annotator uncertainty, the recommendation here is to quantify aleatoric
uncertainty [25, 23]. This is the uncertainty estimated and attempted to be
removed when aggregating data from different annotators. With a quantitative
value of uncertainty for a datapoint, a decision can be made to use or not use
that point, or perhaps weight its influence appropriately. The process can
include an assessment of how familiar annotators are with the domain, as well
the social and cultural norms associated with the type of data being
annotated.

Dataset Revision Process. The above problems mean that at some point in a
dataset’s life-cycle, it will need to revised. A revision of a dataset can
remove problematic information, document what has been removed from the
previous version, and provide an explanation for why the revision was
necessary. This may not completely remove all problems from a dataset,
especially in a big data context, but it is a way to address specific problems
as they are raised. Revision control systems for data should be developed to
ease this process. Importantly, a standard for dataset revision should be
defined and adopted by the community at large. “Datasheets for Datasets” [19]
is a framework that has attracted attention for this, but researchers and
practitioners have been slow to put this into practice.

##  4 Evaluating the Model

The field of machine learning is founded upon dataset-based evaluation. On the
one hand, using standard datasets provide a common basis for comparison across
different algorithms, as well as sufficient data for self-contained
evaluations. On the other hand, because datasets are self-contained worlds
they often misrepresent algorithm performance and can result in misleading
findings if datasets are too heavily relied on. Exacerbating these problems
are intentional or unintentional misuses of learning algorithms. Here we make
specific recommendations on how evaluation can be improved informed by common
practices in other experimental fields and previous observations from the
machine learning literature that have gone unheeded. These recommendations are
made for artificial neural networks, but in some cases can apply to other
learning algorithms as well.

###  4.1 Problems with Model Evaluation

Lack of Statistical Significance Testing. Statistically verifying the results
of a scientific paper is essential to its publication and the credibility of
its results. Despite the attention to rigor in the algorithm developement
process, many of the most popular and successful machine learning models do
not apply statistically rigorous techniques in evaluation. A machine learning
model is usually considered successful if it can exceed state-of-the-art
performance on standard datasets, but researchers rarely pay attention to the
statistical significance of their results. Making judgments based on a small
number of data points with little attention to the significance of their
findings leads researchers to incorrectly over- or under-value work. In the
worst cases, the lack of significance testing leads to a failure to notice
that methodologies do not beat basic null models for the task.

Random Seed Effects. A largely unacknowledged problem in neural network
experiments is the lack of $k$-fold testing by varying the value of the random
seed during training. Rarely do we find a paper that reports error based on
this form of evaluation. It is well known that different random
initializations of the elementary parameters can lead to drastically different
results [35]. For datasets with fixed training, validation, and testing
partitions, this presents a dilemma. An honest experimenter may get lucky or
unlucky, depending on the choice of the seed. Thus the reported results may
not reflect what happens on average for a series of training runs. A dishonest
experimenter may attempt to mine seeds for an extended period of time, looking
for a favorable starting place that leads to good results on the test set.

There are varying views on this. The random seed is a valid hyperparameter
that can be changed by the experimenter at will. However the point of training
should be to achieve good performance in the real world, not just post the
best results on a benchmark dataset. If a good result on a test set is the
product of chance and is not reproducible over many training runs, then it is
invalid. Conversely, it is possible that a poor result from a rare bad seed is
underrating the performance of an approach.

Learning Datasets. An age old problem in machine learning is the ability of an
algorithm to learn a dataset at training time, instead of a function to solve
an intended problem. Folk wisdom has been passed down to several generations
of machine learning researchers in the form of the apocryphal story of the
military neural network meant to detect tanks in images, which ended up
learning the time of day from the background [20]. Allegedly, it just so
happened that tanks only appeared under low-light conditions in the dataset,
making it appear that the tank detector was working as expected. While this
exact experiment likely never happened, it has some basis in reality. The
ability of algorithms to learn spurious correlations in a dataset leads to
illusory effects that confirm an experimenter’s hypothesis.

A real example of this problem is the recent “Criminality from Face” research
where deep learning was reported to be able to determine whether or not
somebody is a criminal based on a photo of their face [49, 21]. Bowyer et al.
[10] demonstrated that accurate results can be achieved for this task because
of the organization of the datasets used: mugshot photos from government data
sources are labeled “criminal” and ordinary public photos crawled from the web
are labeled “non-criminal.” Given the large mismatch in visual appearance
between these two sets, it’s possible to be misled by the assigned labels. In
the now classic paper an “Unbiased Look at Dataset Bias,” Torralba and Efros
[45] warned about this very problem. That paper has largely been remembered
for kicking off the study of individual biases within computer vision
datasets, but it made a more important observation about datasets as self-
contained worlds: they possess a certain visual style at a global-level, which
can easily be learned. If an experimenter is labeling the datasets to suit
their needs, either intentionally or unintentionally, this observation can be
exploited.

Lack of 3rd Party Evaluation. The academic peer review process is meant to
verify the scientific integrity of work, which includes reproducibility.
Earlier in this paper, we noted that the vast majority of published deep
learning models for medical image analysis cannot be independently verified
[30]. It is now standard practice by corporate research labs to not publish
code or data with their papers in the interest of protecting intellectual
property. This means that it is impossible to verify the claims made without
re-implementing the work described in the paper from scratch, which is
sometimes impossible if corresponding configurations are not made available to
the public, or if details have been omitted. Even when the code is made
available, replications of machine learning research often fail [38].

Aggravating this problem has been the dramatic increase in papers submitted to
machine learning-oriented publication venues. CVPR alone went from 2,123
papers in 2015 to 6,424 in 2020. Reviewers, who are overburdened even with the
average load of 5-6 papers, do not have time to work with any available code
under these conditions.

On the commercial side, companies are under no obligation to submit their
products to peer review. In other industries, it is common practice to seek
3rd party evaluation of a product for quality or standards certification. This
has rarely been the case in the AI industry. When blackbox machine learning
products are released without 3rd party evaluation, problems that might
otherwise have been identified in testing can emerge in operation. In one
example, Microsoft released a blackbox web app for photo captioning that
reproduced racial stereotypes which are prevalent in computer vision datasets
composed of celebrity photos [40]. In another case, Twitter’s blackbox photo
cropping algorithm demonstrated racial bias, and led someone to start a public
experiment on the platform [36]. Presently, we only see change when there is
large public outcry, but as machine learning-based systems proliferate, this
will not be a sustainable check on the problem.

###  4.2 Case Study: Neural Architecture Search and Randomly Wired Neural
Networks

Neural architecture search (NAS) is a field of deep learning that tries to
optimize the neural network architecture and find networks that produce better
results when trained. Unfortunately neural architecture search by and large
does not work, and many NAS researchers seem to have not noticed this fact due
to poor statistical practice.

The landmark paper Xie et al. [51] points out that that NAS papers typically
compare against each other with no external reference points or null models to
compare to. To address this gap, they train neural networks whose
computational graphs are generated randomly. To decrease the influence of any
bias or knowledge that the authors have on the null models, they decide to
generate graphs using standard random graph generators from social network
analysis. These random graph generators have the additional benefit of having
never been applied to NAS before. The surprising result was that some of the
random network generators achieved “state-of-the-art” performance.

Unfortunately, the neural architecture search community does not seem to have
learned much from — or even understood — the paper. At the time of writing Xie
et al. [51] had 144 citations. Of those, more than $75\%$ of the papers cite
it as an example of neural architecture search being effective, with more than
$25\%$ explicitly describing it as a state-of-the-art methodology.

Despite the fact that hundreds of NAS papers have been published since Xie et
al. [51], subsequent research by Li and Talwalkar [28] and White et al. [48]
has continued to validate the fact that basic null models based on random
search remain the state-of-the-art for NAS. This term itself is a bit of a
misnomer though, as any methodology that fails to outperform random networks
is at a very basic level failing to do optimization at all.

###  4.3 Best Practices for Model Evaluation

Statistically Validate Results. A better option than reporting results with
just a point estimate is to report the average and standard error of the
estimator. This can be done by varying the random seed, subsampling the test
dataset, or by hypothesis testing. While some of these techniques are
beginning to catch on, there do not currently exist cultural norms enforcing
meaningful levels of statistic rigor in published research.

Train a Dataset Classifier as a Control Model. In order to determine whether
or not a dataset or data source is being learned instead of the intended
function, our recommendation is to perform a control experiment that swaps the
target labels in the training set for dataset labels. This will lead to the
creation of a dataset classifier at training time if a dataset is being
learned. This process is especially important in cases where multiple datasets
are being used in the training set. If this detector is able to correctly
identify the source of new instances, it is likely learning the statistical
differences between datasets rather than the property of interest.

Conduct Third Party Evaluation. Authors of machine learning papers deserve
another set of eyes on their work. Conferences and journals should give
reviewers access to the exact code, data and configuration for the experiments
described in papers introducing new algorithms. Area chairs and area editors
should expect reviewers to run and tinker with the code, and referee reports
should include an analysis of reviewer experiments focusing on replication and
sensitivity to free parameters (including a change of data). This will
inevitably slow down the rate of publication, but improve the quality of
published work (and perhaps ease the burden on the conference reviewing
process). It is very much in-line with recent calls for machine learning to
join the Slow Science movement [8]. With respect to commercial products, an
organization similar to Underwriters Laboratories (https://www.ul.com/) should
be established to certify the safety and correctness of machine learning
products. There is already some precedent for this in computer security [54].

##  5 Takeaways for Doing Better Data Science

Machine learning as a field is where computer networking was in the 1990s:
something akin to the Wild West. With very little in the way of mature
protocols, standards, and best practices, numerous problems in usability and
security abounded on the Internet until the community took action. The same is
true right now in machine learning. The examples we have provided in this
paper show that the field needs to be tamed. The conversation about problems
in data-driven models needs to extend beyond dataset bias to put the
recommendations we make in this paper into practice. The community needs to
think seriously about forming standards committees. To make this happen,
cooperation will need to be fostered between corporations, academia,
governments, and civil society — something that we have not seen to date due
to competing interests.

###  5.1 Short-Term Takeaways

A short-term recommendation is one that the reader can go back to the office
with and immediately start using. The most actionable short-term solutions we
have discussed are thinking about whether or not a problem needs to be solved
via an algorithm, statistically validating results, performing hypothesis
driven data collection, auditing datasets, and training a dataset classifier
as a control model. These recommendations can be implemented on a project-by-
project basis without significant external support. Finally, although many of
our suggestions require larger scale changes than one person can accomplish
alone, you can encourage the development of the necessary norms and cultural
attitudes by asking these questions in peer review, posing them to project
leads, and making your personal commitment to them known. Even if people do
not yield to your questions, raising them and normalizing them as a point of
discussion is an important first step.

###  5.2 Medium-Term Takeaways

A medium-term recommendation is one that the reader can take back to their
company and work to achieve policy change. Medium-term solutions that we have
proposed here include democratizing notions of stakeholdership, integrating
stakeholder feedback into the default ML development cycle, subjecting
research to ethical review, and quantifying annotator uncertainty.
Implementing these solutions will require spending time to setup processes and
protocols before implementation and will potentially require engagement with
multiple levels of management within an organization.

###  5.3 Long-Term Takeaways

A long-term recommendation is one that takes the form of large-scale change
across an entire community. Long-term solutions that we have proposed here
include defining a standard for a dataset revision process, establishing
systematized third party evaluation, and developing cultural norms enforcing
the shorter-term recommendations. Putting these things into place may take
years, with preparatory workshops and committee meetings convening at major
machine learning conferences to set the agenda. These are the most ambitious
recommendations, but ones that will certainly pay off by benefiting the
largest number of stakeholders.

## References

  * Afifi [2019] Mahmoud Afifi.  11k hands: gender recognition and biometric identification using a large dataset of hand images.  _Multimedia Tools and Applications_ , 78(15):20835–20854, 2019.
  * Agrawal and Davis [2020] Parag Agrawal and Dantley Davis.  Transparency around image cropping and changes to come.  _Twitter Blog_ , 2020.
  * Agüera y Arcas et al. [2018] Blaise Agüera y Arcas, Alexander Todorov, and Margaret Mitchell.  Do algorithms reveal sexual orientation or just expose our stereotypes?  _medium.com_ , 2018.
  * Ajunwa [2019] Ifeoma Ajunwa.  Automated employment discrimination.  _Available at SSRN 3437631_ , 2019.
  * Albert et al. [2020] Kendra Albert, Jon Penney, Bruce Schneier, and Ram Shankar Siva Kumar.  Politics of adversarial machine learning.  In _Towards Trustworthy ML: Rethinking Security and Privacy for ML Workshop, Eighth International Conference on Learning Representations (ICLR)_ , 2020.
  * Argamon et al. [2009] Shlomo Argamon, Moshe Koppel, James W Pennebaker, and Jonathan Schler.  Automatically profiling the author of an anonymous text.  _Communications of the ACM_ , 52(2):119–123, 2009\.
  * Bauer et al. [2017] Greta R Bauer, Jessica Braimoh, Ayden I Scheim, and Christoffer Dharma.  Transgender-inclusive measures of sex/gender for population surveys: Mixed-methods evaluation and recommendations.  _PloS one_ , 12(5):e0178043, 2017.
  * Bengio [2020] Yoshua Bengio.  Time to rethink the publication process in machine learning.  https://yoshuabengio.org/2020/02/26/time-to-rethink-the-publication-process-in-machine-learning/, 2020\.  Accessed: 2020-9-20.
  * Biderman et al. [2020] Stella Biderman, Anima Anandkumar, Aylin Caliskan, Catherine D’Ignazio, and Ram Shankar Siva Kumar.  Ai ethics and bias panel.  The AI Village at DEF CON, 2020.  URL https://www.youtube.com/watch?v=7zswHvHR9cA.
  * Bowyer et al. [2020] Kevin W Bowyer, Michael King, and Walter Scheirer.  The criminality from face illusion.  _arXiv preprint arXiv:2006.03895_ , 2020.
  * Burke [2019] L. Burke.  Your interview with AI.  _Inside Higher Ed_ , 2019.
  * Chen et al. [2017] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song.  Targeted backdoor attacks on deep learning systems using data poisoning.  _arXiv preprint arXiv:1712.05526_ , 2017.
  * Coalition for Critical Technology [2020] Coalition for Critical Technology.  Abolish the #techtoprisonpipeline.  https://medium.com/@CoalitionForCriticalTechnology/abolish-the-techtoprisonpipeline-9b5b14366b16, 2020\.  Accessed: 2020-9-20.
  * Das and Chakrabarty [2015] Deepjoy Das and Alok Chakrabarty.  Human gait based gender identification system using hidden markov model and support vector machines.  In _International Conference on Computing, Communication & Automation_, pages 268–272. IEEE, 2015.
  * de Vries et al. [2019] Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten.  Does object recognition work for everyone?  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 52–59, 2019.
  * Do et al. [2020] Trung Dung Do, Van Huan Nguyen, and Hakil Kim.  Real-time and robust multiple-view gender classification using gait features in video surveillance.  _Pattern Analysis and Applications_ , 23(1):399–413, 2020.
  * Eidam [2016] Eyragon Eidam.  The role of data analytics in predictive policing.  _Government Technology_ , 2016.
  * Gattal et al. [2020] Abdeljalil Gattal, Chawki Djeddi, Ameur Bensefia, and Abdellatif Ennaji.  Handwriting based gender classification using cold and hinge features.  In _International Conference on Image and Signal Processing_ , pages 233–242. Springer, 2020.
  * Gebru et al. [2018] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford.  Datasheets for datasets.  _FAT ML_ , 2018.
  * Gwern [2020] Gwern.  The Neural Net Tank Urban Legend .  https://www.gwern.net/Tanks, 2020.  Accessed: 2020-9-20.
  * Hashemi and Hall [2020] Mahdi Hashemi and Margeret Hall.  Criminal tendency detection from facial images and the gender bias effect.  _Journal of Big Data_ , 7(1):1–16, 2020.
  * Jo and Gebru [2020] Eun Seo Jo and Timnit Gebru.  Lessons from archives: strategies for collecting sociocultural data in machine learning.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 306–316, 2020.
  * Kendall and Gal [2017] Alex Kendall and Yarin Gal.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?  In _NeurIPS_ , 2017.
  * Keyes [2018] Os Keyes.  The misgendering machines: Trans/hci implications of automatic gender recognition.  _Proceedings of the ACM on Human-Computer Interaction_ , 2(CSCW):1–22, 2018.
  * Kiureghian and Ditlevsen [2009] Armen Der Kiureghian and Ove Ditlevsen.  Aleatory or epistemic? Does it matter?  _Structural Safety_ , 31(2):105–112, March 2009\.
  * Kotsiantis et al. [2006] Sotiris Kotsiantis, Dimitris Kanellopoulos, Panayiotis Pintelas, et al.  Handling imbalanced datasets: A review.  _GESTS International Transactions on Computer Science and Engineering_ , 30(1):25–36, 2006.
  * Kumar et al. [2019] Deepak Kumar, Rajat Gupta, Ashirwad Sharma, and Sushil Kumar Saroj.  Gender classification using skin patterns.  In _Proceedings of 2nd International Conference on Advanced Computing and Software Engineering (ICACSE)_ , 2019.
  * Li and Talwalkar [2020] Liam Li and Ameet Talwalkar.  Random search and reproducibility for neural architecture search.  In _Uncertainty in Artificial Intelligence_ , pages 367–377. PMLR, 2020.
  * Littman [2019] Johnathan Littman.  Waze hijacked l.a. in the name of convenience. can anyone put the genie back in the bottle?  _Los Angeles Magazine_ , 2019.
  * Liu et al. [2019] Xiaoxuan Liu, Livia Faes, Aditya U Kale, Siegfried K Wagner, Dun Jack Fu, Alice Bruynseels, Thushika Mahendiran, Gabriella Moraes, Mohith Shamdas, Christoph Kern, et al.  A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.  _The Lancet Digital Health_ , 1(6):e271–e297, 2019.
  * Lopez [2018] Steve Lopez.  Column: How waze and google maps turned an encino neighborhood into a speedway.  _Los Angeles Times_ , 2018.
  * McKenzie [2010] Patrick McKenzie.  Falsehoods programmers believe about names.  _Kalzumeus Software_ , 2010.
  * Miller [1998] George A Miller.  _WordNet: An electronic lexical database_.  MIT press, 1998.
  * Peterson et al. [2019] Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky.  Human uncertainty makes classification more robust.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 9617–9626, 2019.
  * Pinto et al. [2009] Nicolas Pinto, David Doukhan, James J DiCarlo, and David D Cox.  A high-throughput screening approach to discovering good forms of biologically inspired visual representation.  _PLoS Comput Biol_ , 5(11):e1000579, 2009.
  * Prabhu [2020] Vinay Prabhu.  https://twitter.com/vinayprabhu/status/1307460502017028096?s=20, 2020.  Accessed: 2020-9-20.
  * Prabhu and Birhane [2020] Vinay Uday Prabhu and Abeba Birhane.  Large image datasets: A pyrrhic win for computer vision?  _arXiv preprint arXiv:2006.16923_ , 2020.
  * Raff [2019] Edward Raff.  A step toward quantifying independently reproducible machine learning research.  In _Advances in Neural Information Processing Systems_ , pages 5485–5495, 2019.
  * Rodrigues et al. [2017] Filipe Rodrigues, Lourenco Mariana, Bernardete Ribeiro, and Francisco C. Pereira.  Learning supervised topic models for classification and regression from crowds.  _IEEE T-PAMI_ , 39(12):2409 – 2422, 5 2017.
  * Scheirer [2020] W. Scheirer.  How to make ai less racist.  _Bulletin of the Atomic Scientists_ , 2020.
  * Sloane et al. [2020] Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano.  Participation is not a design fix for machine learning.  _arXiv preprint arXiv:2007.02423_ , 2020.
  * Spiel et al. [2019] Katta Spiel, Oliver L Haimson, and Danielle Lottridge.  How to do better with gender on surveys: a guide for hci researchers.  _interactions_ , 26(4):62–65, 2019.
  * Sünderhauf et al. [2018] Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, et al.  The limits and potentials of deep learning for robotics.  _The International Journal of Robotics Research_ , 37(4-5):405–420, 2018.
  * Torralba et al. [2020] A. Torralba, R. Fergus, and B. Freeman.  Tiny Images Dataset Retraction.  https://groups.csail.mit.edu/vision/TinyImages/, 2020.  Accessed: 2020-9-20.
  * Torralba and Efros [2011] Antonio Torralba and Alexei A Efros.  Unbiased look at dataset bias.  In _CVPR 2011_ , pages 1521–1528. IEEE, 2011.
  * Torralba et al. [2008] Antonio Torralba, Rob Fergus, and William T Freeman.  80 million tiny images: A large data set for nonparametric object and scene recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 30(11):1958–1970, 2008.
  * Wang and Kosinski [2018] Yilun Wang and Michal Kosinski.  Deep neural networks are more accurate than humans at detecting sexual orientation from facial images.  _Journal of personality and social psychology_ , 114(2):246, 2018.
  * White et al. [2020] Colin White, Sam Nolen, and Yash Savani.  Local search is state of the art for nas benchmarks.  _arXiv preprint arXiv:2005.02960_ , 2020.
  * Wu and Zhang [2016] Xiaolin Wu and Xi Zhang.  Automated inference on criminality using face images.  _arXiv preprint arXiv:1611.04135_ , pages 4038–4052, 2016.
  * Xie et al. [2012] Jin Xie, Lei Zhang, Jane You, David Zhang, and Xiaofeng Qu.  A study of hand back skin texture patterns for personal identification and gender classification.  _Sensors_ , 12(7):8691–8709, 2012.
  * Xie et al. [2019] Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He.  Exploring randomly wired neural networks for image recognition.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1284–1293, 2019.
  * Youssef et al. [2013] Amira E Youssef, Ahmed S Ibrahim, and A Lynn Abbott.  Automated gender identification for arabic and english handwriting.  In _5th International Conference on Imaging for Crime Detection and Prevention (ICDP 2013)_ , pages 1–6. IET, 2013.
  * Yu et al. [2009] Shiqi Yu, Tieniu Tan, Kaiqi Huang, Kui Jia, and Xinyu Wu.  A study on gait-based gender classification.  _IEEE Transactions on image processing_ , 18(8):1905–1910, 2009.
  * Zetter [2020] Kim Zetter.  A famed hacker is grading thousands of programs — and may revolutionize software in the process.  The Intercept, https://rb.gy/igtlzz, 2020.  Accessed: 2020-9-20.
  * Zhang [2020] Amy X. Zhang.  https://twitter.com/amyxzh/status/1307505876396158976?s=20, 2020\.  Accessed: 2020-9-19.