††thanks: Present address: Department of Physics, Massachusetts Institute of
Technology, Cambridge, Massachusetts 02139, USA††thanks: Present address:
Department of Physics, University of Maryland, College Park, Maryland 20742,
USA††thanks: Present address: School of Science, Westlake University, 18
Shilongshan Road, 310024 Hangzhou, China

# An electronic band sculpted by oxygen vacancies and indispensable for dilute
superconductivity

Benoît Fauqué1 Clément Collignon1,2 Hyeok Yoon3,4 Ravi2 Xiao Lin 2 Igor I.
Mazin5 Harold Y. Hwang3,4 Kamran Behnia2 (1) JEIP (USR 3573 CNRS), Collège de
France, 75005 Paris, France
(2) Laboratoire de Physique et d’Etude de Matériaux (CNRS)
ESPCI Paris, Université PSL, 75005 Paris, France
(3) Stanford Institute for Materials and Energy Sciences, SLAC National
Accelerator Laboratory, Menlo Park, Stanford, California 94025, USA
(4) Department of Applied Physics, Stanford University, Stanford, California
94305, USA
(5) Department of Physics and Astronomy and Quantum Science and Engineering
Center, George Mason University, Fairfax, Virginia 22030, USA

###### Abstract

Dilute superconductivity survives in bulk strontium titanate when the Fermi
temperature falls well below the Debye temperature. Here, we show that the
onset of the superconducting dome is dopant-dependent. When mobile electrons
are introduced by removing oxygen atoms, the superconducting transition
survives down to $2\times 10^{17}$ cm-3, but when they are brought by
substituting Ti with Nb, the threshold density for superconductivity is an
order of magnitude higher. Our study of quantum oscillations reveals a
difference in the band dispersion between the dilute metals made by these
doping routes and our band calculations demonstrate that the rigid band
approximation does not hold when mobile electrons are introduced by oxygen
vacancies. We identify the band sculpted by these vacancies as the exclusive
locus of superconducting instability in the ultra-dilute limit.

##  I Introduction

Discovered half a century ago [1], superconductivity in strontium titanate is
attracting renewed attention [2, 3]. It lies beyond the Migdal-Eliashberg [4]
framework, since the Fermi temperature of electrons falls below the Debye
temperature of phonons [5, 6]. The dilute metal going through this
superconducting instability has non-trivial transport properties [7, 8, 9,
10]. The insulating parent, identified as a quantum paraelectric [11], has a
huge permittivity, displays an unusual low-temperature thermal transport [12]
including a thermal Hall component [13]. Soft transverse optical phonons,
hybridizing with transverse acoustic phonons at low temperatures [14, 15], are
suspected to play a major role.

The proximity to a ferroelectric instability is often invoked in explaining
superconductivity [16, 17, 18]. Several experimental studies have documented a
constructive interplay between superconductivity and ferroelectricity [19, 20,
21, 22] and recent theoretical accounts of Cooper pairing invoke the exchange
of two [23, 24, 25] or one [26, 27] soft phonons between pairing electrons. A
two-phonon exchange scenario, first invoked decades ago [28], leads to
instantaneous electron-electron attraction and can explain [24] the
experimentally-observed persistence of superconductivity at densities below
1018 cm-3 [5, 29].

Here, we present a study of superconducting transition and quantum
oscillations in oxygen-reduced and Nb-substituted strontium titanate in the
dilute limit. We find that when $n<5\times 10^{18}$ cm-3, superconductivity is
present in SrTiO3-δ, but absent in SrTi1-xNbxO3, either single-crystalline or
custom-made thin films. The dilute metals generated by alternative doping
routes are also different. In particular, the Lifshitz transition in
SrTi1-xNbxO3 occurs at a lower density. We present numerical evidence that the
virtual crystal approximation used in the LDA band calculations [30] holds for
Nb doping, but not for oxygen deficiency. We conclude that while both doping
routes eventually lead to a superconducting dome, dilute superconductivity
(arising in a metal where e-e distance is of the order 10 nm) occurs only when
the occupied band is sculpted by oxygen vacancies. A larger coupling between
the electrons of the vacancy-sculpted band and soft TO phonons, or a larger
electronic density of states, or a combination of both, may be the origin.

Figure 1: Different thresholds for emergence of superconductivity:  a)
Temperature dependence of electrical resistivity ($\rho_{xx}$) in SrTi1-xNbxO3
and SrTiO3-δ (orange) sample with similar carrier density n${}_{H}\simeq$1.6
$\times 10^{18}$cm-3. Neither the thin film nor the single crystalline
SrTi1-xNbxO3 shows a superconducting transition at this carrier density. b)
Doping evolution of the superconducting critical temperature (Tc). Solid
symbols represent samples studied in the present work. Open symbols represent
what was reported previously [2, 22, 31].

SrTiO3 can be n-doped by a variety of atomic substitutions. The focus of the
present study is to compare substitution of tetravalent Ti4+ by pentavalent
Nb5+ with creation of oxygen vacancies. While both lead to n-doping, there is
no a priori reason to believe that they affect the electronic structure in the
same way [32]. In presence of an O vacancy, the two neighboring Ti ions are
left with one extra electron each. What can happen with these electrons? They
can either get collectively extended over all Ti sites or both occupy the
vacancy. In a recent DFT+DMFT calculation, Souto-Casares, Spaldin and Ederer
[33] found that if the Hubbard repulsion on Ti $and$ the concentration are
both large, one or even two electrons will prefer to avoid the Hubbard
repulsion on Ti and “hide” in the vacancy. However, the effective
concentration of vacancies in that work [33] was one per four formula units,
which is several orders of magnitude higher than the relevant experimental
range. At very small concentrations, Hubbard correlation between the doped
electrons, included in the DMFT method and not in the DFT, are much less
important than geometric changes such as lattice relaxation. Keeping this in
mind, we have performed full structural relaxation in the density functional
theory (DFT), using an $8\times 8\times 8$ supercell of 64 f.u., and either
replacing one Ti with Nb or removing one O. As discussed below, we found that
the resulting electronic structure near the bottom of the conduction band is
dramatically different in the two cases.

Figure 2: Distinct Fermi surfaces: a) Magnetoresistance
($\frac{\Delta\rho_{xx}}{\rho_{0}}$) vs B in SrTi1-xNbxO3 (orange) and
SrTiO3-δ (blue) with the same Hall carrier density nH=1.65e18cm-3 at T = 120
mK. b) Traces of the quantum oscillations in
$\frac{\partial^{2}\rho_{xx}}{\partial B^{2}}$ as function of B-1 for both
samples. c) Amplitude of the Fourier transform (FT) vs frequency (F) deduced
from b). The amplitude was normalized by the peak value and shift for clarity.
In both samples, two frequencies (and their harmonics) can be identified,
respectively labelled F1 and F2. Figure 3: Doping evolution of the Fermi
surface of SrTi1-xNbxO3 and SrTiO3-δ : a) Frequencies of the quantum
oscillations of the two sub-bands F1 (circles) and F2 (squares) vs. the Hall
carrier density (nH) in SrTi1-xNbxO3, in SrTiO3-δ [29] and in Sr1-xLaxTiO3
[34]. The Lifshitz transition occurs at different threshold densities. b)
Ratio of the carrier density extracted from the Hall coefficient to the
carrier density deduced from quantum oscillations frequency assuming two
spherical pockets. A difference persists in all samples. This implies distinct
geometries for the Fermi surface in the two cases. Figure 4: Quantum
oscillations in SrTi1-xNbxO3 with n${}_{H}=4.4\times 10^{18}$cm-3 : a)
$\frac{\Delta\rho_{xx}}{\rho_{0}}$ vs B at different temperatures from 80 mK
to 2.1 K. b) $\frac{\partial^{2}\rho_{xx}}{\partial B^{2}}$ as a function of
B-1 displaying quantum oscillations. c) FT of the data at T = 80 mK.

##  II Results

###  II.1 Dopant dependent superconductivity

Fig.1a shows the low-temperature electrical resistivity ($\rho_{xx}$) of three
samples with the same Hall carrier density. One can see that SrTiO3-δ becomes
a superconductor, but not SrTi1-xNbxTiO3 (at least not down to 30mK). Note
that, the oxygen deficient sample has a residual resistivity lower than the
thin film and larger than the crystalline SrTi1-xNbxTiO3. This rules out
disorder as the driver of this dichotomy. Studying four different Nb-doped
SrTiO3 thin films [35] with a thickness of $\approx$ 900 nm and nominal Nb
concentrations of 0.02 and 0.04 atomic percent, we found that none was
superconducting (See the supplement for more details about samples and data
[36]).

Fig.1.b draws the superconducting phase diagram of SrTiO3-δ and in
SrTi1-xNbxTiO3 according to the available data reported by different groups
[2, 22, 31]. The critical temperature does not evolve identically. The peak in
Tc of the principal dome is higher in Nb-doped strontium titanate, while a
lower superconducting dome is present for SrTiO3-δ and absent in SrTi1-xNbxO3.

Evidence for superconductivity in this density range is limited to the
detection of zero resistivity. The critical temperature of bulk probes of
superconductivity, such as specific heat [37], thermal conductivity [37] and
diamagnetism [38] is consistently lower than the temperature at which
resistivity drops to zero. Given the strong variation of critical temperature
with pressure or strain [18], this points to the survival of filamentary
superconductivity with a higher critical temperature at specific locations
like dislocations or domain boundaries. Nevertheless, oxygen-reduced samples
display bulk superconductivity for a carrier density as low as 4.5$\times
10^{18}$cm-3 [19, 22] and the Nb-substituted ones do not show any transition
(resistive or else) at this concentration. Therefore, there is a genuine
difference of the ground states, irrespective of the origin of filamentary
superconductivity.

###  II.2 Dopant dependent metallicity

Let us now compare the normal state of these two dilute metals. Structurally,
the distribution of oxygen vacancies is known to be less homogeneous than the
distribution of Nb atoms [39]. However, in both cases, electrons belong to a
single Fermi sea (and not to a plurality of puddles). Since the Bohr radius is
as long as several hundreds of nanometer [40], disorder is averaged over a
volume containing many dopants. In both oxygen-reduced and Nd-substituted
strontium titanate, the carrier mean-free-path is much longer than the
interdopant distance and quantum oscillations are visible at moderate
($\approx 2$ T) magnetic fields.

We carried out a careful examination of the Fermi surface by looking at the
Shubnikov-de Haas effect in the two cases. Magnetoresistance
($\frac{\Delta\rho_{xx}}{\rho_{0}}$) at T = 120 mK and for B$\parallel$ [001]
(See Fig.2a)) reveals a difference. Quantum oscillations are visible on top of
a monotonic background in both. As seen in Fig.2b, however, despite the quasi-
equality of their Hall carrier density, the second derivative of resistivity
$\frac{\partial^{2}\rho_{xx}}{\partial B^{2}}$ shows different patterns. Two
main frequencies, labelled F1 and F2 can be resolved, in qualitative agreement
with previous studies of quantum oscillations [41, 34, 5]. However, F1 and F2
are not identical in the two cases, implying a difference in the structure of
the Fermi surface between superconducting SrTiO3-δ and non-superconducting
SrTi1-xNbxTiO3.

By studying three other Nb-doped thin-film samples with nH ranging from
8.6$\times 10^{17}$ cm-3 to 4.4$\times 10^{18}$cm-3 (see [36] for sample
details), we found that the evolution of F1 and F2 in SrTi1-xNbxO3 and in
SrTiO3-x [29] are not identical (See Fig. 3.a). At a given Hall carrier
density, both frequencies are larger in SrTi1-xNbxO3 compared to SrTiO3-δ (See
the supplement [36] for more details on the analysis).

In the absence of an angle-dependent study, possible multiplicity of
structural domains below 105 K is a source of complication. In multi-domain
samples the orientation of magnetic field can be perpendicular or parallel to
the c-axis according to the orientation of domains. However, a simple
procedure reveals a difference in Fermi surface geometry. Assuming that the
two sub-bands are isotropic, the carrier density can be deduced from the
frequencies of quantum oscillations:
n${}^{sphere}_{SdH}$=$\frac{1}{3\pi^{2}}((\frac{2eF_{1}}{\hbar})^{3/2}+(\frac{2eF_{2}}{\hbar})^{3/2}$)
and then compared to the Hall carrier density, $n_{H}$. Their ratio,
$\frac{n_{H}}{n^{sphere}_{SdH}}$, is a measure of the deviation of the pockets
from perfect spheres. Fig. 3.b compares the two cases at different doping
levels. The ratio is $\approx 1.2$ in SrTiO3-δ and $\approx 0.5$ in
SrTi1-xNbxO3. Thus, the Fermi surface of Nb-doped samples is less spherical
than the Fermi surface of SrTi1-xNbxO3. Moreover, as seen in Fig. 3.a, the
threshold for filling the second band, i.e. the onset of the Lifshitz
transition is also lower in the Nb-doped case.

We note that F1 and F2 of our Nb-doped samples are in good agreement with what
has been found in La-doped samples [34] (See Fig. 3a)). This suggests that the
Fermi surface in La-doped and in Nb-doped strontium titanate are similar and
both differ from the one belonging to oxygen-deficient strontium titanate.
Interestingly, there is no report of a superconducting transition in single
crystals [20] or thin films of Sr1-xLaxTiO3 [21] when the carrier density is
as low as ($\approx 10^{18}$ cm-3).

By studying the temperature dependence of the quantum oscillations and using
the Lifshitz-Kosevitch formalism, we extracted the cyclotron mass of
electrons. Fig. 4 shows the quantum oscillations and their temperature
dependence in the sample with highest carrier concentration
(n${}_{H}=4.4\times 10^{18}$cm-3). The results of the analysis are shown in
the upper panel of Fig. 4c). The effective mass for the lower band is
m${}^{\star}=1.4\pm 0.1$m0, comparable to the value found in previous studies
of Nb-doped [41] (and La-doped [34]) strontium titanate. The effective mass of
the lower band in the oxygen-reduced case at this carrier density is
m${}^{\star}=1.8$ m0 [5] 111The lower band is non-parabolic [30], and
therefore, the effective mass is expected to increase with increasing carrier
density. Experimentally, m⋆ in the oxygen-doped samples becomes $\approx$ 3.5
m0 at n${}_{H}=3\times 10^{19}$cm-3 [29].

###  II.3 Band calculations

Numerical emulation of a regime as dilute as the one explored experimentally
requires unrealistically large supercells. Therefore, a quantitative and
microscopic explanation of the observed difference is challenging. However,
considerable insight can be gained from DFT calculations. To this end, we used
the projected augmented wave pseudo-potential code VASP [43] to fully optimize
the crystal structure, including the lattice parameters, in the $8\times
8\times 8$ supercell of 64 formula units.

In the ground state, at $T=0$, SrTiO3 forms a tetragonal I4/mcm structure,
characterized with alternating rotations of the TiO6 octahedra around the $c$
axis. This is a strong effect well captured by the standard DFT calculations
(Fig. 5). Proper stacking of the rotations along $c$, however, is quite
difficult to obtain, and requires sophisticated setups. Our calculations,
using standard PBE pseudoptentials, and up to 3x3x3 mesh in the Brillouin
zone, reproduce the rotations well, but form a different stacking,
corresponding to a P4/mbm group. The difference is not relevant for our
analysis.

Since the optimized structure is tetragonal, there are two
crystallographically nonequivalent sites, one preserving the tetragonal
symmetry (we shall call it v-T) and the other lowering it to an orthorhombic
one (v-O). In our calculations, the former corresponds to the space group P4/m
(incidentally, removal of Nb also results in the same symmetry), and the
latter to Pmm2. At the level of our accuracy we could not determine with
certainty which vacancy position is energetically preferable (see the
supplement [36] for details).

Figure 5: Effect of impurities on the crystal structure and electron density:
a) Crystal structure with a Nb impurity (green octahedron) b) Same for an O
vacancy (black ball) preserving the tetragonal symmetry. c) The electron
charge distribution around the vacancy; here the red balls are O, the blue-
grey ones Ti and Sr are green. The black ball indicates the approximate
location of the vacancy. The (yellow) isosurfaces are drawn at the level of
electron density of 0.05 $e/$Å3.

However, we were able to make one structural observation: both Nb and v-T case
can be optimized with a restriction forbidding octahedra rotations (v-O
cannot). In the undoped compound these rotations decrease the total energy by
3.4 meV per formula. This energy difference increases, in our supercell, to
4.7 meV for Nb substitution. In contrast, it decreases to 0.9 meV for O
vacancy. As discussed in the next section, this trend is in agreement with
previous experimental findings.

The difference in the electronic structure (Fig. 6) is even more spectacular.
Nb doping has little effect on the band dispersions, but the O vacancies
dramatically modifies them. In case of v-T, the main effect is the relative
shift between the two lower ($xz$ and $yz$) bands and the upper ($xy$) band.
This can be understood by analyzing the change of the local crystal field. As
Fig. 5 shows, Nb introduces no visible additional distortion, while the
vacancy alters the local environment of the two neighboring Ti from octahedral
to pyramidal, and leads to a visible relaxations of the eight closest oxygen
atoms.

Neglecting the small distortion of the octahedra, at the $\Gamma$ point, which
is the only one of interest, the hybridization of a Ti $t_{2g}$ orbital with
the four neighboring O atoms cancels out by symmetry (Fig. 7). However, if one
apical oxygen is removed, the ligand field of the opposite one is
uncompensated, so the corresponding band state is pushed up. In our case, this
means that the $xz$ and $yz$ bands will be pushed up closer to the
(unaffected) $xy$ band at the $\Gamma$ point (but not at the Z point). Of
course, given the delocalization of the doped electrons, the effect on the
entire band is reduced compared to the actual change of the site-local crystal
field, but it still rather large on the scale of interest (i.e., $\sim 10$
meV). Note that no extra electron density is found in the calculations near
the vacancy site (Fig. 7).

Figure 6: Non-relativistic band states in Sr64Ti63NbO192 (a) and in
Sr64Ti64O191 (b,c,d): In all cases, the structure was fully optimized within
DFT. Only in the case of Nb substitution, the dispersion is virtually
identical to the pristine case. In (b), the oxygen vacancy (v-T) keeps the
tetragonal symmetry of the lattice. In (c) and (d), the oxygen vacancy (v-O)
makes the system orthorhombic and as seen in (d) the dispersion along x- and
y-axes are no more identical. The horizontal axis refers to distances in the
momentum space along different orientations expressed in reciprocal lattice
units.  Figure 7: Effect of a tetragonal” v-O vacancy on the ligand crystal
field: Relevant Ti $d$ and O $p$ wave functions for the wave vector k
corresponding to the $\Gamma$ point, in the real space around a Ti site. In
absence of O vacancy, the hybridization of all $t_{2g}$ orbitals with the O
states cancels out by symmetry. Introducing a vacancy does not change this for
the $xy$ orbital (top) but the cancellation is broken by the vacancy for the
$xz$ and $yz$ (not shown) orbitals (bottom). As a consequence, in the momentum
space, the corresponding bands at the $\Gamma$ point shift upward.

The situation for a v-O vacancy is more complicated. Qualitatively, the
perturbation of the Ti environment is the same as depicted in Fig. 7b, but the
picture needs to be rotated by 90∘. Now it is not the $xy$ orbitals that is
being push down with respect to the other two, but $yz$. Since the $xz/yz$
degeneracy with $xy$ is already broken due to octahedra rotation, it leads to
an interesting situation when the states at $\Gamma$ are threefold split
already on the nonrelativistic level, $i.e.$, without spin-orbit.

Interestingly, the calculated threshold for the first Lifshitz transition in
the pristine crystal [30] is n${}_{c1}=6.4\times 10^{17}cm^{-3}$. This is in
quantitative agreement with what we find experimentally for SrTi1-xNbxO3
(n${}_{c1}=6.5\pm 2\times 10^{17}cm^{-3}$; Fig.4a), confirming that the rigid
band approximation holds in this case (but not in SrTiO3-δ).

##  III Discussion

Pristine strontium titanate goes through a cubic-to-tetragonal structural
phase transition [44] at $T_{s}\simeq 105$ K. This is a (remarkably) second-
order [45] structural phase transition driven by the softening of a Transverse
Acoustic (TA) mode at the zone boundary [46]. Previous experiments have found
that $T_{s}$ increases by Nb substitution and decreases by oxygen deficiency
[47, 48]. In oxygen-reduced samples with a carrier density of $0.5\%$ per
formula unit, $T_{s}$ drops by $\sim 10$K [47]. In Nb-doped samples with a
carrier density of $1\%$ per formula unit, $T_{s}$ increases by $\sim 20$K
[48]. This implies that the effect of the two doping routes on the boundary TA
mode. This trend is reproduced by our DFT calculations. They found, as
reported in the previous section, that the energy difference between the cubic
and the tetragonal structures is amplified by Nb substitution and diminished
by oxygen removal. Given the link between $T_{s}$ and energy difference, our
calculations provide a qualitative account of the experimental observation.

The softening of another mode, a Transverse Optical (TO) phonon located at the
center of the Brillouin zone, would lead to ferroelectricity. This mode is
suspected to play a prominent role in driving the superconducting instability
in the dilute limit [23, 24, 25, 26, 27]. Given the contrasting consequences
of the two doping routes for the boundary TA phonon (see above), one may
wonder what would be their effect on the zone-centered ones. The dispersion of
these TO phonons has been found to evolve with doping [7, 49]. But there is no
record of a comparative study of their evolution in oxygen-reduced and Nb-
substituted samples. Future studies may discriminate between the fingerprints
of the two doping routes.

The present study reveals the contrasting outcomes of Nb substitution and
oxygen reduction for metallicity and superconductivity. As we saw, the
experimental data reveals a difference in the Fermi surface geometry in the
two cases near the first Lifshitz transition. This is in qualitative agreement
with our DFT calculations. They show that the band dispersion is different in
the two cases. This agreement constitutes a ‘proof of the concept’. A
quantitative confrontation between theory and experiment is beyond what can be
achieved by the state-of-the-art. Indeed, the theoretical supercell is
equivalent to a doping concentration of $\sim 10^{-2}$ per formula unit. This
exceeds by two orders of magnitude the experimental one and impedes a direct
comparison.

Let us note that very recently, Tomioka et al. [31] have confirmed that below
a threshold density of $\approx 8\times 10^{18}$ cm-3, SrTi1-xNbxO3 and
Sr1-xLa${}_{x}Ti$O3 do not superconduct. On the other hand, replacing Sr with
either Ca or Ba makes them superconducting. This provides additional support
for dopant dependent superconductivity in the dilute limit and calls for an
experimental study of the fermiology in polar metals such as
Nb:Sr0.985Ca0.015TiO3.

Dopant-dependent superconductivity has been reported in other cases. Beyond a
critical threshold of Tl doping, Pb1-xTlxTe is a superconductor, but
Pb1-xNaxTe is non-superconducting [50, 51]. The difference has been tracked to
the presence of Tl impurity states near the Fermi energy. Let us note that the
concentration of dopants is two orders of magnitude lower in strontium
titanate than in lead telluride. The experimental observation of dopant
dependent superconductivity has motivated theoretical scenarios [52, 53],
which may be relevant to our result. Nevertheless, large Hubbard correlations
are unlikely in the extreme dilute limit. Occam’s razor [54] would favor
scenarios which do not require them.

Thus, like lead telluride, strontium titanate demonstrate that the rigid band
approximation does not capture the whole physics when a small concentration of
dopants turns a solid off-stoichiometric. Future studies will tell if the
breakdown of the rigid band approximation has any visible signature in the
thermoelectric response [55], as found in the case of PbTe [56], where doping
by Tl has led to an amplification of the thermoelectric figure of merit.

Sample | $\mu_{H}$ | nH | F1 | F2 | target used | thickness | Undoped cap layer
---|---|---|---|---|---|---|---
| V.cm-2.s-1 | $cm^{-3}$ | T | T |  | nm | nm
$S_{1}$ | 4100 | 4.4e18 | 36 | 137 | 0.04 at.% Nb | 892 | 119
$S_{2}$ | 3162 | 1.65e18 | 16 | 72 | 0.02 at.% Nb | 617 | 102
$S_{3}$ | 1365 | 1.5e18 | 14 | 58 | 0.02 at.% Nb | 895 | 90
$S_{4}$ | 9084 | 8.6e17 | 9 | 33 | 0.02 at.% Nb +undoped | 748 | 100
Table 1: SrTi1-xNbxO3 thin films studied in this work and their properties-
Carrier densities and Hall mobilities are given at 2 K.

In summary, we found that doping strontium titanate by removing oxygen leads
to a dilute metal different from the one obtained by substituting Ti with Nb.
The two detected differences are the band dispersion and the presence and
absence of a superconducting instability. The exceptionally dilute
superconductivity is an instability in a band specifically sculpted by oxygen
vacancies of strontium titanate.

##  IV Materials and methods

Nb-doped films were synthesized on SrTiO3 (001) substrate in pulsed laser
deposition, as detailed previously [57, 35]. The films, which are listed in
table 1, are all sufficiently thick to host a three-dimensional Fermi surface
as found in a previous study of their fermiology [58].

Oxygen-deficient SrTiO3 single crystals were obtained by annealing
commercially bought substrates in a manner similar to what was described in
ref.[59, 5, 29]. Four-contact resistivity measurements were performed in a
dilution refrigerator inserted in a 17 T superconducting magnet.

Band calculations used the Projector Augmented Wave method together with the
Generalized Gradient Correction to the exchange-correlation potential [60], as
implemented in the VASP package [43]. Fig. 5 was generated using the VESTA
program [61].

##  V Acknowledgements

We thank Claude Ederer and Maria Gastiasoro for useful discussions.

This work was supported by the Agence Nationale de la Recherche
(ANR-18-CE92-0020-01; ANR-19-CE30-0014-04), by Jeunes Equipes de l′Institut de
Physique du Collège de France and by a grant attributed by the Ile de France
regional council. It was funded in part by a QuantEmX grant from ICAM and the
Gordon and Betty Moore Foundation through Grant GBMF9616 to KB. The work at
SLAC/Stanford is supported by the U.S. Department of Energy, Office of Basic
Energy Sciences, Division of Materials Sciences and Engineering, under
Contract No. DE-AC02-76SF00515.

## References

  * Schooley  _et al._ [1964] J. F. Schooley, W. R. Hosler, and M. L. Cohen, Superconductivity in semiconducting SrTi${\mathrm{O}}_{3}$, Phys. Rev. Lett. 12, 474 (1964).
  * Collignon  _et al._ [2019] C. Collignon, X. Lin, C. W. Rischau, B. Fauqué, and K. Behnia, Metallicity and superconductivity in doped strontium titanate, Annual Review of Condensed Matter Physics 10, 25 (2019).
  * Gastiasoro  _et al._ [2020] M. N. Gastiasoro, J. Ruhman, and R. M. Fernandes, Superconductivity in dilute SrTiO3: A review, Annals of Physics 417, 168107 (2020).
  * Eliashberg [1960] G. M. Eliashberg, Interactions between electrons and lattice vibrations in a superconductor, Sov. Phys. - JETP 11, 696 (1960).
  * Lin  _et al._ [2013] X. Lin, Z. Zhu, B. Fauqué, and K. Behnia, Fermi surface of the most dilute superconductor, Phys. Rev. X 3, 021002 (2013).
  * Yoon  _et al._ [2021] H. Yoon, A. G. Swartz, S. P. Harvey, H. Inoue, Y. Hikita, Y. Yu, S. B. Chung, S. Raghu, and H. Y. Hwang, Low-density superconductivity in SrTiO3 bounded by the adiabatic criterion, arXiv e-prints , arXiv:2106.10802 (2021).
  * Collignon  _et al._ [2020] C. Collignon, P. Bourges, B. Fauqué, and K. Behnia, Heavy nondegenerate electrons in doped strontium titanate, Phys. Rev. X 10, 031025 (2020).
  * Kumar  _et al._ [2021] A. Kumar, V. I. Yudson, and D. L. Maslov, Quasiparticle and nonquasiparticle transport in doped quantum paraelectrics, Phys. Rev. Lett. 126, 076601 (2021).
  * Nazaryan and Feigel’man [2021] K. G. Nazaryan and M. V. Feigel’man, Conductivity and thermoelectric coefficients of doped ${\mathrm{SrTiO}}_{3}$ at high temperatures, Phys. Rev. B 104, 115201 (2021).
  * Collignon  _et al._ [2021] C. Collignon, Y. Awashima, Ravi, X. Lin, C. W. Rischau, A. Acheche, B. Vignolle, C. Proust, Y. Fuseya, K. Behnia, and B. Fauqué, Quasi-isotropic orbital magnetoresistance in lightly doped ${\mathrm{SrTiO}}_{3}$, Phys. Rev. Materials 5, 065002 (2021).
  * Müller and Burkard [1979] K. A. Müller and H. Burkard, Srti${\mathrm{o}}_{3}$: An intrinsic quantum paraelectric below 4 k, Phys. Rev. B 19, 3593 (1979).
  * Martelli  _et al._ [2018] V. Martelli, J. L. Jiménez, M. Continentino, E. Baggio-Saitovitch, and K. Behnia, Thermal transport and phonon hydrodynamics in strontium titanate, Phys. Rev. Lett. 120, 125901 (2018).
  * Li  _et al._ [2020] X. Li, B. Fauqué, Z. Zhu, and K. Behnia, Phonon thermal hall effect in strontium titanate, Phys. Rev. Lett. 124, 105901 (2020).
  * He  _et al._ [2020] X. He, D. Bansal, B. Winn, S. Chi, L. Boatner, and O. Delaire, Anharmonic eigenvectors and acoustic phonon disappearance in quantum paraelectric ${\mathrm{SrTiO}}_{3}$, Phys. Rev. Lett. 124, 145901 (2020).
  * Fauqué  _et al._ [2022] B. Fauqué, P. Bourges, A. Subedi, K. Behnia, B. Baptiste, B. Roessli, T. Fennell, S. Raymond, and P. Steffens, Mesoscopic fluctuating domains in strontium titanate, Phys. Rev. B 106, L140301 (2022).
  * Rowley  _et al._ [2014] S. E. Rowley, L. J. Spalek, R. P. Smith, M. P. M. Dean, M. Itoh, J. F. Scott, G. G. Lonzarich, and S. S. Saxena, Ferroelectric quantum criticality, Nature Physics 10, 367 (2014).
  * Edge  _et al._ [2015] J. M. Edge, Y. Kedem, U. Aschauer, N. A. Spaldin, and A. V. Balatsky, Quantum critical origin of the superconducting dome in ${\mathrm{SrTiO}}_{3}$, Phys. Rev. Lett. 115, 247002 (2015).
  * Enderlein  _et al._ [2020] C. Enderlein, J. F. de Oliveira, D. A. Tompsett, E. B. Saitovitch, S. S. Saxena, G. G. Lonzarich, and S. E. Rowley, Superconductivity mediated by polar modes in ferroelectric metals, Nature Communications 11, 4852 (2020).
  * Rischau  _et al._ [2017] C. W. Rischau, X. Lin, C. P. Grams, D. Finck, S. Harms, J. Engelmayer, T. Lorenz, Y. Gallais, B. Fauqué, J. Hemberger, and K. Behnia, A ferroelectric quantum phase transition inside the superconducting dome of $\mathrm{Sr_{1-x}Ca_{x}TiO_{3-d}}$, Nature Physics 13, 643 (2017).
  * Tomioka  _et al._ [2019] Y. Tomioka, N. Shirakawa, K. Shibuya, and I. H. Inoue, Enhanced superconductivity close to a non-magnetic quantum critical point in electron-doped strontium titanate, Nature Communications 10, 738 (2019).
  * Ahadi  _et al._ [2019] K. Ahadi, L. Galletti, Y. Li, S. Salmani-Rezaie, W. Wu, and S. Stemmer, Enhancing superconductivity in SrTiO3 films with strain, Science Advances 5, eaaw0120 (2019), https://www.science.org/doi/pdf/10.1126/sciadv.aaw0120 .
  * Rischau  _et al._ [2022] C. W. Rischau, D. Pulmannová, G. W. Scheerer, A. Stucky, E. Giannini, and D. van der Marel, Isotope tuning of the superconducting dome of strontium titanate, Phys. Rev. Research 4, 013019 (2022).
  * van der Marel  _et al._ [2019] D. van der Marel, F. Barantani, and C. W. Rischau, Possible mechanism for superconductivity in doped ${\mathrm{SrTiO}}_{3}$, Phys. Rev. Research 1, 013003 (2019).
  * Kiselov and Feigel’man [2021] D. E. Kiselov and M. V. Feigel’man, Theory of superconductivity due to ngai’s mechanism in lightly doped ${\mathrm{SrTiO}}_{3}$, Phys. Rev. B 104, L220506 (2021).
  * Volkov  _et al._ [2022] P. A. Volkov, P. Chandra, and P. Coleman, Superconductivity from energy fluctuations in dilute quantum critical polar metals, Nature Communications 13, 4599 (2022).
  * Gastiasoro  _et al._ [2022] M. N. Gastiasoro, M. E. Temperini, P. Barone, and J. Lorenzana, Theory of superconductivity mediated by rashba coupling in incipient ferroelectrics, Phys. Rev. B 105, 224503 (2022).
  * Yu  _et al._ [2022] Y. Yu, H. Y. Hwang, S. Raghu, and S. B. Chung, Theory of superconductivity in doped quantum paraelectrics, npj Quantum Materials 7, 63 (2022).
  * Ngai [1974] K. L. Ngai, Two-phonon deformation potential and superconductivity in degenerate semiconductors, Phys. Rev. Lett. 32, 215 (1974).
  * Lin  _et al._ [2014a] X. Lin, G. Bridoux, A. Gourgout, G. Seyfarth, S. Krämer, M. Nardone, B. Fauqué, and K. Behnia, Critical doping for the onset of a two-band superconducting ground state in SrTiO3-δ, Phys. Rev. Lett. 112, 207002 (2014a).
  * van der Marel  _et al._ [2011] D. van der Marel, J. L. M. van Mechelen, and I. I. Mazin, Common fermi-liquid origin of ${T}^{2}$ resistivity and superconductivity in $n$-typeSrTiO3, Phys. Rev. B 84, 205111 (2011).
  * Tomioka  _et al._ [2022] Y. Tomioka, N. Shirakawa, and I. H. Inoue, Superconductivity enhancement in polar metal regions of Sr0.95Ba0.05TiO3 and Sr0.985Ca0.015TiO3 revealed by systematic Nb doping, npj Quantum Materials 7, 111 (2022).
  * Hou and Terakura [2010] Z. Hou and K. Terakura, Defect states induced by oxygen vacancies in cubic SrTiO3: First-principles calculations, Journal of the Physical Society of Japan 79, 114704 (2010), https://doi.org/10.1143/JPSJ.79.114704 .
  * Souto-Casares  _et al._ [2021] J. Souto-Casares, N. A. Spaldin, and C. Ederer, Oxygen vacancies in strontium titanate: A $\mathrm{DFT}+\mathrm{DMFT}$ study, Phys. Rev. Research 3, 023027 (2021).
  * Allen  _et al._ [2013] S. J. Allen, B. Jalan, S. Lee, D. G. Ouellette, G. Khalsa, J. Jaroszynski, S. Stemmer, and A. H. MacDonald, Conduction-band edge and shubnikov–de haas effect in low-electron-density $\mathrm{SrTiO_{3}}$, Phys. Rev. B 88, 045114 (2013).
  * Kozuka  _et al._ [2010] Y. Kozuka, Y. Hikita, C. Bell, and H. Y. Hwang, Dramatic mobility enhancements in doped SrTiO3 thin films by defect management, Applied Physics Letters 97, 012107 (2010).
  * [36] See Supplemental Material for more details and additional data regarding the samples and the methods.
  * Lin  _et al._ [2014b] X. Lin, A. Gourgout, G. Bridoux, F. m. c. Jomard, A. Pourret, B. Fauqué, D. Aoki, and K. Behnia, Multiple nodeless superconducting gaps in optimally doped SrTiO3-δ, Phys. Rev. B 90, 140508 (2014b).
  * Collignon  _et al._ [2017] C. Collignon, B. Fauqué, A. Cavanna, U. Gennser, D. Mailly, and K. Behnia, Superfluid density and carrier concentration across a superconducting dome: The case of strontium titanate, Phys. Rev. B 96, 224506 (2017).
  * Szot  _et al._ [2002] K. Szot, W. Speier, R. Carius, U. Zastrow, and W. Beyer, Localized metallic conductivity and self-healing during thermal reduction of ${\mathrm{SrTiO}}_{3}$, Phys. Rev. Lett. 88, 075508 (2002).
  * Behnia [2015] K. Behnia, On mobility of electrons in a shallow fermi sea over a rough seafloor, Journal of Physics: Condensed Matter 27, 375501 (2015).
  * Uwe  _et al._ [1985] H. Uwe, R. Yoshizaki, T. Sakudo, A. Izumi, and T. Uzumaki, Conduction band structure of $\mathrm{SrTiO_{3}}$, Japanese Journal of Applied Physics 24, 335 (1985).
  * Note [1] The lower band is non-parabolic [30], and therefore, the effective mass is expected to increase with increasing carrier density. Experimentally, m⋆ in the oxygen-doped samples becomes $\approx$ 3.5 m0 at n${}_{H}=3\times 10^{19}$cm-3 [29].
  * Kresse and Hafner [1993] G. Kresse and J. Hafner, Ab initio molecular dynamics for liquid metals, Phys. Rev. B 47, 558 (1993).
  * Lytle [1964] F. W. Lytle, X‐ray diffractometry of low‐temperature phase transformations in strontium titanate, Journal of Applied Physics 35, 2212 (1964).
  * Salje  _et al._ [1998] E. K. H. Salje, M. C. Gallardo, J. Jiménez, F. J. Romero, and J. del Cerro, The cubic-tetragonal phase transition in strontium titanate: excess specific heat measurements and evidence for a near-tricritical, mean field type transition mechanism, Journal of Physics: Condensed Matter 10, 5535 (1998).
  * Shirane and Yamada [1969] G. Shirane and Y. Yamada, Lattice-dynamical study of the 110∘k phase transition in SrTi${\mathrm{O}}_{3}$, Phys. Rev. 177, 858 (1969).
  * Bäuerle and Rehwald [1978] D. Bäuerle and W. Rehwald, Structural phase transitions in semiconducting SrTiO3, Solid State Communications 27, 1343 (1978).
  * Tao  _et al._ [2016] Q. Tao, B. Loret, B. Xu, X. Yang, C. W. Rischau, X. Lin, B. Fauqué, M. J. Verstraete, and K. Behnia, Nonmonotonic anisotropy in charge conduction induced by antiferrodistortive transition in metallic SrTiO3, Phys. Rev. B 94, 035111 (2016).
  * Rehwald [1970] W. Rehwald, Low temperature elastic moduli of strontium titanate, Solid State Communications 8, 1483 (1970).
  * Matsushita  _et al._ [2005] Y. Matsushita, H. Bluhm, T. H. Geballe, and I. R. Fisher, Evidence for charge kondo effect in superconducting tl-doped PbTe, Phys. Rev. Lett. 94, 157002 (2005).
  * Giraldo-Gallo  _et al._ [2018] P. Giraldo-Gallo, P. Walmsley, B. Sangiorgio, S. C. Riggs, R. D. McDonald, L. Buchauer, B. Fauqué, C. Liu, N. A. Spaldin, A. Kaminski, K. Behnia, and I. R. Fisher, Evidence of incoherent carriers associated with resonant impurity levels and their influence on superconductivity in the anomalous superconductor ${\mathrm{Pb}}_{1-x}{\mathrm{Tl}}_{x}\mathrm{Te}$, Phys. Rev. Lett. 121, 207001 (2018).
  * Dzero and Schmalian [2005] M. Dzero and J. Schmalian, Superconductivity in charge kondo systems, Phys. Rev. Lett. 94, 157003 (2005).
  * Kivelson [2020] S. A. Kivelson, Physics of superconducting transition temperatures, Journal of Superconductivity and Novel Magnetism 33, 5 (2020).
  * Mazin [2022] I. Mazin, Inverse Occam’s razor, Nature Physics 18, 367 (2022).
  * Lee and Mahanti [2012] M.-S. Lee and S. D. Mahanti, Validity of the rigid band approximation in the study of the thermopower of narrow band gap semiconductors, Phys. Rev. B 85, 165149 (2012).
  * Heremans  _et al._ [2008] J. P. Heremans, V. Jovovic, E. S. Toberer, A. Saramat, K. Kurosaki, A. Charoenphakdee, S. Yamanaka, and G. J. Snyder, Enhancement of thermoelectric efficiency in PbTe by distortion of the electronic density of states, Science 321, 554 (2008), https://www.science.org/doi/pdf/10.1126/science.1159725 .
  * Inoue  _et al._ [2019] H. Inoue, H. Yoon, T. A. Merz, A. G. Swartz, S. S. Hong, Y. Hikita, and H. Y. Hwang, Delta-doped SrTiO3 top-gated field effect transistor, Applied Physics Letters 114, 231605 (2019).
  * Kim  _et al._ [2011] M. Kim, C. Bell, Y. Kozuka, M. Kurita, Y. Hikita, and H. Y. Hwang, Fermi surface and superconductivity in low-density high-mobility SrTiO3-δ, Phys. Rev. Lett. 107, 106801 (2011).
  * Spinelli  _et al._ [2010] A. Spinelli, M. A. Torija, C. Liu, C. Jan, and C. Leighton, Electronic transport in doped ${\text{SrTiO}}_{3}$: Conduction mechanisms and potential applications, Phys. Rev. B 81, 155110 (2010).
  * Perdew  _et al._ [1996] J. P. Perdew, K. Burke, and M. Ernzerhof, Generalized gradient approximation made simple, Phys. Rev. Lett. 77, 3865 (1996).
  * Momma and Izumi [2011] K. Momma and F. Izumi, VESTA3 for three-dimensional visualization of crystal, volumetric and morphology data, Journal of Applied Crystallography 44, 1272 (2011).


# Why multiple hypothesis test corrections provide poor control of false
positives in the real world

Stanley E. Lazic1,*

1 _Prioris.ai Inc., 459–207 Bank Street, Ottawa, K2P 2N2, Canada_

*Corresponding author<EMAIL_ADDRESS>
## Abstract

Most scientific disciplines use significance testing to draw conclusions about
experimental or observational data. This classical approach provides a
theoretical guarantee for controlling the number of false positives across a
set of hypothesis tests, making it an appealing framework for scientists
seeking to limit the number of false effects or associations that they claim
to observe. Unfortunately, this theoretical guarantee applies to few
experiments, and the true false positive rate (FPR) is much higher. Scientists
have plenty of freedom to choose the error rate to control, the tests to
include in the adjustment, and the method of correction, making strong error
control difficult to attain. In addition, hypotheses are often tested after
finding unexpected relationships or patterns, the data are analysed in several
ways, and analyses may be run repeatedly as data accumulate. As a result,
adjusted p-values are too small, incorrect conclusions are often reached, and
results are harder to reproduce. In the following, I argue why the FPR is
rarely controlled meaningfully and why shrinking parameter estimates is
preferable to p-value adjustments.

## Introduction

Scientists are often taught to analyse data using a null hypothesis
significance testing (NHST) framework. This involves posing the research
question as two mutually exclusive hypotheses. The first is the null
hypothesis ($H_{0}$), which asserts that there is no effect, correlation, or
association. The second is the alternative hypothesis ($H_{1}$), which states
there is an effect, correlation, or association, or more simply, that $H_{0}$
is false. The data are then tested for compatibility with the null hypothesis
by reducing them to a single number, called the test statistic, which is then
used to calculate a p-value. A small p-value indicates that the observed or a
more extreme test statistic is unlikely if the null hypothesis is true (and
all assumptions of the test are met, and there are no unmeasured biases).
Informally, a small p-value indicates that the observed effect, correlation,
or association is larger than expected, if nothing is actually occurring.
Based on this argument, the alternative hypothesis must be more probable
because there are only two alternatives ($H_{0}$ and $H_{1}$), and if one is
discredited, the other must be more likely. This implication does not follow
directly since a p-value is the probability of data, not a hypothesis.
Nevertheless, researchers often interpret a p-value as the probability that a
hypothesis is correct. NHST’s practical and philosophical limitations are not
considered here, as they have been covered elsewhere. [14, 29, 69].

A false positive, also known as a Type I error, occurs when no effect is
present ($H_{0}$ is true), but a researcher incorrectly concludes that an
effect exists. The false positive rate is the number of false positives in a
collection of tests. When using NHST, a predetermined threshold is set to
control the false positive error rate (denoted $\alpha$ and often set to
0.05), and if the p-value is less than $\alpha$, $H_{0}$ is rejected. When
$\alpha=0.05$, there is a 5% chance that an incorrect conclusion will be
reached, by definition, assuming $H_{0}$ is true. If many such tests are
conducted, there is a greater than 5% chance that at _least one_ test in the
collection will be a false positive. In response to this increased error rate,
post-hoc tests or multiple comparison procedures (MCP) have been developed,
which continues to this day. I argue that the MCP framework does not control
the false positive rate (FPR) in many real studies, and therefore contributes
to the poor reproducibility of research.

Researchers have proposed several reasons for irreproducibility, including
publication bias, selective reporting, inappropriate experimental designs and
analyses, and perverse incentives [45, 39, 3, 46]. A more basic explanation
has been overlooked: the true FPR is far from that assumed in many
experiments, even when analyses are conducted appropriately. I describe below
why the FPR is rarely controlled in any meaningful way in many experiments.
Some reasons are well known and have good solutions, but others are
unrecognised. The NHST approach also assumes that there is a single unique
$\alpha$ to control, but I argue that this is a myth. This paper concludes
with suggestions for improving inferences when testing multiple hypotheses.

## Reasons for poor control of false positives

Many experiments will suffer from at least one of the nine reasons given below
for why the FPR may differ from the nominal 0.05 level. These items describe
current practices in biology, which vary from field to field, and do not
necessarily reflect best practices or what researchers should do.

### Freedom to choose the family of comparisons

When using multiple testing procedures, researchers must define a family or
collection of hypothesis tests to be corrected together, and exclude other
hypothesis tests that may be conducted. Correction therefore occurs within
families, not between families. Researchers are free to arbitrarily choose the
family of tests, and although some conventions are followed, this is a often a
cultural practice rather than a principled choice. As an example, suppose an
experiment is designed as a 2-way ANOVA with two genotypes (wildtype and
knock-out) and three doses of a compound (none, low, high). A standard ANOVA
analysis yields three p-values: one for each main effect and another for the
interaction. These three p-values are rarely corrected for multiple testing
[15], but they are a family of related tests since they use the same data, are
part of the same statistical model, and use the same error term when
calculating the p-values. Ryan argued in 1959 that these p-values should be
corrected, but it has not become a standard practice [62]. However,
researchers often correct for multiple testing when comparing levels within a
factor, such as none versus low, and none versus high within the wildtype
group.

Oddly, the family of tests in the above experiment changes if it is a
functional genomics study with many outcome variables, despite having an
identical design. The family is usually a single comparison across all genes,
so all $\sim$20,000 genes tested in the wild-type group comparing none versus
low dose would be considered a family of tests. The p-value adjustment
disregards the other treatment group and genotype. It may seem more rigorous
to adjust 20,000 tests for a single comparison, as opposed to adjusting for
all possible comparisons for one gene. But, doesn’t adjusting for 20,000 genes
times the number of group comparisons make sense as a strategy to limit the
number of false positives across the entire experiment? Studies with more
complex designs have multiple genotypes, treatment groups, time points and
interactions between these, further complicating the determination of the
family of tests [30].

Other reasonable families could be considered, such as all tests relevant to a
research question. For example, does lesioning brain region X in mice affect
their behaviour? Several cognitive, affective, and motor assays, as well as
multiple readouts for each assay could be controlled as part of the same
family. That is, the FPR could be controlled across all behaviours and
readouts. Or, if the research question relates specifically to motor
behaviours, then all the motor assays and the multiple readouts for each motor
assay can be considered a family, and the FPR can be controlled for this
family. It is rare for all the tests that relate to a research question to be
grouped as a family, but it is a sensible way to control error rates. Another
reasonable family consists of all the tests reported in a figure. The graphs
in a multi-panel figure are often related and could be considered a family,
and a researcher may wish to control this overall error rate. A more stringent
approach is to adjust all the p-values reported in a paper, such that the
probability of at least one false positive amongst all the reported hypothesis
tests is 0.05. But what about tests reported in the supplementary material, or
tests conducted but not published, or those published in another paper [61,
57]?

My position is not that other families should be used, but that there are many
sensible options and that current practices are arbitrary [55], and likely
driven by software defaults, which then become cultural norms. Feise argues
that families of tests are not only arbitrary, but also ambiguously and
inconsistently defined [25]. Hence, researchers may choose to group their
hypothesis tests into many small families instead of fewer larger ones so that
less stringent corrections are applied. This provides weaker control of the
FPR and appears to be standard practice. Additionally, researchers may avoid
reporting non-significant comparisons because a less stringent correction is
required for the remaining hypotheses [1]. Such questionable research
practices contribute to the reproducibility crisis and are difficult for peer
reviewers and readers to detect.

### Freedom to choose the error rate to control

Once researchers define the family of tests to be corrected, they can choose
from four main error rates to control. The first is the per-comparison error
rate (PCER), which is the error rate for one comparison and is equal to the
significance threshold defined by the researcher, usually $\alpha=0.05$. The
PCER does not take into account the number of hypotheses tested, and therefore
does not correct for multiple testing. This is the appropriate error rate for
a single planned comparison.

The second and most common one is the familywise error rate (FWER)111Sometimes
the term experimentwise error rate is used. We use the more general term
familywise since an experiment is only one way of defining a family., which is
the probability of having at least one false positive in a collection of
tests, when the null hypothesis is true for all tests. This is the classic
error rate and is calculated as FWER = $1-(1-\alpha)^{m}$, where $\alpha$ is
the error rate for a single test and $m$ is the number of tests conducted.
Thus if $\alpha=0.05$ and $m=15$, we expect $1-(1-0.05)^{15}=0.54$, or a 54%
chance of at least one false positive among the 15 tests. This example
highlights how one of the oldest and simplest MCP works. The Bonferroni
correction controls the FPR using a smaller $\alpha$ cutoff, defined as the
$\alpha$ for a single test divided by the number of tests
($\alpha_{\mathrm{Bonferroni}}=0.05/15=0.0033$). The FWER is then
$1-(1-\alpha/m)^{m}=1-(1-0.0033/15)^{15}=0.0488$, or approximately the target
of 0.05, rounded to two decimal places. Alternatively, instead of using a new
cutoff, the p-values can be adjusted upward, and for the Bonferroni correction
the raw p-value is multiplied by the number of tests, so if the raw p-value is
0.0021, the adjusted value is $0.0021\times 15=0.0315$. Many MCPs work in a
similar way; a new cutoff is calculated from the number of tests (or the
p-value adjusted), but the details vary for each method.

The third and most recent error rate is the false discovery rate (FDR) [4].
The FDR is the proportion of false positives amongst all the tests declared
statistically significant. If 1000 tests are conducted and 200 are deemed
significant at an FDR = 0.05 level, then the interpretation is that 200
$\times$ 0.05 = 10 are false positives and the other 190 are true positives or
true discoveries. The FDR has weaker control over the FPR and is often used
for high throughput and high dimensional “omics” experiments, which have few
samples but many tests.

If controlling the FWER results in no or few significant p-values, researchers
are free to move the goal posts and control the FDR instead, or they can
choose the FDR from the start, thereby ensuring weaker error control. The FDR
is now occasionally used instead of the more rigorous FWER in experiments with
few hypothesis tests.

The fourth and least common error is the per-family error rate (PFER) [62, 70,
42], which is the expected rate at which false positives occur in the family
of tests. It is defined as the number of expected errors out of $m$ tests, so
if the error rate for a single test is $\alpha=0.05$ and 2000 tests are
conducted, the expected number of errors is $2000*0.05=100$. PFER is rarely
selected since it is the most stringent of the four methods, resulting in
fewer significant p-values. However, the popular Bonferroni correction
controls both the PFER and FWER, so it is often used in practice. The
Bonferroni correction is often criticised for being overly conservative, but
it is only conservative for the FWER; it is well calibrated for the PFER [42,
26]. If the goal is to minimise the number of false positives, then the PFER
is recommended over the FWER; whereas the FWER is recommended if, given one
false positive, subsequent false positives are less relevant [62, 26].

### Freedom to choose the correction method

After deciding the family of tests and the error rate, researchers have (too)
many options for the method of correction: for the FWER, options include
Bonferroni [9], progressive Bonferroni [35], Holm [36], Scheffe [64], Sidak
[67], Tukey [78], Student-Newman-Keuls [54, 41], Dunn [22], Duncan [21],
Dunnett [23] and so on; and for the FDR, options include Benjamini-Hochberg
[4], Benjamini-Yekutieli [5], Q-value [72, 73], local FDR, positive FDR [72,
73] marginal FDR [77], conditional FDR [77], empirical FDR [77], Black Box FDR
[74], Beta-Uniform mixture [59], and so on. Dozens of methods exist to correct
for multiple comparisons and researchers are free to choose (subject to
restrictions on the type of comparison), sometimes making their selection
based on the results, and choosing the method that provides significant
p-values for the key comparisons. Modern software makes it easy to click many
options, examine the output, and report the procedure that gives the desired
results (recommendations are available about preferred methods [17, 52]).

More knowledgeable researchers can avail themselves of further options. Many
older MCPs adjust all the hypotheses in a family together, such as the
Bonferroni example above. However, researchers can order hypotheses using
fixed-sequence, fallback, gatekeeping, and recycling procedures, where
hypotheses earlier in the order are subject to fewer, if any, corrections. [2,
83, 84, 20, 38, 48, 37]. As an example, in a study with a low dose, high dose,
and control group, the control versus high dose comparison would be tested
first, and if this is significant, only then would the control versus low dose
group be tested, with no penalty for having conducted the first test. As a
result, by declaring that a “fixed-sequence procedure” was used, researchers
can avoid any real correction for multiple testing. These methods are rarely
used outside of clinical trials, but only because they are unfamiliar to most
researchers and unavailable in popular statistical software. By continually
developing new ways to trade-off false positives and power, methods for
adjusting p-values have become more complex than the statistical models
themselves [11].

Stepping back from the purpose and justification of MCPs and the mathematics
underpinning them, we see that they are merely rank-preserving transformations
of p-values. In other words, these methods monotonically map p-values –
numbers between 0 and 1 – to a new set of numbers, also between 0 and 1.
Graphing p-values on the $x$-axis and adjusted p-valued on the $y$-axis plots
a convex curve, and the various methods only “bend the curve” differently. The
usefulness of controlling false positives by such remapping is open to debate.
Steinfatt proposed a simple alternative in 1979 that has been forgotten but
which would be suitable for many experiments [70]. He suggested dividing the
expected number of false positives by the observed number of significant
results to obtain what he called the Alpha Percentage ($\alpha$%). For
example, taking $\alpha=0.05$ and assuming 1000 tests are conducted, the
expected number of false positives is $1000*0.05=50$, and if 68 significant
results are observed, $\alpha$% = Expected/Observed $\times 100=50/68\times
100\approx 74$%. Hence, about a quarter of significant results are likely
false positives. This percentage will be reduced as the number of observed
significant p-values increases, and hence a lower $\alpha$% is better. Readers
can then interpret the unadjusted p-values in light of this number. Steinfatt
also suggested calculating $\alpha$% for several $\alpha$ values to better
understand how the number of excess significant p-values depends on $\alpha$.

### Procedures that provide poor control of the FPR are often used

Since there are so many correction procedures, it is not surprising that some
control error rates less effectively than others, and that researchers will
sometimes use them. At the extreme end, Fisher’s Least Significant Difference
(LSD) provides no correction for multiple testing. The LSD is a rule-of-thumb
to “only do post-hoc comparisons if the overall ANOVA F-test is significant”.
This if-then rule is rarely used, and even when it is, control of the FPR is
only adequate when there are three groups or comparisons [51]. Furthermore,
the rule-of-thumb is inappropriate for experiments with positive controls
because the overall F-test will always be significant (if not, there is a
problem with the experiment). Despite these problems, researchers frequently
report using Fisher’s LSD. The methods and results sections of journals from
the Public Library of Science, BioMed Central, and Nature Publishing Group
were searched on 22 June 2021 and approximately 3700 papers reported using
this method.

Other procedures such as the Newman-Keuls or Student-Newman-Keuls (SNK)
provide only weak control of the FPR. It’s better than nothing, but the true
FPR is higher than the advertised 5% level, and error control gets worse as
more comparisons are made. SNK is commonly used despite warnings from
statistical software manufacturers: “We offer the Newman-Keuls test for
historical reasons… but we suggest you avoid it because it does not maintain
the family-wise error rate at the specified
level”222https://www.graphpad.com/guides/prism/7/statistics/stat_options_tab_three-
way_anova.htm. Over 10,000 papers reported using this method from the above
publications.

With big effect sizes and few comparisons, many papers using Fisher’s LSD or
SNK would have come to the same conclusion if a better method had been used.
However, since a disproportionate number of published p-values are
suspiciously close to 0.05 [60, 49, 47, 19, 44, 80], more stringent control of
the FPR will no doubt change the conclusion for many.

### Exploratory analyses are uncontrolled

Prior to a formal analysis, statisticians recommend plotting and graphically
exploring the data to better understand the data and check for outliers,
clusters, unusual or impossible values, etc. [12, 13, 43, 46]. Visual
inspection of the data is strongly recommended; but, unanticipated patterns
and relationships are often found, which lead to new hypotheses.
Unfortunately, the same data cannot be used to both generate and test a
hypothesis, at least if control of the FPR matters. This process has been
called “HARKing” – Hypothesising After the Results are Known [40].

All MCPs take the number of tests into account, but this number is unknown
when data have been examined for patterns because it includes all the tests
that might have been done if the pattern was interesting enough [18]. Even
though only one test was conducted, it needs to be corrected for other
potential tests that might have been done, which is impossible to determine.
Hence, there is no way to adjust p-values to account for data-driven
hypotheses when testing for discovered relationships.

### Alternative analyses are uncontrolled

Researchers often try several models before settling on a final analysis. This
includes adding or removing covariates such as age or sex, transforming the
outcome to achieve normality or to stabilise variances, modifying likelihood
and link functions for generalised linear models, and so on. The most
appropriate model cannot always be determined when planning an experiment, so
it makes sense to try alternate models until a suitable one is found. But
researchers can then pick a model that gives the desired results and supports
their theory. Simmons and colleagues showed how easy it is to get significant
results with such alternate analyses [68]. Again, often there is no clearly
defined family of tests and therefore no way to adjust for multiple
alternative analyses.

### Multiple analyses as data accumulate are uncontrolled

In many settings the data accumulate over time and are analysed before the
experiment is complete. Examples include clinical samples, patients, or other
human subjects that are recruited into an experiment; cell culture experiments
where the whole procedure or protocol is repeated several times; or animal
experiments that are run on each litter of animals as they are born. Multiple
“looks” increases the FPR because there are more chances for results to be
significant, especially when an experiment is terminated early because the
desired result was found. Although there are ways to control the FPR in these
situations [8, 65], they are rare outside of clinical trials.

### Analyses of the same data by multiple research groups are uncontrolled

An underappreciated situation that increases the FPR is when multiple
individuals or research groups analyse the same data [76]. This is becoming
more common with data being deposited in repositories and consortia pooling
data, but members analyse them separately. Even if each research group
carefully controls the FPR, the same data is analysed in so many ways by so
many people that every false positive result will likely be found and
reported. For example, there are over 2000 primary publications in PubMed
using data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) and
nearly 15,000 primary publications using the National Health and Nutrition
Examination Survey (NHANES) data. Neither example has a single fixed dataset,
but multiple ones that have accumulated over time, so they suffer both from
“multiple looks” and multiple analysts, making it hard to define a family of
tests. Grootswagers and Robinson report a similar problem in computational
cognitive neuroscience [34].

### Suggestions from people in authority are uncontrolled

Finally, it is not uncommon for peer reviewers, editors, collaborators,
clients, PhD/postdoc supervisors, or other senior individuals to suggest
alternative analyses, tests, transformations, and removal of outliers – often
with the best of intentions. Even if a researcher maintained strict control of
the FPR, the additional requests often come after the analysis is done and can
increase the FPR. These “requests” may be hard to ignore by junior researchers
and are a source of friction for professional data analysts. Although the
suggestion from a more senior and knowledgeable person may be a good idea, it
still weakens strict error control.

## Solutions

The distance between the true and nominal FPR can be reduced in many ways, but
rarely will they be made equal. A simple solution is to avoid MCPs that don’t
control FPR well, such as Fisher’s LSD and Newman-Keuls methods. Researchers
can switch to other methods and journals can discourage their use. Tukey’s or
Holm’s method can be substituted in all cases, or more recent “simultaneous
tests” or “multiple marginal models” that account for correlations between
parameters [10, 58, 56, 81]. If corrections for multiple tests are not used,
this should be clearly stated.

FPRs get inflated when researchers have too much freedom to choose the family,
error rate, and correction method. Defining the analytical decisions before
the experiment is conducted limits the choices. A pre-registered analysis plan
specifies the options and prevents data-driven decisions increasing the FPR.
Nevertheless, the most liberal options can be chosen beforehand, which
mitigates the effectiveness of this approach. In addition, vague language like
“…the main analysis will be followed by post-hoc tests as appropriate” is
pointless and no better than a standard non-preregistered study. Constraints
are best suited for confirmatory experiments where there’s a clear hypothesis,
a primary outcome variable, and prior information to help define a statistical
model. However, since confirmatory experiments tend to have few hypotheses,
the risk of false positives is lower, and so MCPs likely work best where they
are needed the least.

Inflated FPRs also result from uncontrolled analyses: exploratory,
alternative, as data accumulate, by multiple groups, and from suggestions from
others. In these cases it is difficult or impossible to define the family of
tests, and so the standard MCPs cannot be used to control the FPR. One
solution is shrinkage, also known as regularisation or partial pooling, and is
described below.

### Shrink parameter estimates

There’s a debate over whether it’s better to estimate parameters (like effect
sizes) or to test hypotheses [16, 53]. The methods are complementary and many
studies report both p-values and effect sizes with confidence intervals. Here,
I am not arguing that parameter estimation should replace hypothesis testing,
but that parameter estimation helps solve multiple testing. The problem is not
that some p-values are too small, but that some effect sizes are too big.
Therefore the solution is to shrink parameters closer to zero or the null
value [80]. This approach recognises that p-values are not the problem, it’s
the effect sizes. Adjusting p-values is like treating the symptoms of a
disease instead of the cause. Furthermore, adjusted p-values do not fix the
visual impression of exaggerated effects when the data are plotted.

What causes a small p-value when there’s no true effect or association,
assuming that the experiment is carefully controlled and unbiased? The p-value
is based on a 1:1 mapping from a test statistic (along with the degrees of
freedom), so the test statistic must be too big (Fig. 1). What causes a large
test statistic? Even when there is no real difference or association,
sometimes a large effect size is observed due to sampling error or some other
confounding. A second reason is that even if the effect size is small or
moderate, it may be estimated with high precision. Precision is a function of
the sample size (N) and the variability of the data (SD), specifically the
within-group variability. Sometimes the SD is underestimated so that even a
small effect size can lead to a large test statistic and hence a significant
p-value. Would it not make more sense to fix the root cause instead of a
downstream effect?

Figure 1: What causes a small p-value? Ultimately, a large effect size or a
small within-group standard deviation (SD). Shrinkage methods typically adjust
both the effect size and the precision, which is a function of the SD and
sample size (N). Multiple comparison procedures (MCPs) adjust only p-values.

Shrinking parameters follows this logic: say we’re estimating the difference
between the means of two groups, call this $\theta$. Even if $\theta=0$, our
experiment will rarely return a value of exactly 0, sometimes it will be
greater than 0 and sometimes less than 0. Sometimes $\theta$ will be far from
0 because of sampling error and we will incorrectly conclude that $\theta\neq
0$. Knowing this, we can take the observed value of $\theta$ and pull or
shrink it closer to 0, so that we don’t get large values when there’s no
effect, thereby reducing the number of false positives. Although shrinking
might reduce statistical power since true effects will also be shrunk towards
0, often the reduction in false positives will offset that loss. This counter-
intuitive result, where the best estimate of $\theta$ is not the measured or
directly calculated value of $\theta$, is known as Stein’s Paradox [24, 63].

Figure 2 shows shrinkage with simulated data. Data were simulated for two
groups with ten samples each, and for 100 genes. The mean difference between
the groups for each gene was compared with an independent samples t-test. Two
data sets were simulated, one without group differences and one with five
genes differing between groups. Figure 1A shows the estimates (mean
difference) and standard errors for the 100 genes with no true effect. Figure
1B shows the shrunk estimates, which have all been pulled to zero with little
uncertainty, indicating that the model is confident that no effects exist.
Figure 1C shows estimates for the data set with five genes having different
expression levels (all with negative values). Figure 1D shows the shrunk
estimates, and while most genes have been shrunken to zero, four remain
distinct. Note how shrinking corrects the inflated effect sizes.

Figure 2: Shrinking parameters. Estimates (mean difference) and standard
errors for the data with no true effects (A), and the shrunken estimates (B),
which have all been shrunk to 0. Estimates and standard errors for the data
with five true negative effects (C), and the shrunken estimates (D), of which
four remain strongly negative.

Parameter estimates can be shrunk in three ways (Table 1). The first is to
take a Bayesian approach and place informative priors on the parameter
$\theta$ [31, 32]. For example, when comparing several groups to a control,
the parameters representing the difference between group means – call them
$\theta_{i}$ for $i=1,\dots,p$ comparisons – could have independent Normal(0,
$\sigma$) priors (see McElreath for an introduction to Bayesian methods [50]).
The observed mean differences ($\theta_{i}$) would shrink towards zero, with
the amount controlled by $\sigma$ (a small $\sigma$ provides more shrinkage).
This is a general approach but the control of the FPR critically depends on
$\sigma$. Often there is no theory to select a suitable value, but simulations
can help. Data are simulated from hypothetical experiments with no effects
present, a value for $\sigma$ is chosen, and the data analysed as they will be
in the real experiment. Repeat this for many simulated datasets and count the
number of false positives. If this number is greater than a desired threshold
(e.g. 5%), then $\sigma$ needs to be smaller. Repeat this procedure until a
suitable value of $\sigma$ is found, and the sample size can also be
calculated to ensure that statistical power is adequate. This approach works
best when there are few tests, and no connection between them; for example,
when comparing several clinical outcomes between a group of patients and
controls, and each outcome is analysed separately.

Table 1: Three approaches to shrinking parameter estimates. Method | When to use
---|---
Informative priors | Few tests, no relationship between them
Multilevel model | Moderate number of tests, same outcome
Two-step | Many tests, same outcome

The second approach uses a Bayesian or empirical Bayesian
multilevel/hierarchical model [28]. Here, we assume the $\theta_{i}$
parameters come from a common distribution with a mean of zero and a standard
deviation of $\sigma$. Unlike the first approach, $\sigma$ is estimated from
the data, and hence the amount of shrinkage is data-dependent: with little
variation between the group means, $\sigma$ will be small and the shrinkage
will be greater. This approach only works if outcomes are measured on the same
scale since all the parameters are a part of one big model. An example is the
data shown in Figure 2, where the expression of many genes is compared between
groups. A drawback of this approach is that with few groups or comparisons,
$\sigma$ will be hard to estimate, and so it’s best used with a moderate
number of tests.

Empirical Bayes methods are especially useful since there are no options to
select once the model structure is defined. Shrinkage is data-dependent, with
the amount of shrinkage depending on (1) the ratio of variability within and
between groups, (2) the number of parameters, and (3) the number of samples
within each group. A key feature is that the rank order of effects can change,
unlike with p-value adjustments, where the ordering of adjusted p-values stays
the same.

The third approach is a two-step method. First, summary statistics for the
comparisons are computed, such as effect sizes $\theta_{i}$ and their standard
errors. Then, empirical Bayesian methods shrink the effect sizes [75, 33, 71].
This approach is especially useful for “omics” and high-dimensional biology
experiments with many hypotheses, as in Figure 2 [71]. Another approach
developed by van Zwet and Gelman uses the signal-to-noise ratio of similar
published studies to estimate the prior distribution, which is then used to
shrink the parameters [79].

The main drawback with shrinking parameters to control false positives is that
the FPR can’t be determined precisely before experiments are performed,
although it can be estimated with simulation studies. The traditional approach
provides theoretical guarantees (e.g. that the FWER for $n$ tests will be
$\leq\alpha$ when using $\alpha/n$ as the significance threshold), which
rarely applies to real experiments, as argued above. Hence, an approximate
error rate that’s actually achieved is better than an exact theoretical error
rate that doesn’t apply. A second drawback is that these methods are not
implemented in popular software packages and so require more work than ticking
boxes, although they are available in R for omics experiments [71] and
correlation matrices [66].

Several advantages outweigh these drawbacks: results are less dependent on the
number of tests, adjustments can be made when the number of tests is not
defined, and options are fewer, so it’s harder for researchers to select
methods that give desired results. Researchers must choose a prior for the
Bayesian methods, but these can also be specified in advance. Shrinkage is
also a more direct solution since exaggerated effects are brought closer to
the true effects, on average.

## Discussion

The FPR is commonly viewed as a single well-defined entity, but the arguments
above make this idea untenable. First, many error rates exist – per-
comparison, familywise, false discovery, and per-family – so by definition
there cannot be a generic “false positive rate” that a researcher can control.
Multiple comparison procedures critically depend on the number of tests
included in the family, which is often hard to define, such as when the data
accumulate over time, when several people analyse the same data, when multiple
analyses are tried, and for exploratory studies. Berry called these silent or
hidden multiplicities [7, 6]. Consequently, adjusted p-values are impossible
to calculate and therefore meaningless – just like you cannot solve for $y$
when $x$ is unknown in $y=12/x$. It might be better to acknowledge that $y$ is
undefined instead of making up a fictitious value for $x$ to calculate $y$. In
cases where the family can be clearly defined, it’s usually arbitrary, and
other sensible families could have been chosen, leading to different results.
Finally, the bewildering number of procedures available means that different
results can be obtained for the same supposed error rate.

When studies assess the performance of MCPs, they always take the family as
unambiguously given, but since this is an ill-defined entity, performance of
MCPs in the real world is questionable. While there’s no direct proof, I
speculate that one reason for the “reproducibility crisis” is that the classic
MCP approach doesn’t work as well as it should. It is an ill-posed problem
because researchers are trying to control something that cannot be defined,
measured, and may not even exist. This means that the usual fixes such as
improving researcher incentives and training will have little impact on this
aspect of reproducibility. Although, limiting researcher degrees of freedom by
defining the analysis upfront and avoiding methods with poor performance can
help.

The classic MCP approach assumes that effects can be logically separated into
those that exist and those that do not, which is a simplistic view of the
world [27]. Like p-values, MCPs have a social function: they let researchers
make scientific claims. For this reason, p-values are not going away, despite
calls to reform, ban, or abandon their use (see Volume 73 of The American
Statistician [82]). In most fields, a researcher is allowed to claim there’s
an effect or an association if the p-value or adjusted p-value is less than
0.05. This makes MCPs necessary, the argument goes, since the likelihood of
false claims increases when many claims are made. We should keep false claims
under control, but is adjusting p-values really the best way? I argue the best
quantity to adjust is effect sizes and their precision, not p-values.
Shrinkage methods are rarely used to control the FPR in most fields, and so
little is known about their performance in the real world, but it is worth
exploring further. For example, how would these methods work if subsets of a
large dataset were analysed separately, mimicking the situation where multiple
people analyse parts of the data? MCPs are expected to have weaker control for
the subsets since the family of tests is smaller, and hence the FPR will
likely be inflated.

Since estimated effect sizes are too large, shrinking parameters is
fundamentally the right approach to make fewer false scientific claims. Since
p-values are calculated from the effect sizes, the argument is to move the
adjustment upstream. If effect sizes are improved, so are p-values; much like
treating the disease often improves the symptoms. However, shrinkage methods
are harder to apply because they require more knowledge than ticking a box. In
addition, the best approach will depend on the experimental design. Some
experiments are simple with few comparisons, like a clinical trial that
compares a low and high dose of a compound against a placebo. Other
experiments have simple designs (e.g. healthy versus disease) but with
thousands of outcomes, such as gene or protein expression values. Others have
a single outcome but many comparisons, such as a screen testing thousands of
drugs. Finally, some experiments run several assays on the same samples or
subjects, each with various outcomes and factors (e.g. a typical neuroscience
experiment with several behavioural assays and several outcomes per assay).
The best way to shrink estimates will probably vary between designs, and we
don’t have any data comparing the options. Shrinking estimates gives up the
theoretical guarantees of the MCP approach, but since these guarantees do not
apply in many real world settings, little is lost.

The paper encourages scientists to use shrinkage methods in their research,
perhaps along with standard ones, and to compare the two. Eventually,
scientists will be able to compare the relative merits of this approach on the
types of data they routinely generate. We also need quantitative researchers
to assess these methods using simulations to understand when they work well,
when they fail, and if there are sensible default options. This would be more
valuable than developing yet another multiple comparison procedure. In the
end, we shouldn’t assume effects either exist or don’t exist, and try to
reduce false positives; instead, we should minimise the discrepancy between
true and estimated effects, and wrong conclusions will be automatically
controlled.

## References

  * [1] Althouse AD (2016). Adjust for Multiple Comparisons? It’s Not That Simple. _The Annals of thoracic surgery_ 101: 1644–1645.
  * [2] Bauer P, Röhmel J, Maurer W, Hothorn L (1998). Testing strategies in multi-dose experiments including active control. _Statistics in Medicine_ 17(18): 2133–2146.
  * [3] Begley CG, Ioannidis JPA (2015). Reproducibility in Science: Improving the Standard for Basic and Preclinical Research. _Circ Res_ 116(1): 116–126.
  * [4] Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. _Journal of the Royal Statistical Society. Series B_ 57: 289–300.
  * [5] Benjamini Y, Yekutieli D (2005). False discovery rate-adjusted multiple confidence intervals for selected parameters. _Journal of the American Statistical Association_ 100: 71–81.
  * [6] Berry D (2012). Multiplicities in cancer research: ubiquitous and necessary evils. _J Natl Cancer Inst_ 104(15): 1124–1132.
  * [7] Berry DA (2007). The difficult and ubiquitous problems of multiplicities. _Pharm Stat_ 6(3): 155–160.
  * [8] Berry SM, Carlin BP, abd Peter Muller JJL (2010). _Bayesian Adaptive Methods for Clinical Trials_. Boca Raton, FL: CRC Press.
  * [9] Bonferroni CE (1936). Teoria statistica delle classi e calcolo delle probabilità. _Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze_ 8: 3–62.
  * [10] Bretz F, Hothorn T, Westfall P (2011). _Multiple Comparisons Using R_. Boca Ratan, FL: CRC Press.
  * [11] Bretz F, Posch M, Glimm E, Klinglmueller F, Maurer W, Rohmeyer K (2011). Graphical approaches for multiple comparison procedures using weighted Bonferroni, Simes, or parametric tests. _Biometrical Journal_ 53(6): 894–913.
  * [12] Cleveland WS (1993). _Visualizing Data_. New Jersey: Hobart Press.
  * [13] Cleveland WS (1994). _The Elements of Graphing Data_. New Jersey: Hobart Press, revised edn.
  * [14] Cohen J (1994). The earth is round (p $<$ .05). _American Psychologist_ 49(12): 997–1003.
  * [15] Cramer AOJ, van Ravenzwaaij D, Matzke D, Steingroever H, Wetzels R, Grasman RPPP, Waldorp LJ, Wagenmakers EJ (2015). Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies. _Psychonomic Bulletin & Review_ 23(2): 640–647.
  * [16] Cumming G (2013). The new statistics. _Psychological Science_ 25(1): 7–29.
  * [17] Day RW, Quinn GP (1989). Comparisons of Treatments After an Analysis of Variance in Ecology. _Ecological Monographs_ 59(4): 433–463.
  * [18] de Groot A (2014). The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. _Acta Psychologica_ 148: 188–194.
  * [19] de Winter JC, Dodou D (2015). A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too). _PeerJ_ 3: e733.
  * [20] Dmitrienko A, Offen WW, Westfall PH (2003). Gatekeeping strategies for clinical trials that do not require all primary effects to be significant. _Statistics in Medicine_ 22(15): 2387–2400.
  * [21] Duncan DB (1955). Multiple Range and Multiple F Tests. _Biometrics_ 11(1): 1.
  * [22] Dunn OJ (1964). Multiple Comparisons Using Rank Sums. _Technometrics_ 6(3): 241–252.
  * [23] Dunnett CW (1955). A Multiple Comparison Procedure for Comparing Several Treatments with a Control. _Journal of the American Statistical Association_ 50(272): 1096–1121.
  * [24] Efron B, Morris C (1977). Stein’s paradox in statistics. _Scientific American_ 236(5): 119–127.
  * [25] Feise RJ (2002). Do multiple outcome measures require p-value adjustment? _BMC Med Res Methodol_ 2: 8.
  * [26] Frane AV (2015). Are per-family Type I error rates relevant in social and behavioral science? _Journal of Modern Applied Statistical Methods_ 14(1): 12–23.
  * [27] Gelman A (2015). The connection between varying treatment effects and the crisis of unreplicable research: a Bayesian perspective. _Journal of Management_ 41(2): 632–643.
  * [28] Gelman A, Hill J, Yajima M (2012). Why we (usually) don’t have to worry about multiple comparisons. _Journal of Research on Educational Effectiveness_ 5: 189–211.
  * [29] Goodman S (2008). A dirty dozen: twelve p-value misconceptions. _Semin Hematol_ 45(3): 135–140.
  * [30] Grabitz CR, Button KS, Munafò MR, Newbury DF, Pernet CR, Thompson PA, Bishop DVM (2018). Logical and Methodological Issues Affecting Genetic Studies of Humans Reported in Top Neuroscience Journals. _Journal of cognitive neuroscience_ 30: 25–41.
  * [31] Greenland S (1992). A semi-Bayes approach to the analysis of correlated multiple associations, with an application to an occupational cancer-mortality study. _Statistics in Medicine_ 11(2): 219–230.
  * [32] Greenland S, Mansournia MA (2015). Penalization, bias reduction, and default priors in logistic and related categorical and survival regressions. _Statistics in Medicine_ 34(23): 3133–3143.
  * [33] Greenland S, Robins JM (1991). Empirical-Bayes Adjustments for Multiple Comparisons Are Sometimes Useful. _Epidemiology_ 2(4): 244–251.
  * [34] Grootswagers T, Robinson AK (2021). Overfitting the literature to one set of stimuli and data. _Frontiers in Human Neuroscience_ 15\.
  * [35] Hewitt J, Legendre P, McArdle B, Thrush S, Bellehumeur C, Lawrie S (1997). Identifying relationships between adult and juvenile bivalves at different spatial scales. _Journal of Experimental Marine Biology and Ecology_ 216(1-2): 77–98.
  * [36] Holm S (1979). A simple sequentially rejective multiple test procedure. _Scandinavian Journal of Statistics_ 6(2): 65–70.
  * [37] Huque M, Mushti S, Alosh M (2018). _Biopharmaceutical Applied Statistics Symposium: Volume 3 Pharmaceutical Applications_ , chap. Recycling of Significance Levels in Testing Multiple Hypotheses of Confirmatory Clinical Trials, 91–118. Singapore: Springer.
  * [38] Huque MF, Alosh M (2008). A flexible fixed-sequence testing method for hierarchically ordered correlated multiple endpoints in clinical trials. _Journal of Statistical Planning and Inference_ 138(2): 321–335.
  * [39] Ioannidis JPA (2014). How to make more published research true. _PLoS Med_ 11(10): e1001747.
  * [40] Kerr NL (1998). HARKing: hypothesizing after the results are known. _Pers Soc Psychol Rev_ 2(3): 196–217.
  * [41] Keuls M (1952). The use of the ”studentized range”in connection with an analysis of variance. _Euphytica_ 1(2): 112–122.
  * [42] Klockars AJ, Hancock GR (1994). Per-experiment error rates: The hidden costs of several multiple comparison procedures. _Educational and Psychological Measurement_ 54(2): 292–298.
  * [43] Krause A, O’Connell M (Eds.) (2012). _A Picture is Worth a Thousand Tables: Graphics in Life Sciences_. New York, NY: Springer.
  * [44] Krawczyk M (2015). The Search for Significance: A Few Peculiarities in the Distribution of P Values in Experimental Psychology Literature. _PLoS One_ 10(6): e0127872.
  * [45] Landis SC, Amara SG, Asadullah K, Austin CP, Blumenstein R, Bradley EW, Crystal RG, Darnell RB, Ferrante RJ, Fillit H, Finkelstein R, Fisher M, Gendelman HE, Golub RM, Goudreau JL, Gross RA, Gubitz AK, Hesterlee SE, Howells DW, Huguenard J, Kelner K, Koroshetz W, Krainc D, Lazic SE, Levine MS, Macleod MR, McCall JM, Moxley RT, Narasimhan K, Noble LJ, Perrin S, Porter JD, Steward O, Unger E, Utz U, Silberberg SD (2012). A call for transparent reporting to optimize the predictive value of preclinical research. _Nature_ 490(7419): 187–191.
  * [46] Lazic SE (2016). _Experimental Design for Laboratory Biologists: Maximising Information and Improving Reproducibility_. Cambridge, UK: Cambridge University Press.
  * [47] Leggett NC, Thomas NA, Loetscher T, Nicholls MER (2013). The life of p: ”just significant” results are on the rise. _Q J Exp Psychol (Hove)_ 66(12): 2303–2309.
  * [48] Li JD, Mehrotra DV (2008). An efficient method for accommodating potentially underpowered primary endpoints. _Statistics in Medicine_ 27(26): 5377–5391.
  * [49] Masicampo EJ, Lalande DR (2012). A peculiar prevalence of p values just below .05. _Q J Exp Psychol (Hove)_ 65(11): 2271–2279.
  * [50] McElreath R (2016). _Statistical Rethinking: A Bayesian Course with Examples in R and Stan_. Boca Raton, FL: CRC Press.
  * [51] Meier U (2006). A note on the power of Fisher’s least significant difference procedure. _Pharm Stat_ 5(4): 253–263.
  * [52] Midway S, Robertson M, Flinn S, Kaller M (2020). Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test. _PeerJ_ 8: e10387.
  * [53] Morey RD, Rouder JN, Verhagen J, Wagenmakers EJ (2014). Why hypothesis tests are essential for psychological science. _Psychological Science_ 25(6): 1289–1290.
  * [54] Newman D (1939). The distribution of range in samples from a normal population, expressed in terms of an independent estimate of standard deviation. _Biometrika_ 31(1-2): 20–30.
  * [55] O’Keefe DJ (2003). Colloquy: Should Familywise Alpha Be Adjusted? _Human Communication Research_ 29(3): 431–447.
  * [56] Pallmann P, Ritz C, Hothorn LA (2018). Simultaneous small-sample comparisons in longitudinal or multi-endpoint trials using multiple marginal models. _Statistics in medicine_ 37: 1562–1576.
  * [57] Perneger TV (1998). What’s wrong with Bonferroni adjustments. _BMJ_ 316(7139): 1236–1238.
  * [58] Pipper CB, Ritz C, Bisgaard H (2011). A versatile method for confirmatory evaluation of the effects of a covariate in multiple models. _Journal of the Royal Statistical Society Series C (Applied Statistics)_ 61(2): 315–326.
  * [59] Pounds S, Morris SW (2003). Estimating the occurrence of false positives and false negatives in microarray studies by approximating and partitioning the empirical distribution of p-values. _Bioinformatics_ 19(10): 1236–1242.
  * [60] Ridley J, Kolm N, Freckelton RP, Gage MJG (2007). An unexpected influence of widely used significance thresholds on the distribution of reported P-values. _J Evol Biol_ 20(3): 1082–1089.
  * [61] Rothman KJ (1990). No adjustments are needed for multiple comparisons. _Epidemiology_ 1(1): 43–46.
  * [62] Ryan TA (1959). Multiple comparison in psychological research. _Psychological Bulletin_ 56(1): 26–47.
  * [63] Samworth JR (2012). Stein’s paradox. _Eureka_ 62: 38–41.
  * [64] Scheffé H (1959). _The Analysis of Variance_. Wiley-Interscience Publication.
  * [65] Schonbrodt FD, Wagenmakers EJ, Zehetleitner M, Perugini M (2017). Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. _Psychol Methods_ 22(2): 322–339.
  * [66] Schäfer J, Strimmer K (2005). A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. _Statistical Applications in Genetics and Molecular Biology_ 4(1).
  * [67] Sidak Z (1967). Rectangular Confidence Regions for the Means of Multivariate Normal Distributions. _Journal of the American Statistical Association_ 62(318): 626–633.
  * [68] Simmons JP, Nelson LD, Simonsohn U (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. _Psychol Sci_ 22(11): 1359–1366.
  * [69] Stang A, Poole C, Kuss O (2010). The ongoing tyranny of statistical significance testing in biomedical research. _Eur J Epidemiol_ 25(4): 225–230.
  * [70] Steinfatt TM (1979). The alpha percentage and experimentwise error rates in communication research. _Human Communication Research_ 5(4): 366–374.
  * [71] Stephens M (2017). False discovery rates: a new deal. _Biostatistics_ 18(2): 275–294.
  * [72] Storey JD (2002). A direct approach to false discovery rates. _Journal of the Royal Statistical Society, Series B_ 64(3): 479–498.
  * [73] Storey JD (2003). The positive false discovery rate: A Bayesian interpretation and the q-value. _Annals of Statistics_ 31(6): 2013–2035.
  * [74] Tansey W, Wang Y, Blei D, Rabadan R (2018). Black Box FDR. _Proceedings of the 35th International Conference on Machine Learning_ 80: 4867–4876.
  * [75] Thomas DC, Siemiatycki J, Dewar R, Robins J, Goldberg M, Armstrong BG (1985). The problem of multiple inference in studies designed to generate hypotheses. _American Journal of Epidemiology_ 122(6): 1080–1095.
  * [76] Thompson WH, Wright J, Bissett PG, Poldrack RA (2020). Dataset decay and the problem of sequential analyses on open datasets. _eLife_ 9\.
  * [77] Tsai CA, miin Hsueh H, Chen JJ (2003). Estimation of False Discovery Rates in Multiple Testing: Application to Gene Microarray Data. _Biometrics_ 59(4): 1071–1081.
  * [78] Tukey JW (1949). Comparing Individual Means in the Analysis of Variance. _Biometrics_ 5(2): 99.
  * [79] van Zwet E, Gelman A (2021). A proposal for informative default priors scaled by the standard error of estimates. _The American Statistician_ 1–9.
  * [80] van Zwet EW, Cator EA (2021). The significance filter, the winner's curse and the need to shrink. _Statistica Neerlandica_ 1–16.
  * [81] Vogel C, Schaarschmidt F, Ritz C, Koenig F, Hothorn LA (2020). Model-based simultaneous inference for multiple subgroups and multiple endpoints. _arXiv_ .
  * [82] Wasserstein RL, Schirm AL, Lazar NA (2019). Moving to a world beyond ”p $<$ 0.05”. _The American Statistician_ 73(Supp 1): 1–19.
  * [83] Westfall PH, Krishen A (2001). Optimally weighted, fixed sequence and gatekeeper multiple testing procedures. _Journal of Statistical Planning and Inference_ 99(1): 25–40.
  * [84] Wiens BL (2003). A fixed sequence Bonferroni procedure for testing multiple endpoints. _Pharmaceutical Statistics_ 2(3): 211–215.