Do t-Statistic Hurdles Need to be Raised?
Andrew Y. Chen
Federal Reserve Board

Many scholars have called for raising statistical hurdles
to guard against false discoveries in academic publications. I show
these calls may be difficult to justify empirically. Published data
exhibit bias: results that fail to meet existing hurdles are often
unobserved. These unobserved results must be extrapolated, which can
lead to weak identification of revised hurdles. In contrast, statistics
that can target only published findings (e.g. empirical Bayes shrinkage
and the FDR) can be strongly identified, as data on published findings
is plentiful. I demonstrate these results theoretically and in an
empirical analysis of the cross-sectional return predictability literature.

JEL Classification: G0, G1, C1

Keywords: stock market predictability, stock market
anomalies, p-hacking, multiple testing empty

§ INTRODUCTION

Suppose a researcher proposes a “factor” behind a phenomenon.
How do we determine if this factor is worth noting? At least since
Fisher, 1925, researchers have used the following
procedure: (1) construct a statistic that is Student's t-distributed
in the case that the factor is false, and (2) declare the factor a
discovery if this t-statistic exceeds 1.96 in absolute value.[In the traditional language, a “false factor” is called a “true
null hypothesis” while a “true factor” is called a “false
null hypothesis.” Some readers may find the traditional language
confusing, hence my choice of terminology.] More recently, several papers have called for raising this t-statistic
hurdle, or “t-hurdle” to guard against false discoveries (<cit.>;
Chordia et al., 2020), including a paper with dozens of co-authors
(Benjamin et al., 2018).

In this paper, I examine whether these calls can be empirically justified.
An empirical justification is critical, as prior beliefs regarding
cutting edge research are sure to vary across scholars.[For contrasting prior beliefs regarding asset pricing, see Cochrane, 2017
and Barberis, 2018.] Moreover, an empirical justification seems possible, as multiple
testing statistics provide methods for estimating hurdles that control
the false discovery rate (FDR) (Benjamini and Hochberg, 1995).
Indeed, FDR methods are central to <cit.>'s argument
and the concept is also used in Benjamin et al., 2018.

Results summary

I find that raising the t-hurdle may be difficult to justify empirically.
My results differ from previous studies because I explicitly acknowledge
identification problems. Identification is important because the data
on academic discoveries exhibit publication bias: results that fail
to meet the existing t-hurdle are often unobserved. Unobserved results
need to be extrapolated, which can lead to weak identification: likelihood
functions that depend little on certain model parameters (Canova and Sala, 2009).
I characterize this problem in a theoretical analysis that extends
Benjamini and Hochberg, 1995 to a setting with publication bias
(as in Hedges, 1992). In an empirical analysis, I bootstrap
t-hurdle estimates using a rich dataset of published cross-sectional
stock return predictors (Chen and Zimmermann, 2022). Consistent
with the theory, the empirical estimates say little about whether
hurdles should be raised, stay the same, or even be lowered.

At the same time, I find other multiple testing statistics are more
strongly identified. Empirical Bayes shrinkage and the FDR among published
factors (Chen and Zimmermann, 2020, Chen, 2021) focus on the right
tail of t-stats, and this portion of the distribution tends to be
well-observed, in spite of publication bias. For cross-sectional predictors,
I find that published t-stats are biased upward by at most 28% and
that the FDR for published predictors is at most 22%, with 95% confidence.
This strong identification, as well as the weak identification of
t-hurdles, is also found across 10 alternative model specifications.
Robustness is intuitive, as the theoretical results impose few functional
form assumptions. Readers may differ on whether a worst-case FDR of
22% is satisfactory, but at least some will interpret these results
as implying that $t$-hurdles need not be raised for the cross-sectional
predictability literature.

Help reader with counterintuitive result and give fun MT intuition

How can t-hurdles not be raised? This idea seems to fly in
the face of multiple testing logic. If one tests 1,000 factors, 5%
will meet the 1.96 hurdle on average, even if all 1,000 factors are
false. This scenario is visualized in Panel (a) of Figure <ref>,
which depicts the distribution of absolute t-stats implied by a standard
normal. By luck, 50 of the 1,000 false factors meet the classical
hurdle. But since all factors are false, the FDR is 50/50 = 100%.[Here, the FDR is the number of false and significant factors, divided
by the number of significant factors, in expectation (Benjamini and Hochberg, 1995).] Clearly, this scenario implies that the classical hurdle needs to
be raised.

Why t-hurdles may not need to be raised. Panels illustrate
two possible scenarios. Bars are stacked. Panel (a) shows a scenario
with 1,000 factors, all of which are false. Panel (b) shows 1,000
false factors and 1,000 true factors, with the distribution of true
factors selected to match data on published t-statistics in <cit.>.

[All False Scenario]
[Data-Like Scenario]

The problem with this logic is that it ignores the information obtained
from multiple testing. Each additional test is another data point,
which brings news about the veracity of factors overall. This news
can in general be positive or negative. More data does not necessarily
mean more bad news.

The data on published factors looks more like Panel (b).[<cit.> shows the distribution of t-stats
from meta-studies in political science, psychology, economics, and
all fields of science. Of t-stats that exceed 1.96, roughly half also
exceed 3.0. In Figure <ref>, Panel (b), parameters are
selected to match this moment. A similar distribution is seen in cross-sectional
asset pricing (<cit.>).] In this panel, there are once again 1,000 false factors (red), so
50 false factors sneak past the 1.96 hurdle. But now there are an
additional 1,000 true factors (blue), and among all factors, 1,000
have t-stats that exceed 1.96. Thus, the 50 false factors represent
only 5% of the 1,000 factors that meet the classical hurdle. Supposing
that an FDR of 5% is sufficient, as it is in a variety of fields
from genomics to functional imaging (Benjamini, 2020),
the classical hurdle need not be raised at all.

Panel (b) illustrates how multiple testing provides more information.
With a single test, asking the question “what share of t-stats exceed
1.96?” is impossible to answer. But with multiple testing, answering
this question is a straightforward counting exercise. With many tests
one can also estimate the share of factors that are false (e.g. Storey, 2002).
If this share is small, then a t-stat of 1.5 may be due to bad
luck, in which case the classical hurdle can be lowered (Benjamini and Hochberg, 2000).

Detailed theoretical results:

My theoretical results characterize exactly when FDR methods imply
a raising of statistical hurdles. The theory revolves around $\pi_{F}$,
the share of false factors among all factors under consideration.
Assuming that an FDR of 5% is sufficient, the t-hurdle should be
raised if and only if $\pi_{F}$ exceeds the share of t-stats larger
than 1.96. Panel (b) of Figure <ref> shows the knife edge
case in which both of these objects are equal to 50%, and thus the
classical t-hurdle is sufficient.

The key to empirically justifying a higher t-hurdle, then, is to empirically
justify a large $\pi_{F}$. Unfortunately, my theoretical results
suggest that $\pi_{F}$ is weakly identified under publication bias.
I show that, if false and true factors are in a sense distinct, then
values of $\pi_{F}$ ranging from 0 to $2/3$ imply almost identical
distributions in the region in which data is well-observed. As a result,
empirical studies may say little about the proper t-hurdle.

Empirical results

To quantify this problem, I specify a parametric model of biased publication
and fit it to the Chen and Zimmermann, 2022 dataset using quasi-maximum
likelihood. The model includes the key parameter $\pi_{F}$, as well
as additional parameters that fully describe the distribution of t-stats.
The estimated parameters, in turn, provide consistent formulas for
revised t-hurdles. Bootstrapped estimates show that $\pi_{F}$ is
weakly identified: I re-sample the data, re-estimate the model, and
find that the estimated $\pi_{F}$ ranges from 0% to 70% (90% confidence
interval). t-hurdles that control the FDR at 5% range from 0 to 3.0
(90% C.I.).

Strong Identification

More positively, I find that multiple testing statistics that target
published findings are strongly identified, even in the presence of
publication bias. The shrinkage and FDR for published t-stats tend
to be determined by the properties of true factors. And under the
same conditions that imply weak identification of $\pi_{F}$, the
properties of true factors are strongly identified. Intuitively, if
$\pi_{F}$ does not affect the observed distribution, it must be the
properties of true factors that determine the data. Empirical estimates
confirm strong identification for the Chen-Zimmermann dataset. These
results are robust to a cluster-bootstrap that closely mimics correlations
in the empirical data.

An alternative method for handling publication bias is to algorithmically
generate a complete set of factors (Yan and Zheng, 2017, Chordia et al., 2020).
Standard FDR methods can then be applied and $\pi_{F}$ estimated
without the identification problems introduced by publication bias.
This $\pi_{F}$, however, should be interpreted as an upper bound
on the $\pi_{F}$ of the literature, as expert researchers should
be able to find true factors at a higher rate than a simple algorithmic
procedure (Chen, 2021).

Code to replicate all figures and tables can be found at <https://github.com/chenandrewy/qml-pub-bias>.

§.§ Related Literature

Many papers examine multiple testing effects in cross-sectional asset
pricing.[These papers include Harvey et al., 2016, Yan and Zheng, 2017, Chordia et al., 2020, Chen and Zimmermann, 2020, Jacobs and Müller, 2020, Chen, 2021, Chen, 2021, Harvey and Liu, 2021;
and 40.] Among these papers, Harvey et al., 2016; Chordia et al., 2020;
and Harvey and Liu, 2021 focus on t-hurdle corrections. To
these papers, I add a framework for understanding when t-hurdles need
to be raised. The framework shows that most estimates in Harvey et al., 2016
and Chordia et al., 2020 assume that t-stat hurdles
need to be raised, and thus they cannot answer the question of whether
t-hurdles need to be raised. It also highlights how the multiple testing
algorithm perhaps most common in finance (Benjamini and Yekutieli, 2001
Theorem 1.3) is likely too conservative.

I also add to Harvey et al., 2016; Chordia et al., 2020;
and Harvey and Liu, 2021 by examining the identification
problems that come with publication bias (Copas, 1999, Hedges and Vevea, 2005).
These problems would be seen in standard errors on t-hurdles, which
are not provided by Harvey et al., 2016 or Harvey and Liu, 2021.
I revisit their estimates and find that the standard errors are so
wide they say little about whether t-stat hurdles should be raised
or even be lowered.

In a contemporaneous paper, Harvey and Liu, 2021 critique
three assumptions that have been used in this literature: (1) cross-factor
correlations are all equal, (2) the publication probability is a strict
t-stat cutoff, and (3) the selected parametric models may be too restrictive.
I find that none of these issues has a significant effect on the small
shrinkage and FDR estimates found by Chen and Zimmermann, 2020
and 40. I find similar estimates assuming (1)
weak dependence across factors, (2) a smoothly increasing publication
probability, and (3) many distinct parametric forms for latent effect
sizes. Harvey and Liu propose instead to model latent effect sizes
by randomly drawing from empirical data on published and data-mined
factors. This method deviates strongly from the literature (Efron, 2012, Andrews and Kasy, 2019)
and it has not been shown to recover effect sizes either theoretically
or in simulations.

Identification under publication bias is also studied in Andrews and Kasy, 2019,
who prove identification of a non-parametric model. Their proof obtains
the latent distributions by solving a system of ODEs, effectively
assuming an unlimited sample of published tests. My analysis is less
restrictive about data availability. Also unlike Andrews and Kasy,
I connect to the literature on false discovery rates and empirical
Bayes shrinkage. More broadly, my paper helps bridge the literatures
on publication bias (Hedges and Vevea, 2005, McShane et al., 2016)
and multiple testing (Efron, 2012) and provides guidance
on which multiple testing corrections should be applied under publication

There are other arguments against raising t-hurdles. Raising t-hurdles
would lead to published data that is in a sense more distorted. This
reasoning motivates, in part, the push for pre-analysis plans, which
lower t-hurdles provided that the analysis is rigorously pre-specified
(Olken, 2015, Kasy and Spiess, 2023). Raising t-hurdles also
fails to address common misinterpretations of statistical significance
and artificial dichotimization introduced by hypothesis testing (McShane et al., 2019, Chen and Zimmermann, 2023).

§ THEORETICAL RESULTS

I describe a general model (Section <ref>), define
multiple testing statistics (Section <ref>),
and then prove results that illustrate weak and strong identification
(Sections <ref>-<ref>). Section
<ref> explains why some commonly-used FDR
methods cannot answer the question of whether t-stat hurdles should
be raised. Section <ref> discusses false negative

§.§ A Model of Multiple Testing and Publication Bias

A literature is generated in two steps. In the first step, researchers
generate an “unbiased” set of $N$ factors that obey the assumptions
in Benjamini and Hochberg, 1995. Factor $i$ has a t-stat $t_{i}$
that follows

\begin{align}
t_{i}|\mu_{i} & \sim\text{Normal}\left(\mu_{i},1\right)\label{eq:theory-t_mu}
\end{align}

where $\mu_{i}$ is the latent effect size of factor $i$ (e.g. the
expected return) divided by its standard error. $\mu_{i}$ can also
be thought of as the “corrected” t-stat, since it corrects $t_{i}$
for sampling error. $\mu_{i}$ depends on whether factor $i$ is false
($F_{i}$) or true ($T_{i}$)

\begin{align}
\mu_{i}|F_{i} & =0.\label{eq:theory-mu_F}\\
\mu_{i}|T_{i} & \sim g\left(\cdot|\lambda\right),\label{eq:theory-mu_T}
\end{align}

where $g\left(\cdot|\lambda\right)$ is an arbitrary probability distribution
and $\lambda$ is a vector of parameters. Last, and most importantly,
factor $i$ is false with probability $\pi_{F}$.[Benjamini and Hochberg, 1995 leave open the possibility of other
distributional assumptions for Equation (<ref>), but
the standard normal is commonly used in practice (Efron, 2012;
Harvey et al., 2016). Chen, 2021 shows that the standard
normal assumption holds for long-short portfolios from the asset pricing

I describe the factors from the first step as “unbiased” because
$t_{i}|F_{i}\sim\text{Normal}\left(0,1\right),$ and thus the theory
of Fisher, 1925 and Benjamini and Hochberg, 1995
applies. However, the second step of literature generation creates
a bias.

In the second step, factor $i$ is published depending on its statistical

\begin{align}
\Pr\left(\pub_{i}\left||t_{i}|\right.\right) & =s\left(|t_{i}|\right)\label{eq:theory-pubprob}\\
s\left(|t_{i}|\right) & \quad\text{weakly increasing}\label{eq:theory-pubprob-increase}\\
s\left(|t_{i}|\right) & =\bar{s}\quad\text{for \ensuremath{|t_{i}|>\tgood}}\label{eq:theory-pubprob-max}
\end{align}

where $\pub_{i}$ is the event that factor $i$ is published, $s\left(|t_{i}|\right)$
is a function with values in $[0,1]$, and $\bar{s}$ and $\tgood$
are constants. Conceptually, $s(|t_{i}|)$ captures the prevailing
t-stat hurdles that were applied in the generation of the published

This second step implies $t_{i}|\left(F_{i},\pub_{i}\right)\nsim\text{Normal}\left(0,1\right)$,
violating the assumptions of both Fisher, 1925 and
Benjamini and Hochberg, 1995. Thus, to estimate the FDR and related
statistics, one needs to recover the properties of the unbiased factors
generated in the first step.

Equations (<ref>)-(<ref>)
describe a “satisficing” model of publication bias. They say that
a higher t-stat implies a higher probability of publication but once
the t-stat is high enough the community is satisfied and there are
no additional gains. A satisficing model is consistent with the view
that researchers are primarily incentivized to uncover convincing
and interesting mechanisms and that statistical significance plays
a secondary role. This assumption nests the functional forms used
in previous estimates of publication bias (Harvey et al., 2016, Andrews and Kasy, 2019, Chen and Zimmermann, 2020).

I refer to t-stats that exceed $\tgood$ as “well-observed.” $\tgood$
is typically 1.96 (Andrews and Kasy, 2019) or a number
between 1.96 and 3.0 (Harvey et al., 2016, Chen and Zimmermann, 2020).
Relatively few t-stats below $\tgood$ are observed and the data moments
in this region are distorted. In contrast, the likelihood conditional
on $|t_{i}|>\tgood$ is not distorted, so one can make inferences
about $\pi_{F}$ and $\lambda$ directly from the conditional moments
in this region.

In reality, the probability of publication depends not only on $|t_{i}|$
but also on supporting evidence. However, data on supporting evidence
is rarely available, so the publication bias literature focuses on
Equation (<ref>), which ignores supporting evidence.
The net effect of this misspecification is unclear. Omitting supporting
evidence tends to bias estimates of $\pi_{F}$ and $\lambda$ in a
way that implies stronger latent effects, as these parameters must
absorb the larger $|t_{i}|$ values induced by the supporting evidence.
On the other hand, omitting supporting evidence tends to imply weaker
latent effects, as the supporting evidence is by definition a signal
of strong latent effects. Appendix <ref> formalizes
these issues and provides some simulation evidence suggesting that
the net bias is small.

The model abstracts from dynamic issues like out-of-sample decay in
effect size. More generally, the effect size depends on whether it
is measured in the original sample ($\mu_{i}$) or out-of-sample ($\mu_{i}^{\text{OOS}}$).
While researchers typically aim to find factors with stable effect
sizes, empirical evidence in cross-sectional asset pricing finds $\mu_{i}^{\text{OOS}}\approx0.50\mu_{i}$
(McLean and Pontiff, 2016, Chen and Zimmermann, 2020). Thus, finding that
$\mu_{i}$ is close to $t_{i}$ does not necessarily imply out-of-sample

§.§ Hurdles that Control the False Discovery Rate

Calls for raising statistical hurdles often come down to controlling
the false discovery rate with a t-stat hurdle:

\begin{align}
\text{hurdle}\left(5\%\right) & \equiv\min_{h\in\mathbb{R}_{+}}\left\{ h:\Fdr\left(h\right)\le5\%\right\} .\label{eq:theory-hurdle}
\end{align}

where $\Fdr\left(h\right)$ is the Bayesian formulation from Efron, 2008:

\begin{align}
\Fdr\left(h\right) & \equiv\Pr\left(F_{i}\left||t_{i}|>h\right.\right).\label{eq:theory-Fdr-Bayes}
\end{align}

(see also Efron et al., 2001 and Storey, 2002).
Equation (<ref>) finds the lowest t-stat hurdle
that results in less than 5% of factors being false, where 5% is
chosen for ease of exposition (the main results hold for other choices).
$\Fdr\left(h\right)$ is used in both Ioannidis, 2005 and
Benjamin et al., 2018. 40 estimates
an object similar to $\Fdr\left(1.96\right)$ for cross-sectional
return predictors. The [Chen and Zimmermann, 2018] version
of Chen and Zimmermann, 2020 estimates both $\text{hurdle}\left(5\%\right)$
and $\Fdr\left(h\right)$.

As shown by Storey, 2002 and Storey et al., 2004,
the Bayesian formulation is equivalent to the Benjamini-Hochberg ([Benjamini and Hochberg, 1995, Benjamini and Hochberg, 2000])
approach under weak dependence. To see this, define the Benjamini-Hochberg

\begin{align}
\FDR_{BH}\left(h\right) & \equiv\E\left[\frac{\sum_{i=1}^{N}I\left(F_{i}\cap|t_{i}|>h\right)}{\sum_{i=1}^{N}I\left(|t_{i}|>h\right)}\right],\label{eq:theory-FDR-BH}
\end{align}

where $I\left(\cdot\right)$ is an indicator function and I assume
$\sum_{i=1}^{N}I\left(|t_{i}|>h\right)>0$. Then suppose the following
weak law of large numbers hold: For any $h\in\mathbb{R}_{+}$,

\begin{align}
\frac{1}{N}\sum_{i=1}^{N}I\left(|t_{i}|>h\right) & \overset{p}{\rightarrow}\Pr\left(|t_{1}|>h\right)\label{eq:theory-wlln-1}\\
\frac{1}{\pi_{F}N}\sum_{i=1}^{N}I\left(F_{i}\cap|t_{i}|>h\right) & \overset{p}{\rightarrow}\Pr\left(|t_{1}|>h|F_{1}\right).\label{eq:theory-wlln-2}
\end{align}

These expressions say that if you keep counting the shares of factors
that exceed a hurdle, you'll eventually find the probability that
a factor exceeds a hurdle. Rearranging Equation (<ref>)
and plugging in Equations (<ref>)-(<ref>)
leads to Equation (<ref>):

\begin{align}
\E\left[\frac{\sum_{i=1}^{N}I\left(F_{i}\cap|t_{i}|>h\right)}{\sum_{i=1}^{N}I\left(|t_{i}|>h\right)}\right] & =\E\left[\frac{\sum_{i=1}^{N}I\left(F_{i}\cap|t_{i}|>h\right)/\left(\pi_{F}N\right)}{\sum_{i=1}^{N}I\left(|t_{i}|>h\right)/N}\frac{\pi_{F}N}{N}\right]\nonumber \\
& \overset{p}{\rightarrow}\frac{\Pr\left(|t_{i}|>h|F_{i}\right)\Pr\left(F_{i}\right)}{\Pr\left(|t_{i}|>h\right)}=\Pr\left(F_{i}\left||t_{i}|>h\right.\right),\label{eq:theory-FDR-equiv}
\end{align}

where the last equality applies Bayes rule. Harvey et al., 2016's
Section 4 uses Equation (<ref>) in the constraint
of Equation (<ref>) to argue that t-stat hurdles
need to be raised.

§.§ Weak Identification of t-stat Hurdles

As discussed in Benjamini and Hochberg, 2000 (Remark 2), Equations
(<ref>) and (<ref>) imply that
hurdles need not be raised and may even be lowered. The following
proposition characterizes exactly when this occurs:

If $\,\Pr\left(F_{i}\left||t_{i}|>h\right.\right)$
is strictly decreasing in $h$, then

\begin{align}
& \text{hurdle}\left(5\%\right)>1.96\text{ if and only if }\text{\ensuremath{\pi_{F}}}>\Pr\left(|t_{i}|>1.96\right)\label{eq:theory-prop-result-1}\\
& \text{hurdle}\left(5\%\right)=1.96\text{ if and only if }\text{\ensuremath{\pi_{F}}}=\Pr\left(|t_{i}|>1.96\right)\\
& \text{hurdle}\left(5\%\right)<1.96\text{ if and only if }\text{\ensuremath{\pi_{F}}}<\Pr\left(|t_{i}|>1.96\right)\label{eq:theory-prop-result-3}
\end{align}

The proof is in Appendix <ref>.

Proposition <ref> says the t-stat hurdle needs to be raised
if and only if $\pi_{F}$ is larger than $\Pr\left(|t_{i}|>1.96\right)$.
This comparison is visualized in Panel (b) of Figure <ref>.
$\pi_{F}$ is the share of false factors (red) relative to the total
mass, while $\Pr\left(|t_{i}|>1.96\right)$ is the mass to the right
of the vertical line. In this illustration, these probabilities are
equal, so the multiple testing hurdle is equal to the classical one.
In contrast, Panel (a) shows a setting in which the share of the red
mass (100%) is much larger than the mass to the right of the vertical
line (5%), implying that hurdle needs to be raised.

Some readers may have the intuition that $\pi_{F}>\Pr\left(|t_{i}|>1.96\right)$
must be the “right” case of Proposition <ref>. This
intuition is perhaps natural, as statistics is a conservative institution.
However, there is no logical reason for why we must have $\pi_{F}>\Pr\left(|t_{i}|>1.96\right)$.
Assuming that this expression holds amounts to placing additional
restrictions on the model of Section <ref> and it
is unclear how to motivate these restrictions. Thus, it is the role
of data to tell us which case of Proposition <ref> is the
right one. For example, Benjamini et al., 2006 reviews several
methods for estimating $\pi_{F}$.

The methods in Benjamini et al., 2006, however, assume that
all factors are observed. Under publication bias (Equations (<ref>)-(<ref>))
it may be difficult to identify $\pi_{F}$. Indeed, the meta-study
literature on publication bias runs into flat likelihood functions
and estimation problems for other parameters (Copas, 1999
and Hedges and Vevea, 2005). The following proposition shows
publication bias leads to identification problems for $\pi_{F}$:

(Weak Identification) If there exists $\varepsilon$
such that for all $\bar{t}>\tgood$,

\begin{align}
\frac{\Pr\left(|t_{i}|\in[\tgood,\bar{t}]\bigl|F_{i}\right)}{\Pr\left(|t_{i}|\in[\tgood,\bar{t}]|T_{i}\right)} & <\varepsilon\label{eq:theory-weak-cond}
\end{align}

then for any $\pi\in[0,2/3]$, $\pi'\in[0,2/3]$, and $\bar{t}>\tgood$,

\begin{align}
\left|\frac{\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right)|_{\pi_{F}=\pi'}}{\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right)|_{\pi_{F}=\pi}}-1\right| & <2\varepsilon+O\left(\varepsilon^{2}\right),\label{eq:theory-weak-result}
\end{align}

where $\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right)|_{\pi_{F}=\pi}$

\begin{align}
\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right) & =\frac{\pi_{F}\Pr\left(|t_{i}|\in(\tgood,\bar{t}]\bigl|F_{i}\right)+\left(1-\pi_{F}\right)\Pr\left(|t_{i}|\in(\tgood,\bar{t}]\bigl|T_{i}\right)}{\pi_{F}\Pr\left(|t_{i}|>\tgood|F_{i}\right)+\left(1-\pi_{F}\right)\Pr\left(|t_{i}|>\tgood\bigl|T_{i}\right)}.\label{eq:theory-cond-CDF}
\end{align}

evaluated at $\pi_{F}=\pi$.

The proof is in Appendix <ref>.

Proposition 2 says that, under certain assumptions, $\pi_{F}$ has
little effect on model predictions. In particular, Equation (<ref>)
says the distribution of t-stats in the well-observed region changes
by at most a factor of $2\varepsilon$ across values of $\pi_{F}$
ranging from 0 to 2/3. The bound of 2/3 is chosen for illustrative
purposes. More generally, a bound of $\bar{\pi}$ for $\pi_{F}$ implies
an upper bound of $\bar{\pi}/(1-\bar{\pi})\varepsilon$ on the RHS
of Equation (<ref>) (see proof).

The key assumption is Equation (<ref>), which
formalizes the implicit assumption in hypotheses testing that true
and factors are distinct. If true and false factors are not distinct,
then the pursuit of separating true from false factors is in a sense
misguided. In my setting, false factors by definition have $|t_{i}|$
close to zero, so distinctiveness can be thought of as saying that
true factors are more likely to have $|t_{i}|>\tgood$. Equation (<ref>)
says that true factors are $1/\varepsilon$ times more likely to be
found in this region than false factors, where $\varepsilon$ is presumably

Figure <ref> illustrates Proposition <ref> by
overlaying alternative parameterizations of <cit.>'s
parametric model. HLZ's baseline estimate implies $\pi_{F}=0.444$
and $g\left(\cdot|\lambda\right)$ is an exponential distribution
with mean of about 2. This estimate implies that raising t-hurdles
is necessary. According to HLZ's Table 5, the classical t-hurdle of
1.96 needs to be raised to 2.27 to control the FDR at 5%. The distribution
of t-stats implied by HLZ's baseline estimate is shown in the blue
bars of Figure <ref>.

[Figure <ref> about here]

But now consider an alternative model, that uses all the same parameters
as HLZ's baseline but simply changes $\pi_{F}$ to zero. With no false
factors, the alternative model implies that t-hurdles can be lowered,
all the way to 0. The predicted distribution of t-stats is shown in
the red bars of Figure <ref>.

These two models differ markedly in their densities of t-stats near
zero. However, these t-stats are difficult to observe and must be
extrapolated. Indeed, HLZ assume $\tgood=2.57$ (vertical line), and
for $|t_{i}|>\tgood$ the two distributions are nearly identical.
Thus, it is unlikely that the data can tell us whether t-stats should
be raised, stay the same, or even be lowered.

This identification problem should show up in standard error estimates.
They imply that small perturbations to the data imply very different
t-stat hurdles, and thus large standard errors. HLZ do not provide
standard errors for their hurdle estimates. I fill this gap in Section

§.§ Strongly-Identified Multiple Testing Statistics

Revised hurdles are not the only way to deal with a many-factor setting.
Chapter 1 of Efron, 2012's textbook on large scale inference
examines empirical Bayes shrinkage:

\begin{align}
\text{shrinkage}\left(\bar{t}\right) & \equiv\frac{|\bar{t}|-E\left(\mu_{i}\left||t_{i}|=\bar{t}\right.\right)}{|\bar{t}|}\label{eq:theory-shrink-def}
\end{align}

where $\bar{t}\in\mathbb{R}_{+}$. Equation (<ref>)
measures how much you should shrink an observed t-stat ($|t_{i}|$)
toward zero in order to recover its unbiased counterpart ($E\left(\mu_{i}\left||t_{i}|=\bar{t}\right.\right)$).
For example, $\text{shrinkage}\left(4.0\right)=0.25$ means that a
t-stat of 4.0 should be shrunk by 25% and that the unbiased t-stat
is $4.0\times(1-0.25)=3.0$. Since the sample mean return is proportional
to $t_{i}$, the same shrinkage adjustment applies to the sample mean
return. Asset pricing papers that study empirical Bayes shrinkage
include Chen and Zimmermann, 2020, Chinco et al., 2021,
Chen and Velikov, 2023, and 40.

Chapter 2 of Efron, 2012 examines the local FDR:

\begin{align}
\fdr\left(\bar{t}\right) & \equiv\Pr\left(F_{i}\left||t_{i}|=\bar{t}\right.\right).\label{eq:theory-localfdr}
\end{align}

$\fdr\left(\bar{t}\right)$ is just the probability that a factor
with a t-stat of $\bar{t}$ is false. Integrating $\fdr\left(\bar{t}\right)$
from $h$ to infinity leads to Equation (<ref>).

In contrast to $\text{hurdle}\left(5\%\right)$, empirical Bayes shrinkage
and the local fdr can be used to target only published findings (by
selecting $\bar{t}$ to be equal to the t-stats of published factors).
And in the published region, shrinkage and the local fdr depend more
so on the parameter vector that governs true factors ($\lambda$).
To see this, note that the key term in Equation (<ref>)
can be written as

\begin{align}
E\left(\mu_{i}\left||t_{i}|=\bar{t}\right.\right) & =E\left(\mu_{i}\left||t_{i}|=\bar{t},T_{i}\right.\right)\left[\frac{1}{1+\Delta}\right]\label{eq:theory-bias-approx}\\
\Delta & \equiv\left(\frac{\pi_{F}}{1-\pi_{F}}\right)\left[\frac{f_{|t||F}\left(\bar{t}\right)}{f_{|t||T}\left(\bar{t}\right)}\right]\label{eq:theory-bias-approx2}
\end{align}

where $f_{|t||F}\left(\bar{t}\right)$ and $f_{|t||T}\left(\bar{t}\right)$
are the densities of $|t_{i}|$ given that $i$ is false or true,
respectively. Intuitively, shrinkage of the full model is the same
as shrinkage assuming $\pi_{F}=0$, but with a correction that increases
in $\pi_{F}$. A similar argument can be made for the local FDR. Appendix
<ref> provides the derivations.

For empirically relevant $\bar{t}$ and $\lambda$, $\Delta$ is often
small. For example, the HLZ dataset has a mean $|t_{i}|$ of around
4.0, corresponding to $f_{|t||F}\left(4.0\right)=0.00027$. Meanwhile,
their estimates imply $f_{|t||T}\left(4.0\right)=0.076$ , as true
factors are much more likely to have large t-stats. So even if $\pi_{F}$
is as large as 0.9, $\Delta$ is only 0.03. Similarly large t-stats
are commonly found in replications of cross-sectional predictability
papers (Chen and Zimmermann, 2022). More broadly, [Brodeur et al., 2016]
find that t-stats of 4.0 are quite common in the main hypotheses tests
reported in top economics journals.

This analysis rests critically on $\lambda$, which must be estimated
with published data and its associated identification problems. Fortunately,
the following proposition shows that $\lambda$ can be strongly identified

(Strongly Identified Parameters) If there exists
$\varepsilon$ such that for all $\bar{t}>\tgood$,

\begin{align}
\frac{\Pr\left(|t_{i}|\in[\tgood,\bar{t}]\bigl|F_{i}\right)}{\Pr\left(|t_{i}|\in[\tgood,\bar{t}]|T_{i}\right)} & <\varepsilon\label{eq:theory-weak-cond-1}
\end{align}

and $\pi_{F}<2/3$, then

\begin{align}
\left|\frac{P\left(|t_{i}|<\bar{t}||t_{i}|>\tgood\right)}{P\left(|t_{i}|<\bar{t}||t_{i}|>\tgood,T_{i}\right)}-1\right| & \le2\varepsilon+O\left(\varepsilon^{2}\right).\label{eq:theory-strong-result}
\end{align}

The proof is in Appendix <ref>.

Proposition <ref> says that, under the same condition
that lead to weak identification of t-hurdles (Equation (<ref>)),
the model's predictions about well-observed data ($P\left(|t_{i}|<\bar{t}||t_{i}|>\tgood\right)$)
are almost the same as the predictions that come from assuming all
factors are true ($P\left(|t_{i}|<\bar{t}||t_{i}|>\tgood,T_{i}\right)$).
As in the Proposition <ref>, this proposition uses the
$\pi_{F}<2/3$ for illustrative purposes. The more general case in
which $\pi_{F}$ is bounded by $\bar{\pi}$ implies an upper bound
of $\bar{\pi}/(1-\bar{\pi})\varepsilon$ on the RHS of Equation (<ref>)
(see proof).

An implication of Proposition <ref> is that a reasonable
estimate of $\lambda$ could potentially be found by estimating the
model while assuming $\pi_{F}=0$. This result can be seen in Figure
<ref>. The red bars are essentially the model that one would
estimate assuming $\pi_{F}=0$. Thus, assuming $\pi_{F}=0$ would
lead to roughly the same $\lambda$ as HLZ's baseline estimate, in
which $\pi_{F}=0.44$. Intuitively, if $\pi_{F}$ is not at all responsible
for fitting the t-stats to the right of 1.96, it must be the other
parameters in HLZ's estimate that accomplish this.

Taken with Equation (<ref>), Proposition <ref>
implies that the bias in large t-stats could potentially be identified
with just published data. This result is seen in the small standard
errors in Chen and Zimmermann, 2020's shrinkage estimates as well
as the robustness of their estimates to alternative modeling assumptions.
Indeed, Chen and Zimmermann's headline bias estimate of 12% is not
far from Harvey and Liu, 2021's median estimate of 19%,
even though the two papers use very different modeling assumptions
and estimation methods. Similar estimates are also found in Chinco et al., 2021
and 40, who use yet another set of methods (see
discussion in Chen and Zimmermann, 2023). Section <ref>
provides additional evidence that these bias estimates are strongly
identified for the cross-sectional predictability literature.

The key to strong identification is flexibility. Unlike, $\text{hurdle}(5\%)$,
$\text{shrinkage}\left(\bar{t}\right)$ and $\text{\fdr\ensuremath{\left(\bar{t}\right)} }$allow
one to focus on the portion of the data that is well-observed. If
the data are not informative about $\text{shrinkage}\left(1.0\right)$,
one can instead examine $\text{shrinkage}\left(4.0\right)$. The cost
of this flexibility is that $\text{shrinkage}\left(\bar{t}\right)$
and $\text{\fdr\ensuremath{\left(\bar{t}\right)}}$ do not provide
a precise answer to the question, “do t-hurdles need to be raised?”
They provide important evidence (e.g. published t-stats are biased
upward by 12%). But ultimately an additional framework is required
to convert this evidence into a new t-hurdle.

Also unlike $\text{hurdle}(5\%)$, $\text{shrinkage}\left(\bar{t}\right)$
provides economic magnitudes. Thus, it avoids the potentially artificial
dichotomization that can come from hypothesis testing (McShane et al., 2019).
This gain, however, comes at additional analytical costs, namely that
$\text{shrinkage}\left(\bar{t}\right)$ requires selecting $g(\cdot|\lambda)$
and estimating $\lambda$.[Under publication bias, the local FDR can be bounded without estimating
$\lambda$ by using a kind of worst case scenario (Chen, 2021).]

An important caveat of this analysis is that it relies on $\pi_{F}$
being not too close to 1.0. For $\pi_{F}=0.9$ or higher, the $2\varepsilon$
bound becomes a $9\varepsilon$ bound (or higher), and Proposition
<ref> may have little bite. Intuitively, if almost all
factors are false, then even the extreme right tail of the distribution
will be largely determined by false factors. However, we will see
that empirical evidence suggests $\pi_{F}\ge0.9$ is unlikely cross-sectional
return predictors (Section <ref>).

§.§ Conservative FDR Controls

Some FDR methods avoid estimation of $\pi_{F}$ by effectively assuming
$\pi_{F}\ge1.0$. As a result, these conservative methods assume
that t-hurdles need to be raised, and thus cannot answer the question
of whether t-hurdles need to be raised.

These conservative algorithms include Benjamini and Yekutieli, 2001's
(BY's) Theorem 1.3, which is emphasized in HLZ, and is popular in
the finance literature. As shown by Storey, 2002, BY's
Theorem 1.3 is equivalent to replacing the constraint in Equation
(<ref>) with an estimator:[This is seen Equation (13) of Storey, 2002. See also
the Equivalence Theorem of Efron and Tibshirani, 2002. Strictly speaking,
these papers show this equivalence for the Benjamini and Hochberg, 1995
algorithm, however, the BY algorithm simply modifies the Benjamini and Hochberg, 1995
algorithm with a constant factor.]

\begin{align}
h_{BY1.3}\left(5\%\right) & \equiv\min_{h\in\mathbb{R}_{+}}\left\{ h:\widehat{\FDR}\left(h\right)\le5\%\right\} \label{eq:theory-equiv}
\end{align}

\begin{align}
\widehat{\FDR}\left(h\right) & \equiv\frac{\Pr\left(|t_{i}|>h\bigl|F_{i}\right)}{\widehat{\Pr}\left(|t_{i}|>h\right)}\left(\sum_{i=1}^{N}\frac{1}{i}\right)\label{eq:prop-equiv-FDRhat}\\
\widehat{\Pr}\left(|t_{i}|>h\right) & \equiv N^{-1}\sum_{i=1}^{N}I\left(|t_{i}|>h\right).\label{eq:prop-equiv-Prhat}
\end{align}

Compare Equation (<ref>) to the result of applying
Bayes rule to $\Pr\left(F_{i}\left||t_{i}|>h\right.\right)$

\begin{align}
\Pr\left(F_{i}\left||t_{i}|>h\right.\right) & =\frac{\Pr\left(|t_{i}|>h\left|F_{i}\right.\right)}{\Pr\left(|t_{i}|>h\right)}\pi_{F}\label{eq:theory-fdr-bayes}
\end{align}

Thus, $h_{BY1.3}\left(5\%\right)$ effectively assumes that $\pi_{F}=\sum_{i=1}^{N}\frac{1}{i}$.
In HLZ's case, where $N\approx300$, we have $\pi_{F}=6.3$. So this
algorithm implies that, not only is $\pi_{F}$ is large, but it is
six times larger than 1.0. Since $\pi_{F}$>1.0 is physically
impossible, it is hard to justify why such a severe penalty is necessary.
Indeed, $h_{BY1.3}\left(5\%\right)$ is described as “a severe penalty”
and “not really necessary” (Efron, 2012). Even the
original Benjamini and Yekutieli, 2001 paper says this algorithm is
“very often unneeded, and yields too conservative of a procedure.”

An implication of this conservatism is that BY's Theorem 1.3 always
implies that t-hurdles need to be raised. Assuming $\pi_{F}>1$, only
one case of Proposition <ref> is possible:

If $\widehat{\FDR}\left(\cdot\right)$
is strictly decreasing, then $h_{BY1.3}\left(5\%\right)>1.96$.

The proof is in Appendix <ref>.

Similarly, the seminal Benjamini and Hochberg, 1995 (BH95) algorithm
is equivalent to replacing $\left(\sum_{i=1}^{N}\frac{1}{i}\right)$
with 1.0 in Equation (<ref>). Thus, BH95 is
isomorphic to using $\pi_{F}=1.0$ in Equation (<ref>),
and Proposition <ref> implies that BH95 raises t-hurdles
in all cases except for the extreme case in which $\sum_{i=1}^{N}I\left(|t_{i}|>1.96\right)=N$.

The benefit of imposing $\pi_{F}=1.0$ is simplicity. The BH95 hurdle
can be computed for an arbitrary dataset without any additional specifications.
The cost, however, is that BH95 discards all information in the data
that can inform us about $\pi_{F}$. So while BH95 is an excellent
first pass for controlling for multiple testing, it cannot tell us
which case of Proposition <ref> holds, and cannot tell
us whether t-hurdles must be raised.[Unlike the finance literature, the statistics literature tends to
favor the BH95 algorithm over BY's Theorem 1.3 (Benjamini, 2020).
HLZ argue for using BY's Theorem 1.3, as it controls the FDR under
arbitrary dependence. But BH95 controls the FDR under weak dependence
(Storey et al., 2004) and simulations suggest FDR control under
arbitrary dependence if the tests in question use z-statistics (Reiner-Benaim, 2007).]

In their first draft, Benjamini and Hochberg (1995) did not
assume that t-stat hurdles should be raised. As described in Benjamini, 2010,
the 1989 draft recommended a graphical method for estimating $\pi_{F}$.
After many years of rejections, the authors shifted to the conservative
formulation that became the seminal BH95 algorithm. The original algorithm,
along with the result that t-stat hurdles could possibly be lowered,
was eventually published in Benjamini and Hochberg, 2000, and many
statisticians went on to propose additional estimators for $\pi_{F}$
(Efron et al., 2001, Allison et al., 2002, Storey, 2002, Genovese and Wasserman, 2004, Benjamini et al., 2006,
among others).

§.§ False Negative Rates

A natural question is how publication bias affects identification
of false negative rates (FNRs), also known as the type II error rate.
This question can be analyzed using a Bayesian formulation of the

\begin{align}
\Fnr(h) & \equiv\Pr\left(T_{i}\left||t_{i}|\le h\right.\right)\label{eq:theory-fnr}
\end{align}

(see Efron, 2012 Chapter 4.3). $\Fnr(h)$ is the probability
a factor is true, given that the factor fails to meet the hurdle $h$.
One can alternatively define the FNR using factor counts as in Equation
(<ref>) (see Genovese and Wasserman, 2002) but
these definitions are equivalent under weak dependence.

Applying Bayes rule shows how the FNR runs into identification issues:

\begin{align}
\Fnr(h) & =1-\frac{\Pr\left(|t_{i}|\le h\left|F_{i}\right.\right)}{\Pr\left(|t_{i}|\le h\right)}\pi_{F}.\label{eq:theory-fnr-intuition}
\end{align}

In the presence of publication bias, $\pi_{F}$ may be weakly identified
(Proposition <ref>). Moreover, the very definition of publication
bias (Equations (<ref>)-(<ref>))
implies that the data that bears on $\Pr\left(|t_{i}|\le h\right)$
is limited. Intuitively, to estimate a false negative rate one needs
to know the number of insignificant t-stats, and publication bias
means that insignificant t-stats are poorly observed.

An alternative way to address the FNR is to use the FDR control that
comes closest to achieving the constraint in Equation (<ref>).
Achieving this constraint would typically involve forming a point
estimate of $\pi_{F}$ (rather than assuming an upper bound), as is
pursued in this paper.

§ EMPIRICAL ESTIMATES OF T-STAT HURDLES

The theoretical results illustrate weak and strong identification,
but a precise description of these issues requires empirical estimates
of sampling uncertainty. This section provides one such estimate by
bootstrapping estimates using a dataset constructed from asset pricing

§.§ Data

My data begins with 207 published cross-sectional stock return predictors
from the Chen and Zimmermann, 2022 (CZ) dataset (March 2022 release).
I focus on their original predictor portfolios, which consists of
long-short portfolios constructed following the procedures in the
original studies.

This dataset has several advantages as a setting for studying publication
bias. CZ show that the replicated t-stats closely match the originals,
which rules out coding errors, fraud, or other more nefarious sources
of bias. CZ also show that the pairwise correlations cluster around
zero (see also <cit.>), suggesting
that the weak dependence assumptions underlying standard estimation
methods are valid (Wooldridge, 1994), and that modeling
the correlations will add relatively little to estimation efficiency.
Moreover, the monthly returns in this dataset can be used to account
for correlations in my bootstrapped standard errors. Chen, 2021
shows the distribution of t-stats is quite similar to the distribution
found in HLZ, which eases comparison. Last, this dataset is publicly
available at www.openassetpricing.com.

To simplify the baseline model, I drop predictors with in-sample t-stats
that fall below 1.96, leading to a sample of 183 predictors. The approach
of dropping t-stats $<$ 1.96 is also used in HLZ. Carefully modeling
the 24 predictors with smaller t-stats requires a relatively complicated
publication probability function and is likely to introduce more identification
problems (Copas, 1999). In the robustness section, I include
t-stats < 1.96 and find broadly similar results (Section <ref>).

Since all of CZ's factors are cross-sectional return predictors, I
refer to them as “predictors” in what follows.

§.§ Structural Model and Estimation

The structural model adds three assumptions to the model of Section
<ref>. The additional structure can be thought of
as restrictions that help identify the key model objects (e.g. $\pi_{F},$

The first is the unbiased t-stat $\mu_{i}$ is log-normal for true

\begin{align}
\mu_{i}|T_{i} & \sim\text{Log-normal}\left(\lambda_{\mu},\lambda_{\sigma}\right).\label{eq:emp-t-true}
\end{align}

where $\lambda_{\mu}$ and $\lambda_{\sigma}$ are mean and standard
deviation of $\left(\log\mu_{i}\right)|T_{i}$, respectively. This
form is arguably the simplest assumption one can make about $\mu_{i}|T_{i}$
while ensuring that true predictors have an expected return that is
distinct from 0. Section <ref> shows that alternative
distributional assumptions have little effect on the results.

The second assumption is a functional form for the publication probability
(Equation (<ref>)). I assume a staircase function

\begin{align}
\Pr\left(\pub_{i}\left||t_{i}|\right.\right) & =\begin{cases}
\eta\bar{s} & \hspace{1ex}\text{if }|t_{i}|\in(1.96,2.58]\\
\bar{s} & \hspace{1ex}\text{if }|t_{i}|>2.58.
\end{cases}\label{eq:emp-pub-prob}
\end{align}

where the t-stat cutoffs of 1.96 and 2.58 correspond to the traditional
5% and 1% significance cutoffs, respectively, and $\eta$ represents
a “haircut” for marginally-significant predictors. One can think
of this function as the simplest intuitive model of selective publication
for marginally-significant predictors. This functional form is also
assumed in HLZ. Section <ref> shows that a logistic
form (as in Chen and Zimmermann, 2020 and Harvey and Liu, 2021)
leads to similar results.

The last assumption is that $\eta$ lies within an interval obtained
from intuition

\begin{align}
\eta & \in[1/3,2/3].\label{eq:emp-haircut-support}
\end{align}

Equation (<ref>) says that at least some marginally-significant
predictors are missing ($\eta\le2/3$), but at least a significant
minority are reported ($\eta\ge1/3$). HLZ uses the more restrictive
assumption that $\eta=1/2$, though they also examine $\eta=1/3$.
Section <ref> shows that relaxing this assumption
or restricting this assumption has little effect on the main results.

I also assume $\pi_{F}\in[0.01,0.99]$ for technical reasons. I use
numerical integration to compute the log-likelihoods and multiple
testing statistics, and I find that these methods can become poorly
behaved for $\pi_{F}$ that is extremely close to 0 or 1.0. One can
increase the range of $\pi_{F}$ with more complicated numerical methods
but this complication is unlikely to affect the main results.

I estimate $\theta\equiv\left(\pi_{F},\lambda_{\mu},\lambda_{\sigma}\right)$
with quasi-maximum likelihood (QML). I choose $\hat{\theta}$ to maximize
the mean log marginal likelihood of $|t_{i}|$ conditional on $i$
being published. I do not estimate $\bar{s}$, as it is not identified
(it is cancelled out in the conditional likelihood). Under technical
conditions, this QML estimator is consistent (<cit.>).[The key technical assumption is that the log marginal likelihood of
a single observation satisfies the uniform weak law of large numbers.] Intuitively, the model implies that the mean derivatives of the marginal
likelihoods are zero when evaluated at the true parameters, and this
implication can be used as moment conditions to estimate the model.

I confirm that QML is consistent under pairwise correlations as high
as 0.90 in Appendix <ref>. Indeed, in simulated samples
of only 200 t-stats, I find QML is essentially unbiased for $\pi_{F}$
and produces standard errors of around 0.10 to 0.15. These simulations
assume no publication bias and demonstrate that the large standard
errors I find for $\pi_{F}$ are not due to QML.

I measure estimation uncertainty using two bootstraps. My baseline
bootstrap is simple and non-parametric: I draw a full set of t-stats
from the empirical data with replacement and re-run QML. This bootstrap
is transparent and also ensures that the bootstrapped data displays
a similar selection bias and distributional properties as the original
dataset. Moreover, cross-sectional predictor correlations are typically
close to zero (McLean and Pontiff, 2016, Chen and Zimmermann, 2022). These
mild correlations suggest that this simple bootstrap will lead to
similar results as one that accounts for correlations.

For robustness, I also examine a semi-parametric bootstrap that carefully
accounts for correlations. In short, this second approach combines
a cluster bootstrap with a parametric bootstrap. The cluster bootstrap
ensures that the samples closely mimic the correlation structure in
the data while the parametric bootstrap ensures that the samples mimic
the distribution of observed t-stats. Details on the semi-parametric
bootstrap are found in Appendix <ref>

§.§ Parameter Estimates and Intuition

Table <ref> shows the resulting parameter estimates.
The point estimate finds that essentially all factors are true ($\hat{\pi}_{F}=0.01$),
but the 90% confidence interval is huge, ranging from 0.01 to about
0.90. Even the 50% C.I. is quite large, ranging from 0.01 to about
0.70. There results are consistent with Proposition <ref>
and suggest that t-hurdles are weakly identified.

[Table <ref> about here]

Huge uncertainty about $\pi_{f}$ obtains regardless of whether I
use the simple non-parametric bootstrap (shown in the table with no
parentheses) or the semi-parametric cluster bootstrap (shown with
parentheses). For all parameters in Table <ref>, accounting
for correlations has almost no effect on estimation uncertainty. This
result is intuitive, as the typical predictor correlation is close
to zero (Chen and Zimmermann, 2022, see also Figure <ref>
in the Appendix).

In contrast to $\pi_{F}$, the parameters that govern true factors
($\lambda_{\mu}$ and $\lambda_{\sigma}$) are strongly identified.
Table <ref> converts these parameters into the expected
unbiased t-stat $\E\left(\mu_{i}|T_{i}\right)$ and standard deviation
of the unbiased t-stat $\SD\left(\mu_{i}|T_{i}\right)$ for ease of
interpretation.[The lognormal distribution implies

\begin{align}
\E\left(\mu_{i}|T_{i}\right) & =\exp\left(\lambda_{\mu}+\sigma_{\mu}^{2}/2\right)\\
\text{SD}(\SE_{i}) & =\sqrt{\left[\exp(\lambda_{\mu}^{2})-1\right]\exp(2\lambda_{\mu}+\lambda_{\sigma}^{2})}.
\end{align}

] The bootstrapped estimates imply that $\E\left(\mu_{i}|T_{i}\right)$
is between 2.0 and 3.8 while $\SD\left(\mu_{i}|T_{i}\right)$ is between
1.9 and 2.7, with 90% confidence. These results are consistent with
Proposition <ref>.

Figure <ref> provides the intuition behind these results.
It compares the predictions of the point estimate with an alternative
model that also uses QML, but the alternative model fixes $\pi_{F}$
at 2/3. Panel (a) shows that both models fit data to the right of
$2.58$ very well. Recall that this is the subsample of the data that
is well-observed (Equation (<ref>)). Thus, even a
slight perturbation to the observed data can move the point estimate
from $\pi_{F}=0.01$ to $\pi_{F}=2/3$.

[Figure <ref> about here]

The models differ in their predictions about t-stats between 1.96
and 2.58, suggesting the data in this region can help identify $\pi_{F}$.
However, many t-stats in this region may be missing, and it is difficult
know a-priori just how many. Both QML estimates restrict uncertainty
by assuming that the fraction missing is between 1/3 and 2/3 (Equation
(<ref>)). Nevertheless, this restriction is
unable to identify $\pi_{F}$. Section <ref> shows
that even assuming the missing fraction is 0.5 cannot identify $\pi_{F}$

Thus, the key to identification is the distribution of t-stats below
1.96. As seen in Figure <ref>, the two models have very different
predictions for this region. The data tell us very little about which
is the right story, as there are only 24 observations in this region.
Indeed, a close read of Chen and Zimmermann, 2022 shows that even
these 24 observations are poorly defined, as it is unclear if these
observations should even be called “predictors.”

This ambiguity leads me to drop t-stats below 1.96 in the estimation
(though they are still shown in Figure <ref>). Nevertheless,
Section <ref> shows that taking on this additional
data still implies substantial uncertainty. Intuitively, only 7 t-stats
below 1.5 are observed, and it is very hard to say how representative
these t-stats are. This very small sample problem suggests that adding
the standard errors of predictor mean returns to the estimation, which
in principle can non-parametrically identify the publication probability
(Andrews and Kasy, 2019), will still result in weak identification
of $\pi_{F}$. Indeed, previous versions of this paper did include
these standard errors in the estimation and found similar results.[Code for the previous versions can be found at <https://github.com/chenandrewy/t-hurdles>.]

Panel (b) of Figure <ref> illustrates strong identification.
It examines the distribution of true predictors' t-stats $(t_{i}|T_{i})$
in the point estimate and the alternative model. Despite very different
implications about $\pi_{F}$, both models imply similar distributions
of $t_{i}|T_{i}$. Intuitively, a highly dispersed $t_{i}|T_{i}$
is required to fit the long right tail in observed t-stats, regardless
of $\pi_{F}$. Since $t_{i}|T_{i}$ is just $\mu_{i}|T_{i}$ plus
standard normal noise, this result implies that the parameters governing
$\mu_{i}|T_{i}$ are strongly identified, leading to the relatively
narrow confidence bounds in Table <ref>.

Returning to Table <ref>, the probability of publishing
a marginally-significant t-stat ($\eta$) tends to be larger than
50% in its bootstrapped distribution, implying that the asset pricing
literature does not strongly discriminate against predictors with
t-stats between 1.96 and 2.58. This result is intuitive, as the asset
pricing literature contains relatively little discussion of marginal
significance, and instead focuses on economic mechanisms.

Table <ref> shows that the bootstrapped $\eta$ often
bumps against the upper bound of $2/3$ imposed by the model (Equation
(<ref>)). This restriction was imposed to
exclude the possibility that weak identification is due to an excessively
general model. Section <ref> shows that relaxing
these bounds does not change the main results.

As very similar estimates obtain using either the non-parametric or
semi-parametric bootstrap, for the remainder of the paper I discuss
only the non-parametric bootstrap.

§.§ Weak Identification of t-stat Hurdles for Cross-Sectional Predictors

With bootstrapped parameters in hand, I can finally quantify weak
and strong identification of multiple testing statistics.

Figure <ref> plots the bootstrapped distribution
of t-hurdles (Equation (<ref>)). Panel (a) examines
the more common FDR upper bound of 5%, which results in a highly
dispersed t-stat hurdle of between 0 and 3.0. In other words, the
data say little about how to adjust t-hurdles for multiple testing.
Indeed, the classical 5% counterpart of 1.96 lies right in the middle
of the distribution, implying substantial uncertainty about whether
the hurdle should be raised or lowered.

[Figure <ref> about here]

The estimates imply a large mode in t-hurdles close to 0, reflective
of the large mode in bootstrapped estimates of $\pi_{F}$ close to
zero that is implicit in Table <ref>. This mode is
intuitive given the shape of the empirical distribution (Figure <ref>):
This unimodal shape provides little evidence in support of a bimodal
distribution in $\mu_{i}$, and thus simply assuming $\mu_{i}$ is
lognormal ($\pi_{F}=0$) provides a strong fit to the data. This result
is also consistent with Proposition <ref> and Figure
<ref>, which show that the observed distribution of the full
model is approximately the same as a model with $\pi_{F}=0$.

Some readers may find $\pi_{F}=0$ to be implausible, but Figure <ref>
suggests that this mode is immaterial. Excluding the mode at 0, the
distribution of t-hurdle estimates is still highly dispersed, with
values ranging from 1.0 to 3.0. Additionally, Section <ref>
shows that restricting $\pi_{F}\ge0.20$ still results in substantial
uncertainty about the right t-hurdle.

Panel (b) shows that weak identification is also seen if one selects
the unusually restrictive FDR $\le$ 1% t-hurdle. The resulting distribution
of t-hurdles is, once again, highly dispersed, ranging from 0 to 3.5.
The classical 1% hurdle of 2.58 lies well-within the interior of
this distribution.

§.§ Strong Identification of Shrinkage and the Local FDR

Figure <ref> shows that the data are informative
about shrinkage and the local FDR for published predictors. Panel
(a) examines shrinkage by evaluating Equation (<ref>)
using bootstrapped data and then taking the mean across published
t-stats within each bootstrap. The panel shows the distribution of
this mean across bootstraps. The vast majority of the distribution
lies below 26%, which is the upper bound on statistical bias estimated
in McLean and Pontiff, 2016's out-of-sample tests. This result implies
that published sample mean returns are at least 74% due to true expected
returns rather than publication bias, multiple testing, or other related
statistical effects.

[Figure <ref> about here]

The mode of the shrinkage distribution is close to 12%, which is
the point estimate found by Chen and Zimmermann, 2020. This similarity
obtains despite the fact that I assume a bi-modal distribution in
t-stats, providing a counterexample to Harvey and Liu, 2021's
claim that Chen and Zimmermann's estimate is sensitive to their unimodal
assumption. Indeed, Section <ref> shows Chen and Zimmermann's
estimate is robust to 10 alternative models, all of which use bi-modal

Panel (b) shows similar results for the mean local FDR. Panel (b)
computes the mean local FDR by evaluating Equation (<ref>)
at bootstrapped parameter values and taking the mean across published
t-stats within the bootstrap. Quantitatively similar to Panel (a),
Panel (b) finds that published factors are at least 75% true, with
high confidence. The figure displays a large mode near 0, due to the
unimodal shape of the empirical data (see Section <ref>),
but even if this mode were excluded the estimates still imply that
the published predictors are largely true with high confidence.

For comparison, Figure <ref> also plots the
mean local FDR for published predictors implied by HLZ's baseline
estimates. HLZ's estimate implies that only 6% of published findings
are false, close to the middle of the bootstrapped distribution. This
result is surprising, given HLZ's verbal statement that “most claimed
research findings in financial economics are likely false.” However,
this estimate is natural given their numerical estimates.

To understand HLZ's numerical estimates, note that the mean local
FDR for published factors is approximately the Bayesian expression

\begin{align}
\E\left[\text{fdr}\left(t_{i}\right)|\pub_{i}\right] & \approx\Pr\left(F_{i}\left||t_{i}|>1.96\right.\right)\nonumber \\
& =\frac{\Pr\left(|t_{i}|>1.96|F_{i}\right)}{\Pr\left(|t_{i}|>1.96\right)}\pi_{F}.\label{eq:emp-Efdr-approx}
\end{align}

HLZ find that $\pi_{F}=0.444$ (Table 2) and that of 1,378 total t-stats
(also Table 2), 353 t-stats that exceed 1.96 (page 28). Plugging these
numbers into Equation (<ref>) leads to

\begin{align*}
\E\left[\text{fdr}\left(t_{i}\right)|\pub_{i}\right] & \approx\frac{5\%}{353/1378}0.444\\
& =8.6\%.
\end{align*}

This approximation is a touch higher than the 6% found by applying
Equation (<ref>) to their model simulation because
the simulation assumes a more stringent publication criteria (only
half of t-stats between 1.96 and 2.58 are published). Chen, 2021
finds that even HLZ's conservative estimates imply the FDR is rather

§ ROBUSTNESS

Figures <ref> and <ref> depend
on functional form and distributional assumptions. However, Propositions
<ref> and <ref> do not, suggesting that similar
results may obtain under a wide variety of assumptions.

This subsection confirms this robustness for 10 alternative sets of
assumptions. Table <ref> shows the 5th and 95th percentile
of multiple testing statistics, computed across 500 bootstrapped estimates
for each set of assumptions. Most of the 90% confidence intervals
for t-hurdles span 0 and 2.9, implying substantial uncertainty about
the proper correction to the classical hurdle of 1.96. In contrast,
all specifications imply shrinkage is at most 29%, and the local
FDR is at most 25%, with 95% confidence.

[Table <ref> about here]

Indeed, previous versions of this paper found similar results for
additional estimation and modeling assumptions, including estimations
that account for standard errors and t-stats separately, and estimations
that modeled the full distribution of correlations. Additional robustness
can be found using the github code ([a github site]) or using
the github code for the previous draft ([a github site])

The remainder of this section briefly describes each set of assumptions
and their motivations.

§.§ Robustness to Using Small Chen-Zimmerman t-stats

Section <ref> simplifies modeling by dropping the 24
t-stats that fall below 1.96. Specifications (2) and (3) include these
small t-stats—as long as they exceed 0.50. The three t-stats
that fall below 0.50 are 0.06, 0.09, and 0.40. Including these tiny
t-stats leads to shrinkage estimates that divide by tiny numbers (Equation
(<ref>)), leading to extreme outliers that drive
the means.

To accommodate the additional 21 t-stats that fall below 1.96, I add
two additional steps to the staircase publication probability:

\begin{align}
\Pr\left(\pub_{i}\right) & =\begin{cases}
\eta_{a}\bar{s} & \hspace{1ex}\text{if }|t_{i}|\le1.50\\
\eta_{b}\bar{s} & \:\text{if }|t_{i}|\in(1.50,1.96]\\
\eta_{c}\bar{s} & \hspace{1ex}\text{if }|t_{i}|\in(1.96,2.58]\\
\bar{s} & \hspace{1ex}\text{if }|t_{i}|>2.58.
\end{cases}\label{eq:emp-pub-prob-1}
\end{align}

where the additional cutoff of 1.50 is chosen to match the cutoff
used in McLean and Pontiff, 2016. 1.50 is also close to the two-sided
10% hurdle of 1.64.

Specifications (2) and (3) differ in how they restrict the parameters
$\eta_{a}$, $\eta_{b}$, and $\eta_{c}$. Specification (2) makes
no restrictions. Specification (3) assumes $\eta_{c}\in[1/3,2/3]$
(same as in the baseline), and imposes that the other probabilities
are smaller than 1/3.

§.§ Robustness to Assuming $\pi_{F}$ is at Least $X\%$

The baseline estimates show a large mode at $\hat{\pi}_{F}=0.01$
(Table <ref>), leading to large modes in Figures <ref>
and <ref>. Some readers may find such a small
$\pi_{F}$ implausible and would like to restrict $\pi_{F}$ to some
minimum value.

In Table <ref>, Specifications (4) and (5) use the restrictions
$\pi_{F}\ge0.10$ and $\pi_{F}\ge0.20$, respectively.

§.§ Robustness to Alternative Publication Probability Functions

The baseline model restricts $\eta\in[1/3,2/3]$ to show that weak
identification is not due to an excessively general model. Specification
(6) restricts even further to $\eta=0.5$, which is HLZ's baseline

Specification (7) examines a more general $\eta\in[1/3,1.0]$. This
choice is motivated by the fact that the baseline bootstrap bumps
up against the upper bound of $\eta=$2/3 (Table <ref>
), suggesting that $\eta\ge2/3$ is preferred by the data.

Specification (8) assumes a logistic form for the publication probability

\begin{align*}
\Pr\left(\pub_{i}\right) & =\frac{1}{1+\exp\left(-\eta_{b}(|t_{i}|-\eta_{a})\right)},
\end{align*}

which is the same function used in Chen and Zimmermann, 2020 (see
also Cochrane, 2005, Harvey and Liu, 2021).

§.§ Robustness to Alternative Distributional Assumptions

The baseline model assumes $\mu_{i}|T_{i}$ is log-normal, as this
is the simplest assumption one can make that ensures that true predictors
have a positive mean return that is distinct from 0.

In Table <ref>, Specification (9) assumes $\mu_{i}|T_{i}$
is exponential following HLZ and Specification (10) assumes $\mu_{i}|T_{i}$
is a scaled t-distribution, similar to Andrews and Kasy, 2019
and Chen and Zimmermann, 2020. Specification (11) assumes $\mu_{i}|T_{i}$
is a mixture of two normals, which can be motivated by the idea that
there are two distinct types of true predictors, each with a bell-shaped

Specifications (10) and (11) often imply that $\mu_{i}|T_{i}$ can
be either positive or negative, which implies that the sign of $t_{i}$
should be accounted for in the shrinkage estimation (Equation (<ref>)).
For these two rows, the shrinkage shown thus uses

\begin{align}
\text{shrinkage}_{\text{signed}}\left(\bar{t}\right) & \equiv\frac{\bar{t}-E\left(\mu_{i}\left|t_{i}=\bar{t}\right.\right)}{\bar{t}}.\label{eq:theory-bias-1}
\end{align}

which implicitly assumes that the journals use theory to find the
appropriate sign of predictability. Most papers in the Chen-Zimmermann
dataset motivate their signs using theory.

§ CONCLUSION

Motivated by concerns about scientific credibility, a series of papers
call for statistical hurdles to be raised. I show these calls may
be difficult to justify empirically. Publication bias means that the
key parameter in these arguments may be weakly identified. As a result,
empirical data may say little about whether hurdles should be raised,
stay the same, or even be lowered.

More positively, I find that other multiple testing statistics can
be strongly identified. In particular, shrinkage and the local FDR
for published factors may be able to be pinned down without knowledge
of t-stats near zero. For the cross-sectional return predictability
literature, these statistics imply that published findings are at
least 75% true, with high confidence, across many model specifications.

What “at least 75% true” says about the proper $t$-hurdle is
subjective, but at least some readers will argue that a literature
that is mostly true is a healthy one. Still, others may argue that
a literature should strive for trust, and even for achieving “no
questions asked” from their readers.

A caveat about these empirical results is that they are based on the
field of cross-sectional predictability. While other fields of economics
also feature very large t-stats (Brodeur et al., 2016), a rigorous
analysis is required to pin down the FDR. Moreover, cross-sectional
predictability is known for its clean data, standardized methods,
and strong replicability (Chen and Zimmermann, 2022, 40).
These features allay concerns about coding errors and the improper
calculation of t-stats that may remain if shrinkage and FDR estimates
were applied to other fields.

Regardless, my results demonstrate that the debate about scientific
credibility should take multiple testing statistics more seriously.
These statistics do not simply say that statistical hurdles
should be raised, or that a large share of findings are false. Identification
is important, and some multiple testing statistics are more strongly
identified than others. Most important, my paper shows that the debate
should focus on the more strongly identified statistics.

§ A MODEL WITH SUPPORTING EVIDENCE

This section presents a simple model for analyzing the bias the comes
from ignoring supporting evidence.

As in Section <ref>, the t-stat $t_{i}$ provides
a noisy signal of the latent effect $\mu_{i}$, but for simplicity
assume $\mu_{i}$ is normal:

\begin{align*}
t_{i}|\mu_{i} & \sim\text{Normal}\left(\mu_{i},1\right)\\
\mu_{i} & \sim\text{Normal}\left(0,\lambda\right)
\end{align*}

where $\lambda$ in this setting is the variance of $\mu_{i}$. This
is the key parameter that determines shrinkage.

Add to the Section <ref> model supporting evidence,
represented by $z_{i}$:

\begin{align*}
z_{i}|\mu_{i} & \sim\text{Normal}\left(\mu_{i,}1\right)\\
\text{\Cov}\left(t_{i},z_{i}|\mu_{i}\right) & =\rho
\end{align*}

where $\rho$ allows for correlation between the noise components
of $t_{i}$ and $z_{i}$. One can think of $z_{i}$ as representing
aspects of a research paper that are not captured by the primary t-stat.

Now, extend Equation (<ref>) to allow publication
to depend on $z_{i}$:

\begin{align*}
\Pr\left(\pub_{i}\left|t_{i},z_{i}\right.\right) & =\begin{cases}
\bar{s} & \left(t_{i}>2.5\right)\cap\left(z_{i}>z_{\min}\right)\\
0 & \text{otherwise}
\end{cases}
\end{align*}

where the strict cutoffs and the lack of absolute values on $t_{i}$
are chosen for simplicity. The key parameter in this expression is
$z_{\min}$, which captures the quality of supporting evidence required
by the literature.

Let $\hat{\lambda}$ be the following estimate that ignores the supporting

\begin{align}
\widehat{\lambda} & =\arg\min_{\lambda}\left(N_{\pub}^{-1}\sum_{i:\pub_{i}}t_{i}-E\left(t_{i}|t_{i}>2.5;\lambda\right)\right).\label{eq:app-lambdahat}
\end{align}

This expression formalizes the misspecification error. The correct
model moment is $E\left(t_{i}|t_{i}>2.5,z_{i}>z_{\min};\lambda\right)$,
not $E\left(t_{i}|t_{i}>2.5;\lambda\right)$.

In general, $\widehat{\lambda}$ is upward biased. This happens because
the target moment $E\left(t_{i}|t_{i}>2.5,z_{i}>z_{\min};\lambda\right)$
is increasing in $z_{\min}$, due to the fact that both $t_{i}$ and
$z_{i}$ are positively related to $\mu_{i}$. Figure <ref>
visualizes this bias. An upward-biased $\hat{\lambda}$ implies a
downward biased shrinkage.

Supporting Evidence and Bias in $\hat{\lambda}$
Model is described in Appendix <ref>,
with $\rho=0$. Grey is $E\left(t_{i}|t_{i}>2.5;\lambda\right)$ as
a function of $\lambda$ and red is $E\left(t_{i}|t_{i}>2.5,z_{i}>1;\lambda\right)$.
The “Data” line and the choice of $z_{\min}=1.0$ as “actual”
are arbitrary and selected for illustration.

There is, however, is an offsetting bias. Ignoring supporting evidence
in the empirical Bayes estimate of $\mu_{i}$ leads to

\begin{align}
E\left(\mu_{i}|t_{i};\lambda\right) & <E\left(\mu_{i}|t_{i},z_{i};\lambda\right)\quad\text{for }z_{i}>z_{\min}\label{eq:app-muhat}
\end{align}

since $z_{i}$ is positively related to $\mu_{i}$ and published data
tend to have high $z_{i}$. A downward bias in estimates of $\mu_{i}$
leads to an upward bias in shrinkage.

Which bias dominates is not immediately clear. In a simulation analysis,
I find the net bias can go either way, though the magnitudes of the
bias are generally moderate. I examine many simulations with values
of $\lambda$ ranging from 0.5 to 12, $\rho$ from 0 to 0.6, and $z_{\min}$
from 0 to 2.

Net Bias in Shrinkage under Various Parameter Values
I simulate the model in Appendix
<ref> under various parameter values, estimate
$\hat{\lambda}$ ignoring the supporting evidence, and then estimate

Figure <ref> presents the simulation results.
In each simulation, I estimate $\hat{\lambda}$ using the potentially
misspecified Equation (<ref>), compute the potentially
mis-specified empirical Bayes estimate of $\mu_{i}$ (Equation (<ref>)),
and then compute shrinkage by taking the mean estimated $\mu_{i}$
for published $i$ and dividing by the mean $t_{i}$ for published
$i$. Actual shrinkage uses the actual $\mu_{i}$. The actual shrinkage
can be both above and below the estimated shrinkage. But generally
the errors are small relative to the estimated shrinkage.

The errors can become noticeable for $z_{\min}=2.0$, suggesting that
even higher levels of $z_{\min}$ can result in substantial bias.
One can argue, however, that the standards for supporting evidence
should not be as strict as the standards for the main evidence, and
thus $z_{\min}<2.0$ is reasonable.

§ PROOFS

§.§ Proof of Proposition <ref>

In this proof, I consider a more general version of the proposition,
in which the FDR-based and traditional hurdles use different critical
levels. For this more general proof, we need to define a general classical

\begin{align}
h_{0}\left(\alpha_{0}\right) & \equiv\min_{h\in\mathbb{R}_{+}}\left\{ h:\Pr\left(|t_{i}|>h\bigl|F_{i}\right)\le\alpha_{0}\right\} ,\label{eq:theory-single-test}
\end{align}

where $\alpha_{0}$ is the classical significance level. Similarly,
generalize Equation (<ref>) as

\begin{align*}
h_{\FDR}^{\ast}\left(\alpha\right) & \equiv\min_{h\in\mathbb{R}_{+}}\left\{ h:\Pr\left(F_{i}||t_{i}|>h\right)\le\alpha\right\} .
\end{align*}

Since $\Pr\left(F_{i}||t_{i}|>h\right)$ is strictly decreasing, the
solution to the minimization (<ref>) solves $\Pr\left(F_{i}||t_{i}|>h_{\FDR}^{\ast}\left(\alpha\right)\right)=\alpha$.
Rewrite this using Bayes rule:

\begin{align}
\alpha & =\Pr\left(F_{i}||t_{i}|>h_{\FDR}^{\ast}\left(\alpha\right)\right)=\frac{\Pr\left(|t_{i}|>h_{\FDR}^{\ast}\left(\alpha\right)\bigl|F_{i}\right)}{\Pr\left(|t_{i}|>h_{\FDR}^{\ast}\left(\alpha\right)\right)}\pi_{F}\label{eq:app-proof1-01}
\end{align}

Then Equation (<ref>) is solved with Pr $\left(|t_{i}|>h_{0}\left(\alpha_{0}\right)|F_{i}\right)=\alpha_{0}$.
Multiplying both sides by Equation (<ref>) we have

\begin{align}
\Pr\left(|t_{i}|>h_{0}\left(\alpha_{0}\right)|F_{i}\right) & =\Pr\left(|t_{i}|>h_{\FDR}^{\ast}\left(\alpha\right)\bigl|F_{i}\right)\left[\frac{\alpha_{0}}{\alpha}\frac{\pi_{F}}{\Pr\left(|t_{i}|>h_{\FDR}^{\ast}\left(\alpha\right)\right)}\right]\label{eq:app-main-1}
\end{align}

From here, we have three cases for the square bracket term, but the
first case is symmetric with the rest.

For the first case, suppose

\begin{align*}
1 & >\left[\frac{\alpha}{\alpha_{0}}\frac{\Pr\left(|t_{i}|>h_{\FDR}^{\ast}\left(\alpha\right)\right)}{\pi_{F}}\right]
\end{align*}

Note that the above expression is equivalent to the condition in Equation
(<ref>), if we use $\alpha=\alpha_{0}=5\%$.
Multiply both sides of this inequality with (<ref>)
to get the equivalent expression

\begin{align*}
Pr\left(|t_{i}|>h_{0}\left(\alpha_{0}\right)|F_{i}\right) & >\Pr\left(|t_{i}|>h_{\FDR}^{\ast}\left(\alpha\right)\bigl|F_{i}\right).
\end{align*}

Since $Pr\left(|t_{i}|>h|F_{i}\right)$ strictly decreases in $h$,
the above equation is equivalent to

\begin{align*}
h_{0}\left(\alpha_{0}\right) & <h_{\FDR}^{\ast}\left(\alpha\right).
\end{align*}

Thus we have shown the equivalence in Equation (<ref>).

For the other two cases, simply replace $>$ with either $=$ or $<$
in the previous paragraph.

§.§ Proof of Proposition <ref>

For ease of notation, define

\begin{align*}
\delta\left(X\right) & \equiv\frac{\Pr\left(X\bigl|F_{i}\right)}{\Pr\left(X\bigl|T_{i}\right)}\\
L\left(\pi\right) & \equiv\frac{\pi}{(1-\pi)}
\end{align*}

where $X$ is some event. Note that for $X=|t_{i}|\subset[\tgood,\infty)$
we have by assumption (<ref>)

\begin{align*}
\delta\left(X\right) & <\varepsilon.
\end{align*}

Use this notation to rewrite Equation (<ref>):

\begin{align}
\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right) & =\frac{\Pr\left(|t_{i}|\in(\tgood,\bar{t}]\bigl|T_{i}\right)}{\Pr\left(|t_{i}|>\tgood\bigl|T_{i}\right)}\left[\frac{1+L\left(\pi_{F}\right)\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)}{1+L\left(\pi_{F}\right)\delta\left(|t_{i}|>\tgood\right)}\right]\label{eq:app-key}
\end{align}

where all I did was factor $(1-\pi_{F})$ and the probabilities that
condition on $T_{i}$. In the above equation, the front term doesn't
depend on $\pi_{F}$. Thus, if we take the ratio of this expression
evaluated at two different values of $\pi_{F}$, the first term cancels

\begin{align}
\frac{\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right)|_{\pi_{F}=\pi'}}{\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right)|_{\pi_{F}=\pi}} & =\left[\frac{1+L\left(\pi'\right)\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)}{1+L\left(\pi'\right)\delta\left(|t_{i}|>\tgood\right)}\right]\left[\frac{1+L\left(\pi\right)\delta\left(|t_{i}|>\tgood\right)}{1+L\left(\pi\right)\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)}\right]\nonumber \\
& =\left[\frac{1+L\left(\pi'\right)\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)}{1+L\left(\pi\right)\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)}\right]\left[\frac{1+L\left(\pi\right)\delta\left(|t_{i}|>\tgood\right)}{1+L\left(\pi'\right)\delta\left(|t_{i}|>\tgood\right)}\right]\label{eq:app-prop2-01}\\
& =1+\left[L\left(\pi'\right)-L\left(\pi\right)\right]\left[\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)-\delta\left(|t_{i}|>\tgood\right)\right]+O\left(\varepsilon^{2}\right)\label{eq:app-prop2-02}
\end{align}

where the last line uses a Taylor expansion around $\left[\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right),\delta\left(|t_{i}|>\tgood\right)\right]=\left[0,0\right]$
and the fact $\delta\left((\tgood,\bar{t}]\right)<\varepsilon$ and
$\delta\left(|t_{i}|>\tgood\right)<\varepsilon$. The simplicity of
this Taylor expansion comes from the fact that

\begin{align*}
\left.\frac{d}{dx}\left(\frac{1+ax}{1+bx}\right)\right|_{x=0} & =\left.\frac{a}{bx+1}-\frac{b(ax+1)}{(bx+1)^{2}}\right|_{x=0}=a-b.
\end{align*}

So the partial derivatives of Equation (<ref>) simplify
dramatically once $\left[\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right),\delta\left(|t_{i}|>\tgood\right)\right]=\left[0,0\right]$
is plugged in.

Subtracting 1 form both sides, taking absolute values of (<ref>)
and applying the triangle inequality

\begin{align*}
\left|\frac{\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right)|_{\pi_{F}=\pi'}}{\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right)|_{\pi_{F}=\pi}}-1\right| & \le\left|L\left(\pi'\right)-L\left(\pi\right)\right|\left|\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)-\delta\left(|t_{i}|>\tgood\right)\right|+O\left(\varepsilon^{2}\right)\\
& \le\left|L\left(\pi'\right)-L\left(\pi\right)\right|\varepsilon+O\left(\varepsilon^{2}\right)\\
& \le\frac{\bar{\pi}}{1-\bar{\pi}}\varepsilon+O\left(\varepsilon^{2}\right).
\end{align*}

Recall $\bar{\pi}$ is the maximum value that $\pi$ and $\pi'$ can
take. The third line uses the fact that $L\left(\pi\right)$ is strictly

Finally, imposing the assumption that $\bar{\pi}=2/3$ finishes the
proof. Note that using instead $\bar{\pi}=1/2$ results in a factor
of 1 in front of $\varepsilon$, while $\bar{\pi}=9/10$ would result
in a factor of 9. $\bar{\pi}=2/3$ was chosen as an illustrative middle

§.§ Derivation of Shrinkage Expression (<ref>)

Since $i$ is used as a subscript throughout, I drop it for ease of
notation. I use the shorthand notation $p\left(\mu|t\right)$ to denote
the density of $\mu$ given $|t|$, $p\left(\mu|T\right)$ to denote
the density of $\mu$ given the factor is true, and similarly for
other random variables.

Write down the posterior of $\mu|t$ and rewrite the denominator using
the law of total probability

\begin{align*}
p\left(\mu|t\right) & =\frac{p\left(t|\mu\right)p\left(\mu|T\right)\pi_{T}+p\left(t|\mu\right)p\left(\mu|F\right)\pi_{F}}{p\left(t\right)}\\
& =\frac{p\left(t|\mu\right)p\left(\mu|T\right)\pi_{T}+p\left(t|\mu\right)p\left(\mu|F\right)\pi_{F}}{p\left(t|T\right)\pi_{T}+p\left(t|F\right)\pi_{F}}
\end{align*}

Then factor out $p\left(t|T\right)\pi_{T}$

\begin{align*}
p\left(\mu|t\right) & =\left[p\left(t|\mu\right)\frac{p\left(\mu|T\right)}{p\left(t|T\right)}+p\left(t|\mu\right)\frac{p\left(\mu|F\right)\pi_{F}}{p\left(t|T\right)\pi_{T}}\right]\\
& \quad\times\left\{ \frac{1}{1+\frac{p\left(t|F\right)\pi_{F}}{p\left(t|T\right)\pi_{T}}.}\right\} \\
& =\left[p\left(\mu|t,T\right)+p\left(t|\mu\right)\frac{p\left(\mu|F\right)\pi_{F}}{p\left(t|T\right)\pi_{T}}\right]\left\{ \frac{1}{1+\Delta\left(t,\pi_{F},\lambda\right)}\right\}
\end{align*}

\begin{align*}
\Delta\left(t,\pi_{F},\lambda\right) & \equiv\frac{p\left(t|F\right)\pi_{F}}{p\left(t|T\right)\pi_{T}},
\end{align*}

since $p\left(t|\mu,T\right)=p\left(t|\mu\right)$.

Let $h\left(\mu\right)$ be a function s.t. $h\left(0\right)=0$.
Then the posterior expected value of $h\left(\mu\right)$ is

\begin{align}
E\left(h\left(\mu\right)|t,\pi_{F},\lambda\right) & =\left\{ E\left(h\left(\mu\right)|t,T,\lambda\right)+\int d\mu h\left(\mu\right)p\left(t|\mu\right)\frac{p\left(\mu|F\right)\pi_{F}}{p\left(t|T\right)\pi_{T}}\right\} \left\{ \frac{1}{1+\Delta\left(t,\pi_{F},\lambda\right)}\right\} \nonumber \\
& =\left\{ E\left(h\left(\mu\right)|t,T,\lambda\right)+h\left(0\right)p\left(t|0\right)\frac{\pi_{F}}{p\left(t|T\right)\pi_{T}}\right\} \left\{ \frac{1}{1+\Delta\left(t,\pi_{F},\lambda\right)}\right\} \nonumber \\
& =E\left(h\left(\mu\right)|t,T,\lambda\right)\left\{ \frac{1}{1+\Delta\left(t,\pi_{F},\lambda\right)}\right\} \nonumber \\
& =E\left(h\left(\mu\right)|t,\pi_{F}=0,\lambda\right)\left\{ \frac{1}{1+\Delta\left(t,\pi_{F},\lambda\right)}\right\} \label{eq:app-Ehmu}
\end{align}

where the second line comes from the fact that $p\left(\mu|F\right)=\delta\left(\mu\right)$,
where $\delta\left(\cdot\right)$ is the Dirac delta, the third uses
$h\left(0\right)=0$, and the last line uses the fact that the model
with $\pi_{F}=0$ assumes that all factors are true. Choosing $h\left(\mu\right)=\mu$
leads to Equation (<ref>).

A similar argument can be made for the local FDR. The local FDR can
be rewritten as

\begin{align*}
\fdr\left(t\right) & =\frac{p\left(t|F\right)\pi_{F}}{p\left(t|T\right)\pi_{T}+p\left(t|F\right)\pi_{F}}\\
& \le\frac{p\left(t|F\right)}{p\left(t|T\right)\pi_{T}+p\left(t|F\right)\pi_{F}}\\
& =\frac{p\left(t|F\right)}{p\left(t|T\right)}\left[\frac{1}{1+\frac{p\left(t|F\right)\pi_{F}}{p\left(t|T\right)}}\right].
\end{align*}

$\frac{p\left(t|F\right)}{p\left(t|T\right)}$ is, in a way, a bound
on the local fdr assuming that all factors are true. The square brackets
is the correction term, which is just $\Delta\left(t,\pi_{F},\lambda\right)\times\pi_{T}$.

§.§ Proof of Proposition <ref>

The proposition uses the same assumptions as Proposition <ref>,
so I can borrow this expression from its proof

\begin{align*}
\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right) & =\frac{\Pr\left(|t_{i}|\in(\tgood,\bar{t}]\bigl|T_{i}\right)}{\Pr\left(|t_{i}|>\tgood\bigl|T_{i}\right)}\left[\frac{1+L\left(\pi_{F}\right)\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)}{1+L\left(\pi_{F}\right)\delta\left(|t_{i}|>\tgood\right)}\right]
\end{align*}

\begin{align*}
\delta\left(X\right) & \equiv\frac{\Pr\left(X\bigl|F_{i}\right)}{\Pr\left(X\bigl|T_{i}\right)}\\
L\left(\pi\right) & \equiv\frac{\pi}{(1-\pi)}
\end{align*}

and $X$ is some event. Rearranging,

\begin{align*}
\frac{\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right)}{\Pr\left(|t_{i}|\in(\tgood,\bar{t}]\bigl|T_{i},t_{i}|>\tgood\right)} & =\left[\frac{1+L\left(\pi_{F}\right)\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)}{1+L\left(\pi_{F}\right)\delta\left(|t_{i}|>\tgood\right)}\right]\\
& =1+L\left(\pi_{F}\right)\left[\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)-\delta\left(|t_{i}|>\tgood\right)\right]+O\left(\varepsilon^{2}\right)
\end{align*}

\begin{align*}
\frac{\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right)}{\Pr\left(|t_{i}|\in(\tgood,\bar{t}]\bigl|T_{i},t_{i}|>\tgood\right)}-1 & =L\left(\pi_{F}\right)\left[\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)-\delta\left(|t_{i}|>\tgood\right)\right]+O\left(\varepsilon^{2}\right)
\end{align*}

Taking absolute values and manipulating

\begin{align*}
\left|\frac{\Pr\left(|t_{i}|\le\bar{t}||t_{i}|>\tgood\right)}{\Pr\left(|t_{i}|\in(\tgood,\bar{t}]\bigl|T_{i},t_{i}|>\tgood\right)}-1\right| & \le\left|L\left(\pi_{F}\right)\right|\left|\left[\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)-\delta\left(|t_{i}|>\tgood\right)\right]\right|+O\left(\varepsilon^{2}\right)\\
& \le\left|L\left(\pi_{F}\right)\right|\varepsilon+O\left(\varepsilon^{2}\right)\\
& \le\frac{\bar{\pi}}{1-\bar{\pi}}\varepsilon+O\left(\varepsilon^{2}\right)
\end{align*}

where $\bar{\pi}$ is the maximum value under consideration for $\pi_{F}$.
The 2nd line comes from the fact that $\delta\left(|t_{i}|>\tgood\right)>0$
and $\delta\left(|t_{i}|\in(\tgood,\bar{t}]\right)<\varepsilon$ and
the third line uses the fact that $L\left(\pi_{F}\right)$ is strictly
increasing. Plugging in $\bar{\pi}=2/3$ completes the proof.

§.§ Proof of Corollary <ref>

Comparing the definition of $\widehat{\FDR}\left(h\right)$ (Equation
(<ref>)) and Equation (<ref>)
we see that $h_{BY}$ is isomorphic to replacing $\pi_{F}/\Pr\left(|t_{i}|>h\right)$
with $\left(\sum_{i=1}^{N}\frac{1}{i}\right)/\widehat{\Pr}\left(|t_{i}|>h\right)$
in the hurdle definition (Equation (<ref>)). Thus,
Proposition <ref> implies

\begin{align*}
h_{BY}>1.96\text{ if and only if }\left(\sum_{i=1}^{N}\frac{1}{i}\right)>\widehat{\Pr}\left(|t_{i}|>1.96\right) & .
\end{align*}

But since $\sum_{i=1}^{N}\frac{1}{i}>1$ and $\widehat{\Pr}\left(|t_{i}|>1.96\right)\equiv\frac{\sum_{i=1}^{N}I\left(|t_{i}|>1.96\right)}{N}<1$,
we have $h_{BY}>1.96$.

§ ESTIMATION DETAILS

§.§ QML on Simulated Data with Correlations

This section shows that QML works well even if factors have pairwise
correlations as high as 0.9 assuming no publication bias.

The simulation model begin with the model in Section <ref>
and specifies correlations to be AR1 across factor indexes:

\begin{align*}
\varepsilon_{i} & \equiv t_{i}-\mu_{i}\\
\varepsilon_{i} & \sim N\left(\rho\varepsilon_{i-1},\sqrt{1-\rho^{2}}\right)
\end{align*}

This specification ensures Equation (<ref>) holds
and provides a simple way to model correlated factors.

I assume all factors are observed. One can think of this exercise
as demonstrating that the weak identification I find is due to publication
bias rather than QML.

I fix $\lambda_{\mu}$ and $\lambda_{\sigma}$ using the point estimate
from Table <ref>. I vary $\pi_{F}$ and $\rho$ as
shown in Table <ref>. For each of the shown parameter
values, I simulate a sample of 200 factors, apply QML, and repeat
100 times. The table shows the mean $\hat{\pi}_{F}$ across the 100
simulations as well as the standard deviation across simulations.

The table shows that QML is approximately unbiased for $\pi_{F}$
and has moderate standard errors of around 0.10 for $\rho\le0.5$.
Standard errors increase to about 0.15 for the $\rho=$0.90 cases,
but they are still half as large as the standard error of about 0.33
in the empirical estimates. Overall Table <ref> shows
that weak identification of $\pi_{F}$ is not due to QML.

QML on Simulated Data with Correlations

I simulate factors with correlated t-stats that have AR1
coefficient $\rho$ and apply QML. Results show means and standard
deviations across 100 simulations using each parameter set.

1c2[2]*$\pi_F$ truth           3cCross-predictor AR1 coefficient $\rho$
1c0.1     1c0.5     1c0.9
1-13-51c0.10     1lmean $\hat{\pi}_F$     1c          0.13      1c          0.13      1c                                                                          0.12
1lsd $\hat{\pi}_F$     1c          0.10      1c          0.10      1c                                                                          0.12
1c0.50     1lmean $\hat{\pi}_F$     1c          0.48      1c          0.49      1c                                                                          0.49
1lsd $\hat{\pi}_F$     1c          0.15      1c          0.14      1c                                                                          0.17
1c0.90     1lmean $\hat{\pi}_F$     1c          0.90      1c          0.89      1c                                                                          0.87
1lsd $\hat{\pi}_F$     1c          0.07      1c          0.08      1c                                                                          0.15

§.§ Details on the Semi-Parametric Bootstrap

The cluster-bootstrap uses post-1963 panel returns from the Chen-Zimmermann
data. Unlike the main results which use only in-sample data, I include
data outside of the original sample periods for the cluster bootstrap.
This sample selection helps keep the panel balanced and is conservative
because correlations generally increase post-publication (McLean and Pontiff, 2016).
Using this sample, I construct residuals by subtracting the mean return
at the predictor level.

I draw residuals with a cluster bootstrap. I draw 5,000 predictors
(with replacement) and 350 months (with replacement) and construct
a dataset of the corresponding residuals. 350 months corresponds to
the average sample size in the original papers. By construction, the
distribution of correlations in the bootstrapped samples is similar
that of the original data, as seen in Figure <ref>.

I then construct sampling noise $\varepsilon_{i}$ from these residuals
by taking the mean, dividing by the standard deviation, and multiplying
by the square root of the number of observations. I feed these $\varepsilon_{i}$
into the point estimate. That is, I simulate 5,000 $\mu_{i}$ using
the point estimate, calculate $t_{i}=\mu_{i}+\varepsilon_{i}$, and
then obtain published $t_{i}$ by applying Equation (<ref>),
once again using the point estimate. Provided that the model is well-specified,
this semi-parametric bootstrap provides valid inference (Efron and Tibshirani, 1994,
Chapter 6.5).

This relatively complicated bootstrap is required for capturing correlations
because a simple cluster bootstrap would lead to t-statistics that
are too small compared to the empirical data, as it fails to simulate
publication bias. Similarly, a cluster bootstrap that simply applies
a truncation would lead to t-statistics that are too large compared
to empirical data, as this bootstrap effectively assumes that no publication
bias in the empirical data.

Correlations in Cluster-Bootstrapped Return Residuals.
I bootstrap from Chen-Zimmermann panel returns
while ensuring that returns in the same month are always drawn together.
The plot compares the distribution of correlations in the bootstrap
against the correlations in the raw data.

[Allison et al., 2002]
authorAllison, D.B., authorGadbury, G.L.,
authorHeo, M., authorFernández, J.R.,
authorLee, C.K., authorProlla, T.A.,
authorWeindruch, R., year2002.
titleA mixture model approach for the analysis of
microarray gene expression data.
journalComputational Statistics & Data Analysis
volume39, pages1–20.

[Andrews and Kasy, 2019]
authorAndrews, I., authorKasy, M.,
titleIdentification of and correction for publication
journalAmerican Economic Review volume109,

[Barberis, 2018]
authorBarberis, N., year2018.
titlePsychology-based models of asset prices and trading
volume, in: booktitleHandbook of Behavioral Economics:
Applications and Foundations 1. publisherElsevier.
volume volume1, pp. pages79–175.

[Benjamin et al., 2018]
authorBenjamin, D.J., authorBerger, J.O.,
authorJohannesson, M., authorNosek, B.A.,
authorWagenmakers, E.J., authorBerk, R.,
authorBollen, K.A., authorBrembs, B.,
authorBrown, L., authorCamerer, C., et al.,
titleRedefine statistical significance.
journalNature Human Behaviour volume2,

[Benjamini, 2010]
authorBenjamini, Y., year2010.
titleDiscovering the false discovery rate.
journalJournal of the Royal Statistical Society: series B
(statistical methodology) volume72, pages405–416.

[Benjamini, 2020]
authorBenjamini, Y., year2020.
titleSelective inference: The silent killer of
journalHarvard Data Science Review volume2.

[Benjamini and Hochberg, 1995]
authorBenjamini, Y., authorHochberg, Y.,
titleControlling the false discovery rate: a practical and
powerful approach to multiple testing.
journalJournal of the royal statistical society. Series B
(Methodological) , pages289–300.

[Benjamini and Hochberg, 2000]
authorBenjamini, Y., authorHochberg, Y.,
titleOn the adaptive control of the false discovery rate
in multiple testing with independent statistics.
journalJournal of educational and Behavioral Statistics
volume25, pages60–83.

[Benjamini et al., 2006]
authorBenjamini, Y., authorKrieger, A.M.,
authorYekutieli, D., year2006.
titleAdaptive linear step-up procedures that control the
false discovery rate.
journalBiometrika volume93,

[Benjamini and Yekutieli, 2001]
authorBenjamini, Y., authorYekutieli, D.,
titleThe control of the false discovery rate in multiple
testing under dependency.
journalAnnals of statistics ,

[Bessembinder et al., 2021]
authorBessembinder, H., authorBurt, A.,
authorHrdlicka, C.M., year2021.
titleTime series variation in the factor zoo.
journalAaron Paul and Hrdlicka, Christopher M., Time
Series Variation in the Factor Zoo (December 22, 2021) .

[Brodeur et al., 2016]
authorBrodeur, A., authorLé, M.,
authorSangnier, M., authorZylberberg, Y.,
titleStar wars: The empirics strike back.
journalAmerican Economic Journal: Applied Economics
volume8, pages1–32.

[Canova and Sala, 2009]
authorCanova, F., authorSala, L., year2009.
titleBack to square one: Identification issues in dsge
journalJournal of Monetary Economics
volume56, pages431–449.

[Chen, 2021]
authorChen, A.Y., year2021a.
titleThe limits of p-hacking: Some thought experiments.
journalThe Journal of Finance volume76,

[Chen, 2021]
authorChen, A.Y., year2021b.
titleMost claimed statistical findings in cross-sectional
return predictability are likely true.
journalAvailable at SSRN 3912915 .

[Chen and Velikov, 2023]
authorChen, A.Y., authorVelikov, M.,
titleZeroing in on the expected returns of anomalies.
journalJournal of Financial and Quantitative Analysis
volume58, pages968–1004.

[Chen and Zimmermann, 2018]
authorChen, A.Y., authorZimmermann, T.,
titlePublication bias and the cross-section of stock

[Chen and Zimmermann, 2020]
authorChen, A.Y., authorZimmermann, T.,
titlePublication bias and the cross-section of stock
journalThe Review of Asset Pricing Studies
volume10, pages249–289.

[Chen and Zimmermann, 2022]
authorChen, A.Y., authorZimmermann, T.,
titleOpen source cross sectional asset pricing.
journalCritical Finance Review .

[Chen and Zimmermann, 2023]
authorChen, A.Y., authorZimmermann, T.,
titlePublication bias in asset pricing research.

[Chinco et al., 2021]
authorChinco, A., authorNeuhierl, A.,
authorWeber, M., year2021.
titleEstimating the anomaly base rate.
journalJournal of financial economics
volume140, pages101–126.

[Chordia et al., 2020]
authorChordia, T., authorGoyal, A.,
authorSaretto, A., year2020.
titleAnomalies and false rejections.
journalThe Review of Financial Studies
volume33, pages2134–2179.

[Cochrane, 2005]
authorCochrane, J.H., year2005.
titleThe risk and return of venture capital.
journalJournal of financial economics
volume75, pages3–52.

[Cochrane, 2017]
authorCochrane, J.H., year2017.
journalReview of Finance volume21,

[Copas, 1999]
authorCopas, J., year1999.
titleWhat works?: selectivity models and meta-analysis.
journalJournal of the Royal Statistical Society: Series A
(Statistics in Society) volume162, pages95–109.

[Efron, 2008]
authorEfron, B., year2008.
titleMicroarrays, empirical bayes and the two-groups
journalStatistical science , pages1–22.

[Efron, 2012]
authorEfron, B., year2012.
titleLarge-scale inference: empirical Bayes methods for
estimation, testing, and prediction. volume volume1.
publisherCambridge University Press.

[Efron and Tibshirani, 2002]
authorEfron, B., authorTibshirani, R.,
titleEmpirical bayes methods and false discovery rates for
journalGenetic epidemiology volume23,

[Efron et al., 2001]
authorEfron, B., authorTibshirani, R.,
authorStorey, J.D., authorTusher, V.,
titleEmpirical bayes analysis of a microarray experiment.
journalJournal of the American statistical association
volume96, pages1151–1160.

[Efron and Tibshirani, 1994]
authorEfron, B., authorTibshirani, R.J.,
titleAn introduction to the bootstrap.
publisherCRC press.

[Fisher, 1925]
authorFisher, R., year1925.
titleStatistical methods for research workers. .

[Genovese and Wasserman, 2002]
authorGenovese, C., authorWasserman, L.,
titleOperating characteristics and extensions of the false
discovery rate procedure.
journalJournal of the Royal Statistical Society Series B:
Statistical Methodology volume64, pages499–517.

[Genovese and Wasserman, 2004]
authorGenovese, C., authorWasserman, L.,
titleA stochastic process approach to false discovery
journalThe Annals of Statistics volume32,

[Harvey and Liu, 2021]
authorHarvey, C.R., authorLiu, Y.,
titleUncovering the iceberg from its tip: A model of
publication bias and p-hacking.
journalAvailable at SSRN 3865813 .

[Harvey et al., 2016]
authorHarvey, C.R., authorLiu, Y.,
authorZhu, H., year2016.
title... and the cross-section of expected returns.
journalThe Review of Financial Studies
volume29, pages5–68.

[Hedges, 1992]
authorHedges, L.V., year1992.
titleModeling publication selection effects in
journalStatistical Science , pages246–255.

[Hedges and Vevea, 2005]
authorHedges, L.V., authorVevea, J.,
titleSelection method approaches.
journalPublication Bias in Meta-Analysis: Prevention,
Assessment and Adjustments , pages145–174.

[Ioannidis, 2005]
authorIoannidis, J.P., year2005.
titleWhy most published research findings are false.
journalPLoS medicine volume2,

[Jacobs and Müller, 2020]
authorJacobs, H., authorMüller, S.,
titleAnomalies across the globe: Once public, no longer
journalJournal of Financial Economics
volume135, pages213–230.

[40]
authorJensen, T.I., authorKelly, B.T.,
authorPedersen, L.H., yearForthcoming.
titleIs there a replication crisis in finance?
journalJournal of Finance .

[Kasy and Spiess, 2023]
authorKasy, M., authorSpiess, J., year2023.
titleOptimal pre-analysis plans: Statistical decisions
subject to implementability .

[Liu et al., 2020]
authorLiu, L., authorMoon, H.R.,
authorSchorfheide, F., year2020.
titleForecasting with dynamic panel data models.
journalEconometrica volume88,

[McCrary et al., 2016]
authorMcCrary, J., authorChristensen, G.,
authorFanelli, D., year2016.
titleConservative tests under satisficing models of
publication bias.
journalPloS one volume11,

[McLean and Pontiff, 2016]
authorMcLean, R.D., authorPontiff, J.,
titleDoes academic research destroy stock return
journalThe Journal of Finance volume71,

[McShane et al., 2016]
authorMcShane, B.B., authorBöckenholt, U.,
authorHansen, K.T., year2016.
titleAdjusting for publication bias in meta-analysis: An
evaluation of selection methods and some cautionary notes.
journalPerspectives on Psychological Science
volume11, pages730–749.

[McShane et al., 2019]
authorMcShane, B.B., authorGal, D.,
authorGelman, A., authorRobert, C.,
authorTackett, J.L., year2019.
titleAbandon statistical significance.
journalThe American Statistician volume73,

[Olken, 2015]
authorOlken, B.A., year2015.
titlePromises and perils of pre-analysis plans.
journalJournal of Economic Perspectives
volume29, pages61–80.

[Reiner-Benaim, 2007]
authorReiner-Benaim, A., year2007.
titleFdr control by the bh procedure for two-sided
correlated tests with implications to gene expression data analysis.
journalBiometrical Journal volume49,

[Storey, 2002]
authorStorey, J.D., year2002.
titleA direct approach to false discovery rates.
journalJournal of the Royal Statistical Society: Series B
(Statistical Methodology) volume64, pages479–498.

[Storey et al., 2004]
authorStorey, J.D., authorTaylor, J.E.,
authorSiegmund, D., year2004.
titleStrong control, conservative point estimation and
simultaneous conservative consistency of false discovery rates: a unified
journalJournal of the Royal Statistical Society: Series B
(Statistical Methodology) volume66, pages187–205.

[Wooldridge, 1994]
authorWooldridge, J.M., year1994.
titleEstimation and inference for dependent processes.
journalHandbook of econometrics volume4,

[Yan and Zheng, 2017]
authorYan, X.S., authorZheng, L., year2017.
titleFundamental analysis and the cross-section of stock
returns: A data-mining approach.
journalThe Review of Financial Studies
volume30, pages1382–1423.

Weak Identification of t-Hurdles in Harvey, Liu, and Zhu (2016).
HLZ baseline is the baseline estimate in Harvey, Liu, and Zhu's Table
5 (2nd row), which implies that t-hurdles should be raised to 2.27
to ensure FDR $\le$ 5%. The alternative model uses the same model
but changes the share of false factors ($\pi_{F}$) to zero, implying
that even a t-hurdle of 0 ensures FDR $\le5\%$. HLZ assume only t-stats
> 2.57 are well-observed and both models are re-scaled to have the
same density in this region. Despite having very different implications
for t-hurdles, the two models are essentially observationally equivalent,
consistent with Proposition <ref>.
Figure 2hlz demo

Model Fit and Intuition. I examine predictions of the point
estimate (blue) and an estimate that assumes $\pi_{F}=2/3$ (red).
$\pi_{F}$ is the fraction of all predictors that are false. Panel
(a) compares model predictions for all t-stats (lines) with published
t-stats in the CZ data (gray bars). Panel (b) decomposes model predictions
into true predictors (bars) and false predictors (the space between
lines and bars) $\pi_{F}=0.01$ and $\pi_{F}=2/3$ both fit the data
very well, consistent with Proposition <ref>. Consistent
with Proposition <ref>, $\pi_{F}=0.01$ and $\pi_{F}=2/3$
imply similar distributions for true predictors, suggesting $\lambda$
is strongly identified.
Figure 3point est

[Model Predictions vs Data]

[Decomposition of Model Predictions]

Weak Identification of t-hurdles for Cross-Sectional Predictors.
I repeatedly re-sample from the CZ dataset, re-estimate the model
of multiple testing with publication bias, and re-calculate the t-hurdle
(Equation (<ref>)) corresponding to an FDR of 5%
(Panel (a)) and 1% (Panel (b)). $\hat{\pi}_{F}$ is the estimated
probability of drawing a false factor. In both panels, the distribution
of bootstrapped multiple-testing-adjusted t-hurdles is highly dispersed,
with significant mass on both sides of the classical hurdles. Identification
is so weak that the data say little about whether t-stat hurdles should
be raised, stay the same, or even be lowered.
Figure 4boot hurdles

[FDR $\le$ 5%]

[FDR $\le$ 1%]

Strong Identification of the Shrinkage and the local FDR.
I repeatedly re-sample from the CZ dataset, re-estimate the model
of multiple testing with publication bias, and re-calculate shrinkage
(Equation (<ref>), Panel (a)) and the local
FDR (Equation (<ref>), Panel (b)). Shrinkage is
the extent to which t-stats are biased upward due to multiple testing.
The local FDR is the probability a given predictor has an expected
return of zero. Within each bootstrap, shrinkage and local FDR are
averaged across published predictors. The McLean and Pontiff (2016)
bound is taken from their abstract. The HLZ Baseline applies Equation
(<ref>) to simulated data based on their Table
5. Both multiple testing statistics imply that published predictability
is largely true with high confidence and are consistent with external
point estimates.
Figure 5pub stats

[Average Shrinkage for Published t-Stats]

[Local FDR for Published Predictors]

Estimates of a Structural Model of the Cross-Sectional Literature
Table 1parameter estimates

I estimate a structural model of multiple testing with publication
bias using quasi-maximum-likelihood (Section <ref>)
on long-short portfolios from <cit.> (Section
<ref>). The bootstrapped distribution is found by either
re-sampling the empirical t-stats and re-estimating (no parentheses)
or using a semi-parametric cluster-bootstrap that accounts for correlations
(parentheses). $\E\left(\mu_{i}|T_{i}\right)$ and $\SD\left(\mu_{i}|T_{i}\right)$
use textbook lognormal formulas to convert $\lambda_{\mu}$ and $\lambda_{\sigma}$
to moments. The share of false factors $\pi_{F}$ is weakly identified,
with huge confidence bounds, consistent with Proposition <ref>.
In contrast, $\E\left(\mu_{i}|T_{i}\right)$ and $\SD\left(\mu_{i}|T_{i}\right)$
are strongly identified. Accounting for correlations has very little
effect on the results.

Point           7cBootstrapped Distribution
5-11            Estimate           5           25     50     75           95
1-35-57-911-112lProbability false $\pi_F$     0.01           0.01            0.01      0.22      0.67            0.89
(0.01)           (0.01)     (0.31)     (0.65)           (0.85)
11lTrue factors' unbiased t-stats ($\mu_i|T_i$)
1l$\E(\mu_i|T_i)$     2.66           2.00            2.47      2.79      3.21            3.82
(2.00)           (2.46)     (2.80)     (3.13)           (3.56)
1l$\text{SD}(\mu_i|T_i)$     2.29           1.88            2.09      2.25      2.43            2.70
(1.89)           (2.13)     (2.26)     (2.44)           (2.67)

2l$\Pr(\pub_i)$ if marginal ($\eta$)     0.67           0.33            0.46      0.63      0.67            0.67
(0.33)           (0.46)     (0.60)     (0.67)           (0.67)

Table 2robustness

Each row shows the result of bootstrapping t-stat hurdles
(Equation (<ref>)), mean shrinkage for
published t-stats (Equation (<ref>)),
and mean local FDR for published t-stats (Equation (<ref>))
using a different set of modeling assumptions and/or data-inclusion
requirements. Baseline uses the assumptions found throughout Section
<ref>. “P05” and “P95” show 5th and 95th percentiles
across 500 bootstrapped estimations for the robustness tests and 1000
bootstrapped estimations for the baseline. Across 10 alternative modeling
assumptions, the t-hurdle is weakly identified while the shrinkage
and local FDR are strongly identified.

2ct-hurdle           2cShrinkage           2cFDR
2cFDR = 5%           2cfor Pubs           2cfor Pubs
3-46-79-10            P05     P95           P05     P95           P05     P95
3-46-79-10(1)     Baseline     0.0     3.0           8.9     27.9           0.1     22.6

10lIncluding $|t_i| <1.96$
(2)     3-step $\Pr(\pub_i)$     0.0     1.6           2.6     17.7           0.1     4.6
(3)     Restricted 3-step $\Pr(\pub_i)$     0.0     2.4           7.8     22.6           0.1     15.7

10lExcluding the $\pi_F$ very small
(4)     $\pi_F \ge 0.1$     0.9     3.0           9.0     27.8           0.6     23.3
(5)     $\pi_F \ge 0.2$     1.5     3.0           9.2     26.6           1.4     22.0

10lAlternative publication probability function
(6)     $\eta = 0.5$     0.0     2.9           10.9     26.1           0.1     20.7
(7)     $\eta \in [1/3, 1]$     0.0     3.0           6.9     26.8           0.1     22.6
(8)     logistic $\Pr(\pub_i)$     0.0     2.7           6.6     21.2           0.1     17.1

10lAlternative distributional assumptions
(9)     $\mu_i|T_i \sim \text{Exponential}$     0.0     2.9           11.3     28.7           0.1     19.7
(10)     $\mu_i|T_i \sim \text{Student's t}$     0.0     2.9           8.1     22.9           0.1     20.9
(11)     $\mu_i|T_i \sim \text{Mixture-Normal}$     0.0     3.0           7.2     24.1           0.1     24.8