# Simultaneous Hypothesis Testing Using Internal Negative Controls with An
Application to Proteomics

Zijun Gao and Qingyuan Zhao Statistical Laboratory, Department of Pure
Mathematics and Mathematical Statistics, University of Cambridge, UK. Email:
<EMAIL_ADDRESS>Laboratory, Department of Pure Mathematics and
Mathematical Statistics, University of Cambridge, UK. Email:
<EMAIL_ADDRESS>
###### Abstract

Negative control is a common technique in scientific investigations and
broadly refers to the situation where a null effect (“negative result”) is
expected. Motivated by a real proteomic dataset, we will present three
promising and closely connected methods of using negative controls to assist
simultaneous hypothesis testing. The first method uses negative controls to
construct a permutation p-value for every hypothesis under investigation, and
we give several sufficient conditions for such p-values to be valid and
positive regression dependent on the set (PRDS) of true nulls. The second
method uses negative controls to construct an estimate of the false discovery
rate (FDR), and we give a sufficient condition under which the step-up
procedure based on this estimate controls the FDR. The third method, derived
from an existing ad hoc algorithm for proteomic analysis, uses negative
controls to construct a nonparametric estimator of the local false discovery
rate. We conclude with some practical suggestions and connections to some
closely related methods that are propsed recently.

Keywords: multiple testing, negative control, empirical null, exchangeability,
empirical process, proteomics

##  1 Introduction

With the rapid development of high-throughput sequencing technologies, a
common task in modern statistical applications is to test a large number of
hypotheses simultaneously. A wealth of multiple testing procedures have been
proposed in the literature; most of them operate by combining p-values for the
individual hypotheses. Prominent examples include Bonferroni’s correction and
Simes’ test [51] for family-wise error rate (FWER) control, the Benjamini-
Hochberg (BH) procedure for false discovery rate (FDR) control [4], the closed
testing principle [40], and empirical Bayes methods for controlling the local
false discovery rate local-FDR [13].

In many practical situations, however, the validity of these p-values may be
jeopardized by various problems. For example, many high-throughput platforms
for biological experiments are subject to batch effects [32]. Other reasons
for invalid p-values include model misspecification and small sample sizes. As
a consequence, multiple testing procedures that combine these p-values may
fail to control the relevant statistical errors.

In this paper, we employ internal negative controls, of which the null
hypotheses are known to be true, to perform valid simultaneous hypothesis
testing. Motivated by a real proteomic analysis, we present three closely
related methods to use the negative controls. The first method uses, for the
test statistic of each hypothesis under investigation, its Rank Among the
Negative Control (RANC) as a permutation p-value; alternatively, this can be
understood as using negative controls to form a (nonparametric) empirical null
distribution. The second method uses the negative controls to give an estimate
of the false discovery rate of a set of rejected hypotheses. The third method,
derived from an existing ad hoc algorithm for proteomic analysis, uses
negative controls to construct a nonparametric estimator of local-FDR.

###  1.1 Motivating example

Our considerations are motivated by a real proteomic analysis shared by our
collaborating neuroscientists. [50] used proteomic profiling to identify
candidate cell membrane proteins that affect dendrite morphogenesis of
Purkinje cells. To focus on the statistical problem, in this example we will
consider developing cells ($15$ days postnatal) under the treatment condition
(labelled as HRP+H2O2 in the original paper) and the control condition (HRP
only). For each condition, the Purkinje cells of one mouse were extracted,
cultivated under the assigned condition, and prepared for mass spectrometry
that measures the abundance of each protein.

In total, $4,753$ proteins were detected and their subcellular localizations
were annotated in the UniProt database. [50] were interested in determining
proteins annotated with plasma membrane ($740$ in total) that show a higher
level of expression under the treatment condition. As there were no biological
repeats, the authors ranked the membrane proteins by the difference in their
expression under the treatment and control conditions, and then used internal
negative control proteins to determine a cutoff value.111[50] referred to
internal negative control proteins as “false positives” and proteins under
investigation as “true positives”. Here, the internal negative control
proteins are those annotated with nuclear, mitochondrial, or cytoplasmic but
not plasma membrane ($2067$ in total).222The numbers here are slightly
different from those reported in [50] due to an update of the UniProt
knowledgebase.

Figure 1 gives a side-by-side comparison of the test statistics (treatment-
minus-control expressions) of the proteins under investigation and of the
internal negative controls. The bulk of the proteins under investigation is
approximately normally distributed and resemble the internal negative control
proteins, but a number of proteins show a much larger difference compared to
the internal negative controls. Our scientific collaborators informed us that
such a pattern is commonly observed in similar experiments.

One immediate challenge with this dataset is that there is no biological
repeat for each condition. This precludes us from deriving a null distribution
for any single protein based on measurements of just that protein. One
possibile solution is to use an _empirical null_ distribution. Figure 2(a)
shows the histogram of the p-values obtained from a two-sample $t$-test with
the standard error estimated by pooling the proteins. It is apparent from this
plot that the standard error is likely to be overestimated, resulting in
unexpected concentration of p-values around $0.5$. In Section 7 we describe
more details of this and other ways to estimate the null distribution,
including the method suggested by [12].

Figure 1: Histogram of test statistics (abundance differences between the
treatment and control conditions of proteins under investigation (blue) and
internal negative control proteins (grey)). Superimposed are rejection
thresholds (red) with FDR level $q=0.2$ local-FDR level $q=\pi$ ($\pi$ is the
proportion of null hypotheses). See Figure 2 for how the thresholds are
determined.

###  1.2 Overview of the proposed methods

Our first method may be viewed as a nonparametric extension of the methods in
the last paragraph. Specifically, we propose to estimate the null distribution
using the empirical distribution of the negative control test statistics.
Figure 2(b) shows the histogram of what we call the Rank Among Negative
Controls (RANC) p-values obtained from this empirical null distribution.
Compared to Figure 2(a), a striking feature of Figure 2(b) is that the
negative control p-values are almost uniformly distributed over $[0,1]$. This
is expected from this choice of the empirical null; in fact, these p-values
are precisely equal to $\\{1/(m+1),\dotsc,m/(m+1),1\\}$ (assuming no ties).
Among the membrane proteins under investigation, the RANC p-values follow a
desirable pattern: their distribution has a spike near $0$ and is nearly
uniform elsewhere. One may apply familiar multiple testing methods such as the
BH procedure to RANC p-values; this will be justified in Section 3.

(a) Histogram of p-values calculated from a two-sample $t$-test.

(b) Histogram of p-values calculated using rank among negative controls (RANC,
method 1). The FDR rejection threshold can be determined by applying the
Benjamini-Hochberg procedure.

(c) Rejection threshold determined using estimated FDR (method 2).

(d) Rejection threshold determined by maximizing the difference of empirical
complementary CDFs (black curve, method 3).

Figure 2: An illustration of the methods proposed in this article. The red,
vertical lines correspond to rejection thresholds with FDR level $q=0.2$
(solid) and local FDR level $q=\pi$ (dashed, $\pi$ is the proportion of null
hypotheses).

Our second method is motivated by the empirical process perspective of FDR
control [16, 56]. Specifically, we propose to estimate the FDR above (or
below) any rejection threshold by the ratio of the proportions of test
statistics and negative controls that would be rejected by that threshold.
Figure 2(c) illustrates this proposal by showing the estimated FDR curve for
the motivating proteomic analysis, which can be used to select the rejection
threshold for any targeted FDR level $q$ ($q=0.2$ is illustrated in the
figure).

Our third method is motivated by a rule-of-thumb cut-off analysis in the
original study [50] proposed by [28]. This method computes the empirical (in
this case, complementary) cumulative distribution functions of the test
statistics and negative controls and simply chooses the maximizer of their
difference as the rejection threshold; see Figure 2(d) for an illustration.
Although this procedure seems rather ad hoc, the objective that it minimizes
can indeed be viewed as a nonparametric estimate of a weighted mis-
classification risk for multiple testing. Using this heuristic, we show that
the procedure in [28] can be extended to control the local-FDR at the
rejection threshold.

Shortly after releasing the first preprint of this paper, we discovered that
some recent works, stemming from different motivating applications, have
proposed some almost identical ideas above. Specifically, our first method
(RANC p-value) is the same as the conformal p-values to test for outliers
proposed by [3]. Our second method (empirical estimation of FDR) is
essentially the same as the so-called “semi-supervised” multiple testing
proposed by [41], and is closely related to the analysis of knockoff filters
in [63]; they are all essentially derived from the martingale argument in
[56]. Our third method (estimator of local-FDR) is closely related to the
method in [53] which assumes a known null distribution. Thus, our paper in
effect grounds these recent ideas in an exploding and diverging literature on
a simple and concrete biological application described next. Moreover, as we
are motivated by biological applications in which negative controls could
easily be misselected, the theoretical conditions developed below are
generally weaker than the i.i.d. or exchangeability conditions used before. We
discuss the connections and differences with these recent works and the
broader literature in more detail below.

###  1.3 Literature review

When most of the null hypotheses are true (i.e. ${n_{0}}/{n}$ is close to $1$)
and the theoretical model is correct, the bulk of the test statistics should
be close to the theoretical null distribution. However, this is often not the
case in practice. One example is the population stratification in genome-wide
association studies, where systematic ancestry difference may distort the null
distribution [47, 21]. Another example is batch effect or unwanted variation
due to the sequencing platform [32]. [12] argues that if such problems arise,
it may be sensible to estimate the null distribution using the bulk of the
empirical distribution of the test statistics. Other approaches based on more
sophisticated models of the data (such as latent factor models) attempt to
estimate the null distribution empirically by assuming the false hypotheses
are sparse [31, 61] or by using negative controls [15, 61].

Broadly speaking, negative control refers to the situation where a null effect
(“negative result”) is expected. For a given scientific experiment, there are
two types of negative controls: data on internal negative control units in the
same experiment and data from an external control experiment (e.g. the placebo
group in a clinical trial). The motivating example in Section 1.1 contains
both types of negative controls. Proteins annotated with nuclear,
mitochondrial, or cytoplasmic but not plasma membrane are used as internal
negative controls, because the different chemical treatments (HRP+H2O2 and HRP
only) are not expected to change their expression levels. The HRP (horseradish
peroxidase) only condition is used as an external control in [50] because both
HRP and H2O2 are needed to tag cell membrane proteins so that they can be
detected by mass spectrometry.

In this paper we focus on internal negative controls, which essentially
represent prior information about the non-existence of certain causal
connections. Such prior information may come from scientific contexts,
experimental techniques, and previous research studies. In high-throughput
sequencing applications, it is common that the scientific understanding of the
treatment suggests that certain measured units should not be affected by the
treatment. Examples include the non-membrane proteins in our motivating
example [34, 50] and housekeeping genes that are required for the maintenance
of basic cellular function and thus have a stable level of expression [15].
Researchers may also artificially create internal negative controls by adding
units that should not be affected by the treatment. Examples include exogenous
cells or molecules (often called spike-ins) that do not interact with the
treatment in biological experiments [36] and carefully designed questions in
surveys [37].

Although internal negative controls have long been used in scientific
investigations, their utility in statistical methodology has only been
explored since recently. In epidemiology, [37] employed internal negative
controls to detect and remove confounding. This was formally studied by [44]
and is often referred to as “proximal causal inference” in the literature
[59]. In microarray studies, [15] used internal negative control genes to
remove unwanted variation, and their method was analyzed and extended by [62].

There exists a small and scattered literature in computational biology and
biostatistics that attempts to use internal negative controls to control the
number of false discoveries. Some authors proposed to fit a parametric model
to the internal negative controls in order to estimate the null distribution
of the test statistics [45, 38, 52]. Naturally, such methods are sensitive to
the parametric specification. [46] proposed to estimate the local FDR using a
kernel density estimator based on the internal negative controls but did not
provide theoretical justifications of their method. Other authors suggested
heuristic approaches that use the internal negative controls to estimate the
FDR [67, 54].

Conformal inference seeks distribution-free uncertainty quantification of the
predictions from black-box machine learning models. This framework, originally
developed by Vladimir Vovk and collaborators [60], seeks to make predictive
inference based on how close new observations “conform” with the training
data. Conformal inference has received rapidly increasing attention recently;
see [1] for a recent review and some historic notes and [66] for an
interpretation from the viewpoint of classical randomization/permutation
tests. Our work is closely related to conformal inference. In particular, the
RANC p-value proposed here may be viewed as a permutation p-value and is
indeed identical to the conformal p-value for outlier detection in [3]. See
[39] and [35] for some extensions. Another closely related perspective is to
view multiple testing with negative controls as a semi-supervised problem
[41], a term borrowed from the machine learning literature. Motivated by
problems in astrostatistics, [41] proposed to use the same p-value using the
empirical null viewpoint. They analyzed the BH procedure applied to such
p-values using a martingale argument and provided some further optimality
results. To our knowledge, the connection between conformal inference,
negative controls, and empirical null in multiple testing has not been well
recognized yet.

Last but not least, we brieflly review the literature concerning local-FDR.
Local-FDR was first proposed in [13] as a Bayesian alternative to FDR. [57]
showed that controlling local-FDR is closely related to minimizing a
misclassification loss for the multiple testing procedure and developed a
procedure that controls the marginal FDR based on a given estimate of the
Local-FDR curve. Typically, Local-FDR is estimated by fitting parametric or
semiparametric density models to the data; see, for example, [11]. The recent
paper by [53] proposed to estimate local-FDR nonparametrically using the
Grenander’s estimator for monotone density functions and is most closely
related to our third method (and hence the ad hoc procedure in [28]). The main
difference is that [53] assumes the null density function is known, while we
use negative controls to estimate it empirically.

###  1.4 Outline and notation

In Section 2, we describe the mathematical setup of this paper and review some
terminologies in multiple testing. In Section 3, we consider the perspective
of using negative controls to form an empirical null distribution and propose
the RANC p-values. We give sufficient conditions under which the RANC p-values
are valid and satisfy a PRDS property, and discuss how the RANC p-values can
be combined using various multiple testing procedures. In Section 4, we
consider the perspective of using negative controls to form an empirical
estimator of the FDR. We show the step-up procedure with such as estimtor can
control the FDR even if some negative controls are selected incorrectly. In
Section 5, we develop a method that estimates the rejection threshold for a
given level of local-FDR and study its asymptotic properties. In Section 6, we
investigate the performance of RANC p-values using numerical simulations. In
Section 7, we come back to the motivating proteomic dataset and investigate
different choices of the null distribution. In Section 8, we conclude with
some further discussion.

We introduce some mathematical conventions used below. For $x\in\mathbb{R}$,
we use $\lfloor x\rfloor$ to denote the maximal integer smaller or equal to
$x$. For a set $\mathcal{A}$, we denote its cardinality by $|\mathcal{A}|$. We
abbreviate cumulative distribution function as CDF, probability density
function as PDF, almost everywhere as a.s., independently and identically
distributed as i.i.d.. We denote the uniform distribution on the interval
$[0,1]$ by $U[0,1]$, and the normal distribution with mean $\mu$ and variance
$\sigma^{2}$ by $\mathcal{N}(\mu,\sigma^{2})$. We use $\Phi(\cdot)$ to denote
the CDF of the standard normal distribution. For two random variables $X$,
$Y$, $X\stackrel{{\scriptstyle d}}{{=}}Y$ means $X$ and $Y$ follow the same
distribution. If $\mathbb{P}(X\leq t)\leq\mathbb{P}(Y\leq t)$ for all
$t\in\mathbb{R}$, we say $X$ stochastically dominates $Y$ and denote the
relationship by $X\gtrsim Y$ or $Y\lesssim X$. Given a collection of random
variables $X_{i},~{}i=1,\dotsc,n$, the $k$-th order statistic is defined as
its $k$-th smallest value and is denoted as $X_{(k)}$.

##  2 Setup and terminology

Suppose there are ${n}+m$ null hypotheses. The first ${n}$ hypotheses
${\mathcal{I}}=\\{1,\dotsc,{n}\\}$ are under investigation. Let
${\mathcal{I}_{0}}$ denote the set of true null hypotheses and
${n_{0}}=|{\mathcal{I}_{0}}|$; neither ${\mathcal{I}_{0}}$ nor ${n_{0}}$ is
known. The last $m$ hypotheses
${\mathcal{I}_{\text{nc}}}=\\{{n}+1,\dotsc,{n}+m\\}$ are known to be true and
we shall refer to them as the (internal) negative controls. In our motivating
example in Section 1.1, ${n}=740$ and $m=2067$. Each hypothesis ${H_{i}}$ is
associated with a test statistic ${T_{i}}$, and we assume that a small test
statistic provides evidence against that hypothesis. For example, ${T_{i}}$
can be a p-value for ${H_{i}}$ calculated under some possibly misspecified
statistical model. We are interested in identifying as many non-null
hypotheses in ${\mathcal{I}}\setminus{\mathcal{I}_{0}}$ as possible while
maintaining control of some multiple testing error.

Next, we briefly review some error rates that are commonly used for
simultaneous hypothesis testing. Given a set of hypotheses
$\\{{H_{i}}:i\in\mathcal{I}\\}$, let ${R}$ be the total number of rejections,
${V}$ be the number of incorrect rejections, and ${S}$ be the number of the
correct rejections; see also Table 1. FWER is the probability of making at
least one false discovery, FDP is the proportion of false discoveries among
all discoveries, and FDR is the expectation of FDP:

$\displaystyle\text{FWER}:=\mathbb{P}({V}\geq
1),\quad\text{FDP}:=\frac{{V}}{{R}\vee
1},\quad\text{and}\quad\text{FDR}:=\mathbb{E}[\text{FDP}]=\mathbb{E}\left[\frac{{V}}{{R}\vee
1}\right],$

where $R\vee 1$ is the maximum of $R$ and $1$. We say a multiple testing
procedure controls the FWER at level $\alpha$ if $\text{FWER}\leq\alpha$;
similarly, a procedure controls the FDR at $q$ if $\text{FDR}\leq q$. Note
that unlike the other two quantities, FDP is random and can only be controlled
in some probabilistic sense. For example, we say a procedure controls the tail
probability of FDP at $q$ at level $\alpha$ if
$\mathbb{P}(\text{FDP}>q)\leq\alpha$. Finally, it has been shown all
procedures that control FWER or (the tail probability of) FDP can be improved
by the closed testing principle that combines tests of intersection null
hypotheses [40, 18]; an intersection or global null hypothesis
$\cap_{i\in\mathcal{I}}{H_{i}}$ is said to be true if and only if all
individual hypotheses ${H_{i}}$, $i\in\mathcal{I}$ are true.

When discussing FWER, FDR, and FDP, we will assume that the CDF of the test
statistic ${T_{i}}$, denoted by ${F_{i}}$, is a continuous function for all
$i=1,\dotsc,{n}+m$, and there are no ties (with probability one). We discuss
tiebreakers in Section 8.

Table 1: Outcomes in multiple testing. | Not rejected | Rejected | Total
---|---|---|---
${H_{0,i}}$ true | ${U}$ | ${V}$ | ${n_{0}}$
${H_{0,i}}$ false | ${T}$ | ${S}$ | ${n_{1}}$
Total | ${n}-{R}$ | ${R}$ | ${n}$

We will also consider the local-FDR, a Bayesian alternative to traditional
multiple testing criteria. Suppose
$({H_{i}})_{i\in\mathcal{I}}\overset{\text{i.i.d.}}{\sim}\text{Bernoulli}(1-\pi)$,
and the test statistic ${T_{i}}$ follows the distribution

${T_{i}}\mid{H_{i}}=0\sim{F_{0}}\quad\text{and}\quad{T_{i}}\mid{H_{i}}=1\sim{F_{1}}.$

The marginal CDF of ${T_{i}}$ is thus given by the mixture
${F}={\pi}{F_{0}}+(1-{\pi}){F_{1}}$. When discussing local-FDR, we will assume
that the density functions of ${F_{0}}$, ${F_{1}}$, and ${F}$ exist and denote
them by ${f_{0}}$, ${f_{1}}$, and ${f}$, respectively. The local-FDR at $t$ is
simply the posterior probability $\Pr(H_{i}=0\mid T_{i}=t)$ of $H_{i}$ being a
true null given its test statistic is $T_{i}=t$ [13], i.e.

$\displaystyle\text{local-
FDR}(t)=\frac{{\pi}{f_{0}}(t)}{{\pi}{f_{0}}(t)+(1-{\pi}){f_{1}}(t)},$ (1)

We say a procedure controls local-FDR at level $q$ if $\text{local-
FDR}({T_{i}})\leq q$ for all rejected hypotheses ${H_{i}}$.

##  3 The empirical null perspective

###  3.1 Rank among negative controls

Our first method uses negative control statistics to form a nonparametric
estimator of the null distribution. Specifically, we define the RANC p-value
for ${H_{i}},~{}i\in{\mathcal{I}}$ as ${p_{i}}=\hat{F}({T_{i}})$, where

$\displaystyle\hat{F}(t)=\frac{1+\sum_{j\in{\mathcal{I}_{\text{nc}}}}\mathbbm{1}_{\\{{T_{j}}\leq
t\\}}}{1+m}$ (2)

is the empirical cumulative distribution function (CDF) of
$(-\infty,{T_{{n}+1}},\dotsc,{T_{{n}+m}})$. Here, we include a $-\infty$ in
the definition of $\hat{F}$ to ensure that the RANC p-value does not equal
zero with a positive probability. Another way to put this is that ${p_{i}}$ is
simply the normalized rank of ${T_{i}}$ among
$({T_{j}})_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}$:

$\displaystyle{p_{i}}:=\frac{1+\sum_{j\in{\mathcal{I}_{\text{nc}}}}\mathbbm{1}_{\\{{T_{j}}\leq{T_{i}}\\}}}{1+m}=\frac{1+(\text{number
of negative control statistics }\leq T_{i})}{1+(\text{number of negative
control statistics})}.$ (3)

This is why we call ${p_{i}}$ the Rank Among Negative Control (RANC) p-value.

If the negative control statistics resemble the test statistics under the
null, we expect that $\hat{F}$ to be close to the null distribution of the
test statistics and ${p_{i}}=\hat{F}({T_{i}})$ to be approximately uniformly
distributed when $m\to\infty$. If we further assume that

$\text{the collection of random
variables}~{}({T_{i}})_{i\in{\mathcal{I}_{0}}\cup{\mathcal{I}_{\text{nc}}}}~{}\text{is
exchangeable},$ (4)

then ${p_{i}}$ is exactly the p-value of the permutation test of
exchangeability using just $T_{i}$. Thus, the proposed RANC p-value is valid
under (4) in the sense that
$\mathbb{P}({p_{i}}\leq\alpha)\leq\alpha~{}\text{for all}~{}0<\alpha<1$.

Our goal in the rest of this section is to give a more precise, non-asymptotic
analysis of the RANC p-values. In particular, we will give sufficient
conditions under which the RANC p-values are individually valid and satisfy a
PRDS property. To this end, we first give a formal definition of
exchangeability and PRDS on a subset of random variables.

######  Definition 1.

We say a sequence of random variables $(X_{i})_{i\in\mathcal{I}}$ is
exchangeable on a subset $(X_{i})_{i\in\mathcal{J}}$ for some
$\mathcal{J}\subseteq\mathcal{I}$, if for any permutation
$g:\mathcal{I}\to\mathcal{I}$ such that $g(i)=i$ for all
$i\not\in\mathcal{J}$, the distribution of $(X_{g(i)})_{i\in\mathcal{I}}$ is
the same as $(X_{i})_{i\in\mathcal{I}}$. When this holds for
$\mathcal{J}=\mathcal{I}$, we simply say the sequence
$(X_{i})_{i\in\mathcal{I}}$ is exchangeable.

######  Remark 1.

Note that this is equivalent to assuming that $(X_{i})_{i\in\mathcal{J}}$ is
exchangeable conditionally on $(X_{i})_{i\in\mathcal{I}\setminus\mathcal{J}}$.
We introduce this new terminology of exchangeability on a subset to contrast
with the definition of PRDS below.

To define PRDS, we say a set $\mathcal{D}\subseteq\mathbb{R}^{n}$ is
increasing if $\mathcal{D}$ contains all $y\in\mathbb{R}^{n}$ that satisfies
$y_{i}\geq x_{i},~{}1\leq i\leq n$ for some $x\in\mathcal{D}$.

######  Definition 2.

We say a sequence of random variables $(X_{i})_{i\in\mathcal{I}}$ exhibits
Positive Regression Dependence on a Subset (PRDS) $(X_{j})_{j\in\mathcal{J}}$
for some $\mathcal{J}\subseteq\mathcal{I}$, if for any increasing set
$\mathcal{D}\subseteq\mathbb{R}^{|\mathcal{I}|}$ and any $j\in\mathcal{J}$,
the conditional probability
$\mathbb{P}\big{(}(X_{i})_{i\in\mathcal{I}}\in\mathcal{D}\mid X_{j}=x\big{)}$
is increasing in $x$. When this holds for $\mathcal{J}=\mathcal{I}$, we simply
say the sequence $(X_{i})_{i\in\mathcal{I}}$ is PRD.

It follows from the definition that PRDS is preserved by co-monotone
transformtions: given some monotonically increasing (or decreasing) functions
$G_{i}$ for $i\in\mathcal{I}$, the assumption that $(X_{i})_{i\in\mathcal{I}}$
is PRDS implies that $(G_{i}(X_{i}))_{i\in\mathcal{I}}$ is also PRDS.

###  3.2 Validity

######  Proposition 3.1.

Fix a true null hypothesis ${H_{i}}$ for some $i\in{\mathcal{I}_{0}}$ and
suppose the following assumptions are satisfied:

  1. (a)

${F_{i}}(t)\leq{F_{j}}(t)$ for all $j\in{\mathcal{I}_{\text{nc}}}$ and
$t\in\mathbb{R}$;

  2. (b)

$({F_{j}}({T_{j}}))_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}$ is
exchangeable.

Then the RANC p-value ${p_{i}}$ is valid in the sense that
$\mathbb{P}({p_{i}}\leq\alpha)\leq\alpha$ for all $0<\alpha<1$.

A proof of Proposition 3.1 can be found in the Appendix. It uses a monotone
coupling argument and the fact that the rank of exchangeable random variables
is uniformly distributed.

######  Remark 2.

The conditions in Proposition 3.1 are stated in terms of the probability
integral transforms of the test statistics and are weaker than the
exchangeability in (4) in two ways. First, as we are only concerned with the
validity of ${p_{i}}$ for some fixed $i$, it is only necessary to assume that
${T_{i}}$ is exchangeable with the negative control statistics. Second, the
null statistic ${T_{i}}$ is allowed to be stochastically larger than the
internal negative control statistic ${T_{j}}$ for all
$j\in{\mathcal{I}_{\text{nc}}}$. This relaxation is useful for testing one-
sided hypotheses; see Remark 8 below. The exchangeability condition (b) is
satisfied when the transformed test statistics are i.i.d. or follow a mixture
of i.i.d. distributions.

###  3.3 PRDS

Because the RANC p-values ${p_{i}}=\hat{F}({T_{i}}),~{}i=1,\dotsc,{n}$ are
calculated using the same empirical null distribution $\hat{F}(\cdot)$, they
are generally not independent even if the test satistics
${T_{i}},~{}i=1,\dotsc,{n}$ are independent. However, it can be shown that the
RANC p-values may satisfy a desirable PRDS property that is sufficient for the
validity of many multiple hypothesis testing procedures [48, 6].

######  Theorem 3.1.

Suppose one of the two sets of conditions holds:

  1. i.

    1. (a)

${T_{i}}\stackrel{{\scriptstyle d}}{{=}}{T_{j}}$ for any
$i\in{\mathcal{I}_{0}}$ and $j\in{\mathcal{I}_{\text{nc}}}$;

    2. (b)

$\left({T_{i}}\right)_{i\in{\mathcal{I}}}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\left({T_{j}}\right)_{j\in{\mathcal{I}_{\text{nc}}}}$;

    3. (c)

$\left({T_{j}}\right)_{j\in{\mathcal{I}_{\text{nc}}}}$ is mutually
independent;

    4. (d)

$\left({T_{i}}\right)_{i\in{\mathcal{I}}}$ is PRDS on
$\left({T_{i}}\right)_{i\in{\mathcal{I}_{0}}}$;

  2. ii.

$\left({T_{i}}\right)_{i\in{\mathcal{I}}\cup{\mathcal{I}_{\text{nc}}}}$ is
exchangeable on
$\left({T_{i}}\right)_{i\in{\mathcal{I}_{0}}\cup{\mathcal{I}_{\text{nc}}}}$.

Then the RANC p-values are valid and $({p_{i}})_{i\in{\mathcal{I}}}$ is PRDS
on $({p_{i}})_{i\in{\mathcal{I}_{0}}}$.

The validity directly follows from Proposition 3.1. Our proof of the PRDS
property is more involved and is based on the following heuristic: if we swap
any $T_{i},i\in{\mathcal{I}_{0}}$ with the next smallest negative control
statistic, the probability that $\bm{p}\in\mathcal{D}$ for any increasing set
$\mathcal{D}$ can only increase. See the Appendix for more detail.

######  Remark 3.

[3, thm. 2] stated and proved the PRDS property in Theorem 3.1 under the
assumption that
$\left({T_{i}}\right)_{i\in{\mathcal{I}}\cup{\mathcal{I}_{\text{nc}}}}$ is
mutually independent,333[3, thm. 2] does make any assumption on non-null
statistics. We believe this is most likely a typo; see Section B.3 in the
Appendix. which implies the partial/conditional exchangeability condition
(ii). The set of conditions in (i), especially the PRDS condition (i)(i.d) on
the original test statistics, appears to be novel and may be quite useful when
the test statistics are positively dependent.

######  Remark 4.

To our knowledge, the conclusion of Theorem 3.1 does not directly follow from
any existing results about positively dependent distributions. First, the PRDS
property of $({p_{i}})_{i\in{\mathcal{I}}}$ in the conclusion of Theorem 3.1
does not immediately follow from condition (i)(i.d)—the same PRDS property for
$({T_{i}})_{i\in{\mathcal{I}}}$—by applying a co-monotone transformation. This
is because the transformation, defined by the internal negative control
statistics, is random. Second, it is tempting to treat
$({T_{i}})_{i\in{\mathcal{I}_{\text{nc}}}}$ as latent and apply sufficient
conditions for PRDS in latent variable models. However, existing results
either assumes a single latent variable [6], considers only binary random
variables [25], or requires the $\text{MTP}_{2}$ property in [30] that is not
implied by the conditions in Theorem 3.1.

######  Remark 5.

Condition (i)(i.a) in Theorem 3.1 cannot be relaxed to the stochastic
dominance condition (a) in Proposition 3.1; see Section B.1 for a counter-
example.

###  3.4 Multiple testing with RANC p-values

Given the conclusions in Propositions 3.1 and 3.1, we briefly discuss the
multiple testing procedures that can be applied to RANC p-values.

####  3.4.1 Testing an intersection null

We first consider Bonferroni’s test and Simes’ test of an intersection null
${H_{\mathcal{I}}}=\cap_{i=1}^{n}{H_{i}}$ at level $\alpha$. Suppose each
hypothesis ${H_{i}}$ is associated with a valid p-value ${p_{i}}$. Let
${p_{(1)}}\leq\dotsb\leq{p_{(n)}}$ be the ordered p-values. Bonferroni’s test
rejects ${H_{\mathcal{I}}}$ if ${p_{(1)}}\leq\alpha/n$ and controls the type I
error at $\alpha$ as long as the individual p-values are valid. Thus, when
applied to the RANC p-values, Bonferroni’s test is valid if the conditions in
Proposition 3.1 are satisfied for all $i$.

Simes’ test rejects ${H_{\mathcal{I}}}$ if ${p_{(i)}}\leq i\alpha/{n}$ for
some $1\leq i\leq n$ [51]. Obviously, Simes’ test rejects ${H_{\mathcal{I}}}$
whenever Bonferroni’s test rejects ${H_{\mathcal{I}}}$. Simes’ test has been
shown to be valid if the p-values are PRD under $H$ [48]. This property and
Theorem 3.1 lead to the following result.

######  Proposition 3.2.

Suppose ${F_{i}}(t)\leq{F_{j}}(t)$ for all $i\in{\mathcal{I}_{0}}$,
$j\in{\mathcal{I}_{\text{nc}}}$, and $t\in\mathbb{R}$, and one of the
following sets of conditions holds,

  1. i.

(i)(i.b), (i)(i.c), and (i)(i.d) in Theorem 3.1;

  2. ii.

$\left({F_{i}}({T_{i}})\right)_{i\in{\mathcal{I}_{0}}\cup{\mathcal{I}_{\text{nc}}}}$
is exchangeable.

Then Simes’ test applied to the RANC p-values controls the type I error for
testing the intersection null ${H_{\mathcal{I}}}$.

Apart from a relaxation of condition (i)(i.a) in Theorem 3.1 to stochastic
dominance, the conditions in Proposition 3.2 are the same as those in Theorem
3.1 (note that ${\mathcal{I}}={\mathcal{I}_{0}}$ if ${H}$ is true). We cannot
use Theorem 3.1 directly to prove Proposition 3.2 (see Remark 5 above).
However, Proposition 3.2 suggests that the (infeasible) RANC p-values
$(\tilde{{p_{i}}})_{i\in{\mathcal{I}}}$, obtained from test statistics
$({F_{i}}({T_{i}}))_{i\in{\mathcal{I}}}$ and internal negative control
statistics $({F_{i}}({T_{i}}))_{i\in{\mathcal{I}_{\text{nc}}}}$, are PRD under
$H$. So Simes’ test applied to $(\tilde{{p_{i}}})_{i\in{\mathcal{I}}}$
controls the type I error. Under the assumptions in Proposition 3.2, we have
${p_{i}}\geq\tilde{{p_{i}}}$ for all $i\in{\mathcal{I}}$. Thus, if Simes’ test
applied to $({p_{i}})_{i\in{\mathcal{I}}}$ rejects $H$, it must also reject
$H$ when applied to $(\tilde{{p_{i}}})_{i\in{\mathcal{I}}}$. The conclusion in
Proposition 3.2 immediately follows.

There are many other global tests besides Bonferroni’s correction and Simes’
test; a prominent example is Fisher’s combination test that requires
independent p-values [14]. When applied to RANC p-values, however, such
methods may not always control the type I error; in the case of Fisher’s test,
see Section B.2 for a counter-example and [3, sec. 2.1] for a theoretical
characterization of this negative result. Nevertheless, by assuming the
exchangeability in (4), the distribution of any test statistic under the
global null can be obtained by using permutations. RANC p-values come up
naturally in such permutation tests: Proposition A.1 in the Appendix shows
that all permutation tests that are invariant under monotone transformations
and certain permutations can be written as a function of the RANC p-values.

######  Remark 6.

It may be interesting to compare Simes’ test applied to
$({p_{i}})_{i\in{\mathcal{I}}}$ with the permutation test applied to the
statistic $\min_{i\in{\mathcal{I}}}{p_{(i)}}/i$. As both tests use the same
test statistic and the permutation test is exact, Simes’ test can be viewed as
a conservative approximation to the permutation test with a simple rejection
threshold. Numerical simulations shows that this approximation becomes more
accurate when $m$ is much larger than ${n}$ (Figure 3). Heuristically, this is
because Simes’ test is exact when the p-values are independent, and the
dependence of the RANC p-values decreases as the number of negative controls
increases.

(a) ${n}=25$, $m=25$

(b) ${n}=25$, $m=500$

Figure 3: Cumulative distribution functions of the permutation distribution of
the Simes statistic $n\min_{i\in{\mathcal{I}}}({p_{(i)}}/i)$ with different
numbers of internal negative controls ($m=25$ and $m=500$), estimated by
$1000$ random permutations. Simes’ test corresponds to assuming a uniform
distribution.

####  3.4.2 Testing individual hypotheses

We now review methods for testing the individual hypotheses ${H_{i}}$,
$i=1,\dotsc,{n}$. Bonferroni’s correction rejects ${H_{i}}$ if
${p_{i}}\leq\alpha/n$. Holm’s step-down procedure is obtained from closing
Bonferroni’s correction; it orders the hypotheses by ranking p-values from
small to large and keeps on rejecting hypotheses as long as
$p_{(i)}\leq\alpha/({n}-i+1)$. [26]. Both procedures only require that the
p-values are individually valid and thus, when applied to the RANC p-values,
control the FWER if the conditions in Proposition 3.1 are satisfied for all
$i\in{\mathcal{I}_{0}}$. This also applies to graph-based procedures such as
fixed sequence testing and the fallback procedure [65, 7]. Hochberg’s
procedure [23] and Hommels’ procedure [27] are based on closing the Simes’
test. When applied to RANC p-values, they control the FWER under the
conditions in Proposition 3.2.

For FDP control, [33] proposed a step-down procedure that keeps on rejecting
hypotheses as long as $p_{(i)}\leq(\lfloor qi\rfloor+1)\alpha/({n}+\lfloor
qi\rfloor+1-i)$. They showed that this procedure satisfies
$\mathbb{P}(\text{FDP}>q)\leq\alpha$ if the so-called Simes’ inequality is
satisfied [33, thm. 3.2]. Therefore, their procedure applied to the RANC
p-values is valid under the conditions in Proposition 3.2.

As mentioned above, FWER and FDP control are intimately related to testing
intersection nulls. Given valid tests of the intersection nulls, [40] proposed
a closed testing procedure that rejects $H_{i}$ if all intersection nulls
$\\{H_{\mathcal{J}}=\cap_{j\in\mathcal{J}}H_{j}:\mathcal{J}\ni i\\}$ that
logically imply $H_{i}$ are rejected. They showed that this procedure controls
the FWER. [19] extended this procedure to simultaneously estimate the FDP
among any subset of hypotheses [[, see
also]]genovese2004stochastic,genovese06_exceed_contr_false_discov_propor. [18]
showed that all methods that control FWER or some tail probability of FDP can
be written as or uniformly improved by a closed testing procedure. Because
closed testing only requires validity of the tests for the intersection nulls,
in principle it can be applied with Simes’ tests using the RANC p-values when
the conditions in Proposition 3.2 are satisfied.

For FDR control, the BH step-up procedure is most widely used. Let $i$ be the
largest index such that ${p_{(i)}}\leq iq/{n}$; the BH procedure rejects
${H_{j}}$ for all $j$ such that ${p_{j}}\leq{p_{(i)}}$ [4]. The BH procedure
is proven to control the FDR at the nominal level when the p-values are
independent [4] or PRDS on the set of true nulls [6]. This implies the
following result.

######  Corollary 3.1.

Under any sets of conditions in Theorem 3.1, the BH procedure applied to the
RANC p-values controls the FDR.

######  Remark 7.

A concrete example of an exchangeable but non-i.i.d. sequence is the
equicorrelated multivariate normal (EMN) model:

${T_{i}}=\mu_{i}+\sqrt{\rho}Z+\sqrt{1-\rho}X_{i},\quad{i\in{\mathcal{I}}\cup{\mathcal{I}_{\text{nc}}}},$
(5)

where $Z$ and $X_{i}$ are i.i.d. standard normal variables and $0\leq\rho<1$.
Jointly, the test statistics follow a multivariate normal distribution with
all pairwise correlations equal to $\rho$. Suppose $\mu_{i}=0$ for all
$i\in{\mathcal{I}_{0}}\cup{\mathcal{I}_{\text{nc}}}$, then
$({T_{i}})_{i\in{\mathcal{I}_{0}}\cup{\mathcal{I}_{\text{nc}}}}$ is
exchangeable but not independent. It is straightforward to verify that the
exchangeability conditions in Proposition 3.1(b), Theorem 3.1(ii), and
Proposition 3.2(ii) are all satisfied. A simulated example from the EMN model
is given in Section 6.

##  4 The empirical process perspective

An unsatisfactory aspect of Corollary 3.1 is that it requires the marginal
distribution of the test statistic to be the same for all true null hypotheses
and negative controls. In this section, we develop an alternative
justification of this BH procedure that relaxes this assumption. Our argument
is based on using negative controls to form a nonparametric estimator of the
false discovery rate, and is closely related to the empirical process
perspective in [56, 16]. A practical advantage of this approach is that one
can increase the power of the BH procedure by estimating the proportion of
nulls [55].

We first set up the notation to state our main result. Following [56, 16], we
view FDP as an empirical process indexed by the rejection threshold. As the
RANC p-values are invariant under a monotone transformation of the original
test statistics, without loss of generality, we assume ${T_{i}}\in[0,1]$ for
all $i$. The empirical processes for false rejections (${V}$ in Table 1), all
rejections (${R}$ in Table 1), and the FDP are defined as

$\displaystyle{V}(t)$
$\displaystyle:=\sum_{i\in{\mathcal{I}_{0}}}\mathbbm{1}_{\\{{T_{i}}\leq
t\\}},\quad{R}(t):=\sum_{i\in{\mathcal{I}}}\mathbbm{1}_{\\{{T_{i}}\leq
t\\}},\quad\text{and}\quad\text{FDP}(t):=\frac{V(t)}{R(t)\vee 1},\quad 0\leq
t\leq 1.$

For fixed $t$, let the expectation of $\text{FDP}(t)$ be
$\text{FDR}(t)=\mathbb{E}\left[\text{FDP}(t)\right]$. A multiple testing
method such as the BH procedure selects a data-dependent rejection threshold
$\tau$, and we are interested in controlling
$\text{FDR}=\mathbb{E}[\text{FDP}(\tau)]$ at level $q$. This may be achieved
by directly estimating $\text{FDR}(t)$ and stopping the procedure when the
estimated FDR is above $q$. This typically involves estimating the number of
false positives ${V}(t)$. For example, in the usual setting that the test
statistics are p-values and follow $U[0,1]$ under the null, we may estimate
${V}(t)$ conservatively by $nt$, the expectation of ${V}(t)$ when
${\mathcal{I}_{0}}={\mathcal{I}}$.

Compared to previous work [56, 16], a key differerence in our problem is that
the null distribution of the test statistics is unknown and must be estimated
from the negative controls. To this end, let the empirical process for the
negative control rejections and its normalization be, respectively,

$V_{\text{nc}}(t):=\sum_{j\in{\mathcal{I}_{\text{nc}}}}\mathbbm{1}_{\\{{T_{j}}\leq
t\\}},\quad\bar{V}_{\text{nc}}(t):=\frac{n(V_{\text{nc}}(t)+2)}{m+1},\quad
0\leq t\leq 1.$

We propose to estimate $\text{FDR}(t)$ by

$\displaystyle\widehat{\text{FDR}}_{\lambda}(t):=\frac{\hat{\pi}(\lambda)\cdot\bar{V}_{\text{nc}}(t)}{R(t)\vee
1},\quad\text{for some}~{}0<\lambda\leq 1,$ (6)

where $\hat{\pi}(\lambda)$ is the following estimator of the proportion of
true nulls $|{\mathcal{I}_{0}}|/|{\mathcal{I}}|$:

$\displaystyle\hat{\pi}(\lambda)=\begin{dcases}1,&\text{if}~{}\lambda=1,\\\
\frac{{n}+1-{R}(\lambda)}{{n}}\cdot\frac{m+1}{m-{V_{\text{nc}}}(\lambda)},&\text{if}~{}0<\lambda<1.\\\
\end{dcases}$ (7)

Equation (7) is modified from [56, eq. (6)]. It can be shown that
$\widehat{\text{FDR}}_{\lambda}(t)$ is a conservative (i.e. downward biased)
estimator of $\text{FDR}(t)$ when $\lambda=1$; see Proposition A.2 in the
Appendix. Finally, let the rejection threshold be

$\displaystyle{\tau_{q}}=\sup\left\\{0\leq
t\leq\lambda:\widehat{\text{FDR}}_{\lambda}(t)\leq q\right\\},$

so a hypothesis ${H_{i}},~{}i\in{\mathcal{I}}$ is rejected if
${T_{i}}\leq{\tau_{q}}$.

The next proposition relates this rejection threshold with the BH procedure
applied to the RANC p-values.

######  Proposition 4.1.

A hypothesis ${H_{i}},~{}i\in{\mathcal{I}}$ is rejected by the above procedure
when $\lambda=1$ if and only if it is rejected by the BH procedure with the
following modified RANC p-values:

$\tilde{{p_{i}}}=\frac{2+\sum_{j\in{\mathcal{I}_{\text{nc}}}}\mathbbm{1}_{\\{{T_{j}}\leq{T_{i}}\\}}}{1+m}\wedge
1.$

Proposition 4.1 follows from the simple observation that

$\displaystyle\widehat{\text{FDR}}({T_{i}})\leq q\quad\text{if and only
if}\quad\tilde{{p_{i}}}=\frac{2+V_{\text{nc}}({T_{i}})}{1+m}\wedge
1\leq\frac{R({T_{i}})}{{n}}q.$

Compared to the original RANC p-value $p_{i}$ defined in (3), an extra $1$ is
added to the numerator of $\tilde{{p_{i}}}$. This subtle modification is
needed because, unlike the problem with a known null distribution studied by
[56] and others, due to the discreteness of $V_{\text{nc}}(t)$ (and hence
$\bar{V}_{\text{nc}}(t)$), it is generally not true that
$\widehat{\text{FDR}}_{\lambda}({\tau_{q}})\leq q$. Although this modification
only makes a minuscule difference in most practical problems, it is needed in
the super-martingale proof of the next Theorem.

The final piece we need to state the main theorem of this section is a
stronger notion of stochastic dominance [68].

######  Definition 3 (Uniform stochastic dominance).

For two random variables $X$, $Y$ supported on $[0,1]$, we say $X$ is
uniformly stochastically larger than $Y$ if $\mathbb{P}(X\leq t)>0$,
$\mathbb{P}(Y\leq t)>0$, and $\mathbb{P}(X\leq s\mid X\leq
t)\leq\mathbb{P}(Y\leq s\mid Y\leq t)$ for all $0<s\leq t\leq 1$.

Heuristically, uniform stochastic dominance just means stochastic dominance
after conditioning on the variable is less than $t$ for all $t$. It is
satisfed if the distributions of $X$ and $Y$ are in a family with monotone
likelihood ratio; more examples and results can be found in [64, 68].

######  Theorem 4.1.

Suppose the following conditions are true:

  1. (a)

${T_{i}}$ is uniformly stochastically larger than ${T_{j}}$ for all
$i\in{\mathcal{I}_{0}}$ and $j\in{\mathcal{I}_{\text{nc}}}$;

  2. (b)

$({T_{i}})_{i\in{\mathcal{I}}\cup{\mathcal{I}_{\text{nc}}}}$ is mutually
independent.

Then for any fixed $0<\lambda\leq 1$, the step-up procedure with rejection
threshold ${\tau_{q}}$ controls the FDR at level $q$.

We provide a sketch proof of this result by modifying the martingale argument
in [56]; more details can be found in Section A.2. We consider the time-
reversals of ${V}(t)$, ${S}(t)$, and ${V_{\text{nc}}}(t)$ starting from $t=1$
and define the backward filtrations as
$\mathcal{F}_{t}=\sigma\big{(}{V}(s),{S}(s),{V_{\text{nc}}}(s):t\leq s\leq
1\big{)}$ for $0\leq t\leq 1$. Our proof rests on showing the following
process is a backward super-martingale:

$\displaystyle M(t)=\frac{{V}(t)}{(1+{V_{\text{nc}}}(t))/(1+m)}.$ (8)

This extends the martingale $V(t)/t$ in [56] when the null CDF is known to be
$F_{0}(t)=t$. More precisely, it is proved in the Appendix that

$\displaystyle\mathbb{E}\left[M(s)\mid\mathcal{F}_{t}\right]\leq
M(t)\cdot\left(1-\left(1-p_{t}^{s}\right)^{{V_{\text{nc}}}(t)+1}\right)\leq
M(t)~{}\text{for some}~{}0\leq p_{t}^{s}\leq 1~{}\text{and all}~{}0\leq s\leq
t\leq 1.$ (9)

Theorem 4.1 then follows from applying the optional stopping theorem.

######  Remark 8.

Since Theorem 4.1 does not require exchangeability of the test statistics, it
can be applied to one-sided tests. Specifically, suppose $T_{i}$ is the
likelihood-ratio statistic for testing ${H_{0}}:\theta_{i}\leq\theta_{0}$ vs.
${H_{1}}:\theta_{i}>\theta_{0}$ in a one-dimensional exponential family with
natural (or mean) parameter $\theta$ (and thus has a monotone likelihood
ratio). We have $\theta_{i}\leq\theta_{0}$ for $i\in{\mathcal{I}_{0}}$ by
definition. If $\theta_{i}\geq\theta_{0}$ for all
$i\in{\mathcal{I}_{\text{nc}}}$, then condition (a) in Theorem 4.1 is
satisfied. This suggests another useful aspect of the stochastic dominance
condition (a): the definition of “negative controls” can be relaxed and they
do not need to be true null hypotheses. In other words, the RANC p-values are
robust to incorrect selection of negative controls in the sense that the FDR
may still be controlled at the nominal level. However, when too many negative
controls are not true nulls, the multiple testing procedure may have very
little power.

######  Remark 9.

A similar martingale argument was developed by [41] to prove that the BH
procedure ($\lambda=1$) applied to the RANC p-values controls the FDR. A main
distinction is that they require partial exchangeability of the test
statistics (condition (ii) in Theorem 3.1), which is weaker than the
independence condition (b) in Theorem 4.1 but does not allow the case of
uniformly stochastic dominance in condition (a).444Although by using de
Finetti’s theorem, the independence condition (b) in Theorem 4.1 can be easily
relaxed to (conditional) $\infty$-extendability [8]. The uniformly stochastic
dominance condition (a) arises naturally in our proof of the first inequality.
It remains unclear to us whether this can be allowed in the proof in [41], as
their martingale is not indexed by the rejection threshold. When the marginal
distributions of the null test statistics and internal negative controls are
the same, the gap $M(t)-\mathbb{E}\left[M(s)\mid\mathcal{F}_{t}\right]$ is
small for large ${V_{\text{nc}}}(t)$, so $M(t)$ is almost a martingale. This
can be used to prove a lower bound on the FDR; see [41, thm. 3.1].

##  5 Local-FDR control

We now turn to our third method motivated by the ad hoc procedure in [28]. We
will consider the two-mixture setup for multiple testing in Section 2. More
specifically, we will assume for the rest of this section that
$({H_{i}},{T_{i}}),~{}i=1,\dotsc,{n}$ are i.i.d.,
${H_{i}}\sim\text{Bernoulli}(1-\pi)$, and
${T_{i}}\mid{H_{i}}\sim{F_{{H_{i}}}}$, so the marginal CDF of
$T_{i},~{}i\in{\mathcal{I}}$ is given by $F(t)=\pi F_{0}(t)+(1-\pi)F_{1}(t)$.
To simplify the discussion, we will assume the null proportion ${\pi}$ is
known. In practice, ${\pi}$ is generally unknown and a wealth of estimators of
${\pi}$ have been proposed in the literature [49, 24, 22, 58, 5, 55, 29]; see
[16] for a review of their asymptotic properties.

###  5.1 PDF-based methods

Currently, most practical applications estimate the local-FDR by plugging in
estimators of ${f}$ (and ${f_{0}}$ if it is unknown) into the definition of
local-FDR in (1); we will call such methods PDF-based, to contrast with our
proposal below that is CDF-based. A major limitation of the PDF-based approach
is that the estimated densities may be very inaccurate at the tail where the
rejection threshold is likely to be located. This is worsened by the fact that
the marginal density $f(t)$ appears in the denominator of the definition of
$\text{local-FDR}(t)$. Another problem is that PDF-based estimators of the
local-FDR are generally not invariant to, monotone transformation of the test
statistics, but the definition of local-FDR is. Thus, very different rejection
sets may be obtained if the investigator chooses to use different
transformations of the test statistics.

We illustrate the performance of PDF-based methods with a simple simulation
example. We generate ${n}=400$ test statistics with ${\pi}=0.5$,
${F_{0}}=t_{10}$ (t-distribution with $10$ degrees of freedom), and
${F_{1}}=\text{Exp}(1)$, and an independent set of $m=1000$ internal negative
controls from ${F_{0}}$. We assume ${\pi}$ is known and set ${q}=0.3$. Figure
4 shows that when a simple kernel density estimator is used to estimate both
${F}$ and ${F_{0}}$, the estimated local-FDR is highly variable at the left
tail and the step-down rejection threshold (dashed verticle line) is too
conservative to be useful. Transforming the test statistics to z-scores
improves the performance of this PDF-based method, but the rejection threshold
is still too small (Figure 4).

###  5.2 CDF-based methods

Next, we relate the CDF-based cut-off analysis in [28] to local-FDR control.
To this end, consider the following optimization problem for some given
$\lambda=q/{\pi}$ and $0<q<1$:

$\displaystyle{\hat{\tau}_{\lambda,{n},m}}:=\mathop{\mathrm{argmin}}_{{t}}{{F}_{0,m}}({t})-\lambda{{F}_{n}}({t}),$
(10)

where the objective function is a weighted difference between the empirical
CDFs of the negative controls and test statistics:

${{F}_{0,m}}(t)=\frac{1}{m}\sum_{j\in{\mathcal{I}_{\text{nc}}}}\mathbbm{1}_{\\{{T_{j}}\leq
t\\}},~{}{{F}_{n}}(t)=\frac{1}{{n}}\sum_{i\in{\mathcal{I}}}\mathbbm{1}_{\\{{T_{i}}\leq
t\\}}.$

Thus, the procedure in [28] (see Figure 2(d)) corresponds to using $\lambda=1$
or equivalently $q=\pi$. In a moment, it will be clear that $q$ can be
understood as the local-FDR level targeted by using the rejection threshold
${\hat{\tau}_{\lambda,{n},m}}$.

By the Glivenko-Cantelli theorem, ${{F}_{0,m}}$ and ${{F}_{n}}$ converge
uniformly to ${F_{0}}{}$ and ${F}$, respectively. Thus, we expect
${\hat{\tau}_{\lambda,{n},m}}$ to converge to the minimizer

$\tau_{\lambda}^{*}:=\mathop{\mathrm{argmin}}_{t}{F_{0}}({t})-\lambda{F}({t}).$
(11)

By setting the derivative of Eq. (11) to zero, we obtain
${f_{0}}(\tau_{\lambda}^{*})=(q/{\pi}){f}(\tau_{\lambda}^{*})$, or
equivalently,

$\text{local-
FDR}(\tau_{\lambda}^{*})=\frac{\pi{f_{0}}(\tau_{\lambda}^{*})}{{f}(\tau_{\lambda}^{*})}=q,$

where ${f_{0}}{}$ and ${f}$ are the density functions corresponding to
${F_{0}}{}$ and ${F}$. As mentioned above, the procedure in [28] corresponds
to using $\lambda=1$, or equivalent $q=\pi$. Intuitively, this is a sensible
choice because $\pi$ is simply the probability of making a false discovery by
rejecting a random hypothesis.

(c) PDF-based

(d) PDF-based, transformed ${T_{i}}$

(e) CDF-based

Figure 4: Comparison of CDF-based and PDF-based methods where ${n}=400$,
$m=1000$, $\pi=0.5$, $F_{0}=t_{10}$, $F_{1}=\text{Exp}(1)$, and $q=0.3$. Panel
(a) shows the result of a PDF-based method, where the null density and the
marginal density are estimated by a kernel estimator (R function density with
default options). In panel (b), we apply the PDF-based method to the
transformed test statistics $\Phi^{-1}({F_{0}}({T_{i}}))$, where the null
distribution of the transformed test statistics is $\mathcal{N}(0,1)$. In
panel (c), we showcase the proposed CDF-based method (10). We plot the
weighted type I error $(1-{q}){F_{0}}({t})$ (dark red), the weighted type II
error ${q}(1-{\pi})/{\pi}(1-{F_{1}}({t}))$ (dark green), and superimpose the
population risk (blue)—the sum of the two weighted errors up to a constant
shift, and the empirical objective (black). (The population and empirical risk
curves have been scaled for visualization purposes.)

The heuristic argument in the previous paragraph can be made precise with the
help of the empirical process theory. By assuming that the density functions
$f_{0}$ and $f$ are differentiable at $\tau_{\lambda}^{*}$ and
$(f_{0}/f)^{\prime}(\tau_{\lambda}^{*})>0$, it can be shown that the estimated
threshold ${\hat{\tau}_{\lambda,{n},m}}$ converges to $\tau_{\lambda}^{*}$ at
rate $({n}\wedge m)^{-1/3}$ when $n,m\to\infty$. Morevoer, by assuming
monotone likelihood ratio (i.e. $({f_{0}}{}/{f})^{\prime}>0$) and additional
regularity conditions, it can be shown that such convergence is uniform over
$\lambda$. See Section A.3 for some technical results.

The optimization problem (11) can be rewritten as a decision-theoretic
problem: let $\hat{H}_{i}(t)=\mathbbm{1}_{\\{T_{i}\leq{t}\\}}$ be the one-
sided decision rule, then

$\tau_{\lambda}^{*}=\mathop{\mathrm{argmin}}_{t}\mathbb{E}_{{F_{0}}}\left[\hat{H}_{i}(t)\right]-\lambda\mathbb{E}_{{F}}\left[\hat{H}_{i}(t)\right]=\mathop{\mathrm{argmin}}_{t}\mathbb{E}\left[{w_{0}}\mathbbm{1}_{\\{{H_{i}}<\hat{{H_{i}}}(t)\\}}+{w_{1}}\mathbbm{1}_{\\{{H_{i}}>\hat{{H_{i}}}(t)\\}}\right],$
(12)

where ${w_{0}}=1-q$ is the loss of a false positive and ${w_{1}}=q$ is the
loss of a false negative. This connection was pointed out by [57].

In contrast to PDF-based methods, the CDF-based method proposed here is free
of tuning parameters and invariant to monotone transformations of the test
statistics. Figure 4 illustrates the practical performance of this procedure
using the same simulation example above, by showing the population and
empirical risk in (12) as curves of the rejection threshold $t$. It is evident
that the CDF-based rejection threshold ${\hat{\tau}_{\lambda,{n},m}}$ is close
to $\tau_{\lambda}^{*}$ and its risk is close to the oracle.

We offer some additional remarks on the CDF-based method above.

######  Remark 10.

By using the order statistics ${p_{(1)}}<\dotsb<{p_{({n})}}$ of the RANC
p-values, the optimization problem (10) can also be rewritten as as

$\displaystyle{\hat{\tau}_{\lambda,{n},m}}=T_{(i^{*})},~{}\text{where}~{}i^{*}=\mathop{\mathrm{argmin}}_{i}\frac{m+1}{m}{p_{(i)}}-\frac{\lambda
i}{{n}}.$

This is almost identical to the method in [53]. More precisely, [53] assumes
an independent sequence of p-values that are uniformly distributed under the
null (so $F_{0}(t)=t$ is known) is given and does not include the factor
$(m+1)/m$ in the last equation. [53] not only proved the same ${n}^{-1/3}$
convergence rate for their estimator of the rejection threshold but also
showed that the expected maximum local-FDR of the rejected hypotheses is
controlled if the likelihood ratio is monotone. Our numerical simulation
suggests that the method proposed here might also be able to control the
maximum local-FDR, but we are unable use the technique developed by [53] to
prove this because the RANC p-values are not independent.

######  Remark 11.

The method above can be extended to estimate the local-FDR curve, which shall
be denoted by $q(t)$. In fact,
$\hat{\tau}({q}):=\hat{\tau}_{\lambda=q/{\pi},{n},m}$ is an increasing and
piece-wise constant function. Thus, the local-FDR curve can be estimated by
inverting $\hat{\tau}({q})$,

$\displaystyle{\hat{q}({t})}=\inf_{q}\\{{q}:\hat{\tau}(q)\geq{t}\\},\quad\text{for
any}~{}{t}.$ (13)

The resulting local-FDR curve ${\hat{q}({t})}$ is increasing, piece-wise
constant, left-continuous, and the jump points are contained in
$({T_{i}})_{i\in{\mathcal{I}}}$. The estimated curve $\hat{q}(t)$ is
essentially a light modification of Grenander’s estimator [20] of a monotone
density function. It is well known that Grenander’s estimator converges at the
rate $n^{-1/3}$; see [9] for a recent review.

######  Remark 12.

Because the objective function (11) is typicaly locally convex at its optimum,
it is expected that the regret
$\\{{F_{0}}({\hat{\tau}_{\lambda,{n},m}})-\lambda{F}({\hat{\tau}_{\lambda,{n},m}})\\}-\\{{F_{0}}(\tau_{\lambda}^{*})-\lambda{F}(\tau_{\lambda}^{*})\\}$
usually converges to zero twice as fast as ${\hat{\tau}_{\lambda,{n},m}}$
converges to $\tau_{\lambda}^{*}$. However, even when the risk is almost flat
around $\tau_{\lambda}^{*}$, a simple argument using the
Dvoretzky–Kiefer–Wolfowitz (DKW) inequality shows that the regret converges at
least at the rate ${n}^{-1/2}$; see Proposition A.3 in the Appendix. Thus, the
simple CDF-based method almost always has a small regret, even if the
rejection threshold $\tau_{\lambda}^{*}$ is not estimated very accurately.

##  6 Simulations

When using multiple testing methods based on negative controls, a concern in
practice is that they may be not very powerful. We investigate this using
numerical simulations.

We generate a set of baseline p-values
$({T_{i}})_{i\in{\mathcal{I}}\cup{\mathcal{I}_{\text{nc}}}}$ that might be
individually invalid, and compare three variations of the BH procedure:

  1. 1.

the standard BH procedure (BH) that assumes the validity of the baseline
p-values and directly aggregates $({T_{i}})_{i\in{\mathcal{I}}}$;

  2. 2.

the BH procedure with RANC p-values (BH RANC) that first computes the RANC
p-values $({p_{i}})_{i\in{\mathcal{I}}}$ using
$({T_{i}})_{i\in{\mathcal{I}}\cup{\mathcal{I}_{\text{nc}}}}$ and then applies
the BH procedure to $({p_{i}})_{i\in{\mathcal{I}}}$;

  3. 3.

the oracle BH procedure (BH oracle) applies the BH procedure to the p-values
corrected by the true null CDF.

We experiment with $6=3\times 2$ joint distributions of the baseline p-values
with different marginal distributions and dependency structures.

  * •

Marginal distribution. We consider three marginal distributions of null
baseline p-values: $T\overset{d}{=}\Phi(Z+\mu)$ where $Z\sim\mathcal{N}(0,1)$
and $\mu\in\\{-0.5,0,0.5\\}$, corresponding respectively to anti-conservative
(anti-csvr.), exact (exact), and conservative (csvr.) p-values. The marginal
distribution of the non-null baseline p-values is set to $\Phi(Z-3)$.

  * •

Dependency. We consider two types of dependencies: the independent setting
(ind.) where $({T_{i}})_{i\in{\mathcal{I}}\cup{\mathcal{I}_{\text{nc}}}}$ are
mutually independent, and the exchangeable setting (exch.) where
$({T_{i}})_{i\in{\mathcal{I}}\cup{\mathcal{I}_{\text{nc}}}}$ are generated
from an EMN model in Remark 7 with correlation parameter $\rho=0.5$.

We generate ${n_{0}}=100$ null test statistics, ${n_{1}}=10$ non-null test
statistics, and $m=200$ internal negative controls. In each trial, we record
the FDP and the true positive rate (the number of true discoveries divided by
the total number of non-nulls) of each method. All configurations are repeated
$10^{4}$ times. We set the target FDR level to $q=0.2$.

Table 2: FDR and power analysis. We compare the validity and the power of three variants of the BH procedure (FDR level $q=0.2$) across $6$ joint distributions of the baseline p-values. The standard deviation of the FDP and the true positive rate is recorded in the bracket. All settings are repeated $10^{4}$ times. | BH | BH RANC | BH oracle
---|---|---|---
FDR | power | FDR | power | FDR | power
ind. | csvr. | 0.047 | 0.8 | 0.17 | 0.9 | 0.18 | 0.93
(0.072) | (0.15) | (0.13) | (0.13) | (0.12) | (0.087)
exact | 0.18 | 0.82 | 0.16 | 0.76 | 0.18 | 0.82
(0.13) | (0.14) | (0.14) | (0.21) | (0.13) | (0.14)
anti-csvr. | 0.49 | 0.87 | 0.16 | 0.53 | 0.18 | 0.63
(0.13) | (0.12) | (0.16) | (0.28) | (0.15) | (0.19)
exch. | csvr. | 0.044 | 0.76 | 0.17 | 1 | 0.13 | 0.9
(0.14) | (0.31) | (0.13) | (0.018) | (0.25) | (0.2)
exact | 0.13 | 0.77 | 0.17 | 0.98 | 0.13 | 0.77
(0.25) | (0.31) | (0.13) | (0.047) | (0.25) | (0.31)
anti-csvr. | 0.31 | 0.77 | 0.17 | 0.91 | 0.13 | 0.57
(0.35) | (0.31) | (0.13) | (0.12) | (0.25) | (0.38)

Table 2 reports the result of this simulation study. The BH RANC controls FDR
in all $6$ settings, while the standard BH fails when the individual p-values
are invalid (in two anti-csvr. settings). In terms of statistical power, the
BH RANC is comparable to the BH oracle when the p-values are independent, and,
perhaps surprisingly, is more powerful when the p-values are positively
dependent (in three exch. settings). In fact, BH RANC is more powerful when
the baseline p-values are positively dependent than when they are independent.
We believe this surprising gain of power is due to the fact that the RANC
p-values are invariant to monotone transformations and are hence invariant to
the shared latent factor $Z$ in the EMN model (5). In other words, the RANC
p-values effectively have a larger signal-to-noise-ratio in the dependent
case.

An important question in practice is how many negative controls are needed to
use RANC p-values and have decent power. In order for BH RANC to reject all
non-nulls, a necessary condition is that $p_{({n_{1}})}\leq q{n_{1}}/{n}$.
Since the minimal RANC p-value is at least $1/(1+m)$, the above constraint
implies that we need $m\geq C{n}/(q{n_{1}})$ for some multiple $C>1$ that may
depend on the signal strength. By varying the number of internal negative
controls in the the simulation setup above, we find that $C=2$ is a good rule
of thumb for the power of BH RANC to be close to that of BH oracle; see Figure
C.1 in the Appendix. When the non-nulls have a smaller effect size, a larger
$C$ may be required. See also the simulation study in [41] and a related rule-
of-thumb developed there.

##  7 Real data analysis

We now return to the motivating proteomic data analysis described in Section
1.1. As mentioned already, there are ${n}=740$ proteins under investigation
and $m=2,067$ internal negative control proteins. We denote the protein
abundance under the treatment and the control condition by
${Y_{i}^{\text{t}}}$ and ${Y_{i}^{\text{c}}}$, respectively. For each protein
$i\in{\mathcal{I}}=\\{1,\dotsc,{n}\\}$, we would like to test the one-sided
hypothesis:

$\displaystyle{H_{i,0}}:\mathbb{E}[{Y_{i}^{\text{t}}}]\leq\mathbb{E}[{Y_{i}^{\text{c}}}]\quad\text{v.s.}\quad{H_{i,1}}:\mathbb{E}[{Y_{i}^{\text{t}}}]>\mathbb{E}[{Y_{i}^{\text{c}}}].$

For internal negative control protein
$j\in{\mathcal{I}_{\text{nc}}}=\\{741,\dotsc,2807\\}$, its expression is
anticipated to be the same over the two conditions, i.e.
$\mathbb{E}[{Y_{j}^{\text{t}}}]=\mathbb{E}[{Y_{j}^{\text{c}}}]$.

###  7.1 Falsification of negative controls

The validity of the internal negative controls can be falsified by comparing
the empirical distribution of ${Y_{j}^{\text{t}}}-{Y_{j}^{\text{c}}}$ over
different subgroups of negative controls. In this example, Figure 5 shows that
the test statistics for proteins annotated with different non-membrane
subcellular locations are distributed similarly, thereby supporting the usage
of them as internal negative controls.

Figure 5: Quantile-quantile plots of the abundance differences across internal
negative control subgroups. We partition the internal negative controls into
three subgroups according to the subcellular location keywords “nuclear”,
“mitochondrial”, and “cytoplasmic”. The K-S test p-values for all three pair-
wise subgroup comparisons are non-significant at level $0.05$.

###  7.2 Choice of empirical null

We investigate different choices of the empirical null distributions in this
proteomic dataset. We apply different normally distributed null distributions
obtained using the method described next. For each
$\mathcal{J}\in\\{\mathcal{I},\mathcal{I}\cup\mathcal{I}_{\text{nc}},\mathcal{I}_{\text{nc}}\\}$
(corresponding to using, respectively, all test statistics under
investigation, all test statistics observed, and only the negative control
statistics), we consider three different estimators of the mean $\mu$ and
standard deviation $\sigma$ of the null distribution:

  1. 1.

MAD1: $\mu=0$,
$\sigma=\sqrt{\\{\texttt{MAD}(({Y_{i}^{\text{t}}})_{i\in\mathcal{J}})\\}^{2}+\\{\texttt{MAD}(({Y_{i}^{\text{c}}})_{i\in\mathcal{J}})^{2}\\}}$,
where mad computes the median of the absolute deviations from the median
multiplied by a factor of 1.4826 that ensures consistency for the normal
distribution;

  2. 2.

MAD2: $\mu=0$,
$\sigma=\texttt{MAD}(({Y_{i}^{\text{t}}}-{Y_{i}^{\text{c}}})_{i\in\mathcal{J}})$;

  3. 3.

Efron: the method described in [12], in which a Poisson regression is first
applied to $({Y_{i}^{\text{t}}}-{Y_{i}^{\text{c}}})_{i\in\mathcal{J}}$ to
estimate the density function and $\mu$ and $\sigma$ are then obtained from
the mode and the half-width of the center peak of the estimated density
function.

We then compute the one-sided p-values using each empirical null distribution.
Our proposed RANC p-values correspond to using the empirical cumulative
distribution function (ECDF) of
$({Y_{i}^{\text{t}}}-{Y_{i}^{\text{c}}})_{i\in\mathcal{J}}$ for
$\mathcal{J}=\mathcal{I}_{\text{nc}}$. To investigate the performance of each
empirical null, we used the Kolmogorov-Smirnov and Anderson-Darling tests to
assess whether the p-values that are between 0.5 and 0.99 are approximately
uniformly distributed.

Table 3: A comparison of different choices of the empirical null distribution. NC: Negative controls; KS test: p-value of the Kolmogorov-Smirnov test of uniformity; AD test: p-value of the Anderson-Darling test of uniformity; BH rejections: number of rejections using the BH procedure with target FDR level $q=0.2$. Statistics | Method | Empirical null | KS test | AD test |  BH rejections
---|---|---|---|---|---
Testing ($\mathcal{I}$) | MAD1 | $\text{N}(0,0.4)$ | $<10^{-16}$ | $1.9\times 10^{-6}$ | $102$
| MAD2 | $\text{N}(0,0.3)$ | $3.3\times 10^{-16}$ | $2.0\times 10^{-6}$ | $144$
| Efron | $\text{N}(-0.02,0.22)$ | $1.8\times 10^{-5}$ | $2.5\times 10^{-5}$ | $200$
All ($\mathcal{I}\cup\mathcal{I}_{\text{nc}}$) | MAD1 | $\text{N}(0,0.39)$ | $<10^{-16}$ | $1.9\times 10^{-6}$ | $103$
| MAD2 | $\text{N}(0,0.23)$ | $1.5\times 10^{-5}$ | $5.3\times 10^{-6}$ | $182$
| Efron | $\text{N}(-0.05,0.21)$ | $4.5\times 10^{-5}$ | $3.5\times 10^{-5}$ | $234$
NC ($\mathcal{I}_{\text{nc}}$) | MAD1 | $\text{N}(0,0.4)$ | $<10^{-16}$ | $1.9\times 10^{-6}$ | $102$
| MAD2 | $\text{N}(0,0.2)$ | $0.004$ | $0.002$ | $211$
| Efron | $\text{N}(-0.02,0.18)$ | $0.039$ | $0.023$ | $240$
| ECDF (RANC) |  | $0.019$ | $0.063$ | $214$

Table 3 reports the empirical null obtained by each method above, the p-values
from the two tests of uniformity, and the number of rejections made by the BH
procedure ($q=0.2$). Among these empirical nulls, only Efron’s method applied
to negative controls and the RANC p-values procedure produce p-values whose
bulk are nearly uniformly distributed; see also Figure C.3 in the Appendix.
Between these methods, Efron’s method is more aggressive and produces more
rejections. However, a normal distribution does not seem to fit the
distribution of negative controls very well; see Figure C.4 in the Appendix.

##  8 Discussion

Motivated by a real proteomic data anslysis, we have suggested three model-
free methods for simultaneous hypothesis testing using internal negative
controls. They can be used to control various multiple testing error rates and
have appealing theoretical properties. Moreover, these methods offer
competitive practical performane as long as there are a decent number of
number of negative controls. Another attractive property is that the methods
proposed here are all invariant to monotone transformations of the data and
requires no subjective evaluation of the goodness-of-fit of the empirical null
distribution.

As mentioned in the Introduction, several recent articles have proposed
similar ideas about using negative controls in multiple testing [3, 41, 53].
These proposals arise from different applied domains and different
terminologies are used. An advantage to use “negative control” to refer to the
observations that resemble the null is that negative control is an integral
component of scientific methods and the nature of the method can be
immediately understood by across different contexts. As a consequence, it is
straightforward to understand the assumptions involved and assess them in
practice. In fact, this prompted us to develop weaker theoretical conditions
that allow dependent test statistics (see Remark 3) or misspecfified negative
controls (see Remark 8).

Of course, the statistical power of RANC p-values depend closely on the
quantity and quality of the internal negative controls. With a moderate number
of negative controls, the BH procedure applied to the RANC p-values rejects
almost as many hypotheses as the BH procedure applied to the “oracle”
p-values, which are calculated using the unknown true null distribution. When
the signals are strong, the FDR level is $q=0.2$, and the proportion of non-
nulls is $n_{1}/n=0.1$, a good rule of thumb for the number of negatve
controls is $m\geq 2{n}/(q{n_{1}})=100$; see also [41] for related discussion
on the number of negative controls.

Although the RANC p-value is robust to certain misclassifications of the
negative controls, using too many negative controls of poor quality may lead
to low power. Additionally, one can falsify the crucial exchangeability
assumption by considering subgroups of negative controls defined by domain
knowledge; see Figure 5 in Section 7 for an illustration of this idea using
the proteomic dataset.

For convenience, we have assumed throughout the article that there are no ties
among the test statistics. Although we regard this assumption as
inconsequential for most practical applications, one may also consider
randomly breaking ties in the definition of RANC p-values. It can be easily
shown that Proposition 3.1 still holds. However, it remains unclear if Theorem
3.1 holds as our proof assumes that the test statistics have continuous
distributions. Also, the empirical process argument in Section 4 does not
directly extend because the process ${V_{\text{nc}}}(t)$ may have large jumps
in the presence of ties. See the supplementary materials of [3] for related
discussion.

## Acknowledgements

Qingyuan Zhao and Zijun Gao are partly supported by EPSRC (grant
EP/V049968/1). We thank Jing Ren and Zora Chan for bringing the proteomic
application to us and Jiefu Li for sharing the dataset analyzed in Sections
1.1 and 7. We thank Rajen Shah and Richard Samworth for helpful comments and
Aaditya Ramdas for pointing us to the conformal inference literature and in
particular the paper by [3].

## Data and computer programs

The data and computer programs to reproduce the analysis and figures are
available at https://github.com/ZijunGao.

## References

  * [1] Anastasios N Angelopoulos and Stephen Bates  “A gentle introduction to conformal prediction and distribution-free uncertainty quantification”  In _arXiv preprint arXiv:2107.07511_ , 2021
  * [2] Yannick Baraud  “Bounding the expectation of the supremum of an empirical process over a (weak) VC-major class”  In _Electronic journal of statistics_ 10.2 Institute of Mathematical StatisticsBernoulli Society, 2016, pp. 1709–1728
  * [3] Stephen Bates et al.  “Testing for Outliers With Conformal P-Values” to appear  In _Annals of Statistics_ , 2021  arXiv: http://arxiv.org/abs/2104.08279v3
  * [4] Yoav Benjamini and Yosef Hochberg  “Controlling the false discovery rate: a practical and powerful approach to multiple testing”  In _Journal of the Royal statistical society: series B (Methodological)_ 57.1 Wiley Online Library, 1995, pp. 289–300
  * [5] Yoav Benjamini and Yosef Hochberg  “On the adaptive control of the false discovery rate in multiple testing with independent statistics”  In _Journal of educational and Behavioral Statistics_ 25.1 Sage Publications Sage CA: Los Angeles, CA, 2000, pp. 60–83
  * [6] Yoav Benjamini and Daniel Yekutieli  “The control of the false discovery rate in multiple testing under dependency”  In _The Annals of Statistics_ 29.4 Institute of Mathematical Statistics, 2001, pp. 1165–1188
  * [7] Frank Bretz, Willi Maurer, Werner Brannath and Martin Posch  “A graphical approach to sequentially rejective multiple test procedures”  In _Statistics in medicine_ 28.4 Wiley Online Library, 2009, pp. 586–604
  * [8] Persi Diaconis and David Freedman  “Finite exchangeable sequences”  In _The Annals of Probability_ 8.4, 1980, pp. 745–764
  * [9] Cécile Durot and Hendrik P Lopuhaä  “Limit theory in monotone function estimation”  In _Statistical Science_ 33.4 JSTOR, 2018, pp. 547–567
  * [10] Aryeh Dvoretzky, Jack Kiefer and Jacob Wolfowitz  “Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator”  In _The Annals of Mathematical Statistics_ 27.3 JSTOR, 1956, pp. 642–669
  * [11] Bradley Efron  “Large-scale inference: empirical Bayes methods for estimation, testing, and prediction”  Cambridge University Press, 2012
  * [12] Bradley Efron  “Large-scale simultaneous hypothesis testing: the choice of a null hypothesis”  In _Journal of the American Statistical Association_ 99.465 Taylor & Francis, 2004, pp. 96–104
  * [13] Bradley Efron, Robert Tibshirani, John D Storey and Virginia Tusher  “Empirical Bayes analysis of a microarray experiment”  In _Journal of the American statistical association_ 96.456 Taylor & Francis, 2001, pp. 1151–1160
  * [14] Ronald Aylmer Fisher  “Statistical methods for research workers”  Edinburgh, UK: OliverBoyd, 1925
  * [15] Johann A Gagnon-Bartsch and Terence P Speed  “Using control genes to correct for unwanted variation in microarray data”  In _Biostatistics_ 13.3 Oxford University Press, 2012, pp. 539–552
  * [16] Christopher Genovese and Larry Wasserman  “A stochastic process approach to false discovery control”  In _The annals of statistics_ 32.3 Institute of Mathematical Statistics, 2004, pp. 1035–1061
  * [17] Christopher R Genovese and Larry Wasserman  “Exceedance Control of the False Discovery Proportion”  In _Journal of the American Statistical Association_ 101.476, 2006, pp. 1408–1417  DOI: 10.1198/016214506000000339
  * [18] Jelle J Goeman, Jesse Hemerik and Aldo Solari  “Only closed testing procedures are admissible for controlling false discovery proportions”  In _The Annals of Statistics_ 49.2 Institute of Mathematical Statistics, 2021, pp. 1218–1238
  * [19] Jelle J Goeman and Aldo Solari  “Multiple testing for exploratory research”  In _Statistical Science_ 26.4 Institute of Mathematical Statistics, 2011, pp. 584–597
  * [20] Ulf Grenander  “On the theory of mortality measurement: part ii”  In _Scandinavian Actuarial Journal_ 1956.2 Taylor & Francis, 1956, pp. 125–153
  * [21] Jacklyn N. Hellwege et al.  “Population Stratification in Genetic Association Studies”  In _Current Protocols in Human Genetics_ 95.1, 2017  DOI: 10.1002/cphg.48
  * [22] Nicolas W Hengartner and Philip B Stark  “Finite-sample confidence envelopes for shape-restricted densities”  In _The Annals of Statistics_ 23.2 JSTOR, 1995, pp. 525–550
  * [23] Yosef Hochberg  “A Sharper Bonferroni Procedure for Multiple Tests of Significance”  In _Biometrika_ 75.4, 1988, pp. 800–802  DOI: 10.1093/biomet/75.4.800
  * [24] Yosef Hochberg and Yoav Benjamini  “More powerful procedures for multiple significance testing”  In _Statistics in medicine_ 9.7 Wiley Online Library, 1990, pp. 811–818
  * [25] Paul W Holland and Paul R Rosenbaum  “Conditional association and unidimensionality in monotone latent variable models”  In _The Annals of Statistics_ 14.4 JSTOR, 1986, pp. 1523–1543
  * [26] Sture Holm  “A simple sequentially rejective multiple test procedure”  In _Scandinavian journal of statistics_ 6.2 JSTOR, 1979, pp. 65–70
  * [27] G. Hommel  “A Stagewise Rejective Multiple Test Procedure Based on a Modified Bonferroni Test”  In _Biometrika_ 75.2, 1988, pp. 383–386  DOI: 10.1093/biomet/75.2.383
  * [28] Victoria Hung et al.  “Proteomic Mapping of the Human Mitochondrial Intermembrane Space in Live Cells Via Ratiometric Apex Tagging”  In _Molecular Cell_ 55.2, 2014, pp. 332–341  DOI: 10.1016/j.molcel.2014.06.003
  * [29] Jiashun Jin and T Tony Cai  “Estimating the null and the proportion of nonnull effects in large-scale multiple comparisons”  In _Journal of the American Statistical Association_ 102.478 Taylor & Francis, 2007, pp. 495–506
  * [30] Samuel Karlin and Yosef Rinott  “Classes of orderings of measures and related correlation inequalities. I. Multivariate totally positive distributions”  In _Journal of Multivariate Analysis_ 10.4 Elsevier, 1980, pp. 467–498
  * [31] Jeffrey T Leek and John D Storey  “Capturing Heterogeneity in Gene Expression Studies By Surrogate Variable Analysis”  In _PLoS Genetics_ 3.9, 2007, pp. e161  DOI: 10.1371/journal.pgen.0030161
  * [32] Jeffrey T. Leek et al.  “Tackling the Widespread and Critical Impact of Batch Effects in High-Throughput Data”  In _Nature Reviews Genetics_ 11.10, 2010, pp. 733–739  DOI: 10.1038/nrg2825
  * [33] Erich Leo Lehmann and Joseph P Romano  “Generalizations of the familywise error rate”  In _The Annals of Statistics_ 33.3 Institute of Mathematical Statistics, 2005, pp. 1138–1154
  * [34] Jiefu Li et al.  “Cell-surface proteomic profiling in the fly brain uncovers wiring regulators”  In _Cell_ 180.2 Elsevier, 2020, pp. 373–386
  * [35] Ziyi Liang, Matteo Sesia and Wenguang Sun  “Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers”  In _arXiv preprint arXiv:2208.11111_ , 2022
  * [36] Katrice A Lippa et al.  “Exploring the use of internal and externalcontrols for assessing microarray technical performance”  In _BMC Research Notes_ 3.349 BioMed Central, 2010, pp. 1–14
  * [37] Marc Lipsitch, Eric J Tchetgen Tchetgen and Ted Cohen  “Negative controls: a tool for detecting confounding and bias in observational studies”  In _Epidemiology_ 21.3 NIH Public Access, 2010, pp. 383–388
  * [38] Jennifer Listgarten et al.  “A powerful and efficient set test for genetic markers that handles confounders”  In _Bioinformatics_ 29.12 Oxford University Press, 2013, pp. 1526–1533
  * [39] Ariane Marandon, Lihua Lei, David Mary and Etienne Roquain  “Machine Learning Meets False Discovery Rate”  In _CoRR_ , 2022  arXiv: http://arxiv.org/abs/2208.06685v2
  * [40] Ruth Marcus, Peritz Eric and K Ruben Gabriel  “On closed testing procedures with special reference to ordered analysis of variance”  In _Biometrika_ 63.3 Oxford University Press, 1976, pp. 655–660
  * [41] David Mary and Etienne Roquain  “Semi-Supervised Multiple Testing”  In _Electronic Journal of Statistics_ 16.2, 2022  DOI: 10.1214/22-ejs2050
  * [42] Pascal Massart  “Concentration inequalities and model selection: volume 1896 of Lecture Notes in Mathematics”  Springer, 2007
  * [43] Pascal Massart  “The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality”  In _The Annals of Probability_ 18.3 JSTOR, 1990, pp. 1269–1283
  * [44] Wang Miao, Zhi Geng and Eric J Tchetgen Tchetgen  “Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder”  In _Biometrika_ 105.4, 2018, pp. 987–993  DOI: 10.1093/biomet/asy038
  * [45] David A Nix, Samir J Courdy and Kenneth M Boucher  “Empirical methods for controlling false positives and estimating confidence in ChIP-Seq peaks”  In _BMC bioinformatics_ 9.523 Springer, 2008, pp. 1–9
  * [46] Matthew M Parks, Benjamin J Raphael and Charles E Lawrence  “Using controls to limit false discovery in the era of big data”  In _BMC bioinformatics_ 19.1 Springer, 2018, pp. 1–8
  * [47] Alkes L Price et al.  “Principal Components Analysis Corrects for Stratification in Genome-Wide Association Studies”  In _Nature Genetics_ 38.8, 2006, pp. 904–909  DOI: 10.1038/ng1847
  * [48] Sanat K Sarkar and Chung-Kuei Chang  “The Simes method for multiple hypothesis testing with positively dependent test statistics”  In _Journal of the American Statistical Association_ 92.440 Taylor & Francis, 1997, pp. 1601–1608
  * [49] Tore Schweder and Eil Spjøtvoll  “Plots of p-values to evaluate many tests simultaneously”  In _Biometrika_ 69.3 Oxford University Press, 1982, pp. 493–502
  * [50] S Andrew Shuster et al.  “In situ cell-type-specific cell-surface proteomic profiling in mice”  In _Neuron_ 110 Elsevier, 2022, pp. 1–14
  * [51] R John Simes  “An improved Bonferroni procedure for multiple tests of significance”  In _Biometrika_ 73.3 Oxford University Press, 1986, pp. 751–754
  * [52] Matthew Slattery et al.  “Cofactor binding evokes latent differences in DNA binding specificity between Hox proteins”  In _Cell_ 147.6 Elsevier, 2011, pp. 1270–1282
  * [53] Jake A. Soloff, Daniel Xiang and William Fithian  “The Edge of Discovery: Controlling the Local False Discovery Rate At the Margin”, 2022  arXiv: http://arxiv.org/abs/2207.07299v1
  * [54] Jun S Song et al.  “Model-based analysis of two-color arrays (MA2C)”  In _Genome biology_ 8.8 BioMed Central, 2007, pp. 1–13
  * [55] John D Storey  “A direct approach to false discovery rates”  In _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ 64.3 Wiley Online Library, 2002, pp. 479–498
  * [56] John D Storey, Jonathan E Taylor and David Siegmund  “Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach”  In _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ 66.1 Wiley Online Library, 2004, pp. 187–205
  * [57] Wenguang Sun and T Tony Cai  “Oracle and adaptive compound decision rules for false discovery rate control”  In _Journal of the American Statistical Association_ 102.479 Taylor & Francis, 2007, pp. 901–912
  * [58] Jan WH Swanepoel  “The limiting behavior of a modified maximal symmetric $2s$-spacing with applications”  In _The Annals of Statistics_ 27.1 Institute of Mathematical Statistics, 1999, pp. 24–35
  * [59] Eric J Tchetgen Tchetgen et al.  “An Introduction To Proximal Causal Learning”, 2020  arXiv: http://arxiv.org/abs/2009.10982v1
  * [60] Vladimir Vovk, Alexander Gammerman and Glenn Shafer  “Algorithmic learning in a random world”  Springer, 2005
  * [61] Jingshu Wang, Qingyuan Zhao, Trevor Hastie and Art B. Owen  “Confounder Adjustment in Multiple Hypothesis Testing”  In _The Annals of Statistics_ 45.5, 2017  DOI: 10.1214/16-aos1511
  * [62] Jingshu Wang, Qingyuan Zhao, Trevor Hastie and Art B Owen  “Confounder adjustment in multiple hypothesis testing”  In _The Annals of Statistics_ 45.5 Institute of Mathematical Statistics, 2017, pp. 1863–1894
  * [63] Asaf Weinstein, Rina Barber and Emmanuel Candes  “A Power and Prediction Analysis for Knockoffs With Lasso Statistics”  In _CoRR_ , 2017  arXiv: http://arxiv.org/abs/1712.06465v1
  * [64] Ward Whitt  “Uniform conditional stochastic order”  In _Journal of Applied Probability_ 17.1 Cambridge University Press, 1980, pp. 112–123
  * [65] Brian L. Wiens  “A Fixed Sequence Bonferroni Procedure for Testing Multiple Endpoints”  In _Pharmaceutical Statistics_ 2.3, 2003, pp. 211–215  DOI: 10.1002/pst.64
  * [66] Yao Zhang and Qingyuan Zhao  “What is a randomization test?”  In _arXiv preprint arXiv:2203.10980_ , 2022
  * [67] Yong Zhang et al.  “Model-based analysis of ChIP-Seq (MACS)”  In _Genome biology_ 9.9 BioMed Central, 2008, pp. 1–9
  * [68] Qingyuan Zhao, Dylan S Small and Weijie Su  “Multiple testing when many p-values are uniformly conservative, with application to testing qualitative interaction in educational interventions”  In _Journal of the American Statistical Association_ 114.527 Taylor & Francis, 2019, pp. 1291–1304

##  Appendix A Technical proofs

###  A.1 Results in Section 3

###### Proof of Proposition 3.1.

Step 1. We prove the validity if ${F_{i}}(t)={F_{j}}(t)$ for all
$j\in{\mathcal{I}_{\text{nc}}}$ and $t\in\mathbb{R}$. By condition (b),
$({T_{j}})_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}$ is exchangeable. Since
there are no ties among $({T_{j}})_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}$
a.s., then the rank of ${T_{i}}$ among
$({T_{j}})_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}$ is uniformly
distributed on $\\{1,2,\ldots,1+m\\}$. By Definition (3), for $0\leq t\leq 1$,

$\displaystyle\mathbb{P}({p_{i}}\leq t)$
$\displaystyle=\mathbb{P}\left(\sum_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}\mathbbm{1}_{\\{{T_{j}}\leq{T_{i}}\\}}\leq
t(1+m)\right)$ $\displaystyle=\sum_{k=1}^{\lfloor
t(1+m)\rfloor}\mathbb{P}\left(\sum_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}\mathbbm{1}_{\\{{T_{j}}\leq{T_{i}}\\}}=k\right)=\frac{\lfloor
t(1+m)\rfloor}{1+m}\leq t.$

If we break ties randomly,
$({T_{j}})_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}$ is still exchangeable
and the rank of $({T_{j}})_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}$ is a
permutation of $1$, $2$, $\ldots$, $m+1$. The marginal distribution of the
rank of ${T_{i}}$ is still uniform and the proof holds.

Step 2. We prove the validity if ${F_{i}}(t)\leq{F_{j}}(t)$ for all
$j\in{\mathcal{I}_{\text{nc}}}$ and $t\in\mathbb{R}$. By condition (b) and
step 1, the RANC p-value ${p_{i}}^{\prime}$ based on
$({F_{j}}({T_{j}}))_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}$ is valid.
Define ${T_{j}}^{\prime}={F_{j}}^{-1}({F_{j}}({T_{j}}))$, where
${F_{j}}^{-1}(p):=\inf\\{t:{F_{j}}(t)\geq p\\}$. Then
${T_{j}}^{\prime}={F_{j}}^{-1}({F_{j}}({T_{j}}^{\prime}))$,
${T_{j}}^{\prime}\leq{T_{j}}$, and $\mathbb{P}({T_{j}}^{\prime}<{T_{j}})=0$.
On $\\{{T_{j}}^{\prime}={T_{j}}\\}$, by condition (a),

$\displaystyle{F_{j}}({T_{j}}^{\prime})\leq{F_{i}}({T_{i}})\implies{F_{j}}({T_{j}}^{\prime})\leq{F_{j}}({T_{i}})\implies{T_{j}}^{\prime}={F_{j}}^{-1}({F_{j}}({T_{j}}^{\prime}))\leq{F_{j}}^{-1}{F_{j}}({T_{i}})\leq{T_{i}}.$

Therefore,

$\displaystyle\mathbb{P}({p_{i}}\leq t)$
$\displaystyle=\mathbb{P}\left(\sum_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}\mathbbm{1}_{\\{{T_{j}}^{\prime}\leq{T_{i}}\\}}\leq
t(1+m)\right)$
$\displaystyle\leq\mathbb{P}\left(\sum_{j\in\\{i\\}\cup{\mathcal{I}_{\text{nc}}}}\mathbbm{1}_{\\{{F_{j}}({T_{j}}^{\prime})\leq{F_{i}}({T_{i}})\\}}\leq
t(1+m)\right)=\mathbb{P}\left({p_{i}}^{\prime}\leq t\right)\leq t.$

∎

###### Proof of Theorem 3.1.

Without loss of generality, we assume
${\mathcal{I}_{0}}=\\{1,\dotsc,{n_{0}}\\}$ and
${\mathcal{I}_{\text{nc}}}=\\{{n}+1,\dotsc,{n}+m\\}$. When ${n_{0}}>0$,
${T_{1}}$ corresponds to a true null. Under both conditions in Theorem 3.1,
${F_{i}}={F_{j}}$ for all
$i,j\in{\mathcal{I}_{0}}\cup{\mathcal{I}_{\text{nc}}}$. Without loss of
generality, we assume the null distribution is $U[0,1]$ so ${F_{i}}(t)=t$ for
all $0\leq t\leq 1$ and $i\in{\mathcal{I}_{0}}\cup{\mathcal{I}_{\text{nc}}}$;
otherwise, we can replace all statistics ${T_{i}}$ by ${F_{1}}({T_{i}})$.
Since the PRDS property is invariant by co-monotone transformations, with a
slight abuse of notation, it suffices to prove the PRDS property is satisfied
by the unnormalized ranks

$\displaystyle{p_{i}}:=\sum_{j\in{\mathcal{I}_{\text{nc}}}}\mathbbm{1}_{\\{{T_{j}}\leq{T_{i}}\\}},~{}i\in{\mathcal{I}}.$

Under both sets of conditions,
$({T_{i}})_{i\in\\{1\\}\cup{\mathcal{I}_{\text{nc}}}}$ is exchangeable.
Therefore, for any $0\leq k\leq m-1$,
$\mathbb{P}({p_{1}}=k)=\mathbb{P}({p_{1}}=k+1)=1/(1+m)$. Then it suffices to
show

$\displaystyle\mathbb{P}\left({\bm{p}}\in\mathcal{D},~{}{p_{1}}=k\right)\leq\mathbb{P}\left({\bm{p}}\in\mathcal{D},~{}{p_{1}}=k+1\right)$

for all fixed $0\leq k\leq m-1$ and increasing set
$\mathcal{D}\subseteq\mathbb{R}^{{n}+m}$.

Let $\bm{C}=(C_{1},\dotsc,C_{m})=(T_{{n}+1},\dotsc,T_{{n}+m})$ denote all
negative control statistics. Let $C_{(1)}<\dotsb<C_{(m)}$ be the order
statistics. Let $\bm{C}_{-(k+1)}$ denote all negative control test statistics
excluding $C_{(k+1)}$. It suffices to show that

$\displaystyle\mathbb{P}\left({\bm{p}}\in\mathcal{D},~{}{p_{1}}=k\mid\bm{C}_{-(k+1)}=\bm{c}_{-(k+1)}\right)\leq\mathbb{P}\left({\bm{p}}\in\mathcal{D},~{}{p_{1}}=k+1\mid\bm{C}_{-(k+1)}=\bm{c}_{-(k+1)}\right)$
(14)

for all $\bm{c}_{-(k+1)}$. As a convention, let $c_{(0)}=0$ and $c_{(m+1)}=1$.
Define the bi-variate function

$\displaystyle
f(t,c)=\mathbb{P}\left({\bm{p}}\in\mathcal{D}\mid\bm{C}_{-(k+1)}=\bm{c}_{-(k+1)},C_{(k+1)}=c,{T_{1}}=t\right)\quad\text{for}~{}c_{(k)}<c,t<c_{(k+2)}.$

Let $g(c,t)$ denote the density function of $(C_{(k+1)},T_{1})$ given
$\bm{C}_{-(k+1)}=\bm{c}_{-(k+1)}$ and $c_{(k)}<T_{1}<c_{(k+2)}$. To prove
(14), it suffices to show that

$\int_{c_{(k)}}^{c_{(k+2)}}\int_{c_{(k)}}^{c_{(k+2)}}f(t,c)g(t,c)\mathbbm{1}_{\\{t<c\\}}\,dt\,dc\leq\int_{c_{(k)}}^{c_{(k+2)}}\int_{c_{(k)}}^{c_{(k+2)}}f(t,c)g(t,c)\mathbbm{1}_{\\{t>c\\}}\,dt\,dc.$

This follows from the next three Lemmas.

######  Lemma A.1.

The function $g$ is symmetric, i.e. $g(t,c)=g(c,t)$ for all
$c_{(k)}<c<t<c_{(k+2)}$.

######  Lemma A.2.

$f(c,t)\leq f(t,c)$ for all $c_{(k)}<c<t<c_{(k+2)}$.

######  Lemma A.3.

Let $h:[0,1]^{2}\to\mathbb{R}$ be a bi-variate function and $h(x,y)\leq
h(y,x)$ for all $0<x<y<1$. Then

$\displaystyle\int_{0}^{1}\int_{0}^{1}h(x,y)\mathbbm{1}_{\\{x<y\\}}\,dx\,dy\leq\int_{0}^{1}\int_{0}^{1}h(x,y)\mathbbm{1}_{\\{x>y\\}}\,dx\,dy$

###### Proof of Lemma A.1.

This is obviously true under the exchangeability condition in (ii). For the
first set of conditions, the independence conditions in (i)(i.b) and (i)(i.c)
imply that

$\left(C_{(k+1)},T_{1}\right)\mid\bm{C}_{-(k+1)}=\bm{c}_{-(k+1)}\sim
U\left(\left[c_{(k)},c_{(k+2)}\right]^{2}\right),$

where $U\left(\left[c_{(k)},c_{(k+2)}\right]^{2}\right)$ is the uniform
distribution over the square $\left[c_{(k)},c_{(k+2)}\right]^{2}$. ∎

###### Proof of Lemma A.2.

Under the first set of conditions, we claim that $f(t,c)$ is decreasing in $c$
and increasing in $t$, so the desired conclusion follows. The first
observation follows from the definition of $\bm{p}$ and the assumption that
$\mathcal{D}$ is increasing. The second claim follows from the PRDS property
in condition (i)(i.d). To see this, given $\bm{C}=\bm{c}$, the event
$\bm{p}\in\mathcal{D}$ can be rewritten as an event
$\bm{T}\in\mathcal{D}^{\prime}$ where
$\mathcal{D}^{\prime}=\mathcal{D}^{\prime}(\bm{c})$ is an increasing set that
depends on $\bm{c}$. Using conditions (i)(i.b) and (i)(i.d), we have

$\displaystyle f(t,c)=$
$\displaystyle\mathbb{P}\left({\bm{p}}\in\mathcal{D}\mid\bm{C}_{-(k+1)}=\bm{c}_{-(k+1)},C_{(k+1)}=c,{T_{1}}=t\right)$
$\displaystyle=$
$\displaystyle\mathbb{P}\left(\bm{T}\in\mathcal{D}^{\prime}(\bm{c})\mid\bm{C}_{-(k+1)}=\bm{c}_{-(k+1)},C_{(k+1)}=c,{T_{1}}=t\right)$
$\displaystyle=$
$\displaystyle\mathbb{P}\left(\bm{T}\in\mathcal{D}^{\prime}(\bm{c})\mid{T_{1}}=t\right)$

is increasing in $t$. Now consider the exchangeability condition (ii), which
implies that $(T_{2},\dotsc,T_{{n}})$ has the same conditional distribution
given

$\bm{C}_{-(k+1)}=\bm{c}_{-(k+1)},C_{(k+1)}=c,{T_{1}}=t,$

and given

$\bm{C}_{-(k+1)}=\bm{c}_{-(k+1)},C_{(k+1)}=t,{T_{1}}=c,$

for all $c_{(k)}<c,t<c_{(k+2)}$. The conclusion then follows from the fact
that the RANC p-values $\bm{p}$ only become smaller when we swap $C_{(k+1)}=c$
with $T_{1}=t$ if $c<t$ and the assumption that $\mathcal{D}$ is increasing. ∎

###### Proof of Lemma A.3.

The conclusion follows from rewritting one of the integrals as follows,

$\displaystyle\int_{0}^{1}\int_{0}^{1}h(x,y)\mathbbm{1}_{\\{x<y\\}}\,dx\,dy-\int_{0}^{1}\int_{0}^{1}h(x,y)\mathbbm{1}_{\\{x>y\\}}\,dx\,dy$
$\displaystyle=$
$\displaystyle\int_{0}^{1}\int_{0}^{1}h(x,y)\mathbbm{1}_{\\{x<y\\}}\,dx\,dy-\int_{0}^{1}\int_{0}^{1}h(y,x)\mathbbm{1}_{\\{y>x\\}}\,dx\,dy$
$\displaystyle=$
$\displaystyle\int_{0}^{1}\int_{0}^{1}(h(x,y)-h(y,x))\mathbbm{1}_{\\{x<y\\}}\,dx\,dy$
$\displaystyle\leq$ $\displaystyle 0.$

∎

∎

######  Proposition A.1.

If the permutation test statistic ${G(\cdot)}$ satisfies

  1. 1.

${G({\bm{T}})}={G(g({\bm{T}}))}$ for arbitrary strictly increasing function
$g(\cdot):\mathbb{R}\to\mathbb{R}$,
$g({\bm{T}})=(g({T_{i}}))_{i\in{\mathcal{I}}\cup{\mathcal{I}_{\text{nc}}}}$;

  2. 2.

${G({\bm{T}})}={G({\bm{T}}_{P})}$, where ${\bm{T}}_{P}$ is a permutation of
${\bm{T}}$ where $({T_{i}})_{i\in{\mathcal{I}}}$ are not exchanged with
$({T_{j}})_{j\in{\mathcal{I}_{\text{nc}}}}$,

then ${G({\bm{T}})}$ is a function of the set of RANC p-values
$\\{{p_{i}}\\}_{i\in{\mathcal{I}}}$.

###### Proof of Proposition A.1.

The mapping from ${\bm{T}}$ with no ties to its rank $R({\bm{T}})$ is a
strictly increasing function. Since ${G({\bm{T}})}={G(g({\bm{T}}))}$ for
arbitrary strictly increasing function $g(\cdot)$, then
${G({\bm{T}})}={G(R({\bm{T}}))}$. For ${\bm{T}}$ and ${\bm{T}}^{\prime}$, we
can always permute $({T_{i}})_{i\in{\mathcal{I}}}$ so that the ordering within
$({T_{i}})_{i\in{\mathcal{I}}}$ equals that of
$({T_{i}}^{\prime})_{i\in{\mathcal{I}}}$. Similarly for
$({T_{i}})_{i\in{\mathcal{I}_{\text{nc}}}}$. Denote the permuted test
statistics by ${\bm{T}}_{P}$, and then
$\\{{p_{i}}\\}_{i\in{\mathcal{I}}}=\\{{p_{P,i}}\\}_{i\in{\mathcal{I}}}$. Since
${G(\cdot)}$ is invariant to permutations within
$({T_{i}})_{i\in{\mathcal{I}}}$ and
$({T_{i}})_{i\in{\mathcal{I}_{\text{nc}}}}$, then
${G({\bm{T}})}={G({\bm{T}}_{P})}$. Note that
$\\{{p_{i}}\\}_{i\in{\mathcal{I}}}=\\{{p_{P,i}}\\}_{i\in{\mathcal{I}}}=\\{{p_{i}}^{\prime}\\}_{i\in{\mathcal{I}}}$
implies $R({\bm{T}}_{P})=R({\bm{T}}^{\prime})$, and further
${G({\bm{T}})}={G({\bm{T}}_{P})}={G(R({\bm{T}}_{P}))}={G(R({\bm{T}}^{\prime}))}={G({\bm{T}}^{\prime})}$.
Therefore, ${G({\bm{T}})}$ is a function of
$\\{{p_{i}}\\}_{i\in{\mathcal{I}}}$. ∎

###  A.2 Results in Section 4

The next proposition suggests that $\widehat{\text{FDR}}_{\lambda}(t)$ is a
conservative estimate of $\text{FDR}(t)$. The proposition is valid for
replacing the $+2$ by $+1$ in the numerator of
$\widehat{\text{FDR}}_{\lambda}(t)$.

######  Proposition A.2.

Assume $({T_{i}})_{i\in{\mathcal{I}}\cup{\mathcal{I}_{\text{nc}}}}$ are
independent and ${T_{i}}\stackrel{{\scriptstyle d}}{{=}}{C_{j}}$ for
$i\in{\mathcal{I}_{0}}$, $j\in{\mathcal{I}_{\text{nc}}}$, then
$\mathbb{E}\left[\widehat{\text{FDR}}_{\lambda}(t)\right]\geq\text{FDR}(t)$
when $\lambda=1$.

###### Proof of Proposition A.2.

For $0<\lambda<1$, $\mathbb{P}(V_{\text{nc}}(\lambda)=m)>0$ and
$\mathbb{E}\left[\widehat{\text{FDR}}_{\lambda}(t)\right]=\infty>\text{FDR}(t)$.
Thus, it is left to show the inequality is true for $\lambda=1$. For
simplicity, we write $\widehat{\text{FDR}}_{1}(t)$ as
$\widehat{\text{FDR}}(t)$. We denote the distribution of true null and
negative control test statistics by ${F_{0}}(t)$. Since $V_{\text{nc}}(t)\leq
m$,

$\displaystyle\mathbb{E}\left[\frac{V_{\text{nc}}(t)+1}{m+1}\right]\geq\mathbb{E}\left[\frac{V_{\text{nc}}(t)}{m}\right]={F_{0}}(t).$
(15)

The result in [56, Theorem 1] of p-values following $U[0,1]$ can extend to
test statistics of arbitrary distribution with continuous CDF,

$\displaystyle\mathbb{E}\left[\frac{{n_{0}}{F_{0}}(t)-V(t)}{R(t)\vee
1}\right]\geq 0.$ (16)

Since the negative control test statistics are independent of the test
statistics under investigation and ${n}\geq{n_{0}}$,

$\displaystyle\mathbb{E}\left[\widehat{\text{FDR}}(t)\right]$
$\displaystyle=\mathbb{E}\left[\mathbb{E}\left[\widehat{\text{FDR}}(t)\mid
R(t)\right]\right]=\mathbb{E}\left[\frac{{n}\mathbb{E}\left[\frac{V_{\text{nc}}(t)+1}{m+1}\mid
R(t)\right]}{R(t)\vee 1}\right]$
$\displaystyle\geq\mathbb{E}\left[\frac{{n}{F_{0}}(t)}{R(t)\vee
1}\right]\geq\mathbb{E}\left[\frac{V(t)}{R(t)\vee 1}\right]=\text{FDR}(t).$

∎

###### Proof of Proposition 4.1.

Let ${T_{q}}^{\prime}$, ${T_{q}}$ be the largest test statistic under
investigation rejected by applying the BH procedure to the RANC p-values and
the empirical-process-based step-up procedure, respectively. It is
straightforward to show the event that ${T_{q}}^{\prime}$ does not exist is
equivalent to the event that ${T_{q}}$ does not exist. It is left to show the
equivalence holds when both ${T_{q}}^{\prime}$ and ${T_{q}}$ are well-defined.
On one hand,

$\displaystyle\frac{V_{\text{nc}}({T_{q}}^{\prime})+2}{m+1}\wedge
1\leq\frac{R({T_{q}}^{\prime})}{{n}}q\implies\frac{{n}\frac{V_{\text{nc}}({T_{q}}^{\prime})+2}{m+1}}{R({T_{q}}^{\prime})\vee
1}\leq q\implies{\tau_{q}}\geq{T_{q}}^{\prime},$

which further implies ${T_{q}}\geq{T_{q}}^{\prime}$ by the definition of
${T_{q}}$. On the other hand, by the discussion before Proposition 4.1, there
exists $\varepsilon>0$ such that ${\tau_{q}}-\varepsilon>{T_{q}}$,
$V_{\text{nc}}({\tau_{q}}-\varepsilon)\geq V_{\text{nc}}({T_{q}})$,
$R({\tau_{q}}-\varepsilon)=R({T_{q}})$, and
$\widehat{\text{FDR}}({\tau_{q}}-\varepsilon)\leq q$. Then,

$\displaystyle\widehat{\text{FDR}}({T_{q}})=\frac{{n}\cdot\frac{V_{\text{nc}}({T_{q}})+2}{m+1}}{R({T_{q}})\vee
1}\leq\frac{{n}\cdot\frac{V_{\text{nc}}({\tau_{q}}-\varepsilon)+2}{m+1}}{R({\tau_{q}}-\varepsilon)\vee
1}=\widehat{\text{FDR}}({\tau_{q}}-\varepsilon)\leq q.$

Therefore, ${T_{q}}\leq{T_{q}}^{\prime}$.

∎

###### Proof of Theorem 4.1.

Without loss of generality, we assume the test statistics are supported on
$(0,1)$. Otherwise, we can always apply the transformation
$t\to\frac{e^{t}}{1+e^{t}}$. Define a decreasing family of $\sigma$-algebras
$\mathcal{F}_{t}=\sigma({V}(s),{S}(s),{V_{\text{nc}}}(s),~{}t\leq s\leq 1)$
for $0\leq t\leq 1$. By definition, $M(t)$ in Eq. (8) is measurable with
respect to $\mathcal{F}_{t}$, $0\leq M(t)\leq(m+1){n_{0}}$, $M(0)=0$, and
$M(1)={n_{0}}$.

We show $M(t)$ is a backward super-martingale with respect to
$\\{\mathcal{F}_{t}\\}$. For $0\leq s<t$, by condition (b), $({V}(s),{V}(t))$,
$({V}_{\text{nc}}(s),{V}_{\text{nc}}(t))$, and $({S}(s),{S}(t))$ are mutually
independent. On $\left\\{{V}(t)=ck,{V_{\text{nc}}}(t)+1=c\right\\}$ with non-
zero probability,

$\displaystyle\begin{split}\mathbb{E}\left[\frac{{V}(s)}{{V_{\text{nc}}}(s)+1}\mid\mathcal{F}_{t}\right]&=\mathbb{E}\left[\frac{{V}(s)}{{V_{\text{nc}}}(s)+1}\mid{V}(t)=ck,~{}{V_{\text{nc}}}(t)+1=c\right]\\\
&=\underbrace{\mathbb{E}\left[V(s)\mid
V(t)=ck\right]}_{:=\text{(I)}}\cdot\underbrace{\mathbb{E}\left[\frac{1}{{V_{\text{nc}}}(s)+1}\mid{V_{\text{nc}}}(t)+1=c\right]}_{:=\text{(II)}}.\end{split}$
(17)

By condition (a), there exists $0\leq p_{t}^{s}\leq 1$ such that

$\displaystyle\max_{i\in{\mathcal{I}_{0}}}\frac{{F_{i}}(s)}{{F_{i}}(t)}\leq
p_{t}^{s}\leq\min_{j\in{\mathcal{I}_{\text{nc}}}}\frac{{F_{j}}(s)}{{F_{j}}(t)}.$
(18)

By condition (b), and Eq. (18),

$\displaystyle{V}(s)\mid{V}(t)$ $\displaystyle\lesssim Z,\quad
Z\sim\text{Binomial}({V}(t),p_{t}^{s}),$ (19)
$\displaystyle\frac{1}{1+{V_{\text{nc}}}(s)}\mid\frac{1}{1+{V_{\text{nc}}}(t)}$
$\displaystyle\lesssim\frac{1}{1+Z^{\text{nc}}},\quad
Z^{\text{nc}}\sim\text{Binomial}({V_{\text{nc}}}(t),p_{t}^{s}).$ (20)

For term (I) in Eq. (17), by Eq. (19),

$\displaystyle\text{(I)}\leq{V}(t)\cdot p_{t}^{s}=ckp_{t}^{s}.$ (21)

For term (II) in Eq. (17), by Eq. (20),

$\displaystyle\begin{split}\text{(II)}&\leq\sum_{i=0}^{c-1}\frac{1}{i+1}\binom{c-1}{i}(p_{t}^{s})^{i}(1-p_{t}^{s})^{c-1-i}\\\
&=\frac{1}{cp_{t}^{s}}\sum_{i=0}^{c-1}\binom{c}{i+1}(p_{t}^{s})^{i+1}(1-p_{t}^{s})^{c-1-i}=\frac{1}{cp_{t}^{s}}\left(1-(1-p_{t}^{s})^{c}\right).\end{split}$
(22)

Plug Eq. (21), Eq. (22) into Eq. (17),

$\displaystyle\mathbb{E}\left[\frac{{V}(s)}{{V_{\text{nc}}}(s)+1}\mid\mathcal{F}_{t}\right]\leq
ckp_{t}^{s}\cdot\frac{1}{cp_{t}^{s}}\left(1-(1-p_{t}^{s})^{c}\right)\leq
k=\frac{{V}(t)}{{V_{\text{nc}}}(t)+1}.$

Next, we show ${\tau_{q}}$ is a stopping time with respect to
$\\{\mathcal{F}_{t}\\}$. By definition, if $\lambda=1$, then for $0\leq t\leq
1$,

$\displaystyle\\{{\tau_{q}}>t\\}=\left\\{\sup\left\\{s\geq
0:\frac{\frac{{n}}{m+1}({V_{\text{nc}}}(s)+2)}{({V}(s)+{S}(s))\vee 1}\leq
q\right\\}>t\right\\},$

is $\mathcal{F}_{t}$ measurable. If $0<\lambda<1$, for $t\geq\lambda$,
$\\{{\tau_{q}}>t\\}=\emptyset$; for $t<\lambda$, $\hat{\pi}(\lambda)$ is
$\mathcal{F}_{t}$ measurable. Thus, ${\tau_{q}}$ is a stopping time regarding
$\\{\mathcal{F}_{t}\\}$.

Since $M(t)$ is bounded and thus uniformly integrable, we can apply the
optional stopping time theorem

$\displaystyle\mathbb{E}\left[M\left({\tau_{q}}\right)\mid\mathcal{F}_{1}\right]\leq
M(1)={n_{0}}.$ (23)

We next show

$\displaystyle\frac{{V}({\tau_{q}})}{({V}({\tau_{q}})+{S}({\tau_{q}}))\vee
1}\leq\frac{q}{\hat{\pi}(\lambda)}\cdot\frac{m+1}{{n}}\cdot\frac{{V}({\tau_{q}})}{{V_{\text{nc}}}({\tau_{q}})+1}.$
(24)

It suffices to discuss the case where ${\tau_{q}}>0$. Let
$\mathcal{S}_{q}=\left\\{0\leq
t\leq\lambda:\frac{\hat{\pi}(\lambda){n}\frac{V_{\text{nc}}(t)+2}{m+1}}{(V(t)+S(t))\vee
1}\leq q\right\\}$, then ${\tau_{q}}=\sup\mathcal{S}_{q}$. If
${\tau_{q}}\in\mathcal{S}_{q}$, then Eq. (24) is obviously true. If
${\tau_{q}}\notin\mathcal{S}_{q}$, then there exists a sequence
$(t_{k})\subseteq\mathcal{S}_{q}$ such that $t_{k}\to{\tau_{q}}$ as
$k\to\infty$, and ${V}(t_{k})+{S}(t_{k})\leq{V}({\tau_{q}})+{S}({\tau_{q}})$.
Since a Poisson process increases at most by one at a time, then
$\lim_{k\to\infty}{V_{\text{nc}}}(t_{k})+2\geq{V_{\text{nc}}}({\tau_{q}})+1$.
Therefore,

$\displaystyle q\cdot\frac{m+1}{{n}}$
$\displaystyle\geq\lim_{k\to\infty}\frac{\hat{\pi}(\lambda)({V_{\text{nc}}}(t_{k})+2)}{({V}(t_{k})+{S}(t_{k}))\vee
1}\geq\frac{\hat{\pi}(\lambda)\left({V_{\text{nc}}}\left({\tau_{q}}\right)+1\right)}{\left({V}\left({\tau_{q}}\right)+{S}\left({\tau_{q}}\right)\right)\vee
1}.$

Finally, we compute the FDR. By Eq. (24),

$\displaystyle\text{FDR}=\mathbb{E}\left[\frac{{V}({\tau_{q}})}{({V}({\tau_{q}})+{S}({\tau_{q}}))\vee
1}\mid\mathcal{F}_{1}\right]\leq
q\cdot\frac{m+1}{{n}}\cdot\mathbb{E}\left[\frac{1}{\hat{\pi}(\lambda)}\cdot\frac{{V}({\tau_{q}})}{{V_{\text{nc}}}({\tau_{q}})+1}\mid\mathcal{F}_{1}\right].$

By the tower property and Eq. (23),

$\displaystyle\mathbb{E}\left[\frac{1}{\hat{\pi}(\lambda)}\cdot\frac{{V}({\tau_{q}})}{{V_{\text{nc}}}({\tau_{q}})+1}\mid\mathcal{F}_{1}\right]=\mathbb{E}\left[\frac{1}{\hat{\pi}(\lambda)}\cdot\mathbb{E}\left[\frac{{V}({\tau_{q}})}{{V_{\text{nc}}}({\tau_{q}})+1}\mid\mathcal{F}_{\lambda}\right]\mid\mathcal{F}_{1}\right]\leq\mathbb{E}\left[\frac{1}{\hat{\pi}(\lambda)}\cdot\frac{{V}(\lambda)}{{V_{\text{nc}}}(\lambda)+1}\mid\mathcal{F}_{1}\right].$

If $\lambda=1$, then

$\displaystyle\text{FDR}\leq q\cdot\frac{m+1}{{n}}\cdot\frac{{n_{0}}}{m+1}\leq
q.$

If $0<\lambda<1$, we plug in the definition of $\hat{\pi}(\lambda)$ and by
condition (b),

FDR $\displaystyle\leq
q\cdot\mathbb{E}\left[\frac{m-{V_{\text{nc}}}(\lambda)}{{n}-{S}(\lambda)-{V}(\lambda)+1}\cdot\frac{{V}(\lambda)}{{V_{\text{nc}}}(\lambda)+1}\mid\mathcal{F}_{1}\right]$
$\displaystyle=q\cdot\mathbb{E}\left[\frac{m-{V_{\text{nc}}}(\lambda)}{{V_{\text{nc}}}(\lambda)+1}\mid\mathcal{F}_{1}\right]\cdot\mathbb{E}\left[\frac{{V}(\lambda)}{{n}-{S}(\lambda)-{V}(\lambda)+1}\mid\mathcal{F}_{1}\right].$

By a similar argument of Eq. (22),

$\displaystyle\text{FDR}\leq
q\cdot\frac{1-p_{1}^{\lambda}}{p_{1}^{\lambda}}(1-(1-p_{1}^{\lambda})^{m})\cdot\frac{p_{1}^{\lambda}}{1-p_{1}^{\lambda}}(1-(p_{1}^{\lambda})^{{n_{0}}})\leq
q.$

∎

######  Remark 13.

In the proof of Theorem 4.1, it may be tempting to use the alternative process

$\frac{{V}(t)}{({V_{\text{nc}}}(t)\vee 1)/m}$

as $M(t)$. However, this is not a super-martingale.

###  A.3 Results in Section 5

######  Proposition A.3.

For any ${F}$, ${F_{0}}$, and $0<\varepsilon<1$,

$\displaystyle\mathbb{P}\left({R_{\lambda}}({\hat{\tau}_{\lambda,{n},m}})-{R_{\lambda}}({\tau^{*}_{\lambda}})>2(1+\lambda)\sqrt{\frac{\log(4/\varepsilon)}{2(m\wedge{n})}}\right)\leq\varepsilon.$

###### Proof of Proposition A.3.

By the triangle inequality and the Dvoretzky–Kiefer–Wolfowitz (DKW) inequality
[10, 43], for any $\varepsilon^{\prime}>0$,

$\displaystyle\begin{split}&\quad~{}\mathbb{P}\left(\sup_{{t}\in\mathbb{R}}\left|{{\ell}_{\lambda,{n},m}}({t})-{R_{\lambda}}({t})\right|>(1+\lambda)\varepsilon^{\prime}\right)\\\
&\leq\mathbb{P}\left(\sup_{{t}\in\mathbb{R}}\left|{F_{0}}({t})-{{F}_{0,m}}({t})\right|>\varepsilon^{\prime}\right)+\mathbb{P}\left(\sup_{{t}\in\mathbb{R}}\left|\lambda{F}({t})-\lambda{{F}_{n}}({t})\right|>\lambda\varepsilon^{\prime}\right)\\\
&\leq 2e^{-2m\varepsilon^{\prime 2}}+2e^{-2{n}\varepsilon^{\prime 2}}\leq
4e^{-2(m\wedge{n})\varepsilon^{\prime 2}}.\end{split}$ (25)

Since ${\hat{\tau}_{\lambda,{n},m}}$ minimizes ${{\ell}_{\lambda,{n},m}}$,

$\displaystyle\begin{split}&\quad~{}{R_{\lambda}}({\hat{\tau}_{\lambda,{n},m}})-{R_{\lambda}}({\tau^{*}_{\lambda}})\\\
&=\left({R_{\lambda}}({\hat{\tau}_{\lambda,{n},m}})-{{\ell}_{\lambda,{n},m}}({\hat{\tau}_{\lambda,{n},m}})\right)+\left({{\ell}_{\lambda,{n},m}}({\hat{\tau}_{\lambda,{n},m}})-{{\ell}_{\lambda,{n},m}}({\tau^{*}_{\lambda}})\right)+\left({{\ell}_{\lambda,{n},m}}({\tau^{*}_{\lambda}})-{R_{\lambda}}({\tau^{*}_{\lambda}})\right)\\\
&\leq
2\sup_{{t}\in\mathbb{R}}\left|{{\ell}_{\lambda,{n},m}}({t})-{R_{\lambda}}({t})\right|.\end{split}$
(26)

Therefore, we combine Eq. (25), Eq. (26), and take
$\varepsilon^{\prime}=\sqrt{\frac{\log(4/\varepsilon)}{2(m\wedge{n})}}$,

$\displaystyle\quad~{}\mathbb{P}\left({R_{\lambda}}({\hat{\tau}_{\lambda,{n},m}})-{R_{\lambda}}({\tau^{*}_{\lambda}})>2(1+\lambda)\sqrt{\frac{\log(4/\varepsilon)}{2m\wedge{n}}}\right)$
$\displaystyle\leq\mathbb{P}\left(\sup_{{t}\in\mathbb{R}}\left|{{\ell}_{\lambda,{n},m}}({t})-{R_{\lambda}}({t})\right|>(1+\lambda)\sqrt{\frac{\log(4/\varepsilon)}{2(m\wedge{n})}}\right)=\varepsilon.$

∎

######  Proposition A.4.

Consider the setting in Section 5 and assume the following assumptions hold:

  1. 1.

${f}({\tau^{*}_{\lambda}})$, ${f_{0}}({\tau^{*}_{\lambda}})>0$ and
${f}^{\prime}({\tau^{*}_{\lambda}})$, ${f_{0}}^{\prime}({\tau^{*}_{\lambda}})$
exist;

  2. 2.

${f}({t})\leq{f_{\max}}$, ${f_{0}}({t})\leq{f_{0,\max}}$, in
$|{t}-{\tau^{*}_{\lambda}}|\leq D$ for some ${f_{\max}}$, ${f_{0,\max}}$,
$D>0$;

  3. 3.

the derivative $({f_{0}}/{f})^{\prime}({\tau^{*}_{\lambda}})$ is positive;

  4. 4.

for any $\delta>0$, there exists $\varepsilon_{\delta}>0$ such that
${F_{0}}({t})-\lambda{F}({t})>{F_{0}}({\tau^{*}_{\lambda}})-\lambda{F}({\tau^{*}_{\lambda}})+\varepsilon_{\delta}$
for $|{t}-{\tau^{*}_{\lambda}}|>\delta$.

Then there exists a constant $C^{*}>0$ such that with probability at least
$1-C^{*}/\log({n}\wedge m)$,

$\displaystyle\left|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}\right|\leq\frac{\log({n}\wedge
m)}{({n}\wedge m)^{1/3}}\quad\text{for all}~{}{n},m\geq 5.$ (27)

######  Lemma A.4.

Under Assumptions 1, 3, there exists $C^{\prime}>0$, $D^{\prime}>0$ such that
for any ${t}$ satisfying $\left|{t}-{\tau^{*}_{\lambda}}\right|\leq
D^{\prime}$,

$\displaystyle{R_{\lambda}}({\tau^{*}_{\lambda}})\leq{R_{\lambda}}({t})-C^{\prime}({t}-{\tau^{*}_{\lambda}})^{2}.$
(28)

###### Proof of Lemma A.4.

We perform Taylor expansion of ${R_{\lambda}}({t})$ at ${\tau^{*}_{\lambda}}$
with the Peano’s form of remainder,

$\displaystyle{R_{\lambda}}({t})$
$\displaystyle={R_{\lambda}}({\tau^{*}_{\lambda}})+{R_{\lambda}}^{\prime}({\tau^{*}_{\lambda}})({t}-{\tau^{*}_{\lambda}})+\frac{{R_{\lambda}}^{\prime\prime}({\tau^{*}_{\lambda}})}{2}({t}-{\tau^{*}_{\lambda}})^{2}+o(({t}-{\tau^{*}_{\lambda}})^{2}).$
(29)

Since ${\tau^{*}_{\lambda}}$ minimizes ${R_{\lambda}}({t})$, by the KKT
condition,

$\displaystyle{R_{\lambda}}^{\prime}({\tau^{*}_{\lambda}})$
$\displaystyle={F_{0}}^{\prime}({\tau^{*}_{\lambda}})-\lambda{F}^{\prime}({\tau^{*}_{\lambda}})={f_{0}}({\tau^{*}_{\lambda}})-\lambda{f}({\tau^{*}_{\lambda}})=0.$
(30)

Note that

$\displaystyle
0<\left(\frac{{f_{0}}}{{f}}\right)^{\prime}({\tau^{*}_{\lambda}})=\frac{{f_{0}}^{\prime}{}({\tau^{*}_{\lambda}}){f}({\tau^{*}_{\lambda}})-{f_{0}}{}({\tau^{*}_{\lambda}}){f}^{\prime}({\tau^{*}_{\lambda}})}{{f}^{2}({\tau^{*}_{\lambda}})}.$
(31)

By Eq. (30) and (31),

$\displaystyle\begin{split}{R_{\lambda}}^{\prime\prime}({\tau^{*}_{\lambda}})&={f_{0}}^{\prime}({\tau^{*}_{\lambda}})-\lambda{f}^{\prime}({\tau^{*}_{\lambda}})={f_{0}}^{\prime}({\tau^{*}_{\lambda}})-\frac{{f_{0}}({\tau^{*}_{\lambda}})}{{f}({\tau^{*}_{\lambda}})}{f}^{\prime}({\tau^{*}_{\lambda}})=\frac{{f_{0}}^{\prime}({\tau^{*}_{\lambda}}){f}({\tau^{*}_{\lambda}})-{f_{0}}({\tau^{*}_{\lambda}}){f}^{\prime}({\tau^{*}_{\lambda}})}{{f}({\tau^{*}_{\lambda}})}>0.\end{split}$
(32)

Plug Eq. (30) and (32) into Eq. (29) and we have proved that there exists
$D^{\prime}>0$,
$C^{\prime}={R_{\lambda}}^{\prime\prime}({\tau^{*}_{\lambda}})/4>0$, such that
Eq. (28) is valid. ∎

######  Lemma A.5.

Under the assumptions in Proposition A.4,
${\hat{\tau}_{\lambda,{n},m}}\to{\tau^{*}_{\lambda}}$ in probability as ${n}$,
$m\to\infty$.

###### Proof of Lemma A.5.

Let ${{\ell}_{\lambda,{n},m}}({t}):={{F}_{0,m}}({t})-\lambda{{F}_{n}}({t})$.
For any $\delta>0$, there exists $\varepsilon>0$ such that
${R_{\lambda}}({\tau^{*}_{\lambda}})<{R_{\lambda}}({t})-\varepsilon$ for any
$|{t}-{\tau^{*}_{\lambda}}|>\delta$. By Eq. (25) with
$\varepsilon^{\prime}=\varepsilon/2(1+\lambda)$, we have with probability at
least $1-8e^{-2(m\wedge{n})\varepsilon^{\prime 2}}$, for any
$|{t}-{\tau^{*}_{\lambda}}|>\delta$,

$\displaystyle\quad~{}{{\ell}_{\lambda,{n},m}}({\tau^{*}_{\lambda}})-{{\ell}_{\lambda,{n},m}}({t})$
$\displaystyle=({{\ell}_{\lambda,{n},m}}({\tau^{*}_{\lambda}})-{R_{\lambda}}({\tau^{*}_{\lambda}}))+({R_{\lambda}}({\tau^{*}_{\lambda}})-{R_{\lambda}}({t}))+({R_{\lambda}}({t})-{{\ell}_{\lambda,{n},m}}({t}))$
$\displaystyle<-\varepsilon+2(1+\lambda)\varepsilon^{\prime}=0.$

Therefore,
$\mathbb{P}(|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|\leq\delta)\to
1$. ∎

######  Lemma A.6.

Under the assumptions in Proposition A.4, there exists $C>0$ such that for any
$0<\delta\leq D$, ${n}\wedge m\geq 5$,

$\displaystyle\quad~{}\mathbb{E}\left[\sup_{|{t}-{\tau^{*}_{\lambda}}|\leq\delta}\left|\left({{\ell}_{\lambda,{n},m}}({t})-{R_{\lambda}}{}({t})\right)-\left({{\ell}_{\lambda,{n},m}}({\tau^{*}_{\lambda}})-{R_{\lambda}}{}({\tau^{*}_{\lambda}})\right)\right|\right]$
(33) $\displaystyle\leq
C(1+\lambda)\left(\sqrt{\frac{\delta\left({f_{\max}}\vee{f_{0,\max}}\right)\log({n}\wedge
m)}{{n}\wedge m}}+\frac{\log({n}\wedge m)}{{n}\wedge m}\right).$ (34)

###### Proof of Lemma A.6.

By the triangle inequality,

$\displaystyle\quad~{}\sup_{|{t}-{\tau^{*}_{\lambda}}|\leq\delta}\left|\left({{\ell}_{\lambda,{n},m}}({t})-{R_{\lambda}}{}({t})\right)-\left({{\ell}_{\lambda,{n},m}}({\tau^{*}_{\lambda}})-{R_{\lambda}}{}({\tau^{*}_{\lambda}})\right)\right|$
$\displaystyle\leq\sup_{|{t}-{\tau^{*}_{\lambda}}|\leq\delta}\left|\left({{F}_{0,m}}({t})-{F_{0}}({t})\right)-\left({{F}_{0,m}}({\tau^{*}_{\lambda}})-{F_{0}}({\tau^{*}_{\lambda}})\right)\right|$
$\displaystyle\quad~{}+\sup_{|{t}-{\tau^{*}_{\lambda}}|\leq\delta}\lambda\left|\left({{F}_{n}}{}({t})-{F}({t})\right)-\left({{F}_{n}}({\tau^{*}_{\lambda}})-{F}{}({\tau^{*}_{\lambda}})\right)\right|.$

We show there exists $C>0$, for any $0<\delta\leq D$, ${n}\geq 5$,

$\displaystyle\mathbb{E}\left[\sup_{|{t}-{\tau^{*}_{\lambda}}|\leq\delta}\left|\left({{F}_{n}}({t})-{F}({t})\right)-\left({{F}_{n}}({\tau^{*}_{\lambda}})-{F}({\tau^{*}_{\lambda}})\right)\right|\right]\leq
C\left(\sqrt{\frac{\delta{f_{\max}}\log({n})}{{n}}}+\frac{\log({n})}{{n}}\right).$
(35)

The analysis also applies to the null CDF part and combining the two parts
yields the desired result.

For ${t}<{\tau^{*}_{\lambda}}$,

$\displaystyle\left({{F}_{n}}({t})-{F}({t})\right)-\left({{F}_{n}}({\tau^{*}_{\lambda}})-{F}({\tau^{*}_{\lambda}})\right)=\frac{1}{n}\sum_{i=1}^{n}\mathbbm{1}_{\\{{T_{i}}\in({t},{\tau^{*}_{\lambda}}]\\}}-\mathbb{P}\left({T_{i}}\in({t},{\tau^{*}_{\lambda}}]\right),$

and similarly for ${t}>{\tau^{*}_{\lambda}}$. To bound the left hand side of
Eq. (35), we define

$\displaystyle\sigma_{\delta}:=\sup_{|{t}-{\tau^{*}_{\lambda}}|\leq\delta}\sqrt{\mathsf{Var}\left(\mathbbm{1}_{\\{{T_{i}}\in[{t}\wedge{\tau^{*}_{\lambda}},{t}\vee{\tau^{*}_{\lambda}}]\\}}\right)},$
$\displaystyle N_{\delta}:=$
$\displaystyle\left\\{\left(i,\mathbbm{1}_{\\{{T_{i}}\in[{t}\wedge{\tau^{*}_{\lambda}},{t}\vee{\tau^{*}_{\lambda}}]\\}}\right),~{}|{t}-{\tau^{*}_{\lambda}}|<\delta\right\\},\quad\pi_{\delta}:=\mathbb{E}\left[\log(2|N_{\delta}|)\right].$

We bound $\sigma_{\delta}$ and $\pi_{\delta}$ separately. For
$\sigma_{\delta}$, by Assumption 2,

$\displaystyle\sigma_{\delta}\leq\sqrt{\mathbb{P}\left({T_{i}}\in[{\tau^{*}_{\lambda}}-\delta,{\tau^{*}_{\lambda}}+\delta]\right)}\leq\sqrt{2\delta{f_{\max}}}.$
(36)

For $\pi_{\delta}$, notice that
$\mathbbm{1}_{\\{{T_{i}}\in[{\tau^{*}_{\lambda}},{t}]\\}}$ is binary and
increases with regard to ${t}$, then

$\displaystyle\left|\left\\{\left(i,\mathbbm{1}_{\\{{T_{i}}\in[{\tau^{*}_{\lambda}},{t}]\\}}\right),~{}0\leq{t}-{\tau^{*}_{\lambda}}\leq\delta\right\\}\right|\leq{n}+1,$

and similarly for $-\delta\leq{t}-{\tau^{*}_{\lambda}}\leq 0$. Combine the two
parts, $\left|N_{\delta}\right|\leq 2{n}+2$, and
$\pi_{\delta}\leq\log(4{n}+4)$. By [42, Lemma 6.4] or [2, Theorem 3.1],

$\displaystyle\quad~{}\mathbb{E}\left[\sup_{|{t}-{\tau^{*}_{\lambda}}|\leq\delta}\left|\frac{1}{n}\sum_{i=1}^{n}\mathbbm{1}_{\\{{T_{i}}\in[{t}\wedge{\tau^{*}_{\lambda}},{t}\vee{\tau^{*}_{\lambda}}]\\}}-\mathbb{P}\left({T_{i}}\in[{t}\wedge{\tau^{*}_{\lambda}},{t}\vee{\tau^{*}_{\lambda}}]\right)\right|\right]$
$\displaystyle\leq
2\sigma_{\delta}\sqrt{\frac{2\pi_{\delta}}{{n}}}+\frac{8\pi_{\delta}}{{n}}\leq
4\sqrt{\frac{\delta{f_{\max}}\log(4{n}+4)}{{n}}}+\frac{8\log(4{n}+4)}{{n}}.$

Take $C=16$, Eq. (35) is valid for $n\geq 5$. ∎

###### Proof of Proposition A.4.

By the proof of Lemma A.5, for ${n}\wedge m$ large enough,
$\mathbb{P}(|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|>D^{\prime}/2)\leq
1/\log({n}\wedge m)$ for the $D^{\prime}$ in Lemma A.4. Define the rate
$r_{{n}\wedge m}=({{n}\wedge m}/\log({{n}\wedge m}))^{-1/3}$, and shells

$\displaystyle S_{{{n}\wedge m},j}=\left\\{\tau:\frac{2^{j-1}}{r_{{n}\wedge
m}}\leq|{t}-{\tau^{*}_{\lambda}}|\leq\frac{2^{j}}{r_{{n}\wedge
m}}\right\\},\quad j\geq 1.$

If the event $\\{|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|\geq
2^{t}/r_{{n}\wedge m}\\}$ happens for some $t\geq 0$, then there exists $j\geq
t$ such that ${\hat{\tau}_{\lambda,{n},m}}\in S_{{{n}\wedge m},j}$. Therefore,
for ${n}\wedge m$ large enough,

$\displaystyle\quad~{}\mathbb{P}\left(|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|\geq\frac{2^{t}}{r_{{n}\wedge
m}}\right)$
$\displaystyle\leq\mathbb{P}\left(|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|\geq\frac{2^{t}}{r_{{n}\wedge
m}},~{}|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|\leq
D^{\prime}/2\right)+\mathbb{P}\left(|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|>D^{\prime}/2\right)$
$\displaystyle\leq\sum_{j\geq t,~{}\frac{2^{j-1}}{r_{{n}\wedge m}}\leq
D^{\prime}/2}\mathbb{P}\left({\hat{\tau}_{\lambda,{n},m}}\in S_{{n}\wedge
m,j}\right)+\frac{1}{\log({n}\wedge m)}$ $\displaystyle\leq\sum_{j\geq
t,~{}\frac{2^{j-1}}{r_{{n}\wedge m}}\leq
D^{\prime}/2}\mathbb{P}\left(\exists~{}{t}\in S_{{n}\wedge
m,j},~{}{{\ell}_{{n},m}}({t})<{{\ell}_{{n},m}}({\tau^{*}_{\lambda}})\right)+\frac{1}{\log({n}\wedge
m)}.$

Notice that if ${{\ell}_{{n},m}}({t})<{{\ell}_{{n},m}}({\tau^{*}_{\lambda}})$
and $|{t}-{\tau^{*}_{\lambda}}|\leq D^{\prime}$, by Lemma A.4,

$\displaystyle{{\ell}_{{n},m}}({t})-{R}({t})-({{\ell}_{{n},m}}({\tau^{*}_{\lambda}})-{R}({\tau^{*}_{\lambda}}))$
$\displaystyle\leq{R}({\tau^{*}_{\lambda}})-{R}({t})\leq-C^{\prime}({t}-{\tau^{*}_{\lambda}})^{2}\leq-C^{\prime}\frac{4^{j-1}}{r_{{n}\wedge
m}^{2}}.$

Then by Markov inequality and Lemma A.6,

$\displaystyle~{}\quad\mathbb{P}\left(\exists~{}{t}\in S_{{n}\wedge
m,j},~{}{{\ell}_{{n},m}}({t})<{{\ell}_{{n},m}}({\tau^{*}_{\lambda}})\right)$
$\displaystyle\leq\mathbb{P}\left(\sup_{{t}\in S_{{n}\wedge
m,j}}\left|{{\ell}_{{n},m}}({t})-{R}({t})-({{\ell}_{{n},m}}({\tau^{*}_{\lambda}})-{R}({\tau^{*}_{\lambda}}))\right|\geq
C^{\prime}\frac{4^{j-1}}{r_{{n}\wedge m}^{2}}\right)$
$\displaystyle\leq\frac{\mathbb{E}\left[\sup_{{t}\in S_{{n}\wedge
m,j}}\left|{{\ell}_{{n},m}}({t})-{R}({t})-({{\ell}_{{n},m}}({\tau^{*}_{\lambda}})-{R}({\tau^{*}_{\lambda}}))\right|\right]}{C^{\prime}\frac{4^{j-1}}{r_{{n}\wedge
m}^{2}}}$
$\displaystyle\leq\frac{C(1+\lambda)\left(\sqrt{\frac{(2^{j}/r_{{n}\wedge
m})({f_{\max}}\vee{f_{0,\max}})\log({n}\wedge m)}{{n}\wedge
m}}+\frac{\log({n}\wedge m)}{{n}\wedge
m}\right)}{C^{\prime}\frac{4^{j-1}}{r_{{n}\wedge m}^{2}}}$
$\displaystyle\leq\frac{8C(1+\lambda)\sqrt{{f_{\max}}\vee{f_{0,\max}}\vee
1})}{C^{\prime}}\cdot\sqrt{\frac{\log({n}\wedge m)}{{n}\wedge m}}\cdot
r_{{n}\wedge m}^{3/2}\cdot 2^{-3j/2}$
$\displaystyle=\frac{8C(1+\lambda)\sqrt{{f_{\max}}\vee{f_{0,\max}}\vee
1})}{C^{\prime}}\cdot 2^{-3j/2},$

where we use $\sqrt{\frac{(2^{j}/r_{{n}\wedge m})\log({n}\wedge m)}{{n}\wedge
m}}\geq\frac{\log({n}\wedge m)}{{n}\wedge m}$. Finally, we sum over shells
$S_{{n}\wedge m,j}$,

$\displaystyle\mathbb{P}\left(|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|\geq\frac{2^{t}}{r_{{n}\wedge
m}}\right)$
$\displaystyle\leq\frac{8C(1+\lambda)\sqrt{{f_{\max}}\vee{f_{0,\max}}\vee
1})}{C^{\prime}}\sum_{j\geq t}^{\infty}2^{-3j/2}+\frac{1}{\log({n}\wedge m)}$
$\displaystyle\leq\frac{8C(1+\lambda)\sqrt{{f_{\max}}\vee{f_{0,\max}}\vee
1})}{C^{\prime}}\cdot\frac{2^{-3t/2}}{1-2^{-3/2}}+\frac{1}{\log({n}\wedge
m)}.$

Take $2^{3t/2}=\log({n}\wedge m)$,

$\displaystyle\mathbb{P}\left(|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|\geq\frac{2^{t}}{r_{{n}\wedge
m}}\right)=\mathbb{P}\left(|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|\geq\frac{\log({n}\wedge
m)}{({n}\wedge m)^{1/3}}\right)\leq\frac{C^{*}}{\log({n}\wedge m)},$

where we let $C^{*}=\frac{8C(1+\lambda)\sqrt{{f_{\max}}\vee{f_{0,\max}}\vee
1})}{(1-2^{-3/2})C^{\prime}}+1$. ∎

######  Proposition A.5.

Let $\Lambda=(a,b)\subseteq(0,{\pi})$ and
$\mathcal{T}=\\{{\tau^{*}_{\lambda}}:\lambda\in\Lambda\\}$. Suppose there
exists a neighbourhood
$\mathcal{T}_{\Delta}=(\inf{\mathcal{T}}-\Delta,\sup{\mathcal{T}}+\Delta)$ for
some $\Delta>0$ and some $0<{f}_{\min}<{f}_{\max}<\infty$ such that

  1. 1.

$\inf_{\lambda\in\Lambda}\inf_{{t}\notin\mathcal{T}_{\Delta}}({F_{0}}({t})-\lambda{F}({t}))-({F_{0}}({\tau^{*}_{\lambda}})-\lambda{F}({\tau^{*}_{\lambda}}))>0$;

  2. 2.

$0<{f_{\min}}\leq{f}(t),{f_{0}}(t)\leq{f_{\max}}<\infty$ for all
${t}\in\mathcal{T}_{\Delta}$;

  3. 3.

${f_{0}}^{\prime}$, $f^{\prime}$, $({f_{0}}/{f})^{\prime}$ exist and
$0<f_{\min}^{\prime}\leq{f_{0}}^{\prime}({t}),{f}^{\prime}({t}),({f_{0}}/{f})^{\prime}({t})\leq
f_{\max}^{\prime}<\infty$ for all ${t}\in\mathcal{T}_{\Delta}$.

Then there exists a constant $C^{*}>0$ such that with probability at least
$1-C^{*}/\log({n}\wedge m)$,

$\displaystyle\sup_{\lambda\in\Lambda}\left|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}\right|\leq\frac{\log({n}\wedge
m)}{({n}\wedge m)^{1/3}},\quad\text{for all}{n}\wedge m\geq 5.$ (37)

###### Proof of Proposition A.5.

We can follow the proof of Proposition A.4 and substitute Lemma A.4, Lemma
A.5, and Lemma A.6 with their corresponding uniform results provided below.

Analogous to Lemma A.4, we have under Assumption 2 and Assumption 3, there
exists $C^{\prime}>0$, $D^{\prime}>0$ such that for any $\lambda\in\Lambda$,
any ${t}$ satisfying $\left|{t}-{\tau^{*}_{\lambda}}\right|\leq D^{\prime}$,

$\displaystyle{R_{\lambda}}({\tau^{*}_{\lambda}})\leq{R_{\lambda}}({t})-C^{\prime}({t}-{\tau^{*}_{\lambda}})^{2}.$
(38)

In fact, by Assumption 3, we perform Taylor expansion of ${R_{\lambda}}({t})$
at ${\tau^{*}_{\lambda}}$. For
$|{t}-{\tau^{*}_{\lambda}}|<\Delta\wedge(f_{\min}^{\prime}f_{\min}/2(f_{\max}^{\prime})^{2})$,

$\displaystyle{R_{\lambda}}({t})$
$\displaystyle={R_{\lambda}}({\tau^{*}_{\lambda}})+{R_{\lambda}}^{\prime}({\tau^{*}_{\lambda}})({t}-{\tau^{*}_{\lambda}})+\frac{{R_{\lambda}}^{\prime\prime}({t}^{\prime})}{2}({t}-{\tau^{*}_{\lambda}})^{2},$
(39)

for some
${t}^{\prime}\in[t\wedge{\tau^{*}_{\lambda}},t\vee{\tau^{*}_{\lambda}}]$.
Since ${\tau^{*}_{\lambda}}$ minimizes ${R_{\lambda}}({t})$, by the KKT
condition,

$\displaystyle{R_{\lambda}}^{\prime}({\tau^{*}_{\lambda}})$
$\displaystyle={F_{0}}^{\prime}({\tau^{*}_{\lambda}})-\lambda{F}^{\prime}({\tau^{*}_{\lambda}})={f_{0}}({\tau^{*}_{\lambda}})-\lambda{f}({\tau^{*}_{\lambda}})=0.$
(40)

By Assumption 3,

$\displaystyle
f_{\min}^{\prime}<\left(\frac{{f_{0}}}{{f}}\right)^{\prime}({t}^{\prime})=\frac{{f_{0}}^{\prime}{}({t}^{\prime}){f}({t}^{\prime})-{f_{0}}{}({t}^{\prime}){f}^{\prime}({t}^{\prime})}{{f}^{2}({t}^{\prime})}.$
(41)

By Eq. (40), (41), and Assumption 2, Assumption 3,

$\displaystyle\begin{split}{R_{\lambda}}^{\prime\prime}({t}^{\prime})&={f_{0}}^{\prime}({t}^{\prime})-\lambda{f}^{\prime}({t}^{\prime})={f_{0}}^{\prime}({t}^{\prime})-\frac{{f_{0}}({\tau^{*}_{\lambda}})}{{f}({\tau^{*}_{\lambda}})}{f}^{\prime}({t}^{\prime})\\\
&={f_{0}}^{\prime}({t}^{\prime})-\frac{{f_{0}}({t}^{\prime})}{{f}({t}^{\prime})}{f}^{\prime}({t}^{\prime})+\frac{{f_{0}}({t}^{\prime})}{{f}({t}^{\prime})}{f}^{\prime}({t}^{\prime})-\frac{{f_{0}}({\tau^{*}_{\lambda}})}{{f}({\tau^{*}_{\lambda}})}{f}^{\prime}({t}^{\prime})\\\
&>\frac{{f_{0}}^{\prime}({t}^{\prime}){f}({t}^{\prime})-{f_{0}}({t}^{\prime}){f}^{\prime}({t}^{\prime})}{{f}({t}^{\prime})}-|{t}^{\prime}-{\tau^{*}_{\lambda}}|\sup_{{t}\in\mathcal{T}_{\Delta}}\left(\frac{{f_{0}}}{{f}}\right)^{\prime}({t})\sup_{{t}\in\mathcal{T}_{\Delta}}{f}^{\prime}({t})\\\
&>f_{\min}^{\prime}f_{\min}-|{t}^{\prime}-{\tau^{*}_{\lambda}}|(f_{\max}^{\prime})^{2}.\end{split}$
(42)

Plug Eq. (40) and (42) into Eq. (39) and we have proved that Eq. (38) is valid
for $C^{\prime}=f_{\min}^{\prime}f_{\min}/4>0$ and
$D^{\prime}<\Delta\wedge(f_{\min}^{\prime}f_{\min}/2(f_{\max}^{\prime})^{2})$.

Analogous to Lemma A.5, under the assumptions in Proposition A.5,
$\sup_{\lambda\in\Lambda}|{\hat{\tau}_{\lambda,{n},m}}-{\tau^{*}_{\lambda}}|\to
0$ in probability as ${n}$, $m\to\infty$. In fact, by Assumption 3,
${R_{\lambda}}({t})$ decreases on
$(\inf\mathcal{T}-\Delta,{\tau^{*}_{\lambda}}]$ and increases on
$[{\tau^{*}_{\lambda}},\sup\mathcal{T}+\Delta)$. By Assumption 1 and (38), for
any $0<\delta<D^{\prime}$, $\lambda\in\Lambda$,
${t}-{\tau^{*}_{\lambda}}>\delta$,

$\displaystyle{R_{\lambda}}({\tau^{*}_{\lambda}})-{R_{\lambda}}({t})\leq{R_{\lambda}}({\tau^{*}_{\lambda}})-{R_{\lambda}}({\tau^{*}_{\lambda}}+\delta)\leq-C^{\prime}\delta^{2}.$

Similarly for ${t}-{\tau^{*}_{\lambda}}<-\delta$. The rest of the proof is
similar to that of Lemma A.5.

Analogous to Lemma A.6, we have under Assumption 2, there exists $C>0$,
$\delta>0$, such that for ${n}\wedge m\geq 5$,

$\displaystyle\begin{split}&\quad~{}\mathbb{E}\left[\sup_{\lambda\in\Lambda,|{t}-{\tau^{*}_{\lambda}}|\leq\delta}\left|\left({{\ell}_{\lambda,{n},m}}({t})-{R_{\lambda}}{}({t})\right)-\left({{\ell}_{\lambda,{n},m}}({\tau^{*}_{\lambda}})-{R_{\lambda}}{}({\tau^{*}_{\lambda}})\right)\right|\right]\\\
&\leq C(1+\lambda)\left(\sqrt{\frac{\delta{f_{\max}}\log({n}\wedge
m)}{{n}\wedge m}}+\frac{\log({n}\wedge m)}{{n}\wedge m}\right).\end{split}$

In fact, the outline of the proof is similar to that of Lemma A.6. It is left
to characterize the complexity of the function class
$\left\\{\mathbbm{1}_{\\{{T_{i}}\in[{t}\wedge{\tau^{*}_{\lambda}},{t}\vee{\tau^{*}_{\lambda}}]\\}},\lambda\in\Lambda,|t-{\tau^{*}_{\lambda}}|<\delta\right\\}$.
We define

$\displaystyle\sigma_{\delta}:=\sup_{\lambda\in\Lambda,|{t}-{\tau^{*}_{\lambda}}|\leq\delta}\sqrt{\mathsf{Var}\left(\mathbbm{1}_{\\{{T_{i}}\in[{t}\wedge{\tau^{*}_{\lambda}},{t}\vee{\tau^{*}_{\lambda}}]\\}}\right)},$
$\displaystyle N_{\delta}:=$
$\displaystyle\left\\{\left\\{i:\mathbbm{1}_{\\{{T_{i}}\in[{t}\wedge{\tau^{*}_{\lambda}},{t}\vee{\tau^{*}_{\lambda}}]\\}}\right\\}:~{}\lambda\in\Lambda,~{}|{t}-{\tau^{*}_{\lambda}}|<\delta\right\\},\quad\pi_{\delta}:=\mathbb{E}\left[\log(2|N_{\delta}|)\right].$

We bound $\sigma_{\delta}$ and $\pi_{\delta}$ separately. For
$\sigma_{\delta}$, by Assumption 2,

$\displaystyle\sigma_{\delta}\leq\sqrt{\mathbb{P}\left({T_{i}}\in[{\tau^{*}_{\lambda}}-\delta,{\tau^{*}_{\lambda}}+\delta]\right)}\leq\sqrt{2\delta{f_{\max}}}.$

For $\pi_{\delta}$, notice that
$\\{[{t}\wedge{\tau^{*}_{\lambda}},{t}\vee{\tau^{*}_{\lambda}}]:\lambda\in\Lambda,|{t}-{\tau^{*}_{\lambda}}|<\delta\\}$
is a subset of all intervals of $\mathbb{R}$, then

$\displaystyle\left|\left\\{\left\\{i:\mathbbm{1}_{\\{{T_{i}}\in[{t}\wedge{\tau^{*}_{\lambda}},{t}\vee{\tau^{*}_{\lambda}}]\\}}\right\\}:~{}\lambda\in\Lambda,~{}|{t}-{\tau^{*}_{\lambda}}|<\delta\right\\}\right|\leq
1+\binom{{n}+1}{2}\leq{n}^{2},$

for ${n}\geq 5$, and immediately $\pi_{\delta}\leq\log(2{n}^{2})$. ∎

######  Remark 14.

In the cases with non-monotone likelihood ratios, we identify the test
statistic values $t$ that minimize the objective function
${{F}_{0,m}}({t})-\lambda{{F}_{n}}({t})$ within a neighborhood
$[t-h_{n},t+h_{n}]$, where $h_{n}$ denotes a vanishing bandwidth. If there are
a finite number of test statistic values $\tau_{\lambda,l}^{*}$ such that
$\text{local-FDR}(\tau_{\lambda,l}^{*})=q$, an analogous convergence result to
Proposition A.4 can be obtained.

##  Appendix B Counter-examples

###  B.1 Stochastic dominance and PRDS

We provide an example to show that condition (i)(i.a) in Theorem 3.1 cannot be