# Simultaneous directional inference

Ruth Heller
Department of Statistics and Operations Research, Tel-Aviv University
<EMAIL_ADDRESS>
Aldo Solari
Department of Economics, Management and Statistics, University of Milano-
Bicocca
<EMAIL_ADDRESS>The authors gratefully acknowledge the Rita Levi-
Montalcini prize for Scientific Cooperation between Italy and Israel.

(August 2023)

###### Abstract

We consider the problem of inference on the signs of $n>1$ parameters. We aim
to provide $1-\alpha$ post-hoc confidence bounds on the number of positive and
negative (or non-positive) parameters. The guarantee is simultaneous, for all
subsets of parameters. Our suggestion is as follows: start by using the data
to select the direction of the hypothesis test for each parameter; then,
adjust the $p$-values of the one-sided hypotheses for the selection, and use
the adjusted $p$-values for simultaneous inference on the selected $n$ one-
sided hypotheses. The adjustment is straightforward assuming that the
$p$-values of one-sided hypotheses have densities with monotone likelihood
ratio, and are mutually independent. We show that the bounds we provide are
tighter (often by a great margin) than existing alternatives, and that they
can be obtained by at most a polynomial time. We demonstrate the usefulness of
our simultaneous post-hoc bounds in the evaluation of treatment effects across
studies or subgroups. Specifically, we provide a tight lower bound on the
number of studies which are beneficial, as well as on the number of studies
which are harmful (or non-beneficial), and in addition conclude on the effect
direction of individual studies, while guaranteeing that the probability of at
least one wrong inference is at most 0.05.

Keywords: Conditional inference; Directional decisions; Meta-analysis;
Multiple testing; Partitioning principle; Simultaneous confidence bounds.

##  1 Introduction

Let $\theta=(\theta_{1},\ldots,\theta_{n})$ be a vector of $n$ unknown real-
valued parameters. A conventional analysis is two-sided multiple testing,
e.g., testing the family of point null hypotheses
$H_{1}:\theta_{1}=0,\ldots,H_{n}:\theta_{n}=0$ with a procedure that
guarantees familywise error rate (FWER) control at a pre-specified level
$\alpha$. If the $i$th null hypothesis is rejected, the conclusion is that
$\theta_{i}\neq 0$. Tukey, (1991) argued that such a conclusion is
unsatisfactory, and the analysis should instead conclude on the sign of the
parameter, i.e., that $\theta_{i}>0$ or that $\theta_{i}<0$. Following
rejection, it is tempting to conclude on the sign of the parameter based on
the data. For example, for a rejected point null hypothesis, conclude that it
is positive if its point estimate is positive; otherwise conclude that it is
negative. However, a directional error, referred to often as a type III error
may occur if we decide after rejection of the point null hypothesis
$\theta_{i}=0$ that $\theta_{i}>0$ when in fact $\theta_{i}<0$ (or that
$\theta_{i}<0$ when in fact $\theta_{i}>0$). Therefore, the probability of
making at least one type I or type III error, henceforth referred to as the
directional FWER (dFWER), may be larger than $\alpha$ even though
$\textrm{FWER}\leq\alpha$. See Shaffer, (1980) for an example of lack of
control using Holm’s procedure (Holm,, 1979) for the family of point null
hypotheses when the test statistics are independent but Cauchy distributed.

For independent test statistics satisfying monotone likelihood ratio (MLR) for
$\theta_{i}$ (Karlin and Rubin,, 1956), existing two-sided multiple testing
procedures have been shown to control the dFWER: Holm’s procedure (Shaffer,,
1980), Hochberg’s procedure (Finner,, 1994; Liu,, 1997) and, more generally,
any closed testing procedure (Finner,, 1999). For example, for continuous
exponential families (where the $p$-value is a monotone transformation of the
sufficient statistic), the family of $p$-value densities satisfies the MLR
property. MLR is also satisfied when the observations are normally distributed
with unknown variance when testing a single mean, or assuming equal unknown
variance when comparing two means (Lehmann and Romano,, 2005).

Rather than adopting existing procedures for two-sided tests for the purpose
of conclusion on the sign of each parameter, one can consider the problem of
testing $2n$ one-sided hypotheses given by the following $n$ pairs:

$\displaystyle H_{i}^{-}:\theta_{i}\leq 0,\quad H_{i}^{+}:\theta_{i}\geq
0,\quad i=1,\ldots,n.$ (1)

The directional decisions are built-in in the rejections of multiple testing
procedures for the family of $2n$ hypotheses in (1), so the conventional error
rate (FWER) coincides with the directional error rather (dFWER). Let $p_{i}$
and $q_{i}$ be the $p$-values for $H_{i}^{-}$ and $H_{i}^{+}$, respectively,
$i=1,\ldots,n$.

Shaffer, (1974) considered (1) for $n=1$, referring to it as a directional
hypothesis-pair. Multiple directional hypothesis-pairs were considered in
Holm, (1979); Shaffer, (1980, 2002); Bauer et al., (1986); Finner, (1999); Guo
and Romano, (2015), among others. Bauer et al., (1986) showed that without
additional distributional assumptions, it is not possible to improve much over
Holm’s procedure on the $2n$ $p$-values of the family of hypotheses in (1),
for level $\alpha$ FWER control. Unfortunately, the number of rejections can
be considerably smaller than Holm’s procedure on two sided $p$-values (which
requires more distributional assumptions for validity), see details in § 1.1.

In this work, we provide powerful procedures not only for making positive and
negative discoveries, but also for bounding the total number of positive and
negative parameters, as well as for bounding the number of positive and
negative parameters in any index set $\mathcal{I}\subseteq\\{1,\ldots,n\\}$.
Let $n^{+}(\mathcal{I})$ and $n^{-}(\mathcal{I})$ be the number of parameters
in $\mathcal{I}$ with positive values and negative values, respectively:

$\displaystyle n^{+}(\mathcal{I})=|\\{i\in\mathcal{I}:\theta_{i}>0\\}|,\quad
n^{-}(\mathcal{I})=|\\{i\in\mathcal{I}:\theta_{i}<0\\}|.$

For simplicity of notation, we use $n^{+}$ and $n^{-}$ instead of $n^{+}([n])$
and $n^{-}([n])$, where $[n]$ is the index set $\\{1,\ldots,n\\}$. Our first
goal is to provide tight lower bounds for $n^{+}(\mathcal{I})$ and
$n^{-}(\mathcal{I})$, for any $\mathcal{I}$. More formally, we aim to provide
functionals $\ell_{\alpha}^{+}(\cdot)$, $\ell_{\alpha}^{-}(\cdot)$, so that
the following guarantee holds:

$\displaystyle\mathbb{P}_{\theta}\Big{(}n^{+}(\mathcal{I})\geq\ell_{\alpha}^{+}(\mathcal{I}),\,\,n^{-}(\mathcal{I})\geq\ell_{\alpha}^{-}(\mathcal{I}),\mathrm{\,\,for\,\,all\,\,}\mathcal{I}\Big{)}\geq
1-\alpha.$ (2)

We shall demonstrate in this work that much tighter lower bounds are possible
than the bounds based on the number of discoveries in each direction in
$\mathcal{I}$ (i.e., $|i\in\mathcal{I}:H_{i}^{-}\textrm{ is rejected}|$ for
the lower bound for $n^{+}(\mathcal{I})$; $|i\in\mathcal{I}:H_{i}^{+}\textrm{
is rejected}|$ for the lower bound for $n^{-}(\mathcal{I})$) .

Tight bounds are important in various applications. Consider, for example, the
case of evaluating a treatment effect on multiple subgroups (or cohorts). So
$\theta_{i}$ is the average treatment effect for subgroup $i$, and a positive
lower bound on both the number of subgroups for which $\theta_{i}>0$
($n^{+}$), and on the number of subgroups for which $\theta_{i}<0$ ($n^{-}$),
conveys the heterogeneity of the treatment effect. This can be of major
clinical importance, since it provides the researcher with the knowledge that
at least for some subgroups the treatment is harmful rather than effective
(i.e., that there is a qualitative interaction, Gail and Simon, 1985; Zhao et
al., 2019).

In some settings it is enough to infer on positive and non-positive findings,
i.e., the interest is only in lower bounds on $n^{+}(\mathcal{I})$ and on
$n^{-}(\mathcal{I})+n^{0}(\mathcal{I})$, where
$n^{0}(\mathcal{I})=|\mathcal{I}|-n^{+}(\mathcal{I})-n^{-}(\mathcal{I})=|i\in\mathcal{I}:\theta_{i}=0|$.
For example, in systematic reviews in clinical trials, when it is important to
evaluate the bounds on the number of studies with a positive treatment effect
(i.e., $\theta_{i}>0$), and a positive lower bound on $n^{-}+n^{0}$ is of
concern since it suggests that the treatment may have no effect or may even be
harmful (IntHout et al.,, 2016). More generally, in replicability analysis
(Bogomolov and Heller,, 2022), if interest lies in establishing that there is
a positive association (i.e., $\theta_{i}>0$) in more than one study, then
providing with $1-\alpha$ confidence the lower bound on $n^{+}(\mathcal{I})$
as well as on $n^{-}(\mathcal{I})+n^{0}(\mathcal{I})$ is of great interest:
the lower bound on $n^{+}(\mathcal{I})$ provides the minimal extent of
replicability in the desired positive direction that can be stated with
confidence; the lower bound on $n^{-}(\mathcal{I})+n^{0}(\mathcal{I})$
provides the evidence of the limit on replicability – a positive lower bound
means that at least some studies do not replicate.

Thus we would like to provide functionals $\tilde{\ell}_{\alpha}^{+}(\cdot)$,
$\tilde{\ell}_{\alpha}^{-}(\cdot)$ so that the following guarantee holds:

$\mathbb{P}_{\theta}\Big{(}n^{+}(\mathcal{I})\geq\tilde{\ell}_{\alpha}^{+}(\mathcal{I}),\,\,n^{-}(\mathcal{I})+n^{0}(\mathcal{I})\geq\tilde{\ell}_{\alpha}^{-}(\mathcal{I}),\mathrm{\,\,for\,\,all\,\,}\mathcal{I}\Big{)}\geq
1-\alpha.$ (3)

Since $n^{-}_{\alpha}(\mathcal{I})\leq
n^{-}_{\alpha}(\mathcal{I})+n^{0}(\mathcal{I}),$ the functionals
$\ell_{\alpha}^{+}(\mathcal{I})$ and $\ell_{\alpha}^{-}(\mathcal{I})$ that
satisfy (2) also satisfy (3). So it should be possible to improve the lower
bounds when aiming at the more relaxed requirement (3). For this purpose, we
modify $H_{i}^{+}$ in (1) by removing the equality sign. So the hypotheses
considered for $i=1,\ldots,n$ are:

$\displaystyle H_{i}^{-}:\theta_{i}\leq 0,\quad K_{i}:\theta_{i}>0,\quad
i=1,\ldots,n.$ (4)

Exactly one of the hypotheses pairs is true, i.e., in the family of $2n$ null
hypotheses defined in (4), there are exactly $n$ true one-sided null
hypotheses. Therefore, for making positive and non-positive discoveries more
powerful procedures than those that are typically used with two-sided testing
can be devised (Bauer et al.,, 1986; Guo and Romano,, 2015). Bauer et al.,
(1986) showed that for testing the $2n$ hypotheses defined in (4) using
Bonferroni, discoveries are made with the Bonferroni cut-off $\alpha/n$, which
is twice as large as as the Bonferroni cut-off for the family of hypotheses
(1). This can be understood from the work of Shaffer, (1986), who showed that
when performing a Bonferroni procedure, the proper correction factor for FWER
control is not the number of hypotheses tested, but the maximum number of null
hypotheses that can simultaneously be true (Goeman et al.,, 2010). In § 1.1 we
review additional FWER and FDR controlling procedures for (4).

The bounds we develop are uniformly tighter when considering the pairs of
hypotheses in (4) rather than in (1). Therefore, the approach based on the
pairs of hypotheses in (4) should always be preferred – unless it is important
to infer on the positive as well as the negative parameters. Support for
testing (4) is also given by Tukey, (1991). With regard to a single pair of
hypotheses, Tukey, (1991) argued as follows: “Statisticians classically asked
the wrong question – and were willing to answer with a lie, one that was often
a downright lie. They asked “Are the effects of A and B different?” and they
were willing to answer “no.” All we know about the world teaches us that the
effects of A and B are always different–in some decimal place–for any A and B.
Thus asking “Are the effects different?” is foolish. What we should be
answering first is “Can we tell the direction in which the effects of A differ
from the effects of B?” In other words, can we be confident about the
direction from A to B? Is it “up,” “down” or “uncertain”?”. So in cases where
the point null hypotheses are never true, the hypotheses in (4) should clearly
be preferred over the hypotheses in (1). According to Tukey, (1991); Jones and
Tukey, (2000), this is often the case.

###  1.1 FWER and FDR controlling procedures for testing (1) and (4)

Let $x_{i}=\min(p_{i},q_{i}),i=1,\ldots,n$ be the $n$ smallest $p$-values of
the $2n$ one-sided hypotheses. Let $x_{(1)}\leq\ldots\leq x_{(n)}$ be their
ordered values. For critical values $\alpha_{1}\leq\ldots\leq\alpha_{n}$, the
procedure for directional inference is as follows:

  1. 1.

Let

$\displaystyle R=\begin{cases}\max\\{r:x_{(r)}\leq\alpha_{i},\ \forall i\leq
r\\}&\text{for a step-down procedure },\\\
\max\\{r:x_{(r)}\leq\alpha_{r}\\}&\text{for a step-up procedure }.\end{cases}$
(5)

  2. 2.

For $r\leq R$, if $x_{(r)}$ is the $p$-value for $H_{i}^{-}$, declare
$\theta_{i}>0$, otherwise declare $\theta_{i}<0$ if testing (1) or
$\theta_{i}\leq 0$ if testing (4).

For testing (1) with FWER control, a valid procedure for any dependence
structure among the test statistics is Holm’s procedure on the $2n$
$p$-values, i.e., the step-down procedure with critical values
$\alpha_{i}=\alpha/(2n-i+1),i=1,\ldots,n$.

The next procedures were shown to provide FWER control under the additional
distributional assumption of uniform validity (Guo and Romano,, 2015; Zhao et
al.,, 2019; Ellis et al.,, 2020). A valid $p$-value $x_{i}$ is uniformly valid
if for all $\tau\in(0,1)$, $x_{i}/\tau$ given $x_{i}\leq\tau$ is valid, i.e.,
if the null hypothesis is true then

$\mathbb{P}(x_{i}\leq t\mid x_{i}\leq\tau)\leq\frac{t}{\tau}\ \ \forall\ \
0\leq t\leq\tau\leq 1.$

A sufficient condition for uniform validity is the MLR property of the test
statistic for $\theta_{i}$. It is in fact sufficient to require that the
inequality above be satisfied for $\tau=1/2$ (this is the weaker condition of
conditional validity, defined in $(A1)$ of § 3). These procedures are detailed
in the first two rows of Table 1.

Note that the procedure in row 1 has larger critical values than Holm’s
procedure on the $2n$ $p$-values (which has critical values
$\alpha_{i}=\alpha/(2n-i+1),\ i=1,\ldots,n$), and hence can lead to
considerably more discoveries. Moreover, the procedure in row 2 has critical
values that are almost twice as large as the procedure in row 1, so it should
be preferred if it is enough to discover positive and non-positive (rather
than negative) parameters.

The procedures in the last two rows of Table 1 provide FDR control on positive
and negative discoveries (row 5) and on positive and non-positive discoveries
(row 6). The procedure in row 6 provides a uniform improvement, in terms of
the number of discoveries, over the procedure in row 5, since the critical
values are twice as large.

Table 1: Overview of existing and proposed procedures for error criterion and
the type of inference considered. The critical values refer to the step-wise
procedure outlined in Equation (5). The procedures in the first four rows
require conditional validity and independence of the $n$ test statistics,
i.e., conditions (A1)-(A2) in § 3. The procedures in the last two rows require
only independence of the $n$ test statistics.

Criterion | Procedure | Type of inference | Critical values $\alpha_{i}$
---|---|---|---
FWER | Holm $n$ two-sided (Shaffer,, 1980) | positive/negative | $\alpha/\\{2(n-i+1)\\}$
Guo-Romano (Guo and Romano,, 2015) | positive/non-positive | $\alpha/(n-i+1+\alpha)$
FDP | Directional closed testing (§ 3) | positive/negative | -
Partitioning with adaptive tests (§ 4) | positive/non-positive | -
FDR | Benjamini-Hochberg (Benjamini and Yekutieli,, 2005) | positive/negative | $i\alpha/2n$
Guo-Romano (Guo and Romano,, 2015) | positive/non-positive | $i\alpha/n$

We conclude this section by a brief overview of our contributions, which
include the two procedures in rows 3 and 4 of Table 1.

###  1.2 A brief overview of our contributions

Closed testing procedures are useful not only for making discoveries, but also
in order to provide simultaneous confidence lower bounds on the number of true
discoveries in $\mathcal{I}$, or equivalently simultaneous confidence upper
bounds on the false discovery proportion (FDP) in $\mathcal{I}$ (Goeman and
Solari, 2011; Blanchard et al., 2020, reviewed in § 2). These bounds are often
referred to as post-hoc confidence bounds. When testing point null hypotheses
with a closed testing procedure, the number of true discoveries are the number
of parameters which are not zero. A lower bound on the number of parameters
which are not zero does not reveal information about the number of positive
and negative parameters in the set, and therefore may not be satisfactory.
More specifically, from having a functional $\ell_{\alpha}(\cdot)$ that
satisfies

$\displaystyle\mathbb{P}_{\theta}\Big{(}n^{+}(\mathcal{I})+n^{-}(\mathcal{I})\geq\ell_{\alpha}(\mathcal{I}),\mathrm{\,\,for\,\,all\,\,}\mathcal{I}\Big{)}\geq
1-\alpha.$ (6)

it is not clear how to extract the lower bounds to satisfy (2).

For finding $\ell_{\alpha}^{+}(\cdot)$ and $\ell_{\alpha}^{-}(\cdot)$ that
satisfy (2), we suggest the following two step approach, which we refer to
henceforth as directional closed testing (DCT): first, select from each pair
the hypothesis to test (based on the data); second, on the selected $n$ one-
sided hypotheses, apply an $\alpha$ level closed testing procedure (reviewed
in § 2). Of course, the second step has to adjust for the first step of
selection from the same data. With this approach, we can provide simultaneous
confidence bounds on $n^{+}(\mathcal{I})$ and $n^{-}(\mathcal{I})$ for any
$\mathcal{I}$, building upon the work of Goeman and Solari, (2011) that showed
how to obtain simultaneous bounds for a closed testing procedure. In
particular, we can provide tight lower bounds on $n^{+}$ and $n^{-}$, as well
as positive and negative discoveries, with the guarantee that the probability
of any wrong inference is at most $\alpha$.

Interestingly, a closed testing procedure on two-sided $p$-values (suggested
by Finner, (1999) for dFWER control) implicitly starts by the first step in
the DCT procedure. However, viewing closed testing on two-sided $p$-values as
a DCT procedure has several advantages. First, by realizing that the first
step is the selection of the one-sided hypotheses to test, the follow-up
inference is explicit on the selected one-sided hypotheses and therefore
controls the usual error rate (FWER), so there is no need for a special
definition of directional error rate (dFWER), see Benjamini, (2010) for a
discussion of the potential harms in having too many concepts of error rates.
Second, in addition to concluding on the signs of individual parameters DCT
provides simultaneous lower bounds on $n^{+}(\mathcal{I})$ and
$n^{-}(\mathcal{I})$ as described in (11). Third, the sufficient assumption
$(A1)$ in § 3 is more general than the distributional assumption for dFWER
control considered in Shaffer, (1980); Finner, (1999); Sarkar et al., (2004);
Guo and Romano, (2015).

Another advantage of DCT is that the second step in the DCT procedure can
start by testing for qualitative interactions (i.e., testing the null
hypothesis that all parameters are non-positive or all parameters are non-
negative) using the state-of-art approach suggested in Zhao et al., (2019).
The power to detect qualitative interactions will thus be as in Zhao et al.,
(2019), but in addition DCT provides (at no additional cost for multiplicity)
discoveries and confidence lower bounds on the number of positive and negative
parameters for any subset of parameters of interest, see § 5.2 for details.

It is worth noting that once we have obtained the confidence lower bounds on
the number of positive and negative parameters, we can automatically derive
the corresponding upper bounds, i.e. if
$n^{+}(\mathcal{I})\geq\ell_{\alpha}^{+}(\mathcal{I}),n^{-}(\mathcal{I})\geq\ell_{\alpha}^{-}(\mathcal{I})$
holds, then also
$n^{+}(\mathcal{I})\leq|\mathcal{I}|-\ell^{-}_{\alpha}(\mathcal{I}),n^{-}_{\alpha}(\mathcal{I})\leq|\mathcal{I}|-\ell^{+}_{\alpha}(\mathcal{I})$
holds.

For finding $\tilde{\ell}_{\alpha}^{+}(\cdot)$ and
$\tilde{\ell}_{\alpha}^{-}(\cdot)$ that satisfy (3), we suggest applying the
partitioning principle (reviewed in § 2) to the $2^{n}$ orthants defined by
$\\{(-\infty,0],(0,\infty)\\}^{n}$. Each orthant defines the orthant
hypothesis that $\theta$ is in the orthant. There are $2^{n}$ orthant
hypotheses, but exactly one orthant hypothesis is true. Therefore, it is
enough to test each at level $\alpha$. This is in contrast to DCT, where
multiple testing adjustments are made (adjustments are necessary, since for
the zero vector $\theta=0$ the $2n$ hypotheses in (1) are true).

We show the general approach for extracting the bounds, as well as the
specific polynomial time algorithm that uniformly improves over the DCT bounds
$\ell_{\alpha}^{+}(\cdot),\ell_{\alpha}^{-}(\cdot)$, so that
$\tilde{\ell}_{\alpha}^{+}(\cdot)\geq\ell_{\alpha}^{+}(\cdot)$,
$\tilde{\ell}_{\alpha}^{-}(\cdot)\geq\ell_{\alpha}^{-}(\cdot)$.

In § 2, we review the necessary statistical background and some of the
notation and definitions we are going to use throughout this work. Then, we
explain in detail DCT in § 3, and partitioning in § 4. In § 5 we provide
usages of our suggested methods: for subgroup analysis, for providing
inference following the test of qualitative interactions, and for enhancing
meta-analysis. In § 6 we compare and contrast the methods in simulations, and
provide some practical recommendations. Moreover, we quantify the improvement
upon the methods targeting FWER or FDR control in Table 1 (their lower bounds
are computed merely by counting the number of rejections in each direction).
Final remarks are provided in § 7. All proofs are in § A of the Supplementary
Material (SM). R code reproducing the examples is available at
`https://aldosolari.github.io/directionalinference/`.

##  2 Background on simultaneous inference

Let $H_{1},\ldots,H_{n}$ be $n$ base null hypotheses, or base hypotheses,
which form the basic units for the inference. The corresponding $p$-values are
$x_{1},\ldots,x_{n}$. Each $x_{i}$ is a valid $p$-value for its respective
null hypothesis $H_{i}$, i.e., its distribution is uniform or stochastically
larger than uniform when $H_{i}$ is true. For
$\mathcal{I}\subseteq\\{1,\ldots,n\\}$, let
$H_{\mathcal{I}}:\bigcap_{i\in\mathcal{I}}H_{i}$ be the _intersection
hypothesis_ of the $|\mathcal{I}|$ base hypotheses in $\mathcal{I}$. This
hypothesis is true if $\forall i\in\mathcal{I},\ H_{i}$ are true. The level
$\alpha$ test of $H_{\mathcal{I}}$, denoted by
$\phi_{\mathcal{I}}\in\\{0,1\\},$ is called the _local test_. It satisfies
that $\mathbb{P}_{\theta}(\phi_{\mathcal{I}}=1)\leq\alpha$ when $\theta\in
H_{\mathcal{I}}$.

#### The closed testing procedure

The closed testing procedure corrects the local tests for multiple testing by
rejecting $H_{\mathcal{I}}$ only if it is rejected by all intersection
hypotheses that include $\mathcal{I}$, i.e., if $\bar{\phi}_{\mathcal{I}}=1$,
where
$\bar{\phi}_{\mathcal{I}}=\min\\{\phi_{\mathcal{J}}:\mathcal{J}\supseteq\mathcal{I}\\}.$
Marcus et al., (1976) showed that the adjusted tests
$\bar{\phi}_{\mathcal{I}}$ have FWER control:
$\mathbb{P}_{\theta}\Big{(}\bar{\phi}_{\mathcal{I}}=0\mathrm{\,\,for\,\,all\,\,}\mathcal{I}\subseteq\mathcal{T}\Big{)}\geq
1-\alpha,$ where $\mathcal{T}$ is the (unknown) index set of the true base
hypotheses. The _discoveries_ are the base hypotheses rejected,
$\mathcal{D}_{\alpha}=\\{i\in[n]:\bar{\phi}_{i}=1\\}$.

#### Simultaneous post-hoc bounds

Goeman and Solari, (2011) showed that the lower bound for the number of true
discoveries in $\mathcal{I}$, $|\mathcal{I}\setminus\mathcal{T}|$, is

$\ell_{\alpha}(\mathcal{I})=\min(|\mathcal{I}\setminus\mathcal{J}|:\mathcal{J}\subseteq\mathcal{I},\bar{\phi}_{\mathcal{J}}=0),$
(7)

with the guarantee that
$\mathbb{P}_{\theta}\Big{(}|\mathcal{I}\setminus\mathcal{T}|\geq\ell_{\alpha}(\mathcal{I}),\mathrm{\,\,for\,\,all\,\,}\mathcal{I}\Big{)}\geq
1-\alpha.$ For two-sided hypotheses, this is the guarantee in (6). Note that
$\ell_{\alpha}(\cdot)$ can be used to upper bound the FDP, which is
$|\mathcal{I}\cap\mathcal{T}|/|\mathcal{I}|$: with at least $(1-\alpha)$
confidence, $|\mathcal{I}\cap\mathcal{T}|/|\mathcal{I}|$ is at most
$(|\mathcal{I}|-\ell_{\alpha}(\mathcal{I}))/|\mathcal{I}|$.

#### The combining function for testing an intersection hypothesis

The combining function $f:[0,1]^{|\mathcal{I}|}\longrightarrow[0,1]$, maps the
$|\mathcal{I}|$ $p$-values for $H_{i},i\in\mathcal{I}$, into a valid $p$-value
for testing $H_{\mathcal{I}}$. The local test is

$\phi_{\mathcal{I}}=\mathds{1}\\{f(\\{x_{i},i\in\mathcal{I}\\})\leq\alpha\\}.$

For mutually independent $p$-values, a popular combining function is Fisher,
which has excellent power properties for a wide range of signals (Benjamini
and Heller,, 2008; Hoang and Dickhaus,, 2022) is
$f_{\mathrm{Fisher}}(x_{1},\ldots,x_{d})=\mathbb{P}(\chi^{2}_{2d}\geq-2\sum_{i=1}^{d}\log
x_{i})$, where $\chi^{2}_{2d}$ is a chi-squared random variable with $2d$
degrees of freedom. Let $x_{(1)}\leq\ldots\leq x_{(d)}$ be the ordered values.
Another popular combining function is Simes111This combining method produces a
valid $p$-value for the intersection hypothesis also if the $p$-values being
combined are positive dependent, Goeman et al., (2019).,
$f_{\mathrm{Simes}}(x_{1},\ldots,x_{d})=\min_{1\leq i\leq
d}\left\\{\frac{d}{i}x_{(i)}\right\\}$. It is essentially based on a single
quantile, and thus does not take full advantage of the pooled evidence when
the number of positive (or negative) parameters is large. Bogomolov, (2023)
suggested the the following combination function, $\min_{1\leq i\leq
d}\left\\{\frac{2(\sum_{l=1}^{d}\mathds{1}(x_{l}>0.5)+1)}{i}x_{(i)}\right\\}$,
which uses the plug-in estimator for the number of null hypotheses combined.
It may be much smaller than Simes, except in the case where the intersection
is only of two hypotheses. Therefore, it makes sense to switch to the Simes
combining method in that case. So the modified Simes (mSimes) combining
function we shall consider is :

$f_{\mathrm{mSimes}}(x_{1},\ldots,x_{d})=\begin{cases}\min_{1\leq i\leq
d}\left\\{\frac{2(\sum_{l=1}^{d}\mathds{1}(x_{l}>0.5)+1)}{i}x_{(i)}\right\\}&\text{if
}d>2,\\\ \min_{1\leq i\leq
d}\left\\{\frac{d}{i}x_{(i)}\right\\}&\text{otherwise}.\end{cases}$ (8)

Our assumption regarding $f(\cdot)$, satisfied by all the combining functions
above, is:

$(A0)$

The combining function satisfies monotonicity: $f(x_{1},\ldots,x_{d})\leq
f((x^{\prime}_{1},\ldots,x^{\prime}_{d}))$ for $x_{i}\leq x_{i}^{\prime}$ for
all $i=1,\ldots,d$; and symmetry:
$f(x_{1},\ldots,x_{d})=f(x_{j_{1}},\ldots,x_{j_{d}})$ for any permutation
$(j_{1},\ldots,j_{d})$ of $(1,\ldots,d)$.

#### The computational complexity for evaluating post-hoc bounds

Computing $\ell_{\alpha}(\mathcal{I})$ involves the evaluation of
exponentially many tests, which hinders its practical application. For
specific combining functions, however, there exist shortcuts to derive the
closed testing results in polynomial time. Quadratic time shortcuts for
combining functions satisfying $(A0)$ have been developed for FWER control
(Dobriban,, 2020) and simultaneous FDP control (Goeman and Solari,, 2011;
Goeman et al.,, 2021). Tian et al., (2021) showed that if the combining
function satisfies also the separability condition (see Appendix B of Tian et
al., (2021) for the definition), both FDP and FWER shortcuts can be reduced to
linear time, after an initial sorting of the $p$-values. Separability is
satisfied by the Fisher’s combining function, as well by the combining
function based on generalized means of $p$-values (Vovk and Wang,, 2020). For
the Simes’ combining function, the shortcut described in Goeman et al., (2019)
allows calculation of $\ell_{\alpha}(\mathcal{I})$ in linear time, after an
initial preparatory step that takes linearithmic time.

#### The partitioning principle

Closed testing is a fundamental principle for controlling familywise error,
but there is another equally fundamental principle known as _partitioning
principle_ (Finner and Strassburger,, 2002). The key idea is to partition the
parameter space into disjoint regions and simultaneously test the hypotheses
that the true parameter value lies within each region. Since the true value
lies within exactly one of these regions, there is only one true null
hypothesis, and all the others are false. Therefore, if the test for the true
null hypothesis controls the Type I error probability, then the FWER is also
controlled. The partition procedure for the base hypotheses
$H_{1},\ldots,H_{n}$ defines for every $\mathcal{I}\subseteq[n]$ the
_partitioning hypothesis_

$J_{\mathcal{I}}=\big{\\{}\bigcap_{i\in\mathcal{I}}H_{i}\big{\\}}\cap\big{\\{}\bigcap_{j\in[n]\setminus\mathcal{I}}H_{j}^{c}\big{\\}},$

where $H_{j}^{c}$ is the $j$th alternative hypothesis, i.e. the complement of
$H_{j}$. The partitioning hypothesis $J_{\mathcal{I}}$ is true if and only if
all $H_{i}$, $i\in\mathcal{I}$, are true and all $H_{i}$,
$i\in[n]\setminus\mathcal{I}$, are false. The discoveries are all indices
$i\in[n]$ for which all $J_{\mathcal{I}}$ with $\mathcal{I}\ni i$ are rejected
at level $\alpha$.

Closed testing and partitioning are equivalent principles for FWER control:
for every closed testing procedure, there exists a partitioning procedure that
rejects exactly the same hypotheses. Likewise, for every partitioning
procedure, there exists a closed testing procedure that rejects exactly the
same hypotheses (Goeman et al.,, 2021). The partitioning principle has been
argued to be more convenient than closed testing for obtaining confidence
intervals and sets in Stefansson et al., (1988); Finner et al., (2021). Since
the family of hypotheses in (4) has $n$ true null hypotheses which are in
exactly one of the $2^{n}$ possible orthants, we suggest in § 4 to use the
partitioning principle in order to obtain confidence sets for
$n^{+}(\mathcal{I})$ and $n^{0}(\mathcal{I})+n^{-}(\mathcal{I})$.

##  3 Directional closed testing

Let $p=(p_{1},\ldots,p_{n})$ denote the $p$-value vector for
$(H_{1}^{-},\ldots,H_{n}^{-})$, and $q=(q_{1},\ldots,q_{n})$ denote the
$p$-value vector for $(H_{1}^{+},\ldots,H_{n}^{+})$. For simplicity, we assume
that the test statistics are continuous and $q_{i}=1-p_{i}$, $i\in[n]$. Our
assumptions for the $p$-values are:

$(A1)$

Each $p$-value is conditionally valid:

$\displaystyle\sup_{\theta\in H^{-}_{i}}\mathbb{P}_{\theta}(2p_{i}\leq x\mid
p_{i}\leq 1/2)\leq x\quad\forall x\in[0,1],$ $\displaystyle\sup_{\theta\in
H^{+}_{i}}\mathbb{P}_{\theta}(2q_{i}\leq x\mid p_{i}>1/2)\leq x\quad\forall
x\in[0,1].$

$(A2)$

$p_{1},\ldots,p_{n}$ are mutually independent.

We are interested in simultaneous inference on the selected family of $n$
directional hypotheses $\\{H_{i}^{-}:p_{i}\leq
1/2\\}\cup\\{H_{i}^{+}:p_{i}>1/2\\}.$ We propose the following procedure.

######  Procedure 3.1 (Directional closed testing).

  1. Step 1

Select the $n$ one-sided hypotheses for testing: $\\{H_{i}^{-}:p_{i}\leq
1/2\\}\cup\\{H_{i}^{+}:p_{i}>1/2\\}$. Let $\mathcal{S}^{-}=\\{i:p_{i}\leq
1/2\\}$ and $\mathcal{S}^{+}=\\{i:p_{i}>1/2\\}$ be, respectively, the indices
for which we test $H_{i}^{-}$ and $H_{i}^{+}$.

  2. Step 2

Apply a level $\alpha$ closed testing procedure on the family of $n$ one-sided
selected hypotheses , using the conditional $p$-values
$\\{2p_{i},i\in\mathcal{S}^{-}\\}\cup\\{2q_{j},j\in\mathcal{S}^{+}\\}.$ The
intersection hypothesis

$\displaystyle
H_{\mathcal{I}}:\big{\\{}\bigcap_{i\in\mathcal{I}\cap\mathcal{S}^{-}}H_{i}^{-}\big{\\}}\cap\big{\\{}\bigcap_{i\in\mathcal{I}\cap\mathcal{S}^{+}}H_{i}^{+}\big{\\}}$

is true if and only if all selected hypotheses with $i\in\mathcal{I}$ are
true. For all $\mathcal{I}\subseteq[n]$, testing $H_{\mathcal{I}}$ at level
$\alpha$ is done by the local test

$\displaystyle\phi_{\mathcal{I}}=\mathds{1}\\{f(\\{2p_{i},i\in\mathcal{I}\cap\mathcal{S}^{-}\\}\cup\\{2q_{j},j\in\mathcal{I}\cap\mathcal{S}^{+}\\})\leq\alpha\\}$
(9)

using a combining function $f(\cdot)$.

Note that
$f(\\{2p_{i},i\in\mathcal{I}\cap\mathcal{S}^{-}\\}\cup\\{2q_{j},j\in\mathcal{I}\cap\mathcal{S}^{+}\\}=f(2\min(p_{i},q_{i}),i\in\mathcal{I})$,
so the test of $H_{\mathcal{I}}$ coincides with the test of the intersection
of two-sided null hypotheses in $\mathcal{I}$ using two-sided $p$-values. We
denote by $\mathcal{S}$ the vector of signs that we condition on in Step 1 of
the DCT Procedure 3.1:

$\mathcal{S}=(sign(p_{1}-1/2),\ldots,sign(p_{n}-1/2)),\quad
sign(p_{i}-1/2)=\begin{cases}-1&\text{if }p_{i}\leq 1/2,\\\ 1&\text{if
}p_{i}>1/2.\end{cases}$

Conditional on the vector of signs $\mathcal{S}\in\\{-1,1\\}^{n},$ the
probability of rejecting the intersection hypotheses of the true nulls among
the selected is at most $\alpha$, since in the local test in (9) we combine
(conditional) $p$-values that have each a distribution that is at least as
large as uniform given $\mathcal{S}$. We formalize this in the next
proposition.

######  Proposition 3.1.

Let $p_{1},\ldots,p_{n}$ fulfil conditions $(A1)-(A2)$, and let $\theta$ be
the true unknown vector of parameters. So among the $n$ selected for testing
in Step 1 of Procedure 3.1, $\mathcal{T}=\\{\\{i:\theta_{i}\in
H_{i}^{-}\\}\cap\mathcal{S}^{-}\\}\cup\\{\\{i:\theta_{i}\in
H_{i}^{+}\\}\cap\mathcal{S}^{+}\\}$ are the indices of true one sided null
hypotheses. Then the test of the intersection of the hypotheses in
$\mathcal{T}$ satisfies conditional type I error control:

$\sup_{\theta^{\prime}\in
H_{\mathcal{T}}}\mathbb{P}_{\theta^{\prime}}\Big{(}\phi_{\mathcal{T}}=1\mid\mathcal{S}\Big{)}\leq\alpha.$
(10)

Proposition 3.1 implies that with probability at least $1-\alpha$ all the
rejections of intersection hypotheses in Procedure 3.1 are correct,
conditionally on $\mathcal{S}$ and therefore also unconditionally:
$\mathbb{P}_{\theta}\left(\bar{\phi}_{\mathcal{I}}=0\ \forall\
\mathcal{I}\subseteq\mathcal{T}\mid\mathcal{S}\right)\geq 1-\alpha$,
$\mathbb{P}_{\theta}\left(\bar{\phi}_{\mathcal{I}}=0\ \forall\
\mathcal{I}\subseteq\mathcal{T}\right)\geq 1-\alpha$, where
$\bar{\phi}_{\mathcal{I}}=\min\\{\phi_{\mathcal{J}}:\mathcal{J}\supseteq\mathcal{I}\\}$
for $\phi_{\mathcal{I}}$ in (9).

We now show explicitly how to obtain discoveries and confidence bounds on
$n^{+}(\mathcal{I})$ and $n^{-}(\mathcal{I})$ from Procedure 3.1. Let
$\mathcal{D}_{\alpha}^{+}=\\{i\in\mathcal{S}^{-}:\bar{\phi}_{i}=1\\}$ and
$\mathcal{D}_{\alpha}^{-}=\\{i\in\mathcal{S}^{+}:\bar{\phi}_{i}=1\\}$ be the
index sets of the positive and negative discoveries, respectively, where
$\bar{\phi}_{i}=\min\\{\phi_{\mathcal{J}}:\mathcal{J}\ni i\\}$. The procedure
concludes $\theta_{i}>0$ for all $i\in\mathcal{D}_{\alpha}^{+}$ and
$\theta_{i}<0$ for all $i\in\mathcal{D}_{\alpha}^{-}$ while guaranteeing that
the probability of at least one false conclusion on the sign of the parameter,
given the vector of signs $\mathcal{S}$, is at most $\alpha$.

More generally, for any $\mathcal{I}\subseteq[n]$, we can use (7) as follows.
For $\mathcal{I}\cap\mathcal{S}^{-}$,
$\ell_{\alpha}(\mathcal{I}\cap\mathcal{S}^{-})$ is the lower bound on the
number of positive parameters:
$n^{+}(\mathcal{I}\cap\mathcal{S}^{-})\geq\ell_{\alpha}(\mathcal{I}\cap\mathcal{S}^{-})$.
Similarly, for $\mathcal{I}\cap\mathcal{S}^{+}$: $\
n^{-}(\mathcal{I}\cap\mathcal{S}^{+})\geq\ell_{\alpha}(\mathcal{I}\cap\mathcal{S}^{+}).$
From (7) and (10) it follows that the bounds

$\displaystyle\ell_{\alpha}^{+}(\mathcal{I})=\ell_{\alpha}(\mathcal{I}\cap\mathcal{S}^{-}),\quad\ell_{\alpha}^{-}(\mathcal{I})=\ell_{\alpha}(\mathcal{I}\cap\mathcal{S}^{+})$
(11)

are such that

$\displaystyle\mathbb{P}_{\theta}\Big{(}n^{+}(\mathcal{I})\geq\ell_{\alpha}^{+}(\mathcal{I}),\,\,n^{-}(\mathcal{I})\geq\ell_{\alpha}^{-}(\mathcal{I})\mathrm{\,\,for\,\,all\,\,}\mathcal{I}\mid\mathcal{S}\Big{)}\geq
1-\alpha.$ (12)

In particular for $\mathcal{I}=[n]$, the $1-\alpha$ confidence bounds for
$n^{+}$ and $n^{-}$ are $\ell_{\alpha}^{+}=\ell_{\alpha}(\mathcal{S}^{-})$ and
$\ell_{\alpha}^{-}=\ell_{\alpha}(\mathcal{S}^{+})$, respectively. Note that
the positive and negative discoveries are given by
$\mathcal{D}^{+}_{\alpha}=\\{i:\ell_{\alpha}^{+}(i)=1\\},\mathcal{D}^{-}_{\alpha}=\\{i:\ell_{\alpha}^{-}(i)=1\\},$
and $\ell_{\alpha}^{+}(\mathcal{I})\geq|\mathcal{D}^{+}_{\alpha}|$,
$\ell_{\alpha}^{-}(\mathcal{I})\geq|\mathcal{D}^{-}_{\alpha}|$ always holds.
In § 5 \- § 6 we show that the gap between the lower bound and the number of
discoveries can be large for $\mathcal{I}=[n]$, and it is larger for Fisher’s
combining function than for mSimes and Simes. Finally, note that
$\ell_{\alpha}(\mathcal{I})$ serves as a lower bound for the number of non-
zero parameters $n^{+}(\mathcal{I})+n^{-}(\mathcal{I})$, and
$\ell_{\alpha}(\mathcal{I})\geq\ell_{\alpha}^{+}(\mathcal{I})+\ell_{\alpha}^{-}(\mathcal{I})$
always holds.

Computing (11) involves at most the evaluation of exponentially many local
tests (9), but for specific combining functions there exist shortcuts to
derive the closed testing results in polynomial time, as detailed in § 2.
Therefore, the computational time is reasonable even for moderately large $n$.

###  3.1 Dependent $p$-values

DCT is valid if the local test satisfies (10). Sufficient conditions are
(A0)-(A2). However, for specific dependencies among the $p$-values or test
statistics, it may be that (10) is satisfied even if (A2) is not satisfied.

A specific setting with dependent $p$-values for which we have a local test
that satisfies (10) is the setting of testing $n$ population normal means,
where the population variance $\sigma^{2}$ is unknown, but identical across
all populations. This is the common one-way analysis of variance setting. We
shall describe this setting using the notation $N(\mu,\tau)$,
$\chi^{2}_{\nu}$, $t_{\nu}$, and $F_{k,\nu}$ to denote the normal, chi-
squared, central t, and central F distributions, respectively (with their
respective parameters or degrees of freedom), and $t_{\nu,x}$, $F_{k,\nu,x}$
denote the $x$-quantile of $t_{\nu}$ and $F_{k,\nu}$, respectively.

The test statistic for the mean $\theta_{i}$ is $y_{i}/(k_{i}s)$, where
$y_{i}\sim N(\theta_{i},k_{i}\sigma)$ and $\nu
s^{2}/\sigma^{2}\sim\chi^{2}_{\nu}$ for the appropriate $k_{i}$ and $\nu$ that
depend on the sample sizes, $i=1,\ldots,n$. So $y_{i}/(k_{i}s)\sim t_{\nu}$
and
$\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}\frac{y_{i}^{2}}{k_{i}^{2}s^{2}}\sim
F_{|\mathcal{I}|,\nu}$ for the zero vector $\theta=0$, for all
$\mathcal{I}\subseteq[n]$. Although $y_{1},\ldots,y_{n}$ are independent, the
test statistics $y_{i}/(k_{i}s),i=1,\ldots,n$ are dependent since $\sigma$ is
estimated using all observations. The $p$-values are
$p_{i}=\mathbb{P}(t_{\nu}\geq y_{i}/(k_{i}s))$ and
$q_{i}=\mathbb{P}(t_{\nu}\leq y_{i}/(k_{i}s))$.

A natural combining function is the one for the F-test

$f(\\{2p_{i},i\in\mathcal{I}\cap\mathcal{S}^{-}\\}\cup\\{2q_{j},j\in\mathcal{I}\cap\mathcal{S}^{+}\\}=f(2\min(p_{i},q_{i}),i\in\mathcal{I})=\mathbb{P}\left(F_{|\mathcal{I}|,\nu}\geq\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}\frac{y_{i}^{2}}{k_{i}^{2}s^{2}}\right).$

To see that this is a valid combining function, note that for any $\theta\in
H_{\mathcal{I}}$ (defined in Step 2 of the DCT procedure 3.1):

$\displaystyle\mathbb{P}_{\theta}\left(\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}\frac{y_{i}^{2}}{k_{i}^{2}s^{2}}\geq
F_{|\mathcal{I}|,\nu,1-x}\geq\mid\mathcal{S}\right)=\mathbb{E}\left(\mathbb{P}_{\theta}\left(\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}\frac{y_{i}^{2}}{k_{i}^{2}s^{2}}\geq
F_{|\mathcal{I}|,\nu,1-x}\mid\mathcal{S},s\right)\right)$
$\displaystyle\leq\mathbb{E}\left(\mathbb{P}_{0}\left(\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}\frac{y_{i}^{2}}{k_{i}^{2}s^{2}}\geq
F_{|\mathcal{I}|,\nu,1-x}\mid\mathcal{S},s\right)\right)=\mathbb{P}_{0}\left(\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}\frac{y_{i}^{2}}{k_{i}^{2}s^{2}}\geq
F_{|\mathcal{I}|,\nu,1-x}\geq\mid\mathcal{S}\right)\leq x,$

where the expectation is over the distribution of $s$ (which is independent of
$\mathcal{S}$), the first inequality in the second row follows from Lemma 1 of
Al Mohamad et al., (2020), and the last inequality follows from the validity
of the F test when the vector of means $\theta=0$. The DCT procedures
coincides with the closed F-test procedure on two-sided $p$-values suggested
by Finner, (1999) for dFWER control. It complements the analysis for dFWER
control suggested by Finner, (1999) by providing also simultaneous post-hoc
bounds for $n^{+}(\mathcal{I})$ and $n^{-}(\mathcal{I})$ for all
$\mathcal{I}\subseteq[n].$

##  4 Simultaneous inference on positive and non-positive parameters

###  4.1 The general recipe for the simultaneous confidence lower bounds

We derive simultaneous $(1-\alpha)$-confidence lower bounds
$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})$ and
$\tilde{\ell}_{\alpha}(\mathcal{I})$ for $n^{+}(\mathcal{I})$ and
$n^{-}(\mathcal{I})+n^{0}(\mathcal{I})$ by using the partitioning principle,
reviewed in § 2. The main idea is to partition the parameter space
$\mathbb{R}^{n}$ into disjoint subspaces: consider the $2^{n}$ orthants

$\displaystyle\Theta_{\mathcal{K}}$ $\displaystyle=$
$\displaystyle\\{\theta\in\mathbb{R}^{n}:\theta_{i}>0\mathrm{\,\,for\,\,all\,\,}i\in\mathcal{K},\theta_{j}\leq
0\mathrm{\,\,for\,\,all\,\,}j\notin\mathcal{K}\\}$

for all $\mathcal{K}$, so that exactly one $\Theta_{\mathcal{K}}$ contains the
true parameter $\theta$.

Each orthant $\Theta_{\mathcal{K}}$ has a corresponding null hypothesis
$J_{\mathcal{K}}:\theta\in\Theta_{\mathcal{K}}$ that the orthant includes the
true parameter. This hypothesis is true if and only if all $K_{i}$,
$i\in\mathcal{K}$ are true and all $H^{-}_{j}$, $j\notin\mathcal{K}$ are true:

$\displaystyle
J_{\mathcal{K}}:\\{\bigcap_{i\in\mathcal{K}}K_{i}\\}\cap\\{\bigcap_{i\notin\mathcal{K}}H^{-}_{i}\\}.$

Let $\psi_{\mathcal{K}}\in\\{0,1\\}$ be the level $\alpha$ local test for
hypotheses $J_{\mathcal{K}}$, for all $\mathcal{K}$. The hypotheses
$J_{\mathcal{K}}$ are all disjoint, and only one of them is true. For the true
$J_{\mathcal{K}}$,
$\displaystyle\sup_{\theta\in\Theta_{\mathcal{K}}}\mathbb{P}_{\theta}(\psi_{\mathcal{K}}=1)\leq\alpha$.

Consider the hypothesis $J_{v}(\mathcal{I}):n^{+}(\mathcal{I})=v$, which can
be equivalently expressed as
$J_{v}(\mathcal{I}):\bigcup_{\mathcal{K}:|\mathcal{K}\cap\mathcal{I}|=v}J_{\mathcal{K}}$.
Then $J_{v}(\mathcal{I})$ is rejected if and only if all $J_{\mathcal{K}}$
with $|\mathcal{K}\cap\mathcal{I}|=v$ are rejected, and denote by

$\displaystyle\psi_{v}(\mathcal{I})=\min\\{\psi_{\mathcal{K}}:\mathcal{K}\subseteq[n]\textrm{
and }|\mathcal{K}\cap\mathcal{I}|=v\\}$ (13)

the statistical test for $J_{v}(\mathcal{I})$. The collection of values $v$
for which we failed to reject $J_{v}(\mathcal{I})$ at level $\alpha$
constitutes a $(1-\alpha)$ confidence set for $n^{+}(\mathcal{I})$.

######  Proposition 4.1.

Let
$\mathcal{N}^{+}(\mathcal{I})_{\alpha}=\\{v\in\\{0,\ldots,|\mathcal{I}|\\}:\psi_{v}(\mathcal{I})=0\\}.$
Then
$\mathbb{P}_{\theta}(n^{+}(\mathcal{I})\in\mathcal{N}^{+}(\mathcal{I})_{\alpha}\mathrm{\,\,for\,\,all\,\,}\mathcal{I})\geq
1-\alpha$ for any $\theta\in\mathbb{R}^{n}$. Furthermore, the lower bounds
given by
$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})=\min(\mathcal{N}^{+}(\mathcal{I})_{\alpha}),\quad\tilde{\ell}^{-}_{\alpha}(\mathcal{I})=|\mathcal{I}|-\max(\mathcal{N}^{+}(\mathcal{I})_{\alpha})$
satisfy (3).

The construction described above is very general, for any valid local test. In
particular, this means that if the $p$-values are dependent, the confidence
set can be computed as long as the local test used is valid for the dependency
of the $n$ $p$-values combined for each orthant hypothesis. See § C for an
example with dependent $p$-values.

Next, we show that by mapping the local tests used in DCT to local tests for
testing orthant hypotheses, we can get a uniform improvement over DCT.

###  4.2 A uniform improvement over DCT for the bounds on $n^{+}(\mathcal{I})$
and $n^{-}(\mathcal{I})+n^{0}(\mathcal{I})$

We can use the DCT local tests, henceforth referred to as the adaptive local
tests, for testing orthant hypothesis $J_{\mathcal{K}}$, as follows. Instead
of testing this intersection hypothesis, test the larger intersection
hypothesis that only combines $p$-values from $\mathcal{S}^{-}$ and
$\mathcal{S}^{+}$, i.e., test only the intersection of one-sided hypotheses
selected in the first step of the DCT procedure. The larger intersection
hypothesis has indices in
$\\{\mathcal{K}^{c}\cap\mathcal{S}^{-}\\}\cup\\{\mathcal{K}\cap\mathcal{S}^{+}\\}$.
The adaptive local test is

$\displaystyle\psi_{\mathcal{K}}$ $\displaystyle=$
$\displaystyle\phi_{\\{\mathcal{K}^{c}\cap\mathcal{S}^{-}\\}\cup\\{\mathcal{K}\cap\mathcal{S}^{+}\\}}$
$\displaystyle=$
$\displaystyle\mathds{1}\\{f(\\{2p_{i},i\in\mathcal{K}^{c}\cap\mathcal{S}^{-}\\}\cup\\{2q_{j},j\in\mathcal{K}\cap\mathcal{S}^{+}\\})\leq\alpha\\}.$

This test is a valid local test for $J_{\mathcal{K}}$ since

$\big{\\{}\bigcap_{i\in\mathcal{K}^{c}}H_{i}^{-}\big{\\}}\cap\big{\\{}\bigcap_{j\in\mathcal{K}}K_{j}\big{\\}}\
\ \subseteq\ \
\big{\\{}\bigcap_{i\in\mathcal{K}^{c}\cap\mathcal{S}^{-}}H_{i}^{-}\big{\\}}\cap\big{\\{}\bigcap_{j\in\mathcal{K}\cap\mathcal{S}^{+}}H^{+}_{j}\big{\\}}.$

Is this a good local test for an orthant? Zhao et al., (2019) argued that the
power of a test of an intersection hypothesis that combines only the
conditional $p$-values of $p$-values below a threshold has excellent power
properties. Since for testing each orthant, we combine only conditional
$p$-values for which the data favours the one-sided alternative, we expect the
adaptive local test to have the power advantage observed in Zhao et al.,
(2019). Specifically, we expect to have superior power over the non-adaptive
local test that combines all $p$-values in the intersection when the parameter
vector has mixed signs, and not lose much power if the parameter vector does
not have mixed signs.

However, the computational problem remains: direct application of (13) takes
exponential time since there are $\binom{|\mathcal{I}|}{v}$ orthants with
exactly $v$ positive parameters. For a combining function $f(\cdot)$
satisfying condition $(A0)$, the reduction to polynomial computation time is
possible by first sorting the conditional $p$-values in each of the following
sets: $\mathcal{S}^{-}\cap\mathcal{I}$, $\mathcal{S}^{+}\cap\mathcal{I}$,
$\mathcal{S}^{-}\cap\mathcal{I}^{c}$, $\mathcal{S}^{+}\cap\mathcal{I}^{c}$.
Consider the problem of testing that $n^{+}(\mathcal{I})=v$ and exactly $k$
positive parameters are in $\mathcal{S}^{+}\cap\mathcal{I}$. Since $f(\cdot)$
is monotone, the orthant with the largest $p$-value is the one that combines:
the $|\mathcal{I}\cap\mathcal{S}^{-}|-(v-k)$ largest $2p_{i}$’s in
$\mathcal{I}\cap\mathcal{S}^{-}$; the $k$ largest $2q_{i}$’s in
$\mathcal{I}\cap\mathcal{S}^{+}$; between $0$ and
$|\mathcal{I}^{c}\cap\mathcal{S}^{-}|$ largest $2p_{i}$’s in
$\mathcal{I}^{c}\cap\mathcal{S}^{-}$; between $0$ and
$|\mathcal{I}^{c}\cap\mathcal{S}^{+}|$ largest $2q_{i}$’s in
$\mathcal{I}^{c}\cap\mathcal{S}^{+}$. So we can find the largest combination
$p$-value by considering
$|\mathcal{S}^{-}\cap\mathcal{I}^{c}|\times|\mathcal{S}^{+}\cap\mathcal{I}^{c}|$
possible combinations. Considering all possible values of $k$ and $v$, we see
that the complexity is $\mathcal{O}(|\mathcal{I}|^{2}|\mathcal{I}^{c}|^{2})$.
See § D for the detailed algorithm.

######  Proposition 4.2.

Let $p_{1},\ldots,p_{n}$ fulfil conditions $(A1)-(A2)$, and the combining
function $f(\cdot)$ fulfils condition $(A0)$. Then using the adaptive local
tests (4.2), the partitioning procedure (AP) returns the lower bounds
$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})$ and
$\tilde{\ell}^{-}_{\alpha}(\mathcal{I})$ for $n^{+}(\mathcal{I})$ and
$n^{-}(\mathcal{I})+n^{0}(\mathcal{I})$, respectively, with at most
$\mathcal{O}(|\mathcal{I}|^{2}\cdot\max(1,|\mathcal{I}^{c}|^{2}))$
computation. The bounds satisfy the following conditional (on the vector of
signs, $\mathcal{S}$) and unconditional coverage guarantees, respectively:

$\mathbb{P}_{\theta}\Big{(}n^{+}(\mathcal{I})\geq\tilde{\ell}_{\alpha}^{+}(\mathcal{I}),\,\,n^{-}(\mathcal{I})+n^{0}(\mathcal{I})\geq\tilde{\ell}_{\alpha}^{-}(\mathcal{I}),\mathrm{\,\,for\,\,all\,\,}\mathcal{I}\mid\mathcal{S}\Big{)}\geq
1-\alpha,$ (15)
$\displaystyle\mathbb{P}_{\theta}\Big{(}n^{+}(\mathcal{I})\geq\tilde{\ell}_{\alpha}^{+}(\mathcal{I}),\,\,n^{-}(\mathcal{I})+n^{0}(\mathcal{I})\geq\tilde{\ell}_{\alpha}^{-}(\mathcal{I}),\mathrm{\,\,for\,\,all\,\,}\mathcal{I}\Big{)}\geq
1-(1-2^{-n})\alpha>1-\alpha.$ (16)

The conditional $(1-\alpha)$ confidence bounds, i.e., the bounds that satisfy
(15), have an unconditional $1-\alpha(1-2^{-n})$ confidence guarantee, i.e.,
they satisfy (16). Therefore, adaptive local tests with partitioning can be
carried out at level $\tilde{\alpha}=\alpha/(1-2^{-n})$ if the conditional
guarantee is not necessary. Using $\tilde{\alpha}$ falls within the framework
of the holistic approach of Goeman and Solari, (2022), that view selection and
conditioning as means for useful inferences with unconditional error
guarantees.

With $(1-\alpha)$ confidence,
$\tilde{\ell}_{\tilde{\alpha}}^{+}(\mathcal{I})\leq
n^{+}(\mathcal{I})\leq|\mathcal{I}|-\tilde{\ell}_{\tilde{\alpha}}^{-}(\mathcal{I})$.
These are tighter bounds than with $(1-\tilde{\alpha})$ confidence.
Specifically, the overall lower bounds for $n^{+}$ or $n^{-}+n^{0}$ are
positive if $f(\\{2p_{i},i\in\mathcal{S}^{-}\\})\leq\tilde{\alpha}$ or
$f(\\{2q_{j},j\in\mathcal{S}^{+}\\})\leq\tilde{\alpha}$, respectively. In
contrast, with DCT more testing takes place to correct for multiplicity, and
moreover the local tests are carried out at level $\alpha<\tilde{\alpha}$.
Therefore, there is less power to detect positive lower bounds for $n^{+}$ and
$n^{-}+n^{0}$ with DCT. The number of discoveries for FWER control at level
$\alpha$ with partitioning may also be greater than with DCT, but only because
it is carried out at level $\tilde{\alpha}>\alpha$. The multiplicity
correction is essentially the same, since a discovery is made for $i\in[n]$
using partitioning only if all adaptive local tests that include it are
rejected, and these are precisely the local tests for all $\mathcal{I}\ni i$.
By this same reasoning, the lower bounds are the same when using partitioning
and DCT, with the same level $\alpha$ local tests, for $n^{+}(\mathcal{I})$ if
$\mathcal{I}\subseteq\mathcal{S}^{-}$, and for
$n^{-}(\mathcal{I})+n^{0}(\mathcal{I})$ if
$\mathcal{I}\subseteq\mathcal{S}^{+}$.

######  Proposition 4.3.

For a given combining function satisfying $(A0)$, let
$\ell_{\alpha}^{+}(\cdot),\ell_{\alpha}^{-}(\cdot)$ be the DCT lower bounds
that satisfy (11), and
$\tilde{\ell}_{\alpha}^{+}(\cdot),\tilde{\ell}_{\alpha}^{-}(\cdot)$ be the
bounds extracted from applying the level $\alpha$ adaptive local tests in
(4.2). Then $\tilde{\ell}_{\alpha}^{+}(\cdot)\geq\ell_{\alpha}^{+}(\cdot)$,
and $\tilde{\ell}_{\alpha}^{-}(\cdot)\geq\ell_{\alpha}^{-}(\cdot)$. Moreover,
if $\mathcal{I}\subseteq\mathcal{S}^{-}$ then
$\tilde{\ell}_{\alpha}^{+}(\mathcal{I})=\ell_{\alpha}^{+}(\mathcal{I})$ and
$\tilde{\ell}_{\alpha}^{-}(\mathcal{I})=\ell_{\alpha}^{-}(\mathcal{I})=0$; if
$\mathcal{I}\subseteq\mathcal{S}^{+}$ then
$\tilde{\ell}_{\alpha}^{-}(\mathcal{I})=\ell_{\alpha}^{-}(\mathcal{I})$ and
$\tilde{\ell}_{\alpha}^{+}(\mathcal{I})=\ell_{\alpha}^{+}(\mathcal{I})=0$.

###  4.3 Inference for $n=2$

To enhance our understanding of the partitioning procedure with adaptive local
tests, we examine the basic case of $n=2$. Table 2 presents the adaptive local
tests used as a function of the selection event, while the top-left plot in
Figure 1 displays the regions that provide inference about $n^{+}$ using
Simes’ combining function $f_{\mathrm{Simes}}$.

Table 2:  For each orthant hypothesis (rows), the adaptive local tests utilized by the partitioning procedure for the case of $n=2$ if the realized vector of signs is $\mathcal{S}=(sign(p_{1}-1/2),sign(p_{2}-1/2))$ (columns). | $\mathcal{S}=(-1,-1)$ | $\mathcal{S}=(-1,1)$ | $\mathcal{S}=(1,-1)$ | $\mathcal{S}=(1,1)$
---|---|---|---|---
$H^{-}_{1}\cap H^{-}_{2}$ | $f(2p_{1},2p_{2})$ | $2p_{1}$ | $2p_{2}$ | 1
$H^{-}_{1}\cap K_{2}$ | $2p_{1}$ | $f(2p_{1},2q_{2})$ | 1 | $2q_{2}$
$K_{1}\cap H^{-}_{2}$ | $2p_{2}$ | 1 | $f(2q_{1},2p_{2})$ | $2q_{1}$
$K_{1}\cap K_{2}$ | 1 | $2q_{2}$ | $2q_{1}$ | $f(2q_{1},2q_{2})$

By examining the top row of Figure 1, we can clearly observe the power gain
for the partitioning procedure with adaptive local tests (top-left plot)
compared to DCT (top-right plot). One notable aspect is that the partitioning
region expands significantly at the top-left and bottom-right quadrants,
thereby enhancing the partitioning procedure’s capability to detect both
positive and non-positive effects. This improved performance is accompanied by
the advantage of operating at a higher significance level, as determined by
$\tilde{\alpha}=(1-(1-2^{-n}))\alpha=4\alpha/3$. This increase in power comes
with the restriction that the partitioning procedure can only draw inferences
about non-positive effects. For instance, if $p_{1}>1-\frac{\alpha}{4}$, the
partitioning procedure would infer that $\theta_{1}\leq 0$. In contrast, the
DCT procedure can provide a more conclusive statement by inferring that
$\theta_{1}<0$.

Partitioning with adaptive tests$2\alpha/3$$\alpha/3$$p_{1}$$p_{2}$Directional
closed testing$\alpha/2$$\alpha/4$$p_{1}$$p_{2}$$n^{+}=2$$n^{+}\geq
1$$n^{+}=1$$n^{+}\leq 1$$n^{+}=0$

Directional closed
testing$\alpha/2$$\alpha/4$$p_{1}$$p_{2}$$H_{1}^{-},H_{2}^{-}$$H_{1}^{-},H_{2}^{+}$$H_{1}^{+},H_{2}^{+}$$H_{1}^{+},H_{2}^{-}$$H_{1}^{-}$$H_{1}^{+}$$H_{2}^{-}$$H_{2}^{+}$Partitioning
with adaptive
tests$2\alpha/3$$\alpha/3$$p_{1}$$p_{2}$$H_{1}^{-},H_{2}^{-}$$H_{1}^{-},K_{2}$$K_{1},K_{2}$$K_{1},H_{2}^{-}$$H_{1}^{-}$$K_{1}$$H_{2}^{-}$$K_{2}$

Figure 1:  The top row illustrates regions leading to inference about $n^{+}$
with $(1-\alpha)$ confidence for the partitioning procedure with adaptive
local tests (top-left plot) and the directional closed testing procedure (top-
right plot) with Simes’ combining function. The gray scale indicates the type
of inference. The bottom row illustrates regions leading to the rejection of
base hypotheses for partitioning with adaptive local tests (bottom left plot)
and directional closed testing (bottom right plot). The pattern indicates the
number of hypotheses rejected at level $\alpha$: one (dots) or two (diagonal
lines). Each plot is based on $\alpha=0.2$.

The bottom-left plot of Figure 1 displays the rejection region for the base
hypotheses: $H_{1}^{-}$, $H_{2}^{-}$, $K_{1}$, and $K_{2}$. Notably, if we
observe $(p_{1},p_{2})$ such that $1/2<p_{1}<1-2\alpha/3$ and
$\alpha/3<p_{2}<2\alpha/3$, we refrain from rejecting any base hypothesis,
even if we can infer that $n^{+}\geq 1$. In contrast, in DCT using Simes’
combining function (bottom-right plot), whenever there is an informative
inference about $n^{+}$, it is accompanied by the rejection of base hypotheses
$H_{1}^{-}$, $H_{2}^{-}$, $K_{1}$, and $K_{2}$. This is not the case for $n>2$
or by using Fisher’s combining function for $n=2$ (see § B for this and the
comparison with other FWER controlling procedures). The shape of the region in
the left plot of Figure 1 is non-monotone, in the sense that a larger pair of
$p$-values may result in a larger lower bound on $n^{+}$. For example, if we
observe $(p_{1},p_{2})$ such that $1/2<p_{1}<1-2\alpha/3$ and
$\alpha/3<p_{2}<2\alpha/3$, we would conclude that $n^{+}\geq 1$ by rejecting
$H_{1}^{-}\cap H_{2}^{-}$ because $2p_{2}\leq\tilde{\alpha}$ (see column 3 in
Table 2). However, if $p_{1}$ decreases to $\alpha/3<p_{1}<1/2$, we would not
draw any inference. This outcome is rather counter-intuitive since a smaller
value of $p_{1}$ corresponds to stronger evidence supporting $\theta_{1}>0$.
This is a result of employing adaptive tests rather than non-adaptive ones,
i.e. of performing selection. The procedure is nevertheless attractive, since
conditional on the selection (of which $p$-values of one-sided hypotheses to
combine), the rejection region is monotone.

Figure 2 shows the power advantage of partitioning over DCT, for a range of
parameters $(\theta_{1},\theta_{2})$ and $\alpha=5\%$. The $p$-values are
derived from independent test statistics $\hat{\theta}_{i}$ following a
Gaussian distribution $N(\theta_{i},1)$, $i=1,2$. The non-trivial bounds
probability (left panel) refers to the probability that $(p_{1},p_{2})$ lies
in the non-white area of the top plots in Figure 1. The expected number of
discoveries (right panel) is the weighted average of the number of rejections
displayed in the bottom plots of Figure 1, with weights determined by their
respective probabilities. . The gain is substantial, particularly in terms of
the non-trivial bounds probability when the parameters have opposite signs.
For example, with $(\theta_{1},\theta_{2})=(2,-1.5)$, the DCT has a
probability of 54.1%, while partitioning has 71.6%, resulting in a difference
of 17.5%. This difference is nearly triple the value of $61.7\%-55.1\%=6.6\%$
observed when both effects are positive, with
$(\theta_{1},\theta_{2})=(2,1.5)$.

$-3$$-2$$-1$$0$$1$$2$$3$$0$$0.2$$0.4$$0.6$$0.8$$1$$\theta_{2}$Non-trivial
bounds probabilityDCT ($\theta_{1}=0$)AP ($\theta_{1}=0$)DCT
($\theta_{1}=2$)AP ($\theta_{1}=2$)

$-3$$-2$$-1$$0$$1$$2$$3$$0$$0.2$$0.4$$0.6$$0.8$$1$$1.2$$1.4$$\theta_{2}$Average
number of discoveriesDCT ($\theta_{1}=0$)AP ($\theta_{1}=0$)DCT
($\theta_{1}=2$)AP ($\theta_{1}=2$)

Figure 2: Probability of observing non-trivial bounds (left plot) and the
expected number of discoveries (right plot) versus $\theta_{2}$ with
directional closed testing (DCT, solid line) and adaptive local tests with
partitioning (AP, dashed line) at the confidence level of 95%. Black curves
correspond to $\theta_{1}=0$, and blue curves to $\theta_{1}=2$.

##  5 Applications

###  5.1 Subgroup Analysis

In subgroup analysis, it is common to apply two-sided tests in order to
conclude about the intervention effect in each subgroup. An easier, yet
informative goal, is to provide lower bounds on $n^{+}$ and either $n^{-}$ or
$n^{-}+n^{0}$ . The data of Fisher et al., (1983), re-analysed in Gail and
Simon, (1985), consists of the difference in disease-free survival
probabilities at 3 years for breast cancer patients who underwent a PFT
treatment (L-Phenylalanine mustard, 5-Fluorouracil, and Tamoxifen) compared to
those who received a PF treatment (L-Phenylalanine mustard and
5-Fluorouracil). The study includes a total of 1260 patients divided in $n=4$
subgroups, classified based on age and progesterone receptor levels. Let
$m_{i}^{j}$ be the sample size for treatment
$j\in\\{\mathrm{PFT},\mathrm{PF}\\}$ in the $i$th subgroup. Table 3 provides
$\hat{\pi}_{i}^{j}$, the Kaplan-Meier estimate of $\pi_{i}^{j}$, the disease-
free survival probability at 3 years for patients who underwent treatment $j$
and belong to subgroup $i$. The corresponding standard error
$\mathrm{SE}(\hat{\pi}^{j}_{i})$ is calculated using Greenwood’s formula
(Greenwood et al.,, 1926).

However, the variable $\sqrt{m^{j}_{i}}(\hat{\pi}^{j}_{i}-\pi^{j}_{i})$
converges in distribution to a Normal variable with a mean of 0 and a variance
that depends on the (unknown) survival probability $\pi^{j}_{i}$, which
simplifies to $\pi_{i}^{j}(1-\pi_{i}^{j})$ if there is no censoring. For
censored data, Cutler and Ederer, (1958) suggested replacing $m^{j}_{i}$ with
the effective sample size
$\tilde{m}^{j}_{i}=\hat{\pi}^{j}_{i}(1-\hat{\pi}^{j}_{i})/\mathrm{SE}^{2}(\hat{\pi}^{j}_{i})$.
Anderson et al., (1982) showed that
$\sqrt{\tilde{m}^{j}_{i}}(\hat{\pi}^{j}_{i}-\pi^{j}_{i})$ converges in
distribution to $N(0,\pi^{j}_{i}(1-\pi^{j}_{i}))$. This motivates the arcsine-
square root transformation
$g(\hat{\pi}^{j}_{i})=\arcsin{\sqrt{\hat{\pi}^{j}_{i}}}$: by the delta method,
the asymptotic variance of
$\sqrt{\tilde{m}^{j}_{i}}\\{g(\hat{\pi}^{j}_{i})-g(\pi^{j}_{i})\\}$ is $1/4$,
making it an effective variance-stabilizing transformation. Then, the large-
sample normal approximation for the standardized difference
$\hat{\theta}_{i}=\\{g(\hat{\pi}^{\mathrm{PFT}}_{i})-g(\hat{\pi}^{\mathrm{PF}}_{i})\\}/\sqrt{1/(4\tilde{m}^{\mathrm{PFT}}_{i})+1/(4\tilde{m}^{\mathrm{PF}}_{i})}$
can be used to calculate the $p$-value $p_{i}=1-\Phi(\hat{\theta}_{i})$ for
$H_{i}^{-}$, where $\Phi(\cdot)$ is the $N(0,1)$ CDF (Klein et al.,, 2007).

Table 3:  Analysis of disease-free survival probabilities at 3 years $\hat{\pi}_{i}^{j}$ for breast cancer patients who underwent treatment $j$th treatment ($j=$PFT, PF) and belong to subgroup $i$ defined by age and progesterone receptor (PR) levels ($i=1$: Age $<50$, PR $<10$; $i=2$: Age $\geq 50$, PR $<10$; $i=3$: Age $<50$, PR $\geq 10$; $i=4$: Age $\geq 50$, PR $\geq 10$). Reproduced from Table 2 in Gail and Simon, (1985). The last two rows are the (standardized) differences between PFT and PF disease-free survival probabilities at 3 years (arcsin-square root transformed) and the $p$-value for $H_{i}^{-}$ computed using the large sample normal approximation. Subgroup $i$ | 1 | 2 | 3 | 4
---|---|---|---|---
Treatment $j$ | PFT | PF | PFT | PF | PFT | PF | PFT | PF
$\hat{\pi}_{i}^{j}$ | .599 | .436 | .526 | .639 | .651 | .698 | .639 | .790
$\mathrm{SE}(\hat{\pi}^{j}_{i})$ | .0542 | .0572 | .0510 | .0463 | .0431 | .0438 | .0386 | .0387
$\hat{\theta}_{i}$ | 2.051 | -1.635 | -0.764 | -2.708
$p_{i}$ | .0202 | .9490 | .7774 | .9966

Applying the first step of hypothesis selection based on the direction favored
by the data, the DCT procedure selects
$H_{1}^{-},H_{2}^{+},H_{3}^{+},H_{4}^{+}$. The $p$-values used for testing in
the second step, are therefore: $0.0202,1-0.9490,1-0.7774$, and $1-0.9966$,
for groups 1, 2, 3, and 4, respectively. The procedure is illustrated in the
top panel of Figure 3.

Applying DCT at level $\alpha=0.05$, we have $0\leq n^{+}$ and $1\leq n^{-}$
with 95% confidence. Furthermore, we conclude that $\theta_{4}<0$, since
$H^{+}_{4}$ is rejected by closed testing. We obtain a trivial lower bound for
$n^{+}$ because $H^{-}_{1}$ is not rejected by closed testing, and a non-
trivial lower bound for $n^{-}$ because $H^{+}_{2}\cap H^{+}_{3}\cap
H_{4}^{+}$ is rejected by closed testing.

Procedure 3.1 (Directional closed testing)$H_{1}^{-}\cap H_{2}^{+}\cap
H_{3}^{+}\cap H_{4}^{+}$$0.0271$$(0.0271)$$H_{1}^{-}\cap H_{2}^{+}\cap
H_{3}^{+}$$0.1209$$(0.1158)$$H_{1}^{-}\cap H_{2}^{+}\cap
H_{4}^{+}$$0.0203$$(0.0271)$$H_{1}^{-}\cap H_{3}^{+}\cap
H_{4}^{+}$$0.0203$$(0.0271)$$H_{2}^{+}\cap H_{3}^{+}\cap
H_{4}^{+}$$0.0203$$(0.0271)$$H_{1}^{-}\cap
H_{2}^{+}$$0.0806$$(0.1209)$$H_{1}^{-}\cap
H_{3}^{+}$$0.0806$$(0.1209)$$H_{1}^{-}\cap
H_{4}^{+}$$0.0135$$(0.0271)$$H_{2}^{+}\cap
H_{3}^{+}$$0.2039$$(0.2039)$$H_{2}^{+}\cap
H_{4}^{+}$$0.0135$$(0.0271)$$H_{3}^{+}\cap
H_{4}^{+}$$0.0135$$(0.0271)$$H_{1}^{-}$$0.0403$$(0.1209)$$H_{2}^{+}$$0.1020$$(0.2039)$$H_{3}^{+}$$0.4451$$(0.4451)$$H_{4}^{+}$$0.0068$$(0.0271)$Procedure
5.1 (Closed testing for qualitative interactions)$H_{1}^{-}\cap H_{2}^{+}\cap
H_{3}^{+}\cap H_{4}^{+}$$0.0403$$(0.0403)$$H_{1}^{-}\cap H_{2}^{+}\cap
H_{3}^{+}$$0.0403$$(0.0403)$$H_{1}^{-}\cap H_{2}^{+}\cap
H_{4}^{+}$$0.0403$$(0.0403)$$H_{1}^{-}\cap H_{3}^{+}\cap
H_{4}^{+}$$0.0403$$(0.0403)$$H_{2}^{+}\cap H_{3}^{+}\cap
H_{4}^{+}$$0.0403$$(0.0403)$$H_{1}^{-}\cap
H_{2}^{+}$$0.0403$$(0.0403)$$H_{1}^{-}\cap
H_{3}^{+}$$0.0403$$(0.0403)$$H_{1}^{-}\cap
H_{4}^{+}$$0.0403$$(0.0403)$$H_{2}^{+}\cap
H_{3}^{+}$$0.2039$$(0.2039)$$H_{2}^{+}\cap
H_{4}^{+}$$0.0135$$(0.0403)$$H_{3}^{+}\cap
H_{4}^{+}$$0.0135$$(0.0403)$$H_{1}^{-}$$0.0403$$(0.0403)$$H_{2}^{+}$$0.1020$$(0.2039)$$H_{3}^{+}$$0.4451$$(0.4451)$$H_{4}^{+}$$0.0068$$(0.0403)$
Figure 3: The example from Table 3. Top Plot: Procedure 3.1 (Directional
closed testing) with Simes’ local tests. Bottom Plot: Procedure 5.1 (Closed
testing for qualitative interactions) with Zhao et al., (2019) test for
$\\{H_{1}^{-}\\}\cup\\{H_{2}^{+}\cap H_{3}^{+}\cap H_{4}^{+}\\}$. Each node
represents a closed testing intersection hypothesis, and the arrows indicate
subset relationships. Below each intersection hypothesis, the corresponding
$p$-value and the closed testing adjusted $p$-value (in brackets). Gray nodes
are hypotheses rejected by the $5\%$ level closed testing procedure.

The procedure using the same adaptive local tests with partitioning is
illustrated in Figure 4. We can extract the confidence set for $n^{+}$ from
these non-rejected orthant hypotheses as follows. Since the number of positive
parameters in each non-rejected orthant hypothesis is at least one and at most
three, the bounds are $1\leq n^{+}\leq 3$ with 95% confidence. This is an
improvement over the bounds for $n^{+}$ with DCT. Alternatively, we can see
that the lower bound is one because $H_{1}^{-}\cap H_{2}^{-}\cap H_{3}^{-}\cap
H_{4}^{-}$ is rejected (by a local test of $H_{1}^{-}$, since for the
remaining coordinates $K_{2},K_{3},K_{4}$ were selected for testing).
Moreover, since the fourth individual hypothesis in each non-rejected
partition null hypothesis is non-positive, we further conclude that
$\theta_{4}\leq 0$. This is a weaker conclusion compared to $\theta_{4}<0$
obtained by DCT.

$K_{1}\cap K_{2}\cap K_{3}\cap K_{4}$$0.0203$$K_{1}\cap K_{2}\cap K_{3}\cap
H_{4}^{-}$$0.2039$$K_{1}\cap K_{2}\cap H_{3}^{-}\cap K_{4}$$0.0135$$K_{1}\cap
H_{2}^{-}\cap K_{3}\cap K_{4}$$0.0135$$H_{1}^{-}\cap K_{2}\cap K_{3}\cap
K_{4}$$0.0271$$K_{1}\cap K_{2}\cap H_{3}^{-}\cap H_{4}^{-}$ $0.1020$$K_{1}\cap
H^{-}_{2}\cap K_{3}\cap H_{4}^{-}$ $0.4451$$K_{1}\cap H^{-}_{2}\cap
H_{3}^{-}\cap K_{4}$ $0.0068$$H_{1}^{-}\cap K_{2}\cap K_{3}\cap H_{4}^{-}$
$0.1209$$H_{1}^{-}\cap K_{2}\cap H_{3}^{-}\cap K_{4}$ $0.0203$$H_{1}^{-}\cap
H_{2}^{-}\cap K_{3}\cap K_{4}$ $0.0203$$K_{1}\cap H_{2}^{-}\cap H_{3}^{-}\cap
H_{4}^{-}$$1$$H_{1}^{-}\cap K_{2}\cap H_{3}^{-}\cap
H_{4}^{-}$$0.0806$$H_{1}^{-}\cap H_{2}^{-}\cap K_{3}\cap
H_{4}^{-}$$0.0806$$H_{1}^{-}\cap H_{2}^{-}\cap H_{3}^{-}\cap
K_{4}$$0.0135$$H_{1}^{-}\cap H_{2}^{-}\cap H_{3}^{-}\cap H_{4}^{-}$$0.0403$
Figure 4: The example from Table 3. Each box represents an orthant hypothesis.
Below each orthant hypothesis, the corresponding $p$-value obtained by the
Simes local test. Gray boxes are orthant hypotheses rejected by the
partitioning procedure at the $5\%$ level.

###  5.2 Testing for qualitative interactions with follow-up inference

If testing for mixed signs of the parameter vector $\theta$ is of primary
concern, the approach described in § 5.3 can be adjusted so that DCT is
applied only if the hypothesis of no qualitative interaction,
$H_{0}:\\{n^{+}=0\\}\cup\\{n^{-}=0\\},$ is rejected. Gail and Simon, (1985)
analyzed the data of Table 3 to see whether they could demonstrate a
qualitative interaction at the $\alpha=5\%$ level. However, the likelihood
ratio test proposed by Gail and Simon, (1985) results in a $p$-value of
$0.0877$.

The test of Zhao et al., (2019), with selection threshold $\tau=1/2$, rejects
$H_{0}:\\{n^{+}=0\\}\cup\\{n^{-}=0\\}$ if $H_{\mathcal{S}^{-}}$ is rejected
with $f(\\{2p_{i},i\in\mathcal{S}^{-}\\})$ and $H_{\mathcal{S}^{+}}$ is
rejected with $f(\\{2q_{i},i\in\mathcal{S}^{+}\\})$. This is a valid test
since $\\{n^{+}=0\\}\subseteq H_{\mathcal{S}^{-}}$ and $\\{n^{-}=0\\}\subseteq
H_{\mathcal{S}^{+}}$. The $p$-value is $\max(0.0403,0.0203)=0.0403$ with
Simes’ combination test.

We propose the following procedure tailored for qualitative interactions:
start with Zhao et al., (2019) global test for $H_{0}$ and, if rejected,
continue with the DCT procedure.

######  Procedure 5.1 (Closed testing for qualitative interactions).

  1. Step 1

Apply Step 1 of procedure 2.1

  2. Step 2

Apply a level $\alpha$ closed testing procedure on the family of hypotheses
$\\{H_{i}^{-}:p_{i}\leq 1/2\\}\cup\\{H_{i}^{+}:p_{i}>1/2\\}$. Testing
$H_{\mathcal{I}}$ at level $\alpha$ is done by the local test

$\displaystyle\phi_{\mathcal{I}}=\left\\{\begin{array}[]{cc}\mathds{1}\\{\max\\{f(\\{2p_{i},i\in\mathcal{S}^{-}\\}),f(\\{2q_{i},i\in\mathcal{S}^{+}\\})\\}\leq\alpha\\}&\mathrm{if\,\,}\mathcal{I}\supseteq\mathcal{S}^{-}\mathrm{\,\,or\,\,}\mathcal{I}\supseteq\mathcal{S}^{+}\\\
\mathds{1}\\{f(\\{2p_{i},i\in\mathcal{I}\cap\mathcal{S}^{-}\\}\cup\\{2q_{j},j\in\mathcal{I}\cap\mathcal{S}^{+}\\})\leq\alpha\\}&\mathrm{otherwise}\\\
\end{array}\right.$

If $H_{0}$ is rejected, then the lower bounds for $n^{+}$ and $n^{-}$ are both
informative, i.e. $n^{+}>0,n^{-}>0$. From the bottom plot of Figure 3 we see
that at level $5\%$ procedure 5.1 rejects all hypotheses but $H_{2}^{+}\cap
H_{3}^{+}$, $H_{2}^{+}$ and $H_{3}^{+}$, giving $1\leq n^{+}\leq 3$, $1\leq
n^{-}\leq 3$ and $\theta_{1}>0$, $\theta_{4}<0$. This result improves the one
obtained by the DCT procedure 3.1 ($1\leq n^{-}\leq 4$ and $\theta_{4}<0$).
However, with procedure 5.1, rejecting $H_{0}$ is crucial: if $H_{0}$ is not
rejected, the closed testing procedure 5.1 does not reject any intersection
hypothesis. For instance, at $\alpha=3\%$, procedure 3.1 gives the same
conclusions as at $\alpha=5\%$, while procedure 5.1 is uninformative since no
rejections are made.

Note that the partitioning procedure with adaptive tests may employ the local
tests of procedure 5.1, giving

$\displaystyle\psi_{\mathcal{K}}=\left\\{\begin{array}[]{cc}\mathds{1}\\{\max\\{f(\\{2p_{i},i\in\mathcal{S}^{-}\\}),f(\\{2q_{i},i\in\mathcal{S}^{+}\\})\\}\leq\alpha\\}&\mathrm{if\,\,}\mathcal{K}\subseteq\mathcal{S}^{-}\mathrm{\,\,or\,\,}\mathcal{K}\subseteq\mathcal{S}^{+}\\\
\mathds{1}\\{f(\\{2p_{i},i\in\mathcal{K}^{c}\cap\mathcal{S}^{-}\\}\cup\\{2q_{j},j\in\mathcal{K}\cap\mathcal{S}^{+}\\})\leq\alpha\\}&\mathrm{otherwise}\\\
\end{array}\right.$

Applied to the data in Table 3, this procedure at level $\alpha=5\%$ gives the
same conclusions as Procedure 5.1 at the same level.

###  5.3 Enhancing meta-analysis

The first step in a meta-analysis is often the computation of a global null
$p$-value, which communicates the strength of the evidence against the null
hypothesis of no association in any of the studies. Rejecting the global null
does not rule out the possibility that this is due to a non-zero association
only in a single study (e.g., due to the particulars of the cohort in that
study, or due to bias). Therefore, it is of interest provide tight lower and
upper bounds on the number of studies with, e.g., a positive effect. Moreover,
for non-trivial bounds it is natural also to try to identify the studies where
there is an association. DCT should be used for this purpose if inference on
positive and negative associations are desired. Adaptive local tests with
partitioning should be used if it is enough to infer the lower and upper
bounds on $n^{+}$, and to identify positive and non-positive associations.
Since in a typical meta-analysis the studies are independent and the test
statistic in each study is approximately normal, the required conditions
$(A1)-(A2)$ are satisfied.

Fore a concrete example, we consider the data of Cooper et al., (2003), made
public by Konstantopoulos, (2011), where the effect of a modified school
calendar (with more frequent but shorter breaks) on student achievement is
examined. The meta-analysis used studies of $n=56$ schools in 11 districts.
The experimental design is a two-group comparison (modified calendar vs
traditional calendar) which involves computing the standardized mean
difference $y_{i}\sim N(\theta_{i},1)$, where $\theta_{i}>0$ indicates a
positive effects, i.e. a higher level of achievement in the group following
the modified school calendar. Zhao et al., (2019) showed that there is
evidence of a qualitative interaction at the $5\%$ level, i.e., that the
effect of the modified school calendar on student achievement is positive in
at least one study, and negative in at least one study.

Table 4 shows the results of our analysis. With DCT, the upper bound on
$n^{+}$ is the trivial bound of $n=56$. Adaptive local tests with partitioning
gives much more informative bounds: using Fisher’s combining method, the 95%
confidence bound for $n^{+}$ is 10 to 53, and we have four individual
discoveries; using mSimes, the 95% confidence bound for $n^{+}$ is 8 to 54,
and we have eight individual discoveries. The bounds are tighter than the
bounds obtained by counting discoveries in each direction, since local tests
used for the bounds can reject the intersection hypotheses even when base
hypotheses are not rejected. Using the procedure of Guo and Romano, (2015)
which is targeted for FWER control, the 95% confidence bound for $n^{+}$ is 8
to 55, with nine individual discoveries. Using the procedure of Guo and
Romano, (2015) which is targeted for FDR control we get the narrowest bounds
for $n^{+}$, 12 to 52. However, these are not 95% confidence bounds since this
procedure does not provide a simultaneous confidence guarantee.

Post-hoc bounds are obtained for $n^{+}(\mathcal{I}_{k})$ for $k=1,\ldots,n$
using Fisher’s combining function, where $\mathcal{I}_{k}$ represents the
indices of the top $k=|\mathcal{I}_{k}|$ schools with the largest (in absolute
value) estimated effects $|y_{i}|$.

Table 4:  The effect of modified school calendars on student achievement: intervals for $n^{+}$ and $n^{-}$ and corresponding number of discoveries $|\mathcal{D}^{+}|$ and $|\mathcal{D}^{-}|$, for different combining methods and procedures, with $95\%$ confidence. Results for Procedure 3.1 (DCT), the adaptive local tests with partitioning (AP), and the procedures in Guo and Romano, (2015) for FWER and FDR control (last two rows). $|\mathcal{D}^{-}|$ is the number of discoveries of negative parameters for DCT and non-positive parameters otherwise.  | $n^{+}$ | $n^{-}\ \backslash\ n^{-}+n^{0}$
---|---|---
Local test | Procedure | lower bound | upper bound | $|\mathcal{D}^{+}|$ | lower bound | upper bound | $|\mathcal{D}^{-}|$
Fisher | DCT | 9 | 56 | 4 | 0 | 47 | 0
AP | 10 | 53 | 3 | 46
Simes | DCT | 8 | 56 | 8 | 0 | 48 | 0
AP | 8 | 55 | 1 | 48
modified | DCT | 8 | 56 | 8 | 0 | 48 | 0
Simes | AP | 8 | 54 | 2 | 48
adaptive | DCT | 9 | 56 | 5 | 0 | 47 | 0
LRT | AP | 11 | 53 | 3 | 45
Guo-Romano | FWER | 8 | 55 | 8 | 1 | 48 | 1
FDR | 12 | 52 | 12 | 4 | 44 | 4

##  6 Simulations

We carry out simulation studies in order to evaluate the performance of our
suggested methods. Our main focus is on the evaluation of the DCT procedure
3.1 and the adaptive local tests with partitioning (AP) procedure, with
different combining methods of potential interest. Proposition 4.3 states that
the bounds of AP are at least as tight as with DCT. We aim to quantify the
potential gain in terms of tighter bounds. We use two representative combining
methods for this purpose: Fisher, which is a sum-based method, and modified
Simes (8), which is quantile based. Sum based tests, such as Fisher, are
better for the bounds but worse for discoveries. In the SM, we further
consider the adaptive likelihood ratio test (Al Mohamad et al.,, 2020), which
is a sum based test for normal test statistics, and Simes – it is inferior to
mSimes for the bounds as well as for (directional) decisions in our
simulations.

We compare our procedures to the FWER and FDR controlling procedures in Guo
and Romano, (2015), which were introduced in § 1.1. These procedures aim to
identify the positive and non-positive parameters, so they control the
directional error rates. We extract the lower bounds for $n^{+}(\mathcal{I})$
and $n^{-}(\mathcal{I})+n^{0}(\mathcal{I})$ from these procedures by counting
the number of discoveries that are claimed to positive, and non-positive,
respectively. The FWER controlling procedure is a good competitor for
discoveries, but does poorly for the bounds. The FDR controlling procedure is
less relevant, since unlike all the other procedures considered, the
probability that the number of positive or non-positive discoveries are
greater than the true number of positive and non-positive parameters in the
set of discoveries can be much larger than $\alpha$, see Figure 10 in the SM.
However, we included it since it is interesting that for the bounds it is not
necessarily better, even though there is no confidence guarantee.

Our data generating mechanism is as follows. For a total of $n=50$ parameters,
there are $n^{+}\in\\{0,\ldots,n\\}$ positive parameters and
$n^{-}\in\\{0,\ldots,n-n^{+}\\}$ negative parameters. The remaining parameters
are zero. The test statistics are generated independently from a Gaussian
distribution with standard deviation one and mean centered at the parameter
value, $\hat{\theta}_{i}\sim N(\theta_{i},1)$. The one sided $p$-value for
$H_{i}^{-}$ is $p_{i}=1-\Phi(\hat{\theta}_{i}),i=1,\ldots,n$. Our analysis is
based on $B=2000$ data generating mechanisms, and $\alpha=0.05$.

We display the average sum of the lower bounds on the number of positive and
non-positive parameters (the average length of the bounds is $n$ minus this
average) and the number of parameters discovered for each procedure, in Figure
5 and supplementary Figure 11. Our key findings are the following. First, AP
provides much tighter bounds than DCT. Second, the sum-based tests are better
for the bounds than the quantile based tests and than the FWER controlling
procedure in Guo and Romano, (2015). They can also be much better for the
bounds than the FDR controlling procedure in Guo and Romano, (2015) in data
generating mechanisms with fairly weak signal. Third, for making discoveries,
mSimes is competitive with the FWER controlling procedure in Guo and Romano,
(2015), and they are both much better than the sum-based tests. From the
simulations, we can support the following general guidance: mSimes is a good
choice if making discoveries is important in addition to the bounds, or if it
is expected that the fraction of parameters far enough from zero is small.
Otherwise it is best to use a sum based combining method, such as Fisher or
the ALRT.

0123$0$$5$$10$$15$$20$$25$Signal-to-noise ratio$\ell^{+}+\ell^{-}$

0123$0$$5$$10$$15$$20$$25$Signal-to-noise
ratio$|\mathcal{D}^{+}|+|\mathcal{D}^{-}|$

0123$0$$5$$10$$15$$20$$25$Signal-to-noise ratio$\ell^{+}+\ell^{-}$Fisher
APmSimes APFisher DCTmSimes DCTGR FWERGR FDR

0123$0$$5$$10$$15$$20$$25$Signal-to-noise
ratio$|\mathcal{D}^{+}|+|\mathcal{D}^{-}|$FishermSimesGR FWERGR FDR

Figure 5: The average sum of the lower bounds on positive and non-positive
parameters (left column, higher is better) and number of base hypotheses
rejected (right column, higher is better), versus the signal to noise ratio
(i.e., the absolute value of the non-null $\theta_{i}$ value), for the
following methods: AP and DCT with combining functions Fisher (AP FISHER and
DCT Fisher), and mSimes (AP mSIMES and DCT mSimes); the Guo-Romano procedure
for FWER control (GR FWER) and for FDR control (GR FDR). $n^{+}=n^{-}=15$ in
the first row; $n^{+}=30$ and $n^{-}=0$ in the second row. In the right
column, the absence of DCT procedures is due to the fact that they coincide
with the AP procedures (Proposition 4.3). Note that for GR-FWER and GR-FDR,
the sum of lower bounds on the number of positive and non-positive parameters
is their number of discoveries, so their curves are the same across panels in
each row.

##  7 Concluding remarks

In Procedure 3.1, the selection Step 1 has a cost of inflating the $p$-values
by a factor of 2. It is a reasonable cost, since it is no worse than the cost
of 2-sided testing. It is possible to consider more generally the selection of
$\\{H_{i}^{+}:p_{i}\leq\tau\\}\cup\\{H_{i}^{-}:p_{i}>\tau\\}$ for a
$\tau\in(0,1)$. Such a selection step inflates the $p_{i}$s by $1/\tau$, and
the $q_{i}$s by $1/(1-\tau)$. It may be useful when it is more important to
detect one direction over another. More generally, each parameter may be
accompanied by its own predefined selection threshold $\tau_{i},i=1,\ldots,n$.
In particular, it is possible to set $\tau_{i}\in\\{0,1\\}$ if the direction
for inference is a-priori known for parameter $\theta_{i}$. Selection is thus
more general, and the directional closed testing or partitioning procedures
are carried out as described, using the conditional $p$-values
$\\{p_{i}/\tau_{i}:p_{i}\leq\tau_{i}\\}\cup\\{q_{i}/(1-\tau_{i}):p_{i}>\tau_{i}\\}$.
Importantly, the same computational shortcuts can be used on these conditional
$p$-values, and inference is valid as long as $p_{1},\ldots,p_{n}$ are
independent and uniformly valid.

An open problem is how to deal with dependent $p$-values. For very specific
structured dependencies, as discussed in § 3.1, the procedures that condition
on the vector of signs may be applied with the appropriate local test. Without
conditioning, it is possible to apply the partitioning procedure as detailed
in § 4.1, as long as the local test is valid for the specific dependency
between the $p$-values. We provide a small example in § C, and leave for
future research the more comprehensive examination of unconditional methods
for dependent test statistics.

This work concentrated on post-hoc bounds. For simplicity of exposition, the
examples we presented were small scale. But in principle the methods can be
applied also in moderate and large scale testing. As shown in § 6, the bounds
can be tighter than bounds obtained from FDR controlling procedures when the
signal is weak and sparse. Specifically, our procedures can be useful for
studies that examine longitudinal data on independent individuals. For
example, in neuroimaging, the $p$-value from an experiment on an individual in
a region of interest in the brain may be reported for hundreds of individuals.
The lower bounds on $n^{+}(\mathcal{I})$ and $n^{-}(\mathcal{I})$ or
$n^{-}(\mathcal{I})+n^{0}(\mathcal{I})$ may be of great interest, for
$\mathcal{I}=[n]$ as well as for various $\mathcal{I}\subset[n]$.

For large scale testing, methods that use negative controls or masking have
been suggested for inference on the signs. For example, Leung and Tran, (2023)
(building upon the work of Lei and Fithian, 2018) suggests reflecting the
$p$-values of the one-sided hypotheses around 0.25 or 0.75. For FDR control,
these methods build on the knockoff+ method of Barber and Candès, (2015), so
the estimated false discovery proportion in $\mathcal{I}$ is the estimated
number of negative controls in $\mathcal{I}$ plus one, divided by
$|\mathcal{I}|$. The added one in the numerator makes these procedures less
relevant for small scale testing or when very few parameters are non-zero. An
open question is how to extend their inference to post-hoc simultaneous bounds
on $n^{+}(\mathcal{I})$ and $n^{-}(\mathcal{I})$. For this purpose, it may be
possible to build upon the work of Li et al., (2022). It will then be
interesting to compare the two approaches: conditioning on the signs and using
adjusted $p$-values as in the current manuscript, versus conditioning on the
masked $p$-values and using a knockoff approach.

In a high dimensional meta-analysis, where the studies are independent but
each study examines many (say $m$) features, can we first select features, and
then provide simultaneous post-hoc bounds? For example, for the neuroimaging
application, how can we address multiple regions of interest, and provide
tight bounds for each region while controlling a relevant error measure? If
$|\mathcal{R}|$ features out of $m$ are selected, and the post-hoc bounds are
constructed at level $|\mathcal{R}|\alpha/m$, then a relevant error is
controlled. This error is related to the average FWER over the selected, i.e.,
the expected proportion of features with at least one erroneous rejection out
of all the selected features, suggested in Benjamini and Bogomolov, (2014).
Briefly, the error we consider is the expected proportion of at least one non-
covering post-hoc bound, over the selected features. We plan to carry out in
the future a detailed analysis of this procedure, and its theoretical
guarantees for specific and general dependencies.

## References

  * Al Mohamad et al., (2020) Al Mohamad, D., Van Zwet, E. W., Cator, E., and Goeman, J. J. (2020).  Adaptive critical value for constrained likelihood ratio testing.  Biometrika, 107(3):677–688.
  * Anderson et al., (1982) Anderson, J. R., Bernstein, L., and Pike, M. C. (1982).  Approximate confidence intervals for probabilities of survival and quantiles in life-table analysis.  Biometrics, pages 407–416.
  * Barber and Candès, (2015) Barber, R. F. and Candès, E. J. (2015).  Controlling the false discovery rate via knockoffs.  The Annals of Statistics, 43(5):2055 – 2085.
  * Bauer et al., (1986) Bauer, P., Hackl, P., Hommel, G., and Sonnemann, E. (1986).  Multiple testing of pairs of one-sided hypotheses.  Metrika, 33(1):121–127.
  * Benjamini, (2010) Benjamini, Y. (2010).  Simultaneous and selective inference: Current successes and future challenges.  Biometrical Journal, 52(6):708–721.
  * Benjamini and Bogomolov, (2014) Benjamini, Y. and Bogomolov, M. (2014).  Selective inference on multiple families of hypotheses.  Journal of the Royal Statistical Society. Series B (Statistical Methodology), 76(1):297–318.
  * Benjamini and Heller, (2008) Benjamini, Y. and Heller, R. (2008).  Screening for partial conjunction hypotheses.  Biometrics, 64(4):1215–1222.
  * Benjamini et al., (2009) Benjamini, Y., Heller, R., and Daniel, Y. (2009).  Selective inference in complex research.  Journal of the royal statistical society, series A, 367(1906):4255––4271.
  * Benjamini and Yekutieli, (2005) Benjamini, Y. and Yekutieli, D. (2005).  False discovery rate-adjusted multiple confidence intervals for selected parameters.  Journal of the American Statistical Association, 100(469):71–81.
  * Blanchard et al., (2020) Blanchard, G., Neuvial, P., and Roquain, E. (2020).  Post hoc confidence bounds on false positives using reference families.  The Annals of Statistics, 48(3):1281 – 1303.
  * Bogomolov, (2023) Bogomolov, M. (2023).  Testing partial conjunction hypotheses under dependency, with applications to meta-analysis.  Electronic Journal of Statistics, 17(1):102 – 155.
  * Bogomolov and Heller, (2022) Bogomolov, M. and Heller, R. (2022).  Replicability across multiple studies.  arXiv:2210.00522.
  * Bohrer and Schervish, (1980) Bohrer, R. and Schervish, M. J. (1980).  An optimal multiple decision rule for signs of parameters.  Proceedings of the National Academy of Sciences, 77(1):52–56.
  * Bretz et al., (2016) Bretz, F., Hothorn, T., and Westfall, P. (2016).  Multiple comparisons using R.  CRC press.
  * Cooper et al., (2003) Cooper, H., Valentine, J. C., Charlton, K., and Melson, A. (2003).  The effects of modified school calendars on student achievement and on school and community attitudes.  Review of educational research, 73(1):1–52.
  * Cutler and Ederer, (1958) Cutler, S. J. and Ederer, F. (1958).  Maximum utilization of the life table method in analyzing survival.  Journal of chronic diseases, 8(6):699–712.
  * Dobriban, (2020) Dobriban, E. (2020).  Fast closed testing for exchangeable local tests.  Biometrika, 107(3):761–768.
  * Dunnett, (1955) Dunnett, C. W. (1955).  A multiple comparison procedure for comparing several treatments with a control.  Journal of the American Statistical Association, 50(272):1096–1121.
  * Ellis et al., (2020) Ellis, J. L., Pecanka, J., and Goeman, J. J. (2020).  Gaining power in multiple testing of interval hypotheses via conditionalization.  Biostatistics, 21(2):e65–e79.
  * Finner, (1994) Finner, H. (1994).  Testing multiple hypotheses: general theory, specific problems, and relationship to other multiple decision procedures.  Habilitationsschrift, Fachbereich IV Mathematik, Univ. Trier.
  * Finner, (1999) Finner, H. (1999).  Stepwise multiple test procedures and control of directional errors.  The Annals of Statistics, 27(1):274–289.
  * Finner and Strassburger, (2002) Finner, H. and Strassburger, K. (2002).  The partitioning principle: a powerful tool in multiple decision theory.  Annals of statistics, pages 1194–1213.
  * Finner et al., (2021) Finner, H., Tang, S.-Y., Cui, X., and Hsu, J. C. (2021).  Partitioning for confidence sets, confident directions, and decision paths.  In Handbook of Multiple Comparisons, pages 57–90. Chapman and Hall/CRC.
  * Fisher et al., (1983) Fisher, B., Redmond, C., Brown, A., Wickerham, D., Wolmark, N., Allegra, J., Escher, G., Lippman, M., Savlov, E., Wittliff, J., Fisher, E., Plotkin, D., Bowman, D., Wolter, J., Bornstein, R., Desser, R., and Frelick, R. (1983).  Influence of tumor estrogen and progesterone receptor levels on the response to tamoxifen and chemotherapy in primary breast cancer.  Journal of Clinical Oncology, 1(4):227–241.
  * Fisher, (1934) Fisher, R. A. (1934).  Statistical methods for research workers. Fifth Edition. Oliver and Boyd, Edinburgh and London.
  * Gail and Simon, (1985) Gail, M. and Simon, R. (1985).  Testing for qualitative interactions between treatment effects and patient subsets.  Biometrics, pages 361–372.
  * Goeman and Solari, (2022) Goeman, J. and Solari, A. (2022).  Conditional versus unconditional approaches to selective inference.  Preprint, available at https://arxiv.org/abs/2207.13480.
  * Goeman et al., (2010) Goeman, J., Solari, A., and Stijnen, T. (2010).  Three-sided hypothesis testing: Simultaneous testing of superiority, equivalence and inferiority.  Statistics in Medicine, 29:2117–2125.
  * Goeman et al., (2021) Goeman, J. J., Hemerik, J., and Solari, A. (2021).  Only closed testing procedures are admissible for controlling false discovery proportions.  The Annals of Statistics, 49(2):1218–1238.
  * Goeman et al., (2019) Goeman, J. J., Meijer, R. J., Krebs, T. J., and Solari, A. (2019).  Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing.  Biometrika, 106(4):841–856.
  * Goeman and Solari, (2011) Goeman, J. J. and Solari, A. (2011).  Multiple Testing for Exploratory Research.  Statistical Science, 26(4):584–597.
  * Greenwood et al., (1926) Greenwood, M. et al. (1926).  A report on the natural duration of cancer.  A Report on the Natural Duration of Cancer., (33).
  * Guo and Romano, (2015) Guo, W. and Romano, J. P. (2015).  On stepwise control of directional errors under independence and some dependence.  Journal of Statistical Planning and Inference, 163:21–33.
  * Hoang and Dickhaus, (2022) Hoang, A.-T. and Dickhaus, T. (2022).  Combining independent p-values in replicability analysis: A comparative study.  Journal of Statistical Computation and Simulation.
  * Holm, (1979) Holm, S. (1979).  A simple sequentially rejective multiple test procedure.  Scandinavian journal of statistics, pages 65–70.
  * IntHout et al., (2016) IntHout, J., Ioannidis, J. P. A., Rovers, M. M., and Goeman, J. J. (2016).  Plea for routinely presenting prediction intervals in meta-analysis.  BMJ OPEN, 6(7).
  * Jaljuli et al., (2022) Jaljuli, I., Benjamini, Y., Shenhav, L., Panagiotou, O. A., and Heller, R. (2022).  Quantifying replicability and consistency in systematic reviews.  Statistics in Biopharmaceutical Research, 0(0):1–14.
  * Jones and Tukey, (2000) Jones, L. and Tukey, J. (2000).  A sensible formulation of the significance test.  Psychol Methods, 5(4):411–414.
  * Karlin and Rubin, (1956) Karlin, S. and Rubin, H. (1956).  The theory of decision procedures for distributions with monotone likelihood ratio.  The Annals of Mathematical Statistics, pages 272–299.
  * Klein et al., (2007) Klein, J. P., Logan, B., Harhoff, M., and Andersen, P. K. (2007).  Analyzing survival curves at a fixed point in time.  Statistics in medicine, 26(24):4505–4519.
  * Konstantopoulos, (2011) Konstantopoulos, S. (2011).  Fixed effects and variance components estimation in three-level meta-analysis.  Research Synthesis Methods, 2(1):61–76.
  * Lehmann and Romano, (2005) Lehmann, E. L. and Romano, J. P. (2005).  Testing statistical hypotheses.  Springer Texts in Statistics. Springer, third edition.
  * Lei and Fithian, (2018) Lei, L. and Fithian, W. (2018).  AdaPT: An Interactive Procedure for Multiple Testing with Side Information.  Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(4):649–679.
  * Leung and Tran, (2023) Leung, D. and Tran, N. (2023).  Adaptive procedures for directional false discovery rate control.  Preprint, available at https://arxiv.org/abs/2201.05828.
  * Li et al., (2022) Li, J., Maathuis, M., and Goeman, J. (2022).  Simultaneous false discovery proportion bounds via knockoffs and closed testing.  Preprint, available at https://arxiv.org/abs/2212.12822.
  * Liu, (1997) Liu, W. (1997).  Control of directional errors with step-up multiple tests.  Statistics & probability letters, 31(3):239–242.
  * Marcus et al., (1976) Marcus, R., Peritz, E., and Gabriel, K. R. (1976).  Closed testing procedures with special reference to ordered analysis of variance.  Biometrika, 63(3):655–660.
  * Romano et al., (2011) Romano, J. P., Shaikh, A., and Wolf, M. (2011).  Consonance and the closure method in multiple testing.  The International Journal of Biostatistics, 7(1).
  * Sarkar et al., (2004) Sarkar, S. K., Sen, P. K., and Finner, H. (2004).  On two results in multiple testing.  Lecture Notes-Monograph Series, pages 89–99.
  * Shaffer, (1974) Shaffer, J. P. (1974).  Bidirectional unbiased procedures.  Journal of the American Statistical Association, 69(346):437–439.
  * Shaffer, (1980) Shaffer, J. P. (1980).  Control of directional errors with stagewise multiple test procedures.  The Annals of Statistics, 8(6):1342–1347.
  * Shaffer, (1986) Shaffer, J. P. (1986).  Modified sequentially rejective multiple test procedures.  Journal of the American Statistical Association, 81(395):826–831.
  * Shaffer, (2002) Shaffer, J. P. (2002).  Multiplicity, directional (type iii) errors, and the null hypothesis.  Psychological methods, 7(3):356.
  * Spjotvoll, (1972) Spjotvoll, E. (1972).  On the optimality of some multiple comparison procedures.  The Annals of Mathematical Statistics, 43(2):398–411.
  * Stefansson et al., (1988) Stefansson, G., Kim, W., and Hsu, J. (1988).  On confidence sets in multiple comparisons. statistical decision theory and related topics iv (ss gupta and jo berger, eds.) 2, 89–104.
  * Tian et al., (2021) Tian, J., Chen, X., Katsevich, E., Goeman, J., and Ramdas, A. (2021).  Large-scale simultaneous inference under dependence.  arXiv preprint arXiv:2102.11253.
  * Tukey, (1991) Tukey, J. (1991).  The philosophy of multiple comparisons.  Statistical Science, 6(1):100–116.
  * Vovk et al., (2022) Vovk, V., Wang, B., and Wang, R. (2022).  Admissible ways of merging p-values under arbitrary dependence.  The Annals of Statistics, 50(1):351–375.
  * Vovk and Wang, (2020) Vovk, V. and Wang, R. (2020).  Combining p-values via averaging.  Biometrika, 107(4):791–808.
  * Zhao et al., (2019) Zhao, Q., Small, D. S., and Su, W. (2019).  Multiple testing when many $p$-values are uniformly conservative, with application to testing qualitative interaction in educational interventions.  Journal of the American Statistical Association, 114(527):1291–1304.

##  Appendix A Proofs

###  A.1 Proof of Proposition 3.1

###### Proof.

From step 2 of the procedure, it follows that $\phi_{\mathcal{T}}$ is a
function of
$\\{2p_{i},i\in\mathcal{S}^{-}\cap\mathcal{T}\\}\cup\\{2q_{j},j\in\mathcal{S}^{+}\cap\mathcal{T}\\}$.
The validity of the local test given the vector of signs follows from the fact
that it combines valid $p$-values, since the validity of each conditional
$p$-values is guaranteed by assumption $(A1)$, and they are independent by
assumption $(A2)$. So $\forall\ \theta\in\mathbb{R}^{n}$,

$\mathbb{P}_{\\{\theta_{i}:i\in\mathcal{T}\\}}(\phi_{\mathcal{T}}=1\mid\\{sign(p_{i}-1/2):i\in\mathcal{T}\\})\leq\alpha.$

Moreover, since the $p$-values are independent (assumption ($A2$)), it follows
that

$\displaystyle\mathbb{P}_{\theta}(\phi_{\mathcal{T}}=1\mid\mathcal{S})=\mathbb{P}_{\\{\theta_{i}:i\in\mathcal{T}\\}}(\phi_{\mathcal{T}}=1\mid\\{sign(p_{i}-1/2):i\in\mathcal{T}\\}).$

Thus completing the proof.

∎

###  A.2 Proof of Proposition 4.1

###### Proof.

Suppose that the true $\theta$ lies in $\Theta_{\mathcal{K}}$ for some
$\mathcal{K}\subseteq[n]$. We have
$\psi_{|\mathcal{K}\cap\mathcal{I}|}(\mathcal{I})\leq\psi_{\mathcal{K}}$ for
every $\mathcal{I}$. Therefore, if $\psi_{\mathcal{K}}=0$, then
$|\mathcal{K}\cap\mathcal{I}|=n^{+}(\mathcal{I})\in\mathcal{N}^{+}(\mathcal{I})_{\alpha}$
for all $\mathcal{I}$, thus

$\mathbb{P}_{\theta}(n^{+}(\mathcal{I})\in\mathcal{N}^{+}(\mathcal{I})_{\alpha}\mathrm{\,\,}\forall\mathcal{I})\geq\mathbb{P}_{\theta}(\psi_{\mathcal{K}}=0)\geq
1-\alpha.$

Furthermore, the event
$\\{n^{+}(\mathcal{I})\in\mathcal{N}^{+}(\mathcal{I})_{\alpha}\\}$ implies
$\\{\min(\mathcal{N}^{+}(\mathcal{I})_{\alpha})\leq
n^{+}(\mathcal{I})\leq\max(\mathcal{N}^{+}(\mathcal{I})_{\alpha})\\}$ for all
$\mathcal{I}$, thus
$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})=\min(\mathcal{N}^{+}(\mathcal{I})_{\alpha})$,
$\tilde{\ell}^{-}_{\alpha}(\mathcal{I})=|\mathcal{I}|-\max(\mathcal{N}^{+}(\mathcal{I})_{\alpha})$
satisfy (3). ∎

###  A.3 Proof of Proposition 4.2

###### Proof.

Proposition D.1 shows that Algorithm 1 returns the bounds
$\tilde{\ell}_{\alpha}^{+}(\mathcal{I})$ and
$\tilde{\ell}_{\alpha}^{-}(\mathcal{I})$ derived from the partitioning
procedure with adaptive local tests (4.2) with at most
$O(|\mathcal{I}|^{2}\cdot\max(1,|\mathcal{I}^{c}|^{2}))$ computation.

Suppose the true $\theta\in\Theta_{\mathcal{K}}$ for some
$\mathcal{K}\subseteq[n]$, and let
$\tilde{\mathcal{T}}=\\{\mathcal{K}^{c}\cap\mathcal{S}^{-}\\}\cup\\{\mathcal{K}\cap\mathcal{S}^{+}\\}$.
The partitioning procedure testing the orthant hypothesis $J_{\mathcal{K}}$ by
using $\psi_{\mathcal{K}}=\phi_{\tilde{\mathcal{T}}}$, and

$\displaystyle\sup_{\theta\in\Theta_{\mathcal{K}}}\mathbb{P}_{\theta}(\phi_{\mathcal{K}}=1\mid\mathcal{S})\leq\sup_{\theta\in
H_{\tilde{\mathcal{T}}}}\mathbb{P}_{\theta}(\phi_{\tilde{\mathcal{T}}}=1\mid\mathcal{S})\leq\alpha$

where the first inequality follows from $J_{\mathcal{K}}\subseteq
H_{\tilde{\mathcal{T}}}$ and the second inequality follows from Proposition
3.1. This establishes control of type I error at level $\alpha$, conditional
on the vector of signs $\mathcal{S}$, for the test $\phi_{\mathcal{K}}$ of the
true orthant hypothesis $J_{\mathcal{K}}$.

Furthermore,

$\displaystyle\mathbb{P}_{\theta}(\tilde{\mathcal{T}}=\emptyset)=\mathbb{P}_{\theta}(\mathcal{S}^{-}=\mathcal{K})=$
$\displaystyle\prod_{i\in\mathcal{K}}\mathbb{P}_{\theta}(p_{i}\leq
0.5)\prod_{j\notin\mathcal{K}}\mathbb{P}_{\theta}(p_{j}>0.5)\geq\mathbb{P}_{0}(\mathcal{S}^{-}=\mathcal{K})\geq
2^{-n}.$ (19)

Thus

$\mathbb{P}_{\theta}(\psi_{\mathcal{K}}=1)=\sum_{\mathcal{S}}\mathbb{P}_{\theta}(\phi_{\tilde{\mathcal{T}}}=1|\mathcal{S})\mathbb{P}_{\theta}(\mathcal{S})\leq\alpha(1-\mathbb{P}_{\theta}(\tilde{\mathcal{T}}=\emptyset))\leq\alpha(1-2^{-n}).$

This ensures unconditional control of type I error at level
$\tilde{\alpha}=\alpha(1-2^{-n})$ for the test $\phi_{\mathcal{K}}$ of the
true orthant hypothesis $J_{\mathcal{K}}$. See also § 1.2 of the Supplementary
Material to Al Mohamad et al., (2020) for the unconditional type I error
control with the adaptive likelihood ratio test.

The previous results together with Proposition 4.1 imply the conditional and
unconditional coverage (15) and (16).

∎

###  A.4 Proof of Proposition 4.3

###### Proof.

In order to prove that
$\ell^{+}_{\alpha}(\mathcal{I})\leq\tilde{\ell}^{+}_{\alpha}(\mathcal{I})$ for
every $\mathcal{I}$, we proceed by contradiction. Assume that
$\exists\,\mathcal{I}$ such that
$\ell^{+}_{\alpha}(\mathcal{I})>\tilde{\ell}^{+}_{\alpha}(\mathcal{I})$. We
have
$\ell^{+}_{\alpha}(\mathcal{I})=\ell^{+}_{\alpha}(\mathcal{I}\cap\mathcal{S}^{-})$
and by Lemma 1 in Goeman et al., (2021):

$\displaystyle\ell^{+}_{\alpha}(\mathcal{I\cap\mathcal{S}^{-}})=\min_{\mathcal{V}\subseteq[n]}(|(\mathcal{I}\cap\mathcal{S}^{-})\setminus\mathcal{V}|:\phi_{\mathcal{V}}=0).$
(20)

Because the null hypothesis that $n^{+}(\mathcal{I})$ is equal to
$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})$ is not rejected at level $\alpha$,
then $\exists\,\mathcal{K}$ such that
$|\mathcal{K}\cap\mathcal{I}|=\tilde{\ell}^{+}_{\alpha}(\mathcal{I})$ for
which $J_{\mathcal{K}}$ is not rejected at level $\alpha$. Equivalently,
$\exists\,\mathcal{U}=(\mathcal{K}^{c}\cap\mathcal{S}^{-})\cup(\mathcal{K}\cap\mathcal{S}^{+})$
such that $\phi_{\mathcal{U}}=0$ and

$|(\mathcal{I}\cap\mathcal{S}^{-})\setminus\mathcal{U}|=|\mathcal{I}\cap\mathcal{S}^{-}\cap\mathcal{K}|\leq\tilde{\ell}^{+}_{\alpha}(\mathcal{I})<\ell^{+}_{\alpha}(\mathcal{I}).$

On the other hand, from equation (20) it follows that
$\ell^{+}_{\alpha}(\mathcal{I})\leq|(\mathcal{I}\cap\mathcal{S}^{-})\setminus\mathcal{U}|$
contradicting the assumption that
$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})<\ell^{+}_{\alpha}(\mathcal{I})$.

$\ell^{-}_{\alpha}(\mathcal{I})\leq\tilde{\ell}^{-}_{\alpha}(\mathcal{I})$ for
every $\mathcal{I}$ can be proved in analogous way.

In order to prove that
$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})=\ell^{+}_{\alpha}(\mathcal{I})$ for
every $\mathcal{I}\subseteq\mathcal{S}^{-}$, write:

$\displaystyle\tilde{\ell}_{\alpha}^{+}(\mathcal{I})$ $\displaystyle=$
$\displaystyle\min\\{v\in\\{0,\ldots,n\\}:\psi_{v}(\mathcal{I})=0\\}$
$\displaystyle=$
$\displaystyle\min\\{v\in\\{0,\ldots,n\\}:\min_{\mathcal{K}\subseteq[n]:|\mathcal{K}\cap\mathcal{I}|=v}\phi_{(\mathcal{K}^{c}\cap\mathcal{S}^{-})\cup(\mathcal{K}\cap\mathcal{S}^{+})}=0\\}$
$\displaystyle=$
$\displaystyle\min_{\mathcal{K}\subseteq[n]}\\{|\mathcal{K}\cap\mathcal{I}|:\phi_{(\mathcal{K}^{c}\cap\mathcal{S}^{-})\cup(\mathcal{K}\cap\mathcal{S}^{+})}=0\\}$
$\displaystyle=$
$\displaystyle\min_{\mathcal{V}\subseteq[n]}\\{|\mathcal{V}^{c}\cap\mathcal{I}|:\phi_{(\mathcal{V}\cap\mathcal{S}^{-})\cup(\mathcal{V}^{c}\cap\mathcal{S}^{+})}=0\\}$

If $\mathcal{I}\subseteq\mathcal{S}^{-}$, then
$\mathcal{I}\setminus\\{(\mathcal{V}\cap\mathcal{S}^{-})\cup(\mathcal{V}^{c}\cap\mathcal{S}^{+})\\}=|\mathcal{I}\setminus\mathcal{V}|$
for any $\mathcal{V}$, thus

$\tilde{\ell}_{\alpha}^{+}(\mathcal{I})=\min_{\mathcal{V}\subseteq[n]}\\{|\mathcal{I}\setminus\mathcal{V}|:\phi_{(\mathcal{V}\cap\mathcal{S}^{-})\cup(\mathcal{V}^{c}\cap\mathcal{S}^{+})}=0\\}=\min_{\mathcal{V}\subseteq[n]}\\{|\mathcal{I}\setminus\mathcal{V}|:\phi_{\mathcal{V}}=0\\}=\ell_{\alpha}^{+}(\mathcal{I}).$

$\tilde{\ell}^{-}_{\alpha}(\mathcal{I})=\ell^{-}_{\alpha}(\mathcal{I})$ for
every $\mathcal{I}\subseteq\mathcal{S}^{+}$ can be proved in analogous way.

∎

##  Appendix B Case $n=2$

Figure 6 displays the rejection regions for base hypotheses for the
partitioning with adaptive local tests (top left plot) and directional closed
testing (top right plot) using Fisher’s combining functions.

We call a closed testing procedure _consonant_ if the rejection of the
intersection hypothesis $H_{\mathcal{I}}$ implies the rejection of at least
one of its component hypotheses $H_{i}$, $i\in\mathcal{I}$ (Romano et al.,,
2011). From the top-right and bottom-right plots of Figure 6, we can observe
that the directional closed testing procedure with Fisher’s combination is
also not consonant. For instance, when $\alpha=0.2$ as shown in the plots, if
$(p_{1},p_{2})=(0.101,0.101)$, then $H_{1}^{-}\cap H_{2}^{-}$ is rejected by
Fisher’s combination test $f_{\mathrm{Fisher}}(2p_{1},2p_{2})=0.1713<\alpha$.
However, neither $H_{1}^{-}$ nor $H_{2}^{-}$ is rejected because
$2p_{i}>\alpha$, $i=1,2$.

The directional closed testing procedure with Simes’ local tests is also not
consonant for $n>2$, although it is consonant for $n=2$, as can be seen from
Figure 1.

Directional Closed Testing$\alpha/2$$p_{1}$$p_{2}$Partitioning with Adaptive
Tests$2\alpha/3$$p_{1}$$p_{2}$$n^{+}=2$$n^{+}\geq 1$$n^{+}=1$$n^{+}\leq
1$$n^{+}=0$

Directional Closed
Testing$H_{1}^{-},H_{2}^{-}$$H_{1}^{-},H_{2}^{+}$$H_{1}^{+},H_{2}^{+}$$H_{1}^{+},H_{2}^{-}$$H_{1}^{-}$$H_{1}^{+}$$H_{2}^{-}$$H_{2}^{+}$$p_{1}$$p_{2}$$\alpha/2$Partitioning
with Adaptive
Tests$H_{1}^{-},H_{2}^{-}$$H_{1}^{-},K_{2}$$K_{1},K_{2}$$K_{1},H_{2}^{-}$$H_{1}^{-}$$K_{1}$$H_{2}^{-}$$K_{2}$$p_{1}$$p_{2}$$\alpha/2$

Figure 6: The top row illustrates regions leading to inference about $n^{+}$
with $(1-\alpha)$ confidence for the partitioning procedure with adaptive
tests (top-left plot) and the directional closed testing procedure (top-right
plot) with Fisher’s combining function. The gray scale indicates the type of
inference. The bottom row illustrates regions leading to the rejection of base
hypotheses for partitioning with adaptive local tests (bottom left plot) and
directional closed testing (bottom right plot). The pattern indicates the
number of hypotheses rejected at level $\alpha$: one (dots) or two (diagonal
lines). Each plot is based on $\alpha=0.2$.

From Figure 7, it can be observed that when $\alpha=0.3$ and
$(p_{1},p_{2})=(0.19,0.19)$, we are able to reject both $H_{1}^{-}\cap K_{1}$
and $K_{1}\cap H_{2}^{-}$ because $2p_{i}<2\alpha/3$, where $i=1,2$. However,
we cannot reject $H_{1}^{-}\cap H_{2}^{-}$ since
$f_{\mathrm{Fisher}}(2p_{1},2p_{2})=0.423>\alpha$. Therefore, we conclude that
$n^{+}$ cannot be equal to 1, i.e.,
$n^{+}\in\mathcal{N}^{+}_{\alpha}=\\{0,2\\}$. In this case, however, if we
calculate the lower and upper bounds for $n^{+}$ as
$\min(\mathcal{N}^{+}_{\alpha})$ and $\max(\mathcal{N}^{+}_{\alpha})$,
respectively, we obtain the uninformative result of $0$ and $2$.

Partitioning with Adaptive Tests$2\alpha/3$$p_{1}$$p_{2}$ Figure 7: The
partitioning procedure with adaptive tests and Fisher’s combining function
concludes $n^{+}\in\\{0,2\\}$ with $(1-\alpha)$ confidence when
$(p_{1},p_{2})$ belongs to the regions in black. The plot is based on
$\alpha=0.3$.

Finally, Figure 8 illustrates the rejection regions for $n=2$ of two FWER-
controlling procedures for the family of hypotheses (4). The first is the
Holm-type procedure proposed by Guo and Romano, (2015), which requires
conditions (A1)-(A2) in § 3. The second procedure is the
$\mathrm{\check{S}id\acute{a}k}$-type procedure, also known as the “Sp cross”
originally introduced by Spjotvoll, (1972) and modified by Bohrer and
Schervish, (1980), which requires require only independence of the $n$
$p$-values. See § C for its derivation as a partitioning procedure.

Spjotvoll, (1972) $1-(1-\alpha)^{1/2}$$p_{1}$$p_{2}$Guo and Romano,
(2015)$\alpha/(1+\alpha)$$\alpha/(2+\alpha)$$p_{1}$$p_{2}$ Figure 8:  Holm-
type procedure of Guo and Romano, (2015) (left plot) and
$\mathrm{\check{S}id\acute{a}k}$-type procedure of Spjotvoll, (1972) (right
plot). Each plot is based on $\alpha=0.2$.

##  Appendix C The partitioning procedure based on unconditional $p$-values

The partitioning procedure based on unconditional $p$-values
$p_{1},\ldots,p_{n}$ and $q_{1},\ldots,q_{n}$ tests each partitioning
hypothesis $J_{\mathcal{K}}$ by

$\displaystyle\psi_{\mathcal{K}}$ $\displaystyle=$
$\displaystyle\mathds{1}\\{f(\\{p_{i},i\in\mathcal{K}^{c}\\}\cup\\{q_{j},j\in\mathcal{K}\\})\leq\alpha\\}.$

If we assume that $p_{1},\ldots,p_{n}$ are independent, we can use
$\mathrm{\check{S}id\acute{a}k}$’s combining function
$f_{\mathrm{\check{S}id\acute{a}k}}(x_{1},\ldots,x_{d})=1-(1-\min(x_{1},\ldots,x_{d}))^{d}$
and obtain the following FWER controlling procedure: if $p_{i}\leq
1-(1-\alpha)^{(1/n)}$, declare $\theta_{i}>0$; if $q_{i}\leq
1-(1-\alpha)^{(1/n)}$, declare $\theta_{i}\leq 0$. The right plot in Figure 8
illustrates the rejection region for $n=2$.

Several combining function are valid for arbitrary dependence among $p$-values
$p_{1},\ldots,p_{n}$ (Vovk et al.,, 2022). The most well-known one is arguably
the Bonferroni’s combining function
$f_{\mathrm{Bonferroni}}(x_{1},\ldots,x_{d})=d\min(x_{1},\ldots,x_{d})\wedge
1$. When $f_{\mathrm{Bonferroni}}$ is used in the partitioning procedure, it
gives the FWER controlling procedure of Bauer et al., (1986): if
$p_{i}\leq\alpha/n$, declare $\theta_{i}>0$; if $q_{i}\leq\alpha/n$, declare
$\theta_{i}\leq 0$.

Positively correlated $p$-values $p_{1},\ldots,p_{n}$ arises in comparing $n$
groups with a common control group in a one-way layout. For simplicity, let’s
assume that there are $m$ observations per group. The parameters of interest
are $\theta_{i}=\mu_{i}-\mu_{0}$ with corresponding test statistics
$\hat{\theta}_{i}=(\hat{\mu}_{i}-\hat{\mu}_{0})/\sqrt{2\hat{\sigma}^{2}/m}$
following $t_{nm-1}$, a Student’s $t$ distribution with $nm-1$ degrees of
freedom. The $p$-values are $p_{i}=\mathbb{P}(t_{nm-1}\geq\hat{\theta}_{i})$,
for $i=1,\ldots,n$. It is worth noting that the $p$-values are correlated
since $(\hat{\theta}_{1},\ldots,\hat{\theta}_{n})$ follows an $n$-variate $t$
distribution with $nm-1$ degrees of freedom and correlation matrix with off-
diagonal elements equal to $1/2$. Dunnett, (1955) test for $J_{\mathcal{K}}$
is given by

$\displaystyle\psi_{\mathcal{K}}$ $\displaystyle=$
$\displaystyle\mathds{1}\\{\max(\\{\hat{\theta}_{i},i\in\mathcal{K}^{c}\\}\cup\\{-\hat{\theta}_{j},j\in\mathcal{K}\\})>c_{\alpha}(\mathcal{K})\\}$
(21)

where $c_{\alpha}(\mathcal{K})$ represents the $(1-\alpha)$-quantile of the
maximum of an $n$-variate $t$ distribution with a correlation matrix that has
off-diagonal elements whose sign depends on $\mathcal{K}$.

To illustrate this approach, we consider the recovery data from Bretz et al.,
(2016). A company conducted a study to evaluate the effectiveness of
specialized heating blankets in assisting post-surgical body heat. Four types
of blankets, labeled b0, b1, b2, and b3, were tested on surgical patients to
measure their impact on recovery times. The b0 blanket served as the standard
option already utilized in different hospitals. The main focus was to assess
the recovery time, measured in minutes, of patients randomly assigned to one
of the four treatments. A shorter recovery time would indicate a more
effective treatment. The key question addressed in this study is whether
blanket types b1, b2, or b3 modify recovery time compared to b0. To perform a
formal analysis of the data, we assume an one-way layout

$y_{ij}=\gamma+\mu_{i}+\epsilon_{ij}$

where $\gamma+\mu_{i}$ represents the expected recovery time for blanket b$i$,
$i=0,1,2,3$, with i.i.d. errors $\epsilon_{ij}\sim N(0,\sigma^{2})$.

Dunnett, (1955)’s procedure is the standard procedure used to address the
many-to-one comparisons problem. In the absence of specific information about
the direction of the effect, we typically formulate two-sided null hypotheses
of the form $H_{i}:\theta_{i}=0$, where $\theta_{i}=\mu_{i}-\mu_{0}$ and
$i=1,2,3$.

Table 5 presents the multiplicity-adjusted $p$-values for both single-step and
step-down Dunnett’s procedures applied to the point null hypotheses
$H_{i}:\theta_{i}=0$. Although the step-down Dunnett procedure is more
powerful, the single-step Dunnett’s procedure provides simultaneous confidence
intervals. These intervals enable directional conclusions to be drawn without
the risk of Type III errors (Finner et al.,, 2021). At level $\alpha=0.1$, we
conclude that blanket b2 leads to significantly lower recovery times as
compared to the standard blanket b0. The results of one-sided Dunnett’s
procedures are also provided for comparison, assuming a priori specific
information about the direction of the effect.

Table 5: Dunnett’s procedures for the recovery data: single-step and step-down adjusted $p$-values for two-sided and one-sided hypotheses. Hypothesis | Test statistic |  $p$-value | Dunnett’s adjusted $p$-value
---|---|---|---
|  |  | single-step | step-down
$H_{1}:\theta_{1}=0$ | $-1.330$ | 0.191 | 0.456 | 0.192
$H_{2}:\theta_{2}=0$ | $-4.656$ | $<0.001$ | $<0.001$ | $<0.001$
$H_{3}:\theta_{3}=0$ | $-1.884$ | 0.067 | 0.884 | 0.127
$H^{+}_{1}:\theta_{1}\geq 0$ | $-1.330$ | 0.096 | 0.241 | 0.096
$H^{+}_{2}:\theta_{2}\geq 0$ | $-4.656$ | $<0.001$ | $<0.001$ | $<0.001$
$H^{+}_{3}:\theta_{3}\geq 0$ | $-1.884$ | 0.034 | 0.092 | 0.064
$H^{-}_{1}:\theta_{1}\leq 0$ | $-1.330$ | 0.904 | 0.996 | 0.996
$H^{-}_{2}:\theta_{2}\leq 0$ | $-4.656$ | 1 | 1 | 1
$H^{-}_{3}:\theta_{3}\leq 0$ | $-1.884$ | 0.966 | 1 | 0.997

Table 6 shows the $p$-value for the orthant hypotheses using Dunnett’s tests
(21). The computation of $p$-values is performed with the glht function of the
multcomp R package. Table 7 gives multiplicity adjusted $p$-values for each
pair $H_{i}^{-}:\theta_{i}\leq 0$, $K_{i}:\theta_{i}>0$. At level
$\alpha=0.1$, we conclude that blanket b2 and b3 lead to recovery times that
are at most as large as the standard blanket b0.

Table 6: Partitiong procedure using Dunnett’s tests: $p$-values for orthant hypotheses for the recovery data. Orthant hypothesis | Test statistic | Critical value $\alpha=0.1$ |  $p$-value
---|---|---|---
$K_{1}\cap K_{2}\cap K_{3}$ | $4.656$ | 1.843 | $<0.001$
$K_{1}\cap K_{2}\cap H_{3}^{-}$ | $4.656$ | $1.873$ | $<0.001$
$H_{1}^{-}\cap K_{2}\cap K_{3}$ | $4.656$ | $1.867$ | $<0.001$
$K_{1}\cap H_{2}^{-}\cap K_{3}$ | $1.884$ | $1.867$ | $0.097$
$K_{1}\cap H_{2}^{-}\cap H_{3}^{-}$ | $1.330$ | $1.867$ | $0.259$
$H_{1}^{-}\cap K_{2}\cap H_{3}^{-}$ | $4.656$ | $1.866$ | $<0.001$
$H_{1}^{-}\cap H_{2}^{-}\cap K_{3}$ | $1.884$ | $1.874$ | $0.098$
$H_{1}^{-}\cap H_{2}^{-}\cap H_{3}^{-}$ | $-1.330$ | $1.843$ | $0.996$
Table 7: Partitiong procedure using Dunnett’s tests: adjusted $p$-values for base hypotheses for the recovery data. Hypothesis | Adjusted $p$-value | Hypothesis | Adjusted $p$-value
---|---|---|---
$H^{-}_{1}:\theta_{1}\leq 0$ | 0.996 | $K_{1}:\theta_{1}>0$ | 0.259
$H^{-}_{2}:\theta_{2}\leq 0$ | 0.996 | $K_{2}:\theta_{2}>0$ | $<0.001$
$H^{-}_{3}:\theta_{3}\leq 0$ | 0.996 | $K_{3}:\theta_{3}>0$ | 0.098

##  Appendix D Exact shortcut for the partitioning procedure with adaptive
tests

We first discuss the computation of the lower bound
$\tilde{\ell}^{+}_{\alpha}$ for $n^{+}$ and $\tilde{\ell}^{-}_{\alpha}$ for
$n^{-}+n^{0}$. Suppose we have observed $\mathcal{S}^{-}$ of size
$|\mathcal{S}^{-}|=s$, and we want to check whether the test in (4.2) rejects
all the hypotheses $J_{\mathcal{K}}$ with $|\mathcal{K}|=k$ at level $\alpha$:

$\displaystyle
f_{k}=\max_{\mathcal{K}:|\mathcal{K}|=k}f(\\{2p_{i},i\in\mathcal{S}^{-}\cap\mathcal{K}^{c}\\}\cup\\{2q_{i},i\in\mathcal{S}^{+}\cap\mathcal{K}\\})$
$\displaystyle\leq$ $\displaystyle\alpha.$

Vandermonde’s convolution

$\displaystyle{n\choose k}$ $\displaystyle=$
$\displaystyle\sum_{v=0}^{k}{s\choose k-v}{n-s\choose v}$

states that for each $v=0,\ldots,k$ there are ${s\choose k-v}{n-s\choose v}$
sets $\mathcal{K}$ of size $k$ such that $|\mathcal{S}^{+}\cap\mathcal{K}|=v$
and $|\mathcal{S}^{-}\cap\mathcal{K}|=k-v$ (or equivalently,
$|\mathcal{S}^{-}\cap\mathcal{K}^{c}|=s-k+v$). Then, the maximization problem
becomes

$f_{k}=\max_{v\in\\{\max(0,k-s),\ldots,\min(k,n-s)\\}}\mathop{\max_{\mathcal{K}:|\mathcal{S}^{+}\cap\mathcal{K}|=v,}}_{|\mathcal{S}^{-}\cap\mathcal{K}|=k-v}f(\\{2p_{i},i\in\mathcal{S}^{-}\cap\mathcal{K}^{c}\\}\cup\\{2q_{i},i\in\mathcal{S}^{+}\cap\mathcal{K}\\})$

For any increasing function $f$, the maximum has solution with the $v$ largest
$p$-values $q_{i}$ with $i\in\mathcal{S}^{+}$ and the $s-k+v$ largest
$p$-values $p_{i}$ with $i\in\mathcal{S}^{-}$, i.e.

$\displaystyle
f_{k}=\max_{v\in\\{\max(0,k-s),\ldots,\min(k,n-s)\\}}f(\\{2p_{(k-v+1)},\ldots,2p_{(s)}\\}\cup\\{2q_{(n-s-v+1)},\ldots,2q_{(n-s)}\\}).$
(22)

In order to compute $f_{k}$ in (22) for $k=0,\ldots,n$, we can use a nested
loop, where the number of iterations of the inner loop (i.e. the index $v$ in
(22) from $\max(0,k-s)$ to $\min(k,n-s)$) depends on the value of the outer
loop’s index (i.e. $k$ from 0 to $n$) and the size $s$ of $\mathcal{S}^{-}$.
The total complexity for the two loops is $O(n^{2})$. The maximum number of
iterations happens when $s\in\\{n/2-1,n/2,n/2+1\\}$ if $n$ is even, and
$s\in\\{(n-1)/2,(n+1)/2\\}$ if $n$ is odd; the mininum number of iterations
happens when $s\in\\{0,n\\}$.

The procedure we have just discussed is just a special case of the following 1
for the derivation of the lower bounds
$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})$ and
$\tilde{\ell}^{-}_{\alpha}(\mathcal{I})$ for a generic subset $\mathcal{I}$.

Input : right-tailed $p$-values $p_{1},\ldots,p_{n}$; combining function
$f(\cdot)$; subset $\mathcal{I}\subset[n]$; level $\alpha$

Output : Confidence bounds $l^{+}_{\alpha}(\mathcal{I})$ and
$u^{+}_{\alpha}(\mathcal{I})$, $p$-values $f_{v}(\mathcal{I})$ for
$J_{v}(\mathcal{I}):n^{+}(\mathcal{I})=v$, for $v=0,\ldots,|\mathcal{I}|$

Initialize :

$\mathcal{S}^{-}=\\{i\in[n]:p_{i}\leq 1/2\\}$, $|\mathcal{S}^{-}|=s$,
$f_{s}(\mathcal{I})=0$ ;

$\mathbf{a}=(a_{1},\ldots,a_{|\mathcal{S}^{-}\cap\mathcal{I}|})$ with
$a_{1}\geq\ldots\geq a_{|\mathcal{S}\cap\mathcal{I}|}$ sorted values of
$\\{2p_{i}:i\in\mathcal{S}^{-}\cap\mathcal{I}\\}$ ;

$\mathbf{b}=(b_{1},\ldots,b_{|\mathcal{S}^{-}\cap\mathcal{I}^{c}|})$ with
$b_{1}\geq\ldots\geq b_{|\mathcal{S}^{-}\cap\mathcal{I}^{c}|}$ sorted values
of $\\{2p_{i}:i\in\mathcal{S}^{-}\cap\mathcal{I}^{c}\\}$ ;

$\mathbf{c}=(c_{1},\ldots,c_{|\mathcal{S}^{+}\cap\mathcal{I}|})$ with
$c_{1}\geq\ldots\geq c_{|\mathcal{S}^{+}\cap\mathcal{I}|}$ sorted values of
$\\{2q_{i}:i\in\mathcal{S}^{+}\cap\mathcal{I}\\}$ ;

$\mathbf{d}=(d_{1},\ldots,d_{|\mathcal{S}^{+}\cap\mathcal{I}^{c}|})$ with
$d_{1}\geq\ldots\geq d_{|\mathcal{S}^{+}\cap\mathcal{I}^{c}|}$ sorted values
of $\\{2q_{i}:i\in\mathcal{S}^{+}\cap\mathcal{I}^{c}\\}$ ;

1 for  _$v\leftarrow 0$ to $|\mathcal{I}|$_ do

2 for  _$u\leftarrow 0$ to $|\mathcal{I}^{c}|$_ do

3

4 for  _$k\leftarrow\max(0,v-|\mathcal{S}^{-}\cap\mathcal{I}|)$ to
$\min(v,|\mathcal{S}^{+}\cap\mathcal{I}|)$_ do

5 for  _$j\leftarrow\max(0,u-|\mathcal{S}^{-}\cap\mathcal{I}^{c}|)$ to
$\min(u,|\mathcal{S}^{+}\cap\mathcal{I}^{c}|)$_ do

6

7
$f_{k,j}=f(\\{a_{1},\ldots,a_{k-v+|\mathcal{S}^{-}\cap\mathcal{I}|}\\}\cup\\{b_{1},\ldots,b_{j-u+|\mathcal{S}^{-}\cap\mathcal{I}^{c}|}\\}\cup\\{c_{1},\ldots,c_{k}\\}\cup\\{d_{1},\ldots,d_{j}\\})$
;

8

9 end for

10

11 end for

12

13 $f_{v,u}(\mathcal{I})\leftarrow\max\\{f_{k,j}\\}$ ;

14

15 end for

16
$f_{v}(\mathcal{I})=\max\\{f_{v,u}(\mathcal{I}),u=0,\ldots,|\mathcal{I}^{c}|\\}$
;

17

18 end for

19$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})\leftarrow\min(v\in\\{0,\ldots,s\\}:f_{v}(\mathcal{I})>\alpha)$
;

20

21$\tilde{\ell}^{-}_{\alpha}(\mathcal{I})\leftarrow|\mathcal{I}|-\max(v\in\\{s,\ldots,|\mathcal{I}|\\}:f_{v}(\mathcal{I})>\alpha)$
;

22

23return $\tilde{\ell}^{+}_{\alpha}(\mathcal{I})$,
$\tilde{\ell}^{-}_{\alpha}(\mathcal{I})$,
$f_{0}(\mathcal{I}),\ldots,f_{|\mathcal{I}|}(\mathcal{I})$ ;

Algorithm 1 Shortcut for computing the lower bounds
$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})$ and
$\tilde{\ell}^{-}_{\alpha}(\mathcal{I})$ derived from adaptive local tests
with partitioning.

######  Proposition D.1.

For any $\mathcal{I}\subseteq[n]$, Algorithm 1 returns the lower bounds
$\tilde{\ell}^{+}_{\alpha}(\mathcal{I})$ and
$\tilde{\ell}^{-}_{\alpha}(\mathcal{I})$ derived from the partitioning
procedure with adaptive local tests (4.2), with at most
$O(|\mathcal{I}|^{2}\cdot\max(1,|\mathcal{I}^{c}|^{2}))$ computation. In
particular, calculation of the adjusted $p$-values for $H^{-}_{i}$ and $K_{i}$
requires $O(n^{2})$ time.

###### Proof.

Suppose we have observed $\mathcal{S}^{-}$ of size $|\mathcal{S}^{-}|=s$, and
for any $\mathcal{I}\subseteq[n]$ and any $v\in\\{0,\ldots,|\mathcal{I}|\\}$
we want to check whether the test in (4.2) rejects all the hypotheses
$J_{\mathcal{K}}$ with $|\mathcal{K}\cap\mathcal{I}|=v$ at level $\alpha$ (or,
equivalently, if $J_{v}(\mathcal{I}):n^{+}(\mathcal{I})=v$ is rejected at
level $\alpha$):

$f_{v}(\mathcal{I})=\max_{\mathcal{K}:|\mathcal{K}\cap\mathcal{I}|=v}f(\\{2p_{i},i\in\mathcal{S}^{-}\cap\mathcal{K}^{c}\\}\cup\\{2q_{i},i\in\mathcal{S}^{+}\cap\mathcal{K}\\})\leq\alpha.$

Algorithm 1 computes

$f_{v,u}(\mathcal{I})=\mathop{\max_{\mathcal{K}:|\mathcal{K}\cap\mathcal{I}|=v,}}_{|\mathcal{K}\cap\mathcal{I}^{c}|=u}f(\\{2p_{i},i\in\mathcal{S}^{-}\cap\mathcal{K}^{c}\\}\cup\\{2q_{i},i\in\mathcal{S}^{+}\cap\mathcal{K}\\})$

so that
$f_{v}(\mathcal{I})=\max\\{f_{v,u}(\mathcal{I}),u=0,\ldots,|\mathcal{I}^{c}|\\}$.
The function $f(\cdot)$ in (9) combines $2p_{i}$ with
$i\in\mathcal{S}^{-}\cap\mathcal{K}^{c}$ and $2q_{i}$ with
$i\in\mathcal{S}^{+}\cap\mathcal{K}$. Writing
$\mathcal{K}=(\mathcal{K}\cap\mathcal{I})\cup(\mathcal{K}\cap\mathcal{I}^{c})$
and
$\mathcal{K}^{c}=(\mathcal{K}^{c}\cap\mathcal{I})\cup(\mathcal{K}^{c}\cap\mathcal{I}^{c})$
gives

$\displaystyle f_{\mathcal{K}}(\mathcal{I})$ $\displaystyle=$ $\displaystyle
f(\\{2p_{i},i\in\mathcal{S}^{-}\cap\mathcal{K}^{c}\\}\cup\\{2q_{i},i\in\mathcal{S}^{+}\cap\mathcal{K}\\})$
$\displaystyle=$ $\displaystyle
f(\\{2p_{i},i\in\mathcal{S}^{-}\cap\mathcal{K}^{c}\cap\mathcal{I}\\}\cup\\{2p_{i},i\in\mathcal{S}^{-}\cap\mathcal{K}^{c}\cap\mathcal{I}^{c}\\}$
$\displaystyle\quad\,\cup\\{2q_{i},i\in\mathcal{S}^{+}\cap\mathcal{K}\cap\mathcal{I}\\}\cup\\{2q_{i},i\in\mathcal{S}^{+}\cap\mathcal{K}\cap\mathcal{I}^{c}\\}).$

Consider Vandermonde’s convolutions:

${|\mathcal{I}|\choose
v}=\sum_{k=0}^{v}{|\mathcal{S}^{-}\cap\mathcal{I}|\choose
v-k}{|\mathcal{S}^{+}\cap\mathcal{I}|\choose k},\quad{|\mathcal{I}^{c}|\choose
u}=\sum_{j=0}^{u}{|\mathcal{S}^{-}\cap\mathcal{I}^{c}|\choose
u-j}{|\mathcal{S}^{+}\cap\mathcal{I}^{c}|\choose j}.$

The first convolution states that for each $k\in\\{0,\ldots,v\\}$, there are
${|\mathcal{S}^{-}\cap\mathcal{I}|\choose
v-k}{|\mathcal{S}^{+}\cap\mathcal{I}|\choose k}$ sets $\mathcal{K}$ such that
$|\mathcal{K}\cap\mathcal{I}|=v$ with
$|\mathcal{S}^{+}\cap\mathcal{K}\cap\mathcal{I}|=k$ and
$|\mathcal{S}^{-}\cap\mathcal{K}\cap\mathcal{I}|=v-k$. Likewise, the second
convolution states that for each $j\in\\{0,\ldots,u\\}$, there are
${|\mathcal{S}^{-}\cap\mathcal{I^{c}}|\choose
u-j}{|\mathcal{S}^{+}\cap\mathcal{I}^{c}|\choose j}$ sets $\mathcal{K}$ such
that $|\mathcal{K}\cap\mathcal{I}^{c}|=u$ with
$|\mathcal{S}^{+}\cap\mathcal{K}\cap\mathcal{I}^{c}|=j$ and
$|\mathcal{S}^{-}\cap\mathcal{K}\cap\mathcal{I}^{c}|=u-j$. Then, the
maximization problem becomes

$\displaystyle
f_{v,u}(\mathcal{I})=\mathop{\max_{k\in\\{k_{1},\ldots,k_{2}\\},}}_{j\in\\{j_{1},\ldots,j_{2}\\}}\mathop{\max_{\mathcal{K}:|\mathcal{S}^{+}\cap\mathcal{K}\cap\mathcal{I}|=k,|\mathcal{S}^{-}\cap\mathcal{K}\cap\mathcal{I}|=v-k,}}_{|\mathcal{S}^{+}\cap\mathcal{K}\cap\mathcal{I}^{c}|=j,|\mathcal{S}^{-}\cap\mathcal{K}\cap\mathcal{I}^{c}|=u-j}f_{\mathcal{K}}(\mathcal{I})$

where $k_{1}=\max(0,v-|\mathcal{S}^{-}\cap\mathcal{I}|)$,
$k_{2}=\min(v,|\mathcal{S}^{+}\cap\mathcal{I}|)$,
$j_{1}=\max(0,u-|\mathcal{S}^{-}\cap\mathcal{I}^{c}|)$ and
$j_{2}=\min(u,|\mathcal{S}^{+}\cap\mathcal{I}^{c}|)$.

For any increasing function $f(\cdot)$, the maximum has solution with largest
$k$ $p$-values $q_{i}$ with $i\in\mathcal{S}^{+}\cap\mathcal{I}$, the largest
$k-v+|\mathcal{S}^{-}\cap\mathcal{I}|$ $p$-values $p_{i}$ with
$i\in\mathcal{S}^{-}\cap\mathcal{I}^{c}$, the largest $j$ $p$-values $q_{i}$
with $i\in\mathcal{S}^{+}\cap\mathcal{I}$ and the largest
$j-u+|\mathcal{S}^{-}\cap\mathcal{I}^{c}|$ $p$-values $p_{i}$ with
$i\in\mathcal{S}^{-}\cap\mathcal{I}$:

$f_{v,u}(\mathcal{I})=\mathop{\max_{k\in\\{k_{1},\ldots,k_{2}\\}}}_{j\in\\{j_{1},\ldots,j_{2}\\}}f(\\{a_{1},\ldots,a_{k-v+|\mathcal{S}\cap\mathcal{I}|}\\}\cup\\{b_{1},\ldots,b_{j-u+|\mathcal{S}\cap\mathcal{I}^{c}|}\\}\cup\\{c_{1},\ldots,c_{k}\\}\cup\\{d_{1},\ldots,d_{j}\\})$

where $a_{1}\geq\ldots\geq a_{|\mathcal{S}^{-}\cap\mathcal{I}|}$,
$b_{1}\geq\ldots\geq b_{|\mathcal{S}^{-}\cap\mathcal{I}^{c}|}$,
$c_{1}\geq\ldots\geq c_{|\mathcal{S}^{+}\cap\mathcal{I}|}$ and
$d_{1}\geq\ldots\geq d_{|\mathcal{S}^{+}\cap\mathcal{I}^{c}|}$ denote the
sorted values of $\\{2p_{i}:i\in\mathcal{S}^{-}\cap\mathcal{I}\\}$,
$\\{2p_{i}:i\in\mathcal{S}^{-}\cap\mathcal{I}^{c}\\}$,
$\\{2q_{i}:i\in\mathcal{S}^{+}\cap\mathcal{I}\\}$ and
$\\{2q_{i}:i\in\mathcal{S}^{+}\cap\mathcal{I}^{c}\\}$, respectively.

Algorithm 1 evaluates $f_{v,u}(\mathcal{I})$ with a nested loop. The outer
loop executes
$\min(v,|\mathcal{S}^{+}\cap\mathcal{I}|)-\max(0,v-|\mathcal{S}^{-}\cap\mathcal{I}|)$
times. Every time the outer loop executes, the inner loop executes
$\min(v,|\mathcal{S}^{+}\cap\mathcal{I}|)-\max(0,u-|\mathcal{S}\cap\mathcal{I}^{c}|)$
times. As a result, the complexity for evaluating $f_{v,u}(\mathcal{I})$ is
$O(|\mathcal{I}||\mathcal{I}^{c}|)$.

The complexity for computing $f_{v}(\mathcal{I})$ for
$v=0,\ldots,|\mathcal{I}|$ is $O(|\mathcal{I}|^{2}|\mathcal{I}^{c}|^{2})$
because it requires to compute $f_{v,u}(\mathcal{I})$ for
$u=0,\ldots,|\mathcal{I}^{c}|$ and $v=0,\ldots,|\mathcal{I}|$. If
$\mathcal{I}=\\{i\\}$, it takes $O(n^{2})$ to compute the adjusted $p$-values
$\bar{p}_{i}=f_{0}(\\{i\\})$ and $\bar{q}_{i}=f_{1}(\\{i\\})$ for $H^{-}_{i}$
and $K_{i}$, respectively.

∎

##  Appendix E Adaptive lower bounds for $n^{+}$ and $n^{-}+n^{0}$

We derive lower bounds $l_{\alpha}^{+}$ and an $l_{\alpha}^{-}$ for $n^{+}$
and $n^{-}+n^{0}$ such that

$\displaystyle\mathbb{P}_{\theta}\Big{(}l_{\alpha}^{+}\leq n^{+},\
l_{\alpha}^{-}\leq n^{-}+n^{0}\mid\mathcal{S}\Big{)}\geq 1-\alpha.$ (23)

These bounds may be tighter than taking $\mathcal{I}=[n]$ with adaptive
partitioning. Interestingly, the gap between these bounds and the bounds for
$\mathcal{I}=[n]$ in the adaptive partitioning procedure is minimal, so
adaptive partitioning is recommended if interest lies in positive and non-
positive discoveries. This finding is based on a wide range of data
generations examined (omitted for brevity). Typically, the analyst is
interested in positive and non-positive discoveries in addition to the lower
bounds for $n^{+}$ and $n^{-}+n^{0}$. Since the cost of further inferences is
minimal, we suggest using adaptive partitioning in order to find the lower
bounds for $n^{+}$ and $n^{-}+n^{0}$ and provide discoveries. Nevertheless,
procedure E.1 below should be used if interest is only in lower bounds for
$n^{+}$ and $n^{-}+n^{0}$, since it is computationally much simpler and the
bounds are at least as tight as with $\mathcal{I}=[n]$ in procedure 1, as
formalized in Proposition E.2.

We briefly review tests of partial conjunction (PC) hypotheses, in order to
set in context the novel Procedure E.1. Let $H^{r/n}:n^{+}\leq r-1$, be the PC
null hypothesis that at most $r-1$ hypotheses among
$H^{-}_{1},\ldots,H^{-}_{n}$ are false; and $K^{r/n}:n^{-}+n^{0}\leq r-1$, the
PC null hypothesis that at most $r-1$ hypotheses among $K_{1},\ldots,K_{n}$
are false. For $r=1$, $H^{r/n}$ is the global null hypothesis that none of the
parameters are positive. For $r=2$, rejection of $H^{r/n}$ leads to
establishing minimal replicability in the positive direction (Benjamini et
al.,, 2009; Jaljuli et al.,, 2022).

Since $H^{r/n}$ is false if and only if every intersection hypothesis of size
$n-r+1$ is false (Benjamini et al.,, 2009), a valid $p$-value $p^{r/n}$ for
$H^{r/n}$ is the largest intersection hypothesis $p$-value, over all
intersections of $n-r+1$ null hypotheses:

$p^{r/n}=\max_{\\{\mathcal{I}:\mathcal{I}\subseteq[n],|\mathcal{I}|=n-r+1\\}}p_{\mathcal{I}},$