# Too Many, Too Improbable: testing joint hypotheses and closed testing
shortcuts

Phillip B. Mogensen and Bo Markussen

###### Abstract

Hypothesis testing is a key part of empirical science and multiple testing as
well as the combination of evidence from several tests are continued areas of
research. In this article we consider the problem of combining the results of
multiple hypothesis tests to i) test global hypotheses and ii) make marginal
inference while controlling the $k$-FWER. We propose a new family of
combination tests for joint hypotheses, called the ‘Too Many, Too Improbable’
(TMTI) statistics, which we show through simulation to have higher power than
other combination tests against many alternatives. Furthermore, we prove that
a large family of combination tests – which includes the one we propose but
also other combination tests – admits a quadratic shortcut when used in a
Closed Testing Procedure, which controls the FWER strongly. We develop an
algorithm that is linear in the number of hypotheses for obtaining confidence
sets for the number of false hypotheses among a collection of hypotheses and
an algorithm that is cubic in the number of hypotheses for controlling the
$k$-FWER for any $k$ greater than one.

##  1 Introduction

The problem of combining $p$-values from tests of a family of hypotheses
$\\{H_{i}\\}_{i\in\mathcal{I}}$ indexed by a finite set $\mathcal{I}$ has long
been a field of study and remains an active area of research today. In 1925,
Fisher proposed a method of combining independent $p$-values by observing that
minus two times the sum of $(\log p_{i})_{i\in\mathcal{I}}$ follows a
$\chi_{2|\mathcal{I}|}^{2}$-distribution (Fisher, 1992). Fisher’s combination
test is asymptotically Bahadur-optimal among the class of all combination
tests (Littell and Folks, 1973). Still, the Fisher Combination Test can
potentially be outperformed by other combination tests for any given finite
sample. In 1973, Brown devised an extension of the Fisher Combination Test or
when the underlying test statistics are jointly Gaussian with a known
covariance matrix and the hypotheses are one-tailed (Brown, 1975). Kost and
McDermott (2002) further relaxed the assumptions on the dependence structure
by deriving an approximation of the distribution of the Fisher Combination
Test when the underlying tests statistics are jointly $T$-distributed with a
common denominator. In recent years the Cauchy Combination Test (Liu and Xie,
2020) and the Harmonic Mean $p$-value (Wilson, 2019) have been proposed and
Vovk and Wang (2020) derive a large family of combination tests by using the
Kolmogorov generalized $f$-mean.

These combination-based methods for testing the global null hypothesis
$H_{0}\coloneqq\bigcap_{i\in\mathcal{I}}H_{i}$ follow the overall recipe of
finding a mapping $f:[0,1]^{|\mathcal{I}|}\to[0,1]$ of
$\bm{p}\coloneqq(p_{i})_{i\in\mathcal{I}}$ such that $f(\bm{p})$ is again a
$p$-value under $H_{0}$. That is, such that for any choice of $\alpha\in[0,1]$
it holds that ${\mathbb{P}}(f(\bm{p})\leq\alpha)\leq\alpha$ when $H_{0}$ is
true. One particular way of obtaining this property is to choose any function,
say $f_{1}$, that maps the hypercube $[0,1]^{|\mathcal{I}|}$ to any subset of
the real line and then subsequently transform the resulting random variable by
its cumulative distribution function (CDF), say $f_{2}$. The composite mapping
$f_{2}\circ f_{1}$ is then a valid combination test. The Fisher Combination
Test is an example of this; first, we map $\bm{p}$ onto the positive real line
by the mapping $\bm{p}\mapsto-2\sum_{i\in\mathcal{I}}\log p_{i}$, which is
then transformed back onto the unit interval using the CDF of a
$\chi_{2|\mathcal{I}|}^{2}$-distribution. Another simple way of constructing
valid combination tests is to use the minimal $p$-value from any procedure
that controls the family-wise error rate (FWER). For example, we may use the
minimal $p$-value of the Bonferroni corrected $p$-values, corresponding to the
mapping $\bm{p}\mapsto\min\left({|\mathcal{I}|\cdot\min(\bm{p}),1}\right)$.

In this paper, we introduce a family of combination tests – the ‘Too Many, Too
Improbable’ (TMTI) tests – that strongly controls the Type I error at level
$\alpha$, for any choice of $\alpha\in(0,1)$. In brief, these statistics are
obtained by ordering the observed $p$-values, transforming them by the CDFs of
beta distributions and returning a local minimum. The $p$-value is then the
local minimum transformed by its CDF. We derive analytical expressions for the
null CDFs of the TMTI test statistics under an assumption of independence and
show through simulation that the TMTI tests can have higher power than other
common combination tests under many alternatives. Additionally, we give an
$\mathcal{O}(m^{2})$ shortcut for carrying out a full Closed Testing Procedure
for all elementary hypotheses for a large family of test statistics, obtained
by considering test statistics of the form
$Z=h(F_{(1)}(p_{(1)}),\dots,F_{(m)}(p_{(m)}))$ under mild assumptions on the
functions $F_{(1)},\dots,F_{(m)}$ and $h$. Using prior work by Goeman and
Solari (2011), we show how these shortcuts can be used to obtain $k$-FWER
control for elementary hypotheses as well as construct confidence sets for the
number of false hypotheses in a rejection set. Finally, we discuss how mixing
different local tests across a Closed Testing Procedure can be used to
increase power.

##  2 The ‘Too Many, Too Improbable’ family of test statistics

###  2.1 Notation and setup

Let $\mathcal{I}\coloneqq\\{1,\dots,m\\}$ be an index set with cardinality $m$
and let $\\{H_{i}\\}_{i\in\mathcal{I}}$ be hypotheses. Let
$(P_{i})_{i\in\mathcal{I}}$ be random variables on probability spaces
$(\Omega_{i},\mathbb{B}_{i},\mathbb{P}_{i})_{i\in\mathcal{I}}$ with
$\Omega_{i}\subseteq[0,1]$. In most situations we will have $\Omega_{i}=[0,1]$
and have $\mathbb{B}_{i}$ be the Borel sigma-algebra, although this need not
be the case. We denote by $p_{i}$ an outcome of $P_{i}$ and call $p_{i}$ the
$p$-value for the test of $H_{i}$. For a given subset of indices,
$\mathcal{J}\subseteq\mathcal{I}$, we consider the task of testing the joint
hypothesis $H_{\mathcal{J}}\coloneqq\bigcap_{j\in\mathcal{J}}H_{j}$ by using
the marginal $p$-values,
$\bm{P}^{\mathcal{J}}\coloneqq(P_{j})_{j\in\mathcal{J}}$. The set
$\mathcal{J}$ can be chosen freely according to what kind of hypothesis one
wishes to test. If we choose $\mathcal{J}$ with $|\mathcal{J}|=1$, no
adjustment needs to be made, as we are simply testing a marginal hypothesis,
for which we already have a $p$-value. If we choose $\mathcal{J}=\mathcal{I}$,
we are considering the global null hypothesis of $\mathcal{I}$. Anything in
between those two extremes corresponds to testing a particular joint
hypothesis. E.g., if $(p_{i})_{i\in\mathcal{I}}$ are the $p$-values output
from a genome-wide association study, then $\mathcal{J}$ could correspond to a
particular region, which is of special interest, e.g., a gene or chromosome.

In order to test $H_{\mathcal{J}}$, we construct a test statistic, denoted by
$Z$, with corresponding $p$-value $P$ that satisfies

$H_{\mathcal{J}}\text{
true}\implies\forall\alpha\in[0,1]:~{}{\mathbb{P}}(P\leq\alpha)\leq\alpha.$
(1)

The above statement is called Type I error control and means that whenever the
joint hypothesis $H_{\mathcal{J}}$ is true, the probability that we reject
$H_{\mathcal{J}}$ at level $\alpha$ is at most $\alpha$.

###  2.2 Definition of the TMTI statistics

Let $P_{(1)}^{\mathcal{J}}\leq\dots,P_{(|\mathcal{J}|)}^{\mathcal{J}}$ denote
an ordering of $\bm{P}^{\mathcal{J}}$. This ordering is possibly not unique.
Let $\beta(a,b)(x)$ denote the CDF of the $\beta(a,b)$-distribution.When the
shape and scale parameters are integers, we can write

$\forall i,m\in\mathbb{N}:\hskip
28.45274pt\beta(i,m+1-i)(x)=\sum_{k=i}^{m}\binom{m}{k}x^{k}(1-x)^{m-k}.$ (2)

We construct the collection
$\bm{Y}^{\mathcal{J}}\coloneqq(Y^{\mathcal{J}}_{k})_{k=0}^{|\mathcal{J}|+1}$
of random variables by $Y_{0}^{\mathcal{J}}\coloneqq 2$,
$Y_{|\mathcal{J}|+1}^{\mathcal{J}}\coloneqq 2$, and

$\forall k\in\\{1,\dots,|\mathcal{J}|\\}:\hskip
28.45274ptY_{k}^{\mathcal{J}}\coloneqq\beta(k,|\mathcal{J}|+1-k)(P_{(k)}^{\mathcal{J}}).$

If all variables in $\bm{P}^{\mathcal{J}}$ are independent and exactly
uniform, then each $Y^{\mathcal{J}}_{k}$ is uniformly distributed on $[0,1]$
for $k=1,\dotsc,|\mathcal{J}|$, as it is well known that the order statistics
of i.i.d. $U(0,1)$ variables are $\beta$-distributed.

Let $c\leq|\mathcal{J}|$ be an integer. We then consider the first
$Y^{\mathcal{J}}_{k}$ among the first $c$ variables that is strictly smaller
than the following $n\in\mathbb{N}\cup\\{\infty\\}$, i.e.,

$L_{n,c}\coloneqq\min\\{l\in\\{1,\dots,c\\}:Y_{l}^{\mathcal{J}}<Y_{k}^{\mathcal{J}}~{}~{}\text{for
all}~{}~{}k=l+1,\dots,\min(l+n,|\mathcal{J}|+1)\\}.$

If $Y_{1}^{\mathcal{J}}\geq\dots\geq Y_{c}^{\mathcal{J}}$ we set $L_{n,c}=c$.
We think of $L_{n,c}$ as the index of a kind of local minimum of
$Y_{1}^{\mathcal{J}},\dots,Y_{c}^{\mathcal{J}}$, in the sense that
$Y_{L_{n,c}}^{\mathcal{J}}$ is always a local minimum, but it further needs to
satisfy that it is smaller than the following $n$ terms. In particular,
$Y_{L_{1,c}}^{\mathcal{J}}$ is the first local minimum of
$Y_{1}^{\mathcal{J}},\dots,Y_{c}^{\mathcal{J}}$ and
$Y_{L_{\infty,c}}^{\mathcal{J}}$ is the global minimum of
$Y_{1}^{\mathcal{J}},\dots,Y_{c}^{\mathcal{J}}$. The construction of
$Y^{\mathcal{J}}_{0}$ and $Y^{\mathcal{J}}_{|\mathcal{J}|+1}$ is a technical
one, meant only to ensure the existence of $L^{\mathcal{J}}_{n,c}$. To ease
the notational burden, we omit the subscripted $n$ and $c$ and the
superscripted $\mathcal{J}$ when the particular choices of $n$, $c$ and
$\mathcal{J}$ are not of importance or unambiguous from context.

######  Definition 1.

Let $n\in\mathbb{N}\cup\\{\infty\\}$ and let $c\leq|\mathcal{J}|$ be an
integer. The ‘Too Many, Too Improbable’ test statistic is then defined as

$Z_{n,c}^{\mathcal{J}}\coloneqq Y_{L_{n,c}}^{\mathcal{J}}.$

Small values of $Z_{n,c}^{\mathcal{J}}$ are critical and the $p$-value for the
test of $H_{\mathcal{J}}$ is obtained by evaluating the test statistic in its
CDF under $H_{\mathcal{J}}$. We denote by $\gamma_{n,c}^{\mathcal{J}}(x)$ the
CDF of $Z_{n,c}^{\mathcal{J}}$ under $H_{\mathcal{J}}$.

Generally, we will only consider cases in which $n=1$ or $n=\infty$, as these
are the most natural choices of $n$. However, the setup allows for other
choices of $n$. Choosing $1<n<\infty$ can potentially increase the power of
the procedure in cases where signals are fairly sparse, but sufficiently weak
that the first local minimum falls ‘too early’ by chance. However, we do not
investigate this further, but simply remark that it is possible to choose $n$
different from what we consider in the remainder of this paper.

######  Remark 1.

Testing the joint hypothesis $H_{\mathcal{J}}$ using any TMTI test satisfies
the statement in (1) by the probability integral transform. That is, the TMTI
test controls the Type I error.

######  Remark 2.

Whenever $|\mathcal{J}|=1$, say $\mathcal{J}=\\{j\\}$, the TMTI transform is
simply the identity transform, i.e., $\gamma(Z)=P_{j}$.

######  Remark 3.

If the variables in $\bm{P}^{\mathcal{I}}$ are exchangeable, i.e., if any two
subsets of equal size have the same joint distribution, it follows, that for
any two sets $\mathcal{J}_{1},\mathcal{J}_{2}\subset\mathcal{I}$ with
$|\mathcal{J}_{1}|=|\mathcal{J}_{2}|$, we have
$\gamma^{\mathcal{J}_{1}}=\gamma^{\mathcal{J}_{2}}$. Thus, for exchangeable
$p$-values, the CDF of the TMTI statistic depends only on the choice of
$\mathcal{J}$ through its cardinality.

###  2.3 Truncation procedures and the TMTI

The Truncated Product Method of Zaykin et al. (2002) and the Rank Truncated
Product Method of Dudbridge and Koeleman (2003) are two notable variants of
the Fisher Combination Test, that also test the global null hypothesis but
against different alternatives.

The Truncated Product Method is a combination test that uses only the
$p$-values that are smaller than some predefined threshold $\tau\in(0,1)$. The
alternative hypothesis is therefore, that there is at least one false
hypothesis among those hypotheses that gave rise to $p$-values below $\tau$.
The Rank Truncated Product Method is also a combination test, but this uses
only the smallest $K$ $p$-values, for some predefined $K<|\mathcal{J}|$. Thus,
the alternative hypothesis is, that there is at least one false hypothesis
among those, that gave rise to the $K$ smallest $p$-values.

The TMTI family of test statistics includes similar procedures. For any
$c<|\mathcal{J}|$, the alternative hypothesis is that there is at least one
false hypothesis among those that gave rise to the $c$ smallest $p$-values.
Thus, setting $c=K$ for some integer $K<|\mathcal{J}|$, the TMTI procedure
uses only the first $K$ $p$-values in the construction of the test statistic
and therefore tests the joint hypothesis $H_{\mathcal{J}}$ against the same
alternative as the Rank Truncated Product Method. We call this procedure the
rank truncated TMTI.

By setting
$c=\max(\\{j\in\\{1,\dots,|\mathcal{J}|\\}:p_{(j)}\leq\tau\\}\cup\\{1\\})\eqqcolon\bar{\tau}$,
for some value $\tau\in(0,1)$, the TMTI procedure uses only the $p$-values
that are marginally significant at level $\tau$, and thus tests against the
same alternative as the Truncated Product Method. We call this procedure the
truncated TMTI. In the event that no $p$-values are smaller than $\tau$, $c$
becomes $1$ and uses instead the smallest of the available $p$-values.

We write TMTIn to denote the TMTI statistic $Z_{n,c}^{\mathcal{J}}$ with
$c=|\mathcal{J}|$, tTMTIn to denote the truncated TMTI statistic and rtTMTIn
to denote the rank truncated TMTI statistic.

There are two potential advantages to using a truncated procedure (i.e.,
$c<|\mathcal{J}|$) over a non-truncated procedure. First, for large $m$ (say,
$m\geq 10^{6}$), it is non-trivial to compute the TMTI∞-statistic, because its
computation involves sorting $m$ different $p$-values and computing $m$
different $\beta$-transformations. Using a truncation procedure instead
reduces the computational cost, because fewer $p$-values need to be
considered. Thus, only a partial sorting is required and fewer
$\beta$-transformations need to be computed. Second, as we outline below,
using a truncation procedure can potentially have higher power than its non-
truncated version.

######  Lemma 1.

Let $n\in\mathbb{N}\cup\\{\infty\\}$ and $x\in(0,1)$. Let $\mathcal{I}$ be an
index set with cardinality $m$. It follows that

$\forall c<m:\hskip 28.45274pt\gamma_{n,c}(x)<\gamma_{n,m}(x)$

When using $n=1$, i.e., considering the first local minimum, and when using
moderate values of $\tau$ and $K$, it is likely that $\gamma_{1,\tau}$,
$\gamma_{1,K}$ and $\gamma_{1}$ are going to be nearly identical, as the first
local minimum is likely to lie early in the sequence $\bm{Y}$. This implies
that the $p$-values of the tTMTI1, the rtTMTI1 and TMTI1 tests are nearly
identical. Thus, the TMTI1 by itself can be thought of as a truncation method.
However, if using the global minimum, we expect that there can be a large
difference between the methods, as the global minimum is likely to lie further
along the sequence $\bm{Y}$. Thus, applying either the tTMTI∞ with a low
$\tau$ or rtTMTI∞ with a low $K$ is going to be roughly equivalent to applying
the TMTI1. These properties are demonstrated in Figure 1 for the case of
independent and exactly uniform $p$-values.

From Figure 1 we conclude, that if the global null hypothesis is indeed false,
and if the global minimum of the sequence $\bm{Y}$ happens to fall within the
first $K$ or $\bar{\tau}$ indices of $\bm{Y}$, there is a potential for a
large power gain by applying either the tTMTI∞ or rtTMTI∞ over the standard
TMTI∞. This is because the procedures will all be considering the same test
statistic, $Z$, but they will evaluate it under different $\gamma$ functions,
thereby yielding different $p$-values. By Lemma 1, the $\gamma$ functions from
the tTMTI∞ and rtTMTI∞ procedures are dominated by the $\gamma$ function from
the TMTI∞, implying that $p$-value resulting from applying the truncation
procedures will be lower than those of the standard procedure, giving rise to
higher power.

Figure 1: Comparison of $\gamma_{n}^{\mathcal{I}}$,
$\gamma_{n,\tau}^{\mathcal{I}}$ and $\gamma_{n,K}^{\mathcal{I}}$ for
$m=10^{5}$, $\tau=0.01$ and $K=5$ in the case of independent and exactly
uniform $p$-values. The solid red lines are TMTI, the dotted green and blue
lines are tTMTI and rtTMTI, respectively.

##  3 Computation of $\gamma$

###  3.1 An analytical expression of the CDF of TMTI∞ in the i.i.d. case

In the case where the $p$-values are independent under the null hypothesis, we
can derive an analytical expression of $\gamma_{\infty,c}$.

######  Theorem 1.

Let $P_{1},\dots,P_{m}$ be i.i.d. uniformly distributed on $[0,1]$. For every
$i\in\\{1,\dots,m\\}$, let $x_{i}$ be the $x$-quantile of the $\beta(i,m+1-i)$
distribution and define the polynomial

$Q_{i}(x;\bm{a})\coloneqq\sum_{j=1}^{i}\frac{a_{j}}{(i+1-j)!}x^{i+1-j}.$

Define $\bar{Q}_{1}\coloneqq x_{1}$ and define recursively

$\displaystyle\forall i\in\\{2,\dots,m\\}:\hskip 28.45274pt\bar{Q}_{i,c}$
$\displaystyle\coloneqq
Q_{i}(x_{\min(i,c)};(1,-\bar{Q}_{1,c},\dots,-\bar{Q}_{i-1,c})).$

If $c\leq m$ is a fixed integer, then

$\gamma_{\infty,c}(x)=x_{c}^{m}+\sum_{i=1}^{m-1}\frac{m!}{(m-i)!}\bar{Q}_{i,c}(1-x_{c}^{m-i}).$
(3)

Furthermore, let $\tau\in(0,1)$ and define $\tilde{Q}_{1}\coloneqq\tau$ and

$\forall i\in\\{2,\dots,m\\}:\hskip 28.45274pt\tilde{Q}_{i}\coloneqq
Q_{i}(\tau;(1,-\bar{Q}_{1,m},\dots,\bar{Q}_{i-1,m})).$

If $c$ is a random variable given by
$c=\max(\\{i\in\\{1,\dots,m\\}:P_{(i)}<\tau\\}\cup\\{1\\})$, then

$\begin{split}\gamma_{\infty,c}(x)&=(1-\tau)^{m}\frac{x-\beta(1,m)(\tau)}{1-\beta(1,m)(\tau)}I(x_{1}>\tau)\\\
&+\sum_{i=1}^{m}\left[\binom{m}{i}\tau^{i}(1-\tau)^{m-i}\left\\{1-\frac{i!}{\tau^{i}}\left(\tilde{Q}_{i}-\bar{Q}_{i,m}\right)I(x_{i}\leq\tau)\right\\}\right]\end{split}$
(4)

The above can be readily implemented by recursively computing the
$\bar{Q}_{i,c}$ and $\tilde{Q}_{i}$ terms, e.g.,in a for-loop.

In the special case of $c=1$, we have $\gamma_{n,1}(x)=x$ by construction,
regardless of $m$ and $n$. In this setting, the TMTI procedure is then a
minimum-$p$ method. This has the advantage, that the procedure can be applied
directly in high-dimensional settings, if the assumption of independence
holds. In particular, it is easy to show, that when $c=1$ the critical value
of the TMTI test is $1-(1-\alpha)^{1/m}$, and it is thus equivalent to using
the Šidák correction (Šidák, 1967) for testing the global null hypothesis.

###  3.2 A bootstrap scheme for the CDF of TMTIn in the i.i.d. case

Although it is easy to implement Equations (3) and (4), numerical difficulties
may arise when $m$ is large, say $m>100$, due to the presence of the
factorials $1!,\dots,m!$ in the computations. Essentially, the $\bar{Q}_{i,c}$
terms are all very small, because they include the reciprocals of factorials,
but they are scaled up by another factorial. Although this is well-defined,
numerical instabilities will occur in implementations in standard double-
precision arithmetic. For larger $m$, one can perform the calculations in
arbitrary precision, but the added computational cost of doing so can be
enormous. Instead, a simple bootstrapping scheme can be employed by; i)
drawing $m$ values independently from a $U(0,1)$ distribution; ii)
transforming the values from step i as described in Equation 2.2 and saving
the desired TMTI statistic as $Z_{b}$, where $b$ indexes the iteration; iii)
repeating steps i and ii sufficiently many times, say $B$, and; iv) using
$\hat{\gamma}(x)\coloneqq\frac{1}{B}\sum_{b=1}^{B}I(Z_{b}\leq x)$ as an
approximation of $\gamma$. This bootstrap scheme can be applied regardless of
the choice of $n$ and $c$.

###  3.3 The non-independent case

The derivation of the CDF, $\gamma$, in Section 3.1 relied on the assumption
that all $p$-values are independent. Chen et al. (2020) argue that combining
methods that are Valid under Arbitrary Dependence (VAD) structures have lower
power than combining methods that are Valid under Independent (VI) $p$-values,
if the underlying, $p$-values are in fact independent. However, if the
underlying $p$-values are not independent, VI methods may fail to hold level,
whereas their VAD counterparts will hold level for any dependence structure.
Thus, the choice of combining method should depend on the scientific question
of interest. For instance, in genome-wide association studies (GWAS), it is
unreasonable to assume independence, as base-pairs are likely to be locally
dependent (Dudbridge and Koeleman, 2003).111It is often possible to filter the
$p$-values in a manner such that the remaining $p$-values are likely to be
independent, e.g., using a distance-based filtering.

In some cases, however, it is possible to apply the methods described in
Sections 3.1 and 3.2, even if the $p$-values are not independent. For
instance, if the underlying tests, $Z$, are jointly Gaussian with a known
covariance matrix, $\Sigma$, these can be decorrelated by performing an
eigendecomposition, $\Sigma=Q\Lambda Q^{T}$, and then constructing
$\tilde{Z}\coloneqq(Q\Lambda^{-1/2}Q^{T})^{T}Z$. Then, the components of
$\tilde{Z}$ are jointly independent (see, e.g., Kessy et al. (2018)) and the
methods described in Sections 3.1 and 3.2 can be directly applied to the
$p$-values obtained from the test statistics $\tilde{Z}$. This remains true
for any rotation, $\bar{Z}\coloneqq R(Q\Lambda^{-1/2}Q^{T})^{T}Z$, where $R$
is an orthogonal matrix.

If the $p$-values are not independent, and if the decorrelation procedure
described above is not appropriate, one can still try to apply the TMTI
directly. However, level of the test (i.e., Equation 1) is no longer
guaranteed, and thus there is a chance that the Type I error is increased. How
much the Type I error increases depends entirely on the dependence structure
of the $p$-values. In B.1, we investigate the level of the TMTI tests under
three different types of dependencies: autoregressive $p$-values (i.e.,
$\operatorname{cor}(P_{i},P_{j})=\rho^{|i-j|}$), equicorrelated $p$-values
(i.e., $\operatorname{cor}(P_{i},P_{j})=\rho$, for all $i,j$), and block-
diagonally correlated $p$-values (i.e., $\operatorname{cor}(\bm{P})$ has a
block-diagonal structure, where all off-diagonal entries are $\rho$ if in the
same block and $0$ else). We note, however, that rtTMTIn seems to either have
the correct level or be conservative, no matter the dependence structure.
Overall, the TMTI tests hold level only under weak autoregressive and block-
diagonal dependency structures, and fails to hold level for stronger
dependency structures and equicorrelated $p$-values (see B.1 and Figures 5 and
6 for a full account of the results). Thus, the TMTI tests can potentially
still be applied in settings with weak dependence, but it is not appropriate
in settings with strong dependencies.

Finally, one can apply any VI combination method under arbitrary dependence,
if one is able to sample from the joint distribution of $\bm{P}$ under the
global null. This can, for instance, be done if one assumes an underlying
parametric model or by employing a resampling bootstrap procedure. In B.2, we
give an example of how this can be done in a case where the marginal
hypotheses of interest are $T$-tests for the parameters in a linear model
being zero.

##  4 Power of the TMTI – a simulation study

In this section, we show by means of simulation, that many of the TMTI tests
have high power against a wide range of alternative hypotheses. In particular,
we find that the TMTI∞ and tTMTI∞ tests have high power both in cases where
signals are sparse but strong and in cases where signals are dense but weak.

We consider $m=10^{5}$ independent tests, of which
$N_{\operatorname{false}}\in\\{10^{0},\cdots,10^{4}\\}$ are false. In order to
investigate situations in which the $p$-values from true hypotheses are
conservative, we generate these as $p_{\operatorname{true}}\coloneqq
U^{\delta}$, where $U\sim U(0,1)$ and $\delta\in[0,1]$. When $\delta=1$, this
corresponds to the true $p$-values being exactly uniform, and when
$\delta\in(0,1)$, it corresponds to the true $p$-values being strictly
conservative. The degree of conservatism increases as $\delta$ decreases and
the extreme case in which $\delta=0$ corresponds to the degenerate case where
all true $p$-values are equal to one, meaning that no hypothesis can ever be
rejected, no matter the significance level. Situations in which the $p$-values
are strictly conservative occur in many places. For instance, in a GWAS with
dichotomous traits, the $p$-values will be conservative (Wu et al., 2011).
Conservative $p$-values also occur in Invariant Causal Prediction, where a
$p$-value for invariance is obtained as the minimum of Bonferroni-corrected
$p$-values from multiple environments (Peters et al., 2016). We generate
$p$-values for the false hypotheses by independently sampling $Z$-scores,
$Z_{1},\dots,Z_{n_{\operatorname{false}}}\stackrel{{\scriptstyle
i.i.d.}}{{\sim}}N(\mu_{\operatorname{false}},1)$, for different values of
$\mu_{\operatorname{false}}$ and then letting
$p_{i,\operatorname{false}}\coloneqq 2\times(1-\Phi(|Z_{i}|))$, where $\Phi$
is the CDF of a $N(0,1)$-distribution. The values of
$\mu_{\operatorname{false}}$ are chosen equidistantly between the two values,
which satisfy that a Bonferroni test has either 5% or 99% power to reject the
global null hypothesis in a setting with no conservatism.

For comparison, we include the Fisher Combination Test (which is known to lose
power in the presence of conservative $p$-values (Zaykin et al., 2002)) and
its truncated versions, and the Cauchy Combination Test and Harmonic Mean
$p$-value (which are known to have high power in settings with sparse, strong
signals (Liu and Xie, 2020; Wilson, 2019)). In all settings, we employ a
significance level of $\alpha=0.05$, and for the truncation procedures, we use
$\tau=0.05$ and $K=10$. We include TMTI∞ and both truncation variants, as well
as TMTI1. For TMTI1, we do not include any truncation variants, as we expect
these to be roughly equal to the non-truncated version (per Figure 1).

Figure 2: Power curves for different TMTI tests and competing methods.
Generally, the TMTI∞ and tTMTI∞ work well in all settings. The values of
$\mu_{\operatorname{false}}$ are chosen equidistantly between the two values,
which satisfy that a Bonferroni test has either 5% or 99% power to reject the
global null hypothesis in a setting with no conservatism.

The results of the simulations are displayed in Figure 2. Overall, there are
three things to notice.

First, the TMTI∞ and tTMTI∞ generally work well no matter how many false
hypotheses there are. When there is only a single false hypothesis, these
methods have less power than, e.g., a Bonferroni correction, which has the
highest power in this scenario, but both methods have considerably higher
power than both the Fisher Combination Test and Truncated Product Method. When
there are more false hypotheses, TMTI∞ and tTMTI∞ perform on par with the
Fisher Combination Test and Truncated Product Method, having considerably
higher power than the remaining methods. No other methods exhibit this
property; the Cauchy Combination Test, Harmonic Mean $p$-value, Compound
Bonferroni Arithmetic Mean and Bonferroni test all work well when signals are
sparse, but have low power when there are many weak signals. In contrast, the
Fisher Combination Test and Truncated Product Method work well when signals
are dense and weak, but have almost no power when signals are sparse and
strong. Thus, the TMTI∞ and tTMTI∞ appear to have high power against all
alternative hypotheses, mimicking the properties of, e.g., the ACAT-O, a test
which is shown to have high power against both sparse and dense alternatives
(Liu et al., 2019). The ACAT-O, however, is designed specifically for
sequencing studies and works by leveraging information about the minor-allele
counts of a sequencing study, and thus cannot be directly applied in other
settings. In contrast, TMTI∞ and tTMTI∞ work as regular combination tests and
can be applied to any type of data, given that the assumption of independence
is satisfied.

Second, TMTI1 and rtTMTI∞ work well when signals are sparse, although not
better than a Bonferroni correction. When signals are dense, these methods
have less power than the TMTI∞, tTMTI∞, Fisher Combination Test and Truncated
Product Method, but higher power than the Cauchy Combination Test and the
Harmonic Mean $p$-value. The TMTI1 and Rank Truncated Product Method have
almost identical power in all settings.

Third, all methods are, in some degree, affected by conservatism, in the sense
that all methods generally have less power when the $p$-values from true
hypotheses are conservative. When there are few false hypotheses
($N_{\operatorname{false}}\leq 10^{2}$), the Fisher Combination Test and
Truncated Product Method have almost no power even under mild conservatism.
When there are sufficiently many false hypotheses
($N_{\operatorname{false}}=10^{4}$), the effect of conservatism is less
pronounced. Overall, it appears that the TMTI tests are less affected by
conservatism than the Fisher tests.

It is in line with the intuition behind the TMTI tests that TMTI∞ and tTMTI∞
do not perform as well as its competitors in situations where signals are
sparse but strong, because these achieve their power from ‘too many’ of the
marginal hypotheses being false. In contrast, minimum-$p$ based tests, such as
the Bonferroni procedure, need only a single, very strong signal to detect
that the global null is false. Similarly, Liu and Xie (2020) argue that the
Cauchy Combination Test only makes use of the first few small $p$-values to
represent the overall significance. The same holds true for the Harmonic Mean
$p$-value (Wilson, 2019). The reason that TMTI1 and rtTMTI∞ still perform
similarly to these three methods is that the first local minimum of the sorted
and transformed $p$-values is likely to lie early on when there are only a
few, very small $p$-values, and likely to coincide with the global minimum of
the $K$ smallest $p$-values, if $K$ is sufficiently small. Thus, TMTI1 and
rtTMTI∞ share the property, that they are influenced the most by a few of the
smallest $p$-values. The global minimum, however, need not lie early on, when
there are only a few false hypotheses, meaning that the few signals that we do
observe can potentially be missed when assessing the overall significance
using TMTI∞ or tTMTI∞.

In B.3 we repeated the experiment of this section in a setting with non-
constant $\mu$ values (i.e., when $N_{\operatorname{false}}>1$, the values of
$\mu_{\operatorname{false}}$ were allowed to differ for each false marginal
hypothesis), finding results similar to those shown in Figure 2.

##  5 Multiple testing and strong FWER control

In this section, we consider a common task in statistics. Given $p$-values for
a collection of hypotheses, which hypotheses can safely be rejected? As each
$p$-value gives marginal Type I error control by definition, a naive approach
would be to set a level, $\alpha$, and reject any hypothesis if its
corresponding $p$-value falls below $\alpha$. However, as the number of tests
conducted increases, more Type I errors will be made, which makes it necessary
to employ methods that control for multiple testing. Popular targets one may
wish to control for include the False Discovery Rate (Benjamini and Hochberg,
1995) and the Family-Wise Error Rate (FWER). To control the FWER the
Bonferroni correction is often used, as it is easy to implement and guarantees
strong FWER control. However, the Bonferroni correction has received criticism
for, among other things, heavily increasing the risk of making Type II errors,
i.e., failing to reject false hypotheses (Perneger, 1998). A general approach
for turning global testing procedures into a procedure that controls the FWER
for elementary hypotheses is the Closed Testing Procedure of Marcus et al.
(1976). We briefly review the theory on Closed Testing Procedures.

######  Definition 2.

Let $\mathcal{I}$ be a set of indices and let $\\{H_{i}\\}_{i\in\mathcal{I}}$
denote a collection of hypotheses. For any subset $\mathcal{J}\subseteq I$,
let $H_{\mathcal{J}}\coloneqq\bigcap_{j\in\mathcal{J}}H_{j}$ be the joint
hypothesis. Let $\phi^{\mathcal{J}}$ be a random variable on $[0,1]$
satisfying

$H_{\mathcal{J}}\text{ true }\implies\forall\alpha\in[0,1]:\hskip
28.45274pt{\mathbb{P}}(\phi^{\mathcal{J}}\leq\alpha)\leq\alpha.$

That is, $\phi^{\mathcal{J}}$ is a valid $p$-value for the test of the joint
hypothesis $H_{\mathcal{J}}$. Furthermore, for any subset
$\mathcal{J}\subseteq\mathcal{I}$ we define the closure of $\mathcal{J}$ in
$\mathcal{I}$ to be

$\mathcal{J}_{\mathcal{I}}^{*}\coloneqq\bigcup_{{\mathcal{K}}:{\mathcal{K}}\subseteq\mathcal{I},~{}\mathcal{J}\subseteq{\mathcal{K}}}\\{{\mathcal{K}}\\}.$

That is, $\mathcal{J}_{\mathcal{I}}^{*}$ is the set of all supersets of
$\mathcal{J}$ that are contained in $\mathcal{I}$. A Closed Testing Procedure
for the test of the joint hypothesis $H_{\mathcal{J}}$ is one that rejects
$H_{\mathcal{J}}$ at level $\alpha$ if and only if every superset of
$\mathcal{J}$ in $\mathcal{I}$ is also rejected at level $\alpha$, i.e.,

$(H_{\mathcal{J}}\text{ is rejected at level
}\alpha)=\bigcap_{{\mathcal{K}}\in\mathcal{J}_{\mathcal{I}}^{*}}\left(\phi^{\mathcal{K}}\leq\alpha\right).$

That is, the event that we reject $H_{\mathcal{J}}$ occurs if and only if we
reject all supersets of $\mathcal{J}$ in $\mathcal{I}$ marginally.

From the above, we see that a Closed Testing Procedure is more strict than
marginal testing. That is, it becomes more difficult to reject any hypothesis,
as we now need to reject all hypotheses that include the hypothesis of
interest – not only the hypothesis itself. The upside is that we obtain strong
control of the FWER.

######  Theorem 2 (Marcus et al. (1976)).

Let $\mathcal{J}_{1},\dots,\mathcal{J}_{m}$ be distinct subsets of a larger
set of indices $\mathcal{I}$. Testing $H_{\mathcal{J}_{1}}$ through
$H_{\mathcal{J}_{m}}$ each at level $\alpha$ by means of a closed testing
procedure controls the FWER at level $\alpha$ in the strong sense.

Given any general method to construct tests of joint hypotheses from marginal
tests, we can employ these in a Closed Testing Procedure to obtain strict
control of the FWER. It is generally accepted that Closed Testing Procedures
are more powerful than other methods that control the FWER (Grechanovsky and
Hochberg, 1999), although this power increase comes at the cost of a heavy
computational burden. Given $m$ marginal hypotheses which we want to test, we
need to perform $\sum_{i=1}^{m}\binom{m}{i}=2^{m}-1$ tests. This is because we
need to test all possible intersection hypotheses, which corresponds to the
powerset of all hypotheses, minus the empty set. Thus, in many cases, it is
not feasible to perform a Closed Testing Procedure when $m$ is even slightly
large. Indeed, even with just $m=300$ marginal tests, the number of tests to
be performed in a full Closed Testing Procedure is $2^{300}-1\approx 2\cdot
10^{90}$ – roughly $10$ billion times the number of atoms in the observable
universe. Thus, with many procedures, one seeks to find a shortcut so that
only a subset of the powerset of hypotheses needs to be tested. This is often
possible (Grechanovsky and Hochberg, 1999) and considerably reduces the
computational complexity of carrying out a Closed Testing Procedure.

Zaykin et al. (2002) introduce a shortcut for the Truncated Product Method,
reducing the computational complexity of the Closed Testing Procedure from
$\mathcal{O}(2^{m})$ to $\mathcal{O}(m^{2})$. In a recent result, Tian et al.
(2021) give the same shortcut for a family of combination tests that are sums
of marginal tests. Dobriban (2020) gives a shortcut for test statistics that
are monotone and symmetric. Here, we provide a shortcut for class of
combination tests, which are monotone but not necessarily symmetric, and not
necessarily sums of marginal tests. Furthermore, we show that TMTI∞, tTMTI∞
and rtTMTI∞ all admit this shortcut.

######  Lemma 2.

Let $\bm{p}_{\mathcal{I}}$ be a set of observed $p$-values with
$\mathcal{I}\coloneqq\\{1,\dots,m\\}$ and $m\geq 2$. Let $\mathcal{J}^{k}$ be
the set of all subsets of $\mathcal{I}$ with $|\mathcal{J}|=k$. Let
$\mathcal{X}\subseteq\mathbb{R}$ be a set and let
$F_{(1)}:[0,1]\to\mathcal{X},\dots,F_{(k)}:[0,1]\to\mathcal{X}$ be a sequence
of functions that satisfy

$\displaystyle\forall j\in\mathcal{I}~{}~{}\forall
x\in\mathcal{X}~{}~{}\forall\epsilon\geq 0:$ $\displaystyle\hskip
28.45274ptF_{(j)}(x)\leq F_{(j)}(x+\epsilon)$ (C1) and $\displaystyle\forall
j\in\\{1,\dots,m-1\\}~{}~{}\forall x\in\mathcal{X}:$ $\displaystyle\hskip
28.45274ptF_{(j)}(x)\geq F_{(j+1)}(x).$ (C2)

Define for all $\mathcal{J}\in\mathcal{J}^{k}$ the random variable
$\bm{Y}^{\mathcal{J}}\coloneqq(F_{(1)}(p_{(1)}),\dots,F_{(k)}(p_{(k)}))$ and
let $h:\mathcal{X}^{k}\to[0,1]$ be a function satisfying

$\forall\bm{x}\in\mathcal{X}^{k}~{}~{}\forall\bm{\epsilon}\in\mathbb{R}_{+}^{k}:\hskip
28.45274pth(\bm{x})\leq h(\bm{x}+\bm{\epsilon}),$ (C3)

Let $\eta:\mathcal{I}\to\mathcal{I}$ be a bijection ordering
$\bm{p}_{\mathcal{I}}$, i.e., $p_{\eta(1)}\leq\dots\leq p_{\eta(m)}$. It then
follows that for any two sets,
$\mathcal{J}_{1},\mathcal{J}_{2}\in\mathcal{J}^{k}$

$\eta(\mathcal{J}_{1})\leq\eta(\mathcal{J}_{2})\implies
h(\bm{Y}^{\mathcal{J}_{1}})\leq h(\bm{Y}^{\mathcal{J}_{2}}).$

In the above, the operation $\leq$ applied to the sets $\eta(\mathcal{J}_{1})$
and $\eta(\mathcal{J}_{2})$ is taken to mean element-wise less than or equal
to.

Lemma 2 states that whenever we consider $k$ $p$-values, we will obtain a
smaller test statistic if we substitute one or more of them with smaller
$p$-values. In the context of closed testing, this implies that when
considering the closure of an atom, say $\\{j\\}$, then among all subsets of
size $k$ in $\\{j\\}_{\mathcal{I}}^{*}$ we do not need to test all
$m!/((m-k)!k!)$ intersection hypotheses. This is because we know that the
largest (smallest) test statistic is obtained by considering the $p$-value
$p_{j}$ combined with the $k-1$ largest (smallest) remaining $p$-values.
Assuming that the underlying distribution of the $p$-values is exchangeable,
this implies that the $p$-values for the combination tests obey the same
inequalities as the test statistics, and thus we need only consider the
intersection hypothesis which we know will yield the largest $p$-value.

######  Remark 4.

The same result as in Lemma 2 can be obtained by reversing the inequalities in
Equations (C1), (C2) and (C3). In contrast, we can obtain a version which
gives

$\eta(\mathcal{J}_{1})\leq\eta(\mathcal{J}_{2})\implies
h(\bm{Y}^{\mathcal{J}_{1}})\geq h(\bm{Y}^{\mathcal{J}_{2}})$

if we reverse the inequalities in Equations (C1) and (C2) and keep Equation
(C3), or if we reverse the inequality in Equation (C3) and keep Equations (C1)
and C2. The choice of which version to use depends on whether small or large
values of the test statistic are critical.

######  Theorem 3.

TMTI∞, tTMTI∞ and rtTMTI∞ all satisfy the conditions of Lemma 2.

######  Remark 5.

Even though the TMTI∞ variants all satisfy the conditions in Lemma 2, not all
TMTIn variants do. To see this, consider two sets of $p$-values,
$\bm{p}_{1}=(0.25,0.50,0.75)$ and $\bm{p}_{2}=(0.2,0.5,0.75)$. Then
$\bm{Y}_{1}=(0.58,0.5,0.42)$, making $0.42$ the first local minimum of
$\bm{Y}_{1}$, and $\bm{Y}_{2}=(0.49,0.5,0.42)$, making $0.49$ the first local
minimum of $\bm{Y}_{2}$. Thus, we have $\bm{Y}_{2}\leq\bm{Y}_{1}$ but
$h_{TMTI_{1}}(\bm{Y_{2}})>h_{TMTI_{1}}(\bm{Y_{1}})$.

######  Theorem 4.

Let $p_{\mathcal{I}}$, $F_{(1)},\dots,F_{|\mathcal{I}|}$, $\mathcal{X}$, $h$
and $\bm{Y}^{\mathcal{J}}$ be defined as in Lemma 2. Assume that the
underlying distribution of $p_{\mathcal{I}}$ is exchangeable. If Equations
(C1), (C2) and (C3) are satisfied, then a Closed Testing Procedure using
$h(\bm{Y}^{\mathcal{J}})$ as test statistic can be used to obtain control of
the FWER for all marginal hypotheses in at most $\frac{1}{2}m(m-1)$ steps.

######  Remark 6.

The result in Lemma 2 and its converse in Remark 4 does not only apply to TMTI
statistics. For example, letting $F_{(1)},\dots,F_{(m)}$ be the identity
mappings and $h(\bm{x})\coloneqq-2\sum_{i=1}^{m}\log x_{i}$ we obtain the
Fisher Combination Test, which then gives us the well-known
$\mathcal{O}(m^{2})$ shortcut described e.g.,in Zaykin et al. (2002).
Similarly, letting $h(\bm{x})=\frac{1}{m}\sum_{i=1}^{m}\tan((0.5-x_{i})\pi)$,
we obtain the unweighted Cauchy Combination Test.

In Algorithm 1, we give an example of how this shortcut procedure can be
implemented to return adjusted $p$-values for the tests of all marginal
hypotheses.

Input: Sorted $p$-values $p_{1}\leq\dots\leq p_{m}$ for tests of hypotheses
$H_{1},\dots,H_{m}$, a significance level $\alpha$

Output: Adjusted $p$-values for the tests of $H_{1},\dots,H_{m}$

Construct an empty $m\times m$ matrix $Q$.

for  _$i=1,\dots,m-1$ _ do

$c\leftarrow m$.

$Q_{i,c}\leftarrow p_{i}$.

for  _$j=m,\dots,i+1$ _ do

$c\leftarrow c-1$.

Test the hypothesis $\left(\bigcap_{k=m}^{j}H_{k}\right)\cap H_{i}$ and save
the $p$-value as $p_{i,m:j}$.

$Q_{i,c}\leftarrow p_{i,m:j}$.

for  _$i=1,\dots,m-1$ _ do

$Q_{i,(i+1):m}\leftarrow Q_{i,i}$

$\tilde{p}_{i}\leftarrow\max Q_{1:m,i}$

$\tilde{p}_{m}\leftarrow\max Q_{1:m,m}$

Return $\tilde{p_{1}},\dots,\tilde{p}_{m}$ as adjusted $p$-values for
$H_{1},\dots,H_{m}$.

Algorithm 1 Shortcut Closed Testing Procedure for statistics satisfying
Conditions (C1), (C2) and (C3)

If the practitioner is content with having only a lower bound on the adjusted
$p$-value, whenever the test is not rejected at a chosen level, Theorem 4
provides an upper bound on the number of steps required to complete the Closed
Testing Procedure. For instance, if the global hypothesis cannot be rejected,
no marginal hypothesis can be rejected, and the procedure can therefore be
stopped and the $p$-value for the test of the global hypothesis can be used as
a lower bound for all adjusted $p$-values. Stopping the procedure early can
speed up computations considerably, especially when $m$ is large but very few
hypotheses can be rejected. Furthermore, Lemma 2 also implies that an adjusted
$p$-value for a single elementary hypothesis can be obtained in only $m$
steps. In practice, one is often only interested in obtaining adjusted
$p$-values for the hypotheses, for which the marginal $p$-value is significant
(as the remaining hypotheses cannot be rejected in a Closed Testing
Procedure). Thus, if there are, say, $n$ elementary hypotheses for which the
marginal $p$-value is significant, these can be adjusted in $\mathcal{O}(nm)$
complexity.

We have exemplified in Figure 3 how the reduced test-tree looks for the Closed
Testing Procedure when applying any test that satisfies the conditions of
Lemma 2 in the case of $m=4$ total tests, where the hypotheses are labeled
such that $H_{i}$ corresponds to the $i$th lowest $p$-value. To obtain an
adjusted $p$-value for any marginal hypothesis, say $H_{i}$, one takes the
maximal $p$-value from the test of all ancestral hypotheses in the graph. For
example, the adjusted $p$-value for the test of $H_{2}$ would be the largest
of the $p$-values from the tests of $H_{2}$, $H_{2,4}$, $H_{2,3,4}$ and
$H_{1,2,3,4}$.

${{H_{1,2,3,4}}}$${{H_{1,3,4}}}$${{H_{2,3,4}}}$${{H_{1,4}}}$${{H_{2,4}}}$${{H_{3,4}}}$${H_{1}}$${H_{2}}$${H_{3}}$${H_{4}}$
Figure 3: Test procedure with $m=4$

###  5.1 A remark on mixture strategies in Closed Testing Procedures

Definition 2 of a Closed Testing Procedure and the subsequent Theorem 2 on
FWER control make no assumptions on the choice of local tests,
$\phi^{\mathcal{J}}$, and these can in principle vary across all intersection
hypotheses to be tested. We only require that every local test,
$\phi^{\mathcal{J}}$, is a valid $\alpha$-level test. The natural choice is to
use the same kind of test at every intersection hypothesis, e.g., TMTI∞.
However, we can in principle employ any choice of local test. In some cases,
we argue, it is reasonable to use different local tests for different kinds of
intersection hypotheses. When we go through a Closed Testing Procedure, we are
going to consider tests of many different kinds of hypotheses, and in
particular different kinds of alternative hypotheses. As previously discussed,
TMTI∞ has slightly lower power than other methods in situations where signals
are extremely sparse. According to the shortcut strategy outlined in Algorithm
1, we need only consider the $|\mathcal{J}|-1$ largest $p$-values alongside
the $j$th $p$-value, when testing all supersets of $\\{j\\}$ of size
$|\mathcal{J}|$. When $|\mathcal{J}|$ is small or when false hypotheses are
sparse, it is likely that these intersection hypotheses will consist of a
single false hypothesis (if any) with all the remaining hypotheses being true.
It makes sense, then, to employ a different test in these situations, which
has power against alternatives of sparse signals. However, this alternative
test needs to satisfy the same shortcut as TMTI∞ in order to be employed
across all supersets of equal size. One such choice is rtTMTI∞ with a low
choice of $K$ – e.g., $K=1$ – as this method satisfies the same shortcut as
TMTI∞, but has higher power against sparse alternatives, as discussed in
Section 4. Given a number of hypotheses, say $m$, of which we expect $F<m$ to
be false, we could for example conduct the Closed Testing Procedure by
employing rtTMTI∞ with a small $K$ whenever we consider supersets of size at
most $m-F$ and TMTI∞ whenever we consider supersets of size greater than
$m-F$. We call such a strategy a mixture Closed Testing Procedure. We return
to mixture Closed Testing Procedures in Section 8, where we analyse a real
dataset. The reasoning is that once we start considering supersets of size
greater than $m-F$, the intersections hypotheses considered in the shortcut
procedure will potentially include multiple false hypotheses, while they will
include at most one false hypothesis when considering supersets of size less
than $m-F$.

We stress that when employing a mixture Closed Testing Procedure, the choice
of local tests should be made a priori and not be data-driven, so as not to
incur new multiplicity problems.

##  6 The number of false hypotheses in a rejection set and $k$-FWER control

Given a set of hypotheses, indexed by $\mathcal{J}$, such that we can safely
reject the joint hypothesis $H_{\mathcal{J}}$ – i.e., we conclude that at
least one of the hypotheses in $\mathcal{J}$ is false – the natural question
is then how many of the hypotheses in $\mathcal{J}$ are false. To answer this,
Goeman and Solari (2011) provide a simple way of generating a $1-\alpha$
confidence set for the number of false hypotheses contained in $\mathcal{J}$
when using Closed Testing Procedures. Let
$\mathcal{R}\subseteq\operatorname{powerset}(\mathcal{J})\backslash\\{\emptyset\\}$
be the set of all intersection hypotheses that can be rejected by any Closed
Testing Procedure. Define $\tau$ to be the number of true hypotheses in
$\mathcal{J}$ and
$t_{\alpha}\coloneqq\max\\{|\mathcal{K}|:\mathcal{K}\subseteq\mathcal{J},\mathcal{K}\not\in\mathcal{R}\\}$
to be the size of the largest intersection hypothesis in $\mathcal{J}$ that
can not be rejected by the Closed Testing Procedure.

######  Theorem 5 (Goeman and Solari (2011)).

The sets $\\{0,\dots,t_{\alpha}\\}$ and
$\\{|\mathcal{J}|-t_{\alpha},\dots,|\mathcal{J}|\\}$ are $1-\alpha$ confidence
sets for the number of true hypotheses, $\tau$, and the number of false
hypotheses, $|\mathcal{J}|-\tau$, respectively.

This remarkably simple theorem has the implication, that we can generate
confidence sets for the number of false hypotheses among all those tested in
only $t_{\alpha}$ steps when using any test procedure satisfying the
conditions of Lemma 2, assuming that the $p$-values are realized from an
exchangeable distribution. We describe in Algorithm 2 how to do this.

The quantity $t_{\alpha}$ depends on the choice of test used on each
intersection. Different tests have power against different alternatives, and a
test that has low power for a particular intersection hypothesis will be more
likely to not reject that hypothesis. Thus, if the chosen test has low power
for the particular data, the resulting confidence set for the number of false
hypotheses will be conservative. In contrast, if the chosen test has high
power against the particular alternative, the confidence set tightens.

As noted in Goeman and Solari (2011), we can also apply Theorem 5 as a way of
controlling the $k$-FWER, i.e., the probability of making at least $k$ false
rejections. To control the $k$-FWER, we find the largest $i$ such that
$t_{\alpha}<k$ when calculated for the set of hypotheses yielding the $i$
smallest $p$-values. That is, if we find that $F$ hypotheses in a set, say
$\mathcal{J}$, are false with $1-\alpha$ confidence, then we can reject every
hypothesis in that set while controlling the $k$-FWER at
$k=|\mathcal{J}|-F+1$. This can be done in $\mathcal{O}(m^{3})$ time. This is
because determining $t_{\alpha}$ from a set $\mathcal{J}$ is done in
$\mathcal{O}(|\mathcal{J}|^{2})$ time, as described in Algorithm 2, and we now
need to do this for subsets of increasing (or decreasing, depending on the
search direction) size. It is worth noting, that the $k$ at which the
practitioner wishes to control the $k$-FWER need not be chosen a priori, as
the confidence sets are simultaneous for all choices of $\mathcal{J}$, simply
because the closure of each $\mathcal{J}$ is contained within the full
closure.

Input: Hypotheses $(H_{i})_{i\in\mathcal{I}}$, an ordered set
$\mathcal{J}\subseteq\mathcal{I}$, ordered $p$-values $p_{1}\leq\dots\leq
p_{|\mathcal{I} |}$ .

Output: A $1-\alpha$ confidence set for the number of false hypotheses in
$\mathcal{J}$.

Initialization  __

$t_{\alpha}\leftarrow 0$

$m\leftarrow|\mathcal{J}|$

$\tilde{p}\leftarrow(p_{j})_{j\in\mathcal{J}}$

if  _$m=|\mathcal{I}|$ _ then

for  _$i=m,\dots,1$ _ do

$\tilde{\mathcal{J}}\leftarrow\\{m-i+1,\dots,m\\}$

Let $p_{\tilde{\mathcal{J}}}$ be the $p$-value for the test of
$\bigcap_{j\in\tilde{\mathcal{J}}}H_{j}$.

if  _$p_{\tilde{\mathcal{J}}}\geq\alpha$ _ then

Set $t_{\alpha}\leftarrow i$

Break the loop and return
$\\{|\mathcal{J}|-t_{\alpha},\dots,|\mathcal{J}|\\}$.

else

for  _$i=m,\dots,1$ _ do

$\tilde{\mathcal{J}}\leftarrow\\{m-i+1,\dots,m\\}$

$\hat{p}\leftarrow(p_{i})_{i\in\mathcal{I}\backslash\tilde{\mathcal{J}}}$

Let $p_{\tilde{\mathcal{J}}}$ be the $p$-value for the test of
$\bigcap_{j\in{\tilde{\mathcal{J}}}}H_{j}$

for  _$j=1,\dots,|\mathcal{I}\backslash\tilde{\mathcal{J}}|$ _ do

Define ${\tilde{\mathcal{J}}}_{2}$ as $\tilde{\mathcal{J}}$ appended with the
$j$ largest values of $\hat{\mathcal{I}}$.

Update $p_{\tilde{\mathcal{J}}}$ as the $p$-value from the test of
$\bigcap_{j\in{\tilde{\mathcal{J}}}_{2}}H_{j}$.

if  _$p_{\tilde{\mathcal{J}}}\geq\alpha$ _ then

Set $t_{\alpha}\leftarrow i$

Break the loop and return
$\\{|\mathcal{J}|-t_{\alpha},\dots,|\mathcal{J}|\\}$.

Return $\\{|\mathcal{J}|-t_{\alpha},\dots,|\mathcal{J}|\\}$.

Algorithm 2 Confidence set for the number of false hypotheses among
$\mathcal{J}\subseteq\mathcal{I}$

##  7 Additional computational considerations

The computational time of a carrying out a Closed Testing Procedure when using
the above shortcuts is manageable for reasonable values of $m$. For example,
computing adjusted $p$-values for a set of $m=100$ $p$-values take roughly two
seconds on a standard laptop using single-threaded computations. Still, there
is a considerable amount of computational effort involved in carrying out a
Closed Testing Procedure when $m$ is sufficiently large, in part because the
CDFs of the TMTI statistics will have to be bootstrapped. To further reduce
the computational burden, we offer the following result.

######  Lemma 3.

Let $\mathcal{J}_{1}$ and $\mathcal{J}_{2}$ be sets such that
$\mathcal{J}_{1}\subsetneq\mathcal{J}_{2}$. Then

$\forall x\in(0,1):\hskip
28.45274pt\gamma^{\mathcal{J}_{1}}(x)<\gamma^{\mathcal{J}_{2}}(x).$

That is, a conservative $p$-value for the test of $H_{\mathcal{J}_{1}}$ can be
obtained by using $\gamma^{\mathcal{J}_{2}}$ instead of
$\gamma^{\mathcal{J}_{1}}$ when computing the $p$-value.

The purpose of Lemma 3 is that the user can choose to skip the bootstrap at
several layers of the Closed Testing Procedure and instead simply use the CDF
of a higher layer. This improves the running time of the algorithm at the cost
of using conservative $p$-values at the layers where the bootstrap was
skipped. Exactly how costly this trade-off is, depends on how many layers are
skipped each time. We conjecture that the $p$-value will only be slightly
conservative if the number of layers skipped is small relative to the size of
the subsets considered.

##  8 An example using real data

In this section, we give an example of how TMTI∞ performs against other
methods when applied to real data. For this purpose, we consider data from the
National Assessment of Educational Progress on the state-wise changes in
eighth-grade mathematics achievements from 1990 to 1992. This data is, among
other places, presented in Williams et al. (1999), where the authors compute
two-sided $T$-tests for the mean change in mathematical achievements over the
two-year period, to quantify whether or not any particular state has
progressed or worsened during that time period. The original data includes one
$p$-value of exactly $0$. We have rounded this to be $0.00001$ here to ensure
Type I error control. The same data is used in Benjamini and Hochberg (2000),
where the authors find that mathematics achievements have changed
significantly in $24$ of the $34$ states. However, the authors control the
more lenient False Discovery Rate, and thus their results are not directly
comparable to the ones presented here. Williams et al. (1999) apply a
significance level of $0.025$ instead of the usual $0.05$. In Benjamini and
Hochberg (2000), the authors apply the usual $0.05$ significance level but
have doubled all $p$-values such that the results are comparable. We have done
the same here. The data, adapted from Williams et al. (1999), is presented in
Table 1. Given that we only have access to summary statistics, we assume that
all $p$-values are independent. Whether this is a reasonable assumption can be
debated.

There are several questions regarding this data that may be of interest to the
practitioner:

  1. 1.

Did mathematics achievements change significantly in any state from 1990 to
1992?

  2. 2.

In how many states did mathematics achievements change significantly?

  3. 3.

In what states did mathematics achievements change significantly?

To answer the first question, we can, for example, apply TMTI∞ to obtain a
$p$-value. Doing so results in a $p$-value of $1.58\times 10^{-13}$. Thus, we
find evidence that mathematics education has changed significantly in at least
one of the $34$ states.

To answer the remaining two questions, we apply TMTI∞ in a Closed Testing
Procedure as well as a mixture Closed Testing Procedure, using rtTMTI∞ with
$K=1$ whenever we consider fewer than $15$ hypotheses and TMTI∞ when
considering more. For comparison, we also apply the standard Bonferroni
correction as well as the Fisher Combination Test in a Closed Testing
Procedure and a Rank Truncated Product Method/Fisher Combination Test (denoted
rTPM/FCT) mixture, using the Rank Truncated Product Method with $K=1$ for
intersection hypotheses of size at most $15$. The choice of $K=1$ for
intersection hypotheses smaller than $15$ corresponds to a belief that at most
$15$ hypotheses are true. Here, we have chosen $15$ at random, but a
practitioner with subject matter knowledge can choose this based on prior
knowledge.

By applying Algorithm 2 with TMTI∞ we find that $\\{23,\dots,34\\}$ is a
$95\%$ confidence set for the number of states in which mathematics
achievements have changed. In contrast, using the Fisher Combination Test, the
confidence set is $\\{19,\dots,34\\}$. The same confidence set is found when
applying the mixture strategies. That is, we can say with $95\%$ confidence,
that mathematics achievements have changed significantly in at least $23$
states when using TMTI∞. The improved performance of TMTI∞ over the Fisher
Combination Test here is likely due to TMTI∞ generally having high power
across a wide range of settings (see Section 4), whereas the Fisher
Combination Test lacks power in settings with sparse, strong signals. When
carrying out a Closed Testing Procedure, we test many different kinds of joint
hypotheses (i.e., some containing more false hypotheses than others), and it
is thus beneficial that the employed test has high power in many different
settings.

To determine which of the hypotheses we can say with certainty are false while
controlling the FWER, we applied Algorithm 1. Here, TMTI∞, the Fisher
Combination Test and the Bonferroni correction perform identically and are
capable of rejecting the bottom four hypotheses. In contrast, when we apply
the two mixture strategies, we can reject the bottom seven hypotheses. Thus,
by incorporating prior knowledge, we can increase the size of the rejection
set for which we have FWER control.

To find which of the hypotheses can be rejected while controlling the more
general $k$-FWER, we consider sets of increasing size of the smallest
$p$-values, each time calculating $t_{\alpha}$. For any chosen set, say
$\mathcal{J}$, we can reject the entire set while controlling the $k$-FWER at
$t_{\alpha}+1$. Doing so, we find that the 11 hypotheses giving rise to the
smallest $p$-values can be rejected by the TMTI∞ while controlling the
$k$-FWER at $k=2$. In contrast, the mixture Closed Testing Procedure is only
capable of rejecting the bottom eight hypotheses while controlling the
$k$-FWER at $k=2$. If we are willing to accept a more lenient $k$-FWER control
of $k=5$, we are capable of rejecting the bottom 22 hypotheses using the TMTI∞
and the bottom 11 hypotheses using the rtTMTI∞/TMTI∞ mixture. In Figure 4, we
have summarized the associated $k$ at which we control the $k$-FWER at, when
rejecting the bottom $t$ hypotheses, for $t=1,\dots,34$, when using TMTI∞, the
rtTMTI∞/TMTI∞ mixture, the Fisher Combination Test and the rTPM/FCT mixture,
respectively. Here, we note that TMTI∞ is weakly better than the Fisher
Combination Test except for at a single rejection set, $\\{1,\dots,13\\}$.

State | GA | AR | AL | NJ | NE | ND | DE | MI
---|---|---|---|---|---|---|---|---
$p$-value (%) | 85.628 | 60.282 | 44.008 | 41.998 | 38.640 | 36.890 | 31.162 | 23.522
| LA | IN | WI | VA | WV | MD | CA | OH
| 20.964 | 19.388 | 15.872 | 14.374 | 10.026 | 8.226 | 7.912 | 6.590
| NY | PA | FL | WY | NM | CT | OK | KY
| 5.802 | 5.572 | 5.490 | 4.678 | 4.650 | 4.104 | 2.036 | 0.964
| AZ | ID | TX | CO | IA | NH | NC | HI
| 0.904 | 0.748 | 0.404 | 0.282 | 0.200 | 0.180 | 0.002 | 0.002
| MN | RI |  |  |  |  |  |
| 0.002 | 0.001 |  |  |  |  |  |
Table 1: States and their $p$-values for $T$-tests of changes in mathematics
achievements from 1990 to 1992. Figure 4: Overview of the different Closed
Testing Procedure methods employed and the $k$ at which they control the
$k$-FWER, when rejecting the bottom $t$ hypotheses, for $t=1,\dots,34$. The
two mixture strategies, rtTMTI∞/TMTI∞ and rTPM/FCT, are colored the same, as
their results are identical.

That the two mixture strategies have higher power to detect differences when
controlling the FWER than TMTI∞ and the Fisher Combination Test, but lower
power to detect differences when using the more lenient $k$-FWER control, may
seem counter-intuitive and requires some exposition. The difference lies in
what intersection hypotheses need to be considered in the full Closed Testing
Procedure test tree. When controlling the FWER, we are in principle looking
through all intersection hypotheses, and then we use the maximal $p$-values
along the closure of each atom as the adjusted $p$-values. As outlined in
Lemma 2, however, we need only consider the part of the closure that contains
subsets containing only the atom and the largest $p$-values. When we apply
Algorithm 2 iteratively to obtain $k$-FWER control, we are considering the
closures, not of atoms, but of intersections. Put differently, consider the
index set $\mathcal{J}\coloneqq\\{1,\dots,t\\}$, for some $t$, of the $t$
smallest $p$-values. We are then going to consider the closure of $\\{t\\}$ in
$\mathcal{J}$, i.e., $\\{t\\}_{\mathcal{J}}^{*}$ at first. For all sets in
this closure, say $\tilde{\mathcal{J}}\in\\{t\\}_{\mathcal{J}}^{*}$, we are
then going to calculate the adjusted $p$-value as the maximal $p$-value along
the closure of $\tilde{\mathcal{J}}$ in $\mathcal{I}$, i.e.,
$\tilde{\mathcal{J}}_{\mathcal{I}}^{*}$. These sets are not, as they were in
the ordinary Closed Testing Procedure, sets consisting of an atom unioned with
the largest $p$-values, but rather several, possibly neighboring, atoms
unioned with the largest $p$-values. We constructed the mixture strategies to
have higher power in situations in which we considered intersection hypotheses
with only a single false hypothesis in them. Using this method, we are now
considering sets that possibly have multiple false hypotheses in them, even
when the total number of hypotheses included in the set is low – which is when
TMTI∞ gains its power. In contrast, rtTMTI∞ with a small $K$ loses power, when
there are more than $K$ false hypotheses present in the intersection
hypothesis.

A detailed table with adjusted $p$-values for all of the tests employed here
can be found in C.

##  9 Conclusion

We have introduced the ‘Too Many, Too Improbable’ (TMTI) family of combination
test statistics for testing joint hypotheses among $m$ marginal hypotheses.
The TMTI family includes truncation-based tests, similar to those of Zaykin et
al. (2002) and Dudbridge and Koeleman (2003), for testing global hypotheses
against sparse alternatives. We have shown in Section 4 that the TMTI tests
outperforms other combination tests in many situations. In particular, we
found that TMTI∞ and tTMTI∞ were the only tests that were able to achieve high
power both when signals are dense but weak and when signals are sparse but
strong. Although we found in all scenarios that there was at least one other
test that performed equally as well as TMTI∞ tTMTI∞, no other combination
tests had similar performance across all scenarios. This property is useful,
e.g., if one has no a priori knowledge about the sparsity and strength of
signals and for generating $1-\alpha$ confidence sets for the number of false
hypotheses in a rejection set.

In Section 5, we have given an $\mathcal{O}(m^{2})$ shortcut for controlling
the Family-Wise Error Rate using Closed Testing Procedures (Marcus et al.,
1976) for a large class of test statistics, which includes the TMTI family of
test statistics, but also the Cauchy Combination Test among others. Using this
shortcut, we use the work of Goeman and Solari (2011) in Section 6 to develop
an $\mathcal{O}(m^{3})$ algorithm for controlling the generalized FWER as well
as an $\mathcal{O}(m)$ algorithm for obtaining $1-\alpha$ confidence sets for
the number of false hypotheses among all hypotheses.

In Section 8, we applied a TMTI test in a Closed Testing Procedure, as well as
a mixture – i.e., varying local tests across the Closed Testing Procedure – of
two TMTI tests, to a real dataset and compared it to the Fisher Combination
Test applied in a Closed Testing Procedure. Here we found that all TMTI tests
were able to reject the same hypotheses as the Fisher Combination Test, but
that the TMTI test generated a narrower confidence set for the number of false
hypotheses among the collection of considered hypotheses. Additionally, we
found that by employing mixture strategies, we were able to reject more
hypotheses than with standard methods. However, the mixture strategies
performed worse when controlling the $k$-FWER with $k\geq 2$.

## Supplementary material

Proofs of all lemmas and theorems are supplied in the appendix. Furthermore,
the appendix includes further simulations to support those of Section 4 and a
detailed table containing the adjusted $p$-values of the procedures applied in
Section 8.

The TMTI family of test statistics and the shortcuts for $k$-FWER and
confidence sets is implemented in the R package TMTI available on CRAN.

## References

  * Benjamini and Hochberg (1995) Benjamini, Y., Hochberg, Y., 1995\.  Controlling the false discovery rate: a practical and powerful approach to multiple testing.  Journal of the Royal statistical society: series B (Methodological) 57, 289–300.
  * Benjamini and Hochberg (2000) Benjamini, Y., Hochberg, Y., 2000\.  On the adaptive control of the false discovery rate in multiple testing with independent statistics.  Journal of educational and Behavioral Statistics 25, 60–83.
  * Brown (1975) Brown, M.B., 1975.  400: A method for combining non-independent, one-sided tests of significance.  Biometrics , 987–992.
  * Chen et al. (2020) Chen, Y., Liu, P., Tan, K.S., Wang, R., 2020.  Trade-off between validity and efficiency of merging p-values under arbitrary dependence.  arXiv preprint arXiv:2007.12366 .
  * Dobriban (2020) Dobriban, E., 2020.  Fast closed testing for exchangeable local tests.  Biometrika 107, 761–768.
  * Dudbridge and Koeleman (2003) Dudbridge, F., Koeleman, B.P., 2003\.  Rank truncated product of p-values, with application to genomewide association scans.  Genetic Epidemiology: The Official Publication of the International Genetic Epidemiology Society 25, 360–366.
  * Fisher (1992) Fisher, R.A., 1992.  Statistical methods for research workers, in: Breakthroughs in statistics. Springer, pp. 66–70.
  * Goeman and Solari (2011) Goeman, J.J., Solari, A., 2011\.  Multiple testing for exploratory research.  Statistical Science 26, 584–597.
  * Grechanovsky and Hochberg (1999) Grechanovsky, E., Hochberg, Y., 1999\.  Closed procedures are better and often admit a shortcut.  Journal of Statistical Planning and Inference 76, 79–91.
  * Kessy et al. (2018) Kessy, A., Lewin, A., Strimmer, K., 2018.  Optimal whitening and decorrelation.  The American Statistician 72, 309–314.
  * Kost and McDermott (2002) Kost, J.T., McDermott, M.P., 2002\.  Combining dependent p-values.  Statistics & Probability Letters 60, 183–190.
  * Littell and Folks (1973) Littell, R.C., Folks, J.L., 1973\.  Asymptotic optimality of fisher’s method of combining independent tests ii.  Journal of the American Statistical Association 68, 193–194.
  * Liu et al. (2019) Liu, Y., Chen, S., Li, Z., Morrison, A.C., Boerwinkle, E., Lin, X., 2019.  Acat: a fast and powerful p value combination method for rare-variant analysis in sequencing studies.  The American Journal of Human Genetics 104, 410–421.
  * Liu and Xie (2020) Liu, Y., Xie, J., 2020.  Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures.  Journal of the American Statistical Association 115, 393–402.
  * Marcus et al. (1976) Marcus, R., Eric, P., Gabriel, K.R., 1976.  On closed testing procedures with special reference to ordered analysis of variance.  Biometrika 63, 655–660.
  * Perneger (1998) Perneger, T.V., 1998.  What’s wrong with bonferroni adjustments.  Bmj 316, 1236–1238.
  * Peters et al. (2016) Peters, J., Bühlmann, P., Meinshausen, N., 2016.  Causal inference by using invariant prediction: identification and confidence intervals.  Journal of the Royal Statistical Society: Series B (Statistical Methodology) 78, 947–1012.
  * Šidák (1967) Šidák, Z., 1967.  Rectangular confidence regions for the means of multivariate normal distributions.  Journal of the American Statistical Association 62, 626–633.
  * Tian et al. (2021) Tian, J., Chen, X., Katsevich, E., Goeman, J., Ramdas, A., 2021.  Large-scale simultaneous inference under dependence.  arXiv preprint arXiv:2102.11253 .
  * Vovk and Wang (2020) Vovk, V., Wang, R., 2020.  Combining p-values via averaging.  Biometrika 107, 791–808.
  * Williams et al. (1999) Williams, V.S., Jones, L.V., Tukey, J.W., 1999.  Controlling error in multiple comparisons, with examples from state-to-state differences in educational achievement.  Journal of educational and behavioral statistics 24, 42–69.
  * Wilson (2019) Wilson, D.J., 2019.  The harmonic mean p-value for combining dependent tests.  Proceedings of the National Academy of Sciences 116, 1195–1200.
  * Wu et al. (2011) Wu, M.C., Lee, S., Cai, T., Li, Y., Boehnke, M., Lin, X., 2011.  Rare-variant association testing for sequencing data with the sequence kernel association test.  The American Journal of Human Genetics 89, 82–93.
  * Zaykin et al. (2002) Zaykin, D.V., Zhivotovsky, L.A., Westfall, P.H., Weir, B.S., 2002\.  Truncated product method for combining p-values.  Genetic Epidemiology: The Official Publication of the International Genetic Epidemiology Society 22, 170–185.

##  Appendix A Proofs

###  A.1 Proof of Lemma 1

This holds trivially, as the first $Y$ smaller than the following $n$ must
necessarily be larger when we only consider the first $c<|\mathcal{I}|$ values
of $\bm{Y}^{\mathcal{I}}$ than if we consider the full sequence.

###  A.2 Proof of Theorem 1

Assume first that $c=m$. The order statistics $(P_{(1)},\dots,P_{(m)})$ have a
constant joint density $m!$ on the simplex
$\\{\bm{p}\in[0,1]^{m}:p_{1}\leq\dots\leq p_{m}\\}$. Thus

$\displaystyle\gamma^{\mathcal{I}}(x)\coloneqq$
$\displaystyle{\mathbb{P}}(\min(\beta(1,m)(P_{(1)}),\dots,\beta(m,1)(P_{(m)}))\leq
x)$ $\displaystyle=$ $\displaystyle
1-{\mathbb{P}}(\beta(1,m)(P_{(1)})>x,\dots,\beta(m,1)(P_{(m)})>x)$
$\displaystyle=$ $\displaystyle
1-{\mathbb{P}}(P_{(1)}>{\beta^{-1}(1,m)(x)},\dots,P_{(m)}>{\beta^{-1}(m,1)(x)})$
$\displaystyle=$ $\displaystyle
1-{\mathbb{P}}(P_{(1)}>x_{1},\dots,P_{(m)}>x_{m})$ $\displaystyle=$
$\displaystyle
1-m!\int_{x_{m}}^{1}\int_{x_{m-1}}^{q_{m}}\dotsc\int_{x_{2}}^{q_{3}}\int_{x_{1}}^{q_{2}}~{}dq_{1}~{}dq_{2}\dotsc
dq_{m-1}~{}dq_{m}.$ (5)

The integral in Equation 5 can be expressed as

$\int_{x_{m}}^{1}\int_{x_{m-1}}^{q_{m}}\dots\int_{x_{2}}^{q_{3}}\int_{x_{1}}^{q_{2}}~{}dq_{1}~{}dq_{2}\dots
dq_{m-1}~{}dq_{m}=Q_{m}(1;(1,-\bar{Q}_{1,c},\dots,-\bar{Q}_{i-1,c}))-\bar{Q}_{m,c},$

which in turn implies that

$\displaystyle\gamma^{\mathcal{I}}(x)$
$\displaystyle=1-m!(Q_{m}(1;(1,-\bar{Q}_{1,c},\dots,-\bar{Q}_{i-1,c}))-\bar{Q}_{m,c})$
$\displaystyle=1-m!\left(\frac{1}{m!}(1-x_{m}^{m})-\sum_{i=1}^{m-1}\bar{Q}_{i,c}\frac{1}{(m-i)!}(1-x_{m}^{m-i})\right)$
$\displaystyle=x_{m}^{m}+\sum_{i=1}^{m-1}\bar{Q}_{i,c}\frac{m!}{(m-i)!}(1-x_{m}^{m-i}).$

Now, assume that $c<m$. Then the expression is identical to the one given in
Equation (5), with the exception that the lower bound on all integrals after
the $c$’th from the inside out will be $x_{c}$. The CDF is therefore

$\gamma_{\infty,c}(x)=x_{c}^{m}+\sum_{i=1}^{m-1}\bar{Q}_{i,c}\frac{m!}{(m-i)!}(1-x_{c}^{m-i}).$

Now, let $c$ be a random variable given by
$c=\max\\{i\in\\{1,\dots,m\\}:p_{(i)}<\tau\\}$. Assume without loss of
generality that $p_{1}\leq\dots\leq p_{m}$. For any fixed integer $i\leq m$ we
note that $p_{1},\dots,p_{i}\mid c=i\sim U(0,\tau)$ and that the joint
distribution of $(p_{(1)},\dots,p_{(i)})$ conditional on $c=i$ has density
$i!/\tau^{i}$ on the simplex $\\{\bm{p}\in[0,1]^{i}:p_{1}\leq\dots\leq
p_{i}<\tau\\}$. By the Law of Total Probability, we can write

$\gamma_{\infty,c}(x)=\sum_{i=0}^{m}{\mathbb{P}}(Z_{\infty,c}<x\mid
c=i){\mathbb{P}}(c=i).$

The distribution of $c$ is binomial with probability parameter $\tau$ and size
$m$, i.e.,

${\mathbb{P}}(c=i)=\binom{m}{i}\tau^{i}(1-\tau)^{m-i}.$

Consider first the case of $i\geq 1$. We see that

$\displaystyle{\mathbb{P}}(Z_{\infty,i}\leq x\mid c=i)$
$\displaystyle=1-{\mathbb{P}}(P_{1}>x_{1},\dots,P_{i}>x_{i}\mid c=i)$
$\displaystyle=1-\frac{i!}{\tau^{i}}\int_{x_{i}}^{\tau}\int_{x_{i-1}}^{q_{i}}\dotsc\int_{x_{2}}^{q_{3}}\int_{x_{1}}^{q_{2}}~{}dq_{1}~{}dq_{2}\dotsc
dq_{i-1}~{}dq_{i}I(x_{i}<\tau)$
$\displaystyle=1-\frac{i!}{\tau^{i}}\left(\tilde{Q}_{i}-\bar{Q}_{i,m}\right)I(x_{i}<\tau).$

In the case of $i=0$, we see that ${\mathbb{P}}(Z_{\infty,c}<x\mid
c=0)={\mathbb{P}}(\beta(1,m)(P_{1})<x\mid c=0)$. The distribution of $P_{1}$
conditionally on no $p$-values falling below $\tau$ is uniform on the interval
$(\tau,1)$. Thus, we have $\beta(1,m)(P_{1})\mid c=0\sim
U(\beta(1,m)(\tau),1)$ and therefore

${\mathbb{P}}(Z_{\infty,c}<x\mid
c=0)=\frac{x-\beta(1,m)(\tau)}{1-\beta(1,m)(\tau)}I(x_{1}>x).$

Combining all of the above, we obtain

$\begin{split}\gamma_{\infty,c}(x)&=(1-\tau)^{m}\frac{x-\beta(1,m)(\tau)}{1-\beta(1,m)(\tau)}I(x_{1}>x)\\\
&+\sum_{i=1}^{m}\left[\binom{m}{i}\tau^{i}(1-\tau)^{m-i}\left\\{1-\frac{i!}{\tau^{i}}\left(\tilde{Q}_{i}-\bar{Q}_{i,m}\right)I(x_{i}\leq\tau)\right\\}\right]\end{split}$

which proves the claim.

###  A.3 Proof of Lemma 2

Assume without loss of generality that the $p$-values $\bm{p}_{\mathcal{I}}$
are already sorted, i.e., that $p_{1}\leq\dots\leq p_{m}$. Then $\eta$ is the
identity function and can therefore be omitted entirely. Fix a set
$\mathcal{J}\coloneqq\\{j_{1},\dots,j_{k}\\} \in\mathcal{J}^{k}$ and assume
without loss of generality that $j_{1}<\dots<j_{k}$. Fix $j\in\mathcal{J}$ and
$l\in\mathcal{I}\backslash\mathcal{J}$ such that $l<j$ and let
$\mathcal{J}_{l}^{-j}\coloneqq(\mathcal{J}\backslash\\{j\\})\cup\\{l\\}$,
i.e., $\mathcal{J}_{l}^{-j}$ is the set obtained by substituting $j$ for $l$
in $\mathcal{J}$. It suffices to show that

$h(\bm{Y}^{\mathcal{J}_{l}^{-j}})\leq h(\bm{Y}^{\mathcal{J}}).$

There are two cases to consider:

Case 1; $j=\min\mathcal{J}$: If $j$ is the smallest element in $\mathcal{J}$
then substituting it for $l$ does not change the ordering of $\mathcal{J}$. By
Condition (C1), it holds that

$F_{(1)}(p_{(l)})=:Y_{1}^{\mathcal{J}_{l}^{-j}}\leq
Y_{1}^{\mathcal{J}}\coloneqq F_{(1)}(p_{(j)}).$

As all other values for $\bm{Y}^{\mathcal{J}_{l}^{-j}}$ and
$\bm{Y}^{\mathcal{J}}$ are unchanged, it follows from Condition C3 that
$h(\bm{Y}^{\mathcal{J}_{l}^{-j}})\leq h(\bm{Y}^{\mathcal{J}})$.

Case 2; $j>\min\mathcal{J}$: Define by $\tilde{j}$ the largest index in
$\mathcal{J}$ smaller than $j$, i.e.

$\tilde{j}\coloneqq\max\\{i\in\mathcal{J}:i<j\\}.$

Suppose first that $\tilde{j}<l<j$. In this case, the ordering of
$\mathcal{J}$ is unchanged when substituting $j$ for $l$, making this case
isomorphic to case 1. If, on the other hand, $l<\tilde{j}$ the ordering of
$\mathcal{J}$ changes when substituting $j$ for $l$. Let $\hat{j}$ be the
smallest index in $\mathcal{J}$ larger than $l$, i.e.,

$\hat{j}\coloneqq\min\\{i\in\mathcal{J}:i>l\\}.$

Then we must show two things

  1. 1.

$Y_{l}^{\mathcal{J}_{l}^{-j}}\leq Y_{\hat{j}}^{\mathcal{J}}$.

  2. 2.

For all $i>\hat{j}$ it holds that $Y_{i}^{\mathcal{J}_{l}^{-j}}\leq
Y_{i}^{\mathcal{J}}$.

Let $\eta_{\mathcal{A}}:\mathcal{A}\to\mathcal{A}$ be a function sorting the
elements of a set $\mathcal{A}$. That is, $\eta_{\mathcal{A}}(a)=b$ if and
only if $a$ is the $b$’th lowest element in $\mathcal{A}$. We then see that
$\eta_{\mathcal{J}_{l}^{-j}}(l)=\eta_{\mathcal{J}}(\hat{j})$ and thus that the
first point above is satisfied as we have $p_{(l)}\leq p_{(\hat{j})}$ and
therefore $Y_{l}^{\mathcal{J}_{l}^{-j}}\leq Y_{\hat{j}}^{\mathcal{J}}$ by
Condition (C1). The second point above is satisfied by Condition (C2), as
$\eta_{\mathcal{J}_{l}^{-j}}(h)=\eta_{\mathcal{J}}(h)+1$ for any $h>\hat{j}$.
This proves Lemma 2.

###  A.4 Proof of Theorem 3

We start by reminding the reader that all three TMTI statistics have
$F_{(i)}(x)=\beta(i,m+1-i)(x)$, for $i=1,\dots,m$, regardless of the choice of
$n$. These functions are weakly increasing, as they are CDFs, thus satisfying
Condition (C1). Next, fix $x\in(0,1)$ and $i\in\mathcal{I}$ with $i<m$. We
then see that

$\displaystyle F_{(i+1)}(x)$
$\displaystyle=\sum_{h=i+1}^{k}\binom{k}{h}x^{h}(1-x)^{k-h}$
$\displaystyle=\beta(i+1,k+1-(i+1))(x)$
$\displaystyle<\binom{k}{i}x^{i}(1-x)^{k-i}+\sum_{h=i+1}^{k}\binom{k}{h}x^{h}(1-x)^{k-h}$
$\displaystyle=\sum_{h=i}^{k}\binom{k}{h}x^{h}(1-x)^{k-h}$
$\displaystyle=\beta(i,k+1-i)(x)$ $\displaystyle=F_{(i)}(x),$

by using Equation (2). Thus, Condition (C2) is satisfied.

Let $\mu>1$ and note that

$\displaystyle h_{TMTI_{\infty}}(\bm{Y})$ $\displaystyle\coloneqq\min\bm{Y}$
$\displaystyle h_{tTMTI_{\infty}}(\bm{Y})$
$\displaystyle\coloneqq\min(Y_{1}+\mu
I(\beta^{-1}(1,m)(Y_{1})>\alpha),\dots,Y_{m}+\mu
I(\beta^{-1}(m,1)(Y_{m})\geq\alpha))$ $\displaystyle
h_{rtTMTI_{\infty}}(\bm{Y})$ $\displaystyle\coloneqq\min(Y_{1},\dots,Y_{K}).$

It is immediate that both $h_{TMTI_{\infty}}$ and $h_{rtTMTI_{\infty}}$
satisfy Condition (C3), as the mapping $\bm{x}\mapsto\min\bm{x}$ is weakly
increasing in every coordinate. To see that $h_{tTMTI_{\infty}}$ satisfies
Condition (C3), note that for fixed $i\in\mathcal{I}$ it holds that
$\beta^{-1}(i,m+1-i)$ is strictly increasing, thus making $x\mapsto x+\mu
I(\beta^{-1}(i,m+1-i)(x)>\alpha)$ a weakly increasing mapping and therefore
also making $h_{tTMTI_{\infty}}$ a weakly increasing mapping.

###  A.5 Proof of Theorem 4

Assume without loss of generality that $\mathcal{I}=\\{1,\dots,m\\}$ and
$p_{1}\leq\dots\leq p_{m}$ and denote by
$\mathbb{P}_{h(\bm{Y}^{\mathcal{J}})}$ the CDF of $h(\bm{Y}^{\mathcal{J}})$.
The adjusted $p$-value for the test of some $H_{i}$ is the maximal $p$-value
across all intersection hypotheses in the closure of $\\{i\\}$ in
$\mathcal{I}$, i.e.

$p_{i}^{\star}\coloneqq\max_{\mathcal{J}\in\\{i\\}_{\mathcal{I}}^{*}}\mathbb{P}_{h(\bm{Y}^{\mathcal{J}})}\circ
h(\bm{Y}^{\mathcal{J}}).$

Let $\mathcal{J}_{\\{i\\}_{\mathcal{I}}^{*}}^{k}$ denote the set of all sets
in $\\{i\\}_{\mathcal{I}}^{*}$ of size $k$. Then

$\operatorname*{argmax}_{\mathcal{J}\in\mathcal{J}_{\\{i\\}_{\mathcal{I}}^{*}}^{k}}\mathbb{P}_{h(\bm{Y}^{\mathcal{J}})}\circ
h(\bm{Y}^{\mathcal{J}})=\begin{cases}\\{m-k,\dots,m\\},&\text{if }i\geq m-k\\\
\\{i,m-k+1,\dots,m\\},&\text{else}\end{cases}$

by Lemma 2. Let
$\overline{\\{i\\}_{\mathcal{I}}^{*}}\coloneqq\\{\\{i,m\\},\\{i,m-1,m\\},\dots,\\{i,\dots,m\\},\\{i-1,\dots,m\\},\dots,\\{1,\dots,m\\}\\}$.
Then, by Lemma 2

$p_{i}^{\star}=\max_{\mathcal{J}\in\overline{\\{i\\}_{\mathcal{I}}^{*}}}\mathbb{P}_{h(\bm{Y}^{\mathcal{J}})}\circ
h(\bm{Y}^{\mathcal{J}}).$

Thus, the adjusted $p$-value for any hypothesis $H_{i}$ can be obtained in
$|\overline{\\{i\\}_{\mathcal{I}}^{*}}|=m-1$ steps222Disregarding the first
step, as a marginal $p$-value for the test of $H_{i}$ is already supplied..
However, note for any $i$ that
$|\overline{\\{1\\}_{\mathcal{I}}^{*}}\cap\overline{\\{i\\}_{\mathcal{I}}^{*}}|=i-1$.
Therefore, the number of steps required to obtain an adjusted $p$-value for
all hypotheses is $\sum_{i=1}^{m}(m-i)=\frac{1}{2}m(m-1)$, as claimed.

###  A.6 Proof of Lemma 3

Assume without loss of generality that $\mathcal{J}_{1}=\\{1,\dots,m_{1}\\}$
and $\mathcal{J}_{2}=\\{1,\dots,m_{2}\\}$. Then $m_{1}<m_{2}$. Let
$L\coloneqq\operatorname*{argmin}_{j\in\mathcal{J}_{1}}Y_{j}^{\mathcal{J}_{1}}$.
Then

$Z^{\mathcal{J}_{1}}=\beta(L,m_{1}+1-L)(P_{(L)})<\beta(L,m_{2}+1-L)(P_{(L)}),$

which implies that $\min\bm{Y}^{\mathcal{J}_{2}}<\min\bm{Y}^{\mathcal{J}_{1}}$
and therefore
${\mathbb{P}}(\min\bm{Y}^{\mathcal{J}_{1}}<x)<{\mathbb{P}}(\min\bm{Y}^{\mathcal{J}_{2}}<x)$,
which is the claimed result.

##  Appendix B Further simulation studies

###  B.1 An investigation of the robustness of the TMTI CDFs against different
dependency structures

In this section we investigate how well the analytical expression of the TMTI
CDFs under the null distribution derived in Section 3.1 under an i.i.d.
assumption approximates the actual CDF of the TMTI statistics under different
dependency structures.

In the following, we let $\bm{X}\sim\mathcal{N}(\bm{0},\Sigma)$ be an
$m$-dimensional random vector with coordinate means of zero, where
$\operatorname{diag}(\Sigma)=(1~{}\dots~{}1)$. We then calculate $p$-values
$\bm{P}$ as

$\forall i\in\\{1,\dots,m\\}:\hskip 28.45274ptP_{i}\coloneqq
2(1-\Phi(|X_{i}|)),$

where $x\mapsto\Phi(x)$ is the CDF of a $N(0,1)$ distribution. This ensures
that each $P_{i}$ is uniform on $(0,1)$ and that the dependency structure of
$\bm{P}$ is fully determined by the covariance matrix $\Sigma$. We consider
three structures of $\Sigma$:

  1. 1.

Equicorrelated tests, where $\Sigma_{i,j}=\rho I(i\neq j)+I(i=j)$, for all
$i,j$ and some $\rho\in(0,1)$.

  2. 2.

Block-diagonal tests, where

$\Sigma=\begin{pmatrix}\Sigma_{1}&\dots&0\\\ \vdots&\ddots&\vdots\\\
0&\dots&\Sigma_{g}\end{pmatrix}$

for some $g<m$ and $\Sigma_{1},\dots,\Sigma_{g}$ are themselves equicorrelated
with parameter $\rho$.

  3. 3.

Autoregressive tests, where $\Sigma_{i,j}=\rho^{|i-j|}$.

The first point above can happen in a scenario like the simulation study
performed in B.2 where we combine $T$-tests performed on independent variables
but where the standard error is estimated on the basis of a number of
covariates. It is unlikely to that the correlation between the tests is high,
but it is none-zero. Nevertheless, we try this scenario even for large values
of $\rho$ in order to investigate what happens under extreme dependencies.

The second point in the above represents a scenario in which we have performed
multiple tests within $g$ groups or individuals that are independent from one
another, but where tests performed within the same group are not independent.
The last point represents, for example, a design where the tests are spatially
correlated, where dependence is highest for neighboring plots.

We perform the experiments with $m=200$ tests and different values of $\rho$.
For the block-diagonal experiment we set $g=40$, corresponding to $40$ groups
of five tests each. In each experiment we bootstrap the CDFs of TMTI∞,
tTMTI$\infty$ with $\tau=0.05$ and rtTMTI∞ with $K=5$ and $K=1$ both under an
assumption of i.i.d. tests and under the actual dependency structure. For
comparison, we have also included the Cauchy Combination Test and Harmonic
Mean $p$-value tests. We then plot calibration curves, i.e., the curve
$x\mapsto(\text{actual }\operatorname{CDF}(x),\text{i.i.d.\
}\operatorname{CDF}(x))$. If the i.i.d. CDF is robust against departures from
independence then this curve will lie exactly on the diagonal of the unit
square. If the i.i.d. CDF is conservative it will lie above the diagonal of
the unit square and if it is anti-conservative it will lie below the diagonal
of the unit square. The results are presented in Figure 5 and again in Figure
6 where we have zoomed in on the square $(0,0.1)\times(0,0.1)$. From the
figures, we see that weak dependencies generally do not affect the CDFs of the
TMTI statistics by much but stronger dependencies have a large effect on the
CDFs. This is similar to what we see for the Cauchy Combination Test and
Harmonic Mean $p$-value statistics. Although both of these are claimed to be
robust against dependence (Liu and Xie, 2020; Wilson, 2019), we see that both
of these also become anti-conservative in their lower tails when there is
sufficiently strong dependence. The Harmonic Mean $p$-value appears to
generally be more anti-conservative than the TMTI statistics, while the Cauchy
Combination Test performs slightly better than rtTMTI∞ with $K=5$, but worse
than rtTMTI∞ with $K=1$. In particular, we see that when dependencies are
strong, TMTI∞ is very anti-conservative in its lower tail implying a loss of
Type I error control. However, its truncated variants, tTMTI∞ and rtTMTI∞, are
less affected by the dependencies and are only slightly anti-conservative in
many cases with strong dependencies. In particular, rtTMTI∞ with $K=1$,
corresponding to only using the smallest $p$-value, is very robust against
most dependencies and, in contrast to the other CDFs presented, never anti-
conservative. Only in the equicorrelated experiment when correlations are
strong does it deviate from the unit square diagonal and here it is
conservative. We also note that the CDFs of all statistics are conservative at
their upper tails. However, it is generally only interesting to see how
conservative or anti-conservative the i.i.d. approximations are in the regions
around typical significance level as conservativeness or anti-conservativeness
here can be the difference between making a Type I or II error and making no
error.

These experiments suggest that rtTMTI∞ with $K=1$ is robust against departures
from independence and is thus reasonable to use when making no assumptions on
the dependency structure of the $p$-values in question. In contrast, the other
TMTI statistics can be used under some dependency structures if we believe
that the dependencies are not too strong, and the truncated TMTI∞ variants are
generally more robust against dependency than TMTI∞.

Figure 5: Calibration curves for the CDFs of different TMTI statistics under
an i.i.d. assumption versus different dependency structures. CCT = Cauchy
Combination Test, HMP = Harmonic Mean $p$-value. Figure 6: Calibration curves
for the CDFs of different TMTI statistics under an i.i.d. assumption versus
different dependency structures, zoomed to only show the region
$(0,0.1)\times(0,0.1)$. CCT = Cauchy Combination Test, HMP = Harmonic Mean
$p$-value.

###  B.2 An example of applying the TMTI to non-independent data

In this section, we give an example of how the TMTI can be applied to non-
independent data in a particular setting. We consider a variation of the
simulation study conducted in Section 4, where the marginal $p$-values now
come from dependent $T$-scores instead of independent $Z$-scores. Furthermore,
we investigate the power of this procedure under different alternatives by
means of simulation. Throughout this section, we consider a significance level
of $\alpha=0.05$.

We consider the following setup: let $\bm{X}$ be an $n\times g$-dimensional
binary matrix of full rank satisfying that every row-sum of $\bm{X}$ equals
one and every column-sum of $\bm{X}$ equals $k$, for some $k\in\mathbb{N}$.
Fix a vector $\mu\in\mathbb{R}^{g}$ and let $\epsilon\sim N_{n}(0,I_{n})$ be a
random variable, where $I_{n}$ is the $n\times n$ identity matrix . We then
define

$\bm{W}\coloneqq\bm{X}\mu+\epsilon.$

The interpretation of this experiment is that we have recorded $k$
observations of some random variable, $W$, in $m$ different groups to obtain a
total of $N=mk$ samples. By altering the number of non-zero elements in $\mu$
we control the number of groups that affect the outcome, $\bm{W}$. As the
power of any test in this scenario is directly associated with the magnitude
of the coefficients, $\mu$, we will only consider constant values of $\mu$.
That is, the elements in $\mu$ which we allow to be non-zero, will all be
equal. We then consider the global hypothesis
$H_{0}:\bigcap_{i=1}^{m}(\mu_{i}=0),$ which is the hypothesis that the outcome
$W$ is not affected by any of the $m$ groups. The goal is now to estimate the
power of the TMTI, i.e., the quantity ${\mathbb{P}}_{H_{A}}(\text{reject
}H_{0}),$ under different alternative hypotheses $H_{A}$, i.e., different
$\mu$ vectors. For every alternative hypothesis considered here, we will
compute the $p$-value under four tests from the TMTI family; TMTI∞ and its
truncated and rank truncated versions (with $\tau=0.05$ and $K=5$), and TMTI1.
For TMTI1, we do not consider any truncated variants, as we expect these to be
roughly equal (per Figure 1).

For comparison, we consider a number of combination-based test procedures. For
this, we will compute the marginal $T$-tests of the hypotheses
$H_{i}:\mu_{i}=0$ for $i=1,\dots,m$ under the joint model. Then, we will apply
the following procedures; the Cauchy Combination Test (Liu and Xie, 2020); the
Harmonic Mean $p$-value (Wilson, 2019); the Compound Bonferroni Arithmetic
Mean (Vovk and Wang, 2020); a Bonferroni correction; and the Rank Truncated
Product Method. Additionally, we include the $F$-test, as this would often be
the natural choice of test in this particular setup. However, the $F$-test
offers less flexibility compared to a combination test, e.g., when used in a
Closed Testing Procedure (see Section 5), as this would require fitting
$2^{m}$ linear models.

In order to calculate $p$-values for the TMTI statistics, we estimate the
$\gamma$ functions under $H_{0}$ by employing a bootstrapping scheme: For
$i=1,\dots,10^{5}$ we first define
$\tilde{\bm{W}}\coloneqq\bm{W}-(X^{T}X)^{-1}X^{T}\bm{W}X^{T}$. That is,
$\tilde{\bm{W}}$ is $\bm{W}$ with the group means subtracted. This ensures
that each group of $\tilde{\bm{W}}$ has mean zero. We then construct
$\bm{W}_{i}$ by resampling $\bm{W}$ uniformly with replacement in order to
introduce variation. We then compute the relevant marginal $T$-test
statistics, compute $p$-values and finally output the TMTI statistic. We then
use the empirical CDFs of the bootstrapped TMTI statistics. In Figure 7 we
show the simulated sizes of all included tests. From this we conclude that all
tests have approximately the correct size or lower, except for the Fisher
Combination Test and the Truncated Product Method (both of which assume
independence of the $p$-values), and these are therefore left out from further
simulations. One can apply the same bootstrapping scheme as described above to
obtain valid $p$-values for both of these tests. However, we do refrain from
doing that here. We note also that the Rank Truncated Product Method has a
slightly increased Type I error, but it is sufficiently little that this may
be attributed to chance. The increased variance in the estimates of the Type I
errors for the TMTI statistics is due to the critical values of these being
estimated by bootstrapping.

Figure 7: Estimated sizes of the included tests in B.2, i.e., the probability
of rejecting the global null hypothesis when it is true. The Truncated Product
Methodand Rank Truncated Product Methodare denoted as the truncated FCT and
rank truncated FCT, respectively.

Figure 8 contains the results of a simulation with
$N_{\operatorname{false}}\in\\{10^{0},\dots,10^{3}\\}$. Generally, the results
are similar to those displayed in Figure 2: TMTI∞ and tTMTI∞ both perform well
in all settings, although not as well as some methods when
$N_{\operatorname{false}}=1$, but still better than the $F$-test. When
$N_{\operatorname{false}}>1$, we generally find that TMTI∞ and tTMTI∞ perform
either as good or better than the best of the competing methods. TMTI1 and
rtTMTI∞ have similar performance: when there are less than ten false
hypotheses, these perform on par or better than with the best of the competing
methods, but when there are multiple false hypotheses, they slightly
outperform the Cauchy Combination Test and Harmonic Mean $p$-value, although
they are both outperformed by the $F$-test, TMTI∞ and tTMTI∞.

Figure 8: Power curves for different TMTI tests and competing methods.
Generally, TMTI∞ and tTMTI∞ work well in all settings, while TMTI1 and rtTMTI∞
performs best when signals are sparse.

As in Section 4, the results of these simulations indicate that TMTI∞ and
tTMTI∞ offer an alternative to current tests that is powerful against a wide
range of alternative hypotheses. This is useful if one has no a priori
knowledge of the sparsity and strength of signals, as well as when employing
the tests in a Closed Testing Procedure. If one has a priori knowledge, that
the signals are sparse and strong, one should rather employ a test such as
TMTI1, rtTMTI∞, Harmonic Mean $p$-value, Cauchy Combination Test or a
Bonferroni test.

######  Remark 7.

The above example generalizes to a situation in which we have two sets of
covariates, $\bm{X}$ and $\tilde{\bm{X}}$, such that
$\bm{W}\coloneqq\bm{X}\mu+\tilde{\bm{X}}\tilde{\mu}+\epsilon,$ where
$\tilde{\bm{X}}$ are covariates we would simply like to adjust for when
computing our test, but not variables that are of interest. In this setting,
we could also compute the $p$-values corresponding to the tests of
$H_{i}:\tilde{\mu}_{i}=0$, although we would not care whether they are true or
not. Thus, we would select a subset of the $p$-values, say $\mathcal{J}$,
corresponding to those related to $\mu$, and compute our test only for those.

###  B.3 The effects of mixed $\mu$ values

In this section, we repeat the experiment performed in Section 4, with the
change that the $\mu$ values are allowed to differ between the false marginal
hypotheses. Thus, for $N_{\operatorname{false}}$ false hypotheses, we generate
$p$-values by sampling $X_{\operatorname{false},i}\sim N(\mu/i,1)$ (i.e., the
signal with the largest effect has mean $\mu$ and the signal with the weakest
effect has mean $\mu/N_{\operatorname{false}}$), where $\mu$ is chosen
equidistantly between the values that satisfy that a Bonferroni test has
either 5% or 99% power to reject the global null hypothesis in a setting with
no conservatism. The results are displayed in Figure 9. The comments to this
figure are the same as the comments to Figure 2.

Figure 9: Power curves for different TMTI tests and competing methods, in a
scenario where the signal strength is allowed to differ between each false
marginal hypothesis. The values of $\mu_{\operatorname{false}}$ are chosen
equidistantly between the two values, which satisfy that a Bonferroni test has
either 5% or 99% power to reject the global null hypothesis in a setting with
no conservatism.

##  Appendix C Table of adjusted $p$-values for all tests employed in Section
8

State (change) |  $p$-value |  Bonferroni |  TMTI∞ |  rtTMTI∞/ TMTI∞ |  FCT |  rTPM/ FCT
---|---|---|---|---|---|---
GA (-0.323) |  0.85628 |  1.00000 |  0.87219 |  0.93682 |  0.85753 |  0.93775
AR (-0.777) |  0.60282 |  1.00000 |  0.87219 |  0.93682 |  0.85753 |  0.93775
AL (-1.568) |  0.44008 |  1.00000 |  0.85873 |  0.93682 |  0.81333 |  0.93775
NJ (1.565) |  0.41998 |  1.00000 |  0.85873 |  0.93682 |  0.80157 |  0.93775
NE (1.334) |  0.38640 |  1.00000 |  0.85873 |  0.93682 |  0.78021 |  0.93775
ND (1.526) |  0.36890 |  1.00000 |  0.85873 |  0.93682 |  0.76813 |  0.93775
DE (1.374) |  0.31162 |  1.00000 |  0.85873 |  0.92675 |  0.72551 |  0.92768
MI (2.215) |  0.23522 |  1.00000 |  0.80175 |  0.88412 |  0.66845 |  0.88500
LA (2.637) |  0.20964 |  1.00000 |  0.78923 |  0.88412 |  0.64602 |  0.88500
IN (2.149) |  0.19388 |  1.00000 |  0.78923 |  0.88412 |  0.63076 |  0.88500
WI (2.801) |  0.15872 |  1.00000 |  0.78923 |  0.85060 |  0.59172 |  0.85144
VA (2.858) |  0.14374 |  1.00000 |  0.77357 |  0.84467 |  0.57388 |  0.84550
WV (2.331) |  0.10026 |  1.00000 |  0.68933‡ |  0.74677 |  0.51177‡ |  0.74750
MD (3.339) |  0.08226 |  1.00000 |  0.68933‡ |  0.69934 |  0.48059‡ |  0.71026
CA (3.777) |  0.07912 |  1.00000 |  0.68454‡ |  0.70957 |  0.47464‡ |  0.71026
OH (3.466) |  0.06590 |  1.00000 |  0.62312‡ |  0.64033 |  0.44713‡ |  0.64096
NY (4.893) |  0.05802 |  1.00000 |  0.58342‡ |  0.59203 |  0.42838‡ |  0.59262
PA (4.303) |  0.05572 |  1.00000 |  0.58342‡ |  0.57683 |  0.42250‡ |  0.57739
FL (3.784) |  0.05490 |  1.00000 |  0.58342‡ |  0.57129 |  0.42036‡ |  0.57185
WY (2.226) |  0.04678* |  1.00000 |  0.58342‡ |  0.51259 |  0.39755‡ |  0.51308
NM (2.334) |  0.04650* |  1.00000 |  0.58342‡ |  0.51043 |  0.39671‡ |  0.51092
CT (3.204) |  0.04104* |  1.00000 |  0.55925‡ |  0.46666 |  0.37939‡ |  0.46711
OK (4.181) |  0.02036* |  0.69224 |  0.42037‡ |  0.26549 |  0.29050‡ |  0.26573
KY (4.326) |  0.00964* |  0.32776 |  0.28899† |  0.13524‡ |  0.21234† |  0.13535‡
AZ (4.993) |  0.00904* |  0.30736 |  0.27561† |  0.12735‡ |  0.20643† |  0.12747‡
ID (2.956) |  0.00748* |  0.25432 |  0.23899† |  0.10651‡ |  0.18974† |  0.10659‡
TX (5.645) |  0.00404* |  0.13736 |  0.17114† |  0.05892† |  0.14480† |  0.05897†
CO (4.326) |  0.00282* |  0.09588 |  0.12797† |  0.04148* |  0.12286† |  0.04150*
IA (4.811) |  0.00200* |  0.06800 |  0.11058† |  0.02958* |  0.10453† |  0.02961*
NH (4.422) |  0.00180* |  0.06120 |  0.10121† |  0.02666* |  0.09939† |  0.02667*
NC (7.265) |  0.00002* |  0.00068* |  0.00346* |  0.00346* |  0.00843* |  0.00064*
HI (5.550) |  0.00002* |  0.00068* |  0.00346* |  0.00346* |  0.00843* |  0.00064*
MN (6.421) |  0.00002* |  0.00068* |  0.00346* |  0.00346* |  0.00843* |  0.00064*
RI (5.094) |  0.00001* |  0.00034* |  0.00198* |  0.00198* |  0.00551* |  0.00044*
Table 2: State-wise changes in mathematics achievements from 1990 to 1992 and
the $p$-values for the corresponding $T$-tests. Values that are significant at
$\alpha=0.05$ are marked with an asterisk. Hypotheses that can be rejected
while controlling the $k$-FWER at $k=2$ are marked with a $\dagger$.