#  A Bootstrap Hypothesis Test for High-Dimensional Mean Vectors

Alexander Giessing Department of Statistics, University of Washington,
Seattle, WA. E-mail<EMAIL_ADDRESS>Jianqing Fan Department of ORFE,
Princeton University, Princeton, NJ. E-mail<EMAIL_ADDRESS>
###### Abstract

This paper is concerned with testing global null hypotheses about population
mean vectors of high-dimensional data. Current tests require either strong
mixing (independence) conditions on the individual components of the high-
dimensional data or high-order moment conditions. In this paper, we propose a
novel class of bootstrap hypothesis tests based on $\ell_{p}$-statistics with
$p\in[1,\infty]$ which requires neither of these assumptions. We study
asymptotic size, unbiasedness, consistency, and Bahadur slope of these tests.
Capitalizing on these theoretical insights, we develop a modified bootstrap
test with improved power properties and a self-normalized bootstrap test for
elliptically distributed data. We then propose two novel bias correction
procedures to improve the accuracy of the bootstrap test in finite samples,
which leverage measure concentration and hypercontractivity properties of
$\ell_{p}$-norms in high dimensions. Numerical experiments support our
theoretical results in finite samples.

Keywords: Bootstrap Test; High-Dimensional Data; Elliptically Distributed
Data; Gaussian Approximation; Variances of $\ell_{p}$-Norms of Gaussian Random
Vectors; Spherical Bootstrap.

##  1 Introduction

###  1.1 Testing high-dimensional global null hypotheses

Let $X_{1},\ldots,X_{n}\in\mathbb{R}^{d}$ be i.i.d. random vectors with
unknown mean $\mu_{0}\in\mathbb{R}^{d}$ and unknown positive semi-definite
covariance matrix $\Sigma\neq\mathbf{0}\in\mathbb{R}^{d\times d}$. In this
paper, we consider testing high-dimensional linear restrictions

$\displaystyle
H_{0}:\>\>R\mu_{0}=r\quad{}\quad{}\mathrm{vs.}\quad{}\quad{}H_{1}:\>\>R\mu_{0}\neq
r,$ (1)

for some deterministic matrix $R\in\mathbb{R}^{t\times d}$ and vector
$r\in\mathbb{R}^{t}$ when dimension $d$ and number of restrictions $t$ are
potentially much larger than the sample size $n$, i.e. $d,t\gg n$.

Global null hypothesis testing problems such as (1) arise frequently in
scientific applications. For instance, many biological processes involve
regulation of multiple genes with small effect sizes on individual variants
(Manolio et al., 2009). In such cases, analyzing genes grouped according to
their biological functions or chromosomal location can increase the power of
statistical tests, reduce the complexity of the analysis, and lead to a better
understanding of the underlying genetic mechanisms (Huang et al., 2022). The
problem of assessing whether a group of many genes is differentially expressed
from another group of genes ultimately reduces to testing a high-dimensional
population mean vector as in (1). Another example are cross-sectional and
longitudinal studies in financial econometrics (Fan et al., 2015; Gagliardini
et al., 2016). In these studies the parameter of interest is usually not a
simple population mean but a high-dimensional regression vector. The time
series character of such data sets adds additional complications. However,
often, the testing problems can still be recast as in (1).

There are two common approaches to testing (1) in high dimensions: one based
on $\ell_{2}$-norms, i.e. sum-of-squares of the sample mean (e.g. Bai and
Saranadasa, 1996; Chen et al., 2010; Wang et al., 2015; Xu et al., 2017; He et
al., 2021; Huang et al., 2022), and the other one based on
$\ell_{\infty}$-norms, i.e. component-wise maxima of the sample mean (e.g.
Chernozhukov et al., 2013; Cai et al., 2014; Chen, 2018; Xue and Yao, 2020;
Lopes et al., 2020). Typically, tests based on the $\ell_{2}$-norm are valid
only under mixing (independence) conditions on the individual components of
the high-dimensional random vectors, whereas tests based on the
$\ell_{\infty}$-norm often require higher-order moment conditions.

In this paper, we develop a new class of bootstrap hypothesis tests based on
$\ell_{p}$-norms, $p\in[1,\infty]$, which is valid without mixing conditions
and under mild lower-order moment conditions. Along with a thorough analysis
of the theoretical properties of the test (Sections 2.1-2.4) we address the
following questions:

  * •

It is statistical folklore that $\ell_{2}$-type statistics have high power
against dense alternatives whose signals are spread out over a large number of
coordinates while $\ell_{\infty}$-type statistics have high power against
sparse alternatives with only a few strong signals (Cai et al., 2014; Fan et
al., 2015; Wang et al., 2015; He et al., 2021). _Does this apply to the
bootstrap test as well and what is the use of other $\ell_{p}$-norms, $p\geq
1$? Can we combine $\ell_{p}$-norms to obtain a test that has high power
against, both, sparse and dense alternatives?_ (Section 3.1)

  * •

The validity of bootstrapping procedures (in, both, classical as well as
modern high-dimensional settings) hinges on moment conditions (Giné and Zinn,
1990; Giessing, 2023b). _Can we improve the theoretical and practical
performance of the bootstrap test by leveraging distributional assumptions
commonly used in high-dimensional statistics?_ (Section 3.2)

  * •

The proposed bootstrap test depends on the data via an estimate of the high-
dimensional covariance matrix. Estimating high-dimensional covariance matrices
can be challenging (Cai et al., 2010; Cai and Liu, 2011; Avella-Medina et al.,
2018). _How does the estimated covariance matrix affect the validity of the
bootstrap test?_ (Section 4.1)

  * •

The proposed test statistic is non-pivotal; in particular, the sampling
distribution of the test statistic depends on the unknown population
covariance matrix. In practice, when bootstrapping non-pivotal test statistics
the actual level of the test often differs substantially from the nominal one.
Several (computationally expensive) bias correction schemes have been proposed
in classical settings (Davison et al., 1986; Beran, 1987; Hall and Martin,
1988; Shi, 1992). _Can we draw on insights from high-dimensional probability
theory to develop new (computationally more efficient) correction schemes?_
(Sections 4.2-4.3)

The reader interested in only the answers to these questions and related open
problems, may directly jump to the discussion in Section 6.

###  1.2 A bootstrap hypothesis test based on $\ell_{p}$-norms

We propose to test hypothesis (1) on the basis of the $\ell_{p}$-statistic

$\displaystyle
T_{n,p}:=\|RS_{n}-\sqrt{n}r\|_{p},\quad{}\quad{}S_{n}=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}X_{i},\quad{}\quad{}p\geq
1,$

and, given a nominal level $\alpha\in(0,1)$, reject the null hypothesis if and
only if

$\displaystyle T_{n,p}\geq c^{*}_{p}(1-\alpha;\widehat{\Omega}_{n}),$

where
$c^{*}_{p}(\alpha;\widehat{\Omega}_{n})=\inf\left\\{t\in\mathbb{R}:\mathrm{P}(T_{n,p}^{*}\leq
t\mid X_{1},\ldots,X_{n})\geq\alpha\right\\}$ is the conditional
$\alpha$-quantile of the Gaussian proxy statistic

$\displaystyle T_{n,p}^{*}:=\|Z_{n}\|_{p},\quad{}\quad{}Z_{n}\mid
X_{1},\ldots,X_{n}\sim N(0,\widehat{\Omega}_{n}),$

and $\widehat{\Omega}_{n}$ is a positive semi-definite estimate of
$\Omega=R\Sigma R^{\prime}$. We call above testing procedure the _bootstrap
test based on $T_{n,p}$ at level $\alpha$_ and define

$\displaystyle\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n}):=\mathbf{1}\left\\{T_{n,p}\geq
c^{*}_{p}(1-\alpha;\widehat{\Omega}_{n})\right\\}.$

The bootstrap test based on $T_{n,p}$ rejects $H_{0}$ at level $\alpha$ if and
only if $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})=1$.

To understand the rationale for the test, consider the low-dimensional case
first: If dimensions $d,t$ are fixed and the data are i.i.d. with finite
second moments, the CLT and the continuous mapping theorem imply that
$T_{n,p}\overset{d}{\rightarrow}\|Z\|_{p}$, where $Z\sim
N\left(0,\Omega\right)$. Hence, in this case, the Gaussian proxy statistic
$T_{n,p}^{*}$ is just the parametric bootstrap estimate of the limiting random
variable $\|Z\|_{p}$. Of course, if $d,t\geq\sqrt{n}$, then the CLT does not
apply and the limiting random variable $Z$ needs not to exist. The reason why
the bootstrap is nonetheless valid in high dimensions is that a CLT is not
needed. Indeed, validity of the bootstrap test follows already from
convergence of the laws of $T_{n,p}$ and $T_{n,p}^{*}$ with respect to the
Kolmogorov distance (see Appendix A). Since two sequences of laws are close in
Kolmogorov distance if they have the same cluster points, this does not imply
(or necessitate) existence of a limit law.

In the remainder of the paper we will study the asymptotic properties of this
test, develop modifications for special scenarios, and provide answers to the
questions raised in Section 1.1.

###  1.3 Contributions and outline of the paper

In detail, our _contributions to statistical theory_ are as follows: First, we
show that the bootstrap tests $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$
have asymptotic correct size for all $p\in[1,\infty]$. By inverting the test
statistics we then obtain Scheffé-type simultaneous confidence regions for
multiple testing problems (Section 2.2). Second, we establish that the
bootstrap tests are asymptotically unbiased and consistent against certain
alternatives (Section 2.3). In particular, we show that for each exponent
$p\in[1,\infty]$ there exist at least three different classes of alternatives
characterized by their signal strength relative to noise level and expected
size of the Gaussian proxy statistic $T_{n,p}^{*}$. Third, to assess the power
of the bootstrap tests (against sparse and dense alternatives and for
different exponents $p\geq 1$) we adapt the classical concept of the Bahadur
slope to the high-dimensional setting. We then show that
$\varphi_{\alpha}(T_{n,2},\widehat{\Omega}_{n})$ is most powerful among all
bootstrap tests when testing against dense alternatives whose signals are
spread out over a large number of coordinates whereas
$\varphi_{\alpha}(T_{n,\log t},\widehat{\Omega}_{n})$ is most powerful among
all bootstrap tests when testing against sparse alternatives with only a few
strong signals (Section 2.4). Fourth, we discuss three modifications of the
basic bootstrap tests: We construct a bootstrap test with power against, both,
sparse and dense alternatives (Section 3.1), propose a self-normalized test
for elliptically (heavy-tailed) data (Section 3.2), and briefly discuss
bootstrap tests for population parameters other than the mean (Section 3.3).

The key _methodological innovations_ of this paper are two new bias correction
schemes to improve the accuracy of the bootstrap test. Typically, when
implementing bootstrap methods via repeated sampling from the empirical
distribution or, as in our case, repeated sampling from the Gaussian proxy
statistic $T_{n,p}^{*}$ the actual level of the test differs from the nominal
level $\alpha$ even if the number of samples $B$ is large. We develop a three-
fold bias decomposition (Section 4.1) and propose methods to mitigate two of
the biases: First, we show how to incorporate structural information of the
high-dimensional covariance matrix of the data into the bootstrap test
(Section 4.2). Second, we propose a novel spherical multiplier bootstrap test
which exploits hypercontractivity properties of $\ell_{p}$-norms of high-
dimensional random vectors (Section 4.3). We provide supporting results from
Monte Carlo experiments in Section 5.

The main _probabilistic contribution_ of this paper is that the bootstrap test
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ is indeed valid _for all
$p\geq 1$_ even if dimensions $d,t$ far exceed the sample size $n$. We develop
the necessary technical tools in Appendix A. These include, among other
things, Gaussian approximation and anti-concentration inequalities for high-
dimensional $\ell_{p}$-statistics (Appendix A.1) and lower bounds on the
variances of $\ell_{p}$-norms of (an-isotropic) Gaussian random vectors
(Appendix A.2). For the reader’s convenience, we also provide a brief
exposition of the relevant results from the two companion papers Giessing
(2023a, b) (Appendix A.3).

###  1.4 Notation

We denote the $\ell_{p}$-norm, $p\in[1,\infty)$, and the $\ell_{\infty}$-norm
of a vector $x\in\mathbb{R}^{d}$ by
$\|x\|_{p}=(\sum_{k=1}^{d}|x_{k}|^{p})^{1/p}$ and $\|x\|_{\infty}=\max_{1\leq
k\leq d}|x_{k}|$, respectively. For a symmetric matrix $A$, we denote its
operator, Frobenius, and $q,p$-matrix norm by $\|A\|_{op}$, $\|A\|_{F}$, and
$\|A\|_{q\rightarrow p}:=\sup_{\|u\|_{q}=1}\|Au\|_{p}$, $1\leq p,q\leq\infty$.
For two deterministic sequences $(a_{n})_{n\geq 1}$ and $(b_{n})_{n\geq 1}$,
we write $a_{n}\lesssim b_{n}$ if $a_{n}=o(b_{n})$. Also, $a_{n}\asymp b_{n}$
if there exist absolute constants $C_{1},C_{2}>0$ such that $C_{1}b_{n}\leq
a_{n}\leq C_{2}b_{n}$ for all $n\geq 1$. We denote the set of mean vectors
which are equivalent under the null hypothesis by
$\mathcal{H}_{0}=\\{\mu\in\mathbb{R}^{d}:R\mu=r\\}$ and the set of those
belonging to the alternative hypothesis by
$\mathcal{H}_{1}=\mathcal{H}_{0}^{c}$.

##  2 Properties of the bootstrap hypothesis test

###  2.1 High-level assumptions and simple sufficient conditions

To carry out the theoretical analysis of the bootstrap tests
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ we need a set of minimal
assumption which we present in this section (Assumptions 1–3). We include
three examples (Lemmas 1–3) which illustrate these minimal assumptions for
concrete data generating processes.

Throughout, $X,X_{1},\ldots,X_{n}\in\mathbb{R}^{d}$ are i.i.d. random vectors
with mean $\mu_{n}\in\mathbb{R}^{d}$ and positive semi-definite covariance
matrix $\Sigma\neq\mathbf{0}\in\mathbb{R}^{d\times d}$, $Z\sim N(0,\Omega)$
with $\Omega\equiv R\Sigma R^{\prime}\neq\mathbf{0}\in\mathbb{R}^{t\times t}$
and $R\in\mathbb{R}^{t\times d}$. We allow the dimension $d$ and the number of
restrictions $t$ to vary as a function of the sample size $n$; in particular,
it is possible that $d,t\rightarrow\infty$ as $n\rightarrow\infty$.

The high-level assumptions that we impose are conditions on the moments of the
data. These moment conditions originate from the technical results concerning
the approximation of the sampling distribution of $T_{n,p}$ (see Appendix A)
and implicitly restrict the growth rates of dimensions $d,t$ relative to the
sample size $n$.

######  Assumption 1 (Control of third moments).

The $\ell_{p}$\- and Gaussian proxy statistic satisfy

$\displaystyle\left(\mathrm{E}[\|R(X-\mu_{n})\|_{p}^{3}]\right)^{1/3}\vee\left(\mathrm{E}[\|Z\|_{p}^{3}]\right)^{1/3}\lesssim
n^{1/6}\mathrm{Var}(\|Z\|_{p})^{1/2}.$

######  Assumption 2 (Control of ratio of moments).

There exists $s>3$ such that

$\displaystyle\frac{\left(\mathrm{E}[\|R(X-\mu_{n})\|_{p}^{s}]\right)^{1/s}}{\left(\mathrm{E}[\|R(X-\mu_{n})\|_{p}^{3}]\right)^{1/3}}\lesssim
n^{1/3-1/s}.$

######  Assumption 3 (Consistent estimate of covariance matrix).

There exists a nonparametric statistic $\widehat{\Omega}_{n}$ based on the
$X_{i}$’s only such that

$\displaystyle\|\widehat{\Omega}_{n}-\Omega\|_{q\rightarrow
p}=o_{p}\big{(}\mathrm{Var}(\|Z\|_{p})\big{)},\quad{}\quad{}1/p+1/q=1,$

where $\|M\|_{q\rightarrow p}:=\sup_{\|u\|_{q}=1}\|Mu\|_{p}$ for matrices $M$.

######  Remark 1 (On Assumption 3).

We require $\widehat{\Omega}_{n}$ to be a nonparametric estimator to guarantee
that the Gaussian proxy statistic $T_{n,p}^{*}$ is ancillary. Ancilliarity of
$T_{n,p}^{*}$ is necessary for computing critical values for composite
hypotheses and confidence regions. If we were interested in testing only one
null and alternative hypothesis at a time, we could do without it, e.g. we
could use
$\widehat{\Omega}_{n}=n^{-1}\sum_{i=1}^{n}R(X_{i}-\mu)(X_{i}-\mu)^{\prime}R^{\prime}$,
where $\mu\in\mathbb{R}^{d}$ depends on the (null) hypothesis under
consideration.

These assumptions are mild and can easily be verified to hold in a range of
scenarios. In the following we give three examples with simple sufficient
conditions. First, we consider data with sub-Gaussian tails which is a common
assumption in high dimensional statistics. In the second example we introduce
data with log-concave distribution which is a useful assumption for proving
several subtle properties of the bootstrap test in Sections 2.3 and 2.4. The
third example is about heavy-tailed data; it is important in applications and
becomes relevant in Section 3.2 where we introduce a modified test statistic
for elliptically distributed data which allows us to further relax Assumptions
1–3. To keep the exposition simple, we only consider exponents
$p\in\\{2,\infty\\}$. However, in principle, all three examples can be
extended to all $p\in[1,\infty]$. In the following,
$\widehat{\Omega}_{n}=n^{-1}\sum_{i=1}^{n}R(X_{i}-\bar{X}_{n})(X_{i}-\bar{X}_{n})^{\prime}R^{\prime}$
denotes the (nonparametric) sample covariance matrix of the transformed data
$RX_{1},\ldots,RX_{n}\in\mathbb{R}^{t}$.

######  Lemma 1 (Sub-Gaussian design).

Let $X,X_{1},\ldots X_{n}\in\mathbb{R}^{d}$ be a simple random sample of sub-
Gaussian random vectors with mean $\mu_{n}\in\mathbb{R}^{d}$ and positive
semi-definite covariance matrix $\Sigma\in\mathrm{R}^{d\times d}$ such that
$\|(X-\mu_{n})^{\prime}u\|_{\psi_{2}}\lesssim\|\Sigma^{1/2}u\|_{2}$ for all
$u\in\mathbb{R}^{d}$. Then, Assumptions 1–3 are satisfied

  * (i)

for $p=2$ if

$\displaystyle\frac{r(\Omega)}{\sqrt{r(\Omega^{2})}}=o(n^{1/6})\quad{}\quad{}\mathrm{and}\quad{}\quad{}\frac{r(\Omega)}{r(\Omega^{2})}\left(\sqrt{\frac{r(\Omega)}{n}}\vee\frac{r(\Omega)}{n}\right)=o(1),$

where $r(M):=\mathrm{tr}(M)/\|M\|_{op}$ is the effective rank of
$M\in\\{\Omega,\Omega^{2}\\}$;

  * (ii)

for $p=\infty$ if

$\displaystyle\frac{\omega_{(t)}}{\omega_{(1)}}\frac{\sqrt{\log
t}}{n^{1/6}}\vee\frac{\omega_{(t)}^{2}}{\omega_{(1)}^{2}}\frac{\log
t}{n^{1/6}}=o(1)\quad{}\quad{}\mathrm{and}\quad{}\quad{}\frac{\omega_{(t)}^{2}}{\omega_{(1)}^{2}}\sqrt{\frac{\log
t}{n}}\vee\frac{\omega_{(t)}^{4}}{\omega_{(1)}^{4}}\sqrt{\frac{(\log
t)^{3}}{n}}=o(1),$

where $\omega_{(1)}^{2}\leq\ldots\leq\omega_{(t)}^{2}$ are the ordered
diagonal elements of $\Omega$.

######  Remark 2 (On the exponent $p$ and covariance matrix $\Omega$).

The conditions in Lemma 1 (i) on $\Omega$ are easily satisfied for matrices
with variance decay or bounded effective rank such that
$\mathrm{r}(\Omega),\mathrm{r}(\Omega^{2})<\infty$ independent of or slowly
increasing in the dimensions $d,t$. We refer to Lopes et al. (2020) for more
explicit examples. In Lemma 1 (ii) the ratio
$\omega_{(t)}^{2}/\omega_{(1)}^{2}$ can be upper bounded by the conditioning
number of $\Omega$, i.e. ratio of its largest to smallest eigenvalue.

######  Lemma 2 (Log-concave design).

Let $X,X_{1},\ldots X_{n}\in\mathbb{R}^{d}$ be a simple random sample of
random vectors with log-concave density, i.e. the density $f$ satisfies
$f=e^{-\varphi}$ with $\varphi$ convex. Then, Assumptions 1–3 are satisfied

  * (i)

for $p=2$ if

$\displaystyle\frac{r(\Omega)}{\sqrt{r(\Omega^{2})}}=o(n^{1/6})\quad{}\quad{}\mathrm{and}\quad{}\quad{}\frac{r(\Omega)}{r(\Omega^{2})}\left(\left(\frac{r(\Omega)}{n}\right)^{1/4}\vee\frac{r(\Omega)}{n}\right)=o(1),$

where $r(M):=\mathrm{tr}(M)/\|M\|_{op}$ is the effective rank of
$M\in\\{\Omega,\Omega^{2}\\}$;

  * (ii)

for $p=\infty$ if

$\displaystyle\frac{\omega_{(t)}}{\omega_{(1)}}\frac{\log(tK_{\varphi})}{n^{1/6}}\vee\frac{\omega_{(t)}^{2}}{\omega_{(1)}^{2}}\frac{\sqrt{\log(tK_{\varphi})^{3}}}{n^{1/6}}=o(1)\quad{}\mathrm{and}\quad{}\frac{\omega_{(t)}^{2}}{\omega_{(1)}^{2}}\left(\frac{(\log
t)^{5}}{n}\right)^{1/4}\vee\frac{\omega_{(t)}^{4}}{\omega_{(1)}^{4}}\sqrt{\frac{(\log
t)^{3}}{n}}=o(1),$

where $\omega_{(1)}^{2}\leq\ldots\leq\omega_{(t)}^{2}$ are the ordered
diagonal elements of $\Omega$ and $K_{\varphi}>1$ is a constant depending on
(the marginals of) the density $f=e^{-\varphi}$.

######  Remark 3 (On the convexity of $\varphi$).

If $\varphi$ is strictly convex in the sense that
$\varphi^{\prime\prime}\geq\lambda I_{d}$ for some $\lambda>0$, then the
$X_{i}$’s are sub-Gaussian and the bounds of Lemma 1 apply. For details (in
particular, how the sub-Gaussian tail bounds depend on $\lambda>0$) we refer
to Theorems 5.2 and 5.3 in Ledoux (2001).

######  Lemma 3 (Heavy-tailed design).

Let $X,X_{1},\ldots X_{n}\in\mathbb{R}^{d}$ be a simple random sample of
random vectors. Then, Assumptions 1–3 are satisfied

  * (i)

for $p=2$ if there exists $s>3$,

$\displaystyle
m_{2,3}\sqrt{\frac{r(\Omega)}{r(\Omega^{2})}}=o(n^{1/6}),\quad{}\frac{m_{2,s}}{m_{2,3}}=o(n^{1/3-1/s}),\quad{}\frac{r(\Omega)}{r(\Omega^{2})}\left(\sqrt{\frac{M_{n}^{2}(\log
n)}{n}}\vee\frac{M_{n}^{2}(\log n)}{n}\right)=o(1),$

where $r(M):=\mathrm{tr}(M)/\|M\|_{op}$ is the effective rank of
$M\in\\{\Omega,\Omega^{2}\\}$, $M_{n}^{2}:=\mathbb{E}[\max_{1\leq i\leq
n}\|X_{i}-\mu_{n}\|_{2}^{2}]/\|\Omega\|_{op}$, and
$m_{2,s}^{s}:=\mathbb{E}[\|X-\mu_{n}\|_{2}^{s}]/\|\Omega\|_{op}^{s/2}$;

  * (ii)

for $p=\infty$ if there exists $s>3$,

$\displaystyle
m_{\infty,3}\left(\frac{\omega_{(t)}}{\omega_{(1)}}\vee\frac{\omega_{(t)}^{2}}{\omega_{(1)}^{2}}\sqrt{\frac{\log
t}{n}}\right)=o(n^{1/6}),\quad{}\frac{m_{\infty,s}}{m_{\infty,3}}=o(n^{1/3-1/s}),$
$\displaystyle\frac{\omega_{(t)}^{2}}{\omega_{(1)}^{2}}\left(\frac{(\log
t)^{5}}{n}\right)^{1/4}\vee\frac{\omega_{(t)}^{4}}{\omega_{(1)}^{4}}\sqrt{\frac{(\log
t)^{3}}{n}}=o(1),$

where $\omega_{(1)}^{2}\leq\ldots\leq\omega_{(t)}^{2}$ are the ordered
diagonal elements of $\Omega$ and
$m_{\infty,s}^{s}:=\mathbb{E}[\|X-\mu_{n}\|_{\infty}^{s}]/\omega_{(t)}^{s}$.

######  Remark 4 (On the $(\log n)$-factor).

If the $\|X_{i}\|_{2}\leq M_{n}$ almost surely, we can remove the $(\log
n)$-factors in Lemma 3 (i). Using an alternative argument based on the proof
of Lemma 2, we can replace the third relation in Lemma 3 (i) by
$\frac{r(\Omega)}{r(\Omega^{2})}\Big{(}\big{(}\frac{r(\Omega)}{n}\big{)}^{1/4}\vee\frac{r(\Omega)}{n}\Big{)}=o(1)$.

###  2.2 Asymptotic size

As a first result we show that the bootstrap test
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ has asymptotic correct size,
i.e. asymptotically the bootstrap test controls the type 1 error. By inverting
the hypothesis tests we then obtain asymptotic $1-\alpha$ confidence sets and
Scheffé-type simultaneous confidence intervals for multiple comparison
problems.

######  Theorem 4 (Asymptotic size $\alpha$ test).

Suppose that Assumptions 1–3 hold. Then, for $1\leq p\leq\infty$,

$\displaystyle\lim_{n\rightarrow\infty}\sup_{\alpha\in(0,1)}\sup_{\mu\in\mathcal{H}_{0}}\Big{|}\mathrm{E}_{\mu}[\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})]-\alpha\Big{|}=0.$

Since
$T_{n,p}=\|RS_{n}-\sqrt{n}r\|_{p}=\sup_{\|u\|_{q}=1}u^{\prime}(RS_{n}-\sqrt{n}r)$
for $1/q+1/p=1$, we immediately obtain the following corollary:

######  Corollary 5.

Recall the conditions of Theorem 4. Let $u\in\mathbb{R}^{t}$ be a (possibly)
random vector not necessarily independent of $X_{1},\ldots,X_{n}$. Then, for
$U_{n,p}:=u^{\prime}(RS_{n}-\sqrt{n}r)/\|u\|_{q}$ with $1/p+1/q=1$,

$\displaystyle\lim_{n\rightarrow\infty}\sup_{\mu\in\mathcal{H}_{0}}\mathrm{E}_{\mu}[\varphi_{\alpha}(U_{n,p},\widehat{\Omega}_{n})]\leq\alpha.$

Corollary 5 is of some interest because it implies that we can use the
bootstrap test $\varphi_{\alpha}(U_{n,p},\widehat{\Omega}_{n})$ to test
(simple) data dependent hypotheses while still controlling the type 1 error:

######  Example 1 (Post-selective test of subsets of a population mean
vector).

Let $S_{n,(1)}\leq\ldots\leq S_{n,(d)}$ be the absolute values of the entries
of $S_{n}\in\mathbb{R}^{d}$ arranged in ascending order. Let
$\widehat{J}=\\{i:S_{n,(i)}\geq S_{n,(B)}\\}\subset\\{1,\ldots,d\\}$ be the
random subset of the $B\geq 1$ largest entries in $S_{n}$ and consider the
data dependent hypotheses $H_{0}:\mu_{0,\widehat{J}}=0$ vs.
$H_{1}:\mu_{0,\widehat{J}}\neq 0$. Since
$\|S_{n,\widehat{J}}\|_{p}\leq\sup_{\|u\|_{q}=1}u^{\prime}S_{n}=\|S_{n}\|_{p}$
we can test this hypothesis using above Corollary with $R=I$ and $r=0$ (such
tests are relevant in the context of exploratory gene expression/ gene set/
pathway analyses, e.g. Mathur et al., 2018). Since this test has a high
threshold for rejecting the null hypothesis (i.e. the $(1-\alpha)$ quantile of
the Gaussian proxy statistic
$T_{n,p}^{*}=\|Z_{n}\|_{p}=\sup_{\|u\|_{q}=1}u^{\prime}Z_{n}$), it is
conservative and under-powered. It is not a panacea to all post-selective
inference problems.

A test with higher power is the one based on the $(1-\alpha)$ quantile of the
Gaussian proxy statistic
$\sup_{J\subset\\{1,\ldots,d\\},\>|J|=B}\|Z_{n,J}\|_{p}$. Intuitively, this
proxy statistic captures the maximum spurious signal of $B$ variables. The set
$\widehat{J}$ is considered significant only if its signal is stronger than
the maximum spurious signal of $B$ variables. Under properly adjusted
Assumptions 1–3, it is trivial to modify the proofs of Theorem 4 and Corollary
5 to verify that this test has size at most $\alpha$, too.

Another easy consequence of Theorem 4 is the following:

######  Corollary 6.

Recall the conditions of Theorem 4. Suppose that in addition there exists a
random matrix $\widehat{R}_{n}\in\mathbb{R}^{t\times d}$, not necessarily
independent of $X_{1},\ldots,X_{n}$, such that

$\displaystyle\|\widehat{R}_{n}-R\|_{q\rightarrow
p}\|S_{n}-\sqrt{n}\mu_{0}\|_{p}=o_{p}\left(\sqrt{\mathrm{Var}(\|Z\|_{p})}\right),\quad{}\quad{}1/p+1/q=1.$
(2)

Then, for
$\widetilde{T}_{n,p}:=\|\widehat{R}_{n}(S_{n}-\sqrt{n}\mu_{0})\|_{p}$ and all
$1\leq p\leq\infty$,

$\displaystyle\lim_{n\rightarrow\infty}\sup_{\alpha\in(0,1)}\sup_{\mu\in\mathcal{H}_{0}}\Big{|}\mathrm{E}_{\mu}[\varphi_{\alpha}(\widetilde{T}_{n,p},\widehat{\Omega}_{n})]-\alpha\Big{|}=0.$

Corollary 6 shows that (simple) preprocessing of the data does not affect the
size of the bootstrap test. A common preprocessing step is to studentize the
data. Studentization is particularly appropriate if the data set has been
obtained by aggregating data from various sources with different noise levels
or is known to be heteroscedastic. Moreover, in the classical setting with
fixed dimensions, (bootstrap) tests based on studentized statistics are known
to be more accurate (Hall, 1986). We have evidence from simulations that the
same holds true in high dimensions.

######  Example 2 (Tests based on studentized statistics).

Suppose that Assumptions 1–3 hold with $R^{2}=\mathrm{diag}(\Sigma)^{-1}$ and
that eq. (2) is satisfied with
$\widehat{R}_{n}^{2}=\mathrm{diag}(\widehat{\Sigma}_{n})^{-1}$, where
$\widehat{\Sigma}_{n}=(\hat{\sigma}_{kj})_{k,j=1}^{d}$ is the sample
covariance matrix based on the $X_{i}$’s. By Corollary 6 we can test
$H_{0}:\mu=\mu_{0}$ versus $H_{1}:\mu\neq\mu_{0}$ at level $\alpha$ via
$\varphi_{\alpha}(\widetilde{T}_{n,p},\widehat{\Omega}_{n})$, where
$\widetilde{T}_{n,p}:=\|\widetilde{S}_{n}-\tilde{\mu}_{0}\|_{p}$ is the
studentized $\ell_{p}$-statistic with
$\tilde{\mu}_{0}=(\tilde{\mu}_{0,k}/\hat{\sigma}_{kk})_{k=1}^{d}$,
$\widetilde{X}_{i}=(X_{ik}/\hat{\sigma}_{kk})_{k=1}^{d}$, and
$\widetilde{S}_{n}=n^{-1/2}\sum_{i=1}^{n}\widetilde{X}_{i}$. Notice that
$\|\widehat{R}_{n}-R\|_{q\rightarrow p}\asymp\max_{1\leq k\leq
d}|\hat{\sigma}_{kk}^{2}-\sigma_{kk}^{2}|$ for all $p\in\\{2\\}\cup[\log
d,\infty]$. Thus, for these exponents, eq. (2) is easily verified by combining
the results in Section A.2 with the commonly used moment conditions in high-
dimensional statistics.

By inverting the test $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ we
obtain asymptotic $1-\alpha$ bootstrap confidence regions and Scheffé-type
simultaneous bootstrap confidence intervals. The following theorem is
essentially a restatement of Theorem 4 and therefore does not need a proof.

######  Proposition 7 (Asymptotic confidence regions and simultaneous
confidence intervals).

Suppose that Assumptions 1–3 hold. Set
$\hat{\mu}_{n}=n^{-1/2}S_{n}=n^{-1}\sum_{i=1}^{n}X_{i}$. Then, for all
$\alpha\in(0,1)$ and $1\leq p,q\leq\infty$ such that $1/p+1/q=1$ the following
holds.

  * (i)

The $\ell_{p}$-norm ellipsoid

$\displaystyle\mathcal{E}_{n,p}(R,\widehat{\Omega}_{n}):=\left\\{\mu\in\mathbb{R}^{d}:\sqrt{n}\|R(\hat{\mu}_{n}-\mu)\|_{p}\leq
c^{*}_{p}(1-\alpha;\widehat{\Omega}_{n})\right\\}$

is an asymptotic $(1-\alpha)$-confidence region for $\mu_{0}$,
$\lim_{n\rightarrow\infty}\mathrm{P}\left(\mu_{0}\in\mathcal{E}_{n,p}(R,\widehat{\Omega}_{n})\right)=1-\alpha$.

  * (ii)

The intervals

$\displaystyle\mathcal{I}_{n,p}(h;R,\widehat{\Omega}_{n}):=\left[h^{\prime}R\hat{\mu}_{n}-\frac{\|h\|_{q}}{\sqrt{n}}c^{*}_{p}(1-\alpha;\widehat{\Omega}_{n}),\>h^{\prime}R\hat{\mu}_{n}+\frac{\|h\|_{q}}{\sqrt{n}}c^{*}_{p}(1-\alpha;R,\widehat{\Omega}_{n})\right],\quad{}h\in\mathbb{R}^{d},$

are asymptotic simultaneous $(1-\alpha)$-confidence intervals for all linear
combinations $h^{\prime}R\mu_{0}$, $h\in\mathbb{R}^{d}$, i.e.
$\lim_{n\rightarrow\infty}\mathrm{P}\left(h^{\prime}R\mu_{0}\in\mathcal{I}_{n,p}(h;R,\widehat{\Omega}_{n}),\>\forall
h\in\mathbb{R}^{d}\right)=1-\alpha$.

######  Remark 5 (On the matrix of linear restrictions $R$).

Under the conditions of Corollary 6 the deterministic matrix
$R\in\mathbb{R}^{t\times d}$ can be replaced by a random matrix
$\widehat{R}_{n}$. Thus, analogous to Corollary 6, we can construct confidence
regions nd simultaneous confidence intervals for the studentized mean vector.

Apart from asymptotic exact $1-\alpha$ confidence sets for the mean $\mu_{0}$
and linear combinations $h^{\prime}\mu_{0}$ Proposition 7 also yields
asymptotic level $1-\alpha$ confidence intervals for the norm
$\|\mu_{0}\|_{p}$:

######  Example 3 (Conservative confidence intervals for $\|\mu_{0}\|_{p}$).

Suppose that Assumptions 1–3 hold with $R=I_{d}\in\mathbb{R}^{d\times d}$. Let
$\widehat{\Sigma}_{n}=(\hat{\sigma}_{kj})_{k,j=1}^{d}$ be the sample
covariance matrix based on the $X_{i}$’s. By the triangle inequality we have
$\big{|}\|\hat{\mu}_{n}\|_{p}-\|\mu_{0}\|_{p}\big{|}\leq\|\hat{\mu}_{n}-\mu_{0}\|_{p}$.
Hence, by Proposition 7 (i)

$\displaystyle\mathcal{C}_{n,p}:=\left[\|\hat{\mu}_{n}\|_{p}-n^{-1/2}c^{*}_{p}(1-\alpha;\widehat{\Sigma}_{n}),\>\|\hat{\mu}_{n}\|_{p}+n^{-1/2}c^{*}_{p}(1-\alpha;\widehat{\Sigma}_{n})\right],\quad{}p\geq
1,$

is an asymptotic level $1-\alpha$ interval for $\|\mu_{0}\|_{p}$, i.e.
$\lim_{n\rightarrow\infty}\mathrm{P}\left(\|\mu_{0}\|_{p}\in\mathcal{C}_{n,p}\right)\geq
1-\alpha$.

If one is interested in inference about a single linear combination of a high-
dimensional mean vector, say $x^{\prime}\mu_{0}$, it is advisable to use
Proposition 7 (i) with $R=x^{\prime}$ (not Proposition 7 (ii) with $h=x$ and
$R=I_{d}$) in order to obtain intervals with short width:

######  Example 4 (Confidence interval for a single linear combination of a
high-dimensional mean vector).

Suppose that Assumptions 1–3 hold with $R=x^{\prime}$. Let
$\widehat{\Omega}_{x}$ be an estimate of the scalar variance $x^{\prime}\Sigma
x$. Then, by Proposition 7 (i),

$\displaystyle\left[x^{\prime}\hat{\mu}_{n}-n^{-1/2}c^{*}_{p}(1-\alpha;\widehat{\Omega}_{x}),\>x^{\prime}\hat{\mu}_{n}+n^{-1/2}c^{*}_{p}(1-\alpha;\widehat{\Omega}_{x})\right]$

is an asymptotic $(1-\alpha)$-confidence interval for $x^{\prime}\mu_{0}$.
Notice that this result holds without sparsity assumptions of $x$ or
$\mu_{0}$.

###  2.3 Asymptotic unbiasedness and consistency

Next, we establish unbiasedness and consistency of the bootstrap test
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$. Both properties turn out to
be more nuanced in high dimensions than the classical setting with fixed
dimensions. We conclude the section with a discussion of the effect of the
exponent $p\in[1,\infty]$ on the consistency of
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ against sparse and dense
alternatives.

The following theorem shows that (asymptotically) no alternative in
$\mathcal{H}_{1}$ has probability of rejection less than the size of the
bootstrap test. This means that we accept the alternative with higher
probability when it is correct than when it is false.

######  Theorem 8 (Unbiasedness).

Suppose that Assumptions 1–3 hold and, in addition, the $X_{i}$’s are
symmetric around their mean $\mu_{n}$ and have a log-concave density, i.e. the
density satisfies $f=e^{-\varphi}$ with $\varphi$ convex and
$\varphi(\mu_{n}-x)=\varphi(\mu_{n}+x)$ for all $x\in\mathbb{R}^{d}$. Then,
for all $1\leq p\leq\infty$ and all $\alpha\in(0,1)$,

$\displaystyle\lim_{n\rightarrow\infty}\mathrm{E}_{\mu}[\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})]\geq\alpha\quad\quad\forall\mu\in\mathcal{H}_{1}.$

######  Remark 6 (On the assumptions on the density).

In low dimensions, when $S_{n}=n^{-1/2}\sum_{i=1}^{n}R(X_{i}-\mu_{n})$
converges weakly under the alternative hypothesis to a Gaussian random vector,
the additional condition on the density of the $X_{i}$’s can be dropped since
the density of the limiting distribution $N(0,\Omega)$ is automatically
symmetric and log-concave. The situation is more complex in high dimensions
when $S_{n}$ does not converge to a Gaussian random vector and is not even
Gaussian approximable under the alternative hypothesis. In this case, symmetry
and log-concavity of the $X_{i}$’s allow us to invoke Anderson’s lemma and
reduce the problem to Gaussian approximation under the null hypothesis which
falls within the scope of Theorem 3.

It is straightforward to verify that in the classical setting with fixed
dimensions $d,t<\infty$ the bootstrap test
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ is asymptotically consistent
against any alternative $\mu\in\mathcal{H}_{1}$. In high dimensions with
$d,t\rightarrow\infty$ as $n\rightarrow\infty$ asymptotic consistency is a
more delicate property because the critical value
$c^{*}_{n,p}(1-\alpha;\widehat{\Omega}_{n})$ depends on the dimensions $d,t$
and diverges as $n\rightarrow\infty$. Thus, intuitively, in high dimensions
the bootstrap test is consistent only if the signal of the alternative
diverges faster than the critical value as $n\rightarrow\infty$. This is the
content of the next theorem.

######  Theorem 9 (Consistency under high-dimensional alternatives).

Suppose that Assumptions 1–3 hold. Then, for all $1\leq p\leq\infty$ the
following is true.

  * (i)

For any $\alpha\in(0,1)$,

$\displaystyle\lim_{n\rightarrow\infty}\sup_{(\mu_{n})_{n\geq
1}\in\mathcal{A}_{p}}\mathrm{E}_{\mu_{n}}[\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})]=\alpha,$

where $\mathcal{A}_{p}=\left\\{\left(\mu_{n}\right)_{n\geq
1}:\sqrt{n}\|R\mu_{n}-r\|_{p}\lesssim\mathrm{Var}(\|Z\|_{p})^{1/2}\right\\}$.

  * (ii)

For any $\alpha\in(0,1/2)$,

$\displaystyle\lim_{n\rightarrow\infty}\sup_{(\mu_{n})_{n\geq
1}\in\mathcal{B}_{p}}\mathrm{E}_{\mu_{n}}[\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})]<1,$

where $\mathcal{B}_{p}=\left\\{\left(\mu_{n}\right)_{n\geq
1}:\sqrt{n}\|R\mu_{n}-r\|_{p}\lesssim\mathrm{E}\|Z\|_{p}\right\\}$.

  * (iii)

For any $\alpha\in(0,1)$,

$\displaystyle\lim_{n\rightarrow\infty}\inf_{(\mu_{n})_{n\geq
1}\in\mathcal{C}_{p}}\mathrm{E}_{\mu_{n}}[\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})]=1,$

where $\mathcal{C}_{p}=\left\\{\left(\mu_{n}\right)_{n\geq
1}:\mathrm{E}\|Z\|_{p}\lesssim\sqrt{n}\|R\mu_{n}-r\|_{p}\right\\}$.

In above theorem, $\mathcal{A}_{p}$ is the set of alternatives (i.e. sequences
of mean vectors $(\mu_{n})_{n\geq 1}$) whose signals
$\sqrt{n}\|R\mu_{n}-r\|_{p}$ are asymptotically negligible compared to the
noise level of the Gaussian proxy statistic $\|Z\|_{p}$ with $Z\sim
N(0,\Omega)$. As one would expect, alternatives in $\mathcal{A}_{p}$ are
undetectable in the sense that the (asymptotic) probability of rejection is
not larger than the (asymptotic) size of the test
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$. The set $\mathcal{B}_{p}$
contains those alternatives whose signals are asymptotically smaller than the
expected value of $\|Z\|_{p}$. The bootstrap test is inconsistent for such
alternatives, but unlike in the case of $\mathcal{A}_{p}$ the power function
is not necessarily tacked to size $\alpha$. Lastly, the set $\mathcal{C}_{p}$
contains alternatives whose signals asymptotically dominate noise level and
expected value of the Gaussian proxy statistic $\|Z\|_{p}$. The bootstrap test
is asymptotically consistent for these alternatives.

From Theorem 9 it is conceivable that there exist alternatives
$(\mu_{n})_{n\geq 1}\in\mathcal{H}_{1}$ and exponents $p,q\geq 1$ such that
the bootstrap test based on $T_{n,p}$ is consistent whereas the test based on
$T_{n,q}$ is inconsistent. In the following we show that this intuition is
indeed correct.

######  Definition 1 (Sparse or dense alternatives).

Given a sparsity level $s\in\\{1,\ldots,t\\}$ and a signal strength
$\delta>0$, define

$\displaystyle\mathcal{D}_{\delta,s}=\left\\{\left(\mu_{n}\right)_{n\geq
1}:\|R\mu_{n}-r\|_{0}=s,\>|(R\mu_{n}-r)_{k}|\asymp\delta\sqrt{\mathrm{E}[Z_{k}^{2}]},\>k\in\\{j:(R\mu_{n}-r)_{j}\neq
0\\}\right\\},$

where $Z\sim N(0,\Omega)$. The set $\mathcal{D}_{\delta,s}$ contains “sparse”
alternatives if $s\ll t$ and “dense” alternatives if $s\asymp t$.

######  Remark 7 (On the uniform signal-to-noise ratio in Definition 1).

In the definition of sparse and dense alternatives we require that
asymptotically the signal-to-noise ratios of all non-zero entries in the
alternative are of the same order $\delta>0$. We believe that this is a
realistic assumption, since in applications the high-dimensional data is
either collected from a single data generating mechanism or, if aggregated
from different sources, it is standardized to guarantee that the measurements
have comparable levels of variability.

######  Proposition 10.

Consider the setup in Definition 1. Recall the notation of Theorem 9. Suppose
that Assumptions 1–3 hold and let $1\leq p<\log t\leq q\leq\infty$. Denote by
$\omega_{(1)}^{2}\leq\ldots\leq\omega_{(t)}^{2}$ the ordered values of
$\omega_{k}^{2}=\mathrm{E}[|Z_{k}|^{2}]$ where $Z\sim N(0,\Omega)$ and define
$a_{t}^{2}=\omega_{(t)}/\omega_{(1)}$ and
$b_{t}^{p}=\big{(}t^{-1}\sum_{k=1}^{t}\omega_{(k)}^{p}\big{)}/\big{(}s^{-1}\sum_{k=1}^{s}\omega_{(k)}^{p}\big{)}$
for $s\geq 1$.

  * (i)

If $s\ll t$ and $\sqrt{\frac{a_{t}^{2}\log
t}{n}}\lesssim\delta\lesssim\sqrt{\frac{b_{t}^{2}(t/s)^{2/p}}{n}}$, then

$\displaystyle\mathcal{D}_{\delta,s}\subseteq\mathcal{B}_{p}\cap\mathcal{C}_{q}.$

Thus, there exist signal strengths $\delta>0$ such that
$\varphi_{\alpha}(T_{n,q},\widehat{\Omega}_{n})$ with $q\geq\log t$ is
consistent and $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ with $p<\log
t$ is inconsistent against sparse $(\mu_{n})_{n\geq
1}\in\mathcal{D}_{\delta,s}(\Omega)$.

  * (ii)

If $s\asymp t$ and
$\sqrt{\frac{b_{t}^{2}p}{n}}\lesssim\delta\lesssim\sqrt{\frac{a_{t}^{2}\log
t}{n}}$, then

$\displaystyle\mathcal{D}_{\delta,s}\subseteq\mathcal{C}_{p}\cap\mathcal{B}_{q}.$

Thus, there exist signal strengths $\delta>0$ such that
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ with $p<\log t$ is consistent
and $\varphi_{\alpha}(T_{n,q},\widehat{\Omega}_{n})$ with $q\geq\log t$ is
inconsistent against dense $(\mu_{n})_{n\geq 1}\in\mathcal{D}_{\delta,s}$.

Above proposition substantiates the statistical folklore that tests based on
sum-of-squares type statistics (i.e. $T_{n,p}$ with $p<\log t$) have good
power against dense alternatives whose signals $R\mu_{n}-r$ are spread out
over a large number of coordinates, whereas tests based on maximum type
statistics (i.e. $T_{n,p}$ with $p\geq\log t$) are more powerful against
“sparse” alternatives with only a few strong signals $R\mu_{n}-r$.

Moreover, part (ii) of the proposition shows that the bootstrap tests
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ with $p<\log t$ are
consistent against dense alternatives with coordinate-wise effects as small as
$O(1/\sqrt{n})$ (provided that $b_{t}=O(1)$). Thus, these tests can detect
deviations from the global null hypothesis when the coordinate-wise effects
are well below the detection threshold once traditional multiple comparison
adjustments such as Bonferoni-based family-wise error rate control are taken
into account. This is useful for establishing the existence of significant
aggregate effects composed of otherwise negligible individual effects.

###  2.4 Worst case Bahadur slope

In the preceding section we showed that there exist alternatives and test
statistics $T_{n,p}$ and $T_{n,q}$ such the bootstrap test based on $T_{n,p}$
is consistent and the one based on $T_{n,q}$ is inconsistent. In such a
situation, we will always choose the consistent over the inconsistent test.
But how can we choose between two statistics $T_{n,p}$ and $T_{n,q}$ if the
associated tests $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ and
$\varphi_{\alpha}(T_{n,q},\widehat{\Omega}_{n})$ are both consistent against
the alternative under consideration?

In the classical setting with fixed dimensions $d,t<\infty$ and in certain
high-dimensional settings when the test statistic has a (Gaussian) limit
distribution (e.g. Chen et al., 2010; Cai et al., 2014; Wang et al., 2015; He
et al., 2021) this question can be easily answered by computing the asymptotic
relative efficiency of the tests based on Pitman alternatives. Since our test
statistics do not have a limiting distribution and the Gaussian approximation
results hold only under the null hypothesis, we cannot pursue this approach.
Instead, we suggest comparing the tests
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ based on their _observed
significance levels_ as this requires only knowledge of the sampling
distribution under the null hypothesis. Observed significance levels can be
interpreted as a measure of strength of the observed sample as evidence
against the null hypothesis. In particular, the smaller an observed
significance level, the greater the evidence against the null hypothesis. The
disadvantage of using observed significance levels is that they depend on the
(unknown) distribution of the data. To obtain concrete results, we will
therefore impose additional distributional assumptions.

To formalize this approach, let $F_{n,\mu}$ be the sampling distribution of
$T_{n,p}$ if the $X_{i}$’s have mean $\mu$. Since the bootstrap test
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ is significant for large
values of $T_{n,p}$, the observed significance level is

$\displaystyle\sup_{\mu\in\mathcal{H}_{0}}\mathrm{P}_{\mu}\left(T_{n,p}\geq
u\right)\mid_{u=T_{n,p}}=\sup_{\mu\in\mathcal{H}_{0}}\big{(}1-F_{n,\mu}(T_{n,p})\big{)}.$
(3)

If the $X_{i}$’s were drawn from $P_{\mu}$ with
$\mu\in\mathcal{H}_{0}=\\{\mu_{0}\\}$ the observed significance level would be
uniformly distributed on $(0,1)$, whereas if the $X_{i}$’s were drawn from
$P_{\mu}$ with $\mu\notin\mathcal{H}_{0}$ (and dimensions $d,t<\infty$ fixed)
it would converge to zero at an exponential rate by Cramér’s large deviation
principle. It is therefore of interest to understand the behavior of

$\displaystyle-\frac{1}{n^{\gamma}}\log\Big{(}\sup_{\mu\in\mathcal{H}_{0}}\big{(}1-F_{n,\mu}(T_{n,p})\big{)}\Big{)}\quad{}\quad{}\mathrm{as}\quad{}\quad{}n\rightarrow\infty\quad{}\quad{}\mathrm{for\>some\>\>}\gamma\in(0,1].$
(4)

The limit, if it exists, is called the Bahadur slope (Serfling, 1980, Chapter
10.4). The larger the Bahadur slope of a test, the “faster” the rate at which
the test gathers evidence against the null hypothesis at any given sample size
$n$. In high dimensions with $d,t\rightarrow\infty$ as $n\rightarrow\infty$
obtaining the exact limit is extremely challenging as it requires a large
deviation principle for high-dimensional random vectors. We believe that
developing such a large deviation principle should be addressed in a separate,
future paper. Here, we only derive an asymptotic lower bound on the quantity
in eq. (4) which affords ranking tests based on their “worst case” Bahadur
slopes.

######  Theorem 11 (Asymptotic bounds on Bahadur slope).

Suppose that Assumptions 1–3 hold and
$\limsup_{n\rightarrow\infty}\mathrm{E}[\|Z\|_{p}]/(\sqrt{n}\|R\mu_{n}-r\|_{p})<1$
for $1\leq p\leq\infty$.

  * (i)

If the $X_{i}$’s have a log-concave density, i.e. the density satisfies
$f=e^{-\varphi}$ with $\varphi$ convex, then

$\displaystyle-\frac{1}{\sqrt{n}}\log\Big{(}\sup_{\mu\in\mathcal{H}_{0}}\big{(}1-F_{n,\mu}(T_{n,p})\big{)}\Big{)}\gtrsim\frac{\|R\mu_{n}-r\|_{p}}{\|\Omega^{1/2}\|_{2\rightarrow
p}}+o_{p}(1).$

  * (ii)

If the $X_{i}$’s have a strictly log-concave density, i.e. the density
satisfies $f=e^{-\varphi}$ where $\varphi^{\prime\prime}\geq\lambda I$ with
$\lambda>0$, then

$\displaystyle-\frac{1}{n}\log\Big{(}\sup_{\mu\in\mathcal{H}_{0}}\big{(}1-F_{n,\mu}(T_{n,p})\big{)}\Big{)}\gtrsim\left(\frac{\|R\mu_{n}-r\|_{p}}{\|\Omega^{1/2}\|_{2\rightarrow
p}/\sqrt{\lambda}}\right)^{2}+o_{p}(1).$

  * (iii)

If the $X_{i}$’s are Gaussian random vectors and, in addition,
$\log\left(\|R\mu_{n}-r\|_{p}/\|\Omega^{1/2}\|_{2\rightarrow p}\right)=o(n)$,
then

$\displaystyle-\frac{1}{n}\log\Big{(}\sup_{\mu\in\mathcal{H}_{0}}\big{(}1-F_{n,\mu}(T_{n,p})\big{)}\Big{)}\asymp\left(\frac{\|R\mu_{n}-r\|_{p}}{\|\Omega^{1/2}\|_{2\rightarrow
p}}\right)^{2}+o_{p}(1).$

######  Remark 8 (On the distributional assumptions).

As in Theorem 8 we impose log-concavity to navigate around certain
shortcomings of the available technical tools. If there existed a large
deviation principle for random vectors whose dimension $d$ varies with the
sample size $n$, we would not need this assumption. However, in the absence of
such a result, log-concavity conveniently guarantees via Borell’s lemma that
$\ell_{p}$-norms of the averages of random vectors have sub-exponential tails
for all $d$ and $n$. In the case of Gaussian random vectors we are able to
further strengthen the result and derive a matching upper bound. Notice that
the condition $\log\left(\|R\mu_{n}-r\|_{p}/\|\Omega^{1/2}\|_{2\rightarrow
p}\right)=o(n)$ imposes an implicit constraint on the growth rate of the
dimension $d$ and the Poincaré constant $\|\Omega^{1/2}\|_{2\rightarrow p}$.
This hints at some of the challenges of obtaining large deviation principles
for (arbitrary) random vectors whose dimension is a function of the sample
size.

######  Remark 9 (On the Poincaré constant $\|\Omega^{1/2}\|_{2\rightarrow
p}$).

Since $\sqrt{\mathrm{Var}(\|Z\|_{p})}\leq\|\Omega^{1/2}\|_{2\rightarrow p}$,
one might wonder whether the lower bounds in Theorem 11 should in fact depend
on the variance of the Gaussian proxy statistic $\|Z\|_{p}$, just as the
results in Section 2.3. Statement (iii) of Theorem 11 shows that this is not
the case and that the Poincaré constant
$\|\Omega^{1/2}\|_{2\rightarrow\infty}$ is indeed the correct quantity. While
the results in Section 2.3 rely on Gaussian approximation arguments and thus
“local” deviations characterized by the standard deviation
$\sqrt{\mathrm{Var}(\|Z\|_{p})}$, Theorem 11 depends on large deviations and
concentration of measure type arguments involving the Poincaré constant.

Theorem 11 complements Theorem 9 insofar as it introduces another qualitative
measure by which we can compare tests: While Theorem 9 enables us to compare
tests based on whether they are consistent or not, Theorem 11 allows to rank
consistent tests based on their Bahadur slopes. We formalize this idea in the
following definition:

######  Definition 2 (Bahadur preferred test).

Consider two tests $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ and
$\varphi_{\alpha}(T_{n,p^{\prime}},\widehat{\Omega}_{n})$, both consistent
against alternatives in $\mathcal{H}\subseteq\mathcal{H}_{1}$. We say that
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ is Bahadur preferred over
$\varphi_{\alpha}(T_{n,p^{\prime}},\widehat{\Omega}_{n})$ when testing against
alternatives in $\mathcal{H}$ if, for all alternatives in $\mathcal{H}$, the
lower bound on the Bahadur slope of
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ is at least as large as the
lower bound on the Bahadur slope of
$\varphi_{\alpha}(T_{n,p^{\prime}},\widehat{\Omega}_{n})$.

To illustrate this approach of ranking tests, we continue the discussion of
sparse and dense alternatives started in Proposition 10.

######  Proposition 12.

Consider the setup in Definition 1. Recall the notation of Theorem 9. Suppose
that Assumptions 1–3 and any of the sufficient conditions of Theorem 11 hold.
In addition, suppose that $\Omega$ is a diagonal matrix.

  * (i)

If $p\in[\log t,\infty]$, $s\ll t$, and $\sqrt{(\log t)/n}\lesssim\delta$,
then

$\displaystyle\mathcal{D}_{\delta,s}\subseteq\mathcal{C}_{\log
t}\cap\mathcal{C}_{p}\quad{}\quad{}\mathrm{and}\quad{}\quad{}\frac{\|R\mu_{n}-r\|_{\log
t}}{\|\Omega^{1/2}\|_{2\rightarrow\log
t}}\geq\frac{\|R\mu_{n}-r\|_{p}}{\|\Omega^{1/2}\|_{2\rightarrow
p}}\quad{}\forall\>(\mu_{n})_{n\geq 1}\in\mathcal{D}_{\delta,s}.$

Thus, when testing against sparse alternatives $\mathcal{D}_{\delta,s}$ there
exist signal strengths $\delta>0$ such that $\varphi_{\alpha}(T_{n,\log
t},\widehat{\Omega}_{n})$ is Bahadur preferred over all tests
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ with $p\in[\log t,\infty]$.

  * (ii)

If $p,q\in[1,\log t]$, $p\leq q$, $s\asymp t$, and $\sqrt{q/n}\lesssim\delta$,
then

$\displaystyle\mathcal{D}_{\delta,s}\subseteq\mathcal{C}_{2}\cap\mathcal{C}_{p}\quad{}\quad{}\mathrm{and}\quad{}\quad{}\frac{\|R\mu_{n}-r\|_{2}}{\|\Omega^{1/2}\|_{2\rightarrow
2}}\geq\frac{\|R\mu_{n}-r\|_{p}}{\|\Omega^{1/2}\|_{2\rightarrow
p}}\quad{}\forall\>(\mu_{n})_{n\geq 1}\in\mathcal{D}_{\delta,s}.$

Thus, when testing against dense alternatives $\mathcal{D}_{\delta,s}$ there
exist signal strengths $\delta>0$ such that
$\varphi_{\alpha}(T_{n,2},\widehat{\Omega}_{n})$ is Bahadur preferred over all
tests $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ with $p\in[1,q]$.

######  Remark 10 (On the assumption on $\Omega$).

Definition 2 is only useful if we can compute the Poincaré constant
$\|\Omega^{1/2}\|_{2\rightarrow p}$. Evaluating
$\|\Omega^{1/2}\|_{2\rightarrow p}$ for arbitrary positive semi-definite
$\Omega$ is difficult and can even be NP-hard (Foucart and Rauhut, 2013).
Therefore, the assumption that $\Omega$ is diagonal is a compromise: on the
one hand, it is specific enough to simplify the computation of
$\|\Omega^{1/2}\|_{2\rightarrow p}$; on the other hand, it is broad enough to
cover a large class of tests.

######  Remark 11.

Obviously, case (i) also holds for $s\asymp t$. But then case (ii) simply
implies that $\varphi_{\alpha}(T_{n,2},\widehat{\Omega}_{n})$ is Bahadur
preferred over all tests $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$,
$p\in[1,\infty]$.

Proposition 12 is a surprising result: Unlike one could have expected, neither
$\varphi_{\alpha}(T_{n,\infty},\widehat{\Omega}_{n})$ nor
$\varphi_{\alpha}(T_{n,1},\widehat{\Omega}_{n})$ are Bahadur preferred tests.
Instead, under the setup of Definition 1 the bootstrap tests based on
$T_{n,\log t}$ and $T_{n,2}$ are “best” against sparse and dense alternatives,
respectively. The simulation study in Section 5 further corroborates this
conclusion even in finite samples.

######  Remark 12 (On the relation of observed significance level and
$p$-value).

Observed significance level and $p$-value are two different concepts that
yield the same numbers only in a few special situations. Given an observed
test statistic $T_{n,p}$ the $p$-value of
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ is defined as

$\displaystyle\inf\\{\alpha\in(0,1):\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})=1\\}=\mathrm{P}\left(T^{*}_{n,p}\geq
u\mid X_{1},\ldots,X_{n}\right)\mid_{u=T_{n,p}}=1-\widehat{G}_{n}(T_{n,p}),$

where $\widehat{G}_{n}$ is the conditional distribution of the Gaussian proxy
statistic

$\displaystyle T_{n,p}^{*}:=\|Z_{n}\|_{p},\quad{}\quad{}Z_{n}\mid
X_{1},\ldots,X_{n}\sim N(0,\widehat{\Omega}_{n}).$

Typically, $\widehat{G}_{n}\neq F_{n,\mu}$, and hence $p$-value and observed
significance level of $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ are
different.

From Theorem 3 we know that $\widehat{G}_{n}$ and $F_{n,\mu}$ are close in
Kolomgorov-Smirnov distance for all $\mu\in\mathcal{H}_{0}$. This implies that
$p$-values and observed significance levels are close as well. However, this
does not imply that the $p$-value and observed significance level accumulate
evidence against the null hypothesis at the same rate. Indeed, from Theorem 11
(iii) we infer that

$\displaystyle-\frac{1}{n}\log(1-\widehat{G}_{n}(T_{n,p}))\gtrsim\left(\frac{\|R\mu_{n}-r\|_{p}}{\|\widehat{\Omega}_{n}^{1/2}\|_{2\rightarrow
p}}\right)^{2}+o_{p}(1),$

independent of the sampling distribution of $T_{n,p}$. Incidentally, under the
distributional assumptions of Theorem 11, ranking tests
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ by the rates at which their
$p$-values vanish yields the same order as ranking them by their worst case
Bahadur slopes.

##  3 Three modifications of the basic bootstrap hypothesis test

###  3.1 A test with power against sparse and dense alternatives

The first modification of the basic bootstrap hypothesis test that we consider
is motivated by the theoretical analysis in Section 2.4 and aims at enhancing
the power of the test.

Consider the implications of Proposition 12 when the signal strength satisfies
$\delta\asymp\sqrt{(\log t)/n}$: In the case of dense alternatives with
$s\asymp t$ we easily verify (by Theorem 9) that all bootstrap tests based on
$\ell_{p}$-statistics are consistent. Hence, combining Proposition 12 (i) and
(ii) (with $q=\log t$) we deduce that the test based on $T_{n,2}$ is “best”
among all tests based on $\ell_{p}$-statistics with any $p\in[1,\infty]$.
Moreover, in the case of sparse alternatives with $s\ll t$, Theorem 9 implies
that tests based on $T_{n,p}$ with $p\in[1,\log t)$ are inconsistent. Hence,
by Proposition 12 (i) the bootstrap test based on $T_{n,\log t}$ is “best” not
just among all tests based on $\ell_{p}$-statistics with $p\in[\log t,\infty]$
but with any $p\in[1,\infty]$.

We therefore propose to combine test statistics $T_{n,2}$ and $T_{n,\log t}$
into the single test statistic

$\displaystyle W_{n}:=T_{n,2}+T_{n,\log
t}=\|RS_{n}-\sqrt{n}r\|_{2}+\|RS_{n}-\sqrt{n}r\|_{\log
t},\quad{}\quad{}S_{n}=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}X_{i}.$

Given a nominal level $\alpha\in(0,1)$, we reject the null hypothesis if and
only if

$\displaystyle W_{n}\geq c^{*}_{W}(1-\alpha;\widehat{\Omega}_{n}),$

where
$c^{*}_{W}(\alpha;\widehat{\Omega}_{n})=\inf\left\\{s\in\mathbb{R}:\mathrm{P}(W_{n}^{*}\leq
s\mid X_{1},\ldots,X_{n})\geq\alpha\right\\}$ is the conditional
$\alpha$-quantile of the Gaussian proxy statistic

$\displaystyle W_{n}^{*}:=\|Z_{n}\|_{2}+\|Z_{n}\|_{\log
t},\quad{}\quad{}Z_{n}\mid X_{1},\ldots,X_{n}\sim N(0,\widehat{\Omega}_{n}),$

and $\widehat{\Omega}_{n}$ is a positive semi-definite estimate of
$\Omega=R\Sigma R^{\prime}$. We define the _modified bootstrap test based on
$W_{n}$ at level $\alpha$_ as

$\displaystyle\varphi_{\alpha}(W_{n},\widehat{\Omega}_{n}):=\mathbf{1}\left\\{W_{n}\geq
c^{*}_{W}(1-\alpha;\widehat{\Omega}_{n})\right\\}.$

Given the theoretical results in Appendix A it is straightforward to show that
this test has asymptotically the correct size $\alpha$:

######  Proposition 1 (Asymptotic size $\alpha$ test).

Suppose that Assumptions 1–3 hold for $p\in\\{2,\log t\\}$. Then,

$\displaystyle\lim_{n\rightarrow\infty}\sup_{\alpha\in(0,1)}\sup_{\mu\in\mathcal{H}_{0}}\Big{|}\mathrm{E}_{\mu}[\varphi_{\alpha}(W_{n},\widehat{\Omega}_{n})]-\alpha\Big{|}=0.$

Moreover, by simple modification of Theorem 9 we conclude that the modified
bootstrap test is consistent against both, dense and sparse alternatives.

######  Proposition 2.

Suppose that Assumptions 1–3 hold for $p\in\\{2,\log t\\}$. Then, for
$\alpha\in(0,1)$ arbitrary,

$\displaystyle\lim_{n\rightarrow\infty}\inf_{(\mu_{n})_{n\geq
1}\in\mathcal{W}}\mathrm{E}_{\mu_{n}}[\varphi_{\alpha}(W_{n},\widehat{\Omega}_{n})]=1,$

where $\mathcal{W}=\left\\{\left(\mu_{n}\right)_{n\geq
1}:\mathrm{E}\|Z\|_{2}+\mathrm{E}\|Z\|_{\log
t}\lesssim\sqrt{n}\|R\mu_{n}-r\|_{2}+\sqrt{n}\|R\mu_{n}-r\|_{\log
t}\right\\}$.

We easily verify that $\mathcal{W}\supset\mathcal{D}_{\delta,s}$ for sparse
alternatives characterized by sparsity level $s\lesssim t$ and signal strength
$\delta\gtrsim\sqrt{(\log t)/n}$ as well as for dense alternatives
characterized by $s\asymp t$ and $\delta\gtrsim 1/\sqrt{n}$ (recall notation
of Definition 1). Thus, the modified bootstrap test
$\varphi_{\alpha}(W_{n},\widehat{\Omega}_{n})$ is consistent against dense and
sparse alternatives simultaneously.

######  Remark 1 (Comparison with the _power enhancement method_ by Fan et al.
(2015)).

The starting point for Fan et al.’s (2015) power enhancement method is the
observation that sum-of-square type statistics based on (such as
$T_{n,2}^{2}$) tend to have low power against sparse alternatives. To rectify
this, they suggest to add a _power enhancement component_ $J_{0}$ to the
initial sum-of-square type statistics. Such $J_{0}$ should satisfy three
intuitive properties: First, non-negativity, i.e. $J_{0}\geq 0$ almost surely;
second, no size distortion, i.e.
$\inf_{\mu\in\mathcal{H}_{0}}\mathrm{P}_{\mu}(J_{0}=0)\rightarrow 1$; third,
power enhancement, i.e. $J_{0}\rightarrow\infty$ in probability under some
alternative $\mathcal{H}_{1}$. The power enhancement component typically
depends on a tuning parameter $\delta>0$ that determines the minimal
(asymptotic) signal strength of an alternative $\mathcal{H}_{1}$ for which
$J_{0}\rightarrow\infty$ in probability. Adapted to our setup, Fan et al.
(2015) propose to set $\delta\asymp(\log\log n)\sqrt{(\log t)/n}$.

The philosophy of our bootstrap test
$\varphi_{\alpha}(W_{n},\widehat{\Omega}_{n})$ and the power enhancement by
Fan et al. (2015) are thus very different. Our bootstrap test is not based on
the idea of enhancing the power the test statistic $T_{n,2}$ without
distorting the size of the test (with the critical value being the
$\alpha$-quantile of the sampling distribution of $T_{n,2}$). Instead, our
bootstrap test is based on $T_{n,2}+T_{n,\log t}$ and the critical value is
the $\alpha$-quantile of the sampling distribution of $T_{n,2}+T_{n,\log t}$.
Hence, size distortion is no issue. Moreover, the bootstrap test does not
depend on a tuning parameter and the minimal signal strength that the
bootstrap test can detect can be as small as $\delta\gtrsim\sqrt{(\log t)/n}$
(sparse alternative) and $\delta\gtrsim 1/\sqrt{n}$ (dense alternative).

###  3.2 A test statistic for elliptically distributed data

We have so far kept the assumptions on the data generating process minimal and
have mostly relied on the three moment conditions Assumptions 1–3. These
moment conditions implicitly constrain the growth rates of the dimensions
$d,t$ relative to the sample size $n$. In this section, we propose a test
statistic based on self-normalized random vectors which has asymptotic correct
size under stronger assumptions on the data generating process but
considerably weaker implicit constraints on $d,t,n$. Consequently, the self-
normalized test performs well in even higher dimensions.

The results in this section hold under only the following assumption, i.e. in
the following Assumptions 1–3 are not needed.

######  Assumption 4 (Elliptically distributed data).

Let $X,X_{1},\ldots,X_{n}\in\mathbb{R}^{d}$ be a simple random sample of
elliptically distributed random vectors, i.e.

$\displaystyle X=\mu_{n}+\Gamma\theta U^{(s)},$

where $U^{(s)}\in\mathbb{R}^{s}$ is a random vector uniformly distributed on
the $\ell_{2}$-norm unit sphere $\mathbb{S}^{s-1}$, $\theta$ is a non-negative
random variable independent of $U$, $\mu_{n}\in\mathbb{R}^{d}$, and
$\Gamma\in\mathbb{R}^{d\times s}$. In particular, $X$ has mean $\mu_{n}$ and
covariance matrix
$\Sigma=s^{-1}\mathrm{E}[\theta^{2}]\Gamma\Gamma^{\prime}\in\mathbb{R}^{d\times
d}$ with $\mathrm{rank}(\Sigma)=s\wedge d$.

######  Remark 2 (On elliptically distributed data).

The class of elliptical distributions includes many classical non-Gaussian
multivariate distributions such as multivariate t-distribution, multivariate
logistic distribution, Kotz-type multivariate distribution, and Pearson II
type multivariate distribution (Fang et al., 1990). Elliptical distributions
have proved useful for modeling tail dependence (i.e. clustering of extremes)
in financial data (McNeil et al., 2005). Assumption 4 does not (!) imply
Assumptions 1–3.

Let $X_{1},\ldots,X_{n}\in\mathbb{R}^{d}$ be a simple random sample of
elliptically distributed random vectors and denote by
$\widetilde{X},\widetilde{X}_{1},\ldots,\widetilde{X}_{n}\in\mathbb{R}^{t}$
their projections on the Euclidean unit sphere with center
$r\in\mathbb{R}^{t}$,

$\displaystyle\widetilde{X}:=\begin{cases}\frac{RX-r}{\|RX-r\|_{2}},&\|RX-r\|_{2}\neq
0,\\\ 0,&o/w.\end{cases}$

From the characterization in Assumption 4 it follows that $\widetilde{X}\sim
R\Gamma U^{(s)}$, where $U^{(s)}\in\mathbb{R}^{s}$ is a random vector
uniformly distributed on the $\ell_{2}$-norm unit sphere $\mathbb{S}^{s-1}$
for all $\mu\in\mathcal{H}_{0}$. In particular,
$\mathrm{E}_{\mu}[\widetilde{X}]=0$ for all $\mu\in\mathcal{H}_{0}$. Moreover,
in the proofs to below results we will show that the projection
$\widetilde{X}$ satisfies the simple sufficient conditions of Lemma 3. We
therefore use these projections $\widetilde{X}_{1},\ldots,\widetilde{X}_{n}$
as building blocks for a _self-normalized test statistic_ : Define

$\displaystyle
V_{n}:=\left\|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\widetilde{X}_{i}\right\|_{2},$

and, given a nominal level $\alpha\in(0,1)$, we reject the null hypothesis if
and only if

$\displaystyle V_{n}\geq c^{*}_{V}(1-\alpha;\widetilde{\Omega}_{n}),$

where
$c^{*}_{V}(\alpha;\widetilde{\Omega}_{n})=\inf\left\\{s\in\mathbb{R}:\mathrm{P}(V_{n}^{*}\leq
s\mid X_{1},\ldots,X_{n})\geq\alpha\right\\}$ is the conditional
$\alpha$-quantile of the Gaussian proxy statistic

$\displaystyle V_{n}^{*}:=\|Z_{n}\|_{2},\quad{}\quad{}Z_{n}\mid
X_{1},\ldots,X_{n}\sim N(0,\widetilde{\Omega}_{n}),$

and $\widetilde{\Omega}_{n}$ is a positive semi-definite estimate of
$\Omega=\mathrm{E}[\widetilde{X}\widetilde{X}^{\prime}]$. We define the _self-
normalized bootstrap test based on $V_{n}$ at level $\alpha$_ as

$\displaystyle\varphi_{\alpha}(V_{n},\widetilde{\Omega}_{n}):=\mathbf{1}\left\\{V_{n}\geq
c^{*}_{V}(1-\alpha;\widetilde{\Omega}_{n})\right\\}.$

We have the following result:

######  Proposition 3 (Asymptotic size $\alpha$ test).

Let
$\widetilde{\Omega}_{n}=n^{-1}\sum_{i=1}^{n}\widetilde{X}_{i}\widetilde{X}_{i}^{\prime}$.
If Assumption 4 holds and
$\|R\Gamma\Gamma^{\prime}R^{\prime}\|_{op}=o(n^{1/6}s^{-1}\|R\Gamma\Gamma^{\prime}R^{\prime}\|_{F})$,
then

$\displaystyle\lim_{n\rightarrow\infty}\sup_{\alpha\in(0,1)}\sup_{\mu\in\mathcal{H}_{0}}\Big{|}\mathrm{E}_{\mu}[\varphi_{\alpha}(V_{n},\widetilde{\Omega}_{n})]-\alpha\Big{|}=0.$

Since $s\geq\mathrm{rank}(R\Gamma\Gamma^{\prime}R^{\prime})$, we typically
have $s^{-1}\|R\Gamma\Gamma^{\prime}R^{\prime}\|_{F}=O(1)$. Therefore, the
requirement
$\|R\Gamma\Gamma^{\prime}R^{\prime}\|_{op}=o(n^{1/6}s^{-1}\|R\Gamma\Gamma^{\prime}R^{\prime}\|_{F})$
is very mild and easily satisfied even when the covariance matrix $\Sigma$ has
full rank and/ or is dense. In particular, the dimensions $d,t,s$ can be
arbitrarily large relative to the sample size $n$. However, these mild
requirements come at a slight disadvantage: The estimate
$\widetilde{\Omega}_{n}=n^{-1}\sum_{i=1}^{n}\widetilde{X}_{i}\widetilde{X}_{i}^{\prime}$
depends on the hypothesis $R\mu_{n}=r$. Thus, the Gaussian proxy statistic
$V^{*}_{n}$ is no longer ancillary. Consequently, we can establish consistency
of the self-normalized bootstrap test only for one alternative hypothesis at a
time, not uniformly over all alternatives. We make this explicit in the next
proposition by indexing the estimate $\widetilde{\Omega}_{n}$ by the
alternative $\mu_{n}$.

######  Proposition 4 (Consistency under high-dimensional alternatives).

For arbitrary $\mu\in\mathbb{R}^{d}$ define
$\widetilde{\Omega}_{\mu}:=n^{-1}\sum_{i=1}^{n}\widetilde{X}_{i,\mu}\widetilde{X}_{i,\mu}^{\prime}$,
where $\widetilde{X}_{i,\mu}=R(X_{i}-\mu)/\|R(X_{i}-\mu)\|_{2}$. If Assumption
4 holds and
$\|R\Gamma\Gamma^{\prime}R^{\prime}\|_{op}=o(n^{1/6}s^{-1}\|R\Gamma\Gamma^{\prime}R^{\prime}\|_{F})$,
then for $\alpha\in(0,1)$ arbitrary,

$\displaystyle\lim_{n\rightarrow\infty}\inf_{(\mu_{n})_{n\geq
1}\in\mathcal{E}}\mathrm{E}_{\mu_{n}}[\varphi_{\alpha}(V_{n},\widetilde{\Omega}_{\mu_{n}})]=1,$

where $\mathcal{E}=\left\\{\left(\mu_{n}\right)_{n\geq
1}:\lim_{n\rightarrow\infty}\sqrt{n}\left\|\mathrm{E}_{\mu_{n}}\left[\frac{RX-r}{\|RX-r\|_{2}}\right]\right\|_{2}=\infty\right\\}$.

Notice that $\mathrm{E}[\|Z\|_{2}]\leq 1$ for $Z\sim
N(0,\widetilde{\Omega}_{\mu_{n}})$ (see proof of Proposition 3). Hence, we can
re-write the set of alternatives as
$\mathcal{E}=\\{\left(\mu_{n}\right)_{n\geq
1}:\mathrm{E}[\|Z\|_{2}]=o\big{(}\sqrt{n}\|\mathrm{E}_{\mu_{n}}[(RX-r)/\|RX-r\|_{2}]\|_{2}\big{)}\\}$.
In this form, $\mathcal{E}$ matches the intuition developed in Section 2.2 and
Theorem 9: An alternative $\mu_{n}\in\mathcal{E}$ is detectable because its
signal $\sqrt{n}\|\mathrm{E}_{\mu_{n}}[(RX-r)/\|RX-r\|_{2}]\|_{2}$ dominates
the expected value of the Gaussian proxy statistic $\|Z\|_{2}$.

######  Remark 3 (Comparison with the _high-dimensional nonparametric test_ by
Wang et al. (2015)).

Our self-normalized bootstrap test is inspired by Wang et al. (2015) who, to
the best of our knowledge, were the first to propose a sum-of-square type
statistic for elliptically distributed data based on the normalized random
vectors $X_{1}/\|X_{1}\|_{2},\ldots,X_{n}/\|X_{n}\|_{2}$.

The most interesting and consequential difference between their test and ours
lies in the theoretical analysis. The asymptotic theory in Wang et al. (2015)
is driven by “asymptotics over the dimension”. By this we mean that the test
statistic is constructed in such a way that increasing the dimension reduces
the variability of each individual summand of the test statistic. Then,
provided that the entries in the high-dimensional random vectors are not be
too correlated, a Martingale CLT applies. Earlier attempts in this direction
by Bai and Saranadasa (1996) and Chen et al. (2010) required data generated by
a factor model and strong moment conditions. The assumptions in Wang et al.
(2015) are much weaker, but they are violated if $\mathrm{tr}(\Sigma)=O(1)$,
i.e. if the data lie in a (approximate) low-dimensional subspace (a.k.a. if
the data are highly correlated). In contrast, the asymptotic theory of the
bootstrap test relies on Gaussian approximation which is related to the
classical “asymptotics over the sample”. As seen in Propositions 3 and 4 this
results in very mild assumptions on the dimension and the covariance matrix
$\Sigma$.

###  3.3 Bootstrap test statistics based on approximate sample averages

The basic bootstrap test (including above two modifications) is based on the
average of i.i.d. random vectors. However, in many applications, the “natural”
statistic to base an $\ell_{p}$-statistic on is a nonlinear transformation of
the data, i.e. $T_{n,p}=\|F(X_{1},\ldots,X_{n})\|_{p}$ with
$F:\bigtimes_{i=1}^{n}\mathbb{R}^{d}\rightarrow\mathbb{R}^{t}$ and $F$
nonlinear. For example, $F$ could be a function that captures certain pre-
processing steps such as removal of biases or handling of missingness and
heteroscedasticity in a data set. Or $F$ could be a procedure that returns an
estimate of a high-dimensional regression vector, e.g. solution to a
generalized estimating equation.

In classical statistics with fixed dimension, asymptotic theory can be
developed whenever $F$ is sufficiently smooth via continuous mapping theorem,
delta method, or linearization. An analogous result holds true for the high-
dimensional bootstrap test:

######  Proposition 5.

Consider a random sample $Y_{1},\ldots,Y_{n}$. Let
$F:\bigtimes_{i=1}^{n}\mathbb{R}^{d}\rightarrow\mathbb{R}^{t}$. If, under the
null hypothesis,

$\displaystyle
F(Y_{1},\ldots,Y_{n})-\sqrt{n}r=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}R(X_{i}-\mu)+W_{n}\quad{}\quad{}\mathrm{and}\quad{}\quad{}\|W_{n}\|_{p}=o_{p}\left(\sqrt{\mathrm{Var}(\|Z\|_{p})}\right),$
(5)

where $X_{1},\ldots X_{n}\in\mathbb{R}^{d}$ are i.i.d. random vectors
(possibly changing with $n$) with mean $\mu\in\mathbb{R}^{d}$ and positive
semi-definite covariance matrix $\Sigma\neq\mathbf{0}\in\mathbb{R}^{d\times
d}$, $Z\sim N(0,\Omega)$ with $\Omega\equiv R\Sigma
R^{\prime}\neq\mathbf{0}\in\mathbb{R}^{t\times t}$ and
$R\in\mathbb{R}^{t\times d}$ (possibly changing with $n$), then Theorems 4 and
9 and Propositions 7, 1, and 2 hold with the test statistic
$\|F(Y_{1},\ldots,Y_{n})-\sqrt{n}r\|_{p}$ substituted for
$T_{n,p}=\|RS_{n}-\sqrt{n}r\|_{p}$.

Intuitively, this result says that the bootstrap hypothesis test is valid
whenever $F$ affords a linear expansion with a remainder term $W_{n}$ that is
asymptotically negligible compared to the noise level the leading term of the
linear expansion (which is captured by $\sqrt{\mathrm{Var}(\|Z\|_{p})}$).
Theorems 8 and 11 do not necessarily hold under the condition in eq. (5)
because they require stronger non-asymptotic lower bounds on tail
probabilities of log-concave measures.

##  4 A practical guide to the bootstrap hypothesis test

###  4.1 Three sources of bias

To test a global null hypothesis such as (1) via the bootstrap test
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ we need to compute the
conditional $(1-\alpha)$-quantile $c^{*}_{p}(1-\alpha;\widehat{\Omega}_{n})$
of the Gaussian proxy statistic $T_{n,p}^{*}$. Since there does not exist an
analytic expression of the distribution of $T_{n,p}^{*}$, we suggest
approximating $c^{*}_{p}(1-\alpha;\widehat{\Omega}_{n})$ via Monte Carlo
simulations. Thus, in practice, we test hypothesis (1) using the _Monte Carlo
bootstrap hypothesis test_ defined as

$\displaystyle\varphi_{\alpha,B}(T_{n,p},\widehat{\Omega}_{n}):=\mathbf{1}\left\\{T_{n,p}\geq
c^{*}_{p,B}(1-\alpha;\widehat{\Omega}_{n})\right\\},$

where $c^{*}_{p,B}(1-\alpha;\widehat{\Omega}_{n})$ denotes the
$\lfloor(1-\alpha)B\rfloor$th order statistic of the random sample
$T_{n,p,1}^{*},\ldots,T_{n,p,B}^{*}$ with $T_{n,p,b}^{*}:=\|Z_{n,b}\|_{p}$ and
$Z_{n,b}\mid X_{1},\ldots,X_{n}\sim_{iid}N(0,\widehat{\Omega})$ for all
$b=1,\ldots,B$.

Since the test statistic is non-pivotal (i.e. the sampling distribution of the
test statistic depends on the unknown population covariance $\Omega$)
theoretical and empirical results from the classical, low-dimensional
bootstrap literature suggest that the Monte Carlo bootstrap test might be
severely biased (e.g. Davison et al., 1986; Beran, 1987; Hall and Martin,
1988; Shi, 1992; Shao and Tu, 1995). In the following we therefore discuss
three sources of potential biases in our high-dimensional setting. This
discussion lays the foundation for developing strategies to mitigate two of
these biases (Sections 4.2 and 4.3).

Consider the following decomposition of the bias of the Gaussian Monte Carlo
bootstrap test:

$\displaystyle\begin{split}\mathrm{E}_{\mu}\left[\varphi_{\alpha,B}(T_{n,p},\widehat{\Omega}_{n})\right]-\alpha&=\underbrace{\mathrm{E}_{\mu}\left[\varphi_{\alpha}(T_{n,p},\Omega)\right]-\alpha}_{\mathrm{Gaussian\>approximation\>error}}+\>\>\>\underbrace{\mathrm{E}_{\mu}\left[\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})\right]-\mathrm{E}_{\mu}\left[\varphi_{\alpha}(T_{n,p},\Omega)\right]}_{\mathrm{Gaussian\>comparison\>error}}\\\
&\quad{}+\underbrace{\mathrm{E}_{\mu}\left[\varphi_{\alpha,B}(T_{n,p},\widehat{\Omega}_{n})\right]-\mathrm{E}_{\mu}\left[\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})\right]}_{\mathrm{Monte\>Carlo\>error}}.\end{split}$
(6)

The “Gaussian approximation error” in above display is the error of
approximating the $(1-\alpha)$th quantile of the test statistic $T_{n,p}$ with
the $(1-\alpha)$th quantile of the Gaussian proxy statistic $\|Z\|_{p}$, where
$Z\sim N(0,\Omega)$. In other words, this error measures the distance between
the distributions of $T_{n,p}$ and $\|Z\|_{p}$. Since in the classical setting
with fixed dimensions bootstrapping procedures are known to be consistent if
and only if the bootstrapped statistic satisfies a CLT (Mammen, 1993; van der
Vaart and Wellner, 1996, Theorem 3.6.1), it is intuitive that the Gaussian
approximation error is part of the bias decomposition (6). Theorem 1 (in
Appendix A.1) provides an upper bound on this error uniformly in
$\alpha\in(0,1)$. Under Assumptions 1 and 2 this upper bound vanishes as
$n,d,t\rightarrow\infty$. The rate at which the Gaussian approximation error
vanishes is only of theoretical interest as it is independent of the Monte
Carlo procedure and the data $X_{1},\ldots,X_{n}$. For all practical purposes,
the only way to reduce this error is by increasing the sample size $n$.

The “Gaussian comparison error” in bias decomposition (6) arises from
comparing the $(1-\alpha)$th quantiles of the Gaussian proxy statistics
$\|Z\|_{p}$ and $\|Z_{n}\|_{p}$, where $Z\sim N(0,\Omega)$ and $Z_{n}\mid
X_{1},\ldots,X_{n}\sim N(0,\widehat{\Omega}_{n})$, respectively. From Theorem
2 (in Appendix A.1) we deduce that the rate at which this error vanishes
depends on how fast $\|\widehat{\Omega}_{n}-\Omega\|_{q\rightarrow
p}/\mathrm{Var}(\|Z\|_{p})\rightarrow 0$ in probability, where $1/p+1/q=1$.
Consequently, the choice of the estimator $\widehat{\Omega}_{n}$ has
significant impact on the Gaussian comparison error. We discuss in Section 4.2
how to exploit this simple observation to improve the accuracy of the
bootstrap hypothesis test.

The third term in bias decomposition (6) is the “Monte Carlo error” of
approximating the $(1-\alpha)$th quantile of the Gaussian proxy statistic
$T_{n,p}^{*}$. Under the mild assumption that the test statistic $T_{n,p}$ has
a density (or probability mass function) we have the following result:

######  Theorem 1 (Gaussian Monte Carlo Error).

Let $M_{n,p}={\mathrm{ess}\sup}_{z,\mu}f_{n,p,\mu}(z)$, where $f_{n,p,\mu}$ is
the density of the test statistics $T_{n,p}$ when the $X_{i}$’s have mean
$\mu$. Then, for all $1\leq p\leq\infty$,

$\displaystyle\sup_{\alpha\in(0,1)}\sup_{\mu}\left|\mathrm{E}_{\mu}\left[\varphi_{\alpha,B}(T_{n,p},\widehat{\Omega}_{n})\right]-\mathrm{E}_{\mu}\left[\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})\right]\right|$
$\displaystyle\lesssim B^{-1/2}(\log
B)\sqrt{M_{n,p}^{2}\mathrm{E}\left[\mathrm{Var}(T_{n,p}^{*}\mid
X_{1},\ldots,X_{n})\right]},$

where $\lesssim$ hides an absolute constant independent of $p,n,d,t,B$, and
the distribution of the $X_{i}$’s.

######  Remark 1 (On size, consistency, and power of the Gaussian Monte Carlo
bootstrap test).

The upper bound on the Monte Carlo error increases in the conditional variance
of the Gaussian proxy statistic $T_{n,p}^{*}$ and the mode $M_{n,p}$ of the
density of the statistic $T_{n,p}$, but decreases in the number of Monte Carlo
samples $B$. Thus, Theorems 4 and 9 and Propositions 7, 1–5 continue to hold
if we substitute the Monte Carlo Bootstrap test
$\varphi_{\alpha,B}(T_{n,p},\widehat{\Omega}_{n})$ for the infeasible
bootstrap test $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$ and choose
$B\gtrsim n\vee\left(M_{n,p}^{2}\mathrm{E}\left[\mathrm{Var}(T_{n,p}^{*}\mid
X_{1},\ldots,X_{n})\right]\right)^{1+\gamma}$, $\gamma>0$ arbitrary.

The dependence of the upper bound in Theorem 1 on the number of Monte Carlo
samples $B$ and the conditional variance $\mathrm{Var}(T_{n,p}^{*}\mid
X_{1},\ldots,X_{n})$ is intuitively obvious. It is, however, noteworthy,
because its proof applies to any Monte Carlo sampling scheme and does not rely
on the specific Gaussian proxy statistic $T_{n,p}^{*}\equiv\|Z_{n}\|_{p}$,
$Z_{n}\mid X_{1},\ldots,X_{n}\sim N(0,\widehat{\Omega}_{n})$. This raises the
prospect that it might be possible to reduce the Monte Carlo error by
designing an alternative (potentially non-Gaussian) Monte Carlo sampling
scheme based on proxy statistics that have smaller variances than the Gaussian
proxy statistics $T_{n,p}^{*}$.

Finding such an alternative Monte Carlo sampling scheme is a challenging task
since for certain exponents $p$ and covariance matrices $\widehat{\Omega}_{n}$
the Gaussian proxy statistic $T_{n,p}^{*}$ is “superconcentrated”, i.e. for
certain exponents and covariance matrices the variance of $T_{n,p}^{*}$ is a
decreasing function of the dimension of the Gaussian random vector $Z_{n}$ and
hence (asymptotically) already very small (see Remark 2). In Section 4.3 we
succeed in developing a simple and practical alternative using
hypercontractivity tools for Markov semigroups (i.e. Cordero-Erausquin and
Ledoux, 2012; Tanguy, 2017). Numerical experiments in Section 5 show that this
alternative Monte Carlo sampling scheme substantially improves the accuracy of
the bootstrap tests in finite samples.

######  Remark 2 (On the superconcentration of $\ell_{p}$-norms of Gaussian
random vectors in $\mathbb{R}^{d}$).

In the case of isotropic Gaussian random vectors, $Z\sim N(0,I_{d})$, the
variance of $\|Z\|_{p}$ has several phase transitions depending on the
interplay between dimension $d$ and exponent $p$ (Lytova and Tikhomirov,
2019). In particular, $\mathrm{Var}(\|Z\|_{2})\asymp 1$ (i.e. not
superconcentrated) and $\mathrm{Var}(\|Z\|_{\infty})\asymp(\log d)^{-1}$ (i.e.
superconcentrated), which, by Theorem 1, lends theoretical support to the
empirical observation that the bias of the Gaussian Monte Carlo hypothesis
test based on the $\ell_{\infty}$-norm is significantly smaller than the one
based on the $\ell_{2}$-norm.

In the statistically more relevant case of an-isotropic Gaussian random
vectors, $Z\sim N(0,\Sigma)$, the variance of $\|Z\|_{p}$ is less well
studied. In principle, one can obtain upper bounds on
$\mathrm{Var}(\|Z\|_{p})$ via Talagrand’s $L_{1}$-$L_{2}$ inequality (Cordero-
Erausquin and Ledoux, 2012). For $\ell_{2}$\- and $\ell_{\infty}$-norms the
following useful estimates have been derived via ad-hoc approaches:
$\mathrm{Var}(\|Z\|_{2})\asymp\mathrm{tr}(\Sigma^{2})/\mathrm{tr}(\Sigma)$
(Valettas, 2019, p. 344) and
$\mathrm{Var}(\|Z\|_{\infty})\lesssim\sigma^{2}(\log\log 2d/\log
2d)^{1/4}+\sigma^{2}(\log\sum_{j,k}(2d)^{-2/(1+r_{jk})}/\log 2d)^{1/4}$, where
$\Sigma=(\sigma_{jk})_{j,k=1}^{d}$, $\sigma_{jj}=\sigma^{2}$ and
$r_{jk}=\sigma_{jk}/\sigma^{2}$ for all $1\leq j,k\leq d$ (Chatterjee, 2008,
Theorem 1.11). While less obvious than in the isotropic case, the takeaway of
the an-isotropic case is similar: $\|Z\|_{\infty}$ is superconcentrated
(provided that the correlations between the entries in the vector $Z$ are
“small”), whereas $\|Z\|_{2}$ is not superconcentrated (irrespective of the
correlations). Thus, by way of Theorem 1 this provides again a partial
explanation for why the Gaussian Monte Carlo hypothesis test based on the
$\ell_{\infty}$-norm is more accurate than the one based on the
$\ell_{2}$-norm.

###  4.2 Bias reduction via structured covariance matrix estimation

In the preceding section we argued that the rate at which
$\|\widehat{\Omega}_{n}-\Omega\|_{q\rightarrow
p}/\mathrm{Var}(\|Z\|_{p})\rightarrow 0$ impacts the accuracy of the (Monte
Carlo) bootstrap test via the Gaussian approximation error. We now show how to
reduce this error by leveraging additional structural information about the
population covariance $\Omega$. Note that reducing the Gaussian approximation
error is tantamount to relaxing the sufficient conditions of Lemmas 1–3. The
results in this section are far from comprehensive, they are meant to be
illustrative.

Throughout, let $\Omega=R\Sigma
R^{\prime}=(\omega_{jk})_{j,k=1}^{t}\in\mathbb{R}^{t\times t}$ and
$\widehat{\Omega}_{n}^{\mathrm{naive}}=n^{-1}\sum_{i=1}^{n}R(X_{i}-\bar{X}_{n})(X_{i}-\bar{X}_{n})^{\prime}R^{\prime}$
be the population and (naive) sample covariance matrix of
$RX_{1},\ldots,RX_{n}\in\mathbb{R}^{t}$, respectively. Consider the following
canonical structural assumption on $\Omega$:

######  Assumption 5 (Approximately sparse covariance matrix).

There exist constants $\gamma\in[0,1]$ and $R_{\gamma}>0$ such that
$\max_{1\leq j\leq t}\sum_{k=1}^{t}|\omega_{jk}|^{\gamma}\leq R_{\gamma}$.

Under Assumption 5 it is natural to estimate the covariance matrix via (hard)
thresholding of the sample covariance as
$\mathcal{T}_{\lambda}(\widehat{\Omega}_{n}^{\mathrm{naive}})$, where

$\displaystyle\mathcal{T}_{\lambda}(\Omega):=\big{(}\omega_{jk}\mathbf{1}\\{|\omega_{jk}|>\lambda\\}\big{)}_{j,k=1}^{t},$

and $\lambda>0$ a suitably chosen thresholding parameter. Since the bootstrap
test requires a positive semi-definite estimate of the covariance matrix, we
project this estimate onto the cone of positive semi-definite matrices. The
resulting positive semi-definite projection
$\mathcal{T}_{\lambda}^{+}(\widehat{\Omega}_{n}^{\mathrm{naive}})$ maintains
the same order of error as the original thresholding estimate (Avella-Medina
et al., 2018, p. 275).

We have the following result; it should be compared to the case $p=2$ in
Lemmas 1–3:

######  Lemma 2.

Let $X_{1},\ldots X_{n}\in\mathbb{R}^{d}$ be a simple random sample of random
vectors with mean $\mu_{n}\in\mathbb{R}^{d}$ and positive semi-definite
covariance matrix $\Sigma\in\mathrm{R}^{d\times d}$. Suppose that
$\Omega=R\Sigma R^{\prime}$ satisfies Assumption 5. Let
$\widehat{\Omega}_{n}:=\mathcal{T}_{\lambda}^{+}(\widehat{\Omega}_{n}^{\mathrm{naive}})$
for $\lambda>0$ to be specified below. Then, Assumptions 1–3 hold for exponent
$p=2$

  * (i)

if the $X_{i}$’s are sub-Gaussian, $\lambda\asymp\sqrt{(\log t)/n}\vee(\log
t)/n$, and

$\displaystyle\frac{r(\Omega)}{\sqrt{r(\Omega^{2})}}=o(n^{1/6})\quad{}\quad{}\mathrm{and}\quad{}\quad{}\frac{r(\Omega)}{r(\Omega^{2})}\frac{R_{\gamma}}{\omega_{(t)}^{2\gamma}}\lambda^{1-\gamma}=o(1);$

  * (ii)

if the $X_{i}$’s have a log-concave density, $\lambda\asymp\big{(}(\log
t)/n\big{)}^{1/4}\vee\sqrt{(\log t)/n}$, and

$\displaystyle\frac{r(\Omega)}{\sqrt{r(\Omega^{2})}}=o(n^{1/6})\quad{}\quad{}\mathrm{and}\quad{}\quad{}\frac{r(\Omega)}{r(\Omega^{2})}\frac{R_{\gamma}}{\omega_{(t)}^{\gamma-1}}\lambda^{1-\gamma}=o(1);$

  * (i)

if the $X_{i}$’s are heavy-tailed, $\lambda\asymp\big{(}(\log
t)/n\big{)}^{1/4}\vee\sqrt{(\log t)/n}$, and there exists $s>3$ such that

$\displaystyle
m_{2,3}\sqrt{\frac{r(\Omega)}{r(\Omega^{2})}}=o(n^{1/6}),\quad{}\frac{m_{2,s}}{m_{2,3}}=o(n^{1/3-1/s}),\quad{}\frac{r(\Omega)}{r(\Omega^{2})}\frac{R_{\gamma}}{\omega_{(t)}^{\gamma-1}}\lambda^{1-\gamma}=o(1),$

where $r(M):=\mathrm{tr}(M)/\|M\|_{op}$ is the effective rank of
$M\in\\{\Omega,\Omega^{2}\\}$ and
$m_{2,s}^{s}:=\mathbb{E}[\|X_{i}-\mu_{n}\|_{2}^{s}]/\|\Omega\|_{op}^{s/2}$.

Three comments are in order: First, the rates in above lemma are less
stringent than those in Lemmas 1–3, i.e. the thresholding estimator reduces
the Gaussian approximation error in an asymptotic sense. Numerical evidence
shows that the improvement is sizable even in finite samples (see Section 5).
Second, we only provide results for exponent $p=2$ because structural
assumptions on the population covariance $\Omega$ do not improve the rates for
exponent $p=\infty$. In fact, the rates in above lemma for $p=2$ reduce
approximately to the rates of $p=\infty$ in Lemmas 1–3 raised to the power
$1-\gamma$. Third, other structural assumptions on $\Omega$ such as bandedness
or low-rank result in similar improvements of the asymptotic rates. Above
lemma is merely intended as a proof of concept.

######  Remark 3 (Comparison with the _Gaussian multiplier bootstrap_ by
Chernozhukov et al. (2013)).

The bootstrap test is at least as accurate as the Gaussian multiplier test by
Chernozhukov et al. (2013). Moreover, if additional information about the
population covariance matrix is available, the bootstrap test can
substantially outperform the Gaussian multiplier test. To see this, recall
that the Gaussian multiplier bootstrap is defined as

$\displaystyle\varphi_{\alpha}^{m}(T_{n,p},X_{1},\ldots,X_{n}):=\mathbf{1}\left\\{T_{n,p}\geq
c^{m}_{p}(1-\alpha;X_{1},\ldots,X_{n})\right\\},$

where
$c^{m}_{p}(\alpha;X_{1},\ldots,X_{n})=\inf\left\\{u\in\mathbb{R}:\mathrm{P}(T_{n,p}^{m}\leq
u\mid X_{1},\ldots,X_{n})\geq\alpha\right\\}$ is the conditional
$\alpha$-quantile of the Gaussian multiplier proxy statistic

$\displaystyle
T_{n,p}^{m}:=\left\|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\xi_{i}R(X_{i}-\bar{X}_{n})\right\|_{p},\quad\quad\xi_{1},\ldots,\xi_{n}\sim_{iid}N(0,1),$

and the Gaussian multipliers $\xi_{1},\ldots,\xi_{n}$ are independent of the
data $X_{1},\ldots,X_{n}$. Since $n^{-1/2}\sum_{i=1}^{n}\xi_{i}\\\
R(X_{i}-\bar{X}_{n})\mid X_{1},\ldots,X_{n}\sim
N(0,\widehat{\Omega}_{n}^{\mathrm{naive}})$, the Gaussian multiplier test
$\varphi_{\alpha}^{m}(T_{n,p},X_{1},\ldots,X_{n})$ is equivalent to the
bootstrap test
$\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n}^{\mathrm{naive}})$. Thus, the
Gaussian multiplier bootstrap does not offer a way to leverage structural
information about the population covariance matrix.

###  4.3 Improved efficiency via spherical bootstrapping

As detailed in Section 4.1, we can, in theory, reduce the bias of the Monte
Carlo bootstrap test by using a more efficient non-Gaussian sampling scheme
based on highly superconcentrated proxy statistics. When implementing this
idea, we have to trade off two conflicting goals: On the one hand, we need to
find a non-Gaussian distribution such that the proxy statistics are “more”
superconcentrated than their Gaussian counterparts. On the other hand, the
non-Gaussian distribution must not be “too far” from a Gaussian distribution
with mean zero and covariance $\widehat{\Omega}_{n}$ or else our proof
strategy based on Gaussian proxy statistics fails (viz. Gaussian approximation
and Gaussian comparison, see Section 4.1).

Formalizing this bias-variance trade-off as a constrained optimization problem
and finding the optimal non-Gaussian distribution is a formidable task. Here,
we only propose one possible solution to this problem; namely, the _spherical
bootstrap hypothesis test_ defined as

$\displaystyle\varphi_{\alpha}^{s}(T_{n,p},\widehat{\Gamma}_{n}):=\mathbf{1}\left\\{T_{n,p}\geq
c^{s}_{p}(1-\alpha;\widehat{\Gamma}_{n})\right\\},$

where
$c^{s}_{p}(\alpha;\widehat{\Gamma}_{n})=\inf\left\\{u\in\mathbb{R}:\mathrm{P}(T_{n,p}^{s}\leq
u\mid X_{1},\ldots,X_{n})\geq\alpha\right\\}$ is the conditional
$\alpha$-quantile of the _spherical proxy statistic_

$\displaystyle
T_{n,p}^{s}:=\sqrt{s}\|\widehat{\Gamma}_{n}U^{(s)}\|_{p},\quad{}\quad{}U^{(s)}\sim\mathrm{Unif}(\mathbb{S}^{s-1}),\quad{}\quad{}\widehat{\Gamma}_{n}\widehat{\Gamma}_{n}^{\prime}=\widehat{\Omega}_{n}\in\mathbb{R}^{t\times
t},$

where $\mathbb{S}^{s-1}$ is the $\ell_{2}$-norm unit sphere in
$\mathbb{R}^{s}$, $2\leq s\leq t$, and $\widehat{\Omega}_{n}$ is a positive
semi-definite estimate of $\Omega$. The corresponding feasible _spherical
Monte Carlo bootstrap hypothesis test_ is

$\displaystyle\varphi_{\alpha,B}^{s}(T_{n,p},\widehat{\Gamma}_{n}):=\mathbf{1}\left\\{T_{n,p}\geq
c^{s}_{p,B}(1-\alpha;\widehat{\Gamma}_{n})\right\\},$

where $c^{s}_{p,B}(1-\alpha;\widehat{\Gamma}_{n})$ denotes the
$\lfloor(1-\alpha)B\rfloor$th order statistic of the random sample
$T_{n,p,1}^{s},\ldots,T_{n,p,B}^{s}$ with
$T_{n,p,b}^{s}:=\sqrt{s}\|\widehat{\Gamma}_{n}U_{b}^{(s)}\|_{p}$ and
$U_{b}^{(s)}\sim\mathrm{Unif}(\mathbb{S}^{s-1})$ for all $b=1,\ldots,B$.

We expect the spherical bootstrap procedure to perform well for two reasons:
First, the first two moments of $\sqrt{s}\widehat{\Gamma}_{n}U^{(s)}$ and
$Z_{n}\mid X_{1},\ldots,X_{n}\sim N(0,\widehat{\Omega}_{n})$ coincide. Thus,
the spherical proxy statistic $T_{n,p}^{s}$ and the Gaussian proxy statistic
$T_{n,p}^{*}$ are close in Kolmogorov distance (Theorem 1, Appendix A.1).
Second, the entries in the vector $U^{(s)}$ are “negatively associated”, i.e.
we deduce from Theorem 3.3 in Fang et al. (1990) that all moments of entries
in $U^{(s)}$ are non-positively correlated. This suggests that
$\|\widehat{\Gamma}_{n}U^{(s)}\|_{p}$ is more superconcentrated than
$\|Z_{n}\|_{p}\overset{d}{=}\|\widehat{\Gamma}_{n}Z^{(s)}\|_{p}$, where
$Z^{(s)}\sim N(0,I_{s})$ (for a more rigorous statement, see Lemma 25 in
Appendix A.4).

The following two theorems confirm that this heuristic reasoning is indeed
correct. The main takeaway is that the spherical Monte Carlo bootstrap test
has asymptotic correct size, is consistent against high-dimensional
alternatives, has the same power properties as the (Gaussian) bootstrap test,
and has smaller (asymptotic) Monte Carlo error than the (Gaussian) Monte Carlo
bootstrap test. Thus, for all practical purposes, we recommend using the
spherical bootstrapping procedure.

######  Theorem 3.

Suppose that $\|\widehat{\Gamma}_{n}\|_{2\rightarrow
p}=o_{p}\left(s^{1/4}\sqrt{\mathrm{Var}(\|Z\|_{p})}\right)$. Then, Theorems 4
and 9 and Propositions 7, 1–5 hold with the spherical bootstrap test
$\varphi_{\alpha}^{s}(T_{n,p},\widehat{\Gamma}_{n})$ substituted for the
(Gaussian) bootstrap test $\varphi_{\alpha}(T_{n,p},\widehat{\Omega}_{n})$.

######  Theorem 4 (Spherical Monte Carlo Error).

Let $M_{n,p}={\mathrm{ess}\sup}_{z,\mu}f_{n,p,\mu}(z)$, where $f_{n,p,\mu}$ is
the density of the test statistics $T_{n,p}$ when the $X_{i}$’s have mean
$\mu$. Then, for all $1\leq p\leq\infty$,

$\displaystyle\sup_{\alpha\in(0,1)}\sup_{\mu}\left|\mathrm{E}_{\mu}\left[\varphi_{\alpha,B}^{s}(T_{n,p},\widehat{\Gamma}_{n})\right]-\mathrm{E}_{\mu}\left[\varphi_{\alpha}^{s}(T_{n,p},\widehat{\Gamma}_{n})\right]\right|$
$\displaystyle\lesssim B^{-1/2}s^{-1}(\log
B)\sqrt{M_{n,p}^{2}\mathrm{E}\big{[}\|\widehat{\Gamma}_{n}\|_{2\rightarrow
p}^{2}\big{]}},$

where $\lesssim$ hides an absolute constant independent of $p,n,d,t,s,B$, and
the distribution of the $X_{i}$’s.

######  Remark 4 (On size, consistency, and power of the spherical Monte Carlo
bootstrap test).

Combined with Theorem 3 the bound on the spherical Monte Carlo error implies
that Theorems 4 and 9 and Propositions 7, 1–5 to hold if we substitute the
spherical Monte Carlo Bootstrap test
$\varphi_{\alpha,B}^{s}(T_{n,p},\widehat{\Gamma}_{n})$ for the infeasible
$\varphi_{\alpha}^{s}(T_{n,p},\widehat{\Gamma}_{n})$ and choose $B\gtrsim
n\vee\left(s^{-2}M_{n,p}^{2}\mathrm{E}\big{[}\|\widehat{\Gamma}_{n}\|_{2\rightarrow
p}^{2}]\right)^{1+\gamma}$, $\gamma>0$ arbitrary.

######  Remark 5 (On the rank $s$ of
$\widehat{\Gamma}_{n}\in\mathbb{R}^{t\times s}$).

If we estimate the covariance matrix $\Omega\in\mathbb{R}^{t\times t}$ via the
sample analogue
$\widehat{\Omega}_{n}=n^{-1}\sum_{i=1}^{n}R(X_{i}-\bar{X}_{n})(X_{i}-\bar{X}_{n})^{\prime}R^{\prime}$,
then there exists $\widehat{\Gamma}_{n}\in\mathbb{R}^{t\times(n-1)}$ such that
$\widehat{\Gamma}_{n}\widehat{\Gamma}_{n}^{\prime}=\widehat{\Omega}_{n}$.
Thus, in this case rank $s=n-1$ and the condition in Theorem 3 is typically
satisfied whenever Assumption 3 holds. Moreover, for $s=n-1$ the upper bound
in Theorem 4 is (asymptotically) significantly smaller than the corresponding
upper bound on the (Gaussian) Monte Carlo error of Theorem 1. If we estimate
$\Omega\in\mathbb{R}^{t\times t}$ using a more sophisticated estimator (viz.
Section 4.2), rank $s$ may be smaller or larger than $n-1$. The smaller rank
$s$, the smaller the variance reduction and, thus, the smaller the gains of
using the spherical Monte Carlo bootstrap test. Evidence from simulations
supports this theoretical insight.

######  Remark 6 (Comparison with the _non-Gaussian multiplier bootstrap_ by
Deng and Zhang (2020)).

The rationale for the non-Gaussian (multiplier) bootstrap procedures in Deng
and Zhang (2020) is very different from the one for the spherical bootstrap:
Deng and Zhang (2020) do not consider the Monte Carlo error of implementing a
bootstrap procedure. Instead, they are concerned with reducing the theoretical
biases corresponding to Gaussian approximation and Gaussian comparison error.
By matching up to five moments of the distributions of test and proxy
statistic, they achieve some improvements in the rates at which these
theoretical biases vanish (compared to Chernozhukov et al., 2013). However,
the most recent results in Chernozhukov et al. (2019, 2020) show that the
Gaussian multiplier bootstrap can achieve the same (or even better) rates.

To the best of our knowledge, we are the first to propose a method to mitigate
the Monte Carlo error associated with a high-dimensional bootstrap procedure.
The practical gains and the theoretical justification of the spherical
bootstrap hold regardless of any (future) improvements in the rates at which
Gaussian approximation and Gaussian comparison errors vanish.

##  5 Numerical Experiments

###  5.1 Monte Carlo setup

We study the performance of the bootstrap test and its modifications in four
different data generating processes (DGPs). To get our main points across as
concise as possible, we consider testing problem 1 with
$R=I_{d}\in\mathbb{R}^{d\times d}$ and $r=\mathbf{0}\in\mathbb{R}^{d}$ (i.e.
we test $H_{0}:\>\mu_{0}=\mathbf{0}$ versus $H_{1}:\>\mu_{0}\neq\mathbf{0}$)
and only report results for test statistics with exponents
$p\in\\{2,\infty\\}$. In DGPs 1–3 we generate vectors
$X_{i}=(X_{i1},\ldots,X_{id})^{\prime}$ via the Gaussian copula model

$\displaystyle X_{ij}=F^{-1}\left(\Phi(Y_{ij})\right)-1,\quad\quad
Y_{i}=(Y_{i1},\ldots,Y_{id})^{\prime}\sim N(0,\Sigma),\quad\quad 1\leq j\leq
d,\quad\quad 1\leq i\leq n,$

where the random vectors $Y_{1},\ldots,Y_{n}\in\mathbb{R}^{d}$ are independent
copies of each other and $F$ denotes the cdf of the Gamma distribution with
shape and rate parameters $\alpha=\beta=1$. The parameters $\alpha,\beta$ are
chosen such that the Gamma distribution has mean and variance equal to one,
skew equal to two, and an excess kurtosis of six. This ensures that the
$X_{i}$’s are distributed markedly different from symmetric Gaussian or
spherical random vectors and hence makes the bootstrap procedures a non-
trivial exercise. We consider three different covariance structures for
$\Sigma=(\Sigma_{jk})_{j,k=1}^{d}$: equicorrelated entries with
$\Sigma_{jk}=0.8+0.2\times\mathbf{1}\\{j=k\\}$ (non-mixing dependencies) in
DGP 1, weakly dependent entries with $\Sigma_{jk}=0.8^{|j-k|}$ (Toeplitz
structure of an autoregressive process of order one) in DGP 2, and
$m$-dependent entries with $\Sigma_{jk}=\max\left\\{1-|j-k|/\lceil
d^{1/3}/2\rceil,0\right\\}$ (banded structure of an moving average process
with 2, 3, and 4 lags for dimension 100, 500, 1000, respectively) in DGP 3. In
DGP 4 we generate vectors $X_{i}\in\mathbb{R}^{d}$ from a multivariate
$t(4)$-distribution. We choose 4 degrees of freedom because this guarantees
that the first $3+\delta$ moments exist, which is just enough for our
theoretical results to hold. For the covariance matrix $\Sigma$ we choose the
same Toeplitz structure as in DGP 2.

Under DGP 1, 2, and 4 we estimate the covariance matrix via the naive sample
covariance. Under DGP 3 we use a thresholding estimate of the covariance
matrix. Simply thresholding the sample covariance matrix and projecting it
onto the cone of positive semi-definite matrices as suggested in Section 4.2
results in large biases. We therefore use the Cholesky-based regularization
approach by Rothman et al. (2010) which is guaranteed to produce positive
semi-definite estimates. To reduce the computational burden, we assume that
the correct bandwidth is known.

For DGPs 1–3 we report results for dimensions $d\in\\{100,500,1000\\}$ and
sample sizes $n\in\\{20,50,100\\}$; for DGP 4 we present results for
dimensions $d\in\\{500,1000,5000\\}$ and $n\in\\{20,50,100\\}$. We implement
all bootstrap procedures with $B=5000$ bootstrap samples. The QQ-plots in
Section 5.2 compare ‘nominal’ and ‘actual’ sizes of different bootstrap tests.
We choose 99 ‘nominal’ sizes $\alpha\in\\{0.01,0.02,\ldots,0.99\\}$ and
compute the corresponding ‘actual’ sizes as the relative frequency of false
positives in 1000 independent Monte Carlo samples.

###  5.2 Results

In Figures 1 and 2 we compare the accuracy of the basic Gaussian bootstrap
test from Section 1.2 and the bias-corrected spherical bootstrap test from
Section 4.3 for exponents $p\in\\{2,\infty\\}$. In the case of equicorrelated
design (DGP 1) both tests are rather accurate, especially for the (important)
small quantiles $\alpha\in\\{0.01,\ldots,0.1\\}$. The test statistic based on
the $\ell_{2}$-norm appears to be slightly more accurate that the test
statistic based on the $\ell_{\infty}$-norm. For Toeplitz matrices (DGP 2) the
spherical bootstrap substantially outperforms the Gaussian bootstrap across
all quantile levels $\alpha\in\\{0.01,0.02,\ldots,0.99\\}$ for test statistic
based on the $\ell_{2}$-norm. For test statistics based on the
$\ell_{\infty}$-norm the spherical and Gaussian bootstrap test perform
similarly for small quantiles $\alpha$, but for large quantiles $\alpha>0.5$
the spherical test is significantly more conservative than the Gaussian test.

In Figure 3 we show results for the Modified bootstrap test (Section 3.1)
under DGP 2 and the self-normalized test (Section 3.2) under DGP 4. In both
cases, we observe that the spherical bootstrap test is more accurate than the
Gaussian test. As predicted by our theory, the validity of the (spherical)
self-normalized test is virtually unaffected by dimension. Notice that the
fact that the spherical bootstrap performs so well for the self-normalized
test is not entirely trivial: while each normalized vector $\widetilde{X}_{i}$
follows a spherical distribution, the (rescaled) sum of of spherical random
vectors is not spherically distributed.

In Figure 4 we compare the accuracy of the basic Gaussian bootstrap test, the
bias-corrected spherical bootstrap test, and the Gaussian bootstrap test under
DGP 3 with thresholded covariance matrix (Section 4.2) for exponents
$p\in\\{2,\infty\\}$. In accordance with our theory, the thresholded
covariance matrix does indeed reduce the bias. However, it results in a
conservative test whose actual level is consistently less than the targeted
nominal level. Among the three tests, the spherical bootstrap test is still
the most accurate one.

Figure 1: QQ-Plots of ‘nominal’ and ‘actual’ sizes of the Gaussian and
spherical bootstrap test under equicorrelated design (DGP 1). Panel (A) shows
simulation results for the bootstrap test with exponent $p=2$, panel (B) shows
simulation results for the bootstrap test with exponent $p=\infty$.  Figure 2:
QQ-Plots of ‘nominal’ and ‘actual’ sizes of the Gaussian and spherical
bootstrap test under Toeplitz design (DGP 2). Panel (A) shows simulation
results for the bootstrap test with exponent $p=2$, panel (B) shows simulation
results for the bootstrap test with exponent $p=\infty$.  Figure 3: Panel (A)
shows QQ-Plots of ‘nominal’ and ‘actual’ sizes of the Gaussian and spherical
modified test bootstrap Section 3.1 under Toeplitz design (DGP 2). Panel (B)
shows shows QQ-Plots of ‘nominal’ and ‘actual’ sizes of the Gaussian and
spherical self-normalized test from Section 3.2 for data with
$t(4)$-distribution (DGP 4). Figure 4: QQ-Plots of ‘nominal’ and ‘actual’
sizes of the Gaussian bootstrap test, the Gaussian bootstrap test with
thresholded covariance matrix, and the spherical bootstrap test for banded
covariance design (DGP 3). Panel (A) shows simulation results for the
bootstrap test with exponent $p=2$, panel (B) shows simulation results for the
bootstrap test with exponent $p=\infty$.

##  6 Discussion

In Section 1.1 we raised four questions about the bootstrap hypothesis test
that motivated us to write this paper. Synthesizing our findings from Sections
2-5 we can now answer these questions:

  * •

_On the choice of exponent $p\geq 1$ and the power of the test:_ Our analysis
indicates that exponents $p\in\\{2,\log t\\}$ are the most relevant ones, with
$\varphi_{\alpha}(T_{n,2},\widehat{\Omega}_{n})$ having high power against
dense alternatives and $\varphi_{\alpha}(T_{n,\log t},\widehat{\Omega}_{n})$
high power against sparse alternatives. Since $\ell_{\infty}$\- and
$\ell_{\log t}$-norms are equivalent for $t$-dimensional vectors, this result
is in line with the literature (Cai et al., 2014; Fan et al., 2015; He et al.,
2021); however, for high-dimensional bootstrap tests this question was
unsettled until now. Since the high-dimensional bootstrap tests do not satisfy
a CLT (but only the weaker Gaussian approximability property, see Appendices
A.1 and A.3), the power analysis is based on the Bahadur slope of the tests.

  * •

_Leveraging distributional assumptions:_ Our discussion of the self-normalized
bootstrap test for elliptically distributed data shows that it is indeed
possible to exploit distributional assumptions in way that dramatically
improves theoretical and empirical performance of the basic bootstrap test. If
one accepts the stronger distributional assumption, then the additional
sufficient condition for type 1 error control and consistency is extremely
mild and, in many cases, holds for arbitrarily large dimensions $d,t$. It
might well be possible to develop similar tests for other data generating
processes (see Wang et al., 2015).

  * •

_The effect of the estimated covariance matrix:_ The effect of the estimated
covariance matrix on the validity of the bootstrap is fully captured in
Assumption 3. Intuitively, this assumption says that the estimation error
$\|\widehat{\Omega}_{n}-\Omega\|_{q\rightarrow p}$ has to be asymptotically
negligible compared to the variance of the Gaussian proxy statistic
$\|Z\|_{p}$. If the dimensions $d,t$ are fixed, $\mathrm{Var}(\|Z\|_{p})$
plays no role. However, typically, $\mathrm{Var}(\|Z\|_{p})\downarrow 0$
whenever $d,t\rightarrow\infty$ and $p>2$ (see Appendix A.2); hence,
Assumption 3 can be difficult to satisfy in high dimensions.

  * •

_Utilizing insights from high-dimensional probability theory to mitigate
biases:_ The bias decomposition of the (Monte Carlo) bootstrap test exposes
three sources of potential bias: a Gaussian approximation error, a Gaussian
comparison error, and a Monte Carlo error. The first error is unavoidable and
can only be reduced by collecting more samples $n$. The second error can be
mitigated by sampling from Gaussian proxy statistics $T_{n,p}^{*}$ that employ
structured covariance matrix estimators to leverage low rank, bandedness, or
approximate sparsity of the population covariance. The third error can be
attenuated by improving the efficiency of the Monte Carlo sampling procedure.
We propose the spherical bootstrap which is motivated by hypercontractivity
properties of $\ell_{p}$-norms of high-dimensional spherically distributed
random vectors. Simulation studies show that both bias correction schemes tend
to produce bootstrap tests that have better control of type 1 error.

We conclude with several comments and questions that arose while working on
this paper:

  * •

_How much of the theory carries over to statistics other than
$\ell_{p}$-norms?_ The practical power of the bootstrap principle is its
ability to perform inference in complicated settings involving highly non-
linear statistics. While $\ell_{p}$-norms are non-linear functions, they can
also be expressed as simple suprema over their respective dual-norm unit
balls. This duality relationship allows us to apply the general theory from
Giessing (2023a, b) and drives the majority of the theoretical results. This
general theory can be applied to a virtually any statistic and function class.
However, it remains a major challenge to derive sharp upper bounds on the
third moments of arbitrary statistics and non-trivial lower bounds on the
variances of the corresponding Gaussian proxy statistics.

  * •

_Why do all rates (in this paper) depend on $\mathrm{Var}(\|Z\|_{p})$?_ The
fact that all rates in this paper (Appendices A.1 and A.3) feature the
variance of the Gaussian proxy statistic $\|Z\|_{p}$ is intriguing and
deserves further investigation. There appears to be a connection between
“super-concentration” (Chatterjee, 2014), “anti-concentration” (Le Cam, 1986),
and “universality laws” (e.g. classical CLTs, high-dimensional Gaussian
approximation results, or Mossel et al.’s (2010) multilinear invariance
principle). For a brief discussion we refer to Appendix A.2.

  * •

_What are necessary conditions for bootstrap consistency in high dimensions?_
We know that in the classical, low-dimensional setting the (empirical or wild)
bootstrap is valid if and only if the involved function classes are Donsker
(Mammen, 1992; van der Vaart and Wellner, 1996). This and the bias
decomposition in Section 4.1 seem to suggest that Gaussian approximability of
the function classes might not only be a sufficient but also a necessary
condition for bootstrap consistency in high-dimensional settings. However,
this need not be true. In fact, the proofs in the theoretical companion papers
Giessing (2023a, b) use Gaussianity only twice, once when applying an anti-
concentration argument and a second time when invoking Stein’s lemma. Other
distributions (might) satisfy similar properties that can be used to replace
the original arguments.

  * •

_What are the connections between Efron’s empirical, the wild, and the
spherical bootstrap?_ Numerical results from simulations (not included in this
paper) show that Efron’s empirical bootstrap is valid for
$\ell_{p}$-statistics with large exponents $p\gtrsim\log d$ but fails for
small exponents $p\lesssim\log d$. This is broadly in line with results by
Chernozhukov et al. (2013) and El Karoui and Purdom (2018). We can contribute
the following insight: Recall that Efron’s empirical bootstrap can be
interpreted as a multiplier bootstrap with multinomial multipliers (van der
Vaart and Wellner, 1996, p.345). The multipliers are thus negatively
correlated and have correlation coefficients $-n^{-1}$. When bootstrapping the
$\ell_{p}$-norm with $p\gtrsim\log d$ this results in an additional bias term
of order $O(n^{-1})$ (modulo $(\log d)$-terms); whereas, when bootstrapping
the $\ell_{p}$-norm with $p\lesssim\log d$ this yields a non-vanishing bias
term of order $O(1)$. El Karoui and Purdom (2018) arrive at a similar
conclusion and therefore advocate using uncorrelated multipliers in high
dimensions. Our spherical bootstrap can now be viewed as just another
multiplier bootstrap with uncorrelated multipliers. However, unlike the
multipliers proposed by El Karoui and Purdom (2018) ours are dependent which,
together with the hypercontractivity of $\ell_{p}$-norms, improves the
efficiency and accuracy of the bootstrap procedure. Numerical results (not
included in this paper) show that even in classical, low-dimensional settings
the spherical bootstrap tends to outperform Efron’s empirical bootstrap.
Existing theory on exchangeable multiplier bootstrap procedures does not cover
the spherical bootstrap because it requires non-negative multipliers (van der
Vaart and Wellner, 1996, p. 353ff). In this paper we have focused on validity
and consistency of the spherical bootstrap procedure; a more comprehensive and
comparative study of its efficiency and accuracy would be highly desirable.

## Acknowledgement

Alexander Giessing’s research is supported by NSF grant DMS-2310578, Jianqing
Fan’s research by ONR grant N00014-19-1-2120 and the NSF grants DMS-2052926,
DMS-2053832, and DMS-2210833.

## References

  * ben (2003) On the dependence of the Berry–Esseen bound on dimension.  _Journal of Statistical Planning and Inference_ , 113(2):385 – 402, 2003.  ISSN 0378-3758.
  * Anderson (1955) T. W. Anderson.  The integral of a symmetric unimodal function over a symmetric convex set and some probability inequalities.  _Proceedings of the American Mathematical Society_ , 6(2):170–176, 1955.
  * Avella-Medina et al. (2018) M. Avella-Medina, H. S. Battey, J. Fan, and Q. Li.  Robust estimation of high-dimensional covariance and precision matrices.  _Biometrika_ , 105(2):271–284, 03 2018.
  * Bai and Saranadasa (1996) Z. Bai and H. Saranadasa.  Effect of high dimension: by an example of a two sample problem.  _Statistica Sinica_ , 6(2):311–329, 1996.
  * Beran (1987) R. Beran.  Prepivoting to reduce level error of confidence sets.  _Biometrika_ , 74(3):457–468, 1987.
  * Biau and Mason (2015) G. Biau and D. M. Mason.  High-dimensional $p$-norms.  In _Mathematical Statistics and Limit Theorems_ , pages 21–40. Springer, 2015.
  * Boucheron et al. (2013) S. Boucheron, G. Lugosi, and P. Massart.  _Concentration Inequalities: A Nonasymptotic Theory of Independence_.  Oxford University Press, Oxford, 2013.
  * Bullen et al. (1988) P. Bullen, D. Mitrinovic, M. Vasic, and P. Vasić.  _Means and Their Inequalities_.  Springer, 1988.
  * Cacoullos (1982) T. Cacoullos.  On Upper and Lower Bounds for the Variance of a Function of a Random Variable.  _The Annals of Probability_ , 10(3):799 – 809, 1982.
  * Cai and Liu (2011) T. Cai and W. Liu.  Adaptive Thresholding for Sparse Covariance Matrix Estimation.  _Journal of the American Statistical Association_ , 106(494):672–684, 2011.
  * Cai et al. (2010) T. T. Cai, C.-H. Zhang, and H. H. Zhou.  Optimal rates of convergence for covariance matrix estimation.  _Annals of Statistics_ , 38(4):2118–2144, 08 2010\.
  * Cai et al. (2014) T. T. Cai, W. Liu, and Y. Xia.  Two-sample test of high dimensional means under dependence.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 76(2):349–372, 2014.
  * Chatterjee (2008) S. Chatterjee.  Chaos, concentration, and multiple valleys, 2008.
  * Chatterjee (2014) S. Chatterjee.  _Superconcentration and Related Topics_.  Springer Monographs in Mathematics. Springer, 2014.
  * Chen et al. (2010) S. X. Chen, Y.-L. Qin, et al.  A two-sample test for high-dimensional data with applications to gene-set testing.  _The Annals of Statistics_ , 38(2):808–835, 2010\.
  * Chen (2018) X. Chen.  Gaussian and bootstrap approximations for high-dimensional u-statistics and their applications.  _Ann. Statist._ , 46(2):642–678, 04 2018.
  * Chernozhukov et al. (2013) V. Chernozhukov, D. Chetverikov, and K. Kato.  Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors.  _The Annals of Statistics_ , 41(6):2786–2819, 12 2013.
  * Chernozhukov et al. (2014a) V. Chernozhukov, D. Chetverikov, and K. Kato.  Anti-concentration and honest, adaptive confidence bands.  _The Annals of Statistics_ , 42(5):1787–1818, 2014a.
  * Chernozhukov et al. (2014b) V. Chernozhukov, D. Chetverikov, and K. Kato.  Gaussian approximation of suprema of empirical processes.  _The Annals of Statistics_ , 42(4):1564–1597, 08 2014b.
  * Chernozhukov et al. (2015) V. Chernozhukov, D. Chetverikov, and K. Kato.  Comparison and anti-concentration bounds for maxima of gaussian random vectors.  _Probability Theory and Related Fields_ , 162(1):47–70, Jun 2015.
  * Chernozhukov et al. (2017) V. Chernozhukov, D. Chetverikov, and K. Kato.  Central limit theorems and bootstrap in high dimensions.  _The Annals of Probability_ , 45(4):2309–2352, 07 2017.
  * Chernozhukov et al. (2019) V. Chernozhukov, D. Chetverikov, K. Kato, and Y. Koike.  Improved central limit theorem and bootstrap approximations in high dimensions.  _arXiv preprint, arXiv:1912.10529_ , 12 2019.
  * Chernozhukov et al. (2020) V. Chernozhukov, D. Chetverikov, and Y. Koike.  Nearly optimal central limit theorem and bootstrap approximations in high dimensions.  2020\.
  * Cordero-Erausquin and Ledoux (2012) D. Cordero-Erausquin and M. Ledoux.  _Hypercontractive Measures, Talagrand’s Inequality, and Influences_ , pages 169–189.  Springer, 2012.
  * Cule and Samworth (2010) M. Cule and R. Samworth.  Theoretical properties of the log-concave maximum likelihood estimator of a multidimensional density.  _Electronic Journal of Statistics_ , 4(none):254 – 270, 2010.
  * Davison et al. (1986) A. C. Davison, D. V. Hinkley, and E. Schechtman.  Efficient bootstrap simulation.  _Biometrika_ , 73(3):555–566, 1986.
  * Deng and Zhang (2020) H. Deng and C.-H. Zhang.  Beyond gaussian approximation: Bootstrap for maxima of sums of independent random vectors.  _arXiv preprint, arXiv:1705.09528_ , 2020.
  * Ding et al. (2015) J. Ding, R. Eldan, and A. Zhai.  On multiple peaks and moderate deviations for the supremum of a Gaussian field.  _The Annals of Probability_ , 43(6):3468 – 3493, 2015.
  * Dudley (2014) R. Dudley.  _Uniform Central Limit Theorems_.  Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2014.
  * El Karoui and Purdom (2018) N. El Karoui and E. Purdom.  Can We Trust the Bootstrap in High-dimensions? The Case of Linear Models.  _Journal of Machine Learning Research_ , 19(5):1–66, 2018.
  * Fan et al. (2015) J. Fan, Y. Liao, and J. Yao.  Power enhancement in high-dimensional cross-sectional tests.  _Econometrica_ , 83(4):1497–1541, 2015.
  * Fang et al. (1990) K. Fang, S. Kotz, and K. Ng.  _Symmetric Multivariate and Related Distributions_.  Chapman & Hall, 1990.
  * Fang and Koike (2021) X. Fang and Y. Koike.  High-dimensional central limit theorems by Stein’s method.  _The Annals of Applied Probability_ , 31(4):1660 – 1686, 2021.
  * Foucart and Rauhut (2013) S. Foucart and H. Rauhut.  _A Mathematical Introduction to Compressive Sensing_.  Applied and Numerical Harmonic Analysis. Springer New York, 2013.
  * Fournier and Guillin (2015) N. Fournier and A. Guillin.  On the rate of convergence in wasserstein distance of the empirical measure.  _Probability Theory and Related Fields_ , 162(3):707–738, 2015.
  * Gagliardini et al. (2016) P. Gagliardini, E. Ossola, and O. Scaillet.  Time-varying risk premium in large cross-sectional equity data sets.  _Econometrica_ , 84(3):985–1046, 2016.
  * Giessing (2023a) A. Giessing.  Anti-Concentration of Suprema of Gaussian Processes and Gaussian Order Statistics.  _Working Paper_ , 2023a.
  * Giessing (2023b) A. Giessing.  Gaussian and Bootstrap Approximations for Empirical Processes.  _Working Paper_ , 2023b.
  * Giné and Zinn (1990) E. Giné and J. Zinn.  Bootstrapping General Empirical Measures.  _The Annals of Probability_ , 18(2):851 – 869, 1990.
  * Hall (1986) P. Hall.  On the Bootstrap and Confidence Intervals.  _The Annals of Statistics_ , 14(4):1431 – 1452, 1986.
  * Hall and Martin (1988) P. Hall and M. A. Martin.  On bootstrap resampling and iteration.  _Biometrika_ , 75(4):661–671, 1988.
  * Hardy et al. (1988) G. Hardy, J. Littlewood, and G. Pólya.  _Inequalities_.  Cambridge Mathematical Library. Cambridge University Press, 1988.
  * He et al. (2021) Y. He, G. Xu, C. Wu, and W. Pan.  Asymptotically independent U-statistics in high-dimensional testing.  _The Annals of Statistics_ , 49(1):154 – 181, 2021.
  * Houdre and Perez-Abreu (1995) C. Houdre and V. Perez-Abreu.  Covariance identities and inequalities for functionals on wiener and poisson spaces.  _The Annals of Probability_ , 23(1):400–419, 1995\.
  * Huang et al. (2022) Y. Huang, C. Li, R. Li, and S. Yang.  An overview of tests on high-dimensional means.  _Journal of Multivariate Analysis_ , 188:104813, 2022.
  * Koltchinskii and Lounici (2017) V. Koltchinskii and K. Lounici.  Concentration inequalities and moment bounds for sample covariance operators.  _Bernoulli_ , 23(1):110 – 133, 2017.
  * Le Cam (1986) L. Le Cam.  _Asymptotic Methods in Statistical Decision Theory_.  Springer Series in Statistics. Springer New York, 1986.
  * Ledoux (2001) M. Ledoux.  _The concentration of measure phenomenon_.  AMS, 2001.
  * Ledoux and Talagrand (1991) M. Ledoux and M. Talagrand.  _Probability in Banach Spaces_.  Springer, 1991.
  * Lopes et al. (2020) M. E. Lopes, Z. Lin, and H.-G. Müller.  Bootstrapping max statistics in high dimensions: Near-parametric rates under weak variance decay and application to functional and multinomial data.  _Annals of Statistics_ , 48(2):1214–1229, 04 2020\.
  * Lytova and Tikhomirov (2019) A. Lytova and K. Tikhomirov.  The variance of the $\ell^{n}_{p}$-norm of the Gaussian vector, and Dvoretzky’s theorem.  _St. Petersburg Mathematical Journal_ , (30):699–722, 2019.
  * Major (1978) P. Major.  On the invariance principle for sums of independent identically distributed random variables.  _Journal of Multivariate Analysis_ , 8(4):487–517, 1978.
  * Mammen (1992) E. Mammen.  Bootstrap, wild bootstrap, and asymptotic normality.  93(4):439–455, 1992.
  * Mammen (1993) E. Mammen.  Bootstrap and Wild Bootstrap for High Dimensional Linear Models.  _Ann. Statist._ , 21(1):255–285, 03 1993.
  * Manolio et al. (2009) T. A. Manolio, F. S. Collins, N. J. Cox, D. B. Goldstein, L. A. Hindorff, D. J. Hunter, M. I. McCarthy, E. M. Ramos, L. R. Cardon, A. Chakravarti, J. H. Cho, A. E. Guttmacher, A. Kong, L. Kruglyak, E. Mardis, C. N. Rotimi, M. Slatkin, D. Valle, A. S. Whittemore, M. Boehnke, A. G. Clark, E. E. Eichler, G. Gibson, J. L. Haines, T. F. C. Mackay, S. A. McCarroll, and P. M. Visscher.  Finding the missing heritability of complex diseases.  _Nature_ , 4611(7265):747–753, 2009.
  * Mathur et al. (2018) R. Mathur, D. Rotroff, J. Ma, A. Shojaie, and A. Motsinger-Reif.  Gene set analysis methods: a systematic comparison.  _BioData Mining_ , 11(1), 2018.
  * McNeil et al. (2005) A. McNeil, R. Frey, and P. Embrechts.  _Quantitative Risk Management: Concepts, Techniques and Tools - Revised Edition_.  Princeton University Press, 2005.
  * Mendelson (2010) S. Mendelson.  Empirical processes with a bounded $\psi_{1}$ diameter.  _Geometric and Functional Analysis_ , 20(4):988–1027, 2010.
  * Mossel et al. (2010) E. Mossel, R. O’Donnell, and K. Oleszkiewicz.  Noise stabiklity on functions with low influences: invariance and optimality.  _Annals of Mathematics_ , 171(1):295–341, 2010\.
  * Paouris and Valettas (2018) G. Paouris and P. Valettas.  On Dvoretzky’s theorem for subspaces of ${L}_{p}$.  _Journal of Functional Analysis_ , 275(8):2225 – 2252, 2018.  ISSN 0022-1236.
  * Rothman et al. (2010) A. J. Rothman, E. Levina, and J. Zhu.  A new approach to cholesky-based covariance regularization in high dimensions.  _Biometrika_ , 97(3):539–550, 2010.
  * Saumard and Wellner (2014) A. Saumard and J. A. Wellner.  Log-concavity and strong log-concavity: A review.  _Statistics Surveys_ , 8:45–114, 2014.
  * Serfling (1980) R. Serfling.  _Approximation Theorems of Mathematical Statistics_.  Wiley Series in Probability and Statistics. Wiley, 1980.
  * Shao and Tu (1995) J. Shao and D. Tu.  The jackknife and bootstrap.  1995\.
  * Shi (1992) S. G. Shi.  Accurate and efficient double-bootstrap confidence limit method.  _Computational Statistics and Data Analysis_ , 13(1):21–32, 1992.
  * Tanguy (2017) K. Tanguy.  _Quelques inégalités de superconcentration: théorie et applications_.  PhD thesis, Université Paul Sabatier-Toulouse III, 2017.
  * Valettas (2019) P. Valettas.  On the tightness of gaussian concentration for convex functions.  _Journal d’Analyse Mathématique_ , 139(1):341–367, 2019.
  * van der Vaart and Wellner (1996) A. W. van der Vaart and J. A. Wellner.  _Weak Convergence and Empirical Processes_.  Springer, 1996.
  * Vershynin (2012) R. Vershynin.  Introduction to the non-asymptotic analysis of random matrices.  In Y. Eldar and G. Kutyinok, editors, _Compressed Sensing, Theory and Applications_ , pages 210–268, Cambridge, 2012. Cambridge University Press.
  * Wainwright (2019) M. J. Wainwright.  _High-dimensional statistics: A non-asymptotic viewpoint_.  Cambridge University Press, 2019.
  * Wang et al. (2015) L. Wang, B. Peng, and R. Li.  A high-dimensional nonparametric multivariate test for mean vector.  _Journal of the American Statistical Association_ , 110(512):1658–1669, 2015.
  * Xu et al. (2017) G. Xu, L. Lin, P. Wei, and W. Pan.  An adaptive two-sample test for high-dimensional means.  _Biometrika_ , 103(3):609–624, 03 2017.
  * Xu et al. (2019) M. Xu, D. Zhang, and W. B. Wu.  Pearson’s chi-squared statistics: approximation theory and beyond.  _Biometrika_ , 106:716–723, 2019.
  * Xue and Yao (2020) K. Xue and F. Yao.  Distribution and correlation-free two-sample test of high-dimensional means.  _The Annals of Statistics_ , 48(3):1304 – 1328, 2020.
  * Zhang and Wu (2017) D. Zhang and W. B. Wu.  Gaussian approximation for high dimensional time series.  _Ann. Statist._ , 45(5):1895–1919, 10 2017.

Supplementary Materials for “A bootstrap hypothesis test for high-dimensional
mean vectors” Alexander Giessing11footnotemark: 1 Jianqing Fan22footnotemark:
2

Contents

[sections] [sections]l1

##  Appendix A Bootstrapping high-dimensional $\ell_{p}$-statistics

In this section we present Gaussian and bootstrap approximation for high-
dimensional $\ell_{p}$-statistics (Appendix A.1), lower bounds on the variance
of $\ell_{p}$-norms of (an-isotropic) Gaussian random vectors (Appendix A.2),
and essential results from our companion papers Giessing (2023a, b) (Appendix
A.3).

###  A.1 Approximating sampling distribution and quantiles

Unless otherwise stated, all parameters in this section may be thought of as
indexed by the sample size $n$. Since we present non-asymptotic bounds that
hold for all $n\geq 1$ we leave this dependence implicit. The results in this
section are consequences of the more general results in our two companion
papers Giessing (2023a, b) summarized in Appendix A.3.

At the core of any bootstrap procedure is a CLT. Thus, the following theorem
is key to all theoretical results in this paper:

######  Theorem 1 (Gaussian approximation).

Let $X,X_{1},\ldots X_{n}\in\mathbb{R}^{d}$ be i.i.d. random vectors with mean
zero and positive semi-definite covariance matrix
$\Sigma\neq\mathbf{0}\in\mathbb{R}^{d\times d}$. Set
$S_{n}=n^{-1/2}\sum_{i=1}^{n}X_{i}$ and $Z\sim N(0,\Sigma)$. Then, for $1\leq
p\leq\infty$, $n\geq 1$,

$\displaystyle\sup_{s\geq 0}\Big{|}\mathrm{P}\left(\|S_{n}\|_{p}\leq
s\right)-\mathrm{P}\left(\|Z\|_{p}\leq s\right)\Big{|}$
$\displaystyle\quad{}\quad{}\quad{}\lesssim\frac{(\mathrm{E}[\|X\|_{p}^{3}])^{1/3}}{n^{1/6}\sqrt{\mathrm{Var}(\|Z\|_{p})}}+\frac{\mathrm{E}\left[\|X\|_{p}^{3}\mathbf{1}\\{\|X\|_{p}^{3}>n\>\mathrm{E}[\|X\|_{p}^{3}]\\}\right]}{\mathrm{E}\left[\|X\|_{p}^{3}\right]}+\frac{\mathrm{E}[\|Z\|_{p}]}{\sqrt{n\mathrm{Var}(\|Z\|_{p})}},$

where $\lesssim$ hides an absolute constant independent of $p,n,d$, and the
distribution of the $X_{i}$’s.

This Gaussian approximation differs from the ones in the literature (e.g.
Chernozhukov et al., 2013, 2014a, 2014b, 2015, 2017, 2019, 2020; Zhang and Wu,
2017; Xu et al., 2019; Lopes et al., 2020; Xue and Yao, 2020; Fang and Koike,
2021) in three ways: First, it holds for all $\ell_{p}$-norms with exponents
$p\in[1,\infty]$ whereas existing results apply only to the
$\ell_{\infty}$-norm or the square of the $\ell_{2}$-norm. Second, the non-
asymptotic upper bound is dimension-free, i.e. it does not explicitly depend
on the dimension $d$. Since the quantities $\mathrm{E}[\|X\|_{p}^{3}]$ and
$\mathrm{E}[\|Z\|_{p}]$ can often be upper bounded in terms of the trace of
the covariance matrix (e.g. see Lemmas 1–3), this opens the possibility of
leveraging the eigen-structure of the covariance matrix. Third, the result
applies to degenerate distributions that do not have a strictly positive
definite covariance matrix.

The most striking feature of this Gaussian approximation result is the
dependence of the upper bound on the variance of the Gaussian proxy statistic
$\mathrm{Var}(\|Z\|_{p})$. The magnitude of this variance depends in a
delicate way on the exponent $p$ and the dimensions $d$ and hints at an
interesting connection between universality laws for high-dimensional data,
anti-concentration, and super-concentration. We refer to Section A.2 for
details.

The dependence of the upper bounds on the sample size $n$ is most likely sub-
optimal. For example, for $p=\infty$ and under additional assumptions on the
moments of the data, sharper rates have been derived in Lopes et al. (2020)
and Chernozhukov et al. (2020).

Specializing to the $\ell_{2}$-norm and comparing the upper bound in Theorem 1
with the upper bound in Theorem 1.1 in ben (2003) we observe that our bound is
better by a factor $d^{1/4}$. The reason for this discrepancy is that Gaussian
approximation result is in fact a weaker a statement than the ben’s (2003)
Berry-Esseen-type result and the two results are incomparable: In Theorem 1 we
consider the supremum over all $\ell_{2}$-balls with center at the origin,
whereas in Theorem 1.1 ben (2003) considers the supremum over all
$\ell_{2}$-balls.

Since the Gaussian distribution is fully characterized by its first two
moments, Theorem 1 instantly suggests that it should be possible to
approximate the sampling distribution of $\|S_{n}\|_{p}$ with the sampling
distribution of $\|Z_{n}\|_{p}$, where $Z_{n}\mid X_{1},\ldots,X_{n}\sim
N(0,\widehat{\Sigma}_{n})$ and $\widehat{\Sigma}_{n}$ is a positive semi-
definite estimate of $\Sigma$. The next theorem formalizes this idea:

######  Theorem 2 (Gaussian process bootstrap approximation).

Let $X,X_{1},\ldots X_{n}\in\mathbb{R}^{d}$ be i.i.d. random vectors with mean
zero and positive semi-definite covariance matrix
$\Sigma\neq\mathbf{0}\in\mathbb{R}^{d\times d}$. Let
$\widehat{\Sigma}_{n}\equiv\widehat{\Sigma}_{n}(X_{1},\ldots,X_{n})$ be any
positive semi-definite estimate of $\Sigma$. Set
$S_{n}=n^{-1/2}\sum_{i=1}^{n}X_{i}$, $Z\sim N(0,\Sigma)$, and $Z_{n}\mid
X_{1},\ldots X_{n}\sim N(0,\widehat{\Sigma}_{n})$. Then, for $1\leq
p\leq\infty$, $n\geq 1$,

$\displaystyle\sup_{s\geq 0}\Big{|}\mathrm{P}\left(\|S_{n}\|_{p}\leq
s\right)-\mathrm{P}\left(\|Z_{n}\|_{p}\leq s\mid
X_{1},\ldots,X_{n}\right)\Big{|}$
$\displaystyle\quad{}\lesssim\frac{(\mathrm{E}[\|X\|_{p}^{3}])^{1/3}}{n^{1/6}\sqrt{\mathrm{Var}(\|Z\|_{p})}}+\frac{\mathrm{E}\left[\|X\|_{p}^{3}\mathbf{1}\\{\|X\|_{p}^{3}>n\>\mathrm{E}[\|X\|_{p}^{3}]\\}\right]}{\mathrm{E}\left[\|X\|_{p}^{3}\right]}+\frac{\mathrm{E}[\|Z\|_{p}]}{\sqrt{n\mathrm{Var}(\|Z\|_{p})}}+\left(\frac{\|\widehat{\Sigma}_{n}-\Sigma\|_{q\rightarrow
p}}{\mathrm{Var}(\|Z\|_{p})}\right)^{1/3},$

where $1/p+1/q=1$ and $\lesssim$ hides an absolute constant independent of
$p,n,d$, and the distribution of the $X_{i}$’s.

To test statistical hypotheses and construct confidence intervals, we do not
need to estimate the entire sampling distribution but (only) specific
quantiles. Since the covariance matrix $\Sigma$ is unknown, the quantiles of
$\|Z\|_{p}$ with $Z\sim N(0,\Sigma)$ are infeasible. Instead, we use quantiles
of $\|Z_{n}\|_{p}$ where $Z_{n}\mid X_{1},\ldots,X_{n}\sim
N(0,\widehat{\Sigma}_{n})$: For $\alpha\in(0,1)$ arbitrary, we define

$\displaystyle c_{p}^{*}(\alpha;\widehat{\Sigma}_{n}):=\inf\left\\{s\geq
0:\mathrm{P}\left(\|Z_{n}\|_{p}\leq s\mid
X_{1},\ldots,X_{n}\right)\geq\alpha\right\\}.$

Since this quantity is random, it is not immediately obvious that it is a
valid approximation of the $\alpha$-quantile of the sampling distribution of
$\|S_{n}\|_{p}$. However, combing Theorem 2 with (by now) standard arguments
(e.g. Chernozhukov et al., 2013), we obtain the following result:

######  Theorem 3 (Gaussian process bootstrap approximation of quantiles).

Recall the conditions of Theorem 2. Let $(\Theta_{n})_{n\geq 1}\in\mathbb{R}$
be a sequence of arbitrary random variables, not necessarily independent of
$X_{1},\ldots,X_{n}$. For $1\leq p\leq\infty$, $n\geq 1$,

$\displaystyle\sup_{\alpha\in(0,1)}\Big{|}\mathrm{P}\left(\|S_{n}\|_{p}+\Theta_{n}\leq
c_{p}^{*}(\alpha;\widehat{\Sigma}_{n})\right)-\alpha\Big{|}$