splits using Bonferroni correction (see, e.g. Vovk and Wang,, 2020, for other
methods of combining $p$-values). However the resulting test is expected to be
conservative as it does not use the dependence structure between $p$-values.

#### Calibration under symmetry condition.

On the other hand, it is relatively straightforward to calibrate the test
statistic using many splits under the symmetry condition. One key observation
is that, when $X$ and $-X$ have the same distribution, any test statistic,
let’s say $T(X_{1},\ldots,X_{n})$, has the same distribution as
$T(\varepsilon_{1}X_{1},\ldots,\varepsilon_{n}X_{n})$ where
$\\{\varepsilon_{i}\\}_{i=1}^{n}$ are i.i.d. Rademacher random variables
independent of the data. We are then able to use exchangeability among
different test statistics associated with different Rademacher random
variables and produce a valid level $\alpha$ test as in (13). We note that,
similar to the permutation procedure for two-sample testing or independence
testing, this flip-sign procedure works as a wrapper around any statistic
including the average of statistics from multiple splits and provides a valid
$p$-value in finite samples.

Another way, which is more computationally efficient but more conservative, is
to use the concentration bound as before in Corollary 2. Let
$T_{\text{mean},(1)},\ldots,T_{\text{mean},(K)}$ be the test statistic in (7)
computed over different splits. We denote
$V_{(i)}:=\sqrt{m_{1}}T_{\text{mean},(i)}/\big{\\{}m_{1}+T^{2}_{\text{mean},(i)}\big{\\}}^{1/2}$
for $i=1,\ldots,K$ and define a test

$\displaystyle\phi_{\text{{sym}},K}:=\mathds{1}\Bigg{\\{}\frac{1}{K}\sum_{i=1}^{K}e^{\lambda_{i}V_{(i)}-\lambda_{i}^{2}/2}>\frac{1}{\alpha}\Bigg{\\}},$

for some $\lambda_{i}\in\mathbb{R}$ and $i=1,\ldots,K$. Then Markov’s
inequality shows that
$\sup_{P\in\mathcal{P}_{\text{sym},n}}\mathbb{E}_{P}[\phi_{\text{sym},K}]\leq\alpha$
since $\mathbb{E}_{P}[\exp(\lambda_{i}V_{(i)}-\lambda_{i}^{2}/2)]\leq 1$ for
each $i=1,\ldots,K$ under the symmetry assumption. We note that a similar idea
appears in Wasserman et al., (2020) where they study likelihood ratio tests
based on sample-splitting.

##  Appendix B Proofs of all results

In this section we collect technical details and proofs of our main results.
In addition to the notation introduced in Section 1.5, we need to introduce
some additional notation used throughout this section.

#### Additional notation.

For a sequence of random variables $X_{n}$ and constants $a_{n}$, we write
$X_{n}=O_{P}(a_{n})$ if $a_{n}^{-1}X_{n}$ is stochastically bounded and
$X_{n}=o_{P}(a_{n})$ if $a_{n}^{-1}X_{n}$ converges to zero in probability or
equivalently $a_{n}^{-1}X_{n}\overset{p}{\longrightarrow}0$. For real
sequences $a_{n}$ and $b_{n}$, we write $a_{n}\lesssim b_{n}$ or $b_{n}\gtrsim
a_{n}$ if there exists a constant $C$ such that $a_{n}\leq Cb_{n}$ for all
$n$.

To simplify the presentation, we often suppress dependence on $P$ and write
$\mathbb{E}$ (and $\mathbb{P}$) instead of $\mathbb{E}_{P}$ (and
$\mathbb{P}_{P}$).

###  B.1 Details on Assumption 1

In this subsection, we demonstrate that Assumption 1 is implied by the model
assumption made in Bai and Saranadasa, (1996); Chen and Qin, (2010); Wang and
Xu, (2019). We first recall their multivariate model assumption:

$\displaystyle X=\mu+\Gamma Z,$ (37)

where $\Gamma$ is a $d\times r$ matrix with some $r\geq d$ and thus
$\mathrm{Cov}(X)=\Gamma\Gamma^{\top}$. Here $Z=(z_{1},\ldots,z_{r})^{\top}$
satisfies $\mathbb{E}[Z]=0$, $\text{Var}[Z]=I$,
$\mathbb{E}[z_{i}^{4}]=3+\Delta<\infty$ and $\mathbb{E}[z_{i}^{8}]<C$ for
$i=1,\ldots,r$ where $\Delta$ is a difference between the fourth moment of
$z_{i}$ and $N(0,1)$ and $C$ is some positive constant. Furthermore it is
assumed that

$\displaystyle\mathbb{E}[z_{\ell_{1}}^{\alpha_{1}}z_{\ell_{2}}^{\alpha_{2}}\cdots
z_{\ell_{q}}^{\alpha_{q}}]=\mathbb{E}[z_{\ell_{1}}^{\alpha_{1}}]\mathbb{E}[z_{\ell_{2}}^{\alpha_{2}}]\cdots\mathbb{E}[z_{\ell_{q}}^{\alpha_{q}}]$

for a positive integer $q$ such that $\sum_{\ell=1}^{q}\alpha_{\ell}\leq 8$
and $\\{\ell_{1},\ldots,\ell_{q}\\}$ are distinct indices.

Now, for a given nonzero $u\in\mathbb{R}^{d}$ and by assuming $\mu=0$, Lemma 1
of Wang and Xu, (2019) yields

$\displaystyle\mathbb{E}[(u^{\top}X)^{4}]=\mathbb{E}[(Z^{\top}\Gamma^{\top}uu^{\top}\Gamma
Z)^{2}]\leq(3+\Delta)\text{tr}^{2}(\Gamma^{\top}uu^{\top}\Gamma)=(3+\Delta)\\{\mathbb{E}[(u^{\top}X)^{2}]\\}^{2}.$

This result together with Jensen’s inequality implies

$\displaystyle\left[\frac{\\{\mathbb{E}|u^{\top}X|^{3}\\}^{1/3}}{\\{\mathbb{E}|u^{\top}X|^{2}\\}^{1/2}}\right]^{4}\leq\frac{\mathbb{E}[(u^{\top}X)^{4}]}{\\{\mathbb{E}[(u^{\top}X)^{2}]\\}^{2}}\leq
3+\Delta.$

Thus, under the model (37), Assumption 1 is satisfied with
$K=(3+\Delta)^{1/4}$.

In order to demonstrate the dimension-agnostic property of $T_{\mathrm{cov}}$
under a non-Gaussian setting, we define a class of distributions for which
bound (21) holds with a potentially different constant.

######  Definition B.1 (Covariance model).

Define a class of distribution $\mathcal{P}_{0,n,\mathrm{cov}}$ where a random
vector $X\sim P\in\mathcal{P}_{0,n,\mathrm{cov}}$ is continuous and satisfies
model (37) with $\mu=0$, $d=r$ and $\Gamma=I$. Moreover, assume that there
exists a small constant $c>0$ such that $\Delta+2>c$.

###  B.2 Proof of Theorem 1

First note that
$\mathbb{E}_{P}[f_{\text{mean}}^{2}(X)|\mathcal{X}_{2}]=\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}$
is strictly greater zero and finite almost surely under Assumption 1. Thus,
based on Lemma 1, the studentized statistic has the following conditional
guarantee

$\displaystyle\sup_{-\infty<z<\infty}\big{|}\mathbb{P}_{P}(T_{\text{mean}}\leq
z|\mathcal{X}_{2})-\Phi(z)\big{|}\leq\frac{C\mathbb{E}_{P}[|f_{\text{mean}}^{3}(X)||\mathcal{X}_{2}]}{\\{\mathbb{E}_{P}[f_{\text{mean}}^{2}(X)|\mathcal{X}_{2}]\\}^{3/2}\sqrt{m_{1}}}\leq\frac{C_{K}}{\sqrt{m_{1}}}\quad\text{a.s.,}$

where the second inequality uses Assumption 1. Now the claim holds since

$\displaystyle\sup_{-\infty<z<\infty}\big{|}\mathbb{E}_{P}[\mathbb{P}_{P}(T_{\text{mean}}\leq
z|\mathcal{X}_{2})]-\Phi(z)\big{|}\leq\mathbb{E}_{P}\bigg{[}\sup_{-\infty<z<\infty}\big{|}\mathbb{P}_{P}(T_{\text{mean}}\leq
z|\mathcal{X}_{2})-\Phi(z)\big{|}\bigg{]}$

and $C_{K}$ does not depend on $P\in\mathcal{P}_{n}$.

###  B.3 Proof of Theorem 2

By recalling $\widehat{\mu}_{2}=m_{2}^{-1}\sum_{i=m_{1}+1}^{n}X_{i}$ and
denoting $\widehat{\mu}_{1}=m_{1}^{-1}\sum_{i=1}^{m_{1}}X_{i}$, the power
function can be written as

$\displaystyle\mathbb{P}_{P}(T_{\text{mean}}>z_{1-\alpha})$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\left(\frac{\sqrt{m_{1}}\widehat{\mu}_{2}^{\top}(\widehat{\mu}_{1}-\mu)}{\sqrt{m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{\widehat{\mu}_{2}^{\top}(X_{i}-\widehat{\mu}_{1})\\}^{2}}}>z_{1-\alpha}-\frac{\sqrt{m_{1}}\widehat{\mu}_{2}^{\top}\mu}{\sqrt{{m_{1}}^{-1}\sum_{i=1}^{m_{1}}\\{\widehat{\mu}_{2}^{\top}(X_{i}-\widehat{\mu}_{1})\\}^{2}}}\right).$

Note that $X-\mu$ satisfies Assumption 1 under the Gaussian assumption and so
it follows that

$\displaystyle\frac{\sqrt{m_{1}}\widehat{\mu}_{2}^{\top}(\widehat{\mu}_{1}-\mu)}{\sqrt{{m_{1}}^{-1}\sum_{i=1}^{m_{1}}\\{\widehat{\mu}_{2}^{\top}(X_{i}-\widehat{\mu}_{1})\\}^{2}}}\overset{d}{\longrightarrow}N(0,1).$
(38)

Now we claim that under Assumption 2:

  1. (C1).

$\displaystyle\sqrt{m_{1}}\widehat{\mu}_{2}^{\top}\mu=\sqrt{m_{1}}\mu^{\top}\mu+O_{P}\left(\sqrt{\frac{m_{1}}{m_{2}}\mu^{\top}\Sigma\mu}\right)$,

  2. (C2).

$\displaystyle\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}=\frac{1}{m_{2}}\text{tr}(\Sigma^{2})+\mu^{\top}\Sigma\mu+O_{P}\left(\sqrt{\frac{2}{m_{2}^{2}}\text{tr}(\Sigma^{4})+\frac{4}{m_{2}}\mu^{\top}\Sigma^{3}\mu}\right)$,

  3. (C3).

$\displaystyle\frac{\widehat{\mu}_{2}^{\top}\widehat{\Sigma}\widehat{\mu}_{2}}{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}\overset{p}{\longrightarrow}1$
where
$\displaystyle\widehat{\Sigma}=m_{1}^{-1}\sum_{i=1}^{m_{1}}(X_{i}-\widehat{\mu}_{1})(X_{i}-\widehat{\mu}_{1})^{\top}$.

Once these conditions are given, it follows that

$\displaystyle\frac{\sqrt{m_{1}}\widehat{\mu}_{2}^{\top}\mu}{\sqrt{{m_{1}}^{-1}\sum_{i=1}^{m_{1}}\\{\widehat{\mu}_{2}^{\top}(X_{i}-\widehat{\mu}_{1})\\}^{2}}}\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\frac{\sqrt{m_{1}}\widehat{\mu}_{2}^{\top}\mu}{\sqrt{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}}\times\sqrt{\frac{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}{\widehat{\mu}_{2}^{\top}\widehat{\Sigma}\widehat{\mu}_{2}}}$
$\displaystyle\overset{\text{(i)}}{=}\leavevmode\nobreak\ $
$\displaystyle\frac{\sqrt{m_{1}}\widehat{\mu}_{2}^{\top}\mu}{\sqrt{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}}\\{1+o_{P}(1)\\}$
$\displaystyle\overset{\text{(ii)}}{=}\leavevmode\nobreak\ $
$\displaystyle\frac{\sqrt{m_{1}}\mu^{\top}\mu+O_{P}(n^{-1/4})}{\sqrt{m_{2}^{-1}\text{tr}(\Sigma^{2})+O_{P}(n^{-1/2})}}\\{1+o_{P}(1)\\}$
$\displaystyle\overset{\text{(iii)}}{=}\leavevmode\nobreak\ $
$\displaystyle\frac{\sqrt{m_{1}}\mu^{\top}\mu}{\sqrt{m_{2}^{-1}\text{tr}(\Sigma^{2})}}+o_{P}(1),$

where step (i) uses (C3) with the continuous mapping theorem and step (ii)
follows by (C1) and (C2) along with Assumption 2. The last step (iii) holds
since $\sqrt{m_{1}}\mu^{\top}\mu=O(1)$ and $m_{2}^{-1}\text{tr}(\Sigma^{2})$
is strictly greater than zero and finite under the eigenvalue condition in
Assumption 2. Combining the last equality (iii) with the normal approximation
(38), we arrive at the conclusion that

$\displaystyle\mathbb{P}_{P}(T_{\text{mean}}>z_{1-\alpha})\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\Phi\left(z_{\alpha}+\frac{\sqrt{m_{1}}\mu^{\top}\mu}{\sqrt{m_{2}^{-1}\text{tr}(\Sigma^{2})}}\right)+o(1)$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\Phi\left(z_{\alpha}+\frac{n\sqrt{\kappa(1-\kappa)}\mu^{\top}\mu}{\sqrt{\text{tr}(\Sigma^{2})}}\right)+o(1).$

To complete the proof, we verify the conditions (C1), (C2) and (C3) in order.

$\bullet$ Verification of (C1) and (C2). It is clear to see that

$\displaystyle\mathbb{E}_{P}[\sqrt{m_{1}}\widehat{\mu}_{2}^{\top}\mu]=\sqrt{m_{1}}\mu^{\top}\mu\quad\text{and}\quad\text{Var}_{P}[\sqrt{m_{1}}\widehat{\mu}_{2}^{\top}\mu]=\frac{m_{1}}{m_{2}}\mu^{\top}\Sigma\mu.$

The condition (C1) follows since
$X_{n}=\mathbb{E}_{P}[X_{n}]+O_{P}(\sqrt{\text{Var}_{P}[X_{n}]})$.

For (C2), we first note the following moment calculations

$\displaystyle\mathbb{E}_{P}[\xi^{\top}\Lambda\xi]\leavevmode\nobreak\
=\leavevmode\nobreak\ \text{tr}(\Lambda\Sigma)+\mu^{\top}\Lambda\mu,$
$\displaystyle\text{Var}_{P}[\xi^{\top}\Lambda\xi]\leavevmode\nobreak\
=\leavevmode\nobreak\
2\text{tr}(\Lambda\Sigma\Lambda\Sigma)+4\mu^{\top}\Lambda\Sigma\Lambda\mu,$

where $\xi\sim N(\mu,\Sigma)$. Then, using the fact that
$\widehat{\mu}_{2}\sim N(\mu,m_{2}^{-1}\Sigma)$, we can obtain

$\displaystyle\mathbb{E}_{P}[\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}]\leavevmode\nobreak\
=\leavevmode\nobreak\
\frac{1}{m_{2}}\text{tr}(\Sigma^{2})+\mu^{\top}\Sigma\mu\quad\text{and}\quad\text{Var}_{P}[\widehat{\mu}_{2}^{\top}\Sigma]\leavevmode\nobreak\
=\leavevmode\nobreak\
\frac{2}{m_{2}^{2}}\text{tr}(\Sigma^{4})+\frac{4}{m_{2}}\mu^{\top}\Sigma^{3}\mu.$

Thus the condition (C2) follows similarly as (C1).

$\bullet$ Verification of (C3). We want to prove that for any $\epsilon>0$,

$\displaystyle\mathbb{P}_{P}\bigg{(}\bigg{|}\frac{\widehat{\mu}_{2}^{\top}\widehat{\Sigma}\widehat{\mu}_{2}}{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}-1\bigg{|}>\epsilon\bigg{)}\rightarrow
0\quad\text{as $n\rightarrow\infty$.}$

By the law of total expectation and conditional Chebyshev’s inequality,

$\displaystyle\mathbb{P}_{P}\bigg{(}\bigg{|}\frac{\widehat{\mu}_{2}^{\top}\widehat{\Sigma}\widehat{\mu}_{2}}{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}-1\bigg{|}>\epsilon\bigg{)}\leavevmode\nobreak\
\leq\leavevmode\nobreak\
\frac{1}{\epsilon^{2}}\mathbb{E}_{P}\bigg{[}\mathbb{E}_{P}\bigg{\\{}\bigg{(}\frac{\widehat{\mu}_{2}^{\top}\widehat{\Sigma}\widehat{\mu}_{2}-\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}\bigg{)}^{2}\bigg{|}\widehat{\mu}_{2}\bigg{\\}}\bigg{]}$

and the bias-variance decomposition gives

$\displaystyle\mathbb{E}_{P}\bigg{\\{}\bigg{(}\frac{\widehat{\mu}_{2}^{\top}\widehat{\Sigma}\widehat{\mu}_{2}-\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}\bigg{)}^{2}\bigg{|}\widehat{\mu}_{2}\bigg{\\}}\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\frac{1}{m_{1}^{2}}+\text{Var}_{P}\bigg{\\{}\frac{\widehat{\mu}_{2}^{\top}\widehat{\Sigma}\widehat{\mu}_{2}}{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}\bigg{|}\widehat{\mu}_{2}\bigg{\\}}.$

Using the variance formula of the sample variance (e.g. Lee,, 1990) and by
letting
$\widetilde{\mu}_{2}=\widehat{\mu}_{2}/\sqrt{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}$,

$\displaystyle\text{Var}_{P}\bigg{\\{}\frac{\widehat{\mu}_{2}^{\top}\widehat{\Sigma}\widehat{\mu}_{2}}{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}\bigg{|}\widehat{\mu}_{2}\bigg{\\}}\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\frac{(m_{1}-1)^{2}}{m_{1}^{2}}\bigg{\\{}\frac{\mathbb{E}_{P}[\\{\widetilde{\mu}_{2}^{\top}(X-\mu)\\}^{4}|\widehat{\mu}_{2}]}{m_{1}}-\frac{\\{\mathbb{E}_{P}[(\widetilde{\mu}_{2}^{\top}(X-\mu))^{2}|\widehat{\mu}_{2}]\\}^{4}(m_{1}-3)}{m_{1}(m_{1}-1)}\bigg{\\}}$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\frac{(m_{1}-1)^{2}}{m_{1}^{2}}\bigg{\\{}\frac{3}{m_{1}}-\frac{(m_{1}-3)}{m_{1}(m_{1}-1)}\bigg{\\}},$

where we use the following identities:

$\displaystyle\mathbb{E}_{P}[\\{\widetilde{\mu}_{2}^{\top}(X-\mu)\\}^{4}|\widehat{\mu}_{2}]\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\text{Var}_{P}[\\{\widetilde{\mu}_{2}^{\top}(X-\mu)\\}^{2}|\widehat{\mu}_{2}]+\\{\mathbb{E}_{P}[\\{\widetilde{\mu}_{2}^{\top}(X-\mu)\\}^{2}|\widehat{\mu}_{2}]\\}^{2}$
$\displaystyle=\leavevmode\nobreak\ $ $\displaystyle
2\text{tr}(\widetilde{\mu}_{2}\widetilde{\mu}_{2}^{\top}\Sigma\widetilde{\mu}_{2}\widetilde{\mu}_{2}^{\top}\Sigma)+\text{tr}^{2}(\widetilde{\mu}_{2}\widetilde{\mu}_{2}^{\top}\Sigma)=3,$

and

$\displaystyle\mathbb{E}_{P}[\\{\widetilde{\mu}_{2}^{\top}(X-\mu)\\}^{2}|\widehat{\mu}_{2}]\leavevmode\nobreak\
=\leavevmode\nobreak\
\text{tr}(\widetilde{\mu}_{2}^{\top}\Sigma\widetilde{\mu}_{2})=1.$

Combining the results yields

$\displaystyle\mathbb{E}_{P}\bigg{[}\mathbb{E}_{P}\bigg{\\{}\bigg{(}\frac{\widehat{\mu}_{2}^{\top}\widehat{\Sigma}\widehat{\mu}_{2}-\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}{\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}\bigg{)}^{2}\bigg{|}\widehat{\mu}_{2}\bigg{\\}}\bigg{]}\leavevmode\nobreak\
=\leavevmode\nobreak\
\frac{1}{m_{1}^{2}}+\frac{2(m_{1}-1)}{m_{1}^{2}}\rightarrow 0\quad\text{as
$m_{1}\rightarrow\infty$,}$

which verifies (C3). This completes the proof of Theorem 2.

###  B.4 Proof of Theorem 3

Recall that $\widehat{\mu}_{1}=m_{1}^{-1}\sum_{i=1}^{m_{1}}X_{i}$ and
$\widehat{\mu}_{2}=m_{2}^{-1}\sum_{j=m_{1}+1}^{n}X_{j}$ and note that under
Assumption 1, the denominator of $T_{\text{mean}}$ is greater than zero almost
surely. Consequently we can reformulate the type II error as

$\displaystyle\mathbb{E}_{P}[1-\phi_{\text{mean}}]\leavevmode\nobreak\
=\leavevmode\nobreak\
\mathbb{P}_{P}\bigg{\\{}\widehat{\mu}_{2}^{\top}\widehat{\mu}_{1}\leq
z_{1-\alpha}\times\bigg{(}m_{1}^{-2}\sum_{i=1}^{m_{1}}\\{\widehat{\mu}_{2}^{\top}(X_{i}-\widehat{\mu}_{1})\\}^{2}\bigg{)}^{1/2}\bigg{\\}}.$
(39)

Then, using Markov’s inequality, we can show that the following event
$\mathcal{A}$:

$\displaystyle\mathcal{A}=\Bigg{\\{}\frac{1}{m_{1}^{2}}\sum_{i=1}^{m_{1}}\\{\widehat{\mu}_{2}^{\top}(X_{i}-\widehat{\mu}_{1})\\}^{2}<\frac{2}{\beta}\bigg{[}\frac{\text{tr}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\mu^{\top}\Sigma\mu}{m_{1}}\bigg{]}\Bigg{\\}}$
(40)

holds with probability at least $\beta/2$ for some $\beta>0$ specified later.
We also note that the mean and the variance of
$\widehat{\mu}_{2}^{\top}\widehat{\mu}_{1}$ are

$\displaystyle\mathbb{E}_{P}\big{[}\widehat{\mu}_{2}^{\top}\widehat{\mu}_{1}\big{]}\leavevmode\nobreak\
=\leavevmode\nobreak\ $ $\displaystyle\|\mu\|_{2}^{2},$
$\displaystyle\text{Var}_{P}\big{[}\widehat{\mu}_{2}^{\top}\widehat{\mu}_{1}\big{]}\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\left(\frac{1}{m_{1}}+\frac{1}{m_{2}}\right)\mu^{\top}\Sigma\mu+\frac{\text{tr}(\Sigma^{2})}{m_{1}m_{2}}$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\left(\frac{1}{m_{1}}+\frac{1}{m_{2}}\right)\|\mu\|_{2}^{2}\lambda^{\ast}+\frac{\text{tr}(\Sigma^{2})}{m_{1}m_{2}},$
(41)

where the inequality uses the definition of $\lambda^{\ast}$. Therefore, when
$z_{1-\alpha}>0$, the type II error is bounded by

$\displaystyle\mathbb{E}_{P}[1-\phi_{\text{mean}}]\leavevmode\nobreak\
\leq\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\left(\widehat{\mu}_{2}^{\top}\widehat{\mu}_{1}\leq
z_{1-\alpha}\sqrt{\frac{2}{\beta}\bigg{[}\frac{\text{tr}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\mu^{\top}\Sigma\mu}{m_{1}}\bigg{]}}\right)+\mathbb{P}_{P}(\mathcal{A}^{c})$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\left(\widehat{\mu}_{2}^{\top}\widehat{\mu}_{1}\leq
z_{1-\alpha}\sqrt{\frac{2}{\beta}\bigg{[}\frac{\text{tr}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\|\mu\|_{2}^{2}\lambda^{\ast}}{m_{1}}\bigg{]}}\right)+\frac{\beta}{2}.$

Observe that by taking $\beta=1/K_{n}^{2(1-\delta)}$ for a small $\delta>0$
such that $1-\delta>0$ and $n$ sufficiently large, we can bound

$\displaystyle\frac{\|\mu\|_{2}^{2}}{2}\leavevmode\nobreak\
\geq\leavevmode\nobreak$ $\displaystyle
z_{1-\alpha}\sqrt{\frac{2}{\beta}\bigg{[}\frac{(\lambda^{\ast})^{2}}{\kappa_{\ast}(1-\kappa^{\ast})}\frac{d}{n^{2}}+\frac{\|\mu\|_{2}^{2}\lambda^{\ast}}{\kappa_{\ast}n}\bigg{]}}$
(42) $\displaystyle\leavevmode\nobreak\
\overset{\text{($\ast$)}}{\geq}\leavevmode\nobreak$ $\displaystyle
z_{1-\alpha}\sqrt{\frac{2}{\beta}\bigg{[}\frac{\text{tr}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\|\mu\|_{2}^{2}\lambda^{\ast}}{m_{1}}\bigg{]}},$

where step ($\ast$) uses the conditions:
$\lambda_{\text{max}}(\Sigma)\leq\lambda^{\ast}$ and $\kappa_{\ast}\leq
m_{1}/n\leq\kappa^{\ast}$. Thus

$\displaystyle\mathbb{P}_{P}\left(\widehat{\mu}_{2}^{\top}\widehat{\mu}_{1}\leq
z_{1-\alpha}\sqrt{\frac{2}{\beta}\bigg{[}\frac{\text{tr}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\|\mu\|_{2}^{2}\lambda^{\ast}}{m_{1}}\bigg{]}}\right)$
$\displaystyle\leavevmode\nobreak\
\overset{\text{(i)}}{\leq}\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\left(\widehat{\mu}_{2}^{\top}\widehat{\mu}_{1}-\|\mu\|_{2}^{2}\leq-\frac{\|\mu\|_{2}^{2}}{2}\right)$
$\displaystyle\leavevmode\nobreak\
\overset{\text{(ii)}}{\leq}\leavevmode\nobreak\ $
$\displaystyle\frac{4}{\|\mu\|_{2}^{4}}\text{Var}_{P}\big{[}\widehat{\mu}_{2}^{\top}\widehat{\mu}_{1}\big{]}\leavevmode\nobreak\
\overset{\text{(iii)}}{\leq}\leavevmode\nobreak\
\frac{4}{\|\mu\|_{2}^{4}}\Bigg{[}\left(\frac{1}{m_{1}}+\frac{1}{m_{2}}\right)\|\mu\|_{2}^{2}\lambda^{\ast}+\frac{(\lambda^{\ast})^{2}d}{m_{1}m_{2}}\Bigg{]}$
$\displaystyle\overset{\text{(iv)}}{\lesssim}\leavevmode\nobreak\ $
$\displaystyle\frac{1}{n\|\mu\|_{2}^{2}}+\frac{d}{n^{2}\|\mu\|_{2}^{4}}\leavevmode\nobreak\
\overset{(v)}{\lesssim}\frac{1}{K_{n}^{2}\sqrt{d}}+\frac{1}{K_{n}^{4}},$

where step (i) uses the inequality (42), step (ii) follows by Chebyshev’s
inequality and step (iii) follows by the upper bound for the variance (41)
along with the condition on the maximum eigenvalue. Step (iv) is true since
$\kappa_{\ast}\leq m_{1}/n\leq\kappa^{\ast}$ and finally step (v) uses the
condition that $\|\mu\|_{2}\geq K_{n}d^{1/4}/n^{1/2}$. Now by taking the
supremum over $\mathcal{Q}_{\mathrm{mean}}(\epsilon_{n})$, we can achieve the
uniform type II error guarantee.

Since the type II error rate is decreasing in $\alpha$, the result also
follows when $z_{1-\alpha}<0$. This completes the proof of Theorem 3.

###  B.5 Proof of Corollary 2

The first claim follows by our discussion in the main text. To show the second
claim, note that $1-\Phi(t)\leq\exp(-t^{2}/2)$ for all $t\in\mathbb{R}$, which
implies $\sqrt{2\log(1/\alpha)}\geq z_{1-\alpha}$ for any $\alpha\in(0,1)$.
This means that $\phi_{\text{sym}}\leq\phi_{\text{{mean}}}$ and thus
$\lim_{n\rightarrow\infty}\sup_{P\in\mathcal{P}_{0,n}}\mathbb{E}_{P}[\phi_{\text{{sym}}}]\leq\alpha$
due to Theorem 3. Note that we assume $\alpha$ is fixed and the proof for the
uniform type II error of $\phi_{\text{mean}}$ goes through even when
$z_{1-\alpha}$ is replaced by some other positive constant. Therefore the
second claim also holds.

###  B.6 Proof of Theorem 4

$\bullet$ Type I error. Since we assume that $X$ has a continuous
distribution, $\widehat{v}$ is nonzero with probability one. Thus, for any
$P\in\mathcal{P}_{0,n}$, we have

$\displaystyle\frac{\mathbb{E}_{P}[|f_{\text{sparse}}^{3}(X)||\mathcal{X}_{2}]}{\\{\mathbb{E}_{P}[f_{\text{sparse}}^{2}(X)|\mathcal{X}_{2}]\\}^{3/2}\sqrt{m_{1}}}=\frac{\mathbb{E}_{P}[|\widehat{v}^{\top}X|^{3}|\mathcal{X}_{2}]}{\\{\mathbb{E}_{P}[(\widehat{v}^{\top}X)^{2}|\mathcal{X}_{2}]\\}^{3/2}\sqrt{m_{1}}}\leq\frac{K^{3}}{\sqrt{m_{1}}}\quad\text{a.s.}$

Therefore the same Berry–Esseen bound used earlier gives us

$\displaystyle\sup_{P\in\mathcal{P}_{0,n}}\sup_{-\infty<z<\infty}\big{|}\mathbb{P}_{P}(T_{\text{{sparse}}}\leq
z)-\Phi(z)\big{|}\leq\frac{C_{K}}{\sqrt{m_{1}}},$

which in turn implies that type I error control:

$\displaystyle\sup_{P\in\mathcal{P}_{0,n}}\big{|}\mathbb{E}_{P}[\phi_{\text{{sparse}}}]-\alpha\big{|}=O\left(\frac{1}{\sqrt{m_{1}}}\right)=o(1).$

$\bullet$ Type II error. Turning to type II error control, assume without loss
of generality that $|\mu^{(j)}|>\epsilon_{n}$ for
$j\in\mathcal{J}:=\\{1,\ldots,J\\}$ where $1\leq J\leq d$. For each
$j\in\mathcal{J}$, let us define an event
$\mathcal{A}_{j}:=\big{\\{}\widehat{\mu}_{2}^{(j)}>\max_{i\in\mathcal{J}^{c}}|\widehat{\mu}_{2}^{(i)}|\big{\\}}$
if $\mu^{(j)}>0$ and
$\big{\\{}-\widehat{\mu}_{2}^{(j)}>\max_{i\in\mathcal{J}^{c}}|\widehat{\mu}_{2}^{(i)}|\big{\\}}$
if $\mu^{(j)}<0$. Then the type II error of our test is bounded above by

$\displaystyle\mathbb{P}_{P}(T_{\text{sparse}}\leq
z_{1-\alpha})\leavevmode\nobreak\ \leq\leavevmode\nobreak\ $
$\displaystyle\underbrace{\mathbb{P}_{P}(T_{\text{sparse}}\leq
z_{1-\alpha},\cup_{j\in\mathcal{J}}\mathcal{A}_{j})}_{(I)}+\underbrace{\mathbb{P}_{P}(\cap_{j\in\mathcal{J}}\mathcal{A}_{j}^{c})}_{(II)}.$

We analyze $(I)$ and $(II)$ separately. To start with the term $(I)$, the
event $\cup_{j\in\mathcal{J}}\mathcal{A}_{j}$ implies that
$i^{\star}=\arg\max_{1\leq j\leq d}|\widehat{\mu}_{2}^{(j)}|$ should belong to
$\mathcal{J}$. Thus by letting events $\mathcal{B}_{j}:=\\{i^{\star}=j\\}$ for
$j=1,\ldots,J$, which are disjoint with probability one, we have

$\displaystyle\mathbb{P}_{P}(T_{\text{sparse}}\leq
z_{1-\alpha},\cup_{j\in\mathcal{J}}\mathcal{A}_{j})\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\sum_{k\in\mathcal{J}}\mathbb{P}_{P}(T_{\text{sparse}}\leq
z_{1-\alpha},\cup_{j\in\mathcal{J}}\mathcal{A}_{j},\mathcal{B}_{k})$ (43)
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\sum_{k\in\mathcal{J}}\mathbb{P}_{P}(\mathcal{B}_{k},\cup_{j\in\mathcal{J}}\mathcal{A}_{j})\mathbb{P}_{P}(T_{\text{sparse}}\leq
z_{1-\alpha}\leavevmode\nobreak\
|\cup_{j\in\mathcal{J}}\mathcal{A}_{j},\mathcal{B}_{k}).$

Now, by letting
$\widehat{s}_{1,k}^{2}=m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{f_{\text{sparse}}(X_{i})-\overline{f}_{\text{sparse}}\\}^{2}$
when $i^{\star}=k$ and assuming $\mu^{(k)}>0$ (and the same argument holds for
$\mu^{(k)}<0$), we can simply write

$\displaystyle\mathbb{P}_{P}(T_{\text{sparse}}\leq
z_{1-\alpha}\leavevmode\nobreak\
|\cup_{j\in\mathcal{J}}\mathcal{A}_{j},\mathcal{B}_{k})$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\bigg{\\{}\text{sign}\Big{(}\widehat{\mu}_{2}^{(k)}\Big{)}\times\frac{\sqrt{m_{1}}\widehat{\mu}_{1}^{(k)}}{\widehat{s}_{1,k}}\leq
z_{1-\alpha}\leavevmode\nobreak\
\Big{|}\cup_{j\in\mathcal{J}}\mathcal{A}_{j},\mathcal{B}_{k}\bigg{\\}}$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\bigg{\\{}\frac{\sqrt{m_{1}}(\widehat{\mu}_{1}^{(k)}-\mu^{(k)})}{\widehat{s}_{1,k}}\leq
z_{1-\alpha}-\
\frac{\sqrt{m_{1}}\mu^{(k)}}{\widehat{s}_{1,k}}\leavevmode\nobreak\
\Big{|}\cup_{j\in\mathcal{J}}\mathcal{A}_{j},\mathcal{B}_{k}\bigg{\\}}.$

Define another event $\mathcal{C}_{k}:=\\{\widehat{s}_{1,k}<\nu^{-1}\sigma\\}$
where $\nu>0$ will be specified later. Notice that the sub-Gaussian parameter
$\sigma^{2}$ is lower bounded by the variance of $X^{(j)}$ for all
$j=1,\ldots,d$. Then, by Markov’s inequality, it holds
$\mathbb{P}_{P}\\{\mathcal{C}_{k}^{c}\leavevmode\nobreak\
|\cup_{j\in\mathcal{J}}\mathcal{A}_{j},\mathcal{B}_{k}\\}\leq\nu$ and thus we
may further bound the above equation by

$\displaystyle\mathbb{P}_{P}(T_{\text{sparse}}\leq
z_{1-\alpha}\leavevmode\nobreak\
|\cup_{j\in\mathcal{J}}\mathcal{A}_{j},\mathcal{B}_{k})$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\bigg{\\{}\frac{\sqrt{m_{1}}(\widehat{\mu}_{1}^{(k)}-\mu^{(k)})}{\widehat{s}_{1,k}}\leq
z_{1-\alpha}-\
\frac{\sqrt{m_{1}}\epsilon_{n}}{\nu^{-1}\sigma}\leavevmode\nobreak\
\Big{|}\cup_{j\in\mathcal{J}}\mathcal{A}_{j},\mathcal{B}_{k}\bigg{\\}}+\nu$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\Phi\bigg{\\{}z_{1-\alpha}-\
\frac{\sqrt{m_{1}}\epsilon_{n}}{\nu^{-1}\sigma}\bigg{\\}}+\frac{C}{\sqrt{m_{1}}}+\nu,$

where the first inequality uses our assumption that
$\mu^{(k)}\geq\epsilon_{n}>0$ and the second inequality uses the Berry–Esseen
bound for the studentized statistic (Bentkus and Götze,, 1996). Here $C$ is a
universal constant and one can take $K^{3}$ where $K$ is given in Assumption
1. Combining this bound with the equation (43) yields

$\displaystyle\mathbb{P}_{P}(T_{\text{sparse}}\leq
z_{1-\alpha},\cup_{j\in\mathcal{J}}\mathcal{A}_{j})\leavevmode\nobreak\
\leq\leavevmode\nobreak\ $ $\displaystyle\Phi\bigg{\\{}z_{1-\alpha}-\
\frac{\sqrt{m_{1}}\epsilon_{n}}{\nu^{-1}\sigma}\bigg{\\}}+\frac{C}{\sqrt{m_{1}}}+\nu$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\Phi\bigg{\\{}z_{1-\alpha}-\
\frac{\sqrt{m_{1}}}{\nu^{-1}\sigma}\times K_{n}\sigma\sqrt{\frac{\log
d}{n}}\bigg{\\}}+\frac{C}{\sqrt{m_{1}}}+\nu,$

where the last inequality holds due to $\epsilon_{n}\geq
K_{n}\sigma\sqrt{n^{-1}\log d}$. Note that the upper bound does not depend on
$P$. Therefore, by taking $\nu=K_{n}^{-q}$ where $0<q<1$ and assuming
$\kappa_{\ast}<m_{1}/n<\kappa^{\ast}$, we see that

$\displaystyle\sup_{P\in\mathcal{Q}_{\text{sparse}}(\epsilon_{n})}\mathbb{P}_{P}(T_{\text{sparse}}\leq
z_{1-\alpha},\cup_{j\in\mathcal{J}}\mathcal{A}_{j})\rightarrow 0\quad\text{as
$n\rightarrow\infty$.}$ (44)

Next turning to the term $(II)$, we first bound $(II)$ by
$\mathbb{P}_{P}(\cap_{j\in\mathcal{J}}\mathcal{A}_{j}^{c})\leq\mathbb{P}_{P}(\mathcal{A}_{1}^{c})$.
For the case when $\mu^{(1)}>0$ (and similarly for $\mu^{(1)}<0$),

$\displaystyle\mathbb{P}_{P}(\mathcal{A}_{1}^{c})\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\big{\\{}\widehat{\mu}_{2}^{(1)}\leq\max_{i\in\mathcal{J}^{c}}|\widehat{\mu}_{2}^{(i)}|\big{\\}}=\mathbb{P}_{P}\big{\\{}\mu^{(1)}\leq\max_{i\in\mathcal{J}^{c}}|\widehat{\mu}_{2}^{(i)}|-(\widehat{\mu}_{2}^{(1)}-\mu^{(1)})\big{\\}}$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\big{\\{}\mu^{(1)}\leq\max_{i\in\mathcal{J}^{c}}|\widehat{\mu}_{2}^{(i)}|+|\widehat{\mu}_{2}^{(1)}-\mu^{(1)}|\big{\\}}$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\big{\\{}\mu^{(1)}\leq 2\max_{1\leq j\leq
d}|\widehat{\mu}_{2}^{(i)}-\mu^{(i)}|\big{\\}}.$

We note that $\widehat{\mu}_{2}^{(i)}-\mu^{(i)}$ is sub-Gaussian with
parameter $\sigma^{2}/m_{2}$. Therefore, together with the union bound,

$\displaystyle\mathbb{P}_{P}\big{\\{}\mu^{(1)}\leq 2\max_{1\leq j\leq
d}|\widehat{\mu}_{2}^{(i)}-\mu^{(i)}|\big{\\}}\leavevmode\nobreak\
\leq\leavevmode\nobreak\ $
$\displaystyle\sum_{i=1}^{d}\mathbb{P}_{P}\\{\mu^{(1)}/2\leq|\widehat{\mu}_{2}^{(i)}-\mu^{(i)}|\\}$
$\displaystyle\leq\leavevmode\nobreak\ $ $\displaystyle
2d\exp\bigg{\\{}-\frac{m_{2}(\mu^{(1)})^{2}}{8\sigma^{2}}\bigg{\\}},$

Now by our choice of $\mu^{(1)}\geq\epsilon_{n}$, the upper bound converges to
zero uniformly over $\mathcal{Q}_{\text{sparse}}(\epsilon_{n})$ and so

$\displaystyle\sup_{P\in\mathcal{Q}_{\text{sparse}}(\epsilon_{n})}\mathbb{P}_{P}(\cap_{j\in\mathcal{J}}\mathcal{A}_{j}^{c})\rightarrow
0\quad\text{as $n\rightarrow\infty$.}$

Combining this with (44) completes the proof of Theorem 4.

###  B.7 Proof of Proposition 2

We start by noting that $f_{\mathrm{cov}}(x)$ can be written as

$\displaystyle
f_{\mathrm{cov}}(x)=x^{\top}(S-I)x-\mathbb{E}_{P_{0}}[X^{\top}(S-I)X|\mathcal{X}_{2}],$

where $S=m_{2}^{-1}\sum_{j=m_{1}+1}^{n}X_{j}X_{j}^{\top}$. Based on this
reformulation and using the moment expression for a Gaussian quadratic form
(e.g. Lemma 3 of Cai and Ma,, 2013), we have

$\displaystyle\mathbb{E}_{P_{0}}[|f_{\mathrm{cov}}(X)|^{2}|\mathcal{X}_{2}]=\text{Var}_{P_{0}}[f_{\mathrm{cov}}(X)|\mathcal{X}_{2}]=2\text{tr}\\{(S-I)^{2}\\},$
$\displaystyle\mathbb{E}_{P_{0}}[|f_{\mathrm{cov}}(X)|^{4}|\mathcal{X}_{2}]=48\text{tr}\\{(S-I)^{4}\\}+12\text{tr}^{2}\\{(S-I)^{2}\\}.$

Therefore the right-hand side of (20) becomes

$\displaystyle\frac{\mathbb{E}_{P_{0}}[|f_{\mathrm{cov}}(X)|^{4}|\mathcal{X}_{2}]}{\\{\mathbb{E}_{P_{0}}[|f_{\mathrm{cov}}(X)|^{2}|\mathcal{X}_{2}]\\}^{2}m_{1}}\leavevmode\nobreak\
=\leavevmode\nobreak\
\frac{12\text{tr}\\{(S-I)^{4}\\}}{m_{1}\text{tr}^{2}\\{(S-I)^{2}\\}}+\frac{3}{m_{1}}\quad\text{a.s.}$

Furthermore, by Cauchy-Schwarz inequality, we have
$\text{tr}\\{(S-I)^{4}\\}\leq\text{tr}^{2}\\{(S-I)^{2}\\}$, which leads to

$\displaystyle\frac{\mathbb{E}_{P_{0}}[|f_{\mathrm{cov}}(X)|^{4}|\mathcal{X}_{2}]}{\\{\mathbb{E}_{P_{0}}[|f_{\mathrm{cov}}(X)|^{2}|\mathcal{X}_{2}]\\}^{2}m_{1}}\leq\frac{15}{m_{1}}\quad\text{a.s.}$
(45)

For a general distribution $P$ in $\mathcal{P}_{0,n,\mathrm{cov}}$,
Proposition A.1 of Chen et al., (2010) yields that

$\displaystyle\mathbb{E}_{P}[|f_{\mathrm{cov}}(X)|^{2}|\mathcal{X}_{2}]\geq
c\text{tr}\\{(S-I)^{2}\\},$
$\displaystyle\mathbb{E}_{P}[|f_{\mathrm{cov}}(X)|^{4}|\mathcal{X}_{2}]\leq
C\text{tr}^{2}\\{(S-I)^{2}\\},$

where constant $c$ is given in Definition B.1 and $C$ is another constant
given in Proposition A.1 of Chen et al., (2010). Therefore, for some constant
$C^{\prime}>0$, it holds

$\displaystyle\sup_{P\in\mathcal{P}_{0,n,\mathrm{cov}}}\frac{\mathbb{E}_{P}[|f_{\mathrm{cov}}(X)|^{4}|\mathcal{X}_{2}]}{\\{\mathbb{E}_{P}[|f_{\mathrm{cov}}(X)|^{2}|\mathcal{X}_{2}]\\}^{2}m_{1}}\leq\frac{C^{\prime}}{m_{1}}\quad\text{a.s.}$
(46)

This result together with the Berry–Esseen bound (19) completes the proof of
Proposition 2.

###  B.8 Proof of Theorem 5

To study the power of $\phi_{\mathrm{cov}}$, we need to understand the
limiting behavior of $T_{\mathrm{cov}}$ under the alternative. To this end,
let us modify $f_{\mathrm{cov}}(x)$ in (17) and define

$\displaystyle
f_{\mathrm{cov}}^{\ast}(x):=f_{\mathrm{cov}}(x)-\text{tr}\\{(\Sigma-I)(S-I)\\},$

where $S=m_{2}^{-1}\sum_{j=m_{1}+1}^{n}X_{j}X_{j}^{\top}$. By putting
$\overline{f}_{\mathrm{cov}}^{\ast}=m_{1}^{-1}\sum_{i=1}^{m_{1}}f_{\mathrm{cov}}^{\ast}(X_{i})$,
consider a studentized statistic

$\displaystyle
T_{\mathrm{cov}}^{(a)}:=\frac{\sqrt{m_{1}}\overline{f}_{\mathrm{cov}}^{\ast}}{\sqrt{m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{f_{\mathrm{cov}}^{\ast}(X_{i})-\overline{f}_{\mathrm{cov}}^{\ast}\\}^{2}}}=T_{\mathrm{cov}}-\frac{\sqrt{m_{1}}\text{tr}\\{(\Sigma-I)(S-I)\\}}{\sqrt{m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{f_{\mathrm{cov}}^{\ast}(X_{i})-\overline{f}_{\mathrm{cov}}^{\ast}\\}^{2}}}.$

In the rest of this proof, we prove the following claims:

  1. (D1).

$\displaystyle T_{\mathrm{cov}}^{(a)}\overset{d}{\longrightarrow}N(0,1).$

  2. (D2).

$\displaystyle\frac{m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{f_{\mathrm{cov}}^{\ast}(X_{i})-\overline{f}_{\mathrm{cov}}^{\ast}\\}^{2}}{2\text{tr}\\{(\Sigma(S-I))^{2}\\}}=1+o_{P}(1)$.

  3. (D3).

$\displaystyle\frac{\sqrt{m_{1}}\text{tr}\\{(\Sigma-I)(S-I)\\}}{\sqrt{2\text{tr}\\{(\Sigma(S-I))^{2}\\}}}=\frac{n\sqrt{\kappa(1-\kappa)}\|\Sigma-I\|_{F}^{2}}{\sqrt{2}{\text{tr}}(\Sigma^{2})}+o_{P}(1)=O_{P}(1)$.

Once these claims are proved, the power function of $\phi_{\mathrm{cov}}$
approximates

$\displaystyle\mathbb{E}_{P}[\phi_{\mathrm{cov}}]\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\left(T_{\mathrm{cov}}^{(a)}>z_{1-\alpha}-\frac{\sqrt{m_{1}}\text{tr}\\{(\Sigma-I)(S-I)\\}}{\sqrt{m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{f_{\mathrm{cov}}(X_{i})-\overline{f}_{\mathrm{cov}}\\}^{2}}}\right)$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\left(T_{\mathrm{cov}}^{(a)}>z_{1-\alpha}-\frac{\sqrt{m_{1}}\text{tr}\\{(\Sigma-I)(S-I)\\}}{\sqrt{2\text{tr}\\{(\Sigma(S-I))^{2}\\}}}\\{1+o_{P}(1)\\}\right)$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\Phi\left(z_{\alpha}+\frac{n\sqrt{\kappa(1-\kappa)}\|\Sigma-I\|_{F}^{2}}{\sqrt{2}{\text{tr}}(\Sigma^{2})}\right)+o(1),$

which completes the proof. The rest of the proof is dedicated to verifying
these claims.

$\bullet$ Verification of (D1). Since
$\mathbb{E}_{P}[f_{\mathrm{cov}}^{\ast}(X)|\mathcal{X}_{2}]=0$, the
conditional Berry–Esseen bound for a studentized statistic implies

$\displaystyle\sup_{-\infty<z<\infty}\big{|}\mathbb{P}_{P}(T_{\mathrm{cov}}^{(a)}\leq
z|\mathcal{X}_{2})-\Phi(z)\big{|}\leq
C_{1}\frac{\mathbb{E}_{P}[|f_{\mathrm{cov}}^{\ast}(X)|^{3}|\mathcal{X}_{2}]}{\\{\mathbb{E}_{P}[|f_{\mathrm{cov}}^{\ast}(X)|^{2}|\mathcal{X}_{2}]\\}^{3/2}\sqrt{m_{1}}},$

for some $C_{1}>0$. Note that $f_{\mathrm{cov}}^{\ast}(X)$ can be succinctly
written as

$\displaystyle
f_{\mathrm{cov}}^{\ast}(x)=x^{\top}(S-I)x-\mathbb{E}_{P}[X^{\top}(S-I)X|\mathcal{X}_{2}]$

and its second and fourth moments can be computed as

$\displaystyle\mathbb{E}_{P}[|f_{\mathrm{cov}}^{\ast}(X)|^{2}|\mathcal{X}_{2}]=\text{Var}_{P}[f_{\mathrm{cov}}^{\ast}(X)|\mathcal{X}_{2}]=2\text{tr}\\{(\Sigma(S-I))^{2}\\},$
(47)
$\displaystyle\mathbb{E}_{P}[|f_{\mathrm{cov}}^{\ast}(X)|^{4}|\mathcal{X}_{2}]=48\text{tr}\\{(\Sigma(S-I))^{4}\\}+12\text{tr}^{2}\\{(\Sigma(S-I))^{2}\\}.$

Therefore, similarly to Proposition 2,

$\displaystyle\sup_{-\infty<z<\infty}\big{|}\mathbb{P}_{P}(T_{\mathrm{cov}}^{(a)}\leq
z)-\Phi(z)\big{|}\leq\frac{C_{2}}{\sqrt{m_{1}}},$

which shows that $T_{\mathrm{cov}}^{(a)}$ converges to the standard normal
distribution.

$\bullet$ Verification of (D2). Our goal is to prove that for any $\epsilon>0$

$\displaystyle\mathbb{P}_{P}\left(\bigg{|}\frac{m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{f_{\mathrm{cov}}^{\ast}(X_{i})-\overline{f}_{\mathrm{cov}}^{\ast}\\}^{2}}{2\text{tr}\\{(\Sigma(S-I))^{2}\\}}-1\bigg{|}\geq\epsilon\right)\rightarrow
0\quad\text{as $m_{1}\rightarrow\infty$.}$ (48)

To verify this claim, we follow the same steps used in the proof of Theorem 2.
By the law of total expectation and conditional Chebyshev’s inequality,

$\displaystyle\mathbb{P}_{P}\bigg{(}\bigg{|}\frac{m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{f_{\mathrm{cov}}^{\ast}(X_{i})-\overline{f}_{\mathrm{cov}}^{\ast}\\}^{2}}{2\text{tr}\\{(\Sigma(S-I))^{2}\\}}-1\bigg{|}>\epsilon\bigg{)}$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\frac{1}{\epsilon^{2}}\mathbb{E}_{P}\bigg{[}\mathbb{E}_{P}\bigg{\\{}\bigg{(}\frac{m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{f_{\mathrm{cov}}^{\ast}(X_{i})-\overline{f}_{\mathrm{cov}}^{\ast}\\}^{2}-2\text{tr}\\{(\Sigma(S-I))^{2}\\}}{2\text{tr}\\{(\Sigma(S-I))^{2}\\}}\bigg{)}^{2}\bigg{|}\mathcal{X}_{2}\bigg{\\}}\bigg{]}$

and the bias-variance decomposition along with the variance formula of the
sample variance (e.g. Lee,, 1990) gives

$\displaystyle\mathbb{E}_{P}\bigg{\\{}\bigg{(}\frac{m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{f_{\mathrm{cov}}^{\ast}(X_{i})-\overline{f}_{\mathrm{cov}}^{\ast}\\}^{2}-2\text{tr}\\{(\Sigma(S-I))^{2}\\}}{2\text{tr}\\{(\Sigma(S-I))^{2}\\}}\bigg{)}^{2}\bigg{|}\mathcal{X}_{2}\bigg{\\}}$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\frac{1}{m_{1}^{2}}+\text{Var}_{P}\bigg{\\{}\frac{m_{1}^{-1}\sum_{i=1}^{m_{1}}\\{f_{\mathrm{cov}}^{\ast}(X_{i})-\overline{f}_{\mathrm{cov}}^{\ast}\\}^{2}}{2\text{tr}\\{(\Sigma(S-I))^{2}\\}}\bigg{|}\mathcal{X}_{2}\bigg{\\}}$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\frac{1}{m_{1}^{2}}+\frac{(m_{1}-1)^{2}}{m_{1}^{2}}\Bigg{\\{}\frac{\mathbb{E}_{P}[|f_{\mathrm{cov}}^{\ast}(X_{1})|^{4}|\mathcal{X}_{2}]}{4m_{1}\text{tr}^{2}\\{(\Sigma(S-I))^{2}\\}}-\frac{(m_{1}-3)}{m_{1}(m_{1}-1)}\Bigg{\\}}.$

Now leveraging the expression (LABEL:Eq:_fourth_moment_of_g1) and Cauchy-
Schwarz inequality yield

$\displaystyle\frac{\mathbb{E}_{P}[|f_{\mathrm{cov}}^{\ast}(X_{1})|^{4}|\mathcal{X}_{2}]}{4m_{1}\text{tr}^{2}\\{(\Sigma(S-I))^{2}\\}}\leavevmode\nobreak\
\leq\leavevmode\nobreak\ \frac{15}{m_{1}}\quad\text{a.s.}$

Putting things together, we see that the convergence result (48) holds and
thus (D2) follows.

$\bullet$ Verification of (D3). We begin by analyzing the numerator of

$\displaystyle\frac{\sqrt{m_{1}}\text{tr}\\{(\Sigma-I)(S-I)\\}}{\sqrt{2\text{tr}\\{(\Sigma(S-I))^{2}\\}}}.$
(49)

The first two moments of the numerator are computed as

$\displaystyle\mathbb{E}_{P}[\text{tr}\\{(\Sigma-I)(S-I)\\}]\leavevmode\nobreak\
=\leavevmode\nobreak\ $ $\displaystyle\|\Sigma-I\|_{F}^{2},$
$\displaystyle\text{Var}_{P}[\text{tr}\\{(\Sigma-I)(S-I)\\}]\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\frac{2}{m_{2}}\text{tr}\\{(\Sigma-I)\Sigma(\Sigma-I)\Sigma\\}$
$\displaystyle\leavevmode\nobreak\ =\leavevmode\nobreak\ $
$\displaystyle\frac{2}{m_{2}}\sum_{i=1}^{d}\lambda_{i}^{2}(\Sigma)\\{\lambda_{i}(\Sigma)-1\\}^{2}$
$\displaystyle\leavevmode\nobreak\ \leq\leavevmode\nobreak\ $
$\displaystyle\frac{2\lambda_{d}^{2}(\Sigma)}{m_{2}}\|\Sigma-I\|_{F}^{2},$

where we recall that $\lambda_{1}(\Sigma)\leq\ldots\leq\lambda_{d}(\Sigma)$
are eigenvalues of $\Sigma$. Therefore

$\displaystyle\text{tr}\\{(\Sigma-I)(S-I)\\}$
$\displaystyle\leavevmode\nobreak\ =\leavevmode\nobreak\
\|\Sigma-I\|_{F}^{2}+O_{P}\left(\sqrt{\frac{2\lambda_{d}^{2}(\Sigma)}{m_{2}}}\|\Sigma-I\|_{F}\right)$
$\displaystyle\leavevmode\nobreak\ =\leavevmode\nobreak\
\|\Sigma-I\|_{F}^{2}+O_{P}\big{(}m_{2}^{-1/2}\big{)},$

where the second equality follows since the eigenvalues are uniformly bounded
and $\|\Sigma-I\|_{F}=O(1)$. Next we turn to the denominator of the quantity
(49). First of all, we compute the expectation

$\displaystyle\mathbb{E}_{P}[2\text{tr}\\{(\Sigma(S-I))^{2}\\}]\leavevmode\nobreak\
=\leavevmode\nobreak\ $ $\displaystyle
2\mathbb{E}_{P}[\text{tr}\\{(\Sigma(S-\Sigma))^{2}\\}]+4\mathbb{E}[\text{tr}\\{\Sigma(\Sigma-I)\Sigma(S-\Sigma)\\}]$
$\displaystyle+\leavevmode\nobreak\ $ $\displaystyle
2\sum_{i=1}^{d}\lambda_{i}^{2}(\Sigma)\\{\lambda_{i}(\Sigma)-1\\}^{2}$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\frac{2\text{tr}(\Sigma^{4})}{m_{2}}+\frac{2\text{tr}^{2}(\Sigma^{2})}{m_{2}}+2\sum_{i=1}^{d}\lambda_{i}^{2}(\Sigma)\\{\lambda_{i}(\Sigma)-1\\}^{2},$

where the second term asymptotically dominates the other terms under the given
conditions. It can be seen that the variance is bounded by

$\displaystyle\text{Var}_{P}[2\text{tr}\\{(\Sigma(S-I))^{2}\\}]\leavevmode\nobreak\
\lesssim\leavevmode\nobreak\ $
$\displaystyle\text{Var}_{P}[\text{tr}\\{(\Sigma(S-\Sigma))^{2}\\}]+\text{Var}_{P}[\text{tr}\\{\Sigma(\Sigma-I)\Sigma(S-\Sigma)\\}]$
$\displaystyle\lesssim\leavevmode\nobreak\ $
$\displaystyle\frac{1}{m_{2}^{3}}\Big{\\{}\text{tr}(\Sigma^{8})+\text{tr}^{2}(\Sigma^{4})+\text{tr}^{2}(\Sigma^{2})\text{tr}(\Sigma^{4})\Big{\\}}$
$\displaystyle+\leavevmode\nobreak\ $
$\displaystyle\frac{1}{m_{2}^{2}}\Big{\\{}\text{tr}^{2}(\Sigma^{8})+\text{tr}(\Sigma^{16})\Big{\\}}$
$\displaystyle+\leavevmode\nobreak\ $
$\displaystyle\frac{1}{m_{2}}\text{tr}(\Sigma^{8})+\frac{1}{m_{2}}\text{tr}\\{(\Sigma^{3}(\Sigma-I))^{2}\\}.$

Using the bounded eigenvalue condition along with the conditions on $\kappa$
and $\tau$, the variance is further upper bounded by

$\displaystyle\text{Var}_{P}[2\text{tr}\\{(\Sigma(S-I))^{2}\\}]\leavevmode\nobreak\
\lesssim\leavevmode\nobreak\
\frac{d^{3}}{m_{2}^{3}}+\frac{d^{2}}{m_{2}^{2}}+\frac{d}{m_{2}}=O(1).$

To summarize the results, we have obtained that

$\displaystyle\text{tr}\\{(\Sigma-I)(S-I)\\}\leavevmode\nobreak\
=\leavevmode\nobreak\ \|\Sigma-I\|_{F}^{2}+O_{P}\big{(}m_{2}^{-1/2}\big{)}$

and

$\displaystyle 2m_{1}^{-1}\text{tr}\\{(\Sigma(S-I))^{2}\\}\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\frac{2\text{tr}(\Sigma^{4})}{m_{1}m_{2}}+\frac{2\text{tr}^{2}(\Sigma^{2})}{m_{1}m_{2}}+\frac{2}{m_{1}}\sum_{i=1}^{d}\lambda_{i}^{2}(\Sigma)\\{\lambda_{i}(\Sigma)-1\\}^{2}+O_{P}\big{(}m_{1}^{-1}\big{)}$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\frac{2\text{tr}^{2}(\Sigma^{2})}{n^{2}\kappa(1-\kappa)}+O_{P}\big{(}n^{-1}\big{)}.$

Therefore the quantity (49) approximates

$\displaystyle\frac{\sqrt{m_{1}}\text{tr}\\{(\Sigma-I)(S-I)\\}}{\sqrt{2\text{tr}\\{(\Sigma(S-I))^{2}\\}}}=\frac{n\sqrt{\kappa(1-\kappa)}\|\Sigma-I\|_{F}^{2}}{\sqrt{2}\text{tr}(\Sigma^{2})}+o_{P}(1).$

Moreover the first term of the right-hand side is bounded, which verifies
(D3). This completes the proof of Theorem 5.

###  B.9 Proof of Proposition 3

The proof follows similar lines of argument to the proof of Theorem 3. First
of all, type I error control is a direct consequence of Proposition 2 and so
we focus on type II error control. Due to the continuity, the type II error
rate can be written as

$\displaystyle\mathbb{E}_{P}[1-\phi_{\mathrm{cov}}]=\mathbb{P}_{P}\Bigg{(}\overline{f}_{\mathrm{cov}}\leq
z_{1-\alpha}\sqrt{\frac{1}{m_{1}^{2}}\sum_{i=1}^{m_{1}}\big{\\{}f_{\mathrm{cov}}(X_{i})-\overline{f}_{\mathrm{cov}}\big{\\}}^{2}}\Bigg{)}.$

Next applying Markov’s inequality with the moment expression
(LABEL:Eq:_fourth_moment_of_g1) yields

$\displaystyle\mathbb{P}_{P}\left(\frac{1}{m_{1}^{2}}\sum_{i=1}^{m_{1}}\big{\\{}f_{\mathrm{cov}}(X_{i})-\overline{f}_{\mathrm{cov}}\big{\\}}^{2}\geq
t\right)\leavevmode\nobreak\ \leq\leavevmode\nobreak\ $
$\displaystyle\frac{2\mathbb{E}_{P}[\text{tr}\\{(\Sigma(S-I))^{2}\\}]}{tm_{1}}$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\frac{2}{t}\Bigg{[}\frac{\text{tr}(\Sigma^{4})}{m_{1}m_{2}}+\frac{\text{tr}^{2}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\sum_{i=1}^{d}\lambda_{i}^{2}(\lambda_{i}-1)^{2}}{m_{1}}\Bigg{]}.$

Then, for some $\beta>0$ (specified later), define the event $\mathcal{B}$
such that

$\displaystyle\mathcal{B}=\Bigg{\\{}\frac{1}{m_{1}^{2}}\sum_{i=1}^{m_{1}}\big{\\{}f_{\mathrm{cov}}(X_{i})-\overline{f}_{\mathrm{cov}}\big{\\}}^{2}<\frac{4}{\beta}\Bigg{[}\frac{\text{tr}(\Sigma^{4})}{m_{1}m_{2}}+\frac{\text{tr}^{2}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\sum_{i=1}^{d}\lambda_{i}^{2}(\lambda_{i}-1)^{2}}{m_{1}}\Bigg{]}\Bigg{\\}},$

which holds with probability at least $1-\beta/2$ due to Markov’s inequality.
On the other hand, the mean and the variance of $\overline{f}_{\mathrm{cov}}$
are

$\displaystyle\mathbb{E}_{P}[\overline{f}_{\mathrm{cov}}]\leavevmode\nobreak\
=\leavevmode\nobreak\ $ $\displaystyle\|\Sigma-I\|_{F}^{2},$
$\displaystyle\text{Var}_{P}[\overline{f}_{\mathrm{cov}}]\leavevmode\nobreak\
\leq\leavevmode\nobreak\ $
$\displaystyle\frac{\text{tr}(\Sigma^{4})}{m_{1}m_{2}}+\frac{\text{tr}^{2}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\sum_{i=1}^{d}\lambda_{i}^{2}(\lambda_{i}-1)^{2}}{m_{1}}.$

Since $\text{tr}(\Sigma^{4})\leq\text{tr}^{2}(\Sigma^{2})$ and
$\text{tr}^{2}(\Sigma^{2})\leq 8\|\Sigma-I\|_{F}^{4}+8d^{2}$, which can be
verified by the basic inequality $(a+b)^{2}\leq 2a^{2}+2b^{2}$ for all
$a,b\in\mathbb{R}$, we can further upper bound the variance as

$\displaystyle\text{Var}_{P}[\overline{f}_{\mathrm{cov}}]\leavevmode\nobreak\
\lesssim\leavevmode\nobreak$
$\displaystyle\frac{\|\Sigma-I\|_{F}^{4}}{m_{1}m_{2}}+\frac{d^{2}}{m_{1}m_{2}}+\frac{\lambda_{1}^{2}\|\Sigma-I\|_{F}^{2}}{m_{1}}$
(50) $\displaystyle\leavevmode\nobreak\ \lesssim\leavevmode\nobreak$
$\displaystyle\frac{\|\Sigma-I\|_{F}^{4}}{n^{2}}+\frac{d^{2}}{n^{2}}+\frac{\|\Sigma-I\|_{F}^{4}}{n}+\frac{\|\Sigma-I\|_{F}^{2}}{n},$

where the second inequality uses the condition that $\kappa_{\ast}\leq
m_{1}/n\leq\kappa^{\ast}$ for some $\kappa_{\ast},\kappa^{\ast}>0$ and
$\lambda_{1}\leq\|\Sigma-I\|_{F}+1$.

Suppose that $z_{1-\alpha}>0$. Armed with the preliminary results, we can
bound the type II error of $\phi_{\mathrm{cov}}$ by

$\displaystyle\mathbb{E}_{P}[1-\phi_{\mathrm{cov}}]\leavevmode\nobreak\
\leq\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\left(\overline{f}_{\mathrm{cov}}\leq
z_{1-\alpha}\sqrt{\frac{4}{\beta}\Bigg{[}\frac{\text{tr}(\Sigma^{4})}{m_{1}m_{2}}+\frac{\text{tr}^{2}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\sum_{i=1}^{d}\lambda_{i}^{2}(\lambda_{i}-1)^{2}}{m_{1}}\Bigg{]}}\right)+\mathbb{P}_{P}(\mathcal{B}^{c})$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\left(\overline{f}_{\mathrm{cov}}\leq
z_{1-\alpha}\sqrt{\frac{4}{\beta}\Bigg{[}\frac{\text{tr}(\Sigma^{4})}{m_{1}m_{2}}+\frac{\text{tr}^{2}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\sum_{i=1}^{d}\lambda_{i}^{2}(\lambda_{i}-1)^{2}}{m_{1}}\Bigg{]}}\right)+\frac{\beta}{2}.$

For a small value of $\delta>0$ such that $1-\delta>0$, we take
$\beta=\max\\{n^{\delta-1},K_{n}^{-1}\\}$. Then by the inequality (50) and
taking sufficiently large $n$ and $K_{n}$, we can bound

$\displaystyle
z_{1-\alpha}\sqrt{\frac{4}{\beta}\Bigg{[}\frac{\text{tr}(\Sigma^{4})}{m_{1}m_{2}}+\frac{\text{tr}^{2}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\sum_{i=1}^{d}\lambda_{i}^{2}(\lambda_{i}-1)^{2}}{m_{1}}\Bigg{]}}$
$\displaystyle\lesssim\leavevmode\nobreak\ $
$\displaystyle\frac{z_{1-\alpha}\|\Sigma-I\|_{F}^{2}}{\sqrt{\beta
n}}+\frac{z_{1-\alpha}d}{\sqrt{\beta}n}+\frac{z_{1-\alpha}\|\Sigma-I\|_{F}}{\sqrt{\beta
n}}$ $\displaystyle\lesssim\leavevmode\nobreak\ $
$\displaystyle\frac{z_{1-\alpha}\|\Sigma-I\|_{F}^{2}}{\sqrt{n^{\delta}}}+\frac{K_{n}^{1/2}z_{1-\alpha}d}{n}+\frac{K_{n}^{1/2}z_{1-\alpha}\|\Sigma-I\|_{F}}{\sqrt{n}},$

where the last inequality uses our choice of $\beta$. Therefore, for large $n$
and $K_{n}$, we have

$\displaystyle
z_{1-\alpha}\sqrt{\frac{4}{\beta}\Bigg{[}\frac{\text{tr}(\Sigma^{4})}{m_{1}m_{2}}+\frac{\text{tr}^{2}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\sum_{i=1}^{d}\lambda_{i}^{2}(\lambda_{i}-1)^{2}}{m_{1}}\Bigg{]}}\leq\frac{\|\Sigma-I\|_{F}^{2}}{2},$

since $\|\Sigma-I\|_{F}\geq K_{n}\sqrt{d/n}$. Then building on Chebyshev’s
inequality together with the above inequality yields

$\displaystyle\mathbb{P}_{P}\left(\overline{f}_{\mathrm{cov}}\leq
z_{1-\alpha}\sqrt{\frac{4}{\beta}\Bigg{[}\frac{\text{tr}(\Sigma^{4})}{m_{1}m_{2}}+\frac{\text{tr}^{2}(\Sigma^{2})}{m_{1}m_{2}}+\frac{\sum_{i=1}^{d}\lambda_{i}^{2}(\lambda_{i}-1)^{2}}{m_{1}}\Bigg{]}}\right)$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\mathbb{P}_{P}\left(\overline{f}_{\mathrm{cov}}-\|\Sigma-I\|_{F}^{2}\leq-\frac{\|\Sigma-I\|_{F}^{2}}{2}\right)$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\frac{4\text{Var}_{P}[\overline{f}_{\mathrm{cov}}]}{\|\Sigma-I\|_{F}^{4}}\leavevmode\nobreak\
\lesssim\leavevmode\nobreak\
\frac{1}{\|\Sigma-I\|_{F}^{4}}\Bigg{[}\frac{\|\Sigma-I\|_{F}^{4}}{n^{2}}+\frac{d^{2}}{n^{2}}+\frac{\|\Sigma-I\|_{F}^{4}}{n}+\frac{\|\Sigma-I\|_{F}^{2}}{n}\Bigg{]}$
$\displaystyle\lesssim\leavevmode\nobreak\ $
$\displaystyle\frac{1}{n^{2}}+\frac{1}{K_{n}^{4}}+\frac{1}{n}+\frac{1}{K_{n}^{2}}.$

Furthermore, since $\beta=\max\\{n^{\delta-1},K_{n}^{-1}\\}\rightarrow 0$, we
conclude that

$\displaystyle\lim_{n\rightarrow\infty}\sup_{P\in\mathcal{Q}_{\mathrm{cov}}(\epsilon_{n})}\mathbb{E}_{P}[1-\phi_{\mathrm{cov}}]=0.$

Since the type II error rate is decreasing in $\alpha$, the result also
follows when $z_{1-\alpha}<0$. This completes the proof of Proposition 3.

###  B.10 Proof of Theorem 6

Our strategy to prove Theorem 6 is as follows. We begin by considering the
conditionally standardized statistic, that is

$\displaystyle
T_{\text{stand},h}=\frac{\sqrt{m}\overline{f}_{h}}{\sqrt{\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]}},$

and show that it converges to $N(0,1)$ in distribution. Next we establish that
the difference between $T_{\text{stand},h}$ and $T_{h}$ defined in (34) is
asymptotically negligible and hence $T_{h}$ converges to $N(0,1)$ as well.

In the proof below, the probability measure $P$, kernel $h$ and other
associated quantities such as a sequence of eigenvalues
$\\{\lambda_{i}\\}_{i=1}^{\infty}$ are allowed to change with $n$. Hence these
are implicitly dependent on $n$. For notational convenience, however, the
dependence on $n$ will be suppressed. We also assume that $2m=n$ in this
proof, which does not affect the asymptotic result.

$\bullet$ Verification of Lyapunov condition for $T_{\text{stand},h}$. Notice
that $\overline{f}_{h}$ is the sample mean of i.i.d. random variables
conditional on $\mathcal{X}_{2}$. Therefore Lyapunov condition shows that if

$\displaystyle\frac{\mathbb{E}_{P}[f_{h}^{4}(X_{1})|\mathcal{X}_{2}]}{\\{\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]\\}^{2}m}\overset{p}{\longrightarrow}0,$
(51)

the standardized statistic $T_{\text{stand},h}$ converges to $N(0,1)$
conditional on $\mathcal{X}_{2}$. Furthermore, since the limiting distribution
is continuous, Lemma 2.11 of Van der Vaart, (2000) shows

$\displaystyle\sup_{-\infty<z<\infty}\big{|}\mathbb{P}_{P}(T_{\text{stand},h}\leq
z|\mathcal{X}_{2})-\Phi(z)\big{|}\overset{p}{\longrightarrow}0.$

Additionally, since $X_{n}\overset{p}{\longrightarrow}0$ implies
$\mathbb{E}_{P}[X_{n}]\rightarrow 0$ when $X_{n}$ is bounded, we also have

$\displaystyle\sup_{-\infty<z<\infty}\big{|}\mathbb{P}_{P}(T_{\text{stand},h}\leq
z)-\Phi(z)\big{|}\rightarrow 0.$

Hence it is enough to show (51) to establish the unconditional central limit
theorem for $T_{\text{stand},h}$. To prove that Lyapunov condition (51) holds
under the given conditions, let us consider the following decomposition:

$\displaystyle\frac{\mathbb{E}_{P}[f_{h}^{4}(X_{1})|\mathcal{X}_{2}]}{\\{\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]\\}^{2}m}=\underbrace{\frac{m\mathbb{E}_{P}[f_{h}^{4}(X_{1})|\mathcal{X}_{2}]}{\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}}}_{(I)}\times\underbrace{\frac{\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}}{\\{\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]\\}^{2}m^{2}}}_{(II)}.$
(52)

Note that the expectation of the first term $(I)$ is

$\displaystyle\mathbb{E}_{P}[(I)]\leavevmode\nobreak\ =\leavevmode\nobreak\ $
$\displaystyle\frac{m\mathbb{E}_{P}[\mathbb{E}_{P}[f_{h}^{4}(X_{1})|\mathcal{X}_{2}]]}{\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}}$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\frac{m}{\\{\mathbb{E}_{P}[h^{2}(X,Y)]\\}^{2}}\Bigg{\\{}\frac{1}{m^{3}}\mathbb{E}_{P}[h^{4}(X_{1},X_{2})]+\frac{3m(m-1)}{m^{4}}\mathbb{E}_{P}[h^{2}(X_{1},X_{3})h^{2}(X_{2},X_{3})]\Bigg{\\}}.$

Therefore, by Markov’s inequality, the first term converges to zero in
probability when

$\displaystyle\frac{\mathbb{E}_{P}[h^{4}(X_{1},X_{2})]n^{-1}+\mathbb{E}_{P}[h^{2}(X_{1},X_{3})h^{2}(X_{2},X_{3})]}{n\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}}\rightarrow
0.$ (53)

In fact this is exactly the condition (29b). Hence all we need to show is that
the second term $(II)$ is stochastically bounded. Recall that
$\lambda^{\dagger}$ is the limiting value of
$\lambda_{1}/\sqrt{\sum_{k=1}^{\infty}\lambda_{k}}$. For the second term, we
need a different analysis depending on whether $\lambda^{\dagger}>0$ or
$\lambda^{\dagger}=0$.

1) When $\lambda^{\dagger}>0$. We start with the case where
$\lambda^{\dagger}>0$. Recall that $h(x,y)$ can be written as

$\displaystyle h(x,y)=\sum_{k=1}^{\infty}\lambda_{k}\phi_{k}(x)\phi_{k}(y).$

Therefore we may write

$\displaystyle\frac{m\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]}{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]}=\sum_{k=1}^{\infty}\widetilde{\lambda}_{k}\Bigg{(}\bigg{\\{}\frac{1}{\sqrt{m}}\sum_{j=m+1}^{n}\phi_{k}(X_{j})\bigg{\\}}\Bigg{)}^{2},$

where
$\widetilde{\lambda}_{k}=\lambda_{k}^{2}/\sum_{k^{\prime}=1}^{\infty}\lambda_{k^{\prime}}^{2}$.
Then we notice that

$\displaystyle(II)^{1/2}\leavevmode\nobreak\ =\leavevmode\nobreak\ $
$\displaystyle\frac{1}{\sum_{k=1}^{\infty}\widetilde{\lambda}_{k}\big{\\{}\frac{1}{\sqrt{m}}\sum_{j=m+1}^{n}\phi_{k}(X_{j})\big{\\}}^{2}}$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\frac{1}{\widetilde{\lambda}_{1}\big{\\{}\frac{1}{\sqrt{m}}\sum_{j=m+1}^{n}\phi_{1}(X_{j})\big{\\}}^{2}},$

which is stochastically bounded as we prove below. Using the
eigendecomposition (28) and Fubini–Tonelli’s theorem yields

$\displaystyle\frac{\mathbb{E}_{P}[h^{2}(X_{1},X_{3})h^{2}(X_{2},X_{3})]}{n\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}}\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\frac{\sum_{k=1}^{\infty}\lambda_{k}^{4}\mathbb{E}_{P}[\phi_{k}^{4}(X)]+\sum_{k=1,k^{\prime}=1,k\neq
k^{\prime}}^{\infty}\lambda_{k}^{2}\lambda_{k^{\prime}}^{2}\mathbb{E}_{P}[\phi_{k}^{2}(X)\phi_{k^{\prime}}^{2}(X)]}{n\\{\sum_{k=1}^{\infty}\lambda_{i}^{2}\\}^{2}}$
$\displaystyle\geq\leavevmode\nobreak\ $
$\displaystyle\widetilde{\lambda}_{1}\frac{\mathbb{E}_{P}[\phi_{1}^{4}(X)]}{n}.$

By the condition (29b) and
$\widetilde{\lambda}_{1}\rightarrow(\lambda^{\dagger})^{2}>0$, we have
$n^{-1}\mathbb{E}_{P}[\phi_{1}^{4}(X)]\rightarrow 0$. This guarantees that
$\frac{1}{\sqrt{m}}\sum_{j=m+1}^{n}\phi_{1}(X_{j})$ converges to $N(0,1)$ by
Lyapunov central limit theorem. Furthermore, by Slutsky’s theorem and
continuous mapping theorem,

$\displaystyle\frac{1}{\widetilde{\lambda}_{1}\big{\\{}\frac{1}{\sqrt{m}}\sum_{j=m+1}^{n}\phi_{1}(X_{j})\big{\\}}^{2}}\overset{d}{\longrightarrow}\frac{1}{(\lambda^{\dagger})^{2}\xi^{2}}=O_{P}(1),$

where $\xi\sim N(0,1)$. This result implies that the second term is
stochastically bounded, i.e. $(II)=O_{P}(1)$. Combining this with (53) yields
our claim (51).

2) When $\lambda^{\dagger}=0$. Next we consider the setting where
$\lambda^{\dagger}=0$. Write

$\displaystyle g(x,y)=\mathbb{E}_{P}[h(x,X)h(y,X)].$

Then after some algebra, we obtain the identity

$\displaystyle\frac{m\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]}{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]}\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\frac{1}{m}\sum_{j=m+1}^{n}\frac{g(X_{j},X_{j})}{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]}+\frac{1}{m}\sum_{m+1\leq
j_{1}\neq j_{2}\leq
n}\frac{g(X_{j_{1}},X_{j_{2}})}{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]}.$

The mean and the variance of the above quantity satisfy

$\displaystyle\mathbb{E}_{P}\bigg{\\{}\frac{m\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]}{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]}\bigg{\\}}\leavevmode\nobreak\
=\leavevmode\nobreak\ 1\quad\text{and}$
$\displaystyle\text{Var}_{P}\bigg{\\{}\frac{m\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]}{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]}\bigg{\\}}\leavevmode\nobreak\
\lesssim\leavevmode\nobreak\
\frac{\mathbb{E}_{P}[h^{2}(X_{1},X_{3})h^{2}(X_{2},X_{3})]}{m\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}}+\frac{\mathbb{E}_{P}[g^{2}(X_{1},X_{2})]}{\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}}.$
(54)

Consequently, as long as the right-hand side of (54) converges to zero, we
have

$\displaystyle(II)=\frac{1}{1+o_{P}(1)}=O_{P}(1).$

Combining this with the previous result (53), we see that when

$\displaystyle\frac{\mathbb{E}_{P}[h^{4}(X_{1},X_{2})]n^{-1}+\mathbb{E}_{P}[h^{2}(X_{1},X_{3})h^{2}(X_{2},X_{3})]}{n\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}}\rightarrow
0\quad\text{and}$
$\displaystyle\frac{\mathbb{E}_{P}[g^{2}(X_{1},X_{2})]}{\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}}=\frac{\sum_{k=1}^{\infty}\lambda_{k}^{4}}{\\{\sum_{k=1}^{\infty}\lambda_{k}^{2}\\}^{2}}\leq\frac{\lambda_{1}^{2}}{\sum_{k=1}^{\infty}\lambda_{k}^{2}}\rightarrow(\lambda^{\dagger})^{2}=0.$

This verifies the claim (51).

$\bullet$ Central Limit Theorem for $T_{h}$. So far we have established that
$T_{\text{stand},h}$ converges to $N(0,1)$. Therefore, once we prove

$\displaystyle\frac{m^{-1}\sum_{i=1}^{m}\\{f_{h}(X_{i})-\overline{f}_{h}\\}^{2}}{\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]}\overset{p}{\longrightarrow}1,$
(55)

Slutsky’s theorem together with the continuous mapping theorem implies that
$T_{h}$ converges to $N(0,1)$ as well. To show this, let us consider the
decomposition

$\displaystyle\frac{(m-1)^{-1}\sum_{i=1}^{m}\\{f_{h}(X_{i})-\overline{f}_{h}\\}^{2}-\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]}{\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]}$
$\displaystyle=\leavevmode\nobreak\ $
$\displaystyle\underbrace{\frac{\sum_{i=1}^{m}\\{f_{h}(X_{i})-\overline{f}_{h}\\}^{2}-(m-1)\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]}{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]}}_{(I)^{\prime}}\times\underbrace{\frac{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]}{(m-1)\mathbb{E}_{P}[f_{h}^{2}(X_{1})|\mathcal{X}_{2}]}}_{(II)^{\prime}}.$

From the previous calculations for verifying (51), we already know that
$(II)^{\prime}=O_{P}(1)$. Therefore all we need to show is
$(I)^{\prime}\overset{p}{\longrightarrow}0$. Note that the expectation of
$(I)^{\prime}$ is equal to zero and the variance of $(I)^{\prime}$ can be
bounded by

$\displaystyle\text{Var}_{P}[(I)^{\prime}]\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\frac{(m-1)^{2}}{\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}}\mathbb{E}\Bigg{\\{}\text{Var}_{P}\bigg{[}\frac{1}{m-1}\sum_{i=1}^{m}\\{f_{h}(X_{i})-\overline{f}_{h}\\}^{2}\Big{|}\mathcal{X}_{2}\bigg{]}\Bigg{\\}}$
$\displaystyle\leq\leavevmode\nobreak\ $
$\displaystyle\frac{m\mathbb{E}_{P}[f_{h}^{4}(X_{1})]}{\\{\mathbb{E}_{P}[h^{2}(X_{1},X_{2})]\\}^{2}},$

where the inequality uses the variance formula for the sample variance (e.g.
Lee,, 1990). The upper bound is equivalent to the expectation of $(I)$ in
(52), which converges to zero as we proved before. As a result, we see that
$(I)^{\prime}\times(II)^{\prime}\overset{p}{\longrightarrow}0$ that further
implies our claim (55) since $(m-1)/m\rightarrow 1$.

$\bullet$ Concluding the proof. Following our previous argument, which holds
for a row-wise i.i.d. triangular array, we have

$\displaystyle\sup_{-\infty<z<\infty}\big{|}\mathbb{P}_{P_{n}}(T_{h}\leq
z)-\Phi(z)\big{|}\rightarrow 0,$ (56)

for any sequence $P_{n}$ in $\mathcal{P}_{n,h}$. To finish the proof, for each
$n$, choose $P_{n}^{\prime}\in\mathcal{P}_{n,h}$ such that

$\displaystyle\sup_{P\in\mathcal{P}_{n,h}}\sup_{-\infty<z<\infty}\big{|}\mathbb{P}_{P}(T_{h}\leq
z)-\Phi(z)\big{|}\leq\sup_{-\infty<z<\infty}\big{|}\mathbb{P}_{P_{n}^{\prime}}(T_{h}\leq
z)-\Phi(z)\big{|}+n^{-1}.$

Now it follows that the right-hand side converges to zero based on the result
(56) and thus we arrive at the conclusion of Theorem 6.

###  B.11 Proof of Proposition 4

We again assume that $2m=n$ for simplicity, which does not affect our
asymptotic results. We start with type I error control and then move to type
II error control. For this proof, we often need to take the expectation
involving random variables from different distributions and so we suppress
dependence on $P$ in the expectation to simplify the notation.

$\bullet$ Type I error control. We use Theorem 6 to prove type I error
control. More specifically we verify that the kernel (36) satisfies
$\mathbb{E}[h_{\text{Gau},\nu_{n}}(X,y)]=0$, the condition (29b) and
$\lambda_{1}^{2}/\sum_{k=1}^{\infty}\lambda_{k}^{2}\rightarrow 0$ under the
given assumptions. The first condition is trivially true by the definition of
the kernel. The latter two conditions were in fact verified in Li and Yuan,
(2019) (see the proof of their theorem 1) by noting that the condition (29a)
is equivalent to
$\lambda_{1}^{2}/\sum_{k=1}^{\infty}\lambda_{k}^{2}\rightarrow 0$ since

$\displaystyle\Bigg{[}\frac{\lambda_{1}^{2}}{\sum_{k=1}^{\infty}\lambda_{k}^{2}}\Bigg{]}^{2}\leq\frac{\sum_{k=1}^{\infty}\lambda_{k}^{4}}{\big{\\{}\sum_{k=1}^{\infty}\lambda_{k}^{2}\big{\\}}^{2}}\leq\frac{\lambda_{1}^{2}}{\sum_{k=1}^{\infty}\lambda_{k}^{2}}.$

Hence the test statistic converges to the standard normal distribution and the
result follows.

$\bullet$ Type II error control. To prove type II error control, we recall
that the type II error is

$\displaystyle\mathbb{P}(T_{\text{MMD}}\leq
z_{1-\alpha})=\mathbb{P}\bigg{\\{}\overline{f}_{h}\leq
z_{1-\alpha}\bigg{[}m^{-2}\sum_{i=1}^{m}\\{f_{h}(X_{i})-\overline{f}_{h}\\}^{2}\bigg{]}^{1/2}\bigg{\\}},$

where $f_{h}$ is defined in (33) with $h(x,y)=h_{\text{Gau},\nu_{n}}(x,y)$ and
$\overline{f}_{h}=m^{-1}\sum_{i=1}^{m}f_{h}(X_{i})$. Following the proof of
Theorem 3, the type II error of the test can be arbitrary small once the
expectation of $\overline{f}_{h}$ is sufficiently larger than its standard
deviation and its threshold. More specifically, suppose that

$\displaystyle\big{\\{}\mathbb{E}\big{[}\overline{f}_{h}\big{]}\big{\\}}^{2}\leavevmode\nobreak\
\gtrsim\leavevmode\nobreak\
\text{Var}[\overline{f}_{h}]\quad\text{and}\quad\big{\\{}\mathbb{E}\big{[}\overline{f}_{h}\big{]}\big{\\}}^{2}\leavevmode\nobreak\
\gtrsim\leavevmode\nobreak\
\mathbb{E}\bigg{[}m^{-2}\sum_{i=1}^{m}\\{f_{h}(Y_{i})-\overline{f}_{h}\\}^{2}\bigg{]}.$
(57)

Then by following the details in the proof of Theorem 3, Chebyshev’s
inequality along with Markov inequality shows that

$\displaystyle\lim_{n\rightarrow\infty}\sup_{P\in\mathcal{Q}_{\text{density}}(\epsilon_{n})}\mathbb{P}_{P}(T_{\text{MMD}}\leq
z_{1-\alpha})=0.$

In what follows, we analyze $\overline{f}_{h}$ and the sample variance of
$f_{h}$ and show that both of the conditions in (57) hold.

1) Verification of the first condition in (57). To start with the mean, note
that the expectation of $\overline{f}_{h}$ is the same as the expectation of
the U-statistic in Li and Yuan, (2019). Therefore we can use the result of Li
and Yuan, (2019) and obtain

$\displaystyle\mathbb{E}[\overline{f}_{h}]\leavevmode\nobreak\
\gtrsim\leavevmode\nobreak\ \nu_{n}^{-d/2}\|p-q\|_{L_{2}}^{2}\quad\text{for
sufficiently large $n$.}$ (58)

Consider the decomposition of $h_{\text{Gau},\nu_{n}}(x,y)$

$\displaystyle h_{\text{Gau},\nu_{n}}(x,y)\leavevmode\nobreak\
=\leavevmode\nobreak$ $\displaystyle
G_{\nu_{n}}(x,y)-\mathbb{E}[G_{\nu_{n}}(x,Y)]-\mathbb{E}[G_{\nu_{n}}(Y,y)]+\mathbb{E}[G_{\nu_{n}}(Y,Y^{\prime})]$
(59) $\displaystyle-\leavevmode\nobreak$
$\displaystyle\mathbb{E}[G_{\nu_{n}}(x,X)]+\mathbb{E}[G_{\nu_{n}}(x,Y)]+\mathbb{E}[G_{\nu_{n}}(Y,X)]-\mathbb{E}[G_{\nu_{n}}(Y,Y^{\prime})]$
$\displaystyle-\leavevmode\nobreak$
$\displaystyle\mathbb{E}[G_{\nu_{n}}(X,y)]+\mathbb{E}[G_{\nu_{n}}(y,Y)]+\mathbb{E}[G_{\nu_{n}}(Y,X)]-\mathbb{E}[G_{\nu_{n}}(Y,Y^{\prime})]$
$\displaystyle+\leavevmode\nobreak$
$\displaystyle\mathbb{E}[h_{\text{Gau},\nu_{n}}(Y,Y^{\prime})],$

where $X,X^{\prime}\sim q$ and $Y,Y^{\prime}\sim p$. Let
$Y_{1},\ldots,Y_{n}\overset{i.i.d.}{\sim}p$. Then the variance of
$\overline{f}_{h}$ is bounded by

$\displaystyle\text{Var}[\overline{f}_{h}]\leavevmode\nobreak\
\lesssim\leavevmode\nobreak\ $
$\displaystyle\text{Var}\Bigg{[}\frac{1}{m^{2}}\sum_{i=1}^{m}\sum_{j=m+1}^{n}G_{\nu_{n}}(Y_{i},Y_{j})-\mathbb{E}[G_{\nu_{n}}(Y_{i},Y)]-\mathbb{E}[G_{\nu_{n}}(Y,Y_{j})]+\mathbb{E}[G_{\nu_{n}}(Y,Y^{\prime})]\Bigg{]}$
$\displaystyle+\leavevmode\nobreak\ $
$\displaystyle\text{Var}\Bigg{[}\frac{1}{m}\sum_{i=1}^{m}\mathbb{E}[G_{\nu_{n}}(Y_{i},X)]-\mathbb{E}[G_{\nu_{n}}(Y_{i},Y)]-\mathbb{E}[G_{\nu_{n}}(Y,X)]+\mathbb{E}[G_{\nu_{n}}(Y,Y^{\prime})]\Bigg{]}$
$\displaystyle+\leavevmode\nobreak\ $
$\displaystyle\text{Var}\Bigg{[}\frac{1}{m}\sum_{j=m+1}^{n}\mathbb{E}[G_{\nu_{n}}(Y_{j},X)]-\mathbb{E}[G_{\nu_{n}}(Y_{j},Y)]-\mathbb{E}[G_{\nu_{n}}(Y,X)]+\mathbb{E}[G_{\nu_{n}}(Y,Y^{\prime})]\Bigg{]}$
$\displaystyle\lesssim\leavevmode\nobreak\ $
$\displaystyle\frac{M^{2}}{n^{2}\nu_{n}^{d/2}}+\frac{M}{n\nu_{n}^{3d/4}}\|p-q\|_{L_{2}}^{2},$

where the last inequality follows by Li and Yuan, (2019). Combining this upper
bound with the lower bound (58), we see that the first condition of (57) is
fulfilled when $\|p-q\|_{L_{2}}\gtrsim\nu_{n}^{d/8}/\sqrt{n}$. Now recall that
$\nu_{n}\asymp n^{4/(d+4s)}$ and $\|p-q\|_{L_{2}}\geq\epsilon_{n}\gtrsim
n^{2s/(d+4s)}$ by the given conditions. Hence the first condition of (57)
holds.

2) Verification of the second condition in (57). To verify the second
condition, notice that

$\displaystyle\mathbb{E}\bigg{[}m^{-2}\sum_{i=1}^{m}\\{f_{h}(Y_{i})-\overline{f}_{h}\\}^{2}\bigg{]}\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\frac{m-1}{m^{2}}\mathbb{E}\\{\text{Var}[f_{h}(Y_{1})|Y_{m+1},\ldots,Y_{n}]\\}$
$\displaystyle\leavevmode\nobreak\ =\leavevmode\nobreak\ $
$\displaystyle\frac{m-1}{m^{2}}\mathbb{E}\bigg{\\{}\text{Var}\bigg{[}\frac{1}{m}\sum_{j=m+1}^{n}h_{\text{Gau},\nu_{n}}(Y_{1},Y_{j})|Y_{m+1},\ldots,Y_{n}\bigg{]}\bigg{\\}}.$

Now using the decomposition (59),

$\displaystyle\mathbb{E}\bigg{\\{}\text{Var}\bigg{[}\frac{1}{m}\sum_{j=m+1}^{n}h_{\text{Gau},\nu_{n}}(Y_{1},Y_{j})|Y_{m+1},\ldots,Y_{n}\bigg{]}\bigg{\\}}$
$\displaystyle\lesssim\leavevmode\nobreak\ $
$\displaystyle\mathbb{E}\bigg{\\{}\text{Var}\bigg{[}\frac{1}{m}\sum_{j=m+1}^{n}G_{\nu_{n}}(Y_{1},Y_{j})-\mathbb{E}[G_{\nu_{n}}(Y_{1},Y)]\big{|}Y_{m+1},\ldots,Y_{n}\bigg{]}\bigg{\\}}$
$\displaystyle+\leavevmode\nobreak\ $
$\displaystyle\text{Var}\Big{\\{}\mathbb{E}[G_{\nu_{n}}(Y_{1},X)]-\mathbb{E}[G_{\nu_{n}}(Y_{1},Y)]\Big{\\}}$
$\displaystyle\lesssim\leavevmode\nobreak\ $
$\displaystyle\frac{1}{n}\mathbb{E}[G_{\nu_{n}}^{2}(Y_{1},Y_{2})]+\mathbb{E}\Big{\\{}\mathbb{E}[G_{\nu_{n}}(Y_{1},X)]-\mathbb{E}[G_{\nu_{n}}(Y_{1},Y)]\Big{\\}}^{2}$
$\displaystyle\lesssim\leavevmode\nobreak\ $
$\displaystyle\frac{M^{2}}{n\nu_{n}^{d/2}}+\frac{M}{\nu_{n}^{3d/4}}\|p-q\|_{L_{2}}^{2}.$

Combining this upper bound with the lower bound (58), we again see that the
second condition of (57) is satisfied once
$\|p-q\|_{L_{2}}\gtrsim\nu_{n}^{d/8}/\sqrt{n}$. As noted before, this follows
under the given conditions. This completes the proof of Proposition
Proposition 4.

###  B.12 Proof of Proposition 5

By the multivariate central limit theorem, we have

$\displaystyle\sqrt{m}\begin{pmatrix}\widehat{\mu}_{1}\\\
\widehat{\mu}_{2}\end{pmatrix}\overset{d}{\longrightarrow}N\left(\begin{bmatrix}0\\\
0\end{bmatrix},\begin{bmatrix}\Sigma&0\\\ 0&\Sigma\end{bmatrix}\right).$

For nonzero $x,y\in\mathbb{R}^{d}$, let us define
$g(x,y):=x^{\top}y/\sqrt{x^{\top}\Sigma x}+x^{\top}y/\sqrt{y^{\top}\Sigma y}$.
Then the continuous mapping theorem yields

$\displaystyle\frac{m\widehat{\mu}_{1}^{\top}\widehat{\mu}_{2}}{\sqrt{m\widehat{\mu}_{1}^{\top}\Sigma\widehat{\mu}_{1}}}+\frac{m\widehat{\mu}_{1}^{\top}\widehat{\mu}_{2}}{\sqrt{m\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}}\overset{d}{\longrightarrow}\frac{\bm{\xi}_{1}^{\top}\Sigma\bm{\xi}_{2}}{\sqrt{\bm{\xi}_{1}^{\top}\Sigma^{2}\bm{\xi}_{1}}}+\frac{\bm{\xi}_{1}^{\top}\Sigma\bm{\xi}_{2}}{\sqrt{\bm{\xi}_{2}^{\top}\Sigma^{2}\bm{\xi}_{2}}}.$
(60)

Next let
$S_{1}:=m^{-1}\sum_{i=1}^{m}(X_{i}-\widehat{\mu}_{1})(X_{i}-\widehat{\mu}_{1})^{\top}$
and
$S_{2}:=m^{-1}\sum_{i=m+1}^{n}(X_{i}-\widehat{\mu}_{2})(X_{i}-\widehat{\mu}_{2})^{\top}$.
Since the dimension is fixed, the law of large numbers shows that
$\|S_{i}-\Sigma\|_{\text{op}}=o_{P}(1)$ for $i=1,2$ where $\|A\|_{\text{op}}$
denotes the operator norm of a matrix $A$. Since we can write

$\displaystyle
T_{\text{mean}}^{\text{cross}}=\frac{m\widehat{\mu}_{1}^{\top}\widehat{\mu}_{2}}{\sqrt{m\widehat{\mu}_{1}^{\top}(S_{2}-\Sigma)\widehat{\mu}_{1}+m\widehat{\mu}_{1}^{\top}\Sigma\widehat{\mu}_{1}}}+\frac{m\widehat{\mu}_{1}^{\top}\widehat{\mu}_{2}}{\sqrt{m\widehat{\mu}_{2}^{\top}(S_{1}-\Sigma)\widehat{\mu}_{2}+m\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}}$

and $|m\widehat{\mu}_{i}^{\top}(S_{j}-\Sigma)\widehat{\mu}_{i}|\leq
m\widehat{\mu}_{i}^{\top}\widehat{\mu}_{i}\times\|S_{j}-\Sigma\|_{\text{op}}=o_{P}(1)$
for $i\neq j\in\\{1,2\\}$,

$\displaystyle T_{\text{mean}}^{\text{cross}}\leavevmode\nobreak\
=\leavevmode\nobreak\ $
$\displaystyle\frac{m\widehat{\mu}_{1}^{\top}\widehat{\mu}_{2}}{\sqrt{m\widehat{\mu}_{1}^{\top}\Sigma\widehat{\mu}_{1}+o_{P}(1)}}+\frac{m\widehat{\mu}_{1}^{\top}\widehat{\mu}_{2}}{\sqrt{m\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}+o_{P}(1)}}$
$\displaystyle\leavevmode\nobreak\ =\leavevmode\nobreak\ $
$\displaystyle\frac{m\widehat{\mu}_{1}^{\top}\widehat{\mu}_{2}}{\sqrt{m\widehat{\mu}_{1}^{\top}\Sigma\widehat{\mu}_{1}}}+\frac{m\widehat{\mu}_{1}^{\top}\widehat{\mu}_{2}}{\sqrt{m\widehat{\mu}_{2}^{\top}\Sigma\widehat{\mu}_{2}}}+o_{P}(1),$

by the Taylor expansion. Finally Slutsky’s theorem along with the convergence
(60) proves the claim.


# Low-Resource Music Genre Classification with
Cross-Modal Neural Model Reprogramming

###### Abstract

Transfer learning (TL) approaches have shown promising results when handling
tasks with limited training data. However, considerable memory and
computational resources are often required for fine-tuning pre-trained neural
networks with target domain data. In this work, we introduce a novel method
for leveraging pre-trained speech models for low-resource music classification
based on the concept of Neural Model Reprogramming (NMR). NMR aims at re-
purposing a pre-trained model from a source domain to a target domain by
modifying the input of a frozen pre-trained models for cross-modal adaptation.
In addition to the known, input-independent, reprogramming method, we propose
an new reprogramming paradigm:  Input-dependent NMR, to increase adaptability
to complex input data such as musical audio. Experimental results suggest that
a neural model pre-trained on large-scale datasets can successfully perform
music genre classification by using this reprogramming method. The two
proposed Input-dependent NMR TL methods outperform fine-tuning-based TL
methods on a small genre classification dataset.

##  1 Introduction

Large-scale datasets are often recognized as one key component in successfully
building powerful deep neural network (DNN) prediction models [1]. For
example, the ImageNet dataset [2] with 14 million samples can be used to train
several benchmark image classification systems [3] for visual perception. For
sequence modeling and language processing tasks, the representation power of
models pre-trained on millions of language data such as BERT [4] demonstrates
promising results on few-shot tasks and low-resource prediction. In speech and
acoustic understanding applications, the performance of acoustic models also
benefits considerably from large-scale datasets such as human speech commands
[5] and audio event prediction datasets (e.g., AudioSet [6]).

In the music domain, however, the lack of large-scale training data has been a
critical problem. This not only hinders the development of data-driven
approaches, but also makes the investigation of novel deep learning
architectures (e.g., transformer-based models) developed in other domains
challenging since they often require massive amounts of training data. To
tackle this problem, Transfer Learning (TL) is a popular solution. For
example, the MusiCNN [7] and JukeBox [8] models, pre-trained large-scale music
datasets, have achieved promising performance on several low-resource MIR
downstream tasks [9]. VGGish and the OpenL3, pre-trained on the large-scale
audio dataset, AudioSet [10, 11], also have shown their effectiveness on
various music information retrieval (MIR) downstream tasks [12, 13].

Although these TL methods could demonstrate decent performance in downstream
tasks, they suffer from some drawbacks. For example, the learned
representation does not guarantee to contain task-specific information. Recent
results show, e.g., that models pre-trained on music auto-tagging might lack
of information for key detection [14]. Moreover, it seems that learned
representations might still lead to inferior performance compared to task-
specific models designed for the purpose [14]. Theoretically, fine-tuning the
pre-trained model should allow to solve this problem. However, training large-
scale models such as JukeBox or VGGish requires considerable computational
resources.

In this work, we attempt to tackle this problem from a different angle. We
take advantage of a newly proposed method called Neural Model Reprogramming
(NMR) [15, 16, 17], to adopt a pre-trained model for downstream tasks. NMR is
an alternative TL technique that has been confirmed to provide good or even
state-of-the-art results [16, 17] in numerous machine learning tasks. It aims
to re-purpose a (frozen) pre-trained model on a task-specific dataset with
only a small amount of training parameters without modifying the whole model.
Unlike the original NMR training scheme, which proposes adding an universal
trainable noise directly to the input sequence, we further propose two novel
input-dependent methods to learn sample-dependent information from different
input signals and improve the NMR training strategy on musical data.

We investigate the proposed method on two models pre-trained on audio and
speech, respectively: Audio Spectrogram Transformer (AST) [18] and SpeechATT
[19], for the task of music genre classification. By using the proposed
methods, we achieve new state-of-the-art results on the small music genre
classification dataset, GTZAN [20].

##  2 Related Work

(a) Training pipeline for the baseline input-independent NMR by adding an
universal noise to the input waveform (II-NMR).

(b) Training pipeline for the proposed input-dependent NMR by transforming the
input feature (ID-NMR).

(c) Training pipeline for the proposed input-dependent NMR method with skip
connection (IDS-NMR).

Fig. 1: The overview of the training pipeline for the three NMR methods used
in this work.

The concept of NMR is originally related to adversarial attacks [21]. By
applying “trainable perturbations” against the loss function of a target model
on the input data, an adversarial noise can maliciously manipulate output
predictions of neural network based classification model. Meanwhile, Elsayed
et al. [15] proposed using this trainable noise to reprogram a model pre-
trained on a specific “source” task (e.g., ImageNet) to a new “target” task
(e.g., MNIST or CIFAR-10). After the reprogramming model is successfully
trained on the target data, these reprogramming noises can be directly applied
to a frozen pre-trained model input during inference time. These trainable
perturbations can thus be considered a new program to empower cross-domain
transfer learning toward new tasks. NMR also demonstrates its flexibility in
an application-oriented setting [22], such as reprogramming a frozen traffic
sign image prediction API into a medical image classifier without knowing the
parameters of a source model.

In addition to image data, the NMR method has been proven effective in various
domains. In natural language processing, NMR has been proposed for machine
translation and sentiment classification [neekhara2018adversarial, 23]. Vinod
et al. further explore the possibility of cross-domain reprogramming by
adopting NLP models to molecule learning tasks in biochemistry [17]. In terms
of time series processing, the recent advances [16, 25] of Voice2Series show
an English speech command model could be re-programmed to become either a
sensor data predictor or a multilingual recognizer on some low-resource
language such as Lithuanian with state-of-the-art performance.

The aforementioned techniques provide both preliminary empirical findings and
theoretical foundations to further motivate us to study NMR techniques toward
music applications, addressing the limitations of insufficient training data
and infeasible large-scale neural architectures.

##  3 Neural Model Reprogramming

This work introduces two new reprogramming methods. While traditional
reprogramming simply adds noise to the input of a system as described in Sect.
3.1, our methods transform the input signal non-linearly and utilize
intermediate representations to account for the variability and complexity of
musical input signals. As shown in Fig. 1, all our pre-trained systems include
input waveform, preprocessing step to extract a high-level representation,
deep neural network, and a classifier to predict the probability of each
label. The output of the classifier is then mapped by a many-to-one label
mapping layer to map the probabilities of $n$ source classes to a target
class. That is, for a target label $y_{\mathrm{T}}\in Y_{\mathrm{T}}$, the
prediction will be the averaged class predictions
($\\{y_{1},y_{2},...,y_{n}\\}\in Y_{\mathrm{S}}$) over the set of source
labels from the original pre-trained model assigned to it. All of the steps
mentioned above will not be updated during training. Instead, a trainable
transformation function $\mathcal{H}$ will be added to introduce ”noise” to
the training pipeline. We will introduce how to add $\mathcal{H}$.

###  3.1 Input-independent NMR: Baseline Method

Since the applicability of NMR to music signals remains unexplored, we first
investigate the efficiency and potential of NMR for music signals by
establishing a baseline. We first follow the same training scheme proposed by
Yang et al. [16] as one simple baseline of waveform-level reprogramming. As
shown in Fig. 1(a), a trainable universal parameter, $\theta$, is added as a
”noise” to transform the whole waveform:
$x^{\prime}=\mathcal{H}(x;\theta)\coloneqq x+\theta$, where $x$ and
$x^{\prime}$ represent the waveform before and after transformation,
respectively. Since the universal parameter is independent of the input, we
refer to this method as II-NMR.

###  3.2 Input-dependent NMR

Although most previous NMR studies focus on adding a universal transformation
on input waveform, we suspect that a more elaborate transformation is required
for the music signal. Hence, we propose the following two adjustments.

First, music signals are fundamentally more complex than other audio data
(e.g., speech command). To capture complicated harmonic relations, MIR tasks
usually rely on perceptually meaningful features, such as Mel-spectrogram,
instead of using raw audio. Moreover, the categorization of music is usually
not determined by only one or two factors. In genre classification, for
example, the same instrumentation or chord progression can appear in
completely different music genres. Therefore, we propose that the trainable
parameters should be related to features instead of the raw waveform.
Moreover, the trainable parameters should be non-linear and “input-dependent”
to capture the complex musical features of each training sample.111We also
experimented on transforming the waveform or adding noise to the features, but
these methods resulted in low accuracy. To achieve these criteria, we add a
transformation function ($T$) composed of convolutional layers to transform
the original feature, as shown in Fig 1(b). The transformation function will
learn how to add ”noise” to the input features depending on each input sample.
That is, $X^{\prime}=\mathcal{H}(X;\mathrm{w})\coloneqq T(X)$, where $w$ is
the parameters of $T$. We refer to this adjustment input-dependent NMR (ID-
NMR).

Second, there are few potential drawbacks of directly modifying either
waveform or input features. For example, to update parameters of the
transformation function, the reprogramming layer still needs the gradient from
the whole model during training, which leads to slower training time.
Moreover, compared to the raw waveform or input features, middle layers of the
pre-trained models sometime represent more high-level information which is
more critical for classification. Hence, we propose using skip connections to
transform the output of the middle layer $v$ of the model, and add the
transformed feature $v^{\prime}$ (or noise) to the classifier for
classification, as shown in Figure 1(c). By using skip connections, we expect
the adaptive information from the input-dependent features from the skip-
connection to directly influence the classifier. We will refer to this method
later as IDS-NMR.

###  3.3 Source Models

In this section, we introduce two DNN models that we use in the experiment. To
explore the probability of cross domain reprogramming, we choose SpeeechATT
and AST models pre-trained on speech and audio data, respectively.222The
preprocessing step, DNN model and classifier all follow the implementation of
the original models.

SpeechATT: we choose SpeechATT as the efficient baseline model since Yang et
al. has achieved several state-of-the-art results on time-series data by using
the NMR training scheme and the SpeechATT model. The model is pre-trained on
the Google Speech Command dataset [5], which contains 105,829 utterances of 35
words. Different than other speech recognition tasks, utterances usually
contain very short time frames, so the input of this model is only a one-
second audio sample. In music genre classification tasks, the input audio is
usually longer than one second. To satisfy the data format of pre-trained
speech command model, we chunked the input sample into non-overlapping one-
second pieces. The final probability of each label is attained by averaging
the probabilities over all chunks.

Audio Spectrogram Transformer (AST): In recent years, transformer
architectures have been successfully applied to a variety of tasks [4, 26]. In
this paper, we experiment with the recently proposed AST model [18] for
reprogramming. AST is a purely attention-based model for audio classification
with BERT-alike patch-wise feature learning. To further leverage a larger
scale of training data, AST is pre-trained on ImageNet [3] and fine-tuned
AudioSet [10]. Although other large-scale models, such as VGGish model [10],
have been commonly used in several MIR downstream tasks [13, 12], we choose
AST since it has outperformed the VGGish model on AudioSet classification.
Moreover, reprogramming VGG-based architectures has shown worse performance
due to the disadvantage of the multi-channel feature aggregation used in VGG-
based architectures [16]. The input of AST model is an audio recording of
length 10 s. We apply a similar chunking approach as for SpeechATT: audio is
chunked into non-overlapping 10 s segments.

##  4 Experiment

###  4.1 Dataset

We choose GTZAN, a small but popular dataset for genre classification, for the
experiment. It contains 10 musical genres, with each genre having 100 audio
snippets of 30 s length, resulting in a total of 1000 snippets. Since the
total hours of this dataset is only around 8 hours, it is suitable to
represent the scenario of low-resource training data. We adopt the “fault-
filtered” split from [27] which addresses some of the reported issues with
this dataset [20], resulting in 443 pieces for training, 197 pieces for
validation, and 290 pieces for testing. The same dataset split is also used in
the MIREX baseline systems [12].

###  4.2 Experimental Setup

Four baseline methods are included in the experiment for comparison. The first
baseline is a simple CNN architecture (BL-CNN). There are plenty of ways to
build the baseline CNN. We choose ResNet since it has shown its efficiency on
audio-tagging tasks [28]. We simply concatenate four ResNet blocks with kernel
size 3 and channel size 50.333The parameters are roughly tuned to produce the
best result. For the second baseline, we simply fine-tune the original pre-
trained model with the datasets mentioned above as another common TL method.
We call this method BL-FT-AST in short. The above mentioned methods together
with our approaches can fit the model on task-specific datasets.

Another common transfer learning method is to extract the representation from
the last layer of the pre-trained model and train a shallow supervised model
for classification. Although this method could not tune the pre-trained models
on task-specific datasets, it is included for comparison. Following the work
from [14], the mean pooling representations across time are used to train an
one-layer MLPs with 512 hidden units for classification. Except for
experimenting on the representation extracted from AST model (BL-R-AST), we
also include the representation extracted from VGGish [10] model (BL-R-VGGish)
for comparison. We use the Pytorch version of the VGGish
model444https://github.com/harritaylor/torchvggish for this experiment.

Each setting is trained 100 epochs, and the model that performs the best on
the validation set is picked for evaluation. Adam optimizer with 0.0001
learning rate is used for training. Label mapping layers have $n=2$ for
SpeechATT and $n=5$ for AST after parameter search. Each method is trained
five times with different random seeds. Following previous works [12, 29, 7,
14] we report the mean and standard deviation of accuracy as the evaluation
metrics. For the convolutional layers ($T$), we use a similar architecture as
the BL-CNN, with a kernel size of 3 and 136 channels, to roughly match the
total parameters as the BL-CNN model. For the convolutional layers ($T$), we
use a similar architecture as the BL-CNN, with a kernel size of 3 and 136
channels, to roughly match the total parameters as the BL-CNN model.

Model |  SpeechATT${}_{\text{audio}}$ |  AST${}_{\text{audio + vision}}$
---|---|---
II-NMR |  0.399 $\pm$ 0.013 |  0.503 $\pm$ 0.018
ID-NMR |  0.609 $\pm$ 0.005 |  0.802 $\pm$ 0.021
IDS-NMR |  0.657 $\pm$ 0.021 |  0.828 $\pm$ 0.017
Table 1: Results on GTZAN dataset comparing different NMR methods and pre-trained models. Method | Pre-trained Model | Accuracy
---|---|---
FT-CNN [7] | MusiCNN | 0.790
FT-CNN [29] | SampleCNN | 0.821
Emb.+SVM [12] |  [7]+[30] | 0.801
Probing [14] | JukeBox | 0.797
BL-CNN | CNN${}_{\text{audio}}$ | 0.666 $\pm$ 0.034
BL-R-VGGish | VGGish${}_{\text{audio}}$ | 0.765 $\pm$ 0.017
BL-FT-AST | AST${}_{\text{audio + vision}}$ | 0.772 $\pm$ 0.013
BL-R-AST | AST${}_{\text{audio + vision}}$ | 0.831 $\pm$ 0.005
IDS-NMR | AST${}_{\text{audio + vision}}$ | 0.828 $\pm$ 0.017
Table 2: Result on GTZAN dataset comparing existing works and other baseline
methods with the IDS-NMR method.

##  5 Result & Discussion

We then compare the results between different NMR methods. The performance of
different methods by reprogramming the SpeechATT and AST models can be seen in
Table. 1. We can observe that for both models, directly adding trainable noise
(NMR-Noise) does not work well. However, our proposed input-dependent methods
can largely improve the performance. After adding skip-connection for
training, both models achieve obvious performance gain. SpeechATT generally
performs worse than AST since the original model is trained only one-second
audio. For music genre classification, it is hard to capture context
information within such a short audio clip. Moreover, SpeechATT has fewer
parameters and is trained with a smaller and music-unrelated dataset.

In the end, we compare baseline methods and existing TL methods with our
input-dependent skip connection method. Result can be seen in Table 2. We can
observe that although our proposed method utilizes a model pre-trained on
partially music-related datasets, AudioSet and ImageNet, it outperforms
existing models pre-trained on music-specific datasets, such as MSD [7, 29],
and model pre-trained on millions of music data [14]. Our IDS-NMR method also
outperforms fine-tuning method (BL-FT-AST). Although our methods perform on
par with the representation method (BL-R-AST), our IDS-NMR method gives models
the flexibility to learn both mid-level and high-level information. As extra
findings, our IDS-NMR method with jointly ImageNet and AudioSet pre-training
could achieve one highest test accuracy (85.1%) and outperforms AudioSet-only
pre-training by 2.8% within all the random seeds. The representation from AST
model (BL-R-AST) performs better than VGGish model (BL-R-VGGish). This result
suggests that the utilization of the AST model could thus lead to improved
results for a variety of MIR applications in the near future, replacing VGGish
features which have been commonly used in many music downstream tasks.

Methods |  # Trainable Params |  Speed (sec)
---|---|---
BL-FT |  88,132,063 |  -
II-NMR |  160,000 |  10:08
ID-NMR |  234,623 |  10:11
IDS-NMR |  235,680 |  03:50
Table 3: Training speed and number of parameters by using different training
methods on SpeechATT and AST model.

We also compare the number of parameters and training speed for the proposed
methods. We use a single GeForce RTX 2080 GPU to train the GTZAN dataset and
report how many seconds it takes to complete one epoch. In each epoch, the
models visit each training sample once. We can see from Table 3 that although
with or without skip connection has similar number of trainable parameters for
our input-dependent methods, with skip connecting is faster in training. Fine-
tuning AST model although can achieve good result, it could not fit on a
single GPU for training.

##  6 Conclusion

In this work, we proposed a new TL training scheme to leverage the pre-trained
models on downstream tasks. We prob into two pre-trained models, SpeechATT and
AST, by using both baseline and improved NMR methods. Experiment result on
low-resource music genre classification shows that the proposed input-
dependent method not only outperforms the fine-tuning transfer learning method
but also achieve higher accuracy than existing models pre-trained on music-
specific datasets. In the future, we plan to explore the proposed methods on
other MIR tasks which also suffer from data limitation problem. Our
implementation will be available at https://github.com/biboamy/music-repro.

## References

  * [1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [2] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Proc. CVPR_ , 2009, pp. 248–255.
  * [3] O. Russakovsky _et al._ , “Imagenet large scale visual recognition challenge,” _International Journal of Computer Vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proc. NAACL-HLT_ , 2019.
  * [5] P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” _arXiv preprint arXiv:1804.03209_ , 2018.
  * [6] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _Proc. ICASSP_ , 2017, pp. 776–780.
  * [7] J. Pons and X. Serra, “musicnn: Pre-trained convolutional neural networks for music audio tagging,” in _Late-breaking/demo session in ISMIR_ , 2019.
  * [8] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” _arXiv preprint arXiv:2005.00341_ , 2020.
  * [9] F. Korzeniowski, O. Nieto, M. McCallum, M. Won, S. Oramas, and E. Schmidt, “Mood classification using listening data,” in _Proc. ISMIR_ , 2020, pp. 542––549.
  * [10] S. Hershey _et al._ , “Cnn architectures for large-scale audio classification,” in _Proc. ICASSP_ , 2017.
  * [11] J. Cramer, H.-H. Wu, J. Salamon, and J. P. Bello, “Look, listen, and learn more: Design choices for deep audio embeddings,” in _Proc. ICASSP_ , 2019, pp. 3852–3856.
  * [12] K. C. B. Liang, Q. M. BU, and T. M. Entertainment, “Do user preference data benefit music genre classification tasks?” _MIREX_ , 2020.
  * [13] E. Koh and S. Dubnov, “Comparison and analysis of deep audio embeddings for music emotion recognition,” _AAAI Workshop on Affective Content Analysis_ , 2021.
  * [14] R. Castellon, C. Donahue, and P. Liang, “Codified audio language modeling learns useful representations for music information retrieval,” in _Proc. ISMIR_ , 2021.
  * [15] G. F. Elsayed, I. Goodfellow, and J. Sohl-Dickstein, “Adversarial reprogramming of neural networks,” in _Proc. ICLR_ , 2018.
  * [16] C.-H. H. Yang, Y.-Y. Tsai, and P.-Y. Chen, “Voice2series: Reprogramming acoustic models for time series classification,” in _Proc. ICML_ , 2021, pp. 11 808–11 819.
  * [17] R. Vinod, P.-Y. Chen, and P. Das, “Reprogramming language models for molecular representation learning,” _arXiv preprint arXiv:2012.03460_ , 2020.
  * [18] Y. Gong, Y.-A. Chung, and J. Glass, “AST: Audio spectrogram transformer,” in _Proc. Interspeech_ , 2021.
  * [19] D. C. de Andrade, S. Leo, M. L. D. S. Viana, and C. Bernkopf, “A neural attention model for speech command recognition,” _arXiv preprint arXiv:1808.08929_ , 2018.
  * [20] B. L. Sturm, “The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use,” _arXiv preprint arXiv:1306.1461_ , 2013\.
  * [21] I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _Proc. ICLR_ , 2015.
  * [22] Y.-Y. Tsai, P.-Y. Chen, and T.-Y. Ho, “Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources,” in _Proc. ICML_ , 2020, pp. 9614–9624.
  * [23] K. Hambardzumyan, H. Khachatrian, and J. May, “Warp: Word-level adversarial reprogramming,” in _Proc. ACL-IJCNLP_ , 2021, pp. 4921–4933.
  * [24] K.-W. Chang, W.-C. Tseng, S.-W. Li, and H.-y. Lee, “An exploration of prompt tuning on generative spoken language model for speech processing tasks,” _arXiv_ , 2022.
  * [25] H. Yen, P.-J. Ku, C.-H. H. Yang, H. Hu, S. M. Siniscalchi, P.-Y. Chen, and Y. Tsao, “A study of low-resource speech commands recognition based on adversarial reprogramming,” _arXiv preprint arXiv:2110.03894_ , 2021.
  * [26] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran, “Image transformer,” in _Proc. ICML_ , 2018, pp. 4055–4064.
  * [27] C. Kereliuk, B. L. Sturm, and J. Larsen, “Deep learning and music adversaries,” _IEEE Trans. on Multimedia_ , vol. 17, no. 11, pp. 2059–2071, 2015.
  * [28] M. Won, A. Ferraro, D. Bogdanov, and X. Serra, “Evaluation of cnn-based automatic music tagging models,” in _Proc. SMC_ , 2020.
  * [29] J. Lee, J. Park, K. L. Kim, and J. Nam, “Samplecnn: End-to-end deep convolutional neural networks using very small filters for music classification,” _Applied Sciences_ , vol. 8, no. 1, p. 150, 2018.
  * [30] P. Covington, J. Adams, and E. Sargin, “Deep neural networks for youtube recommendations,” in _Proc. ACM RecSys_ , 2016, pp. 191–198.