Let $T_{n}^{(1)},\dots,T_{n}^{(L)}$ be exchangeable statistics under
$(X,\Omega)\sim P^{n}\times P_{\Omega}$ for $P\in\mathcal{P}_{0}$. Under 1 and
1, we have

$\|\widehat{F}_{n}-F_{0}\|_{\infty}\rightarrow_{p}0,\quad\|\widehat{G}_{n}-G_{P}\|_{\infty}\rightarrow_{p}0$

as $n\rightarrow\infty$.

  * Proof.

Take $F=F_{0}$ and $G=G_{P}$ in Lemma C.2. Because $F_{0}$ and $G_{P}$ are
continuous, by van der Vaart (2000, Lem. 2.11) 1 implies
$\|F_{m,P}-F_{0}\|_{\infty}\rightarrow 0$ and Assumption 1 implies
$\|G_{m,P}-G_{P}\|_{\infty}\rightarrow 0$. Therefore, with probability tending
to one, the bounds in Lemma C.2 tend to zero, which yields the result. ∎

###  C.2 Rank transform

In this section, we show that the rank transform does not affect the validity
of subsampling approximation under the null. Recall that $\widetilde{G}_{n}$
is the empirical distribution of $\\{\widetilde{S}_{b}\\}$, where
$\widetilde{S}_{b}$ is the result of applying aggregation $S$ to the $b$-th
row of the rank-transformed matrix $\widetilde{\mathbf{H}}$.

######  Lemma C.3.

Given $x_{1},\dots,x_{n}\in\mathbb{R}$,
$x_{1}^{\prime},\dots,x_{n}^{\prime}\in\mathbb{R}$ and $\varepsilon>0$ such
that $|x_{i}-x_{i}^{\prime}|\leq\varepsilon$ for all $i$, for every $x\in R$
it holds that

$|\mathbb{F}_{\\{x_{i}\\}}(x)-\mathbb{F}_{\\{x_{i}^{\prime}\\}}(x)|\leq\mathbb{F}_{\\{x_{i}\\}}(x+\varepsilon)-\mathbb{F}_{\\{x_{i}\\}}(x-\varepsilon).$

  * Proof.

As
$|\mathbb{F}_{\\{x_{i}\\}}(x)-\mathbb{F}_{\\{x_{i}^{\prime}\\}}(x)|=n^{-1}\sum_{i=1}^{n}(\mathbb{I}\\{x_{i}\leq
x\\}-\mathbb{I}\\{x_{i}^{\prime}\leq x\\})$, we know
$|\mathbb{F}_{\\{x_{i}\\}}(x)-\mathbb{F}_{\\{x_{i}^{\prime}\\}}(x)|$ is
bounded above by the fraction of $i$ such that $x$ is between $x_{i}$ and
$x_{i}^{\prime}$. Due to the fact that
$|x_{i}-x_{i}^{\prime}|\leq\varepsilon$, this fraction is further bounded
above by the fraction of $x_{i}$ such that
$x\in[x_{i}-\varepsilon,x_{i}+\varepsilon)$, or equivalently,
$x_{i}\in(x-\varepsilon,x+\varepsilon]$, which is the right-hand side of the
display above. ∎

######  Lemma C.4 (High probability bound on rank-transformed subsampling).

Let $T_{n}^{(1)},\dots,T_{n}^{(L)}$ be exchangeable statistics under
$(X,\Omega)\sim P^{n}\times P_{\Omega}$ for $P\in\mathcal{P}_{0}$. Suppose $S$
is chosen such that 2 holds. Let

$\displaystyle\varepsilon^{F,P}_{n}:=1/K_{n}+\|F_{m,P}-F_{0}\|_{\infty}+1/(2\,B_{n}\,L),$
$\displaystyle\varepsilon^{G,P}_{n}:=1/K_{n}+\|G_{m,P}-G_{P}\|_{\infty},$

where $B_{n}=J\lfloor n/m\rfloor$. Then, under 1 and 1, we have the following
results.

  1. (1)

For $F_{0}=\mathrm{unif}(0,1)$, with probability tending to one, it holds that

$\|\widetilde{G}_{n}-G_{P}\|_{\infty}\leq
2\,g_{\max,P}\,\varepsilon^{F,P}_{n}+3\,\varepsilon^{G,P}_{n}.$

  2. (2)

For $F_{0}=\mathcal{N}(0,1)$, with probability tending to one, it holds that

$\|\widetilde{G}_{n}-G_{P}\|_{\infty}\leq\frac{2\,g_{\max,P}}{|\Phi^{-1}(\varepsilon^{F,P}_{n})|}+4L\,\varepsilon^{F,P}_{n}+3\,\varepsilon^{G,P}_{n}+\frac{1}{B_{n}}.$

  * Proof.

In what follows, let $\mathcal{A}_{n}$ be the event that the bounds in Lemma
C.2 hold with $F=F_{0}$ and $G=G_{P}$. Note that
$\operatorname{\mathbb{P}}_{P}(\mathcal{A}_{n})\rightarrow 1$.

    1. (1)

Under $F_{0}=\mathrm{unif}(0,1)$, the rank transform Eq. 4 becomes

$\widetilde{H}_{b,l}=\widehat{F}_{n}(\widehat{H}_{b,l})-\frac{1}{2B_{n}L},\quad
b=1,\dots,B_{n},\quad l=1,\dots,L.$

On $\mathcal{A}_{n}$, we have

$sup_{x\in[0,1]}\|\widehat{F}_{n}(x)-x\|\leq
1/K_{n}+\|F_{m,P}-F_{0}\|_{\infty}$

and hence, by the rank transform,

$\max_{b,l}|\widetilde{H}_{b,l}-\widehat{H}_{b,l}|\leq\varepsilon^{F,P}_{n}.$

By 2, it implies

$\max_{b}|\widetilde{S}_{b}-\widehat{S}_{b}|\leq\varepsilon^{F,P}_{n}.$

Recall that on $\mathcal{A}_{n}$,
$\|\widehat{G}_{n}-G_{P}\|_{\infty}\leq\varepsilon^{G,P}_{n}$. Note that
$\widetilde{G}_{n}$ and $\widehat{G}_{n}$ are respectively the distribution
functions of $\\{\widetilde{S}_{b}\\}$ and $\\{\widehat{S}_{b}\\}$. Applying
Lemma C.3, we have that for every $x$,

$\displaystyle|\widetilde{G}_{n}(x)-\widehat{G}_{n}(x)|$
$\displaystyle\leq\widehat{G}_{n}(x+\varepsilon^{F,P}_{n})-\widehat{G}_{n}(x-\varepsilon^{F,P}_{n})$
$\displaystyle\leq
G_{P}(x+\varepsilon^{F,P}_{n})-G_{P}(x-\varepsilon^{F,P}_{n})+2\varepsilon^{G,P}_{n}$
$\displaystyle\leq
2g_{\max,P}\,\varepsilon^{F,P}_{n}+2\,\varepsilon^{G,P}_{n}\quad\text{on
}\mathcal{A}_{n},$

where the last step follows from Assumption 1 and the mean value theorem. Note
that the final bound above does not depend on $x$. Consequently, on
$\mathcal{A}_{n}$, we have

$\displaystyle\|\widetilde{G}_{n}-G_{P}\|_{\infty}$
$\displaystyle\leq\|\widetilde{G}_{n}-\widehat{G}_{n}\|_{\infty}+\|\widehat{G}_{n}-G_{P}\|_{\infty}$
$\displaystyle\leq
2\,g_{\max,P}\,\varepsilon^{F,P}_{n}+3\,\varepsilon^{G,P}_{n},$

which occurs with probability tending to one.

    2. (2)

Under $F_{0}=\Phi$, the rank transform in Eq. 4 becomes

$\widetilde{H}_{b,l}=\Phi^{-1}\left(\widehat{F}_{n}(\widehat{H}_{b,l})-1/(2B_{n}L)\right).$

By a Taylor expansion of the outer function at $\Phi(\widehat{H}_{b,l})$,

$\widetilde{H}_{b,l}=\widehat{H}_{b,l}+\frac{1}{\phi(\Phi^{-1}(\xi))}\left[\widehat{F}_{n}(\widehat{H}_{b,l})-1/(2B_{n}L)-\Phi(\widehat{H}_{b,l})\right]$
(24)

for some $\xi$ between $\widehat{F}_{n}(\widehat{H}_{b,l})-1/(2B_{n}L)$ and
$\Phi(\widehat{H}_{b,l})$. Let

$\Gamma_{\varepsilon}:=\left\\{(b,l):\widehat{F}_{n}(\widehat{H}_{b,l})-1/(2B_{n}L)\in[0,2\varepsilon^{F,P}_{n})\cup(1-2\varepsilon^{F,P}_{n},1]\right\\}.$

Because $\\{\widehat{F}_{n}(\widehat{H}_{b,l})\\}$ is the set of normalised
ranks, we have

$|\Gamma_{\varepsilon}|\leq 4\varepsilon^{F,P}_{n}B_{n}L+1.$ (25)

Observe that
$\|\widehat{F}_{n}-1/(2B_{n}L)-\Phi\|_{\infty}\leq\varepsilon^{F,P}_{n}$ on
$\mathcal{A}_{n}$. Then on $\mathcal{A}_{n}$, for every
$(b,l)\in\Gamma_{\varepsilon}^{c}$, we know $\xi$ in Eq. 24 is in
$[\varepsilon^{F,P}_{n},1-\varepsilon^{F,P}_{n}]$. By Mills’ ratio (Lemma
B.2), we have

$\phi(\Phi^{-1}(\xi))\geq\phi(\Phi^{-1}(1-\varepsilon_{F,P}))>\varepsilon^{F,P}_{n}|\Phi^{-1}(\varepsilon^{F,P}_{n})|.$

It then follows from Eq. 24 that, on $\mathcal{A}_{n}$,

$|\widetilde{H}_{b,l}-\widehat{H}_{b,l}|\leq\frac{1}{|\Phi^{-1}(\varepsilon^{F,P}_{n})|},\quad(b,l)\in\Gamma_{\varepsilon}^{c}.$
(26)

Define

$\Xi_{\varepsilon}:=\left\\{b:(b,l)\in\Gamma_{\varepsilon}\text{ for some
}l\in\\{1,\dots,L\\}\right\\}.$

By 2, on $\mathcal{A}_{n}$, Eq. 26 implies

$|\widetilde{S}_{b}-\widehat{S}_{b}|\leq\delta_{n}:=\frac{1}{|\Phi^{-1}(\varepsilon^{F,P}_{n})|},\quad
b\in\Xi_{\varepsilon}^{c}.$ (27)

In light of $|\Xi_{\varepsilon}|\leq|\Gamma_{\varepsilon}|$ and Eq. 25,
observe that

$\rho_{\varepsilon}:=|\Xi_{\varepsilon}|/B_{n}\leq
4\varepsilon^{F,P}_{n}L+1/B_{n}.$ (28)

We now bound the distance between the empirical distributions of the rank-
transformed and untransformed statistics. We have

$\begin{split}\|\widetilde{G}_{n}-\widehat{G}_{n}\|_{\infty}&=\left\|\mathbb{F}_{\\{\widetilde{S}_{b}\\}}-\mathbb{F}_{\\{\widehat{S}_{b}\\}}\right\|_{\infty}\\\
&=\left\|(1-\rho_{\varepsilon})\mathbb{F}_{\widetilde{S}_{\Xi_{\varepsilon}^{c}}}+\rho_{\varepsilon}\mathbb{F}_{\widetilde{S}_{\Xi_{\varepsilon}}}-(1-\rho_{\varepsilon})\mathbb{F}_{\widehat{S}_{\Xi_{\varepsilon}^{c}}}-\rho_{\varepsilon}\mathbb{F}_{\widehat{S}_{\Xi_{\varepsilon}}}\right\|_{\infty}\\\
&\leq(1-\rho_{\varepsilon})\left\|\mathbb{F}_{\widetilde{S}_{\Xi_{\varepsilon}^{c}}}-\mathbb{F}_{\widehat{S}_{\Xi_{\varepsilon}^{c}}}\right\|_{\infty}+\rho_{\varepsilon}\left\|\mathbb{F}_{\widetilde{S}_{\Xi_{\varepsilon}}}-\mathbb{F}_{\widehat{S}_{\Xi_{\varepsilon}}}\right\|_{\infty}\\\
&\leq(1-\rho_{\varepsilon})\left\|\mathbb{F}_{\widetilde{S}_{\Xi_{\varepsilon}^{c}}}-\mathbb{F}_{\widehat{S}_{\Xi_{\varepsilon}^{c}}}\right\|_{\infty}+\rho_{\varepsilon},\end{split}$
(29)

where the last step uses the fact that Kolmogorov distance is bounded by one.
Given Eq. 27, by Lemma C.3, at any $x$, we have

$\begin{split}\left|\mathbb{F}_{\widetilde{S}_{\Xi_{\varepsilon}^{c}}}(x)-\mathbb{F}_{\widehat{S}_{\Xi_{\varepsilon}^{c}}}(x)\right|&\leq\frac{\left|\\{b\in\Xi_{\varepsilon}^{c}:\widehat{S}_{b}\in(x-\delta_{n},x+\delta_{n}]\right|}{|\Xi_{\varepsilon}^{c}|}\\\
&\leq\frac{\left|\\{b:\widehat{S}_{b}\in(x-\delta_{n},x+\delta_{n}]\right|}{|\Xi_{\varepsilon}^{c}|}\\\
&=\frac{\left|\\{b:\widehat{S}_{b}\in(x-\delta_{n},x+\delta_{n}]\right|}{B(1-\rho_{\varepsilon})}\\\
&=\frac{1}{1-\rho_{\varepsilon}}\left[\widehat{G}_{m,n,P}(x+\delta_{n})-\widehat{G}_{m,n,P}(x-\delta_{n})\right],\end{split}$

where $\rho_{\varepsilon}$ is defined in Eq. 28. Further, on
$\mathcal{A}_{n}$, we also have
$\|\widehat{G}_{n}-G_{P}\|_{\infty}\leq\varepsilon^{G,P}_{n}$ and hence

$\left|\mathbb{F}_{\widetilde{S}_{\Xi_{\varepsilon}^{c}}}(x)-\mathbb{F}_{\widehat{S}_{\Xi_{\varepsilon}^{c}}}(x)\right|\leq\frac{1}{1-\rho_{\varepsilon}}\left[G_{P}(x+\delta_{n})-G_{P}(x-\delta_{n})+2\varepsilon^{G,P}_{n}\right].$

Taking supremum over $x$ and using Assumption 1, we have

$\left\|\mathbb{F}_{\widetilde{S}_{\Xi_{\varepsilon}^{c}}}-\mathbb{F}_{\widehat{S}_{\Xi_{\varepsilon}^{c}}}\right\|_{\infty}\leq\frac{1}{1-\rho_{\varepsilon}}(2g_{\max,P}\delta_{n}+2\varepsilon^{G,P}_{n}).$

Hence, by substituting the above, Eq. 27 and (28) into Eq. 29, we arrive at

$\displaystyle\|\widetilde{G}_{n}-\widehat{G}_{n}\|_{\infty}$
$\displaystyle\leq
2g_{\max,P}\delta_{n}+2\varepsilon^{G,P}_{n}+4L\varepsilon^{F,P}_{n}+1/B_{n}$
$\displaystyle=\frac{2g_{\max,P}}{|\Phi^{-1}(\varepsilon^{F,P}_{n})|}+2\varepsilon^{G,P}_{n}+4L\varepsilon^{F,P}_{n}+\frac{1}{B_{n}}\quad\text{on
}\mathcal{A}_{n}.$

Finally, combining the above and
$\|\widehat{G}_{n}-G_{P}\|_{\infty}\leq\varepsilon^{G,P}_{n}$, we conclude
that on $\mathcal{A}_{n}$,

$\|\widetilde{G}_{n}-G_{P}\|_{\infty}\leq\frac{2g_{\max,P}}{|\Phi^{-1}(\varepsilon^{F,P}_{n})|}+4L\varepsilon^{F,P}_{n}+3\varepsilon^{G,P}_{n}+1/B_{n},$

whose probability tends to one.

∎

######  Remark C.1.

The normal case needs a separate proof because when $S$ is Lipschitz,
$(t_{1},\dots,t_{L})\mapsto S(\Phi^{-1}(t_{1}),\dots,\Phi^{-1}(t_{L}))$ need
not be Lipschitz.

###  C.3 Proof of Theorem 3.1

  * Proof.

Observe that with $m=\lfloor n/\log n\rfloor$ and $B_{n}=J\lfloor n/m\rfloor$,
the bounds on $\|\widetilde{G}_{n}-G_{P}\|_{\infty}$ in Lemma C.4 tend to zero
as $n\rightarrow\infty$. We have
$\|\widetilde{G}_{n}-G_{P}\|_{\infty}\rightarrow_{p}0$.

    1. (i)

This follows from the proof of (ii) by taking $\mathcal{P}_{0}=\\{P\\}$.

    2. (ii)

Note first that by Lemma C.8 and van der Vaart (2000, Lem. 2.11), we have

$\sup_{P\in\mathcal{P}_{0}}\|F_{n,P}-F_{0}\|_{\infty}\rightarrow
0,\quad\sup_{P\in\mathcal{P}_{0}}\|G_{n,P}-G_{P}\|_{\infty}\rightarrow 0.$

By Lemma C.4 and the given assumptions, there exists a sequence
$\gamma_{n}\rightarrow 0$ that does not depend on $P$, such that

$\sup_{P\in\mathcal{P}_{0}}\varepsilon^{F,P}_{n}\leq\gamma_{n},\quad\sup_{P\in\mathcal{P}_{0}}\varepsilon^{G,P}_{n}\leq\gamma_{n},$

where $\varepsilon^{F,P}_{n}$ and $\varepsilon^{G,P}_{n}$ are defined in Lemma
C.4. Also, because the probability in Lemma C.2 does not depend on $P$, there
exists sequences $\lambda_{n}\rightarrow 0$ and $\delta_{n}\rightarrow 0$,
which do not depend on $P$, such that for every $P\in\mathcal{P}_{0}$,

$\mathcal{D}_{n,P}:=\\{\|\widetilde{G}_{n}-G_{P}\|_{\infty}<\lambda_{n}\\}$

occurs with probability at least $1-\delta_{n}$. Let
$\mathcal{B}_{n}:=\left\\{S_{n}>\widetilde{G}_{n}^{-1}(1-\alpha)\right\\}$.

We first show that
$\limsup_{n}\sup_{P\in\mathcal{P}_{0}}\mathbb{P}_{P}(\mathcal{B}_{n})\leq\alpha$.
Observe that

$1-\alpha\leq\widetilde{G}_{n}\left(\widetilde{G}_{n}^{-1}(1-\alpha)\right)\leq
G_{P}\left(\widetilde{G}_{n}^{-1}(1-\alpha)\right)+\|\widetilde{G}_{n}-G_{P}\|_{\infty}$

using Lemma C.6, which implies

$G_{P}\left(\widetilde{G}_{n}^{-1}(1-\alpha)\right)\geq
1-\alpha-\|\widetilde{G}_{n}-G_{P}\|_{\infty}.$ (30)

It follows that on $\mathcal{B}_{n}$,

$G_{P}(S_{n})\geq G_{P}\left(\widetilde{G}_{n}^{-1}(1-\alpha)\right)\geq
1-\alpha-\|\widetilde{G}_{n}-G_{P}\|_{\infty}.$

Hence on $\mathcal{B}_{n}\cap\mathcal{D}_{n}$,

$S_{n}>G_{P}^{-1}(1-\alpha-\lambda_{n}).$

We have

$\displaystyle\mathbb{P}_{P}(\mathcal{B}_{n})$
$\displaystyle\leq\mathbb{P}_{P}(\mathcal{B}_{n}\cap\mathcal{D}_{n})+\mathbb{P}_{P}(\mathcal{D}_{n}^{c})$
$\displaystyle\leq\mathbb{P}_{P}\left\\{S_{n}>G_{P}^{-1}(1-\alpha-\lambda_{n})\right\\}+\delta_{n}$
$\displaystyle=1-G_{n,P}\left(G_{P}^{-1}(1-\alpha-\lambda_{n})\right)+\delta_{n}$
$\displaystyle\leq
1-G_{P}\left(G_{P}^{-1}(1-\alpha-\lambda_{n})\right)+\|G_{P}-G_{n,P}\|_{\infty}+\delta_{n}$
(by Lemma C.6)
$\displaystyle\leq\alpha+\lambda_{n}+\|G_{P}-G_{n,P}\|_{\infty}+\delta_{n},$

where $G_{n,P}$ is the distribution function of $S_{n}$. Because $\lambda_{n}$
and $\delta_{n}$ do not depend on $P$, it follows that

$\sup_{P\in\mathcal{P}_{0}}\mathbb{P}_{P}(\mathcal{B}_{n})\leq\alpha+\lambda_{n}+\sup_{P\in\mathcal{P}_{0}}\|G_{P}-G_{n,P}\|_{\infty}+\delta_{n},$

and hence
$\limsup_{n}\sup_{P\in\mathcal{P}_{0}}\mathbb{P}_{P}(\mathcal{B}_{n})\leq\alpha$
because $\lambda_{n}\rightarrow 0$, $\delta_{n}\rightarrow 0$ and
$\sup_{P\in\mathcal{P}_{0}}\|G_{P}-G_{n,P}\|_{\infty}\rightarrow 0$.

We now show that
$\liminf_{n}\inf_{P\in\mathcal{P}_{0}}\mathbb{P}_{P}(\mathcal{B}_{n})\geq\alpha$,
which will yield the final conclusion. Let

$y_{n,P}:=\sup\left\\{x:G_{P}(x)\leq 1-\alpha+\lambda_{n}+1/B_{n}\right\\}$
(31)

and consider $\mathcal{F}_{n,P}:=\\{S_{n}>y_{n,P}\\}$. Observe that
$\mathcal{F}_{n,P}$ implies $G_{P}(S_{n})-\lambda_{n}>1-\alpha+1/B$. Then,
$\mathcal{F}_{n,P}\cap\mathcal{D}_{n,P}$ implies
$\widetilde{G}_{n}(S_{n})>1-\alpha+1/B_{n}$, which further leads to

$S_{n}>\widetilde{G}_{n}^{-1}(1-\alpha)$

because $\widetilde{G}_{n}$ is an empirical measure with $B$ atoms. Hence, we
have
$\mathcal{F}_{n,P}\cap\mathcal{D}_{n,P}\subseteq\mathcal{B}_{n}\cap\mathcal{D}_{n,P}$
and therefore

$\displaystyle\mathbb{P}_{P}(\mathcal{B}_{n})$
$\displaystyle\geq\mathbb{P}_{P}(\mathcal{B}_{n}\cap\mathcal{D}_{n,P})$
$\displaystyle\geq\mathbb{P}_{P}(\mathcal{F}_{n,P}\cap\mathcal{D}_{n,P})$
$\displaystyle\geq\mathbb{P}_{P}(\mathcal{F}_{n,P})-\mathbb{P}_{P}(\mathcal{D}_{n,P}^{c})$
$\displaystyle=1-G_{n,P}(y_{n,P})-\mathbb{P}_{P}(\mathcal{D}_{n,P}^{c})$
$\displaystyle\geq 1-G_{n,P}(y_{n,P})-\delta_{n}$ $\displaystyle\geq
1-G_{P}(y_{n,P})-\|G_{P}-G_{n,P}\|_{\infty}-\delta_{n}$
$\displaystyle\stackrel{{\scriptstyle(i)}}{{=}}1-(1-\alpha+\lambda_{n}+1/B_{n})-\|G_{P}-G_{n,P}\|_{\infty}-\delta_{n}$
$\displaystyle=\alpha-\lambda_{n}-1/B_{n}-\|G_{P}-G_{n,P}\|_{\infty}-\delta_{n},$

where (i) uses Eq. 31 and the continuity of $G_{P}$. Because sequences
$\lambda_{n}$, $B$ and $\delta_{n}$ do not depend on $P$, it then follows that

$\inf_{P\in\mathcal{P}_{0}}\mathbb{P}_{P}(\mathcal{B}_{n})\geq\alpha-\lambda_{n}-1/B-\sup_{P\in\mathcal{P}_{0}}\|G_{P}-G_{n,P}\|_{\infty}-\varepsilon_{n},$

and hence
$\liminf_{n}\inf_{P\in\mathcal{P}_{0}}\mathbb{P}_{P}(\mathcal{B}_{n})\geq\alpha$
because $\lambda_{n}\rightarrow 0$, $B\rightarrow\infty$,
$\delta_{n}\rightarrow 0$ and
$\sup_{P\in\mathcal{P}_{0}}\|G_{P}-G_{n,P}\|_{\infty}\rightarrow 0$.

∎

######  Remark C.2.

The proof above needs no assumption on $G_{P}$ being strictly increasing at
$G_{P}^{-1}(1-\alpha)$; see also Politis et al. (1999, Remark 1.2.1).

###  C.4 Proof of Theorem 3.2

We first prove the following lemma.

######  Lemma C.5.

Under the assumptions of Theorem 3.2 and Assumption 2, it holds that
$\|\widetilde{Q}_{n}-Q_{P}\|_{\infty}\rightarrow_{p}0$, where $Q_{P}$ is the
distribution function of $\max\\{G_{P}^{1}(S^{1}),\dots,G_{P}^{W}(S^{W})\\}$.

  * Proof.

We show the result by relating $R_{n}$ to

$R_{n}^{0}:=\max(G_{P}^{1}(S_{n}^{1}),\dots,G_{P}^{W}(S_{n}^{1}))$

and relating $\widetilde{Q}_{n}:=\mathbb{F}_{\\{\widetilde{R}_{b}\\}}$ to the
subsampling estimate $\widehat{Q}_{n}:=\mathbb{F}_{\\{\widehat{R}_{b}\\}}$
with $\\{\widehat{R}_{b}\\}$ computed as follows. Applying $S^{1},\dots,S^{W}$
to the $b$-th row of the untransformed matrix $\widehat{\mathbf{H}}$, we get

$\widehat{S}_{b}^{1}:=S^{1}(\widehat{H}_{b,1},\dots,\widehat{H}_{b,L}),\quad\dots,\quad\widehat{S}_{b}^{W}:=S^{W}(\widehat{H}_{b,1},\dots,\widehat{H}_{b,L}).$

Recall that for $w=1,\dots,W$, $G_{P}^{w}$ is the asymptotic distribution
function of $S_{n}^{w}$. We define

$\widehat{R}_{b}:=\max\left\\{G_{P}^{1}(\widehat{S}_{b}^{1}),\dots,G_{P}^{W}(\widehat{S}_{b}^{W})\right\\},\quad
b=1,\dots,B_{n};$

compare this with the definition of $\widetilde{R}_{b}$ in Algorithm 3.

First note that $Q_{P}$ has a density that is bounded by $W$. Indeed, fix
$1\geq t>u\geq 0$ and set $\mathcal{T}_{w}:=\\{G_{P}^{w}(S_{n}^{w})\leq t\\}$
and $\mathcal{U}_{w}:=\\{G_{P}^{w}(S_{n}^{w})\leq u\\}$ for $w=1,\ldots,W$.
Then

$\displaystyle Q_{P}(t)-Q_{P}(u)$
$\displaystyle=\operatorname{\mathbb{P}}_{P}\left(\cap_{w}\mathcal{T}_{w}\right)-\operatorname{\mathbb{P}}_{P}(\cap_{w}\mathcal{U}_{w})$
$\displaystyle\leq\operatorname{\mathbb{P}}_{P}\left\\{\left(\cap_{w}\mathcal{T}_{w}\right)\setminus\left(\cap_{w}\mathcal{U}_{w}\right)\right\\}.$

Now

$\displaystyle\left(\cap_{w}\mathcal{T}_{w}\right)\setminus\left(\cap_{w}\mathcal{U}_{w}\right)$
$\displaystyle=\cap_{w}\mathcal{T}_{w}\cap\left(\cap_{w}\mathcal{U}_{w}\right)^{c}$
$\displaystyle=\cap_{w}\mathcal{T}_{w}\cap\left(\cup_{w}\mathcal{U}_{w}^{c}\right)$
$\displaystyle=\cup_{w}\left(\cap_{w}\mathcal{T}_{w}\cap\mathcal{U}_{w}^{c}\right)$
$\displaystyle\subseteq\cup_{w}\left\\{u<G_{P}^{w}(S_{n}^{w})\leq t\right\\}.$

Thus by a union bound,

$Q_{P}(t)-Q_{P}(u)\leq W(t-u),$

proving the claim.

By the joint stability in Assumption 2, the distribution of $R^{0}_{n}$
converges to $Q_{P}$. By definition, $\widehat{Q}_{n}$ is the natural
subsampling estimate of $Q_{P}$. Let

$\mathcal{Q}_{n,P}:=\left\\{\|\widehat{Q}_{n}-Q_{P}\|_{\infty}\leq
1/K_{n}+\|Q_{n,P}-Q_{P}\|_{\infty}\right\\},$

where $K_{n}$ is defined in Eq. 23 and $Q_{n,P}$ is the distribution function
of $R_{n}^{0}$. By Assumption 2, $\|Q_{n,P}-Q_{P}\|_{\infty}\rightarrow 0$. By
an argument similar to Lemma C.2, we know
$\mathbb{P}_{P}(\mathcal{Q}_{n,P})\rightarrow 1$. In what follows, let
$\mathcal{E}_{n}^{w}$ be the event that the corresponding bound in Lemma C.4
on $\|\widetilde{G}^{w}_{n}-G_{P}^{w}\|$ holds, namely

$\mathcal{E}_{n,P}^{w}:=\left\\{\|\widetilde{G}_{n}^{w}-G_{P}^{w}\|_{\infty}\leq\Delta_{n,P}^{w}\right\\},$

where

$\Delta_{n,P}^{w}=\begin{cases}2\,g^{w}_{\max,P}\,\varepsilon^{F,P}_{n}+3\,\varepsilon^{w}_{G,P},\quad&F_{0}=\mathrm{unif}(0,1)\\\
2\,g^{w}_{\max,P}/|\Phi^{-1}(\varepsilon^{F,P}_{n})|+4L\,\varepsilon^{F,P}_{n}+3\,\varepsilon^{w}_{G,P}+1/B_{n},\quad&F_{0}=\mathcal{N}(0,1)\end{cases}$
(32)

with $\varepsilon^{F,P}_{n}$ and $\varepsilon^{w}_{G,P}$ given by Lemma C.4.
From Lemma C.4, we know $\mathbb{P}_{P}(\mathcal{E}_{n,P}^{w})\rightarrow 1$.
We now prove $\|\widetilde{Q}_{n}-Q_{P}\|_{\infty}\rightarrow_{p}0$.

We first show the $F_{0}=\mathrm{unif}(0,1)$ case. Let $\mathcal{A}_{n,P}$ be
the event that the bound in Lemma C.2 holds, which implies
$\|\widehat{F}_{n}-F_{0}\|_{\infty}\leq\varepsilon^{F,P}_{n}$; we have
$\mathbb{P}_{P}(\mathcal{A}_{n,P})\rightarrow 1$. Because 2 holds for
$S^{1},\dots,S^{W}$, we have that on $\mathcal{A}_{n,P}$,

$|\widetilde{S}_{b}^{w}-\widehat{S}_{b}^{w}|\leq\varepsilon^{F,P}_{n},\quad
b=1,\dots,B_{n},\quad w=1,\dots,W.$

Then, we have for $w=1,\dots,W$, on the event
$\mathcal{A}_{n,P}\cap\mathcal{E}_{n,P}^{w}$,

$\displaystyle|\widetilde{G}_{n}^{w}(\widetilde{S}_{b}^{w})-G_{P}^{w}(\widehat{S}_{b}^{w})|$
$\displaystyle\leq|\widetilde{G}_{n}^{w}(\widetilde{S}_{b}^{w})-G_{P}^{w}(\widetilde{S}_{b}^{w})|+|G_{P}^{w}(\widetilde{S}_{b}^{w})-G_{P}^{w}(\widehat{S}_{b}^{w})|$
$\displaystyle\leq\Delta_{n,P}^{w}+g_{P,\max}^{w}\,\varepsilon^{F,P}_{n},\quad
b=1,\dots,B_{n}.$

Further, on $\mathcal{A}_{n,P}\cap(\cap_{w}\mathcal{E}_{n,P}^{w})$, it holds
that

$|\widetilde{G}_{n}^{w}(\widetilde{S}_{b}^{w})-G_{P}^{w}(\widehat{S}_{b}^{w})|\leq(\max_{w}\Delta_{n,P}^{w})+(\max_{w}g_{P,\max}^{w})\,\varepsilon^{F,P}_{n},\quad
b=1,\dots,B_{n},\quad w=1,\dots,W,$

and, because $\max(\cdot)$ is 1-Lipschitz in $\|\cdot\|_{\infty}$, that

$|\widetilde{R}_{b}-\widehat{R}_{b}|\leq(\max_{w}\Delta_{n,P}^{w})+(\max_{w}g_{P,\max}^{w})\,\varepsilon^{F,P}_{n},\quad
b=1,\dots,B_{n}.$

Because $\widetilde{Q}_{n}:=\mathbb{F}_{\\{\widetilde{R}_{b}\\}}$ and
$\widehat{Q}_{n}:=\mathbb{F}_{\\{\widehat{R}_{b}\\}}$, using Lemma C.3 one can
show that on $\mathcal{A}_{n,P}\cap(\cap_{w}\mathcal{E}_{n,P}^{w})$,

$\|\widetilde{Q}_{n}-\widehat{Q}_{n}\|_{\infty}\leq
2q_{\max,P}\left[(\max_{w}\Delta_{n,P}^{w})+(\max_{w}g_{P,\max}^{w})\,\varepsilon^{F,P}_{n}\right]+2\|\widehat{Q}_{n}-Q_{P}\|_{\infty}.$

It follows that, on
$\mathcal{Q}_{n,P}\cap\mathcal{A}_{n,P}\cap(\cap_{w}\mathcal{E}_{n,P}^{w})$,

$\|\widetilde{Q}_{n}-Q_{P}\|_{\infty}\leq
2W\left[(\max_{w}\Delta_{n,P}^{w})+(\max_{w}g_{P,\max}^{w})\,\varepsilon^{F,P}_{n}\right]\\\
+3\left(1/K_{n}+\|Q_{n,P}-Q_{P}\|_{\infty}\right).$ (33)

The bound vanishes because $\|Q_{n,P}-Q_{P}\|_{\infty}\rightarrow 0$ (by
Assumption 2), $\varepsilon^{F,P}_{n}\rightarrow 0$,
$\max_{w}\varepsilon^{w}_{G,P}\rightarrow 0$ (by Lemma C.4, 1 and 2),
$\max_{w}\Delta_{n,P}^{w}\rightarrow 0$ (by Eq. 32) and $K\rightarrow\infty$
(by definition). Since
$\mathbb{P}_{P}(\mathcal{Q}_{n,P}\cap\mathcal{A}_{n,P}\cap(\cap_{w}\mathcal{E}_{n,P}^{w}))\rightarrow
1$ by the union bound, we conclude that
$\|\widetilde{Q}_{n}-Q_{P}\|_{\infty}\rightarrow_{p}0$.

The $F_{0}=\mathcal{N}(0,1)$ case can be shown by an argument similar to the
above and the argument used for proving the corresponding case in Lemma C.4. ∎

  * Proof of Theorem 3.2.

We continue to use the notation for proving the last lemma.

    1. (i)

This follows from the proof of (ii) by taking $\mathcal{P}_{0}=\\{P\\}$.

    2. (ii)

Note first that by Lemma C.8 and van der Vaart (2000, Lem. 2.11), we have

$\sup_{P\in\mathcal{P}_{0}}\|F_{n,P}-F_{0}\|_{\infty}\rightarrow
0,\quad\max_{w}\sup_{P\in\mathcal{P}_{0}}\|G^{w}_{n,P}-G^{w}_{P}\|_{\infty}\rightarrow
0,\quad\sup_{P\in\mathcal{P}_{0}}\|Q_{n,P}-Q_{P}\|_{\infty}\rightarrow 0.$

Now fix $P\in\mathcal{P}_{0}$. Observe that on the event
$\cap_{w}\mathcal{E}_{n,P}^{w}$, we have

$|R_{n}^{0}-R_{n}|\leq\max_{w}\Delta_{n,P}^{w},$

where each $\Delta_{n,P}^{w}$ in Eq. 32 tends to zero and
$\mathbb{P}_{P}(\cap_{w}\mathcal{E}_{n,P}^{w})\rightarrow 1$. Now let $P$ vary
in $\mathcal{P}_{0}$. Under the given assumptions, observe that there exists a
sequence $\delta_{n}\rightarrow 0$, which does not depend on $P$, such that
$\sup_{P\in\mathcal{P}_{0}}\max_{w}\Delta_{n,P}^{w}\leq\delta_{n}$. Further,
because the probability in Lemma C.2 does not depend on $P$, also observe that

$\sup_{P\in\mathcal{P}_{0}}\mathbb{P}_{P}\left\\{\cup_{w}(\mathcal{E}_{n,P}^{w})^{c}\right\\}\rightarrow
0$

holds under the given assumptions; see also the proof of Lemma C.4. Hence,
with $\delta_{n}\rightarrow 0$ that does not depend on $P$, we have

$\sup_{P\in\mathcal{P}_{0}}\mathbb{P}_{P}\left(|R_{n}-R_{n}^{0}|>\delta_{n}\right)\rightarrow
0.$

Also, observe from the proof of (1), e.g., Eq. 33 for the case of
$F_{0}=\mathrm{unif}(0,1)$, that there exists sequence $\lambda_{n}\rightarrow
0$, which does not depend on $P$, such that

$\sup_{P\in\mathcal{P}_{0}}\mathbb{P}_{P}\left(\|\widetilde{Q}_{n}-Q_{P}\|_{\infty}>\lambda_{n}\right)\rightarrow
0.$

The result then follows from an argument similar to that for proving (ii) of
Theorem 3.1.

∎

###  C.5 Proofs of power results

######  Proposition C.1.

Let $(T_{n}^{(1)},\dots,T_{n}^{(L)})$ be $L$ exchangeable test statistics.
Suppose $\phi_{n}(T_{n}^{(1)},\dots,T_{n}^{(L)})$ is a most powerful at test
level $\alpha$ for testing certain $P_{0}\in\mathcal{P}_{0}$ against certain
$P_{1}\in\mathcal{P}\setminus\mathcal{P}_{0}$. Then $\phi_{n}$ equals a
symmetric function of $(T_{n}^{(1)},\dots,T_{n}^{(n)})$ $P_{0}$-almost
everywhere.

  * Proof.

Let $p_{0}$ and $p_{1}$ be respectively the densities of $P_{0}$ and $P_{1}$
with respect to some dominating measure $\mu$. By the Neyman–Pearson lemma
(Lehmann and Romano, 2005, Theorem 3.2.1), $\phi_{n}$ must be equal to
$\mathbb{I}\\{p_{1}(T_{n}^{(1)},\dots,T_{n}^{(L)})>kp_{0}(T_{n}^{(1)},\dots,T_{n}^{(L)})\\}$
for some $k$, $\mu$-almost everywhere. By exchangeability, $p_{1}$ and $p_{0}$
are symmetric in $(T_{n}^{(1)},\dots,T_{n}^{(L)})$. The result follows. ∎

  * Proof of Theorem 3.3.

We prove the result by showing that
$\|\widetilde{G}_{n}-G_{P_{0}}\|_{\infty}\rightarrow_{p}0$. Throughout the
proof, let $P_{n}\in\mathcal{P}$ be an arbitrary sequence such that
$P_{n}\rightarrow P_{0}$ and let $(X,\Omega)\sim P_{n}^{n}\times P_{\Omega}$.

We first prove for the case when $F_{0}=\mathrm{unif}(0,1)$. Recall that in
subsampling, for each $b=1,\dots,B_{n}$ and each $l=1,\dots,L$, we have

$\widehat{H}_{b,l}=T_{m}^{(l)}(X_{i_{1,b}},\dots,X_{i_{m,b}}),$

whose distribution function is $F_{m,P_{n}}$ because data is drawn iid from
$P_{n}$. With $K_{n}$ defined in Eq. 23, let

$\mathcal{A}_{n}:=\left\\{\|\mathbb{F}_{\widehat{\mathbf{H}}}-F_{m}^{P_{n}}\|_{\infty}\leq
1/K_{n}\right\\}.$

Using Lemma C.1, it holds that $\mathbb{P}_{P_{n}}(\mathcal{A}_{n})\rightarrow
1$. Also recall from Algorithm 2 that

$\widetilde{H}_{b,l}=F_{0}^{-1}\left(\mathbb{F}_{\widehat{\mathbf{H}}}(\widehat{H}_{b,l})-1/(2B_{n}L)\right)=\mathbb{F}_{\widehat{\mathbf{H}}}(\widehat{H}_{b,l})-1/(2B_{n}L),\quad
b=1,\dots,B_{n},\;l=1,\dots,L.$

Let $\check{H}_{b,l}:=F_{m,P_{n}}(\widehat{H}_{b,l})$ accordingly. By
definitions of $\widetilde{H}_{b,l}$, $\check{H}_{b,l}$ and $\mathcal{A}_{n}$,
observe that on $\mathcal{A}_{n}$,

$\max_{b,l}|\widetilde{H}_{b,l}-\check{H}_{b,l}|\leq
1/K_{n}+1/(2B_{n}L)=:\varepsilon_{n},$

where $\varepsilon_{n}\rightarrow 0$. Recall that
$\widetilde{S}_{b}:=S(\widetilde{H}_{b,1},\dots\widetilde{H}_{b,L})$ for
$b=1,\dots,B_{n}$. Accordingly, let
$\check{S}_{b}:=S(\check{H}_{b,1},\dots\check{H}_{b,L})$. By 2 on $S$, it
follows that on $\mathcal{A}_{n}$,

$\max_{b}|\widetilde{S}_{b}-\check{S}_{b}|\leq\varepsilon_{n}.$ (34)

In view of the local uniform convergence of the copula in Eq. 6, observe for
every $b$, it holds that

$\displaystyle(\check{H}_{b,1},\dots,\check{H}_{b,L})$
$\displaystyle=\left(F_{m,P_{n}}(T_{m}^{(1)}(X_{i_{1,b}},\dots,X_{i_{m,b}})),\dots,F_{m,P_{n}}(T_{m}^{(L)}(X_{i_{1,b}},\dots,X_{i_{m,b}}))\right)$
$\displaystyle\rightarrow_{d}(C^{(1)},\dots,C^{(L)})$

because along every subsequence of $m$, the subsequence of $P_{n}$ converges
to $P_{0}$. By the continuous mapping theorem, it then follows that

$\check{S}_{b}\rightarrow_{d}S(C^{(1)},\dots,C^{(L)})\sim G_{P_{0}},\quad
b=1,\dots,B_{n}.$ (35)

To see that $G_{P_{0}}$ is simply the distribution function of
$S(C^{(1)},\dots,C^{(L)})$, consider again the convergence of copula, this
time with a constant sequence $Q_{n}\equiv P_{0}$. Let

$\mathcal{B}_{n}:=\left\\{\|\mathbb{F}_{\\{\check{S}_{b}\\}}-G_{P_{0}}\|_{\infty}\leq
1/K_{n}+\|G_{m,P_{n}}-G_{P_{0}}\|_{\infty}=:\delta_{n}\right\\},$

where $G_{m,P_{n}}$ is the distribution function of
$S(F_{m,P_{n}}(T_{m}^{(1)}),\dots,F_{m,P_{n}}(T_{m}^{(L)}))$ under
$(X,\Omega)\sim P_{n}^{m}\times P_{\Omega}$. By an argument similar to the
proof of Lemma C.2, we have that
$\mathbb{P}_{P_{n}}(\mathcal{B}_{n})\rightarrow 1$. Further, in light of Eq.
35 where $\check{S}_{b}\sim G_{m,P_{n}}$ and the continuity of $G_{P_{0}}$ by
Assumption 1, we know $\|G_{m,P_{n}}-G_{P_{0}}\|_{\infty}\rightarrow 0$ and
hence $\delta_{n}\rightarrow 0$.

Recall that $\widetilde{G}_{n}:=\mathbb{F}_{\\{\widetilde{S}_{b}\\}}$. Now, by
definition of $\mathcal{B}_{n}$ and Eq. 34, on
$\mathcal{A}_{n}\cap\mathcal{B}_{n}$, using Lemma C.3, for every $x$ it holds
that

$\displaystyle|\widetilde{G}_{n}(x)-\mathbb{F}_{\\{\check{S}_{b}\\}}(x)|$
$\displaystyle\leq\mathbb{F}_{\\{\check{S}_{b}\\}}(x+\varepsilon_{n})-\mathbb{F}_{\\{\check{S}_{b}\\}}(x-\varepsilon_{n})$
$\displaystyle\leq
G_{P_{0}}(x+\varepsilon_{n})-G_{P_{0}}(x-\varepsilon_{n})+2\delta_{n}$
$\displaystyle\leq 2g_{\max,P_{0}}\varepsilon_{n}+2\delta_{n}$

and consequently,

$\|\widetilde{G}_{n}-G_{P_{0}}\|_{\infty}\leq
2g_{\max,P_{0}}\varepsilon_{n}+3\delta_{n}\rightarrow 0$

since $\varepsilon_{n}\rightarrow 0$, $\delta_{n}\rightarrow 0$ and
$g_{\max,P_{0}}<\infty$ (by Assumption 1). Because
$\mathbb{P}_{P_{n}}(\mathcal{A}_{n}\cap\mathcal{B}_{n})\rightarrow 1$ by a
union bound, our desired result that
$\|\widetilde{G}_{n}-G_{P_{0}}\|_{\infty}\rightarrow_{p}0$ follows.

The proof for the case when $F_{0}=\mathcal{N}(0,1)$ follows analogously using
an argument similar to that employed to prove case (2) of Lemma C.4. ∎

###  C.6 Test after rejection sampling

  * Proof of Proposition 4.1.

Let $\widehat{Q}_{n}$ be the distribution defined through
$\mathop{}\\!\mathrm{d}\widehat{Q}_{n}/\mathop{}\\!\mathrm{d}P=\widehat{r}_{n}$.
Note that $\widehat{Q}_{n}$ is random because $\widehat{r}_{n}$ is random. Let
$\gamma_{n}$ be a sequence of positive integers such that
$\gamma_{n}/n\rightarrow 1/C$. Consider a triangular array consisting of rows
$(\widetilde{X}^{n}_{1},\dots,\widetilde{X}^{n}_{\gamma_{n}})$ drawn iid from
$\widehat{Q}_{n}$ for $n=1,2,\dots$, i.e., the $n$-th row is drawn from
$\widehat{Q}_{n}^{\gamma_{n}}$.

We first show that
$T_{\gamma_{n}}(\widetilde{X}^{n}_{1},\dots,\widetilde{X}^{n}_{\gamma_{n}})\rightarrow_{d}T$.
Let $\pi_{n}$ be the total variation coupling between
$(\widetilde{X}_{1}^{n},\dots,\widetilde{X}_{\gamma_{n}}^{n})\sim\widehat{Q}_{n}^{\gamma_{n}}$
and $(\widetilde{X}_{1}^{\ast},\dots,\widetilde{X}_{\gamma_{n}}^{\ast})\sim
Q^{\gamma_{n}}$, where $\widehat{Q}_{n}^{\gamma_{n}}$ and $Q^{\gamma_{n}}$ are
respectively the $\gamma_{n}$-fold product measures of $\widehat{Q}_{n}$ and
$Q$. Under $\pi_{n}$ it holds that

$\displaystyle\operatorname{\mathbb{P}}\left(T_{\gamma_{n}}(\widetilde{X}_{1}^{n},\dots,\widetilde{X}_{\gamma_{n}}^{n})\neq
T_{\gamma_{n}}(\widetilde{X}_{1}^{\ast},\dots,\widetilde{X}_{\gamma_{n}}^{\ast})\mid\widehat{r}_{n}\right)$
$\displaystyle\leq\operatorname{\mathbb{P}}\left((\widetilde{X}_{1}^{n},\dots,\widetilde{X}_{\gamma_{n}}^{n})\neq(\widetilde{X}_{1}^{\ast},\dots,\widetilde{X}_{\gamma_{n}}^{\ast})\mid\widehat{r}_{n}\right)$
$\displaystyle=\operatorname*{\mathrm{d}_{\text{TV}}}(\widehat{Q}_{n}^{\gamma_{n}},\,Q^{\gamma_{n}})$
$\displaystyle\leq\gamma_{n}\operatorname*{\mathrm{d}_{\text{TV}}}(\widehat{Q}_{n},Q)=\gamma_{n}\int|\widehat{r}_{n}-r|\mathop{}\\!\mathrm{d}P,$

where we used total variation’s tensorisation property. Taking expectation
over $\widehat{r}_{n}$, we have

$\displaystyle\operatorname{\mathbb{P}}\left(T_{\gamma_{n}}(\widetilde{X}_{1}^{n},\dots,\widetilde{X}_{\gamma_{n}}^{n})\neq
T_{\gamma_{n}}(\widetilde{X}_{1}^{\ast},\dots,\widetilde{X}_{\gamma_{n}}^{\ast})\right)$
$\displaystyle\leq\gamma_{n}\operatorname{\mathbb{E}}\int|\widehat{r}_{n}-r|\mathop{}\\!\mathrm{d}P$
$\displaystyle=(\gamma_{n}/n)\,n\operatorname{\mathbb{E}}\int|\widehat{r}_{n}-r|\mathop{}\\!\mathrm{d}P\rightarrow
0,$

where the last step used Eq. 14 and $\gamma_{n}/n\rightarrow 1/C$. Again,
using the optimal coupling formulation of total variation, it follows that the
total variation between the law of
$T_{\gamma_{n}}(\widetilde{X}_{1}^{n},\dots,\widetilde{X}_{\gamma_{n}}^{n})$
and the law of
$T_{\gamma_{n}}(\widetilde{X}_{1}^{\ast},\dots,\widetilde{X}_{\gamma_{n}}^{\ast})$
tends to zero. Because total variation is a strong metric and
$T_{\gamma_{n}}(\widetilde{X}_{1}^{\ast},\dots,\widetilde{X}_{\gamma_{n}}^{\ast})\rightarrow_{d}T$
by our assumption, we have that
$T_{\gamma_{n}}(\widetilde{X}_{1}^{n},\dots,\widetilde{X}_{\gamma_{n}}^{n})\rightarrow_{d}T$.

By the properties of rejection sampling (von Neumann, 1951), conditioned on
$\Gamma_{n}=\gamma_{n}$, the accepted sample
$\widetilde{X}_{1}^{n},\dots,\widetilde{X}_{\gamma_{n}}^{n}$ are iid from
$\widehat{Q}_{n}$. Further, $\Gamma_{n}\sim\mathrm{binom}(n,1/C)$ and hence
$\Gamma_{n}/n\rightarrow_{p}1/C$. Consider the sequence
$\widetilde{T}_{n}:=T_{\Gamma_{n}}(\widetilde{X}_{1}^{n},\dots,\widetilde{X}_{\Gamma_{n}}^{n})$.
For every subsequence $n_{i}\nearrow\infty$, there is a further subsequence
$n_{i_{k}}$ such that $\Gamma_{n_{i_{k}}}/n_{i_{k}}\rightarrow 1/C$ almost
surely. This completes the proof because by the result we showed earlier
$\widetilde{T}_{n_{i_{k}}}$ has the same limit law for every such subsequence.
∎

######  Proposition C.2 (Parametric density ratio and simple test statistic).

Consider the setting of Proposition 4.1 without requiring Eq. 14 to hold.
Suppose $r(x)=r(x;\beta_{0})$ holds for some parametric family $r(x;\beta)$
differentiable in $\beta$ at $\beta_{0}$, such that
$\|\beta-\beta_{0}\|\rightarrow 0$ implies
$\int|r(x;\beta)-r(x;\beta_{0})|^{2}\mathop{}\\!\mathrm{d}P(x)\rightarrow 0$.
Further, let $\widehat{r}_{n}(x):=r(x;\widehat{\beta}_{n})$, where
$\widehat{\beta}_{n}$ is a consistent, asymptotically normal estimator of
$\beta_{0}$ fitted on a separate sample of size $n$ from $P$. Let $f$ be a
real-valued function of $X$ such that
$\operatorname{\mathbb{E}}_{P}f^{4}(X)<\infty$. Define
$T_{n}:=n^{-1/2}\sum_{i=1}^{n}\\{f(X_{i})-\operatorname{\mathbb{E}}_{Q}f(X)\\}$
and $T_{0}:=0$. Then we have

$T_{\Gamma_{n}}(\widetilde{X}^{n}_{1},\dots,\widetilde{X}^{n}_{\Gamma_{n}})\rightarrow_{d}\mathcal{N}\left(0,\,\operatorname{\mathrm{var}}_{Q}f(X)+v_{\beta_{0}}^{\top}\Sigma
v_{\beta_{0}}/C\right),$

where $\Sigma$ is the asymptotic covariance of $\widehat{\beta}_{n}$ and
$v_{\beta_{0}}=\nabla_{\beta}\operatorname{\mathbb{E}}_{P}f(X)r(X;\beta)\mid_{\beta=\beta_{0}}$.

  * Proof.

Let $\widehat{Q}_{n}$ be the distribution defined through
$\mathop{}\\!\mathrm{d}\widehat{Q}_{n}/\mathop{}\\!\mathrm{d}P=\widehat{r}_{n}$.
Note that $\widehat{Q}_{n}$ is random because $\widehat{r}_{n}$ is random. Let
$\gamma_{n}$ be a sequence of positive integers such that
$\gamma_{n}/n\rightarrow 1/C$. Consider a triangular array consisting of rows
$(\widetilde{X}^{n}_{1},\dots,\widetilde{X}^{n}_{\gamma_{n}})$ drawn iid from
$\widehat{Q}_{n}$ for $n=1,2,\dots$, i.e., the $n$-th row is drawn iid from
$\widehat{Q}_{n}$. By the proof of Proposition 4.1, it suffices to show that

$\widetilde{T}_{n}:=T_{\gamma_{n}}(\widetilde{X}^{n}_{1},\dots,\widetilde{X}^{n}_{\gamma_{n}})\rightarrow_{d}\mathcal{N}\left(0,\,\operatorname{\mathrm{var}}_{Q}f(X)+v_{\beta_{0}}^{\top}\Sigma
v_{\beta_{0}}/C\right).$

By Lévy’s continuity theorem, to show the above, it suffices to show that the
characteristic function of $\widetilde{T}_{n}$ converges to the characteristic
function of the limit distribution pointwise. Using

$T_{n}=\gamma_{n}^{-1/2}\sum_{i=1}^{\gamma_{n}}\left\\{f(\widetilde{X}^{n}_{i})-\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(\widetilde{X}_{1}^{n})\right\\}+\sqrt{\gamma_{n}}\left\\{\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)-\operatorname{\mathbb{E}}_{Q}f(X)\right\\},$

we can write the characteristic function of $\widetilde{T}_{n}$ as

$\varphi_{n}(t):=\operatorname{\mathbb{E}}\exp(it\widetilde{T}_{n})=\operatorname{\mathbb{E}}\left[A_{n}(t)\exp\left(it\sqrt{\gamma_{n}}\\{\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)-\operatorname{\mathbb{E}}_{Q}f(X)\\}\right)\right],$
(36)

where

$A_{n}(t):=\operatorname{\mathbb{E}}\left[\exp\left(it\gamma_{n}^{-1/2}\sum_{i=1}^{\gamma_{n}}\\{f(\widetilde{X}^{n}_{i})-\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)\\}\right)\mid\\{\widehat{Q}_{n}\\}_{n=1}^{\infty}\right].$

By the Lyapunov CLT, we have

$\gamma_{n}^{-1/2}\sum_{i=1}^{\gamma_{n}}\frac{f(\widetilde{X}^{n}_{i})-\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)}{\sqrt{\operatorname{\mathrm{var}}_{\widehat{Q}_{n}}f(X)}}\mid\\{\widehat{Q}_{n}\\}_{n=1}^{\infty}\rightarrow_{d}\mathcal{N}(0,1).$
(37)

To see this, note that $\operatorname{\mathbb{E}}_{P}f^{4}(X)<\infty$ and
$\widehat{r}_{n}\leq C$ imply
$\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}|f(X)|^{3}=\int\widehat{r}_{n}|f^{3}|\mathop{}\\!\mathrm{d}P<\infty$,
which ensures
$\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}|f(X)-\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)|^{3}<\infty$.
Further, observe that

$\displaystyle|\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f^{2}(X)-\operatorname{\mathbb{E}}_{Q}f^{2}(X)|$
$\displaystyle\leq\int|r(x;\widehat{\beta}_{n})-r(x;\beta_{0})|f^{2}(x)\mathop{}\\!\mathrm{d}P(x)$
$\displaystyle\leq\left(\int|r(x;\widehat{\beta}_{n})-r(x;\beta_{0})|^{2}\mathop{}\\!\mathrm{d}P(x)\right)^{1/2}(\operatorname{\mathbb{E}}_{P}f^{4}(X))^{1/2}\rightarrow_{p}0$

by $\widehat{\beta}_{n}\rightarrow_{p}\beta_{0}$ and our assumption on
$r(x;\beta)$ and $f$. This implies
$\operatorname{\mathrm{var}}_{\widehat{Q}_{n}}f(X)\rightarrow_{p}\operatorname{\mathrm{var}}_{Q}f(X)$.
Let $n^{\prime}$ be an arbitrary subsequence of $n$. Then there exists a
further subsequence $n^{\prime\prime}$ along which
$\operatorname{\mathrm{var}}_{\widehat{Q}_{n^{\prime\prime}}}f(X)\rightarrow_{a.s.}\operatorname{\mathrm{var}}_{Q}f(X)$.
By Slutsky’s theorem, along the subsequence $n^{\prime\prime}$, Eq. 37 implies

$\gamma_{n^{\prime\prime}}^{-1/2}\sum_{i=1}^{\gamma_{n^{\prime\prime}}}\frac{f(\widetilde{X}^{n^{\prime\prime}}_{i})-\operatorname{\mathbb{E}}_{\widehat{Q}_{n^{\prime\prime}}}f(\widetilde{X}_{1}^{n^{\prime\prime}})}{\sqrt{\operatorname{\mathrm{var}}_{Q}f(X)}}\mid\\{\widehat{Q}_{n}\\}_{n=1}^{\infty}\rightarrow_{d}\mathcal{N}(0,1)$

and hence

$\gamma_{n^{\prime\prime}}^{-1/2}\sum_{i=1}^{\gamma_{n^{\prime\prime}}}\left\\{f(\widetilde{X}^{n^{\prime\prime}}_{i})-\operatorname{\mathbb{E}}_{\widehat{Q}_{n^{\prime\prime}}}f(\widetilde{X}_{1}^{n^{\prime\prime}})\right\\}\mid\\{\widehat{Q}_{n}\\}_{n=1}^{\infty}\rightarrow_{d}\mathcal{N}(0,\operatorname{\mathrm{var}}_{Q}f(X)).$

It follows from Lévy’s continuity theorem that at every $t$,
$A_{n^{\prime\prime}}(t)\rightarrow_{a.s.}\exp\left\\{-\operatorname{\mathrm{var}}_{Q}f(X)\,t^{2}/2\right\\}$
along $n^{\prime\prime}$. Because $n^{\prime}$ is an arbitrary subsequence of
$n$, we conclude that

$A_{n}(t)\rightarrow_{p}\exp\left\\{-\operatorname{\mathrm{var}}_{Q}f(X)\,t^{2}/2\right\\}$

along the original sequence $n$. Rewrite Eq. 36 as

$\varphi_{n}(t)=\exp\left\\{-\operatorname{\mathrm{var}}_{Q}f(X)\,t^{2}/2\right\\}\operatorname{\mathbb{E}}\exp\left(it\sqrt{\gamma_{n}}\\{\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)-\operatorname{\mathbb{E}}_{Q}f(X)\\}\right)+\\\
\operatorname{\mathbb{E}}\left\\{\left[A_{n}(t)-\exp\left\\{-\operatorname{\mathrm{var}}_{Q}f(X)\,t^{2}/2\right\\}\right]\exp\left(it\sqrt{\gamma_{n}}\\{\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)-\operatorname{\mathbb{E}}_{Q}f(X)\\}\right)\right\\}.$

For every $t$,
$(A_{n}(t)-\exp\left\\{-\operatorname{\mathrm{var}}_{Q}f(X)\,t^{2}/2\right\\})\rightarrow_{p}0$.
Because the moduli of both $A_{n}(t)$ and
$\exp\left(it\sqrt{\gamma_{n}}\\{\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)-\operatorname{\mathbb{E}}_{Q}f(X)\\}\right)$
are bounded by one, by DCT, we have

$\operatorname{\mathbb{E}}\left\\{\left[A_{n}(t)-\exp\left\\{-\operatorname{\mathrm{var}}_{Q}f(X)\,t^{2}/2\right\\}\right]\exp\left(it\sqrt{\gamma_{n}}\\{\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)-\operatorname{\mathbb{E}}_{Q}f(X)\\}\right)\right\\}\rightarrow
0$

and hence

$\varphi_{n}(t)\rightarrow\exp\left\\{-\operatorname{\mathrm{var}}_{Q}f(X)\,t^{2}/2\right\\}\,\lim_{n}\operatorname{\mathbb{E}}\exp\left(it\sqrt{\gamma_{n}}\\{\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)-\operatorname{\mathbb{E}}_{Q}f(X)\\}\right).$

Because
$\sqrt{n}(\widehat{\beta}_{n}-\beta_{0})\rightarrow_{d}\mathcal{N}(0,\Sigma)$,
by the delta method, we have

$\sqrt{n}\\{\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)-\operatorname{\mathbb{E}}_{Q}f(X)\\}\rightarrow_{d}\mathcal{N}(0,v_{\beta_{0}}^{\top}\Sigma
v_{\beta_{0}})$

and hence at every $t$,

$\operatorname{\mathbb{E}}\exp\left(it\sqrt{\gamma_{n}}\\{\operatorname{\mathbb{E}}_{\widehat{Q}_{n}}f(X)-\operatorname{\mathbb{E}}_{Q}f(X)\\}\right)\rightarrow\exp\left[-(v_{\beta_{0}}^{\top}\Sigma
v_{\beta_{0}}/C)t^{2}/2\right].$

Finally, at every $t$, $\varphi_{n}(t)$ converges to the characteristic
function of
$\mathcal{N}(0,\,\operatorname{\mathrm{var}}_{Q}f(X)+v_{\beta_{0}}^{\top}\Sigma
v_{\beta_{0}}/C)$ and this completes the proof. ∎

######  Remark C.3.

In addition to the variance of $f(X)$ under $Q$, the estimation of the density
ratio also contributes to the asymptotic variance of the CLT in Proposition
C.2. While it may be difficult to form a self-normalised $Z$-statistic using
the accepted sample, we expect other forms of normalisation, such as using a
permutation test (as employed in our numerical studies), to behave properly.
For permutations, the additional asymptotic variance equally affects
$T_{\Gamma_{n}}$ and its permutation counterparts.

###  C.7 Variance estimator from rank-transformed subsampling

  * Proof of Proposition 4.2.

Fix $\varepsilon>0$. Let
$\Pi:=\left\\{(b,l):\mathbb{F}_{\widehat{\theta}_{m,b}^{(l)}}(\widehat{\theta}_{m,b}^{(l)})\in(\varepsilon/2,1-\varepsilon/2)\right\\}$.
Observe that $|\Pi|>(1-\varepsilon)B_{n}L\nearrow\infty$. Let
$\widetilde{\beta}$ be the least squares estimate of the slope from linearly
regressing $\widetilde{H}_{b,l}$ on
$\widehat{H}_{b,l}=\sqrt{m/L}(\widehat{\theta}_{m,b}^{(l)}-\theta_{0})/\sigma$
and an intercept term using those points in $\Pi$. It suffices to show that
$\widetilde{\beta}\rightarrow_{p}1$. By inspecting the proof for case (2) of
Lemma C.4, because $\varepsilon^{F,P}_{n}\rightarrow 0$, observe that when $n$
is sufficiently large, we have $2\varepsilon^{F,P}_{n}<\varepsilon$ and, with
probability tending to one, $|\widetilde{H}_{b,l}-\widehat{H}_{b,l}|\leq
1/|\Phi^{-1}(\varepsilon^{F,P}_{n})|$ for every $(b,l)\in\Pi$. Let
$\widehat{h}_{0}:=|\Pi|^{-1}\sum_{(b,l)\in\Pi}\widehat{H}_{b,l}$ and note that

$\sum_{(b,l)\in\Pi}\widehat{H}_{b,l}(\widehat{H}_{b,l}-\widehat{h}_{0})=\sum_{(b,l)\in\Pi}(\widehat{H}_{b,l}-\widehat{h}_{0})^{2}.$

Thus

$\widetilde{\beta}-1=\frac{\sum_{(b,l)\in\Pi}(\widetilde{H}_{b,l}-\widehat{H}_{b,l})(\widehat{H}_{b,l}-\widehat{h}_{0})}{\sum_{(b,l)\in\Pi}(\widehat{H}_{b,l}-\widehat{h}_{0})^{2}}\leq\frac{1}{|\Phi^{-1}(\varepsilon^{F,P}_{n})|}\frac{1}{\sqrt{|\Pi|^{-1}\sum_{(b,l)\in\Pi}(\widehat{H}_{b,l}-\widehat{h}_{0})^{2}}}$

with probability tending to one. Noting that
$|\Phi^{-1}(\varepsilon^{F,P}_{n})|\rightarrow\infty$ and that
$|\Pi|^{-1}\sum_{(b,l)\in\Pi}(\widehat{H}_{b,l}-\widehat{h}_{0})^{2}$ tends to
a positive constant by Eq. 17, it follows that
$\widetilde{\beta}\rightarrow_{p}1$. ∎

###  C.8 Auxiliary lemmas

The following standard result follows directly from the definition of
$F^{-1}$.

######  Lemma C.6.

Given any distribution function $F$, we have $F^{-1}(F(x))\leq x$ and
$F(F^{-1}(\alpha))\geq\alpha$.

###  C.9 Other results

######  Proposition C.3.

Suppose $T_{n}^{(1)},\dots,T_{n}^{(L)}$ are exchangeable and $S$ satisfies 2.
Then, under 1, the sequence $(S_{n})$ is uniformly tight.

  * Proof.

We know from Prohorov’s theorem (van der Vaart, 2000, Thm. 2.4) and 1 that the
sequence $(T_{n}^{(1)})$ is uniformly tight, so given $\varepsilon>0$, there
exists $M\geq 0$ such that
$\sup_{n}\operatorname{\mathbb{P}}_{P}(|T_{n}^{(1)}|>M)<\varepsilon/L$.

Now from 2 we have that $|S_{n}-S(0,\ldots,0)|\leq\|T_{n}\|_{\infty}$. Thus,

$\displaystyle\sup_{n}\operatorname{\mathbb{P}}_{P}\left(|S_{n}|>M+|S(0,\ldots,0)|\right)$
$\displaystyle\leq\sup_{n}\operatorname{\mathbb{P}}_{P}\left(\|T_{n}\|_{\infty}>M\right)$
$\displaystyle\leq
L\sup_{n}\operatorname{\mathbb{P}}_{P}\left(|T_{n}^{(1)}|>M\right)<\varepsilon,$

applying a union bound and appealing to exchangeability in the final line. ∎

######  Lemma C.7.

Suppose that for all $P\in\mathcal{P}$, distribution functions $H_{n,P}$
converge uniformly to a continuous distribution function $H_{P}$, i.e., for
all $x\in\mathbb{R}$,

$\sup_{P\in\mathcal{P}}|H_{n,P}(x)-H_{P}(x)|\to 0.$

Assume that each $H_{P}$ has density $h_{P}$ and
$\sup_{P\in\mathcal{P}}\|h_{P}\|_{\infty}=:h_{\max}<\infty$. Further, suppose
$\\{H_{P}:P\in\mathcal{P}\\}$ is tight. Then we have

$\sup_{P\in\mathcal{P}}\|H_{n,P}-H_{P}\|_{\infty}\to 0.$

  * Proof.

Given $\varepsilon>0$, by tightness of $\\{H_{P}:P\in\mathcal{P}\\}$, there
exists $M\geq 0$ such that $\sup_{P\in\mathcal{P}}H_{P}(-M)<\varepsilon$ and
$\sup_{P\in\mathcal{P}}\\{1-H_{P}(M)\\}<\varepsilon$. Next set
$\delta:=\varepsilon/h_{\max}$ and $x_{j}=\delta(j-1)-M$ for
$j=1,\ldots,\lceil 2M/\delta\rceil=:J-1$. Additionally set $x_{0}:=-\infty$
and $x_{J}:=\infty$. Fix $P\in\mathcal{P}$ and note that
$H_{P}(x_{j})-H_{P}(x_{j-1})\leq\varepsilon$ for all $j$.

Now for each $x\in\mathbb{R}$, there exists $j$ with $x_{j-1}\leq x\leq
x_{j}$, and

$\displaystyle H_{n,P}(x)-H_{P}(x)$ $\displaystyle\leq
H_{n,P}(x_{j})-H_{P}(x_{j-1})\leq H_{n,P}(x_{j})-H_{P}(x_{j})+\varepsilon$
$\displaystyle H_{n,P}(x)-H_{P}(x)$ $\displaystyle\geq
H_{n,P}(x_{j-1})-H_{P}(x_{j})\geq
H_{n,P}(x_{j-1})-H_{P}(x_{j-1})-\varepsilon,$

so

$\|H_{n,P}-H_{P}\|_{\infty}\leq\max_{j=0,\ldots,J}|H_{n,P}(x_{j})-H_{P}(x_{j})|+\varepsilon.$

Thus

$\limsup_{n\to\infty}\sup_{P\in\mathcal{P}}\|H_{n,P}-H_{P}\|_{\infty}\leq\limsup_{n\to\infty}\sup_{P\in\mathcal{P}}\max_{j=0,\ldots,J}|H_{n,P}(x_{j})-H_{P}(x_{j})|+\varepsilon=\varepsilon.$

As $\varepsilon$ was arbitrary, we have the desired result. ∎

######  Lemma C.8.

Let $G_{n,p}$ and $G_{P}$ for $P\in\mathcal{P}_{0}$ be as in (3) of Theorem
3.1. Then

$\sup_{P\in\mathcal{P}_{0}}\|G_{n,P}-G_{P}\|_{\infty}\to 0.$

  * Proof.

By Lemma C.7 it suffices to show that for all $\varepsilon>0$, there exists
$M\geq 0$ such that $\sup_{P\in\mathcal{P}_{0}}G_{P}(-M)<\varepsilon$ and
$\sup_{P\in\mathcal{P}_{0}}\\{1-G_{P}(M)\\}<\varepsilon$.

We have that given $\varepsilon>0$, there exists $M^{\prime}\geq 0$ such that
$F_{0}(-M^{\prime})+F_{0}(M^{\prime})<\varepsilon/L$. Then, arguing similarly
to the proof of Proposition C.3, for all $P\in\mathcal{P}_{0}$,

$\operatorname{\mathbb{P}}_{P}(|S_{n}|>M+|S(0,\ldots,0)|)\leq
L\operatorname{\mathbb{P}}_{P}\left(|T_{n}^{(1)}|>M\right).$

Taking suprema over $P\in\mathcal{P}_{0}$ and limits as $n\to\infty$,

$\sup_{P\in\mathcal{P}_{0}}\\{G_{P}(-M^{\prime})+G_{P}(M^{\prime})\\}<\varepsilon,$

as required. ∎

######  Proposition C.4.

Let $W_{1},\dots,W_{n}$ be iid from $P$. Suppose for $f(W;\beta_{0})$ it holds
that $\operatorname{\mathbb{E}}f(W;\beta_{0})=0$,
$\sigma^{2}:=\operatorname{\mathbb{E}}f^{2}(W;\beta_{0})<\infty$ and that
$f(W;\beta)$ permits a second order expansion in $\beta$ in a neighbourhood of
$\beta_{0}$. Let $\widehat{\beta}$ be a consistent, asymptotically linear
estimator of $\beta_{0}$ from $W_{1},\dots,W_{n}$. Then it holds that

$\frac{\sum_{i}f(W_{i};\widehat{\beta})}{\sqrt{\sum_{i}f^{2}(W_{i};\widehat{\beta})}}\rightarrow_{d}\mathcal{N}(0,1).$

  * Proof.

In view of

$\frac{\sum_{i}f(W_{i};\widehat{\beta})}{\sqrt{\sum_{i}f^{2}(W_{i};\widehat{\beta})}}=\frac{n^{-1/2}\sum_{i}f(W_{i};\widehat{\beta})}{\sqrt{n^{-1}\sum_{i}f^{2}(W_{i};\widehat{\beta})}},$

it suffices to show that $n^{-1}\sum_{i}f(W_{i};\widehat{\beta})$ is a
consistent, asymptotically linear estimator of zero, i.e.,

$n^{-1/2}\sum_{i}f(W_{i};\widehat{\beta})=n^{-1/2}\sum_{i}\varphi(W_{i})+o_{p}(1),$

for some $\varphi$ such that $\operatorname{\mathbb{E}}\varphi(W)=0$,
$\operatorname{\mathbb{E}}\varphi^{2}(W)<\infty$. Let $h(W):=(\partial
f(W;\beta)/\partial\beta)|_{\beta=\beta_{0}}$. Using

$f(W;\widehat{\beta})=f(W;\beta_{0})+h(W)^{\top}(\widehat{\beta}-\beta_{0})+O(\|\widehat{\beta}-\beta_{0}\|^{2}),$

we have

$\displaystyle n^{1/2}\sum_{i}f(W_{i};\widehat{\beta})$
$\displaystyle=n^{-1/2}\sum_{i}\left[f(W_{i};\beta_{0})+h(W_{i})^{\top}(\widehat{\beta}-\beta_{0})+O(\|\widehat{\beta}-\beta_{0}\|^{2})\right]$
$\displaystyle=n^{-1/2}\sum_{i}f(W_{i};\beta_{0})+\sqrt{n}(\widehat{\beta}-\beta_{0})^{\top}\left[n^{-1}\sum_{i}h(W_{i})\right]+O(n^{1/2}\|\widehat{\beta}-\beta_{0}\|^{2})$
$\displaystyle=n^{-1/2}\sum_{i}f(W_{i};\beta_{0})+\sqrt{n}\left[\operatorname{\mathbb{E}}h^{\top}(W)\right](\widehat{\beta}-\beta_{0})+O_{p}(n^{-1/2})$
$\displaystyle=n^{-1/2}\sum_{i}\left\\{f(W_{i};\beta_{0})+\left[\operatorname{\mathbb{E}}h^{\top}(W)\right]\gamma(W_{i})\right\\}+o_{p}(1),$

where $\gamma(W)$ is the influence function for $\widehat{\beta}$. Hence, we
have $\varphi(W)=f(W;\beta_{0})+(\operatorname{\mathbb{E}}h)^{\top}\gamma(W)$
and it is easy to check that $\operatorname{\mathbb{E}}\varphi(W)=0$ and
$\operatorname{\mathbb{E}}\varphi^{2}(W)<\infty$. ∎

##  Appendix D Additional numerical results

###  D.1 Derandomization of Example 2

Fig. D.1 shows probability of non-replication (i.e. two applications of the
same test on the same data leads to one acceptance and one rejection) for the
numerical experiment in Section 4.1.1.

Figure D.1: Probability of non-replication for testing $\mu=\mathbf{0}$ in
Example 2 at level $\alpha=0.05$, where $\mu=\tau n^{-1/2}v_{1}$. Tests based
on $S_{n}=(T_{n}^{(1)}+\dots+T_{n}^{(L)})/L$ significantly reduces the chance
of non-replication from using the single-split test.

###  D.2 Goodness-of-fit testing for quantile regression

In this section, we include additional numerical results for goodness-of-fit
testing of quantile regression considered in Section 4.1.3. Fig. D.2 reports
the results when the non-linear function in Eq. 11 is chosen as
$\eta(X)=2\exp(-1-X_{2}-X_{3})$.

Figure D.2: Testing goodness-of-fit of a quantile regression model
$q_{0.5}(X)=\beta_{0}+\beta^{\top}X$: power (95% CI) at level $\alpha=0.05$
(dashed horizontal) under $n=1000$. The model is well-specified if and only if
$v=0$. The non-linear function in Eq. 11 is $\eta(X)=2\exp(-1-X_{2}-X_{3})$.

###  D.3 Comparison with conservative $p$-value merging

In this section, we include numerical results that compare our method to
various conservative, deterministic merging methods for non-independent
p-values introduced by Vovk and Wang (2020). In what follows, single refers to
the p-value resulting from a single application of a randomized test (e.g.,
one single data split). Labels avg and min respectively refer to subsampling
based inference for the average p-value and the minimum p-value.
Conservatively merged p-values are labelled as M.xx for the following merging
functions $p=f(p_{1},\dots,p_{L})$:

Arithmetic (M.arith)

$p=2\,(p_{1}+\dots+p_{L})/L$,

Geometric (M.geom)

$p=e\,(p_{1}\dots p_{L})^{1/L}$,

Bonferroni (M.Bonf)

$p=L\,\min(p_{1},\dots,p_{L})$,

Compound Bonferroni-geometric (M.Bonf-geom)

$p=2\min\\{L\min(p_{1},\dots,p_{L}),e\,(p_{1}\dots p_{L})^{1/L}\\}$.

These merging functions are precise or asymptotically precise, which roughly
mean that the multiplicative constants in their definitions cannot be improved
in general; see Vovk and Wang (2020).

In below, Fig. D.3 shows results for detecting multivariate unimodality
considered in Section 4.1.2; Fig. D.4 shows results for goodness-of-fit
testing for quantile regression considered in Section 4.1.3; Figs. D.5 and D.6
show results for testing generalised conditional independence in the two
settings considered in Section 4.2.

Figure D.3: Supplement to Fig. 4.3: comparison of single-split p-value
(single), subsampling p-value (S=adaptive) and conservative $p$-value merging
functions (M.xx, dashed curves) for testing multivariate unimodality in $p$
dimensions considered in Section 4.1.2. The null hypothesis holds when
$\tau=0$ and the level is 0.05. Subsampling and conservative merging are based
on $L=50$ exchangeable p-values. Figure D.4: Supplement to Figs. 4.5 and D.2:
comparison of single-split p-value (single), subsampling p-value (S=avg) and
conservative $p$-value merging functions (M.xx, dashed curves) for goodness-
of-fit testing of quantile regression considered in Section 4.1.3. The null
hypothesis holds when $v=0$ and the level is 0.05. Subsampling and
conservative merging are based on $L=50$ exchangeable p-values. The non-linear
function in Eq. 11 is: (a) $\eta(X)=4\sqrt{X_{1}^{2}+X_{2}^{2}}$, (b)
$\eta(X)=2\exp(-1-X_{2}-X_{3})$. Figure D.5: Supplement to Fig. 4.7:
comparison of single-split p-value (single), subsampling averaged p-value
(avg) and conservative $p$-value merging functions (M.xx, dashed curves) for
testing a generalised conditional independence; see Section 4.2 for the
setting. The null hypothesis holds when $\tau=0$ and the level is 0.05.
Subsampling and conservative merging are based on $L=20$ exchangeable
p-values. Figure D.6: Supplement to Fig. 4.8: comparison of single-split
p-value (single), subsampling averaged p-value (avg) and conservative
$p$-value merging functions (M.xx, dashed curves) for testing a generalised
conditional independence under the more difficult setting Eq. 16; see Section
4.2. The null hypothesis holds when $\tau=0$ and the level is 0.05.
Subsampling and conservative merging are based on $L=20$ exchangeable
p-values.

###  D.4 Calibration of DML

This section provides details for the numerical example in Section 4.3. In the
partially linear model, we let $X\sim\mathcal{N}(0,1)$ and
$s(X):=4\sqrt{|X|}$. The heteroscedastic errors are drawn as

$V=s(X)\,\varepsilon_{V},\quad\xi=s(X)\,\varepsilon_{\xi}$

with $\varepsilon_{V}\sim\mathsf{Ga}(0.5,1)-0.5$ and
$\varepsilon_{V}\sim\mathsf{Ga}(0.3,1)-0.3$. The regression functions are
chosen as

$\displaystyle m_{0}(x)$ $\displaystyle=x+\cos(x)+\exp(x)/(1+\exp(x)),$
$\displaystyle g_{0}(x)$
$\displaystyle=\left[-10x+3\cos(4x)x^{2}/(1+\exp(x/6))\right]/10.$