$\displaystyle\leq$ $\displaystyle
2\cdot\underbrace{\frac{1}{N}\sum_{i=1}^{I}\left\lVert\xi_{i}\right\rVert\left\lVert\varepsilon_{i}\right\rVert\left\lVert\mathcal{E}_{i}^{(\xi)}\right\rVert\left\lVert\mathcal{E}_{i}^{(\varepsilon)}\right\rVert}_{=:G_{I}^{(\text{{I}})}}+2\cdot\underbrace{\frac{1}{N}\sum_{i=1}^{I}\left(\left\lVert\xi_{i}\right\rVert\left\lVert\varepsilon_{i}\right\rVert\left\\{\left\lVert\mathcal{E}_{i}^{(\xi)}\right\rVert\left\lVert\varepsilon_{i}\right\rVert+\left\lVert\mathcal{E}_{i}^{(\varepsilon)}\right\rVert\left\lVert\xi_{i}\right\rVert\right\\}\right)}_{=:G_{I}^{(\text{{II}})}}$
$\displaystyle\quad+2\cdot\underbrace{\frac{1}{N}\sum_{i=1}^{I}\left\lVert\mathcal{E}_{i}^{(\xi)}\right\rVert^{2}\left\lVert\mathcal{E}_{i}^{(\varepsilon)}\right\rVert^{2}}_{=:G_{I}^{(\text{{III}})}}+4\cdot\underbrace{\frac{1}{N}\sum_{i=1}^{I}\left(\left\lVert\mathcal{E}_{i}^{(\xi)}\right\rVert^{2}\left\lVert\varepsilon_{i}\right\rVert^{2}+\left\lVert\mathcal{E}_{i}^{(\varepsilon)}\right\rVert^{2}\left\lVert\xi_{i}\right\rVert^{2}\right)}_{=:G_{I}^{(\text{{IV}})}}$

by the triangle equality. We proceed by showing that each term on the right
hand side is $o_{P}(1)$. In the case of Assumption (a) we have

$\displaystyle\mathbb{E}\left[G_{I}^{(\text{{I}})}\right]$
$\displaystyle\leq\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\xi_{i}\|_{2}^{2}\|\varepsilon_{i}\|_{2}^{2}\right]^{\frac{1}{2}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{2}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}^{2}\right]^{\frac{1}{2}}$
$\displaystyle\leq
C_{\varepsilon}^{\frac{1}{4}}C_{\xi}^{\frac{1}{4}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{4}\right]^{\frac{1}{4}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}^{4}\right]^{\frac{1}{4}}=o(1),$

by the Cauchy–Schwarz inequality (twice). We can similarly show (again by
multiple uses of the Cauchy–Schwarz inequality) that
$\mathbb{E}[G_{I}^{(\text{{II}})}],\mathbb{E}[G_{I}^{(\text{{III}})}],\mathbb{E}[G_{I}^{(\text{{IV}})}]=o(1)$.
It then follows by Markov’s inequality (alongside Lemma 12) that
$G_{I}^{(\text{{I}})},G_{I}^{(\text{{II}})},G_{I}^{(\text{{III}})},G_{I}^{(\text{{IV}})}=o_{P}(1)$,
and so $\sup_{\psi\in\Psi}\left|G_{I}(\psi)\right|=o_{P}(1)$. If alternatively
Assumption (b) holds then we consider each term separately:

##### Term $G_{I}^{(\text{{I}})}$:

$\displaystyle\mathbb{E}\left[\big{(}G_{I}^{(\text{{I}})}\big{)}^{\frac{2}{3}}\right]$
$\displaystyle\leq
2^{\frac{2}{3}}N^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\xi_{i}\|_{2}^{\frac{2}{3}}\|\varepsilon_{i}\|_{2}^{\frac{2}{3}}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{\frac{2}{3}}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}^{\frac{2}{3}}\right]$
$\displaystyle\leq
N^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\varepsilon_{i}\|_{2}^{2}\|\xi_{i}\|_{2}^{2}\right]^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}\|\mathcal{E}_{i}^{(\xi)}\|_{2}\right]^{\frac{2}{3}}$
$\displaystyle\leq
N^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}^{2}\right]^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{2}\right]^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\varepsilon_{i}\|_{2}^{4}\right]^{\frac{1}{6}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\xi_{i}\|_{2}^{4}\right]^{\frac{1}{6}}$
$\displaystyle=o(1),$

the first inequality follows as
$(\sum_{i}x_{i})^{\eta}\leq\sum_{i}x_{i}^{\eta}$ for any
$\eta\in(0,1],x_{i}\geq 0$, the second follows by Hölder’s inequality, and the
third by the Cauchy–Schwarz inequality.

##### Term $G_{I}^{(\text{{II}})}$:

$\displaystyle\mathbb{E}\left[\big{(}G_{I}^{(\text{{II}})}\big{)}^{\frac{4}{5}}\right]$
$\displaystyle\leq
N^{\frac{1}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\xi_{i}\|_{2}^{\frac{4}{5}}\|\varepsilon\|_{2}^{\frac{8}{5}}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{\frac{4}{5}}\right]+N^{\frac{1}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\xi_{i}\|_{2}^{\frac{8}{5}}\|\varepsilon\|_{2}^{\frac{4}{5}}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}^{\frac{4}{5}}\right]$
$\displaystyle\leq
N^{\frac{1}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\xi_{i}\|_{2}^{\frac{4}{3}}\|\varepsilon_{i}\|_{2}^{\frac{8}{3}}\right]^{\frac{3}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{2}\right]^{\frac{2}{5}}$
$\displaystyle+N^{\frac{1}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\xi_{i}\|_{2}^{\frac{8}{3}}\|\varepsilon_{i}\|_{2}^{\frac{4}{3}}\right]^{\frac{3}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}^{2}\right]^{\frac{2}{5}}$
$\displaystyle\leq
N^{\frac{1}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{2}\right]^{\frac{2}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{2}\right]^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\xi_{i}\|_{2}^{4}\right]^{\frac{1}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\varepsilon_{i}\|_{2}^{4}\right]^{\frac{2}{5}}$
$\displaystyle+N^{\frac{1}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}^{2}\right]^{\frac{2}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{2}\right]^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\xi_{i}\|_{2}^{4}\right]^{\frac{2}{5}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\varepsilon_{i}\|_{2}^{4}\right]^{\frac{1}{5}}$
$\displaystyle=o(1),$

where the first inequality follows as
$(\sum_{i}x_{i})^{\eta}\leq\sum_{i}x_{i}^{\eta}$ for any
$\eta\in(0,1],x_{i}\geq 0$, the second follows by the Cauchy–Schwarz
inequality, and the third by Hölder’s inequality.

##### Term $G_{I}^{(\text{{III}})}$:

$\mathbb{E}\left[\big{(}G_{I}^{(\text{{III}})}\big{)}^{\frac{1}{2}}\right]\leq
N^{\frac{1}{2}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{2}\right]^{\frac{1}{2}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}^{2}\right]^{\frac{1}{2}}=o(1),$

by $(\sum_{i}x_{i})^{\eta}\leq\sum_{i}x_{i}^{\eta}$ for any
$\eta\in(0,1],x_{i}\geq 0$ and using the Cauchy–Schwarz inequality.

##### Term $G_{I}^{(\text{{IV}})}$:

$\displaystyle\mathbb{E}\left[\big{(}G_{I}^{(\text{{IV}})}\big{)}^{\frac{2}{3}}\right]$
$\displaystyle\leq
N^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{\frac{4}{3}}\|\varepsilon_{i}\|_{2}^{\frac{4}{3}}\right]+N^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}^{\frac{4}{3}}\|\xi_{i}\|_{2}^{\frac{4}{3}}\right]$
$\displaystyle\leq
N^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\xi)}\|_{2}^{2}\right]^{\frac{2}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\varepsilon_{i}\|_{2}^{4}\right]^{\frac{1}{3}}$
$\displaystyle\quad+N^{\frac{1}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\mathcal{E}_{i}^{(\varepsilon)}\|_{2}^{2}\right]^{\frac{2}{3}}\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{I}\|\xi_{i}\|_{2}^{4}\right]^{\frac{1}{3}}$

By applying Markov’s inequality to each of the above terms we have
$G_{I}^{\text{{I}}},G_{I}^{\text{{II}}},G_{I}^{\text{{III}}},G_{I}^{\text{{IV}}}=o_{P}(1)$,
and therefore with Lemma 12 we have
$\sup_{\psi\in\Psi}\left|G_{I}(\psi)\right|=o_{P}(1)$.

Therefore, combining the results above with the triangle inequality

$\sup_{\psi\in\Psi}\left|A_{I}(\psi)\right|=o_{P}(1)\quad\text{and}\quad\sup_{\psi\in\Psi}\left|B_{I}(\psi)\right|=o_{P}(1).$
(24)

Now, by the triangle inequality

$\displaystyle\sup_{\psi\in\Psi}\left|\hat{L}_{\mathrm{SL},I}(\psi)-L_{\mathrm{SL}}(\psi)\right|$
$\displaystyle\leq\left(\sup_{\psi\in\Psi}\left|\frac{1}{\left(\mathbb{E}\left[\xi^{\top}W(\psi)(X)\xi\right]+B_{I}(\psi)\right)^{2}}-\frac{1}{\left(\mathbb{E}\left[\xi^{\top}W(\psi)(X)\xi\right]\right)^{2}}\right|\right)\cdot$
$\displaystyle\cdot\left(\sup_{\psi\in\Psi}\mathbb{E}\left[\left(\xi^{\top}W(\psi)(X)\varepsilon\right)^{2}\right]+\sup_{\psi\in\Psi}\left|A_{I}(\psi)\right|\right)+\frac{\sup_{\psi\in\Psi}\left|A_{I}(\psi)\right|}{\inf_{\psi\in\Psi}\left(\mathbb{E}\left[\xi^{\top}W(\psi)(X)\xi\right]\right)^{2}}.$

We have already shown that
$\mathbb{E}\left[\sup_{\psi\in\Psi}\left(\xi^{\top}W(\psi)(X)\varepsilon\right)^{2}\right]<\infty$
and it is similarly straightforward to show that
$\mathbb{E}\left[\inf_{\psi\in\Psi}\xi^{\top}W(\psi)(X)\xi\right]>0$.
Therefore, it remains to show that

$\sup_{\psi\in\Psi}\left|\frac{1}{\left(\mathbb{E}\left[\xi^{\top}W(\psi)(X)\xi\right]+B_{I}(\psi)\right)^{2}}-\frac{1}{\left(\mathbb{E}\left[\xi^{\top}W(\psi)(X)\xi\right]\right)^{2}}\right|=o_{P}(1).$
(25)

To show this, it will be helpful to define

$\displaystyle c$
$\displaystyle:=\mathbb{E}\bigg{[}\inf_{\psi\in\Psi}\xi^{\top}W(\psi)(X)\xi\bigg{]}>0,$
$\displaystyle C$
$\displaystyle:=\mathbb{E}\bigg{[}\sup_{\psi\in\Psi}\xi^{\top}W(\psi)(X)\xi\bigg{]}<\infty$

Then, further defining
$B_{I}^{\sup}:=\sup_{\psi\in\Psi}\left|B_{I}(\psi)\right|$ it follows that

$\displaystyle\mathbb{P}_{P}\Bigg{(}\sup_{\psi\in\Psi}\Bigg{|}$
$\displaystyle\frac{1}{\left(\mathbb{E}\left[\xi^{\top}W(\psi)(X)\xi\right]+B_{I}(\psi)\right)^{2}}-\frac{1}{\left(\mathbb{E}\left[\xi^{\top}W(\psi)(X)\xi\right]\right)^{2}}\Bigg{|}>\epsilon\Bigg{)}$
$\displaystyle\leq\mathbb{P}_{P}\left(\frac{2cB_{I}^{\sup}-\left(B_{I}^{\sup}\right)^{2}}{C^{4}\left(1+c^{-1}B_{I}^{\sup}\right)^{2}}>\epsilon\right)$
$\displaystyle\leq\mathbb{P}_{P}\left(\frac{2cB_{I}^{\sup}-\left(B_{I}^{\sup}\right)^{2}}{C^{4}\left(1+c^{-1}B_{I}^{\sup}\right)^{2}}>\tilde{\epsilon}\right)$
where $\tilde{\epsilon}:=\min\left(\epsilon,\frac{1}{6}C^{-4}c^{2}\right)$
$\displaystyle\leq\mathbb{P}_{P}\left(B_{I}^{\sup}>\lambda\right)$ where
$\lambda:=\left(1+c^{-2}C^{4}\tilde{\epsilon}\right)^{-1}\left(c-c^{-1}C^{4}\tilde{\epsilon}-\sqrt{c^{2}-3C^{4}\tilde{\epsilon}}\right)>0$
$\displaystyle=\mathbb{P}_{P}\left(\sup_{\psi\in\Psi}\left|B_{I}(\psi)\right|>\lambda\right)=o(1)$
by (24).

The penultimate step follows by solving the quadratic inequality in
$B_{I}^{\sup}$. This proves the claim (25) and hence (23), completing the
proof.

##  Appendix C Further details on sandwich boosting

###  C.1 Working correlation structures that exhibit computational benefits

We outline three common working correlation structures that allow for
computationally fast sandwich boosting, at computational order $O(N)$ as
opposed to $O\big{(}\sum_{i}n_{i}^{3}\big{)}$.

##### Equicorrelated:

The equicorrelated working correlation structure (and its scaled inverse) is
given by

$\displaystyle C_{\theta}(X_{i})$
$\displaystyle=\left(\mathbbm{1}_{\\{j=k\\}}+\mathbbm{1}_{\\{j\neq
k\\}}\rho_{\theta}\right)_{(j,k)\in[n_{i}]^{2}},$ $\displaystyle
C^{-1}_{\theta}(X_{i})$
$\displaystyle=\left(\mathbbm{1}_{\\{j=k\\}}-\frac{\theta}{1+\theta
n_{i}}\right)_{(j,k)\in[n_{i}]^{2}},$

where the constant correlation $\rho_{\theta}=\theta/(1+\theta)$ is given in
terms of the parameter $\theta\in[0,\infty)$. This working correlation is
particularly popular for repeated measures data, with this class of weights in
the homoscedastic case (setting $\sigma^{*}\equiv s^{*}\equiv 1$) the same
class as used when fitting an intercept only mixed effects model.

##### Autoregressive, AR(1):

The autoregressive structure is given by the working correlation matrix (and
its scaled inverse) of an $\text{AR}(1)$ process

$\displaystyle C_{\theta}(X_{i})$
$\displaystyle=\left(\theta^{|j-k|}\right)_{(j,k)\in[n_{i}]^{2}},$
$\displaystyle C^{-1}_{\theta}(X_{i})$
$\displaystyle=\begin{cases}\left(\mathbbm{1}_{\\{j=k\\}}+\theta^{2}\mathbbm{1}_{\\{1<j=k<n_{i}\\}}-\theta\mathbbm{1}_{\\{|j-k|=1\\}}\right)_{(j,k)\in[n_{i}]^{2}}&\text{if
}n_{i}>1\\\ (1-\theta^{2})&\text{if }n_{i}=1\end{cases}.$

This working correlation is therefore of particular relevance for longitudinal
datasets where the correlation between observations is bounded by a function
that decays exponentially with separation.

##### Hierarchical / Nested:

The nested working correlation models hierarchical groupings as follows.
Consider the $i$th group (of size $n_{i}$) to be further partitioned into $M$
sub-groups of sizes $(n_{i1},\ldots,n_{iM})$ (such that
$\sum_{m=1}^{M}n_{im}=n_{i}$). The nested group structure is then given by the
block matrix

$C_{\theta}(X_{i})=\begin{pmatrix}E_{i,11}&\cdots&E_{i,1M}\\\
\vdots&\ddots&\vdots\\\ E_{i,M1}&\cdots&E_{i,MM}\\\ \end{pmatrix},\quad
E_{i,mm^{\prime}}=\begin{cases}\rho_{1}J_{n_{im}\times
n_{im}}+(1-\rho_{1})I_{n_{im}}&\text{if }m=m^{\prime}\vspace{1mm}\\\
\rho_{2}J_{n_{im}\times n_{im^{\prime}}}&\text{if }m\neq
m^{\prime},\end{cases}$

for the vector $\rho=(\rho_{1},\rho_{2})\in[0,1)\times[0,1)$ satisfying
$\rho_{2}\leq\rho_{1}$ (suppressing the dependence on $\theta$ in
$\rho_{\theta}$), where $J_{a\times b}$ denotes an $a\times b$ dimensional
matrix of ones and $I_{a}$ denotes the $a\times a$ dimensional identity
matrix. This nested correlation can be reparametrised in terms of
$\theta=(\theta_{1},\theta_{2}):=\left(\frac{\rho_{1}-\rho_{2}}{1-\rho_{1}},\frac{\rho_{2}}{1-\rho_{1}}\right)\in[0,\infty)\times[0,\infty)$
as

$\begin{gathered}C_{\theta}^{-1}(X_{i})=(1+\theta_{1}+\theta_{2})I_{n_{i}}-\mathcal{B}_{i}(\theta),\\\
\mathcal{B}_{i}(\theta)=\begin{pmatrix}\mathcal{A}_{i,11}(\theta)J_{n_{i1}\times
n_{i1}}&\mathcal{A}_{i,12}(\theta)J_{n_{i1}\times
n_{i2}}&\cdots&\mathcal{A}_{i,1M}(\theta)J_{n_{i1}\times n_{iM}}\\\
\mathcal{A}_{i,21}(\theta)J_{n_{i2}\times
n_{i1}}&\mathcal{A}_{i,22}(\theta)J_{n_{i2}\times
n_{i2}}&\cdots&\mathcal{A}_{i,2M}(\theta)J_{n_{i2}\times n_{iM}}\\\
\vdots&\vdots&\ddots&\vdots\\\ \mathcal{A}_{i,M1}(\theta)J_{n_{iM}\times
n_{i1}}&\mathcal{A}_{i,M2}(\theta)J_{n_{iM}\times
n_{i2}}&\cdots&\mathcal{A}_{i,MM}(\theta)J_{n_{iM}\times n_{iM}}\\\
\end{pmatrix},\\\
\mathcal{A}_{i,mm^{\prime}}(\theta)=\frac{\theta_{1}}{1+\theta_{1}n_{im}}\mathbbm{1}_{\\{m=m^{\prime}\\}}+\frac{\theta_{2}}{1+\theta_{2}\sum_{l=1}^{M}\frac{n_{il}}{1+\theta_{1}n_{il}}}\cdot\frac{1}{\left(1+\theta_{1}n_{im}\right)\left(1+\theta_{1}n_{im^{\prime}}\right)}\quad(m,m^{\prime}\in[M]),\\\
\end{gathered}$

calculated by the Woodbury matrix identity.

###  C.2 Calculation of s-scores for computationally efficient sandwich
boosting

Note that, as discussed in Section 2, the scores for sandwich boosting can be
calculated in $O\big{(}\sum_{i}n_{i}^{3}\big{)}$ computations for a generic
(inverse) working correlation $C_{\theta}^{-1}(\cdot)$, but the three working
correlations introduced are special cases that can be calculated in $O(N)$
operations. We first introduce a general algorithm to calculate scores for a
generic working correlation parametrised by $\theta$, and then present this
theory applied to the special cases of the working correlations introduced
(where we have a reduction in computational order).

Input: Index set $\mathcal{I}$ for boosting training; Working correlation
structure $\mathcal{C}_{i}$ (26); Estimates of (grouped) errors
$(\hat{\xi}_{i},\hat{\varepsilon}_{i},X_{i})_{i\in\mathcal{I}}$; Estimates of
$(s,\theta)$ at which scores to be calculated.

Calculate $A^{(1)}$, $A^{(2)}$:

for  _$i\in\mathcal{I}$_ do

$A^{(1)}_{i}:=\hat{\xi}_{i}^{\top}W_{i}\hat{\xi}_{i}=\sum_{k=1}^{n_{i}}\sum_{k^{\prime}=1}^{n_{i}}\mathcal{C}_{ikk^{\prime}}(\theta)s(X_{ik})s(X_{ik^{\prime}})\hat{\xi}_{ik}\hat{\xi}_{ik^{\prime}},$

$A^{(2)}_{i}:=\hat{\xi}_{i}^{\top}W_{i}\hat{\varepsilon}_{i}=\sum_{k=1}^{n_{i}}\sum_{k^{\prime}=1}^{n_{i}}\mathcal{C}_{ikk^{\prime}}(\theta)s(X_{ik})s(X_{ik^{\prime}})\mathcal{C}_{ikk^{\prime}}(\theta)\hat{\xi}_{ik}\hat{\varepsilon}_{ik^{\prime}}.$

end for

Calculate $A^{(3)},A^{(4)}$:

for  _$i\in\mathcal{I}$ and $j\in[n_{i}]$_ do

$A^{(3)}_{ij}:=2\sum_{k=1}^{n_{i}}\mathcal{C}_{ijk}(\theta)s(X_{ik})\hat{\xi}_{ij}\hat{\xi}_{ik},$

$A^{(4)}_{ij}:=\sum_{k=1}^{n_{i}}\mathcal{C}_{ijk}(\theta)s(X_{ik})\left(\hat{\xi}_{ij}\hat{\varepsilon}_{ik}+\hat{\xi}_{ik}\hat{\varepsilon}_{ij}\right).$

end for

Calculate $A^{(1)}_{\text{sum}}:=\sum_{i\in\mathcal{I}}A^{(1)}_{i}$ and
$A^{(2)}_{\text{sum.sq}}:=\sum_{i\in\mathcal{I}}A^{(2)^{2}}_{i}$.

Calculate $s$-scores:

for  _$i\in\mathcal{I}$ and $j\in[n_{i}]$_ do

$U_{\text{SL}}^{(s)}(s,\theta)(X_{ij})=-2\left(A^{(1)}_{\text{sum}}\right)^{-3}\left[A^{(2)}_{\text{sum.sq}}A^{(3)}_{ij}-A^{(1)}_{\text{sum}}A^{(2)}_{i}A^{(4)}_{ij}\right].$

end for

Calculate the $\theta$-score:

$U_{\text{SL}}^{(\theta)}(s,\theta)=-2\left(A^{(1)}_{\text{sum}}\right)^{-3}\Bigg{[}A^{(2)}_{\text{sum.sq}}\left(\sum_{i\in\mathcal{I}}\sum_{k=1}^{n_{i}}\sum_{k^{\prime}=1}^{n_{i}}\frac{\partial\mathcal{C}_{ikk^{\prime}}(\theta)}{\partial\theta}s(X_{ik})s(X_{ik^{\prime}})\hat{\xi}_{ik}\hat{\xi}_{ik^{\prime}}\right)$

$-A^{(1)}_{\text{sum}}\left(\sum_{i\in\mathcal{I}}A_{i}^{(2)}\sum_{k=1}^{n_{i}}\sum_{k^{\prime}=1}^{n_{i}}\left(\frac{\partial\mathcal{C}_{ikk^{\prime}}(\theta)}{\partial\theta}s(X_{ik})s(X_{ik^{\prime}})\hat{\xi}_{ik}\hat{\varepsilon}_{ik^{\prime}}\right)\right)\Bigg{]}.$

Output: The set of $N$ $s$-scores
$\left(X_{ij},\,U_{\mathrm{SL}}^{(s)}(s,\theta)(X_{ij})\right)_{i\in\mathcal{I},j\in[n_{i}]}$
and $\theta$-score $U_{\mathrm{SL}}^{(\theta)}(s,\theta)$.

Algorithm 3 Computationally lean sandwich boosting score calculation for an
arbitrary working correlation

Therefore, the order of computation to calculate the score functions for
sandwich boosting is $O\big{(}\sum_{i}n_{i}^{3}\big{)}$ for a general
correlation matrix structure whose inverse $C_{\theta}^{-1}(\cdot)$ is not
known (and so needs to be computed analytically). In the special case of
Toeplitz correlation structures, or where the inverse correlation structure
$C_{\theta}^{-1}(\cdot)$ is known, this order of computation can be reduced to
$O\big{(}\sum_{i}n_{i}^{2}\big{)}$. Note that this computational order can be
further reduced to $O(N)$ in the three special cases of correlation structures
posed in this paper. We give an example of these, implementing Algorithm 3 to
the equicorrelation structure, with scores calculated in $O(N)$ operations as
in Algorithm 4. For notational simplicity for these algorithms we introduce
the notation

$\mathcal{C}_{ijk}(\theta):=\left\\{C_{\theta}^{-1}(X_{i})\right\\}_{jk},$
(26)

for the inverse working correlation structure.

Input: Index set $\mathcal{I}$ for boosting training; Estimates of (grouped)
errors $(\hat{\xi}_{i},\hat{\varepsilon}_{i},X_{i})_{i\in\mathcal{I}}$;
Estimates of $(s,\theta)$ at which scores to be calculated.

Calculate $A^{(1)},A^{(2)},B^{(1)},B^{(2)}$:

for  _$i\in\mathcal{I}$_ do

$B^{(1)}_{i}:=\sum_{k=1}^{n_{i}}s(X_{ik})\hat{\xi}_{ik},$

$B^{(2)}_{i}:=\sum_{k=1}^{n_{i}}s(X_{ik})\hat{\varepsilon}_{ik},$

$A^{(1)}_{i}:=\hat{\xi}_{i}^{\top}W_{i}\hat{\xi}_{i}=\sum_{k=1}^{n_{i}}s^{2}(X_{ik})\hat{\xi}_{ik}^{2}-\frac{\theta}{1+\theta
n_{i}}\left(\sum_{k=1}^{n_{i}}s(X_{ik})\hat{\xi}_{ik}\right)^{2},$

$A^{(2)}_{i}:=\hat{\xi}_{i}^{\top}W_{i}\hat{\varepsilon}_{i}=\sum_{k=1}^{n_{i}}s^{2}(X_{ik})\hat{\xi}_{ik}\hat{\varepsilon}_{ik}-\frac{\theta}{1+\theta
n_{i}}\left(\sum_{k=1}^{n_{i}}s(X_{ik})\hat{\xi}_{ik}\right)\Big{(}\sum_{k=1}^{n_{i}}s(X_{ik})\hat{\varepsilon}_{ik}\Big{)}.$

end for

Calculate $A^{(3)}$ and $A^{(4)}$:

for  _$i\in\mathcal{I}$ and $j\in[n_{i}]$_ do

$A^{(3)}_{ij}:=2s(X_{ij})\hat{\xi}_{ij}^{2}-\frac{2\theta}{1+\theta
n_{i}}B^{(1)}_{i}\hat{\xi}_{ij},$

$A^{(4)}_{ij}:=2s(X_{ij})\hat{\xi}_{ij}\hat{\varepsilon}_{ij}-\frac{\theta}{1+\theta
n_{i}}\left(B^{(2)}_{i}\hat{\xi}_{ij}+B^{(1)}_{i}\hat{\varepsilon}_{ij}\right).$

end for

Calculate $A^{(1)}_{\text{sum}}:=\sum_{i=1}^{I}A^{(1)}_{i}$ and
$A^{(2)}_{\text{sum.sq}}:=\sum_{i=1}^{I}A^{(2)^{2}}_{i}$.

Calculate $s$-scores:

for  _$i\in\mathcal{I}$ and $j\in[n_{i}]$_ do

$U_{\text{SL}}^{(s)}(s,\theta)(X_{ij})=-2\left(A^{(1)}_{\text{sum}}\right)^{-3}\left[A^{(2)}_{\text{sum.sq}}A^{(3)}_{ij}-A^{(1)}_{\text{sum}}A^{(2)}_{i}A^{(4)}_{ij}\right].$

end for

Calculate the $\theta$-score:

$U_{\text{SL}}^{(\theta)}(s,\theta)=-2\left(A^{(1)}_{\text{sum}}\right)^{-3}\left[A^{(2)}_{\text{sum.sq}}\left(\sum_{i=1}^{I}\frac{\left(B^{(1)}_{i}\right)^{2}}{\left(1+\theta
n_{i}\right)^{2}}\right)-A^{(1)}_{\text{sum}}\left(\sum_{i=1}^{I}\frac{A^{(2)}_{i}B^{(1)}_{i}B^{(2)}_{i}}{\left(1+\theta
n_{i}\right)^{2}}\right)\right].$

Output: The set of $N$ $s$-scores
$\left(X_{ij},\,U_{\mathrm{SL}}^{(s)}(s,\theta)(X_{ij})\right)_{i\in\mathcal{I},j\in[n_{i}]}$
and $\theta$-score $U_{\mathrm{SL}}^{(\theta)}(s,\theta)$.

Algorithm 4 Computationally lean sandwich boosting score calculation for the
equicorrelated working correlation

Similar algorithms exist for the autoregressive $\text{AR}(1)$ and nested
groups working correlations (and are also working correlation cases that
achieve the faster $O(N)$ computational rates when calculating scores for
performing sandwich boosting).

###  C.3 Theoretical results under working correlation structures

The uniform asymptotic convergence results of Section 4 works on the general
working covariance model parametrised by $(\sigma,\rho)$ functions and / or
parameters. For computational speed and stability for sandwich boosting we
consider the weight class reparametrised in terms of $(s,\theta)$. For
completeness we reformulate the assumptions on the deterministic limits
$(\sigma^{*},\rho^{*})$ of Assumption A2 in terms of the deterministic limits
$(s^{*},\theta^{*})$, with a particular focus on the three prototypical
working correlation parametrisations considered in Section C.

######  Corollary 6.

Consider the setting of Theorem 4 where the weight class $\mathcal{W}$ takes
one of the three forms outlined in Section C in terms of $(s,\theta)$.
Further, suppose there exists the deterministic function
$s^{*}:\mathbb{R}^{d}\to\mathbb{R}$ and vector $\theta^{*}$ whose estimators
in place of Assumptions (A3.1)-(A3.3) in Assumption A3 satisfy

  1. (A3.1’)

$\mathcal{R}_{s^{*}}:=\underset{j\in[n]}{\max}\,\mathbb{E}_{P}\left[\left(c^{*}\hat{s}^{(1)}(X_{j})-s^{*}(X_{j})\right)^{2}\,\Big{|}\,\hat{s}^{(1)}\right]=o_{\mathcal{P}}(1)$
where
$c^{*}:=\underset{c>0}{\textrm{arginf}}\,\underset{j\in[n]}{\max}\,\mathbb{E}_{P}\left[\left(c\hat{s}^{(1)}(X_{j})-s^{*}(X_{j})\right)^{2}\right]$,

  2. (A3.2’)

$\mathcal{R}_{\theta^{*}}:=\underset{k\in[K]}{\max}\big{\|}\hat{\theta}^{(k)}-\theta^{*}\big{\|}_{2}=o_{\mathcal{P}}(1)$,

  3. (A3.3’)

For all $k\in[K]$, $\hat{s}^{(k)}$ are bounded above and below uniformly by
some positive constants and, for the relevant working correlation:

     * •

Equicorrelation: $\hat{\theta}^{(k)}$ is bounded above and below uniformly by
constants in $[0,\infty)$,

     * •

Longitudinal: $\hat{\theta}^{(k)}$ is bounded above and below uniformly by
constants in $[0,1)$,

     * •

Hierarchical / Nested: $\hat{\theta}^{(k)}$ is component-wise bounded above
and below uniformly by constants in $[0,\infty)$.

Then the results of Theorem 4 hold in this alternative setup.

A proof of this corollary is given in Section D.4.

##  Appendix D Proof of results in Section 4

In this section we prove a stronger version of Theorem 4, extending it to non-
identically grouped data of varying group size, and allowing group sizes to
diverge at sufficiently slow rates with $N$. To allow for non-identical groups
we work on a sequence of families of probability distributions
$(\mathcal{P}_{I})_{I\in\mathbb{N}}$ that the grouped observations follow
(satisfying Assumption A1 with us suppressing dependence on $i$), and re-
define the nuisance function error rates given in Assumptions A2 and A3 with

$\mathcal{R}_{l}:=\max_{k\in[K]}\max_{i\in\mathcal{I}_{k}}\max_{j\in[n_{i}]}\,\mathbb{E}_{P}\left[\left(\hat{l}^{(k)}(X_{ij})-l_{0}(X_{ij})\right)^{2}\,\bigg{|}\,\hat{l}^{(k)}\right],$

and similarly for $\mathcal{R}_{m},\mathcal{R}_{\sigma}$ and
$\mathcal{R}_{\rho}$. This notation then recovers that in Assumption A2 when
the groups are identically distributed. We also introduce an additional
parameter $\tau$ of the working correlation structure and re-define

$V:=\left(\frac{1}{N}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\xi_{i}^{\top}W_{i}^{*}\xi_{i}\right]\right)^{-2}\left(\frac{1}{N}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left(\varepsilon_{i}^{\top}W_{i}^{*}\xi_{i}\right)^{2}\right]\right),$

with $W_{i}^{*}:=W(\sigma^{*},\rho^{*})(X_{i})$.

######  Assumption A3 Extended (Additional assumption on weight function).

Additionally to the Assumptions ((A3.1))-((A3.4)) of Assumption A3 suppose we
also make the assumption

  * (A3.5)

The working correlation function $\rho^{*}$ satisfies
$\underset{k\in[K]}{\max}\underset{i\in\mathcal{I}_{k}}{\max}\underset{j\in[n_{i}]}{\max}\sum_{j^{\prime}\neq
j}^{n_{i}}\rho^{*}(X_{ij},X_{ij^{\prime}})\lesssim n_{i}^{\tau}$ for some
$\tau\in[0,1]$ almost surely.

We can now specify the asymptotic regimes on the maximum group size
$n_{\max}:=\max_{i\in[I]}n_{i}$ for which Theorem 4 holds, in terms of the
nuisance error rates
$(\mathcal{R}_{m},\mathcal{R}_{l},\mathcal{R}_{\sigma},\mathcal{R}_{\rho})$,
the working correlation metadata $(\alpha,\kappa,\gamma,\tau)$, and the true
data generation mechanism metadata $(\delta,\mu_{\Sigma},\mu_{\Omega})$.

######  Assumption A4 (Maximum Group Size).

The maximum group size $n_{\max}:=\max_{i\in[I]}n_{i}$ of the grouped
observations $\left\\{S_{i}=(Y_{i},D_{i},X_{i})\right\\}_{i\in[I]}$ satisfies

$n_{\max}=o\Bigg{(}N^{\frac{\delta}{4(1+\kappa)+(2+\kappa)\delta}\wedge\frac{1}{3+2\kappa}}\wedge\mathcal{R}_{m}^{-\left(\frac{1}{\kappa+\mu_{\Omega}}\wedge\frac{1}{2(\kappa+\alpha)}\wedge\frac{1}{\gamma}\right)}\wedge\left(N\mathcal{R}_{m}^{-1}\right)^{\frac{1}{2\gamma+\mu_{\Omega}}}\wedge\mathcal{R}_{l}^{-\frac{1}{\kappa+\mu_{\Omega}}}\\\
\wedge\mathcal{R}_{\sigma}^{-\left(\frac{1}{2\kappa+2\mu_{\Sigma}+\tau+1}\wedge\frac{1}{2\gamma+\tau+1}\right)}\wedge\mathcal{R}_{\rho}^{-\left(\frac{1}{2(\kappa+\mu_{\Sigma}+1)}\wedge\frac{1}{2(\gamma+1)}\right)}\wedge\left(N^{-1}\mathcal{R}_{m}^{-1}\left(\mathcal{R}_{l}\vee\mathcal{R}_{m}\right)^{-1}\right)^{\frac{1}{\kappa}}\Bigg{)}.$

This assumption reduces to the rates given in Section 4 by setting
$(\mu_{\Sigma},\mu_{\Omega},\gamma,\tau)=(1,1,1,1)$ in the equicorrelated
working correlation setting, and
$(\mu_{\Sigma},\mu_{\Omega},\alpha,\kappa,\gamma,\tau)=(0,1,0,0,0,0)$ in the
autoregressive $\text{AR}(1)$ working correlation setting.

We introduce the following additional notation relevant for the proofs in this
section. For a distribution $P$ governing the distribution of a random vector
$A\in\mathbb{R}^{d}$, we write $\left\lVert
A\right\rVert_{P,p}:=(\mathbb{E}_{P}\|A\|_{2}^{p})^{1/p}$, the subscript $P$
in the expectation indicating the dependence on $P$; we will similarly write
$\mathbb{P}_{P}$. We extend this definition to random matrices $A$ where then
$\left\lVert
A\right\rVert_{P,p}:=(\mathbb{E}_{P}\|A\|_{\mathrm{op}}^{p})^{1/p}$, where
$\left\lVert\cdot\right\rVert_{\mathrm{op}}$ is the operator norm.

###  D.1 Proof of Theorem 4

The proof of Theorem 4 amounts to proving the two claims:

  1. (i)

$\lim\limits_{I\to\infty}\sup\limits_{P\in\mathcal{P}_{I}}\sup\limits_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(\sqrt{N/V}\big{(}\hat{\beta}-\beta\big{)}\leq
t\right)-\Phi\left(t\right)\right|=0;$

  2. (ii)

$\lim\limits_{I\to\infty}\sup\limits_{P\in\mathcal{P}_{I}}\sup\limits_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(\sqrt{N/\hat{V}}\big{(}\hat{\beta}-\beta\big{)}\leq
t\right)-\Phi\left(t\right)\right|=0.$

We will prove each in turn.

###### Proof of Theorem 4 (i).

Consider separating the nuisance functions into ‘fixed effects’ functions
$\eta:=(l,m)$ and weight functions $\nu:=(\sigma,\rho)$ (analogous to the
working covariance in the GEE setup or random effects in the ML setup). Define
the terms

$\begin{split}\varphi\big{(}S_{i};\tilde{\beta},\eta,\nu\big{)}&:=\big{(}D_{i}-m(X_{i})\big{)}^{\top}W_{i}(\nu)\big{(}Y_{i}-l(X_{i})-\left(D_{i}-m(X_{i})\right)\tilde{\beta}\big{)},\\\
\phi\big{(}S_{i};\eta,\nu\big{)}&:=\big{(}D_{i}-m(X_{i})\big{)}^{\top}W_{i}(\nu)\big{(}D_{i}-m(X_{i})\big{)},\end{split}$

where $W_{i}(\nu):=W(\nu)(X_{i})$. Further, recall the notation that we denote
the $k^{\text{th}}$ fold estimators for the ‘fixed effects’
$\eta_{0}:=(l_{0},m_{0})$ as
$\hat{\eta}^{(k)}:=(\hat{l}^{(k)},\hat{m}^{(k)})$, and for the weights the
estimators $\hat{\nu}^{(k)}:=(\hat{\sigma}^{(k)},\hat{\rho}^{(k)})$ that
converge to a deterministic limit $\nu^{*}:=(\sigma^{*},\rho^{*})$, each with
corresponding weights $\hat{W}_{i}=W(\hat{\nu}^{(k)})(X_{i})$ (with
$k$-dependence implicit through $i$) and $W_{i}^{*}=W(\nu^{*})(X_{i})$. By
scaling we may assume without loss of generality that in Assumption (A3.1) we
have $c^{*}=1$.

Then $\hat{\beta}$ in Algorithm 1 can be expanded as

$\displaystyle N^{\frac{1}{2}}\big{(}\hat{\beta}-\beta\big{)}=$
$\displaystyle\left(N^{-1}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\phi\big{(}S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}\right)^{-1}\left(N^{-\frac{1}{2}}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\varphi\big{(}S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}\right)$
$\displaystyle=$
$\displaystyle\left(\frac{N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]}{\left(N^{-1}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\phi(S_{i};\eta_{0},\nu^{*})\right]\right)^{2}}\right)^{\frac{1}{2}}\left(\frac{N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}\phi\left(S_{i};\eta_{0},\nu^{*}\right)+M_{2}}{N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\phi(S_{i};\eta_{0},\nu^{*})\right]}\right)^{-1}$
$\displaystyle\qquad\cdot\left(\frac{N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i=1}^{I}\varphi(S_{i};\beta,\eta_{0},\nu^{*})+M_{1}}{\left(N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\right)^{\frac{1}{2}}}\right),$
(27)

where

$\displaystyle M_{1}$
$\displaystyle:=N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\left[\varphi\big{(}S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}-\varphi\left(S_{i};\beta,\eta_{0},\nu^{*}\right)\right],$
$\displaystyle M_{2}$
$\displaystyle:=N^{-1}n_{\max}^{\gamma}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\left[\phi\big{(}S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}-\phi\left(S_{i};\eta_{0},\nu^{*}\right)\right].$

We claim that by Lemma 17 we have

$\lim_{I\to\infty}\sup_{P\in\mathcal{P}_{I}}\sup_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(\left(\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\right)^{-\frac{1}{2}}\left(\sum_{i=1}^{I}\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right)\leq
t\right)-\Phi(t)\right|=0.$ (28)

We must verify the condition for Lemma 17 to hold; we will show that

$\sup_{P\in\mathcal{P}_{I}}\frac{\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right|^{2+2\Delta}\right]}{\left(\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\right)^{1+\Delta}}=o(1),$
(29)

holds for $\Delta=\frac{\delta}{4}$. First, note that for an arbitrary
$P\in\mathcal{P}_{I}$,

$\displaystyle\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|\varphi\left(S_{i};\beta,\eta_{0},\nu^{*}\right)\right|^{2+2\Delta}\right]=$
$\displaystyle\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|\varepsilon_{i}^{\top}W_{i}^{*}\xi_{i}\right|^{2+2\Delta}\right]$
$\displaystyle\leq$
$\displaystyle\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left\lVert
W_{i}^{*}\right\rVert^{2+2\Delta}_{\mathrm{op}}\left\lVert\xi_{i}\right\rVert_{2}^{2+2\Delta}\left\lVert\varepsilon_{i}\right\rVert_{2}^{2+2\Delta}\right]$
$\displaystyle\lesssim$
$\displaystyle\sum_{i=1}^{I}\left\lVert\xi_{i}\right\rVert_{P,4+4\Delta}^{2+2\Delta}\left\lVert\varepsilon_{i}\right\rVert_{P,4+4\Delta}^{2+2\Delta}\lesssim\sum_{i=1}^{I}n_{i}^{2+2\Delta}\leq
Nn_{\max}^{1+2\Delta},$
$\displaystyle\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]=$
$\displaystyle\sum_{i=1}^{I}\mathbb{E}_{P}\left[\mathrm{tr}\left(\Sigma_{i}W_{i}^{*}\xi_{i}\xi_{i}^{\top}W_{i}^{*}\right)\right]$
$\displaystyle\geq$
$\displaystyle\sum_{i=1}^{I}\mathbb{E}_{P}\left[\Lambda_{\min}\left(W_{i}^{*\frac{1}{2}}\Sigma_{i}W_{i}^{*\frac{1}{2}}\right)\left\lVert
W_{i}^{*}\right\rVert_{\mathrm{op}}\mathrm{tr}\left(\xi_{i}\xi_{i}^{\top}\right)\right]$
$\displaystyle\gtrsim$
$\displaystyle\sum_{i=1}^{I}n_{i}^{-\kappa}\mathbb{E}_{P}\left[\mathrm{tr}\left(\xi_{i}\xi_{i}^{\top}\right)\right]=\sum_{i=1}^{I}n_{i}^{-\kappa}\mathbb{E}_{P}\left[\mathrm{tr}\left(\Omega_{i}\right)\right]$
$\displaystyle\geq$
$\displaystyle\sum_{i=1}^{I}n_{i}^{1-\kappa}\mathbb{E}_{P}\left[\Lambda_{\min}\left(\Omega_{i}\right)\right]\gtrsim\sum_{i=1}^{I}n_{i}^{1-\kappa}\geq
Nn_{\max}^{-\kappa},$

where the bounding of the numerator follows by the Cauchy–Schwarz inequality
and Assumptions (A1.1) and (A3.3), and the bounding of the denominator follows
by Assumptions (A1.2) and (A3.3). Therefore,

$\displaystyle\sup_{P\in\mathcal{P}_{I}}\frac{\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right|^{2+2\Delta}\right]}{\left(\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\right)^{1+\Delta}}\leq
N^{-\Delta}n_{\max}^{(1+\kappa)+(2+\kappa)\Delta}=o(1)$

by Assumption A4, with $\Delta=\frac{\delta}{4}$. Therefore, the condition
(29) holds, and so the claim (28) is proved.

Further, $M_{1}=o_{\mathcal{P}}(1)$ by Lemma 8 and (as shown already) for
arbitrary $P\in\mathcal{P}_{I}$

$\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\gtrsim
1$

therefore, by Lemma 18 in conjunction with (28) we have that

$\lim_{I\to\infty}\sup_{P\in\mathcal{P}_{I}}\sup_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(\frac{N^{-1}n_{\max}^{\frac{\kappa}{2}}\sum_{i=1}^{I}\varphi(S_{i};\beta,\eta_{0},\nu^{*})+M_{1}}{\left(N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\right)^{\frac{1}{2}}}\leq
t\right)-\Phi\left(t\right)\right|=0.$ (30)

Next, we claim that

$\frac{N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}\phi\left(S_{i};\eta_{0},\nu^{*}\right)+M_{2}}{N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\phi(S_{i};\eta_{0},\nu^{*})\right]}=1+o_{\mathcal{P}}(1),$
(31)

which we proceed to show. We claim that

$N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}\phi\left(S_{i};\eta_{0},\nu^{*}\right)=N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\phi\left(S_{i};\eta_{0},\nu^{*}\right)\right]+o_{\mathcal{P}}(1).$

This follows by Lemma 15 alongside

$\begin{split}&\mathbb{E}_{P}\left[\bigg{(}N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}\left(\phi\left(S_{i};\eta_{0},\nu^{*}\right)-\mathbb{E}_{P}\left[\phi\left(S_{i};\eta_{0},\nu^{*}\right)\right]\right)\bigg{)}^{2}\right]\\\
\leq&N^{-2}n_{\max}^{2\gamma}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left(\phi\left(S_{i};\eta_{0},\nu^{*}\right)-\mathbb{E}_{P}\left[\phi\left(S_{i};\eta_{0},\nu^{*}\right)\right]\right)^{2}\right]\\\
\leq&N^{-2}n_{\max}^{2\gamma}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left(\phi\left(S_{i};\eta_{0},\nu^{*}\right)\right)^{2}\right]\leq
N^{-2}n_{\max}^{2\gamma}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left\lVert
W_{i}^{*}\right\rVert_{\mathrm{op}}^{2}\left\lVert\xi_{i}\right\rVert_{2}^{4}\right]\\\
\lesssim&N^{-2}n_{\max}^{2\gamma}\sum_{i=1}^{I}n_{i}^{2}\leq
N^{-1}n_{\max}^{1+2\gamma}=o(1),\end{split}$

by the Cauchy–Schwarz inequality and Assumptions (A1.1), (A3.3) and A4.
Because also $M_{2}=o_{\mathcal{P}}(1)$ by Lemma 9, as well as that

$\begin{split}N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\phi(S_{i};\eta_{0},\nu^{*})\right]&=N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\mathrm{tr}\left(W_{i}^{*}\xi_{i}\xi_{i}^{\top}\right)\right]\\\
&\geq
N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\Lambda_{\min}\left(W_{i}^{*}\right)\mathrm{tr}\left(\xi_{i}\xi_{i}^{\top}\right)\right]\\\
&\gtrsim
N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}n_{i}^{-\gamma}\mathbb{E}_{P}\left[\mathrm{tr}\left(\Omega_{i}\right)\right]\\\
&\gtrsim N^{-1}n_{\max}^{\gamma}\sum_{i=1}^{I}n_{i}^{1-\gamma}\geq
1,\end{split}$

we have, by Lemma 12, proved the claim (31).

Finally, combining (30) with (31) with Lemma 18 completes the proof;

$\lim_{I\to\infty}\sup_{P\in\mathcal{P}_{I}}\sup_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(N^{\frac{1}{2}}{V}^{-\frac{1}{2}}\big{(}\hat{\beta}-\beta\big{)}\leq
t\right)-\Phi\left(t\right)\right|=0.$

∎

###### Proof of Theorem 4 (ii).

Note that

$\hat{V}=\bigg{(}N^{-1}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\phi\big{(}S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}\bigg{)}^{-2}\bigg{(}N^{-1}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\varphi^{2}\big{(}S_{i};\hat{\beta},\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}\bigg{)},$

and so we can decompose

$\displaystyle
N^{\frac{1}{2}}\hat{V}^{-\frac{1}{2}}\big{(}\hat{\beta}-\beta\big{)}=$
$\displaystyle\left(\frac{N^{-1}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\varphi^{2}\big{(}S_{i};\hat{\beta},\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}}{N^{-1}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]}\right)^{-\frac{1}{2}}\cdot$
$\displaystyle\qquad\cdot\left(\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\right)^{-\frac{1}{2}}\left(\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\varphi\big{(}S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}\right)$
(32) $\displaystyle=$
$\displaystyle\left(\frac{N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]+M_{3}+M_{4}}{N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]}\right)^{-\frac{1}{2}}\cdot$
(33)
$\displaystyle\qquad\cdot\left(\frac{N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i=1}^{I}\varphi(S_{i};\beta,\eta_{0},\nu^{*})+M_{1}}{\left(N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\right)^{\frac{1}{2}}}\right),$
(34)

where

$\begin{split}M_{3}&:=N^{-1}n_{\max}^{\kappa}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\left[\varphi^{2}\big{(}S_{i};\hat{\beta},\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}-\varphi^{2}\left(S_{i};\beta,\eta_{0},\nu^{*}\right)\right],\\\
M_{4}&:=N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})-\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\right].\end{split}$

Note that the latter term in this decomposition is considered in the proof of
Theorem 4; specifically the result (30). Therefore it remains for us to show
that

$\frac{N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]+M_{3}+M_{4}}{N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]}=1+o_{\mathcal{P}}(1).$
(35)

From Lemmas 10 and 11 we have that $M_{3}$ and $M_{4}$ are both
$o_{\mathcal{P}}(1)$. Further (as shown in the proof of Theorem 4),

$\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\gtrsim
1,$

and so combining these results with Lemmas 12 and 14 yields the result (35).
Finally, the results (30) and (35) in conjunction with Lemma 18 completes the
proof.

∎

###  D.2 Auxiliary results for the proof of Theorem 4: bounding terms

######  Lemma 7.

Under the setup and notation of Theorem 4, we have that

$\big{\|}\hat{W}_{i}-W_{i}^{*}\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},2}\lesssim
n_{i}^{\frac{1+\tau}{2}}\mathcal{R}_{\sigma}^{\frac{1}{2}}+n_{i}\mathcal{R}_{\rho}^{\frac{1}{2}}.$

###### Proof.

We define

$\Sigma_{i}(\sigma,\rho):=\Big{(}\big{(}\mathbbm{1}_{\\{j=k\\}}+\mathbbm{1}_{\\{j\neq
k\\}}\rho(X_{ij},X_{ik})\big{)}\sigma(X_{ij})\sigma(X_{ik})\Big{)}_{(j,k)\in[n_{i}]^{2}},$

and further $\Sigma_{i}^{*}:=\Sigma_{i}(\sigma^{*},\rho^{*})$ and
$\hat{\Sigma}_{i}:=\Sigma_{i}(\hat{\sigma}^{(k)},\hat{\rho}^{(k)})$
(suppressing dependence on $k\in[K]$).

Note that, as
$\hat{W}_{i}-W_{i}^{*}\asymp\hat{\Sigma}^{-1}_{i}-\Sigma^{*^{-1}}_{i}$,

$\big{\|}\hat{W}_{i}-W_{i}^{*}\big{\|}_{\mathrm{op}}\lesssim\big{\|}\hat{\Sigma}^{-1}_{i}\big{(}\Sigma^{*}_{i}-\hat{\Sigma}_{i}\big{)}\Sigma^{*^{-1}}_{i}\big{\|}_{\mathrm{op}}\leq\big{\|}\Sigma^{*^{-1}}_{i}\big{\|}_{\mathrm{op}}\big{\|}\hat{\Sigma}^{-1}_{i}\big{\|}_{\mathrm{op}}\big{\|}\hat{\Sigma}_{i}-\Sigma^{*}_{i}\big{\|}_{\mathrm{op}}\lesssim\big{\|}\hat{\Sigma}_{i}-\Sigma^{*}_{i}\big{\|}_{\mathrm{op}},$

and further,

$\displaystyle\big{\|}\hat{\Sigma}_{i}-\Sigma^{*}_{i}\big{\|}_{\mathrm{op}}^{2}$
$\displaystyle\leq\big{\|}\hat{\Sigma}_{i}-\Sigma^{*}_{i}\big{\|}_{F}^{2}$
$\displaystyle=\sum_{j=1}^{n_{i}}\Big{(}\hat{\sigma}^{(k)}(X_{ij})^{2}-\sigma^{*}(X_{ij})^{2}\Big{)}^{2}$
$\displaystyle\quad+\sum_{j=1}^{n_{i}}\sum_{j^{\prime}=1,j^{\prime}\neq
j}^{n_{i}}\Big{(}\hat{\rho}^{(k)}(X_{ij},X_{ij^{\prime}})\hat{\sigma}^{(k)}(X_{ij})\hat{\sigma}^{(k)}(X_{ij^{\prime}})-\rho^{*}(X_{ij},X_{ij^{\prime}})\sigma^{*}(X_{ij})\sigma^{*}(X_{ij^{\prime}})\Big{)}^{2}$
$\displaystyle\lesssim\sum_{j=1}^{n_{i}}\Big{(}\hat{\sigma}^{(k)}(X_{ij})-\sigma^{*}(X_{ij})\Big{)}^{2}\bigg{(}1+\sum_{j^{\prime}=1,j^{\prime}\neq
j}^{n_{i}}\hat{\rho}^{(k)}(X_{ij},X_{ij^{\prime}})^{2}\bigg{)}$
$\displaystyle\quad+\sum_{j=1}^{n_{i}}\sum_{j^{\prime}=1,j^{\prime}\neq
j}^{n_{i}}\big{(}\hat{\rho}^{(k)}(X_{ij},X_{ij^{\prime}})-\rho^{*}(X_{ij},X_{ij^{\prime}})\big{)}^{2},$

where $\left\lVert\cdot\right\rVert_{F}$ denotes the Frobenius norm, and so

$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\big{\|}\hat{\Sigma}_{i}-\Sigma^{*}_{i}\big{\|}_{\mathrm{op}}^{2}\right]$
$\displaystyle\lesssim
n_{i}^{1+\tau}\mathcal{R}_{\sigma}+n_{i}^{2}\mathcal{R}_{\rho},$

and hence

$\big{\|}\hat{W}_{i}-W_{i}^{*}\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},2}=\left(\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\big{\|}\hat{W}_{i}-W_{i}^{*}\big{\|}_{\mathrm{op}}^{2}\right]\right)^{\frac{1}{2}}\lesssim
n_{i}^{\frac{1+\tau}{2}}\mathcal{R}_{\sigma}^{\frac{1}{2}}+n_{i}\mathcal{R}_{\rho}^{\frac{1}{2}}.$

completing the proof. ∎

######  Lemma 8.

Under the setup of Theorem 4,

$M_{1}:=N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\left[\varphi\big{(}S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}-\varphi\left(S_{i};\beta,\eta_{0},\nu^{*}\right)\right]=o_{\mathcal{P}}(1).$

###### Proof.

For $k\in[K]$ and $i\in\mathcal{I}_{k}$, define the
$(\hat{l}^{(k)},\hat{m}^{(k)})$ errors

$\begin{gathered}\mathcal{E}_{m}^{(k)}(X_{i}):=\hat{m}^{(k)}\left(X_{i}\right)-m\left(X_{i}\right),\qquad\mathcal{E}_{l}^{(k)}(X_{i}):=\hat{l}^{(k)}\left(X_{i}\right)-l\left(X_{i}\right),\\\
\mathcal{E}_{l-\beta
m}^{(k)}(X_{i}):=\mathcal{E}_{l}^{(k)}(X_{i})-\beta\mathcal{E}_{m}^{(k)}(X_{i}).\end{gathered}$

The term $M_{1}$ can then be decomposed as $M_{1}=\sum_{k=1}^{K}M_{1,k}$,
where

$M_{1,k}:=\underbrace{N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})}_{\textup{I}}+\underbrace{N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\xi_{i}}_{\textup{II}}\\\
+\underbrace{N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\mathcal{E}_{m}^{(k)}(X_{i})^{\top}W_{i}^{*}\varepsilon_{i}}_{\textup{III}}+\underbrace{N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\left(\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\right)^{\top}\left(\hat{W}_{i}-W_{i}^{*}\right)\varepsilon_{i}}_{\textup{IV}}.$

We show that, for an arbitrary $k\in[K]$, each of these four terms are
$o_{\mathcal{P}}(1)$ in turn:

##### Term I:

Note that for an arbitrary $P\in\mathcal{P}_{I}$,

$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\bigg{|}N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\bigg{|}\right]$
$\displaystyle\leq$ $\displaystyle
N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\sum_{i\in\mathcal{I}_{k}}\left|\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right|\right]$
$\displaystyle\leq$ $\displaystyle
N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})\right]^{\frac{1}{2}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\mathcal{E}_{m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right]^{\frac{1}{2}}$
$\displaystyle\lesssim$ $\displaystyle
N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\big{\|}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},2}\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},2}$
$\displaystyle\leq$ $\displaystyle
N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}n_{i}\mathcal{R}_{m}^{\frac{1}{2}}\left(\mathcal{R}_{l}+\mathcal{R}_{m}\right)^{\frac{1}{2}}=N^{\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\mathcal{R}_{m}^{\frac{1}{2}}\left(\mathcal{R}_{l}+\mathcal{R}_{m}\right)^{\frac{1}{2}}=o_{\mathcal{P}}(1),$

by Assumption A4. The result that this term I is $o_{\mathcal{P}}(1)$ follows
directly by Lemma 15.

##### Term II:

First note that
$\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\xi_{i}\right]=0$ for any
$P\in\mathcal{P}_{I}$ and $i\in\mathcal{I}_{k}$. Then we have

$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\bigg{(}N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\xi_{i}\bigg{)}^{2}\right]$ $\displaystyle=$
$\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\xi_{i}\right)^{2}\right]$ $\displaystyle=$
$\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\Omega_{i}\hat{W}_{i}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})\right]$ $\displaystyle\leq$ $\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\Lambda_{\max}(\Omega_{i})\left\lVert\hat{W}_{i}\right\rVert_{\mathrm{op}}^{2}\big{\|}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},2}^{2}\right]$
$\displaystyle\lesssim$ $\displaystyle
N^{-1}n_{\max}^{\kappa+\mu_{\Omega}}\left(\mathcal{R}_{l}+\mathcal{R}_{m}\right)=o_{\mathcal{P}}(1),$

by Assumption A4. The result that this term II is $o_{\mathcal{P}}(1)$ follows
directly by Lemma 15.

##### Term III:

First note that
$\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\mathcal{E}^{m}_{k}(X_{i})^{\top}\hat{W}_{i}\varepsilon_{i}\right]=0$
for any $P\in\mathcal{P}_{I}$ and $i\in\mathcal{I}_{k}$. Then

$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\bigg{(}N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\mathcal{E}^{m}_{k}(X_{i})^{\top}W_{i}^{*}\varepsilon_{i}\bigg{)}^{2}\right]$
$\displaystyle=$ $\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\left(\mathcal{E}^{m}_{k}(X_{i})^{\top}W_{i}^{*}\varepsilon_{i}\right)^{2}\right]$
$\displaystyle=$ $\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\mathcal{E}_{m}^{(k)}(X_{i})^{\top}W_{i}^{*}\Sigma_{i}W_{i}^{*}\mathcal{E}_{m}^{(k)}(X_{i})\right]$
$\displaystyle\leq$ $\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\Lambda_{\max}(W_{i}^{*\frac{1}{2}}\Sigma_{i}W_{i}^{*\frac{1}{2}})\left\lVert
W_{i}^{*}\right\rVert_{\mathrm{op}}\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{2}^{2}\right]$
$\displaystyle\lesssim$ $\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}n_{i}^{1+\alpha}\mathcal{R}_{m}\leq
n_{\max}^{\kappa+\alpha}\mathcal{R}_{m}=o_{\mathcal{P}}(1),$

by Assumption A4. The result that this term II is $o_{\mathcal{P}}(1)$ follows
directly by Lemma 15.

##### Term IV:

Similarly, note that
$\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\Big{[}\Big{(}\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\Big{)}^{\top}\big{(}\hat{W}_{i}-W_{i}^{*}\big{)}\varepsilon_{i}\Big{]}=0$
for any $P\in\mathcal{P}_{I}$ and $i\in\mathcal{I}_{k}$. Then

$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\bigg{[}\Big{(}\Big{(}\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\Big{)}^{\top}\big{(}\hat{W}_{i}-W_{i}^{*}\big{)}\varepsilon_{i}\Big{)}^{2}\bigg{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\left(\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\right)^{\top}\big{(}\hat{W}_{i}-W_{i}^{*}\big{)}\Sigma_{i}\big{(}\hat{W}_{i}-W_{i}^{*}\big{)}\left(\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\right)\right]$
$\displaystyle\leq$
$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\Lambda_{\max}(\Sigma_{i})\big{\|}\hat{W}_{i}-W_{i}^{*}\big{\|}_{\mathrm{op}}^{2}\left(\left\lVert\xi_{i}\right\rVert_{2}^{2}+\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{2}^{2}\right)\right]$
$\displaystyle\leq$
$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\Lambda_{\max}(\Sigma_{i})\big{\|}\hat{W}_{i}-W_{i}^{*}\big{\|}_{\mathrm{op}}\left(\left\lVert\xi_{i}\right\rVert_{2}^{2}+\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{2}^{2}\right)\right]$
$\displaystyle\lesssim$ $\displaystyle
n_{i}^{\mu_{\Sigma}}\big{\|}\hat{W}_{i}-W_{i}^{*}\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},2}\Big{(}\left\lVert\xi_{i}\right\rVert_{P|S_{\mathcal{I}_{k}^{c}},4}^{2}+\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},4}^{2}\Big{)}$
$\displaystyle\lesssim$ $\displaystyle
n_{i}^{1+\mu_{\Sigma}}\Big{(}n_{i}^{\frac{1+\tau}{2}}\mathcal{R}_{\sigma}^{\frac{1}{2}}+n_{i}\mathcal{R}_{\rho}^{\frac{1}{2}}\Big{)}$

with the second inequality holding because
$\big{\|}\hat{W}_{i}-W_{i}^{*}\big{\|}_{\mathrm{op}}\leq\big{\|}\hat{W}_{i}\big{\|}_{\mathrm{op}}+\big{\|}W_{i}^{*}\big{\|}_{\mathrm{op}}\lesssim
1$ almost surely, and the fourth inequality obtained by invoking Lemma 7.
Therefore

$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\bigg{[}\Big{(}N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\left(\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\right)^{\top}\big{(}\hat{W}_{i}-W_{i}^{*}\big{)}\varepsilon_{i}\Big{)}^{2}\bigg{]}$
$\displaystyle\leq$ $\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\bigg{[}\Big{(}\left(\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\right)^{\top}\big{(}\hat{W}_{i}-W_{i}^{*}\big{)}\varepsilon_{i}\Big{)}^{2}\bigg{]}$
$\displaystyle\lesssim$ $\displaystyle
n_{\max}^{\kappa+\mu_{\Sigma}}\Big{(}n_{\max}^{\frac{1+\tau}{2}}\mathcal{R}_{\sigma}^{\frac{1}{2}}+n_{i}\mathcal{R}_{\rho}^{\frac{1}{2}}\Big{)}=o_{\mathcal{P}}(1),$

by Assumption A4. The result that this term IV is $o_{\mathcal{P}}(1)$ follows
directly by Lemma 15.

All four terms are therefore $o_{\mathcal{P}}(1)$ and so applying Lemma 12
completes the proof. ∎

######  Lemma 9.

Under the setup of Theorem 4,

$M_{2}:=N^{-1}n_{\max}^{\gamma}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\left[\phi\big{(}S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}-\phi\left(S_{i};\eta_{0},\nu^{*}\right)\right]=o_{\mathcal{P}}(1).$

###### Proof.

We adopt the same notation as in Lemma 8. Then we can decompose $M_{2}$ into
$M_{2}=\sum_{k=1}^{K}M_{2,k}$, where

$M_{2,k}:=\underbrace{N^{-1}n_{\max}^{\gamma}\sum_{i\in\mathcal{I}_{k}}\mathcal{E}_{m}^{(k)}(X_{i})\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})}_{\textup{V}}-2\underbrace{N^{-1}n_{\max}^{\gamma}\sum_{i\in\mathcal{I}_{k}}\xi_{i}^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})}_{\textup{VI}}\\\
+\underbrace{N^{-1}n_{\max}^{\gamma}\sum_{i\in\mathcal{I}_{k}}\xi_{i}^{\top}\left(\hat{W}_{i}-W_{i}^{*}\right)\xi_{i}}_{\textup{VII}}.$

For an arbitrary $k\in[K]$ we will show that each of these three terms are
each $o_{\mathcal{P}}(1)$:

##### Term V:

We have that

$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\bigg{[}N^{-1}n_{\max}^{\gamma}\sum_{i\in\mathcal{I}_{k}}\mathcal{E}_{m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\bigg{]}\lesssim$
$\displaystyle
N^{-1}n_{\max}^{\gamma}\sum_{i\in\mathcal{I}_{k}^{c}}\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},2}^{2}$
$\displaystyle\lesssim$ $\displaystyle
n_{\max}^{\gamma}\mathcal{R}_{m}=o_{\mathcal{P}}(1),$

and so (by Lemma 15) this term is $o_{\mathcal{P}}(1)$.

##### Term VI:

$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\bigg{[}\Big{(}N^{-1}n_{\max}^{\gamma}\sum_{i\in\mathcal{I}_{k}}\mathcal{E}_{m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\xi\Big{)}^{2}\bigg{]}$
$\displaystyle=$ $\displaystyle
N^{-2}n_{\max}^{2\gamma}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\mathcal{E}_{m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\Omega_{i}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})^{\top}\right]$
$\displaystyle\lesssim$ $\displaystyle
N^{-2}n_{\max}^{2\gamma}\sum_{i\in\mathcal{I}_{k}}n_{i}^{1+\mu_{\Omega}}\mathcal{R}_{m}\leq
N^{-1}n_{\max}^{2\gamma+\mu_{\Omega}}\mathcal{R}_{m}=o_{\mathcal{P}}(1),$

and so (again by Lemma 15) this term is $o_{\mathcal{P}}(1)$.

##### Term VII:

$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\bigg{[}\Big{|}N^{-1}n_{\max}^{\gamma}\sum_{i\in\mathcal{I}_{k}}\xi_{i}^{\top}(\hat{W}_{i}-W_{i}^{*})\xi\Big{|}\bigg{]}$
$\displaystyle\leq$ $\displaystyle
N^{-1}n_{\max}^{\gamma}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\big{\|}\hat{W}_{i}-W_{i}^{*}\big{\|}_{\mathrm{op}}\left\lVert\xi_{i}\right\rVert_{2}^{2}\right]$
$\displaystyle\lesssim$ $\displaystyle
n_{\max}\Big{(}n_{\max}^{\frac{1+\tau}{2}}\mathcal{R}_{\sigma}^{\frac{1}{2}}+n_{\max}\mathcal{R}_{\rho}^{\frac{1}{2}}\Big{)}=o_{\mathcal{P}}(1),$

and so similarly this term is $o_{\mathcal{P}}(1)$.

Combining these results with Lemma 12 proves the claim. ∎

######  Lemma 10.

Under the setup of Theorem 4,

$M_{3}:=N^{-1}n_{\max}^{\kappa}\sum_{k=1}^{K}\sum_{i\in\mathcal{I}_{k}}\left[\varphi^{2}\big{(}S_{i};\hat{\beta},\hat{\eta}^{(k)},\hat{\nu}^{(k)}\big{)}-\varphi^{2}\left(S_{i};\beta,\eta_{0},\nu^{*}\right)\right]=o_{\mathcal{P}}(1).$

###### Proof.

First, decompose the $M_{3}$ term as $M_{3}=\sum_{k=1}^{K}M_{3,k}$, where

$\displaystyle M_{3,k}$
$\displaystyle:=N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left[\varphi^{2}(S_{i};\hat{\beta},\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]$
$\displaystyle=N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left(\varphi(S_{i};\hat{\beta},\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right)^{2}$
$\displaystyle\quad+N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\varphi(S_{i};\beta,\eta_{0},\nu^{*})\left(\varphi(S_{i};\hat{\beta},\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right)$
$\displaystyle\leq
2N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left(\varphi(S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right)^{2}$
$\displaystyle\quad+2N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left(\varphi(S_{i};\hat{\beta},\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi(S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)})\right)^{2}$
$\displaystyle\quad+N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\varphi(S_{i};\beta,\eta_{0},\nu^{*})\left(\varphi(S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right)$
$\displaystyle\quad+N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\varphi(S_{i};\beta,\eta_{0},\nu^{*})\left(\varphi(S_{i};\hat{\beta},\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi(S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)})\right)$
$\displaystyle=\underbrace{2N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left(\varphi(S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right)^{2}}_{\textup{VIII}}$
$\displaystyle\quad+\underbrace{N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\varphi(S_{i};\beta,\eta_{0},\nu^{*})\left(\varphi(S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right)}_{\textup{IX}}$
$\displaystyle\quad+\underbrace{2N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\phi^{2}(S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)})\big{(}\hat{\beta}-\beta\big{)}^{2}}_{\textup{X}}$
$\displaystyle\quad+\underbrace{N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\varphi(S_{i};\beta,\eta_{0},\nu^{*})\phi(S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)})\big{(}\hat{\beta}-\beta\big{)}}_{\textup{XI}}$

We claim that each term is $o_{\mathcal{P}}(1)$, and prove this for each term
separately:

##### Term VIII:

We decompose this term into the four subsequent terms

$\displaystyle
2N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left(\varphi(S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right)^{2}$
$\displaystyle\lesssim$
$\displaystyle\underbrace{N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left|\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right|^{2}}_{\textup{VIII}\text{a}}+\underbrace{N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\xi_{i}\right)^{2}}_{\textup{VIII}\text{b}}$
$\displaystyle+\underbrace{N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left(\mathcal{E}_{m}^{(k)}(X_{i})^{\top}W_{i}^{*}\varepsilon_{i}\right)^{2}}_{\textup{VIII}\text{c}}+\underbrace{N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left(\left(\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\right)^{\top}\big{(}\hat{W}_{i}-W_{i}^{*}\big{)}\varepsilon_{i}\right)^{2}}_{\textup{VIII}\text{d}}.$

Note that

$\displaystyle\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\bigg{(}N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\left|\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right|^{2}\bigg{)}^{\frac{1}{2}}\right]$
$\displaystyle\leq$ $\displaystyle
N^{-\frac{1}{2}}n_{\max}^{\frac{\kappa}{2}}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\left|\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right|\right]=o_{\mathcal{P}}(1)$

as shown when dealing with Term I in the decomposition for Lemma 8, and so the
square root of VIIIa and hence VIIIa are both $o_{\mathcal{P}}(1)$. Further,
the expectation under the law of $P|S_{\mathcal{I}_{k}^{c}}$ of the terms
VIIIb, VIIIc and VIIId are each shown in Lemma 8 to be $o_{\mathcal{P}}(1)$,
and so the terms VIIIb, VIIIc and VIIId are all $o_{\mathcal{P}}(1)$, thus
(again by Lemma 15) the term V is $o_{\mathcal{P}}(1)$.

##### Term IX:

Note that this term can be further decomposed into the four terms

$\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\varphi(S_{i};\beta,\eta_{0},\nu^{*})\left(\varphi(S_{i};\beta,\hat{\eta}^{(k)},\hat{\nu}^{(k)})-\varphi(S_{i};\beta,\eta_{0},\nu^{*})\right)$
$\displaystyle=$
$\displaystyle\underbrace{N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right)}_{\textup{IX}\text{a}}$
$\displaystyle\quad+\underbrace{N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}^{*}\xi_{i}\right)}_{\textup{IX}\text{b}}+\underbrace{N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}\left(\mathcal{E}_{m}^{(k)}(X_{i})^{\top}W_{i}^{*}\varepsilon_{i}\right)}_{\textup{IX}\text{c}}$
$\displaystyle\quad+\underbrace{N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\Big{(}\big{(}\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\big{)}^{\top}W_{i}^{*}\varepsilon_{i}\Big{)}\left(\xi_{i}^{\top}\big{(}\hat{W}_{i}-W_{i}^{*}\big{)}\varepsilon_{i}\right)}_{\textup{IX}\text{d}}.$

We claim that each of these four terms are $o_{\mathcal{P}}(1)$; we will show
each in turn. For the first term IXa note that
$\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\left(\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\right)\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right)\right]=0$
for all $P\in\mathcal{P}_{I}$ and $i\in\mathcal{I}_{k}$. We consider the cases
of $\delta\in[0,4)$ and $\delta\geq 4$ separately. If $\delta\in[0,4)$ then by
the von Bahr–Esseen inequality

$\displaystyle\mathbb{E}_{P}\bigg{[}\Big{|}N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right)\Big{|}^{1+\frac{\delta}{4}}\bigg{]}$
$\displaystyle\leq$
$\displaystyle\left(2-I^{-1}\right)N^{-1-\frac{\delta}{4}}n_{\max}^{\kappa\left(1+\frac{\delta}{4}\right)}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P}\left[\left|\big{(}\xi_{i}^{\top}W_{i}\varepsilon_{i}\big{)}\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right)\right|^{1+\frac{\delta}{4}}\right]$
$\displaystyle\lesssim$ $\displaystyle
N^{-1-\frac{\delta}{4}}n_{\max}^{\kappa\left(1+\frac{\delta}{4}\right)}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P}\left[\left\lVert\xi_{i}\right\rVert_{2}^{1+\frac{\delta}{4}}\left\lVert\varepsilon_{i}\right\rVert_{2}^{1+\frac{\delta}{4}}\big{\|}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})\big{\|}_{2}^{1+\frac{\delta}{4}}\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{2}^{1+\frac{\delta}{4}}\right]$
$\displaystyle\lesssim$ $\displaystyle
N^{-1-\frac{\delta}{4}}n_{\max}^{\kappa\left(1+\frac{\delta}{4}\right)}\sum_{i\in\mathcal{I}_{k}}\left\lVert\xi_{i}\right\rVert_{P,4+\delta}^{1+\frac{\delta}{4}}\left\lVert\varepsilon_{i}\right\rVert_{P,4+\delta}^{1+\frac{\delta}{4}}\big{\|}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},4+\delta}^{1+\frac{\delta}{4}}\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},4+\delta}^{1+\frac{\delta}{4}}$
$\displaystyle\lesssim$ $\displaystyle
N^{-1-\frac{\delta}{4}}n_{\max}^{\kappa\left(1+\frac{\delta}{4}\right)}\sum_{i\in\mathcal{I}_{k}}n_{i}^{2+\frac{\delta}{2}}\leq
N^{-\frac{\delta}{4}}n_{\max}^{(1+\kappa)+(2+\kappa)\frac{\delta}{4}},$

for any $P\in\mathcal{P}_{I}$, and thus

$\displaystyle\sup_{P\in\mathcal{P}_{I}}\mathbb{E}_{P}\bigg{[}\Big{|}N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right)\Big{|}^{1+\frac{\delta}{4}}\bigg{]}$
$\displaystyle\lesssim
N^{-\frac{\delta}{4}}n_{\max}^{(1+\kappa)+(2+\kappa)\frac{\delta}{4}}$
$\displaystyle=o(1),$

by Assumption A4. If on the other hand $\delta\geq 4$ then note that

$\displaystyle\mathbb{E}_{P}\bigg{[}\Big{(}N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right)\Big{)}^{2}\bigg{]}$
$\displaystyle=$ $\displaystyle
N^{-2}n_{\max}^{2\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P}\left[\left|\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right)\right|^{2}\right]$
$\displaystyle\lesssim$ $\displaystyle
N^{-2}n_{\max}^{2\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P}\left[\left\lVert\xi_{i}\right\rVert_{2}^{2}\left\lVert\varepsilon_{i}\right\rVert_{2}^{2}\big{\|}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})\big{\|}_{2}^{2}\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{2}^{2}\right]$
$\displaystyle\lesssim$ $\displaystyle
N^{-2}n_{\max}^{2\kappa}\sum_{i\in\mathcal{I}_{k}}\left\lVert\xi_{i}\right\rVert_{P,8}^{2}\left\lVert\varepsilon_{i}\right\rVert_{P,8}^{2}\big{\|}\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},8}^{2}\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{P|S_{\mathcal{I}_{k}^{c}},8}^{2}$
$\displaystyle\lesssim$ $\displaystyle
N^{-2}n_{\max}^{2\kappa}\sum_{i\in\mathcal{I}_{k}}n_{i}^{4}\leq
N^{-1}n_{\max}^{3+2\kappa},$

for all $P\in\mathcal{P}_{I}$, and thus

$\sup_{P\in\mathcal{P}_{I}}\mathbb{E}_{P}\bigg{[}\Big{(}N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\mathcal{E}_{m}^{(k)}(X_{i})\right)\Big{)}^{2}\bigg{]}\lesssim
N^{-1}n_{\max}^{3+2\kappa}=o(1),$

by Assumption A4. Hence, by Lemma 15, the term IXa is $o_{\mathcal{P}}(1)$.

For the term IXb, note that as
$\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\left(\xi_{i}^{\top}W_{i}\varepsilon_{i}\right)\left(\mathcal{E}_{l-\beta
m}^{(k)}(X_{i})^{\top}\hat{W}_{i}\xi_{i}\right)\right]=0$ for any
$P\in\mathcal{P}_{I}$ and $i\in\mathcal{I}_{k}$, and so the same methodology
as for the term IXa can be used for IXb, showing that this term is also
$o_{\mathcal{P}}(1)$.

For the term IXc, note that

$\begin{split}&\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\bigg{[}\Big{|}N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}\left(\mathcal{E}_{m}^{(k)}(X_{i})^{\top}W_{i}^{*}\varepsilon_{i}\right)\Big{|}\bigg{]}\\\
\leq&N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\left|\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}\left(\mathcal{E}_{m}^{(k)}(X_{i})^{\top}W_{i}^{*}\varepsilon_{i}\right)\right|\right]\\\
\leq&N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\big{(}\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\big{)}^{2}\right]^{\frac{1}{2}}E_{P|S_{\mathcal{I}_{k}^{c}}}\left[\left(\mathcal{E}_{m}^{(k)}(X_{i})^{\top}W_{i}^{*}\varepsilon_{i}\right)^{2}\right]^{\frac{1}{2}}\\\
=&N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\xi_{i}^{\top}W_{i}^{*}\Sigma_{i}W_{i}^{*}\xi_{i}\right]^{\frac{1}{2}}E_{P|S_{\mathcal{I}_{k}^{c}}}\left[\mathcal{E}_{m}^{(k)}(X_{i})^{\top}W_{i}^{*}\Sigma_{i}W_{i}^{*}\mathcal{E}_{m}^{(k)}(X_{i})\right]^{\frac{1}{2}}\\\
\leq&N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}n_{i}^{1+\alpha}\mathcal{R}_{m}^{\frac{1}{2}}\leq
n_{\max}^{\kappa+\alpha}\mathcal{R}_{m}^{\frac{1}{2}}=o_{\mathcal{P}}(1),\end{split}$

Similarly, for the term IXd,

$\begin{split}&\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\bigg{[}\Big{|}N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\big{(}\xi_{i}^{\top}W^{*}_{i}\varepsilon_{i}\big{)}\Big{(}\Big{(}\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\Big{)}^{\top}\big{(}\hat{W}_{i}-W^{*}_{i}\big{)}\varepsilon_{i}\Big{)}\Big{|}\bigg{]}\\\
\leq&N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\bigg{[}\Big{|}\big{(}\xi_{i}^{\top}W^{*}_{i}\varepsilon_{i}\big{)}\Big{(}\Big{(}\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\Big{)}^{\top}\big{(}\hat{W}_{i}-W^{*}_{i}\big{)}\varepsilon_{i}\Big{)}\Big{|}\bigg{]}\\\
\leq&N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\Big{[}\big{(}\xi_{i}^{\top}W^{*}_{i}\varepsilon_{i}\big{)}^{2}\Big{]}^{\frac{1}{2}}E_{P|S_{\mathcal{I}_{k}^{c}}}\bigg{[}\Big{(}\Big{(}\xi_{i}+\mathcal{E}_{m}^{(k)}(X_{i})\Big{)}^{\top}\big{(}\hat{W}_{i}-W^{*}_{i}\big{)}\varepsilon_{i}\Big{)}^{2}\bigg{]}^{\frac{1}{2}}\\\
\lesssim&n_{\max}^{\kappa+\frac{\alpha}{2}+\frac{\mu_{\Sigma}}{2}}\Big{(}n_{\max}^{\frac{1+\tau}{2}}\mathcal{R}_{\sigma}^{\frac{1}{2}}+n_{\max}\mathcal{R}_{\rho}^{\frac{1}{2}}\Big{)}=o_{\mathcal{P}}(1),\end{split}$

by Assumption A4. Hence, by Lemma 15, the term IXd is $o_{\mathcal{P}}(1)$.

##### Term X:

Note that

$\displaystyle\mathbb{E}_{P}\bigg{[}\Big{|}2N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\phi^{2}(S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)})\big{(}\hat{\beta}-\beta\big{)}^{2}\Big{|}\bigg{]}$
$\displaystyle\leq$ $\displaystyle
2N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P}\left[\phi^{2}(S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)})\big{(}\hat{\beta}-\beta\big{)}^{2}\right]$
$\displaystyle\lesssim$ $\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\Big{(}\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\left\lVert\xi_{i}\right\rVert_{2}^{4}\big{(}\hat{\beta}-\beta\big{)}^{2}\right]+\mathbb{E}_{P|S_{\mathcal{I}_{k}^{c}}}\left[\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{2}^{4}\big{(}\hat{\beta}-\beta\big{)}^{2}\right]\Big{)}$
$\displaystyle\leq$ $\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\Big{(}\left\lVert\xi_{i}\right\rVert_{P,4+\delta}^{4}\big{\|}\hat{\beta}-\beta\big{\|}_{P,\frac{2(4+\delta)}{\delta}}^{2}+\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{P,4+\delta}^{4}\big{\|}\hat{\beta}-\beta\big{\|}_{P,\frac{2(4+\delta)}{\delta}}^{2}\Big{)}$
$\displaystyle\leq$ $\displaystyle
N^{-2}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}n_{i}^{2}\bigg{(}\Big{\|}n_{i}^{-\frac{1}{2}}\xi_{i}\Big{\|}_{P,4+\delta}^{4}\left\lVert
N^{\frac{1}{2}}\big{(}\hat{\beta}-\beta\big{)}\right\rVert_{P,\frac{2(4+\delta)}{\delta}}^{2}$
$\displaystyle\hskip
120.00018pt+\Big{\|}n_{i}^{-\frac{1}{2}}\mathcal{E}_{m}^{(k)}(X_{i})\Big{\|}_{P,4+\delta}^{4}\left\lVert
N^{\frac{1}{2}}\big{(}\hat{\beta}-\beta\big{)}\right\rVert_{P,\frac{2(4+\delta)}{\delta}}^{2}\bigg{)}$
$\displaystyle\lesssim$ $\displaystyle
N^{-2}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}n_{i}^{2}\leq
N^{-1}n_{\max}^{1+\kappa},$

by using the fact that any finite moment of a normal distribution is finite.
Thus

$\sup_{P\in\mathcal{P}_{I}}\mathbb{E}_{P}\bigg{[}\Big{|}2N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\phi^{2}(S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)})\big{(}\hat{\beta}-\beta\big{)}^{2}\Big{|}\bigg{]}\lesssim
N^{-1}n_{\max}^{1+\kappa}=o(1),$

by Assumption A4. By Lemmas 12 and 15 we have that term X is
$o_{\mathcal{P}}(1)$.

##### Term XI:

Note that

$\displaystyle\mathbb{E}_{P}\bigg{[}\Big{|}N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\varphi(S_{i};\beta,\eta_{0},\nu^{*})\phi(S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)})\big{(}\hat{\beta}-\beta\big{)}\Big{|}\bigg{]}$
$\displaystyle\lesssim$ $\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\mathbb{E}_{P}\left[\left\lVert\xi_{i}\right\rVert_{2}^{2}\left(\left\lVert\xi_{i}\right\rVert_{2}+\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{2}\right)\left\lVert\varepsilon_{i}\right\rVert_{2}\big{|}\hat{\beta}-\beta\big{|}\right]$
$\displaystyle\leq$ $\displaystyle
N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\Big{(}\left\lVert\xi_{i}\right\rVert_{P,4+\delta}^{3}\left\lVert\varepsilon_{i}\right\rVert_{P,4+\delta}\big{\|}\hat{\beta}-\beta\big{\|}_{P,\frac{4+\delta}{\delta}}$
$\displaystyle\hskip
60.00009pt+\left\lVert\xi_{i}\right\rVert_{P,4+\delta}^{2}\left\lVert\varepsilon_{i}\right\rVert_{P,4+\delta}\big{\|}\mathcal{E}_{m}^{(k)}(X_{i})\big{\|}_{P,4+\delta}\big{\|}\hat{\beta}-\beta\big{\|}_{P,\frac{4+\delta}{\delta}}\Big{)}$
$\displaystyle\lesssim$ $\displaystyle
N^{-\frac{3}{2}}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}n_{i}^{2}\big{\|}N^{\frac{1}{2}}\big{(}\hat{\beta}-\beta\big{)}\big{\|}_{P,\frac{4+\delta}{\delta}}$
$\displaystyle\lesssim$ $\displaystyle
N^{-\frac{3}{2}}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}n_{i}^{2}\leq
N^{-\frac{1}{2}}n_{\max}^{1+\kappa},$

by Hölder’s inequality (twice), and thus

$\sup_{P\in\mathcal{P}_{I}}\mathbb{E}_{P}\bigg{[}\Big{|}N^{-1}n_{\max}^{\kappa}\sum_{i\in\mathcal{I}_{k}}\varphi(S_{i};\beta,\eta_{0},\nu^{*})\phi(S_{i};\hat{\eta}^{(k)},\hat{\nu}^{(k)})\big{(}\hat{\beta}-\beta\big{)}\Big{|}\bigg{]}\lesssim
N^{-\frac{1}{2}}n_{\max}^{1+\kappa}=o(1)$

and Assumption A4. It then follows by Lemmas 12 and 15 that the term XI is
$o_{\mathcal{P}}(1)$. As all terms are $o_{\mathcal{P}}(1)$, applying Lemma 12
completes the proof.

∎

######  Lemma 11.

Under the setup of Theorem 4,

$M_{4}:=N^{-1}n_{\max}^{\kappa}\sum_{i=1}^{I}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})-\mathbb{E}_{P}\left[\varphi^{2}(S_{i};\beta,\eta_{0},\nu^{*})\right]\right]=o_{\mathcal{P}}(1).$

###### Proof.

We will consider the two cases $\delta\in(0,4)$ and $\delta\geq 4$ separately.
If $\delta\in(0,4)$ then, by the von Bahr–Esseen inequality, for all
$P\in\mathcal{P}_{I}$

$\displaystyle\mathbb{E}_{P}\left[\left|M_{4}\right|^{1+\frac{\delta}{4}}\right]\leq$
$\displaystyle
N^{-\left(1+\frac{\delta}{4}\right)}n_{\max}^{\kappa\left(1+\frac{\delta}{4}\right)}\mathbb{E}_{P}\left[\bigg{|}\sum_{i=1}^{I}\varphi^{2}\left(S_{i};\beta,\eta_{0},\nu^{*}\right)-\mathbb{E}_{P}\left[\varphi^{2}\left(S_{i};\beta,\eta_{0},\nu^{*}\right)\right]\bigg{|}^{1+\frac{\delta}{4}}\right]$
$\displaystyle\leq$ $\displaystyle
2N^{-\left(1+\frac{\delta}{4}\right)}n_{\max}^{\kappa\left(1+\frac{\delta}{4}\right)}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|\varphi^{2}\left(S_{i};\beta,\eta_{0},\nu^{*}\right)-\mathbb{E}_{P}\left[\varphi^{2}\left(S_{i};\beta,\eta_{0},\nu^{*}\right)\right]\right|^{1+\frac{\delta}{4}}\right]$
$\displaystyle\leq$ $\displaystyle
2N^{-\left(1+\frac{\delta}{4}\right)}n_{\max}^{\kappa\left(1+\frac{\delta}{4}\right)}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|\varphi\left(S_{i};\beta,\eta_{0},\nu^{*}\right)\right|^{2+\frac{\delta}{2}}\right]$
$\displaystyle\leq$ $\displaystyle
2N^{-\left(1+\frac{\delta}{4}\right)}n_{\max}^{\kappa\left(1+\frac{\delta}{4}\right)}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left\lVert
W_{i}^{*}\right\rVert_{\mathrm{op}}^{2+\frac{\delta}{2}}\left\lVert\xi_{i}\right\rVert_{2}^{2+\frac{\delta}{2}}\left\lVert\varepsilon_{i}\right\rVert_{2}^{2+\frac{\delta}{2}}\right]$
$\displaystyle\lesssim$ $\displaystyle
N^{-\left(1+\frac{\delta}{4}\right)}n_{\max}^{\kappa\left(1+\frac{\delta}{4}\right)}\sum_{i=1}^{I}\left\lVert\xi_{i}\right\rVert_{P,4+\delta}^{2+\frac{\delta}{2}}\left\lVert\varepsilon_{i}\right\rVert_{P,4+\delta}^{2+\frac{\delta}{2}}$
$\displaystyle\lesssim$ $\displaystyle
N^{-\left(1+\frac{\delta}{4}\right)}n_{\max}^{\kappa\left(1+\frac{\delta}{4}\right)}\sum_{i=1}^{I}n_{i}^{2+\frac{\delta}{2}}\leq
N^{-\frac{\delta}{4}}n_{\max}^{(1+\kappa)+(2+\kappa)\frac{\delta}{4}},$

and so

$\sup_{P\in\mathcal{P}_{I}}\mathbb{E}_{P}\left[\left|M_{4}\right|^{1+\frac{\delta}{4}}\right]\lesssim
N^{-\frac{\delta}{4}}n_{\max}^{(1+\kappa)+(2+\kappa)\frac{\delta}{4}}=o(1),$

by Assumption A4.

If $\delta\geq 4$ then, again for all $P\in\mathcal{P}_{I}$,

$\displaystyle\mathbb{E}_{P}\left[M_{4}^{2}\right]\leq$ $\displaystyle
N^{-2}n_{\max}^{2\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\varphi^{4}\left(S_{i};\beta,\eta_{0},\nu^{*}\right)\right]$
$\displaystyle\leq$ $\displaystyle
N^{-2}n_{\max}^{2\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|\xi_{i}^{\top}W_{i}^{*}\varepsilon_{i}\right|^{4}\right]$
$\displaystyle\leq$ $\displaystyle
N^{-2}n_{\max}^{2\kappa}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left\lVert
W_{i}^{*}\right\rVert_{\mathrm{op}}^{4}\left\lVert\xi_{i}\right\rVert_{2}^{4}\left\lVert\varepsilon_{i}\right\rVert_{2}^{4}\right]$
$\displaystyle\lesssim$ $\displaystyle
N^{-2}n_{\max}^{2\kappa}\sum_{i=1}^{I}\left\lVert\xi_{i}\right\rVert_{P,8}^{4}\left\lVert\varepsilon_{i}\right\rVert_{P,8}^{4}$
$\displaystyle\lesssim$ $\displaystyle
N^{-2}n_{\max}^{2\kappa}\sum_{i=1}^{I}n_{i}^{4}\leq
N^{-1}n_{\max}^{3+2\kappa},$

and so

$\sup_{P\in\mathcal{P}_{I}}\mathbb{E}_{P}\left[M_{4}^{2}\right]\lesssim
N^{-1}n_{\max}^{3+2\kappa}=o(1),$

by Assumptions (A1.1) and (A3.3).

Thus, in each case we have by applying Markov’s inequality that
$M_{4}=o_{\mathcal{P}}(1)$. ∎

###  D.3 Auxiliary uniform convergence results

For the uniform convergence results of Theorem 4 we require the following
lemmata. Given a sequence of families of probability distributions
$\left(\mathcal{P}_{I}\right)_{I\in\mathbb{N}}$ we use the shorthand
$\left(A_{I}\right)_{I\in\mathbb{N}}$ to denote a sequence of random variables
$\left(A_{P,I}\right)_{P\in\mathcal{P}_{I},I\in\mathbb{N}}$.

######  Lemma 12.

Let $\left(A_{I}\right)_{I\in\mathbb{N}}$ and
$\left(B_{I}\right)_{I\in\mathbb{N}}$ be sequences of real-valued random
variables. If $A_{I}=o_{\mathcal{P}}(1)$ and $B_{I}=o_{\mathcal{P}}(1)$ then
$A_{I}+B_{I}=o_{\mathcal{P}}(1)$.

######  Lemma 13.

Let $\left(A_{I}\right)_{I\in\mathbb{N}}$ and
$\left(B_{I}\right)_{I\in\mathbb{N}}$ be sequences of real-valued random
variables. Suppose that $A_{I}=o_{\mathcal{P}}(1)$ and $B_{I}\leq A_{I}$ for
all $I\in\mathbb{N}$ uniformly over all $P\in\mathcal{P}_{I}$. Then
$B_{I}=o_{\mathcal{P}}(1)$.

######  Lemma 14.

Let $\left(A_{I}\right)_{I\in\mathbb{N}}$ and
$\left(B_{I}\right)_{I\in\mathbb{N}}$ be sequences of real-valued random
variables. If $A_{I}=o_{\mathcal{P}}(1)$ and $B_{I}=O_{\mathcal{P}}(1)$ then
$A_{I}B_{I}=o_{\mathcal{P}}(1)$.

######  Lemma 15.

Let $\left(A_{I}\right)_{I\in\mathbb{N}}$ and
$\left(B_{I}\right)_{I\in\mathbb{N}}$ be a sequences of real-valued random
variables. If there exists some $\alpha\geq 1$ such that
$\mathbb{E}_{P}(\left|A_{I}\right|^{\alpha}\,|\,B_{I})=o_{\mathcal{P}}(1)$,
then $A_{I}=o_{\mathcal{P}}(1)$.

For proofs of these four lemmas see, for example, lundborg2022projected

######  Lemma 16.

Let $(A_{I}^{(1)})_{I\in\mathbb{N}},\ldots,(A_{I}^{(K)})_{I\in\mathbb{N}}$ be
sequences of real-valued random variables satisfying
$A_{I}^{(k)}=o_{\mathcal{P}}(1)$ for all $k\in[K]$, where $K\in\mathbb{N}$ is
finite. Then $\max_{k\in[K]}A_{I}^{(k)}=o_{\mathcal{P}}(1)$.

###### Proof.

For any $\epsilon>0$

$\sup_{P\in\mathcal{P}}\mathbb{P}_{P}\left(\left|\max_{k\in[K]}A_{I}^{(k)}\right|>\epsilon\right)\leq\sup_{P\in\mathcal{P}}\mathbb{P}_{P}\left(\max_{k\in[K]}\big{|}A_{I}^{(k)}\big{|}>\epsilon\right)\leq\sum_{k=1}^{K}\sup_{P\in\mathcal{P}}\mathbb{P}_{P}\left(\big{|}A_{I}^{(k)}\big{|}>\epsilon\right)=o(1),$

completing the proof. ∎

######  Lemma 17.

Let $\left(A_{I,i}\right)_{I\in\mathbb{N},i\in[I]}$ be a triangular array of
real-valued random variables satisfying:

  * •

$A_{I,1},\ldots,A_{I,I}$ are independent;

  * •

$\mathbb{E}_{P}\left[A_{I,i}\right]=0\hskip 11.38109pt(\forall
I\in\mathbb{N},i\in[I])$;

  * •

There exists some $\Delta>0$ such that

$\lim_{I\to\infty}\sup_{P\in\mathcal{P}_{I}}\frac{\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|A_{I,i}\right|^{2+\Delta}\right]}{\left(\sum_{i=1}^{I}\mathbb{E}_{P}\left[A_{I,i}^{2}\right]\right)^{1+\frac{\Delta}{2}}}=0.$

Then
$S_{I}:=\left(\sum_{i=1}^{I}\mathbb{E}_{P}\left[A_{I,i}^{2}\right]\right)^{-\frac{1}{2}}\left(\sum_{i=1}^{I}A_{I,i}\right)$
converges uniformly to $N(0,1)$, i.e.

$\lim_{I\to\infty}\sup_{P\in\mathcal{P}_{I}}\sup_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(S_{I}\leq
t\right)-\Phi(t)\right|=0.$

###### Proof.

For each $I\in\mathbb{N}$, let $P\in\mathcal{P}_{I}$ satisfy

$\sup_{P\in\mathcal{P}_{I}}\sup_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(S_{I}\leq
t\right)-\Phi(t)\right|\leq\sup_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(S_{I}\leq
t\right)-\Phi(t)\right|+I^{-1}.$ (36)

Let
$T_{I,i}:=\left(\sum_{i=1}^{I}\mathbb{E}_{P}\left[A_{I,i}^{2}\right]\right)^{-\frac{1}{2}}\left(A_{I,i}\right)$
for each $i\in[I]$. Note that by construction $T_{I,1},\ldots,T_{I,I}$ are
independent and mean zero and $\sum_{i=1}^{I}\mathbb{E}_{P}(T_{I,i}^{2})=1$.
By the Lindeberg–Feller central limit theorem for triangular arrays
(Proposition 2.27, vandervaart) we have

$\lim_{I\to\infty}\sup_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(S_{I}\leq
t\right)-\Phi(t)\right|=0,$

provided the condition for the Lindeberg–Feller theorem holds. This condition
holds as, for all $\epsilon>0$,

$\displaystyle\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|T_{I,i}\right|^{2}\mathbbm{1}_{\left\\{\left|T_{I,i}\right|>\epsilon\right\\}}\right]$
$\displaystyle\leq\sum_{i=1}^{I}\left(\mathbb{E}_{P}\left[\left|T_{I,i}\right|^{2+\Delta}\right]\right)^{\frac{2}{2+\Delta}}\big{(}\mathbb{P}_{P}\left(\left|T_{I,i}\right|>\epsilon\right)\big{)}^{\frac{\Delta}{2+\Delta}}$
$\displaystyle\leq\epsilon^{-\Delta}\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|T_{I,i}\right|^{2+\Delta}\right]$
$\displaystyle=\epsilon^{-\Delta}\cdot\frac{\sum_{i=1}^{I}\mathbb{E}_{P}\left[\left|A_{I,i}\right|^{2+\Delta}\right]}{\left|\sum_{i=1}^{I}\mathbb{E}_{P}[A_{I,i}^{2}]\right|^{1+\frac{\Delta}{2}}}$

by Hölder’s inequality followed by Markov’s inequality. Taking limits of (36)
therefore completes the proof. ∎

######  Lemma 18 (shahpeters, Lemma 20).

Let $\left(A_{I}\right)_{I\in\mathbb{N}}$ and
$\left(B_{I}\right)_{I\in\mathbb{N}}$ be sequences of real-valued random
variables. Suppose

$\lim_{I\to\infty}\sup_{P\in\mathcal{P}_{I}}\sup_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(A_{I}\leq
t\right)-\Phi(t)\right|=0.$

Then we have:

  1. (a)

$\text{If }B_{I}=o_{\mathcal{P}}(1)\text{ then
}\lim_{I\to\infty}\sup_{P\in\mathcal{P}_{I}}\sup_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(A_{I}+B_{I}\leq
t\right)-\Phi(t)\right|=0;$

  2. (b)

$\text{If }B_{I}=1+o_{\mathcal{P}}(1)\text{ then
}\lim_{I\to\infty}\sup_{P\in\mathcal{P}_{I}}\sup_{t\in\mathbb{R}}\left|\mathbb{P}_{P}\left(A_{I}/B_{I}\leq
t\right)-\Phi(t)\right|=0.$

###  D.4 Proof of Corollary 6

###### Proof of Theorem 6.

We aim to show that Assumption (A3.1’) implies (A3.1) and (A3.2’) implies
(A3.2). Note that for all $k\in[K]$, $i\in\mathcal{I}_{k}$, $j\in[n_{i}]$,

$\mathbb{E}_{P}\left[\left(\hat{\sigma}^{(k)}(X_{ij})-\sigma^{*}(X_{ij})\right)^{2}\right]=\mathbb{E}_{P}\left[\frac{\left(\hat{s}^{(k)}(X_{ij})-s^{*}(X_{ij})\right)^{2}}{\hat{s}^{(k)}(X_{ij})^{2}s^{*}(X_{ij})^{2}}\right]\lesssim\mathbb{E}_{P}\left[\left(\hat{s}^{(k)}(X_{ij})-s^{*}(X_{ij})\right)^{2}\right],$

and so $\mathcal{R}_{\sigma}\lesssim\mathcal{R}_{s^{*}}$, hence Assumption
(A3.1’) implies (A3.1).

To show that Assumption (A3.2’) implies (A3.2), we consider each case
separately:

##### Equicorrelation Case:

Note that for all $k\in[K]$,

$\left|\hat{\rho}^{(k)}-\rho^{*}\right|=\frac{\left|\hat{\theta}^{(k)}-\theta^{*}\right|}{|1+\theta^{*}|\cdot|1+\hat{\theta}^{(k)}|}\lesssim\left|\hat{\theta}^{(k)}-\theta^{*}\right|,$

and therefore $\mathcal{R}_{\rho}\lesssim\mathcal{R}_{\theta^{*}}$.

##### Hierarchical / Nested Case:

Note that for all $k\in[K]$,

$\displaystyle\left|\hat{\rho}^{(k)}_{1}-\rho^{*}_{1}\right|$
$\displaystyle=\frac{\left|\hat{\theta}^{(k)}_{1}+\hat{\theta}^{(k)}_{2}-\theta^{*}_{1}-\theta^{*}_{2}\right|}{\left|1+\hat{\theta}^{(k)}_{1}+\hat{\theta}^{(k)}_{2}\right|\cdot\left|1+\theta^{*}_{1}+\theta^{*}_{2}\right|}\lesssim\left|\hat{\theta}^{(k)}_{1}-\theta^{*}_{1}\right|+\left|\hat{\theta}^{(k)}_{2}-\theta^{*}_{2}\right|\leq\sqrt{2}\left\lVert\hat{\theta}^{(k)}-\theta^{*}\right\rVert_{2},$
$\displaystyle\left|\hat{\rho}^{(k)}_{2}-\rho^{*}_{2}\right|$
$\displaystyle=\frac{\left|\left(1+\theta^{*}_{1}\right)\left(\hat{\theta}_{2}-\theta^{*}_{2}\right)-\theta^{*}_{2}\left(\hat{\theta}^{(k)}_{1}-\theta^{*}_{1}\right)\right|}{\left|1+\hat{\theta}^{(k)}_{1}+\hat{\theta}^{(k)}_{2}\right|\cdot\left|1+\theta^{*}_{1}+\theta^{*}_{2}\right|}\lesssim\sqrt{2}\left\lVert\hat{\theta}^{(k)}-\theta^{*}\right\rVert_{2},$

and therefore $\mathcal{R}_{\rho}\lesssim\mathcal{R}_{\theta^{*}}$.

##### Longitudinal $\text{AR}(1)$ Case:

Note that for all $k\in[K]$, $i\in\mathcal{I}_{k}$, $j\in[n_{i}]$,
$j^{\prime}\in[n_{i}]\backslash\left\\{j\right\\}$,

$\displaystyle\left|\hat{\rho}^{(k)}(X_{ij},X_{ij^{\prime}})-\rho^{*}(X_{ij},X_{ij^{\prime}})\right|=\left|\hat{\theta}^{(k)^{\left|j-j^{\prime}\right|}}-\theta^{*^{\left|j-j^{\prime}\right|}}\right|$
$\displaystyle\leq$
$\displaystyle\left|\hat{\theta}^{(k)}-\theta\right|\cdot\left|\sum_{l=1}^{\left|j-j^{\prime}\right|-1}\hat{\theta}^{(k)^{l}}\theta^{*^{\left|j-j^{\prime}\right|-1-l}}\right|\leq\left|\hat{\theta}^{(k)}-\theta^{*}\right|\cdot\left|j-j^{\prime}\right|\cdot\left(\hat{\theta}^{(k)}\vee\theta^{*}\right)^{\left|j-j^{\prime}\right|-1}$
$\displaystyle\leq$
$\displaystyle\left|\hat{\theta}^{(k)}-\theta^{*}\right|\cdot\left|j-j^{\prime}\right|\left(1-c\right)^{\left|j-j^{\prime}\right|-1}\leq\left(\log\left(\frac{1}{1-c}\right)\right)^{-1}(1-c)^{\frac{1}{\log\left(\frac{1}{1-c}\right)}-1}\left|\hat{\theta}^{(k)}-\theta^{*}\right|$
$\displaystyle\lesssim$
$\displaystyle\left|\hat{\theta}^{(k)}-\theta^{*}\right|,$

for some $c>0$, with the last inequality following by elementary calculus
(optimising the function $N\mapsto N(1-c)^{N-1}$). Therefore
$\mathcal{R}_{\rho}\lesssim\mathcal{R}_{\theta^{*}}$.

Hence, for each working correlation, Assumption (A3.2’) implies (A3.2).

Finally, we must show that for each working correlation
$\Lambda_{\max}(W_{i}^{*})\lesssim 1$, $\Lambda_{\min}(W_{i}^{*})\gtrsim
n^{-\gamma}$ for all $i\in[I]$. We consider each separately.

##### Equicorrelation Case:

Consider the decomposition of the matrix $W_{i}^{*}$ as the sum of two
matrices $W_{i}=W_{i}^{*(A)}+W_{i}^{*(B)}$

$\displaystyle\big{(}W_{i}^{*(A)}\big{)}_{jj^{\prime}}$
$\displaystyle=\delta_{jj^{\prime}}s^{*}(X_{ij})s^{*}(X_{ij^{\prime}})$
$\displaystyle\big{(}W_{i}^{*(B)}\big{)}_{jj^{\prime}}$
$\displaystyle=-\frac{\theta}{1+\theta
n_{i}}s^{*}(X_{ij})s^{*}(X_{ij^{\prime}}),$

for $j,j^{\prime}\in[n_{i}]$ Then, we have that

$\Lambda_{\max}(W_{i}^{*})\leq\max_{j\in[n_{i}]}s^{*}(X_{ij})^{2}+0\lesssim
1,$

and further that, by Weyl’s inequality,

$\Lambda_{\min}(W_{i}^{*})\geq\min_{j\in[n_{i}]}s^{*}(X_{ij})^{2}-\frac{\theta^{*}}{1+\theta^{*}n_{i}}\left\lVert
s^{*}(X_{i})\right\rVert_{2}^{2}\\\
\geq\left(1-\frac{\theta^{*}n_{i}}{1+\theta^{*}n_{i}}\right)\min_{j\in[n_{i}]}s^{*}(X_{ij})^{2}\gtrsim\frac{1}{1+\theta^{*}n_{i}}\geq\frac{1}{1+\theta^{*}}\cdot\frac{1}{n_{i}}\gtrsim
n_{i}^{-1}.$

##### Hierarchical / Nested Case:

First consider the decomposition of $W_{i}^{*}=W_{i}^{*(A)}+W_{i}^{*(B)}$
where

$\displaystyle\big{(}W_{i}^{*(A)}\big{)}_{jj^{\prime}}$
$\displaystyle=(1+\theta^{*}_{1}+\theta^{*}_{2})\delta_{jj^{\prime}}s^{*}(X_{ij})s^{*}(X_{ij^{\prime}}),$
$\displaystyle\big{(}W_{i}^{*(B)}\big{)}_{jj^{\prime}}$
$\displaystyle=-\mathcal{B}_{i,jj^{\prime}}s^{*}(X_{ij})s^{*}(X_{ij^{\prime}}),$

for $j,j^{\prime}\in[n_{i}]$ and $\mathcal{B}$ as defined in Section 3.2. Then

$\Lambda_{\max}(W_{i}^{*})\leq\max_{j\in[n_{i}]}s^{*}(X_{ij})^{2}+0\lesssim 1$

Further, by considering $W_{i}^{*^{-1}}$ parametrised by
$(\rho^{*}_{1},\rho^{*}_{2})=\left(\frac{\theta^{*}_{1}+\theta^{*}_{2}}{1+\theta^{*}_{1}+\theta^{*}_{2}},\frac{\theta^{*}_{2}}{1+\theta^{*}_{1}+\theta^{*}_{2}}\right)$,
we can similarly show that $\Lambda_{\max}(W_{i}^{*^{-1}})\lesssim n_{i}$, and
therefore $\Lambda_{\min}(W_{i}^{*})\gtrsim n_{i}^{-1}$, completing the proof.

##### Longitudinal $\text{AR}(1)$ Case:

By the Gershgorin circle theorem alongside the triangle inequality,

$\displaystyle\Lambda_{\max}(W_{i}^{*})\leq$
$\displaystyle(1+2\theta^{*})\max_{j\in[n_{i}]}s^{*}(X_{ij})^{2}\lesssim 1,$
$\displaystyle\Lambda_{\max}(W_{i}^{*^{-1}})\leq$
$\displaystyle\left(\min_{j\in[n_{i}]}s^{*}(X_{ij})^{2}\right)^{-1}+2\left(\min_{j\in[n_{i}]}s^{*}(X_{ij})^{2}\right)^{-1}\sum_{j=1}^{n_{i}-1}\theta^{j}$
$\displaystyle=$
$\displaystyle\left(1+\frac{2\theta^{*}(1-\theta^{*^{n_{i}-1}})}{1-\theta^{*}}\right)\left(\min_{j\in[n_{i}]}s^{*}(X_{ij})^{2}\right)^{-1}$
$\displaystyle\leq$
$\displaystyle\left(1+\frac{2}{1-\theta^{*}}\right)\left(\min_{j\in[n_{i}]}s^{*}(X_{ij})^{2}\right)^{-1}\lesssim
1,$

and so
$\Lambda_{\min}(W_{i}^{*})=\big{(}\Lambda_{\max}(W_{i}^{*^{-1}})\big{)}^{-1}\gtrsim
1$ as required.

∎

##  Appendix E Practical details of numerical results in Section 5

###  E.1 Details of weight estimators studied in 5.1

The ‘heteroscedastic ML’ estimator used in Section 5 follows Algorithm 1 but
with weights constructed as follows. Adopting the notation in Algorithm 1 with
weights as in (13), we construct

$\displaystyle(\hat{\phi}_{\mathrm{ML}},\hat{\rho}_{\mathrm{ML}},\hat{\beta}_{\mathrm{ML}})=\underset{(\phi,\rho,\beta)\in\mathbb{R}^{5}\times(-1,1)\times\mathbb{R}}{\textrm{argmin}}l_{\mathrm{ML}}(\phi,\rho,\beta),$
$\displaystyle
l_{\mathrm{ML}}(\phi,\rho,\beta):=\sum_{i\in\mathcal{I}_{k}^{c}}\Big{(}-\log\det
W(\phi,\rho)(X_{i})+\big{(}\tilde{R}^{Y}_{i}-\beta\tilde{R}^{D}_{i}\big{)}^{\top}W(\phi,\rho)(X_{i})\big{(}\tilde{R}^{Y}_{i}-\beta\tilde{R}^{D}_{i}\big{)}\Big{)},$
$\displaystyle\left\\{W(\phi,\rho)(X_{i})\right\\}^{-1}=D_{\sigma_{\phi}}(X_{i})C_{\theta_{\rho}}D_{\sigma_{\phi}}(X_{i}),\qquad\sigma_{\phi}(x)=\exp\big{(}\phi^{\top}(1,x,x^{2},x^{3},x^{4})\big{)},$

with working correlations
$(C_{\theta_{\rho}})_{jk}=\mathbbm{1}_{\left\\{j=k\right\\}}+\rho\cdot\mathbbm{1}_{\left\\{j\neq
k\right\\}}$ for examples 5.1.1 and 5.1.4 (equicorrelated) and
$(C_{\theta_{\rho}})_{jk}=\rho^{|j-k|}$ for examples 5.1.2 and 5.1.3
($\text{AR}(1)$). These are estimated using the nlme package [nlme-code].
Weights are then constructed as

$\big{\\{}\hat{W}_{\mathrm{ML}}^{(k)}(\cdot)\big{\\}}^{-1}=D_{\hat{\sigma}_{\mathrm{ML}}}(\cdot)C_{\theta_{\hat{\rho}_{\mathrm{ML}}}}D_{\hat{\sigma}_{\mathrm{ML}}}(\cdot),\qquad\hat{\sigma}_{\mathrm{ML}}(x):=\exp\big{(}\hat{\phi}_{\mathrm{ML}}^{\top}(1,x,x^{2},x^{3},x^{4})\big{)}$

and used in Algorithm 1 to construct the final $\hat{\beta}$. The
‘homoscedastic ML’ estimator is constructed as above fixing
$\hat{\phi}_{\mathrm{ML}}={\bf 0}$.

The ‘heteroscedastic GEE’ estimator also follow the setup of Algorithm 1 with
weights constructed as follows. First we generate a generalised additive cubic
spline estimator for the variance and estimator for the correlation parameter
as

$\displaystyle\hat{\sigma}_{\mathrm{GEE}}$
$\displaystyle:=\underset{\sigma\in\Pi}{\textrm{argmin}}\sum_{i\in\mathcal{I}_{k}^{c}}\sum_{j=1}^{n_{i}}\big{(}\tilde{\varepsilon}_{ij}^{2}-\sigma^{2}(X_{ij})\big{)}^{2}+\lambda
J(\sigma^{2}),$ $\displaystyle
J(f)=\int\left(f^{\prime\prime}(x)\right)^{2}dx,$
$\displaystyle\hat{\rho}_{\mathrm{GEE}}$
$\displaystyle:=\underset{\rho\in(-1,1)}{\textrm{argmin}}\sum_{i\in\mathcal{I}_{k}^{c}}\sum_{\begin{subarray}{c}j,k=1\\\
j\neq
k\end{subarray}}^{n_{i}}\Big{(}\tilde{\varepsilon}_{ij}\tilde{\varepsilon}_{ik}-\hat{\sigma}_{\mathrm{GEE}}(X_{ij})\hat{\sigma}_{\mathrm{GEE}}(X_{ik})(C_{\theta_{\rho}})_{jk}\Big{)}^{2},$

where $\Pi$ is the set of $\mathbb{R}_{>0}$ valued functions that have
absolutely continuous first derivative, $\lambda$ is a tuning parameter that
minimises the generalised cross-validation (GCV) score, and
$C_{\theta_{\rho}}$ takes the equicorrelated or $\text{AR}(1)$ structure for
each example as outlined above. In all the simulations in Section 5.1 we fit
$\hat{\sigma}_{\mathrm{GEE}}$ using the gam package [gam-code], and
$\hat{\rho}_{\mathrm{GEE}}$ using the geepack package [geepack]. Weights are
then constructed as

$\big{\\{}\hat{W}_{\mathrm{GEE}}^{(k)}(\cdot)\big{\\}}^{-1}=D_{\hat{\sigma}_{\mathrm{GEE}}}(\cdot)C_{\theta_{\hat{\rho}_{\mathrm{GEE}}}}D_{\hat{\sigma}_{\mathrm{GEE}}}(\cdot),$

and used in Algorithm 1 to construct $\hat{\beta}$. The ‘homoscedastic GEE’
estimator is estimated in the same way except we set
$\hat{\sigma}_{\mathrm{GEE}}\equiv 1$.

###  E.2 Sandwich boosting with variable step size

In Section 3.2 we outlined the sandwich boosting algorithm 2 in terms of a
fixed step size $\lambda^{(s)}$, with an optimal number of boosting iterations
$m_{\text{stop}}$ selected by cross-validation. In this setup $\lambda^{(s)}$
acts as a hyperparameter; if it is set to be too large one can be susceptible
to under-fitting (being unable to attain a local minimum), and if set too
small it can take an unreasonably large $m_{\text{stop}}$ to achieve
convergence. An adaptation of Algorithm 2 that may be helpful in practice
would be to replace the $\lambda^{(s)}$ constant step size with a variable
step size

$\displaystyle\lambda^{(s)}_{m}\approx\underset{\lambda\in\Lambda}{\textrm{argmin}}\,\hat{L}_{\mathrm{SL}}(\hat{s}_{m}-\lambda\hat{u}_{m},\hat{\theta}_{m}),$

over some closed interval $\Lambda$ of positive step sizes, attained by
minimising the second order expansion of
$\lambda\mapsto\hat{L}_{\mathrm{SL}}(\hat{s}_{m}-\lambda\hat{u}_{m},\hat{\theta}_{m})$.
The $m$th iterative update of the $s$-function is then taken as

$\displaystyle\hat{s}_{m+1}=(\hat{s}_{m}-\mu^{(s)}\lambda^{(s)}_{m}\hat{u}_{m})\vee\epsilon,$

for some constant shrinkage parameter $\mu^{(s)}\in(0,1)$, whose purpose is to
avoid overfitting, and small constant $\epsilon>0$ which ensures that
$\hat{s}_{m+1}$ is always strictly positive everywhere; in all our
experiments, we set $\epsilon=0.1$. With this setup boosting, can then be
performed until a convergence criterion is achieved. This can eliminate the
need to determine an approximately optimal $(\lambda^{(s)},m_{\text{stop}})$,
as well as achieve faster convergence of sandwich boosting. We implement this
variable step size scheme in some of the numerical examples studied in Section
5.

###  E.3 Details of numerical results in Section 5

Sandwich boosting for the numerical results is setup as follows.

Increasing Model Complexity (Section 5.1.1):  The variable step size
adaptation of sandwich boosting is used, with $\Lambda=[0.001,10]$ and
$\lambda^{(\theta)}=0.1$.

Increasing Covariance Misspecification (Section 5.1.2):  A constant step size
$\lambda^{(s)}=10$ is used, with $\lambda^{(\theta)}=0.1$ and
$m_{\text{stop}}=200$ selected by cross-validation.

Correlation Misspecification (Section 5.1.3):  As this simulated example
involves optimisation of the sandwich loss over a parameteric weight class in
terms of a single $\text{AR}(1)$ parameter, there is no need to select a
$(\lambda^{(s)},m_{\text{stop}})$.

Conditional Variance Misspecification (Section 5.1.4):  A constant step size
$\lambda^{(s)}=100$ is used, with $\lambda^{(\theta)}=0.1$ and
$m_{\text{stop}}=500$ selected by cross-validation.

Real world data scenarios (Section 5.2):  We carry out sandwich boosting for
both with the variable step size adaptation within the interval
$\Lambda=[0.01,1]$, with $s$-shrinkage parameter $\mu^{(s)}=0.1$ and
$\theta$-step size $\lambda^{(\theta)}=0.1$.

When analysing the real world data discussed in Section 5.2, we alleviate the
dependence of the cross-fitting method on the randomness of the sample splits
by repeating the procedure of Algorithm 1 $\mathcal{S}=50$ times for different
random sample splits. Suppose for the repeat indexed by $s\in[\mathcal{S}]$ we
have $\beta$-estimator $\hat{\beta}_{s}$ and asymptotic variance estimator
$\hat{V}_{s}$. Then we aggregate the results of these repeats to produce a
final estimate of $\beta$ and associated estimate of its variance via

$\hat{\beta}:=\underset{s\in[\mathcal{S}]}{\text{median}}\,\hat{\beta}_{s},\qquad\hat{V}:=\underset{s\in[\mathcal{S}]}{\text{median}}\,\big{(}\hat{V}_{s}+(\hat{\beta}-\hat{\beta}_{s})^{2}\big{)};$

see for example chern, emmenegger.

###  E.4 Choice of parameters for cross-fitting

As outlined in Section 5, for the simulations in Section 5.1 we perform cross-
fitting with $K=2$ folds and for $\mathcal{S}=1$ random sample splits. The
choice of $(K,\mathcal{S})$ does not affect the asymptotic mean squared error
of the resulting estimator (see Theorem 4), and thus for sufficiently large
sample size the choice of cross-fitting parameters does not affect the
resulting mean squared error. Figure 6 (left panel) shows a simulation in
Section 5.1 (specifically Example 5.1.4 with group size $n_{i}\equiv 8$) for
cross-fitting parameters $K\in\\{2,5,10\\}$. We see the choice of $K$ does not
have a significant effect on the mean squared error of $\hat{\beta}$ at the
sample sizes considered in Section 5.1, therefore justifying our choice of
$(K,\mathcal{S})=(2,1)$ (although any other choices would be valid, at a
further computational cost). We do however note that, while such a simple,
computationally lean choice is justified for the simulations in Section 5.1,
this does not necessarily hold for smaller sample sizes; see for example
Figure 6 (right). In such settings a larger choice of either (or indeed both)
$K$ and $\mathcal{S}$ can improve the mean squared error, by reducing the
variance in the nuisance function estimators, in addition to the variance
induced by the sample split(s) used for cross-fitting. Therefore, when
analysing the real data scenarios of Section 5.2 we choose a larger choice of
$(K,\mathcal{S})=(5,50)$ (see Section E.3 for details).

Figure 6: Mean squared error of the sandwich boosted $\beta$-estimators for
different cross-fitting parameter choices (the number of folds used for cross-
fitting, $K\in\\{2,5,10\\}$, and the number of repeats over different sample
splits, $\mathcal{S}\in\\{1,5\\}$). We present these for two simulations:
(Left) Example 5.1.4, with $I=2^{12}$ groups each of size $n_{i}\equiv 8$; and
(Right) The same data generating mechanism as in Example 5.1.4 but with
$I=500$ groups each of size $n_{i}\equiv 4$. For all settings nominal $95\%$
confidence intervals for the mean squared errors (over 500 simulations) are
plotted.

##  Appendix F Additional numerical results

Tables 5–8 present the coverage probabilities of nominal 95% confidence
intervals based on the construction in Algorithm 1 for the simulation settings
in Section 5.1.

$\lambda$ |  | Sandwich
---
Boosting
Het ML | Hom ML | Het GEE | Hom GEE | Unweighted | Oracle
0.1 | 0.952 | 0.954 | 0.950 | 0.950 | 0.950 | 0.946 | 0.950
0.3 | 0.956 | 0.950 | 0.956 | 0.950 | 0.956 | 0.946 | 0.950
0.5 | 0.944 | 0.952 | 0.946 | 0.952 | 0.946 | 0.952 | 0.950
0.7 | 0.950 | 0.950 | 0.948 | 0.948 | 0.948 | 0.956 | 0.948
0.9 | 0.950 | 0.958 | 0.948 | 0.950 | 0.948 | 0.954 | 0.948
1.1 | 0.956 | 0.952 | 0.950 | 0.958 | 0.950 | 0.956 | 0.958
1.3 | 0.952 | 0.946 | 0.952 | 0.962 | 0.952 | 0.950 | 0.954
1.5 | 0.958 | 0.942 | 0.948 | 0.958 | 0.948 | 0.946 | 0.962
1.7 | 0.962 | 0.946 | 0.946 | 0.970 | 0.946 | 0.948 | 0.962
1.9 | 0.958 | 0.942 | 0.944 | 0.970 | 0.944 | 0.944 | 0.966
2.1 | 0.962 | 0.938 | 0.942 | 0.954 | 0.942 | 0.948 | 0.954
2.3 | 0.954 | 0.936 | 0.946 | 0.964 | 0.946 | 0.950 | 0.956
2.5 | 0.946 | 0.942 | 0.952 | 0.948 | 0.952 | 0.946 | 0.952
Table 5: Coverage of nominal $95\%$ confidence intervals $\hat{C}(0.05)$ for $\beta$ in Example 5.1.1. $\eta$ |  | Sandwich
---
Boosting
Het ML | Hom ML | Het GEE | Hom GEE | Unweighted
10 | 0.982 | 0.974 | 0.974 | 0.974 | 0.974 | 0.958
20 | 0.976 | 0.978 | 0.984 | 0.952 | 0.984 | 0.952
30 | 0.982 | 0.962 | 0.970 | 0.948 | 0.972 | 0.946
40 | 0.984 | 0.960 | 0.976 | 0.930 | 0.974 | 0.944
50 | 0.972 | 0.946 | 0.968 | 0.910 | 0.966 | 0.940
60 | 0.972 | 0.942 | 0.958 | 0.910 | 0.960 | 0.932
70 | 0.972 | 0.932 | 0.958 | 0.892 | 0.958 | 0.934
80 | 0.970 | 0.934 | 0.962 | 0.906 | 0.960 | 0.952
90 | 0.970 | 0.908 | 0.964 | 0.884 | 0.964 | 0.956
100 | 0.968 | 0.914 | 0.960 | 0.868 | 0.958 | 0.932
Table 6: Coverage of nominal $95\%$ confidence intervals $\hat{C}(0.05)$ for $\beta$ in Example 5.1.2. | Group
---
Size
Hom SL | Hom ML | Hom GEE | Unweighted
$2^{1}$ | 0.948 | 0.948 | 0.948 | 0.946
$2^{2}$ | 0.946 | 0.946 | 0.948 | 0.944
$2^{3}$ | 0.950 | 0.946 | 0.936 | 0.942
$2^{4}$ | 0.942 | 0.960 | 0.934 | 0.958
$2^{5}$ | 0.952 | 0.956 | 0.936 | 0.948
$2^{6}$ | 0.950 | 0.946 | 0.936 | 0.950
$2^{7}$ | 0.944 | 0.942 | 0.932 | 0.946
$2^{8}$ | 0.942 | 0.944 | 0.938 | 0.938
Table 7: Coverage of nominal $95\%$ confidence intervals $\hat{C}(0.05)$ for $\beta$ in Example 5.1.3. | Group
---
Size
| Sandwich
---
Boosting
Het ML | Hom ML | Het GEE | Hom GEE | Unweighted
$2^{1}$ | 0.938 | 0.936 | 0.932 | 0.948 | 0.932 | 0.926
$2^{2}$ | 0.938 | 0.938 | 0.940 | 0.954 | 0.940 | 0.922
$2^{3}$ | 0.938 | 0.936 | 0.938 | 0.956 | 0.938 | 0.926
$2^{4}$ | 0.938 | 0.940 | 0.944 | 0.954 | 0.944 | 0.924
$2^{5}$ | 0.938 | 0.940 | 0.936 | 0.966 | 0.936 | 0.934
$2^{6}$ | 0.942 | 0.942 | 0.948 | 0.968 | 0.948 | 0.928
$2^{7}$ | 0.946 | 0.948 | 0.950 | 0.966 | 0.950 | 0.916
$2^{8}$ | 0.952 | 0.952 | 0.952 | 0.968 | 0.952 | 0.926
Table 8: Coverage of nominal $95\%$ confidence intervals $\hat{C}(0.05)$ for
$\beta$ in Example 5.1.4.

##  Appendix G Further details on the generalised sandwich loss of Section 6

Consider the setup of Section 6 where we wish to estimate function $\beta(X)$
in model (18), a task for which by our assumption (19), it suffices to
estimate coefficient vector $\boldsymbol{\phi}$. Below we briefly outline how
this may be performed. For intuition, suppose the regression functions
$(l_{0},m_{0})$ (defined as in the rest of the paper) are known, and define
$R_{i}^{Y}:=Y_{i}-l_{0}(X_{i})$ and $R_{i}^{D}:=D_{i}-m_{0}(X_{i})$. Then

$R_{i}^{Y}=M_{i}\boldsymbol{\phi}+\varepsilon_{i},\qquad
M_{i}:=\big{(}\varphi_{l}(X_{ij})R_{ij}^{D}\big{)}_{j\in[n],l\in[L]}.$

An estimate of $\boldsymbol{\phi}$ can then be obtained via a potentially
weighted multivariate linear regression of the $R^{Y}_{i}$ onto the $M_{i}$.
In reality, estimates $(\hat{l},\hat{m})$ would be used in place of the
unknown $(l_{0},m_{0})$ in a cross-fitting scheme similar to that in Section
3.1. We may obtain an estimate $\hat{V}_{\boldsymbol{\phi}}(W)$ as a function
of the weight function, and then seek to minimise the generalised sandwich
loss objective (21) via sandwich boosting.

Algorithm 5 outlines how to calculate the $s$-scores for sandwich boosting in
this setup for an arbitrary inverse working correlation
$\mathcal{C}_{ijk}(\theta)$ as in (26). The order of computation of this score
calculation can be reduced to $O(L^{2}N)$ when using one of the working
correlation parameterisations in Section C; see Algorithm 6 for the example of
the equicorrelated working correlation.

Input: Index set $\mathcal{I}$ for boosting training; Working correlation
structure $\mathcal{C}_{i}$ (26); Estimates of (grouped) errors
$(\hat{\xi}_{i},\hat{\varepsilon}_{i},X_{i})_{i\in\mathcal{I}}$; Estimates of
$(s,\theta)$ at which scores to be calculated; $\Phi\in\mathbb{R}^{L\times L}$
matrix (see (20)).

Calculate $A^{(1)}\in\mathbb{R}^{|\mathcal{I}|\times L\times L}$ and
$A^{(2)}\in\mathbb{R}^{|\mathcal{I}|\times L\times L}$:

for  _$i\in\mathcal{I}$_ do

for  _$(l,l^{\prime})\in[L]^{2}$_ do

$A^{(1)}_{ill^{\prime}}:=\sum_{k=1}^{n_{i}}\sum_{k^{\prime}=1}^{n_{i}}\mathcal{C}_{ikk^{\prime}}(\theta)s(X_{ik})s(X_{ik^{\prime}})\hat{\xi}_{ik}\hat{\xi}_{ik^{\prime}}\varphi_{l}(X_{ik})\varphi_{l^{\prime}}(X_{ik^{\prime}})$,

$A^{(2)}_{ill^{\prime}}:=\sum_{k=1}^{n_{i}}\sum_{k^{\prime}=1}^{n_{i}}\mathcal{C}_{ikk^{\prime}}(\theta)s(X_{ik})s(X_{ik^{\prime}})\hat{\xi}_{ik}\hat{\varepsilon}_{ik^{\prime}}\varphi_{l}(X_{ik})\varphi_{l^{\prime}}(X_{ik^{\prime}})$.

end for

end for

Calculate $A^{(1)}_{\text{sum}}:=\sum_{i\in\mathcal{I}}A^{(1)}_{i}$ and
$A^{(2)}_{\text{sum.sq}}:=\sum_{i\in\mathcal{I}}A^{(2)}_{i}A^{(2)T}_{i}$.

Calculate $A^{(3)}$ and $A^{(4)}$:

for  _$i\in\mathcal{I}$ and $j\in[n_{i}]$_ do

for  _$(l,l^{\prime})\in[L]^{2}$_ do

$A^{(3)}_{ijll^{\prime}}:=\sum_{k=1}^{n_{i}}\mathcal{C}_{ijk}(\theta)s(X_{ik})\hat{\xi}_{ij}\hat{\xi}_{ik}\Big{(}\varphi_{l}(X_{ij})\varphi_{l^{\prime}}(X_{ik})+\varphi_{l^{\prime}}(X_{ij})\varphi_{l}(X_{ik})\Big{)}$,

$A^{(4)}_{ijll^{\prime}}:=\sum_{k=1}^{n_{i}}\mathcal{C}_{ijk}(\theta)s(X_{ik})\left(\hat{\xi}_{ij}\hat{\varepsilon}_{ik}\varphi_{l}(X_{ij})\varphi_{l^{\prime}}(X_{ik})+\hat{\xi}_{ik}\hat{\varepsilon}_{ij}\varphi_{l^{\prime}}(X_{ij})\varphi_{l}(X_{ik})\right)$.

end for

end for

Calculate $s$-scores:

for  _$i\in\mathcal{I}$ and $j\in[n_{i}]$_ do

$U_{\text{SL}}^{(s)}(s,\theta)(X_{ij}):=-\mathrm{tr}\bigg{(}\Phi\cdot\left(A_{\text{sum}}^{(1)}\right)^{-1}\Big{[}A_{ij}^{(3)}\left(A_{\text{sum}}^{(1)}\right)^{-1}A_{\text{sum.sq}}^{(2)}$

$\qquad+A_{\text{sum.sq}}^{(2)}\left(A_{\text{sum}}^{(1)}\right)^{-1}A_{ij}^{(3)}-A_{ij}^{(4)}A_{i}^{(2)}-A_{i}^{(2)}A_{ij}^{(4)}\Big{]}\left(A_{\text{sum}}^{(1)}\right)^{-1}\bigg{)}$.

end for

Calculate $A^{(5)}_{\text{sum}}$ and $A^{(6)}$:

for  _$(l,l^{\prime})\in[L]^{2}$_ do

$\big{(}A^{(5)}_{\text{sum}}\big{)}_{ll^{\prime}}=\sum_{i\in\mathcal{I}}\sum_{k=1}^{n_{i}}\sum_{k^{\prime}=1}^{n_{i}}\frac{\partial\mathcal{C}_{ikk^{\prime}}}{\partial\theta}s(X_{ik})s(X_{ik^{\prime}})\hat{\xi}_{ik}\hat{\xi}_{ik^{\prime}}\varphi_{l}(X_{ik})\varphi_{l^{\prime}}(X_{ik^{\prime}})$,

for  _$i\in\mathcal{I}$_ do

$A^{(6)}_{ill^{\prime}}:=\sum_{k=1}^{n_{i}}\sum_{k^{\prime}=1}^{n_{i}}\frac{\partial\mathcal{C}_{ikk^{\prime}}}{\partial\theta}s(X_{ik})s(X_{ik^{\prime}})\hat{\xi}_{ik}\hat{\varepsilon}_{ik^{\prime}}\varphi_{l}(X_{ik})\varphi_{l^{\prime}}(X_{ik^{\prime}})$.

end for

end for

Calculate
$A^{(6)}_{\text{sum.sq.diff}}:=\sum_{i\in\mathcal{I}}A_{i}^{(2)}A_{i}^{(6)}+A_{i}^{(6)}A_{i}^{(2)}$.

Calculate the $\theta$-score:

$U_{\text{SL}}^{(\theta)}(s,\theta)=-\mathrm{tr}\bigg{(}\Phi\cdot\left(A_{\text{sum}}^{(1)}\right)^{-1}\Big{[}A_{\text{sum}}^{(5)}\left(A_{\text{sum}}^{(1)}\right)^{-1}A_{\text{sum.sq}}^{(2)}+A_{\text{sum.sq}}^{(2)}\left(A_{\text{sum}}^{(1)}\right)^{-1}A_{\text{sum}}^{(5)}$

$\qquad-
A_{\text{sum.sq.diff}}^{(6)}\Big{]}\left(A_{\text{sum}}^{(1)}\right)^{-1}\bigg{)}$.

Output: The set of $N$ $s$-scores
$\left(X_{ij},\,U_{\mathrm{SL}}^{(s)}(s,\theta)(X_{ij})\right)_{i\in\mathcal{I},j\in[n_{i}]}$
and $\theta$-score $U_{\mathrm{SL}}^{(\theta)}(s,\theta)$.

Algorithm 5 Generalised sandwich boosting: Score calculation for arbitrary
working correlation

Input: Index set $\mathcal{I}$ for boosting training; Estimates of (grouped)
errors $(\hat{\xi}_{i},\hat{\varepsilon}_{i},X_{i})_{i\in\mathcal{I}}$;
Estimates of $(s,\theta)$ at which scores to be calculated;
$\Phi\in\mathbb{R}^{L\times L}$ matrix (see (20)).

Calculate $A^{(1)}\in\mathbb{R}^{|\mathcal{I}|\times L\times L}$,
$A^{(2)}\in\mathbb{R}^{|\mathcal{I}|\times L\times L}$,
$B^{(1)}\in\mathbb{R}^{|\mathcal{I}|\times L}$ and
$B^{(2)}\in\mathbb{R}^{|\mathcal{I}|\times L}$:

for  _$i\in\mathcal{I}$_ do

for  _$l\in[L]$_ do

$B_{il}^{(1)}:=\sum_{k=1}^{n_{i}}s(X_{ik})\hat{\xi}_{ik}\varphi_{l}(X_{ik})$,

$B_{il}^{(2)}:=\sum_{k=1}^{n_{i}}s(X_{ik})\hat{\varepsilon}_{ik}\varphi_{l}(X_{ik})$,

end for

for  _$(l,l^{\prime})\in[L]^{2}$_ do

$A^{(1)}_{ill^{\prime}}:=\sum_{k=1}^{n_{i}}s^{2}(X_{ik})\hat{\xi}_{ik}^{2}\varphi_{l}(X_{ik})\varphi_{l^{\prime}}(X_{ik})-\frac{\theta}{1+\theta
n_{i}}B_{il}^{(1)}B^{(1)}_{il^{\prime}}$,

$A^{(2)}_{ill^{\prime}}:=\sum_{k=1}^{n_{i}}s^{2}(X_{ik})\hat{\xi}_{ik}\hat{\varepsilon}_{ik}\varphi_{l}(X_{ik})\varphi_{l^{\prime}}(X_{ik})-\frac{\theta}{1+\theta
n_{i}}B_{il}^{(1)}B_{il^{\prime}}^{(2)}$.

end for

end for

Calculate $A^{(1)}_{\text{sum}}:=\sum_{i\in\mathcal{I}}A^{(1)}_{i}$ and
$A^{(2)}_{\text{sum.sq}}:=\sum_{i\in\mathcal{I}}A^{(2)}_{i}A^{(2)T}_{i}$.

Calculate $A^{(3)}$ and $A^{(4)}$:

for  _$i\in\mathcal{I}$ and $j\in[n_{i}]$_ do

for  _$(l,l^{\prime})\in[L]^{2}$_ do

$A^{(3)}_{ijll^{\prime}}:=2s(X_{ij})\hat{\xi}_{ij}^{2}\varphi_{l}(X_{ij})\varphi_{l^{\prime}}(X_{ij})-\frac{\theta}{1+\theta
n_{i}}\hat{\xi}_{ij}\left(B_{il}^{(1)}\varphi_{l^{\prime}}(X_{ij})+B^{(1)}_{il^{\prime}}\varphi_{l}(X_{ij})\right)$,

$A^{(4)}_{ijll^{\prime}}:=2s(X_{ij})\hat{\xi}_{ij}\hat{\varepsilon}_{ij}\varphi_{l}(X_{ij})\varphi_{l^{\prime}}(X_{ij})-\frac{\theta}{1+\theta
n_{i}}\left(B_{il}^{(1)}\hat{\varepsilon}_{ij}\varphi_{l^{\prime}}(X_{ij})+B_{il^{\prime}}^{(2)}\hat{\xi}_{ij}\varphi_{l}(X_{ij})\right)$.

end for

end for

Calculate $s$-scores:

for  _$i\in\mathcal{I}$ and $j\in[n_{i}]$_ do

$U_{\text{SL}}^{(s)}(s,\theta)(X_{ij}):=-\mathrm{tr}\bigg{(}\Phi\cdot\left(A_{\text{sum}}^{(1)}\right)^{-1}\Big{[}A_{ij}^{(3)}\left(A_{\text{sum}}^{(1)}\right)^{-1}A_{\text{sum.sq}}^{(2)}$

$\qquad+A_{\text{sum.sq}}^{(2)}\left(A_{\text{sum}}^{(1)}\right)^{-1}A_{ij}^{(3)}-A_{ij}^{(4)}A_{i}^{(2)}-A_{i}^{(2)}A_{ij}^{(4)}\Big{]}\left(A_{\text{sum}}^{(1)}\right)^{-1}\bigg{)}.$

end for

Calculate the $\theta$-score:

$U_{\text{SL}}^{(\theta)}(s,\theta)=\mathrm{tr}\Bigg{(}\Phi\cdot\left(A_{\text{sum}}^{(1)}\right)^{-1}\Bigg{[}\left(\underset{i\in\mathcal{I}}{\sum}\frac{B_{i}^{(1)}B_{i}^{(1)T}}{\left(1+\theta
n_{i}\right)^{2}}\right)\left(A_{\text{sum}}^{(1)}\right)^{-1}A_{\text{sum.sq}}^{(2)}$

$\qquad+A_{\text{sum.sq}}^{(2)}\left(A_{\text{sum}}^{(1)}\right)^{-1}\left(\underset{i\in\mathcal{I}}{\sum}\frac{B_{i}^{(1)}B_{i}^{(1)T}}{\left(1+\theta
n_{i}\right)^{2}}\right)-\underset{i\in\mathcal{I}}{\sum}\frac{A_{i}^{(2)}B_{i}^{(1)}B_{i}^{(2)T}+B_{i}^{(2)}B_{i}^{(1)T}A_{i}^{(2)}}{\left(1+\theta
n_{i}\right)^{2}}\Bigg{]}\left(A_{\text{sum}}^{(1)}\right)^{-1}\Bigg{)}$.

Output: The set of $N$ $s$-scores
$\left(X_{ij},\,U_{\mathrm{SL}}^{(s)}(s,\theta)(X_{ij})\right)_{i\in\mathcal{I},j\in[n_{i}]}$
and $\theta$-score $U_{\mathrm{SL}}^{(\theta)}(s,\theta)$.

Algorithm 6 Generalised sandwich boosting: Score calculation for
equicorrelated working correlation

## References

  * Chernozhukov et al. [2018] V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins.  Double/debiased machine learning for treatment and structural parameters.  _The Econometrics Journal_ , 21(1):C1–C68, 2018\.  doi: 10.1111/ectj.12097.
  * Emmenegger and Bühlmann [2023] C. Emmenegger and P. Bühlmann.  Plug-in machine learning for partially linear mixed-effects models with repeated measurements.  _Scandinavian Journal of Statistics_ , 2023.  doi: 10.1111/sjos.12639.
  * Halekoh et al. [2006] U. Halekoh, S. Højsgaard, and J. Yan.  The r package geepack for generalized estimating equations.  _Journal of Statistical Software_ , 15/2:1–11, 2006.
  * Jennrich [1969] R. I. Jennrich.  Asymptotic Properties of Non-Linear Least Squares Estimators.  _The Annals of Mathematical Statistics_ , 40(2):633 – 643, 1969.  doi: 10.1214/aoms/1177697731.
  * Lundborg et al. [2022] A. R. Lundborg, I. Kim, R. D. Shah, and R. J. Samworth.  The projected covariance measure for assumption-lean variable significance testing.  _arXiv preprint arXiv:2211.02039_ , 2022.
  * Pinheiro et al. [2022] J. Pinheiro, D. Bates, and R Core Team.  _nlme: Linear and Nonlinear Mixed Effects Models_ , 2022.  URL https://CRAN.R-project.org/package=nlme.  R package version 3.1-161.
  * Shah and Peters [2020] R. D. Shah and J. Peters.  The hardness of conditional independence testing and the generalised covariance measure.  _The Annals of Statistics_ , 48(3), June 2020.  doi: 10.1214/19-aos1857.
  * van der Vaart [1998] A. W. van der Vaart.  _Asymptotic Statistics_.  Cambridge University Press, 1998.
  * Wood [2017] S. Wood.  _Generalized Additive Models: An Introduction with R_.  Chapman and Hall/CRC, 2 edition, 2017.