$\displaystyle\mathsf{W}^{2}_{2}(\Theta^{\intercal}Z,\Theta^{\intercal}Z^{*})$
$\displaystyle\leq
2k\left(\alpha(X)+\alpha(Y)+\frac{2d_{x}\beta_{2}^{2}(X)}{\mathrm{tr}(\Sigma_{X})}+\frac{2d_{y}\beta_{2}^{2}(X)}{\mathrm{tr}(\Sigma_{Y})}\right)$
(21)

where $\alpha(X)$ and $\alpha(Y)$ are defined in Lemma 10.

It remains to transform the MI term $\mathsf{I}(\Theta;\Theta^{\intercal}Z)$
in (18) into $\mathsf{I}(\Theta_{*};\Theta_{*}^{\intercal}Z+\sqrt{t}N)$, where
$N\sim\mathcal{N}(0,\mathrm{I}_{2k})$ and
$\Theta_{*}=\mathrm{A}_{*}\oplus\mathrm{B}_{*}$ with $\mathrm{A}_{*}$ and
$\mathrm{B}_{*}$ matrices of dimension $d_{x}\times k$ and $d_{y}\times k$ and
entries i.i.d. according to $\mathcal{N}(0,1/d_{x})$ and
$\mathcal{N}(0,1/d_{y})$, respectively. Using the polar decomposition of
Gaussian matrices, we know that $\smash{\mathrm{A}_{*}\stackrel{{\scriptstyle
d}}{{=}}\mathrm{A}(\mathrm{A}_{*}^{\intercal}\mathrm{A}_{*})^{1/2}}$, where
$\mathrm{A}\sim\sigma_{k,d_{x}}$, i.e., it is uniformly distributed over
$\mathrm{St}(k,d_{x})$. A similar claim holds for $\mathrm{B}_{*}$. By
invariance of MI to invertible transformations
($\mathrm{A}_{*}^{\intercal}\mathrm{A}_{*}$ and
$\mathrm{B}_{*}^{\intercal}\mathrm{B}_{*}$ are a.s. invertible), we have

$\mathsf{I}(\Theta;\Theta^{\intercal}Z)=\mathsf{I}(\mathrm{A},\mathrm{B};\mathrm{A}^{\intercal}X,\mathrm{B}^{\intercal}Y)=\mathsf{I}(\mathrm{A}_{*},\mathrm{B}_{*};\mathrm{A}_{*}^{\intercal}X,\mathrm{B}_{*}^{\intercal}Y)=\mathsf{I}(\Theta_{*};\Theta_{*}^{\intercal}Z).$

Next, we introduce the noise into the latter MI as follows. Denote the
distribution of $\Theta_{*}$ by $\gamma$ and consider

$\displaystyle\mathsf{I}(\Theta_{*};\Theta_{*}^{\intercal}Z)-\mathsf{I}(\Theta_{*};W)$
$\displaystyle=\mathsf{h}(\Theta_{*}^{\intercal}Z)-\mathsf{h}(W)+\mathsf{h}(W|\Theta_{*})-\mathsf{h}(\Theta_{*}^{\intercal}Z|\Theta_{*})$
$\displaystyle\leq\int\mathsf{h}(W|\Theta_{*}=\theta)-\mathsf{h}(\Theta_{*}^{\intercal}Z|\Theta_{*}=\theta)d\gamma(\theta)$
$\displaystyle\leq\int\sqrt{\mathsf{J}(\theta^{\intercal}Z)}\mathsf{W}_{2}\big{(}\theta^{\intercal}Z,\theta^{\intercal}Z+\sqrt{t}N\big{)}d\gamma(\theta)$
$\displaystyle\leq\sqrt{2kt\int\mathsf{J}(\theta^{\intercal}Z)d\gamma(\theta)},$
(22)

where the first inequality follow since
$\mathsf{h}(W)\geq\mathsf{h}(W|N)=\mathsf{h}(\Theta_{*}^{\intercal}Z)$ (as
conditioning cannot increase differential entropy), the second inequality
follows from Lemma 1, while the last step upper bounds the 2-Wasserstein
distance by $\big{(}\mathbb{E}[\|\sqrt{t}N\|^{2}]\big{)}^{1/2}=\sqrt{2kt}$ and
applies Jensen’s inequality.

To bound the expected Fisher information, let $\theta^{\dagger}$ denote the
pseudo-inverse of $\theta$. Using the data processing inequality once more, we
have

$\displaystyle\int\mathsf{J}(\theta^{\intercal}Z)d\gamma(\theta)$
$\displaystyle\leq\int\mathrm{tr}\big{(}\theta^{\dagger}\mathrm{J}_{\mathrm{F}}(Z)(\theta^{\dagger})^{\intercal}\big{)}d\gamma(\theta)$
$\displaystyle=\int\mathrm{tr}\big{(}\mathrm{J}_{\mathrm{F}}(Z)(\theta\theta^{\intercal})^{-1}\big{)}d\gamma(\theta)$
$\displaystyle=\|\mathrm{J}_{\mathrm{F}}(Z)\|_{\operatorname{op}}\,\mathbb{E}\big{[}\mathrm{tr}\big{(}(\Theta_{*}^{\intercal}\Theta_{*})^{-1}\big{)}\big{]}$
$\displaystyle=\|\mathrm{J}_{\mathrm{F}}(Z)\|_{\operatorname{op}}\,\left(d_{x}\mathbb{E}\,\big{[}\mathrm{tr}\big{(}(\tilde{\mathrm{A}}_{*}^{\intercal}\tilde{\mathrm{A}}_{*})^{-1}\big{)}\big{]}+d_{y}\mathbb{E}\,\big{[}\mathrm{tr}\big{(}(\tilde{\mathrm{B}}_{*}^{\intercal}\tilde{\mathrm{B}}_{*})^{-1}\big{)}\big{]}\right),$

where $\tilde{\mathrm{A}}_{*}$ and $\tilde{\mathrm{B}}_{*}$ are random
Gaussian matrices of dimensions $d_{x}\times k$ and $d_{y}\times k$,
respectively, with i.i.d. $\mathcal{N}(0,1)$ entries. Consequently, note that
$\tilde{\mathrm{A}}_{*}^{\intercal}\tilde{\mathrm{A}}_{*}$ and
$\tilde{\mathrm{B}}_{*}^{\intercal}\tilde{\mathrm{B}}_{*}$ follow the $k\times
k$ Wishart distribution with $d_{x}$ and $d_{y}$ degrees of freedom,
respectively. For $d_{x}>k+1$ the mean of the inverse is
$\mathbb{E}\big{[}(\mathrm{A}^{\intercal}_{*}\mathrm{A}_{*})^{-1}\big{]}=\frac{1}{d_{x}-k-1}\mathrm{I}_{2k}$
and so
$\mathbb{E}\big{[}\mathrm{tr}\big{(}(\mathrm{A}^{\intercal}_{*}\mathrm{A}_{*})^{-1}\big{)}\big{]}=\frac{k}{d_{x}-k-1}$;
cf. e.g., [46] (and similarly for $\tilde{\mathrm{B}}_{*}$). Inserting this
into (22) and combining with (19) and (21) yields the result.∎

###  B.2 Proof of Lemma 11

The following bounds follow by the exact same argument of Lemmas 4 and 5 from
[32], respectively.

######  Lemma 12.

We have

$\mathsf{I}(\Theta_{*};W)\leq\kappa\int_{\mathbb{R}^{2k}}\sqrt{\mathrm{Var}(p_{W|\Theta_{*}}(w|\Theta_{*}))}dz$

where $\kappa=\sup_{x\in(0,\infty)}\log(1+x)/\sqrt{x}\approx 0.80474$.

######  Lemma 13.

Let $f:\mathbb{R}^{d}\to\mathbb{R}_{\geq 0}$ be a non-negative integrable
function and denote its $p$th moment by $\eta_{p}[f]:=\int\|z\|^{p}f(z)dz$. If
$\eta_{d-1}[f]$, $\eta_{d+1}[f]<\infty$, then

$\int\sqrt{f(z)}dz\leq\sqrt{\frac{2\pi^{\frac{d}{2}+1}}{\Gamma\left(\frac{d}{2}\right)}}\big{(}\eta_{d-1}[f]\eta_{d+1}[f]\big{)}^{\frac{1}{4}},$

where $\Gamma(z)$ is the Gamma function.

Let $\varphi_{t}$ denote the density of $\mathcal{N}(0,t\mathrm{I}_{d})$; the
dimension is suppressed and should be understood from the context, while the
subscript is omitted when $t=1$. Define the following quantities:

$\displaystyle m_{p}(W,\Theta_{*})$
$\displaystyle:=\frac{\int_{\mathbb{R}^{2k}}\|w\|^{p}\mathrm{Var}\big{(}p_{W|\Theta_{*}}(w|\Theta_{*})\big{)}dw}{\big{(}\int_{\mathbb{R}^{k}}\varphi^{2}(w)dw\big{)}\big{(}\int_{\mathbb{R}^{k}}\|w\|^{p}\varphi^{2}(w)dw\big{)}}$
$\displaystyle M(W,\Theta_{*})$
$\displaystyle:=\sqrt{m_{2k-1}(W,\Theta_{*})m_{2k+1}(W,\Theta_{*})}.$

The following lemma is adapted from Lemma 6 of [32] to accommodate our
$M(W,\Theta_{*})$, definition which slightly differs from theirs.

######  Lemma 14.

If the conditional distribution of $W$ given $\Theta_{*}$, $p_{W|\Theta_{*}}$,
is absolutely continuous w.r.t. $\mathrm{Leb}$ and $M(W,\Theta_{*})<\infty$,
we have

$\mathsf{I}(W;\Theta_{*})\leq\kappa\left(\frac{3\pi
k}{8}\right)^{\frac{1}{4}}\sqrt{M(W,\Theta_{*})},$

where $\kappa$ is as defined in Lemma 12.

###### Proof.

Lemmas 12 and 13 together imply

$\displaystyle\mathsf{I}(W;\Theta_{*})$
$\displaystyle\leq\kappa\sqrt{\frac{2\pi^{k+1}}{\Gamma(k)}}\Big{(}\eta_{2k-1}\big{[}\mathrm{Var}\big{(}p_{W|\Theta_{*}}(w|\Theta_{*})\big{)}\big{]}\cdot\eta_{2k+1}\big{[}\mathrm{Var}\big{(}p_{W|\Theta_{*}}(w|\Theta_{*})\big{)}\big{]}\Big{)}^{\frac{1}{4}},$
$\displaystyle=\kappa\sqrt{\frac{2\pi^{k+1}}{\Gamma(k)}}\sqrt{M(W,\Theta_{*})\int_{\mathbb{R}^{k}}\varphi^{2}(w)dw}$
$\displaystyle\qquad\qquad\qquad\qquad\times\left(\int_{\mathbb{R}^{k}}\|w\|^{2k-1}\varphi^{2}(w)dw\int_{\mathbb{R}^{k}}\|w\|^{2k+1}\varphi^{2}(w)dw\right)^{\frac{1}{4}},$
$\displaystyle=\kappa\sqrt{M(W,\Theta_{*})}\sqrt{2^{1-2k}\pi}\left(\frac{\Gamma(\frac{3k-1}{2})\Gamma(\frac{3k+1}{2})}{\Gamma^{2}(\frac{k}{2})\Gamma^{2}(k)}\right)^{\frac{1}{4}},$
$\displaystyle\leq\kappa\sqrt{M(W,\Theta_{*})}({6k\pi^{2}})^{\frac{1}{4}}\left(2^{-2k}\frac{\Gamma(\frac{3k-1}{2})}{\Gamma(\frac{k}{2})\Gamma(k)}\right)^{\frac{1}{2}},$
$\displaystyle\leq\frac{\kappa}{2}\sqrt{M(W,\Theta_{*})}({6k\pi})^{\frac{1}{4}},$

where we have substituted in $M(W,\Theta_{*})$ as defined above, have noted
that
$\int\|w\|^{p}\varphi^{2}(w)dw=(4\pi)^{-\frac{k}{2}}\Gamma(\frac{k+p}{2})/\Gamma(\frac{k}{2})$.
The last step observes that
$2^{-2k}\frac{\Gamma(\frac{3k-1}{2})}{\Gamma(\frac{k}{2})\Gamma(k)}$ is a
decreasing in $k\geq 1$. This can be verified by by using the fact that that
as $\Gamma(z+1)=z\Gamma(z)$, increasing $k$ by 2 will decrease
$2^{-2k}\frac{\Gamma(\frac{3k-1}{2})}{\Gamma(\frac{k}{2})\Gamma(k)}$. ∎

Given the bound in Lemma 14, we next bound the moment $M(W,\Theta_{*})$. To
that end, we control $\eta_{p}(W,\Theta_{*})$. For convenience of notation, we
set $Z=(Z_{1}^{\intercal}\ Z_{2}^{\intercal})^{\intercal}$, i.e., $Z_{1}=X$,
and $Z_{2}=Y$, and $d_{1}=d_{x}$, $d_{2}=d_{y}$. We consider two independent
copies of $Z$, denoted by $Z^{(1)}$ and $Z^{(2)}$. With this notation, we have
the following lemma.

######  Lemma 15.

For any $p\geq 0$, we have

$\displaystyle m_{p}(W,\Theta_{*})$
$\displaystyle=\mathbb{E}\left[(V_{a,1}-R_{1})^{-\frac{1}{2}}(V_{a,2}-R_{2})^{-\frac{1}{2}}\left(\left(\frac{V_{g,1}^{2}-R_{1}^{2}}{V_{a,1}-R_{1}}\right)^{\frac{p}{2}}+\left(\frac{V_{g,2}^{2}-R_{2}^{2}}{V_{a,2}-R_{2}}\right)^{\frac{p}{2}}\right)\right]$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad-\mathbb{E}\left[V_{a,1}^{-\frac{1}{2}}V_{a,2}^{-\frac{1}{2}}\left(\left(\frac{V_{g,1}^{2}}{V_{a,1}}\right)^{\frac{p}{2}}+\left(\frac{V_{g,2}^{2}}{V_{a,2}}\right)^{\frac{p}{2}}\right)\right],$

where

$\displaystyle V_{a,i}$
$\displaystyle=t+\frac{1}{2d_{i}}\|Z_{i}^{(1)}\|^{2}+\frac{1}{2d_{i}}\|Z_{i}^{(2)}\|^{2},$
$\displaystyle V_{g,i}$
$\displaystyle=\sqrt{\left(t+\frac{1}{d_{i}}\|Z_{i}^{(1)}\|^{2}\right)\left(t+\frac{1}{d_{i}}\|Z_{i}^{(2)}\|^{2}\right)},$
$\displaystyle R_{i}$ $\displaystyle=\frac{1}{d_{i}}\langle
Z_{i}^{(1)},Z_{i}^{(2)}\rangle.$

###### Proof.

By the definition of $W$, we have
$p_{W|\Theta_{*}}(w|\theta)=\mathbb{E}\big{[}\varphi_{t}(w-\theta^{\intercal}Z)\big{]}$,
whereby

$p^{2}_{W|\Theta_{*}}(w|\theta)=\mathbb{E}\big{[}\varphi_{t}(w-\theta^{\intercal}Z^{(1)})\varphi_{t}(w-\theta^{\intercal}Z^{(2)})\big{]}.$

Taking the expectation over the distribution of $\Theta_{*}$ and swapping the
order of expectation yields

$\mathbb{E}\big{[}p^{2}_{W|\Theta_{*}}(w|\Theta_{*})\big{]}=\mathbb{E}\big{[}\nu(w,Z^{(1)},Z^{(2)})\big{]}$

where
$\nu(y,z^{(1)},z^{(2)})=\mathbb{E}[\varphi_{t}(w-\Theta_{*}^{\intercal}z^{(1)})\varphi_{t}(w-\Theta_{*}^{\intercal}z^{(2)})]$.
Note that since $z^{(1)},z^{(2)}$ are fixed,

$\left[\begin{array}[]{c}\Theta_{*}^{\intercal}z^{(1)}\\\
\Theta_{*}^{\intercal}z^{(2)}\end{array}\right]\sim\mathcal{N}(0,\Sigma)$

where

$\Sigma=\left[\begin{array}[]{cccc}\|z_{1}^{(1)}\|^{2}/d_{1}&0&\frac{\langle
z_{1}^{(1)},z_{1}^{(2)}\rangle}{d_{1}}&0\\\
0&\|z_{2}^{(1)}\|^{2}/d_{2}&0&\frac{\langle
z_{2}^{(1)},z_{2}^{(2)}\rangle}{d_{2}}\\\ \frac{\langle
z_{1}^{(1)},z_{1}^{(2)}\rangle}{d_{1}}&0&\|z_{1}^{(2)}\|^{2}/d_{1}&0\\\
0&\frac{\langle
z_{2}^{(1)},z_{2}^{(2)}\rangle}{d_{2}}&0&\|z_{2}^{(2)}\|^{2}/d_{2}\end{array}\right]\otimes\mathrm{I}_{k}.$

The proof of Lemma 7 from [32] shows that

$\displaystyle\nu(w,z^{(1)},z^{(2)})=(2\pi)^{2k}\left|\Sigma+t\,\mathrm{I}_{4k}\right|^{-\frac{1}{2}}\exp\left(-\frac{1}{2}\left\|(\Sigma+t\,\mathrm{I}_{4k})^{-\frac{1}{2}}\left[\begin{array}[]{c}w\\\
w\end{array}\right]\right\|^{2}\right).$ (25)

It is convenient to transform $\Sigma$ into a block-diagonal form. To that
end, let us consider the (orthonormal) permutation matrix

$\mathrm{P}=\left[\begin{array}[]{cccc}1&0&0&0\\\ 0&0&1&0\\\ 0&1&0&0\\\
0&0&0&1\end{array}\right],$

and set
$\Sigma^{\prime}=(\mathrm{P}\otimes\mathrm{I}_{k})\Sigma(\mathrm{P}\otimes\mathrm{I}_{k})^{\intercal}$.
This gives

${\Sigma}^{\prime}=\left[\begin{array}[]{cccc}\|z_{1}^{(1)}\|^{2}/d_{1}&\frac{\langle
z_{1}^{(1)},z_{1}^{(2)}\rangle}{d_{1}}&0&0\\\ \frac{\langle
z_{1}^{(1)},z_{1}^{(2)}\rangle}{d_{1}}&\|z_{1}^{(2)}\|^{2}/d_{1}&0&0\\\
0&0&\|z_{2}^{(1)}\|^{2}/d_{2}&\frac{\langle
z_{2}^{(1)},z_{2}^{(2)}\rangle}{d_{2}}\\\ 0&0&\frac{\langle
z_{2}^{(1)},z_{2}^{(2)}\rangle}{d_{2}}&\|z_{2}^{(2)}\|^{2}/d_{2}\end{array}\right]\otimes\mathrm{I}_{k}.$

Note that since $\mathrm{P}$ is a permutation matrix, the eigenvalues of
$\Sigma$ and $\tilde{\Sigma}$ are equal, hence
$\left|\Sigma+t\mathrm{I}\right|=\left|\tilde{\Sigma}+t\mathrm{I}\right|=(v_{g,1}^{2}-r_{1}^{2})^{k}(v_{g,2}^{2}-r_{2}^{2})^{k}$.
We also obtain

$\frac{1}{2}\left\|(\Sigma+t\,\mathrm{I}_{4k})^{-\frac{1}{2}}\mspace{-3.0mu}\left[\begin{array}[]{c}w\\\
w\end{array}\right]\right\|^{2}\mspace{-6.0mu}=\mspace{-3.0mu}\frac{1}{2}\left\|(\tilde{\Sigma}+t\,\mathrm{I}_{4k})^{-\frac{1}{2}}\mspace{-4.0mu}\left[\begin{array}[]{c}w_{1}\\\
w_{1}\\\ w_{2}\\\
w_{2}\end{array}\right]\right\|^{2}\mspace{-6.0mu}=\mspace{-3.0mu}\left(\frac{v_{a,1}-r_{1}}{v_{g,1}^{2}-r_{1}^{2}}\right)\|w_{1}\|^{2}+\left(\frac{v_{a,2}-r_{2}}{v_{g,2}^{2}-r_{2}^{2}}\right)\|w_{2}\|^{2},$

where $(v_{a,i},v_{g,i},r_{i})$ are as defined in the lemma statement and we
have observed that $\mathrm{P}^{\intercal}\mathrm{P}=\mathrm{I}_{4}$ and
$(\mathrm{P}\otimes\mathrm{I}_{k})\left[\begin{array}[]{c}w\\\
w\end{array}\right]=\left[\begin{array}[]{c}w_{1}\\\ w_{1}\\\ w_{2}\\\
w_{2}\end{array}\right]$. Substituting into (25) and simplifying yields

$\nu(w,Z^{(1)},Z^{(2)})=(V_{a,1}-R_{1})^{-\frac{k}{2}}(V_{a,2}-R_{2})^{-\frac{k}{2}}{U_{1}^{\frac{k}{2}}U_{2}^{\frac{k}{2}}}\varphi_{k,1}^{2}(U_{1}^{-\frac{1}{2}}y_{1})\varphi_{k,1}^{2}(U_{2}^{-\frac{1}{2}}y_{2})$

where $U_{i}=(V_{g,i}^{2}-R_{i}^{2})/(V_{a,i}-R_{i})$. Then the $p$th moment
of $\nu$ with respect to $w$ is (using change of variables) is give by

$\displaystyle\eta_{p}\big{[}\mathbb{E}\big{[}p^{2}_{W|\Theta_{*}}(w|\Theta_{*})\big{]}\big{]}$
$\displaystyle\qquad=\mathbb{E}\left[\int\|w\|^{p}\nu(w,Z^{(1)},Z^{(2)})dw\right]$
$\displaystyle\qquad=\mathbb{E}\left[(V_{a,1}-R_{1})^{-\frac{k}{2}}(V_{a,2}-R_{2})^{-\frac{k}{2}}\int
w_{1}^{p}{U_{1}^{\frac{k}{2}}U_{2}^{\frac{k}{2}}}\varphi_{1}^{2}(U_{1}^{-\frac{1}{2}}w_{1})\varphi_{1}^{2}(U_{2}^{-\frac{1}{2}}z_{2})dw\right.$
$\displaystyle\qquad\qquad\qquad\left.+(V_{a,1}-R_{1})^{-\frac{k}{2}}(V_{a,2}-R_{2})^{-\frac{k}{2}}\int
w_{2}^{p}{U_{1}^{\frac{k}{2}}U_{2}^{\frac{k}{2}}}\varphi_{k,1}^{2}(U_{1}^{-\frac{1}{2}}w_{1})\varphi_{k,1}^{2}(U_{2}^{-\frac{1}{2}}w_{2})dw\right]$
$\displaystyle\qquad=\eta_{0}[\varphi^{2}]\eta_{p}[\varphi^{2}]\mathbb{E}\mspace{-3.0mu}\left[(V_{a,1}-R_{1})^{-\frac{k}{2}}(V_{a,2}-R_{2})^{-\frac{k}{2}}\left(\left(\frac{V_{g,1}^{2}-R_{1}^{2}}{V_{a,1}-R_{1}}\right)^{\frac{p}{2}}\mspace{-3.0mu}+\mspace{-3.0mu}\left(\frac{V_{g,2}^{2}-R_{2}^{2}}{V_{a,2}-R_{2}}\right)^{\frac{p}{2}}\right)\right]\mspace{-3.0mu}.$
(26)

Next, we find the $p$th moment of the unconditional squared density
$p_{W}^{2}$. First, as in the conditional case, we write
$p_{W}^{2}(w)=\mathbb{E}[\tilde{\nu}(w,z^{(1)},z^{(2)})]$, where

$\tilde{\nu}(w)=\mathbb{E}\big{[}\varphi_{t}(w-\Theta_{*}^{(1)}z^{(1)})\varphi_{t}(w-\Theta_{*}^{(2)}z^{(2)})\big{]}$

with $\Theta_{*}^{(i)}$, for $i=1.2$, being independent copies of
$\Theta_{*}$. This independence in turn decorrelates $\Theta_{*}^{(1)}z^{(1)}$
and $\Theta_{*}^{(2)}z^{(2)}$, i.e.,

$\left[\begin{array}[]{c}\Theta_{*}^{(1)}z^{(1)}\\\
\Theta_{*}^{(2)}z^{(2)}\end{array}\right]\sim\mathcal{N}\big{(}0,\mathrm{diag}(\Sigma)\big{)}.$

Proceeding as in the correlated case above yields

$\eta_{p}\big{[}\mathbb{E}[p_{W}^{2}(w)]\big{]}=\eta_{0}[\varphi^{2}]\eta_{p}[\varphi^{2}]=\mathbb{E}\left[V_{a,1}^{-\frac{k}{2}}V_{a,2}^{-\frac{k}{2}}\left(\left(\frac{V_{g,1}^{2}}{V_{a,1}}\right)^{\frac{p}{2}}+\left(\frac{V_{g,2}^{2}}{V_{a,2}}\right)^{\frac{p}{2}}\right)\right],$

and combining this with (26) gives

$\displaystyle m_{p}(W,\Theta_{*})$
$\displaystyle=\frac{\eta_{p}\big{[}\mathrm{Var}\big{(}p^{2}_{W|\Theta_{*}}(w|\Theta_{*})\big{)}\big{]}}{\eta_{0}[\varphi^{2}]\eta_{p}[\varphi^{2}]}$
$\displaystyle=\mathbb{E}\left[(V_{a,1}-R_{1})^{-\frac{k}{2}}(V_{a,2}-R_{2})^{-\frac{k}{2}}\left(\left(\frac{V_{g,1}^{2}-R_{1}^{2}}{V_{a,1}-R_{1}}\right)^{\frac{p}{2}}+\left(\frac{V_{g,2}^{2}-R_{2}^{2}}{V_{a,2}-R_{2}}\right)^{\frac{p}{2}}\right)\right]$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad-\mathbb{E}\left[V_{a,1}^{-\frac{k}{2}}V_{a,2}^{-\frac{k}{2}}\left(\left(\frac{V_{g,1}^{2}}{V_{a,1}}\right)^{\frac{p}{2}}+\left(\frac{V_{g,2}^{2}}{V_{a,2}}\right)^{\frac{p}{2}}\right)\right].$

∎

It remains to bound the expectation in Lemma 15. We start with the following
bound.

######  Lemma 16.

For any $p\geq 0$, we have

$m_{p}(W,\Theta_{*})\leq\mathbb{E}\left[V_{a,2}^{-\frac{k}{2}}V_{a,1}^{-\frac{k-p}{2}}g_{p}\left(\frac{R_{1}}{V_{a,1}},\frac{R_{2}}{V_{a,2}}\right)+V_{a,1}^{-\frac{k}{2}}V_{a,2}^{-\frac{k-p}{2}}g_{p}\left(\frac{R_{2}}{V_{a,2}},\frac{R_{1}}{V_{a,1}}\right)\right],$

where $g_{k,p}:(-1,1)\rightarrow\mathbb{R}$ is given by
$g_{k,p}(u,v):=(1-v)^{-\frac{k}{2}}(1-u)^{-\frac{k}{2}}(1+u)^{\frac{p}{2}}-1$.

###### Proof.

Note that for $i,j=1,2$ with $i\neq j$, we have

$\displaystyle(V_{a,j}-R_{j})^{-\frac{k}{2}}$
$\displaystyle(V_{a,i}-R_{i})^{-\frac{k}{2}}\left(\frac{V_{g,i}^{2}-R_{i}^{2}}{V_{a,i}-R_{i}}\right)^{\frac{p}{2}}-V_{a,i}^{-\frac{k}{2}}V_{a,j}^{-\frac{k}{2}}\left(\frac{V_{g,i}^{2}}{V_{a,i}}\right)^{\frac{p}{2}}$
$\displaystyle=V_{a,i}^{-\frac{k+p}{2}}V_{a,j}^{-\frac{k}{2}}V_{g,i}^{p}\left[\left(1-\frac{R_{j}}{V_{a,j}}\right)^{-\frac{k}{2}}\left(1-\frac{R_{i}}{V_{a,i}}\right)^{-\frac{k}{2}}\left({1+\frac{R_{i}}{V_{g,i}}}\right)^{\frac{p}{2}}-1\right]$
$\displaystyle=g_{p}\left(\frac{R_{i}}{V_{a,i}},\frac{R_{j}}{V_{a,j}}\right)V_{a,j}^{-\frac{k}{2}}V_{a,i}^{-\frac{k+p}{2}}V_{g,i}^{p}$
$\displaystyle\leq
g_{p}\left(\frac{R_{i}}{V_{a,i}},\frac{R_{j}}{V_{a,j}}\right)V_{a,j}^{-\frac{k}{2}}V_{a,i}^{-\frac{k-p}{2}},$

where we have noted that $V_{g,i}\leq V_{a,i}$ since the geometric mean is
upper bounded by the arithmetic mean. Substituting into Lemma 15 completes the
proof. ∎

To make the expectation of $g_{p}$ tractable we next upper bound it by a
quadratic function.

######  Lemma 17.

For any $t>0$ and $(r_{1},r_{2})$ such that $|r_{1}|\leq c_{1},|r_{2}|\leq
c_{2}$, for some $c_{1},c_{2}$, we have

$g_{p}\left(\frac{r_{i}}{t+c_{i}},\frac{r_{j}}{t+c_{j}}\right)\leq\frac{k+p}{2}\frac{r_{i}}{t+c_{i}}+\frac{k}{2}\frac{r_{j}}{t+c_{j}}+t^{-k}(t+2c_{i})^{\frac{p}{2}}(t+c_{i})^{\frac{k-p}{2}}(t+c_{j})^{\frac{k}{2}}\left(\frac{r_{i}^{2}+r_{j}^{2}}{c_{i}^{2}\wedge
c_{j}^{2}}\right),$

where $i,j=1,2$ with $i\neq j$.

###### Proof.

Since $g_{p}(0,0)=0$, we decompose

$g_{p}(u,v)=\big{(}\nabla
g_{p}(0,0)\big{)}^{\intercal}\left[\begin{array}[]{c}u\\\
v\end{array}\right]+h_{p}(u,v)(u^{2}+v^{2}),$

where $h_{p}(u,v)=\left(g_{p}(u,v)-\nabla
g_{p}(0,0)^{\intercal}\left[\begin{array}[]{c}u\\\
v\end{array}\right]\right)/(u^{2}+v^{2})$. It can be verified that $h_{p}$ is
non-negative and nondecreasing in both arguments. Hence, for all $-1<u\leq
z_{u}<1$, $-1<v\leq z_{v}<1$, we have

$g_{p}(u,v)\leq\nabla g_{p}(0,0)^{\intercal}\left[\begin{array}[]{c}u\\\
v\end{array}\right]+h_{p}(z_{u},z_{v})(u^{2}+v^{2}).$

Furthermore, for $z_{u},z_{v}>0$,

$h_{p}(z_{u},z_{v})\leq\frac{g_{p}(z_{u},z_{v})}{z_{u}^{2}+z_{v}^{2}}=\frac{1}{z_{u}^{2}+z_{v}^{2}}(1-z_{v})^{-\frac{k}{2}}(1-z_{u})^{-\frac{k}{2}}(1+z_{u})^{\frac{p}{2}}.$

Using $z_{u}=c_{i}/(t+c_{i})$, $z_{v}=c_{j}/(t+c_{j})$, we obtain

$\displaystyle h_{p}(z_{u},z_{v})$
$\displaystyle\leq\frac{1}{2\min(z_{u}^{2},z_{v}^{2})}t^{-k}(t+c_{j})^{\frac{k}{2}}(t+c_{i})^{\frac{k-p}{2}}(t+2c_{i})^{\frac{p}{2}},$

from which the result follows.

∎

Next, we provide a bound on $M(W,\Theta_{*})$ subject to a.s. boundedness
assumption on the squared norms of the random variables. The subsequently
presented Lemma 19 then relaxes this assumption to a bound on the MI term of
interest.

######  Lemma 18.

Suppose that
$\lambda_{\min}\leq\frac{\|\mathrm{X}\|^{2}}{d_{x}}\wedge\frac{\|\mathrm{Y}\|^{2}}{d_{y}}\leq\frac{\|\mathrm{X}\|^{2}}{d_{x}}\vee\frac{\|\mathrm{Y}\|^{2}}{d_{y}}\leq\lambda_{\max}$
a.s. Then

$M(W,\Theta_{*})\leq
2^{\frac{1}{4}}\left(\frac{\lambda_{\max}}{\lambda_{\min}}\right)^{\frac{k}{2}}\left[4k\frac{\beta_{1}(Z_{1})+\beta_{1}(Z_{2})}{\lambda_{\min}}+2\left(1+\frac{2\lambda_{\max}}{t}\right)^{k}\frac{\beta_{2}^{2}(Z_{1})+\beta_{2}^{2}(Z_{2})}{\lambda_{\min}^{2}}\right].$

###### Proof.

Using Lemma 17 and the definitions of $\beta_{r}^{r}$, $R_{i}$, and $V_{a,i}$,
for $p=2k-1$, we have

$\displaystyle\mathbb{E}\left[V_{a,j}^{-\frac{k}{2}}V_{a,i}^{-\frac{k-p}{2}}g_{p}\left(\frac{R_{i}}{V_{a,i}},\frac{R_{j}}{V_{a,j}}\right)\right]\leq(t+\lambda_{\min})^{-\frac{k}{2}}(t+$
$\displaystyle\lambda_{\max})^{\frac{k-1}{2}}\left(\frac{3k-1}{2}\frac{\beta_{1}(Z_{i})}{\lambda_{\min}}+\frac{k}{2}\frac{\beta_{1}(Z_{j})}{\lambda_{\min}}\right)$
$\displaystyle+t^{-k}(t+2\lambda_{\max})^{\frac{2k-1}{2}}\frac{\beta_{2}^{2}(Z_{i})+\beta_{2}^{2}(Z_{j})}{\lambda_{\min}^{2}}.$

By Lemma 16, this yields

$\displaystyle m_{2k-1}(W,\Theta_{*})\leq(t+\lambda_{\min})^{-\frac{k}{2}}$
$\displaystyle(t+\lambda_{\max})^{\frac{k-1}{2}}(4k-1)\frac{\beta_{1}(Z_{1})+\beta_{1}(Z_{2})}{\lambda_{\min}}$
$\displaystyle\qquad\qquad\qquad+2t^{-k}(t+2\lambda_{\max})^{\frac{2k-1}{2}}\frac{\beta_{2}^{2}(Z_{1})+\beta_{2}^{2}(Z_{2})}{\lambda_{\min}^{2}}.$

Similarly for $p=2k+1$,

$\displaystyle m_{2k+1}(W,\Theta_{*})\leq(t+\lambda_{\min})^{-\frac{k}{2}}$
$\displaystyle(t+\lambda_{\max})^{\frac{k+1}{2}}(4k+1)\frac{\beta_{1}(Z_{1})+\beta_{1}(Z_{2})}{\lambda_{\min}}$
$\displaystyle\qquad\qquad\qquad+2t^{-k}(t+2\lambda_{\max})^{\frac{2k+1}{2}}\frac{\beta_{2}^{2}(Z_{1})+\beta_{2}^{2}(Z_{2})}{\lambda_{\min}^{2}}.$

By the definition of
$M(W,\Theta_{*})=\sqrt{m_{2k-1}(W,\Theta_{*})m_{2k+1}(W,\Theta_{*})}$, we
obtain

$\displaystyle M(W,\Theta_{*})\leq
2^{\frac{1}{4}}\left(\frac{\lambda_{\max}}{\lambda_{\min}}\right)^{\frac{k}{2}}\left[4k\frac{\beta_{1}(Z_{1})+\beta_{1}(Z_{2})}{\lambda_{\min}}+2\left(1+\frac{2\lambda_{\max}}{t}\right)^{k}\frac{\beta_{2}^{2}(Z_{1})+\beta_{2}^{2}(Z_{2})}{\lambda_{\min}^{2}}\right].$

where we have used the fact that the geometric mean is upper bounded by the
arithmetic mean.

∎

The derivation is concluded by adapting Lemma 12 of [32] to our notation and
setting.

######  Lemma 19.

Let $\mathcal{E}\subseteq\mathbb{R}^{d_{x}+d_{y}}$ be measurable. Then

$\displaystyle\mathsf{I}(W;\Theta_{*})\leq\frac{k}{2}\log\left(1+\frac{\lambda}{t}\right)\left(\mu_{Z}(\mathcal{E}^{c})+\frac{\alpha(X)+\alpha(Y)}{\lambda}\right)+\mu_{Z}(\mathcal{E})\mathsf{I}(W;\Theta_{*}|Z\in\mathcal{E}).$

###### Proof.

Letting $U=\mathds{1}_{\mathcal{E}}(Z)$, the MI chain rule gives

$\mathsf{I}(W,U;\Theta_{*})=\mathsf{I}(W;\Theta_{*})+\mathsf{I}(U;\Theta_{*}|W)=\mathsf{I}(W;\Theta_{*}|U)+\mathsf{I}(U;\Theta_{*}).$

Since $\mathsf{I}(U;\Theta_{*})=0$, we have
$\mathsf{I}(W;\Theta_{*})\leq\mathsf{I}(W;\Theta_{*}|U)$, and expanding the
conditioning yields

$\mathsf{I}(W;\Theta_{*})\leq\mu_{Z}(\mathcal{E}^{c})\mathsf{I}(W;\Theta_{*}|Z\notin\mathcal{E})+\mu_{Z}(\mathcal{E})\mathsf{I}(W;\Theta_{*}|Z\in\mathcal{E}).$

Recall that $W=\Theta_{*}^{\intercal}Z+\sqrt{t}N$, where $\Theta_{*}$ and $N$
are independent Gaussians. Therefore, conditioned on $Z$, $(W,\Theta_{*})$ is
jointly Gaussian and we have

$\displaystyle\mathsf{I}(W;\Theta_{*}|Z\notin\mathcal{E})$
$\displaystyle\leq\mathsf{I}(W;\Theta_{*}|Z,Z\notin\mathcal{E})$
$\displaystyle=\mathsf{h}(W|Z,Z\notin\mathcal{E})-\mathsf{h}(W|\Theta_{*},Z,Z\notin\mathcal{E})$
$\displaystyle=\frac{k}{2}\mathbb{E}\left[\left.\log\left(\frac{t+\frac{1}{d_{x}}\|X\|^{2}}{t}\right)+\log\left(\frac{t+\frac{1}{d_{y}}\|Y\|^{2}}{t}\right)\right|Z\notin{\mathcal{E}}\right]$
$\displaystyle=\frac{k}{2\mu_{Z}(\mathcal{E}^{c})}\left(\mspace{-2.0mu}\mathbb{E}\left[\left(\log\left(\frac{t\mspace{-2.0mu}+\mspace{-2.0mu}\frac{1}{d_{x}}\|X\|^{2}}{t+\lambda}\right)\mspace{-2.0mu}+\mspace{-2.0mu}\log\left(\frac{t\mspace{-2.0mu}+\mspace{-2.0mu}\frac{1}{d_{y}}\|Y\|^{2}}{t+\lambda}\right)\right)\mspace{-2.0mu}{\mathds{1}_{\mathcal{E}^{c}}}(Z)\right]\mspace{-2.0mu}+\mspace{-2.0mu}\log\left(1\mspace{-2.0mu}+\mspace{-2.0mu}\frac{\lambda}{t}\right)\mu_{Z}(\mathcal{E}^{c})\right)$
$\displaystyle\leq\frac{k}{2\mu_{Z}(\mathcal{E}^{c})}\log\left(1+\frac{\lambda}{t}\right)\left(\mu_{Z}(\mathcal{E}^{c})+\frac{\alpha(X)+\alpha(Y)}{\lambda}\right),$

where the last inequality follows from Lemma 19 of [32]. Combining expressions
yields the lemma. ∎

To use the bound on $M(W,\Theta_{*})$ from Lemma 18, we therefore let

$\mathcal{E}_{i}=\left\\{w_{i}\in\mathbb{R}^{d_{i}}:\left|\frac{1}{d_{i}}\|z_{i}\|^{2}-\lambda\right|\leq\frac{\epsilon}{2}\lambda\right\\},$

where $\epsilon\in(0,1]$. Markov’s inequality implies
$\mu_{Z_{i}}(\mathcal{E}_{i}^{c})\leq\frac{2}{\epsilon\lambda}\alpha(Z_{i})$.
Define $\mathcal{E}=\mathcal{E}_{1}\times\mathcal{E}_{2}$, so that by the
union bound
$\mu_{Z}(\mathcal{E}^{c})\leq\mu_{Z_{1}}(\mathcal{E}_{1}^{c})+\mu_{Z_{2}}(\mathcal{E}_{2}^{c})\leq\frac{4}{\epsilon\lambda}\alpha(Z_{1})\vee\alpha(Z_{2})$.
Let $Z^{\prime}$ be drawn according to the conditional distribution of $Z$
given $Z\in\mathcal{E}$, and set
$W^{\prime}=\Theta_{*}^{\intercal}Z^{\prime}+\sqrt{t}N$. By Lemma 18, we have

$M(W^{\prime},\Theta_{*})\leq
2^{\frac{1}{4}}\left(\frac{1+\epsilon}{1-\epsilon}\right)^{\frac{k}{2}}\left[4k\frac{\beta_{1}(Z^{\prime}_{1})+\beta_{1}(Z^{\prime}_{2})}{\lambda}+2\left(1+\frac{2(1+\epsilon)\lambda}{t}\right)^{k}\frac{\beta_{2}^{2}(Z^{\prime}_{1})+\beta_{2}^{2}(Z^{\prime}_{2})}{\lambda^{2}}\right].$

Hence

$\displaystyle\mathsf{I}(W;\Theta_{*}|Z\in\mathcal{E})$
$\displaystyle\leq\frac{\kappa}{2}(6\sqrt{2}k\pi)^{\frac{1}{4}}\left(\frac{1+\epsilon}{1-\epsilon}\right)^{\frac{k}{4}}\left[2\sqrt{2k}\sqrt{\frac{\beta_{1}(Z^{\prime}_{1})\vee\beta_{1}(Z^{\prime}_{2})}{\lambda}}\right.$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\ \
+\left.2\left(1+\frac{2(1+\epsilon)\lambda}{t}\right)^{\frac{k}{2}}\frac{\beta_{2}(Z^{\prime}_{1})\vee\beta_{2}(Z^{\prime}_{2})}{\lambda}\right],$

and applying Lemma 19, while noting that
$\beta^{r}_{r}(Z_{i}^{\prime})\leq\frac{\beta^{r}_{r}(Z_{i})}{\mu_{Z_{i}}^{2}(\mathcal{E}_{i})}\leq\frac{\beta^{r}_{r}(Z_{i})}{\mu_{Z}^{2}(\mathcal{E})}$,
for $i=1,2$ and $r=1,2$, yields the result of Lemma 11.