q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\widehat{\bm{\beta}}_{\mathcal{M}^{c}}}+\lambda({\bf
A}^{\rm T}\widehat{{\bf z}})+{\bf A}_{1}^{\rm T}{\bf C}^{\rm
T}\bm{\mu}_{4}=\bf 0.$ (A.34)

There exist $\delta\in[4R_{1}\tau_{1}\sqrt{\hbox{log}(p)/n}/\lambda,1]$ so
that $\|({\bf A}^{\rm T}\widehat{{\bf z}})_{(\mathcal{M}\cup
S)^{c}}\|_{\infty}\leq 1-\delta$, where we name $({\bf A}^{\rm T}\widehat{{\bf
z}})$ as extended sub-gradient. In addition,

$\displaystyle
n\geq\max[\hbox{log}(p)\tau_{1}^{2}(m+k)^{2}/(\alpha_{1}-\mu)^{2},4\hbox{log}(p)\tau_{1}^{2}\\{t_{1}+(2/\delta+1/2)\sqrt{k}\\}^{4}/(\alpha_{1}-\mu)^{2}],~{}~{}\alpha_{1}>\mu$

and $\bm{\beta}_{\mathcal{M}^{c}}$ is $k$ sparse.

Under the above conditions and the conditions (a) and (b), (7) has a unique
local minimizer $\widehat{\bm{\beta}}$.

Proof: We follow the primal-dual witness construction introduced in Wainwright
(2009).

Step i: Following Lemma B.10 in the supplementary material, let
$\widehat{\bm{\beta}}_{\mathcal{M}\cup S}$ be the minimizer for (14). We can
easily check that when replacing $\bm{\beta}$ by $\bm{\beta}_{\mathcal{M}\cup
S}$, ${\bf X}$ by ${\bf X}_{\mathcal{M}\cup S}$, and $p$ by $m+k$, all the
conditions of Theorem 1 are still satisfied. Here to check the condition
regarding $\alpha_{1}$, we note that for any $\bm{\beta}$,

$\displaystyle\min_{\|\bm{\beta}\|_{2}\leq R_{2},\|\bm{\beta}\|_{1}\leq
R_{1}}\alpha_{\min}[E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf X}_{i}{\bf
X}_{i}^{\rm T}\\}]$ $\displaystyle=$
$\displaystyle\min_{\|\bm{\beta}\|_{2}\leq R_{2},\|\bm{\beta}\|_{1}\leq
R_{1}}\inf_{\|{\bf v}\|_{2}=1,{\bf v}\in\mathbb{R}^{p}}{\bf v}^{\rm
T}E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf
v}$ $\displaystyle\leq$ $\displaystyle\min_{\|\bm{\beta}\|_{2}\leq
R_{2},\|\bm{\beta}\|_{1}\leq R_{1}}\inf_{\|{\bf v}\|_{2}=1,{\bf
v}\in\mathbb{R}^{m+k}}({\bf v}^{\rm T},{\bf 0}^{\rm
T})E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm
T}\\}({\bf v}^{\rm T},{\bf 0}^{\rm T})^{\rm T}$ $\displaystyle=$
$\displaystyle\min_{\|\bm{\beta}\|_{2}\leq R_{2},\|\bm{\beta}\|_{1}\leq
R_{1}}\alpha_{\min}[E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf
X}_{i,\mathcal{M}\cup S}{\bf X}_{i,\mathcal{M}\cup S}^{\rm T}\\}]$
$\displaystyle=$ $\displaystyle\min_{\|\bm{\beta}_{\mathcal{M}\cup
S}\|_{2}\leq R_{2},\|\bm{\beta}_{\mathcal{M}\cup S}\|_{1}\leq
R_{1}}\alpha_{\min}[E\\{\exp(\bm{\beta}_{\mathcal{M}\cup S}^{\rm T}{\bf
X}_{i,\mathcal{M}\cup S}){\bf X}_{i,\mathcal{M}\cup S}{\bf
X}_{i,\mathcal{M}\cup S}^{\rm T}\\}].$

Therefore, Theorem 1 applied to the $m+k$ dimensional case leads to

$\displaystyle\|\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\widecheck{\bm{\beta}}_{\mathcal{M}\cup S}\|_{1}\leq(4\sqrt{k}+t_{1})t\leq
R_{1}/2,$

and

$\displaystyle\|\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\widecheck{\bm{\beta}}_{{\mathcal{M}\cup S}}\|_{2}\leq t\leq R_{2}/2.$

Therefore

$\displaystyle\|\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}\|_{1}\leq\|\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\widecheck{\bm{\beta}}_{{\mathcal{M}\cup
S}}\|_{1}+\|\widecheck{\bm{\beta}}\|_{1}<R_{1}$

and

$\displaystyle\|\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}\|_{2}\leq\|\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\widecheck{\bm{\beta}}_{{\mathcal{M}\cup
S}}\|_{2}+\|\widecheck{\bm{\beta}}\|_{2}<R_{2}.$

Hence $\widehat{\bm{\beta}}_{\mathcal{M}\cup S}$ and $\widehat{\bm{\beta}}$
must be in the interior of the feasible region.

Step ii: We show that $\widehat{\bm{\beta}}$ is a local minimum for (7) by
verifying the conditions in Lemma B.12 in the supplementary material. Because

$\displaystyle\mathcal{L}(\bm{\beta})+\rho_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})$
$\displaystyle=$
$\displaystyle\mathcal{L}(\bm{\beta})-q_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})+\lambda\|\bm{\beta}_{\mathcal{M}^{c}}\|_{1},$

we can write $f=\mathcal{L}$, $g=q_{\lambda}$, and $({\bf x}^{*},{\bf
v}^{*},{\bf w}_{1}^{*},{\bf
w}_{2}^{*},\mu_{1}^{*},\mu_{2}^{*},\bm{\mu}_{3}^{*})=(\widehat{\bm{\beta}},\widehat{{\bf
z}},{\bf A}^{\rm T}\widehat{{\bf z}},\widehat{\bf z}_{1},0,0,\bm{\mu}_{4})$,
where ${\bf z}_{1}\in\partial\|\widehat{\bm{\beta}}\|_{2}$. Lemma B.4 in the
supplementary material ensures the concavity and differentiability of $g({\bf
x})-\mu/2\|{\bf x}\|_{2}^{2}$. Further, since $\mu_{1}^{*}=\mu_{2}^{*}=0$,
(B.2) and (B.3) are satisfied. (B.4) is satisfied by our construction in
(A.34). Therefore, it remains to verify (B.5). We first show that
$G^{*}\subseteq{\mathbb{R}}^{\mathcal{M}\cup S}$ so that (B.5) only needs to
be satisfies for the vectors belong to ${\mathbb{R}}^{\mathcal{M}\cup S}$.
Suppose this does not hold, let $\bm{\nu}\in G^{*}$ such that
$\hbox{supp}(\bm{\nu})\not\subseteq\mathcal{M}\cup S$. This implies there is
an index $j\in{(\mathcal{M}\cup S)^{c}}$ such that $\nu_{j}\neq 0$.

Now we define ${\bf A}^{\rm T}{{\bf z}}^{\prime}$ such that $({\bf A}^{\rm
T}{{\bf z}}^{\prime})_{k}=({\bf A}^{\rm T}\widehat{{\bf z}})_{k}$ for $k\neq
j$, and $({\bf A}^{\rm T}{{\bf z}}^{\prime})_{j}={\rm sign}(\nu_{j})$, where
$a_{k}$ is the $k$th element in vector ${\bf a}$. Clearly ${{\bf
z}}^{\prime}\in\partial\|\widehat{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}$ and

$\displaystyle\lambda\bm{\nu}^{\rm T}{\bf A}^{\rm T}{{\bf
z}}^{\prime}>\lambda\bm{\nu}^{\rm T}{\bf A}^{\rm T}\widehat{{\bf z}}$

because $\|({\bf A}^{\rm T}\widehat{{\bf z}})_{(\mathcal{M}\cup
S)^{c}}\|_{\infty}<1$. Therefore,

$\displaystyle\bm{\nu}^{\rm
T}\left[\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}}-\left\\{{\bf
A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\widehat{\bm{\beta}}_{\mathcal{M}^{c}}}\right\\}\right]+\lambda\bm{\nu}^{\rm
T}{\bf A}^{\rm T}{{\bf z}}^{\prime}+\bm{\nu}^{\rm T}{\bf A}_{1}^{\rm T}{\bf
C}^{\rm T}\bm{\mu}_{4}$ $\displaystyle>$ $\displaystyle\bm{\nu}^{\rm
T}\left[\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}}-\left\\{{\bf
A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\widehat{\bm{\beta}}_{\mathcal{M}^{c}}}\right\\}\right]+\lambda\bm{\nu}^{\rm
T}{\bf A}^{\rm T}\widehat{{\bf z}}+\bm{\nu}{\bf A}_{1}^{\rm T}{\bf C}^{\rm
T}\bm{\mu}_{4}=0.$

Now because $\bm{\nu}\in G^{*}$, we have

$\displaystyle\bm{\nu}{\bf A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{4}=0.$

This implies

$\displaystyle\bm{\nu}^{\rm
T}\left[\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}}-\left\\{{\bf
A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\widehat{\bm{\beta}}_{\mathcal{M}^{c}}}\right\\}\right]+\lambda\bm{\nu}^{\rm
T}{\bf A}^{\rm T}{{\bf z}}^{\prime}>0$

which contradicts with the requirement of $G^{*}$ that

$\displaystyle\sup_{{\bf
v}\in\partial\|\widehat{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}}\bm{\nu}^{\rm
T}\left[\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}}-\left\\{{\bf
A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\widehat{\bm{\beta}}_{\mathcal{M}^{c}}}\right\\}\right]+\lambda\bm{\nu}^{\rm
T}{\bf A}^{\rm T}{\bf v}={0}.$

Therefore, $G^{*}\subseteq\mathbb{R}^{\mathcal{M}\cup S}$. Now by construction
$\hbox{supp}(\widehat{\bm{\beta}})\subset\mathcal{M}\cup S$, using Lemma B.10
in the supplementary material, we conclude that (B.5) holds with $\kappa=\mu$.
Hence, all conditions of Lemma B.12 in the supplementary material are
satisfied, so we conclude $\widehat{\bm{\beta}}$ is an isolated local minimum
of (7).

Now by Lemma B.13, because
$\hbox{supp}(\widehat{\bm{\beta}})\subseteq\mathcal{M}\cup S$ and
$\widehat{\bm{\beta}}$ is an interior minimizer, the support of any stationary
point of (7) is a subset of $\mathcal{M}\cup S$. Hence, we can write any
stationary point in the form of
$\widetilde{\bm{\beta}}=\\{\widetilde{\bm{\beta}}_{\mathcal{M}\cup S}^{\rm
T},{\bf 0}_{p-m-k}^{\rm T}\\}^{\rm T}$, where
$\widetilde{\bm{\beta}}_{\mathcal{M}\cup S}$ is a stationary point for (14).
Further, note that (14) is strictly convex by Lemma B.10, and hence
$\widetilde{\bm{\beta}}_{\mathcal{M}\cup S}$ is unique in the feasible set and
therefore $\widehat{\bm{\beta}}_{\mathcal{M}\cup S}$ and
$\widetilde{\bm{\beta}}$ are unique. Hence
$\widehat{\bm{\beta}}=\widetilde{\bm{\beta}}$ is the unique local minimum.
This proves the result. ∎

####  A.3.2 Supporting Theorem of Theorem 4 and its Proof

######  Theorem A.2.

Let $\alpha_{1}=\min_{\|\bm{\beta}\|_{1}\leq R_{1},\|\bm{\beta}\|_{2}\leq
R_{2}}\alpha_{\min}[E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf X}_{i}{\bf
X}_{i}^{\rm T}\\}]/2$. Assume $\rho_{\lambda}$ is $\mu$-amenable, for
$\mu<\alpha_{1}$, and Conditions (C1) – (C6) in the supplementary material
hold. Define ${\bf A}=\\{{\bf 0}_{(p-m)\times m},{\bf
I}_{(p-m)\times(p-m)}\\}$ and ${\bf A}_{1}=\\{{\bf I}_{m\times m},{\bf
0}_{m\times(p-m)}\\}$. Further assume that

(a) The parameters $\lambda,R_{1},R_{2}$ satisfy

$\displaystyle
4\max\left\\{\|\partial\mathcal{L}(\bm{\beta}_{t})/\partial\bm{\beta}\|_{\infty},\alpha_{1}(\hbox{log}(p)/n)^{1/4}\right\\}\leq\lambda\leq\left[\alpha_{1}\left\\{\vphantom{\int_{0}^{1}}16(4\sqrt{k}+\sqrt{m})\frac{6\sqrt{k}+\sqrt{m})}{4\alpha_{1}-3\mu}\right\\}^{-1}\right]^{1/2},$
$\displaystyle\max\left\\{\vphantom{\int_{0}^{1}}2\|\bm{\beta}_{t}\|_{1},2(4\sqrt{k}+\sqrt{m})\lambda\frac{6\sqrt{k}+2\sqrt{m}}{4\alpha_{1}-3\mu}\right\\}\leq
R_{1}$
$\displaystyle\leq\min\left(\frac{\alpha_{1}}{8\lambda},[n\alpha_{1}^{2}/\\{64\tau_{1}^{2}\hbox{log}(p)\\}]^{1/4},[n\alpha_{1}^{4}/\\{16\tau_{1}^{4}\hbox{log}(p)\\}]^{1/4}\right).$

with

$\displaystyle\|\bm{\beta}_{t}\|_{1}\neq(4\sqrt{k}+\sqrt{m})\frac{6\lambda\sqrt{k}+2\lambda\sqrt{m}}{4\alpha_{1}-3\mu}$

and

$\displaystyle\max\left\\{2\|\bm{\beta}_{t}\|_{2},2\frac{6\lambda\sqrt{k}+2\lambda\sqrt{m}}{4\alpha_{1}-3\mu}\right\\}\leq
R_{2},$

with

$\displaystyle\|\bm{\beta}_{t}\|_{2}\neq\frac{6\lambda\sqrt{k}+2\lambda\sqrt{m}}{4\alpha_{1}-3\mu}.$

(b) Let $\widehat{\bm{\beta}}_{a\mathcal{M}\cup S}$ be the minimizer for (15),
$\widehat{\bm{\beta}}_{a}=(\widehat{\bm{\beta}}_{a\mathcal{M}\cup S}^{\rm
T},{\bf 0}_{p-m-k}^{\rm T})^{\rm T}$, $\widehat{{\bf z}}$ satisfy

$\displaystyle\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}}_{a})}{\partial\bm{\beta}_{a}}\right\\}-\left\\{{\bf
A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{a\mathcal{M}^{c}})}{\partial\widehat{\bm{\beta}}_{a\mathcal{M}^{c}}}\right\\}+\lambda({\bf
A}^{\rm T}\widehat{{\bf z}})=\bf 0.$ (A.35)

There exist $\delta\in[4R_{1}\tau_{1}\sqrt{\hbox{log}(p)/n}/\lambda,1]$ so
that $\|({\bf A}^{\rm T}\widehat{{\bf z}})_{(\mathcal{M}\cup
S)^{c}}\|_{\infty}\leq 1-\delta$, where we name $({\bf A}^{\rm T}\widehat{{\bf
z}})$ as extended sub-gradient. In addition,

$\displaystyle n$ $\displaystyle\geq$
$\displaystyle\max[\hbox{log}(p)\tau_{1}^{2}(m+k)^{2}/(\alpha_{1}-\mu)^{2},$
$\displaystyle
4\hbox{log}(p)\tau_{1}^{2}\\{\sqrt{m}+(2/\delta+1/2)\sqrt{k}\\}^{4}/(\alpha_{1}-\mu)^{2}],~{}~{}\alpha_{1}>\mu$

and $\bm{\beta}_{\mathcal{M}^{c}}$ is $k$ spar se. Under the above conditions
and the conditions (a) and (b), (8) has a unique local minimizer
$\widehat{\bm{\beta}}_{a}$.

Proof: The proof follows the same argument as those lead to Theorem A.1 hence
is omitted.

####  A.3.3 Definition needed to prove Theorems 3 and 4

Let $\bm{\beta}^{*}$ is the point on the line connecting
$\widehat{\bm{\beta}}$ and $\widecheck{\bm{\beta}}$. We write

$\displaystyle\widehat{{\bf
Q}}(\bm{\beta}^{*})=\left\\{\begin{array}[]{cc}\widehat{{\bf
Q}}_{(\mathcal{M}\cup S)(\mathcal{M}\cup S)}(\bm{\beta}^{*})&\widehat{{\bf
Q}}_{(\mathcal{M}\cup S)(\mathcal{M}\cup S)^{c}}(\bm{\beta}^{*})\\\
\widehat{{\bf Q}}_{(\mathcal{M}\cup S)^{c}(\mathcal{M}\cup
S)}(\bm{\beta}^{*})&\widehat{{\bf Q}}_{(\mathcal{M}\cup S)^{c}(\mathcal{M}\cup
S)^{c}}(\bm{\beta}^{*})\end{array}\right\\}$ (A.38)

Then by the construction in (A.34), let
$\widehat{\bm{\beta}}=(\widehat{\bm{\beta}}_{\mathcal{M}\cup S}^{\rm T},{\bf
0}_{(\mathcal{M}\cup S)^{c}}^{\rm T})^{\rm T}$,
$\widehat{\bm{\beta}}_{\mathcal{M}\cup S}$ is the minimizer for (14), then we
have

$\displaystyle\widehat{{\bf
Q}}(\bm{\beta}^{*})(\widehat{\bm{\beta}}-\widecheck{\bm{\beta}})+\left[\begin{array}[]{l}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}-\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}\\\
\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{(\mathcal{M}\cup
S)^{c}}-\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{(\mathcal{M}\cup
S)^{c}}\end{array}\right]$
$\displaystyle+\lambda\left\\{\begin{array}[]{l}({\bf A}^{\rm T}\widehat{{\bf
z}})_{\mathcal{M}\cup S}\\\ ({\bf A}^{\rm T}\widehat{{\bf
z}})_{(\mathcal{M}\cup
S)^{c}}\end{array}\right\\}+\left\\{\begin{array}[]{c}({\bf A}_{1}^{\rm T}{\bf
C}^{\rm T}\bm{\mu}_{3}^{*})_{\mathcal{M}\cup S}\\\ \bf 0_{(\mathcal{M}\cup
S)^{c}}\end{array}\right\\}={\bf 0}.$

Taking the upper $m+k$ non-zero component, we get

$\displaystyle\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\widecheck{\bm{\beta}}_{\mathcal{M}\cup S}$ $\displaystyle=$
$\displaystyle\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\left[-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}+\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}\right.$ (A.41) $\displaystyle\left.-\lambda({\bf A}^{\rm T}\widehat{{\bf
z}})_{\mathcal{M}\cup S}-({\bf A}_{1}^{\rm T}{\bf C}^{\rm
T}\bm{\mu}_{3}^{*})_{\mathcal{M}\cup S}\vphantom{\int_{0}^{1}}\right],$

while taking the lower $p-m-k$ components, this leads to

$\displaystyle({\bf A}^{\rm T}\widehat{{\bf z}})_{(\mathcal{M}\cup S)^{c}}$
$\displaystyle=$ $\displaystyle\lambda^{-1}\left[\left\\{{\bf A}^{\rm
T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{(\mathcal{M}\cup
S)^{c}}-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{(\mathcal{M}\cup
S)^{c}}\right]$ $\displaystyle-\lambda^{-1}\widehat{{\bf Q}}_{(\mathcal{M}\cup
S)^{c}(\mathcal{M}\cup
S)}(\bm{\beta}^{*})(\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\bm{\beta}_{\mathcal{M}\cup S})$ $\displaystyle=$
$\displaystyle\lambda^{-1}\left[\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{(\mathcal{M}\cup
S)^{c}}-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{(\mathcal{M}\cup
S)^{c}}\right]$ $\displaystyle+\lambda^{-1}\widehat{{\bf Q}}_{(\mathcal{M}\cup
S)^{c}(\mathcal{M}\cup S)}(\bm{\beta}^{*})\\{\widehat{{\bf
Q}}_{(\mathcal{M}\cup S)(\mathcal{M}\cup S)}(\bm{\beta}^{*})\\}^{-1}$
$\displaystyle\times\left(\left[\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}-\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}\right]+\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup S}+({\bf
A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{3}^{*})_{\mathcal{M}\cup S}\right).$

Further by Condition (A4), we have

$\displaystyle\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{(\mathcal{M}\cup
S)^{c}}$ $\displaystyle=$
$\displaystyle\left[\left\\{\frac{\partial\lambda|\widehat{\beta}_{j}|}{\partial\beta_{j}}\right\\}-\left\\{\frac{\partial\rho_{\lambda}(\widehat{\beta}_{j})}{\partial\beta_{j}}\right\\},j\in(\mathcal{M}\cup
S)^{c}\right]^{\rm T}$ $\displaystyle=$ $\displaystyle\bf 0.$

Therefore, we have

$\displaystyle({\bf A}^{\rm T}\widehat{{\bf z}})_{(\mathcal{M}\cup S)^{c}}$
$\displaystyle=$
$\displaystyle\lambda^{-1}\left[-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{(\mathcal{M}\cup
S)^{c}}\right]+\lambda^{-1}\widehat{{\bf Q}}_{(\mathcal{M}\cup
S)^{c}(\mathcal{M}\cup S)}(\bm{\beta}^{*})\\{\widehat{{\bf
Q}}_{(\mathcal{M}\cup S)(\mathcal{M}\cup S)}(\bm{\beta}^{*})\\}^{-1}$
$\displaystyle\times\left(\left[\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}-\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}\right]+\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup S}+({\bf
A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{3}^{*})_{\mathcal{M}\cup S}\right).$

By the condition that ${\bf C}\widehat{\bm{\beta}}_{\mathcal{M}}={\bf
C}\widecheck{\bm{\beta}}={\bf t}$, from (A.41), we have

$\displaystyle\bf 0$ $\displaystyle=$ $\displaystyle{\bf C}[{\bf I}_{m\times
m},{\bf 0}_{m\times k}]\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\left[-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}+\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}-\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup S}\right]$
$\displaystyle+{\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times
k}]\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]^{\rm
T}{\bf C}^{\rm T}\bm{\mu}_{3}^{*},$

which leads to

$\displaystyle\bm{\mu}_{3}^{*}$ $\displaystyle=$ $\displaystyle({\bf C}[{\bf
I}_{m\times m},{\bf 0}_{m\times k}]\\{\widehat{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1}[{\bf I}_{m\times m},{\bf
0}_{m\times k}]^{\rm T}{\bf C}^{\rm T})^{-1}{\bf C}[{\bf I}_{m\times m},{\bf
0}_{m\times k}]\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}$
$\displaystyle\times\left[-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}+\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\widecheck{\bm{\beta}}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}-\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup S}\right]$

and

$\displaystyle({\bf A}_{1}{\bf C}^{\rm T}\bm{\mu}_{3}^{*})_{\mathcal{M}\cup
S}$ $\displaystyle=$ $\displaystyle{\bf A}_{2}\\{\widehat{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\left[-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}+\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}-\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup S}\right],$

where

$\displaystyle{\bf A}_{2}=[{\bf I}_{m\times m},{\bf 0}_{m\times k}]^{\rm
T}{\bf C}^{\rm T}({\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times
k}]\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]^{\rm
T}{\bf C}^{\rm T})^{-1}{\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}].$

Hence, to use Theorem A.1, we must show that $\|({\bf A}^{\rm T}\widehat{{\bf
z}})_{(\mathcal{M}\cup S)^{c}}\|_{\infty}<1$. Define ${{\bf
Q}}(\bm{\beta})=E\\{\exp(\bm{\beta}^{\rm T}{\bf X}){\bf X}{\bf X}^{\rm T}\\}$,
and

$\displaystyle{\bf A}_{2}^{*}=[{\bf I}_{m\times m},{\bf 0}_{m\times k}]^{\rm
T}{\bf C}^{\rm T}({\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]\\{{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}(\bm{\beta})\\}^{-1}[{\bf I}_{m\times
m},{\bf 0}_{m\times k}]^{\rm T}{\bf C}^{\rm T})^{-1}{\bf C}[{\bf I}_{m\times
m},{\bf 0}_{m\times k}].$

####  A.3.4 Proof of Theorem 3

First of all, $\widehat{\bm{\beta}}_{(\mathcal{M}\cup S)^{c}}=\bf 0$ by
construction. For any unit vector, recall that

$\displaystyle{\bf v}^{\rm
T}\frac{\partial^{2}\mathcal{L}(\bm{\beta})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}{\bf w}=\exp(\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta}/2){\bf v}^{\rm T}\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}-\bm{\Omega}\\}{\bf w}$

Denoting $\nabla^{3}\mathcal{L}(\bm{\beta})$ to be the third order gradient of
$\mathcal{L}$, we have

$\displaystyle{\bf v}^{\rm T}\nabla^{3}\mathcal{L}(\bm{\beta}){\bf w}$
$\displaystyle=$ $\displaystyle n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{\rm
T}{\bf W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta}/2){\bf v}^{\rm
T}\\{({\bf W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}-\bm{\Omega}\\}{\bf
w}\\{{\bf W}_{i}-\bm{\Omega}\bm{\beta}\\}^{\rm T}$
$\displaystyle-n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta}/2)\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}){\bf w}^{\rm T}\bm{\Omega}+{\bf w}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}){\bf v}^{\rm T}\bm{\Omega}\\}.$

Hence define vectors ${\bf v},{\bf w}$ such that their $j$th element
$|v_{j}|>0,|w_{j}|>0$ for $j\in\mathcal{M}\cup S$, and $|v_{j}|=|w_{j}|=0$ for
$j\notin\mathcal{M}\cup S$, and $\|{\bf v}\|_{2}=\|{\bf w}\|_{2}=1$. Firstly
by Theorem 1 and the condition that $\|{\bf
h}_{n}\|_{2}=O\\{\sqrt{\max(m+k-r,r)/n}\\}$, we have

$\displaystyle\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2}$ $\displaystyle=$
$\displaystyle\|\widehat{\bm{\beta}}-\widecheck{\bm{\beta}}\|_{2}+\|\widecheck{\bm{\beta}}-\bm{\beta}_{t}\|_{2}$
$\displaystyle\leq$ $\displaystyle
C_{1}\lambda\max(\sqrt{r},\sqrt{m-r},\sqrt{k})+\sqrt{(m+k)/n}$
$\displaystyle\leq$ $\displaystyle
C_{2}\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}\sqrt{m+k}$

for some constants $C_{1},C_{2}>0$. Further recall that
${\mathbb{K}}\equiv\\{{\bf v}\in{\mathbb{R}}^{\mathcal{M}\cup S}:\|{\bf
v}\|_{2}\leq 1\\}$.

$\displaystyle\sup_{{\bf v},{\bf w}\in{\mathbb{K}}}\bigg{|}{\bf v}^{\rm
T}\left\\{\widehat{{\bf
Q}}(\bm{\beta}^{*})-\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}\right\\}{\bf w}\bigg{|}$ (A.44) $\displaystyle\leq$
$\displaystyle\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}\left[\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}^{\rm
T}\\{({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}-\bm{\Omega}\\}{\bf
w}\\{{\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}\\}^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}\right.$
$\displaystyle\left.+\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf w}^{\rm T}\bm{\Omega}+{\bf w}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf v}^{\rm
T}\bm{\Omega}\\}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}\right]$
$\displaystyle\leq$ $\displaystyle\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf w}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})$ $\displaystyle-{\bf v}^{\rm
T}\bm{\Omega}{\bf w}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\\}\bigg{|}$ $\displaystyle+\sup_{{\bf
v},{\bf w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf w}^{\rm T}\bm{\Omega}+{\bf
w}^{\rm T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf v}^{\rm
T}\bm{\Omega}\\}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$
$\displaystyle\leq$ $\displaystyle\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}\bigg{|}\frac{n^{-1}}{2}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\frac{({\bf
v}+{\bf w})^{\rm T}}{\sqrt{2}}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes
2}\frac{{\bf v}+{\bf w}}{\sqrt{2}}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v},{\bf
w}\in{\mathbb{K}}}\bigg{|}\frac{n^{-1}}{2}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\frac{({\bf
v}-{\bf w})^{\rm T}}{\sqrt{2}}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes
2}\frac{{\bf v}-{\bf w}}{\sqrt{2}}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v},{\bf w}\in{\mathbb{K}}}\bigg{|}\frac{({\bf v}+{\bf w})^{\rm
T}}{\sqrt{2}}\bm{\Omega}\frac{{\bf v}+{\bf
w}}{\sqrt{2}}\frac{n^{-1}}{2}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v},{\bf w}\in{\mathbb{K}}}\bigg{|}\frac{({\bf v}-{\bf w})^{\rm
T}}{\sqrt{2}}\bm{\Omega}\frac{{\bf v}-{\bf
w}}{\sqrt{2}}\frac{n^{-1}}{2}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v},{\bf w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf w}^{\rm T}\bm{\Omega}+{\bf
w}^{\rm T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf v}^{\rm
T}\bm{\Omega}\\}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$
$\displaystyle\leq$ $\displaystyle\sup_{{\bf
v}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v}\in{\mathbb{K}}}\bigg{|}{\bf v}^{\rm T}\bm{\Omega}{\bf
v}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf W}_{i}-\bm{\beta}^{*\rm
T}\bm{\Omega}\bm{\beta}^{*}/2)({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v},{\bf w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf w}^{\rm T}\bm{\Omega}+{\bf
w}^{\rm T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf v}^{\rm
T}\bm{\Omega}\\}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$
$\displaystyle\leq$ $\displaystyle\sup_{{\bf
v}\in{\mathbb{K}}}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}|({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})|$ $\displaystyle+\sup_{{\bf
v}\in{\mathbb{K}}}{\bf v}^{\rm T}\bm{\Omega}{\bf
v}\bigg{\|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})\bigg{\|}_{2}\|(\widehat{\bm{\beta}}-\bm{\beta}_{t})\|_{2}$
$\displaystyle+2\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})\\{{\bf w}^{\rm
T}\bm{\Omega}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\\}\bigg{|}.$

Now for (A.44), because by Condition (C1) in the supplementary material, we
have

$\displaystyle|({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})|$ $\displaystyle=$
$\displaystyle|({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})/\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2}|\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2}$
$\displaystyle\leq$ $\displaystyle
M_{W}\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2}+\|\bm{\Omega}\|_{2}\|\bm{\beta}^{*}\|_{2}\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2}$
$\displaystyle\leq$ $\displaystyle
2M_{W}\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2}.$

The last equality holds because $\|\bm{\Omega}\|_{2}=O(1)$,
$\|\bm{\beta}^{*}\|_{2}\leq\|\bm{\beta}_{t}\|_{2}+\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2}=O_{p}(1)+O_{p}\left\\{{\hbox{log}(p)}/{n}\right\\}^{1/4}\sqrt{m+k}=o_{p}(\sqrt{m+k})$.
From Corollary B.4 in the supplementary material, we have

$\displaystyle\sup_{{\bf
v}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}\sup_{i}|({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm T}(\widehat{\bm{\beta}}-\bm{\beta})|$
$\displaystyle\vphantom{\int_{0}^{1}}-E\\{\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}\\}\sup_{i}|({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta})|\bigg{|}$ $\displaystyle\leq$
$\displaystyle
2M_{W}\sqrt{(m+k)/n}\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}\|_{2}$

with probability $1-O[\exp\\{-(m+k)\\}]$. Further since $\sup_{{\bf
v}\in{\mathbb{K}}}E\\{\exp(\bm{\beta}^{*\rm T}{\bf W}_{i}-\bm{\beta}^{*\rm
T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}\\}=O(1)$ due to Condition
C1(b), hence we get

$\displaystyle\sup_{{\bf
v}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta})\bigg{|}$ $\displaystyle\leq$
$\displaystyle\sup_{{\bf
v}\in{\mathbb{K}}}\left\\{n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}\right\\}\sup_{i}|({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm T}(\widehat{\bm{\beta}}-\bm{\beta})|$
$\displaystyle\leq$
$\displaystyle\\{2M_{W}\sqrt{(m+k)/n}+O_{p}(1)\\}\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}\|_{2}$
$\displaystyle=$ $\displaystyle
O_{p}(\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}\|_{2}).$

For (A.44), we first have

$\displaystyle\|n^{-1}\sum_{i=1}^{n}\exp(2\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*})({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}_{\mathcal{M}\cup S}$
$\displaystyle-E\\{\exp(2\bm{\beta}^{*\rm T}{\bf W}_{i}-\bm{\beta}^{*\rm
T}\bm{\Omega}\bm{\beta}^{*})({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes
2}_{\mathcal{M}\cup S}\\}\|_{2}$ $\displaystyle=$ $\displaystyle
O_{p}\\{\sqrt{(m+k)/n}\\}$

by Corollary B.3 in the supplementary material. Further
$\|E\\{\exp(2\bm{\beta}^{*\rm T}{\bf W}_{i}-\bm{\beta}^{*\rm
T}\bm{\Omega}\bm{\beta}^{*})({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes
2}_{\mathcal{M}\cup S}\\}\|_{2}=O(1)$. Hence (A.44) is of order
$O_{p}(\|\widehat{\bm{\beta}}-\bm{\beta}\|_{2})$. Now for (A.44), because

$\displaystyle\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})\bigg{|}^{2}$ $\displaystyle\leq$
$\displaystyle n^{-1}\sum_{i=1}^{n}\\{\exp(2\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v},$

by Corollary B.3 in the supplementary material, we have

$\displaystyle n^{-1}\sum_{i=1}^{n}\exp(2\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}$
$\displaystyle-E\\{\exp(2\bm{\beta}^{*\rm T}{\bf W}_{i}-\bm{\beta}^{*\rm
T}\bm{\Omega}\bm{\beta}^{*}){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}\\}=O_{p}(\sqrt{(m+k)/n})$

with probability of the order $1-O\\{\exp(m+k)\\}$. Further because
$E\\{\exp(2\bm{\beta}^{*\rm T}{\bf W}_{i}-\bm{\beta}^{*\rm
T}\bm{\Omega}\bm{\beta}^{*}){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}\\}=O(1)$, the third term
of the last line is of the order
$O_{p}(\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2})$. Therefore, we have

$\displaystyle\sup_{{\bf v},{\bf w}\in{\mathbb{K}}}\bigg{|}{\bf v}^{\rm
T}\left\\{\widehat{{\bf
Q}}(\bm{\beta}^{*})-\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}_{t}^{\rm
T}}\right\\}{\bf
w}\bigg{|}=O_{p}(\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2})$

Hence for some positive constants $C_{3}$,

$\displaystyle\sup_{{\bf v},{\bf w}\in{\mathbb{K}}}\bigg{|}{\bf v}^{\rm
T}\left\\{\widehat{{\bf
Q}}(\bm{\beta}^{*})-\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}\right\\}{\bf w}\bigg{|}\leq
C_{3}\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)$ (A.45)

with probability $1-O[\exp\left\\{-(m+k)\right\\}]$. Further, by Corollary B.2
in the supplementary material, we also have

$\displaystyle\sup_{{\bf v},{\bf w}\in{\mathbb{K}}}\bigg{|}{\bf v}^{\rm
T}\left[\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}-{\bf Q}(\bm{\beta}_{t})\right]{\bf w}\bigg{|}$ $\displaystyle=$
$\displaystyle\sup_{{\bf v},{\bf w}\in{\mathbb{K}}}\bigg{|}{\bf
v}_{\mathcal{M}\cup S}^{\rm
T}\left[\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}-{\bf Q}(\bm{\beta}_{t})\right]_{\mathcal{M}\cup S,\mathcal{M}\cup S}{\bf
w}_{\mathcal{M}\cup S}\bigg{|}$ (A.46) $\displaystyle=$ $\displaystyle
O_{p}\\{\sqrt{(m+k)/n}\\}$ $\displaystyle=$ $\displaystyle
o_{p}\left[\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)\right]$

with probability $1-2\exp\left\\{-(m+k)\right\\}$. Further, recall that
${\mathbb{K}}_{1}\equiv\\{{\bf v}\in{\mathbb{R}}^{(\mathcal{M}\cup
S)^{c}}:\|{\bf v}\|_{2}=1,\|{\bf v}\|_{0}=1\\}$.

$\displaystyle\sup_{{\bf v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}{\bf v}_{1}^{\rm T}\left\\{\widehat{{\bf
Q}}(\bm{\beta}^{*})-\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}\right\\}{\bf w}\bigg{|}$ (A.50) $\displaystyle\leq$
$\displaystyle\sup_{{\bf v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\left[\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm
T}\\{({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}-\bm{\Omega}\\}{\bf
w}\\{{\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}\\}^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}\right.$
$\displaystyle\left.+\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}_{1}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf w}^{\rm T}\bm{\Omega}+{\bf
w}^{\rm T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf v}^{\rm
T}\bm{\Omega}\\}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}\right]$
$\displaystyle\leq$ $\displaystyle\sup_{{\bf v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}_{1}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf w}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})$ $\displaystyle-{\bf v}_{1}^{\rm
T}\bm{\Omega}{\bf w}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\\}\bigg{|}$ $\displaystyle+\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}_{1}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf w}^{\rm T}\bm{\Omega}$
$\displaystyle+{\bf w}^{\rm T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf
v}_{1}^{\rm T}\bm{\Omega}\\}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$
$\displaystyle\leq$ $\displaystyle\sup_{{\bf v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf w}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm
T}\bm{\Omega}{\bf w}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}_{1}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf w}^{\rm T}\bm{\Omega}$
$\displaystyle+{\bf w}^{\rm T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf
v}_{1}^{\rm T}\bm{\Omega}\\}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$
$\displaystyle\leq$ $\displaystyle\sup_{{\bf v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}n^{-1}\sum_{i=1}^{n}\bigg{|}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf w}\bigg{|}\bigg{|}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm
T}\bm{\Omega}{\bf w}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}_{1}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf w}^{\rm T}\bm{\Omega}$
$\displaystyle+{\bf w}^{\rm T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf
v}_{1}^{\rm T}\bm{\Omega}\\}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$
$\displaystyle\leq$ $\displaystyle\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1}}\frac{1}{2}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf
v}_{1}\sup_{i}\bigg{|}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
w}\in{\mathbb{K}}}\frac{1}{2}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf w}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf w}\sup_{i}\bigg{|}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm
T}\bm{\Omega}{\bf w}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle+\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{{\bf v}_{1}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf w}^{\rm T}\bm{\Omega}$
$\displaystyle+{\bf w}^{\rm T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*}){\bf
v}_{1}^{\rm T}\bm{\Omega}\\}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}.$

For (A.50), from Corollary B.4, we have

$\displaystyle\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}_{1}\sup_{i}|({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})|$
$\displaystyle\vphantom{\int_{0}^{1}}-E\\{\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}_{1}\\}\sup_{i}|({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})|\bigg{|}$ $\displaystyle\leq$
$\displaystyle
2M_{W}\sqrt{\hbox{log}(p)/n}\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2}$

with probability $1-O[\exp\\{-\hbox{log}(p)\\}]$. Further since $\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1}}E\\{\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}_{1}\\}=O(1)$ due to
Condition C1(b), hence we get

$\displaystyle\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1}}\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}_{1}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})\bigg{|}$ $\displaystyle\leq$
$\displaystyle\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1}}\left\\{n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf
v}_{1}\right\\}\sup_{i}|({\bf W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\rm
T}(\widehat{\bm{\beta}}-\bm{\beta}_{t})|$ $\displaystyle\leq$
$\displaystyle\\{2M_{W}\sqrt{\hbox{log}(p)/n}+O_{p}(1)\\}\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2}$
$\displaystyle=$ $\displaystyle
O_{p}(\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}\|_{2}).$

For (A.50), we use the same argument as those lead to the order of (A.44), we
have (A.50) is of order
$O_{p}(\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2})$. For (A.50), we
use the same argument as those lead to the order of (A.44), we have (A.50) is
of order $O_{p}(\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2})$.

Now for (A.50), because

$\displaystyle\bigg{|}n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2){\bf v}_{1}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})\bigg{|}^{2}$ $\displaystyle\leq$
$\displaystyle n^{-1}\sum_{i=1}^{n}\\{\exp(2\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}){\bf v}_{1}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}_{1},$

by Corollary B.3 in the supplementary material, we have

$\displaystyle n^{-1}\sum_{i=1}^{n}\exp(2\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}){\bf v}_{1}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}_{1}$
$\displaystyle-E\\{\exp(2\bm{\beta}^{*\rm T}{\bf W}_{i}-\bm{\beta}^{*\rm
T}\bm{\Omega}\bm{\beta}^{*}){\bf v}_{1}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf
v}_{1}\\}=O_{p}(\sqrt{\hbox{log}(p)/n})$

with probability of the order $1-O[\exp\\{-\hbox{log}(p)\\}]$. Further because
$E\\{\exp(2\bm{\beta}^{*\rm T}{\bf W}_{i}-\bm{\beta}^{*\rm
T}\bm{\Omega}\bm{\beta}^{*}){\bf v}_{1}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}{\bf v}_{1}\\}=O(1)$, and because
of the same argument as those lead to (A.50), we conclude that (A.50) is of
the order $O_{p}(\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2})$. Hence follow
(A.44), and by Conditions (C1), Corollaries B.2–B.5 in the supplementary
material, for positive constants $C_{4}$ we have

$\displaystyle\sup_{{\bf v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}{\bf v}_{1}^{\rm T}\left\\{\widehat{{\bf
Q}}(\bm{\beta}^{*})-\frac{\partial^{2}\mathcal{L}(\bm{\beta})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}\right\\}{\bf w}\bigg{|}$ $\displaystyle=$ $\displaystyle
O_{p}(\sqrt{m+k}\|\widehat{\bm{\beta}}-\bm{\beta}_{t}\|_{2})$ (A.51)
$\displaystyle\leq$ $\displaystyle
C_{4}\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)$

and

$\displaystyle\sup_{{\bf v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}{\bf v}_{1}^{\rm
T}\left[\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}-{\bf Q}(\bm{\beta}_{t})\right]{\bf w}\bigg{|}$ (A.52) $\displaystyle=$
$\displaystyle\sup_{{\bf v}_{1}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}\bigg{|}{\bf v}_{1(\mathcal{M}\cup S)^{c}}^{\rm
T}\left[\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}-{\bf Q}(\bm{\beta}_{t})\right]_{(\mathcal{M}\cup S)^{c},\mathcal{M}\cup
S}{\bf w}_{\mathcal{M}\cup S}\bigg{|}$ $\displaystyle=$ $\displaystyle
O_{p}[\sqrt{\max\\{(m+k),\hbox{log}(p)\\}/n}]$ $\displaystyle=$ $\displaystyle
o_{p}\left[\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)\right],$

with probabilities to the order of $1-2\exp[-\max\\{(m+k),\hbox{log}(p)\\}]$.

Combine (A.45) and (A.46) with Lemma B.11 in the supplementary material,

$\displaystyle\|(\\{\widehat{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S})(\bm{\beta}^{*})\\}^{-1}-\\{{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\|_{2}$ (A.53) $\displaystyle\leq$
$\displaystyle\frac{\|\\{{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\|_{2}^{2}\|\widehat{\bf Q}_{(\mathcal{M}\cup
S),\mathcal{M}\cup S}(\bm{\beta}^{*})-{\bf Q}_{(\mathcal{M}\cup
S),\mathcal{M}\cup S}(\bm{\beta}_{t})\|_{2}}{\\{1-\|\\{{\bf
Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\|_{2}\|\widehat{\bf Q}_{(\mathcal{M}\cup
S),\mathcal{M}\cup S}(\bm{\beta}^{*})-{\bf Q}_{(\mathcal{M}\cup
S),\mathcal{M}\cup S}(\bm{\beta}_{t})\|_{2}\\}}$ $\displaystyle=$
$\displaystyle O_{p}(\|\widehat{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}^{*})-{\bf Q}_{0(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\|_{2})$ $\displaystyle=$ $\displaystyle
C_{9}\left[\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)\right],$

and hence

$\displaystyle\|(\\{\widehat{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S})(\bm{\beta}^{*})\\}^{-1}-\\{{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\|_{\infty}$ (A.54) $\displaystyle=$ $\displaystyle
C_{9}(m+k)^{3/2}\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}.$

Further,

$\displaystyle\bigg{\|}\widehat{\bf Q}_{(\mathcal{M}\cup
S)^{c},\mathcal{M}\cup S}(\bm{\beta}^{*})\\{\widehat{\bf Q}_{(\mathcal{M}\cup
S),\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{\infty}$ $\displaystyle\leq$ $\displaystyle\bigg{\|}[\widehat{\bf
Q}_{(\mathcal{M}\cup S)^{c},\mathcal{M}\cup S}(\bm{\beta}^{*})-{\bf
Q}_{(\mathcal{M}\cup S)^{c},\mathcal{M}\cup S}(\bm{\beta}_{t})]\\{\widehat{\bf
Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{\infty}$ $\displaystyle+\bigg{\|}{\bf Q}_{(\mathcal{M}\cup
S)^{c},\mathcal{M}\cup S}(\bm{\beta}_{t})(\\{\widehat{\bf Q}_{(\mathcal{M}\cup
S),\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1}$ $\displaystyle-\\{{\bf
Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1})\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{\infty}$ $\displaystyle+\bigg{\|}{\bf Q}_{(\mathcal{M}\cup
S)^{c},\mathcal{M}\cup S}(\bm{\beta}_{t})\\{{\bf Q}_{(\mathcal{M}\cup
S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{\infty}$ $\displaystyle\leq$ $\displaystyle\sup_{{\bf
v}_{1}\in{\mathbb{K}}_{1},{\bf w}\in{\mathbb{K}}}\bigg{|}{\bf
v}_{1(\mathcal{M}\cup S)^{c}}^{\rm T}[\widehat{\bf Q}_{(\mathcal{M}\cup
S)^{c},\mathcal{M}\cup S}(\bm{\beta}^{*})-{\bf Q}_{(\mathcal{M}\cup
S)^{c},\mathcal{M}\cup S}(\bm{\beta}_{t})]{\bf w}_{\mathcal{M}\cup S}\bigg{|}$
$\displaystyle\times\|\\{\widehat{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\|_{2}\bigg{\|}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{2}$ $\displaystyle+\|{\bf Q}_{(\mathcal{M}\cup
S)^{c},\mathcal{M}\cup S}(\bm{\beta}_{t})\|_{2}\bigg{\|}\\{\widehat{\bf
Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1}-\\{{\bf
Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\bigg{\|}_{2}\bigg{\|}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{2}$ $\displaystyle+\bigg{\|}{\bf Q}_{(\mathcal{M}\cup
S)^{c},\mathcal{M}\cup S}(\bm{\beta}_{t})\\{{\bf Q}_{0(\mathcal{M}\cup
S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{\infty}$ $\displaystyle\leq$ $\displaystyle
C_{10}\left[\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)\right]\sqrt{m+k}\sqrt{\hbox{log}(p)/n}$
$\displaystyle+C_{11}\left[\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)\right]\sqrt{m+k}\sqrt{\hbox{log}(p)/n}$
$\displaystyle+\bigg{\|}{\bf Q}_{(\mathcal{M}\cup S)^{c},\mathcal{M}\cup
S}(\bm{\beta}_{t})\\{{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{\infty}.$

(A.3.4) is obtained by using (A.51) and (A.52) and the fact that

$\displaystyle\|\\{\widehat{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\|_{2}$ $\displaystyle\leq$
$\displaystyle\|\\{\widehat{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}-[{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})]^{-1}\|_{2}+\|\\{{\bf Q}_{(\mathcal{M}\cup
S),\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}\|_{2}$ $\displaystyle\leq$
$\displaystyle O_{p}(1).$

(A.3.4) is obtained by using (A.45), (A.46) and the fact that $\|{\bf
Q}_{(\mathcal{M}\cup S)^{c},\mathcal{M}\cup S}(\bm{\beta})\|_{2}=O(1)$.
Therefore, now note that $n\geq c_{\infty}(m+k)^{4}\hbox{log}(p)$ and
$\lambda=O[\\{\hbox{log}(p)/n\\}^{1/4}]$ by the statement assumption, together
with Condition (C4), we have

$\displaystyle\bigg{\|}\widehat{\bf Q}_{(\mathcal{M}\cup
S)^{c},\mathcal{M}\cup S}(\bm{\beta}^{*})\\{\widehat{\bf Q}_{(\mathcal{M}\cup
S),\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{\infty}$ (A.57) $\displaystyle\leq$ $\displaystyle\bigg{\|}{\bf
Q}_{(\mathcal{M}\cup S)^{c},\mathcal{M}\cup S}(\bm{\beta}_{t})\\{{\bf
Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{\infty}+o_{p}(\lambda)$ $\displaystyle\leq$
$\displaystyle\bigg{\|}{\bf Q}_{(\mathcal{M}\cup S)^{c},\mathcal{M}\cup
S}(\bm{\beta})\\{{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\bigg{\|}_{2}\bigg{\|}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{2}+o_{p}(\lambda)$ $\displaystyle=$
$\displaystyle\sqrt{m+k}\sqrt{\hbox{log}(p)/n}+o_{p}(\lambda)$
$\displaystyle=$ $\displaystyle o_{p}(\lambda).$

Therefore

$\displaystyle\|({\bf A}^{\rm T}\widehat{{\bf z}})_{(\mathcal{M}\cup
S)^{c}}\|_{\infty}$ (A.58) $\displaystyle=$
$\displaystyle\|\lambda^{-1}\left[-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{(\mathcal{M}\cup
S)^{c}}\right]+\lambda^{-1}\widehat{{\bf Q}}_{(\mathcal{M}\cup
S)^{c}(\mathcal{M}\cup S)}(\bm{\beta}^{*})\\{\widehat{{\bf
Q}}_{(\mathcal{M}\cup S)(\mathcal{M}\cup S)}(\bm{\beta}^{*})\\}^{-1}$
$\displaystyle\times\left(\left[\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}-\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}\right]+\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup S}+({\bf
A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{3}^{*})_{(\mathcal{M}\cup
S)}\right)\|_{\infty}$ $\displaystyle\leq$
$\displaystyle\lambda^{-1}\bigg{\|}\left[-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{(\mathcal{M}\cup
S)^{c}}\right]\bigg{\|}_{\infty}$
$\displaystyle+\lambda^{-1}\bigg{\|}\widehat{{\bf Q}}_{(\mathcal{M}\cup
S)^{c}(\mathcal{M}\cup S)}(\bm{\beta}^{*})\\{\widehat{{\bf
Q}}_{(\mathcal{M}\cup S)(\mathcal{M}\cup
S)}(\bm{\beta}^{*})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\bigg{\|}_{\infty}$ $\displaystyle+\lambda^{-1}\bigg{\|}\widehat{{\bf
Q}}_{(\mathcal{M}\cup S)^{c}(\mathcal{M}\cup
S)}(\bm{\beta}^{*})\\{\widehat{{\bf Q}}_{(\mathcal{M}\cup S)(\mathcal{M}\cup
S)}(\bm{\beta}^{*})\\}^{-1}\left(\left[-\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}\right]+\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup
S}\right)\bigg{\|}_{\infty}$ $\displaystyle+\lambda^{-1}\|\widehat{{\bf
Q}}_{(\mathcal{M}\cup S)^{c}(\mathcal{M}\cup
S)}(\bm{\beta}^{*})\\{\widehat{{\bf Q}}_{(\mathcal{M}\cup S)(\mathcal{M}\cup
S)}(\bm{\beta}^{*})\\}^{-1}({\bf A}_{1}^{\rm T}{\bf C}^{\rm
T}\bm{\mu}_{3}^{*})_{\mathcal{M}\cup S}\|_{\infty}$ $\displaystyle=$
$\displaystyle\lambda^{-1}\bigg{\|}\widehat{{\bf Q}}_{(\mathcal{M}\cup
S)^{c}(\mathcal{M}\cup S)}(\bm{\beta}^{*})\\{\widehat{{\bf
Q}}_{(\mathcal{M}\cup S)(\mathcal{M}\cup
S)}(\bm{\beta}^{*})\\}^{-1}\left(\left[-\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}\right]+\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup
S}\right)\bigg{\|}_{\infty}$ $\displaystyle+\lambda^{-1}\bigg{\|}\widehat{{\bf
Q}}_{(\mathcal{M}\cup S)^{c}(\mathcal{M}\cup
S)}(\bm{\beta}^{*})\\{\widehat{{\bf Q}}_{(\mathcal{M}\cup S)(\mathcal{M}\cup
S)}(\bm{\beta}^{*})\\}^{-1}{\bf A}_{2}\\{\widehat{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1}$
$\displaystyle\times\left[\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}-\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup
S}\right]\bigg{\|}_{\infty}+o(1).$

Now recall (A.41) and

$\displaystyle({\bf A}_{1}{\bf C}^{\rm T}\bm{\mu}_{3}^{*})_{\mathcal{M}\cup
S}$ $\displaystyle=$ $\displaystyle{\bf A}_{2}\\{\widehat{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\left[-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}+\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}-\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup S}\right].$

Hence

$\displaystyle\|\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\widecheck{\bm{\beta}}_{\mathcal{M}\cup S}\|_{\infty}$ (A.60)
$\displaystyle\leq$ $\displaystyle\|\\{\widehat{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\|_{\infty}$ $\displaystyle+\bigg{\|}\\{\widehat{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1}\left[\left\\{{\bf A}^{\rm
T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}-\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup
S}\right]\bigg{\|}_{\infty}$ $\displaystyle+\|\\{\widehat{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1}{\bf
A}_{2}\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}$
$\displaystyle\times\left[-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}+\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}-\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup
S}\right]\|_{\infty}$ $\displaystyle\leq$ $\displaystyle
o_{p}(\lambda)+\lambda\|\\{\widehat{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\|_{\infty}+\lambda\|\\{\widehat{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1}{\bf
A}_{2}\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\|_{\infty}$ $\displaystyle=$ $\displaystyle
o_{p}(\lambda)+2\lambda c_{\infty}+2\lambda c_{\infty},$

where $o_{p}(\lambda)$ in the second inequality is obtained by using the same
argument as those lead to (A.57). The other two terms are obtained by using
Lemma B.4 such that

$\bigg{\|}\left[\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}-\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup
S}\right]\bigg{\|}_{\infty}\leq\lambda.$

The last equality holds because

$\displaystyle\|\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}\|_{\infty}$ $\displaystyle\leq$
$\displaystyle\|\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}-\\{{\bf Q}_{(\mathcal{M}\cup S),\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\|_{\infty}+\|\\{{\bf Q}_{(\mathcal{M}\cup
S),\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}\|_{\infty}$ $\displaystyle\leq$
$\displaystyle 2c_{\infty}$

by (A.54). And similarly, we have

$\displaystyle\|\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}{\bf A}_{2}\\{\widehat{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1}\|_{\infty}$ $\displaystyle\leq$
$\displaystyle\|\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}{\bf A}_{2}\\{\widehat{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1}$ $\displaystyle-\\{{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}{\bf
A}_{2}\\{{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\|_{\infty}$ $\displaystyle+\|\\{{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}{\bf
A}_{2}\\{{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\|_{\infty}$ $\displaystyle=$ $\displaystyle
2c_{\infty}.$

Now because $\min(|\beta_{j}|)\geq\lambda(\gamma+5c_{\infty})$ for $j\in S$,
we have

$\displaystyle\min|\widehat{\beta}_{j}|\geq\lambda\gamma$

and $q_{\lambda}^{\prime}(\widehat{\beta})_{j}=\lambda{\rm
sign}(\widehat{\beta}_{j})=\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{j}$.
Hence by Lemma B.3 in the supplementary material, we have

$\displaystyle\left\\{{\bf A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}_{\mathcal{M}\cup
S}-\lambda({\bf A}^{\rm T}\widehat{{\bf z}})_{\mathcal{M}\cup S}=\bf 0.$
(A.61)

Inserting (A.61) into (A.58), the first two terms of (A.58) are zero hence
$\|({\bf A}^{\rm T}{\bf z})_{(\mathcal{M}\cup S)^{c}}\|=o_{p}(1)<1.$ This
implies that $\widehat{\bm{\beta}}$ has support in $\mathcal{M}\cup S$.
Further, because of (A.60), $\widehat{\bm{\beta}}$ is unique in the feasible
set. Therefore, using (A.41) together with (A.61) and (A.3.4), we have

$\displaystyle\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\widecheck{\bm{\beta}}_{\mathcal{M}\cup S}$ $\displaystyle=$
$\displaystyle(\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}^{*})\\}^{-1}-\\{\widehat{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1}{\bf A}_{2}\\{\widehat{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}(\bm{\beta}^{*})\\}^{-1})$
$\displaystyle\times\left[-\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\right]$ $\displaystyle=$ $\displaystyle-(\\{{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta})\\}^{-1}-\\{{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}{\bf A}_{2}^{*}\\{{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1})$
$\displaystyle\times\left[\left\\{\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\right]\\{1+o_{p}(1)\\}$

and $\widehat{\bm{\beta}}_{(\mathcal{M}\cup S)^{c}}=\bf 0$. Now because

$\displaystyle\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}-\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}}$
$\displaystyle=$
$\displaystyle\frac{\partial^{2}\mathcal{L}(\bm{\beta}^{*})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}(\widecheck{\bm{\beta}}-\bm{\beta}_{t})$ $\displaystyle=$
$\displaystyle{\bf
Q}(\bm{\beta})(\widecheck{\bm{\beta}}-\bm{\beta}_{t})+\left\\{\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}-{\bf
Q}(\bm{\beta}_{t})\right\\}(\widecheck{\bm{\beta}}-\bm{\beta}_{t})+\left\\{\frac{\partial^{2}\mathcal{L}(\bm{\beta}^{*})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}-\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}\right\\}(\widecheck{\bm{\beta}}-\bm{\beta}_{t})$ $\displaystyle=$
$\displaystyle{\bf
Q}(\bm{\beta}_{t})(\widecheck{\bm{\beta}}-\bm{\beta}_{t})\\{1+o_{p}(1)\\}.$

The last equality holds because

$\displaystyle\|\left\\{\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}-{\bf Q}(\bm{\beta}_{t})\right\\}\|_{2}=o_{p}(1),$

by (A.46),

$\displaystyle\|\left\\{\frac{\partial^{2}\mathcal{L}(\widetilde{\bm{\beta}})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}-\frac{\partial^{2}\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}\right\\}\|_{2}=o_{p}(1)$

by (A.45), and $\|{\bf Q}(\bm{\beta}_{t})\|_{2}=O(1)$. Therefore,

$\displaystyle\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\widecheck{\bm{\beta}}_{\mathcal{M}\cup S}$ $\displaystyle=$
$\displaystyle-(\\{{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}-\\{{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}{\bf A}_{2}^{*}\\{{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1})$
$\displaystyle\times\left[\left\\{\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}}+{\bf
Q}(\bm{\beta}_{t})(\widecheck{\bm{\beta}}-\bm{\beta}_{t})\right\\}_{\mathcal{M}\cup
S}\right]\\{1+o_{p}(1)\\}.$

Further, we have

$\displaystyle\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\bm{\beta}_{t\mathcal{M}\cup S}$ $\displaystyle=$
$\displaystyle\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\widecheck{\bm{\beta}}_{\mathcal{M}\cup S}-[({\bf C}{\bf C}^{\rm
T})^{-1}{\bf C},{\bf 0}_{r\times k}]^{\rm T}{\bf h}_{n}$ $\displaystyle=$
$\displaystyle-(\\{{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})^{-1}\\}-\\{{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}{\bf A}_{2}^{*}\\{{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1})\left[\left\\{\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\right.$ $\displaystyle\left.+{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})[({\bf C}{\bf C}^{\rm T})^{-1}{\bf C},{\bf 0}_{r\times
k}]^{\rm T}{\bf h}_{n}\vphantom{\int_{0}^{1}}\right]\\{1+o_{p}(1)\\}-[({\bf
C}{\bf C}^{\rm T})^{-1}{\bf C},{\bf 0}_{r\times k}]^{\rm T}{\bf h}_{n}$
$\displaystyle=$ $\displaystyle-\\{({{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}-\\{{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}{\bf A}_{2}^{*}\\{{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1})\left\\{\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\\{1+o_{p}(1)\\}$ $\displaystyle-\\{{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}{\bf A}_{2}^{*}[\\{({\bf C}{\bf
C}^{\rm T})^{-1}{\bf C},{\bf 0}_{r\times k}\\}^{\rm T}{\bf
h}_{n}]\\{1+o_{p}(1)\\}.$

∎

####  A.3.5 Proof of Theorem 4

The proof is very similar to that of Theorem 3 hence is omitted.

###  A.4 Proof of Results in Section 4.4

####  A.4.1 Proof of Lemma 1

First note that

$\displaystyle\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\bm{\omega}_{n}$ $\displaystyle=$
$\displaystyle-\sqrt{n}\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}^{-1}(\bm{\beta}_{t})\left\\{\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}$ $\displaystyle=$ $\displaystyle\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t})$
$\displaystyle\times n^{-1/2}\sum_{i=1}^{n}\\{Y_{i}{\bf
W}_{i}-\exp(\bm{\beta}_{t}^{\rm T}{\bf W}_{i}-\bm{\beta}_{t}^{\rm
T}\bm{\Omega}\bm{\beta}_{t}/2)({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{t})\\}_{\mathcal{M}\cup S}.$

It is easy to see that

$\displaystyle\sum_{i=1}^{n}\hbox{cov}\left(\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t})\right.$
$\displaystyle\left.\times n^{-1/2}\\{Y_{i}{\bf
W}_{i}-\exp(\bm{\beta}_{t}^{\rm T}{\bf W}_{i}-\bm{\beta}_{t}^{\rm
T}\bm{\Omega}\bm{\beta}_{t}/2)({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{t})\\}_{\mathcal{M}\cup S}\right)={\bf
I}_{r\times r}.$ (A.62)

Further,

$\displaystyle
r^{1/4}\sum_{i=1}^{n}E\|n^{-1/2}\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t})$ (A.63)
$\displaystyle\times\\{Y_{i}{\bf W}_{i}-\exp(\bm{\beta}_{t}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{t}^{\rm T}\bm{\Omega}\bm{\beta}_{t}/2)({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{t})\\}_{\mathcal{M}\cup S}\|_{2}^{3}$
$\displaystyle\leq$ $\displaystyle
r^{1/4}\sum_{i=1}^{n}E\|n^{-1/2}\\{Y_{i}\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t}){\bf
W}_{i\mathcal{M}\cup S}$ $\displaystyle-\exp(\bm{\beta}_{t}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{t}^{\rm
T}\bm{\Omega}\bm{\beta}_{t}/2)\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t})({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{t})_{\mathcal{M}\cup S}\\}\|_{2}^{3}$
$\displaystyle\leq$ $\displaystyle
r^{1/4}\sum_{i=1}^{n}E\|n^{-1/2}\\{Y_{i}\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t}){\bf
W}_{i\mathcal{M}\cup S}$ $\displaystyle-\exp(\bm{\beta}_{t}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{t}^{\rm
T}\bm{\Omega}\bm{\beta}_{t}/2)\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t})({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{t})_{\mathcal{M}\cup S}\\}\|_{2}^{3}$
$\displaystyle\leq$ $\displaystyle
r^{1/4}D\sum_{i=1}^{n}n^{-3/2}(\|\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t}){\bf
W}_{i\mathcal{M}\cup S}\|_{2}^{3}$
$\displaystyle+\|\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf Q},\bm{\beta}_{t}){\bf
C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup
S}^{-1}(\bm{\beta}_{t})(\bm{\Omega}\bm{\beta}_{t})_{\mathcal{M}\cup
S}\|_{2}^{3})$ $\displaystyle=$ $\displaystyle o_{p}(1),$

where $D$ is a positive constant. The second to the last inequality holds by
Condition (D1). Also because

$\displaystyle\|\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf Q},\bm{\beta}_{t}){\bf
C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t}){\bf W}_{i\mathcal{M}\cup S}\|_{2}$
$\displaystyle\leq$ $\displaystyle\|\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t})\|_{2}\|{\bf
W}_{i\mathcal{M}\cup S}\|_{2}$ $\displaystyle\leq$
$\displaystyle\|\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf Q},\bm{\beta}_{t}){\bf
C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t})\|_{2}|{\bf W}_{i}^{\rm T}{\bf
v}|/\|{\bf v}\|_{2}$ $\displaystyle\leq$ $\displaystyle
O(1)M_{W}\sqrt{m+k}=O(\sqrt{m+k}),$

where ${\bf v}$ is a $p$ dimensional vector with $\|{\bf v}\|_{0}=m+k$, the
last inequality holds by Condition (C1) (a). Further since
$\|\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf Q},\bm{\beta}_{t}){\bf C}[{\bf
I}_{m\times m},{\bf 0}_{m\times k}]{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup
S}^{-1}(\bm{\beta}_{t})(\bm{\Omega}\bm{\beta}_{t})_{\mathcal{M}\cup
S}\|_{2}=O(1)$,

$\displaystyle
r^{1/4}D\sum_{i=1}^{n}n^{-3/2}(\|\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t}){\bf
W}_{i\mathcal{M}\cup S}\|_{2}^{3}$
$\displaystyle+\|\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf Q},\bm{\beta}_{t}){\bf
C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup
S}^{-1}(\bm{\beta}_{t})(\bm{\Omega}\bm{\beta}_{t})_{\mathcal{M}\cup
S}\|_{2}^{3})$ $\displaystyle=$ $\displaystyle
O\\{r^{1/4}(m+k)^{3/2}n^{-1/2}\\}=o_{p}(1),$

by the Condition that $n\geq c_{\infty}(m+k)^{4}\hbox{log}(p)$. Combine
(A.4.1) and (A.63), and using Lemma B.14 in Section B.4.3, let ${\bf Z}$ is a
standard Gaussian random variable we have

$\displaystyle\lim_{n\to\infty}\sup_{\mathcal{C}}|\Pr(\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\bm{\omega}_{n}\in{\mathcal{C}})-\Pr({\bf
Z}\in{\mathcal{C}})|=0.$ (A.64)

Now we choose

$\displaystyle{\mathcal{C}_{x}}=\\{{\bf z}:\|{\bf
z}-\sqrt{n}\bm{\Psi}^{-1/2}{\bf h}_{n}\|_{2}\leq x\\},$

then

$\displaystyle\lim_{n\to\infty}|\Pr(\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\bm{\omega}_{n}\in{\mathcal{C}_{x}})-\Pr({\bf
Z}\in{\mathcal{C}}_{x})|=0,$

which implies

$\displaystyle\lim_{n\to\infty}|\Pr(T_{0}\leq x)-\Pr\\{\chi^{2}(r,n{\bf
h}_{n}^{\rm T}\bm{\Psi}^{-1}({\bf\Sigma},{\bf Q},\bm{\beta}_{t}){\bf
h}_{n})\leq x\\}|=0,$

where $\chi^{2}(r,\gamma)$ is a non-central chi-square distribution, with non-
centrality parameter $\gamma$. ∎

####  A.4.2 Proof of Theorem 5

From Theorem 4, we have

$\displaystyle\widehat{\bm{\beta}}_{a\mathcal{M}\cup
S}-\bm{\beta}_{t\mathcal{M}\cup S}$ $\displaystyle=$ $\displaystyle-\\{{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\\{1+o_{p}(1)\\}$

Hence,

$\displaystyle\sqrt{n}{\bf
C}(\widehat{\bm{\beta}}_{a\mathcal{M}}-\bm{\beta}_{t\mathcal{M}})$
$\displaystyle=$ $\displaystyle-\sqrt{n}{\bf C}[{\bf I}_{m\times m},{\bf
0}_{m\times k}]\\{{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\\{1+o_{p}(1)\\}$ $\displaystyle=$
$\displaystyle\bm{\omega}_{n}\\{1+o_{p}(1)\\}.$

Further, because ${\bf C}\bm{\beta}_{t\mathcal{M}}={\bf t}+{\bf h}_{n}$, we
have

$\displaystyle\sqrt{n}\bm{\Psi}^{-1/2}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})({\bf C}\widehat{\bm{\beta}}_{a\mathcal{M}}-{\bf
t})=\bm{\Psi}^{-1/2}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})\bm{\omega}_{n}\\{1+o_{p}(1)\\}+\sqrt{n}\bm{\Psi}^{-1/2}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a}){\bf h}_{n}.$

Further because ${\bf h}_{n}=O\\{\sqrt{\max(m+k-r,r)/n}\\}$,

$\displaystyle n({\bf C}\widehat{\bm{\beta}}_{a\mathcal{M}}-{\bf t})^{\rm
T}\bm{\Psi}^{-1}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})({\bf C}\widehat{\bm{\beta}}_{a\mathcal{M}}-{\bf
t})$ (A.65) $\displaystyle=$ $\displaystyle(\bm{\omega}_{n}+\sqrt{n}{\bf
h}_{n})^{\rm T}\bm{\Psi}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n})\\{1+o_{p}(1)\\}$
$\displaystyle+(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n})^{\rm
T}\\{\bm{\Psi}^{-1}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})-\bm{\Psi}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\\}(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n})$
$\displaystyle\leq$ $\displaystyle(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n})^{\rm
T}\bm{\Psi}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n})\\{1+o_{p}(1)\\}$
$\displaystyle+\|\bm{\omega}_{n}+\sqrt{n}{\bf
h}_{n}\|_{2}^{2}\|\bm{\Psi}^{-1}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})-\bm{\Psi}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\|_{2}$ $\displaystyle=$ $\displaystyle T_{0}+o_{p}(r).$

The last equality holds because $T_{0}$ converge in distribution to a non-
central chi-square distribution with degree freedom $r$ as shown in Lemma 1
and hence $T_{0}$ is of the order $O_{p}(r)$. Also because

$\displaystyle\|\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n}\|_{2}^{2}$
$\displaystyle\leq$ $\displaystyle(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n})^{\rm
T}\bm{\Psi}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})(\bm{\omega}_{n}+\sqrt{n}{\bf
h}_{n})/\alpha_{\min}\\{\bm{\Psi}^{-1}({\bf\Sigma},{\bf Q},\bm{\beta}_{t})\\}$
$\displaystyle\leq$ $\displaystyle(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n})^{\rm
T}\bm{\Psi}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n})/c_{\bm{\Psi}}$

by Condition (D2), we have

$\displaystyle\|\bm{\Psi}^{-1}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})-\bm{\Psi}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\|_{2}\|\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n}\|_{2}^{2}$
$\displaystyle\leq$
$\displaystyle\left[\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)\right]O_{p}(T_{0}).$

Therefore, $T_{W}-T_{0}=o_{p}(r)$. ∎

####  A.4.3 Proof of Theorem 6

From Theorem 3, we have

$\displaystyle\widehat{\bm{\beta}}_{\mathcal{M}\cup
S}-\bm{\beta}_{t\mathcal{M}\cup S}$ $\displaystyle=$ $\displaystyle-(\\{{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}-\\{{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}{\bf
A}_{2}^{*}\\{{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1})\left\\{\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\\{1+o_{p}(1)\\}$ $\displaystyle+\\{{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}(\bm{\beta}_{t})\\}^{-1}{\bf A}_{2}^{*}[\\{({\bf C}{\bf
C}^{\rm T})^{-1}{\bf C},{\bf 0}_{r\times k}\\}^{\rm T}{\bf
h}_{n}]\\{1+o_{p}(1)\\}$

and $\widehat{\bm{\beta}}_{(\mathcal{M}\cup S)^{c}}=\bf 0$. By Taylor
expansion we have

$\displaystyle\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}$ $\displaystyle=$
$\displaystyle\left\\{\frac{\partial\mathcal{L}({\bm{\beta}_{t}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}+\left\\{\frac{\partial^{2}\mathcal{L}({\bm{\beta}}^{*})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}\right\\}_{\mathcal{M}\cup
S}(\widehat{\bm{\beta}}-\bm{\beta}_{t})_{\mathcal{M}\cup S}$ $\displaystyle=$
$\displaystyle\left\\{\frac{\partial\mathcal{L}({\bm{\beta}_{t}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}+{\bf Q}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})(\widehat{\bm{\beta}}-\bm{\beta}_{t})_{\mathcal{M}\cup
S}\\{1+o_{p}(1)\\}$ $\displaystyle=$ $\displaystyle{\bf A}_{2}^{*}\\{{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\\{1+o_{p}(1)\\}$ $\displaystyle+\\{{\bf A}_{2}^{*}[\\{({\bf C}{\bf C}^{\rm
T})^{-1}{\bf C},{\bf 0}_{r\times k}\\}^{\rm T}{\bf h}_{n}]\\{1+o_{p}(1)\\}$

where $\bm{\beta}^{*}$ is a point in between $\bm{\beta}_{t}$ and
$\widehat{\bm{\beta}}$. The second equality holds by (A.45) and (A.46) in
Appendix. Therefore, we have

$\displaystyle\sqrt{n}\bm{\Psi}(\widehat{{\bf\Sigma}},\widehat{{\bf
Q}},\widehat{\bm{\beta}})^{-1/2}{\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times
k}]\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\widehat{\bm{\beta}})\\}^{-1}{\bf A}_{2}^{*}\\{{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\left\\{\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}$ $\displaystyle=$
$\displaystyle\sqrt{n}\bm{\Psi}(\widehat{{\bf\Sigma}},\widehat{{\bf
Q}},\widehat{\bm{\beta}})^{-1/2}{\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times
k}]\\{{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\bm{\beta}_{t})\\}^{-1}\left\\{\frac{\partial\mathcal{L}({\bm{\beta}_{t}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}\\{1+o_{p}(1)\\}$ $\displaystyle=$
$\displaystyle\bm{\Psi}(\widehat{{\bf\Sigma}},\widehat{{\bf
Q}},\widehat{\bm{\beta}})^{-1/2}\bm{\omega}_{n}\\{1+o_{p}(1)\\},$

The first equality holds by (A.45) and (A.46) in the supplementary material.
And similarly

$\displaystyle\sqrt{n}\bm{\Psi}(\widehat{{\bf\Sigma}},\widehat{{\bf
Q}},\widehat{\bm{\beta}})^{-1/2}{\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times
k}]\\{\widehat{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\widehat{\bm{\beta}})\\}^{-1}\\{{\bf A}_{2}^{*}[\\{({\bf C}{\bf C}^{\rm
T})^{-1}{\bf C},{\bf 0}_{r\times k}\\}^{\rm T}{\bf h}_{n}]$ $\displaystyle=$
$\displaystyle\sqrt{n}\bm{\Psi}(\widehat{{\bf\Sigma}},\widehat{{\bf
Q}},\widehat{\bm{\beta}})^{-1/2}{\bf h}_{n}\\{1+o_{p}(1)\\}$

and hence

$\displaystyle T_{S}=(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n})^{\rm
T}\bm{\Psi}(\widehat{{\bf\Sigma}},\widehat{{\bf
Q}},\widehat{\bm{\beta}})^{-1}(\bm{\omega}_{n}+\sqrt{n}{\bf
h}_{n})\\{1+o_{p}(1)\\}$

Now the same steps in (A.65) lead to $T_{S}-T_{0}=o_{p}(r)$. ∎

## Appendix B

##  B.1 Some Lemmas on the Penalty Function

######  Lemma B.1.

Conditions (A1)–(A4) imply that $\rho_{\lambda}$ is $\lambda$-Lipschitz and
all sub gradients and derivatives of $\rho_{\lambda}$ are bounded by $\lambda$
in magnitude. Conditions (A1)–(A5) imply

$\displaystyle\lambda\|\bm{\beta}_{\mathcal{M}^{c}}\|_{1}\leq\rho_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})+\mu/2\|\bm{\beta}_{\mathcal{M}^{c}}\|_{2}^{2},\forall\bm{\beta}_{\mathcal{M}^{c}}\in\mathbb{{\bf
R}}^{p-m}$

Proof: This lemma is a direct consequence of Lemma 4 in Loh & Wainwright
(2015).

######  Lemma B.2.

Suppose $\rho_{\lambda}$ satisfies Conditions (A1)–(A5). Let ${\bf
v}\in\mathbb{R}^{p-m}$, let $\mathcal{A}$ be the index set of $k$ largest
elements of ${\bf v}$ in magnitude, and let $\mathcal{A}^{c}$ be the index set
of the remaining $p-m-k$ elements of ${\bf v}$. Suppose $\xi>0$ and satisfies

$\displaystyle\xi\rho_{\lambda}({\bf v}_{\mathcal{A}})-\rho_{\lambda}({\bf
v}_{\mathcal{A}^{c}})\geq 0.$

Then

$\displaystyle\xi\rho_{\lambda}({\bf v}_{\mathcal{A}})-\rho_{\lambda}({\bf
v}_{\mathcal{A}^{c}})\leq\lambda(\xi\|{\bf v}_{\mathcal{A}}\|_{1}-\|{\bf
v}_{\mathcal{A}^{c}}\|_{1}).$

Moreover, if $\bm{\beta}_{t\mathcal{M}^{c}}\in\mathbb{R}^{p-m}$ is $k$-sparse,
then for a vector $\bm{\beta}_{\mathcal{M}^{c}}\in\mathbb{R}^{p-m}$ such that
$\xi\rho_{\lambda}(\bm{\beta}_{t\mathcal{M}^{c}})-\rho_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})>0$
and $\xi\geq 1$, we have

$\displaystyle\xi\rho_{\lambda}(\bm{\beta}_{t\mathcal{M}^{c}})-\rho_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})\leq\lambda(\xi\|{\bf
v}_{\mathcal{A}}\|_{1}-\|{\bf v}_{\mathcal{A}^{c}}\|_{1}),$

where ${\bf v}=\bm{\beta}_{\mathcal{M}^{c}}-\bm{\beta}_{t\mathcal{M}^{c}}$,
$\mathcal{A}$ is the index set of the $k$ largest elements of ${\bf v}$ in
magnitude, and $\mathcal{A}^{c}$ is the index set of the remaining $p-m-k$
elements of ${\bf v}$.

Proof: This lemma is a direct consequence of Lemma 5 in Loh & Wainwright
(2015). ∎

######  Lemma B.3.

Suppose $\rho_{\lambda}$ is $(\mu,\gamma)$-amenable,
$|\widehat{\beta}_{j}|\geq\lambda\gamma$ for $j\in\mathcal{M}\cup S$, then
$q_{\lambda}^{\prime}(\widehat{\beta}_{j})=\lambda{\rm
sign}(\widehat{\beta}_{j})$.

Proof: Because $\rho_{\lambda}$ is $(\mu,\gamma)$-amenable,
$\rho^{\prime}(\widehat{\beta}_{j})=0$ by Condition (A6) and (A1). Hence
$q_{\lambda}^{\prime}(\widehat{\beta}_{j})=\partial\lambda|\widehat{\beta}_{j}|/\partial\widehat{\beta}_{j}=\lambda{\rm
sign}(\widehat{\beta}_{j})$. This proves the result. ∎

######  Lemma B.4.

Consider a $\mu$-amenable regularizer $\rho_{\lambda}$. Then
(a) $|\rho_{\lambda}^{\prime}(t)|\leq\lambda$ for all $t\neq 0$.
(b) The function $q_{\lambda}(t)-\mu/2t^{2}$ is concave and everywhere
differentiable, where $q_{\lambda}(t)=\lambda|t|-\rho_{\lambda}(t)$.

Proof: This lemma is a direct consequence of Lemma 5 in Loh & Wainwright
(2017). ∎

##  B.2 Some Lemmas on the Criterion Function

######  Lemma B.5.

Assume Conditions (C1) – (C4) hold. There exists a constant $c_{1}>0$ so that

$\displaystyle\hbox{pr}\left[n^{-1}\|\sum_{i=1}^{n}\\{Y_{i}{\bf
W}_{i}-\exp(\bm{\beta}_{t}^{\rm T}{\bf W}_{i}-\bm{\beta}_{t}^{\rm
T}\bm{\Omega}\bm{\beta}_{t}/2)({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{t})\\}\|_{\infty}>c_{1}\sqrt{\hbox{log}(p)/n}\vphantom{\int_{0}^{1}}\right]\leq
6p^{-1}.$

Proof: The lemma is the direct consequence of Corollary 1 in Jiang & Ma
(2021). We omit the proof here.

######  Lemma B.6.

Assume that Conditions (C1), (C4) and (C6) hold, then for any $\bm{\beta}$
with $\|\bm{\beta}\|_{2}\leq R_{2}$, and for sufficiently large $n$ and $p$,
with probability $1-O[\exp\\{-\sqrt{n\hbox{log}p}\\}]$,
$n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta}/2)\\{({\bf W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes
2}-\bm{\Omega}\\}$ satisfies the lower and upper-RE conditions with

$\alpha_{1}=\min_{\|\bm{\beta}\|_{1}\leq R_{1},\|\bm{\beta}\|_{2}\leq
R_{2}}\alpha_{\min}[E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf X}_{i}{\bf
X}_{i}^{\rm T}\\}]/2,$ $\alpha_{2}=\max_{\|\bm{\beta}\|_{1}\leq
R_{1},\|\bm{\beta}\|_{2}\leq R_{2}}3\alpha_{\max}[E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}]/2$

and $\tau(n,p)=\tau_{1}\sqrt{\hbox{log}(p)/n}$ for a bounded positive constant
$\tau_{1}$.

Proof: The lemma is a direct consequence of Lemma 12 in Jiang & Ma (2021) with
$c=1$. ∎

######  Corollary B.1.

Assume Conditions (C1)–(C4) to hold, $\|{\bf
h}_{n}\|_{2}=O\\{\sqrt{\max(m+k-r,r)/n}\\}$ and $m+k=o(n^{1/3})$, then there
exists a positive constant $c_{10}$ so that

$\displaystyle\|\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\|_{\infty}\leq
c_{10}\max[\sqrt{(m+k)/n},\sqrt{\hbox{log}(p)/n}],$

with probability $1-O(p^{-1})-O[\exp\\{-\sqrt{n\hbox{log}(p)}\\}]$.

Proof:

$\displaystyle\|\frac{\partial\mathcal{L}(\widecheck{\bm{\beta}})}{\partial\bm{\beta}}\|_{\infty}$
$\displaystyle=$ $\displaystyle n^{-1}\|\sum_{i=1}^{n}\\{Y_{i}{\bf
W}_{i}-\exp(\widecheck{\bm{\beta}}^{\rm T}{\bf
W}_{i}-\widecheck{\bm{\beta}}^{\rm T}\bm{\Omega}\widecheck{\bm{\beta}}/2)({\bf
W}_{i}-\bm{\Omega}\widecheck{\bm{\beta}})\\}\|_{\infty}$ $\displaystyle\leq$
$\displaystyle n^{-1}\|\sum_{i=1}^{n}\\{Y_{i}{\bf
W}_{i}-\exp({\bm{\beta}_{t}}^{\rm T}{\bf W}_{i}-{\bm{\beta}_{t}}^{\rm
T}\bm{\Omega}{\bm{\beta}_{t}}/2)({\bf
W}_{i}-\bm{\Omega}{\bm{\beta}_{t}})\\}\|_{\infty}$
$\displaystyle+n^{-1}\|\sum_{i=1}^{n}\exp(\widecheck{\bm{\beta}}^{\rm T}{\bf
W}_{i}-\widecheck{\bm{\beta}}^{\rm T}\bm{\Omega}\widecheck{\bm{\beta}}/2)({\bf
W}_{i}-\widecheck{\bm{\beta}}^{\rm
T}\bm{\Omega})\\}-\exp({\bm{\beta}_{t}}^{\rm T}{\bf
W}_{i}-{\bm{\beta}_{t}}^{\rm T}\bm{\Omega}{\bm{\beta}_{t}}/2)({\bf
W}_{i}-\bm{\Omega}{\bm{\beta}_{t}})\\}\|_{\infty}.$

Let $\mathbb{K}_{0}=\\{{\bf v}\in{\mathbb{R}}^{p}:{\bf
v}_{\mathcal{M}^{c}}=0,\|{\bf v}\|_{2}=1\\}.$ Then,

$\displaystyle n^{-1}\|\sum_{i=1}^{n}\exp(\widecheck{\bm{\beta}}^{\rm T}{\bf
W}_{i}-\widecheck{\bm{\beta}}^{\rm T}\bm{\Omega}\widecheck{\bm{\beta}}/2)({\bf
W}_{i}-\widecheck{\bm{\beta}}^{\rm T}\bm{\Omega})-\exp({\bm{\beta}_{t}}^{\rm
T}{\bf W}_{i}-{\bm{\beta}_{t}}^{\rm T}\bm{\Omega}{\bm{\beta}_{t}}/2)({\bf
W}_{i}-\bm{\Omega}{\bm{\beta}_{t}})\|_{\infty}$ $\displaystyle\leq$
$\displaystyle n^{-1}\|\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf
W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes
2}-\bm{\Omega}\\}\\{\widecheck{\bm{\beta}}-{\bm{\beta}_{t}}\\}\|_{2}$
$\displaystyle\leq$ $\displaystyle\sup_{{\bf
v}\in\mathbb{K}}n^{-1}\sum_{i=1}^{n}{\bf v}^{\rm T}\exp(\bm{\beta}^{*\rm
T}{\bf W}_{i}-\bm{\beta}^{*\rm T}\bm{\Omega}\bm{\beta}^{*}/2)\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}-\bm{\Omega}\\}{\bf
v}\|\widecheck{\bm{\beta}}-{\bm{\beta}_{t}}\|_{2}$ $\displaystyle\leq$
$\displaystyle 3\alpha_{\max}[E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf
X}_{i}{\bf X}_{i}^{\rm T}\\}]/2\|{\bf C}^{\rm T}({\bf C}{\bf C}^{\rm
T})^{-1}\|_{2}\|{\bf h}_{n}\|_{2}+\tau_{1}m\sqrt{\hbox{log}(p)/n}\|{\bf
C}^{\rm T}({\bf C}{\bf C}^{\rm T})^{-1}\|_{2}\|{\bf h}_{n}\|_{2}$
$\displaystyle\leq$ $\displaystyle
c_{10}\max[\sqrt{(m+k)/n},\sqrt{\hbox{log}(p)/n}]$

for some constants $c_{10}$. The first inequality holds by the Taylor
expansion with $\bm{\beta}^{*}$ be the point on the line connecting
$\widecheck{\bm{\beta}}$ and $\bm{\beta}$. The second inequality holds because
$\widecheck{\bm{\beta}}-\bm{\beta}$ only has $m$ nonzero elements supported in
$\mathcal{M}$, and hence only the corresponding $m\times m$ sub-matrix in
$n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{*\rm T}{\bf W}_{i}-\bm{\beta}^{*\rm
T}\bm{\Omega}\bm{\beta}^{*}/2)\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta}^{*})^{\otimes 2}-\bm{\Omega}\\}$ contributes to
the $L_{2}$ norm. The third inequality holds by Lemma B.6 and the fact that
${\bf v}$ only as $m$ non-zero elements. The last equality holds because
$\|{\bf h}_{n}\|_{2}=O\\{\sqrt{\max(m+k-r,r)/n}\\}\leq O(\sqrt{m+k/n})$, and
$m\sqrt{m+k/n}\to 0$ because $m+k=o(n^{1/3})$. This proves the results. ∎

######  Lemma B.7.

Assume that Conditions (C1) and (C6) hold. If ${\bf X}_{i},{\bf
U}_{i}\in{\mathbb{R}}^{p}$, define ${\mathbb{K}}\equiv\\{{\bf
v}\in{\mathbb{R}}^{\mathcal{M}\cup S}:\|{\bf v}\|_{2}\leq 1\\}$,
${\mathbb{K}}_{1}\equiv\\{{\bf v}\in{\mathbb{R}}^{(\mathcal{M}\cup
S)^{c}}:\|{\bf v}\|_{2}=1,\|{\bf v}\|_{0}=1\\}$

$\displaystyle\Pr\left(\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}]|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{nt^{2}}{324e^{2}M_{4}},\frac{nt}{36eM_{5}\hbox{log}(n)}\right]+2(m+k)\hbox{log}(9)\right),$
$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}]|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{nt^{2}}{36e^{2}M_{4}},\frac{nt}{12eM_{5}\hbox{log}(n)}\right]+(m+k)\hbox{log}(9)+\hbox{log}(p)\right),$

and

$\displaystyle\Pr\left(\sup_{{\bf v}\in\mathbb{K}_{1},{\bf
w}\in\mathbb{K}_{1}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}]|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{nt^{2}}{16e^{2}M_{4}},\frac{nt}{8eM_{5}\hbox{log}(n)}\right]+2\hbox{log}(p)\right).$

Proof: By Lemma 1 statement 3 and Lemma 3 statement 3 in Jiang & Ma (2021), we
can see that the square of a conditional sub-Gaussian variable is sub-
exponential. Now because ${\bf v}^{\rm T}({\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega})$ given ${\bf X}_{i}$ and $\bm{\beta}^{\rm T}{\bf W}_{i}$ is
normal, and hence ${\bf v}^{\rm T}({\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega})^{\otimes 2}{\bf v}$ is conditional sub-exponential. Now by the
Cauchy–Schwarz inequality, and without loss of generality we assume ${\bf
v}^{\rm T}({\bf W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega})^{\otimes 2}{\bf
v}\geq{\bf w}^{\rm T}({\bf W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega})^{\otimes
2}{\bf w}$, we have

$\displaystyle|{\bf v}^{\rm T}({\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega})^{\otimes 2}{\bf w}|\leq{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega})^{\otimes 2}{\bf v}.$

Hence, by Lemma 3 statement 3 in Jiang & Ma (2021), we have for some bounded
positive $K_{3}(\bm{\beta}^{\rm T}{\bf W}_{i},{\bf X}_{i})$,

$\displaystyle E[\exp\\{|{\bf v}^{\rm T}({\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega})^{\otimes 2}{\bf w}|/K_{3}(\bm{\beta}^{\rm T}{\bf W}_{i},{\bf
X}_{i})\\}|\bm{\beta}^{\rm T}{\bf W}_{i},{\bf X}_{i}]$ $\displaystyle\leq$
$\displaystyle E[\exp\\{|{\bf v}^{\rm T}({\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega})^{\otimes 2}{\bf v}|/K_{3}(\bm{\beta}^{\rm T}{\bf W}_{i},{\bf
X}_{i})\\}|\bm{\beta}^{\rm T}{\bf W}_{i},{\bf X}_{i}]\leq e$

Hence, ${\bf v}^{\rm T}({\bf W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega})^{\otimes
2}{\bf w}$ is also conditional sub-exponential variable. Therefore, we have
that

$\displaystyle g({\bf W}_{i},\bm{\beta},{\bf v},{\bf w})-E\\{g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})|\bm{\beta}^{\rm T}{\bf W}_{i},{\bf
X}_{i}\\}$

is centered sub-exponential. Then using the same argument as those that lead
to Lemma 8 in Jiang & Ma (2021) and Condition (C6), for any unit vectors ${\bf
v},{\bf w}$, we can show that

$\displaystyle\Pr\left(|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}]|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{nt^{2}}{16e^{2}M_{4}},\frac{nt}{8eM_{5}\hbox{log}(n)}\right]\right).$

Now we define $\mathcal{B}=\\{{\bf u}_{1},\ldots,{\bf
u}_{r}\\}\subset{\mathbb{K}}$ to be a 1/3-cover of ${\mathbb{K}}$, if for
every ${\bf v}\in{\mathbb{K}}$, there is some ${\bf u}_{i}\in\mathcal{B}$ such
that $\|{\bf v}-{\bf u}_{i}\|_{2}\leq 1/3$. Define $\Delta{\bf v}={\bf v}-{\bf
u}_{j}$ where ${\bf u}_{j}=\arg\min_{{\bf u}_{i}}\|{\bf v}-{\bf u}_{i}\|_{2}$.
We have $\|\Delta{\bf v}\|_{2}\leq 1/3$. Similarly define ${\bf
u}_{k}=\arg\min_{{\bf u}_{i}}\|{\bf w}-{\bf u}_{i}\|_{2}$ for ${\bf
w}\in\mathbb{K}$. By ledoux2013, we can construct $\mathcal{B}$ with
$|\mathcal{B}|<9^{(m+k)}$. Now for ${\bf v}_{1},{\bf v}_{2}\in\mathbb{K}$,
define

$\displaystyle\Phi({\bf v}_{1},{\bf v}_{2})={\bf v}_{1}^{\rm
T}\left[\sum_{i=1}^{n}\frac{A(\bm{\beta}^{\rm T}{\bf W}_{i})\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes
2}-\bm{\Omega}\\}-E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf X}_{i}{\bf
X}_{i}^{\rm T}\\}}{n}\right]{\bf v}_{2}.$

We have

$\displaystyle|\Phi({\bf v},{\bf w})|$ $\displaystyle=$
$\displaystyle|\Phi(\Delta{\bf v}+{\bf u}_{j},\Delta{\bf w}+{\bf u}_{k})|$
$\displaystyle\leq$ $\displaystyle\max_{j,k}|\Phi({\bf u}_{j},{\bf
u}_{k})|+\max_{i}|\Phi(\Delta{\bf v},{\bf u}_{i})|+\max_{i}|\Phi({\bf
u}_{i},\Delta{\bf w})|+|\Phi(\Delta{\bf v},\Delta{\bf w})|.$

Since $\|3\Delta{\bf v}\|_{2}\leq 1$ and $\hbox{supp}(3\Delta{\bf
v})\subseteq\mathbb{K}$, $3\Delta{\bf v}\in\mathbb{K}$. It follows that

$\displaystyle\sup_{{\bf v},{\bf w}\in\mathbb{K}}|\Phi({\bf v},{\bf w})|$
$\displaystyle\leq$ $\displaystyle\max_{j,k}|\Phi({\bf u}_{j},{\bf
u}_{k})|+1/3\sup_{{\bf v}\in\mathbb{K}}\max_{i}|\Phi(3\Delta{\bf v},{\bf
u}_{i})|+1/3\sup_{{\bf w}\in\mathbb{K}}\max_{i}|\Phi({\bf u}_{k},3\Delta{\bf
w})|+1/9\sup_{{\bf v},{\bf w}\in\mathbb{K}}|\Phi(3\Delta{\bf v},3\Delta{\bf
w})|$ $\displaystyle\leq$ $\displaystyle\max_{j,k}|\Phi({\bf u}_{j},{\bf
u}_{k})|+2/3\\{\sup_{{\bf v}\in\mathbb{K}}|\Phi(3\Delta{\bf v},3\Delta{\bf
v})|\\}^{1/2}\\{\max_{i}|\Phi({\bf u}_{i},{\bf u}_{i})|\\}^{1/2}+1/9\sup_{{\bf
v},{\bf w}\in\mathbb{K}}|\Phi({\bf v},{\bf w})|$ $\displaystyle\leq$
$\displaystyle\max_{j,k}|\Phi({\bf u}_{j},{\bf u}_{k})|+\sup_{{\bf v},{\bf
w}\in\mathbb{K}}\\{2/3|\Phi({\bf v},{\bf w})|+1/9|\Phi({\bf v},{\bf w})|\\}.$

Hence, $\sup_{{\bf v},{\bf w}\in\mathbb{K}}|\Phi({\bf v},{\bf w})|\leq
9/2\max_{j,k}|\Phi({\bf u}_{j},{\bf u}_{k})|$. By a union bound while
considering $|\mathcal{B}|<9^{2(m+k)}$, we have

$\displaystyle\Pr\left(\sup_{{\bf v},{\bf
w}\in\mathbb{K}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}]|>9/2nt\right)$
$\displaystyle\leq$
$\displaystyle\Pr\left(\max_{j,k}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf
W}_{i})g({\bf W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm
T}E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf
w}]|>nt\right)$ $\displaystyle\leq$
$\displaystyle\Pr\left(\max_{j,k}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf
W}_{i})g({\bf W}_{i},\bm{\beta},{\bf v}/\|{\bf v}\|_{2},{\bf w}/\|{\bf
w}\|_{2})-{\bf v}/\|{\bf v}\|_{2}^{\rm T}E\\{\exp(\bm{\beta}^{\rm T}{\bf
X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}/\|{\bf w}\|_{2}]|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{nt^{2}}{16e^{2}M_{4}},\frac{nt}{8eM_{5}\hbox{log}(n)}\right]+2(m+k)\hbox{log}(9)\right).$

It also follows that

$\displaystyle\sup_{{\bf v}\in\mathbb{K}_{1},{\bf w}\in\mathbb{K}}|\Phi({\bf
v},{\bf w})|$ $\displaystyle\leq$ $\displaystyle\max_{{\bf
v}\in\mathbb{K}_{1},j}|\Phi({\bf v},{\bf u}_{j})|+1/3\sup_{{\bf
v}\in\mathbb{K}_{1},{\bf w}\in\mathbb{K}}|\Phi({\bf v},3\Delta{\bf w})|$
$\displaystyle\leq$ $\displaystyle\max_{{\bf v}\in\mathbb{K}_{1},j}|\Phi({\bf
v},{\bf u}_{j})|+1/3\sup_{{\bf v}\in\mathbb{K}_{1},{\bf
w}\in\mathbb{K}}\\{|\Phi({\bf v},{\bf w})|,$

so $\sup_{{\bf v}\in\mathbb{K}_{1},{\bf w}\in\mathbb{K}}|\Phi({\bf v},{\bf
w})|\leq 3/2\max_{{\bf v}\in\mathbb{K}_{1},j}|\Phi({\bf v},{\bf u}_{j})|$.
Because ${\bf v}\in\mathbb{K}_{1}$ is a vector with a single nonzero entry
$1$, there are only $p-m-k$ elements in $\mathbb{K}_{1}$. We thus have

$\displaystyle\Pr\left(\sup_{{\bf v}\in\mathbb{K}_{1},{\bf
w}\in\mathbb{K}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}]|>3/2nt\right)$
$\displaystyle\leq$ $\displaystyle\Pr\left(\max_{{\bf v}\in\mathbb{K}_{1},{\bf
u}_{j}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf u}_{j})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf u}_{j}]|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{nt^{2}}{16e^{2}M_{4}},\frac{nt}{8eM_{5}\hbox{log}(n)}\right]+(m+k)\hbox{log}(9)+\hbox{log}(p)\right).$

This proves the result. Further, since ${\mathbb{K}}_{1}$ contains only unit
vectors, we have

$\displaystyle\Pr\left(\sup_{{\bf v}\in\mathbb{K}_{1},{\bf
w}\in\mathbb{K}_{1}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}]|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{nt^{2}}{16e^{2}M_{4}},\frac{nt}{8eM_{5}\hbox{log}(n)}\right]+2\hbox{log}(p)\right).$

∎

######  Corollary B.2.

Assume that Conditions (C1), (C4) and (C6) hold and
$m+k=o\\{n/\hbox{log}^{2}(n)\\}$. If ${\bf X}_{i},{\bf
U}_{i}\in{\mathbb{R}}^{p}$, define ${\mathbb{K}}\equiv\\{{\bf
v}\in{\mathbb{R}}^{\mathcal{M}\cup S}:\|{\bf v}\|_{2}\leq 1\\}$,
${\mathbb{K}}_{1}\equiv\\{{\bf v}\in{\mathbb{R}}^{(\mathcal{M}\cup
S)^{c}}:\|{\bf v}\|_{2}=1,\|{\bf v}\|_{0}=1\\}$, there are
$a_{0}>0,a_{1}>0,b_{1}>0$ such that

$\displaystyle\Pr\left(\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$ $\displaystyle\left.-{\bf v}^{\rm
T}E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf
w}]|>na_{0}\sqrt{\frac{m+k}{n}}\right)$ $\displaystyle\leq$ $\displaystyle
2\exp\left\\{-(m+k)\right\\},$ $\displaystyle\Pr\left(\sup_{{\bf
v}\in{\mathbb{K}}_{1},{\bf w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm
T}{\bf W}_{i})g({\bf W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$
$\displaystyle\left.-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm T}{\bf
X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf
w}]|>na_{1}\sqrt{\frac{\max[m+k,\hbox{log}(p)]}{n}}\right)$
$\displaystyle\leq$ $\displaystyle 2\exp[-\max\\{\hbox{log}(p),(m+k)\\}],$

and

$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}_{1}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$ $\displaystyle\left.-{\bf v}^{\rm
T}E\\{\exp(\bm{\beta}^{\rm T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf
w}]|>nb_{1}\sqrt{\frac{\hbox{log}(p)}{n}}\right)$ $\displaystyle\leq$
$\displaystyle 2\exp[-\hbox{log}(p)].$

Proof: By Lemma B.7, take $t=a_{0}\sqrt{(m+k)/n}$, we have

$\displaystyle\Pr\left(\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}]|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{a_{0}^{2}(m+k)}{324e^{2}M_{4}},\frac{a_{0}\sqrt{n(m+k)}}{36eM_{5}\hbox{log}(n)}\right]+2(m+k)\hbox{log}(9)\right)$
$\displaystyle=$ $\displaystyle
2\exp\left(-\frac{a_{0}^{2}(m+k)}{324e^{2}M_{4}}+2(m+k)\right)$
$\displaystyle=$ $\displaystyle 2\exp\left\\{-(m+k)\right\\}$

The second to the last equality holds because
$m+k=o\\{n/\hbox{log}^{2}(n)\\}$. The last equality holds by choosing
$a_{0}=\sqrt{972e^{2}M_{4}}$.

Further, by the second relation in Lemma B.7, take
$t=a_{1}\sqrt{\max\\{\hbox{log}(p),(m+k)\\}/n}$, we have

$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}]|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{a_{1}^{2}\max\\{\hbox{log}(p),(m+k)\\}}{36e^{2}M_{4}},\right.\right.$
$\displaystyle\left.\left.\frac{a_{1}\sqrt{n\max\\{\hbox{log}(p),(m+k)\\}}}{12eM_{5}\hbox{log}(n)}\right]+(m+k)\hbox{log}(9)+\hbox{log}(p)\right)$
$\displaystyle=$ $\displaystyle
2\exp\left\\{-\min\left(\frac{a_{1}^{2}\max\\{\hbox{log}(p),(m+k)\\}}{36e^{2}M_{4}},\right.\right.$
$\displaystyle\left.a_{1}\max[\hbox{log}(p)\sqrt{n}/\\{12eM_{5}\sqrt{\hbox{log}(p)}\hbox{log}(n)\\},\sqrt{n(m+k)}/\\{12eM_{5}\hbox{log}(n)\\}]\right)$
$\displaystyle\left.+2(m+k)+\hbox{log}(p)\vphantom{\int_{0}^{1}}\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left(\frac{a_{1}^{2}\max\\{\hbox{log}(p),(m+k)\\}}{36e^{2}M_{4}},\right.\right.$
$\displaystyle\left.a_{1}\max[\hbox{log}(p)/(12eM_{5}C),\sqrt{n(m+k)}/\\{12eM_{5}\hbox{log}(n)\\}]\right)$
$\displaystyle\left.+2(m+k)+\hbox{log}(p)\vphantom{\int_{0}^{1}}\right\\}$
$\displaystyle\leq$ $\displaystyle
2\exp\left\\{-\min\left[\frac{a_{1}^{2}\max\\{\hbox{log}(p),(m+k)\\}}{36e^{2}M_{4}},\right.\right.$
$\displaystyle\left.\left.a_{1}\max\\{\hbox{log}(p)/(12eM_{5}C),(m+k)/(12eM_{5})\\}\right]+3\max\\{(m+k),\hbox{log}(p)\\}\vphantom{\int_{0}^{1}}\right\\}$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-a_{2}\max\\{\hbox{log}(p),(m+k)\\}+3\max\\{\hbox{log}(p),(m+k)\\}\right)$
$\displaystyle=$ $\displaystyle 2\exp[-\max\\{\hbox{log}(p),(m+k)\\}],$

where
$a_{2}=\min\\{a_{1}^{2}/(36e^{2}M_{4}),a_{1}/(12eM_{5}C),a_{1}/(12eM_{5})\\}$,
and we select $a_{1}$ such that $a_{2}\geq 4$. The third equality holds
because $\hbox{log}(n)\leq C\sqrt{n/\hbox{log}(p)}$ by Condition (C4). The
fourth equality holds $(m+k)=o\\{n/\hbox{log}^{2}(n)\\}$.

In addition, by the third relation in Lemma B.7, take
$t=b_{1}\sqrt{\hbox{log}(p)/n}$, we have

$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}_{1}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm
T}{\bf X}_{i}){\bf X}_{i}{\bf X}_{i}^{\rm T}\\}{\bf w}]|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{b_{1}^{2}\hbox{log}(p)}{16e^{2}M_{4}},\frac{b_{1}\sqrt{n\hbox{log}(p)}}{8eM_{5}\hbox{log}(n)}\right]+2\hbox{log}(p)\right)$
$\displaystyle=$ $\displaystyle
2\exp\left\\{-\min\left(\frac{b_{1}^{2}\hbox{log}(p)}{16e^{2}M_{4}},b_{1}\hbox{log}(p)\sqrt{n}/\\{8eM_{5}\sqrt{\hbox{log}(p)}\hbox{log}(n)\\}\right)+2\hbox{log}(p)\vphantom{\int_{0}^{1}}\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left(\frac{b_{1}^{2}\hbox{log}(p)}{16e^{2}M_{4}},b_{1}\hbox{log}(p)/(8eM_{5}C)\right)+2\hbox{log}(p)\vphantom{\int_{0}^{1}}\right\\}$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-b_{12}\hbox{log}(p)+2\hbox{log}(p)\right)$ $\displaystyle=$
$\displaystyle 2\exp[-\hbox{log}(p)],$

where $b_{12}=\min\\{b_{1}^{2}/(16e^{2}M_{4}),b_{1}/(8eM_{5}C)\\}$, and we
select $b_{1}$ such that $b_{12}\geq 3$. The second inequality holds because
$\hbox{log}(n)\leq C\sqrt{n/\hbox{log}(p)}$ by Condition (C4). ∎

######  Lemma B.8.

Assume that Conditions (C1) and (C7) hold. If ${\bf X}_{i},{\bf
U}_{i}\in{\mathbb{R}}^{p}$, define ${\mathbb{K}}\equiv\\{{\bf
v}\in{\mathbb{R}}^{\mathcal{M}\cup S}:\|{\bf v}\|_{2}\leq 1\\}$,
${\mathbb{K}}_{1}\equiv\\{{\bf v}\in{\mathbb{R}}^{(\mathcal{M}\cup
S)^{c}}:\|{\bf v}\|_{2}=1,\|{\bf v}\|_{0}=1\\}$. Then

$\displaystyle\Pr\left(\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A^{2}(\bm{\beta}^{\rm T}{\bf
W}_{i})g_{1}({\bf W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$
$\displaystyle\left.-{\bf v}^{\rm T}E\\{\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}\\}{\bf w}]>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\min\left[\frac{nt^{2}}{324e^{2}M_{6}},\frac{nt}{36eM_{7}\hbox{log}(n)}\right]+2(m+k)\hbox{log}(9)\right),$
$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A^{2}(\bm{\beta}^{\rm T}{\bf
W}_{i})g_{1}({\bf W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$
$\displaystyle\left.-{\bf v}^{\rm T}E\\{\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}\\}{\bf
w}]|>nt\vphantom{\int_{0}^{1}}\right)$ $\displaystyle\leq$ $\displaystyle
2\exp\left\\{-\min\left[\frac{nt^{2}}{36e^{2}M_{6}},\frac{nt}{12eM_{7}\hbox{log}(n)}\right]+(m+k)\hbox{log}(9)+\hbox{log}(p)\right\\},$

and

$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}_{1}}|\sum_{i=1}^{n}[A^{2}(\bm{\beta}^{\rm T}{\bf
W}_{i})g_{1}({\bf W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$
$\displaystyle\left.-{\bf v}^{\rm T}E\\{\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}\\}{\bf
w}]|>nt\vphantom{\int_{0}^{1}}\right)$ $\displaystyle\leq$ $\displaystyle
2\exp\left\\{-\min\left[\frac{nt^{2}}{16e^{2}M_{6}},\frac{nt}{8eM_{7}\hbox{log}(n)}\right]+2\hbox{log}(p)\right\\}.$

Proof: The lemma holds by using the same arguments as those lead to Lemma B.7.
∎

######  Corollary B.3.

Assume that Conditions (C1), (C4) (C7) hold, $m+k=o\\{n/\hbox{log}^{2}(n)\\}$.
If ${\bf X}_{i},{\bf U}_{i}\in{\mathbb{R}}^{p}$, define
${\mathbb{K}}\equiv\\{{\bf v}\in{\mathbb{R}}^{\mathcal{M}\cup S}:\|{\bf
v}\|_{2}\leq 1\\}$, ${\mathbb{K}}_{1}\equiv\\{{\bf
v}\in{\mathbb{R}}^{(\mathcal{M}\cup S)^{c}}:\|{\bf v}\|_{2}=1,\|{\bf
v}\|_{0}=1\\}$. There are $a_{2}>0,a_{3}>0,b_{3}>0$ such that

$\displaystyle\Pr\left(\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A^{2}(\bm{\beta}^{\rm T}{\bf
W}_{i})g_{1}({\bf W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$
$\displaystyle\left.-{\bf v}^{\rm T}E\\{\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}\\}{\bf
w}]|>na_{2}\sqrt{\frac{(m+k)}{n}}\right)$ $\displaystyle\leq$ $\displaystyle
2\exp\left\\{-(m+k)\right\\},$ $\displaystyle\Pr\left(\sup_{{\bf
v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A^{2}(\bm{\beta}^{\rm T}{\bf
W}_{i})g_{1}({\bf W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$
$\displaystyle-{\bf v}^{\rm T}E\\{\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}\\}{\bf w}]|$
$\displaystyle\left.>na_{3}\sqrt{\frac{\max[(m+k),\hbox{log}(p)]}{n}}\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left[-\max\\{\hbox{log}(p),(m+k)\\}\right].$

and

$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}_{1}}|\sum_{i=1}^{n}[A^{2}(\bm{\beta}^{\rm T}{\bf
W}_{i})g_{1}({\bf W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$
$\displaystyle-{\bf v}^{\rm T}E\\{\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}\\}{\bf w}]|$
$\displaystyle\left.>nb_{3}\sqrt{\frac{\hbox{log}(p)}{n}}\right)\leq
2\exp\\{-\hbox{log}(p)\\}.$

Proof: The corollary follows the same arguments as those lead to Corollary
B.2.∎

######  Corollary B.4.

Assume that Conditions (C1), (C4) (C7) hold, $m+k=o\\{n/\hbox{log}^{2}(n)\\}$.
If ${\bf X}_{i},{\bf U}_{i}\in{\mathbb{R}}^{p}$, define
${\mathbb{K}}\equiv\\{{\bf v}\in{\mathbb{R}}^{\mathcal{M}\cup S}:\|{\bf
v}\|_{2}\leq 1\\}$, ${\mathbb{K}}_{1}\equiv\\{{\bf
v}\in{\mathbb{R}}^{(\mathcal{M}\cup S)^{c}}:\|{\bf v}\|_{2}=1,\|{\bf
v}\|_{0}=1\\}$. There are $a_{21}>0,a_{31}>0,b_{31}>0$ such that

$\displaystyle\Pr\left(\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g_{1}({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$ $\displaystyle\left.-{\bf v}^{\rm
T}E\\{\exp(\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta}/2)({\bf W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes
2}\\}{\bf w}]|>na_{21}\sqrt{\frac{(m+k)}{n}}\right)$ $\displaystyle\leq$
$\displaystyle 2\exp\left\\{-(m+k)\right\\},$
$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf W}_{i})g_{1}({\bf
W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$ $\displaystyle-{\bf v}^{\rm
T}E\\{\exp(\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta}/2)({\bf W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes
2}\\}{\bf w}]|$
$\displaystyle\left.>na_{31}\sqrt{\frac{\max[(m+k),\hbox{log}(p)]}{n}}\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left[-\max\\{\hbox{log}(p),(m+k)\\}\right],$

and

$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}_{1}}|\sum_{i=1}^{n}[A(\bm{\beta}^{\rm T}{\bf
W}_{i})g_{1}({\bf W}_{i},\bm{\beta},{\bf v},{\bf w})\right.$
$\displaystyle-{\bf v}^{\rm T}E\\{\exp(\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta}/2)({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}\\}{\bf w}]|$
$\displaystyle\left.>nb_{31}\sqrt{\frac{\hbox{log}(p)}{n}}\right)\leq
2\exp\\{-\hbox{log}(p)\\}.$

Proof: The corollary follows the same arguments as those that lead to
Corollary B.3.∎

######  Lemma B.9.

Assume that Conditions (C1) holds, $m+k=o\\{n/\hbox{log}^{2}(n)\\}$. If ${\bf
X}_{i},{\bf U}_{i}\in{\mathbb{R}}^{p}$, ${\mathbb{K}}\equiv\\{{\bf
v}\in{\mathbb{R}}^{\mathcal{M}\cup S}:\|{\bf v}\|_{2}=1\\}$,
${\mathbb{K}}_{1}\equiv\\{{\bf v}\in{\mathbb{R}}^{(\mathcal{M}\cup
S)^{c}}:\|{\bf v}\|_{2}=1,\|{\bf v}\|_{0}=1\\}$. Then there exists constants
$c_{2},c_{3},c_{4},c_{5},c_{6},c_{7},c_{8},c_{9},c_{10}>0$

$\displaystyle\Pr\left(\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w}-E({\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w})\\}|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-c_{2}n\min(c_{3}t^{2},c_{4}t)+2(m+k)\hbox{log}(9)\right),$
$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w}-E({\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w})\\}|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-c_{7}n\min(c_{5}t^{2},c_{6}t)+(m+k)\hbox{log}(9)+\hbox{log}(p)\right),$

and

$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}_{1}}|\sum_{i=1}^{n}\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w}-E({\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w})\\}|>nt\right)$
$\displaystyle\leq$ $\displaystyle
2\exp\left(-c_{10}n\min(c_{8}t^{2},c_{9}t)+2\hbox{log}(p)\right).$

Proof: The lemma is a consequence of Lemma 15 in Loh & Wainwright (2012) and
using the same arguments as those lead to Lemma B.7.∎

######  Corollary B.5.

Assume that Conditions (C1) holds. If ${\bf X}_{i},{\bf
U}_{i}\in{\mathbb{R}}^{p}$, ${\mathbb{K}}\equiv\\{{\bf
v}\in{\mathbb{R}}^{\mathcal{M}\cup S}:\|{\bf v}\|_{2}\leq 1\\}$,
${\mathbb{K}}_{1}\equiv\\{{\bf v}\in{\mathbb{R}}^{(\mathcal{M}\cup
S)^{c}}:\|{\bf v}\|_{2}=1,\|{\bf v}\|_{0}=1\\}$. There are constants
$a_{4}>0,a_{5}>0,b_{5}>0$ such that

$\displaystyle\Pr\left(\sup_{{\bf v},{\bf
w}\in{\mathbb{K}}}|\sum_{i=1}^{n}\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w}-E({\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w})\\}|\right.$
$\displaystyle\left.>na_{4}\sqrt{\frac{(m+k)}{n}}\right)\leq
2\exp\left\\{-(m+k)\right\\},$ $\displaystyle\Pr\left(\sup_{{\bf
v}\in{\mathbb{K}}_{1},{\bf w}\in{\mathbb{K}}}|\sum_{i=1}^{n}\\{{\bf v}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w}-E({\bf v}^{\rm
T}({\bf W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w})\\}|\right.$
$\displaystyle\left.>na_{5}\sqrt{\frac{\max[(m+k),\hbox{log}(p)]}{n}}\right)\leq
2\exp[-\max\\{\hbox{log}(p),(m+k)\\}],$

and

$\displaystyle\Pr\left(\sup_{{\bf v}\in{\mathbb{K}}_{1},{\bf
w}\in{\mathbb{K}}_{1}}|\sum_{i=1}^{n}\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w}-E({\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}{\bf w})\\}|\right.$
$\displaystyle\left.>nb_{5}\sqrt{\frac{\hbox{log}(p)}{n}}\right)\leq
2\exp\\{-\hbox{log}(p)\\}.$

Proof: The corollary follows the same arguments as those lead to Corollary
B.2.∎

##  B.3 Lemmas on Criterion Function and Penalty Function

######  Lemma B.10.

Consider a $\mu$-amenable regularizer $\rho_{\lambda}$, with $\mu<\alpha_{1}$
and $n>\hbox{log}(p)\tau_{1}^{2}(m+k)^{2}/(\alpha_{1}-\mu)^{2}$, where
$\alpha_{1},\tau_{1}$ are defined in Lemma B.6. Then the function
$\mathcal{L}(\bm{\beta})-\mu\|\bm{\beta}_{\mathcal{M}^{c}}\|_{2}^{2}/2$ and
$\mathcal{L}(\bm{\beta})+\rho_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})$ are
strictly convex on $\bm{\beta}\in\mathbb{R}^{\mathcal{M}\cup S}$, and hence
the restricted program (14) is also strictly convex.

Proof: First define a vector ${\bf v}\in{\mathbb{R}}^{p}$ with the $j$th
element $|v_{j}|>0$ if $j\in\mathcal{M}\cup S$, and $\|v_{j}\|=0$, otherwise.
By Lemma B.6, we have for $\bm{\beta}$ in the feasible set of program (14),

$\displaystyle{\bf v}^{\rm
T}\frac{\partial^{2}\mathcal{L}(\bm{\beta})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}{\bf v}\geq\alpha_{1}\|{\bf
v}\|_{2}^{2}-\tau_{1}\sqrt{\frac{\hbox{log}(p)}{n}}\|{\bf v}\|_{1}^{2},$

and $\|{\bf v}\|_{1}\leq\sqrt{m+k}\|{\bf v}\|_{2}$, and hence we have

$\displaystyle{\bf v}^{\rm
T}\frac{\partial^{2}\mathcal{L}(\bm{\beta})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}{\bf
v}\geq\left\\{\alpha_{1}-\tau_{1}(m+k)\sqrt{\frac{\hbox{log}(p)}{n}}\right\\}\|{\bf
v}\|_{2}^{2}.$

Therefore,

$\displaystyle{\bf v}_{\mathcal{M}\cup S}^{\rm
T}\left\\{\frac{\partial^{2}\mathcal{L}(\bm{\beta})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}\right\\}_{(\mathcal{M}+S)(\mathcal{M}+S)}{\bf v}_{\mathcal{M}\cup
S}-\mu{\bf v}_{S}^{\rm T}{\bf v}_{S}$ $\displaystyle\geq$
$\displaystyle\left\\{\alpha_{1}-\mu-\tau_{1}(m+k)\sqrt{\frac{\hbox{log}(p)}{n}}\right\\}\|{\bf
v}\|_{2}^{2},$

where
$\left\\{\partial^{2}\mathcal{L}(\bm{\beta})/(\partial\bm{\beta}\partial\bm{\beta}^{\rm
T})\right\\}_{(\mathcal{M}\cup S),(\mathcal{M}\cup S)}$ is the
$(m+k)\times(m+k)$ block of
$\left\\{\partial^{2}\mathcal{L}(\bm{\beta})/\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}\right\\}$ corresponding to $\mathcal{M}\cup S$. Hence
$\mathcal{L}(\bm{\beta})-\mu\|\bm{\beta}_{\mathcal{M}^{c}}\|_{2}^{2}/2$ is
strictly convex on $\mathbb{R}^{\mathcal{M}\cup S}$.

Finally, because

$\displaystyle\mathcal{L}(\bm{\beta})-q_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})=(\mathcal{L}(\bm{\beta})-\mu\|\bm{\beta}_{\mathcal{M}^{c}}\|_{2}^{2}/2)+\\{\mu\|\bm{\beta}_{\mathcal{M}^{c}}\|_{2}^{2}/2-q_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})\\},$

where the second part is convex over $\mathbb{R}^{\mathcal{M}\cup S}$ by Lemma
B.4, hence $\mathcal{L}(\bm{\beta})-q_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})$
restricted to $\mathbb{R}^{\mathcal{M}\cup S}$ is strictly convex. Because
$\mathcal{L}(\bm{\beta})+\rho_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})=\mathcal{L}(\bm{\beta})-q_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})+\lambda\|\bm{\beta}_{\mathcal{M}^{c}}\|_{1}$,
the strict convexity of
$\mathcal{L}(\bm{\beta})+\rho_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})$ over
$\mathbb{R}^{\mathcal{M}\cup S}$ follows. This proves the result. ∎

Now as we know ${\bf C}$ contains $r$ independent columns, without loss of
generality, we write ${\bf C}=({\bf C}_{r},{\bf C}_{m-r})$ where ${\bf C}_{r}$
is a full rank square matrix.

######  Lemma B.11.

Let ${\bf A},{\bf B}\in\mathbb{R}^{p\times p}$ be invertible. For any matrix
norm $\|\|$, we have

$\displaystyle\|{\bf A}^{-1}-{\bf B}^{-1}\|\leq\frac{\|{\bf
A}^{-1}\|^{2}\|{\bf A}-{\bf B}\|}{1-\|{\bf A}^{-1}\|\|{\bf A}-{\bf B}\|}.$

In particular, if $\|{\bf A}^{-1}\|\|{\bf A}-{\bf B}\|\leq 1/2$, then $\|{\bf
A}^{-1}-{\bf B}^{-1}\|=O(\|{\bf A}^{-1}\|^{2}\|{\bf A}-{\bf B}\|)$.

Proof: This lemma is Lemma 11 in Loh & Wainwright (2017). ∎

######  Lemma B.12.

Suppose ${\bf x}^{*}$ is feasible for the program

$\displaystyle\min_{{\bf x}}\\{f({\bf x})-g({\bf
x}_{\mathcal{M}^{c}})+\lambda\|{\bf x}_{\mathcal{M}^{c}}\|_{1}\\},\text{ such
that }\|{\bf x}\|_{1}\leq R_{1},\|{\bf x}\|_{2}\leq R_{2},\text{ and }{\bf
C}{\bf x}_{\mathcal{M}}={\bf t},$ (B.1)

where $f\in C^{2}$, $g\in C^{1}$ and $g({\bf
x}_{\mathcal{M}^{c}})-\kappa/2\|{\bf x}_{\mathcal{M}^{c}}\|_{2}^{2}$ is
concave and ${\bf C}$ is an $r\times m$ matrix. Define ${\bf A}=\\{{\bf
0}_{(p-m)\times m},{\bf I}_{(p-m)\times(p-m)}\\}$ and ${\bf A}_{1}=\\{{\bf
I}_{m\times m},{\bf 0}_{m\times(p-m)}\\}$. Assume there are ${\bf
v}^{*}\in\partial\|{\bf x}^{*}_{\mathcal{M}^{c}}\|_{1}$, ${\bf
w}_{1}^{*}\in\partial\|{\bf x}^{*}\|_{1}$, ${\bf w}_{2}^{*}\in\partial\|{\bf
x}^{*}\|_{2}$, $\mu_{1}^{*}\geq 0$, $\mu_{2}^{*}\geq 0$, $\bm{\mu}_{3}^{*}$
such that

$\displaystyle\mu_{1}^{*}(R_{1}-\|{\bf x}^{*}\|_{1})=0$ (B.2)
$\displaystyle\mu_{2}^{*}(R_{2}-\|{\bf x}^{*}\|_{2})=0$ (B.3)
$\displaystyle\frac{\partial f({\bf x}^{*})}{\partial{\bf x}^{*}}-{\bf A}^{\rm
T}\frac{\partial g({\bf x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf
x}^{*}_{\mathcal{M}^{c}}}+\lambda{\bf A}^{\rm T}{\bf v}^{*}+\mu_{1}^{*}{\bf
w}_{1}^{*}+\mu_{2}^{*}{\bf w}_{2}^{*}+{\bf A}_{1}^{\rm T}{\bf C}^{\rm
T}\bm{\mu}_{3}^{*}=0$ (B.4) $\displaystyle{\bf s}^{\rm
T}\frac{\partial^{2}f({\bf x})}{\partial{\bf x}\partial{\bf x}^{\rm T}}{\bf
s}>\kappa,\forall{\bf s}\in G^{*},$ (B.5)

where

$\displaystyle G^{*}$ $\displaystyle:=$ $\displaystyle\left\\{{\bf
s}\in{\mathbb{R}}^{p}:\|{\bf s}\|_{2}=1;\sup_{{\bf w}_{1}\in\partial\|{\bf
x}^{*}\|_{1}}{\bf s}^{\rm T}{\bf w}_{1}\leq 0\text{ if }\|{\bf
x}^{*}\|_{1}=R_{1};\sup_{{\bf w}_{2}\in\partial\|{\bf x}^{*}\|_{2}}{\bf
s}^{\rm T}{\bf w}_{2}\leq 0\text{ if }\|{\bf x}^{*}\|_{2}=R_{2};\right.$
$\displaystyle\sup_{{\bf v}\in\partial\|{\bf
x}^{*}_{\mathcal{M}^{c}}\|_{1}}{\bf s}^{\rm T}\left(\frac{\partial f({\bf
x}^{*})}{\partial{\bf x}^{*}}-{\bf A}^{\rm T}\frac{\partial g({\bf
x}_{\mathcal{M}^{c}}^{*})}{\partial{\bf
x}_{\mathcal{M}^{c}}^{*}}\right)+\lambda{\bf s}^{\rm T}{\bf A}^{\rm T}{\bf
v}={0};$ $\displaystyle\left.\mu_{1}^{*}\sup_{{\bf w}_{1}\in\partial\|{\bf
x}^{*}\|_{1}}{\bf s}^{\rm T}{\bf w}_{1}=0,\mu_{2}^{*}\sup_{{\bf
w}_{2}\in\partial\|{\bf x}^{*}\|_{2}}{\bf s}^{\rm T}{\bf w}_{2}=0;{\bf C}{\bf
s}_{\mathcal{M}}=\bf 0\right\\}.$

Then ${\bf x}^{*}$ is an isolated local minimum of the program (B.1).

Proof: The proof of this lemma is similar to the proof of Theorem 3 in
Fletcher & Watson (1980) and that of Lemma 10 in Loh & Wainwright (2017),
except that we allow additional constraints $\|{\bf x}\|_{2}\leq R_{2}$ and
${\bf C}{\bf x}_{\mathcal{M}}={\bf t}$.

Suppose ${\bf x}^{*}$ is not an isolated local minimum. Then there is a
sequence $({\bf x}^{k})$, so that ${\bf x}^{k}\to{\bf x}^{*}$ and

$\displaystyle\phi({\bf x}^{k})\leq\phi({\bf x}^{*}),$

where $\phi({\bf x})=f({\bf x})-g({\bf x}_{\mathcal{M}^{c}})+\lambda\|{\bf
x}_{\mathcal{M}^{c}}\|_{1}$. Let ${\bf s}^{k}:=({\bf x}^{k}-{\bf
x}^{*})/\|{\bf x}^{k}-{\bf x}^{*}\|_{2}$, so $({\bf s}^{k})$ is a set of
feasible directions. Since $({\bf s}^{k})\subset\mathbb{B}_{2}(1)$, where
$\mathbb{B}_{2}(1)$ is the ball with radius 1, the set must possess a point of
accumulation ${\bf s}\in\mathbb{B}_{2}(1)$, and we can extract a convergence
subsequence such that $({\bf s}^{k})\to{\bf s}$. With a slight abuse of
notation, we still use $({\bf s}^{k})$ to denote the subsequence. We will show
that ${\bf s}\in G^{*}$.

First of all by the construction, ${\bf x}^{k}$’s are all feasible, and hence
${\bf C}{\bf x}^{k}_{\mathcal{M}}={\bf t}$. Therefore, ${\bf C}{\bf
s}^{k}_{\mathcal{M}}=0$, take the limits on the left and right of the equation
implies

$\displaystyle{\bf C}{\bf s}_{\mathcal{M}}=\bf 0.$ (B.6)

Further, because the feasible region is closed, ${\bf s}$ is also in the
feasible direction at ${\bf x}^{*}$. If $\|{\bf x}^{*}\|_{1}=R_{1}$, by the
sub-gradient of convexity function $\|{\bf x}^{*}\|_{1}$ we have

$\displaystyle 0\geq\|{\bf x}^{k}\|_{1}-\|{\bf x}^{*}\|_{1}=\|{\bf
x}^{*}+\|{\bf x}^{k}-{\bf x}^{*}\|_{2}{\bf s}^{k}\|_{1}-\|{\bf
x}^{*}\|_{1}\geq\|{\bf x}^{k}-{\bf x}^{*}\|_{2}{{\bf s}^{k}}^{\rm T}{\bf
w}_{1},$

for any ${\bf w}_{1}\in\partial\|{\bf x}^{*}\|_{1}$. When $k\to\infty$, this
leads to

$\displaystyle\sup_{{\bf w}_{1}\in\partial\|{\bf x}^{*}\|_{1}}{\bf s}^{\rm
T}{\bf w}_{1}\leq 0.$ (B.7)

Further (B.2) also implies that if $\|{\bf x}^{*}\|_{1}\neq R_{1}$, then
$\mu_{1}^{*}=0$. Since $\mu_{1}^{*}\geq 0$, hence we have

$\displaystyle\mu_{1}^{*}\sup_{{\bf w}_{1}\in\partial\|{\bf x}^{*}\|_{1}}{\bf
s}^{\rm T}{\bf w}_{1}\leq 0.$ (B.8)

Similarly, by the sub-gradient of convexity function $\|{\bf x}^{*}\|_{2}$ we
have

$\displaystyle 0\geq\|{\bf x}^{k}\|_{2}-\|{\bf x}^{*}\|_{2}=\|{\bf
x}^{*}+\|{\bf x}^{k}-{\bf x}^{*}\|_{2}{\bf s}^{k}\|_{2}-\|{\bf
x}^{*}\|_{2}\geq\|{\bf x}^{k}-{\bf x}^{*}\|_{2}{{\bf s}^{k}}^{\rm T}{\bf
w}_{2},$

for any ${\bf w}_{2}\in\partial\|{\bf x}^{*}\|_{2}$. When $k\to\infty$, this
leads to

$\displaystyle\sup_{{\bf w}_{2}\in\partial\|{\bf x}^{*}\|_{2}}{\bf s}^{\rm
T}{\bf w}_{2}\leq 0.$ (B.9)

Further (B.3) also implies that if $\|{\bf x}^{*}\|_{2}\neq R_{2}$, then
$\mu_{2}^{*}=0$. Since $\mu_{2}^{*}\geq 0$, hence we have

$\displaystyle\mu_{2}^{*}\sup_{{\bf w}_{2}\in\partial\|{\bf x}^{*}\|_{2}}{\bf
s}^{\rm T}{\bf w}_{2}\leq 0.$ (B.10)

Further by (B.4), we have

$\displaystyle{\bf s}^{\rm T}\frac{\partial f({\bf x}^{*})}{\partial{\bf
x}^{*}}-{\bf s}^{\rm T}{\bf A}^{\rm T}\frac{\partial g({\bf
x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf x}^{*}_{\mathcal{M}^{c}}}+{\bf s}^{\rm
T}\lambda{\bf A}^{\rm T}{\bf v}^{*}+\mu_{1}^{*}{\bf s}^{\rm T}{\bf
w}_{1}^{*}+\mu_{2}^{*}{\bf s}^{\rm T}{\bf w}_{2}^{*}+{\bf s}^{\rm T}{\bf
A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{3}^{*^{\rm T}}=0$

which, together with (B.6), implies

$\displaystyle{\bf s}^{\rm T}\frac{\partial f({\bf x}^{*})}{\partial{\bf
x}^{*}}-{\bf s}^{\rm T}{\bf A}^{\rm T}\frac{\partial g({\bf
x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf x}^{*}_{\mathcal{M}^{c}}}+{\bf s}^{\rm
T}\lambda{\bf A}^{\rm T}{\bf v}^{*}=-\mu_{1}^{*}{\bf s}^{\rm T}{\bf
w}_{1}^{*}-\mu_{2}^{*}{\bf s}^{\rm T}{\bf w}_{2}^{*}\geq 0.$ (B.11)

By the definition of sub-gradient, we have

$\displaystyle\|{\bf x}_{\mathcal{M}^{c}}^{*}+\|{\bf x}^{k}-{\bf
x}^{*}\|_{2}{\bf A}{\bf s}^{k}\|_{1}-\|{\bf
x}_{\mathcal{M}^{c}}^{*}\|_{1}\geq\|{\bf x}^{k}-{\bf x}^{*}\|_{2}{\bf s}^{k\rm
T}{\bf A}^{\rm T}{\bf v}$

for all ${\bf v}\in\partial\|{\bf x}_{\mathcal{M}^{c}}^{*}\|_{1}$ and all $k$.
Further because $\|{\bf x}_{\mathcal{M}^{c}}^{*}+\|{\bf x}^{k}-{\bf
x}^{*}\|_{2}{\bf A}{\bf s}^{k}\|_{1}-\|{\bf
x}_{\mathcal{M}^{c}}^{*}\|_{1}=\|{\bf x}_{\mathcal{M}^{c}}^{k}\|_{1}-\|{\bf
x}_{\mathcal{M}^{c}}^{*}\|_{1}$, we have

$\displaystyle{\bf s}^{\rm T}{\bf A}^{\rm T}{\bf v}=\lim_{k\to\infty}{\bf
s}^{k\rm T}{\bf A}^{\rm T}{\bf v}\leq\lim_{k\to\infty}\frac{\|{\bf
x}_{\mathcal{M}^{c}}^{k}\|_{1}-\|{\bf x}_{\mathcal{M}^{c}}^{*}\|_{1}}{\|{\bf
x}^{k}-{\bf x}^{*}\|_{2}}$ (B.12)

for all ${\bf v}\in\partial\|{\bf x}_{\mathcal{M}^{c}}^{*}\|_{1}$.
Furthermore,

$\displaystyle{\bf s}^{\rm T}\frac{\partial f({\bf x}^{*})}{\partial{\bf
x}^{*}}-{\bf s}^{\rm T}{\bf A}^{\rm T}\frac{\partial g({\bf
x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf x}^{*}_{\mathcal{M}^{c}}}$ (B.13)
$\displaystyle=$ $\displaystyle\lim_{k\to\infty}{\bf s}^{k\rm T}\frac{\partial
f({\bf x}^{*})}{\partial{\bf x}^{*}}-{\bf s}^{k\rm T}{\bf A}^{\rm
T}\frac{\partial g({\bf x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf
x}^{*}_{\mathcal{M}^{c}}}$ $\displaystyle=$
$\displaystyle\lim_{k\to\infty}\frac{\langle{\bf x}^{k}-{\bf
x}^{*},\frac{\partial f({\bf x}^{*})}{\partial{\bf x}^{*}}-{\bf A}^{\rm
T}\frac{\partial g({\bf x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf
x}^{*}_{\mathcal{M}^{c}}}\rangle}{\|{\bf x}^{k}-{\bf x}^{*}\|_{2}}$
$\displaystyle=$ $\displaystyle\lim_{k\to\infty}\frac{f({\bf x}^{k})-g({\bf
x}_{\mathcal{M}^{c}}^{k})-f({\bf x}^{*})+g({\bf
x}_{\mathcal{M}^{c}}^{*})}{\|{\bf x}^{k}-{\bf x}^{*}\|_{2}},$

since ${\bf x}^{k}\to{\bf x}^{*}$ and $f({\bf x}^{k})-g({\bf
x}_{\mathcal{M}^{c}}^{k})\in C^{1}$. Combining (B.12) and (B.13), we conclude
that

$\displaystyle\sup_{{\bf v}\in\partial\|{\bf
x}_{\mathcal{M}^{c}}^{*}\|_{1}}{\bf s}^{\rm T}\frac{\partial f({\bf
x}^{*})}{\partial{\bf x}^{*}}-{\bf s}^{\rm T}{\bf A}^{\rm T}\frac{\partial
g({\bf x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf
x}^{*}_{\mathcal{M}^{c}}}+\lambda{\bf s}^{\rm T}{\bf A}^{\rm T}{\bf v}$
$\displaystyle\leq$ $\displaystyle\lim_{k\to\infty}\frac{\phi({\bf
x}^{k})-\phi({\bf x}^{*})}{\|{\bf x}^{k}-{\bf x}*\|}\leq 0.$

Combining with (B.11), we have

$\displaystyle 0$ $\displaystyle\leq$ $\displaystyle{\bf s}^{\rm
T}\frac{\partial f({\bf x}^{*})}{\partial{\bf x}^{*}}-{\bf s}^{\rm T}{\bf
A}^{\rm T}\frac{\partial g({\bf x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf
x}^{*}_{\mathcal{M}^{c}}}+\lambda{\bf s}^{\rm T}{\bf A}^{\rm T}{\bf v}^{*}$
$\displaystyle\leq$ $\displaystyle\sup_{{\bf v}\in\partial\|{\bf
x}^{*}_{\mathcal{M}^{c}}\|_{1}}{\bf s}^{\rm T}\frac{\partial f({\bf
x}^{*})}{\partial{\bf x}^{*}}-{\bf s}^{\rm T}{\bf A}^{\rm T}\frac{\partial
g({\bf x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf
x}^{*}_{\mathcal{M}^{c}}}+\lambda{\bf s}^{\rm T}{\bf A}^{\rm T}{\bf v}$
$\displaystyle\leq$ $\displaystyle 0,$

hence

$\displaystyle\sup_{{\bf v}\in\partial\|{\bf
x}^{*}_{\mathcal{M}^{c}}\|_{1}}{\bf s}^{\rm T}\frac{\partial f({\bf
x}^{*})}{\partial{\bf x}^{*}}-{\bf s}^{\rm T}{\bf A}^{\rm T}\frac{\partial
g({\bf x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf
x}^{*}_{\mathcal{M}^{c}}}+\lambda{\bf s}^{\rm T}{\bf A}^{\rm T}{\bf v}=0$
(B.14)

Now together with (B.4), we have

$\displaystyle\mu_{1}^{*}{\bf s}^{\rm T}{\bf w}_{1}^{*}+\mu_{2}^{*}{\bf
s}^{\rm T}{\bf w}_{2}^{*}=0.$

Further by (B.8) and (B.10), we have

$\displaystyle\mu_{1}^{*}\sup_{{\bf w}_{1}\in\partial\|{\bf x}^{*}\|_{1}}{\bf
s}^{\rm T}{\bf w}_{1}=\mu_{2}^{*}\sup_{{\bf w}_{2}\in\partial\|{\bf
x}^{*}\|_{2}}{\bf s}^{\rm T}{\bf w}_{2}=0.$ (B.15)

Combine (B.6), (B.7), (B.9), (B.14) and (B.15), we conclude that ${\bf s}\in
G^{*}$.

By the convexity of the $L_{1}$ norm, we have

$\displaystyle\|{\bf x}^{k}\|_{1}-\|{\bf x}^{*}\|_{1}=\|{\bf x}^{*}+({\bf
x}^{k}-{\bf x}^{*})\|_{1}-\|{\bf x}^{*}\|_{1}\geq({\bf x}^{k}-{\bf
x}^{*})^{\rm T}{\bf w}_{1}={\bf x}^{k\rm T}{\bf w}_{1}-\|{\bf x}^{*}\|_{1}$

for all ${\bf w}_{1}\in\partial\|{\bf x}^{*}\|_{1}$. Therefore, ${\bf x}^{k\rm
T}{\bf w}_{1}^{*}\leq\|{\bf x}^{k}\|_{1}\leq R_{1}$. Similarly, by the
convexity of the $L_{2}$ norm, we have

$\displaystyle\|{\bf x}^{k}\|_{2}-\|{\bf x}^{*}\|_{2}=\|{\bf x}^{*}+({\bf
x}^{k}-{\bf x}^{*})\|_{2}-\|{\bf x}^{*}\|_{2}\geq({\bf x}^{k}-{\bf
x}^{*})^{\rm T}{\bf w}_{2}={\bf x}^{k\rm T}{\bf w}_{2}-\|{\bf x}^{*}\|_{2}$

for all ${\bf w}_{2}\in\partial\|{\bf x}^{*}\|_{2}$. Therefore, ${\bf x}^{k\rm
T}{\bf w}_{2}^{*}\leq\|{\bf x}^{k}\|_{2}\leq R_{2}$. Further, ${\bf x}^{k\rm
T}{\bf A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{3}^{*}-{\bf t}^{\rm
T}\bm{\mu}_{3}^{*}=0$. Hence

$\displaystyle\phi({\bf x}^{k})$ $\displaystyle=$ $\displaystyle f({\bf
x}^{k})-g({\bf x}_{\mathcal{M}^{c}}^{k})+\lambda\|{\bf
x}_{\mathcal{M}^{c}}^{k}\|_{1}$ $\displaystyle\geq$ $\displaystyle f({\bf
x}^{k})-g({\bf x}_{\mathcal{M}^{c}}^{k})+\lambda{\bf
x}_{\mathcal{M}^{c}}^{k\rm T}{\bf v}^{*}+\mu_{1}^{*}({\bf x}^{k\rm T}{\bf
w}_{1}^{*}-R_{1})+\mu_{2}^{*}({\bf x}^{k\rm T}{\bf w}_{2}^{*}-R_{2})$
$\displaystyle+{\bf x}^{k\rm T}{\bf A}_{1}^{\rm T}{\bf C}^{\rm
T}\bm{\mu}_{3}^{*}-{\bf t}^{\rm T}\bm{\mu}_{3}^{*}$

for all ${\bf v}^{*}\in\partial\|{\bf x}_{\mathcal{M}^{c}}^{*}\|_{1}$, and

$\displaystyle\phi({\bf x}^{*})$ $\displaystyle=$ $\displaystyle f({\bf
x}^{*})-g({\bf x}_{\mathcal{M}^{c}}^{*})+\lambda{\bf
x}_{\mathcal{M}^{c}}^{*\rm T}{\bf v}^{*}+\mu_{1}^{*}({\bf x}^{*\rm T}{\bf
w}_{1}^{*}-R_{1})+\mu_{2}^{*}({\bf x}^{*\rm T}{\bf w}_{2}^{*}-R_{2})$
$\displaystyle+{\bf x}^{*\rm T}{\bf A}_{1}^{\rm T}{\bf C}^{\rm
T}\bm{\mu}_{3}^{*}-{\bf t}^{\rm T}\bm{\mu}_{3}^{*}.$

The equality holds because $\mu_{1}^{*}({\bf x}^{*\rm T}{\bf
w}_{1}^{*}-R_{1})=0$ and $\mu_{2}^{*}({\bf x}^{*\rm T}{\bf
w}_{2}^{*}-R_{2})=0$. Hence

$\displaystyle\lim_{k\to\infty}\\{\phi({\bf x}^{k})-\phi({\bf
x}^{*})\\}/\|{\bf x}^{k}-{\bf x}^{*}\|_{2}^{2}$ (B.16) $\displaystyle\geq$
$\displaystyle\lim_{k\to\infty}\\{f({\bf x}^{k})-f({\bf x}^{*})-g({\bf
x}_{\mathcal{M}^{c}}^{k})+g({\bf x}_{\mathcal{M}^{c}}^{*})+\langle\lambda{\bf
A}^{\rm T}{\bf v}^{*}+\mu_{1}^{*}{\bf w}_{1}^{*}+\mu_{2}^{*}{\bf w}_{2}^{*}$
$\displaystyle+{\bf A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{3}^{*},{\bf
x}^{k}-{\bf x}^{*}\rangle\\}/\|{\bf x}^{k}-{\bf x}^{*}\|_{2}^{2}$
$\displaystyle=$ $\displaystyle\lim_{k\to\infty}\\{f({\bf x}^{k})-f({\bf
x}^{*})-g({\bf x}_{\mathcal{M}^{c}}^{k})+g({\bf
x}_{\mathcal{M}^{c}}^{*})\\}/\|{\bf x}^{k}-{\bf x}^{*}\|_{2}^{2}$
$\displaystyle-\left\langle\frac{\partial f({\bf x}^{*})}{\partial{\bf
x}^{*}}-{\bf A}^{\rm T}\frac{\partial g({\bf
x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf x}^{*}_{\mathcal{M}^{c}}},{\bf
x}^{k}-{\bf x}^{*}\right\rangle/\|{\bf x}^{k}-{\bf x}^{*}\|_{2}^{2}$
$\displaystyle=$ $\displaystyle\lim_{k\to\infty}\left\\{f({\bf x}^{k})-f({\bf
x}^{*})-\left\langle\frac{\partial f({\bf x}^{*})}{\partial{\bf x}^{*}},{\bf
x}^{k}-{\bf x}^{*}\right\rangle\right\\}/\|{\bf x}^{k}-{\bf x}^{*}\|_{2}^{2}$
$\displaystyle-\left\\{g({\bf x}_{\mathcal{M}^{c}}^{k})-g({\bf
x}_{\mathcal{M}^{c}}^{*})-\left\langle{\bf A}^{\rm T}\frac{\partial g({\bf
x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf x}^{*}_{\mathcal{M}^{c}}},{\bf
x}^{k}-{\bf x}^{*}\right\rangle\right\\}/\|{\bf x}^{k}-{\bf x}^{*}\|_{2}^{2}.$