By the concavity of $g({\bf x}_{\mathcal{M}^{c}})-\kappa/2\|{\bf
x}_{\mathcal{M}^{c}}\|^{2}_{2}$, we have

$\displaystyle\left\\{g({\bf x}_{\mathcal{M}^{c}}^{k})-g({\bf
x}_{\mathcal{M}^{c}}^{*})-\left\langle{\bf A}^{\rm T}\frac{\partial g({\bf
x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf x}^{*}_{\mathcal{M}^{c}}},{\bf
x}^{k}-{\bf x}^{*}\right\rangle\right\\}\leq\kappa/2\|{\bf
x}^{k}_{\mathcal{M}^{c}}-{\bf x}^{*}_{\mathcal{M}^{c}}\|_{2}^{2}.$

Further note that $\phi({\bf x}^{k})-\phi({\bf x}^{*})\leq 0$. Combine with
(B.16), we have

$\displaystyle\lim_{k\to\infty}\left\\{f({\bf x}^{k})-f({\bf
x}^{*})-\left\langle\frac{\partial f({\bf x}^{*})}{\partial{\bf x}^{*}},{\bf
x}^{k}-{\bf x}^{*}\right\rangle\right\\}/\|{\bf x}^{k}-{\bf x}^{*}\|_{2}^{2}$
$\displaystyle-\kappa/2\|{\bf x}^{k}_{\mathcal{M}^{c}}-{\bf
x}^{*}_{\mathcal{M}^{c}}\|_{2}^{2}/\|{\bf x}^{k}-{\bf x}^{*}\|_{2}^{2}\leq 0,$

which by Taylor expansion implies

$\displaystyle{\bf s}^{\rm T}\frac{\partial^{2}f({\bf x}^{*})}{\partial{\bf
x}\partial{\bf x}^{\rm T}}{\bf s}$ $\displaystyle=$
$\displaystyle\lim_{k\to\infty}({\bf x}^{k}-{\bf x}^{*})^{\rm
T}\frac{\partial^{2}f({\bf x}^{*})}{\partial{\bf x}\partial{\bf x}^{\rm
T}}({\bf x}^{k}-{\bf x}^{*})/\|{\bf x}^{k}-{\bf x}^{*}\|_{2}^{2}$
$\displaystyle\leq$ $\displaystyle\lim_{k\to\infty}\\{\kappa\|{\bf
x}^{k}_{\mathcal{M}^{c}}-{\bf x}^{*}_{\mathcal{M}^{c}}\|_{2}^{2}+o(\|({\bf
x}^{k}_{\mathcal{M}^{c}}-{\bf x}^{*}_{\mathcal{M}^{c}})\|_{2}^{2})\\}/\|{\bf
x}^{k}-{\bf x}^{*}\|_{2}^{2}$ $\displaystyle\leq$ $\displaystyle\kappa,$

which contradicts with (B.5). Hence ${\bf x}^{*}$ must be an isolated local
minimum. ∎

######  Corollary B.6.

Suppose ${\bf x}^{*}$ is feasible for the program

$\displaystyle\min_{{\bf x}}\\{f({\bf x})-g({\bf
x}_{\mathcal{M}^{c}})+\lambda\|{\bf x}_{\mathcal{M}^{c}}\|_{1}\\},\text{ such
that }\|{\bf x}\|_{1}\leq R_{1},\|{\bf x}\|_{2}\leq R_{2},$ (B.17)

where $f\in C^{2}$, $g\in C^{1}$ and $g({\bf
x}_{\mathcal{M}^{c}})-\kappa/2\|{\bf x}_{\mathcal{M}^{c}}\|_{2}^{2}$ is
concave and ${\bf C}$ is an $r\times m$ matrix. Define ${\bf A}=\\{{\bf
0}_{(p-m)\times m},{\bf I}_{(p-m)\times(p-m)}\\}$ and ${\bf A}_{1}=\\{{\bf
I}_{m\times m},{\bf 0}_{m\times(p-m)}\\}$. Assume there are ${\bf
v}^{*}\in\partial\|{\bf x}^{*}_{\mathcal{M}^{c}}\|_{1}$, ${\bf
w}_{1}^{*}\in\partial\|{\bf x}^{*}\|_{1}$, ${\bf w}_{2}^{*}\in\partial\|{\bf
x}^{*}\|_{2}$, $\mu_{1}^{*}\geq 0$, $\mu_{2}^{*}\geq 0$ such that

$\displaystyle\mu_{1}^{*}(R_{1}-\|{\bf x}^{*}\|_{1})=0$
$\displaystyle\mu_{2}^{*}(R_{2}-\|{\bf x}^{*}\|_{2})=0$
$\displaystyle\frac{\partial f({\bf x}^{*})}{\partial{\bf x}^{*}}-{\bf A}^{\rm
T}\frac{\partial g({\bf x}^{*}_{\mathcal{M}^{c}})}{\partial{\bf
x}^{*}_{\mathcal{M}^{c}}}+\lambda{\bf A}^{\rm T}{\bf v}^{*}+\mu_{1}^{*}{\bf
w}_{1}^{*}+\mu_{2}^{*}{\bf w}_{2}^{*}=0$ $\displaystyle{\bf s}^{\rm
T}\frac{\partial^{2}f({\bf x})}{\partial{\bf x}\partial{\bf x}^{\rm T}}{\bf
s}>\kappa,\forall{\bf s}\in G^{*},$

where

$\displaystyle G^{*}$ $\displaystyle:=$ $\displaystyle\left\\{{\bf
s}\in{\mathbb{R}}^{p}:\|{\bf s}\|_{2}=1;\sup_{{\bf w}_{1}\in\partial\|{\bf
x}^{*}\|_{1}}{\bf s}^{\rm T}{\bf w}_{1}\leq 0\text{ if }\|{\bf
x}^{*}\|_{1}=R_{1};\sup_{{\bf w}_{2}\in\partial\|{\bf x}^{*}\|_{2}}{\bf
s}^{\rm T}{\bf w}_{2}\leq 0\text{ if }\|{\bf x}^{*}\|_{2}=R_{2};\right.$
$\displaystyle\sup_{{\bf v}\in\partial\|{\bf
x}^{*}_{\mathcal{M}^{c}}\|_{1}}{\bf s}^{\rm T}\left(\frac{\partial f({\bf
x}^{*})}{\partial{\bf x}^{*}}-{\bf A}^{\rm T}\frac{\partial g({\bf
x}_{\mathcal{M}^{c}}^{*})}{\partial{\bf
x}_{\mathcal{M}^{c}}^{*}}\right)+\lambda{\bf s}^{\rm T}{\bf A}^{\rm T}{\bf
v}={0};$ $\displaystyle\left.\mu_{1}^{*}\sup_{{\bf w}_{1}\in\partial\|{\bf
x}^{*}\|_{1}}{\bf s}^{\rm T}{\bf w}_{1}=0,\mu_{2}^{*}\sup_{{\bf
w}_{2}\in\partial\|{\bf x}^{*}\|_{2}}{\bf s}^{\rm T}{\bf w}_{2}=0\right\\}.$

Then ${\bf x}^{*}$ is an isolated local minimum of the program (B.17).

Proof: The Corollary holds by using the same argument as those lead to Lemma
B.12, while ignoring the equality constraint.∎

######  Lemma B.13.

Assume Conditions (C1)–(C6) hold, $\lambda\leq\alpha_{1}/(8R_{1})$,
$R_{1}\leq[n\alpha_{1}^{2}/\\{64\tau_{1}^{2}\hbox{log}(p)\\}]^{1/4}$,
$\delta\in[4R_{1}\tau_{1}\sqrt{\hbox{log}(p)/n}/\lambda,1]$, $n\geq
4\hbox{log}(p)\tau_{1}^{2}\\{\sqrt{r}\|{\bf C}_{r}^{-1}{\bf
C}_{m-r}\|_{2}+\sqrt{m-r}+(2/\delta+1/2)\sqrt{k}\\}^{4}/(\alpha_{1}-\mu)^{2}$,
and $\alpha_{1}>\mu$. Define ${\bf A}=\\{{\bf 0}_{(p-m)\times m},{\bf
I}_{(p-m)\times(p-m)}\\}$ and ${\bf A}_{1}=\\{{\bf I}_{m\times m},{\bf
0}_{m\times(p-m)}\\}$. Suppose $\widetilde{\bm{\beta}}$ is a stationary point
of program (7) and $\widehat{\bm{\beta}}$ is the interior local minimizer of
(7) and satisfies $\hbox{supp}(\widehat{\bm{\beta}})\subseteq\mathcal{M}\cup
S$. Then $\hbox{supp}(\widetilde{\bm{\beta}})\subseteq\mathcal{M}\cup S$.

Proof: Let $\widetilde{{\bf v}}:=\widetilde{\bm{\beta}}-\widehat{\bm{\beta}}$,
by the Taylor expansion of the first order derivative

$\displaystyle\\{\partial\mathcal{L}(\widetilde{\bm{\beta}})/\partial\bm{\beta}^{\rm
T}-\partial\mathcal{L}(\widehat{\bm{\beta}})/\partial\bm{\beta}^{\rm
T}\\}\widetilde{{\bf v}}=\widetilde{\bf v}^{\rm
T}\partial^{2}\mathcal{L}(\bm{\beta}^{*})/\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}\widetilde{\bf v},$

where $\bm{\beta}^{*}$ is a point on the line connecting
$\widehat{\bm{\beta}}$ and $\widetilde{\bm{\beta}}$ and hence in the feasible
set. Therefore, by Lemma B.6 we have

$\displaystyle\\{\partial\mathcal{L}(\widetilde{\bm{\beta}})/\partial\bm{\beta}^{\rm
T}-\partial\mathcal{L}(\widehat{\bm{\beta}})/\partial\bm{\beta}^{\rm
T}\\}\widetilde{{\bf v}}\geq\alpha_{1}\|\widetilde{\bf
v}\|_{2}^{2}-\tau_{1}\sqrt{\hbox{log}(p)/n}\|\widetilde{\bf v}\|_{1}^{2}.$

We first show that $\|\widetilde{{\bf v}}\|_{2}\leq 1$. Suppose that
$\|\widetilde{{\bf v}}\|_{2}>1$, we have

$\displaystyle\\{\partial\mathcal{L}(\widetilde{\bm{\beta}})/\partial\bm{\beta}^{\rm
T}-\partial\mathcal{L}(\widehat{\bm{\beta}})/\partial\bm{\beta}^{\rm
T}\\}\widetilde{{\bf v}}\geq\alpha_{1}\|\widetilde{\bf
v}\|_{2}-2\tau_{1}R_{1}\sqrt{\hbox{log}(p)/n}\|\widetilde{\bf v}\|_{1}.$
(B.18)

The first order condition implies

$\displaystyle\left\\{\partial\mathcal{L}(\widetilde{\bm{\beta}})/\partial\bm{\beta}+{\bf
A}^{\rm
T}\partial\rho_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}\right\\}^{\rm
T}(\widehat{\bm{\beta}}-\widetilde{\bm{\beta}})\geq 0$ (B.19)

and hence

$\displaystyle\partial\mathcal{L}(\widetilde{\bm{\beta}})/\partial\bm{\beta}^{\rm
T}\widetilde{{\bf v}}\leq-{\bf A}^{\rm
T}\partial\rho_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}\widetilde{{\bf v}}.$

Combine with (B.18), we have

$\displaystyle\\{-{\bf A}^{\rm
T}\partial\rho_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}-\partial\mathcal{L}(\widehat{\bm{\beta}})/\partial\bm{\beta}\\}^{\rm
T}\widetilde{{\bf v}}\geq\alpha_{1}\|\widetilde{\bf
v}\|_{2}-2\tau_{1}R_{1}\sqrt{\hbox{log}(p)/n}\|\widetilde{\bf v}\|_{1}.$
(B.20)

Further because $\widehat{\bm{\beta}}$ is an interior local minimum, we have

$\displaystyle\partial\mathcal{L}(\widehat{\bm{\beta}})/\partial\bm{\beta}+{\bf
A}^{\rm
T}\partial\rho_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}+{\bf
A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{4}=\bf 0.$

for some Lagrange multiplier $\bm{\mu}_{4}>{\bf 0}$. Note that $({\bf
A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{4})^{\rm T}\widetilde{\bf
v}=\bm{\mu}_{t}^{\rm T}{\bf
C}(\widetilde{\bm{\beta}}_{\mathcal{M}}-\widehat{\bm{\beta}}_{mM})={\bf 0}$.
Therefore, plug in (B.20), we have

$\displaystyle\alpha_{1}\|\widetilde{\bf
v}\|_{2}-2\tau_{1}R_{1}\sqrt{\hbox{log}(p)/n}\|\widetilde{\bf v}\|_{1}$
$\displaystyle\leq$ $\displaystyle\\{{\bf A}^{\rm
T}\partial\rho_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}+{\bf
A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{4}-{\bf A}^{\rm
T}\partial\rho_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}\\}^{\rm
T}\widetilde{{\bf v}}$ $\displaystyle=$ $\displaystyle\\{{\bf A}^{\rm
T}\partial\rho_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}-{\bf
A}^{\rm
T}\partial\rho_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}\\}^{\rm
T}\widetilde{{\bf v}}$ $\displaystyle\leq$ $\displaystyle\\{\|{\bf A}^{\rm
T}\partial\rho_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}\|_{\infty}+\|{\bf
A}^{\rm
T}\partial\rho_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}\|_{\infty}\\}\|\widetilde{{\bf
v}}\|_{1}$ $\displaystyle\leq$ $\displaystyle 2\lambda\|\widetilde{{\bf
v}}\|_{1},$

where $\|{\bf A}^{\rm
T}\partial\rho_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}\|_{\infty}\leq\lambda$
holds by Lemma B.4. Hence we have

$\displaystyle\|\widetilde{{\bf
v}}\|_{2}\leq\\{2\lambda+2\tau_{1}R_{1}\sqrt{\hbox{log}(p)/n}\\}\|\widetilde{{\bf
v}}\|_{1}/\alpha_{1}\leq
2R_{1}\\{2\lambda+2\tau_{1}R_{1}\sqrt{\hbox{log}(p)/n}\\}/\alpha_{1}.$

The right hand side is at most 1 because $\lambda\leq\alpha_{1}/(8R_{1})$, and
$n\geq\hbox{log}(p)64\tau_{1}^{2}R_{1}^{4}/\alpha_{1}^{2}$, which contradicts
with $\|\widetilde{\bf v}\|_{2}>1$. Therefore $\|\widetilde{\bf v}\|_{2}\leq
1$.

Now note that

$\displaystyle\mathcal{L}(\bm{\beta})-q_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})=(\mathcal{L}(\bm{\beta})-\mu\|\bm{\beta}_{\mathcal{M}^{c}}\|_{2}^{2}/2)+\\{\mu\|\bm{\beta}_{\mathcal{M}^{c}}\|_{2}^{2}/2-q_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})\\}$

and
$\\{\mu\|\bm{\beta}_{\mathcal{M}^{c}}\|_{2}^{2}/2-q_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})\\}$
is convex by Lemma B.4, and hence for any $\bm{\beta}$ in the feasible set, we
have

$\displaystyle\widetilde{{\bf v}}^{\rm
T}\frac{\partial^{2}\\{\mathcal{L}(\bm{\beta})-q_{\lambda}(\bm{\beta}_{\mathcal{M}^{c}})\\}}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}\widetilde{{\bf v}}$ $\displaystyle\geq$ $\displaystyle\widetilde{{\bf
v}}^{\rm
T}\frac{\partial^{2}\mathcal{L}(\bm{\beta})}{\partial\bm{\beta}\partial\bm{\beta}^{\rm
T}}\widetilde{{\bf v}}-\mu\widetilde{{\bf v}}^{\rm T}{\bf A}^{\rm T}{\bf
A}\widetilde{{\bf v}}$ (B.21) $\displaystyle\geq$
$\displaystyle(\alpha_{1}-\mu)\|\widetilde{{\bf
v}}\|_{2}^{2}-\tau_{1}\sqrt{\frac{\hbox{log}(p)}{n}}\|\widetilde{{\bf
v}}\|_{1}^{2}.$

Further by (B.19), we have

$\displaystyle
0\leq\left\\{\frac{\partial\mathcal{L}(\widetilde{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf
A}\right\\}(\widehat{\bm{\beta}}-\widetilde{\bm{\beta}})+\lambda\widetilde{{\bf
z}}^{\rm T}{\bf A}(\widehat{\bm{\beta}}-\widetilde{\bm{\beta}}),$

where $\widetilde{{\bf
z}}\in\partial\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}$. Further,
because $\widehat{\bm{\beta}}$ is an interior local minimum, for
$\widehat{{\bf z}}\in\partial\|\widehat{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}$,
we have

$\displaystyle{\bf 0}$ $\displaystyle=$
$\displaystyle\partial\mathcal{L}(\widehat{\bm{\beta}})/\partial\bm{\beta}+{\bf
A}^{\rm
T}\partial\rho_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})/\partial\bm{\beta}_{\mathcal{M}^{c}}+{\bf
A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{4}$ $\displaystyle=$
$\displaystyle\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}}-{\bf
A}^{\rm T}\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}}\right\\}+\lambda{\bf
A}^{\rm T}\widehat{{\bf z}}+{\bf A}_{1}^{\rm T}{\bf C}^{\rm T}\bm{\mu}_{4},$

which leads to

$\displaystyle\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf v}}+\lambda(\widehat{{\bf z}}^{\rm T}{\bf
A})\widetilde{{\bf v}}=\bf 0.$

Hence

$\displaystyle 0$ $\displaystyle\leq$
$\displaystyle\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf
v}}-\left\\{\frac{\partial\mathcal{L}(\widetilde{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf v}}$ $\displaystyle+\lambda(\widehat{{\bf
z}}^{\rm T}{\bf
A})(\widetilde{\bm{\beta}}-\widehat{\bm{\beta}})-\lambda(\widetilde{{\bf
z}}^{\rm T}{\bf A})(\widetilde{\bm{\beta}}-\widehat{\bm{\beta}})$
$\displaystyle=$
$\displaystyle\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf
v}}-\left\\{\frac{\partial\mathcal{L}(\widetilde{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf v}}$ $\displaystyle+\lambda(\widehat{{\bf
z}}^{\rm T}{\bf
A})\widetilde{\bm{\beta}}-\lambda\|\widehat{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}-\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}+\lambda(\widetilde{{\bf
z}}^{\rm T}{\bf A})\widehat{\bm{\beta}},$

which implies

$\displaystyle\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}-\lambda(\widehat{{\bf
z}}^{\rm T}{\bf A})\widetilde{\bm{\beta}}$ (B.22) $\displaystyle\leq$
$\displaystyle\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf
v}}-\left\\{\frac{\partial\mathcal{L}(\widetilde{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf
v}}-\lambda\|\widehat{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}+\lambda(\widetilde{{\bf
z}}^{\rm T}{\bf A})\widehat{\bm{\beta}}$ $\displaystyle\leq$
$\displaystyle\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf
v}}-\left\\{\frac{\partial\mathcal{L}(\widetilde{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf v}}$ $\displaystyle\leq$
$\displaystyle\tau_{1}\sqrt{\frac{\hbox{log}(p)}{n}}\|\widetilde{{\bf
v}}\|_{1}^{2}-(\alpha_{1}-\mu)\|\widetilde{{\bf v}}\|_{2}^{2}.$

The second inequality holds because $|(\widetilde{{\bf z}}^{\rm T}{\bf
A})\widehat{\bm{\beta}}|=|\widetilde{{\bf z}}^{\rm
T}\widehat{\bm{\beta}}_{\mathcal{M}^{c}}|\leq\|\widetilde{{\bf
z}}\|_{\infty}\|\widehat{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}\leq\|\widehat{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}$,
and last inequality holds by the Taylor expansion and (B.21).

Now we first assume that the following statement holds: If $\|({\bf A}^{\rm
T}\widehat{{\bf z}})_{\mathcal{M}\cup S}\|_{\infty}\leq 1-\delta$ and
$\lambda\geq 4R_{1}\tau_{1}\sqrt{\hbox{log}(p)/(n)}/\delta$, then

$\displaystyle\|\widetilde{\bf v}\|_{1}\leq\\{\sqrt{r}\|{\bf C}_{r}^{-1}{\bf
C}_{m-r}\|_{2}+\sqrt{m-r}+(2/\delta+1/2)\sqrt{k}\\}\|\widetilde{\bf v}\|_{2}.$

We then will have

$\displaystyle\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}-\lambda(\widehat{{\bf
z}}^{\rm T}{\bf A})\widetilde{\bm{\beta}}$ $\displaystyle\leq$
$\displaystyle\left[\tau_{1}\sqrt{\frac{\hbox{log}(p)}{n}}\\{\sqrt{r}\|{\bf
C}_{r}^{-1}{\bf
C}_{m-r}\|_{2}+\sqrt{m-r}+(2/\delta+1/2)\sqrt{k}\\}^{2}-(\alpha_{1}-\mu)\right]\|\widetilde{\bf
v}\|_{2}^{2}.$

Now because $n\geq 4\hbox{log}(p)\tau_{1}^{2}\\{\sqrt{r}\|{\bf C}_{r}^{-1}{\bf
C}_{m-r}\|_{2}+\sqrt{m-r}+(2/\delta+1/2)\sqrt{k}\\}^{4}/(\alpha_{1}-\mu)^{2}$,
we have

$\displaystyle
0=\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}-\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}\leq\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}-\lambda(\widehat{{\bf
z}}^{\rm T}{\bf
A})\widetilde{\bm{\beta}}\leq-(\alpha_{1}-\mu)/2\|\widetilde{\bf
v}\|_{2}^{2}\leq 0.$

The first inequality holds by the fact that $(\widehat{{\bf z}}^{\rm T}{\bf
A})\widetilde{\bm{\beta}}=\widehat{{\bf z}}^{\rm
T}\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\leq\|\widehat{{\bf
z}}\|_{\infty}\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}\leq\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}$.
Hence we have

$\displaystyle\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}=\lambda(\widehat{{\bf
z}}^{\rm T}{\bf A})\widetilde{\bm{\beta}}.$

Now because $\|(\widehat{{\bf z}}^{\rm T}{\bf A})_{(\mathcal{M}\cup
S)^{c}}\|_{\infty}<1$, we conclude that
$\widetilde{\bm{\beta}}_{(\mathcal{M}\cup S)^{c}}=\bf 0$. Hence,
$\hbox{supp}(\widetilde{\bm{\beta}})\subset\mathcal{M}\cup S$. This would
prove the claim in the statement.

Thus, we only need to show that if $\|({\bf A}^{\rm T}\widehat{{\bf
z}})_{\mathcal{M}\cup S}\|_{\infty}\leq 1-\delta$ and $\lambda\geq
4R_{1}\tau_{1}\sqrt{\hbox{log}(p)/(n)}/\delta$, then

$\displaystyle\|\widetilde{\bf v}\|_{1}\leq\\{(\sqrt{r}\|{\bf C}_{r}^{-1}{\bf
C}_{m-r}\|_{2}+\sqrt{m-r})+(2/\delta+1/2)\sqrt{k}\\}\|\widetilde{\bf
v}\|_{2}.$

First from (B.22), we have

$\displaystyle(\alpha_{1}-\mu)\|\widetilde{{\bf
v}}\|_{2}^{2}-\tau_{1}\sqrt{\frac{\hbox{log}(p)}{n}}\|\widetilde{{\bf
v}}\|_{1}^{2}$ $\displaystyle\leq$
$\displaystyle\left\\{\frac{\partial\mathcal{L}(\widetilde{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widetilde{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf
v}}-\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}-\frac{\partial
q_{\lambda}(\widehat{\bm{\beta}}_{\mathcal{M}^{c}})}{\partial\bm{\beta}_{\mathcal{M}^{c}}^{\rm
T}}{\bf A}\right\\}\widetilde{{\bf v}}$ (B.23) $\displaystyle\leq$
$\displaystyle\lambda(\widehat{{\bf z}}^{\rm T}{\bf
A})\widetilde{\bm{\beta}}-\lambda\|\widehat{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}-\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}+\lambda(\widetilde{{\bf
z}}^{\rm T}{\bf A})\widehat{\bm{\beta}}$ $\displaystyle=$
$\displaystyle\lambda(\widetilde{{\bf z}}^{\rm T}{\bf
A})\widehat{\bm{\beta}}-\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}+\lambda(\widehat{{\bf
z}}^{\rm T}{\bf A})\widetilde{{\bf v}}.$

The second inequality holds by (B.3). Now since
$\hbox{supp}(\widehat{\bm{\beta}})\subseteq\mathcal{M}\cup S$, we have

$\displaystyle\lambda(\widetilde{{\bf z}}^{\rm T}{\bf
A})\widehat{\bm{\beta}}-\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}$
(B.24) $\displaystyle\leq$
$\displaystyle\lambda\|\widehat{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}-\lambda\|\widetilde{\bm{\beta}}_{\mathcal{M}^{c}}\|_{1}$
$\displaystyle=$
$\displaystyle\lambda\|\widehat{\bm{\beta}}_{S}\|_{1}-\lambda\|\widetilde{\bm{\beta}}_{S}\|_{1}-\lambda\|\widetilde{\bm{\beta}}_{(\mathcal{M}\cup
S)^{c}}\|_{1}$ $\displaystyle=$
$\displaystyle\lambda\|\widehat{\bm{\beta}}_{S}\|_{1}-\lambda\|\widetilde{\bm{\beta}}_{S}\|_{1}-(\lambda\|\widetilde{\bm{\beta}}_{(\mathcal{M}\cup
S)^{c}}\|_{1}-\lambda\|\widehat{\bm{\beta}}_{(\mathcal{M}\cup S)^{c}}\|_{1})$
$\displaystyle=$
$\displaystyle\lambda\|\widetilde{\bm{\beta}}_{S}+\widehat{\bm{\beta}}_{S}-\widetilde{\bm{\beta}}_{S}\|_{1}-\lambda\|\widetilde{\bm{\beta}}_{S}\|_{1}-(\lambda\|\widetilde{\bm{\beta}}_{(\mathcal{M}\cup
S)^{c}}\|_{1}-\lambda\|\widetilde{\bm{\beta}}_{(\mathcal{M}\cup
S)^{c}}+\widehat{\bm{\beta}}_{(\mathcal{M}\cup
S)^{c}}-\widetilde{\bm{\beta}}_{(\mathcal{M}\cup S)^{c}}\|_{1})$
$\displaystyle\leq$ $\displaystyle\lambda\|\widetilde{{\bf
v}}_{S}\|_{1}-\lambda\|\widetilde{{\bf v}}_{(\mathcal{M}\cup S)^{c}}\|_{1}.$

In addition,

$\displaystyle\lambda(\widehat{{\bf z}}^{\rm T}{\bf A})\widetilde{{\bf v}}$
$\displaystyle=$ $\displaystyle\lambda\\{({\bf A}^{\rm T}\widehat{{\bf
z}})_{S}\\}^{\rm T}\widetilde{{\bf v}}_{S}+\lambda\\{({\bf A}^{\rm
T}\widehat{{\bf z}})_{(\mathcal{M}\cup S)^{c}}\\}^{\rm T}\widetilde{{\bf
v}}_{(\mathcal{M}\cup S)^{c}}$ (B.25) $\displaystyle\leq$
$\displaystyle\lambda\|({\bf A}^{\rm T}\widehat{{\bf
z}})_{S}\|_{\infty}\|\widetilde{{\bf v}}_{S}\|_{1}+\lambda\|({\bf A}^{\rm
T}\widehat{{\bf z}})_{(\mathcal{M}\cup S)^{c}}\|_{\infty}\|\widetilde{{\bf
v}}_{(\mathcal{M}\cup S)^{c}}\|_{1}$ $\displaystyle\leq$
$\displaystyle\lambda\|\widetilde{{\bf
v}}_{S}\|_{1}+\lambda(1-\delta)\|\widetilde{{\bf v}}_{(\mathcal{M}\cup
S)^{c}}\|_{1}.$

Combine (B.23), (B.24), and (B.25), we have

$\displaystyle-\tau_{1}\sqrt{\frac{\hbox{log}(p)}{n}}\|\widetilde{{\bf
v}}\|_{1}^{2}$ $\displaystyle\leq$
$\displaystyle(\alpha_{1}-\mu)\|\widetilde{{\bf
v}}\|_{2}^{2}-\tau_{1}\sqrt{\frac{\hbox{log}(p)}{n}}\|\widetilde{{\bf
v}}\|_{1}^{2}$ $\displaystyle\leq$ $\displaystyle\lambda\|\widetilde{{\bf
v}}_{S}\|_{1}-\lambda\|\widetilde{{\bf v}}_{(\mathcal{M}\cup
S)^{c}}\|_{1}+\lambda\|\widetilde{{\bf
v}}_{S}\|_{1}+\lambda(1-\delta)\|\widetilde{{\bf v}}_{(\mathcal{M}\cup
S)^{c}}\|_{1}$ $\displaystyle=$ $\displaystyle\lambda 2\|\widetilde{{\bf
v}}_{S}\|_{1}-\lambda\delta\|\widetilde{{\bf v}}_{(\mathcal{M}\cup
S)^{c}}\|_{1}.$

Now because $\delta\lambda/2\geq
2\tau_{1}R_{1}\sqrt{\hbox{log}(p)/n}\geq\tau_{1}\sqrt{\hbox{log}(p)/n}\|\widetilde{{\bf
v}}\|_{1}$, the above display implies

$\displaystyle-2^{-1}\delta\lambda\|\widetilde{{\bf v}}_{S}\|_{1}\leq\lambda
2\|\widetilde{{\bf v}}_{S}\|_{1}-\lambda\delta\|\widetilde{{\bf
v}}_{(\mathcal{M}\cup S)^{c}}\|_{1}$

which leads to

$\displaystyle\delta\|\widetilde{{\bf v}}_{(\mathcal{M}\cup
S)^{c}}\|_{1}\leq(2+\delta/2)\|\widetilde{{\bf v}}_{S}\|_{1}.$

Then

$\displaystyle\|\widetilde{\bf v}\|_{1}$ $\displaystyle=$
$\displaystyle\|\widetilde{\bf v}_{\mathcal{M}}\|_{1}+\|\widetilde{\bf
v}_{S}\|_{1}+\|\widetilde{\bf v}_{(\mathcal{M}\cup S)^{c}}\|_{1}$
$\displaystyle\leq$ $\displaystyle\|\widetilde{\bf
v}_{\mathcal{M}}\|_{1}+\|\widetilde{\bf
v}_{S}\|_{1}+(4/\delta+1)\|\widetilde{\bf v}_{S}\|_{1}$ $\displaystyle\leq$
$\displaystyle\|\widetilde{\bf
v}_{\mathcal{M}}\|_{1}+(2/\delta+1/2)\sqrt{k}\|\widetilde{\bf v}\|_{2}$
$\displaystyle\leq$ $\displaystyle\\{\sqrt{r}\|{\bf C}_{r}^{-1}{\bf
C}_{m-r}\|_{2}+\sqrt{m-r}+(2/\delta+1/2)\sqrt{k}\\}\|\widetilde{\bf v}\|_{2}.$

The last equality holds by using the same argument as those lead to (A.30).
This completes the proof. ∎

######  Corollary B.7.

Assume Conditions (C1)–(C6) hold, $\lambda\leq\alpha_{1}/(8R_{1})$,
$R_{1}\leq[n\alpha_{1}^{2}/\\{64\tau_{1}^{2}\hbox{log}(p)\\}]^{1/4}$,
$\delta\in[4R_{1}\tau_{1}\sqrt{\hbox{log}(p)/n}/\lambda,1]$, $n\geq
4\hbox{log}(p)\tau_{1}^{2}\\{\sqrt{m}+(2/\delta+1/2)\sqrt{k}\\}^{4}/(\alpha_{1}-\mu)^{2}$,
and $\alpha_{1}>\mu$. Suppose $\widetilde{\bm{\beta}}_{a}$ is a stationary
point of program (8) and $\widehat{\bm{\beta}}_{a}$ is the interior local
minimize of (8) and satisfies
$\hbox{supp}(\widehat{\bm{\beta}}_{a})\subseteq\mathcal{M}\cup S$. Then
$\hbox{supp}(\widetilde{\bm{\beta}}_{a})\subseteq\mathcal{M}\cup S$.

Proof: The Corollary holds by using the same arguments as those lead to Lemma
B.13, while using the consistency result in Theorem 2. ∎

##  B.4 Supporting Results related to Test Statistics

###  B.4.1 Some Definitions

We define

$\displaystyle{\bf\Sigma}(\bm{\beta})$ $\displaystyle\equiv$ $\displaystyle
E[\\{Y_{i}{\bf W}_{i}-\exp(\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta}/2)({\bf W}_{i}-\bm{\Omega}\bm{\beta})\\}^{\otimes 2}]$

By using the relation (2)–(4) and the additional relations

$\displaystyle E\\{\exp(2\bm{\beta}_{t}^{\rm T}{\bf W}_{i}-\bm{\beta}_{t}^{\rm
T}\bm{\Omega}\bm{\beta}_{t})\mid{\bf X}_{i}\\}$ $\displaystyle=$
$\displaystyle\exp(2\bm{\beta}_{t}^{\rm T}{\bf X}_{i}),$ $\displaystyle
E\\{\exp(2\bm{\beta}_{t}^{\rm T}{\bf W}_{i}-\bm{\beta}_{t}^{\rm
T}\bm{\Omega}\bm{\beta}_{t})({\bf W}_{i}-\bm{\Omega}\bm{\beta}_{t})\mid{\bf
X}_{i}\\}$ $\displaystyle=$ $\displaystyle\exp(2\bm{\beta}_{t}^{\rm T}{\bf
X}_{i}){\bf X}_{i},$ $\displaystyle E[\exp(2\bm{\beta}_{t}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{t}^{\rm T}\bm{\Omega}\bm{\beta}_{t})\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{t})^{\otimes 2}-\bm{\Omega}/2\\}\mid{\bf
X}_{i}]$ $\displaystyle=$ $\displaystyle\exp(2\bm{\beta}_{t}^{\rm T}{\bf
X}_{i}){\bf X}_{i}^{\otimes 2},$

we have

$\displaystyle{\bf\Sigma}(\bm{\beta}_{t})$ $\displaystyle=$ $\displaystyle
E[\\{\exp(\bm{\beta}_{t}^{\rm T}{\bf X}_{i})-\exp(2\bm{\beta}_{t}^{\rm T}{\bf
X}_{i})\\}({\bf X}_{i}{\bf X}_{i}^{\rm T}+\bm{\Omega})$
$\displaystyle-\exp(2\bm{\beta}_{t}^{\rm T}{\bf
X}_{i})\bm{\Omega}\bm{\beta}_{t}{\bf X}_{i}^{\rm T}-\exp(2\bm{\beta}_{t}^{\rm
T}{\bf X}_{i}){\bf X}_{i}\bm{\beta}_{t}^{\rm
T}\bm{\Omega}+\exp(2\bm{\beta}_{t}^{\rm T}{\bf W}_{i}-\bm{\beta}_{t}^{\rm
T}\bm{\Omega}\bm{\beta}_{t})({\bf W}_{i}-\bm{\Omega}\bm{\beta}_{t})^{\otimes
2}]$

and the sample version

$\displaystyle\widehat{\bf\Sigma}(\bm{\beta})$ $\displaystyle=$ $\displaystyle
n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta}/2)\\{({\bf W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes
2}-\bm{\Omega}\\}$ $\displaystyle-\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}-\bm{\Omega}/2\\}$
$\displaystyle+\exp(\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta}/2)\bm{\Omega}-\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})\bm{\Omega}$
$\displaystyle-\exp(2\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta})\bm{\Omega}\bm{\beta}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\rm T}-\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})({\bf
W}_{i}-\bm{\Omega}\bm{\beta})\bm{\beta}^{\rm T}\bm{\Omega}$
$\displaystyle+\exp(2\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta})({\bf W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}$
$\displaystyle=$ $\displaystyle n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}^{\rm
T}{\bf W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta}/2)\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}\\}$
$\displaystyle-\exp(2\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta})\\{({\bf W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes
2}+\bm{\Omega}/2\\}$ $\displaystyle-\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})\bm{\Omega}\bm{\beta}({\bf
W}_{i}-\bm{\Omega}\bm{\beta})^{\rm T}-\exp(2\bm{\beta}^{\rm T}{\bf
W}_{i}-\bm{\beta}^{\rm T}\bm{\Omega}\bm{\beta})({\bf
W}_{i}-\bm{\Omega}\bm{\beta})\bm{\beta}^{\rm T}\bm{\Omega}$
$\displaystyle+\exp(2\bm{\beta}^{\rm T}{\bf W}_{i}-\bm{\beta}^{\rm
T}\bm{\Omega}\bm{\beta})({\bf W}_{i}-\bm{\Omega}\bm{\beta})^{\otimes 2}.$

Further, let

$\displaystyle{\bm{\Psi}}({\bf\Sigma},{\bf Q},\bm{\beta})=({\bf C}[{\bf
I}_{m\times m},{\bf 0}_{m\times k}]{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}^{-1}({\bm{\beta}}){\bf\Sigma}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}({\bm{\beta}}){{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}^{-1}({\bm{\beta}})[{\bf I}_{m\times m},{\bf 0}_{m\times k}]^{\rm T}{\bf
C}^{\rm T}),$

and

$\displaystyle T_{0}=(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n})^{\rm
T}\bm{\Psi}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})(\bm{\omega}_{n}+\sqrt{n}{\bf h}_{n}),$

where

$\displaystyle\bm{\omega}_{n}=-\sqrt{n}{\bf C}[{\bf I}_{m\times m},{\bf
0}_{m\times k}]{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}^{-1}(\bm{\beta}_{t})\left\\{\frac{\partial\mathcal{L}(\bm{\beta}_{t})}{\partial\bm{\beta}_{t}}\right\\}_{\mathcal{M}\cup
S}$

Further define Wald statistics as

$\displaystyle T_{W}=n({\bf C}\widehat{\bm{\beta}}_{a\mathcal{M}}-{\bf
t})^{\rm T}{\bm{\Psi}}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})^{-1}({\bf
C}\widehat{\bm{\beta}}_{a\mathcal{M}}-{\bf t});$

the score statistics

$\displaystyle T_{S}$ $\displaystyle=$ $\displaystyle
n\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}^{\rm
T}}\right\\}_{\mathcal{M}\cup S}({\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times
k}]{{\bf Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}^{-1}(\widehat{\bm{\beta}}))^{\rm T}$
$\displaystyle\times\bm{\Psi}^{-1}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}^{-1}(\widehat{\bm{\beta}})\left\\{\frac{\partial\mathcal{L}(\widehat{\bm{\beta}})}{\partial\bm{\beta}}\right\\}_{\mathcal{M}\cup
S}.$

Because without knowing the distribution of ${\bf X}$, the full likelihood is
unknown and hence we do not discuss the likelihood ratio test here.

###  B.4.2 Assumptions

  1. (D1)

Assume

$\displaystyle\max_{1\leq i\leq n}(\|\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t}){\bf C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf
Q}}_{\mathcal{M}\cup S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t}){\bf
W}_{i\mathcal{M}\cup S}\|_{2}^{3}$
$\displaystyle+\|\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf Q},\bm{\beta}_{t}){\bf
C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup
S}^{-1}(\bm{\beta}_{t})(\bm{\Omega}\bm{\beta}_{t})_{\mathcal{M}\cup
S}\|_{2}^{3})^{-1}$ $\displaystyle\times
E[\|Y_{i}\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf Q},\bm{\beta}_{t}){\bf C}[{\bf
I}_{m\times m},{\bf 0}_{m\times k}]{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t}){\bf W}_{i}-\exp(\bm{\beta}_{t}^{\rm
T}{\bf W}_{i}-\bm{\beta}_{t}^{\rm T}\bm{\Omega}\bm{\beta}_{t}/2)$
$\displaystyle\times\bm{\Psi}^{-1/2}({\bf\Sigma},{\bf Q},\bm{\beta}_{t}){\bf
C}[{\bf I}_{m\times m},{\bf 0}_{m\times k}]{{\bf Q}}_{\mathcal{M}\cup
S,\mathcal{M}\cup S}^{-1}(\bm{\beta}_{t})({\bf W}_{i}-\bm{\beta}_{t}^{\rm
T}\bm{\Omega})_{\mathcal{M}\cup S}\|_{2}^{3}|{\bf W}_{i}]=O(1)$

  2. (D2)

$c_{\bm{\Psi}}\leq\alpha_{\min}\\{{\bm{\Psi}}({\bf\Sigma},{\bf
Q},\bm{\beta})\\}\leq\alpha_{\max}\\{{\bm{\Psi}}({\bf\Sigma},{\bf
Q},\bm{\beta})\\}\leq C_{\bm{\Psi}}.$

###  B.4.3 Some Lemmas

######  Lemma B.14.

Suppose ${\bf X}_{1},\ldots,{\bf X}_{n}$ are independent $m$ dimensional
random vector which satisfies, $E(X_{i})=0$ and $\sum_{i=1}^{n}\hbox{cov}({\bf
X}_{i})={\bf I}_{m}$. Let ${\bf Z}$ be a m-dimensional standard multivariate
normal vector, then

$\displaystyle\sup_{\mathcal{C}}|\Pr(\sum_{i=1}^{n}{\bf
X}_{i}\in{\mathcal{C}})-\Pr({\bf
Z}\in\mathcal{C})|=O(m^{1/2}\sum_{i=1}^{n}E\|{\bf X}_{i}\|_{2}^{3}).$

Proof: The lemma follows Theorem 1.1 in Bentkus (2005).

######  Lemma B.15.

Assume Conditions (C1) – (C7) hold, then

$\displaystyle\|{\bm{\Psi}}^{-1}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})-{\bm{\Psi}}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\|_{2}=\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k),$

and

$\displaystyle\|{\bm{\Psi}}^{-1}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}})-{\bm{\Psi}}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\|_{2}=\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k).$

Proof: First

$\displaystyle\|{\bm{\Psi}}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})-{\bm{\Psi}}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\|_{2}$ (B.26) $\displaystyle=$ $\displaystyle
O_{p}\\{\max(\|\widehat{\bf Q}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\widehat{\bm{\beta}}_{a})^{-1}-{\bf Q}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}({\bm{\beta}_{t}})^{-1}\|_{2},$
$\displaystyle\|\widehat{\bf\Sigma}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}(\widehat{\bm{\beta}}_{a})-{\bf\Sigma}_{\mathcal{M}\cup S,\mathcal{M}\cup
S}({\bm{\beta}_{t}})\|_{2}\\}.$

Using the same arguments as those lead to (A.53), we have $\|\widehat{\bf
Q}(\widehat{\bm{\beta}}_{a})^{-1}-{\bf
Q}({\bm{\beta}_{t}})^{-1}\|_{2}=\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)$.
Now recall that ${\mathbb{K}}\equiv\\{{\bf v}\in{\mathbb{R}}^{\mathcal{M}\cup
S}:\|{\bf v}\|_{2}\leq 1\\}$, for ${\bf v}\in{\mathbb{K}}$, by the Taylor
expansion

$\displaystyle{\bf v}^{\rm
T}\\{\widehat{\bf\Sigma}(\widehat{\bm{\beta}}_{a})-\widehat{{\bf\Sigma}}({\bm{\beta}_{t}})\\}{\bf
v}$ $\displaystyle=$ $\displaystyle
n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}_{a}^{\rm T}{\bf W}_{i}-\bm{\beta}_{a}^{\rm
T}\bm{\Omega}\bm{\beta}_{a}/2){\bf v}^{\rm T}\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\otimes 2}\\}{\bf v}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\rm
T}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle+n^{-1}\sum_{i=1}^{n}2\exp(\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm
T}\bm{\Omega}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle-n^{-1}\sum_{i=1}^{n}2\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm T}\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\otimes 2}+\bm{\Omega}/2\\}{\bf v}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\rm
T}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle-n^{-1}\sum_{i=1}^{n}2\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm
T}\bm{\Omega}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle-n^{-1}\sum_{i=1}^{n}2\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm
T}\bm{\Omega}\bm{\beta}_{a}({\bf W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\rm T}{\bf
v}({\bf W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\rm
T}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle-n^{-1}\sum_{i=1}^{n}2\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a})\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm T}\bm{\Omega}-{\bf v}^{\rm
T}\bm{\Omega}\bm{\beta}_{a}{\bf v}^{\rm
T}\bm{\Omega}\\}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle-n^{-1}\sum_{i=1}^{n}2\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a})\bm{\beta}_{a}^{\rm T}\bm{\Omega}{\bf v}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\rm
T}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle-n^{-1}\sum_{i=1}^{n}\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a})\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm T}\bm{\Omega}-{\bf v}^{\rm
T}\bm{\Omega}\bm{\beta}_{a}{\bf v}^{\rm
T}\bm{\Omega}\\}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle+n^{-1}\sum_{i=1}^{n}\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm T}\\{({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\otimes 2}\\}{\bf v}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\rm
T}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle+n^{-1}\sum_{i=1}^{n}2\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm
T}\bm{\Omega}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$ $\displaystyle=$
$\displaystyle n^{-1}\sum_{i=1}^{n}\exp(\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}/2){\bf v}^{\rm
T}\\{({\bf W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\otimes 2}\\}{\bf v}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\rm
T}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle+n^{-1}\sum_{i=1}^{n}2\exp(\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}/2){\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm
T}\bm{\Omega}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle-n^{-1}\sum_{i=1}^{n}2\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm
T}\bm{\Omega}{\bf v}/2({\bf W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\rm
T}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle-n^{-1}\sum_{i=1}^{n}4\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm
T}\bm{\Omega}\bm{\beta}_{a}({\bf W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\rm T}{\bf
v}({\bf W}_{i}-\bm{\Omega}\bm{\beta}_{a})^{\rm
T}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t})$
$\displaystyle-n^{-1}\sum_{i=1}^{n}2\exp(2\bm{\beta}_{a}^{\rm T}{\bf
W}_{i}-\bm{\beta}_{a}^{\rm T}\bm{\Omega}\bm{\beta}_{a})\\{{\bf v}^{\rm T}({\bf
W}_{i}-\bm{\Omega}\bm{\beta}_{a}){\bf v}^{\rm T}\bm{\Omega}-{\bf v}^{\rm
T}\bm{\Omega}\bm{\beta}_{a}{\bf v}^{\rm
T}\bm{\Omega}\\}(\widehat{\bm{\beta}}_{a}-\bm{\beta}_{t}),$

where $\bm{\beta}_{a}$ is a point in between $\widehat{\bm{\beta}}_{a}$ and
$\bm{\beta}_{t}$. Using the same arguments as those lead to (A.45) and (A.46),
we have

$\displaystyle\|\widehat{\bf\Sigma}(\widehat{\bm{\beta}}_{a})-\widehat{{\bf\Sigma}}({\bm{\beta}_{t}})\|_{2}=O_{p}\left[\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)\right],$

and

$\displaystyle\|\widehat{\bf\Sigma}(\bm{\beta})-{{\bf\Sigma}}({\bm{\beta}_{t}})\|_{2}=o_{p}\left[\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)\right].$

Therefore,

$\displaystyle\|\widehat{\bf\Sigma}(\widehat{\bm{\beta}}_{a})-\widehat{{\bf\Sigma}}({\bm{\beta}_{t}})\|_{2}=O_{p}\left[\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)\right].$

Combine with (B.26) and the fact that $\|\widehat{\bf
Q}(\widehat{\bm{\beta}}_{a})^{-1}-{\bf
Q}({\bm{\beta}_{t}})^{-1}\|_{2}=\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k)$,
by Lemma B.11 we have

$\displaystyle\|{\bm{\Psi}}^{-1}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})-{\bm{\Psi}}^{-1}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\|_{2}$ $\displaystyle=$ $\displaystyle
O_{p}\\{\|{\bm{\Psi}}(\widehat{\bf\Sigma},\widehat{\bf
Q},\widehat{\bm{\beta}}_{a})-{\bm{\Psi}}({\bf\Sigma},{\bf
Q},\bm{\beta}_{t})\|_{2}\\}$ $\displaystyle=$
$\displaystyle\left\\{\frac{\hbox{log}(p)}{n}\right\\}^{1/4}(m+k).$

The second relation in the statement holds by using the same arguments as
those lead the above results. This proves the result.∎

##  B.5 The composite ROIs

Based on Braak & Braak (1991), Landau et al. (2016), Schöll et al. (2016), we
define the composite regions as follow, where letter $L$ and $R$ represent the
left and right hemispheres, respectively.

  * •

Braak 1 and 2 composite region (Braak12): L_entorhinal, R_entorhinal

  * •

Braak 3 and 4 composite region (Braak34):
L_parahippocampal, L_fusiform, L_lingual, L_amygdala, R_parahippocampal,
R_fusiform, R_lingual, R_amygdala, L_middletemporal, L_caudantcing,
L_rostantcing, L_postcing, L_isthmuscing, L_insula, L_inferiortemporal,
L_temppole, R_middletemporal, R_caudantcing, R_rostantcing, R_postcing,
R_isthmuscing, R_insula, R_inferiortemporal, R_temppole.

  * •

Braak 5 and 6 composite region (Braak56):
L_superior_frontal, L_lateral_orbitofrontal, L_medial_orbitofrontal,
L_frontal_pole, L_caudal_middle_frontal, L_rostral_middle_frontal,
L_pars_opercularis, L_pars_orbitalis, L_pars_triangularis, L_lateraloccipital,
L_parietalsupramarginal, L_parietalinferior, L_superiortemporal,
L_parietalsuperior, L_precuneus, L_bankSuperiorTemporalSulcus, L_tranvtemp,
R_superior_frontal, R_lateral_orbitofrontal, R_medial_orbitofrontal,
R_frontal_pole, R_caudal_middle_frontal, R_rostral_middle_frontal,
R_pars_opercularis, R_pars_orbitalis, R_pars_triangularis, R_lateraloccipital,
R_parietalsupramarginal, R_parietalinferior,
R_superiortemporal, R_parietalsuperior, R_precuneus,
R_bankSuperiorTemporalSulcus, R_tranvtemp, L_pericalcarine, L_postcentral,
L_cuneus, L_precentral, L_paracentral, R_pericalcarine, R_postcentral,
R_cuneus, R_precentral, R_paracentral

##  B.6 Simulations when measurement error covariance is unknown

To evaluate the proposed adjustment estimator in Section 6, we added the
following three simulation settings where the covariance matrices of the
measurement errors contain different numbers of unknown parameters.

  1. 1.

$\bm{\Omega}$ is a matrix with $p/4$ unknown parameters, corresponding to
$p/4$ nonzero diagonal entries.

  2. 2.

$\bm{\Omega}$ is a matrix with $p/2$ unknown parameters, corresponding to
$p/2$ nonzero entries.

  3. 3.

$\bm{\Omega}$ is a matrix with $6p-15$ unknown parameters. Specifically,
$\bm{\Omega}=(\sigma_{ij})_{p\times p}$, where $\sigma_{ij}=0.05(1-|i-j|/5)$
for $|i-j|\leq 5$ and $\sigma_{ij}=0$ for $|i-j|>5$.

In these settings, the number of unknown parameters in $\bm{\Omega}$
increases, while all other settings are the same as in Section 5.2. We
evaluate the empirical sizes and powers of the Wald test for $n=300$, $p=50$
in Table B.9 and $p=350$ in Table B.10. In the first two settings, where
$\bm{\Omega}$ contains relatively few parameters hence their convergence is
sufficiently fast, the empirical sizes are well controlled around the nominal
level 0.05 in all hypothesis tests. In the last setting, the parameter number
is too large to achieve sufficiently fast convergence, hence the Wald test
cannot control the Type I errors. These results imply that the proposed
adjustment can control Type I error rate when the error covariance matrix
contains small or moderate number of unknown parameters.

Table B.9:  The empirical size and power of Wald tests for linear hypothesis testing with $n=300$. NUP standards for the number of unknown parameter in $\bm{\Omega}$.  |  |  NUP=$p/4$ |  NUP=$p/2$ |  NUP $=6p-15$
---|---|---|---|---
$p=50$ | $\beta_{2}$ | $H_{0,1}:\beta_{2}=-0.75,\ v.s.\ H_{a,1}:\beta_{2}\neq-0.75$
-0.75 | 0.054 | 0.044 | 0.262
-0.65 | 0.334 | 0.384 | 0.348
-0.55 | 0.832 | 0.852 | 0.542
-0.35 | 1.000 | 1.000 | 0.910
$\beta_{3}$ | $H_{0,2}:\beta_{3}=0,\ v.s.\ H_{a,2}:\beta_{3}\neq 0$
0.0 | 0.064 | 0.044 | 0.558
0.1 | 0.262 | 0.244 | 0.330
0.2 | 0.714 | 0.768 | 0.320
0.4 | 1.000 | 0.994 | 0.644
$\beta_{p}$ | $H_{0,3}:\beta_{p}=0,\ v.s.\ H_{a,3}:\beta_{p}\neq 0$
0.0 | 0.074 | 0.066 | 0.070
0.1 | 0.302 | 0.316 | 0.228
0.2 | 0.778 | 0.804 | 0.676
0.4 | 1.000 | 1.000 | 0.992
$\beta_{1}+\beta_{2}$ | $H_{0,4}:\beta_{1}+\beta_{2}=0,\ v.s.\ H_{a,4}:\beta_{1}+\beta_{2}\neq 0$
0.0 | 0.048 | 0.052 | 0.056
0.1 | 0.164 | 0.152 | 0.220
0.2 | 0.488 | 0.484 | 0.650
0.4 | 0.966 | 0.972 | 0.998
$\beta_{3}+\beta_{4}$ | $H_{0,5}:\beta_{3}+\beta_{4}=0,\ v.s.\ H_{a,5}:\beta_{3}+\beta_{4}\neq 0.$
0.0 | 0.056 | 0.056 | 0.542
0.1 | 0.182 | 0.156 | 0.326
0.2 | 0.456 | 0.518 | 0.278
0.4 | 0.972 | 0.952 | 0.588
$\beta_{1}+\beta_{p}$ | $H_{0,6}:\beta_{1}+\beta_{p}=0.75,\ v.s.\ H_{a,6}:\beta_{1}+\beta_{p}\neq 0.75$
0.75 | 0.050 | 0.048 | 0.328
0.85 | 0.154 | 0.168 | 0.338
0.95 | 0.472 | 0.486 | 0.456
1.15 | 0.974 | 0.974 | 0.668
$\beta_{2}+\beta_{3}$ | $H_{0,7}:\beta_{2}+\beta_{3}=-0.75,\ v.s.\ H_{a,7}:\beta_{2}+\beta_{3}\neq-0.75$
-0.75 | 0.048 | 0.040 | 0.246
-0.65 | 0.170 | 0.166 | 0.292
-0.55 | 0.480 | 0.490 | 0.414
-0.35 | 0.976 | 0.968 | 0.856
Table B.10:  The empirical size and power of Wald tests for linear hypothesis testing with $n=300$. NUP standards for the number of unknown parameter in $\bm{\Omega}$. |  |  NUP=$p/4$ |  NUP=$p/2$ |  NUP $=6p-15$
---|---|---|---|---
$p=350$ | $\beta_{2}$ | $H_{0,1}:\beta_{2}=-0.75,\ v.s.\ H_{a,1}:\beta_{2}\neq-0.75$
-0.75 | 0.060 | 0.052 | 0.672
-0.65 | 0.326 | 0.314 | 0.792
-0.55 | 0.846 | 0.818 | 0.910
-0.35 | 0.898 | 0.886 | 0.738
$\beta_{3}$ | $H_{0,2}:\beta_{3}=0,\ v.s.\ H_{a,2}:\beta_{3}\neq 0$
0.0 | 0.046 | 0.062 | 0.466
0.1 | 0.290 | 0.294 | 0.150
0.2 | 0.774 | 0.732 | 0.150
0.4 | 0.996 | 0.998 | 0.762
$\beta_{p}$ | $H_{0,3}:\beta_{p}=0,\ v.s.\ H_{a,3}:\beta_{p}\neq 0$
0.0 | 0.054 | 0.068 | 0.044
0.1 | 0.298 | 0.286 | 0.206
0.2 | 0.790 | 0.778 | 0.662
0.4 | 0.986 | 0.986 | 1.000
$\beta_{1}+\beta_{2}$ | $H_{0,4}:\beta_{1}+\beta_{2}=0,\ v.s.\ H_{a,4}:\beta_{1}+\beta_{2}\neq 0$
0.0 | 0.044 | 0.052 | 0.060
0.1 | 0.190 | 0.184 | 0.226
0.2 | 0.506 | 0.498 | 0.648
0.4 | 0.872 | 0.860 | 0.998
$\beta_{3}+\beta_{4}$ | $H_{0,5}:\beta_{3}+\beta_{4}=0,\ v.s.\ H_{a,5}:\beta_{3}+\beta_{4}\neq 0.$
0.0 | 0.064 | 0.066 | 0.508
0.1 | 0.168 | 0.180 | 0.164
0.2 | 0.472 | 0.478 | 0.116
0.4 | 0.950 | 0.944 | 0.684
$\beta_{1}+\beta_{p}$ | $H_{0,6}:\beta_{1}+\beta_{p}=0.75,\ v.s.\ H_{a,6}:\beta_{1}+\beta_{p}\neq 0.75$
0.75 | 0.056 | 0.054 | 0.680
0.85 | 0.182 | 0.152 | 0.738
0.95 | 0.476 | 0.464 | 0.778
1.15 | 0.966 | 0.960 | 0.512
$\beta_{2}+\beta_{3}$ | $H_{0,7}:\beta_{2}+\beta_{3}=-0.75,\ v.s.\ H_{a,7}:\beta_{2}+\beta_{3}\neq-0.75$
-0.75 | 0.068 | 0.056 | 0.678
-0.65 | 0.172 | 0.182 | 0.680
-0.55 | 0.492 | 0.446 | 0.696
-0.35 | 0.950 | 0.942 | 0.900

## References

  * [1]
  * Baker et al. [2017] Baker, S. L., Lockhart, S. N., Price, J. C., He, M., Huesman, R. H., Schonhaut, D., Faria, J., Rabinovici, G. & Jagust, W. J. [2017], ‘Reference tissue–based kinetic evaluation of 18f-av-1451 for tau imaging’, Journal of Nuclear Medicine 58(2), 332–338.
  * Belloni, Chernozhukov & Kaul [2017] Belloni, A., Chernozhukov, V. & Kaul, A. [2017], ‘Confidence bands for coefficients in high dimensional linear models with error-in-variables’, arXiv preprint arXiv:1703.00469 .
  * Belloni & Rosenbaum [2016] Belloni, A. & Rosenbaum, M. [2016], ‘An $\\{l_{1}$, $l_{2}$, $l_{\infty}\\}$-regularization approach to high-dimensional errors-in-variables models’, Electronic Journal of Statistics 10, 1935–7524.
  * Belloni, Rosenbaum & Tsybakov [2017] Belloni, A., Rosenbaum, M. & Tsybakov, A. B. [2017], ‘Linear and conic programming estimators in high dimensional errors-in-variables models’, Journal of the Royal Statistical Society: Series B (Statistical Methodology) 79(3), 939–956.
  * Benjamini & Hochberg [1995] Benjamini, Y. & Hochberg, Y. [1995], ‘Controlling the false discovery rate: a practical and powerful approach to multiple testing’, Journal of the Royal statistical society: series B (Methodological) 57(1), 289–300.
  * Bentkus [2005] Bentkus, V. [2005], ‘A lyapunov-type bound in $r^{d}$’, Theory of Probability & Its Applications 49(2), 311–323.
  * Bickel & Levina [2008] Bickel, P. J. & Levina, E. [2008], ‘Covariance regularization by thresholding’, The Annals of Statistics 36(6), 2577–2604.
  * Braak & Braak [1991] Braak, H. & Braak, E. [1991], ‘Neuropathological stageing of alzheimer-related changes’, Acta neuropathologica 82(4), 239–259.
  * Brown et al. [2019] Brown, B., Weaver, T. & Wolfson, J. [2019], ‘Meboost: Variable selection in the presence of measurement error’, Statistics in medicine 38(15), 2705–2718.
  * Cai & Liu [2011] Cai, T. & Liu, W. [2011], ‘Adaptive thresholding for sparse covariance matrix estimation’, Journal of the American Statistical Association 106(494), 672–684.
  * Carroll et al. [2006] Carroll, R. J., Ruppert, D., Stefanski, L. A. & Crainiceanu, C. M. [2006], Measurement error in nonlinear models: a modern perspective, Chapman and Hall/CRC, New Tork.
  * Chen & Gu [2014] Chen, L. & Gu, Y. [2014], ‘The convergence guarantees of a non-convex approach for sparse recovery’, IEEE Transactions on Signal Processing 62(15), 3754–3767.
  * Cook & Stefanski [1995] Cook, J. & Stefanski, L. A. [1995], ‘A simulation extrapolation method for parametric measurement error models’, Journal of the American Statistical Association 89, 1314–1328.
  * Datta & Zou [2017] Datta, A. & Zou, H. [2017], ‘Cocolasso for high-dimensional error-in-variables regression’, The Annals of Statistics 45(6), 2400–2426.
  * Duchi et al. [2008] Duchi, J., Shalev-Shwartz, S., Singer, Y. & Chandra, T. [2008], Efficient projections onto the l 1-ball for learning in high dimensions, in ‘Proceedings of the 25th international conference on Machine learning’, ACM, pp. 272–279.
  * Fallah et al. [2011] Fallah, N., Mitnitski, A. & Rockwood, K. [2011], ‘Applying neural network poisson regression to predict cognitive score changes’, Journal of Applied Statistics 38(9), 2051–2062.
  * Fan & Li [2001] Fan, J. & Li, R. [2001], ‘Variable selection via nonconcave penalized likelihood and its oracle properties’, Journal of the American statistical Association 96(456), 1348–1360.
  * Fan et al. [2011] Fan, J., Liao, Y. & Mincheva, M. [2011], ‘High dimensional covariance matrix estimation in approximate factor models’, Annals of statistics 39(6), 3320.
  * Fletcher & Watson [1980] Fletcher, R. & Watson, G. A. [1980], ‘First and second order conditions for a class of nondifferentiable optimization problems’, Mathematical Programming 18(1), 291–307.
  * Jiang & Ma [2021] Jiang, F. & Ma, Y. [2021], ‘Poisson regression with error corrupted high dimensional features’, Statistica Sinica .  In press.
  * Jiang et al. [2021] Jiang, F., Zhou, Y. & Ma, Y. [2021], ‘Code and data for high dimensional poisson models with noisy data: hypothesis testing for nonlinear nonconvex optimization’.  https://github.com/feigroup/high-dimenisional-inference-for-count-data-with-errors.
  * Katz et al. [2021] Katz, M. J., Wang, C., Nester, C. O., Derby, C. A., Zimmerman, M. E., Lipton, R. B., Sliwinski, M. J. & Rabin, L. A. [2021], ‘T-moca: A valid phone screen for cognitive impairment in diverse community samples’, Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring 13(1), e12144.
  * Landau et al. [2016] Landau, S., Ward, T. J., Murphy, A. & Jagust, W. [2016], ‘Flortaucipir (av-1451) processing methods’.
https://adni.bitbucket.io/reference/docs/UCBERKELEYAV1451/UCBERKELEY_AV1451_Methods_Aug2018.pdf

  * Leisman et al. [2016] Leisman, G., Moustafa, A. A. & Shafir, T. [2016], ‘Thinking, walking, talking: integratory motor and cognitive brain function’, Frontiers in public health 4, 94.
  * Li et al. [2021] Li, M., Li, R. & Ma, Y. [2021], ‘Inference in high dimensional linear measurement error models’, Journal of Multivariate Analysis in press.
  * Loh & Wainwright [2012] Loh, P.-L. & Wainwright, M. J. [2012], ‘High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity’, The Annals of Statistics pp. 1637–1664.
  * Loh & Wainwright [2015] Loh, P.-L. & Wainwright, M. J. [2015], ‘Regularized m-estimators with nonconvexity: Statistical and algorithmic theory for local optima’, Journal of Machine Learning Research 16, 559–616.
  * Loh & Wainwright [2017] Loh, P.-L. & Wainwright, M. J. [2017], ‘Support recovery without incoherence: A case for nonconvex regularization’, The Annals of Statistics 45(6), 2455–2482.
  * McCullagh & Nelder [2019] McCullagh, P. & Nelder, J. A. [2019], Generalized linear models, Routledge.
  * Mitnitski et al. [2014] Mitnitski, A. B., Fallah, N., Dean, C. B. & Rockwood, K. [2014], ‘A multi-state model for the analysis of changes in cognitive scores over a fixed time interval’, Statistical methods in medical research 23(3), 244–256.
  * Ning & Liu [2017] Ning, Y. & Liu, H. [2017], ‘A general theory of hypothesis tests and confidence regions for sparse high dimensional models’, The Annals of Statistics 45, 158–195.
  * Schöll et al. [2016] Schöll, M., Lockhart, S. N., Schonhaut, D. R., O’Neil, J. P., Janabi, M., Ossenkoppele, R., Baker, S. L., Vogel, J. W., Faria, J., Schwimmer, H. D. et al. [2016], ‘Pet imaging of tau deposition in the aging human brain’, Neuron 89(5), 971–982.
  * Sentürk & Müller [2005] Sentürk, D. & Müller, H.-G. [2005], ‘Covariate-adjusted regression’, Biometrika 92(1), 75–89.
  * Shi et al. [2019] Shi, C., Song, R., Chen, Z., Li, R. et al. [2019], ‘Linear hypothesis testing for high dimensional generalized linear models’, The Annals of statistics 47(5), 2671–2703.
  * Smith & Kosslyn [2008] Smith, E. & Kosslyn, S. [2008], Cognitive Psychology: Mind and Brain, Pearson Education, Pearson Prentice Hall.
https://books.google.com/books?id=YIPSNwAACAAJ

  * Sørensen et al. [2015] Sørensen, Ø., Frigessi, A. & Thoresen, M. [2015], ‘Measurement error in lasso: Impact and likelihood bias correction’, Statistica sinica pp. 809–829.
  * Sørensen et al. [2018] Sørensen, Ø., Hellton, K. H., Frigessi, A. & Thoresen, M. [2018], ‘Covariate selection in high-dimensional generalized linear models with measurement error’, Journal of Computational and Graphical Statistics 27(4), 739–749.
  * Stefanski [1989] Stefanski, L. A. [1989], ‘Unbiased estimation of a nonlinear function a normal mean with application to measurement err oorf models’, Communications in Statistics-Theory and Methods 18(12), 4335–4358.
  * Van de Geer et al. [2014] Van de Geer, S., Bühlmann, P., Ritov, Y. & Dezeure, R. [2014], ‘On asymptotically optimal confidence regions and tests for high-dimensional models’, Annals of Statistics 42(3), 1166–1202.
  * Vial [1982] Vial, J.-P. [1982], ‘Strong convexity of sets and functions’, Journal of Mathematical Economics 9(1-2), 187–205.
  * Wainwright [2009] Wainwright, M. J. [2009], ‘Sharp thresholds for high-dimensional and noisy sparsity recovery using $l_{1}$ \- constrained quadratic programming (lasso)’, IEEE transactions on information theory 55(5), 2183–2202.
  * Weiner et al. [2017] Weiner, M. W., Veitch, D. P., Aisen, P. S., Beckett, L. A., Cairns, N. J., Green, R. C., Harvey, D., Jack Jr, C. R., Jagust, W., Morris, J. C. et al. [2017], ‘The alzheimer’s disease neuroimaging initiative 3: Continued innovation for clinical trial improvement’, Alzheimer’s & Dementia 13(5), 561–571.
  * Zhang et al. [2012] Zhang, C.-H., Zhang, T. et al. [2012], ‘A general theory of concave regularization for high-dimensional sparse estimation problems’, Statistical Science 27(4), 576–593.
  * Zhang & Cheng [2017] Zhang, X. & Cheng, G. [2017], ‘Simultaneous inference for high-dimensional linear models’, Journal of the American Statistical Association 112(518), 757–768.


# Potential Energy based Mixture Model for Noisy Label Learning

Zijia Wang
Dell Technologies
Shanghai, China
<EMAIL_ADDRESS>
&Wenbin Yang
Dell Technologies
Shanghai, China
<EMAIL_ADDRESS>
&Zhisong Liu
Dell Technologies
Shanghai, China
<EMAIL_ADDRESS>
&Zhen Jia
Dell Technologies
Shanghai, China
<EMAIL_ADDRESS>

###### Abstract

Training deep neural networks (DNNs) from noisy labels is an important and
challenging task. However, most existing approaches focus on the corrupted
labels and ignore the importance of inherent data structure. To bridge the gap
between noisy labels and data, inspired by the concept of potential energy in
physics, we propose a novel Potential Energy based Mixture Model (PEMM) for
noise-labels learning. We innovate a distance-based classifier with the
potential energy regularization on its class centers. Embedding our proposed
classifier with existing deep learning backbones, we can have robust networks
with better feature representations. They can preserve intrinsic structures
from the data, resulting in a superior noisy tolerance. We conducted extensive
experiments to analyze the efficiency of our proposed model on several real-
world datasets. Quantitative results show that it can achieve state-of-the-art
performance.

##  1 Introduction

With the recent emergence of large-scale datasets, deep neural networks (DNNs)
have exhibited impressive performance in numerous machine learning tasksWang
et al., (2019). However, labeling large-scale datasets is a costly and error-
prone process, and even high-quality datasets are likely to contain noisy
labels Wang et al., 2018b . Recent studies Arazo et al., (2019); Wang et al.,
(2019) show that noisy labels severely degrade the generalization performance
of deep neural networks. Therefore, learning from noisy labels , or we also
refer it as robust learning, is becoming an important task.

Currently, several works focus on robust learning mechanisms to reduce the
effects caused by noise. There are many approaches that resolve the problem
from different aspects. For example, Zhang and Sabuncu, (2018); Wang et al.,
(2019); Ghosh et al., (2017) propose noise-robust losses and robust
regularizers to avoid exceptions caused by outliers. Jiang et al., (2018);
Malach and Shalev-Shwartz, (2017); Wang et al., 2018b  realize co-teaching
deep neural networks for knowledge transfer. Tong et al., (2015); Vahdat,
(2017); Li et al., (2017) utilize label correction to improve the data
robustness. However, these approaches only focus on the technologies that
separate the clean data from the noisy ones, and ignore the inherent
information of data samples, which may cause over-confidence on noisy
labelsWang et al., (2019); Li et al., (2020).

In this paper, we investigate the problem in a “natural way". Our method is
motivated by the following observation: clean datasets are often low-rankLi et
al., (2019), which means they are in a “stable state" with the lowest entropy
Yu et al., (2020), while the existence of noisy label will break the inherent
low-rank state. Coincidentally, things in the world always tend to exist in
such a steady state because of the potential energy (PE) among molecules.
Similar phenomenon also exists in the real world where the potential energy
(PE) keeps the molecules in a steady state.

The mechanism of PE is illustrated in Fig. 1: for two nuclei, there exists a
distance that makes the system most stable, while in this state, the potential
energy becomes minimal. Inspired by this theory, we propose a Potential Energy
based Mixture Model (PEMM) for robust learning. Concretely, we propose a
distance based classifiers constrained by a novel Potential Energy based
Mixture Model (PEMM). It could preserve the inherent structures of data
samples, so the training process becomes less dependent on class labels. In
the Experiment section, we will demonstrate the strengths of our approach on
several datasets. We will also provide ablation studies for detailed analysis.

Figure 1: An illustration of potential energy. The energy of the molecule is a
function of the distance between the two nuclei. The most stable
state(potential energy is -436KJ/mol) is achieved at the distance 74 pm.

In summary, the advantages and contributions of the algorithm proposed in this
paper are:

  * •

An explainable noisy labels learning method. Compared with other methods with
hand-crafted combined loss, our PEMM framework is based on essence of the
machine learning, which makes the algorithm more explainable.

  * •

Out of box. Compared to existing approaches that often involve architectural
or non-trivial algorithmic modifications, our method is easy to use. The PEMM
can be easily incorporated into existing methods to significantly improve
their performance.

  * •

Satisfied performance. Empirically, it is demonstrated that the PEMM can
achieve better robustness than state-of-the-art methods.

##  2 Related Work

###  2.1 Noisy-robust training

Several methods that aim to mitigate label noise have been proposed. Here, we
summarize some of the recent approaches to noise-robust learning. For simply,
this method can be roughly divided into three categories: 1) label correction
, 2) loss adjustment , and 3) training strategies modification.

Label correction One common label correction approach is to correct data
labels via a clean label inference step using complex noise models
characterized by some probabilistic models, like graphical models Tong et al.,
(2015), conditional random fields Vahdat, (2017) or knowledge graphs Li et
al., (2017). However, the ground-truth noise models are neither always
available in practice nor easy to obtain accurate estimation.

Loss adjustment A family of studies modify the regularization to improve the
robustness. For example, PHuberMenon et al., (2019) proposes a composite loss-
based gradient clipping for label noise robustness. Moreover, Label Smoothing
Regularization (LSR)Szegedy et al., (2016) adopts soft labels to avoid
overfitting to noisy labels. Recently, the mixupZhang et al., (2017)
regularizes the DNN to favor simple linear behaviors in raining examples.

Meanwhile, researchers have attempted to design robust loss functions. Work in
Ghosh et al., (2017) shows that Mean Absolute Error (MAE) is theoretically
robust against label noise. However, MAE may lead to significantly longer
convergence time and performance dropZhang and Sabuncu, (2018). To overcome
the problem, Generalized Cross Entropy (GCE)Zhang and Sabuncu, (2018) loss
applies a Box-Cox transformation to probabilities can behave like a weighted
MAE. Furthermore, Wang proposed Symmetric Cross Entropy loss (SCE)Wang et al.,
(2019), which combines CE loss with Reverse Cross Entropy loss.

Training strategies modification There are some methods design new learning
paradigms for noisy labels. Work in Jiang et al., (2018) proposed a framework
that supervises the training of a student-net by a learned sample weighting
scheme in favor of probably correct labels. Furthermore, the iterative
learning framework Wang et al., 2018b  iteratively detects and isolates noisy
samples during the learning process. However, these methods either rely on
complex interventions into the learning process, which may be challenging to
adapt and tune.

###  2.2 Larger margin learning

The concept of large margin learning aims to keep the value of empirical risk
fixed and minimize the confidence intervalVapnik and V., (1999). The
classification margin is closely related to adversarial robustness and
generalization ability. However, larger margin learning methods, like large
margin cosine loss(LMCL)Wang et al., 2018a  and additive angular margin
loss(AAML)Deng et al., (2019), are not developed for noisy labels problems
directly and their decision boundaries are affected by noisy labels.

##  3 Problem Statement

Given a $K$-class dataset $\mathcal{D}=\left\\{\bm{x};y\right\\}^{N}_{n=1}$,
with $\bm{x}\in\bm{X}\subseteq R^{d}$ and
$y\in\mathcal{Y}=\left\\{1;...;K\right\\}$ representing a $d$-dimensional
input and its associated label respectively. For each sample $\bm{x}$, a
classifier $f(\bm{x})$ output its probability of each label
$k\in\left\\{1;...;K\right\\}:p(k|\bm{x})$. Denoting the ground-truth
distribution over labels for sample $\bm{x}$ by $q(k|\bm{x})$, and
$\sum_{k=1}^{K}q(k|\bm{x})=1$. Then the cross entropy (CE) loss for the data
sample $\bm{x}_{n}$ can be expressed as:

$l_{ce}(\bm{x}_{n})=-\sum_{k=1}^{K}q(k|\bm{x}_{n})\log(p(k|\bm{x}_{n}))$ (1)

Although easy to converge, the CE loss may easily be overfit to noise labels
Wang et al., (2019). In CE, samples with less congruence on provided labels
are implicitly weighed more in the gradient update. This implicit weighting
scheme is desirable for training with clean data, but can cause overfitting to
noisy labels.

##  4 Solution

The main idea behind the proposed method is that the inherent data structure
can be modeled by fitting a mixture model. Then, with representations that
preserve intrinsic structures from the data, training relies less on class
labels and becomes more robust.

###  4.1 Inherent data information

In general, the machine learning (ML) tasks are trying to estimate the real
data distribution $p(\bm{x})$ with a parametric model. Formally, we rewrite
the estimation process with latent feature $\bm{z}$ and data label $y$ as

$\displaystyle
KL\big{(}p(\bm{x},\bm{z},y)\big{\|}q(\bm{x},\bm{z},y)\big{)}=\sum_{y}p(\bm{z},y|\bm{x})\tilde{p}(\bm{x})\ln\frac{p(\bm{z},y|\bm{x})\tilde{p}(\bm{x})}{q(\bm{z},y|\bm{x})\tilde{q}(\bm{x})}d\bm{z}d\bm{x}$
(2)

where $\tilde{p}(\bm{x})$ and $\tilde{q}(\bm{x})$ are the data distribution
estimated with finite data samples and the given data distribution. From our
problem, we can set $\tilde{p}(\bm{x})=\tilde{q}(\bm{x})$.

Then, in a deep learning framework, it is natural to assume that

$\displaystyle p(\bm{z},y|\bm{x})$
$\displaystyle=p(y|\bm{z})p(\bm{z}|\bm{x}),\quad q(\bm{z},y|\bm{x})$
$\displaystyle=q(\bm{z}|y)q(y|\bm{x})$ (3)

Then, we can rewrite the eq.2 as a form of expectation:

$\displaystyle\mathbb{E}_{\bm{x}\sim\tilde{p}(\bm{x})}\Big{[}\sum_{y}p(y|\bm{z})\log$
$\displaystyle\frac{p(y|\bm{z})p(\bm{z}|\bm{x})}{q(\bm{z}|y)q(y|\bm{x})}\Big{]},\quad\bm{z}\sim
p(\bm{z}|\bm{x})$ (4)

where $\bm{z}\sim p(\bm{z}|\bm{x})$ is the reparameterize operation. One can
further rewrite the eq. 4 as

$\displaystyle\mathbb{E}_{\bm{x}\sim\tilde{p}(\bm{x})}$
$\displaystyle\Big{[}-\sum_{y}p(y|\bm{z})\log
q(\bm{z}|y)-\sum_{y}p(y|\bm{z})\log q(y|\bm{x})$ (5)
$\displaystyle+\sum_{y}p(y|\bm{z})\log p(z|\bm{x})+\sum_{y}p(y|\bm{z})\log
p(y|\bm{z})\Big{]},\bm{z}\sim p(\bm{z}|\bm{x})$

where $q(y|\bm{z})\log p(\bm{z}|y)$ is essentially the CE loss,
$p(y|\bm{z})\log q(y|\bm{x})$ is the loss when given a reference distribution
$p(y|\bm{x})$, $p(y|\bm{z})\log p(z|\bm{x})$ will be minimized when the
features are well separated, $p(y|\bm{z})\log p(y|\bm{z})$ is a constant.

From eq.5, one can find that the widely used CE loss only uses part of the
information. In the context of noisy labels, $q(y|x)$ does not represent the
true class distribution well.

To fully utilize the inherent structures information of the data, i.e.
approximate the expectation in eq.5, we propose the learning framework shown
in Fig. 2. The classifier(section 4.2) with Potential Energy (PE) regularizer
(section 4.3) on its class centers is utilized for noisy robust learning.

Figure 2: The framework for noisy robust learning.

###  4.2 Distance based classifier

In some recent works about few-shot learning Fort, (2017), it is reported that
the distance based classifier is more explainable. Assuming the
$\bm{C}=[\bm{c}_{1},\dots,\bm{c}_{K}]^{T}$ is the matrix containing all the
class centers, the distance between sample $\bm{x}_{n}$ and the $k$-th center
vector $\bm{c}_{k}$ can be expressed as:

$d_{n,k}=exp(-\frac{\left\|\bm{z}_{n}-\bm{c}_{k}\right\|_{2}^{2}}{\sigma^{2}})$
(6)

where $\bm{z}_{n}=f(\bm{x}_{n})$ is the latent space feature before
classifier, $\|\|_{2}$ stands for the $l_{2}$ norm of a vector, $\bm{c}_{k}$
is the center of $k$-th class, and $\sigma$ is the hyper-parameter of the
radius of the Euclidean ball. In the sequence, the posterior probability of
$\bm{x}_{n}$ can then be calculated as

$p(y_{n,k}|\bm{x}_{n})=\frac{d_{n,k}}{\sum_{k=1}^{K}d_{n,k}}$ (7)

Then, we noticed that, the second term in eq.5 reflect the penalty of
distribution $q(y|\bm{x})$ given the prediction distribution $p(y|\bm{z})$.
Similar to Wang et al., (2019), one can use Reverse Cross Entropy (RCE) to
measure the penalty effectively. The RCE loss for a sample $\bm{x}$ is:

$\displaystyle
l_{rce}(\bm{x}_{n})=-\sum_{k=1}^{K}p(k|\bm{x}_{n})\log(q(k|\bm{x}_{n}))$ (8)

In the sequence, we adopt the RCE lossWang et al., (2019) and give the final
classifier loss as

$\displaystyle l_{clf}(\bm{x}_{n})=$
$\displaystyle-\alpha\sum_{k=1}^{K}q(k|\bm{x}_{n})\log(p(k|\bm{x}_{n}))-\sum_{k=1}^{K}p(k|\bm{x}_{n})\log(q(k|\bm{x}_{n}))$
(9)

where $\alpha$ is the parameter for flexible exploration on the robustness.
Intuitively, the $l_{clf}$ will cover the first term and second term in eq. 5.

###  4.3 Potential energy based centers regularization

To cover the third term in eq.5, we introduce the potential energy based
center regularization. Potential energy is a concept in physics. As shown in
Fig. 1, there exists a distance($r_{0}$) that makes the system most stable
McCall, (2010). We adapt this concept and establish the stable state for
robust feature learning Li et al., (2017); Yu et al., (2020).

In this paper, we use the potential energy expression:

$E(r)=\frac{1}{r^{u}}-\frac{\xi}{r^{v}}$ (10)

where $r>0$ is the distance between any two nuclei, $v\in\mathbb{N}^{+}$ and
$u\in\mathbb{N}^{+}$ represents the power of push and attractive force,
respectively. $\xi$ is the weight of attractive force. For computing
effectiveness, we choose $u=3$ ,$v=2$ and $\xi=2$ for a approximation to the
real-word potential energy curve C, (2009). The $r_{0}$ under these settings
is calculated as 0.75.

Then we adapt this equation into the loss function of class centers matrix
$\bm{C}=[\bm{c}_{1};...;\bm{c}_{K}]$ as :

$\displaystyle{L}_{pem}(\bm{C})=$
$\displaystyle\sum_{i=0}^{K-1}\sum_{j=i+1}^{K}\big{[}{l}_{pem}(\bm{c}_{i},\bm{c}_{j})\big{]}$
(11)

where

$\displaystyle{l}_{pem}(\bm{c}_{i},\bm{c}_{j})=\frac{1}{(\|\bm{c}_{i}-\bm{c}_{j}\|_{2}+\beta)^{3}}-\frac{2}{(\|\bm{c}_{i}-\bm{c}_{j}\|_{2}+\beta)^{2}}$
(12)

where $\bm{c}_{0}=\bm{0}$; $K>2$ is the number of classes to be compared,
$0<\beta<r_{0}$ are the hyper-parameters to control the loss value.

Fig. 3 illustrates the effectiveness of PE constrains. The PE constrains try
to make the system stable by minimizing the potential energy. After some
iterations, the system will achieve “aesthetically pleasing” layout, which can
be captured by two criteria: "all the edge lengths ought to be the same, and
the layout should display as much symmetry as possible." Kobourov, (2012).

Figure 3: The establishment process of co-stable state: starting from random
positions, then each node will look for a stable position automatically.

###  4.4 Training

The loss function covers all terms in eq. 5, then it can be expressed as:

$L=\sum_{n=1}^{B}(t)\bigg{[}L_{clf}(\bm{x}_{n})\bigg{]}+\lambda
L_{pem}(\bm{C})$ (13)

where $B$ is a batch-size of samples, $\lambda$ is trade-off parameter to
balance the contributions of the $l_{pem}$. Formally, our method can is
express as Algorithm 1. It should be noticed that, one can also reweight the
data samples according to the estimated distribution of our PEMM as the state-
of-the-artsLi et al., (2020); Yao et al., (2020).

Input : Noisy dataset $\\{{\bm{x},y}\\}\in\mathcal{D}$;epoch number
${T}$;learning rate $\delta$

Output : Robust model

Initialize: Network parameters $\bm{\Theta}$

for  _$t\leftarrow 1$ to $T$_ do

for  _each mini batch B_ do

obtain features $\bm{z}$

compute distance ${\bm{d}}$ (eq.6)

compute the classify loss ${l_{clf}}$ (eq.9)

compute potential energy loss ${l_{pem}}$(eq.11)

compute loss ${L}$ of the batch (eq.13)

update $\bm{\Theta}$ using back propagation

return

Algorithm 1 Robust training of PEMM

##  5 Comparison with large margin learning

Intuitively, PE based class center regularization is similar to a larger
margin learning methodsGuo and Zhang, (2021). The larger margin learning will
enforce large margins among centers across different categories. However,
different form the well-known large-margin loss Guo and Zhang, (2021), which
utilize the pre-defined boundary, our method is more flexible by directly
minimize the potential energy. Moreover, a class center based discriminative
loss based on Euclidean distance is also proposed in Chen et al., (2019):

$l_{sq}(\bm{c}_{i},\bm{c}_{j})=max(0,r_{0}-\|\bm{c}_{i}-\bm{c}_{j}\|_{2}).$
(14)

As shown in the Fig.4, the loss proposed in Chen et al., (2019) and eq.12 are
both related to the distance $r_{0}$. However, for the loss in eq.14, when the
class distance bigger than $r_{0}$, the penalty will no longer change, while
the PE loss is asymmetric and only reaches its minimization (most stable
state) at $r_{0}$.

Figure 4: Illustration of the difference between the loss in Chen et al.,
(2019)(red) and our PE loss(black). The PE loss value is asymmetric.

The asymmetric property of PE is important to establish a co-stable system. In
the high-dimensional space, the PE loss will help the class center to form
such a polyhedron whose distance between any two vertices keep the same.
Obviously, such a polyhedron system is co-table. Without loss of generality,
consider a simple binary classification system shown in the Fig.5. Due to the
PE loss, the distance between A, B, and O (coordinate origin) are
$d=r_{0}-\beta$. When change the position of class A, one has to change the
position of class B at the same time. Obviously, these changes will be
equivalent to rotating A and B at the same time. So, such a co-stable state
system makes full use of complementary information between classes Song et
al., (2021). In addition, such a co-stable system is also an output of
implicit early-stopping mechanism. According to eq.11, the limitation of the
sum of class centers distances as

$\sum_{i=0}^{K-1}\sum_{j=i+1}^{K}\|\mathbf{c}_{i}-\mathbf{c}_{j}\|_{2}\Rightarrow\frac{K(K+1)}{2}(r_{0}-\beta).$
(15)

The memorization nature Li et al., (2019) indicates that the DNNs can avoid
overfitting to noisy label by early stopping. For our model, the PE naturally
result in an equivalent early stooping due to the limitation in eq.15, which
will make the training process robust to the noisy labels.

Figure 5: Change process of a co-stable process.(left to right)

##  6 Experiment

In this section, we describe the experiments performed to evaluate our method.
We verify our method by comparing with other recent baseline methods, varying
experimental settings in terms of dataset and type and ratio of noise in the
training data.

###  6.1 Empirical Understanding

We conduct experiments on CIFAR-10 dataset with symmetric noise towards a
deeper understanding of PEMM. Symmetric noisy labels are generated by flipping
the labels of a given proportion of training samples to one of the other class
labels uniformly.

Experimental setup: An 8-layer DNN with 6 convolutional layers followed by 2
fully connected layers are tested in our experiment. All networks are trained
using SGD with momentum 0.9, weight decay 10-4. The initial learning rate is
established as 0.01, which is divided by 10 after 40 and 80 epochs (120 epochs
in total). In our network setting , all the features $\bm{z}$ will be
normalized before eq.6, and we will set $\sigma$ to 1 in all experiments. The
parameter $\alpha$, $\beta$ and $\lambda$ in PEMM are set to 0.1,0.3,1
respectively.

Representations: We investigate the representations learned by PEMM compared
to that learned by CE. The t-SNEVan der Maaten and Hinton, (2008) are utilized
to project the representation at the second last dense layer to a 2D. The
projected representations for 40% noisy labels are shown in Fig. 6 .One can
easily found that, the representations learned by PEMM have significantly
better quality than that of CE with more separated class centers and clearly
bounded clusters.

Figure 6: Representations learned by CE and PEMM with 40% symmetric noisy
labels for training ,(top) and validation (bottom) in CIFAR-10 dataset with
(a)(c) cross-entropy loss from Eq. 1 (test accuracy 78.7%), (b)(d) the
proposed PEMM (test accuracy 87.9%).

Parameter analysis: We tune the parameters of PEMM: $\alpha$, $\beta$ and
$\lambda$. As $\lambda$ can be reflected by $\beta$, thus we only show results
of $\alpha$ and $\beta$. We tested $\alpha\in[0.01;1]$ and $\beta\in[0.1;1]$
on CIFAR-10 under 40% noisy labels. Fig.7 (a) shows that large $\alpha$ (e.g.,
1.0) tends to cause more overfitting, while small $\alpha$ (e.g., 0.01) will
some the robustness to noisy. For this reason, a relatively large $\alpha$ may
help to convergence on more complex datasets. As for parameter $\beta$, one
can see that PEMM is not very sensitive to $\beta$ (Fig.7 (b)) when
$0.2<\beta<r_{0}$. However, if $\beta<0.2$ or $\beta>r_{0}$, the performance
of our PEMM will decrease due of the properties of PE function.

Figure 7: The effect of different values of $\alpha$(a) and $\beta$(b).

Ablation study: For a comprehensive understanding of each component in PEMM,
we further conduct a series of ablation experiments on CIFAR-10 under 40%
noisy labels. Figure8 presents the following experiments: a) replace SCE with
CE loss ; b) remove CE loss; c) remove the PEM term; d) remove the
reconstruction term.

For a better understanding, we visualize the feature distribution with t-SNE
again. In Fig 8(a), one can observe that even without SCE loss function, our
method still has better noisy robustness than the CE-loss based method. Then,
compare Fig.8(a) and Fig.8(b), the reverse-CE term in SCE loss has more impact
on the performance improvement. Moreover, it is witnessed in 8(c) that the PEM
loss term can enlarge the decision boundary.

(a) Remove RCE loss loss (accuracy 83.72%)

(b) Remove CE loss (accuracy 86.50%)

(c) Remove the PEM term (accuracy 83.80%)

Figure 8: Ablation experiments with 40% symmetric noisy labels for in CIFAR-10
train dataset.All the t-SNE visualization in performed in test dataset

###  6.2 Robustness to Noisy Labels

We compare PEMM with 4 recently proposed noisy label learning methods as well
as the standard CE loss: (1) Forward Patrini et al., (2017): Training with
label correction by multiplying the network prediction with the ground truth
noise matrix;(2) Bootstrap Reed et al., (2014): Training with new labels
generated by a convex combination of the raw labels and the predicted labels;
(3) GCE Zhang and Sabuncu, (2018): Training with a noise robust loss
encompassing both MAE and CE; (4) SCE Wang et al., (2019): Training with the
novel symmetric cross entropy loss;and (5) CE: Training with standard cross
entropy loss.

Table 1: Test accuracy (%) of different models on benchmark datasets with
various rates of symmetric and asymmetric noisy labels. The average accuracy
and standard deviation of 5 random runs are reported and the best results are
in bold.

Datasets | Methods | Symmetric Noise | Asymmetric Noise
---|---|---|---
Noise Rate | Noise Rate
0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 0.2 | 0.3 | 0.4
CIFAR-10 | CE | 89.26 $\pm$ 0.03 | 82.96 $\pm$ 0.05 | 78.70 $\pm$ 0.07 | 66.62 $\pm$ 0.15 | 34.80 $\pm$ 0.25 | 85.98 $\pm$ 0.03 | 83.53 $\pm$ 0.08 | 78.51 $\pm$ 0.05
Bootstrap | 88.77 $\pm$ 0.06 | 83.95 $\pm$ 0.10 | 79.97 $\pm$ 0.07 | 71.65 $\pm$ 0.05 | 41.44 $\pm$ 0.49 | 86.57 $\pm$ 0.08 | 84.86 $\pm$ 0.05 | 79.76 $\pm$ 0.07
Forward | 89.39 $\pm$ 0.04 | 85.83 $\pm$ 0.05 | 81.37 $\pm$ 0.03 | 73.59 $\pm$ 0.08 | 47.10 $\pm$ 0.14 | 87.68 $\pm$ 0.01 | $\bm{86.86\pm 0.06}$ | ${85.73\pm 0.04}$
GCE | 86.76 $\pm$ 0.03 | 84.86 $\pm$ 0.06 | 82.42 $\pm$ 0.10 | 75.20 $\pm$ 0.09 | 40.81 $\pm$ 0.24 | 84.61 $\pm$ 0.09 | 82.11 $\pm$ 0.13 | 75.32 $\pm$ 0.10
SCE | $\bm{91.05\pm 0.04}$ | $\bm{90.88\pm 0.06}$ | $\bm{86.34\pm 0.07}$ | $\bm{80.07\pm 0.02}$ | $\bm{53.81\pm 0.27}$ | $\bm{88.24\pm 0.05}$ | 85.36 $\pm$ 0.14 | 80.64 $\pm$ 0.10
PEMM | $\bm{89.28\pm 0.04}$ | $\bm{89.01\pm 0.06}$ | $\bm{88.54\pm 0.07}$ | $\bm{85.25\pm 0.03}$ | $\bm{59.34\pm 0.28}$ | $\bm{89.05\pm 0.03}$ | 87.11 $\pm$ 0.15 | $\bm{85.78\pm 0.15}$
CIFAR-100 | CE | 64.34 $\pm$ 0.37 | 59.26 $\pm$ 0.39 | 50.82 $\pm$ 0.19 | 25.39 $\pm$ 0.09 | 5.27 $\pm$ 0.06 | 62.97 $\pm$ 0.19 | 63.12 $\pm$ 0.16 | 61.85 $\pm$ 0.35
Bootstrap | 63.26 $\pm$ 0.39 | 57.91 $\pm$ 0.42 | 48.17 $\pm$ 0.18 | 12.27 $\pm$ 0.11 | $1.00\pm 0.01$ | $63.44\pm 0.35$ | $63.18\pm 0.35$ | $62.08\pm 0.22$
Forward | 63.99 $\pm$ 0.52 | 59.75 $\pm$ 0.34 | 53.13 $\pm$ 0.28 | 24.70 $\pm$ 0.26 | $2.65\pm 0.03$ | $64.09\pm 0.61$ | $64.00\pm 0.32$ | $60.91\pm 0.36$
GCE | $64.43\pm 0.20$ | $59.06\pm 0.27$ | $53.25\pm 0.65$ | $36.16\pm 0.74$ | $8.43\pm 0.80$ | $63.03\pm 0.22$ | $63.17\pm 0.26$ | $61.69\pm 1.15$
SCE | $\bm{66.75\pm 0.04}$ | $\bm{60.01\pm 0.19}$ | $\bm{53.69\pm 0.07}$ | $\bm{41.47\pm 0.04}$ | $\bm{15.00\pm 0.04}$ | $\bm{65.58\pm 0.06}$ | $\bm{65.14\pm 0.05}$ | $\bm{63.10\pm 0.13}$
PEMM | $\bm{67.00\pm 0.05}$ | $\bm{62.10\pm 0.18}$ | $\bm{65.75\pm 0.12}$ | $\bm{56.25\pm 0.05}$ | $\bm{30.03\pm 0.05}$ | $\bm{65.78\pm 0.08}$ | $\bm{65.80\pm 0.03}$ | $\bm{64.81\pm 0.18}$

Experimental setup: We test two types of label noise: symmetric (uniform)
noise and asymmetric (class-dependent) noise on CIFAR-10 Krizhevsky et al.,
(2009) and CIFAR-100 Krizhevsky et al., (2009) dataset. For asymmetric noisy
labels, we flip labels within a specific set of classes.We flip the class
labels as the setting in Wang et al., (2019).

We use the same network as Section 5.1 for CIFAR-10 , a ResNet-44 He et al.,
(2016) for CIFAR-100 and ResNet-32 He et al., (2016) for Imagenette.
Parameters for the baselines are configured according to their original
papers. For our PEMM, we have the same setting as Section 5.1. All networks
are trained using SGD with momentum 0.9, weight decay 5 × 10-3 and an initial
learning rate of 0.1. The learning rate is divided by 10 after 20 and 50
epochs for CIFAR-10 (80 epochs in total),after 80 and 120 epochs for CIFAR-100
(150 epochs in total) and after 50 ,100 and 150 epochs for Imagenette (200
epochs in total). Simple data augmentation techniques (width/height shift and
horizontal flip) are applied on all datasets.

Next, we also test our proposed method on Clothing1M Xiao et al., (2015). The
Clothing1M dataset contains 1 million images and the overall accuracy of the
labels is ${\sim 61.54\%}$ and may contain both symmetric and asymmetric label
noise. The classification accuracy on the $10k$ clean testing data is used as
the evaluation metric.

The ResNet-50 with pretrained weights is used for Clothing1M dataset. We
follow the same image processing setting in Wang et al., (2019). We train the
models with batch size 64 and initial learning rate $10^{-3}$, which is
reduced by $1/10$ after 5 epochs (10 epochs in total). SGD with a momentum 0.9
and weight decay $10^{-3}$ are adopted in our experiment. Other settings are
the same as we did in CIFAR-100 dataset.

Robustness performance: All the results are shown in Table1 and Table2.

In Table 1, one can see that PEMM shows a clear improvement over other
methods. This is likely because that, other methods care too much about the
noisy labels itself, while PEMM ensures richer representations that preserves
intrinsic structures from the data, relies less on class labels.Moreover, the
equivalent early stopping property also benefit our result and avoid over-
learning on noisy class labels.

In Table 2, it is witnessed that our methods have competitive performance and
does not require extra auxiliary information for noisy label correction.

Table 2: Accuracy (%) of different models on real-world noisy dataset
Clothing1M. The best results are in bold.

Methods CE Bootstrap Forward GCE SCE PEMM Acc 68.80 68.94 69.84 69.75 71.02
$\bm{72.80}$

##  7 Conclusions

We present a PEMM based training algorithm that it can handle noisy labels
without requiring priors from clean data. A robust mixture model is proposed
to approximate the latent distribution of data without supervision. It is
implemented by the distance-based classifier with the PE loss on its class
centers. We provide both theoretical and empirical analysis on PEMM, and
demonstrate its effectiveness against various types and rates of label noise
on both benchmark and real-world datasets. Overall, due to its simplicity and
ease of implementation, we believe that PEMM shows great potential for noisy
data training. In the future, it can be merged with other advanced techniques
for real-world applications.

## References

  * Arazo et al., (2019) Arazo, E., Ortego, D., Albert, P., O’Connor, N. E., and McGuinness, K. (2019).  Unsupervised label noise modeling and loss correction.
  * C, (2009) C, J. (2009).  Textbook Of Engineering Physics.  Prentice-Hall Of India Pvt. Limited.
  * Chen et al., (2019) Chen, C., Chen, Z., Jiang, B., and Jin, X. (2019).  Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3296–3303.
  * Deng et al., (2019) Deng, J., Guo, J., Xue, N., and Zafeiriou, S. (2019).  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699.
  * Fort, (2017) Fort, S. (2017).  Gaussian prototypical networks for few-shot learning on omniglot.
  * Ghosh et al., (2017) Ghosh, A., Kumar, H., and Sastry, P. S. (2017).  Robust loss functions under label noise for deep neural networks.
  * Guo and Zhang, (2021) Guo, Y. and Zhang, C. (2021).  Recent advances in large margin learning.
  * He et al., (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
  * Jiang et al., (2018) Jiang, L., Zhou, Z., Leung, T., Li, L.-J., and Fei-Fei, L. (2018).  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In International Conference on Machine Learning, pages 2304–2313. PMLR.
  * Kobourov, (2012) Kobourov, S. G. (2012).  Spring embedders and force directed graph drawing algorithms.  Computer Science.
  * Krizhevsky et al., (2009) Krizhevsky, A., Hinton, G., et al. (2009).  Learning multiple layers of features from tiny images.
  * Li et al., (2020) Li, J., Socher, R., and Hoi, S. C. H. (2020).  Dividemix: Learning with noisy labels as semi-supervised learning.
  * Li et al., (2019) Li, M., Soltanolkotabi, M., and Oymak, S. (2019).  Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks.
  * Li et al., (2017) Li, Y., Yang, J., Song, Y., Cao, L., Luo, J., and Li, L.-J. (2017).  Learning from noisy labels with distillation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1910–1918.
  * Malach and Shalev-Shwartz, (2017) Malach, E. and Shalev-Shwartz, S. (2017).  Decoupling "when to update" from "how to update".
  * McCall, (2010) McCall, R. (2010).  Physics of the Human Body.  Johns Hopkins University Press.
  * Menon et al., (2019) Menon, A. K., Rawat, A. S., Reddi, S. J., and Kumar, S. (2019).  Can gradient clipping mitigate label noise?  In International Conference on Learning Representations.
  * Patrini et al., (2017) Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., and Qu, L. (2017).  Making deep neural networks robust to label noise: A loss correction approach.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944–1952.
  * Reed et al., (2014) Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., and Rabinovich, A. (2014).  Training deep neural networks on noisy labels with bootstrapping.  arXiv preprint arXiv:1412.6596.
  * Song et al., (2021) Song, H., Kim, M., Park, D., Shin, Y., and Lee, J.-G. (2021).  Learning from noisy labels with deep neural networks: A survey.
  * Szegedy et al., (2016) Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016).  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826.
  * Tong et al., (2015) Tong, X., Tian, X., Yi, Y., Chang, H., and Wang, X. (2015).  Learning from massive noisy labeled data for image classification.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Vahdat, (2017) Vahdat, A. (2017).  Toward robustness against label noise in training deep discriminative neural networks.  arXiv preprint arXiv:1706.00038.
  * Van der Maaten and Hinton, (2008) Van der Maaten, L. and Hinton, G. (2008).  Visualizing data using t-sne.  Journal of machine learning research, 9(11).
  * Vapnik and V., (1999) Vapnik and V., N. (1999).  An overview of statistical learning theory.  IEEE Trans Neural Netw, 10(5):988–999.
  * (26) Wang, H., Wang, Y., Zhou, Z., Ji, X., Gong, D., Zhou, J., Li, Z., and Liu, W. (2018a).  Cosface: Large margin cosine loss for deep face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265–5274.
  * (27) Wang, Y., Liu, W., Ma, X., Bailey, J., and Xia, S. T. (2018b).  Iterative learning with open-set noisy labels.  IEEE.
  * Wang et al., (2019) Wang, Y., Ma, X., Chen, Z., Luo, Y., Yi, J., and Bailey, J. (2019).  Symmetric cross entropy for robust learning with noisy labels.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 322–330.
  * Xiao et al., (2015) Xiao, T., Xia, T., Yang, Y., Huang, C., and Wang, X. (2015).  Learning from massive noisy labeled data for image classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2691–2699.
  * Yao et al., (2020) Yao, Q., Yang, H., Han, B., Niu, G., and Kwok, J. T.-Y. (2020).  Searching to exploit memorization effect in learning with noisy labels.  In International Conference on Machine Learning, pages 10789–10798. PMLR.
  * Yu et al., (2020) Yu, Y., Chan, K. H. R., You, C., Song, C., and Ma, Y. (2020).  Learning diverse and discriminative representations via the principle of maximal coding rate reduction.  Advances in Neural Information Processing Systems, 33.
  * Zhang et al., (2017) Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2017).  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412.
  * Zhang and Sabuncu, (2018) Zhang, Z. and Sabuncu, M. R. (2018).  Generalized cross entropy loss for training deep neural networks with noisy labels.