# A note on sign-changing solutions to supercritical Yamabe-type equations

Jurgen Julio-Batalla  Universidad Industrial de Santander, Carrera 27 calle 9,
680002, Bucaramanga, Santander, Colombia<EMAIL_ADDRESS>
###### Abstract.

On a closed Riemannian manifold $(M^{n},g)$, we consider the Yamabe-type
equation $-\Delta_{g}u+\lambda u=\lambda|u|^{q-1}u$, where
$\lambda\in\mathbb{R}_{+}$ and $q>1$. We assume that $M$ admits a proper
isoparametric function $f$ with focal submanifolds of positive dimension. If
$k>0$ is the minimum of the dimensions of the focal submanifolds of $f$, we
let $q^{*}=\frac{n-k+2}{n-k-2}$. We prove the existence of infinite
$f$-invariant sign-changing solutions to the equation when $1<q<q^{*}$.

The author was supported by project 3756 of Vicerrectoría de Investigación y
Extensión of Universidad Industrial de Santander

##  1\. Introduction

Let $(M^{n},g)$ be a closed connected Riemannian manifold of dimension $n\geq
3$. We let $\Delta=div(\nabla)$ be the non-positive Laplace operator. In this
paper we will study nodal (sign-changing) solutions of the Yamabe-type
equation

(1.1) $-\Delta_{g}u+\lambda u=\lambda|u|^{q-1}u,$

where $\lambda\in\mathbb{R}_{+}$ and $q>1$.

If the scalar curvature of $g$, ${\textbf{s}}_{g}$, is constant equal to
$\frac{n-2}{4(n-1)}\lambda$ and $q=p_{n}=\frac{n+2}{n-2}$ is the critical
Sobolev exponent, then (1.1) is the Yamabe equation. In this case, if $u$ is a
positive solution of the equation, then $u^{\frac{4}{n-2}}g$ is a Riemannian
metric, conformal to $g$, which also has constant scalar curvature ${\bf
s}_{g}$. It is an important result obtained in several steps by H. Yamabe
[23], N. Trudinger [20], T. Aubin [3] and R. Schoen [19], that given any
closed Riemannian manifold $(M^{n},g)$ of dimension at least 3, there is a
conformal metric of constant scalar curvature. Such conformal metric is unique
if it has non-positive scalar curvature. But in the positive case the solution
in general is not unique and many important results have been obtained about
the space of conformal constant scalar curvature metrics, which amounts to
multiplicity results for positive solutions of the Yamabe equation. Many
results have also been obtained about nodal solutions of the Yamabe equation.
If $u$ is a nodal solution of Equation (1.1) then $u$ vanishes somewhere and
$|u|^{\frac{4}{n-2}}g$ is not a Riemannian metric. However such solutions
still have geometric and analytic interest. In the case of round sphere
$(\mathbb{S}^{n},g_{0})$ the first results about the existence of nodal
solutions to the Yamabe equation were obtained by Y. Ding [8], using the rich
family of symmetries of the sphere. Later, more results were obtained for
instance by M. del Pino, M. Musso, F. Pacard, A. Pistoia [7]; M. Medina, M.
Musso [16] and by J. C. Fernández, J. Petean [10]. In the context of general
Riemannian manifolds existence of nodal solutions to the Yamabe equation under
very general conditions was obtained by B. Ammann and E. Humbert [2]. For
other results on nodal solutions of this equation see the articles by G. Henry
[13], J. Vétois [21], J. Julio-Batalla, J. Petean [15] and M. J. Gursky, S.
Pérez-Ayala in [12].

When $q<p_{n}$, Equation (1.1) is called subcritical. Solutions of Equation
1.1 on the closed Riemannian manifold $(M^{n},g)$ in the subcritical range
yield solutions of the Yamabe equation for appropriate manifolds which fiber
over $M$ (in particular certain Riemannian products with $M$). See for
instance the articles by G. Henry, J. Petean [14], by R. Bettiol, P. Piccione
[5] and by L. L. de Lima, P. Piccione , M. Zedda [6] for positive solutions.
And the article by J. Petean [18] for nodal solutions. Supercritical equations
are more difficult to treat, since most of the techniques employed in the
subcritical and critical cases, do not apply in the supercritical range. But
there are some results, see for instance the article by J. C. Fernández, O.
Palmas and J. Petean [9] in the case of the sphere.

In this article we will consider the situation when the Riemannian manifold
$(M^{n},g)$ admits a proper isoparametric function $f$ with positive
dimensional level sets. Recall that a non-constant, smooth function
$f:M\rightarrow[t_{0},t_{1}]$ is called isoparametric if there exist smooth
functions $a,b$ (defined on $[t_{0},t_{1}]$) which verify that $|\nabla
f|^{2}=b(f)$ and $\Delta f=a(f)$.

From the general theory of isoparametric functions, introduced by Q-M. Wang in
[22], it is known that the only zeros of the function
$b:[t_{0},t_{1}]\rightarrow\mathbb{R}_{\geq 0}$ are $t_{0}$ and $t_{1}$, which
means that the only critical values of $f$ are its minimum and its maximum. It
is also known that $M_{0}=f^{-1}(t_{0})$ and $M_{1}=f^{-1}(t_{1})$ are smooth
submanifolds; they are called the focal submanifolds of $f$. We let $d_{i}$ be
the dimension of $M_{i}$. The isoparametric function $f$ is called proper when
$d_{0},d_{1}\leq n-2$. In this case, all the level sets $f^{-1}(t)$ are
connected.

We will say that a function $u$ on $M$ is $f$-invariant if it is constant
along the level sets of $f$, which is equivalent to say that $u=\varphi\circ
f$ for a function $\varphi:[t_{0},t_{1}]\rightarrow\mathbb{R}$. We will look
for solutions of Equation (1.1) which are $f$-invariant. Many results have
been obtained regarding solutions of the Yamabe-type equations invariant by an
isoparametric function. In [14] G. Henry and J. Petean considered the Yamabe
equation on Riemannian products with the sphere. They proved multiplicity
results for conformal constant scalar curvature metrics on the products by
constructing solutions of subcritical Yamabe-type equations on the sphere
invariant under a fixed isoparametric function. They reduce the equation to an
ordinary differential equation and use global bifurcation theory. In [13] G.
Henry proved the existence of a $f$-invariant nodal solution of the Yamabe
equation which minimizes energy among nodal solutions. In [10] J. C. Fernández
and J. Petean also use the reduction of the equation to an ordinary
differential equation to show multiplicity results for nodal solutions of the
Yamabe equation on the round sphere which are $f$-invariant. J. C. Fernández,
O. Palmas and J. Petean proved in [9] multiplicity results for nodal solutions
of supercritical Yamabe-type equations on the sphere, also using the reduction
to an ordinary differential equation.

For a fixed, proper, isoparametric function $f$ on the closed Riemannian
manifold $(M^{n},g)$ we let $k=\min\\{d_{0},d_{1}\\}$, the minimum of the
dimensions of the focal submanifolds. We let $q^{*}=\frac{n-k+2}{n-k-2}$.

We define the energy of a solution $u$ of Equation (1.1) as

$E(u)=\int_{M}|u|^{q+1}dv_{g},$

where $dv_{g}$ is the volume element of $(M^{n},g)$.

Our main result is the following

######  Theorem 1.1.

Consider a proper isoparametric function $f$ on a closed Riemannian manifold
$(M^{n},g)$. Assume that the minimum dimension of level sets of $f$, $k$, is
strictly positive. Assume that $1<q<\frac{n-k+2}{n-k-2}$. Then there exist
infinite solutions of Equation (1.1) with arbitrarily large energy.

Note that since $p_{n}=\frac{n+2}{n-2}<\frac{n-k+2}{n-k-2}$, this result
applies to critical and supercritical Yamabe-type equations. It was proved in
[4, Proposition 4.1] that there is a $C^{0}$-bound for positive $f$-invariant
solutions to the Equation (1.1) when $q<q^{*}$. It then follows that only a
finite number of the solutions given by Theorem 1.1 can be positive solutions,
and we have:

######  Corollary 1.2.

Consider a proper isoparametric function $f$ on a closed Riemannian manifold
$(M^{n},g)$. Assume that the minimum dimension of level sets of $f$, $k$, is
strictly positive. Assume that $1<q<\frac{n-k+2}{n-k-2}$. Then there exist
infinite nodal solutions of Equation (1.1).

Our result is similar to the result obtained by J. Vétois in [21, Corollary
1.1]. Indeed, J. Vétois guarantees the existence of large-energy solutions to
the Schrödinger-Yamabe type equation

$-\Delta_{g}u+hu=|u|^{p_{n}-1}u,$

where $h$ is a Hölder continuous function, assuming that
$h<\frac{n-2}{4(n-1)}\textbf{s}_{g}$.

We will prove Theorem 1.1 by applying the classical theorem of A. Ambrosetti
and P. Rabinowitz about the existence of mountain pass critical points [1,
Theorem 2.13]. This will be carried out in Section 3. In Section 2 we will
recall some information about isoparametric functions and give some
preliminary results.

##  2\. Preliminaries on isoparametric functions

Let $(M^{n},g)$ be a closed connected Riemannian manifold. Assume that $M$
admits a proper isoparametric function $f:M\rightarrow[t_{0},t_{1}]$. The
general theory of isoparametric functions on Riemannian manifolds was
introduced by Q-M Wang in [22] following the classical theory, which
considered the case of space forms. Recall that the fact that $f$ is an
isoparametric function means that there are smooth functions $a$ and $b$ such
that $\Delta(f)=a\circ f$ and $\|\nabla f\|^{2}=b\circ f$. It is proved in
[22] that the only critical values of $f$ are its minimum and maximum. The
critical level sets, $M_{0}=f^{-1}(t_{0})$ and $M_{1}=f^{-1}(t_{1})$, are
called the focal submanifolds of $f$, and it was also proved in [22] that they
are actually smooth submanifolds. The regular level sets of $f$, $f^{-1}(t)$
for $t\in(t_{0},t_{1})$, are called isoparametric hypersurfaces. We let
$d_{i}$ be the dimension of the focal submanifold $M_{i}$ and
$k=\min\\{d_{0},d_{1}\\}$. We will assume later that $k>0$. J. Ge and Z. Tang
proved in [11] that if $k\leq n-2$ then all the level sets of $f$ are
connected. In this case the isoparametric function $f$ is called proper.

We consider on $M$ the (normalized) Yamabe-type equation

(2.1) $-\Delta_{g}u+\lambda u=|u|^{q-1}u,$

where $1<q<(n-k+2)/(n-k-2)$ and $\lambda$ is a positive constant.

In this note we will find solutions of the Yamabe-type equation in the space
of $f$-invariant functions.

######  Definition 2.1.

Given an isoparametric function $f$ on the closed manifold $(M^{n},g)$ we
denote by $S_{f}$ the space of $f$-invariant functions on $M$,
$S_{f}=\\{\varphi\circ f:\varphi:[t_{0},t_{1}]\rightarrow\mathbb{R}\\}$.

Let $L^{2}_{1}(M)$ denote the usual Sobolev space of $L^{2}$-functions on $M$
which admit one derivative also in $L^{2}(M)$. Then we let
$L^{2}_{1,f}(M)=L^{2}_{1}(M)\cap S_{f}$ be the space of $L^{2}_{1}$-functions
on $M$ which are $f$-invariant.

Consider the functional $I:L^{2}_{1}(M)\rightarrow\mathbb{R}$ defined as

$I(u)=\frac{1}{2}\int_{M}\left(|\nabla u|^{2}+\lambda
u^{2}\right)dv_{g}-\frac{1}{q+1}\int_{M}|u|^{q+1}dv_{g}.$

Critical points $u$ of the functional $I$ are weak solutions of Equation
(2.1): they verify

$DI(u)[v]=\int_{M}(\langle\nabla u,\nabla v\rangle+\lambda uv)\
dv_{g}-\int_{M}u|u|^{q-1}v\ dv_{g}=0,$

for all $v\in L^{1}_{2}(M)$.

It is important to point out that $L^{2}_{1,f}(M)$ is a closed subset of
$L^{2}_{1}(M)$. Also note that if $u=\varphi\circ f\in S_{f}$ is a
$C^{2}$-function, then $\Delta u=\Delta(\varphi\circ
f)=\left(\varphi^{\prime\prime}b+\varphi^{\prime}a\right)\circ f\in S_{f}$.

Let $H$ be the $L^{2}$-orthogonal complement of $L^{2}_{1,f}(M)$ in
$L^{2}_{1}(M)$. Then for any $v\in H$ and any $u\in L^{2}_{1,f}(M)\cap
C^{2}(M)$ we have that

$\int_{M}\langle\nabla u,\nabla v\rangle dv_{g}=-\int_{M}\ (\Delta u)v\
dv_{g}=0.$

Since $L^{2}_{1,f}(M)\cap C^{2}(M)$ is dense in $L^{2}_{1,f}(M)$ it follows
that for any $u\in L^{2}_{1,f}(M)$ and any $v\in H$ we have that

$DI(u)[v]=\int_{M}(\langle\nabla u,\nabla v\rangle+\lambda uv)\
dv_{g}-\int_{M}u|u|^{q-1}v\ dv_{g}=0.$

Therefore we have:

######  Lemma 2.2.

If $u\in L^{2}_{1,f}(M)$ is a critical point of the restriction of $I$ of
$L^{2}_{1,f}(M)$, $I:L^{2}_{1,f}(M)\rightarrow\mathbb{R}$, then $u$ is a
strong solution of Equation (2.1).

The previous comments show that if $u\in L^{2}_{1,f}(M)$ is a critical point
of the restriction of $I$ of $L^{2}_{1,f}(M)$, then $u$ is a critical point in
$L^{2}_{1}(M)$, and therefore a weak solution of Equation (2.1). Then it
follows by classical elliptic regularity that $u$ is a strong solution.

It was proved in [13, Section 6] that if $1<q<(n-k+2)/(n-k-2)$ then
$L^{2}_{1,f}(M)\subset L_{f}^{q+1}(M)$ and moreover the inclusion is compact.
The following lemma is part of the previous statement, we include a short
proof for completeness since we will need it in the next section:

######  Lemma 2.3.

Assume that $1<q<(n-k+2)/(n-k-2)$. There exists a positive constant $C$ such
that if $u\in L^{2}_{1,f}(M)$, then

$|u|_{L^{q+1}(M)}\leq C|u|_{L^{2}_{1}(M)}.$

###### Proof.

Since the focal submanifolds $M_{0},M_{1}$ of the function $f$ have positive
dimensions ($d_{0},d_{1}$, respectively), it is well-known that $M^{n}$ can be
identified as a union of two disk bundles $D_{0},D_{1}$, each one over $M_{0}$
and $M_{1}$ respectively. More precisely, for $i\in\\{0,1\\}$ let
$exp_{M_{i}}$ be the normal exponential map of $M_{i}$ in $M$. Following the
notation used by R. Miyaoka in [17] the manifold $M^{n}$ is diffeomorphic to

$N_{\leq a}M_{0}\cup N_{\leq a}M_{1},$

where

$N_{\leq
a}Q=\\{exp_{Q}(s\eta)/\quad|\eta|=1,s<a\\}\cup\\{\exp_{Q}(a\eta)/\quad|\eta|=1\\},$

and $2a=d_{g}(M_{0},M_{1})$.

From the definition of disk bundle, we can choose a coordinate system
$(U_{j},\varphi_{j})$ on $N_{\leq a}M_{0}$ (and similar ones on $N_{\leq
a}M_{1}$) such that $U_{j}=\pi^{-1}(U^{\prime}_{j})$ for a finite cover
$\\{U^{\prime}_{j}\\}$ of $M_{0}$ and each $\varphi_{j}$ is a diffeomorphism
defined by

$\varphi_{j}:U_{j}\rightarrow U^{\prime}_{j}\times D^{n-m_{0}}_{0}(a),$

where $D^{n-m_{0}}_{0}(a)$ is the disk of radius $a$ in the normal bundle of
$M_{0}$.

Without loss of generality, we cover $M^{n}$ by a finite number $m$ of these
type of charts with a uniform bound for the metric tensor $g$ i.e. there
exists a constant $c>1$ with

$c^{-1}\ Id\leq g^{l}_{ij}\leq c\ Id\quad\text{for}\quad
l\in\\{1,\cdots,m\\}.$

Let $s=n-d_{0}$.

For any $f$-invariant function $u$ we have that $u$ only depends on the normal
coordinates on $N_{\leq a}M_{0}$ and $N_{\leq a}M_{1}$. In particular, using
the previous charts $(U_{j},\varphi_{j})$ on $N_{\leq a}M_{0}$, the function
$u$ only depends on $D^{s}_{0}(a)$ (similarly, $u$ only depends on
$D_{0}^{n-d_{1}}(a)$ in $N_{\leq a}M_{1}$).

For some positive constants $k_{0}$ and $k_{1}$ we have:

$\displaystyle\int_{U_{l}}u^{q+1}dv_{g}$
$\displaystyle=\int\limits_{U^{\prime}_{l}\times
D_{0}^{s}(a)}u^{q+1}\sqrt{detg^{l}_{ij}}dxdy$
$\displaystyle=k_{0}\int\limits_{D_{0}^{s}(a)}u^{q+1}\sqrt{detg^{l}_{ij}}dy$
$\displaystyle\leq k_{1}\int\limits_{D_{0}^{s}(a)}u^{q+1}dy.$

Therefore for some positive constant $k_{2}$ we have:

$|u|_{L^{q+1}(U_{l})}\leq k_{2}|u|_{L^{q+1}(D_{0}^{s}(a))}.$

Since $g^{l}_{ij}$ is also bounded from below, with a similar argument we can
get that:

$|u|_{L^{2}_{1}(U_{l})}\geq k_{3}|u|_{L^{2}_{1}(D_{0}^{s}(a))},$

for some positive constant $k_{3}$.

Applying the usual Sobolev inequalities in Euclidean space
$D_{0}^{s}(a)\subset\mathbb{R}^{n-m_{0}}$ we have that for some positive
constant $k_{4}$ we have:

$|u|_{L^{q+1}(D_{0}^{s}(a))}\leq k_{4}|u|_{L^{2}_{1}(D_{0}^{s}(a))},$

and therefore,

$|u|_{L^{q+1}(U_{l})}\leq C(l)|u|_{L^{2}_{1}(U_{l})}.$

Then the inequality in the lemma follows by taking a partition of unity on $M$
subordinate to $\\{U_{l}\\}_{l}$ and adding up.

∎

##  3\. Mountain pass critical points

In this section we will give the proof of Theorem 1.1. We will apply the
classical min-max method of Ambrosetti-Rabinowitz to the functional
$I:L^{2}_{1,f}(M)\rightarrow\mathbb{R}$ defined as

$I(u)=\frac{1}{2}\int_{M}\left(|\nabla u|^{2}+\lambda
u^{2}\right)dv_{g}-\frac{1}{q+1}\int_{M}|u|^{q+1}dv_{g}.$

We briefly recall the min-max Theorem 2.13 in [1]:

Let $B(\rho)$ be the open ball of center at $0$ and radius $\rho>0$ in the
space $L_{1,f}^{2}(M)$ and

$\Gamma^{*}=\\{h:L_{1,f}^{2}(M)\rightarrow L_{1,f}^{2}(M)/\;I(h(B(1))\geq
0\;,\;h\;\text{is an odd homeomorphism}\\}.$

If the functional $I$ satisfies the conditions

  1. (1)

$I(0)=0$ and there exist $\rho,\delta>0$ such that $I>0$ in $B(\rho)-\\{0\\}$
and $I\geq\delta$ on $\partial B(\rho);$

  2. (2)

the Palais-Smale condition: if $u_{m}$ is a sequence in $L^{2}_{1,f}(M)$ such
that $I(u_{m})$ is bounded and $DI(u_{m})\rightarrow 0$, then it has a
convergent subsequence;

  3. (3)

$I(u)=I(-u)$ for any $u\in L^{2}_{1,f}(M)$, and for any finite dimensional
subspace $E$ in $L_{1,f}^{2}(M)$, $E\cap\\{I\geq 0\\}$ is bounded.

Then the content of Theorem 2.13 in [1] is that given a sequence $E_{m}$ of
subspaces of $E$, $E_{m}\subset E_{m+1}$ ($m$ = dim($E_{m}$)), there exists an
increasing sequence of critical values $c_{m}$ of $I$, given by

$c_{m}=\sup\limits_{h\in\Gamma^{*}}\inf\limits_{u\in
E_{m-1}^{\perp}\cap\;\partial B(1)}I(h(u)).$

First we will verify that $I$ in fact satisfies conditions 1, 2 and 3.

Indeed,

-Condition 1:

Note that since $\lambda>0$, the operator $P:=-\Delta+\lambda$ is coercive,
and therefore there exists a positive constant $C_{1}$ such that for any
function $u\in L_{1,f}^{2}(M)$ we have:

$\int_{M}uP(u)dv_{g}\geq C_{1}|u|^{2}_{L_{1}^{2}(M)}.$

Moreover, by Lemma 2.3 there exists a positive constant $C_{2}$ such that for
any function $u\in L_{1,f}^{2}(M)$ we have:

$\int_{M}u^{q+1}dv_{g}\leq C_{2}|u|^{q+1}_{L^{2}_{1}(M)}.$

Hence

$I(u)\geq C_{3}|u|^{2}_{L_{1}^{2}(M)}-C_{4}|u|^{q+1}_{L^{2}_{1}(M)},$

for positive constants $C_{3},C_{4}$.

Consider a function $u\in L^{2}_{1,f}(M)$ with $|u|_{L^{2}_{1}(M)}=1$. It
follows from the previous inequality that for any $t>0$, $I(tu)\geq
t^{2}C_{3}-t^{q+1}C_{4}$. Then we can deduce that the assumption that $q+1>2$
implies the existence of a small $\rho>0$ such that the functional $I$
satisfies Condition 1.

-Condition 2:

Consider a Palais-Smale sequence $u_{m}$ in $L^{2}_{1,f}(M)$ for the
functional $I$. Then it follows that the sequence is bounded in
$L^{2}_{1,f}(M)$, and therefore it has a subsequence which is weakly
convergent. Then by the compactness of the embedding of $L^{2}_{1,f}(M)$ in
$L_{f}^{q+1}(M)$ [13, Lemma 6.1] we have a convergent subsequence.

-Condition 3:

Let $E_{m}$ be a subspace of $L^{2}_{1,f}(M)$ of dimension $m$. There is a
positive constant $C$ so that for all $u$ in $E_{m}$ we have

$\int_{M}uP(u)dv_{g}\leq C|u|^{q+1}_{L^{q+1}(M)}.$

Then for any $u\in E_{m}$ with $\int_{M}uP(u)dv_{g}=1$ and for $t>0$ we have

$I(tu)\leq\frac{t^{2}}{2}-\frac{t^{q+1}}{(q+1)C}.$

Thus $E_{m}\cap\\{I\geq 0\\}$ is bounded.

Therefore we can apply Theorem 2.13 in [1] to obtain that there exists a
sequence $u_{m}\in L^{2}_{1,f}$ of critical points of the functional
$I:L^{2}_{1,f}(M)\rightarrow\mathbb{R}$, associated to the mountain pass level
$c_{m}$. The critical points are strong solutions of the Yamabe-type equation
(1.1) by Lemma 2.2.

It follows that to conclude the proof of Theorem 1.1 we only need to prove
that the solutions $u_{m}$ have large energy, i. e. that the increasing
sequence $c_{m}$ goes to infinity.

Let $\\{e_{i}\\}_{i\geq 1}$ be an orthonormal basis of $L^{2}_{1,f}(M)$ and
$E_{m}=\langle e_{1},...,e_{m}\rangle$.

Let

$N=\left\\{u\in
L^{2}_{1,f}(M)-\\{0\\}\;/\quad\frac{1}{2}\int_{M}uP(u)dv_{g}=\frac{1}{q+1}\int_{M}|u|^{q+1}dv_{g}\right\\},$

and consider

$d_{m}=\inf\left\\{\left(\int_{M}uP(u)dv_{g}\right)^{1/2}\;/\;u\in N\cap
E_{m}^{\perp}\right\\}.$

Clearly $d_{m}$ is a non-decreasing sequence. We will show that
$d_{m}\rightarrow\infty$. Assume that $(d_{m})$ is bounded. Then there exists
$v_{m}\in N\cap E_{m}^{\perp}$ and a positive constant $d$ such that

$\int_{M}v_{m}P(v_{m})dv_{g}\leq d,\quad\forall m.$

Since $P$ is coercive it follows that the sequence $v_{m}$ is bounded in
$L^{2}_{1,f}(M)$ and therefore it is weakly convergent to $v\in
L^{2}_{1,f}(M)$. Since $v$ is orthogonal to every $E_{m}$ we must have that
$v=0$. Since the inclusion $L^{2}_{1,f}(M)\subset L_{f}^{q+1}(M)$ is compact,
a subsequence satisfies that $v_{m}\rightarrow 0$ strongly in
$L^{q+1}_{f}(M)$.

Also by Lemma 2.3 there exists a constant $K>0$ such that

$K<\dfrac{|v_{m}|_{L^{2}_{1}(M)}}{|v_{m}|_{L^{q+1}(M)}}.$

Using that $P$ is coercive and the definition of $N$ we have that there exists
a positive constant $K_{0}$ such that

$\dfrac{|v_{m}|_{L^{2}_{1}(M)}}{|v_{m}|_{L^{q+1}(M)}}\leq
K_{0}\dfrac{\left(\int_{M}v_{m}P(v_{m})dv_{g}\right)^{1/2}}{\left(\int_{M}v_{m}P(v_{m})dv_{g}\right)^{1/(q+1)}}.$

Then since $q+1>2$ it follows that

$\int_{M}v_{m}P(v_{m})dv_{g}$

is bounded from below by a positive constant. Again by definition of $N$, the
sequence

$|v_{m}|_{L^{q+1}(M)}$

also must be bounded from below by a positive constant. This is a
contradiction.

Therefore $d_{m}\rightarrow\infty$.

Now, following a similar approach used in [1] we will compare the values
$d_{m}$ and $c_{m}$:

First note that for all $u\in L_{1,f}^{2}(M)-\\{0\\}$ there exists a constant
$C>1$ such that

$\left(\int_{M}uP(u)dv_{g}\right)^{1/2}<C|u|_{L^{2}_{1}(M)}.$

Moreover we can find a unique positive constant $\alpha(u)$ such that
$\alpha(u)u\in N$. In particular

(3.1)
$\frac{1}{2(\alpha(u))^{q-1}}\int_{M}uP(u)dv_{g}=\frac{1}{q+1}\int_{M}|u|^{q+1}dv_{g}.$

Additionally, if $u\in E_{m}^{\perp}$ then

(3.2) $d_{m}\leq\left(\int_{M}\alpha uP(\alpha u)dv_{g}\right)^{1/2}<\alpha
C|u|_{L^{2}_{1}(M)}.$

These constants are fundamental in the next construction

######  Lemma 3.1.

[1] For each $m$ there exists $h_{m}\in\Gamma^{*}$ such that

$\inf\limits_{u\in E_{m}^{\perp}\cap\partial B(1)}I(h_{m}(u))\geq kd_{m}^{2}$

for some constant $k>0.$

###### Proof.

Using 3.1,3.2 we have that, for each $u\in E_{m}^{\perp}$ with
$|u|_{L^{2}_{1}(M)}\leq 1$,

$\displaystyle I\left(\frac{d_{m}}{C}u\right)=$
$\displaystyle\frac{d_{m}^{2}}{2C^{2}}\int_{M}uP(u)dv_{g}-\left({\frac{d_{m}}{C}}\right)^{q+1}\frac{1}{2\alpha^{q-1}}\int_{M}uP(u)dv_{g}$
$\displaystyle=$
$\displaystyle\frac{d_{m}^{2}}{C^{2}}\left(1-\left(\frac{d_{m}}{C\alpha}\right)^{q-1}\right)\left(\frac{1}{2}\int_{M}uP(u)dv_{g}\right)$
$\displaystyle\geq$ $\displaystyle d_{m}^{2}k_{0}\int_{M}uP(u)dv_{g}$

for some $k_{0}>0$.

Thus

(3.3) $\frac{d_{m}}{C}(E_{m}^{\perp}\cap B(1))\subset\\{I>0\\}\cup\\{0\\}.$

Furthermore, from the coercivity of $P$ it follows that

$I\left(\frac{d_{m}}{C}u\right)\geq kd_{m}^{2},$

for some $k>0$ and for all $u\in E_{m}^{\perp}\cap\partial B(1)$.

We can suppose that there is some $\epsilon>0$ for which

(3.4) $Z_{\epsilon}:=\frac{d_{m}}{C}(E_{m}^{\perp}\cap
B(1))\bigoplus\epsilon(E_{m}\cap B(1))\subset\\{I>0\\}\cup\\{0\\}.$

Indeed, if it is not the case, there exist $\epsilon_{i}$ and $u_{i}\in
Z_{\epsilon_{i}}-\\{0\\}$ such that $\epsilon_{i}\rightarrow 0$ and
$u_{i}\notin\\{I>0\\}$. The sequence $(u_{i})$ is bounded in $L_{1,f}^{2}(M)$,
so as we discussed above $u_{i}$ converges weakly in $L_{1,f}^{2}(M)$ and
strongly in $L^{q+1}_{f}(M)$ to $u_{0}$. Again by Lemma 2.3 and coercivity of
$P$,

$0<K<\dfrac{\int_{M}u_{i}P(u_{i})dv_{g}}{|u_{i}|^{2}_{L^{q+1}(M)}}.$

Since $I(u_{i})\leq 0$ and $q>1$ it follows that $|u_{i}|_{L^{q+1}(M)}$ is
uniformly bounded from below and therefore $\int_{M}u_{0}^{q+1}dv_{g}>0$. In
particular $u_{0}\neq 0$ and it follows from 3.3 that $I(u_{0})>0$. On the
other hand since $u_{0}$ is the weak limit in $L^{2}_{1,f}(M)$ of $u_{i}$, we
have that $I(u_{0})\leq 0$. This is a contradiction and therefore we can pick
a positive $\epsilon$ such that 3.4.

We can fix such $\epsilon$ and define the linear map $h_{m}$ from
$L^{2}_{1,f}(M)$ into $L^{2}_{1,f}(M)$ as

$h_{m}(u):=\frac{d_{m}}{C}u_{1}+\epsilon u_{2},$

where $u=u_{1}+u_{2}$ for $u_{1}\in E_{m}^{\perp}$, $u_{2}\in E_{m}$.

By construction $h_{m}(B(1))\subset Z_{\epsilon}\subset\\{I\geq 0\\}$. Then
$h_{m}\in\Gamma^{*}$.

Finally,

$\inf\limits_{u\in E_{m}^{\perp}\cap\partial B(1)}I(h_{m}(u))\geq kd_{m}^{2}.$

∎

It is clear from the definition of $c_{m}$ and Lemma 3.1 that $c_{m+1}\geq
kd_{m}^{2}$ and hence the proof of Theorem 1.1 is complete.

## References

  * [1] A. Ambrosetti, P. H. Rabinowitz, Dual variational methods in critical point theory and applications, J. Funct. Anal., 14, (1973) 349–381.
  * [2] B. Ammann, E. Humbert, The second Yamabe invariant, J. Funct. Anal. 235 (2006), 377-412.
  * [3] T. Aubin, Equations differentielles non-lineaires et probleme de Yamabe concernant la courbure scalaire, J. Math. Pures Appl. 55 (1976), 269-296.
  * [4] A. Betancourt de la Parra, J. Julio-Batalla, J. Petean, Global bifurcation techniques for Yamabe type equations on Riemannian manifolds, Nonlinear Ana.202 (2021) 112140.
  * [5] R. Bettiol, P. Piccione, Multiplicity of solutions to the Yamabe problem on collapsing Riemannian submersions, Pacific J. Math. 266 (2013), 1-21.
  * [6] L. L. de Lima, P. Piccione, M. Zedda, On bifurcation of solutions of the Yamabe problem on product manifolds, Ann. Inst. H. Poincaré Anal. Non Linéaire 29 (2012) , 261-277.
  * [7] M. del Pino, M. Musso, F. Pacard, A. Pistoia, Large energy entire solutions for the Yamabe equation, J. Differential Equations 251 (2011), 2568-2597.
  * [8] Y. Ding, On a conformally invariant elliptic equation on $\mathbb{R}^{n}$, Comm. Math. Phys. 107 (1986), no. 2, 331-335.
  * [9] J. C. Fernández, O. Palmas, J.Petean, Supercritical elliptic problems on the round sphere and nodal solutions to the Yamabe problem in projective spaces, Discrete Contin. Dyn. Syst. 40 (2020), pp. 2495–2514.
  * [10] J. C. Fernández, J. Petean, Low energy nodal solutions to the Yamabe equation, J. Differential Equations. 268 (11) (2020), pp. 6576-6597
  * [11] J. Ge, Z. Tang, Isoparametric functions and exotic spheres, J. Reine Angew. Math. 683 (2013), 161-180.
  * [12] M.J. Gursky, S. Pérez-Ayala .Variational properties of the second eigenvalue of the conformal Laplacian J. Funct. Anal., 282 (8) (2022)
  * [13] G. Henry.Isoparametric functions and nodal solutions of the Yamabe equation . Ann. of Global Anal. and Geometry. 56, 203–219 (2019)
  * [14] G. Henry, J. Petean, Isoparametric hypersurfaces and metrics of constant scalar curvature, Asian J. Math. 18 (2014), 53-67.
  * [15] J. Julio-Batalla, J. Petean. Nodal solutions of Yamabe- type equations on positive Ricci curvature manifolds. Proc. Amer. Math. Soc. 149, 2021, 4419–4429
  * [16] M. Medina, M. Musso Doubling nodal solutions to the Yamabe equation in $\mathbb{R}^{n}$ with maximal rank J. Math. Pures Appl. (9), 152 (2021), pp. 145-188
  * [17] R. Miyaoka. Transnormal functions on a Riemannian manifold. Diff. Geometry and its Appl. 31 (2013) 130–139.
  * [18] J. Petean, Nodal solutions of the Yamabe equation on products, J. Geom. Phys. 59 (2009), 1395-1401.
  * [19] R. Schoen, Conformal deformation of a Riemannian metric to constant scalar curvature, J. Differential Geometry 20 (1984), 479-495.
  * [20] N. Trudinger, Remarks concerning the conformal deformation of Riemannian structures on compact manifolds, Ann. Scuola Norm. Sup. Pisa (3) 22 (1968), 265–274.
  * [21] J. Vétois. Multiple solutions for nonlinear elliptic equations on compact Riemannian manifolds. Int. J. Math., 18 (9) (2007), pp. 1071-1111
  * [22] Q-M. Wang, Isoparametric functions on Riemannian manifolds, Math. Ann. 277 (1987), 639-646.
  * [23] H. Yamabe, On a deformation of Riemannian structures on compact manifolds, Osaka Math. J. 12 (1960), 21-37.


# Coordinated Sparse Recovery of Label Noise

Yukun Yang, , Naihao Wang, Haixin Yang, Ruirui Li Yukun Yang, Naihao Wang,
Ruirui Li are with the Department of Information Science and Technology,
Beijing University of Chemical Technology, Beijing. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Haixin Yang is with the
Department of Mathematical Sciences, Peking University. E-mail:
<EMAIL_ADDRESS>received April 19, 2005; revised August
26, 2015.

###### Abstract

Label noise is a common issue in real-world datasets that inevitably impacts
the generalization of models. This study focuses on robust classification
tasks where the label noise is instance-dependent. Estimating the transition
matrix accurately in this task is challenging, and methods based on sample
selection often exhibit confirmation bias to varying degrees. Sparse over-
parameterized training (SOP) has been theoretically effective in estimating
and recovering label noise, offering a novel solution for noise-label
learning. However, this study empirically observes and verifies a technical
flaw of SOP: the lack of coordination between model predictions and noise
recovery leads to increased generalization error. To address this, we propose
a method called Coordinated Sparse Recovery (CSR). CSR introduces a
collaboration matrix and confidence weights to coordinate model predictions
and noise recovery, reducing error leakage. Based on CSR, this study designs a
joint sample selection strategy and constructs a comprehensive and powerful
learning framework called CSR+. CSR+ significantly reduces confirmation bias,
especially for datasets with more classes and a high proportion of instance-
specific noise. Experimental results on simulated and real-world noisy
datasets demonstrate that both CSR and CSR+ achieve outstanding performance
compared to methods at the same level.

###### Index Terms:

Noisy label learning, noise recovery, sparse modeling, over-parameterization,
collaboration matrix, confidence weight, sample-selection

##  1 Introduction

Deep learning has achieved remarkable success in various fields such as
computer vision and medical image processing, primarily relying on high-
quality labeled training data. Unfortunately, manually annotating data is
costly and time-consuming, inevitably resulting in the introduction of noisy
labels due to human bias and lack of expert knowledge. Alternatively, using
web scraping or crowdsourcing to acquire labeled data often leads to a high
proportion of label noise. Training neural networks directly on noisy data
yields deteriorated performance, since over-parameterized deep neural networks
have enough capacity to fit even completely random labels[1],[2],[3].
Consequently, the research area known as learning with noisy labels (LNL) [4],
which seeks to address the adverse impact of label noise, has attracted
considerable attention in recent years [5],[6],[7],[8],[9],[10],[11],[12].

Some regularization methods [13], [14], [15] aim to prevent overfitting to
noise by imposing additional constraints, but this may introduce permanent
bias. Noise transition matrix can be employed to represent the transfer
relationship from clean labels to noisy labels. Although methods based on
noise transition matrix have theoretical statistical consistency, accurately
estimating them is a highly challenging task [16], [17], especially for
instance-dependent label noise. Recent research has explored the use of memory
effects through early learning
methods[3],[9],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28] in LNL
tasks. The most widely used approach is sample selection, which typically
utilizes the confidence of the early-stage deep neural networks (DNNs) to
select reliable samples for network update. However, these methods not only
require prior knowledge of the noise-label ratio but also have the potential
to select incorrect samples, leading to varying degrees of confirmation bias.
Ways to make better use of early learning to reduce confirmation bias need
further exploration.

In recent years, there has been a significant focus on the investigation of
learning behavior in over-parameterized networks. Research [29] has discovered
that wider over-parameterized networks demonstrate increased robustness,
whereas the robustness of deeper over-parameterized networks relies on
specific initialization methods and learning rates. In order to enhance early
learning and unleash its potential, recent studies have introduced a robust
training approach known as sparse over-parameterization (SOP). This approach
leverages the inherent generalization ability of over-parameterization and
integrates implicit regularization techniques to model and recover noisy
labels that may differ across instances. Theoretical analyses demonstrate that
SOP is effective in handling instance-dependent noise without introducing
additional bias. However, SOP is beset by technical flaws and suffers from a
non-coordinated learning issue between model prediction and label noise
recovery. This issue results in error leakage, leading to overfitting and the
memorization of noise, ultimately compromising the network’s generalization
capability.

This study analyzes this phenomenon and empirically concludes the following
findings: 1) Sparse recovery in over-parameterized training exhibits a general
lack of coordination between the learning of model parameters and that of
noise parameters. 2) This lack of coordination contributes to error leakage
and increased model prediction errors. Building upon these findings, we
present a novel approach for coordinating sparse noise recovery through a
collaboration matrix. It also suggests tuning the balance between the updates
of model parameters and those of noise parameters by leveraging confident
weights, consequently enhancing the reliability of model predictions. As a
result, we propose the Coordinated Sparse Recovery (CSR) method. Comparing CSR
with SOP, CSR effectively mitigates the issue of error memorization resulting
from insufficient coordination and enhances generalization performance.
Moreover, a joint sample selection strategy is introduced, based on the loss
distribution and noise probability distribution derived during the CSR
learning process. Constructing a robust learning framework using this strategy
significantly mitigates confirmation bias and improves test accuracy. Our
approach achieved superior performance on datasets with diverse proportions of
instance-dependent noise, as well as on challenging real-world noisy datasets.

In summary, this paper presents several contributions, including but not
limited to:

$\bullet$ Introducing the collaboration matrix to address the issue of non-
coordinated learning between model prediction and label noise recovery, thus
reducing the leakage of error gradients.

$\bullet$ Proposing the use of confidence weight to tune the balance between
model parameter updates and sparse noise parameter updates, thereby improving
the reliability of model predictions.

$\bullet$ Developing a novel joint sample selection strategy based on the loss
distribution and noise probability distribution obtained from the Coordinated
Sparse Recovery (CSR), capable of partitioning samples into clean, hard, and
noisy subsets.

$\bullet$ Conducting comprehensive experiments on synthetic instance-dependent
noise datasets, as well as real-world noise datasets, to demonstrate the
effectiveness of the proposed method.

##  2 RELATED WORK

###  2.1 Learning From Noisy Labels

We focus on discussing methods that do not require additional clean labeled
datasets, nor consider methods for noisy label learning on open sets[30].

The method designed for learning with noisy labels can be roughly summarized
into three categories: robust loss functions, loss correction methods, and
sample selection methods.

Robust loss functions are designed to maintain robustness in the presence of
label noise by using a cost function. This category of methods includes MAE
loss [8], improved reweighted MAE loss [31], general cross-entropy loss [9],
symmetric cross-entropy loss [10], and loss functions designed based on
information theory like $L_{DIM}$ [32]. Loss correction methods explicitly
adjust the loss function to consider the noise distribution represented by a
transition matrix of error probabilities [33],[34],[35],[36]. The transition
matrix has statistical consistency in theory, but it is difficult to
accurately estimate them in the case of instance-dependent noise. The existing
techniques of robust loss and loss correction have not fully explored the
phenomenon of early learning[2], that is, deep convolutional neural networks
tend to fit clean annotations before memorizing erroneous labels. Early
learning can be explored through sample selection, where the outputs of these
models in the early learning stage are used to predict whether the samples are
incorrectly or correctly labeled, assuming that samples with erroneous labels
tend to have higher loss values [3, 37].

###  2.2 Sample Selection Methods

Co-teaching [21] employs two parallel networks to select samples and update
one network using the gradients of the samples with small-loss samples from
the other network, in order to reduce error accumulation. However, this
selection process may lead to the selection of overly simple and non-diverse
samples, thus affecting the model’s ability to generalize. Another approach
involves correcting labels using models obtained during the early stage of
learning. This is where Co-correcting [38] comes into play, generating pseudo-
labels for noisy samples by blending the probabilities of the one-hot labels
with the model’s prediction probabilities. DivideMix [22] uses two Gaussian
mixture models to estimate the predictive losses of two networks and selects
samples based on this, combined with the MixMatch [39] semi-supervised
learning framework. To mitigate confirmation bias, many studies design
optimized sample selection strategies. For instance, BARE [40] leverages an
adaptive sample selection technique that statistics the samples within a
batch. UNICON [23] introduces a uniform selection mechanism based on JS
divergence in the embedding space. DISC [41] utilizes a dynamic and instance-
specific threshold for sample selection.

In contrast to the explicit sample selection methods mentioned above that rely
on specific strategies, the proposed CSR in this paper can be viewed as a loss
correction method that dynamically explores and utilizes early learning
characteristics. Through the collaboration matrix and confident weight, it
achieves coordinated and reliable implicit sample selection and label
correction.

###  2.3 Sparse Over-Parameterization and its Variants

The learning behavior of over-parameterized networks has garnered significant
attention from researchers, particularly in light of the remarkable success of
deep neural networks. Deep neural networks, being an archetypal case of over-
parameterization, exhibit a phenomenon called double descent [42]. This
phenomenon is characterized by an initial decrease in test error, followed by
an increase, and eventually a gradual decrease again. Some studies [43, 44]
suggest that this double descent phenomenon is associated with label noise,
which aligns with the observations in the early learning stage.

Sparse over-parameterization [45] can be viewed as an implicit regularization
technique. Unlike explicit regularization methods that introduce persistent
bias, sparse over-parameterization exploits the inclination of over-
parameterized networks towards low-rank and sparse solutions to separate and
recover corrupted labels. Initially applied in the domain of image restoration
[46], this approach can restore noisy images without any prior knowledge about
the noise. In the realm of image classification, if we assume that the noise
is sparse and dissimilar to the network trained on clean labels, we can employ
the same idea to recover the labels. Subsequent research [47] has utilized
this regularization term to train robustly and calculated the distance between
noisy samples and class centers in the embedding space to generate pseudo
labels. However, until now, there has been no investigation into the
generalization degradation resulting from the lack of coordination of over-
parameterized training with noisy labels.

##  3 Preliminary

Sparse Over-Parameterization (SOP) is a principled approach that aims to
identify and recover sparse corrupted labels for each sample by adding a noise
parameter vector $s_{i}$. It corrects the loss by adding the noise parameter
vector rather than multiplying it with the noise transition matrix.
Consequently, robust learning under noisy labels can be formulated as the
following empirical risk minimization problem:

$\min\limits_{\theta,\\{u_{i},v_{i}\\}_{i=1}^{N}}\frac{1}{N}\sum_{i=1}^{N}l(f(x_{i};\theta)+s_{i},y_{i}).$
(1)

Let $f$ denote the estimated probability of parameter $\theta$ given the input
$x_{i}$, $s_{i}$ represent the noise vector, $y_{i}$ be the one-hot encoding
of the label for the $i$-th sample, $N$ be the total number of samples, and
$l$ denote the loss function. The aim of SOP is to utilize the previously
learned robust model in the early stage to update $s_{i}$, and subsequently
utilize the learned $s_{i}$ to mitigate overfitting of noise in the
classification model. Given parameter $s_{i}$, denoting sparse noise,
initialized with a negligible value near zero, and employing an optimal
learning rate during the Stochastic Gradient Descent (SGD) for parameter
updates, it becomes feasible to restore the noise parameter $s_{i}$ under the
over-parameterization assumption with implicit regularization. The proof of
this theory is detailed in reference [45].

The transformation of $s_{i}$ can be expressed by the following formula, in
order to decouple the instance-dependent label noise into non-negative vectors
$u_{i}\odot u_{i}$ for identifying the noise and $v_{i}\odot v_{i}$ for
recovering the noise:

$s_{i}=u_{i}\odot u_{i}\odot y_{i}-v_{i}\odot v_{i}\odot(1-y_{i}).$ (2)

In this context, $\odot$ is the Hadamard product. The vector $u_{i}$ is
proportional to the one-hot label $y_{i}$, while the vector $v_{i}$ is
associated with $(1-y_{i})$.

SOP utilizes the Cross-Entropy (CE) loss function for updating the parameters
$\theta$ and $u$, and employs the Mean Squared Error (MSE) loss function for
updating the parameter $v$. Prior to calculating the CE loss, SOP applies
normalization to the sum of $f(x_{i};\theta)$ and $s_{i}$. On the other hand,
when learning the parameter $v$ with the MSE loss, SOP applies the one-hot
transformation to $f(x_{i};\theta)$, which promotes a more cautious learning
approach.

##  4 Coordinated Sparse Recovery

###  4.1 Analysis of the Learning Pace

As expressed in Eq. 2, the transition of the predicted probability of noisy
labels can be decoupled into two components: adding the label noise
probability and subtracting the predictions of the true labels. As a result,
the SOP approach necessitates simultaneous learning of the network parameters
$\theta$, and noise vectors $u$ and $v$ within the same network. If the
learning pace of $u$ or $v$ is not synchronized with that of $\theta$, it may
introduce substantial bias into the transition process, causing the inclusion
of inaccurate information in the gradient update of $\theta$ and resulting in
erroneous gradient leakage. This leakage subsequently impedes the proper
learning of $\theta$, leading to more pronounced error accumulation.

To analyze the learning paces of these three groups of parameters, we computed
the distribution of their gradient updates throughout the training. The
proportion of gradient updates indicated as $g_{t}$ at the $t$-th epoch, can
be expressed as:

$g_{t}=\frac{|z_{t}|}{\sum^{T}_{t=1}|z_{t}|},t=1,2,\cdots T.$ (3)

A higher value of $g_{t}$ implies a stronger engagement in the learning tasks
during the $t$-th epoch. $|z_{t}|$ signifies the cumulative sum of absolute
gradients for the parameter set. $T$ represents the total number of epochs.
Figure 1a$\sim$h exhibits the distribution of gradient update for different
parameter groups when trained on CIFAR-10/100 with varying ratios of instant-
dependant noisy labels using the hyperparameters given in reference[45].

Figure 1:  Empirical analysis of the incoordination phenomenon of $v$-lag
learning and its impact on test error on CIFAR-10/100 Datasets with different
label noises: variation trends of parameter gradient with respect to training
time (a)$\sim$(h) and correlation between parameter learning incoordination
and model generalization (i)$\sim$(l).

It is apparent that, regardless of different datasets and label noise ratios,
a consistent trend emerges: the learning of $\theta$ and $u$ show substantial
overlap, while the learning of $v$ noticeably falls behind $\theta$ and $u$.
This suggests that during the learning processes of $\theta$ and $u$, there
exists a significant duration where $v$ remains unchanged at a negligible
level. Within this timeframe, if $f(x_{i},\theta)$ exhibits considerable
confidence for a specific term in the vector $(1-y_{i})$, it will generate an
erroneous gradient updating in the opposite direction. Consequently, the
model’s confidence in the correct category will then decrease, causing
fluctuations back and forth.

The one-hot operation plays a key role in causing the delayed learning of $v$.
In situations where mislabeled samples are present and the model is uncertain
about selecting the correct class, applying the one-hot transformation will
result in all predictions for classes other than the labeled class $y_{i}$
being zero. This leads to a zero gradient for updating $v$ at a specific time
point. Only when the model becomes confident enough in selecting the correct
class, it will generate gradients that can update $v$, along with gradient
errors that update $\theta$. In fact, the one-hot operation is crucial for the
SOP algorithm, and removing this transformation would result in a significant
increase in the model’s test error. The delay in learning $v$ can also be
attributed to using the flatter Mean Squared Error (MSE) loss function instead
of the Cross-Entropy (CE) loss update. Appendix B offers a detailed derivation
and explanation as to why the adoption of the CE loss function is not
feasible, despite the fact that utilizing MSE further worsens the challenge of
coordinating the noise parameter $v$ and the model parameter $\theta$.

###  4.2 Influence to Generalization

This paper examines the extent of incoordination in learning paces by
employing the Chi-Square[48] distance to quantify the disparity in gradient
distributions. Given that $g^{\theta}$ and $g^{v}$ denote the gradient
distributions of two distinct parameter groups, and $g^{\theta}_{t}$ and
$g^{v}_{t}$ denote the proportion in the $t$-th epoch respectively, we define
the ratio of incoordination as follows:

$I=\frac{\sum^{T}_{t=1}|g^{\theta}_{t}-g^{v}_{t}|}{\sum^{T}_{t=1}g^{\theta}_{t}+g^{v}_{t}}.$
(4)

When the distributions of the gradients are perfectly identical, the value of
$I$ is $0$. Conversely, when there is no overlap at all between the
distributions of the gradients, the value of $I$ is $1$.

This research conducts approximate simulation experiments to analyze the
association between the incoordination ratio $I$ and the test error $e$. The
experiments are carried out on the CIFAR-10/100 datasets, with varying ratios
of instance-dependent noise as well as real-world noise. The experiment setup
involves training the network initially and storing the values and gradients
of $v$ for each epoch. Subsequently, 30 epochs of $v$ are repeatedly shifted
backward, where after each shift, the update of $v$ remains fixed, and the
values of $v$ are utilized for training the model parameters $\theta$ and
sparse parameters $u$. Then, the testing errors of the model are computed
after each shift.

Figure 1i$\sim$l showcases partial experimental findings on the CIFAR-100
dataset, incorporating various ratios of instance-dependent noise and real-
world noise. The $x$-axis denotes the level of incoordination $I$, while the
$y$-axis represents the test error $e$. It can be observed that the change
curve exhibits a monotonic increase, indicating that as the value of $I$
augments, the test error rises. Accordingly, the experiment affirms that the
incoordination ratio is in fact linked to the test error of the model, thereby
compromising its generalization capability.

###  4.3 Collaboration Matrix

In order to tackle the problem of learning incoordination during over-
parameterized training, this paper introduces an innovative approach, namely
the collaboration matrix, which serves to dynamically counteract error
leakage. The label estimation process is modeled as the multiplication of
predictions by the collaboration matrix, followed by the addition of a sparse
noise vector. More specifically, we redefine the optimization problem for
empirical risk minimization in the following manner:

$\min\limits_{\theta,\\{u_{i},v_{i}\\}_{i=1}^{N}}\frac{1}{N}\sum_{i=1}^{N}l(f(x_{i};\theta)\times
M+s_{i},y_{i})$ (5)

, where $M\in R^{K\times K}$ is the collaboration matrix. The matrix $M$ is
learnable. It is initialized with the identity matrix and updated using
stochastic gradient descent along with other parameters. The CE loss is
employed to update the $M$ matrix, and the specific update formula is as
follows:

$M\longleftarrow M-\tau_{m}\cdot\frac{\partial L_{CE}}{\partial
M},i=1,\dots,N,$ (6)

where $\tau_{m}$ represents the learning rate of the collaboration matrix $M$.
We employ loss calculation and gradient update methods analogous to SOP. The
distinction lies in the inclusion of matrix $M$ in the calculation and update
processes. Moreover, to mitigate issues of gradient explosion and gradient
disappearance during training, we additionally apply matrix normalization.
This is demonstrated in the formula provided below.

$\bar{M}=(M-min(M))/(\gamma-min(M)),$ (7)

in which $\bar{M}$ is the normalized matrix and $\gamma$ is a learnable scale
parameter.

Within the training of this research, $M$ primarily focuses on updating the
parameters located on the main diagonal of the matrix, whereas the other off-
diagonal parameters tend to approach $0$. Figure 2b provides a visual
representation of $M$ after 50 epochs under 40% IDN noise on CIFAR-100. From
this figure, we can easily observe this phenomenon. Our speculation is that
this behavior may stem from the distinct update stages of the decoupled
orthogonal vectors $u$ and $v$, which are responsible for noise prediction and
noise correction respectively in the context of SOP. This disparity also
allows $M$ to play a collaborative coordinating role in $f$. Specifically,
during the period of delayed learning for $v$, multiplying $f$ by $M$
effectively narrows the gap for mislabeled samples, thereby reducing the
contamination of incorrect gradients.

We also analyze the fluctuation of $M$. Figure 2a illustrates the average
value on the main diagonal of $M$ obtained through training on CIFAR-100 with
various ratios of instance-dependent noise. Notably, these curves demonstrate
a consistent trend followed by a rapid decline starting from an initialization
value of $1$. Particularly, during the phase of delayed learning for vector
$v$, the mean value of $M$ experiences a sharp decrease. As $v$ begins to
update, the mean value of $M$ undergoes a gradual and slow decline until it
reaches a stable state.

Figure 2: Visualization and change analysis of M matrix values: (a) average
value variations in M matrix over time; (b) visualization of M value at 50
epoch time: mainly concentrated on the diagonal

We adopt a unified $M$ for all samples in our learning process. Each element
on the diagonal of $M$ is approximated to represent the adjusted weight for
its corresponding category. Figure 3 illustrates the learning process for
mislabeled samples, highlighting the difference between using $M$ and not
using it. When $M$ is not applied, the predicted probability $f$ for the
correct category cannot be effectively corrected by $v$, resulting in a
significant loss that should ideally be avoided. However, by multiplying $M$
with the predicted probability, this erroneous loss is greatly reduced. This
helps in mitigating gradient updates in the wrong direction and slowing down
the learning of $\theta$. It is worth noting that $M$ does not significantly
affect the loss computation for correctly labeled samples. In cases where
$s_{i}$ tends close to 0, only the product of $M$ and $f$ needs to be
computed, which undergoes minimal change after normalization.

Figure 3: Comparison on an incorrectly annotated sample (b) with and (a)
without the use of a collaboration matrix. Without the collaboration matrix,
there would be an error loss of 0.69, whereas, after the utilization of the
collaboration matrix, the error loss decreases significantly to 0.18.

###  4.4 Confidence Weight

To further improve the robustness for $\theta$, we introduce a technique
called confidence weighting that adjusts the learning proportion for different
groups of parameters. Represented as $\omega$, the weight is a floating-point
value ranging from 0 to 1. We multiply $\omega$ with the gradient of the model
parameters and $(1-\omega)$ with the gradient of the noise parameters. Our
assumption is that when the predicted probability closely matches the given
label, the confidence weight tends to be $1.0$. This indicates that the sample
is likely to be accurately labeled and justifies updating the model
parameters. Conversely, when the confidence weight approaches 0, it suggests
that the sample is likely mislabeled and requires an update to the noise
parameters.

To ensure more accurate predicted probabilities, we employ a momentum-based
exponential moving average (EMA) on the results of $\Delta T$ time-series
predictions. Suppose $t$ is the current epoch, $q^{\prime}_{t}$ denotes the
model prediction at $t$, and $q_{t}$ is the temporarily ensembled prediction
and adopts the following formula for the recursive operation.

$q_{t}=\beta\cdot q_{t-1}+(1-\beta_{t})\cdot q^{\prime}_{t},$ (8)
$\beta_{t}=\frac{(\beta_{init}-1)\cdot t}{T}+1,$ (9)

where $\beta_{t}$ represents the momentum, which is dynamically computed to
prioritize the predictions of the model at a nearer time and ensure that the
smoothed outcome reflects this consideration more prominently. $\beta_{init}$
is a hyperparameter, and $T$ is the overall number of training epochs.

Let $p_{i}=q_{\Delta T}$ denote the predicted probability after applying
temporal EMA smoothing over $\Delta T$ windows. The confidence weight can be
computed using the following formula.

$\begin{split}\omega_{i}=1-Normalize_{[0,1]}(L_{CE}(p_{i},y_{i})),\\\
\Omega=\\{\omega_{i};i\in(1,\dots,N)\\},\ \ \ \ \ \ \ \ \ \ \end{split}$ (10)

where $L_{CE}$ is the cross-entropy loss, $y_{i}$ is the given label, and
$\Omega$ represents the set of all training samples.

###  4.5 Overall Algorithm of CSR

Figure 4: The architecture of CSR.

We put forward a resilient training approach for overparameterized networks,
referred to as Coordinated Sparse Recovery (CSR), that revolves around the
collaboration matrix and the confidence weighting. The configuration of CSR is
depicted in Figure 4 and the overall algorithm is shown in Figure 5.

Figure 5: Overall Algorithm of CSR.

In each batch, we update the network parameters $\theta$, the scale factor
$\gamma$, and the collaboration matrix $M$. In each epoch, we update $u_{i}$
and $v_{i}$ for each sample. The emphasis of the updates is determined by
calculating the confidence weight $\omega_{i}$ of each sample.

##  5 Extension of CSR

CSR can serve as a standalone powerful learning method and also as a component
in combination with semi-supervised techniques to devise a novel sample-
selection-based learning framework.

###  5.1 Small-$u$ Sample Selection

In sample selection, it is commonly employed to utilize a Gaussian mixture
model to estimate the distribution of sample losses, followed by picking out
clean samples with small losses using a given threshold, denoted as $\sigma$.
Let $GMM()$ represent the process of fitting a Gaussian mixture model, and
$L_{CE}(f(x_{i}),y_{i})$ the cross-entropy loss between the prediction $f$ and
its corresponding label $y_{i}$ for a sample $x_{i}$. The set of selected
clean samples can be computed using the following formula.

$\begin{split}S_{loss}=\\{x_{i},y_{i}|GMM(L_{CE}(f(x_{i}),y_{i}))>\sigma\\},\end{split}$
(11)

The precision of sample selection by this strategy may not be optimal, as
difficult-to-learn samples also have big losses. On the other hand, during CSR
learning, we utilize $u$ to model the probability of label noise, hence the
distribution of $u$ can serve as a criterion for sample selection. To address
this, we propose the small-$u$ strategy. Specifically, we use $GMM$ to fit the
distribution of $u$, and then apply a threshold cutoff on the results to
filter out clean samples, which is defined as follows:

$\begin{split}S_{u}=\\{x_{i},y_{i}|GMM(u_{i})>\sigma\\},\end{split}$ (12)

Figure 6: Comparison of precision and recall for small-loss sample selection
and small-$u$ sample selection.

Figure 6 demonstrates the selection performance on the CIFAR-100N dataset with
real-world label noise, employing the small-loss strategy and small-$u$
strategy respectively. As depicted in Fig. 6a, the precision of samples chosen
via GMM, fitted to the $u$-distribution, consistently improves with longer
training time, eventually reaching a precision of more than 90%. Nevertheless,
the recall of the small-$u$ selection exhibits a perpetual decline, as shown
in Fig. 6b, unlike the stable recall observed in the small-loss sample
selection strategy.

###  5.2 Joint Sample Selection

This paper presents a methodology for conducting joint sample selection. By
utilizing set operations, the complete sample dataset is ultimately divided
into three well-defined subsets, namely the clean set, hard set, and noisy
set. This approach aims to simultaneously address both accuracy and diversity
concerns in sample selection. Given that $S_{loss}$ is derived from Eq. 11 and
$S_{u}$ is derived from Eq. 12, the computation methods for the clean set
$S_{clean}$, the hard set $S_{hard}$, and the noisy set $S_{noisy}$ can be
defined as follows in equations 13 to 15.

$S_{clean}=\\{S_{loss}\cap S_{u}\\},$ (13) $S_{hard}=\\{S_{loss}\cup
S_{u}-S_{loss}\cap S_{u}\\},$ (14) $S_{noisy}=\\{\Omega-S_{loss}\cup
S_{u}\\}.$ (15)

We employ different processing strategies for subsets of clean, hard, and
noisy samples. Given the high reliability of sample labels in $S_{clean}$, we
utilize supervised learning with both strong and weak augmentation, as well as
Mixup augmentation. This approach aims to enhance the model’s performance by
leveraging trustworthy labels to their fullest extent. On the other hand, the
samples in $S_{hard}$ exhibit better diversity but less reliability compared
to $S_{clean}$. To address this, we only apply weak and Mixup augmentation
with supervised learning. This strategy aims to achieve a more robust decision
boundary by locally smoothing. As for the samples in $S_{noisy}$, their labels
are considered unreliable, and thus, we treat them as unlabeled samples. To
handle this, we generate pseudo-labels for the noisy samples using a dynamic
threshold. We then include the samples with pseudo-labels into the clean set
for the subsequent round of training. This process helps us progressively
improve the model’s performance on the noisy samples by gradually
incorporating them into the training process.

###  5.3 Framework of CSR+

The CSR+ framework is designed by integrating multiple techniques including
joint sample selection, consistent regularization with strong and weak
augmentation, Mixup augmentation, and label correction techniques. Figure 7
depicts the architecture of CSR+, featuring a multi-task learning network with
three losses. In order to provide a comprehensive understanding, this paper
will briefly outline the technical aspects of consistent regularization,
Mixup, and label correction.

Figure 7: CSR+ architecture.

####  5.3.1 Consistency Regularization

The consistency regularization used in the CSR+ framework is identical to the
consistency regularization employed in the previous work [22],[49]. It
involves generating a new sample by applying random augmentations to the
original sample, and then calculating the CE loss with its corresponding
label. Let $x_{i}^{s}$ represent the augmented sample of $x_{i}$, and let
$y_{i}$ denote its corresponding label. The consistency regularization can be
calculated using the following formula:

$L_{cr}=L_{CE}(f(x_{i}^{s},\theta),y_{i}).$ (16)

####  5.3.2 Mixup

The CSR+ framework employs Mixup augmentation technique to generate new
training instances using weakly augmented samples. Let $x_{i}^{w}$ and
$x_{j}^{w}$ be two weakly augmented samples, and $y_{i}$ and $y_{j}$ be their
corresponding labels. The newly generated training instance
$\\{x_{i}^{m},y_{i}^{m}\\}$ can be expressed as follows:

$\begin{split}x_{i}^{m}=\delta\cdot x_{i}^{w}+(1-\delta)\cdot x_{j}^{w},\\\
y_{i}^{m}=\delta\cdot y_{i}+(1-\delta)\cdot y_{j},\ \ \end{split}$ (17)

where $\delta\in[0,1]$, $\delta\sim Beta(\alpha,\alpha)$ is a hyperparameter
representing the mixing ratio. Then, the Mixup loss is defined as follows:

$L_{Mix}=L_{CE}(f(x_{i}^{m},\theta),y_{i}^{m}).$ (18)

Until now, the overall loss can be calculated as follows:

$L_{overall}=L_{CSR}+\alpha(L_{CR}+L_{Mix}),$ (19)

where $L_{CSR}$ is the loss function for the training of CSR and $\alpha$ is a
weight factor hyperparameter set to increase linearly from 0 to 1.

####  5.3.3 Noisy Label Correction

The CSR+ framework employs the dynamic threshold sample correction method
proposed by DISC[41] to rectify the samples within the set of noisy samples.
This correction method consists of three steps.

Step1: Employ a weighted averaging technique to combine the predicted
probabilities derived from the strong and weak augmented predictions.

$p_{ws}=\epsilon\cdot p_{w}+(1-\epsilon)\cdot p_{s}.$ (20)

The variables mentioned in the provided formula include the following:
$p_{w}$, which represents weakly augmented prediction; $p_{s}$, which
represents strongly augmented prediction; $p_{ws}$, which represents the newly
obtained prediction probability; and $\epsilon$, indicating the weight
coefficient set to 0.5.

Step 2: Calculate the dynamic threshold value.

$\phi(t)=max(\phi_{ws}(t)+0.5,0.99),$ (21)

where 0.5 is a small positive offset value, 0.99 is an upper threshold and
$\phi_{ws}(t)$ is the historical smoothing of $p_{ws}$

Step 3: Employ the dynamic threshold to filter the obtained new predicted
probabilities, replacing the original labels with the highest predicted
probability that surpasses the threshold. This revised probability will then
serve as the pseudo-label.

$\tilde{y_{i}}=argmax(p_{ws})|min(p_{ws})>\phi(t).$ (22)

##  6  EXPERIMENT AND ANALYSIS

To evaluate the effectiveness of CSR and CSR+, we conducted extensive
experiments on synthetic datasets containing instance-dependent label noise at
different noise ratios sourced from CIFAR-10/100. Additionally, we evaluated
these models on five real-world datasets with label noise. While our primary
objective is to enhance the model’s robustness to instance-dependent label
noise, it is noteworthy that our approach also exhibits significant efficacy
in handling simpler types of label noise.

In this chapter, we begin by comparing CSR with other principle approaches
(without using any additional training techniques) to evaluate the
generalizability of CSR. Then, we discuss the contributions of the two core
modules of CSR and the learning rate settings of the collaboration matrix in
the training process. Following that, we compare CSR+ with other state-of-the-
art methods to provide a comprehensive performance assessment. Subsequently,
we conduct ablation studies to investigate the contributions of joint sample
selection and other training techniques employed in CSR+. Lastly, we analyze
the complexities of CSR and CSR+, including time complexity and space
complexity.

###  6.1 Datasets

The experiments make use of the following datasets:

$\bullet$ CIFAR-10/100 [50] are datasets comprising of 32 × 32 × 3 color
images. Both of them consist of a total of 50,000 training images and 10,000
test images.

$\bullet$ CIFAR-10N/100N datasets [51] are real-world noisy datasets that have
been manually annotated. Each image in CIFAR-10N has three annotated labels.
The label denoted as Random $i$ represents the $i$-th submitted label for each
image. The label denoted as Aggregate is obtained through aggregating the
three noisy labels by majority voting. The dataset with the highest noise rate
is referred to as Worst. We have selected the Worst label from CIFAR-10N,
which has a noise rate of 40.21%. On the other hand, each image in CIFAR-100N
is associated with a coarse label and a fine label, both of which are provided
by human annotators. We have selected the noisy fine labels from CIFAR-100N,
which exhibit an overall noise level of 40.20%.

$\bullet$ Clothing1M[52] is a real-world dataset that comprises of 1 million
training images extracted from online shopping sites. It consists of 14
categories. The estimated noise level in this dataset is 38.5%.

$\bullet$ WebVision [53] and ILSVRC12 are datasets derived from the ImageNet
Large Scale Visual Recognition Challenge. The ImageNet dataset contains 1,000
classes and 2.4 million images. In our experiments, we follow the Mini setting
and use the data from the initial 50 classes for training. The estimated noise
level in these datasets is 20%. To evaluate our model, we utilize the
validation sets of WebVision and ILSVRC12, which consist of 50,000 and 2,500
images respectively.

###  6.2 Synthetic Noise

To synthesize instance-dependent noise, we employ the method mentioned earlier
[16], which depends on part-specific label noise to generate noise that is
specific to each instance. Our experiments were carried out using three
different noise rates, namely 20%, 40%, and 60%.

###  6.3 Experimental setup

Experiments were conducted using a GeForce RTX3090 GPU in the PyTorch v1.10
environment. The SGD optimizer was utilized, and the learning rate settings
were following those of SOP. The experimental setups for each dataset were as
follows:

For the CIFAR-10/100 and CIFAR-10/100N datasets, the PreActResNet-18 network
was employed as the backbone. Learning rate for this network was set to 0.02
for CSR and 0.03 for CSR+, with a weight decay of 0.0005 and a batch size of
128. The network was trained for 300 epochs. For the CIFAR-10 and CIFAR-10N
datasets, the learning rate for the matrix was set to 0.000001, and a warm-up
period of 10 epochs was applied. For the CIFAR-100 and CIFAR-100N datasets,
The learning rate for the matrix was set to 0.001, and a warm-up period of 20
epochs was applied.

In the case of the Clothing1M dataset, a pre-trained ResNet50 served as the
backbone network. The learning rate for this network was set to 0.002, with a
weight decay of 0.001 and a batch size of 32. The learning rate for the matrix
was set to 0.001, with a warm-up period of 1 epoch. The overall training
process lasted for 10 epochs.

For the WebVision and ILSVRC12 datasets, the Inception ResNetv2 was used as
the backbone network. The learning rate for this network was set to 0.01, with
a weight decay of 0.01 and a batch size of 32. The learning rate for the
matrix was set to 0.001, with a warm-up period of 5 epochs. The overall
training process lasted for 80 epochs.

###  6.4 Comparison of CSR With Other Principle Methods

To begin with, we conduct a comparative analysis of CSR with several other
principle methods, namely CE [54] loss, Forward method [35] that integrates
noise estimation and loss correction, Co-teaching [21] which adopts the dual
network sample selection technique, GCE [9] loss which combines CE loss and
MAE loss[55], ELR [25] with the early learning regularizer, and SOP [45] using
sparse over-parameterized training.

TABLE I: Comparison of CSR With Other Principle Methods Dataset | CIFAR-10 |  10N | CIFAR-100 |  100N
---|---|---|---|---
Noise type | Instance dependent |  Worst | Instance dependent |  Noise fine
Noise rate | 0.2 | 0.4 | 0.6 |  about 0.4 | 0.2 | 0.4 | 0.6 |  about 0.4
CE | 83.93 | 67.64 | 43.83 | 77.69 | 57.35 | 43.17 | 24.42 | 55.50
Forward | 87.22 | 79.37 | 66.56 | 79.79 | 58.19 | 42.80 | 27.91 | 57.01
Co-teaching | 88.87 | 73.00 | 62.51 | 83.83 | 43.30 | 23.21 | 12.58 | 60.37
GCE | 89.80 | 78.95 | 60.76 | 80.66 | 58.01 | 45.69 | 35.08 | 56.73
ELR | 92.35 | 88.74 | 67.97 | 83.58 | 72.66 | 62.04 | 44.38 | 59.81
SOP | 93.91 | 91.95 | 72.12 | 86.76 | 74.09 | 66.52 | 50.20 | 61.09
CSR | 94.08 | 92.56 | 74.69 | 87.37 | 75.93 | 71.75 | 58.98 | 64.62

Table I presents a comprehensive analysis of the experimental results obtained
by comparing CSR with the other approaches. To facilitate analysis and enhance
understanding, these results have been graphically represented in a curve
graph, as shown in Figure 8. The CSR method demonstrates superior performance
compared to other methods. As the noise ratio increases, the test accuracy of
the comparative methods declines significantly. For instance, when tested on
the CIFAR-10 IDN dataset, the classical Co-teaching method experienced a drop
in test accuracy from 88.87 to 62.51. Similarly, the SOP method showed a
decrease in test accuracy from 74.09 to 50.20 on CIFAR-100 IDN dataset. In
contrast, the CSR method exhibited the smallest decline in test accuracy on
both datasets, indicating its robustness. Further analysis consistently ranks
the SOP method as the second-best in all metrics, closely following the CSR
method. However, as the noise ratio increases, the performance gap between CSR
and SOP widens. Moreover, the CSR method demonstrates more notable performance
improvement on the CIFAR-100 dataset compared to CIFAR-10. When tested on
CIFAR-10 with a 60% IDN, CSR outperforms SOP with a 2.57 percentage point
increase. Conversely, when tested on CIFAR-100 with a 60% IDN, this
improvement escalates to 8.78. These empirical findings undeniably establish
the advantages of the CSR method, especially in more challenging scenarios.

Figure 8: Visualization of comparative results of principle methods on a) CIFAR-10 and b) CIFAR-100 with varying IDN ratios. TABLE II: Ablation Experiments of Various Modules of CSR Dataset | CIFAR-10 |  10N | CIFAR-100 |  100N
---|---|---|---|---
Noise type | Instance dependent |  Worst | Instance dependent |  Noise fine
Noise rate | 0.2 | 0.4 | 0.6 |  about 0.4 | 0.2 | 0.4 | 0.6 |  about 0.4
CSR | 94.08 | 92.56 | 74.69 | 87.37 | 75.93 | 71.75 | 58.98 | 64.62
CSR w/o cw | 94.04 | 92.07 | 73.24 | 86.78 | 75.57 | 71.57 | 56.15 | 63.09
CSR w/o $M$ | 93.95 | 92.27 | 75.21 | 87.12 | 74.38 | 68.62 | 53.94 | 63.24
SOP | 93.91 | 91.95 | 72.12 | 86.76 | 74.09 | 66.52 | 50.20 | 61.09

####  6.4.1 Ablation Study of CSR

CSR adds the collaboration matrix and confidence weight to SOP. In order to
further evaluate the respective contributions of these two modules, we
conducted an ablation study on CSR. Table II presents the results of this
study. It becomes evident that the removal of either module significantly
impairs performance. However, the elimination of the collaboration matrix has
a more detrimental effect on performance compared to the removal of the
confidence weights. This observation further aligns with our previous
findings. The collaboration matrix effectively diminishes generalization
errors by mitigating erroneous gradient leakage, while the confidence weight
enables the network to learn and generate more reliable predictions. When used
together, the test accuracy can be maximized.

We believe that either the collaboration matrix or the confidence weight holds
the potential to effectively mitigate overfitting to noisy labels in the
model. To validate our hypothesis, we have introduced a novel evaluation
metric termed Noise Fitting Rate (NFR). This metric provides a quantitative
measure of the samples in which the predictions accurately align with
incorrect labels, relative to the overall number of mislabeled samples.
Assuming $\Omega_{e}$ is the collection of mislabeled samples, its calculation
is defined by the following formula:

$NFR=|\ \\{i|f_{i}(x_{i},\theta)=y_{i},i\in\Omega_{e}\\}\ |\ /|\Omega_{e}|.$
(23)

Within this context, the variable $i$ represents the $i$-th individual sample
within the sample collection $\Omega_{e}$. Meanwhile, $f_{i}(x_{i},\theta)$
denotes the predicted result generated by the model for the $i$-th sample, and
$y_{i}$ signifies the noisy label associated with that particular sample.
Moreover, the notation $|\Omega_{e}|$ indicates the size of elements within
the set $\Omega_{e}$.

To visually illustrate the fluctuation of this Noise Fitting Rate (NFR) over
time, we have constructed a graph, as depicted in Figure 9. This figure
observes and demonstrates the level of noise fitting by methods in the
ablation study under 40% instance-dependent noise on CIFAR-100. It is apparent
that the baseline method SOP, represented by the red curve, struggles to
withstand the detrimental impact of label noise in the absence of the
collaboration matrix and the confidence weight. As the training proceeds, the
network tends to increasingly overfit the noisy data. Conversely, the green
and yellow curves depict learning processes incorporating the collaboration
matrix and the confidence weight, respectively. These approaches successfully
mitigate the negative effects of label noise during the early stage of
training and enable robust learning throughout the subsequent training
process. The proposed CSR, represented by the blue curve, demonstrates the
highest level of robustness against label noise.

Figure 9: Curves of Noise Fitting Rate over time on the CIFAR-100 40% IDN
dataset.

####  6.4.2 Choice of $\tau_{m}$

In order to make the collaboration matrix effective, we experimentally
determine the best value of the learning rate $\tau_{m}$. We conducted
experiments where we updated matrix $M$ using various learning rates under
CIFAR-100 40% IDN, observing the impact on test performance. We discovered
that a learning rate of 0.001 yielded the best performance for $M$. We utilize
similar experiments to determine the learning rate of the collaboration matrix
on other datasets. Specifically, for CIFAR-10, we ultimately settled on a
learning rate of 0.000001 for the collaboration matrix. On other real
datasets, the learning rate $\tau_{m}$ is set to 0.001.

TABLE III: Ablation Experiment of $\tau_{m}$ Under 40% IDN on CIFAR-100 lr for $M$ | 0.1 | 0.01 | 0.001 | 0.0001 | 0.00001
---|---|---|---|---|---
test acc | 42.35 | 69.88 | 71.75 | 69.33 | 68.79

###  6.5 Comparison of CSR+ with SOTA

We compare CSR+ with the state-of-the-art methods in the field of noisy label
learning to evaluate the effectiveness of the proposed robust learning
framework. To guarantee fairness, the comparative methods were tested using
the most advanced configuration, referred to as the Plus version. The results
of these comparisons are presented in Table IV in chronological order, with
certain outcomes derived from publicly accessible data mentioned in the
literature [41].

TABLE IV: Comparison of CSR+ With SOTA on the CIFAR-10/100 and CIFAR-10N/100N Datasets Dataset | CIFAR-10 | 10N | CIFAR-100 |  100N
---|---|---|---|---
Noise type | Instance dependent | Worst | Instance dependent |  Noise fine
Noise rate | 0.2 | 0.4 | 0.6 |  about 0.4 | 0.2 | 0.4 | 0.6 |  about 0.4
Co-teaching+ |  89.80 ±0.28 |  73.38 ±1.39 |  59.22 ±6.34 |  83.83 ±1.28 |  41.71 ±0.78 |  24.45 ±0.71 |  12.58 ±0.58 |  60.37 ±0.27
ELR+ |  95.09 ±0.21 |  94.06 ±0.32 |  91.22 ±0.51 |  90.91 ±0.28 |  77.09 ±0.20 |  74.50 ±0.32 |  66.99 ±0.79 |  67.73 ±0.41
DivideMix |  93.24 ±0.12 |  94.37 ±0.22 |  92.19 ±0.63 |  92.56 ±0.42 |  79.04 ±0.21 |  76.08 ±0.35 |  46.72 ±1.32 |  71.13 ±0.48
UNICON |  95.92 ±0.01 |  95.72 ±0.02 |  95.34 ±0.04 |  94.48 ±0.01 |  78.22 ±0.09 |  77.71 ±0.10 |  71.07 ±0.32 |  70.84 ±0.11
SOP+ |  96.19 ±0.12 |  95.51 ±0.16 |  84.86 ±0.19 |  93.24 ±0.21 |  78.26 ±0.12 |  72.79 ±0.24 |  62.25 ±0.31 |  67.81 ±0.23
CC |  93.68 ±0.12 |  94.97 ±0.09 |  94.95 ±0.11 | - |  79.61 ±0.19 |  76.58 ±0.25 |  59.40 ±0.46 | -
ProMix |  97.12 ±0.04 |  96.56 ±0.12 |  94.15 ±0.23 |  95.61 ±0.15 |  81.84 ±0.14 |  79.50 ±0.23 |  72.88 ±0.41 |  72.68 ±0.21
DISC |  96.48 ±0.04 |  95.94 ±0.04 |  95.05 ±0.05 |  90.37 ±0.13 |  80.12 ±0.13 |  78.44 ±0.19 |  69.57 ±0.14 |  69.51 ±0.21
CSR+ |  97.25 ±0.04 |  96.88 ±0.07 |  94.57 ±0.16 |  95.66 ±0.14 |  82.44 ±0.09 |  81.10 ±0.13 |  75.85 ±0.23 |  72.74 ±0.14

The mean and standard deviation of three independently run experiments are
presented. It can be seen that CSR+ outperforms most of the compared state-of-
the-art methods on CIFAR-10/100 with various ratios of noise as well as
CIFAR-10N/100N, demonstrating good generalization. CSR+ not only performs
exceptionally well on instances with low noise ratios, but it also maintains
relatively high test accuracy on datasets with 60% instance-dependent noise.
In comparison, Co-teaching+ and SOP+ show significant performance decline as
the noise ratio increases. The DISC method exhibits unstable performance,
although it achieves the highest test accuracy on the CIFAR-10 60% IDN
dataset, even surpassing our CSR+ method by 0.48 percentage points. However,
it experiences noticeable declines on the CIFAR-100 60% IDN dataset and two
real-world noise datasets. Its performance on real datasets is even inferior
to DivideMix, proposed in 2020. The CC method does not provide results on
real-world datasets, but it also performs poorly on CIFAR-100, indicating that
the assumptions of such methods may not hold in more complex scenarios.
Overall, CSR+ has the best comprehensive performance. It is effective not only
on data with more categories, but also on data with higher ratios of IDN noise
and real-world noise.

Table V presents the experimental results of CSR+ on real-world datasets such
as Clothing1M, WebVision, and ILSVRC12. Similarly, in terms of classification
accuracy, our method outperforms all other state-of-the-art methods in terms
of top-1 test accuracy on all three datasets. This further demonstrates the
wide applicability and effectiveness of CSR+ in handling label noise in real-
world scenarios. However, in terms of top-5 accuracy, CSR+ slightly lags
behind CC’s method. This could be attributed to CC’s training approach that
maximizes representation consistency and minimizes classification consistency,
whereas CSR+ only utilizes a consistency regularization term.

TABLE V: Comparison of CSR+ With SOTA on the Clothing1M, Webvision and ILSVRC12 Datasets Method | Clothing1M | WebVision | ILSVRC12
---|---|---|---
| top1 | top5 | top1 | top5
Co-teaching | 69.21 | 63.58 | 85.20 | 61.48 | 84.70
ELR+ | 74.81 | 77.78 | 91.68 | 70.29 | 89.76
DivideMix | 74.76 | 77.32 | 91.64 | 75.20 | 90.84
CC | 74.54 | 79.36 | 93.64 | 76.08 | 93.86
DISC | 74.79 | 80.28 | 92.28 | 77.44 | 92.28
CSR+ | 74.92 | 80.47 | 92.51 | 77.46 | 95.28

####  6.5.1 Ablation Study on Sample Selection Strategies

Due to the use of various latest training techniques in comparative methods,
it is hard to evaluate the effectiveness of our sample selection strategy
based purely on the final leaderboard. In order to further assess the proposed
joint sample selection strategy, we conducted ablation experiments on
different sample selection strategies within the unified CSR+ framework. The
representative sample selection strategies we employed included BARE’s
selection based on Batch statistics, GMM+loss selection in DivideMix, mutual
information selection in Co-teaching, JS divergence selection in UNICON.

The outcomes of the experiments are presented in Table VI. The utilization of
GMM+loss proves to be extremely efficient when the noise ratio is low.
However, as the noise ratio increases, this selection strategy runs the risk
of misclassifying erroneous samples as correct, therefore amplifying the
potential for confirmation bias. In scenarios characterized by a multitude of
categories, the accurate computation of class centers becomes challenging,
thus impeding the preciseness of sample selection in methods relying on latent
space distance metrics. Conversely, mutual information-based methods address
this issue by training two independent networks, thereby mitigating the
accumulation of errors to a certain extent. In comparison, the joint sample
selection strategy presented in this paper is more effective and yields
superior outcomes in comparison to methods based on mutual information
selection.

TABLE VI: Comparison of Joint Sample Selection With the Other Sample Selection Methods Dataset | CIFAR-100 |  CIFAR-100N
---|---|---
Noise type | Instance dependent |  Noise fine
Noise rate | 0.2 | 0.4 | 0.6 |  about 0.4
batch statistics | 73.78 | 72.06 | 67.95 | 63.37
GMM+loss | 82.34 | 78.63 | 73.08 | 71.25
mutual information | 81.74 | 79.56 | 74.00 | 71.14
JS divergence | 82.21 | 78.83 | 71.64 | 70.41
joint sample selection | 82.44 | 81.10 | 75.85 | 72.74

####  6.5.2 Ablation Study of CSR+

TABLE VII: Ablation Experiments on CSR+ Dataset | CIFAR-100 |  CIFAR-100N
---|---|---
Noise type | Instance dependent |  Noise fine
Noise rate | 0.2 | 0.4 | 0.6 |  about 0.4
CSR | 75.93 | 71.75 | 58.98 | 64.62
\+ CR | 77.26 | 73.34 | 65.44 | 68.18
\+ Mixup | 81.17 | 78.26 | 66.50 | 70.16
\+ Label Correction | 81.94 | 79.60 | 71.45 | 71.45
\+ Co-train | 82.44 | 81.10 | 75.85 | 72.74

We conducted ablation experiments on the modules of CSR+ to observe the
contributions of the training techniques employed. We used CSR as the baseline
method and gradually introduced noise-robust techniques such as consistency
regularization, Mixup, Label Correction, and Co-train. The experiments were
performed on two types of noise, namely real noise on CIFAR-100N and instance
noises on CIFAR-100. The results are presented in Table VII. It can be
observed that each noise-robust technique improved the testing accuracy of
CSR, leading to the best overall performance. The joint sample selection
strategy required the combination of consistency regularization and Mixup to
be effective. From the ablation experiments, it is evident that these two
modules provided the most significant improvement in testing accuracy. Label
correction and Co-train techniques also increased the accuracy, albeit to a
lesser extent. The ablation experiments suggest that the CSR+ framework is not
merely a stack of training tricks, but a robust framework for learning under
noisy labels built around CSR learning and joint sample selection.

###  6.6 Complexity Analysis

During our experiments on CIFAR-100 with 40% IDN labels, we conducted
complexity analysis, which included time complexity and space complexity. We
tabulated the changes in time consumption and memory usage for each module
added after the SOP. The second column represents the time taken to train one
epoch, the third column denotes the total memory consumption during training,
the fourth column indicates the number of parameters, and the fifth column
displays the testing accuracy. It can be observed that, compared to SOP, CSR
does not significantly increase training time and memory consumption. However,
it led to a significant improvement in performance, with the testing accuracy
increasing from 66.52 to 71.75, a gain of 5.23 percentage points. CSR+
utilizes more parameters and consumes more time and memory during training.
Nonetheless, it can further enhance the performance by almost 10 percentage
points.

TABLE VIII: Complexity Analysis Method | Time(s) | Memory(MB) | Parameter(MB) | Accuracy(%)
---|---|---|---|---
SOP | 15.3 | 835.9 | 18.1000 | 66.52
CSR w/ cw | 15.4 | 836.0 | 18.1009 | 71.57
CSR w/ M | 15.8 | 856.0 | 18.1000 | 68.62
CSR | 16.2 | 856.0 | 18.1009 | 71.75
CSR+ | 105.2 | 2433.4 | 36.2018 | 81.10

##  7 Conclusion

Label noise is widely present in datasets and can significantly harm the
generalization performance of models. Particularly in over-parameterized
networks, it can also lead to a greater demand for more data, longer training
duration, and larger network parameters. To address this issue, this paper
proposes a coordinated sparse recovery (CSR) method based on sparse modeling
of noise and over-parameterized training. This method resolves the inherent
and widespread learning incoordination between model prediction and noise
recovery. Furthermore, we extend this method to build a complete and robust
learning framework. Extensive comparative and ablation experiments are
conducted, which demonstrate that the proposed CSR method indeed reduces
erroneous gradient updates, makes better use of early learning, and achieves
improved generalization performance. The joint sample selection we construct
is a highly effective strategy for sample selection, which, when combined with
common semi-supervised training techniques, can achieve the best test accuracy
on datasets with a higher proportion of instance-level noise. In the future,
our research will focus on expanding the proposed method to address the
openset problem and exploring better integration with pre-trained large-scale
models.

## Acknowledgments

I would like to express my gratitude to Mr.JAHANZAIB YAQOOB for proofreading
this manuscript, as well as to the laboratory staff for their valuable
discussions and support in this endeavor.

## References

  * [1] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning (still) requires rethinking generalization,” _Communications of the ACM_ , vol. 64, no. 3, pp. 107–115, 2021.
  * [2] D. Arpit, S. Jastrzębski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio _et al._ , “A closer look at memorization in deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 233–242.
  * [3] L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei, “Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels,” in _International conference on machine learning_. PMLR, 2018, pp. 2304–2313.
  * [4] H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee, “Learning from noisy labels with deep neural networks: A survey,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [5] N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari, “Learning with noisy labels,” _Advances in neural information processing systems_ , vol. 26, 2013.
  * [6] N. Manwani and P. Sastry, “Noise tolerance under risk minimization,” _IEEE transactions on cybernetics_ , vol. 43, no. 3, pp. 1146–1151, 2013\.
  * [7] B. Van Rooyen, A. Menon, and R. C. Williamson, “Learning with symmetric label noise: The importance of being unhinged,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [8] A. Ghosh, H. Kumar, and P. S. Sastry, “Robust loss functions under label noise for deep neural networks,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 31, no. 1, 2017.
  * [9] Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [10] Y. Wang, X. Ma, Z. Chen, Y. Luo, J. Yi, and J. Bailey, “Symmetric cross entropy for robust learning with noisy labels,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 322–330.
  * [11] X. Ma, H. Huang, Y. Wang, S. Romano, S. Erfani, and J. Bailey, “Normalized loss functions for deep learning with noisy labels,” in _International conference on machine learning_. PMLR, 2020, pp. 6543–6553.
  * [12] Y. Wang, X. Sun, and Y. Fu, “Scalable penalized regression for noise detection in learning with noisy labels,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 346–355.
  * [13] C. M. Bishop, “Training with noise is equivalent to tikhonov regularization,” _Neural Computation_ , vol. 7, no. 1, pp. 108–116, 1995.
  * [14] L. Gu and T. Kanade, “A generative shape regularization model for robust face alignment,” in _Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part I 10_. Springer, 2008, pp. 413–426.
  * [15] J. Hoffman, D. A. Roberts, and S. Yaida, “Robust learning with jacobian regularization,” _arXiv preprint arXiv:1908.02729_ , 2019.
  * [16] X. Xia, T. Liu, B. Han, N. Wang, M. Gong, H. Liu, G. Niu, D. Tao, and M. Sugiyama, “Part-dependent label noise: Towards instance-dependent label noise,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 7597–7610, 2020.
  * [17] S. Yang, E. Yang, B. Han, Y. Liu, M. Xu, G. Niu, and T. Liu, “Estimating instance-dependent label-noise transition matrix using dnns,” 2021.
  * [18] E. Arazo, D. Ortego, P. Albert, N. O’Connor, and K. McGuinness, “Unsupervised label noise modeling and loss correction,” in _International conference on machine learning_. PMLR, 2019, pp. 312–321.
  * [19] H. Cheng, Z. Zhu, X. Li, Y. Gong, X. Sun, and Y. Liu, “Learning with instance-dependent label noise: A sample sieve approach,” _arXiv preprint arXiv:2010.02347_ , 2020.
  * [20] E. Englesson and H. Azizpour, “Generalized jensen-shannon divergence loss for learning with noisy labels,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 30 284–30 297, 2021.
  * [21] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” _Advances in neural information processing systems_ , vol. 31, 2018\.
  * [22] J. Li, R. Socher, and S. C. Hoi, “Dividemix: Learning with noisy labels as semi-supervised learning,” _arXiv preprint arXiv:2002.07394_ , 2020.
  * [23] N. Karim, M. N. Rizve, N. Rahnavard, A. Mian, and M. Shah, “Unicon: Combating label noise through uniform selection and contrastive learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9676–9686.
  * [24] J. Li, C. Xiong, and S. C. Hoi, “Learning from noisy data with robust representation learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9485–9494.
  * [25] S. Liu, J. Niles-Weed, N. Razavian, and C. Fernandez-Granda, “Early-learning regularization prevents memorization of noisy labels,” _Advances in neural information processing systems_ , vol. 33, pp. 20 331–20 342, 2020.
  * [26] D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa, “Joint optimization framework for learning with noisy labels,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5552–5560.
  * [27] X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama, “How does disagreement help generalization against label corruption?” in _International Conference on Machine Learning_. PMLR, 2019, pp. 7164–7173.
  * [28] Y. Zhang, S. Zheng, P. Wu, M. Goswami, and C. Chen, “Learning with feature-dependent label noise: A progressive approach,” _arXiv preprint arXiv:2103.07756_ , 2021.
  * [29] Z. Zhu, F. Liu, G. Chrysos, and V. Cevher, “Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization),” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 36 094–36 107, 2022.
  * [30] H. Wei, L. Tao, R. Xie, and B. An, “Open-set label noise can improve robustness against inherent label noise,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 7978–7992, 2021.
  * [31] X. Wang, Y. Hua, E. Kodirov, and N. M. Robertson, “Imae for noise-robust learning: Mean absolute error does not treat examples equally and gradient magnitude’s variance matters,” _arXiv preprint arXiv:1903.12141_ , 2019.
  * [32] Y. Xu, P. Cao, Y. Kong, and Y. Wang, “L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [33] J. Goldberger and E. Ben-Reuven, “Training deep neural-networks using a noise adaptation layer,” in _International conference on learning representations_ , 2016.
  * [34] X. Xia, T. Liu, N. Wang, B. Han, C. Gong, G. Niu, and M. Sugiyama, “Are anchor points really indispensable in label-noise learning?” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [35] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, “Making deep neural networks robust to label noise: A loss correction approach,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1944–1952.
  * [36] R. Tanno, A. Saeedi, S. Sankaranarayanan, D. C. Alexander, and N. Silberman, “Learning from noisy labels by regularized estimation of annotator confusion,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 11 244–11 253.
  * [37] E. Malach and S. Shalev-Shwartz, “Decoupling" when to update" from" how to update",” _Advances in neural information processing systems_ , vol. 30, 2017\.
  * [38] J. Liu, R. Li, and C. Sun, “Co-correcting: noise-tolerant medical image classification via mutual label correction,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 12, pp. 3580–3592, 2021.
  * [39] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [40] D. Patel and P. Sastry, “Adaptive sample selection for robust learning under label noise,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 3932–3942.
  * [41] Y. Li, H. Han, S. Shan, and X. Chen, “Disc: Learning from noisy labels via dynamic instance-specific selection and correction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 24 070–24 079.
  * [42] P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever, “Deep double descent: Where bigger models and more data hurt,” _Journal of Statistical Mechanics: Theory and Experiment_ , vol. 2021, no. 12, p. 124003, 2021\.
  * [43] K. Wen, J. Teng, and J. Zhang, “Benign overfitting in classification: Provably counter label noise with larger models,” in _The Eleventh International Conference on Learning Representations_ , 2022.
  * [44] D. A. Nguyen, R. Levie, J. Lienen, G. Kutyniok, and E. Hüllermeier, “Memorization-dilation: Modeling neural collapse under noise,” _arXiv preprint arXiv:2206.05530_ , 2022.
  * [45] S. Liu, Z. Zhu, Q. Qu, and C. You, “Robust training under label noise by over-parameterization,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 14 153–14 172.
  * [46] C. You, Z. Zhu, Q. Qu, and Y. Ma, “Robust recovery via implicit bias of discrepant learning rates for double over-parameterization,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 17 733–17 744, 2020\.
  * [47] F. A. Wani, M. S. Bucarelli, and F. Silvestri, “Combining distance to class centroids and outlier discounting for improved learning with noisy labels,” _arXiv preprint arXiv:2303.09470_ , 2023.
  * [48] X. C. F. Z. Y. L. Wei Yang, Luhui Xu, “Chi-squared distance metric learning for histogram data,” _Mathematical Problems in Engineering_ , 2015.
  * [49] H. Wang, R. Xiao, Y. Dong, L. Feng, and J. Zhao, “Promix: combating label noise via maximizing clean sample utility,” _arXiv preprint arXiv:2207.10276_ , 2022.
  * [50] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [51] J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu, “Learning with noisy labels revisited: A study using real-world human annotations,” _arXiv preprint arXiv:2110.12088_ , 2021.
  * [52] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang, “Learning from massive noisy labeled data for image classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 2691–2699.
  * [53] W. Li, L. Wang, W. Li, E. Agustsson, and L. Van Gool, “Webvision database: Visual learning and understanding from web data,” _arXiv preprint arXiv:1708.02862_ , 2017.
  * [54] P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein, “A tutorial on the cross-entropy method,” _Annals of operations research_ , vol. 134, pp. 19–67, 2005.
  * [55] C. J. Willmott and K. Matsuura, “Advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance,” _Climate research_ , vol. 30, no. 1, pp. 79–82, 2005.

|  Yukun Yang received the BE degree from Qingdao University, China. He is
currently working toward the MS degree with the Department of Information
Science and Technology, Beijing University of Chemical Technology. His
research interests include deep learning, robust learning and computer vision.
---|---
|  Ruirui Li received the Ph.D. degree in computer science and technology from
Tsinghua University, Beijing, China, in 2014. She joined as a Postdoctoral
with the College of Information Science and Technology, Beijing University of
Chemical Technology. She was appointed as an Associate Professor with the same
institute in 2020. Her research interests include machine learning, pattern
analysis, and computer vision.
---|---
|  Naihao Wang received the BE degree from Beijing University of Chemical
Technology, China. He is currently working toward the MS degree with the
Department of Information Science and Technology, Beijing University of
Chemical Technology. His research interests include deep learning, robust
learning and computer vision.
---|---
|  Haixin Yang received the BE degree from Peking University, China. He is
currently working toward the Ph.D. degree with the School of Mathematical
Sciences, Peking University. His research interests include deep learning,
robust learning and computer vision.
---|---