# Correlated Insulator Collapse due to Quantum Avalanche via In-Gap Ladder
States

Jong E. Han<EMAIL_ADDRESS>Department of Physics, State University of
New York at Buffalo, Buffalo, New York 14260, USA  Camille Aron  Laboratoire
de Physique de l’École Normale Supérieure, ENS, Université PSL, CNRS, Sorbonne
Université, Université Paris Cité, F-75005 Paris, France  Jae-Ho Han  Center
for Theoretical Physics of Complex Systems, Institute for Basic Science(IBS),
Daejeon 34126, South Korea  Ki-Seok Kim  Department of Physics, POSTECH,
Pohang, Gyeongbuk 37673, South Korea  Ishiaka Mansaray  Department of Physics,
State University of New York at Buffalo, Buffalo, New York 14260, USA  Michael
Randle  Department of Electrical Engineering, State University of New York at
Buffalo, Buffalo, New York 14260, USA  Jonathan P. Bird  Department of
Physics, State University of New York at Buffalo, Buffalo, New York 14260, USA
Department of Electrical Engineering, State University of New York at Buffalo,
Buffalo, New York 14260, USA

###### Abstract

We propose a microscopic mechanism to resolve the long-standing puzzle of the
insulator-to-metal transition in correlated electronic systems, most notably
charge-density-wave (CDW) materials and Mott insulators, driven far-from-
equilibrium by a DC electric field. By introducing a generic model of
electrons coupled to an inelastic medium of phonons, we demonstrate that an
electron avalanche can occur in the bulk limit of such insulators at
arbitrarily small electric field. The quantum avalanche arises by the
generation of a ladder of in-gap states, created by a multi-phonon emission
process. Hot-phonons in the avalanche trigger a premature and partial collapse
of the correlated gap. The details of the phonon spectrum dictate two-stage
versus single-stage mechanisms which we associate with CDW and Mott resistive
transitions, respectively. The electron and phonon temperatures, as well as
the temperature dependence of the threshold fields, point to the quantum
nature of this nonequilibrium phase transition.

In spite of recent progress in our understanding of the nonequilibrium many-
body state of matter, one of the long-standing problems that has remained
unresolved concerns the microscopic mechanism behind the insulator-to-metal
transition (IMT) of strongly-correlated electronic systems, driven by a DC
electric field. For more than five decades, the community has fiercely debated
the origin of the dielectric breakdown in charge-density-wave (CDW) systems
bardeenPT ; bardeen1989 ; ong1979 ; grunerRMP ; maki1977 ; thornehistory , for
which the reported threshold electric fields are orders of magnitude smaller
than theoretical estimates based on the Landau-Zener zener  mechanism. In the
late seventies, this problem led to the formulation of the classical theory of
depinning, in which the CDW order parameter is understood as being pinned by
the presence of disorder and can be abruptly unpinned under the action of a
static electric field fukuyama1978 ; lee1979 ; fisherCDW . More recently, a
similar mismatch between theory and experiments has also been found in studies
of transition-metal compounds such as Mott insulators janod ; stoliar ; guiot
; Ridley ; sblee . In spite of the emerging potential of these materials for
applications such as non-volatile neuromorphic computing del_valle , the lack
of understanding of the microscopic origins of their resistive transitions is
a bottleneck to the development of such technologies.

From the early days of CDW research, the theoretical paradigm for the
resistive transition in correlated systems has been the Landau-Zener mechanism
ong1979 ; zener . This model predicts switching fields on the order of

$E_{\rm LZ}\sim\frac{\Delta^{2}}{e\hbar v_{F}},$ (1)

with the (zero-field) insulating gap $\Delta$ and the Fermi-velocity $v_{F}$.
Over the years, various theoretical attempts have been made to improve the
description of the resistive switching transition in a many-body context.
Existing theories include the explicit time-evolution of the Hubbard chain
oka2003 , the multi-band Hubbard model mazza , disorder-driven sugimoto  and
nonequilibrium han2018  phase-transitions, spatial inhomogeneity nanolett ,
and Coulomb blockade of multi-solitons miller2000 ; maki1977 . These various
microscopic models have been unable to settle the debate as to the importance
of thermal zimmers  versus electronic giorgianni2019 ; jager  processes to the
resistive transition, and it has been speculated that an important ingredient
must be missing.

Despite the differences in CDW and Mott phenomenology, it is often hard to
disentangle one mechanism from the other zhuCDW . The colossal mismatch of the
switching fields in the field-driven transition in both systems motivates us
to look for a generic mechanism. We propose that the IMT in CDW and Mott
insulators can be explained in terms of the role of inelastic many-body
states. We show how these phonon-emitting transitions can lead to the
generation of in-gap states, resulting in a quantum avalanche mechanism and a
destruction of the correlated gap, at much smaller electric field scales than
previously predicted.

Fig. 1: Avalanche Mechanism a, (left) Electronic band located above the Fermi
level by $\Delta$ at zero electric field, $E=0$. Hybridization to a fermionic
bath provides a finite decay rate $\Gamma$. (right) The local density of
states (LDOS) for electrons that occupy only bath states (red shading). b,
(left) Electronic ladder-like states after successive phonon-emission events
(blue) form quantum pathways for avalanche at $E>E_{T}$. (right) Once the in-
gap ladder states are formed, electrons are pumped through them into the
conduction band by the electric-field. A finite density of electronic
excitation $n_{\rm ex}(\omega)$ (red shading) now populates the in-gap LDOS.

For a conceptual understanding of the avalanche mechanism, we start by
considering a one-dimensional conduction band of mass $m$ separated by a gap
$\Delta$ from the Fermi energy, see Fig. 1 a. The electrons are subject to
three processes. First, they are accelerated by a DC electric field $E$.
Second, they can emit optical phonons of energy $\hbar\omega_{\rm ph}$ by
means of an electron-phonon scattering process controlled by the coupling
parameter $g_{\rm ep}$. Finally, dephasing introduces a finite lifetime
$\hbar/\Gamma$ to the electrons, setting an upper limit on the time scale over
which they can form the multi-phonon state. The dephasing may arise from
electron tunneling to a substrate, coupling to higher electronic states, and
other many-body mechanisms separate from the electron-phonon interaction.
Hereafter, we adopt a unit system in which $\hbar=1$, the electric charge
$e=1$, the Boltzmann constant $k_{B}=1$, and the lattice constant $a=1$.

Let us now give a back-of-the-envelope criterion for an avalanche that can be
triggered by a combination of the three aforementioned electronic processes.
We note that, in an insulator subject to an external bias, there will exist
dilute, yet nonetheless finite, charge excitations. Importantly, electron-
phonon scattering processes will create a ladder of intermediate replica bands
that are equally spaced by $\omega_{\rm ph}$. The minimal number of successive
phonon-scattering events to bridge the gap is $N_{\rm
ladder}\sim\Delta/\omega_{\rm ph}$, see Fig. 1b. The timescale it takes to
accelerate an electron in a dispersive band, to gain the energy of a single
phonon, can be estimated as $\tau\sim(m\,\omega_{\rm ph})^{1/2}/E$. The
electron-phonon scattering rate is proportional to the dimensionless electron-
phonon mass-renormalization factor $\lambda\equiv(g_{\rm ep}/\omega_{\rm
ph})^{2}\nu_{0}$, where the typical density of states $\nu_{0}\sim m$. The
number of phonons that are emitted during the electronic lifetime $1/\Gamma$
is then $N_{\rm ep}\sim{\lambda}/{\Gamma\tau}$. Consequently, the criterion
for the avalanche becomes $N_{\rm ep}=N_{\rm ladder}$, yielding the following
electric field required to trigger the avalanche

$E_{\rm av}\sim\frac{\hbar\Gamma\Delta}{eg_{\rm
ep}^{2}}\left[\frac{(\hbar\omega_{\rm ph})^{3}}{m}\right]^{1/2}.$ (2)

This outcome is highly non-trivial in that the dissipative electronic decay
rate $\Gamma$ plays a crucial role in the onset of this nonequilibrium quantum
phase transition. The switching field is very different in nature from that of
Eq. (1), and we note that the avalanche via inelastic scattering is also
distinct from the conventional mechanism that has been discussed for high-
field transport in semiconductors wolff1954 . Most importantly, as we shall
see below, the avalanche fields predicted by Eq. (2) are much smaller than
those expected from Eq. (1).

Results

Quantum avalanche in a rigid-band model
To demonstrate the existence of the proposed avalanche mechanism, we
investigate a model of a one-band tight-binding chain under a DC electric-
field $E$, in the Coulomb gauge liprl2015 ; ligraphene

$H^{\rm 1D}_{0,\rm
el}=\sum_{i}\left[-t(d^{\dagger}_{i+1}d_{i}+d^{\dagger}_{i}d_{i+1})+(2t+\Delta-
Ex_{i})d^{\dagger}_{i}d_{i}\right],$ (3)

where $d^{\dagger}_{i}/d_{i}$ is the creation/annihilation operator for an
electron at site $i$, for which the site position $x_{i}=ia$. The electrons
are locally coupled to phonons, which are modeled by a collection of harmonic
oscillators given by the Hamiltonian

$H_{0,\rm ph}=\frac{1}{2}\sum_{k}(p_{k}^{2}+\omega_{k}^{2}\varphi_{k}^{2}),$
(4)

with $\varphi_{k}$ the amplitude, $p_{k}$ the momentum, $k$ the continuum
index, and $\omega_{k}$ the frequency of the phonon. In this section we
consider the case of optical phonons, with $\omega_{k}=\omega_{\rm ph}$,
deferring a discussion of acoustic phonons until later. The on-site electron-
phonon coupling is given by

$H_{\rm ep}=g_{\rm ep}\sum_{i}\varphi_{i}d^{\dagger}_{i}d_{i},$ (5)

with the coupling constant $g_{\rm ep}$.

We use a Schwinger-Keldysh formulation of the dynamical mean-field theory
(DMFT georges_DMFT1996 ; aoki ; liprl2015 ) which bypasses the transient
dynamics and directly yields the homogeneous nonequilibrium steady-state of
the many-body dynamics aron2012prl ; prb2013b . The fermion baths enter the
computation of the electronic Green’s function via local retarded and lesser
self-energies at site $i$ as

$\Sigma^{R}_{0,i}(\omega)=-i\Gamma,\quad\Sigma^{<}_{0,i}(\omega)=2i\Gamma
f_{0}(\omega+Ex_{i}),$ (6)

while the Ohmic baths weiss  enter the phonon Green’s function via local self-
energies as

$\Pi^{R}_{0,i}(\omega)=-2i\tau_{P}^{-1}\omega,\quad\Pi^{<}_{0,i}(\omega)=-4i\tau_{P}^{-1}\omega
n_{0}(\omega).$ (7)

In the above expressions, $f_{0}(\omega)=(e^{\omega/T}+1)^{-1}$ and
$n_{0}(\omega)=(e^{\omega/T}-1)^{-1}$ are the Fermi-Dirac and Bose-Einstein
distributions at the bath temperature $T$, respectively, and $\tau_{P}$
khurgin2007  is the phonon decay rate. We compute the second-order self-energy
to electrons and phonons on the same footing. We refer the reader to the
Methods Section and to the Supplementary Information for further details on
the electron-phonon calculations.

Fig. 2: Avalanche controlled by electron life-time a, Electron excitation per
site $n_{\rm ex}$ versus $E$ at several values of the fermion decay rate
$\Gamma=1,2,4,6\times 10^{-3}$ (from left to right). The inset shows the same
data on a semi-logarithmic scale. The avalanche field $E_{\rm av}$ is
proportional to $\Gamma$. The curves with filled (open) symbols are computed
without (with) the non-equilibrium self-energy of the phonons. b, Post-
avalanche spectra of electrons and phonons. The electron spectral function
$A_{\rm el}(\omega)$ shows the band edge, and the distribution function
$f_{\rm el}(\omega)$ reveals the in-gap state occupation with multi-phonon
edges (marked by red arrows) separated by $\omega_{\rm ph}$. The phonon
distribution function $n_{\rm ph}(\omega)$ is equilibrium-like.

In the calculations that follow, we choose the model parameters $\Delta=1$,
$t=1$, $g_{\rm ep}=0.25$, $\omega_{\rm ph}=0.3$, $\tau_{P}^{-1}=0.001$, and
$T=0.001$ in units of eV. We use the lattice constant $a=5$ Å to compute the
electric-field. Fig. 2a shows the density of electronic excitations $n_{\rm
ex}$ per site ($0\leq n_{\rm ex}\leq 1$) when ramping up the electric field
starting from the insulator state. Filled (empty) symbols denote results
obtained by discarding (including) the nonequilibrium self-energy of the
phonons. At the onset of the avalanche at the threshold field $E_{\rm av}$,
$n_{\rm ex}$ increases abruptly in a similar fashion as in critical phenomena.
Notably, $E_{\rm av}$ is found to be roughly proportional to $\Gamma$,
consistent with the back-of-the-envelope estimate made in Eq. (2).
Furthermore, the pre-avalanche excitations are inversely proportional to
$E_{\rm av}$ [see inset to (a)], which demonstrates that the avalanche is not
initiated by thermal excitation but is of quantum origin. Remarkably, the
onset of the avalanche is not affected by hot-phonon effects since they are
not activated in the insulating regime, while the incoherence of phonons
impedes the avalanche resulting in a linear increase of $n_{\rm ex}$ after the
avalanche.

The nontrivial singular fixed-point behavior at $\Gamma=0$ eckstein2010  (see
Fig. 2a) should be distinguished from phonon-assisted tunneling kleinman1965 ;
vdovin2016 , which is commonly manifested in resonant tunneling in
heterostructures. With the electron occupation quickly reaching beyond 0.1
after the avalanche, we are in a metallic bulk limit through a phase
transition, instead of in the perturbative tunneling regime. The avalanche
shown here only emerges after a fully self-consistent solution is reached
after hundreds of iterations, unlike what is expected from a low-order
perturbative theory kleinman1965 .

Figure 2b elucidates nature of the nonequilibrium state after the avalanche.
The energy distribution $f_{\rm el}(\omega)$ reveals a sizable nonequilibrium
occupation of the in-gap states around energies at multiples of $\omega_{\rm
ph}$ away from the band edge. This confirms the involvement of multi-phonon
emission in the avalanche mechanism. Moreover, the strong deviation of $f_{\rm
el}(\omega)$ from a Fermi-Dirac distribution points to the non-thermal nature
of the avalanche. In contrast, the phonon distribution $n_{\rm ph}(\omega)$
appears mostly thermal.

Insulator-to-metal transition induced by avalanche

We now turn our attention to the implications of the avalanche for the
resistive transition in CDW and Mott insulators. The strong charge
fluctuations initiated by the avalanche are expected to generate phonon
excitations, which then profoundly perturb the inter-orbital mixing and
destabilize the charge gap. To address this point, we extend our model to a
two-band correlated insulator where the gap $\Delta$ between the bands is
generated by electronic interactions. (See Fig. 1)a.) Specifically, we
consider the following Hamiltonian

$\displaystyle H^{\rm 2D}_{\rm el}$ $\displaystyle=$
$\displaystyle-t\sum_{\langle
ij\rangle}\sum_{\alpha=1,2}(-1)^{\alpha}(d^{\dagger}_{\alpha i}d_{\alpha
j}+d^{\dagger}_{\alpha j}d_{\alpha i})$ (8)
$\displaystyle+\sum_{i}\sum_{\alpha}[(-1)^{\alpha}(2t-\mu)-Ex_{i}]d^{\dagger}_{\alpha
i}d_{\alpha i}$
$\displaystyle+\sum_{i}\left[\xi(d^{\dagger}_{1i}d_{2i}+d^{\dagger}_{2i}d_{1i})+\frac{\xi^{2}}{2U}\right].$

Here, $\alpha$ is the band index and $i$ is the site index on a square lattice
with $\langle ij\rangle$ denoting nearest neighbors. The last term in Eq. (8)
corresponds to the mean-field decoupling of an inter-orbital Hubbard-like
interaction, where the order parameter $\xi$ is determined from the condition

$\xi=U\langle d^{\dagger}_{1i}d_{2i}+d^{\dagger}_{2i}d_{1i}\rangle.$ (9)

The electron-phonon coupling is given by $H_{\rm ep}=g_{\rm
ep}\sum_{i,\alpha}\varphi_{\alpha i}d^{\dagger}_{\alpha i}d_{\alpha i},$ with
the independent phonons coupling to each orbital. The fermion/phonon baths are
set up as described previously with the Fermi energy in the center of the gap.
Note that this generic model allows us to address both the CDW Peierls
transition and Mott physics on an equal footing.

Fig. 3:  CDW-like two-stage IMT, a, (Top) Band-crossing at the Fermi level
(dotted line), (Bottom) Charge gap formation due to inter-orbital interaction,
b, Density of states at $E=0$: well-defined charge gap $\Delta_{0}=1$. c,
After the first threshold, $E>E_{\rm T}$: pseudo-gap formation. The inset is a
close-up (solid red shading: the frequency-resolved electronic occupation of
the in-gap states). d, After the second threshold, $E>E_{\rm IMT}$: metallic
density of states with a fully collapsed gap. e, $I-V$ characteristics for
various temperatures $T$ proceeds in two stages: a CDW-like continuous
transition to a metal at $E=E_{\rm T}$ followed by a discontinuous current
jump at $E=E_{\rm IMT}$. f, Effective temperatures of the electrons and
phonons. At $E=E_{\rm T}$, the electrons go through a transition without
heating up the phonons, while at $E_{\rm IMT}$ the two temperatures
equilibrate. g, Phase diagram in the $E$–$T$ plane. The $E_{\rm T}$ and
$E_{\rm IMT}$ curves delimit a pseudo-gap metal region. (See the main text for
the parameters.)

The spectrum of the phonons has important consequences for the IMT. Let us
first discuss the case of optical phonons with energy $\omega_{\rm ph}$ and
later turn to the case of acoustic phonons. With increasing electric field,
the insulating system undergoes a two-stage transition to a metal as
manifested by the spectral function in Fig. 3b-d. Here, we define the
electron-phonon coupling $g_{\rm ep}$ such that, given $U=2$, the initial gap
$\Delta_{0}$ is tuned to $1.0$ at $E=0$ and at the lowest temperature. The
strength of this coupling corresponds to a moderate (dimensionless) mass-
renormalization factor of $\lambda\approx(g_{\rm ep}/\omega_{\rm
ph})^{2}\nu_{0}=0.32$ with the phonon energy $\omega_{\rm ph}=0.3$ eV and the
2D density of states $\nu_{0}\approx(8t)^{-1}$. (The damping is set to
$\Gamma=\tau_{P}^{-1}=0.001$ eV, and the chemical potential to $\mu=t$.) In
the low-field limit, the spectrum features a well-defined charge gap. As $E$
increases beyond the first threshold field $E_{T}$, in-gap states develop via
the avalanche mechanism presented above. This results in a pseudo-gap phase
where the system becomes conducting while the charge gap mostly remains
intact. The energy-resolved electron occupation $n_{\rm
ex}(\omega)=(2\pi)^{-1}{\rm Im}G^{<}(x_{i}=0,\omega)$ (the red shaded area
shown in the inset to panel (b)) indicates that the electric current is
carried mainly by the above-gap states while the in-gap states provide the
pathway for this population inversion. The pseudo-gap regime persists until a
full IMT occurs at $E=E_{\rm IMT}$, when the gap $\Delta$ collapses to zero in
a strongly discontinuous transition. Unambiguous observation of a pseudogap in
the NbSe3 system was recently made nicholson2017 ; nicholson2020 , consistent
with our model in the small-field limit.

The $I-V$ relation in Fig. 3e shows evidence of a two-stage IMT. First, the
system continuously becomes conducting at the threshold field $E=E_{T}$.
Later, at the higher threshold $E_{\rm IMT}$, the current rises
discontinuously. The avalanche behavior discussed in Fig. 2 is responsible for
the threshold behavior at $E_{T}$. This behavior is similar to that in CDW
systems, in which it has been widely attributed to the depinning transition
fukuyama1978 ; lee1979 . Here, we propose an alternative mechanism of electron
avalanche via inelastic scattering that does not require any disorder or
reduced dimensionality. With coupling to optical phonons, the avalanche is not
sufficiently disruptive to the gap, and the intermediate gapped state is
instead sustained over a wide range of electric field ($E_{T}<E<E_{\rm IMT}$).
It is remarkable that $E_{T}$ is around two orders of magnitude smaller than
the switching field expected for zero electron-phonon coupling strength
$E_{\rm IMT}(\lambda=0,\Delta_{0}=1)\approx 1.6$ MeV/cm.

The non-thermal nature of the avalanche transition is illustrated by the form
of the effective electron and phonon temperature, as shown in Fig. 3f. (See
Methods for the definition of effective temperatures.) As the electric field
is increased beyond $E=E_{T}$, the electrons heat immediately while the
phonons stay cold. This clearly demonstrates that heat exchange does not
trigger the avalanche. On the other hand, the full IMT at $E_{\rm IMT}$ is
initiated once the electron and phonon temperatures equilibrate, suggesting
that this second transition can be described in terms of a thermal mechanism.
Note how the rapid rise of electron temperature near $E_{T}$ is suppressed at
higher $T$, where only the transition near $E_{\rm IMT}$ remains visible.

The phase diagram defined by the two switching fields ($E_{T}$ and $E_{\rm
IMT}$) is plotted in Fig. 3g. The most remarkable observation here is that
$E_{T}$ remains constant near zero temperature, but increases at higher
temperatures until it merges with $E_{\rm IMT}$. This may seem counter-
intuitive, since the gap is expected to decrease with increasing $T$. The gap,
however, remains nearly constant until close to a critical temperature
$T_{c}$, meaning that a thermal argument is not applicable. We find that, as
suggested by the hot-phonon effects apparent in Fig. 2, thermal decoherence
may be detrimental to the avalanche, meaning that a stronger electric field is
required to induce an avalanche. This is further evidence that the threshold
behavior is of quantum-mechanical origin.

The existence of bias-driven multiple-stage transitions in CDW systems has
been intensely debated bardeenPT ; zaitsev-zotov2004 ; zaitsev-zotov2001 ;
itkis1990 ; fleming1980 , with the threshold behavior observed at low fields
having been shown to exhibit a similar temperature dependence to that of
$E_{T}(T)$ which we discuss. Recent works on semiconducting CDW TiS3
randleACSnano ; randleAPL2022 , for example, have shown an increase of
$E_{T}(T)$ with increasing temperature. While realistic bands with multiple
order parameters would produce more complex thresholds, our work demonstrates
that a single order-parameter theory still supports the existence of a multi-
stage IMT.

Fig. 4: Mott-like abrupt IMT a, $I-V$ relation in the limit of weak coupling
to acoustic phonons, displaying a single-stage discontinuous IMT. b, $E_{\rm
IMT}$ versus $T$ diagram for (a). c, $I-V$ relation in the strong-coupling
limit showing non-monotonic dependence of $E_{\rm IMT}$ on $T$. d, Non-
monotonic $E_{\rm IMT}$ versus $T$. Increasing $E_{\rm IMT}$ with $T$ at low
$T$ demonstrates the non-thermal nature of the IMT in the strong-coupling
limit.

Finally, let us discuss the case of electron coupling to acoustic phonons,
with a continuous spectrum of the form $\omega_{k}\propto k$ and cutoff Debye
energy chosen as $\omega_{D}=0.6$ eV. Compared to optical phonons, the
influence of acoustic phonons on the nonequilibrium dynamics is more dramatic.
We discuss this situation for both the weak and strong limits of electron-
phonon coupling, as shown in Fig. 4. In the weak-coupling limit
($\lambda=0.113$), corresponding to panels (a) and (b), the IMT is dominated
by a strongly discontinuous collapse of the gap. While the signature of the
avalanche is still present as a precursor to the IMT, the range of the
avalanche region is much narrower than that found for coupling to optical
phonons. The avalanche current is also very small so that its effect is
insignificant. $E_{\rm IMT}$ varies monotonically with temperature and the
$E-T$ phase diagram is conventional. In the strong-coupling limit
($\lambda=0.602$), in contrast, $E_{\rm IMT}$ is strongly nonmonotonic,
increasing with $T$ in the low-$T$ limit. As is especially apparent from panel
(c), the intermediate field region between $E_{T}$ and $E_{\rm IMT}$ has
diminished dramatically so that the IMT appears as a fully single-stage
transition, as often observed in correlated transition-metal systems janod ;
stoliar ; zimmers ; guenon ; htkim . Basically, in this limit, the strong low-
energy excitations of acoustic modes cause the IMT to bypass the CDW-like
pseudo-gap state.

Strongly discontinuous IMTs have been observed in various transition-metal
oxides and chalcogenides, in which $E_{\rm IMT}(T)$ vanishes with concave
curvature as temperature is increased towards $T_{c}$ tlwu ; maki1986 ,
consistent with both limits of Fig. 4. At low temperatures, not much is known
about the behavior of $E_{\rm IMT}(T)$ and further experimental efforts are
called for.

Conclusions

We have demonstrated a quantum-avalanche mechanism as a generic scenario for
the nonequilibrium IMT in gapped systems and anticipate rich physics by
controlling hot-phonon effects. The existence of the in-gap ladder states may
be directly verified experimentally by using transient electrical pulses of
varying duration to control the amount of hot-phonon generation. While we have
discussed this mechanism with electron-phonon coupling for the sake of
robustness, the avalanche may also arise via coupling to more general bosonic
excitations, such as the Goldstone modes associated with the order parameter
responsible for opening of the charge gap. Such a scenario may provide a more
direct path for the destruction of the correlated gap. Here we have presented
a minimal framework for the quantum avalanche that is fundamentally different
from the Landau-Zener mechanism. Study of the avalanche with spatial
inhomogeneity, and its interplay with disorder, is left for future research.

Methods

Calculation of Self-Energy and Effective Temperature:  The many-body
calculations in this work are based on the nonequilibrium Green’s function
technique approximated by the DMFT scheme aoki ; liprl2015 ; ligraphene , in
which we limit the self-energy to be diagonal in site and orbital indices.
Full details on the nonequilibrium DMFT are given in the Supplementary
Information. The electron and phonon self-energies, expressed respectively as

$\displaystyle\Sigma^{\lessgtr}_{{\rm ep},\alpha}({\bf r},\omega)$
$\displaystyle=$ $\displaystyle ig_{\rm ep}^{2}\int\frac{{\rm
d}\omega^{\prime}}{2\pi}{\cal G}^{\lessgtr}_{\alpha\alpha}({\bf
r},\omega-\omega^{\prime})D^{\lessgtr}_{\alpha}(\omega^{\prime}),$ (10)
$\displaystyle\Pi^{\lessgtr}_{{\rm ep},\alpha}(\omega)$ $\displaystyle=$
$\displaystyle-2ig_{\rm ep}^{2}\int\frac{{\rm
d}\omega^{\prime}}{2\pi}G^{\lessgtr}_{\alpha\alpha}({\bf
r},\omega+\omega^{\prime})G^{\gtrless}_{\alpha\alpha}({\bf
r},\omega^{\prime}),$ (11)

are iterated to convergence. (The factor 2 in the phonon self-energy accounts
for spin degeneracy.) Once we achieve full convergence, we compute the
distribution functions as

$f_{\rm el}(\omega)=-\frac{1}{2}\frac{\sum_{\alpha}{\rm
Im}G^{<}_{\alpha\alpha}({\bf r}=0,\omega)}{\sum_{\alpha}{\rm
Im}G^{R}_{\alpha\alpha}({\bf r}=0,\omega)},\quad n_{\rm
ph}(\omega)=\frac{1}{2}\frac{{\rm Im}D^{<}(\omega)}{{\rm Im}D^{R}(\omega)}.$
(12)

We define the effective temperature for electrons and phonons in terms of the
first moment of the distribution, which correctly reduces to the bath
temperature in the equilibrium limit, as

$\displaystyle T_{\rm el}^{2}$ $\displaystyle=$
$\displaystyle\frac{6}{\pi^{2}}\int^{\infty}_{-\infty}\omega[f_{\rm
el}(\omega)-\Theta(-\omega)]{\rm d}\omega$ $\displaystyle T_{\rm ph}^{2}$
$\displaystyle=$ $\displaystyle\frac{6}{\pi^{2}}\int_{0}^{\infty}\omega n_{\rm
ph}(\omega){\rm d}\omega,$ (13)

with the step-function $\Theta(x)$. As shown in Fig. 2b, electronic
distribution functions may deviate strongly from the Fermi-Dirac form, in
which cases $T_{\rm el}$ provides an approximate measure of nonequilibrium
energy excitation.

## References

  * (1) Bardeen, J.  Superconductivity and other macroscopic quantum phenomena.  _Physics Today_ 25 (1990).
  * (2) Bardeen, J.  Classical versus quantum models of charge-density-wave depinning in quasi-one-dimensional metals.  _Phys. Rev. B_ 39, 3528–3532 (1989).  URL https://link.aps.org/doi/10.1103/PhysRevB.39.3528.  Publisher: American Physical Society.
  * (3) Ong, N. P. _et al._ Effect of impurities on the anomalous transport properties of NbSe3.  _Phys. Rev. Lett._ 42, 811–814 (1979).  URL https://link.aps.org/doi/10.1103/PhysRevLett.42.811.
  * (4) Grüner, G.  The dynamics of charge-density waves.  _Rev. Mod. Phys._ 60, 1129–1181 (1988).  URL https://link.aps.org/doi/10.1103/RevModPhys.60.1129.  Publisher: American Physical Society.
  * (5) Maki, K.  Creation of soliton pairs by electric fields in charge-density-wave condensates.  _Phys. Rev. Lett._ 39, 46–48 (1977).  URL https://link.aps.org/doi/10.1103/PhysRevLett.39.46.  Publisher: American Physical Society.
  * (6) Thorne, R. E.  A history of the I-V characteristic of CDW conductors.  _Journal de Physique IV (Proceedings)_ 131, 89–94 (2005).  URL http://www.edpsciences.org/10.1051/jp4:2005131020.
  * (7) Zener, C.  Non-adiabatic crossing of energy levels.  _Proc. R. Soc. A_ 137, 696–702 (1932).
  * (8) Fukuyama, H. & Lee, P.  Dynamics of charge-density wave. 1. Impurity pinning in a single chain.  _Phys. Rev. B_ 17, 535–541 (1978).  URL http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=mekentosj&SrcApp=Papers&DestLinkType=FullRecord&DestApp=WOS&KeyUT=A1978EN01600017.
  * (9) Lee, P. A. & Rice, T. M.  Electric field depinning of charge density waves.  _Phys. Rev. B_ 19, 3970–3980 (1979).  URL https://link.aps.org/doi/10.1103/PhysRevB.19.3970.
  * (10) Fisher, D. S.  Sliding charge-density waves as a dynamic critical phenomenon.  _Phys. Rev. B_ 31, 1396–1427 (1985).  URL https://link.aps.org/doi/10.1103/PhysRevB.31.1396.
  * (11) Janod, E. _et al._ Resistive switching in Mott insulators and correlated systems.  _Adv. Func. Mater._ 25, 6287–6305 (2015).
  * (12) Stoliar, P. _et al._ Universal electric-field-driven resistive transition in narrow-gap Mott insulators.  _Adv. Mater._ 25, 3222–3226 (2013).
  * (13) Guiot, V. _et al._ Avalanche breakdown in GaTa4Se8-xTex narrow-gap Mott insulators.  _Nat. Comm._ 4, 1722–6 (2013).
  * (14) Ridley, B. K.  Specific negative resistance in solids.  _Proc. Phys. Soc._ 82, 954 (1963).
  * (15) Lee, S. B. _et al._ Strong resistance nonlinearity and third harmonic generation in the unipolar resistance switching of NiO thin films.  _Appl. Phys. Lett._ 93, 252102 (2008).
  * (16) del Valle, J., Ramirez, J. G., Rozenberg, M. J. & Schuller, I. K.  Challenges in materials and devices for resistive-switching-based neuromorphic computing.  _J. Appl. Phys._ 124, 211101–24 (2018).  URL http://aip.scitation.org/doi/10.1063/1.5047800.  Publisher: AIP Publishing LLC.
  * (17) Oka, T., Arita, R. & Aoki, H.  Breakdown of a Mott insulator: A nonadiabatic tunneling mechanism.  _Phys. Rev. Lett._ 91, 66406 (2003).
  * (18) Mazza, G., Amaricci, A., Capone, M. & Fabrizio, M.  Field-driven Mott gap collapse and resistive switch in correlated insulators.  _Phys. Rev. Lett._ 117, 176401 (2016).
  * (19) Sugimoto, N., Onoda, S. & Nagaosa, N.  Field-induced metal-insulator transition and switching phenomenon in correlated insulators.  _Phys. Rev. B_ 78, 155104 (2008).
  * (20) Han, J. E., Li, J., Aron, C. & Kotliar, G.  Nonequilibrium mean-field theory of resistive phase transitions.  _Phys. Rev. B_ 98, 035145 (2018).  URL https://arxiv.org/abs/1804.10733.
  * (21) Li, J., Aron, C., Kotliar, G. & Han, J. E.  Microscopic theory of resistive switching in ordered insulators: Electronic versus thermal mechanisms.  _Nano Lett._ 17, 2994–2998 (2017).  URL http://pubs.acs.org/doi/abs/10.1021/acs.nanolett.7b00286.
  * (22) Miller, J., Ordonez, C. & Prodan, E.  Time-correlated soliton tunneling in charge and spin density waves.  _Phys. Rev. Lett._ 84, 1555–1558 (2000).  URL https://link.aps.org/doi/10.1103/PhysRevLett.84.1555.
  * (23) Zimmers, A. _et al._ Role of thermal heating on the voltage induced insulator-metal transition in VO2.  _Phys. Rev. Lett._ 110, 056601 (2013).
  * (24) Giorgianni, F., Sakai, J. & Lupi, S.  Overcoming the thermal regime for the electric-field driven Mott transition in Vanadium Sesquioxide.  _Nat. Comm._ 10, 1–6 (2019).  URL http://dx.doi.org/10.1038/s41467-019-09137-6.  Publisher: Springer US.
  * (25) Jager, M. F. _et al._ Tracking the insulator-to-metal phase transition in VO2 with few-femtosecond extreme UV transient absorption spectroscopy.  _Proc. Nat. Aca. Sci._ 114, 9558–9563 (2017).
  * (26) Zhu, X., Cao, Y., Zhang, J., Plummer, E. W. & Guo, J.  Classification of charge density waves based on their nature.  _Proceedings of the National Academy of Sciences_ 112, 2367–2371 (2015).  URL http://www.pnas.org/lookup/doi/10.1073/pnas.1424791112.
  * (27) Wolff, P. A.  Theory of electron multiplication in Silicon and Germanium.  _Phys. Rev._ 95, 1415 (1954).
  * (28) Li, J., Camille Aron, Kotliar, G. & Han, J. E.  Electric-field-driven resistive switching in the dissipative Hubbard model.  _Phys. Rev. Lett._ 114, 226403 (2015).  URL http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.114.226403.
  * (29) Li, J. & Han, J. E.  Nonequilibrium excitations and transport of Dirac electrons in electric-field-driven graphene.  _Phys. Rev. B_ 97, 205412 (2018).  URL https://link.aps.org/doi/10.1103/PhysRevB.97.205412.  Publisher: American Physical Society.
  * (30) Georges, A., Kotliar, G., Krauth, W. & Rozenberg, M.  Dynamical mean-field theory of strongly correlated fermion systems and the limit of infinite dimensions.  _Rev. Mod. Phys._ 68, 13–125 (1996).
  * (31) Aoki, H. _et al._ Nonequilibrium dynamical mean-field theory and its applications.  _Rev. Mod. Phys._ 86, 779–837 (2014).
  * (32) Aron, C., Kotliar, G. & Weber, C.  Dimensional crossover driven by an electric field.  _Phys. Rev. Lett._ 108, 086401 (2012).
  * (33) Han, J. E. & Li, J.  Energy dissipation in a DC-field-driven electron lattice coupled to fermion baths.  _Phys. Rev. B_ 88, 075113 (2013).  URL http://link.aps.org/doi/10.1103/PhysRevB.88.075113.
  * (34) Weiss, U.  _Quantum Dissipative Systems_ (World Scientific, London, 2008).
  * (35) Khurgin, J., Ding, Y. J. & Jena, D.  Hot phonon effect on electron velocity saturation in GaN: A second look.  _Appl. Phys. Lett._ 91, 252104–3 (2007).  URL http://aip.scitation.org/doi/10.1063/1.2824872.
  * (36) Eckstein, M., Oka, T. & Werner, P.  Dielectric breakdown of Mott insulators in dynamical mean-field theory.  _Phys. Rev. Lett._ 105, 146404 (2010).
  * (37) Kleinman, L.  Theory of phonon-assisted tunneling in semiconductors.  _Phys. Rev._ 140, A637–A648 (1965).  URL https://link.aps.org/doi/10.1103/PhysRev.140.A637.
  * (38) Vdovin, E. . I. _et al._ Phonon-assisted resonant tunneling of electrons in graphene-Boron Nitride transistors.  _Phys. Rev. Lett._ 116, 186603 (2016).  URL https://link.aps.org/doi/10.1103/PhysRevLett.116.186603.
  * (39) Nicholson, C. . I. _et al._ Dimensional crossover in a charge density wave material probed by angle-resolved photoemission spectroscopy.  _Phys. Rev. Lett._ 118, 206401 (2017).  URL http://link.aps.org/doi/10.1103/PhysRevLett.118.206401.
  * (40) Nicholson, C. W. _et al._ Role of a higher-dimensional interaction in stabilizing charge density waves in quasi-one-dimensional NbSe3 revealed by angle-resolved photoemission spectroscopy.  _Phys. Rev. B_ 101, 045412 (2020).  URL https://link.aps.org/doi/10.1103/PhysRevB.101.045412.
  * (41) Zaitsev-Zotov, S. V.  Finite-size effects in quasi-one-dimensional conductors with a charge-density wave.  _Physics-Uspekhi_ 47, 533–554 (2004).  URL http://stacks.iop.org/1063-7869/47/i=6/a=R01?key=crossref.ac4e2eeb5551029c28c63e82d102319c.
  * (42) Zaitsev-Zotov, S. V., Pokrovskii, V. Y. & Monceau, P.  Transition to 1D conduction with decreasing thickness of the crystals of TaS3 and NbSe3 quasi-1D conductors.  _JETP Lett._ 73, 25–27 (2001).  URL http://link.springer.com/10.1134/1.1355400.
  * (43) Itkis, M. E., Nad’, F. Y. & Monceau, P.  Nonlinear conductivity of quasi-one-dimensional TaS${}_{\textrm{3}}$ at low temperatures.  _J. Phys. Condens. Matter_ 2, 8327–8335 (1990).  URL https://iopscience.iop.org/article/10.1088/0953-8984/2/42/010.
  * (44) Fleming, R. M.  Electric-field depinning of charge-density waves in NbSe3.  _Phys. Rev. B_ 22, 5606–5612 (1980).  URL https://link.aps.org/doi/10.1103/PhysRevB.22.5606.
  * (45) Randle, M. _et al._ Gate-controlled metal-insulator transition in TiS3 nanowire field-effect transistors.  _ACS Nano_ 13, 803–811 (2018).  URL http://pubs.acs.org/doi/10.1021/acsnano.8b08260.  Publisher: American Chemical Society.
  * (46) Randle, M. D. _et al._ High-electric-field behavior of the metal-insulator transition in TiS${}_{\textrm{3}}$ nanowire transistors.  _Appl. Phys. Lett._ 120, 073102 (2022).  URL https://aip.scitation.org/doi/10.1063/5.0083166.
  * (47) Guénon, S. _et al._ Electrical breakdown in a V2O3 device at the insulator-to-metal transition.  _Europhys. Lett._ 101, 57003 (2013).
  * (48) Kim, H.-T. _et al._ Electrical oscillations induced by the metal-insulator transition in VO2.  _J. Appl. Phys._ 107, 023702 (2010).
  * (49) Wu, T.-L., Whittaker, L., Banerjee, S. & Sambandamurthy, G.  Temperature and voltage driven tunable metal-insulator transition in individual WxV1-xO2 nanowires.  _Phys. Rev. B_ 83, 073101 (2011).
  * (50) Maki, K.  Thermal fluctuations of the order parameter in charge-density waves.  _Phys. Rev. B_ 33, 2852–2854 (1986).  URL http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=mekentosj&SrcApp=Papers&DestLinkType=FullRecord&DestApp=WOS&KeyUT=A1986A018700099.

Acknowledgements

JEH is grateful for computational support from the CCR at Buffalo. CA
acknowledges the support from the French ANR “MoMA” project ANR-19-CE30-0020
and from the project 6004-1 of the Indo-French Centre for the Promotion of
Advanced Research (IFCPAR). JHH acknowledges the support from the Institute
for Basic Science in the Republic of Korea through the project IBS-R024-D1.
KSK was supported by the Ministry of Education, Science, and Technology
(Grants No. NRF-2021R1A2C1006453 and No. NRF-2021R1A4A3029839) of the National
Research Foundation of Korea (NRF) and by TJ Park Science Fellowship of the
POSCO TJ Park Foundation. We are much grateful to Sambandamurthy Ganapathy,
Han-Woong Yeom, Emmanuel Baudin for helpful discussions.

Author Information

Contributions:  JEH conceived the project and produced the theoretical data.
CA and IM helped development of the code. CA, IM, JHH and KSK contributed to
theoretical discussions. MR and JPB contributed with experimental discussions
and with interpretation of the theoretical results. The manuscript was written
by JEH, CA, MR, and JPB with contributions from all authors.

Corresponding authors: Correspondence to Jong E Han (jonghan@buffalo.edu).


# Transform consistency for learning with noisy labels

Rumeng Yi, Yaping Huang Rumeng Yi, Yaping Huang are with the Beijing Key
Laboratory of Traffic Data Analysis and Mining, Beijing Jiaotong University,
Beijing 100044, China (e-mail<EMAIL_ADDRESS>yphuang@bjtu.edu.cn).

###### Abstract

It is crucial to distinguish mislabeled samples for dealing with noisy labels.
Previous methods such as “Co-teaching” and “JoCoR” introduce two different
networks to select clean samples out of the noisy ones and only use these
clean ones to train the deep models. Different from these methods which
require to train two networks simultaneously, we propose a simple and
effective method to identify clean samples only using one single network. We
discover that the clean samples prefer to reach consistent predictions for the
original images and the transformed images while noisy samples usually suffer
from inconsistent predictions. Motivated by this observation, we introduce to
constrain the transform consistency between the original images and the
transformed images for network training, and then select small-loss samples to
update the parameters of the network. Furthermore, in order to mitigate the
negative influence of noisy labels, we design a classification loss by using
the off-line hard labels and on-line soft labels to provide more reliable
supervisions for training a robust model. We conduct comprehensive experiments
on CIFAR-10, CIFAR-100 and Clothing1M datasets. Compared with the baselines,
we achieve the state-of-the-art performance. Especially, in most cases, our
proposed method outperforms the baselines by a large margin.

##  I Introduction

Deep Neural Networks (DNNs) achieve a remarkable success on many computer
vision tasks due to the large-scale datasets with reliable and clean
annotations [1] [2] [3]. However, collecting such datasets with precise
annotations is expensive and time-consuming. There are two alternative
solutions to alleviate this issue: crowd-sourcing from non-experts and online
queries by search engines. Unfortunately, the obtained annotations inevitably
contain noisy labels. As DNNs have the capability to memorize all training
samples, they will eventually overfit the noisy labels, leading to poor
generalization performance [4] [5].

(a) (a) Symmetric 50%

(b) (b) Asymmetric 40%

Figure 1: KL Divergence between two inputs (original and horizontally flip
images) on CIFAR-10 under (a) symmetric 50% and (b) asymmetric 40% label noise
for clean and noisy samples.

In order to reduce the effect of noisy labels, an efficient strategy is to
select clean samples out of the noisy ones and then only use these clean ones
to update the parameters of the network. Following this research line, many
studies have shown impressive progresses, where Decoupling [6], Co-teaching
[7] and Co-teaching+ [8] are three representative methods. All these methods
train two networks simultaneously, but the difference is that Decoupling and
Co-teaching+ select training samples depending on the disagreement between two
different networks, while in Co-teaching, each network views its small-loss
instances as clean ones, and teaches them to its peer network. Recently, the
state-of-the-art method named JoCoR [9] achieves the excellent performance.
JoCoR applies an agreement maximization principle that trains two networks
with a joint loss (including the conventional supervised loss and the co-
regularization loss) and use the joint loss to select small-loss instances.

In this paper, different from the above methods which require to train two
networks simultaneously, we propose a simple and effective method to
distinguish clean samples only using one single network. We find that the
prediction consistency under different image transforms (such as scaling,
rotation, flipping) in one network is beneficial to select clean samples. As
shown in Figure 1, we feed the original and transformed (horizontally flip)
images into one single network, and observe the Kullback-Leibler (KL)
Divergence of clean and noisy samples on CIFAR-10 dataset with 50% symmetric
noise and 40% asymmetric noise. From Figure 1, we can see that the KL
Divergence of clean samples are much smaller than noisy samples under two
different noise levels. It suggests that the network can reach an agreement on
the original and transformed images for clean samples during training. While
for noisy samples, the KL Divergence remains a larger value. This observation
motivates us to distinguish clean and noisy samples by considering the KL
Divergence between the original images and the transformed images.

Based on the above observation, we propose a novel approach to identify
mislabeled samples. Specifically, we follow the agreement maximization
principle to train a network with a joint loss, including the classification
losses and the KL loss between two inputs (the original images and transformed
images). Moreover, we further adapt a self-ensembling method to construct a
teacher model, the predictions of the teacher model are used as the on-line
soft labels. Then we combine off-line hard labels with on-line soft labels to
provide robust supervisory information for training and further alleviate the
noise effects.

Intuitively, our proposed method can be regarded as adding some perturbations
on the training samples. Since we observe that the perturbation can have
stronger effect on the noisy samples than that on clean samples, making clean
and noisy samples being distinguished more easily. Furthermore, our method
only requires to train one single network, which is more suitable for real
applications.

In summary, our contribution in this paper are three-fold:

  * •

We propose a simple and effective approach to learn from noisy labels, which
introduces the prediction consistency of the images under different spatial
transforms (such as scaling, rotation, flipping) in one single network to
select clean samples efficiently.

  * •

We design a classification loss by using the off-line hard labels and on-line
soft labels to provide reliable supervisions for network training.

  * •

We conduct comprehensive experiments on CIFAR-10, CIFAR-100 and Clothing1M
datasets, and achieve the state-of-the-art performance. In most cases, such as
symmetry-50%, symmetry-80% and asymmetry-40% on CIFAR-10, and symmetry-20%,
symmetry-50% and symmetry-80% on CIFAR-100, the performance of our method
exceeds the second best method by up to 3.70%-12.78%.

##  II Related Works

###  II-A Learning with Noisy Labels

The techniques to alleviate the effect of noisy labels can be divided into two
categories: (1) detecting noisy labels and then cleansing potential noisy
labels or reduce their impacts. (2) directly training noise-robust models with
noisy labels.

One solution of detecting and cleansing noisy labels is to select clean
samples out of the noisy ones, and then use them to update the network. For
example, Decoupling [6], Co-teaching [7], Co-teaching+ [8] and the recent
state-of-the-art JoCoR [9] train two networks by introducing “agreement” or
“disagreement” strategies to select clean samples for training. O2U-Net [10]
adjusts the hyperparameters of the network and ranks the normalized average
loss of every sample to detect the noisy samples. MentorNet [11] learns a
data-driven curriculum to supervise the training of a student network. The
second attempt is based on re-weighting or relabeling methods. Han et al. [12]
propose a deep self-learning framework to relabel the noisy samples by
extracting multiple prototypes for one category. Ren et al. [13] weights each
sample in the loss function based on the gradient directions compared to those
on a clean set. Arazo et al. [14] model the per-sample loss distribution with
a beta mixture model and correct the loss by relying on the network
prediction. In a similar spirit, Li et al. [15] use two networks to perform
sample selection via a gaussian mixture model and further apply the semi-
supervised learning technique, MixMatch [16], to handle noisy labels.

For the second type, Xiao et al. [17] model the relationships between images,
class labels and label noises with a probabilistic graphical model and further
integrate it into an end-to-end deep learning model. Guo et al. [18] propose a
CurriculumNet, where training data are divided into several subsets by ranking
their complexity via distribution density, and then the subsets are formed as
a curriculum to teach the model in understanding noisy labels gradually. Li et
al. [19] propose a meta-learning based method to avoid the model overfitting
to the specific noise by generating synthetic noisy labels.

Motivated by the above studies, we propose a simple and effective method to
select clean samples out of the noisy ones by observing the category
distribution and visual attention map between the original images and the
transformed images in one single network. This achieves an aggregated
inference which combines the predictions from different spatial transforms to
improve the classification accuracy.

###  II-B Image Transformation

Human visual perception shows good consistency under certain spatial
transforms, such as scaling, rotation and flipping, which has motivated the
data augmentation strategy widely used in DNNs. [20] learns image
representations by training convolutional networks to recognize the geometric
transformation that is applied to the input image. [21] proposes a two-branch
network with original images and its transformed images as inputs and
introduces a new attention consistency loss that measures the attention
heatmap consistency between two branches and achieves a new state-of-the-art
for multi-label image classification. [22] augments original labels via self-
supervision of input transformation to learn a single unified task with
respect to the joint distribution of the original and self-supervised labels.

In this paper, we propose the first usage of the transform consistency for
noisy-labeled image classification. Specifically, we regard the transformation
operation performed on the images as the added disturbance. For clean samples,
the network’s predictions are not affected by the added disturbance, resulting
in a consistent prediction between two inputs, while for noisy samples, the
network’s predictions may be inconsistent or oscillate strongly, which allow
us to distinguish the clean samples out of the noisy ones.

Figure 2: Overview of the proposed approach. Specifically, an original image
and its transformed image are both fed into one single network, and then apply
a joint loss, _i.e._ , two classification loss ($\mathcal{L}_{c}^{h}$ and
$\mathcal{L}_{c}^{s}$) and one KL loss (${{\mathcal{L}}_{KL}}$) to train the
network. Finally we select the samples with small joint loss to update the
model’s parameters ($\theta$). Besides, we use the exponential moving average
(EMA) to construct a teacher model, the prediction of the two inputs are used
as the on-line soft labels for optimizing $\theta$.

###  II-C Self-ensemble Learning

Self-ensemble has been widely studied in semi-supervised learning. These
methods form a consensus prediction of the unknown labels using the outputs of
the network in training on different epochs. For example, temporal ensembling
[23] maintains an exponential moving average (EMA) of label predictions on
each training example, and penalizes predictions that are inconsistent with
this target. Mean teacher [24] averages model weights over training steps to
form a target-generating teacher model. Recently, the self-ensembling strategy
is used for noisy-labeled image classification. MLNT [19] designs a noise-
tolerant algorithm which constructs a teacher model to give more reliable
predictions unaffected by the synthetic noise. SELF [25] attempts to identify
clean labels progressively using self-forming ensembles of models and
predictions.

To effectively mitigate the negative influence of noisy labels, in this paper,
we introduce the self-ensembling method to construct a teacher model by using
the exponential moving average (EMA) of the model snapshots in each training
iterations. Then the predictions of the teacher model are used as the on-line
soft labels, which provide a more stable supervisory signal than noisy hard
labels to guide the model’s training. Note that this teacher model does not
require additional training.

In this section, we give the details of our proposed method. Our approach aims
at identifying potentially noisy labels during training and keeping the
network from receiving supervision of the clean labels. The overall framework
is illustrated in Figure 2. Specifically, an original image and its
transformed image are both fed into one single network, and then we apply a
joint loss, _i.e._ , two classification losses and one KL loss, to train the
network. Based on our observed principle that the model can keep consistent
predictions on two inputs of clean samples and inconsistent predictions on
noisy labels, we select the samples with small joint loss to update the
model’s parameters. In order to further alleviate the negative influence of
label noise, apart from the off-line hard labels, our framework further
incorporates on-line soft pseudo labels by using the self-ensembling strategy
into the training process.

Problem statement. Image classification can be formulated with a training set
$\mathcal{D}=\\{({{x}_{1}},{{y}_{1}}),...,({{x}_{n}},{{y}_{n}})\\}$, where
${{x}_{i}}$ is an image and ${{y}_{i}}\in{{\\{0,1\\}}^{C}}$ is the one-hot
label over $C$ classes which may contain noise. Let $f({{x}_{i}};\theta)$
denote the model’s output softmax probability parameterized by $\theta$.

Classification loss. In our framework, the two inputs of the network are
${{x}_{i}}$ and ${{\tilde{x}}_{i}}$, respectively, where
${{\tilde{x}}_{i}}=t({{x}_{i}})$ and $t$ represents the transformation
operation such as scaling, rotation and flipping. Their label confidences can
be predicted as $f({{x}_{i}};\theta)$ and $f({{\tilde{x}}_{i}};\theta)$. In
general, the objective function is the empirical risk of cross-entropy loss,
which is formulated by:

${{\mathcal{L}}_{c}^{h}}({{x}_{i}})=-\frac{1}{N}\sum\limits_{i=1}^{N}{{{y}_{i}}\cdot\log(f({{x}_{i}};\theta))}$
(1)
${{\mathcal{L}}_{c}^{h}}({{\tilde{x}}_{i}})=-\frac{1}{N}\sum\limits_{i=1}^{N}{{{y}_{i}}\cdot\log(f({{{\tilde{x}}}_{i}};\theta))}$
(2)

where $N$ is the total number of samples. However, since ${{y}_{i}}$ contains
noise, the model would overfit the noisy labels and result a poor
classification performance.

To address this problem, we construct a teacher model parameterized by
$\tilde{\theta}$ following the self-ensembling method to generate reliable
soft pseudo labels for supervising the model’s training. Specifically, at
current iteration $T$, we update ${{\tilde{\theta}}^{(T)}}$ with

${{\tilde{\theta}}^{(T)}}=\gamma{{\tilde{\theta}}^{(T-1)}}+(1-\gamma)\theta$
(3)

where $\gamma$ is a smoothing coefficient hyperparameter within the range
[0,1), and ${{\tilde{\theta}}^{(T-1)}}$ indicates the parameter of the teacher
model in the previous iteration $(T-1)$. Then the soft classification loss for
optimizing $\theta$ with the soft pseudo labels generated from the teacher
model can be formulated as:

$\mathcal{L}_{c}^{s}({{x}_{i}})=-\frac{1}{N}\sum\limits_{i=1}^{N}{f({{x}_{i}};\tilde{\theta})}\cdot\log(f({{x}_{i}};\theta))$
(4)
$\mathcal{L}_{c}^{s}({{\tilde{x}}_{i}})=-\frac{1}{N}\sum\limits_{i=1}^{N}{f({{{\tilde{x}}}_{i}};\tilde{\theta})}\cdot\log(f({{\tilde{x}}_{i}};\theta))$
(5)

Therefore, the classification loss of ${{x}_{i}}$ and ${{\tilde{x}}_{i}}$ are
given by:

${{\mathcal{L}}_{c}}({{x}_{i}})=\mathcal{L}_{c}^{h}({{x}_{i}})+\mathcal{L}_{c}^{s}({{x}_{i}})$
(6)

${{\mathcal{L}}_{c}}({{\tilde{x}}_{i}})=\mathcal{L}_{c}^{h}({{\tilde{x}}_{i}})+\mathcal{L}_{c}^{s}({{\tilde{x}}_{i}})$
(7)

KL loss. Based on the observation that the model can make consistent
predictions on the original and transformed images of clean samples, and
inconsistent predictions on most noisy samples, we apply the Kullback-Leibler
(KL) Divergence as KL loss to measure the probability distributions of the
network for two inputs. In practice, the KL loss can be expressed as

${{\mathcal{L}}_{KL}}={{D}_{KL}}({{p}_{x}}\parallel{{p}_{{\tilde{x}}}})+{{D}_{KL}}({{p}_{{\tilde{x}}}}\parallel{{p}_{x}})$
(8)

where

${{D}_{KL}}({{p}_{x}}\parallel{{p}_{{\tilde{x}}}})=\sum\nolimits_{i=1}^{N}{\sum\nolimits_{j=1}^{C}{{{p}^{j}}({{x}_{i}})\log\frac{{{p}^{j}}({{x}_{i}})}{{{p}^{j}}({{{\tilde{x}}}_{i}})}}}$
(9)
${{D}_{KL}}({{p}_{{\tilde{x}}}}\parallel{{p}_{x}})=\sum\nolimits_{i=1}^{N}{\sum\nolimits_{j=1}^{C}{{{p}^{j}}({{{\tilde{x}}}_{i}})\log\frac{{{p}^{j}}({{{\tilde{x}}}_{i}})}{{{p}^{j}}({{x}_{i}})}}}.$
(10)

Here $C$ is the number of categories. ${{p}^{j}}({{x}_{i}})$ and
${{p}^{j}}({{\tilde{x}}_{i}})$ represent the predicted probability
distributions, _i.e._ , ${{p}}({{x}_{i}})={{f}}({{x}_{i}};\theta)$ and
${{p}}({{\tilde{x}}_{i}})={{f}}({{\tilde{x}}_{i}};\theta)$, repectively.

Overall loss. We integrate the above-mentioned losses. The total loss can be
formulated as:

${{\mathcal{L}}_{total}}=(1-\lambda)({{\mathcal{L}}_{c}}({{x}_{i}})+{{\mathcal{L}}_{c}}({{\tilde{x}}_{i}}))+\lambda{{\mathcal{L}}_{KL}}$
(11)

where $\lambda$ is the parameter weighting the classification loss and KL
loss.

Small-loss selection. We use the total loss to select the small-loss
instances. Specifically, the network feeds forward a mini-batch of data
$\bar{\mathcal{D}}=\\{({{x}_{1}},{{y}_{1}}),({{x}_{2}},{{y}_{2}}),...,({{x}_{B}},{{y}_{B}})\\}$
from $\mathcal{D}$ first, then we select a small proportion of instances
${{\bar{\mathcal{D}}}^{{}^{\prime}}}$ in $\bar{\mathcal{D}}$ which have small
total loss:

${{\bar{\mathcal{D}}}^{{}^{\prime}}}=\arg{{\min}_{{{\mathcal{D}}^{{}^{\prime}}}:|{{\mathcal{D}}^{{}^{\prime}}}|\geq
R(T)|\bar{\mathcal{D}}|}}{{\mathcal{L}}_{total}}({{\mathcal{D}}^{{}^{\prime}}})$
(12)

where $R(T)$ is a parameter that controls how many small-loss instances should
be selected. For $R(T)$, we follow the same update strategy as [7] [8] [9].
Specifically, since the DNNs start to learn from clean samples in initial
phases and gradually adapt to noisy ones during training, we apply a large
$R(T)$ to use more small-loss instances to train the model in the beginning.
With the increase of epoch, as the DNNs will overfit the noisy data gradually,
we should update the $R(T)$ with a smaller value to keep less small-loss
instances selected in each mini batch. The update strategy for $R(T)$ is given
by:

$R(T)=1-\min\\{\frac{T}{{{T}_{k}}}\tau,\tau\\}$ (13)

where $T$ is the current iteration, ${{T}_{k}}$ represents how many epochs for
linear drop rate, and $\tau$ is the actual noise rate in the whole dataset
which is closely related to the noise ratio in the noisy classes. If $\tau$ is
not known in advanced, $\tau$ can be inferred using validation set [26] [27].

Algorithm 1

1:$\theta$ and $\tilde{\theta}$, training set $\mathcal{D}$, learning rate
$\eta$, fixed $\tau$, epoch ${{T}_{k}}$ and ${{T}_{max}}$, iteration
${{N}_{max}}$, ensembling momentum $\gamma$ ;

2:for $T$=1,…,${{T}_{max}}$ do

3: Shuffle training set $\mathcal{D}$;

4: for $N$=1,…,${{N}_{max}}$ do

5: Fetch mini-batch ${\bar{\mathcal{D}}}$ from $\mathcal{D}$;

6: Calculate the total loss ${{\mathcal{L}}_{total}}$ by Eq. (11);

7: Obtain small-loss sets ${{{\bar{\mathcal{D}}}}^{{}^{\prime}}}$ by Eq. (12)
from ${\bar{\mathcal{D}}}$;

8: Obtain $\mathcal{L}$ by Eq. (14) on
${{{\bar{\mathcal{D}}}}^{{}^{\prime}}}$;

9: Update ${\theta\text{ = }\theta\text{ - }\eta\nabla\mathcal{L}}$;

10: end for

11: Update ${{\tilde{\theta}}^{(T)}}=\text{
}\gamma{{\tilde{\theta}}^{(T-1)}}\text{ }+\text{ }(1-\gamma)\theta$ by Eq.
(3);

12: Update ${R(T)=1-\min\\{\frac{T}{{{T}_{k}}}\tau,\tau\\}}$ by Eq. (13);

13:end for

14:$\theta$ and $\tilde{\theta}$.

Finally, we calculate the average loss of the selected instances
${{\bar{\mathcal{D}}}^{{}^{\prime}}}$ and back-propagate them to update the
network’s parameters:

$\mathcal{L}=\frac{1}{\left|{{{\bar{\mathcal{D}}}}^{{}^{\prime}}}\right|}\sum\nolimits_{x\in{{{\bar{\mathcal{D}}}}^{{}^{\prime}}}}{{{\mathcal{L}}_{total}}(x)}$
(14)

Algorithm 1 delineates the overall training process.

##  III Experiments

###  III-A Datasets and implementation details

Datasets. We extensively evaluate our approach on CIFAR-10, CIFAR-100 [28] and
Clothing1M [17] datasets. Both CIFAR-10 and CIFAR-100 contain 50K training
images and 10K test images of size 32 $\times$ 32, which involve 10 classes
and 100 classes, respectively. Clothing1M contains 1 million images, which
consists of 50k training images, 14k validation images and 10k test images.
Human annotators are asked to mark a set of 25k labels as a clean set, but we
do not use these in our experiments. Note that the overall label accuracy of
Clothing1M is 61.54%.

Implementation. For all experiments in our paper, we follow the same setting
as JoCoR [9]. Specifically, for CIFAR-10 and CIFAR-100, we use a 7-layer CNN
network architecture. For Clothing1M, we use a 18-layer ResNet.

We use the Adam optimizer with a momentum of 0.9 for all experiments. For
CIFAR-10 and CIFAR-100, the network is trained for 200 epochs. We set the
initial learning rate as 0.001 and the batch size as 128. For Clothing1M, the
network is trained for 15 epochs. We set the constant learning rate as
$5\times{{10}^{-4}}$ and the batch size as 64. In all experiments, we take the
horizontal flipping as the transformation operation, and the ensemble momentum
$\gamma$ in Eq. 3 is set to 0.999. The hyperparameters of total loss weight
$\lambda$ and linear drop rate ${{T}_{k}}$ for all experiments are given in
Table I.

TABLE I: The hyperparameters of total loss weight $\lambda$ and linear drop rate ${{T}_{k}}$ for all experiments. Dataset | Flipping-Rate | $\lambda$ | ${{T}_{k}}$
---|---|---|---
CIFAR-10 | Symmetry-20% |  0.8 |  10
Symmetry-50% | 0.5 | 20
Symmetry-80% | 0.3 | 40
Asymmetry-40% | 0.7 | 50
CIFAR-100 | Symmetry-20% | 0.8 | 40
Symmetry-50% | 0.8 | 30
Symmetry-80% | 0.2 | 30
Asymmetry-40% | 0.9 | 30
Clothing1M | 0.6 | 5

Following previous works [7] [8] [9], we verify our method with two types of
label noise: _symmetric_ and _asymmetric_. The symmetric label noise is
generated by using a random one-hot vector to replace the ground-truth label
of a sample. The asymmetric label noise is designed to mimic the structure of
real-world label noise, such as CAT$\leftrightarrow$DOG,
BIRD$\leftrightarrow$AIRPLANE. In each experiment, we also follow the same
setting to assume the noise rate $r$ is known, therefore the $\tau$ in Eq.
(13) can be obtained. Specifically, for symmetric label noise, $\tau$ equals
to $r$ (_i.e._ , $\tau=r$). For asymmetric label noise, due to the fact that
the actual noise rate $\tau$ in the whole dataset is half of the noisy rate in
the noisy classes, we set $\tau=r/2$.

Measurements. We use the test accuracy and label prediction to measure the
performance, _i.e._ , _test accuracy = (# of correct predictions) / (# of test
dataset)_ and _label precision = (# of clean labels) / (# of all selected
labels)_. Intuitively, the algorithm with higher label precision is more
robust to the label noise. All experiments are repeated five times and the
error bar for STD in each figure has been highlighted as a shade.

Baseline. We use the conventional training with the cross-entropy loss on
noisy datasets (abbreviated as Standard) as our baseline and compare our
method with following state-of-the-art approaches:

  1. -

Decoupling [6], which trains two networks simultaneously, and then updates
models only using the instances that have different predictions from these two
networks.

  2. -

Co-teaching [7], which trains two networks simultaneously, and each network
uses its small-loss instances to update its peer network’s parameters.

  3. -

Co-teaching+ [8], which trains two networks simultaneously using the
disagreement-update step (data update) and cross-update step (parameters
update).

  4. -

JoCoR [9], which trains two networks with a pseudo-siamese paradigm and update
their parameters simultaneously by a joint loss.

(a) (a) Symmetry-20%

(b) (b) Symmetry-50%

(c) (c) Symmetry-80%

(d) (d) Asymmetry-40%

Figure 3: Results on CIFAR-10 dataset. Top: test accuracy (%) vs. epochs; bottom: label precision (%) vs. epochs. TABLE II: Average test accuracy (%) on CIFAR-10 over the last 10 epochs. Flipping-Rate | Standard | Decoupling | Co-teaching | Co-teaching+ | JoCoR | Ours
---|---|---|---|---|---|---
Symmetry-20% | $74.32\pm 0.07$ | $85.72\pm 0.05$ | $86.21\pm 0.04$ | $86.88\pm 0.06$ | $88.00\pm 0.02$ | $\textbf{89.85}\pm\textbf{0.06}$
Symmetry-50% | $49.98\pm 0.08$ | $77.50\pm 0.03$ | $83.15\pm 0.06$ | $80.80\pm 0.03$ | $82.23\pm 0.04$ | $\textbf{86.85}\pm\textbf{0.05}$
Symmetry-80% | $24.03\pm 0.09$ | $55.80\pm 0.07$ | $63.01\pm 0.04$ | $56.31\pm 0.04$ | $60.59\pm 0.05$ | $\textbf{73.84}\pm\textbf{0.08}$
Asymmetry-40% | $72.29\pm 0.15$ | $66.34\pm 0.12$ | $79.23\pm 0.17$ | $72.74\pm 0.26$ | $77.81\pm 0.21$ | $\textbf{85.87}\pm\textbf{0.11}$

###  III-B Comparison with state-of-the-art methods

####  III-B1 Results on CIFAR-10 dataset

The test accuracy on CIFAR-10 dataset is shown in Table II. From the
comparison results, we can clearly see that our method performs the best in
all four cases. Specifically, in the easiest Symmetry-20% case, all methods
work well, but our method still achieves an improvement more than 1.85% over
the best baseline method JoCoR (89.85% vs. 88.00%). When the noise rate raises
to 50%, the test accuracy of Decoupling decreases lower than 80%, but the
other three methods remain above 80%, where Co-teaching performs better than
JoCoR and Co-teaching+. Our method exceeds Co-teaching by 3.70% (86.85% vs.
83.15%). In the Symmetry-80% case, which means that the network trains with
extremely noisy labels, methods based on “disagreement” strategy, _i.e._ ,
Decoupling and Co-teaching+, cannot work well in this case. Co-teaching and
JoCoR only reach to 63.01% and 60.59% respectively. Surprisingly, our method
largely exceeds these two methods by 10.83% and 13.25% respectively. In the
hardest Asymmetry-40%, Decoupling and Co-teaching+ also fail, where Decoupling
works much worse than the Standard method. In contrast, Co-teaching, JoCoR and
our method perform better. Among them, our method is still the best and
exceeds the second best method Co-teaching by 6.64% (85.87% vs. 79.23%).

In the top of Figure 3, we show test accuracy vs. number of epochs, which can
help us to clearly see the memorization effects of DNNs. For example, the
Standard method first learns from clean samples in initial stages and then
gradually adapts to noisy ones, therefore the curve of test accuracy first
reaches a high level and then decreases gradually. While for Decoupling, the
curve of test accuracy first rises from 0 to 100 epochs, then gradually
decreases from 100 epochs in two cases (_i.e._ , Symmetry-50% and
Symmetry-80%). In other two cases (_i.e._ , Symmetry-20% and Asymmetry-40%),
the test accuracy gradually increases and finally stabilizes. The test
accuracy curves of the other four methods follow the similar trends, _i.e._ ,
they first increase and then level off, but the Co-teaching+ drops slightly in
Asymmetry-40%.

In order to further explain such good performance, we plot the label precision
vs. number of epochs on the bottom of Figure 3. Only Decoupling, Co-teaching,
Co-teaching+, JoCoR and our method are considered here because these
algorithms contain the operation of clean instances selection. We can see that
the label precision curves of Decoupling and Co-teaching+ fail to select the
clean instances in all four cases. The label precision curve of Co-teaching+
first increases at a high level and then gradually decreases, while for
Decoupling, the label precision is always at a lower level. These curves show
that the “disagreement” strategy cannot deal with noisy labels at all because
this strategy does not utilize the memorization effects of DNNs during
training. In contrast, the label precision curves of the other three methods
increase during the training, and then remain a high precision, which means
that they can pick up clean instances successfully. Among them, our method can
achieve a higher label precision only after training a few epochs, and has
always provided more accurate supervisions for the subsequent training
process, resulting in a best classification performance.

####  III-B2 Results on CIFAR-100 dataset

Table III shows the test accuracy on CIFAR-100 dataset. Different from
CIFAR-10 dataset, CIFAR-100 dataset is more difficult for noisy-labeled image
classification because it contains 100 classes. Similarly, the test accuracy
of our method on CIFAR-100 dataset achieves the excellent performance again.
Specifically, in the easiest Symmetry-20% case, all methods work well, but our
method exceeds the second best method JoCoR 4.85% (62.00% vs. 57.15%). When
the noise rate raises to 50%, our method is still the best and outperforms the
second method JoCoR 6.59% (55.54% vs. 48.95%). In the extreme Symmetry-80%
case, Decoupling, Co-teaching+ and JoCoR fail to achieve good performance,
_i.e._ , they are all below 20%. Co-teaching also only achieves 22.08%, but
our method achieves 34.86%, which is much higher than Co-teaching by 12.78%.
In the hardest case (_i.e._ , Asymmetry-40%), Decoupling, Co-teaching and
JoCoR are below 35%. Co-teaching+ achieves the best performance (45.19%), but
our method is comparable with Co-teaching+ (43.56%). Meanwhile, our method
achieves the better performance on label precision than Co-teaching+ in this
case.

(a) (a) Symmetry-20%

(b) (b) Symmetry-50%

(c) (c) Symmetry-80%

(d) (d) Asymmetry-40%

Figure 4: Results on CIFAR-100 dataset. Top: test accuracy (%) vs. epochs; bottom: label precision (%) vs. epochs. TABLE III: Average test accuracy (%) on CIFAR-100 over the last 10 epochs. Flipping-Rate | Standard | Decoupling | Co-teaching | Co-teaching+ | JoCoR | Ours
---|---|---|---|---|---|---
Symmetry-20% | $38.15\pm 0.07$ | $45.80\pm 0.05$ | $52.58\pm 0.10$ | $54.82\pm 0.07$ | $57.15\pm 0.04$ | $\textbf{62.00}\pm\textbf{0.20}$
Symmetry-50% | $20.58\pm 0.05$ | $37.98\pm 0.05$ | $45.62\pm 0.06$ | $46.51\pm 0.07$ | $48.95\pm 0.03$ | $\textbf{55.54}\pm\textbf{0.20}$
Symmetry-80% | $6.88\pm 0.03$ | $17.56\pm 0.04$ | $22.08\pm 0.04$ | $12.16\pm 0.03$ | $18.63\pm 0.03$ | $\textbf{34.86}\pm\textbf{0.17}$
Asymmetry-40% | $30.08\pm 0.11$ | $31.60\pm 0.03$ | $32.22\pm 0.04$ | $\textbf{45.19}\pm\textbf{0.04}$ | $34.00\pm 0.04$ | $43.56\pm 0.20$

Figure 4 shows the test accuracy and label precision vs. epochs. We can
observe the memorization effects of DNNs using different methods again.
Similar with the results on CIFAR-10 dataset, the Standard method first
increases a high level and then decreases gradually. Decoupling and Co-
teaching+ always worse than the other three approaches in Symmetry-20%, 50%
and 80%, and the label precision curves of them also explain the results that
they have no ability to deal with the noisy labels. Instead, Co-teaching,
JoCoR and our method mitigate the memorization issue. They can select clean
instances out of the noisy ones successfully, resulting in a good performance.
Only in Asymmetry-40%, our method is slightly lower than Co-teaching+, but our
method achieves the better performance on label precision than Co-teaching+.
Besides, by observing the convergence process of the model trained with five
methods, we can see that our method can stabilize and achieve the better
classification performance only after about 100 epochs.

####  III-B3 Results on Clothing1M dataset

We demonstrate the efficiency of our proposed method on real-world noisy
labels using Clothing1M dataset. The results are shown in Table IV. We report
the _best_ test accuracy across all epochs and the _last_ test accuracy at the
end of training. Our method gains the best performance compared to the state-
of-the-art methods in _best_ and _last_ test accuracy and achieves a
significant improvement in _last_ accuracy of 3.14% over Standard method, and
an improvement of 0.24% over the second best method JoCoR.

TABLE IV: Classification accuracy (%) on the Clothing1M test set. Methods |   _Best_ |   _Last_
---|---|---
Standard |  67.44 |  66.99
Decoupling |  68.48 |  67.32
Co-teaching |  69.21 |  68.51
Co-teaching+ |  59.32 |  58.79
JoCoR |  70.67 |  69.89
Ours |  70.77 |  70.13

###  III-C Ablation study

In this section, we evaluate two components, _i.e._ , transform consistency
and soft classification loss, in our proposed method by conducting ablation
studies on CIFAR-10 and CIFAR-100 datasets. Results are shown in Table V. We
create a baseline model that utilizes only off-line hard labels for training
(the first row).

TABLE V: Ablation study results in terms of test accuracy (%) on CIFAR-10 and CIFAR-100 datasets. Dataset | CIFAR-10 | CIFAR-100
---|---|---
Noise type | Symmetry | Asymmetry | Symmetry | Asymmetry
Methods/Noise ratio | 20% | 50% | 80% | 40% | 20% | 50% | 80% | 40%
$\lambda\text{=}0$ & w/o ${\tilde{\theta}}$ |  | 87.38
---
$\pm 0.16$
| 84.21
---
$\pm 0.09$
| 72.64
---
$\pm 0.12$
| 79.26
---
$\pm 0.21$
| 55.52
---
$\pm 0.09$
| 48.32
---
$\pm 0.17$
| 25.20
---
$\pm 0.14$
| 34.79
---
$\pm 0.16$
$\lambda\text{=}0$ |  | 87.92
---
$\pm 0.08$
| 84.87
---
$\pm 0.05$
| 73.45
---
$\pm 0.11$
| 80.25
---
$\pm 0.25$
| 55.58
---
$\pm 0.17$
| 48.89
---
$\pm 0.20$
| 25.99
---
$\pm 0.17$
| 35.62
---
$\pm 0.22$
w/o ${\tilde{\theta}}$ |  | 89.23
---
$\pm 0.07$
| 86.21
---
$\pm 0.11$
| 72.85
---
$\pm 0.20$
| 84.37
---
$\pm 0.38$
| 61.62
---
$\pm 0.16$
| 54.09
---
$\pm 0.19$
| 33.15
---
$\pm 0.13$
| 42.42
---
$\pm 0.18$
Ours |  | 89.85
---
$\pm\textbf{0.06}$
| 86.85
---
$\pm\textbf{0.05}$
| 73.84
---
$\pm\textbf{0.08}$
| 85.87
---
$\pm\textbf{0.11}$
| 62.00
---
$\pm\textbf{0.20}$
| 55.54
---
$\pm\textbf{0.20}$
| 34.86
---
$\pm\textbf{0.17}$
| 43.56
---
$\pm\textbf{0.20}$
TABLE VI: Test accuracy (%) base00d0 on various transforms on CIFAR-10 and CIFAR-100 datasets. Dataset | CIFAR10 | CIFAR100
---|---|---
Noise type | Symmetry | Asymmetry | Symmetry | Asymmetry
Transform/Noise ratio | 20% | 50% | 80% | 40% | 20% | 50% | 80% | 40%
Scaling |  | 83.06
---
$\pm 0.19$
| 81.78
---
$\pm 0.17$
| 69.63
---
$\pm 0.28$
| 71.96
---
$\pm 0.41$
| 49.52
---
$\pm 0.37$
| 43.53
---
$\pm 0.29$
| 22.72
---
$\pm 0.25$
| 34.18
---
$\pm 0.33$
Rotation-$90{}^{\circ}$ |  | 86.51
---
$\pm 0.14$
| 83.02
---
$\pm 0.14$
| 68.70
---
$\pm 0.15$
| 78.77
---
$\pm 0.26$
| 54.97
---
$\pm 0.28$
| 49.40
---
$\pm 0.25$
| 29.13
---
$\pm 0.31$
| 35.93
---
$\pm 0.28$
Rotation-$180{}^{\circ}$ |  | 87.23
---
$\pm 0.08$
| 82.36
---
$\pm 0.22$
| 67.24
---
$\pm 0.28$
| 78.51
---
$\pm 0.16$
| 57.46
---
$\pm 0.25$
| 51.77
---
$\pm 0.26$
| 32.88
---
$\pm 0.17$
| 40.21
---
$\pm 0.27$
Rotation-$270{}^{\circ}$ |  | 86.38
---
$\pm 0.16$
| 83.09
---
$\pm 0.15$
| 68.37
---
$\pm 0.10$
| 77.86
---
$\pm 0.19$
| 54.83
---
$\pm 0.25$
| 48.70
---
$\pm 0.26$
| 29.56
---
$\pm 0.29$
| 35.87
---
$\pm 0.35$
Rotation-$360{}^{\circ}$ |  | 87.31
---
$\pm 0.12$
| 83.33
---
$\pm 0.16$
| 71.63
---
$\pm 0.30$
| 80.29
---
$\pm 0.22$
| 56.99
---
$\pm 0.24$
| 50.53
---
$\pm 0.30$
| 31.41
---
$\pm 0.28$
| 41.87
---
$\pm 0.22$
Vertical flipping |  | 86.13
---
$\pm 0.11$
| 80.59
---
$\pm 0.17$
| 67.03
---
$\pm 0.17$
| 78.25
---
$\pm 0.16$
| 55.95
---
$\pm 0.22$
| 49.09
---
$\pm 0.21$
| 30.09
---
$\pm 0.29$
| 38.07
---
$\pm 0.32$
Horizontal flipping |  | 89.85
---
$\pm\textbf{0.06}$
| 86.85
---
$\pm\textbf{0.05}$
| 73.84
---
$\pm\textbf{0.08}$
| 85.87
---
$\pm\textbf{0.11}$
| 62.00
---
$\pm\textbf{0.20}$
| 55.54
---
$\pm\textbf{0.20}$
| 34.86
---
$\pm\textbf{0.17}$
| 43.56
---
$\pm\textbf{0.20}$

Effectiveness of transform consistency. To observe the effectiveness of the KL
loss, we set $\lambda=0$ in Eq.11. The results are shown in the second row of
Table V (denoted as $\lambda=0$). The test accuracy of four cases drops by
0.39% to 5.62% on CIFAR-10 dataset and 6.42% to 8.87% on CIFAR-100 dataset.
These experiments fully demonstrate that the clean and noisy samples can be
distinguished more easily by adding KL loss and further verify the
effectiveness of transform consistency.

Effectiveness of soft classification loss. In order to investigate the
effectiveness of the soft classification loss, we perform the experiment by
only utilizing the off-line hard labels as the supervisory information (_i.e._
, $\mathcal{L}_{c}^{s}({{x}_{i}})=0$ in Eq. 6 and
$\mathcal{L}_{c}^{s}({{\tilde{x}}_{i}})=0$ in Eq. 7). The performance is
presented in the third row of Table V (denoted as w/o ${\tilde{\theta}}$). The
test accuracy of four cases drops by 0.62% to 1.50% on CIFAR-10 dataset and
0.38% to 1.71% on CIFAR-100 dataset. These results suggest that the on-line
soft labels are more reliable due to the better decoupling between the past
average models of the two networks, which can effectively mitigate the
negative influence of hard noisy labels and avoid bias amplification even when
the networks have much erroneous outputs in the early training epochs.

###  III-D Effectiveness of different transforms

We conduct experiments to investigate that the prediction consistency under
different image transforms on CIFAR-10 and CIFAR-100 datasets. Specifically,
we focus on a subset of frequently used transforms: _scaling, rotation_ , and
_flipping_. Scaling means the input images are resized to 48 $\times$ 48, and
then cropped to 32 $\times$ 32\. Rotation contains $90{}^{\circ}$,
$180{}^{\circ}$, $270{}^{\circ}$ and $360{}^{\circ}$, and flipping contains
vertical flipping and horizontal flipping. The results are shown in Table VI.
Among them, the test accuracy of horizontal flipping gains the best
performance on two datasets, and the test accuracy of scaling is the worst on
two datasets except two cases on CIFAR-10, _i.e._ , Symmetry-50% and
Symmetry-80%. While for rotation and vertical flipping, the results in four
cases on two datasets present comparable performance.

##  IV Conclusion

In this paper, we propose a simple and effective approach that utilize the
transform consistency to identify mislabeled samples. Specifically, we train
one single network with a joint loss between two inputs (the original image
and its transformed image), which includes two classification losses and one
KL loss. Furthermore, we design a classification loss by using the off-line
hard labels and on-line soft labels to provide more reliable supervisions for
training a robust model. We conduct comprehensive experiments on CIFAR-10,
CIFAR-100 and Clothing1M datasets and achieve the state-of-the-art
performance.

## References

  * [1] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in ECCV, pp. 801–818, 2018.
  * [2] R. Girshick, “Fast r-cnn,” in ICCV, pp. 1440–1448, 2015.
  * [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, pp. 770–778, 2016.
  * [4] D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa, “Joint optimization framework for learning with noisy labels,” in CVPR, pp. 5552–5560, 2018.
  * [5] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” arXiv preprint arXiv:1611.03530, 2016.
  * [6] E. Malach and S. Shalev-Shwartz, “Decoupling” when to update” from” how to update”,” in NeurIPS, pp. 960–970, 2017.
  * [7] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” in NeurIPS, pp. 8527–8537, 2018.
  * [8] X. Yu, B. Han, J. Yao, G. Niu, I. W. Tsang, and M. Sugiyama, “How does disagreement help generalization against label corruption?,” arXiv preprint arXiv:1901.04215, 2019.
  * [9] H. Wei, L. Feng, X. Chen, and B. An, “Combating noisy labels by agreement: A joint training method with co-regularization,” in CVPR, pp. 13726–13735, 2020.
  * [10] J. Huang, L. Qu, R. Jia, and B. Zhao, “O2u-net: A simple noisy label detection approach for deep neural networks,” in ICCV, pp. 3326–3334, 2019.
  * [11] L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei, “Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels,” in ICML, pp. 2304–2313, 2018.
  * [12] J. Han, P. Luo, and X. Wang, “Deep self-learning from noisy labels,” in ICCV, pp. 5138–5147, 2019.
  * [13] M. Ren, W. Zeng, B. Yang, and R. Urtasun, “Learning to reweight examples for robust deep learning,” arXiv preprint arXiv:1803.09050, 2018.
  * [14] E. Arazo, D. Ortego, P. Albert, N. E. O’Connor, and K. McGuinness, “Unsupervised label noise modeling and loss correction,” arXiv preprint arXiv:1904.11238, 2019.
  * [15] J. Li, R. Socher, and S. C. Hoi, “Dividemix: Learning with noisy labels as semi-supervised learning,” arXiv preprint arXiv:2002.07394, 2020.
  * [16] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” in NeurIPS, pp. 5049–5059, 2019.
  * [17] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang, “Learning from massive noisy labeled data for image classification,” in CVPR, pp. 2691–2699, 2015.
  * [18] S. Guo, W. Huang, H. Zhang, C. Zhuang, D. Dong, M. R. Scott, and D. Huang, “Curriculumnet: Weakly supervised learning from large-scale web images,” in ECCV, pp. 135–150, 2018.
  * [19] J. Li, Y. Wong, Q. Zhao, and M. S. Kankanhalli, “Learning to learn from noisy labeled data,” in CVPR, pp. 5051–5059, 2019.
  * [20] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” arXiv preprint arXiv:1803.07728, 2018.
  * [21] H. Guo, K. Zheng, X. Fan, H. Yu, and S. Wang, “Visual attention consistency under image transforms for multi-label image classification,” in CVPR, pp. 729–739, 2019.
  * [22] H. Lee, S. J. Hwang, and J. Shin, “Self-supervised label augmentation via input transformations,” in Proceedings of Machine Learning and Systems 2020, pp. 3537–3547, 2020.
  * [23] S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” arXiv preprint arXiv:1610.02242, 2016.
  * [24] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in NeurIPS, pp. 1195–1204, 2017.
  * [25] D. T. Nguyen, C. K. Mummadi, T. P. N. Ngo, T. H. P. Nguyen, L. Beggel, and T. Brox, “Self: Learning to filter noisy labels with self-ensembling,” arXiv preprint arXiv:1910.01842, 2019.
  * [26] T. Liu and D. Tao, “Classification with noisy labels by importance reweighting,” PAMI, vol. 38, no. 3, pp. 447–461, 2015.
  * [27] X. Yu, T. Liu, M. Gong, K. Batmanghelich, and D. Tao, “An efficient and provable approach for mixture proportion estimation using linear independence assumption,” in CVPR, pp. 4480–4489, 2018.
  * [28] A. Krizhevsky, G. Hinton, et al., “Learning multiple layers of features from tiny images,” 2009.