# Topological phases in N-layer ABC-graphene boron-nitride moire superlattices

David Andrés Galeano González  Instituto de Física, Universidad de Antioquia,
AA 1226, Medellín, Colombia  Department of Physics, University of Seoul, Seoul
02504, Korea  Bheema Lingam Chittari  Department of Physical Sciences, Indian
Institute of Science Education and Research Kolkata, Mohanpur 741246, West
Bengal, India  Youngju Park  Department of Physics, University of Seoul, Seoul
02504, Korea  Jin-Hua Sun  The Research Institute of Advanced Technologies,
Ningbo University, Zhejiang 315211, P. R. China  Jeil Jung<EMAIL_ADDRESS>Department of Physics, University of Seoul, Seoul 02504, Korea  Department of
Smart Cities, University of Seoul, Seoul 02504, Korea

###### Abstract

Rhombohedral $N=3$ trilayer graphene on hexagonal boron nitride (TLG/BN) hosts
gate-tunable, valley-contrasting, nearly flat topological bands that can
trigger spontaneous quantum Hall phases under appropriate conditions of the
valley and spin polarization. Recent experiments have shown signatures of C =
2 valley Chern bands at 1/4 hole filling, in contrast to the predicted value
of C = 3. We discuss the low-energy model for rhombohedral N-layer graphene (N
= 1, 2, 3) aligned with hexagonal boron nitride (hBN) subject to off-diagonal
moire vector potential terms that can alter the valley Chern numbers. Our
analysis suggests that topological phase transitions of the flat bands can be
triggered by pseudomagnetic vector field potentials associated to moire strain
patterns, and that a nematic order with broken rotational symmetry can lead to
valley Chern numbers that are in agreement with recent Hall conductivity
observations.

##  I Introduction

In recent years, magic-angle twisted bilayer graphene Bistritzer and MacDonald
(2011); Cao  _et al._ (2018a, b) (tBG) has emerged as a platform for exploring
correlated insulating phases and unconventional superconductivity in
compositionally simple systems, owing to the possibility of achieving
extremely narrow bandwidths where Coulomb repulsion energies can dominate the
kinetic energy of electrons. Bistritzer and MacDonald (2011); Wong  _et al._
(2015); Cao  _et al._ (2018a, b); Yankowitz  _et al._ (2019); Cao  _et al._
(2020); Serlin  _et al._ (2019); Bultinck  _et al._ (2020); Sharpe  _et al._
(2019); Kim  _et al._ (2017); Choi  _et al._ (2019); Liu  _et al._ (2019);
Carr  _et al._ (2018); Chittari  _et al._ (2018a); Leconte  _et al._ (2019);
Zhao  _et al._ (2020); Fidrysiak  _et al._ (2018); Kang and Vafek (2019);
Koshino  _et al._ (2018); Tarnopolsky  _et al._ (2019); Po  _et al._ (2019);
You and Vishwanath (2019); Park  _et al._ (2019); Chichinadze  _et al._
(2020); Shi  _et al._ (2020); Crasto de Lima  _et al._ (2019); Ma  _et al._
(2020); Hesp  _et al._ (2019); Fernandes and Venderbos (2020); Stepanov  _et
al._ (2020); Zhang  _et al._ (2020); Song  _et al._ (2019) The pool of moire
materials exhibiting such behaviors has rapidly expanded beyond twisted
bilayer graphene to include trilayer graphene on hexagonal boron nitride
(TLG/BN), Chittari  _et al._ (2019); Chen  _et al._ (2019a, b, 2020);
Yankowitz  _et al._ (2016); Lin and Ni (2019); Yin  _et al._ (2019); Zhang and
Mao (2020); Kim  _et al._ (2018); Repellin  _et al._ (2020); Zhang and Senthil
(2019); Abouelkomsan  _et al._ (2020); Kindermann  _et al._ (2012); Bokdam
_et al._ (2014); San-Jose  _et al._ (2014); Wallbank  _et al._ (2013); Zhu
_et al._ (2018) or double bilayer graphene (tDBG) Chebrolu  _et al._ (2019);
Liu  _et al._ (2020); Lee  _et al._ (2019); Shen  _et al._ (2020); Choi and
Choi (2019); Koshino (2019); Li  _et al._ (2020); Zhang  _et al._ (2019a) as
representative systems where the perpendicular electric field can control the
flatness of the low-energy moire bands and achieve the narrow bandwidths over
a wider range of the twist angles, without requiring high-precision rotation
as in tBG. Studies have evaluated the feasibility of engineering moire flat
bands in gapped Dirac materials, Javvaji  _et al._ (2020); kun Shi  _et al._
(2020) suggesting TMDC bilayers Pan  _et al._ (2018); Gong  _et al._ (2014);
Huang  _et al._ (2014); Duan  _et al._ (2014); Song and Gabor (2018) as
platforms for identifying nearly flat bands where one can benefit from the
aforementioned looser constraints on twist angle precision. Javvaji  _et al._
(2020); Naik and Jain (2018); Wu  _et al._ (2018, 2019); Jin  _et al._
(2019a); Seyler  _et al._ (2019); Bi and Fu (2019); Zhang  _et al._ (2019b);
Yu  _et al._ (2019); Sushko  _et al._ (2019); Tang  _et al._ (2019); Jin  _et
al._ (2019b); An  _et al._ (2019)

The finite valley Chern number of the moire bands underlie the anomalous Hall
effects observed in transport experiments when the degeneracy of the bands are
lifted via Coulomb interactions. Zhang  _et al._ (2019a); Chittari  _et al._
(2018b) Charge Hall conductivity signals were observed in twisted bilayer
graphene nearly aligned with hexagonal boron nitride (tBG/BN) at 3/4 filling
densities, Sharpe  _et al._ (2019); Serlin  _et al._ (2019) and closely
related traces of quantum anomalous Hall effects were observed in TLG/BN. Chen
_et al._ (2019a) Contrary to the expectations of a charge Hall conductivity of
$\sigma_{xy}=3e^{2}/h$, which is consistent with the predicted $K$ valley
Chern number $C=3$ of hole bands for TLG/BN, Chittari  _et al._ (2018b) the
experiments showed a quantized anomalous Hall conductivity of
$\sigma_{xy}=2e^{2}/h$, which is consistent with $C=2$. Chen  _et al._ (2019a)
In this paper we explore the valley Chern number phase diagram of G/BN, BG/BN,
and ABC stacked TLG/BN structures, that can be described using the low-energy
$N$-chiral Dirac model subject to moire patterns, in an attempt to identify
the system parameter conditions that can alter the valley Chern numbers. The
manuscript is structured as follows. In Sec. II we introduce the model
Hamiltonian, in Sec. III we present the electronic structure results
comprising the valley Chern number phase diagrams, and lastly in Sec. IV we
present the summary and discussions.

##  II Model Hamiltonian

The low-energy model Hamiltonian of rhombohedral $N$-layer graphene on
hexagonal boron nitride at the $K$-valley subject to substrate moire patterns
$H_{\xi}^{M}$ is given by Chittari  _et al._ (2019)

$H_{N}^{\nu,\xi}=\frac{\upsilon_{0}^{N}}{(-t_{1})^{N-1}}\begin{bmatrix}0&\left(\pi^{\dagger}\right)^{N}\\\
\pi^{N}&0\end{bmatrix}+\Delta\sigma_{z}+H^{\rm R}_{N}+H_{\xi}^{M}$ (1)

where $\xi=\pm 1$ distinguishes the two possible (0∘ and 60∘) alignments
between the layers of graphene and BN. The first term in equation (1)
describes the low-energy N-layer graphene $2\times 2$ Hamiltonian containing
the momentum operator $\pi=\nu p_{x}+ip_{y}$, where $\nu=1$ is used for the
principal valley $K$ of graphene, unless stated otherwise. The second term of
equation (1) introduces the interlayer potential difference proportional to
the mass term through $\Delta$. The third term, $H^{\rm R}_{N}$, describes the
remote hopping term corrections for the N-layers. We model the remote hopping
term corrections for TLG with $N=3$, Jung  as shown below:

$\displaystyle H^{\rm R}_{3,\nu}$ $\displaystyle=$
$\displaystyle\left[\left(\frac{2\upsilon_{0}\upsilon_{3}\pi^{2}}{t_{1}}+t_{2}\right)\sigma_{x}\right]$
$\displaystyle+$
$\displaystyle\left[\frac{2\upsilon_{0}\upsilon_{4}\pi^{2}}{t_{1}}-\Delta^{\prime}+\Delta^{\prime\prime}\left(1-\frac{3\upsilon_{0}^{2}\pi^{2}}{t_{1}^{2}}\right)\right]\mathbb{1}.$

The effective hopping parameters for rhombohedral trilayer graphene that fits
the local density approximation (LDA) bands are $t_{0}=-2.62$ eV,
$t_{1}=0.358$ eV, $t_{2}=-0.0083$ eV, $t_{3}=0.293$ eV, and $t_{4}=-0.144$ eV,
where associated velocities are defined as
$\upsilon_{m}=\sqrt{3}a\,|t_{m}|/2\hbar$, using $a=2.46\,\AA$ as the lattice
constant of graphene, and the constants in the diagonal terms are
$\Delta^{\prime}=0.0122$ eV and $\Delta^{\prime\prime}=0.0095$ eV.

In the case of $N=2$, the parameters $t_{2}$ and $\Delta^{\prime\prime}$ drop
out, and $\Delta^{\prime}=0.015$ eV is used for bands obtained within the LDA
Jung and MacDonald (2013) and the remote hopping terms are captured as

$\displaystyle H^{\rm R}_{2,\nu}$ $\displaystyle=$
$\displaystyle-\upsilon_{3}\left(\begin{matrix}0&\pi\\\
\pi^{\dagger}&0\end{matrix}\right)+\frac{\upsilon_{3}}{4\sqrt{3}}\left(\begin{matrix}0&\pi^{\dagger
2}\\\ \pi^{2}&0\end{matrix}\right)$ $\displaystyle+$
$\displaystyle\frac{\upsilon_{0}}{t_{1}}\left[\frac{\Delta^{\prime}\upsilon_{0}}{t_{1}}+2\upsilon_{4}\right]\left(\begin{matrix}\pi^{\dagger}\pi&0\\\
0&\pi\pi^{\dagger}\end{matrix}\right).$

In the case of $N=1$, all the remote hopping terms drop out and $H^{\rm
R}_{1,\nu}=0$.

The last term of equation (1)

$H_{\xi}^{M}=H_{\xi}^{V}+H_{\xi}^{A}$ (3)

is the effective moire potential term induced by the hBN layer that consists
of the diagonal $H_{\xi}^{V}$ and off-diagonal $H_{\xi}^{A}$ terms. We use the
local commensurate stacking vector $\vec{d}=(d_{x},d_{y})$ between the
substrate and the contacting graphene layer, where the stacking vector
$\vec{d}$ and the real space position $\vec{r}$ are related through

$\displaystyle\vec{d}(\vec{r})\simeq\varepsilon\vec{r}+\theta\hat{z}\times\vec{r}$
(4)

in the small angle approximation. Jung  _et al._ (2014) Here, $\theta$ is the
relative twist and $\epsilon=(a-a_{BN})/a_{BN}$ is the lattice constant
mismatch between the graphene and hBN layers, where $a$ and $a_{BN}$ are the
lattice constants of graphene and hBN layers respectively.

The diagonal term of the Hamiltonian in real space is given by

$\displaystyle
H_{\xi}^{V}(\vec{r})=V_{AA/BB}^{M}(\vec{r})\left[\frac{\mathbb{1}+\xi\sigma_{z}}{2}\right]$
(5)

where the moire potential function is given by

$\displaystyle V_{AA/BB}^{M}(\vec{r})=2C_{AA/BB}{\rm
Re}\left[e^{i\phi_{AA/BB}}f^{\xi}(\vec{r})\right],$ (6)

which in turn depends on the auxiliary function

$\displaystyle
f^{\xi}(\vec{r})=\sum_{m=1}^{6}e^{i\xi{{\tilde{G}_{m}}}\cdot{\vec{r}}}\frac{\left(1+(-1)^{m}\right)}{2}$
(7)

expressed using six moire reciprocal lattices
$\tilde{G}_{m=1..6}=\hat{R}_{2\pi(m-1)/3}{\tilde{G}_{1}}$ successively rotated
by 60∘. The moire reciprocal lattice vector
${\tilde{G}_{1}}\approx\epsilon\vec{G}_{1}-\theta\vec{z}\times\vec{G}_{1}$ is
related to the following reciprocal lattice vector of graphene
$\vec{G}_{1}=[0,4\pi/(\sqrt{3}a)]$. For the diagonal terms of the moire
potentials we use the parametrization of a G/BN interface Jung  _et al._
(2014) projected onto only one of the sublattices. Chittari  _et al._ (2019)
The moire potential in Eq. (6) can be broken down in terms of inversion
symmetric and asymmetric patterns, see Refs. [Wallbank  _et al._ , 2013; Jung
_et al._ , 2017] and appendix A for more details.

The intersublattice off-diagonal term $H_{\xi}^{A}(\vec{r})$ introduces
coherence between the low energy sublattices in the system and could arise in
multilayer graphene with BN interfaced systems due to higher order
perturbation corrections that couple the diagonal moire pattern terms with
interlayer tunngling. This term can be modeled as

$\displaystyle
H_{\xi}^{A}(\vec{r})=\vec{A}^{\xi}(\vec{r})\cdot\sigma^{\xi}_{xy}$ (8)

where $\sigma^{\xi}_{xy}=(\sigma_{x},\xi\sigma_{y})$ is the Pauli matrix
vector and the pseudomagnetic vector potential that we model through

$\displaystyle\vec{A}^{\xi}(\vec{r})=V^{M}_{AB}\,\vec{\mathbb{\nabla}}_{\vec{r}}\,{\rm
Re}\left[e^{i\phi_{AB}}f^{\xi}(\vec{r})\right],$ (9)

where the prefactor

$\displaystyle
V^{M}_{AB}=2C_{AB}\left[\cos(\tilde{\theta})\hat{z}\times\frac{\mathbb{1}}{|\tilde{G}|}-\sin(\tilde{\theta})\frac{\mathbb{1}}{|\tilde{G}|}\right]$
(10)

depends on twist angle and lattice constant mismatch

$\displaystyle\cos({\tilde{\theta}})=\frac{\alpha\cos(\theta)-1}{\beta};~{}~{}\sin(\tilde{\theta})=\frac{\alpha\sin(\theta)}{\beta}$
$\displaystyle\beta=\sqrt{\alpha^{2}-2\alpha\cos(\theta)+1};~{}~{}\alpha=1+\epsilon.$

We will show that this off-diagonal vector potential $\vec{A}^{\xi}(\vec{r})$
in Eq. (8) can alter the Chern number phase diagram by breaking the rotational
symmetry and modify the valley Chern numbers depending on system parameter
values.

In a previous report Chittari  _et al._ (2019) on the electronic structure of
TLG/BN the moire potentials were modeled to act at the low energy sublattice
of the contacting graphene layer and zero direct interlayer coherence between
the low energy sites of the top and bottom layers was assumed by using
$C_{AB}=0$ zero off-diagonal term. There the $C=N$ proportionality to layer
number was verified to up to three layers for either valence or conduction
bands depending on the sign of the electric field and hBN substrate alignment
orientation. As a matter of convention, in this work we assume that the $C$
valley Chern number is associated to the $K$ valley of the multilayer graphene
layer while a time reversal symmetric counterpart is assumed for the
$K^{\prime}$ counterpart. The zero or finite integer value of the valley Chern
number in each band is attributed to the sum of the primary and secondary
Chern weights typically concentrated near the gap opening points in the mBZ
leading to the total valley Chern number, such that
$C^{e/h}=w^{e/h}_{P}+w^{e/h}_{S}$. While the primary Chern weight near charge
neutrality is set by the interlayer potential difference, the secondary Chern
weights near the mBZ boundaries depend on the moire potential parameters that
generate the avoided secondary gaps at the mBZ corners. We will show in our
analysis that the secondary Chern weights are easily altered based on the
choice of the off-diagonal moire potential terms.

Figure 1: (color online) Band structures, valley Chern numbers, Berry
curvatures and the Fermi surfaces at the vHS of the low energy bands in TLG/BN
when a finite off-diagonal term as in Eq. (8) is added to the Hamiltonian. (a)
Band structures for $\xi=1$, $N=3$ with $\Delta=0.01$ eV, for different values
of the off-diagonal pseudomagnetic field patterns proportional to $C_{AB}=0$
(black), $C_{AB}=0.007$ eV (red), and $C_{AB}=0.01$ eV (green). For all three
cases we use the $\phi_{AB}=0$ phase term. The respective $K$ valley Chern
numbers for the valence bands are for $C_{AB}=0$, $C=3$ (black), for
$C_{AB}=0.007$ eV, $C=0$ (red) and for $C_{AB}=0.01$ eV, $C=2$ (green). The
valley Chern number of the conduction bands are in all cases $C=0$. (b)
Topological phase transitions as a function $C_{AB}$, keeping $\phi_{AB}=0$
for the phase term when $N=3$ and $N=2$ for $\xi=1$ and $\Delta=0.01$ eV. We
note that for $N=3$ there is a $C=3\rightarrow 2$ transition for the valence
bands and from $C=0\rightarrow-2$ for the conduction bands. For $N=2$ there is
a $C=2\rightarrow 1$ transition in the valence bands and from
$C=0\rightarrow-1$ for the conduction bands. (c) The Berry curvatures for the
three valence bands giving the different Chern numbers. The closure of the
band gap associated with the topological phase transitions can be identified
to take place between $\tilde{K}^{\prime}$ and $\tilde{K}$. (d) Fermi surface
contours at the vHS for the low energy valence and conduction bands where we
can observe breaking of triangular rotational symmetry when $C_{\rm AB}\neq
0$.  Figure 2: (color online) The local density of states (LDOS,
$D({\vec{r}},E)$) along with the total density of states (DOS, $D(E)$ )
projected vertically to account the LDOS with respect to the inclusion of off-
diagonal term in the moire Hamiltonian in TLG/BN. For $\xi=1$ with
$\Delta=0.01$ eV, the LDOS and DOS are obtained for different off-diagonal
terms ($C_{AB}=0,0.007,0.01$ eV), that are discussed in Fig. 1. The real space
representation of the normalized LDOS ($\tilde{D}({\vec{r}},E)$) is presented
for each Van Hove singularity (vHS) at the valence and conduction bands
$C_{AB}=0$, and for the cases when $C_{AB}\neq 0$ we calculate their
differences $\delta\tilde{D}({\vec{r}},E)=\tilde{D}_{C_{AB}\neq
0}({\vec{r}},E)-\tilde{D}_{C_{AB}=0}({\vec{r}},E)$. The normalized LDOS is
defined as $\tilde{D}({\vec{r}},E)=D({\vec{r}},E)$/max$[D({\vec{r}},E)]$ and
we have plotted them at the van Hove singularity energies $E=E_{vHS}$. Figure
3: (color online) The valley Chern number phase diagrams for the low energy
conduction and valence bands of $N$-layer graphene boron nitride moire
superlattices in the parameter space that defines the strength and shape of
off-diagonal terms through the parameters ($C_{AB}$ ($0\sim 0.025$) eV and
$\phi_{AB}$ ($0^{\circ}\sim 180^{\circ}$)) in the moire Hamiltonian. The
valley Chern number phase diagrams are obtained for $\xi=1$ (top panels) and
$\xi=-1$ (bottom panels) with $N=3$ (left), $N=2$ (middle) and $N=1$ (right).

##  III Results and Discussions

Here we present the electronic structure of ABC multilayer graphene on hBN
under the effect of an off-diagonal moire pattern term in Eq. (8) by allowing
variations of the parameters between $C_{AB}=0\sim 0.025$ eV and
$\phi_{AB}=0\sim\pi$ for $N=1,2,3$ systems. We have verified that changing the
diagonal moire parameters alone in a similar range of magnitude and phase did
not modify the valley Chern numbers of the low energy bands which are 0 or
$N$. In the following we assess the impact of this term in particular in the
valley Chern number of the low energy bands for fixed diagonal moire patterns
and discuss the real-space anisotropies introduced in the local density of
states.

###  III.1 Topological phase transitions in the low energy bands

The main finding in this work is that the introduction of an off-diagonal
interlayer coherence moire patterm term through a finite $C_{AB}\neq 0$ can
switch the valley Chern number of the low energy bands in the system as we
illustrate in Fig. 1, where the vector moire potential term
$H_{\xi}^{A}(\vec{d})$ defines a pseudospin field on the graphene layer
contacting hBN.

The inclusion of off-diagonal terms causes significant changes in the
electronic structure and triggers topological phase transitions, as shown in
Fig. 1(a). The band structures for $N=3$ (TLG/BN) were obtained with $\xi=1$
with a constant value of interlayer potential difference $\Delta=0.01$ eV. The
continuous variation of the $C_{AB}$ term with $\phi_{AB}=0$ leads to a gap
closure between the low energy valence band and the higher energy bands
between $\tilde{K}$ and $\tilde{K^{\prime}}$ for a value of $C_{AB}=$ 0.007
eV, and the gap reopens for larger values of $C_{AB}$ as shown in Fig. 1(a).
Under the electric field, the gap at the primary Dirac point and the avoided
gaps at the moire mini-Brillouin-zone (mBZ) boundaries isolate the low-energy
bands near charge neutrality.

Introduction of a finite $C_{AB}$ parameter also breaks the triangular
rotational symmetry of the moire Brillouin zone that can be visualized in the
Berry curvature distribution in Fig. 1(b). The Berry curvatures of the low
energy $n^{th}$ low-energy bands are calculated through D. Xiao and Niu (2010)

$\Omega_{n}(\vec{k})=-2\sum_{n^{\prime}\neq n}Im\left[\langle
u_{n}|\frac{\partial H}{\partial k_{x}}|u_{n^{\prime}}\rangle\langle
u_{n^{\prime}}|\frac{\partial H}{\partial
k_{y}}|u_{n}\rangle/\left(E_{n^{\prime}}-E_{n}\right)^{2}\right],$ (11)

where, for every $k$-point, we obtain sums through all the neighboring
$n^{\prime}$ bands; $|u_{n}\rangle$ are the moire superlattice Bloch states,
and $E_{n}$ are the eigenvalues. Based on the Berry curvature, the valley
Chern number of the $n^{th}$ band is obtained from $C=\int{\rm
d^{2}}\vec{k}\,\,\Omega_{n}(\vec{k})/(2\pi)$ integrated in the moire Brillouin
zone. The Berry curvatures for the valence band for three values of $C_{AB}$
(before, after and at the transition) are compared in Fig. 1(b). Band closure
between the low energy and higher energy bands is observed between moire
Brillouin zone corners
$\tilde{K}^{\prime}_{1}=2\pi(-\frac{2}{3},\frac{1}{\sqrt{3}})$ and
$\tilde{K}_{1}=2\pi(\frac{1}{3},\frac{1}{\sqrt{3}})$, and is absent between
another set of $\tilde{K}^{\prime}_{2}=2\pi(\frac{2}{3},0)$ and
$\tilde{K}_{2}=2\pi(\frac{1}{3},\frac{1}{\sqrt{3}})$ in the mBZ. This
asymmetry between the initially equivalent $\tilde{K}_{1}$ and $\tilde{K}_{2}$
or $\tilde{K}^{\prime}_{1}$ and $\tilde{K}^{\prime}_{2}$ mini-valleys
indicates the rotational symmetry breaking introduced by the $C_{AB}$ term.

The band closure as a function of $C_{AB}$ indicates a possible topological
transition of the bands. Indeed, the valley Chern number $C=3$ with $C_{AB}=0$
has changed to $C=2$ after the band closure and re-opening. We present the low
energy bands valley Chern numbers in Fig. 1(c), where the valley Chern number
very near the band closure point ($C_{AB}$ = 0.007 eV) is $C=0$. The
conduction band valley Chern number undergoes a transition from $C=0$ to a
non-zero valley Chern number $C=-2$ as shown in Fig. 1 (c).

For $\xi=1$ hBN alignment and $N=2$ bilayer graphene (BG/BN) where we allow
$C_{AB}$ to change while keeping $\phi_{AB}$ = 0 we see that the valence band
exhibits a topological phase transition from $C=2$ to $C=1$ as shown in Fig.
1(d). However, for $N=1$ monolayer graphene aligned with hBN (G/BN) the
valence band retains the same valley Chern number of $C=1$ even with
increasing values of $C_{AB}$.

###  III.2 Broken rotational symmetry nematic local density of states

We had shown in Fig. 1 for a particular case of TLG/BN with $\xi=1$ and
$\Delta=0.01$ eV that the addition of an off-diagonal term in the moire
Hamiltonian can trigger a topological phase transition. Here we report
calculations of local density of states (LDOS) in order to distinguish how the
added off-diagonal terms can modify the LDOS profile $D(\vec{r},{E})$. In Fig.
2 we show the LDOS and density of states (DOS) for the low energy valence and
conduction bands for theree values of $C_{AB}$, namely 0, 0.007, and 0.01 eV.
The van Hove singularities in the DOS near the charge neutrality indicate the
presence of flat bands. The conduction band has a localization at BA stacking,
whereas the valence band has localization at AA stacking. With a finite off
diagonal term $C_{AB}\neq 0$ the vHS peak is broadened slightly but the
localization remains at the same stacking as illustrated in the LDOS plots. We
use the normalized LDOS defined as
$\tilde{D}(\vec{r},E)=D(\vec{r},E)$/max$[D(\vec{r},E)]$ in our plots and find
the influence of off-diagonal moire $C_{AB}$ term on the LDOS map through the
difference $\delta\tilde{D}(\vec{r},E)=\tilde{D}_{C_{AB}\neq
0}(\vec{r},E)-\tilde{D}_{C_{AB}=0}(\vec{r},E)$. This quantity shows how the
inclusion of a finite off-diagonal term leads to anisotropic LDOS profiles and
breaks the triangular rotational symmetry of the solutions.

###  III.3 Pattern shape dependent Chern number phase diagrams

The moire pattern shapes that mix inversion symmetric and asymmetric
components can be calibrated through the phase parameter $\phi$ that in turn
controls the shapes of the triangular moire patterns in the first harmonic
expansion, see the appendix for more details. In Fig. 1 we had noted the
change in the valley Chern numbers with $C_{AB}\neq 0$ and $\phi_{AB}=0$
combinations. However, there could be additional valley Chern number phases
when $\phi_{AB}$ is allowed to take a finite value. Indeed, we find this is
the case and we have presented in Fig. 3 the valley Chern number phase diagram
in the parameter space of $C_{AB}=0\sim 0.025$ eV and $\phi_{AB}=0\sim\pi$ for
TLG/BN, BG/BN and G/BN as concrete examples of $N$-chiral systems up to three
layers.

The results markedly depend on the orientation of the BN layer and for $\xi=1$
we generally obtain nonzero valley Chern number equal to $C=N$ while the
conduction band is trivial with $C=0$ for $N=1,2,3$. When we change the
orientation of the substrate moire pattern by setting $\xi=-1$ the conduction
band becomes non-trivial acquiring an opposite sign valley Chern number
$C=-N$, while the valence band acquires $C=0$. It is noted that the valley
Chern number of the valence/conduction bands are sensitive to both the off-
diagonal parameters $C_{AB}\neq 0$ and $\phi_{AB}\neq 0$ provided that the
former is sufficiently large and allows for additional valley Chern number
transitions for both initially trivial and non-trivial bands.

In the case of TLG/BN, $N=3$ with $\xi=1$, the valley Chern number of the
valence (conduction) band is $C=3\,(0)$ for $C_{AB}$ = 0. The valley Chern
number of the valence (conduction) band changed to $C=2\,(-2)$ for
sufficiently large $C_{AB}$ with $\phi_{AB}$ = 0. For $\xi=-1$, the valley
Chern number of valence (conduction) band is $C=0\,(-3)$ for $C_{AB}$ = 0, and
changed to $C=1\,(-2)$ with increasing value of $C_{AB}$.

Similarly, for BG/BN, $N=2$ with $\xi=1$ the valence (conduction) band has a
phase transition from $C=2\,(0)$ to $C=1\,(-1)$. However, with $\xi=-1$, the
valence/conduction band shows a transition from $C=0\,(-2)$ to $C=-1\,(-1)$.

For G/BN, $N=1$ with $\xi=1$, the valence/conduction band has a valley Chern
number $C=1\,(0)$. Even though the valley Chern number of the valence band did
not change with increasing $C_{AB}$, the conduction band valley Chern number
changed to $C=1$. For the case of $\xi=-1$, the valence band valley Chern
number remains unchanged while it does change for the conduction band when we
modify $C_{AB}$.

##  IV Summary and discussion

We have explored the phase diagram map for the valley Chern numbers of the low
energy valence and conduction bands of rhombohedral $N=1,2,3$ layer graphene
boron-nitride superlattices for different moire patterns. The intra-sublattice
diagonal moire patterns produces low energy bands whose valley Chern number
magnitudes are zero or proportional to layer number $N$. The absolute value of
the maximum valley Chern number followed the number of graphene layers
$N=1,2,3$ for all possible moire patterns within the first harmonic
approximation. For zero off-diagonal patterns we find that the valence band
has a valley Chern number equal to the number of layers $C=N$ when $\xi=1$,
while the conduction band is $C=-N$ when $\xi=-1$.

However, introduction of off-diagonal interlayer coherence moire pattern terms
captured through the magnitude $C_{AB}$ and phase $\phi_{AB}$ parameters
allows to trigger topological phase transitions giving rise to valley Chern
numbers that are smaller than the number of layers, a behavior that can be
traced mainly to the variations in the electron-hole secondary Chern weights
near the moire Brillouin zone boundaries. Thus, experimentally observed
quantum anomalous Hall effects in TLG/BN Chen  _et al._ (2019a) compatible
with $C=2$ rather than the expected $C=3$ from layer number could result from
rotational symmetry breaking interlayer coherence terms introduced by the
Coulomb interactions. Our model off-diagonal contributions could also result
from higher order corrections of the moire potentials mediated by interlayer
tunneling. Broken rotational symmetry in the mBZ upon inclusion of off-
diagonal terms in the Hamiltonian is evidenced from the Berry curvature
distribution in momentum space and the LDOS maps.

In summary, we have presented valley Chern number phase diagrams in the
parameter space that defines the off-diagonal moire pattern of the model
Hamiltonian of rhombohedral $N$-chiral multilayer graphene subject to moire
scalar and vector potentials. The vector potential moire patterns which are
often ignored in the low energy Hamiltonian models of $N$-chiral multilayer
graphene BN superlattices turned out to play a more prominent role than the
scalar potentials for triggering topological transitions to phases with valley
Chern numbers that are different to those dictated by layer number. Our
results points to the possibility of triggering topological phase transitions
by breaking the triangular rotational symmetry through pseudomagnetic field
vector potentials generated by moire strain patterns. These can result for
example from real strain fields whose bond distortions give rise to unequal
electron hopping probabilities to the neighboring atoms, or from virtual
effective strains due to high order interlayer electron hopping processes or
Coulomb interactions.

###### Acknowledgements.

D.A.G.G. acknowledges partial support from the Universidad de Antioquia,
Colombia, under initiative CODI ES84180154 Estrategia de sostenibilidad del
Grupo de Física Atómica y Molecular and projects CODI-251594 and 2019-24770.
Additionally, the authors are grateful to Professor Dr. Jorge Mahecha from the
University of Antioquia for his advice and assistance in this work. We
acknowledge financial support from the National Research Foundation of Korea
(NRF) through grants NRF-2020R1A2C3009142 for B.L.C., NRF-2020R1A5A1016518 for
Y. P., the Zhejiang Provincial Natural Science Foundation of China (Grant No.
LY19A040003) for J. H. S., and the Basic study and Urban convergence R&D
research fund of the University of Seoul (2019) for J. J. We acknowledge
computational support from KISTI through grant KSC-2020-CRE-0072.

## References

  * Bistritzer and MacDonald (2011) R. Bistritzer and A. H. MacDonald, Proceedings of the National Academy of Sciences 108, 12233 (2011).
  * Cao  _et al._ (2018a) Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Nature 556, 43 (2018a).
  * Cao  _et al._ (2018b) Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, _et al._ , Nature 556, 80 (2018b).
  * Wong  _et al._ (2015) D. Wong, Y. Wang, J. Jung, S. Pezzini, A. M. DaSilva, H.-Z. Tsai, H. S. Jung, R. Khajeh, Y. Kim, J. Lee, S. Kahn, S. Tollabimazraehno, H. Rasool, K. Watanabe, T. Taniguchi, A. Zettl, S. Adam, A. H. MacDonald, and M. F. Crommie, Phys. Rev. B 92, 155409 (2015).
  * Yankowitz  _et al._ (2019) M. Yankowitz, S. Chen, H. Polshyn, Y. Zhang, K. Watanabe, T. Taniguchi, D. Graf, A. F. Young, and C. R. Dean, Science 363, 1059 (2019).
  * Cao  _et al._ (2020) Y. Cao, D. Chowdhury, D. Rodan-Legrain, O. Rubies-Bigorda, K. Watanabe, T. Taniguchi, T. Senthil, and P. Jarillo-Herrero, Phys. Rev. Lett. 124, 076801 (2020).
  * Serlin  _et al._ (2019) M. Serlin, C. L. Tschirhart, H. Polshyn, Y. Zhang, J. Zhu, K. Watanabe, T. Taniguchi, L. Balents, and A. F. Young, Science (2019), 10.1126/science.aay5533.
  * Bultinck  _et al._ (2020) N. Bultinck, S. Chatterjee, and M. P. Zaletel, Phys. Rev. Lett. 124, 166601 (2020).
  * Sharpe  _et al._ (2019) A. L. Sharpe, E. J. Fox, A. W. Barnard, J. Finney, K. Watanabe, T. Taniguchi, M. A. Kastner, and D. Goldhaber-Gordon,  365, 605 (2019).
  * Kim  _et al._ (2017) K. Kim, A. DaSilva, S. Huang, B. Fallahazad, S. Larentis, T. Taniguchi, K. Watanabe, B. J. LeRoy, A. H. MacDonald, and E. Tutuc, Proceedings of the National Academy of Sciences 114, 3364 (2017).
  * Choi  _et al._ (2019) Y. Choi, J. Kemmer, Y. Peng, A. Thomson, H. Arora, R. Polski, Y. Zhang, H. Ren, J. Alicea, G. Refael, F. von Oppen, K. Watanabe, T. Taniguchi, and S. Nadj-Perge, Nature Physics 15, 1174 (2019).
  * Liu  _et al._ (2019) J. Liu, Z. Ma, J. Gao, and X. Dai, Phys. Rev. X 9, 031021 (2019).
  * Carr  _et al._ (2018) S. Carr, S. Fang, P. Jarillo-Herrero, and E. Kaxiras, Phys. Rev. B 98, 085144 (2018).
  * Chittari  _et al._ (2018a) B. L. Chittari, N. Leconte, S. Javvaji, and J. Jung, Electronic Structure 1, 015001 (2018a).
  * Leconte  _et al._ (2019) N. Leconte, S. Javvaji, J. An, and J. Jung, (2019), arXiv:1910.12805 [cond-mat.mes-hall] .
  * Zhao  _et al._ (2020) X.-J. Zhao, Y. Yang, D.-B. Zhang, and S.-H. Wei, Phys. Rev. Lett. 124, 086401 (2020).
  * Fidrysiak  _et al._ (2018) M. Fidrysiak, M. Zegrodnik, and J. Spałek, Phys. Rev. B 98, 085436 (2018).
  * Kang and Vafek (2019) J. Kang and O. Vafek, Phys. Rev. Lett. 122, 246401 (2019).
  * Koshino  _et al._ (2018) M. Koshino, N. F. Q. Yuan, T. Koretsune, M. Ochi, K. Kuroki, and L. Fu, Phys. Rev. X 8, 031087 (2018).
  * Tarnopolsky  _et al._ (2019) G. Tarnopolsky, A. J. Kruchkov, and A. Vishwanath, Phys. Rev. Lett. 122, 106405 (2019).
  * Po  _et al._ (2019) H. C. Po, L. Zou, T. Senthil, and A. Vishwanath, Phys. Rev. B 99, 195455 (2019).
  * You and Vishwanath (2019) Y.-Z. You and A. Vishwanath, npj Quantum Materials 4, 16 (2019).
  * Park  _et al._ (2019) M. J. Park, Y. Kim, G. Y. Cho, and S. Lee, Phys. Rev. Lett. 123, 216803 (2019).
  * Chichinadze  _et al._ (2020) D. V. Chichinadze, L. Classen, and A. V. Chubukov, Phys. Rev. B 101, 224513 (2020).
  * Shi  _et al._ (2020) H. Shi, Z. Zhan, Z. Qi, K. Huang, E. v. Veen, J. Á. Silva-Guillén, R. Zhang, P. Li, K. Xie, H. Ji, M. I. Katsnelson, S. Yuan, S. Qin, and Z. Zhang, Nature Communications 11, 371 (2020).
  * Crasto de Lima  _et al._ (2019) F. Crasto de Lima, R. H. Miwa, and E. Suárez Morell, Phys. Rev. B 100, 155421 (2019).
  * Ma  _et al._ (2020) C. Ma, Q. Wang, S. Mills, X. Chen, B. Deng, S. Yuan, C. Li, K. Watanabe, T. Taniguchi, X. Du, F. Zhang, and F. Xia, Nano Letters 20, 6076 (2020).
  * Hesp  _et al._ (2019) N. C. H. Hesp, I. Torre, D. Rodan-Legrain, P. Novelli, Y. Cao, S. Carr, S. Fang, P. Stepanov, D. Barcons-Ruiz, H. Herzig-Sheinfux, K. Watanabe, T. Taniguchi, D. K. Efetov, E. Kaxiras, P. Jarillo-Herrero, M. Polini, and F. H. L. Koppens, (2019), arXiv:1910.07893 [cond-mat.str-el] .
  * Fernandes and Venderbos (2020) R. M. Fernandes and J. W. F. Venderbos, Science Advances 6 (2020), 10.1126/sciadv.aba8834.
  * Stepanov  _et al._ (2020) P. Stepanov, I. Das, X. Lu, A. Fahimniya, K. Watanabe, T. Taniguchi, F. H. L. Koppens, J. Lischner, L. Levitov, and D. K. Efetov, Nature 583, 375 (2020).
  * Zhang  _et al._ (2020) Y. Zhang, Y. Gao, and D. Xiao, Phys. Rev. B 101, 041410 (2020).
  * Song  _et al._ (2019) Z. Song, Z. Wang, W. Shi, G. Li, C. Fang, and B. A. Bernevig, Phys. Rev. Lett. 123, 036401 (2019).
  * Chittari  _et al._ (2019) B. L. Chittari, G. Chen, Y. Zhang, F. Wang, and J. Jung, Phys. Rev. Lett. 122, 016401 (2019).
  * Chen  _et al._ (2019a) G. Chen, A. L. Sharpe, P. Gallagher, I. T. Rosen, E. J. Fox, L. Jiang, B. Lyu, H. Li, K. Watanabe, T. Taniguchi, J. Jung, Z. Shi, D. Goldhaber-Gordon, Y. Zhang, and F. Wang, Nature 572, 215 (2019a).
  * Chen  _et al._ (2019b) G. Chen, L. Jiang, S. Wu, B. Lyu, H. Li, B. L. Chittari, K. Watanabe, T. Taniguchi, Z. Shi, J. Jung, Y. Zhang, and F. Wang, Nature Physics 15, 237 (2019b).
  * Chen  _et al._ (2020) G. Chen, A. L. Sharpe, E. J. Fox, Y.-H. Zhang, S. Wang, L. Jiang, B. Lyu, H. Li, K. Watanabe, T. Taniguchi, Z. Shi, T. Senthil, D. Goldhaber-Gordon, Y. Zhang, and F. Wang, Nature 579, 56 (2020).
  * Yankowitz  _et al._ (2016) M. Yankowitz, K. Watanabe, T. Taniguchi, P. San-Jose, and B. J. LeRoy, Nature communications 7, 13168 (2016).
  * Lin and Ni (2019) X. Lin and J. Ni, Phys. Rev. B 100, 195413 (2019).
  * Yin  _et al._ (2019) L.-J. Yin, L.-J. Shi, S.-Y. Li, Y. Zhang, Z.-H. Guo, and L. He, Phys. Rev. Lett. 122, 146802 (2019).
  * Zhang and Mao (2020) Y.-H. Zhang and D. Mao, Phys. Rev. B 101, 035122 (2020).
  * Kim  _et al._ (2018) H. Kim, N. Leconte, B. L. Chittari, K. Watanabe, T. Taniguchi, A. H. MacDonald, J. Jung, and S. Jung, Nano Letters 18, 7732 (2018).
  * Repellin  _et al._ (2020) C. Repellin, Z. Dong, Y.-H. Zhang, and T. Senthil, Phys. Rev. Lett. 124, 187601 (2020).
  * Zhang and Senthil (2019) Y.-H. Zhang and T. Senthil, Phys. Rev. B 99, 205150 (2019).
  * Abouelkomsan  _et al._ (2020) A. Abouelkomsan, Z. Liu, and E. J. Bergholtz, Phys. Rev. Lett. 124, 106803 (2020).
  * Kindermann  _et al._ (2012) M. Kindermann, B. Uchoa, and D. L. Miller, Phys. Rev. B 86, 115415 (2012).
  * Bokdam  _et al._ (2014) M. Bokdam, T. Amlaki, G. Brocks, and P. J. Kelly, Phys. Rev. B 89, 201404 (2014).
  * San-Jose  _et al._ (2014) P. San-Jose, A. Gutiérrez-Rubio, M. Sturla, and F. Guinea, Phys. Rev. B 90, 075428 (2014).
  * Wallbank  _et al._ (2013) J. R. Wallbank, A. A. Patel, M. Mucha-Kruczyński, A. K. Geim, and V. I. Fal’Ko, Phys. Rev. B 87, 1 (2013).
  * Zhu  _et al._ (2018) G.-Y. Zhu, T. Xiang, and G.-M. Zhang, Science Bulletin 63, 1087 (2018).
  * Chebrolu  _et al._ (2019) N. R. Chebrolu, B. L. Chittari, and J. Jung, Phys. Rev. B 99, 235417 (2019).
  * Liu  _et al._ (2020) X. Liu, Z. Hao, E. Khalaf, J. Y. Lee, Y. Ronen, H. Yoo, D. Haei Najafabadi, K. Watanabe, T. Taniguchi, A. Vishwanath, and P. Kim, Nature 583, 221 (2020).
  * Lee  _et al._ (2019) J. Y. Lee, E. Khalaf, S. Liu, X. Liu, Z. Hao, P. Kim, and A. Vishwanath, Nature Communications 10, 5333 (2019).
  * Shen  _et al._ (2020) C. Shen, Y. Chu, Q. Wu, N. Li, S. Wang, Y. Zhao, J. Tang, J. Liu, J. Tian, K. Watanabe, T. Taniguchi, R. Yang, Z. Y. Meng, D. Shi, O. V. Yazyev, and G. Zhang, Nature Physics 16, 520 (2020).
  * Choi and Choi (2019) Y. W. Choi and H. J. Choi, Phys. Rev. B 100, 201402 (2019).
  * Koshino (2019) M. Koshino, Phys. Rev. B 99, 235406 (2019).
  * Li  _et al._ (2020) X. Li, F. Wu, and S. Das Sarma, Phys. Rev. B 101, 245436 (2020).
  * Zhang  _et al._ (2019a) Y.-H. Zhang, D. Mao, Y. Cao, P. Jarillo-Herrero, and T. Senthil, Phys. Rev. B 99, 075127 (2019a).
  * Javvaji  _et al._ (2020) S. Javvaji, J.-H. Sun, and J. Jung, Phys. Rev. B 101, 125411 (2020).
  * kun Shi  _et al._ (2020) L. kun Shi, J. Ma, and J. C. W. Song, 2D Materials 7, 015028 (2020).
  * Pan  _et al._ (2018) Y. Pan, S. Fölsch, Y. Nie, D. Waters, Y.-C. Lin, B. Jariwala, K. Zhang, K. Cho, J. A. Robinson, and R. M. Feenstra, Nano letters 18, 1849 (2018).
  * Gong  _et al._ (2014) Y. Gong, J. Lin, X. Wang, G. Shi, S. Lei, Z. Lin, X. Zou, G. Ye, R. Vajtai, B. I. Yakobson, _et al._ , Nature materials 13, 1135 (2014).
  * Huang  _et al._ (2014) C. Huang, S. Wu, A. M. Sanchez, J. J. Peters, R. Beanland, J. S. Ross, P. Rivera, W. Yao, D. H. Cobden, and X. Xu, Nature materials 13, 1096 (2014).
  * Duan  _et al._ (2014) X. Duan, C. Wang, J. C. Shaw, R. Cheng, Y. Chen, H. Li, X. Wu, Y. Tang, Q. Zhang, A. Pan, _et al._ , Nature nanotechnology 9, 1024 (2014).
  * Song and Gabor (2018) J. C. W. Song and N. M. Gabor, Nature Nanotechnology 13, 986 (2018).
  * Naik and Jain (2018) M. H. Naik and M. Jain, Phys. Rev. Lett. 121, 266401 (2018).
  * Wu  _et al._ (2018) F. Wu, T. Lovorn, E. Tutuc, and A. H. MacDonald, Phys. Rev. Lett. 121, 026402 (2018).
  * Wu  _et al._ (2019) F. Wu, T. Lovorn, E. Tutuc, I. Martin, and A. H. MacDonald, Phys. Rev. Lett. 122, 086402 (2019).
  * Jin  _et al._ (2019a) C. Jin, E. C. Regan, A. Yan, M. Iqbal Bakti Utama, D. Wang, S. Zhao, Y. Qin, S. Yang, Z. Zheng, S. Shi, K. Watanabe, T. Taniguchi, S. Tongay, A. Zettl, and F. Wang, Nature 567, 76 (2019a).
  * Seyler  _et al._ (2019) K. L. Seyler, P. Rivera, H. Yu, N. P. Wilson, E. L. Ray, D. G. Mandrus, J. Yan, W. Yao, and X. Xu, Nature 567, 66 (2019).
  * Bi and Fu (2019) Z. Bi and L. Fu, (2019), arXiv:1911.04493 [cond-mat.str-el] .
  * Zhang  _et al._ (2019b) L. Zhang, Z. Zhang, F. Wu, D. Wang, R. Gogna, S. Hou, K. Watanabe, T. Taniguchi, K. Kulkarni, T. Kuo, S. R. Forrest, and H. Deng, (2019b), arXiv:1911.10069 [cond-mat.mes-hall] .
  * Yu  _et al._ (2019) H. Yu, M. Chen, and W. Yao, National Science Review (2019), 10.1093/nsr/nwz117.
  * Sushko  _et al._ (2019) A. Sushko, K. D. Greve, T. I. Andersen, G. Scuri, Y. Zhou, J. Sung, K. Watanabe, T. Taniguchi, P. Kim, H. Park, and M. D. Lukin, (2019), arXiv:1912.07446 [cond-mat.mes-hall] .
  * Tang  _et al._ (2019) Y. Tang, L. Li, T. Li, Y. Xu, S. Liu, K. Barmak, K. Watanabe, T. Taniguchi, A. H. MacDonald, J. Shan, and K. F. Mak, (2019), arXiv:1910.08673 [cond-mat.mtrl-sci] .
  * Jin  _et al._ (2019b) C. Jin, E. C. Regan, D. Wang, M. Iqbal Bakti Utama, C.-S. Yang, J. Cain, Y. Qin, Y. Shen, Z. Zheng, K. Watanabe, T. Taniguchi, S. Tongay, A. Zettl, and F. Wang, Nature Physics 15, 1140 (2019b).
  * An  _et al._ (2019) L. An, X. Cai, M. Huang, Z. Wu, J. Lin, Z. Ying, Z. Ye, X. Feng, and N. Wang, (2019), arXiv:1907.03966 [cond-mat.mes-hall] .
  * Chittari  _et al._ (2018b) B. L. Chittari, N. Leconte, S. Javvaji, and J. Jung, Electronic Structure 1, 015001 (2018b).
  * (78) J. Jung, Unpublished .
  * Jung and MacDonald (2013) J. Jung and A. H. MacDonald, Phys. Rev. B 87, 195450 (2013).
  * Jung  _et al._ (2014) J. Jung, A. Raoux, Z. Qiao, and A. H. MacDonald, Phys. Rev. B 89, 205414 (2014).
  * Jung  _et al._ (2017) J. Jung, E. Laksono, A. M. DaSilva, A. H. MacDonald, M. Mucha-Kruczyński, and S. Adam, Phys. Rev. B 96, 085442 (2017).
  * D. Xiao and Niu (2010) M.-C. C. D. Xiao and Q. Niu, Rev. Mod. Phys. 82, 1959 (2010).

##  Appendix A Inversion symmetric and asymmetric moire patterns

Here we discuss in more detail the breakdown of the moire patterns into
inversion symmetric and asymmetric terms Wallbank  _et al._ (2013) for the
diagonal scalar moire pattern in Eq. (5) and off-diagonal vector potential-
like term in Eq. (8). Illustrations of the diagonal and off-diagonal moire
patterns for different phase angles are in Fig. 4.

Figure 4: Intralayer Hamiltonian elements as a function of sliding vector
$\vec{d}$ for different phase angles $\varphi_{ii}$ and $\varphi_{ij}$. The
black hexagon represents the real space presentation of moire supercell. In
the real space the inversion symmetric potential for the diagonal term
$H_{ii}$ is associated with $\varphi_{ii}=n\pi$ but the inversion asymmetric
potential for the off-diagonal term $H_{ij}$ is associated with
$\varphi_{ij}=(2n+1)\pi/2$ where $N$ is an integer. Figure 5: Dependence of
the scalar moire potential term $M(\vec{r})=2C\,{\rm
Re}\left[f(\vec{r})e^{i\phi}\right]$ on the local stacking vector $\vec{d}$
and the phase angle $\phi$. The local stacking coordinate vector and real
space position is related through
$\vec{d}(\vec{r})=\varepsilon\vec{r}+\theta\hat{z}\times\vec{r}$ in the small
angle approximation. The moire pattern in Eq. (A) will be symmetric for
$\phi_{ii}=n\pi$ and asymmetric when $\phi_{ii}=(2n+1)\pi/2$ for integer $n$
values, and a combination of both for intermediate phase angles.

The diagonal element of the moire potential in Eq. (5) involving a scalar
function $V_{AA/BB}^{M}(\vec{r})$ is defined in terms of
$C_{AA/BB}=C_{ii},~{}\phi_{AA/BB}=\phi_{ii}$ and for $\xi=1$ we have

$\displaystyle M_{ii}(\vec{r})$ $\displaystyle=$ $\displaystyle 2C_{ii}\,{\rm
Re}\left[f(\vec{r})e^{i\phi_{ii}}\right]$ $\displaystyle=$ $\displaystyle
C_{ii}\left(f_{1}(\vec{r})\cos{\phi_{ii}}+f_{2}(\vec{r})\sin{\phi_{ii}}\right),$

with $f_{1}(\vec{r})=\sum_{m=1}^{6}e^{i{\tilde{G}}_{m}\vec{r}}$ (symmetric
function) and
$f_{2}(\vec{r})=i\sum_{m=1}^{6}(-1)^{m-1}e^{i{\tilde{G}}_{m}\vec{r}}$
(antisymmetric function), where we use six moire reciprocal lattices
$\tilde{G}_{m=1..6}=\hat{R}_{2\pi(m-1)/3}{\tilde{G}_{1}}$ successively rotated
by 60∘ as introduced in the main text. Hence, the moire pattern in Eq. (A)
will be symmetric for $\phi_{ii}=n\pi$ and asymmetric when
$\phi_{ii}=(2n+1)\pi/2$ for integer $N$ values, and a combination of both for
intermediate phase angles. In Fig. 5, we illustrate the $M(\vec{r})$ patterns
in real space for different values of the angle $\phi_{ii}$.

The off-diagonal $H_{ij}(\vec{r})$ term with $i\neq j$ given in Eq. (8) has a
vector potential term $\vec{A}(\vec{r})$ defined in Eq. (A) that can be
represented by its magnitude and orientation as in Fig. 6,

$\displaystyle A^{\xi}(\vec{r})$ $\displaystyle=$ $\displaystyle
V^{M}_{AB}\vec{\nabla}_{\vec{r}}{\rm Re}\left[e^{i\phi_{AB}}f(\vec{r})\right]$
$\displaystyle=$ $\displaystyle V^{M}_{AB}\vec{\nabla}_{\vec{r}}{\rm
Re}\left[\sum_{m=1}^{6}e^{i\xi{{\tilde{G}_{m}}}\cdot{\vec{r}}+\phi_{AB}}\left(1+\frac{\left(1+(-1)^{m}\right)}{2}\right)\right].$

Figure 6: Dependence of the vector moire potential $\vec{A}(\vec{r})$ on phase
angle $\phi_{AB}$ that decide the relative presence of inversion symmetric and
asymmetric pattern components as in Fig. 5 for the diagonal terms. The real
and imaginary parts of the inter-sub lattice tunneling sets the direction of
the pseudo magnetic vector potential (arrows) which strongly depends on the
phase angle $\phi_{AB}$.


# Collaborative Label Correction via
Entropy Thresholding

Hao Wu, Jiangchao Yao, Jiajie Wang, Yinru Chen, Ya Zhang, Yanfeng Wang
Cooperative Medianet Innovation Center
Shanghai Jiao Tong University
{howiethepeanut, sunarker, ww1024, chenyinru990504, ya_zhang,
<EMAIL_ADDRESS>
###### Abstract

Deep neural networks (DNNs) have the capacity to fit extremely noisy labels
nonetheless they tend to learn data with clean labels first and then memorize
those with noisy labels. We examine this behavior in light of the Shannon
entropy of the predictions and demonstrate the low entropy predictions
determined by a given threshold are much more reliable as the supervision than
the original noisy labels. It also shows the advantage in maintaining more
training samples than previous methods. Then, we power this entropy criterion
with the Collaborative Label Correction (CLC) framework to further avoid
undesired local minimums of the single network. A range of experiments have
been conducted on multiple benchmarks with both synthetic and real-world
settings. Extensive results indicate that our CLC outperforms several state-
of-the-art methods.

##  I Introduction

Large-scale supervised training datasets have significantly driven the success
of DNNs in multiple fields [1, 2, 3]. However, accurate labels provided by the
human experts are not always feasible in the real-world applications.
Alternative ways of obtaining annotations [4, 5] inevitably introduce label
noise to the dataset. As DNNs are capable of fitting extremely noisy labels
[6], training DNNs robustly with noisy supervision [7, 8, 9, 10] has been an
important and vibrant area of research. Recent works have proposed two
following popular ways for learning with noisy labels. One way is to find the
clean instances out of the noisy ones for training via _sample selection_. For
example, Decouple [11] selects samples on which the two models make different
predictions. Co-teaching [12] uses two models to select small loss samples as
supervision for each other. The other way is by leveraging the predictions of
the model to rectify the noisy labels, i.e., _label correction_. For instance,
Distillation [13] combines the predictions of the teacher net with the
original labels as supervision for the student net. Co-distillation [14, 15]
collaboratively distills knowledge between the two networks.

However, for current label correction methods, the predictions of the model
are rigidly utilized without verifying their validity. The performance still
suffers from the influence of the inaccurate supervision.

Figure 1: The framework of Collaborative Label Correction. We maintain two
networks $f$ and $g$. On the panels below, the x-axis represents ten classes
in CIFAR-10 and the y-axis represents the entropy of the predictions. We plot
the entropy of predictions in a mini-batch as red crosses which indicate
correct predictions and green crosses which indicate incorrect predictions.

On the other hand, although sample selection methods use different criteria to
guarantee the label precision of the remaining samples, the increase of noise
ratio will result in the decreasing number of samples. Thus sample selection
methods are naturally forced to discard training samples that could be
potentially utilized to boost the performance of the classifier. This presents
a challenge to learning with noisy labels—can we leverage more reliable
supervisions from the noisy dataset while maintain sufficient samples for
training?

To tackle the issues of both ways, we first provide a new method that selects
trustworthy predictions to correct the noisy labels, and then propose the
Collaborative Label Correction (CLC) framework shown in Fig. 1. As can be
seen, two models utilize their low entropy predictions determined by a given
threshold to correct labels for each other. We empirically demonstrate the
entropy criterion that stems from the memorization effects [16], can guarantee
the validity of the predictions and thus avoid the poor quality predictions.
Treating the low entropy predictions as pseudo labels, CLC manages to achieve
better or comparable label precision with sample selection methods. Moreover,
CLC shows advantages in keeping much more training samples compared to sample
selection methods which averts the loss of data. Simultaneously, CLC greatly
benefits from the dual network structure that avoids undesired local minimums
of the single network. Our whole model is trained in an end-to-end manner.

The main contributions are summarized as follows:

  * •

We propose entropy thresholding to select the low entropy predictions as the
supervision of the classifier. It guarantees the validity of low entropy
predictions leveraging the memorization effects [16] and simultaneously
maintains sufficient training samples.

  * •

We propose a Collaborative Label Correction framework that incorporates the
above advantages of the entropy criterion. Moreover, the dual network
structure of CLC avoids undesired local minimums of the single network.

  * •

We empirically verify the proposed method on on multiple benchmarks with both
synthetic and real-world settings. Comprehensive experiments demonstrate that
CLC outperforms several state-of-the-art methods.

##  II Collaborative Label Correction

For noisy supervision, consider a set
$\mathcal{D}=\\{(x_{n},y_{n})\\}^{N}_{n=1}$ of $N$ samples from $c$ classes,
where $x_{n}$ denotes an image and $y_{n}\in\\{0,1\\}^{c}$ is the
corresponding noisy label. To address the issue of inaccurate predictions in
label correction methods and loss of data in sample selection methods, our
goal is to select the most trustworthy predictions as pseudo labels and retain
as many reliable predictions as possible in the training.

###  II-A Memorization Effects in Light of Shannon Entropy

In this section, we will demonstrate that entropy thresholding can help us
find reliable predictions out of all model predictions. In a standard training
process, DNNs tend to learn samples with clean labels first and then memorize
those with noisy labels, which is regarded as the memorization effects [16].
We examine this process in light of Shannon entropy. For a given input $x$ and
a network $f$, $p_{f}=f(x)$ is denoted as a probabilistic prediction after the
softmax output. Its Shannon entropy is given by
$H(f,x)=-\sum_{i=1}^{c}p_{fi}\log p_{fi}$. Generally, at the beginning of
training the model does not learn anything thus the prediction distribution is
uniformly distributed which yields the highest entropy. While in an advanced
stage of training, the model fits all the labels thus the predictions place
all probability in one class for every sample which yields the lowest entropy.
To visualize the prediction entropy in the training process, we conduct the
toy experiments on CIFAR-10 with the synthetic symmetric noise [17] and depict
the corresponding curve on the left panel of Fig. 2. Besides, we plot the
variation of the entropy for both correct predictions and incorrect
predictions, since we have the ground-truth labels. As can be seen in the
training process, the entropy of predictions generally decreases to zero. The
entropy of the correct predictions decreases fast while the entropy of the
incorrect predictions decreases at a rather slow pace. The entropy gap between
the correct and the incorrect predictions becomes larger at first then narrows
down which matches the behavior of the DNNs stated in the memorization effects
[16].

Figure 2: Left: The evolution of the average entropy of correct predictions,
incorrect predictions and all predictions through the training process. Right:
Number of samples with the low entropy predictions determined by a given
threshold vs. number of epochs.

If we only analyze the low entropy predictions under a given threshold $1.0$,
we will obtain the curve on the right panel of Fig. 2. We can see that the
number of the correct predictions increases sharply and persists at a high
level. Conversely, the number of the incorrect predictions remains at a low
level for a period and then increases as the training proceeds. This
demonstrates the low entropy predictions are trustworthy before the model
gradually fits noise. Utilizing this behavior, we propose entropy thresholding
to discriminate the reliable low entropy predictions from the unreliable high
entropy predictions. We further leverage the low entropy predictions to
robustly train our classifier.

###  II-B The Collaborative Label Correction Framework

In this section, we propose the Collaborative Label Correction (CLC) framework
where two models collaboratively utilize their low entropy predictions to
correct the original labels for each other. For a given threshold $\Gamma$, we
firstly obtain the low entropy set as well as the high entropy set from a
given mini-batch $\overline{\mathcal{D}}$ out of $\mathcal{D}$:

$\displaystyle\overline{\mathcal{D}}_{f,low}=\\{(x_{n},p_{fn})|H(f,\overline{\mathcal{D}})\leq\Gamma\\},$
(1)
$\displaystyle\overline{\mathcal{D}}_{g,low}=\\{(x_{n},p_{gn})|H(g,\overline{\mathcal{D}})\leq\Gamma\\},$
(2)
$\displaystyle\overline{\mathcal{D}}_{f,high}=\\{(x_{n},y_{n})|H(f,\overline{\mathcal{D}})>\Gamma\\},$
(3)
$\displaystyle\overline{\mathcal{D}}_{g,high}=\\{(x_{n},y_{n})|H(g,\overline{\mathcal{D}})>\Gamma\\},$
(4)

with respect to the predictions from the two networks $f$ and $g$. Note that
we only correct labels with model predictions in the low entropy set and leave
the labels unchanged in the high entropy set. The objective functions for the
two networks $f$ and $g$ are defined as follows:

$\displaystyle L(f,\overline{\mathcal{D}})=$ $\displaystyle
L_{ce}(f,\overline{\mathcal{D}}_{g,low})+\alpha
H(f,\overline{\mathcal{D}}_{f,low})+$ $\displaystyle+$ $\displaystyle\beta
L_{ce}(f,\overline{\mathcal{D}}_{g,high}),$ (5) $\displaystyle
L(g,\overline{\mathcal{D}})=$ $\displaystyle
L_{ce}(g,\overline{\mathcal{D}}_{f,low})+\alpha
H(g,\overline{\mathcal{D}}_{g,low})+$ $\displaystyle+$ $\displaystyle\beta
L_{ce}(g,\overline{\mathcal{D}}_{f,high}),$ (6)

Algorithm 1 The CLC Algorithm

0: A noisy dataset $\mathcal{D}$, two networks $f$ and $g$, $\Gamma$,
$\alpha$, $\beta$

1: for epoch $i=\text{StartEpoch}$ to MaxEpoch do

2: for batch $j=1$ to $\frac{|\mathcal{D}|}{\text{BatchSize}}$ do

3: Calculate $H(f,\overline{\mathcal{D}})$

4: Calculate $H(g,\overline{\mathcal{D}})$

5: Obtain $\overline{\mathcal{D}}_{f,low}$ and
$\overline{\mathcal{D}}_{f,high}$ by Eq. (1) and Eq. (3)

6: Obtain $\overline{\mathcal{D}}_{g,low}$ and
$\overline{\mathcal{D}}_{g,high}$ by Eq. (2) and Eq. (4)

7: Update $f$ by optimizing $L(f,\overline{\mathcal{D}})$ in Eq. (5)

8: Update $g$ by optimizing $L(g,\overline{\mathcal{D}})$ in Eq. (6)

9: end for

10: end for

where $L_{ce}$ is the cross-entropy loss. In the first term of Eq. 5, the
network $f$ is supervised by the low entropy predictions from network $g$,
which is the key module of this paper. The second term of Eq. 5 is a
regularization term to prevent the output of the network $f$ from degenerating
to the uniform distribution. We minimize this term in order to make $f$
maintain the already reliable predictions. For the third part of Eq. 5, it is
an auxiliary term that guarantees $f$ to use the original labels in the early
phase, since the number of low entropy predictions in this stage is small. As
the training proceeds, the number of low entropy predictions will grow and
conversely the third term will be squeezed out. Like [15], we empirically
assign weights to $\alpha$ and $\beta$ to save human labor. The training
process is summarized in Algorithm 1.

###  II-C Prediction Selection vs. Sample Selection

In this section, we present the advantage of prediction selection in CLC over
sample selection. When treating all model predictions as pseudo labels,
prediction selection which selects samples with the correct pseudo labels
seems similar to sample selection which selects samples with correct original
labels. However, for sample selection, the optimal situation is that all of
the clean instances in the dataset are selected and the noisy instances are
discarded, thus the upper bound of the number of selected samples is equal to
the amount of clean samples collected in the original dataset. Conversely, for
prediction selection, we can easily obtain a suboptimal model that simply fits
all the labels according to the memorization effects [16]. Thus the upper
bound of the number of correct pseudo labels is no less than the amount of the
correct original labels. The optimal situation is that a well generalized
model corrects all the labels and the least upper bound is the total amount of
samples. Namely, prediction selection has the potential to maintain all the
training samples including the originally noisy ones. Conversely, sample
selection is forced to discard training samples that could be potentially
leveraged to enhance the performance of the classifier.

###  II-D Dual Network vs. Single Network

We choose to maintain two networks collaboratively for CLC. Naturally a
question would occur that if we can use only one network to learn from its own
low entropy predictions, namely Self-paced Label Correction (SLC). However,
there reside two problems as follows. One problem is that although entropy
thresholding can guarantee the validity of most predictions, the few errors
from the prediction bias can not be filtered by their own predictor. Thus the
errors will accumulate through training. This will cause the self-paced
network to be stuck in _undesired local minimums_ as soon as all of the labels
are corrected by the low entropy predictions. CLC naturally addresses this
problem as two networks have the potential to filter out the biases residing
in each other. The other problem is that stochastic tweaking may cause some of
the correct low entropy predictions to go wrong even though we explicitly
encourage the model to minimize the entropy of the already low entropy
predictions. These alterations are presumably irreversible. Take “gene
mutation” as a supportive example that we have only one good genotype
corresponding to the correct category and much more bad genotypes
corresponding to the incorrect categories. The rate of mutation will be much
higher than that of reverse mutation. Since one model is not aware of the
prediction errors of its own, the accumulation of the “mutations” may cause
SLC to collapse into _undesired local minimums_ where all predictions are
random low entropy predictions. For CLC, incorrect predictions caused by non-
trivial tweaking will not be directly used as supervision for the original
predictor. Thus CLC avoids the accumulation of the “mutations” and enhances
the stability of label correction.

##  III Experiments

TABLE I: Summary of the 6 datasets used in the experiments. dataset | # of training | # of testing | # of class | data type | noise type
---|---|---|---|---|---
CIFAR-10 | 50,000 | 10,000 | 10 | image | synthetic
CIFAR-100 | 50,000 | 10,000 | 100 | image | synthetic
Clothing1M | 1M | 74k | 14 | image | real-world
Food-101N | 310k | 25,250 | 101 | image | real-world
20 Newsgroups | 11,314 | 7,532 | 20 | text | synthetic
TREC | 5,452 | 500 | 6 | text | synthetic

###  III-A Datasets and Baselines

TABLE II: Average test accuracy (%) on CIFAR-10 & CIFAR-100 over the last ten epochs. | CIFAR-10 | CIFAR-100
---|---|---
| pairwise | asymmetric | symmetric | pairwise | asymmetric | symmetric
setting | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12
noise ratio | 0.2 | 0.45 | 0.2 | 0.45 | 0.2 | 0.5 | 0.2 | 0.45 | 0.2 | 0.45 | 0.2 | 0.5
Standard | 76.54 | 49.73 | 82.86 | 69.52 | 76.71 | 49.91 | 50.47 | 32.51 | 50.79 | 31.47 | 48.63 | 25.44
Bootstrap | 76.19 | 49.75 | 82.90 | 70.23 | 76.66 | 48.83 | 51.01 | 31.17 | 50.35 | 32.42 | 47.87 | 24.74
Forward | 78.06 | 58.69 | 83.29 | 69.59 | 77.56 | 51.70 | 52.67 | 30.14 | 54.99 | 28.41 | 47.00 | 25.94
Decouple | 77.30 | 49.23 | 83.53 | 70.24 | 78.03 | 48.97 | 51.28 | 31.49 | 50.50 | 31.72 | 46.19 | 23.29
MentorNet | 80.32 | 58.67 | 83.62 | 70.28 | 80.73 | 68.70 | 50.89 | 32.26 | 50.27 | 32.35 | 52.12 | 37.96
Co-teaching | 83.24 | 72.74 | 85.12 | 76.02 | 82.09 | 74.06 | 54.74 | 34.08 | 52.81 | 34.77 | 53.84 | 41.34
Co-distillation | 81.36 | 51.87 | 85.28 | 72.15 | 81.49 | 56.76 | 56.40 | 34.93 | 55.27 | 35.27 | 54.36 | 32.41
CLC | 88.97 | 84.26 | 89.71 | 78.29 | 85.73 | 80.97 | 62.12 | 54.62 | 62.37 | 55.46 | 57.45 | 41.17

The general information of the datasets can be found in Table I. For CIFAR10
and CIFAR100 [18], pairwise noise [12], asymmetric noise [19] and symmetric
noise [17] are injected. We use $r$ to denote the noise ratio. Images in
Clothing1M [20] are crawled from online shopping websites with labels
generated by surrounding texts. The overall label accuracy is 61.54%.
Food-101N [21] is generated through Google, Bing, Yelp and TripAdvisor. The
overall label accuracy is estimated to be 80%. 20 Newsgroups [20newsgroup] is
a collection of 20 newsgroups and TREC [22] is a collection of questions from
6 coarse classes. Pairwise noise and symmetric noise are injected. We compare
CLC with the following methods: Standard, which directly trains a vanilla
classifier on noisy datasets; Bootstrap [23], which linearly combines the
model predictions and original labels; Forward [19], which uses a noise
transition matrix to modify model predictions; Decouple [11], which updates
the parameters when two models disagree with each other; MentorNet [24]. We
use self-paced MentorNet that the model learns with its small loss samples;
Co-teaching [12], which uses two models to select small loss samples for each
other.; Co-distillation [14, 15], which uses two models to distill knowledge
for each other.

###  III-B Implementation

For CIFAR-10 and CIFAR-100, we align the implementation in [12] that a 9-layer
CNN is used as the classifier. We generally deploy the Adam optimizer [25] and
set the learning rate to 0.001. We set the batch size to 128 and run 200
epochs in total. For 20 Newsgroups and TREC, a 3-layer MLP is used and the
dimension of the hidden layers is set to 300. We use the pre-trained word
embeddings from GloVe [26]. For Clothing1M and Food-101N, ResNet-50 [2] is
used. Images are resized to 256 with respect to shorter sides. We perform
random crop of 224$\times$224, random flip, brightness, contrast and
saturation. We set the batch size to 64 and run 15 epochs on Clothing1M and 30
epochs on Food-101N. To be fair, all methods are pre-trained for several
epochs as warming-up following [19, 15, 12]. Specifically the models are
trained as Standard for 2 epochs on Clothing1M and 10 epochs on all other
datasets. For Forward, the pre-trained model is used to estimate the
transition matrix [19] on all datasets except for Clothing1M where we use the
normalized ground-truth confusion matrix provided in [20]. For MentorNet and
Co-teaching, the noise ratio $r$ is required as side information [12]. For
CLC, we empirically use the average entropy of the pre-trained model
predictions as $\Gamma$. We empirically set $\alpha=0.1$ and $\beta=0.5$. We
use test accuracy to measure the performances. For sample selection methods
along with CLC, we also report the number of selected samples and their label
precision on the CIFAR datasets as we have the ground-truth labels.

Figure 3: Left: Illustration of test accuracy vs. number of epochs on CIFAR-10
with pairwise noise. ($r=0.45$). Right: Illustration of test accuracy vs.
number of epochs on CIFAR-100 with pairwise noise. ($r=0.45$).

###  III-C Results on CIFAR-10 and CIFAR-100

Table II summarizes the results of all methods on the CIFAR datasets. Our CLC
achieves the best performance in most of the settings. Specifically for high-
level pairwise noise ($r=0.45$), CLC outperforms the best baseline by 11.52%
on CIFAR-10 and 19.69% on CIFAR-100 even impressively surpassing all baselines
under low-level noise. For asymmetric noise, CLC manages to outperform the
best baseline by 20.19% when $r=0.45$ on CIFAR-100 and again surpass all
baselines under low-level noise. In Fig. 3, we trace the test accuracy of all
baselines and CLC under high-level pairwise noise on CIFAR-10 on the left
panel. For all baselines, the test accuracy increases first and then decreases
as the training proceeds which matches the memorization effects.

Figure 4: Left: Comparison of CLC and SLC on CIFAR-10 with pairwise noise
($r=0.45$). Right: Comparison of CLC and SLC on CIFAR-10 with symmetric noise
($r=0.5$).

However, the accuracy of CLC keeps increasing and persists at a high level
after a long time of training, which demonstrates the robustness of CLC under
extreme label noise. On the right panel, we trace the test accuracy on
CIFAR-100 with high-level pairwise noise. Similarly, CLC keeps improving for
around 60 epochs and persists at a high level till the end which indicates CLC
is effective in dealing with multiple classes.

Figure 5: Average number of selected samples and average label precision (%)
over the last ten epochs on the CIFAR datasets of 12 different settings. The
indices of x-axis corresponds to the settings in Table II.

In Fig. 5, we show the average number of selected samples and average label
precision. Decouple only selects a few samples and the label precision is low.
MentorNet and Co-teaching are programmed to discard a proportion $r$ of
samples thus almost half of the samples are discarded when $r$ is large. Co-
teaching achieves better label precision compared to MentorNet taking
advantage of collaborative learning. CLC retains all 50000 of the samples as
well as achieving comparable or even better label precision in most of the
settings. Setting 12 is the only one setting where the label precision of CLC
is much lower than that of Co-teaching. However, CLC maintains twice the
number of samples than Co-teaching. Thus the relatively low label precision is
compensated by the number advantage.

In Fig. 4 we compare CLC with SLC. For high-level pairwise noise, the test
accuracy of SLC is 78.35%, which is better than all baselines. Moreover, SLC
retains all training samples and shows robustness against label noise which
resembles the behavior of CLC. However, CLC addresses the prediction bias in
SLC by collaboratively correcting labels for each, therefore CLC outperforms
SLC by 5.91%. For high-level symmetric noise, SLC works fine in the early
stage and manages to correct 49785 samples out of 50000 samples with label
precision of 75.85% at the 19th epoch. Then, SLC degenerates rapidly within 6
epochs due to the stochastic tweaking and is eventually stuck in an undesired
local minimum where all predictions are random low entropy predictions. For
CLC, the dual model structure is naturally robust to the stochastic tweaking
and can be trained stably.

In Fig. 6, we illustrate the confusion matrices with predictions of CLC models
and the noisy labels in setting 3 and 4. The ground-truth transition is given
on the left panel in each setting. Although CLC does not explicitly model the
noise transition, the induced confusion matrices closely resemble the ground-
truth transition. Specifically labels of the 4th category “cat” and 6th
category “dog” randomly flip between each other forming a hard noise structure
visualized as a block (bounded in a red box). Nonetheless the same block can
be observed in the confusion matrix. This demonstrates that CLC is capable of
capturing hard noise patterns under extreme label noise.

Figure 6: The ground-truth transition matrices and the confusion matrices of
CLC models on CIFAR-10. The horizontal indices denote the noisy classes while
the vertical indices denote the ground-truth classes or predicted classes.

###  III-D Results on Clothing1M and Food-101N

Table III summarizes the results on two real-world datasets. For Clothing1M,
Decouple has a poor performance. Co-teaching and MentorNet adopt the same
criterion while Co-teaching leverages two models to cope with the sample
selection bias and performs better. Maintaining two models, Co-distillation
shows comparable results with Co-teaching. Among all baselines, Forward
achieves the best performance with the use of the ground-truth transition
matrix. However CLC outperforms Forward by 2.16% without using any side
information. For Food-101N, Bootstrap degenerates the performance with the
unverified predictions. Without the knowledge of the ground-truth transition
matrix, Forward improves little compared to Standard. Maintaining two
networks, Co-distillation shows comparable results with Co-teaching . However,
CLC manages to outperform the best baseline by 1.46%.

In the top panel of Fig. 7 we present several examples of label corrections of
CLC between two similar classes “dress” and “vest”. Both “dress” and “vest”
cover upper body and lack sleeves while only “dress” covers lower body. CLC
successfully captures the difference between the two categories and correct
the noisy labels with its own predictions. In the bottom panel of Fig. 7 we
present several examples between two similar classes “cake” and “mousse”. They
contain the same ingredient but differ in shapes and CLC manages to rectify
the incorrect labels. These examples prove that samples with originally noisy
labels are potentially useful.

TABLE III: Average test accuracy (%) on Clothing1M and Food-101N over the last five epochs. method | Clothing1M | Food-101N
---|---|---
Standard | 66.14 | 75.45
Bootstrap | 66.70 | 74.75
Forward | 67.70 | 75.92
Decouple | 64.44 | 72.47
MentorNet | 60.00 | 76.93
Co-teaching | 67.30 | 77.87
Co-distillation | 67.23 | 77.31
CLC | 69.86 | 79.33
Figure 7: Examples of label corrections of CLC on Clothing1M and Food-101N.

###  III-E Results on 20 Newsgroups and TREC

The results of all methods on 20 Newsgroups and TREC are reported in Table IV.
Generally, the performance of Standard model is poor especially when noise
ratio is high. For label correction methods Bootstrap and Co-distillation,
since they heavily rely on the quality of the model predictions, only little
improvement is made compared to Standard and failures occur in multiple
settings. Forward also leverages the Standard model to estimate the transition
matrix thus the performance suffers from the inaccurate estimations in setting
2 and setting 6. Conversely, sample selection methods like Decouple and
MentorNet show good robustness against label noise. Adopting the same small
loss trick, Co-teaching does not perform as well as MentorNet and failed in
setting 2. This indicates the small loss samples selected by one model may not
be useful information for the other model. In all settings, CLC achieves the
best performance which demonstrates that CLC is also effective in handling
textual data under noisy supervision. Impressively, the result of CLC on 20
Newsgroups with high-level symmetric noise even surpasses the results of all
baselines under low-level symmetric noise.

TABLE IV: Average test accuracy (%) on 20 Newsgroups over the last ten epochs. | 20 Newsgroups | TREC
---|---|---
| pairwise | symmetric | pairwise | symmetric
setting | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
noise ratio | 0.2 | 0.45 | 0.2 | 0.5 | 0.2 | 0.45 | 0.2 | 0.5
Standard | 53.31 | 40.60 | 50.80 | 30.13 | 70.20 | 44.82 | 70.16 | 50.18
Bootstrap | 52.57 | 40.88 | 50.90 | 30.16 | 71.10 | 44.10 | 70.82 | 45.88
Forward | 60.55 | 30.69 | 54.94 | 42.26 | 72.22 | 39.20 | 71.86 | 58.22
Decouple | 61.15 | 41.34 | 61.28 | 42.12 | 80.02 | 45.62 | 78.12 | 51.22
MentorNet | 63.19 | 44.96 | 64.23 | 58.93 | 79.82 | 61.98 | 79.42 | 71.40
Co-teaching | 62.82 | 39.98 | 63.48 | 59.83 | 79.78 | 61.82 | 78.14 | 71.54
Co-distillation | 55.36 | 38.41 | 51.89 | 32.92 | 72.18 | 47.76 | 70.59 | 49.66
CLC | 66.42 | 49.24 | 67.22 | 65.02 | 80.81 | 71.80 | 80.12 | 73.42

In figure 8, we analyze the learned representation via t-SNE [27]. We
visualize the the last layer of the classifier before softmax in setting 8.
When the noisy labels are superimposed, the Standard feature representations
are well clustered in the view of noisy training labels, which indicates
Standard completely fits the noisy supervision. This is consistent with the
memorization effects. While for CLC, the clusters contain massive samples from
different classes that indicates CLC does not directly rely on the noisy
labels. When the true labels are superimposed, clusters of the Standard model
turn into mixtures of samples in the view of the ground-truth labels.
Conversely, the feature representations of CLC are much better clustered. This
demonstrates that CLC can guarantee the model to be trained with reliable
supervision under label noise.

Figure 8: The t-SNE visualizations of the last layer before softmax in setting
8. We visualize the feature representation of Standard and CLC with noisy and
clean labels superimposed.

##  IV Conclusion and Future Work

In this paper, we present a simple entropy based criterion to select low
entropy predictions which are more reliable as supervision than the original
noisy labels. Incorporating this entropy based criterion, we propose the CLC
framework where two models correct labels for each other with their low
entropy predictions. CLC has the potential to maintain much more samples than
sample selection methods and the dual network structure of CLC can avoid the
undesired local minimums of the single network. We conduct a range of
experiments on visual and textual datasets with synthetic or real-world noise
to demonstrate that CLC trains models robustly even under extreme label noise.
Future directions of research include automatically adjusting threshold
$\Gamma$ by the network itself and other criterion that verifies the validity
of predictions for CLC.

## Acknowledgment

This work is supported by The High Technology Research and Development Program
of China (2015AA015801), NSFC (61521062), and STCSM (18DZ2270700).

## References

  * [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _NIPS_ , 2012.
  * [2] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [3] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _NIPS_ , 2014.
  * [4] V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy, “Learning from crowds,” _Journal of Machine Learning Research_ , 2010.
  * [5] R. Fergus, L. Fei-Fei, P. Perona, and A. Zisserman, “Learning object categories from internet image searches,” _Proceedings of the IEEE_ , 2010\.
  * [6] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” in _ICLR_ , 2016.
  * [7] J. Yao, H. Wu, Y. Zhang, I. W. Tsang, and J. Sun, “Safeguarded dynamic label regression for noisy supervision,” in _AAAI_ , 2019.
  * [8] J. Yao, J. Wang, I. W. Tsang, Y. Zhang, J. Sun, C. Zhang, and R. Zhang, “Deep learning from noisy image labels with quality embedding,” _IEEE Transactions on Image Processing_ , 2018.
  * [9] B. Han, J. Yao, G. Niu, M. Zhou, I. Tsang, Y. Zhang, and M. Sugiyama, “Masking: A new perspective of noisy supervision,” in _NIPS_ , 2018.
  * [10] J. Cheng, T. Liu, K. Ramamohanarao, and D. Tao, “Learning with bounded instance-and label-dependent label noise,” _arXiv preprint arXiv:1709.03768_ , 2017.
  * [11] E. Malach and S. Shalev-Shwartz, “Decoupling” when to update” from” how to update”,” in _NIPS_ , 2017.
  * [12] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” in _NIPS_ , 2018.
  * [13] Y. Li, J. Yang, Y. Song, L. Cao, J. Luo, and L.-J. Li, “Learning from noisy labels with distillation,” in _ICCV_ , 2017.
  * [14] R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton, “Large scale distributed neural network training through online distillation,” in _ICLR_ , 2018.
  * [15] G. Song and W. Chai, “Collaborative learning for deep neural networks,” in _NIPS_ , 2018.
  * [16] D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio _et al._ , “A closer look at memorization in deep networks,” in _ICML_ , 2017.
  * [17] B. Van Rooyen, A. Menon, and R. C. Williamson, “Learning with symmetric label noise: The importance of being unhinged,” in _NIPS_ , 2015.
  * [18] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Citeseer, Tech. Rep., 2009.
  * [19] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, “Making deep neural networks robust to label noise: A loss correction approach,” in _CVPR_ , 2017.
  * [20] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang, “Learning from massive noisy labeled data for image classification,” in _CVPR_ , 2015.
  * [21] K.-H. Lee, X. He, L. Zhang, and L. Yang, “Cleannet: Transfer learning for scalable image classifier training with label noise,” in _CVPR_ , 2018.
  * [22] X. Li and D. Roth, “Learning question classifiers,” in _COLING_ , 2002.
  * [23] S. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich, “Training deep neural networks on noisy labels with bootstrapping,” in _ICLR_ , 2014.
  * [24] L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei, “Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels,” in _ICML_ , 2018.
  * [25] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2015.
  * [26] J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in _EMNLP_ , 2014.
  * [27] L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” _Journal of machine learning research_ , 2008.