# Asymmetric Co-teaching with Multi-view Consensus
for Noisy Label Learning

Fengbei Liu1 Yuanhong Chen1 Chong Wang 1 Yu Tian2 Gustavo Carneiro3 1
Australian Institute for Machine Learning, University of Adelaide 2 Harvard
Medical School, Harvard University
3 CVSSP, University of Surrey

###### Abstract

Learning with noisy-labels has become an important research topic in computer
vision where state-of-the-art (SOTA) methods explore: 1) prediction
disagreement with co-teaching strategy that updates two models when they
disagree on the prediction of training samples; and 2) sample selection to
divide the training set into clean and noisy sets based on small training
loss. However, the quick convergence of co-teaching models to select the same
clean subsets combined with relatively fast overfitting of noisy labels may
induce the wrong selection of noisy label samples as clean, leading to an
inevitable confirmation bias that damages accuracy. In this paper, we
introduce our noisy-label learning approach, called Asymmetric Co-teaching
(AsyCo), which introduces novel prediction disagreement that produces more
consistent divergent results of the co-teaching models, and a new sample
selection approach that does not require small-loss assumption to enable a
better robustness to confirmation bias than previous methods. More
specifically, the new prediction disagreement is achieved with the use of
different training strategies, where one model is trained with multi-class
learning and the other with multi-label learning. Also, the new sample
selection is based on multi-view consensus, which uses the label views from
training labels and model predictions to divide the training set into clean
and noisy for training the multi-class model and to re-label the training
samples with multiple top-ranked labels for training the multi-label model.
Extensive experiments on synthetic and real-world noisy-label datasets show
that AsyCo improves over current SOTA methods.

##  1 Introduction

Figure 1: Comparison of methods Decoupling [20], Co-teaching+ [36], JoCoR
[29], and our AsyCo. AsyCo co-teaches the multi-class model A and the multi-
label model B with different training strategies (denoted by the different
colours of A&B). The training samples for A and B, represented by the green
and red arrows, are formed by our proposed multi-view consensus that uses
label views from the training set and model predictions to estimate the
variables $\mathbf{w}$ and $\hat{\mathbf{y}}$, which selects clean/noisy
samples for training A and iteratively re-labels samples for training B,
respectively.

Deep neural network (DNN) has achieved remarkable success in many fields,
including computer vision [15, 11], natural language processing (NLP) [7, 35]
and medical image analysis [17, 28]. However, the methods from those fields
often require massive amount of high-quality annotated data for supervised
training [6], which is challenging and expensive to acquire. To alleviate such
problem, some datasets have been annotated via crowdsourcing [32], from search
engines [27], or with NLP from radiology reports [28]. Although these cheaper
annotation processes enable the construction of large-scale datasets, they
inevitably introduce noisy labels for model training, resulting in DNN model
performance degradation. Therefore, novel learning algorithms are required to
robustly train DNN models when training sets containing noisy labels.

Previous methods tackle noisy-label learning from different perspectives. For
example, some approaches focus on prediction disagreement  [36, 29, 20], which
rely on jointly training two models to update their parameters when they
disagree on the predictions of the same training samples. These two models
generally use the same training strategy, so even though they are trained
using samples with divergent predictions, both models will quickly converge to
select similar clean samples during training, which neutralises the
effectiveness of prediction disagreement. Other noisy-label learning methods
are based on sample selection [16, 9, 1] to find clean and noisy-label samples
that are treated differently in the training process. Sample-selection
approaches usually assume that samples with small training losses are
associated with clean labels, which is an assumption verified only at early
training stages [18, 37]. However, such assumption is unwarranted in later
training stages because DNN models can overfit any type of noisy label after a
certain number of epochs, essentially reducing the training loss for all
training samples. State-of-the-art (SOTA) noisy-label learning approaches [16]
have been designed to depend on both prediction disagreement and sample
selection methods to achieve better performance than either method alone.
Nevertheless, these SOTA methods are still affected by the fast convergence of
both models and label noise overfitting, which raises the following questions:
1) Are there more effective ways to maximise the prediction disagreement
between both models, so they consistently produce divergent results during the
training procedure? 2) Is there a sample selection approach that can better
integrate prediction disagreements than the small loss strategy?

Motivated by traditional multi-view learning [3, 26] and multi-label learning
[24], we propose a new noisy-label learning method that aims to answer the two
questions above. Our method, named Asymmetric Co-teaching (AsyCo) and depicted
in Fig. 1, is based on two models trained with different learning strategies
to maximise their prediction disagreement. One model, the classification net,
is trained with conventional multi-class learning by minimising a cross
entropy loss and provide single-class prediction, and the other, the reference
net, is trained with a binary cross entropy loss to enable multi-label
learning that is used to estimate the top-ranked labels that represent the
potentially clean candidate labels for each training sample. The original
training labels and the predictions by the training and reference nets enable
the formation of three label views for each training sample, allowing us to
formulate the multi-view consensus that is tightly integrated with the
prediction disagreement to select clean and noisy samples for training the
multi-class model and to iteratively re-label samples with multiple top-ranked
labels for training the multi-label model. In summary, our main contributions
are:

  * •

The new noisy-label co-teaching method AsyCo designed to maximise the
prediction disagreement between the training of a multi-class and a multi-
label model; and

  * •

The novel multi-view consensus that uses the disagreements between training
labels and model predictions to select clean and noisy samples for training
the multi-class model and to iteratively re-label samples with multiple top-
ranked labels for training the multi-label model.

We conduct extensive experiments on both synthetic and real-world noisy
datasets that show that AsyCo provides substantial improvements over previous
state-of-the-art (SOTA) methods.

##  2 Related Work

Prediction disagreement approaches seek to maximise model performance by
exploring the prediction disagreements between models trained from the same
training set. In general, these methods [20, 36, 29, 13] train two models
using samples that have different predictions from both models to mitigate the
problem of confirmation bias (i.e., a mistake being reinforced by further
training from the same mistake) that particularly affects single-model
training. Furthermore, the cross teaching of two models can help escape local
minima. Most of the prediction-disagreement methods also rely on sample-
selection techniques, as we explain below, but in general, they use the same
training strategy to train two models, which limits the ability of these
approaches to maximise the divergence between the models.

Sample selection approaches aim to automatically classify training samples
into clean or noisy and treat them differently during the training process.
Previous papers [18, 37] have shown that when training with noisy label, DNN
fits the samples with clean labels first and gradually overfits the samples
with noisy labels later. Such training loss characterisation allowed
researchers to assume that samples with clean labels have small losses,
particularly at early training stages – this is known as the small-loss
assumption. For examples, M-correction [1] automatically selects clean samples
by modelling the training loss distribution with a Beta Mixture model (BMM).
Sample selection has been combined with prediction disagreement in several
works, such as Co-teaching [9] and Co-teaching+ [36] that train two networks
simultaneously, where in each mini-batch, it selects small-loss samples to be
used in the training of the other model. JoCoR [29] improves upon Co-teaching+
by using a contrastive loss to jointly train both models. DivideMix [16] has
advanced the area with a similar combination of sample selection and
prediction disagreement using semi-supervised learning, co-teaching and small-
loss detection with a Gaussian Mixture Model (GMM). InstanceGM [8] combines
graphical model with DivideMix to achieve promising results. These methods
show that sample selection based on the small-loss assumption is one of the
core components for achieving SOTA performance. However, the small loss signal
used to select samples is poorly integrated with prediction disagreement since
both models will quickly converge to produce similar loss values for all
training samples, resulting in little disagreement between models, which
increases the risk of confirmation bias.

Transition matrix methods aim to estimate a noise transition matrix to
guarantee that the classifier learned from the noisy data is consistent with
the optimal classifier [31, 22, 5] F-correction [22] uses a two-step solution
to heuristically estimate the noise transition matrix. T-revision [31] argues
that anchor points are not necessary for estimating the transition matrix and
proposes a solution for selecting reliable samples to replace anchor points.
kMEIDTM [5] proposes an anchor-free method for estimating instance-dependent
transition matrix by applying manifold regularization during the training. The
main issue with the methods above is that it is challenging to estimate the
transition matrix accurately, particularly an instance-dependent transition
matrix that contains little support from the training set. Furthermore, real-
world scenarios often contain out-of-distribution samples that are hard to
represent in the transition matrix.

Multi-view learning (MVL) studies the integration of knowledge from different
views of the data to capture consensus and complementary information across
different views. Traditional MVL methods [3, 26] aimed to encourage the
convergence of patterns from different views. For example, Co-training [3]
uses two views of web-pages (i.e., text and hyperlinks on web-pages) to allow
the use of inexpensive unlabelled data to augment a small labelled data.
Considering that the quality and importance of different views could vary for
real-world applications, recent methods [10] weight the contribution of each
view based on the estimated uncertainty. In our paper, we explore this multi-
view learning strategy to select clean and noisy samples and to iteratively
re-label training samples, where the views are represented by the training
labels, and the predictions by the two models that are trained using different
learning strategies.

##  3 Method

###  3.1 Problem Definition

We denote the noisy training set as
$\mathcal{D}=\\{(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\\}_{i=1}^{|\mathcal{D}|}$,
where $\mathbf{x}_{i}\in\mathcal{X}\subset\mathbb{R}^{H\times W\times C}$ is
the input image of size $H\times W$ with $C$ colour channels, and
$\tilde{\mathbf{y}}_{i}\in\mathcal{Y}\subset\\{0,1\\}^{|\mathcal{Y}|}$ is the
one-hot (or multi-class) label representation. The goal of is to learn the
classification net $n_{\theta}:\mathcal{X}\to\mathcal{L}$, parameterised by
$\theta\in\Theta$, that outputs the logits
$\mathbf{l}\in\mathcal{L}\subset\mathbb{R}^{|\mathcal{Y}|}$ for an image
$\mathbf{x}\in\mathcal{X}$. Following the prediction-disagreement strategy, we
also define the reference net denoted by $r_{\phi}:\mathcal{X}\to\mathcal{L}$,
parameterised by $\phi\in\Phi$, to be jointly trained with $n_{\theta}(.)$.

AsyCo111Algorithm in supplementary material. is based on alternating the
training of the multi-class model $n_{\theta}(.)$ and the multi-label model
$r_{\phi}(.)$, which allows the formation of three label views for the
training samples $\\{\mathbf{x}_{i}\\}_{i=1}^{|\mathcal{D}|}$: 1) the original
training label $\tilde{\mathbf{y}}_{i}$, 2) the classification net multi-class
prediction $\tilde{\mathbf{y}}^{(n)}_{i}$, and 3) the reference net multi-
label prediction $\tilde{\mathbf{y}}^{(r)}_{i}$. Using these views, we
introduce new methods to estimate the sample-selection variable $\mathbf{w}$
that classifies training samples into clean or noisy, and the re-labelling
variable $\hat{\mathbf{y}}$ that holds multiple top-ranked labels for training
samples, where $\mathbf{w}$ is used for training the multi-class model
$n_{\theta}(.)$, and $\hat{\mathbf{y}}$ for training the multi-label model
$r_{\phi}(.)$. Fig. 2 depicts AsyCo, in comparison with prediction
disagreement methods based on co-teaching and small-loss sample selection.

###  3.2 Asymmetric Co-teaching Optimisation

Figure 2: Comparison between traditional small-loss sample selection (top) and
our AsyCo, consisting of prediction disagreement between the multi-class model
A and multi-label model B (bottom). Traditional methods utilises the small-
loss assumption for classifying samples as clean or noisy, while our multi-
view sample selection uses prediction disagreements to update the sample-
selection variable $\mathbf{w}$ for classifying samples as clean, noisy or
unmatched (U) to train the classification net A. Our multi-view re-labelling
selects ambiguous samples and maximise disagreement by updating the re-
labelling variable $\mathbf{\hat{y}}$ for training the reference net B.

Our Asymmetric co-teaching optimisation trains a multi-class model with the
usual cross-entropy (CE), but the other model is trained with multi-label
learning [23] that associates samples with multiple labels and utilises binary
cross-entropy (BCE) to train for each label independently. We have two goals
with the multi-label model: 1) maximise the disagreement with the multi-class
model, and 2) formulate a mechanism to find the most likely clean labels by
selecting multiple top-ranked labels of training samples. While the first goal
is motivated by the training strategy differences, the second goal is
motivated by the hypothesis that a possible cause of the overfitting of noisy
labels is the single-class constraint that forces multi-class models to fit
only one class. By removing this constraint, the true clean label is likely to
be within the top-ranked candidate labels222Training strategy visualization in
supplementary material.. Our AsyCo optimisation starts with a warmup stage of
supervised learning to train both networks with:

$\begin{split}\theta^{\dagger}&=\arg\min_{\theta}\sum_{(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}}\ell_{\mathrm{CE}}(\tilde{\mathbf{y}}_{i},\sigma_{sm}(n_{\theta}(\mathbf{x}_{i}))),\\\
\phi^{\dagger}&=\arg\min_{\phi}\sum_{(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}}\ell_{\mathrm{BCE}}(\tilde{\mathbf{y}}_{i},\sigma_{sg}(r_{\phi}(\mathbf{x}_{i}))),\end{split}$
(1)

where $\sigma_{sm}(.)$ and $\sigma_{sg}(.)$ are the softmax and sigmoid
activation functions, respectively, $\ell_{\mathrm{CE}}(.)$ represents the CE
loss for multi-class learning, and $\ell_{\mathrm{BCE}}$ denotes the BCE loss
for multi-label learning. The two models from (1) will provide predictions as
follows:

$\begin{split}\tilde{\mathbf{y}}_{i}^{(n)}&=\mathrm{OneHot}(n_{\theta^{\dagger}}(\mathbf{x}_{i})),\\\
\tilde{\mathbf{y}}_{i}^{(r)}&=\mathrm{TopK}(r_{\phi^{\dagger}}(\mathbf{x}_{i})),\end{split}$
(2)

where $\tilde{\mathbf{y}}_{i}^{(n)}\in\mathcal{Y}$ is the one-hot single-label
prediction by $n_{\theta^{\dagger}}(\mathbf{x}_{i})$, and
$\tilde{\mathbf{y}}_{i}^{(r)}\in\\{0,1\\}^{|\mathcal{Y}|}$ is the top-$K$
multi-label prediction of $r_{\phi^{\dagger}}(\mathbf{x}_{i})$ (i.e., the
largest $K$ values from $r_{\phi^{\dagger}}(.)$ will set
$\tilde{\mathbf{y}}_{i}^{(r)}$ to $1$ and the rest are set to $0$). However,
removing the single-class constraint from multi-class classification
inevitably weakens the model performance. Thus, we aim to extract useful
information from top-ranked candidate labels to help training $n_{\theta}$
with multi-view consensus, explained below, which uses the label views
produced by the predictions from $n_{\theta}$ and $r_{\phi}$ and the training
labels, to select samples for training $n_{\theta}$ and re-label samples for
training $r_{\phi}$.

###  3.3 Multi-view Consensus

One of the objectives of maximising prediction disagreement between models is
to improve sample selection accuracy for co-teaching. We propose a new sample
selection based on multi-view consensus, where each sample $\mathbf{x}_{i}$
has three label views: the single-label training label
$\tilde{\mathbf{y}}_{i}$, the single-label one-hot prediction
$\tilde{\mathbf{y}}_{i}^{(n)}$, and the multi-label top-$K$ prediction
$\tilde{\mathbf{y}}_{i}^{(r)}$. These multiple views allow us to build
training subsets given prediction disagreements, as shown in Tab. 1, where the
Agreement Degree (AG) score is defined as:

$\text{AG}(\tilde{\mathbf{y}},\tilde{\mathbf{y}}^{(n)},\tilde{\mathbf{y}}^{(r)})=\tilde{\mathbf{y}}^{\top}\tilde{\mathbf{y}}^{(n)}+{\tilde{\mathbf{y}}^{(n)}}^{\top}\tilde{\mathbf{y}}^{(r)}+\tilde{\mathbf{y}}^{\top}\tilde{\mathbf{y}}^{(r)}$
(3)

Table 1: Three possible label views: the training label
$\tilde{\mathbf{y}}_{i}$, the single-label one-hot prediction
$\tilde{\mathbf{y}}_{i}^{(n)}$, and the multi-label top-$K$ prediction
$\tilde{\mathbf{y}}_{i}^{(r)}$. The combination of these multiple views form
the subsets, defined in the first column, with agreement scores
$\text{AG}(.)$, from (3), in the last column.

Subsets | $\tilde{\mathbf{y}}^{\top}\tilde{\mathbf{y}}^{(n)}$ | ${\tilde{\mathbf{y}}^{(n)}}^{\top}\tilde{\mathbf{y}}^{(r)}$ | $\tilde{\mathbf{y}}^{\top}\tilde{\mathbf{y}}^{(r)}$ | $\text{AG}(.)$
---|---|---|---|---
Core (C) | 1 | 1 | 1 | 3
Side-Core (SC) | 0 | 1 | 1 | 2
NY | 1 | 0 | 0 | 1
NR | 0 | 1 | 0 | 1
RY | 0 | 0 | 1 | 1
Unmatched (U) | 0 | 0 | 0 | 0

The training of the classification net $n_{\theta}(.)$ has the goals of
producing the testing model and of maximising the disagreement with
$r_{\phi}(.)$. This training employs a semi-supervised learning strategy [2],
which requires the division of the training set into clean and noisy sets.
Unlike previous methods that rely on the small-loss assumption to classify
training samples into clean or noisy [16, 9, 1], we utilize the subsets
created by prediction disagreements from the multiple label views shown in
Tab. 1. For training $n_{\theta}(.)$, we first discard all samples in the
subset $\mathrm{Unmatched}$ given their high level of uncertainty because both
models disagree with each other and with the training label. For the remaining
samples, we seek label agreements between pair of views beyond its own
prediction. More specifically, training samples are classified as clean when
$\tilde{\mathbf{y}}^{\top}\tilde{\mathbf{y}}^{(r)}=1$, which indicates that
the training label matches one of the top ranked predictions by $r_{\phi}(.)$.
Such agreement from label views $\tilde{\mathbf{y}}$ and
$\tilde{\mathbf{y}}^{(r)}$ indicates that the training label
$\tilde{\mathbf{y}}$ is within the top-ranked predictions by $r_{\phi}(.)$,
but may not match the prediction by $n_{\theta}(.)$. Therefore, classifying
such samples as clean can help maximise the disagreement with $r_{\phi}$ and
alleviate confirmation bias. The remaining samples with
$\tilde{\mathbf{y}}^{\top}\tilde{\mathbf{y}}^{(r)}=0$ are classified as noisy
because of the insufficient support by $r_{\phi}(.)$ for the training label
$\tilde{\mathbf{y}}$. Therefore, based on the criterion described above, the
classification net $n_{\theta}$ is trained with
$\\{\mathrm{C},\mathrm{SC},\mathrm{RY}\\}$ as clean and
$\\{\mathrm{NY},\mathrm{NR}\\}$ as noisy, defined by the following sample-
selection variable:

$\mathbf{w}_{i}=\left\\{\begin{array}[]{lll}+1,&\text{ if
}\text{AG}(\tilde{\mathbf{y}}_{i},\tilde{\mathbf{y}}^{(n)}_{i},\tilde{\mathbf{y}}^{(r)}_{i})>0\text{
and }\tilde{\mathbf{y}}_{i}^{\top}\tilde{\mathbf{y}}_{i}^{(r)}=1,\\\ 0,&\text{
if
}\text{AG}(\tilde{\mathbf{y}}_{i},\tilde{\mathbf{y}}^{(n)}_{i},\tilde{\mathbf{y}}^{(r)}_{i})>0\text{
and }\tilde{\mathbf{y}}_{i}^{\top}\tilde{\mathbf{y}}_{i}^{(r)}=0,\\\
-1,&\text{ if
}\text{AG}(\tilde{\mathbf{y}}_{i},\tilde{\mathbf{y}}^{(n)}_{i},\tilde{\mathbf{y}}^{(r)}_{i})=0,\end{array}\right.$
(4)

where $\mathbf{w}_{i}\in\\{+1,0,-1\\}$ denotes a clean, noisy, and unmatched
training sample, respectively.

The training of $n_{\theta}(.)$ is performed by

$\begin{split}\theta^{*}&=\arg\min_{\theta}\sum_{\begin{subarray}{c}(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}\\\
\mathbf{w}_{i}=+1\end{subarray}}\ell_{CE}(\tilde{\mathbf{y}}_{i},\sigma_{sm}(n_{\theta}(\mathbf{x}_{i})))\\\
&+\lambda\sum_{\begin{subarray}{c}(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}\\\
\mathbf{w}_{i}=0\end{subarray}}\ell_{MSE}(\upsilon(\sigma_{sm}(n_{\theta}(\mathbf{x}_{i})),T),\sigma_{sm}(n_{\theta}(\mathbf{x}_{i}))),\end{split}$
(5)

where $\upsilon(.,T)$ is a sharpening function [16] parameterised by the
temperature $T$, and $\lambda$ is the weight to control the strength of the
unsupervised learning with the noisy labels, and $\ell_{MSE}(.)$ denotes the
mean square error loss function.

The training of the reference net $r_{\phi}(.)$ has the goals of maximising
the disagreement with $n_{\theta}(.)$ using the multi-view consensus from Tab.
1, and maintaining the top-ranked labels of training samples as clean label
candidates. To achieve that, we focus on designing a new supervisory training
signal by re-labelling the samples where predictions by $n_{\theta}(.)$ and
$r_{\phi}(.)$ match (i.e.,
${\tilde{\mathbf{y}}^{(n)}}^{\top}\tilde{\mathbf{y}}^{(r)}=1$) and the
prediction by $n_{\theta}(.)$ does not match the training label
$\tilde{\mathbf{y}}$ (i.e.,
${\tilde{\mathbf{y}}}^{\top}\tilde{\mathbf{y}}^{(n)}=0$). The training samples
that meet this condition can be regarded as hard to fit by $n_{\theta}(.)$,
with the top-ranked predictions by $\tilde{\mathbf{y}}^{(r)}$ being likely to
contain the hidden clean label. The conditions above indicates that we select
samples from $\mathrm{SC}\bigcup\mathrm{NR}$ from Tab. 1 for re-labelling. For
samples in $\mathrm{SC}$, since $n_{\theta}(.)$ is trained with supervised
learning in (5), the maximisation of prediction disagreement is achieved by
re-labelling the sample to $\tilde{\mathbf{y}}^{(n)}$. For samples in
$\mathrm{NR}$, $n_{\theta}(.)$ is trained with unsupervised learning in (5),
so the prediction disagreement is maximised by re-labelling the sample to
$\tilde{\mathbf{y}}+\tilde{\mathbf{y}}^{(n)}$, forming a multi-label target.
We define the re-labelling variable $\hat{\mathbf{y}}$ to represent the new
supervisory training signal, as follows:

$\hat{\mathbf{y}}_{i}=\left\\{\begin{array}[]{lll}\tilde{\mathbf{y}}_{i}^{(n)},&\text{
if }(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathrm{SideCore},\\\
\tilde{\mathbf{y}}_{i}+\tilde{\mathbf{y}}_{i}^{(n)},&\text{ if
}(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathrm{NR},\\\
\tilde{\mathbf{y}}_{i},&\text{otherwise},\end{array}\right.$ (6)

with training of $r_{\phi}(.)$ achieved with:

$\phi^{*}=\arg\min_{\phi}\sum_{i=1}^{|\mathcal{D}|}\ell_{BCE}(\hat{\mathbf{y}}_{i},\sigma_{sg}(r_{\phi}(\mathbf{x}_{i}))).$
(7)

Note that this re-labelling is iteratively done at every epoch. The testing
procedure depends exclusively on the classification net $n_{\theta}(.)$.

##  4 Experiments

We show the results of extensive experiments on instance-dependent synthetic
noise benchmarks with datasets CIFAR10 and CIFAR100 [14] with various noise
rates and on three real-world datasets, namely: Animal-10N [27], Red Mini-
ImageNet [12] and Clothing1M [32].

###  4.1 Datasets

CIFAR10/100. For CIFAR10 and CIFAR100 [14], the training set contains 50K
images and testing set contains 10K images of size 32 $\times$ 32 $\times$ 3\.
CIFAR10 has 10 classes and CIFAR100 has 100 classes. We follow previous work
[30] for generating instance-dependent noise with rates in {0.2, 0.3, 0.4,
0.5}. Red Mini-ImageNet is proposed by [12] based on Mini-ImageNet [6]. The
images and their corresponding labels are annotated by Google Cloud Data
Labelling Service. This dataset is proposed to study real-world web-based
noisy label. Red Mini-ImageNet has 100 classes with each class containing 600
images from ImageNet. The images are resized to 32 $\times$ 32 from the
original 84 $\times$ 84 pixels to allow a fair comparison with other baselines
[33, 12]. We test our method on noise rates in {20%, 40%, 60%, 80%}. Animal
10N is a real-world dataset proposed in [27], which contains 10 animal species
with similar appearances (wolf and coyote, hamster and guinea pig, etc.). The
training set size is 50K and testing size is 10K, where we follow the same
setup as [27]. Clothing 1M is a real-world dataset with 100K images and 14
classes. The labels are generated from surrounding text with an estimated
noise ratio of 38.5%. We follow a common setup using a training image size of
224 $\times$ 224 pixels. The dataset also contains clean training, clean
validation and clean test sets with 50K, 14K and 10K images. We do not use
clean training and clean validation, only the clean testing is used for
measuring model performance.

Methods | CIFAR10 | CIFAR100
---|---|---
0.2 | 0.3 | 0.4 | 0.5 | 0.2 | 0.3 | 0.4 | 0.5
CE | 75.81 | 69.15 | 62.45 | 39.42 | 30.42 | 24.15 | 21.34 | 14.42
Mixup [38] | 73.17 | 70.02 | 61.56 | 48.95 | 32.92 | 29.76 | 25.92 | 21.31
Forward [22] | 74.64 | 69.75 | 60.21 | 46.27 | 36.38 | 33.17 | 26.75 | 19.27
T-Revision [31] | 76.15 | 70.36 | 64.09 | 49.02 | 37.24 | 36.54 | 27.23 | 22.54
Reweight [19] | 76.23 | 70.12 | 62.58 | 45.46 | 36.73 | 31.91 | 28.39 | 20.23
PTD-R-V [30] | 76.58 | 72.77 | 59.50 | 56.32 | 65.33 | 64.56 | 59.73 | 56.80
Decoupling [20] | 78.71 | 75.17 | 61.73 | 50.43 | 36.53 | 30.93 | 27.85 | 19.59
Co-teaching [9] | 80.96 | 78.56 | 73.41 | 45.92 | 37.96 | 33.43 | 28.04 | 23.97
MentorNet [13] | 81.03 | 77.22 | 71.83 | 47.89 | 38.91 | 34.23 | 31.89 | 24.15
CausalNL [34] | 81.79 | 80.75 | 77.98 | 78.63 | 41.47 | 40.98 | 34.02 | 32.13
CAL [40] | 92.01 | - | 84.96 | - | 69.11 | - | 63.17 | -
kMEIDTM [5] | 92.26 | 90.73 | 85.94 | 73.77 | 69.16 | 66.76 | 63.46 | 59.18
DivideMix [16] $\theta^{(1)}$ test † | 94.62 | 94.49 | 93.50 | 89.07 | 74.43 | 73.53 | 69.18 | 57.52
Ours | 96.00 | 95.82 | 95.01 | 94.13 | 76.02 | 74.02 | 68.96 | 60.35
DivideMix [16] † | 94.80 | 94.60 | 94.53 | 93.04 | 77.07 | 76.33 | 70.80 | 58.61
Ours 2$\times n_{\theta}$ test | 96.56 | 96.11 | 95.53 | 94.86 | 78.50 | 77.32 | 73.32 | 65.96

Table 2: Test accuracy (%) of different methods on CIFAR10/100 with instance-
dependent noise [30]. Results reproduced from publicly available code are
presented with $\dagger$. Best single/ensemble inference results are labelled
with red/green.

###  4.2 Implementation

For CIFAR10/10 and Red Mini-ImageNet we use Preact-ResNet18 [11] and train it
for 200 epochs with SGD with momentum=0.9, weight decay=5e-4 and batch
size=128. The initial learning rate is 0.02 and reduced by a factor of 10
after 150 epochs. The warmup period for all three datasets is 10 epochs. We
set $\lambda=25$ in (5) for CIFAR10 and Red Mini-ImageNet, and $\lambda=100$
for CIFAR100. In (2), we set $K=1$ for CIFAR10 and $K=3$ for CIFAR100 and Red
Mini-ImageNet. These values are fixed for all noise rates. For data
augmentations, we use random cropping and random horizontal flipping for all
three datasets.

For Animal 10N, we follow a common setup used by previous methods with a
VGG-19BN [25] architecture, trained for 100 epochs with SGD with momentum=0.9,
weight decay=5e-4 and batch size=128. The initial learning rate is 0.02, and
reduced by a factor of 10 after 50 epochs. The warmup period is 10 epochs. We
set $\lambda=25$ and $K=2$. For data augmentations, we use random cropping and
random horizontal flipping.

For Clothing1M, we use ImageNet [6] pre-trained ResNet50 [11] and train it for
80 epochs with SGD with momentum=0.9, weight decay=1e-3 and batch size=32. The
warmup period is 1 epoch. The initial learning rate is set to 0.002 and
reduced by a factor of 10 after 40 epochs. Following DivideMix [16], we also
sample 1000 mini-batches from the training set to ensure the training set is
pseudo balanced. We set $K=4$. For data augmentation, we first resize the
image to 256 $\times$ 256 pixels, then random crop to 224 $\times$ 224 and
random horizontal flipping.

For the semi-supervised training of $n_{\theta}(.)$, we use MixMatch [2] from
DivideMix [16]. We also extend our method to train two $n_{\theta}(.)$ models
and use ensemble prediction at inference time, similarly to DivideMix [16]. We
denoted this variant as $2\times n_{\theta}$. Our code is implemented in
Pytorch [21] and all experiments are performed on an RTX 3090333Time of
Different sample selection comparison in supplementary.

###  4.3 Comparison with SOTA Methods

We compare our AsyCo with the following methods: 1) CE, which trains the
classification network with standard CE loss on the noisy dataset; 2) Mixup
[38], which employs mixup on the noisy dataset; 3) Forward [22], which
estimates the noise transition matrix in a two-stage training pattern; 4)
T-Revision [31], which finds reliable samples to replace anchor points for
estimating transition matrix; 5) Reweight [19], which utilizes a class-
dependent transition matrix to correct the loss function; 6) PTD-R-V [30],
which proposes a part-dependent transition matrix for accurate estimation; 7)
Decoupling [20], which trains two networks on samples whose predictions from
the network are different; 8) Co-teaching [9], which trains two networks and
select small-loss samples as clean samples; 9) MentorNet [13], which utilizes
a teacher network for selecting noisy samples; 10) CausalNL [34], which
discovers a causal relationship in noisy dataset and combines it with Co-
Teaching; 11) CAL [40], which uses second-order statistics with a new loss
function; 12) kMEIDTM [5], which learns instance-dependent transition matrix
by applying manifold regularization during the training; 13) DivideMix [16],
which combines semi-supervised learning, sample selection and Co-Teaching to
achieve SOTA results; 14) FaMUS [33], which is a meta-learning method that
learns the weight of training samples to improve the meta-learning update
process; 15) Nested [4], which is a novel feature compression method that uses
nested dropout to regularize features when training with noisy label–this
approach can be combined with existing techniques such as Co-Teaching [9]; and
16) PLC [39], which is a method that produces soft pseudo label when learning
with label noise.

###  4.4 Experiment Results

Synthetic Noise Benchmarks. The experimental results of our proposed AsyCo
with instance-dependent noise on CIFAR10/100 are shown in Tab. 2. We reproduce
DivideMix [16] in this setup with single model at inference time denoted by
$\theta^{(1)}$ and also the original ensemble inference. Compared with the
best baselines, our method achieves large improvements for all noise rates. On
CIFAR10, we achieve $\approx 1.5\%$ improvements for low noise rates and
$\approx 1\%$ to $5\%$ improvements for high noise rates. For CIFAR100, we
improve between $\approx 1.5\%$ and $\approx 7\%$ for many noise rates. Note
that our result is achieved without using small-loss sample selection, which
is a fundamental technique for most noisy label learning methods [16, 9, 13].
The superior performance of AsyCo indicates that our multi-view consensus for
sample selection and top-rank re-labelling are effective when learning with
label noise.

Method | Noise rate
---|---
0.2 | 0.4 | 0.6 | 0.8
CE | 47.36 | 42.70 | 37.30 | 29.76
Mixup [38] | 49.10 | 46.40 | 40.58 | 33.58
DivideMix [16] | 50.96 | 46.72 | 43.14 | 34.50
MentorMix [12] | 51.02 | 47.14 | 43.80 | 33.46
FaMUS [33] | 51.42 | 48.06 | 45.10 | 35.50
Ours | 59.40 | 55.08 | 49.78 | 41.02
Ours 2$\times n_{\theta}$ test | 61.98 | 57.46 | 51.86 | 42.58

Table 3: Test accuracy (%) of different methods on Red Mini-ImageNet with
different noise rates. Baselines results are from FaMUS [33]. Best results
with single/ensemble inferences are labelled with red/green.

Method | Accuracy
---|---
CE | 79.4
Nested [4] | 81.3
Dropout + CE [4] | 81.1
SELFIE [27] | 81.8
PLC [39] | 83.4
Nested + Co-Teaching [4] | 84.1
Ours | 85.6
Ours 2$\times n_{\theta}$ | 86.3

Table 4: Test accuracy (%) of different methods on Animal-10N. Baselines
results are presented with Nested Dropout [4]. Best single/ensemble inference
results are labelled with red/green.

Single | Methods | CE | Forward [22] | PTD-R-V [30] | ELR [18] | kMEIDTM [5] | Ours
---|---|---|---|---|---|---|---
Accuracy | 68.94 | 69.84 | 71.67 | 72.87 | 73.34 | 73.60
Ensemble | Methods | Co-Teaching [9] | Co-Teaching+ [36] | JoCoR [29] | CausalNL [34] | DivideMix [16] | Ours 2$\times n_{\theta}$
Accuracy | 69.21 | 59.3 | 70.3 | 72.24 | 74.60 | 74.43

Table 5: Test accuracy (%) of different methods on Clothing1M. Best
single/ensemble inference results are labelled with red/green.

Real-world Noisy-label Datasets. In Tab. 3, we present results on Red Mini-
ImageNet [12]. Our method achieves SOTA results for all noise rates with 4% to
8% improvements in single model inference and 7% to 10% in ensemble inference.
The improvement is significant compared with FaMUS [33] with a gap of more
than 6%. Compared with DivideMix [16], our method achieves between 6% and 10%
improvements. In Tab. 4, we present the results for Animal 10N [27], where the
previous SOTA method was Nested Dropout + Co-Teaching [4], which achieves
84.1% accuracy. Our method achieves 85.6% accuracy, which is 2.2% higher than
previous SOTA. Additionally, our ensemble version achieves 86.34% accuracy,
which improves 1% more compared to our single inference model, yielding a new
SOTA result. In Tab. 5, we show our result on Clothing1M [32]. In the single
model setup, our model outperforms all previous SOTA methods. In the ensemble
inference setup, our model shows comparable performance with the SOTA method
DivideMix [16] and outperforms all other methods. Compared with other methods
based on prediction disagreement [9, 36, 29], our model improves by at least
3%. The performance on these three real-world datasets indicates the
superiority of our proposed AsyCo.

##  5 Ablation Study

For the ablation study, we first visualise the training losses of subsets from
Tab. 1 that are used by our multi-view consensus approach. We also compare the
accuracy of GMM selected clean samples and our multi-view selected samples.
Then we test alternative approaches for multi-view sample selection and re-
labelling. We perform all ablation experiments on the instance-dependent
CIFAR10/100 [30].

(a) CIFAR100 0.2 loss

(b) CIFAR100 0.2 Accuracy

(c) CIFAR100 0.5 loss

(d) CIFAR100 0.5 Accuracy

Figure 3:  (a) and (c) are sample loss histograms for the subsets in Tab. 1
for CIFAR100 with 0.2 and 0.5 instance-dependent noise after warmup. Vertical
dot line is GMM threshold. (b) and (d) are accuracy of clean set selected by
GMM and our multi-view strategy. (b) and (d) also show accuracy of whether
hidden clean labels within $r_{\phi}$ top-ranked prediction or not for multi-
view re-labelling and not re-labelling.

Model | Ablation | CIFAR10 | CIFAR100
---|---|---|---
0.2 | 0.3 | 0.4 | 0.5 | 0.2 | 0.3 | 0.4 | 0.5
$n_{\theta}$ |  $\mathbf{w}_{i}=0$ if  $(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathrm{RY}$ | 93.28 | 93.85 | 92.54 | 82.60 | 73.58 | 71.51 | 65.51 | 56.65
$\mathbf{w}_{i}=0$ if  $(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathrm{U}$ | 95.71 | 94.88 | 94.34 | 91.60 | 75.10 | 72.64 | 67.42 | 57.55
$\mathbf{w}_{i}=+1$ if  $(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathrm{U}$ | 95.20 | 95.14 | 94.72 | 90.27 | 75.34 | 73.21 | 66.09 | 55.95
Small-loss subsets | 92.37 | 91.80 | 90.93 | 78.53 | 70.10 | 69.52 | 64.69 | 56.35
$r_{\phi}$ | CE | 95.22 | 94.83 | 83.48 | 64.96 | 73.33 | 69.29 | 63.82 | 54.83
Frozen after warmup | 91.19 | 88.97 | 84.72 | 67.57 | 68.73 | 65.36 | 58.88 | 48.13
$\mathbf{\hat{y}_{i}}=\mathbf{\tilde{y}_{i}}$ | 95.42 | 94.69 | 90.53 | 84.95 | 74.43 | 71.75 | 62.25 | 53.69
$\mathbf{\hat{y}_{i}}=\mathbf{\tilde{y}^{(n)}_{i}}$ | 94.29 | 94.23 | 94.13 | 93.67 | 74.55 | 73.71 | 68.21 | 57.84
AsyCo original result: | 96.00 | 95.82 | 95.01 | 94.13 | 76.02 | 74.02 | 68.96 | 60.35

Table 6: Ablation study for the classification net $n_{\theta}$ and reference
net $r_{\phi}$.

Fig. 3(a) and Fig. 3(c) show the loss histograms after warmup for each subset
in Tab. 1. To compare with small-loss sample selection approaches, we adopt
the sample-selection approach by DivideMix [16] that is based on a Gaussian
Mixture Model (GMM) to divide the training set into clean and noisy subsets
(the vertical black dotted line is the threshold estimated by DivideMix).
These graphs show that the subsets’ loss histograms are relatively consistent
in different noise rates. Specifically, $\mathrm{C}$ always has the smallest
loss values among all subsets, which shows that our multi-view sample
selection is able to confidently extract clean samples. We also observe that
$\mathrm{NY}$ has small loss values in both graphs. However, using
$\mathrm{NY}$ as clean set does not produce promising performance, as shown in
Tab. 6, row ’Small-loss subsets’, which represents the use of almost all
samples in C and NY as clean samples (since they are on the left-hand side of
the GMM threshold). This indicates that the small-loss samples in
$\mathrm{NY}$ are likely to contain overfitted noisy-label samples, whereas
our multi-view sample selection successfully avoids selecting these samples.
In Fig. 3(b) and Fig. 3(d), we show the accuracy of the clean set selected by
the GMM-based small-loss strategy of DivideMix and by our multi-view consensus
during the training stages. We observe that multi-view selection performs
consistently better than GMM in both graphs. We also validate the accuracy of
the hidden clean label produced by the top ranked predictions of $r_{\phi}(.)$
by comparing the re-labelling produced by Eq. 6 versus no re-labelling (i.e.,
train $r_{\phi}(.)$ with the original training labels.) Our multi-view re-
labelling consistently improves the label accuracy overtime, which indicates
the effectiveness of our method.

Tab. 6 shows a study on the selection of different subsets from Tab. 1 for the
sample-selection when training the classification net $n_{\theta}(.)$. First,
we test the importance of classifying the samples in $\mathrm{RY}$ as clean
for training $n_{\theta}(.)$ by, instead, treating these samples as noisy in
Eq. (5) (i.e., by setting $\mathbf{w}_{i}=0$). This new sample selection
causes a large drop in performance for all cases, which suggests that
$\mathrm{RY}$ contains informative samples that are helpful for training
$n_{\theta}(.)$. Second, we test whether using the unmatched samples in
$\mathrm{U}$ can improve model training, where we include them as clean or
noisy samples by setting $\mathbf{w}_{i}=+1,0$, respectively. Both studies
lead to worse results compared to the original AsyCo that discards
$\mathrm{U}$ samples (see last row). Despite this result, we also notice that
in low noise rates (0.2, 0.3), treating $\mathrm{U}$ as clean leads to
slightly better accuracy than treating $\mathrm{U}$ as noisy. These results
suggest that the high uncertainty and lack of view agreements by the samples
in $\mathrm{U}$ lead to poor supervisory training signal, which means that
discarding these samples is currently the best option. Finally, the histograms
of Fig. 3 indicate that $\mathrm{NY}$ also contains small-loss samples.
Therefore, we make the traditional small-loss assumption to train our AsyCo
and use the subsets $\mathrm{C}$ and $\mathrm{NY}$ as clean and treat the
other subsets as noisy. As shown in the ”Small-loss subset” row of Tab. 6, the
accuracy is substantially lower, which suggests that the small-loss samples
may contain overfitted noisy-label samples.

We analyse the training of $r_{\phi}(.)$ with different training losses and
re-labelling strategies in Tab. 6. We first study how the multi-label training
loss provided by the BCE loss helps mitigate label noise by training our
reference net $r_{\theta}(.)$ with the CE loss $\ell_{CE}(.)$ in Eq. (1) and
(7), while keeping the multi-view sample selection and re-labelling strategies
unchanged. We observed that by training $r_{\theta}(.)$ with $\ell_{CE}(.)$
leads to a significant drop in accuracy for most cases, where for CIFAR10 with
low noise rate (20% and 30%), $\ell_{CE}(.)$ maintains the accuracy of
$\ell_{BCE}(.)$, but for larger noise rates, such as 40% and 50%,
$\ell_{CE}(.)$ is not competitive with $\ell_{BCE}(.)$ because it reduces the
prediction disagreements between $n_{\theta}(.)$ and $r_{\phi}(.)$,
facilitating the overfitting to the same noisy-label samples by both models.
For CIFAR100, $\ell_{CE}(.)$ leads to worse results than $\ell_{BCE}(.)$ for
all cases. These results suggest that to effectively co-teach two models with
prediction disagreement, the use of different training strategies is an
important component. Next, we study a training, where $r_{\phi}(.)$ is frozen
after warmup, but we still train $n_{\theta}(.)$. The result drops
significantly which indicates that $r_{\phi}(.)$ needs to be trained in
conjunction with $n_{\theta}(.)$ to achieve reasonable performance. We study
different re-labelling strategies by first setting
$\hat{\mathbf{y}}_{i}=\tilde{\mathbf{y}}$ for training $r_{\phi}(.)$, which
leads to comparable results for low noise rates, but worse results for high-
noise rates, suggesting that that only training with $\tilde{\mathbf{y}}$ is
not enough to achieve good performance. Finally, by setting
$\hat{\mathbf{y}}_{i}=\mathbf{\tilde{y}}^{(n)}$, we notice better but slightly
worse results than our proposed re-labelling from Eq. (6).

##  6 Conclusion

In this work, we introduced a new noisy label learning method called AsyCo.
Unlike previous SOTA noisy label learning methods that train two models with
the same strategy and select small-loss samples, AsyCo explores two different
training strategies and use multi-view consensus for sample selection. We show
in experiments that AsyCo outperforms previous methods in both synthetic and
real-world benchmarks. In the ablation study, we explore various subset
selection strategies for sample selection and re-labelling, which show the
importance of our design decisions. For future work, we will explore lighter
models for the reference net as only rank prediction is required. We will also
explore out-of-distribution (OOD) samples in noisy label learning because our
method currently assumes all samples are in-distribution.

## References

  * [1] Eric Arazo, Diego Ortego, Paul Albert, Noel O’Connor, and Kevin McGuinness.  Unsupervised label noise modeling and loss correction.  In International conference on machine learning, pages 312–321. PMLR, 2019.
  * [2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  Advances in neural information processing systems, 32, 2019.
  * [3] Avrim Blum and Tom Mitchell.  Combining labeled and unlabeled data with co-training.  In Proceedings of the eleventh annual conference on Computational learning theory, pages 92–100, 1998.
  * [4] Yingyi Chen, Xi Shen, Shell Xu Hu, and Johan AK Suykens.  Boosting co-teaching with compression regularization for label noise.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2688–2692, 2021.
  * [5] De Cheng, Tongliang Liu, Yixiong Ning, Nannan Wang, Bo Han, Gang Niu, Xinbo Gao, and Masashi Sugiyama.  Instance-dependent label-noise learning with manifold-regularized transition matrix estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16630–16639, 2022.
  * [6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [8] Arpit Garg, Cuong Nguyen, Rafael Felix, Thanh-Toan Do, and Gustavo Carneiro.  Instance-dependent noisy label learning via graphical modelling.  arXiv preprint arXiv:2209.00906, 2022.
  * [9] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  Advances in neural information processing systems, 31, 2018.
  * [10] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou.  Trusted multi-view classification.  arXiv preprint arXiv:2102.02051, 2021.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learningfor image recognition.  ComputerScience, 2015.
  * [12] Lu Jiang, Di Huang, Mason Liu, and Weilong Yang.  Beyond synthetic noise: Deep learning on controlled noisy labels.  In International Conference on Machine Learning, pages 4804–4815. PMLR, 2020.
  * [13] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In International conference on machine learning, pages 2304–2313. PMLR, 2018.
  * [14] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [16] Junnan Li, Richard Socher, and Steven CH Hoi.  Dividemix: Learning with noisy labels as semi-supervised learning.  arXiv preprint arXiv:2002.07394, 2020.
  * [17] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez.  A survey on deep learning in medical image analysis.  Medical image analysis, 42:60–88, 2017.
  * [18] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda.  Early-learning regularization prevents memorization of noisy labels.  Advances in neural information processing systems, 33:20331–20342, 2020.
  * [19] Tongliang Liu and Dacheng Tao.  Classification with noisy labels by importance reweighting.  IEEE Transactions on pattern analysis and machine intelligence, 38(3):447–461, 2015.
  * [20] Eran Malach and Shai Shalev-Shwartz.  Decoupling” when to update” from” how to update”.  Advances in neural information processing systems, 30, 2017.
  * [21] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [22] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu.  Making deep neural networks robust to label noise: A loss correction approach.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944–1952, 2017.
  * [23] Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor.  Asymmetric loss for multi-label classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 82–91, 2021.
  * [24] Min Shi, Yufei Tang, Xingquan Zhu, and Jianxun Liu.  Multi-label graph convolutional network representation learning.  IEEE Transactions on Big Data, 2020.
  * [25] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [26] Vikas Sindhwani, Partha Niyogi, and Mikhail Belkin.  A co-regularization approach to semi-supervised learning with multiple views.  In Proceedings of ICML workshop on learning with multiple views, volume 2005, pages 74–79. Citeseer, 2005.
  * [27] Hwanjun Song, Minseok Kim, and Jae-Gil Lee.  Selfie: Refurbishing unclean samples for robust deep learning.  In International Conference on Machine Learning, pages 5907–5915. PMLR, 2019.
  * [28] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017.
  * [29] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An.  Combating noisy labels by agreement: A joint training method with co-regularization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13726–13735, 2020.
  * [30] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama.  Part-dependent label noise: Towards instance-dependent label noise.  Advances in Neural Information Processing Systems, 33:7597–7610, 2020.
  * [31] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama.  Are anchor points really indispensable in label-noise learning?  Advances in Neural Information Processing Systems, 32, 2019.
  * [32] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang.  Learning from massive noisy labeled data for image classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2691–2699, 2015.
  * [33] Youjiang Xu, Linchao Zhu, Lu Jiang, and Yi Yang.  Faster meta update strategy for noise-robust deep learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 144–153, 2021.
  * [34] Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang.  Instance-dependent label-noise learning under a structural causal model.  Advances in Neural Information Processing Systems, 34:4409–4420, 2021.
  * [35] Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria.  Recent trends in deep learning based natural language processing.  ieee Computational intelligenCe magazine, 13(3):55–75, 2018.
  * [36] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama.  How does disagreement help generalization against label corruption?  In International Conference on Machine Learning, pages 7164–7173. PMLR, 2019.
  * [37] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  Communications of the ACM, 64(3):107–115, 2021.
  * [38] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, 2017.
  * [39] Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen.  Learning with feature-dependent label noise: A progressive approach.  arXiv preprint arXiv:2103.07756, 2021.
  * [40] Zhaowei Zhu, Tongliang Liu, and Yang Liu.  A second-order approach to learning with instance-dependent label noise.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10113–10123, 2021.


* McIntyre et al. (2023) McIntyre, J. R., Chen, C. H. K., & Larosa, A. 2023, The Astrophysical Journal, 957, 111, doi: 10.3847/1538-4357/acf3dd
  * McKinney et al. (2010) McKinney, W., et al. 2010, in Proceedings of the 9th Python in Science Conference, Vol. 445, Austin, TX, 51–56
  * Meyrand et al. (2016) Meyrand, R., Galtier, S., & Kiyani, K. H. 2016, Phys. Rev. Lett., 116, 105002, doi: 10.1103/PhysRevLett.116.105002
  * Meyrand et al. (2023) Meyrand, R., Squire, J., Mallet, A., & Chandran, B. D. G. 2023, Reflection-driven turbulence in the super-Alfvénic solar wind.  https://arxiv.org/abs/2308.10389
  * Meyrand et al. (2021) Meyrand, R., Squire, J., Schekochihin, A., & Dorland, W. 2021, Journal of Plasma Physics, 87, 535870301, doi: 10.1017/S0022377821000489
  * Moncuquet et al. (2020) Moncuquet, M., Meyer-Vernet, N., Issautier, K., et al. 2020, The Astrophysical Journal Supplement Series, 246, 44, doi: 10.3847/1538-4365/ab5a84
  * Monin & Jaglom (1987) Monin, A. S., & Jaglom, A. M. 1987, Statistical fluid mechanics. 2, 3rd edn. (Cambridge, Mass.: MIT Pr)
  * Montgomery & Turner (1981) Montgomery, D., & Turner, L. 1981, Physics of Fluids, 24, 825, doi: 10.1063/1.863455
  * Müller & Biskamp (2000) Müller, W.-C., & Biskamp, D. 2000, Phys. Rev. Lett., 84, 475, doi: 10.1103/PhysRevLett.84.475
  * Müller et al. (2003) Müller, W.-C., Biskamp, D., & Grappin, R. 2003, Phys. Rev. E, 67, 066302, doi: 10.1103/PhysRevE.67.066302
  * Müller & Grappin (2005) Müller, W.-C., & Grappin, R. 2005, Phys. Rev. Lett., 95, 114502, doi: 10.1103/PhysRevLett.95.114502
  * Ng & Bhattacharjee (1996) Ng, C. S., & Bhattacharjee, A. 1996, ApJ, 465, 845, doi: 10.1086/177468
  * Oboukhov (1962) Oboukhov, A. M. 1962, Journal of Fluid Mechanics, 13, 77, doi: 10.1017/S0022112062000506
  * Osman et al. (2012) Osman, K. T., Matthaeus, W. H., Wan, M., & Rappazzo, A. F. 2012, Phys. Rev. Lett., 108, 261102, doi: 10.1103/PhysRevLett.108.261102
  * Oughton & Matthaeus (2020) Oughton, S., & Matthaeus, W. H. 2020, ApJ, 897, 37, doi: 10.3847/1538-4357/ab8f2a
  * Oughton et al. (2017) Oughton, S., Matthaeus, W. H., & Dmitruk, P. 2017, ApJ, 839, 2, doi: 10.3847/1538-4357/aa67e2
  * Oughton et al. (2017) Oughton, S., Matthaeus, W. H., & Dmitruk, P. 2017, The Astrophysical Journal, 839, 2, doi: 10.3847/1538-4357/aa67e2
  * Palacios et al. (2022) Palacios, J. C., Bourouaine, S., & Perez, J. C. 2022, ApJ, 940, L20, doi: 10.3847/2041-8213/ac92f6
  * Parashar et al. (2019) Parashar, T. N., Cuesta, M., & Matthaeus, W. H. 2019, The Astrophysical Journal, 884, L57, doi: 10.3847/2041-8213/ab4a82
  * Parashar et al. (2020) Parashar, T. N., Goldstein, M. L., Maruca, B. A., et al. 2020, The Astrophysical Journal Supplement Series, 246, 58, doi: 10.3847/1538-4365/ab64e6
  * Parker & Tidman (1958) Parker, E. N., & Tidman, D. A. 1958, Physical Review, 111, 1206, doi: 10.1103/PhysRev.111.1206
  * Passot & Sulem (2019) Passot, T., & Sulem, P. L. 2019, Journal of Plasma Physics, 85, 905850301, doi: 10.1017/S0022377819000187
  * Passot et al. (2022) Passot, T., Sulem, P. L., & Laveder, D. 2022, Journal of Plasma Physics, 88, 905880312, doi: 10.1017/S0022377822000472
  * Passot et al. (2018) Passot, T., Sulem, P. L., & Tassi, E. 2018, Physics of Plasmas, 25, 042107, doi: 10.1063/1.5022528
  * Perez & Boldyrev (2009) Perez, J. C., & Boldyrev, S. 2009, Phys. Rev. Lett., 102, 025003, doi: 10.1103/PhysRevLett.102.025003
  * Perez & Chandran (2013) Perez, J. C., & Chandran, B. D. G. 2013, The Astrophysical Journal, 776, 124, doi: 10.1088/0004-637X/776/2/124
  * Perez et al. (2012) Perez, J. C., Mason, J., Boldyrev, S., & Cattaneo, F. 2012, Phys. Rev. X, 2, 041005, doi: 10.1103/PhysRevX.2.041005
  * Perez et al. (2014) —. 2014, The Astrophysical Journal Letters, 793, L13, doi: 10.1088/2041-8205/793/1/L13
  * Podesta (2009) Podesta, J. J. 2009, The Astrophysical Journal, 698, 986, doi: 10.1088/0004-637x/698/2/986
  * Podesta & Bhattacharjee (2010) Podesta, J. J., & Bhattacharjee, A. 2010, ApJ, 718, 1151, doi: 10.1088/0004-637X/718/2/1151
  * Podesta & Borovsky (2010) Podesta, J. J., & Borovsky, J. E. 2010, Physics of Plasmas, 17, 112905, doi: 10.1063/1.3505092
  * Podesta et al. (2009) Podesta, J. J., Chandran, B. D. G., Bhattacharjee, A., Roberts, D. A., & Goldstein, M. L. 2009, Journal of Geophysical Research (Space Physics), 114, A01107, doi: 10.1029/2008JA013504
  * Politano & Pouquet (1995) Politano, H., & Pouquet, A. 1995, Phys. Rev. E, 52, 636, doi: 10.1103/PhysRevE.52.636
  * Pucci & Velli (2014) Pucci, F., & Velli, M. 2014, ApJ, 780, L19, doi: 10.1088/2041-8205/780/2/L19
  * Pulupa et al. (2017) Pulupa, M., Bale, S. D., Bonnell, J. W., et al. 2017, Journal of Geophysical Research: Space Physics, 122, 2836, doi: 10.1002/2016JA023345
  * Roberts et al. (1987) Roberts, D. A., Goldstein, M. L., Klein, L. W., & Matthaeus, W. H. 1987, Journal of Geophysical Research: Space Physics, 92, 12023, doi: https://doi.org/10.1029/JA092iA11p12023
  * Roberts et al. (1992) Roberts, D. A., Goldstein, M. L., Matthaeus, W. H., & Ghosh, S. 1992, \jgr, 97, 17115, doi: 10.1029/92JA01144
  * Ruffolo et al. (2020) Ruffolo, D., Matthaeus, W. H., Chhiber, R., et al. 2020, ApJ, 902, 94, doi: 10.3847/1538-4357/abb594
  * Ruzmaikin et al. (1995) Ruzmaikin, A. A., Feynman, J., Goldstein, B. E., Smith, E. J., & Balogh, A. 1995, J. Geophys. Res., 100, 3395, doi: 10.1029/94JA02808
  * Sahraoui et al. (2009) Sahraoui, F., Goldstein, M. L., Robert, P., & Khotyaintsev, Y. V. 2009, Phys. Rev. Lett., 102, 231102, doi: 10.1103/PhysRevLett.102.231102
  * Salem et al. (2009) Salem, C., Mangeney, A., Bale, S. D., & Veltri, P. 2009, ApJ, 702, 537, doi: 10.1088/0004-637X/702/1/537
  * Schekochihin (2022) Schekochihin, A. A. 2022, Journal of Plasma Physics, 88, 155880501, doi: 10.1017/S0022377822000721
  * Schekochihin et al. (2009a) Schekochihin, A. A., Cowley, S. C., Dorland, W., et al. 2009a, \apjs, 182, 310, doi: 10.1088/0067-0049/182/1/310
  * Schekochihin et al. (2009b) —. 2009b, The Astrophysical Journal Supplement Series, 182, 310, doi: 10.1088/0067-0049/182/1/310
  * Shankarappa et al. (2023) Shankarappa, N., Klein, K. G., & Martinović, M. M. 2023, The Astrophysical Journal, 946, 85, doi: 10.3847/1538-4357/acb542
  * She & Leveque (1994) She, Z.-S., & Leveque, E. 1994, Phys. Rev. Lett., 72, 336, doi: 10.1103/PhysRevLett.72.336
  * Shebalin et al. (1983) Shebalin, J. V., Matthaeus, W. H., & Montgomery, D. 1983, Journal of Plasma Physics, 29, 525–547, doi: 10.1017/S0022377800000933
  * Shi et al. (2023) Shi, C., Sioulas, N., Huang, Z., et al. 2023, arXiv e-prints, arXiv:2308.12376, doi: 10.48550/arXiv.2308.12376
  * Sioulas (2023) Sioulas, N. 2023, MHDTurbPy, 0.1.0, Zenodo, doi: 10.5281/zenodo.7572468
  * Sioulas et al. (2022) Sioulas, N., Huang, Z., Velli, M., et al. 2022, The Astrophysical Journal, 934, 143, doi: 10.3847/1538-4357/ac7aa2
  * Sioulas et al. (2023) Sioulas, N., Velli, M., Huang, Z., et al. 2023, The Astrophysical Journal, 951, 141, doi: 10.3847/1538-4357/acc658
  * Sioulas et al. (2023) Sioulas, N., Huang, Z., Shi, C., et al. 2023, ApJ, 943, L8, doi: 10.3847/2041-8213/acaeff
  * Soni et al. (2024) Soni, S. L., Akhavan-Tafti, M., Suen, G. H. H., et al. 2024, Switchback Patches Evolve into Microstreams via Magnetic Relaxation.  https://arxiv.org/abs/2402.13964
  * Sorriso-Valvo et al. (1999) Sorriso-Valvo, L., Carbone, V., Veltri, P., Consolini, G., & Bruno, R. 1999, Geophysical Research Letters, 26, 1801, doi: https://doi.org/10.1029/1999GL900270
  * Sorriso-Valvo et al. (2023) Sorriso-Valvo, L., Marino, R., Foldes, R., et al. 2023, A&A, 672, A13, doi: 10.1051/0004-6361/202244889
  * Sorriso-Valvo et al. (2021) Sorriso-Valvo, L., Yordanova, E., Dimmock, A. P., & Telloni, D. 2021, The Astrophysical Journal Letters, 919, L30, doi: 10.3847/2041-8213/ac26c5
  * Squire et al. (2020) Squire, J., Chandran, B. D. G., & Meyrand, R. 2020, The Astrophysical Journal Letters, 891, L2, doi: 10.3847/2041-8213/ab74e1
  * Squire et al. (2022a) Squire, J., Meyrand, R., Kunz, M. W., et al. 2022a, Nature Astronomy, 6, 715, doi: 10.1038/s41550-022-01624-z
  * Squire et al. (2022b) —. 2022b, Nature Astronomy, 6, 715, doi: 10.1038/s41550-022-01624-z
  * Stix (1992) Stix, T. H. 1992, Waves in plasmas
  * Strauss (1976) Strauss, H. R. 1976, Physics of Fluids, 19, 134, doi: 10.1063/1.861310
  * Telloni (2022) Telloni, D. 2022, Frontiers in Astronomy and Space Sciences, 9, doi: 10.3389/fspas.2022.917393
  * Tenerani et al. (2021) Tenerani, A., Sioulas, N., Matteini, L., et al. 2021, The Astrophysical Journal Letters, 919, L31, doi: 10.3847/2041-8213/ac2606
  * Tu & Marsch (1995) Tu, C. Y., & Marsch, E. 1995, Space Sci. Rev., 73, 1, doi: 10.1007/BF00748891
  * Tu et al. (1990) Tu, C. Y., Marsch, E., & Rosenbauer, H. 1990, Geophys. Res. Lett., 17, 283, doi: 10.1029/GL017i003p00283
  * Turner et al. (2011) Turner, A. J., Gogoberidze, G., Chapman, S. C., Hnat, B., & Müller, W. C. 2011, Phys. Rev. Lett., 107, 095002, doi: 10.1103/PhysRevLett.107.095002
  * Uzdensky & Loureiro (2016) Uzdensky, D. A., & Loureiro, N. F. 2016, Phys. Rev. Lett., 116, 105003, doi: 10.1103/PhysRevLett.116.105003
  * Van Rossum & Drake Jr (1995) Van Rossum, G., & Drake Jr, F. L. 1995, Python reference manual (Centrum voor Wiskunde en Informatica Amsterdam)
  * Vech & Chen (2016) Vech, D., & Chen, C. H. K. 2016, The Astrophysical Journal Letters, 832, L16, doi: 10.3847/2041-8205/832/1/L16
  * Vech et al. (2018) Vech, D., Mallet, A., Klein, K. G., & Kasper, J. C. 2018, The Astrophysical Journal Letters, 855, L27, doi: 10.3847/2041-8213/aab351
  * Velli (1993) Velli, M. 1993, A&A, 270, 304
  * Velli et al. (1989) Velli, M., Grappin, R., & Mangeney, A. 1989, Physical Review Letters, 63, 1807, doi: 10.1103/PhysRevLett.63.1807
  * Velli et al. (1990) Velli, M., Grappin, R., & Mangeney, A. 1990, Computer Physics Communications, 59, 153, doi: 10.1016/0010-4655(90)90165-W
  * Velli et al. (1991) Velli, M., Grappin, R., & Mangeney, A. 1991, Geophysical & Astrophysical Fluid Dynamics, 62, 101, doi: 10.1080/03091929108229128
  * Verdini & Grappin (2012) Verdini, A., & Grappin, R. 2012, Phys. Rev. Lett., 109, 025004, doi: 10.1103/PhysRevLett.109.025004
  * Verdini & Grappin (2015) Verdini, A., & Grappin, R. 2015, ApJ, 808, L34, doi: 10.1088/2041-8205/808/2/L34
  * Verdini et al. (2019) Verdini, A., Grappin, R., Alexandrova, O., et al. 2019, MNRAS, 486, 3006, doi: 10.1093/mnras/stz1041
  * Verdini et al. (2018) Verdini, A., Grappin, R., Alexandrova, O., & Lion, S. 2018, ApJ, 853, 85, doi: 10.3847/1538-4357/aaa433
  * Verdini et al. (2009) Verdini, A., Velli, M., & Buchlin, E. 2009, ApJ, 700, L39, doi: 10.1088/0004-637X/700/1/L39
  * Vinogradov et al. (2023) Vinogradov, A., Alexandrova, O., Démoulin, P., et al. 2023, Embedded coherent structures from MHD to sub-ion scales in turbulent solar wind at 0.17 au.  https://arxiv.org/abs/2307.10478
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261, doi: 10.1038/s41592-019-0686-2
  * Völk & Aplers (1973) Völk, H. J., & Aplers, W. 1973, Ap&SS, 20, 267, doi: 10.1007/BF00642204
  * Wang et al. (2020) Wang, T., He, J., Alexandrova, O., Dunlop, M., & Perrone, D. 2020, ApJ, 898, 91, doi: 10.3847/1538-4357/ab99ca
  * Wang et al. (2023) Wang, X., Huang, L., Wang, Y., & Yuan, H. 2023, Universe, 9, doi: 10.3390/universe9090399
  * Wang et al. (2022) Wang, Y., Chhiber, R., Adhikari, S., et al. 2022, Strategies for determining the cascade rate in MHD turbulence: isotropy, anisotropy, and spacecraft sampling, arXiv, doi: 10.48550/ARXIV.2209.00208
  * Wicks et al. (2011) Wicks, R. T., Horbury, T. S., Chen, C. H. K., & Schekochihin, A. A. 2011, Phys. Rev. Lett., 106, 045001, doi: 10.1103/PhysRevLett.106.045001
  * Wicks et al. (2013a) Wicks, R. T., Mallet, A., Horbury, T. S., et al. 2013a, Phys. Rev. Lett., 110, 025003, doi: 10.1103/PhysRevLett.110.025003
  * Wicks et al. (2013b) Wicks, R. T., Roberts, D. A., Mallet, A., et al. 2013b, The Astrophysical Journal, 778, 177, doi: 10.1088/0004-637X/778/2/177
  * Wu et al. (2022) Wu, H., He, J., Yang, L., et al. 2022, On the scaling and anisotropy of two subranges in the inertial range of solar wind turbulence, arXiv, doi: 10.48550/ARXIV.2209.12409
  * Wu et al. (2023) Wu, H., Huang, S., Wang, X., et al. 2023, The Astrophysical Journal Letters, 947, L22, doi: 10.3847/2041-8213/acca20
  * Wu et al. (2013) Wu, P., Perri, S., Osman, K., et al. 2013, ApJ, 763, L30, doi: 10.1088/2041-8205/763/2/L30
  * Yao et al. (2011) Yao, S., He, J. S., Marsch, E., et al. 2011, ApJ, 728, 146, doi: 10.1088/0004-637X/728/2/146
  * Zhang et al. (2022) Zhang, J., Huang, S. Y., He, J. S., et al. 2022, The Astrophysical Journal Letters, 924, L21, doi: 10.3847/2041-8213/ac4027
  * Zhao et al. (2022) Zhao, G. Q., Meyrand, R., Feng, H. Q., Wu, D. J., & Kasper, J. C. 2022, The Astrophysical Journal, 938, 124, doi: 10.3847/1538-4357/ac9380
  * Zhao et al. (2023) Zhao, S., Yan, H., Liu, T. Z., Yuen, K. H., & Wang, H. 2023, arXiv e-prints, arXiv:2301.06709, doi: 10.48550/arXiv.2301.06709
  * Zhdankin et al. (2016) Zhdankin, V., Boldyrev, S., & Chen, C. H. K. 2016, MNRAS, 457, L69, doi: 10.1093/mnrasl/slv208