# Multi-Label Gold Asymmetric Loss Correction with Single-Label Regulators

Cosmin Octavian Pene
Delft University of Technology
Delft, The Netherlands
<EMAIL_ADDRESS>
&Amirmasoud Ghiassi
Delft University of Technology
Delft, The Netherlands
<EMAIL_ADDRESS>
&Taraneh Younesian
Delft University of Technology
Delft, The Netherlands
<EMAIL_ADDRESS>
&Robert Birke
ABB Research
Baden-Dättwil, Switzerland
<EMAIL_ADDRESS>
&Lydia Y.Chen
Delft University of Technology
Delft, The Netherlands
<EMAIL_ADDRESS>

###### Abstract

Multi-label learning is an emerging extension of the multi-class
classification where an image contains multiple labels. Not only acquiring a
clean and fully labeled dataset in multi-label learning is extremely
expensive, but also many of the actual labels are corrupted or missing due to
the automated or non-expert annotation techniques. Noisy label data decrease
the prediction performance drastically. In this paper, we propose a novel Gold
Asymmetric Loss Correction with Single-Label Regulators (GALC-SLR) that
operates robust against noisy labels. GALC-SLR estimates the noise confusion
matrix using single-label samples, then constructs an asymmetric loss
correction via estimated confusion matrix to avoid overfitting to the noisy
labels. Empirical results show that our method outperforms the state-of-the-
art original asymmetric loss multi-label classifier under all corruption
levels, showing mean average precision improvement up to 28.67% on a real
world dataset of MS-COCO, yielding a better generalization of the unseen data
and increased prediction performance.

##  1 Introduction

Real-world images naturally contain multiple objects corresponding to diverse
labels, which elevates deep learning models for multi-label learning. Multi-
label classification is an extension of multi-class classification where the
input is not assigned only a single label, but multiple ones.

It is extremely time consuming and expensive to collect high quality labels
for single-label images. Even long-standing and highly curated datasets, e.g.
CIFAR (Krizhevsky et al., 2009), contain wrong labels (Chen et al., 2019a).
Multi-label learning, where each image has multiple possible label
combinations, exacerbates this problem (Zhao and Gomes, 2021). For instance,
(Veit et al., 2017) shows that the Open Images dataset (Krasin et al., 2016),
which is widely used for multi-label and multi-class image classification,
contains 26.6% false positives among the training label set.

In single-label classification, label noise has been widely studied (Algan and
Ulusoy, 2021). Due to the memorization effect Deep Neural Networks (DNNs) can
overfit to noise degrading significantly their performance (Zhang et al.,
2017). Several techniques have been proposed to counter the effect of wrong
labels (Hendrycks et al., 2018; Han et al., 2018a; Patrini et al., 2017; Yao
et al., 2019). However, multi-label classification is a more complex problem.
As depicted in Fig. 1(a), each image comes with multiple labels including some
wrong and some clean ones. This has a negative impact on the performance of
DNNs. According to (Song et al., 2020), existing noise resilient methods for
single-label are not able to learn the correlation among multiple labels. Even
so little attention has been given to evaluating multi-label classifiers with
noisy labels (Zhao and Gomes, 2021).

To fill the gap in noise-resilient multi-label classifiers, we propose Gold
Asymmetric Loss Correction with Single-Label Regulators (GALC-SLR). GALC-SLR
assumes that a small subset of the training samples can be trusted. We use
this additional information to accurately estimate the noise corruption
matrix. Due to class imbalance and label correlations, learning the noise in
real-world multi-label datasets is more complicated than in real-world single-
label datasets. Hence, we introduce a novel method that uses single label
regulators to rebalance the predictions towards a targeted label. This leads
to accurate noise estimations used to correct the wrong labels during training
making the model robust to label noise even in the more challenging multi-
label setting.

In comparison to the state-of-the-art Asymmetric Loss (ASL) multi-label
classifier (Baruch et al., 2020) GALC-SLR is significantly more accurate under
label noise. ASL balances the probabilities of different samples by treating
positive and negative samples differently, i.e. asymmetrically. In empirical
evaluation on the MS-COCO dataset (Lin et al., 2014) GALC-SLR outperforms ASL
under all tested noise ratios from 0% to 60%. GALC-SLR improves the mean
Average Precision (mAP) over ASL on average by 13.81% and up to 28.67%.

The contributions of this paper are summarized as follows:

  * •

We design a noise estimation technique that uses trusted multi-label and
single-label data in order to calculate the corruption matrix.

  * •

Using our noise estimation, we design a robust multi-label classifier, GALC-
SLR, based on a loss correction approach.

  * •

We compare GALC-SLR against a state-of-the-art classifier under noisy labels
and study the behaviour of GALC-SLR in target ablation study experiments.

(a) Example of image with wrong labels

(b) Impact of increasing wrong label ratios

Figure 1: Wrong labels and their impact in multi-label classification.

###  1.1 Motivation example

Our motivation stems from the detrimental effects label noise in training data
can have on the model performance. We demonstrate this using the state-of-the-
art ASL (Baruch et al., 2020) method to train a TResNet-M (Ridnik et al.,
2021a) network on the MS-COCO dataset (Lin et al., 2014). ASL applied on
TResNet ranks top on the leader board for MLC on MS-
COCO111https://paperswithcode.com/sota/multi-label-classification-on-ms-coco
visited June 24, 2021.. We inject symmetric label noise (details in Section
4.1) at various corruption levels, from 0% to 60%, and report the mean average
precision to asses the impact of wrong labels. mAP is considered by many
recent works (Lanchantin et al., 2020; Chen et al., 2019b) as important metric
for performance evaluation in multi-label classification, since it takes into
consideration both false-negative and false-positive rates (Baruch et al.,
2020). Fig. 1(b) shows the results: each additional 10% noisy labels leads to
a 5%-8% reduction in mAP score. Since it is hard and costly to avoid label
noise (Zhao and Gomes, 2021), it is vital to develop robust classifiers that
can avoid overfitting to the label noise in the training data.

##  2 Related Work

Recent literature has shown increased interest towards robustness against
noisy labels in training, much more in single-label classification than in
multi-label learning. We first investigate the robust learning solutions in
single-label classification, followed by an analysis of the multi-label
context.

Single-label Classification Many researchers have been tackling the problem of
noisy labels in single-label classification. (Song et al., 2020) distinguishes
between five categories of robust DNNs. Some classifiers such as C-model
(Goldberger and Ben-Reuven, 2017), Contrastive-Additive Noise Network (Yao et
al., 2019) and Robust Generative Classifier (RoG) (Lee et al., 2019) rely on a
noise adaptation layer at the top of the softmax layer. Another solution is to
use regularization techniques such as data augmentation (Shorten and
Khoshgoftaar, 2019), weight decay (Krogh and Hertz, 1992), dropout (Srivastava
et al., 2014) and batch normalization (Ioffe and Szegedy, 2015). These methods
perform well on low to moderate noise, but fail on datasets with higher noise
(Tanno et al., 2019). Other works use Sample Selection showing impressive
results even under heavy noise. Examples are MentorNet (Jiang et al., 2018),
Co-teaching (Han et al., 2018a) and Co-teaching+ (Yu et al., 2019). Another
stream of work estimates the noise corruption matrix to correct the labels
during training, without, e.g. Forward (Patrini et al., 2017) and Masking (Han
et al., 2018b), or with, e.g. Golden Loss Correction (Hendrycks et al., 2018),
a small portion of trusted, i.e. verified clean, data.

Multi-label Classification In multi-label learning, little attention has been
given to the consequences of label noise (Song et al., 2020). Few papers treat
noisy labels in the multi-label context. (Sun et al., ) learns by a low-rank
and sparse decomposition approach to obtain ground-truth and irrelevant label
matrices. (Fang and Zhang, 2019) introduces label confidence to recover the
true labels. (Xie and Huang, 2020) formalizes the two objective of recovering
ground-truth and identify noisy labels via a unified regulators-based
framework. The topic is gaining traction with some more works available as
technical reports. (Zhao and Gomes, 2021) leverages context to identify noisy
labels. (Li et al., 2020) proposes a two-step noise correction. (Bai et al.,
2020) acknowledges and evaluates the impact of noisy labels even if it is not
the main goal of the authors.

In contrast to the above, GALC-SLR aims to leverage single-label regulators
together with a small fraction of trusted data to avoid overfitting to noisy
labels in multi-label classification.

##  3 Methodology

###  3.1 Notation

Consider the multi label dataset
$\mathcal{D}=\\{(\bm{x}_{i},\tilde{\bm{y}}_{i})\\}_{i=1}^{N}$ where
$\bm{x}_{i}\subset\mathbb{R}^{d}$ denotes the $i^{th}$ sample out of $N$ with
$d$ features. $\tilde{\bm{y}}_{i}\subset[0,1]^{K}$ denotes the corresponding
label vector over $K$ classes. The label vector is affect by noise, hence
$\tilde{\bm{y}}$ can be clean ($\bm{y}$) or noisy ($\hat{\bm{y}}$). Similar to
GLC (Hendrycks et al., 2018), we assume that a subset of the data, i.e. gold
dataset $\mathcal{G}\subset\mathcal{D}$, can be trusted. $|\mathcal{G}|$
contains samples $(\bm{x},\bm{y})$ with no corrupted labels. We refer to the
rest of the samples $(\bm{x},\hat{\bm{y}})$ with potentially corrupted labels
as silver dataset $\mathcal{S}=\mathcal{D}-\mathcal{G}$. We define the trusted
fraction as the ratio $\frac{|\mathcal{G}|}{|\mathcal{G}|+|\mathcal{S}|}$.
Furthermore, we assume that a small dataset of clean single-label images
$\mathcal{G}_{S}$ is available. We use these sets to train a _silver_
$f(.;\theta)$ and a _gold_ $g(.;\phi)$ classifier and estimate the noise given
by a $K\times K$ noise corruption matrix C. The elements $\bm{C}_{ij}$ are the
probability of label $i$ to be flipped into label $j$, formally:

$\bm{C}_{ij}=p(\widehat{y}_{j}=1\wedge\widehat{y}_{i}=0|y_{j}=0\wedge
y_{i}=1)$

###  3.2 Overview of GALC-SLR

We propose a novel Gold Asymmetric Loss Correction with Single-Label
Regulators training approach. GALC-SLR combines an asymmetric loss approach
with a gold loss correction approach to counter noisy labels. The asymmetric
loss treats relevant and irrelevant labels differently and has been shown to
obtain impressive results on several MLC datasets (Baruch et al., 2020). The
gold loss correction (GLC) assumes that a small subset of trusted samples is
available to accurately estimate the true corruption matrix. It is a powerful
method that achieves impressive results under both symmetric and asymmetric
noise in the single-label setting (Hendrycks et al., 2018). The original GLC
assumes conditional independence of $\bm{y}$ given $\bm{x}$. This assumption
holds when $\bm{y}$ is deterministic in $\bm{x}$. This does not hold for
multi-label classification, because an image can have multiple labels.
Furthermore, in multi-label classification there can be label correlations
that the original formula does not take into account, making it impossible to
target a specific label. To derive an accurate multi-label, rather than
single-label, noise corruption matrix GALC-SLR uses single-label regulators
and sigmoid classification which gives a more reliable representation of the
noise in the multi-label context.

Figure 2 presents an overview of the GALC-SLR method. It includes three steps.
Step 1: we train a classifier $f(.;\theta)$ using ASL loss on the noisy
samples in $\mathcal{S}$. Step 2 is the heart of GALC-SLR. We use $f$ with the
trusted samples in $\mathcal{G}$ to estimate the noise corruption matrix and
correct it via single-label regulators derived via the samples in
$\mathcal{G}_{S}$. Step 3: we train the final classifier $g(.;\phi)$ using ASL
loss on samples from $\mathcal{G}$ and corrected samples from $\mathcal{S}$.

Figure 2: Overview of GALC-SLR

###  3.3 Noise Corruption Matrix Estimation

First, we train a silver classifier
$f(\bm{x};\theta)=\widehat{p}(\hat{\bm{y}}|\bm{x})$ on $\mathcal{S}$, using
the asymmetric loss from (Baruch et al., 2020):

$\mathcal{L}_{ASL}=\left\\{\begin{array}[]{@{}ll@{}}L_{+}=(1-p)^{\gamma_{+}}\log(p)\\\
L_{-}=(p_{m})^{\gamma_{-}}\log(1-p_{m})\end{array}\right.$

where $L_{+}$ and $L_{-}$ are the positive and negative loss parts used for
relevant and irrelevant labels, respectively. $p$ is the network output
probability and $\gamma_{+},\gamma_{-}$ are the focusing parameters. Finally,
$p_{m}=\max(p-m,0)$ denotes the shifted probability by a margin hyper
parameter $m$. Given the labels in $\mathcal{S}$ are potentially corrupted,
$f$ is not a reliable classifier for our final predictions. However, we can
use $f$ to estimate our multi-label corruption matrix $\widehat{\bm{C}}$.

Algorithm 1 GALC-SLR $\mathbf{\widehat{C}}$ estimation

1:Input: Untrusted data $\mathcal{S}$, trusted single-label data
$\mathcal{G}_{S}$, silver classifier $f$

2:Output: Estimated $\widehat{\bm{C}}$

3:Fill $Reg\in\mathbb{R}^{K\times K}$ with zeros /* Calculate Regulator for
each label */

4:for $k=1,...,K$ do

5: $num\\_examples=0$

6: for $(\bm{x}_{i},\bm{y}_{i})\in\mathcal{G}_{S}$ such that $\bm{y}_{ik}=1$
do /* Mean of silver softmax predictions */

7: $num\\_examples\mathrel{+}=1$

8: $Reg_{k\bullet}\mathrel{+}=f_{soft}(\bm{x}_{i})$

9: end for

10: $Reg_{k\bullet}\mathrel{/}=num\\_examples$

11:end for

12:Fill $\widehat{\bm{C}}\in\mathbb{R}^{K\times K}$ with zeros /* Estimate
multi-label corruption matrix */

13:for $k=1,...,K$ do

14: $num\\_examples=0$

15: for $(\bm{x}_{i},\bm{y}_{i})\in\mathcal{S}$ such that $\bm{y}_{ik}=1$ do
/* For each image that contains label k */

16: $num\\_examples\mathrel{+}=1$, $num\\_other\\_labels=0$

17: Fill $regulators\in\mathbb{R}^{K}$ with zeros

18: for $p=1,...,K$ such that $p\neq k\And\bm{y}_{ip}=1$ do /* Sum up other
labels */

19: $num\\_other\\_labels\mathrel{+}=1$

20: $regulators\mathrel{+}=Reg_{p\bullet}$

21: end for

22: $\widehat{\bm{C}}_{k\bullet}\mathrel{+}=f_{sig}(\bm{x}_{i})-regulators$ /*
Correct sigmoid prediction via regulators */

23:
$\widehat{\bm{C}}_{k\bullet}\mathrel{+}=Reg_{k\bullet}*num\\_other\\_labels$
/* Rebalance towards target label $k$*/

24: end for

25: $\widehat{\bm{C}}_{k\bullet}\mathrel{/}=num\\_examples$

26:end for

27:$\widehat{\bm{C}}=sig(\widehat{\bm{C}})$ /* Final scaling */

Algorithm 1 depicts our novel multi-label corruption matrix estimation. First,
we calculate the single-label regulators by taking the average of our silver
softmax predictions for each label (lines 3-10):

$Reg_{k\bullet}=\frac{\sum_{i=1}^{N}f_{soft}(\bm{x_{i}})}{N},\,\,\,\forall
k\in K$

where $Reg\subset\mathbb{R}^{K\times K}$ and $Reg_{k\bullet}$ denotes the
$k^{th}$ matrix row. This is the main step in our method. It not only allows
to target a specific label $k$ for its noise corruption estimation, but also
to regulate the label correlations from the multi-label images. The next step
is to explicitly estimate the noise corruption matrix. For each label $k$, we
sum the rows in $Reg$ for the labels which are present in the image except $k$
(line 18-21). Next, for the each row of the noise corruption matrix, we
regulate the sigmoid predictions of $f$ by subtracting the summed regulators
(line 22). Finally, we rebalance each row using the number of other labels in
the image (line 23) and take the average over the number of samples (line 25)
scaled via a sigmoid (line 27). Fig. 3 compares the multi-label corruption
matrix estimated by GALC-SLR for 40% symmetric label noise against the
injected –ground truth– one, and the GLC estimated one. We observe that GALC-
SLR’s estimation is more resistant to imbalanced data with respect to GLC’s
estimation. This can be seen from the darker, closer to the truth, diagonal
values and the more pronounced difference with respect to the off-diagonal
values. Note that to highlight this effect we avoid the last sigmoid scaling
for better contrast in the figures.

With the estimated noise corruption matrix $\mathbf{\widehat{C}}$, we finally
train the robust gold classifier $g(.;\phi)$. We correct labels of the samples
in $\mathcal{S}$ via $\mathbf{\widehat{C}}$ while leveraging samples in
$\mathcal{G}$ as is. The loss function follows as:

$\begin{array}[]{@{}lll@{}}\ell&=\mathcal{L}_{ASL}(\widehat{\bm{C}}^{T}g_{sig}(\bm{x}),\hat{\bm{y}}),&\forall
x\in\mathcal{S}\\\ \ell&=\mathcal{L}_{ASL}(g(\bm{x}),\bm{y})&\forall
x\in\mathcal{G}.\end{array}$

##  4 Evaluation

###  4.1 Experiment setup

Datasets. We evaluate GALC-SLR using MS-COCO (Lin et al., 2014) dataset. MS-
COCO is a popular real-world dataset of common objects in context widely used
for evaluation of multi-label classification. The training and validation
datasets contain 82K and 40K images, respectively. Each image is tagged on
average with 2.9 labels belonging to 80 classes.

To explicitly test GALC-SLR in the more challenging multi-label setting, we
remove all single label images from both the training and validation datasets.
This does not only elicit a more reliable evaluation, but it also allows to
collect single-label samples to construct $\mathcal{G}_{S}$. The number of
images per class label varies from 1, for unpopular classes, to 1234, for the
most popular class with an average of 210.2 images per class. More single-
label samples allow to estimate more accurate regulators which in turn leads
to a more robust classifier. Finally, we split the training data into a gold
($\mathcal{G}$) and a silver ($\mathcal{S}$) datasets. As base we use 10% as
gold data, leading to 6.5K clean samples and 58.7K samples injected with noisy
labels.

Label Noise. Label noise in multi-label data is more complex than in a single-
label context since each sample has an arbitrary number of labels. We follow
previous works (Jiang et al., 2018; Patrini et al., 2017) and inject symmetric
noise, but with an extra step. Specifically, we select a fraction $\eta$, i.e.
the noise ratio, of labels and flip them to another class with uniform
probability. This corresponds to a noise corruption matrix having elements
$C_{ij}$ as follows:

$C_{ij}=\Bigg{\\{}\begin{array}[]{rl}1-\eta&\text{if}\;i=j\\\
\dfrac{\eta}{K-1}&\text{if}\;i\neq j\\\ \end{array}$

In order to ensure wrong label injection, we test whether or not the new label
is already associated with the image. If it does, we repeatedly elect a new
label until we select one which is not yet present. In order to evaluate how
robust GALC-SLR is to noise, we test our method against multiple noise ratios
–from 0% to 60%.

(a) True

(b) GALC-SLR

(c) GLC

Figure 3: Comparison between multi-label corruption matrices with 40% noise.

Evaluation Metrics. For a comprehensive and reliable evaluation, we follow
conventional settings and report the following metrics: mean average precision
(mAP), average per-class F1 (CF1) and average overall F1 (OF1). These metrics
have been widely used in literature to evaluate multi-label classification
(Baruch et al., 2020; Song et al., 2020; Wang et al., 2020). and have been
shown to dramatically decrease with label noise (Zhao and Gomes, 2021). Note
that only the training set is affected by noise, whereas the evaluation
metrics are computed on the clean testing set.

DNN Architecture. As base architecture for the DNN we use TResNet (Ridnik et
al., 2021a). TResNet network is a high performance GPU-dedicated architecture
based on ResNet50 designed to increase the model prediction performance,
without increasing training or inference time. The TResNet network is pre-
trained on the ImageNet-21K dataset for better generalizability and increased
prediction accuracy (Ridnik et al., 2021b). In particular we use the TResNet-M
version.

Baseline. As baseline we compare against ASL (Baruch et al., 2020) using the
code provided by the authors. GALC-SLR assumes access to a small subset of
clean samples. For a fair comparison, we test ASL and GALC-SLR on the same
datasets with the same label noise. The only additional knowledge of our
method is which labels are trusted, i.e. belonging to the small golden dataset
$\mathcal{G}$, and which are potentially corrupted.

Implementation Details. We use PyTorch v1.9.0 for both GALC-SLR and ASL, and
the default parameters provided in (Baruch et al., 2020) except that we always
take the last trained model due to the memorization effect. The number of
training epochs is an important parameter for a reliable evaluation,
especially in a noisy setting. DNNs are shown to present the so-called
memorization effect (Xu et al., 2021; Feldman and Zhang, 2020; Feldman, 2020)
benefiting in general from this factor to achieve a better prediction
performance in atypical samples. However, (Arpit et al., 2017) suggests that
with noisy data, DNNs prioritize learning simple patterns first. From
preliminary experiments we see that 80 epochs are enough for the learning to
stabilize.

###  4.2 Results

In this subsection we empirically compare the performance of GALC-SLR to the
performance of ASL under 0% to 60% symmetric noise. We aim to show the
effectiveness of our GALC-SLR in robustly learning from noisy data.

Fig. 4 shows the comparison results. The performance of both systems decreases
under increasing noise levels, but GALC-SLR is significantly more robust. In
terms of mAP GALC-SLR consistently outperforms ASL for all noise ratios (see
Fig. 4(a)). ASL’s performance drops an average of 5.34% points with each 10%
noise, while GALC-SLR’s performance decreases with only 1.07% points. Under
severe noise, i.e. 60%, the gap between GALC-SLR and ASL is more than 28%
points and only 9.6% points worse than without noise. In comparison ASL drops
by 38.1% points from 0% to 60% noise. This shows that GALC-SLR is robust even
to high noise levels. Similar results apply for both CF1 and OF1, see Fig.
4(b) and Fig. 4(c), respectively. Even if ASL is slightly better in the no
noise case, the performance quickly degrades with additional noise. At 60%
GALC-SLR is better by 24.3% and 26.1% points for CF1 and OF1, respectively.

(a) Mean Average Precision (mAP)

(b) Average per-class F1 (CF1)

(c) Average overall F1 (OF1)

(d) Memorization effect under 40% noise

Figure 4: Evaluation of GALC-SLR and ASL on MS-COCO with symmetric label
noise.

To reliably assess the correctness of GALC-SLR, we also investigate the
observed memorization effect for GALC-SLR (depicted in Fig. 4(d)). Both GALC-
SLR and ASL follow the same trend. First they learn the easy patterns,
achieving a high accuracy after just a few epochs. However, afterwards the
performance slowly degrades over training effort and finally stabilizes after
60 epochs. The figure clearly shows the advantage of GALC-SLR over ASL in the
different levels at which they plateau. Moreover, one can observe that GALC-
SLR has a slight delay in learning at the beginning of the training, i.e.
GALC-SLR peaks at epoch 10, while ASL at epoch 6. This observation indicates
that GALC-SLR does not help in terms of learning speed nor in reaching a
higher performance during training, but by preventing to overfit to the noisy
labels. This makes the DNN more resistant to wrong label information. This
suggests that our method can also be applied to other existing classifiers and
domains.

###  4.3 Ablation Study

To better understand the performance of GALC-SLR, we perform extra ablation
studies to investigate the effects of: i) errors in the noise corruption
matrix estimation; ii) impact of the gold dataset size (both studied in
experiment I); and iii) impact of number of single-label images (studied in
experiment II). The base setup of the experiments is the same as in Section
4.1 with the only changes specifically mentioned.

Experiment I: Fig. 3 shows visually the difference between the true and our
estimated noise corruption matrix. To assess also quantitatively how good our
estimation method works, we train classifier $g$ with the true corruption
matrix. Fig. 5(a) compares the achieved mAP results under 40% noise. $g$
trained with the true corruption matrix represents the upper performance bound
achievable by noise corruption matrix estimators. Fig. 5(a) shows the results
across the bars of different colors. Since GALC-SLR uses trusted data to
estimate the noise corruption matrix and train the robust classifier $g$, we
expect the size of $\mathcal{G}$ to have an impact on the estimation accuracy
and consequently on model performance. We investigate this effect by repeating
the previous experiments with halve the fraction of trusted data, i.e. 5%.
This corresponds to 3,263 clean samples and 62,005 samples injected with noisy
labels. Fig. 5(a) shows these results via the two different bar plot groups.

With estimated noise corruption matrix we reach 68.73% and 69.94% mAP and
69.11% and 70.30% using the true noise correction matrix under 5% and 10% of
trusted data, respectively. The difference between GALC-SLR and the upper
bound (True-Correction) is below 0.5% points in both cases. This shows that
our noise estimation is able to capture almost perfectly the impact of the
noise corruption matrix, and that it works even with reduced amount of trusted
data.

Experiment II: To assess the impact of trusted single-label images on the
estimation of the corruption matrix, we conduct two extra experiments with
varying images per class. In addition to the previous case using all single-
label images, we limit the number of single-label images per class to 50 and
10, referred to as GALC-SLR-L50 and GALC-SLR-L10, respectively. This results
in a total of 2824 for GALC-SLR-L50 and 721 for GALC-SLR-L10 single-label
images used.

Fig. 5(b) shows the impact on the mAP over training epochs. One can observe
that limiting the number of single-label images has only a minor impact on the
performance of GALC-SLR. Hence our proposed method is not only robust to wrong
labels in multi label learning but it also can estimate an accurate noise
corruption matrix by using only a small proportion of trusted single-label
data. In other words, GALC-SLR has a limited dependency on the amount of clean
single-label data.

(a) Impact of trusted data

(b) Impact of single-label data

Figure 5: Ablation study of GALC-SLR on MS-COCO

##  5 Conclusion

In this paper, we show the impact of wrong label information on multi-label
classification. Motivated by this, we propose the Gold Asymmetric Loss
Correction with Single-Label Regulators, a multi-label method that is robust
against label noise. This method assumes access to a small set of clean multi-
label examples as well as to a small set of clean single-label samples. GALC-
SLR uses this additional information in order to accurately model the label
noise distribution in a multi-label setting. Through a novel regularization
technique that rebalances predictions towards a targeted label, GALC-SLR
estimates a noise corruption matrix close to the true matrix. We evaluate
GALC-SLR on a real-world dataset under label noise, at multiple corruption
levels from low to heavy noise. Results show that GALC-SLR is a powerful
method that significantly improves robustness against label noise in multi-
label classification.

## References

  * Krizhevsky et al. [2009] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-10 (canadian institute for advanced research).  2009\.  URL http://www.cs.toronto.edu/~kriz/cifar.html.
  * Chen et al. [2019a] Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang.  Understanding and utilizing deep neural networks trained with noisy labels.  In _ICML_ , volume 97, pages 1062–1070, 2019a.
  * Zhao and Gomes [2021] W. Zhao and Carla P. Gomes.  Evaluating multi-label classifiers with noisy labels.  _ArXiv_ , abs/2102.08427, 2021.
  * Veit et al. [2017] Andreas Veit, N. Alldrin, Gal Chechik, Ivan Krasin, A. Gupta, and Serge J. Belongie.  Learning from noisy large-scale datasets with minimal supervision.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6575–6583, 2017.
  * Krasin et al. [2016] Ivan Krasin, Tom Duerig, Neil Alldrin, Andreas Veit, Sami Abu-El-Haija, Serge Belongie, David Cai, Zheyun Feng, Vittorio Ferrari, Victor Gomes, Abhinav Gupta, Dhyanesh Narayanan, Chen Sun, Gal Chechik, and Kevin Murphy.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  _Dataset available fromhttps://github.com/openimages_, 2016\.
  * Algan and Ulusoy [2021] G. Algan and I. Ulusoy.  Image classification with deep learning in the presence of noisy labels: A survey.  _Knowl. Based Syst._ , 215:106771, 2021.
  * Zhang et al. [2017] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  In _ICLR_ , 2017.
  * Hendrycks et al. [2018] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel.  Using trusted data to train deep networks on labels corrupted by severe noise.  In _NIPS_ , pages 10456–10465, 2018.
  * Han et al. [2018a] B. Han, Quanming Yao, Xingrui Yu, Gang Niu, M. Xu, Weihua Hu, I. Tsang, and Masashi Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In _NIPS_ , 2018a.
  * Patrini et al. [2017] Giorgio Patrini, A. Rozza, A. Menon, R. Nock, and Lizhen Qu.  Making deep neural networks robust to label noise: A loss correction approach.  _CVPR_ , pages 2233–2241, 2017.
  * Yao et al. [2019] J. Yao, J. Wang, I. Tsang, Ya Zhang, J. Sun, C. Zhang, and R. Zhang.  Deep learning from noisy image labels with quality embedding.  _IEEE Transactions on Image Processing_ , 28:1909–1922, 2019.
  * Song et al. [2020] Hwanjun Song, Minseok Kim, Dongmin Park, and J. Lee.  Learning from noisy labels with deep neural networks: A survey.  _ArXiv_ , abs/2007.08199, 2020.
  * Baruch et al. [2020] Emanuel Ben Baruch, T. Ridnik, Nadav Zamir, Asaf Noy, Itamar Friedman, M. Protter, and Lihi Zelnik-Manor.  Asymmetric loss for multi-label classification.  _ArXiv_ , abs/2009.14119, 2020.
  * Lin et al. [2014] Tsung-Yi Lin, M. Maire, Serge J. Belongie, James Hays, P. Perona, D. Ramanan, Piotr Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , 2014.
  * Ridnik et al. [2021a] Tal Ridnik, Hussam Lawen, Asaf Noy, Emanuel Ben Baruch, Gilad Sharir, and Itamar Friedman.  Tresnet: High performance gpu-dedicated architecture.  In _IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 1400–1409, 2021a.
  * Lanchantin et al. [2020] Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi.  General multi-label image classification with transformers.  _ArXiv_ , abs/2011.14027, 2020.
  * Chen et al. [2019b] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Y. Guo.  Multi-label image recognition with graph convolutional networks.  _CVPR_ , pages 5172–5181, 2019b.
  * Goldberger and Ben-Reuven [2017] J. Goldberger and E. Ben-Reuven.  Training deep neural-networks using a noise adaptation layer.  In _ICLR_ , 2017.
  * Lee et al. [2019] Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, B. Li, and Jinwoo Shin.  Robust inference via generative classifiers for handling noisy labels.  In _ICML_ , 2019.
  * Shorten and Khoshgoftaar [2019] Connor Shorten and Taghi Khoshgoftaar.  A survey on image data augmentation for deep learning.  _Journal of Big Data_ , 6, 07 2019.
  * Krogh and Hertz [1992] Anders Krogh and John Hertz.  A simple weight decay can improve generalization.  In _NIPS_ , volume 4. Morgan-Kaufmann, 1992.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _ICML_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pages 448–456, 2015.
  * Tanno et al. [2019] Ryutaro Tanno, A. Saeedi, S. Sankaranarayanan, D. Alexander, and N. Silberman.  Learning from noisy labels by regularized estimation of annotator confusion.  _CVPR_ , pages 11236–11245, 2019.
  * Jiang et al. [2018] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In _ICML_ , pages 2309–2318, 2018.
  * Yu et al. [2019] Xingrui Yu, B. Han, Jiangchao Yao, Gang Niu, I. Tsang, and Masashi Sugiyama.  How does disagreement help generalization against label corruption?  In _ICML_ , volume 97, pages 7164–7173, 2019.
  * Han et al. [2018b] Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor W. Tsang, Ya Zhang, and Masashi Sugiyama.  Masking: A new perspective of noisy supervision.  In _NIPS_ , pages 5841–5851, 2018b.
  * [28] Lijuan Sun, Songhe Feng, Tao Wang, Congyan Lang, and Yi Jin.  Partial multi-label learning by low-rank and sparse decomposition.  In _AAAI_ , pages 5016–5023.
  * Fang and Zhang [2019] Jun-Peng Fang and Min-Ling Zhang.  Partial multi-label learning via credible label elicitation.  In _AAAI_ , pages 3518–3525. AAAI Press, 2019.
  * Xie and Huang [2020] Ming-Kun Xie and Sheng-Jun Huang.  Partial multi-label learning with noisy label identification.  In _AAAI_ , pages 6454–6461, 2020.
  * Li et al. [2020] Junnan Li, Caiming Xiong, Richard Socher, and Steven C. H. Hoi.  Towards noise-resistant object detection with noisy annotations.  _CoRR_ , abs/2003.01285, 2020.
  * Bai et al. [2020] Junwen Bai, Shufeng Kong, and Carla P. Gomes.  Disentangled variational autoencoder based multi-label classification with covariance-aware multivariate probit model.  In _IJCAI_ , pages 4313–4321, 2020.
  * Wang et al. [2020] Ya Wang, Dongliang He, Fu Li, Xiang Long, Zhichao Zhou, Jinwen Ma, and Shilei Wen.  Multi-label classification with label graph superimposing.  In _AAAI_ , pages 12265–12272, 2020.
  * Ridnik et al. [2021b] Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  _CoRR_ , abs/2104.10972, 2021b.
  * Xu et al. [2021] Han Xu, Xiaorui Liu, Wentao Wang, Wenbiao Ding, Zhongqin Wu, Zitao Liu, Anil Jain, and Jiliang Tang.  Towards the memorization effect of neural networks in adversarial training, 2021.
  * Feldman and Zhang [2020] Vitaly Feldman and Chiyuan Zhang.  What neural networks memorize and why: Discovering the long tail via influence estimation.  In _NIPS_ , 2020.
  * Feldman [2020] V. Feldman.  Does learning require memorization? a short tale about a long tail.  _ACM SIGACT Symposium on Theory of Computing_ , 2020.
  * Arpit et al. [2017] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien.  A closer look at memorization in deep networks.  In _ICML_ , volume 70, pages 233–242, 2017.