# Evaluating Multi-label Classifiers with Noisy Labels

Wenting Zhao  Carla Gomes

###### Abstract

Multi-label classification (MLC) is a generalization of standard
classification where multiple labels may be assigned to a given sample. In the
real world, it is more common to deal with noisy datasets than clean datasets,
given how modern datasets are labeled by a large group of annotators on
crowdsourcing platforms, but little attention has been given to evaluating
multi-label classifiers with noisy labels. Exploiting label correlations now
becomes a standard component of a multi-label classifier to achieve
competitive performance. However, this component makes the classifier more
prone to poor generalization – it overfits labels as well as label
dependencies. We identify three common real-world label noise scenarios and
show how previous approaches perform poorly with noisy labels. To address this
issue, we present a Context-Based Multi-Label Classifier (CbMLC) that
effectively handles noisy labels when learning label dependencies, without
requiring additional supervision. We compare CbMLC against other domain-
specific state-of-the-art models on a variety of datasets, under both the
clean and the noisy settings. We show CbMLC yields substantial improvements
over the previous methods in most cases.

Machine Learning, ICML

##  1 Introduction

Unlike a traditional single-output learning problem where we assign one label
to a given sample, multi-label classification (MLC) aims to predict a set of
target objects simultaneously. MLC has been an active research topic in
domains including computer vision (Wang et al., 2016), natural language
processing (Nam et al., 2014), and recommendation systems (Liu et al., 2017).
Notable applications are assigning tags to online images, identifying topics
for web articles, and generating product recommendations. Developing an
effective MLC technique can lead to improvements in many real-world problems.

One challenge MLC faces is the introduction of complex noise in the annotation
process. A common way to collect labels is to use crowdsourcing platforms such
as Amazon Mechanical Turk (MTurk). Because a large-scale dataset could be
labeled by potentially thousands of different human annotators, the labels
they provide could naturally be in varying qualities (Snow et al., 2008;
Ipeirotis et al., 2010; Raykar et al., 2010; Yuen et al., 2011); the interface
a crowdsourcing platform provides can also introduce bias (Li et al., 2017).
We refer readers to Krasin et al. (2017) for an example of a data collection
process for a dataset of millions of samples, where 26.6% labels are false
positive (Veit et al., 2017)! Another source of labels is citizen science
projects. For example, in the _eBird_ project (Sullivan et al., 2009),
amateurs observe and record birds at locations they have been to, but one may
misidentify a species, and it is difficult to differentiate between one
species not being a habitant of a place and the species simply not yet being
observed at the place. To develop an accurate MLC method, it is important to
consider how label noise is introduced and how it may affect performance.

Figure 1: Mean average precision (mAP) curves on Pascal VOC produced by ML-GCN
and our method, CbMLC. We inject different types of asymmetric noise on the
trainval set and test with the clean test set. ML-GCN quickly overfits the
trainval set in the first ten epochs, whereas CbMLC generalizes well on the
test set as the trainval accuracy grows.

Considering label noise for the single-label setting is straightforward, as
each instance only has one positive label. That is, a mislabeled instance has
exactly two classes that are flipped. However, label noise under the multi-
label setting is more complicated, as there is an unknown number of positive
labels associated to an instance. In other words, the number of mislabeled
classes is arbitrary. Despite the prevalence of label noise in MLC, little
attention has been given to evaluate MLC with noisy labels. Among the several
works (Li et al., 2021; Bai et al., 2020; Yao et al., 2018) that consider
noisy labels, they only evaluate with uniform noise that is symmetric on
positive and negative labels. In the real world, positive labels in MLC are
very sparse, and therefore it is unreasonable to assume symmetric noise across
positive and negative labels. We are in need of a way to model noisy labels in
MLC that is closer to real-world scenarios.

Recent successes in MLC are the outcome of leveraging label correlations via
graph neural networks (GNNs) (Scarselli et al., 2008) which model how labels
interact with each other (Lanchantin et al., 2019; Chen et al., 2019; Wang et
al., 2020), or via label covariance matrices (Chen et al., 2017; Bai et al.,
2020; Tang et al., 2018). However, these methods are particularly prone to
errors. For example, ML-GCN (Chen et al., 2019), a state-of-the-art multi-
label classification model for image recognition, pre-computes a label
correlation matrix purely based on the label co-occurrences in the training
set, and uses the matrix to guide information propagation among label nodes
during training. Given the over-parameterized nature of deep neural networks,
it is easy to overfit label noise, leading to poor generalization (Zhang et
al., 2017a). In Figure 1, we show the mean averaged precision (mAP) curves for
ML-GCN and our proposed method, CbMLC, on a corrupted version of _Pascal VOC_
, where we inject different types of asymmetric noise in the trainval set. ML-
GCN quickly overfits the noisy trainval set, and test accuracy starts dropping
in the first 10 epochs. However, using the proposed regularization technique,
CbMLC substantially shrinks the generalization gap.

Our contributions are (1) We identify three real-world noise scenarios and
show how state-of-the-art MLC techniques perform poorly under these scenarios.
We test with both symmetric and asymmetric noise types, and combined noise
types with varying ratios on different instances. (2) We propose a Context-
based Multi-label Classifier (CbMLC), an end-to-end framework that leverages
large pretrained word embeddings to perform context-based regularization for
multi-label classification to avoid overfitting. It employs an encoder-to-
decoder network architecture with an asymmetric loss function, a domain-
specific encoder to encode features into a latent vector, and a message
passing decoder operating on a label graph to learn label dependencies. This
proposed framework not only benefits from employing a graph neural network to
learn label correlations, but also alleviates the overfitting issue _without
requiring access to any clean labels_. (3) We conduct extensive experiments on
public datasets in a variety of application domains including computer vision
(CV) and natural language processing (NLP). On clean datasets, our framework
outperforms (or is comparable to) the state-of-the-art methods. On datasets
with noise, CbMLC produces substantial improvements over previous methods in
most cases. We provide an ablation study that shows CbMLC learns better label
semantics guided by the context-based regularization.

##  2 Modeling Real-World Noise in MLC

Though there is a vast amount of research focusing on deep learning for
single-label classification (Song et al., 2020), little attention has been
drawn to the multi-label setting. In particular, few works have examined how
to model noise in a learning algorithm so that it is closer to real-world
scenarios, nor is it a common practice for researchers to evaluate their MLC
methods with noisy labels.

To identify possible noise types, we first discuss how noise is generated. Due
to ever-increasing dataset sizes, it becomes impossible to have human experts
to annotate all available data; hence, we turn to crowd-sourcing platforms
(Yuen et al., 2011). A problem these platforms suffer from is that the
reliability of individual workers varies, and there are individuals who are
careless and give very noisy responses (Snow et al., 2008). Other automatic
methods such as $k$-NN induction are used (Chua et al., 2009). Additionally,
in citizen science projects, even people with more expertise may also make
mistakes due to various reasons such as environments and perceptual errors
(McNicol, 1972).

We consider label noise from a statistical point of view, inspired by the
noise taxonomy for the single-label setting Frénay & Verleysen (2013). To
model the label noise process, we introduce three random variables: $Y$ is the
true label for one class being either 0 or 1, $\tilde{Y}$ is the annotated
label, and $E$ is a binary variable indicating whether an error occurred
(i.e., $Y\neq\tilde{Y}$). We focus on two statistical models of label noise.
The first model is the _noisy completely at random (NCAR)_ model, where the
occurrence of an error $E$ is independent of $Y$. Formally, the error rate, or
the probability of a true label different from the observed label is,
$p_{e}=P(E=1)=P(Y\neq\tilde{Y})$. In the case of MLC, we assume a biased coin
is flipped independently for every class label of an instance to decide
whether the observed label is correct or not. Therefore, if the error rate is
$p_{e}$ in a labeling process, an annotated instance will in expectation have
$|\mathcal{Y}|\cdot p_{e}$ corrupted labels, where $|\mathcal{Y}|$ is the
number of labels. The second model is the _noisy at random (NAR)_ model, where
the probability of an error $E$ depends on $Y$. In other words, this model can
represent anisotropic noise varying from classes to classes. Formally, the
error rate for a label in an instance is $p_{e}=P(E=1)=P(Y=1)\cdot
P(E=1|Y=1)+P(Y=0)\cdot P(E=1|Y=0)$. We note these two noise models are
independent of features. While features could also impact how noise is
introduced because mislabeled samples are likely similar to examples of
another class (Beigman Klebanov & Beigman, 2009), or have labels with low
density (Denoeux, 2000), due to its complexity we leave it to future work.

Based on the two noise models, we evaluate MLC methods with the following
noise types that are likely to occur in the real-world annotation processes:

  1. 1.

Under NCAR, we consider an example with $x\%$ labels uniformly flipped. This
is by far the most widely evaluated noise type in the existing MLC literature
(Li et al., 2021; Bai et al., 2020).

  2. 2.

Under NAR, we consider an example with $x\%$ positive labels uniformly
flipped. It is common to see this type of noise in citizen science projects
(Sullivan et al., 2009). In general, it is more often to label positive
classes as negative in MLC (Ben-Baruch et al., 2020).

  3. 3.

Under NAR, we consider an example with only one positive label that is
selected from all positive labels. This is a standard assumption made when
collecting data from the web (Li et al., 2017). In fact, because of ignoring
the intrinsic multi-label nature of the images, the classification performance
on ImageNet (Russakovsky et al., 2015) produced by state-of-the-art
classifiers is under-estimated (Stock & Cisse, 2018).

We thoroughly evaluate state-of-the-art MLC methods and CbMLC with the three
noise types in Section 4. We note that in the real world, it is rare to have
uniform noise or to have only one noise type in a dataset, and thus we test
the MLC algorithms with a single noise type as well as combined noise types
with varying ratios.

##  3 Background

#### Notations.

Let $\mathcal{D}$ denote the dataset
$\\{(\bm{x}_{i},\bm{y}_{i})\\}_{i=1}^{N}$, where $\bm{x}_{i}\in\mathbb{R}^{S}$
is an input, and $\bm{y}_{i}\in\\{0,1\\}^{L}$ is an output associated with
sample $i$. Input $\bm{x}_{i}$ can be either an ordered or an unordered set,
and output $\bm{y}_{i}$ has $L$ binary labels with 1 indicating the presence
of a label and 0 otherwise. We note that in the noisy learning setting,
$\bm{y}_{i}$ could be flipped from the actual truth value. Our goal is to make
few mistakes on unseen data even when there are corrupted labels.

#### Word Embeddings.

Word embedding is a language modeling technique where words from the
vocabulary are mapped to vectors of real values. They are often trained from
very large online text corpora in an unsupervised manner. Common word
embeddings include GloVe (Pennington et al., 2014), GoogleNews (Mikolov et
al., 2013), FastText (Joulin et al., 2016). we will use GloVe, as it features
a nearest-neighbors property, where words that are semantically similar will
have smaller Euclidean distances.

#### Attention-based Message Passing Neural Networks (MPNN).

_Self-attention_ , or _intra-attention_ , is a mechanism that learns to assign
different importance weights to each part of a larger piece to focus on more
important portions (Vaswani et al., 2017). Let $G=(V,E)$ be a directed graph
where $V$ is the node set and $E$ is the edge set. We assume $G$ to be a
complete graph. To define self-attention on a graph in which each node attends
to every other nodes and all edges carry attention weights, attention weight
$a_{lj}^{t}$ for a node pair $(\bm{v}_{l},\bm{v}_{j})$ is computed as:

$\displaystyle
e^{t}_{lj}=a(\bm{v}^{t}_{l},\bm{v}^{t}_{j})=\frac{(\mathbf{W}^{q}\bm{v}^{t}_{l})^{\top}(\mathbf{W}^{u}\bm{v}^{t}_{j})}{\sqrt{\smash[b]{d}}}$
(1)
$\displaystyle\alpha^{t}_{lj}=\textrm{softmax}_{j}(e^{t}_{lj})=\frac{\textrm{exp}(e^{t}_{lj})}{\sum_{k\in\mathcal{N}(l)}{\textrm{exp}(e^{t}_{lk})}},$
(2)

where $a(\cdot)$ is a dot product with node-wise linear transformations
$\mathbf{W}^{q}\in\mathbb{R}^{d\times d}$ on node $\bm{v}_{l}^{t}$ and
$\mathbf{W}^{u}\in\mathbb{R}^{d\times d}$ on node $\bm{v}_{j}^{t}$, scaled by
$\sqrt{d}$; $e_{lj}^{t}$ represents the raw importance of label $j$ to label
$l$ and is normalized by $\textrm{softmax}(\cdot)$ to obtain
$\alpha^{t}_{lj}$.

We briefly introduce LaMP, which applies self-attention to learn label
interactions for MLC (Lanchantin et al., 2019). Each $v\in V$ associates with
a label. To predict whether a label is present, LaMP collects states from its
neighbors based on the computed attention weights and makes an update on the
current state of the node. If an attention weight is large from one label to
another, it suggests the one label has a strong dependency on the other label.

We describe this process formally. In the $t$-th self-attention layer, each
label is represented by a node $\bm{v}_{l}^{t}\in\mathbb{R}^{d}$. LaMP first
generates the message $\bm{m}_{l}^{t}$ of $\bm{v}_{l}^{t}$:

$\displaystyle
M_{\textrm{atn}}(\bm{v}^{t}_{l},\bm{v}^{t}_{j})=\alpha^{t}_{lj}\mathbf{W}^{v}\bm{v}^{t}_{j},$
(3)
$\displaystyle\bm{m}^{t}_{l}=\bm{v}^{t}_{l}+\sum_{j\in\mathcal{N}(l)}M_{\textrm{atn}}(\bm{v}^{t}_{l},\bm{v}^{t}_{j}),$
(4)

where $\mathbf{W}^{v}\in\mathbb{R}^{d\times d}$ is a node-wise linear
transformation.

After going through the $t$-th feed forward layer $U^{t}$, we obtain
$\bm{v}_{l}^{t+1}$ in the $(t+1)$-th self-attention layer as:

$\displaystyle{\bm{v}}^{t+1}_{l}$
$\displaystyle=\bm{m}^{t}_{l}+U^{t}(\bm{m}^{t}_{l};\bm{W}).$ (5)

Finally, we can feed this into a multi-layer perceptron (MLP) to obtain a
probability for label $\bm{v}$. In later sections, we will refer to this
module as an attention-based MPNN.

#### Asymmetric Loss Functions.

The most widely-used loss function for MLC is binary cross entropy (BCE). For
each training sample, BCE is defined as follows:

$\textrm{BCE}(\bm{y},\hat{\bm{y}})=\frac{1}{L}\sum_{l=0}^{L-1}-(y_{l}\log(\hat{y}_{l})+(1-y_{l})\log(1-\hat{y}_{l}))$

(6)

where $\bm{y}$ is the annotated label vector and $\hat{\bm{y}}$ is the
predicted label vector. The label class is indexed by $l$. An issue BCE
introduces is the penalty for misclassifying every label class is equal, and
thus the loss reduced by easy negative labels may outweigh the penalty from
the rare positive samples. Therefore, a loss to decouple the focusing levels
of the positive and negative classes would be useful. Ben-Baruch et al.
propose asymmetric BCE that enables this flexibility:

$\textrm{ASL}(\bm{y},\hat{\bm{y}})=\frac{1}{L}\sum_{l=0}^{L-1}-y_{l}(1-\hat{y}_{l})^{\gamma^{+}}\log(\hat{y}_{l})\\\
-(1-y_{l})\hat{y}_{l}^{\gamma^{-}}\log(1-\hat{y}_{l}))$

(7)

where $\gamma^{+}$, $\gamma^{-}$ are the focusing parameters for positive,
negative classes, respectively. By setting $\gamma^{+}<\gamma^{-}$, we are
able to concentrate the optimization on making correct classifications on
positive classes.

Furthermore, to make additional efforts for reducing the effect of very easy
negative samples, probability shifting is introduced:

$\hat{y}_{l}=\textrm{max}(\hat{y}_{l}-m,0)$ (8)

where $m$ is a hyper-parameter that controls how many negative samples to
drop. When a class probability is below $m$, we drop any loss penalty for the
class.

Figure 2: Model architecture of CbMLC. (A) Overall network architecture is in
an encoder-to-decoder style. The encoder can be parameterized by any function
depending on the application domain, and the decoder is a message passing
network. (B) The underlying graph for the decoder. Each node corresponds to a
label represented by an embedded vector, and there is an edge connecting every
pair of nodes in both directions, meaning messages are passed both ways. (C)
An example of label node obtaining information from both $\bm{z}$ and other
label nodes. Label 0 first receives $\bm{z}$ and collects messages from all
other nodes with the attention weights $a^{01}$ through $a^{0L-1}$. (D)
Decoder layers. Each of the $T$ layers computes a multi-head attention,
followed by a feedforward operation. Layer normalization is used throughout to
stabilize training. Finally, a readout layer is applied to produce the final
label probability.

##  4 A Context-Based Multi-label Classifier

We propose CbMLC, an end-to-end learning framework for multi-label
classification, extending robust predictive performance on real-world datasets
with noisy labels. The key to its robustness is that CbMLC exploits rich
contextual information in large-scale pretrained word embeddings by
encouraging label embeddings similar to but not the same as word embeddings.
The context-based regularization provides the following benefits: (1) over-
parameterized neural networks can easily overfit noisy and less frequent
labels, and incorporating word similarity information from pretrained word
embeddings helps CbMLC to avoid spurious label correlations; (2) when we are
given a dataset with zero prior knowledge, the pretrained word embeddings
guide learning in a promising direction; and (3) regularizing based on the
word embeddings alleviates the over-smoothing problem in GNNs. In addition to
the regularization, CbMLC also leverages recent advances in MLC such as
asymmetric losses and employs self-attention in label graphs.

###  4.1 Context-Based Regularization

The key to context-based regularization is that we inject the word
similarities learned by pre-trained word embeddings on label embeddings, which
is done by adjusting the $\mathcal{L}2$ distance between the word embeddings
and the label embeddings. To begin, we have a word embedding function
$g:\Delta\mapsto\mathbb{R}^{P}$, where $\Delta$ is a dictionary of word
tokens, and $P$ is the dimensionality of the embedding space. $g(\cdot)$ is
often pretrained from large online corpora in an unsupervised fashion; thus,
acquiring this contextual information does not require any additional
supervision. We denote a label embedding by $\bm{e}_{l}$ for the $l$th label.
To construct initial label embeddings, we define a mapping $h$ from word
tokens of a label class $s_{l}$ to its embedding in $\mathbb{R}^{P}$. If
$s_{l}$ is a single word token $d$ and $s_{l}=d\in\Delta$, then
$h(s_{l})=g(d)$; however, if $s_{l}\notin\Delta$, then $h(s_{l})\sim
N(\mu,\sigma)^{P}$, where $\mu,\sigma$ are the mean and the variance of all
the label tokens in $\Delta$. If $s_{l}$ is a label of multiple word tokens
$\\{d1,...,d_{K}\\}$, then $h(s_{l})=\frac{1}{K}\sum_{1}^{K}h(d_{k})$.
$\bm{e}_{l}$ is initially set as $h(s_{l})$ and updated at every gradient
step. We define the context-based regularization as follows:

$\mathcal{L}_{CB}=\sum_{l=0}^{L-1}(\bm{e}_{l}-h(s_{l}))^{2}$ (9)

With this regularizer, CbMLC can learn label embeddings customized to each
dataset while still maintaining some degree of global word similarities
obtained from pre-trained word embeddings to prevent over-fitting.

###  4.2 Model Architecture

Figure 2 presents the model architecture of the CbMLC framework. In Figure 2A,
the overall network flow follows an encoder-to-decoder structure: the encoder
first maps input $x$ to a latent space, and the decoder takes the latent
vector $\bm{z}$ to produce the final label probabilities. The encoder can be
parameterized by any function depending on the specific tasks, and the decoder
is an attention-based MPNN on a label graph, where each node corresponds to a
label. From Figure 2B, we can see that the label graph is by default a
complete graph, where every pair of nodes is connected by an edge in both
directions. We can also remove edges if two labels are known to have no
correlations.

In Figure 2C, label node 0 first receives $\bm{z}$ and treats it as a single
message to make an updates. Then, label node 0 collects messages from its
neighbors based on their attention weights to make the second update. Figure
2D demonstrates how to make one update. Each label node has hidden states that
are represented as embedded vectors
$\\{\bm{v}^{t}_{0},\bm{v}^{t}_{1},\dots,\bm{v}^{t}_{L-1}\\}$, where
$\bm{v}^{t}_{l}\in\mathbb{R}^{P}$ and $t$ is the time step, and
$\bm{v}^{0}_{l}=\bm{e}_{l}$. Self-attention is computed based on Equations
1-4, and feedforward is computed using Equation 5. Specifically, we use multi-
head self-attention (Vaswani et al., 2017), so that a node can focus on
different parts simultaneously. Each of the attention and feedforward
sublayers is followed by layer normalization (Ba et al., 2016) to reduce
training instability. Because at every $t$, $\bm{v}^{t}_{l}$ needs to update
twice, there are $2T$ layers. After the $2T$ layers, a readout layer predicts
each label $\hat{y}_{l}$, where a readout function $R$ projects
$\bm{v}_{l}^{2T}$ using a projection matrix
$\textbf{W}^{o}\in\mathbb{R}^{d\times d}$. The $l$th row of $\textbf{W}^{o}$
is denoted by $\textbf{W}^{o}_{l}$. The resulting vector of size $L\times 1$
is then fed through an element-wise sigmoid function to produce the final
probabilities of all labels:

$\hat{y}_{l}=R({\bm{v}}^{2T}_{l};{\textbf{W}^{o}})=\textrm{sigmoid}({\textbf{W}^{o}_{l}}{\bm{v}}^{2T}_{l}).$
(10)

We note that $R(\cdot)$ is shared across all nodes; therefore, the prediction
result is invariant to any node permutation.

###  4.3 Loss Function

The loss function $\mathcal{L}$ consists of two parts: the first part is a 0/1
loss that penalizes incorrect predictions, and the second part is the context-
based (CB) regularizer described in Sec. 4.1. $\mathcal{L}$ is as follows:

$\mathcal{L}=\mathcal{L}_{\textrm{ASL}}+\lambda\mathcal{L}_{CB}$ (11)

$\mathcal{L}_{ASL}$ is the asymmetric cross entropy loss defined in Sec. 3.
Hyperparameter $\lambda$ controls to which degree we encourage labels to
follow the word similarity structure from the pretrained word embbedings. In a
dataset with more training examples, we can set a small $\lambda$ as CbMLC has
more examples to learn how labels interact with each other. In summary, the
parameters to be optimized are the model parameters together with the label
embeddings.

Dataset |  F1 score | MLKNN | SLEEC | Seq2Seq | LaMP | MPVAE | ML-GCN | resnet101-ce | resnet101-ASL | CbMLC
---|---|---|---|---|---|---|---|---|---|---
bibtex |  ebF1 | 0.1826 | 0.4490 | 0.393 | 0.447 | 0.4534 | - | - | - |  0.4566
miF1 | 0.1782 | 0.4074 | 0.384 | 0.473 | 0.4800 | - | - | - |  0.4837
maF1 | 0.0727 | 0.2937 | 0.282 | 0.3760 | 0.3863 | - | - | - |  0.4014
reuters |  ebF1 | - | - | 0.8917 | 0.8950 | - | - | - | - |  0.9064
miF1 | - | - | 0.8545 | 0.8770 | - | - | - | - |  0.8893
maF1 | - | - | 0.4567 | 0.5600 | - | - | - | - |  0.6024
delicious |  ebF1 | 0.2590 | 0.3081 | 0.3200 | 0.3720 |  0.3732 | - | - | - | 0.3558
miF1 | 0.2639 | 0.3333 | 0.3290 | 0.3860 |  0.3934 | - | - | - | 0.3699
maF1 | 0.0526 | 0.1418 | 0.1660 | 0.196 | 0.1814 | - | - | - |  0.1984
rcv1 |  ebF1 | - | - |  0.8899 | 0.8870 | - | - | - | - | 0.8868
miF1 | - | - |  0.8797 | 0.8770 | - | - | - | - | 0.8735
maF1 | - | - | 0.7258 | 0.7400 | - | - | - | - |  0.7460
VOC |  ebF1 | - | - | - | - | - | 0.9019 | 0.8903 | 0.8839 |  0.9067
miF1 | - | - | - | - | - | 0.8904 | 0.8800 | 0.8743 |  0.8933
maF1 | - | - | - | - | - | 0.8784 | 0.8657 | 0.8599 |  0.8822

Table 1: Performance comparison (based on example- and label-based F1 scores)
of CbMLC to state-of-the-art models on five datasets with clean labels. Higher
scores are better. Highlighted numbers are the best of in each row.

##  5 Experiments

Figure 3: Performance comparison (based on label-based F1 scores) of CbMLC to
SOTA MLC models on three datasets with uniform noise across all labels (top)
and with uniform noise across positive labels (bottom). The blue lines are
miF1 scores, and the red lines are the maF1 scores. The dashed lines represent
CbMLC; the dotted lines represent the best SOTA methods for the corresponding
datasets.

We perform a thorough comparison of CbMLC to the state of the art (SOTA) on a
variety of datasets ranging from NLP to CV under both clean and noisy
settings. We also present an ablation study to demonstrate the effectiveness
of the context-based regularization.

#### Datasets and Evaluation Metrics.

We run experiments on the following datasets available for download online
111http://mulan.sourceforge.net/datasets-mlc.html
222http://host.robots.ox.ac.uk/pascal/VOC/: _reuters-21578_ and _rcv1-v2_
(Lewis et al., 2004), natural language sequences with predefined categories
based on their content, _bibtex_ (Katakis et al., 2008) and _delicious_ (Bi &
Kwok, 2013), collections of unordered text objects associated with tags, and
_Pascal VOC_ (Everingham et al., 2015), RGB images with multiple objects
simultaneously present. For _reuters-21578_ , _rcv1-v2_ , _bibtex_ , and
_delicious_ , we split them into a training set, a validation set, and a test
set in an identical way as that in Bai et al. (2020) and Lanchantin et al.
(2019). For _Pascal VOC_ (for brevity, we later refer to it as _VOC_), we
follow the exact training settings used in Chen et al. (2019) and Ben-Baruch
et al. (2020).

Following conventional MLC settings, we report example-based F1 (ebF1) scores,
label-based micro-averaged F1 scores, and label-based macro-averaged F1
scores. When there are imbalanced classes, high miF1 scores indicate better
performance on frequent labels, and high maF1 scores indicate strong results
on less frequent labels.

#### Baseline Comparisons.

We compare CbMLC with domain specific SOTA MLC methods: Seq2Seq (Nam et al.,
2017) and LaMP (Lanchantin et al., 2019) are two SOTA approaches for multi-
label text classification, and ML-GCN (Chen et al., 2019) is the SOTA in image
recognition. MLKNN (Zhang & Zhou, 2007), SLEEC (Bhatia et al., 2015), and
MPVAE (Bai et al., 2020) are general MLC techniques, but they cannot handle
image or sequence inputs. For CV datasets, we also compare to the resnet101
(He et al., 2016), with either BCE or ASL as the loss function. Among all
baseline methods, _MPVAE claims to be a noise-robust model_. We follow the
same training settings and tune hyper-parameters to achieve the best
performance for every baseline method. We note that, we do not directly
compare to single-label methods for dealing with noise because 1) many such
works assume a softmax output layer (Zhang & Sabuncu, 2018; Lyu & Tsang, 2020;
Jindal et al., 2019) or access to a small set of clean data (Zhang et al.,
2020), and 2) CbMLC is general in that the decoder and the context-based
regularization can be combined with any feature extractor, so CbMLC can be
incorporated with other noise learning methods such as meta-learning
approaches (Garcia et al., 2016). However, we do modify one SOTA single-label
noise method using loss correction and include the comparison results in
supplementary material.

#### Training and Implementation Details, etc.

We run experiments on NVIDIA Tesla V100 GPUs, each with 16GB memory. We train
CbMLC for up to 50 epochs with the Adam (Kingma & Ba, 2015) optimizer and use
a weight decay of $1e^{-5}$. The number of decoder layers is 4. For NLP
datasets, we adopt a transformer-style encoder used in Lanchantin et al.
(2019) and set the learning rate to be $2^{-4}$. For CV datasets, we use
resnet101 (He et al., 2016) as the encoder, and the learning rate is
$1e^{-5}$. CbMLC only requires minimal tuning; we only search $\lambda$ from
$\\{0.001,0.01,0.1,1\\}$ and $\gamma^{+}$ as well as $\gamma^{-}$ from
$\\{1,2,4\\}$. $m$ is $0.05$ or $0.1$.

###  5.1 Experiments on Clean Datasets

Table 1 presents the performance comparison of CbMLC to other SOTA MLC methods
on the five datasets. On all five datasets, CbMLC produces superior or at
least comparable performance to that of previous works. Furthermore, we have
the highest maF1 score for every dataset; that is, our method tends to learn
better for the classes with less positive labels. CbMLC produces relatively
smaller improvements in miF1 scores; this could be because there are already
many samples with frequent labels, so it is possible to learn how the labels
interact from label co-occurrences alone.

###  5.2 Experiments on Datasets with Injected Noise

We compare CbMLC to other SOTA MLC methods with injected noise discussed in
Section 2. For type 1 noise, we test with $\\{1\%,2\%,3\%,4\%,5\%\\}$ uniform
noise for the NLP datasets and $\\{10\%,20\%,30\%,40\%,50\%\\}$ uniform noise
for the CV datasets across both positive and negative labels (since there are
far fewer label classes for the CV datasets, adding as little noise as we add
to the NLP datasets leads to very minor drop in performance). For type 2
noise, we test with $\\{10\%,20\%,30\%,40\%,50\%\\}$ uniform noise on all the
datasets across all positive labels. We also test on type 3 noise, where each
training sample only has one positive label that is randomly chosen. Due to
the page limit, we select one dataset from each task to present the results
for type 1 and type 2 noise. We include comprehensive results for all the
datasets in the supplementary material. To produce a noisy setting close to
the real world, we also test the methods with combined noise: for each
instance, there is 1/3 chance for it to be corrupted by one type of noise; for
type 1 noise, 0% to 10% positive and negative labels are modified, and for
type 2 noise, 0% to 50% positive negative labels are modified.

We present the results for type 1 noise and type 2 noise in Figure 3. The
dashed lines show the performance of CbMLC, and the dotted lines show the
performance of the best performing SOTA MLC method on the corresponding
dataset. The blue lines represent the miF1 scores, and the red lines are the
maF1 scores. In most cases, we produce a substantial improvement. On _reuters_
and _bibtex_ , light noise (e.g., 1%, 2% type 1 noise and 10%, 20% type 2
noise) does not lead to much drop in the classification results; they are even
comparable to running other SOTA methods on the clean datasets. We also
highlight how our method performs on _Pascal VOC_. When there is 30% type 1
noise, we improve the maF1 score by the best SOTA method by nearly 30%, and
when there is 40% type 1 noise, we improve the miF1 score by the best SOTA
method by 30% as well.

We summarize the results for type 3 noise in Table 2 and the results for the
combined noise in Table 3. We outperform the best SOTA method on every dataset
(except for the miF1 score on _rcv1_ with type 3 noise). When the noise
becomes more complex, the performance gaps between the best SOTA method and
CbMLC further increase. Therefore, context-based regularization is a simple
yet very effective trick to prevent overfit label noise. To summarize, CbMLC
produce relativelly small improvements on the clean datasets; however, when
there are noisy labels, CbMLC is more robust than other SOTA methods.

|  F1 score | bibtex | reuters | delicious | rcv1 | VOC
---|---|---|---|---|---|---
SOTA |  miF1 | 0.3572 | 0.8165 | 0.2737 |  0.8483 | 0.8350
maF1 | 0.2387 | 0.4000 | 0.0363 | 0.6770 | 0.8302
CbMLC |  miF1 |  0.4210 |  0.8581 |  0.2805 | 0.8448 |  0.8447
maF1 |  0.3049 |  0.4758 |  0.0636 |  0.7007 |  0.8442

Table 2: Performance comparison (based on label-based F1 scores) of CbMLC to
SOTA MLC models on the five datasets with injected noise where each training
sample only has one positive label.

|  F1 score | bibtex | reuters | delicious | rcv1 | VOC
---|---|---|---|---|---|---
SOTA |  miF1 | 0.3238 | 0.7484 | 0.1973 | 0.8579 | 0.7138
maF1 | 0.1888 | 0.2768 | 0.0682 | 0.6646 | 0.6281
CbMLC |  miF1 |  0.3282 |  0.8200 |  0.3290 |  0.8580 |  0.7621
maF1 |  0.1993 |  0.3325 |  0.0811 |  0.6735 |  0.7281

Table 3: Performance comparison (based on label-based F1 scores) of CbMLC to
SOTA MLC models on the five datasets injected with combined noise types.
Figure 4: Embedding distances between label pairs averaged over top 100 most
frequently co-occurring label pairs for each dataset. A smaller distance for a
label pair indicates the two labels have stronger correlation. Using word
embeddings with the context-based regularizer for label embeddings produces
smaller distances for label pairs with frequent co-occurrences than not using
word embeddings, suggesting CbMLC learns better label semantics.

|  F1 score | bibtext | reuters | delicious | rcv1 | VOC
---|---|---|---|---|---|---
Fixed WordEmb |  ebF1 | 0.4484 | 0.9045 |  0.3634 | 0.8867 | 0.9023
miF1 | 0.4814 | 0.8830 |  0.3749 | 0.8734 | 0.8901
maF1 | 0.3834 | 0.5530 | 0.1949 | 0.7320 | 0.8797
Regularized WordEmb |  ebF1 |  0.4538 |  0.9064 | 0.3558 |  0.8868 |  0.9049
miF1 |  0.4893 |  0.8893 | 0.3699 |  0.8735 |  0.8919
maF1 |  0.3940 |  0.6024 |  0.1984 |  0.7460 |  0.8822

Table 4: Comparison on whether the context-based regularization yields an
improvement over having fixed word embeddings. Highlighted numbers are the
best in each row.

###  5.3 Ablation Study

#### Regularized vs. Fixed Word Embeddings.

To verify context-based regularization provides a benefit, we first show a
performance comparison of applying regularized word embeddings to applying
fixed word embeddings as label embeddings in Table 4. Except for _delicious_ ,
all other datasets favor using context-based regularization.

#### Measuring Label Correlations.

Additionally, we measure the Euclidean distance between pairs of label
embeddings. In particular, we sort label pairs based on the number of times
they co-occur in the training set and compute the average distance for the 100
pairs with most co-occurrences for the label embeddings with random
initialization and for the label embeddings with context-based regularization.
We observe that training with randomly initialized label embeddings often
suffers from a more severe over-smoothing problem; in order to reduce this
effect, we normalize the distance before comparing Euclidean distances across
different embedding spaces. For a highly correlated label pair, having a
closer distance in the embedding space suggests the model successfully
captures their correlation. We present the results in Figure 4. Out of the
five datasets, we learn more accurate label similarities on four of them.

##  6 Related Work

Existing MLC approaches that are related to our work incorporate label
correlations, use regularization or an unconventional loss, or consider MLC
under a noisy label setting.

#### Learning Label Dependencies.

Our technique is within the line of work where conditional label dependencies
are considered when making label predictions, as opposed to the methods that
decompose a MLC problem into a set of independent binary classification
problems (Boutell et al., 2004; Zhang & Zhou, 2007).

Probabilistic chain classifiers (PCCs) stack binary classifiers sequentially
and produce one label at a time conditioned on all previous outputs (Cheng et
al., 2010; Read et al., 2011). Followup works extend PCCs to recurrent neural
networks (RNNs) (Wang et al., 2016; Nam et al., 2017). There are two major
drawbacks of these proposed models. They are not parallelizable due to the
auto-regressive nature, so either training or performing inference is time-
cosuming. Additionally, MLC labels are unordered sets, but sequential
prediction is highly order-dependent; therefore, ordering labels differently
could significantly impact performance.

Chen et al.; Chen et al.; Bai et al. develop deep multi-variate probit models
(DMVPs) to tackle MLC problems. DMVPs first compute a probability of each
label being present and use a label covariance matrix to make adjustments.
However, because of covariance matrices, DMVPs can only model pairwise label
correlations, and real-world label interactions often happen beyond second
order (Wang et al., 2020).

Building label graphs on GNNs overcome the prior obstacles: their graphical
structures enables to learn label dependencies in an order-invariant manner,
and they have the potential to capture high-order label dependencies by
stacking multiple layers together. Chen et al.; Chang et al.; Wang et al.;
Wang et al. represent each node of the graph as a word embedding of the
corresponding label. This may be problematic: word embeddings are fixed for
all datasets, but the pretrained word embeddings are unlikely to fully capture
the relations between words for every dataset. The learning capacities are
thus degraded. Lanchantin et al.; Wang et al.; You et al. model label
interactions using attention-based message passing neural networks (MPNNs)
(Gilmer et al., 2017), where they learn label representations from scratch.
However, this may easily overfit when there are noisy labels.

#### Losses and Regularization Techniques.

In addition to BCE, other losses have been proposed to train multi-label
classifiers. Gong et al. (2014) perform extensive experiments on evaluating
softmax (Guillaumin et al., ), pairwise ranking (Joachims, 2002), and weighted
approximate ranking (Weston et al., 2011) for MLC. (Lin et al., 2017) present
focal loss to weight positive and negative labels differently. Regularization
techniques are an important research direction to prevent overfitting on
training sets. Zhang et al. (2017b); Krichene et al. (2019); Guo et al. (2019)
maximize the distances between all pairs of label embeddings to account for
sparse labels in extreme MLC.

#### MLC with Noisy Labels

To handle noisy labels in MLC, Garcia et al. (2016) propose to adjust
incorrectly estimated label transition probabilities by quality embedding.
Partial multi-label learning (PMLL) setting is also related (Xie & Huang,
2018), where each example is annotated with a candidate label set that
contains relevant labels and noisy labels. To identify the noisy labels,
(Zhang & Fang, 2020) recover true labels by introducing labeling confidence,
(Sun et al., 2019) exploit a low-rank and sparse decomposition scheme, and
(Xie & Huang, 2020) optimize the multi-label classifier and noisy label
identifier under a unified framework.

##  7 Conclusion

While label noise is a common problem in MLC, few works have evaluated MLC
algorithms with noisy labels. We identify three common noise scenarios and
create a noise setting to perform evaluations with combined noise types. As
the MLC methods that learn label interactions can easily be affected by noisy
data, we present CbMLC, which employs an asymmetric loss function with
context-based regularization. Our experiments show substantial improvements
over other SOTA MLC techniques on the noisy datasets.

## References

  * Ba et al. (2016) Ba, J. L., Kiros, J. R., and Hinton, G. E.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Bai et al. (2020) Bai, J., Kong, S., and Gomes, C.  Disentangled variational autoencoder based multi-label classification with covariance-aware multivariate probit model.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pp. 4313–4321, 7 2020.  doi: 10.24963/ijcai.2020/595.  Special track on AI for CompSust and Human well-being.
  * Beigman Klebanov & Beigman (2009) Beigman Klebanov, B. and Beigman, E.  From annotator agreement to noise models.  _Computational Linguistics_ , 35(4):495–503, 2009\.
  * Ben-Baruch et al. (2020) Ben-Baruch, E., Ridnik, T., Zamir, N., Noy, A., Friedman, I., Protter, M., and Zelnik-Manor, L.  Asymmetric loss for multi-label classification.  _arXiv preprint arXiv:2009.14119_ , 2020.
  * Bhatia et al. (2015) Bhatia, K., Jain, H., Kar, P., Varma, M., and Jain, P.  Sparse local embeddings for extreme multi-label classification.  In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 28_ , pp. 730–738. Curran Associates, Inc., 2015.
  * Bi & Kwok (2013) Bi, W. and Kwok, J.  Efficient multi-label classification with many labels.  In Dasgupta, S. and McAllester, D. (eds.), _Proceedings of the 30th International Conference on Machine Learning_ , volume 28 of _Proceedings of Machine Learning Research_ , pp. 405–413, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.  URL http://proceedings.mlr.press/v28/bi13.html.
  * Boutell et al. (2004) Boutell, M. R., Luo, J., Shen, X., and Brown, C. M.  Learning multi-label scene classification.  _Pattern recognition_ , 37(9):1757–1771, 2004\.
  * Chang et al. (2020) Chang, W.-C., Yu, H.-F., Zhong, K., Yang, Y., and Dhillon, I. S.  _Taming Pretrained Transformers for Extreme Multi-Label Text Classification_ , pp. 3163–3171.  Association for Computing Machinery, New York, NY, USA, 2020.  ISBN 9781450379984.
  * Chen et al. (2017) Chen, D., Xue, Y., Fink, D., Chen, S., and Gomes, C. P.  Deep multi-species embedding.  In _Proceedings of the 26th International Joint Conference on Artificial Intelligence_ , pp. 3639–3646, 2017.
  * Chen et al. (2018) Chen, D., Xue, Y., and Gomes, C.  End-to-end learning for the deep multivariate probit model.  In _International Conference on Machine Learning_ , pp. 932–941, 2018.
  * Chen et al. (2019) Chen, Z.-M., Wei, X.-S., Wang, P., and Guo, Y.  Multi-label image recognition with graph convolutional networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 5177–5186, 2019.
  * Cheng et al. (2010) Cheng, W., Hüllermeier, E., and Dembczynski, K. J.  Bayes optimal multilabel classification via probabilistic classifier chains.  In _Proceedings of the 27th international conference on machine learning (ICML-10)_ , pp. 279–286, 2010.
  * Chua et al. (2009) Chua, T.-S., Tang, J., Hong, R., Li, H., Luo, Z., and Zheng, Y.  Nus-wide: A real-world web image database from national university of singapore.  In _Proceedings of the ACM International Conference on Image and Video Retrieval_ , CIVR ’09, New York, NY, USA, 2009. Association for Computing Machinery.  ISBN 9781605584805.  doi: 10.1145/1646396.1646452.  URL https://doi.org/10.1145/1646396.1646452.
  * Denoeux (2000) Denoeux, T.  A neural network classifier based on dempster-shafer theory.  _IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans_ , 30(2):131–150, 2000.
  * Everingham et al. (2015) Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A.  The pascal visual object classes challenge: A retrospective.  _International Journal of Computer Vision_ , 111(1):98–136, January 2015.
  * Frénay & Verleysen (2013) Frénay, B. and Verleysen, M.  Classification in the presence of label noise: a survey.  _IEEE transactions on neural networks and learning systems_ , 25(5):845–869, 2013.
  * Garcia et al. (2016) Garcia, L. P., de Carvalho, A. C., and Lorena, A. C.  Noise detection in the meta-learning level.  _Neurocomputing_ , 176:14–25, 2016.
  * Gilmer et al. (2017) Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E.  Neural message passing for quantum chemistry.  _arXiv preprint arXiv:1704.01212_ , 2017.
  * Gong et al. (2014) Gong, Y., Jia, Y., Toshev, A., Leung, T., and Ioffe, S.  Deep convolutional ranking for multilabel image annotation.  In _International Conference on Learning Representations_ , 2014.
  * (20) Guillaumin, M., Mensink, T., Verbeek, J., and Schmid, C.  Tagprop: Discriminative metric learning in nearest neighbor models for image auto-annotation.  In _2009 IEEE 12th international conference on computer vision_ , pp. 309–316. IEEE.
  * Guo et al. (2019) Guo, C., Mousavi, A., Wu, X., Holtmann-Rice, D. N., Kale, S., Reddi, S., and Kumar, S.  Breaking the glass ceiling for embedding-based classifiers for large output spaces.  In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 32, pp. 4943–4953. Curran Associates, Inc., 2019.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * Ipeirotis et al. (2010) Ipeirotis, P. G., Provost, F., and Wang, J.  Quality management on amazon mechanical turk.  In _Proceedings of the ACM SIGKDD workshop on human computation_ , pp. 64–67, 2010.
  * Jindal et al. (2019) Jindal, I., Pressel, D., Lester, B., and Nokleby, M.  An effective label noise model for dnn text classification.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 3246–3256, 2019.
  * Joachims (2002) Joachims, T.  Optimizing search engines using clickthrough data.  In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_ , pp. 133–142, 2002.
  * Joulin et al. (2016) Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T.  Fasttext. zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_ , 2016.
  * Katakis et al. (2008) Katakis, I., Tsoumakas, G., and Vlahavas, I.  Multilabel text classification for automated tag suggestion.  _ECML PKDD Discovery Challenge 2008_ , pp. 75, 2008.
  * Kingma & Ba (2015) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  In _ICLR (Poster)_ , 2015.
  * Krasin et al. (2017) Krasin, I., Duerig, T., Alldrin, N., Ferrari, V., Abu-El-Haija, S., Kuznetsova, A., Rom, H., Uijlings, J., Popov, S., Veit, A., Belongie, S., Gomes, V., Gupta, A., Sun, C., Chechik, G., Cai, D., Feng, Z., Narayanan, D., and Murphy, K.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  _Dataset available from https://storage.googleapis.com/openimages/web/index.html_ , 2017.
  * Krichene et al. (2019) Krichene, W., Mayoraz, N., Rendle, S., Zhang, L., Yi, X., Hong, L., Chi, E., and Anderson, J.  Efficient training on very large corpora via gramian estimation.  In _International Conference on Learning Representations_ , 2019.
  * Lanchantin et al. (2019) Lanchantin, J., Sekhon, A., and Qi, Y.  Neural message passing for multi-label classification.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pp. 138–163. Springer, 2019.
  * Lewis et al. (2004) Lewis, D. D., Yang, Y., Rose, T. G., and Li, F.  Rcv1: A new benchmark collection for text categorization research.  _Journal of machine learning research_ , 5(Apr):361–397, 2004.
  * Li et al. (2021) Li, J., Xiong, C., and Hoi, S.  Towards noise-resistant object detection with noisy annotations, 2021\.  URL https://openreview.net/forum?id=TlPHO_duLv.
  * Li et al. (2017) Li, W., Wang, L., Li, W., Agustsson, E., Berent, J., Gupta, A., Sukthankar, R., and Van Gool, L.  Webvision challenge: visual learning and understanding with web data.  _arXiv preprint arXiv:1705.05640_ , 2017.
  * Lin et al. (2017) Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2980–2988, 2017.
  * Liu et al. (2017) Liu, J., Chang, W.-C., Wu, Y., and Yang, Y.  Deep learning for extreme multi-label text classification.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pp. 115–124, 2017.
  * Lyu & Tsang (2020) Lyu, Y. and Tsang, I. W.  Curriculum loss: Robust learning and generalization against label corruption.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=rkgt0REKwS.
  * McNicol (1972) McNicol, D.  What are statistical decisions.  In _A Primer of Signal Detection Theory_ , pp. 1–17. Allen, 1972\.
  * Mikolov et al. (2013) Mikolov, T., Chen, K., Corrado, G., and Dean, J.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ , 2013.
  * Nam et al. (2014) Nam, J., Kim, J., Mencía, E. L., Gurevych, I., and Fürnkranz, J.  Large-scale multi-label text classification—revisiting neural networks.  In _Joint european conference on machine learning and knowledge discovery in databases_ , pp. 437–452. Springer, 2014.
  * Nam et al. (2017) Nam, J., Mencía, E. L., Kim, H. J., and Fürnkranz, J.  Maximizing subset accuracy with recurrent neural networks in multi-label classification.  In _Advances in neural information processing systems_ , pp. 5413–5423, 2017.
  * Pennington et al. (2014) Pennington, J., Socher, R., and Manning, C. D.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pp. 1532–1543, 2014.
  * Raykar et al. (2010) Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., and Moy, L.  Learning from crowds.  _Journal of Machine Learning Research_ , 11(4), 2010.
  * Read et al. (2011) Read, J., Pfahringer, B., Holmes, G., and Frank, E.  Classifier chains for multi-label classification.  _Machine learning_ , 85(3):333, 2011.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Scarselli et al. (2008) Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G.  The graph neural network model.  _IEEE Transactions on Neural Networks_ , 20(1):61–80, 2008.
  * Snow et al. (2008) Snow, R., O’connor, B., Jurafsky, D., and Ng, A. Y.  Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks.  In _Proceedings of the 2008 conference on empirical methods in natural language processing_ , pp. 254–263, 2008.
  * Song et al. (2020) Song, H., Kim, M., Park, D., and Lee, J.-G.  Learning from noisy labels with deep neural networks: A survey, 2020.
  * Stock & Cisse (2018) Stock, P. and Cisse, M.  Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 498–512, 2018.
  * Sullivan et al. (2009) Sullivan, B. L., Wood, C. L., Iliff, M. J., Bonney, R. E., Fink, D., and Kelling, S.  ebird: A citizen-based bird observation network in the biological sciences.  _Biological conservation_ , 142(10):2282–2292, 2009.
  * Sun et al. (2019) Sun, L., Feng, S., Wang, T., Lang, C., and Jin, Y.  Partial multi-label learning by low-rank and sparse decomposition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 5016–5023, 2019.
  * Tang et al. (2018) Tang, L., Xue, Y., Chen, D., and Gomes, C. P.  Multi-entity dependence learning with rich context via conditional variational auto-encoder.  In _AAAI_ , 2018.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  In _Advances in neural information processing systems_ , pp. 5998–6008, 2017.
  * Veit et al. (2017) Veit, A., Alldrin, N., Chechik, G., Krasin, I., Gupta, A., and Belongie, S.  Learning from noisy large-scale datasets with minimal supervision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 839–847, 2017.
  * Wang et al. (2018) Wang, G., Li, C., Wang, W., Zhang, Y., Shen, D., Zhang, X., Henao, R., and Carin, L.  Joint embedding of words and labels for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2321–2331, 2018\.
  * Wang et al. (2016) Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., and Xu, W.  Cnn-rnn: A unified framework for multi-label image classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2285–2294, 2016.
  * (57) Wang, Y., He, D., Li, F., Long, X., Zhou, Z., Ma, J., and Wen, S.  Multi-label classification with label graph superimposing.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, NY, USA, February 7-12, 2020_ , pp. 12265–12272.
  * Wang et al. (2020) Wang, Y., He, D., Li, F., Long, X., Zhou, Z., Ma, J., and Wen, S.  Multi-label classification with label graph superimposing.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 12265–12272, 2020.
  * Weston et al. (2011) Weston, J., Bengio, S., and Usunier, N.  Wsabie: Scaling up to large vocabulary image annotation.  In _Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence - Volume Volume Three_ , IJCAI’11, pp. 2764–2770. AAAI Press, 2011.  ISBN 9781577355151.
  * Xie & Huang (2018) Xie, M.-K. and Huang, S.-J.  Partial multi-label learning.  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018\.
  * Xie & Huang (2020) Xie, M.-K. and Huang, S.-J.  Partial multi-label learning with noisy label identification.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(04):6454–6461, Apr. 2020.  doi: 10.1609/aaai.v34i04.6117.
  * Yao et al. (2018) Yao, J., Wang, J., Tsang, I. W., Zhang, Y., Sun, J., Zhang, C., and Zhang, R.  Deep learning from noisy image labels with quality embedding.  _IEEE Transactions on Image Processing_ , 28(4):1909–1922, 2018.
  * (63) You, R., Guo, Z., Cui, L., Long, X., Bao, Y., and Wen, S.  Cross-modality attention with semantic graph embedding for multi-label classification.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, NY, USA, February 7-12, 2020_ , pp. 12709–12716.
  * Yuen et al. (2011) Yuen, M.-C., King, I., and Leung, K.-S.  A survey of crowdsourcing systems.  In _2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing_ , pp. 766–773. IEEE, 2011.
  * Zhang et al. (2017a) Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O.  Understanding deep learning requires rethinking generalization.  2017a.  URL https://arxiv.org/abs/1611.03530.
  * Zhang & Fang (2020) Zhang, M.-L. and Fang, J.-P.  Partial multi-label learning via credible label elicitation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * Zhang & Zhou (2007) Zhang, M.-L. and Zhou, Z.-H.  ML-KNN: A lazy learning approach to multi-label learning.  _Pattern recognition_ , 40(7):2038–2048, 2007\.
  * Zhang et al. (2017b) Zhang, X., Yu, F. X., Kumar, S., and Chang, S.-F.  Learning spread-out local feature descriptors.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 4595–4603, 2017b.
  * Zhang & Sabuncu (2018) Zhang, Z. and Sabuncu, M. R.  Generalized cross entropy loss for training deep neural networks with noisy labels.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pp. 8792–8802, 2018.
  * Zhang et al. (2020) Zhang, Z., Zhang, H., Arik, S. O., Lee, H., and Pfister, T.  Distilling effective supervision from severe label noise.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9294–9303, 2020.