# Positive Label Is All You Need for Multi-Label Classification

Zhixiang Yuan1 Kaixin Zhang1∗† Tao Huang2
1School of Computer Science and Technology, Anhui University of Technology,
Maanshan 243032, China
2School of Computer Science, Faculty of Engineering, The University of Sydney
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Equal
contributions. †Corresponding author.

###### Abstract

Multi-label classification (MLC) suffers from the inevitable label noise in
training data due to the difficulty in annotating various semantic labels in
each image. To mitigate the influence of noisy labels, existing methods mainly
devote to identifying and correcting the label mistakes via a trained MLC
model. However, these methods still involve annoying noisy labels in training,
which can result in imprecise recognition of noisy labels and weaken the
performance. In this paper, considering that the negative labels are
substantially more than positive labels, and most noisy labels are from the
negative labels, we directly discard all the negative labels in the dataset,
and propose a new method dubbed positive and unlabeled multi-label
classification (PU-MLC). By extending positive-unlabeled learning into MLC
task, our method trains model with only positive labels and unlabeled data,
and introduces adaptive re-balance factor and adaptive temperature coefficient
in the loss function to alleviate the catastrophic imbalance in label
distribution and over-smoothing of probabilities in training. Our PU-MLC is
simple and effective, and it is applicable to both MLC and MLC with partial
labels (MLC-PL) tasks. Extensive experiments on MS-COCO and PASCAL VOC
datasets demonstrate that our PU-MLC achieves significantly improvements on
both MLC and MLC-PL settings with even fewer annotations. Code will be
released.

##  1 Introduction

Deep learning has achieved remarkable success in single-label image
classification task [16, 35, 39]. Recently, multi-label classification (MLC)
[11, 10, 33] has attracted much attention as an image often contains multiple
objects or concepts. Existing methods usually treat MLC as multiple binary
classification tasks and train each binary task with positive and negative
labels. As such, the model is trained to recognize whether an image contains
each class or not.

However, noisy labels inevitably exist in MLC datasets due to the annotation
difficulty [13, 33], which disturb the training and worsen the performance as
a result (see Figure 2(a)-(b)). To alleviate this issue, some methods [40, 33]
propose to first train the model with noisy labels, then correct or discard
the mislabeled labels with the learned model. Nevertheless, the mislabeled
labels are still involved in these methods, which would still cause a negative
impact on training process and mislead the identification of noisy labels to
some extent.

Figure 1: Comparisons of multi-label classification performance on MS-COCO
dataset. Our PU-MLC gains significant improvements consistently on different
known label proportions.

Moreover, the mislabeling problem tends to be severer on multi-label
classification with partial labels (MLC-PL) task [13, 22, 7, 31]. In MLC-PL,
models are trained with a partially labeled dataset to reduce the annotation
cost (see Figure 2(c)), and the model would be more sensitive to the noise on
such a small proportion of labels. To solve the above problem, some works [13,
22] attempt to re-weight the loss of each sample to weaken the influence of
noisy labels; some other works use semantic-aware representations to generate
pseudo labels [7] or blend category-specific semantic representation between
different images [31]. However, similar to the aforementioned methods in MLC,
these MLC-PL methods still use mislabeled samples in training, which would
lead to imprecise evaluation on loss weights and pseudo labels, and thus
degrade the performance.

Figure 2: Comparisons of different learning methods in MLC. (a) an image from
MS-COCO train2014 which has two missing labels. To train the sample image, (b)
missing labels in traditional MLC methods are mistakenly classified as
negative labels; (c) MLC-PL samples a proportion of labels, but still
encounters false negative labels; (d) Our method treats all negative labels as
unlabeled ones. Blue, red, and yellow icons denote positive, negative, and
unknown labels, respectively.

Considering the above discussions, as involving the noisy labels in training
would lead to inferior performance on MLC and MLC-PL, can we just remove them
before training? Frustratingly, we do not have an oracle that can precisely
identify noisy labels. In this paper, we propose a more radical way, that is,
if we cannot identify noisy labels, we just remove all the labels. Inspired by
positive-unlabeled (PU) learning [30, 26, 5], which trains binary classifiers
using labels on one (positive) category only and achieves competitive
performance to traditional positive-negative (PN) learning (see Figure 2(d)),
we propose to discard all the negative labels and train MLC models with only
positive and unlabeled data. Since the negative labels are considerably more
than positive labels in MLC datasets (see Figure 3), removing negative labels
can avoid most of the annotation errors in training. Furthermore, PU learning
is acknowledged to be more robust and accurate than PN learning when the
negative labels are noisy, since the PN may learn to rely on noisy negative
labels and make incorrect predictions. In contrast, PU adopts unbiased risk
estimator to perform a proper approximation to PN without noisy labels, and
the soft labels in estimator is more informative and accurate to label the
sample compared to the hard label in previous methods.

As a result, we propose a method based on PU learning, dubbed as positive and
unlabeled multi-label classification (PU-MLC). Concretely, we extend the
conventional PU learning to the task of combining multiple binary
classifications in MLC. Besides, to deal with the catastrophic imbalance
between positive and negative labels in MLC task, we propose an adaptive re-
balance factor in PU loss to balance the loss weight. Since it could be much
more challenging to train multiple binary classification tasks in MLC than the
simple practice in current PU learning methods, we further propose an adaptive
temperature coefficient module to adjust the sharpness of predicted
probabilities in loss, which effectively avoids the probabilistic distribution
to be over-smooth in the early stage of training, and thus benefits the
optimization.

Our proposed PU-MLC is simple and effective, and applies to both MLC and PU-
MLC tasks. Meanwhile, it can obtain promising performance with only a small
portion of positive labels, which could also reduce the cost of annotating.
Extensive experiments on two benchmark datasets MS-COCO [29] and PASCAL VOC
2007 [14] demonstrate that our PU-MLC achieves significantly improvements on
both MLC and MLC-PL settings, with even fewer annotated labels used. For
example, on MLC-PL, our PU-MLC improves the state-of-the-art SARB [31] by 3.0%
mAP under 10% known label ratio on MS-COCO (see Figure 1); while on MLC, our
PU-MLC achieves an extraordinary 83.6% mAP on MS-COCO, which outperforms our
PN learning baseline by 6.3%.

##  2 Related Work

###  2.1 Multi-Label Classification

Multi-label classification (MLC). Multi-label classification (MLC) task aims
to recognize semantic categories in a given image, which usually contains
multiple objects or concepts. Earlier works [41] treat MLC as a group of
binary classification tasks. However, these methods treat each binary
classification task independently, and thoroughly ignore the abundant
dependencies between labels. Therefore, some works [8, 11, 43] propose to
construct pairwise statistical correlations using the first-order adjacency
matrix obtained by graph convolutional networks (GCN) [25]. Although the above
methods achieve noteworthy success, they cannot extract higher-order
correlations and can attract overfitting on small training sets. Some works
[27, 45] introduce transformer to extract complicated dependencies among
visual features and labels. Meanwhile, to alleviate the mislabeling problem in
MLC datasets, confident learning [40] is introduced into MLC task to obtain
label quality scores and identify mislabeled data in each sample. ASL [33]
rejects mislabeled negative labels through a probability threshold.

Multi-label classification with partial labels (MLC-PL). Traditional multi-
label classification (MLC) tasks rely on fully annotated datasets, and making
such datasets is expensive, time-consuming, and error-prone. To reduce the
cost of annotation, multi-label classification with partial labels (MLC-PL)
attempts to train models with partially-annotated labels per image, which both
contain positive and negative labels. Primitive methods [3, 37, 23] directly
ignore unknown labels or treat them as negative, and use the traditional MLC
method to solve the MLC-PL challenge. However, such simple strategies would
result in inferior performance since they naively ignore the unknown training
samples or assign unreliable pseudo labels to the unknown samples. Therefore,
recent works [13, 22, 7] propose to generate pseudo labels to those unknown
samples based on the learned knowledge in the training model, and then train
the model with ground-truth partial labels and generated pseudo labels. For
example, Durand _et al_. [13] utilize a curriculum learning strategy to
predict the unknown labels, Huynh _et al_. [22] learn a convolutional neural
network to recognize the unknown labels. SST [7] takes semantic information
into account, then use the semantic-aware representations to generate pseudo
labels. Though these recent methods achieve improvements depending on the use
of internal semantic information in the learned model, they still have to
train the model with noisy labels, which would somehow mislead the
optimization and obtain imprecise knowledge in the model. In our paper, we try
to avoid the disturbance of noisy labels by discarding all the negative labels
and performing positive-unlabeled learning, which would have fewer noisy
labels in training.

###  2.2 Positive-Unlabeled (PU) learning

Different from the traditional positive-negative (PN) learning in the binary
classification task, PU learning aims to train the model with only positive
and unknown labels [1]. Recent advances [30, 26, 5, 20] have achieved
remarkable progress in deep learning. Specifically, uPU [30] introduces an
unbiased risk estimator to approximate the original classification risk in PN
learning, but it suffers from the severe overfitting issue in large neural
networks. To solve the problem, nnPU [26] proposes the non-negative risk
estimator to explicitly constrain the training risk of uPU to be non-negative.
However, these methods rely heavily on the class prior estimation. While the
class prior in the training dataset may not always correctly represent the
label distribution in the validation set, and thus performing PU learning
without class prior becomes an emergent topic [5, 18, 4, 15]. For example, vPU
[5] proposes a variational principle to achieve superior performance without
class prior. In this paper, we extend PU learning to MLC task based on vPU
[5].

##  3 Proposed Approach: PU-MLC

In this section, we will formulate our proposed method PU-MLC. We first extend
the existing positive-unlabeled (PU) learning method vPU [5] to MLC tasks with
multiple binary classification sub-tasks. Besides, we find that the
distributions of positive and negative labels in MLC datasets are quite
imbalanced, and directly leveraging the PU loss on such datasets would lead to
inferior performance. Therefore, we propose a re-balance factor on the
unlabeled term in PU loss, to dynamically balance the loss weight according to
the predicted probabilities in each sample. Finally, considering that the
optimization of multiple PU learning tasks in PU-MLC is more difficult than
the single task in traditional PU learning methods, we further propose an
adaptive temperature coefficient module to avoid the probabilistic
distribution to be over-smooth in the early stage of training.

Figure 3: (a) Overview of our proposed PU-MLC. Instead of using positive and
negative labels in the traditional MLC method (red box), our PU-MLC conducts a
positive-unlabeled (PU) learning strategy with only partial positive labels
leveraged. Besides, we introduce mixup regularization loss and the adaptive
temperature coefficient module to further boost the performance. Std: standard
deviation. (b) Histograms of the number of positive and negative labels in
each category. We randomly select 20 categories from MS-COCO train set.

###  3.1 MLC as PU learning

MLC as PN learning. MLC task is usually formulated as multiple binary
classification sub-tasks, and each sub-task aims to recognize whether a
specific category is in the input image. Formally, for a MLC task with $C$
categories, let $\bm{s}\in\mathbb{R}^{N\times C}$ and
$\bm{y}\in\\{-1,+1\\}^{N\times C}$ be the predicted logits and the ground-
truth positive and negative (PN) labels, respectively, where $N$ denotes batch
size, the overall classification loss is formulated as

$\displaystyle\mathcal{L}_{\mathrm{mlc}}=$ $\displaystyle\frac{1}{C\times
N}\sum_{c=1}^{C}\sum_{n=1}^{N}[\mathbbm{1}(y_{n,c}=+1)\mathcal{L}_{+}(\sigma(s_{n,c}))$
(1) $\displaystyle+\mathbbm{1}(y_{n,c}=-1)\mathcal{L}_{-}(\sigma(s_{n,c}))],$

where $\sigma(\cdot)$ is the Sigmoid function, $\mathbbm{1}(\cdot)$ is an
indicator function that takes the value 1 only if the condition is true and 0
otherwise, $\mathcal{L}_{+}$ and $\mathcal{L}_{-}$ denote losses on positive
and negative labels, respectively.

Before presenting our PU-learning based MLC method, we first rewrite the
learning objective of the above positive-negative (PN) classification loss
(equation 1) as the expected risk on the training set. The total risk
$R_{\mathrm{mlc}}$ is accumulated with all PN sub-tasks, and for each task
(category) with the class prior (proportion of positive labels)
$\pi_{\mathrm{p}}$ and $\bm{S}\in\mathbb{R}^{M}$ being its corresponding
logits on the training set with $M$ images, its risk is formulated as

$\displaystyle R_{\mathrm{pn}}$
$\displaystyle=\pi_{\mathrm{p}}\mathbb{E}_{\mathcal{P}}[\mathcal{L}_{+}(\sigma(\bm{S}))]+(1-\pi_{\mathrm{p}})\mathbb{E}_{\mathcal{N}}[\mathcal{L}_{-}(\sigma(\bm{S}))],$
(2)

where the images regarding to their label types are split into positive set
$\mathcal{P}$ and negative set $\mathcal{N}$, and we have the expectations of
positive and negative losses

$\displaystyle\mathbb{E}_{\mathcal{P}}[\mathcal{L}_{+}(\sigma(\bm{S}))]=\frac{1}{|\mathcal{P}|}\sum_{s_{m}\in\mathcal{P}}\mathcal{L}_{+}(\sigma(s_{m})),$
(3)
$\displaystyle\mathbb{E}_{\mathcal{N}}[\mathcal{L}_{-}(\sigma(\bm{S}))]=\frac{1}{|\mathcal{N}|}\sum_{s_{m}\in\mathcal{N}}\mathcal{L}_{-}(\sigma(s_{m})).$

PN to PU. In this paper, we aim to train a MLC model with only positive
labels; i.e., our training set is composed of a positive set $\mathcal{P}$ and
an unlabeled set $\mathcal{U}$ (mixture of unlabeled positive and negative
images). Nevertheless, the negative labels are unavailable in our PU setting,
and therefore we cannot directly optimize equation 2 to obtain our model. In
order to train a classifier with positive and unknown labels, a classical
method uPU [30] introduces an unbiased formulation to the PN learning by
rewriting the expectation of negative classification loss
$\mathbb{E}_{\mathcal{N}}[\mathcal{L}_{-}(\sigma(\bm{S}))]$ to

$\displaystyle(1-\bm{\pi}_{\mathrm{p}})\mathbb{E}_{\mathcal{N}}[\mathcal{L}_{-}(\sigma(\bm{S}))]$
$\displaystyle=\mathbb{E}_{\mathcal{U}}[\mathcal{L}_{-}(\sigma(\bm{S}))]$ (4)
$\displaystyle-\bm{\pi}_{\mathrm{p}}\mathbb{E}_{\mathcal{P}}[\mathcal{L}_{-}(\sigma(\bm{S}))],$

and thus equation 2 could be converted to PU format:

$\displaystyle R_{\mathrm{pu}}$
$\displaystyle=\bm{\pi}_{\mathrm{p}}\mathbb{E}_{\mathcal{P}}[\mathcal{L}_{+}(\sigma(\bm{S}))]$
(5)
$\displaystyle-\bm{\pi}_{\mathrm{p}}\mathbb{E}_{\mathcal{P}}[\mathcal{L}_{-}(\sigma(\bm{S}))]+\mathbb{E}_{\mathcal{U}}[\mathcal{L}_{-}(\sigma(\bm{S}))],$

However, the above method easily causes overfitting in deep neural networks
and rely heavily on the class prior, and we empirically find that it performs
poorly on the multi-label classification task, as the task is more challenging
and many categories have very small class priors. Fortunately, there are some
recent studies [18, 4, 15] propose to mitigate these issues, and our PU-MLC is
developed based on vPU [5], which proposes a novel loss function based on the
variational principle to approximate the ideal classifier without the class
prior:

$R_{\mathrm{var}}=\log{\mathbb{E}_{\mathcal{U}}[\sigma(\bm{S})]}-\mathbb{E}_{\mathcal{P}}[\log{\sigma(\bm{S})}].$
(6)

Therefore, for each category $c$, the classification loss becomes

$\displaystyle\mathcal{L}_{\mathrm{var}}^{(c)}=$
$\displaystyle\log(\frac{1}{|\mathcal{U}^{(c)}_{N}|}\sum_{s_{u}\in\mathcal{U}^{(c)}_{N}}\sigma(s_{u}))$
(7)
$\displaystyle-\frac{1}{|\mathcal{P}^{(c)}_{N}|}\sum_{s_{p}\in\mathcal{P}^{(c)}_{N}}\log\sigma(s_{p}),$

here $\mathcal{P}^{(c)}_{N}$ and $\mathcal{U}^{(c)}_{N}$ denote positive
samples and unlabeled samples of category $c$ in each mini-batch,
respectively.

Besides, vPU also introduces a consistency regularization term based on Mixup
[44], which alleviates the overfitting problem and increases the robustness in
PU learning, i.e.,

$\mathcal{L}_{\mathrm{reg}}^{(c)}=\frac{2}{N}\sum^{N/2}_{n=1}(\log(\sigma(\hat{s}_{n,c}))-\log(\sigma(\tilde{s}_{n,c})))^{2},$
(8)

with

$\displaystyle\mu\overset{iid}{\sim}\mathrm{Beta}(\delta,\delta),$ (9)
$\displaystyle\hat{\bm{s}}=\mu\cdot\bm{s}^{\prime}+(1-\mu)\cdot\bm{s}^{\prime\prime},$
$\displaystyle\tilde{\bm{s}}=\Phi(\mu\cdot\bm{x}^{\prime}+(1-\mu)\cdot\bm{x}^{\prime\prime}),$

where $\bm{s}^{\prime}$ and $\bm{s}^{\prime\prime}$ are the first and last
halves of predicted logits $\bm{s}\in\mathbb{R}^{N\times C}$,
$\bm{x}^{\prime}$ and $\bm{x}^{\prime\prime}$ represent the first and last
halves of input images $\bm{x}$, and $\tilde{\bm{s}}$ denotes the prediction
of the model $\Phi$ with mixed images as input.

As a result, in our PU-MLC, the traditional MLC loss in equation 1 is replaced
with our PU loss, and the overall loss function is formulated as

$\mathcal{L}_{\mathrm{pu-
mlc}}=\sum_{c=1}^{C}(\mathcal{L}_{\mathrm{var}}^{(c)}+\lambda\mathcal{L}_{\mathrm{reg}}^{(c)}),$
(10)

where $\lambda$ is a scalar to balance the losses and we set $\lambda=1$ in
all experiments.

Note that unlike conventional PU learning, we complement all the positive
samples in $\mathcal{P}$ into $\mathcal{U}$ to keep the unlabeled set
$\mathcal{U}$ have a similar label distribution as the traditional training
set, which is important for PU learning (see ablation studies in Section 4.4
for details).

###  3.2 Catastrophic Imbalance of Label Distribution

In MLC datasets, the total number of negative labels are generally much larger
than the positive labels, as summarized in Figure 3. Since our PU-MLC removes
all the negative labels and appends them into the unlabeled set, the number of
samples contributing to the first and second terms of
$\mathcal{L}_{\mathrm{var}}$ in equation 7 are extremely discrepant in each
mini-batch. However, in PU learning methods, the positive and negative samples
are carefully designed to have the same size in one batch, and directly
optimize equation 7 in our method would make the unlabeled term dominate the
optimization result in poor result in MLC-PL when the known label ratio is low
(e.g., only 54.8% mAP with 10% positive labels).

To alleviate the catastrophic imbalance of label distribution, we aim to
narrow down the loss weight of unlabeled term to decrease its importance in
optimization. Inspired by focal loss [28] and ASL [33], we propose a re-
balance factor to dynamically re-weight the unlabeled loss based on the
predicted probabilities on unlabeled samples, and equation 7 is reformulated
as

$\displaystyle\mathcal{L}_{\mathrm{var}}^{(c)}=$ $\displaystyle
p^{\gamma}_{c}\log(\frac{1}{|\mathcal{U}^{(c)}_{N}|}\sum_{s_{u}\in\mathcal{U}^{(c)}_{N}}\sigma(s_{u}))$
(11)
$\displaystyle-\frac{1}{|\mathcal{P}^{(c)}_{N}|}\sum_{s_{p}\in\mathcal{P}^{(c)}_{N}}\log\sigma(s_{p}),$

where $p_{c}^{\gamma}$ denotes our re-balance factor, with
$p_{c}=\frac{1}{|\mathcal{U}|}\sum_{s_{u}\in\mathcal{U}}\sigma(s_{u})$ being
the mean probability of unlabeled samples, and $\gamma$ denotes the
exponential to control the value of the factor. In our experiments, we set
larger $\gamma$ for smaller known label ratios, as the imbalance is severer on
smaller ratios and we need a smaller weight on unlabeled loss to balance the
loss.

Table 1: The comparisons on MS-COCO and VOC 2007 under different known label ratios. Note that our PU-MLC only uses partial positive labels, while other methods train models with the same number of positive labels and additional negative labels. ∗ indicates backbone models using weights pretrained by CLIP [32]. Results except DualCoOp and our method are reported by SARB [31]. Datasets | Methods | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% |  Avg. mAP |  Avg. OF1 |  Avg. CF1
---|---|---|---|---|---|---|---|---|---|---|---|---|---
MS-COCO |  SSGRL [8] | 62.5 | 70.5 | 73.2 | 74.5 | 76.3 | 76.5 | 77.1 | 77.9 | 78.4 | 74.1 | 73.9 | 68.1
ML-GCN [11] | 63.8 | 70.9 | 72.8 | 74.0 | 76.7 | 77.1 | 77.3 | 78.3 | 78.6 | 74.4 | 73.1 | 68.4
KGGR [6] | 66.6 | 71.4 | 73.8 | 76.7 | 77.5 | 77.9 | 78.4 | 78.7 | 79.1 | 75.6 | 73.7 | 69.7
P-GCN [9] | 67.5 | 71.6 | 73.8 | 75.5 | 77.4 | 78.4 | 79.5 | 80.7 | 81.5 | 76.2 | - | -
ASL [33] | 69.7 | 74.0 | 75.1 | 76.8 | 77.5 | 78.1 | 78.7 | 79.1 | 79.7 | 76.5 | 46.7 | 47.9
CL [22] | 26.7 | 31.8 | 51.5 | 65.4 | 70.0 | 71.9 | 74.0 | 77.4 | 78.0 | 60.7 | 61.9 | 48.3
Partial BCE [22] | 61.6 | 70.5 | 74.1 | 76.3 | 77.2 | 77.7 | 78.2 | 78.4 | 78.5 | 74.7 | 74.0 | 68.8
SST [7] | 68.1 | 73.5 | 75.9 | 77.3 | 78.1 | 78.9 | 79.2 | 79.6 | 79.9 | 76.7 | - | -
SARB [31] | 72.5 | 76.0 | 77.6 | 78.7 | 79.6 | 79.8 | 80.0 | 80.5 | 80.8 | 78.4 | 76.8 | 72.7
DualCoOp∗ [38] | 78.7 | 80.9 | 81.7 | 82.0 | 82.5 | 82.7 | 82.8 | 83.0 | 83.1 | 81.9 | 78.1 | 75.3
| PU-MLC | 75.5 | 78.2 | 79.7 | 80.9 | 81.5 | 81.9 | 82.3 | 82.7 | 83.2 | 80.7 | 77.0 | 75.7
| PU-MLC∗ | 80.0 | 82.2 | 83.3 | 84.3 | 84.7 | 85.2 | 85.6 | 86.0 | 86.4 | 84.2 | 80.7 | 78.6
VOC 2007 |  SSGRL [8] | 77.7 | 87.6 | 89.9 | 90.7 | 91.4 | 91.8 | 92.0 | 92.2 | 92.2 | 89.5 | 87.7 | 84.5
ML-GCN [11] | 74.5 | 87.4 | 89.7 | 90.7 | 91.0 | 91.3 | 91.5 | 91.8 | 92.0 | 88.9 | 87.3 | 84.6
KGGR [6] | 81.3 | 88.1 | 89.9 | 90.4 | 91.2 | 91.3 | 91.5 | 91.6 | 91.8 | 89.7 | 86.5 | 84.7
P-GCN [9] | 82.5 | 85.4 | 88.2 | 89.8 | 90.0 | 90.9 | 91.6 | 92.5 | 93.1 | 89.3 | - | -
ASL [33] | 82.9 | 88.6 | 90.0 | 91.2 | 91.7 | 92.2 | 92.4 | 92.5 | 92.6 | 90.5 | 41.0 | 40.9
CL [22] | 44.7 | 76.8 | 88.6 | 90.2 | 90.7 | 91.1 | 91.6 | 91.7 | 91.9 | 84.1 | 83.8 | 75.4
Partial BCE [22] | 80.7 | 88.4 | 89.9 | 90.7 | 91.2 | 91.8 | 92.3 | 92.4 | 92.5 | 90.0 | 87.9 | 84.8
SST [7] | 81.5 | 89.0 | 90.3 | 91.0 | 91.6 | 92.0 | 92.5 | 92.6 | 92.7 | 90.4 | - | -
SARB [31] | 85.7 | 89.8 | 91.8 | 92.0 | 92.3 | 92.7 | 92.9 | 93.1 | 93.2 | 91.5 | 88.3 | 86.0
DualCoOp∗ [38] | 90.3 | 92.2 | 92.8 | 93.3 | 93.6 | 93.9 | 94.0 | 94.1 | 94.2 | 93.2 | 86.3 | 84.2
| PU-MLC | 87.9 | 90.4 | 91.6 | 91.9 | 92.2 | 92.5 | 92.8 | 93.2 | 93.4 | 91.7 | 88.2 | 86.6
| PU-MLC∗ | 92.0 | 92.5 | 93.0 | 93.4 | 93.8 | 94.0 | 94.2 | 94.6 | 94.8 | 93.6 | 89.8 | 88.6

###  3.3 Adaptive Temperature Coefficient

In PU learning, the training model actually acts as an estimator to generate
the probabilistic estimation on each unlabeled samples and optimize them with
the unlabeled loss term [1]. However, the task of learning multiple binary
classifiers in MLC is much more challenging than learning a single binary
classifier in conventional PU learning methods. This gap in learning
difficulty leads to a smaller convergent rate at the earlier phase of training
in our PU-MLC, and thus the predicted probabilistic distribution is over-
smooth, which makes the unlabeled loss less effective.

To adjust the smoothness of probabilistic distribution, we follow [17] and
propose a temperature coefficient $\tau$ to scale the logit values, i.e.,

$\bm{s}_{t}=\bm{s}/\tau,$ (12)

then the $\bm{s}_{t}$ is fed into the PU loss in place of the original
$\bm{s}$.

By setting $\tau<1$, we can make the probabilistic distribution sharper, and
thus provide more informative and effective signals to the loss. However, we
empirically find that fixed temperature coefficient can only improve the
performance on specific known label ratios and dataset (see Table 7). For
example, MS-COCO dataset wants a $\tau<1$ to gain improvements, while $\tau<1$
performs worse than $\tau>1$ on VOC. This implies that, for different datasets
with different learning difficulties, even for different categories in the
same dataset, the optimal $\tau$ are different and needs to be set
individually.

As a result, we propose an adaptive temperature coefficient module to first
measure the sharpness of each category in each batch, then apply independent
temperatures on each category. Formally, given the predicted logits $\bm{s}$,
the sharpness of each category $c$ is measured using the standard deviation of
the logits, and then the temperature is obtained by multiplying a scalar
$\alpha$ onto the sharpness value, i.e.,

$\tau^{(c)}=\min(\alpha\cdot\mathrm{Std}(\bm{s}_{c}),1),$ (13)

where $min(\cdot)$ denotes minimum function to ensure that the $\tau^{(c)}$ is
less than or equal to 1. The temperature is specifically designed to mitigate
the over-smoothing of probabilities. However, when $\tau^{(c)}$ exceeds 1, it
can exacerbate this phenomenon.

The final PU loss $\mathcal{L}_{\mathrm{val}}^{(c)}$ becomes

$\displaystyle\mathcal{L}_{\mathrm{var}}^{(c)}=$ $\displaystyle
p^{\gamma}_{c}\log(\frac{1}{|\mathcal{U}^{(c)}_{N}|}\sum_{s_{u}\in\mathcal{U}^{(c)}_{N}}\sigma(s_{u}/\tau^{(c)}))$
(14)
$\displaystyle-\frac{1}{|\mathcal{P}^{(c)}_{N}|}\sum_{s_{p}\in\mathcal{P}^{(c)}_{N}}\log\sigma(s_{p}/\tau^{(c)}).$

Our adaptive temperature coefficient is suitable for different known label
ratios and datasets, which could gain consistent improvements. The overall
framework of our model is illustrated in Figure 3.

##  4 Experiments

###  4.1 Settings

Datasets. To verify the efficacy of PU-MLC, we conduct extensive experiments
on the two most popular benchmarks MS-COCO [29] and PASCAL VOC [14].

MS-COCO is a large-scale image dataset that is widely used in computer vision
fields such as object detection, semantic segmentation, and image
classification. It contains 123,287 images with 80 categories and an average
of 2.9 positive labels per image. According to the official division standard,
we divided the data into two parts: 82,783 images for training and 40,504
images for testing.

PASCAL VOC 2007 is one of the most widely used datasets for multi-label image
classification tasks. It contains 20 categories and is divided into three
parts: training (2501), validation (2510) and testing (4952). Following
previous works, we use training and validation sets for training and test sets
for evaluating.

Training strategies. Similar to previous works [7, 31], we adopt the
ResNet-101 [16] as the backbone to extract image features, and then obtain
logit scores with the same decoupling module and linear classifier. We
initialize the backbone with parameters pre-trained on the ImageNet [12]
dataset, and fix its parameters of the previous 91 layers. We train the model
for 40 epochs using Adam [24] optimizer and 1-cycle [36] policy, with a
maximum learning rate of 1e-4. During training, we resize the input image to
512 $\times$ 512, and randomly choose a number from 512, 448, 384, 320, 256 as
the width and height to crop a patch. Then, the patch is further resized to
448 $\times$ 448\. Moreover, we also use random horizontal flipping. The
parameter $\alpha$ a is fixed to be 1.0. During testing, we simply resize the
input image to 448 $\times$ 448 for evaluation.

Since experimental datasets are fully annotated, following previous works [13,
22, 31], for a fair comparison, we randomly drop labels ranging from 10% to
90%. Moreover, all the negative labels are dropped to satisfy the PU setting
in our algorithm. At 10% to 90% known label proportions, the hyper-parameter
$\gamma$ is set to 0.55, 0.50, 0.35, 0.30, 0.25, 0.20, 0.15, 0.10, 0.05,
respectively.

Evaluation Metrics. We refer to follow works [7, 31], which compute the mean
average precision (mAP) separately on the proportion of known labels ranging
from 10% to 90%, and report the average mAP. Furthermore, we also adpot the
overall F1-measure (OF1) and per-class F1-measure (CF1) to evaluate the model
comprehensively. Specifically, OF1 and CF1 are calculated as:

$\displaystyle\mathrm{OP}=\frac{\sum_{i}{N_{i}^{c}}}{\sum_{i}{N_{i}^{p}}}\qquad\qquad\mathrm{CP}=\frac{1}{C}\sum_{i}{\frac{N_{i}^{c}}{N_{i}^{p}}}$
(15)
$\displaystyle\mathrm{OR}=\frac{\sum_{i}{N_{i}^{c}}}{\sum_{i}{N_{i}^{g}}}\qquad\qquad\mathrm{CR}=\frac{1}{C}\sum_{i}{\frac{N_{i}^{c}}{N_{i}^{g}}}$
$\displaystyle\mathrm{OF1}=\frac{2\times\mathrm{OP}\times\mathrm{OR}}{\mathrm{OP}+\mathrm{OR}}\;\mathrm{CF1}=\frac{2\times\mathrm{CP}\times\mathrm{CR}}{\mathrm{CP}+\mathrm{CR}}$

where $N_{i}^{c}$ is the number of images that are correctly predicted for the
$i$-th label, $N_{i}^{p}$ is the number of predicted images for the $i$-th
label, and $N_{i}^{g}$ is the number of ground truth images for the $i$-th
label. We also report the average OF1 and CF1 for all known label proportions.

###  4.2 Results on MS-COCO

MLC-PL setting. To demonstrate the effectiveness of the PU-MLC, we compare our
PU-MLC with current published state-of-the-art methods, including SSGRL [8],
ML-GCN [11], KGGR [6], P-GCN [9], ASL [33], CL [22], Partial BCE [22], SST
[7], SARB [31] and DualCoOp [38]. As the experimental results shown in Table
1, our PU-MLC significantly outperforms previous methods under different known
label ratios. For example, we achieve the results of P-GCN with 40% known
labels with only 10% of positive labels. Besides, with 20% positive labels,
our PU-MLC even obtains similar performance as 90% known labels in SSGRL.
Moreover, on a high known label ratio of 90%, we obviously surpass SARB by
2.4% in mAP. Compared with previous methods, our method achieves state-of-the-
art results in average mAP, OF1 and CF1, which are 80.7%, 77.0% and 75.7%,
respectively. DualCoOp uses CLIP [32], a large-scale vision-language pre-
trained model, as its backbone to achieve exceptional performance. For a fair
comparison, by only using the same visual model, our method achieves superior
performance than DualCoOp, which utilizes both visual and language models.

Note that these significant improvements are obtained with even fewer
annotated labels used in training compared to other methods (e.g., with 10%
known label ratio, we only use 10% positive labels, while other methods use
10% positive labels and 10% negative labels), this indicates that our method
is more effective and efficient on limited training annotations. As shown in
Table 2, the number of annotated labels used by PU-MLC in model training is
much smaller than other methods based on PN. Concretely, our method achieves
the best results while decreasing the amount of annotated labels by 96.4% at
each known label ratio.

Table 2: Comparisons of the number of annotated labels used in training on MS-COCO. Reduction: the reduction ratio on used training annotations of our method compared to other methods. Methods | PU-MLC | Others
---|---|---
10% | 50% | 100% | 10% | 50% | 100%
Positive | 24,103 | 120,517 | 241,035 | 24,103 | 120,517 | 241,035
Negative | 0 | 0 | 0 | 638,160 | 3,190,802 | 6,381,605
Total | 24,103 | 120,517 | 241,035 | 662,263 | 3,311,319 | 6,622,640
Reduction | -96.4% | -96.6% | -96.4% | - | - | -

MLC setting. Since our method is designed for both MLC and MLC-PL tasks, we
also conduct experiments to validate our performance on traditional MLC. As
shown in Table 3, we achieve promising performance compared to previous
methods. Similar to MLC-PL, our method in MLC is trained with only positive
labels, and discards a large number of negative labels (negative labels are
$\sim 26.5\times$ more than positive labels), our results can still outperform
those methods trained with full annotations. Besides, compared with our PN
learning baseline ResNet-101 [16], our MLC-PL significantly outperforms it by
6.3% in mAP, which demonstrates that our method is beneficial to MLC setting
by ignoring those noisy negative labels.

Table 3: Comparisons of mAP on MS-COCO in MLC setting. Methods | mAP | OF1 | CF1
---|---|---|---
ResNet-101 [16] | 77.3 | 76.8 | 72.8
Cop [42] | 81.1 | 75.1 | 72.7
DSDL [46] | 81.7 | 75.6 | 73.4
CADM [10] | 82.3 | 79.6 | 77.0
ML-GCN [11] | 83.0 | 80.3 | 78.0
PU-MLC | 83.6 | 77.8 | 78.6

###  4.3 Results on Pascal VOC 2007

Table 1 similarly shows the comparison results between PU-MLC and state-of-
the-art methods on Pascal VOC. Although Pascal VOC has a small size of the
sample and simple categories, and many previous methods achieve splendid
results, we still outperform them on average mAP and CF1. Especially on the
most challenging 10% known labels, our obviously surpass SARB by 2.2% in mAP.
Besides, we achieve the results of KGGR with 40% known labels with only 20% of
positive labels. On high known label ratios, our performance improvements are
not as significant as that in MS-COCO dataset, a possible reason is that VOC
dataset is much easier and smaller than MS-COCO, and using the previous
methods can also obtain impressive performance. Besides, we compare our method
to DualCoOp, which utilizes both the Clip pre-trained visual model and
language models. By using only the same visual model, our approach achieves an
improvement across each known label ratio. Notably, we achieve a significant
1.7% mAP improvement on the 10% known label ratio.

###  4.4 Ablation Studies

Ablation on proposed components. To understand the contribution of each
component in our proposed method, we conduct ablation experiments under three
proportions of known labels: low (10%), medium (50%), and high (100%), where
the 100% known label ratio is equivalent to MLC setting. We treat BCE as PN
learning baseline and it simply ignores unknown labels under MLC-PL setting.
The results are summarized in Table 4. 1st row vs. 2nd row: compared to BCE
loss, directly using vPU achieves poor results on MLC-PL task, especially on
10% known labels. As in the PU setting, when the proportion of known labels
decreases, the quantity gap between the positive and unknown labels becomes
larger, which makes the model be dominated by the unknown labels and neglects
the positive labels. 2nd row vs. 3rd row: In contrast, our re-balance factor
enables original vPU to achieve a great improvement of 19.8% on the 10%
proportion of known labels, which indicates that it greatly alleviates the
contribution imbalance between positive and unknown labels. 3rd row vs. 4th
row: Finally, the addition of the adaptive temperature coefficient further
improves the performance, especially when the proportion of known labels is
10%, as it helps the loss to optimize on sharper predicted probabilities, and
makes the PU loss more effective.

Table 4: Ablation on the proposed components on MS-COCO. RBF: re-balance factor. ATC: adaptive temperature coefficient. Methods | RBF | ATC | 10% | 50% | 100%
---|---|---|---|---|---
BCE | - | - | 73.1 | 78.9 | 80.0
Ours | ✗ | ✗ | 54.8 | 78.9 | 83.0
Ours | ✓ | ✗ | 74.6 | 81.3 | 83.5
Ours | ✓ | ✓ | 75.5 | 81.5 | 83.6

Effect of complementing $\mathcal{P}$ to $\mathcal{U}$. We append all the
samples in the positive set to the unlabeled set to obtain a similar label
distribution to the original PN training dataset. To verify the effectiveness,
we conduct experiments under three proportions of known labels to compare the
performance with or without the complement. As shown in Table 5, the setting
with 90% proportion of known labels obtains a relatively large improvement,
while the gain on 10% proportion is small. The reason behind this is simple:
when we have only a small portion (10%) of positive labels in the positive
set, most of the remaining positive samples (90%) are in the unlabeled set,
and thus the unlabeled set naturally holds a similar label distribution to the
dataset. While on large portions, the label distribution in the unlabeled set
is quite different to the origin (taking an extreme 100% portion as an
example, there are no positive samples in the unlabeled set).

Table 5: Comparisons of adding and not adding $\mathcal{P}$ to $\mathcal{U}$ on MS-COCO. Methods | 10% | 50% | 90%
---|---|---|---
Not adding $\mathcal{P}$ to $\mathcal{U}$ | 75.4 | 81.0 | 82.1
Adding $\mathcal{P}$ to $\mathcal{U}$ | 75.5 | 81.5 | 83.2

Effects of the exponential $\gamma$ in re-balance factor. As shown in Table 6,
we perform analysis of the exponential $\gamma$ on MS-COCO without adaptive
temperature coefficient. Since there are far fewer positive labels than
unknown labels at 10% proportion of known labels, the contribution of positive
labels in training is heavily neglected under $\gamma$ = 0, leading to poor
performance. When we fix $\gamma=0.3$, the above issue is significantly
alleviated. However, as previously discussed, the degree of adjustment is
individual under different proportions of known labels. In our setting, we set
smaller $\gamma$ on larger proportions, allowing the model to achieve better
performance.

Table 6: Comparisons of adopting different $\gamma$ in our loss. Methods | 10% | 50% | 90%
---|---|---|---
Fixed $\bm{\gamma}$ = 0 | 54.8 | 78.9 | 82.1
Fixed $\bm{\gamma}$ = 0.3 | 73.4 | 81.0 | 81.6
Our setting | 74.6 | 81.3 | 83.1

Effects of the temperature coefficient. As shown in Table 7, on MS-COCO,
compared with not using $\tau$ in the model training, the adaptive $\tau$ can
significantly improve the experimental results, especially when the proportion
of known labels is 10%. However, the VOC 2007 shows a different trend, all the
fixed temperatures degrade the performance. This indicates that we must
treating different datasets independently to achieve promising improvements.
In this paper, instead of manually searching the optimal temperature on each
setting, we propose adaptive temperature, which gains improvements
consistently on all settings.

Table 7: Ablation on the temperature coefficient. We report mAP (%) on MS-COCO and VOC 2007.  Methods | MS-COCO | VOC 2007
---|---|---
10% | 50% | 90% | 10% | 50% | 90%
Without $\tau$ | 74.6 | 81.3 | 83.1 | 87.4 | 92.1 | 93.3
Fixed $\tau$ = 0.2 | 75.2 | 81.4 | 82.4 | 85.5 | 91.8 | 92.5
Fixed $\tau$ = 2 | 72.1 | 79.4 | 81.7 | 86.3 | 91.4 | 92.7
Adaptive $\tau$ | 75.5 | 81.5 | 83.2 | 87.9 | 92.2 | 93.4

##  5 Conclusion

In this paper, we propose positive and unlabeled multi-label classification
(PU-MLC). By removing all the negative labels in training, our method benefits
from the cleaner annotations. Besides, we introduce an adaptive re-balance
factor and adaptive temperature coefficient to better adapt PU learning in MLC
task, which achieves significant improvements, especially on small known label
proportions. Extensive experiments on MS-COCO and PASCAL VOC datasets
demonstrate our efficacy. Adopting more advanced PU learning methods and
combining recent approaches on model architectures in MLC would be a potential
direction of improving PU-MLC.

## References

  * [1] Jessa Bekker and Jesse Davis.  Learning from positive and unlabeled data: a survey.  Machine Learning, pages 719–760, 2020.
  * [2] Emanuel Ben-Baruch, Tal Ridnik, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, and Lihi Zelnik-Manor.  Multi-label classification with partial annotations using class-aware selective loss.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4764–4772, 2022.
  * [3] Serhat Selcuk Bucak, Rong Jin, and Anil K. Jain.  Multi-label learning with incomplete class assignments.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2801–2808, 2011.
  * [4] Shizhen Chang, Bo Du, and Liangpei Zhang.  Positive unlabeled learning with class-prior approximation.  Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 2014–2021, 2021.
  * [5] Hui Chen, Fangqing Liu, Yin Wang, Liyue Zhao, and Hao Wu.  A variational approach for learning from positive and unlabeled data.  Advances in Neural Information Processing Systems, pages 14844–14854, 2020.
  * [6] Tianshui Chen, Liang Lin, Riquan Chen, Xiaolu Hui, and Hefeng Wu.  Knowledge-guided multi-label few-shot learning for general image recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1371––1384, 2022.
  * [7] Tianshui Chen, Tao Pu, Hefeng Wu, Yuan Xie, and Liang Lin.  Structured semantic transfer for multi-label recognition with partial labels.  Proceedings of the AAAI conference on artificial intelligence, pages 339–346, 2022.
  * [8] Tianshui Chen, Muxin Xu, Xiaolu Hui, Hefeng Wu, and Liang Lin.  Learning semantic-specific graph representation for multi-label image recognition.  Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 522–531, 2019.
  * [9] Zhaomin Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo.  Learning graph convolutional networks for multi-label recognition and applications.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [10] Zhao-Min Chen, Xiu-Shen Wei, Xin Jin, , and Yanwen Guo.  Multi-label image recognition with joint class-aware map disentangling and label correlation embedding.  2019 IEEE International Conference on Multimedia and Expo (ICME), pages 622–627, 2019.
  * [11] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo.  Multi-label image recognition with graph convolutional networks.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5177–5186, 2019.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  2009 IEEE conference on computer vision and pattern recognition, pages 248–255, 2009.
  * [13] Thibaut Durand, Nazanin Mehrasa, and Greg Mori.  Learning a deep convnet for multi-label classification with partial labels.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 647–657, 2019.
  * [14] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [15] Chen Gong, Qizhou Wang, Tongliang Liu, Bo Han, Jane You, Jian Yang, and Dacheng Tao.  Instance-dependent positive and unlabeled learning with labeling bias estimation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4163–4177, 2021.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [18] Wenpeng Hu, Ran Le, Bing Liu, Feng Ji, Jinwen Ma, Dongyan Zhao, and Rui Yan.  Predictive adversarial learning from positive and unlabeled data.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(9):7806–7814, 2021.
  * [19] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu.  Knowledge distillation from a stronger teacher.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [20] Tao Huang, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu.  Greedynasv2: Greedier search with a greedy path filter.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11902–11911, 2022.
  * [21] Tao Huang, Shan You, Bohan Zhang, Yuxuan Du, Fei Wang, Chen Qian, and Chang Xu.  Dyrep: Bootstrapping training with dynamic re-parameterization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 588–597, 2022.
  * [22] Dat Huynh and Ehsan Elhamifar.  Interactive multi-label cnn learning with partial labels.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9423–9432, 2020.
  * [23] Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache.  Learning visual features from large weakly supervised data.  European Conference on Computer Vision, pages 67––84, 2016.
  * [24] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [25] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * [26] Ryuichi Kiryo, Gang Niu, Marthinus C. du Plessis, and Masashi Sugiyama.  Positive-unlabeled learning with non-negative risk estimator.  Advances in neural information processing systems, pages 1675––1685, 2017.
  * [27] Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi.  General multi-label image classification with transformers.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16478–16488, 2021.
  * [28] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [29] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  European conference on computer vision, pages 740–755, 2014.
  * [30] Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama.  Convex formulation for learning from positive and unlabeled data.  International conference on machine learning, pages 1386–1394, 2015\.
  * [31] Tao Pu, Tianshui Chen, Hefeng Wu, Yongyi Lu, and Liang Lin.  Semantic-aware representation blending for multi-label image recognition with partial labels.  arXiv preprint arXiv:2205.13092, 2022.
  * [32] Alec Radford, Jong Wook Kim, Chris Hallacy abd Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  International conference on machine learning, pages 8748–8763, 2021\.
  * [33] Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor.  Asymmetric loss for multi-label classification.  Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 82–91, 2021.
  * [34] Tal Ridnik, Hussam Lawen, Asaf Noy, Emanuel Ben Baruch, Gilad Sharir, and Itamar Friedman.  Tresnet: High performance gpu-dedicated architecture.  proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1400–1409, 2021.
  * [35] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [36] Leslie N. Smith.  A disciplined approach to neural network hyper-parameters: Part 1 – learning rate, batch size, momentum, and weight decay.  arXiv preprint arXiv:1803.09820, 2018.
  * [37] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017.
  * [38] Ximeng Sun, Ping Hu, and Kate Saenko.  Dualcoop: Fast adaptation to multi-label recognition with limited annotations.  arXiv preprint arXiv:2206.09541, 2022.
  * [39] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [40] Aditya Thyagarajan, Elías Snorrason, Curtis Northcutt, and Jonas Mueller.  Identifying incorrect annotations in multi-label classification data.  arXiv preprint arXiv:2211.13895, 2022.
  * [41] Grigorios Tsoumakas and Ioannis Katakis.  Multi-label classification: An overview.  International Journal of Data Warehousing and Mining (IJDWM), 3(3):1–13, 2007.
  * [42] Shiping Wen, Weiwei Liu, Yin Yang, Pan Zhou, Zhenyuan Guo, Zheng Yan, Yiran Chen, , and Tingwen Huang.  Multilabel image classification via feature/label co-projection.  IEEE Transactions on Systems, Man, and Cybernetics: Systems, 51(11):7250–7259, 2020.
  * [43] Renchun You, Zhiyao Guo, Lei Cui, Xiang Long, Yingze Bao, and Shilei Wen.  Cross-modality attention with semantic graph embedding for multi-label classification.  Proceedings of the AAAI conference on artificial intelligence, 34(7):12709–12716, 2020.
  * [44] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, 2017.
  * [45] Jiawei Zhao, Ke Yan, Yifan Zhao, Xiaowei Guo, Feiyue Huang, and Jia Li.  Transformer-based dual relation graph for multi-label image recognition.  Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 163–172, 2021.
  * [46] Fengtao Zhou, Sheng Huang, and Yun Xing.  Deep semantic dictionary learning for multi-label image classification.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(4):3572–3580, 2021.

##  Appendix A Detailed Training Strategies

Following previous works [7, 31], we adopt ResNet-101 [16] as the backbone to
extract image features, and then obtain logit scores with the same decoupling
module, gated graph neural network and linear classifier. We initialize the
backbone with parameters pre-trained on ImageNet [12] dataset, and the
parameters of the previous 91 layers and the running stats of all batch
normalization layers are fixed. Adam [24] optimizer is adopted with a 1-cycle
[36] learning rate policy. Besides, an exponential moving average (EMA) on
model is adopted. Specifically, for MLC-PL setting, we follow previous works
[13, 22, 31] and randomly drop labels ranging from 10% to 90% in the fully-
annotated MLC datasets to build the partially-annotated datasets. Meanwhile,
all the negative labels are dropped on both MLC and MLC-PL settings to satisfy
the PU setting in our algorithm. For the data augmentations in training, we
first resize the input image to 512 $\times$ 512, and then adopt random
cropping with 512, 448, 384, 320, and 256 pixels, the final input image is
resized to 448 $\times$ 448 followed by random horizontal flipping and
normalization. During test, the input image is simply resized to 448 $\times$
448 without cropping.

On MS-COCO, we train the model for 40 epochs with a maximum learning rate of
1e-4 and a batch size of 100. At 10% to 90% known label proportions, the
hyper-parameter $\gamma$ is set to 0.55, 0.50, 0.35, 0.30, 0.25, 0.20, 0.15,
0.10, and 0.05, respectively. Besides, the $\gamma$ is set to 0.03 in the MLC
task. The parameter $\alpha$ a is fixed to 1.0.

On PASCAL VOC, we train the model for 55 epochs with a maximum learning rate
of 4e-5 and a batch size of 128. At 10% to 90% known label proportions, the
hyper-parameter $\gamma$ is set to 0.85, 0.65, 0.55, 0.50, 0.45, 0.35, 0.15,
0.05, and 0.01, respectively. The parameter $\alpha$ a is fixed to 2.0.

##  Appendix B Effect of ATC in Alleviating Over-Smooth

Figure 4: Comparisons of the discrepancy between the probabilities of positive
and negative labels with or without ATC.

The original form of our PU loss in equation 11 would lead to over-smooth
distributions in the early phase of training due to the learning difficulty.
This small discrepancy between the probabilities of positive and negative
samples would make the unlabeled term of PU loss hard to optimize. To
alleviate the over-smooth problem, we propose adaptive temperature coefficient
(ATC) to adjust the sharpness of distribution.

To prove that the ATC is beneficial to magnify the discrepancy of positive and
negative samples, we measure the Kullback-Leibler divergence between the
predicted probabilities of positive and negative labels in the first 20 epochs
of training on 10% known label ratio. As shown in Figure 4, ATC can
significantly enlarge the discrepancy between the predicted probabilities of
positive and negative labels in the early stage of model training, thereby
alleviating the over-smooth.

The formula of measuring discrepancy:

$D_{KL}(P_{p}||P_{n})=\frac{1}{I\times
C}\sum_{i=1}^{I}\sum_{c=1}^{C}P_{p}^{i,c}\log(\frac{P_{p}^{i,c}}{P_{n}^{i,c}}),$
(16)

where $I$ denotes the number of iterations in an epoch. $P_{p}^{i,c}$ and
$P_{n}^{i,c}$ denote average prediction probabilities of the positive and
negative labels of category $c$ in the $i$-th iteration, respectively.

##  Appendix C More Ablation Studies

###  C.1 Dropping Negative Labels or Positive Labels?

In this paper, we remove negative labels and use positive labels to conduct PU
learning. While we can also remove positive labels and treat negative labels
as “positive” in PU learning. To compare these two choices, we conduct
experiments and report the results of dropping negative labels and dropping
positive labels in Table 8. The results show that, dropping negative labels
obtains obviously better performance. This imply that, the noisy labels in
positive labels are fewer than that in negative labels, and we can achieve
better performance on the cleaner positive labels, though the positive labels
is much fewer than the negative ones.

Table 8: Comparisons of dropping positive labels and negative labels on MS-COCO and VOC. We report the mAP (%).  Dataset | MS-COCO | VOC 2007
---|---|---
Dropping positive | 78.8 | 91.7
Dropping negative | 83.0 | 93.5

###  C.2 Manually Adding Noisy Labels

In this paper, we aim to alleviate the influence of noisy labels in negative
labels and introduce PU-MLC. However, COCO dataset is a finely annotated
dataset and we cannot know how many labels are incorrect in it. To further
validate our efficacy to different degrees of noisy ratios in dataset, we
conduct experiments to manually add noisy labels to the training dataset, and
see how the noisy ratios affect the performance on PU-MLC and our BCE
baseline. As shown in Table 9, we remove 10%, 50%, and 90% positive labels and
labeled them as negatives, and therefore these new negative labels are false
negative noises, we then train our PU-MLC and baseline BCE on these datasets.
The results show that, the impact of noisy labels on our method is less than
BCE at each noisy ratio. Particularly, with 90% positive labels shifted into
noisy labels, the mAP is significantly dropped by 58.4% in BCE, while our PU-
MLC only has a 8.1% reduction. This indicates that our method is more robust
than the traditional PN-based method on noisy labels (false negative).

Table 9: Results of PU-MLC and BCE under different noisy label ratios. We report mAP (%) on COCO. Methods | PU-MLC | BCE
---|---|---
10% | 50% | 90% | 10% | 50% | 90%
Original | 83.6 | 83.6 | 83.6 | 80.0 | 80.0 | 80.0
Adding noise | 83.2 | 81.5 | 75.5 | 79.3 | 74.5 | 21.6
Reduction | -0.4 | -2.1 | -8.1 | -0.7 | -5.5 | -58.4

###  C.3 Freezing BatchNorm Stats

In MLC task, the backbone is initialized with weights pretrained on ImageNet
dataset. Current methods usually set the BatchNorm (BN) layers into train mode
and update the running mean and standard deviation values with MLC datasets.
However, we find that this strategy may not be the optimal in our method. As
summarized in Table 10, we train our model with and without freezing the stats
in BNs, and the results show that, when the known label ratio is high (100%),
the traditional not freezing BN obtains same mAP, while on smaller ratios
especially 10%, freezing BN achieves significant improvements. This implies
that, when with smaller known label ratios, the model is more sensitive and
updating BN stats with the training dataset may make this situation severer,
while fixing the BN stats (set to eval mode) could result in stabler
optimization. As a result, we fix the BN stats in all the experiments of our
PU-MLC.

Table 10: Comparisons of mAPs of freezing and not freezing BatchNorm stats on MS-COCO. Methods | 10% | 50% | 100%
---|---|---|---
Not freezing BN | 74.2 | 81.3 | 83.6
Freezing BN | 75.5 | 81.5 | 83.6

###  C.4 Ablation studies on hyper-parameters $\lambda$ and $\alpha$.

We conduct an analysis on the Pascal VOC dataset to examine the influence of
hyper-parameters, specifically $\lambda$ and $\alpha$, on PU-MLC. Figure 5(a)
demonstrates that our method shows insensitivity to changes in $\lambda$, with
the mAP fluctuating between 87.5% and 87.9%. Therefore, we opt to select the
$\lambda$ value of 1.0 that yields the best performance as our experimental
setting. Additionally, Figure 5(b) illustrates that both large and small
$\alpha$ values have a negative impact on the model. Particularly, when
$\alpha$ exceeds 5.5, the mAP decreases by 1%. Consequently, we chose the
$\alpha$ value of 2.0 as our experimental setting, as it yields the best
performance.

Figure 5: Ablation on hyper-parameters $\lambda$ and $\alpha$. We report the
mAP (%) of different $\lambda$ and $\alpha$ at the 10% known label ratio.

###  C.5 Ablation studies on exponential $\gamma$ selection.

In Section 4.4, we elaborate on the importance of dynamically selecting the
$\gamma$ for each known label ratio. Figure 6 visualizes the process of
$\gamma$ selection at 10% and 50% known label ratios on Pascal VOC. It is
crucial to choose an appropriate $\gamma$ to ensure the re-balance factor
adequately balances the contributions of positive and unknown labels during
training. On the contrary, improper $\gamma$ leads to suboptimal performance
of the re-balance factor. To address this, we carefully choose the $\gamma$
that yields the best performance at each known label ratio.

Figure 6: Ablation studies on exponential $\gamma$ selection. We report the
mAP (%) of different $\gamma$ on VOC 2007.

###  C.6 Compare with CSL

Due to a distinct strategy employ by CSL [2] in making partial label datasets
and its utilization of a unique backbone, a fair comparison between CSL and
other methods in Table 1 is not feasible. We conduct experiments to use the
same training settings in our method. Concretely, we employ the fixed per
class (FPC) of 1000 strategy to generate a partial label dataset and use the
TResNet-M [34] as the backbone. As shown in Table 11, our PU-MLC obtains 85.4%
mAP, significantly outperforming CSL.

Table 11: Compared with CSL on MS-COCO. Methods | FPC=1000
---|---
CSL [2] | 83.4
PU-MLC | 85.4