# Exploring Structured Semantic Prior
for Multi Label Recognition with Incomplete Labels

Zixuan Ding1,4 Ao Wang2,3,411footnotemark: 1 Hui Chen2,3,† Qiang Zhang1
Pengzhang Liu5 Yongjun Bao5 Weipeng Yan5 Jungong Han6,7
1Xidian University 2Tsinghua University 3BNRist
4Hangzhou Zhuoxi Institute of Brain and Intelligence 5JD.com
6Department of Computer Science, the University of Sheffield, UK
7Centre for Machine Intelligence, the University of Sheffield, UK
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>{Paul.yan, baoyongjun,
<EMAIL_ADDRESS>Equal contributions. $\dagger$ Corresponding author.

###### Abstract

Multi-label recognition (MLR) with incomplete labels is very challenging.
Recent works strive to explore the image-to-label correspondence in the
vision-language model, _i.e_., CLIP [22], to compensate for insufficient
annotations. In spite of promising performance, they generally overlook the
valuable prior about the label-to-label correspondence. In this paper, we
advocate remedying the deficiency of label supervision for the MLR with
incomplete labels by deriving a structured semantic prior about the label-to-
label correspondence via a semantic prior prompter. We then present a novel
Semantic Correspondence Prompt Network (SCPNet), which can thoroughly explore
the structured semantic prior. A Prior-Enhanced Self-Supervised Learning
method is further introduced to enhance the use of the prior. Comprehensive
experiments and analyses on several widely used benchmark datasets show that
our method significantly outperforms existing methods on all datasets, well
demonstrating the effectiveness and the superiority of our method. Our code
will be available at https://github.com/jameslahm/SCPNet.

##  1 Introduction

Figure 1: Overview of CNN-based, DualCoOp [26] and our SCPNet. Like DualCoOp,
our SCPNet adopts CLIP as the base model. Differently, our SCPNet aims to
enhance the MLR with the prior about the label-to-label correspondence. MC
means multi-class. CL denotes contrastive learning.

Multi-label recognition (MLR) aims to describe the image content with various
semantic labels [30, 29, 5, 26]. It encodes the visual information into
structured labels, which can benefit the index and fast retrieval of images in
broad practical applications, such as the search engine [24, 27] and the
recommendation system [2, 33].

Benefited from the development of deep learning, MLR has achieved remarkable
progress in recent years. However, collecting high-quality full annotations
becomes very challenging when the label set scales up, which greatly hinders
the wide usage of MLR in real scenarios. Recently, researchers explore more
feasible solutions for MLR. For example, the full label setting is relaxed
with a partial label setting in [21, 3], which merely annotates a few labels
for each training image. One more extreme setting with solely one single
positive label is tackled in [16, 8]. These settings can be unified into a
common issue of incomplete labels, which relieves the burden of the full
annotation and considerably reduces the annotation cost. Therefore, it draws
increasing attention from both academia and industry.

Compared with the full label setting, the incomplete label setting encounters
a dilemma of poor supervision, resulting in severe performance drops for MLR.
Existing methods strive to regain supervision from missing labels by
exhaustively exploring the image-to-label correspondence via semantic-aware
modules [4, 21] or loss calibration methods [8, 32, 16]. A convolutional
neural network (CNN) pretrained on the ImageNet is usually leveraged to
construct the MLR model. Its multi-class softmax layer is often replaced by a
multi-label sigmoid layer (Fig. 1 (a)). Such a replacement wipes out prior
knowledge about the correspondence between images and labels although it is
necessary and inevitable.

Recently, vision-language pretrained models have obtained remarkable success
in various vision tasks [35, 34, 26]. Thanks to their large-scale pretraining,
the vision-language model, _e.g_., CLIP [22], which is trained with 400
million image-text pairs, can well bridge the visual-textual gap [26],
providing rich prior knowledge for the downstream tasks. For the MLR task, Sun
_et al_. [26] propose a DualCoOp method, which is the first work to employ the
CLIP as the MLR base model. Through dual prompts, DualCoOp directly adopts the
text encoder in the CLIP as the multi-label classification head (Fig. 1 (b)),
without abandoning the visual-textual prior in the pretrained CLIP.

Despite its effectiveness, DualCoOp is still limited in remedying the
deficiency of label supervision, which is desired for the MLR with incomplete
labels. Intuitively, it is convenient to reason unknown labels from annotated
labels by leveraging the correspondence among labels, _e.g_., tables are
likely to appear with chairs, and cars are usually accompanied by roads.
Therefore, such a label-to-label correspondence can help survive more label
supervision and thus benefit MLR with incomplete labels. Besides, although
most vision-language models do not encourage the contrastive learning among
texts, they are still abundant in the knowledge about the label-to-label
correspondence because of the large-scale cross-modality training. However,
such a valuable prior is rarely explored in the existing state-of-the-art
method, _i.e_., DualCoOp [26].

In this paper, we aim to mitigate such deficiency of label supervision for MLR
with incomplete labels by leveraging the abundant prior about the label-to-
label correspondence in the CLIP [22]. We present a structured prior prompter
to conveniently derive a structured semantic prior from the CLIP. Then we
propose a novel Semantic Correspondence Prompt network (SCPNet) (Fig. 1 (c)),
which can prompt the structured label-to-label correspondence with a cross-
modality prompter. Our SCPNet also equips a semantic association module to
explore high-order relationships among labels with the guidance of the derived
structured semantic prior. A prior-enhanced self-supervised learning method is
further introduced to comprehensively investigate the valuable prior. As a
result, our method can neatly calibrate its predicted semantic distribution
while maintaining the self-consistency.

To verify the effectiveness of the proposed method for MLR with incomplete
labels, we conduct extensive experiments and analyses on a series of widely
used benchmark datasets, _i.e_., MS COCO[19], PASCAL VOC[11], NUS Wide[7],
CUB[28] and OpenImages[17]. Experimental results show that our method can
significantly outperform state-of-the-art methods on all datasets with a
maximal improvement of $6.8\%/3.4\%$ mAP for the single positive label setting
and the partial label setting, respectively, well demonstrating its
effectiveness and superiority.

Overall, our contributions are four folds.

  * •

We advocate leveraging a structured semantic prior to deal with the deficiency
of label supervision for MLR with incomplete labels. To this end, we extract
such a prior via a structured prior prompter.

  * •

We present a semantic correspondence prompt Network (SCPNet) based on a cross-
modality prompter and a semantic association module. The SCPNet can adequately
explore the structured prior knowledge, thus boosting MLR with incomplete
labels.

  * •

We design a prior-enhanced self-supervised learning method to further
investigate such a structured semantic prior, which can enjoy both
distribution refinement and self-consistency.

  * •

Experimental results show that our method can consistently achieve state-of-
the-art performance on all benchmark datasets, revealing the significant
effectiveness. Thorough analyses also demonstrate the superiority of our
method.

Figure 2: An overview of the proposed method. We design a semantic
correspondence prompt network to explore the structured semantic prior for MlR
with incomplete labels. A prior-enhanced self-supervised learning strategy is
used to enhance such exploration.

##  2 Related work

Multi-label recognition with full annotations. Multi-label Recognition has
long been a hot topic in the computer vision field [30, 1, 21]. A generic
method is to learn multiple binary classifiers [8, 16], which usually takes no
consideration of the label correlation. Recently, the label-to-label
correspondence is established through graph neural networks or transformer
structures [6, 29]. These methods heavily rely on the quality of label
supervision. However, collecting a large-scale dataset with complete labels is
challenging and expensive. In real scenarios, researchers explore much more
practical settings with incomplete labels, _i.e_., MLR with partial labels and
MLR with a single positive label.

Multi-label recognition with incomplete labels. In the partial label setting,
only a few labels need to be annotated for each training image. Durand _et
al_. [9] adopt a curriculum learning based model to predict the missing labels
during the training procedure. Pu _et al_. [21] and Chen _et al_. [4] transfer
predictions of neighboring images via image-image correlation. However, their
performance is not guaranteed in more severe scenarios, _i.e_., single
positive label setting, in which each image is provided with solely one
positive annotation. To tackle the issue of the single positive label, Cole
_et al_. [8] propose a regularized online loss via a joint optimization of
label estimator and image classifier. Zhang _et al_. [32] adopt a label
correction process for the probability exceeding a fixed threshold. Kim _et
al_. [16] propose to reject or correct the large loss samples during training,
which can prevent over-fitting false negative labels. However, different from
our solution, they usually independently calibrate the importance of different
labels [32, 16, 8], taking no consideration of the semantic correspondence
among labels.

Vision-language models in downstream visual tasks. Radford _et al_. [22]
exploit the contrastive learning with large-scale image-text pairs, _i.e_.,
about 400 million pairs, ending up with a powerful vision-language model,
_i.e_., CLIP. Such a model shows remarkable generalization capability in
downstream visual tasks [22]. Therefore, researchers exhaustively explore how
to leverage the abundant vision-language correspondence [23, 12, 14, 26]. Sun
_et al_. [26] also employ CLIP for MLR. They present dual prompts, _i.e_., a
positive prompt and a negative one, to explore the rich image-to-label
correspondence in CLIP. However, different from our motivation, they overlook
the rich label-to-label correspondence in CLIP.

##  3 Methodology

###  3.1 Structured Prior Prompter

For MLR with full annotations, existing methods can achieve fruitful outcomes
by exploring the semantic correspondence between images and labels [6].
However, they require abundant label supervision to obtain accurate label co-
occurrence information for the estimation of label relationships. Therefore,
in MLR with incomplete labels, the scarce label supervision greatly hinders
their capability to explore the semantic correspondence. Benefited from the
development of large-scale pretrained embeddings, _e.g_., Glove [20], or
models, _e.g_., BERT [15] and CLIP [22], we can easily obtain contextual
representations for labels, which can be directly used to derive such a label-
to-label correspondence. Such a annotation-free strategy is notably appealing
when no adequate label supervision is provided. Furthermore, the abundant
correspondence prior in the pretrained model can help associate the annotated
label with unknown labels, which promisingly alleviates the deficiency of
label supervision. Hence, we introduce a structured prior prompter to explore
such a label-to-label correspondence in the pretrained model. Considering the
popularity and the remarkable performance in the computer vision community, we
choose the vision-language model, _i.e_., CLIP [22], as the target.

Specifically, in the proposed structured prior prompter, for a set of to-be-
explored labels $Y=\\{y_{0},y_{1},...,y_{n}\\}$, we derive the label feature
by feeding a prompt template, _i.e_., a photo of a [CLS], into the text
encoder of CLIP. We denote the label feature as $\bm{\bar{z}_{i}}$ for each
$y_{i}$. Then the correlation prior among labels, denoted as
$\bm{A}=(a_{ij})_{n\times n}$, can be derived as:

$a_{ij}=\text{sim}(\bm{\bar{z}_{i}},\bm{\bar{z}_{j}})$ (1)

where $\text{sim}(\cdot,\cdot)$ is the cosine similarity.

For each entry $\bm{a}_{i}$, we select the top $K$ elements and set the rest
to zero, ending up with a sparse matrix,
$\bm{A^{\prime}}=(a^{\prime}_{ij})_{n\times n}$:

$a^{\prime}_{ij}=\left\\{\begin{array}[]{l}a_{ij},\quad{\rm{if}}\
j\in\text{topK}(\bm{a}_{i})\\\ 0,\quad\ \ \,{\rm{if}}\
j\notin\text{topK}(\bm{a}_{i})\end{array}\right.$ (2)

Following [6], we mitigate the over-smoothness of graph representation by
adjusting the sparse graph $\bm{A^{\prime}}$ as follows:

$\bar{a}_{ij}=\left\\{\begin{array}[]{l}(s/\sum\nolimits_{i\neq
j^{\prime}}^{n}a^{\prime}_{ij^{\prime}})\times a^{\prime}_{ij},\quad{\rm{if}}\
i\neq j\\\ 1-s,\qquad\qquad\qquad\quad\ \ \,{\rm{if}}\ i=j\end{array}\right.$
(3)

where $s$ is a hyper-parameter which determines weights assigned to a node
itself and its neighboring nodes. The label correspondence graph $\mathcal{G}$
can be derived as:

$\bm{a}_{ij}^{*}=\frac{\text{I}[\bar{a}_{ij}\neq
0]\exp(\bar{a}_{ij}/\tau^{{}^{\prime}})}{\sum\nolimits_{j}\text{I}[\bar{a}_{ij}\neq
0]\exp(\bar{a}_{ij}/\tau^{{}^{\prime}})}$ (4)

where $\tau^{{}^{\prime}}$ controls the distribution smoothness and
$\text{I}[\cdot]$ is an indicator function. We denote the the adjacency matrix
of $\mathcal{G}$ as $\bm{A^{*}}=(a^{*}_{ij})_{n\times n}$.

We see that $\bm{A^{*}}$ emphasizes the importance of the node itself and
weights other nodes according to their relationships (see Eq. 1). Therefore,
the fruitful label correspondence can be encoded in such a structured graph,
_i.e_., $\bm{A^{*}}$, providing rich structured semantic prior for MLR models.

###  3.2 Semantic Correspondence Prompt Network

As shown in Fig. 2, the SCPNet consists of a cross-modality prompter and a
semantic association module.

Cross-modality prompter (CMP). Previous works [16, 4, 21] usually employ a
convolutional neural network pretrained on ImageNet, _e.g_., ResNet50. During
fine-tuning in the downstream MLR tasks, the prior knowledge about the image-
to-label correspondence is generally discarded due to the semantic shift,
_i.e_., different label sets between the ImageNet and the MLR benchmark
datasets. Differently, we aim to take full use of such an image-label prior
during model optimization. Similar to [26], we resolve the problem of semantic
shift by a cross-modality prompter, based on a vision-language model, _i.e_.,
CLIP [22].

Formally, following [35], given a label set, _i.e_.,
$Y=\\{y_{0},y_{1},...,y_{n}\\}$, we introduce $m$ soft prompt tokens to
extract its representation. For ease of explanation, we denote the prompt as
$\bm{t}_{i}=\\{\bm{v}_{1},\bm{v}_{2},...,\bm{v}_{m},\bm{e}_{i}\\}$, where
$\bm{v}$ with a subscript denotes a soft prompt token and $\bm{e}_{i}$ is the
embedding of $y_{i}$. The label feature of $y_{i}$, denoted as $\bm{z_{i}}$,
can be derived by the text encoder of CLIP. For an input image $\bm{x}$, its
visual representation, denoted as $\bm{f}$, is extracted by the image encoder
of CLIP. The process of feature extraction can be computed as follows:

${\bm{f}}={F}(\bm{x}),{\bm{z_{i}}}={G}(\bm{t_{i}}),$ (5)

where ${F(\cdot)}$ and ${G(\cdot)}$ denote the image encoder and the text
encoder in CLIP, respectively.

Semantic association module (SAM). As CMP still lacks capturing the label-to-
label correspondence, we further equip a semantic association module to
capture high-order relationships among labels. Specifically, with guidance of
the structured semantic prior $\bm{A}^{*}$ (see Eq. 4), we utilize $L$ graph
convolutional network (GCN) layers to progressively refine the input features
$\bm{H}^{0}=\bm{Z}$, where $\bm{Z}=\\{\bm{z}_{0},\bm{z}_{1},...,\bm{z}_{n}\\}$
is a combination of features for $Y$ as in Eq. 5. The $l$-th GCN layer is
updated as follows:

${\bm{H}^{l+1}}=\rho({{\bm{A}}^{*}}{\bm{H}^{l}}{\bm{W}^{l}}),$ (6)

where $\bm{W}$ with a superscript is a learnable parameter matrix and $\rho$
is a non-linear function. $l\in[0,L)$. The final refined label representations
can be obtained through a residual connection, _i.e_.,
$\bm{Z}^{*}=\bm{H}^{0}+\bm{H}^{L}$. The likelihood $p(y_{i}|\bm{x})$ can be
computed as:

$p(y_{i}|\bm{x})=\sigma(\text{sim}(\bm{f},\bm{z_{i}}^{*})/\tau),$ (7)

where ${\bm{z_{i}}}^{*}$ denotes the refined feature for label $y_{i}$.

Benefited from the GCN, the structured label-to-label correspondence in CLIP,
which is represented by $\bm{A}^{*}$, can be progressively refined in the
label representation. Therefore, during the semantic matching between the
image feature and the label feature, _i.e_., Eq. 7, labels with high
correlations will obtain similar likelihoods, enabling a subtle semantic
association.

###  3.3 Prior-Enhanced Self-Supervised Learning

The proposed prior-enhanced self-supervised learning strategy, dubbed PESSL,
aims to make full use of the structured semantic correspondence prior. We
endow the proposed PESSL with a self-supervised consistency loss and a self-
distillation objective that is boosted by a structure-aware semantic
calibration strategy.

Structure-aware semantic calibration. Intuitively, if two labels are
semantically correlated, they may be observed in one image. For MLR, such a
correspondence can help decide potential semantic labels for an input image,
given its predictions. Therefore, we formulate the likelihood of
$p(y_{i}|\bm{x})$ as a weighted combination of likelihoods for correlated
neighboring labels of $y_{i}$:

$p^{*}(y_{i}|\bm{x})=\sum\nolimits_{y_{j}\in\mathcal{N}(y_{i})}w(i,j)\times
p(y_{j}|\bm{x})$ (8)

Here, $w(i,j)$ is a correlation weight indicating the relationship between
$y_{i}$ and $y_{j}$. $\mathcal{N}(y_{i})$ denotes a correlated neighboring set
of labels corresponding to $y_{i}$.

For ease of explanation, we introduce a correlation matrix $\mathbb{W}$ to
represent the whole correlation among labels, _i.e_.,
$\mathbb{W}=(w(i,j))_{n\times n}$. We then customize the whole process as a
function parameterized by $\mathbb{W}\in R^{n\times n}$ and the distribution
over $Y$ given the input $\bm{x}$, _i.e_., $\bm{p}(Y|\bm{x})\in R^{n\times
1}$:

$\text{SASC}(\bm{p}(Y|\bm{x}),\mathbb{W})=\mathbb{W}\bm{p}(Y|\bm{x})$ (9)

Prior-enhanced learning. Existing loss correction methods individually
reweight each label, without taking into consideration the correspondence
among labels. Here, we propose to follow the self-supervised learning
principle [31, 25] and introduce a self-distillation learning strategy to
benefit the MLR model from the structured semantic correspondence among
labels.

Specifically, we derive two different versions for the input image $\bm{x}$
with one weak transformation $\omega(\cdot)$ and one strong transformation
$\Omega(\cdot)$, respectively. Their corresponding semantic distributions,
denoted as $\bm{p}(y|\omega(\bm{x}))$ and $\bm{p}(y|\Omega(\bm{x}))$,
respectively, can be derived by Eq. 7. Then we use a consistency loss to
encourage them to be consistent. Different from [31], which simply regularizes
the model with the most confident label, we construct a set of confident
labels $\mathcal{O}(\bm{x})$ with the top highest probability larger than a
threshold $\mathcal{T}$ in $\bm{p}(y|\omega(\bm{x}))$, _i.e_.,
$\mathcal{O}(\bm{x})=\\{c|c\in\text{topK}(\bm{p}(y|\omega(\bm{x})))\wedge
p(c|\omega(\bm{x}))>\mathcal{T}(c)\\}$. A dynamic threshold strategy is
performed for each label, as [31]. The consistency loss is then derived by:

$\displaystyle\mathcal{L}_{cst}=$
$\displaystyle-\sum\nolimits_{c\in\mathcal{O}(\bm{x})}^{Y}\log
p(c|\Omega(\bm{x}))$ (10)
$\displaystyle-\sum\nolimits_{c\notin\mathcal{O}(\bm{x})}^{Y}\log(1-p(c|\Omega(\bm{x})))$

We calibrate the distribution of the weak-transformed image, _i.e_.,
$\bm{p}(y|\omega(\bm{x}))$, by using the SASC function (see Eq. 9):

$\bm{p^{*}}(y|\omega(\bm{x}))=\text{SASC}(\bm{p}(y|\omega(\bm{x})),\bm{A}^{*})$
(11)

where $\bm{A}^{*}$ represents the structured semantic prior, derived by Eq. 4.
Considering that compared with the weak-transformed image, the strong-
transformed image is usually more difficult to learn. Therefore, we employ a
self-distillation objective to optimize the distribution of the strong-
transformed image $\Omega(\bm{x})$ with the guidance of the calibrated
semantic distribution via the KL-divergence:

$\mathcal{L}_{dstl}=-\sum\limits_{c}^{Y}\left(q^{w}_{c}\log\frac{q_{c}^{s}}{q^{w}_{c}}+{{(1-q^{w}_{c})}\log\frac{1-{q^{s}_{c}}}{1-q^{w}_{c}}}\right)$
(12)

where $q^{w}_{c}=p^{*}(c|\omega(\bm{x}))$ and $q^{s}_{c}=p(c|\Omega(\bm{x}))$.

Overall Objective. Finally, we formulate the prior-enhanced self-supervised
learning as a combination of the consistency objective and the self-
distillation objective:

$\mathcal{L}_{pessl}=\lambda_{cst}\mathcal{L}_{cst}+\lambda_{dstl}\mathcal{L}_{dstl}$
(13)

###  3.4 Network Optimization

During training, we adopt a multi-label classification objective over the
predicted likelihood, _i.e_., $p(y_{i}|\bm{x})$ in Eq. 7, to optimize our
SCPNet, denoted as $\mathcal{L}_{cls}$. We follow [32] to design
$\mathcal{L}_{cls}$. The overall objective for the network optimization is
formulated as follows:

$\mathcal{L}=\mathcal{L}_{cls}+\mathcal{L}_{pessl}$ (14)

Table 1: Comparison with the state-of-the-art methods for MLR with the single positive label (%). Method | LargeLoss setup [16] | SPLC setup [32]
---|---|---
COCO | VOC | NUS | CUB | Avg. | COCO | VOC | NUS | CUB | Avg.
LSAN[8] | 69.2 | 86.7 | 50.5 | 17.9 | 56.1 | 70.5 | 87.2 | 52.5 | 18.9 | 57.3
ROLE[8] | 69.0 | 88.2 | 51.0 | 16.8 | 56.3 | 70.9 | 89.0 | 50.6 | 20.4 | 57.7
LargeLoss[16] | 71.6 | 89.3 | 49.6 | 21.8 | 58.1 | - | - | - | - | -
Hill[32] | - | - | - | - | - | 73.2 | 87.8 | 55.0 | 18.8 | 58.7
SPLC[32] | 72.0 | 87.7 | 49.8 | 18.0 | 56.9 | 73.2 | 88.1 | 55.2 | 20.0 | 59.1
SCPNet (ours) | 75.4 | 90.1 | 55.7 | 25.4 | 61.7 | 76.4 | 91.2 | 62.0 | 25.7 | 63.8
Table 2: Comparison with the state-of-the-art methods for MLR with partial labels (%). Datasets | Method | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | Avg.
---|---|---|---|---|---|---|---|---|---|---|---
COCO | SSGRL[5] | 62.5 | 70.5 | 73.2 | 74.5 | 76.3 | 76.5 | 77.1 | 77.9 | 78.4 | 74.1
GCN-ML[6] | 63.8 | 70.9 | 72.8 | 74.0 | 76.7 | 77.1 | 77.3 | 78.3 | 78.6 | 74.4
SST[4] | 68.1 | 73.5 | 75.9 | 77.3 | 78.1 | 78.9 | 79.2 | 79.6 | 79.9 | 76.7
SARB[21] | 71.2 | 75.0 | 77.1 | 78.3 | 78.9 | 79.6 | 79.8 | 80.5 | 80.5 | 77.9
DualCoOp[26] | 78.7 | 80.9 | 81.7 | 82.0 | 82.5 | 82.7 | 82.8 | 83.0 | 83.1 | 81.9
SCPNet (ours)* | 80.3 | 82.2 | 82.8 | 83.4 | 83.8 | 83.9 | 84.0 | 84.1 | 84.2 | 83.2
SCPNet (ours) | 79.1 | 82.1 | 82.8 | 83.9 | 84.5 | 84.9 | 85.4 | 85.7 | 85.9 | 83.8
VOC2007 | SSGRL[5] | 77.7 | 87.6 | 89.9 | 90.7 | 91.4 | 91.8 | 91.9 | 92.2 | 92.2 | 89.5
GCN-ML[6] | 74.5 | 87.4 | 89.7 | 90.7 | 91.0 | 91.3 | 91.5 | 91.8 | 92.0 | 88.9
SST[4] | 81.5 | 89.0 | 90.3 | 91.0 | 91.6 | 92.0 | 92.5 | 92.6 | 92.7 | 90.4
SARB[21] | 83.5 | 88.6 | 90.7 | 91.4 | 91.9 | 92.2 | 92.6 | 92.8 | 92.9 | 90.7
DualCoOp[26] | 90.3 | 92.2 | 92.8 | 93.3 | 93.6 | 93.9 | 94.0 | 94.1 | 94.2 | 93.2
SCPNet (ours) | 91.1 | 92.8 | 93.5 | 93.6 | 93.8 | 94.0 | 94.1 | 94.2 | 94.3 | 93.5
VG-200 | SSGRL[5] | 34.6 | 37.3 | 39.2 | 40.1 | 40.4 | 41.0 | 41.3 | 41.6 | 42.1 | 39.7
GCN-ML[6] | 32.0 | 37.8 | 38.8 | 39.1 | 39.6 | 40.0 | 41.9 | 42.3 | 42.5 | 39.3
SST[4] | 38.8 | 39.4 | 41.1 | 41.8 | 42.7 | 42.9 | 43.0 | 43.2 | 43.5 | 41.8
SARB[21] | 41.4 | 44.0 | 44.8 | 45.5 | 46.6 | 47.5 | 47.8 | 48.0 | 48.2 | 46.0
SCPNet (ours) | 43.8 | 46.4 | 48.2 | 49.6 | 50.4 | 50.9 | 51.3 | 51.6 | 52.0 | 49.4

##  4 Experiment

###  4.1 Experiment Settings

Datasets. We conduct extensive experiments on several standard benchmarks for
MLR with incomplete labels, including the single positive label setting and
the partial label setting. For the single positive label setting, following
[16, 32], we use MS-COCO (COCO) [19], PASCAL VOC (VOC) [11], NUSWIDE (NUS)
[7], and CUB [28]. For the partial label learning, we adopt MS-COCO (COCO)
[19], PASCAL VOC 2007 (VOC2007) [10] and Visual Genome (VG-200) [18], as [4,
21]. We leave details of benchmark datasets in the supplementary due to the
space limit.

Implementation details. We leverage published CLIP
weights111https://github.com/openai/CLIP to initialize MLR models. To fairly
compare the proposed method with others, we adopt the ResNet50-based CLIP and
the Resnet101-based CLIP for the single positive label and the partial label,
respectively. During training, we tune the image encoder and fix the text
encoder of CLIP. More details are provided in the supplementary.

Evaluation. By default, we employ the mean average precision (mAP) as the
evaluation metric, following previous works [5, 32, 16]. For the single
positive label setting, we perform two different setups, _i.e_., the LargeLoss
setup [16] and the SPLC setup [32], which are common in the community. We
leave the details in the supplementary due to the space limit. For the partial
label setting, following [21], we randomly maintain partial labels for the
training set with a ratio ranging from 10$\%$ to 90$\%$. Apart from
performance on all ratios, we also report the average result.

Table 3: Effect of different modules in the proposed SCPNet method for both the single positive label setting and the partial label setting (%). An average of all metrics is also reported. Model | CMP | SAM | PESSL | Single Positive Label | Partial Label | Avg.
---|---|---|---|---|---|---
$\mathcal{L}_{cst}$ | $\mathcal{L}_{dstl}$ | COCO | VOC | NUS | CUB | COCO | VOC2007 | VG-200
Baseline |  |  |  |  | 73.18 | 88.07 | 55.18 | 19.99 | 77.41 | 88.32 | 46.39 | 64.08
SCPNet | ✓ |  |  |  | 74.36 | 88.46 | 60.66 | 21.42 | 80.90 | 89.16 | 47.55 | 66.07
✓ | ✓ |  |  | 75.12 | 89.09 | 61.08 | 21.66 | 82.12 | 90.16 | 48.11 | 66.76
✓ | ✓ | ✓ |  | 75.70 | 90.92 | 61.75 | 23.67 | 82.85 | 92.50 | 48.70 | 68.01
✓ | ✓ |  | ✓ | 75.84 | 90.92 | 61.56 | 24.51 | 83.35 | 93.21 | 48.83 | 68.32
✓ | ✓ | ✓ | ✓ | 76.42 | 91.16 | 62.04 | 25.71 | 83.76 | 93.49 | 49.36 | 68.85
Table 4: Analysis on the correlation graph. SAM | PESSL | mAP (%)
---|---|---
Static | Static | 76.42
Dynamic | 76.05
No | 75.83
Dynamic | Static | 76.08
Dynamic | 75.84

###  4.2 Comparisons with State-of-the-Arts

MLR with single positive labels. We report the model performance on both the
LargeLoss setup [16] and the SPLC setup [32]. To better reveal the
effectiveness of the proposed method, we also report the average performance
for both setups. As shown in Tab. 1, for both setups, our method can
significantly outperform existing methods on all benchmark datasets, achieving
state-of-the-art performance. Specifically, in the LargeLoss setup, the
proposed SCPNet can obtain a maximal performance improvement of $4.7\%$ (NUS).
As a whole, our method can accomplish an overall performance improvement of
$3.6\%$. In the SPLC setup, the maximal performance improvement achieved by
our method can reach $6.8\%$ (NUS). As a result, our method can accomplish
$4.7\%$ improvement on average.

MLR with partial labels. As shown in Tab. 2, our results also consistently
surpass existing state-of-the-art methods on all benchmark datasets,
especially on the COCO and VG-200. Compared with DualCoOp [26] which also
leverages CLIP to build MLR models, the proposed method can obtain an
improvement of $1.9\%$ mAP on the MS COCO. With a frozen image encoder during
training as DualCoOp, our method, denoted as SCPNet (ours)*, still enjoys
superior performance to DualCoOp. On the VOC2007, our method obtains
comparable performance with $0.3\%$ improvement. However, under small ratios,
our method shows its superiority to DualCoOp, _e.g_., $0.8\%$ improvement with
a ratio of $10\%$. On the VG-200, compared with SARB [21] which enhances the
MLR models with a structure-aware algorithm, our SCPNet can significantly
outperform it with an average performance improvement of $3.4\%$.

These experimental results show that our method can consistently obtain
superior performance in different setups for MLR with incomplete labels, well
demonstrating the effectiveness. To verify the generalization of the proposed
method, we also investigate the effectiveness in the few-shot partial label
setting and the real partial label scenario. We leave them in the
supplementary due to the space limit.

###  4.3 Ablation Study

In order to analyze the effectiveness of each component, we conduct the
ablation study on both the single positive label and the partial label
settings. All results are shown in Tab. 3. We also introduce a model that
directly employs $\mathcal{L}_{cls}$ to optimize a ResNet-based MLR model, as
the baseline. As shown in Tab. 3, each component can obtain consistent
performance improvement in all datasets. Specifically, compared with the
baseline model, our CMP can obtain an average performance of $1.99\%$ mAP,
indicating the superiority of prompting a cross-modality vision-language
model. Augmented by SAM, our method can bring $0.69\%$ mAP improvement. Such
improvements can be attributed to the explicit semantic correspondence among
labels captured by the proposed SAM component. Besides, the consistency
learning, _i.e_., $\mathcal{L}_{cst}$, and the self-distillation objective,
_i.e_., $\mathcal{L}_{dstl}$, can lead to $1.25\%$ and $1.56\%$ performance
improvement, respectively. The overall improvement for the proposed PESSL can
reach $2.09\%$, well demonstrating the strength of incorporating the
structured semantic prior during model optimization. Finally, our proposed
SCPNet can significantly outperform the baseline model with $4.77\%$ mAP
improvement on average, well demonstrating the effectiveness and the
superiority of the proposed method.

Table 5: Analysis on the prior extraction (%). Prior | Dynamic | Image | Glove | BERT | CLIP
---|---|---|---|---|---
mAP | 75.84 | 75.67 | 76.15 | 76.16 | 76.42

###  4.4 Model Analysis

Here, we perform comprehensive inspections for the proposed method. All
experiments are conducted in the single positive label setting on the MS COCO
dataset, by default. Due to the space limit, we provide more analyses in the
supplementary material.

Figure 3: The structured semantic prior (left) and the learnt label
representation (middle: in the baseline, right: in our SCPNet).

Correlation graph construction. We verify the positive effect of the prior
used in the correlation graph construction for both SAM and PESSL. To achieve
this goal, we discuss two kinds of correlation graph: 1) a static one derived
from the pretrained CLIP model (see Eq. 4), which captures the structured
semantic prior, and 2) a dynamic one achieved by the learnable CMP, _i.e_.,
constructing the adjacency matrix with label features ${\bm{z_{i}}}$ computed
by Eq. 5. We also report PESSL without the prior, denoted as “No”. As
illustrated in Tab. 4, we can observe that our method can obtain the optimal
performance by using the static correlation graph for both SAM and PESSL.
Besides, the static graph can substantially achieve better results than the
dynamic one in both components, revealing the advantage of the structured
semantic prior. We claim that in the MLR with incomplete labels, the challenge
of insufficient label supervision makes the dynamic graph sub-optimal, thus
inferior to the static one. By comparing PESSL with the prior (Row 2) and the
one without the prior (Row 4), we can find that the latter achieves inferior
performance, which can demonstrate the benefit of the proposed prior, _i.e_.,
$\bm{A}^{*}$ in Eq. 4.

Prior knowledge extraction. We further investigate the advantage of the
proposed structured semantic prior extracted by CLIP with three other types of
prior knowledge as competitors. For a given label, 1) “Image” averages all
image features corresponding to it; 2) “Glove” represents its feature by
pretrained Glove word embeddings [20]; and 3) “BERT” extracts the label
feature by the prompt learning as ours. We also report the result of dynamic
label-to-label correspondence as the baseline. As shown in Tab. 5, compared
with Dynamic, except Image, Glove, BERT and CLIP show consistent advantage
because of their superior ability to capture the label-to-label
correspondence. Besides, our CLIP achieves the best performance, which reveals
that the CLIP-based structured prior is more matched with the CLIP-based MLR
model due to their consistent knowledge.

Table 6: Prior for MLR models with the ImageNet-based ResNet. Image Encoder | Label Encoder | mAP (%)
---|---|---
ResNet | sigmoid | 73.18
ResNet | Ours | 74.72
Ours | Ours | 76.42

Generalization on the CNN-based architecture. To show the generalization of
the proposed prior-enhanced method, we transfer our design principles to a
vanilla ResNet-based MLR model with a sigmoid layer as the label encoder. We
analyze the impact of replacing the sigmoid layer with ours. As shown in Tab.
6, such modification can result in a performance gain of $1.54\%$,
demonstrating the good generalization ability of the proposed method in the
CNN-based architecture.

Analysis on hyper-parameters. As shown in Tab. 7 and Tab. 8, the best value of
$L$, $\lambda_{cst}$ and $\lambda_{dstl}$ is at $L=3$, $\lambda_{cst}=1/8$,
and $\lambda_{dstl}=1/8$, respectively. More analyses can be found in the
supplementary material.

Table 7: Analysis on the number of GCN Layer, _i.e_., $L$ (%). $L$ | 2 | 3 | 4
---|---|---|---
mAP | 75.88 | 76.42 | 76.22
Table 8: Analysis on $\lambda_{cst}$ and $\lambda_{dstl}$ (%). $\lambda_{cst}$ | 0 | 1/16 | 1/8 | 1/4 | 1/8
---|---|---|---|---|---
$\lambda_{dstl}$ | 0 | 1/8 | 2/8 | 3/8
mAP | 75.12 | 75.56 | 75.70 | 75.13 | 76.42 | 76.40 | 76.17

###  4.5 More Insightful Analysis

To provide more insights about the effectiveness of the proposed method, we
conduct visualization analyses on the structured semantic prior and the label
representation in the latent feature space. First, we present the structured
semantic prior about the label-to-label correspondence by visualizing the
adjacency matrix, _i.e_., $\bm{A}^{*}$ in Eq. 4 on MS COCO. For ease of
explanation, we select two categories, _i.e_., animal and vehicle, and
investigate the label correspondence among labels associated with them. As
shown in Fig. 3, the used structured semantic prior can successfully convey
the similarity among labels although the CLIP is not encouraged in the
contrastive learning on the text. Second, we visualize the label features in
the baseline (weights in the sigmoid layer) and our SCPNet (output of the SAM,
denoted as ${\bm{z_{i}}}^{*}$ in Eq. 7). We can observe that labels belonging
to the same category are more well-aligned together in our SCPNet, compared
with those in the baseline model. This result indicates that our SCPNet can
reasonably derive more discriminative label representations due to the
appliance of the structured semantic prior.

To verify the effect of the proposed structured semantic prior on the issue of
insufficient label supervision, we introduce a competitor model, _i.e_.,
CMP+$\mathcal{L}_{cst}$, which wipes out components involving the prior,
_i.e_., $\bm{A}^{*}$ in Eq. 4. We keep track of the precision of model
predictions on the training set and the mAP result on the test set after each
training epoch. For CMP+$\mathcal{L}_{cst}$, we also visualize the precision
over its calibrated predictions by the $\text{SASC}(\cdot)$ function. As
illustrated in Fig. 4 (left), compared with CMP+$\mathcal{L}_{cst}$, both
SASC$($CMP$+\mathcal{L}_{cst})$ and our SCPNet can obtain consistent
improvements in terms of the label prediction precision. It indicates that the
quality of label supervision can be promoted under the guidance of the
proposed prior, thus benefiting the performance on the test set (see Fig. 4
(right)).

Figure 4: The precision on the training set (left) and the mAP on the test set
(right).

##  5 Conclusion

In this paper, we drive a structured semantic prior about the label-to-label
correspondence from the vision-language model, _i.e_., CLIP [22]. To mitigate
the deficiency of label supervision for MLR with incomplete labels, we
introduce a semantic correspondence prompt network, dubbed SCPNet, which can
explore such a structured semantic prior. It constructs a cross-modality
prompter to leverage the explicit image-to-label correspondence in the CLIP. A
semantic association module is equipped to associate related labels with the
help of such a meaningful structured semantic prior. Furthermore, we propose a
prior-enhanced self-supervised learning method for network optimization.
Experimental results on a series of benchmark datasets for MLR with incomplete
labels show that our method can achieve state-of-the-art performance on both
the partial label setting and the single positive label setting, well
demonstrating its effectiveness and superiority. In the future, we will
further study how to generalize our method to tackle other practical problems,
_e.g_., the domain gap.

Acknowledgement. This work was supported by “Pioneer” and “Leading Goose” R&D
Program of Zhejiang (No. 2023C01038), National Natural Science Foundation of
China (Nos. 62271281, 61773301), Zhejiang Provincial Natural Science
Foundation of China under Grant (No. LDT23F01013F01), China Postdoctoral
Science Foundation (No. BX2021161) and Shanxi Innovation Team Project (No.
2018TD-012).

## References

  * [1] Emanuel Ben-Baruch, Tal Ridnik, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, and Lihi Zelnik-Manor.  Multi-label classification with partial annotations using class-aware selective loss.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4764–4772, 2022.
  * [2] Dolly Carrillo, Vivian F López, and María N Moreno.  Multi-label classification for recommender systems.  Trends in Practical Applications of Agents and Multiagent Systems, pages 181–188, 2013.
  * [3] Tianshui Chen, Liang Lin, Xiaolu Hui, Riquan Chen, and Hefeng Wu.  Knowledge-guided multi-label few-shot learning for general image recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [4] Tianshui Chen, Tao Pu, Hefeng Wu, Yuan Xie, and Liang Lin.  Structured semantic transfer for multi-label recognition with partial labels.  In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 339–346, 2022.
  * [5] Tianshui Chen, Muxin Xu, Xiaolu Hui, Hefeng Wu, and Liang Lin.  Learning semantic-specific graph representation for multi-label image recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 522–531, 2019.
  * [6] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo.  Multi-label image recognition with graph convolutional networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5177–5186, 2019.
  * [7] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng.  Nus-wide: a real-world web image database from national university of singapore.  In Proceedings of the ACM international conference on image and video retrieval, pages 1–9, 2009.
  * [8] Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic.  Multi-label learning from single positive labels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 933–942, 2021.
  * [9] Thibaut Durand, Nazanin Mehrasa, and Greg Mori.  Learning a deep convnet for multi-label classification with partial labels.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 647–657, 2019.
  * [10] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [11] Mark Everingham and John Winn.  The pascal visual object classes challenge 2012 (voc2012) development kit.  Pattern Anal. Stat. Model. Comput. Learn., Tech. Rep, 2007:1–45, 2012.
  * [12] Tianyu Gao, Adam Fisch, and Danqi Chen.  Making pre-trained language models better few-shot learners.  arXiv preprint arXiv:2012.15723, 2020.
  * [13] Dat Huynh and Ehsan Elhamifar.  Interactive multi-label cnn learning with partial labels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9423–9432, 2020.
  * [14] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig.  How can we know what language models know?  Transactions of the Association for Computational Linguistics, 8:423–438, 2020.
  * [15] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  * [16] Youngwook Kim, Jae Myung Kim, Zeynep Akata, and Jungwoo Lee.  Large loss matters in weakly supervised multi-label classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14156–14165, 2022.
  * [17] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  Dataset available from https://github. com/openimages, 2(3):18, 2017\.
  * [18] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73, 2017.
  * [19] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [20] Jeffrey Pennington, Richard Socher, and Christopher Manning.  Glove: Global vectors for word representation.  In EMNLP, pages 1532–1543, 2014.
  * [21] Tao Pu, Tianshui Chen, Hefeng Wu, and Liang Lin.  Semantic-aware representation blending for multi-label image recognition with partial labels.  arXiv preprint arXiv:2203.02172, 2022.
  * [22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [23] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  arXiv preprint arXiv:2010.15980, 2020.
  * [24] Josef Sivic and Andrew Zisserman.  Video google: Efficient visual search of videos.  In Toward category-level object recognition, pages 127–144. Springer, 2006.
  * [25] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  Advances in neural information processing systems, 33:596–608, 2020\.
  * [26] Ximeng Sun, Ping Hu, and Kate Saenko.  Dualcoop: Fast adaptation to multi-label recognition with limited annotations.  arXiv preprint arXiv:2206.09541, 2022.
  * [27] Ivona Tautkute, Tomasz Trzciński, Aleksander P Skorupa, Łukasz Brocki, and Krzysztof Marasek.  Deepstyle: Multimodal search engine for fashion and interior design.  IEEE Access, 7:84613–84628, 2019.
  * [28] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [29] Ya Wang, Dongliang He, Fu Li, Xiang Long, Zhichao Zhou, Jinwen Ma, and Shilei Wen.  Multi-label classification with label graph superimposing.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12265–12272, 2020.
  * [30] Vacit Oguz Yazici, Abel Gonzalez-Garcia, Arnau Ramisa, Bartlomiej Twardowski, and Joost van de Weijer.  Orderless recurrent models for multi-label classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13440–13449, 2020.
  * [31] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  Advances in Neural Information Processing Systems, 34:18408–18419, 2021.
  * [32] Youcai Zhang, Yuhao Cheng, Xinyu Huang, Fei Wen, Rui Feng, Yaqian Li, and Yandong Guo.  Simple and robust loss design for multi-label learning with missing labels.  arXiv preprint arXiv:2112.07368, 2021.
  * [33] Yong Zheng, Bamshad Mobasher, and Robin Burke.  Context recommendation using multi-label classification.  In 2014 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), volume 2, pages 288–295. IEEE, 2014.
  * [34] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816–16825, 2022.
  * [35] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision, 130(9):2337–2348, 2022\.

## Exploring Structured Semantic Prior
for Multi Label Recognition with Incomplete Labels
Supplementary Material

##  Appendix A Methodology

Multi-label classification objective. We use the SPLC + Focal margin[32] loss
to optimize the whole model. For an input image, we denote its visual
representation as $\bm{f}$ and the label feature as $\bm{{z}_{i}}$, which can
be derived by the image encoder and text encoder, respectively (see Eq. (5)).
The multi-classification loss $\mathcal{L}_{cls}$ can be computed as

$\displaystyle{\mathcal{L}_{cls}}=-\sum\limits_{c=1}^{C}$
$\displaystyle\Big{\\{}{y_{c}}{{(1-p_{c}^{m})}^{\alpha}}\log(p_{c}^{m})+(1-{y_{c}})$
(15)
$\displaystyle\big{[}\mathbb{I}(p\leq\beta){p_{c}}^{\alpha}\log(1-{p_{c}})+$
$\displaystyle(1-\mathbb{I}(p\leq\beta)){{(1-{p_{c}})}^{\alpha}}\log({p_{c}})\big{]}\Big{\\}},$

where $p_{c}^{m}=\sigma(\text{sim}(\bm{f},\bm{{z_{i}}})/\tau-m)$ is the
likelihood and $m$ is a margin parameter. $\alpha$ is set to 2 and $\beta$ is
a threshold to identify negative label.

##  Appendix B Experiment Settings

###  B.1 Datesets for MLR with incomplete labels

For the single positive label setting, we conduct experiments on four standard
benchmarks, _i.e_., MS-COCO (COCO), PASCAL VOC 2012 (VOC), NUSWIDE (NUS) and
CUB. The statistics of all benchmark datasets on training datasets are shown
in Tab. 9. COCO contains 82,081 training images with 80 classes and a test set
of 40,137 images. VOC consists of 5,717 training images with 20 classes and
5,823 images for test. NUS is a public multi-label image classification
dataset which contains 269,648 images and each image is manually annotated
with some of 81 categories. CUB consists of 5,994 training images covering 312
categories and 5,794 test images. For a fair comparison with [16], [32], we
perform two different setups. The LargeLoss setup divides the training dataset
into 80$\%$ for training and 20$\%$ for validation. The SPLC setup only trains
on the overall training set and tests on the test set. The validation sets and
test sets are always fully labeled.

For the partial label setting, we adopt three benchmarks, _i.e_., MS-COCO
(COCO), PASCAL VOC 2007 (VOC2007) and Visual Genome (VG-200). COCO dataset is
same as the one used in the single positive label setting. VOC2007 contains a
training set of 5,011 images and a test set of 4,952 images. VG-200 contains a
total of 108,249 images covering tens of thousands of classes, most of which
have only few samples. Following [21], we choose 200 frequent classes as the
VG-200 subset, in which 10,000 images are randomly selected as the test set
and the remaining 82,904 images are used as the training set.

Table 9: The statistics of all benchmark datasets on training sets. Experiment setting | Dataset | Samples | Classes | Labels | Avg.label/img
---|---|---|---|---|---
Single positive label | COCO | 82,081 | 80 | 241,035 | 2.9
VOC | 5,717 | 20 | 8,331 | 1.5
NUS (LargeLoss) | 150,000 | 81 | 284,611 | 1.9
NUS (SPLC) | 119,103 | 81 | 289,460 | 2.4
CUB | 5,994 | 312 | 188,343 | 31.4
Partial labels | COCO | 82,081 | 80 | 241,035 | 2.9
VOC2007 | 5,011 | 20 | 7,306 | 1.5
VG-200 | 82,904 | 200 | 886,618 | 10.7
Real partial labels | OpenImages V3 | 3,552,103 | 5,000 | 13,440,371 | 3.8

###  B.2 Implementation details

For the single positive label setting, we use a single GPU with batch size
128. Each image of ours is uniformly resized to 224 × 224 while other methods
resize to 448 × 448. We use the Adam optimizer and OneCycle learning rate
schedule with the max learning rate of 3e-5. It is trained with 30 epochs in
total.

For the partial label setting, we use two GPUs with batch size 32 and the max
learning rate is 1e-5. We also train for 30 epochs on all benchmark datasets.

For data augmentation, we adopt the random horizontal flip and random resized
crop for the weak transformation and the RandAugment for the strong
transformation.

##  Appendix C More experiments results

###  C.1 Model Analysis

Effect of different modules on the ResNet. To investigate the effectiveness of
our proposed method with CNN-based MLR models, we conduct the ablation study
on replacing the pretrained CLIP with the ResNet model. As shown in Tab. 10,
each component can lead to performance improvement. Compared with the
baseline, introducing SAM can achieve a performance improvement of $1.01\%$,
which shows that exploiting the implicit label-to-label correspondence can
benefit the MLR with incomplete labels. In addition, using the overall
proposed PESSL can accomplish $0.53\%$ mAP improvement, well demonstrating the
advantage of incorporating the structured semantic prior to calibrate the
semantic distribution. Finally, our proposed method significantly outperforms
the baseline model with $1.54\%$ improvement, well indicating the
effectiveness and superiority of our method.

Table 10: Analysis of different modules on the ResNet50 (%). ResNet | SAM | PESSL | mAP
---|---|---|---
✓ |  |  | 73.18
✓ | ✓ |  | 74.19
✓ | ✓ | ✓ | 74.72
Table 11: Analysis on different combination of encoders (%). ResNet50 is pretrained on the ImageNet. ResNet50r is randomly initialized. Image Encoder | Label Encoder | mAP
---|---|---
ResNet50 | Linear | 73.18
CLIP | Linear | 73.33
ResNet50 | CLIP | 64.85
ResNet50r | CLIP | 41.71
CLIP | CLIP | 74.36

Combination of different encoders. We investigate different combinations of
image encoders and label encoders. As shown in Tab. 11, compared with our
method (the last row), replacing the CLIP label encoder with a linear
prediction layer leads to inferior performance for a pretrained/CLIP-based
ResNet50 (Row 1/2). Besides, replacing the CLIP image encoder with a
pretrained/random ResNet50 (Row 3/4) also encounters great performance drops
due to the destruction of image-label correspondence. This evidence shows the
superiority of applying CLIP as the base MLR model.

Effect of the prompt learning. We investigate the effect of the prompt
learning with the CMP model. As shown in Tab. 12, we can observe that with the
random initialization, different prompt length can obtain similar performance.
After leveraging the prompt template, _i.e_., a photo of a, CMP with 4 prompts
can achieve the best performance, indicating the positive effect of the hard
prompt [22] as the prompt initialization.

Table 12: Effect of the prompt in the CMP (%). Length | Initialization | mAP
---|---|---
4 | template | 74.36
4 | Random | 73.85
8 | Random | 73.87
16 | Random | 73.85
Table 13: Analysis on the semantic association module (SAM) (%). Label Feature | $\bm{H}^{L}$ | $\bm{H}^{0},\bm{H}^{L}$ | $\bm{H}^{0}+\bm{H}^{L}$
---|---|---|---
75.01 | 75.65 | 76.42
Correlation Matrix | Original | Sparse | Ours
75.96 | 76.13 | 76.42

Analysis on SAM. For the proposed SAM component, we first investigate how to
construct the label feature. We discuss three variants: 1) directly using
$\bm{H}^{L}$, 2) using $\bm{H}^{0}$ and $\bm{H}^{L}$ in a multi-task learning
strategy, and 3) the proposed residual connection, _i.e_.,
$\bm{H}^{0}+\bm{H}^{L}$. From Tab. 13, we can see that the residual connection
obtains the best performance. We further analyze the effect of different label
correlation matrix. We verify three strategies: 1) using original label
correlation matrix (see Eq. (1)), 2) resulting in a sparse matrix by retaining
the top $K$ elements (see Eq. (2)), and 3) our adjusted the sparse matrix (see
Eq. (4)), which is our proposed structured semantic prior, _i.e_.,
$\bm{A^{*}}$. As shown in Tab. 13, our proposed method can achieve the state-
of-the-art performance, demonstrating that the label-to-label correspondence
can be well captured by the proposed structured semantic prior.

Analysis of the pseudo label selection. We further investigate different
pseudo label selection for weak transformation in the PESSL. We discuss three
strategies: 1) using the weak transformation prediction probabilities as soft
label, 2) setting threshold to filter the pseudo label, and 3) our method
which further selects the top highest probability to construct a set of
confident labels. As shown in Tab. 14, our method obtains the best
performance, demonstrating that the selective construction of confidence
labels is more compatible with the MLR task.

Figure 5: Visualization of example results compared our SCPNet with the baseline model. Red color means results of false recognition and missing recognition. Blue color denotes ours results. GT means ground truth and the precision of recognition is in brackets. Table 14: Different pseudo label selection of PESSL (%). $\mathcal{L}_{cst}$ | Soft Label | Threshold | Ours
---|---|---|---
mAP | 75.88 | 75.80 | 76.42

Stress-testing on domain-specific datasets. We conduct experiments on a common
satellite dataset (AID222https://github.com/Hua-YS/AID-Multilabel-Dataset) and
a common medical dataset (ChestX-
ray14333https://nihcc.app.box.com/v/ChestXray-NIHCC) under the single positive
label setting to perform stress-testing on domain-specific datasets which are
far from those used in CLIP pretraining. As shown in Tab. 15, compared with
CMP, our SCPNet can obtain $5.84\%$ and $5.50\%$ performance improvement on
the AID and ChestX-ray14, respectively. These results show that our method
achieves the best performance although CLIP cannot generalize well in these
datasets, well demonstrating the generalization.

Table 15: Stress-testing on domain-specific datasets (%). Method | AID | ChestX-ray14
---|---|---
SPLC[32] | 71.26 | 25.60
CMP (ours) | 67.48 | 22.42
SCPNet (ours) | 73.32 | 27.92
Table 16: DualCoOp vs. SCPNet in terms of computation cost | DualCoOp | SCPNet
---|---|---
Training speed [iters/sec] | 4.27 | 2.82
Trainable parameters | 1.31M | 3.41M
GPU memory for training | 7.4G | 9.8G
Inference speed [samples/sec] | 318.59 | 322.76
GPU memory for inference | 3.4G | 3.4G
mAP performance on COCO | $81.9\%$ | $83.2\%$

DualCoOp vs. SCPNet. First, in terms of the model performance, for fair
comparison, we implemented our method with a frozen image encoder on
DualCoOp’s code under the same setting, achieving $83.2\%$ mAP (+$1.3\%$). We
also tuned DualCoOp under the SCPNet setting, where DualCoOp achieved inferior
mAP with $82.4\%$ (vs. ours: $83.8\%$). Second, in terms of the computation
efficiency, we compare our method with DualCoOp under the DualCoOp’s setting.
As shown in Tab. 16, during training, SCPNet consumes more resources than
DualCoOp. But the required cost is not unaffordable in practice. During
inference, both methods can derive label features offline. Therefore, SCPNet
is comparable to DualCoOp in terms of computational cost while enjoying the
superior performance. This shows SCPNet is more advanced or at least
comparable when applied in practical scenarios. These results clearly
demonstrate the effectiveness and superiority of our method, compared to
DualCoOp.

Figure 6: Results of the few-shot partial label setting on COCO, VOC, NUS and
CUB dataset (left: in the LargeLoss setup, right: in the SPLC setup).

Hyper-parameters selection. For the single positive label setting, most hyper-
parameters are directly borrowed from COCO, except for some dataset-dependent
hyper-parameters, _e.g_., $K$ in Eq. (2) (best at $60$/$15$/$50$/$280$ for
COCO/VOC/NUS/CUB). $s$ in Eq. (3) is empirically set to $0.2$. For the partial
label setting, most hyper-parameters are directly borrowed from the single
positive label setting, except for the learning rate. Even so, our method can
obtain consistent performance improvements in all scenarios, well
demonstrating the robustness of hyper-parameters. We show that after more
hyperparameter searches, we can obtain slightly better performance than the
reported one, _e.g_., $49.6\%$ vs. $49.4\%$ on VG-200.

###  C.2 Multi-label Recognition Results

Here we present the multi-label recognition results on the single positive
label setting. As shown in Fig. 5, our proposed method can successfully
recognize more accurate labels with lower false identifications (see examples
in the first row). Besides, compared with the baseline model, our method can
achieve fewer missing recognition for difficult labels, _e.g_., “broccoli” in
the middle of the second row. These results further demonstrate the
effectiveness and the superiority of our proposed method.

###  C.3 Few-Shot Single Positive Label Setting

To investigate the effectiveness of the proposed method with a smaller number
of training images, we further conduct the experiments in the few-shot single
positive label setting under both the LargeLoss setup [16] and the SPLC setup
[32].

As illustrated in Fig. 6 (left), in the LargeLoss setup, following [16], we
randomly sample the training images from 10$\%$ to 100$\%$ and conduct
experiments on the COCO dataset. We further compare our method with LargeLoss
[16] on the other benchmark datasets, _i.e_., VOC, NUS and CUB. Only given
10$\%$ of the training images, our method can obtain a maximal performance
improvement of $12.16\%$, $7.47\%$, and $13.48\%$ on COCO, VOC and NUS
dataset, respectively. For CUB dataset, we achieve a maximal improvement of
$9.22\%$ with giving the training images of 20$\%$. Overall, our method can
accomplish an average performance improvement of $7.16\%$, $3.20\%$, $9.35\%$,
and $7.62\%$ on four datasets with the training images from 10$\%$ to 100$\%$.
Besides, as shown in Fig. 6 (right), in the SPLC setup, we present the
comparison results with SPLC [32] on four benchmark datasets as well. The
maximum performance improvement achieved by our method can reach $10.31\%$,
$10.43\%$ and $14.14\%$ on COCO, VOC and NUS dateset, respectively. Our method
can bring a maximum improvement of $9.78\%$ with 20$\%$ training images on CUB
dataset. Our SCPNet can obtain $5.06\%$, $4.79\%$, $8.76\%$, and $7.82\%$
improvement on average for the four datasets, respectively.

These experimental results show that our proposed SCPNet can significantly
achieve state-of-the-art performance in different few-shot single positive
label setting, well indicating the generalization and superiority.

###  C.4 Real Partial Label Scenario

Dataset and implementation details. To analyze the effectiveness of the
proposed method in real partial label scenario, we conduct experiments on the
OpenImage V3[17] dataset with 5,000 classes. The details are shown in Tab. 9,
OpenImage V3 contains 3.5M training images, 42k validation images, and 125k
test images. Follwing [16], we divide the training images into 5 groups, where
G1 has the smallest number of the counted images and G5 is the largest one.
All Gs corresponds to the set of all categories.

Table 17: Results with real partial label on OpenImage V3 dataset.

Method | G1 | G2 | G3 | G4 | G5 | All Gs
---|---|---|---|---|---|---
CL[9] | 70.4 | 71.3 | 76.2 | 80.5 | 86.8 | 77.1
IMCL[13] | 71.0 | 72.6 | 77.6 | 81.8 | 87.3 | 78.1
Naive AN | 77.1 | 78.7 | 81.5 | 84.1 | 88.1 | 82.0
WAN[8] | 71.8 | 72.8 | 76.3 | 79.7 | 84.7 | 77.0
LSAN[8] | 68.4 | 69.3 | 73.7 | 77.9 | 85.6 | 75.0
LargeLoss[16] | 77.7 | 79.3 | 82.1 | 84.7 | 89.4 | 82.6
P-ASL[1] | 73.2 | 78.6 | 85.1 | 87.7 | 90.6 | 83.0
SCPNet (ours) | 79.6 | 81.8 | 85.3 | 87.9 | 92.1 | 85.3

Compared methods. We compare our method with Curriculumn Labeling(CL)[9],
IMCL[13], Naive AN, Weak AN (WAN)[8], Label Smoothing with AN (LSAN)[8],
LargeLoss[16] and P-ASL[1].

Results. As shown in Tab. 17, our method outperforms the state-of-the-art
methods on G1, G2, G3, G4 and G5 with an improvement of $1.9\%$, $2.5\%$,
$0.2\%$, $0.2\%$ and $1.5\%$, respectively. As a whole, our proposed SCPNet
can accomplish a performance improvement of $2.3\%$ on all Gs, well
demonstrating the effectiveness and generalization to practical scenarios.