# Multi-label Image Classification using Adaptive Graph Convolutional
Networks: from a Single Domain to Multiple Domains

Inder Pal Singh<EMAIL_ADDRESS>Enjie Ghorbel Oyebade Oyedotun Djamila
Aouada University of Luxembourg, 2 Av. de l’Universite, 4365 Esch-Belval Esch-
sur-Alzette, Luxembourg  High Institute of Multimedia Arts (ISAMM), University
of Manouba, Tunisia

###### Abstract

This paper proposes an adaptive graph-based approach for multi-label image
classification. Graph-based methods have been largely exploited in the field
of multi-label classification, given their ability to model label
correlations. Specifically, their effectiveness has been proven not only when
considering a single domain but also when taking into account multiple
domains. However, the topology of the used graph is not optimal as it is pre-
defined heuristically. In addition, consecutive Graph Convolutional Network
(GCN) aggregations tend to destroy the feature similarity. To overcome these
issues, an architecture for learning the graph connectivity in an end-to-end
fashion is introduced. This is done by integrating an attention-based
mechanism and a similarity-preserving strategy. The proposed framework is then
extended to multiple domains using an adversarial training scheme. Numerous
experiments are reported on well-known single-domain and multi-domain
benchmarks. The results demonstrate that our approach achieves competitive
results in terms of mean Average Precision (mAP) and model size as compared to
the state-of-the-art. The code will be made publicly available.

##  1 Introduction

Multi-label image classification has been widely investigated by the computer
vision community, given its practical relevance in numerous application
fields, such as human attribute recognition (Li et al., 2016), scene
classification (Shao et al., 2015) and deepfake detection (Singh et al.,
2023). In contrast to traditional image classification approaches that
associate a single label to an input image, multi-label image classification
aims to detect the presence of a set of objects.

Fig. 1: Comparison of our approach (ML-AGCN) without (top) and with UDA (down)
to recent state-of-the-art methods in terms of number of parameters (millions)
and mean Average Precision (mAP) on MS-COCO and Clipart $\rightarrow$ VOC. The
considered state-of-the art methods are: MlTr-m (Cheng et al., 2022),
TResNet-L (Ridnik et al., 2021b), ML-Decoder (Ridnik et al., 2023), ML-GCN
(Chen et al., 2019b), ResNet101 (He et al., 2016), DA-MAIC (Lin et al., 2021),
and DANN (Ganin et al., 2016).

Thanks to the latest progress in deep learning, most of recent methods rely on
a single-stream Deep Neural Networks (DNN), including Convolution Neural
Networks (CNN) (Simonyan and Zisserman, 2015; He et al., 2016; Zhu et al.,
2017; Ge et al., 2018; Gao and Zhou, 2021; Ridnik et al., 2021b), Residual
Neural Networks (RNN) (Wang et al., 2016) and Transformers (Lanchantin et al.,
2021; Cheng et al., 2022; Wang et al., 2022a; Chen et al., 2020).
Nevertheless, their impressive performance comes at the cost of very large
architectures that are unsuitable for memory-constrained environments. As an
alternative, another line of research has tried to integrate priors related to
label correlations (Chen et al., 2019b, a; Li et al., 2019; Wang et al.,
2020b; Singh et al., 2022b; Wang et al., 2022b; Sun et al., 2022). As
demonstrated in (Singh et al., 2022b), such an approach contributes to
improving scalability. In other words, fewer parameters are required to
achieve comparable performance with traditional DNNs.

Graph-based approaches (Chen et al., 2019b; Singh et al., 2022b), are among
the most popular multi-label classification methods that aim at modeling label
correlations. In addition to a standard DNN that extracts image features, a
second stream based on a Graph Convolutional Network (GCN) is used for
generating inter-dependent label classifiers. In particular, the input graph
is used to model label dependencies. Each node represents a label, and each
edge is characterized by the probability of co-occurrence of a label pair.
Graph-based methods have also been shown to be successful in the challenging
context of cross-domain multi-label classification (Lin et al., 2021).
Standard multi-label image classification methods such as (Chen et al., 2019b;
Singh et al., 2022b) usually assume that unseen images and training data are
drawn from the same distribution, i.e. the same domain, hence ignoring a
possible domain shift problem (Lin et al., 2021). This leads, therefore, to
poor generalization capabilities under cross-domain settings. UDA is a
plausible solution to overcome this challenge without relying on costly
annotation efforts (Ganin and Lempitsky, 2015). UDA aims at learning domain
invariant features to bridge the gap between a source domain and a target
domain without access to the associated labels. In particular, one of the most
successful UDA approaches for multi-label classification, namely DA-MAIC (Lin
et al., 2021), leverages graph representations to model label inter-
dependencies and couple it with an adversarial training approach (Ganin et
al., 2016)

Despite their usefulness in both single and cross-domain settings, graph-based
methods (Chen et al., 2019b, a; Li et al., 2019; Wang et al., 2020b; Singh et
al., 2022b; Wang et al., 2022b; Sun et al., 2022; Lin et al., 2021) are
unfortunately subject to three major limitations, namely: (1) The graph
structure is heuristically defined. In particular, it is computed based on the
co-occurrence of labels in the training data. Hence, this topology might not
be ideal for the specific task of multi-label image classification; (2) A
threshold is empirically fixed for discarding edges with a low co-occurrence
probability. This means that infrequent co-occurrences are assumed to be
noisy. Although this might be true in many cases, assuming that any rare event
corresponds to noise does not always hold; and (3) it has been proven in (Jin
et al., 2021) that successive aggregation operations in the GCN usually
dissipate the node similarity in the original feature space, hence potentially
leading to a decrease in terms of performance.

Herein, we posit that by integrating adequate mechanisms in graph-based
approaches for addressing the aforementioned issues, it should be possible to
reduce the network size even more while achieving competitive performance in
both single-domain and cross-domain settings.

In this paper, we propose an adaptive graph-based multi-label classification
method called Multi-Label Adaptive Graph Convolutional Network (ML-AGCN) for
both contexts, single domain and across domains. Our idea consists in: (1)
learning two additional adjacency matrices in an end-to-end manner instead of
solely relying on a heuristically defined graph topology. Note that no
threshold is applied, avoiding the loss of weak yet relevant connections. In
particular, the first learned graph topology computes the importance of each
node pair. This is carried out by employing an attention mechanism similar to
Graph Attention Networks (GAT) (Velickovic et al., 2017). The second is built
based on the similarity between node features and overcomes the information
loss happening through successive convolutions; (2) integrating the proposed
adaptive graph-based architecture in an adversarial domain adaptation
framework for aligning a labeled source domain to an unlabeled target domain.
As shown in Fig. 1, the results suggest that our method is competitive with
respect to the state-of-the-art in terms of both mean Average Precision (mAP)
and network size under the single domain and cross-domain settings.

This paper is an extended version of (Singh et al., 2022a). In comparison to
(Singh et al., 2022a), the main contributions of this article are given below:

  1. 1.

More extensive experiments showing the relevance of the proposed architecture
called Multi-Label Adaptive Graph Convolutional Network (ML-AGCN) for multi-
label image classification.

  2. 2.

An adversarial Domain Adaptation approach integrating an Adaptive Graph
Convolutional Network (DA-AGCN) for multi-label image classification.

  3. 3.

A deep qualitative and quantitative experimental analysis of the proposed
domain adaptation method with respect to the state-of-the-art.

The remainder of this paper is organized as follows. Section 2 reviews the
state-of-the-art on multi-label image classification and domain adaptation for
multi-label classification. Section 3 formulates the problem of graph-based
multi-label image classification and its applicability to different domains.
In Section 4 and Section 5, the proposed method is detailed. Section 6
presents the experimental results and analysis. Finally, Section 7 concludes
this work and highlights interesting future directions.

##  2 Related Works

In this section, we discuss the state-of-the-art approaches for multi-label
image classification in both single and cross-domain settings.

As discussed in Section 1, most multi-label classification methods (Razavian
et al., 2014; Wei et al., 2015; Lanchantin et al., 2021; Cheng et al., 2022;
Wang et al., 2022a; Ridnik et al., 2021b, a) employ a single stream DNN. More
specifically, they mainly take inspiration from successful architectures
proposed in the context of single-label image classification such as CNN, RNN
and transformers. For example, Razavian et al. (2014) have used a pre-trained
OVERFeat (Sermanet et al., 2014) model and have adapted it to multi-label
image classification. Wei et al. (2015) have leveraged the prediction of
multiple CNN architectures pretrained on ImageNet (Deng et al., 2009) such as
AlexNet (Krizhevsky et al., 2012) and VGG-16 (Simonyan and Zisserman, 2015).
Ridnik et al. (Ridnik et al., 2021a) have employed TResNet (Ridnik et al.,
2021b) using a novel loss called Asymmetric Loss (ASL) that focuses more on
positive labels than negative ones. TResNet introduced in (Ridnik et al.,
2021b) is based on a ResNet architecture with a series of modifications for
optimizing the GPU network capabilities while maintaining the performance.

Recently, Lanchantin et al. (2021) attempted to leverage transformers for
modeling complex dependencies among visual features and labels. Similarly,
MlTr (Cheng et al., 2022) combines the pixel attention and the attention among
image patches to better excavate the transformer’s activity in multi-label
image classification. More recently, ML-Decoder (Ridnik et al., 2023) proposed
a transformer-based classification head instead of the standard Global Average
Pooling (GAP), for improving the generalization capability.

Going deeper into the network enables the model to learn more abstract
features from the image. However, this comes at the cost of high memory
requirements. Moreover, these single-stream methods do not explicitly model
the relationship between labels which is an important semantic element to
consider (Chen et al., 2019b).

In order to incorporate the information of label correlations, a second class
of methods have used a second subnetwork in addition to the main backbone. For
instance, Zhu et al. (2017) have introduced a Spatial Regularization Net (SRN)
to learn the underlying relationships between labels by generating label-wise
attention maps. Similarly, Qu et al. (2023) have employed image
representations from intermediate convolutional layers to model both local and
global label semantics.

Graphs can also be an interesting way for modeling label correlations (Chen et
al., 2019b; Singh et al., 2022b; Wang et al., 2020a, 2022a), while keeping the
network size reasonable. Graph-based methods (Chen et al., 2019b; Singh et
al., 2022b; Wang et al., 2020a) usually employ a GCN for learning
interdependent label-wise classifiers and combine it with a standard DNN that
learns discriminative image features. The input graph is heuristically
predefined based on label co-occurrences in the training set, where low values
are ignored. However, following such an empirical strategy might lead to sub-
optimal performances. Additionally, destroying the similarity of node features
through the GCN layers might lead to a decrease in performance, as discussed
in (Jin et al., 2021).

Moreover, similar to any DL method, the aforementioned multi-label
classification methods heavily depend on the availability of a large amount of
annotated data. For mitigating the huge cost caused by data annotation, the
field of unsupervised domain adaptation (Li et al., 2020; Zhang et al., 2019;
Long et al., 2017; Ganin et al., 2016; Long et al., 2018) has been widely
investigated over the last decade. It aims at making use of an existing
labeled dataset from a related domain called source domain to enhance the
model performance on a domain of interest termed target domain, for which only
unlabeled data are provided. Unsupervised domain adaptation methods can be
separated into two main categories. The first one (Li et al., 2020; Zhang et
al., 2019; Long et al., 2017) aims at explicitly reducing the domain gap by
minimizing statistical discrepancy measures between the two domains.
Alternatively, the second class of methods implicitly minimizes this domain
gap by adopting an adversarial training approach (Ganin et al., 2016; Long et
al., 2018). The main idea consists in using a domain classifier to play a min-
max two-player game with the feature generator. This strategy is designed to
enforce the generation of domain-invariant features that are sufficiently
discriminative.

Nevertheless, most existing techniques focus on the task of single-label image
classification. In fact, very few papers have considered domain adaptation for
multi-label image classification (Li et al., 2021; Pham et al., 2021; Chen et
al., 2019b).

Among these rare references, we can mention ML-ANet (Li et al., 2021), which
explicitly minimizes the domain gap by optimizing multi-kernels maximum mean
discrepancies (MK-MMD) in a Reproducing Kernel Hilbert Space (RKHS). More
recently, an adversarial approach has been adopted in (Pham et al., 2021)
where a condition-based domain discriminator similar to conditional-GANs
(Mirza and Osindero, 2014) has been employed. However, similar to the first
category of methods for traditional multi-label image classification, these
two approaches (Li et al., 2021; Pham et al., 2021) neglect the important
information of label dependencies. A graph-based approach called DA-MAIC has
been then proposed as an alternative (Lin et al., 2021). As in ML-GCN (Chen et
al., 2019b), the authors have proposed to build a graph for modeling label
correlations based on label co-occurrences. Additionally, to reduce the domain
shift between the source and target domains, they have used a domain
classifier that is trained in an adversarial manner. Unfortunately, DA-MAIC is
impacted by the same drawbacks affecting ML-GCN (Chen et al., 2019b) as
detailed in Section 1. More details regarding these issues are given in the
next section.

##  3 Background and Problem Formulation

In this section, Graph Convolutional Networks (GCN) are first reviewed. Then,
the problem of multi-label image classification in both single and cross-
domain settings is formulated.

###  3.1 Background: Graph Convolutional Networks (GCN)

CNNs are defined on a regular grid and are, therefore, not directly applicable
to non-Euclidean structures such as graphs. GCN (Kipf and Welling, 2017) have
been proposed as the generalization of traditional CNN to graphs. They have
been very successful in numerous computer vision applications, including human
pose estimation (Cai et al., 2019) and human action recognition (Papadopoulos
et al., 2020a, b). Let us denote a graph by $G=(V,E,\mathbf{F})$. The set
$V=\\{v_{1},v_{2},...,v_{N}\\}$ is formed by $N$ nodes, while
$E=\\{e_{1},e_{2},...,e_{M}\\}$ refers to the set of $M$ edges connecting the
nodes. Finally,
$\mathbf{F}=\\{\mathbf{f}_{1},\mathbf{f}_{2},...,\mathbf{f}_{N}\\}$ represents
the node features such that $\mathbf{f}_{i}\in\mathbb{R}^{d}$ corresponds to
the features of the node $v_{i}$.

Let us assume that $\mathbf{F}^{l}$ is the input node features of the
$l^{\text{th}}$ layer and $\mathbf{A}\in\mathbb{R}^{N\times N}$ the adjacency
matrix of the graph $\mathcal{G}$. Each GCN layer can be seen as a non-linear
function $h(.)$ that computes the node features
$\mathbf{F}^{l+1}\in\mathbb{R}^{N\times d^{\prime}}$ of the
$(l+1)^{\text{th}}$ layer as follows,

$\mathbf{F}^{l+1}=h(\mathbf{A}\mathbf{F}^{l}\mathbf{W}^{l}),$ (1)

where $\mathbf{W}^{l}\in\mathbb{R}^{d\times d^{\prime}}$ is the weight matrix.
Note that $\mathbf{A}$ is normalized before using Eq. (1).

###  3.2 Problem Formulation

####  3.2.1 Graph-based multi-label image classification

The goal of multi-label image classification is to predict the presence or
absence of a set of objects $\mathcal{O}=\\{1,2,...,N\\}$ in a given image
$\mathbf{I}$. This can be done by learning a function $f$ such that,

$\displaystyle f\colon$ $\displaystyle\mathbb{R}^{w\times
h}\to[\\![0,1]\\!]^{N}$
$\displaystyle\mathbf{I}\mapsto\mathbf{y}=[y_{i}]_{i\in\mathcal{O}},$

where $w$ and $h$ define the pixel-wise width and height of the image,
respectively, and $y_{i}=1$ indicates the presence of the label $i$ in
$\mathbf{I}$, in contrast to $y_{i}=0$. Graph-based multi-label image
classification methods such as ML-GCN (Chen et al., 2019b) and IML-GCN (Singh
et al., 2022b) usually involve two subnetworks: (1) A feature generator
denoted by $f_{g}$, and (2) an estimator of $N$ inter-dependent binary
classifiers denoted by $f_{c}$. The generator mostly corresponds to an out-
off-the-shelf CNN network which produces a $d_{f}$-dimensional image feature
representation as described below,

$\begin{split}f_{g}\colon&\mathbb{R}^{w\times h}\to\mathbb{R}^{d_{f}}\\\
&\mathbf{I}\mapsto\mathbf{X}\end{split}.$ (2)

For instance, ML-GCN integrates a ResNet101 (He et al., 2016), while IML-GCN
makes use of TResNet-M (Ridnik et al., 2021b).

On the other hand, the second subnetwork $f_{c}$ is a GCN formed by $L$ layers
which takes a fixed graph $G=(V,{E},\mathbf{F})$ as input where
$\text{card}(V)=N$. In fact, each node $v_{i}\in V$ refers to the label
$i\in\\{1,2,...,N\\}$ and each $\mathbf{f}_{i}\in\mathbf{F}$ corresponds to
its associated label embedding. The adjacency matrix
$\mathbf{A}\in\mathbb{R}^{N\times N}$ of $G$ is usually pre-computed based on
the co-occurrence probabilities that are estimated over the training set
(Singh et al., 2022b; Chen et al., 2019b). In addition, rare co-occurrences
are discarded as they are assumed to be noisy. In other words, given an
empirically fixed threshold $\tau$,

$\mathbf{A}_{ij}=\left\\{\begin{array}[]{l}0,\text{ if }p_{ij}<\tau,\\\
1,\text{ if }p_{ij}\geq\tau\end{array}\right.,$ (3)

with $p_{ij}=p(I_{j}|I_{i})$ being the probability of co-occurrence of label
$j$ and label $i$ in the same image.

Given $\mathbf{F}^{l}\in\mathbb{R}^{d_{l}\times L}$ as the input node features
of the $l^{\text{th}}$ layer, the input features $\mathbf{F}^{l+1}$ of the
$(l+1)^{\text{th}}$ are therefore computed by following Eq. (1).

Finally, the vertex features produced by the last GCN layer, i.e.,
$\mathbf{F}^{L}\in\mathbb{R}^{N\times d_{f}}$, form the $N$ inter-dependent
classifiers.

In summary, $f_{c}$ can be defined as follows,

$\begin{split}f_{c}\colon&\mathcal{G}(N)\to\mathbb{R}^{d_{f}\times N}\\\
&G\mapsto\mathbf{F}^{L}=[F^{L}_{1},...,F^{L}_{N}],\end{split}$ (4)

where $\mathcal{G}(N)$ represents the set of graphs with $N$ nodes, and
$\mathbf{F}_{i}^{L}$ for $i\in\\{1,...,N\\}$ is the generated inter-dependent
binary classifier associated to the label $i$.

Hence, $f$, which returns the final prediction, can be defined as follows,

$\begin{split}f(\mathbf{I})&=\text{sig}(f_{c}(G)f_{g}(\mathbf{I})^{T})\\\
&=\text{sig}(\mathbf{F}^{L}\mathbf{X}^{T}),\end{split}$ (5)

where $\text{sig}(x)=\frac{1}{1+e^{-x}}$ is the sigmoid activation function.

Fig. 2: Architecture of ML-AGCN: On the one hand, the CNN subnet learns
relevant image features from an input image. On the other hand, the GCN subnet
estimates interdependent label classifiers by taking into account one fixed
adjacency matrix $\mathbf{A}$ and two adaptive adjacency matrices
$\mathbf{B}^{(l)}$ and $\mathbf{C}^{(l)}$. Finally, the classifiers are
applied to the CNN features for predicting the labels.

However, as explained in Section 1, three main limitations can be noted in
graph-based methods: (1) the computation of the adjacency matrix $\mathbf{A}$
is made heuristically and is decoupled from the training process; (2) a
threshold $\tau$ is empirically fixed for completely ignoring rare co-
occurrences; and (3) as shown in (Jin et al., 2021), aggregating successively
node features in a graph may induce the loss of the similarity/dissimilarity
information present in the initial feature space.

####  3.2.2 Graph-based unsupervised domain adaptation for multi-label image
classification

Let us consider a source dataset for multi-label image classification
$\\{{(\mathbf{I}_{s}^{k},\mathbf{y}_{s}^{k})}\\}_{k=1}^{n_{s}}$ drawn from a
distribution $\mathcal{D}_{s}$ and a similar unlabelled target dataset
$\\{{(\mathbf{I}^{k}_{t})}\\}_{k=1}^{n_{t}}$ drawn from a different
distribution $\mathcal{D}_{t}$. The matrix
$\mathbf{I}_{s}^{k}\in\mathbb{R}^{w\times h}$ refers to the $k^{\text{th}}$
image sample of $\mathcal{D}_{s}$ and $\mathbf{y}_{s}^{k}\in[\\![0,1]\\!]^{N}$
is its associated label vector, while
$\mathbf{I}_{s}^{t}\in\mathbb{R}^{w\times h}$ denotes the $k^{\text{th}}$
image sample of $\mathcal{D}_{t}$. The variables $n_{s}$ and $n_{t}$ refer
respectively to the total number of samples in $\mathcal{D}_{s}$ and
$\mathcal{D}_{s}$. The goal of unsupervised domain adaptation is to train a
model using labeled source domain data sampled from $\mathcal{D}_{s}$ and
unlabelled target domain samples from $\mathcal{D}_{t}$ for making accurate
predictions on the target domain. Despite its relevance in real-world
applications, few domain adaptation methods have been proposed for multi-label
image classification. Among the most recent and accurate approaches, one can
mention DA-MAIC (Lin et al., 2021), which also takes advantage of the graph
representation for modeling label correlations. It directly extends ML-GCN
(Chen et al., 2019b) by integrating an adversarial training for domain
adaptation. Consequently, DA-MAIC is subject to the same limitations induced
by ML-GCN (Chen et al., 2019b) mentioned in Section 3.2.1.

##  4 Multi-Label Adaptive Graph Convolutional Network (ML-AGCN)

To handle the challenges mentioned in Section 3, a novel graph-based approach
called Multi-Label Adaptive Graph Convolutional Network (ML-AGCN) is
introduced.

###  4.1 Overview of the Proposed Architecture

Similar to (Chen et al., 2019b) and (Singh et al., 2022b), a network formed by
two subnets is adopted as illustrated in Fig. 2. The first is a CNN that
extracts a discriminative representation from a given input image, while the
second is a GCN-based network that learns $N$ interdependent classifiers. As
in (Singh et al., 2022b), TResNet-M which represents a small version of
TResNet (Ridnik et al., 2021b) is employed as a CNN subnetwork. TResNet is a
direct extension of ResNet, which fully exploits the GPU capabilities to boost
the model efficiency. The graph-based subnet, Adaptive Graph Convolutional
Network (AGCN), uses the same image embeddings proposed in (Singh et al.,
2022b) as feature nodes. Nevertheless, in contrast to (Singh et al., 2022b)
and (Chen et al., 2019b), it relies on an end-to-end learned graph topology.
More details regarding this subnetwork are provided in Section 4.2. Similar to
(Singh et al., 2022b), the Asymmetric Loss (ASL) (Ridnik et al., 2021a)
denoted by $\mathcal{L}_{c}$ is used for optimizing ML-AGCN such that,

$\mathcal{L}_{c}=\mathbb{E}_{(\mathbf{I}_{s},\mathbf{y}_{s})\sim\mathcal{D}_{s}}\sum_{i=1}^{N}y_{s}^{(i)}(1-p^{(i)})^{\gamma_{+}}\log(p^{(i)})+\\\
(1-y_{s}^{(i)})(p_{m}^{(i)})^{\gamma_{-}}\log(1-p_{m}^{(i)}),$ (6)

where $\gamma_{+}$ and $\gamma_{-}$ are focusing parameters for positive and
negative samples, respectively, and $y_{s}^{(i)}$ and $p^{(i)}$ are the
respective ground truth and predicted probability with respect to the label
$i$ and $p_{m}^{(i)}$ is the shifted probability given by $\max(p^{(i)}-m,0)$,
where $m$ is a threshold used for reducing the effect of easy negative samples
(Ridnik et al., 2021a).

(a) Fixed graph $\tau=0.1$

(b) Parameterized graph $\tau=0$

Fig. 3: (a) An example of a fixed label graph with a threshold set to
$\tau=0.1$. Dashed (red) edges indicate the ignored edges; (b) The proposed
parameterized graph topology considering all the edges.

###  4.2 Graph-based Subnet: Adaptive Graph Convolutional Network (AGCN)

Our intuition is that by integrating a suitable mechanism, it should be
possible to boost the classification performance and reduce at the same time
the size of graph-based methods. Hence, as illustrated in Fig. 3, we propose
to adaptively learn the graph topology by reformulating Eq. (1) as below,

$\mathbf{F}^{l+1}=\sigma(({\mathbf{A}}+\mathbf{B}^{(l)}+\mathbf{C}^{(l)})\mathbf{F}^{l}\mathbf{W}^{l}),$
(7)

where $\sigma$ is a LeakyRELU activation function.

Instead of relying solely on the adjacency matrix $\mathbf{A}$ defined in
(Chen et al., 2019b), two additional parameterized graphs called attention-
based and similarity adjacency graphs, respectively denoted by
$\mathbf{B}^{(l)}$ and $\mathbf{C}^{(l)}$, are defined. In this case, no
threshold is applied to $\mathbf{A}$ for ignoring rare co-occurrences. It is
also important to note that $\mathbf{A}$ is fixed, while $\mathbf{B}^{(l)}$
and $\mathbf{C}^{(l)}$ vary from one layer to another. In the following, we
detail how $\mathbf{B}^{(l)}$ and $\mathbf{C}^{(l)}$ are computed.

####  4.2.1 Attention-based adjacency matrix

Instead of ignoring rare co-occurrences in $\mathbf{A}$, the matrix
$\mathbf{B}^{(l)}=(b_{ij}^{(l)})_{i,j\in\mathcal{O}}$ is defined based on an
attention mechanism where the importance of each edge is quantified. To that
aim, inspired by (Velickovic et al., 2017), an attention score denoted by
$e_{ij}$ is calculated for each pair of vertices $(v_{i},v_{j})$ as follows,

$e_{ij}=\sigma(\mathbf{a}^{(l)^{T}}\mathbf{(}\mathbf{W}\mathbf{F_{i}}^{(l)}||\mathbf{W}\mathbf{F_{j}}^{(l)}),$
(8)

where $\mathbf{W}\in\mathbb{R}^{d^{(l+1)}\times d^{(l)}}$ represents a
learnable weight matrix, $\mathbf{a}^{(l)^{T}}\in\mathbb{R}^{2d^{(l+1)}}$ are
the learnable attention coefficients and $||$ refers to the concatenation
operation.

A softmax function is then applied to the computed normalized attention scores
such that,

$\alpha_{ij}^{(l)}=\frac{\text{exp}(e_{ij}^{(l)})}{\sum_{k\in\mathcal{N}{(i)}}\text{exp}(e_{ik}^{(l)})},$
(9)

with $\mathcal{N}{(i)}$ defining the neighborhood of the node $i$ and
$\alpha_{ij}$ being the obtained normalized attention score.

We recall that the goal of the GCN subnet is to generate interdependent label
classifiers. This means that each classifier must be predominated by the
information related to the label it belongs to. Hence, the attention score of
the node in question should be maximal. For this purpose, an additional step
called self-importance mechanism is proposed for computing the attention-based
adjacency matrix $\mathbf{B}^{(l)}=(b_{ij}^{(l)})_{i,j\in\mathcal{O}}$ as
follows,

$\begin{cases}b_{ij}^{(l)}=\alpha_{ij}^{(l)}+\max\limits_{k\in\mathcal{O}}(\alpha_{ik}^{(l)})\text{
if }i=j\\\ b_{ij}^{(l)}=\alpha_{ij}^{(l)}\text{ if }i\neq j\end{cases}.$ (10)

(a) Before GCN

(b) After GCN

Fig. 4: Features of graph nodes: the similarity between the node features is
lost after GCN aggregations.

####  4.2.2 Similarity-based adjacency matrix $(\mathbf{C})$

As illustrated in Fig. 4, the GCN aggregation tends to modify the node
similarity in the original feature space (Jin et al., 2021). To overcome this
issue, a node-similarity preserving matrix
$\mathbf{C}^{(l)}=(c_{ij}^{(l)})_{i,j\in\mathcal{O}}$ is proposed. It is
obtained by calculating the cosine similarity $c_{ij}^{(l)}$ between each pair
of vertices $(v_{i}$,$v_{j})$ as follows,

$c_{ij}^{(l)}=\frac{\mathbf{F}_{i}^{(l)}.\mathbf{F}_{j}^{(l)}}{\|\mathbf{F}_{i}^{(l)}\|\|\mathbf{F}_{j}^{(l)}\|},$
(11)

where $\|.\|$ denotes the $L_{2}$ Euclidean norm.

Finally, the output of the final layer $L$ denoted by $\mathbf{F}^{L}$ is used
in Eq. (5) for predicting the labels.

##  5 Domain Adaptation for Multi-label Image Classification using ML-AGCN

Fig. 5: Architecture of the proposed DA-AGCN for multi-label image
classification (best viewed in color). Images from both source and target
datasets are given as input to the CNN subnet that generates image features.
The AGCN-subnet learns in an end-to-end manner the attention and similarity-
based adjacency matrices $\mathbf{B}^{(l)}$ and $\mathbf{C}^{(l)}$,
respectively, and generates accordingly inter-dependent label classifiers
using only labeled source images. In addition, a domain classifier is
considered.

The proposed architecture for multi-label image classification called DA-AGCN
is illustrated in Fig. 5. Similar to ML-AGCN, we make use of TResNet-based
backbone $f_{g}$ to extract discriminative image features and an Adaptive
Graph Convolutional Network $f_{c}$ to learn $N$ interdependent classifiers.
Given source and target input images from $\mathcal{D}_{s}$ and
$\mathcal{D}_{t}$, respectively, the goal of $f_{g}$ is to generate $D$-dim
domain-invariant image features. The latter is used to predict the image
labels, as depicted in Eq. (5). The label classification loss
$\mathcal{L}_{c}$ is therefore defined as in Eq. (6).

For obtaining domain-invariant representations, a domain classifier
$f_{d}\colon\mathbb{R}^{d_{f}}\to[\\![0,1]\\!]$ is employed and trained in an
adversarial manner. Given the features $\mathbf{X}=f_{g}(\mathbf{I})$
extracted from a given image $\mathbf{I}$, $f_{d}$ predicts the domain of the
input image as follows,

$\begin{split}f_{d}\colon&\mathbb{R}^{d_{f}}\to[0,1]\\\
&\mathbf{X}\mapsto\hat{d}.\end{split}$ (12)

where $\hat{d}$ is the predicted domain label of $\mathbf{I}$. Note that the
ground-truth domain label $d=0$ if $\mathbf{I}$ is from the source domain and
$d=1$ if sampled from the target domain.

As in (Ganin et al., 2016), a domain loss is defined as below,

$\mathcal{L}_{d}=\mathbb{E}_{f_{g}(\mathbf{I}_{s})\sim\mathcal{D}_{s}}\log\frac{1}{f_{d}(f_{g}(\mathbf{I}_{s}))}+\\\
\mathbb{E}_{f_{g}(\mathbf{I}_{t})\sim\mathcal{D}_{t}}\log\frac{1}{(1-f_{d}(f_{g}(\mathbf{I}_{t}))}.$
(13)

Given $\mathcal{L}_{c}$ defined in Eq. (6), the final objective function used
to optimize the network is $E(\theta_{g},\theta_{d},\theta_{c})$ defined by,

$E(\theta_{g},\theta_{d},\theta_{c})=\mathcal{L}_{c}(\theta_{c},\theta_{g})+\lambda\mathcal{L}_{d}(\theta_{d},\theta_{g}),$
(14)

where $\theta_{g},\theta_{d}$ and $\theta_{c}$ refer, respectively, to the
parameters of $f_{g}$, $f_{d}$ and $f_{c}$ and $\lambda$ is a hyper-parameter
defining the weight of $\mathcal{L}_{d}$. The network is trained in an
adversarial manner using the GRL for obtaining the optimal parameters
$(\hat{\theta}_{g},\hat{\theta}_{c},\hat{\theta}_{d})$ such that,

$\begin{split}(\hat{\theta}_{g},\hat{\theta}_{c})=\min_{\theta_{g},\theta_{c}}E(\theta_{g},\theta_{c},\hat{\theta}_{d}),\\\
\hat{\theta}_{d}=\max_{\theta_{d}}E(\hat{\theta}_{g},\hat{\theta}_{c},\theta_{d}).\end{split}$
(15)

##  6 Experiments

In this section, the experimental settings and results are presented and
discussed for: (1) single-domain multi-label image classification (Section
6.1); and (2) domain-adaptation for multi-label image classification (Section
6.2).

###  6.1 Multi-label Image Classification

####  6.1.1 Implementation details

As discussed in Section 4, TResNet-M (Ridnik et al., 2021b) is used as the CNN
backbone. In particular, the fully connected layer after the Global Average
Pooling (GAP) layer is removed. The output of the GAP layer produces a
2048-dimensional latent image representation. The dimension of the AGCN output
has also been set to 2048. For the node features, we use the image-based
embeddings proposed in (Singh et al., 2022b) instead of the usual word-based
embeddings used in (Chen et al., 2019b). The model is trained for 40 epochs
using Adam, with a maximum learning rate of $1e-4$ using a cosine decay.

####  6.1.2 Datasets

In the following, the datasets that have been used for the experiments are
presented.

##### MS-COCO

MS-COCO (Lin et al., ) is a widely used large-scale multi-label image dataset.
It contains 80K training images and 40k testing images. Each image is
annotated with multiple object labels from a total of 80 categories.

##### VG-500

The VG-500 dataset (Chen et al., 2020) is a well-known dataset for multi-label
image classification. It includes 500 different objects as categories. The
dataset comes with a training set of 98,249 images and a testing set of 10,000
images.

##### PASCAL-VOC 2007

The PASCAL Visual Object Classes Challenge (Everingham et al., 2010)
introduced in 2007 is one of the most commonly used multi-label image
classification datasets. It contains about 10K image samples with 5011 and
4952 images as training and testing sets, respectively. The images show 20
different object categories with an average of 2.5 categories per image.

####  6.1.3 Quantitative analysis

##### Comparison with state-of-the-art methods in terms of mAP and model size

We compare the performance of ML-AGCN with current state-of-the-art methods by
reporting the mean Average Precision (mAP) as well as the number of model
parameters. Additionally, similar to (Singh et al., 2022b; Chen et al.,
2019b), we report the following evaluation metrics on the MS-COCO dataset,
namely, average per-Class Precision (CP), average per-Class Recall (CR),
average per-Class F1-score (CF1), average Overall Precision (OP), average
overall recall (OR) and average Overall F1-score (OF1).

Table 1: Comparison with the state-of-the-art methods on the MS-COCO dataset.
(Best and second-best performances are indicated in bold and are underlined,
respectively).

Category | Method | Resolution | # params | mAP | CP | CR | CF1 | OP | OR | OF1
---|---|---|---|---|---|---|---|---|---|---
| Multi-Evidence (Ge et al., 2018) | 448x448 | 49.0 | - | 80.4 | 70.2 | 74.9 | 85.2 | 72.5 | 78.4
| SRN (Zhu et al., 2017) | 224x224 | 48.0 | 77.1 | 81.6 | 65.4 | 71.2 | 82.7 | 69.9 | 75.8
| ResNet101 (He et al., 2016) | 448x448 | 44.5 | 77.3 | 80.2 | 66.7 | 72.8 | 83.9 | 70.8 | 76.8
| MCAR (Gao and Zhou, 2021) | 448×448 | 44.9 | 83.8 | 85 | 72.1 | 78 | 88 | 73.9 | 80.3
| MCAR (Gao and Zhou, 2021) | 576x576 | 44.9 | 84.5 | 84.3 | 73.9 | 78.7 | 86.9 | 76.1 | 81.1
CNN | TResNet-L (Ridnik et al., 2021a) | 448x448 | 53.8 | 86.6 | 87.4 | 76.4 | 81.4 | 88.1 | 79.2 | 81.8
RNN | CNN-RNN (Wang et al., 2016) | 224x224 | 66.2 | 61.2 | - | - | - | - | - | -
| ML-GCN (1-layer) (Chen et al., 2019b) | 448x448 | 43.1 | 80.9 | 82.9 | 69.7 | 75.8 | 84.8 | 73.6 | 78.8
| IML-GCN (1-layer) (Singh et al., 2022b) | 448x448 | 29.5 | 81.3 | 81.3 | 72.2 | 76.0 | 86.7 | 77.9 | 82.1
| ML-GCN (Chen et al., 2019b) | 448x448 | 44.9 | 83.0 | 85.1 | 72.0 | 78.0 | 85.8 | 75.4 | 80.3
| A-GCN (Li et al., 2019) | 512x512 | 44.1 | 83.1 | 84.7 | 72.3 | 78.0 | 85.6 | 75.5 | 80.3
| F-GCN (Wang et al., 2020b) | - | 44.3 | 83.2 | 85.4 | 72.4 | 78.3 | 86.0 | 75.7 | 80.5
| CFMIC (Wang et al., 2022b) | 448×448 | 45.1 | 83.8 | 85.8 | 72.7 | 78.7 | 86.3 | 76.3 | 81.0
| SSGRL (Chen et al., 2019a) | 576x576 | 92.2 | 83.8 | 89.9 | 68.5 | 76.8 | 91.3 | 70.8 | 79.7
| FLNet (Sun et al., 2022) | - | 46.0 | 84.1 | 84.9 | 73.9 | 79.0 | 85.5 | 77.4 | 81.1
Graph-based | IML-GCN (Singh et al., 2022b) | 448x448 | 31.5 | 86.6 | 78.8 | 82.6 | 80.2 | 79.0 | 85.1 | 81.9
| MlTr-s (Cheng et al., 2022) | 384x384 | 33.0 | 83.9 | 82.8 | 75.5 | 77.3 | 83 | 78.5 | 79.9
| STMG (Wang et al., 2022a) | 384x384 | 197.0 | 84.3 | 85.8 | 72.7 | 78.7 | 86.7 | 76.8 | 81.5
| C-Tran (Lanchantin et al., 2021) | 576x576 | 120.0 | 85.1 | 86.3 | 74.3 | 79.9 | 87.7 | 76.5 | 81.7
| MlTr-m (Cheng et al., 2022) | 384x384 | 62.0 | 86.8 | 84 | 80.1 | 81.7 | 84.6 | 82.5 | 83.5
Transformer-based | Ml-Decoder (Ridnik et al., 2023) | 448x448 | 51.3 | 88.1 | - | - | - | - | - | -
Ours: ML-AGCN (1-layer) | 448×448 | 29.9 | 86.7 | 79.6 | 82.4 | 80.7 | 79.8 | 84.5 | 82.1
Ours: ML-AGCN (2-layers) | 448×448 | 35.9 | 86.9 | 86.2 | 78.3 | 81.7 | 87.2 | 80.7 | 83.8

Table 1, 2 and 3 report the quantitative comparison of the proposed approach
with respect to state-of-the-art methods on the MS-COCO, the VG-500, and the
VOC datasets, respectively. It can be clearly seen that our method achieves
competitive results as compared to existing methods in terms of mAP while
considerably reducing the model size. More specifically, similar to SSGRL, we
achieve the best mAP performance on VOC-2007 while reducing the number of
parameters from 92.2 to 35.8 million. Moreover, ML-AGCN reaches the second-
best mAP performance on MS-COCO and VG-500. On MS-COCO, the proposed approach
presents a slightly lower mAP as compared to ML-decoder with 86.9% against
88.1% but requires only 35.9 million parameters against 51.3. Similarly, ML-
AGCN only registers a decrease of 0.5% on VG-500 in terms of mAP as compared
to the best-performing approach C-Tran with a considerably lower number of
parameters (almost a quarter). Moreover, this is worth mentioning that ML-AGCN
outperforms the ML-GCN (Chen et al., 2019b) baseline by 3.9%, 6.3%, and 0.5%
in terms of mAP while keeping a comparable number of parameters.

In addition, we also report that the obtained performance when including only
one layer in the graph-subnet of ML-GCN (Chen et al., 2019b), IML-GCN (Singh
et al., 2022b) and ML-AGCN. The obtained results confirm the relevance of the
proposed adaptive learning. In fact, it can be seen that our method
outperforms existing graph-based methods under this setting. More
specifically, ML-AGCN achieves an improvement of 5.7% and 5.3% in terms of mAP
when compared to ML-GCN and IML-GCN, respectively, on the MS-COCO dataset.
Similarly, on the VG-500 benchmark, a significant increase of 20.7% is
recorded in terms of mAP in comparison to IML-GCN.

Table 2: Comparison with the state-of-the-art methods on the VG-500
dataset.(Best and second to the best performance is indicated in bold and
underline respectively).

Method | # params | mAP | CP | CR | CF1 | OP | OR | OF1
---|---|---|---|---|---|---|---|---
ResNet101 (He et al., 2016) | 44.5 | 30.9 | 39\. 1 | 25.6 | 31.0 | 61.4 | 35.9 | 45.4
ML-GCN (Chen et al., 2019b) | 44.9 | 32.6 | 42.8 | 20.2 | 27.5 | 66.9 | 31.5 | 42.8
TResNet-M (Ridnik et al., 2021a) | 29.5 | 33.6 | - | - | - | - | - | -
IML-GCN (Singh et al., 2022b) | 32.1 | 34.5 | - | - | - | - | - | -
SSGRL (Chen et al., 2019a) | 92.2 | 36.6 | - | - | - | - | - | -
KGGR (Chen et al., 2020) | 45.0 | 37.4 | 47.4 | 24.7 | 32.5 | 66.9 | 36.5 | 47.2
C-Tran (Lanchantin et al., 2021) | 120.0 | 38.4 | 49.8 | 27.2 | 35.2 | 66.9 | 39.2 | 49.5
Ours: ML-AGCN (1-layer) | 32.1 | 37.9 | 47.2 | 31.8 | 34.7 | 64.1 | 42.1 | 50.8
Ours: ML-AGCN (2-layers) | 37.4 | 37.1 | 47.3 | 24.7 | 29.0 | 67.6 | 38.1 | 48.7

Table 3: Comparison with the state-of-the-art methods on the VOC-2007
dataset.(Best and second to the best performance is indicated in bold and
underline respectively).

Methods | # params | mAP | aero | bike | bird | boat | bottle | bus | car | cat | chair | cow | table | dog | horse | mbike | person | plant | sheep | sofa | train | tv
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CNN-RNN (Wang et al., 2016) | 66.2 | 84.0 | 96.7 | 83.1 | 94.2 | 92.8 | 61.2 | 82.1 | 89.1 | 94.2 | 64.2 | 83.6 | 70 | 92.4 | 91.7 | 84.2 | 93.7 | 59.8 | 93.2 | 75.3 | 99.7 | 78.6
VeryDeep (Simonyan and Zisserman, 2015) | 138 | 89.7 | 98.9 | 95.0 | 96.8 | 95.4 | 69.7 | 90.4 | 93.5 | 96.0 | 74.2 | 86.6 | 87.8 | 96.0 | 96.3 | 93.1 | 97.2 | 70.0 | 92.1 | 80.3 | 98.1 | 87.0
ResNet101 (He et al., 2016) | 44.5 | 89.9 | 99.5 | 97.7 | 97.8 | 96.4 | 65.7 | 91.8 | 96.1 | 97.6 | 74.2 | 80.9 | 85 | 98.4 | 96.5 | 95.9 | 98.4 | 70.1 | 88.3 | 80.2 | 98.9 | 89.2
HCP (Wei et al., 2015) | 138.0 | 90.9 | 98.6 | 97.1 | 98.0 | 95.6 | 75.3 | 94.7 | 95.8 | 97.3 | 73.1 | 90.2 | 80.0 | 97.3 | 96.1 | 94.9 | 96.3 | 78.3 | 94.7 | 76.2 | 97.9 | 91.5
A-GCN (Li et al., 2019) | 44.1 | 94.0 | 99.4 | 98.5 | 98.6 | 98.0 | 80.8 | 94.7 | 97.2 | 98.2 | 82.4 | 95.5 | 86.4 | 98.2 | 98.4 | 96.7 | 98.9 | 84.8 | 96.6 | 84.4 | 98.9 | 93.7
F-GCN (Wang et al., 2020b) | - | 94.1 | 99.5 | 98.5 | 98.7 | 98.2 | 80.9 | 94.8 | 97.3 | 98.3 | 82.5 | 95.7 | 86.6 | 98.2 | 98.4 | 96.7 | 99.0 | 84.8 | 96.7 | 84.4 | 99.0 | 93.7
FLNet (Sun et al., 2022) | 46.0 | 94.4 | 99.6 | 98.1 | 98.9 | 97.9 | 84.6 | 95.3 | 96.2 | 96.5 | 85.6 | 96.1 | 87.2 | 97.7 | 98.6 | 97.0 | 98.1 | 86.5 | 97.4 | 86.5 | 98.8 | 90.8
TResNet-L (Ridnik et al., 2021a) | 53.8 | 94.6 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | -
CFMIC (Wang et al., 2022b) | 45.1 | 94.7 | 99.7 | 98.5 | 98.8 | 98.3 | 83.9 | 96.5 | 97.5 | 98.8 | 83.1 | 96.1 | 87.4 | 98.6 | 98.9 | 97.2 | 99.0 | 85.4 | 97.1 | 84.9 | 99.2 | 94.2
MCAR (Gao and Zhou, 2021) | 44.9 | 94.8 | 99.7 | 99.0 | 98.5 | 98.2 | 85.4 | 96.9 | 97.4 | 98.9 | 83.7 | 95.5 | 88.8 | 99.1 | 98.2 | 95.1 | 99.1 | 84.8 | 97.1 | 87.8 | 98.3 | 94.8
SSGRL (Chen et al., 2019a) | 92.2 | 95.0 | 99.7 | 98.4 | 98.0 | 97.6 | 85.7 | 96.2 | 98.2 | 98.8 | 82.0 | 98.1 | 89.7 | 98.8 | 98.7 | 97.0 | 99.0 | 86.9 | 98.1 | 85.8 | 99.0 | 93.7
Ours ML-AGCN (1-layer) | 29.5 | 94.5 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Ours ML-AGCN (2-layers) | 35.8 | 95.0 | 99.9 | 98.0 | 98.5 | 98.0 | 81.6 | 96.8 | 96.6 | 98.2 | 85.6 | 99.4 | 88.2 | 99.2 | 99.0 | 96.5 | 98.8 | 84.8 | 99.5 | 88.1 | 98.9 | 94.5

##### Ablation study

In order to analyze the impact of each adjacency matrix used in our approach,
namely, the attention-based matrix $\mathbf{B}$ and similarity-based matrix
$\mathbf{C}$, an ablation study is carried out as shown in Table 4. It can be
noted that by considering $\mathbf{B}$ in addition to $\mathbf{A}$ (without
threshold), an improvement of 5.1%, 20.5%, and 0.04% in terms of mAP is made,
respectively, on MS-COCO, VG-500, and VOC-2007. The magnitude of improvement
seems dependent on the number of classes contained in the considered dataset.
In fact, while VG-500 is formed by 500 classes, MS-COCO and VOC-2007 are
respectively composed of 80 and 20 categories. This highlights the importance
of adaptively modeling label correlations, especially when dealing with a
large number of classes, which is likely to occur in a practical scenario. An
additional mAP improvement of 0.1%, 0.4%, and 0.17% can be seen when including
$\mathbf{C}$ in the AGCN subnet. In conclusion, it can be noted that
$\mathbf{B}$ contributes more importantly to the resulting enhancement. This
could be explained by the fact that $\mathbf{C}$ is less needed, as only two
layers are considered in the graph subnet. This would confirm that the initial
feature similarity is not completely lost through layers.

Table 4: The ablation of adaptively learning $\mathbf{B}$ and $\mathbf{C}$)
for multi-label image classification.

Methods/Dataset (mAP %) | MS-COCO | VG-500 | VOC-2007
---|---|---|---
ML-AGCN (A) | 81.1 | 17 | 94.27
ML-AGCN (A+B) | 86.6 (+5.1%) | 37.5 (+20.5%) | 94.31 (+0.04%)
ML-AGCN (A+B+C) | 86.7 (0.1%) | 37.9 (+0.4%) | 95.0 (+0.69%)

####  6.1.4 Qualitative analysis

In Fig. 6, the Gradient-weighted Class Activation Mapping (Grad-CAM)
(Selvaraju et al., 2017) is visualized for some examples from the VOC dataset.
While the first column of images represents the input images, the second,
third, and fourth columns show, respectively, the Grad-cam visualization from
a model trained using only $\mathbf{A}$, $\mathbf{A}$ and $\mathbf{B}$, and
finally $\mathbf{A}$, $\mathbf{B}$ and $\mathbf{C}$. These qualitative results
are in line with the quantitative ones. As shown in Fig. 6, the use of
$\mathbf{B}$ allows activating more precisely the regions of interest in the
image, while the contribution of $\mathbf{C}$ in this refinement is less
impressive but remains visible.

Fig. 6: Grad-CAM visualization of the predictions using samples from the VOC
dataset using only $\mathbf{A}$, then $\mathbf{A}$ and $\mathbf{B}$, and
finally $\mathbf{A}$, $\mathbf{B}$ and $\mathbf{C}$.

###  6.2 Domain Adaptation for Multi-label Image Classification

####  6.2.1 Implementation details

We reproduce the results of current state-of-the-art methods due to the
limited availability of DA approaches for multi-label image classification. In
particular, we first consider standard multi-label image classification
methods (without DA) and refer to them as MLIC. Since no target images were
employed during the learning process, this is equivalent to source-only
training. Second, we reproduce the outcomes of two state-of-the-art domain
adaptation methods, namely DANN (Ganin et al., 2016) and DA-MAIC (Lin et al.,
2021), that we refer as DA in our experiments. Moreover, we replace the multi-
label softmargin-loss by the traditional cross-entropy loss when training
DANN. We generate Glove-based word embeddings (Pennington et al., 2014) as
node features for training DA-MAIC. Additionally, in order to showcase the
effectiveness of the proposed AGCN subnet, we present the results of DANN and
DA-MAIC by considering the same backbone as ours: an additional experiment
based on TResNet-M instead of the traditional ResNet101 is carried out. The
images have been resized to 224 $\times$ 224, unless stated differently. Our
domain classifier includes one hidden layer of dimension 1024. A maximum
learning rate of $1e-4$ using a cosine decay is considered. The model is
trained for a total of 40 epochs or until convergence.

Table 5: Comparison with the state-of-the-art in terms of mAP and number of
model parameters using two settings, i.e., AID $\rightarrow$ UCM and UCM
$\rightarrow$ AID. (Best performance is indicated in bold and second-best
performance is undelined).

Category | Backbone | Method | # params | AID $\rightarrow$ UCM | UCM $\rightarrow$ AID
---|---|---|---|---|---
mAP | P_C | R_C | F_C | P_O | R_O | F_O | mAP | P_C | R_C | F_C | P_O | R_O | F_O
MLIC | ResNet50 | RESNET (He et al., 2016) | 23.5 | 54.5 | 57.9 | 42.3 | 43.2 | 70.2 | 67.1 | 68.6 | 50.3 | 57.2 | 26.8 | 32.2 | 90.0 | 44.2 | 59.3
ResNet101 | RESNET (He et al., 2016) | 42.5 | 57.5 | 60.0 | 47.5 | 47.0 | 69.1 | 71.5 | 70.3 | 51.7 | 50.6 | 29.6 | 33.9 | 88.0 | 48.5 | 62.5
ResNet101 | ML-GCN (Chen et al., 2019b) | 44.9 | 53.7 | 55.3 | 44.3 | 45.9 | 70.2 | 68.7 | 69.4 | 51.3 | 50.1 | 29.9 | 34.0 | 88.0 | 49.7 | 63.6
TResNet-M | ASL (Ridnik et al., 2021a) | 29.4 | 55.4 | 48.7 | 52.8 | 47.1 | 58.7 | 79.1 | 67.4 | 54.1 | 54.5 | 40.2 | 41.9 | 85.4 | 65.1 | 73.9
TResNet-M | ML-AGCN (Singh et al., 2022a) | 36.6 | 55.2 | 36.6 | 64.9 | 45.1 | 45.0 | 88.1 | 59.6 | 52.1 | 48.2 | 47.4 | 42.9 | 77.1 | 79.8 | 78.4
DA | ResNet101 | DANN (Ganin et al., 2016) | 42.5 | 55.3 | 54.7 | 57.9 | 52.8 | 59.0 | 81.6 | 68.5 | 50.5 | 60.4 | 24.9 | 32.3 | 89.2 | 42.2 | 57.3
DA-MAIC (Lin et al., 2021) | 44.9 | 49.7 | 52.4 | 48.7 | 45.4 | 56.9 | 72.5 | 63.8 | 48.7 | 51.6 | 40.9 | 41.5 | 78.9 | 65.5 | 71.6
TResNet-M | DANN (Ganin et al., 2016) | 29.4 | 52.5 | 59.1 | 31.6 | 36.3 | 70.9 | 53.7 | 61.1 | 51.6 | 52.1 | 23.2 | 27.9 | 83.2 | 27.8 | 41.7
DA-MAIC (Lin et al., 2021) | 31.8 | 54.4 | 55.3 | 37.5 | 38.6 | 68.0 | 67.9 | 67.9 | 50.5 | 51.8 | 22.9 | 29.0 | 91.6 | 35.2 | 50.8
DA-AGCN (Ours) | 36.6 | 59.0 | 54.0 | 59.4 | 52.3 | 57.1 | 82.3 | 67.4 | 56.1 | 55.7 | 42.0 | 43.5 | 82.6 | 64.5 | 72.4

Table 6: Comparison with the state-of-the-art in terms of mAP and number of
model parameters using two settings, i.e., UCM $\rightarrow$ DFC and AID
$\rightarrow$ DFC. (Best performance is indicated in bold and second-best
performance is underlined).

Category | Backbone | Method | # params | AID $\rightarrow$ DFC | UCM $\rightarrow$ DFC
---|---|---|---|---|---
mAP | P_C | R_C | F_C | P_O | R_O | F_O | mAP | P_C | R_C | F_C | P_O | R_O | F_O
MLIC | ResNet50 | RESNET (He et al., 2016) | 23.5 | 52.9 | 52.3 | 47.1 | 44.2 | 52.3 | 50.5 | 51.3 | 67.7 | 58.5 | 31.9 | 34.9 | 65.9 | 39.1 | 49.1
ResNet101 | RESNET (He et al., 2016) | 42.5 | 56.9 | 52.9 | 61.5 | 48.7 | 46.1 | 63.7 | 53.5 | 66.4 | 74.4 | 31.2 | 36.9 | 67.2 | 37.2 | 47.9
ResNet101 | ML-GCN (Chen et al., 2019b) | 44.9 | 58.9 | 56.7 | 57.9 | 45.8 | 45.7 | 65.0 | 53.7 | 64.6 | 72.4 | 32.0 | 35.6 | 64.4 | 38.9 | 48.5
TResNet-M | ASL (Ridnik et al., 2021a) | 29.4 | 56.1 | 49.6 | 68.4 | 49.9 | 43.5 | 74.1 | 54.8 | 68.9 | 66.3 | 53.1 | 44.0 | 52.6 | 57.0 | 54.7
TResNet-M | ML-AGCN (Singh et al., 2022a) | 36.6 | 51.6 | 41.5 | 83.8 | 52.3 | 40.2 | 88.7 | 55.3 | 70.3 | 68.4 | 56.1 | 47.8 | 53.8 | 58.5 | 56.0
DA | ResNet101 | DANN (Ganin et al., 2016) | 42.5 | 64.3 | 57.0 | 65.9 | 51.4 | 48.0 | 65.9 | 55.6 | 67.2 | 72.8 | 44.7 | 48.7 | 56.3 | 49.1 | 52.4
DA-MAIC (Lin et al., 2021) | 44.9 | 50.1 | 54.5 | 45.8 | 38.9 | 44.2 | 48.8 | 46.4 | 65.6 | 69.7 | 54.2 | 52.8 | 57.1 | 58.4 | 57.7
TResNet-M | DANN (Ganin et al., 2016) | 29.4 | 43.0 | 40.7 | 13.6 | 19.3 | 46.0 | 15.6 | 23.3 | 64.1 | 77.3 | 22.6 | 30.1 | 68.6 | 26.5 | 38.2
DA-MAIC (Lin et al., 2021) | 31.8 | 55.4 | 49.8 | 60.4 | 44.7 | 47.3 | 64.1 | 54.4 | 65.8 | 71.4 | 39.3 | 39.7 | 59.9 | 44.6 | 51.1
DA-AGCN (Ours) | 36.6 | 65.7 | 51.8 | 78.1 | 55.7 | 45.2 | 80.8 | 58.0 | 76.5 | 68.5 | 61.7 | 59.0 | 60.0 | 60.2 | 60.1

Table 7: Comparison with the state-of-the-art in terms of mAP and number of
model parameters using the two settings, i.e., VOC $\rightarrow$ Clipart and
Clipart $\rightarrow$ VOC. (Best performance is indicated in bold and second-
best performance is underlined).

Category | Backbone | Method | # params | VOC $\rightarrow$ Clipart | Clipart $\rightarrow$ VOC
---|---|---|---|---|---
mAP | P_C | R_C | F_C | P_O | R_O | F_O | mAP | P_C | R_C | F_C | P_O | R_O | F_O
MLIC | ResNet50 | RESNET (He et al., 2016) | 23.5 | 42.0 | 57.6 | 15.3 | 22.5 | 82.3 | 25.8 | 39.3 | 45.4 | 40.3 | 9.8 | 13.0 | 84.7 | 25.5 | 39.2
ResNet101 | RESNET (He et al., 2016) | 42.5 | 38.0 | 64.8 | 14.3 | 22.5 | 82.3 | 18.3 | 29.9 | 50.1 | 66.2 | 17.5 | 25.5 | 83.9 | 29.6 | 43.7
ResNet101 | ML-GCN (Chen et al., 2019b) | 44.9 | 43.5 | 62.5 | 20.3 | 28.4 | 86.6 | 27.8 | 42.1 | 43.1 | 57.9 | 21.0 | 26.8 | 73.5 | 30.6 | 43.2
TResNet-M | ASL (Ridnik et al., 2021a) | 29.4 | 56.8 | 72.0 | 38.5 | 47.6 | 82.8 | 45.7 | 58.9 | 64.2 | 69.0 | 30.7 | 37.3 | 80.0 | 45.7 | 58.2
TResNet-M | ML-AGCN (Singh et al., 2022a) | 36.6 | 53.7 | 75.5 | 35.5 | 44.4 | 79.1 | 39.9 | 53.1 | 38.0 | 45.5 | 25.1 | 28.2 | 61.8 | 36.6 | 45.9
DA | ResNet101 | DANN (Ganin et al., 2016) | 42.5 | 33.9 | 47.7 | 17.0 | 20.6 | 57.3 | 24.1 | 33.9 | 24.3 | 28.3 | 16.4 | 14.2 | 39.8 | 24.7 | 30.5
DA-MAIC (Lin et al., 2021) | 44.9 | 25.8 | 28.0 | 3.1 | 5.1 | 92.6 | 2.9 | 5.6 | 32.6 | 48.2 | 12.2 | 14.4 | 50.2 | 31.9 | 39.0
TResNet-M | DANN (Ganin et al., 2016) | 29.4 | 40.0 | 82.4 | 17.2 | 27.4 | 93.8 | 17.5 | 29.5 | 67.0 | 76.8 | 23.3 | 32.6 | 93.1 | 20.4 | 33.4
DA-MAIC (Lin et al., 2021) | 31.8 | 62.3 | 77.4 | 42.6 | 51.6 | 83.1 | 51.0 | 63.2 | 74.3 | 84.5 | 53.9 | 63.0 | 83.7 | 57.7 | 68.3
DA-AGCN (Ours) | 36.6 | 62.9 | 75.8 | 46.8 | 54.0 | 80.0 | 50.4 | 61.8 | 74.8 | 73.9 | 66.3 | 67.5 | 72.6 | 65.4 | 68.8

####  6.2.2 Datasets

Similar to DA-MAIC (Lin et al., 2021), we use three multi-label aerial image
datasets in our experiments, namely, AID (Hua et al., 2020), UCM (Chaudhuri et
al., 2018) and DFC15 (Hua et al., 2019). Additionally, due to the limited
number of suitable datasets for the task of DA in MLIC, we convert two well-
known object detection datasets, initially used for DA in the context of
object detection, to multi-label annotations, namely PASCAL VOC 2007
(Everingham et al., 2010) and Clipart1k (Inoue et al., 2018).

##### AID multi-label aerial dataset

The original AID dataset (Xia et al., 2017) contains 10000 high-resolution
aerial images. The images cover a total of 30 categories. A multi-label
version of this dataset was produced in (Hua et al., 2020), where 3000 aerial
images from the original AID dataset have been selected and assigned with
multiple object labels. In total, they include 17 labels: airplane, sand,
pavement, buildings, cars, chaparral, court, trees, dock, tank, water, grass,
mobile-home, ship, bare-soil, sea, and field. 80% and 20% of the images have
been used respectively for training and testing (Hua et al., 2020).

##### UCM multi-label aerial dataset

UCM multi-label dataset (Chaudhuri et al., 2018) is derived from the UCM
dataset (Yang and Newsam, 2010). It consists of images showing 21 land-use
classes. The images have a resolution of 256 x 256 pixels. Later in (Chaudhuri
et al., 2018), 2100 of these aerial images were annotated with multiple tags
in order to generate a multi-label aerial image dataset. This dataset shares
the same number of labels as AID multi-label dataset (Hua et al., 2020) i.e.,
17 labels. In our experiments, 80% and 20% of data are respectively used for
training and testing.

##### DFC15 multi-label aerial dataset

The DFC15 multi-label dataset (Hua et al., 2019) was initially introduced in
2015. It has a total of 3342 high-resolution image samples and includes 8
object labels. In our experiments, the 6 categories in common with UCM and AID
datasets, including water, grass, building, tree, ship, and car, are
considered. 80% and 20% are respectively used for training and testing.

##### VOC and Clipart1k datasets

The Clipart1k (Inoue et al., 2018) dataset contains 20 object categories,
similar to PASCAL-VOC 2007 (Everingham et al., 2010). We create a multi-label
annotation for each image by considering the category of each object bounding
box. Clipart1k provides a total of 1000 image samples. 50% and 50% are
respectively used for training and testing.

####  6.2.3 Quantitative analysis

##### Comparison with state-of-the-art methods in terms of mAP and model size

The proposed domain adaptation approach for multi-label image classification
is compared with state-of-the-art methods. The same metrics described in
Section 6.1.3 are used, including mAP, CP, CR, CF1, OP, OR and OF1. In
addition to the four protocols followed in DA-MAIC (Lin et al., 2021), i.e.,
AID $\rightarrow$ UCM, UCM $\rightarrow$ AID, AID $\rightarrow$ DFC, and UCM
$\rightarrow$ DFC, two more combinations VOC $\rightarrow$ Clipart and Clipart
$\rightarrow$ VOC are provided. Two categories of methods are considered in
our evaluation: (1) the conventional Multi-Label Image Classification (MLIC);
and the Domain Adaptation-based (DA) methods that aim at explicitly reducing
the gap between source and target datasets.

(a) UCM $\rightarrow$ AID

(b) AID $\rightarrow$ UCM

Fig. 7: Grad-CAM visualization of the predictions a) UCM $\rightarrow$ AID,
and b) AID $\rightarrow$ UCM setting using only $\mathbf{A}$, then
$\mathbf{A}$ and $\mathbf{B}$, and finally $\mathbf{A}$, $\mathbf{B}$ and
$\mathbf{C}$.

Table 5 reports the results obtained considering AID and UCM datasets. Two
different settings are followed: AID $\rightarrow$ UCM, where AID is the
source dataset and UCM is the target dataset, and UCM $\rightarrow$ AID, where
UCM is the source dataset and AID is the target dataset. It can be clearly
seen, in both types of DA settings, that the proposed DA-AGCN outperforms
existing state-of-the-art methods in terms of mAP while requiring a lower
number of parameters. More specifically, an improvement of mAP by 10.65% and
4.59% has been respectively recorded for AID $\rightarrow$ UCM and UCM
$\rightarrow$ AID in terms of mAP as compared with the DA-MAIC (Lin et al.,
2021) baseline. This suggests that learning the graph topology in an end-to-
end manner helps improve the performance in the presence of a domain shift.

In Table 6, DA-AGCN is also compared with existing methods by considering the
settings UCM $\rightarrow$ DFC and AID $\rightarrow$ DFC. It can be clearly
noticed that DA-AGCN outperforms the existing works in terms of mAP and model
size for both UCM $\rightarrow$ DFC and AID $\rightarrow$ DFC. More precisely,
an improvement of 1.4% and 6.2% is achieved in terms of mAP compared to the
second-best performing methods.

Table 7 also confirms the superiority of the proposed approach as compared to
other state-of-the-art techniques. However, it is to note that the improvement
given by the adaptive graph remains limited. This might be explained by the
fact that DA for MLIC datasets include a relatively low number of classes (8
to 20 categories). Hence, the interest of adaptively modeling label
correlations might be not entirely visible. This demonstrates the necessity of
creating benchmarks for DA under an MLIC context, including a wider number of
classes.

##### Ablation study

Table 8 reports the obtained mAP when considering each module of DA-AGCN. The
results confirm that adding both an AGCN subnet and an adversarially trained
domain classifier (DC) enhances the performance. Additionally, in Table 9, the
performance when discarding the proposed adjacency matrices $\mathbf{B}$ and
$\mathbf{C}$ is reported using UCM, AID and DFC datasets. The obtained results
confirm the relevance of using the proposed attention-based and similarity-
based adjacency matrices for modeling label correlations.

Table 8: Ablation study: impact of using an adversarial strategy and learning
an adaptive graph topology.

Methods/Dataset (mAP) | UCM $\rightarrow$ AID | AID $\rightarrow$ UCM | AID $\rightarrow$ DFC | UCM $\rightarrow$ DFC
---|---|---|---|---
TResNet only | 34.98 | 37.03 | 62.36 | 54.77
TResNet + AGCN | 53.62 (+18.64) | 53.08 (+16.05) | 58.44 (-3.92) | 74.4 (+19.68)
TResNet + AGCN + DC | 55.79 (+2.17) | 55.37 (+2.29) | 65.11 (+6.67) | 76.49 (+2.04)

Table 9: Ablation study: effect of adaptively learning $\mathbf{B}$ and
$\mathbf{C}$ in the presence of a domain shift.

Methods/Dataset (mAP %) | UCM $\rightarrow$ AID | AID $\rightarrow$ UCM | AID $\rightarrow$ DFC | UCM $\rightarrow$ DFC
---|---|---|---|---
A | 42.82 | 33.64 | 42.56 | 58.26
A+B | 55.35 (+12.53) | 45.14 (+11.5) | 51.47 (+8.91) | 67.86 (+9.6)
A+B+C | 55.79 (+0.44) | 55.37 (+10.23) | 65.11 (+13.64) | 76.49 (+8.63)

####  6.2.4 Qualitative analysis

Grad-CAM (Selvaraju et al., 2017) visualization in the presence of a domain
shift can be seen in Fig. 7. While Fig. 7 (a) demonstrate these results for
UCM $\rightarrow$ AID, Fig. 7 (b) showcases the visualizations for AID
$\rightarrow$ UCM. The left-most column is the input image, and the next three
consecutive columns represent the Grad-CAM visualization of the image using a
model trained with; $\mathbf{A}$, $\mathbf{A}$ and $\mathbf{B}$, and
$\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$ respectively. It can be clearly seen
that adaptively learning the graph topology $\mathbf{B}$ and $\mathbf{C}$
helps activate the most relevant areas of interest, leading to better
classification performance.

####  6.2.5 Failure cases

In Fig. 8 , some failure cases are presented using the Grad-CAM visualization
with $\mathbf{A}$, $\mathbf{A}$ and $\mathbf{B}$ and $\mathbf{A},\mathbf{B}$
and $\mathbf{C}$. It can be noticed that when the targeted object occupies
most of the image, the use of $\mathbf{B}$ and $\mathbf{C}$ makes the model
confuse the object with the background. A good illustration of this can be
seen in the first row of Fig. 8. The water is confused with the background. In
future work, modeling the object occupancy will be investigated for handling
these failure cases.

Fig. 8: Failure cases: Grad-CAM visualization for AID $\rightarrow$ DFC
showcasing performance degradation with adaptive graphs.

##  7 Conclusion

Existing graph-based methods have shown great performances for multi-label
image classification in the context of both single-domain and cross-domain.
However, these methods mostly fix the graph topology heuristically while
discarding edges with rare co-occurrences. Furthermore, it has been
demonstrated in (Jin et al., 2021) that successive GCN aggregations tend to
destroy the initial feature similarity. Hence, as a solution, an adaptive
strategy for learning the graph in an end-to-end manner is proposed. In
particular, attention-based and similarity-preserving mechanisms are adopted.
The proposed framework for multi-label classification in a single domain is
then extended to multiple domains. For that purpose, an adversarial domain
adaptation strategy is employed. The results performed for both single and
cross-domain support the effectiveness of the proposed method in terms of
model performance and size as compared to recent state-of-the-art methods.

## Acknowledgments

This work was funded by the Luxembourg National Research Fund (FNR), under the
project reference BRIDGES2020/IS/14755859/MEETA/Aouada.

## References

  * Cai et al. (2019) Cai, Y., Ge, L., Liu, J., Cai, J., Cham, T.J., Yuan, J., Thalmann, N.M., 2019.  in: In Proceedings of the IEEE/CVF international conference on computer vision, Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks. pp. 2272–2281.
  * Chaudhuri et al. (2018) Chaudhuri, B., Demir, B., Chaudhuri, S., Bruzzone, L., 2018.  “multilabel remote sensing image retrieval using a semisupervised graph-theoretic method,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 2.  pp 1144.
  * Chen et al. (2020) Chen, T., Lin, L., Hui, X., Chen, R., Wu, H., 2020.  Knowledge-guided multi-label few-shot learning for general image recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * Chen et al. (2019a) Chen, T., Xu, M., Hui, X., Wu, H., Lin, L., 2019a.  Learning semantic-specific graph representation for multi-label image recognition, in: In Proceedings of the IEEE/CVF international conference on computer vision, pp. 522–531.
  * Chen et al. (2019b) Chen, Z.M., Wei, X.S., Wang, P., Guo, Y., 2019b.  Multi-label image recognition with graph convolutional networks, in: In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5177–5186.
  * Cheng et al. (2022) Cheng, X., Lin, H., Wu, X., Shen, D., Yang, F., Liu, H., Shi, N., 2022.  Mltr: Multi-label classification with transformer, in: 2022 IEEE International Conference on Multimedia and Expo (ICME), IEEE. pp. 1–6.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.., 2009.  June), in: InIEEE conference on computer vision and pattern recognition, A large-scale hierarchical image database, Imagenet. pp. 248–255.
  * Everingham et al. (2010) Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A., 2010.  The pascal visual object classes (voc) challenge.  International journal of computer vision 88, 303–338.
  * Ganin and Lempitsky (2015) Ganin, Y., Lempitsky, V., 2015.  June. unsupervised domain adaptation by backpropagation, in: International conference on machine learning, PMLR. pp. 1180–1189.
  * Ganin et al. (2016) Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V., 2016.  Domain-adversarial training of neural networks.  The journal of machine learning research 17, 2096–2030.
  * Gao and Zhou (2021) Gao, B.B., Zhou, H.Y., 2021.  Learning to discover multi-class attentional regions for multi-label image recognition.  IEEE Transactions on Image Processing 30, 5920–5932.
  * Ge et al. (2018) Ge, W., Yang, S., Yu, Y., 2018.  Multi-evidence filtering and fusion for multi-label classification, in: In Proceedings of the IEEE conference on computer vision and pattern recognition, object detection and semantic segmentation based on weakly supervised learning. pp. 1277–1286.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Hua et al. (2019) Hua, Y., Mou, L., Zhu, X.X., 2019.  “recurrently exploring class-wise attention in a hybrid convolutional and bidirectional lstm network for multi-label aerial image classification,”.  ISPRS J. Photogramm. Remote Sens 149.
  * Hua et al. (2020) Hua, Y., Mou, L., Zhu, X.X., 2020.  Relation network for multilabel aerial image classification.  IEEE Transactions on Geoscience and Remote Sensing 58, 4558–4572.
  * Inoue et al. (2018) Inoue, N., Furuta, R., Yamasaki, T., Aizawa, K., 2018.  Cross-domain weakly-supervised object detection through progressive domain adaptation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5001–5009.
  * Jin et al. (2021) Jin, W., Derr, T., Wang, Y., Ma, Y., Liu, Z., Tang, J., 2021.  March. node similarity preserving graph convolutional networks, in: In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp. 148–156.
  * Kipf and Welling (2017) Kipf, T.N., Welling, M., 2017.  Semi-Supervised Classification with Graph Convolutional Networks.  ICLR.
  * Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., Hinton, G., 2012.  “imagenet classification with deep convolutional neural networks,”.  in Proc. Neural Inf. Process. Syst 1106.
  * Lanchantin et al. (2021) Lanchantin, J., Wang, T., Ordonez, V., Qi, Y., 2021.  General multi-label image classification with transformers, in: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16478–16488.
  * Li et al. (2021) Li, G., Ji, Z., Chang, Y., Li, S., Qu, X., Cao, D., 2021.  Ml-anet: A transfer learning approach using adaptation network for multi-label image classification in autonomous driving.  Chinese Journal of Mechanical Engineering 34, 1–11.
  * Li et al. (2020) Li, M., Zhai, Y.M., Luo, Y.W., Ge, P.F., Ren, C.X., 2020.  Enhanced transport distance for unsupervised domain adaptation, in: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13936–13944.
  * Li et al. (2019) Li, Q., Peng, X., Qiao, Y., Peng, Q., 2019.  Learning category correlations for multi-label image recognition with graph networks.  arXiv preprint arXiv:1909.13005 .
  * Li et al. (2016) Li, Y., Huang, C., Loy, C.C., Tang, X., 2016.  October. human attribute recognition by deep hierarchical contexts, in: In European conference on computer vision . , Cham, pp. 684–700.
  * Lin et al. (2021) Lin, D., Lin, J., Zhao, L., Wang, Z.J., Chen, Z., 2021.  Multilabel Aerial Image Classification With Unsupervised Domain Adaptation.  IEEE Transactions on Geoscience and Remote Sensing.
  * (26) Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., .  … and zitnick, in: C. L. (2014, September). Microsoft coco: Common objects in context. In European conference on computer vision . , Cham, pp. 740–755.
  * Long et al. (2018) Long, M., Cao, Z., Wang, J., Jordan, M.I., 2018.  Conditional adversarial domain adaptation.  Advances in neural information processing systems 31.
  * Long et al. (2017) Long, M., Zhu, H., Wang, J., Jordan, M.I., 2017.  July. deep transfer learning with joint adaptation networks, in: In International conference on machine learning, PMLR. pp. 2208–2217.
  * Mirza and Osindero (2014) Mirza, M., Osindero, S., 2014.  Conditional generative adversarial nets.  arXiv:1411.1784. arXiv preprint.
  * Papadopoulos et al. (2020a) Papadopoulos, K., Ghorbel, E., Aouada, D., Ottersten, B.., 2020a.  January), in: In25th International Conference on Pattern Recognition (ICPR), Vertex feature encoding and hierarchical temporal modeling in a spatio-temporal graph convolutional network for action recognition. pp. 452--458.
  * Papadopoulos et al. (2020b) Papadopoulos, K., Ghorbel, E., Oyedotun, O., Aouada, D., Ottersten, B., 2020b.  Deepvi: A novel framework for learning deep view-invariant human action representations using a single rgb camera, in: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), IEEE. pp. 138--145.
  * Pennington et al. (2014) Pennington, J., Socher, R., Manning, C.D., 2014.  October, in: In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), Global vectors for word representation, Glove. pp. 1532--1543.
  * Pham et al. (2021) Pham, D.D., Koesnadi, S.M., Dovletov, G., Pauli, J.., 2021.  April), in: Adversarial, U. (Ed.), InIEEE 18th International Symposium on Biomedical Imaging (ISBI), Domain Adaptation for Multi-Label Classification of Chest X-Ray. pp. 1236--1240.
  * Qu et al. (2023) Qu, X., Che, H., Huang, J., Xu, L., Zheng, X., 2023.  Multi-layered semantic representation network for multi-label image classification.  International Journal of Machine Learning and Cybernetics , 1--9.
  * Razavian et al. (2014) Razavian, S., A., A., H., S., 2014.  J, in: In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, an astounding baseline for recognition, and Carlsson, S. CNN features off-the-shelf. pp. 806--813.
  * Ridnik et al. (2021a) Ridnik, T., Ben-Baruch, E., Zamir, N., Noy, A., Friedman, I., Protter, M., Zelnik-Manor, L., 2021a.  Asymmetric loss for multi-label classification, in: In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 82--91.
  * Ridnik et al. (2021b) Ridnik, T., Lawen, H., Noy, A., Baruch, B., E., S., 2021b.  G, in: Tresnet: High performance gpu-dedicated architecture.‘‘ In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, and Friedman, I. pp. 1400--1409.
  * Ridnik et al. (2023) Ridnik, T., Sharir, G., Ben-Cohen, A., Ben-Baruch, E., Noy, A., 2023.  Ml-decoder: Scalable and versatile classification head, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 32--41.
  * Selvaraju et al. (2017) Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D., 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization, in: In Proceedings of the IEEE international conference on computer vision, pp. 618--626.
  * Sermanet et al. (2014) Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y., 2014.  Overfeat: Integrated recognition, localization and detection using convolutional networks.  In ICLR.
  * Shao et al. (2015) Shao, J., Kang, K., Loy, C., 2015.  C. and wang, in: Deeply, X. (Ed.), In Proceedings of the IEEE conference on computer vision and pattern recognition, learned attributes for crowded scene understanding. pp. 4657--4666.
  * Simonyan and Zisserman (2015) Simonyan, K., Zisserman, A., 2015.  Very deep convolutional networks for large-scale image recognition, in: In Y. Bengio and Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015.
  * Singh et al. (2023) Singh, I., Mejri, N., Nygyen, V., Ghorbel, E. anf Aouada, D., 2023.  Multi-type deepfake detection, in: MMSP.
  * Singh et al. (2022a) Singh, I.P., Ghorbel, E., Oyedotun, O., Aouada, D., 2022a.  Multi label image classification using adaptive graph convolutional networks (ml-agcn), in: In IEEE International Conference on Image Processing.
  * Singh et al. (2022b) Singh, I.P., Oyedotun, O., Ghorbel, E., Aouada, D., 2022b.  Iml-gcn: Improved multi-label graph convolutional network for efficient yet precise image classification.  In AAAI-22 Workshop Program-Deep Learning on Graphs: Methods and Applications .
  * Sun et al. (2022) Sun, D., Ma, L., Ding, Z., Luo, B., 2022.  An attention-driven multi-label image classification with semantic embedding and graph convolutional networks.  Cognitive Computation , 1--12.
  * Velickovic et al. (2017) Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y., 2017.  Graph attention networks.  stat 1050.
  * Wang et al. (2016) Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., Xu, W., 2016.  Cnn-rnn: A unified framework for multi-label image classification, in: In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2285--2294.
  * Wang et al. (2020a) Wang, Y., He, D., Li, F., Long, X., Zhou, Z., Ma, J., Wen, S., 2020a.  April, in: In Proceedings of the AAAI Conference on Artificial Intelligence, Multi-label classification with label graph superimposing. (Vol. 34, No. 07. pp. 12265--12272.
  * Wang et al. (2022a) Wang, Y., Xie, Y., Fan, L., Hu, G., 2022a.  Stmg: Swin transformer for multi-label image recognition with graph convolution network.  Neural Computing and Applications 34, 10051--10063.
  * Wang et al. (2020b) Wang, Y., Xie, Y., Liu, Y., Zhou, K., Li, X., 2020b.  Fast graph convolution network based multi-label image recognition via cross-modal fusion, in: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 1575--1584.
  * Wang et al. (2022b) Wang, Y., Xie, Y., Zeng, J., Wang, H., Fan, L., Song, Y., 2022b.  Cross-modal fusion for multi-label image classification with attention mechanism.  Computers and Electrical Engineering 101, 108002.
  * Wei et al. (2015) Wei, Y., Xia, W., Lin, M., Huang, J., Ni, B., Dong, J., Zhao, Y., Yan, S., 2015.  Hcp: A flexible cnn framework for multi-label image classification.  IEEE transactions on pattern analysis and machine intelligence 38, 1901--1907.
  * Xia et al. (2017) Xia, G.S., Hu, J., Hu, F., Shi, B., Bai, X., Zhong, Y., Zhang, L., Lu, X., 2017.  Aid: A benchmark data set for performance evaluation of aerial scene classification.  IEEE Transactions on Geoscience and Remote Sensing 55, 3965--3981.
  * Yang and Newsam (2010) Yang, Y., Newsam, S., 2010.  “bag-of-visual-words and spatial extensions for land-use classification, in: ” in International Conference on Advances in Geographic Information Systems (SIGSPATIAL).
  * Zhang et al. (2019) Zhang, Y., Liu, T., Long, M., Jordan, M., 2019.  May. bridging theory and algorithm for domain adaptation, in: In International Conference on Machine Learning, PMLR. pp. 7404--7413.
  * Zhu et al. (2017) Zhu, F., Li, H., Ouyang, W., Yu, N., Wang, X., 2017.  Learning spatial regularization with image-level supervisions for multi-label image classification, in: In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5513--5522.