11institutetext: Lihi Dery 22institutetext: Ariel University, Ariel. Israel,
22email<EMAIL_ADDRESS>
# Multi-label Ranking: Mining Multi-label and Label Ranking Data

Lihi Dery

###### Abstract

We survey multi-label ranking tasks, specifically multi-label classification
and label ranking classification. We highlight the unique challenges, and re-
categorize the methods, as they no longer fit into the traditional categories
of transformation and adaptation. We survey developments in the last demi-
decade, with a special focus on state-of-the-art methods in deep learning
multi-label mining, extreme multi-label classification and label ranking. We
conclude by offering a few future research directions.

##  1 Introduction

Multi-label ranking (MLR) is the problem of predicting and ranking multiple
labels for a single instance. The predicted labels are known as the instance’s
labelset. MLR can be typically reduced to two sub-problems: The first is
multi-label classification, where the task is to bipartite the data into
relevant labels (the labelset) and irrelevant labels. The second is label
ranking classification, where the task is to rank labels for each instance. A
label ranking may contain ties; in the extreme case relevant labels hold a tie
on first place, and irrelevant labels hold a tie on second place, thus turning
the label ranking classification into a multi-label one.

The first studies of MLR originated from text categorization problems, where
each document was labeled with several predefined topics joachims1998text ;
mccallum1999multi ; schapire2000boostexter . The labels were sometimes ranked
in order of importance ioannou2010 . While multi-label for text categorization
continues to be an active field mironczuk2018 ; mujtaba2019 , multi-label
ranking has spread to many more domains. In bioinformatics, a gene can belong
to multiple functional families clare2001 . In music, a tune can spark many
emotions trohidis2008multi . In medical diagnosis, an x-ray image can have
multiple labels baltruschat2019comparison . In social networks, people may
belong to several interest groups wang2013multi  and in visual object
recognition, objects can be ordered according to their relevance to the
picture bucak2009efficient ; yang2016exploit .

There are six common challenges in MLR that either do not exist in single-
label classification or are intensified in MLR settings.

  * •

High dimensionality in the output space. High dimensionality in the input
space (i.e., data with millions of instances) or in the feature space (i.e.,
data with thousands or millions of features) is also common in single-label
classification, though in MLR it might be harder to solve. However, high
dimensionality in the output space is unique to MLR. The number of possible
labelsets (i.e. label combinations) grows exponentially with the number of
labels. This often leads to sparseness of available data and to class
imbalance, as some labelsets may appear often, while others may be rare or may
not appear in the training set at all. See an example in figure 1.

  * •

Label correlation. This aspect is fundamental in MLR. If there are no
relations between the labels, the problem can be split into multiple binary
classification problems without loss of information. However, in MLR, the
relation between the labels is complex. For example, if two labels have a high
concurrence, the model is supposed to somehow boost the prediction of one
label, if the other is predicted. If two labels have a parallel relation,
i.e., they do not concur, the model is expected to handle that as well. See an
example of label concurrence in figure 2.

  * •

Label imbalance. The label distribution is highly skewed, most labels have
only a handful of positive training instances and a few labels dominate with
many training instances. See an example in figure 3.

  * •

Labelset size imbalance. The labelset size of each instance is highly skewed
rubin2012statistical . Few instances have much more labels than average, while
most instances have very few labels. See an example in figure 4.

  * •

Label importance. Not all labels are equally important to the characterization
of the instance. The label importance is explicitly known if the target class
input labelset is ranked (i.e, if a ranked labels are provided as input in the
training data). Otherwise, the label importance remains to be inferred.

  * •

Zero-shot labels and labelsets. Some labels and labelsets never appear in the
training set.

Figures 1 \- 4, were created using the mldrGUI charte-charte:2015  with the
Genebase dataset diplaris2005protein  as an example. The dataset contains the
classification of proteins into families with similar function. Each instance
is a protein, the attributes are the protein’s motifs. The labels are the
families the protein belongs to. This is a multi-label setting since each
protein can belong to more than one family. In it’s current online version
mulan , the dataset contains 662 instances (the proteins) with 1213 attributes
each. There are 27 possible labels (families) and 32 possible labelsets.

Previous literature reviews on multi-label methods gibaja2014multi ;
gibaja2015tutorial ; herrera2016multilabel ; zhang2013review  and label
ranking methods vembu2010label ; zhou2014taxonomy , while excellent, are
slightly outdated. Our contributions in this survey are three-fold. First, we
refresh the definition of MLR brinker2006unified  and define it’s two sub-
tasks: multi-label classification and label ranking classification. Second, we
suggest to re-categorize MLR methods, as they no longer fit into the
traditional categories of transformation or adaptation. We thus suggest new
categories such as deep learning multi-label methods, extreme multi-label
methods and label ranking methods. Third, we focus on the last demi-decade
which has not yet been surveyed.

The rest of this survey is organized as follows: we first define MLR and place
it in context with other tasks (section 2). Next,we survey recent developments
in multi-label methods, with a special focus on deep learning methods and the
emerging field of extreme multi-label classification (section 3). We then move
on to discuss label ranking (section 4). We present up to date information
about evaluation (section 5) and conclude by offering a few research
directions on open problems (section 6).

Figure 1: High dimensionality in the output space. The number of instances
(y-axis) with a given labelset (x-axis) in the Genebase dataset. Figure 2:
Label correlation: a chord diagram gu2014circlize  showing the concurrence of
13 labels in the Genebase dataset. The arcs represent label concurrence For
example, the protein to the right of 12 o’clock (PDO-0196) co-occurs with only
one other protein (PDO-0199) and that happens less frequently than other
concurrences. Figure 3: Label imbalance. The number of instances (y-axis) with
a given label (x-axis) in the Genebase dataset. Figure 4: Labelset size
imbalance. The number of instances (y-axis) with a given number of labels
(x-axis) in the Genebase dataset.

##  2 Definition and context

We begin with a definition of MLR. Next we place MLR in context by detailing
which problems can be seen as sub cases of MLR, and what MLR is a sub problem
of.

###### Definition 1

Multi-label ranking (MLR). An MLR task is characterized by
$\textbf{x}\in\mathcal{X}$ instances and $l\in L$ labels with the following
properties:

  1. 1.

$\mathcal{X}$ is finite and contains $n$ instances.

  2. 2.

$L$ is finite and contains $m$ labels.

  3. 3.

An ordered labelset $Y=[\,l_{1},...,l_{q}]\,$ with $q\leq|L|$ labels, contains
a subset of the $L$ possible labels. $l_{1}$ is the label with the highest
rank and $l_{q}$ is the label with the lowest rank.

  4. 4.

The labelset size is exponential to the amount of labels:
$Y\subseteq\mathbb{P}(L)$.

  5. 5.

Ties in the labelset ranking are allowed, and in some cases all of the labels
are tied in first place. A threshold $t\in{1,2,..,q}$ indicates the
partitioning of $Y$ into relevant and irrelevant labels. When $t=1$ only the
first label in $Y$ is relevant. When $t=q$, or when the threshold is not
mentioned, all labels in $Y$ are relevant. When $1\leq t\leq q$ the labelset
is bipartite according to $t$.

  6. 6.

The training dataset $\mathcal{D}$ consists of a triplets
$\\{x_{i},Y_{i},t_{i}\\}$

  7. 7.

The goal is to find a mapping function: $h:\mathcal{X}\to Y$ for a given $t$.

In multi-label ranking problems the labelset is bipartite, i.e., the instance
belongs to a ranked subset of the labels, and doesn’t belong to the rest of
the set. For example, this image contains mainly oranges, apples and bananas
in this order, but no pears. Several problems are special cases of multi-label
ranking: binary classification, multi-class classification, multi-label
classification and label ranking. In binary classification, an instance can
belong to one of two possible classes, e.g., this image contains either an
apple or an orange. In multi-class problems, an instance can belong to one out
of multiple possible classes, e.g. this image contains either an apple, an
orange or a banana. In multi-label problems, an instance can belong to many
classes (labels), e.g. this image contains an apple and an orange but not a
banana. In label ranking problems, an instance belongs to a ranked set of
classes (labels), e.g. this image contains oranges, apples and bananas, in
this order.

Multi-label ranking tasks, as defined in definition 1, refer to any problem
whose target class output is a ranked list of labels and a threshold.
Algorithms for solving these tasks are often divided into two sub-groups
according to their target class input. When the input is a ranked set of
labels, it is a label ranking task. When the input is just a set of labels, it
is a multi-label task. Formally:

###### Definition 2

A multi-label (ML) task is an MLR with $1\leq t\leq q$. The labelset is
bipartite according to $t$, with labels $1\leq t$ tied in first place and
considered as relevant.

###### Definition 3

A label ranking (LR) task is an MLR with $t=q$ and no ties in Y.

Table 1 summarizes the differences between MLR and it’s sub-tasks according to
a few parameters:

  * •

Is $m>2$? - There may be many labels available for each instance, or just two.

  * •

Is $q>1$? - It may be possible to assign many labels for each instance, or
just one.

  * •

Are there ties in $Y$? - Is the labelset completely ordered, or can ties
between labels exist?

  * •

Does $t=q$? - Are all labels in $Y$ relevant, or does the threshold $t$
partitions the labels into relevant and irrelevant labels?

Table 1: Table of sub-cases of multi-label ranking |  | $m>2$
---
| $q>1$
---
| ties in $Y$
---
| $t=q$
---
Multi-label Ranking (MLR) | yes | yes | yes | yes
Label ranking (LR) | yes | yes | yes | no
Multi-label (ML) | yes | yes | no | yes
Multi-class (MC) | yes | no | no | yes
Binary classification | no | no | no | no

A recent survey categorizes multi-label mining as a sub-problem in multi-
target learning waegeman2019multi . Related problems in the multi-target
domain, that are not covered in this survey, include:

  * •

Multi variate regression borchani2015  and dyadic prediction where the goal is
to predict a score for the fit between the instance and the label.

  * •

Hierarchical multi-label cerri2011hierarchical  where there is explicit side
information about the dependencies between the labels.

  * •

Multi-instance herrera2016multiple  and multi-instance multi-label
zhou2012multi ; huang2018fast  where the training data is composed of a bag of
instances that are all assigned the same label or labels.

  * •

Multi-view zhao2017multi  is similar to multi-instance but the instances may
have different feature spaces.

##  3 Multi-label algorithms

To date, surveys classify multi-label algorithms as either problem
transformation techniques or algorithm adaptation techniques
charte2019snapshot ; gibaja2015tutorial ; zhang2013review . In the first, the
problem is transformed into a simpler single label classification task. In the
second, an algorithm used for single label classification is adapted to
perform multi-label tasks. Sometimes ensembles techniques are added as a third
class of problems and other times they are listed by their underlying base
classifier’s category (transformation or adaptation).

However, in the last decade algorithms that are specially designed for multi-
label tasks have emerged. These algorithms either: try to maximize a specific
evaluation measure (e.g. petterson2011submodular ; pillai2017designing ;
wu2017progressive ), focus on a certain sub-task (e.g., feature selection
pereira2018categorizing ; spolaor2016systematic ; zhang2019distinguishing ),
or attempt to address specific multi-label challenges such as high
dimensionality output space, label correlation (e.g.gu2011correlated ;
huang2012multi ), imbalance or zero-shot (e.g. rios2018few ).

Moreover, deep learning algorithms designed specifically for multi-label tasks
have been rapidly developing, exhibiting promising results. Indeed, a minority
of these algorithms can be classified as algorithm transformation techniques
(see e.g. the method suggested in yang2016exploit ), but more often, the
algorithm is not adapted but rather specially tailored for the problem at
hand.

As transformation and adaptation techniques have been exquisitely covered
barot2014review ; charte2019snapshot ; gibaja2015tutorial ;
herrera2016multilabel ; zhang2018binary ; zhang2013review  and as there is
only a minor increase in research on them in comparison with other aspects of
MLR, we lightly scan these foundations (sections 3.1 and 3.2), and then direct
our focus on methods from the last demi-decade, specifically on deep learning
(section 3.3) and extreme multi-label methods (section 3.4).

###  3.1 Problem adaptation and problem transformation

Problem adaptation techniques were originally suggested for text
categorization. However the adaptations soon expanded beyond the text domain
and into other scenarios as well. Some of the most noticed adaptations include
: Expectation maximization (EM) mccallum1999multi , SVM elisseeff2002kernel ;
joachims1998text ; xu2013multi ; xu2013fast , k-NN calvo2015improving ;
chiang2012ranking ; huang2017multi ; skryjomski2019speeding ; zhang2007ml ,
decision trees al2014lacova ; clare2001 , association rules thabtah2004mmac
and genetic algorithms gonccalves2018survey .

Problem transformation techniques barot2014review  focus on transforming the
problem into simpler sub-tasks. One classifier is created for each label or
pair of labels. The classifiers are then trained separately, and their output
is combined. The possible transformations are:

  * •

Binary Relevance. The transformation into a binary classification task is
known as binary relevance (BR) zhang2018binary . The first BR solutions
boutell2004learning ; godbole2004discriminative  did not consider label
correlation. However, many correlation-enabling extensions to binary relevance
have been proposed in the past decade. These correlations are classified into
three sub-classes: first order correlation, pairwise correlation, or full
correlation zhang2013review .

  * •

Multi-class. The most known multi-class transformations are the Label Powerset
methods that reduce the problem to a multi-class one by treating each
individual labelset as an independent class label boutell2004learning . In
both BR and multi-class transformations, there is a computational complexity
problem, as the solutions do not scale well as the number of labels increase.
Thus solutions that reduce the number of classifiers were suggested
madjarov2012two ; mencia2008efficient .

  * •

Pairwise label comparisons. Calibrated Label Ranking furnkranz2008multilabel
transform the dataset into pairs of labels and thus train $k(k−1)/2$ binary
classifiers. The output of the classifiers are combined into a ranking of the
output labels, with the highest ranked labels considered as relevant. A
fictional label can be used to automatically create a bi-partition of the
labels into relevant and irrelevant ones brinker2006unified .

###  3.2 Multi-label ensembles

The 2BR method tsoumakas2009correlation  uses BR twice and employs stacking.
It first learns a BR model, and then builds a second, meta-model that takes
the output of the first model and includes a explicit coefficient for
correlated labels. The PruDent method focuses on unnecessary label
dependencies and error-propagation showing improved results over 2BR
alali2015prudent .

In Classifier chains (CC) read2011classifier , the first classifier is trained
on the input attributes. The classifier’s output is then added as a new input
attribute, and a second classifier is trained, and so on. In this way the
classifiers are chained, taking into account the possible label dependencies.
In ensembles of classifier chains (ECC) read2011classifier , a set of CCs with
different orders are trained and the outputs are aggregated.

Hierarchy Of Multi-label classifiERs (HOMER) tsoumakas2008effective  creates a
tree of BR methods, were each leaf contains one label. To classify a new
instance, HOMER begin at the root classifier and passes the instance to each
child only if the parent predicted any of its labels. The union of the
predicted labels by the leaves generates output for the given instance.

AdaBoost.MH schapire2000boostexter  is the multi-label variation of the well
known AdaBoost algorithm freund1997decision . AdaBoost.MH weighs the labels as
well as the instances. Training instances and their corresponding labels that
are hard to predict, get incrementally higher weights in following classifiers
while instances and labels that are easy to classify get lower weights. This
algorithm is designed to minimize the hamming loss. ADTBoost.MH de2003learning
which uses ADTTrees is an extension of AdaBoost.MH.

Random k-Labelsets (RAkEL) Tsoumakas2007  selects a number of random
k-labelsets and learns a Label Powerset classifer for each of them. These are
then aggregated. Enhancements over RAkEL include RAkEL++ rokach2014ensemble
and RAkELLd tsoumakas2010random .

An experimental study on most of the above multi-label ensembles suggests that
ECC, followed by RAkEL, exhibit the best overall performance for all of the
examined metrics moyano2018 .

A few notable ensemble methods were published in the last three years. ML-
FOREST wu2016ml  builds on Random Forest and ML-TSVM chen2016mltsvm  on SVM.
PRAkEL, a cost-sensitive extension of RAkEL that considers the evaluation
criteria and is sensitive to the cost of misclassifying an instance
wu2017progressive . fRAkEL speeds up RAkEL by shrinking the samples with
irrelevant labels xu2016multi . The TSEN ensemble zhang2018_TSEN  is based on
three-way-decisions yao2011superiority . The MULE ensemble
papanikolaou2017large  relies on a heterogeneous ensemble that is composed of
different base models. The assumption is that different labels can be
approximated better by different types of models. MULE incorporates a
statistical test to combine the base models. Most of these methods compare
their performance to earlier methods, mainly to variations of 2BR, ECC,
AdaBoost.MH and RAkEL. However an evaluation of these methods in comparison to
one another is still missing.

###  3.3 Deep learning methods

Deep learning uses multiple layers to represent the abstractions of data and
to automatically discover useful features lecun2015deep ; schmidhuber2015deep
. Deep learning can cope with large amounts of input features, eliminating the
need for feature selection methods. The learnt feature representations are
often accurate for other unseen data as well. While this quality, known as
transfer learning pan2009survey  is not unique to deep learning, in the case
of multiple labels it can provide a solution to the label and labelset
imbalance problems. This also means that parameters for a new model (such as
number of layers and number of nodes) can be learnt from previous successful
models instead of by trial and error. Deep learning models are devised using
different architectures pouyanfar2019survey . Convolutional Neural Network
(CNN) models collobert2011natural ; lecun2010convolutional  are often used for
image processing. Recurrent neural network (RNN) models lai2015recurrent  and
their variant Long Short-Term Memory (LSTM) elman1990finding ; tai2015improved
are often used with text and speech. Generative adversarial networks (GANs)
goodfellow2014generative  and Restricted Boltzmann Machine (RBM)
ackley1985learning ; taylor2007modeling  have been used for unsupervised
multi-label learning tasks but we herein focus on a supervised MLR setting as
defined in definition 1.

Deep learning for multi-label tasks is a growing field, with new papers
appearing frequently. We survey some of the recent developments in two main
multi-label tasks, which belong to the media domain: image annotation and text
annotation.

#### Image annotation

The growing interest in deep learning for multi-label image annotation is
partly driven by new publicly accessible large-scale datasets with quality
labels. A few of the most notable ones incluse:

  * •

The Visual Genome dataset krishna2017visual . Contains over 108K images with
an average of 35 labelled objects, 26 attributes, and 21 pairwise
relationships between objects.

  * •

The ChestX-ray14 dataset wang2017chestx . Contains over 112k chest X-rays from
over 30k patients, labeled with up to 14 pathologies or “No Finding”.

  * •

The MS COCO dataset lin2014microsoft . Contains 328k images with 2.5 million
labelled objects, out of a set of 91 labels.

  * •

The NUS-WIDE dataset chua2009nus . Contains almost 270k Flickr images and
their associated labels, with a total of 5,018 unique labels.

Recent reviews of deep learning for medical images highlight the vast amount
of emerging research in this field feng2019deep ; karako2018medical ;
sahiner2019deep . On the chest X-rays dataset, various CNN based methods have
been suggested. One approach is to transform the problem into multiple single-
label classification problems, to which a CNN architecture is applied
gong2013deep . Another suggestion is Hypotheses-CNNPooling (HCP), where a
number of object (i.e., image) segment hypotheses are taken as the inputs,
then a shared CNN is connected with each hypothesis, and finally the CNN
output results from different hypotheses are aggregated with max pooling to
produce the multi-label predictions wei2015hcp . The CNN-RNN model uses an
underlying RNN model mikolov2010recurrent  to capture the high order label
dependencies. Then, CNN and RNN are combined into one framework to exploit the
label dependencies at the global level wang2016cnn . Other models exist (e.g.
dong2017learning ; guan2018 ; guan2018diagnose ; guendel2018learning ;
rajpurkar2017chexnet ; shin2016learning ), as well as a cascade ensemble
kumar2018cascade . Though multi-instance methods are out of our scope, we note
that a multi-instance method, that integrates the images with other
information about the patients has recently been reported to enhance
performance baltruschat2019comparison .

Various studies have been conducted on non-medical images as well. For a
recent review see Voulodimos et al. voulodimos2018deep . Some of these models
focus on learning the label correlations. A model that learns image-dependent
conditional structures li2016conditional  has been proposed. A Spatial
Regularization Network (SRN) that captures the spatial correlation between
labels as well as the semantic correlation has been suggested zhu2017learning
. A feature attention network (FAN) focuses on more important features and
learn the correlations among convolutional features yan2019 . The Regional
Latent Semantic Dependencies (RLSD) model zhang2018multilabel  specializes in
predicting small objects (alongside prediction of large objects) by first
extracting convolutional features, which are further sent to an RPN-like
(Regional Proposal Network) localization layer. The layer is designed to
localize the regions in an image that may contain multiple semantically
dependent labels. These regions are encoded with a fully-connected neural
network and further sent to an RNN, which captures the latent semantic
dependencies at the regional level. The RNN unit sequentially outputs a multi-
class prediction, based on the outputs of the localization layer and the
outputs of previous recurrent neurons. Finally, a max-pooling operation is
carried out to fuse all the regional outputs as the final prediction. Once
again, multi-instance methods can enhance performance luo2019visual .

Comparisons of cutting-edge image annotation methods are still lacking. A
precedent study compared the performance of ten foreground deep learning
multi-label APIs on the Visual Genome dataset kubany2019semantic . The APIs
were evaluated using various metrics. In addition, a semantic similarity
metric was used allowing for words with similar meaning to be classified as
correct predictions. For example, “bicycle” and “bike” were both classified as
correct for an image of a pedal driven two-wheeler. The study shows that
different APIs excel under different evaluation metrics. Regretfully, the
underlying algorithm of each API is not always publicly available.

#### Text Annotation

One of the earliest models to employ deep learning for text classification was
BP-MLL zhang2006multilabel . It formulated multi-label classification problems
as a neural network with multiple output nodes, one for each label. It was
later suggested nam2014large  to replace the pairwise ranking loss in the
model with a cross-entropy loss instead. However these models do not consider
label dependencies. A CNN model that has a final hidden layer which considers
label co-occurence weights was suggested next kurata_2016 . The model was
analyzed on a small dataset with 103 labels. In the CNN-RNN model
chen2017ensemble , the RNN is set to deal with label co-occurence. The C2AE
algorithm employs a DNN-based label embedding framework and performs joint
feature and label embedding yeh2017learning .

The Seq2seq model yang2018sgm  uses a LSTM to generate labels sequentially,
and predicts the next label based on its previously predicted labels. The
$LSTM^{2}$ model yan2018lstm  utilizes LSTM twice. The algorithm first builds
a representation of the documents in the training set. This is done with a
LSTM network that considers word sequences. For each document in the test set,
the algorithm search for the most similar documents in the training set and
retrieves their labels. The labels are represented as a semantic tree that is
trained with dependency parsing. This tree can capture the correlations
between labels. Based on the document representation, another LSTM is utilized
to rank the document labels.

Recently, text categorization has also been employed in the context of X-ray
images. For example, it has been pointed out that China’s chest X-ray reports
focus more on characterization than on labelling the possible diseases. A
recent study uses LSTM to read the X-ray reports, and output labels of
pathologies yan2019combining . Another study combines the reports and the
instances (a multi-instance model) for the same purpose wang2018tienet .

###  3.4 Extreme multi-label classification

Extreme multi-label (XML) problems, also known as large-scale multi-label
problems, refer to problems with an extremely large set of labels, usually in
the millions. Due to the huge amount of labels, the zero-shot problem, as well
as the label and labelset imbalance problems (see section 1), are intensified.
Deep learning models work well here since they considers both the relatedness
of the representations and the context information.

Currently, all of these problems focus on text classification or on
traditional recommmender system problems that have been reformulated as XML
problems tang2009large ; ueda2003parametric . The instances are of one of the
following kinds:

  * •

Text. Assigning categories (the labels) to a Wikipedia page out of the million
categories (labels) available, recommending bid phrases (the labels) to an
advertiser with a given ad landing page agrawal2013multi , assigning tags (the
labels) to an image. In these cases, the instance, be it a web page or an
image, is represented by a bag of words.

  * •

Item. Assigning a number of categories (the labels) to an Amazon product item.
The instance is represents by item features.

  * •

User. Recommending YouTube videos (the labels) to a user weston2013label . The
instance is represented by user features.

We note that these tasks often require a ranked output, and that a ranked
input might be available. However, to the best of our knowledge, extreme label
ranking which explicitly considers ranked inputs (as opposed to extreme multi-
label) has not yet been explored.

PDSparse yen2016pd  and DiSMEC babbar2017dismec  tackle the sparseness (high
dimensonality) by training one XML model per label. We thus consider them as
XML problem transformation methods.

MLRF agrawal2013multi  designs a multi-label random forest. To cope with the
high dimensonality problem, they assume label independence during the ensemble
construction, but they do consider correlations during prediction. The FastXML
model prabhu2014fastxml  provides a node partitioning formulation that
improves MLRF results in settings with millions of labels. XML-CNN liu2017deep
uses a CNN model for the actual classification.

The SwiftXML model prabhu2018extreme  learns label correlation using a
word2vec embedding. This allows the discovery of similar labels as these
labels are classified closely in the embedded vector. An example given by the
authors is that although the labels of the Wikipedia pages of Einstein and
Newton are very different, the SwiftXML label embedding will learn that the
two are similar, and thus will be able to consider labels given to Einstein’s
page, also for Newton’s page. Authors report to have outperformed SLEEC
bhatia2015sparse  which also employs label embedding and is considered state-
of-the-art. A recent model zhang2018deep  tackles both feature and label
spaces using a non-linear embedding based on a graph structure.

##  4 Label ranking algorithms

Numerous label ranking algorithms were suggested in the literature. One
approach is based on turning the problem into several binary classification
problems and then combining them into output rankings cheng2013labelwise ;
destercke2015cautious ; gurrieri2014alternative ; har2003constraint ;
hullermeier2008label . Another common approach is based on modifying existing
probabilistic algorithms to directly support label ranking. Some main examples
are: naive Bayes models aiguzhinov2010similarity , k-nearest neighbor models
brazdil2003ranking  and decision tree models such as Label Ranking Trees (LRT)
cheng2009decision  and Entropy Based Ranking Trees (ERT) de2015distance .

A few other stand-alone ideas are available as well. RPC (Ranking by Pairwise
Comparison) hullermeier2008label  learns pairwise preferences from which a
ranking is derived. Instance Based Logistic Regression (IBLR) combines
instance-based learning and logistic regression cheng2009decision . Under this
approach, the label statistics of neighboring instances are regarded as
features by the logistic regression classifier. A rule based approach learns a
reduction technique and provides a mapping in the form of logical rules
gurrieri2012label . A recent work korba2018structured  adapts ideas from
structured output prediction. They cast the label ranking problem into the
structured prediction framework and propose embeddings dedicated to ranking
representation. For each embedding they propose a solution to the pre-image
problem. This latter suggestion is a harbinger for a bridge between label
ranking and structured image prediction.

Surveys on label ranking algorithms vembu2010label ; zhou2014taxonomy  capture
some of the earlier methods.

###  4.1 Label ranking ensembles

To the best of our knowledge, only a few papers thus far have investigated the
use of ensembles for label ranking aledo2017tackling ; sa2017label ;
werbin2019beyond ; ZHOU2018 . The ensembles proposed in these papers differ
in: (1) the base label ranking algorithm used, (2) the method used to sample
the data to train each of the simple models (if all models are trained with
the exact same data they will output the exact same results, and then there is
no need for an ensemble), and (3) the aggregation method used to combine the
results of the simple models.

As the base label ranking algorithm aledo2017tackling  used Label Ranking
Trees (LRT) cheng2009decision , sa2017label  used Ranking Trees (RT) and
Entropy Ranking Trees (ERT) de2015distance , ZHOU2018  developed their own
method named Top Label As Class (TLAC) and werbin2019beyond  used both LRT and
Ranking by Pairwise Comparison (RPC) hullermeier2008label .

To select the training data for each simple classifier, aledo2017tackling  and
werbin2019beyond  used a technique known as Bootstrap aggregation or Bagging
breiman1996bagging : they created $b$ different bags by selecting a subset of
the dataset’s instances with replacement. The other two papers sa2017label ;
ZHOU2018  suggested modifications to the well-known Random Forest ensemble
model breiman2001random . As for the aggregation method used, three studies
used a voting rule (either Borda or Modal Ranking). The last study
werbin2019beyond  presents VRS (Voting Rule Selector), a meta-model that
automatically learns the best voting rule to be used. There four works are
summarized in table 2.

| Aledo et al.(2017) | Sa et al. (2017) | Zhou et al. (2018) | Werbin et al. (2019)
---|---|---|---|---
Base algorithm | LRT | ERT, RT | TLAC | LRT, RPC
Data sampling | Bagging | Random Forest | Random Forest | Bagging
Aggregation | Modal ranking | Borda | Borda | Voting Rule Selector
Table 2: Label ranking ensemble frameworks

AdaBoost.MR is the label ranking variation of the well known AdaBoost
algorithm freund1997decision . The algorithm performs pairwise comparisons
between labels. Training instances and their corresponding label pairs that
are hard to predict, get incrementally higher weights in following classifiers
while instances and label pairs that are easy to classify get lower weights.
The algorithm is designed to minimize the ranking loss. Another boosting-based
approach suggests learning a linear utility function for each label, from
which the ranking is deduced dekel2004log . This approach is more general, as
it allows the input to be any sort of preference graph over the labels. A
ranking function assigns a utility score to each label. Then, relevant
pairwise comparisons as deduced from the graph, are performed. Again, the
label weights are updated and additional weight is given to each wrongly
ranked pair. As these approaches rely on pairwise comparisons, they do not
scale well in the case of many labels.

##  5 Evaluation

Evaluating the performance of multi-label algorithms is difficult. For
example, it is may be impossible to decide which mistake of the following two
cases is more serious: one instance with three incorrect labels vs. three
instances each with one incorrect label. Therefore, a number of performance
measures focusing on different aspects have been proposed. Schapire and Singer
schapire2000boostexter  initially suggested five metrics: Hamming loss,
ranking loss, one-error, average precision. The popular micro-F1 and macro-F1
were suggested by Tsoumakas et al. tsoumakas2010random . Instance-F1 and AUC
measures were discussed by Koyejo et al. koyejo2015consistent  A summary of
the 11 most common measures is provided Wu et al. wu2017unified . For label
ranking, the most popular measure is Kendall-tau. The hamming distance is also
often used when permutations represent matching of bipartite graphs.

###  5.1 Dataset repositories

The Mulan repository mulan  contains 27 multi-label datasets on various
subjects. MEKA MEKA  contains datasets in arff format, suitable for Weka. An R
package automates the use of these datasets charte16repository . The KDIS
research group offers a repository of various datasets obtained from different
sources. 111http://www.uco.es/kdis/mllresources/

The Extreme Classification Repository stores 15 extreme multi-label text
datasets, as well as code for various algorithms.
222http://manikvarma.org/downloads/XC/XMLRepository.html$\\#$Prabhu14. LSHTC
series challenge also stores extreme multi-label text datasets
partalasKBAPGAA15 . The SNAP library snapnets  contains social and information
networks, some of which can (and were) utilized for extreme multi-label tasks
(e.g. Amazon products). Lastly, it is possible to generate simulated data via
a multi-label data generator tomas2014framework .

As for datasets with a ranked target class as input, five real world label
ranking datasets can be found in LR_data . The 16 semi-syntetic label ranking
datasets used in cheng10icmla ; cheng2009decision  are stored on the webpage
of one of the authors. 333https://cs.uni-paderborn.de/?id=63912

###  5.2 Stratification of multi-label data

Estimating the accuracy of a model is traditionally done by splitting the data
to training, test, and sometimes also validation subsets. Different techniques
are available, such as cross validation, holdout and bootstrap
efron1994introduction ; refaeilzadeh2009cross . Random sampling works well
when the labels have enough representation in the data. When this is not the
case, a stratification approach assures that all subsets contain contain
approximately the same proportions of labels as the original dataset. For
single-label classification tasks, stratification has been shown to outperform
cross-validation with random sampling kohavi1995study .

For multi-label data, stratification is even more important, as some datasets
may have very few patterns representing them and random sampling might place
them all in either the training or the test data partitions.

Stratification for multi-label data can either consider the distinct labelsets
available in the data, or consider each label separately. Since the number of
labelsets grows exponentially to the number of labels, when the number of
labels is large there might be only one instance for each example in the
labelset, or even no examples at all. For this reason, a method that considers
each label separately was suggested by sechidis2011stratification .
Stratifying the data in this method is a slow procedure, but nonetheless, it
improves the classifier performance.

##  6 Research directions and open problems

There are still many paths to explore for multi-label ranking. We outline a
few of them here.

First, in many real-world scenarios, new labels emerge over time. For example,
the label “valentine2030” will only emerge on year 2030. Multi-Label learning
with emerging new labels has just begun to be considered, and we are aware of
only one pioneering work on the subject zhu2018multi . We have seen herein
that recommender system problems can be reformulated as multi-label ones.
Perhaps it is time to reconsider the other direction as well
yang2012multilabel . The “cold-start” problem is fundamental in recommender
systems. Perhaps their solutions can be adapted to multi-label ranking tasks.
Moreover, evaluation measures need to be fine-tuned to this setting of
emerging labels rios2018few ; xian2017zero .

Second, a recent study bogaert2019evaluating  compared recommender system and
multi-label classification techniques concluding that AdaBoost with CC chains
and BR with multi-label random forest outperform the best recommender system
methods in a given cross-selling setting. However, state-of-the-art multi-
label deep learning methods and extreme multi-label methods should be able to
do even better but have not been considered in the above study. Moreover, it
is interesting to reconsider other recommender and ranking scenarios and
reformulate them as MLR tasks as well.

Third, for single-label problems, a set of complexity measures that calculate
the overlap and separability of classes has been defined ho2002complexity . To
the best of our knowledge, one characterization metric, called TCS
(Theoretical Complexity Score) exists for multi-label tasks charte2016impact
but no complexity measures exist for label ranking, where the emphasis is on
the correct order of labels. In a related context, SCUMBLE (Score of
ConcUrrence among iMBalanced LabEls) is a new metrics that address label co-
occurrence in multi-label datasets charte2019imbalanced . It can and should be
used to gain better understanding of the available data.

Fourth, we have reviewed state-of-the-art deep learning algorithms for image
annotation and separate algorithms for text classification. One cannot but
wonder if these fields can inspire one another and furthermore, if a meta-
method for both tasks can be developed.

Lastly, as explained in section 3.4, extreme label ranking problems where the
target input space is ordered have yet to be explored.

## References

  * (1) David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski.  A learning algorithm for boltzmann machines.  Cognitive science, 9(1):147–169, 1985.
  * (2) Rahul Agrawal, Archit Gupta, Yashoteja Prabhu, and Manik Varma.  Multi-label learning with millions of labels: Recommending advertiser bid phrases for web pages.  In Proceedings of the 22nd international conference on World Wide Web, pages 13–24. ACM, 2013.
  * (3) Artur Aiguzhinov, Carlos Soares, and Ana Paula Serra.  A similarity-based adaptation of naive bayes for label ranking: Application to the metalearning problem of algorithm recommendation.  In International Conference on Discovery Science, pages 16–26. Springer, 2010.
  * (4) Reem Al-Otaibi, Meelis Kull, and Peter Flach.  Lacova: A tree-based multi-label classifier using label covariance as splitting criterion.  In 2014 13th International Conference on Machine Learning and Applications, pages 74–79. IEEE, 2014.
  * (5) Abdulaziz Alali and Miroslav Kubat.  Prudent: A pruned and confident stacking approach for multi-label classification.  IEEE Transactions on Knowledge and Data Engineering, 27(9):2480–2493, 2015.
  * (6) Juan A Aledo, José A Gámez, and David Molina.  Tackling the supervised label ranking problem by bagging weak learners.  Information Fusion, 35:38–50, 2017.
  * (7) Rohit Babbar and Bernhard Schölkopf.  Dismec: Distributed sparse machines for extreme multi-label classification.  In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pages 721–729. ACM, 2017.
  * (8) Ivo M Baltruschat, Hannes Nickisch, Michael Grass, Tobias Knopp, and Axel Saalbach.  Comparison of deep learning approaches for multi-label chest x-ray classification.  Scientific reports, 9(1):6381, 2019.
  * (9) Priyadarshini Barot and Mahesh Panchal.  Review on various problem transformation methods for classifying multi-label data.  International Journal of Data Mining And Emerging Technologies, 4(2):45–52, 2014.
  * (10) Kush Bhatia, Himanshu Jain, Purushottam Kar, Manik Varma, and Prateek Jain.  Sparse local embeddings for extreme multi-label classification.  In Advances in neural information processing systems, pages 730–738, 2015.
  * (11) Matthias Bogaert, Justine Lootens, Dirk Van den Poel, and Michel Ballings.  Evaluating multi-label classifiers and recommender systems in the financial service sector.  European Journal of Operational Research, 2019.
  * (12) Hanen Borchani, Gherardo Varando, Concha Bielza, and Pedro Larrañaga.  A survey on multi-output regression.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(5):216–233, 2015.
  * (13) Matthew R Boutell, Jiebo Luo, Xipeng Shen, and Christopher M Brown.  Learning multi-label scene classification.  Pattern recognition, 37(9):1757–1771, 2004.
  * (14) Pavel B Brazdil, Carlos Soares, and Joaquim Pinto Da Costa.  Ranking learning algorithms: Using ibl and meta-learning on accuracy and time results.  Machine Learning, 50(3):251–277, 2003.
  * (15) Leo Breiman.  Bagging predictors.  Machine learning, 24(2):123–140, 1996.
  * (16) Leo Breiman.  Random forests.  Machine learning, 45(1):5–32, 2001.
  * (17) Klaus Brinker, Johannes Fürnkranz, and Eyke Hüllermeier.  A unified model for multilabel classification and ranking.  In Proceedings of the 2006 conference on ECAI 2006: 17th European Conference on Artificial Intelligence August 29–September 1, 2006, Riva del Garda, Italy, pages 489–493. IOS Press, 2006.
  * (18) Serhat S Bucak, Pavan Kumar Mallapragada, Rong Jin, and Anil K Jain.  Efficient multi-label ranking for multi-class learning: application to object recognition.  In 2009 IEEE 12th International Conference on Computer Vision, pages 2098–2105. IEEE, 2009.
  * (19) Jorge Calvo-Zaragoza, Jose J Valero-Mas, and Juan R Rico-Juan.  Improving knn multi-label classification in prototype selection scenarios using class proposals.  Pattern Recognition, 48(5):1608–1622, 2015.
  * (20) Ricardo Cerri, Rodrigo C Barros, and André CPLF de Carvalho.  Hierarchical multi-label classification for protein function prediction: A local approach based on neural networks.  In 2011 11th International Conference on Intelligent Systems Design and Applications, pages 337–343. IEEE, 2011.
  * (21) David Charte, Francisco Charte, Salvador García, and Francisco Herrera.  A snapshot on nonstandard supervised learning problems: taxonomy, relationships, problem transformations and algorithm adaptations.  Progress in Artificial Intelligence, 8(1):1–14, Apr 2019.
  * (22) Francisco Charte and David Charte.  Working with multilabel datasets in R: The mldr package.  The R Journal, 7(2):149–162, December 2015.
  * (23) Francisco Charte, David Charte, Antonio Rivera, María José del Jesus, and Francisco Herrera.  R ultimate multilabel dataset repository.  In Francisco Martínez-Álvarez, Alicia Troncoso, Héctor Quintián, and Emilio Corchado, editors, Hybrid Artificial Intelligent Systems, pages 487–499, Cham, 2016. Springer International Publishing.
  * (24) Francisco Charte, Antonio Rivera, María José del Jesus, and Francisco Herrera.  On the impact of dataset complexity and sampling strategy in multilabel classifiers performance.  In International Conference on Hybrid Artificial Intelligence Systems, pages 500–511. Springer, 2016.
  * (25) Francisco Charte, Antonio J. Rivera, María J. del Jesus, and Francisco Herrera.  Dealing with difficult minority labels in imbalanced mutilabel data sets.  Neurocomputing, 326-327:39 – 53, 2019.
  * (26) Guibin Chen, Deheng Ye, Zhenchang Xing, Jieshan Chen, and Erik Cambria.  Ensemble application of convolutional and recurrent neural networks for multi-label text categorization.  In 2017 International Joint Conference on Neural Networks (IJCNN), pages 2377–2383. IEEE, 2017.
  * (27) Wei-Jie Chen, Yuan-Hai Shao, Chun-Na Li, and Nai-Yang Deng.  Mltsvm: a novel twin support vector machine to multi-label learning.  Pattern Recognition, 52:61–74, 2016.
  * (28) Weiwei Cheng, Krzysztof Dembczyński, and Eyke Hüllermeier.  Label ranking methods based on the Plackett-Luce model.  In Johannes Fürnkranz and Thorsten Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 215–222, Haifa, Israel, June 2010. Omnipress.
  * (29) Weiwei Cheng, Sascha Henzgen, and Eyke Hüllermeier.  Labelwise versus pairwise decomposition in label ranking.  In LWA, pages 129–136, 2013.
  * (30) Weiwei Cheng, Jens Hühn, and Eyke Hüllermeier.  Decision tree and instance-based learning for label ranking.  In Proceedings of the 26th Annual International Conference on Machine Learning, pages 161–168. ACM, 2009.
  * (31) Tsung-Hsien Chiang, Hung-Yi Lo, and Shou-De Lin.  A ranking-based knn approach for multi-label classification.  In Asian Conference on Machine Learning, pages 81–96, 2012.
  * (32) Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng.  Nus-wide: a real-world web image database from national university of singapore.  In Proceedings of the ACM international conference on image and video retrieval, page 48. ACM, 2009.
  * (33) Amanda Clare and Ross D. King.  Knowledge discovery in multi-label phenotype data.  In Luc De Raedt and Arno Siebes, editors, Principles of Data Mining and Knowledge Discovery, pages 42–53, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg.
  * (34) Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa.  Natural language processing (almost) from scratch.  Journal of machine learning research, 12(Aug):2493–2537, 2011.
  * (35) Francesco De Comité, Rémi Gilleron, and Marc Tommasi.  Learning multi-label alternating decision trees from texts and data.  In International Workshop on Machine Learning and Data Mining in Pattern Recognition, pages 35–49. Springer, 2003.
  * (36) Cláudio Rebelo de Sá, Carla Rebelo, Carlos Soares, and Arno Knobbe.  Distance-based decision tree algorithms for label ranking.  In Portuguese Conference on Artificial Intelligence, pages 525–534. Springer, 2015.
  * (37) Ofer Dekel, Yoram Singer, and Christopher D Manning.  Log-linear models for label ranking.  In Advances in neural information processing systems, pages 497–504, 2004.
  * (38) Sébastien Destercke, Marie-Hélène Masson, and Michael Poss.  Cautious label ranking with label-wise decomposition.  European Journal of Operational Research, 246(3):927–935, 2015\.
  * (39) Sotiris Diplaris, Grigorios Tsoumakas, Pericles A Mitkas, and Ioannis Vlahavas.  Protein classification with multiple algorithms.  In Panhellenic Conference on Informatics, pages 448–456. Springer, 2005.
  * (40) Yuxi Dong, Yuchao Pan, Jun Zhang, and Wei Xu.  Learning to read chest x-ray images from 16000+ examples using cnn.  In Proceedings of the Second IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies, pages 51–57. IEEE Press, 2017.
  * (41) Bradley Efron and Robert J Tibshirani.  An introduction to the bootstrap.  CRC press, 1994.
  * (42) André Elisseeff and Jason Weston.  A kernel method for multi-labelled classification.  In Advances in neural information processing systems, pages 681–687, 2002.
  * (43) Jeffrey L Elman.  Finding structure in time.  Cognitive science, 14(2):179–211, 1990.
  * (44) Yeli Feng, Hui Seong Teh, and Yiyu Cai.  Deep learning for chest radiology: A review.  Current Radiology Reports, 7(8):24, 2019.
  * (45) Yoav Freund and Robert E Schapire.  A decision-theoretic generalization of on-line learning and an application to boosting.  Journal of computer and system sciences, 55(1):119–139, 1997.
  * (46) Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Mencía, and Klaus Brinker.  Multilabel classification via calibrated label ranking.  Machine learning, 73(2):133–153, 2008.
  * (47) Eva Gibaja and Sebastián Ventura.  Multi-label learning: a review of the state of the art and ongoing research.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 4(6):411–444, 2014.
  * (48) Eva Gibaja and Sebastián Ventura.  A tutorial on multilabel learning.  ACM Computing Surveys (CSUR), 47(3):52, 2015.
  * (49) Shantanu Godbole and Sunita Sarawagi.  Discriminative methods for multi-labeled classification.  In Pacific-Asia conference on knowledge discovery and data mining, pages 22–30. Springer, 2004.
  * (50) Eduardo Corrêa Gonçalves, Alex A Freitas, and Alexandre Plastino.  A survey of genetic algorithms for multi-label classification.  In 2018 IEEE Congress on Evolutionary Computation (CEC), pages 1–8. IEEE, 2018.
  * (51) Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey Ioffe.  Deep convolutional ranking for multilabel image annotation.  In nternational Conference on Learning Representations (ICLR), 2014\.
  * (52) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * (53) Quanquan Gu, Zhenhui Li, and Jiawei Han.  Correlated multi-label feature selection.  In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 1087–1096. ACM, 2011.
  * (54) Zuguang Gu, Lei Gu, Roland Eils, Matthias Schlesner, and Benedikt Brors.  circlize implements and enhances circular visualization in r.  Bioinformatics, 30(19):2811–2812, 2014.
  * (55) Qingji Guan and Yaping Huang.  Multi-label chest x-ray image classification via category-wise residual attention learning.  Pattern Recognition Letters, 2018.
  * (56) Qingji Guan, Yaping Huang, Zhun Zhong, Zhedong Zheng, Liang Zheng, and Yi Yang.  Diagnose like a radiologist: Attention guided convolutional neural network for thorax disease classification.  arXiv preprint arXiv:1801.09927, 2018.
  * (57) Sebastian Guendel, Sasa Grbic, Bogdan Georgescu, Siqi Liu, Andreas Maier, and Dorin Comaniciu.  Learning to recognize abnormalities in chest x-rays with location-aware dense networks.  In Iberoamerican Congress on Pattern Recognition, pages 757–765. Springer, 2018.
  * (58) Massimo Gurrieri, Philippe Fortemps, and Xavier Siebert.  Alternative decomposition techniques for label ranking.  In International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, pages 464–474. Springer, 2014.
  * (59) Massimo Gurrieri, Xavier Siebert, Philippe Fortemps, Salvatore Greco, and Roman Słowiński.  Label ranking: A new rule-based label ranking method.  In International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, pages 613–623. Springer, 2012.
  * (60) Sariel Har-Peled, Dan Roth, and Dav Zimak.  Constraint classification for multiclass classification and ranking.  In Advances in neural information processing systems, pages 809–816, 2003.
  * (61) Francisco Herrera, Francisco Charte, Antonio J Rivera, and María J Del Jesus.  Multilabel classification.  In Multilabel Classification, pages 17–31. Springer, 2016.
  * (62) Francisco Herrera, Sebastián Ventura, Rafael Bello, Chris Cornelis, Amelia Zafra, Dánel Sánchez-Tarragó, and Sarah Vluymans.  Multiple instance learning.  In Multiple instance learning, pages 17–33. Springer, 2016.
  * (63) Tin Kam Ho and Mitra Basu.  Complexity measures of supervised classification problems.  IEEE Transactions on Pattern Analysis & Machine Intelligence, 24(3):289–300, 2002.
  * (64) Jun Huang, Guorong Li, Shuhui Wang, Zhe Xue, and Qingming Huang.  Multi-label classification by exploiting local positive and negative pairwise label correlation.  Neurocomputing, 257:164–174, 2017.
  * (65) Sheng-Jun Huang, Wei Gao, and Zhi-Hua Zhou.  Fast multi-instance multi-label learning.  IEEE transactions on pattern analysis and machine intelligence, 2018\.
  * (66) Sheng-Jun Huang and Zhi-Hua Zhou.  Multi-label learning by exploiting label correlations locally.  In Twenty-sixth AAAI conference on artificial intelligence, 2012\.
  * (67) Eyke Hüllermeier, Johannes Fürnkranz, Weiwei Cheng, and Klaus Brinker.  Label ranking by learning pairwise preferences.  Artificial Intelligence, 172(16-17):1897–1916, 2008.
  * (68) George Sakkas Grigorios Tsoumakas Ioannou, Marios and Ioannis Vlahavas.  Obtaining bipartitions from score vectors for multi-label classification.  In In 2010 22nd IEEE International Conference on Tools with Artificial Intelligence, pages 409–416, 2010.
  * (69) Thorsten Joachims.  Text categorization with support vector machines: Learning with many relevant features.  In European conference on machine learning, pages 137–142. Springer, 1998.
  * (70) Kenji Karako, Yu Chen, and Wei Tang.  On medical application of neural networks trained with various types of data.  Bioscience trends, 12(6):553–559, 2018.
  * (71) Ron Kohavi et al.  A study of cross-validation and bootstrap for accuracy estimation and model selection.  In Ijcai, volume 14, pages 1137–1145. Montreal, Canada, 1995.
  * (72) Anna Korba, Alexandre Garcia, and Florence d’Alché Buc.  A structured prediction approach for label ranking.  In Advances in Neural Information Processing Systems, pages 8994–9004, 2018.
  * (73) Oluwasanmi O Koyejo, Nagarajan Natarajan, Pradeep K Ravikumar, and Inderjit S Dhillon.  Consistent multilabel classification.  In Advances in Neural Information Processing Systems, pages 3321–3329, 2015.
  * (74) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision, 123(1):32–73, 2017.
  * (75) Adam Kubany, Shimon Ben Ishay, Ruben-sacha Ohayon, Armin Shmilovici, Lior Rokach, and Tomer Doitshman.  Semantic comparison of state-of-the-art deep learning methods for image multi-label classification.  arXiv preprint arXiv:1903.09190, 2019.
  * (76) Pulkit Kumar, Monika Grewal, and Muktabh Mayank Srivastava.  Boosted cascaded convnets for multilabel classification of thoracic diseases in chest radiographs.  In Aurélio Campilho, Fakhri Karray, and Bart ter Haar Romeny, editors, Image Analysis and Recognition, pages 546–552, Cham, 2018. Springer International Publishing.
  * (77) Gakuto Kurata, Bing Xiang, and Bowen Zhou.  Improved neural network-based multi-label classification with better initialization leveraging label co-occurrence.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 521–526, San Diego, California, June 2016. Association for Computational Linguistics.
  * (78) Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao.  Recurrent convolutional neural networks for text classification.  In Twenty-ninth AAAI conference on artificial intelligence, 2015\.
  * (79) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436–444, 2015.
  * (80) Yann LeCun, Koray Kavukcuoglu, and Clément Farabet.  Convolutional networks and applications in vision.  In Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pages 253–256. IEEE, 2010.
  * (81) Jure Leskovec and Andrej Krevl.  SNAP Datasets: Stanford large network dataset collection.  http://snap.stanford.edu/data, June 2014.
  * (82) Qiang Li, Maoying Qiao, Wei Bian, and Dacheng Tao.  Conditional graphical lasso for multi-label image classification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2977–2986, 2016.
  * (83) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * (84) Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang.  Deep learning for extreme multi-label text classification.  In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 115–124. ACM, 2017\.
  * (85) Yan Luo, Ming Jiang, and Qi Zhao.  Visual attention in multi-label image classification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * (86) Gjorgji Madjarov, Dejan Gjorgjevikj, and Sašo Džeroski.  Two stage architecture for multi-label learning.  Pattern Recognition, 45(3):1019–1034, 2012.
  * (87) Andrew McCallum.  Multi-label text classification with a mixture model trained by em.  In AAAI workshop on Text Learning, pages 1–7, 1999.
  * (88) Eneldo Loza Mencia and Johannes Fürnkranz.  Efficient pairwise multilabel classification for large-scale problems in the legal domain.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 50–65. Springer, 2008.
  * (89) Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur.  Recurrent neural network based language model.  In Eleventh annual conference of the international speech communication association, 2010.
  * (90) Marcin Michał Mirończuk and Jarosław Protasiewicz.  A recent overview of the state-of-the-art elements of text classification.  Expert Systems with Applications, 106:36 – 54, 2018.
  * (91) Jose M. Moyano, Eva L. Gibaja, Krzysztof J. Cios, and Sebastián Ventura.  Review of ensembles of multi-label classifiers: Models, experimental study and prospects.  Information Fusion, 44:33 – 45, 2018.
  * (92) Ghulam Mujtaba, Liyana Shuib, Norisma Idris, Wai Lam Hoo, Ram Gopal Raj, Kamran Khowaja, Khairunisa Shaikh, and Henry Friday Nweke.  Clinical text classification research trends: Systematic literature review and open issues.  Expert Systems with Applications, 116:494 – 520, 2019.
  * (93) Jinseok Nam, Jungi Kim, Eneldo Loza Mencía, Iryna Gurevych, and Johannes Fürnkranz.  Large-scale multi-label text classification—revisiting neural networks.  In Joint european conference on machine learning and knowledge discovery in databases, pages 437–452. Springer, 2014.
  * (94) Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009.
  * (95) Yannis Papanikolaou, Grigorios Tsoumakas, Manos Laliotis, Nikos Markantonatos, and Ioannis Vlahavas.  Large-scale online semantic indexing of biomedical articles via an ensemble of multi-label classification models.  Journal of biomedical semantics, 8(1):43, 2017.
  * (96) Ioannis Partalas, Aris Kosmopoulos, Nicolas Baskiotis, Thierry Artières, George Paliouras, Éric Gaussier, Ion Androutsopoulos, Massih-Reza Amini, and Patrick Gallinari.  LSHTC: A benchmark for large-scale text classification.  CoRR, abs/1503.08581, 2015.
  * (97) Rafael B Pereira, Alexandre Plastino, Bianca Zadrozny, and Luiz HC Merschmann.  Categorizing feature selection methods for multi-label classification.  Artificial Intelligence Review, 49(1):57–78, 2018.
  * (98) James Petterson and Tibério S Caetano.  Submodular multi-label learning.  In Advances in Neural Information Processing Systems, pages 1512–1520, 2011.
  * (99) Ignazio Pillai, Giorgio Fumera, and Fabio Roli.  Designing multi-label classifiers that maximize f measures: State of the art.  Pattern Recognition, 61:394–404, 2017.
  * (100) Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, and SS Iyengar.  A survey on deep learning: Algorithms, techniques, and applications.  ACM Computing Surveys (CSUR), 51(5):92, 2019.
  * (101) Yashoteja Prabhu, Anil Kag, Shilpa Gopinath, Kunal Dahiya, Shrutendra Harsola, Rahul Agrawal, and Manik Varma.  Extreme multi-label learning with label features for warm-start tagging, ranking & recommendation.  In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 441–449. ACM, 2018.
  * (102) Yashoteja Prabhu and Manik Varma.  Fastxml: A fast, accurate and stable tree-classifier for extreme multi-label learning.  In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 263–272. ACM, 2014.
  * (103) Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  arXiv preprint arXiv:1711.05225, 2017.
  * (104) Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank.  Classifier chains for multi-label classification.  Machine learning, 85(3):333, 2011.
  * (105) Jesse Read, Peter Reutemann, Bernhard Pfahringer, and Geoff Holmes.  MEKA: A multi-label/multi-target extension to Weka.  Journal of Machine Learning Research, 17(21):1–5, 2016.
  * (106) Claudio Rebelo de Sa.  Label ranking datasets.  http://dx.doi.org/10.17632/3mv94c8jpc.2, 2018.  Accessed: 2019-10-30.
  * (107) Payam Refaeilzadeh, Lei Tang, and Huan Liu.  Cross-validation.  Encyclopedia of database systems, pages 532–538, 2009.
  * (108) Anthony Rios and Ramakanth Kavuluru.  Few-shot and zero-shot multi-label learning for structured label spaces.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2018, page 3132. NIH Public Access, 2018.
  * (109) Lior Rokach, Alon Schclar, and Ehud Itach.  Ensemble methods for multi-label classification.  Expert Systems with Applications, 41(16):7507–7523, 2014.
  * (110) Timothy N Rubin, America Chambers, Padhraic Smyth, and Mark Steyvers.  Statistical topic models for multi-label document classification.  Machine learning, 88(1-2):157–208, 2012.
  * (111) Cláudio Rebelo Sá, Carlos Soares, Arno Knobbe, and Paulo Cortez.  Label ranking forests.  Expert Systems, 34(1), 2017.
  * (112) Berkman Sahiner, Aria Pezeshk, Lubomir M Hadjiiski, Xiaosong Wang, Karen Drukker, Kenny H Cha, Ronald M Summers, and Maryellen L Giger.  Deep learning in medical imaging and radiation therapy.  Medical physics, 46(1):e1–e36, 2019.
  * (113) Robert E Schapire and Yoram Singer.  Boostexter: A boosting-based system for text categorization.  Machine learning, 39(2-3):135–168, 2000.
  * (114) Jürgen Schmidhuber.  Deep learning in neural networks: An overview.  Neural networks, 61:85–117, 2015.
  * (115) Konstantinos Sechidis, Grigorios Tsoumakas, and Ioannis Vlahavas.  On the stratification of multi-label data.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 145–158. Springer, 2011.
  * (116) Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner-Fushman, Jianhua Yao, and Ronald M Summers.  Learning to read chest x-rays: Recurrent neural cascade model for automated image annotation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2497–2506, 2016.
  * (117) Przemysław Skryjomski, Bartosz Krawczyk, and Alberto Cano.  Speeding up k-nearest neighbors classifier for large-scale multi-label learning on gpus.  Neurocomputing, 354:10–19, 2019.
  * (118) Newton Spolaôr, Maria Carolina Monard, Grigorios Tsoumakas, and Huei Diana Lee.  A systematic review of multi-label feature selection and a new method based on label construction.  Neurocomputing, 180:3–15, 2016.
  * (119) Kai Sheng Tai, Richard Socher, and Christopher D Manning.  Improved semantic representations from tree-structured long short-term memory networks.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556–1566, 2015.
  * (120) Lei Tang, Suju Rajan, and Vijay K Narayanan.  Large scale multi-label classification via metalabeler.  In Proceedings of the 18th international conference on World wide web, pages 211–220. ACM, 2009.
  * (121) Graham W Taylor, Geoffrey E Hinton, and Sam T Roweis.  Modeling human motion using binary latent variables.  In Advances in neural information processing systems, pages 1345–1352, 2007.
  * (122) Fadi A Thabtah, Peter Cowling, and Yonghong Peng.  Mmac: A new multi-class, multi-label associative classification approach.  In Fourth IEEE International Conference on Data Mining (ICDM’04), pages 217–224. IEEE, 2004.
  * (123) Jimena Torres Tomás, Newton Spolaôr, Everton Alvares Cherman, and Maria Carolina Monard.  A framework to generate synthetic multi-label datasets.  Electronic Notes in Theoretical Computer Science, 302:155–176, 2014\.
  * (124) Konstantinos Trohidis, Grigorios Tsoumakas, George Kalliris, and Ioannis P Vlahavas.  Multi-label classification of music into emotions.  In ISMIR, volume 8, pages 325–330, 2008.
  * (125) Grigorios Tsoumakas, Anastasios Dimou, Eleftherios Spyromitros, Vasileios Mezaris, Ioannis Kompatsiaris, and Ioannis Vlahavas.  Correlation-based pruning of stacked binary relevance models for multi-label learning.  In Proceedings of the 1st international workshop on learning from multi-label data, pages 101–116, 2009.
  * (126) Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas.  Effective and efficient multilabel classification in domains with large number of labels.  In Proc. ECML/PKDD 2008 Workshop on Mining Multidimensional Data (MMD’08), volume 21, pages 53–59. sn, 2008.
  * (127) Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas.  Random k-labelsets for multilabel classification.  IEEE Transactions on Knowledge and Data Engineering, 23(7):1079–1089, 2010.
  * (128) Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, Jozef Vilcek, and Ioannis Vlahavas.  Mulan: A java library for multi-label learning.  Journal of Machine Learning Research, 12:2411–2414, 2011.
  * (129) Grigorios Tsoumakas and Ioannis Vlahavas.  Random k-labelsets: An ensemble method for multilabel classification.  In Joost N. Kok, Jacek Koronacki, Raomon Lopez de Mantaras, Stan Matwin, Dunja Mladenič, and Andrzej Skowron, editors, Machine Learning: ECML 2007, pages 406–417, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg.
  * (130) Naonori Ueda and Kazumi Saito.  Parametric mixture models for multi-labeled text.  In Advances in neural information processing systems, pages 737–744, 2003.
  * (131) Shankar Vembu and Thomas Gärtner.  Label ranking algorithms: A survey.  In Preference learning, pages 45–64. Springer, 2010.
  * (132) Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis.  Deep learning for computer vision: A brief review.  Computational intelligence and neuroscience, 2018, 2018.
  * (133) Willem Waegeman, Krzysztof Dembczyński, and Eyke Hüllermeier.  Multi-target prediction: a unifying view on problems and methods.  Data Mining and Knowledge Discovery, 33(2):293–324, 2019.
  * (134) Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu.  Cnn-rnn: A unified framework for multi-label image classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2285–2294, 2016.
  * (135) Xi Wang and Gita Sukthankar.  Multi-label relational neighbor classification using social context features.  In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 464–472. ACM, 2013.
  * (136) Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017.
  * (137) Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M Summers.  Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9049–9058, 2018.
  * (138) Yunchao Wei, Wei Xia, Min Lin, Junshi Huang, Bingbing Ni, Jian Dong, Yao Zhao, and Shuicheng Yan.  Hcp: A flexible cnn framework for multi-label image classification.  IEEE transactions on pattern analysis and machine intelligence, 38(9):1901–1907, 2015.
  * (139) Havi Werbin-Ofir, Lihi Dery, and Erez Shmueli.  Beyond majority: Label ranking ensembles based on voting rules.  Expert Systems with Applications, 2019.
  * (140) Jason Weston, Ameesh Makadia, and Hector Yee.  Label partitioning for sublinear ranking.  In International Conference on Machine Learning, pages 181–189, 2013.
  * (141) Qingyao Wu, Mingkui Tan, Hengjie Song, Jian Chen, and Michael K Ng.  Ml-forest: A multi-label tree ensemble method for multi-label classification.  IEEE transactions on knowledge and data engineering, 28(10):2665–2680, 2016.
  * (142) Xi-Zhu Wu and Zhi-Hua Zhou.  A unified view of multi-label performance measures.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3780–3788. JMLR. org, 2017.
  * (143) Yu-Ping Wu and Hsuan-Tien Lin.  Progressive random k-labelsets for cost-sensitive multi-label classification.  Machine Learning, 106(5):671–694, 2017.
  * (144) Yongqin Xian, Bernt Schiele, and Zeynep Akata.  Zero-shot learning-the good, the bad and the ugly.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4582–4591, 2017.
  * (145) Jianhua Xu.  Fast multi-label core vector machine.  Pattern Recognition, 46(3):885–898, 2013.
  * (146) Suping Xu, Xibei Yang, Hualong Yu, Dong-Jun Yu, Jingyu Yang, and Eric CC Tsang.  Multi-label learning with label-specific feature reduction.  Knowledge-Based Systems, 104:52–61, 2016.
  * (147) Yan Xu, Liping Jiao, Siyu Wang, Junsheng Wei, Yubo Fan, Maode Lai, and Eric I-chao Chang.  Multi-label classification for colon cancer using histopathological images.  Microscopy Research and Technique, 76(12):1266–1277, 2013.
  * (148) Fengqi Yan, Xin Huang, Yao Yao, Mingming Lu, and Maozhen Li.  Combining lstm and densenet for automatic annotation and classification of chest x-ray images.  IEEE Access, 7:74181–74189, 2019.
  * (149) Yan Yan, Ying Wang, Wen-Chao Gao, Bo-Wen Zhang, Chun Yang, and Xu-Cheng Yin.  Lstm: Multi-label ranking for document classification.  Neural Processing Letters, 47(1):117–138, 2018.
  * (150) Z. Yan, W. Liu, S. Wen, and Y. Yang.  Multi-label image classification by feature attention network.  IEEE Access, 7:98005–98013, 2019.
  * (151) Hao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao, Jianxin Wu, and Jianfei Cai.  Exploit bounding box annotations for multi-label object recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 280–288, 2016.
  * (152) Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang.  Sgm: Sequence generation model for multi-label classification.  In Proceedings of the 27th International Conference on Computational Linguistics, pages 3915–3926, 2018.
  * (153) Yiming Yang and Siddharth Gopal.  Multilabel classification with meta-level features in a learning-to-rank framework.  Machine Learning, 88(1-2):47–68, 2012.
  * (154) Yiyu Yao.  The superiority of three-way decisions in probabilistic rough set models.  Information Sciences, 181(6):1080–1096, 2011.
  * (155) Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, and Yu-Chiang Frank Wang.  Learning deep latent space for multi-label classification.  In Thirty-First AAAI Conference on Artificial Intelligence, 2017\.
  * (156) Ian En-Hsu Yen, Xiangru Huang, Pradeep Ravikumar, Kai Zhong, and Inderjit Dhillon.  Pd-sparse: A primal and dual sparse approach to extreme multiclass and multilabel classification.  In International Conference on Machine Learning, pages 3069–3077, 2016.
  * (157) Junjie Zhang, Qi Wu, Chunhua Shen, Jian Zhang, and Jianfeng Lu.  Multilabel image classification with regional latent semantic dependencies.  IEEE Transactions on Multimedia, 20(10):2801–2813, 2018.
  * (158) Min-Ling Zhang, Yu-Kun Li, Xu-Ying Liu, and Xin Geng.  Binary relevance for multi-label learning: an overview.  Frontiers of Computer Science, 12(2):191–202, 2018.
  * (159) Min-Ling Zhang and Zhi-Hua Zhou.  Multilabel neural networks with applications to functional genomics and text categorization.  IEEE transactions on Knowledge and Data Engineering, 18(10):1338–1351, 2006.
  * (160) Min-Ling Zhang and Zhi-Hua Zhou.  Ml-knn: A lazy learning approach to multi-label learning.  Pattern recognition, 40(7):2038–2048, 2007.
  * (161) Min-Ling Zhang and Zhi-Hua Zhou.  A review on multi-label learning algorithms.  IEEE transactions on knowledge and data engineering, 26(8):1819–1837, 2013.
  * (162) Ping Zhang, Guixia Liu, and Wanfu Gao.  Distinguishing two types of labels for multi-label feature selection.  Pattern Recognition, 2019.
  * (163) Wenjie Zhang, Junchi Yan, Xiangfeng Wang, and Hongyuan Zha.  Deep extreme multi-label learning.  In Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, pages 100–107. ACM, 2018.
  * (164) Yuanjian Zhang, Duoqian Miao, Zhifei Zhang, Jianfeng Xu, and Sheng Luo.  A three-way selective ensemble model for multi-label classification.  International Journal of Approximate Reasoning, 103:394 – 413, 2018\.
  * (165) Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun.  Multi-view learning overview: Recent progress and new challenges.  Information Fusion, 38:43–54, 2017.
  * (166) Yangming Zhou, Yangguang Liu, Jiangang Yang, Xiaoqi He, and Liangliang Liu.  A taxonomy of label ranking algorithms.  Journal of Computers, 9(3):557–565, 2014.
  * (167) Yangming Zhou and Guoping Qiu.  Random forest for label ranking.  Expert Systems with Applications, 112:99 – 109, 2018.
  * (168) Zhi-Hua Zhou, Min-Ling Zhang, Sheng-Jun Huang, and Yu-Feng Li.  Multi-instance multi-label learning.  Artificial Intelligence, 176(1):2291–2320, 2012.
  * (169) Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang.  Learning spatial regularization with image-level supervisions for multi-label image classification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5513–5522, 2017.
  * (170) Yue Zhu, Kai Ming Ting, and Zhi-Hua Zhou.  Multi-label learning with emerging new labels.  IEEE Transactions on Knowledge and Data Engineering, 30(10):1901–1914, 2018.