# Composer Style Classification of Piano Sheet Music Images Using Language
Model Pretraining

###### Abstract

This paper studies composer style classification of piano sheet music images.
Previous approaches to the composer classification task have been limited by a
scarcity of data. We address this issue in two ways: (1) we recast the problem
to be based on raw sheet music images rather than a symbolic music format, and
(2) we propose an approach that can be trained on unlabeled data. Our approach
first converts the sheet music image into a sequence of musical “words" based
on the bootleg feature representation, and then feeds the sequence into a text
classifier. We show that it is possible to significantly improve classifier
performance by first training a language model on a set of unlabeled data,
initializing the classifier with the pretrained language model weights, and
then finetuning the classifier on a small amount of labeled data. We train
AWD-LSTM, GPT-2, and RoBERTa language models on all piano sheet music images
in IMSLP. We find that transformer-based architectures outperform CNN and LSTM
models, and pretraining boosts classification accuracy for the GPT-2 model
from 46% to 70% on a 9-way classification task. The trained model can also be
used as a feature extractor that projects piano sheet music into a feature
space that characterizes compositional style.

##  1 Introduction

We’ve all had the experience of hearing a piece of music that we’ve never
heard before, but immediately recognizing the composer based on the piece’s
style. This paper explores this phenomenon in the context of sheet music. The
question that we want to answer is: “Can we predict the composer of a
previously unseen page of piano sheet music based on its compositional style?"

Many previous works have studied the composer classification problem. These
works generally fall into one of two categories. The first category of
approach is to construct a set of features from the music, and then feed the
features into a classifier. Many works use manually designed features that
capture musically meaningful information (e.g. [15][27][1][9]). Other works
feed minimally preprocessed representations of the data (e.g. 2-D piano rolls
[33][34] or tensors encoding note pitch & duration information [35][3]) into a
convolutional model, and allow the model to learn a useful feature
representation. The second category of approach is to train one model for each
composer, and then select the model that has the highest likelihood of
generating a given sequence of music. Common approaches in this category
include N-gram language models [36][11][10] and Markov models [14][23].

Our approach to the composer classification task addresses what we perceive to
be the biggest common obstacle to the above approaches: lack of data. All of
the above approaches assume that the input is in the form of a symbolic music
file (e.g. MIDI or **kern). Because symbolic music formats are much less
widely used than audio, video, and image formats, the amount of training data
that is available is quite limited. We address this issue of data scarcity in
two ways: (1) we re-define the composer classification task to be based on
sheet music images, for which there is a lot of data available online, and (2)
we propose an approach that can be trained on unlabeled data.

Our work takes advantage of recent developments in transfer learning in the
natural language processing (NLP) community. Prior to 2017, transfer learning
in NLP was done in a limited way. Typically, one would use pretrained word
embeddings such as word2vec [20][19] or GloVe [22] vectors as the first layer
in a model. The problem with this paradigm of transfer learning is that the
entire model except the first layer needs to be trained from scratch, which
requires a large amount of labeled data. This is in contrast to the paradigm
of transfer learning in computer vision, where a model is trained on the
ImageNet classification task [26], the final layer is replaced with a
different linear classifier, and the model is finetuned for a different task.
The benefit of this latter paradigm of transfer learning is that the entire
model except the last layer is pretrained, so it can be finetuned with only a
small amount of labeled data. This paradigm of transfer learning has been
widely used in computer vision in the last decade [39] using pretrained models
like VGG [29], ResNet [8], Densenet [13], etc. The switch to ImageNet-style
transfer learning in the NLP community occurred in 2017, when Howard et al.
[12] proposed a way to pretrain an LSTM-based language model on a large set of
unlabeled data, add a classification head on top of the language model, and
then finetune the classifier on a new task with a small amount of labeled
data. This was quickly followed by several other similar language model
pretraining approaches that replaced the LSTM with transformer-based
architectures (e.g. GPT [24], GPT-2 [25], BERT [5]). These pretrained language
models have provided the basis for achieving state-of-the-art results on a
variety of NLP tasks, and have been extended in various ways (e.g.
Transformer-XL [4], XLNet [38]).

Our approach is similarly based on language model pretraining. We first
convert each sheet music image into a sequence of words based on the bootleg
score feature representation [37]. We then feed this sequence of words into a
text classifier. We show that it is possible to significantly improve the
performance of the classifier by training a language model on a large set of
unlabeled data, initialize the classifier with the pretrained language model
weights, and finetune the classifier on a small amount of labeled data. In our
experiments, we train language models on all piano sheet music images in the
International Music Score Library Project (IMSLP)111http://imslp.org/ using
the AWD-LSTM [18], GPT-2 [25], and RoBERTa [16] language model architectures.
By using pretraining, we are able to improve the accuracy of our GPT-2 model
from 46% to 70% on a 9-way classification task.222Code can be found at
https://github.com/tjtsai/PianoStyleEmbedding.

Figure 1: Overview of proxy classifier training. A language model is first
trained on a set of unlabeled data, the classifier is initialized with the
pretrained language model weights, and then the classifier is finetuned on a
small set of labeled data.

##  2 System Description

We will describe our system in the next four subsections. In the first
subsection, we give a high-level overview and rationale behind our approach.
In the following three subsections, we describe the three main stages of
system development: language model pretraining, classifier finetuning, and
inference.

###  2.1 Overview

Figure 1 summarizes our training approach. In the first stage, we convert each
sheet music image into a sequence of words based on the bootleg score
representation [37], and then train a language model on these words. Since
this task does not require labels, we can train our language model on a large
set of unlabeled data. In this work, we train our language model on all piano
sheet music images in the IMSLP dataset. In the second stage, we train a
classifier that predicts the composer of a short fragment of music, where the
fragment is a fixed-length sequence of symbolic words. We do this by adding
one or more dense layers on top of the language model, initializing the
weights of the classifier with the language model weights, and then finetuning
the model on a set of labeled data. In the third stage, we use the classifier
to predict the composer of an unseen scanned page of piano sheet music. We do
this by converting the sheet music image to a sequence of symbolic words, and
then either (a) applying the classifier to a single variable length input
sequence, or (b) averaging the predictions of fixed-length crops sampled from
the input sequence. We will describe each of these three stages in more detail
in the following three subsections.

Figure 2: A short section of sheet music and its corresponding bootleg score.
Staff lines in the bootleg score are shown for reference, but are not present
in the actual feature representation.

The guiding principle behind our approach is to maximize the amount of data.
This impacts our approach in three significant ways. First, it informs our
choice of data format. Rather than using symbolic scores (as in previous
approaches), we instead choose to use raw sheet music images. While this
arguably makes the task much more challenging, it has the benefit of having
much more data available online. Second, we choose an approach that can
utilize unlabeled data. Whereas labeled data is usually expensive to annotate
and limited in quantity, unlabeled data is often extremely cheap and available
in abundance. By adopting an approach that can use unlabeled data, we can
drastically increase the amount of data available to train our models. Third,
we use data augmentation to make the most of the limited quantity of labeled
data that we do have. Rather than fixating on the page classification task, we
instead define a proxy task where the goal is to predict the composer given a
fixed-length sequence of symbolic words. By defining the proxy task in this
way, we can aggressively subsample fragments from the labeled data, resulting
in a much larger number of unique training data points than there are actual
pages of sheet music. Once the proxy task classifier has been trained, we can
apply it to the full page classification task in a straightforward manner.

###  2.2 Language Model Pretraining

The language model pretraining consists of three steps, as shown in the upper
half of Figure 1. These three steps will be described in the next three
paragraphs.

Figure 3: Overview of AWD-LSTM, GPT-2, and RoBERTa language models (top) and
classifiers (bottom). Boxes in blue are trained during the language modeling
phase and used to initialize the classifier.

The first step is to convert the sheet music image into a bootleg score. The
bootleg score is a low-dimensional feature representation of piano sheet music
that encodes the position of filled noteheads relative to the staff lines
[37]. Figure 2 shows an example of a section of sheet music and its
corresponding bootleg score representation. The bootleg score itself is a
$62\times N$ binary matrix, where $62$ indicates the total number of possible
staff line positions in both the left and right hands, and where $N$ indicates
the total number of estimated simultaneous note onset events. Note that the
representation discards a significant amount of information: it does not
encode note duration, key signature, time signature, measure boundaries,
accidentals, clef changes, or octave markings, and it simply ignores non-
filled noteheads (e.g. half or whole notes). Nonetheless, it has been shown to
be effective in aligning sheet music and MIDI [37], and we hypothesize that it
may also be useful in characterizing piano style. The main benefit of using
the bootleg score representation over a full optical music recognition (OMR)
pipeline is processing time: computing a bootleg score only takes about $1$
second per page using a CPU, which makes it suitable for computing features on
the entire IMSLP dataset.333In contrast, the best performing music object
detectors take 40-80 seconds to process each page at inference time using a
GPU [21]. We use the code from [37] as a fixed feature extractor to compute
the bootleg scores.

The second step is to tokenize the bootleg score into a sequence of word or
subword units. We do this differently for different language models. For word-
based language models (e.g. AWD-LSTM [18]), we consider each bootleg score
column as a single word consisting of a 62-character string of 0s and 1s. We
limit the vocabulary to the $30,000$ most frequent words, and map infrequent
words to a special unknown word token <unk>. For subword-based language models
(e.g. GPT-2 [25], RoBERTa [16]), we use a byte pair encoding (BPE) algorithm
[6] to learn a vocabulary of subword units in an unsupervised manner. The BPE
algorithm starts with an initial set of subword units (e.g. the set of unique
characters [28] or the $2^{8}=256$ unique byte values that comprise unicode
characters [7]), and it iteratively merges the most frequently occurring pair
of adjacent subword units until a desired vocabulary size has been reached. We
experimented with both character-level and byte-level encoding schemes (i.e.
representing each word as a string of 62 characters vs. a sequence of 8
bytes), and we found that the byte-level encoding scheme performs much better.
We only report results with the byte-level BPE tokenizer. For both subword-
based language models explored in this work, we use the same shared BPE
tokenizer with a vocabulary size of $30,000$ (which is the vocabularly size
used in the RoBERTa model). At the end of the second step, we have represented
the sheet music image as a sequence of words or subword units.

The third step is to train a language model on a set of unlabeled data. In
this work, we explore three different language models, which are
representative of state-of-the-art models in the last 3-4 years. The top half
of Figure 3 shows a high-level overview of these three language models. The
first model is AWD-LSTM [18]. This is a 3-layer LSTM architecture that makes
heavy use of regularization techniques throughout the model, including four
different types of dropout. The output of the final LSTM layer is fed to a
linear decoder whose weights are tied to the input embedding matrix. This
produces an output distribution across the tokens in the vocabulary. The model
is then trained to predict the next token at each time step. We use the fastai
implementation of the AWD-LSTM model with default parameters. The second model
is openAI’s GPT-2 [25]. This architecture consists of multiple transformer
decoder layers [32]. Each transformer decoder layer consists of a masked self-
attention, along with feedforwards layers, layer normalizations, and residual
connections. While transformer encoder layers allow each token to attend to
all other tokens in the input, the transformer decoder layers only allow a
token to attend to previous tokens.444This is because, in the original machine
translation task [32], the decoder generates the output sentence
autoregressively. Similar to the AWD-LSTM model, the outputs of the last
transformer layer are fed to a linear decoder whose weights are tied to the
input embeddings, and the model is trained to predict the next token at each
time step. We use the huggingface implementation of the GPT-2 model with
default parameters, except that we reduce the vocabulary size from $50,000$ to
$30,000$ (to use the same tokenizer as the RoBERTa model), the amount of
context from 1024 to 512, and the number of layers from 12 to 6. The third
model is RoBERTa [16], which is based on Google’s BERT language model [5].
This architecture consists of multiple transformer encoder layers. Unlike
GPT-2, each token can attend to all other tokens in the input and the goal is
not to predict the next token. Instead, a certain fraction of the input tokens
are randomly converted to a special <mask> token, and the model is trained to
predict the masked tokens. We use the huggingface implementation of RoBERTa
with default parameter settings, except that we reduce the number of layers
from 12 to 6.

###  2.3 Classifier Finetuning

In the second main stage, we finetune a classifier based on a set of labeled
data. The labeled data consists of a set of sheet music images along with
their corresponding composer labels. The process of training the classifier is
comprised of four steps (lower half of Figure 1).

The first two steps are to compute and tokenize a bootleg score into a
sequence of symbolic words. We use the same fixed feature extractor and the
same tokenizer that were used in the language model pretraining stage.

The third step is to sample short, fixed-length fragments of words from the
labeled data. As mentioned in Section 2.1, we define a proxy task where the
goal is to predict the composer given a short, fixed-length fragment of words.
Defining the proxy task in this way has three significant benefits: (1) we can
use sampling to generate many more unique training data points than there are
actual pages of sheet music in our dataset, (2) we can sample the data in such
a way that the classes are balanced, which avoids problems during training,
and (3) using fixed-length inputs allows us to train more efficiently in
batches. Our approach follows the general recommendations of a recent study on
best practices for training a classifier with imbalanced data [2]. Each
sampled fragment and its corresponding composer label constitute a single
$(X_{i},y_{i})$ training pair for the proxy task.

The fourth step is to train the classifier model. The bottom half of Figure 3
shows how this is done with our three models. Our general approach is to add a
classifier head on top of the language model, initialize the weights of the
classifier with the pretrained language model weights, and then finetune the
classifier on the proxy task data. For the AWD-LSTM, we take the outputs from
the last LSTM layer and construct a fixed-size representation by concatenating
three things: (a) the output at the last time step, (b) the result of max
pooling the outputs across the sequence dimension, and (c) the result of
average pooling the outputs across the sequence dimension. This fixed-size
representation (which is three times the hidden dimension size) is then fed
into the classifier head, which consists of two dense layers with batch
normalization and dropout. For the GPT-2 model, we take the output from the
last transformer layer at the last time step, and then feed it into a single
dense (classification) layer. Because the GPT-2 and RoBERTa models require
special tokens during training, we insert special symbols <s> and </s> at the
beginning and end of every training input, respectively. Because of the masked
self-attention, we must use the output of the last token in order to access
all of the information in the input sequence. For the RoBERTa model, we take
the output from the last transformer layer corresponding to the <s> token, and
feed it into a single dense (classification) layer. The <s> takes the place of
the special [CLS] token described in the original paper.

We integrated all models into the fastai framework and finetuned the
classifier in the following manner. We first select an appropriate learning
rate using a range test, in which we sweep the learning rate across a wide
range of values and observe the impact on training loss. We initially freeze
all parameters in the model except for the untrained classification head, and
we gradually unfreeze more and more layers in the model as the training
converges. To avoid overly aggressive changes to the pretrained language model
weights, we use discriminative finetuning, in which earlier layers of the
model use exponentially smaller learning rates compared to later layers in the
model. All training is done with (multiple cycles of) the one cycle training
policy [30], in which learning rate and momentum are varied cyclically over
each cycle. The above practices were proposed in [12] and found to be
effective in finetuning language models for text classification.

###  2.4 Inference

The third main stage is to apply the proxy classifier to the original full
page classification task. We explore two different ways to do this. The first
method is to convert the sheet music image into a bootleg score, tokenize the
bootleg score into a sequence of word or subword units, and then apply the
proxy classifier to a single variable-length input. Note that all of the
models can handle variable-length inputs up to a maximum context length. The
second method is identical to the first, except that it averages the
predictions from multiple fixed-length crops taken from the input sequence.
The fixed-length crops are the same size as is used during classifier
training, and the crops are sampled uniformly with $50\%$ overlap.555We also
experimented with applying a Bayesian prior to the classifier softmax outputs,
as recommended in [2], but found that the results were not consistently
better.

##  3 Experimental Setup

In this section we describe the data collection process and the metrics used
to evaluate our approach.

Figure 4: Statistics on the target dataset. The top two histograms show the
distribution of the number of pages (top left) and number of bootleg score
features (top right) per composer. The bottom figure shows the distribution of
the number of bootleg score features per page.

The data comes from IMSLP. We first scraped the website and downloaded all PDF
scores and accompanying metadata.666We downloaded the data over a span of
several weeks in May of 2018. We filtered the data based on its
instrumentation in order to identify a list of solo piano scores. We then
computed bootleg score features for all of the piano sheet music images using
the XSEDE supercomputing infrastructure [31], and discarded any pages that had
less than a minimum threshold of features. This latter step is designed to
remove non-music pages such as the title page, foreword, or table of contents.
The resulting set of data contained $29,310$ PDFs,777Note that a PDF may
contain multiple pieces (e.g. the complete set of Chopin etudes). $255,539$
pages and a total of $48.5$ million bootleg score features. This set of data
is what we refer to as the IMSLP dataset in this work (e.g. the IMSLP
pretrained language model). For language model training, we split the IMSLP
data by piece, using $90\%$ for training and $10\%$ for validation.

The classification task uses a subset of the IMSLP data. We first identified a
list of composers with a significant amount of data (composers shown in Figure
4). We limited the list to nine composers in order to avoid extreme class
imbalance. Because popular pieces tend to have many sheet music versions in
the dataset, we select one version per piece in order to avoid over-
representation of a small subset of pieces. Next, we manually labeled and
discarded all filler pages, and then computed bootleg score features on the
remaining sheet music images. This cleaned dataset is what we refer to as the
target data in this work (e.g. the target pretrained language model). Figure 4
shows the total number of pages and bootleg score features per composer for
the target dataset, along with the distribution of the number of bootleg score
features per page. For training and testing, we split the data by piece, using
$60\%$ of the pieces for training ($4347$ pages), $20\%$ for validation
($1500$ pages), and $20\%$ for testing ($1304$ pages). To generate data for
the proxy task, we randomly sampled fixed-length fragments from the target
data. We sample the same number of fragments for each composer to ensure class
balance. We experimented with fragment sizes of 64/128/256 and sampled
32400/16200/8100 fragments for training and 10800/5400/2700 fragments for
validation/test, respectively. This sampling scheme ensures the same data
coverage regardless of fragment length. Note that the classification data is
carefully curated, while the IMSLP data requires minimal processing.

We use two different metrics to evaluate our systems. For the proxy task,
accuracy is an appropriate metric since the data is balanced. For the full
page classification task – which has imbalanced data – we report results in
macro F1 score. Macro F1 is a generalization of F1 score to a multi-class
setting, in which each class is treated as a one-versus-all binary
classification task and the F1 scores from all classes are averaged.

##  4 Results & Analysis

In this section we present our experimental results and conduct various
analyses to answer key questions of interest. While the proxy task is an
artificially created task, it provides a more reliable indicator of classifier
performance than the full page classification. This is because the test set of
the full page classification task is both imbalanced and very small ($1304$
data points). Accordingly, we will report results on both the proxy task and
full page classification task.

###  4.1 Proxy Task

We first consider the performance of our models on the proxy classification
task. We would like to understand the effect of (a) model architecture, (b)
pretraining condition, and (c) fragment size.

Figure 5: Model performance on the proxy classification task. This comparison
shows the effect of different pretraining conditions and fragment sizes.

We evaluate four different model architectures. In addition to the AWD-LSTM,
GPT-2, and RoBERTa models previously described, we also measure the
performance of a CNN-based approach recently proposed in [35]. Note that we
cannot use the exact same model in [35] since we do not have symbolic score
information. Nonetheless, we can use the same general approach of computing
local features, aggregating feature statistics across time, and applying a
linear classifier. The design of our 2-layer CNN model roughly matches the
architecture proposed in [35].

We consider three different language model pretraining conditions. The first
condition is with no pretraining, where we train the classifier from scratch
only on the proxy task. The second condition is with target language model
pretraining, where we first train a language model on the target data, and
then finetune the classifier on the proxy task. The third condition is with
IMSLP language model pretraining. Here, we train a language model on the full
IMSLP dataset, finetune the language model on the target data, and then
finetune the classifier on the proxy task.

Figure 5 shows the performance of all models on the proxy task. There are
three things to notice. First, regarding (a), the transformer-based models
generally outperform the LSTM and CNN models. Second, regarding (b), language
model pretraining improves performance significantly across the board.
Regardless of architecture, we see a large improvement going from no
pretraining (condition 1) to target pretraining (condition 2), and another
large improvement going from target pretraining (condition 2) to IMSLP
pretraining (condition 3). For example, the performance of the GPT-2 model
increases from $37.3\%$ to $45.2\%$ to $57.5\%$ across the three pretraining
conditions. Because the data in conditions 1 & 2 is exactly the same, the
improvement in performance must be coming from more effective use of the data.
We can interpret this from an information theory perspective by noting that
the classification task provides the model $log_{2}9=3.17$ bits of information
per fragment, whereas the language modeling task provides $log_{2}V$ bits of
information per bootleg score feature where $V$ is the vocabulary size. The
performance gap between condition 2 and condition 3 can also be interpreted as
the result of providing more information to the model, but here the
information is coming from having additional data. Third, regarding (c),
larger fragments result in better performance, as we might expect.

Figure 6: Results on the full page classification task.

###  4.2 Full Page Classification

Next, we consider performance of our models on the full page classification
task. We would like to understand the effect of (a) model architecture, (b)
pretraining condition, (c) fragment size, and (d) inference type (single vs.
multi-crop). Regarding (d), we found that taking multiple crops improved
results with all models except the CNN. This suggests that this type of test
time augmentation does not benefit approaches that simply average feature
statistics over time. In the results presented below, we only show the optimal
inference type for each model architecture (i.e. CNN with single crop, all
others with multi-crop).

Figure 6 shows model performance on the full page classification task. There
are two things to notice. First, we see the same general trends as in Figure 5
for model architecture and pretraining condition: the transformer-based models
generally outperform the CNN and LSTM models, and pretraining helps
substantially in every case. The macro F1 score of our best model (GPT-2 with
fragment size 64) increases from $0.41$ to $0.51$ to $0.67$ across the three
pretraining conditions. Second, we see the opposite trend as the proxy task
for fragment size: smaller fragments have better page classification
performance. This strongly indicates a data distribution mismatch. Indeed,
when we look at the distribution of the number of bootleg score features in a
single page (Figure 4), we see that a significant fraction of pages have less
than 256 features. Because we only sample fragments that contain a complete
set of 256 words, our proxy task data is biased towards longer inputs. This
leads to poor performance when the classifier is faced with short inputs,
which are never seen in training. Using a fragment size of 64 minimizes this
bias.

Figure 7: t-SNE plot of the RoBERTa model activations for five novel
composers. Each data point corresponds to a single page of sheet music for a
composer that was not considered in the classification task.

###  4.3 t-SNE Plots

Another key question of interest is, “Can we use our model to characterize the
style of any page of piano sheet music?" The classification task forces the
model to project the sheet music into a feature space where the compositional
style of the nine composers can be differentiated. We hypothesize that this
feature space might be useful in characterizing the style of any page of piano
sheet music, even from composers not in the classification task.

To test this hypothesis, we fed data from 5 novel composers into our models
and constructed t-SNE plots[17] of the activations at the second-to-last
layer. Figure 7 shows such a plot for the RoBERTa model. Each data point
corresponds to a single page of sheet music from a novel composer. Even though
we have not trained the classifier to distinguish between these five
composers, we can see that the data points are still clustered, suggesting
that the feature space can describe the style of new composers in a useful
manner.

##  5 Conclusion

We propose a method for predicting the composer of a single page of piano
sheet music. Our method first converts the raw sheet music image into a
bootleg score, tokenizes the bootleg score into a sequence of musical words,
and then feeds the sequence into a text classifier. We show that by
pretraining a language model on a large set of unlabeled data, it is possible
to significantly improve the performance of the classifier. We also show that
our trained model can be used as a feature extractor to characterize the style
of any page of piano sheet music. For future work, we would like to explore
other forms of data augmentation and other model architectures that explicitly
encode musical knowledge.

##  6 Acknowledgments

This work used the Extreme Science and Engineering Discovery Environment
(XSEDE), which is supported by National Science Foundation grant number
ACI-1548562. Large-scale computations on IMSLP data were performed with XSEDE
Bridges at the Pittsburgh Supercomputing Center through allocation TG-
IRI190019. We also gratefully acknowledge the support of NVIDIA Corporation
with the donation of the GPU used for training the models.

## References

  * [1] Andrew Brinkman, Daniel Shanahan, and Craig Sapp.  Musical stylometry, machine learning and attribution studies: A semi-supervised approach to the works of josquin.  In Proc. of the Biennial Int. Conf. on Music Perception and Cognition, pages 91–97, 2016.
  * [2] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski.  A systematic study of the class imbalance problem in convolutional neural networks.  Neural Networks, 106:249–259, 2018.
  * [3] Giuseppe Buzzanca.  A supervised learning approach to musical style recognition.  In Proc. of the international conference on music and artificial intelligence (ICMAI), volume 2002, page 167, 2002.
  * [4] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive language models beyond a fixed-length context.  In Proc. of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, 2019.
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [6] Philip Gage.  A new algorithm for data compression.  C Users Journal, 12(2):23–38, 1994.
  * [7] Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya.  Multilingual language processing from bytes.  In Proc. of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1296–1306, 2016.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proc. of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [9] Dorien Herremans, David Martens, and Kenneth Sörensen.  Composer classification models for music-theory building.  In Computational Music Analysis, pages 369–392. 2016.
  * [10] Ruben Hillewaere, Bernard Manderick, and Darrell Conklin.  String quartet classification with monophonic models.  In Proc. of the International Society for Music Information Retrieval Conference (ISMIR), pages 537–542, 2010.
  * [11] María Hontanilla, Carlos Pérez-Sancho, and Jose M Inesta.  Modeling musical style with language models for composer recognition.  In Iberian Conference on Pattern Recognition and Image Analysis, pages 740–748, 2013.
  * [12] Jeremy Howard and Sebastian Ruder.  Universal language model fine-tuning for text classification.  In Proc. of the 56th Annual Meeting of the Association for Computational Linguistics, pages 328–339, 2018.
  * [13] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proc. of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [14] Maximos A Kaliakatsos-Papakostas, Michael G Epitropakis, and Michael N Vrahatis.  Weighted markov chain model for musical composer identification.  In European Conference on the Applications of Evolutionary Computation, pages 334–343, 2011.
  * [15] Katherine C Kempfert and Samuel WK Wong.  Where does haydn end and mozart begin? composer classification of string quartets.  arXiv preprint arXiv:1809.05075, 2018.
  * [16] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A robustly optimized BERT pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [17] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  Journal of machine learning research, 9(Nov):2579–2605, 2008.
  * [18] Stephen Merity, Nitish Shirish Keskar, and Richard Socher.  Regularizing and optimizing LSTM language models.  arXiv preprint arXiv:1708.02182, 2017.
  * [19] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.
  * [20] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119, 2013.
  * [21] Alexander Pacha, Jan Hajič, and Jorge Calvo-Zaragoza.  A baseline for general music object detection with deep learning.  Applied Sciences, 8(9):1488, 2018.
  * [22] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  GloVe: Global vectors for word representation.  In Proc. of the conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [23] Emanuele Pollastri and Giuliano Simoncelli.  Classification of melodies by composer with hidden markov models.  In Proc. of the First International Conference on WEB Delivering of Music, pages 88–95, 2001.
  * [24] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  OpenAI Blog, 2018.
  * [25] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  OpenAI Blog, 1(8):9, 2019.
  * [26] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Fei-Fei Li.  ImageNet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [27] Pasha Sadeghian, Casey Wilson, Stephen Goeddel, and Aspen Olmsted.  Classification of music by composer using fuzzy min-max neural networks.  In Proc. of the 12th International Conference for Internet Technology and Secured Transactions (ICITST), pages 189–192, 2017.
  * [28] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In Proc. of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725, 2016.
  * [29] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [30] Leslie N Smith.  A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay.  arXiv preprint arXiv:1803.09820, 2018.
  * [31] J. Towns, T. Cockerill, M. Dahan, I. Foster, K. Gaither, A. Grimshaw, V. Hazlewood, S. Lathrop, D. Lifka, G. D. Peterson, R. Roskies, J. R. Scott, and N. Wilkins-Diehr.  XSEDE: Accelerating scientific discovery.  Computing in Science & Engineering, 16(5):62–74, Sept.-Oct. 2014\.
  * [32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [33] Gissel Velarde, Carlos Cancino Chacón, David Meredith, Tillman Weyde, and Maarten Grachten.  Convolution-based classification of audio and symbolic representations of music.  Journal of New Music Research, 47(3):191–205, 2018.
  * [34] Gissel Velarde, Tillman Weyde, Carlos Eduardo Cancino Chacón, David Meredith, and Maarten Grachten.  Composer recognition based on 2d-filtered piano-rolls.  In Proc. of the International Society for Music Information Retrieval Conference (ISMIR), pages 115–121, 2016.
  * [35] Harsh Verma and John Thickstun.  Convolutional composer classification.  In Proc. of the International Society for Music Information Retrieval Conference (ISMIR), pages 549–556, 2019.
  * [36] Jacek Wołkowicz and Vlado Kešelj.  Evaluation of N-gram-based classification approaches on classical music corpora.  In International Conference on Mathematics and Computation in Music, pages 213–225, 2013.
  * [37] Daniel Yang, Thitaree Tanprasert, Teerapat Jenrungrot, Mengyi Shan, and TJ Tsai.  MIDI passage retrieval using cell phone pictures of sheet music.  In Proc. of the International Society for Music Information Retrieval Conference (ISMIR), pages 916–923, 2019.
  * [38] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  XLNet: Generalized autoregressive pretraining for language understanding.  In Advances in neural information processing systems, pages 5754–5764, 2019.
  * [39] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  In Advances in neural information processing systems, pages 3320–3328, 2014.


#  IEEE International Conference on Computer Vision (ICCV) 2019
EGNet: Edge Guidance Network for Salient Object Detection

Jia-Xing Zhao, Jiang-Jiang Liu, Deng-Ping Fan, Yang Cao, Ju-Feng Yang, Ming-
Ming Cheng
TKLNDST, CS, Nankai University
http://mmcheng.net/egnet/ M.M. Cheng<EMAIL_ADDRESS>is the corresponding
author.

###### Abstract

Fully convolutional neural networks (FCNs) have shown their advantages in the
salient object detection task. However, most existing FCNs-based methods still
suffer from coarse object boundaries. In this paper, to solve this problem, we
focus on the complementarity between salient edge information and salient
object information. Accordingly, we present an edge guidance network (EGNet)
for salient object detection with three steps to simultaneously model these
two kinds of complementary information in a single network. In the first step,
we extract the salient object features by a progressive fusion way. In the
second step, we integrate the local edge information and global location
information to obtain the salient edge features. Finally, to sufficiently
leverage these complementary features, we couple the same salient edge
features with salient object features at various resolutions. Benefiting from
the rich edge information and location information in salient edge features,
the fused features can help locate salient objects, especially their
boundaries more accurately. Experimental results demonstrate that the proposed
method performs favorably against the state-of-the-art methods on six widely
used datasets without any pre-processing and post-processing. The source code
is available at http://mmcheng.net/egnet/.

##  1 Introduction

The goal of salient object detection (SOD) is to find the most visually
distinctive objects in an image. It has received widespread attention recently
and been widely used in many vision and image processing related areas, such
as content-aware image editing [6], object recognition [42], photosynth [4],
non-photo-realist rendering [41], weakly supervised semantic segmentation [19]
and image retrieval [15]. Besides, there are many works focusing on video
salient object detection [12, 54] and RGB-D salient object detection [11, 66].

|  |  |
---|---|---|---
|  |  |
Source | Baseline | Ours | GT
Figure 1: Visual examples of our method. After we model and fuse the salient
edge information, the salient object boundaries become clearer.

Inspired by cognitive studies of visual attention [21, 39, 7], early works are
mainly based on the fact that contrast plays the most important role in
saliency detection. These methods benefit mostly from either global or local
contrast cues and their learned fusion weights. Unfortunately, these hand-
crafted features though can locate the most salient objects sometimes, the
produced saliency maps are with irregular shapes because of the undesirable
segmentation methods and unreliable when the contrast between the foreground
and the background is inadequate.

Recently, convolutional neural networks (CNNs) [25] have successfully broken
the limits of traditional hand-crafted features, especially after the emerging
of Fully Convolutional Neural Networks (FCNs) [34]. These CNN-based methods
have greatly refreshed the leaderboards on almost all the widely used
benchmarks and are gradually replacing conventional salient object detection
methods because of the efficiency as well as high performance. In SOD
approaches based on CNNs architecture, the majority of them which regard the
image patches [65, 64] as input use the multi-scale or multi-context
information to obtain the final saliency map. Since the fully convolutional
network is proposed for pixel labeling problems, several end-to-end deep
architectures [18, 17, 28, 23, 31, 60, 67, 50] for salient object detection
appear. The basic unit of output saliency map becomes per pixel from the image
region. On the one hand, the result highlights the details because each pixel
has its saliency value. However, on the other hand, it ignores the structure
information which is important for SOD.

With the increase of the network receptive field, the positioning of salient
objects becomes more and more accurate. However, at the same time, spatial
coherence is also ignored. Recently, to obtain the fine edge details, some SOD
U-Net [40] based works [32, 61, 59, 33] used a bi-directional or recursive way
to refine the high-level features with the local information. However, the
boundaries of salient objects are still not explicitly modeled. The
complementarity between the salient edge information and salient object
information has not been noticed. Besides, there are some methods using pre-
processing (Superpixel) [20] or post-processing (CRF) [28, 17, 33] to preserve
the object boundaries. The main inconvenience with these approaches is their
low inference speed.

In this paper, we focus on the complementarity between salient edge
information and salient object information. We aim to leverage the salient
edge features to help the salient object features locate objects, especially
their boundaries more accurately. In summary, this paper makes three major
contributions:

  * •

We propose an EGNet to explicitly model complementary salient object
information and salient edge information within the network to preserve the
salient object boundaries. At the same time, the salient edge features are
also helpful for localization.

  * •

Our model jointly optimizes these two complementary tasks by allowing them to
mutually help each other, which significantly improves the predicted saliency
maps.

  * •

We compare the proposed methods with 15 state-of-the-art approaches on six
widely used datasets. Without bells and whistles, our method achieves the best
performance under three evaluation metrics.

Figure 2: The pipeline of the proposed approach. We use brown thick lines to
represent information flows between the scales. PSFEM: progressive salient
object features extraction module. NLSEM: non-local salient edge features
extraction module. O2OGM : one-to-one guidance module. FF: feature fusion.
Spv.: supervision.

##  2 Related Works

Over the past years, some methods were proposed to detect the salient objects
in an image. Early methods predicted the saliency map using a bottom-up
pattern by the hand-craft feature, such as contrast [5], boundary background
[57, 68], center prior [44, 24] and so on [22, 44, 51]. More details are
introduced in [2, 1, 9].

Recently, Convolutional neural networks (CNNs) perform their advantages and
refresh the state-of-the-art records in many fields of computer vision.

Li et al. [27] resized the image regions to three different scales to extract
the multi-scale features and then aggregated these multiple saliency maps to
obtain the final prediction map. Wang et al. [45] designed a neural network to
extract the local estimation for the input patches and integrated these
features with the global contrast and geometric information to describe the
image patches. However, the result is limited by the performance of image
patches in these methods. In [34], long et al. firstly proposed a network
(FCN) to predict the semantic label for each pixel. Inspired by FCN, more and
more pixel-wise saliency detection methods were proposed. Wang et al. [47]
proposed a recurrent FCN architecture for salient object detection. Hou et al.
proposed a short connection [17, 18] based on HED [55] to integrate the low-
level features and high-level features to solve the scale-space problem. In
[62], Zhang et al. introduced a reformulated dropout and an effective hybrid
upsampling to learn deep uncertain convolutional features to encourage
robustness and accuracy. In [61], Zhang et al. explicitly aggregated the
multi-level features into multiple resolutions and then combined these feature
maps by a bidirectional aggregation method. Zhang et al. [59] proposed a bi-
directional message-passing model to integrate multi-level features for
salient object detection. Wang et al. [53] leveraged the fixation maps to help
the model to locate the salient object more accurately. In [35], Luo et al.
proposed a U-Net based architecture which contains an IOU edge loss to
leverage the edge cues to detect the salient objects. In other saliency-
related tasks, some methods of using edge cues have appeared. In [26], li et
al. generated the contour of the object to obtain the salient instance
segmentation results. In [29], li et al. leveraged the well-trained contour
detection models to generate the saliency masks to overcome the limitation
caused by manual annotations.

Compared with most of the SOD U-Net based methods [32, 61, 59, 33], we
explicitly model edge information within the network to leverage the edge
cues. Compared with the methods which use the edge cues [14, 69, 58], the
major differences are that we use a single base network and jointly optimize
the salient edge detection and the salient object detection, allowing them to
help each other mutually. which results in better performance. Compared with
NLDF [35], they implemented a loss function inspired by the Mumford-Shah
function [38] to penalize errors on the edges. Since the salient edges are
derived from salient objects through a fixed sober operator, this penalty
essentially only affects the gradient in the neighborhood of salient edges on
feature maps. In this way, the edge details are optimized to some extent, but
the complementarity between salient edge detection and salient object
detection is not sufficiently utilized. In our method, we design two modules
to extract these two kinds of features independently. Then we fuse these
complementary features by a one-to-one guidance module. In this way, the
salient edge information can not only improve the quality of edges but also
make the localization more accurate. The experimental part verifies our
statement.

##  3 Salient Edge Guidance Network

The overall architecture is shown in Fig. 2. In this section, we begin by
describing the motivations in Sec. 3.1, then introduce the adopted salient
object feature extraction module and the proposed non-local salient edge
features extraction module in Sec. 3.2, and finally introduce the proposed
one-to-one guidance module in Sec. 3.3.

###  3.1 Motivation

The pixel-wise salient object detection methods have shown their advantages
compared with region-based methods. However, they ignored the spatial
coherence in the images, resulting in the unsatisfied salient object
boundaries. Most methods [17, 18, 33, 31, 61, 59] hope to solve this problem
by fusing multi-scale information. Some methods [17, 33, 28] used the post-
processing such as CRF to refine the salient object boundaries. In NLDF [35],
they proposed an IOU loss to affect the gradient of the location around the
edge. None of them pay attention to the complementarity between salient edge
detection and salient object detection. A good salient edge detection result
can help salient object detection task in both segmentation and localization,
and vice versa. Based on this idea, we proposed an EGNet to model and fuse the
complementary salient edge information and salient object information within a
single network in an end-to-end manner.

###  3.2 Complementary information modeling

Our proposed network is independent of the backbone network. Here we use the
VGG network suggested by other deep learning based methods [17, 35] to
describe the proposed method. First, we truncate the last three fully
connected layers. Following DSS [17, 18], we connect another side path to the
last pooling layer in VGG. Thus from the backbone network, we obtain six side
features Conv1-2, Conv2-2, Conv3-3, Conv4-3, Conv5-3, Conv6-3. Because the
Conv1-2 is too close to the input and the receptive field is too small, we
throw away this side path $S^{(1)}$. There are five side paths
$S^{(2)},S^{(3)},S^{(4)},S^{(5)},S^{(6)}$ remaining in our method. For
simplicity, these five features could be denoted by a backbone features set
$C$:

$C=\\{C^{(2)},C^{(3)},C^{(4)},C^{(5)},C^{(6)}\\},$ (1)

where $C^{(2)}$ denotes the Conv2-2 features and so on. Conv2-2 preserves
better edge information [61]. Thus we leverage the $S^{(2)}$ to extract the
edge features and other side paths to extract the salient object features.

####  3.2.1 Progressive salient object features extraction

As shown in PSFEM of Fig. 2, to obtain richer context features, we leverage
the widely used architecture U-Net [40] to generate the multi-resolution
features. Different from the original U-Net, in order to obtain more robust
salient object features, We add three convolutional layers (Conv in Fig. 2) on
each side path, and after each convolutional layer, a ReLU layer is added to
ensure the nonlinearity. To illustrate simply, we use the $T$ (Tab. 1) to
denote these convolutional layers and ReLU layers. Besides, deep supervision
is used on each side path. We adopt a convolutional layer to convert the
feature maps to the single-channel prediction mask and use $D$ (Tab. 1) to
denote it. The details of the convolutional layers could be found in Tab. 1.

S | $T_{1}$ | $T_{2}$ | $T_{3}$ | $D$
---|---|---|---|---
2 | 3 | 1 | 128 | 3 | 1 | 128 | 3 | 1 | 128 | 3 | 1 | 1
3 | 3 | 1 | 256 | 3 | 1 | 256 | 3 | 1 | 256 | 3 | 1 | 1
4 | 5 | 2 | 512 | 5 | 2 | 512 | 5 | 2 | 512 | 3 | 1 | 1
5 | 5 | 2 | 512 | 5 | 2 | 512 | 5 | 2 | 512 | 3 | 1 | 1
6 | 7 | 3 | 512 | 7 | 3 | 512 | 7 | 3 | 512 | 3 | 1 | 1
Table 1: Details of each side output. T denotes the feature enhance module
(Conv shown in Fig. 2). Each T contains three convolutional layers: $T_{1}$,
$T_{2}$, $T_{3}$ and three followed ReLu layers. We show the kernel size,
padding and channel number of each convolutional layer. For example, 3, 1, 128
denote a convolutional layer whose kernel size is 3, padding is 1, channel
number is 128. D denotes the transition layer which converts the multi-channel
feature map to one-channel activation map. S denotes the side path.

####  3.2.2 Non-local salient edge features extraction

In this module, we aim to model the salient edge information and extract the
salient edge features. As mentioned above, the Conv2-2 preserves better edge
information. Hence we extract local edge information from Conv2-2. However, in
order to get salient edge features, only local information is not enough.
High-level semantic information or location information is also needed. When
information is progressively returned from the top level to the low level like
the U-Net architecture, the high-level location information is gradually
diluted. Besides, the receptive field of the top-level is the largest, and the
location is the most accurate. Thus we design a top-down location propagation
to propagate the top-level location information to the side path $S^{(2)}$ to
restrain the non-salient edge. The fused features $\bar{C}^{(2)}$ could be
denoted as:

$\bar{C}^{(2)}=C^{(2)}+\mathit{Up}(\phi(\mathit{Trans}(\hat{F}^{(6)};\theta));C^{(2)}),$
(2)

where $\mathit{Trans}(*;\theta)$ is a convolutional layer with parameter
$\theta$, which aims to change the number of channels of the feature, and
$\phi()$ denotes a ReLU activation function. $Up(*;C^{(2)})$ is bilinear
interpolation operation which aims to up-sample * to the same size as
$C^{(2)}$. On the right of the equation, the second term denotes the features
from the higher side path. To illustrate clearly, we use
$\mathit{UpT(\hat{F}^{(i)};\theta,C^{(j)})}$ to represent
$\mathit{Up}(\phi(\mathit{Trans}(\hat{F}^{(i)};\theta));C^{(j)})$.
$\hat{F}^{(6)}$ denotes the enhanced features in side path $S^{(6)}$. The
enhanced features $\hat{F}^{(6)}$ could be represented as
$\mathnormal{f}(C^{(6)};W_{T}^{(6)})$, and the enhanced features in
$S^{(3)},S^{(4)},S^{(5)}$ could be computed as:

$\hat{F}^{(i)}=\mathnormal{f}(C^{(i)}+\mathit{UpT}(\hat{F}^{(i+1)};\theta,C^{(i)});W_{T}^{(i)}),$
(3)

where $W_{T}^{(i)}$ denotes the parameters in $T^{(i)}$ and $f(*;W_{T}^{(i)})$
denotes a series of convolutional and non-linear operations with parameters
$W_{T}^{(i)}$.

After obtaining the guided features $\bar{C}^{(2)}$, similar with other side
paths, we add a series convolutional layers to enhance the guided feature,
then the final  salient edge features  ${F_{E}}$ in $S^{(2)}$ could be
computed as $\mathnormal{f}(\bar{C}^{(2)};W_{T}^{(2)})$. The configuration
details could be found in Tab. 1. To model the salient edge feature
explicitly, we add an extra salient edge supervision to supervise the salient
edge features. We use the cross-entropy loss which could be defined as:

$\displaystyle\mathcal{L}^{(2)}(F_{E};W_{D}^{(2)})=-\sum_{j\in Z_{+}}{\log
Pr(\mathit{y}_{j}=1|F_{E};W_{D}^{(2)})}$ $\displaystyle-\sum_{j\in Z_{-}}{\log
Pr(\mathit{y}_{j}=0|F_{E};W_{D}^{(2)})},$ (4)

where $Z_{+}$ and $Z_{-}$ denote the salient edge pixels set and background
pixels set, respectively. $W_{D}$ denotes the parameters of the transition
layer as shown in Tab. 1. $Pr(\mathit{y}_{j}=1|F_{E};W_{D}^{(2)})$ is the
prediction map in which each value denotes the salient edge confidence for the
pixel. In addition, the supervision added on the salient object detection side
path can be represented as:

$\displaystyle\mathcal{L}^{(i)}(\hat{F}^{(i)};W_{D}^{(i)})=-\sum_{j\in
Y_{+}}{\log Pr(\mathit{y}_{j}=1|\hat{F^{(i)}};W_{D}^{(i)})}$
$\displaystyle-\sum_{j\in Y_{-}}{\log
Pr(\mathit{y}_{j}=0|\hat{F}^{(i)};W_{D}^{(i)})},~{}i\in[3,6],$ (5)

where $Y_{+}$ and $Y_{-}$ denote the salient region pixels set and non-salient
pixels set, respectively. Thus the total loss $\mathbb{L}$ in the
complementary information modeling could be denoted as:

$\mathbb{L}=\mathcal{L}^{(2)}(F_{E};W_{D}^{(2)})+\sum_{i=3}^{6}\mathcal{L}^{(i)}(\hat{F}^{(i)};W_{D}^{(i)}).$
(6)

###  3.3 One-to-one guidance module

After obtaining the complementary salient edge features and salient object
features, we aim to leverage the salient edge features to guide the salient
object features to perform better on both segmentation and localization. The
simple way is to fuse the $F_{E}$ and the $\hat{F}^{(3)}$. It will be better
to sufficiently leverage the multi-resolution salient object features.
However, the disadvantage of fusing the salient edge features and multi-
resolution salient object features progressively from down to top is that
salient edge features are diluted when salient object features are fused.
Besides, the goal is to fuse salient object features and salient edge features
to utilize complementary information to obtain better prediction results.
Hence, we propose a one-to-one guidance module. Moreover, experimental parts
validate our view.

Specifically, we add sub-side paths for $S^{(3)}$, $S^{(4)}$, $S^{(5)}$,
$S^{(6)}$. In each sub-side path, by fusing the salient edge features into
enhanced salient object features, we make the location of high-level
predictions more accurate, and more importantly, the segmentation details
become better. The salient edge guidance features (s-features) could be
denoted as:

${G}^{(i)}=\mathit{UpT}(\hat{F}^{(i)};\theta,F_{E})+F_{E},i\in[3,6].$ (7)

Then similar to the PSFEM, we adopt a series of convolutional layers $T$ in
each sub-side path to further enhance the s-features and a transition layer
$D$ to convert the multi-channel feature map to one-channel prediction map.
Here in order to illustrate clearly, we denote the $T$ and $D$ as $T^{\prime}$
and $D^{\prime}$ in this module. By Eq. (3), we obtain the enhanced s-features
$\hat{G}^{(i)}$.

Here we also add deep supervision for these enhanced s-features. For each sub-
side output prediction map, the loss can be calculated as:

$\displaystyle\mathcal{L}^{(i)^{\prime}}(\hat{G}^{(i)};W_{D^{\prime}}^{(i)})=-\sum_{j\in
Y_{+}}{\log Pr(\mathit{y}_{j}=1|\hat{G}^{(i)};W_{D^{\prime}}^{(i)})}$
$\displaystyle-\sum_{j\in Y_{-}}{\log
Pr(\mathit{y}_{j}=0|\hat{G}^{(i)};W_{D^{\prime}}^{(i)})},~{}i\in[3,6].$ (8)

Then we fuse the multi-scale refined prediction maps to obtain a fused map.
The loss function for the fused map can be denoted as:

$\mathcal{L}_{f}^{\prime}(\hat{G};W_{D^{\prime}})=\sigma(Y,\sum_{i=3}^{6}{\beta_{i}f(\hat{G}^{(i)};W_{D^{\prime}}^{(i)})}),$
(9)

where the $\sigma(*,*)$ represents the cross-entropy loss between prediction
map and saliency ground-truth, which has the same form to Eq. (5). Thus the
loss for this part and the total for the proposed network could be expressed
as:

$\begin{split}&\mathbb{L}^{\prime}=\mathcal{L}_{f}^{\prime}(\hat{G};W_{D^{\prime}})+\sum_{i=3}^{i=6}{\mathcal{L}^{(i)^{\prime}}(\hat{G}^{(i)};W_{D^{\prime}}^{(i)})}\\\
&\mathbb{L}_{t}=\mathbb{L}+\mathbb{L}^{\prime}.\\\ \end{split}$ (10)

|  ECSSD [56] |  PASCAL-S [30] |  DUT-O [57] |  HKU-IS [27] |  SOD [36, 37] |  DUTS-TE [46]
---|---|---|---|---|---|---
|  MaxF $\uparrow$ |  MAE $\downarrow$ |  S $\uparrow$ |  MaxF $\uparrow$ |  MAE $\downarrow$ |  S $\uparrow$ |  MaxF $\uparrow$ |  MAE $\downarrow$ |  S $\uparrow$ |  MaxF $\uparrow$ |  MAE $\downarrow$ |  S $\uparrow$ |  MaxF $\uparrow$ |  MAE $\downarrow$ |  S $\uparrow$ |  MaxF $\uparrow$ |  MAE $\downarrow$ |  S $\uparrow$
VGG-based
DCL∗ [28] | 0.896 | 0.080 |  0.863  | 0.805 | 0.115 |  0.791  | 0.733 | 0.094 |  0.743  | 0.893 | 0.063 |  0.859  | 0.831 | 0.131 |  0.748  | 0.786 | 0.081 | 0.785
DSS∗ [17, 18] | 0.906 | 0.064 |  0.882  | 0.821 | 0.101 |  0.796  | 0.760 | 0.074 |  0.765  | 0.900 | 0.050 |  0.878  | 0.834 | 0.125 |  0.744  | 0.813 | 0.065 | 0.812
MSR [26] | 0.903 | 0.059 |  0.875  | 0.839 | 0.083 |  0.802  | 0.790 | 0.073 |  0.767  | 0.907 | 0.043 |  0.852  | 0.841 | 0.111 |  0.757  | 0.824 | 0.062 | 0.809
NLDF [35] | 0.903 | 0.065 |  0.875  | 0.822 | 0.098 |  0.803  | 0.753 | 0.079 |  0.750  | 0.902 | 0.048 |  0.878  | 0.837 | 0.123 |  0.756  | 0.816 | 0.065 | 0.805
RAS [3] | 0.915 | 0.060 |  0.886  | 0.830 | 0.102 |  0.798  | 0.784 | 0.063 |  0.792  | 0.910 | 0.047 |  0.884  | 0.844 | 0.130 |  0.760  | 0.800 | 0.060 | 0.827
ELD∗ [13] | 0.865 | 0.082 |  0.839  | 0.772 | 0.122 |  0.757  | 0.738 | 0.093 |  0.743  | 0.843 | 0.072 |  0.823  | 0.762 | 0.154 |  0.705  | 0.747 | 0.092 | 0.749
DHS [32] | 0.905 | 0.062 |  0.884  | 0.825 | 0.092 |  0.807  | - | - |  \-  | 0.892 | 0.052 |  0.869  | 0.823 | 0.128 |  0.750  | 0.815 | 0.065 | 0.809
RFCN∗ [48] | 0.898 | 0.097 |  0852  | 0.827 | 0.118 |  0.799  | 0.747 | 0.094 |  0.752  | 0.895 | 0.079 |  0.860  | 0.805 | 0.161 |  0.730  | 0.786 | 0.090 | 0.784
UCF [62] | 0.908 | 0.080 |  0.884  | 0.820 | 0.127 |  0.806  | 0.735 | 0.131 |  0.748  | 0.888 | 0.073 |  0.874  | 0.798 | 0.164 |  0.762  | 0.771 | 0.116 | 0.777
Amulet [61] | 0.911 | 0.062 |  0.894  | 0.826 | 0.092 |  0.820  | 0.737 | 0.083 |  0.771  | 0.889 | 0.052 |  0.886  | 0.799 | 0.146 |  0.753  | 0.773 | 0.075 | 0.796
C2S [29] | 0.909 | 0.057 |  0.891  | 0.845 | 0.081 |  0.839  | 0.759 | 0.072 |  0.783  | 0.897 | 0.047 |  0.886  | 0.821 | 0.122 |  0.763  | 0.811 | 0.062 | 0.822
PAGR [63] | 0.924 | 0.064 |  0.889  | 0.847 | 0.089 |  0.818  | 0.771 | 0.071 |  0.751  | 0.919 | 0.047 |  0.889  | 0.841 | 0.146 |  0.716  | 0.854 | 0.055 | 0.825
Ours | 0.941 | 0.044 |  0.913 | 0.863 | 0.076 |  0.848 | 0.826 | 0.056 |  0.813 | 0.929 | 0.034 |  0.910 | 0.869 | 0.110 |  0.788 | 0.880 | 0.043 | 0.866
ResNet-based
SRM∗ [49] | 0.916 | 0.056 |  0.895  | 0.838 | 0.084 |  0.832  | 0.769 | 0.069 |  0.777  | 0.906 | 0.046 |  0.887  | 0.840 | 0.126 |  0.742  | 0.826 | 0.058 | 0.824
DGRL [52] | 0.921 | 0.043 |  0.906  | 0.844 | 0.075 |  0.839  | 0.774 | 0.062 |  0.791  | 0.910 | 0.036 |  0.896  | 0.843 | 0.103 |  0.774  | 0.828 | 0.049 | 0.836
PiCANet∗ [33] | 0.932 | 0.048 |  0.914 | 0.864 | 0.077 |  0.850 | 0.820 | 0.064 |  0.808 | 0.920 | 0.044 |  0.905 | 0.861 | 0.103 |  0.790 | 0.863 | 0.050 | 0.850
Ours | 0.943 | 0.041 |  0.918 | 0.869 | 0.074 |  0.852 | 0.842 | 0.052 |  0.818 | 0.937 | 0.031 |  0.918 | 0.890 | 0.097 |  0.807 | 0.893 | 0.039 | 0.875
Table 2: Quantitative comparison including max F-measure, MAE, and S-measure
over six widely used datasets. ‘-’ denotes that corresponding methods are
trained on that dataset. $\uparrow\&\downarrow$ denote larger and smaller is
better, respectively. ∗ means methods using pre-processing or post-processing.
The best three results are marked in red, blue, and green, respectively. Our
method achieves the state-of-the-art on these six widely used datasets under
three evaluation metrics.

##  4 Experiments

###  4.1 Implementation Details

We train our model on DUTS [46] dataset followed by [33, 59, 49, 63]. For a
fair comparison, we use VGG [43] and ResNet [16] as backbone networks,
respectively. Our model is implemented in PyTorch. All the weights of newly
added convolution layers are initialized randomly with a truncated normal
($\sigma$ = 0.01), and the biases are initialized to 0. The hyper-parameters
are set as followed: learning rate = 5e-5, weight decay = 0.0005, momentum =
0.9, loss weight for each side output is equal to 1. A back propagation is
processing for each of the ten images. We do not use the validation dataset
during training. We train our model 24 epochs and divide the learning rate by
10 after 15 epochs. During inference, we are able to obtain a predicted
salient edge map and a set of saliency maps. In our method, we directly use
the fused prediction map as the final saliency map.

###  4.2 Datasets and Evaluation Metric

We have evaluated the proposed architecture on six widely used public
benchmark datasets: ECSSD [56], PASCAL-S [30], DUT-OMRON [57], SOD [36, 44],
HKU-IS [27], DUTS [46]. ECSSD [56] contains 1000 meaningful semantic images
with various complex scenes. PASCAL-S [30] contains 850 images which are
chosen from the validation set of the PASCAL VOC segmentation dataset [8].
DUT-OMRON [57] contains 5168 high-quality but challenging images. Images in
this dataset contain one or more salient objects with a relatively complex
background. SOD [36] contains 300 images and is proposed for image
segmentation. Pixel-wise annotations of salient objects are generated by [44].
It is one of the most challenging datasets currently. HKU-IS [27] contains
4447 images with high-quality annotations, many of which have multiple
disconnected salient objects. This dataset is split into 2500 training images,
500 validation images and 2000 test images. DUTS [46] is the largest salient
object detection benchmark. It contains 10553 images for training and 5019
images for testing. Most images are challenging with various locations and
scales. Following most recent works [33, 49, 52], we use the DUTS dataset to
train the proposed model.

We use three widely used and standard metrics, F-measure, mean absolute error
(MAE) [2], and a recently proposed structure-based metric, namely S-measure
[10], to evaluate our model and other state-of-the-art models. F-measure is a
harmonic mean of average precision and average recall, formulated as:

$\displaystyle F_{\beta}$ $\displaystyle=\frac{(1+\beta^{2})Precision\times
Recall}{\beta^{2}\times Precision+Recall},$ (11)

we set $\beta^{2}=0.3$ to weigh precision more than recall as suggested in
[5]. Precision denotes the ratio of detected salient pixels in the predicted
saliency map. Recall denotes the ratio of detected salient pixels in the
ground-truth map. Precision and recall are computed on binary images. Thus we
should threshold the prediction map to binary map first. There are different
precision and recall of different thresholds. We could plot the precision-
recall curve at different thresholds. Here we use the code provided by [17,
18] for evaluation. Following most salient object detection methods [17, 18,
32, 59], we report the maximum F-measure from all precision-recall pairs.

MAE is a metric which evaluates the average difference between prediction map
and ground-truth map. Let $P$ and $Y$ denote the saliency map and the ground
truth that is normalized to [0, 1]. We compute the MAE score by:

$\varepsilon=\frac{1}{W\times H}\sum_{x=1}^{W}\sum_{y=1}^{H}|P(x,y)-Y(x,y)|,$
(12)

where $W$ and $H$ are the width and height of images, respectively.

S-measure focuses on evaluating the structural information of saliency maps,
which is closer to the human visual system than F-measure. Thus we include
S-measure for a more comprehensive evaluation. S-measure could be computed as:

$\mathtt{S}=\gamma\mathtt{S}_{o}+(1-\gamma)\mathtt{S}_{r},$ (13)

where $\mathtt{S}_{o}$ and $\mathtt{S}_{r}$ denotes the region-aware and
object-aware structural similarity and $\gamma$ is set as 0.5 by default. More
details can be found in [10].

Model | SOD | DUTS
---|---|---
MaxF $\uparrow$ |  MAE $\downarrow$ |  S $\uparrow$ |  MaxF $\uparrow$ |  MAE $\downarrow$ |  S $\uparrow$
1\. B | .851 | .116 | .780 | .855 | .060 | .844
2\. B + edge_PROG | .873 | .105 | .799 | .872 | .051 | .851
3\. B + edge_TDLP | .882 | .100 | .807 | .879 | .044 | .866
4\. B + edge_NLDF | .857 | .112 | .794 | .866 | .053 | .860
5\. B + edge_TDLP + MRF_PROG | .882 | .106 | .796 | .880 | .046 | .869
6\. B + edge_TDLP + MRF_OTO | .890 | .097 | .807 | .893 | .039 | .875
Table 3: Ablation analyses on SOD [36] and DUTS-TE [46]. Here, B denotes the
baseline model. edge_PROG, edge_TDLF, edge_NLDF, MRF_PROG, MRF_OTO are
introduced in the Sec. 4.3.

###  4.3 Ablation Experiments and Analyses

In this section, with the DUTS-TR [46] as the training set, we explore the
effect of different components in the proposed network over the relatively
difficult dataset SOD [36] and the recently proposed big dataset DUTS-TE [46].

|  |
---|---|---
(a) DUTS-TE [46] |  (b) DUT-OMRON [57] |  (c) HKU-IS [27]
Figure 3: Precision (vertical axis) recall (horizontal axis) curves on three
popular salient object datasets. It can be seen that the proposed method
performs favorably against state-of-the-arts.

####  4.3.1 The complementary information modeling

In this subsection, we explore the role of salient edge information, which is
also our basic idea. The baseline is the U-Net architecture which integrates
the multi-scale features (From Conv2-2 to Conv6-3) in the way as PSFEM (Fig.
2). We remove the side path $S^{(2)}$ in the baseline and then fuse the final
saliency features $\hat{F}^{(3)}$ (side path from Conv3-3) and the local
Conv2-2 features to obtain the salient edge features. Finally, we integrate
salient edge features and the salient object features $\hat{F}^{(3)}$ to get
the prediction mask. We denote this strategy of using edges as edge_PROG. The
result is shown in the second row of Tab. 3. It proves that the salient edge
information is very useful for the salient object detection task.

####  4.3.2 Top-down location propagation

In this subsection, we explore the role of top-down location propagation.
Compared with edge_PROG mentioned in the previous subsection Sec. 4.3.1, we
leverage the top-down location propagation to extract more accurate location
information from top-level instead of side path $S^{(3)}$. We call this
strategy of using edges as edge_TDLP. By comparing the second and third rows
of Tab. 3, the effect of top-down location propagation could be proved.
Besides, comparing the first row and the third row of Tab. 3, we can find that
through our explicit modeling of these two kinds of complementary information
within the network, the performance is greatly improved on the datasets (3.1%,
2.4% under F-measure) without additional time and space consumption.

|  |  |  |
---|---|---|---|---
|  |  |  |
Source | B | B+edge_NLDF | B+edge_TDLP | GT
Figure 4: Visual examples before and after adding edge cues. B denotes the
baseline model. edge_NLDF and edge_TDLP represent the edges penalty used in
NLDF [35] and the edge modeling method proposed in this paper. The details are
introduced in the Sec. 4.3.

####  4.3.3 Mechanism of using edge cues

To demonstrate the advantages over NLDF [35], in which an IOU loss is added to
the end of the network to punish the errors of edges. We add the same IOU loss
to the baseline. This strategy is called edge_NLDF. The performance is shown
in the 4th row of Tab. 3. Compared with the baseline model, the improvement is
limited. This also demonstrates that the proposed method of using edge
information is more effective. The visualization results are shown in Fig. 4.
Compared with the baseline model without edge constraint, after we add the
edge penalty used in NLDF [35], edge information can only help refine the
boundaries. In particular, this penalty can not help to remove the redundant
parts in saliency prediction mask, nor can it make up for the missing parts.
In contrast, the proposed complementary information modeling method considers
the complementarity between salient edge information and salient object
information, and performs better on both segmentation and localization.

Besides, in order to further prove that salient edge detection and salient
object detection are mutually helpful and complementary. We compare the
salient edges generated by NLDF with the salient edges generated by us. The
pre-trained model and code are both provided by the authors. As shown in Tab.
4, it could be found that the salient edge generated by our method is much
better, especially under the recall and F-measure metrics. It proves that the
edges are more accurate in our methods.

####  4.3.4 The complementary features fusion

After we obtain the salient edge features and multi-resolution salient object
features. We aim to fuse these complementary features. Here we compare three
fusion methods. The first way is the default way, which integrates the salient
edge features ($F_{E}$) and the salient object features $\hat{F}^{(3)}$ which
is on the top of U-Net architecture. The second way is to fuse the multi-
resolution features $\hat{F}^{(3)}$, $\hat{F}^{(4)}$, $\hat{F}^{(5)}$,
$\hat{F}^{(6)}$ progressively, which is called MRF_PROG. The third way is the
proposed one-to-one guidance, which is denoted MRF_OTO. Here MRF denotes the
multi-resolution fusion. The results are shown in the third, fifth, sixth rows
of Tab. 3, respectively. It can be seen that our proposed one-to-one guidance
method is most suitable for our whole architecture.

Model | SOD | DUTS
---|---|---
Recall $\uparrow$ |  Precision $\uparrow$ |  MaxF $\uparrow$ |  Recall $\uparrow$ |  Precision $\uparrow$ |  MaxF $\uparrow$
NLDF | 0.513 | 0.541 | 0.527 | 0.318 | 0.659 | 0.429
Ours | 0.637 | 0.534 | 0.581 | 0.446 | 0.680 | 0.539
Table 4: Comparisons on the salient edge generated by the NLDF and ours.

###  4.4 Comparison with the State-of-the-art

In this section, we compare our proposed EGNet with 15 previous state-of-the-
art methods, including DCL [28], DSS [17, 18], NLDF [35], MSR [26], ELD [13],
DHS [32], RFCN [48], UCF [62], Amulet [61], PAGR [63], PiCANet [33], SRM [49],
DGRL [52], RAS [3] and C2S [29]. Note that all the saliency maps of the above
methods are produced by running source codes or pre-computed by the authors.
The evaluation codes are provided in [17, 18, 10].

|  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
Image | GT | Ours |  PiCANet[33] |  PAGR[63] |  DGRL[52] |  SRM[49] |  UCF[62] |  Amulet[61] |  DSS[17, 18] |  DHS[32]
Figure 5: Qualitative comparisons with state-of-the-arts.

F-measure, MAE, and S-measure. We evaluate and compare our proposed method
with other salient object detection methods in term of F-measure, MAE, and
S-measure as shown in Tab. 2. We could see that different methods may use
different backbone net. Here for a fair comparison, we train our model on the
VGG [43] and ResNet [16], respectively. It can be seen that our model performs
favorably against the state-of-the-art methods under all evaluation metrics on
all the compared datasets especially on the relative challenging dataset SOD
[36, 44] (2.9% and 1.7% improvements in F-measure and S-measure) and the
largest dataset DUTS [46] (3.0% and 2.5%). Specifically, Compared with the
current best approach [33], the average F-measure improvement on six datasets
is 1.9%. Note that this is achieved without any pre-processing and post-
processing.

Precision-recall curves. Besides the numerical comparisons shown in Tab. 2, we
plot the precision-recall curves of all compared methods over three datasets
Fig. 3. As can be seen that the solid red line which denotes the proposed
method outperforms all other methods at most thresholds. Due to the help of
the complementary salient edge information, the results yield sharp edge
information and accurate localization, which results in a better PR curve.

Visual comparison. In Fig. 5, we show some visualization results. It could be
seen that our method performs better on salient object segmentation and
localization. It is worth mentioning that thank to the salient edge features,
our result could not only highlight the salient region but also produce
coherent edges. For instance, for the first sample, due to the influence of
the complex scene, other methods are not capable of localizing and segmenting
salient objects accurately. However, benefiting from the complementary salient
edge features, our method performs better. For the second sample, in which the
salient object is relatively small, our result is still very close to the
ground-truth.

##  5 Conclusion

In this paper, we aim to preserve salient object boundaries well. Different
from other methods which integrate the multi-scale features or leverage the
post-processing, we focus on the complementarity between salient edge
information and salient object information. Based on this idea, we propose the
EGNet to model these complementary features within the network. First, we
extract the multi-resolution salient object features based on U-Net. Then, we
propose a non-local salient edge features extraction module which integrates
the local edge information and global location information to get the salient
edge features. Finally, we adopt a one-to-one guidance module to fuse these
complementary features. The salient object boundaries and localization are
improved under the help of salient edge features. Our model performs favorably
against the state-of-the-art methods on six widely used datasets without any
pre-processing or post-processing. We also provide analyses of the
effectiveness of the EGNet.

##### Acknowledgments.

This research was supported by NSFC (61572264), the national youth talent
support program, and Tianjin Natural Science Foundation (17JCJQJC43700,
18ZXZNGX00110).

## References

  * [1] Ali Borji, Ming-Ming Cheng, Qibin Hou, Huaizu Jiang, and Jia Li.  Salient object detection: A survey.  CVM, 5(2):117–150, 2019.
  * [2] Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li.  Salient object detection: A benchmark.  IEEE TIP, 24(12):5706–5722, 2015.
  * [3] Shuhan Chen, Xiuli Tan, Ben Wang, and Xuelong Hu.  Reverse attention for salient object detection.  In ECCV, pages 234–250, 2018.
  * [4] Tao Chen, Ming-Ming Cheng, Ping Tan, Ariel Shamir, and Shi-Min Hu.  Sketch2photo: Internet image montage.  ACM TOG, 28(5):124:1–10, 2009.
  * [5] Ming Cheng, Niloy J Mitra, Xumin Huang, Philip HS Torr, and Song Hu.  Global contrast based salient region detection.  IEEE TPAMI, 37(3):569–582, 2015.
  * [6] Ming-Ming Cheng, Fang-Lue Zhang, Niloy J Mitra, Xiaolei Huang, and Shi-Min Hu.  Repfinder: finding approximately repeated scene elements for image editing.  ACM TOG, 29(4):83, 2010.
  * [7] Wolfgang Einhäuser and Peter König.  Does luminance-contrast contribute to a saliency map for overt visual attention?  European Journal of Neuroscience, 17(5):1089–1097, 2003.
  * [8] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  IJCV, 88(2):303–338, 2010.
  * [9] Deng-Ping Fan, Ming-Ming Cheng, Jiang-Jiang Liu, Shang-Hua Gao, Qibin Hou, and Ali Borji.  Salient objects in clutter: Bringing salient object detection to the foreground.  In ECCV, pages 186–202. Springer, 2018.
  * [10] Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji.  Structure-measure: A new way to evaluate foreground maps.  In ICCV, pages 4548–4557, 2017.
  * [11] Deng-Ping Fan, Zheng Lin, Jia-Xing Zhao, Yun Liu, Zhao Zhang, Qibin Hou, Menglong Zhu, and Ming-Ming Cheng.  Rethinking rgb-d salient object detection: Models, datasets, and large-scale benchmarks.  arXiv preprint arXiv:1907.06781, 2019.
  * [12] Deng-Ping Fan, Wenguan Wang, Ming-Ming Cheng, and Jianbing Shen.  Shifting more attention to video salient object detection.  In CVPR, pages 8554–8564, 2019.
  * [13] Lee Gayoung, Tai Yu-Wing, and Kim Junmo.  Deep saliency with encoded low level distance map and high level features.  In CVPR, 2016.
  * [14] Wenlong Guan, Tiantian Wang, Jinqing Qi, Lihe Zhang, and Huchuan Lu.  Edge-aware convolution neural network based salient object detection.  IEEE SPL, 26(1):114–118, 2018.
  * [15] Junfeng He, Jinyuan Feng, Xianglong Liu, Tao Cheng, Tai-Hsu Lin, Hyunjin Chung, and Shih-Fu Chang.  Mobile product search with bag of hash bits and boundary reranking.  In CVPR, pages 3005–3012, 2012.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In ICCV, pages 770–778, 2016.
  * [17] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip Torr.  Deeply supervised salient object detection with short connections.  In CVPR, pages 3203–3212, 2017.
  * [18] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip Torr.  Deeply supervised salient object detection with short connections.  IEEE TPAMI, 41(4):815–828, 2019.
  * [19] Qibin Hou, Peng-Tao Jiang, Yunchao Wei, and Ming-Ming Cheng.  Self-erasing network for integral object attention.  In NIPS, 2018.
  * [20] Ping Hu, Bing Shuai, Jun Liu, and Gang Wang.  Deep level sets for salient object detection.  In CVPR, pages 2300–2309, 2017.
  * [21] Laurent Itti and Christof Koch.  Computational modeling of visual attention.  Nature reviews neuroscience, 2(3):194–203, 2001.
  * [22] Laurent Itti, Christof Koch, and Ernst Niebur.  A model of saliency-based visual attention for rapid scene analysis.  IEEE TPAMI, 20(11):1254–1259, 1998.
  * [23] Sen Jia and Neil DB Bruce.  Richer and deeper supervision network for salient object detection.  arXiv preprint arXiv:1901.02425, 2019.
  * [24] Dominik A Klein and Simone Frintrop.  Center-surround divergence of feature statistics for salient object detection.  In ICCV, pages 2214–2219. IEEE, 2011.
  * [25] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [26] Guanbin Li, Yuan Xie, Liang Lin, and Yizhou Yu.  Instance-level salient object segmentation.  In CVPR, 2017.
  * [27] Guanbin Li and Yizhou Yu.  Visual saliency based on multiscale deep features.  In CVPR, pages 5455–5463, 2015.
  * [28] Guanbin Li and Yizhou Yu.  Deep contrast learning for salient object detection.  In CVPR, 2016.
  * [29] Xin Li, Fan Yang, Hong Cheng, Wei Liu, and Dinggang Shen.  Contour knowledge transfer for salient object detection.  In ECCV, pages 355–370, 2018.
  * [30] Yin Li, Xiaodi Hou, Christof Koch, James M Rehg, and Alan L Yuille.  The secrets of salient object segmentation.  In CVPR, pages 280–287, 2014.
  * [31] Zun Li, Congyan Lang, Yunpeng Chen, Junhao Liew, and Jiashi Feng.  Deep reasoning with multi-scale context for salient object detection.  arXiv preprint arXiv:1901.08362, 2019.
  * [32] Nian Liu and Junwei Han.  Dhsnet: Deep hierarchical saliency network for salient object detection.  In CVPR, pages 678–686, 2016.
  * [33] Nian Liu, Junwei Han, and Ming-Hsuan Yang.  Picanet: Learning pixel-wise contextual attention for saliency detection.  In CVPR, pages 3089–3098, 2018.
  * [34] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In CVPR, pages 3431–3440, 2015.
  * [35] Zhiming Luo, Akshaya Kumar Mishra, Andrew Achkar, Justin A Eichel, Shaozi Li, and Pierre-Marc Jodoin.  Non-local deep features for salient object detection.  In CVPR, 2017.
  * [36] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In ICCV, volume 2, pages 416–423, 2001.
  * [37] Vida Movahedi and James H Elder.  Design and perceptual validation of performance measures for salient object segmentation.  In IEEE CVPRW, pages 49–56. IEEE, 2010.
  * [38] David Mumford and Jayant Shah.  Optimal approximations by piecewise smooth functions and associated variational problems.  CPAM, 42(5):577–685, 1989.
  * [39] Derrick Parkhurst, Klinton Law, and Ernst Niebur.  Modeling the role of salience in the allocation of overt visual attention.  Vision research, 42(1):107–123, 2002.
  * [40] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [41] Paul L Rosin and Yu-Kun Lai.  Artistic minimal rendering with lines and blocks.  Graphical Models, 75(4):208–229, 2013.
  * [42] Ueli Rutishauser, Dirk Walther, Christof Koch, and Pietro Perona.  Is bottom-up attention useful for object recognition?  In CVPR, 2004.
  * [43] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [44] Jingdong Wang, Huaizu Jiang, Zejian Yuan, Ming-Ming Cheng, Xiaowei Hu, and Nanning Zheng.  Salient object detection: A discriminative regional feature integration approach.  IJCV, 123(2):251–268, 2017.
  * [45] Lijun Wang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang.  Deep networks for saliency detection via local estimation and global search.  In ICCV, pages 3183–3192, 2015.
  * [46] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan.  Learning to detect salient objects with image-level supervision.  In CVPR, pages 136–145, 2017.
  * [47] Linzhao Wang, Lijun Wang, Huchuan Lu, Pingping Zhang, and Xiang Ruan.  Saliency detection with recurrent fully convolutional networks.  In ECCV, pages 825–841. Springer, 2016.
  * [48] Linzhao Wang, Lijun Wang, Huchuan Lu, Pingping Zhang, and Xiang Ruan.  Saliency detection with recurrent fully convolutional networks.  In ECCV, 2016.
  * [49] Tiantian Wang, Ali Borji, Lihe Zhang, Pingping Zhang, and Huchuan Lu.  A stagewise refinement model for detecting salient objects in images.  In ICCV, pages 4019–4028, 2017.
  * [50] Tiantian Wang, Yongri Piao, Li Xiao, Lihe Zhang, and Huchuan Lu.  Deep learning for light field saliency detection.  In ICCV, 2019.
  * [51] Tiantian Wang, Lihe Zhang, Huchuan Lu, Chong Sun, and Jinqing Qi.  Kernelized subspace ranking for saliency detection.  In ECCV, pages 450–466, 2016.
  * [52] Tiantian Wang, Lihe Zhang, Shuo Wang, Huchuan Lu, Gang Yang, Xiang Ruan, and Ali Borji.  Detect globally, refine locally: A novel approach to saliency detection.  In CVPR, pages 3127–3135, 2018.
  * [53] Wenguan Wang, Jianbing Shen, Xingping Dong, and Ali Borji.  Salient object detection driven by fixation prediction.  In ICCV, pages 1711–1720, 2018.
  * [54] Ziqin Wang, Jun Xu, Li Liu, Fan Zhu, and Ling Shao.  Ranet: Ranking attention network for fast video object segmentation.  In ICCV, Oct 2019.
  * [55] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  In ICCV, pages 1395–1403, 2015.
  * [56] Qiong Yan, Li Xu, Jianping Shi, and Jiaya Jia.  Hierarchical saliency detection.  In CVPR, pages 1155–1162, 2013.
  * [57] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang.  Saliency detection via graph-based manifold ranking.  In CVPR, pages 3166–3173, 2013.
  * [58] Jing Zhang, Yuchao Dai, Fatih Porikli, and Mingyi He.  Deep edge-aware saliency detection.  arXiv preprint arXiv:1708.04366, 2017.
  * [59] Lu Zhang, Ju Dai, Huchuan Lu, You He, and Gang Wang.  A bi-directional message passing model for salient object detection.  In ICCV, pages 1741–1750, 2018.
  * [60] Pingping Zhang, Wei Liu, Huchuan Lu, and Chunhua Shen.  Salient object detection with lossless feature reflection and weighted structural loss.  IEEE TIP, 2019.
  * [61] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Xiang Ruan.  Amulet: Aggregating multi-level convolutional features for salient object detection.  In ICCV, pages 202–211, 2017.
  * [62] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Baocai Yin.  Learning uncertain convolutional features for accurate saliency detection.  In ICCV, pages 212–221. IEEE, 2017.
  * [63] Xiaoning Zhang, Tiantian Wang, Jinqing Qi, Huchuan Lu, and Gang Wang.  Progressive attention guided recurrent network for salient object detection.  In CVPR, pages 714–722, 2018.
  * [64] Jiaxing Zhao, Ren Bo, Qibin Hou, Ming-Ming Cheng, and Paul Rosin.  Flic: Fast linear iterative clustering with active search.  CVM, 4(4):333–348, Dec 2018.
  * [65] Jiaxing Zhao, Bo Ren, Qibin Hou, and Ming-Ming Cheng.  Flic: Fast linear iterative clustering with active search.  In AAAI, 2018.
  * [66] Jia-Xing Zhao, Yang Cao, Deng-Ping Fan, Ming-Ming Cheng, Xuan-Yi Li, and Le Zhang.  Contrast prior and fluid pyramid integration for rgbd salient object detection.  In CVPR, 2019.
  * [67] Kai Zhao, Shanghua Gao, Wenguan Wang, and Ming-Ming Cheng.  Optimizing the f-measure for threshold-free salient object detection.  In ICCV, Oct 2019.
  * [68] Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun.  Saliency optimization from robust background detection.  In CVPR, pages 2814–2821, 2014.
  * [69] Yunzhi Zhuge, Gang Yang, Pingping Zhang, and Huchuan Lu.  Boundary-guided feature aggregation network for salient object detection.  IEEE SPL, 25(12):1800–1804, 2018.