latexFont shape declaration has incorrect series value latexfontFont shape
latexText page latexFont shape latexfontFont shape latexfontSize substitutions

# Salient Objects in Clutter

Deng-Ping Fan, Jing Zhang, Gang Xu, Ming-Ming Cheng, and Ling Shao,
Deng-Ping Fan, Gang Xu and Ming-Ming Cheng are with the CS, Nankai University,
Tianjin, China. (E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Jing Zhang is with Research School of Engineering, the
Australian National University, ACRV, DATA61-CSIRO. (Email<EMAIL_ADDRESS>Ling Shao is with the Inception Institute of Artificial Intelligence, Abu
Dhabi, UAE. (E-mail<EMAIL_ADDRESS>A preliminary version of this
work appeared in ECCV [1]. The major part of this work was done in Nankai
University. Ming-Ming Cheng is the corresponding author.

###### Abstract

In this paper, we identify and address a serious design bias of existing
salient object detection (SOD) datasets, which unrealistically assume that
each image should contain at least one clear and uncluttered salient object.
This design bias has led to a saturation in performance for state-of-the-art
SOD models when evaluated on existing datasets. However, these models are
still far from satisfactory when applied to real-world scenes. Based on our
analyses, we propose a new high-quality dataset and update the previous
saliency benchmark. Specifically, our dataset, called Salient Objects in
Clutter (SOC), includes images with both salient and non-salient objects from
several common object categories. In addition to object category annotations,
each salient image is accompanied by attributes that reflect common challenges
in real-world scenes, which can help provide deeper insight into the SOD
problem. Further, with a given saliency encoder, _e.g._ , the backbone
network, existing saliency models are designed to achieve mapping from the
training image set to the training ground-truth set. We therefore argue that
improving the dataset can yield higher performance gains than focusing only on
the decoder design. With this in mind, we investigate several dataset-
enhancement strategies, including label smoothing to implicitly emphasize
salient boundaries, random image augmentation to adapt saliency models to
various scenarios, and self-supervised learning as a regularization strategy
to learn from small datasets. Our extensive results demonstrate the
effectiveness of these tricks. We also provide a comprehensive benchmark for
SOD, which can be found in our repository: http://dpfan.net/SOCBenchmark.

###### Index Terms:

Salient object detection, SOD, SOC, survey, dataset, benchmark.

##  1 Introduction

This paper considers the task of salient object detection (SOD), which aims to
detect the most attention-grabbing objects in a scene and then extract pixel-
accurate silhouettes for them. The merit of SOD lies in its many applications,
including foreground map evaluation [2, 3, 4], visual tracking [5, 6, 7],
action recognition [8], image retrieval [9, 10], information discovery [11,
12], image contrast enhancement [13], person re-identification [14] image
segmentation [15, 16], video segmentation [17], photo synthesis [18], content-
aware image editing [19], image caption [20], and video compression [21, 22],
style transfer [23, 24], image matching [25], autonomous underwater robots
[26], camouflaged object detection [27], aesthetic scoring [28], self-driving
vehicles [29], plant species identification [30], VR/AR [31]111AR CUT & PASTE:
https://www.youtube.com/watch?v=-N-podTAY9Y., Sony’s BRAVIA XR
TV222https://www.youtube.com/watch?v=4LnCuTAlVno&feature=youtu.be., _etc_.

\begin{overpic}[width=433.62pt]{DataExamp.pdf}
\put(8.0,65.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{non-
salient}}}}}
\put(33.0,65.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{non-
salient}}}}}
\put(58.0,65.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{non-
salient}}}}}
\put(83.0,65.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{non-
salient}}}}}
\par\put(8.0,45.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{carrot}}}}}
\put(1.0,53.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:HO,OC,SC,SO}}}}}
\put(33.0,45.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{dog}}}}}
\put(26.0,53.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:HO,OC,SC,SO}}}}}
\put(58.0,45.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{motorcycle}}}}}
\put(51.0,53.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:HO,SC}}}}}
\put(90.0,40.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{toilet}}}}}
\put(76.0,53.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:HO,OC,AC}}}}}
\put(88.0,48.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{book}}}}}
\par\put(15.0,30.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{bicycle}}}}}
\put(1.0,34.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:OC,SC}}}}}
\put(15.0,23.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{dog}}}}}
\put(32.0,23.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{person}}}}}
\put(26.0,34.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:HO,OC,OV,SC}}}}}
\put(60.0,26.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{wine
glass}}}}}
\put(58.0,23.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{person}}}}}
\put(51.0,34.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:OV}}}}}
\put(79.0,20.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{cell
phone}}}}}
\put(88.0,26.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{banana}}}}}
\put(76.0,34.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:HO,OC,SC}}}}}
\par\put(5.0,8.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{person}}}}}
\put(1.0,15.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:BO,OC}}}}}
\put(30.0,8.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{giraffe}}}}}
\put(26.0,15.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:HO,OC,OV}}}}}
\put(54.0,3.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{person}}}}}
\put(51.0,15.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:HO,OC,AC}}}}}
\put(62.0,1.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{umbrella}}}}}
\put(83.0,12.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{computers}}}}}
\put(76.0,15.0){\contour{black}{{{\color[rgb]{1,1,1}\scriptsize{Attr:HO}}}}}
\put(89.0,7.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{laptop}}}}}
\put(80.0,3.0){\contour{white}{{{\color[rgb]{0,0,0}\scriptsize{keyboard}}}}}
\end{overpic} Figure 1: Examples from our new SOC dataset, including _non-
salient_ (first row) and _salient_ object images (rows 2 to 4). For salient
object images, an instance-level ground-truth map (different color), object
attributes (Attr) and category labels are provided.

However, existing SOD datasets [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42]
are flawed either in their data collection procedure or data quality.
Specifically, most datasets assume that an image should contain at least one
salient object, and thus they discard images that do not contain any salient
objects. We call this _DATA SELECTION BIAS[43]._

Moreover, existing datasets typically contain images with a single object or
several uncluttered objects. These datasets do not adequately reflect the
complexity of real-world images, where scenes usually contain multiple objects
amidst significant clutter. As a result, all top-performing models trained on
the existing large-scale datasets (_e.g._ , DUTS [41]) have nearly saturated
performance (_e.g._ , SCRN [44] has an $S\textrm{-}measure>0.9$ on ECSSD
[37]), but still achieve unsatisfactory results on realistic images (_e.g._ ,
$S\textrm{-}measure<0.8$ on SOC [1]). As the current SOD models are biased
towards ideal conditions, their effectiveness may be impaired once they are
applied to real-world scenes. To solve this problem, it is important to
introduce a dataset with more realistic conditions.

Another issue faced by the RGB SOD community is that only the overall
performance of the models can be analyzed using existing datasets. This is
because none of the datasets contain attributes that reflect different real-
world challenges. Having such attributes would help i) provide deeper insight
into the SOD problem, ii) enable the pros and cons of the SOD models to be
investigated, and iii) allow the model performances to be objectively assessed
from different perspectives. Finally, with a given saliency encoder, _e.g._ ,
the backbone network, existing saliency models are designed to achieve mapping
from the training image set to the training ground-truth set. We thus argue
that efforts on improving the dataset, _e.g._ , fixing the data bias issue,
can yield higher performance gains than focusing only on the decoder design.
Towards this, we investigate several dataset-enhancement strategies, including
label smoothing to highlight salient boundaries, random image augmentation to
adapt saliency models to various scenarios, and self-supervised learning as a
form of regularization to learn from small datasets. Extensive experiments
validate the effectiveness of these tricks.

Our contributions are summarized as follows:

Figure 2: Taxonomy of the saliency detection task. We highlight the scope of
this study in gray. See $\S$ 2 for details.

  * 1)

Dataset. We collect a new high-quality SOD dataset, named “Salient Objects in
Clutter,” or SOC. SOC is the largest instance-level SOD dataset to date,
containing 6,000 images from more than 80 common categories. It differs from
existing datasets in three aspects: i) Salient objects have category
annotations, which can be used for new research problems, such as weakly
supervised SOD. ii) The inclusion of non-salient images and objects makes this
dataset more realistic and challenging than the existing ones. iii) Salient
objects have attributes that reflect various situations encountered in the
real world, such as motion blur, occlusion and background clutter. As a
consequence, SOC narrows the gap between existing datasets and real-world
scenes.

  * 2)

Review & Benchmark. We present the largest scale RGB SOD study, reviewing 201
representative models including 84 algorithms using handcrafted features and
117 deep learning based models. Besides, we also maintain an online benchmark
(_i.e._ , http://dpfan.net/SOCBenchmark.) to dynamically trace the development
of this field. In addition, we provide the most comprehensive benchmark of the
top-100 SOD models. To evaluate the models, for the first time, we not only
present the overall but also an attribute-based performance evaluation. This
allows a deeper understanding of the models and provides a more complete
benchmark.

  * 3)

Strategy. We investigate the biased dataset issue and introduce three dataset-
enhancement strategies; namely, label smoothing to make the model aware of the
salient boundaries, random image augmentation to adapt the saliency models to
various real-world scenarios, and self-supervised learning as a regularization
technique to learn from small datasets. Despite the apparent simplicity of our
strategies, we can achieve an average absolute improvement of 1.1%
$S_{\alpha}$ over five existing cutting-edge models.

  * 4)

Discussions & Future Directions. Based on our SOC, we present the pros and
cons of the current SOD algorithms, discuss several under-investigated open
issues, and provide potential future directions at six levels, _e.g._ , the
dataset level, task level, model level, supervision level, evaluation level,
and application level.

This work extends our previous conference version [1] in the following
aspects. First, we provide more details on our SOC, including sample images
without salient objects, images with attributes, and statistics of the
attributes. Second, we study three novel training dataset related strategies
to fully utilize the non-salient object data and achieve the new state-of-the-
art performance. Third, we conduct the largest-scale (46 traditional and 54
deep learning models) benchmarking of SOD models on our SOC. Finally, based on
our benchmarking results, we highlight several fundamental research directions
and challenges in the SOD.

##  2 Related Work

###  2.1 Scope

Salient object detection originated from the task of fixation prediction (FP)
[45, 46], switching attention regions for accurate object-level regions. SOD
can be traced back to the seminal works [47, 48]. Current algorithms have been
developed for 2D images of limited resolution (width or height $<$ 500
pixels), high-resolution (_i.e._ , 1080p, 4K) [49, 50] and even remote sensing
data [51]. According to the supervision strategy, there are five types of SOD
models: fully supervised [52], semi-supervised [53], weakly supervised [54,
55, 56], unsupervised [57, 58, 59], and self-supervised [60, 59].

Recently, several interesting extensions of SOD have also been introduced,
such as salient instance detection (SID) [61, 62], salient object subitizing
(SOS) [63, 64, 65], and saliency ranking [66, 67]. A taxonomy of the saliency
detection task is shown in Fig. 2. Different from previous SOD reviews [68,
69, 70, 71, 72, 73, 74, 75, 76], we mainly focus on 2D salient object
detection in a fully supervised manner. We highlight the scope of this study
in gray. For other closely related 3D/4D SOD tasks, we refer readers to recent
survey and benchmarking works such as RGB-D SOD [77, 78], Event-RGB SOD
(ERSOD) 333ERSOD: https://github.com/jxr326/ERSOD-Net., Light Field SOD [79],
Co-SOD [80], 360∘Video SOD [81], and Video SOD [17].

Table I: Summary of popular SOD datasets. Our SOC is the only one meeting all requirements. According to [76], these datasets can be grouped into three types: early ($\blacktriangle$), popular/modern ($\blacklozenge$), and special ($\lozenge$). See $\S$ 2.2 for more details.  # | Dataset | Year | Publ. | High-Quality | $\geq 5k$ | Non-Salient | Attribute | Category | Bounding Box | Object | Instance
---|---|---|---|---|---|---|---|---|---|---|---
1 |  MSRA-A, -B [32] $\blacktriangle$ | 2007 | CVPR | ✓ | ✓ | - | - | - | ✓ | ✓ | -
2 |  SED1, SED2 [33] $\blacktriangle$ | 2007 | CVPR | ✓ | - | - | - | - | - | ✓ | -
3 |  ASD [82] $\blacktriangle$ | 2009 | CVPR | ✓ | - | - | - | - | - | ✓ | -
4 |  SOD [83] $\blacklozenge$ | 2010 | CVPRW | ✓ | - | - | - | - | - | ✓ | -
5 |  MSRA10K [84] $\blacklozenge$ | 2011 | CVPR | ✓ | ✓ | - | - | - | - | ✓ | -
6 |  Judd-A [36] $\blacktriangle$ | 2012 | ECCV | ✓ | - | - | - | - | - | ✓ | -
7 |  DUT-O [38] $\blacklozenge$ | 2013 | CVPR | ✓ | ✓ | - | - | - | ✓ | ✓ | -
8 |  ECSSD [37] $\blacklozenge$ | 2013 | CVPR | ✓ | - | - | - | - | - | ✓ | -
9 |  PASCAL-S [39] $\blacklozenge$ | 2014 | CVPR | ✓ | - | - | - | - | - | ✓ | -
10 |  HKU-IS [40] $\blacklozenge$ | 2015 | CVPR | ✓ | - | - | - | - | - | ✓ | -
11 |  SOS [63] $\lozenge$ | 2015 | CVPR | ✓ | ✓ | - | - | - | ✓ | - | -
12 |  MSO [63] $\lozenge$ | 2015 | CVPR | ✓ | - | - | - | - | ✓ | - | -
13 |  XPIE [85] $\lozenge$ | 2017 | CVPR | ✓ | ✓ | - | - | - | - | ✓ | -
14 |  ILSO [61] $\lozenge$ | 2017 | CVPR | - | - | - | - | - | - | ✓ | ✓
15 |  JOT [86] $\lozenge$ | 2017 | FCS | ✓ | ✓ | ✓ | - | - | - | ✓ | -
16 |  DUTS [41] $\blacklozenge$ | 2017 | CVPR | ✓ | ✓ | - | - | - | - | ✓ | -
17 |  SOC (OUR) $\blacklozenge$ | 2021 |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

###  2.2 SOD Datasets

In this section, we briefly discuss existing datasets designed for SOD tasks,
focusing in particular on aspects including annotation type, number of salient
objects per image, number of images, and image quality. These datasets are
listed in Table I.

Early datasets are either limited in their numbers of images or in their
coarse annotations of salient objects. For example, salient objects in the
original version of MSRA-A [32] and MSRA-B [32] are only roughly annotated in
the form of bounding boxes. ASD [82], SED1 [33] and MSRA10K [35] contain only
one salient object in most images, while the SED2 [33] dataset provides two
objects per image but contains only 100 images. In order to improve the
quality of datasets, researchers in recent years have started to collect
images with multiple objects in relatively complex and cluttered backgrounds.
The new datasets include ECSSD [37], DUT-O [38], Judd-A [36], and PASCAL-S
[39]. These datasets are improved in terms of both annotation quality and
number of images, compared to their predecessors. To resolve the shortcomings
still present, some datasets (_e.g._ , HKU-IS [40], XPIE [85], and DUTS [41])
provide large amounts of pixel-wise labeled images (Fig. 3.b) with more than
one salient object per image. However, they ignore non-salient objects (1st
row in Fig. 1) and do not offer instance-level annotations (Fig. 3.c). Jiang
et al.  [86] collected roughly 6K simple background images (most of them are
pure texture images) to cover non-salient scenes. However, their dataset,
named JOT, falls short in capturing the complexity of real-world scenes. The
dataset of ILSO [61] contains instance-level salient object annotations but
only roughly labeled boundaries, as shown in Fig. 7. Beyond the “standard” SOD
datasets, there are also several other special datasets that introduce new
tasks, such as salient object subitizing (_i.e._ , SOS [63] and its subset MSO
[63]).

To sum up, as discussed above, existing datasets mostly focus on images with
clear salient objects and simple backgrounds. Considering the aforementioned
limitations of existing datasets, a more realistic dataset, containing non-
salient objects, textures “in the wild”, and salient objects with attributes,
is needed for future investigations in this field. Such a dataset could offer
deeper insight into the strengths and weaknesses of SOD models, and help
overcome performance saturation. Our SOC is unique in that it provides various
high-quality annotations, as shown in Table I.

\begin{overpic}[width=433.62pt]{AnnotationDifference-min}
\put(5.0,-4.0){\footnotesize{(a) Image}} \put(29.0,-4.0){\footnotesize{(b)
Previous}} \put(56.0,-4.0){\footnotesize{(c) Ours}}
\put(77.0,-4.0){\footnotesize{(d) Segmentation}} \end{overpic} Figure 3:
Previous SOD datasets only annotate the images by drawing pixel-accurate
silhouettes around salient objects (b). Different from object segmentation
datasets [87] (d) where (objects are not necessarily salient), our SOC
provides _salient instances_ (c). We provide a high-quality and large-scale
annotated dataset comprised of images that better capture the properties of
real-world scenes.

###  2.3 SOD Models

We have noticed that, from 1998 to the end of Feb. 2021, more than 10,000
papers on saliency detection or related field have been published. In this
section, we try our best to summarize those published in top conferences
(_e.g._ , NeurIPS, CVPR, ICCV, AAAI) and journals (_e.g._ , TPAMI, TIP, TMM),
as well as some high-quality open-access (_i.e._ , arXiv) works. Instead of
briefly describing the pipeline of each model, we summarize key components to
provide a global view.

As shown in Table II, a number of different approaches have been designed to
tackle SOD using super-pixel, proposal, or edge/boundary annotations under
different levels of supervision, such as unsupervised, semi-supervised, and
fully supervised. Using common aggregation strategies (_e.g._ , linear, non-
linear), these methods mainly focus on pixels, regions, and patches to design
more powerful models. Besides, we note that certain priors (_e.g._ , the
center-surround prior, local/global contrast prior, fore/background prior, and
boundary prior) are frequently used in these methods. Some models also utilize
different post-processing steps (_e.g._ , conditional random field,
morphology, watershed, and max-flow strategies) to further improve the
performance.

Table II:  Summary of popular SOD models using handcrafted features. Agg.: Aggregation strategy, _e.g._ , LN = linear, NL = non-linear, HI = hierarchical, BA = Bayesian, AD = adaptive, LS = least-square solver, EM = energy minimization, and GMRF = Gaussian MRF. SL.: Supervision level, _e.g._ , unsupervised ($\bigstar$), semi-supervised ($\bullet$), weakly supervised ($\leftmoon$), fully supervised ($\circ$), active learning ($A$). Sp.: Whether or not superpixel over-segmentation is used. Pr.: Whether or not proposal methods are used. Ed.: Whether or not edge cues are used. Post-Pros.: Whether post-processing methods (_e.g._ , CRF [88], graph-cut [89], GrabCut [90], Ncut [91]), morphology, max-flow (MF) [92] or only thresholding are used.  | # | Model | Publ. | Scholar | Prior. | Uniqueness | Component | Agg. | SL. | Sp. | Pr. | Ed. | Post-Pros
---|---|---|---|---|---|---|---|---|---|---|---|---|---
2010 - 1998 | 1 |  Itti [45] | TPAMI | link | center-surround | pixel | Color, Intensity, Orientation | LN | $\bigstar$ | - | - | - | -
2 |  GBVS [93] | NeurIPS | link | - | pixel | Markovian | - | $\bigstar$ | - | - | - | -
3 |  FT [82] | CVPR | link | frequency domain | pixel | Color, Luminance | - | $\bigstar$ | - | - | - | -
4 |  SR [94] | CVPR | link | spectral residual | pixel | Log Spectrum | - | $\bigstar$ | - | - | - | -
5 |  AIM [95] | NeurIPS | link | maximizing information | patch | Shannon’s Self-information | - | $\bigstar$ | - | - | - | -
6 |  SUN [96] | JOV | link | self-information | pixel | DoG, ICA-derived features | - | $\bigstar$ | - | - | - | -
7 |  FG [97] | MM | link | local contrast | pixel | Fuzzy Growing | - | $\bigstar$ | - | - | - | -
8 |  AC [98] | ICVS | link | local contrast | multi-patch | Color, Luminance | LN | $\bigstar$ | - | - | - | -
9 |  SEG [99] | ECCV | link | local contrast | pixel | Conditional Probabilistic | - | $\bigstar$ | - | - | - | CRF
10 |  MSSS [100] | ICIP | link | symmetric surround | pixel | Color, Luminance | - | $\bigstar$ | - | - | - | graph-cut
11 |  ICC [101] | ICCV | link | isophote | global structure | curvedness, isocenters, color | LN | $\bigstar$ | - | - | - | graph-cut
12 |  EDS [102] | PR | link | - | pixel | threshold, distance, multi-DoG | - | $\bigstar$ | - | - | $\checkmark$ | -
13 |  RE [103] | ICME | link | local contrast | pixel/patch | Contrast pyramid | - | $\bigstar$ | - | - | - | -
14 |  RSA [104] | MM | link | global contrast | patch |  Polar transfer, NN-GPCA [104] | - | $\bigstar$ | - | - | - | -
15 |  RU [105] | TMM | link | rule based | pixel | denoising, geometric | - | $\bigstar$ | - | - | - | -
16 |  CSM [106] | MM | link | frequency&contrast | pixel | Envelope, Skeleton | - | $\bigstar$ | - | - | - | -
2014 - 2011 | 17 |  LSSC [107] | TIP | link | bayesian | pixel/region | convex hull, subspace clustering | NL | $\bigstar$ | $\checkmark$ | - | - | -
18 |  COV [108] | JOV | link | - | pixel/patch | covariance matrices | NL | $\bigstar$ | - | - | - | -
19 |  GR [109] | SPL | link | contrast, center, smooth | - | convex hull, continuous pair | NL | $\bigstar$ | $\checkmark$ | - | - | -
20 |  MSS [110] | SPL | link | local, integrity, center | - | various gaussian, convex hull | NL | $\bigstar$ | $\checkmark$ | - | - | -
21 |  LSMD [111] | AAAI | link | texture, edge, color | pixel/region | hierarchical clustering, gaussian | - | $\bigstar$ | $\checkmark$ | $\checkmark$ | - | threshold
22 |  BSF [112] | ICIP | link | boundary-based | region | convex hull, soft-segmentation | - | $\bigstar$ | $\checkmark$ | - | - | -
23 |  HC [84] | CVPR | link | global contrast | region | Histogram-based Contrast | - | $\bigstar$ | - | - | - | graph-cut
24 |  RC [84] | CVPR | link | global contrast | region | Region-based Contrast | - | $\bigstar$ | - | - | - | graph-cut
25 |  CA [84] | CVPR | link | context-aware | patch | Four principles | - | $\bigstar$ | - | - | - | -
26 |  MR [38] | CVPR | link | fore/back-ground | pixel/region | graph-based manifold ranking | - | $\bigstar$ | $\checkmark$ | - | - | -
27 |  SF [113] | CVPR | link | element contrast | region | uniqueness, spatial | NL | $\bigstar$ | - | - | - | -
28 |  HS [37] | CVPR | link | global contrast | hi-region | Region-scale, Location heuristic | HI | $\bigstar$ | - | - | - | -
29 |  DRFI [114] | CVPR | link | background descriptor | region | region vector, multi-level | LN | $\circ$ | $\checkmark$ | - | - | -
30 |  RBD [115] | CVPR | link | background weighted | region | background connectivity | LS | $\bigstar$ | $\checkmark$ | - | - | -
31 |  LR [116] | CVPR | link | location, semantic, color | pixel/region | Low rank matrix | NL | $\circ$ | $\checkmark$ | - | - | threshold
32 |  PCA [117] | CVPR | link | center-bias priors | patch | color, pattern, gaussian | NL | $\bigstar$ | $\checkmark$ | - | - | -
33 |  HDCT [118] | CVPR | link | high-dimensional color | pixel | Trimap, color transform | LN | $\bigstar$ | $\checkmark$ | - | - | -
34 |  CRFM [119] | CVPR | link | aggregation | pixel | GIST descriptor | NL | $\circ$ | - | - | - | CRF
35 |  STD [120] | CVPR | link | statistical textural | region | Graph, sparse texture | - | $\bigstar$ | - | - | - | GrabCut
36 |  PDE [121] | CVPR | link | representative elements | region | color, background, center | - | $\bigstar$ | $\checkmark$ | - | - | -
37 |  SUB [122] | CVPR | link | Submodular | region | color, spatial, center | - | $\circ$ | $\checkmark$ | - | - | threshold
38 |  PISA [123] | CVPR | link | spatial | pixel/region | color, structure, orientation | NL | $\bigstar$ | - | - | $\checkmark$ | -
39 |  DSR [124] | ICCV | link | reconstruction errors | multi-region | background, obj./centerGaussian | BA | $\bigstar$ | $\checkmark$ | - | - | -
40 |  MC [125] | ICCV | link | markov random walks | region | Markov Chain | - | $\bigstar$ | $\checkmark$ | - | - | -
41 |  GC [126] | ICCV | link | global cue | region | GMM, appearance, spatial | AD | $\bigstar$ | - | - | - | -
42 |  SVO [127] | ICCV | link | center-surround | patch/region | Graph, Obj. | EM | $\bigstar$ | $\checkmark$ | $\checkmark$ | - | -
43 |  CSD [128] | ICCV | link | center-surround | multi-patch | color, orientation, intensity | LN | $\bigstar$ | - | - | - | -
44 |  UFO [129] | ICCV | link | focus, objectness | pixel/region | Uniqueness, Focusness, Obj. | NL | $\bigstar$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | threshold
45 |  CHM [130] | ICCV | link | center-surround, local | mRegion/patch | SVM, hyperedge | LN | $\bullet$ | $\checkmark$ | - | $\checkmark$ | threshold
46 |  CIO [131] | ICCV | link | objectness | Region | Graph, frequency, Obj. | GMRF | $\bigstar$ | $\checkmark$ | - | - | -
47 |  CC [132] | ICCV | link | convexity context | mRegion | concavity, bounding box | - | $\bigstar$ | $\checkmark$ | - | - | graph-cut
48 |  GS [133] | ECCV | link | boundary, connectivity | patch/region | Geodesic distance transform | - | $\bigstar$ | $\checkmark$ | - | $\checkmark$ | -
49 |  CB [134] | BMVC | link | context, shape, center | mRegion | Iterative energy minimization | LN | $\bigstar$ | $\checkmark$ | $\checkmark$ | - | -
50 |  SLMR [135] | BMVC | link | low-rank matrix | Region | sparse noise | - | $\bigstar$ | $\checkmark$ | - | - | -
2018 - 2015 | 51 |  SMD [136] | TPAMI | link | texture, edge, color | pixel/region | hierarchical clustering, gaussian | - | $\bigstar$ | $\checkmark$ | $\checkmark$ | - | threshold
52 |  RS [137] | TPAMI | link | fore/back-ground | region | manifold ranking, grouping cue | - | $\bigstar$ | ✓ | - | - | -
53 |  BFS [138] | NC | link | fore/back-ground seed | region | Gaussian falloff, threshold | NL | $\bigstar$ | $\checkmark$ | - | - | -
54 |  GLC [139] | PR | link | global/local contrast | region | HOG, LBP, codebook,graph-cut | LN | $\bigstar$ | $\checkmark$ | - | - | -
55 |  DSP [140] | PR | link | propagation | region | sink points, chi-square distance | NL | $\bigstar$ | $\checkmark$ | - | $\checkmark$ | -
56 |  LPS [141] | TIP | link | label propagation-base | pixel/region | three-cue-center, affinity matrix | NL | $\bigstar$ | $\checkmark$ | - | - | -
57 |  MAPM [142] | TIP | link | background | region | Markov absorption probability |  | $\bigstar$ | $\checkmark$ | - | - | -
58 |  MIL [143] | TIP | link | instance | region | multi-instance learning, SVM | - | $\bullet$ | $\checkmark$ | $\checkmark$ | - | -
59 |  RCRR [144] | TIP | link | reversion correction | pixel/region | regular-random walks ranking | - | $\bigstar$ | $\checkmark$ | - | - | -
60 |  FCB [145] | TIP | link | fore/back-ground, center | region | color difference, color volume | NL | $\bigstar$ | $\checkmark$ | - | - | -
61 |  NCS [146] | TIP | link | center bias | pixel/region | Ncut, merging scheme | EM | $\bigstar$ | $\checkmark$ | - | $\checkmark$ | Ncut
62 |  MDC [147] | TIP | link | direction contrast | pixel | OTSU, morphological filter | NL | $\bigstar$ | - | - | - | watershed
63 |  HCCH [148] | TIP | link | closure completeness & reliability | object | hierarchical segmentation | NL | $\bigstar$ | - | - | ✓ | -
64 |  JLSE [149] | TIP | link | exemplar-aided | region | joint latent space embedding | - | $\circ$ | $\checkmark$ | - | - | -
65 |  IFC [150] | TMM | link | boundary homogeneity | pixel/region | linear feedback control system | - | $\bigstar$ | $\checkmark$ | - | - | -
66 |  NIO [151] | TNNLS | link | smoothness, boundary | region | graph, iterative optimization | BA | $\bullet$ | ✓ | - | - | -
67 |  MBS [152] | ICCV | link | barrier distance | pixel | backgroundness cue | - | $\bigstar$ | - | - | - | morphology
68 |  GP [153] | ICCV | link | diffusion based | region/pixel | diffusion/laplacian matrix | - | $\bigstar$ | $\checkmark$ | - | - | -
69 |  BSCA [154] | CVPR | link | color/space contrast | region/pixel | cellular automata, bayesian | - | $\bigstar$ | $\checkmark$ | - | - |  OTSU [155]
70 |  BL [156] | CVPR | link | image prior | mRegion |  SVM, MKB [157], LBP | LN | $\circ$ | $\checkmark$ | - | - | -
71 |  MST [158] | CVPR | link | geometry information | pixel | minimum spanning tree | - | $\bigstar$ | $\checkmark$ | - | - | morphology
72 |  RRWR [159] | CVPR | link | error-boundary removal | pixel/region | regular-random walks ranking | - | $\bigstar$ | $\checkmark$ | - | - | -
73 |  TLLT [160] | CVPR | link | propagation,boundary | region | convex hull, teach-to-learn | - | $\bigstar$ | $\checkmark$ | - | - | -
74 |  WSC [161] | CVPR | link | weighted sparse coding | region | color histogram, dictionary | NL | $\bigstar$ | $\checkmark$ | - | - | -
75 |  PM [162] | ECCV | link | propagation | region | extended random walk | LN | $\bigstar$ | $\checkmark$ | - | - | -
2021 - 2019 | 76 |  TSG [163] | TCSVT | link | regionally spatial consistency | region | Sparse Representation, graph | LN | $\bigstar$ | ✓ | - | - | MF
77 |  LFCS [53] | TCSVT | link | smoothness, boundary | region | Discrete Linear Control System | LN | $\bullet$ | ✓ | - | - | -
78 |  AIGC [164] | TCSVT | link | contrast, object | region | irregular graph | - | $\bigstar$ | ✓ | - | - | -
79 |  FTOE [165] | TMM | link | contrast, center, distribute | pixel/region | fuzzy theory, object enhancement | LN | $\bigstar$ | ✓ | ✓ | - | -
80 |  MSGC [166] | TMM | link | fore/back-ground seed | region | multi-scale, global cue | NL | $\bigstar$ | ✓ | - | - | -
81 |  SIA [167] | TMM | link |  boundary, dhs [168] | - | Cellular Automation | BA | $\bigstar$ | ✓ | - | - | -
82 |  KSR [169] | TIP | link |  trained on[32] | region | R-CNN, Rank-SVM, subspace | - | $A$ | - | ✓ | - | -
83 |  MSR [170] | TIP | link | boundary connectivity | region |  MBD [171] | - | $\bigstar$ | ✓ | - | - | OTSU
84 |  LRR [172] | TIP | link | background | pixel/region |  Celluar Automata [154], FCN32 | Metric | $\bigstar$ | ✓ | - | - | -
Table III:  Summary of popular deep learning based SOD models. See Table II for more detailed descriptions. MB = MSRA-B dataset [32]. M10K = MSRA-10K [35] dataset. P-VOC2010 = PASCAL VOC 2010 semantic segmentation dataset [173]. CRF = Conditional random fields. Clicking the scholar will link to the specific author’s google scholar. | # | Model | Publ. | Scholar | #Training | Training Dataset | Backbone | SL. | Sp. | Pr. | Ed. | CRF
---|---|---|---|---|---|---|---|---|---|---|---|---
2015 | 1 |  SupCNN [174] | IJCV | link | 800 |  ECSSD [37] | - | $\circ$ | ✓ | - | - | -
2 |  LEGS [175] | CVPR | link | 340+3,000 |  PASCAL-S [39]+MB[32] | - | $\circ$ | - | ✓ | - | -
3 |  MDF [40] | CVPR | link | 2,500 |  MB[32] | - | $\circ$ | ✓ | - | ✓ | -
4 |  MC [176] | CVPR | link | 8,000 |  M10K [35] |  GoogLeNet [177] | $\circ$ | ✓ | - | - | -
2016 | 5 |  DSL [178] | TCSVT | link | (5,168+10,000)*80% |  DUT-O [38]+M10K [35] |  LeNet [179]/VGGNet16 | $\circ$ | ✓ | - | - | -
6 |  DISC [180] | TNNLS | link | 9,000 |  M10K [35] | - | $\circ$ | ✓ | - | - | -
7 |  DS [181] | TIP | link | 10,000 |  M10K [35] |  VGGNet [182] | $\circ$ | ✓ | - | ✓ | ✓
8 |  SSD [183] | ECCV | link | 2,500 |  MB[32] |  AlexNet [184] | $\circ$ | ✓ | ✓ | - | -
9 |  CRPSD [185] | ECCV | link | 10,000 |  M10K [35] | VGGNet | $\circ$ | ✓ | - | - | -
10 |  RFCN [186] | ECCV | link | 10,103+10,000 |  P-VOC2010 [173]+M10K [35] | VGGNet | $\circ$ | ✓ | - | ✓ | -
11 |  MAP [187] | CVPR | link |  $\sim$5,500 |  SOS [63] | VGGNet | $\circ$ | - | ✓ | - | -
12 |  SU [188] | CVPR | link | 15,000+10,000 |  SALI [189]+M10K [35] | VGGNet | $\circ$ | - | - | - | ✓
13 |  RACD [190] | CVPR | link | 10,565 |  DUT-O [38]+NJU [191]+NLP [192] | VGGNet | $\circ$ | - | - | - | -
14 |  ELD [193] | CVPR | link | 9,000 |  M10K [35] | VGGNet | $\circ$ | ✓ | - | - | -
15 |  DHS [168] | CVPR | link | 3,500+6,000 |  DUT-O [38]+M10K [35] | VGGNet | $\circ$ | - | - | - | -
16 |  DCL [194] | CVPR | link | 2,500 |  MB[32] | VGGNet | $\circ$ | ✓ | - | - | ✓
2017 | 17 |  DLS [195] | CVPR | link | 10,000 |  M10K [35] | VGGNet | $\circ$ | ✓ | - | - | -
18 |  MSRNet [61] | CVPR | link | (500+)2,500+2,500 |  (ILSO [61]+)MB [32]+HKU-IS [40] | VGGNet | $\circ$ | - | ✓ | ✓ | ✓
19 |  SRM [196] | CVPR | link | 10,553 |  DUTS [41] |  ResNet50 [197] | $\circ$ | - | - | - | -
20 |  NLDF [198] | CVPR | link | 2,500 |  MB[32] | VGGNet | $\circ$ | - | - | ✓ | ✓
21 |  WSS [41] | CVPR | link | 456K |  ImageNet [199] | VGGNet | $\circ$ | ✓ | - | ✓ | ✓
22 |  DSS [200] | CVPR | link | 2,500 |  HKU-IS [40]+MB[32] | VGGNet | $\circ$ | - | - | ✓ | ✓
23 |  FSN [201] | ICCV | link | 10,000 |  M10K [35] | VGGNet | $\circ$ | - | - | - | -
24 |  SVF [202] | ICCV | link | 10,000 |  M10K [35] | VGGNet | $\leftmoon$ | ✓ | - | - | -
25 |  UCF [203] | ICCV | link | 10,000 |  M10K [35] | VGGNet | $\circ$ | - | - | - | -
26 |  AMU [204] | ICCV | link | 10,000 |  M10K [35] | VGGNet | $\circ$ | - | - | ✓ | -
2018 | 27 |  EAR [205] | TCYB | link | 2,500+2,500 |  HKU-IS [40]+MB[32] | VGGNet16 | $\circ$ | - | - | - | -
28 |  Refinet [206] | TMM | link | 3,000 |  MB[32] | VGGNet16 | $\circ$ | ✓ | - | ✓ | ✓
29 |  LICNN [207] | AAAI | link | 456K |  ImageNet [199] | VGGNet | $\circ$ | - | - | - | -
30 |  ASMO [54] | AAAI | link | 82,783+2,500+2,500 |  MsCO [87]+HKU-IS [40]+MB[32] | ResNet101 | $\circ$ | - | - | - | ✓
31 |  RADF [208] | AAAI | link | 10,000 |  M10K [35] | VGGNet | $\circ$ | - | - | - | ✓
32 |  R3Net [209] | IJCAI | link | 10,000 |  M10K [35] |  ResNeXt [210] | $\circ$ | - | - | - | ✓
33 |  C2SNet [211] | ECCV | link | 20,000+10,000 |  Web [211]+M10K [35] | VGGNet | $\circ$ | ✓ | ✓ | - | -
34 |  RAS [212] | ECCV | link | 2,500 |  MB[32] | VGGNet | $\circ$ | - | - | - | -
35 |  LPSNet [213] | CVPR | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | - | -
36 |  RSOD [214] | CVPR | link | 425 |  PASCAL-S [39] | ResNet101 | $\circ$ | - | ✓ | - | -
37 |  DUS [58] | CVPR | link | 2,500 |  MB[32] | ResNet101 | $\leftmoon$ | - | - | - | -
38 |  ASNet [215] | CVPR | link | 15,000+10,000+5,168 |  SALI [189]+M10K [35]+DUT-O [38] | VGGNet | $\circ$ | - | - | - | -
39 |  BMPM [216] | CVPR | link | 10,553 |  DUTS [41] | VGGNet | $\circ$ | - | - | - | -
40 |  DGRL [217] | CVPR | link | 10,553 |  DUTS [41] | ResNet50 | $\circ$ | - | - | - | -
41 |  PiCA [218] | CVPR | link | 10,553 |  DUTS [41] | VGGNet16/ResNet50 | $\circ$ | - | - | - | ✓
42 |  PAGRN [219] | CVPR | link | 10,553 |  DUTS [41] | VGGNet19 | $\circ$ | - | - | - | -
2019 | 43 |  SE2Net [220] | arXiv | link | 10,553 |  DUTS [41] | VGGNet/ResNeXt101 | $\circ$ | - | - | - | -
44 |  DRMC [221] | arXiv | link | 10,533 |  DUTS [41] | VGGNet/ResNet101 | $\circ$ | - | - | - | ✓
45 |  RDSNet [222] | arXiv | link | 10,000+10,553 |  M10K [35]+DUTS [41] | VGGNet/ResNet-152 | $\circ$ | - | - | - | ✓
46 |  AADF [223] | TCSVT | link | 10,553 |  DUTS [41] |  DenseNet161 [224] | $\circ$ | - | - | - | -
47 |  CCAL [225] | TMM | link | 9,000 |  M10K [35] | VGGNet | $\circ$ | - | - | - | -
48 |  DeepUSPS [59] | NeurIPS | link | 2,500 |  MB [32] |  DRN-network [226] | $\leftmoon$ | - | - | - | -
49 |  FBG [227] | TIP | link | 2,500 |  MB [32] | VGGNet16 | $\circ$ | - | - | ✓ | -
50 |  SPA [228] | TIP | link | 4,000 |  HKU-IS [40] | - | $\circ$ | ✓ | - | - | ✓
51 |  ConnNet [229] | TIP | link | 2,500+2,500 |  MB [32]+HKU-IS [40] | ResNet50 | $\circ$ | - | - | - | -
52 |  LFRWS [230] | TIP | link | 10,000 |  M10K [35] | VGGNet16 | $\circ$ | - | - | ✓ | -
53 |  RSR [66] | TPAMI | link | 425 |  Extended of PASCAL-S [39] | ResNet101 | $\circ$ | - | - | - | -
54 |  SSNet [231] | TPAMI | link | 10,000 |  M10K [35] | VGGNet16 | $\leftmoon$ | ✓ | - | - | -
55 |  LVNet [232] | TGRS | link | 600 |  ORSSD [232] | - | $\circ$ | - | - | - | -
56 |  Deepside [233] | NC | link | 2,500+10,553 |  MB [32]+DUTS [41] | VGGNet16 | $\circ$ | ✓ | - | - | -
57 |  SuperVAE [234] | AAAI | link | - | - | VGGNet19 | $\leftmoon$ | ✓ | - | - | -
58 |  DEF [235] | AAAI | link | 10,553 |  DUTS [41] | ResNet101 | $\circ$ | - | - | - | -
59 |  CapSal [55] | CVPR | link | 82,783+5,265 |  MsCO [87]+COCO-CapSal [55] | ResNet101 | $\leftmoon$ | - | - | - | -
60 |  MWS [236] | CVPR | link | 300,000+10,553 |  ImageNet [199]+DUTS [41] | - | $\leftmoon$ | ✓ | - | - | ✓
61 |  MLMS [237] | CVPR | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | ✓ | -
62 |  ICNet [238] | CVPR | link | 10,000 |  M10K [35] | VGGNet16/ResNet50 | $\circ$ | - | - | - | ✓
63 |  AFNet [239] | CVPR | link | 10,533 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | ✓ | -
64 |  PFANet [240] | CVPR | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | ✓ | -
65 |  PAGE [241] | CVPR | link | 10,000 |  M10K [35] | VGGNet16 | $\circ$ | - | - | ✓ | ✓
66 |  CPD [242] | CVPR | link | 10,533 |  DUTS [41] | VGGNet/ResNet50 | $\circ$ | - | - | - | -
67 |  PoolNet [243] | CVPR | link | 10,533 |  DUTS [41] | VGGNet/ResNet | $\circ$ | - | - | ✓ | -
68 |  BASNet [244] | CVPR | link | 10,553 |  DUTS [41] |  ResNet34/Xavier [245] | $\circ$ | - | - | ✓ | -
69 |  JDF [246] | ICCV | link | 2,500 |  MB [32] | VGGNet16 | $\circ$ | - | - | ✓ | -
70 |  DPOR [247] | ICCV | link | 10,533 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | - | -
71 |  JLNet [248] | ICCV | link | 10,582+10,533 |  P-VOC2010 [173]+DUTS [41] | DenseNet169 | $\circ$ | - | - | - | ✓
72 |  GLFN [50] | ICCV | link | 1,600+10,533 |  HRSOD [50]+DUTS [41] | VGGNet | $\circ$ | - | - | - | ✓
73 |  SIBA [249] | ICCV | link | 10,533 |  DUTS [41] | ResNet50 | $\circ$ | - | - | ✓ | -
74 |  SCRNet [44] | ICCV | link | 10,533 |  DUTS [41] | ResNet50 | $\circ$ | - | - | ✓ | -
75 |  EGNet [250] | ICCV | link | 10,533 |  DUTS [41] | VGGNet/ResNet | $\circ$ | - | - | ✓ | -
2020 | 76 |  HUAN [251] | TIP | link | 10,553 |  DUTS [41] | VGGNet/ResNet/ResNetXt | $\circ$ | - | - | - | ✓
77 |  ALM [252] | TIP | link | 10,000+4,447 |  M10K [35]+HKU-IS [40] | DenseNet | $\circ$ | ✓ | - | - | -
78 |  HFFNet [253] | TIP | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | ✓ | -
79 |  DFI [254] | TIP | link | 10,553 |  DUTS [41] | ResNet50 | $\circ$ | - | - | ✓ | -
80 |  R2Net [255] | TIP | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | - | -
81 |  MRNet [256] | TIP | link | 10,553 |  DUTS [41] | ResNet50 | $\circ$ | - | - | - | -
82 |  CIG [257] | TIP | link | 10,000 |  M10K [35] | VGGNet16 | $\circ$ | - | - | ✓ | -
83 |  RASNet [258] | TIP | link | 2,500 |  MB [32] | VGGNet16 | $\circ$ | - | - | - | -
84 |  ASNet [259] | TPAMI | link | 15,000+10,000+5,168 |  SALI [189]+M10K [35]+DUT-O [38] | VGGNet | $\circ$ | - | - | - | -
85 |  DNNet [260] | TCYB | link | 2,500+2,500 |  MB [32]+HKU-IS [40] | - | $\circ$ | - | - | - | -
86 |  CAANet [261] | TCYB | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | - | -
87 |  ROSA [262] | TCYB | link | 2,500+5,168+2,500 |  HKU-IS [40]+DUT-O [38]+MB[32] |  FCN [263] | $\circ$ | ✓ | - | - | -
88 |  DSRNet [264] | TCSVT | link | 10,553 |  DUTS [41] | DenseNet | $\circ$ | - | - | - | -
89 |  EGNL [265] | TCSVT | link | 2,500 |  MB [32] | VGGNet16 | $\circ$ | - | - | ✓ | -
90 |  SACNet [266] | TCSVT | link | 10,553 |  DUTS [41] | ResNet101 | $\circ$ | - | - | - | -
91 |  FLGC [267] | TMM | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | - | -
92 |  TSNet [268] | TMM | link | 4,000 |  MD4K [268] | ResNet50/VGGNet16 | $\circ$ | - | - | - | -
Table IV:  Summary of popular deep learning based SOD models. See Tables II & III for more detailed descriptions.  | # | Model | Publ. | Scholar | #Training | Training Dataset | Backbone | SL. | Sp. | Pr. | Ed. | CRF
---|---|---|---|---|---|---|---|---|---|---|---|---
2020 | 93 |  SUCA [269] | TMM | link | 10,553 |  DUTS [41] | ResNet50 | $\circ$ | - | - | - | -
94 |  MIJR [270] | TMM | link | 2,500+5,000 |  MB [32]+DUTS [41] | VGGNet16 | $\circ$ | ✓ | ✓ | - | ✓
95 |  CAGVgg [271] | PR | link | 10,553 |  DUTS [41] |  VGGNet/ResNet/NASNet [272] | $\circ$ | - | - | - | -
96 |  U2Net [273] | PR | link | 10,553 |  DUTS [41] | UNet | $\circ$ | - | - | - | -
97 |  SalGAN [274] | TII | link | 10,000 |  M10K [35] | VGGNet16 | $\circ$ | - | - | - | -
98 |  ADA [275] | AAAI | link | 2,500+780 |  MB [32]+NIR [275] | VGGNet16 | $\circ$ | - | - | - | -
99 |  PFPNet [276] | AAAI | link | 10,553 |  DUTS [41] | ResNet101 | $\circ$ | - | - | - | -
100 |  GCPANet [277] | AAAI | link | 10,553 |  DUTS [41] | ResNet50 | $\circ$ | - | - | - | -
101 |  F3Net [278] | AAAI | link | 10,553 |  DUTS [41] | ResNet50 | $\circ$ | - | - | ✓ | -
102 |  LDF [279] | CVPR | link | 10,553 |  DUTS [41] | ResNet50 | $\circ$ | - | - | ✓ | -
103 |  ITSD [280] | CVPR | link | 10,553 |  DUTS [41] | VGGNet16/ResNet50 | $\circ$ | - | - | ✓ | -
104 |  SANet [56] | CVPR | link | 10,553 |  DUTS [41] | VGGNet16 | $\leftmoon$ | - | - | ✓ | ✓
105 |  MINet [281] | CVPR | link | 10,553 |  DUTS [41] | VGGNet16/ResNet50 | $\circ$ | - | - | - | -
106 |  ABPNet [282] | ECCV | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | ✓ | -
107 |  CSNet [283] | ECCV | link | 10,553 |  DUTS [41] | - | $\circ$ | - | - | - | -
108 |  GateNet [284] | ECCV | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | - | ✓
2021 | 109 |  DNA [285] | TCYB | link | 10,553 |  DUTS [41] | VGGNet16/ResNet50 | $\circ$ | - | - | - | -
110 |  DAFNet [51] | TIP | link | 1,400 |  EORSSD [51] | VGGNet16 | $\circ$ | - | - | ✓ | -
111 |  HGA [286] | TIP | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | ✓ | -
112 |  HIRN [287] | TIP | link | 10,553 |  DUTS [41] | VGGNet16 | $\circ$ | - | - | ✓ | -
113 |  SCWS [288] | AAAI | link | 10,553 |  SDUTS [56] | ResNet50 | $\leftmoon$ | - | - | - | -
114 |  PFS [289] | AAAI | link | 10,553 |  DUTS [41] | ResNet50 | $\circ$ | - | - | ✓ | -
115 |  KRNet [290] | AAAI | link | 10,553 |  DUTS [41] | ResNet50 | $\circ$ | - | - | ✓ | -
116 |  BAS [31] | arXiv | link | 10,553 |  DUTS [41] | ResNet34/Xavier | $\circ$ | - | - | ✓ | -
117 |  ICON [52] | arXiv | link | 10,553 |  DUTS [41] | ResNet50 | $\circ$ | - | - | - | -

More recently, many deep learning SOD models based on different network
architectures, such as multi-layer perceptrons, fully convolutional networks
(FCNs), hybrid networks and capsules, have been proposed and achieve higher
performance than traditional methods. According to the learning paradigm, most
deep SOD models can be roughly split into two types: single-task learning and
multi-task learning methods. We summarize the training data, backbones, and
other components in Tables III and IV.

We mainly focus on macro-level statistics rather than micro-level
descriptions. We kindly refer readers to the recent architecture review [76].
We hope this comprehensive review can serve as guidance444Research group:
https://github.com/DengPingFan/Saliency-Authors. for future researchers in
this fast-growing field.

###  2.4 Dataset-Enhancement Strategies for Deep Models

Existing deep SOD models focus on designing effective decoders [44, 278, 264,
261, 260, 51, 285] to aggregate features from different levels of the backbone
network [197, 210, 291]. We argue that, as they employ a mapping function from
the input training image set to the output training ground-truth set, deep
models should also focus on dataset-enhancement strategies to improve model
generalization ability. Three different strategies have been widely studied,
including label smoothing [292], image augmentation [293, 294], and self-
supervised learning [295].

Instead of training directly with one-hot supervision, “label smoothing”
techniques learn from smoothed supervision, and can thus relax the supervision
signals using the generated smoothing labels [292] or disturbed labels [296].
Miyato et al.  [297] applied local perturbations to data points to increase
the smoothness of the model distribution. Thulasidasan et al.  [298]
discovered that mix-up training [294] with label smoothing can significantly
improve model calibration. To obtain a more robust and generative model, Xie
et al.  [296] randomly replaced a portion of labels with incorrect values in
each iteration. In addition, Wager et al.  [299] demonstrated that corrupting
training examples with noise from known distributions within the exponential
family can inject appropriate generative assumptions into discriminative
models, thus reducing generalization errors. Peterso et al.  presented a soft-
label dataset (CIFAR10H [300]) aiming at reflecting human perceptual
uncertainty by providing label distributions across categories instead of hard
one-hot labels.

Image augmentation [293] is an effective technique for extending the diversity
of a training dataset, thus improving model generalization ability. Existing
data augmentation techniques can be roughly divided into two categories: 1)
human-designed policies, _e.g._ , rotation or scale transformation, and 2)
learned policies [301, 302]. For the former, a predefined data augmentation
policy is applied to the dataset. Beside the widely used rotation and scale
transformations, other extensively studied methods in this category are
erasing techniques [303, 304], which achieve data augmentation by randomly
erasing part of the image patch. Further, mix-up methods [305, 306] utilize
the mix-up data augmentation strategy to generate new samples from an existing
training dataset to mitigate the uncertainty in prediction. For the latter
[301], the network learns an image-conditioned data augmentation policy, which
is usually parameterized by a deep neural network. In this way, the input
image is fed to the data augmentation network to generate augmented samples
with hyperparameters that control the degree of data augmentation.

Self-supervised learning [307, 295], also termed as consistency learning,
defines an annotation-free pretext task to provide a surrogate supervision
signal for feature learning. Conventionally, self-supervised learning is used
for unsupervised representation learning to learn the feature embedding of the
image or video. Recently, works have defined self-supervised learning as an
auxiliary task, and used it within a weakly supervised [288] or semi-
supervised learning framework [308]. Several recent and representative arts
can be found in [309, 310, 311].

As far as we know, no existing salient object detection works have focused on
exploring the dataset bias issue with dataset-enhancement strategies. In this
paper, we claim that efforts on developing dataset-improvement strategies can
also yield significant performance gains. Further, these solutions are general
and can be easily applied to existing saliency detection networks.

##  3 SOC Dataset

In this section, we present details of our new challenging SOC dataset
designed to reflect real-world scenes. Sample images from SOC are shown in
Fig. 1, while statistics regarding the categories and attributes are shown in
Fig. 4 (a) and Fig. 6, respectively. Based on the strengths and weaknesses of
existing datasets, we identify seven crucial requirements that a comprehensive
and balanced dataset should fulfill.

\begin{overpic}[width=433.62pt]{NewStatistical-min-new}
\put(40.0,65.0){Numbers per category (log scale)}
\put(-0.5,67.0){\small{$10^{2}$} } \put(-0.5,62.0){\small{$10^{3}$} }
\put(-0.5,57.0){\small{$10^{1}$} } \put(-0.5,52.0){\small{$10^{0}$} }
\put(53.0,45.0){(a)} \par\put(19.0,23.0){(b)}
\put(27.5,38.0){\scriptsize{ILSO}} \put(27.5,35.5){\scriptsize{SOC}}
\put(1.0,33.0){\rotatebox{90.0}{proportion}} \put(12.0,25.0){global color
contrast} \par\put(53.0,23.0){(c)} \put(51.0,38.0){\scriptsize{ILSO}}
\put(51.0,35.5){\scriptsize{SOC}}
\put(34.0,33.0){\rotatebox{90.0}{proportion}} \put(46.0,25.0){local color
contrast} \par\put(84.0,23.0){(d)} \par\put(19.0,-1.0){(e)}
\put(13.0,1.0){location distribution}
\put(1.0,10.0){\rotatebox{90.0}{proportion}} \put(24.8,11.5){\small{$r_{o}$}}
\put(24.8,9.8){\small{$r_{m}$}} \put(24.8,8.0){\small{$r_{o}$}}
\put(24.8,6.0){\small{$r_{m}$}} \par\put(84.0,-1.0){(g)}
\put(75.0,11.2){\small Appearance Change (AC)} \put(80.0,2.0){\small Clutter
(CL)} \par\put(53.0,-1.0){(f)} \put(51.0,14.5){\scriptsize{ILSO}}
\put(51.0,12.0){\scriptsize{SOC}} \put(48.0,1.0){instance size}
\put(34.0,10.0){\rotatebox{90.0}{proportion}} \end{overpic} Figure 4: (a)
Number of annotated instances per category in our SOC dataset. (b, c) Global
and local color contrast statistics, respectively. (d) A set of saliency maps
from our dataset and their overlay map. (e) Location distribution of the
salient objects in SOC. (f) Distribution of instance sizes in the SOC and ILSO
[61] datasets. (g) Visual examples of attributes. Best view on screen and
zoomed-in for details.  \begin{overpic}[width=433.62pt]{NonSal-min}
\put(14.0,-2.0){\small{(a)}} \put(46.0,-2.0){\footnotesize{(b)}}
\put(83.0,-2.0){\footnotesize{(c)}} \end{overpic} Figure 5: Examples of non-
salient objects in our dataset. a) Crowded scene, b) motion blur, and c)
background with non-interesting regions.

1) Presence of Non-Salient Objects. Most existing SOD datasets assume that an
image should contain at least one salient object and thus discard images
without salient objects [86]. However, this assumption is only true under
ideal settings, which leads to _data selection bias_. In realistic settings,
images do not always contain salient objects. For example, some images of
amorphous backgrounds, such as sky, grass or textures, contain no salient
objects at all [312]. The non-salient objects or background “stuff” may occupy
the entire scene, and hence heavily constrain the possible locations of a
salient object. Xia et al.  [85] proposed a state-of-the-art SOD model the
determines what is or is not a salient object, indicating that non-salient
objects are crucial for reasoning salient objects. This suggests that non-
salient objects deserve equal attention in SOD. Incorporating images
containing non-salient objects makes a dataset more realistic, and hence more
challenging. We define “ _non-salient objects_ ” as images without salient
objects or images with “stuff” categories. As suggested in [85, 312], the
“stuff” categories include (a) densely distributed similar objects, (b) fuzzy
shapes, and (c) regions without semantics, as illustrated in Fig. 5 (a)-(c),
respectively.

Table V: List of salient object image attributes and their corresponding descriptions. These attributes are derived by studying the characteristics of existing datasets. Some visual examples can be found in Fig. 1 and Fig. 4 (g). For more examples, please refer to the supplementary materials.  Attribute | Description
---|---
AC _(Appearance Change)_ | Obvious illumination change in the object region.
BO _(Big Object)_ | The ratio between the object area and the image area is larger than 0.5.
CL _(Clutter)_ | Foreground and background regions around the object have similar colors. We labeled images with a global
|  color contrast value larger than 0.2 and local color contrast value smaller
than 0.9 as cluttered images ($\S$ 3).
HO _(Heterogeneous Objects)_ | Objects composed of visually distinctive/dissimilar parts.
MB _(Motion Blur)_ | Objects have fuzzy boundaries due to camera shaking or motion.
OC _(Occlusion)_ | Objects are partially or fully occluded.
OV _(Out-of-View)_ | Part of the object is clipped by the image boundaries.
SC _(Shape Complexity)_ |  Objects have complex boundaries, such as thin parts (_e.g._ , the foot of animal) and holes.
SO _(Small Object)_ | The ratio between the object area and the image area is smaller than 0.1.

To avoid data selection bias, we selected images randomly and automatically,
as suggested by Torralba and Efros [43]. Based on the characteristics of non-
salient objects, we randomly collected 783 texture images from the DTD [313]
dataset. To enrich the diversity, 2,217 images including aurora, sky, crowds,
store and many other kinds of realistic scenes were gathered from the Internet
and other datasets [87, 39, 34, 86].

2) Number and Categories of Images. Providing a large number of images is
essential for capturing the diversity and abundance of real-world scenes.
Moreover, with large amounts of data, SOD models can avoid overfitting and
enhance generalization. To this end, we first randomly gathered 3,000 images
from the MS-COCO dataset [87], which contains ‘everyday scenes of common
objects in their natural context.’ Then, more than 80 object categories (see
supplementary materials) were annotated. Note that we separated the process of
data selection and labeling to avoid data selection bias, as discussed in
[43]. Please refer to the subsection “7) High-Quality Salient Object Labeling”
for details on this. Fig. 4 (a) shows the number of salient objects in each
category. As can be seen, the “person” category accounts for a large
proportion of the data, which is reasonable as people usually appear in daily
scenes along with other objects. We divided our dataset (3k non-salient images
and 3k salient images) into training, validation and test sets in the ratio of
6:2:2.

3) Global vs. Local Color Contrast of Salient Objects. As described in [39],
the term “salient” is related to the global/local contrast of the foreground
and background. It is essential to confirm whether the salient objects are
easy to detect. For each object, we compute separate RGB color histograms for
the foreground and background. Then, $\chi^{2}$ distance is utilized to
measure the distance between the two histograms. The global and local color
contrast distributions are shown in Fig. 4 (b) and (c), respectively. Compared
to ILSO, the SOC dataset has a higher proportion of objects with low global
and local color contrast.

4) Locations. _Center bias_ has been identified as one of the most significant
and challenging biases pertaining to saliency detection datasets [69, 39,
314]. Fig. 4 (d) illustrates a set of images and their overlay map (_i.e._ ,
average mask map). As can be seen, although salient objects are located at
different positions, the overlay map shows that somehow these images are still
center biased. Unfortunately, previous benchmarks have often adopted this
incorrect approach to analyze the positional distribution of salient objects.
To avoid this misleading phenomenon, in Fig. 4 (e), we plot the statistics of
two quantities, $r_{o}$ and $r_{m}$, which denote how far an object center and
its farthest (margin) point are from the image center, respectively. Both
$r_{o}$ and $r_{m}$ are divided by half the diagonal length of the image for
normalization, such that $r_{o},r_{m}\in[0,1]$. From these statistics, we
observe that the salient objects in our dataset do not suffer from center
bias.

5) Size of Salient Objects. The size of an instance-level salient object is
defined as the proportion of its pixels to those in the overall image [39]. As
shown in Fig. 4 (f), the sizes of salient objects in our SOC vary greatly
compared with the only other existing instance-level dataset, ILSO [61].
Further, there is a higher proportion of medium-sized objects in SOC.

Figure 6: Left: Attribute distribution over salient object images in our SOC
dataset. Each number in the grid indicates the number of occurrences. Right:
The dominant dependencies among attributes based on the frequency of
occurrences. A larger link width indicates a higher probability of an
attribute occurring with other ones.

6) Salient Objects with Attributes. Having attribute information for the
images in a dataset helps objectively assess the performance of models over
different types of parameters and variations. It also allows for the
inspection of model failures. To this end, we define a set of attributes to
represent specific situations encountered in real-world scenes, such as
_motion blur_ , _occlusion_ and _cluttered background_ (summarized in Table
V). Note that an image can be annotated with multiple attributes as these
attributes are not exclusive.

Inspired by [315], we present the distribution of attributes over the dataset
in the left of Fig. 6. The _SO_ attribute makes up the largest proportion due
to our accurate instance-level annotations (_e.g._ , the tennis racket in Fig.
3). The _HO_ attribute also accounts for a large proportion, because the real-
world scenes are composed of different constituent materials. _Motion blur
(MB)_ is more common in video frames, but also sometimes occurs in still
images. Thus, _MB_ images make up a relatively small proportion of our
dataset. Since a realistic image usually contains multiple attributes, we show
the dominant dependencies among attributes based on the frequency of
occurrence on the right of Fig. 6. For example, a scene containing several
heterogeneous objects is likely to have a large number of objects occluding
each other and forming complex spatial structures. Thus, the _HO_ attribute
has a strong dependency with _OC_ , _OV_ , and _SO_.

7) High-Quality Salient Object Labeling. As noted in [316], training on the
ECSSD dataset (1,000 images) yields better results than when using other
datasets (_e.g._ , MSRA10K with 10,000 images). This is because, besides the
scale, dataset quality is also an important. To obtain a large number of high-
quality images, we randomly selected images from the MS-COCO dataset [87],
which is a large-scale real-world dataset whose objects are annotated with
polygons (_i.e._ , coarse labeling). High-quality labels also play a critical
role in improving the accuracy of SOD models [82]. Towards this end, we re-
labeled the dataset with pixel-wise annotations. Following other famous task-
oriented SOD benchmark datasets [82, 33, 35, 86, 61, 40, 32, 34, 41, 85, 37],
we did not use an eye tracking device. We took two steps to ensure high-
quality annotations: (i) We asked five viewers to annotate objects that they
thought were salient in each image with bounding boxes (bboxes), and (ii) we
kept the images in which the majority ($\geq 3$) of viewers annotated the same
objects (IOU of the bbox $>0.8$). After this first stage, we had 3,000 salient
object images annotated with bboxes. In the second stage, we further manually
labeled accurate silhouettes of the salient objects according to the bboxes.
Note that we had 10 volunteers involved in both steps to cross-check the
quality of annotations. In the end, we kept 3,000 images with high-quality,
instance-level labeled salient objects. As shown in Fig. 7 (b & d), the
boundaries of our object labels are precise, sharp and smooth. During the
annotation process, we also added some new categories (_e.g._ , _computer
monitor, hat, pillow_) that are not labeled in the MS-COCO dataset [87].

\begin{overpic}[width=424.94574pt]{HighQualityAnnotation.pdf}
\put(16.0,26.0){\footnotesize{(a) ILSO} } \put(68.0,26.0){\footnotesize{(b)
SOC} } \put(13.0,2.0){\footnotesize{(c) MS-COCO} }
\put(68.0,2.0){\footnotesize{(d) SOC} } \end{overpic} Figure 7: Compared with
the recent instance-level ILSO dataset [61] (a), which is labeled with
discontinuous coarse boundaries, and MS-COCO dataset [87] (c), which is
labeled with polygons, our SOC dataset (b & d) is labeled with smooth fine
boundaries.

##  4 Our Dataset-Enhancement Strategies

Instead of focusing on designing a strong decoder for feature aggregation, we
introduce three simple dataset-enhancement strategies to achieve better model
generalization ability. We argue that the proposed strategies are easy to
implement by all existing fully supervised SOD models, and yield good
performance with little effort. Let us define the RGB saliency training
dataset as $D=\\{x_{i},y_{i}\\}_{i=1}^{N}$, where $x_{i},y_{i}$ are an input
RGB image and its corresponding ground-truth (GT) saliency map, $i$ indexes
the training images, and $N$ is the size of the training dataset. As SOD is a
binary prediction task, the GT saliency map $y$ is usually a binary map, and
most existing SOD techniques employ a binary (or weighted) cross-entropy loss
function to evaluate the saliency prediction. In this paper, instead of
defining the GT saliency map as a binary segmentation map, we first introduce
“label smoothing” [292] as an effective technique to achieve both efficient
model training and high model performance. Then, we adopt random image
augmentation to generate diverse samples for better model generalization
ability. Finally, as a widely studied technique in semi-supervised or
unsupervised learning [307, 295], we extend the self-supervised learning
solution to fully supervised SOD to achieve a robust model.

###  4.1 Label Smoothing

Label Smoothing and Knowledge Distillation. One of the most important
scenarios in which to apply label smoothing is the teacher-student net [317]
for knowledge distillation. Typically, in a teacher-student net, the teacher
model has a strong learning capacity, while the student model has a lower one.
The teacher model then teaches the student model by providing the latter with
a “soft target”. As discussed in [318], the “soft target” contains a rich
similarity structure over the data, which is essential for producing an
enhanced student model. Further, label smoothing can be treated as a form of
output distribution regularization that prevents the network from overfitting.
As pointed out in [292], hard labels may lead to the overfitting as the model
will assign full probability to each category, which is not guaranteed to
generalize well. With soft labels, the model learns the structure of the data,
thus preventing it from being over-confident. Following the same data setting,
_e.g._ , employing label smoothing, [319] introduced online label smoothing
solution to gradually update the soft labels based on the model’s prediction.

Conventional Setting. Given an input image $x$ and the corresponding ground-
truth saliency map $y$, the conventional deep saliency model $f_{\theta}$ is
trained to achieve saliency prediction $s=f_{\theta}(x)$ by minimizing the
cross-entropy loss:
$\mathcal{L}_{ce}(y,s)=-\sum_{i=1}^{N}\sum_{u,v}y_{i}^{u,v}\log s_{i}^{u,v}$,
where $(u,v)$ index pixels. For the hard label based framework, we have
$y\in\\{0,1\\}$, where 1 indicates the salient foreground and 0 represents the
background.

Label Smoothing Setting. Different from the above hard label setting, in label
smoothing regularization (LSR) [292], a smoothed label $y^{\prime}$ is used
instead of $y$, which is formulated as:

$y^{\prime}=(1-\epsilon)y+\epsilon u(x).$ (1)

Here, $\epsilon$ is the smoothing parameter, and $u(x)$ is a fixed
distribution, which is usually defined as a uniform distribution. The smoothed
label with a uniform distribution $u(x)$ is then defined as:

$y^{\prime}=(1-\epsilon)y+\frac{\epsilon}{K},$ (2)

where $K$ is the number of categories.

Loss Function. Given smoothed label $y^{\prime}$ and hard label $y$, the loss
function with LSR is defined as:

$\mathcal{L}_{ls}=(1-\alpha)\mathcal{L}_{ce}(y,s)+\alpha\mathcal{L}_{ce}(y^{\prime},s),$
(3)

where $\alpha$ is used to balance the contribution of the smoothed and hard
labels, and the smoothed label related loss is defined as
$\mathcal{L}_{lsr}=\mathcal{L}_{ce}(y^{\prime},s)$. Note that, if there exist
other loss functions, the smoothed label can only be used in cross-entropy
loss.

What Does Label Smoothing Really Do? The conventional cross-entropy loss can
be rewritten as:

$\mathcal{L}_{ce}=-\log s.$ (4)

Here, $s$ is the model prediction after sigmoid activation (for binary
classification), which is defined as:

$s_{j}=e^{z_{j}}/\sum_{k=1}^{K}e^{z_{k}}=1/(1+\sum_{k\neq j}e^{z_{k}-z_{j}}).$
(5)

We then substitute $s$ in (Eq. 4) and obtain:

$\mathcal{L}_{ce}=\log(1+\sum_{k\neq j}e^{z_{k}-z_{j}}).$ (6)

Let us define the gap between the correct class and others as $M=z_{k}-z_{j}$.
We can then conclude that the conventional cross-entropy loss aims to maximize
this gap.

For label smoothing setting, as in (Eq. 2), we rewrite the smoothed label
related loss $\mathcal{L}_{lsr}$ as:

$\displaystyle\mathcal{L}_{lsr}=$
$\displaystyle-((1-\epsilon)y+\epsilon/K)\log
s-(1-(1-\epsilon)y-\epsilon/K)\log(1-s)$ (7) $\displaystyle=$
$\displaystyle-(y\log s+(1-y)\log(1-s))+(\epsilon
y-\frac{\epsilon}{K})\log(\frac{s}{1-s}).$

Using the definition of $s$ in (Eq. 5), we have:

$\frac{s_{j}}{1-s_{j}}=\frac{1}{\sum_{k=1}^{K}e^{z_{k}-z_{j}}-1}.$ (8)

We can then combine (Eq. 8) with (Eq. 7) and obtain:

$\mathcal{L}_{lsr}=\mathcal{L}_{ce}(y,s)+(\epsilon
y-\frac{\epsilon}{K})*\frac{1}{\sum_{k=1}^{K}e^{z_{k}-z_{j}}-1}.$ (9)

The first part of (Eq. 9) aims to maximize the gap between the correct class
and the others, which is same as the conventional binary-cross entropy loss as
in (Eq. 6). The second part works in the opposite direction (compared with
(Eq. 6)) to narrow the gap. In this way, the smoothed label related loss works
to balance the gap between the correct class and others, which serves as an
regularization to prevent the model from being over-confident.

###  4.2 Data Augmentation

As an effective data pre-processing technique, data augmentation aims to
generate new samples from an existing dataset, thus producing a model with
good generalization ability. Given the training dataset
$D=\\{x_{i},y_{i}\\}_{i=1}^{N}$, data augmentation produces a new dataset
$D^{\prime}=\\{x^{\prime}_{i},y^{\prime}_{i}\\}_{i=1}^{N^{\prime}}$. As
discussed previously, two main types of data augmentation have received
particular attention. These include the handcrafted policies and learned
policies [301, 302]. For the learned policies, we observe that the augmented
data can change drastically depending on the context, which may not be an
issue for image classification, but will change the salient attributes of an
image. We thus focus only on handcrafted policies.

For handcrafted data augmentation policies, existing works [303, 304, 305,
306] focus on three main directions: 1) image transformation, _e.g._ , scale
or rotation transformation; 2) mix-up to generate new samples, which are
neighbors of the existing samples; and 3) adding noise to the ground-truth.
Similar to learned policies, the mix-up strategy change the context
information of an image, which is harmful for context-based tasks, such as
salient object detection. In this paper, we therefore focus on two very simple
data augmentation techniques, namely image transformation and adding noise to
the ground-truth. For image transformation, we randomly scale, rotate and crop
part of the image (85% of the original image to keep the context information).
For the additive noise solution, we randomly add Gaussian noise of
distribution $\mathcal{N}(0.1,0.3)$ to the ground-truth saliency map, leading
to a noisy ground-truth map. Note that, for image transformation, we transform
image and ground-truth at the same time, while when adding noise to the
ground-truth, we only process the ground-truth saliency maps.

###  4.3 Self-Supervised Learning

Self-supervised learning learns from an image without knowing the task itself
or the ground-truth, making it an unsupervised representation learning
technique. Conventionally, for the supervised learning setting, the loss
function is defined as $\mathcal{L}_{ce}(y,s)$, where $s$ is the model’s
prediction, and $y$ is the ground-truth map. For self-supervised learning, the
final loss function usually includes two main parts: the conventional cross-
entropy loss $\mathcal{L}_{ce}(y,s)$ and an unsupervised loss that serves as a
regularizer, _i.e._ , $\mathcal{L}(g(x),s)$, where $g(x)$ is the
transformation of the original input image $x$. The two studies [295, 308]
introduced a self-supervised loss with rotation estimation as a pretext task.

Similarly, we introduce a scale/rotation consistency loss function to achieve
scale/rotation invariant predictions. Specifically, given an input image $x$,
we define its prediction as $s$. Then, we apply an image transformation (scale
or rotation transformation) and obtain $x^{t}$. We then perform the same
transformation on the prediction $s$ and obtain $s^{\prime}$. We feed $x^{t}$
to the same salient object detection network to get the saliency prediction as
$s^{t}$. We assume that $s^{\prime}$ and $s^{t}$ should be similar. Then, we
adopt the single scale structural similarity index measure (SSIM) [320, 321]
as a similarity measure, and define the self-supervised loss as:

$\mathcal{L}_{ss}=1-SSIM(s^{\prime},s^{t}).$ (10)

###  4.4 Loss Function with the Proposed Strategies.

With the three introduced data-enhancement strategies, we first apply random
data augmentation to both our training image set and training ground-truth
set, as in Section 4.2. Then we generate the smoothed label following (Eq. 1),
with $K=2$ in this paper to represent the salient foreground and background
regions. In addition to the loss function in (Eq. 3), we also introduce a
self-supervised loss $\mathcal{L}_{ss}$. Our final loss function is then
defined as:

$\mathcal{L}=\mathcal{L}_{ls}+\gamma\mathcal{L}_{ss},$ (11)

where $\gamma$ is introduced to balance the self-supervised loss, and is
empirically set to $\gamma=0.3$ in this paper.

##  5 SOC Benchmark

Based on three criteria (_i.e._ , representative pipeline, open-sourced, and
state-of-the-art performance), we select 46 traditional SOD methods and 54
deep learning models from 201 reviewed methods (see $\S$ 2) to conduct our
benchmark. To the best of our knowledge, this benchmark is the most
comprehensive study in the RGB SOD.

###  5.1 Experimental Setup

####  5.1.1 Evaluation Metrics

Note that the GTs of non-salient images in our SOC dataset are all-zero
matrices, so directly using the traditional F-measure [82] will result in very
low and inaccurate scores. Thus, we utilize three golden metrics (_i.e._ , MAE
[322], maximum E-measure [4], and S-measure [3]) to avoid this issue and to
provide a more reliable assessment. Our python evaluation toolboxes are now
publicly available.555https://github.com/mczhuge/SOCToolbox.

  * •

MAE (${M}$) is the mean absolute error metric, which is widely used to measure
the pixel-level difference between the prediction and the GT.

  * •

E-measure ($E_{\xi}^{max}$) is a new perceptual metric that takes both local
and global similarity into consideration.

  * •

S-measure ($S_{\alpha}$) is a standard metric that quantizes the structural
similarity at a region and object level.

Table VI: SOC dataset used in the benchmarking experiments.  | SOC_train | SOC_val | SOC_test | Total
---|---|---|---|---
Salient Objects (Sal) | 1,800 | 600 | 600 | 3,000
Non-Salient Objects (NonSal) | 1,800 | 600 | 600 | 3,000
Total | 3,600 | 1,200 | 1,200 | 6,000

####  5.1.2 Training and Testing Protocols

The statistics of the SOC dataset used in the benchmark are summarized in
Table VI. For traditional algorithms, we directly test their performance on
the SOC-test set (1,200 images). For deep learning models, we first adopt the
pre-trained models with their recommended training parameter settings under
the default training dataset (see Tables III & IV) and then evaluate them on
the SOC_test set to roughly obtain the top-100 models (see Table VII & VIII).
Finally, we provide a quantitative comparison and detailed analysis of 15 SOTA
approaches, including the top-5 traditional methods and top-10 deep learning
models.

Table VII: Comparison of the traditional SOD algorithms on our SOC_test set (1,200 images) in terms of $S_{\alpha}\uparrow$, $E_{\xi}^{max}\uparrow$, and ${M}\downarrow$ The top-3 results are highlighted in red, blue and green, respectively. The superscript of each score is the corresponding ranking. Details of these methods are summarized in Table II. The overall rank index indicates the average ranking of the three metrics. These results are available at: Google Drive.  | # | Model | Code | $S_{\alpha}\uparrow$ | $E_{\xi}^{max}\uparrow$ | ${M}\downarrow$ | Rank
---|---|---|---|---|---|---|---
2014-before | 1 |  SUN [96] | Matlab | $0.475^{46}$ | $0.688^{44}$ | $0.436^{46}$ | 46
2 |  LSSC [107] | Matlab + C | $0.552^{45}$ | $0.714^{43}$ | $0.365^{45}$ | 45
3 |  BSF [112] | Matlab | $0.554^{44}$ | $0.728^{38}$ | $0.353^{44}$ | 44
4 |  GR [109] | Matlab + C | $0.588^{41}$ | $0.715^{42}$ | $0.332^{42}$ | 43
5 |  HS [37] | EXE | $0.601^{40}$ | $0.729^{37}$ | $0.321^{41}$ | 42
6 |  Itti [45] | Matlab | $0.587^{42}$ | $0.736^{30}$ | $0.311^{39}$ | 41
7 |  AIM [95] | Matlab | $0.605^{39}$ | $0.670^{45}$ | $0.250^{24}$ | 39
8 |  GBVS [93] | Matlab | $0.615^{36}$ | $0.733^{35}$ | $0.293^{37}$ | 39
9 |  LR [116] | Matlab | $0.642^{31}$ | $0.723^{40}$ | $0.253^{27}$ | 36
10 |  CA [323] | Matlab + C | $0.606^{38}$ | $0.750^{22}$ | $0.291^{36}$ | 35
11 |  MR [38] | Matlab + C | $0.645^{29}$ | $0.734^{33}$ | $0.259^{31}$ | 32
12 |  SEG [99] | Matlab + C | $0.576^{43}$ | $0.765^{7}$ | $0.352^{43}$ | 32
13 |  FT [82] | C | $0.626^{34}$ | $0.738^{29}$ | $0.236^{20}$ | 28
14 |  MC [125] | Matlab + C | $0.656^{23}$ | $0.736^{30}$ | $0.251^{25}$ | 26
15 |  CB [134] | Matlab + C | $0.653^{25}$ | $0.758^{13}$ | $0.268^{33}$ | 23
16 |  SR [94] | Matlab/C++ | $0.658^{21}$ | $0.661^{46}$ | $0.156^{4}$ | 23
17 |  PCA [117] | Matlab + C | $0.670^{18}$ | $0.741^{28}$ | $0.209^{13}$ | 17
18 |  MSS [110] | Matlab | $0.682^{12}$ | $0.776^{4}$ | $0.231^{19}$ | 10
19 |  SF [113] | C | $0.699^{6}$ | $0.747^{26}$ | $\textbf{{\color[rgb]{1,0,0}0.130}}^{1}$ | 8
20 |  DSR [124] | Matlab + C | $0.702^{5}$ | $0.751^{20}$ | $0.184^{8}$ | 8
21 |  MSSS [100] | C | $0.683^{11}$ | $0.757^{14}$ | $0.164^{5}$ | 7
22 |  HDCT [118] | Matlab | $0.696^{7}$ | $0.774^{5}$ | $0.201^{12}$ | 6
23 |  DRFI [114] | C | $0.709^{4}$ | $\textbf{{\color[rgb]{0,0,1}0.791}}^{2}$ | $0.197^{11}$ | 4
24 |  COV [108] | Matlab | $\textbf{{\color[rgb]{0,1,0}0.711}}^{3}$ | $0.761^{9}$ | $\textbf{{\color[rgb]{0,0,1}0.146}}^{2}$ | 2
25 |  RBD [115] | Matlab | $\textbf{{\color[rgb]{0,0,1}0.716}}^{2}$ | $\textbf{{\color[rgb]{0,1,0}0.784}}^{3}$ | $0.186^{9}$ | 2
2021-2015 | 26 |  WMR [324] | Matlab + C | $0.640^{32}$ | $0.733^{35}$ | $0.269^{34}$ | 38
27 |  MAPM [142] | Matlab + C | $0.644^{30}$ | $0.722^{41}$ | $0.256^{29}$ | 37
28 |  BL [156] | Matlab + C | $0.623^{35}$ | $0.751^{20}$ | $0.296^{38}$ | 32
29 |  RRWR [159] | Matlab | $0.647^{27}$ | $0.735^{32}$ | $0.258^{30}$ | 31
30 |  WLRR [325] | Matlab + C | $0.614^{37}$ | $0.759^{11}$ | $0.312^{40}$ | 30
31 |  RCRR [144] | Matlab | $0.650^{26}$ | $0.734^{33}$ | $0.255^{28}$ | 29
32 |  GP [153] | Matlab + C | $0.632^{33}$ | $0.759^{11}$ | $0.287^{35}$ | 27
33 |  TLLT [160] | Matlab | $0.656^{23}$ | $0.725^{39}$ | $0.214^{15}$ | 25
34 |  BSCA [154] | Matlab + C | $0.657^{22}$ | $0.755^{16}$ | $0.259^{31}$ | 22
35 |  SMD [136] | Matlab | $0.662^{20}$ | $0.748^{25}$ | $0.246^{22}$ | 21
36 |  MDC [147] | C | $0.675^{16}$ | $0.744^{27}$ | $0.219^{17}$ | 20
37 |  DSP [140] | Matlab + C | $0.664^{19}$ | $0.754^{17}$ | $0.248^{23}$ | 17
38 |  MIL [143] | Matlab + C | $0.671^{17}$ | $0.750^{22}$ | $0.236^{20}$ | 17
39 |  MST [158] | C | $0.647^{27}$ | $0.773^{6}$ | $0.251^{25}$ | 16
40 |  GLC [139] | Matlab + C | $0.676^{15}$ | $0.756^{15}$ | $0.223^{18}$ | 15
41 |  MBS [152] | Matlab | $0.678^{14}$ | $0.753^{18}$ | $0.214^{15}$ | 14
42 |  LPS [141] | Matlab + C | $0.694^{9}$ | $0.749^{24}$ | $0.183^{7}$ | 13
43 |  WFD [326] | C | $0.680^{13}$ | $0.760^{10}$ | $0.213^{14}$ | 12
44 |  BFS [138] | Matlab + C | $0.696^{7}$ | $0.753^{18}$ | $0.195^{10}$ | 10
45 |  WSC [161] | Matlab | $0.693^{10}$ | $0.765^{7}$ | $0.179^{6}$ | 5
46 |  HCCH [148] | Matlab | $\textbf{{\color[rgb]{1,0,0}0.736}}^{1}$ | $\textbf{{\color[rgb]{1,0,0}0.794}}^{1}$ | $\textbf{{\color[rgb]{0,1,0}0.149}}^{3}$ | 1
Table VIII:  Evaluation of 54 deep learning based SOD models on our SOC_test set (1,200 images). We adopt the default implementations listed in Table III and Table IV to test their generalization capability. These results are available at: Google Drive.  | # | Model | Code | $S_{\alpha}\uparrow$ | $E_{\xi}^{max}\uparrow$ | ${M}\downarrow$ | Rank
---|---|---|---|---|---|---|---
2015 | 1 |  LEGS [175] | Caffe | $0.679^{53}$ | $0.765^{54}$ | $0.228^{53}$ | 54
2 |  MDF [40] | Caffe | $0.739^{49}$ | $0.768^{53}$ | $0.144^{43}$ | 49
3 |  MC [176] | Caffe | $0.757^{47}$ | $0.823^{43}$ | $0.138^{35}$ | 43
2016 | 4 |  DSL [178] | Caffe | $0.724^{52}$ | $0.810^{47}$ | $0.194^{52}$ | 51
5 |  DISC [180] | Caffe | $0.735^{51}$ | $0.810^{47}$ | $0.175^{50}$ | 50
6 |  DCL [194] | Caffe | $0.771^{44}$ | $0.836^{39}$ | $0.157^{48}$ | 45
7 |  ELD [193] | Caffe | $0.774^{42}$ | $0.836^{39}$ | $0.138^{35}$ | 40
8 |  DS [181] | Caffe | $0.779^{40}$ | $0.860^{24}$ | $0.155^{46}$ | 37
9 |  DHS [168] | Pytorch | $0.800^{32}$ | $0.848^{33}$ | $0.122^{30}$ | 33
10 |  RFCN [186] | Caffe | $0.814^{23}$ | $0.858^{27}$ | $0.113^{23}$ | 25
2017 | 11 |  UCF [203] | Caffe | $0.654^{54}$ | $0.805^{51}$ | $0.285^{54}$ | 53
12 |  AMU [204] | Caffe | $0.737^{50}$ | $0.808^{50}$ | $0.185^{51}$ | 51
13 |  SVF [202] | Caffe | $0.761^{45}$ | $0.816^{45}$ | $0.156^{47}$ | 47
14 |  WSS [41] | Caffe | $0.778^{41}$ | $0.821^{44}$ | $0.140^{39}$ | 42
15 |  DSS [200] | Caffe | $0.807^{30}$ | $0.858^{27}$ | $0.111^{20}$ | 27
16 |  SRM [196] | Caffe | $0.822^{16}$ | $0.859^{26}$ | $0.111^{20}$ | 21
17 |  MSRNet [61] | Caffe | $0.816^{19}$ | $0.871^{16}$ | $0.117^{25}$ | 20
18 |  NLDF [198] | Tensorflow | $0.816^{19}$ | $0.860^{24}$ | $0.104^{13}$ | 16
2018 | 19 |  RAS [212] | Pytorch | $0.759^{46}$ | $0.813^{46}$ | $0.151^{44}$ | 46
20 |  R3Net [209] | Pytorch | $0.773^{43}$ | $0.825^{42}$ | $0.138^{35}$ | 41
21 |  LPSNet [213] | Pytorch | $0.795^{35}$ | $0.838^{38}$ | $0.143^{42}$ | 39
22 |  DGRL-GLN [217] | Caffe | $0.794^{36}$ | $0.845^{36}$ | $0.141^{40}$ | 38
23 |  C2SNet [211] | Caffe | $0.791^{37}$ | $0.845^{36}$ | $0.138^{35}$ | 36
24 |  PiCA-Res [218] | Pytorch | $0.810^{28}$ | $0.858^{27}$ | $0.128^{31}$ | 31
25 |  BMPM [216] | Tensorflow | $0.810^{28}$ | $0.853^{30}$ | $0.119^{27}$ | 29
26 |  ASNet [215] | Keras | $0.817^{18}$ | $0.865^{20}$ | $0.111^{20}$ | 17
2019 | 27 |  MWS [236] | Pytorch | $0.757^{47}$ | $0.828^{41}$ | $0.172^{49}$ | 47
28 |  AFNet [239] | Caffe | $0.812^{24}$ | $0.850^{32}$ | $0.120^{29}$ | 29
29 |  SIBA [249] | Caffe | $0.800^{32}$ | $0.884^{10}$ | $0.130^{33}$ | 26
30 |  Deepside [233] | Caffe | $0.815^{21}$ | $0.861^{23}$ | $0.119^{27}$ | 24
31 |  PFANet [240] | Tensorflow | $0.815^{21}$ | $0.846^{35}$ | $0.101^{8}$ | 22
32 |  PoolNet [243] | Pytorch | $0.829^{13}$ | $0.868^{18}$ | $0.106^{16}$ | 14
33 |  SCRNet [44] | Pytorch | $0.833^{11}$ | $0.872^{15}$ | $0.105^{14}$ | 13
34 |  CPDVgg [242] | Pytorch | $\textbf{{\color[rgb]{0,1,0}0.856}}^{3}$ | $0.889^{6}$ | $\textbf{{\color[rgb]{0,0,1}0.079}}^{2}$ | 2
35 |  EGNet [250] | Pytorch | $\textbf{{\color[rgb]{1,0,0}0.858}}^{1}$ | $\textbf{{\color[rgb]{0,0,1}0.896}}^{2}$ | $\textbf{{\color[rgb]{1,0,0}0.078}}^{1}$ | 1
2020 | 36 |  ABPNet [282] | Pytorch | $0.783^{38}$ | $0.810^{47}$ | $0.153^{45}$ | 44
37 |  U2Net [273] | Pytorch | $0.780^{39}$ | $0.795^{52}$ | $0.105^{14}$ | 35
38 |  GCPANet [277] | Pytorch | $0.807^{30}$ | $0.848^{33}$ | $0.133^{34}$ | 34
39 |  ITSD [280] | Pytorch | $0.798^{34}$ | $0.870^{17}$ | $0.142^{41}$ | 32
40 |  MINet [281] | Pytorch | $0.819^{17}$ | $0.864^{22}$ | $0.117^{25}$ | 22
41 |  SANet [56] | Pytorch | $0.812^{24}$ | $0.868^{18}$ | $0.106^{16}$ | 17
42 |  GateNetVgg [284] | Pytorch | $0.827^{15}$ | $0.865^{20}$ | $0.108^{18}$ | 15
43 |  F3Net [278] | Pytorch | $0.828^{14}$ | $0.891^{5}$ | $0.109^{19}$ | 12
44 |  CSNet [283] | Pytorch | $0.834^{10}$ | $0.876^{14}$ | $0.103^{10}$ | 11
45 |  LDF [279] | Pytorch | $0.835^{9}$ | $0.878^{12}$ | $0.103^{10}$ | 10
46 |  RASNet [258] | Pytorch | $0.832^{12}$ | $0.887^{8}$ | $0.103^{10}$ | 9
47 |  CAGVgg [271] | Keras | $0.837^{8}$ | $0.878^{12}$ | $0.088^{4}$ | 8
48 |  DFI [254] | Pytorch | $0.838^{7}$ | $\textbf{{\color[rgb]{1,0,0}0.903}}^{1}$ | $0.101^{8}$ | 5
49 |  R2Net [255] | Pytorch | $\textbf{{\color[rgb]{0,0,1}0.857}}^{2}$ | $0.885^{9}$ | $\textbf{{\color[rgb]{0,1,0}0.084}}^{3}$ | 4
2021 | 50 |  SCWS [288] | Pytorch | $0.811^{26}$ | $0.851^{31}$ | $0.115^{24}$ | 28
51 |  ICON [52] | Pytorch | $0.811^{26}$ | $\textbf{{\color[rgb]{0,0,1}0.896}}^{2}$ | $0.128^{31}$ | 19
52 |  BAS [31] | Pytorch | $0.842^{5}$ | $0.882^{11}$ | $0.092^{7}$ | 7
53 |  ABP [327] | Pytorch | $0.842^{5}$ | $0.889^{6}$ | $0.091^{6}$ | 6
54 |  CVAE [327] | Pytorch | $0.849^{4}$ | $\textbf{{\color[rgb]{0,1,0}0.892}}^{4}$ | $0.089^{5}$ | 3
\begin{overpic}[width=411.93767pt]{Top-52-min} \end{overpic} Figure 8:
Visualization results of deep learning models.
\begin{overpic}[width=411.93767pt]{TraditionalTop-50-min} \end{overpic} Figure
9: Qualitative results of state-of-the-art traditional approaches.

###  5.2 Quantitative Comparisons

To build a standardized leaderboard (_i.e._ , same image resolution,
thresholding step, and evaluation tool), we provide three golden metrics,
_i.e._ , $S_{\alpha}$, $E_{\xi}^{max}$, and ${M}$.

Table VII shows the performance of 46 SOTA traditional SOD algorithms on our
SOC_test set. In terms of both S-measure (_i.e._ , $S_{\alpha}$) and max
E-measure ($E_{\xi}^{max}$), the HCCH method surpasses all competitors by a
large margin. RBD, COV, and DRFI obtain comparable performance in terms of
$S_{\alpha}$ score. Meanwhile, COV ranks third in terms of $S_{\alpha}$
measure, but ninth in $E_{\xi}^{max}$. In terms of MAE (_i.e._ , ${M}$), the
top-5 approaches are: SF, COV, HCCH, SR, and MSSS. It is worth mentioning that
SF reduces ${M}$ and outperforms all the recent traditional SOD methods. Based
on their overall scores, the top-5 methods are HCCH, RBD, COV, DRFI, and WSC.

The quantitative results of the 54 deep learning SOD models on our SOC_test
dataset are shown in Table VIII. In terms of $S_{\alpha}$, EGNet, R2Net, and
CPDVgg are the top-3 models, with scores of more than 0.85. Roughly 46%
(_i.e._ , 21/45) of model scores are between 0.650 and 0.800. Compared with
the traditional model, which achieves an $S_{\alpha}$ score of 0.736, we can
see continuous improvement over the past few years, with the exception of four
early models (_i.e._ , DISC, DSL, LEGS, and UCF). At the same time, 30 out of
45 models achieve high performance (_e.g._ , 0.800$\leq S_{\alpha}\leq$0.850)
and the average performance is nearly 0.820. Interestingly, in terms of
$E_{\xi}^{max}$, the multi-task learning framework DFI and integrity learning
model have the best and second-best scores of 0.903 and 0.896, respectively.
Consistent with S-measure, in terms of MAE, we obtain the same top-3 models
EGNet, CPDVgg, and R2Net. From our 54 benchmarked models, we find that models
that perform well in terms of S-measure also do well in MAE. Overall, the
top-10 approaches are EGNet, CPDVgg, CVAE, R2Net, DFI, ABP, BAS, CAGVgg,
RASNet, and LDF. In the following section ($\S$ 6), we will provide a more
detailed analysis of these models.

###  5.3 Qualitative Comparisons

Two qualitative comparisons are presented in Fig. 8 and Fig. 9. As can be seen
from Fig. 8, deep models generate saliency maps that are similar to the GTs,
to varying degrees. Specifically, for ASNet, C2SNet, BMPM, DCL, DHS, DSS, DS,
DISC, SVF, RFCN, and PFANet, the position of the object can be well-
identified. However, all these methods generate blurred responses on object
boundaries. PFANet, MDF, MC and LEGS even nearly destroy the integrity of the
object. To better highlight these results, we introduce yellow rectangles to
mark the high-quality segmentation regions and utilize red arrows to point out
the errors. We observe that eight models (ABPNet, AFNet, AMU, NLDF, RAS, SCWS,
UCF, and WSS) can localize the human object but introduce additional noise. We
also notice that CAGNet, CSNet, MINet, DGRL, EGNet, F3Net, ICON, PoolNet, and
R3Net can even capture the small structure of the human elbow. Moreover,
saliency maps from R2Net, Deepside, SIBA, and MSRNet demonstrate better
results than the above-mentioned methods. Amazingly, BAS, U2Net, ABP, CPD,
GateNet, GCPANet, ITSD, LDF, SCRN, and CAVE perform very close to the GT and
result in knife-edge-shaped boundaries in the yellow rectangle region without
any additional noise.

In sharp contrast to the deep learning models, the traditional models (Fig. 9)
all fail without exception. WSC, HCCH, and RBD are the three most promising
approaches. However, their results are still far from the GT map, since they
are mainly based on various prior features extracted from color, orientation,
contrast, _etc_. Further, the center bias prior is not suitable in this case,
since the human is located close to the image boundary, thus making this
example more challenging for these approaches.

Table IX:  Comparison of 14 state-of-the-art approaches in terms of attribute-level performance. For deep learning models, we re-train them on our SOC-Sal_train set (_i.e._ , 1,800 images). Please refer to Tables II, III, & IV for more details. These results are available at: Google Drive.  |  | AC | BO | CL | HO | MB | OC | OV | SC | SO | Avg.
---|---|---|---|---|---|---|---|---|---|---|---
|  Model Attribute | $S_{\alpha}\uparrow$ | ${M}\downarrow$ | $S_{\alpha}\uparrow$ | ${M}\downarrow$ | $S_{\alpha}\uparrow$ | ${M}\downarrow$ | $S_{\alpha}\uparrow$ | ${M}\downarrow$ | $S_{\alpha}\uparrow$ | ${M}\downarrow$ | $S_{\alpha}\uparrow$ | ${M}\downarrow$ | $S_{\alpha}\uparrow$ | ${M}\downarrow$ | $S_{\alpha}\uparrow$ | ${M}\downarrow$ | $S_{\alpha}\uparrow$ | ${M}\downarrow$ | $S_{\alpha}\uparrow$ | ${M}\downarrow$
Traditional |  COV [108] | 0.505 | 0.216 | 0.277 | 0.577 | 0.453 | 0.280 | 0.508 | 0.229 | 0.494 | 0.219 | 0.484 | 0.246 | 0.423 | 0.314 | 0.535 | 0.174 | 0.525 | 0.172 | 0.467 | 0.270
WSC [161] | 0.541 | 0.205 | 0.356 | 0.517 | 0.517 | 0.252 | 0.556 | 0.211 | 0.536 | 0.210 | 0.529 | 0.227 | 0.475 | 0.292 | 0.567 | 0.170 | 0.535 | 0.181 | 0.512 | 0.252
HCCH [148] | 0.585 | 0.199 | 0.354 | .0.525 | 0.537 | 0.254 | 0.615 | 0.197 | 0.547 | 0.202 | 0.552 | 0.225 | 0.468 | 0.298 | 0.595 | 0.165 | 0.588 | 0.162 | 0.538 | 0.247
DRFI [114] | 0.598 | 0.229 | 0.391 | 0.513 | 0.570 | 0.274 | 0.618 | 0.230 | 0.556 | 0.230 | 0.577 | 0.248 | 0.527 | 0.304 | 0.614 | 0.188 | 0.585 | 0.197 | 0.560 | 0.268
RBD [115] | 0.589 | 0.225 | 0.429 | 0.481 | 0.575 | 0.260 | 0.625 | 0.216 | 0.557 | 0.213 | 0.583 | 0.235 | 0.521 | 0.295 | 0.602 | 0.191 | 0.579 | 0.192 | 0.562 | 0.256
Deep Learning |  ABP [327] | 0.767 | 0.092 | 0.592 | 0.315 | 0.742 | 0.125 | 0.787 | 0.101 | 0.742 | 0.095 | 0.740 | 0.112 | 0.746 | 0.132 | 0.759 | 0.083 | 0.741 | 0.080 | 0.735 | 0.126
EGNet [250] | 0.791 | 0.088 | 0.593 | 0.307 | 0.739 | 0.137 | 0.788 | 0.110 | 0.763 | 0.115 | 0.743 | 0.120 | 0.750 | 0.138 | 0.800 | 0.076 | 0.753 | 0.088 | 0.747 | 0.131
CPDVgg [242] | 0.806 | 0.076 | 0.626 | 0.278 | 0.765 | 0.118 | 0.808 | 0.096 | 0.786 | 0.097 | 0.765 | 0.103 | 0.760 | 0.127 | 0.801 | 0.070 | 0.765 | 0.076 | 0.765 | 0.116
CAGVgg [271] | 0.795 | 0.080 | 0.700 | 0.208 | 0.782 | 0.115 | 0.808 | 0.098 | 0.764 | 0.102 | 0.751 | 0.120 | 0.763 | 0.127 | 0.795 | 0.081 | 0.744 | 0.093 | 0.767 | 0.114
RASNet [258] | 0.821 | 0.066 | 0.626 | 0.276 | 0.785 | 0.106 | 0.816 | 0.087 | 0.788 | 0.086 | 0.776 | 0.096 | 0.779 | 0.113 | 0.810 | 0.066 | 0.774 | 0.070 | 0.772 | 0.107
CVAE [327] | 0.813 | 0.075 | 0.688 | 0.217 | 0.790 | 0.107 | 0.816 | 0.092 | 0.784 | 0.091 | 0.771 | 0.104 | 0.776 | 0.115 | 0.820 | 0.069 | 0.767 | 0.080 | 0.781 | 0.106
LDF [279] | 0.819 | 0.071 | 0.697 | 0.212 | 0.796 | 0.105 | 0.824 | 0.088 | 0.792 | 0.085 | 0.781 | 0.098 | 0.790 | 0.107 | 0.780 | 0.073 | 0.801 | 0.072 | 0.787 | 0.101
R2Net [255] | 0.827 | 0.071 | 0.656 | 0.257 | 0.802 | 0.107 | 0.826 | 0.092 | 0.794 | 0.097 | 0.789 | 0.099 | 0.791 | 0.112 | 0.807 | 0.072 | 0.788 | 0.073 | 0.787 | 0.109
BAS [31] | 0.831 | 0.060 | 0.723 | 0.166 | 0.785 | 0.110 | 0.814 | 0.093 | 0.797 | 0.072 | 0.780 | 0.101 | 0.781 | 0.114 | 0.820 | 0.072 | 0.787 | 0.075 | 0.791 | 0.096
Avg. | 0.721 | 0.125 | 0.551 | 0.346 | 0.688 | 0.168 | 0.729 | 0.139 | 0.693 | 0.137 | 0.687 | 0.152 | 0.668 | 0.185 | 0.722 | 0.111 | 0.693 | 0.115 | - | -

##  6 Further Benchmarking

###  6.1 Attribute-Based Evaluation

Based on the top-ranked models presented in Tables VII & VIII, we further re-
train the top-10666 The authors of DFI have only released the test code, so we
only evaluate nine models. deep learning models (using their default settings)
on the SOC-Sal_train set (1,800 images) and then test them on the SOC-Sal_test
set for attribute-based evaluation. In Table IX, we show the performance on
subsets of our dataset characterized by a particular attribute. Due to space
limitations, in the following discussion, we only select a few representative
attributes for further analysis.

Big object (BO) scenes typically occur when objects are close to the camera,
enabling tiny text and patterns to be seen clearly. In this case, models that
prefer to focus on local information are seriously misled, leading to a
considerable decrease in performance (_e.g._ , 8.6% $S_{\alpha}$ reduction for
BAS, 8.7% reduction for CAGVgg, 11.4% reduction for LDF, and 40.7% reduction
for COV) compared with their average performance (Avg.). Among all attributes,
BOs are the most difficult for both traditional and deep learning models.

Small objects (SOs) are tricky for some SOD models. Four models (_i.e._ , BAS,
CVAE, CAGVgg, and RASNet) encounter performance degradation (_e.g._ , from
BAS-0.5% to RASNet-3.6%) because SOs are easily ignored during the
downsampling of CNNs. Other models instead have enhanced performance on SOs,
but significant reduction in performance on BOs.

Heterogeneous objects (HOs) commonly appear in natural scenes. The performance
of all models on HOs improves to some degree, fluctuating from 2.9% to 14.3%.
We suspect this is because, as shown in Fig. 6, HO images make up a
significant proportion of all datasets, so the models are more familiar with
this attribute.

Occlusion (OC) scenes occur when objects are partly obscured. Thus, SOD models
must capture global semantics to make up for the incomplete information of
objects. As observed, traditional models obtain improved performance compared
with their average performance. For deep learning models, in contrast, this
situation is reversed.

As can be seen in the last row of Table IX (average performance of each
attribute), MB and SO have the same $S_{\alpha}$ score. Moreover, the average
scores of AC and SC are very similar. It seems that existing deep learning
based SOD models can effectively address appearance change and shape
complexity. Similar to the attributes of OV and OC, CL and MB remain
challenging for existing methods, generating mid-level (_i.e._ ,
0.65$<S_{\alpha}<$0.70) S-measure scores.

Table X: The contribution of our dataset-enhancement strategies.  Method Metric | $S_{\alpha}\uparrow$ | $E_{\xi}^{max}\uparrow$ | ${M}\downarrow$
---|---|---|---
RASNet [258] | 0.832 | 0.887 | 0.103
Our-RASNet | 0.841 | 0.897 | 0.096
LDF [279] | 0.835 | 0.878 | 0.103
Our-LDF | 0.845 | 0.891 | 0.097
BAS [31] | 0.842 | 0.882 | 0.092
Our-BAS | 0.856 | 0.895 | 0.086
R2Net [255] | 0.857 | 0.885 | 0.084
Our-R2Net | 0.868 | 0.899 | 0.080
CVAE [327] | 0.849 | 0.892 | 0.089
Our-CVAE | 0.863 | 0.902 | 0.086
Table XI: The contribution of each dataset-enhancement strategy.  Method Metric | $S_{\alpha}\uparrow$ | $E_{\xi}^{max}\uparrow$ | ${M}\downarrow$
---|---|---|---
CVAE [327] | 0.849 | 0.892 | 0.089
LS | 0.851 | 0.895 | 0.088
SS | 0.852 | 0.894 | 0.088
RDA | 0.855 | 0.896 | 0.086
Our-CVAE | 0.863 | 0.902 | 0.086
Table XII: Results for cross-dataset generalization in $\S$ 6. UC-Net (CVPR’20) [328] is trained on one dataset and tested on all others. “Sel.”: diagonal score (training and testing on the same dataset). “Oth.”: mean score on all except for self.  Measure |  $S_{\alpha}\uparrow$ [3] |  Drop$\downarrow$
---|---|---
Train Test | SOC | M10K | DU-O | DUTS | ECC | HKU | ILSO | Sel. | Oth. |
SOC [1] | .884 | .768 | .686 | .834 | .749 | .774 | .841 | .884 | .775 | 12%
M10K [35] | .800 | .921 | .784 | .894 | .881 | .882 | .884 | .921 | .854 | 7%
DU-O [38] | .833 | .898 | .854 | .877 | .862 | .867 | .886 | .854 | .871 | -2%
DUTS [41] | .795 | .882 | .793 | .910 | .890 | .903 | .900 | .910 | .861 | 5%
ECC [37] | .791 | .886 | .800 | .901 | .901 | .898 | .903 | .901 | .863 | 4%
HKU [40] | .818 | .892 | .787 | .904 | .883 | .910 | .905 | .910 | .865 | 5%
ILSO [61] | .841 | .888 | .790 | .898 | .882 | .896 | .920 | .920 | .866 | 6%
Oth. | .813 | .869 | .773 | .885 | .858 | .870 | .887 |  |  |

###  6.2 Comparison with Baselines

We introduce three dataset-enhancement strategies to prevent networks from
being overconfident as a result of dataset bias. These include label
smoothing, random data augmentation and self-supervised learning. We argue
that our strategies can be easily used in existing salient object detection
frameworks as general data processing techniques. We thus introduce our
strategies to five benchmark salient object detection models and show the
performance in Table X, where “Our-” represents the benchmark models with our
dataset-enhancement strategies. Further, we investigate the contribution of
each data-enhancement strategy, and show the performance in Table XI, where we
choose CVAE [327] as the base model.

Training & Testing Protocols. We retrain the five models in Table X with their
corresponding training dataset, _e.g._ , MB [32] for RASNet [258], and DUTS
[41] for all the other four models. We follow their original training and
testing settings, _e.g._ , same maximum epoch, learning rate, training and
testing image sizes.

Discussion. Table X shows consistent better performance of models with our
strategies, which illustrates effectiveness of our solutions. Further, in
Table XI, “LS”, “RDA”, “SS” represent adding label smoothing strategy, random
data augmentation and self-supervised learning to the base model respectively.
It shows that the random data augmentation achieves the largest performance
gain, while label smoothing and self-supervised learning achieves comparable
performance improvement. The main reason is that data augmentation introduce
diverse samples to the initial training dataset, which is effective in
improving model generalization ability. For the self-supervised learning
strategy, as the CVAE model [327] has already adopt the multi-scale image as
input strategy, we observe slightly improved performance. However, the better
performance in general can still validate the effectiveness of the proposed
strategy. Label smoothing [292] was introduced to prevent model from over-
confidence, thus achieve well-calibrated model. However, there exists no
saliency metrics to explain the calibration error of the saliency models. We
will investigate in expected calibration error [329] and extend it to saliency
detection task in the future to better explain the calibration error of
saliency models.

###  6.3 Cross-Dataset Generalization

To study the difficulty of existing SOD datasets, we adopt the CDA (cross-data
analysis) method [43]. Given $N$ candidate datasets $\\{D^{n}\\}_{n=1}^{N}$,
we first train a model on the $D_{i}$ dataset, and then test it on the other
datasets (_i.e._ , $\\{D^{n}\\}_{n=1,n\neq i}^{N}$). Following [330, 76], we
randomly select 800 images and 200 images from each dataset as the training
set and testing set, respectively.

We train UC-Net [328] on popular existing datasets that contain more than
1,000 images. Table XII shows the $S_{\alpha}$ score on each dataset. Each
column provides the score of UC-Net tested on a specific dataset and trained
on all others. Each row indicates the performance of UC-Net trained on one
dataset and tested on all others, demonstrating the generalizability of the
dataset adopted for training. We find that our SOC (_e.g._ , Oth. = 0.813) and
DU-O (Oth. = 0.773) datasets are more difficult than other datasets. In
addition, we notice that the model trained on our SOC dataset cannot perform
well (_e.g._ , the Oth. column: 0.775) on other datasets. This is because our
SOC dataset contains many real-world salient objects in open environments,
thus the model cannot overfit the simple or clean scenes in existing datasets.
This is also supported by our core observation that data selection bias is
common in existing datasets.

##  7 Future Directions

Human attention can be influenced by four key factors:

  * •

Visual properties. Our attention may be drawn by basic objects’ unique visual
properties [331].

  * •

Memory. If one knows an object well, it is easier for that object to attract
one’s attention.

  * •

Goal. For example, eye fixation records are quite different from attention
maps, with a specific goal for viewers.

  * •

Emotion. In addition to the above-mentioned factors, we argue that human
attention toward the same scene may be affected by one’s emotion, _e.g._ ,
happiness, sadness, anger.

As demonstrated by Cave [331], attentional control is determined by a
combination of these factors. Unfortunately, the annotations of existing SOD
datasets do not clearly describe which factor they address. Differently, the
ground-truth annotations of our SOC are based on the salicon (free-view task)
dataset777http://salicon.net/, or so-call meaning maps which are used in
recent studies [331, 332, 333]. As concluded by Kalash et al.  [66], the work
to date has addressed a relatively ill-posed problem. Thus, we recommend
several future directions to re-think SOD tasks at six main research levels:

(1) Data Level: Recently, visual saliency detection tasks have attracted
significant interest using 2D (RGB SOD) and 3D (_i.e._ , RGB-D, RGB-T) input
data. However, light field SOD (4D), LIDAR SOD, and 360∘ SOD are still not
well-studied. Establishing new datasets for these types of data will largely
promote the development of this field. Another interesting avenue for
examining saliency detection is to study fine-grained tasks, such as salient
instance detection [61, 62, 334, 335] and part-object visual saliency
detection [336].

(2) Task Level: Multi-task learning has demonstrated strong performance in
recent works [337]. Existing schemes mainly focus on vision tasks, such as
joint salient object detection and camouflaged object detection [338],
detection of salient objects, edges and skeletons simultaneously [254], and
simultaneous detection, ranking, and subitizing of multiple salient objects
[64]. With the success of the transformer technique in natural language
processing (NLP), introducing multi-modality learning into the saliency
detection field may be a feasible way to further incorporate other types of
information, such as CV+NLP (similar to [339]), CV+Audio [340], and CV+other
modality.

(2) Model Level: A huge number of algorithms have been developed to improve
detection accuracy. However, there are several promising directions that could
be further studied such as data augmentation techniques [341], efficient SOD
models (_e.g._ , lightweight models [283, 342]), new loss functions [286,
343], ranking-based models [137, 64], and transformer-based models [344, 345].

(4) Supervision Level: In addition to the most common fully supervised
learning of current SOD models, other supervision strategies, _e.g._ , weakly
supervised (_i.e._ , scribble [56], category [346], and polygon), semi-
supervised [53], self-supervised [60, 347], and unsupervised [58] learning are
also interesting to study.

(5) Evaluation Level: Evaluation metrics are important for model training,
testing, and benchmarking. However, the SOD community still utilizes classical
metrics such as IoU, F-measure, and MAE. These metrics were designed for
universal evaluation rather than for assessing SOD tasks specifically. As a
consequence, they do not work well for certain specific applications, such as
those with high-quality requirements. We envision that introducing a new
metric (_e.g._ , based on the gradient or connectivity error used in [348])
for SOD tasks, such as weighted F-measure [2] and S-measure [3], will be
another important research direction.

(6) Application Level: The SOD task belongs to a more general task called
class-agnostic object detection (CAOD). For simple scenes (_e.g._ , those
containing only one or two clear objects), SOD is identical to CAOD. From this
point of view, SOD models have many potential applications in the real-world
(_e.g._ , Alibaba’s fashion search system [339]), despite their currently
limited number of representative cases [30, 31, 200].

##  8 Conclusion

In this survey, we identified and addressed the long-ignored data selection
bias issue in SOD. Different from previous studies, we aimed to explore the
SOD task in the wild. To achieve this goal, we collected a new challenging and
densely annotated SOC dataset; analyzed a large number ($\sim$200) of
representative models; conducted the most complete (top-100) benchmarking;
devised a series of simple learning strategies to efficiently utilize negative
samples and training data; and identified several current challenges and
future directions. We hope that these contributions will provide the SOD
community an opportunity to explore novel techniques in an open environment.
We have tried to cover the most important works. Nevertheless, it is
impractical to thoroughly investigate all models in this vast field. We will
continue to incorporate new techniques on our website.

## References

  * [1] D.-P. Fan, M.-M. Cheng, J.-J. Liu, S.-H. Gao, Q. Hou, and A. Borji, “Salient objects in clutter: Bringing salient object detection to the foreground,” in _Eur. Conf. Comput. Vis._ , 2018, pp. 186–202.
  * [2] R. Margolin, L. Zelnik-Manor, and A. Tal, “How to evaluate foreground maps?” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2014, pp. 248–255.
  * [3] D.-P. Fan, M.-M. Cheng, Y. Liu, T. Li, and A. Borji, “Structure-measure: A New Way to Evaluate Foreground Maps,” in _Int. Conf. Comput. Vis._ , 2017, pp. 4548–4557.
  * [4] D.-P. Fan, C. Gong, Y. Cao, B. Ren, M.-M. Cheng, and A. Borji, “Enhanced-alignment Measure for Binary Foreground Map Evaluation,” in _Int. Joint Conf. Artif. Intell._ , 2018, pp. 698–704.
  * [5] P. Zhang, T. Zhuo, W. Huang, K. Chen, and M. Kankanhalli, “Online object tracking based on cnn with spatial-temporal saliency guided sampling,” _Neurocomputing_ , vol. 257, pp. 115–127, 2017.
  * [6] A. Borji, S. Frintrop, D. N. Sihite, and L. Itti, “Adaptive object tracking by learning background context,” in _IEEE Conf. Comput. Vis. Pattern Recog. Worksh._ , 2012, pp. 23–30.
  * [7] V. Mahadevan and N. Vasconcelos, “On the connections between saliency and tracking,” in _Adv. Neural Inform. Process. Syst._ , 2012, pp. 1664–1672.
  * [8] A. Abdulmunem, Y.-K. Lai, and X. Sun, “Saliency guided local and global descriptors for effective action recognition,” _Comput. Vis. Media_ , vol. 2, no. 1, pp. 97–106, 2016.
  * [9] J. He, J. Feng, X. Liu, T. Cheng, T.-H. Lin, H. Chung, and S.-F. Chang, “Mobile product search with bag of hash bits and boundary reranking,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2012, pp. 3005–3012.
  * [10] G. Liu and D. Fan, “A model of visual attention for natural image retrieval,” in _IEEE Int. Conf. Inf. Sci. Cloud Comput. Companion_ , 2013, pp. 728–733.
  * [11] J.-Y. Zhu, J. Wu, Y. Xu, E. Chang, and Z. Tu, “Unsupervised object class discovery via saliency-guided multiple class learning,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 37, no. 4, pp. 862–875, 2015.
  * [12] H. Liu, L. Zhang, and H. Huang, “Web-image driven best views of 3d shapes,” _The Vis. Comput._ , vol. 28, no. 3, pp. 279–287, 2012.
  * [13] K. Gu, G. Zhai, X. Yang, W. Zhang, and C. W. Chen, “Automatic contrast enhancement technology with saliency preservation,” _IEEE Trans. Circ. Syst. Video Technol._ , vol. 25, no. 9, pp. 1480–1494, 2015.
  * [14] R. Zhao, W. Ouyang, and X. Wang, “Unsupervised salience learning for person re-identification,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013, pp. 3586–3593.
  * [15] M. Donoser, M. Urschler, M. Hirzer, and H. Bischof, “Saliency driven total variation segmentation,” in _Int. Conf. Comput. Vis._ , 2009, pp. 817–824.
  * [16] S. J. Oh, R. Benenson, A. Khoreva, Z. Akata, M. Fritz, and B. Schiele, “Exploiting saliency for object segmentation from image level labels,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2017, pp. 4410–4419.
  * [17] D.-P. Fan, W. Wang, M.-M. Cheng, and J. Shen, “Shifting more attention to video salient object detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 8554–8564.
  * [18] T. Chen, M.-M. Cheng, P. Tan, A. Shamir, and S.-M. Hu, “Sketch2photo: Internet image montage,” _ACM Trans. Graph._ , vol. 28, no. 5, p. 124, 2009.
  * [19] M.-M. Cheng, F.-L. Zhang, N. J. Mitra, X. Huang, and S.-M. Hu, “Repfinder: finding approximately repeated scene elements for image editing,” _ACM Trans. Graph._ , vol. 29, no. 4, p. 83, 2010.
  * [20] V. Ramanishka, A. Das, J. Zhang, and K. Saenko, “Top-down visual saliency guided by captions,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , vol. 1, no. 2, 2017, p. 7.
  * [21] C. Guo and L. Zhang, “A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression,” _IEEE Trans. Image Process._ , vol. 19, no. 1, pp. 185–198, 2010.
  * [22] H. Hadizadeh and I. V. Bajic, “Saliency-aware video compression,” _IEEE Trans. Image Process._ , vol. 23, no. 1, pp. 19–33, 2014.
  * [23] Y. Liu, Z. Xu, W. Ye, Z. Zhang, S. Weng, C.-C. Chang, and H. Tang, “Image neural style transfer with preserving the salient regions,” _IEEE Access_ , vol. 7, pp. 40 027–40 037, 2019.
  * [24] M.-M. Cheng, X.-C. Liu, J. Wang, S.-P. Lu, Y.-K. Lai, and P. L. Rosin, “Structure-preserving neural style transfer,” _IEEE Trans. Image Process._ , vol. 29, pp. 909–920, 2020.
  * [25] A. Toshev, J. Shi, and K. Daniilidis, “Image matching via saliency region correspondences,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2007, pp. 1–8.
  * [26] M. J. Islam, R. Wang, K. de Langis, and J. Sattar, “Svam: Saliency-guided visual attention modeling by autonomous underwater robots,” _arXiv preprint arXiv:2011.06252_ , 2020.
  * [27] D.-P. Fan, G.-P. Ji, G. Sun, M.-M. Cheng, J. Shen, and L. Shao, “Camouflaged object detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 2777–2787.
  * [28] Y. Tu, L. Niu, W. Zhao, D. Cheng, and L. Zhang, “Image cropping with composition and saliency aware aesthetic score map,” in _AAAI Conf. Art. Intell._ , vol. 34, no. 07, 2020, pp. 12 104–12 111.
  * [29] J. Kim, T. Misu, Y.-T. Chen, A. Tawari, and J. Canny, “Grounding human-to-vehicle advice for self-driving vehicles,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 10 591–10 599.
  * [30] N. Kumar, P. N. Belhumeur, A. Biswas, D. W. Jacobs, W. J. Kress, I. C. Lopez, and J. V. Soares, “Leafsnap: A computer vision system for automatic plant species identification,” in _Eur. Conf. Comput. Vis._ , 2012, pp. 502–516.
  * [31] X. Qin, D.-P. Fan, C. Huang, C. Diagne, Z. Zhang, A. C. Sant’Anna, A. Suàrez, M. Jagersand, and L. Shao, “Boundary-aware segmentation network for mobile and web applications,” _arXiv preprint arXiv:2101.04704_ , 2021.
  * [32] T. Liu, J. Sun, N. Zheng, X. Tang, and H.-Y. Shum, “Learning to detect a salient object,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2007, pp. 1–8.
  * [33] S. Alpert, M. Galun, R. Basri, and A. Brandt, “Image segmentation by probabilistic bottom-up aggregation and cue integration,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2007, pp. 1–8.
  * [34] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in _Int. Conf. Comput. Vis._ , vol. 2, 2001, pp. 416–423.
  * [35] M.-M. Cheng, N. J. Mitra, X. Huang, P. H. S. Torr, and S.-M. Hu, “Global contrast based salient region detection,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 37, no. 3, pp. 569–582, 2015.
  * [36] A. Borji, D. N. Sihite, and L. Itti, “Salient object detection: a benchmark,” in _Eur. Conf. Comput. Vis._ , 2012, pp. 414–429.
  * [37] Q. Yan, L. Xu, J. Shi, and J. Jia, “Hierarchical saliency detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013, pp. 1155–1162.
  * [38] C. Yang, L. Zhang, H. Lu, X. Ruan, and M.-H. Yang, “Saliency detection via graph-based manifold ranking,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013, pp. 3166–3173.
  * [39] Y. Li, X. Hou, C. Koch, J. M. Rehg, and A. L. Yuille, “The secrets of salient object segmentation,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2014, pp. 280–287.
  * [40] G. Li and Y. Yu, “Visual saliency based on multiscale deep features,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2015, pp. 5455–5463.
  * [41] L. Wang, H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, and X. Ruan, “Learning to detect salient objects with image-level supervision,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2017, pp. 136–145.
  * [42] Z. Yang, S. Soltanian-Zadeh, and S. Farsiu, “Biconnet: An edge-preserved connectivity-based approach for salient object detection,” _arXiv preprint arXiv:2103.00334_ , 2021.
  * [43] A. Torralba and A. A. Efros, “Unbiased look at dataset bias,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2011, pp. 1521–1528.
  * [44] Z. Wu, L. Su, and Q. Huang, “Stacked cross refinement network for edge-aware salient object detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 7264–7273.
  * [45] L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual attention for rapid scene analysis,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 20, no. 11, pp. 1254–1259, 1998.
  * [46] A. Borji, “Saliency prediction in the deep learning era: Successes and limitations,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 43, no. 2, pp. 679–700, 2021.
  * [47] T. Liu, J. Sun, N. Zheng, X. Tang, and H. Shum, “Learning to detect a salient object,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2007, pp. 1–8.
  * [48] T. Liu, Z. Yuan, J. Sun, J. Wang, N. Zheng, X. Tang, and H.-Y. Shum, “Learning to detect a salient object,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 33, no. 2, pp. 353–367, 2010.
  * [49] P. Zhang, W. Liu, Y. Zeng, Y. Lei, and H. Lu, “Looking for the detail and context devils: High-resolution salient object detection,” _IEEE Trans. Image Process._ , 2021.
  * [50] Y. Zeng, P. Zhang, J. Zhang, Z. Lin, and H. Lu, “Towards high-resolution salient object detection,” in _Int. Conf. Comput. Vis._ , 2019, pp. 7234–7243.
  * [51] Q. Zhang, R. Cong, C. Li, M.-M. Cheng, Y. Fang, X. Cao, Y. Zhao, and S. Kwong, “Dense attention fluid network for salient object detection in optical remote sensing images,” _IEEE Trans. Image Process._ , 2020.
  * [52] M. Zhuge, D.-P. Fan, N. Liu, D. Zhang, D. Xu, and L. Shao, “Salient object detection via integrity learning,” _arXiv preprint arXiv:2101.07663_ , 2021\.
  * [53] Y. Zhou, S. Huo, W. Xiang, C. Hou, and S.-Y. Kung, “Semi-supervised salient object detection using a linear feedback control system model,” _IEEE Trans. Cybern._ , vol. 49, no. 4, pp. 1173–1185, 2019.
  * [54] G. Li, Y. Xie, and L. Lin, “Weakly supervised salient object detection using image labels,” in _AAAI Conf. Art. Intell._ , vol. 32, no. 1, 2018.
  * [55] L. Zhang, J. Zhang, Z. Lin, H. Lu, and Y. He, “Capsal: Leveraging captioning to boost semantics for salient object detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 6024–6033.
  * [56] J. Zhang, X. Yu, A. Li, P. Song, B. Liu, and Y. Dai, “Weakly-supervised salient object detection via scribble annotations,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 12 546–12 555.
  * [57] Y. Zeng, M. Feng, H. Lu, G. Yang, and A. Borji, “An unsupervised game-theoretic approach to saliency detection,” _IEEE Trans. Image Process._ , vol. 27, no. 9, pp. 4545–4554, 2018.
  * [58] J. Zhang, T. Zhang, Y. Dai, M. Harandi, and R. Hartley, “Deep unsupervised saliency detection: A multiple noisy labeling perspective,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2018, pp. 9029–9038.
  * [59] T. Nguyen, M. Dax, C. K. Mummadi, N. Ngo, T. H. P. Nguyen, Z. Lou, and T. Brox, “Deepusps: Deep robust unsupervised saliency prediction via self-supervision,” in _Adv. Neural Inform. Process. Syst._ , vol. 32, 2019, pp. 204–214.
  * [60] J. Wang, S. Zhu, J. Xu, and D. Cao, “The retrieval of the beautiful: Self-supervised salient object detection for beauty product retrieval,” in _ACM Int. Conf. Multimedia_ , 2019, pp. 2548–2552.
  * [61] G. Li, Y. Xie, L. Lin, and Y. Yu, “Instance-level salient object segmentation,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2017, pp. 2386–2395.
  * [62] R. Fan, M.-M. Cheng, Q. Hou, T.-J. Mu, J. Wang, and S.-M. Hu, “S4net: Single stage salient-instance segmentation,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 6103–6112.
  * [63] J. Zhang, S. Ma, M. Sameki, S. Sclaroff, M. Betke, Z. Lin, X. Shen, B. Price, and R. Mech, “Salient object subitizing,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2015, pp. 4045–4054.
  * [64] M. A. Islam, M. Kalash, and N. D. Bruce, “Revisiting salient object detection: Simultaneous detection, ranking, and subitizing of multiple salient objects,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2018, pp. 7142–7150.
  * [65] S. He, J. Jiao, X. Zhang, G. Han, and R. W. Lau, “Delving into salient object subitizing and detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2017, pp. 1059–1067.
  * [66] M. Kalash, M. A. Islam, and N. D. Bruce, “Relative saliency and ranking: Models, metrics, data and benchmarks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 43, no. 1, pp. 204–219, 2019.
  * [67] A. Siris, J. Jiao, G. K. Tam, X. Xie, and R. W. Lau, “Inferring attention shift ranks of objects for image saliency,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 12 133–12 143.
  * [68] A. Borji and L. Itti, “State-of-the-art in visual attention modeling,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 35, no. 1, pp. 185–207, 2012.
  * [69] A. Borji, M.-M. Cheng, H. Jiang, and J. Li, “Salient object detection: A benchmark,” _IEEE Trans. Image Process._ , vol. 24, no. 12, pp. 5706–5722, 2015.
  * [70] T. V. Nguyen, Q. Zhao, and S. Yan, “Attentive systems: A survey,” _Int. J. Comput. Vis._ , vol. 126, no. 1, pp. 86–110, 2018.
  * [71] A. Borji, M.-M. Cheng, Q. Hou, H. Jiang, and J. Li, “Salient object detection: A survey,” _Comput. Vis. Media_ , vol. 5, no. 2, pp. 117–150, 2019.
  * [72] D. Zhang, H. Fu, J. Han, A. Borji, and X. Li, “A review of co-saliency detection algorithms: Fundamentals, applications, and challenges,” _ACM Trans. Intell. Syst. Technol._ , vol. 9, no. 4, pp. 1–31, 2018.
  * [73] R. Cong, J. Lei, H. Fu, M.-M. Cheng, W. Lin, and Q. Huang, “Review of visual saliency detection with comprehensive information,” _IEEE Trans. Circ. Syst. Video Technol._ , vol. 29, no. 10, pp. 2941–2959, 2018.
  * [74] J. Han, D. Zhang, G. Cheng, N. Liu, and D. Xu, “Advanced deep-learning techniques for salient and category-specific object detection: a survey,” _IEEE Signal Process. Mag._ , vol. 35, no. 1, pp. 84–100, 2018.
  * [75] A. Borji, “Saliency prediction in the deep learning era: An empirical investigation,” _arXiv preprint arXiv:1810.03716_ , vol. 10, 2018.
  * [76] W. Wang, Q. Lai, H. Fu, J. Shen, H. Ling, and R. Yang, “Salient object detection in the deep learning era: An in-depth survey,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2021.
  * [77] D.-P. Fan, Z. Lin, Z. Zhang, M. Zhu, and M.-M. Cheng, “Rethinking rgb-d salient object detection: Models, data sets, and large-scale benchmarks,” _IEEE T. Neural Netw. Learn. Syst._ , 2021.
  * [78] T. Zhou, D.-P. Fan, M.-M. Cheng, J. Shen, and L. Shao, “Rgb-d salient object detection: A survey,” _Comput. Vis. Media_ , pp. 37–69, 2021.
  * [79] Y. Jiang, T. Zhou, G.-P. Ji, K. Fu, Q. Zhao, and D.-P. Fan, “Light field salient object detection: A review and benchmark,” _arXiv preprint arXiv:2010.04968_ , 2020.
  * [80] D.-P. Fan, T. Li, Z. Lin, G.-P. Ji, D. Zhang, M.-M. Cheng, H. Fu, and J. Shen, “Re-thinking co-salient object detection,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2021.
  * [81] J. Li, J. Su, C. Xia, and Y. Tian, “Distortion-adaptive salient object detection in 360∘ omnidirectional images,” _IEEE J. Sel. Top. Signal. Process._ , vol. 14, no. 1, pp. 38–48, 2019.
  * [82] R. Achanta, S. Hemami, F. Estrada, and S. Süsstrunk, “Frequency-tuned salient region detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2009, pp. 1597–1604.
  * [83] V. Movahedi and J. H. Elder, “Design and perceptual validation of performance measures for salient object segmentation,” in _IEEE Conf. Comput. Vis. Pattern Recog. Worksh._ , 2010, pp. 49–56.
  * [84] M.-M. Cheng, G.-X. Zhang, N. J. Mitra, X. Huang, and S.-M. Hu, “Global contrast based salient region detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2011, pp. 409–416.
  * [85] C. Xia, J. Li, X. Chen, A. Zheng, and Y. Zhang, “What is and what is not a salient object? learning salient object detector by ensembling linear exemplar regressors,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2017, pp. 4142–4150.
  * [86] H. Jiang, M.-M. Cheng, S.-J. Li, A. Borji, and J. Wang, “Joint salient object detection and existence prediction,” _Front. Comput. Sci._ , vol. 13, no. 4, pp. 778–788, 2019.
  * [87] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in _Eur. Conf. Comput. Vis._ , 2014, pp. 740–755.
  * [88] P. Krähenbühl and V. Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” in _Adv. Neural Inform. Process. Syst._ , 2011.
  * [89] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 23, no. 11, pp. 1222–1239, 2001.
  * [90] C. Rother, K. Vladimir, and B. Andrew, “Grabcut: interactive foreground extraction using iterated graph cuts,” _ACM Trans. Graph._ , vol. 23, no. 3, pp. 309–314, 2004.
  * [91] J. Shi and M. Jitendra, “Normalized cuts and image segmentation,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 22, no. 8, pp. 888–905, 2000.
  * [92] Y. Boykov and V. Kolmogorov, “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 26, no. 9, pp. 1124–1137, 2004.
  * [93] J. Harel, C. Koch, and P. Perona, “Graph-based visual saliency,” in _Adv. Neural Inform. Process. Syst._ , 2007, pp. 545–552.
  * [94] X. Hou and L. Zhang, “Saliency detection: A spectral residual approach,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2007, pp. 1–8.
  * [95] N. Bruce and J. Tsotsos, “Saliency based on information maximization,” in _Adv. Neural Inform. Process. Syst._ , 2006, pp. 155–162.
  * [96] L. Zhang, M. H. Tong, T. K. Marks, H. Shan, and G. W. Cottrell, “SUN: A Bayesian framework for saliency using natural statistics,” _J. Vis._ , vol. 8, no. 7, pp. 32–32, 2008.
  * [97] Y.-F. Ma and H.-J. Zhang, “Contrast-based image attention analysis by using fuzzy growing,” in _ACM Int. Conf. Multimedia_ , 2003, pp. 374–381.
  * [98] R. Achanta, F. Estrada, P. Wils, and S. Süsstrunk, “Salient region detection and segmentation,” in _Int. Conf. Comput. Vis. Syst._ , 2008, pp. 66–75.
  * [99] E. Rahtu, J. Kannala, M. Salo, and J. Heikkilä, “Segmenting salient objects from images and videos,” in _Eur. Conf. Comput. Vis._ , 2010, pp. 366–379.
  * [100] R. Achanta and S. Süsstrunk, “Saliency detection using maximum symmetric surround,” in _IEEE Int. Conf. Image Process._ , 2010, pp. 2653–2656.
  * [101] R. Valenti, N. Sebe, T. Gevers _et al._ , “Image saliency by isocentric curvedness and color.” in _Int. Conf. Comput. Vis._ , vol. 1, no. 3, 2009, p. 6.
  * [102] P. L. Rosin, “A simple method for detecting salient regions,” _Pattern Recogn._ , vol. 42, no. 11, pp. 2363–2371, 2009.
  * [103] F. Liu and M. Gleicher, “Region enhanced scale-invariant saliency detection,” in _Int. Conf. Multimedia and Expo_ , 2006, pp. 1477–1480.
  * [104] Y. Hu, D. Rajan, and L.-T. Chia, “Robust subspace analysis for detecting visual attention regions in images,” in _ACM Int. Conf. Multimedia_ , 2005, pp. 716–724.
  * [105] Z. Yu and H.-S. Wong, “A rule based technique for extraction of visual attention regions based on real-time clustering,” _IEEE Trans. Multimedia_ , vol. 9, no. 4, pp. 766–784, 2007.
  * [106] H. Yu, J. Li, Y. Tian, and T. Huang, “Automatic interesting object extraction from images using complementary saliency maps,” in _ACM Int. Conf. Multimedia_ , 2010, pp. 891–894.
  * [107] Y. Xie, H. Lu, and M.-H. Yang, “Bayesian saliency via low and mid level cues,” _IEEE Trans. Image Process._ , vol. 22, no. 5, pp. 1689–1698, 2012\.
  * [108] E. Erdem and A. Erdem, “Visual saliency estimation by nonlinearly integrating features using region covariances,” _J. Vis._ , vol. 13, no. 4, pp. 11–11, 2013.
  * [109] C. Yang, L. Zhang, and H. Lu, “Graph-regularized saliency detection with convex-hull-based center prior,” _IEEE Signal Process. Let._ , vol. 20, no. 7, pp. 637–640, 2013.
  * [110] N. Tong, H. Lu, L. Zhang, and X. Ruan, “Saliency detection with multi-scale superpixels,” _IEEE Signal Process. Let._ , vol. 21, no. 9, pp. 1035–1039, 2014.
  * [111] H. Peng, B. Li, R. Ji, W. Hu, W. Xiong, and C. Lang, “Salient object detection via low-rank and structured sparse matrix decomposition,” in _AAAI Conf. Art. Intell._ , 2013.
  * [112] J. Sun, H. Lu, and S. Li, “Saliency detection based on integration of boundary and soft-segmentation,” in _IEEE Int. Conf. Image Process._ , 2012, pp. 1085–1088.
  * [113] F. Perazzi, P. Krähenbühl, Y. Pritch, and A. Hornung, “Saliency filters: Contrast based filtering for salient region detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2012, pp. 733–740.
  * [114] H. Jiang, J. Wang, Z. Yuan, Y. Wu, N. Zheng, and S. Li, “Salient object detection: A discriminative regional feature integration approach,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013, pp. 2083–2090.
  * [115] W. Zhu, S. Liang, Y. Wei, and J. Sun, “Saliency optimization from robust background detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2014, pp. 2814–2821.
  * [116] X. Shen and Y. Wu, “A unified approach to salient object detection via low rank matrix recovery,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2012, pp. 853–860.
  * [117] R. Margolin, A. Tal, and L. Zelnik-Manor, “What makes a patch distinct?” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013, pp. 1139–1146.
  * [118] J. Kim, D. Han, Y.-W. Tai, and J. Kim, “Salient region detection via high-dimensional color transform,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2014, pp. 883–890.
  * [119] L. Mai, Y. Niu, and F. Liu, “Saliency aggregation: A data-driven approach,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013, pp. 1131–1138.
  * [120] C. Scharfenberger, A. Wong, K. Fergani, J. S. Zelek, and D. A. Clausi, “Statistical textural distinctiveness for salient region detection in natural images,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013, pp. 979–986.
  * [121] R. Liu, J. Cao, Z. Lin, and S. Shan, “Adaptive partial differential equation learning for visual saliency detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2014, pp. 3866–3873.
  * [122] Z. Jiang and L. S. Davis, “Submodular salient region detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013, pp. 2043–2050.
  * [123] K. Shi, K. Wang, J. Lu, and L. Lin, “PISA: Pixelwise image saliency by aggregating complementary appearance contrast measures with spatial priors,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2013, pp. 2115–2122.
  * [124] X. Li, H. Lu, L. Zhang, X. Ruan, and M.-H. Yang, “Saliency detection via dense and sparse reconstruction,” in _Int. Conf. Comput. Vis._ , 2013, pp. 2976–2983.
  * [125] B. Jiang, L. Zhang, H. Lu, C. Yang, and M.-H. Yang, “Saliency detection via absorbing markov chain,” in _Int. Conf. Comput. Vis._ , 2013, pp. 1665–1672.
  * [126] M.-M. Cheng, J. Warrell, W.-Y. Lin, S. Zheng, V. Vineet, and N. Crook, “Efficient salient region detection with soft image abstraction,” in _Int. Conf. Comput. Vis._ , 2013, pp. 1529–1536.
  * [127] K.-Y. Chang, T.-L. Liu, H.-T. Chen, and S.-H. Lai, “Fusing generic objectness and visual saliency for salient object detection,” in _Int. Conf. Comput. Vis._ , 2011, pp. 914–921.
  * [128] D. A. Klein and S. Frintrop, “Center-surround divergence of feature statistics for salient object detection,” in _Int. Conf. Comput. Vis._ , 2011, pp. 2214–2219.
  * [129] P. Jiang, H. Ling, J. Yu, and J. Peng, “Salient region detection by ufo: Uniqueness, focusness and objectness,” in _Int. Conf. Comput. Vis._ , 2013, pp. 1976–1983.
  * [130] X. Li, Y. Li, C. Shen, A. Dick, and A. Van Den Hengel, “Contextual hypergraph modeling for salient object detection,” in _Int. Conf. Comput. Vis._ , 2013, pp. 3328–3335.
  * [131] Y. Jia and M. Han, “Category-independent object-level saliency detection,” in _Int. Conf. Comput. Vis._ , 2013, pp. 1761–1768.
  * [132] Y. Lu, W. Zhang, H. Lu, and X. Xue, “Salient object detection using concavity context,” in _Int. Conf. Comput. Vis._ , 2011, pp. 233–240.
  * [133] Y. Wei, F. Wen, W. Zhu, and J. Sun, “Geodesic saliency using background priors,” in _Eur. Conf. Comput. Vis._ , 2012, pp. 29–42.