# DUT-LFSaliency: Versatile Dataset and Light Field-to-RGB Saliency Detection

Yongri Piao, Zhengkun Rong, Shuang Xu, Miao Zhang, and Huchuan Lu Y.R. Piao,
Z.K. Rong and H.C. Lu are with the School of Information and Communication
Engineering, Dalian University of Technology, ChinaM. Zhang and S. Xu are with
the International School of Information and Software Engineering, Dalian
University of Technology, China

###### Abstract

Light field data exhibit favorable characteristics conducive to saliency
detection. The success of learning-based light field saliency detection is
heavily dependent on how a comprehensive dataset can be constructed for higher
generalizability of models, how high dimensional light field data can be
effectively exploited, and how a flexible model can be designed to achieve
versatility for desktop computers and mobile devices. To answer these
questions, first we introduce a large-scale dataset to enable versatile
applications for RGB, RGB-D and light field saliency detection, containing 102
classes and 4204 samples. Second, we present an asymmetrical two-stream model
consisting of the Focal stream and RGB stream. The Focal stream is designed to
achieve higher performance on desktop computers and transfer focusness
knowledge to the RGB stream, relying on two tailor-made modules. The RGB
stream guarantees the flexibility and memory/computation efficiency on mobile
devices through three distillation schemes. Experiments demonstrate that our
Focal stream achieves state-of-the-arts performance. The RGB stream achieves
Top-2 F-measure on DUTLF-V2, which tremendously minimizes the model size by
83% and boosts FPS by 5 times, compared with the best performing method.
Furthermore, our proposed distillation schemes are applicable to RGB saliency
models, achieving impressive performance gains while ensuring flexibility.

###### Index Terms:

Saliency detection, light field, benchmark, knowledge distillation.

##  1 Introduction

Human attentional mechanism (HAM) allows us to focus on interesting regions
and filter out irrelevant ones. This cognitive ability helps us quickly
understand visual scenes out of an overwhelming amount of information. Over
the past decades, many works devote to imitating HAM. This task, namely
saliency detection, is essential for progress in image understanding and has
shown great potential in various computer vision and image processing tasks,
such as image segmentation [1], visual tracking [2, 3], object recognition [4,
5] and robot navigation [6].

The existing saliency detection methods can be roughly divided into three
categories based on the 2D (RGB), 3D (RGB-D) and 4D (light field) input
images. Different from 2D and 3D data, the light field provides multi-view
images of the scene through an array of lenslets and produces a stack of focal
slices, containing abundant spatial parallax information as well as depth
information. A stack of focal slices cater to human visual perception and are
observed in sequence with a combination of eye movements and shifts in visual
attention. Such abundant 4D data provides efficient saliency cues for saliency
detection in challenging scenes such as similar foreground and background,
small salient objects and complex background [7, 8, 9].

Light field not only brings vitality and vigor but also enriches mechanism
design, and meanwhile poses challenges to saliency detection. 1) The
deficiency of light field data in terms of scale, category and element types
limits the generalization of deep saliency models. 2) Light field methods are
both computation-intensive and memory-intensive as high dimensional data are
employed, e.g., the model size of the first deep-learning based light field
saliency detection network is more than 119 MB and FPS is only 2 on a single
1080Ti GPU card [9]. 3) Light field data are less ubiquitous as RGB data taken
by traditional digital cameras, taking advantage of light field data in a
user-friendly way for mobile devices is challenging.

Saliency detection as a pre-processing step for many tasks should be
efficient, versatile and user-friendly. Building on the above observation,
there are three key issues needed to be considered: how do we construct a
comprehensive dataset for achieving higher generalization accuracy of models;
how do we efficiently take advantage of high dimensional light field data; how
do we design the network to make it versatile enough to ensure superior
performance for higher requirements on desktop computers while boosting
flexibility and productivity for mobile devices.

TABLE I: Overview of Light Field Datasets with Various Applications Dataset | Year | Application | Capture | Type | Num | Class | Image
---|---|---|---|---|---|---|---
RGB | Multiview | Focal Stack | Depth
HCI-Old [10] | VMV-13 | Depth Estimation | Blender | Synthetic | 12 | - | $\surd$ | $\surd$ | $\times$ | $\surd$
LFSD [11] | CVPR-14 | Saliency Detection | Lytro I | Real-World | 100 | - | $\surd$ | $\times$ | $\surd$ | $\surd$
LIFFAD [12] | TIP-14 | Face Recognition | Lytro I | Real-World | 80 | - | $\surd$ | $\times$ | $\surd$ | $\surd$
Mobile [13] | CVPR-15 | Depth Estimation | Samsung S3 | Real-World | 13 | 6 | $\surd$ | $\times$ | $\surd$ | $\surd$
TOLF [14] | ICCV-15 | Object Semantation | Camera Array | Real-World | 18 | 7 | $\surd$ | $\surd$ | $\times$ | $\times$
HCI-New [15] | ACCV-16 | Depth Estimation | Blender | Synthetic | 24 | - | $\surd$ | $\surd$ | $\times$ | $\surd$
STFLytro [16] | Stanford-16 | Super-Resolution | Lytro II | Real-World | 300 | - | $\surd$ | $\surd$ | $\surd$ | $\surd$
EPFL [17] | QoMEX-16 | Super-Resolution | Lytro II | Real-World | 118 | 10 | $\surd$ | $\surd$ | $\times$ | $\times$
LFMD [18] | ECCV-16 | Material Recognition | Lytro II | Real-World | 1200 | 12 | $\surd$ | $\surd$ | $\times$ | $\times$
HFUT [19] | TOOM-17 | Saliency Detection | Lytro II | Real-World | 255 | - | $\surd$ | $\surd$ | $\surd$ | $\surd$
LFVSD [20] | ICCV-17 | View Synthesis | Lytro II | Real-World | 3300 | 7 | $\surd$ | $\surd$ | $\times$ | $\times$
LFVD [21] | CVPRW-17 | Depth Estimation | Camera Array | Real-World | 30 | - | $\surd$ | $\surd$ | $\times$ | $\surd$
INRIA [22] | TIP-18 | Light Field Inpainting | Lytro II | Real-World | 40 | - | $\surd$ | $\surd$ | $\times$ | $\surd$
LIFF [23] | CVPR-19 | Feature Detection | Lytro II | Real-World | 850 | 30 | $\surd$ | $\surd$ | $\times$ | $\surd$
SLFD [24] | TIP-19 | Depth Estimation | Blender | Synthetic | 53 | - | $\surd$ | $\surd$ | $\times$ | $\surd$
DLFD [24] | TIP-19 | Depth Estimation | Blender | Synthetic | 43 | - | $\surd$ | $\surd$ | $\times$ | $\surd$
LF-Blur [25] | SPL-19 | Light Field Debluring | Lytro II + | Real-World + | 400 | - | $\surd$ | $\surd$ | $\surd$ | $\times$
Unreal CV | Synthetic
DUTLF-V1 [9] | IJCAI-19 | Saliency Detection | Lytro II | Real-World | 1462 | - | $\surd$ | $\surd$ | $\times$ | $\times$
DUTLF-V2 | - | Saliency Detection | Lytro II | Real-World | 4204 | 102 | $\surd$ | $\surd$ | $\surd$ | $\surd$

Our work aims to shed light on these questions and make further step towards
light field saliency detection. First, we present the largest light field
dataset, extended from [9]. The DUTLF-V2 contains 4204 light field samples
divided into 102 subclasses. There are multiple advantages to this extended
DUTLF-V2: real-life scenarios, 1.2 times more samples than the previous
largest dataset LFVSD [20], 3.4 times more classes than the previous diversest
dataset LIFF [23], a variety of data elements (multiview, focal slices, depth
map and all-in-focus image) being engineered to enable versatile applications
of RGB, RGB-D, and light field saliency detection.

Second, we propose two modules to mimic the prediction mechanism in the brain
perceiving visual information, given two phrases—recruiting and screening that
the eyes process all information in our visual field (Recruiting is the act of
gathering visual resources. Screening is the act of using these resources to
select aspects of visual information). Correspondingly, the multi-focusness
recruiting module (MFRM) is designed to recruit rich saliency features from
every single focal slice for ensuring both effectiveness and diversity, while
the multi-focusness screening module (MFSM) is designed to screen useful
features by scanning salient objects at various locations and emphasizing the
most relevant ones. This designing mechanism ensures higher-performance demand
on desktop computers.

Third, we propose a novel learning strategy leveraging the concept of
knowledge distillation [26], where triple bridges are introduced to transfer
focusness knowledge from the teacher to the student. The proposed multi-
focusness distillation (MFD) encourages multi-focusness consistencies between
the teacher and the student; the attentive focusness distillation scheme (AFD)
mimics the attention map from the teacher; and screened focusness distillation
(SFD) learns complementarity between the screened focusness knowledge from the
teacher and appearance information from the student. The proposed distillation
schemes ensure better absorption and integration of focusness knowledge for
the student. It is noted that with the knowledge learned from the teacher, the
student no longer needs the focal slices but a single RGB image as input. The
lightweight and single-RGB-input student network guarantees flexibility and
productivity for mobile devices.

Furthermore, our distillation schemes can further facilitate other existing
RGB saliency detection approaches to achieve high efficiency while preserving
accuracy. This makes the proposed distillation schemes a universe tool,
generally suitable to a wide range of applications.

Last but not the least, extensive experiments on three light field datasets
demonstrate the effectiveness of the proposed framework. Our teacher network
achieves state-of-the-art results on three datasets, and student network
(VGG16) achieves Top-2 F-measure on DUTLF-V2. The student minimizes the model
size by 83% and boosts the Frame Per Second (FPS) by 5 times, compared with
the best performing method.

An earlier version of this work was published in [9]. The specific changes
implemented in this work are: 1) We expand the DUTLF-V1 [9] to a larger-scale,
higher-diversity and broader-coverage update, which can powerfully assist in
comprehensive scene understanding and higher generalization for RGB, RGB-D and
light field models, and further contributes to studies in the community of
saliency detection and other relative tasks. 2) In order to guide the student
network toward an effective fusion of the multi-focusness features, we propose
the attentive focusness distillation scheme that enforces the student network
to mimic the attention map form the Focal stream. This distillation scheme
further consistently improves the performance of our proposed method. 3) We
apply the proposed three distillation schemes to existing saliency models
under two settings, including no extra parameters setting and few extra
parameters setting. Extensive experiments confirm that the proposed two
application settings are both easy to operate, and can be universally applied
to existing saliency models while achieving significant performance gains over
the original models.

##  2 Related Work

###  2.1 Light Field Dataset

Light field [27, 28] contains richer visual information of the scene, and can
be recorded with a camera array [29], a sensor equipped with lenslet array
[30, 31] or a coded mask [32, 33]. It can produce ample image types that are
available for various visual tasks, such as a stack of focal slices containing
abundant spatial parallax information, depth maps with structural imformation
and multi-view images rich in geometric information. The high-dimensional
representation of light field information offers powerful capabilities for a
variety of applications as shown in Table I, including depth estimation [10,
13, 15, 21, 24], salient object detection [11, 19, 9], face recognition [12],
super-resolution [16, 17], material recognition [18], view synthesis [20],
feature detection [23], object segmentation [14], light field inpainting [22]
and light field debluring [25].

Over the past few years, several light field datasets have been introduced for
salient object detection. Li _et al._ [7], [11] pioneered early work for light
field dataset specially designed for salient object detection, containing 100
samples. Later on, Zhang _et al._ introduced _HFUT_ [19], a more challenging
dataset with 255 real-life scenarios at various conditions. Advances in deep
convolutional neural networks have enabled significant progress of a variety
of large-scale saliency detection models. Although above datasets encouraged
light field salient object detection to various degrees, the small size of the
datasets hardly enables the deep-learning based models that generalize
effectively. In addition, those datasets are constrained in terms of scene
coverage and salient object diversity, failed in well generalizing in real-
world applications. This left a gap in deep-learning based light field
saliency detection for which inadequate information limits reviewers to verify
their proposals. A larger-scale dataset _DUTLF-V1_ [9] remedied the above
limitations to a certain extent. It contains 1462 selected high-quality
samples with a variety of indoor and outdoor scenes. In our work, we take a
further step to extend _DUTLF-V1_ to 4204 light field samples with more
diverse salient objects and more various challenging scenes.

Overall, our _DUTLF-V2_ is clearly distinguishable from above datasets: 1) It
contains a large scale of real-life scenarios captured at various distances
without motion blur and uneven lighting. This can contibute to comprehensive
evaluation of saliency models and pave the way for future deep-learning based
light field studies. 2) Taking diverse salient objects, varied challenging
scenes (e.g., similar foreground and background, complex background and
multiple salient objects), random object position as well as various object
size altogether makes a solid and unique foundation for real-world light field
salient object detection. 3) It consists of richer image types (e.g.,
multiview, focal slices, depth map and all-in-focus image). This makes our
dataset applicable for the field of RGB, RGB-D and light field salient object
detection.

Figure 1: Image elements in the proposed dataset: (a) RGB image, (b) grount
truth, (c) depth map, (d) focal stack ($2^{nd}$ row) and corresponding focus
region ($3^{rd}$ row), (e) micro-image and (f) multiview.

###  2.2 Salient Object Detection

Existing saliency detection models can be generally categorized as 2D (RGB),
3D (RGB-D) and 4D (light field) based approaches. Early 2D methods [34, 35,
36] focus on exploiting low-level visual cues with hand-crafted features, such
as color, region contrast, etc. Benefiting from the development of
convolutional neural networks, numerous 2D methods based on CNNs are proposed.
Li _et al._ [38] developed a contour-to-saliency transferring method which can
convert a pre-trained contour detection model into a saliency detection model.
Liu _et al._ [40] produced detail enriched saliency maps by designing two
pooling-based modules which can progressively exploit high-level features. In
[41], Wu _et al._ came up with a cascaded partial decoder which can improve
both efficiency and accuracy of the existing multi-level feature aggregation
networks. Chen _et al._ [42] proposed a GCPANet to effectively integrate low-
level appearance features, high-level semantic features and global context
features. Pang _et al._ [43] introduced an aggregate interaction module and a
self-interaction module to generate efficient multi-level and multi-scale
features. Wu _et al._ [44] proposed a stacked cross refinement network which
can simultaneously refine multi-level edges and segmentation features. In
[74], Wei _et al._ proposed a cross feature module and a cascaded feedback
decoder to solve the existing big differences between features from different
receptive fields.

Despite the large advance in RGB saliency detection, many scientific
challenges remain for researchers to address. One of concerns is that the
appearance features in RGB data are less predictive regarding challenging
scenes, such as similar foreground and background, low contrast, complex
background and transparent object. To address this issue by exploiting
geometric and structural information, depth cues is introduced to RGB-D
saliency detection. Zhu _et al._ [45] introduced a two-branch network in which
the depth-based features generated from a sub-network are incorporated into a
master network for processing RGB images. Chen _et al._ [46] proposed a
complementarity-aware fusion module which can learn complementary information
from the paired modalities. In [47], Chen _et al._ utilized cross-modal
interactions to encourage complements across both high-level and low-level
features. Chen _et al._ [49] proposed a three-stream network with an
attention-aware mechanism which can adaptively select complementary features.
Zhang _et al._ [71] designed a novel top-down multi-level fusion structure for
effectively utilizing different fusion manners to combine the low-level and
high-level features. In [70], Chen _et al._ proposed a disentangled cross-
modal fusion network for exposing structural and content representations from
both modalities.

Light field data have been demonstrated in favor of saliency detection. Li _et
al._ [11] introduced the first light field saliency detection dataset and
incorporated focusness and objectness cues into saliency detection. Zhang _et
al._ [50] developed the background priors encoded by light field focusness to
enhance the saliency and reduce the background distraction. Li _et al._ [8]
designed a weighted sparse coding framework which can process the heterogenous
input data effectively. Zhang _et al._ [19] utilized multiple light field cues
included all-in-focus images, depth maps, focal slices and multi-view images
to exploit corresponding saliency. Piao _et al._ [9] introduced a light-field-
driven network to extract rich saliency representations and build the
relationship between salient objects and scene understanding. Zhang _et al._
[51] proposed a novel memory-oriented decoder which can comprehensively
exploit internal correlation of focal slices for accurate saliency prediction.
In [68], Zhang _et al._ proposed a light field refinement module and a light
field integration module to take full advantage of light field data. Due to
the unique property of light field, it has shown promising prospects in
saliency detection.

However, higher-dimensional light field data pose new challenges in light
field saliency detection, such as overcoming the memory limitation and
improving computational efficiency. To confront the above difficulties, in
this paper, we design an asymmetrical two-stream network in which teacher
network exploits focal slices for higher-performance demand on desktop
computers, while the lightweight and single-RGB-input student network aims to
achieve flexibility and productivity for mobile devices.

##  3 The DUTLF-V2 Dataset

In order to provide a generic benchmark for salient object detection, we
elaborately built a large-scale, high-diversity and broad-coverage dataset,
namely DUTLF-V2, considered as the updated version of DUTLF-V1. The DUTLF-V2
dataset consists of 4204 light field samples, increased by 187% from the
previous version in number. It covers diverse realistic scenes corresponding
to 10 main classes, which can be further divided into 102 sub-classes. Each
sample of DUTLF-V2 contains 6 elements, including raw light field data, a RGB
image, a saliency mask, a depth map, a stack of 12 focal slices and an array
of multi-view images. A variety of data elements are engineered to enable
versatile applications of DUTLF-V2 for RGB, RGB-D and light field saliency
detection. The complete dataset is available at https://github.com/OIPLab-
DUT/DUTLF-V2. Next, we will show details of DUTLF-V2 regarding the following 4
key aspects.

###  3.1 Dataset Construction

The construction of DUTLF-V2 includes the process of data collection, data
annotation and dataset split. Implementation details are described as follows.

Data Collection. We conducted our data collection with two main goals in mind:
1) to ensure sensory rich and logistically practical data collection, and 2)
to remedy the data deficiency problem while ensuring diversity of the
benchmark dataset. We opt for a commercially available Lytro Illum camera for
our data collection, being consistent with previous works [7], [9], [11],
[19]. Our dataset consists of a variety of indoor and outdoor scenes captured
in the surrounding environments, e.g., offices, supermarkets, campuses,
streets and so on. Moreover, our dataset is recorded over several months under
different time, lighting conditions and camera parameters (e.g., aperture size
and focal length that determine depth of field in an image). To assure the
quality of the data, three participants were employed full time to screen out
disqualified images, such as blurred images and images with disputable salient
objects. In this way, 4204 light field samples are included in DUTLF-V2.

Data Annotation. To ensure the accuracy and consistency for annotation, twenty
participants are instructed to label the salient objects from the all-in-focus
RGB image using a widely-used custom segmentation tool. To further improve
annotation accuracy, all participants are pre-trained with over ten examples.
To achieve a consensus, each three of participants jointly determined the
saliency objects, and cross validated by other three ones. We accept a
consensus of at least 83 percent to be considered a positive label. In the
annotation process, the participants were first asked to draw the coarse
boundary along the salient objects, then check the segmentation results and
refine the boundaries. To this end, we obtained corresponding 4204 accurate
pixel-wise ground-truth masks in total.

Dataset Split. To ensure evaluation consistency, the split protocol of our
dataset is similar with previous works [52], [53]. 4204 light field samples
are randomly generated into the training and testing sets at the ratio of 7:3.
Specifically, 2957 samples are for training and 1247 samples are for testing
with corresponding saliency masks. The proportion of hard samples in the
training set and test set is 57.25% and 60.22%, respectively. We note that
simple and hard samples are well balanced to split into the training and
testing sets.

Figure 2: Dataset statistics: (a) Salient objects categories, including 10
main classes and 102 sub-classes. (b) Scene categories, including complex
scenes and simple scenes, accounting for 58.14% and 41.86%, respectively. (c)
Mutual dependencies among scene categories in (b). (d) Ratio distribution
according to number of slices in each focal stack, the coordinate axis x and y
represent the number of focal slices and coresponding ratio, respectively. (e)
Ratio distribution according to salient object scales, the coordinate axis x
and y represent the salient object scales and coresponding ratio,
respectively. (f) Center distribution of the salient objects depicted in an
image coordinate system.

###  3.2 Dataset Elements

To achieve versatility of our dataset, we decode the light field raw data
using the Lytro Power Tools (LPT). Each light field consists of a RGB (all-in-
focus) image, a corresponding manually labeled ground truth, a depth image, a
stack of focal slices focusing at different depths and an array of multi-view
images. We will elaborate on each element of DUTLF-V2, illustrated in Figure
1.

Raw Light Field Data. Light field data are stored in raw files, which save the
data that come off the image sensor, namely Light Field Raw (LFR) files. The
raw light field data comprise an array of micro-images shown in Figure 1 (e),
each micro image being projected by a respective micro lens of a micro-lens
array. LFR files can be used as initial input for either the Lytro camera
software or any other processing toolbox. In our implementation, we use the
Lytro Power Tools (LPT) to decode the LFR files into RGB (all-in-focus)
images, depth maps, focal stacks and multi views.

RGB Image (All-in-Focus). An all-in-focus image has all parts of the sample in
focus. It can be generated using a Markov random field (MRF) based approach by
integrating the sharpest in-focus pixels across the focal stack. We obtained
4204 all-in-focus images with the resolution of $600\times 400$, which are
applicable for RGB saliency detection.

Saliency Mask. To obtain the pixel-wise ground truth, we manually labeled the
salient objects from the RGB image using a commonly used segmentation tool.
Three annotators are required to determine the salient objects in order to
achieve annotation consensus.

Depth Map. Each pixel in the depth map describes the distance from a viewpoint
to the surfaces of scene objects. Depth maps can be generated using the focal
stack. To highlight the relative position between objects, we normalize the
depth map to the range from 0 to 255. The depth maps together with RGB images,
enable versatile application of our dataset for RGB-D saliency detection.

Focal Stack. A focal stack represents a series of images focusing at different
depths. It caters to human visual perception and is observed in sequence with
a combination of eye movements and shifts in visual attention. In our proposed
dataset, the focal stack has different number varied from 3 to 13 (see Figure
2(d)). Most of the focal stacks contain more than 9 slices, which can
demonstrate the diversity of image depths. We ensure the number of focal
slices in each scene to be 12 by randomly copying for coding requirements.
This allows our dataset to be effectively applied in light field salieny
detection.

Multi View / Micro Image. The light field provides multi-view images of the
scene through an array of lenslets, namely multiview. The multiview is a set
of sub-view images, and each 2D sub-view image ’sees’ the scene from a
slightly different viewpoint, thus providing angular variation which is the
distinctive characteristic of light field imaging. In our DUTLF-V2, each
multiview contains 81 sub-view images, which have a 9$\times$9 angular
resolution and 600$\times$400 pixels of spatial resolution. The sufficient
multiview can facilitate the research of light field saliency detection,
super-resolution and view synthesis.

###  3.3 Dataset Statistics

Our motivation for DUTLF-V2 is to build a versatile dataset in large scale,
high diversity and broad coverage. We conduct descriptive statistics on our
DUTLF-V2 in terms of the following aspects.

Diversity of Salient Objects. High-quality datasets should reflect the
diversity in a broad range of coverage. Our DUTLF-V2 covers a variety of
indoor and outdoor scenes that enable a comprehensive understanding of the
visual systems. Our dataset can be categorized into 10 main classes such as
animal, vehicle, plant, public utilities, artifact, clothes, kitchenware,
furniture, supermarket and human, and 102 sub-classes, such as animal (e.g.,
cat, insect, sheep), vehicle (e.g., bicycle, car), plant (e.g., tree, flower),
public utilities (e.g., bench, road sign), etc., shown in Figure 2(a).
Besides, our dataset reflects the daily situation in a realistic way, further
taking practical application into consideration.

Figure 3: The whole pipeline. Our asymmetrical two-stream network consists of
the Focal stream and the RGB stream. The Focal stream, served as the teacher
network, contains a feature extractor, a Multi-Focusness Recruiting
Module(MFRM) and a Multi-Focusness Screening Module(MFSM). The lightweight
student network learns to replace focal slices with a single RGB image via
three tailored distillation schemes.

Challenging Scenes. A certain amount of hard samples in datasets may guarantee
generalization ability of deep models. With this in mind, we aim to build a
more challenging and representative dataset. Figure 2(b) represents a variety
of challenging scenes in our DUTLF-V2, such as similar foreground and
background, low illumination, small object, complex edge, occlusion, complex
background, multiple objects and transparent object. It can also be seen that
the complex scenes have a larger share in our dataset than the simple scenes.
More importantly, there are mutual dependencies among scene categories shown
in Figure 2(c). That means a scene may belong to multiple complex categories
(e.g., a scene consists of multiple salient objects under low illumination).
The above analysis demonstrates that the proposed DUTLF-V2 is challenging and
presents more challenges to the field of salient object detection.

Scale of Salient Objects. The size of salient objects refers to the proportion
of foreground object pixels over the whole image. Different scale of salient
objects is another factor to determine if the datasets are high qualified.
Figure 2(e) represents that the ratio distribution according to salient object
scales lies in a broad range of [0.001, 0.865] and most objects occupy less
than 35$\%$ area of the image. This attribute further contributes to higher
diversity of our dataset.

Distribution of Center bias. Location of salient objects presents another
challenge in that datasets simply composing centered salient objects may not
be sufficient to generically evaluate saliency models. Figure 2(f) shows the
location distribution for the center of salient objects across the whole
dataset. It can be seen that the salient objects in our DUTLF-V2 appear at a
variety of locations, which avoids the salient objects being easily extracted
by the center.

##  4 Method

To achieve the goal of developing an efficient, versatile and user-friendly
architecture for light field saliency detection, we introduce an asymmetrical
two-stream network based on knowledge distillation, as illustrated in Figure
3. On the one hand, the Focal stream, served as the teacher network, aims to
learn to exploit focal slices for higher-performance demand on desktop
computers. On the other hand, the student network learns to replace focal
slices with a single RGB image for flexibility and productivity on mobile
devices. The feature extractor in the teacher network is based on VGG19 [54],
while the student feature extractor is based on VGG16 [54]. We select the
high-level convolutional features ($F^{3}_{Conv}$, $F^{4}_{Conv}$ and
$F^{5}_{Conv}$) to detect salient objects. The detailed structure of the
feature extractor is shown in Figure 4. In this section, we will discuss the
reasoning Focal stream (Learning to Exploit Focal Stack), and RGB stream
(Learning to Replace Focal Stack) in detail.

###  4.1 Learning to Exploit Focal Stack

In order to enable our teacher network with more accurate prediction for
higher-performance demand on desktop computers, as well as efficient transfer
of rich focusness knowledge to the student, we propose two tailored modules in
the teacher network, which are multi-focusness recruiting module (MFRM) and
multi-focusness screening module (MFSM). The MFRM focuses on explicitly
recruiting saliency information from each focal slice, and the MFSM aims to
select useful features and suppress the unnecessary ones. Detailed discussion
about the effcet of MFRM and MFSM are provided in ablation studies. Next, we
will elaborate each component in the teacher network.

####  4.1.1 Multi-Focusness Recruiting Module (MFRM).

Inspired by the recruiting phase in the brain perceiving visual information,
we aim to gather rich saliency features by processing every single focal
slice. A straightforword solution to this issue is supervising the raw
features with ground truth for avoiding the distraction of non-salient objects
and ensuring the effectiveness of each focusness feature. However, this
strategy could reduce diversity and complementarity between multi-focusness
features. To this end, we design a multi-focusness recruiting module (MFRM)
which encourages the raw multi-focusness features containing adequate
effectiveness and diversity to achieve optimal results. The detail of MFRM is
shown in Figure 3. We first connect a convolutional layer to convert each
focusness feature from 64 channels to 2 channels. Then each focusness feature
is supervised by the following loss function:

$\begin{array}[]{l}L_{R}({f_{k}})=L_{CE}(f_{k},Y)-\lambda\sum\limits_{i=1,i\neq
k}^{N}{L_{MSE}(f_{k},f_{i})},\end{array}$ (1)

where $f_{k}$ represents the $k^{th}$ focusness feature, $Y$ represents the
ground truth and $N$ represents the total number of focal slices. $L_{CE}$ and
$L_{MSE}$ denote cross-entropy and mean squared error loss function,
respectively. The first item encourages effectiveness, the second item
enhances diversity, and the non-negative weight $\lambda$, which is set to 10,
expresses the trade-off between these two items.

Figure 4: Detailed structure of the feature extractor in the Focal stream or
RGB stream.

####  4.1.2 Multi-Focusness Screening Module (MFSM).

Inspired by the screening phase in the brain perceiving visual information,
which concerns with selectivity, we aim to efficiently select useful saliency
information from multi-focusness features. Therefore, by assigning different
weights to different focal slices regarding the salient objects, we propose a
multi-focusness screening module (MFSM) to resemble the screening phase how
human select information of interest from visual resources. Specifically, the
MFSM consists of a ConvLSTM model with an attention mechanism as shown in
Figure 3. The attention module is designed to emphasize the useful features
and suppress the unnecessary ones in order to produce screened features. The
ConvLSTM module aims to summarize the spatial information from the screened
features of historical steps and the current step for accurately identifying
the salient objects. As time step increases, the MFSM can highlight the
salient regions and block the non-salient ones gradually (see Figure 6).
Detailed operations are described as follows.

In each time step $t$, the multi-focusness features
$f=\\{f_{1},f_{2},\cdots,f_{N}\\}$ first go through a feature-wise attention
module and this procedure can be defined as:

$\leavevmode\resizebox{429.28616pt}{}{$\tilde{F}_{t}\\!=\\!\sum\limits_{i=1}^{N}{f\odot}(\Phi(AvgPool(Cat\left[{f_{1};f_{2};\cdots;f_{N}}\right]*W_{f}+H_{t-1}*W_{h})))$},$
(2)

where $\Phi$ represents softmax function. $\odot$ and $*$ mean feature-wise
multiplication and convolution operation, respectively. $H_{t-1}$ represents
the hidden state of the ConvLSTM cell in the $(t-1)^{th}$ step. $W_{f}$ and
$W_{h}$ are parameters of the convolutional kernels. Then the combined
features $\tilde{F}_{t}$ are fed into a ConvLSTM cell. The internal operations
in ConvLSTM are shown in Figure 3. After several time steps, we concatenate
the $O$ gates to summarize the saliency information from the screened
features, and make a final prediction. This procedure can be defined as:

$S_{tea}{\rm{=}}\Phi\left({W_{s}*Cat[O_{1};\cdots,O_{t};\cdots;O_{T}]}\right),$
(3)

where $S_{tea}$ represents the saliency map predicted from the teacher
network, $T$ denotes the total number of time steps and is set to 4. $W_{s}$
denotes the convolution parameter.

###  4.2 Learning to Replace Focal Stack

Most existing methods based on knowledge distillation take as same input for
teacher and student networks. Considering simplifying the heavy focal
computation and accessing focal slices more conveniently, we aim to design a
lightweight network in a user-friendly way by taking a single ubiquitous RGB
image as input to replace focal slices. However, directly transferring the
output from the teacher to the student overlooks the inherent differences
between two types of data. Therefore, we propose three tailored distillation
schemes to replace focal slices with a single RGB image by transferring the
focusness knowledge. In this way, the focusness knowledge is defined as three
parts: 1) The first part is designed to mimic multi-focusness features only
using a single RGB image. 2) The second part is designed to learn to
effectively integrate the multi-focusness features. 3) The third part is
designed to learn complementary information from appearance and screened
focusness knowledge. We also give an in-depth discussion about the effect of
the proposed three distillation schemes in ablation studies. Detailed
description for each distillation scheme is given below.

####  4.2.1 Multi-Focusness Distillation Scheme (MFD).

Unlike directly enforcing the student to mimic the output from the teacher,
the student is first trained to hallucinate multi-focusness features from the
Focal stream by our proposed multi-focusness distillation scheme. This is
mainly driven by the consideration of the inherent differences between input
data for the teacher and student. Moreover, multi-focusness features can be
produced from a single RGB input without explicit focal computation, whis
leads to significantly faster inference. In detail, we reduce the Kullback-
Leibler divergence loss between features of the penultimate layer in RGB
stream and features generated from the MFRM in Focal stream:

$L_{MFD}{\rm{=}}\frac{{\rm{1}}}{N}\sum\limits_{i=1}^{N}{KL\left({f_{i}^{s}\left\|{f_{i}^{t}}\right.}\right)},$
(4)

where $f_{i}^{t}$ represents the feature map of the $i^{th}$ focal slice
produced from the teacher network and $f_{i}^{s}$ represents the $i^{th}$
feature map produced from the student network.

####  4.2.2 Attentive Focusness Distillation Scheme (AFD).

After learning multi-focusness features from the Focal stream, a crisis
problem for the student network is how to effectively integrate the multi-
focusness features. Based on above consideration, we propose the attentive
focusness distillation scheme by which the student is capable to mimic the
attention map from the Focal stream. In this way, the useful multi-focusness
features are highlighted while the unnecessary ones are suppressed, further
leading to more accurate saliency detection results. Specifically, we train
the student network by minimizing the Kullback-Leibler divergence loss between
the attention map of the RGB stream and the MFSM in Focal stream. This loss
can be described as the following equation:

$L_{AFD}{\rm{=}}{KL\left({A_{stu}\left\|{A_{tea}}\right.}\right)},$ (5)

where $A_{tea}$ denotes the attention map of the RGB stream, and $A_{stu}$
denotes the attention map of the MFSM in Focal stream.

####  4.2.3 Screened Focusness Distillation Scheme (SFD).

Our third distillation scheme goes a further step: we align the class
probability of each pixel produced from the student network and the teacher
network, as well as the probability of each pixel between the output of the
student network and the ground truth. We refer this distillation scheme as
screened focusness distillation, which allows the student network to learn
complementary information from appearance and screened focusness information
for accurate prediction. To enhance this process with screened focusness and
appearance information, we train the student network by backpropagating a
linear combination of KL and cross entropy losses through the entire network.
The loss function is given as follows:

$L_{SFD}{\rm{=}}{KL\left({S_{stu}\left\|{S_{tea}}\right.}\right)}+\alpha
L_{CE}(S_{stu},Y),$ (6)

where $S_{tea}$ and $S_{stu}$ represent the saliency map predicted from the
teacher and student networks, respectively. The hyperparameter $\alpha$ is set
to 1.

###  4.3 Training Process

As presented in Algorithm 1, the teacher network is supervised by two losses:
the cross entropy loss $L_{CE}$ with the ground truth and the recruiting loss
$L_{R}{(f_{k})}$ in Eq.(1). During the knowledge distillation process, the
teacher is pre-trained and the parameters are kept frozen. The student is
supervised by a combination of the multi-focusness distillation loss $L_{MFD}$
in Eq.(4), the attentive focusness distillation loss $L_{AFD}$ in Eq.(5) and
the screened focusness distillation loss $L_{SFD}$ in Eq.(6). $W_{T}$ and
$W_{S}$ are parameters for the teacher and student, respectively.

1 Stage 1 : Training the teacher network.

2Inputs : Focal slices.

3
$W_{T}{\rm{=argmin}}_{W_{T}}\left({L_{CE}\left({S_{tea},Y}\right)+\sum\limits_{k=1}^{N}{L_{R}\left({f_{k}}\right)}}\right)$

4Stage 2 : Training the student network.

5Inputs : Single RGB.

$W_{S}{\rm{=argmin}}_{W_{S}}\left({L_{MFD}+L_{AFD}+L_{SFD}}\right)$

Algorithm 1 Training Process of Our Proposed Method

##  5 Experiments and Analyses

In this section, we first presented the adpoted datasets, evaluation metrics
and implementation details. Then we gave a comprehensive comparision of our
DUTLF-V2 with other RGB, RGB-D and light field benchmarks. Next we performed a
number of ablation experiments to analyze the effcet of each component in
teacher and student networks. We also comprehensively compared the performance
of our approach with other advanced saliency models. Besides, detailed studies
are provided about the application of our proposed distillation schemes under
two settings.

###  5.1 Experimental Setup

####  5.1.1 Benchmark Datasets.

We evaluated our approach on the proposed DUTLF-V2 and other two public light
field datasets: HFUT-LFSD [50] and LFSD [7]. DUTLF-V2 consists of 4204 light
field samples under a wide range of indoor and outdoor scenes which can be
divided into 102 classes. HFUT-LFSD and LFSD include 255 and 100 light fields,
respectively. Each light field consists of an all-in-focus image, 12 focal
slices focused at different depths and a corresponding manually labeled ground
truth.

In order to achieve a fair comparison, we randomly genearted 2957 samples from
DUTLF-V2 dataset and additional 100 samples from HFUT-LFSD dataset as the
training set. The remaining samples of DUTLF-V2 and HFUT-LFSD, and the LFSD
dataset are used for testing. To avoid overfitting, we augmented the training
set by flipping, cropping and rotating.

####  5.1.2 Evaluation Metrics.

To evaluate the performance of our approach and other methods, we adopted 5
widely-used metrics: F-measure [55], weighted F-measure [56], Mean Absolute
Error (MAE), S-measure [57], and E-measure [58]. The above evaluation metrics
can provide comprehensive and reliable evaluation results. Specifically, the
F-measure can comprehensively evaluate the quality of saliency maps and the
weighted F-measure is adopted to overcome the interpolation flaw, dependency
flaw and equal-importance flaw for a fair comparison. The MAE computes the
average absolute per-pixel difference between the saliency map and
corresponding ground truth. The S-measure can evaluate the spatial structure
similarities and the E-measure can jointly capture image level statistics and
local pixel matching information. In addition, we adopted model size and
Frames Per Second (FPS) to evaluate the complexity of each method.

####  5.1.3 Implementation Details.

Our method is implemented with Pytorch toolbox and trained on a PC with GTX
1080Ti GPU. We train both teacher network and student network using the SGD
optimization algorithm in which the momentum, weight decay and learning rate
are set to 0.9, 0.0005 and 1e-10, respectively. The hyperparameter temperature
T is set to 20 in all distillation loss functions. The minibatch size is 1 and
maximum iterations of both the teacher network and student network are set to
500000.

###  5.2 Comparison of Benchmarks

In order to offer deeper insights into the proposed dataset, we compared the
DUTLF-V2 to other RGB, RGB-D and light field datasets.

####  5.2.1 Comparison with RGB Datasets

Most existing RGB datasets include SOD [59], ECSSD [60], PASCAL-S [61], DUTS
[52] and MSRA10K [62]. Although a considerable number of samples are contained
in those datasets, they are either randomly selected from the public databases
or collected from the Internet. In contrast, the samples of our DUTLF-V2 are
collected in our daily lives. Our dataset is more closely relevant to real-
life scenarios. This offers an advantage in practical application for the task
of saliency detection.

####  5.2.2 Comparison with RGB-D Datasets

We compared our DUTLF-V2 with existing RGB-D datasets, including NJUD [63],
NLPR [64], RGBD135 [65], DUT-RGBD [53], SSD [66] and STEREO [67]. It is noted
that our DUTLF-V2 shows an advantage in scale over all the above datasets,
numerically increased by 111% compared to the current largest RGB-D dataset
NJUD. This significant increase in scale is beneficial for evaluating deep
saliency models. Additionally, compared to some synthetic or indoor datasets
(e.g., SSD and RGBD135), the DUTLF-V2 shows strength in coverage and
generalization. Specifically, SSD contains 80 samples collected from stereo
movies and RGBD135 contains 135 samples collected from constrained indoor
scenes. While our DUTLF-V2 consists of 4204 samples collected from a variety
of indoor and outdoor scenes that can be divided into 102 classes. The
introduce of our DUTLF-V2 can contribute to facilitate the research and
practices of RGB-D saliency detection.

####  5.2.3 Comparison with Light Field Datasets for Various Tasks

Compared with the public light field datasets for various tasks listed in
Table I, our proposed DUTLF-V2 is significantly improved in terms of scale,
diversity and applicability. We can see from Table I that the scale of our
DUTLF-V2 exceeds all the public light field datasets, increased by 27%
compared to the current largest light field dataset LFVSD [20] for view
synthesis. We also compared the coverage which helps to showcase the high
diversity of our dataset. Our DUTLF-V2 contains 102 classes of samples, over
three times that of the most diverse one [23] which contains 30 classes.
Additionally, we can note that DUTLF-V2 provides a variety of data elements
(e.g., RGB images, multi views, focal stacks, depth maps), while over 88
percent of the listed datasets contain only three or less data types. The
sufficient representation of light fields enables versatile application of our
dataset in RGB, RGB-D and light field saliency detection.

####  5.2.4 Comparison with the Previous Version (DUTLF-V1)

DUTLF-V1 [9] is the existing largest light field saliency detection dataset
and encouraged light field saliency detection to a certain extent. Compared
with DUTLF-V1, our newly proposed DUTLF-V2 has been greatly improved in
several aspects, including scale, diversity and complexity. Specifically,
DUTLF-V2 contains 4204 light field samples, increased by 187% compared to
DUTLF-V1 as shown in Table I. Meanwhile, our DUTLF-V2 covers a wider range of
indoor and outdoor scenes, which can be divided into 102 classes. It is also
noted that the number and type of challenging scenes are greatly increased in
DUTLF-V2, meeting comprehensive training needs as well as bringing up more
challenges for salient object detection. The numerical results of our method
trained on DUTLF-V1 and DUTLF-V2 also confirm that our extended DUTLF-V2 can
powerfully assist in higher generalization of saliency models to other
datasets. As shown in Table II, the proposed method trained on DUTLF-V2
achieves considerable performance gains compared to that trained on DUTLF-V1.
Specifically, in terms of the student network, the MAE is reduced by 24.1% and
9.9% on HFUT and LFSD, respectively. We can see that DUTLF-V2 shows advantages
in training saliency models and encourages more advanced research in this
field.

TABLE II: Comparision of the Training Set of DUTLF-V1 and DUTLF-V2 (red bold: top-1 results)  methods | Training Set | HFUT | LFSD
---|---|---|---
|  |  $F_{\beta}\uparrow$ |  MAE$\downarrow$ |  $F_{\beta}\uparrow$ |  MAE$\downarrow$
Teacher |  DUTLF-V1 |  .705 |  .082 |  .842 |  .080
|  DUTLF-V2 |  .753 |  .069 |  .856 |  .084
Student |  DUTLF-V1 |  .668 |  .091 |  .712 |  .151
|  DUTLF-V2 |  .714 |  .069 |  .738 |  .136
TABLE III: Quantitative Results of the Ablation Analysis for Our Teacher Network (red bold: top-1 results)  Module | Model | DUTLF-V2 | HFUT-LFSD
---|---|---|---
$F^{w}_{\beta}\uparrow$ |  MAE$\downarrow$ |  $F^{w}_{\beta}\uparrow$ |  MAE$\downarrow$
MFRM | Only $L_{CE}$ |  .769 |  .054 |  .664 |  .073
w/MFRM |  .792 |  .050 |  .687 |  .069
MFSM | Step 1 |  .740 |  .059 |  .638 |  .076
Step 2 |  .746 |  .058 |  .653 |  .075
Step 3 |  .778 |  .053 |  .672 |  .071
| Ours (step 4) |  .792 |  .050 |  .687 |  .069
Figure 5: Visual comparisions in MFRM. The 1st to 3rd columns show the focal
slices and corresponding multi-focusness features. The 4th column shows the
ground truth and saliency maps. Figure 6: Visual comparisions of different
steps in MFSM.

###  5.3 Teacher Network Analysis

####  5.3.1 Effect of MFRM.

To validate the MFRM of the teacher network in recruiting comprehensive
saliency information from focal slices, we conducted visual comparisons (see
Figure 5) of the multi-focusness features generated with simple supervision
(denoted as ‘Only $L_{CE}$’) and our proposed MFRM (denoted as ‘w/MFRM’). We
can observe that the multi-focusness features generated by simple supervision
are very similar. This may limit the diversity of features and lead to sub-
optimal results, such as false positives (row 3) or incomplete detection of
salient objects (row 4). In contrast, the MFRM encourages adequate diversity
between features of different focal slices to achieve optimal results (row 5
and row 6). Quantitative results shown in Table III are numerically consistent
that the MFRM brings considerable performance gains. Our MFRM reduces the MAE
by 7.4% and 5.5% on DUTLF-V2 dataset and HFUT-LFSD dataset, respectively.

TABLE IV: Ablation Analysis of the Proposed Distillation Schemes on Student Network (red bold: top-1 results)  Model | DUTLF-V2 | HFUT-LFSD | LFSD
---|---|---|---
|  $F^{w}_{\beta}\uparrow$ |  MAE$\downarrow$ |  $F^{w}_{\beta}\uparrow$ |  MAE$\downarrow$ |  $F^{w}_{\beta}\uparrow$ |  MAE$\downarrow$
Student |  .721 |  .068 |  .651 |  .076 |  .647 |  .150
+MFD |  .743 |  .064 |  .669 |  .073 |  .650 |  .149
+MFD+AFD |  .749 |  .062 |  .667 |  .071 |  .678 |  .138
+MFD+AFD+SFD |  .771 |  .055 |  .678 |  .069 |  .680 |  .136
TABLE V: Ablation Analysis of the Proposed Distillation Schemes Based on Different Backbones (red bold: top-1 results) Model | DUTLF-V2 | HFUT-LFSD | LFSD
---|---|---|---
|  $F^{w}_{\beta}\uparrow$ |  MAE$\downarrow$ |  $F^{w}_{\beta}\uparrow$ |  MAE$\downarrow$ |  $F^{w}_{\beta}\uparrow$ |  MAE$\downarrow$
ShuffleNet |  .591 |  .093 |  .537 |  .099 |  .659 |  .142
+MFD |  .624 |  .084 |  .553 |  .097 |  .661 |  .140
+MFD+AFD |  .689 |  .073 |  .621 |  .079 |  .672 |  .139
+MFD+AFD+SFD |  .691 |  .074 |  .639 |  .082 |  .701 |  .130
ResNet18 |  .637 |  .081 |  .565 |  .094 |  .668 |  .135
+MFD |  .697 |  .065 |  .605 |  .082 |  .676 |  .137
+MFD+AFD |  .730 |  .064 |  .655 |  .074 |  .678 |  .135
+MFD+AFD+SFD |  .761 |  .057 |  .695 |  .069 |  .685 |  .133

####  5.3.2 Effect of MFSM.

To provide evidence for the screening ability of MFSM, we visualized the
saliency maps in different time steps as shown in Figure 6. We can observe
that the attention module contributes more on locating salient object
accurately in step 1 and 2, while the ConvLSTM contributes more on refining
the details of salient object in step 3 and 4. The quantitative results are
listed in Table III. Accumulative improvements are achieved as the time step
increases. These improvements are resonable since the useful features are
emphasized by the attention module and spatial details are refined gradually
by ConvLSTM.

###  5.4 Knowledge Distillation Scheme Analysis

To demonstrate the effectiveness of our proposed learning strategy, we
conducted extensive ablations to explore the three distillation schemes, the
multi-focusness distillation scheme, the attentive focusness distillation
scheme and the screened focusness distillation scheme. The experiments are
conducted on the student network (baseline VGG16) and all distillation schemes
can independently produce fairly comparable results over the original student
model.

####  5.4.1 Multi-Focusness Distillation Scheme (MFD)

The results in Table IV show numerical improvements as we applied the MFD.
Specifically, the performance represents a 5.9%, 3.9% and 0.7% increase on
DUTLF-V2, HFUT-LFSD and LFSD in terms of MAE, respectively. Meanwhile, we can
see from the visualization of multi-focusness features in Figure 7 that the
MFD focuses on transfering more effective multi-focusness features, such as
features with finer boundary of salient object (row 3, column 1). Therefore,
it produces more accurate prediction (row 3, column 4).

Figure 7: Visualization of multi-focusness features of enabling and disabling
MFD. Figure 8: Visual results of enabling and disabling different components
of our distillation system. ‘Student+M’, ‘Student+M+A’ and ‘Student+M+A+S’
refer to ‘Student+MFD’, ‘Student+MFD+AFD’ and ‘Student+MFD+AFD+SFD’,
respectively.

####  5.4.2 Attentive Focusness Distillation Scheme (AFD)

It can be seen that as we employed the AFD, the performance of the student
network achieves accumulative improvements. Specifically, the performance
represents a 3.1%, 2.7% and 7.4% increase on DUTLF-V2, HFUT-LFSD and LFSD in
terms of MAE, respectively. The AFD transfers attentive knowledge that
highlights the foreground, thus leads to more consistent saliency prediction
(column 5 in Figure 8).

####  5.4.3 Screened Focusness Distillation Scheme (SFD)

As we further applied the SFD, the performance represents a 11.3%, 2.8% and
1.4% increase on DUTLF-V2, HFUT-LFSD and LFSD in terms of MAE, respectively.
We also visualized the results employing the SFD shown in Figure 8. The SFD
allows student network to learn complementarity of appearance and screened
focusness information. Therefore, the student with SFD achieves a more
detailed prediction. When all distillation losses are employed, we obtained
the sweet spot of the three. This validates the importance of focusness
information transferring from the Focal steam to the RGB stream.

TABLE VI: Quantitative Comparisons of E-measure, S-measure, Weighted F-measure, F-measure and MAE Scores on Three Light Field Datasets. The best three results are shown in red bold, bule bold, green bold fonts respectively ($*$ represents conventional methods, - means no available results) Type | Methods | Years | Size(M)$\downarrow$ | FPS$\uparrow$ | DUTLF-V2 | HFUT-LFSD | LFSD
---|---|---|---|---|---|---|---
|  |  |  |  |  $E_{s}\uparrow$ |  $S_{\alpha}\uparrow$ |  $F^{w}_{\beta}\uparrow$ |  $F_{\beta}\uparrow$ |  MAE$\downarrow$ |  $E_{s}\uparrow$ |  $S_{\alpha}\uparrow$ |  $F^{w}_{\beta}\uparrow$ |  $F_{\beta}\uparrow$ |  MAE$\downarrow$ |  $E_{s}\uparrow$ |  $S_{\alpha}\uparrow$ |  $F^{w}_{\beta}\uparrow$ |  $F_{\beta}\uparrow$ |  MAE$\downarrow$
4D |  Teacher |  - |  92.5 |  14 |  .924 |  .852 |  .792 |  .852 |  .050 |  .858 |  .778 |  .687 |  .753 |  .069 |  .875 |  .827 |  .799 |  .852 |  .087
2D |  Student |  - |  47 |  105 |  .909 |  .843 |  .771 |  .813 |  .055 |  .850 |  .787 |  .678 |  .714 |  .069 |  .811 |  .737 |  .680 |  .738 |  .136
4D |  LFNet |  TIP’20 |  175.8 |  13 |  .907 |  .870 |  .786 |  .803 |  .049 |  .840 |  .800 |  .678 |  .701 |  .065 |  .866 |  .812 |  .760 |  .814 |  .100
|  DLFS |  IJCAI’19 |  119 |  2 |  .839 |  .786 |  .641 |  .684 |  .087 |  .783 |  .741 |  .590 |  .615 |  .098 |  .806 |  .737 |  .657 |  .715 |  .147
|  MoLF |  NIPS’19 |  186.6 |  5 |  .866 |  .825 |  .709 |  .723 |  .065 |  .829 |  .789 |  .664 |  .678 |  .075 |  .886 |  .830 |  .780 |  .819 |  .089
|  LFS∗ |  TPAMI’17 |  - |  - |  - |  - |  - |  - |  - |  .686 |  .579 |  .264 |  .430 |  .205 |  .771 |  .680 |  .479 |  .740 |  .208
|  MCA∗ |  TOOM’17 |  - |  - |  - |  - |  - |  - |  - |  - |  - |  - |  - |  - |  .841 |  .749 |  - |  .815 |  .150
|  WSC∗ |  CVPR’15 |  - |  - |  - |  - |  - |  - |  - |  - |  - |  - |  - |  - |  .794 |  .706 |  .642 |  .706 |  .156
|  DILF∗ |  IJCAI’15 |  - |  - |  .733 |  .648 |  .388 |  .504 |  .187 |  .736 |  .695 |  .458 |  .555 |  .131 |  .810 |  .755 |  .604 |  .728 |  .168
3D |  DisenFusion |  TIP’20 |  166 |  8 |  .836 |  .781 |  .636 |  .686 |  .093 |  .759 |  .734 |  .571 |  .605 |  .107 |  .831 |  .798 |  .724 |  .780 |  .115
|  ATAFNet |  ACMMM’20 |  291.5 |  21 |  .904 |  .859 |  .775 |  .808 |  .051 |  .856 |  .817 |  .712 |  .736 |  .062 |  .868 |  .806 |  .761 |  .791 |  .103
|  CPFP |  CVPR’19 |  278 |  7 |  .843 |  .764 |  .629 |  .707 |  .075 |  .768 |  .701 |  .536 |  .594 |  .096 |  .669 |  .599 |  .465 |  .524 |  .186
|  TANet |  TIP’19 |  951.9 |  15 |  .849 |  .765 |  .609 |  .688 |  .090 |  .789 |  .744 |  .587 |  .638 |  .096 |  .849 |  .803 |  .727 |  .804 |  .112
|  MMCI |  PR’19 |  929.7 |  19 |  .829 |  .760 |  .545 |  .665 |  .109 |  .787 |  .741 |  .540 |  .645 |  .104 |  .848 |  .799 |  .685 |  .796 |  .128
|  PCA |  CVPR’18 |  533.6 |  15 |  .842 |  .762 |  .601 |  .683 |  .096 |  .782 |  .748 |  .598 |  .644 |  .095 |  .846 |  .807 |  .733 |  .801 |  .112
|  CTMF |  Tcyb’17 |  825.8 |  50 |  .836 |  .780 |  .573 |  .684 |  .104 |  .784 |  .752 |  .544 |  .620 |  .103 |  .856 |  .801 |  .710 |  .791 |  .119
2D |  F3Net |  AAAI’20 |  102.5 |  92 |  .878 |  .841 |  .756 |  .803 |  .063 |  .810 |  .776 |  .673 |  .707 |  .094 |  .737 |  .758 |  .695 |  .742 |  .136
|  SCRN |  ICCV’19 |  96.6 |  29 |  .885 |  .847 |  .743 |  .790 |  .064 |  .823 |  .792 |  .676 |  .720 |  .093 |  .829 |  .796 |  .731 |  .763 |  .117
|  EGNet |  ICCV’19 |  412 |  21 |  .855 |  .821 |  .710 |  .746 |  .078 |  .794 |  .772 |  .634 |  .672 |  .094 |  .776 |  .784 |  .717 |  .762 |  .118
|  CPD |  CVPR’19 |  112 |  66 |  .885 |  .836 |  .754 |  .794 |  .061 |  .810 |  .764 |  .652 |  .689 |  .097 |  .865 |  .846 |  .796 |  .841 |  .083
|  PoolNet |  CVPR’19 |  278.5 |  32 |  .876 |  .832 |  .732 |  .774 |  .069 |  .802 |  .776 |  .652 |  .683 |  .092 |  .786 |  .800 |  .717 |  .769 |  .118
|  PiCANet |  CVPR’18 |  197.2 |  5 |  .850 |  .820 |  .685 |  .738 |  .086 |  .726 |  .781 |  .556 |  .618 |  .115 |  .780 |  .729 |  .621 |  .671 |  .158
|  C2S |  ECCV’18 |  158 |  30 |  .824 |  .787 |  .651 |  .691 |  .098 |  .786 |  .763 |  .630 |  .650 |  .111 |  .820 |  .806 |  .737 |  .749 |  .113
|  R3Net |  IJCAI’18 |  225.3 |  3 |  .807 |  .724 |  .588 |  .643 |  .108 |  .728 |  .727 |  .566 |  .625 |  .151 |  .838 |  .789 |  .717 |  .781 |  .128
|  DSS |  CVPR’17 |  447.3 |  23 |  .828 |  .745 |  .597 |  .669 |  .101 |  .778 |  .715 |  .511 |  .626 |  .133 |  .749 |  .677 |  .570 |  .644 |  .190
Figure 9: Visual comparisons of our method with top-ranking CNNs-based methods
in some challenging scenes.

###  5.5 Different Backbones

In order to further verify the effect and generalization of the three
distillation schemes, we extended our work by replacing the VGG16 with other
two lightweight backbones. As can be seen in Table V, the proposed
distillation schemes contribute to accumulative performance improvements on
all three datasets as the backbone is replaced by ShuffleNet or ResNet18.
Specifically, in the setting with backbone ResNet18, our MFD, AFD and SFD
reduce the MAE by 12.8%, 9.8% and 6.8% on HFUT-LFSD, respectively. From the
above observation, we can conclude that the three distillation schemes can
effectively assist in predicting more accurate saliency results. Also, the
improvement reflects high flexibility of the proposed distillation schemes
that offer more benefits for a wide range of applications.

###  5.6 Comparison with State-of-the-arts

We compared our method with 23 state-of-the-art approaches including both
deep-learning based methods and conventional methods (marked with ‘$*$’).
There are seven 4D light field methods: LFNet [68], DLFS [9], MoLF [51], LFS∗
[7], MCA∗ [19], WSC∗ [8], DILF∗ [50]; seven 3D RGBD methods: DisenFusion [70],
ATAFNet [71], CPFP [72], TANet [49], MMCI [47], PCA [46], CTMF [73]; and nine
top-ranking 2D RGB methods: F3Net [74], SCRN [44], EGNet [75], CPD [41],
PoolNet [40], PiCANet [76], C2S [38], R3Net [77], DSS [78]. For a fair
comparison, the results of the competing methods are generated by authorized
codes or directly provided by authors.

TABLE VII: Application of the Proposed Knowledge Distillation Schemes in Top-ranking RGB Saliency Models (red bold: top-1 results)  Methods | Size(M)$\downarrow$ | DUTLF-V2 | HFUT-LFSD
---|---|---|---
|  |  $F^{w}_{\beta}\uparrow$ |  MAE$\downarrow$ |  $F^{w}_{\beta}\uparrow$ |  MAE$\downarrow$
R3Net-retrain |  214.92 |  .757 |  .053 |  .656 |  .072
R3Net+SFD |  214.92 |  .775 |  .055 |  .680 |  .066
R3Net+MFD+AFD+SFD |  214.93 |  .793 |  .051 |  .716 |  .062
SCRN-retrain |  96.71 |  .747 |  .055 |  .645 |  .078
SCRN+SFD |  96.71 |  .768 |  .055 |  .677 |  .702
SCRN+MFD+AFD+SFD |  96.72 |  .772 |  .054 |  .693 |  .066
CPD-retrain |  111.56 |  .703 |  .071 |  .623 |  .088
CPD+SFD |  111.56 |  .711 |  .068 |  .610 |  .084
CPD+MFD+AFD+SFD |  111.57 |  .773 |  .055 |  .643 |  .072

####  5.6.1 The Teacher Network

As can be seen from the quantitative results in Table VI, our proposed teacher
network can largely outperform other advanced saliency models across all the
datasets in terms of most evaluation metrics. It is worth mentioning that our
significant advantages are achieved on the training set (3057 samples) three
times smaller than the large RGB training set (10553 samples). We also show
some visual results in Figure 9. It can be easily seen that our teacher
network can achieve more complete and accurate predictions in various
challenging scenes, including similar foreground and background (row 1, 2 and
3), small or transparent salient object (row 4 and row 5), complex background
(row 6).

####  5.6.2 The Student Network

Our proposed focusness knowledge distillation schemes can be seen as an
effective replacement for the Focal stream, this leads to considerable results
of the student network (VGG16), such as Top-2 F-measure on DUTLF-V2 dataset
even with a single RGB input. Furthermore, we compared the average execution
time and model size of our method with representative models. It is noted that
the model size of our student network (VGG16) is only 47 MB and mean while FPS
reaches up to 105. Compared to the currently best performing method ATAFNet,
our student network tremedously minimizes the model size by 83% and boosts the
FPS by 5 times. The precision-complexity scatterplot in Figure 10 and 11 also
vividly demonstrate the superior performance of our student network.

Figure 10: Precision-Complexity scatterplot of the compared methods, abscissa
represents model size, ordinate represents F-measure over the DUTLF-V2
dataset. Figure 11: Precision-Complexity scatterplot of the compared methods,
abscissa represents FPS, ordinate represents F-measure over the DUTLF-V2
dataset.

###  5.7 Application

In this section, we applied the three distillation schemes to several top-
ranking RGB saliency models, including R3Net, SCRN and CPD. For a fair
comparison, we retrained the original RGB models on the same training dataset
of our method. The application is conducted via the following two settings
with no or few extra parameters, both of which have obtained impressive
performance gains. Specifically, for the no extra parameters setting, we
trained the RGB models by directly applying the screened focusness
distillation scheme which introduces no extra layers or parameters. For the
few extra parameters setting, we adopted all three distillation schemes by
adding a convolutional layer and a slice-wise attention module to impose the
multi-focusness distillation scheme and attentive focusness distillation
scheme, respectively. The added layers are lightweight and carry negligible
computational burden, referred to Table VII. Next, we will introduce details
of the above two application patterns.

Figure 12: Visualization of application with the no extra parameters setting.
‘R3Net-re’, ‘R3Net+S’ refer to ‘R3Net-retrain’ and ‘R3Net+SFD’, respectively.
Figure 13: Visualization of application with the few extra parameters setting.
‘R3Net+S’, ‘R3Net+M+A+S’ refer to ‘R3Net+SFD’ and ‘R3Net+MFD+AFD+SFD’,
respectively.

####  5.7.1 No Extra Parameters Setting

The first application is to embed the screened focusness distillation scheme
in the existing models. In this way, there is no need to modify the structure
of the original model, and no extra parameters are introduced. This ensures
the flexibility and versatility of the application. It is noted that the
models embedded with SFD are capable of learning complementary information
from appearance and screened focusness information. As shown in Table VII,
R3Net embedded with SFD (denoted as ‘R3Net+SFD’) has a consistent improvement
compared with its original architecture (denoted as ‘R3Net-retrain’), this
application setting also improves the performance of SCRN and CPD. Besides,
some visual comparisons of challenging examples are illustrated in Figure 12.
We can note that after applying SFD, the saliency maps achieve superior
consistence and improved robustness.

####  5.7.2 Few Extra Parameters Setting

We further applied all three distillation schemes to the advanced RGB models.
This application setting slightly modified the original models and introduced
few extra parameters, while achieving a large margin of improvement on
performance. As shown in Table VII, R3Net adopted MFD, AFD and SFD (denoted as
‘R3Net+MFD+AFD+SFD’) outperforms the original R3Net (denoted as ‘R3Net-
retrain’), numerically reducing the MAE by 3.7% and 13.8% on DUTLF-V2 and
HFUT-LFSD, respectively. The performance of other two models also benefits
from our application. Meanwhile, the visual results in Figure 13 illustrate
that this application to RGB models helps them to predict more accurate
saliency maps. It is also noted the comparison results of the two application
settings demonstrate that the application of all three distillation schemes is
more effective with few extra parameters.

##  6 Discussion

In our method, the proposed comprehensive and challenging DUTLF-V2 can assist
in higher generalization for our models, and the three distillation schemes
aim to ensure better absorption and integration of focusness knowledge for the
student. This enables the student no longer need the focal slices but a single
RGB image as input, and guarantees flexibility and productivity for mobile
devices. But interestingly, we noticed that there is a certain performance gap
between the student and the teacher. One possible reason we consider is that
as the teacher network propagates forward and takes up a large percentage of
GPU memory, less computing resources are allocated to the student network and
the batch size is set to 1, thus resulting in suboptimal performance and
generalization of student network[79]. We argue that this problem will be
relieved as computing resources are updated.

##  7 Conclusion

In this paper, we constructed the DUTLF-V2 dataset, which contains a large
scale of samples, diverse realistic scenes and a variety of data elements. Our
proposed dataset provides a generic benchmark for the field of saliency
detection. Furthermore, we presented a novel asymmetrical two-stream network,
which consists of Focal stream and RGB stream, to achieve efficiency and
versatility for both desktop computers and mobile devices. On one hand, we
consider the Focal stream as a teacher network, to learn to exploit focal
slices and produce focusness knowledge tailored for student. Our proposed MFRM
and MFSM recruit and screen useful saliency information effectively and enable
the teacher network to achieve superior performance. On the other hand, we
train the student network using single RGB input, to learn to replace focal
slices relying on three tailor-made distillation schemes. The proposed
distillation schemes allow the student to take advantage of multi-focusness
information and ensure versatility and productivity for mobile devices. Our
evaluation shows the state-of-the-art performance of the teacher and the
flexibility of the student. Additionally, we demonstrate the applicability of
the three proposed distillation schemes under a wide range of applications.
Experimental results confirm the generalization and effectiveness of our
distillation schemes.

## References

  * [1] Y. Li, X. Hou, C. Koch, J. M. Rehg, and A. L. Yuille, “The secrets of salient object segmentation,” pp. 280–287, 2014.
  * [2] S. Hong, T. You, S. Kwak, and B. Han, “Online tracking by learning discriminative saliency map with convolutional neural network,” _CVPR_ , 2015\.
  * [3] A. Smeulders, D. Chu, R. Cucchiara, S. Calderara, A. Dehghan, and M. Shah, “Visual tracking: An experimental survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 11 2013.
  * [4] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2015\.
  * [5] J. Dai, Y. Li, K. He, and J. Sun, “R-fcn: Object detection via region-based fully convolutional networks,” _International Conference on Neural Information Processing Systems (NIPS)_ , p. 379–387, 2016.
  * [6] C. Craye, D. Filliat, and J.-F. Goudou, “Environment exploration for object-based visual saliency learning,” 05 2016.
  * [7] N. Li, J. Ye, Y. Ji, H. Ling, and J. Yu, “Saliency detection on light field.” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 8, pp. 1605–1616, 2017.
  * [8] N. Li, B. Sun, and J. Yu, “A weighted sparse coding framework for saliency detection,” in _CVPR_ , 2015, pp. 5216–5223.
  * [9] Y. Piao, Z. Rong, M. Zhang, X. Li, and H. Lu, “Deep light-field-driven saliency detection from a single view,” 2019.
  * [10] S. Wanner, S. Meister, and B. Goldluecke, “Datasets and benchmarks for densely sampled 4d light fields,” pp. 225–226, 2013.
  * [11] N. Li, J. Ye, Y. Ji, H. Ling, and J. Yu, “Saliency detection on light field,” _IEEE Transactions on Pattern Analysis & Machine Intelligence_, no. 8, pp. 1605–1616, 2017.
  * [12] R. Raghavendra, K. B. Raja, and C. Busch, “Presentation attack detection for face recognition using light field camera,” _IEEE Transactions on Image Processing_ , vol. 24, no. 3, pp. 1060–1075, 2015.
  * [13] S. Suwajanakorn, C. Hernandez, and S. M. Seitz, “Depth from focus with your mobile phone,” pp. 3497–3506, 2015.
  * [14] Y. Xu, H. Nagahara, A. Shimada, and R. ichiro Taniguchi, “Transcut2: Transparent object segmentation from a light-field image,” 2015.
  * [15] K. Honauer, O. Johannsen, D. Kondermann, and B. Goldluecke, “A dataset and evaluation methodology for depth estimation on 4d light fields,” pp. 19–34, 2016\.
  * [16] A. S. Raj, M. Lowney, R. Shah, and G. Wetzstein, “Stanford lytro light field archive,” http://lightfield.stanford.edu/, 2016.
  * [17] M. Rerabek and T. Ebrahimi, “New light field image dataset,” 2016.
  * [18] T. Wang, J. Zhu, E. Hiroaki, M. Chandraker, A. A. Efros, and R. Ramamoorthi, “A 4d light-field dataset and cnn architectures for material recognition,” 2016\.
  * [19] J. Zhang, M. Wang, L. Lin, X. Yang, J. Gao, and Y. Rui, “Saliency detection on light field: A multi-cue approach,” _ACM Transactions on Multimedia Computing, Communications, and Applications_ , vol. 13, no. 3, p. 32, 2017.
  * [20] P. P. Srinivasan, T. Wang, A. Sreelal, R. Ramamoorthi, and R. Ng, “Learning to synthesize a 4d rgbd light field from a single image,” in _ICCV_ , 2017, pp. 2262–2270.
  * [21] N. Sabater, G. Boisson, B. Vandame, P. Kerbiriou, F. Babon, M. Hog, R. Gendrot, T. Langlois, O. Bureller, A. Schubert _et al._ , “Dataset and pipeline for multi-view light-field video,” pp. 1743–1753, 2017.
  * [22] M. L. Pendu, X. Jiang, and C. Guillemot, “Light field inpainting propagation via low rank matrix completion,” 2018.
  * [23] D. G. Dansereau, B. Girod, and G. Wetzstein, “Liff: Light field features in scale and depth,” pp. 8042–8051, 2019.
  * [24] J. Shi, X. Jiang, and C. Guillemot, “A framework for learning depth from a flexible subset of dense and sparse light field views,” _IEEE Transactions on Image Processing_ , vol. 28, no. 12, pp. 5867–5880, 2019.
  * [25] J. S. Lumentut, T. H. Kim, R. Ramamoorthi, and I. K. Park, “Fast and full-resolution light field deblurring using a deep neural network,” _SPL_ , 2019.
  * [26] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [27] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The lumigraph,” in _Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques_ , ser. SIGGRAPH, 1996, p. 43–54.
  * [28] M. Levoy and P. Hanrahan, “Light field rendering,” in _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_ , 1996, pp. 31–42.
  * [29] B. Wilburn, N. Joshi, V. Vaish, E. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy, “High performance imaging using large camera arrays,” _Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques_ , vol. 24, no. 3, pp. 765–776, 2005.
  * [30] E. Adelson and J. Wang, “Single lens stereo with a plenoptic camera,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 14, no. 2, pp. 0–106.
  * [31] D. G. Dansereau, G. M. Schuster, J. E. Ford, and G. Wetzstein, “A wide-field-of-view monocentric light field camera,” _CVPR_ , pp. 3757–3766, 2017.
  * [32] K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar, “Compressive light field photography using overcomplete dictionaries and optimized projections,” _ACM Trans. Graph._ , vol. 32, no. 4, 2013.
  * [33] A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin, “Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing,” _International Conference on Computer Graphics and Interactive Techniques_ , vol. 26, no. 3, p. 69, 2007.
  * [34] W.-C. Tu, S. He, Q. Yang, and S.-Y. Chien, “Real-time salient object detection with a minimum spanning tree,” in _CVPR_ , 2016, pp. 2334–2342.
  * [35] Y. Qin, H. Lu, Y. Xu, and H. Wang, “Saliency detection via cellular automata,” in _CVPR_ , 2015, pp. 110–119.
  * [36] X. Li, H. Lu, L. Zhang, X. Ruan, and M.-H. Yang, “Saliency detection via dense and sparse reconstruction,” in _ICCV_ , 2013, pp. 2976–2983.
  * [37] P. Zhang, D. Wang, H. Lu, H. Wang, and X. Ruan, “Amulet: Aggregating multi-level convolutional features for salient object detection,” in _ICCV_ , 2017, pp. 202–211.
  * [38] X. Li, F. Yang, H. Cheng, W. Liu, and D. Shen, “Contour knowledge transfer for salient object detection,” in _ECCV_ , 2018, pp. 355–370.
  * [39] X. Zhang, T. Wang, J. Qi, H. Lu, and G. Wang, “Progressive attention guided recurrent network for salient object detection,” in _CVPR_ , 2018, pp. 714–722.
  * [40] J.-J. Liu, Q. Hou, M.-M. Cheng, J. Feng, and J. Jiang, “A simple pooling-based design for real-time salient object detection,” _arXiv preprint arXiv:1904.09569_ , 2019.
  * [41] Z. Wu, L. Su, and Q. Huang, “Cascaded partial decoder for fast and accurate salient object detection,” in _CVPR_ , 2019, pp. 3907–3916.
  * [42] Z. Chen, Q. Xu, R. Cong, and Q. Huang, “Global context-aware progressive aggregation network for salient object detection,” 2020.
  * [43] Y. Pang, X. Zhao, L. Zhang, and H. Lu, “Multi-scale interactive network for salient object detection,” 2020.
  * [44] Z. Wu, L. Su, and Q. Huang, “Stacked cross refinement network for edge-aware salient object detection,” in _ICCV_ , 2019, pp. 7264–7273.
  * [45] C. Zhu, X. Cai, K. Huang, T. H. Li, and G. Li, “Pdnet: Prior-model guided depth-enhanced network for salient object detection.” _arXiv preprint arXiv:1803.08636_ , 2018.
  * [46] H. Chen and Y. Li, “Progressively complementarity-aware fusion network for rgb-d salient object detection,” in _CVPR_ , 2018, pp. 3051–3060.
  * [47] H. Chen, Y. Li, and D. Su, “Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for rgb-d salient object detection,” _Pattern Recognition_ , vol. 86, pp. 376–385, 2019.
  * [48] Y. Piao, Z. Rong, M. Zhang, W. Ren, and H. Lu, “A2dele: Adaptive and attentive depth distiller for efficient rgb-d salient object detection,” in _CVPR_ , 2020.
  * [49] H. Chen and Y. Li, “Three-stream attention-aware network for rgb-d salient object detection,” _TIP_ , vol. 28, no. 6, pp. 2825–2835, 2019.
  * [50] J. Zhang, M. Wang, J. Gao, Y. Wang, X. Zhang, and X. Wu, “Saliency detection with a deeper investigation of light field,” in _IJCAI_ , 2015, pp. 2212–2218.
  * [51] M. Zhang, J. Li, W. Ji, Y. Piao, and H. Lu, “Memory-oriented decoder for light field salient object detection,” in _NeurIPS_ , 2019.
  * [52] L. Wang, H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, and X. Ruan, “Learning to detect salient objects with image-level supervision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 136–145.
  * [53] Y. Piao, W. Ji, J. Li, M. Zhang, and H. Lu, “Depth-induced multi-scale recurrent attention network for saliency detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 7254–7263.
  * [54] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _ICLR_ , 2015.
  * [55] R. Achanta, S. Hemami, F. Estrada, and S. Süsstrunk, “Frequency-tuned salient region detection,” in _CVPR_ , no. CONF, 2009, pp. 1597–1604.
  * [56] R. Margolin, L. Zelnik-Manor, and A. Tal, “How to evaluate foreground maps?” in _CVPR_ , 2014, pp. 248–255.
  * [57] D. Fan, M. Cheng, Y. Liu, T. Li, and A. Borji, “Structure-measure: A new way to evaluate foreground maps,” in _ICCV_ , 2017, pp. 4558–4567.
  * [58] D. Fan, C. Gong, Y. Cao, B. Ren, M. Cheng, and A. Borji, “Enhanced-alignment measure for binary foreground map evaluation,” in _IJCAI_ , 2018, pp. 698–704.
  * [59] V. Movahedi and J. H. Elder, “Design and perceptual validation of performance measures for salient object segmentation,” in _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops_. IEEE, 2010, pp. 49–56.
  * [60] Q. Yan, L. Xu, J. Shi, and J. Jia, “Hierarchical saliency detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, pp. 1155–1162.
  * [61] Y. Li, X. Hou, C. Koch, J. M. Rehg, and A. L. Yuille, “The secrets of salient object segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 280–287.
  * [62] M.-M. Cheng, N. J. Mitra, X. Huang, P. H. Torr, and S.-M. Hu, “Global contrast based salient region detection,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 37, no. 3, pp. 569–582, 2014.
  * [63] R. Ju, L. Ge, W. Geng, T. Ren, and G. Wu, “Depth saliency based on anisotropic center-surround difference,” in _2014 IEEE international conference on image processing (ICIP)_. IEEE, 2014, pp. 1115–1119.
  * [64] H. Peng, B. Li, W. Xiong, W. Hu, and R. Ji, “Rgbd salient object detection: a benchmark and algorithms,” in _European conference on computer vision_. Springer, 2014, pp. 92–109.
  * [65] Y. Cheng, H. Fu, X. Wei, J. Xiao, and X. Cao, “Depth enhanced saliency detection method,” in _Proceedings of international conference on internet multimedia computing and service_ , 2014, pp. 23–27.
  * [66] C. Zhu and G. Li, “A three-pathway psychobiological framework of salient object detection using stereoscopic technology,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 3008–3014.
  * [67] Y. Niu, Y. Geng, X. Li, and F. Liu, “Leveraging stereopsis for saliency analysis,” in _2012 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2012, pp. 454–461.
  * [68] M. Zhang, W. Ji, Y. Piao, J. Li, Y. Zhang, S. Xu, and H. Lu, “Lfnet: Light field fusion network for salient object detection,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 6276–6287, 2020.
  * [69] M. Zhang, J. Li, J. Wei, Y. Piao, and H. Lu, “Memory-oriented decoder for light field salient object detection,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 898–908.
  * [70] H. Chen, Y. Deng, Y. Li, T.-Y. Hung, and G. Lin, “Rgbd salient object detection via disentangled cross-modal fusion,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 8407–8416, 2020.
  * [71] M. Zhang, Y. Zhang, Y. Piao, B. Hu, and H. Lu, “Feature reintegration over differential treatment: A top-down and adaptive fusion network for rgb-d salient object detection,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 4107–4115.
  * [72] J.-X. Zhao, Y. Cao, D.-P. Fan, M.-M. Cheng, X.-Y. Li, and L. Zhang, “Contrast prior and fluid pyramid integration for rgbd salient object detection,” in _CVPR_ , 2019.
  * [73] J. Han, H. Chen, N. Liu, C. Yan, and X. Li, “Cnns-based rgb-d saliency detection via cross-view transfer and multiview fusion,” _IEEE transactions on cybernetics_ , no. 99, pp. 1–13, 2017.
  * [74] J. Wei, S. Wang, and Q. Huang, “F3net: Fusion, feedback and focus for salient object detection,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 12 321–12 328.
  * [75] J.-X. Zhao, J.-J. Liu, D.-P. Fan, Y. Cao, J. Yang, and M.-M. Cheng, “Egnet:edge guidance network for salient object detection,” in _ICCV_ , Oct 2019.
  * [76] N. Liu, J. Han, and M.-H. Yang, “Picanet: Learning pixel-wise contextual attention for saliency detection,” in _CVPR_ , 2018, pp. 3089–3098.
  * [77] Z. Deng, X. Hu, L. Zhu, X. Xu, J. Qin, G. Han, and P.-A. Heng, “R3net: Recurrent residual refinement network for saliency detection,” in _IJCAI_ , 2018, pp. 684–690.
  * [78] Q. Hou, M.-M. Cheng, X. Hu, A. Borji, Z. Tu, and P. H. Torr, “Deeply supervised salient object detection with short connections,” in _CVPR_ , 2017, pp. 5300–5309.
  * [79] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _ICML_ , 2015.