# UniST: Towards Unifying Saliency Transformer for Video Saliency Prediction
and Detection

Junwen Xiong1, Peng Zhang1,2 Chuanyue Li1,
Wei Huang3, Yufei Zha1,2, Tao You1 Corresponding author.

###### Abstract

Video saliency prediction and detection are thriving research domains that
enable computers to simulate the distribution of visual attention akin to how
humans perceiving dynamic scenes. While many approaches have crafted task-
specific training paradigms for either video saliency prediction or video
salient object detection tasks, few attention has been devoted to devising a
generalized saliency modeling framework that seamlessly bridges both these
distinct tasks. In this study, we introduce the Unified Saliency Transformer
(UniST) framework, which comprehensively utilizes the essential attributes of
video saliency prediction and video salient object detection. In addition to
extracting representations of frame sequences, a saliency-aware transformer is
designed to learn the spatio-temporal representations at progressively
increased resolutions, while incorporating effective cross-scale saliency
information to produce a robust representation. Furthermore, a task-specific
decoder is proposed to perform the final prediction for each task. To the best
of our knowledge, this is the first work that explores designing a transformer
structure for both saliency modeling tasks. Convincible experiments
demonstrate that the proposed UniST achieves superior performance across seven
challenging benchmarks for two tasks, and significantly outperforms the other
state-of-the-art methods.

## Introduction

As the continuous emergence of massive dynamic data propels deep learning
further towards human vision, a growing attention of research has focused on
video saliency prediction (VSP) (Bak et al. 2018; Wang et al. 2018a; Tsiami,
Koutras, and Maragos 2020) and video salient object detection (VSOD) (Mei et
al. 2021; Zhang et al. 2021; Liu et al. 2022a), which are regarded as the
fundamental tasks in computer vision. Both the tasks essentially aim to
simulate the visual attention distribution of humans perceiving dynamic scenes
by modeling spatio-temporal cues in video content. However, the modeling
paradigms of prior works are tailored for specialized tasks, and thereby lack
the capacity for generalization to address broader tasks.

Figure 1: Comparisons of the traditional different modeling paradigm for VSP
and VSOD tasks, as well as our proposed unified saliency transformer
framework. The VSP and VSOD adopt video encoder and image encoder respectively
as feature extractors, followed by corresponding decoders. Differently, a
unified saliency transformer directly applies an image encoder for feature
processing, and follows a transformer structure for spatio-temporal modeling,
and finally uses different decoders for different tasks.

Previous VSP approaches have made impressive progress in predicting the most
salient visual regions within frame sequences based on video encoder and
decoder (Tsiami, Koutras, and Maragos 2020; Jain et al. 2021; Zhou et al.
2023; Xiong et al. 2023), as depicted in Figure 1(a). To capture spatio-
temporal information from video sequences, these methods uniformly rely on
video models pre-trained on extensive video datasets, e.g., the lightweight
S3D (Xie et al. 2018) and the effective Video Swin Transformer (Liu et al.
2022b). Subsequently, various kinds of decoding strategies have been
successively proposed for processing the obtained spatio-temporal features.
(Min and Corso 2019; Jain et al. 2021) proposed a 3D fully convolutional
decoder with U-Net-like structure to progressively concatenate features along
the temporal dimension. In addition, (Zhou et al. 2023) proposed a 3D
convolutional decoder based on the feature pyramid structure, to build high-
level semantic features at all scales.

Compared to VSP, the video dataset used in the current VSOD task is usually
limited-sized, making it difficult to converge the video encoder-based method
to the optimum (Le and Sugimoto 2017; Fan et al. 2019). As a result, many VSOD
approaches adopt an image encoder and decoder-based training paradigm, where
pre-training is initially performed on the image dataset, and then the pre-
trained weights are transferred to the video dataset to further improve the
model’s generalization capabilities (Gu et al. 2020; Zhang et al. 2021; Liu et
al. 2022a), as shown in the Figure 1(b). To extract the frame-wise features
from video sequences, (Gu et al. 2020; Zhang et al. 2021; Liu et al. 2022a)
employed networks of the ResNet family (He et al. 2016) and the MobileNet
family (Howard et al. 2017) as feature extractors. For the temporal cues
between image feature sequences, (Gu et al. 2020) proposed to use the modified
non-local blocks (Wang et al. 2018b) to obtain the temporal information
between feature sequences. For temporal features extraction in a different
way, (Zhang et al. 2021; Liu et al. 2022a) proposed to construct another
temporal branch from optical flow and fuse with spatial features.

Unfortunately, except for the above achievements on each independent saliency
task, there has not been much prior effort to bridge both tasks for the
construction of a generalized saliency modeling paradigm. Thus, several
questions naturally arise: 1) why it is difficult to unify modeling for video
saliency prediction and detection tasks? and 2) is it possible to build a
unified saliency model generalized to these two different tasks?

As an answer to the questions above, a novel Unifed Saliency Transformer model
(UniST) is proposed, which comprehensively utilizes the essential attributes
of video saliency prediction and video salient object detection tasks. UniST
composes of an image encoder, a saliency-aware transformer and a task-specific
decoder as shown in Figure 1(c), and the incorporated image encoder is to
obtain a generic representation for each image with video sequences. In
addition, a saliency-aware transformer is also introduced to model spatio-
temporal representations of image feature sequences by stacking multiple sal-
transformer blocks, as well as augmenting feature scale progressively.
Subsequently, a task-specific decoder is proposed to leverage the
transformer’s output to make the final prediction for each task. We train
UniST and achieve superior performance to well-established prior works for
both tasks. The main contributions in this work can be summarized as follows:

  * 1)

A novel unified saliency transformer UniST framework is proposed to
comprehensively intrinsically associate both video saliency prediction and
video salient object detection tasks.

  * 2)

An efficient saliency-aware transformer is designed to learn the spatio-
temporal representations at gradually increased resolutions, and incorporate
effective cross-scale saliency information in the meantime.

  * 3)

Convincible experiments have been conducted on different challenging
benchmarks across two tasks, which is able to demonstrate a superior
performance of the proposed UniST in comparison to the other state-of-the-art
works.

## Related Work

### Video Saliency Prediction

Deep learning has led to the emergence of numerous video saliency prediction
methods that focus on modeling continuous motion information across frames.
(Bak et al. 2017) presented a two-stream deep model based on element-wise or
convolutional fusion strategies to learn spatio-temporal features. (Jiang, Xu,
and Wang 2017) designed an object-to-motion CNN network to extract features
and two-layer ConvLSTM with a Bayesian dropout to learn dynamic saliency.
(Wang et al. 2018a) combined the ConvLSTM network with an attention mechanism
to improve training efficiency and performance. However, existing LSTM-based
methods fall short of effectively integrating spatial and temporal information
(Min and Corso 2019). For this challenge, different studies have tried to
incorporate 3D convolutional models or video transformer models into the VSP
task. (Min and Corso 2019) introduced a TASED-Net to leverage the S3D model
(Xie et al. 2018) to simultaneously handle spatial and temporal cues in VSP.
Since then, the training paradigm of 3D convolution has been widely used in
VSP task. (Jain et al. 2021) adopted a 3D encoder-decoder structure,
resembling U-Net, which allows the constant concatenation of decoding features
from various layers(with the corresponding encoder features in the temporal
dimension). (Ma et al. 2022) proposed a video swin transformer-based
framework, which can eliminate the reliance on pre-trained S3D model and
enhance the performance upper bound in VSP task. In a different way, the
proposed UniST directly applies an image encoder for feature processing and
follows a saliency-aware transformer for spatio-temporal modeling.

Figure 2: An overview of the proposed UniST. The visual encoder learns frame-
wise visual representations from the video clips. The multi-scale visual
features are then used as inputs to the saliency-aware transformer for global
spatio-temporal modeling, generating refined and scaled-up spatio-temporal
features for final prediction. c denotes the channel-wise concatenation.

### Video Salient Object Detection

In VSOD, since the content of each frame is highly correlated, it can be
considered whose purpose is to capture long-range feature information among
the adjacency frame. Traditional methods (Girshick et al. 2014; Liu and Han
2016) often rely on conventional heuristics drawn from the domain of image
salient object detection. Recent works (Le and Sugimoto 2017; Wang, Shen, and
Shao 2017) strive to acquire highly semantic representations and usually
perform spatial-temporal detection end-to-end.

By taking temporal information into consideration networks modeling, different
works have been proposed, e.g., ConvLSTM (Li et al. 2018), took optical-flows
as input (Li et al. 2019), or 3D convolution (Miao, Wei, and Yang 2020). But
in real-world scenarios, the temporal cost incurred by introducing three-
dimensional convolution operations is noteworthy, and incorporating optical
flow information may not fully align with the ideal concept of an end-to-end
network. More recently, attention-based mechanisms have gained traction for
refining pairwise relationships between regions across consecutive frames.
(Fan et al. 2019; Gu et al. 2020) proposed a visual-attention-consistent
module and a pyramid constrained self-attention block to better capture the
temporal dynamics cues, respectively. Despite these advancements, the existing
methods remain tailored to specific tasks. Therefore, this paper proposes a
unified framework to address VSOD and VSP tasks more comprehensively.

## Proposed Method

An overview of the proposed UniST is presented in Figure 2. To tackle the
challenges of video saliency prediction and video salient object detection,
the UniST is constructed based on the encoder-decoder architecture which
consists of a visual feature encoder, a saliency-aware transformer and a task-
specific decoder.

To elaborate, video clips are initially fed into the visual feature encoder,
yielding multi-level spatial features for each image. Then, a saliency-aware
transformer is introduced to intricately capture spatio-temporal
representations of image feature sequences. This is achieved through the
stacking of multiple effective sal-transformer stages, which increases the
scale of the feature maps progressively. Finally, a task-specific decoder is
devised to leverage the transformer’s output features, and facilitate the
output predictions for each individual task.

### Visual Feature Encoder

Let $X\in\mathbb{R}^{T\times H\times W\times 3}$ denote an RGB video clip of
length $T$. This is input to a 2D backbone network which produces frame-wise
feature maps. The backbone consists of 4 encoder stages, and outputs 4
hierarchical visual feature maps, illustrated in Figure 2(a). The generated
feature maps are denoted as $\\{f_{X_{i}}\\}\in\mathbb{R}^{T\times h_{i}\times
w_{i}\times C_{i}}$, where $(h_{i},w_{i})=(H,W)/2^{i+1},i=1,...,4$. In
practical implementation, we employ the off-the-shelf MViT(Fan et al. 2021) as
a visual encoder to encode the spatial information of image sequences, which
can also be replaced with other general-purpose encoders, e.g., PVT(Wang et
al. 2021), Swin(Liu et al. 2021).

$\displaystyle\\{f_{X_{1}},f_{X_{2}},f_{X_{3}},f_{X_{4}}\\}=VisualEncoder(X)$
(1)

### Saliency-aware Transformer

For each input video clip, the encoder generates feature maps at four distinct
scales. Nevertheless, the extracted multi-scale feature maps solely capture
information within the spatial domain of the image sequences, neglecting any
temporal domain modeling. Another issue to be noted is that the low spatial
resolution of the visual encoder’s output feature makes it unsuitable for VSP
and VSOD tasks. With these considerations, a saliency-aware transformer is
designed to conduct spatio-temporal modeling, as well as enhance the
resolution of the feature maps progressively.

As shown in Figure 2(b), there are four stages in the saliency-aware
transformer, and each one is a designed sal-transformer stage. The primary
stage of the saliency-aware transformer focus on learning the spatio-temporal
sal-attention of the feature with the lowest resolution, and the subsequent
three stages augment the spatial resolution of the feature maps, and calculate
spatio-temporal sal-attention at higher resolutions. Similarly, the semantic-
guided block and sal-transformer block are used together in the first stage,
but the up embedding block and sal-transformer block are used in the following
three stages.

#### Semantic-Guided Block

Considering the most substantial semantic cues encapsulated by high-level
features(Chang and Zhu 2021; Sun et al. 2022), a semantic-guided block is
introduced to effectively guide the saliency modeling process. Specifically,
as in Figure 2(d), $f_{X_{4}}$ is initially fed into a Conv3d-BN-ReLU block
denoted as $F_{S}$. It consists of a 3D convolutional layer with kernel size
of $3\times 3\times 3$, a 3D batch nomalization layer and a ReLU activation
function. The output of this block is a semantic feature $f_{X_{4}}^{S}$. This
semantic feature is processed by a linear projection layer $F_{P}$ to reduce
the channel dimension to 1, and generate a saliency feature map
$f_{X_{4}}^{P}$. Finally, the semantic and saliency feature maps are
concatenated along the channel dimension, and we use another Conv3d-BN-ReLU
block $F_{C}$ to adjust the channel numbers to the original dimension $C_{4}$
to obtain the combined feature $f_{X_{4}}^{C}\in\mathbb{R}^{T\times
h_{4}\times w_{4}\times C_{4}}$.

$\displaystyle
f_{X_{4}}^{C}=F_{C}(Cat(F_{P}(F_{S}(f_{X_{4}})),F_{S}(f_{X_{4}})))$ (2)

where $Cat(\cdot,\cdot)$ represents the concatenation operation. To feed the
sementic-guided feature into the subsequent sal-transformer block, we flatten
the feature in the spatio-temporal dimension to obtain a 2D feature-token
sequence $f_{X_{4}}^{C}\in\mathbb{R}^{Th_{4}w_{4}\times C_{4}}$.

#### Up Embedding

Since the transformer module typically operates on 2D feature-token sequences,
which may cause the original spatial layout of the feature map has been
corrupted. To reconstruct the spatial details within the feature map, an up
embedding block specifically has been designed for the sal-transformer block.
The input to each up embedding block comes from the 2D feature-token sequence,
which is generated by the previous sal-transformer stage. In the beginning,
the up embedding block reshapes the feature sequence
$f_{X_{i}}\in\mathbb{R}^{Th_{i}w_{i}\times C_{i}}$ into spatial-temporal
feature maps of dimensions $\mathbb{R}^{T\times h_{i}\times w_{i}\times
C_{i}}$. Subsequently, a bilinear interpolation is performed to amplify the
height and width of each spatial feature by $2\times$ along the temporal
dimension, and a Conv-BN-ReLU block $F_{Conv}$ is used to reduce the channel
dimension to $C_{i-1}$ at the same time. For the obtained features of size
$\mathbb{R}^{T\times h_{i-1}\times w_{i-1}\times C_{i-1}}$, they are fused
with the feature maps of previous hierarchical level $f_{X_{i-1}}$ via
addition operation.

$\displaystyle f_{X_{i-1}}=f_{X_{i-1}}+F_{Conv}(Interpolation(f_{X_{i}}))$ (3)

The fused features are reshaped back to feature sequence
$f_{X_{i-1}}\in\mathbb{R}^{Th_{i-1}w_{i-1}\times C_{i-1}}$ as an upsampled
token sequence.

#### Sal-Transformer Block

When the feature maps obtained by the semantic-guided or up embedding block
are input into the sal-transformer block, spatio-temporal feature modeling
starts. Inspired by incorporating multi-scale feature maps to improve model
performance (Li et al. 2019; Jain et al. 2021), sal-transformer block is
proposed which not only models the spatio-temporal domain of features via sal-
attention mechanism, but also integrates the multi-scale attention information
of the previous stage.

Notice that calculating global sal-attention from temporal high-resolution
features has a prohibitively large memory footprint. To alleviate this
problem, in sal-attention (see shown in Figure 2 (e)), a reduction operation
is performed on the size of query $Q$, key $K$, and value $V$ matrices for
attention computation, like prior works (Fan et al. 2021; Wang et al. 2021).
In detail, it utilizes the $Conv3dLN$ operation consisting of a 3D convolution
and a layer normalization for embedding extraction, and obtains $Q$, $K$ and
$V$ in varying dimensions by controlling the convolution’s kernel and stride
sizes. This strategy for dimensionality reduction considerably enhances the
memory and computational efficiency associated with global sal-attention
computation, which makes it feasible to be used by multiple consecutive sal-
transformer blocks.

For an input feature $f_{X_{i}}\in\mathbb{R}^{Th_{i}w_{i}\times C_{i}}$, the
sal-attention first transforms the 2D feature sequence into spatio-temporal
domain, and followed by the operation of $Conv3dLN_{Q}$, $Conv3dLN_{K}$ and
$Conv3dLN_{V}$ to $f_{X_{i}}$. Then, the linear projections
$W_{Q},W_{K},W_{V}\in\mathbb{R}^{C_{i}\times C_{i}}$ are applied to obtain
$Q$, $K$ and $V$ embeddings, respectively. The attention score matrix
$A_{X_{i}}$ of $X_{i}$ feature map can be calculated as:

$\displaystyle
Q_{X_{i}}=W_{Q}(Conv3dLN_{Q}(f_{X_{i}})),Q_{X_{i}}\in\mathbb{R}^{Th_{i}w_{i}\times
C_{i}}$ (4) $\displaystyle
K_{X_{i}}=W_{K}(Conv3dLN_{K}(f_{X_{i}})),K_{X_{i}}\in\mathbb{R}^{T\hat{h}\hat{w}\times
C_{i}}$ $\displaystyle
V_{X_{i}}=W_{V}(Conv3dLN_{V}(f_{X_{i}})),V_{X_{i}}\in\mathbb{R}^{T\hat{h}\hat{w}\times
C_{i}}$ $\displaystyle
A_{X_{i}}=\frac{Q_{X_{i}}(K_{X_{i}})^{T}}{\sqrt{C_{i}}},A_{X_{i}}\in\mathbb{R}^{Th_{i}w_{i}\times
T\hat{h}\hat{w}}$

To further utilize the cross-scale insights from different stages within the
saliency-aware transformer, a saliency transfer mechanism is introduced to
enhance the attention score$A_{X_{i}}$ before the softmax operation by
integrating attention scores from distinct transformer stages. The attention
score $A_{X_{i+1}}$ from the feature map $X_{i+1}$ is utilized to bolster the
attention score $A_{X_{i}}$ of the feature map $X_{i}$. It’s noteworthy that
the second dimension of attention scores across different stages maintains
consistent dimensions because of the size of the convolutional kernel
designed. Specifically, a reshape operation is performed to align the shape of
$A_{X_{i+1}}$ with $\mathbb{R}^{T\times h_{i+1}\times w_{i+1}\times
T\hat{h}\hat{w}}$, and followed by a $2\times$ bilinear interpolation in the
first two spatial dimensions, then finally flatten it to get the attention
matrix $M_{X_{i+1}}\in\mathbb{R}^{Th_{i}w_{i}\times T\hat{h}\hat{w}}$ with
same dimension as $A_{X_{i}}$. The obtained $M_{X_{i+1}}$ is fused with
$A_{X_{i}}$ via addition to obtain the fused attention score with a
convolution operation. Then, we perform a softmax operation on the score and a
dot-product operation with $V$, the final updated feature maps $f_{X_{i}}$
becomes,

$\displaystyle A_{X_{i}}=Conv(A_{X_{i}}+M_{X_{i+1}})$ (5) $\displaystyle
f_{X_{i}}=Softmax(A_{X_{i}})V_{i}+f_{X_{i}}$

Since each sal-transformer stage produces feature maps with varying scales, an
efficient strategy is also proposed to aggregate the multi-scale features. For
each level of features, we employ a 3D convolution and upsample operation. The
channel dimension of feature maps is initially adjusted by 3D convolution, and
then their spatial dimensions are upsampled to align with the dimensions of
the $f_{X_{1}}$ feature map. To obtain the aggregated feature
$f_{X}\in\mathbb{R}^{T\times h_{1}\times w_{1}\times C_{1}}$, the features at
each scale are concatenated along channel dimension and the number of channels
is reduced by 3D convolution.

### Task-Specific Decoder

#### Video Saliency Prediction

For the feature $f_{X}$, a combination of Conv3d-BN-ReLU block and
Conv3d-Sigmoid block is employed. The former compresses the temporal dimension
of the feature to 1, and the latter further reduces the channel dimension of
the feature to 1. The upsampling operation is performed to output the
prediction results $P_{VSP}$ by aligning the spatial dimension of ground-truth
at the meantime.

#### Video Salient Object Detection

As the VSOD task requires the output of $T$-frames, the temporal dimension
compression of the feature $f_{X}$ is not necessary. We directly apply a Conv-
Sigmoid block to reduce the number of channels of $f_{X}$. Like the VSP
decoder, the predictions $P_{VSOD}$ are upsampled to align the spatial
dimensions of the ground-truth.

## Experiments

Method | DHF1K | $\mathit{\textrm{DAVIS}}_{16}$
---|---|---
$\mathit{CC}$ $\uparrow$ |  $\mathit{SIM}$ $\uparrow$ | $\mathit{MAE}\downarrow$ | $S_{m}\uparrow$
UniST baseline | 0.521 | 0.410 | 0.025 | 0.880
UniST w/SAT | 0.532 | 0.417 | 0.022 | 0.891
UniST w/SAT+SGB | 0.536 | 0.419 | 0.019 | 0.898
UniST w/SAT+SGB+ST | 0.541 | 0.423 | 0.018 | 0.904
Performance $\Delta$ | +0.020 | +0.013 | -0.007 | +0.024

Table 1: Ablation Studies. The proposed UniST and its components yield
consistent improvement on different datasets and achieve clear overall
improvement on both tasks. $\downarrow$ means lower better and $\uparrow$
means higher better.

### Datasets

For convincible validation, three popular video datasets, DHF1K(Wang et al.
2018a), Hollywood-2(Marszalek, Laptev, and Schmid 2009) and UCF-
Sports(Rodriguez, Ahmed, and Shah 2008), are used in our experiment. DHF1K
contains 600 training videos, 100 validation videos and 300 testing videos
with a frame rate of 30 fps. The UniST model can only be evaluated on the
validation set of DHF1K due to unavailable annotations of the test set.
Hollywood2 contains 1707 videos extracted from 69 movies with 12 categorized
action classes, 823 videos are used for training and 884 for testing. UCF-
Sports contains 150 videos (103 for training, 47 for testing) collected from
broadcast TV channels, which cover 9 sports, such as diving, weightlifting,
and horse riding.

For VSOD, the proposed model is evaluated on four public benchmark datasets
including DAVIS16(Perazzi et al. 2016), FBMS(Ochs, Malik, and Brox 2013),
ViSal(Wang, Shen, and Shao 2015) and SegTrackV2(Li et al. 2013). DAVIS16 is a
frequently used dataset, which contains 50 videos with a total of 3455 high-
quality pixel-wise annotation frames. FBMS is a test dataset containing 59
videos with 720 sparsely annotated frames. ViSal is a dataset only used for
test containing 19 videos with 193 pixel-wise annotation frames. SegTrackV2 is
also a test dataset with 14 videos and 1,065 annotated frames.

Encoder | DHF1K | $\mathit{\textrm{DAVIS}}_{16}$
---|---|---
$\mathit{AUC}$-$J$ $\uparrow$ |  $\mathit{CC}$ $\uparrow$ |  $\mathit{SIM}$ $\uparrow$ | $\mathit{MAE}\downarrow$ | $S_{m}\uparrow$ | $F_{\beta}\uparrow$
Image MViT | 0.920 | 0.541 | 0.423 | 0.018 | 0.904 | 0.873
Video MViT | 0.921 | 0.547 | 0.428 | 0.069 | 0.729 | 0.590
Image Swin | 0.909 | 0.483 | 0.371 | 0.025 | 0.885 | 0.854
Video Swin | 0.911 | 0.510 | 0.384 | 0.077 | 0.708 | 0.546

Table 2: Performance comparison of using using different types and families of
transformer encoder structures in UniST on DHF1K and DAVIS16 datasets.

### Implementation Details

To facilitate implementation, the pre-trained MViT-small model(Fan et al.
2021) on ImageNet(Deng et al. 2009) is employed. The input images of the
network are all resized to $224\times 384$ for training and testing. We follow
the experimental settings of prior works (Jain et al. 2021; Liu et al. 2022a).
For the VSP task, we first pre-train the UniST model on the DHF1K dataset and
then fine-tune it on the Hollywood2 and UCF-Sports datasets. And as for the
VSOD task, we choose to pre-train the entire model on the image DUTS dataset
(Wang et al. 2017), and then fine-tune it on the DAVIS16 dataset.

All experimental training process chooses Adam as the optimizer with a
learning rate of $1e-4$. The computation platform is configured by two NVIDIA
GeForce RTX 4090 GPUs in a distributed fashion, using PyTorch. More
implementation details are in the appendix.

### Evaluation Metrics

For video saliency detection, we use AUC-Judd $\mathit{AUC}$-$J$, Similarity
Metric $\mathit{SIM}$, Linear Correlation Coefficient $\mathit{CC}$, and
Normalized Scanpath Saliency $\mathit{NSS}$, following existing work(Hu et al.
2023). For video salient object detection, we adopt three evaluation metrics
for comparison, including the mean absolute error $\mathit{MAE}$, F-measure
$maxF$(Achanta et al. 2009), and S-measure $\mathit{S_{m}}$(Fan et al. 2017).

Method | DHF1K | Hollywood2 | UCF-Sports
---|---|---|---
$\mathit{CC}\uparrow$ | $\mathit{NSS}\uparrow$ |  $\mathit{AUC}$-$J$ $\uparrow$ | $\mathit{SIM}\uparrow$ | $\mathit{CC}\uparrow$ | $\mathit{NSS}\uparrow$ |  $\mathit{AUC}$-$J$ $\uparrow$ | $\mathit{SIM}\uparrow$ | $\mathit{CC}\uparrow$ | $\mathit{NSS}\uparrow$ |  $\mathit{AUC}$-$J$ $\uparrow$ | $\mathit{SIM}\uparrow$
$\textrm{TASED-Net}_{\mathit{\rm ICCV}^{\prime}2019}$ | 0.440 | 2.541 | 0.898 | 0.351 | 0.646 | 3.302 | 0.918 | 0.507 | 0.582 | 2.920 | 0.899 | 0.469
$\textrm{UNIVSAL}_{\mathit{\rm ECCV}^{\prime}2020}$ | 0.431 | 2.435 | 0.900 | 0.344 | 0.673 | 3.901 | 0.934 | 0.542 | 0.644 | 3.381 | 0.918 | 0.523
$\textrm{ViNet}_{\mathit{\rm IROS}^{\prime}2020}$ | 0.460 | 2.557 | 0.900 | 0.352 | 0.693 | 3.730 | 0.930 | 0.550 | 0.673 | 3.620 | 0.924 | 0.522
$\textrm{VSFT}_{\mathit{\rm TCSVT}^{\prime}2021}$ | 0.462 | 2.583 | 0.901 | 0.360 | 0.703 | 3.916 | 0.936 | 0.577 | - | - | - | -
$\textrm{ECANet}_{\mathit{\rm NeuroComputing}^{\prime}2022}$ | - | - | - | - | 0.673 | 3.380 | 0.929 | 0.526 | 0.636 | 3.189 | 0.917 | 0.498
$\textrm{STSANet}_{\mathit{\rm TMM}^{\prime}2022}$ | - | - | - | - | 0.721 | 3.927 | 0.938 | 0.579 | 0.705 | 3.908 | 0.936 | 0.560
$\textrm{TinyHD-S}_{\mathit{\rm WACV}^{\prime}2023}$ | 0.492 | 2.873 | 0.907 | 0.388 | 0.690 | 3.815 | 0.935 | 0.561 | 0.624 | 3.280 | 0.918 | 0.510
$\textrm{TMFI-Net}_{\mathit{\rm TCSVT}^{\prime}2023}$ | 0.524 | 3.006 | 0.918 | 0.410 | 0.739 | 4.095 | 0.940 | 0.607 | 0.707 | 3.863 | 0.936 | 0.565
UniST(Ours) | 0.541 | 3.113 | 0.920 | 0.423 | 0.777 | 4.397 | 0.951 | 0.632 | 0.706 | 3.718 | 0.932 | 0.576

Table 3: Comparisons of our method with the other state-of-the-arts on VSP
datasets. Our UniST significantly outperforms the previous state-of-the-arts
by a large margin.

### Ablation Studies

To conduct an in-depth analysis of the proposed UniST framework, a range of
model baselines and variants are defined as outlined in Table 1. (i) ”UniST
baseline” denotes a strong saliency model of the proposed UniST framework. It
uses MViT-small encoder and task-specific decoders for VSP and VSOD tasks. It
also combines multi-scale features from the encoder to help boost performance.
(ii) ”UniST w/SAT” indicates adding the pure saliency-aware transformer upon
”UniST baseline”. The pure saliency-aware transformer refers to the
replacement of the semantic-guided block with a standard 3D convolution, while
omitting the saliency transfer operation. (iii) ”UniST w/SAT+SGB” indicates
adding the proposed semantic-guided block upon ”UniST w/SAT”. Similarly,
”UniST w/SAT+SGB+ST” denotes the full model after integrating the saliency
transfer mechanism.

To analyze the effectiveness of each part in this work, we investigate the
performance of the UniST baseline and its model variants on both DHF1K and
DAVIS16 datasets, as shown in Table 1. It can be observed that the SAT, SGB
and ST modules all achieve clear improvement. Specifically, as the core module
of the UniST framework, SAT significantly improves the VSP task by 0.011($CC$)
on DHF1K, and the VSOD task by 0.011($S_{m}$) on DAVIS16. Finally, the full
UniST model achieves remarkable performance gain compared to the UniST
baseline.

#### Effect of the Number of Sal-Transformer Stages

There are four stages in the proposed saliency-aware transformer as a default
configuration. Figure LABEL:fig_transformer_stages illustrates the impact of
varying the number of sal-transformer stages on task performance across the
DHF1K and DAVIS16 datasets. Notably, the most optimal performance is attained
when the number of sal-transformer stages is set to 4. These results indicate
that UniST needs to gradually fuse the feature of each scale and increase the
feature resolution.

Method | $\mathit{\textrm{DAVIS}}_{16}$ | FBMS | ViSal | SegV2
---|---|---|---|---
$\mathit{MAE}\downarrow$ | $S_{m}\uparrow$ | $maxF\uparrow$ | $\mathit{MAE}\downarrow$ | $S_{m}\uparrow$ | $maxF\uparrow$ | $\mathit{MAE}\downarrow$ | $S_{m}\uparrow$ | $maxF\uparrow$ | $\mathit{MAE}\downarrow$ | $S_{m}\uparrow$ | $maxF\uparrow$
$\textrm{SSAV}_{\mathit{\rm CVPR}^{\prime}2019}$ | 0.028 | 0.893 | 0.861 | 0.040 | 0.879 | 0.865 | 0.020 | 0.943 | 0.939 | 0.023 | 0.851 | 0.801
$\textrm{CAS}_{\mathit{\rm TNNLS}^{\prime}2020}$ | 0.032 | 0.873 | 0.860 | 0.056 | 0.856 | 0.863 | - | - | - | 0.029 | 0.820 | 0.847
$\textrm{PCSA}_{\mathit{\rm AAAI}^{\prime}2020}$ | 0.022 | 0.902 | 0.880 | 0.040 | 0.868 | 0.837 | 0.017 | 0.946 | 0.940 | 0.025 | 0.865 | 0.810
$\textrm{FSNet}_{\mathit{\rm ICCV}^{\prime}2021}$ | 0.020 | 0.920 | 0.907 | 0.041 | 0.890 | 0.888 | - | - | - | 0.023 | 0.870 | 0.772
$\textrm{ReuseVOS}_{\mathit{\rm CVPR}^{\prime}2021}$ | 0.019 | 0.883 | 0.865 | 0.027 | 0.888 | 0.884 | 0.020 | 0.928 | 0.933 | 0.025 | 0.844 | 0.832
$\textrm{TransVOS}_{\mathit{\rm PrePrint}^{\prime}2021}$ | 0.018 | 0.885 | 0.869 | 0.038 | 0.867 | 0.886 | 0.021 | 0.917 | 0.928 | 0.024 | 0.816 | 0.800
$\textrm{UFO}_{\mathit{\rm TMM}^{\prime}2023}$ | 0.036 | 0.864 | 0.828 | 0.028 | 0.894 | 0.890 | 0.011 | 0.953 | 0.940 | 0.022 | 0.892 | 0.863
UniST(Ours) | 0.018 | 0.904 | 0.873 | 0.027 | 0.902 | 0.884 | 0.011 | 0.952 | 0.952 | 0.017 | 0.897 | 0.854

Table 4: Comparisons of our method with the other state-of-the-arts on VSOD
datasets. Our UniST outperforms the previous state-of-the-arts on most of the
metrics on these four datasets.

#### Image Encoder vs. Video Encoder

A further performance comparison is conducted using image and video encoders
from two transformer families. As shown in Table 2, there is minimal variation
in the model’s performance when employing either the image or video encoder on
the DHF1K dataset. Conversely, on the DAVIS16 dataset, the model employing the
image encoder surpasses its video encoder-based counterpart by a significant
margin. Due to the limited size of the DAVIS16 dataset, it is difficult to
converge the model to an optimal state by training on this dataset directly.
Therefore, the image MViT model with the best performance on both datasets is
chosen as the default encoder in our work.

Figure 4: Comparing the visualization results of attention scores in different
sal-transformer stages with and without the saliency transfer mechanism.

#### Further Analysis of Saliency Transfer

Figure 4 shows the visualized results of the attention scores in various sal-
transformer stages with/without the saliency transfer mechanism. Compared to
configurations without saliency transfer, the gradual integration of the
attention score significantly helps the model learn a more discriminative
feature, thus resulting in better visualization results.

### Comparision with State-of-the-arts

Figure 5: Qualitative results of our method compared with other state-of-the-
art methods on VSP and VSOD tasks. GT means the ground-truth.

#### Video Saliency Prediction

The proposed UniST is compared with recent state-of-the-art works on three
video saliency datasets as shown in Table 3. Experimental results in the table
highlight the superiority of the proposed unified scheme, as it outperforms
the other comparable works on almost all datasets and metrics. Notably, UniST
significantly surpasses the previous top-performing methods, such as TMFI-Net
(Zhou et al. 2023) and VSFT (Ma et al. 2022). Compared to TMFI-Net, UniST
improves the average $CC$ and $SIM$ performance on the DHF1K and Hollywood2
datasets by 4.2% and 3.1%, respectively, and becomes the new state-of-the-art
on both benchmarks. Moreover, on the UCF-Sports dataset, our method
demonstrates competitive performance comparable to TMFI-Net. Figure 5(a) shows
that the prediction results of the proposed work are more likely as the
ground-truth in comparison to other works. These significant improvements
indicate that our saliency-aware transformer is well suited for the spatio-
temporal modeling required in video saliency prediction.

#### Video Saliency Object Detection

Table 4 shows a comparison of the proposed UniST method against existing
state-of-the-art, including UFO (Su et al. 2023), TransVOS (Mei et al. 2021)
and ReuseVOS (Park et al. 2021), on four video datasets. The UniST is able to
achieve the best performance across a majority of metrics for all four
datasets. As opposed to the UFO model, which introduces the additional COCO-
SEG dataset(Lin et al. 2014), the UniST simply uses the image saliency dataset
DUTS (Wang et al. 2017) for pre-training to achieve comparable performance.
Compared to TransVOS (Mei et al. 2021), the UniST achieves $2.1\%$ and $4\%$
$S_{m}$ improvement on the two major datasets, DAVIS16 and FBMS, respectively.
These results clearly show that our UniST can successfully utilize a
generalized framework to solve the video salient object detection problem.
Figure 5(b) shows some qualitative results which indicate that our method is
closer to the ground-truth. More visualization results can be found in the
supplementary.

## Conclusion

In this paper, we propose a novel unified saliency transformer framework,
UniST, to unify the modeling paradigms for video saliency prediction and video
saliency object detection tasks. By capturing the spatial features of video
frame sequences through a visual feature encoder, the subsequent saliency-
aware transformer not only helps the model to capture the spatio-temporal
information in the sequence of image features but also progressively increases
the scale of the features for saliency prediction. Finally, a task-specific
decoder is devised to leverage the transformer’s output features, and
facilitate the output predictions for each individual task. Extensive
experiments demonstrated the effectiveness of the proposed method and also
showed its significantly better performance on seven popular benchmarks
compared to the previous state-of-the-art methods.

#### Limitations

Although our model has good performance in saliency prediction, the
improvements on detection are not such significant. We conjecture that this is
due to the lack of temporal information in the image dataset used for VSOD
pre-training, making it difficult for the saliency-aware transformer to
provide spatio-temporal modeling. We believe that the UniST can be further
improved after adding more video datasets.

#### Acknowledgments

This work is supported by the National Natural Science Foundation of China
(No.61971352, No.62271239), Ningbo Natural Science Foundation (No.2021J048,
No.2021J049), Jiangxi Double Thousand Plan (No.JXSQ2023201022), Fundamental
Research Funds for the Central Universities (No.D5000220190), Innovative
Research Foundation of Ship General Performance (No.25522108).

## References

  * Achanta et al. (2009) Achanta, R.; Hemami, S.; Estrada, F.; and Susstrunk, S. 2009.  Frequency-tuned salient region detection.  In _2009 IEEE conference on computer vision and pattern recognition_ , 1597–1604. IEEE.
  * Bak et al. (2017) Bak, C.; Kocak, A.; Erdem, E.; and Erdem, A. 2017.  Spatio-temporal saliency networks for dynamic saliency prediction.  _IEEE Transactions on Multimedia_ , 20(7): 1688–1698.
  * Bak et al. (2018) Bak, C.; KOÇAK, A.; ERDEM, M.; and Erdem, İ. 2018.  Spatio-Temporal Saliency Networks for Dynamic Saliency Prediction.  _IEEE TRANSACTIONS ON MULTIMEDIA_ , 20(7).
  * Chang and Zhu (2021) Chang, Q.; and Zhu, S. 2021.  Temporal-spatial feature pyramid for video saliency detection.  _arXiv preprint arXiv:2105.04213_.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Fan et al. (2017) Fan, D.-P.; Cheng, M.-M.; Liu, Y.; Li, T.; and Borji, A. 2017.  Structure-measure: A new way to evaluate foreground maps.  In _Proceedings of the IEEE international conference on computer vision_ , 4548–4557.
  * Fan et al. (2019) Fan, D.-P.; Wang, W.; Cheng, M.-M.; and Shen, J. 2019.  Shifting more attention to video salient object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 8554–8564.
  * Fan et al. (2021) Fan, H.; Xiong, B.; Mangalam, K.; Li, Y.; Yan, Z.; Malik, J.; and Feichtenhofer, C. 2021.  Multiscale vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 6824–6835.
  * Girshick et al. (2014) Girshick, R.; Donahue, J.; Darrell, T.; and Malik, J. 2014.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 580–587.
  * Gu et al. (2020) Gu, Y.; Wang, L.; Wang, Z.; Liu, Y.; Cheng, M.-M.; and Lu, S.-P. 2020.  Pyramid constrained self-attention network for fast video salient object detection.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, 10869–10876.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Howard et al. (2017) Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_.
  * Hu et al. (2023) Hu, F.; Palazzo, S.; Salanitri, F. P.; Bellitto, G.; Moradi, M.; Spampinato, C.; and McGuinness, K. 2023.  TinyHD: Efficient Video Saliency Prediction with Heterogeneous Decoders using Hierarchical Maps Distillation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2051–2060.
  * Jain et al. (2021) Jain, S.; Yarlagadda, P.; Jyoti, S.; Karthik, S.; Subramanian, R.; and Gandhi, V. 2021.  Vinet: Pushing the limits of visual modality for audio-visual saliency prediction.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 3520–3527. IEEE.
  * Jiang, Xu, and Wang (2017) Jiang, L.; Xu, M.; and Wang, Z. 2017.  Predicting video saliency with object-to-motion CNN and two-layer convolutional LSTM.  _arXiv preprint arXiv:1709.06316_.
  * Le and Sugimoto (2017) Le, T.-N.; and Sugimoto, A. 2017.  Deeply Supervised 3D Recurrent FCN for Salient Object Detection in Videos.  In _BMVC_ , volume 1, 3.
  * Li et al. (2013) Li, F.; Kim, T.; Humayun, A.; Tsai, D.; and Rehg, J. M. 2013.  Video segmentation by tracking many figure-ground segments.  In _Proceedings of the IEEE international conference on computer vision_ , 2192–2199.
  * Li et al. (2018) Li, G.; Xie, Y.; Wei, T.; Wang, K.; and Lin, L. 2018.  Flow guided recurrent neural encoder for video salient object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3243–3252.
  * Li et al. (2019) Li, H.; Chen, G.; Li, G.; and Yu, Y. 2019.  Motion guided attention for video salient object detection.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 7274–7283.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , 740–755. Springer.
  * Liu et al. (2022a) Liu, J.; Wang, J.; Wang, W.; and Su, Y. 2022a.  DS-Net: Dynamic spatiotemporal network for video salient object detection.  _Digital Signal Processing_ , 130: 103700.
  * Liu and Han (2016) Liu, N.; and Han, J. 2016.  Dhsnet: Deep hierarchical saliency network for salient object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 678–686.
  * Liu et al. (2021) Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021\.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 10012–10022.
  * Liu et al. (2022b) Liu, Z.; Ning, J.; Cao, Y.; Wei, Y.; Zhang, Z.; Lin, S.; and Hu, H. 2022b.  Video swin transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3202–3211.
  * Ma et al. (2022) Ma, C.; Sun, H.; Rao, Y.; Zhou, J.; and Lu, J. 2022.  Video saliency forecasting transformer.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 32(10): 6850–6862.
  * Marszalek, Laptev, and Schmid (2009) Marszalek, M.; Laptev, I.; and Schmid, C. 2009.  Actions in context.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2929–2936. IEEE.
  * Mei et al. (2021) Mei, J.; Wang, M.; Lin, Y.; Yuan, Y.; and Liu, Y. 2021.  Transvos: Video object segmentation with transformers.  _arXiv preprint arXiv:2106.00588_.
  * Miao, Wei, and Yang (2020) Miao, J.; Wei, Y.; and Yang, Y. 2020.  Memory aggregation networks for efficient interactive video object segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10366–10375.
  * Min and Corso (2019) Min, K.; and Corso, J. J. 2019.  Tased-net: Temporally-aggregating spatial encoder-decoder network for video saliency detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2394–2403.
  * Ochs, Malik, and Brox (2013) Ochs, P.; Malik, J.; and Brox, T. 2013.  Segmentation of moving objects by long term video analysis.  _IEEE transactions on pattern analysis and machine intelligence_ , 36(6): 1187–1200.
  * Park et al. (2021) Park, H.; Yoo, J.; Jeong, S.; Venkatesh, G.; and Kwak, N. 2021.  Learning dynamic network using a reuse gate function in semi-supervised video object segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 8405–8414.
  * Perazzi et al. (2016) Perazzi, F.; Pont-Tuset, J.; McWilliams, B.; Van Gool, L.; Gross, M.; and Sorkine-Hornung, A. 2016.  A benchmark dataset and evaluation methodology for video object segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 724–732.
  * Rodriguez, Ahmed, and Shah (2008) Rodriguez, M. D.; Ahmed, J.; and Shah, M. 2008.  Action mach a spatio-temporal maximum average correlation height filter for action recognition.  In _2008 IEEE conference on computer vision and pattern recognition_ , 1–8. IEEE.
  * Su et al. (2023) Su, Y.; Deng, J.; Sun, R.; Lin, G.; Su, H.; and Wu, Q. 2023.  A unified transformer framework for group-based segmentation: Co-segmentation, co-saliency detection and video salient object detection.  _IEEE Transactions on Multimedia_.
  * Sun et al. (2022) Sun, F.; Zhou, W.; Ye, L.; and Yu, L. 2022.  Hierarchical decoding network based on swin transformer for detecting salient objects in RGB-T images.  _IEEE Signal Processing Letters_ , 29: 1714–1718.
  * Tsiami, Koutras, and Maragos (2020) Tsiami, A.; Koutras, P.; and Maragos, P. 2020.  Stavis: Spatio-temporal audiovisual saliency network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4766–4776.
  * Wang et al. (2017) Wang, L.; Lu, H.; Wang, Y.; Feng, M.; Wang, D.; Yin, B.; and Ruan, X. 2017.  Learning to detect salient objects with image-level supervision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 136–145.
  * Wang et al. (2018a) Wang, W.; Shen, J.; Guo, F.; Cheng, M.-M.; and Borji, A. 2018a.  Revisiting video saliency: A large-scale benchmark and a new model.  In _Proceedings of the IEEE Conference on computer vision and pattern recognition_ , 4894–4903.
  * Wang, Shen, and Shao (2015) Wang, W.; Shen, J.; and Shao, L. 2015.  Consistent video saliency using local gradient flow optimization and global refinement.  _IEEE Transactions on Image Processing_ , 24(11): 4185–4196.
  * Wang, Shen, and Shao (2017) Wang, W.; Shen, J.; and Shao, L. 2017.  Video salient object detection via fully convolutional networks.  _IEEE Transactions on Image Processing_ , 27(1): 38–49.
  * Wang et al. (2021) Wang, W.; Xie, E.; Li, X.; Fan, D.-P.; Song, K.; Liang, D.; Lu, T.; Luo, P.; and Shao, L. 2021.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 568–578.
  * Wang et al. (2018b) Wang, X.; Girshick, R.; Gupta, A.; and He, K. 2018b.  Non-local neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7794–7803.
  * Xie et al. (2018) Xie, S.; Sun, C.; Huang, J.; Tu, Z.; and Murphy, K. 2018.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In _Proceedings of the European conference on computer vision (ECCV)_ , 305–321.
  * Xiong et al. (2023) Xiong, J.; Wang, G.; Zhang, P.; Huang, W.; Zha, Y.; and Zhai, G. 2023.  CASP-Net: Rethinking Video Saliency Prediction From an Audio-Visual Consistency Perceptual Perspective.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 6441–6450.
  * Zhang et al. (2021) Zhang, M.; Liu, J.; Wang, Y.; Piao, Y.; Yao, S.; Ji, W.; Li, J.; Lu, H.; and Luo, Z. 2021.  Dynamic context-sensitive filtering network for video salient object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 1553–1563.
  * Zhou et al. (2023) Zhou, X.; Wu, S.; Shi, R.; Zheng, B.; Wang, S.; Yin, H.; Zhang, J.; and Yan, C. 2023\.  Transformer-based Multi-scale Feature Integration Network for Video Saliency Prediction.  _IEEE Transactions on Circuits and Systems for Video Technology_.

##  Appendix A More Implementation Details

To facilitate implementation, the pre-trained MViT-small model(Fan et al.
2021) on ImageNet(Deng et al. 2009) is employed. The input images of the
network are all resized to $224\times 384$ for training and testing. (1): On
video saliency detection, we set batch size to 6 and set $T$ to 16 and use the
annotations in the middle of the $T$ frames for supervision. Like prior works
(Jain et al. 2021; Ma et al. 2022), to evaluate the Hollywood2 and UCF-Sports
benchmarks, the whole model needs to be pre-trained on the DHF1K dataset
first, and then fine-tuned on the Hollywood2 and UCF-Sports training sets.
(2): On video salient object detection, we set batch size to 16 and set $T$ to
5. We follow the common practice for most methods(Zhang et al. 2021; Liu et
al. 2022a), which first pre-trained their models on the static saliency
dataset, such as DUTS(Wang et al. 2017). And we then proceed to fine-tune the
UniST model on the DAVIS16 training set. All experimental training process
chooses Adam as the optimizer with a learning rate of $1e-4$. The computation
platform is configured by two NVIDIA GeForce RTX 4090 GPUs in a distributed
fashion, using PyTorch.

### Saliency-aware Transformer

There are four sal-transformer stages in saliency-aware transformer. Each
stage has one sal-transformer block, where the number of multi-heads is set to
2 by default. The convolution kernel $K$ and stride size $S$ of the
$Conv3dLN_{Q}$ operation are constantly 3 and 1, respectively, while both are
of size $\\{2,4,8,16\\}$ for the $Conv3dLN_{K}$ and $Conv3dLN_{V}$ operations,
respectively, see table below for specific configuration parameters.

| $Conv3dLN_{Q}$ | $Conv3dLN_{K}$ | $Conv3dLN_{V}$
---|---|---|---
Stage1 |  | $K=3\times 3\times 3$
---
$S=1\times 1\times 1$
$P=1\times 1\times 1$
| $K=2\times 2\times 2$
---
$S=2\times 2\times 2$
$P=0\times 0\times 0$
| $K=2\times 2\times 2$
---
$S=2\times 2\times 2$
$P=0\times 0\times 0$
Stage2 |  | $K=3\times 3\times 3$
---
$S=1\times 1\times 1$
$P=1\times 1\times 1$
| $K=4\times 4\times 4$
---
$S=4\times 4\times 4$
$P=0\times 0\times 0$
| $K=4\times 4\times 4$
---
$S=4\times 4\times 4$
$P=0\times 0\times 0$
Stage3 |  | $K=3\times 3\times 3$
---
$S=1\times 1\times 1$
$P=1\times 1\times 1$
| $K=8\times 8\times 8$
---
$S=8\times 8\times 8$
$P=0\times 0\times 0$
| $K=8\times 8\times 8$
---
$S=8\times 8\times 8$
$P=0\times 0\times 0$
Stage4 |  | $K=3\times 3\times 3$
---
$S=1\times 1\times 1$
$P=1\times 1\times 1$
| $K=16\times 16\times 16$
---
$S=16\times 16\times 16$
$P=0\times 0\times 0$
| $K=16\times 16\times 16$
---
$S=16\times 16\times 16$
$P=0\times 0\times 0$

Figure 6: Qualitative results of our method compared with other state-of-the-
art methods on Video Saliency Prediction task.  Figure 7: Qualitative results
of our method compared with other state-of-the-art methods on Video Salient
Object Detection task.

### Training Objectives of Video Saliency Detection

We refer to the training paradigm of multiple loss functions in (Tsiami,
Koutras, and Maragos 2020; Chang and Zhu 2021), which contains: Kullback-
Leibler ($KL$) divergence, Linear Correlation Coefficient ($CC$) and
Similarity Metric ($SIM$). Assuming that the predicted saliency map is
$P_{VSP}\in[0,1]$, the labeled binary fixation map is $G_{fix}\in\\{0,1\\}$,
and the dense saliency map generated by the fixation map is $G_{den}\in[0,1]$,
then $L_{KL}$, $L_{CC}$, and $L_{NSS}$ are employed to signify three different
loss functions, respectively. The first is the $KL$ loss between the predicted
map $P_{VSP}$ and the dense map $G_{den}$:

$L_{KL}(P_{VSP},S_{den})=\sum_{x}G_{den}(x)ln\frac{G_{den}(x)}{P_{VSP}(x)}$
(6)

where $x$ represents the spatial domain of a saliency map. The second loss
function is based on the $CC$ that has been widely used in saliency
evaluation, and used to measure the linear relationship between the predicted
saliency map $P_{VSP}$ and the dense map $G_{den}$:

$L_{CC}(P_{VSP},G_{den})=-\frac{cov(P_{VSP},G_{den})}{\rho(P_{VSP})\rho(G_{den})}$
(7)

where $cov(\cdot)$ and $\rho(\cdot)$ represent the covariance and the standard
deviation respectively. The last one is derived from the $SIM$, which can
measure the similarity between two distributions:

$\displaystyle
L_{SIM}(P_{VSP},G_{den})=\sum_{x}min\\{\zeta(P_{VSP}(x)),\zeta(G_{den}(x))\\}$
(8)

where $\zeta$ represents the normalization operation. The weighted summation
of the above $KL$, $CC$ and $SIM$ is taken to represent the final loss
function:

$L_{VSP}=L_{KL}+\lambda_{1}L_{CC}+\lambda_{2}L_{SIM}$ (9)

where $\lambda_{1}$, $\lambda_{2}$ are the weights of $CC$ and $SIM$,
respectively. We set $\lambda_{1}=\lambda_{2}=-0.1$ in our implementation.

### Training Objectives of Video Salient Object Detection

We refer to the training paradigm in (Gu et al. 2020) that uses the binary
cross entropy loss function. We denote our VSOD prediction as $P_{VSOD}$, and
the Ground Truth of the saliency map is $G_{VSOD}$. Then binary cross entropy
loss $L_{VSOD}$ can be defined as

$\displaystyle L_{VSOD}(P_{VSOD},G_{VSOD})=$
$\displaystyle-\frac{1}{N}\sum_{i=1}^{N}[g_{i}log(p_{i})$ (10)
$\displaystyle+(1-g_{i})log(1-p_{i})]$

where $N$ is the number of pixels.

##  Appendix B More Qualitative Results

In Figures 6 and 7, we show the prediction results of the proposed UniST
model, as well as the predictions of other SOTA methods. The UniST produces
significantly better results than the other methods, being closer to the
ground-truth on both tasks.