Previous video object segmentation approaches mainly focus on using simplex solutions between appearance and motion, limiting feature collaboration efficiency among and across these two cues.

In this work, we study a novel and efficient full-duplex strategy network () to address this issue, by considering a better mutual restraint scheme between motion and appearance in exploiting the cross-modal features from the fusion and decoding stage. Specifically, we introduce the
relational cross-attention module (RCAM) to achieve bidirectional message propagation across embedding sub-spaces. To improve the model's robustness and update the inconsistent features from the spatial-temporal embeddings, we adopt the bidirectional purification module (BPM) after the RCAM.

Extensive experiments on five popular benchmarks show that our  is
robust to various challenging scenarios (, motion blur, occlusion) and
achieves favourable performance against existing cutting-edges both in the video object segmentation and video salient object detection tasks.
The project is publicly available at:

Video Object Segmentation, Salient Object Detection, Visual Attention

§ INTRODUCTION

Comparison between three strategies for embedding appearance and motion patterns before the fusion and decoding stage.

(a) Direction-independent strategy [44] without information transmission,

(b) Simplex strategy [141] with only unidirectional information transmission, , using motion guides appearance or vice versa, and

(c) our full-duplex strategy
with simultaneously bidirectional information transmission.

This paper mainly focuses on discussing directional modelling (b & c) in the deep learning era.

Over the past three years, social platforms have accumulated a large number of short videos. Analyzing these videos efficiently and intelligently has become a challenging issue today.

Video object segmentation (VOS) [115, 16, 118, 41] is a fundamental technique to address this issue, whose purpose is to delineate pixel-level moving object[We use `foreground object' & `target object' interchangeably.] masks in each frame.
Besides video analysis, many other applications have also benefited from VOS, such as  robotic manipulation [1], autonomous cars [70], video editing [43], action segmentation [103], optical flow estimation [24], medical diagnosis [45], interactive segmentation [14, 72, 37, 131, 19], URVOS [87], and video captioning [77].

Recently, we have witnessed rapid development in addressing video object understanding by exploiting the relationships between the frames' appearance-aware [132, 13, 57, 84, 101] and motion-aware [42, 95, 91].

Unfortunately, short-term dependency prediction [42, 95] generates unreliable estimations and suffers the common ordeals [39] (, noise, deformation, and diffusion). In addition, the capability of appearance-based modelling like recurrent neural network [97, 71, 30] (RNN) is severely hindered by blurred foregrounds or cluttered backgrounds [18].

Those conflicts are prone to accumulating inaccuracies and the propagation of spatial-temporal embeddings, which cause the short-term feature drifting problem [129].

As shown in fig:teaser_figure (a), the direction-independent strategy [48, 44, 21, 97, 122] is the earliest solution by encoding the appearance and motion features individually and fuse them directly.

However, this intuitive way will implicitly cause feature conflicts since the motion- and appearance-aware features are derived from two distinctive modalities, which is extracted from separate branches.

An alternative way is to integrate them in a guided manner. As illustrated in  fig:teaser_figure (b), several recent methods opt for the simplex strategy [141, 100, 65, 74, 61, 39, 80], which is either appearance-based or motion-guided.

Although these two strategies have achieved promising results, they both fail to consider the mutual restraint between the appearance and motion features that both guide human visual attention allocation during dynamic observation, according to previous studies in cognitive psychology [50, 119, 99] and computer vision [107, 44].

Intuitively, appearance and motion characteristics should be homogeneous to a certain degree for the same object within a short time. As seen in fig:front_figure_1, the foreground region of appearance and motion intrinsically share the correlative patterns about perceptions, including semantic structure, movement trends.

Nevertheless, misguided knowledge in the individual modality, , static shadow under the chassis and small car in the background, will produce inaccuracies during the feature propagation. Thus, it easily stains the result (see blue boxes).

Visual comparison between the simplex (, (a) appearance-refined motion and (b) motion-refined appearance) and our full-duplex strategy under our framework.
In contrast, our  offers a collaborative way to leverage the appearance and motion cues under the mutual restraint of full-duplex strategy, thus providing more accurate structure details and alleviating the short-term feature drifting issue [129].

To address these challenges, we introduce a novel modality transmission strategy (full-duplex [5]) between spatial- and temporal-aware, instead of embedding them individually.

The proposed strategy is the bidirectional attention scheme across motion and appearance cues, which explicitly incorporates the appearance and motion patterns in a unified framework as depicted in fig:teaser_figure (c).

As seen in fig:front_figure_1, our method visually performs better than the one with a simplex strategy (a & b).

To fully investigate the simplex and full-duplex strategies of our framework, we present the following contributions:

* We propose a unified framework Full-duplex Strategy Network () for robust video object segmentation, which makes full use of spatial-temporal representations.

* We adopt a bidirectional interaction module, dubbed the relational cross-attention module (RCAM), to extract discriminative features from the appearance and motion branches, which ensures mutual restraint between each other.

To improve the model robustness, we introduce a
bidirectional purification module (BPM), which is equipped with an interlaced decremental connection to update inconsistent features between the spatial-temporal embeddings automatically.

* We demonstrate that our  achieves favourable performance on five mainstream benchmarks, especially for our  ($N$=4, CRF) outperforms the SOTA U-VOS model (, MAT [141]) on the DAVIS$_{16}$ [82] leaderboard by a margin of 2.4% in terms of Mean-$\mathcal{F}$ score, with less training data (, Ours-13K vs. MAT-16K).

As an extension of our ICCV-2021 version [46], we incorporate more details to provide a better understanding of our novel framework as follow:

* To provide our community with a comprehensive study, we have made a lot of efforts to improve the presentations (, fig:teaser_figure, fig:front_figure_1, and Fig:performance1) and discussions (see sec:discussion) of our manuscript.

* We investigate the self-purification mode of BPM under our  (see fig:self_purf and sec:self_purf), the relation between RCAM and BPM (see sec:relation_RCAM_BPM), and the training effectiveness with less data (see sec:scale_of_dataset). The results further demonstrate the validity and rationality of our current design under various conditions.

* We provide more details to the conference version, such as backbone details (see sec:backbone_details), evaluation metrics (see sec:metric), prediction selection (see sec:prediction_selection), and post-processing techniques (see sec:crf_post_process).

* We further present more performances under different thresholds used in the final results (, PR curve in fig:pr_curve). Additional test results (, DAVSOD$_{19}$-Normal25, DAVSOD$_{19}$-Difficult20) on a recent challenging dataset still show that our framework is superior to existing SOTA models (see tab:davsod_benchmark).

§ RELATED WORKS

Depending on whether or not the first frame of ground truth is given, the VOS task can be divided into two scenarios, , semi-supervised [109] and unsupervised/zero-shot [71].

Some typical semi-supervised VOS models can be referred to [134, 85, 133, 53, 64, 88, 6, 128, 126, 12, 38, 25, 47, 121, 102, 20, 11, 81].

This paper studies the unsupervised setting [141, 140], leaving the semi-supervised setting as our future work.

§.§ Unsupervised VOS

Although there are many works addressing the VOS task in a semi-supervised manner, , by supposing an object mask annotation is given in the first frame, other researchers have attempted to address the more challenging unsupervised VOS (U-VOS) problem.

Early U-VOS models resort to low-level handcrafted features for heuristic segmentation inference, such as long sparse point trajectories [10, 75, 31, 90, 111], object proposals [58, 59, 69, 83], saliency priors [108, 27, 106], optical flow [100], or superpixels [32, 33, 123].

These traditional models have limited generalizability and thus low accuracy in highly dynamic and complex scenarios due to their lack of semantic information and high-level content understanding. Recently, RNN-based models [93, 113, 126, 138, 97, 4] have become popular due to their better capability of capturing long-term dependencies and their use of deep learning.

In this case, U-VOS is formulated as a recurrent modelling issue over time, where spatial features are jointly exploited with long-term temporal context.

The architecture of our  for video object segmentation. The Relational Cross-Attention Module (RCAM) abstracts more discriminative representations between the motion and appearance cues using the full-duplex strategy.
Then four Bidirectional Purification Modules (BPM) are stacked to further re-calibrate inconsistencies between the motion and appearance features. Finally, we utilize the decoder to generate our prediction.

How to combine motion cues with appearance features is a long-standing problem in this field. To this end, Tokmakov  [96] proposed to use the motion patterns required from the video simply. However, their method cannot accurately segment objects between two similar consecutive frames since it relies heavily on the guidance of optical flow.

To resolve this, several works [97, 21, 92] have integrated the spatial and temporal features from the parallel network, which can be viewed as plain feature fusion from the independent spatial and temporal branch with an implicit modelling strategy.

Li  [62] proposed a multi-stage processing method to tackle U-VOS, which first utilizes a fixed appearance-based network to generate objectness and then feeds this into the motion-based bilateral estimator to segment the objects.

§.§ Attention-based VOS

The attention-based VOS task is closely related to U-VOS since it extracts attention-aware object(s) from a video clip.

Traditional methods [112, 142, 125, 40] first compute the single-frame saliency based on various hand-crafted static and motion features and then conduct spatial-temporal optimization to preserve coherency across consecutive frames.

Recent works [110, 55, 73] aim to learn a highly semantic representation and usually perform spatial-temporal detection end-to-end.

Many schemes have been proposed to employ deep networks that consider temporal information, such as ConvLSTM [93, 30, 60], take optical-flows/adjacent-frames as input [61, 110], 3D convolutional [55, 73], or directly exploit temporally concatenated deep features [56].

Besides, long-term influences are often taken into account and combined with deep learning. Li  [63] proposed a key-frame strategy to locate representative high-quality video frames of salient objects [7, 139] and diffused their saliency to ill-detected non-key frames.

Chen  [15] improved saliency detection by leveraging long-term spatial-temporal information, where high-quality “beyond-the-scope frames” are aligned with the current frames. Both types of information are fed to deep neural networks for classification.

Besides considering how to better leverage temporal information, other researchers have attempted to address different problems in video salient object detection (V-SOD), such as reducing the data labelling requirements [127], developing semi-supervised approaches [94], or investigating relative saliency [116].

Fan  [30] recently introduced a V-SOD model equipped with a saliency shift-aware ConvLSTM, together with an attention-consistent V-SOD dataset with high-quality annotations.

Zhao  [137] build a large-scale with scribble annotation for weakly supervised video salient object detection. They propose an appearance-motion fusion module to aggregate the spatial-temporal features attentively.

§ METHODOLOGY

§.§ Overview

Suppose that a video clip contains $T$ consecutive frames $\{\mathbf{A}^t\}_{t=1}^{T}$.

We first utilize optical flow field generator $\mathcal{H}$, , FlowNet 2.0 [42], to generate $T-1$ optical flow maps $\{\mathbf{M}^{t}\}_{t=1}^{T-1}$, which are all computed by two adjacent frames ($\mathbf{M}^t = \mathcal{H}[\mathbf{A}^{t},\mathbf{A}^{t+1}]$).

To ensure the inputs match, we discard the last frame in the pipeline.

Thus, the proposed pipeline takes both the appearance image $\{\mathbf{A}^{t}\}_{t=1}^{T-1}$ and its paired motion map $\{\mathbf{M}^{t}\}_{t=1}^{T-1}$ as the input.

First, $\mathbf{M}^{t}$ & $\mathbf{A}^{t}$ pairs at frame $t$[Here, we omit the superscript “$t$” for the convenient expression.] are fed to two independent ResNet-50 [36] branches (, motion and appearance blocks in Fig:overall_framework).

The appearance features $\{\mathcal{X}_k\}_{k=1}^{K}$ and motion features $\{\mathcal{Y}_k\}_{k=1}^{K}$ extracted from $K$ layers are then sent to the Relational Cross-Attention Modules (RCAMs), which allows the network to embed spatial-temporal cross-modal features.

Next, we employ the Bidirectional Purification Modules (BPMs) with $N$ cascaded units. BPMs focus on distilling representative carriers from fused features $\{\mathbf{F}_k^n\}^{N}_{n=1}$ and motion-based features $\{\mathbf{G}_k^n\}_{n=1}^{N}$.

Finally, the predictions (, $\mathbf{S}_M^t$ and $\mathbf{S}_A^t$) at frame $t$ are generated from two decoder blocks.

Illustration of our Relational Cross-Attention Module (RCAM) with a simplex (a & b) and full-duplex (c) strategy.

§.§ Relational Cross-Attention Module

As discussed in sec:introduction, a single-modality (, motion or appearance) guided stimulation may cause the model to make incorrect decisions.

To alleviate this, we design a cross-attention module (RCAM) via the channel-wise attention mechanism, which focuses on distilling out effectively squeezed cues from two modalities and then modulating each other.

As shown in Fig:framework_RCAM (c), the two inputs of RCAM are appearance features $\{\mathcal{X}_k\}_{k=1}^{K}$ and motion features $\{\mathcal{Y}_k\}_{k=1}^{K}$, which are obtained from the two different branches of the standard ResNet-50 [36].

Specifically, for each k-level, we first perform global average pooling (GAP) to generate channel-wise vectors $\mathcal{V}^{X}_{k}$ and $\mathcal{V}^{Y}_{k}$ from each $\mathcal{X}_k$ and $\mathcal{Y}_k$.

Next, two 1$\times$1 conv, , $\phi(x;\mathbf{W}_\phi)$ and $\theta(x;\mathbf{W}_\theta)$, with learnable parameters $\mathbf{W}_\phi$ and $\mathbf{W}_\theta$, generate two discriminated global descriptors.

The sigmoid function $\sigma[x]=e^{x}/(e^{x}+1),~x\in\mathbb{R}$ is then applied to convert the final descriptors into the interval [0, 1], , into the valid attention vector for channel weighting.

Then, we perform outer product $\otimes$ between $\mathcal{X}_k$ and $\sigma\left[\theta(\mathcal{V}_{k}^{Y};\mathbf{W}_\theta)\right]$ to generate a candidate feature $\mathcal{Q}_{k}^{X}$, and vice versa, as follows:

\begin{equation}
\mathcal{Q}_{k}^{X} = \mathcal{X}_k \otimes \sigma\left[\theta(\mathcal{V}_{k}^{Y};\mathbf{W}_\theta)\right],
\end{equation}

\begin{equation}
\mathcal{Q}_{k}^{Y} = \mathcal{Y}_k \otimes \sigma\left[\phi(\mathcal{V}_{k}^{X};\mathbf{W}_\phi)\right].
\end{equation}

Then, we combine $\mathcal{Q}_{k}^{X}$, $\mathcal{Q}_{k}^{Y}$, and lower-level fused feature $\mathcal{Z}_{k-1}$ for in-depth feature extraction.

With an element-wise addition operation $\oplus$, conducted in the corresponding k-th level block $\mathcal{B}_k[x]$ in the ResNet-50, we finally obtain the fused features $\mathcal{Z}_k$ that contain comprehensive spatial-temporal correlations:

\begin{equation}\label{equ:3}
\mathcal{Z}_k = \mathcal{B}_k \left[ \mathcal{Q}^{X}_{k} \oplus \mathcal{Q}^{Y}_{k} \oplus \mathcal{Z}_{k-1} \right],
\end{equation}

where $k \in \{1:K\}$ denotes different feature hierarchies in the backbone.

Note that $\mathcal{Z}_0$ denotes the zero tensor.

In our implementation, we use the top four feature pyramids, , $K=4$, suggested by [117, 135].

§.§ Bidirectional Purification Module

In addition to the RCAM described above, which integrates common cross-modality features, we further introduce the bidirectional purification module (BPM) to improve the model robustness.

Following the standard in action recognition [89] and saliency detection [120], our bidirectional purification phase comprises $N$ BPMs connected in a cascaded manner.

As shown in Fig. <ref>, we first employ the feature allocator $\psi_{\{F,G\}}(x;\mathbf{W}_\psi^{\{F,G\}})$ to unify the feature representations from the previous stage:

\begin{equation}\label{equ:allocator}
\mathbf{F}_{k}^{n} = \psi_{F}(\mathcal{Z}_k;\mathbf{W}_\psi^{F}),~
\mathbf{G}_{k}^{n} = \psi_{G}(\mathcal{Y}_k;\mathbf{W}_\psi^{G}),
\end{equation}

where $k \in \{1:K\}$ and $n \in \{1:N\}$ denote different feature hierarchies and number of BPM, respectively.

To be specific, $\psi_{\{F,G\}}(x;\mathbf{W}_{\psi}^{\{F,G\}})$ is composed of two 3$\times$3 conv, each with 32 filters to reduce the feature channels.

Note that the allocator is conducive to reduce the computational burden as well as facilitate various element-wise operations.

Here, we consider a bidirectional attention scheme (see Fig:framework_bpm (c)) that contains two simplex strategies (see Fig:framework_bpm (a & b)) in the BPM.

On the one hand, the motion features $\mathbf{G}_{k}^{n}$ contain temporal cues and can be used to enrich the fused features $\mathbf{F}_{k}^{n}$ by the concatenation operation.

On the other, the distractors in the motion feature $\mathbf{G}_{k}^{n}$ can be suppressed by multiplicating the fused features $\mathbf{F}_{k}^{n}$.

Besides acquiring robust feature representation, we introduce an efficient cross-modal fusion strategy in this scheme, which broadcasts high-level, semantically strong features to low-level, semantically weak features via interlaced decremental connection (IDC) with a top-down pathway [66].

Specifically, as the first part, the spatial-temporal feature combination branch (see Fig:framework_bpm (b)) is formulated as:

\begin{equation}\label{equ:5}
\mathbf{F}_{k}^{n+1} = \mathbf{F}_{k}^{n} \oplus \bigcup_{i=k}^{K} \left[ \mathbf{F}_{k}^{n},\mathcal{P} ( \mathbf{G}_{i}^{n}) \right],
\end{equation}

where $\mathcal{P}$ is an up-sampling operation followed by a 1$\times$1 convolutional layer (conv) to reshape the candidate guidance to a consistent size with $\mathbf{F}_{k}^{n}$.

Symbols $\oplus$ and $\bigcup$ respectively denote element-wise addition and concatenation operations with an IDC strategy[For instance, $\mathbf{\bar{G}}_{2}^{n} = \bigcup_{i=2}^{K=4} [\mathbf{F}_{2}^{n}, \mathcal{P} ( \mathbf{G}_{i}^{n})] = \mathbf{F}_{2}^{n} \odot \mathcal{P} ( \mathbf{G}_{2}^{n}) \odot \mathcal{P} (\mathbf{G}_{3}^{n}) \odot \mathcal{P} ( \mathbf{G}_{4}^{n})$ when $k=2$ and $K=4$.], followed by a 1$\times$1 conv with 32 filters.

For the other part, we formulate the temporal feature re-calibration branch (see Fig:framework_bpm (a)) as:

\begin{equation}
\mathbf{G}_{k}^{n+1} = \mathbf{G}_{k}^{n} \oplus \bigcap_{j=k}^{K}[ \mathbf{G}_{k}^{n},\mathcal{P}(\mathbf{F}_{j}^{n})],
\end{equation}

where $\bigcap$ denotes element-wise multiplication with an IDC strategy, followed by a 1$\times$1 conv with 32 filters.

Illustration of our Bidirectional Purification Module (BPM) with a simplex and full-duplex strategy.

§.§ Decoder

After feature aggregation and re-calibration with multi-pyramidal interaction, the last BPM unit produces two groups of discriminative features (, $\mathbf{F}_{k}^{N}$ & $\mathbf{G}_{k}^{N}$) with a consistent channel number of 32.

We integrate pyramid pooling module (PPM) [136] into each skip connection of the U-Net [86] as our decoder, and only adopt the top four layers in our implementation ($K=4$).

Since the features are fused from high to low level, global information is well retained at different scales of the designed decoder:

\begin{equation}\label{equ:ppm_1}
\hat{\mathbf{F}}_{k}^{N} = \mathcal{C} [ \mathbf{F}_{k}^{N} \odot \mathcal{UP}(\hat{\mathbf{F}}_{k+1}^{N}) ],
\end{equation}

\begin{equation}\label{equ:ppm_2}
\hat{\mathbf{G}}_{k}^{N} = \mathcal{C} [ \mathbf{G}_{k}^{N} \odot \mathcal{UP}(\hat{\mathbf{G}}_{k+1}^{N}) ].
\end{equation}

Here, $\mathcal{UP}$ indicates the upsampling operation after the pyramid pooling layer, while $\odot$ is a concatenation operation between two features. Then, a conv $\mathcal{C}$ is used for reducing the channels from 64 to 32.

Lastly, we use a 1$\times$1 conv with a single filter after the upstream output (, $\hat{\mathbf{F}}_{1}^{N}$ & $\hat{\mathbf{G}}_{1}^{N}$), followed by a sigmoid activation function to generate the predictions (, $\mathbf{S}_{A}^{t}$ & $\mathbf{S}_{M}^{t}$) at frame $t$.

§.§ Learning Objective

Given a group of predictions $\mathbf{S}^{t} \in \{\mathbf{S}_{A}^{t}, \mathbf{S}_{M}^{t}\}$ and the corresponding ground-truths $\mathbf{G}^{t}$ at frame $t$, we employ the standard binary cross-entropy loss $\mathcal{L}_{bce}$ to measure the dissimilarity between output and target, which computes:

\begin{equation}
\begin{aligned}
\mathcal{L}_{bce}(\mathbf{S}^{t},\mathbf{G}^{t}) =
&-\sum_{(x, y)}[\mathbf{G}^{t}(x, y) \log (\mathbf{S}^{t}(x, y)) \\
&+ (1 - \mathbf{G}^{t}(x, y)) \log (1-\mathbf{S}^{t}(x, y))],
\end{aligned}
\end{equation}

where $(x,y)$ indicates a coordinate in the frame. The overall loss function is then formulated as:

\begin{equation}
\mathcal{L}_{total} = \mathcal{L}_{bce}(\mathbf{S}_{A}^{t},\mathbf{G}^{t}) + \mathcal{L}_{bce}(\mathbf{S}_{M}^{t},\mathbf{G}^{t}).
\end{equation}

For final prediction, we use $\mathbf{S}_{A}^{t}$ since our experiments show that it performs better when combining appearance and motion cues.

§.§ Implementation Details

§.§.§ Backbone Details

Without any modification, three standard ResNet-50 [36] (removing the top-three layers: average pooling, fully-connected, and softmax layers) backbones are adopted for the appearance branch, the motion branch and the merging branch.

Each ResNet-50 backbone results in $K=4$ hierarchies inspired by previous work [117].

After removing the top fully connected layers, the feature hierarchies ($\{\mathcal{X}_k, \mathcal{Y}_k, \mathcal{Z}_k\}, k \in \{2:5\}$) from shallow to deep are extracted from the conv2_3 ($k=2$), conv3_4 ($k=3$), conv4_6 ($k=4$), and conv5_3 ($k=5$) layers of the ResNet-50, respectively.

Note that we have also tried a two-branches setting, namely removing the merging branch and letting $\mathcal{Z}_k = \mathcal{Q}^X_k \oplus \mathcal{Q}^{Y}_{k} \oplus \mathcal{Z}_{k-1}$ instead of $\mathcal{Z}_k = \mathcal{B}_k [ \mathcal{Q}_k^X \oplus \mathcal{Q}^Y_x \oplus \mathcal{Z}_{k-1}]$ in equ:3.

Unfortunately, this leads to a 2.5% performance drop in performance concerning $S_\alpha$ on the DAVIS$_{16}$ [82] dataset.

This is because the third merging branch can sequentially enhance and promote the spatial-temporal features from RCAMs, leading to better segmentation accuracy.

§.§.§ Training Settings

We implement our model in PyTorch [79], accelerated by an NVIDIA RTX TITAN GPU.

All the inputs are uniformly resized to 352$\times$352.

To enhance the stability and generalizability of our learning algorithm, we employ the multi-scale (, $\{0.75,1,1.25\}$) training strategy [35] in the training phase.

As can be seen from the experimental results in tab:number_of_bpm, the variant with $N$=4 (the number of BPM) achieves the best performance.

We utilize the stochastic gradient descent (SGD) algorithm to optimize the entire network, with a momentum of $0.9$, a learning rate of $2e^{-3}$, and a weight decay of $5e^{-4}$.

The learning rate decreased by 10% per 20 epochs.

§.§.§ Testing Settings and Runtime

Given a frame along with its motion map, we resize them to 352$\times$352 and feed them into the corresponding branch.

Similar to [141, 68, 113], We employ the conditional random field (CRF) [52] post-processing technique.

The inference time of our method is 0.08s per frame, regardless of flow generation and CRF post-processing.

§ EXPERIMENTS

§.§ Experimental Protocols

§.§.§ Datasets

We evaluate the proposed model on four widely used VOS datasets.

DAVIS$_{16}$ [82] is the most popular of these, and consists of 50 (30 training and 20 validation) high-quality and densely annotated video sequences.

MCL [49] contains 9 videos and is mainly used as testing data.

FBMS [76] includes 59 natural videos, in which 29 sequences are
used as the training set and 30 are for testing.
SegTrack-V2 [59] is one of the earliest VOS datasets and consists of 13 clips.
In addition, DAVSOD$_{19}$ [30] was specifically designed for the V-SOD task.

It is the most challenging visual attention consistent V-SOD dataset with high-quality annotations and diverse attributes.

§.§.§ Training

Following a similar multi-task training setup as [61], we divide our training procedure into three steps:

* We first adopt a well-known static saliency dataset DUTS [104] to train the spatial branch to avoid over-fitting, like in [110, 93, 30]. This step lasts for 50 epoches with a batch size of 8 under the same training settings mentioned in sec:training_scheme.

* We then train the temporal branch on the generated optical flow maps. This step lasts for 50 epoches with a batch size of 8 under the same training settings mentioned in sec:training_scheme.

* We finally load the weights pre-trained on two sub-tasks into the spatial and temporal branches, and thus, the whole network is end-to-end trained on the training set of DAVIS$_{16}$ (30 clips) and FBMS (29 clips). The last step takes about 4 hours and converges after 20 epochs with a batch size of 8 under same the training settings mentioned in sec:training_scheme.

§.§.§ Testing

We follow the standard benchmarks [82, 30] to test our model on the validation set (20 clips) of DAVIS$_{16}$, the test set of FBMS (30 clips), the test set (Easy35 split) of DAVSOD$_{19}$ (35 clips), the whole of MCL (9 clips), and the whole of SegTrack-V2  (13 clips).

§.§ Evaluation Metrics

We define a prediction map at frame $t$ as $\mathbf{S}_{A}^{t}$ and its corresponding ground-truth mask as $\mathbf{G}^t$. The formulations of the metrics are given as follows.

§.§.§ Metrics for U-VOS task

Following [129], we utilize two standard metrics to evaluate the performance of U-VOS models. Note that all prediction maps are ensured to be binary in the U-VOS task.

* Mean Region Similarity: This metric, also called jaccard similarity coefficient, is defined as the intersection-over-union of the prediction map and the ground-truth mask. The formulation is defined as:

\begin{equation}\label{equ:region_sim}
\mathcal{J} = \frac{|\mathbf{S}_{A}^{t} \cap \mathbf{G}^t|}{|\mathbf{S}_{A}^{t} \cup \mathbf{G}^t|},
\end{equation}

where $|\cdot|$ is the number of pixels in the area. In all of our experiments, we also report the mean value of Mean-$\mathcal{J}$, similar to [129].

* Mean Contour Accuracy: Here, the contour accuracy metric we used is also called the contour F-measure. We compute the contour-based precision and recall between the contour points of $c(\mathbf{S}_{A}^{t})$ and $c(\mathbf{G}^t)$, where $c(\cdot)$ is the extraction of contour points of a mask. The formulation is defined as:

\begin{equation}\label{equ:contour}
\mathcal{F} = \frac{2 \times \textrm{Precision}_{c} \times \textrm{Recall}_{c}}{\textrm{Precision}_c + \textrm{Recall}_c},
\end{equation}

where $\textrm{Precision}_{c} = |c(\mathbf{S}_{A}^{t}) \cap c(\mathbf{G}^t)|/|c(\mathbf{S}_{A}^{t})|$, $\textrm{Recall}_c = |c(\mathbf{S}_{A}^{t}) \cap c(\mathbf{G}^t))| / |c(\mathbf{G}^t)|$.

Similar to [129], we also report the mean value of Mean-$\mathcal{F}$ in all of our experiments.

§.§.§ Metrics for V-SOD task

Unlike the U-VOS task, the prediction map can be non-binary in the V-SOD benchmarking. More details refer to sec:evaluation_on_davis16.

* Mean Absolute Error (MAE): It is a typical pixel-wise measure, which is defined as:

\begin{equation}\label{equ:mae}
\mathcal{M} = \frac{1}{W \times H} \sum_{x}^{W} \sum_{y}^{H} |\mathbf{S}_{A}^{t}(x,y) - \mathbf{G}^t(x,y)|,
\end{equation}

where $W$ and $H$ are the width and height of ground-truth $\mathbf{G}^t$, and $(x,y)$ are the coordinates of a pixel in $\mathbf{G}^t$.

* Precision-Recall (PR) Curve: Precision and recall [2, 22, 8] are defined as:

\begin{equation}
\textrm{Precision}=\frac{|\mathbf{S}_{A}^{t}(T) \cap \mathbf{G}^t|}{|\mathbf{S}_{A}^{t}(T)|},
\end{equation}

\begin{equation}
\textrm{Recall} =\frac{|\mathbf{S}_{A}^{t}(T)\cap \mathbf{G}^t|}{|\mathbf{G}^t|},
\end{equation}

where $\mathbf{S}_{A}^{t}(T)$ is the binary mask obtained by directly thresholding the prediction map $\mathbf{S}_{A}^{t}$ with the threshold $T \in [0,255]$, and $|\cdot|$ is the total area of the mask inside the map. By varying $T$, a precision-recall curve can be obtained.

* Maximum F-measure: This is defined as:

\begin{equation}\label{equ:max_f}
F_\beta = \frac{(1+\beta^2) \textrm{Precision} \times \textrm{Recall}} {\beta^{2} \times \textrm{Precision} + \textrm{Recall}},
\end{equation}

where $\beta^2$ is set to 0.3 to focus more on the precision value than the recall value, as recommended in [9]. We convert the non-binary prediction map into binary masks with threshold values from 0 to 255. In this paper, we report the maximum (, $F_{\beta}^{max}$)  of a series of F-measure values calculated from the precision-recall curve by iterating over all the thresholds.

* Maximum Enhanced-Alignment Measure: As a recently proposed metric, $E_{\xi}$ [3] is used to evaluate both the local and global similarity between two binary maps. The formulation is as follows:

\begin{equation}\label{equ:e_m}
E_{\xi} = \frac{1}{W \times H} \sum_{x}^{W} \sum_{y}^{H} \phi\left[\mathbf{S}_{A}^{t}(x, y), \mathbf{G}^t(x, y)\right],
\end{equation}

where $\phi$ is the enhanced-alignment matrix. Similar to $F_\beta^{max}$, we report the maximum $E_{\xi}$ value computed from all the thresholds in all of our comparisons.

* Structure Measure: Fan  [28] proposed a metric to measure the structural similarity between a non-binary saliency map and a ground-truth mask:

\begin{equation}\label{equ:s_m}
S_\alpha = (1-\alpha) \times S_o(\mathbf{S}_{A}^{t}, \mathbf{G}^t) + \alpha \times S_r(\mathbf{S}_{A}^{t}, \mathbf{G}^t),
\end{equation}

where $\alpha$ balances the object-aware similarity $S_o$ and region-aware similarity $S_r$. We use the default setting ($\alpha=0.5$) suggested in [28].

Video object segmentation (VOS) performance of our , compared with 14 SOTA unsupervised models and seven semi-supervised models on DAVIS$_{16}$ [82] validation set.
`w/ Flow': the optical flow algorithm is used.
`w/ CRF': conditional random field [52] is used for post-processing.
The best scores are marked in bold.

16c|Unsupervised     7cSemi-supervised

0*     0*Metrics

MAT        AGNN          AnDiff        COS        AGS
EpO+       MOA           LSMO          ARP           LVO
LMP        SFL           ELM           FST

AGA      RGM       FEEL      FA	     OS     MSK

[141]
[105]
[129]     [68]     [113]     [26]     [92]
[98]         [51]     [97]
[96]        [21]      [54]     [78]

[130]
[47]      [121]       [102]      [20]	     [11]
[81]

w/ Flow

w/ CRF

Mean-$\mathcal{J} \uparrow$
mygray83.4     82.1
82.4     80.7
81.7     80.5     79.7     80.6     77.2
78.2     76.2     75.9
70.0     67.4     61.8     55.8
mygray85.3     81.5     81.5     81.1     82.4     79.8     79.7

Mean-$\mathcal{F} \uparrow$
83.1     mygray83.3
80.7     79.1
80.5     79.5     77.4     75.5     77.4
75.9     70.6     72.1
65.9     66.7     61.2     51.1
mygray86.9     82.2     82.0     82.2     79.5     80.6     75.4

Video salient object detection (V-SOD) performance of our , compared with 13 SOTA models on three popular V-SOD datasets, including DAVIS$_{16}$ [82], MCL [49], and FBMS [76].

`$\dag$' denotes that we generate non-binary saliency maps without CRF [52] for a fair comparison.
`N/A' means the results are not available.

4c|DAVIS$_{16}$ [82]   4c|MCL [49]
4cFBMS [76]

-2*     1*Model
$S_{\alpha}\uparrow$     $E_{\xi}^{max}\uparrow$
$F_{\beta}^{max}\uparrow$     $\mathcal{M}\downarrow$
$S_{\alpha}\uparrow$     $E_{\xi}^{max}\uparrow$
$F_{\beta}^{max}\uparrow$     $\mathcal{M}\downarrow$
$S_{\alpha}\uparrow$     $E_{\xi}^{max}\uparrow$
$F_{\beta}^{max}\uparrow$     $\mathcal{M}\downarrow$

MBN [62]
0.887       0.966       0.862       0.031
0.755       0.858       0.698       0.119
0.857       0.892       0.816       0.047

FGRN [60]
0.838       0.917       0.783       0.043
0.709       0.817       0.625       0.044
0.809       0.863       0.767       0.088

SCNN [94]
0.761       0.843       0.679       0.077
0.730       0.828       0.628       0.054
0.794       0.865       0.762       0.095

DLVS [110]
0.802       0.895       0.721       0.055
0.682       0.810       0.551       0.060
0.794       0.861       0.759       0.091

SCOM [17]
0.814       0.874       0.746       0.055
0.569       0.704       0.422       0.204
0.794       0.873       0.797       0.079

RSE [125]
0.748       0.878       0.698       0.063
0.682       0.657       0.576       0.073
0.670       0.790       0.652       0.128

SRP [23]
0.662       0.843       0.660       0.070
0.689       0.812       0.646       0.058
0.648       0.773       0.671       0.134

MESO [124]
0.718       0.853       0.660       0.070
0.477       0.730       0.144       0.102
0.635       0.767       0.618       0.134

LTSI [15]
0.876       0.957       0.850       0.034
0.768       0.872       0.667       0.044
0.805       0.871       0.799       0.087

SPD [63]
0.783       0.892       0.763       0.061
0.685       0.794       0.601       0.069
0.691       0.804       0.686       0.125

0.893       0.948       0.861       0.028
0.819       0.889       0.773       0.026
0.879       0.926       0.865       0.040

0.886       0.947       0.848       0.027
0.820       0.895       0.742       0.028
0.872       0.905       0.859       0.053

0.902       0.961       0.880       0.022
N/A          N/A        N/A         N/A
0.868       0.920       0.837       0.040

0.920       0.970       0.907       0.020
0.864       0.924       0.821       0.023
0.890       0.935       0.888       0.041

§.§ U-VOS and V-SOD tasks

§.§.§ Evaluation on DAVIS$_{16}$ dataset

As shown in tab:davis16, we compare our  with 14 SOTA U-VOS models on the DAVIS$_{16}$ public leaderboard.

We also compare it with seven recent semi-supervised approaches as reference.

We use a threshold of 0.5 to generate the final binary maps for a fair comparison, as recommended by [129].

Our  outperforms the best model (AAAI'20-MAT [141]) by a margin of 2.4% in Mean-$\mathcal{F}$ and 1.0% in Mean-$\mathcal{J}$, achieving the new SOTA performance.

Notably, the proposed U-VOS model also outperforms the semi-supervised model (, AGA [47]), even though it utilizes the first ground-truth mask to reference object location.

We also compare  against 13 SOTA V-SOD models. The non-binary saliency maps[Note that all compared maps in the V-SOD task, including ours, are non-binary.] are obtained from the standard benchmark [30].

This can be seen from tab:score_VOS, our method consistently outperforms all other models since 2018 on all metrics. In particular, for the $S_\alpha$ and $F_\beta^{max}$ metrics, our method improves the performance by $\sim$2.0% compared with the best AAAI'20-PCAS [34] model.

§.§.§ Evaluation on MCL dataset

This dataset has fuzzy object boundaries in the low-resolution frames due to fast object movements.

Therefore, the overall performance is lower than on DAVIS$_{16}$. As shown in tab:score_VOS, our method still stands out in these extreme circumstances, with a 3.0$\sim$8.0% increase in all metrics compared with ICCV'19-RCR [127] and CVPR'19-SSAV [30].

§.§.§ Evaluation on FBMS dataset

This is one of the most popular VOS datasets with diverse attributes, such as interacting objects, dynamic backgrounds, and no per-frame annotation. As shown in tab:score_VOS, our model achieves competitive performance in terms of $\mathcal{M}$.

Further, compared to the previous best-performing SSAV [30], it obtains improvements in other metrics, including $\mathcal{S}_{\alpha}$ (0.890 vs. SSAV=0.879) and $E_{\xi}^{max}$ (0.935 vs. SSAV=0.926), making it more suitable to the human visual system (HVS) as mentioned in [28, 29].

Benchmarking results of 13 state-of-the-art V-SOD models on three subsets of DAVSOD$_{19}$ [30]. `$\dag$' denotes that we generate non-binary saliency maps without CRF [52] for a fair comparison. `N/A' means the results are not available.

1c$S_\alpha\uparrow$      1c$E_\xi^{max}\uparrow$    1c$F_\beta^{max}\uparrow$     1c|$\mathcal{M}\downarrow$
1c$S_\alpha\uparrow$      1c$E_\xi^{max}\uparrow$    1c$F_\beta^{max}\uparrow$     1c|$\mathcal{M}\downarrow$
1c$S_\alpha\uparrow$      1c$E_\xi^{max}\uparrow$    1c$F_\beta^{max}\uparrow$     1c$\mathcal{M}\downarrow$

MBN [62]
0.646     0.694     0.506    0.109     0.597     0.665     0.436    0.127     0.561     0.635     0.352    0.140

FGRN [60]
0.701     0.765     0.589    0.095     0.638     0.700     0.468    0.126     0.608     0.698     0.390    0.131

SCNN [94]
0.680     0.745     0.541    0.127     0.589     0.685     0.425    0.193     0.533     0.677     0.345    0.234

DLVS [110]
0.664     0.737     0.541    0.129     0.599     0.670     0.416    0.147     0.571     0.687     0.336    0.128

SCOM [17]
0.603     0.669     0.473    0.219     N/A     N/A     N/A     N/A     N/A     N/A     N/A     N/A

RSE [125]
0.577     0.663     0.417    0.146     0.549     0.590     0.360    0.170     0.555     0.644     0.306    0.130

SRP [23]
0.575     0.655     0.453    0.146     0.545     0.601     0.387    0.169     0.555     0.682     0.341    0.123

MESO [124]
0.549     0.673     0.360    0.159     0.542     0.597     0.354    0.165     0.556     0.661     0.310    0.127

LTSI [15]
0.695     0.769     0.585    0.106     0.658     0.723     0.499    0.128     0.618     0.718     0.406    0.112

SPD [63]
0.626     0.685     0.500    0.138     0.596     0.633     0.443    0.171     0.574     0.688     0.345    0.137

0.755     0.806     0.659    0.084     0.661     0.723     0.509    0.117     0.619     0.696     0.399    0.114

0.741     0.803     0.653    0.087     0.674     0.729     0.533    0.118     0.644     0.768     0.444     0.094

0.741     0.793     0.656    0.086     N/A     N/A     N/A     N/A     N/A     N/A     N/A     N/A

0.773     0.825     0.685     0.072
0.707     0.764     0.597     0.104
0.662     0.752     0.487     0.099

§.§.§ Evaluation on SegTrack-V2 dataset

This is the earliest VOS dataset from the traditional era. Thus, only a limited number of deep U-VOS models have been tested on it. We only compare our  against the top-3 models: AAAI'20-PCAS [34] ($S_\alpha$=0.866), ICCV'19-RCR [127] ($S_\alpha$=0.842), and CVPR'19-SSAV [30] ($S_\alpha$=0.850). Our method achieves the best performance ($S_\alpha$=0.870).

§.§.§ Evaluation on DAVSOD$_{19}$ dataset

Recently published, DAVSOD$_{19}$ [30] is the most challenging visual attention consistent V-SOD dataset with high-quality annotations and diverse attributes.

It contains diversified challenging scenarios due to the video sequences containing shifts in attention.

DAVSOD$_{19}$ is divided into three subsets, according to difficulty: DAVSOD$_{19}$-Easy-35 (35 clips), DAVSOD$_{19}$-Normal25 (25 clips), and DAVSOD$_{19}$-Difficult20 (20 clips).

Note that, in the saliency field, non-binary maps are required for evaluation; thus, we only report the results of  without CRF post-processing in benchmarking the V-SOD task.

In this document, we adopt the four metrics mentioned in sec:metric_vsod, including $S_\alpha$, $E_{\xi}^{max}$, $F_\beta^{max}$, and $\mathcal{M}$.

For showing the robustness of , in tab:davsod_benchmark, we also make the first effort to benchmark all 11 SOTA models since 2018, in terms of the three difficulty levels:

Precision-recall curves of SOTA V-SOD methods and the proposed  across six datasets. Zoom in for details and the best view in color for friendlier observation.

* Easy35 subset: Most of the video sequences are similar to those in the DAVIS$_{16}$ dataset, which also consists of a large number of single video objects. We see that
outperforms all the reported algorithms across all metrics.

As shown in tab:davsod_benchmark, compared with the recent method (PCSA), our model achieves large improvements of 3.2% in terms of $S_{\alpha}$.

* Normal25 subset: Different from previous subsets, this one includes multiple moving salient objects. Thus, it is more difficult than traditional V-SOD datasets due to the attention shift phenomena [30]. As expected,  still obtains the best performance, with significant improvement, , 6.4% for $F_{\beta}^{max}$ metric.

* Difficult20 subset: This is the most challenging subset in existing V-SOD datasets since it contains a large number of attention shift sequences under cluttered scenarios.

Therefore, from the results shown in tab:davsod_benchmark, the performances of all the compared models decrease dramatically (, $F_\beta^{max}\leq$ 0.5).

Even though our framework is not specifically designed for the V-SOD task, we still easily obtain the best performance in two metrics (, $\mathcal{S}_{\alpha}$ and $F_\beta^{max}$).

Different from the best two models, which utilize additional training data (, RCR leverages pseudo-labels, SSAV utilizes the validation set), our model does not use any additional training data and still outperforms the SSAV model by 8.8% ($F_{\beta}^{max}$), and achieves comparable performance to the second-best RCR (ICCV'19) model.

These results are also supported by recent conclusions that “human visual attention should be an underlying mechanism that drives U-VOS and V-SOD” (TPAMI'20 [107]).

§.§.§ PR Curve

As shown in fig:pr_curve, we further investigate the precision-recall curves of different models on six V-SOD datasets, including DAVIS$_{16}$ [82], MCL [49], FBMS [76], and DAVSOD$_{19}$ [30] (, Easy35, Normal25, and Difficult20).

Note that the higher and more to the right in the PR curve, the more accurate performance.

Even though existing SOTA methods have achieved significant progress in the V-SOD task on three typical benchmark datasets, we still obtain the best performance under all thresholds.

Besides, as a recent and challenging dataset, the overall performances on the three subsets of DAVSOD$_{19}$ [30] are relatively poor. However, our  again achieves more satisfactory performance by large margins.

Qualitative results on five datasets, including  DAVIS$_{16}$ [82], MCL [49],  FBMS [76], SegTrack-V2 [59],
and DAVSOD$_{19}$ [30].

§.§.§ Qualitative Results

Some qualitative results on the five datasets are shown in Fig:performance1, validating that our method achieves high-quality U-VOS and V-SOD results.

As can be seen in the 1$^{st}$ row, the behind camel did not move, so it does not get noticed.

Interestingly, as our full-duplex strategy model considers both appearance and motion bidirectionally, it can automatically predict the dominated camel in the centre of the video instead of the camel behind.

A similar phenomenon is also presented in the 5$^{th}$ row, our method successfully detects dynamic skiers with the video clip rather than the static man in the background.

Overall, for these challenging situations, ,
dynamic background (1$^{st}$ & 5$^{th}$ rows),
fast-motion (4$^{rd}$ row),
out-of-view (6$^{rd}$ & 7$^{nd}$ row),
occlusion (7$^{nd}$ row), and
deformation (8$^{th}$ row),
our model is able to infer the real target object(s) with fine-grained details.

From this point of view, we demonstrate that  is a general framework for both U-VOS and V-SOD tasks.

§.§ Ablation Study

In this section, we conduct ablation studies to analyse our , including stimulus selection (sec:stimulus_selection), effectiveness of RCAM (sec:effectiveness_of_RCAM) and BPM (sec:effectiveness_of_bpm), number of cascaded BPMs (sec:number_of_bpms), and effectiveness of full-duplex strategy (sec:effectivess_of_FS).

§.§.§ Stimulus Selection

We explore the influence of different stimuli (appearance only vs. motion only) in our framework. We use only video frames or motion maps (using [42]) to train the ResNet-50 [36] backbone together with the proposed decoder block (see sec:decoder).

As shown in tab:ablation_icam, $Mo.$ performs slightly better than $App.$ in terms of $S_\alpha$ on DAVIS$_{16}$, which suggests that the “optical flow” setting can learn more visual cues than “video frames”.

Nevertheless, $App.$ outperforms $Mo.$ in $\mathcal{M}$ metric on MCL.

This motivates us to explore how to use appearance and motion cues simultaneously effectively.

§.§.§ Effectiveness of RCAM

To validate our RCAM (Rel.) effectiveness, we replace our fusion strategy with the vanilla fusion (Vanilla) using a concatenate operation followed by a convolutional layer to fuse two modalities.

As expected (tab:ablation_icam), the proposed Rel. performs consistently better than the vanilla fusion strategy on both DAVIS$_{16}$ and MCL datasets.
We would like to point out that our RCAM has two important properties:

* It enables mutual correction and attention.

* It can alleviate error propagation within a network to an extent due to the mutual correction and bidirectional interaction.

Ablation studies (sec:stimulus_selection, sec:effectiveness_of_RCAM, & sec:effectiveness_of_bpm) for our components on DAVIS$_{16}$ and MCL. We set $N=4$ for BPM.

4c|Component Settings     2c|DAVIS$_{16}$    2cMCL

Appearance     Motion     RCAM     BPM
$\mathcal{S}_{\alpha}\uparrow$    $\mathcal{M}\downarrow$
$\mathcal{S}_{\alpha}\uparrow$     $\mathcal{M}\downarrow$

$App.$                                                     0.834     0.047     0.754     0.038

$Mo.$                                                     0.858     0.039     0.763     0.053

Vanilla                                    0.871     0.035     0.776     0.046

Rel.                  0.900     0.025     0.833     0.031

Bi-Purf.                 0.904     0.026     0.855     0.023

$^\dag$                0.920     0.020     0.864     0.023

Ablation study for the number ($N$) of BPMs on DAVIS$_{16}$ [82] and MCL [49], focusing on parameter and FLOPs of BPMs, and runtime of .

Param.     FLOPs     Runtime     2c|DAVIS$_{16}$    2cMCL

(M)     (G)     (s/frame)
$\mathcal{S}_{\alpha}\uparrow$     $\mathcal{M}\downarrow$
$\mathcal{S}_{\alpha}\uparrow$     $\mathcal{M}\downarrow$

$N=0$     0.000     0.000      0.03     0.900      0.025     0.833      0.031

$N=2$     0.507     1.582     0.05     0.911     0.026     0.843     0.028

1.015     3.163     0.08     0.920      0.020     0.864     0.023

$N=6$     1.522     4.745     0.10     0.918      0.023     0.863      0.023

$N=8$     2.030     6.327     0.13     0.920    0.023     0.864     0.023

§.§.§ Effectiveness of BPM

To illustrate the effectiveness of the BPM (with $N=4$), we derive two different models: Rel. and , referring to the framework without or with BPM. We observe that the model with BPM gains 2.0$\sim$3.0% than the one without BPM, according to the statistics in tab:ablation_icam. We attribute this improvement to BPM's introduction of an interlaced decremental connection, enabling it to fuse the different signals effectively.

Similarly, we remove the RCAM and derive another pair of settings (Vanilla & Bi-Purf.) to test the robustness of our BPM. The results show that even using the bidirectional vanilla fusion strategy (Bi-Purf.) can still enhance the stability and generalization of the model. This benefits from the purification forward process and re-calibration backward process in the whole network.

Ablation study for the simplex and full-duplex strategies on DAVIS$_{16}$ [82] and MCL [49]. We set $N=4$ for BPM.

2c|Direction Setting     2c|DAVIS$_{16}$    2cMCL

1c|RCAM     1c|BPM
$\mathcal{S}_{\alpha}\uparrow$      $\mathcal{M}\downarrow$
$\mathcal{S}_{\alpha}\uparrow$     $\mathcal{M}\downarrow$

4* simplex
$App. \Rightarrow Mo.$        $(App.+Mo.) \Rightarrow Mo.$
0.896     0.026     0.816     0.038

$App. \Rightarrow Mo.$        $(App.+Mo.) \Leftarrow Mo.$
0.902     0.025     0.832     0.031

$App. \Leftarrow Mo.$         $(App.+Mo.) \Rightarrow Mo.$
0.891     0.029     0.806     0.039

$App. \Leftarrow Mo.$         $(App.+Mo.) \Leftarrow Mo.$
0.897     0.028     0.840     0.028

self-purf.    $App. \Leftrightarrow Mo.$    $(App.+Mo.) \nLeftrightarrow Mo.$     0.899     0.026     0.854     0.023

full-dup.     $App. \Leftrightarrow Mo.$     $(App.+Mo.) \Leftrightarrow Mo.$
0.920     0.020     0.864     0.023

§.§.§ Number of Cascaded BPMs

Naturally, more cascaded BPMs should lead to better boost performance. This is investigated and the evaluation results are shown in tab:number_of_bpm, where $N=\{0,2,4,6,8\}$.

Note that $N=0$ means that NO BPM is used.

Clearly, as can be seen from fig:front_figure_2 (red star), we compare four variants of our , including $N$=0 (Mean-$\mathcal{J}$=76.4, Mean-$\mathcal{F}$=76.8), $N$=2 (Mean-$\mathcal{J}$=80.4, Mean-$\mathcal{F}$=81.4), $N$=4 (Mean-$\mathcal{J}$=82.1, Mean-$\mathcal{F}$=83.3), and $N$=4, CRF (Mean-$\mathcal{J}$=83.4, Mean-$\mathcal{F}$=83.1).

It demonstrates that more BPMs leads to better results, but the performance reaches saturation after $N=4$.

Further, too many BPMs (, $N>4$) will cause high model-complexity and increase the over-fitting risk. As a trade-off, we use $N=4$ throughout our experiments.

§.§.§ Effectiveness of Full-Duplex Strategy

To investigate the effectiveness of the RCAM and BPM modules with the full-duplex strategy, we study two unidirectional (, simplex strategy in Fig:framework_RCAM & Fig:framework_bpm) variants of our model.
In tab:ablation_bi_model, the symbols $\Rightarrow$, $\Leftarrow$, and $\Leftrightarrow$ indicate the feature transmission directions in the designed RCAM
or BPM.

Specifically, $App. \Leftarrow Mo.$ indicates that the attention vector in the optical flow branch weights the features in the appearance branch and vice versa. $(App. + Mo.) \Leftarrow Mo.$ indicates that motion cues are used to guide the fused features extracted from both appearance and motion.
The comparison results show that our elaborately designed modules (RCAM and BPM) jointly cooperate in a full-duplex fashion and outperform all simplex (unidirectional) settings.

§.§ Further Discussion

§.§.§ Prediction Selection

Which is the final prediction, $\mathbf{S}_{A}^{t}$ or $\mathbf{S}_{M}^{t}$?

As mentioned in sec:training, we choose $\mathbf{S}_{A}^{t}$ as our final segmentation result instead of $\mathbf{S}_{M}^{t}$.

The major reasons for doing so can be summarized as follows:

* We employ the auxiliary supervision for the motion-based branch to learn more motion patterns inspired by [96].

* More informative appearance and motion cues are contained in another branch at the phase of bidirectional purification.

Ablation study (sec:prediction_selection) for the choice of final segmentation result on DAVIS$_{16}$ [82] and MCL [49] dataset.

2c|DAVIS$_{16}$    2cMCL

$\mathcal{S}_{\alpha}\uparrow$     $\mathcal{M}\downarrow$
$\mathcal{S}_{\alpha}\uparrow$     $\mathcal{M}\downarrow$

(a) $\mathbf{S}_{M}^{t}$ as result
0.920     0.022     0.862     0.024

(b) $(\mathbf{S}_{A}^{t}+\mathbf{S}_{M}^{t})/2$ as result
0.920     0.022     0.863     0.023

(c) $\mathbf{S}_{A}^{t}$ as result (Ours)
0.920     0.022     0.864     0.023

As shown in tab:choice_of_result, three experiments are conducted to verify our assumption:
(a) choosing $\mathbf{S}_{M}^{t}$ as the final result,
(b) choosing $(\mathbf{S}_{A}^{t}+\mathbf{S}_{M}^{t})/2$ as the final result, and
(c) choosing $\mathbf{S}_{A}^{t}$ as the final result (Ours).

As can be seen in tab:choice_of_result, all three choices achieve very similar results, while $\mathbf{S}_{A}^{t}$ performs slightly better than the other two.

Besides, considering the reduction of unnecessary computational cost, we choose $\mathbf{S}_{A}^{t}$ as our final result for comparison with other methods.

§.§.§ Effectiveness of CRF

From fig:front_figure_2 we can see that our  without CRF post-processing technique, ,  ($N$=4), still outperforms the best model AAAI'20-MAT in terms of Mean-$\mathcal{F}$ metric.

This means that our initial method (,  without CRF) can distinguish hard samples around the object boundaries without post-processing techniques.

When equipped with the CRF post-processing technique [52], our  ($N$=4, CRF) achieves the best performance in terms of both Mean-$\mathcal{J}$ and Mean-$\mathcal{F}$ metrics.

(74, 1)$\mathcal{J}$
(3, 64)90$\mathcal{F}$

Mean contour accuracy ($\mathcal{F}$) vs. mean region similarity ($\mathcal{J}$) scores on DAVIS$_{16}$ dataset [82].
Circles indicate U-VOS methods.
Four variants are shown in bold-italic, in which `$N$' indicates the number of bidirectional purification modules (BPM) and `CRF' means that using CRF [52] post-processing technique.
Compared with the best unsupervised VOS model (, MAT [141] also with CRF),
the proposed method  ($N$=4, CRF) achieves the new SOTA by a large margin.

Illustration of self-purification strategy (a) and the proposed bidirectional purification strategy (b). Note that sub-figure (b) is same as the Fig:framework_bpm (c) for convenient comparison. Note that $\oplus$, $\otimes$, and $\odot$ denote element-wise addition, multiplication, and concatenation, respectively.

§.§.§ Training Effectiveness with Less Data

As shown in fig:front_figure_2, the proposed method, ,  ($N$=4, CRF), surpasses the best U-VOS model MAT [141] (also with CRF), while our  with less labelled data in the training phase (, Ours-13K vs. MAT-16K).

Besides, we also observe that the recently proposed method 3DC-Seg [71], based on a 3D convolutional network, can achieve the new state-of-the-art (Mean-$\mathcal{J}$=84.3, Mean-$\mathcal{F}$=84.7), while relies on a massive amount of labelled training samples as expert knowledge in the fine-tuning phase, including 158K images (, COCO [67] + YouTube-VOS [126] + DAVIS$_{16}$ [82]).

It requires about ten times more training data than the best model MAT [141] (16K images) in the fine-tuning phase.

Thus, it demonstrates the efficient training process in our pipeline.

§.§.§ Self-Purification Strategy in BPM

We provide more details on the different variants mentioned in sec:effectivess_of_FS

including $(App. + Mo.) \Leftarrow Mo.$, $(App. + Mo.) \Rightarrow Mo.$ and $(App.+Mo.) \Leftrightarrow Mo.$ in BPM.

The implementation of $App. \Leftarrow Mo.$ and $App. \Rightarrow Mo.$ in RCAM are illustrated in Fig:framework_RCAM (a) & (b), whereas the structure implementation of $(App. + Mo.) \Leftarrow Mo.$ & $(App. + Mo.) \Rightarrow Mo.$ in BPM are illustrated in Fig:framework_bpm (a) & (b).

Here, note that all of these variants indicate unidirectional refinement, in contrast to the proposed bi-directional schemes.

Last but not least, to validate that the gains of bi-directional schemes in practice DO COME FROM the bi-directional procedure and not more complex model structures, we implement another variant using the same complex structures but without any branch interactions before the decoding stage.

This is done by exchanging the places of $\mathbf{G}_{k}^n$ and $\mathbf{F}_{k}^n$ as illustrated in Fig. <ref> (b), leading to a kind of “self-purification” strategy. Symbol “$\nLeftrightarrow$” in Fig. <ref> (a) means that there is NO interaction between the two branches, , there is only interaction within itself.

Comparisons of the uni-/bidirectional strategies are shown in tab:ablation_bi_model. The comparison results show that
our elaborately designed modules (, RCAM and BPM) jointly cooperate in a bidirectional manner and outperform all unidirectional settings.

Besides, our bidirectional purification scheme (, `full-dup.' in tab:ablation_bi_model) also achieves very notable improvement (2.1% and 1.0% gains in $S_\alpha$ on DAVIS$_{16}$ [82] and MCL [49], respectively) against the “self-purification” variant (, `self-purf.' in tab:ablation_bi_model), which has a similar complex structure, further validating the benefit of the bidirectional behavior claimed in this study.

§.§.§ Relation Between RCAM and BPM

The two introduced modules, , RCAM and BPM, focus on using appearance and motion features while ensuring the information flow between them.

They can work collaboratively under the mutual restraint of our full-duplex strategy, but they cannot be substituted for one another.

This is due to the RCAM transmits the features at each level in a point-to-point manner (, $\mathcal{X}_1 \rightarrow \mathcal{Y}_1 $), and thus, it fits with the progressive feature extraction in the encoder.

The BPM, on the other hand, broadcasts high-level features to low-level features via an interlaced decremental connection in a set-to-point manner (,  $ \{ \mathbf{F}_2^n, \mathbf{F}_3^n, \mathbf{F}_4^n\} \rightarrow \mathbf{G}_2^n$), which is more suitable for the multi-level feature interaction in the decoder.

§ CONCLUSION

In this paper, we present a simple yet efficient framework, termed full-duplex strategy network (), that fully leverages the mutual constraints of appearance and motion cues to address the video object segmentation problem.

It consists of two core modules:

the relational cross-attention module (RCAM) in the encoding stage and the efficient bidirectional purification module (BPM) in the decoding stage.

The former one is used to abstract features from a dual-modality, while the latter is utilized to re-calibrate inconsistant features step-by-step.

We thoroughly validate functional modules of our architecture via extensive experiments, leading to several interesting findings.

Finally,  acts as a unified solution that significantly advances SOTA models for both U-VOS and V-SOD tasks.

In the future, we may extend our scheme to learn short-term and long-term information in an efficient Transformer-like framework [114, 143] to further boost the accurarcy.

[1]
A. Abramov, K. Pauwels, J. Papon, F. Wörgötter, and B. Dellen.
Depth-supported real-time video segmentation with the kinect.
In IEEE WACV, pages 457–464, 2012.

[2]
R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk.
Frequency-tuned salient region detection.
In IEEE CVPR, pages 1597–1604, 2009.

[3]
R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk.
Frequency-tuned salient region detection.
In IEEE CVPR, pages 1597–1604, 2009.

[4]
N. Ballas, L. Yao, C. Pal, and A. Courville.
Delving deeper into convolutional networks for learning video
In ICLR, 2016.

[5]
D. Bharadia, E. McMilin, and S. Katti.
Full duplex radios.
In ACM SIGCOMM, pages 375–386, 2013.

[6]
G. Bhat, F. J. Lawin, M. Danelljan, A. Robinson, M. Felsberg, L. Van Gool, and
R. Timofte.
Learning what to learn for video object segmentation.
In ECCV, pages 777–794, 2020.

[7]
A. Borji, M.-M. Cheng, Q. Hou, H. Jiang, and J. Li.
Salient object detection: A survey.
CVMJ, 5(2):117–150, 2019.

[8]
A. Borji, M.-M. Cheng, H. Jiang, and J. Li.
Salient object detection: A benchmark.
IEEE TIP, 24(12):5706–5722, 2015.

[9]
A. Borji, M.-M. Cheng, H. Jiang, and J. Li.
Salient object detection: A benchmark.
IEEE TIP, 24(12):5706–5722, 2015.

[10]
T. Brox and J. Malik.
Object segmentation by long term analysis of point trajectories.
In ECCV, pages 282–295, 2010.

[11]
S. Caelles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and
L. Van Gool.
One-shot video object segmentation.
In IEEE CVPR, pages 221–230, 2017.

[12]
S. Caelles, J. Pont-Tuset, F. Perazzi, A. Montes, K.-K. Maninis, and
L. Van Gool.
The 2019 davis challenge on vos: Unsupervised multi-object
arXiv preprint arXiv:1905.00737, 2019.

[13]
J. Chang, D. Wei, and J. W. Fisher.
A video representation using temporal superpixels.
In IEEE CVPR, pages 2051–2058, 2013.

[14]
B. Chen, H. Ling, X. Zeng, G. Jun, Z. Xu, and S. Fidler.
Scribblebox: Interactive annotation framework for video object
In ECCV, pages 293–310, 2020.

[15]
C. Chen, G. Wang, C. Peng, X. Zhang, and H. Qin.
Improved robust video saliency detection based on long-term
spatial-temporal information.
IEEE TIP, 29:1090–1100, 2019.

[16]
X. Chen, Z. Li, Y. Yuan, G. Yu, J. Shen, and D. Qi.
State-aware tracker for real-time video object segmentation.
In IEEE CVPR, pages 9384–9393, 2020.

[17]
Y. Chen, W. Zou, Y. Tang, X. Li, C. Xu, and N. Komodakis.
Scom: Spatiotemporal constrained optimization for salient object
IEEE TIP, 27(7):3345–3357, 2018.

[18]
Z. Chen, C. Guo, J. Lai, and X. Xie.
Motion-appearance interactive encoding for object segmentation in
unconstrained videos.
IEEE TCSVT, 30(6):1613–1624, 2019.

[19]
H. K. Cheng, Y.-W. Tai, and C.-K. Tang.
Modular interactive video object segmentation: Interaction-to-mask,
propagation and difference-aware fusion.
In IEEE CVPR, pages 5559–5568, 2021.

[20]
J. Cheng, Y.-H. Tsai, W.-C. Hung, S. Wang, and M.-H. Yang.
Fast and accurate online video object segmentation via tracking
In IEEE CVPR, pages 7415–7424, 2018.

[21]
J. Cheng, Y.-H. Tsai, S. Wang, and M.-H. Yang.
Segflow: Joint learning for video object segmentation and optical
In IEEE ICCV, pages 686–695, 2017.

[22]
M.-M. Cheng, N. J. Mitra, X. Huang, P. H. Torr, and S.-M. Hu.
Global contrast based salient region detection.
IEEE TPAMI, 37(3):569–582, 2015.

[23]
R. Cong, J. Lei, H. Fu, F. Porikli, Q. Huang, and C. Hou.
Video saliency detection via sparsity-based reconstruction and
IEEE TIP, 28(10):4819–4831, 2019.

[24]
M. Ding, Z. Wang, B. Zhou, J. Shi, Z. Lu, and P. Luo.
Every Frame Counts: Joint Learning of Video Segmentation and Optical
In AAAI, pages 10713–10720, 2020.

[25]
B. Duke, A. Ahmed, C. Wolf, P. Aarabi, and G. W. Taylor.
Sstvos: Sparse spatiotemporal transformers for video object
In IEEE CVPR, pages 5912–5921, 2021.

[26]
M. Faisal, I. Akhter, M. Ali, and R. Hartley.
Epo-net: Exploiting geometric constraints on dense trajectories for
motion saliency.
In IEEE WACV, pages 1873–1882, 2020.

[27]
A. Faktor and M. Irani.
Video segmentation by non-local consensus voting.
In BMVC, volume 2, page 8, 2014.

[28]
D.-P. Fan, M.-M. Cheng, Y. Liu, T. Li, and A. Borji.
Structure-measure: A new way to evaluate foreground maps.
In IEEE ICCV, pages 4548–4557, 2017.

[29]
D.-P. Fan, G.-P. Ji, X. Qin, and M.-M. Cheng.
Cognitive vision inspired object segmentation metric and loss
SCIENTIA SINICA Informationis, 2021.

[30]
D.-P. Fan, W. Wang, M.-M. Cheng, and J. Shen.
Shifting more attention to video salient object detection.
In IEEE CVPR, pages 8554–8564, 2019.

[31]
K. Fragkiadaki, G. Zhang, and J. Shi.
Video segmentation by tracing discontinuities in a trajectory
In IEEE CVPR, pages 1846–1853, 2012.

[32]
F. Galasso, R. Cipolla, and B. Schiele.
Video segmentation with superpixels.
In ACCV, pages 760–774, 2012.

[33]
M. Grundmann, V. Kwatra, M. Han, and I. Essa.
Efficient hierarchical graph-based video segmentation.
In IEEE CVPR, pages 2141–2148, 2010.

[34]
Y. Gu, L. Wang, Z. Wang, Y. Liu, M. Cheng, and S. Lu.
Pyramid constrained self-attention network for fast video salient
object detection.
In AAAI, volume 34, pages 10869–10876, 2020.

[35]
K. He, X. Zhang, S. Ren, and J. Sun.
Spatial pyramid pooling in deep convolutional networks for visual
IEEE TPAMI, 37(9):1904–1916, 2015.

[36]
K. He, X. Zhang, S. Ren, and J. Sun.
Deep residual learning for image recognition.
In IEEE CVPR, pages 770–778, 2016.

[37]
Y. Heo, Y. J. Koh, and C.-S. Kim.
Interactive video object segmentation using global and local transfer
In ECCV, pages 297–313, 2020.

[38]
L. Hu, P. Zhang, B. Zhang, P. Pan, Y. Xu, and R. Jin.
Learning position and target consistency for memory-based video
object segmentation.
In IEEE CVPR, pages 4144–4154, 2021.

[39]
P. Hu, G. Wang, X. Kong, J. Kuen, and Y.-P. Tan.
Motion-guided cascaded refinement network for video object
IEEE TPAMI, pages 1400–1409, 2020.

[40]
Y.-T. Hu, J.-B. Huang, and A. G. Schwing.
Unsupervised video object segmentation using motion saliency-guided
spatio-temporal propagation.
In ECCV, pages 786–802, 2018.

[41]
X. Huang, J. Xu, Y.-W. Tai, and C.-K. Tang.
Fast video object segmentation with temporal aggregation network and
dynamic template matching.
In IEEE CVPR, pages 8879–8889, 2020.

[42]
E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox.
Flownet 2.0: Evolution of optical flow estimation with deep networks.
In IEEE CVPR, pages 2462–2470, 2017.

[43]
S. Jain and K. Grauman.
Click carving: Segmenting objects in video with point clicks.
In AAAI (Human Computation and Crowdsourcing), pages 89–98,

[44]
S. D. Jain, B. Xiong, and K. Grauman.
Fusionseg: Learning to combine motion and appearance for fully
automatic segmentation of generic objects in videos.
In IEEE CVPR, pages 2117–2126, 2017.

[45]
G.-P. Ji, Y.-C. Chou, D.-P. Fan, G. Chen, D. Jha, H. Fu, and L. Shao.
Progressively normalized self-attention network for video polyp
In MICCAI, 2021.

[46]
G.-P. Ji, K. Fu, Z. Wu, D.-P. Fan, J. Shen, and L. Shao.
Full-duplex strategy for video object segmentation.
In IEEE ICCV, 2021.

[47]
J. Johnander, M. Danelljan, E. Brissman, F. S. Khan, and M. Felsberg.
A generative appearance model for end-to-end video object
In IEEE CVPR, pages 8953–8962, 2019.

[48]
A. Khoreva, R. Benenson, E. Ilg, T. Brox, and B. Schiele.
Lucid data dreaming for object tracking.
In IEEE CVPRW, 2017.

[49]
H. Kim, Y. Kim, J.-Y. Sim, and C.-S. Kim.
Spatiotemporal saliency detection for video sequences based on random
walk with restart.
IEEE TIP, 24(8):2552–2564, 2015.

[50]
C. Koch and S. Ullman.
Shifts in selective visual attention: towards the underlying neural
In Matters of intelligence, pages 115–141. 1987.

[51]
Y. J. Koh and C.-S. Kim.
Primary object segmentation in videos based on region augmentation
and reduction.
In IEEE CVPR, pages 7417–7425, 2017.

[52]
P. Krähenbühl and V. Koltun.
Efficient inference in fully connected crfs with gaussian edge
In NeurIPS, pages 109–117, 2011.

[53]
M. Lan, Y. Zhang, Q. Xu, and L. Zhang.
E3SN: Efficient End-to-End Siamese Network for Video Object
In IJCAI, pages 701–707, 2020.

[54]
D. Lao and G. Sundaramoorthi.
Extending layered models to 3d motion.
In ECCV, pages 435–451, 2018.

[55]
T.-N. Le and A. Sugimoto.
Deeply supervised 3d recurrent fcn for salient object detection in
In BMVC, volume 1, page 3, 2017.

[56]
T.-N. Le and A. Sugimoto.
Video salient object detection using spatiotemporal deep features.
IEEE TIP, 27(10):5002–5015, 2018.

[57]
S.-H. Lee, W.-D. Jang, and C.-S. Kim.
Contour-constrained superpixels for image and video processing.
In IEEE CVPR, pages 2443–2451, 2017.

[58]
Y. J. Lee, J. Kim, and K. Grauman.
Key-segments for video object segmentation.
In IEEE ICCV, pages 1995–2002, 2011.

[59]
F. Li, T. Kim, A. Humayun, D. Tsai, and J. M. Rehg.
Video segmentation by tracking many figure-ground segments.
In IEEE ICCV, pages 2192–2199, 2013.

[60]
G. Li, Y. Xie, T. Wei, K. Wang, and L. Lin.
Flow guided recurrent neural encoder for video salient object
In IEEE CVPR, pages 3243–3252, 2018.

[61]
H. Li, G. Chen, G. Li, and Y. Yu.
Motion guided attention for video salient object detection.
In IEEE ICCV, pages 7274–7283, 2019.

[62]
S. Li, B. Seybold, A. Vorobyov, X. Lei, and C.-C. Jay Kuo.
Unsupervised video object segmentation with motion-based bilateral
In ECCV, pages 207–223, 2018.

[63]
Y. Li, S. Li, C. Chen, A. Hao, and H. Qin.
Accurate and robust video saliency detection via self-paced
IEEE TMM, 22(5):1153–1167, 2020.

[64]
Y. Li, Z. Shen, and Y. Shan.
Fast video object segmentation using the global context module.
In ECCV, pages 735–750, 2020.

[65]
F. Lin, Y. Chou, and T. Martinez.
Flow adaptive video object segmentation.
Image and Vision Computing, 94:103864, 2020.

[66]
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie.
Feature pyramid networks for object detection.
In IEEE CVPR, pages 2117–2125, 2017.

[67]
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Dollár, and C. L. Zitnick.
Microsoft coco: Common objects in context.
In ECCV, pages 740–755, 2014.

[68]
X. Lu, W. Wang, C. Ma, J. Shen, L. Shao, and F. Porikli.
See more, know more: Unsupervised video object segmentation with
co-attention siamese networks.
In IEEE CVPR, pages 3623–3632, 2019.

[69]
T. Ma and L. J. Latecki.
Maximum weight cliques with mutex constraints for video object
In IEEE CVPR, pages 670–677, 2012.

[70]
W. Maddern, G. Pascoe, C. Linegar, and P. Newman.
1 year, 1000 km: The oxford robotcar dataset.
IJRR, 36(1):3–15, 2017.

[71]
S. Mahadevan, A. Athar, A. Osep, S. Hennen, L. Leal-Taixé, and B. Leibe.
Making a case for 3d convolutions for object segmentation in videos.
In BMVC, 2020.

[72]
J. Miao, Y. Wei, and Y. Yang.
Memory aggregation networks for efficient interactive video object
In IEEE CVPR, pages 10366–10375, 2020.

[73]
K. Min and J. J. Corso.
Tased-net: Temporally-aggregating spatial encoder-decoder network for
video saliency detection.
In IEEE ICCV, pages 2394–2403, 2019.

[74]
D. Nilsson and C. Sminchisescu.
Semantic video segmentation by gated recurrent flow propagation.
In IEEE CVPR, pages 6819–6828, 2018.

[75]
P. Ochs and T. Brox.
Higher order motion models and spectral clustering.
In IEEE CVPR, pages 614–621, 2012.

[76]
P. Ochs, J. Malik, and T. Brox.
Segmentation of moving objects by long term video analysis.
IEEE TPAMI, 36(6):1187–1200, 2013.

[77]
Y. Pan, T. Yao, H. Li, and T. Mei.
Video captioning with transferred semantic attributes.
In IEEE CVPR, pages 6504–6512, 2017.

[78]
A. Papazoglou and V. Ferrari.
Fast object segmentation in unconstrained video.
In IEEE ICCV, pages 1777–1784, 2013.

[79]
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen,
Z. Lin, N. Gimelshein, L. Antiga, et al.
Pytorch: An imperative style, high-performance deep learning library.
In NeurIPS, pages 8024–8035, 2019.

[80]
Q. Peng and Y.-M. Cheung.
Automatic video object segmentation based on visual and motion
IEEE TMM, 21(12):3083–3094, 2019.

[81]
F. Perazzi, A. Khoreva, R. Benenson, B. Schiele, and A. Sorkine-Hornung.
Learning video object segmentation from static images.
In IEEE CVPR, pages 2663–2672, 2017.

[82]
F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and
A. Sorkine-Hornung.
A benchmark dataset and evaluation methodology for video object
In IEEE CVPR, pages 724–732, 2016.

[83]
F. Perazzi, O. Wang, M. Gross, and A. Sorkine-Hornung.
Fully connected object proposals for video segmentation.
In IEEE ICCV, pages 3227–3234, 2015.

[84]
M. Reso, J. Jachalsky, B. Rosenhahn, and J. Ostermann.
Temporally consistent superpixels.
In IEEE ICCV, pages 385–392, 2013.

[85]
A. Robinson, F. J. Lawin, M. Danelljan, F. S. Khan, and M. Felsberg.
Learning fast and robust target models for video object segmentation.
In IEEE CVPR, 2020.

[86]
O. Ronneberger, P. Fischer, and T. Brox.
U-net: Convolutional networks for biomedical image segmentation.
In MICCAI, pages 234–241, 2015.

[87]
S. Seo, J.-Y. Lee, and B. Han.
URVOS: Unified Referring Video Object Segmentation Network with a
Large-Scale Benchmark.
In ECCV, volume 12360, pages 208–223, 2020.

[88]
H. Seong, J. Hyun, and E. Kim.
Kernelized memory network for video object segmentation.
In ECCV, volume 12367, pages 629–645, 2020.

[89]
L. Sevilla-Lara, Y. Liao, F. Güney, V. Jampani, A. Geiger, and M. J. Black.
On the integration of optical flow and action recognition.
In GCPR, pages 281–297, 2018.

[90]
J. Shi and J. Malik.
Motion segmentation and tracking using normalized cuts.
In IEEE ICCV, pages 1154–1160, 1998.

[91]
J. Shi and C. Tomasi.
Good features to track.
In IEEE CVPR, pages 593–600, 1994.

[92]
M. Siam, C. Jiang, S. Lu, L. Petrich, M. Gamal, M. Elhoseiny, and M. Jagersand.
Video object segmentation using teacher-student adaptation in a human
robot interaction (hri) setting.
In IEEE ICRA, pages 50–56, 2019.

[93]
H. Song, W. Wang, S. Zhao, J. Shen, and K.-M. Lam.
Pyramid dilated deeper convlstm for video salient object detection.
In ECCV, pages 715–731, 2018.

[94]
Y. Tang, W. Zou, Z. Jin, Y. Chen, Y. Hua, and X. Li.
Weakly supervised salient object detection with spatiotemporal
cascade neural networks.
IEEE TCSVT, 29(7):1973–1984, 2018.

[95]
Z. Teed and J. Deng.
Raft: Recurrent all-pairs field transforms for optical flow.
In ECCV, volume 12347, pages 402–419, 2020.

[96]
P. Tokmakov, K. Alahari, and C. Schmid.
Learning motion patterns in videos.
In IEEE CVPR, pages 3386–3394, 2017.

[97]
P. Tokmakov, K. Alahari, and C. Schmid.
Learning video object segmentation with visual memory.
In IEEE ICCV, pages 4481–4490, 2017.

[98]
P. Tokmakov, C. Schmid, and K. Alahari.
Learning to segment moving objects.
IJCV, 127(3):282–301, 2019.

[99]
A. M. Treisman and G. Gelade.
A feature-integration theory of attention.
Cognitive psychology, 12(1):97–136, 1980.

[100]
Y.-H. Tsai, M.-H. Yang, and M. J. Black.
Video segmentation via object flow.
In IEEE CVPR, pages 3899–3908, 2016.

[101]
M. Van den Bergh, G. Roig, X. Boix, S. Manen, and L. Van Gool.
Online video seeds for temporal window objectness.
In IEEE ICCV, pages 377–384, 2013.

[102]
P. Voigtlaender, Y. Chai, F. Schroff, H. Adam, B. Leibe, and L.-C. Chen.
Feelvos: Fast end-to-end embedding learning for video object
In IEEE CVPR, pages 9481–9490, 2019.

[103]
H. Wang, C. Deng, F. Ma, and Y. Yang.
Context modulated dynamic networks for actor and action video
segmentation with language queries.
In AAAI, pages 12152–12159, 2020.

[104]
L. Wang, H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, and X. Ruan.
Learning to detect salient objects with image-level supervision.
In IEEE CVPR, pages 136–145, 2017.

[105]
W. Wang, X. Lu, J. Shen, D. J. Crandall, and L. Shao.
Zero-shot video object segmentation via attentive graph neural
In IEEE ICCV, 2019.

[106]
W. Wang, J. Shen, X. Li, and F. Porikli.
Robust video object cosegmentation.
IEEE TIP, 24(10):3137–3148, 2015.

[107]
W. Wang, J. Shen, X. Lu, S. C. Hoi, and H. Ling.
Paying attention to video object pattern understanding.
IEEE TPAMI, 43(7):2413–2428, 2020.

[108]
W. Wang, J. Shen, and F. Porikli.
Saliency-aware geodesic video object segmentation.
In IEEE CVPR, pages 3395–3402, 2015.

[109]
W. Wang, J. Shen, F. Porikli, and R. Yang.
Semi-supervised video object segmentation with super-trajectories.
IEEE TPAMI, 41(4):985–998, 2018.

[110]
W. Wang, J. Shen, and L. Shao.
Video salient object detection via fully convolutional networks.
IEEE TIP, 27(1):38–49, 2017.

[111]
W. Wang, J. Shen, J. Xie, and F. Porikli.
Super-trajectory for video segmentation.
In IEEE ICCV, pages 1671–1679, 2017.

[112]
W. Wang, J. Shen, R. Yang, and F. Porikli.
Saliency-aware video object segmentation.
IEEE TPAMI, 40(1):20–33, 2017.

[113]
W. Wang, H. Song, S. Zhao, J. Shen, S. Zhao, S. C. Hoi, and H. Ling.
Learning unsupervised video object segmentation through visual
In IEEE CVPR, pages 3064–3074, 2019.

[114]
W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and
L. Shao.
Pyramid vision transformer: A versatile backbone for dense prediction
without convolutions.
In IEEE ICCV, 2021.

[115]
Y. Wang, Z. Xu, X. Wang, C. Shen, B. Cheng, H. Shen, and H. Xia.
End-to-end video instance segmentation with transformers.
In IEEE CVPR, 2021.

[116]
Z. Wang, X. Yan, Y. Han, and M. Sun.
Ranking video salient object detection.
In ACM MM, pages 873–881, 2019.

[117]
J. Wei, S. Wang, and Q. Huang.
F3net: Fusion, feedback and focus for salient object detection.
In AAAI, pages 12321–12328, 2020.

[118]
P. Wen, R. Yang, Q. Xu, C. Qian, Q. Huang, R. Cong, and J. Si.
DMVOS: Discriminative matching for real-time video object
In ACM MM, 2020.

[119]
J. M. Wolfe, K. R. Cave, and S. L. Franzel.
Guided search: an alternative to the feature integration model for
visual search.
J EXP PSYCHOL HUMAN, 15(3):419, 1989.

[120]
Z. Wu, L. Su, and Q. Huang.
Stacked cross refinement network for edge-aware salient object
In IEEE ICCV, pages 7264–7273, 2019.

[121]
S. Wug Oh, J.-Y. Lee, K. Sunkavalli, and S. Joo Kim.
Fast video object segmentation by reference-guided mask propagation.
In IEEE CVPR, pages 7376–7385, 2018.

[122]
H. Xiao, B. Kang, Y. Liu, M. Zhang, and J. Feng.
Online meta adaptation for fast video object segmentation.
IEEE TPAMI, 42(5):1205–1217, 2019.

[123]
C. Xu, C. Xiong, and J. J. Corso.
Streaming hierarchical video segmentation.
In ECCV, pages 626–639, 2012.

[124]
M. Xu, B. Liu, P. Fu, J. Li, and Y. H. Hu.
Video saliency detection via graph clustering with motion energy and
spatiotemporal objectness.
IEEE TMM, 21(11):2790–2805, 2019.

[125]
M. Xu, B. Liu, P. Fu, J. Li, Y. H. Hu, and S. Feng.
Video salient object detection via robust seeds extraction and
multi-graphs manifold propagation.
IEEE TCSVT, 30(7):2191–2206, 2019.

[126]
N. Xu, L. Yang, Y. Fan, J. Yang, D. Yue, Y. Liang, B. Price, S. Cohen, and
T. Huang.
Youtube-vos: Sequence-to-sequence video object segmentation.
In ECCV, pages 585–601, 2018.

[127]
P. Yan, G. Li, Y. Xie, Z. Li, C. Wang, T. Chen, and L. Lin.
Semi-supervised video salient object detection using pseudo-labels.
In IEEE ICCV, pages 7284–7293, 2019.

[128]
L. Yang, Y. Fan, and N. Xu.
Video instance segmentation.
In IEEE ICCV, pages 5188–5197, 2019.

[129]
Z. Yang, Q. Wang, L. Bertinetto, W. Hu, S. Bai, and P. H. Torr.
Anchor diffusion for unsupervised video object segmentation.
In IEEE ICCV, pages 931–940, 2019.

[130]
Z. Yang, Y. Wei, and Y. Yang.
Collaborative video object segmentation by foreground-background
In ECCV, volume 12350, pages 332–348, 2020.

[131]
Z. Yin, J. Zheng, W. Luo, S. Qian, H. Zhang, and S. Gao.
Learning to recommend frame for interactive video object segmentation
in the wild.
In IEEE CVPR, pages 15445–15454, 2021.

[132]
L. Zelnik-Manor and M. Irani.
Event-based analysis of video.
In IEEE CVPR, volume 2, pages II–II, 2001.

[133]
K. Zhang, L. Wang, D. Liu, B. Liu, Q. Liu, and Z. Li.
Dual temporal memory network for efficient video object segmentation.
In ACM MM, pages 1515–1523, 2020.

[134]
Y. Zhang, Z. Wu, H. Peng, and S. Lin.
A transductive approach for video object segmentation.
In IEEE CVPR, pages 6949–6958, 2020.

[135]
Z. Zhang, X. Zhang, C. Peng, X. Xue, and J. Sun.
Exfuse: Enhancing feature fusion for semantic segmentation.
In ECCV, pages 269–284, 2018.

[136]
H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.
Pyramid scene parsing network.
In IEEE CVPR, pages 2881–2890, 2017.

[137]
W. Zhao, J. Zhang, L. Li, N. Barnes, N. Liu, and J. Han.
Weakly supervised video salient object detection.
In IEEE CVPR, pages 16826–16835, 2021.

[138]
J. Zheng, W. Luo, and Z. Piao.
Cascaded convlstms using semantically-coherent data synthesis for
video object segmentation.
IEEE Access, 7:132120–132129, 2019.

[139]
T. Zhou, D.-P. Fan, M.-M. Cheng, J. Shen, and L. Shao.
Rgb-d salient object detection: A survey.
CVMJ, pages 1–33, 2021.

[140]
T. Zhou, J. Li, S. Wang, R. Tao, and J. Shen.
Matnet: Motion-attentive transition network for zero-shot video
object segmentation.
IEEE TIP, 29:8326–8338, 2020.

[141]
T. Zhou, S. Wang, Y. Zhou, Y. Yao, J. Li, and L. Shao.
Motion-attentive transition for zero-shot video object segmentation.
In AAAI, pages 13066–13073, 2020.

[142]
X. Zhou, Z. Liu, C. Gong, and W. Liu.
Improving video saliency detection via localized estimation and
spatiotemporal refinement.
IEEE TMM, 20(11):2993–3007, 2018.

[143]
M. Zhuge, D. Gao, D.-P. Fan, L. Jin, B. Chen, H. Zhou, M. Qiu, and L. Shao.
Kaleido-bert: Vision-language pre-training on fashion domain.
In IEEE CVPR, pages 12647–12657, 2021.