Cagan Selim<EMAIL_ADDRESS>Oguzhan<EMAIL_ADDRESS>Jordi<EMAIL_ADDRESS>Fatma<EMAIL_ADDRESS>KUIS AI
Koc University
Istanbul, Turkey Bogazici University
Istanbul, Turkey Google Research
Zurich, Switzerland Two-Level Temporal Relation Model for online VIS

# Two-Level Temporal Relation Model for Online Video Instance Segmentation

###### Abstract

In Video Instance Segmentation (VIS), current approaches either focus on the
quality of the results, by taking the whole video as input and processing it
offline; or on speed, by handling it frame by frame at the cost of competitive
performance. In this work, we propose an online method that is on par with the
performance of the offline counterparts. We introduce a message-passing graph
neural network that encodes objects and relates them through time. We
additionally propose a novel module to fuse features from the feature pyramid
network with residual connections. Our model, trained end-to-end, achieves
state-of-the-art performance on the YouTube-VIS dataset within the online
methods. Further experiments on DAVIS demonstrate the generalization
capability of our model to the video object segmentation task. Code is
available at: https://github.com/caganselim/TLTM

##  1 Introduction

Video Instance Segmentation (VIS) is the task of concurrently detecting,
segmenting, and tracking object instances in videos. The recent progress in
VIS is mainly driven by large datasets [Yang et al.(2019a)Yang, Fan, and Xu,
Xu et al.(2018)Xu, Yang, Fan, Yang, Yue, Liang, Price, Cohen, and Huang, Pont-
Tuset et al.(2017)Pont-Tuset, Perazzi, Caelles, Arbeláez, Sorkine-Hornung, and
Van Gool, Caelles et al.(2019)Caelles, Pont-Tuset, Perazzi, Montes, Maninis,
and Van Gool, Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and
Zisserman] that allow solving these tasks together. The existing methods can
be categorized as _offline_ , when they take the whole video clip as input, or
_online_ , when they process each frame or pair of frames sequentially. Online
methods typically follow the tracking-by-segmentation paradigm by first
performing instance segmentation and then merging instances through time via
an association algorithm [Yang et al.(2021)Yang, Fang, Wang, Li, Fang, Shan,
Feng, and Liu, Fang et al.(2021)Fang, Yang, Wang, Li, Fang, Shan, Feng, and
Liu, Li et al.(2021)Li, Li, Li, and Zhang, Liu et al.(2021)Liu, Cui, Tan, and
Chen, Wu et al.(2021)Wu, Cao, Song, Wang, Yang, and Yuan, Wang et
al.(2021a)Wang, Xu, Chen, and Lin]. While frame-by-frame processing is fast,
it lacks temporal context which results in a large number of ID switches due
to e.g. occlusions. In contrast, offline methods are able to better leverage
the spatio-temporal information from all the frames in the video [Athar et
al.(2020)Athar, Mahadevan, Osep, Leal-Taixé, and Leibe, Bertasius and
Torresani(2020), Lin et al.(2021)Lin, Wu, Liu, Lu, and Jia, Wang et
al.(2021b)Wang, Xu, Wang, Shen, Cheng, Shen, and Xia]. While temporal
information leads to stronger performance, it hurts efficiency compared to the
online counterparts, which might not be suitable for real-time applications.

In this paper, we propose an efficient online method that can reach the
performance of offline methods. We achieve that by modeling the changes to the
representation of the objects at two levels to allow the flow of temporal
information between frames: a Graph Neural Network (GNN) at the object level
and a spatio-temporal feature encoder, coined ResFuser, at the feature level.
We build on top of a one-stage instance segmentation network, CenterMask [Lee
and Park(2020)] and at the feature level, we aggregate features with the
proposed ResFuser module, which uses a skip connection to predict changes from
one frame to the next. At the object level, we estimate the changes to the
states of objects through time and model the interactions between them with a
GNN [Kipf et al.(2019)Kipf, van der Pol, and Welling]: Each object becomes a
node on the graph represented by its encoded state. The states of objects are
updated via message passing between all the objects in consecutive frames. In
the end, we associate the objects with an online tracker across multiple
frames [Yang et al.(2019a)Yang, Fan, and Xu]. Despite the two-level temporal
aggregation, our network can be trained end-to-end.

Our comprehensive experiments on Youtube-VIS show the importance of both the
feature level and the object level aggregation for the competitive results.
Further experiments on DAVIS show our method’s generalization capability to
one more task: Video Object Segmentation (VOS). We summarize our main
contributions as follows:

  * •

ResFuser for feature sharing in the Feature Pyramid Network (FPN) [Lin et
al.(2017)Lin, Dollár, Girshick, He, Hariharan, and Belongie] with residual
connections: This module enables to predict the changes to the feature maps of
consecutive frames at each level of the FPN to relate the two frames at the
feature-level.

  * •

A message-passing GNN that encodes objects and relates them across time: This
module exploits the fact that an object is still the most similar to itself
from one frame to the next, despite some changes which typically occur due to
the objects interacting with each other and with the environment. These
interactions lead to some changes in the appearance of the object which we
learn to estimate with a GNN.

##  2 Related Work

##### Video Instance Segmentation (VIS).

Methods in VIS can be categorized as online or offline, depending on whether
they operate frame-by-frame or take the whole video as input.

Online methods typically add a tracking head to a segmentation network such as
Mask R-CNN [He et al.(2017)He, Gkioxari, Dollár, and Girshick]. In Mask-Track
R-CNN [Yang et al.(2019a)Yang, Fan, and Xu], tracking is formulated as a
multi-class classification problem into one of the already identified
instances or a new unseen instance. This methodology, called tracking-by-
segmentation, extends an instance segmentation method with a matching
algorithm and a history queue. QueryInst [Fang et al.(2021)Fang, Yang, Wang,
Li, Fang, Shan, Feng, and Liu] treats instances as learnable queries with a
multi-stage end-to-end framework and adopts the tracking method proposed in
[Yang et al.(2019a)Yang, Fan, and Xu]. SipMask [Cao et al.(2020)Cao, Anwer,
Cholakkal, Khan, Pang, and Shao] follows the same association strategy by
utilizing a single-stage segmentation network. Single-stage methods for VIS
have progressed quite significantly in recent years, leveraging spatio-
temporal feature fusion techniques. CrossVIS [Yang et al.(2021)Yang, Fang,
Wang, Li, Fang, Shan, Feng, and Liu] uses features of instances in a frame to
localize the same instances in another frame at the pixel level. STMask [Li et
al.(2021)Li, Li, Li, and Zhang] employs a frame-level feature calibration
module between predicted and ground-truth boxes and a temporal fusion module
between consecutive frames to improve the inference of instance masks. SG-Net
[Liu et al.(2021)Liu, Cui, Tan, and Chen] proposes a one-stage framework based
on FCOS [Tian et al.(2019)Tian, Shen, Chen, and He] to improve mask quality by
dividing target instances into sub-regions and performing instance
segmentation on each sub-region.

We propose an online method by using the single-stage CenterMask [Lee and
Park(2020)] as the instance segmentation network. We extend the temporal
context with two modules, ResFuser and GNN, at the feature and object levels,
respectively. VisSTG [Wang et al.(2021a)Wang, Xu, Chen, and Lin] also uses a
GNN to relate the frames, but at the pixel level, which is too costly due to
message passing between many pixels. This, in particular, entails that larger
backbones cannot be utilized due to memory constraints. In contrast, our
method relates objects with a GNN but still benefits from correlations at the
feature level with the ResFuser, which is significantly less costly.

Offline methods operate on the whole video or on multiple frames to make
better use of temporal information but this typically hinders efficiency and
prevents their use in real-time scenarios. STEm-Seg [Athar et al.(2020)Athar,
Mahadevan, Osep, Leal-Taixé, and Leibe], for example, models a video clip as a
spatio-temporal volume where instances are represented as clusters inside the
volume. Instead of costly 3D volume processing, we use a single-stage
segmentation network in the detection phase. Based on DETR [Carion et
al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko], VisTR
[Wang et al.(2021b)Wang, Xu, Wang, Shen, Cheng, Shen, and Xia] uses
transformers, but is not fully end-to-end trainable and has slow convergence
as explained in [Wu et al.(2022)Wu, Yarram, Liang, Lan, Yuan, Eledath, and
Medioni]. The performance of EfficientVIS [Wu et al.(2022)Wu, Yarram, Liang,
Lan, Yuan, Eledath, and Medioni] heavily depends on the number of input
frames. As shown in our experiments, our method taking just 2 frames as input
outperforms the results of EfficientVIS with 9 frames.

##### Video Object Segmentation (VOS).

VOS is the task of segmenting and tracking arbitrary, novel objects without
considering their semantic categories. Depending on the input at test time,
VOS can be divided into semi-supervised (or one-shot), where an initial mask
of the object of interest is given [Maninis et al.(2018)Maninis, Caelles,
Chen, Pont-Tuset, Leal-Taixé, Cremers, and Gool, Perazzi et al.(2017)Perazzi,
Khoreva, Benenson, Schiele, and Sorkine-Hornung, Voigtlaender et
al.(2019)Voigtlaender, Chai, Schroff, Adam, Leibe, and Chen, Wang et
al.(2019a)Wang, Zhang, Bertinetto, Hu, and Torr, Wang et al.(2019d)Wang, Xu,
Liu, Zhu, and Shao], and unsupervised (or zero-shot), without any initial mask
[Lu et al.(2019)Lu, Wang, Ma, Shen, Shao, and Porikli, Ventura et
al.(2019)Ventura, Bellver, Girbau, Salvador, Marques, and Giro-i Nieto, Wang
et al.(2019c)Wang, Song, Zhao, Shen, Zhao, Hoi, and Ling, Zhou et
al.(2020)Zhou, Li, Wang, Tao, and Shen, Yang et al.(2019b)Yang, Wang,
Bertinetto, Hu, Bai, and Torr]. Our method falls into the zero-shot category.
While some benchmarks consider only the dominant object in the scene for
segmenting and tracking [Perazzi et al.(2017)Perazzi, Khoreva, Benenson,
Schiele, and Sorkine-Hornung], in the more generic case the videos have
multiple objects [Pont-Tuset et al.(2017)Pont-Tuset, Perazzi, Caelles,
Arbeláez, Sorkine-Hornung, and Van Gool, Xu et al.(2018)Xu, Yang, Fan, Yang,
Yue, Liang, Price, Cohen, and Huang, Caelles et al.(2019)Caelles, Pont-Tuset,
Perazzi, Montes, Maninis, and Van Gool]. AGNN [Wang et al.(2019b)Wang, Lu,
Shen, Crandall, and Shao] builds a fully connected graph where nodes
correspond to frames and a mask on each frame is related through message
passing between them via an attention mechanism. While this works well for the
case where there is a single foreground object moving in the video, we focus
on the more realistic setting with multiple objects moving and interacting.

##  3 Two-Level Temporal Relation Model

Figure 1: Overall Framework. Our method takes two consecutive frames as input
and relates them temporally at two levels to better segment and track
instances in videos: (i) feature level with ResFuser at each level of the
feature pyramid and (ii) object level with a GNN after encoding objects with
the object encoder.

Our framework consists of two main parts: (i) we modify an instance
segmentation model to aggregate spatio-temporal information between two
consecutive frames and (ii) we use an object encoder to represent the objects
and then relate them with a GNN-based transition model to learn changes to the
representation of the objects. Below we explain the details of each part in
detail.

###  3.1 Spatio-Temporal Feature Aggregation (ResFuser)

We start with an anchor-free image object segmentation method, CenterMask [Lee
and Park(2020)], based on a single-stage object detector, FCOS [Tian et
al.(2019)Tian, Shen, Chen, and He]. FCOS utilizes multi-level predictions with
a Feature Pyramid Network (FPN) to detect objects of various sizes.

Given an input frame at time $t$, FCOS extracts features $\mathbf{p}_{i}^{t}$
at each level $i$. For spatio-temporal feature aggregation, we introduce the
ResFuser module with the following functionality:

$\displaystyle\mathbf{p}_{i}^{(t+1)}=f_{i}(\mathbf{p}_{i}^{(t)},\mathbf{p}_{i}^{(t+1)})+\mathbf{p}_{i}^{(t+1)}$
(1)

where each $f_{i}$ is implemented as a two-layer CNN to learn residual changes
at each level $i=\\{3,\dots,7\\}$. The residual connections allow the fusing
of missing information from the previous frame at each level.

A common approach in the design of segmentation networks is to add a separate
head for segmentation in addition to the detection and the classification
heads of an object detector, e.g. as in Mask R-CNN [He et al.(2017)He,
Gkioxari, Dollár, and Girshick]. CenterMask adds a spatial attention-guided
mask (SAG-Mask) branch to the anchor-free one-stage object detector FCOS [Tian
et al.(2019)Tian, Shen, Chen, and He]. Given the bounding boxes from the
detector, the SAG-Mask branch predicts a segmentation mask for each box with
the spatial attention map. The goal of the attention map is to learn to focus
on the pixels belonging to the object and learn to ignore the others around
the object. The input to the SAG-Mask layer is the features inside the
predicted region of interest (RoI) extracted by RoI Align [He et al.(2017)He,
Gkioxari, Dollár, and Girshick]. Finally, the result of this operation is
upsampled and used to predict class-specific masks. We use the same
classification, regression, centerness, and mask losses as CenterMask. We use
the result of RoI Align also to represent objects as explained next.

###  3.2 Encoding and Relating Objects

##### Object Encoder.

Our object encoder takes the pooled object-centric features
$\mathbf{o}_{t}^{k}$ of CenterMask as input for each object $k$ at time $t$.
These features are used to estimate the mask of the object but are high
dimensional, e.g. $256\times 14\times 14$ with a ResNet-50 (the number of
channels times the size of the spatial grid). We first encode these features
into a low-dimensional latent representation $\mathbf{z}^{k}_{t}$, for each
object $k$ on both frames $t$ and $t+1$:

$\mathbf{z}_{t}^{k}=\mathcal{E}(\mathbf{o}_{t}^{k})\qquad\mathbf{z}_{t+1}^{k}=\mathcal{E}(\mathbf{o}_{t+1}^{k})$
(2)

with an object encoder $\mathcal{E}$ which is implemented as a simple two-
layer CNN followed by an MLP. We relate the objects to each other in this
latent representation as explained next.

##### Relating Objects.

Inspired by C-SWM [Kipf et al.(2019)Kipf, van der Pol, and Welling], we relate
the objects using a Graph Neural Network (GNN). GNN models the state
transitions of objects in the latent space by considering pairwise
interactions between them. While we cannot explicitly model the action on the
object which causes the transition as in C-SWM, we take advantage of an object
being the most similar to itself from one frame to the next.

We construct a fully-connected graph withe the objects as nodes. Our goal is
to learn the state transitions, i.e. the change in the state representation of
each object on the graph from one frame to the next, and we do so by passing
messages between them. The key observation is that the state transition of a
node in the GNN should match the residual in between the object representation
in two consecutive frames. We use the same object encoder in the following
frame and enforce the difference between the state representations of the
first frame and the next to be the state transition estimated by the GNN.

We use the encoded latent representations $\mathbf{z}_{t}^{k}$ as node
features for each object $k\in\\{1,\dots,K\\}$ where $K$ is the number of
objects segmented on frame $t$. Our goal is to learn the transition
$\Delta\mathbf{z}_{t}^{k}$ to model the change from frame $t$ to $t+1$:

$\mathbf{z}_{t+1}^{k}=\mathbf{z}_{t}^{k}+\Delta\mathbf{z}_{t}^{k}$ (3)

at each node/object $k$. The output of the GNN is the set of transitions for
all $K$ objects at frame $t$,
$\Delta\mathbf{z}_{t}=\\{\Delta\mathbf{z}_{t}^{1},\dots,\Delta\mathbf{z}_{t}^{K}\\}$.

While learning the transitions, GNN relates objects to each other by passing
messages between them. The message passing operation is performed by
iteratively updating the node representations:

$\displaystyle\mathbf{e}_{t}^{i,j}=f_{(}\mathbf{z}_{t}^{i},~{}\mathbf{z}_{t+1}^{j})$
$\displaystyle\hfill\Delta\mathbf{z}_{t}^{j}=f_{n}(\mathbf{z}_{t}^{j},~{}\sum_{i\neq
j}\mathbf{e}_{t}^{i,j})$ (4)

where $f_{n}$ and $f_{e}$ are the node and the edge update functions,
respectively; and $\mathbf{e}_{t}^{i,j}$ is the edge representation for the
edge between the node $i$ and $j$ at time $t$. After one step of message
passing, nodes become more aware of the other objects in the scene as well as
their transitions.

We train the network a binary-cross-entropy loss for associating instances in
a video. We define a positive edge between the ground-truth instance node of
an object in the first frame and a proposal node that belongs to the same
object in the next frame. During inference, we employ the same tracking scheme
as MaskTrack R-CNN [Yang et al.(2019a)Yang, Fan, and Xu] with the following
modification on the score matrix: We use the predicted edge scores instead of
the inner-product of object vectors as done in [Yang et al.(2019a)Yang, Fan,
and Xu].

##  4 Experiments

###  4.1 Training

##### Pre-training.

We first train CenterMask [Lee and Park(2020)] on a combination of COCO [Lin
et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick],
YouTube-VIS [Yang et al.(2019a)Yang, Fan, and Xu] and Open Images [Kuznetsova
et al.(2020)Kuznetsova, Rom, Alldrin, Uijlings, Krasin, Pont-Tuset, Kamali,
Popov, Malloci, Kolesnikov, et al.] for the 40 classes of YouTube-VIS 2019,
similarly to [Luiten et al.(2020)Luiten, Zulfikar, and Leibe]. We used 19
overlapping classes of COCO and we map 60 similar classes of Open Images to
the 40 classes of YouTube-VIS (see the Supplementary for the details of the
mapping), i.e. similar classes are represented with samples from both
datasets.

##### Training.

For data augmentation, we apply random affine transformations and motion blur
as in [Athar et al.(2020)Athar, Mahadevan, Osep, Leal-Taixé, and Leibe] to
simulate video movement from static images. We use the same losses as in
CenterMask [Lee and Park(2020)] except MaskIoU [Huang et al.(2019)Huang,
Huang, Gong, Huang, and Wang], which we have observed to add little to the
performance (see the Supplementary for details). We train our method using
Stochastic Gradient Descent (SGD) for 180K iterations with a batch size of 16
and an initial learning rate of $5e^{-3}$. We decrease the learning rate by a
factor of 10 after 100K and 150K iterations. We will share the code and the
pre-trained models upon publication.

Method (ResNet-50) | Aug. | FPS | AP | AP50 | AP75 | AR1 | AR10
---|---|---|---|---|---|---|---
MaskTrack R-CNN [Yang et al.(2019a)Yang, Fan, and Xu] | - | 33 | 30.3 | 51.1 | 32.6 | 31.0 | 35.5
STEm-Seg [Athar et al.(2020)Athar, Mahadevan, Osep, Leal-Taixé, and Leibe] | - | 7 | 30.6 | 50.7 | 33.5 | 31.6 | 37.1
SipMask [Cao et al.(2020)Cao, Anwer, Cholakkal, Khan, Pang, and Shao] | - | 34 | 32.5 | 53.0 | 33.3 | 33.5 | 38.9
CompFeat [Fu et al.(2020)Fu, Yang, Liu, Huang, and Shi] | - |  $<$33 | 35.3 | 56.0 | 38.6 | 33.1 | 40.3
TraDeS [Wu et al.(2021)Wu, Cao, Song, Wang, Yang, and Yuan] | - | 26 | 32.6 | 52.6 | 32.8 | 29.1 | 36.6
CrossVIS [Yang et al.(2021)Yang, Fang, Wang, Li, Fang, Shan, Feng, and Liu] | - | 40 | 34.8 | 54.6 | 37.9 | 34.0 | 39.0
STMask [Li et al.(2021)Li, Li, Li, and Zhang] |  DCN [Dai et al.(2017)Dai, Qi, Xiong, Li, Zhang, Hu, and Wei] | 29 | 33.5 | 52.1 | 36.9 | 31.1 | 39.2
SG-Net [Liu et al.(2021)Liu, Cui, Tan, and Chen] | MS | 23 | 34.8 | 56.1 | 36.8 | 35.8 | 40.8
Ours | MS | 15 | 35.8 | 57.1 | 38.0 | 34.7 | 41.2
QueryInst [Fang et al.(2021)Fang, Yang, Wang, Li, Fang, Shan, Feng, and Liu] | MS | 32 | 36.2 | 56.7 | 39.7 | 36.1 | 42.9
CrossVIS [Yang et al.(2021)Yang, Fang, Wang, Li, Fang, Shan, Feng, and Liu] | MS | 40 | 36.3 | 56.8 | 38.9 | 35.6 | 40.7
VisSTG [Wang et al.(2021a)Wang, Xu, Chen, and Lin] | MS | 22 | 36.5 | 58.6 | 39.0 | 35.5 | 40.8

Method (ResNet-101) | Aug. | FPS | AP | AP50 | AP75 | AR1 | AR10
---|---|---|---|---|---|---|---
MaskTrack R-CNN [Yang et al.(2019a)Yang, Fan, and Xu] | - | 29 | 31.9 | 53.7 | 32.3 | 32.5 | 37.7
SRNet [Ying et al.(2021)Ying, Li, and Chuah] | - | 35 | 32.3 | 50.2 | 34.8 | 32.3 | 40.1
STEm-Seg [Athar et al.(2020)Athar, Mahadevan, Osep, Leal-Taixé, and Leibe] | - | 7 | 34.6 | 55.8 | 37.9 | 34.4 | 41.6
CrossVIS [Yang et al.(2021)Yang, Fang, Wang, Li, Fang, Shan, Feng, and Liu] | - | 36 | 36.6 | 57.3 | 39.7 | 36.0 | 42.0
SipMask [Cao et al.(2020)Cao, Anwer, Cholakkal, Khan, Pang, and Shao] | MS | 24 | 35.8 | 56.0 | 39.0 | 35.4 | 42.4
STMask [Li et al.(2021)Li, Li, Li, and Zhang] |  DCN [Dai et al.(2017)Dai, Qi, Xiong, Li, Zhang, Hu, and Wei] | 23 | 36.8 | 56.8 | 38.0 | 34.8 | 41.8
SG-Net [Liu et al.(2021)Liu, Cui, Tan, and Chen] | MS | 20 | 36.3 | 57.1 | 39.6 | 35.9 | 43.0
Ours | MS | 14 | 38.1 | 61.9 | 40.6 | 37.0 | 44.4

Table 1: Results on YouTube-VIS 2019. We compare our method to other online
state-of-the-art methods on YouTube-VIS 2019 val set using ResNet-50 (top) and
ResNet-101 (bottom) as backbones. “MS” denotes multi-scale training.

###  4.2 Benchmarks

##### YouTube-VIS 2019 [Yang et al.(2019a)Yang, Fan, and Xu].

The YouTube Video Instance Segmentation 2019 dataset is the first large
benchmark introduced for the VIS task. It consists of 2883 high-quality
YouTube videos containing 4883 unique objects annotated with approximately
131k object masks corresponding to 40 predetermined object categories. The
task is to segment and classify each object instance while consistently
tracking them across frames. The evaluation metrics are Average Precision (AP)
and Average Recall (AR). These metrics are evaluated over 10 IoU thresholds
from 50% to 95% at a step of 5%, and calculated firstly by category and
finally averaged over the category set.

##### DAVIS 2019 [Caelles et al.(2019)Caelles, Pont-Tuset, Perazzi, Montes,
Maninis, and Van Gool].

Densely Annotated Video Segmentation 2019 is a high-quality dataset that is
popular in the VOS task. In its 2019 version, there are a total of 90
sequences, 60 for training and 30 for validation. We evaluate our framework on
the Unsupervised Video Object Segmentation (UVOS) task, i.e. to segment and
track foreground objects without classifying them, unlike VIS. The main
evaluation metric is the $J\&F$ score, which is the mean of $J$-score and
$F$-score. The former is based on the IoU between the predicted and the ground
masks whereas $F$-score measures the accuracy of the predicted mask
boundaries.

###  4.3 Quantitative Results

####  4.3.1 Video Instance Segmentation

Table 1 compares our method to online and real-time state-of-the-art VIS
methods on YouTube-VIS 2019. Following the literature, we perform a separate
comparison by using a small (ResNet-50 [He et al.(2016)He, Zhang, Ren, and
Sun]) and a large (ResNet-101 [He et al.(2016)He, Zhang, Ren, and Sun])
backbone. The results show that our model outperforms the other models by 1.8
points with the ResNet-101 backbone while preserving a competitive performance
with ResNet-50. . Our model extends the capability of online models to benefit
more from the temporal context with the two proposed modules, GNN and
ResFuser. Furthermore, our model can better utilize a large number of
parameters as shown in the SoTA performance with the large backbone. Since
these two modules have additional parameters, more data is required, both in
terms of quantity and variety as shown in our ablations (Section 4.4). The
only other model that employs a GNN for feature aggregation is VisSTG [Wang et
al.(2021a)Wang, Xu, Chen, and Lin] but their results are not reported with the
ResNet-101 backbone, probably due to overly increased computational and memory
requirements.

Note that offline methods which make use of the full sequence are not included
in Table 1 for a fair comparison. For reference, VisTR [Wang et
al.(2021b)Wang, Xu, Wang, Shen, Cheng, Shen, and Xia], a real-time and end-to-
end trainable encoder-decoder transformer-based framework, reports a score of
35.6 mAP with ResNet-50 backbone and 38.6 mAP with ResNet-101 backbone with
$T=36$ frames as input, which is the largest amount of annotated frames on
YouTube-VIS-19 dataset. Similarly, the state-of-the-art EfficientVIS [Wu et
al.(2022)Wu, Yarram, Liang, Lan, Yuan, Eledath, and Medioni] achieves 37.9 mAP
with ResNet-50 and 39.8 mAP with ResNet-101 backbone using $T=36$. These high
scores can be attributed to rich information available on the whole video
level as proven by a score of 35.3 mAP when reducing the temporal window to
$T=9$ input frames [Wu et al.(2022)Wu, Yarram, Liang, Lan, Yuan, Eledath, and
Medioni], while our method can obtain 35.8 mAP with just $T=2$ frames.
Although STEm-Seg [Athar et al.(2020)Athar, Mahadevan, Osep, Leal-Taixé, and
Leibe] is also a whole-video-level method, we still include it in our
comparisons because, to the best of our knowledge, it is the first method that
reports results on both VIS and VOS tasks like us.

####  4.3.2 Video Object Segmentation

Method | Online | E2E | J & F | Mean | Recall | Decay | Mean | Recall | Decay
---|---|---|---|---|---|---|---|---|---
RVOS [Ventura et al.(2019)Ventura, Bellver, Girbau, Salvador, Marques, and Giro-i Nieto] |  |  | 41.2 | 36.8 | 40.2 | 0.5 | 45.7 | 46.4 | 1.7
KIS [Cho et al.(2019)Cho, Hong, Kang, and Kim] | - | 59.9 | - | - | - | - | - | -
Ours |  | 61.9 | 60.7 | 70.3 | -2.4 | 63.1 | 70.9 | 1.9
AGNN [Wang et al.(2019b)Wang, Lu, Shen, Crandall, and Shao] | - |  - | 61.1 | 58.9 | 65.7 | 11.7 | 63.2 | 67.1 | 14.3
STEm-Seg [Athar et al.(2020)Athar, Mahadevan, Osep, Leal-Taixé, and Leibe] |  | 64.7 | 61.5 | 70.4 | -4 | 67.8 | 75.5 | 1.2
UnOVOST [Luiten et al.(2020)Luiten, Zulfikar, and Leibe] | - | 67.9 | 66.4 | 76.4 | -0.2 | 69.3 | 76.9 | 0.01
Propose-Reduce [Lin et al.(2021)Lin, Wu, Liu, Lu, and Jia] |  | 68.3 | 65.0 | - | - | 71.6 | - | -

Table 2: Results on DAVIS-19. We compare our method to state-of-the-art
methods, online at the top and offline at the bottom, on DAVIS-19 Unsupervised
val set. All results are reported with the ResNet-101 backbone.

We evaluate our model’s performance on the additional task of VOS in Table 2
without training on it. Our method considerably outperforms previous online
methods RVOS [Ventura et al.(2019)Ventura, Bellver, Girbau, Salvador, Marques,
and Giro-i Nieto] and KIS [Cho et al.(2019)Cho, Hong, Kang, and Kim], and
achieves a slightly better $J\&F$ score with a noticeably less decay than
offline AGNN [Wang et al.(2019b)Wang, Lu, Shen, Crandall, and Shao]. Our
method is only 2.5 points below the offline STEm-Seg, despite the 14-frame
video input used by their method. With $T=4$ input frames, STEm-Seg reports a
$J\&F$ score of 62.2 which is similar to our method’s performance (61.9) with
only $T=2$ input frames. Furthermore, STEm-Seg is trained on both image (COCO
[Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and
Zitnick] and PASCAL [Everingham et al.(2010)Everingham, Van Gool, Williams,
Winn, and Zisserman]) and video datasets (YouTube-VIS and DAVIS) whereas our
method is not even trained on DAVIS, which shows a better generalization
performance to the video object segmentation task. Although the offline
UnOVOST [Luiten et al.(2020)Luiten, Zulfikar, and Leibe] method achieves an
impressive score of 67.9, it cannot be trained end-to-end due to its complex
post-processing based on heuristics that are specific to this benchmark. Also,
it is significantly slower (1 vs. 14 FPS). Propose-Reduce [Lin et
al.(2021)Lin, Wu, Liu, Lu, and Jia], current state of the art on DAVIS-19, is
another offline method based on performing segmentation on keyframes and then
propagating it with a two-stage network. Both the performance and efficiency
are highly dependent on the number of keyframes.

###  4.4 Ablation Study

| YTVIS-19 (mAP) | DAVIS-19 (J&F)
---|---|---
No Open Images | 28.0 | 54.7
Open Images | 34.3 | 59.1
Open Images + ResFuser | 34.9 | 59.4
Open Images + ResFuser + GNN | 35.8 | 60.5
Table 3: Ablation Study. We show the effect of pre-training on OpenImages
[Kuznetsova et al.(2020)Kuznetsova, Rom, Alldrin, Uijlings, Krasin, Pont-
Tuset, Kamali, Popov, Malloci, Kolesnikov, et al.], and our two contributions,
ResFuser and GNN on YoutubeVIS and DAVIS. All results are presented with the
ResNet-50 backbone.

Table 3 presents our ablation experiments to isolate the effect of each of our
contributions. The first row is a baseline with CenterMask pre-trained only on
YouTube-VIS-19 [Yang et al.(2019a)Yang, Fan, and Xu] and COCO [Lin et
al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick].
Here, we use an IoU-based tracker by removing the Kalman Filter from the well-
known SORT algorithm [Bewley et al.(2016)Bewley, Ge, Ott, Ramos, and Upcroft].
In the second row, we pre-train CenterMask [Lee and Park(2020)] additionally
on Open Images [Kuznetsova et al.(2020)Kuznetsova, Rom, Alldrin, Uijlings,
Krasin, Pont-Tuset, Kamali, Popov, Malloci, Kolesnikov, et al.] to see the
effect of a larger-scale image instance segmentation dataset on video
segmentation problems. Although Open Images is not typically used for video
segmentation problems, our framework greatly benefits from this additional
data as it relies on instance detection and segmentation on the frame level.
We observe a significant boost of +6.3 mAP on YouTube-VIS-19 and +4.4 $J\&F$
on DAVIS-19.

We then add our feature-sharing module ResFuser which is a simple feature-
sharing procedure between the consecutive frames to improve detection and
segmentation by exploiting similar feature representations of corresponding
instances. Despite being conceptually very simple, ResFuser consistently
improves the results on both datasets too (+0.6 mAP on YouTube-VIS-19 and +0.3
$J\&F$ on DAVIS-19).

Finally, the last row shows that linking objects across time with a GNN
further improves the performance by a noticeable margin (+0.9 mAP in YouTube-
VIS-19). The gain on the DAVIS-19 unsupervised validation set is even higher
(+1.1 $J\&F$). This reinforces our claim that GNN enhances instance
segmentation and tracking in multiple objects, which is the case for DAVIS-19
as it contains a large number of sequences with 2 or 3 objects (the average
number of objects per sequence is 2.2 on the validation set [Caelles et
al.(2019)Caelles, Pont-Tuset, Perazzi, Montes, Maninis, and Van Gool]). Our
method of relating objects with a GNN shows promising improvements for both
video segmentation tasks.

###  4.5 Qualitative Analysis

We show two qualitative examples in Fig. 2 (see Supplementary for the videos)
comparing the results with our GNN module versus those without. In the case of
VIS, our model with GNN can correctly segment and track the tennis racket and
the two players despite frequent exiting and re-entering throughout the video.
The model without GNN, on the other hand, assigns a different ID to each
object when they re-enter the view, showing the benefits of our object-level
reasoning in the GNN.

In the VOS video, we demonstrate our model’s capability to segment and track
object instances with similar appearances and frequent occlusions. The two
fish occluding each other on the first frame (1 and 5) can be separated
correctly with the help of GNN while they are treated as a single object
without GNN. In the next frame, a fish makes a fast turn and in the third
frame, another fish swims at the bottom edge of the frame. These situations
significantly affect object appearances, and GNN continues to detect and track
these fish as opposed to the model without GNN. Finally, in the last frame,
our model with GNN can segment and identify the fish entering the frame (6) as
a new object despite being heavily camouflaged.

Figure 2: Qualitative Results.

##  5 Conclusion

We proposed a new end-to-end trainable framework for simultaneously detecting,
segmenting, classifying, and tracking objects in videos in an online fashion.
We introduced two key novelties: an effective skip-connection module
(ResFuser) that allows feature sharing between consecutive frames and an
object-relating GNN. The two together reach state-of-the-art results on the
challenging YouTube-VIS dataset, with an AP of 38.1 at 14 frames per second.
We showed that our framework generalizes well to the VOS task by achieving
competitive results without ever seeing the corresponding training data.

## References

  * [Athar et al.(2020)Athar, Mahadevan, Osep, Leal-Taixé, and Leibe] Ali Athar, Sabarinath Mahadevan, Aljosa Osep, Laura Leal-Taixé, and Bastian Leibe.  Stem-seg: Spatio-temporal embeddings for instance segmentation in videos.  In _Proc. of the European Conf. on Computer Vision (ECCV)_ , 2020\.
  * [Bertasius and Torresani(2020)] Gedas Bertasius and Lorenzo Torresani.  Classifying, segmenting, and tracking object instances in video with mask propagation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [Bewley et al.(2016)Bewley, Ge, Ott, Ramos, and Upcroft] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft.  Simple online and realtime tracking.  In _Proc. IEEE International Conf. on Image Processing (ICIP)_ , 2016\.
  * [Caelles et al.(2019)Caelles, Pont-Tuset, Perazzi, Montes, Maninis, and Van Gool] Sergi Caelles, Jordi Pont-Tuset, Federico Perazzi, Alberto Montes, Kevis-Kokitsi Maninis, and Luc Van Gool.  The 2019 DAVIS challenge on VOS: Unsupervised multi-object segmentation.  _arXiv:1905.00737_ , 2019.
  * [Cao et al.(2020)Cao, Anwer, Cholakkal, Khan, Pang, and Shao] Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao.  Sipmask: Spatial information preservation for fast image and video instance segmentation.  In _Proc. of the European Conf. on Computer Vision (ECCV)_ , 2020\.
  * [Carion et al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _Proc. of the European Conf. on Computer Vision (ECCV)_ , 2020\.
  * [Cho et al.(2019)Cho, Hong, Kang, and Kim] Donghyeon Cho, Sungeun Hong, Sungil Kang, and Jiwon Kim.  Key instance selection for unsupervised video object segmentation.  _arXiv preprint arXiv:1906.07851_ , 2019.
  * [Dai et al.(2017)Dai, Qi, Xiong, Li, Zhang, Hu, and Wei] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2017.
  * [Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _International Journal of Computer Vision (IJCV)_ , 88(2):303–338, 2010.
  * [Fang et al.(2021)Fang, Yang, Wang, Li, Fang, Shan, Feng, and Liu] Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu.  Instances as queries.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2021.
  * [Fu et al.(2020)Fu, Yang, Liu, Huang, and Shi] Yang Fu, Linjie Yang, Ding Liu, Thomas S Huang, and Humphrey Shi.  Compfeat: Comprehensive feature aggregation for video instance segmentation.  _arXiv preprint arXiv:2012.03400_ , 2020.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * [He et al.(2017)He, Gkioxari, Dollár, and Girshick] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2017.
  * [Huang et al.(2019)Huang, Huang, Gong, Huang, and Wang] Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang.  Mask scoring r-cnn.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Kipf et al.(2019)Kipf, van der Pol, and Welling] Thomas Kipf, Elise van der Pol, and Max Welling.  Contrastive learning of structured world models.  _arXiv preprint arXiv:1911.12247_ , 2019.
  * [Kuznetsova et al.(2020)Kuznetsova, Rom, Alldrin, Uijlings, Krasin, Pont-Tuset, Kamali, Popov, Malloci, Kolesnikov, et al.] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al.  The open images dataset v4.  _International Journal of Computer Vision (IJCV)_ , 128(7):1956–1981, 2020.
  * [Lee and Park(2020)] Youngwan Lee and Jongyoul Park.  Centermask : Real-time anchor-free instance segmentation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [Li et al.(2021)Li, Li, Li, and Zhang] Minghan Li, Shuai Li, Lida Li, and Lei Zhang.  Spatial feature calibration and temporal fusion for effective one-stage video instance segmentation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [Lin et al.(2021)Lin, Wu, Liu, Lu, and Jia] Huaijia Lin, Ruizheng Wu, Shu Liu, Jiangbo Lu, and Jiaya Jia.  Video instance segmentation with a propose-reduce paradigm.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2021.
  * [Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Proc. of the European Conf. on Computer Vision (ECCV)_ , 2014\.
  * [Lin et al.(2017)Lin, Dollár, Girshick, He, Hariharan, and Belongie] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [Liu et al.(2021)Liu, Cui, Tan, and Chen] Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingjie Chen.  Sg-net: Spatial granularity network for one-stage video instance segmentation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [Lu et al.(2019)Lu, Wang, Ma, Shen, Shao, and Porikli] Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli.  See more, know more: Unsupervised video object segmentation with co-attention siamese networks.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Luiten et al.(2020)Luiten, Zulfikar, and Leibe] Jonathon Luiten, Idil Esen Zulfikar, and Bastian Leibe.  UnOVOST: Unsupervised offline video object segmentation and tracking.  In _Proc. of the IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2020.
  * [Maninis et al.(2018)Maninis, Caelles, Chen, Pont-Tuset, Leal-Taixé, Cremers, and Gool] Kevis-Kokitsi Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool.  Video object segmentation without temporal information.  _IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI)_ , 2018.
  * [Perazzi et al.(2017)Perazzi, Khoreva, Benenson, Schiele, and Sorkine-Hornung] Federico Perazzi, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, and Alexander Sorkine-Hornung.  Learning video object segmentation from static images.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [Pont-Tuset et al.(2017)Pont-Tuset, Perazzi, Caelles, Arbeláez, Sorkine-Hornung, and Van Gool] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  _arXiv preprint arXiv:1704.00675_ , 2017.
  * [Tian et al.(2019)Tian, Shen, Chen, and He] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  FCOS: Fully convolutional one-stage object detection.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2019.
  * [Ventura et al.(2019)Ventura, Bellver, Girbau, Salvador, Marques, and Giro-i Nieto] Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques, and Xavier Giro-i Nieto.  Rvos: End-to-end recurrent network for video object segmentation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Voigtlaender et al.(2019)Voigtlaender, Chai, Schroff, Adam, Leibe, and Chen] Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and Liang-Chieh Chen.  Feelvos: Fast end-to-end embedding learning for video object segmentation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Wang et al.(2019a)Wang, Zhang, Bertinetto, Hu, and Torr] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr.  Fast online object tracking and segmentation: A unifying approach.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019a.
  * [Wang et al.(2021a)Wang, Xu, Chen, and Lin] Tao Wang, Ning Xu, Kean Chen, and Weiyao Lin.  End-to-end video instance segmentation via spatial-temporal graph neural networks.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2021a.
  * [Wang et al.(2019b)Wang, Lu, Shen, Crandall, and Shao] Wenguan Wang, Xiankai Lu, Jianbing Shen, David J Crandall, and Ling Shao.  Zero-shot video object segmentation via attentive graph neural networks.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2019b.
  * [Wang et al.(2019c)Wang, Song, Zhao, Shen, Zhao, Hoi, and Ling] Wenguan Wang, Hongmei Song, Shuyang Zhao, Jianbing Shen, Sanyuan Zhao, Steven C. H. Hoi, and Haibin Ling.  Learning unsupervised video object segmentation through visual attention.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019c.
  * [Wang et al.(2021b)Wang, Xu, Wang, Shen, Cheng, Shen, and Xia] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia.  End-to-end video instance segmentation with transformers.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2021b.
  * [Wang et al.(2019d)Wang, Xu, Liu, Zhu, and Shao] Ziqin Wang, Jun Xu, Li Liu, Fan Zhu, and Ling Shao.  RANet: Ranking attention network for fast video object segmentation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019d.
  * [Wu et al.(2021)Wu, Cao, Song, Wang, Yang, and Yuan] Jialian Wu, Jiale Cao, Liangchen Song, Yu Wang, Ming Yang, and Junsong Yuan.  Track to detect and segment: An online multi-object tracker.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [Wu et al.(2022)Wu, Yarram, Liang, Lan, Yuan, Eledath, and Medioni] Jialian Wu, Sudhir Yarram, Hui Liang, Tian Lan, Junsong Yuan, Jayan Eledath, and Gerard Medioni.  Efficient video instance segmentation via tracklet query and proposal.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [Xu et al.(2018)Xu, Yang, Fan, Yang, Yue, Liang, Price, Cohen, and Huang] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang.  YouTube-VOS: Sequence-to-sequence video object segmentation.  In _Proc. of the European Conf. on Computer Vision (ECCV)_ , 2018\.
  * [Yang et al.(2019a)Yang, Fan, and Xu] Linjie Yang, Yuchen Fan, and Ning Xu.  Video instance segmentation.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2019a.
  * [Yang et al.(2021)Yang, Fang, Wang, Li, Fang, Shan, Feng, and Liu] Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu.  Crossover learning for fast online video instance segmentation.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2021.
  * [Yang et al.(2019b)Yang, Wang, Bertinetto, Hu, Bai, and Torr] Zhao Yang, Qiang Wang, Luca Bertinetto, Weiming Hu, Song Bai, and Philip HS Torr.  Anchor diffusion for unsupervised video object segmentation.  In _Proc. of the IEEE International Conf. on Computer Vision (ICCV)_ , 2019b.
  * [Ying et al.(2021)Ying, Li, and Chuah] Xiaowen Ying, Xin Li, and Mooi Choo Chuah.  Srnet: Spatial relation network for efficient single-stage instance segmentation in videos.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021.
  * [Zhou et al.(2020)Zhou, Li, Wang, Tao, and Shen] Tianfei Zhou, Jianwu Li, Shunzhou Wang, Ran Tao, and Jianbing Shen.  MATNet: Motion-attentive transition network for zero-shot video object segmentation.  _IEEE Trans. on Image Processing (TIP)_ , 29, 2020.