Subhabrata<EMAIL_ADDRESS>Laurynas
<EMAIL_ADDRESS>Iro<EMAIL_ADDRESS>Andrea
<EMAIL_ADDRESS>Christian<EMAIL_ADDRESS>Visual Geometry Group
University of Oxford
Oxford, UK Guess What Moves

# Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating
Motion

###### Abstract

Motion, measured via optical flow, provides a powerful cue to discover and
learn objects in images and videos. However, compared to using appearance, it
has some blind spots, such as the fact that objects become invisible if they
do not move. In this work, we propose an approach that combines the strengths
of motion-based and appearance-based segmentation. We propose to supervise an
image segmentation network with the pretext task of predicting regions that
are likely to contain simple motion patterns, and thus likely to correspond to
objects. As the model only uses a single image as input, we can apply it in
two settings: unsupervised video segmentation, and unsupervised image
segmentation. We achieve state-of-the-art results for videos, and demonstrate
the viability of our approach on still images containing novel objects.
Additionally we experiment with different motion models and optical flow
backbones and find the method to be robust to these change. Project page and
code available at https://www.robots.ox.ac.uk/~vgg/research/gwm.

**footnotetext: Authors contributed equally.

##  1 Introduction

The motion of objects in a video can be detected by methods such as optical
flow and used to discover and segment them. A key benefit is that optical flow
is object-agnostic: because it relies on low-level visual properties, it can
extract a signal even before the objects are discovered, and can thus be used
to establish an understanding of objectness.

The potential of motion as a cue is epitomized in _video segmentation_
problems, where the input is a generic video sequence and the task is to
extract the main object(s) in the video. In fact, some methods [Yang et al.
(2021a), Meunier et al. (2022)] adopt a motion-_only_ approach to video object
segmentation, arguing that motion patterns are much easier to model and
interpret than appearance. However, this approach ignores appearance cues and
is ‘blind’ to stationary objects.

Instead, we propose to use motion as _supervision_ to discover objects in
videos _and_ still images without the need for manual annotations. We observe
that different objects tend to generate distinctive optical flow patterns
which can be well approximated by small parametric models, such as affine or
quadratic. We use this fact to train a segmentation network that, given a
_single RGB frame_ as input, predicts _which image regions_ are likely to
contain such patterns. The idea is that these regions would then separate the
objects from the background.

This approach has several useful properties. First, while motion is used for
supervising the network, the latter implicitly learns the appearance of the
objects, regularizing the segmentation. Second, because the network works with
a single image as input, it does not observe the motion directly. The model
must anticipate what _could_ move, extracting objects even if they are not in
motion. Third, the network avoids predicting the objects’ motion directly,
which is a highly-ambiguous task given a single image as input; instead, it
predicts only the support regions of the motion patterns, and the training
loss measures the compatibility of such regions with the observed motion
according to the assumed motion model.

While we are not the first to consider motion as a cue for decomposing an
image into objects, our particular way of modeling motion is simple and
versatile, and allows two application modes of our approach. First, we
consider _internal learning for unsupervised motion segmentation_ [Ulyanov et
al. (2020)]. Given one or more videos as input (without labels), we optimize a
network, as described above, to output a segmentation of the videos,
effectively ‘observing’ motion via backpropagation. Our approach achieves
state-of-the-art performance on standard benchmarks for unsupervised motion
segmentation [Yang et al. (2019), Yang et al. (2021a)].

The second mode is _transductive learning for unsupervised image segmentation_
, which is intended to assess the generalization capabilities of our model as
an image segmenter. In this case, the network is first trained on a number of
training videos and then evaluated on a disjoint set of images. Since only
appearance information is available at test time, the problem solved is not
motion segmentation, but image segmentation. In this scenario, our model
segments novel objects not observed during training, demonstrating the
viability of our approach.

##  2 Related Work

Our work aims to combine motion and appearance cues for unsupervised object
discovery, in that motion can be used as a cue to learn a general object
segmenter for both videos and images. As such, there exist several related
areas in literature, which we review next.

#### Unsupervised Video Object Segmentation.

The aim of video object segmentation (VOS) is to densely label objects present
in a video. Current VOS benchmarks [Perazzi et al. (2016), Li et al. (2013),
Ochs et al. (2014)] usually define the problem as foreground-background
separation, where the foreground comprises the most salient objects. Efforts
to reduce the amount of supervision follow two main directions, semi-
supervised and unsupervised VOS. Semi-supervised methods require manual
annotations for the object(s) of interest in an initial frame during
inference; the goal is to re-localize these objects across the video [Caelles
et al. (2017)]. Unsupervised VOS aims to discover object(s) of interest
without the initial targets [Faktor and Irani (2014), Papazoglou and Ferrari
(2013), Tokmakov et al. (2019), Jain et al. (2017), Li et al. (2018), Lu et
al. (2019)]. However, most unsupervised VOS methods use, in fact, some form of
supervised pre-training on external data.

#### Motion Segmentation.

In videos, the background is usually relatively static whereas objects in the
scene have independent motion, thus providing a strong ‘objectness’ signal.
Thus, many works approach unsupervised video object segmentation as a motion
segmentation problem. Several earlier methods address this problem by grouping
point trajectories [Brox and Malik (2010a), Sundaram et al. (2010), Ochs and
Brox (2012), Keuper et al. (2015), Keuper et al. (2020), Ochs and Brox
(2011)], motion boundaries [Papazoglou and Ferrari (2013)], voting [Faktor and
Irani (2014)] and layered models [Chang and III (2013), Jojic and Frey
(2001)]. More recently, Xie et al. (2022); Lamdouar et al. (2021) train motion
models on generated scenes with synthetic 2D objects and generalize to real
videos. CIS Yang et al. (2019) proposes an adversarial framework, where an
inpainter is tasked with predicting the optical flow of a segment based on
context, while the generator aims to create segments with zero mutual
information such that the context becomes uninformative. DyStaB Yang et al.
(2021b) extends CIS using the segmentation output of a dynamic model to
bootstrap a static one. In contrast to our method, this yields two separate
models to choose from based on the application (_i.e_ , video or static image
segmentation). Instead, AMD Liu et al. (2021) employs a single model with
separate appearance and motion ‘pathways’ and performs unsupervised test-time
adaptation for video segmentation. Finally, MG Yang et al. (2021a) abandons
the appearance pathway altogether, directly segmenting optical flow inputs
with a Slot Attention-like architecture Locatello et al. (2020).

Closer to our approach, another line of work uses various motion models to
group image regions. Early methods Jepson and Black (1993); Torr (1998)
consider mixture models of flow to account for the fact that a region may
contain multiple motion patterns. Another line of work Bideau and Learned-
Miller (2016); Bideau et al. (2018) segments object translation directions
from motion angle field obtained by correcting for estimated rotation of the
camera. Mahendran et al. (2018) employ an affine flow model, using the entropy
of flow magnitude histograms for loss to deal with noisy flow in real world.
Meunier et al. (2022) consider affine and quadratic motion models, however
their method uses flows as input which makes it suitable only for videos
during inference.

#### Unsupervised Image Segmentation.

While we use motion as a learning signal, our method yields a general-purpose
image segmentation network, separating an image into foreground and
background, without using ground truth masks for supervision. Early work in
unsupervised image segmentation makes use of hand-crafted priors, _e.g_ color
contrast Cheng et al. (2015); Wei et al. (2012), while some recent methods
also combine handcrafted heuristics to generate pseudo-masks and use them to
train using deep networks Zhang et al. (2018); Zeng et al. (2019); Nguyen et
al. (2019). Others address this problem via mutual information maximization
between different views of the input Ji et al. (2019); Ouali et al. (2020). A
recently emerging line of work Bielski and Favaro (2019); Chen et al. (2019);
Kanezaki (2018); Benny and Wolf (2020); Voynov et al. (2021); Melas-Kyriazi et
al. (2022b) explores generative models to obtain segmentation masks. Many of
them Bielski and Favaro (2019); Chen et al. (2019); Kanezaki (2018); Benny and
Wolf (2020) are based on the idea of generating foreground and background as
separate layers and combine them to obtain a real image. Others Voynov et al.
(2021); Melas-Kyriazi et al. (2022b) analyze large-scale unsupervised GANs
(_e.g_ BigGAN Brock et al. (2019)) and find implicit foreground-background
structure in them to generate a synthetic annotated training dataset.
Alternative line of work explores feature maps of self-supervised Vision
Transformers, such as DINO Caron et al. (2021). For example, STEGO Hamilton et
al. (2022) supports segmenting multiple _classes_ in an image, performing
_semantic_ segmentation, by distilling features and class centroids from DINO.
In Melas-Kyriazi et al. (2022a) and TokenCut Wang et al. (2022), authors model
image patches with an affinity graph based on DINO feature alignment and
perform further analysis on this graph to extract masks. Shin et al. (2022)
cluster features of a variety of self-supervised backbones to produce
candidate masks, using them to train a segmenter. Instead, our model is
trained on video data using optical flow as a supervisory signal. However,
since it only requires a single image as input at test time, we show that our
method is applicable to this task, providing an alternative approach to
unsupervised object segmentation.

#### Unsupervised Object Discovery.

While the above methods often aim to segment the most salient object(s) in an
image, unsupervised multi-object segmentation explores the problem of
decomposing a scene into parts, which typically include each individual
foreground object and the background. The usual approach is to learn
structured object-centric representations, _i.e_ to model the scene with
latent variables (slots) operating on a common representation Greff et al.
(2019); Locatello et al. (2020); Emami et al. (2021); Lin et al. (2020);
Crawford and Pineau (2019); Burgess et al. (2019); Engelcke et al. (2020,
2021); Jiang and Ahn (2020); Singh et al. (2022a). While these methods are
image-based, extensions to video also exist Kosiorek et al. (2018); Jiang* et
al. (2020); Crawford and Pineau (2020); Kabra et al. (2021); Zablotskaia et
al. (2020); Besbinar and Frossard (2021); Min et al. (2021); Bear et al.
(2020); Kipf et al. (2021); Singh et al. (2022b); Monnier et al. (2021). These
methods often operate in an auto-encoding fashion with inductive bias to
separate objects derived from a reconstruction bottleneck (Burgess et al.,
2019), that is often dependent on the architecture and the latent variable
model. We similarly impose a reconstruction bottleneck on the flow but use a
simple model grounded in projective geometry, with a known closed-form
solution. It is also important to note that unsupervised _multi_ -object
segmentation appears to be significantly more challenging, with current
methods exploiting the simplicity of synthetic scenes Johnson et al. (2017);
Girdhar and Ramanan (2020), while struggling on more realistic data Karazija
et al. (2021). Recently, Bao et al. (2022) explore an extension of Slot
Attention Locatello et al. (2020), guided by an external supervised motion
segmentation algorithm, to real-world data. However, due to the difficulty of
the problem, they operate in a constrained domain (autonomous driving) and
consider only a limited number of object categories. We instead focus on wide
variety of categories and settings encountered in common video segmentation
datasets and consider both motion and appearance jointly.

##  3 Method

Figure 1: Model Diagram. We train a segmentation network to partition an image
into $K$ components without manual annotations. Our model is trained using
individual frames from video as input and pre-computed optical flow as
supervision. The predicted segments are used to approximate the input flow
with piecewise quadratic flow models and the training loss is formulated as
the error between the reconstructed and the input flow. Appearance features
from the backbone are used to merge the predicted $K$ segments into foreground
and background components. Motion information is not required at test time and
inference can be performed on still images. Optical flow is colorized for
visualization only.

In this paper, we present a method that uses motion anticipation to discover
and segment objects in images without the need for human annotations (overview
in Fig. 1). We use optical flow from video sequences as supervision for this
problem. However, rather than predicting the flow directly, we task a general
image segmentation network to predict image regions where motion may be
explained by a simple coherent model. Such regions should align with optical
flow patterns produced by objects that _could_ move (but do not have to).

###  3.1 Segmentation by Motion Anticipation

Let $I\in\mathbb{R}^{3\times H\times W}=(\mathbb{R}^{3})^{\Omega}$ be an RGB
image defined on a lattice $\Omega=\\{1,\dots,H\\}\times\\{1,\dots,W\\}.$
Assume that the image is a frame in a video sequence and let
$F\in(\mathbb{R}^{2})^{\Omega}$ be the corresponding optical flow (extracted
from the video by means of an off-the-shelf optical flow network, such as RAFT
Teed and Deng (2020)). The goal is to decompose the image into $K$ components
(or regions), which is a classic segmentation problem. Hence, we learn a
segmentation network $\Phi(I)\in([0,1]^{K})^{\Omega}$ that, given the image
$I$ as input, assigns each pixel $u$ to one of $K$ components in a soft
manner, with probabilities:

$P(m_{u}=k\mid
I,\Phi)=[(\Phi(I))_{k}]_{u},~{}~{}~{}u\in\Omega,\;k\in\\{1,...,K\\}.$ (1)

$m_{u}=k$ in Eq. 1 denotes the predicted mask corresponding to component $k$
indexed by $u$. In particular, we seek to separate the foreground and
background, for which one may choose $K=2$, although as we show later (Section
3.2), this need not be the case.

More specifically, we train $\Phi$ to partition pixels according to the
Gestalt principle of common fate Spelke (1990); Wagemans et al. (2012). This
is done by associating each region $k\in\\{1,...,K\\}$ to a model $\theta_{k}$
of the optical flow observed within it. That is, the optical flow
corresponding to an input frame can be approximated by piece-wise parametric
models, representing the motion or _flow pattern_ , of each component
independently. According to the common fate principle, pixels within the same
region are expected to exhibit _coherent_ motion.

A variety of motion models exist for describing the 2D flow of an object
(Mahendran et al. (2018); Meunier et al. (2022)). These are generally of the
form $F_{u}\approx Au+b$, where parameters $A,b$ can be recovered by solving a
system of linear equations. One common choice is an affine model (where
$u=[x,y]$ are pixel coordinates), which is sufficient if objects are smaller
and further away from camera. The affine model, however, struggles if the
depth of an object varies significantly resulting in more complex flow
patterns. To factor out unknown depth information, each object can be modeled
as a plane with a quadratic 8-parameter model (Adiv, 1985). Here, we allow for
more complex geometry than planes, by using a simplified 12-parameter
quadratic model $\theta_{k}=(A_{k},b_{k})$ with $A_{k}\in\mathbb{R}^{2\times
5}$ and $b_{k}\in\mathbb{R}^{2}$ per region $k$. In this case,
$u=[x,~{}x^{2},~{}y,~{}y^{2},~{}xy]\in\mathbb{R}^{5}$ includes quadratic and
mixed terms of the pixel coordinates to model quadratic dependencies. The
12-parameter model also allows treating each flow direction independently. We
assume that the model predicts the flow up to isotropic i.i.d. Gaussian noise,
which results in a simple $L^{2}$ fitting loss:

$-\log p(F_{u}\mid\theta_{k})\propto\|F_{u}-A_{k}u-b_{k}\|^{2}.$ (2)

Summing over all pixels, learning minimizes the energy function:

$\mathcal{L}(F\mid\theta,I,\Phi)\propto\sum_{u\in\Omega}\sum_{k}\|F_{u}-A_{k}u-b_{k}\|^{2}\cdot
p(m_{u}=k\mid I,\Phi).$ (3)

In the expression above, we _do not_ know the flow parameters $\theta_{k}$ as
the network only predicts the regions’ extent. Instead, we _min-out_ the
parameters $\theta_{k}$ in the loss itself and compute

$\mathcal{L}(F\mid
I,\Phi)=\min_{\theta_{k\in\\{1,...,K\\}}}\mathcal{L}(F\mid\theta,I,\Phi).$ (4)

The energy in Eq. 2 is quadratic in $\theta_{k}$, resulting in a weighted
least squares problem that can be efficiently solved in closed form (see
supplementary material).

Our model is learned from a large collection $\mathcal{T}$ of video frame-
optical flow pairs $(I,F)$, minimizing the empirical risk:

$\Phi^{*}=\operatornamewithlimits{argmin}_{\Phi}\frac{1}{|\mathcal{T}|}\sum_{(I,F)\in\mathcal{T}}\mathcal{L}(F\mid
I,\Phi)$ (5)

###  3.2 Over-segmentation

While the 12-parameter model is more powerful than an affine one, it is still
not sufficient to model arbitrary flow patterns. In complex scenes that
contain foreground and background clutter, we often observe motion parallax
effects. Additionally, non-rigid objects and self-occlusions can result in
complex flow patterns within the object that are not captured accurately by
the quadratic model.

To account for such complexity, we propose to _over-segment_ the input image
into $K>2$ regions. Over-segmentation enables the model to use additional
regions to explain several moving objects and to approximate varyingly moving
parts of a single non-rigid object as well as motion parallax. To achieve a
binary segmentation output, one needs a criterion to merge a number of
predicted regions down to foreground and background.

We devise a criterion based on _appearance_ cues to avoid the ambiguity
associated with merging regions based on motion. To this end, we use a pre-
trained self-supervised image encoder, such as DINO-ViT Caron et al. (2021),
to obtain dense features for the input image and merge the segments predicted
by $\Phi$ based on feature similarity. Formally, let $V_{u}$ denote the
feature vector of pixel $u$ obtained by the self-supervised encoder. Then,
$\bar{V}_{k}=\sum_{u}V_{u}p(m_{u}=k\mid I,\Phi)/\sum_{v}p(m_{v}=k\mid I,\Phi)$
is the average feature vector for segment $k$, where pixels are weighed by
their probability with which they belong to the segment. We compute the
pairwise similarities of different regions via an affinity matrix
$\Pi\in\mathbb{R}^{K\times K}$, where entries corresponding to segments $i$
and $j$ are set as

$(\Pi)_{ij}=\max\left(\epsilon,\left\langle\frac{\bar{V}_{i}}{||\bar{V}_{i}||_{2}},\frac{\bar{V}_{j}}{||\bar{V}_{j}||_{2}}\right\rangle\right),$
(6)

where only feature vectors pointing in the same direction are considered and
$\epsilon=10^{-12}$ is a small constant that keeps the graph connected. We
then perform spectral clustering Cheeger (1969); Shi and Malik (2000); Melas-
Kyriazi et al. (2022a) into two components using the affinity $\Pi$.

###  3.3 Two Scenarios: Motion _vs_ Image Segmentation

We experiment with two modes of application of our model. The first scenario
is _internal learning for unsupervised video segmentation_ , where the network
is evaluated on the same video sequences that have been used for optimization.
This is effectively an unsupervised motion segmentation algorithm because the
network not only receives as input appearance information, but incorporates
motion information via backpropagation, observing indirectly optical flow too.
While not explicitly stated in the respective papers, prior motion
segmentation works such as Yang et al. (2021a); Meunier et al. (2022) also
operate in this mode, while directly observing moving objects, often using
optical flow as input.

The second scenario is _transductive learning for image segmentation_. In this
case, the network is first trained using a number of unlabelled videos, and
then used for single-image foreground object segmentation on an _independent_
validation/test set of still images. In this scenario, motion is only used as
a supervisory signal: when the network is applied at test time, motion is not
considered anymore and the network operates purely as an image-based
segmenter. As for any transductive learning setting, the goal is to assess the
generalization performance of the network on new images.

##  4 Experiments

|  | Inf. Input | Input | Flow | Runtime | DAVIS | STv2 | FBMS
---|---|---|---|---|---|---|---|---
|  | RGB | Flow | Resolution | Method |  sec $\downarrow$ | $\mathcal{J}\uparrow$ | $\mathcal{J}\uparrow$ | $\mathcal{J}\uparrow$
Wang et al. (2018) | SAGE | ✓ | ✓ | – |  LDOF Brox and Malik (2010b) | $0.9$ | 42.6 | 57.6 | 61.2
Faktor and Irani (2014) | NLC | ✓ | ✓ | – |  SIFTFlow Liu et al. (2010) | $11$ | 55.1 | 67.2 | 51.5
Keuper et al. (2015) | CUT | ✓ | ✓ | – |  LDOF Brox and Malik (2010b) | $103$ | 55.2 | 54.3 | 57.2
Papazoglou and Ferrari (2013) | FTS | ✓ | ✓ | – |  LDOF Brox and Malik (2010b) | $0.5$ | 55.8 | 47.8 | 47.7
Yang et al. (2019) | CIS | ✓ | ✓ | $192\times 384$ |  PWCNet Sun et al. (2018) | $0.1$ | 59.2 | 45.6 | 36.8
Liu et al. (2021) | AMD | ✓ | ✗ | $128\times 224$ | ✗ | – | 57.8 | 57.0 | 47.5
Yang et al. (2021a) | MG | ✗ | ✓ | $128\times 224$ |  RAFT Teed and Deng (2020) | $0.012$ | 68.3 | 58.6 | 53.1
Meunier et al. (2022) | EM | ✗ | ✓ | $128\times 224$ |  RAFT Teed and Deng (2020) | – | 69.3 | 55.5 | 57.8
Xie et al. (2022) | OCLR | ✓ | ✓ | $480\times 832$ |  RAFT Teed and Deng (2020) | – | 78.9 | 71.6 | 68.7
Ye et al. (2022) |  DS‡ | ✓ | ✓ | $240\times 426$ |  RAFT Teed and Deng (2020) | $1800~{}(22.5)^{\ddagger}$ | 79.1 | 72.1 | 71.8
Ours (UNet) | ✓ | ✗ | $128\times 224$ |  RAFT Teed and Deng (2020) | $0.027$ | 78.3 | 76.8 | 72.0
Ours (MaskFormer) | ✓ | ✗ | $128\times 224$ |  RAFT Teed and Deng (2020) | $0.059$ | 79.5 | 78.3 | 77.4
Yang et al. (2019) |  CIS† | ✓ | ✓ | $192\times 384$ |  PWCNet Sun et al. (2018) | $11$ | 71.5 | 62.0 | 63.5
Yang et al. (2021b) |  DyStaB†* | ✓ | ✓ | $192\times 384$ |  RAFT Teed and Deng (2020) | – | 80.0 | 74.2 | 73.2
Ours† (w/ CRF) | ✓ | ✗ | $128\times 224$ |  RAFT Teed and Deng (2020) | $3.73$ | 80.7 | 78.9 | 78.4
Table 1:  Unsupervised video segmentation on DAVIS2016, SegTrack-v2 (STv2),
and FBMS59.
$\dagger$ denotes the usage of CRFs and other extra significant post-
processing (_e.g_ , multi-step flow, multi-crop, temporal smoothing for CIS
Yang et al. (2019)). $\ddagger$ DS is optimized per sequence; authors report
30 min training time for 80-frame video. * DyStaB utilises supervised pre-
training.

As discussed above, our formulation allows us to evaluate our method in two
settings: video object segmentation and general image/object segmentation. We
show that learning a network that guesses what moves not only results in
state-of-the-art performance in video segmentation, but also generalizes to
image segmentation without further training.

###  4.1 Experimental Setup

#### Architecture.

Our formulation enables us to use any standard image segmentation architecture
for the model $\Phi$. This has two main benefits: while training the model
needs optical flow (and thus video data), inference can be performed on single
images alone just like any image segmentation method. Second, using a standard
architecture allows us to benefit from (self-)supervised pretraining, ensuring
better convergence and broader generalization. We experiment with both
convolutional and transformer-based architectures.

#### Datasets.

For the _video segmentation_ task, we use three popular datasets: DAVIS2016
(DAVIS) Perazzi et al. (2016), SegTrackV2 (STv2) Li et al. (2013), as well as
FBMS Ochs et al. (2014). For the _image segmentation_ task, we consider the
Caltech-UCSD Birds-200 (CUB) dataset Welinder et al. (2010) and three saliency
detection benchmarks: DUTS Wang et al. (2017), ECSSD Shi et al. (2016), and
DUT-OMRON Yang et al. (2013).

Figure 2: Qualitative Comparison on DAVIS, STv2, and FBMS. †– indicates use of
CRF. Our method correctly segments objects in challenging conditions including
strong parallax (2nd, 3rdseq.), small objects (4th), background motion (5th),
camouflaged appearance (6th), non-rigid motion (7th) or no motion at all (8th
seq.). In the failure cases, our method is confused by ripples and reflection
in the water, the front wheel rotating in a different direction and multiple
disconnected objects.

#### Optical Flow.

Our method derives its learning signal from optical flow. We estimate optical
flow for all frames on DAVIS, STv2, and FBMS following the practice of
MotionGrouping Yang et al. (2021a). We employ RAFT Teed and Deng (2020)
(supervised) using the original resolution for our main experiments. Please
see the supplement for experiments with other flow methods.

#### Training Details.

We use MaskFormer Cheng et al. (2021) as our segmentation network, and use
only the segmentation head. For the backbone and appearance features $V$, we
leverage a ViT-B transformer, pre-trained on ImageNet Russakovsky et al.
(2015) in a self-supervised manner using DINO Caron et al. (2021) to avoid any
external sources of supervision. We set the number of components to $K=4$
unless otherwise noted. Please see the supplement for all details and hyper-
parameter settings.

###  4.2 Unsupervised Video Segmentation

In Table 1 we report our performance on the DAVIS, STv2, and FBMS datasets and
compare to other unsupervised video segmentation approaches. Our method
achieves state-of-the-art performance, even without CRF post-processing. Fig.
2 provides a qualitative comparison of the results. Our model provides better
segmentation with sharper boundaries despite complex non-rigid motion,
parallax effects or lack-of-motion. However, on challenging scenarios our
method still struggles to segment small details or non-connected instances.

Our method is not restricted to a specific segmentation architecture. To
investigate, MaskFormer is replaced with a simple convolutional U-Net
architecture Ronneberger et al. (2015), as in EM Meunier et al. (2022), and
trained from scratch for a fair comparison. The U-Net based model achieves
comparable results on DAVIS and FBMS and 76.8 on STv2 (Table 1), outperforming
earlier methods even without transformers.

###  4.3 Flow Model and Number of Components

Flow Model | $K$ |  DAVIS ($\mathcal{J}$$\uparrow$) |  DAVIS ($\mathcal{J}_{\text{oracle}}$$\uparrow$)
---|---|---|---
Constant ($A=0$) | 4 | 76.8 | 77.7
Affine ($u=[x,~{}y]$) | 4 | 77.1 | 78.8
Quadratic (Eq. 2) | 4 | 79.5 | 81.5
Quadratic (Eq. 2) | 2 | 74.5 | 74.5
Quadratic (Eq. 2) | 3 | 77.8 | 79.5
Quadratic (Eq. 2) | 4 | 79.5 | 81.5
Quadratic (Eq. 2) | 5 | 76.0 | 79.9
Table 2: Flow Model and Number of Components. We ablate the choice of flow
model and the number of components $K$. More complex flow models improve
performance, and over-segmentation helps until the assignment problem between
components and the final binary segmentation becomes too difficult at $K=5$.
To evaluate the quality of the clustering of components we also report the
oracle clustering performance as an upper bound.

Using DAVIS, we now study the effectiveness of the individual components of
the method. In Table 2 we evaluate the performance of the model under
different flow models: constant, affine, and quadratic. We find that more
complex models lead to improved performance, likely due to the fact that manny
scenes in the DAVIS benchmark are highly dynamic with complex objects and
backgrounds. Additionally, in the same table we evaluate how the number of
components, $K$, influences the final performance after clustering. With $K=2$
the model directly performs foreground-background separation but needs to
model each with a single component which is often difficult, _e.g_ due to
complex motions of deformable objects and/or parallax effects. Increasing the
number of components is beneficial up to $K=4$, after which the assignment
problem from over-segmentation to foreground and background becomes too
difficult for simple spectral clustering. This can be seen by evaluating the
segmentation performance under an optimal oracle assignment of the components
to foreground and background (oracle column in Table 2). In all cases $K<=4$,
spectral clustering nearly reaches oracle performance.

| CUB
---|---
Ours GT Image |

| DUT-OMRON
---|---
Ours GT Image |

| DUTS
---|---
Ours GT Image |

| ECSSD
---|---
Ours GT Image |

Figure 3: Qualitative Comparison. Our method can extract salient object in various environments and works even for novel object that were not included in the training data.  |  | CUB | DUTS | ECSSD | OMRON
---|---|---|---|---|---
|  | Acc | $\mathcal{J}\uparrow$ | $maxF_{\beta}\uparrow$ | Acc | $\mathcal{J}\uparrow$ | $F_{\beta}\uparrow$ | Acc | $\mathcal{J}\uparrow$ | $F_{\beta}\uparrow$ | Acc | $\mathcal{J}\uparrow$ | $F_{\beta}\uparrow$
Voynov et al. (2021) |  Voynov _et al._ | 94.0 | 71.0 | 80.7 | 88.1 | 51.1 | 60.0 | 90.6 | 68.4 | 79.0 | 86.0 | 46.4 | 53.3
Liu et al. (2021) | AMD | – | – | – | – | – | 60.2 | – | – | – | – | – | –
Melas-Kyriazi et al. (2022b) |  Kyriazi _et al._ | 92.1 | 66.4 | 78.3 | 89.3 | 52.8 | 61.4 | 91.5 | 71.3 | 80.6 | 88.3 | 50.9 | 58.3
Melas-Kyriazi et al. (2022a) |  Kyriazi _et al._ | – | 76.9 | – | – | 51.4 | – | – | 73.3 | – | – | 56.7 | –
Yang et al. (2021b) |  DyStaB† | – | – | – | – | – | – | – | – | 88.1 | – | – | 73.9
Wang et al. (2022) | TokenCut | – | – | – | 90.3 | 57.6 | – | 91.8 | 71.2 | – | 88.0 | 53.3 | –
Shin et al. (2022) | SelfMask | – | – | – | 92.3 | 62.6 | – | 94.4 | 78.1 | – | 90.1 | 58.2 | –
Ours | 93.5 | 64.6 | 80.9 | 91.5 | 49.2 | 65.6 | 88.5 | 56.1 | 74.3 | 89.3 | 41.31 | 56.3
Table 3: Unsupervised object segmentation benchmark CUB and three saliency
detection benchmarks: DUTS, ECSSD, and DUT-OMRON (OMRON). $\dagger$ DyStaB
uses CRF post-processing, supervised pre-training, and self-training on each
dataset. (SoTA only table - please see the supplement for a complete version
of this table including many older methods.)

###  4.4 Unsupervised Image Segmentation

While the main aim of our work is object segmentation in videos, we also
assess the image segmentation performance on common image segmentation and
saliency benchmarks: CUB, DUTS, DUT-OMRON, and ECSSD. For this experiment, we
train our model on all three motion segmentation datasets (DAVIS, FBMS and
STv2) jointly and apply the resulting network to the image segmentation
benchmarks without any further fine-tuning. In Table 3, we report the
performance of our method and compare to the current state of the art. It is
worth noting that most prior work (except Melas-Kyriazi et al. (2022b, a);
Wang et al. (2022)) relies on dataset-specific training, self-training, post-
processing or supervised pre-training to achieve image segmentation.

Finally, we evaluate the model qualitatively in Fig. 3 on all four benchmarks.
We observe our model works well on a diverse set of classes, such as
buildings, certain animals and plants, even though they were not part of the
foreground (moving) objects in the training data.

##  5 Conclusions

We have proposed a simple approach to exploit the synergies between motion in
videos and objectness for segmenting visual objects without supervision. The
key idea is using motion anticipation as a learning signal: we train an image
segmentation network to predict regions that likely contain simple optical
flow patterns, as these have a high chance to correspond to objects. We find
that the complexity of the motion model is important to model complicated flow
patters that can arise even for rigid objects. Our results show that this
approach achieves state-of-the-art performance in video segmentation
benchmarks. Future work could thus consider extensions to more sophisticated
motion models, accounting for the 3D shape of objects, and to separate
multiple objects.

#### Acknowledgments

S. C. is supported by a scholarship sponsored by Facebook. L. K. is funded by
EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and
Systems EP/S024050/1. I. L. and A. V. are supported by the European Research
Council (ERC) grant IDIU-638009. I. L. is also supported by EPSRC grant
VisualAI EP/T028572/1. C. R. is supported by Innovate UK (project 71653) on
behalf of UK Research and Innovation (UKRI).

## References

  * Adiv (1985) Gilad Adiv.  Determining three-dimensional motion and structure from optical flow generated by several moving objects.  _IEEE transactions on pattern analysis and machine intelligence_ , (4):384–401, 1985.
  * Bao et al. (2022) Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert.  Discovering objects that can move.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11789–11798, June 2022.
  * Bear et al. (2020) Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li Fei-Fei, Jiajun Wu, Josh Tenenbaum, and Daniel L. Yamins.  Learning physical graph representations from visual scenes.  In _Advances in Neural Information Processing Systems_ , volume 33, 2020.
  * Benny and Wolf (2020) Yaniv Benny and Lior Wolf.  Onegan: Simultaneous unsupervised learning of conditional image generation, foreground segmentation, and fine-grained clustering.  In _European Conference on Computer Vision_ , pages 514–530. Springer, 2020.
  * Besbinar and Frossard (2021) Beril Besbinar and Pascal Frossard.  Self-supervision by prediction for object discovery in videos.  In _2021 IEEE International Conference on Image Processing (ICIP)_ , pages 1509–1513. IEEE, 2021.
  * Bideau and Learned-Miller (2016) Pia Bideau and Erik Learned-Miller.  It’s moving! a probabilistic model for causal motion segmentation in moving camera videos.  In _European Conference on Computer Vision_ , pages 433–449. Springer, 2016.
  * Bideau et al. (2018) Pia Bideau, Aruni RoyChowdhury, Rakesh R Menon, and Erik Learned-Miller.  The best of both worlds: Combining cnns and geometric constraints for hierarchical motion segmentation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 508–517, 2018.
  * Bielski and Favaro (2019) Adam Bielski and Paolo Favaro.  Emergence of object segmentation in perturbed generative models.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Brock et al. (2019) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _ArXiv_ , abs/1809.11096, 2019.
  * Brox and Malik (2010a) Thomas Brox and Jitendra Malik.  Object segmentation by long term analysis of point trajectories.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2010a.
  * Brox and Malik (2010b) Thomas Brox and Jitendra Malik.  Large displacement optical flow: descriptor matching in variational motion estimation.  _IEEE transactions on pattern analysis and machine intelligence_ , 33(3):500–513, 2010b.
  * Burgess et al. (2019) Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner.  Monet: Unsupervised scene decomposition and representation.  _arXiv preprint arXiv:1901.11390_ , 2019.
  * Caelles et al. (2017) Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool.  One-shot video object segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 221–230, 2017.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9650–9660, October 2021.
  * Chang and III (2013) Jason Chang and John W. Fisher III.  Topology-constrained layered tracking with latent flow.  _2013 IEEE International Conference on Computer Vision_ , pages 161–168, 2013.
  * Cheeger (1969) Jeff Cheeger.  A lower bound for the smallest eigenvalue of the laplacian.  In _Proceedings of the Princeton conference in honor of Professor S. Bochner_ , pages 195–199, 1969.
  * Chen et al. (2019) Mickaël Chen, Thierry Artières, and Ludovic Denoyer.  Unsupervised object segmentation by redrawing.  _Advances in neural information processing systems_ , 32, 2019.
  * Cheng et al. (2021) Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Cheng et al. (2015) Ming-Ming Cheng, Niloy J. Mitra, Xiaolei Huang, Philip H. S. Torr, and Shi-Min Hu.  Global contrast based salient region detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 37(3):569–582, 2015.
  * Crawford and Pineau (2019) Eric Crawford and Joelle Pineau.  Spatially invariant unsupervised object detection with convolutional neural networks.  In _The Thirty-Third AAAI Conference on Artificial Intelligence_ , pages 3412–3420. AAAI Press, 2019.
  * Crawford and Pineau (2020) Eric Crawford and Joelle Pineau.  Exploiting spatial invariance for scalable unsupervised object tracking.  In _Thirty-Fourth AAAI Conference on Artificial Intelligence_ , 2020\.
  * Emami et al. (2021) Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan.  Efficient iterative amortized inference for learning symmetric and disentangled multi-object representations.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 2970–2981. PMLR, 18–24 Jul 2021.
  * Engelcke et al. (2020) Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner.  GENESIS: generative scene inference and sampling with object-centric latent representations.  In _International Conference on Learning Representations_. OpenReview.net, 2020.
  * Engelcke et al. (2021) Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner.  Genesis-v2: Inferring unordered object representations without iterative refinement.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Faktor and Irani (2014) Alon Faktor and Michal Irani.  Video segmentation by non-local consensus voting.  In _Proceedings of the British Machine Vision Conference_. BMVA Press, 2014.
  * Girdhar and Ramanan (2020) Rohit Girdhar and Deva Ramanan.  Cater: A diagnostic dataset for compositional actions & temporal reasoning.  In _International Conference on Learning Representations_ , 2020.
  * Greff et al. (2019) Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner.  Multi-object representation learning with iterative variational inference.  In _Proceedings of the International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2424–2433. PMLR, 2019.
  * Hamilton et al. (2022) Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman.  Unsupervised semantic segmentation by distilling feature correspondences.  In _International Conference on Learning Representations_ , 2022.
  * He et al. (2021) Xingzhe He, Bastian Wandt, and Helge Rhodin.  Ganseg: Learning to segment by unsupervised hierarchical image generation.  _arXiv preprint arXiv:2112.01036_ , 2021.
  * Jain et al. (2017) Suyog Jain, Bo Xiong, and Kristen Grauman.  Fusionseg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos.  _arXiv preprint arXiv:1701.05384_ , 2017.
  * Jepson and Black (1993) Allan D. Jepson and Michael J. Black.  Mixture models for optical flow computation.  _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition_ , pages 760–761, 1993.
  * Ji et al. (2019) Xu Ji, Joao F Henriques, and Andrea Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9865–9874, 2019.
  * Jiang and Ahn (2020) Jindong Jiang and Sungjin Ahn.  Generative neurosymbolic machines.  _Advances in Neural Information Processing Systems_ , 33:12572–12582, 2020.
  * Jiang* et al. (2020) Jindong Jiang*, Sepehr Janghorbani*, Gerard De Melo, and Sungjin Ahn.  Scalor: Generative world models with scalable object representations.  In _International Conference on Learning Representations_ , 2020.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2901–2910, 2017.
  * Jojic and Frey (2001) N. Jojic and B.J. Frey.  Learning flexible sprites in video layers.  In _Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001_ , volume 1, pages I–I, 2001\.
  * Kabra et al. (2021) Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P Burgess.  SIMONe: View-invariant, temporally-abstracted object representations via unsupervised video decomposition.  _arXiv preprint arXiv:2106.03849_ , 2021.
  * Kanezaki (2018) Asako Kanezaki.  Unsupervised image segmentation by backpropagation.  In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 1543–1547. IEEE, 2018.
  * Karazija et al. (2021) Laurynas Karazija, Iro Laina, and Christian Rupprecht.  Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * Keuper et al. (2015) Margret Keuper, Bjoern Andres, and Thomas Brox.  Motion trajectory segmentation via minimum cost multicuts.  In _2015 IEEE International Conference on Computer Vision (ICCV)_ , pages 3271–3279, 2015.
  * Keuper et al. (2020) Margret Keuper, Siyu Tang, Bjoern Andres, Thomas Brox, and Bernt Schiele.  Motion segmentation amp; multiple object tracking by correlation co-clustering.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 42(1):140–153, 2020.
  * Kipf et al. (2021) Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff.  Conditional object-centric learning from video.  _arXiv preprint arXiv:2111.12594_ , 2021.
  * Kosiorek et al. (2018) Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner.  Sequential attend, infer, repeat: Generative modelling of moving objects.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * Lamdouar et al. (2021) Hala Lamdouar, Weidi Xie, and Andrew Zisserman.  Segmenting invisible moving objects.  In _BMVC_ , 2021.
  * Li et al. (2013) Fuxin Li, Taeyoung Kim, Ahmad Humayun, David Tsai, and James M. Rehg.  Video segmentation by tracking many figure-ground segments.  _2013 IEEE International Conference on Computer Vision_ , pages 2192–2199, 2013.
  * Li et al. (2018) Siyang Li, Bryan Seybold, Alexey Vorobyov, Alireza Fathi, Qin Huang, and C-C Jay Kuo.  Instance embedding transfer to unsupervised video object segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6526–6535, 2018.
  * Lin et al. (2020) Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn.  SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition.  In _International Conference on Learning Representations_. OpenReview.net, 2020.
  * Liu et al. (2010) Ce Liu, Jenny Yuen, and Antonio Torralba.  Sift flow: Dense correspondence across scenes and its applications.  _IEEE transactions on pattern analysis and machine intelligence_ , 33(5):978–994, 2010.
  * Liu et al. (2020) Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang.  Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Liu et al. (2021) Runtao Liu, Zhirong Wu, Stella Yu, and Stephen Lin.  The emergence of objectness: Learning zero-shot segmentation from videos.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Locatello et al. (2020) Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric learning with slot attention.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 11525–11538, 2020.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2018.
  * Lu et al. (2019) Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli.  See more, know more: Unsupervised video object segmentation with co-attention siamese networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3623–3632, 2019.
  * Mahendran et al. (2018) Aravindh Mahendran, James Thewlis, and Andrea Vedaldi.  Self-supervised segmentation by grouping optical-flow.  In _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , September 2018.
  * Melas-Kyriazi et al. (2022a) Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8364–8375, June 2022a.
  * Melas-Kyriazi et al. (2022b) Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Finding an unsupervised image segmenter in each of your deep generative models.  In _International Conference on Learning Representations_ , 2022b.
  * Meunier et al. (2022) Etienne Meunier, Anaïs Badoual, and Patrick Bouthemy.  Em-driven unsupervised learning for efficient motion segmentation.  _CoRR_ , abs/2201.02074, 2022.
  * Min et al. (2021) Cheol-Hui Min, Jinseok Bae, Junho Lee, and Young Min Kim.  Gatsbi: Generative agent-centric spatio-temporal object interaction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3074–3083, 2021.
  * Monnier et al. (2021) Tom Monnier, Elliot Vincent, Jean Ponce, and Mathieu Aubry.  Unsupervised layered image decomposition into object prototypes.  In _ICCV_ , 2021.
  * Nguyen et al. (2019) Tam Nguyen, Maximilian Dax, Chaithanya Kumar Mummadi, Nhung Ngo, Thi Hoai Phuong Nguyen, Zhongyu Lou, and Thomas Brox.  Deepusps: Deep robust unsupervised saliency prediction via self-supervision.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Ochs and Brox (2011) Peter Ochs and Thomas Brox.  Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions.  In _2011 international conference on computer vision_ , pages 1583–1590. IEEE, 2011.
  * Ochs and Brox (2012) Peter Ochs and Thomas Brox.  Higher order motion models and spectral clustering.  In _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 614–621. IEEE, 2012.
  * Ochs et al. (2014) Peter Ochs, Jitendra Malik, and Thomas Brox.  Segmentation of moving objects by long term video analysis.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 36:1187–1200, 2014.
  * Ouali et al. (2020) Yassine Ouali, Celine Hudelot, and Myriam Tami.  Autoregressive unsupervised image segmentation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , August 2020.
  * Papazoglou and Ferrari (2013) Anestis Papazoglou and Vittorio Ferrari.  Fast object segmentation in unconstrained video.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , December 2013.
  * Perazzi et al. (2016) F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In _Computer Vision and Pattern Recognition_ , 2016.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pages 234–241. Springer, 2015.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Shi and Malik (2000) Jianbo Shi and Jitendra Malik.  Normalized cuts and image segmentation.  _IEEE Transactions on pattern analysis and machine intelligence_ , 22(8):888–905, 2000.
  * Shi et al. (2016) Jianping Shi, Qiong Yan, Li Xu, and Jiaya Jia.  Hierarchical image saliency detection on extended CSSD.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 38(4):717–729, 2016.
  * Shin et al. (2022) Gyungin Shin, Samuel Albanie, and Weidi Xie.  Unsupervised salient object detection with spectral cluster voting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 3971–3980, June 2022.
  * Singh et al. (2022a) Gautam Singh, Fei Deng, and Sungjin Ahn.  Illiterate DALL-E learns to compose.  In _International Conference on Learning Representations_ , 2022a.
  * Singh et al. (2022b) Gautam Singh, Yi-Fu Wu, and Sungjin Ahn.  Simple unsupervised object-centric learning for complex and naturalistic videos.  _arXiv preprint arXiv:2205.14065_ , 2022b.
  * Spelke (1990) Elizabeth S Spelke.  Principles of object perception.  _Cognitive science_ , 14(1):29–56, 1990.
  * Sun et al. (2018) Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * Sundaram et al. (2010) Narayanan Sundaram, Thomas Brox, and Kurt Keutzer.  Dense point trajectories by GPU-Accelerated large displacement optical flow.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2010.
  * Teed and Deng (2020) Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Tokmakov et al. (2019) Pavel Tokmakov, Cordelia Schmid, and Karteek Alahari.  Learning to segment moving objects.  _Int. J. Comput. Vision_ , 127(3):282–301, mar 2019.  ISSN 0920-5691.
  * Torr (1998) Philip H. S. Torr.  Geometric motion segmentation and model selection.  _Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences_ , 356:1321 – 1340, 1998.
  * Ulyanov et al. (2020) Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky.  Deep image prior.  _IJCV_ , 128(7):1867–1888, 2020.
  * Voynov et al. (2021) Andrey Voynov, Stanislav Morozov, and Artem Babenko.  Object segmentation without labels with large-scale generative models.  In _International Conference on Machine Learning_ , pages 10596–10606. PMLR, 2021.
  * Wagemans et al. (2012) Johan Wagemans, Jacob Feldman, Sergei Gepshtein, Ruth Kimchi, James R Pomerantz, Peter A Van der Helm, and Cees Van Leeuwen.  A century of gestalt psychology in visual perception: Ii. conceptual and theoretical foundations.  _Psychological bulletin_ , 138(6):1218, 2012.
  * Wang et al. (2017) Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan.  Learning to detect salient objects with image-level supervision.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Wang et al. (2018) Wenguan Wang, Jianbing Shen, Ruigang Yang, and Fatih Porikli.  Saliency-aware video object segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(1):20–33, 2018.
  * Wang et al. (2022) Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz.  Self-supervised transformers for unsupervised object discovery using normalized cut.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 14543–14553, June 2022.
  * Wei et al. (2012) Yichen Wei, Fang Wen, Wangjiang Zhu, and Jian Sun.  Geodesic saliency using background priors.  In _ECCV_ , 2012.
  * Welinder et al. (2010) P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010\.
  * Xia and Kulis (2017) Xide Xia and Brian Kulis.  W-net: A deep model for fully unsupervised image segmentation.  _arXiv preprint arXiv:1711.08506_ , 2017.
  * Xie et al. (2022) Jun Xie, Weidi Xie, and Andrew Zisserman.  Segmenting moving objects via an object-centric layered representation.  _ArXiv_ , abs/2207.02206, 2022.
  * Yang et al. (2021a) Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie.  Self-supervised video object segmentation by motion grouping.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7177–7188, 2021a.
  * Yang et al. (2013) Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang.  Saliency detection via graph-based manifold ranking.  In _Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on_ , pages 3166–3173. IEEE, 2013.
  * Yang et al. (2019) Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto.  Unsupervised moving object detection via contextual information separation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Yang et al. (2021b) Yanchao Yang, Brian Lai, and Stefano Soatto.  Dystab: Unsupervised object segmentation via dynamic-static bootstrapping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2826–2836, 2021b.
  * Ye et al. (2022) Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely.  Deformable sprites for unsupervised video decomposition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2657–2666, June 2022.
  * Yu et al. (2021) Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu.  Unsupervised foreground extraction via deep region competition.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Zablotskaia et al. (2020) Polina Zablotskaia, Edoardo A Dominici, Leonid Sigal, and Andreas M Lehrmann.  Unsupervised video decomposition using spatio-temporal iterative inference.  _arXiv preprint arXiv:2006.14727_ , 2020.
  * Zeng et al. (2019) Yu Zeng, Yunzhi Zhuge, Huchuan Lu, Lihe Zhang, Mingyang Qian, and Yizhou Yu.  Multi-source weak supervision for saliency detection.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019\.
  * Zhang et al. (2018) Jing Zhang, T. Zhang, Yuchao Dai, Mehrtash Harandi, and Richard I. Hartley.  Deep unsupervised saliency detection: A multiple noisy labeling perspective.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9029–9038, 2018.

## Supplementary Material

In this supplementary material, we provide further details on our training
parameters in Appendix A. Appendix B contains the closed form solution of the
fitting of the flow model $\theta$. Expanded experiments and ablations are
found in Appendix C. Finally, more qualitative results are presented in
Appendix D. See the project page,
https://www.robots.ox.ac.uk/~vgg/research/gwm, for additional visualizations,
code and models.

##  Appendix A Experimental Setup

#### Network.

We use MaskFormer Cheng et al. (2021) as our segmentation
network111Implementation from https://github.com/facebookresearch/MaskFormer.,
and use only the segmentation head. As MaskFormer predicts masks at 4 times
lower resolution than input, we modify the PixelDecoder by appending [Conv(3),
UpsampleNN(2), Conv(1)]$\times 2$ to its output layers to bring the masks back
up to the input resolution.

For the backbone and appearance features $V$, we leverage a ViT-8 transformer,
pre-trained on ImageNet Russakovsky et al. (2015) in a self-supervised manner
using DINO Caron et al. (2021) to avoid any external sources of supervision.
For the hierarchical backbone features to decoder we use the key feature
outputs from layers 6, 8, 10, 12.

The input RGB images are interpolated (bi-cubic) to $128\times 224$ resolution
for input to the network. We interpolate (nearest neighbor) the optical flow
to $480\times 854$ for the loss. Output segmentation logits are up-sampled
using bi-linear interpolation to the flow resolution for training and again to
annotation resolution for evaluation.

#### Training Hyperparameters.

The networks are optimised using AdamW Loshchilov and Hutter (2018), with
learning rate of $1.5\times 10^{-4}$, a schedule of linear warm-up from
$1.0\times 10^{-6}$ to $1.5\times 10^{-4}$ over 1.5k iteration and polynomial
decay afterwards. We use batch size of 8 and train for 15k iterations. We
additionally employ gradient clipping when the 2-norm exceeds 0.01 for
stability. The loss multiplier is $0.03$.

#### UNet.

For experiments using U-Net222Implementation from
https://github.com/milesial/Pytorch-UNet., we use the standard 4-layer
version. The batch-size is increased to 16 and learning rate to $7.0\times
10^{-4}$. We also clip the gradients only when 2-norm exceeds 5.0. All other
settings, including optimizer and learning rate schedules, are kept the same.
U-Net is not pre-trained and trained from scratch.

#### Optical Flow.

Our method derives its learning signal from optical flow estimated using off-
the-shelf frozen networks. We estimate optical flow for all frames on DAVIS,
STv2, and FBMS following the practice of MotionGrouping Yang et al. (2021a).
We employ RAFT Teed and Deng (2020) (supervised) using the original resolution
for our main experiments, and gaps between frames of $\\{-2,-1,1,2\\}$ for
DAVIS and STv2, and $\\{-6,-3,3,6\\}$ on FBMS. When multiple flows are
associated with a single frame (multiple gaps), we sample one at random for
each iteration.

##  Appendix B Quadratic Flow Model: Closed Form Solution

Consider one of $K$ regions $m$ and define $w_{u}\propto P(m_{u}=k|I,\Phi)$
the posterior probability for that region, normalized so that
$\sum_{u\in\Omega}w_{u}=1$ (the scaling factor does not matter for the purpose
of finding the minimizer). We can obtain the minimizer $(A^{*},b^{*})$ and
minimum of the energy

$E(A,b)=\sum_{u\in\Omega}w_{u}\|F_{u}-Au-b\|^{2}$ (7)

as follows. Defining

$\bar{u}:=\begin{bmatrix}u\\\
1\end{bmatrix},~{}~{}~{}~{}M:=\begin{bmatrix}A&~{}~{}b\end{bmatrix}\in\mathbb{R}^{2\times
6}$

allows rewriting the energy as

$E(M)=\sum_{u\in\Omega}w_{u}\|F_{u}-M\bar{u}\|^{2}=\operatorname{tr}\left(\Lambda_{FF}-M\Lambda_{\bar{\Omega}F}-\Lambda_{F\bar{\Omega}}M^{\top}+M\Lambda_{\bar{\Omega}\bar{\Omega}}M^{\top}\right),$

where

$\Lambda_{FF}=\sum_{u\in\Omega}w_{u}F_{u}F_{u}^{\top},~{}~{}\Lambda_{F\bar{\Omega}}=\sum_{u\in\Omega}w_{u}F_{u}\bar{u}^{\top},~{}~{}\Lambda_{\bar{\Omega}F}=\Lambda_{F\bar{\Omega}}^{\top},~{}~{}\Lambda_{\bar{\Omega}\bar{\Omega}}=\sum_{u\in\Omega}w_{u}\bar{u}\bar{u}^{\top}.$

are the (uncentered) second moment matrices of the flow $F_{u}$ and
homogeneous coordinate vectors $\bar{u}$. By inspection of the trace term, the
gradient of the energy is given by:

$\frac{dE(M)}{dM}=2\left(\Lambda_{F\bar{\Omega}}-M\Lambda_{\bar{\Omega}\bar{\Omega}}\right)$

Hence, the optimal regression matrix $M^{*}$ and corresponding energy value
are

$M^{*}=\Lambda_{F\bar{\Omega}}\Lambda_{\bar{\Omega}\bar{\Omega}}^{-1},~{}~{}~{}~{}E(M^{*})=\operatorname{tr}\left(\Lambda_{FF}-M^{*}\Lambda_{\Omega\bar{F}}\right).$

Somewhat more intuitive results can be obtained by centering the moments and
resolving for $A$ and $b$ instead of $M$. Specifically, define:

$\mu_{\Omega}:=\sum_{u\in\Omega}w_{u}u,~{}~{}~{}\mu_{F}:=\sum_{u\in\Omega}w_{u}F_{u}.$

The covariance matrices of the vectors are:

$\Sigma_{FF}=\sum_{u\in\Omega}w_{u}(F_{u}-\mu_{F})(F_{u}-\mu_{F})^{\top},~{}~{}~{}\Sigma_{F\Omega}=\sum_{u\in\Omega}w_{u}(F_{u}-\mu_{F})(u-\mu_{\Omega})^{\top},\\\
\Sigma_{\Omega
F}=\Lambda_{F\Omega}^{\top},~{}~{}~{}\Sigma_{\Omega\Omega}=\sum_{u\in\Omega}w_{u}(u-\mu_{\Omega})(u-\mu_{\Omega})^{\top}.$

It is easy to check that

$\Lambda_{FF}=\Sigma_{FF}+\mu_{F}\mu_{F}^{\top},~{}~{}~{}\Lambda_{F\bar{\Omega}}=\begin{bmatrix}\Sigma_{F\Omega}+\mu_{F}\mu_{\Omega}^{\top}&~{}~{}\mu_{F}\end{bmatrix},~{}~{}~{}\Lambda_{\bar{\Omega}\bar{\Omega}}=\begin{bmatrix}\Sigma_{\Omega\Omega}+\mu_{\Omega}\mu_{\Omega}^{\top}&~{}~{}\mu_{\Omega}\\\
\mu_{\Omega}^{\top}&~{}~{}1\end{bmatrix}.$

From this:

$M^{*}=\Lambda_{F\bar{\Omega}}\Lambda_{\bar{\Omega}\bar{\Omega}}^{-1}=\begin{bmatrix}\Sigma_{F\Omega}+\mu_{F}\mu_{\Omega}^{\top}&~{}~{}\mu_{F}\end{bmatrix}\begin{bmatrix}\Sigma_{\Omega\Omega}+\mu_{\Omega}\mu_{\Omega}^{\top}&~{}~{}\mu_{\Omega}\\\
\mu_{\Omega}^{\top}&~{}~{}1\\\ \end{bmatrix}^{-1}\\\
=\begin{bmatrix}\Sigma_{F\Omega}+\mu_{F}\mu_{\Omega}^{\top}&~{}~{}\mu_{F}\end{bmatrix}\begin{bmatrix}\Sigma_{\Omega\Omega}^{-1}&~{}~{}-\Sigma_{\Omega\Omega}^{-1}\mu_{\Omega}\\\
-\mu_{\Omega}^{\top}\Sigma_{\Omega\Omega}^{-1}&~{}~{}1+\mu_{\Omega}^{\top}\Sigma_{\Omega\Omega}^{-1}\mu_{\Omega}\\\
\end{bmatrix}\\\
=\begin{bmatrix}\Sigma_{F\Omega}\Sigma_{\Omega\Omega}^{-1}&~{}~{}\mu_{F}-\Sigma_{F\Omega}\Sigma_{\Omega\Omega}^{-1}\
\mu_{\Omega}\end{bmatrix}=\begin{bmatrix}A^{*}&~{}~{}b^{*}\end{bmatrix}.$

Hence, the optimal regression coefficients and energy value are also given by:

$A^{*}=\Sigma_{F\Omega}\Sigma_{\Omega\Omega}^{-1},~{}~{}~{}b^{*}=\mu_{F}-A^{*}\mu_{\Omega}.$

##  Appendix C Further Experiments

###  C.1 Generalization in Unsupervised Video Segmentation

| Model | Flow |  DAVIS ($\mathcal{J}$$\uparrow$) |  FBMS ($\mathcal{J}$$\uparrow$)
---|---|---|---|---
Liu et al. (2021) | AMD (100 steps) | ✗ | 57.8 | 47.5
|  Ours (Zero shot) | ARFlow | 62.5 | 65.4
|  Ours (20 steps) | ARFlow | 65.2 | 67.6
Meunier et al. (2022) | EM | RAFT | 69.3 | 57.8
|  Ours (Zero shot) | RAFT | 66.8 | 73.2
|  Ours (20 steps) | RAFT | 76.3 | 77.1
Table 4: Generalization performance on unseen _videos_. Few unsupervised
methods operate in this setting. AMD trains on YT-VOS, followed by 100 test-
time adaptation steps, while EM trains on FlyingThings3D using flow as input.
We use (fully unsupervised) ARFlow for fair comparison with AMD. Our method
shows better performance after observing motion. (Test-time adaptation uses
the training loss. No GT is involved at any point.)

We also test our model in a video _generalization_ setting. In contrast to the
protocol of Yang et al. (2019, 2021a), where evaluation set is observed
together with training to infer masks jointly333Note, no annotations are
observed at any point., here we train only on frames from the training set. We
report performance on unseen videos. In this case, our method independently
segments a collection of frames from a new video, with no way to incorporate
motion information.

To “observe” motion on _unseen_ inputs, we also report results after taking 20
test-time adaptation steps (using our unsupervised loss) for each evaluation
sequence in isolation (c.f. AMD Liu et al. (2021) takes 100 test-time
adaptations steps). That is after training, we follow our training setup
(optimizer, rate, batch size) and feed frames from the evaluation video and
corresponding optical flow, calculate loss and take gradient steps. Despite
other methods using much larger training sets, our approach shows better
performance (Table 4).

###  C.2 Ablation Studies

#### Pretraining.

Backbone | Backbone |  | DAVIS | STv2 | FBMS
---|---|---|---|---|---
model | pretraining | Sup. |  $\mathcal{J}$$\uparrow$ |  $\mathcal{J}$$\uparrow$ |  $\mathcal{J}$$\uparrow$
ViT-8 | ImageNet DINO | ✗ | 79.5 | 78.3 | 77.4
UNet | None | ✗ | 78.3 | 76.8 | 72.0
SWIN-tiny | ImageNet MOBY | ✗ | 78.3 | 77.4 | 74.6
SWIN-tiny | ImageNet CLS | ✓ | 78.9 | 77.7 | 75.5
SWIN-tiny | None | ✗ | 78.3 | 75.2 | 68.8
Resnet-50 | ImageNet CLS | ✓ | 77.5 | 75.8 | 72.9
Table 5: Effect of Pretraining/Backbone. Our method with MaskFormer benefits from pretraining, with slight improvement offered by supervised (CLS) over unsupervised (MOBY) pretraining (usng SWIN transformer). Comparable results can be obtained with training from scratch. Best results are obtained using DINO features. |  |  | DAVIS | STv2 | FBMS
---|---|---|---|---|---
Model | $K$ | Merge |  $\mathcal{J}$$\uparrow$ |  $\mathcal{J}$$\uparrow$ |  $\mathcal{J}$$\uparrow$
Ours | $K=4$ | ✓ | 79.5 | 78.3 | 77.4
Spectral clustering | $K=2$ | ✗ | 15.79 | 14.89 | 27.45
K-Means | $K=4$ | ✓ | 41.79 | 34.84 | 48.80
K-Means | $K=2$ | ✗ | 20.24 | 21.14 | 38.25
Table 6: Feature Clustering without Motion. We experiment with offline
clustering of DINO features to assess the importance of our motion-based
formulation. Simply clustering DINO features using K-Means or spectral
clustering Melas-Kyriazi et al. (2022a) into 2 clusters performs worse. Over-
clustering and merging using our cluster-merging approach performs better but
still fails to reach our performance.

Compared to recent methods for video segmentation Yang et al. (2021a); Meunier
et al. (2022), one of the benefits of our formulation is that we can leverage
unsupervised pretraining for the segmentation network (_e.g_ , for the ViT
backbone of MarkFormer). This enables our method to be trained in only 15k
iterations. Here, we investigate the importance of the backbone. To this end
we replace ViT with Swin-tiny pretrained using MOBY (self-supervised) in Table
5. The performance differences are small.

Additionally, we investigate the effect of other pretraining strategies on the
performance. Switching to a model pretrained on ImageNet with image-level
supervision (_i.e_ a classification task) only slightly improves performance
showing that the method does not need to rely on supervised pre-training.
Finally, we train the model using same settings for 20k iterations from
scratch, without any pre-training. This results in comparable performance on
DAVIS but reduced performance on the smaller datasets. Comparing backbones
without pre-training, UNet gives better results than SWIN-tiny, likely due to
smaller networks being easier to train on small datasets.

#### Feature Clustering without Motion.

To demonstrate the potential of using motion for discovering objects, in Table
6, we compare to additional baselines that only rely on clustering visual
features. Spectral feature clustering with $K=2$ (based on Melas-Kyriazi et
al. (2022a)), on the same visual features we use to merge segments (_i.e_ ,
DINO) after over-clustering, shows (somewhat unsurprisingly) that learning
from motion is important for motion segmentation. Similarly, K-means ($K=2$)
on the same features also falls behind our method. Yet, we show that K-means
also benefits from over-clustering ($K=4$) and then merging.

#### Flow Estimation.

Opt. Flow | Sup. |  DAVIS ($\mathcal{J}$$\uparrow$)
---|---|---
Liu et al. (2020) | ARFlow | ✗ | 66.9
Sun et al. (2018) | PWCNet | ✓ | 74.9
Teed and Deng (2020) | RAFT | ✓ | 79.5
Table 7: Choice of Optical Flow Method. Measuring the influence of the method
to extract optical flow.

Method |  DAVIS ($\mathcal{J}$$\uparrow$)
---|---
Yang et al. (2021a) | MG | 53.2
Liu et al. (2021) | AMD | 57.8
| Ours | 66.9
Table 8: Fully Unsupervised Video Object Segmentation. Comparison to the state
of the art in unsupervised VOS without reliance on any supervision

|  | CUB | DUTS | ECSSD | OMRON
---|---|---|---|---|---
|  | Acc | $\mathcal{J}\uparrow$ | $maxF_{\beta}\uparrow$ | Acc | $\mathcal{J}\uparrow$ | $F_{\beta}\uparrow$ | Acc | $\mathcal{J}\uparrow$ | $F_{\beta}\uparrow$ | Acc | $\mathcal{J}\uparrow$ | $F_{\beta}\uparrow$
Xia and Kulis (2017) |  WNet† | – | 24.8 | – | – | – | – | – | – | – | – | – | –
Ji et al. (2019) | IIC-seg | – | 36.5 | – | – | – | – | – | – | – | – | – | –
Bielski and Favaro (2019) | PertGAN | – | 38.0 | – | – | – | – | – | – | – | – | – | –
Chen et al. (2019) | ReDO | 84.5 | 42.6 | – | – | – | – | – | – | – | – | – | –
Kanezaki (2018) | UISB | – | 44.2 | – | – | – | – | – | – | – | – | – | –
Benny and Wolf (2020) | OneGAN | – | 55.5 | – | – | – | – | – | – | – | – | – | –
Yu et al. (2021) | DRC | – | 56.4 | – | – | – | – | – | – | – | – | – | –
He et al. (2021) | GANSeg | – | 62.9 | – | – | – | – | – | – | – | – | – | –
Voynov et al. (2021) |  Voynov _et al._ | 94.0 | 71.0 | 80.7 | 88.1 | 51.1 | 60.0 | 90.6 | 68.4 | 79.0 | 86.0 | 46.4 | 53.3
Liu et al. (2021) | AMD | – | – | – | – | – | 60.2 | – | – | – | – | – | –
Melas-Kyriazi et al. (2022b) |  Kyriazi _et al._ | 92.1 | 66.4 | 78.3 | 89.3 | 52.8 | 61.4 | 91.5 | 71.3 | 80.6 | 88.3 | 50.9 | 58.3
Melas-Kyriazi et al. (2022a) |  Kyriazi _et al._ | – | 76.9 | – | – | 51.4 | – | – | 73.3 | – | 56.7 | – |
Yang et al. (2021b) |  DyStaB† | – | – | – | – | – | – | – | – | 88.1 | – | – | 73.9
Wang et al. (2022) | TokenCut | – | – | – | 90.3 | 57.6 | – | 91.8 | 71.2 | – | 88.0 | 53.3 | –
Shin et al. (2022) | SelfMask | – | – | – | 92.3 | 62.6 | – | 94.4 | 78.1 | – | 90.1 | 58.2 | –
Ours | 93.5 | 64.6 | 80.9 | 91.5 | 49.2 | 65.6 | 88.5 | 56.1 | 74.3 | 89.3 | 41.31 | 56.3
Table 9: Expanded unsupervised object segmentation benchmark CUB and three
saliency detection benchmarks: DUTS, ECSSD, and DUT-OMRON (OMRON). $\dagger$
DyStaB uses CRF post-processing, supervised pre-training, and self-training on
each dataset.

Finally, our method relies on optical flow estimated by frozen, off-the-shelf
networks. So far we have been using RAFT Teed and Deng (2020), as such optical
flow network was adopted in our baselines. In Table 8, we also consider PWCNet
Sun et al. (2018) and fully-unsupervised ARFlow Liu et al. (2020). We observe
that the performance of the flow estimator has an impact on the final
performance of our method. Finally, we compare our _fully_ unsupervised model
(which uses self-supervised pretraining and flow) to fully unsupervised state-
of-the-art methods. Appearance-Motion Decomposition (AMD) Liu et al. (2021)
works end-to-end and directly extracts motion features from pairs of images
with a PWCNet-like architecture, while MotionGrouping (MG) Yang et al. (2021a)
and our method use ARFlow Liu et al. (2020) for optical flow estimation. In
Table 8 we show that our method achieves a significant improvement over
previous approaches.

##  Appendix D Additional Results and Discussion

We provide a further breakdown of our results in Tables 10, 12 and 12,
reporting per sequence evaluation results on the video segmentation tasks.

#### Video object segmentation and egomotion.

We note that some sequences have pronounced egomotion (_e.g_ , camera shaking
in libby of DAVIS or inside a moving car in camel01 of FBMS). Our model
performs well on these sequences, demonstrating that it can handle egomotion.
When _only_ the camera is moving, the resulting optical flow would still
highlight objects due to parallax. This provides a learning signal, however,
it would likely be weaker for objects farther away from the camera. As our
method works on a per-frame basis and does not _require_ flow during
inference, this should not have an impact at test time. However, fine-tuning
on scenes with only egomotion (see Section C.1 for experiments investigating
test-time adaptation) and only small or far away objects, might lead to the
model learning to ignore them.

#### Image segmentation.

For unsupervised image segmentation, we show some additional qualitative
results for CUB in Fig. 4, DUT-OMRON in Fig. 5, DUTS in Fig. 6, and ECSSD in
Fig. 7. Our model, trained on a combined dataset of DAVIS, FBMS and STv2, is
robust enough to handle a wide array of classes from the above datasets in
varying context. Our model can segment both stationary and non-stationary
objects and works well when multiple objects are in the foreground. In Fig. 8,
we show a few failure cases for all datasets, where the model struggles mostly
with ambiguous foreground objects and, in particular, with close-ups of
stationary objects, _e.g_ signs (ECSSD) and buildings (DUT-OMRON). The model
also has issues with boundaries for many objects, _i.e_ the foreground objects
are correctly identified but the model fails to fully segment them. For
example, in DUTS, the snake in the first image has a well segmented head,
however, the model does not segment its body accurately.

| w/o CRF | w/ CRF
---|---|---
Sequence |  $\mathcal{J}$(M) |  $\mathcal{J}$(R) |  $\mathcal{J}$(D) |  $\mathcal{J}$(M) |  $\mathcal{J}$(R) |  $\mathcal{J}$(D)
blackswan | 67.0 | 100.0 | -0.8 | 67.4 | 100.0 | 1.1
bmx-trees | 58.2 | 76.9 | 19.9 | 59.8 | 76.9 | 17.5
breakdance | 86.2 | 100.0 | 4.9 | 87.4 | 100.0 | 5.2
camel | 89.4 | 100.0 | 5.7 | 90.6 | 100.0 | 5.5
car-roundabout | 81.4 | 90.4 | 26.7 | 81.2 | 90.4 | 25.8
car-shadow | 84.3 | 100.0 | 9.0 | 83.9 | 100.0 | 8.0
cows | 90.4 | 100.0 | 3.4 | 91.3 | 100.0 | 3.2
dance-twirl | 87.4 | 100.0 | -7.1 | 88.8 | 100.0 | -6.2
dog | 92.9 | 100.0 | -1.7 | 93.9 | 100.0 | -1.6
drift-chicane | 78.6 | 98.0 | 2.2 | 82.0 | 100.0 | 2.6
drift-straight | 80.6 | 100.0 | 7.2 | 82.1 | 100.0 | 8.2
goat | 78.6 | 100.0 | 1.7 | 75.8 | 100.0 | 4.5
horsejump-high | 84.9 | 100.0 | 6.4 | 88.0 | 100.0 | 4.6
kite-surf | 64.4 | 97.9 | 4.5 | 67.5 | 97.9 | 3.1
libby | 82.9 | 100.0 | 8.6 | 84.5 | 100.0 | 8.6
motocross-jump | 74.1 | 78.9 | 4.1 | 75.1 | 81.6 | 4.1
paragliding-launch | 62.2 | 65.4 | 33.5 | 64.1 | 66.7 | 35.8
parkour | 86.1 | 100.0 | -4.5 | 88.1 | 100.0 | -3.1
scooter-black | 82.1 | 97.6 | -4.3 | 82.1 | 100.0 | -4.3
soapbox | 79.2 | 100.0 | -2.8 | 81.0 | 100.0 | -0.4
Average | 79.5 | 95.3 | 5.8 | 80.7 | 95.7 | 6.1
Table 10: Result breakdown on DAVIS16 validation sequences. (M), (R), and (D)
are mean, recall and decay of IoU, respectively

| w/o CRF | w/ CRF
---|---|---
Sequence |  $\mathcal{J}$(M) |  $\mathcal{J}$(M)
drift | 86.1 | 86.5
birdfall | 67.8 | 57.1
girl | 84.5 | 86.3
cheetah | 57.0 | 50.8
worm | 83.7 | 84.0
parachute | 90.6 | 93.2
monkeydog | 22.9 | 22.6
hummingbird | 57.3 | 57.2
soldier | 77.4 | 77.4
bmx | 76.4 | 77.5
frog | 84.1 | 86.7
penguin | 77.7 | 76.8
monkey | 75.0 | 75.8
bird of paradise | 92.3 | 94.0
Seq. Avg. | 73.8 | 73.3
Frame Avg. | 78.3 | 78.9
Table 11: Sequence breakdown on SegTrackv2 dataset.

| w/o CRF | w/ CRF
---|---|---
Sequence |  $\mathcal{J}$(M) |  $\mathcal{J}$(M)
camel01 | 86.8 | 91.0
cars1 | 86.9 | 86.8
cars10 | 64.6 | 64.8
cars4 | 81.5 | 82.4
cars5 | 81.6 | 82.1
cats01 | 87.7 | 89.5
cats03 | 69.4 | 63.2
cats06 | 66.5 | 67.4
dogs01 | 76.3 | 75.6
dogs02 | 85.3 | 86.4
farm01 | 90.8 | 90.5
giraffes01 | 82.1 | 83.9
goats01 | 79.9 | 83.7
horses02 | 80.4 | 83.6
horses04 | 59.8 | 60.5
horses05 | 72.8 | 74.5
lion01 | 75.1 | 75.0
marple12 | 81.9 | 81.6
marple2 | 84.4 | 85.9
marple4 | 81.1 | 82.4
marple6 | 95.1 | 95.1
marple7 | 76.6 | 77.6
marple9 | 95.4 | 96.3
people03 | 90.1 | 91.0
people1 | 85.3 | 87.2
people2 | 88.1 | 89.7
rabbits02 | 91.2 | 91.2
rabbits03 | 81.5 | 84.4
rabbits04 | 43.8 | 44.1
tennis | 73.3 | 74.2
Seq. Avg. | 79.8 | 80.7
Frame Avg. | 77.4 | 78.4
Table 12: Sequence breakdown on FBMS59 dataset

| CUB
---|---
Ours GT Image Ours GT Image Ours GT Image |
Figure 4: Qualitative Comparison on CUB. We train our model on a combined dataset of DAVIS, FBMS and STv2. Our method can extract birds in different environments and poses. Our model can segment different species of birds | DUT-OMRON
---|---
Ours GT Image Ours GT Image Ours GT Image |
Figure 5: Qualitative Comparison on DUT-OMRON. We train our model on a combined dataset of DAVIS, FBMS and STv2. Our model can segment both stationary and non-stationary objects and is robust enough to work on a wide range of classes  | DUTS
---|---
Ours GT Image Ours GT Image Ours GT Image |
Figure 6: Qualitative Comparison on DUTS. We train our model on a combined dataset of DAVIS, FBMS and STv2. We can segment a wide array of classes. Our model performs well on scenes where multiple objects are in the foreground | ECSSD
---|---
Ours GT Image Ours GT Image Ours GT Image |
Figure 7: Qualitative Comparison on ECSSD. We train our model on a combined dataset of DAVIS, FBMS and STv2. Our model can segment objects from different classes in complex poses Ours GT Image Ours GT Image Ours GT Image Ours GT Image |
---|---
Figure 8: Qualitative Comparison of Failure Cases. We train our model on a
combined dataset of DAVIS, FBMS and STv2. Our method can extract salient
object in various environments. The model has difficulty where the foreground
object is ambiguous — when there are multiple prominent objects but only few
are annotated as salient object. The model also has issues with predicting the
object boundaries well for some instances