(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Visual Geometry Group, University of Oxford22institutetext:
CMIC, Shanghai Jiao Tong University
22email<EMAIL_ADDRESS>https://www.robots.ox.ac.uk/~vgg/research/flowsam/

# Moving Object Segmentation:
All You Need Is SAM (and Flow)

Junyu Xie 11 Charig Yang 11 Weidi Xie 1122 Andrew Zisserman 11

###### Abstract

The objective of this paper is motion segmentation – discovering and
segmenting the moving objects in a video. This is a much studied area with
numerous careful, and sometimes complex, approaches and training schemes
including: self-supervised learning, learning from synthetic datasets, object-
centric representations, amodal representations, and many more. Our interest
in this paper is to determine if the Segment Anything model (SAM) can
contribute to this task.

We investigate two models for combining SAM with optical flow that harness the
segmentation power of SAM with the ability of flow to discover and group
moving objects. In the first model, we adapt SAM to take optical flow, rather
than RGB, as an input. In the second, SAM takes RGB as an input, and flow is
used as a segmentation prompt. These surprisingly simple methods, without any
further modifications, outperform all previous approaches by a considerable
margin in both single and multi-object benchmarks. We also extend these frame-
level segmentations to sequence-level segmentations that maintain object
identity. Again, this simple model outperforms previous methods on multiple
video object segmentation benchmarks.

###### Keywords:

Motion Segmentation Video Object Segmentation Motion-based Object Discovery

##  1 Introduction

Figure 1:  Adapting SAM for Video Object Segmentation by incorporating flow.
(a) Flow-as-Input: FlowI-SAM takes in optical flow only and predicts frame-
level segmentation masks. (b) Flow-as-Prompt: FlowP-SAM takes in RGB and
applies flow information as a prompt for frame-level segmentation. (c)
Sequence-level mask association: as a post-processing step, the multi-mask
selection module autoregressively transforms frame-level mask outputs from
FlowI-SAM and/or FlowP-SAM and produces sequence-level masks in which object
identities are consistent throughout the sequence.

Recent research in image segmentation has been transformative, with the
Segment Anything Model (SAM) [12] emerging as a significant breakthrough.
Leveraging large-scale datasets and scalable self-labelling, SAM enables
flexible image-level segmentation across many scenarios [23, 42, 34, 36, 53,
4], facilitated by user prompts such as boxes, texts, and points. In videos,
optical flow has played an important and successful role for moving object
segmentation – in that it can (i) discover moving objects, (ii) provide crisp
boundaries for segmentation, and (iii) group parts of objects together if they
move together. It has formed the basis for numerous methods of moving object
discovery by self-supervised learning [47, 26, 43, 15, 25, 10, 35]. However,
it faces segmentation challenges if objects are momentarily motionless, and in
distinguishing foreground objects from background ‘noise’. This naturally
raises the question: “How can we leverage the power of SAM with flow for
moving object segmentation in videos?”.

To this end, we explore two distinct yet simple variants to effectively tailor
SAM for motion segmentation. First, we introduce FlowI-SAM (Fig. 1a), an
adaption of the original SAM that directly processes optical flow as a three-
channel input image for segmentation, where points on a uniform grid are used
as prompts. This approach leverages the ability of SAM to accurately segment
moving objects against the static background, by exploiting the distinct
textures and clear boundaries present in optical flow fields. However, it has
less success in scenes where the optical flow arises from multiple interacting
objects as the flow only contains limited information for separating them.
Second, building on the strong ability of SAM on RGB image segmentation, we
propose FlowP-SAM (Fig. 1b) where the input is an RGB frame, and flow guides
SAM for moving object segmentation as prompts, produced by a trainable prompt
generator. This method effectively leverages the ability of SAM on RGB image
segmentation, with flow information acting as a selector of moving
objects/regions within a frame. Additionally, we extend these methods from
frame-level to sequence-level video segmentation (Fig. 1c) so that object
identities are consistent throughout the sequence. We do this by introducing a
matching module that auto-regressively chooses whether to select a new object
or propagate the old one based on temporal consistency.

In summary, this paper introduces and explores two models to leverage SAM for
moving object segmentation in videos, enabling the principal moving objects to
be discriminated from background motions. Our contributions are threefold:

  * $\bullet$

The FlowI-SAM model, which utilises optical flow as a three-channel input
image for precise per-frame segmentation and moving object identification.

  * $\bullet$

The FlowP-SAM model, a novel combination of dual-stream (RGB and flow) data,
that employs optical flow to generate prompts, guiding SAM to identify and
localise the moving objects in RGB images.

  * $\bullet$

New state-of-the-art performance by a large margin on moving object
segmentation benchmarks, including DAVIS, YTVOS, and MoCA, at both frame and
sequence levels.

##  2 Related Work

Video object segmentation (VOS) is an extensively studied task in computer
vision. The objective is to segment the primary object(s) in a video sequence.
Numerous benchmarks are developed for evaluating VOS performance, catering to
both single-object [31, 18, 29, 16] and multi-object [33, 46] scenarios. Two
major protocols are widely explored in VOS research, namely unsupervised [49,
47, 21, 38, 8, 19, 22, 20] and semi-supervised VOS [39, 14, 13, 27, 40, 11, 2,
30, 50, 6]. Notably, the term “unsupervised” exclusively indicates that no
groundtruth annotation is used during inference time (i.e., no inference-time
supervision). In contrast, the semi-supervised VOS employs first-frame
groundtruth annotations to initiate the object tracking and mask propagation
in subsequent frames. This paper focuses on unsupervised VOS tasks and
utilises motion as a crucial cue for object discovery.

Motion Segmentation focuses on discovering objects through their movement and
generating corresponding segmentation masks. Existing benchmarks for motion
segmentation largely overlap with those used for VOS evaluation, especially in
the single-object case. For multi-object motion segmentation, datasets [43,
44] have been specifically curated from VOS benchmarks to exclusively focus on
sequences with dominant locomotion. There are two major setups in the motion
segmentation literature: one that relies on motion information only to
distinguish moving elements from the background through spatial clustering
[47, 25, 26] or explicit supervision [15, 43]; the other [1, 24, 48, 10, 44,
32] that enhances motion-based object discovery by incorporating appearance
information. We term these two approaches “flow-only” and “RGB-based”
segmentation, respectively, and explore both setups in this work.

Segment Anything Model (SAM) [12] has demonstrated impressive ability on image
segmentation across various scenarios. It was trained on the SA-1B datasets
with over one billion self-labelled masks and $11$ million images. Such large-
scale training renders it a strong zero-shot generalisability to unseen
domains. Many works adapt the SAM model to perform different tasks, such as
tracking [7], change detection [56], and 3D segmentation [3]. Some other works
extend SAM towards more promptability [58], more efficient models [51, 55,
45], or more domains [23, 42, 34, 4]. Our work extends the robust SAM
framework to moving object segmentation by adapting it with optical flow
inputs and flow prompts.

##  3 SAM Preliminaries

The Segment Anything Model (SAM) is engineered for high-precision image
segmentation, accommodating both user-specified prompts and a fully autonomous
operation mode. When guided by user input, SAM accepts various forms of
prompts including points, boxes, masks, or textual descriptions to accurately
delineate the segmentation targets. Alternatively, in its automatic mode, SAM
uses points on a uniform grid as prompts, to propose all plausible
segmentation masks that capture objects and their hierarchical
subdivisions—objects, parts, and subparts. In this case, the inference is
repeated for each prompt of the grid, generating masks for each prompt in
turn, and the final mask selection is guided by the predicted mask IoU scores.

Architecturally, SAM is underpinned by three foundational components: (i)
Image encoder extracts strong image features via a heavy Vision Transformer
(ViT) backbone, which is pre-trained by the Masked Auto-Encoder (MAE)
approach; (ii) The prompt encoder converts the input prompts into positional
information which helps with locating the segmentation target; (iii) Mask
decoder features a light-weight two-way transformer that takes in a
combination of encoded prompt tokens, learnable mask tokens, and an IoU
prediction token as input queries. These queries iteratively interact with the
dense spatial features from image encoder, leading to the final mask
predictions and IoU estimations. In the next sections, we describe two
distinct, yet simple variants to effectively tailor SAM for motion
segmentation.

##  4 Frame-Level Segmentation I: Flow as Input

Figure 2: Overview of FlowI-SAM. (a) Inference pipeline of FlowI-SAM. (b)
Architecture of FlowI-SAMwith trainable parameters labelled. The point prompt
token is generated by a frozen prompt encoder.

In this section, we focus on discovering moving objects from individual frames
by exploiting motion information only, to yield corresponding segmentation
masks. Formally, given the optical flow input $F_{t}\in\mathbb{R}^{H\times
W\times 3}$ at frame $t$, we aim to predict a segmentation mask
${M^{i}_{t}}\in\mathbb{R}^{H\times W}$ together with a foreground object IoU
(fIoU) score $s_{\text{fIoU},t}^{i}\in\mathbb{R}^{1}$ for each object $i$,

$\\{M^{i}_{t},s_{\text{fIoU},t}^{i}\\}_{i=0}^{N}=\Phi_{\text{{\tt FlowI-
SAM}}}(F_{t})$ (1)

To adapt SAM for this new task, we formulate FlowI-SAM ($\Phi_{\text{{\tt
FlowI-SAM}}}$) by finetuning it on optical Flow Inputs, and re-purpose the
original IoU prediction head to instead predict the fIoU, as illustrated in
Fig. 2b.

Flow Inputs with Multiple Frame Gaps. To mitigate the effect of noisy optical
flow, i.e. complicated flow fields due to stationary parts, articulated
motion, and object interactions, etc., we consider multiple flow inputs
$\\{F_{t,g}\\}$ with different frame gaps (e.g.,
$g\in\\{(1,$-$1),(2,$-$2)\\}$) for both training and evaluation stages. These
multi-gap flow inputs are independently processed by the image encoder to
obtain dense spatial features $\\{d_{t,g}\\}$ at a lower resolution $h\times
w$, which are then combined by averaging the spatial feature maps across
different flow gaps, i.e.,
$d_{t}=\text{Average}_{g}(\\{d_{t,g}\\})\in\mathbb{R}^{h\times w\times d}$.

FlowI-SAM Inference. To discover all moving objects from flow input, the
FlowI-SAM model is prompted by points on a uniform grid. Each point prompt
outputs a pair of mask and objectness score predictions. This mechanism is the
same as in the original SAM formulation, and is illustrated in Fig. 2a. The
final segmentation is selected using Non-Maximum Suppression (NMS) based on
the predicted fIoU and overlap ratio.

FlowI-SAM Training. To adapt the pre-trained SAM model for optical flow
inputs, we finetune the lightweight mask decoder, while the image encoder and
the prompt encoder remain frozen. The overall loss is formulated as:

$\mathcal{L}_{\text{{\tt FlowI-
SAM}}}=\frac{1}{NT}\sum_{i,t}^{N,T}\left(\mathcal{L}_{\text{BCE}}(M^{i}_{t},\hat{M}^{i}_{t})+\lambda_{f}\lVert
s_{\text{fIoU},t}^{i}-\hat{s}_{\text{fIoU},t}^{i}\lVert^{2}\right)$ (2)

where $\hat{M}^{i}_{t}$ and $\hat{s}_{\text{fIoU},t}^{i}$ denote the
groundtruth segmentation masks and fIoU, and $\lambda_{f}$ is a scale factor.

##  5 Frame-Level Segmentation II: Flow as Prompt

In this section, we adapt SAM to video object segmentation by processing RGB
frames, with optical flow as a prompt. We term this frame-level segmentation
architecture FlowP-SAM for Flow as Prompt SAM. As shown in Fig. 3b, FlowP-SAM
encompasses two major modules, namely the flow prompt generator and the
segmentation module. The flow prompt generator takes optical flow as inputs,
and produces flow prompts that can be used as supplemental queries to infer
frame-level segmentation masks ${M^{i}_{t}}$ from RGB inputs $I_{t}$.
Formally,

$\\{M^{i}_{t},\,s_{\text{fIoU},t}^{i},\,s_{\text{MOS},t}^{i}\\}_{i=0}^{N}=\Phi_{\text{{\tt
FlowP-SAM}}}(F_{t},I_{t})$ (3)

where $s_{\text{MOS},t}^{i}$ indicates the moving object score (MOS) predicted
by the flow prompt generator, while $s_{\text{fIoU},t}^{i}$ denotes the
foreground object IoU (fIoU) estimated by the segmentation module.
Specifically, MOS specifies whether the input point prompt belongs to moving
objects ($s_{\text{MOS},t}^{i}=1$) or the stationary region
($s_{\text{MOS},t}^{i}=0$). On the other hand, fIoU follows the same
formulation as in the FlowI-SAM, i.e., predicting IoUs for foreground objects
and yielding $0$ for background regions.

Figure 3: Overview of FlowP-SAM. (a) Inference pipeline of FlowP-SAM. (b)
Architecture of FlowP-SAM. The flow prompt generator produces flow prompts to
be injected into a SAM-like RGB-based segmentation module. Both modules take
in the same point prompt token, which is obtained from a frozen prompt
encoder. (c) Detailed architecture of the flow transformer. The input tokens
function as queries within a lightweight transformer decoder, iteratively
attending to dense flow features. The output moving object score (MOS) token
is then processed by an MLP-based head to predict a score indicating whether
the input point prompt corresponds to a moving object.

Flow Prompt Generator consists of (i) a frozen SAM image encoder, where the
dense spatial features are extracted from optical flow inputs at different
frame gaps, followed by an averaging across frame gaps; (ii) a flow
transformer, with the detailed architecture depicted in Fig. 3c, where we
first concatenate the input point prompt (i.e., a positional embedding) with
learnable flow prompts and moving object score (MOS) tokens to form queries.
These queries then iteratively attend to dense flow features in a lightweight
transformer decoder. There are two outputs from the flow prompt generator,
namely, the refined flow prompts, and an MOS token, which is subsequently
processed by an MLP-based head to yield a final moving object score.

Segmentation Module. The overall structure of the segmentation module
resembles the original SAM, except for two adaptions: (i) The IoU-prediction
head is re-purposed to predict foreground object scores (fIoU) (same as the
FlowI-SAM); (ii) The outputs tokens from flow prompt generator are injected as
additional query inputs.

FlowP-SAM Inference. Similar to FlowI-SAM, we prompt FlowP-SAM with single
point prompts from a uniform grid to iteratively predict possible segmentation
masks, together with MOS and fIoU estimations. These predicted scores are
averaged, i.e., $(s_{\text{MOS},t}^{i}+s_{\text{fIoU},t}^{i})/2$, and then
utilised to guide post-processing which involves the NMS and overlay of
selected masks.

FlowP-SAM Training. We train FlowP-SAM in an end-to-end fashion while keeping
the SAM pre-trained prompt encoder and image encoders frozen. The flow
transformer is trained from scratch, and the mask decoder in the segmentation
module is finetuned on top of the pre-trained SAM. The overall loss is:

$\displaystyle\mathcal{L}_{\text{{\tt FlowP-SAM}}}=$
$\displaystyle\frac{1}{NT}\sum_{i,t}^{N,T}\Big{(}\Big{.}\mathcal{L}_{\text{BCE}}(M^{i}_{t},\hat{M}^{i}_{t})$
$\displaystyle+$
$\displaystyle\lambda_{m}\mathcal{L}_{\text{BCE}}(s_{\text{MOS},t}^{i},\hat{s}_{\text{MOS},t}^{i})+\lambda_{f}\lVert
s_{\text{fIoU},t}^{i}-\hat{s}_{\text{fIoU},t}^{i}\lVert^{2}\Big{)}\Big{.}$ (4)

where $\hat{M}^{i}_{t}$ corresponds to the groundtruth mask.
$\hat{s}_{\text{MOS},t}^{i}$ and $\hat{s}_{\text{fIoU},t}^{i}$ indicate the
groundtruth of two predicted scores, with $\lambda_{m}$ and $\lambda_{f}$
being the scale factors.

##  6 Sequence-level Mask Association

In this section, we outline our method for linking the frame-wise predictions
for each moving object into a continuous track throughout the sequence.
Specifically, we compute two types of masks: frame-wise masks $M$ at the
current frame using the model (FlowI-SAM and/or FlowP-SAM); and sequence-level
masks $\mathscr{M}$, that are obtained by propagating the initial frame
prediction with optical flow, we then update the mask of the current frame by
making a comparison between the two. The following section details our update
mechanism.

Update Mechanism. This process aims to associate object masks across frames,
as well as to determine whether the sequence-level results at a particular
frame should be obtained directly from frame-level predictions at that frame
or by propagating from previous frame results.

Specifically, given a sequence-level mask for object $i$ at frame $t-1$ (i.e.,
$\mathscr{M}^{i}_{t-1}$), we first warp it to frame $t$ using optical flow,

$\mathscr{M}^{i}_{t\leftarrow t-1}=\text{warp}(\mathscr{M}^{i}_{t-1},F_{t-1})$
(5)

We then consider three sets of masks: (i) the warped masks
$\\{\mathscr{M}^{i}_{t\leftarrow t-1}\\}$; (ii) the frame-level predictions
$\\{{M}^{i}_{t}\\}$ at frame $t$; (iii) the frame-level predictions from
neighboring frames (with $\Delta{t}$ gap) after aligning them to the current
frame using optical flow (i.e., $\\{{M}^{i}_{t\leftarrow t+\Delta{t}}\\}$).
For each pair of mask sets, we perform a pairwise Hungarian matching based on
the IoU score, resulting in three pairings in total. The Hungarian matched
pairs can then reflect the temporal consistency across these predictions based
on the transitivity principle, _i.e_. if object $i$ in (i) matches with object
$j$ in (ii) and object $k$ in (iii), then the latter two objects must also
match with each other. If such transitivity holds, we set the consistency
score $c_{i}=1$, and $c_{i}=0$ otherwise.

This matching process is repeated for $\Delta{t}\in\\{1,2,-1,-2\\}$, resulting
in an averaged consistency score $\bar{c}_{i}$, which guides the following
mask update:

$\mathscr{M}^{i}_{t}=\begin{cases}{M}^{i}_{t}&\text{if }\bar{c}_{i}\geq 0.5\\\
\mathscr{M}^{i}_{t\leftarrow t-1}&\text{if }\bar{c}_{i}<0.5\end{cases}$ (6)

where $\mathscr{M}^{i}_{t}$ denotes the sequence-level mask prediction for
object $i$ at frame $t$.

The rationale behind this is that the two methods have their own strengths and
drawbacks: propagation is safe in preserving the object identity, but the mask
quality degrades over time, whereas updating ensures high-quality masks but
comes with the risk of mis-associating object identities. Thus, if the current
frame-wise mask is temporally consistent, then we can be reasonably confident
to update the mask, if not, then we choose the safer option and propagate the
previous mask.

Note that we do this separately for each object $i\in N$, so each object gets
updated or propagated independently. We lastly layer all the objects back
together (to its original order) and remove any overlaps to obtain the final
sequence-level predictions.

##  7 Experiments

###  7.1 Datasets

Single-Object Benchmarks. For single-object motion segmentation, we adopt
standard datasets, including DAVIS2016 [31], SegTrackv2 [18], FBMS-59 [28],
and MoCA [16]. Although SegTrackv2 and FBMS-59 include a few multi-object
sequences, following the common practice [47, 15], we treat them as single-
object benchmarks by grouping all moving objects into a single foreground
mask. MoCA stands for Moving Camouflaged Animals, designed as a camouflaged
object detection benchmark. Following [47, 15, 43], we adopt a filtered MoCA
dataset by excluding videos with predominantly no locomotion.

Multi-Object Benchmarks. In terms of multi-object segmentation, we report the
performance on DAVIS2017 [33], DAVIS2017-motion [33, 43], and YouTube-
VOS2018-motion [46, 44], where DAVIS2017 is characterised by predominantly
moving objects, each annotated as distinct entities. For example, a man riding
a horse would be separately labelled. In contrast, DAVIS2017-motion re-
annotates the objects based on their joint movements such that objects with
shared motion are annotated as a single entity. For example, a man riding a
horse is annotated as a single entity due to their shared motion.

The YouTubeVOS2018-motion [44] dataset is a curated subset of the original
YouTubeVOS2018 [46]. It specifically excludes video sequences involving common
motion, severe partial motion, and stationary objects, making it ideally
suited for motion segmentation evaluation. Whereas, the original dataset also
annotates many stationary objects and only provides partial annotations for a
subset of moving objects.

Summary of Evaluation Datasets. To investigate the role of motion in object
discovery and segmentation, we adopt all aforementioned benchmarks, which
consist of only predominantly moving object sequences. Notably, for the
evaluation of FlowI-SAM, we exclude the class-labelled DAVIS2017 dataset, as
commonly moving objects cannot be separated solely based on motion cues.

Training Datasets. To adapt the RGB pre-trained SAM for moving object
discovery and motion segmentation, we train both FlowI-SAM and FlowP-SAM first
on the synthetic dataset introduced by [43], as described below, and then on
real-world video datasets, including DAVIS2016, DAVIS2017, and
DAVIS2017-motion.

###  7.2 Evaluation Metrics

To assess the accuracy of predicted masks, we report intersection-over-union
($\mathcal{J}$), except for MoCA where only the ground-truth bounding boxes
are given, we instead follow the literature [47] and report the detection
success rate (SR). Regarding multi-object benchmarks, we additionally report
the contour accuracy ($\mathcal{F}$) in Appendix 0.D.

In this work, we differentiate between the frame-level and sequence-level
methods, and adopt two distinct evaluation protocols: (i) Since frame-level
methods generate the segmentation independently for each frame, we apply per-
frame Hungarian matching to match the object masks between predictions with
the groundtruth, before the evaluation; (ii) Conversely, sequence-level
methods employ an extra step to link object masks across frames. As a result,
the Hungarian matching is conducted globally for each sequence, i.e., the
object IDs between predicted and groundtruth masks are matched once per
sequence. Given the added complexity and potential errors during frame-wise
object association, sequence-level predictions are often considered a greater
challenge.

###  7.3 Implementation Details

In this section, we summarise the experimental setting in our frame-level
segmentation models. For more information regarding detailed architectures,
hyperparameter settings, and sequence-level mask associations, please refer to
Appendix 0.A.

Flow Computation. We adopt an off-the-shelf method (RAFT [37]) to estimate
optical flow with multiple frame gaps at $(1,$-$1)$ and $(2,$-$2)$, except for
YTVOS18-m and FBMS-59, where higher frame gaps at $(3,$-$3)$ and $(6,$-$6)$
are used to compensate for slow motion.

Model Settings. For both FlowI-SAM and FlowP-SAM, we follow the default SAM
setting and adopt the first output mask token (out of four) for mask
predictions. For FlowI-SAM, we deploy two versions of the pre-trained SAM
image encoder, specifically ViT-B and ViT-H, to extract optical flow features.

Regarding FlowP-SAM, for efficiency reasons, we utilise ViT-B to encode
optical flows and employ ViT-H as the image encoder for RGB frames. We
initialise the flow prompt generator with $4$ learnable flow prompt tokens,
which are subsequently processed by a light-weight two-layer transformer
decoder in the flow transformer module.

Evaluation Settings. At inference time, for FlowI-SAM with flow-only inputs,
we input independent point prompts over a $10\times 10$ uniform grid, while
for FlowP-SAM, to take account for more complicated RGB textures, we consider
a large grid size of $20\times 20$.

Mask Selection over Multiple Point Prompts. During post-processing, we utilise
the predicted scores (fIoU for FlowI-SAM, and an average of MOS and fIoU for
FlowP-SAM) as guidance throughout the mask selection process: (i) Non-Maximum
Suppression (NMS) is applied to filter out repeating masks and keep the ones
with higher scores; (ii) The remaining masks are then ranked according to
their scores and top-$n$ masks are retained ($n=5$ for FlowI-SAM, and $n=10$
for FlowP-SAM); (iii) These $n$ masks are overlaid by allocating masks with
higher scores at the front.

Training Settings. The training is performed in two stages, which involves
synthetic pre-training on the dataset proposed by [43], followed by finetuning
on the real DAVIS sequences, as detailed in Section 0.A.1. YTVOS is not used
for fine-tuning as there is only a low proportion of moving object sequences.
We train both models in an end-to-end manner using the Adam Optimiser at a
learning rate of $3e^{-5}$. The training was conducted on a single NVIDIA A40
GPU, with each mode taking roughly $3$ days to reach full convergence.

###  7.4 Ablation Study

We present a series of ablation studies on key designs in the per-frame FlowI-
SAM
and FlowP-SAM models, as well as on our sequence-level method. For a more
detailed ablation analysis, we refer the reader to Appendix 0.B.

Optical flow frame gaps |  DAVIS17-m $\mathcal{J}\uparrow$ |  DAVIS16 $\mathcal{J}\uparrow$
---|---|---
$1,$ -$1$ | $64.5$ | $78.0$
$2,$ -$2$ | $65.3$ | $77.7$
$1,$-$1,2,$-$2$ | $65.7$ | $79.1$

Table 1:  The frame gaps of optical flow inputs to FlowI-SAM. The SAM ViT-H
image encoder is adopted. The results are shown for frame-level predictions.

Flow features combination |  DAVIS17-m $\mathcal{J}\uparrow$ |  DAVIS16 $\mathcal{J}\uparrow$
---|---|---
Taking max | $65.0$ | $78.2$
Averaging | $65.7$ | $79.1$

Table 2: The combination of dense flow features in FlowI-SAM. The SAM ViT-H
image encoder is adopted. The results are shown for frame-level predictions.

Ablation Studies for FlowI-SAM. Below are the investigations focusing on input
optical flows and how the dense flow features are combined.

  * $\bullet$

Optical Flow Frame Gaps. As demonstrated in Table 1, utilizing optical flow
inputs with multiple frame gaps (i.e., $1$,-$1$,$2$,-$2$) results in
noticeable performance boosts across both multi- and single-object benchmarks.
This improvement is attributed to the consistency of motion information over
extended temporal ranges, which effectively mitigates the impact of noise in
optical flow inputs caused by slow movements, partial motions, etc.

  * $\bullet$

Combination of Flow Features. We have explored two combination schemes: (i)
taking the maximum; and (ii) averaging across different frame gaps. According
to Table 2, the averaging approach yields superior results.

Stage | Predicted scores | Flow | FT mask | DAVIS17 | DAVIS16
---|---|---|---|---|---
for post-processing | prompt | decoder |  $\mathcal{J}$ $\uparrow$ |  $\mathcal{J}$ $\uparrow$
| IoU | ✗ | ✗ | $25.2$ | $30.3$
+ Train flow prompt generator | IoU | $\checkmark$ | ✗ | $61.9$ | $80.6$
(MOS+IoU)/2 | $\checkmark$ | ✗ | $63.7$ | $81.4$
+ Finetune segment -ation module | (MOS+IoU)/2 | $\checkmark$ | $\checkmark$ | $65.5$ | $81.5$
(MOS+fIoU)/2 | $\checkmark$ | $\checkmark$ | $69.9$ | $86.1$

Table 3: Ablation analysis of FlowP-SAM. The study starts from the vanilla SAM
checkpoint and progressively introduces new components (labelled in blue).
“MOS” is short for the moving object score, and “fIoU” indicates the
foreground object IoU. The results are shown for frame-level predictions.

Ablation Studies for FlowP-SAM. As illustrated in Table 3, we start from the
vanilla SAM and progressively add our proposed components. Note that, we adopt
the same inference pipeline (i.e., the same point prompts and post-processing
steps) for all predictions shown. Since the foreground object IoU (fIoU) is
not predicted by the vanilla SAM, we instead apply default IoU predictions to
guide the mask selection.

We train the flow prompt generator to simultaneously predict flow prompt
tokens and moving object scores (MOS). The injection of flow prompts into the
standard RGB-based SAM architecture results in notable enhancements, verifying
the value of motion information for accurately determining object positions
and shapes. Additionally, employing MOS as additional post-processing guidance
yields further improvements.

Upon finetuning the segmentation module, we observe a slight enhancement in
performance. Finally, substituting the default IoU predictions with fIoU
scores achieves more precise mask selection, as evidenced by the improved
results.

###  7.5 Quantitative Results

Given the distinct evaluation protocols outlined in Sec. 7.2, we report our
method separately, with a frame-level analysis for FlowI-SAM (introduced in
Sec. 4) and FlowP-SAM (introduced in Sec. 5), followed by a sequence-level
evaluation.

|  |  | Multi-object benchmarks | Single-object benchmarks
---|---|---|---|---
|  |  |  YTVOS18-m $\mathcal{J}$ $\uparrow$ |  DAVIS17-m $\mathcal{J}$ $\uparrow$ |  DAVIS17 $\mathcal{J}$ $\uparrow$ |  DAVIS16 $\mathcal{J}$ $\uparrow$ |  STv2 $\mathcal{J}$ $\uparrow$ |  FBMS $\mathcal{J}$ $\uparrow$ |  MoCA SR $\uparrow$
Model | Flow | RGB
Flow-only methods |  |  |  |
COD [16] | $\checkmark$ | ✗ | $-$ | $-$ | $-$ | $65.3$ | $-$ | $-$ | $0.236$
†MG [47] | $\checkmark$ | ✗ | $37.0$ | $38.4$ | $-$ | $68.3$ | $58.6$ | $53.1$ | $0.484$
†EM [25] | $\checkmark$ | ✗ | $-$ | $-$ | $-$ | $69.3$ | $55.5$ | $57.8$ | $-$
FlowI-SAM (ViT-B) | $\checkmark$ | ✗ | $56.7$ | $63.2$ | $-$ | $\mathbf{79.4}$ | $69.0$ | $72.9$ | $\mathbf{0.628}$
FlowI-SAM (ViT-H) | $\checkmark$ | ✗ | $\mathbf{58.6}$ | $\mathbf{65.7}$ | $-$ | $79.1$ | $\mathbf{70.1}$ | $\mathbf{75.1}$ | $0.625$
RGB-based methods |  |  |  |
†VideoCutLER [41] | ✗ | $\checkmark$ | $59.0$ | $57.4$ | $41.7$ | $-$ | $-$ | $-$ | $-$
†Safadoust et al. [35] | ✗ | $\checkmark$ | $-$ | $59.3$ | $-$ | $-$ | $-$ | $-$ | $-$
MATNet [57] | $\checkmark$ | $\checkmark$ | $-$ | $-$ | $56.7$ | $82.4$ | $50.4$ | $76.1$ | $0.544$
DystaB [48] | ✗ | $\checkmark$ | $-$ | $-$ | $-$ | $82.8$ | ${74.2}$ | $75.8$ | $-$
TransportNet [52] | $\checkmark$ | $\checkmark$ | $-$ | $-$ | $-$ | $84.5$ | $-$ | $78.7$ | $-$
TMO [9] | $\checkmark$ | $\checkmark$ | $-$ | $-$ | $-$ | ${85.6}$ | $-$ | ${79.9}$ | $-$
FlowP-SAM | $\checkmark$ | $\checkmark$ | ${76.9}$ | ${78.5}$ | ${69.9}$ | ${86.1}$ | ${83.9}$ | ${87.9}$ | $\mathbf{0.645}$
FlowP-SAM+FlowI-SAM | $\checkmark$ | $\checkmark$ | $\mathbf{77.4}$ | $\mathbf{80.0}$ | $\mathbf{71.6}$ | $\mathbf{86.2}$ | $\mathbf{84.2}$ | $\mathbf{88.7}$ | $\mathbf{0.645}$

Table 4:  Frame-level comparison on video object segmentation benchmarks.
“${{\dagger}}$” indicates models that are trained without human annotations.
For the results in the last row, we combine frame-level predictions from
FlowP-SAM and FlowI-SAM (ViT-H).

Frame-Level Performance. Table 4 distinguishes between flow-only and RGB-based
methods, where the former adopts optical flow as the only input modality, and
the latter takes in RGB frames with optional flow inputs. Note that, the
performance for some recent self-supervised methods is also reported, owing to
the lack of the supervised baselines.

In terms of the flow-only segmentation, our FlowI-SAM (with both SAM image
encoders) outperforms the previous methods by a large margin (>$10\%$).
Regarding the RGB-based segmentation, our FlowP-SAM also achieves state-of-
the-art performance, particularly excelling at multi-object benchmarks. By
combining these two frame-level predictions (FlowI-SAM+FlowP-SAM), we observe
further performance boosts. This suggests the complementary roles of the flow
and RGB modalities in frame-level segmentation, particularly when there are
multiple moving objects involved. In particular, we show that using both
models in tandem by layering FlowI-SAM’s predictions behind that of FlowP-SAM
allows the model to fill in on missed predictions (such as motion blur, poor
lighting, or small objects).

|  |  | Multi-object benchmarks | Single-object benchmarks
---|---|---|---|---
|  |  |  YTVOS18-m $\mathcal{J}$ $\uparrow$ |  DAVIS17-m $\mathcal{J}$ $\uparrow$ |  DAVIS17 $\mathcal{J}$ $\uparrow$ |  DAVIS16 $\mathcal{J}$ $\uparrow$ |  STv2 $\mathcal{J}$ $\uparrow$ |  FBMS $\mathcal{J}$ $\uparrow$ |  MoCA SR $\uparrow$
Model | Flow | RGB
Flow-only methods |  |  |  |
†SIMO [15] | $\checkmark$ | ✗ | $-$ | $-$ | $-$ | $67.8$ | $62.0$ | $-$ | $0.566$
†Meunier et al. [26] | $\checkmark$ | ✗ | $-$ | $-$ | $-$ | $73.2$ | $55.0$ | $59.4$ | $-$
†OCLR [43] | $\checkmark$ | ✗ | $46.5$ | $54.5$ | $-$ | $72.1$ | $67.6$ | $70.0$ | $0.599$
OCLR-real [43] | $\checkmark$ | ✗ | $49.5$ | $55.7$ | $-$ | $73.3$ | $65.9$ | $70.5$ | $0.605$
FlowI-SAM (seq, ViT-B) | $\checkmark$ | ✗ | $51.9$ | $60.0$ | $-$ | $\mathbf{78.4}$ | $66.9$ | $69.0$ | $\mathbf{0.615}$
FlowI-SAM (seq, ViT-H) | $\checkmark$ | ✗ | $\mathbf{53.8}$ | $\mathbf{61.5}$ | $-$ | $78.0$ | $\mathbf{67.7}$ | $\mathbf{71.5}$ | $0.604$
RGB-based methods |  |  |  |
UnOVOST [22] | ✗ | $\checkmark$ | $-$ | $-$ | $66.4$ | $-$ | $-$ | $-$ | $-$
Propose-Reduce [20] | ✗ | $\checkmark$ | $-$ | $-$ | $67.0$ | $-$ | $-$ | $-$ | $-$
OCLR-flow [43] \+ SAM [12] | $\checkmark$ | $\checkmark$ | $57.0$ | $62.0$ | $-$ | $80.6$ | $71.5$ | $79.2$ | $-$
PMN [17] | $\checkmark$ | $\checkmark$ | $-$ | $-$ | $-$ | ${85.6}$ | $-$ | ${77.8}$ | $-$
Xie et al. [44] \+ SAM [12] | $\checkmark$ | $\checkmark$ | $71.1$ | $70.9$ | $-$ | $86.6$ | $\mathbf{81.3}$ | $\mathbf{85.7}$ | $-$
DEVA [5] | ✗ | $\checkmark$ | $-$ | $-$ | $70.4$ | $87.6$ | $-$ | $-$ | $-$
FlowP-SAM+FlowI-SAM (seq) | $\checkmark$ | $\checkmark$ | $\mathbf{74.7}$ | $\mathbf{74.3}$ | $\mathbf{71.0}$ | $\mathbf{87.7}$ | $80.1$ | $82.8$ | $\mathbf{0.647}$

Table 5: Sequence-level comparison on video object segmentation benchmarks.
“${{\dagger}}$” indicates models that are trained without human annotations.
“seq” indicates that our sequence-level predictions with object masks matched
across frames. We adopt FlowP-SAM and FlowI-SAM (ViT-H) to obtain the results
in the last row.

Sequence-Level Performance. For flow-based segmentation, we apply the mask
association technique introduced in Sec. 6 to obtain sequence-level
predictions from per-frame FlowI-SAM results. To ensure a fair comparison, we
additionally finetune the synthetic-trained OCLR [43] model on the real-world
dataset (DAVIS) with groundtruth annotations provided, resulting in “OCLR-
real” results. As shown by the top part of Table 5, FlowI-SAM (seq)
demonstrates superior performance against OCLR-real, benefiting from the
robust prior knowledge in pre-trained SAM.

For RGB-based segmentation, we obtain our sequence-level predictions by FlowI-
SAM+FlowP-SAM. As can be seen from the lower part of Table 5, our method
achieves outstanding performance across both single- and multi-object
benchmarks.

Figure 4: Qualitative comparison of flow-only segmentation methods on DAVIS
(left), YTVOS (middle), and MoCA (right) sequences. Our FlowI-SAM (seq)
successfully identifies moving objects from noisy optical flow background
(e.g., the ducks in the fourth column).  Figure 5: Qualitative comparison of
RGB-based segmentation methods on DAVIS (left), YTVOS (middle), and SegTrackv2
(right). While the previous method (the third row) struggles to disentangle
multiple moving objects (e.g., mixed gold fishes in the second column), our
FlowP-SAM+FlowI-SAM (seq) accurately separates and segments all moving
objects.

###  7.6 Qualitative Visualisations

In this section, example visualisations are provided across multiple datasets,
where we mainly present sequence-level results, owing to their marginal
differences from frame-level predictions. More visualisations across different
evaluation datasets can be found in Appendix 0.E.

Fig. 4 illustrates the segmentation predictions based on only optical flow
inputs. Compared to OCLR-real, our FlowI-SAM accurately identifies and
disentangles the moving objects from the noisy backgrounds (e.g., the person
in the first column and the ducks in the fourth column), as well as extracts
fine structures (e.g., the camouflaged insect in the fifth column) from
optical flow.

Fig. 5 further provides the visualisations of the RGB-based method, where the
prior work (Xie et al. [44] \+ SAM [12]) sometimes fails to (i) identify the
moving objects (e.g., missing leopard in the fifth column); (ii) distinguish
between multiple objects (e.g., entangled object segmentation in the second
and fourth columns), while our FlowI-SAM+FlowP-SAM (seq) incorporates RGB-
based prediction with flow prompts, resulting in the accurate localisation and
segmentation of moving objects.

##  8 Conclusion

In this paper, we focus on moving object segmentation in real-world videos, by
incorporating per-frame SAM with motion information (optical flow) in two
ways: (i) for flow-only segmentation, we introduce FlowI-SAM that directly
takes in optical flow as inputs; (ii) for RGB-based segmentation, we utilise
motion information to generate flow prompts as guidance. Both approaches
deliver state-of-the-art performance in frame-level segmentation across
single- and multi-object benchmarks. Additionally, we develop a frame-wise
association method that amalgamates predictions from FlowI-SAM and FlowP-SAM ,
achieving sequence-level segmentation predictions that outperform all prior
methods on the DAVIS, YTVOS, and MoCA benchmarks.

The major limitation of this work is its extended running time, attributed to
the computationally heavy image encoder in the vanilla SAM. However, our
approach is generally applicable to other prompt-based segmentation models.
With the emergence of more efficient versions of SAM, we anticipate a
significant reduction in inference time.

## Acknowledgements

This research is supported by the UK EPSRC CDT in AIMS (EP/S024050/1), the
Clarendon Scholarship, and the UK EPSRC Programme Grant Visual AI
(EP/T028572/1).

## References

  * [1] Bideau, P., Learned-Miller, E.: It’s moving! a probabilistic model for causal motion segmentation in moving camera videos. In: ECCV (2016)
  * [2] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV (2021)
  * [3] Cen, J., Fang, J., Yang, C., Xie, L., Zhang, X., Shen, W., Tian, Q.: Segment any 3d gaussians. arXiv preprint arXiv:2312.00860 (2023)
  * [4] Chen, T., Zhu, L., Ding, C., Cao, R., Zhang, S., Wang, Y., Li, Z., Sun, L., Mao, P., Zang, Y.: Sam-adapter: Adapting segment anything in underperformed scenes. In: ICCV Workshop (2023)
  * [5] Cheng, H.K., Oh, S.W., Price, B., Schwing, A., Lee, J.Y.: Tracking anything with decoupled video segmentation. In: ICCV (2023)
  * [6] Cheng, H.K., Schwing, A.G.: XMem: Long-term video object segmentation with an atkinson-shiffrin memory model. In: ECCV (2022)
  * [7] Cheng, Y., Li, L., Xu, Y., Li, X., Yang, Z., Wang, W., Yang, Y.: Segment and track anything. arXiv preprint arXiv:2305.06558 (2023)
  * [8] Cho, D., Hong, S., Kang, S., Kim, J.: Key instance selection for unsupervised video object segmentation. arXiv preprint arXiv:1906.07851 (2019)
  * [9] Cho, S., Lee, M., Lee, S., Park, C., Kim, D., Lee, S.: Treating motion as option to reduce motion dependency in unsupervised video object segmentation. In: WACV (2023)
  * [10] Choudhury, S., Karazija, L., Laina, I., Vedaldi, A., Rupprecht, C.: Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion. In: BMVC (2022)
  * [11] Jabri, A., Owens, A., Efros, A.A.: Space-time correspondence as a contrastive random walk. In: NeurIPS (2020)
  * [12] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollar, P., Girshick, R.: Segment anything. In: ICCV (2023)
  * [13] Lai, Z., Lu, E., Xie, W.: Mast: A memory-augmented self-supervised tracker. In: CVPR (2020)
  * [14] Lai, Z., Xie, W.: Self-supervised learning for video correspondence flow. In: BMVC (2019)
  * [15] Lamdouar, H., Xie, W., Zisserman, A.: Segmenting invisible moving objects. In: BMVC (2021)
  * [16] Lamdouar, H., Yang, C., Xie, W., Zisserman, A.: Betrayed by motion: Camouflaged object discovery via motion segmentation. In: ACCV (2020)
  * [17] Lee, M., Cho, S., Lee, S., Park, C., Lee, S.: Unsupervised video object segmentation via prototype memory network. In: WACV (2023)
  * [18] Li, F., Kim, T., Humayun, A., Tsai, D., Rehg, J.M.: Video segmentation by tracking many figure-ground segments. In: ICCV (2013)
  * [19] Li, S., Seybold, B., Vorobyov, A., Fathi, A., Huang, Q., Kuo, C.C.J.: Instance embedding transfer to unsupervised video object segmentation. In: CVPR (2018)
  * [20] Lin, H., Wu, R., Liu, S., Lu, J., Jia, J.: Video instance segmentation with a propose-reduce paradigm. In: ICCV (2021)
  * [21] Lu, X., Wang, W., Ma, C., Shen, J., Shao, L., Porikli, F.: See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In: CVPR (2019)
  * [22] Luiten, J., Zulfikar, I.E., Leibe, B.: Unovost: Unsupervised offline video object segmentation and tracking. In: WACV (2020)
  * [23] Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications (2024)
  * [24] Mahendran, A., Thewlis, J., Vedaldi, A.: Self-supervised segmentation by grouping optical-flow. In: ECCV (2018)
  * [25] Meunier, E., Badoual, A., Bouthemy, P.: Em-driven unsupervised learning for efficient motion segmentation. IEEE TPAMI (2022)
  * [26] Meunier, E., Bouthemy, P.: Unsupervised space-time network for temporally-consistent segmentation of multiple motions. In: CVPR (2023)
  * [27] Miao, B., Bennamoun, M., Gao, Y., Mian, A.: Self-supervised video object segmentation by motion-aware mask propagation. In: ICME (2022)
  * [28] Ochs, P., Malik, J., Brox, T.: Segmentation of moving objects by long term video analysis. IEEE TPAMI (2014)
  * [29] Ochs, P., Brox, T.: Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions. In: ICCV (2011)
  * [30] Oh, S.W., Lee, J.Y., Xu, N., Kim, S.J.: Video object segmentation using space-time memory networks. In: ICCV (2019)
  * [31] Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: CVPR (2016)
  * [32] Ponimatkin, G., Samet, N., Xiao, Y., Du, Y., Marlet, R., Lepetit, V.: A simple and powerful global optimization for unsupervised video object segmentation. In: WACV (2023)
  * [33] Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., Gool, L.V.: The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675 (2017)
  * [34] Ren, S., Luzi, F., Lahrichi, S., Kassaw, K., Collins, L.M., Bradbury, K., Malof, J.M.: Segment anything, from space? In: WACV (2024)
  * [35] Safadoust, S., Güney, F.: Multi-object discovery by low-dimensional object motion. In: ICCV (2023)
  * [36] Tang, L., Xiao, H., Li, B.: Can sam segment anything? when sam meets camouflaged object detection. arXiv preprint arXiv:2304.04709 (2023)
  * [37] Teed, Z., Deng, J.: Raft: Recurrent all-pairs field transforms for optical flow. In: ECCV (2020)
  * [38] Ventura, C., Bellver, M., Girbau, A., Salvador, A., Marques, F., Giro-i Nieto, X.: RVOS: End-to-end recurrent network for video object segmentation. In: CVPR (2019)
  * [39] Vondrick, C., Shrivastava, A., Fathi, A., Guadarrama, S., Murphy, K.: Tracking emerges by colorizing videos. In: ECCV (2018)
  * [40] Wang, X., Jabri, A., Efros, A.A.: Learning correspondence from the cycle-consistency of time. In: CVPR (2019)
  * [41] Wang, X., Misra, I., Zeng, Z., Girdhar, R., Darrell, T.: Videocutler: Surprisingly simple unsupervised video instance segmentation. arXiv preprint arXiv:2308.14710 (2023)
  * [42] Wu, J., Ji, W., Liu, Y., Fu, H., Xu, M., Xu, Y., Jin, Y.: Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.12620 (2023)
  * [43] Xie, J., Xie, W., Zisserman, A.: Segmenting moving objects via an object-centric layered representation. In: NeurIPS (2022)
  * [44] Xie, J., Xie, W., Zisserman, A.: Appearance-based refinement for object-centric motion segmentation. arXiv:2312.11463 (2023)
  * [45] Xiong, Y., Varadarajan, B., Wu, L., Xiang, X., Xiao, F., Zhu, C., Dai, X., Wang, D., Sun, F., Iandola, F., Krishnamoorthi, R., Chandra, V.: Efficientsam: Leveraged masked image pretraining for efficient segment anything. arXiv:2312.00863 (2023)
  * [46] Xu, N., Yang, L., Fan, Y., Yue, D., Liang, Y., Yang, J., Huang, T.: Youtube-vos: A large-scale video object segmentation benchmark. In: ECCV (2018)
  * [47] Yang, C., Lamdouar, H., Lu, E., Zisserman, A., Xie, W.: Self-supervised video object segmentation by motion grouping. In: ICCV (2021)
  * [48] Yang, Y., Lai, B., Soatto, S.: Dystab: Unsupervised object segmentation via dynamic-static bootstrapping. In: CVPR (2021)
  * [49] Yang, Z., Wang, Q., Bertinetto, L., Bai, S., Hu, W., Torr, P.H.: Anchor diffusion for unsupervised video object segmentation. In: ICCV (2019)
  * [50] Yang, Z., Yang, Y.: Decoupling features in hierarchical propagation for video object segmentation. In: NeurIPS (2022)
  * [51] Zhang, C., Han, D., Qiao, Y., Kim, J.U., Bae, S.H., Lee, S., Hong, C.S.: Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289 (2023)
  * [52] Zhang, K., Zhao, Z., Liu, D., Liu, Q., Liu, B.: Deep transport network for unsupervised video object segmentation. In: ICCV (2021)
  * [53] Zhang, X., Gu, C., Zhu, S.: Sam-helps-shadow:when segment anything model meet shadow removal. arXiv preprint arXiv:2306.06113 (2023)
  * [54] Zhao, S., Sheng, Y., Dong, Y., Chang, E.I.C., Xu, Y.: Maskflownet: Asymmetric feature matching with learnable occlusion mask. In: CVPR (2020)
  * [55] Zhao, X., Ding, W., An, Y., Du, Y., Yu, T., Li, M., Tang, M., Wang, J.: Fast segment anything. arXiv preprint arXiv:2306.12156 (2023)
  * [56] Zheng, Z., Zhong, Y., Zhang, L., Ermon, S.: Segment any change. arXiv:2402.01188 (2024)
  * [57] Zhou, T., Wang, S., Zhou, Y., Yao, Y., Li, J., Shao, L.: Motion-attentive transition for zero-shot video object segmentation. In: AAAI (2020)
  * [58] Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718 (2023)

Appendix

This Appendix consists of the following sections: (i) In Appendix 0.A, we
provide implementation details regarding architectural designs and
experimental settings; (ii) In Appendix 0.B, we conduct comprehensive ablation
studies for both frame-level and sequence-level segmentation; (iii) In
Appendix 0.C (combining motion segmentation with motion-guided segmentation),
we discuss the rationale of combining results from FlowI-SAM and FlowP-SAM
into a single prediction with some example cases; (iv) In Appendix 0.D,
additional quantitative comparisons are provided; (v) In Appendix 0.E,
visualisations and failure cases are discussed.

##  Appendix 0.A Implementation Details

In this section, we summarise the experimental settings in our models,
including hyperparameter choices, architecture details, and training datasets,
with the detailed settings for frame-level and sequence-level segmentation
separately discussed. The official code will be released upon acceptance.

###  0.A.1 Frame-Level Segmentation

Training and Evaluation Datasets. For both FlowI-SAM and FlowP-SAM, there are
two major training stages: a synthetic pre-training on the simulated dataset
proposed by [43], followed by finetuning on the real DAVIS sequences. These
two datasets are adopted owing to their predominantly moving objects in their
training sequences. A more detailed summary of the training datasets and
corresponding evaluation benchmarks is listed in Table 6. For evaluation,
apart from the DAVIS validation sequences, we assess the zero-shot performance
on YTVOS18-m, STv2, FBMS, and MoCA datasets. Notably, owing to occasional
multi-object sequences in STv2 and FBMS, we evaluate the model that is trained
on multi-object DAVIS sequences.

Model |  Training datasets | Evaluation datasets
---|---|---
FlowI-SAM | Syn+DAVIS16 | DAVIS16, MoCA
Syn+DAVIS17-m | DAVIS17-m, YTVOS18-m, STv2, FBMS
FlowP-SAM | Syn+DAVIS16 | DAVIS16, MoCA
Syn+DAVIS17 | DAVIS17, YTVOS18-m, STv2, FBMS
Syn+DAVIS17-m | DAVIS17-m
Table 6:  Training datasets and corresponding evaluation benchmarks. Datasets
in italic indicate zero-shot generalisation during evaluation.

Architectural Details. For FlowI-SAM, we preserve the architecture of the
original SAM and directly finetune it with optical flow inputs. For FlowP-SAM,
as detailed in the main text, a new flow prompt generator is introduced, with
the details provided in Algorithm 1.

Algorithm 1 Pseudo Code for Flow Prompt Generator

    # Number of frame gaps g = 4 (for 1,-1,2,-2)
    # Image resolution H = W = 1024; Image channel size C = 3
    # Feature resolution h = w = 64; Feature channel size c = 256

    """ Flow feature encoding """
    f = SAM_image_encoder(F)      # b g C H W (input flow F)
                                  # b g c h w (dense flow features f)
    f = Average(f)                # b c h w, averaging over frame gaps

    """ Point Prompt encoding """
    pp = SAM_prompt_encoder(p)    # b 2 (point coordinates p)
                                  # b c (point prompt token pp)

    """ Query initialisation """
    mos = Embedding(b, c)         # b c (learnable moving object score token mos)
    fp = Embedding(b, 4, c)       # b 4 c (4 learnable flow prompt tokens fp)
    q = Concat(mos, fp, pp)       # b 6 c, concatenating all tokens to form queries q

    """ Flow transformer """
    for i in range(2):            # Two layers
        # A standard transformer decoder layer with query, key, value inputs
        # Feed-forward layer dimension = 512; Number of heads = 8
        q = transformer_decoder_layer(q, f, f)   # b 6 c
    mos, fp, _ = q                # Output tokens

    # Output flow prompt token fp (b 4 c) will be injected into the segmentation module.

    # A three-layer MLP with a sigmoid function as the last activation function
    # Feed-forward layer dimension = 256
    mos_score = MLP(mos)          # b 1 (moving object score estimation mos_score)

Hyperparameter Settings. Regarding the input resolutions, we follow the
default SAM settings to pad and resize the images to $1024\times 1024$. After
the frozen SAM encoder, the resultant dense spatial features are of size
$64\times 64$, with feature dimensions at $256$. During training of FlowI-SAM,
we adopt a loss factor $\lambda_{f}=0.01$ for fIoU loss. For FlowP-SAM, we set
both loss factors ($\lambda_{f}$ for fIoU loss and $\lambda_{f}$ for MOS loss)
to $0.01$.

###  0.A.2 Sequence-Level Association

In this section, we present the code snip used for sequence-level association
(Algorithm 2). We will release the full code upon publication.

Algorithm 2 Pseudo Code for Sequence-Level Association

    def threeway_hungarian(m1, m2, m3):
        ious_23 = iou(m2, m3)
        _, idx_23 = linear_sum_assignment(-ious_23)
        m3_aligned = m3[idx_23]
        ious_13 = iou(m1, m3_aligned)
        ious_12 = iou(m1, m2)
        _, idx_13 = linear_sum_assignment(-ious_13)
        _, idx_12 = linear_sum_assignment(-ious_12)
        return m2[idx_12], (idx_12 == idx_13)

    def temp_consistency(p, m, b1, b2, f1, f2):
        m_aligned, c1 = threeway_hungarian(p, m, b1)
        _, c2 = threeway_hungarian(p, m, b2)
        _, c3 = threeway_hungarian(p, m, f1)
        _, c4 = threeway_hungarian(p, m, f2)
        c = ((c1+c2+c3+c4)/4 >= 0.5)
        return m_aligned * c + p * (1-c)

##  Appendix 0.B Ablation Study

###  0.B.1 Frame-Level Segmentation: FlowI-SAM

Optical Flow Estimation Method. To ensure a fair comparison, we follow prior
works [47, 43] and apply RAFT [37] as the flow estimation method. Table 7
demonstrates how the input flow quality affects the segmentation results,
which also verifies our default choice of RAFT for flow estimation.

Optical flow method |  DAVIS17-m $\mathcal{J}\uparrow$ |  DAVIS16 $\mathcal{J}\uparrow$
---|---|---
MaskFlownet [54] | $61.7$ | $76.1$
RAFT [37] | $65.7$ | $79.1$
Table 7:  Comparison of optical flow methods. The results are predicted by
frame-level FlowI-SAM, where the SAM ViT-H image encoder is adopted to encode
optical flow with frame gaps $\\{1$,-$1$,$2$,-$2\\}$. We adopt RAFT as the
default optical flow estimation method.

###  0.B.2 Frame-Level Segmentation: FlowP-SAM

Comparison with SAM [12]. Since the vanilla SAM is not trained to explicitly
identify moving foreground objects, for a fair comparison, we consider an
alternative setup by specifying the objects with their groundtruth centroids
as point prompt inputs. We apply this setting to both SAM and FlowP-SAM, with
the performance summarised in Table 8.

Notably, the original SAM formulation employs four mask tokens to generate
masks at different semantic levels, including default, sub-parts, parts, and
whole. For object-level segmentation, we examine the performance of the
$1^{\text{st}}$ (default) and $4^{\text{th}}$ (whole) output mask channels.
According to Table 8, while the $4^{\text{th}}$ output channel shows a
superior performance in SAM, we observe an opposite trend for FlowP-SAM, where
the $1^{\text{st}}$ mask token yields better results. Therefore, the
$1^{\text{st}}$ output mask token is adopted as our default setting.

Model |  Output mask token/channel |  DAVIS17-m $\mathcal{J}\uparrow$ |  DAVIS17 $\mathcal{J}\uparrow$ |  DAVIS16 $\mathcal{J}\uparrow$
---|---|---|---|---
SAM [12] | $1^{\text{st}}$ | $46.6$ | $48.3$ | $42.7$
SAM [12] | $4^{\text{th}}$ | $71.4$ | $68.0$ | $73.6$
FlowP-SAM | $4^{\text{th}}$ | $79.5$ | $72.6$ | $82.9$
FlowP-SAM | $1^{\text{st}}$ | $80.0$ | $73.4$ | $85.6$
Table 8: Quantitative comparison when the centroid of each groundtruth object
mask is provided as a point prompt. Both SAM and FlowP-SAM are evaluated
following this setting. The results are shown for frame-level predictions, and
our default FlowP-SAM utilises the $1^{\text{st}}$ mask token.

Comparison with FlowI-SAM \+ SAM [12]. An alternative method for moving object
segmentation is to apply SAM to refine the flow-predicted masks by FlowI-SAM.
However, such a two-stage method prevents the interaction between RGB and flow
information, leading to inferior performance compared to the end-to-end FlowP-
SAM, as can be seen in Table 9.

Method |  DAVIS17-m $\mathcal{J}\uparrow$ |  DAVIS16 $\mathcal{J}\uparrow$
---|---|---
FlowI-SAM \+ SAM [12] | $72.6$ | $82.4$
FlowP-SAM | $78.5$ | $86.1$
Table 9: Comparison with a two-stage method. The results are shown for frame-
level predictions.

Number of Transformer Decoder Layers. We further investigate the effects of
the number of transformer decoder layers in the flow prompt generator (in
FlowP-SAM). As can be observed from Table 10, increasing the layer number from
$2$ (default) to $4$ does not contribute to a noticeable performance change.

Transformer decoder layers |  DAVIS17-m $\mathcal{J}\uparrow$ |  DAVIS17 $\mathcal{J}\uparrow$ |  DAVIS16 $\mathcal{J}\uparrow$
---|---|---|---
$2$ | $78.5$ | $69.9$ | $86.1$
$4$ | $79.2$ | $69.1$ | $85.1$
Table 10:  The number of layers in the transformer decoder of flow prompt
generator in FlowP-SAM. The results are shown for frame-level predictions, and
by default, we adopt $2$ transformer decoder layers.

###  0.B.3 Sequence-Level Mask Association

In this section, we conduct ablation studies on our sequence-level mask
association module. We show comparison against two baselines that constitute
our method: (i) propagating the previous mask only, and (ii) Hungarian-
matching between past and present masks only. We also perform ablations to
show the performance gains by averaging the confidence scores across different
neighbouring frames as compared to picking one neighbouring frame. The results
are shown in Table 11.

We observe that (i) Hungarian matching alone presents a strong baseline for
RGB-based tracking (using FlowP-SAM \+ FlowI-SAM), where frame-level
predictions are largely consistent; (ii) However, in flow-only cases (using
FlowI-SAM) where object identities might disappear, Hungarian matching is
prone to lose track of the object. This issue of losing track can be solved
using our temporal consistency method, where choosing to propagate appropriate
masks helps maintain object permanence. We observe that flow-only method
(using FlowI-SAM) benefits considerably from this. We postulate further that
using more frames may help even further, but this will involve computing new
optical flows, whereas we only currently use those that are already used as
input to FlowI-SAM and/or FlowP-SAM.

| RGB-based | Flow-only
---|---|---
|  DAVIS17 $\mathcal{J}$ $\uparrow$ |  DAVIS17-m $\mathcal{J}$ $\uparrow$
Propagation only | 34.9 | 29.3
Hungarian only | 68.1 | 48.5
Temporal consistency ($-1$) | 69.0 | 48.7
Temporal consistency ($\pm 1$) | 69.4 | 51.7
Temporal consistency ($\pm 1$, $\pm 2$) | 71.0 | 61.5
Table 11: Ablation study on sequence-level mask association. The frame-wise
masks are predicted by FlowP-SAM \+ FlowI-SAM (ViT-H) for RGB-based, and
FlowI-SAM (ViT-H) for flow-only segmentation, with different methods for
sequence-level mask association.

##  Appendix 0.C Combining Motion Segmentation with Motion-Guided Segmentation

In the main paper, we have quantitatively shown that combining the results of
FlowI-SAM and FlowP-SAM simply by layering the latter behind the former yields
better results. Here, we discuss the reason why this is the case, and also
provide visualisation examples.

We observe that one of the main failure modes of FlowP-SAM (and other RGB-
based methods) is when the model completely fails to identify the object due
to poor appearance, such as occlusion, camouflage, motion blur, small object
size, or bad lighting conditions. In these cases, FlowI-SAM (and other flow-
only methods) shine as they are agnostic to appearance and objectness.

In such cases, layering the motion segmentation masks behind RGB-based
segmentation masks is a very simple and sensible solution. Specifically, we
concatenate the FlowI-SAM predictions behind that of FlowP-SAM, followed by
removing any overlaps. In this way, the regions predicted by both models will
always belong to FlowP-SAM.

We show the effectiveness of this method in Figure 6. In each example case,
the predicted mask from FlowP-SAM missed an object, whereas the predicted mask
from FlowI-SAM grouped all moving objects as a single object. We show that
layering these two masks together allows the prediction from FlowI-SAM to
‘fill in’ the gap that FlowP-SAM failed to predict. Notably, the object
identities are also correctly separated. This is because FlowI-SAM , being
layered behind, does not over-segment regions that are already segmented by
FlowP-SAM.

Figure 6: Combining FlowI-SAM and FlowP-SAM. This example shows combining the
two predictions by layering FlowI-SAM’s prediction behind FlowP-SAM allows
recovery of lost objects undetected by FlowP-SAM due to small object (left),
occlusion (middle), and motion blur (right). Note that both models
individually make wrong predictions.

##  Appendix 0.D Quantitative Results

For multi-object segmentation, we report the performance on both IoU
($\mathcal{J}$) and contour accuracy ($\mathcal{F}$), with Table 12 and Table
13 comparing across frame-level and sequence-level methods, respectively.

|  |  | YTVOS18-m | DAVIS17-m | DAVIS17
---|---|---|---|---|---
|  Flow |  RGB |  $\mathcal{J}\&\mathcal{F}$ $\uparrow$ |  $\mathcal{J}$ $\uparrow$ |  $\mathcal{F}$ $\uparrow$ |  $\mathcal{J}\&\mathcal{F}$ $\uparrow$ |  $\mathcal{J}$ $\uparrow$ |  $\mathcal{F}$ $\uparrow$ |  $\mathcal{J}\&\mathcal{F}$ $\uparrow$ |  $\mathcal{J}$ $\uparrow$ |  $\mathcal{F}$ $\uparrow$
Flow-only methods |  |  |  |  |  |
†MG [47] | $\checkmark$ | ✗ | $33.3$ | $37.0$ | $29.6$ | $35.8$ | $38.4$ | $33.2$ | $-$ | $-$ | $-$
FlowI-SAM (ViT-B) | $\checkmark$ | ✗ | $53.9$ | $56.7$ | $51.2$ | $62.3$ | $63.2$ | $61.3$ | $-$ | $-$ | $-$
FlowI-SAM (ViT-H) | $\checkmark$ | ✗ | $\mathbf{56.4}$ | $\mathbf{58.6}$ | $\mathbf{54.2}$ | $\mathbf{64.8}$ | $\mathbf{65.7}$ | $\mathbf{63.9}$ | $-$ | $-$ | $-$
RGB-based methods |  |  |  |  |  |
†VideoCutLER [41] | ✗ | $\checkmark$ | $57.0$ | $59.0$ | $55.1$ | $57.3$ | $57.4$ | $57.2$ | $43.6$ | $41.7$ | $45.5$
†Safadoust et al. [35] | ✗ | $\checkmark$ | $-$ | $-$ | $-$ | $59.2$ | $59.3$ | $59.2$ | $-$ | $-$ | $-$
MATNet [57] | $\checkmark$ | $\checkmark$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $58.6$ | $56.7$ | $60.4$
FlowP-SAM | $\checkmark$ | $\checkmark$ | $76.7$ | $76.9$ | $76.4$ | $78.9$ | $78.5$ | $79.3$ | $72.7$ | $69.9$ | $75.4$
FlowP-SAM+FlowI-SAM | $\checkmark$ | $\checkmark$ | $\mathbf{77.4}$ | $\mathbf{77.4}$ | $\mathbf{77.5}$ | $\mathbf{80.1}$ | $\mathbf{80.0}$ | $\mathbf{80.2}$ | $\mathbf{74.6}$ | $\mathbf{71.6}$ | $\mathbf{77.6}$

Table 12: Frame-level comparison on multi-object segmentation benchmarks.
“${{\dagger}}$” indicates models that are trained without human annotations.
For the results in the last row, we combine frame-level predictions from
FlowP-SAM and FlowI-SAM (ViT-H).

|  |  | YTVOS18-m | DAVIS17-m | DAVIS17
---|---|---|---|---|---
|  Flow |  RGB |  $\mathcal{J}\&\mathcal{F}$ $\uparrow$ |  $\mathcal{J}$ $\uparrow$ |  $\mathcal{F}$ $\uparrow$ |  $\mathcal{J}\&\mathcal{F}$ $\uparrow$ |  $\mathcal{J}$ $\uparrow$ |  $\mathcal{F}$ $\uparrow$ |  $\mathcal{J}\&\mathcal{F}$ $\uparrow$ |  $\mathcal{J}$ $\uparrow$ |  $\mathcal{F}$ $\uparrow$
Flow-only methods |  |  |  |  |  |
†OCLR [43] | $\checkmark$ | ✗ | $45.3$ | $46.5$ | $44.1$ | $55.1$ | $54.5$ | $55.7$ | $-$ | $-$ | $-$
OCLR-real [43] | $\checkmark$ | ✗ | $47.5$ | $49.5$ | $45.5$ | $56.2$ | $55.7$ | $56.7$ | $-$ | $-$ | $-$
FlowI-SAM (seq, ViT-B) | $\checkmark$ | ✗ | $50.2$ | $51.9$ | $48.4$ | $59.3$ | $60.0$ | $58.6$ | $-$ | $-$ | $-$
FlowI-SAM (seq, ViT-H) | $\checkmark$ | ✗ | $\mathbf{52.1}$ | $\mathbf{53.8}$ | $\mathbf{50.3}$ | $\mathbf{61.0}$ | $\mathbf{61.5}$ | $\mathbf{60.5}$ | $-$ | $-$ | $-$
RGB-based methods |  |  |  |  |  |
UnOVOST [22] | ✗ | $\checkmark$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $67.9$ | $66.4$ | $69.3$
Propose-Reduce [20] | ✗ | $\checkmark$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $70.4$ | $67.0$ | $73.8$
OCLR-flow [43] \+ SAM [12] | $\checkmark$ | $\checkmark$ | $58.5$ | $57.0$ | $60.0$ | $64.2$ | $62.0$ | $66.4$ | $-$ | $-$ | $-$
Xie et al. [44] \+ SAM [12] | $\checkmark$ | $\checkmark$ | $70.6$ | $71.1$ | $70.2$ | $71.5$ | $70.9$ | $72.1$ | $-$ | $-$ | $-$
DEVA [5] | ✗ | $\checkmark$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $73.4$ | $70.4$ | $76.4$
FlowP-SAM+FlowI-SAM (seq) | $\checkmark$ | $\checkmark$ | $\mathbf{75.2}$ | $\mathbf{74.7}$ | $\mathbf{75.7}$ | $\mathbf{75.0}$ | $\mathbf{74.3}$ | $\mathbf{75.6}$ | $\mathbf{73.6}$ | $\mathbf{71.0}$ | $\mathbf{76.1}$

Table 13: Sequence-level comparison on multi-object segmentation benchmarks.
“${{\dagger}}$” indicates models that are trained without human annotations.
“seq” indicates that our sequence-level predictions with object masks matched
across frames. We adopt FlowP-SAM and FlowI-SAM (ViT-H) to obtain the results
in the last row.

##  Appendix 0.E Qualitative Results and Failure Cases

Additional visualisations are provided for our sequence-level segmentation
results on various datasets, including DAVIS17 (Fig. 7), YTVOS18-m (Fig. 8),
MoCA (Fig. 9), STv2 (Fig. 10), and FBMS (Fig. 11). Note that, we adopt FlowI-
SAM (seq) as the flow-only method, and FlowP-SAM \+ FlowI-SAM (seq) for RGB-
based segmentation.

Failure Cases. For FlowI-SAM, one common failure case is related to the
uninformative optical flow inputs. For instance, in the third sequence in Fig.
8 and the second sequence in Fig. 11, (partially) stationary objects are not
captured by the motion fields, therefore leading to missing objects/parts in
the resultant flow-only segmentation.

Another limitation in this work is that the sequence-wise association fails in
some cases, as shown in Fig. 12. Here, the occlusion is long (10 frames), and
the model has lost track of the object. This can indeed be improved by giving
a longer temporal context in the temporal consistency, though our method does
provide a strong baseline.

Figure 7: Qualitative visualisation on DAVIS sequences. The sequence-level
predictions are shown, with FlowI-SAM (seq) and FlowP-SAM \+ FlowI-SAM (seq)
being our flow-only and RGB-based methods, respectively. Our flow-only method
correctly identifies multiple moving objects based on noisy optical flow
inputs (e.g., three pigs in the bottom left), while our RGB-based method
yields more accurate segmentation masks.  Figure 8: Qualitative visualisation
on YTVOS sequences. The last sequence provides a partial motion example, where
flow-only segmentation fails to recover the whole object mask.  Figure 9:
Qualitative visualisation on MoCA sequences. Both flow-only and RGB-based
methods (with flow prompts) are capable of discovering the camouflaged object
by leveraging motion information.  Figure 10: Qualitative visualisation on
STv2 sequences. With the predominant object locomotion (therefore clean
optical flow fields), both flow-only and RGB-based methods yield accurate
segmentation masks.  Figure 11: Qualitative visualisation on FBMS sequences.
As shown in the second sequence (i.e., the giraffes), the flow-only method is
not capable of discovering the occasionally stationary object (the little
giraffe), whereas the RGB-based method identifies both foreground giraffes
correctly.  Figure 12: A failure case. This example sequence is from FBMS. For
a clearer demonstration, the occluder (a horse foot) is labelled with the red
box, and the object ID mis-matching is indicated by different colours (orange
and green) of object masks.