# Betrayed by Attention: A Simple yet Effective Approach for
Self-supervised Video Object Segmentation

Shuangrui Ding1† Rui Qian1† Haohang Xu2 Dahua Lin1,3 Hongkai Xiong2
1The Chinese University of Hong Kong 2Shanghai Jiao Tong University
3Shanghai Artificial Intelligence Laboratory
{dsr1212<EMAIL_ADDRESS>

###### Abstract

In this paper, we propose a simple yet effective approach for self-supervised
video object segmentation (VOS). Our key insight is that the inherent
structural dependencies present in DINO-pretrained Transformers can be
leveraged to establish robust spatio-temporal correspondences in videos.
Furthermore, simple clustering on this correspondence cue is sufficient to
yield competitive segmentation results. Previous self-supervised VOS
techniques majorly resort to auxiliary modalities or utilize iterative slot
attention to assist in object discovery, which restricts their general
applicability and imposes higher computational requirements. To deal with
these challenges, we develop a simplified architecture that capitalizes on the
emerging objectness from DINO-pretrained Transformers, bypassing the need for
additional modalities or slot attention. Specifically, we first introduce a
single spatio-temporal Transformer block to process the frame-wise DINO
features and establish spatio-temporal dependencies in the form of self-
attention. Subsequently, utilizing these attention maps, we implement
hierarchical clustering to generate object segmentation masks. To train the
spatio-temporal block in a fully self-supervised manner, we employ semantic
and dynamic motion consistency coupled with entropy normalization. Our method
demonstrates state-of-the-art performance across multiple unsupervised VOS
benchmarks and particularly excels in complex real-world multi-object video
segmentation tasks such as DAVIS-17-Unsupervised and YouTube-VIS-19. The code
and model checkpoints will be released at https://github.com/shvdiwnkozbw/SSL-
UVOS.

Figure 1: Attention leaks the object’s position! We visualize the self-
attention maps of different queries (red prompt) from the video sequence (a).
The frame-wise DINO attention maps (b) highlight image regions corresponding
to the queried object. A randomly initialized spatio-temporal Transformer
block on top of DINO produces noisy spatio-temporal attention maps (c) that
coarsely track objects over time. Our method diminishes noise in the learned
spatio-temporal attention maps (d) which derive temporally coherent object
segmentation.

††${\dagger}$ Equal contribution.

##  1 Introduction

Representing the visual scene with objects as the basic elements has been long
acknowledged as a core cognitive capability of an intelligent agent. In the
realm of computer vision, Video Object Segmentation (VOS) tasks require a
model to segment and track specified objects within a video sequence, striving
to emulate this foundational function. This task holds significant importance
in various real-world vision systems, including but not limited to, autonomous
driving [18] and surveillance security [47]. However, traditional methods for
VOS [6, 49] typically follow a fully supervised paradigm and entail
substantial costs for obtaining pixel-level per-frame annotations. This factor
significantly limits their feasibility for large-scale applications. Thus, the
researchers turn their attention to self-supervised learning [21, 8, 51, 52,
10, 11], a more efficient approach that leverages unlabeled data.

Despite its promising potential, self-supervised VOS poses significant
challenges. Firstly, previous methods tend to incorporate additional signals
such as optical flow [68, 37, 66], or depth cues [27, 15], to furnish object-
related clues. However, these auxiliary signals within videos can often be
difficult to extract, thus compromising their availability. Secondly, most
existing models [53, 1, 73] adopt slot attention [39] to parse the frame into
objects. Nevertheless, the requirement for a predefined quantity of learnable
slot queries limits its adaptability to in-the-wild data, especially when the
number of objects is unknown. Additionally, the use of either external
modality or slot attention imposes an extra computational burden on the model.
To address these limitations, we recall the recent self-supervised learning
technique DINO [7, 46], which demonstrates emerging objectness in the
attention maps of pretrained Vision Transformer (ViT) [13]. As depicted in
Fig. 1(b), DINO attention maps encode spatial dependencies between diverse
patches. These dependencies present different patterns for various objects,
providing abundant cues for object segmentation. This phenomenon motivates us
with an intriguing possibility: Can we utilize the object-aware attributes of
DINO to learn robust spatio-temporal dependencies and thereby produce coherent
object segmentations in videos?

In this paper, we delve into the potential of this idea with a simple
architecture. Based on DINO pretrained ViT, we introduce a single spatio-
temporal Transformer block to process frame-wise DINO features and calculate
spatio-temporal attention maps. Remarkably, these attention maps in Fig. 1(c)
showcase the ability to coarsely discriminate and track object parts even with
random initialization. This observation leads us to consider attention maps as
valuable indicators that reveal the positions likely to be associated with the
same object over time. Then, we encourage both semantic and motion alignment
among patches belonging to the same object while distinguishing those from
different objects in a fully self-supervised manner. The training
significantly diminishes noise in the learned spatio-temporal dependencies as
displayed in Fig. 1(d). Building on this insight, in inference, we directly
apply Hierarchical Clustering [26] to these spatio-temporal attention maps to
derive object segmentation masks within a sequence. Interestingly, this naive
clustering method yields unexpectedly competitive results. And the
effectiveness of this approach can be attributed to clustering on the spatio-
temporal attention maps, instead of frame-by-frame feature clustering as in
slot attention. This fundamentally ensures the temporal coherence of produced
segmentations and guarantees the object’s permanence.

In summary, our proposed method offers three distinct advantages: (1) Our
approach solely requires RGB frames and operates entirely under self-
supervision without the need for external modality. This characteristic
renders our method highly accessible and adaptable in real-world scenarios;
(2) The architecture is remarkably simple and systematically efficient. We
only introduce a single learnable Transformer block and harness a parameter-
free clustering algorithm to discover the objects. It facilitates easy
generalization to novel scenes, without the constraints imposed by slot
attention; (3) Our method achieves state-of-the-art results across a bunch of
unsupervised VOS benchmarks, including single object segmentation tasks like
DAVIS-16 [48], SegTrack-v2 [32], FBMS-59 [44], as well as multi-object
segmentation tasks such as MOVi-E [19], DAVIS-17-Unsupervised [50], and
YouTube-VIS-19 [67]. Notably, we surpass the previous state-of-the-art method,
TimeT [54], by 6.6% on YTVIS-19 and by 6.1% on DAVIS-17 in terms of the FG-ARI
metric.

##  2 Related Work

Figure 2: Our architecture overview. Given the video frames and a DINO
pretrained Transformer, we first use a temporal correlator to construct the
spatio-temporal correspondence. We then utilize these attention maps as a
clustering metric and apply hierarchical clustering across all frames to
generate segmentation masks in one shot. During training, for each patch, we
sample a positive/negative set and assign an importance weight based on its
corresponding attention map. We promote alignment within the positive set
while differentiating the representations from the negative set. The final
loss, computed with importance weights, only trains the temporal correlator
with the DINO ViT remaining frozen.

Video Object Segmentation aims to segment objects coherently in a video
sequence [6, 17, 14, 29, 41, 45, 49, 23, 34, 2, 25]. In video object
segmentation (VOS), there are two prevalent protocols for evaluating the
learned models: Semi-supervised VOS and Unsupervised VOS. In semi-supervised
VOS, the algorithm is provided with the object masks in the first frame, and
required to track them in subsequent frames. In contrast, unsupervised VOS
aims to identify and segment salient objects from the background without any
specific reference. In this paper, we focus on the more challenging
unsupervised VOS setting without using any kind of manual annotations in
either training or inference. Recently, there emerges a line of self-
supervised algorithms for unsupervised VOS [63, 68, 35, 59, 66, 69, 9, 27,
71]. CIS [68] facilitates fully unsupervised motion segmentation, discarding
object mask supervision during training. By formulating a min-max game of
mutual information, the generator is motivated to create segmentations that
effectively distinguish foreground objects from the background. AMD [38]
minimizes the warping synthesis error to train appearance and motion pathways
without any supervision. GWM [9] uses a single RGB image as input with optical
flow acting as supervision to highlight moving areas. MG [66] and OCLR [64]
solely leverage optical flow as input to generate object-centric layered
representations, with each layer indicating a potential object. Despite the
impressive performance of state-of-the-art self-supervised VOS methods [53,
54, 1] in discovering single objects, they still face limitations when it
comes to solving multi-object discovery tasks. In contrast, by leveraging the
rich object cues in attention maps, our method can not only pick out the most
salient foreground areas for single object discovery but also discern multiple
objects in real-world scenarios.

Self-supervised Spatio-temporal Correspondence learning usually leverages free
temporal supervision signals in videos to learn representations that
facilitate accurate pixel or object tracking across space and time [30, 29,
58, 60, 24, 4, 22, 65, 7, 36]. Vondrick et al. [58] employ the natural
temporal color coherence to train a colorization model on grayscale videos,
thereby establishing fine-grained correspondence between current and future
frames. CRW [24] presents a self-supervised learning approach for dense
correspondence in raw videos. It uses space-time graph-based random walks and
cycle consistency to implicitly supervise chains of comparisons. Hu et al.
[22] independently learn semantics and temporal correspondence from two
pathways and fuse them at a later stage. UME [33] respectively designs short-
term appearance and long-term semantic consistency to learn generalizable
correspondence. Taking a step further by integrating high-level semantics with
low-level temporal correlation, SMTC [53] develops two-stage slot attention to
establish dense correspondence with more emphasis on foreground objects. In
this work, we explicitly initialize the spatio-temporal correspondence on top
of the DINO pretrained Transformer. This initialization effectively guides the
learning process with a self-supervised pixel-level consistency.

Object-aware DINO Features have shown efficacy in object localization tasks,
particularly in image domain [56, 5, 20, 74, 42, 72, 61]. DINO’s original
paper [7] reveals that the learned representations and attention maps carry
substantial object cues that can be harnessed for object discovery. The
groundbreaking work, LOST [56], utilizes these features from DINO to perform
object segmentation by constructing a graph wherein objects are segmented
using the inverse degrees of nodes. TokenCut [62] employs DINO features for
applying the Normalized Cut algorithm [55], thereby obtaining foreground
segments in an image. Motivated by the successful application of DINO features
in image settings, recent works have expanded their use to video segmentation
tasks [62, 53, 1, 54]. SMTC [53] utilizes the rich semantics and
correspondence cues in the DINO features to provide a reliable reference for
object decomposition in consecutive frames. TimeT [54] introduces a feature
forwarding process to propagate dense DINO features across time and establish
temporal consistency. SOLV [1] applies invariant slot attention on DINO
features to distill object-centric representations and merge similar slots
into candidate objects. Most of the existing works focus on leveraging DINO
feature vectors for optimization and adaptation to videos. While in our work,
we lay more emphasis on the self-attention maps. We extend the frame-wise DINO
attention maps to spatio-temporal domain, which explicitly encode space-time
structural dependencies and guide temporally coherent object discovery in
videos.

##  3 Method

The aim of this paper is to develop a self-supervised approach for video
object segmentation without relying on either other modalities [64] or slot
attention [39]. We begin with an initial exploration of clustering the frame-
wise attention maps and features from DINO in Section 3.1, observing that the
attention map is remarkably more effective for VOS task than the features.
Motivated by this, in Section 3.2, we straightforwardly construct the spatio-
temporal correspondence in the form of self-attention, and directly apply
clustering on these spatio-temporal structural dependencies for video
segmentation tasks. Our main pipeline is illustrated in Fig. 2. To train our
proposed module, we propose a dual semantic and motion consistency protocol
coupled with entropy normalization, as detailed in Section 3.3.

Clustering Metric | Cosine similarity | KL divergence
---|---|---
$F$ | $A$ | $A_{v}$ | $\tilde{A}_{v}$
DAVIS-16 | 47.4 | 60.3 | 51.1 | 75.4
DAVIS-17 | 14.7 | 26.7 | 20.9 | 39.2
Table 1: Preliminary video object segmentation results using varying
clustering metrics. $F$ represents features pretrained with DINO. $A$ denotes
per-frame attention maps from DINO, while $A_{v}$ and $\tilde{A}_{v}$ denote
spatio-temporal attention maps produced by a randomly initialized and a final
learned temporal correlator respectively, on top of DINO. The IoU
($\mathcal{J}$ score) is reported for both DAVIS-16 and DAVIS-17-Unsupervised
datasets.

###  3.1 Preliminary on DINO Attention Maps

The self-supervised ViT, as achieved by DINO [7], has exhibited a rich
emergence of objectness in the attention maps. To elucidate this, we visualize
the self-attention map from the final Transformer block of the DINO pretrained
ViT-S/8 as a representative. As illustrated in Fig. 1(b), an image patch
exhibits a high degree of attention dependencies with patches that reside
within the same object. Conversely, image patches of distinct objects tend to
manifest different attention distributions. Encouraged by this observation, we
pose the following question: could it be possible to directly harness these
attention maps to achieve object discovery without relying on annotations?

In response to this question, we conduct a straightforward initial exploration
with these attention maps. We denote the attention map of the last Transformer
Block as $A\in\mathbb{R}^{HW\times HW}$ and its output features
$F\in\mathbb{R}^{HW\times C}$, where $H$, $W$ and $C$ respectively denote the
height, width, and channel dimension of the feature map111Here we omit the CLS
token, and average the multi-head attention along the head dimension.. For an
intuitive comparison, we apply unsupervised clustering on both attention maps
$A$ and features $F$ to yield the object segmentation masks. For the detailed
procedure, please refer to the Sec. 3.2. Then, we report frame-wise evaluation
results on the video object segmentation datasets DAVIS-16 [48] and
DAVIS-17-Unsupervised [50] in Table 1. Interestingly, we observe that the
attention maps significantly excel in decoupling object components compared to
features, exhibiting an improvement of over 10 points on both datasets in
terms of IoU. We surmise this is because the attention maps preserve intact
correspondences, which are more suitable for fine-grained tasks such as object
grouping and discrimination. Therefore, creating spatio-temporal structural
dependencies that expand beyond spatial DINO attention maps appears promising
for video object segmentation tasks.

###  3.2 Our Simple Architecture for VOS

Motivated by the frame-wise preliminary result, we introduce a temporal
correlator instantiated with a spatio-temporal Transformer block [3] on top of
the DINO pretrained ViT backbone, to extend the self-attention calculation to
the spatio-temporal domain. Specifically, given the frame-wise DINO features
$\\{F_{1},...,F_{T}\\}$ of $T$ frames, we process them through a spatio-
temporal Transformer block $\Phi_{\text{st}}$ as follows:

$\displaystyle\\{\tilde{F}_{1},...,\tilde{F}_{T}\\}=\Phi_{\text{st}}({F_{1}+\texttt{pe}_{1},...,F_{T}+\texttt{pe}_{T}}),$
(1)

where pe represents learnable temporal positional embeddings. The resulting
temporally fused features $\tilde{F}_{t}\in\mathbb{R}^{HW\times C}$ maintain
the same dimension as the input. It is important to note that
$\Phi_{\text{st}}$ also produces an intermediate output, the spatio-temporal
attention maps $A_{v}\in\mathbb{R}^{THW\times THW}$, where $A_{v}[i]$
signifies the correlations between the $i$-th position and all spatio-temporal
patches. In this way, we establish spatio-temporal structural dependencies
beyond frame-wise DINO features and have the potential to improve object
permanence across the entire time span.

Since $A_{v}[i]\in\mathbb{R}^{THW}$ is a proximity distribution that describes
the structural dependencies between the $i$-th patch and all other positions,
we adopt symmetric KL divergence to measure the distance
$\mathcal{M}\in\mathbb{R}^{THW\times THW}$ between any two patches as:

$\displaystyle\mathcal{M}[i,j]=\mathcal{D}_{KL}(A_{v}[i]||A[j])+\mathcal{D}_{KL}(A_{v}[j]||A_{v}[i]),$
(2)

where $\mathcal{D}_{KL}$ denotes KL divergence. Based on this distance matrix
$\mathcal{M}$, we then apply Hierarchical Clustering [26] to obtain the object
segmentation masks. Please consult our appendix for an detailed implementation
of the hierarchical clustering algorithm. In this way, we iteratively merge
$THW$ patches into $K$ centroids $A_{c}\in\mathbb{R}^{K\times THW}$. Since
each centroid likely represents the spatial distribution of an object, we
consider the cluster assignments as potential candidates for the object
segmentation mask. Specifically, we calculate the KL divergence distance
between the $A_{v}$ and $A_{c}$ as $D\in\mathbb{R}^{THW\times K}$, and use
argmin operation to produce the cluster assignments $Z$ within the video
sequence:

$\displaystyle Z=\texttt{argmin}(D,\texttt{dim=1})\in\\{1,...,K\\}^{THW}.$ (3)

Then, we employ a bipartite matching mechanism, following the standard
evaluation protocol in DAVIS-17 Challenge [50], to match the predicted masks
with ground truth for evaluation.

There are two noteworthy aspects in our clustering stage. First, in the
clustering process, we only set a distance threshold hyper-parameter without
specifying the number of cluster centroids and the hierarchical clustering
algorithm can adaptively generate different numbers of clusters according to
the complexity of the video. Second, our reference to the entire spatio-
temporal distribution enables coherent object segmentation across multiple
frames. As a result, the generated segmentation masks effortlessly track
corresponding objects over time without extra association procedures.

###  3.3 Training

Noticing that even a randomly initialized attention map $A_{v}$ shown in
Figure 1(c), exhibits an initial ability to differentiate parts and track
objects over time, we leverage these maps as indicators for positive and
negative sets sampling. Mathematically, given the attention maps
$A_{v}\in\mathbb{R}^{THW\times THW}$, we first reshape $A_{v}$ into shape
$THW\times T\times HW$ then use top-k operation to retrieve $K_{p}$ position
indexes in each frame with highest attention scores:

$\displaystyle
I_{p}=\texttt{top-k}(\texttt{reshape}(A_{v}),K_{p},\texttt{dim=-1}).$ (4)

In this way, we obtain a position index set $I_{p}\in\mathbb{R}^{THW\times
T\times K_{p}}$, which represents positions potentially associated with the
same object as the $i$-th patch. The reshape operation ensures that we
consistently extract the same number of patches in each frame. This, in turn,
guarantees effective object tracking within consecutive frames and promotes
consistent temporal correspondence learning. Similarly, we perform this
operation to find $K_{n}$ positions in each frame with the lowest attention
scores $I_{n}\in\mathbb{R}^{THW\times T\times K_{n}}$. Each $I_{n}[i]$
represents positions least likely to belong to the same object as the $i$-th
patch. After obtaining the positive and negative index sets, we apply self-
supervised correspondence learning in two aspects: semantic consistency and
dynamic motion consistency.

Semantic consistency. Intutively, we take the temporally fused features
$F_{v}=\\{\tilde{F}_{1},...,\tilde{F}_{T}\\}\in\mathbb{R}^{THW\times C}$ as
the semantic representation. We promote the semantic alignment between the
features of the same object and distinguish features from different objects.
Given the $i$-th patch, we first refer to the positive pair indexes $I_{p}[i]$
in Eq. 4 to gather the features that are likely from the same object. Then we
calculate the cosine distance between the query feature and all positive
feature vectors, and produce the distance matrix
$\mathcal{S}_{p}[i]\in\mathbb{R}^{TK_{p}}$. Similarly, we gather distinct
features using $I_{n}[i]$, and form negative distance matrix
$\mathcal{S}_{n}[i]$. We employ a simple margin loss to enforce higher
consistency between corresponding object areas:

$\displaystyle\mathcal{L}_{\text{s}}[i]=\sum_{j=1}^{TK_{p}}\sum_{k=1}^{TK_{n}}\max(\mathcal{S}_{p}[i,j]-\mathcal{S}_{n}[i,k]+\lambda_{1},0),$
(5)

where $\lambda_{1}$ is the margin hyper-parameter.

Dynamic motion consistency. Regarding dynamic motions, rather than calculating
optical flow, inter-frame feature correlations, or latent cost volumes as
motion representations, we turn to readily accessible attention maps $A_{v}$.
These maps exhibit spatio-temporal correlations of specific patches, serving
as an effective latent representation of temporal dynamics. Therefore, it is
feasible to directly employ these attention maps to bolster motion consistency
between corresponding objects. Similar to the semantic consistency learning,
given the attention map of $i$-th patch $A_{v}[i]$ as query, we also leverage
the positive (negative) pair indexes $I_{p}[i]$ ($I_{n}[i]$) to gather the
attention maps of the corresponding (distinct) objects. Then we calculate the
symmetric KL divergence as in Eq. 2 between the query and all positive
(negative) attention maps, obtaining the distance matrix
$\mathcal{M}_{p}[i]\in\mathbb{R}^{TK_{p}}$ and
$\mathcal{M}_{n}[i]\in\mathbb{R}^{TK_{n}}$. We utilize the same margin loss
for optimization, with $\lambda_{2}$ as the margin hyper-parameter:

$\displaystyle\mathcal{L}_{\text{m}}[i]=\sum_{j=1}^{TK_{p}}\sum_{k=1}^{TK_{n}}\max(\mathcal{M}_{p}[i,j]-\mathcal{M}_{n}[i,k]+\lambda_{2},0).$
(6)

Overall objectives. Besides, we evaluate the importance of each spatio-
temporal patch and lay more emphasis on the informative regions. Formally, we
employ the entropy of the attention maps as a measure of the information
content within each patch:

$\displaystyle e[i]=\sum_{k=1}^{THW}-A_{v}[i,k]\log A_{v}[i,k],$ (7)

with $e[i]$ denoting the entropy of $i$-th patch. A higher entropy value
indicates more ambiguous spatio-temporal correlations, which provide less
information. Hence, we perform softmax normalization on the reversed entropy
to produce the importance weight $w$ for each patch as:

$\displaystyle w[i]=\frac{\exp(-e[i])}{\sum_{j=1}^{THW}\exp(-e[j])}.$ (8)

Then, the overall learning objectives on semantic and motion consistency can
be formulated as the weighted summation over all spatio-temporal patches:

$\displaystyle\mathcal{L}=\sum_{i=1}^{THW}w[i](\mathcal{L}_{\text{s}}[i]+\mathcal{L}_{\text{m}}[i]).$
(9)

Since we only introduce one additional temporal correlator and an entirely
parameter-free clustering strategy, we exclusively tune a single standard
spatio-temporal Transformer block with the pretrained ViT frozen throughout
the training process. This exceptionally simple method not only maintains a
minimal number of learnable parameters but also yields remarkable results in
video object segmentation.

##  4 Experiments

###  4.1 Datasets

For training, we adopt the challenging video dataset YouTube-VIS-19 [67] for
video instance segmentation. YouTube-VIS-19 consists of 2,883 high-resolution
YouTube videos and each video usually contains multiple object instances of
distinct semantics. During the inference, we evaluate our method on two lines
of tasks:

Unsupervised Single Object Segmentation: We utilize DAVIS-16 [48], SegTrack-v2
[32], and FMBS-59 [44] datasets. Following the standard protocol [68], we
merge all individual masks into one unified foreground mask for SegTrack-v2
and FMBS-59, reporting the mean Intersection over Union (mIoU) on the
validation set. Note that the evaluation is in the zero-shot manner, as the
model does not see the whole dataset before testing.

Unsupervised Multiple Object Segmentation: We use the synthetic MOVi-E [19]
dataset and real-world datasets like DAVIS-17-Unsupervised [50] and YouTube-
VIS-19 [67], reporting Foreground Adjusted Rand Index (FG-ARI) and mIoU. For
MOVi-E, we integrate its training set into our training due to significant
distribution shifts. For the rest, we maintain solely training on the YouTube-
VIS-19 set.

###  4.2 Implementation Details

In training, we sample $T=3$ frames with temporal stride $4$ as the input
clip. Each frame is augmented with random crop and color jitter, and resized
to $192\times 384$. We adopt DINO pretrained ViT-S/8 [7] as the frame encoder,
which is then followed by a single spatio-temporal Transformer Encoder block
with 8 heads as temporal correlator $\Phi_{\text{st}}$. We set the number of
sampled positive and negative pairs to $K_{p}=10$ and $K_{n}=50$ in default.
For optimization, we adopt AdamW [40] with a learning rate $1\times 10^{-4}$,
and train the model for a total of 30k iterations with a batch size of 16. In
the inference stage, our model can be flexibly applied to video sequences of
arbitrary lengths. For the task of multi-object segmentation, we set the
distance threshold in hierarchical clustering to 1.0. We follow the standard
evaluation protocol outlined in the DAVIS-17 challenge [50] to match the
predicted masks with the ground truth. In benchmarks for single object
segmentation, where all objects are annotated as a whole, we adjust the
distance threshold to 0.6 in order to merge all foreground objects into a
single cluster.

|  |  | YTVIS-19 | DAVIS-17
---|---|---|---|---
$\mathcal{L}_{\text{s}}$ | $\mathcal{L}_{\text{m}}$ | $w$ | FG-ARI | mIoU | $\mathcal{J}\&\mathcal{F}$ | FG-ARI
- | - | - | 23.4 | 25.5 | 23.1 | 18.4
✓ | ✗ | ✓ | 30.1 | 32.4 | 29.6 | 27.5
✗ | ✓ | ✓ | 39.9 | 46.1 | 38.6 | 37.3
✓ | ✓ | ✓ | 44.3 | 50.1 | 43.9 | 40.1
✓ | ✓ | ✗ | 43.1 | 48.1 | 42.5 | 38.1
Table 2: Ablation on different combinations of semantic consistency loss $\mathcal{L}_{\text{s}}$, motion consistency loss $\mathcal{L}_{\text{m}}$ and entropy weight $w$. The first row represents the results obtained from a randomly initialized spatio-temporal Transformer block. Dynamic motion alignment plays a more fundamental role in our pixel-level segmentation tasks compared to semantic constraints. Besides, entropy normalization indeed refines the training process and yields superior results. Clustering Metric | YTVIS-19 | DAVIS-17
---|---|---
FG-ARI | mIoU | $\mathcal{J}\&\mathcal{F}$ | FG-ARI
$F$ | 22.1 | 23.8 | 16.1 | 19.4
$F_{v}$ | 33.9 | 31.4 | 33.4 | 32.1
$A$ | 30.1 | 34.8 | 29.3 | 30.5
$A_{v}$ | 44.3 | 50.1 | 43.9 | 40.1
Table 3: Ablation on different clustering metrics. Features $F$ and attention
maps $A$ are directly obtained from DINO. $F_{v}$ and $A_{v}$ are generated
from our learned spatio-temporal Transformer block. Either the attention map
$A$ from DINO or our spatio-temporal attention map $A_{v}$ can serve as a more
effective cue for clustering compared to their corresponding feature.

###  4.3 Ablation Study

In the ablation study, we utilize DINO ViT-S/8 as the frozen backbone. We
report FG-ARI and IoU on YouTube-VIS-2019 and $\mathcal{J}\&\mathcal{F}$ on
DAVIS-17-Unsupervised for multi-object segmentation. For additional ablation
studies, please refer to the appendix.

Training objective $\mathcal{L}_{\text{s}}$, $\mathcal{L}_{\text{m}}$, and
$w$. To understand the contribution of each component to the VOS performance,
we individually evaluate each loss item as depicted in Table 2. Compared to
the baseline without training, the inclusion of semantic consistency loss
$\mathcal{L}_{\text{s}}$ with entropy-based weighting mechanism $w$ (second
row) elevates FG-ARI from 23.4 to 30.1 on YouTube-VIS-19. Furthermore, the
motion consistency loss $\mathcal{L}_{\text{m}}$ (third row) yields even more
substantial improvements, resulting in FG-ARI reaching 39.9. This echos with
our preliminary observations that the dynamic motions, as a manifestation of
spatio-temporal dependencies, plays a more significant role than semantics in
fine-grained object segmentation. Combining all components (fourth row)
provides the best results, highlighting the synergy between the components.
Additionally, removing the weighting function $w$ (fifth row) slightly
decreases the performance on both two benchmarks, affirming the necessity of
entropy normalization to prioritize informative regions.

Clustering metrics. To further explore whether our clustering strategy
optimally leverages the learned spatio-temporal properties, we select four
types of clustering metrics for comparison. Specifically, we adopt both
feature vectors and attention maps directly from DINO and those from our
learned spatio-temporal Transformer block. Table 3 reveals two key
observations. First, our learning paradigm consistently outperforms the off-
the-shelf DINO in terms of both features and attention maps, improving all
metrics by over 10 points. This strongly supports our motivation that
establishing spatio-temporal correspondence is crucial for video object
segmentation. Second, utilizing the intermediate attention maps directly for
clustering proves to be more effective than employing the final features. We
speculate this occurs because the attention maps encode fine-grained
structural dependencies between various positions, whereas feature vectors
primarily capture high-level semantics.

|  | YTVIS-19 | DAVIS-17
---|---|---|---
$T$ | Stride | FG-ARI | mIoU | $\mathcal{J}\&\mathcal{F}$ | FG-ARI
3 | 1 | 37.4 | 43.6 | 37.1 | 33.6
3 | 4 | 44.3 | 50.1 | 43.9 | 40.1
3 | 8 | 44.2 | 50.2 | 43.3 | 40.7
5 | 4 | 44.8 | 50.6 | 44.3 | 41.5
Table 4: Ablation on the number of frames $T$ and temporal stride. When
sampling more than 3 frames with a larger stride, the method starts to exhibit
a diminishing effect.

Number of frames $T$ and temporal stride. To investigate whether a wider
temporal receptive field can reinforce temporal coherence and enhance object
segmentations, we experiment by varying the number of input frames and
temporal stride in training. Our results, displayed in Table 4, reveal that
the performance significantly deteriorates when the stride is very small, such
as a temporal stride of 1 in the first row. However, it begins to plateau upon
reaching 4. This is because a small temporal stride leads to very subtle
temporal dynamics in consecutive frames, resulting in a trivial learning task
while a larger temporal stride provides richer dynamics, effectively
encouraging the model to capture temporal consistency and maintain object
permanence. Besides, introducing more frames leads to slight improvement. This
phenomenon suggests that a modest temporal receptive field can provide
sufficient temporal context for video object segmentation.

###  4.4 Comparison with State-of-the-art

Model | RGB | Flow | DAVIS | ST-v2 | FBMS
---|---|---|---|---|---
NLC [16] | ✓ | ✓ | 55.1 | 67.2 | 51.5
CIS [68] | ✓ | ✓ | 59.2 | 45.6 | 36.8
TokenCut [62] | ✓ | ✓ | 64.3 | 60.2 | 59.6
MOD [12] | ✓ | ✓ | 73.9 | 62.2 | 61.3
DyStaB [69] | ✓ | ✓ | 80.0 | 74.2 | 73.2
DeSprite [71] | ✓ | ✓ | 79.1 | 72.1 | 71.8
RCF [37] | ✓ | ✓ | 80.9 | 76.7 | 69.9
SIMO [31] | ✗ | ✓ | 67.8 | 62.0 | -
MG [66] | ✗ | ✓ | 68.3 | 58.6 | 53.1
EM [43] | ✗ | ✓ | 69.3 | 55.5 | 57.8
OCLR [64] | ✗ | ✓ | 72.1 | 67.6 | 65.4
AMD [38] | ✓ | ✗ | 57.8 | 57.0 | 47.5
SMTC [53] | ✓ | ✗ | 71.8 | 69.3 | 68.4
Ours | ✓ | ✗ | 75.4 | 74.8 | 73.3
Table 5: Quantitative results on single object video segmentation. The
tick($\checkmark$) and cross(✗) labels under the RGB and Flow columns indicate
whether a method utilizes the corresponding modality during training or
inference. We compare per frame mean IoU on DAVIS-16, SegTrack-v2 and FBMS-59
without any post-processing (e.g., spectral clustering, test-time adaptation,
CRF [28]).

Unsupervised Single Object Segmentation. We first present the quantitative
results on unsupervised single object discovery in Table 5. Note that all the
compared methods are trained on the target dataset, while our model is only
trianed on YouTube-VIS-19 and directly transferred to these single object
segmentation benchmarks in a _zero-shot_ manner. Despite this, our method
still achieves the best performance among those only using RGB data. Though
SMTC [53] proposes a sophisticated VOS framework based on slot attention, our
method outperforms it by approximately 5 points. The superiority demonstrates
the generalization ability of our approach simply guided by attention. As for
the counterparts that resort to optical flow, some of them achieve very
promising performance on three benchmarks [70, 37, 71]. This is because
optical flow provides strong prior in highlighting the moving areas in videos,
which well aligns with the single object segmentation task. However, optical
flow can be challenging to obtain and may be unreliable, especially in complex
scenes. In contrast, our method does not require any additional data
modalities and can be flexibly extended to various scenarios.

| MOVi-E | YTVIS-19 | DAVIS-17
---|---|---|---
Model | FG-ARI | mIoU | FG-ARI | mIoU | FG-ARI | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$
SAVi [27] | 42.8 | 16.0 | 11.1 | 12.7 | - | - | - | -
STEVE [57] | 50.6 | 26.6 | 20.0 | 20.9 | - | - | - | -
OCLR [64] | - | - | 15.9 | 32.5 | 14.7 | - | 34.6 | -
VideoSAUR [73] | 73.9 | 35.6 | 39.5 | 29.1 | - | - | - | -
SOLV [1] | 80.8 | - | 29.1 | 45.3 | 32.2 | - | 30.2 | -
SMTC [53] | - | - | 31.4 | 38.8 | 33.3 | 40.5 | 36.4 | 44.6
TimeT⋆ [54] | - | - | 37.9 | 40.4 | 35.5 | 40.0 | 35.8 | 44.2
Ours | 83.4 | 40.2 | 44.3 | 50.1 | 40.1 | 43.9 | 39.2 | 48.6
Ours† | 84.4 | 40.7 | 44.5 | 50.1 | 41.6 | 43.7 | 39.4 | 48.0
Table 6: Quantitative results on multiple object video segmentation. For
MOVi-E and YouTube-VIS-19 datasets, we report FG-ARI and mIOU. Besides, we
report Region Similarity $\mathcal{J}$ and Contour Accuracy $\mathcal{F}$ on
DAVIS-17-Unsupervised. $\star$ The original paper of TimeT [54] includes the
IoU of background category, we rerun the evaluation to exclude background
here. $\dagger$ We use DINOv2 ViT-S/14 as our frozen frame encoder.

Unsupervised Multiple Object Segmentation. We also evaluate our method on more
challenging multi-object segmentation, which aims to segment different objects
in a temporally coherent manner. Among the compared methods, SAVi [27] and
OCLR [64] employ optical flow as supervision to discriminate motions of
different objects. STEVE [57], VideoSAUR [73], SOLV [1] and SMTC [53] develop
variants of slot attention with temporal constraints to bind to distinct
instances. TimeT [54] propagates dense DINO features across time to enhance
temporal consistency and applies clustering on learned features to produce
object segmentations. In contrast, our method directly uses spatio-temporal
attention maps for object segmentation. As shown in Table 6, our method
achieves state-of-the-art results on both synthetic (MOVi-E) and realistic
video dataset (YTVIS-19 and DAVIS-17), indicating the promising generalization
ability. Specifically, comparing to the prior arts, we gain an approximately
5-point advantage over both VideoSAUR [73] on YTVIS-19 and SMTC [53] and TimeT
[54] on DAVIS-17 in terms of FG-ARI. The superior results under our simple
architecture reveal the feasibility of utilizing the spatio-temporal
dependencies to mine object cues and derive reliable video object segmentation
results. Besides, we also report the results using DINOv2 ViT-S/14 as the
frame encoder. Despite the larger patch size, our method still achieves
competitive performance. This demonstrates the robustness of our approach with
respect to pretraining models.

Figure 3: Qualitative comparison with DINO baseline on DAVIS-17-Unsupervised.
Different color denotes different clusters.

Qualitative Comparison. To visually demonstrate the effectiveness of our
proposed method, we contrast our segmentation results with those of the DINO
counterpart in Fig. 3. For the DINO baseline, we cluster the patches through
the attention map $A$. As can be observed, our method is able to segment with
significantly less noise compared to DINO. For instance, in the motocross-jump
sequence, our method nearly outlines the entire motorbike and the rider,
whereas DINO misses most of the bike’s body and only identifies its tyre. This
shows that inferring the masks by associating a collection of RGB features and
constructing spatio-temporal correspondences greatly enhances the segmentation
result. Additionally, with the benefit of temporal correspondence, our model
can differentiate between distinct object instances with the same semantics.
Consider the soapbox sequence displayed in the right column; our method
distinguishes two different people - one running behind the soapbox and
another sitting in the soapbox throughout the video. This level of distinction
cannot be obtained from DINO’s spatial attention map, further emphasizing the
effectiveness of introducing rich spatio-temporal relations in video
segmentation tasks.

##  5 Conclusion

In this paper, we introduce a simple yet powerful approach for self-supervised
video object segmentation. Specifically, our proposed architecture includes
only a single spatio-temporal Transformer block that ingests frame-wise DINO
features and constructs the spatio-temporal correspondence through the
attention layer. During training, we adopt a dual self-supervised consistency
objective that encompasses both semantic consistency and dynamic motion
consistency, supplemented with entropy normalization. In inference, we utilize
hierarchical clustering on spatio-temporal attention maps to generate
temporally coherent object segmentation masks. This extremely straightforward
approach nevertheless delivers state-of-the-art results across many VOS
benchmarks, including exceptional performance on challenging real-world multi-
object segmentation tasks such as DAVIS-17-Unsupervised and YouTube-VIS-19.

## References

  * Aydemir et al. [2023] Görkay Aydemir, Weidi Xie, and Fatma Guney.  Self-supervised object-centric learning for videos.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Bao et al. [2018] Linchao Bao, Baoyuan Wu, and Wei Liu.  Cnn in mrf: Video object segmentation via inference in a cnn-based higher-order spatio-temporal mrf.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5977–5986, 2018.
  * Bertasius et al. [2021] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In _ICML_ , page 4, 2021.
  * Bian et al. [2022] Zhangxing Bian, Allan Jabri, Alexei A Efros, and Andrew Owens.  Learning pixel trajectories with multiscale contrastive random walks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6508–6519, 2022.
  * Bielski and Favaro [2022] Adam Bielski and Paolo Favaro.  Move: Unsupervised movable object segmentation and detection.  _Advances in Neural Information Processing Systems_ , 35:33371–33386, 2022.
  * Caelles et al. [2017] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool.  One-shot video object segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 221–230, 2017.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9650–9660, 2021.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Choudhury et al. [2022] Subhabrata Choudhury, Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht.  Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion.  In _British Machine Vision Conference (BMVC)_ , 2022.
  * Ding et al. [2022a] Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu, Qingyi Chen, Jue Wang, and Hongkai Xiong.  Motion-aware contrastive video representation learning via foreground-background merging.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9716–9726, 2022a.
  * Ding et al. [2022b] Shuangrui Ding, Rui Qian, and Hongkai Xiong.  Dual contrastive learning for spatio-temporal representation.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 5649–5658, 2022b.
  * Ding et al. [2022c] Shuangrui Ding, Weidi Xie, Yabo Chen, Rui Qian, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian.  Motion-inductive self-supervised object discovery in videos.  _arXiv preprint arXiv:2210.00221_ , 2022c.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Dutt Jain et al. [2017] Suyog Dutt Jain, Bo Xiong, and Kristen Grauman.  Fusionseg: Learning to combine motion and appearance for fully automatic segmentation of generic objects in videos.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3664–3673, 2017.
  * Elsayed et al. [2022] Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf.  Savi++: Towards end-to-end object-centric learning from real-world videos.  _Advances in Neural Information Processing Systems_ , 35:28940–28954, 2022.
  * Faktor and Irani [2014] Alon Faktor and Michal Irani.  Video segmentation by non-local consensus voting.  In _BMVC_ , page 8, 2014.
  * Fan et al. [2019] Deng-Ping Fan, Wenguan Wang, Ming-Ming Cheng, and Jianbing Shen.  Shifting more attention to video salient object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8554–8564, 2019.
  * Feng et al. [2020] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer.  Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges.  _IEEE Transactions on Intelligent Transportation Systems_ , 22(3):1341–1360, 2020.
  * Greff et al. [2022] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi.  Kubric: a scalable dataset generator.  2022\.
  * Hamilton et al. [2022] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman.  Unsupervised semantic segmentation by distilling feature correspondences.  In _International Conference on Learning Representations_ , 2022.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * Hu et al. [2022] Yingdong Hu, Renhao Wang, Kaifeng Zhang, and Yang Gao.  Semantic-aware fine-grained correspondence.  In _European Conference on Computer Vision_ , pages 97–115. Springer, 2022.
  * Hu et al. [2018] Yuan-Ting Hu, Jia-Bin Huang, and Alexander G Schwing.  Videomatch: Matching based video object segmentation.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 54–70, 2018.
  * Jabri et al. [2020] Allan Jabri, Andrew Owens, and Alexei Efros.  Space-time correspondence as a contrastive random walk.  _Advances in neural information processing systems_ , 33:19545–19560, 2020.
  * Johnander et al. [2019] Joakim Johnander, Martin Danelljan, Emil Brissman, Fahad Shahbaz Khan, and Michael Felsberg.  A generative appearance model for end-to-end video object segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8953–8962, 2019.
  * Johnson [1967] Stephen C Johnson.  Hierarchical clustering schemes.  _Psychometrika_ , 32(3):241–254, 1967.
  * Kipf et al. [2022] Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff.  Conditional object-centric learning from video.  In _International Conference on Learning Representations_ , 2022.
  * Lafferty et al. [2001] John Lafferty, Andrew McCallum, and Fernando CN Pereira.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  2001\.
  * Lai and Xie [2019] Z. Lai and W. Xie.  Self-supervised learning for video correspondence flow.  In _BMVC_ , 2019.
  * Lai et al. [2020] Zihang Lai, Erika Lu, and Weidi Xie.  Mast: A memory-augmented self-supervised tracker.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6479–6488, 2020.
  * Lamdouar et al. [2021] Hala Lamdouar, Weidi Xie, and Andrew Zisserman.  Segmenting invisible moving objects.  In _British Machine Vision Association_ , 2021.
  * Li et al. [2013] Fuxin Li, Taeyoung Kim, Ahmad Humayun, David Tsai, and James M Rehg.  Video segmentation by tracking many figure-ground segments.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2192–2199, 2013.
  * Li et al. [2023] Liulei Li, Wenguan Wang, Tianfei Zhou, Jianwu Li, and Yi Yang.  Unified mask embedding and correspondence learning for self-supervised video segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18706–18716, 2023.
  * Li and Loy [2018] Xiaoxiao Li and Chen Change Loy.  Video object segmentation with joint re-identification and attention-aware mask propagation.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 90–105, 2018.
  * Li et al. [2019a] Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang.  Joint-task self-supervised learning for temporal correspondence.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2019a.
  * Li et al. [2019b] Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang.  Joint-task self-supervised learning for temporal correspondence.  _Advances in Neural Information Processing Systems_ , 32, 2019b.
  * Lian et al. [2023] Long Lian, Zhirong Wu, and Stella X Yu.  Bootstrapping objectness from videos by relaxed common fate and visual grouping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14582–14591, 2023.
  * Liu et al. [2021] Runtao Liu, Zhirong Wu, Stella Yu, and Stephen Lin.  The emergence of objectness: Learning zero-shot segmentation from videos.  In _Advances in Neural Information Processing Systems_ , pages 13137–13152. Curran Associates, Inc., 2021.
  * Locatello et al. [2020] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric learning with slot attention.  _Advances in Neural Information Processing Systems_ , 33:11525–11538, 2020.
  * Loshchilov and Hutter [2018] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2018.
  * Maninis et al. [2018] K-K Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool.  Video object segmentation without temporal information.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(6):1515–1530, 2018.
  * Melas-Kyriazi et al. [2022] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8364–8375, 2022.
  * Meunier et al. [2022] Etienne Meunier, Anaïs Badoual, and Patrick Bouthemy.  Em-driven unsupervised learning for efficient motion segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(4):4462–4473, 2022.
  * Ochs et al. [2013] Peter Ochs, Jitendra Malik, and Thomas Brox.  Segmentation of moving objects by long term video analysis.  _IEEE transactions on pattern analysis and machine intelligence_ , 36(6):1187–1200, 2013.
  * Oh et al. [2019] Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim.  Video object segmentation using space-time memory networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9226–9235, 2019.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Patil et al. [2021] Prashant W Patil, Akshay Dudhane, Ashutosh Kulkarni, Subrahmanyam Murala, Anil Balaji Gonde, and Sunil Gupta.  An unified recurrent video object segmentation framework for various surveillance environments.  _IEEE Transactions on Image Processing_ , 30:7889–7902, 2021.
  * Perazzi et al. [2016] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 724–732, 2016.
  * Perazzi et al. [2017] Federico Perazzi, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, and Alexander Sorkine-Hornung.  Learning video object segmentation from static images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2663–2672, 2017.
  * Pont-Tuset et al. [2017] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  _arXiv preprint arXiv:1704.00675_ , 2017.
  * Qian et al. [2021] Rui Qian, Yuxi Li, Huabin Liu, John See, Shuangrui Ding, Xian Liu, Dian Li, and Weiyao Lin.  Enhancing self-supervised video representation learning via multi-level feature optimization.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 7990–8001, 2021.
  * Qian et al. [2022] Rui Qian, Shuangrui Ding, Xian Liu, and Dahua Lin.  Static and dynamic concepts for self-supervised video representation learning.  In _European Conference on Computer Vision_ , pages 145–164. Springer, 2022.
  * Qian et al. [2023] Rui Qian, Shuangrui Ding, Xian Liu, and Dahua Lin.  Semantics meets temporal correspondence: Self-supervised object-centric learning in videos.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 16675–16687, 2023.
  * Salehi et al. [2023] Mohammadreza Salehi, Efstratios Gavves, Cees GM Snoek, and Yuki M Asano.  Time does tell: Self-supervised time-tuning of dense image representations.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 16536–16547, 2023.
  * Shi and Malik [2000] Jianbo Shi and Jitendra Malik.  Normalized cuts and image segmentation.  _IEEE Transactions on pattern analysis and machine intelligence_ , 22(8):888–905, 2000.
  * Siméoni et al. [2021] Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce.  Localizing objects with self-supervised transformers and no labels.  In _BMVC 2021-32nd British Machine Vision Conference_ , 2021.
  * Singh et al. [2022] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn.  Simple unsupervised object-centric learning for complex and naturalistic videos.  _Advances in Neural Information Processing Systems_ , 35:18181–18196, 2022.
  * Vondrick et al. [2018] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy.  Tracking emerges by colorizing videos.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 391–408, 2018.
  * Wang et al. [2019a] Ning Wang, Yibing Song, Chao Ma, Wengang Zhou, Wei Liu, and Houqiang Li.  Unsupervised deep tracking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1308–1317, 2019a.
  * Wang et al. [2019b] Xiaolong Wang, Allan Jabri, and Alexei A Efros.  Learning correspondence from the cycle-consistency of time.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2566–2576, 2019b.
  * Wang et al. [2023a] Xudong Wang, Rohit Girdhar, Stella X. Yu, and Ishan Misra.  Cut and learn for unsupervised object detection and instance segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3124–3134, 2023a.
  * Wang et al. [2023b] Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz.  Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023b.
  * Xie et al. [2019] Christopher Xie, Yu Xiang, Zaid Harchaoui, and Dieter Fox.  Object discovery in videos as foreground motion clustering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9994–10003, 2019.
  * Xie et al. [2022] Junyu Xie, Weidi Xie, and Andrew Zisserman.  Segmenting moving objects via an object-centric layered representation.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Xu and Wang [2021] Jiarui Xu and Xiaolong Wang.  Rethinking self-supervised correspondence learning: A video frame-level similarity perspective.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10075–10085, 2021.
  * Yang et al. [2021a] Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie.  Self-supervised video object segmentation by motion grouping.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7177–7188, 2021a.
  * Yang et al. [2019a] Linjie Yang, Yuchen Fan, and Ning Xu.  Video instance segmentation.  In _ICCV_ , 2019a.
  * Yang et al. [2019b] Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto.  Unsupervised moving object detection via contextual information separation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019b.
  * Yang et al. [2021b] Yanchao Yang, Brian Lai, and Stefano Soatto.  Dystab: Unsupervised object segmentation via dynamic-static bootstrapping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2826–2836, 2021b.
  * Yang et al. [2021c] Yanchao Yang, Brian Lai, and Stefano Soatto.  Dystab: Unsupervised object segmentation via dynamic-static bootstrapping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2826–2836, 2021c.
  * Ye et al. [2022] Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely.  Deformable sprites for unsupervised video decomposition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2657–2666, 2022.
  * Zadaianchuk et al. [2023a] Andrii Zadaianchuk, Matthaeus Kleindessner, Yi Zhu, Francesco Locatello, and Thomas Brox.  Unsupervised semantic segmentation with self-supervised object-centric representations.  In _The Eleventh International Conference on Learning Representations_ , 2023a.
  * Zadaianchuk et al. [2023b] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius.  Object-centric learning for real-world videos by predicting temporal feature similarities.  In _Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023)_ , 2023b.
  * Ziegler and Asano [2022] Adrian Ziegler and Yuki M Asano.  Self-supervised learning of object parts for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14502–14511, 2022.

Supplementary Material

##  Appendix A More Implementation Details

Dataset. Firstly, we benchmark on three popular datasets designed for single
object segmentation. DAVIS-16 [48] consists of 50 high quality videos, 3455
frames in total. Every frame is annotated with a pixel-level accurate
segmentation mask. SegTrack-v2 [32] contains 14 sequences and 947 fully-
annotated frames. Each sequence involves 1-6 moving objects and presents
challenges including motion blur, appearance change, complex deformation,
occlusion, slow motion, and interacting objects. FBMS-59 [44] has 59 sequences
with greatly varied resolution and annotates every 20th frame. Many sequences
contain multiple moving objects. Following previous evaluation metric [68,
64], we merge objects of SegTrackv2 and FBMS-59 into a single one for video
object segmentation. We calculate the mean per-frame the Jaccard Index
$\mathcal{J}$ over the validation set.

For multi-object segmentation in video, we evaluate our method on one
synthetic and two real-world video datasets. MOVi-E [19] dataset is a
synthetic dataset with granular control over data complexity and comprehensive
ground truth annotations. MOVi-E scenes contain up to 23 objects and introduce
simple linear camera movement. Our evaluation of learned features extends to
real-world datasets DAVIS-17 [50] and YouTube-VIS-19 [67]. DAVIS-17, an
expansion of DAVIS-16, includes 40 additional video sequences along with
multi-object segmentation annotations. We utilize 30 validation videos on
DAVIS-2017 for evaluation. As for YouTube-VIS-19, due to the lack of mask
annotations in the validation or test set, following previous works [1, 73],
we select 300 out of the whole 2,883 videos in training set for evaluation.
For MOVi-E and YouTube-VIS-19, we report the Foreground Adjusted Rand Index
(FG-ARI) and mean Intersection over Union (mIoU). Furthermore, for DAVIS-2017,
we adhere to the standard protocol [50] and report both Region Similarity
($\mathcal{J}$) and Contour Accuracy ($\mathcal{F}$).

Hierarchical Clustering Algorithm. We present our Hierarchical Clustering
based inference in Alg. 1. Given the spatio-temporal attention maps
$A_{v}\in\mathbb{R}^{THW\times THW}$, the algorithm finally outputs the
cluster centers $A_{c}\in\mathbb{R}^{K\times THW}$ and cluster assignments
$Z\in\\{1,...,K\\}^{T^{\prime}HW}$ which serve as predicted object
segmentation masks. Specifically, each attention map is treated as a separate
cluster. The process then cycles through each attention map (or current
’cluster’), calculating distances between it and all other clusters using the
KL-divergence metric. It identifies the clusters that are close to it (i.e.,
those whose distance is less than the threshold) and combines them to form a
new, larger cluster, represented by their updated centroid. This updated
cluster set then replaces the initial set of attention maps, and the process
continues iteratively until no more clusters can be merged. Finally, the
algorithm assigns each original attention map to the cluster whose center it
is closest to, yielding the final cluster assignments. Note that executing
inference on extensive video sequences with a large $T$ value might cause the
self-attention matrix to become redundant, thereby requiring significant
computational resources. To address this limitation, we sparsely sample
$T^{\prime}$ frames ($T^{\prime}<<T$) as key, with the original densely
sampled frames as query, and calculate the cross-attention
$A^{\prime}_{v}\in\mathbb{R}^{THW\times T^{\prime}HW}$. By applying clustering
to more compact $A^{\prime}_{v}$, we linearly reduce memory requirements and
maintain stable performance as shown in Sec. 9.

Algorithm 1 Hierarchical Clustering

Input: Spatio-temporal attention maps $A_{v}\in\mathbb{R}^{THW\times
T^{\prime}HW}$, distance threshold $\tau$

Output: Cluster assignment $Z\in\\{1,...,K\\}^{THW}$

Initialize cluster centers $A_{c}\leftarrow A_{v}$

while the number of clusters in $A_{c}$ changes do

Initialize updated clusters $A_{p}\leftarrow\\{\\}$

for all $x\in A_{c}$ do

Compute distances:
$\mathcal{M}\leftarrow\texttt{calculate\\_distance}(x,A_{c})$

Identify proximal members:
$\mathcal{I}\leftarrow\\{i\mid\mathcal{M}[i]<\tau\\}$

Calculate new cluster centroid:
$x\leftarrow\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}A_{c}[i]$

Add new centroid to updated clusters: $A_{p}\leftarrow A_{p}\cup\\{x\\}$

Remove merged attention maps from current cluster set: $A_{c}\leftarrow
A_{c}\backslash A_{c}[i],\forall i\in\mathcal{I}$

end for

Update the clusters: $A_{c}\leftarrow A_{p}$

end while

Compute final distances:
$\mathcal{M}\leftarrow\texttt{calculate\\_distance}(A_{v},A_{c})$

Compute final cluster assignments:
$Z\leftarrow\texttt{argmin}(\mathcal{M},\texttt{dim=1})$

##  Appendix B More Ablation Study

| YTVIS-19 | DAVIS-17
---|---|---
Model | FG-ARI | mIoU | $\mathcal{J}\&\mathcal{F}$ | FG-ARI
DINO ViT-S/16 | 42.5 | 47.2 | 41.7 | 38.3
DINO ViT-S/8 | 44.3 | 50.1 | 43.9 | 40.1
DINO ViT-B/8 | 43.5 | 50.2 | 42.8 | 40.7
DINOv2 ViT-S/14 | 44.1 | 49.7 | 43.1 | 40.5
DINOv2 ViT-B/14 | 44.5 | 50.1 | 43.7 | 41.6
Table 7: Ablation on different pretrained backbones. We show the results on
various DINO and DINOv2 pretrained ViT encoders with different patch sizes.

Pretrained backbones. We present the ablation studies on different pretrained
backbones in Table. 7. We show the results on both DINO and DINOv2 pretrained
ViT encoders with different patch sizes. Generally, our method achieves
competitive results on all variants of visual encoders. Comparing the first
two lines, i.e., DINO ViT-S/16 vs. DINO ViT-S/8, smaller patch size
contributes to notable performance improvements, approximately 2 points on two
benchmarks, due to more fine-grained segmentation predictions. Comparing DINO
and DINOv2, despite larger patch size, more advanced DINOv2 pretrained
backbones reach comparable performance. This reveals the robustness and
flexibility of our method to different backbones.

Motion representations. In training, we use the spatio-temporal attention maps
as latent motion representations for dynamic motion alignment
$\mathcal{L}_{\text{m}}$. There are some alternatives to formulate this motion
representation. Here we explore two variants in Table 8. The one is simple
global feature correlation, which directly calculates the dot product between
spatio-temporal feature vectors, i.e.,
$C_{g}=\tilde{F}\tilde{F}^{T}\in\mathbb{R}^{THW\times THW}$. The other is
local feature correlation, where we sample a sliding local window with size
$T_{p}\times H_{p}\times W_{p}$ for each grid, and obtain the local
correlation matrix $C_{l}\in\mathbb{R}^{THW\times T_{p}H_{p}W_{p}}$. It is
obvious that using the self-attention maps as motion representation
substantially outperforms the performance of the other two methods. Compared
to the global correlation, the learnable parameters in attention layers
enhance the modeling capacity, which enables the self-attention maps to
capture more comprehensive spatio-temporal dependencies and serve as a better
motion representation. The local correlation only captures temporal dynamics
in limited reception fields, restricting the ability for long-term perception.

| YTVIS-19 | DAVIS-17
---|---|---
MotionRep | FG-ARI | mIoU | $\mathcal{J}\&\mathcal{F}$ | FG-ARI
Local Correlation | 30.5 | 33.8 | 29.6 | 27.7
Global Correlation | 39.1 | 49.2 | 36.4 | 33.9
Self-Attention | 44.3 | 50.1 | 43.9 | 40.1
Table 8: Ablation on different motion representations in calculating motion consistency loss $\mathcal{L}_{\text{m}}$. We compare using the self-attention matrix, simple local and global feature correlation as the latent motion representation. | YTVIS-19 | DAVIS-17 | Speed
---|---|---|---
Ratio | FG-ARI | mIoU | $\mathcal{J}\&\mathcal{F}$ | FG-ARI | Ratio
0.1 | 43.5 | 49.7 | 42.8 | 40.0 |  2.3$\times$
0.2 | 44.1 | 50.0 | 43.4 | 40.2 |  2.0$\times$
0.5 | 44.3 | 50.2 | 43.8 | 40.1 |  1.4$\times$
1.0 | 44.3 | 50.1 | 43.9 | 40.1 |  1.0$\times$
Table 9: Ablation on different numbers of key frames sampled for calculating
the spatio-temporal attention matrix. We compare performance under different
ratios.

Number of key frames. As stated in the above section, it is feasible to
sparsely sample video frames as key to reduce computation costs in inference.
We present the ablation study in Table 9. We report the multiple object
segmentation performance as well as the inference throughput ratio (including
the whole feature extraction, attention calculation and clustering process).
Interestingly, our findings suggest that promising results for video object
segmentation can still be achieved even when only 10% of the frames are
sampled. This sparse sampling approach leads to a remarkable $2.3\times$
speedup in inference. For illustration, given a video with $T$ frames, we
uniformly sample $T^{\prime}=0.1T$ frames as key, with the original $T$ frames
as query, and calculate the cross-attention
$A^{\prime}_{v}\in\mathbb{R}^{THW\times T^{\prime}HW}$. This linearly reduces
the channel dimension of each attention map. Then we perform hierarchical
clustering on these $THW$ samples with reduced channel dimension and produce
the cluster assignments (segmentation masks) for all frames within the video
in one shot. The underlying reason is that video frames are highly redundant,
sparse sampling could provide an abundant temporal reference for spatio-
temporal dependency calculation. Hence, by sampling a small percentage of the
entire video, we achieve comparable performance with a substantial reduction
in computational cost, leading to faster inference speeds.

##  Appendix C More Visualization Results

Results of different clustering hierarchies. In our inference stage, the
hierarchical clustering algorithm produces different clustering hierarchies.
We explore whether there exists an interpretable phenomenon during the
clustering process in Fig. LABEL:hierarchy. Interestingly, we observe that our
model is able to segment objects at different granularities across
hierarchies. Generally, it results in more fine-grained object segmentation in
lower hierarchies and vice versa. For example, in Fig. LABEL:motohuman, the
model discerns two distinct objects - the motorbike and the human - at a lower
hierarchy, subsequently merging them into a cohesive foreground area at a
higher hierarchy. Similarly, Fig. LABEL:parkour shows that the clustering
isolates different sections of the human figure at a lower hierarchy, before
integrating them to form a holistic human body at a higher hierarchy. Such
interpretable hierarchical clustering outcomes yield multi-layered object
segmentations, potentially resolving the ambiguities in annotations.

Results on consecutive frames. We additionally show our segmentation results
on video sequences with object occlusion, disappearance and reappearance,
which is prevalent and challenging in real-world scenarios. In Fig.
LABEL:occlusion, we present three typical cases. The first is a cat-girl
sequence, where there exist mutual occlusions between two objects. Our model
is able to accurately segment the object parts despite severe occlusion. The
second is a kid-football sequence, where the football disappears in the second
frame and reappears in later frames. Since our method refers to the spatio-
temporal dependencies across the whole temporal range, it is able to recognize
that the ball in the first frame and those in later frames belong to the same
instance. This enables our model to process real-world videos with complex
temporal dynamics. The third is a very challenging sequence consisting of two
lizards, which share very similar colors, body shapes and textures and only
vary in sizes and positions. Moreover, the smaller one is severely occluded by
the human hand in the latter three frames. Despite these challenges, our
method is still able to distinguish these two lizards and accurately track
specific instances over time. These examples demonstrate the applicability of
our method to general video scenes.


* Freire, (2019) Freire, O. J. (2019).  David Bohm. A Life Dedicated to Understanding the Quantum World.  Springer.
  * Freire, (2024) Freire, O. J. (2024).  On the vicissitudes of Bohm’s 1952 interpretation of quantum mechanics.  In Oldofredi, A., editor, Guiding Wavesin Quantum Mechanics: 100 Years of de Broglei-Bohm Pilot-Wave Theory. Oxford University Press.
  * Heilbron, (1988) Heilbron, J. (1988).  The Earliest Missionaries of the Copenhagen Spirit.  In Ullmann-Margalit, E., editor, Science in Reflection. Boston Studies in the Pilosophy of Science, volume 110, pages 201–233. Springer, Dordrecht.
  * Heisenberg, (1927) Heisenberg, W. (1927).  Über den anschaulichen Inhalt der quantentheoretischen Kinematik und Mechanik.  Zeitschrift für Physik, 43:172–198.
  * Heisenberg, (1949) Heisenberg, W. (1949).  The Physical Principles of the Quantum Theory.  (Originally Published by Chicago University Press in 1930). Dover Publications.
  * Heisenberg, (1955) Heisenberg, W. (1955).  The development of the interpretation of the quantum theory.  In Pauli, W., editor, Niels Bohr and the Development of Physics: Essays Dedicated to Niels Bohr on the Occasion of his Seventieth Birthday, pages 12–29. McGraw-Hill.
  * Heisenberg, (1958) Heisenberg, W. (1958).  Physics and Beyond. Encounters and Conversations.  Harper and Row.
  * Hilgevoord and Uffink, (2016) Hilgevoord, J. and Uffink, J. (2016).  The Uncertainty Principle.  Stanford Encyclopedia of Philosophy.
  * Howard, (1994) Howard, D. (1994).  What Makes a Classical Concept Classical? Towards a Reconstruction of Niels Bohr’s Philosophy of Physics.  In Faye, J. and Folse, H., editors, Niels Bohr and Contemporary Philosophy, pages 201–229. Dordrecht: Kluwer.
  * Howard, (2004) Howard, D. (2004).  Who invented the “Copenhagen interpretation”? A study in mithology.  Philosophy of Science, 71(5):669–682.
  * Howard, (2012) Howard, D. (2012).  Popper and Bohr on Realism in Quantum Mechanics.  Quanta, 1(1):33–57.
  * Kragh, (1990) Kragh, H. (1990).  Dirac. A Scientific Biography.  Cambridge University Press.
  * Lam and Wütrich, (2020) Lam, V. and Wütrich, C. (2020).  Spacetime foliations from a realist perspective.  Synthese, 199:335–353.
  * Lazarovici et al., (2018) Lazarovici, D., Esfeld, M., and Oldofredi, A. (2018).  Observables and unobservables in quantum mechanics: How the no-hidden-variables theorems support the Bohmian particle ontology.  Entropy, 20(5).
  * Lombardi and Labarca, (2005) Lombardi, O. and Labarca, M. (2005).  The Ontological Autonomy of the Chemical World.  Foundations of Chemistry, 7:125–148.
  * Maudlin, (1995) Maudlin, T. (1995).  Three measurement problems.  Topoi, 14:7–15.
  * Myrvold, (2003) Myrvold, W. (2003).  On some early objections to Bohm’s theory.  Intertional Studies in the Philosophy of Science, 17(1):7–24.
  * Oldofredi and Esfeld, (2019) Oldofredi, A. and Esfeld, M. (2019).  Observability, Unobservability and the Copenhagen Interpretation in Dirac’s Methodology of Physics.  Quanta, 8(1):68–87.
  * Osnaghi et al., (2009) Osnaghi, S., Freitas, F., and Freire, O. J. (2009).  The origin of the Everettian heresy.  Studies in History and Philosophy of Science Part B: Studies in History and Philosophy of Modern Physics, 40:97–123.
  * Passon, (2018) Passon, O. (2018).  On a Common Misconception Regarding the de Broglie-Bohm Theory.  Entropy, 20:440.
  * Peat, (1997) Peat, D. (1997).  Infinite Potential: The Life and Times of David Bohm.  Basic Books.
  * Rescher, (1988) Rescher, N. (1988).  Rationality. A philosophical inquiry into the nature and the rationale of reason.  Oxford University Press.
  * Ruphy, (2011) Ruphy, S. (2011).  From Hacking’s Plurality of Styles of Scientific Reasoning to “Foliated” Pluralism: A Philosophically Robust Form of Ontologico-Methodological Pluralism.  Philosophy of Science, 78(5):1212–1222.
  * Sakurai, (1994) Sakurai, J. J. (1994).  Modern Quantum Mechanics.  Addison-Wesley Publishing Company.
  * Scerri, (2007) Scerri, E. (2007).  The ambiguity of reduction.  HYLE - International Journal for Philosophy of Chemistry, 13(2):67–81.
  * Schrödinger, (1935) Schrödinger, E. (1935).  Die gegenwärtige Situation in der Quantenmechanik.  Naturwissenschaften, 23(48):807–812.
  * Šeselja, (2017) Šeselja, D. (2017).  Scientific Pluralism and Inconsistency Toleration.  Humanamente. Journal of Philosophical Studies, 32:1–29.
  * Struyve, (2010) Struyve, W. (2010).  Pilot-wave approaches to quantum field theory.  Journal of Physics: Conference Series, 306:012047.
  * Tahko, (2018) Tahko, T. (2018).  Fundamentality.  Stanford Encyclopedia of Philosophy.
  * Talbot, (2017) Talbot, C., editor (2017).  David Bohm: Causality and Chance, Letters to Three Women.  Springer.
  * Tumulka, (2022) Tumulka, R. (2022).  Foundations of Quantum Mechanics.  Springer.
  * Valentini, (1991) Valentini, A. (1991).  Signal-locality, uncertainty, and the subquantum H-theorem II.  Physics Letters A, 158(1-2):1–8.
  * (67) van Strien, M. (2020a).  Bohm’s theory of quantum mechanics and the notion of classicality.  Studies in History and Philosophy of Science Part B: Studies in History and Philosophy of Modern Physics, 71:72–86.
  * (68) van Strien, M. (2020b).  Pluralism and anarchism in quantum physics: Paul Feyerabend’s writings on quantum physics in relation to his general philosophy of science.  Studies in History and Philosophy of Science Part A, 80:72–81.
  * van Strien, (2024) van Strien, M. (2024).  Why Bohm was never a determinist.  In Oldofredi, A., editor, Guiding Wavesin Quantum Mechanics: 100 Years of de Broglei-Bohm Pilot-Wave Theory. Oxford University Press.
  * von Meyenn, (1996) von Meyenn, K., editor (1996).  Wolfgang Pauli. Scientific Correspondence with Bohr, Einstein, Heisenberg, a.o. Volume IV, Part I: 1950-1952.  Springer.
  * von Meyenn, (1999) von Meyenn, K., editor (1999).  Wolfgang Pauli. Scientific Correspondence with Bohr, Einstein, Heisenberg, a.o. Volume IV, Part II: 1953-1954. Springer.
  * von Neumann, (1955) von Neumann, J. (1955).  Mathematical Foundations of Quantum Mechanis.  Princeton University Press.