*mainfig/extern/

# SPOT: Self-Training with Patch-Order Permutation for
Object-Centric Learning with Autoregressive Transformers

Ioannis Kakogeorgiou1 Spyros Gidaris2 Konstantinos Karantzalos1 Nikos
Komodakis3,4,5
1National Technical University of Athens 2valeo.ai
3University of Crete 4IACM-Forth 5Archimedes/Athena RC

###### Abstract

Unsupervised object-centric learning aims to decompose scenes into
interpretable object entities, termed slots. Slot-based auto-encoders stand
out as a prominent method for this task. Within them, crucial aspects include
guiding the encoder to generate object-specific slots and ensuring the decoder
utilizes them during reconstruction. This work introduces two novel
techniques, (i) an attention-based self-training approach, which distills
superior slot-based attention masks from the decoder to the encoder, enhancing
object segmentation, and (ii) an innovative patch-order permutation strategy
for autoregressive transformers that strengthens the role of slot vectors in
reconstruction. The effectiveness of these strategies is showcased
experimentally. The combined approach significantly surpasses prior slot-based
autoencoder methods in unsupervised object segmentation, especially with
complex real-world images. We provide the implementation code at
https://github.com/gkakogeorgiou/spot.

##  1 Introduction

Decomposing a scene into separate objects is crucial for AI progress. While
current AI methods often use labeled segmentations or diverse signals like
text, video, motion, or depth, humans can typically achieve scene
decomposition with visual cues alone. Unsupervised object-centric learning,
inspired by human abilities and utilizing abundant unlabeled image data, seeks
to represent a scene as a composition of distinct objects using only visual
information.

Auto-encoding-based frameworks stand at the forefront of object-centric
learning approaches [17, 6, 14, 35, 15, 37, 48, 39, 40]. Here, the emergence
of object-centric representations is enabled due to architectural inductive
biases such as the use of bottleneck modules that force the network to
prioritize the encoding of salient object features. Their simple design,
combined with the ability to operate unsupervised, makes them a standout
choice. One notable advance in this regard involves approaches employing
‘slot’-structured bottlenecks [37, 48, 45]. These auto-encoders, characterized
by their slot-based architecture, consist of two core components. First is the
encoder, responsible for transforming input data into a set of latent vectors
referred to as ‘slots’, each intended to represent an individual object within
an image. The second is the decoder, burdened with the challenging task of
reconstructing the input based on information derived from the extracted
slots, guiding the learning of object-centric representations.

To advance this paradigm, our work introduces SPOT – a dual-stage strategy
designed to elevate object-centric learning so as to more effectively handle
complex real-world images, which is one of the pivotal challenges in this
domain [71]. SPOT aims to refine both components of slot-based auto-encoders,
enhancing both the encoder’s precision in generating object-specific ‘slots’
and the decoder’s ability to utilize these slots during reconstruction. To
that end, it makes two key technical advancements.

Improving slot generation through self-training. The encoding of object-
specific information into the slots is achieved through an iterative attention
mechanism that forces slots to compete over image patches. This competition
leads to the generation of slot-based attention masks that indicate the
association of each image patch with a specific slot [37, 45]. Besides the
encoder, slot-based attention masks are also generated at the decoder side.
For autoregressive transformer decoders [48, 45], these are produced by the
cross-attention module between each output patch and the extracted slots.
Empirically, we observe that masks produced during decoding demonstrate
superior object decomposition, i.e., better object segmentation, compared to
those from encoding [45]. Building on this insight, we propose a self-training
scheme that distills slot-based attention masks from the decoder to the
encoder, thereby enhancing the object segmentation information captured by the
slots.

Enhanced autoregressive decoders with sequence permutations. Object-centric
models commonly utilize weak slot-wise MLP decoders [60]. Drawing inspiration
from large language models [5], Singh et al [48] propose the adoption of more
expressive autoregressive transformer decoders in the context of object-
centric learning [48, 49], surpassing the performance of MLP-based
counterparts. Moreover, multiple studies have underscored the importance of
increasing decoder capacity to effectively apply object-centric learning to
complex scenes [45, 8, 65, 50, 27, 66, 24]. However, autoregressive
transformer models may face overfitting challenges, particularly when
accustomed to teacher-forcing training [64], relying excessively on past
ground-truth tokens. This tendency leads them to neglect slot vectors,
offering weaker and less robust supervisory signals for their learning. To
mitigate this, we propose the introduction of sequence permutations, altering
the autoregressive transformer’s prediction order. This modification amplifies
the role of slot vectors in the reconstruction process, resulting in improved
object-centric representations.

In summary, our contributions are threefold:

  1. 1.

We enhance unsupervised object-centric learning in slot-based autoencoders by
introducing self-training. This involves distilling slot-attention masks from
the initially trained teacher model’s decoder to the slot-attention module of
the student model, thereby improving precision in generating object-specific
slots.

  2. 2.

We amplify the role of slot vectors in the reconstruction process by
introducing sequence permutations that alter the prediction order of the
autoregressive transformer decoder. This modification leads to a more robust
supervisory signal for object-centric learning.

  3. 3.

Empirical evidence demonstrates the synergistic effectiveness of the above
strategies. The combined approach forms the SPOT framework, significantly
outperforming prior slot-based autoencoder methods in unsupervised object
segmentation, particularly with complex real-world images.

##  2 Related work

Unsupervised object-centric learning aims to decompose multi-object scenes
into meaningful object entities. Previous studies utilize auto-encoding
frameworks [17, 6, 14, 35, 15, 37, 48, 39, 40], with slot-attention
bottlenecks [37] emerging as a prominent paradigm. Despite their notability,
early object-centric models struggle with complex (real-world) scenes [71,
28]. To address this, some methods focus on improving precision and stability
of the encoder’s slot-attention module [29, 51, 32, 26, 8], employing bi-level
optimization [8, 26], architectural modifications [29, 32], or regularization
losses [51]. Another line of work focuses on designing a decoder that supports
good decomposition [48, 27, 66], where SLATE [48] utilizes an autoregressive
transformer decoder, and others propose diffusion-based methods [27, 66].
DINOSAUR [45] uses self-supervised pre-trained features as reconstruction
targets, proving more effective in complex scenes. Additionally, some works
extend slot-based auto-encoders to exploit video data [62, 31, 49, 1, 73, 53],
motion [3, 4], depth [13], or text [67]. Our work also aims at improving the
effectiveness of object centric learning in handling complex real-world data
but does so using static images and is thus complementary to the above
approaches.

Furthermore, object-centric learning has been explored through contrastive
frameworks [63, 23, 22, 2, 38], primarily for pre-training representations.

Autoregressive transformer decoders [57] excel in natural language processing
[72, 5, 52] and have recently proven effective in computer vision, exemplified
by models like iGPT [9]. Notably, SLATE [48] and STEVE [49] have applied them
to object-centric learning, highlighting their efficacy in handling intricate
visual scenes. However, despite their success, autoregressive transformers
face training stability challenges in object-centric learning, potentially due
to their high capacity [45]. They tend to overly depend on past ground-truth
tokens, neglecting input from the encoder, as observed in pretraining image
encoders with image caption tasks [54]. In our analysis, we validate this
issue in the context of object-centric learning and propose a simple patch-
order permutation strategy, which is very easy to integrate with existing
autoregressive transformer models.

Self-training approaches refine models using their own predictions on
unlabeled data, enhancing the training set. They are widely used in semi-
supervised learning for tasks like image classification [68, 61], semantic
segmentation [12, 74, 69], and object detection [70, 56, 43], as well as in
unsupervised domain adaptation [75, 41, 36] and unsupervised localization [58,
46, 47]. Additionally, they enhance performance in supervised classification.
This work introduces self-training to the domain of unsupervised object-
centric learning, demonstrating its notable effectiveness in this specific
context.

##  3 Method

Figure 1:  _Enhancing unsupervised object-centric learning via self-training_.
Our two-stage approach starts with exclusive training in the initial stage
(not depicted) using the reconstruction loss $L_{\mathrm{REC}}$. In the
following stage, shown here, a teacher-student framework is applied. The
teacher model, trained in the first stage, guides the student model with an
additional loss $L_{\mathrm{ATT}}$, distilling attention masks
$A_{\mathrm{DEC}}$ from the teacher’s decoder to the slot-attention masks
$A_{\mathrm{SLOT}}$ in the student’s encoder.

Slot-based auto-encoders. Following the common practice in object-centric
learning, we use a slot-based auto-encoding framework [37]. This framework’s
encoder consists of two modules: an image encoder extracting $n$ patch-wise
features from image $X$ and a slot-attention module that groups these features
into $k<n$ latent vectors
$U=(\mathbf{u}_{1};\dots;\mathbf{u}_{k})\in{}^{k\times d_{u}}$ referred to as
‘slots’, each representing an object in the image. The decoder aims to
reconstruct a target signal, such as the original image $X$, from these slot
vectors. Here, following DINOSAUR [45], we employ self-supervised pre-trained
feature encoders (e.g., DINO [7]) for instantiating the image encoder as well
as extracting the reconstruction targets. Defining $Y,\hat{Y}\in{}^{n\times
d_{y}}$ as the target features and the predicted reconstructions,
respectively, the model is trained by minimizing the reconstruction loss:

$L_{\mathrm{REC}}=\frac{1}{n\cdot d_{y}}||Y-\hat{Y}||_{2}^{2}.$ (1)

Defining the reconstruction task with high-level features provides a valuable
training signal for learning object-centric representations from real-world
data [45].

Auto-regressive transformer decoders. A crucial element in slot-based auto-
encoders is the decoder’s design [49, 48, 27, 66]. Here, we employ an
autoregressive transformer [57] as our decoder. Autoregressive transformers
predict the feature $\hat{\mathbf{y}}_{i}$ at position $i$ based on prior
target features $Y_{<i}$ and slots $U$. This prediction is jointly performed
for all token positions using teacher-forcing111While joint prediction is
typically a training-only practice, in our context, is extended to testing due
to the availability of target features.:

$\hat{Y}={\textsc{Decoder}}(Y_{<};U),$ (2)

where $Y_{<}\in{}^{n\times d_{y}}$ is the decoder’s input, consisting of
target features $Y$ right-shifted by one position (excluding the last token),
with a learnable Beginning-Of-Sentence ([BOS]) token prepended:
$Y_{<}=(\mathbf{y}_{{\textsc{[BOS]}}};\mathbf{y}_{1};\dots;\mathbf{y}_{n-1})$.
The decoder is composed of a sequence of transformer blocks [57], each
incorporating a causal self-attention layer (to avoid attending to ‘future’
tokens), a patch-to-slot cross-attention layer allowing to utilize information
from slot vectors $U$, and a feed-forward layer (see Fig. 4).

Objective. The primary objective of slot-based auto-encoders is to decompose
the input image into its individual objects. This is typically evaluated by
examining masks linked to each slot, indicating the association of each image
patch with a specific slot / object. These masks, called slot-attention masks,
can be derived from either the slot-attention module or the employed decoder.

In this work, we present SPOT, a novel two-stage training method that enhances
object-centric learning on real-world data through self-training (Sec. 3.1)
and sequence permutation in the autoregressive decoder (Sec. 3.2).

###  3.1 Self-training via slot-attention distillation

We use the matrix $A\in[0,1]^{n\times k}$ to denote slot attention masks. Each
row of $A$ is a probability distribution in a $k$-dimensional simplex,
indicating the assignment of each image patch to $k$ slots. As previously
outlined, we can derive slot-attention masks from two places:

Slot-attention module.

This module employs an iterative attention-based approach that begins with the
initial query slot vectors $\tilde{U}$ to produce the output slot vectors $U$.
At its core, the module incorporates a modified slot-to-patch cross-attention
layer. The slot attention module’s matrix $A$ is derived from this layer,
specifically from the last iteration’s cross-attention map

$A_{\mathrm{SLOT}}={\textsc{Softmax}}\left(\frac{Q_{E}K_{E}^{\top}}{\sqrt{d_{p}}}\right)^{\top}\in\mathbb{R}^{n\times
k},$ (3)

where $K_{E}\in\mathbb{R}^{n\times d_{p}}$ are keys computed from the patch-
wise features extracted by the image encoder from the input image $X$, and
$Q_{E}\in\mathbb{R}^{k\times d_{p}}$ are queries computed from slot vectors of
the previous iteration. The Softmax is applied along the slots dimension to
enforce competition. The initial slots vectors $\tilde{U}$ are either
independently sampled from a Gaussian distribution [37] or are trainable
parameters [26].

Decoder module.

As mentioned earlier, transformer-based decoders incorporate patch-to-slot
cross-attention layers for leveraging information from slot vectors $U$. Here
the attention masks $A$, denoted as $A_{\mathrm{DEC}}\in{}^{n\times k}$, are
computed as the average (across $H$ heads) of patch-to-slot cross-attention
maps from the final transformer layer:

$A_{\mathrm{DEC}}=\frac{1}{H}\sum_{j=1}^{H}{\textsc{Softmax}}\left(\frac{Q_{j}K_{j}^{\top}}{\sqrt{d_{h}}}\right),$
(4)

where $K_{j}\in{}^{k\times d_{h}}$ are keys computed from the slots $U$, and
$Q_{j}\in{}^{n\times d_{h}}$ are queries computed from the transformed (from
previous layers) decoder input $Y_{<}$. The Softmax is applied along the
dimension of slots.

In our empirical analysis of the attention masks from these two modules (see
Tab. 1 entry (a) in Sec. 4), we note that the decoder masks exhibit superior
performance in grouping patches into object-centric slots. This observation
aligns with the findings reported by Seitzer et al. [45].

Two-stage training with slot-attention distillation. Motivated by this
observation, we propose to improve the slot-attention module with a self-
training scheme depicted in Fig. 1. Our training approach involves two stages.
In the initial stage, the model is trained exclusively using the
$L_{\mathrm{REC}}$ loss. In the second stage, we employ a teacher-student
framework. The pre-trained model serves as the teacher, guiding the training
of a new model (referred to as the student) with an additional loss
$L_{\mathrm{ATT}}$ that distills attention masks $A_{\mathrm{DEC}}$ from the
teacher’s decoder, denoted as $A_{T}$, to the attentions masks
$A_{\mathrm{SLOT}}$ of the student’s encoder, denoted as $A_{S}$.

This distillation enhances the grouping capability of the student’s slot-
attention module, resulting in improved slot representations.

Slot-attention distillation loss $L_{\mathrm{ATT}}$. To distill the teacher’s
attention masks $A_{T}$ to the student, we first convert them from soft to
hard-assignment masks by applying row-wise to $A_{T}$ the argmax and one-hot
operators:

$A^{\prime}_{T}=\texttt{one-hot}(\texttt{argmax}(A_{T}))\in\\{0,1\\}^{n\times
k}.$ (5)

Then, we use Hungarian matching [33] to map the $k$ slots in the teacher’s
masks $A^{\prime}_{T}$ with the $k$ slots in the student’s masks $A_{S}$,
using the IoU between the masks as cost function. The matching process results
in the $A^{\prime\prime}_{T}$ masks. Finally, we apply the cross-entropy loss
between $A^{\prime\prime}_{T}$ and $A_{S}$:

$L_{\mathrm{ATT}}=\frac{1}{n}\langle
A^{\prime\prime}_{T},\log(A_{S})\rangle_{F},$ (6)

where $\langle\cdot,\cdot\rangle_{F}$ denotes the Frobenius inner product.

The total training loss in the second training stage is

$L=L_{\mathrm{REC}}+\lambda L_{\mathrm{ATT}},$ (7)

where $\lambda$ is the loss weight of the distillation objective.

Stabilizing image encoder training with $L_{\mathrm{ATT}}$. We employ self-
supervised pre-trained Vision Transformer [11] (ViT) models as image encoders,
empirically shown to achieve superior object-centric learning results [45].
During the first training stage, it is crucial, as emphasized in [45], to keep
the ViT image encoder frozen for training stability to achieve good results.
Our analysis in Sec. 4.2 reveals an additional advantage of the self-training
loss: serving as a stabilization factor, it facilitates fine-tuning of ViT in
the second training stage, maximizing its learning capacity. We attribute this
stabilizing effect to the self-training loss explicitly guiding the encoder to
generate object-specific slot masks, in contrast to the reconstruction loss
$L_{\mathrm{REC}}$.

###  3.2 Autoregressive transformer decoder with sequence permutations

|  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
(a) Left to right | (b) Top to bottom | (c) Top to bottom | (d) Right to left | (e) Bottom to top | (f) Right to left | (g) Bottom to top | (h) Left to right | (i) Spiral
Top to bottom | Left to right | Right to left | Top to bottom | Right to left | Bottom to top | Left to right | Bottom to top |
Figure 2: _Sequence permutations in SPOT_. The sequence of patches used for autoregressive-based decoder predictions. |  |
---|---|---
(a) Decoder w/o permutations | (b) Decoder with permutations |
grads. of default perm. | grads. of random perm. |
Figure 3:  $L_{1}$ gradients norms for each patch’s reconstruction loss with
respect to the decoder’s input slots (aggregated across all the slots, four
decoder blocks, and the entire COCO validation set). Subplots show gradients
with: (a) default permutation and (b) randomly sampled sequence permutations.

Autoregressive transformer decoders outperform simpler MLPs with spatial-
broadcasting decoders [45], thanks to their effectiveness in ensuring global
consistency in predictions. However, their high representational capacity, as
discussed in [54], may limit the encoder’s effective learning by diminishing
the strength of the supervisory signal backpropagated to it. This is
manifested as follows: initial patch tokens heavily rely on slot vector
information due to limited context from ‘past’ tokens. As decoding progresses,
dependence on features of earlier tokens grows, reducing the significance of
slot vector information. Consequently, later tokens provide a weaker
supervisory signal for learning slot vectors in the encoder. Fig. 3(a)
illustrates this variability in signal strength based on token location. In
this figure, the magnitude of the $L_{\mathrm{REC}}$ loss gradient with
respect to slots (averaged over data samples) is presented, highlighting that
tokens in the first row and column contribute significantly higher magnitude
gradients during backpropagation to slots compared to their later
counterparts.

Autoregressive decoding with permuted sequences. To tackle this issue, we
propose introducing sequence permutations to alter the order in which the
autoregressive transformer predicts, based on a predefined set of
permutations. In essence, employing a sequence permutation requires the
transformer to make predictions not only in the traditional left-to-right and
top-to-bottom patch-order (as depicted in Fig. 2(a)), but also with additional
patch-orderings shown in Fig. 2(b-i). The set of sequence permutations
comprises eight ordering variations, each associated with a distinct starting
patch and direction (horizontal/vertical), complemented by an additional
spiral permutation originating from a patch at the center of the image.

During each training step, we randomly select one sequence permutation from
the set to guide predictions by the autoregressive transformer. This
introduces variability in token positions, as tokens later in the sequence in
the default order may now occupy initial positions in the permuted sequence.
Consequently, these tokens must rely more on slot information, offering
stronger supervisory signal to the encoder after the permutation. This
improvement is evident in Fig. 3(b), where the magnitudes of the gradients
w.r.t. the slots of the model that use sequence permutations in the decoder
are less sensitive w.r.t. the token location they are coming from (as opposed
to the model without permutations Fig. 3(a)). Thus, the model trained with
sequence permutations benefits from a more uniform reliance on the slots
across all token positions.

Figure 4: _Autoregressive (AR) decoding via sequence permutations_. Violet
boxes indicate differences from typical AR decoder.

We illustrate the workflow of our autoregressive decoder with sequence
permutations in Fig. 4. To enable the decoder to recognize the current
sequence ordering and thus what is the next-patch that it must predict, a
different [BOS] token is used for each different permutation. More formally,
we define 9 different sequence permutations
$\sigma_{j}:\\{1,\dots,n\\}\rightarrow\\{1,\dots,n\\}$, $j=1,\dots,9$. For
each permutation $j$, the permuted input to the autoregressive transformer is

$Y_{<\sigma_{j}}=(\mathbf{y}_{{\textsc{[BOS]}}_{j}};\mathbf{y}_{\sigma_{j}(1)};\dots;\mathbf{y}_{\sigma_{j}(n-1)}),$
(8)

where $\mathbf{y}_{{\textsc{[BOS]}}_{j}}$ is the [BOS] token for the $j$
permutation. The decoder produces predictions $\hat{Y}_{\sigma_{j}}$ for this
$j$ permutation

$\hat{Y}_{\sigma_{j}}=(\mathbf{y}_{\sigma_{j}(1)};\dots;\mathbf{y}_{\sigma_{j}(n)})={\textsc{Decoder}}(Y_{<\sigma_{j}};U),$
(9)

which are then re-ordered to $\hat{Y}$ using the inverse permutation
$\sigma_{j}^{-1}$, followed by applying the $L_{\mathrm{REC}}$ loss.

| SP | ST | ENS | Epochs | Decoder | Slot Attention | Max$($Dec, Slot Att$)$
---|---|---|---|---|---|---|---
| mBOi | mBOc | mIoU | FG-ARI | mBOi | mBOc | mIoU | FG-ARI | mBOi | mBOc | mIoU | FG-ARI
(a) |  |  |  | 50 |  32.0$\pm$0.1 |  41.4$\pm$0.3 |  30.0$\pm$0.1 |  32.3$\pm$0.6 |  30.0$\pm$0.2 |  38.9$\pm$0.4 |  28.1$\pm$0.2 |  29.5$\pm$0.6 |  32.0$\pm$0.1 |  41.4$\pm$0.3 |  30.0$\pm$0.1 |  32.3$\pm$0.6
(b) | ✓ |  |  | 50 |  32.7$\pm$0.2 |  40.9$\pm$0.5 |  30.8$\pm$0.1 |  35.6$\pm$0.5 |  31.1$\pm$0.1 |  38.9$\pm$0.3 |  29.4$\pm$0.1 |  33.4$\pm$0.4 |  32.7$\pm$0.2 |  40.9$\pm$0.5 |  30.8$\pm$0.1 |  35.6$\pm$0.5
(c) | ✓ |  | ✓ | 50 |  32.9$\pm$0.2 |  41.2$\pm$0.5 |  31.0$\pm$0.1 |  36.0$\pm$0.5 |  31.1$\pm$0.1 |  38.9$\pm$0.3 |  29.4$\pm$0.1 |  33.4$\pm$0.4 |  32.9$\pm$0.2 |  41.2$\pm$0.5 |  31.0$\pm$0.1 |  36.0$\pm$0.5
(d) |  |  |  | 100 |  32.3$\pm$0.3 |  42.1$\pm$0.2 |  30.2$\pm$0.3 |  31.8$\pm$0.9 |  30.5$\pm$0.2 |  39.8$\pm$0.1 |  28.5$\pm$0.3 |  30.0$\pm$0.7 |  32.3$\pm$0.3 |  42.1$\pm$0.2 |  30.2$\pm$0.3 |  31.8$\pm$0.9
(e) |  | ✓ |  | 50+50 |  30.1$\pm$0.5 |  38.2$\pm$1.0 |  28.3$\pm$0.4 |  22.5$\pm$1.6 |  33.2$\pm$0.1 |  43.6$\pm$0.2 |  31.1$\pm$0.0 |  34.4$\pm$0.5 |  33.2$\pm$0.1 |  43.6$\pm$0.2 |  31.1$\pm$0.0 |  34.4$\pm$0.5
(f) | ✓ | ✓ |  | 50+50 | 34.7$\pm$0.1 | 44.3$\pm$0.3 | 32.7$\pm$0.1 | 36.6$\pm$0.3 | 33.7$\pm$0.1 | 43.1$\pm$0.4 | 31.8$\pm$0.1 | 37.8$\pm$0.5 | 34.7$\pm$0.1 | 44.3$\pm$0.3 | 32.7$\pm$0.1 | 37.8$\pm$0.5
(g) | ✓ | ✓ | ✓ | 50+50 | 35.0$\pm$0.1 | 44.7$\pm$0.3 | 33.0$\pm$0.1 | 37.0$\pm$0.2 | 33.7$\pm$0.1 | 43.1$\pm$0.4 | 31.8$\pm$0.1 | 37.8$\pm$0.5 | 35.0$\pm$0.1 | 44.7$\pm$0.3 | 33.0$\pm$0.1 | 37.8$\pm$0.5
Table 1: _Ablation study on COCO_. Results for slot masks generated by
Decoder, Slot Attention, and their max (Max$($Dec, Slot Att$)$) using mean and
std over 3 seeds. SP: sequence permutation, ST: self-training, ENS: ensebling
of nine permutations at test-time.

Therefore, in our decoder, the sole modifications involve permuting the inputs
and outputs and introducing multiple [BOS] tokens (one per permutation)
instead of a single one. Other than that, the core architecture of the decoder
remains unchanged. During training, a single randomly sampled permutation per
training step is employed, maintaining the training cost. At inference, we
simply use the default permutation (Fig. 2(a)) preserving the same inference
cost. Alternatively, computing attention masks $A_{\mathrm{DEC}}$ for each
permutation and averaging them (i.e., test-time ensembling using the
permutations), as explored in Sec. 4, yields a slight performance improvement.

###  3.3 Incorporating permutations into self-training

We employ sequence permutation to enhance the transformer decoder in both
training stages. This improves the 1st-stage decoder, which thus serves as a
better teacher for the 2nd self-training stage. In the 2nd stage, beyond the
student, we also incorporate random sequence permutations in the teacher’s
decoder when generating the target slot-attention masks for the self-training
loss.

For the 1st-stage training, we use as initial slots independently sampled
vectors from a Gaussian distribution, following [37, 45]. For the 2nd stage,
we initialize slots as learnable vectors employing bi-level optimization [8,
26].

##  4 Experiments

(a) Loss Weight $\lambda$ |  (b) Permutations |  (c) Enc. fine-tuning
---|---|---
0 | 0.002 | 0.005 | 0.01 | 0.02 | Default | Random | ✗ | ✓
30.7$\pm$2.2 |  34.6$\pm$0.3 | 34.7$\pm$0.1 |  34.1$\pm$0.5 |  33.4$\pm$1.5 |  34.6$\pm$0.2 | 34.7$\pm$0.1 |  32.4$\pm$0.2 | 34.7$\pm$0.1

Table 2: _Analysis of self-training hyper-parameters on COCO_. Results are the
mean and standard deviation of the decoder’s mBOi over 3 seeds. Section (a)
studies the impact of the loss weight $\lambda$, section (b) the impact of
using random or the default permutation for generating the target mask $A_{T}$
with the teacher model, and section (c) the impact of fine-tuning the image
encoder during the self-training stage.

###  4.1 Setup

Datasets. We utilized the MS COCO 2017 [34] dataset for its diverse collection
of real-world images, each featuring multiple co-occurring objects. This
dataset poses a significant challenge for object-centric learning models due
to the complexity of the scenes. We also considered the PASCAL VOC 2012 [16]
dataset, which comprises images often containing a single or just a few
salient objects, offering a comparatively more straightforward evaluation.
Furthermore, we used the synthetic datasets MOVi-C and MOVi-E [18], which
contain approximately 1000 realistic 3D-scanned objects. MOVi-C includes
scenes with 3-10 objects, whereas MOVi-E contains scenes with 11-23 objects
per scene. Although MOVi-C/E are originally video-based, we adapt them by
selecting random frames following [45].

Metrics To assess object-centric learning, we use Mean Best Overlap at the
instance (mBOi) and class (mBOc) levels, Foreground Adjusted Rand index (FG-
ARI), and mean Intersection over Union (mIoU). mBOi identifies the best
overlap for each ground truth mask, while mIoU employs Hungarian matching for
a one-to-one correspondence between predicted and ground truth segments. Our
main focus is on mBO and mIoU metrics because they consider background pixels,
comprehensively evaluating how closely masks fit around objects. In contrast,
FG-ARI, a cluster similarity metric, exclusively focuses on foreground pixels,
potentially giving a misleading impression of segmentation quality by ignoring
the fidelity of predicted masks while also promoting over-segmentation. While
we report FG-ARI, it is not our primary focus, aligning with concerns raised
in other studies [66, 14, 28, 44]. We provide a more detailed discussion about
FG-ARI unreliability in the Appendix A.

Implementation details. We employ the Adam optimizer [30] with
$\beta_{1}=0.9$, $\beta_{2}=0.999$, no weight decay, and a batch size of 64.
For each training stage on COCO and PASCAL, we use 50 and 560 training epochs,
respectively. For MOVi-C/E experiments, we use 65 and 30 epochs for the first
and second stages, respectively.

We implement our SPOT models using ViT-B/16 [11] for the encoder (by default
initialized with DINO [7]) and 4 transformer layers in our decoder. Unless
stated otherwise, loss weight $\lambda$ is 0.005. Following [45], on COCO,
PASCAL, MOVi-C, and MOVi-E we use 7, 6, 11, and 24 slots, respectively. All
models are trained on a single GPU with 24 Gbytes. We provide learning rate
schedules and further implementation details in the Appendix C.

###  4.2 Analysis

In this section, we analyze our approach, emphasizing the impact of self-
training and sequence-permutation in the autoregressive decoder, along with
related design choices. Our primary experimentation focuses on the challenging
COCO dataset, complemented by evaluations on MOVi-C.

(A) Sequence-permutation impact. Tab. 1 evaluates the influence of sequence-
permutation with and without self-training. Without self-training, sequence
permutations (Tab. 1 (a)$\rightarrow$(b)) enhances instance-specific metrics
(mBOi, mIoU, FG-ARI) on both the decoder and slot-attention. The class-
specific metric mBOc remains stable (within std).

When self-training is enabled, the significance of sequence permutations
becomes even more evident (Tab. 1 (e)$\rightarrow$(f)). Omitting them results
in a notable performance drop across all metrics, especially impacting
decoder-specific ones (Decoder cols.). Notably, without sequence-permutations,
self-training boosts the slot-attention module (Slot Attention cols.) but
hampers decoder performance (Decoder cols.). This discrepancy highlights the
autoregressive decoder’s susceptibility to neglecting slot input during
reconstruction tasks. An overfitting behavior of this type may be attributed
to the accelerated learning dynamics of slot vectors during training, caused
by the self-training loss $L_{\mathrm{ATT}}$ applied to them, making it
challenging for the autoregressive decoder to effectively leverage them.
Sequence permutations play a crucial role in compelling the decoder to
prioritize slots, underscoring their importance in autoregressive decoders for
slot-centric learning.

Test-time ensebling of permutations. Comparing models (b) to (c) and (f) to
(g) in Tab. 1, we observe a slight performance gain by employing test-time
ensembling of permutations. However, this comes at the expense of increased
inference time. Nonetheless, our method demonstrates very robust performance
even without test-time ensembling, emphasizing that the core influence of
sequence-permutation lies in enhancing the effectiveness and stability of
training slot-based auto-encoders with autoregressive decoders.

Comparison to other AR training approaches. Motivated by CapPa [54], we
explored employing a training procedure that switches between autoregressive
and parallel non-autoregressive decoding222Here, given as input position
embeddings, the transformer predicts at all positions in parallel without
causal masking in self-attention., with the latter being used at 25% of
training steps, as an alternative to sequence permutations for enhancing the
autoregressive decoder. Our findings, outlined in Tab. 3, reveal that, in the
context of object-centric learning, parallel decoding during training does not
help, unlike our sequence permutation approach.

Decoder | mBOi | mIoU | FG-ARI
---|---|---|---
Transformer | 32.0 | 30.0 | 32.3
Transformer w/ PA | 27.8 | 26.5 | 35.3
Transformer w/ SP | 32.7 | 30.8 | 35.6
Table 3: _Autoregressive decoder comparison on COCO_. Evaluation metrics for
slot masks generated by the autoregressive transformer decoder trained
conventionally (Transformer), with sequence permutation during training
(Transformer w/ SP), or parallel prediction for 25% of training iterations
(Transformer w/ PA), as discussed in [54]. No self-training is employed.

Last, the positive impact of sequence permutations is also demonstrated in
MOVi-C (results in the appendix Sec. B.2).

(B) Self-training impact. Comparing entries (f) to (b) in Tab. 1, we see that
self-training yields significant gains. It also helps the max performance
(columns Max$($Dec, Slot Att$)$) in the case of not using sequence
permutation.

Decoder | ST | mBOi | mBOc | mIoU | FG-ARI
---|---|---|---|---|---
MLP | ✗ | 26.7 | 30.3 | 25.6 | 38.7
✓ | 28.4 | 32.4 | 27.0 | 42.5
Table 4: _Self-training with MLP decoder on COCO_. Evaluation metrics for slot masks generated by the decoder. ST: self-training.  Method | COCO | PASCAL | MOVi-C | MOVi-E
---|---|---|---|---
mBOi | mBOc | mBOi | mBOc | mBOi | mIoU | mBOi | mIoU
SA [37]† | 17.2 | 19.2 | 24.6 | 24.9 |  26.2$\pm$1.0 | - |  24.0$\pm$1.2 | -
SLASH [29] | - | - | - | - | - |  27.7$\pm$5.9 | - | -
SLATE [48]† | 29.1 | 33.6 | 35.9 | 41.5 |  39.4$\pm$0.8 |  37.8$\pm$0.7 |  30.2$\pm$1.7 |  28.6$\pm$1.7
CAE [39]† | - | - |  32.9$\pm$0.9 |  37.4$\pm$1.0 | - | - | - | -
DINOSAUR [45] |  32.3$\pm$0.4 |  38.8$\pm$0.4 |  44.0$\pm$1.9 |  51.2$\pm$1.9 | 42.4 | - | - | -
DINOSAUR-MLP [45] |  27.7$\pm$0.2 |  30.9$\pm$0.2 |  39.5$\pm$0.1 |  40.9$\pm$0.1 |  39.1$\pm$0.2 | - |  35.5$\pm$0.2 | -
Rotating Features [40] | - | - |  40.7$\pm$0.1 |  46.0 $\pm$0.1 | - | - | - | -
SlotDiffusion [66] | 31.0 | 35.0 | 50.4 | 55.3 | - | - | 30.2 | 30.2
(Stable-)LSD [27] | 30.4 | - | - | - |  45.6$\pm$0.8 |  44.2$\pm$0.9 |  39.0$\pm$0.5 |  37.6$\pm$0.5
SPOT w/o ENS (ours) | 34.7$\pm$0.1 | 44.3$\pm$0.3 | 48.1$\pm$0.4 | 55.3$\pm$0.4 | 47.0$\pm$1.2 | 46.4$\pm$1.2 | 39.9$\pm$1.1 | 39.0$\pm$1.1
SPOT w/ ENS (ours) | 35.0$\pm$0.1 | 44.7$\pm$0.3 | 48.3$\pm$0.4 | 55.6$\pm$0.4 | 47.3$\pm$1.2 | 46.7$\pm$1.3 | 40.1$\pm$1.2 | 39.3$\pm$1.2
Table 5: _Comparison with object-centric methods on COCO, PASCAL, MOVi-C and
MOVi-E datasets_. SPOT results are the mean and std over 3 seeds. DINOSAUR
uses an autoregressive decoder and DINO [7] ViT encoder (ViT-B/16 for PASCAL
and MOVi-C, ViT-S/8 for COCO). DINOSAUR-MLP uses an MLP decoder and DINO ViT
encoder (ViT-B/16 for COCO and PASCAL, ViT-S/8 for MOVi-C/E). †: COCO and
PASCAL results of SA and SLATE are from [66], MOVi-C/E results are from [45]
for SA and from [27] for SLATE, PASCAL results of CAE are from [40].

Autoregressive and MLP-based decoders. As discussed earlier, when it comes to
autoregressive decoder performance, optimal results are obtained when self-
training is paired with sequence-permutation, which is crucial for preventing
the decoder from ignoring slot information and overfitting. Notably, our self-
distillation scheme also extends beyond autoregressive decoders. To showcase
this, we apply it with MLP-based decoders where it brings significant
performance improvements as demonstrated in Tab. 4.

In Tab. 2, section (a) illustrates the sensitivity of the self-distillation
scheme to the loss weight $\lambda$, with $\lambda$=0.005 producing optimal
results. Performance remains relatively stable for loss weights around this
value.

In section (b) of Tab. 2, we investigate the impact of extracting the
teacher’s slot-mask $A_{T}$ with a standard or random permutation during the
self-distillation scheme. This design choice does not appear to have a
significant impact, with a random permutation being our default choice.

Finally, in section (c) of Tab. 2, we highlight an additional benefit derived
from the self-training stage, particularly in the successful fine-tuning of
the image encoder during that stage. We stress that, without the self-training
loss, allowing the image encoder to be fine-tuned becomes unstable, yielding
subpar results (Tab. 2 (a) for $\lambda=0$ in $L_{\mathrm{ATT}}$). This
observation aligns with Seitzer et al.’s findings [45]. The self-training loss
acts as a stabilizing factor, enabling further utilization of the pre-trained
image encoder. We provide further analysis of the training stability of the
image encoder in appendix Sec. B.3.

###  4.3 Comparison with object-centric methods

|  |  |
---|---|---|---
(a) Ground Truth
|  |  |
(b) SPOT
|  |  |
(c) SPOT w/o self-training
|  |  |
(d) SPOT w/o self-training & w/o sequence permutation
Figure 5: Example results on COCO 2017, using 7 slots.

Method comparison. In Tab. 5, we compare our object-centric learning method
SPOT with prior approaches across the MOVi-C, MOVi-E, PASCAL, and COCO
datasets using mBOi and, when applicable, mBOc metrics. More detailed results,
including the FG-ARI metric are provided in the Appendix B. Our method
outperforms others in all scenarios except PASCAL’s mBOi, where it ranks
second to SlotDiffusion [66]. Notably, on the demanding COCO dataset, our
approach excels, surpassing the prior state-of-the-art by 2.7 and 5.9 points
in mBOi and mBOc metrics, respectively. This underscores its superiority in
unsupervised object-centric learning with real-world data.

Encoder | Method | mBOi | mBOc | mIoU | FG-ARI
---|---|---|---|---|---
DINO | DINOSAUR |  31.6$\pm$0.7 |  39.7$\pm$0.9 | - |  34.1$\pm$1.0
SPOT w/o ENS |  34.7$\pm$0.1 |  44.3$\pm$0.3 |  32.7$\pm$0.1 |  36.6$\pm$0.3
SPOT w/ ENS | 35.0$\pm$0.1 | 44.7$\pm$0.3 | 33.0$\pm$0.1 |  37.0$\pm$0.2
MoCo-v3 | DINOSAUR |  31.4$\pm$0.2 |  38.5$\pm$0.5 | - |  35.2$\pm$0.2
SPOT w/o ENS |  32.7$\pm$0.2 |  41.7$\pm$0.4 |  30.7$\pm$0.2 |  34.4$\pm$0.2
SPOT w/ ENS |  32.9$\pm$0.2 |  42.0$\pm$0.4 |  30.9$\pm$0.2 |  34.8$\pm$0.3
MAE | DINOSAUR |  30.2$\pm$1.8 |  33.2$\pm$1.8 | - |  32.8$\pm$3.7
SPOT w/o ENS |  33.3$\pm$0.3 |  40.7$\pm$0.7 |  31.4$\pm$0.3 |  37.5$\pm$1.0
SPOT w/ ENS |  33.4$\pm$0.3 |  40.9$\pm$0.7 |  31.6$\pm$0.3 | 37.7$\pm$1.0
Table 6: _Evaluation with various pre-trained encoders on COCO_. SPOT results
are the mean and standard deviation over 3 seeds.

Comparing with other pre-trained image features. In our experiments, we used
DINO [7] features for the image encoder and reconstruction targets $Y$. We
also explored MOCO-v3 [10] and MAE [21] as alternatives, comparing SPOT with
DINOSAUR on the COCO dataset. Results in Tab. 6 show SPOT outperforming
DINOSAUR across all examined pre-trained encoders, except with MOCO-v3 and FG-
ARI metric. DINO excels in mBOi, mBOc, and mIoU metrics, while MAE performs
best in the FG-ARI metric.

###  4.4 Qualitative results

In Fig. 5, we present the effects of self-training and sequence permutation.
This combination effectively mitigates over-segmentation issues while
preserving a high degree of detailed segmentation. More examples of the
efficacy of SPOT are provided in LABEL:fig:teaser, showcasing its robust
performance across various images.

##  5 Conclusion

In conclusion, SPOT advances unsupervised object-centric learning for real-
world images by enhancing slot-based auto-encoders through two key strategies.
Firstly, a self-training scheme uses decoder-generated attention masks to
improve slot attention in the encoder. Secondly, a novel patch-order
permutation strategy for autoregressive transformers boosts the decoder’s
performance without additional training cost. The synergistic application of
these strategies enables SPOT to achieve state-of-the-art results in real-
world object-centric learning.

We note that sequence permutation decoding might be more broadly beneficial to
other computer vision tasks employing autoregressive decoders.

#### Acknowledgements

This research work was supported by the Hellenic Foundation for Research and
Innovation (HFRI) and the General Secretariat of Research and Innovation
(GSRI) under the 4th Call for H.F.R.I. Scholarships to PhD Candidates (grant:
11252). It was also partially supported by the RAMONES and iToBos EU Horizon
2020 projects under grants 101017808 and 965221, respectively. NTUA thanks
NVIDIA for the support with the donation of GPU hardware.

## References

  * Aydemir et al. [2023] Görkay Aydemir, Weidi Xie, and Fatma Guney.  Self-supervised object-centric learning for videos.  In _NeurIPS_ , 2023.
  * Baldassarre and Azizpour [2022] Federico Baldassarre and Hossein Azizpour.  Towards self-supervised learning of global and object-centric representations.  In _ICLRW_ , 2022.
  * Bao et al. [2022] Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert.  Discorying object that can move.  In _CVPR_ , 2022.
  * Bao et al. [2023] Zhipeng Bao, Pavel Tokmakov, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert.  Object discovery from motion-guided tokens.  In _CVPR_ , 2023.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In _NeurIPS_ , 2020.
  * Burgess et al. [2019] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner.  Monet: Unsupervised scene decomposition and representation.  _arXiv preprint arXiv:1901.11390_ , 2019.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _ICCV_ , 2021.
  * Chang et al. [2022] Michael Chang, Tom Griffiths, and Sergey Levine.  Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation.  In _NeurIPs_ , 2022.
  * Chen et al. [2020] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In _ICML_ , 2020.
  * Chen et al. [2021] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In _ICCV_ , 2021.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Jakob Uszkoreit, Mostafa Dehghani Neil Houlsby, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * Du et al. [2022] Ye Du, Yujun Shen, Haochen Wang, Jingjing Fei, Wei Li, Liwei Wu, Rui Zhao, Zehua Fu, and Qingjie LIU.  Learning from future: A novel self-training framework for semantic segmentation.  In _NeurIPS_ , 2022.
  * Elsayed et al. [2022] Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf.  SAVi++: Towards end-to-end object-centric learning from real-world videos.  In _NeurIPS_ , 2022.
  * Engelcke et al. [2020] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner.  Genesis: Generative scene inference and sampling with object-centric latent representations.  In _ICLR_ , 2020.
  * Eslami et al. [2016] SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al.  Attend, infer, repeat: Fast scene understanding with generative models.  _NeurIPs_ , 2016.
  * Everingham et al. [2009] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ , 2009.
  * Greff et al. [2019] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner.  Multi-object representation learning with iterative variational inference.  In _ICML_ , 2019.
  * Greff et al. [2022] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi.  Kubric: A scalable dataset generator.  In _CVPR_ , 2022.
  * Hamilton et al. [2022] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman.  Unsupervised semantic segmentation by distilling feature correspondences.  In _ICLR_ , 2022.
  * Hariharan et al. [2011] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik.  Semantic contours from inverse detectors.  In _ICCV_ , 2011.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _CVPR_ , 2022.
  * Hénaff et al. [2021] Olivier J Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira.  Efficient visual pretraining with contrastive detection.  In _ICCV_ , 2021.
  * Hénaff et al. [2022] Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović.  Object discovery and representation networks.  In _ECCV_ , 2022.
  * Jabri et al. [2023] Allan Jabri, Sjoerd van Steenkiste, Emiel Hoogeboom, Mehdi SM Sajjadi, and Thomas Kipf.  Dorsal: Diffusion for object-centric representations of scenes et al.  _arXiv preprint arXiv:2306.08068_ , 2023.
  * Ji et al. [2019] Xu Ji, Joao F. Henriques, and Andrea Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In _ICCV_ , 2019.
  * Jia et al. [2022] Baoxiong Jia, Yu Liu, and Siyuan Huang.  Improving object-centric learning with query optimization.  In _ICLR_ , 2022.
  * Jiang et al. [2023] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn.  Object-centric slot diffusion.  In _NeurIPS_ , 2023.
  * Karazija et al. [2021] Laurynas Karazija, Iro Laina, and Christian Rupprecht.  Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation.  In _NeurIPS Datasets and Benchmarks Track_ , 2021.
  * Kim et al. [2023] Jinwoo Kim, Janghyuk Choi, Ho-Jin Choi, and Seon Joo Kim.  Shepherding slots to objects: Towards stable and robust object-centric learning.  In _CVPR_ , 2023.
  * Kingma and Ba [2015] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _ICLR_ , 2015.
  * Kipf et al. [2022] Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff.  Conditional object-centric learning from video.  In _ICLR_ , 2022.
  * Kori et al. [2023] Avinash Kori, Francesco Locatello, Francesca Toni, and Ben Glocker.  Unsupervised conditional slot attention for object centric learning.  _arXiv preprint arXiv:2307.09437_ , 2023.
  * Kuhn [1955] Harold W Kuhn.  The hungarian method for the assignment problem.  _Naval Research Logistics Quarterly_ , 2(1-2):83–97, 1955.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , 2014.
  * Lin et al. [2020] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn.  Space: Unsupervised object-oriented scene representation via spatial attention and decomposition.  In _ICLR_ , 2020.
  * Liu et al. [2021] Hong Liu, Jianmin Wang, and Mingsheng Long.  Cycle self-training for domain adaptation.  In _NeurIPS_ , 2021.
  * Locatello et al. [2020] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric learning with slot attention.  In _NeurIPS_ , 2020.
  * Löwe et al. [2020] Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf.  Learning object-centric video models by contrasting sets.  _arXiv preprint arXiv:2011.10287_ , 2020.
  * Löwe et al. [2022] Sindy Löwe, Phillip Lippe, Maja Rudolph, and Max Welling.  Complex-valued autoencoders for object discovery.  _Transactions on Machine Learning Research_ , 2022.
  * Löwe et al. [2023] Sindy Löwe, Phillip Lippe, Francesco Locatello, and Max Welling.  Rotating features for object discovery.  _NeurIPs_ , 2023.
  * Mei et al. [2020] Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang.  Instance adaptive self-training for unsupervised domain adaptation.  In _ECCV_ , 2020.
  * Melas-Kyriazi et al. [2022] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization.  In _CVPR_ , 2022.
  * Minderer et al. [2023] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby.  Scaling open-vocabulary object detection, 2023.
  * Monnier et al. [2021] Tom Monnier, Elliot Vincent, Jean Ponce, and Mathieu Aubry.  Unsupervised Layered Image Decomposition into Object Prototypes.  In _ICCV_ , 2021.
  * Seitzer et al. [2023] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello.  Bridging the gap to real-world object-centric learning.  In _ICLR_ , 2023.
  * Siméoni et al. [2021] Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce.  Localizing objects with self-supervised transformers and no labels.  In _BMVC_ , 2021.
  * Siméoni et al. [2023] Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonin Vobecky, Éloi Zablocki, and Patrick Pérez.  Unsupervised object localization: Observing the background to discover objects.  In _CVPR_ , 2023.
  * Singh et al. [2022a] Gautam Singh, Fei Deng, and Sungjin Ahn.  Illiterate dall-e learns to compose.  In _ICLR_ , 2022a.
  * Singh et al. [2022b] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn.  Simple unsupervised object-centric learning for complex and naturalistic videos.  In _NeurIPS_ , 2022b.
  * Singh et al. [2023] Gautam Singh, Yeongbin Kim, and Sungjin Ahn.  Neural systematic binder.  In _ICLR_ , 2023.
  * Stange et al. [2023] Andrew Stange, Robert Lo, Abishek Sridhar, and Kousik Rajesh.  Exploring the role of the bottleneck in slot-based models through covariance regularization.  _arXiv preprint arXiv:2306.02577_ , 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and Brian Fuller.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Traub et al. [2023] Manuel Traub, Sebastian Otte, Tobias Menge, Matthias Karlbauer, Jannik Thuemmel, and Martin V. Butz.  Learning what and where: Disentangling location and identity tracking without supervision.  In _ICLR_ , 2023.
  * Tschannen et al. [2023] Michael Tschannen, Manoj Kumar, Andreas Peter Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer.  Image captioners are scalable vision learners too.  In _NeurIPs_ , 2023.
  * Van Gansbeke et al. [2021] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool.  Unsupervised semantic segmentation by contrasting object mask proposals.  In _ICCV_ , 2021.
  * Vandeghen et al. [2023] Renaud Vandeghen, Gilles Louppe, and Marc Van Droogenbroeck.  Adaptive self-training for object detection.  In _ICCV Workshop_ , 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Wang et al. [2023] Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra.  Cut and learn for unsupervised object detection and instance segmentation.  _arXiv preprint arXiv:2301.11320_ , 2023.
  * Watters et al. [2019a] Nick Watters, Loic Matthey, Chris P. Burgess, and Alexander Lerchner.  Spatial broadcast decoder: A simple architecture for disentangled representations in VAEs.  In _ICLR workshops_ , 2019a.
  * Watters et al. [2019b] Nick Watters, Loic Matthey, Chris P. Burgess, and Alexander Lerchner.  Spatial broadcast decoder: A simple architecture for disentangled representations in VAEs.  In _ICLR workshops_ , 2019b.
  * Wei et al. [2021] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang.  Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning.  In _CVPR_ , 2021.
  * Weis et al. [2021] Marissa A. Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S. Ecker.  Benchmarking unsupervised object representations for video sequences.  _Journal of Machine Learning Research_ , 2021.
  * Wen et al. [2022] Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi.  Self-supervised visual representation learning with semantic grouping.  _NeurIPs_ , 2022.
  * Williams and Zipser [1989] Ronald J Williams and David Zipser.  A learning algorithm for continually running fully recurrent neural networks.  _Neural computation_ , 1(2):270–280, 1989.
  * Wu et al. [2023a] Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg.  Slotformer: Unsupervised visual dynamics simulation with object-centric models.  In _ICLR_ , 2023a.
  * Wu et al. [2023b] Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg.  Slotdiffusion: Object-centric generative modeling with diffusion models.  In _NeurIPS_ , 2023b.
  * Xu et al. [2022] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  In _CVPR_ , 2022.
  * Yalniz et al. [2019] I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan.  Billion-scale semi-supervised learning for image classification.  _arXiv preprint arXiv:1905.00546_ , 2019.
  * Yang et al. [2022] Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao.  St++: Make self-training work better for semi-supervised semantic segmentation.  In _CVPR_ , 2022.
  * Yang et al. [2021] Qize Yang, Xihan Wei, Biao Wang, Xian-Sheng Hua, and Lei Zhang.  Interactive self-training with mean teachers for semi-supervised object detection.  In _CVPR_ , 2021.
  * Yang and Yang [2022] Yafei Yang and Bo Yang.  Promising or elusive? unsupervised object segmentation from real-world single images.  In _NeurIPS_ , 2022.
  * Yang et al. [2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _NeurIPs_ , 2019.
  * Zadaianchuk et al. [2023] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius.  Object-centric learning for real-world videos by predicting temporal feature similarities.  In _NeurIPS_ , 2023.
  * Zhu et al. [2021] Yi Zhu, Zhongyue Zhang, Chongruo Wu, Zhi Zhang, Tong He, Hang Zhang, R Manmatha, Mu Li, and Alexander J Smola.  Improving semantic segmentation via efficient self-training.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Zou et al. [2018] Yang Zou, Zhiding Yu, B. V. K. Vijaya Kumar, and Jinsong Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In _ECCV_ , 2018.

##  Appendix A Discussion about FG-ARI unreliability

|  |  |
---|---|---|---
(a) Ground Truth
|  |  |
(b) SPOT
|  |  |
(c) SPOT w/ MLP-based decoder
Figure 6: Example results on COCO 2017, using 7 slots.  Method | mBOi | mBOc | mIoU | FG-ARI
---|---|---|---|---
MOVi-C
11-block Mask | 19.5 | - | 18.2 | 42.7
SA [37]† |  26.2$\pm$1.0 | - | - |  43.8$\pm$0.3
SLASH [29] | - | - |  27.7$\pm$5.9 |  51.9$\pm$4.0
SLATE [48]† |  39.4$\pm$0.8 | - |  37.8$\pm$0.7 |  49.5$\pm$1.4
DINOSAUR (MLP) [45] |  39.1$\pm$0.2 | - | - | 68.6$\pm$0.4
DINOSAUR [45] | 42.4 | - | - | 55.7
LSD [27] |  45.6$\pm$0.8 | - |  44.2$\pm$0.9 |  52.0$\pm$3.5
SPOT w/o ENS (ours) | 47.0$\pm$1.2 | - | 46.4$\pm$1.2 | 52.1$\pm$3.3/57.9$\pm$2.0
SPOT w/ ENS (ours) | 47.3$\pm$1.2 | - | 46.7$\pm$1.3 | 52.3$\pm$3.3/57.9$\pm$2.0
MOVi-E
24-block Mask | 20.4 | - | 18.8 | 41.9
SA [37]† |  24.0$\pm$1.2 | - | - |  45.0$\pm$1.7
SLATE [48]† |  30.2$\pm$1.7 | - |  28.6$\pm$1.7 |  46.1$\pm$3.3
DINOSAUR (MLP) [45] |  35.5$\pm$0.2 | - | - | 65.1$\pm$1.2
SlotDiffusion [66] | 30.2 | - | 30.2 | 60.0
LSD [27] |  39.0$\pm$0.5 | - |  37.6$\pm$0.5 |  52.2$\pm$0.9
SPOT w/o ENS (ours) | 39.9$\pm$1.1 | - | 39.0$\pm$1.1 | 56.4$\pm$4.1/59.9$\pm$0.4
SPOT w/ ENS (ours) | 40.1$\pm$1.2 | - | 39.3$\pm$1.2 | 56.8$\pm$4.3/59.9$\pm$0.4
PASCAL
1-block Mask | 19.1 | 23.0 | 17.4 | 49.9
6-block Mask | 24.7 | 25.9 | 24.2 | 13.9
SA [37]† | 24.6 | 24.9 | - | 12.3
SLATE [48]† | 35.9 | 41.5 | - | 15.6
CAE [39]† |  32.9$\pm$0.9 |  37.4$\pm$1.0 | - | -
DINOSAUR (MLP) [45] |  39.5$\pm$0.1 |  40.9$\pm$0.1 | - |  24.6$\pm$0.2
DINOSAUR [45] |  44.0$\pm$1.9 |  51.2$\pm$1.9 | - | 24.8$\pm$2.2
Rotating Features [40] |  40.7$\pm$0.1 |  46.0 $\pm$0.1 | - | -
SlotDiffusion [66] | 50.4 | 55.3 | - | 17.8
SPOT w/o ENS (ours) | 48.1$\pm$0.4 | 55.3$\pm$0.4 | 46.5$\pm$0.4 | 19.4$\pm$0.7/19.7$\pm$0.4
SPOT w/ ENS (ours) | 48.3$\pm$0.4 | 55.6$\pm$0.4 | 46.8$\pm$0.4 | 19.9$\pm$0.9/19.7$\pm$0.4
COCO
7-block Mask | 16.8 | 19.5 | 15.9 | 22.7
SA [37]† | 17.2 | 19.2 | - | 21.4
SLATE [48]† | 29.1 | 33.6 | - | 32.5
DINOSAUR [45] |  32.3$\pm$0.4 |  38.8$\pm$0.4 | - |  34.3$\pm$0.5
SlotDiffusion [66] | 31.0 | 35.0 | - | 37.2
Stable-LSD [27] | 30.4 | - | - | 35.0
SPOT w/o ENS (ours) | 34.7$\pm$0.1 | 44.3$\pm$0.3 | 32.7$\pm$0.1 | 36.6$\pm$0.3/37.8$\pm$0.5
SPOT w/ ENS (ours) | 35.0$\pm$0.1 | 44.7$\pm$0.3 | 33.0$\pm$0.1 | 37.0$\pm$0.2/37.8$\pm$0.5
Table 7: _Comparison with object-centric methods on COCO, PASCAL, MOVi-C and
MOVi-E datasets_. SPOT results are the mean and std over 3 seeds. For SPOT FG-
ARI, we report results from both _decoder/slot encoder_ masks. DINOSAUR uses
an autoregressive decoder and DINO [7] ViT encoder (ViT-B/16 for PASCAL and
MOVi-C, ViT-S/8 for COCO). DINOSAUR-MLP uses an MLP decoder and DINO ViT
encoder (ViT-B/16 for COCO and PASCAL, ViT-S/8 for MOVi-C/E). †: COCO and
PASCAL results of SA and SLATE are from [66], MOVi-C/E results are from [45]
for SA and from [27] for SLATE, PASCAL results of CAE are from [40]. We bold
the best and underline the second-best results.

FG-ARI has been a common metric in assessing predicted object masks against
ground-truth segmentation in previous works. However, several recent works
have raised concerns about its reliability [66, 14, 28, 44, 45]. Notably, FG-
ARI is criticized for its unreliability, which may favor either over-
segmentation [14, 44] or under-segmentation. Additionally, the fact that it
ignores background pixels makes it unable to probe a model’s effectiveness in
object segmentation [28, 44].

Consequently, FG-ARI can be misleading in assessing segmentation quality. We
illustrate this in Tab. 7, where we show that we can achieve the highest 49.9
FG-ARI score in the Pascal dataset _by trivially assigning all pixels to a
single slot-mask (1-block mask)_. This highlights the metric’s unreliability,
particularly in datasets with scenes featuring few or single objects.

To further examine FG-ARI, Fig. 6 compares qualitative results of SPOT with an
autoregressive transformer decoder (the standard setup) and SPOT with an MLP-
based decoder. While the standard SPOT setup outperforms SPOT with an MLP
decoder in mBOi, mBOc, and mIoU metrics, SPOT with MLP achieves a higher FG-
ARI score (42.5 vs. 37.8) (see Table 1 and Table 4 in the main paper).
However, the qualitative results in Fig. 6 indicate that SPOT with MLP is
distinctly inferior to the standard SPOT. This is particularly evident in the
first three images, showing notable over-segmentation (as seen with the bear,
horse, and dog) and incorrect object grouping (such as the horse and rider).

These observations emphasize the inadequacy of FG-ARI in measuring the
segmentation quality of predicted object masks. Unsupervised object-centric
methods should place greater reliance on mBO and mIoU metrics.

| SP | ST | Decoder | Slot Attention
---|---|---|---|---
| mBOi | mIoU | FG-ARI | mBOi | mIoU | FG-ARI
(a) |  |  |  45.3$\pm$1.8 |  44.6$\pm$1.7 |  50.6$\pm$4.3 |  42.8$\pm$0.1 |  42.0$\pm$0.1 |  55.4$\pm$0.6
(b) | ✓ |  |  46.1$\pm$0.9 |  45.2$\pm$0.9 |  51.8$\pm$2.7 |  42.5$\pm$0.1 |  41.6$\pm$0.1 |  57.6$\pm$0.7
(c) | ✓ | ✓ | 47.3$\pm$1.2 | 46.7$\pm$1.3 | 52.3$\pm$3.3 | 46.2$\pm$0.8 | 45.4$\pm$0.7 | 57.9$\pm$2.0
Table 8: _Ablation study on MOVi-C_. Metrics for slot masks generated by Decoder and Slot Attention. Results are mean and standard dev. over 3 seeds. SP: sequence permutation with ensebling of nine permutations at test-time, ST: self-training. BO-QSA [26] | ST | mBOi
---|---|---
|  | Collapse
✓ |  |  30.7$\pm$2.2
✓ | ✓ | 34.7$\pm$0.1
Table 9: _Image encoder training stability results on COCO_. Results are the
mean and standard deviation of the decoder’s mBOi over 3 seeds. ST: self-
training studies the impact of the distillation loss $L_{\mathrm{ATT}}$, and
BO-QSA the impact of using trainable initialization of slots along with bi-
level optimization [26, 8]. All models use sequence permutation in the
autoregressive decoder.

##  Appendix B Additional experimental results

###  B.1 Comparison with prior object-centric methods

In Table 7, we present extended benchmark results, including the mean
Intersection over Union (mIoU) and Foreground Adjusted Rand index (FG-ARI)
across all datasets. We show results for SPOT on FG-ARI derived from both
decoder and slot encoder attention masks, offering a more comprehensive view
of its capabilities.

SPOT outperforms other methods in mBOi, mBOc and mIoU across all datasets,
except for the mBOi in the Pascal setting, where it ranks second-best.
Concerning FG-ARI, SPOT achieves the best results in COCO and second-best in
MOVi-C and MOVi-E (within the standard deviation). However, as discussed in
Appendix A, FG-ARI is unreliable. For instance, in the Pascal dataset, the
1-block mask (_i.e_., the trivial solution where the entire image is covered
by a single mask) achieves the highest FG-ARI score, doubling the score of the
second-best.

###  B.2 Ablations in MOVi-C

In Tab. 8, we analyze the effects of self-training and sequence permutations
on the MOVi-C dataset. Both of these approaches enhance performance,
resulting, for instance, in a 2-point increase in mBOi.

###  B.3 Image encoder training stability analysis

In Tab. 9, we notice that fine-tuning the image encoder, without self-
training, can avoid training collapse by using trainable initial slots and bi-
level optimization (BO-QSA [26]). However, the achieved mBOi score, 30.7, is
notably lower compared to not fine-tuning the image encoder and using randomly
initialized slots, where the mBOi is 32.7 (refer to Tab. 1 entry (b) in the
main paper). Furthermore, fine-tuning the image encoder solely with BO-QSA and
without self-training exhibits a high standard deviation of 2.2, indicating
there is still a training stability issue. The introduction of self-training
not only enhances performance significantly (from 30.7 to 34.7) but also
stabilizes the training process, as evident from the drop in standard
deviation from 2.2 to 0.1. This underscores the crucial role of self-training
in ensuring training stability.

05101520253035404550556065707580859095100010203040506070Instance size (% of
total image area)mBOi

Figure 7: _Analysis of SPOT performance across different instance sizes._ We demonstrate SPOT’s performance, measured in mBOi on COCO, across varied instance sizes. The instance sizes are expressed as a percentage of the total image area, categorized into distinct bins: 0-0.5%, 0.5-1%, 1-5%, 5-10%, 10-20%, 20-50%, 50-80%, 80-100%. |  |  |
---|---|---|---
(a) Large size instances
|  |  |
(b) Medium size instances
|  |  |
(c) Small size instances
Figure 8: Example SPOT’s results on COCO 2017, using 7 slots, for large,
medium and small instance sizes.

###  B.4 Performance across different instance sizes

In Fig. 7, we present a detailed analysis of SPOT’s performance across various
instance sizes. We observe that SPOT performs optimally when instances occupy
between 20% and 80% of the input image area. For larger instances (exceeding
80%), SPOT continues to yield favorable results. Conversely, a decline in
performance is noted as the size of the instances decreases. We note that this
decrease in performance for smaller instance sizes is expected due to the
coarse resolution of ViT encoders.

In Fig. 8, we provide examples for large, medium and small instance mask
sizes. For the large and medium-sized examples, we observe good performance.
For small instances, there is often a tendency to be grouped together or be
part of the background.

##  Appendix C Implementation details

###  C.1 MLP decoder

In the MLP-based decoder, we employ the spatial broadcast mechanism as
described in prior works [45, 59], in which each slot is expanded into $n$
tokens that correspond to the patches. Next, learnable positional information
is added to these tokens for spatial identification. These tokens are then
processed individually by a four-layer MLP with ReLU activations. This process
yields both the reconstruction and an alpha map for each slot ($d_{y}$ \+ 1
dimensions). The alpha map serves as an attention map that indicates the
active regions of the slot. The final reconstructed output is obtained by
aggregating these individual slot reconstructions, using the alpha maps as
weighting factors. In the MLP decoder, the slot-attention masks are the
predicted alpha maps.

###  C.2 SPOT models

We employ the Adam optimizer [30] with $\beta_{1}=0.9$, $\beta_{2}=0.999$, no
weight decay, and a batch size of 64. The learning rate ($lr$) follows a
linear warm-up from 0 to a peak value for $10000$ training iterations and then
decreases via a cosine annealing schedule. For experiments on COCO and PASCAL
using DINO and MoCo-v3 encoders and for both training stages, the peak
learning rate is $4\times 10^{-4}$ and the low value is $4\times 10^{-7}$. For
both stages with MAE on COCO and the first stage of MOVi-C/E experiments, the
peak value is $2\times 10^{-4}$ and the low value is $4\times 10^{-5}$. For
the second stage of MOVi-C/E experiments, the peak value is $2\times 10^{-4}$
and the low value is $1.5\times 10^{-4}$. For each training stage on COCO and
PASCAL, we use 50 and 560 training epochs, respectively. For MOVi-C/E
experiments, we use 65 and 30 epochs for the first and second stages,
respectively.

We implement our SPOT models using ViT-B/16 [11] for the encoder (by default
initialized with DINO [7]), without applying the final layer norm. In the
autoregressive transformer decoder, we use 4 transformer blocks each one with
6 heads. For the MLP-based decoder, we use 2048 hidden layer size. For all
experiments, we use 3 iterations in the slot attention module with the
dimension of slot $d_{u}$ being 256 and the slot attention’s MLP hidden
dimension being 1024. Unless stated otherwise, loss weight $\lambda$ is 0.005.
For MLP-based decoder, MoCo-v3 [10], and MAE [21] encoders, $\lambda$ is
0.001. At the self-training stage, we fine-tune the last four ViT encoder’s
blocks.

Following [45], on COCO, PASCAL, MOVi-C, and MOVi-E we use 7, 6, 11, and 24
slots, respectively.

In the main paper, we explored a training approach inspired by CapPa [54]. To
tune this approach, we experimented with different percentages of parallel
decoding —25%, 50%, and 100%— and observed mBOi scores of 27.8, 25.9, and
23.6, respectively. We found that a lower percentage of parallel decoding
correlates with better performance. Consequently, we used a 25% parallel
decoding for the experiment in the main paper.

###  C.3 Datasets and evaluation

Here, we provide details about the employed datasets and the evaluation
protocol.

COCO We use COCO 2017 dataset for training, which consists of 118,287 images,
and evaluate SPOT on the validation set of 5,000 images. During training, we
resize the minor axis of the images to 224 pixels, perform center cropping at
224$\times$224 and then random horizontal flipping with 0.5 probability. For
the evaluation, we use both types of masks: instance masks for the mBOi, mIoU,
FG-ARI and segmentation masks for the mBOc metrics. For consistency with
previous studies [45, 25, 19], we scale the minor axis to 320 pixels, perform
center cropping, and evaluate the masks at 320$\times$320 resolution.

PASCAL We use PASCAL VOC 2012 “trainaug” set for training, which consists of
10,582 images. The "trainaug" variant contains 1,464 images from the
segmentation train set and 9,118 from the SBD dataset [20]. We use this split,
for consistency with previous works [45, 55, 42]. During training, we resize
the minor axis of the images to 224 pixels, perform simple random cropping at
224$\times$224 and then random horizontal flipping with 0.5 probability. We
evaluate on the official segmentation validation set, which consists of 1,449
images. We ignore the unlabeled pixels during evaluation. Similar to COCO, we
resize the minor axis to 320 pixels, perform center cropping, and evaluate the
masks at 320$\times$320 resolution.

MOVi-C/E We transform MOVi-C/E datasets, originally consisting of videos, into
image datasets by selecting nine random frames from each clip of the train
set. From this process, we obtain 87,633 images for training with MOVi-C and
87,741 images with MOVi-E. During training, we resize the images at
224$\times$224 resolution. For evaluation, we use every frame from the 250
clips of the validation set consisting of 6,000 images, being consistent with
prior works [45, 31, 13]. We evaluate the masks at the full 128$\times$128
resolution.

n-block Mask : As in [45], to generate block mask patterns, we first divide
the image into columns and then subdivide these columns to obtain the required
number of slot masks. Specifically, for MOVi-C, we utilize 3 columns to
produce a total of 11 block masks. For MOVi-E, we use 4 columns to accommodate
24 masks. In the case of PASCAL VOC 2012, we use 2 columns for 6 masks. For
COCO, we use 2 columns for 7 masks. The number of block masks is aligned with
the number of slots utilized for each respective dataset. For the 1-block
mask, we employ just one column and one mask; in other words, the entire image
is covered by a single mask.


##  pocoMC: A Python package for accelerated Bayesian inference in astronomy
and cosmology

Minas Karamanis 1222E-mail<EMAIL_ADDRESS>David Nabergoj 2,
Florian Beutler 1,

John A. Peacock 1, and Uroš Seljak 3

1Institute for Astronomy, University of Edinburgh, Royal Observatory,
Blackford Hill,
Edinburgh EH9 3HJ, UK
2Faculty of Computer and Information Science, University of Ljubljana, Večna
pot 113, 1000 Ljubljana, Slovenia
3Physics Department, University of California and Lawrence Berkeley National
Laboratory Berkeley, CA 94720, USA

###### Summary

pocoMC is a Python package for accelerated Bayesian inference in astronomy and
cosmology. The code is designed to sample efficiently from posterior
distributions with non–trivial geometry, including strong multimodality and
non–linearity. To this end, pocoMC relies on the Preconditioned Monte Carlo
algorithm which utilises a Normalising Flow in order to decorrelate the
parameters of the posterior. It facilitates both tasks of parameter estimation
and model comparison, focusing especially on computationally expensive
applications. It allows fitting arbitrary models defined as a log–likelihood
function and a log–prior probability density function in Python. Compared to
popular alternatives (e.g. nested sampling) pocoMC can speed up the sampling
procedure by orders of magnitude, cutting down the computational cost
substantially. Finally, parallelisation to computing clusters manifests linear
scaling. The code is publicly available at https://github.com/minaskar/pocomc.

##  1 Statement of need

Over the past few decades, the volume of astronomical and cosmological data
has increased substantially. At the same time, theoretical and
phenomenological models in these fields have grown even more complex. As a
response to that, a number of methods aiming at efficient Bayesian computation
have been developed with the sole task of comparing those models to the
available data (Trotta,, 2017; Sharma,, 2017). In the Bayesian context,
scientific inference proceeds though the use of Bayes’ theorem:

$\mathcal{P}(\theta)=\frac{\mathcal{L}(\theta)\pi(\theta)}{\mathcal{Z}}$ (1)

where the posterior $\mathcal{P}(\theta)\equiv p(\theta|d,\mathcal{M})$ is the
probability of the parameters $\theta$ given the data $d$ and the model
$\mathcal{M}$. The other components of this equation are: the likelihood
function $\mathcal{L}(\theta)\equiv p(d|\theta,\mathcal{M})$, the prior
$\pi(\theta)\equiv p(\theta|\mathcal{M})$, and the model evidence
$\mathcal{Z}=p(d|\mathcal{M})$. The prior and the likelihood are usually
provided as input in this equation and one seeks to estimate the posterior and
the evidence. Knowledge of the posterior, in the form of samples, is paramount
for the task of parameter estimation whereas the ratio of model evidences
yields the Bayes factor which is the cornerstone of Bayesian model comparison.

Markov chain Monte Carlo (MCMC) has been established as the standard tool for
Bayesian computation in astronomy and cosmology, either as a standalone
algorithm or as part of another method (e.g. nested sampling (Skilling,,
2006)). However, as MCMC relies on the local exploration of the posterior, the
presence of non-linear correlation between parameters and multimodality can at
best hinder its performance and at worst violate its theoretical guarantees of
convergence (i.e. ergodicity). Usually, those challenges are partially
addressed by reparameterising the model using a common change–of–variables
parameter transformation. However, guessing the right kind of
reparameterisation a priori is not trivial as it often requires a deep
knowledge of the physical model and its symmetries. These problems are usually
complicated further by the substantial computational cost of evaluating
astronomical and cosmological models. pocoMC is designed to tackle exactly
these kinds of difficulties by automatically reparameterising the model such
that the parameters of the model are approximately uncorrelated and standard
techniques can be applied. As a result, pocoMC produces both samples from the
posterior distribution and an unbiased estimate of the model evidence thus
facilitating both scientific tasks with excellent efficiency and robustness.
Compared to popular alternatives such as nested sampling, pocoMC can reduce
the computational cost, and thus, the total run time of the analysis by orders
of magnitude, in both artificial and realistic applications (Karamanis et
al.,, 2022). Finally, the code is well-tested and is currently used for
research work in the field of gravitational wave parameter estimation
(Vretinaris et al.,, 2022).

Figure 1: Logo of pocoMC.

##  2 Method

pocoMC implements the Preconditioned Monte Carlo (PMC) algorithm. PMC combines
the popular Sequential Monte Carlo (SMC) (Del Moral et al.,, 2006) method with
a Normalising Flow (NF) (Papamakarios et al.,, 2021). The latter works as a
preconditioner for the target distribution of the former. As SMC evolves a
population of particles, starting from the prior distribution and gradually
approaching the posterior distribution, the NF transforms the parameters of
the target distribution such that any correlation between parameters or
presence of multimodality is removed. The effect of this bijective
transformation is the substantial rise in the sampling efficiency of the
algorithm as the particles are allowed to sample freely from the target
without being hindered by its locally–curved geometry. The method is explained
in detail in the accompanying publication (Karamanis et al.,, 2022) and we
provide only a short summary here.

###  2.1 Sequential Monte Carlo

The basic idea of basic SMC is to sample from the posterior distribution
$\mathcal{P}(\theta)$ by first defining a path of intermediate distributions
starting from the prior $\pi(\theta)$. In the case of pocoMC the path has the
form:

$p_{t}(\theta)=\pi(\theta)^{1-\beta_{t}}\mathcal{P}(\theta)^{\beta_{t}}$ (2)

where $0=\beta_{1}<\beta_{2}<\dots<\beta_{T}=1$. Starting from the prior, each
distribution with density $p_{t}(\theta)$ is sampled in turn using a
collection of particles propagated by a number of MCMC steps. Prior to MCMC
sampling, the particles are re-weighted using importance sampling and then re-
sampled to account for the transition from $p_{t}(\theta)$ to
$p_{t+1}(\theta)$. pocoMC utilises the importance weights of this step to
define an estimator for the effective sample size (ESS) of the population of
particles. Maintaining a fixed value of ESS during the run allows pocoMC to
adaptively specify the $\beta_{t}$ schedule.

###  2.2 Preconditioned Monte Carlo

In vanilla SMC, standard MCMC methods (e.g. Metropolis-Hastings) are used to
update the positions of the particles during each iteration. This however can
become highly inefficient if the distribution $p_{t}(\theta)$ is characterised
by a non–trivial geometry. pocoMC, which is based on PMC, utilises a NF to
learn an invertible transformation that simplifies the geometry of the
distribution by mapping $p_{t}(\theta)$ into a zero-mean unit-variance normal
distribution. Sampling then proceeds in the latent space in which correlations
are substantially reduced. The positions of the particles are transformed back
to the original parameter space at the end of each iteration. This way, PMC
and pocoMC are able to sample from very challenging posteriors very
efficiently using simple Metropolis-Hastings updates in the
preconditioned/uncorrelated latent space.

##  3 Features

  * •

User–friendly black-box API (only the log-likelihood, log-prior and some prior
samples required from the user)

  * •

Default configuration sufficient for most applications (no tuning is required
but is possible for experienced users)

  * •

Posterior corner, trace, and run plotting tools

  * •

Support for both MAF and RealNVP normalising flows with added regularisation
(Papamakarios et al.,, 2017; Dinh et al.,, 2016)

  * •

Straightforward parallelisation using MPI or multiprocessing

  * •

Continuous integration, unit tests and wide range of examples available

  * •

Extensive documentation available online http://pocomc.readthedocs.io

## Acknowledgements

MK would like to thank Jamie Donald-McCann and Richard Grumitt for providing
constructive comments and George Vretinaris for feedback on an early version
of the code. This project has received funding from the European Research
Council (ERC) under the European Union’s Horizon 2020 research and innovation
program (grant agreement 853291), and by the U.S. Department of Energy, Office
of Science, Office of Advanced Scientific Computing Research under Contract
No. DE-AC02-05CH11231 at Lawrence Berkeley National Laboratory to enable
research for Data-intensive Machine Learning and Analysis. FB is a University
Research Fellow.

## References

  * Del Moral et al., (2006) Del Moral, Pierre, Doucet, Arnaud, & Jasra, Ajay. 2006.  Sequential monte carlo samplers.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(3), 411–436.
  * Dinh et al., (2016) Dinh, Laurent, Sohl-Dickstein, Jascha, & Bengio, Samy. 2016.  Density estimation using real nvp.  arXiv preprint arXiv:1605.08803.
  * Karamanis et al., (2022) Karamanis, Minas, Beutler, Florian, Peacock, John A, Nabergoj, David, & Seljak, Uroš. 2022.  Accelerating astronomical and cosmological inference with Preconditioned Monte Carlo.  in prep.
  * Papamakarios et al., (2017) Papamakarios, George, Pavlakou, Theo, & Murray, Iain. 2017.  Masked autoregressive flow for density estimation.  Advances in neural information processing systems, 30.
  * Papamakarios et al., (2021) Papamakarios, George, Nalisnick, Eric T, Rezende, Danilo Jimenez, Mohamed, Shakir, & Lakshminarayanan, Balaji. 2021.  Normalizing Flows for Probabilistic Modeling and Inference.  J. Mach. Learn. Res., 22(57), 1–64.
  * Sharma, (2017) Sharma, Sanjib. 2017.  Markov chain Monte Carlo methods for Bayesian data analysis in astronomy.  arXiv preprint arXiv:1706.01629.
  * Skilling, (2006) Skilling, John. 2006.  Nested sampling for general Bayesian computation.  Bayesian analysis, 1(4), 833–859.
  * Trotta, (2017) Trotta, Roberto. 2017.  Bayesian methods in cosmology.  arXiv preprint arXiv:1701.01467.
  * Vretinaris et al., (2022) Vretinaris, George, Vretinaris, Stamatis, Mermigkas, Christos, Karamanis, Minas, & Stergioulas, Nikolaos. 2022.  Robust and fast parameter estimation of gravitational waves from neutron star merger remnants.  in prep.