# Towards Interpretable Controllability
in Object-Centric Learning

Jinwoo Kim1∗, Janghyuk Choi2∗, Jaehyun Kang1, Changyeon Lee1,
0Ho-Jin Choi2 & Seon Joo Kim1
1Yonsei University 2KAIST

###### Abstract

The binding problem in artificial neural networks is actively explored with
the goal of achieving human-level recognition skills through the comprehension
of the world in terms of symbol-like entities. Especially in the field of
computer vision, object-centric learning (OCL) is extensively researched to
better understand complex scenes by acquiring object representations or slots.
While recent studies in OCL have made strides with complex images or videos,
the interpretability and interactivity over object representation remain
largely uncharted, still holding promise in the field of OCL. In this paper,
we introduce a novel method, Slot Attention with Image Augmentation (SlotAug),
to explore the possibility of learning interpretable controllability over
slots in a self-supervised manner by utilizing an image augmentation strategy.
We also devise the concept of sustainability in controllable slots by
introducing iterative and reversible controls over slots with two proposed
submethods: Auxiliary Identity Manipulation and Slot Consistency Loss.
Extensive empirical studies and theoretical validation confirm the
effectiveness of our approach, offering a novel capability for interpretable
and sustainable control of object representations. Code will be available
soon.

**footnotetext: Equal contribution.

##  1 Introduction

Compositional comprehension of visual scenes (Marr, 2010; Tenenbaum et al.,
2011; Johnson et al., 2017; Fischler & Elschlager, 1973), essential for
various computer vision tasks such as localization (Cho et al., 2015) and
reasoning (Mao et al., 2019), requires human-like understanding of complex
world (Treisman, 1996; Spelke & Kinzler, 2007; Lake et al., 2017). In response
to this, object-centric learning (OCL) has emerged as an active research area
(Locatello et al., 2020; Kipf et al., 2021; Greff et al., 2016). OCL aims to
enable a model to decompose an image into its components in terms of an
object, and to acquire object representations or slots, without relying on
human-annotated labels.

In pursuit of a deeper understanding of images, interpretable and controllable
object representation has been studied (Greff et al., 2019; Burgess et al.,
2019; Singh et al., 2023). Nevertheless, the previous approaches face
limitations in achieving interpretable controllability as they require
additional processes to figure out how to interact with slots, such as
exploring a connection between specific values in slots and object properties
by a manual exhaustive search, and training a feature selector with ground-
truth object properties (Fig. 1(a)). This issue arises due to a training-
inference discrepancy, wherein interactions with slots are only considered
during the inference stage. This discrepancy problem brings ambiguity in how
to interact with object representations, hindering interpretable
controllability. Furthermore, learning interpretable controllability is
followed by another challenge: object representation should be intact even
after multiple manipulations by humans. In this context, we devise the concept
of sustainability referring to the ability to preserve the nature of slots,
allowing for iterative manipulations.

In this work, we advance the field of OCL with respect to the interpretability
of object representation. To achieve interpretable controllability, we propose
a method that enables the manipulation of object representation through
semantically interpretable instructions in a self-supervised manner. We
address the training-inference discrepancy problem by incorporating image
augmentation into our training pipeline (Fig. 1(c)). By involving the slot
manipulation in the training, we can resolve the discrepancy problem and
streamline the way to interact with slots in the inference stage (Fig. 1(b)
and (d)).

Figure 1:  Overview of our method compared to the previous methods. (a)
Previous methods require an additional process to manipulate slots such as
feature selection during inference. (b) Our model, however, has the shared
process of manipulating slots between the training and inference stages. (c)
To ensure homogeneity between the training and inference stages, we
incorporate scenarios involving image manipulation into the training phase.
This includes the application of simple image augmentation techniques such as
scaling, translating, and color shifting. (d) Upon completion of the training,
our model achieves interpretable controllability, enabling users to manipulate
individual objects according to their intentions.

Second, to attain sustainability in object representation, we introduce
Auxiliary Identity Manipulation (AIM) and Slot Consistency Loss (SCLoss). AIM
is a methodology designed to facilitate the learning of the concept of multi-
round manipulation. AIM is implemented by incorporating an auxiliary
manipulation process into the intermediate stage of slot manipulation, where
the auxiliary manipulation introduces no semantic changes to object properties
such as zero-pixel translations. This simple auxiliary process can expose our
model to multi-round manipulation: we can make two-round manipulations with
one instruction from the augmentation and the other from the auxiliary
manipulation. Additionally, SCLoss allows our model to learn the concept of
reversible manipulation, such as the relationship between moving an object 1
pixel to the right and moving it 1 pixel to the left. After being trained with
SCLoss, our model produces consistent and reusable representations that can
undergo multiple modifications and enhance their usability. With AIM and
SCLoss, our model achieves sustainability in object representation.

Extensive experiments are shown to demonstrate the interpretable and
sustainable controllability of our model. To assess interpretability, we
conduct object manipulation experiments where slots are guided by semantically
interpretable instructions. In evaluating sustainability, we introduce novel
experiments, including the durability test. Our evaluation encompasses not
only pixel space assessments such as image editing via object manipulation,
but also slot space analyses such as property prediction, to provide a
comprehensive examination of our approach.

The main contributions of the paper can be summarized as follows: (i) Towards
interpretable controllability in object-centric learning, we incorporate image
augmentation into the training process to explore the possibility of
interpretable controllability over object representation. (ii) We introduce a
novel concept of sustainability in object representation, strengthening the
interaction between neural networks and humans, which is an important aspect
of interpretable controllability. (iii) Two novel methods are introduced to
ensure sustainability in object representation: Auxiliary Identity
Manipulation and Slot Consistency Loss. (iv) We extensively validate the
effectiveness of our method through empirical studies including novel
experiments and analyses on pixel and slot space.

##  2 Methods

Figure 2:  Architecture of our model. From a given image $img_{ref}$, we first
generate an augmented image $img_{aug}$ (leftmost part of the figure), and the
corresponding instruction $insts_{ref2aug}$ and its inverse $insts_{aug2ref}$.
Our model produces slots from $img_{ref}$ and decodes the slots to reconstruct
the original image ($recon_{ref}$). The slots are also manipulated with
SlotManipulation module which takes $insts_{ref2aug}$ as the other input. We
incorporate Auxiliary Identity Manipulation (AIM) into this manipulation
process. The details are provided in the right part of the figure. The
manipulated slots are then simultaneously 1) decoded into a reconstruction of
the augmented image $recon_{aug}$, and 2) re-manipulated by SlotManipulation
with $insts_{aug2ref}$. Our total loss consists of the reconstruction losses
of reference and augmented images, and the slot-level cycle consistency.

###  2.1 Preliminary: Slot Attention

Slot Attention (SA) (Locatello et al., 2020) introduces the concept of slots,
a set of $K$ vectors of dimension $D_{slot}$, that serves as the object
representation. The slots are initialized by a Gaussian distribution with
learnable mean $\mu$ and sigma $\sigma$, and are updated over $T$ iterations
by the slot attention module. The final slots are then decoded to reconstruct
the target image. To provide a comprehensive understanding of our method, we
describe the mechanism of spatial binding by Slot Attention in Alg. 2 in the
Appendix. Each updated slot is then decoded individually into an RGBA image
using a spatial broadcast decoder (Watters et al., 2019) which is shared
across slots. The decoded images are blended into a single image using alpha
masks to reconstruct the input image. The training objective is the mean
squared error (MSE) between the original input image and the reconstructed
image, following a self-supervised learning approach.

###  2.2 Self-Supervised Learning for Interpretable Controllability

Data augmentation. We introduce a simple data augmentation scheme that, for a
given input image or a reference image $img_{ref}\in\mathbb{R}^{H\times
W\times 3}$, generates an augmented image $img_{aug}\in\mathbb{R}^{H\times
W\times 3}$ and the transformation instructions between them,
$insts_{ref2aug}$ and $insts_{aug2ref}\in\mathbb{R}^{K\times L}$, where $L$
indicates the total number of values to represent the object properties.
$img_{aug}$ is produced by a random translation, scaling, or color shifting on
$img_{ref}$. To transform $img_{ref}$ into $img_{aug}$, we employ a set of
instructions known as $insts_{ref2aug}$. These instructions comprise a list of
values that dictate the augmentation process, including translation values, a
scaling factor, and color shift values in the HSL color space. We also have
the inverse instructions, $insts_{aug2ref}$, which allow us to revert
$img_{aug}$ back to $img_{ref}$. Details for the data augmentation are
described in the Appendix.

Training. We propose a novel training process that leverages image
augmentation (Fig. 2). Our training scheme enables learning interpretable
controllability which allows us to interact with the model via semantically
interpretable instructions. Our training process involves data augmentation,
spatial binding, slot manipulation, and image reconstruction via slot
decoding. For a given input image, we initially perform data augmentation to
yield $img_{ref}$, $img_{aug}$, $insts_{ref2aug}$, and $insts_{aug2ref}$.
Then, the model performs spatial binding on $img_{ref}$ to produce
$slots_{ref}$.

Thereafter, the model conducts SlotManipulation (Alg. 1) to modify
$slots_{ref}$ based on $insts_{ref2aug}$. In the SlotManipulation, we utilize
a newly introduced component called PropertyEncoder, which is 3-layer multi-
layer perceptrons (MLPs). This PropertyEncoder is responsible for generating
vector representations, $\texttt{inst}\\_\texttt{vec}$, which capture the
essence of transformation instructions. Each $\text{PropertyEncoder}_{j}$
generates an $\texttt{inst}\\_\texttt{vec}_{\texttt{j}}$ that encodes the
values of $insts_{ref2aug}$ for the $j$-th property. These vectors are then
added to $slots_{ref}$ to reflect the effect of $insts_{ref2aug}$. This
addition is followed by a residual connection, along with layer normalization
and another MLP to generate $slots_{ref2aug}$.

Algorithm 1 Our slot manipulation algorithm in pseudo code. The algorithm
takes slots and insts as input, where insts contains the information for
modifications. J represents the number of object properties, while
$\texttt{P}_{j,f}$ and $\texttt{P}_{j,l}$ indicate the first and last indices
of the j-th object property values. The PropertyEncoder outputs a vector of
the same dimension as slots, $K\times D_{slots}$.

1:function SlotManipulation($\texttt{slots}\in\mathbb{R}^{K\times D_{slots}},\
\texttt{insts}\in\mathbb{R}^{K\times L}$)

2: for $j=0\ldots J$ do

3: $\texttt{inst}_{j}=\texttt{insts}[:,\texttt{P}_{j,f}:\texttt{P}_{j,l}]$

4:
$\texttt{inst\\_vec}_{j}=\texttt{PropertyEncoder}_{j}(\texttt{LayerNorm}(\texttt{inst}_{j}))$

5: $\texttt{slots}=\texttt{slots}+\texttt{inst\\_vec}_{j}$

6: end for

7: $\texttt{slots}=\texttt{slots}+\texttt{MLP(LayerNorm(slots))}$

8: return slots

9:end function

Lastly, $slots_{ref2aug}$ is decoded by the decoder to create the
$recon_{aug}$, the reconstruction image for the $img_{aug}$. The MSE between
$img_{aug}$ and $recon_{aug}$ serves as a training loss,
$\mathcal{L}_{\text{aug}}$. To ensure stable training, we also adopt an
additional loss, $\mathcal{L}_{\text{ref}}$, the MSE between the $img_{ref}$
and $recon_{ref}$, the reconstructed reference image decoded from
$slots_{ref}$. Accordingly, our training loss for image reconstruction is
defined as
$\mathcal{L}_{\text{recon}}=\mathcal{L}_{\text{ref}}+\mathcal{L}_{\text{aug}}$.

Inference. To perform object manipulation, we provide the model with the
position of the target object, along with the instruction to be carried out.
When given the position of the target object, we use the Hungarian algorithm
(Kuhn, 1955) to find the slot for the object closest to the given position. To
predict the position of an object encoded in a slot, we compute the center of
mass acquired from the alpha mask by the decoder or from the attention map
between the visual encodings and the slot. After figuring out the desired
slot, we perform slot manipulation with the given instructions.

###  2.3 Sustainability in Object Representation

In this work, we introduce sustainability which stands for the concept that
object representations should sustain their integrity even after undergoing
iterative manipulations. Therefore, sustainability is a key feature that
contributes to the reliable and reusable object representation.

Auxiliary Identity Manipulation (AIM) serves as the identity operation for
slot manipulation, indicating no changes in object properties. By manipulating
slots with instructions that include zero values for translation, one for
scaling, and so on, AIM is supposed to make each slot preserve the original
identity of the object. We incorporate AIM into the training process to make
the model recognize and maintain the intrinsic characteristics of objects
during iterative manipulations. AIM is applied to the slot manipulation
process as follows:

$\begin{split}slots^{\prime}_{ref2aug}=f(f(slots_{ref},insts_{ref2aug}),insts_{identity})\\\
=f(slots_{ref2aug},insts_{identity}),\phantom{,,,,000000}\end{split}$ (1)

where $f$ represents the SlotManipulation function, and $insts_{identity}$ is
the instruction that contains the identity elements for manipulating object
properties. In the followings, $slots^{\prime}_{ref2aug}$ is notated as
$slots_{ref2aug}$ for simplicity if not mentioned.

Slot Consistency Loss (SCLoss) addresses the issue of a slot diverging
significantly from its original state after iterative manipulations, even when
a user intends to restore the corresponding object to its original state. To
implement SCLoss, we introduce $slots_{revisited}$, which is derived by
executing a series of SlotManipulation operations on $slots_{ref}$ using
$insts_{ref2aug}$ and $insts_{aug2ref}$. Supposed that our goal is to ensure
$slots_{ref}$ and $slots_{revisited}$ have the same representation, we set the
MSE between them as SCLoss. As a result, the model learns to keep the two
distinct slots representing the same object as close as possible and to be
robust against multiple rounds of manipulation. The equation of SCLoss,
$\mathcal{L}_{\text{cycle}}$, and the total training loss,
$\mathcal{L}_{\text{total}}$, are as follows:

$\mathcal{L}_{\text{cycle}}=\frac{1}{K}\|f(f(slots_{ref},insts_{ref2aug}),insts_{aug2ref})-slots_{ref}\|_{2}^{2},$
(2)
$\mathcal{L}_{\text{total}}=w_{recon}\mathcal{L}_{\text{recon}}+w_{cycle}\mathcal{L}_{\text{cycle}},$
(3)

where $K$ is the number of slots, $f$ is the SlotManipulation function, and
$w_{recon}$ and $w_{cycle}$ are the weights for the corresponding loss.

##  3 Related Works

The binding problem in artificial neural networks (Greff et al., 2020),
inspired by cognitive science (Treisman, 1996; Feldman, 2013), is a subject of
active exploration, aiming to attain human-like recognition abilities by
understanding the world in terms of symbol-like entities (like objects). In
computer vision, object-centric learning (OCL) focuses on comprehending visual
scenes by considering objects and their relationships without labeled data
(Xie et al., 2022; Engelcke et al., 2021; Wu et al., 2022). MONet (Burgess et
al., 2019), IODINE (Greff et al., 2019), and GENESIS (Engelcke et al., 2019)
have adopted autoencoding architectures (Baldi, 2012; Kingma & Welling, 2013;
Makhzani et al., 2015) to accomplish self-supervised OCL, and Slot Attention
(Locatello et al., 2020) introduced the concept of slot competition, which
enables parallel updates of slots with a single visual encoding and decoding
stage. Recent studies have leveraged large-scale models to learn object
representations in complex images (Singh et al., 2021; Seitzer et al., 2022),
multi-view images (Sajjadi et al., 2022a), and videos (Kipf et al., 2021;
Singh et al., 2022). Other recent works have utilized object-related inductive
biases to improve the OCL models. SLASH (Kim et al., 2023) addressed the
instability in background separation using a learnable low-pass filter to
solidify the object patterns in the attention maps. SysBinder (Singh et al.,
2023) introduced a factor-level slot, called block, to disentangle object
properties and enhance the interpretability in OCL.

Several studies have shown the possibility of interacting with object
representation to manipulate the objects. VAE-based models such as IODINE
(Greff et al., 2019) and Slot-VAE (Wang et al., 2023b) showed that adjusting
the values of slots can change object properties. SysBinder (Singh et al.,
2023) demonstrated that replacing factor-level slot, called block, between
slots exchanges the corresponding properties. However, these works have
difficulties in determining ways to interact with slots as they require manual
efforts to identify the features associated with specific properties. ISA
(Biza et al., 2023) incorporates spatial symmetries of objects using slot-
centric reference frames into the spatial binding process, enhancing
interactivity of object representation for spatial properties such as position
and scale. Meanwhile, our method itself has no constraint on the types of the
target property, showing its expandability toward extrinsic properties such as
the shape and material of objects if there exist proper image augmentation
skills or labeled data. In another direction, MulMon (Li et al., 2020) and
COLF (Smith et al., 2022) showed the manipulation of extrinsic object
properties, such as position and z-axis rotation, by utilizing a novel view
synthesis with a multi-view dataset. In contrast, our work accomplishes direct
and interpretable controllability over object representation in single-view
images without requiring multi-view datasets.

##  4 Experiments

Datasets. We evaluate models on four multi-object datasets: Tetrominoes
(Rishabh et al., 2019), CLEVR6 (Johnson et al., 2017), CLEVRTEX6 (Karazija et
al., 2021) and PTR (Hong et al., 2021). For Tetrominoes, we use 60K and 15K
samples as a train and a test set, respectively. CLEVR6 is a subset of CLEVR
dataset, where $6$ stands for the maximum number of objects in a scene. We use
35K samples for training and 7.5K samples for testing. CLEVRTEX6 is a subset
of CLEVRTEX which is a complex variant of CLEVR, having complicated shapes,
textures, materials, and backgrounds. CLEVRTEX6 contains 20K and 5K samples
for training and testing. PTR is a dataset consisting of 52K training and 9K
test samples, containing complex objects with part-whole hierarchies.

Training. Unless stated otherwise, the training setup follows the methodology
in Slot Attention (Locatello et al., 2020). The number of epochs is 1000 with
20 warm-ups and 200 decaying epochs. We adopt AdamW (Loshchilov & Hutter,
2019) as the optimizer. The number of slots ($K$) is set to 7 and the input
image size ($H\times W)$ is set to 128 $\times$ 128, except for Tetrominoes
where $K=7$ and $H=W=64$. The weights for the training loss are set as
$w_{recon}=1.0$ and $w_{cycle}=0.1$. The details of the training process
including the data augmentation setting are stated in the Appendix.

Models. We employ the same model architecture as Slot Attention. However, for
CLEVRTEX6 and PTR, which are more complex datasets, we replace the encoder
with ViT (Dosovitskiy et al., 2020) pretrained by MAE (He et al., 2022) and
the decoder with that of SRT (Sajjadi et al., 2022b) while using an increased
size of the slot attention module. The additional details for adopting large
models are described in the Appendix. To clarify the methods used in ablative
studies, we categorize our model into three versions: v1, which is exclusively
trained with image augmentation; v2, which improves upon v1 with AIM; v3,
which extends v1 with both AIM and SCLoss. For qualitative studies, we use the
v3 model.

###  4.1 Human-Interpretable Control over Object Representation

Figure 3:  (a) Object manipulation with human-interpretable instruction. The
first and second columns are the ground-truth and reconstruction images,
respectively. The following columns are the results of the controls along the
instructions. Here, instructions are described with the text for easy
understanding. The actual instantiation of the instructions can be found in
the Appendix. From the first row onwards, the results are for Tetrominoes,
CLEVR, CLEVR, and PTR, respectively. (b) Conditional image composition. From
given source images, we can collect specific objects, which are indicated by
white numbers, and manipulate them to generate a novel image.

####  4.1.1 Image Editing by Object Manipulation

As shown in Fig. 3(a), our model can manipulate individual objects. We can
control not only multiple objects in a scene but also multiple properties of
an object with the specific intent of users through the instructions. In the
second row of the figure, we observe that our model can restore the cropped
part of the object (the pink one) when we pull the object inward after
changing the color of the object. Based on this observation, we can ascertain
that our object representations retain the intrinsic properties of objects
seamlessly even after manipulation. Furthermore, the interpretable
controllability is accomplished with a neglectable compromise of the
performance on both the object discovery and image reconstruction tasks, as
demonstrated in Tab. 2.

One may wonder how our model can excel in controlling individual slots while
being trained solely on image-level manipulation without any explicit object-
level supervision. We attribute this successful transition from image
augmentation to object manipulation to the slots’ ability to focus their
attention effectively on each distinct object. This capability is achieved by
the following key factors. Firstly, using slot-wise decoder (Locatello et al.,
2020) enables independent decoding for each slot, eliminating dependencies on
other slots. Secondly, using Attention Refining Kernel (ARK) (Kim et al.,
2023) allows our model to efficiently discover individual objects without any
concerns of attention leakage. These factors collectively ensure that the
slots remain directed toward their corresponding objects, thereby facilitating
precise object-level manipulations. We claim that these factors enable our
model to seamlessly extend the knowledge learned from image-level augmentation
to object-level manipulation. More discussions including theoretical proof and
empirical results are provided in the Appendix to substantiate our claim.

####  4.1.2 Conditional Image Composition

We introduce conditional image composition, an advanced version of a
downstream task called compositional generation (Singh et al., 2021). From
compositional generation, or image composition, we can evaluate the
reusability and robustness of slots obtained from different images. As shown
in Fig. 3(b), our task aims to generate novel images by not only combining but
also manipulating slots collected from various images.

Owing to the direct controllability, our model is capable of rendering objects
along with human intention by modifying the objects in accordance with
instructions that contain values for the desired change. As shown in the row
of Fig. 3(b), the number of slots for image composition (9 objects) can be
expanded beyond the quantity for which the model was originally trained (up to
6 objects). We attribute this to the characteristics of slots in Slot
Attention. Moreover, our model also can resolve the conflicts among multiple
images regarding the relative position (or depth) of the objects as
illustrated in the object 3, 5 and 8 in the third row in Fig. 3(b). From these
observations, we claim that the proposed method can effectively manipulate and
combine slots without sacrificing the original nature and robustness.

Table 1:  Results on object discovery. Two metrics, mean Intersection over Union (mIoU) and Adjusted Rand Index (ARI) are reported in % (mean $\pm$ std for 3 trials).  | mIoU ($\uparrow$) | ARI ($\uparrow$)
---|---|---
Previous methods |  |
SA (Locatello et al., 2020) | 47.3 $\pm$ 23.2 | 63.1 $\pm$ 54.5
+ARK (Kim et al., 2023) | 68.8 $\pm$ 00.4 | 95.4 $\pm$ 00.5
Ours (SlotAug) |  |
Base model (v1) | 68.9 $\pm$ 00.1 | 95.7 $\pm$ 00.2
\+ AIM (v2) | 68.5 $\pm$ 00.1 | 95.3 $\pm$ 00.1
\+ AIM + SCLoss (v3) | 68.5 $\pm$ 00.1 | 95.2 $\pm$ 00.7

Table 2:  Results on durability test with MSE on CLEVR6.  | Slot ($\downarrow$) | Obj. Pos. ($\downarrow$)
---|---|---
| Single step (x8)
v1 | 50.8 | 0.14
v2 | 39.7 | 0.15
v3 | 0.25 | 0.01
| Multiple steps (x4)
v1 | 54.0 | 0.16
v2 | 41.4 | 0.11
v3 | 0.31 | 0.02

###  4.2 Sustainability in Object Representation

####  4.2.1 Iterative manipulation

Fig. 4 shows the results of iterative manipulation applied to a specific
object along with a series of instructions including "Stay" referring to
$insts_{identity}$. We can observe that all our models succeed in manipulating
the target object, demonstrating that our proposed training scheme works
properly. Nevertheless, it is also clear that models v1 and v2 fail to follow
the instructions along the consecutive manipulations. In the case of v1,
object appearances deteriorate with the emergence of abnormal artifacts from
the third round. Whereas, in v2, although the collapsing issue is mitigated,
an out-of-interest property, color, changes despite no instruction for such
modification. These unexpected results are also triggered by the "Stay"
instruction, which is intended to maintain the current state of the object.
However, in the case of v3, we finally achieve optimal results that adhere to
the instructions, including "Stay". Based on these observations, we argue that
both AIM and SCLoss significantly contribute to sustainable controllability.

Figure 4:  Iterative slot manipulation.  The leftmost image is the initial
image from which the iterative manipulation begins. The text on each column
states the instruction used for manipulation. Each row shows the results of
the manipulation by v1, v2, and v3 models, respectively. Center areas are
cropped for better visibility.  Figure 5:  Durability test.  The leftmost
image is the initial image from which the test begins. The top three rows show
the results of the single-step tests where each model is instructed to
alternately move the target object up and down four times each. In the multi-
step test, as shown in the last row, the model performs two round-trip
manipulations, each involving moving the target object down, changing its
color, reverting the color, and returning the object to its original position.

####  4.2.2 Durabiltiy test

In the durability test, we evaluate how many manipulations a model can endure
while preserving object representation intact. Our durability test consists of
two types: single- and multi-step tests. In the single-step test, we
repeatedly manipulate slots with two instructions: one to modify a specific
object property and another to revert the object to its initial state. The
multi-step test involves a series of instructions to modify an object and
another series to restore it to its initial state.

As depicted in Fig. 5, our findings align with the previous experiment (Sec.
4.2.1). While v1 fails to keep the color after the second round and the color
gradually deviates as the rounds progress, v2 relatively preserves the color
well for the fourth round. Nevertheless, from the fifth round, the texture
progressively diverges from its original. Different from the v1 and v2, v3
demonstrates strong durability despite a greater number of manipulations.

We also perform quantitative evaluations on 100 randomly selected samples in
CLEVR6 to measure the intrinsic deformity of slots (Tab. 2). We conduct 8
single-step and 4 multi-step round trip manipulations, both resulting in a
total of 16 manipulations. We assess the durability test results by measuring
the difference, using L2 distance, between the original state and the
manipulated state for two aspects: the slot vector and object position vector.
Both qualitative and quantitative results lead us to that our model can
achieve better sustainability as the model evolves from v1 to v2 and v3.

###  4.3 Slot Space Analysis: Property Prediction

In addition to the pixel space analysis in the previous sections, we also
perform slot space analysis to comprehensively assess the effectiveness of our
method. We conduct property prediction to evaluate the quality of slots
concerning human-interpretable object properties such as size, color,
material, shape, and position. Through the property prediction, we can examine
how well slots are distributed in the slot space along the properties of
corresponding objects.

Figure 6:  t-SNE of slots on property prediction for color. The upper row is the results for the baseline model, SA + ARK, and the lower row is the results for our method, SlotAug. The first column is the result of the original slots obtained from the spatial binding process. The second and third columns are the results of the intermediate outputs from the first and second MLP layers of the property predictor, respectively. The results of t-SNE for other properties are shown in the Appendix.  Table 3:  Results of property prediction. Each column reports the F1 score (%) for predicting size, color, material, shape, and position, respectively. The numbers inside the parenthesis indicate the number of classes. For the position, we set two distance thresholds indicated as ‘pos@threshold’.  | size (2) | color (8) | material (2) | shape (3) |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---|---|---|---
SA + ARK | 69.7 | 63.5 | 70.4 | 59.1 | 71.1 | 51.8
SlotAug | 82.2 | 78.2 | 82.6 | 73.0 | 84.2 | 77.2

A property predictor, consisting of 3-layer MLPs, takes slots as input and
predicts a property of objects. Each property predictor is trained by
supervised learning using the ground truths. To investigate the effectiveness
of object representations learned through the proposed method, the OCL models
to produce slots are frozen during property prediction. As shown in Tab. 3,
our model outperforms the baseline method (Kim et al., 2023) across all
properties including those, like material and shape, that are not addressed
during training. Moreover, in Fig. 6, qualitative results using t-SNE (Van der
Maaten & Hinton, 2008) show that while the original slots themselves do not
appear to be well-clustered, slots obtained by SlotAug exhibit better
adaptability to the downstream task compared to those from the baseline model,
reinforcing the quantitative findings. Based on these results, we assert that
our method enhances interpretability not only in the pixel space but also in
the slot space.

##  5 Conclusion

We presented an OCL framework, SlotAug, for exploring the potential of
interpretable controllability in slots. To resolve the lack of labeled data,
we employed image augmentation for self-supervised learning of our model.
Moreover, we introduced a concept of sustainability in slots, achieved by the
proposed method AIM and SCLoss. We substantiated the effectiveness of our
methods by providing extensive empirical studies and theoretical evidence in
the Appendix. These empirical studies include pixel- and slot-space analyses
on tasks such as the durability test and property prediction. Though our work
remains several questions detailed in the Appendix and represents just one
step on a long journey of OCL, we firmly believe that our work is a
foundational piece in the field of interpretable OCL and propel the ongoing
effort to equip machines with human-like comprehension abilities.

##  Appendix A Limitations and Future Works

Advanced object manipulation and real-world datasets. In this research, we
leverage image augmentation techniques to create pseudo labels for object
manipulation at the image level, effectively addressing the absence of object-
level ground truths. Consequently, we gain control over objects within the
scope of augmentation-related properties, including attributes such as color,
position, and size. Nonetheless, relying solely on image augmentation to
generate supervision signals has its inherent limitations. These limitations
encompass factors like the diversity of target properties and the extent of
manipulation that can be effectively covered. Exploring avenues for
incorporating more informative open-source datasets, such as those used for
image captions, holds promise for manipulating object representations more
broadly. We will elaborate this potential in Section E.1.

Position insensitive representation. In our research, it is observed that the
Slot Attention algorithm generates slots exhibiting sensitivity to the
positioning of objects. Notably, this phenomenon persists even when we exclude
the soft positional encoding in the visual encoder. To achieve a more
interpretable object representation, exploring the generation of well-balanced
slots across various properties, rather than solely focusing on position,
shows potential for future work.

State-aware slot manipulation. In our slot manipulation process, the Property
Encoder encodes each property without taking into account the current state of
the manipulated slot. For instance, when modifying the size of an object, the
PropertyEncoder produces a property vector, inst_vec, irrespective of the
color or position of the target object. By incorporating the current state of
the target slot into the slot manipulation process, the precision and
complexity of the algorithm could be potentially enhanced.

##  Appendix B Preliminary: Spatial Binding in Slot Attention

Algorithm 2 Spatial binding in slot attention algorithm in pseudo-code format.
The input image is encoded into a set of $N$ vectors of dimension $D_{input}$
which is mapped to a set of $K$ vectors with dimension $D_{slot}$. Slots are
initialized from a Gaussian distribution with learned parameters
$\mu,\sigma\in\mathbb{R}^{D_{slot}}$. The number of iterations is set to
$T=3$.

1:function SpatialBinding($\texttt{img}\in\mathbb{R}^{H\times W\times 3}$)

2: $\texttt{inputs}=\texttt{Encoder(img)}$

3: $\texttt{inputs}=\texttt{LayerNorm(inputs)}$

4: for $t=0\ldots T$ do

5: $\texttt{slots\\_prev}=\texttt{slots}$

6: $\texttt{slots}=\texttt{LayerNorm(slots)}$

7:
$\texttt{attn}=\texttt{Softmax}(\frac{1}{\sqrt{D_{slot}}}k(\texttt{inputs})\cdot
q(\texttt{slots})^{T},\texttt{axis=`slots'})$

8: $\texttt{updates}=\texttt{WeightedMean(weights=attn+}\epsilon\texttt{,
values=}v\texttt{(inputs))}$

9: $\texttt{slots}=\texttt{GRU(state=slots\\_prev, inputs=updates)}$

10: $\texttt{slots}=\texttt{slots}+\texttt{MLP(LayerNorm(slots))}$

11: end for

12: return slots

13:end function

The core mechanism of the slot attention, the spatial binding, is described in
Alg. 2. Given an input image img $\in\mathbb{R}^{H\times W\times 3}$,
convolutional neural networks (CNNs) encoder generates a visual feature map
input $\in\mathbb{R}^{N\times D_{enc}}$, where $H$, $W$, $N$, and
$D_{\text{enc}}$ are the height and width of the input image, the number of
pixels in the input image ($=HW$), and the channel of the visual feature map.
The slot attention module takes slots and inputs, and projects them to
dimension $D_{slot}$ through linear transformations $k$, $q$, and $v$. Dot-
product attention is applied to generate an attention map, attn, with query-
wise normalized coefficients, enabling slots to compete for the most relevant
pixels of the visual feature map. The attention map coefficients weight the
projected visual feature map to produce updated slots, updates. With the
iterative mechanism of the slot attention module, the slots can gradually
refine their representations.

##  Appendix C Implementation and experimental details

###  C.1 Training

We use a single V100 GPU with 16GB of RAM with 1000 epochs and a batch size of
64. The training takes approximately 65 hours (wall-clock time) using 12GB of
RAM for the CLEVR6 dataset, and 22 hours using 9GB of RAM for the Tetrominoes
dataset, both with 16-bit precision.

###  C.2 Image Augmentation

Upon receiving an input image $img_{input}$, we produce four outputs: a
reference image, denoted as $img_{ref}$, an augmented image, represented as
$img_{aug}$, and the transformation instructions between them, indicated as
$insts_{ref2aug}$ and $insts_{aug2ref}$.

In the data augmentation process, three pivotal variables are defined. The
first is the template size $\mathcal{T}$, employed for the initial cropping of
$img_{input}$ prior to the application of transformation (240 for CLEVR6 and
80 for Tetrominoes). Next, the crop size $\mathcal{C}$ is used to crop the
transformed image before resizing it to $\mathcal{M}$ (192 for CLEVR6 and 64
for Tetrominoes). This two-stage cropping procedure mitigates the zero-padding
that results from transformations. Lastly, the image size $\mathcal{M}$
denotes the final image size post data augmentation (128 for CLEVR6 and 64 for
Tetrominoes).

In the training phase, $img_{ref}$ is obtained by applying a center-crop
operation on $img_{input}$ using $\mathcal{C}$ and then resizing it to
$\mathcal{M}$. The generation of $img_{aug}$ is more complex, entailing the
application of a random transformation from a set of three potential
transformations. Initially, $img_{input}$ is cropped using $\mathcal{T}$, and
the transformation process is implemented. Following this, the transformed
image is cropped by $\mathcal{C}$ and then resized to $\mathcal{M}$, yielding
$img_{aug}$. The detailed description for each transformation is as follows:

Translating. We set a maximum translation value
$d_{max}=\frac{\mathcal{T}-\mathcal{C}}{2}$. A value is randomly chosen within
the range of $(-d_{max},d_{max})$ for translation along the $x$-axis ($d_{x}$)
and the $y$-axis ($d_{y}$) respectively.

Scaling. The maximum and minimum scaling factors, $s_{max}$ and $s_{min}$, are
computed by $\frac{\mathcal{T}}{\mathcal{C}}$ and
$\frac{\mathcal{C}}{\mathcal{T}}$, respectively. A float value $s$, serving as
a scaling factor, is then randomly sampled from within the range of
$(s_{max},s_{min})$. One thing to note is that calculating the transformation
instructions is not straightforward due to the potential translation of
objects during scaling. Thus, to calibrate the instructions, we infer
translation values from the predicted object positions before scaling. The
position prediction is calculated as the weighted mean on the attention maps
between the visual encodings and slots. With this position prediction, we add
the translation term into the scaling process so that the model should perform
both object-level scaling and translating: $\vec{d}=(s-1)(\vec{p}-\vec{c})$,
where $\vec{d}$ represents the vector of the translation value, $\vec{p}$
refers to the vector of the predicted object position, and $\vec{c}$ is the
vector corresponding to the position of image center.

Color shifting. In this study, we employ the HSL (hue, saturation, and
lightness) color space for effective object color manipulation. The input
image, initially in RGB space, is converted to HSL space. We adjust the hue by
rotating it using randomly sampled angles that span the entire hue space. For
saturation, we apply a scaling factor, determined by the exponential of a
value randomly drawn from (-1, 1), a hyper parameter. Our primary focus lies
on the internal color of objects, leaving lightness untouched. Nonetheless,
adjustments to lightness can be made if necessary.

Instruction. Each transformation instruction is a list of 6 values: one
scaling factor $(\Lambda\text{scale})$, two translation parameters $(\Delta
x,\Delta y)$, and three color shifting parameters in HSL
$(\Delta\text{hue},\Lambda\text{saturation},\Lambda\text{lightness})$ where
$\Lambda$ and $\Delta$ means the multiplicative and additive factor for the
corresponding values, respectively. The identity instruction,
$insts_{identity}$, contains the base values for each transformation. Thus,
$insts_{identity}$ has $1$ for scaling, $(0,0)$ for translation, and $(0,1,1)$
for color shifting. For the inverse instruction , $insts_{aug2ref}$ has the
values of $-insts_{ref2aug}$ for additive factors, and
$\frac{1}{insts_{ref2aug}}$ for multiplicative factors.

###  C.3 Model

Basically, our model framework is built on Slot Attention (Locatello et al.,
2020), thereby the encoder, decoder, and slot attention module are the same as
that of Slot Attention except for the inclusion of the Attention Refining
Kernel (ARK) from SLASH (Kim et al., 2023). For Tetorminoes and CLEVR, we
employ a 4-layer CNN encoder and a 6-layer Spatial Broadcast (SB) decoder
(Watters et al., 2019) with a hidden dimension of 64. Within the slot
attention module, we set the slot dimension to 64, perform the binding process
for 3 iterations, and use a kernel size of 5 for the ARK. Please refer to the
original papers (Kim et al., 2023; Locatello et al., 2020) for additional
details for Slot Attention.

For CLEVRTEX6 and PTR datasets which include more complicated objects, we
adopt larger models with a slot dimension, $D_{slot}$, of 256. As encoders, we
use 1) Resnet34 (He et al., 2016) following (Elsayed et al., 2022; Biza et
al., 2023) and 2) ViT-base (Dosovitskiy et al., 2020), with the patch size of
8, pretrained via MAE (He et al., 2022) As decoders, we use an increased size
of SB decoder consisting of 8-layer CNNs with a hidden dimension of 128, and a
Transformer-based decoder proposed in Sajjadi et al. (2022b). The results of
using large models are described in Sec. E.

In Alg. 2 of the main paper, the Property Encoder (PropertyEncoder) takes as
input the values that correspond to specific properties. Accordingly, the
input size for the property encoder is 1 for scaling, 2 for translation, and 3
for color shifting. Each property is encoded via Property Encoder, a 3-layer
MLP with ReLU activation functions, resulting in a inst_vec, a vector of
dimension $D_{slot}$.

##  Appendix D From the Image-level Training to Object-level Inference

To begin with, we would like to highlight our unique approach to the training
procedure. While our training incorporates manipulations at the image-level,
it can be perceived as training the model at the individual object-level. In
this section, we discuss on how this transition is achieved without the need
for an additional tuning process, and present empirical results that support
our claim.

As we discussed shortly in Sec. 3.1. in the main paper, the success of
transitioning from image-level augmentation during training to object-level
manipulation during inference can be attributed primarily to the fact that the
entire process for each slot, including object discovery and decoding,
exclusively influences the reconstruction of its respective object. To
substantiate our claim, a mathematical proof is provided below to show how an
image-level reconstruction loss can be disentangled into object-level
reconstruction losses.

$\mathcal{L}_{\text{recon}}=\|\hat{\mathcal{I}}-\mathcal{I}\|_{2}^{2}$ (4)

$=\|\sum_{k=1}^{\mathcal{K}}(\hat{\mathcal{I}}_{k}^{rgb}\odot\hat{\mathcal{I}}_{k}^{\alpha})-\mathcal{I}\|_{2}^{2}$
(5)

$=\|\sum_{k=1}^{\mathcal{K}}(\hat{\mathcal{I}}_{k}^{rgb}\odot\hat{\mathcal{I}}_{k}^{\alpha})-\sum_{k=1}^{\mathcal{K}}(\mathcal{I}\odot\hat{\mathcal{I}}_{k}^{\alpha})\|_{2}^{2}$
(6)

$=\|\sum_{k=1}^{\mathcal{K}}(\hat{\mathcal{I}}_{k}^{rgb}\odot\hat{\mathcal{I}}_{k}^{\alpha}-\mathcal{I}\odot\hat{\mathcal{I}}_{k}^{\alpha})\|_{2}^{2}$
(7)

$\approx\|\sum_{k=1}^{\mathcal{K}}(\hat{\mathcal{O}}_{k}-\mathcal{O}_{k})\|_{2}^{2},$
(8)

$=\sum_{k=1}^{\mathcal{K}}\|(\hat{\mathcal{O}}_{k}-\mathcal{O}_{k})\|_{2}^{2}+\sum_{\begin{subarray}{c}i,j=1\\\
i\neq
j\end{subarray}}^{\mathcal{K}}(\hat{\mathcal{O}}_{i}\cdot\hat{\mathcal{O}}_{j}-2\
\hat{\mathcal{O}}_{i}\cdot\mathcal{O}_{j}+\mathcal{O}_{i}\cdot\mathcal{O}_{j})$
(9)

$\approx\sum_{k=1}^{\mathcal{K}}\|(\hat{\mathcal{O}}_{k}-\mathcal{O}_{k})\|_{2}^{2},$
(10)

where $\mathcal{K}$ is the number of slots,
$\hat{\mathcal{I}}\in\mathbb{R}^{H\times W\times 3}$ represents the
reconstructed image, and $\mathcal{I}\in\mathbb{R}^{H\times W\times 3}$
represents the input image. $\hat{\mathcal{I}}_{k}^{rgb}\in\mathbb{R}^{H\times
W\times 3}$ and $\hat{\mathcal{I}}_{k}^{\alpha}\in\mathbb{R}^{H\times W\times
1}$ are the reconstruction results generated by the decoder using the k-th
slot as input: an RGB and an alpha map (or an attention mask), respectively.
$\hat{\mathcal{O}}_{k}\in\mathbb{R}^{H\times W\times 3}$ is the predicted
image for the specific object that is bounded with the k-th slot, while
$\mathcal{O}_{k}\in\mathbb{R}^{H\times W\times 3}$ is the corresponding
ground-truth object image.

From Eq. (4) to Eq. (5), we follow the decoding process of Slot Attention
(Locatello et al., 2020). In particular, each k-th slot is decoded
independently, resulting in the reconstructed RGB image
$\hat{\mathcal{I}}_{k}^{rgb}$ and the reconstructed alpha map
$\hat{\mathcal{I}}_{k}^{\alpha}$. The final reconstruction image
$\hat{\mathcal{I}}$ is generated by aggregating $\hat{\mathcal{I}}_{k}^{rgb}$
using a pixel-level weighted average, where the weights are determined by
$\hat{\mathcal{I}}_{k}^{\alpha}$. It is crucial to recognize that
$\hat{\mathcal{I}}_{k}^{\alpha}$ serves as an attention mask, as elaborated
below:

$\sum_{k=1}^{\mathcal{K}}\hat{\mathcal{I}_{k}^{\alpha}}(x,y)=1\phantom{00}\text{for
all}\phantom{0}x,y,$ (11)

where $\hat{\mathcal{I}}_{k}^{\alpha}(x,y)$ is a value for the position
$(x,y)$. This characteristic plays a pivotal role in our approach,
facilitating the transition from Eq. (5) to Eq. (6). In this transformation,
the input image $\mathcal{I}$ is effectively weighted by the set of
$\mathcal{K}$ alpha maps, denoted as $\hat{\mathcal{I}}_{k}^{\alpha}$, where
$k$ spans from 1 to $\mathcal{K}$. Then, as both the first and second terms in
Eq. (6) involve the same sigma operations, we can simplify the expression by
combining the individual subtraction operations into a single sigma operation
(Eq. (7)).

Subsequently, we approximate Eq. (7) as Eq. (8) to get an object-level
disentangled version of the reconstruction loss. Here we assume that both
$\hat{\mathcal{O}}_{k}$ and $\mathcal{O}_{k}$ only consist of a specific
region of interest within the input image. This region corresponds to the
target object which is bound to the $k$-th slot, while the remaining areas are
masked out and assigned a value of zero. We can make this assumption based on
the successful performance of the previous object-centric learning model,
SLASH (Kim et al., 2023). SLASH has demonstrated effective capabilities in
focusing on and capturing specific objects of interest within an image, by
introducing the Attention Refining Kernel (ARK). By incorporating ARK into our
model, we confidently assume that $\hat{\mathcal{O}}_{k}$ and
$\mathcal{O}_{k}$ primarily represent the target object while masking out
other irrelevant parts as zero as shown in Fig. 7.

Here, Eq. (8) can be broken down into two separate summations. The first one
is our target term that is the sum of object-level MSE losses, and the second
term is the residual term. Lastly, the transition from Eq. (9) to Eq. (10)
constitutes a significant simplification in the representation of the loss
function. This is a valid transformation under the assumption follows:

$\hat{\mathcal{O}}_{i}\cdot\hat{\mathcal{O}}_{j}=\hat{\mathcal{O}}_{i}\cdot\mathcal{O}_{j}=\mathcal{O}_{i}\cdot\mathcal{O}_{j}=0\phantom{00}\text{if}\phantom{0}i\neq
j.$ (12)

This assumption postulates that the inner product of different object images,
whether they are predicted or ground-truth, is always zero. We assert that
this assumption is justifiable, much like the previous one, given the
promising results obtained in our object discovery experiments. The loss
computation is thus decomposed into individual components for each slot, which
lends itself to an interpretation of object-level loss.

The conversion from image-level MSE loss to a sum of individual object-level
MSE losses provides a new perspective on our training method. Despite the use
of image-level manipulations, the underlying core of the training process
inherently engages with object-level representations. This demonstrates how a
simple methodological addition, incorporating image augmentation into the
training process, can lead to considerable gains in the model’s capacity for
user-intention-based object manipulation.

Fig. 7 empirically demonstrates the effectiveness of our model, leveraging
Slot Attention for controllability over slots. Conversely, it was noted that
the well-known alternative framework for object-centric learning, SLATE (Singh
et al., 2021), employing image tokenization from Discrete VAE (dVAE) (Im Im et
al., 2017) and Transformer-based auto-regressive decoding (Vaswani et al.,
2017), struggled with the manipulation of slots, as illustrated in Fig. 8. The
same slot manipulation strategy via Property Encoder was used for comparison.
Other training environments are just the same as the official paper (Singh et
al., 2021) except for the addition of the training loss for the reconstruction
of the augmented images.

##  Appendix E Further experimental results

More qualitative results on object-level manipulation using various backbone
encoders (basic CNN, Resnet34, and ViT-base) and decoders (basic SB are shown
in Fig. 9 and Fig. 10). Furthermore, we showcase several additional
experimental results below.

###  E.1 Fully Supervised Training on Materials and Shapes

To explore the capabilities of our method when provided with human-annotated
labels, we demonstrate object manipulation examples related to materials and
shapes. We utilize the CLEVR render
111https://github.com/facebookresearch/clevr-dataset-gen to generate datasets
having ground truth in terms of property modification. Using datasets
containing precise annotations for the target properties (materials and
shapes), we can explicitly train a model through object-level supervision.

As shown in Fig. 11, the model can effectively acquire knowledge of extrinsic
properties, such as material and shape, when provided with appropriate
supervision signals. In future work, one can aim to enhance the proposed
training scheme by leveraging more informative datasets, such as those for
image captioning (Chen et al., 2015; Wang et al., 2023a), to train a more
human-interactive framework. This process may entail elaborate data processing
since the datasets were not originally designed for the purposes of object
manipulation. However, we firmly believe that pursuing this path holds great
promise.

###  E.2 Quantitative Evaluation on Object Manipulation

Table 4:  Results of object-level manipulation on the rendered dataset. We evaluate the object-level manipulation by assessing metric scores over three sorts of images: reference images (ref.); manipulated images (manip.); and reversion or restored images (rev.). We use mIoU and ARI for the object discovery task, and MSE for the image generation task. | mIoU |  | ARI |  | MSE
---|---|---|---|---|---
| ref. | manip. | rev. |  | ref. | manip. | rev. |  | ref. | manip. | rev.
v1 | 89.4 | 69.9 | 68.9 |  | 97.9 | 79.3 | 80.1 |  | 7.3e-4 | 4.6e-3 | 6.3e-3
v2 | 87.9 | 71.4 | 80.0 |  | 96.1 | 79.4 | 90.2 |  | 7.5e-4 | 4.2e-3 | 3.1e-3
v3 | 85.4 | 70.5 | 79.8 |  | 95.3 | 78.8 | 89.8 |  | 7.8e-4 | 3.5e-3 | 2.1e-3

The supplementary quantitative results on object manipulation are shown in
Tab. 4. Given the absence of an evaluation benchmark dataset, we opt to employ
the same CLEVR render as described in Sec. E.1, to generate a set of 1500
triplets. These triplets consist of a reference image, instructions for object
manipulation, and the resulting manipulated image. Additionally, it is worth
noting that there exists no prior research specifically addressing slot
manipulation through direct human-interpretable instructions. Consequently,
our performance comparisons are restricted to different versions of our model:
v1 (the base model with image augmentation only), v2 (image augmentation +
AIM), and v3 (image augmentation + AIM + SCLoss).

One can observe that all three models successfully execute object discovery
and object manipulation tasks with minimal differences in their performance
scores, highlighting the effectiveness of our training approach leveraging
image augmentation. However, in the context of the reversion task, wherein the
models are instructed to revert the manipulated objects to their original
state, both v2 and v3 outperform v1, demonstrating the effectiveness of the
proposed AIM. Furthermore, in terms of image editing, v3 surpasses both v1 and
v2 by a significant margin, underscoring the effectiveness of the proposed
SCLoss.

###  E.3 Ablation Study on Loss Weihgts

We conduct an ablation study on training losses using the v3 model (image
augmentation + AIM + SCLoss). Tab. 5 shows the results of training models with
three different loss weights for the SCLoss ($w_{cycle}$) while maintaining
the $w_{recon}$ as 1.0. For the balanced training result, considering both
image reconstruction and object discovery, we opted for 0.1 due to its
balanced performance.

Table 5:  Results of ablation studies on weights for the training loss. The leftmost column shows the values of the weight for SC-Loss ($w_{cycle}$), while the weight for the reconstruction loss ($w_{recon}$)is set to $1.0$. The other columns show the training losses when training is finished and the scores of the validation metrics for the object discovery task. Each row shows the results of using the weight for SC-Loss with $1.0$, $0.1$, and $0.01$, respectively.  $w_{cycle}$ | Train | Val
---|---|---
loss_recon_ref | loss_recon_aug | loss_cycle | mIoU | ARI
1.0* | 3.5e-4 | 4.9e-4 | 2.2e-6 | 66.5 | 94.1
0.1 | 3.2e-4 | 3.8e-4 | 1.8e-5 | 68.5 | 95.2
0.01 | 3.1e-4 | 4.4e-4 | 6.2e-5 | 68.7 | 95.4

###  E.4 Extreme Durability Test

We evaluate our v3 model with two stringent versions of the durability test.
Fig. 12 displays the first extreme durability test, wherein we manipulate all
objects within a given scene across a total of 24 manipulation steps. The
complete manipulation process encompasses four cycles of round-trip
manipulations, each cycle comprising three sequential forward manipulations
followed by three recovery manipulations. Despite the fact that the appearance
of each object tends to deviate from its initial state as manipulations
accumulate, it is notable that our model demonstrates substantial robustness
against multiple rounds of manipulations.

In the second durability test, we manipulate a target object through 50 steps
of manipulation, which consists of 25 cycles of a single forward manipulation
(translation, scaling, or color shifting) and its corresponding recovery
manipulation. Fig. 13 demonstrates our model’s significant endurance against
numerous manipulation steps. We observe that our model exhibits greater
robustness in translating objects compared to scaling and color shifting.
However, it is noteworthy that the resilience of our model against both
scaling and color shifting is impressive, as it withstands around 20 steps of
manipulations without significant distortion in object appearance. The figure
further includes qualitative results that gauge slot divergence. It becomes
clear that our model’s durability improves gradually as it evolves from
version v1 to v2 and finally to v3.

###  E.5 Toy Application: Object-centric Image Retrieval

With the acquisition of object-level controllability, we can extend object-
centric learning to a newly introduced downstream task, called object-centric
image retrieval. Object-centric image retrieval aims to retrieve an image
having an object that is most relevant to a target object that is given by the
user’s intention.

The retrieval process is as follows. First, we acquire slots for a target
object and candidate objects from the corresponding images by conducting
object discovery. Then, to reduce the effect of spatial properties such as
object position or size, we neutralize the slots by performing slot
manipulation with the instructions for moving the objects to the central
position and for scaling the objects to the unified size. After
neutralization, we generate a object image by decoding a neutralized slot. The
relevance scores between the target object image and candidate object images
are computed along the given metric, specifically the MSE. Finally, object-
centric image retrieval can be accomplished by finding the image containing
the top-k objects as shown in Fig. 14.

###  E.6 Additional t-SNE Results

Additional t-SNE results from the property prediction are shown in Fig. 15.
Similar to the result on the color property in the main paper, we can observe
that the proposed SlotAug produces more well-clustered slots in the earlier
layer in the property predictors.

0

Figure 7:  Training results of our method. The leftmost column is the
reference images, $img_{ref}$. The second leftmost column is the
reconstruction of the reference images, $recon_{ref}$. The middle columns show
the object discovery results where each column corresponds to a single slot in
$slots_{ref}$. The second rightmost column is the augmented images,
$img_{aug}$. The rightmost column is the reconstruction of the augmented
images, $recon_{aug}$.  Figure 8:  Training results of SLATE (Singh et al.,
2021) for slot manipulation. The leftmost column is the reference images,
$img_{ref}$; The second leftmost column is the reconstruction of the reference
images, $recon_{ref}$. The middle columns show the object discovery results
where each column corresponds to a single slot in $slots_{ref}$. The second
rightmost column is the augmented images, $img_{aug}$. The rightmost column is
the reconstruction of the augmented images, $recon_{aug}$.  Figure 9:
Visualization of object manipulation in CLEVR6 and Tetrominoes dataset. The
leftmost column features the initial images, serving as the starting point for
the manipulation process. The subsequent columns depict the results of object-
level manipulation, following the instructions represented as text above the
images.  Figure 10:  Visualization of object manipulation in CLEVRTEX and PTR
with larger encoder and decoder. The first three rows are for CLEVRTEX and the
last is for PTR dataset. Regarding the encoder, we initialize ResNet34
randomly, while ViT is pre-trained using MAE (He et al., 2022) on the target
datasets. For the decoder, we employ both the Spatial Broadcast (SB) decoder
and the SRT decoder (Sajjadi et al., 2022b). In case (b), we enhance the size
of the SB decoder with a hidden dimension of 128 and a depth of 8.
Additionally, for SRT, we adopt a slot-wise decoding strategy akin to the SB
decoder.  Figure 11:  Results of manipulating materials and shapes of objects.
For this experiment, the training datasets are crafted using the CLEVR
renderer, wherein we modify target properties such as materials and shapes
while keeping other properties unchanged.  Figure 12:  The results of the
durability test are depicted in the figure, wherein all objects in the scene
are manipulated in accordance with the instructions, provided as text at the
top of the figure. The leftmost column presents the initial state of the
image. The subsequent three columns comprise three distinct manipulations:
translation, scaling, and color shifting. The right three columns feature
three recovery manipulations intended to restore the image to its original
state. We perform 4 cycles of these round trip processes, leading to a total
of 24 manipulations.  Figure 13:  The results of the single-step durability
test. The top image represents the initial state prior to any manipulation.
Each column depicts the results of the single-step durability test with
translation, scaling, and color shifting, respectively. The left images in
each column illustrate the outcome of the manipulation corresponding to the
column name, while the right images in each column display the results of the
recovery, or inverse, manipulation. Each row represents the results after a
series of manipulations, with the number of manipulations corresponding to the
row number.  Figure 14:  Visualization of object-centric image retrieval. The
top row displays query images, indicating the target objects to be retrieved.
Below each query image, you can find the top 5 retrieval results. Each
retrieval result consists of the original image on the left and an attention
map on the right. The attention map, associated with the slot corresponding to
the target object, emphasizes the specific region within the image.  Figure
15:  The further t-SNE results from the property prediction. We visualize
t-SNE for three additional properties: (a) materials (2 types), (b) shape (3
types), and size (2 types).

## References

  * Baldi (2012) Pierre Baldi.  Autoencoders, unsupervised learning, and deep architectures.  In _Proceedings of ICML workshop on unsupervised and transfer learning_ , pp. 37–49. JMLR Workshop and Conference Proceedings, 2012.
  * Biza et al. (2023) Ondrej Biza, Sjoerd van Steenkiste, Mehdi SM Sajjadi, Gamaleldin F Elsayed, Aravindh Mahendran, and Thomas Kipf.  Invariant slot attention: Object discovery with slot-centric reference frames.  _arXiv preprint arXiv:2302.04973_ , 2023.
  * Burgess et al. (2019) Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner.  Monet: Unsupervised scene decomposition and representation.  _arXiv preprint arXiv:1901.11390_ , 2019.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Cho et al. (2015) Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce.  Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1201–1210, 2015.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Elsayed et al. (2022) Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf.  Savi++: Towards end-to-end object-centric learning from real-world videos.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 35:28940–28954, 2022.
  * Engelcke et al. (2019) Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner.  Genesis: Generative scene inference and sampling with object-centric latent representations.  _arXiv preprint arXiv:1907.13052_ , 2019.
  * Engelcke et al. (2021) Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner.  Genesis-v2: Inferring unordered object representations without iterative refinement.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 34:8085–8094, 2021.
  * Feldman (2013) Jerome Feldman.  The neural binding problem (s).  _Cognitive neurodynamics_ , 7:1–11, 2013.
  * Fischler & Elschlager (1973) Martin A Fischler and Robert A Elschlager.  The representation and matching of pictorial structures.  _IEEE Transactions on computers_ , 100(1):67–92, 1973.
  * Greff et al. (2016) Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber.  Tagger: Deep unsupervised perceptual grouping.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 29, 2016.
  * Greff et al. (2019) Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner.  Multi-object representation learning with iterative variational inference.  In _International Conference on Machine Learning (ICML)_ , pp. 2424–2433. PMLR, 2019.
  * Greff et al. (2020) Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber.  On the binding problem in artificial neural networks.  _arXiv preprint arXiv:2012.05208_ , 2020.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Hong et al. (2021) Yining Hong, Li Yi, Joshua B Tenenbaum, Antonio Torralba, and Chuang Gan.  Ptr: A benchmark for part-based conceptual, relational, and physical reasoning.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Im Im et al. (2017) Daniel Im Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio.  Denoising criterion for variational auto-encoding framework.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 31, 2017.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2901–2910, 2017.
  * Karazija et al. (2021) Laurynas Karazija, Iro Laina, and Christian Rupprecht.  Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Kim et al. (2023) Jinwoo Kim, Janghyuk Choi, Ho-Jin Choi, and Seon Joo Kim.  Shepherding slots to objects: Towards stable and robust object-centric learning.  _arXiv preprint arXiv:2303.17842_ , 2023.
  * Kingma & Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kipf et al. (2021) Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff.  Conditional object-centric learning from video.  _arXiv preprint arXiv:2111.12594_ , 2021.
  * Kuhn (1955) Harold W Kuhn.  The hungarian method for the assignment problem.  _Naval research logistics quarterly_ , 1955.
  * Lake et al. (2017) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman.  Building machines that learn and think like people.  _Behavioral and brain sciences_ , 40:e253, 2017.
  * Li et al. (2020) Nanbo Li, Cian Eastwood, and Robert Fisher.  Learning object-centric representations of multi-object scenes from multiple views.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 33:5656–5666, 2020.
  * Locatello et al. (2020) Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric learning with slot attention.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 33:11525–11538, 2020.
  * Loshchilov & Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Makhzani et al. (2015) Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey.  Adversarial autoencoders.  _arXiv preprint arXiv:1511.05644_ , 2015.
  * Mao et al. (2019) Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu.  The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision.  _arXiv preprint arXiv:1904.12584_ , 2019.
  * Marr (2010) David Marr.  _Vision: A computational investigation into the human representation and processing of visual information_.  MIT press, 2010.
  * Rishabh et al. (2019) Kabra Rishabh, Burgess Chris, Matthey Loic, LopezKaufman Raphael, Greff Klaus, Reynolds Malcolm, and Lerchner Alexander.  Multi-object datasets.  _https://github.com/deepmind/multi_object_datasets/_ , 2019.
  * Sajjadi et al. (2022a) Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf.  Object scene representation transformer.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 35:9512–9524, 2022a.
  * Sajjadi et al. (2022b) Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al.  Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 6229–6238, 2022b.
  * Seitzer et al. (2022) Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al.  Bridging the gap to real-world object-centric learning.  _arXiv preprint arXiv:2209.14860_ , 2022.
  * Singh et al. (2021) Gautam Singh, Fei Deng, and Sungjin Ahn.  Illiterate dall-e learns to compose.  _arXiv preprint arXiv:2110.11405_ , 2021.
  * Singh et al. (2022) Gautam Singh, Yi-Fu Wu, and Sungjin Ahn.  Simple unsupervised object-centric learning for complex and naturalistic videos.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 35:18181–18196, 2022.
  * Singh et al. (2023) Gautam Singh, Yeongbin Kim, and Sungjin Ahn.  Neural systematic binder.  In _International Conference on Learning Representations (ICLR)_ , 2023.
  * Smith et al. (2022) Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Fredo Durand, Joshua B Tenenbaum, Jiajun Wu, and Vincent Sitzmann.  Unsupervised discovery and composition of object light fields.  _arXiv preprint arXiv:2205.03923_ , 2022.
  * Spelke & Kinzler (2007) Elizabeth S Spelke and Katherine D Kinzler.  Core knowledge.  _Developmental science_ , 10(1):89–96, 2007.
  * Tenenbaum et al. (2011) Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman.  How to grow a mind: Statistics, structure, and abstraction.  _Science_ , 331(6022):1279–1285, 2011.
  * Treisman (1996) Anne Treisman.  The binding problem.  _Current opinion in neurobiology_ , 6(2):171–178, 1996.
  * Van der Maaten & Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 30, 2017.
  * Wang et al. (2023a) Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al.  The all-seeing project: Towards panoptic visual recognition and understanding of the open world.  _arXiv preprint arXiv:2308.01907_ , 2023a.
  * Wang et al. (2023b) Yanbo Wang, Letao Liu, and Justin Dauwels.  Slot-vae: Object-centric scene generation with slot attention.  In _International Conference on Machine Learning (ICML)_ , 2023b.
  * Watters et al. (2019) Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner.  Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes.  _arXiv preprint arXiv:1901.07017_ , 2019.
  * Wu et al. (2022) Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg.  Slotformer: Unsupervised visual dynamics simulation with object-centric models.  _arXiv preprint arXiv:2210.05861_ , 2022.
  * Xie et al. (2022) Sirui Xie, Ari S Morcos, Song-Chun Zhu, and Ramakrishna Vedantam.  Coat: Measuring object compositionality in emergent representations.  In _International Conference on Machine Learning (ICML)_ , pp. 24388–24413. PMLR, 2022.