# Refine and Represent: Region-to-Object Representation Learning

Akash Gokul  Konstantinos Kallidromitis  Shufan Li
Yusuke Kato  Digital and AI Technology Center, Technology Division, Panasonic
Holdings Co.  Kazuki Kozuka  Digital and AI Technology Center, Technology
Division, Panasonic Holdings Co.  Trevor Darrell  Berkeley AI Research
Colorado J Reed  Berkeley AI Research

###### Abstract

Recent works in self-supervised learning have demonstrated strong performance
on scene-level dense prediction tasks by pretraining with object-centric or
region-based correspondence objectives. In this paper, we present Region-to-
Object Representation Learning (R2O) which unifies region-based and object-
centric pretraining. R2O operates by training an encoder to dynamically refine
region-based segments into object-centric masks and then jointly learns
representations of the contents within the mask. R2O uses a region refinement
module to group small image regions, generated using a region-level prior,
into larger regions which tend to correspond to objects by clustering region-
level features. As pretraining progresses, R2O follows a region-to-object
curriculum which encourages learning region-level features early on and
gradually progresses to train object-centric representations. Representations
learned using R2O lead to state-of-the art performance in semantic
segmentation for PASCAL VOC ($+0.7$ mIOU) and Cityscapes ($+0.4$ mIOU) and
instance segmentation on MS COCO ($+0.3$ APmk). Further, after pretraining on
ImageNet, R2O pretrained models are able to surpass existing state-of-the-art
in unsupervised object segmentation on the Caltech-UCSD Birds 200-2011 dataset
($+2.9$ mIoU) without any further training. We provide the code/models from
this work at https://github.com/KKallidromitis/r2o.

##  1 Introduction

Recent progress in self-supervised pretraining in computer vision has led to a
wide range of improvements on transfer learning tasks such as image
classification, object detection, and segmentation [8, 6, 46, 28]. Instance
discrimination pretext objectives have driven many of these advancements,
whereby contrasting augmented views of an input image, a network learns
representations that are invariant to predefined image-level data
augmentations. However, much of this work has focused on scene-level
pretraining, which as recent works have shown, is not optimal for many pixel-
level prediction tasks, such as semantic segmentation [56, 63, 28].

Many state-of-the-art methods have extended image-level pretraining to learn
local features by leveraging either region-based [56, 60] or object-centric
similarity tasks [28, 2]. Region-based pretraining methods enforce predictive
invariance between random image patches that are present in both views and
thus learn from all regions in an image, regardless of size or content.
Similarly, object-centric pretraining uses segmentation heuristics to localize
objects and enforces representational invariance to the features corresponding
to the objects in the scene. Both types of representation learning have led to
improvements in downstream dense prediction tasks [28, 2, 57, 65].

In this paper, we find that pretraining for dense prediction tasks benefits
from both region-based and object-centric objectives at different stages of
the pretraining process. Specifically, we present Region-to-Object
representation learning (R2O): a framework that learns region-based and
object-centric features by dynamically refining segmentation masks throughout
pretraining and identifying dense representations in these segmented regions.
R2O uses a region refinement module to transform the predicted masks from
multiple small scale image regions that capture consistency in color and
spatial position, to larger, object-centric regions. Predicted masks are then
used to enforce representational invariance for the features corresponding to
the discovered segments.

Finally, we introduce a region-to-object curriculum, which gradually reduces
the number of segments within predicted masks, in order to encourage region-
level pretraining for small image regions early on and increasingly object-
centric representation learning as pretraining progresses (see Figure 1).

We study R2O on a range of datasets and tasks, including object detection,
instance segmentation, and semantic segmentation. Our contributions are
summarized as follows:

  * •

We present R2O, which unifies region-based and object-centric instance
discrimination pretraining by training an encoder to dynamically refine
region-based segments into object-centric masks and then jointly learns
representations of the contents within the mask.

  * •

We introduce a region-to-object pretraining curriculum for R2O that begins by
training local features corresponding to simple image regions, e.g. adjacent
pixels sharing similar color values, and gradually progresses to learn object
centric features.

  * •

R2O pretraining improves transfer performance in tasks such as object
detection and instance segmentation on MS COCO ($+0.3$ APmk) as well as
exceeding previous methods in PASCAL VOC ($+0.7$ mIOU) and Cityscapes ($+0.4$
mIOU) semantic segmentation. R2O further improves state-of-the-art
unsupervised segmentation performance on Caltech-UCSD Birds 200-2011
(CUB-200-2011) by $+2.9$ mIoU despite not finetuning on CUB-200-2011.

Figure 1: R2O unifies region-based and object-centric pretraining by jointly
learning to discover and represent regions and objects. Our region refinement
process allows object discovery using scheduled K-means clustering over the
embeddings of neighboring regions. Following this schedule, masks early in
training consist of random image segments ($K=128$) which gradually develop
into object centered masks ($K=4$). By the end of pretraining, R2O enables
object-centric discovery and representation without requiring advanced
segmentation algorithms to localize objects in a scene.

##  2 Related Works

#### Image-level Self-Supervised Pretraining

Self-supervised learning algorithms aim to learn representations from
unlabelled data that are effective for a desired task, such as a transfer task
that may include a limited number of labels [3]. Early works in discriminative
visual representation learning focused on training encoders to predict the
position of an image patch [17, 41], predict the angle of rotation [21], and
predict the colors in a grayscale image [67]. Current methods have shifted
away from explicit rotation or color predicition, and instead, train an
encoder to learn representations that are invariant to data augmentations [26,
8, 23, 9, 6, 45, 40]. This form of pretraining, known as instance
discrimination pretraining, is often achived through contrastive learning [43]
or a teacher-student dual-network formulation[23]. The contrastive objective
encourages representations from the same image to be invariant to data
augmentations and dissimilar to representations from other images (commonly
referred to as negatives). Unlike contrastive objectives, Siamese
representational learning methods do not rely on negatives and are thus less
reliant on batch size [23, 9].

#### Region-Based and Object-Centric Self-Supervised Pretraining

Instance discrimination pretraining has primarily focused on learning image-
level representations. While this is suitable for downstream tasks such as
image classification, learning local or object-centric features during
pretraining is critical for tasks such as object detection and semantic
segmentation. Recent works [56, 63, 60, 57, 31, 42, 2, 64, 61] have extended
the instance discrimination paradigm to concurrently learn region-based
features. These methods enforce representational invariance to image regions
in both views without taking into consideration the content in these regions.
Examples include DenseCL [56], which jointly trains the local features be
invariant across views while performing the standard image-level
representation training, and ReSim [60], which performs region-level
representation learning by sliding a window across the image regions present
in both views and enforcing region-based representational similarity.

Xie et al. [63] go beyond simple feature-to-feature matching and incorporate
spatial sensitivity when matching local features that are close in the
embedding space. Our work extends region-based pretraining by refining small
image regions into object centric masks over training. Concurrent to this line
of work are methods [28, 57, 2, 65, 53] that use heuristics to locate objects
and enforce representational invariance for the features corresponding to
these objects [19, 51]. The closest related work is DetCon [28], which uses
unsupervised segmentation algorithms to train object-level representations. In
contrast, our method is able to discover and represent objects by refining
small image regions instead of relying on advanced segmentation heuristics to
localize objects. Other works in this area [57, 53, 2, 65] focus on training
object-centric features for a specific task, such as object detection,
semantic segmentation, and salient object detection, and do not train a
network to learn features that can transfer to multiple tasks. Concurrent to
our work, Hénaff et al. [29] and Wen et al. [58] propose methods to directly
discover objects, using K-means or learned semantic prototypes [37], and
represent them without incorporating object-level heuristics or region-based
pretraining.

#### Region Clustering

In this work, we use clustering to refine small image regions into object-
centric masks. Clustering has a long history in unsupervised semantic
segmentation [19, 1, 16, 30, 14, 13]. This trend has continued in the era of
deep learning. Current methods utilize clustering assignments in order to
learn spatially and semantically consistent embeddings [35, 12, 32, 34].
Kanezaki [35], Hwang et al. [32] train pixelwise embeddings by clustering
these embeddings and using the cluster assignments as pseudo labels. Cho et
al. [12] and Ji et al. [34] train embedding using a similarity loss and task-
specific data augmentations. Our work does not focus on the task of
unsupervised semantic segmentation, and instead, focuses on using clustering
to localize objects for learning transferable representations. Our method uses
K-means clustering [36] due to its simplicity and effectiveness.

##  3 Method

We present R2O, a self-supervised pretraining method that enables region-level
and object-centric representation learning. At a high level, R2O transforms
region-level image priors into object-centric masks through a region
refinement module and encourages representational invariance for the features
corresponding to the contents of the discovered masks (see Figure 2). The
training process begins by passing an image through the encoder as well as
clustering the image into fine-grained regions using a simple algorithm such
as SLIC [1]. Next, features for each region are pooled and clustered using
K-means to create masks. A high number of clusters, $K$, leads to masks that
group small image regions together. Conversely, we have found that using a
lower number of clusters leads to neighboring image regions that are grouped
together in order to form semantically meaningful segmentations which tend to
correspond to objects. R2O then promotes representational similarity for the
features corresponding to the refined masks, by aligning the features for
refined regions across different views of the image. Finally, a region-to-
object curriculum enables region-level and object-centric representation
learning by gradually progressing masks from oversegmented regional fragments
to accurate object-centric masks (Figure 1).

Figure 2:  R2O architecture. R2O consists of two steps: (1) Region Refinement,
where small image regions (given by a region-level prior) are transformed into
object-centric masks and (2) Representation Learning, where object-level
representations are learned. Step 1 produces object-centric segmentations by
computing features for each region given by the region-level prior and
performing K-means clustering on these region-level embeddings. Step 2
encourages representation similarity for the features corresponding to the
refined mask in each view. R2O also uses a region-to-object curriculum which
gradually decrease the number of clusters ($K$) during region refinement from
a very high value ($K=128$) to a low value ($K=4$), which as we show, enables
region-based pretraining early on during training and slowly evolves to
object-centric pretraining.

#### Formulation

We formulate the pretraining objective as a bilevel optimization problem where
the upper-level objective refines region-level priors into object-centric
masks (Section 3.1) and the lower-level objective optimizes representational
invariance for mask-level features (Section 3.2). Let $X$ refer to a given
image and $R$ be the region-level prior for the image (e.g. SLIC superpixels).
The outer loop computes a cluster assignment, $M$, which is used to segment
the image $X$, according to a mask-level objective
($\mathcal{L_{\text{mask}}}$) given the representation $f_{\theta}(X)$ and the
region-level prior $R$ (Eq. 1). The inner loop optimizes the parameters,
$\theta$, of the encoder, $f_{\theta}$, following any representation learning
objective ($\mathcal{L_{\text{repr}}}$) with respect to the segmentations
created using $M$ (Eq. 2).

$\displaystyle\min_{M}\quad$
$\displaystyle\mathcal{L_{\text{mask}}}(M;f_{\theta^{*}}(X),R)$ (1) s.t.
$\displaystyle\theta^{*}=\operatorname*{argmin}_{\theta}\mathcal{L_{\text{repr}}}(f_{\theta}(X);M)$
(2)

###  3.1 Object-Centric Segmentation via Region Refinement

Our Region Refinement step refines a region-level prior, such as SLIC
superpixels, into object-centric segmentations by clustering region-level
features. Specifically, we define an object-centric segmentation as a set of
clusters, $M$, which satisfy criteria such as color consistency, spatial
proximity, and representational similarity. Region-level priors provide
segmentations which satisfy image-level criteria, such as color consistency,
and can be derived solely from a given image. However, as illustrated in
Figure 2, these simple region-level priors do not accurately segment objects
in the scene, but rather, group together small neighboring regions. Recent
works have shown that representations learned from self-supervised pretraining
can encode object-level semantics [66, 62, 52]. Thus, we incorporate the
features learned during pretraining as part of our clustering objective in
order to transform our region-level prior into object-centric segmentations.

We approximate the solution to the upper-level optimization problem (Eq. 1) by
performing K-Means clustering on the set of region-level features to discover
an object-centric segmentation. For each region, we compute an embedding,
$p_{i}$, using a mask-pooling operation, where each region has a corresponding
binary mask that selects the corresponding features in the convolutional
feature map. This generates a set of region-level embeddings
$P=\\{p_{1},...,p_{|R|}\\}$ where $|R|$ is the number of distinct regions
generated by the region-level prior $R$. Next, we perform the refinement step
shown in Figure 2 by applying K-Means clustering to the region-level
embeddings in $P$ in order to compute clusters $M=\\{M_{1},...,M_{K}\\}$ (Eq.
3). Below, we denote $|M_{i}|$ as the number of points assigned to cluster
$M_{i}$ and $\mu_{i}$ is the mean of the points in $M_{i}$. Given cluster
assignments $M$, we produce an object-centric segmentation, $m$, by assigning
each pixel in the representation $f_{\theta}(X)\in\mathbb{R}^{H\times W\times
D}$ to its nearest cluster center (see Mask in Figure 2).

$\mathcal{L_{\text{mask}}}(M;f_{\theta}(X),R)=\frac{1}{K}\sum_{k=1}^{K}\frac{1}{|M_{k}|}\sum_{p\in
M_{k}}||p-\mu_{k}||^{2}\\\ $ (3)

###  3.2 Object-Centric Representation Learning

Our representation learning process is similar to the BYOL pipeline [23].
Since optimizing this inner objective to convergence can be expensive, we
approximate $\theta^{*}$ by minimizing
$\mathcal{L_{\text{repr}}}(f_{\theta}(X);M)$ for one epoch using mini-batch
gradient descent.

#### Setup

Given an image $X$, we create two views: $x_{1}=\mathcal{T}_{\text{aug}}(X)$,
$x_{2}=\mathcal{T}_{\text{aug}}(X)$, where $\mathcal{T}_{\text{aug}}$ is any
image augmentation policy. We use $m$ to obtain $m_{1}$, $m_{2}$ corresponding
to views $x_{1}$, $x_{2}$. Further, we use a Siamese architecture with two
similar networks: an online encoder $f_{\theta}$ (denoted by parameters
$\theta$) and a target encoder $f_{\xi}$ (denoted by parameters $\xi$). The
target network is not trained directly, and is updated using an exponential
moving average of the online network’s parameters.

#### Pretraining

We begin by computing $f_{\theta}(x_{1})$,
$f_{\xi}(x_{2})\in\mathbb{R}^{H\times W\times D}$. Next, we replace the
commonly used global-pooling with mask-pooling [28]. This operation generates
the average embeddings within the view-specific masks $m_{1}$ and $m_{2}$. We
compute $h_{\theta,1}$ and $h_{\xi,2}$ by applying mask-pooling to
$f_{\theta,1}$ and $f_{\xi,2}$. Following the Siamese architecture, we compute
$z_{\theta,1}$ and $z_{\xi,2}$ by forward-passing $h_{\theta,1}$, $h_{\xi,2}$
through the online network’s projector and the target network’s projector
respectively. Finally, we forward-pass $z_{\theta,1}$ through the online
network’s predictor to obtain $q_{\theta}(z_{\theta,1})$. We adopt the L2 loss
according to the BYOL architecture (4). To make this loss symmetric for the
view, we perform the same computation after swapping the two views to obtain
$z_{\theta,2},z_{\xi,1}$. Our final loss is provided in equation (5).

$\mathcal{L}_{\text{BYOL}}(z_{\theta},z_{\xi})=2-2\cdot\frac{q_{\theta}(z_{\theta})\cdot
z_{\xi}}{\left\lVert q_{\theta}(z_{\theta})\right\rVert_{2}\cdot\left\lVert
z_{\xi}\right\rVert_{2}}$ (4)

$\mathcal{L}_{repr}=\mathcal{L}_{\text{BYOL}}(z_{\theta,1},z_{\xi,2})+\mathcal{L}_{\text{BYOL}}(z_{\theta,2},z_{\xi,1})$
(5)

###  3.3 Region-to-Object Curriculum

Empirically, we have found that varying the number of segments in the refined
masks, from a high number to a low number, results in improved downstream
performance and more accurate masks. Thus, we introduce a region-to-object
curriculum which allows for R2O pretraining to learn both region based and
object centric features. Specifically, we schedule the number of clusters $K$
introduced in the region refinement step (Section 3.1). We have found that a
region-to-object curriculum where $K$ begins at a high value, e.g. $K=128$,
and is reduced to a low value, e.g. $K=4$, over pretraining works best in
terms of downstream performance. A visualization of the refined mask (Figure
1) illustrates how this curriculum allows for refined mask to become
increasingly object centric (see Appendix A.8 for more examples). In this
work, we explore a piecewise, linear, and cosine schedule for $K$.

##  4 Results

We evaluate R2O via common transfer learning paradigms by pretraining on the
train set of the ImageNet ILSVRC-2012 dataset [48]. We evaluate
representations when finetuned for object detection and instance segmentation
on COCO and semantic segmentation on PASCAL VOC and Cityscapes. Furthermore,
we perform several ablations exploring the various formulations and
implementation specifics of R2O discussed below. We also investigate
unsupervised object segmentation on Caltech-UCSD Birds 200-2011 [55] using our
ImageNet pretrained models. The license, PII, and consent details of each
dataset are in the respective papers. We provide the code/models from this
work at https://github.com/KKallidromitis/r2o.

#### Architecture

We use a ResNet-50 [24] architecture for all encoders. Following, BYOL, all
projector and predictions heads are 2 layer MLPs which use batch normalization
[33] after the hidden layer. Our region refinement step uses the C4 output
from our target network. For efficiency purposes, we perform K-means over the
mini-batch on each GPU. Our refined mask $m$ is of shape (14, 14). We use
RoIAlign [25] to align $m$ with views $x_{1}$ and $x_{2}$. Our region-to-
object representation learning step applies mask-pooling to the C5 output of
the ResNet-50 similar to the existing use of global pooling.

#### Optimization

We follow the optimization details of BYOL (full details in Supplementary
A.1). Specifically, we use a base learning rate of 0.3, which is scaled
linearly to batch size, and a weight decay of $10^{-6}$. The learning rate is
decayed by a cosine schedule after a warmup period which is one percent of the
total pretraining time, e.g. 300 epochs of pretraining uses 3 warmup epochs.
Our target network’s parameter $\tau$, which details the tradeoff between the
current parameters in the target and online networks, is increased over
training following a cosine schedule. We use the LARS optimizer with a global
batch size of 4096 distributed over 128 NVIDIA V100 GPUs. Our models are
trained using PyTorch [44].

#### Refinement

We generate $100$ segments for each image using SLIC. We perform K-means with
$K$ initially set to 128. We use a cosine scheduler that decreases the value
of K from 128 to 4 over the course of training. See Appendix A.1 for
additional experimental details.

###  4.1 Transfer Learning Details

#### Object Detection and Instance Segmentation: MS COCO

We finetune our encoder as the backbone of a Mask-RCNN (R50-FPN) [25] using
Detectron2 [59]. We follow the 1$\times$ (12 epochs) and 2$\times$ (24 epochs)
training schedule, training on MS COCO’s train2017 dataset and evaluating on
the val2017 dataset. We report average precision for bounding box predictions
(APbb) and mask predictions (APmk). Hyperparameter details and variance across
multiple random seeds are reported in Appendix A.1.

#### Semantic Segmentation: PASCAL VOC and Cityscapes.

We evaluate our semantic segmentation performance by finetuning the encoder as
the backbone of a FCN [38] using MMSegmentation [15]. For PASCAL VOC, we
finetune using the train_aug2012 dataset for 45 epochs. For Cityscapes, we
finetune using the train_fine dataset for 160 epochs. Our evaluation pipeline
follows the architecture and hyperparameters found in He et al. [26].
Performance is measured by mean intersection-over-union (mIOU) on val2012 and
val_fine respectively. To ensure fair comparision [22], we tune the
hyperparameters used in evaluation for all related methods. Full details
regarding hyperparameters and standard deviation of performance over random
trials are in Appendix A.3.

###  4.2 ImageNet-1K Transfer

We use the BYOL data augmentation policy to generate two views during
pretraining which lasts for 300 epochs. Additionally, we compute SLIC masks
using a $224\times 224$ resized version of the original image.

#### Object Detection and Instance Segmentation

Table 1 shows our performance on COCO object detection and instance
segmentation. Our method outperforms state-of-the-art methods when using the
1$\times$ schedule ($+0.2$ APbb, $+0.3$ APmk) and is competitive with these
methods when using the 2$\times$ schedule ($+0.1$ APbb, $+0.1$ APmk). We
attribute our state-of-the-art performance on instance segmentation to the
nature of our pretraining. R2O segments a scene by refining a large number of
small regions based on image-level priors to a few larger regions that are
semantically meaningful using learned representations while learning
representations corresponding to each of the discovered segments. Thus,
segmentation based tasks should benefit from this form of pretraining process
as representations learned during pretraining rely upon the boundaries of the
discovered masks instead of rectangular bounding boxes.

| COCO 1$\times$ |  | COCO 2$\times$
---|---|---|---
Method | APbb | APmk |  | APbb | APmk
Supervised | 38.9 | 35.4 |  | 40.6 | 36.8
MoCo v2 [10] | 38.9 | 35.4 |  | 40.9 | 37.0
BYOL† [23] | 40.6 | 37.5 |  | 42.0 | 38.7
DenseCL [56] | 40.3 | 36.4 |  | 41.2 | 37.3
ReSim [60] | 39.3 | 35.7 |  | 41.1 | 37.1
PixPro [63] | 41.4 | - |  | - | -
DetCon _B_ † [28] | 41.5 | 38.0 |  | 42.1 | 38.9
DetCo [61] | 39.4 | 34.4 |  | 41.4 | 35.8
LEWEL [31] | 41.3 | 37.4 |  | 42.2 | 38.2
R2O | 41.7 | 38.3 |  | 42.3 | 39.0
Table 1: Performance on COCO object detection and instance segmentation
following ImageNet pretraining. All methods pretrained a ResNet-50 which later
served as the backbone of a Mask R-CNN R50-FPN finetuned on train2017 for 12
epochs (1$\times$ schedule) or 24 epochs (2$\times$ schedule). We report
Average Precision on bounding box (APbb) and mask (APmk) predictions for
val2017. †: Results from re-implementation.

#### Semantic Segmentation

As seen in Table 2, we outperform existing methods on PASCAL VOC ($+0.7$ mIOU)
and Cityscapes ($+0.4$ mIOU) semantic segmentation. We believe our
improvements in semantic segmentation demonstrate the benefits of pretraining
region-level and object-centric representations. R2O improves transfer
performance on this task as pixel-level features are learned while also
training the object-centric representations needed to identify the objects in
the scene. In particular, the benefits of R2O is demonstrated by our state-of-
the-art performance on the PASCAL VOC semantic segmentation task, a pixel-
level classification task which is object-focused as most of its classes are
also present in ImageNet-1K.

Method | PASCAL VOC | Cityscapes
---|---|---
Supervised | 72.4 | 74.7
MoCo v2 [10] | 73.9 | 75.6
BYOL† [23] | 75.0 | 75.8
DenseCL [56] | 73.8 | 76.1
DetCon _B_ † [28] | 76.0 | 76.2
ReSim [60] | 74.3 | 75.5
PixPro [63] | 74.2 | 75.9
DetCo [61] | 74.3 | 74.9
LEWEL [31] | 75.5 | 75.4
R2O | 76.7 | 76.6
Table 2: Performance on PASCAL VOC and Cityscapes semantic segmentation (mIOU)
following ImageNet pretraining. Unless otherwise stated, results are computing
using pretrained weights officially released by the authors. †: Results from
re-implementation of pretraining method.

###  4.3 Unsupervised Object Segmentation

We investigate the quality of segmentations from R2O pretrained features by
evaluating unsupervised object segmentation on Caltech-UCSD Birds 200-2011
(CUB-200-2011). We use our ImageNet pretrained ResNet-50 and generate object
segmentations by applying K-means clustering to image features with $K=5$. We
use Hungarian matching to assign one segment as the foreground and the rest
segments are considered as background. This approach accounts for the high
variance in the background (e.g., trees, ground, sky, etc.). Table 3 shows our
performance, measured by mean intersection-over-union (mIOU), when segmenting
the foreground and background on a test from CUB-200-2011 used in prior works
[7]. Table 3 also provides self-supervised pretrained baselines using MoCo v2
[10] and BYOL [23] following our evaluation protocol. In comparison to
existing unsupervised object segmentation methods [5, 7, 4, 54], segmentation
via K-means clustering of R2O pretrained ResNet-50 features surpasses state-
of-the-art ($+2.9$ mIOU) without finetuning on CUB-200-2011. Similar to our
protocol, Melas-Kyriazi et al. [39] extract accurate segmentations from
ImageNet pretrained GANs, without finetuning on CUB-200-2011, in order to
train a UNet [47]. R2O outperforms Melas-Kyriazi et al. [39] ($+4.8$ mIOU)
using only K-means clustering on features from R2O pretrained encoders.

Method | CUB-200-2011
---|---
Self-Supervised Pretraining Methods |
MoCo v2 [10] | 63.5
BYOL† [23] | 65.1
Unsupervised Object Segmentation Methods |
PerturbGAN [5] | 38.0
ReDO [7] | 42.6
OneGAN [4] | 55.5
Voynov et al. [54] | 68.3
Melas-Kyriazi et al. [39] | 66.4
R2O | 71.2
Table 3: Performance on CUB-200-2011 segmentation. Results report the mean
intersection-over-union (mIOU) for foreground-background segmentations of
images in a Caltech-UCSD Birds 200-2011 (CUB-200-2011) test set [7]. Unlike
[5, 7, 4, 54], we are able to produce state-of-the-art quality segmentations
($+2.9$ mIOU) using our ImageNet pretrained encoder and without finetuning on
CUB-200-2011. †: Results from re-implementation.

###  4.4 Ablation and Analysis

To examine the importance of region-to-object pretraining, we experiment with
different schedule types. Furthermore, we also evaluate the effectiveness of
our region refinement module. The models used in these ablations are
pretrained on ImageNet100 [50] for 100 epochs and evaluated on PASCAL VOC
semantic segmentation using the details in Section 4.1. We provide ablations
on the effect of the type of scheduler (e.g. cosine, linear, piecewise) and
type of neighborhood prior (SLIC versus spatial segmentation) in the Appendix
(A.6, A.7). The Appendix also includes ablations examining the use of a
contrastive representation learning objective (Appendix A.4) and transfer
performance when using a Vision Transformer (ViT) encoder [18] (Appendix A.5).

#### Quantifying objectness of the masks throughout pretraining.

We use the Average Best Overlap (ABO) [51] to measure the improvement of masks
throughout training using semantic segmentation annotations for ImageNet-1K
provided by LUSS [20]. As shown in Figure 3, our ABO improved by $0.25$ over
the course of pretraining. As a reference, Figure 3 includes the ABO of our
SLIC prior as well. Masks refined by R2O are consistently better than SLIC
segments, suggesting that the refinement step is able to improve small region
priors into more object-like masks that are semantically meaningful.

Figure 3: R2O refined masks become increasingly object-centric over
pretraining. We compute average best overlap (ABO) for ImageNet segmentations
(using LUSS annotations) generated over the course of pretraining. For
reference, we also display the ABO of our region-level prior (SLIC).

Schedule | PASCAL VOC (mIOU)
---|---
reg $\rightarrow$ obj | 16 $\rightarrow$ 4 | 61.1
64 $\rightarrow$ 4 | 61.5
128 $\rightarrow$ 4 | 62.3
256 $\rightarrow$ 4 | 61.7
obj $\rightarrow$ reg | 4 $\rightarrow$ 16 | 60.2
4 $\rightarrow$ 64 | 60.8
4 $\rightarrow$ 128 | 61.0
4 $\rightarrow$ 256 | 61.1

Table 4: Comparing region-to-object, object-to-region, and fixed $K$
curriculum. We pretrain an encoder on ImageNet100 while using alternate
schedules for our region refinement step. We report mean intersection-over-
union (mIOU) after finetuning on PASCAL VOC semantic segmentation.

#### Evaluating Alternate Schedules.

Table 4 demonstrates that a region-to-object schedule leads to the best
transfer performance on PASCAL VOC ($+1.2$ mIOU) for semantic segmentation.
This schedule outperforms using a fixed $K$ and an opposite schedule which
increases $K$ from a low value to a high value. Using a fixed $K$ does not
allow for learning both region based features, corresponding to image patches
without objects, and object centric representation learning. If the fixed
value of $K$ is high, say $K=64$, then refined masks will have many distinct
regions and the representation learning step will train features corresponding
to small image regions or object parts. A small fixed value of $K$, say $K=4$,
has the opposite effect as representations learned will simply match large
image regions across views and cannot learn small region based features
necessary for pixel level prediction tasks. The opposite schedule, denoted as
object-to-region, also leads to suboptimal performance. We believe this is
because such a schedule does not allow for objects to be found and
represented. Instead, early refined masks are just large image regions without
any semantic meaning as the network has undergone very little training.

Region Refinement | SLIC Prior | Region-to-Object Schedule | PASCAL VOC (mIOU)
---|---|---|---
✗ | ✗ | ✗ | 61.2
✓ | ✗ | ✗ | 61.5
✗ | ✗ | ✓ | 61.4
✓ | ✗ | ✓ | 61.5
✗ | ✓ | ✓ | 61.7
✓ | ✓ | ✗ | 61.4
✓ | ✓ | ✓ | 62.3
Table 5: Importance of R2O components. We ablate the key components of the R2O
pretraining process, namely: the region refinement module, the use of a prior
(SLIC), and the region-to-object schedule. As shown, the combination of using
Region Refinement, SLIC Prior, and Region-to-Object Schedule are necessary for
the top performing result (62.3 mIOU) – removing any component decreases
performance by at least $0.6$ mIOU.

#### Effect of of Region Refinement

Table 5 details the results of ablating each key component in R2O pretraining.
The baseline method, which does not use a SLIC prior, region-to-object
schedule, nor the region refinmenet, performs the worst. This baseline method
is a region-to-region form of pretraining, similar to DenseCL [56] but without
the method’s feature matching strategy. We find that our Region-to-Object
Schedule has the largest impact on performance ($+0.9$ mIOU) because this
element combines the region level learning and object level learning.
Moreover, Table 5 shows that the combination of using Region Refinement, SLIC
Prior, and Region-to-Object Schedule are necessary for the top performing
result ($62.3$ mIOU); removing any component decreases performance.

#### Limitations

While our experiments and ablations indicate that R2O learns object-centric
masks throughout pretraining, we do not have a guarantee that such masks will
necessarily identify objects. As shown by several examples in Appendix A.8,
there are several failure cases where R2O segmentations group different
objects together or separates parts of objects that have significantly
different color, texture, or brightness compared to the rest of the object.
R2O is an unsupervised pretraining algorithm, and without semantic grounding
from object-level annotations, it cannot necessarily distinguish between the
semantics of what constitutes an object. Nevertheless, as indicated by our
strong experimental results, R2O learns useful representations that can be
finetuned for dense downstream prediction tasks.

##  5 Conclusion

We present Region-to-Object Representation Learning (R2O) which unifies
region-based and object-centric instance pretraining by learning both region-
level and object-centric representations using only simple region-level
priors. While existing methods have either focused exclusively on region-based
or object-centric pretraining, R2O shows that a region-to-object curriculum
which transitions between these two objectives leads to strong representations
for a variety of downstream tasks such as detection and segmentation.
Furthermore, we have found R2O pretraining generates object-centric
segmentations during pretraining and our ImageNet pretrained encoders
establish new state-of-the-art in unsupervised object segmentations on
different downstream datasets.

##  6 Broader Impacts and Acknowledgements

Our method can learn object-centric features using unsupervised pretraining.
As demonstrated in Section 4, this can benefit downstream tasks such as object
detection and segmentation. However, R2O does not guarantee the output of
object-centric masks. Thus, representations learned using R2O could exhibit
the negative artifacts seen in image-level pretraining, such as only focusing
on background pixels [49]. Caution must be exercised in safety-critical
applications of our method.

Acknowledgements: We thank Tete Xiao for valuable discussions and feedback.
This work was supported in part by DoD including DARPA’s LwLL, PTG and/or
SemaFor programs, as well as BAIR’s industrial alliance programs. We thank the
authors of DetCon [28] for assisting us with reproducing their baselines.

## References

  * Achanta et al. [2012] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk.  Slic superpixels compared to state-of-the-art superpixel methods.  _IEEE transactions on pattern analysis and machine intelligence_ , 34(11):2274–2282, 2012.
  * Bar et al. [2021] A. Bar, X. Wang, V. Kantorov, C. J. Reed, R. Herzig, G. Chechik, A. Rohrbach, T. Darrell, and A. Globerson.  Detreg: Unsupervised pretraining with region priors for object detection.  _arXiv preprint arXiv:2106.04550_ , 2021.
  * Bengio et al. [2013] Y. Bengio, A. Courville, and P. Vincent.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1798–1828, 2013.
  * Benny and Wolf [2020] Y. Benny and L. Wolf.  Onegan: Simultaneous unsupervised learning of conditional image generation, foreground segmentation, and fine-grained clustering.  In _European Conference on Computer Vision_ , pages 514–530. Springer, 2020.
  * Bielski and Favaro [2019] A. Bielski and P. Favaro.  Emergence of object segmentation in perturbed generative models.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Caron et al. [2021] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9650–9660, 2021.
  * Chen et al. [2019] M. Chen, T. Artières, and L. Denoyer.  Unsupervised object segmentation by redrawing.  _Advances in neural information processing systems_ , 32, 2019.
  * Chen et al. [2020a] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020a.
  * Chen and He [2021] X. Chen and K. He.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15750–15758, 2021.
  * Chen et al. [2020b] X. Chen, H. Fan, R. Girshick, and K. He.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_ , 2020b.
  * Chen et al. [2021] X. Chen, S. Xie, and K. He.  An empirical study of training self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9640–9649, 2021.
  * Cho et al. [2021] J. H. Cho, U. Mall, K. Bala, and B. Hariharan.  Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16794–16804, 2021.
  * Comaniciu and Meer [1997] D. Comaniciu and P. Meer.  Robust analysis of feature spaces: Color image segmentation.  In _Proceedings of IEEE computer society conference on computer vision and pattern recognition_ , pages 750–755. IEEE, 1997.
  * Comaniciu and Meer [1999] D. Comaniciu and P. Meer.  Mean shift analysis and applications.  In _Proceedings of the seventh IEEE international conference on computer vision_ , volume 2, pages 1197–1203. IEEE, 1999.
  * Contributors [2020] M. Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * Dhanachandra et al. [2015] N. Dhanachandra, K. Manglem, and Y. J. Chanu.  Image segmentation using k-means clustering algorithm and subtractive clustering algorithm.  _Procedia Computer Science_ , 54:764–771, 2015.
  * Doersch et al. [2015] C. Doersch, A. Gupta, and A. A. Efros.  Unsupervised visual representation learning by context prediction.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1422–1430, 2015.
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Felzenszwalb and Huttenlocher [2004] P. F. Felzenszwalb and D. P. Huttenlocher.  Efficient graph-based image segmentation.  _International journal of computer vision_ , 59(2):167–181, 2004.
  * Gao et al. [2021] S.-H. Gao, Z.-Y. Li, M.-H. Yang, M.-M. Cheng, J. Han, and P. Torr.  Large-scale unsupervised semantic segmentation.  _arXiv preprint arXiv:2106.03149_ , 2021.
  * Gidaris et al. [2018] S. Gidaris, P. Singh, and N. Komodakis.  Unsupervised representation learning by predicting image rotations.  _arXiv preprint arXiv:1803.07728_ , 2018.
  * Goyal et al. [2019] P. Goyal, D. Mahajan, A. Gupta, and I. Misra.  Scaling and benchmarking self-supervised visual representation learning.  In _Proceedings of the ieee/cvf International Conference on computer vision_ , pages 6391–6400, 2019.
  * Grill et al. [2020] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in Neural Information Processing Systems_ , 33:21271–21284, 2020.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2017] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * He et al. [2020] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * He et al. [2022] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Hénaff et al. [2021] O. J. Hénaff, S. Koppula, J.-B. Alayrac, A. van den Oord, O. Vinyals, and J. Carreira.  Efficient visual pretraining with contrastive detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10086–10096, 2021.
  * Hénaff et al. [2022] O. J. Hénaff, S. Koppula, E. Shelhamer, D. Zoran, A. Jaegle, A. Zisserman, J. Carreira, and R. Arandjelović.  Object discovery and representation networks.  _arXiv preprint arXiv:2203.08777_ , 2022.
  * Hou et al. [2016] J. Hou, H. Gao, and X. Li.  Dsets-dbscan: A parameter-free clustering algorithm.  _IEEE Transactions on Image Processing_ , 25(7):3182–3193, 2016.
  * Huang et al. [2022] L. Huang, S. You, M. Zheng, F. Wang, C. Qian, and T. Yamasaki.  Learning where to learn in cross-view self-supervised learning.  _arXiv preprint arXiv:2203.14898_ , 2022.
  * Hwang et al. [2019] J.-J. Hwang, S. X. Yu, J. Shi, M. D. Collins, T.-J. Yang, X. Zhang, and L.-C. Chen.  Segsort: Segmentation by discriminative sorting of segments.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7334–7344, 2019.
  * Ioffe and Szegedy [2015] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _Proceedings of the 32nd International Conference on Machine Learning_ , Proceedings of Machine Learning Research. PMLR, 2015.
  * Ji et al. [2019] X. Ji, J. F. Henriques, and A. Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9865–9874, 2019.
  * Kanezaki [2018] A. Kanezaki.  Unsupervised image segmentation by backpropagation.  In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 1543–1547. IEEE, 2018.
  * Lloyd [1982] S. Lloyd.  Least squares quantization in pcm.  _IEEE transactions on information theory_ , 28(2):129–137, 1982.
  * Locatello et al. [2020] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf.  Object-centric learning with slot attention.  _Advances in Neural Information Processing Systems_ , 33:11525–11538, 2020.
  * Long et al. [2015] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3431–3440, 2015.
  * Melas-Kyriazi et al. [2021] L. Melas-Kyriazi, C. Rupprecht, I. Laina, and A. Vedaldi.  Finding an unsupervised image segmenter in each of your deep generative models.  _arXiv preprint arXiv:2105.08127_ , 2021.
  * Misra and Maaten [2020] I. Misra and L. v. d. Maaten.  Self-supervised learning of pretext-invariant representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6707–6717, 2020.
  * Noroozi and Favaro [2016] M. Noroozi and P. Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In _European conference on computer vision_ , pages 69–84. Springer, 2016.
  * O Pinheiro et al. [2020] P. O. O Pinheiro, A. Almahairi, R. Benmalek, F. Golemo, and A. C. Courville.  Unsupervised learning of dense visual representations.  _Advances in Neural Information Processing Systems_ , 33:4489–4500, 2020.
  * Oord et al. [2018] A. v. d. Oord, Y. Li, and O. Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Reed et al. [2021] C. J. Reed, S. Metzger, A. Srinivas, T. Darrell, and K. Keutzer.  Selfaugment: Automatic augmentation policies for self-supervised learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2674–2683, June 2021.
  * Reed et al. [2022] C. J. Reed, X. Yue, A. Nrusimha, S. Ebrahimi, V. Vijaykumar, R. Mao, B. Li, S. Zhang, D. Guillory, S. Metzger, K. Keutzer, and T. Darrell.  Self-supervised pretraining improves self-supervised pretraining.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 2584–2594, January 2022.
  * Ronneberger et al. [2015] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pages 234–241. Springer, 2015.
  * Russakovsky et al. [2015] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Selvaraju et al. [2021] R. R. Selvaraju, K. Desai, J. Johnson, and N. Naik.  Casting your model: Learning to localize improves self-supervised representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11058–11067, 2021.
  * Tian et al. [2019] Y. Tian, D. Krishnan, and P. Isola.  Contrastive multiview coding.  _arXiv preprint arXiv:1906.05849_ , 2019.
  * Uijlings et al. [2013] J. R. Uijlings, K. E. Van De Sande, T. Gevers, and A. W. Smeulders.  Selective search for object recognition.  _International journal of computer vision_ , 104(2):154–171, 2013.
  * Van Gansbeke et al. [2020] W. Van Gansbeke, S. Vandenhende, S. Georgoulis, M. Proesmans, and L. Van Gool.  Scan: Learning to classify images without labels.  In _European conference on computer vision_ , pages 268–285. Springer, 2020.
  * Van Gansbeke et al. [2021] W. Van Gansbeke, S. Vandenhende, S. Georgoulis, and L. Van Gool.  Unsupervised semantic segmentation by contrasting object mask proposals.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10052–10062, 2021.
  * Voynov et al. [2021] A. Voynov, S. Morozov, and A. Babenko.  Object segmentation without labels with large-scale generative models.  In _International Conference on Machine Learning_ , pages 10596–10606. PMLR, 2021.
  * Wah et al. [2011] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * Wang et al. [2021] X. Wang, R. Zhang, C. Shen, T. Kong, and L. Li.  Dense contrastive learning for self-supervised visual pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3024–3033, 2021.
  * Wei et al. [2021] F. Wei, Y. Gao, Z. Wu, H. Hu, and S. Lin.  Aligning pretraining for detection via object-level contrastive learning.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Wen et al. [2022] X. Wen, B. Zhao, A. Zheng, X. Zhang, and X. Qi.  Self-supervised visual representation learning with semantic grouping.  _arXiv preprint arXiv:2205.15288_ , 2022.
  * Wu et al. [2019] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * Xiao et al. [2021] T. Xiao, C. J. Reed, X. Wang, K. Keutzer, and T. Darrell.  Region similarity representation learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10539–10548, 2021.
  * Xie et al. [2021a] E. Xie, J. Ding, W. Wang, X. Zhan, H. Xu, P. Sun, Z. Li, and P. Luo.  Detco: Unsupervised contrastive learning for object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8392–8401, 2021a.
  * Xie et al. [2021b] J. Xie, X. Zhan, Z. Liu, Y. S. Ong, and C. C. Loy.  Unsupervised object-level representation learning from scene images.  _Advances in Neural Information Processing Systems_ , 34:28864–28876, 2021b.
  * Xie et al. [2021c] Z. Xie, Y. Lin, Z. Zhang, Y. Cao, S. Lin, and H. Hu.  Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16684–16693, 2021c.
  * Xu et al. [2021] Y. Xu, Q. Zhang, J. Zhang, and D. Tao.  Regioncl: Can simple region swapping contribute to contrastive learning?  _arXiv preprint arXiv:2111.12309_ , 2021.
  * Yang et al. [2021] C. Yang, L. Huang, and E. J. Crowley.  Contrastive object-level pre-training with spatial noise curriculum learning.  _arXiv preprint arXiv:2111.13651_ , 2021.
  * Zhang et al. [2021] F. Zhang, P. Torr, R. Ranftl, and S. Richter.  Looking beyond single images for contrastive semantic segmentation learning.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Zhang et al. [2016] R. Zhang, P. Isola, and A. A. Efros.  Colorful image colorization.  In _European conference on computer vision_ , pages 649–666. Springer, 2016.

## Appendix

##  A.1 Pretraining Details

#### Data Augmentation

We follow the BYOL [23] data augmentation pipeline. The SLIC masks are
generated using a resized copy of the input image. To generate the augmented
views, we implement the following operations using torchvision.

  1. 1.

Random Crop: Generate two distinct random crops from the original image.
Afterwards we resize both views to $224\times 224$.

  2. 2.

Horizontal Flip: We flip the views horizontally with a $0.5$ probability.

  3. 3.

Color Jitter: With an implementation probability of $0.8$ we set brightness
equal to $0.4$, contrast = $0.4$, saturation = $0.2$ and hue = $0.1$.

  4. 4.

Grayscale: We convert the images from RGB to Grayscale with a $0.2$
probability.

  5. 5.

Gaussian Blur: We always apply blurring for the first view and with a $0.1$
probability for the second view. We use a $23\times 23$ kernel with a standard
deviation chosen uniformly at random between ranges $[0.1,2.0]$.

  6. 6.

Solarize: Solarize only implemented on view 2 with a probability of $0.2$
using the PIL package and a threshold of $128$.

#### Refinement

We employ a cosine schedule for the number of segments, $K$, which starts at
$K_{0}$ at epoch 0 and gradually decreases to $K_{f}$ between epoch
$t_{\alpha}$ and $T_{epoch}$, where $T_{epoch}$ is the total number of epochs
during the pretraining. Equation A1 expresses $K$ as a function of $t$, the
epoch number. We choose $K_{0}=128,K_{f}=4,t_{\alpha}=40,T_{epoch}=300$ in our
experiment.

$K(t)=\left\\{\begin{array}[]{lr}K_{0},&\text{if }t<t_{\alpha}\\\
K_{f}+cos(\frac{2(t-t_{\alpha})}{(T_{epoch}-t_{\alpha})\cdot\pi})(K_{0}-K_{f})&\text{otherwise
}\end{array}\right\\}$ (A1)

#### Optimization

We followed BYOL [23] optimization. The online network is updated by a LARS
optimizer with a base learning rate of 0.3 and and a weight decay of
$10^{-6}$. The actual maximum learning rate during training is
$\text{base\\_lr}\times\frac{\text{global\\_batch\\_size}}{256}$. The learning
rate increases linearly during the warmup period and then decreases following
a cosine schedue. In our experiment, the global batch size is 4096 and the
warmup period is 3 epochs. The target network parameter $\xi$ is updated by
$\xi^{t}\leftarrow(1-\tau^{t})\cdot\theta^{t}+\tau^{t}\cdot\xi^{t-1}$ where
$\theta$ is the online network parameter and $\tau$ increases from $0.99$ to
$1.00$ following a cosine schedule.

##  A.2 Transfer Learning Details

#### Object Detection and Instance Segmentation: MS COCO

We adopt Mask-RCNN [25] architecture with a ResNet-50 backbone and FPN. In the
fine tuning stage, images are randomly flipped and resized to $u\cdot 1024$ on
the longest side where $u$ is uniformly sampled in $[0.8,1.25]$. The images
are then padded or cropped to $1024\times 1024$. The aspect ratio is kept the
same as original image. During the evaluation, the images are resized to
$1024$ on the longest side. We fine tuned on COCO for the 1$\times$ and
2$\times$ schedule using stochastic gradient descent. We swept between
learning rates of $0.003,0.1,0.3$ for our model and our DetCon implementation
to ensure a fair comparison. We use a momentum of $0.9$, a weight decay of
$4\cdot 10^{-5}$, and a global batch size of $32$. The learning rate increases
linearly in the first $1000$ epochs and drops twice by a factor of $10$ after
$\frac{2}{3}$ and $\frac{8}{9}$ of the total training time.

#### Semantic Segmentation: PASCAL VOC and Cityscapes

We adopt FCN [38] architecture with a ResNet-50 backbone. We made the
following architecture changes on the backbone according to MOCO [26]. The 3×3
convolutions in conv5 blocks have dilation 2 and stride 1. This is followed by
two extra 3×3 convolutions of 256 channels with BN and ReLU, and then a 1×1
convolution for per-pixel classification. We set dilation = 6 in the two extra
3×3 convolutions.All other implementation details are the default of
MMSegmentation [15], For PASCAL VOC, we fine tuned on train_aug2012 and
evaluated on val2012 dataset. The images are randomly flipped, scaled by a
ratio randomly sampled in $[0.5,2.0]$, and then cropped to $513\times 513$
during the training. The evaluation is performed on the original image size.
We fine tuned for 30k iterations with a batch size of 16 and a weight decay of
$10^{-4}$. The learning rate is dropped by a factor of 10 at $\frac{7}{10}$
and $\frac{9}{10}$ of the total training time. For Cityscapes, we fine tuned
on train_fine and evaluated on val_fine dataset. The images are randomly
flipped, resized to $2049\times 1025$, scaled by a ratio randomly sampled in
$[0.5,2.0]$, and then cropped to $769\times 769$ during the training. The
evaluation is performed on a resolution of $2049\times 1025$. We fine tuned
for 30k iterations with a batch size of 16, and a weight decay of $10^{-4}$.
The learning rate is dropped by a factor of 10 at $\frac{7}{10}$ and
$\frac{9}{10}$ of the total training time.To ensure a fair comparison, we
followed the practice of [22] and evaluate each pretraining method under a
variety of learning rates and report the best results. We also report the
standard deviation across three random seeds. For PASCAL VOC, we swept between
the learning rate 0.001, 0.003, 0.01, 0.03, 0.1. For Cityscapes, we swept
between learning rate 0.004, 0.01, 0.04.

##  A.3 Extended Experimental Results

We conduct additional experiments and report the average performance on
semantic segmentation and instance detection/segmentation with standard
deviation across 3 different seeds in table A1 and A2.

Method | PASCAL VOC | Cityscapes
---|---|---
Supervised | 72.4 | 74.7
MoCo v2 [10] | 73.9 $\pm$ 0.12 | 75.6 $\pm$ 0.09
BYOL† [23] | 75.0 $\pm$ 0.22 | 75.8 $\pm$ 0.31
DenseCL [56] | 73.8 $\pm$ 0.2 | 76.1 $\pm$ 0.12
DetCon _B_ † [28] | 76.0 $\pm$ 0.14 | 76.2 $\pm$ 0.09
ReSim [60] | 74.3 $\pm$ 0.28 | 75.5 $\pm$ 0.15
PixPro [63] | 74.2 $\pm$ 0.44 | 75.9 $\pm$ 0.37
DetCo [61] | 74.3 $\pm$ 0.07 | 74.9 $\pm$ 0.46
LEWEL [31] | 75.5 $\pm$ 0.23 | 75.4 $\pm$ 0.07
R2O |  76.7 $\pm$ 0.09 |  76.6 $\pm$ 0.05
Table A1: Performance on PASCAL VOC and Cityscapes semantic segmentation (mIoU) with standard deviation across 3 seeds. †: Results from re-implementation of pretraining method. Method | Epochs | APbb | APmk
---|---|---|---
BYOL (reproduced) | 300 | 40.3 ($\pm$ 0.10) | 37.5 ($\pm$ 0.08)
DetCon _B_ (reproduced) | 300 | 41.5 ($\pm$ 0.03) | 38.0 ($\pm$ 0.09)
Ours | 300 | 41.7 ($\pm$ 0.02) | 38.3 ($\pm$ 0.07)
Table A2: Performance on COCO object detection and instance segmentation with
standard deviation following the 1$\times$ schedule averaged across 3 seeds.

##  A.4 Exploring Contrastive Representation Learning

We investigate using a contrastive objective function for $L_{repr}$. We use
the NT-Xent objective used by the authors of SimCLR [8] and employ the same
architecture, data augmentation policy, and optimization details. We adopt the
mask sampling strategy employed by Hénaff et al. [28] in order to reduce the
memory overhead caused by the cross-GPU gather operation used in contrastive
learning. Table A3 compares the performance when using the SimCLR objective or
the BYOL objective. To ensure fair comparision, both methods were pretrained
for 300 epochs on ImageNet100 using the $K=128\rightarrow K=4$ schedule. The
BYOL objective leads to an improvement of $+4.51$ on PASCAL and $+1.44$ on
Cityscapes for the task of semantic segmentation. We note that the authors of
DetCon [28] demonstrated similar results when using a contrastive objective,
with DetcConS requiring up to 3 times the pretraining in order to match the
performance of DetConB.

Objective | PASCAL VOC | Cityscapes
---|---|---
SimCLR NT-Xent | 62.99 $\pm 0.02$ | 69.99 $\pm 0.17$
BYOL |  67.5 $\pm 0.1$ |  71.43 $\pm 0.25$
Table A3: Comparing contrastive and Siamese representation learning
objectives. We investigate using contrastive learning, via the Simclr NT-Xent
objective, during R2O pretraining. We find that using a Siamese representation
learning objective such as BYOL outperforms its contrastive counterpart when
finetuned for PASCAL VOC ($+4.51$) and Cityscapes ($+1.44$) semantic
segmentation. Results report mean mean intersection-over-union (mIOU).

##  A.5 Exploring Vision Transformer Architectures

Inspired by the recent progress in applications of Vision Transformer (ViT)
architectures [27, 6, 11, 18], we experiment with applying R2O pretraining to
a ViT-Base (Vit-B) architecture.We apply additonal transpose convolution
layers during the pretraining on the feature maps in the final transformer
block to get 7x7 and 14x14 feature maps to mimic the output of ResNet C5 and
C4 output. We did not make any other changes to the architecture or the loss.
We pretrained the R2O and MAE for 300 epochs on ImageNet100 with a global
batch size of 1024. We used the exact same optimization set up as our ResNet
experiments. The base learning rate is 0.3 following a cosine decay schedule.
The momentum is 0.9 and weight decay is $10^{-6}$. For MAE, we strictly
followed the optimization setting of [27]. We evaluate the pretrained weights
of MS COCO for instance detection and segmentation. We fined tuned for
1$\times$ schedule (12 epoch) using AdamW optimizer. The learning rate is
3e-4, the layer-wise decay rate is 0.75, and the drop path rate is 0.2. The
learning rate is multiplied by 0.1 at epochs 9 and 11. Table A4 compares MAE
[27] to R2O when using a ViT-B encoder after pretraining on ImageNet100 for
300 epochs. The R2O pretrained encoder provides a $+5.7$ increase in APbb and
a $+4.9$ increase in APmk relative to MAE. Thus, the benefits of R2O
pretraining for dense prediction tasks also apply to newer architectures such
as ViTs.

Objective | APbb | APmk
---|---|---
MAE [27] | 34.8 | 31.7
R2O | 40.5 | 36.6
Table A4: Examining COCO object detection and instance segmentation
performance when pretraining a ViT. We report Average Precision on bounding
box (APbb) and mask (APmk) predictions for val2017

##  A.6 Impact of Scheduler Type

We also experiment with the type of region-to-object scheduler (cosine,
piecewise or linear). In our ImageNet-1K experiment, we used a cosine
scheduler due to its performance in small-scale experiments. When pretraining
on ImageNet100, our linear and piecewise schedule slightly outperform the
cosine schedule ($+0.5$ and $+0.3$ on PASCAL VOC mIOU) (Table A5). This shows
the benefit of region-to-object learning. Most importantly, R2O pretraining
leads to better performance regardless of the type of scheduler.

##  A.7 Importance of Region-Level Prior

Compared with spatial heuristics, which divide an image into say a fixed
number of square patches, the SLIC prior which take into account color
resemblance leads to better performance ($+0.8$ mIOU) when finetuned for
PASCAL VOC semantic segmentation (Table A5). This suggest that using a broader
set of image-level criteria in equation will improve our learning process. The
performance benefits of SLIC also suggest that finding good region priors is
important for the refinement process. When comparing $10\times 10$ spatial
prior with $2\times 2$ spatial prior, we see that the encoder pretrained using
the $10\times 10$ prior performs because this prior introduces weaker spatial
constraints than its $2\times 2$ counterpart.

Prior | Scheduler | K schedule | PASCAL VOC (mIOU)
---|---|---|---
SLIC | Linear | 128 $\rightarrow$ 4 | 62.3
SLIC | Cosine | 128 $\rightarrow$ 4 | 61.8
SLIC | Piecewise | 128 $\rightarrow$ 4 | 62.1
SLIC | Fixed | 128 | 61.4
SLIC | Fixed | 64 | 61.4
SLIC | Fixed | 4 | 60.7
Spatial ($10\times 10$) | Linear | 128 $\rightarrow$ 4 | 61.5
Spatial ($2\times 2$) | Linear | 128 $\rightarrow$ 4 | 60.1
Table A5: Effect of varying region-level prior and type of region-to-object
scheduler. We pretrain on ImageNet100 for 100 epochs using various region-
level priors and region-to-object schedulers (e.g. cosine, linear, or
piecewise schedule). Results display the mean intersection-over-union (mIOU)
after finetuning on PASCAL VOC for semantic segmentation.

##  A.8 Visualization of Mask Refinement

#### ImageNet-1K Pretraining

In Figure A1, we visualize the masks generated by R2O in the region refinement
step during pretraining. The number of segments, $K$, starts at 128 and
gradually decreases to 4 following a cosine schedule. At 100 epoch, $K=119$.
At 200 epoch, $K=74$. At 300 epoch, $K=4$.

In Figure A2, we visualize some failure cases where the masks failed to
capture objects. In these examples, masks group pixels that are similar in
color and texture but do not belong to the same object (examples 1, 4). They
also tend to over-segment an object into multiple parts if different parts
have large disparities in color and texture (example 2). In a few extreme
cases, the masks groups all the pixels in entire image as belonging to the
same object (example 3). Since we are performing K-means over the batch, this
could happen when there is large variance across region-level features in the
batch while the variance in region-level features from the given image is
small. However, even in this worst case scenario, our representation learning
step amounts to using the standard BYOL objective with a global-pooled
embedding.

#### COCO Pretraining

We perform an additional experiment to investigate the mask refinement when
pretraining on COCO, a scene-centric dataset. We use the default region-to-
object scheduler used in our ImageNet-1K experiments, i.e. $K=128\rightarrow
K=4$, and find the quality of refined masks to be comparable to those
generated during ImageNet-1K pretraining ($0.46$ versus $0.49$ final ABO
respectively). Figure A3 visualizes some of the masks by the end of COCO
pretraining.

#### Caltech-UCSD Birds 200-2011 Segmentation

We perform unsupervised segmentation on Caltech-UCSD with K-means clustering
on feature space with K=5 using R2O ImageNet pre-trained checkpoints. Figure
A4 visualizes some of the masks.

Figure A1:  Visualization of refined masks generated during ImageNet
pretraining after 100, 200 and 300 epochs. We demonstrate early masks consists
of random image segments which gradually become object-centric. Figure A2:
Visualization of Failure Cases. We visualize some of the poorly segmented
masks generated during pretraining. From left to right: In example 1, one mask
groups the dog and part of the background together because of similar colors.
In example 2, the masks oversegment the TV screen into multiple regions. In
example 3, the jaguar blends into the background. In example 4, the masks fail
to distinguish the actual object and its reflection.  Figure A3:
Visualization of COCO Pretraining. We pretrain the model on COCO and visualize
the K-means mask during the training. The columns from left to right are
original images, SLIC regions priors and clustering results.  Figure A4:
Visualization of Caltech-UCSD Birds 200-2011 Segmentation. We perform
unsupervised segmentation by clustering feature representations on a per-image
basis with $K=5$. The best match is considered as the foreground.