11institutetext: Sungkyunkwan University

# Progressive Proxy Anchor Propagation for Unsupervised Semantic Segmentation

Hyun Seok Seong  0000-0002-7952-2017 WonJun Moon  0000-0003-2805-0926 SuBeen
Lee  0009-0005-1470-1160 Jae-Pil Heo Corresponding author 0000-0001-9684-7641

###### Abstract

The labor-intensive labeling for semantic segmentation has spurred the
emergence of Unsupervised Semantic Segmentation. Recent studies utilize patch-
wise contrastive learning based on features from image-level self-supervised
pretrained models. However, relying solely on similarity-based supervision
from image-level pretrained models often leads to unreliable guidance due to
insufficient patch-level semantic representations. To address this, we propose
a Progressive Proxy Anchor Propagation (PPAP) strategy. This method gradually
identifies more trustworthy positives for each anchor by relocating its proxy
to regions densely populated with semantically similar samples. Specifically,
we initially establish a tight boundary to gather a few reliable positive
samples around each anchor. Then, considering the distribution of positive
samples, we relocate the proxy anchor towards areas with a higher
concentration of positives and adjust the positiveness boundary based on the
propagation degree of the proxy anchor. Moreover, to account for ambiguous
regions where positive and negative samples may coexist near the positiveness
boundary, we introduce an instance-wise ambiguous zone. Samples within these
zones are excluded from the negative set, further enhancing the reliability of
the negative set. Our state-of-the-art performances on various datasets
validate the effectiveness of the proposed method for Unsupervised Semantic
Segmentation. Our code is available at https://github.com/hynnsk/PPAP.

###### Keywords:

Unsupervised Semantic Segmentation, Contrastive Learning

##  1 Introduction

Semantic Segmentation plays a vital role in various fields, including robotics
and autonomous driving [6, 56, 52, 49, 8, 37, 36, 18]. With the abundant data
available in media, developing high-quality semantic segmentation models has
become feasible [32], though this has also increased the demand for extensive
human annotations. Likewise, the increasing burden on human labor has spurred
the emergence of Unsupervised Semantic Segmentation (USS) [27, 13, 22, 55, 46,
31, 45, 48, 29, 30].

(a) Illustration comparing HP and Ours.

(b) Trustworthiness.

Figure 1:  (a) Illustration of how positive and negative sets are determined
in HP [46] and Ours. Different colors indicate different potential classes. In
HP, $k$-th nearest neighbor on a per-sample basis becomes an instance-wise
positiveness criterion, and all other samples farther than $k$-th neighbor are
considered as the negatives. On the other hand, we progressively propagate the
proxy anchor to be relocated in the region surrounded by semantically similar
samples. Consequently, we enable the trustworthy positive collection with
numerous samples in dense regions. Additionally, we define an ambiguous region
around the positiveness boundary where semantic boundaries might be ambiguous.
By excluding the samples in the ambiguous region in training, we avoid the
undesired repulsion between the anchor and the possibly false positives in the
negative set. (b) The number of positives and their precision with respect to
the ground truth label for a randomly sampled subset of the dataset. The
X-axis represents anchors within the range of [0%, $q$%] in the anchor list,
sorted in ascending order by the number of identified positives. The bar plot
displays the average number of gathered positives, and the line plot
illustrates their precision, as determined by the ground truth labels of both
the anchor and the positives.

The main challenge in USS stems from the lack of supervision to train the
model. To overcome this, prior works [22, 46, 55] suggested first learning the
image-level representation space and then leveraging this knowledge to develop
the ability of pixel-level understanding. Likewise, utilizing self-supervised
pretrained models [24, 3, 4] to provide supervision in USS became mainstream.
By employing these foundation models, previous techniques have demonstrated
promising results, particularly by learning the relationship among image
patches in the dataset [46, 22, 55].

Yet, we notice that existing methods still encounter challenges in discovering
trustworthy relationships between patches. For instance, HP [46], the latest
USS technique based on contrastive learning, utilized $k$-th nearest neighbor
of each anchor to determine an appropriate boundary for positive set
selection. While discretizing samples with such a boundary provides an
intuitive basis, it often leads to unreliable supervision. This is because
they exclusively rely on the similarity metric on a per-patch basis within an
imperfect embedding learned in an unsupervised manner at the image-level [4].
Consequently, as illustrated in Fig. 1(a), this approach may cause anchors to
gather false positives (FP) 111The italic ‘false positives (FP)’ and ‘true
positives (TP)’ [51] represent samples that incorrectly and correctly included
in the set, respectively, throughout the paper. in the positive set especially
when they are located in data-sparse areas or near the semantic boundaries
while encouraging anchors in dense regions to repel FP in the negative set.
Specifically, in Fig. 1(b), we present a quantitative comparison between HP
and our method, focusing on the number and precision of collected positives.
As observed, we note that only 77.5% of the positives identified by HP match
the ground truth labels, even though the number of positives is insufficient.
Precision further decreases for instances in data-sparse regions; samples with
a small number of gathered positives exhibit even lower precision (bottom 10%
samples retain only 33.81% precision on discovered positives). This indicates
the inclusion of a substantial number of FP in the positive set, thereby
attracting semantically dissimilar samples and leading to unstable learning.

To mitigate these issues, we propose a Progressive Proxy Anchor Propagation
(PPAP) strategy to deal with the vulnerability of the per-patch-based
similarity metric in an image-level pretrained embedding space. Our goal is to
establish a reliable proxy anchor by considering the data distribution
surrounding each anchor, thereby gathering patches with more trustworthy
positive and negative relationships minimizing ambiguity. This approach can
also obtain a larger number of training guidance as it enhances the precision
of gathered relationships. Specifically, to discover the position of the proxy
anchor, we begin by defining a tight boundary around each anchor to construct
a small, reliable positive set. The rationale behind establishing a tight
boundary is rooted in the observation that samples within closely adjacent
regions are highly likely to share similar semantics even within the image-
level pretrained embedding space. Subsequently, we iteratively undertake the
following two steps to enlarge a trustworthy positive set per anchor: 1) Re-
define the position of a proxy anchor based on the distribution of identified
positive samples, 2) Lower the similarity threshold for the positiveness
criterion, i.e., expand the boundary, according to the reliability of the new
proxy anchor position, and gather the updated positive set. Likewise, by
discovering the samples with similar semantics and moving the proxy anchor
towards the center point of such samples, we expect the assembly of
trustworthy positives. This strategy enables collecting a large number of
positive patches with high precision, as shown in Fig 1(b). Still, a
positiveness boundary might not be a perfect measure to detect all the
positive samples. In other words, there exists a degree of ambiguity around
the boundary, where both positive and negative instances might coexist. To
address this, we expand the original binary relationship categorization of
samples, i.e., positive and negative, for contrastive learning into tri-
partite groups, i.e., positive, negative, and ambiguous. The size of the
ambiguous set is determined based on the reliability of the relocated proxy
anchor. Consequently, while utilizing the positive and negative sets in
contrastive learning, we disregard the ambiguous set, as including FP in the
negative set often disrupts the stable training [11].

Overall, our contributions are summarized as follows:

  * •

We propose Progressive Proxy Anchor Propagation (PPAP), which systematically
gathers trustworthy positive samples for each anchor by progressively
analyzing the distribution of the positive samples.

  * •

We establish an ambiguity-excluded negative set based on the propagated proxy
anchor, defining a semantically ambiguous zone for each anchor. This approach
effectively eliminates potential FP in the negative set.

  * •

The efficacy of our trustworthy contrastive learning is validated by achieving
new state-of-the-art performances across diverse datasets.

##  2 Related Work

###  2.1 Unsupervised Semantic Segmentation

Semantic segmentation aims to classify the semantics of individual pixels
within an image [56, 7, 5, 8, 59, 17, 37, 44, 42]. In recent years, the
integration of transformers into semantic segmentation has emerged as a
promising research direction [12, 52, 35]. However, achieving pixel-wise
supervision requires extensive human labor. The necessity of learning semantic
segmentation without supervision has become apparent in recent literature [27,
13, 55, 46, 45, 29, 48, 30]. Earlier trials [27, 13] learned to maintain
consistent semantics across the paired features. In contrast, recent
techniques [55, 22, 46] have employed Vision Transformer (ViT) models trained
in a self-supervised manner as backbone networks to transfer knowledge to the
segmentation head. For instance, transFGU [55] grouped target datasets based
on prior knowledge and generated pseudo-labels to train the segmentation
model. STEGO [22] tried to maintain the patch relationships in the
segmentation head by distilling feature correspondences to segmentation
correspondences. HP [46], on the other hand, identified hidden positives using
the $k$-th nearest neighbor criterion to guide the contrastive objective. Our
goal aligns with previous works in seeking pseudo-supervision by considering
patch relationships. However, the key difference lies in our approach of
considering data distribution to find trustworthy pseudo-supervision within
the imperfect embedding space.

###  2.2 Self-supervised Representation Learning

Self-supervised representation learning has long been spotlighted for its
effectiveness in providing a decent initialization point for various
downstream tasks [4, 22]. There are several prevalent approaches in this
domain, including pretext tasks which learn the representation by
reconstructing the original input from augmented images [40, 34, 58, 20, 16,
41, 9], relation-based approaches [10, 24, 3, 21, 39] and masked-modeling
approaches [23, 53, 47, 54]. While masked-modeling approaches excel at
preserving local context, they are often less efficient in learning
discriminative representations [23, 54, 25]. Therefore, relation-based
approaches [4, 57], particularly DINO [4], are popularly employed in the realm
of USS [22, 55, 46]. Although the features from DINO are powerful in
describing semantics for the whole image, their direct use for semantic
segmentation proves effective due to the model being trained at the image-
level, as shown in Fig 1(b). In this regard, we have developed algorithms to
complement the representation of DINO for the promotion of such features to
reflect pixel-level semantics.

##  3 Method

Figure 2:  Overall procedure of Progressive Proxy Anchor Propagation (PPAP).
Our backbone consists of two branches: one for acquiring the training
guidance, and the other for task adaptive finetuning. Specifically, the former
feature extractor produces feature $\mathbf{f}$ used to compute training
guidance via trustworthy positive and ambiguity-excluded negative sets by
PPAP, and its parameters are frozen for stable guidance. On the other hand,
the latter branch is being finetuned with the training guidance to learn task-
adaptive feature $\mathbf{z}$.

###  3.1 Background and Overview

Recently, it has become mainstream to utilize the positive relationships among
patches in training for Unsupervised Semantic Segmentation (USS) [22, 55, 46].
They exploited the patch-wise embeddings from a pretrained foundation model.
However, we claim that they heavily relied on the similarity measured in
imperfect embedding space for inferring patch-level training guidance.
Instead, in this paper, we suggest the importance of considering the data
distribution; since not all anchors are highly likely to be densely surrounded
by semantically similar patch features in the embedding space, we aim to
search for better spots to gather a sufficient number of trustworthy positive
and negative samples.

The architecture of our method is illustrated in Fig 2. Following the recent
works [22, 55, 46], our goal is to learn an appropriate projection function
for the features extracted from a pretrained model suitable to the USS task.
To achieve this, we define two streams using the pretrained ViT; for the first
stream we keep all the blocks frozen to provide reliable supervision, and for
the other stream we finetune the last block for adapting features to the
semantic segmentation task. Given a mini-batch of images
$\\{\mathbf{x}_{b}\\}_{b=1}^{B}$, the former stream computes pairs of $B\times
H\times W$ patch features $\mathbf{f}_{i}\in\mathbb{R}^{D}$ where $H\times W$
is the number of patch features for an image and $D$ stands for the dimension
of embedding space. On the other hand, the latter stream producing projected
patch features $\mathbf{z}_{i}\in\mathbb{R}^{D}$ is being finetuned with the
gathered positive and negative sets. Specifically, the process begins by
determining the positive set $\mathcal{P}_{i}$ with $\mathbf{f}_{i}$. Through
the iterative process of positive gathering and proxy anchor relocation, we
construct trustworthy positive set. Afterward, we determine the ambiguity-
excluded negative set to train $\mathbf{z}_{i}$. In the following sections, we
discuss the Progressive Proxy Anchor Propagation strategy to obtain
trustworthy positive and ambiguity-excluded negative sets. in Sec. 3.2 and
Sec. 3.3, respectively.

###  3.2 Progressive Proxy Anchor Propagation

Collecting a sufficient amount of trustworthy pseudo-supervision is a
cumbersome task but crucial for the performance in USS [11].

To this end, we propose a Progressive Proxy Anchor Propagation algorithm to
identify the reliable region for each anchor, where semantically similar
samples to each anchor are densely located, as described in Fig. 2. The
propagation process begins by forming an initial positive set comprising
samples that are highly adjacent to each anchor. Subsequently, the algorithm
employs an iterative process composed of two following steps: 1) relocate the
proxy anchor towards more densely populated regions identified with the
distribution of gathered positives, and 2) identify positive samples around
the proxy anchor according to the expanded boundary. Note that the proxy
anchor (i.e., relocated anchor) provides a positive collection criterion on
behalf of the anchor where its boundary for positive collection is
proportional to the reliability of the proxy anchor’s new position that is
measured by the propagation degree of the proxy anchor. The proxy anchor
position is considered more reliable if it does not move significantly,
suggesting it is already surrounded by samples with similar semantics. This
enables each anchor to gather numerous trustworthy positive samples.

Specifically, the initial positive set $\mathcal{P}_{i}^{0}$ of a given anchor
$\mathbf{f}_{i}$ is obtained by applying the initial positiveness criterion
$\Phi^{0}$ to gather as below:

$\mathcal{P}_{i}^{0}=\\{j\mid\mathbf{f}_{i}\cdot\mathbf{f}_{j}>\Phi_{i}^{0},j\in\mathcal{B}\\},\;\forall_{i}\Phi_{i}^{0}=\Phi^{0},$
(1)

where $\mathcal{B}$ denotes the set containing all patch features within the
mini-batch, i.e., $|\mathcal{B}|=B\times H\times W$, and $\left(\cdot\right)$
refers to the similarity measure between two vectors (typically the cosine
similarity). Here, the criterion $\Phi^{0}$ is to decide whether all other
patch features in the mini-batch are positive or not, based on the similarity
threshold. This initial threshold is set to be big enough to make a tight
criterion and is shared across all anchors. Such a tight boundary from a large
initial criterion is for stable proxy anchor relocation since the samples with
close proximity are more likely to be semantically similar.

Then, to propagate the proxy anchor toward the positives-dominant region, we
derive the new proxy anchor position by averaging the collected positive set.
This way, we take the distribution of the gathered positives into account.
Formally, out of total $T$ steps of relocation, we present the $t$-th
relocated position of an anchor $\mathbf{f}_{i}$ as $\mathbf{v}^{t}_{i}$, as
follows:

$\mathbf{v}_{i}^{t}=\frac{1}{|\mathcal{P}^{t-1}_{i}|}\sum_{j\in\mathcal{P}^{t-1}_{i}}\mathbf{f}_{j}.$
(2)

To account for using the average points of the positive set, we posit that the
close vicinity of the proxy anchor is highly likely to retain the same
semantic. Thus, we claim that the center point of the gathered positives will
move the proxy anchor closer to a dense region populated with semantically
similar samples.

Furthermore, the positiveness criterion $\Phi_{i}^{t}$ should be reliability-
adaptively adjusted. We determine the reliability of $\mathbf{v}_{i}^{t}$
based on the similarity to the previous proxy anchor position
$\mathbf{v}_{i}^{t-1}$ since we assume that the proxy anchor point is
converged to the center point of semantically similar patches if the scope of
the propagation in a single step is limited (i.e., high similarity between
$\mathbf{v}_{i}^{t-1}$ and $\mathbf{v}_{i}^{t}$). With such intuition, we
revise the criterion $\Phi_{i}^{t}$ to be loosened when there is high
reliability ($\mathbf{v}_{i}^{t}$ and $\mathbf{v}_{i}^{t-1}$ are in close
proximity) on the position $\mathbf{v}_{i}^{t}$ as follows:

$\Phi_{i}^{t}=\Phi_{i}^{t-1}-(1-(\mathbf{v}_{i}^{t-1}\cdot\mathbf{v}_{i}^{t}))/\sigma_{\text{pos}},$
(3)

where $\sigma_{\text{pos}}$ is a coefficient used to prevent excessive
reduction of the criterion. Note that $\mathbf{v}_{i}^{0}=\mathbf{f}_{i}$.
Consequently, the positive set at iteration $t$ is expressed with the revised
criterion $\Phi_{i}^{t}$ with the new proxy anchor point $\mathbf{v}_{i}^{t}$
as follows:

$\mathcal{P}_{i}^{t}=\\{j\mid\mathbf{v}_{i}^{t}\cdot\mathbf{f}_{j}>\Phi_{i}^{t},j\in\mathcal{B}\\}.$
(4)

The process above is iteratively performed (Eq. 2 \- Eq. 4) for $T$ times to
discover a reliable zone to sample the positives $\mathcal{P}_{i}^{T}$.

###  3.3 Ambiguity-excluded Negative Set

Along with the importance of gathering trustworthy positive sets for
contrastive learning, preserving the reliability of the negative sets is
another important factor [11]. Accordingly, we utilize the propagated proxy
anchor $\mathbf{v}_{i}^{T}$ as the base to compose the negative set to prevent
the conflict to the positive set $\mathcal{P}_{i}^{T}$.

However, we point out the presence of an ambiguous zone for each anchor where
positives and negatives are intermixed, making it unclear to categorize them
exactly on one side. When the samples in such a zone are considered negatives,
the model may face unwanted repulsion. Derived from such motivation, we
additionally define an ambiguous set for each anchor that is neither included
in the positive set nor the negative set, thereby excluding them from the
learning process.

The method to establish the ambiguous set is similar to the process for
positive set sampling: we update the criterion over the $T$ steps of the proxy
anchor propagation in an anchor-dependent manner and define the set according
to this criterion. However, the key difference lies in how we determine the
initial ambiguity criterion $\Psi^{0}$. Unlike the boundary $\Phi$ for
positive selection, we set $\Psi$ to a small value at the initial step to
serve as a loose boundary since the vicinal areas of an initial anchor
$\mathbf{f}_{i}$ might not be reliable. This criterion is then progressively
raised in the subsequent steps.

Given the initial anchor point $\mathbf{v}_{i}^{0}$ as $\mathbf{f}_{i}$ and
the $t$-th propagated proxy anchor point $\mathbf{v}_{i}^{t}$ through proxy
anchor propagation (Eq. 2), we progressively adjust the ambiguity criterion
by:

$\Psi_{i}^{t}=\Psi_{i}^{t-1}+(1-(\mathbf{v}_{i}^{t-1}\cdot\mathbf{v}_{i}^{t}))/\sigma_{\text{amb}},$
(5)

where $\sigma_{\text{amb}}$ is a coefficient to prevent excessive increase of
the criterion and $\forall_{i}\Psi_{i}^{0}=\Psi^{0}$. Through $t$ steps,
$\Psi_{i}^{t}$ tightens the boundary if the position of the proxy anchor
becomes densely surrounded by the positives. In other words, if the relocated
proxy anchor is positioned in a densely populated area, there is less
probability of having semantically alike samples outside the positive sampling
region formed with Eq. 3.

After determining the ambiguity criterion through $T$ steps, we then proceed
to define the ambiguous set. Using the $T$-th relocated proxy anchor
$\mathbf{v}_{i}^{T}$, the ambiguous set $\mathcal{A}_{i}$ for the anchor
$\mathbf{f}_{i}$ is defined as follows:

$\mathcal{A}_{i}=\\{j\mid(\mathbf{v}_{i}\cdot\mathbf{f}_{j}>\Psi_{i}^{T})\land(\mathbf{v}_{i}\cdot\mathbf{f}_{j}<\Phi_{i}^{T}),j\in\mathcal{B}\\}.$
(6)

Finally, the negative set $\mathcal{N}$ is organized as follows:

$\mathcal{N}_{i}=\\{j\mid(j\notin\mathcal{P}_{i})\land(j\notin\mathcal{A}_{i}),j\in\mathcal{B}\\}.$
(7)

###  3.4 Training Objective

Following existing works [13, 46, 27, 22] in USS, we utilize contrastive
learning objecitve [10, 28]. With the aim of distinguishing the semantically
similar positive set $\mathcal{P}_{i}^{T}$ and dissimilar negative set
$\mathcal{N}_{i}$, the objective is expressed as:

$L^{\text{con}}_{i}=\frac{-1}{|\mathcal{P}^{T}_{i}|}\sum_{p\in\mathcal{P}^{T}_{i}}\text{log}\frac{\text{exp}(\mathbf{z}_{i}\cdot\mathbf{z}_{p}/\tau)}{\sum\limits_{n\in(\mathcal{N}_{i}\cup\mathcal{P}_{i}^{T})}\text{exp}(\mathbf{z}_{i}\cdot\mathbf{z}_{n}/\tau)},$
(8)

where $\tau$ is a temperature parameter, and $\mathcal{P}^{T}_{i}$ and
$\mathcal{N}_{i}$ denote positive and negative sets for $i$-th anchor,
respectively.

##  4 Experiments

###  4.1 Experimental Settings

####  4.1.1 Datasets.

Following previous protocols [27, 13, 22, 46, 19], we evaluate our method on
COCO-stuff [1] and Cityscapes [14], Potsdam-3, and ImageNet-S [19] datasets.
Further details can be found in the Appendix. COCO-stuff is a dataset for
scene understanding tasks, e.g., semantic segmentation, detection, and image
captioning, that consists of 172 classes. Among them, the COCO-stuff benchmark
for USS utilizes 27 classes. Cityscapes is another large-scale dataset for
scene understanding that consists of 30 classes captured across 50 different
cities. Similarly to COCO-stuff, 27 subclasses are used for the benchmark. In
addition, Potsdam-3 contains satellite images that are divided into 3 classes.
Lastly, ImageNet-S is a large-scale dataset which has 1.2 million training
images with 919 semantic classes.

####  4.1.2 Evaluation Protocols.

For COCO-stuff, Cityscapes, and Potsdam-3 datasets, we adopt two evaluation
methods: clustering (unsupervised) and linear probe [22, 46]. Clustering
evaluates the alignment between the prediction and the ground truth with the
Hungarian matching algorithm. On the other hand, the linear probe utilizes an
additional fully connected layer for classification. For both evaluations, we
apply the post-processing step using a Conditional Random Field (CRF) [33] to
refine the predictions. Accuracy (Acc.) and mean Intersection over Union
(mIoU) are used to measure the performances. For the evaluation on the
ImageNet-S dataset, we adopt mIoU with distance matching i.e., the k-nearest
neighbors classifier with k=10, following the evaluation protocol from PASS
[19].

Table 1: Experimental results on COCO-stuff dataset. Method | Backbone | Unsupervised | Linear
---|---|---|---
Acc. | mIoU | Acc. | mIoU
DC [2] | R18+FPN | 19.9 | - | - | -
MDC [2] | R18+FPN | 32.2 | 9.8 | 48.6 | 13.3
IIC [27] | R18+FPN | 21.8 | 6.7 | 44.5 | 8.4
PiCIE [13] | R18+FPN | 48.1 | 13.8 | 54.2 | 13.9
PiCIE+H [13] | R18+FPN | 50.0 | 14.4 | 54.8 | 14.8
DINO [4] | ViT-S/8 | 28.7 | 11.3 | 68.6 | 33.9
TransFGU [55] | ViT-S/8 | 52.7 | 17.5 | - | -
STEGO [22] | ViT-S/8 | 48.3 | 24.5 | 74.4 | 38.3
HP [46] | ViT-S/8 | 57.2 | 24.6 | 75.6 | 42.7
PPAP (Ours) | ViT-S/8 | 59.0 | 27.2 | 76.9 | 46.3
DINO [4] | ViT-S/16 | 22.0 | 8.0 | 50.3 | 18.1
STEGO [22] | ViT-S/16 | 52.5 | 23.7 | 70.6 | 34.5
HP [46] | ViT-S/16 | 54.5 | 24.3 | 74.1 | 39.1
PPAP (Ours) | ViT-S/16 | 62.9 | 26.5 | 76.0 | 43.3

###  4.2 Experimental Results

####  4.2.1 Quantitative Result.

We compare the performances of our method with various baselines [2, 27, 13,
55, 22, 46]. In Tab. 1, we observe that the recent works with the ViT backbone
outperform the other ones, and among them, our approach demonstrates state-of-
the-art performances across all metrics. In particular, our PPAP, equipped
only with the sampling strategies for both the positive and negative samples,
exceeds HP [46] that utilizes contrastive learning also with locality learning
in a task-specific perspective.

Table 2: Experimental results on Cityscapes dataset. Method | Backbone | Unsupervised | Linear
---|---|---|---
Acc. | mIoU | Acc. | mIoU
MDC [2] | R18+FPN | 40.7 | 7.1 | - | -
IIC [27] | R18+FPN | 47.9 | 6.4 | - | -
PiCIE [13] | R18+FPN | 65.5 | 12.3 | - | -
DINO [4] | ViT-S/8 | 34.5 | 10.9 | 84.6 | 22.8
TransFGU [55] | ViT-S/8 | 77.9 | 16.8 | - | -
HP [46] | ViT-S/8 | 80.1 | 18.4 | 91.2 | 30.6
PPAP (Ours) | ViT-S/8 | 82.0 | 19.6 | 90.8 | 31.5
DINO [4] | ViT-B/8 | 43.6 | 11.8 | 84.2 | 23.0
STEGO [22] | ViT-B/8 | 73.2 | 21.0 | 90.3 | 26.8
HP [46] | ViT-B/8 | 79.5 | 18.4 | 90.9 | 33.0
PPAP (Ours) | ViT-B/8 | 83.3 | 21.2 | 91.4 | 36.5

Table 3: Experimental results on Potsdam-3 dataset. Method | Backbone | Unsupervised
---|---|---
Acc. | mIoU
Rand. CNN [27] | VGG11 | 38.2 | -
K-Means [43] | VGG11 | 45.7 | -
SIFT [38] | VGG11 | 38.2 | -
CP [15] | VGG11 | 49.6 | -
CC [26] | VGG11 | 63.9 | -
DeepCluster [2] | VGG11 | 41.7 | -
IIC [27] | VGG11 | 65.1 | -
DINO [4] | ViT-B/8 | 53.0 | -
STEGO [22] | ViT-B/8 | 77.0 | -
HP [46] | ViT-B/8 | 82.4 | 69.7
PPAP (Ours) | ViT-B/8 | 83.2 | 71.0

Table 4: Experimental results on ImageNet-S validation set. $\dagger$: reproduced performance. Method | Backbone | IN-S |  IN-S300 |  IN-S50
---|---|---|---|---
SwAV [3] | ResNet-50 | 15.1 | 22.4 | -
PASS [19] | ResNet-50 | 15.6 | 25.1 | -
DINO† [4] | ViT-S/16 | 7.3 | 12.0 | 22.8
HP† [46] | ViT-S/16 | 8.6 | 14.4 | 29.5
PPAP (Ours) | ViT-S/16 | 25.7 | 37.3 | 59.3

Table 5: Ablation study varying each component.  PPAP | Unsupervised
---|---
TPS | ANS | Acc. | mIoU
- | - | 49.1 | 22.9
✓ | - | 53.7 | 25.0
- | ✓ | 55.5 | 25.1
✓ | ✓ | 62.9 | 26.5

In terms of the backbone, we achieve greater improvements with the ViT-S/16,
which uses larger-sized patch features. We attribute these results to the
robust property of our PPAP. Unlike PPAP, other methods are shown to yield
better results with the small patch size (ViT-S/8 backbone) since the patches
with the larger size are more likely to include a mixture of semantics. Yet,
our proposed PPAP is guided to search for semantically similar patches to
learn its prototypical proxy point and even disregard the patches that retain
an ambiguous relationship with the given anchor. As a result, PPAP achieves
promising results by a margin up to 15.41% and 9.05% compared to HP [46] in
unsupervised Acc. and mIoU, respectively.

Performance comparison on the Cityscapes dataset is displayed in Tab. 3.
Similar to the results on the COCO-stuff dataset, our proposed PPAP achieves
new state-of-the-art results except in one case. These results further verify
the applicability of our components to different datasets.

PPAP also shows improvement on the Potsdam-3 as shown in Tab. 3, but the
difference is modest due to the dataset’s limited three distinct semantic
classes, well clustered by pretrained ViT features. However, our method
demonstrates greater enhancements on datasets with more and less distinct
semantic classes, not as effectively distinguished by the pretrained
backbones.

On the ImageNet-S dataset, PPAP significantly outperforms existing methods,
PASS [19] and HP [46], as shown in Tab. 5. We also conduct experiments on the
subsets ImageNet-S300 and ImageNet-S50, which contain 300 and 50 classes,
respectively. The superior performance of PPAP on these datasets further
demonstrates its scalability compared to existing methods.

Our PPAP algorithm shares similarities with k-means clustering in that both
methods iteratively find and relocate points through averaging. However, there
are three critical differences: 1) PPAP aims to discover patch-wise proxy
anchor rather than having instances within a cluster share a single proxy. 2)
PPAP enables a stable initial relocation process by using only the very close
nearest neighbors of an anchor. 3) PPAP accurately determines the positiveness
and negativeness of each anchor based on the degree of its relocation. In
contrast, simply applying k-means clustering and assigning positive
relationships to all features within each cluster leads to high recall but low
precision in the positive set. For example, we observed that the precision of
positives with k-means is only 22%, even when the number of classes is given
as a prior. In comparison, PPAP achieves a precision of 72% on the COCO-stuff
dataset using the DINO pretrained ViT-S/16 model without class prior.

####  4.2.2 Qualitative Result.

We display our quantitative results in comparison to STEGO [22] and HP [46] in
Fig. 3. Considering the complexity of the scenes, we plot simple to
complicated scenes in order from left to right. To be brief, our PPAP shows
consistent results to have fewer mispredicted pixels compared to the
baselines. Particularly, our proposed method is robust to pixel-wise noises
because each anchor is progressively propagated to search for reliable points
that address the vulnerability of per-sample basis inference in the imperfect
embedding space learned in an unsupervised manner at the image-level.

Figure 3:  Qualitative comparison results of PPAP (Ours) with STEGO and HP on
the COCO-stuff dataset with DINO pretrained ViT-S/8 backbone.

###  4.3 Ablation Study

We provide ablation studies to evaluate the individual components and key
hyperparameters. The primary components under study are: 1) Trustworthy
Positive Set (TPS) obtained based on PPAP and 2) Ambiguity-excluded Negative
Set (ANS). Additionally, we examined the impact of hyperparameters,
specifically: 1) coefficients $\sigma_{\text{pos}}$ and $\sigma_{\text{amg}}$
to regulate the criteria and 2) initial criteria $\Phi^{0}$ and $\Psi^{0}$.

####  4.3.1 Varying Components of PPAP.

We perform an ablation study to assess the individual contributions of each
component, which are presented in Tab. 5. These experiments are performed on
the COCO-stuff dataset with ViT-S/16 backbone. For the baseline, we train the
model using contrastive loss based on positives determined with the initial
positiveness criterion $\mathcal{P}^{0}$ and negatives comprising all
remaining samples. Incorporating the TPS yields improvements of 9.36% in
unsupervised accuracy and 9.17% in mIoU. For the experiment on the third row,
the ambiguous set for contrastive loss is defined after $T$-step propagation,
while the positive set is defined with the initial positiveness criterion
$\mathcal{P}^{0}$. This brings 13.03% and 9.6% enhancements over the baseline.
These results verify that both components significantly contribute to
performance enhancement. Consequently, with both components combined, we
observe a notable overall improvement of 28.11% in accuracy and 15.72% in
mIoU.

(a) $\sigma_{\text{pos}}$

(b) $\sigma_{\text{amb}}$

(c) $\Phi^{0}$

(d) $\Psi^{0}$

Figure 4:  Ablation studies of various coefficients on three different
datasets. Whereas the X-axis denotes the value of each hyperparameter, the
Y-axis shows the performance.

####  4.3.2 Varying $\sigma_{\text{pos}}$ and $\sigma_{\text{amb}}$.

In Fig 4(a) and 4(b), we carry out an ablation study on the coefficient
$\sigma$, which is crucial for controlling the degree of reduction and
increase in positiveness and ambiguity criteria, respectively. A smaller
$\sigma_{\text{pos}}$ leads to a more intensive reduction in the positiveness
criterion, while a bigger $\sigma_{\text{pos}}$ results in a more gradual
reduction. Such property is reflected in the outcomes presented in Fig 4(a). A
smaller $\sigma_{\text{pos}}$ tends to lower the performance due to $FP$
erroneously included in the positive set. On the other hand, moderately larger
$\sigma_{\text{pos}}$ helps to mitigate the aforementioned problem.
$\sigma_{\text{amb}}$ follows the same principle: a smaller value results in a
more substantial increase of the criterion, escalating the possibility of
erroneously repelling $FP$ in the negative set. And a bigger value leads to a
more moderate increase. A proper $\sigma$ can prevent both the excessive
changes in the criteria. We note that it is advisable to choose values around
3 to ensure stability.

####  4.3.3 Varying $\Phi^{0}$ and $\Psi^{0}$.

Ablation studies for varying $\Phi^{0}$ and $\Psi^{0}$ are shown in Fig. 4(c)
and 4(d). These parameters act as the initial criteria for selecting positive
and ambiguous samples and serve as key hyperparameters in our method. For
$\Phi^{0}$, as samples in close vicinity to the anchor are highly likely to
share the same semantics, setting its initial value low incurs the existence
of $FP$ in the positive set. Still, setting the $\Phi^{0}$ too high results in
selecting only a few positives that the anchor relocation is implemented only
within its very close proximity. Regarding $\Psi^{0}$, setting $\Psi^{0}$ too
small significantly reduces the size of the negative set and leads to a
shortage of hard negative samples in the negative set, while too large
$\Psi^{0}$ increases $FP$ in the negative set. We found that values of 0.55
for $\Phi^{0}$ and 0.15 for $\Psi^{0}$ consistently perform well across all
datasets and models. Note that we also found that higher $\Phi^{0}$ requires
more propagation steps (i.e., bigger $T$).

Table 6: Comparing both TP in positive set ($\mathcal{P}$) and FP in negative set ($\mathcal{N}$) between HP and Ours on 3 datasets. Numbers and percentages are the results averaged from subsamples of each dataset. S/8 and B/8 indicate ViT-S/8 and ViT-B/8 backbone, respectively. | Cityscapes (S/8) | COCO-stuff27 (S/8) | Potsdam-3 (B/8)
---|---|---|---
|  HP [46] | Ours |  HP [46] | Ours |  HP [46] | Ours
Number of positives in $\mathcal{P}$ | 78 | 1838 | 142 | 288 | 136 | 194
% of TP in $\mathcal{P}$ | 88.83 | 90.18 | 81.98 | 87.15 | 83.60 | 88.62
Number of negatives in $\mathcal{N}$ | 50K | 43K | 50K | 46K | 50K | 30K
% of FP in $\mathcal{N}$ | 21.33 | 14.22 | 7.34 | 5.92 | 34.08 | 29.01

###  4.4 Trustworthiness of Positive and Negative Sets.

If excessive FP and FN are present in contrastive loss, the model may face
undesired attraction and repulsion, respectively. We contend that mitigating
these issues can enhance the trustworthiness of contrastive learning. To
verify the robust trustworthiness of our method, previously, we conducted a
precision comparison of the positive sets between ours and HP [46] on Fig.
1(b). Notably, ours not only collects more positive samples than HP but also
achieves a higher precision (i.e., ratio of TP). For further demonstration, we
present the ratios of TP in the positive set ($\mathcal{P}$) and FP in the
negative set ($\mathcal{N}$) for three datasets in Tab. 6. As shown, ours got
a higher ratio of TP in $\mathcal{P}$ and lower ratio of FP in $\mathcal{N}$
than HP, even with a much larger size of the positive set. For example, we can
observe that Ours got 23$\times$ more positives with a higher ratio of TP on
Cityscapes dataset with ViT-S/8 backbone. Likewise, we ensure the
trustworthiness of contrastive learning under more reliable positive and
negative sets.

###  4.5 Visualization of Patches in Positive Set

Figure 5:  Comparison of gathered positives between HP and PPAP (Ours) with
the visualizations. In all examples, blue boxes indicate the selected anchor
patch and the red boxes denote the patches that are considered as positive to
an anchor. In (d), yellow dotted circles exist to highlight the region where
the $FP$ are detected.

We illustrate how HP [46] and our proposed PPAP organize the positive samples
for each anchor through visualizations in Fig. 5. As depicted throughout the
visualizations, there is a tendency for our method to collect more positive
patches compared to the baseline. On top of that, we can also observe that
positive samples that are semantically identical are progressively obtained
through the propagation steps in (a), (b), and (c). Lastly, along with the
numerically measured difference in the precision of the positive sets, we find
that falsely detected positives by the baseline in (d) are not considered
positive in ours. To account for such a phenomenon, we claim that the presence
of ambiguous zones enables the filtering of the hard negative samples.

##  5 Conclusion

To tackle the challenge of USS, previous approaches have primarily relied on
exploiting patch relationships to guide the training process. In this work, we
extend this mainstream to ensure the reliability of the gathered guidance.
Specifically, we consider the data distribution around the anchor to identify
densely crowded regions containing samples with similar semantics. By
relocating the proxy anchor to these regions, we expect it to be surrounded by
trustworthy positives, creating a large positive set with high precision. In
addition, we address instance-wise ambiguous zones where samples with similar
and dissimilar semantics coexist. By excluding samples from these regions
during training, we aim to eliminate FP in the negative set, preventing
unstable training. Our state-of-the-art results verify the importance of
ensuring the reliability of the supervision in USS.

Acknowledgements. This work was supported in part by MSIT&KNPA/KIPoT (Police
Lab 2.0, No. 210121M06), MSIT/IITP (No. 2022-0-00680, 2019-0-00421,
2020-0-01821, RS-2024-00437102), and SEMES-SKKU collaboration funded by SEMES.

## References

  * [1] Caesar, H., Uijlings, J., Ferrari, V.: Coco-stuff: Thing and stuff classes in context. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1209–1218 (2018)
  * [2] Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: Proceedings of the European conference on computer vision (ECCV). pp. 132–149 (2018)
  * [3] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems 33, 9912–9924 (2020)
  * [4] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9650–9660 (2021)
  * [5] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062 (2014)
  * [6] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR (2015)
  * [7] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence 40(4), 834–848 (2017)
  * [8] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV). pp. 801–818 (2018)
  * [9] Chen, P., Liu, S., Jia, J.: Jigsaw clustering for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
  * [10] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [11] Chen, T.S., Hung, W.C., Tseng, H.Y., Chien, S.Y., Yang, M.H.: Incremental false negative detection for contrastive learning. arXiv preprint arXiv:2106.03719 (2021)
  * [12] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022)
  * [13] Cho, J.H., Mall, U., Bala, K., Hariharan, B.: Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16794–16804 (2021)
  * [14] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223 (2016)
  * [15] Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE international conference on computer vision. pp. 1422–1430 (2015)
  * [16] Dosovitskiy, A., Springenberg, J.T., Riedmiller, M., Brox, T.: Discriminative unsupervised feature learning with convolutional neural networks. In: Advances in Neural Information Processing Systems. Citeseer (2014)
  * [17] Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene labeling. IEEE transactions on pattern analysis and machine intelligence 35(8), 1915–1929 (2012)
  * [18] Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.: Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3146–3154 (2019)
  * [19] Gao, S., Li, Z.Y., Yang, M.H., Cheng, M.M., Han, J., Torr, P.: Large-scale unsupervised semantic segmentation. IEEE transactions on pattern analysis and machine intelligence 45(6), 7457–7476 (2022)
  * [20] Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: International Conference on Learning Representations (2018), https://openreview.net/forum?id=S1v4N2l0-
  * [21] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33, 21271–21284 (2020)
  * [22] Hamilton, M., Zhang, Z., Hariharan, B., Snavely, N., Freeman, W.T.: Unsupervised semantic segmentation by distilling feature correspondences. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=SaKO6z6Hl0c
  * [23] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16000–16009 (2022)
  * [24] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729–9738 (2020)
  * [25] Huang, Z., Jin, X., Lu, C., Hou, Q., Cheng, M.M., Fu, D., Shen, X., Feng, J.: Contrastive masked autoencoders are stronger vision learners. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
  * [26] Isola, P., Zoran, D., Krishnan, D., Adelson, E.H.: Learning visual groups from co-occurrences in space and time. arXiv preprint arXiv:1511.06811 (2015)
  * [27] Ji, X., Henriques, J.F., Vedaldi, A.: Invariant information clustering for unsupervised image classification and segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9865–9874 (2019)
  * [28] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Advances in Neural Information Processing Systems 33, 18661–18673 (2020)
  * [29] Kim, C., Han, W., Ju, D., Hwang, S.J.: Eagle: Eigen aggregation learning for object-centric unsupervised semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3523–3533 (2024)
  * [30] Kim, J., Shim, K., Lee, I., Shim, B.: Expand-and-quantize: Unsupervised semantic segmentation using high-dimensional space and product quantization. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 2768–2776 (2024)
  * [31] Kim, J., Lee, B.K., Ro, Y.M.: Causal unsupervised semantic segmentation. arXiv preprint arXiv:2310.07379 (2023)
  * [32] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)
  * [33] Krähenbühl, P., Koltun, V.: Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems 24 (2011)
  * [34] Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: European conference on computer vision. Springer (2016)
  * [35] Li, Z., Wang, W., Xie, E., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P., Lu, T.: Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1280–1289 (2022)
  * [36] Lin, G., Milan, A., Shen, C., Reid, I.: Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1925–1934 (2017)
  * [37] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3431–3440 (2015)
  * [38] Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the seventh IEEE international conference on computer vision. vol. 2, pp. 1150–1157. Ieee (1999)
  * [39] Misra, I., Maaten, L.v.d.: Self-supervised learning of pretext-invariant representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6707–6717 (2020)
  * [40] Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: European conference on computer vision. Springer (2016)
  * [41] Noroozi, M., Vinjimoor, A., Favaro, P., Pirsiavash, H.: Boosting self-supervised learning via knowledge transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
  * [42] Park, S., Lee, S., Hyun, S., Seong, H.S., Heo, J.P.: Task-disruptive background suppression for few-shot segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 4442–4449 (2024)
  * [43] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in python. the Journal of machine Learning research 12, 2825–2830 (2011)
  * [44] Pinheiro, P., Collobert, R.: Recurrent convolutional neural networks for scene labeling. In: International conference on machine learning. pp. 82–90. PMLR (2014)
  * [45] Seitzer, M., Horn, M., Zadaianchuk, A., Zietlow, D., Xiao, T., Simon-Gabriel, C.J., He, T., Zhang, Z., Schölkopf, B., Brox, T., et al.: Bridging the gap to real-world object-centric learning (2023)
  * [46] Seong, H.S., Moon, W., Lee, S., Heo, J.P.: Leveraging hidden positives for unsupervised semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19540–19549 (2023)
  * [47] Shi, Y., Siddharth, N., Torr, P., Kosiorek, A.R.: Adversarial masking for self-supervised learning. In: International Conference on Machine Learning. pp. 20026–20040. PMLR (2022)
  * [48] Sick, L., Engel, D., Hermosilla, P., Ropinski, T.: Unsupervised semantic segmentation through depth-guided feature correlation and sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3637–3646 (2024)
  * [49] Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: Transformer for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7262–7272 (2021)
  * [50] Wang, Y., Fei, J., Wang, H., Li, W., Bao, T., Wu, L., Zhao, R., Shen, Y.: Balancing logit variation for long-tailed semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19561–19573 (2023)
  * [51] Wikipedia: False positives and false negatives — Wikipedia, the free encyclopedia. http://en.wikipedia.org/w/index.php?title=False%20positives%20and%20false%20negatives&oldid=1205293928 (2024), [Online; accessed 06-March-2024]
  * [52] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems 34, 12077–12090 (2021)
  * [53] Xie, J., Li, W., Zhan, X., Liu, Z., Ong, Y.S., Loy, C.C.: Masked frequency modeling for self-supervised visual pre-training. In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=9-umxtNPx5E
  * [54] Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., Hu, H.: Simmim: A simple framework for masked image modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9653–9663 (2022)
  * [55] Yin, Z., Wang, P., Wang, F., Xu, X., Zhang, H., Li, H., Jin, R.: Transfgu: a top-down approach to fine-grained unsupervised semantic segmentation. In: European Conference on Computer Vision. pp. 73–89. Springer (2022)
  * [56] Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 (2015)
  * [57] Yun, S., Lee, H., Kim, J., Shin, J.: Patch-level representation learning for self-supervised vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8354–8363 (2022)
  * [58] Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: European conference on computer vision. Springer (2016)
  * [59] Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2881–2890 (2017)
  * [60] Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., Kong, T.: ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 (2021)

##  6 Limitations.

The training guidance derived from patch-wise representation still has
limitations in capturing intricate pixel-level details, especially along
object edges. This issue becomes more pronounced with larger patch sizes, as
ViT-S/16 exhibits lower mIoU compared to ViT-S/8 in this regard.

##  7 Inference

We introduce the overall flow of our model at the inference stage in Fig. 6.
During training, we finetune the last block of the Vision Transformer (ViT)
and the projection head which outputs the projected vector $\mathbf{z}$. While
the projected vector $\mathbf{z}$ is used for the training, we use the feature
from the backbone for the inference as did in [10, 3, 4].

Figure 6:  Illustration of the training and inference process.

##  8 Implementation Details

In line with existing works [22, 46, 55], we utilize a DINO-pretrained ViT as
our backbone network. The embedding dimension of the projection layer ($D$) is
384 for ViT-small and 768 for ViT-base. We set $T$ to 2, 3, and 1 for the
experiments on COCO-stuff, Cityscapes, and Potsdam-3 datasets, respectively.
Tab. 7 shows the hyperparameter set that yields the best performance for each
dataset and backbone. To reduce the training cost, we use only a random subset
of the patch features [31]. Specifically, we use 1/4 of all patch features for
the ViT-S/16 backbone and 1/16 for the other backbones. For the experiments on
the ImageNet-S dataset, we use the same hyperparameter settings as for the
COCO-stuff dataset.

Table 7: Hyperparameter settings for each experiment. | COCO-stuff | Cityscapes | Potsdam-3
---|---|---|---
| ViT-S/8 | ViT-S/16 | ViT-S/8 | ViT-B/8 | ViT-B/8
$\Phi$ | 0.55 | 0.55 | 0.6 | 0.6 | 0.55
$\Psi$ | 0.2 | 0.15 | 0.2 | 0.2 | 0.15
$\sigma_{\text{pos}}$ | 3 | 3 | 3 | 3 | 5
$\sigma_{\text{amb}}$ | 3 | 4 | 3 | 2 | 3

##  9 Additional Study

###  9.1 Study on Effects to Class Frequency

In dense prediction tasks, the variation in the frequency between different
semantics is a very natural phenomenon. However, this typically leads to the
problem of long-tailed data distribution which triggers the large performance
gap between classes of high and low frequencies [50]. To further analyze the
strength of our proposed method, we compare with HP [46] on the effects from
the perspective of the class frequency. Specifically, we divide the classes in
the COCO-stuff dataset into three groups, i.e., Few, Medium, and Many,
according to the data frequency and measure the performances for each group.
Results are reported in Tab. 8. As shown, we find that our method is
particularly notable in learning classes with fewer samples compared to the
baseline. For such a result, we attribute a reason that the number and the
precision of the gathered positives are similar between samples as shown in
Fig. 1 of the main paper.

Table 8: Experimental results considering long-tailed distribution. Method | Few | Medium | Many | All
---|---|---|---|---
HP | 28.9 | 47.1 | 75.6 | 42.0
Ours | 32.9 | 50.9 | 76.2 | 45.6

###  9.2 Different Pretrained Backbones.

The results in Tab. 9 demonstrate consistent performance improvements with
various backbones pretrained in a self-supervised manner (e.g., iBoT [60],
SelfPatch [57]). We observed that backbones trained with inter-image
relationships consistently enhance performance. However, models such as MAE
[23] struggle to preserve globally shared semantics in each patch feature
across all images (e.g., 4.3% of U.mIoU for MAE alone) due to their lack of
inter-image relationship modeling [25].

Table 9: Experimental results with various pretrained backbones.${\dagger}$: reproduced. Method | Backbone | Unsupervised
---|---|---
Acc. | mIoU
iBoT [60] | ViT-S/16 | 39.2 | 11.8
\+ HP† [46] | ViT-S/16 | 53.2 | 23.0
\+ PPAP (Ours) | ViT-S/16 | 62.4 | 26.0
iBoT [60] | ViT-B/16 | 35.7 | 15.0
\+ HP† [46] | ViT-B/16 | 51.1 | 22.4
\+ PPAP (Ours) | ViT-B/16 | 63.4 | 27.6
SelfPatch [57] | ViT-S/16 | 35.1 | 12.3
\+ STEGO [22] | ViT-S/16 | 52.4 | 22.2
\+ HP [46] | ViT-S/16 | 56.1 | 23.2
\+ PPAP (Ours) | ViT-S/16 | 57.8 | 23.7

###  9.3 Contribution of CRF

Conditional Random Field (CRF) utilizes pixel position and RGB color
information to smooth the predicted label of each pixel across its neighboring
pixels, thereby effectively enhancing the performance of semantic segmentation
[33]. Following the previous works [22, 46], we incorporate CRF as a post-
processing step in our method. Tab. 10 presents a performance comparison
between HP [46] and our method, both with and without the application of CRF.
While the use of CRF leads to performance boosts in all experiments, we
highlight the superiority of our PPAP in that it outperforms HP with CRF even
without CRF.

Table 10: Experimental results on COCO-stuff dataset with and without CRF. Backbone | Method | CRF | Unsupervised | Linear
---|---|---|---|---
Acc. | mIoU | Acc. | mIoU
ViT-S/8 | HP [46] | - | 56.2 | 23.9 | 73.7 | 40.4
✓ | 57.2 | 24.6 | 75.6 | 42.7
Ours | - | 57.4 | 26.6 | 76.0 | 45.4
✓ | 59.0 | 27.2 | 76.9 | 46.3
ViT-S/16 | HP [46] | - | 52.7 | 23.2 | 71.9 | 37.1
✓ | 54.5 | 24.3 | 74.1 | 39.1
Ours | - | 59.9 | 25.0 | 74.3 | 42.1
✓ | 62.9 | 26.5 | 76.0 | 43.3

###  9.4 Qualitative Results without CRF

We provide visualizations comparing the predictions of our proposed PPAP with
those of existing methods, i.e., STEGO [22], and HP [46], without CRF. The
results without CRF are shown in Fig. 7 and 8. As can be noticed, we point out
that existing methods tend to be prone to noise, whereas our method
demonstrates robustness against pixel-wise noises even without applying the
CRF.

Figure 7:  Qualitative results without CRF.  Figure 8:  Qualitative results
without CRF.


# Using Deep Reinforcement Learning to Learn High-Level Policies on the ATRIAS
Biped

Tianyu Li1, Akshara Rai2, Hartmut Geyer2, Christopher G. Atkeson2 1Mechanical
Engineering, Carnegie Mellon University, USA<EMAIL_ADDRESS>Institute, School of Computer Science, Carnegie Mellon University, USA,
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Learning controllers for bipedal robots is a challenging problem, often
requiring expert knowledge and extensive tuning of parameters that vary in
different situations. Recently, deep reinforcement learning has shown promise
at automatically learning controllers for complex systems in simulation. This
has been followed by a push towards learning controllers that can be
transferred between simulation and hardware, primarily with the use of domain
randomization. However, domain randomization can make the problem of finding
stable controllers even more challenging, especially for underactuated bipedal
robots. In this work, we explore whether policies learned in simulation can be
transferred to hardware with the use of high-fidelity simulators and
structured controllers. We learn a neural network policy which is a part of a
more structured controller. While the neural network is learned in simulation,
the rest of the controller stays fixed, and can be tuned by the expert as
needed. We show that using this approach can greatly speed up the rate of
learning in simulation, as well as enable transfer of policies between
simulation and hardware. We present our results on an ATRIAS robot and explore
the effect of action spaces and cost functions on the rate of transfer between
simulation and hardware. Our results show that structured policies can indeed
be learned in simulation and implemented on hardware successfully. This has
several advantages, as the structure preserves the intuitive nature of the
policy, and the neural network improves the performance of the hand-designed
policy. In this way, we propose a way of using neural networks to improve
expert designed controllers, while maintaining ease of understanding.

##  I INTRODUCTION

Reinforcement learning is a continuously learning and adapting framework that
can generalize to multiple robots and task scenarios. Such a framework is
essential for moving towards autonomous agents that can respond to changes in
their environment and learn to perform well on new tasks. This potential has
sparked great interest in studying reinforcement learning approaches on
complex, real world problems. Recently deep reinforcement learning (DRL) has
achieved very impressive results and beyond human performance on several
complex long-horizon games such as AlphaGo [1] and even starting with no human
input AlphaGoZero[2]. Similar progress has been achieved in the domain of
continuous control too, dealing with very high dimensional state and action
spaces. For example, Deep Deterministic Policy Gradients (DDPG) [3] and Trust
Region Policy Optimization (TRPO) [4] can solve several control challenges in
the Mujoco simulation environment [5]. [6] show that a variant of the Proximal
Policy Optimization (PPO) [7] can learn to control a very high degree of
freedom humanoid robot in Mujoco, even in very challenging environments.
Similarly, [8] use a variant of DDPG to learn to do dexterous manipulation
with a high degree of freedom hand in simulation.

Figure 1: Our test platform is CMU’s ATRIAS robot in a planar setup.

While simulation results for deep RL are very impressive, the simulation-
hardware gap makes most controllers learned in simulation unsuitable to be
implemented on hardware. This is partly because simulations are not perfect
representations of real systems, but also because learning approaches often
tend to exploit simulation inaccuracies to achieve better performance. Recent
work learns parameters of expert controllers on hardware sample-efficiently,
for example [9] and [10] use Bayesian Optimization. While these are promising,
and robust even to hardware damage, they are limited by the controllers they
can represent. On the other hand, high-dimensional neural network policies can
approximate arbitrarily complex controllers, but too expensive to optimize on
hardware. One approach to learn policies in simulation and deploy on hardware
is domain randomization [11]. It can be used to learn robust neural network
policies in simulation by applying random perturbations to the dynamics and
other properties of the simulator. This approach has been applied to
manipulation problems [12], as well as quadrupedal locomotion problems [13].
However, typical domain randomization can lead to controllers that perform
worse than controllers trained without randomization [13], as well as make the
learning problem much harder [14] taking over 100 years of simulation time to
learn successful policies.

Domain randomization from scratch is especially challenging for under-actuated
bipedal robots, as the basin of stability around a controller is typically
very small. As a result, it is very difficult to randomly explore the space of
controllers to find successful controllers that stabilize a wide range of
dynamic models, and other disturbances. Moreover, the learned controllers
would typically be quasi-static in nature, similar to [11] in behavior, which
is not very impressive in performance.

In this work, we study if a high-fidelity simulator and structured neural
network controllers can eliminate the need for domain randomization in
learning controllers for bipedal robots. We use a high-fidelity simulation of
the ATRIAS robot [15] with ground height disturbances to learn a walking
controller in simulation. Then we test these controllers on an actual bipedal
robot and evaluate if they are capable of realizing a stable walking gait on
hardware.

We experiment with different controller structures, and study their effect on
transfer from simulation to hardware, without any domain randomization. We
show that structured neural network controllers have a fast training rate in
simulation as well as higher rate of transfer to hardware. The structure also
gives the user the power to modify the controller, if required, without having
to re-train the neural network from scratch.

The main contributions of this paper are as follows:

  1. 1.

We demonstrate a successful neural network policy learned using deep
reinforcement learning on a bipedal robot hardware

  2. 2.

We study the effect of action space on the success of transfer between
simulation and hardware

  3. 3.

We present a way of using deep reinforcement learning and neural networks to
improve user-designed policies, while maintaining their intuitive structure

##  II Learning Locomotion Controllers

In this section, we give a brief introduction of deep reinforcement learning
and explain how we train our neural network policies in simulation.

###  II-A Background

We formulate our locomotion control problem as a Markov Decision Process (MDP)
which can be represented as a tuple: $M=\left\\{S,A,r,\tau,T\right\\}$. Here
$S\subseteq\mathbb{R}^{n}$ is the set of states, $A\subseteq\mathbb{R}^{m}$ is
the set of actions, $r:S\times A\to\mathbb{r}$ is the reward function,
$\tau:S\times A\to S$ is the transition function which is robot’s dynamics in
our setting, $T$ is the maximum episode length. We wish to find a policy
$\pi:S\to A$ that can maximize the expected sum of the reward over a
trajectory:

$\eta(\pi)=\mathbb{E}_{\pi}[\,\displaystyle{\sum_{i=1}^{T}}\,r_{i}\,]$

We use an actor-critic framework to solve this problem, similar to [8]. The
policy is modeled using a neural network that takes as input the partial state
of the robot and outputs desired control actions. A second neural network
models the value function of the robot, which is the expected total reward
from the current state.

###  II-B Imitation Learning

Exploration is one of the challenges in RL methods; we need to try different
actions in order to learn from and learn about their consequences. If we start
from nothing, the learning process could be very slow [16] since the
algorithms put a lot of effort on exploring state-action space. One way to
reduce this issue is using behavior cloning [17]. Behavior cloning learns a
policy over state-action pairs from expert trajectory [18] by finding the
policy parameters $\theta$ that solve the following maximum-likelihood
optimization problem [16]:

$maximize_{\theta}\sum_{(s,a^{*})\in\rho_{D}}ln\,\pi_{\theta}(a^{*}|s)$ (1)

where $\rho_{D}$ indicates the expert’s trajectory distribution, with $a^{*}$
is actions that expert took, and $s$ are the states visited.

Similarly, rewards accumulated during the expert’s demonstrations can be used
to pre-train a Q-function by minimizing TD error [19]. As Q-function is the
guidance for policy update, a well pre-trained Q-function benefits actor
training as well.

In practice, imitation learning does not guarantee that the cloned policy can
work as good as the expert due to compounding error caused by distributional
shift in states seen during training and testing [16], [20]. Moreover, the
expert controller might not perform sufficiently well in settings where it is
difficult to design a controller. For example, while our expert controller is
very stable on flat ground walking, rough ground seems to present a challenge.
This means that the learned policy actually has to be updated to perform well
in this situation. Nevertheless, the expert is a good initialization for the
policy and makes the learning process much faster.

###  II-C Proximal Policy Optimization (PPO)

We used PPO [6], [7] as our deep RL algorithm. PPO is a stable on-policy
method that uses importance sampling and a clipped objective function to
update policy parameters. As a result, the updated policy is close to the
initializing policy and the training is stable.

##  III CONTROLLERS DESCRIPTION

In this section, we first describe a feedback based reactive policy which is
also our expert controller, used for initializing our neural network policies.
Then we introduce two types of neural network controllers using a similar
state and action representation as the feedback based reactive policy. The
first neural network controller has a general architecture, with little
structure, while the second has a very similar structure to the expert.

###  III-A Feedback Based Reactive Policy (Expert Controller)

This feedback based reactive policy [21], is designed for controlling the CoM
height and torso pitch in stance, and the leg placement location in swing:

$\displaystyle F_{x}=K_{pt}(\theta_{des}-\theta)+K_{dt}(-\dot{\theta})$ (2)
$\displaystyle F_{z}=K_{pz}(z_{des}-z)+K_{dz}(-\dot{z})$ (3) $\displaystyle
x_{p}=k(v_{act}-v_{tgt})+0.5\cdot v\cdot T$ (4)

Here, $F_{x}$ is the desired horizontal ground reaction force (GRF), $K_{pt}$
and $K_{dt}$ are the feedback gains on the torso pitch angle $\theta$ and
velocity $\dot{\theta}$. $F_{z}$ is the desired vertical GRF, $K_{pz}$ and
$K_{dz}$ are the feedback gains on the CoM vertical height $z$ and velocity
$\dot{z}$. $\theta_{des}$ and $z_{des}$ are desired the torso lean and
vertical CoM height. $x_{p}$ is the desired swing leg foot placement location,
$v_{act}$ is the current measured horizontal velocity of CoM and $v_{tgt}$ is
the target velocity. T is the swing time and the term $0.5\cdot v\cdot T$ is a
feedforward term similar to the Raibert controller [22]. Our controller tries
to maintain a fixed CoM height and torso pitch in stance, and regulate a
forward velocity through leg placement in swing.

The desired GRFs ($F_{x}$,$F_{z}$) are then sent to an inverse dynamics model
of ATRIAS to generate desired motor torques that realize the GRFs [23]. These
torques are tracked with low level motor velocity-based feedback loop that
generates the desired current in the motors.

Figure 2: Pipeline of the expert controller for controlling ATRIAS.

The swing trajectory starts from the current leg position and terminates at
the calculated foot position $x_{fp}$ with ground-speed matching. Inverse
kinematics then translates this trajectory to desired joint positions and
velocities. The joints are controlled by sending a velocity commands to the
robot.

This feedback based reactive policy can walk in simulation without any
perturbations. However, this controller is not robust enough, since in an
environment with ground height disturbances or torque noise it often falls
after a few steps. In addition, this controller cannot be directly implemented
on hardware. Due to differences between the simulation and hardware such as
error in models and parameters, that do not transfer, this controller is not
robust enough to overcome these differences. This implies that further
training is essential to achieve a better performance. We discuss two
different types of controller that can be trained to obtain a better and more
robust policy.

###  III-B Neural Network Policy

In our first setting, we create a general neural network policy without much
structure. Similar to the expert controller, we design the action space of the
neural network to be horizontal and vertical ground reaction forces (GRFs)
$F_{z}$, $F_{x}$ and the swing leg foot place location $x_{p}$. The input
space is tha state of the CoM:
(x,$v_{act}$,$z$,$\dot{z}$,$\theta$,$\dot{\theta}$). This means that the
structure of the neural network policy becomes:

$\pi(x,v_{act},z,\dot{z},\theta,\dot{\theta})_{NN}\rightarrow(F_{z},F_{x},x_{p})$
(5)

The stance and swing control pipeline is similar to the expert and shown in
Figure 3.

Figure 3: Pipeline of our first method for controlling ATRIAS. Here
$\tau_{RF},\tau_{RB},\tau_{LF},\tau_{LB}$ are torques in the right leg’s front
and back and left leg’s front and back motors.

While this policy still has the information about dynamics constraints of the
robot through the inverse dynamics and inverse kinematics blocks, it loses the
well-defined structure of the expert. This means that the output of the neural
network is unpredictable, and the user does not have a lot of control over the
behavior of the robot. Roboticists typically prefer controllers that can be
understood and predictable, as well as can be modified if needed. In our
second controller, we try to emulate this shared autonomy, which lets the user
control the overall behavior of the robot, and the neural network helps
improve the user-designed policy through deep RL.

###  III-C Neural Network in the Heuristic Policy

In our second setting, instead of directly predicting the vertical and
horizontal ground reaction force $(F_{z},F_{x},x_{p})$ as well as desired
swing leg foot place location $x_{p}$, we use a neural network as part of the
feedback based reactive stepping policy, described in Section III-A. While the
original policy has a fixed desired torso lean $\theta_{des}$, desired CoM
height $z_{des}$ and fixed structure for $x_{p}$, we learn these as a function
of the state of the CoM using a neural network. The neural network now takes
the state of the CoM as input, and predicts the desired torso pitch, CoM
height and an offset on the footstep location.

$\displaystyle\pi(x,v_{act},z,\dot{z},\theta,\dot{\theta})_{HP}\rightarrow(\theta_{NN},z_{NN},x_{NN})$
(6)

When inserted into the expert controller, this gives a structured neural
network policy, where neural network outputs are used as part of a heuristic
policy. It is worth noting that this policy is capable of generating the same
outputs as the general neural network policy. For example, for any desired
$F_{x}$ profile, it is always possible to find a $\theta_{NN}$ given $K_{pt}$,
$K_{dt}$, $\theta$ and $\dot{\theta}$. The pipeline is illustrated in Figure
4.

$\displaystyle F_{x}=K_{pt}(\theta_{NN}-\theta)+K_{dt}(-\dot{\theta})$ (7)
$\displaystyle F_{z}=K_{pz}(z_{NN}-z)+K_{dz}(-\dot{z})$ (8) $\displaystyle
x_{p}=k(v_{act}-v_{tgt})+0.5\cdot v\cdot T+x_{NN}$ (9)

Figure 4: Pipeline of our second method using a heuristic policy and neural
network for controlling ATRIAS.

There are several benefits to having such structure in our policies. First and
foremost, the user can predict the behavior and understand the outputs of this
policy. This is important for policies implemented on robots to ensure safety
of the robot and its environment. Moreover, if the user wants to test a
slightly different setting than the simulation, she can easily tune other
parameters of this policy, keeping the neural network fixed. Lastly, if the
policy fails on hardware, it is easy for the user to understand why that might
be, and even possible to fix other parts of the controller without re-training
the neural network.

##  IV EXPERIMENT

In this section we describe our experimental platform - the ATRIAS robot,
followed by our simulation and hardware experiments. Our experiments compare a
general neural network policy with a more structured policy, where the neural
network helps modulate an expert controller. We compare both policies in
simulation and hardware.

###  IV-A ATRIAS robot

The ATRIAS robot (Figure.1) is our test bipedal robot platform. It weights
about 64kg, with most of its mass concentrated around its torso, and its
rotational inertia about its center of mass (CoM) is about 2.2$kgm^{2}$. In
this work, we only focus on planar walking, enforced with the help of a boom.
Details of the robot and our simulator can be found in [15]. The simulator is
designed in MATLAB Simulink environment and is high-fidelity. It has non-
linear contact models, as described in [15] for realistic contact forces, as
well as detailed actuator dynamics, like spring deflections and gear dynamics.

###  IV-B SIMULATION EXPERIMENTS

We trained our neural network controllers in simulation before implementing on
hardware. This is because we use Proximal Policy Optimization (PPO) as our
learning algorithm. PPO needs to collect a large amount of data points at each
iteration. When our current policy is not good enough, we would not be able to
collect much data since the robot might fall at the very beginning of
experiment. In order to collect enough number of data points, we need to do a
large number of trials, making it near impossible to train on hardware.
However, since we train our policies in simulation, the resulting controllers
might work well in simulation environment but when implemented on hardware,
they might fall. In order to overcome this issue, we add ground height
disturbances to learn robust controllers in simulation.

Figure 5: Simulation of ATRIAS with ground height disturbance. Different
colors indicate different ground height disturbances. Maximum ground height
disturbance is $\pm$ 10 cm.

The reward function is defined focusing on matching the desired walking speed
and preventing large angular velocity of the torso. We also add a large
penalty if the controller falls:

$reward_{t}=\left\\{\begin{array}[]{rcl}-C_{1}\cdot(v_{act,t}-v_{tgt})^{2}-C_{2}\cdot(\dot{\theta}_{t})^{2}+1,\
{if\ walk}\\\ -C_{3}*T_{sim},\ {if\ fall}\end{array}\right.$ (10)

where $v_{act,t}$ is the vector is the actual forward velocity of the robot at
time $t$, $v_{tgt}$ is the target velocity, $\dot{\theta}_{t}$ is the angular
velocity of torso pitch, $T_{sim}$ is the simulation time, $C_{1},C_{2},C_{3}$
are positive fixed parameters. This kind of cost function is similar to prior
work on optimizing locomotion controllers, and it can easily distinguish
points that walk from points that fall. The longer the controller can survive,
the closer to the target speed, the less torso pitch the higher total reward
the controller can get. In this setting,$C_{1}=1$, $C_{2}=0.3$, $C_{3}=0.01$.
In our observation, often the highest scoring controllers in simulation tend
to exploit simulation inaccuracies. For example, some controllers tend to jerk
the torso to achieve speeds closer to the target speed. While these perform
well in simulation, they tend to fail on hardware.

####  IV-B1 Neural Network Policy

The first set of simulation experiments were with the Neural Network Policy
(NN Policy), described in Section III-B. The first step for Neural Network
Policy was behavior cloning. After initialization, our NN policy could walk on
flat ground, but it had difficulties with ground height disturbances. This was
unsurprising because the ’expert controller’ also cannot walk on ground with
ground height disturbance. Hence, we had to train the policy further to
achieve acceptable performance.

Figure 6: Sequence of frames from a single walking cycle (left to right).

We trained 10 different NN policies, the rewards for which are shown on Figure
8. The rewards shown is the averaged reward of one iteration which consisted
on 30,000 data points. As shown in the plot, the reward keeps growing as
training goes on, starting from the average cost of the expert. This shows
that PPO achieves a very stable training with little to no forgetting of the
initial expert.

Figure 7: A time lapse of Neural Network Policy walking around the boom with
ground height disturbance in simulation.  Figure 8: Training plot of Neural
Network Policy (green) and Neural Network with Heuristic Policy (blue). Each
of them is averaged of 5 trials data. In our training, we collected 30000 data
in one iteration. For each simulation episode, we simulated 10 seconds if the
controller keeps walking, and in simulation our time step was 0.001s. Thus, we
could collect no more than 10000 data in a single episode, which also means in
one iteration we needed data at least from 3 different episodes. By collecting
data from more than one episodes, we could average the total rewards to
determine the performance of the controller, also eliminating ‘lucky’ runs.
Figure 9: A plot of the stance leg SEA torques measured during one run of the
NN policy on hardware. The torques are 0 during swing and follow a double hump
pattern in stance. Note the high initial torques in the first two steps as
compared to the rest of the walking cycle.

The NN policy achieved good reward improvement compared to the initialization.
After training, the NN policy could walk with ground height disturbance with a
reasonable success rate. Figure 6 and Figure 7 shows the NN policy walking
with ground height disturbance.

####  IV-B2 Neural network with heuristic policy

The second set of simulation experiment uses a neural network in a heuristic
policy, described in section III-C. By initializing this policy to the target
values of the ’expert controller’, the initial policy was able to walk on flat
ground but cannot walk on rough ground.

The training plot of Heuristic Policy is shown on Figure 8. As shown on the
plot, the Heuristic Policy reward improves faster than the NN policy but
reaches its peak in very few iterations. It then remains at that reward for
the rest of iterations. The initial fast improvement is expected because as
the heuristic policy uses the structure of the feedback-based reactive
stepping policy, the search for reasonable parameters is simple. Hence,
walking controllers are found much faster. However, the same structure also
imposes a strong bias on the search, and as a result, it is hard to find
controllers that can perform as well as the pure NN controller in simulation.
However, as we describe later, even though the best reward achieved by
Heuristic controllers is lower, the rate of transfer to hardware is much
higher than the NN policy.

Figure 10: A time lapse of ATRIAS walking around the boom during a run of a NN
with heuristic policy.

###  IV-C HARDWARE EXPERIMENTS

We tested 5 NN policies and 5 heuristic policies in our first set of hardware
experiment, with the reward function described in Function 10.

There is a large mismatch between simulation and hardware starting conditions
for our system. Starting from rest is a significant challenge for bipedal
robots, especially underactuated robots like ATRIAS. ATRIAS is incapable of
balancing by standing in place, and needs to continue stepping to stay
upright. Typically, a start-up routine is designed for these robots to reach
some initial walking gait, and then the controller to be tested is initiated.
For example [24] start by executing a stepping motion in air, and a human
holds on to the robot as its lowered onto the ground. We train our controllers
to start from rest in both simulation and hardware, to avoid having to design
a start-up routine. However, our simulation leg is not initially loaded, so
the simulation has to make an initial push to keep the CoM from going too low.
On the other hand, on hardware the robot starts in position control in the air
and then is lowered. So as stance starts, there is already sufficient force in
the leg, and the initial push (from simulation) makes the robot leave ground.
Such a controller falls on hardware. This behavior is illustrated in Figure 9
where the first two steps on hardware experience higher leg forces than the
rest of the steps.

####  IV-C1 Neural network policy

2 out of 5 of the NN policy could walk on hardware. But all of the NN policies
shared the same issue that at the beginning of walking they would jump up.
This behavior can also be seen in our simulation. The initializing ’expert
controller’ has a jump up behavior at the beginning of walking, and our NN
policy also inherits this behavior. In simulation, this behavior would not
lead to falling, but on hardware, ATRIAS would fall as soon as it tries to
start walking because of the initial jump. On hardware, 2 out of 5 controllers
could overcome the initial disturbance and then started normal walking
pattern. 3 out 5 controllers, however could not recover from this disturbance.
This led to a $40\%$ rate of transfer of learned policies between simulation
and hardware.

####  IV-C2 Neural network with heuristic policy

4 out 5 controllers trained with neural network as part of the heuristic
structure were able to walk on hardware. There are a few reasons for the
success of the NN with heuristic structure policy, over the pure NN policy.
Firstly, the initial discrepancy between simulation and hardware leg force is
already compensated for by the heuristic structure. In the simulation, the NN
predicts a desired height, and the feedback on this desired height results in
a high leg force, as the actual height of the robot is lower. On hardware, the
actual height of the robot is close to this desired height, so it
automatically reduces the initial leg force. Unlike pure NN policy, where the
forces were high on both simulation and hardware, now the initial forces are
only high when the actual state of the robot is different from the desired.
This structure in our policy, hence, makes the generalization from simulation
to hardware very simple and intuitive.

Secondly, since the structure of the policy is readable by an expert, the
expert can still edit the policy after being trained in simulation. For
example, if the NN was trained on a lower velocity but the expert wanted to
run the robot on a higher velocity, the heuristic structure allows to directly
edit this target. Similarly, we observed that some of our controllers were
falling because of speeding, and we increased the feedback gain $k$ on
velocity feedback, described in Equation 9. This modification significantly
improved the success rate of policies learned in simulation on hardware.

These experiments highlight the advantages of structure in learned policies.
While the NN helped us improve our heuristic policy, as shown in Figure 8, by
learning a state-dependant desired height, pitch and footstep location, it
also kept the intuitive structure of the heuristic. So, it was able to better
reject disturbances between simulation and hardware, as well as be modified
for slightly different test situations, without needing re-learning. This
allowed a $80\%$ rate of transfer of learned policies between simulation and
hardware.

##  V CONCLUSIONS AND DISCUSSION

In this work, we used deep reinforcement learning to learn two neural network
policies to control the ATRIAS biped in simulation, and study their
effectiveness at transferring to hardware. One of the policies uses a general
neural network, while the second builds on the structure of a heuristic
policy. We show that introducing structure into neural network policies can
vastly improve the transfer rate of policies between simulation and hardware.
A structured policy can guarantee the safety of the robot, as well as, allow a
user to intervene if the controller fails. In our experiments, such a
controller led to a $80\%$ rate of transfer between simulation and hardware,
as compared to $20\%$ for a traditional NN policy. These results show that
incorporating neural networks into heuristic policies can help improve the
performance of the policy, and increase the rate of transfer of NN policies
between simulation and hardware.

In our hardware setup, the initial condition in simulation and hardware was
very different, which led to a lot of controllers failing on hardware. This is
a common problem in locomotion systems, especially bipeds. One way of learning
more robust policies to such disturbances can be to use cost functions that
encourage conservative controllers. We conducted such experiments with a cost
that penalized high torques and found that the transfer improved to a $100\%$
for NN policy with heuristic structure. In general, it is worth studying
control frameworks that can switch between conservative and high performing
controllers, based on the mismatch between simulation and hardware. For
example, we might want to start with a conservative controller, and then
transition to a high-performance controller once we have achieved a cyclic
gait. We leave this for future work.

## References

  * [1] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot _et al._ , “Mastering the game of go with deep neural networks and tree search,” _nature_ , vol. 529, no. 7587, p. 484, 2016.
  * [2] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton _et al._ , “Mastering the game of go without human knowledge,” _Nature_ , vol. 550, no. 7676, p. 354, 2017\.
  * [3] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [4] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in _International Conference on Machine Learning_ , 2015, pp. 1889–1897.
  * [5] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on_. IEEE, 2012, pp. 5026–5033.
  * [6] N. Heess, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, A. Eslami, M. Riedmiller _et al._ , “Emergence of locomotion behaviours in rich environments,” _arXiv preprint arXiv:1707.02286_ , 2017.
  * [7] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [8] A. Rajeswaran, K. Lowrey, E. V. Todorov, and S. M. Kakade, “Towards generalization and simplicity in continuous control,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 6550–6561.
  * [9] R. Antonova, A. Rai, and C. G. Atkeson, “Deep kernels for optimizing locomotion controllers,” in _Conference on Robot Learning_ , 2017, pp. 47–56.
  * [10] A. Cully, J. Clune, D. Tarapore, and J.-B. Mouret, “Robots that can adapt like animals,” _Nature_ , vol. 521, no. 7553, pp. 503–507, 2015.
  * [11] I. Mordatch, K. Lowrey, and E. Todorov, “Ensemble-cio: Full-body dynamic motion planning that transfers to physical humanoids,” in _Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on_. IEEE, 2015, pp. 5307–5314.
  * [12] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” _arXiv preprint arXiv:1710.06537_ , 2017.
  * [13] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, “Sim-to-real: Learning agile locomotion for quadruped robots,” _arXiv preprint arXiv:1804.10332_ , 2018.
  * [14] OpenAI, :, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, “Learning Dexterous In-Hand Manipulation,” _ArXiv e-prints_ , Aug. 2018.
  * [15] W. C. Martin, A. Wu, and H. Geyer, “Robust spring mass model running for a physical bipedal robot,” in _Robotics and Automation (ICRA), 2015 IEEE International Conference on_. IEEE, 2015, pp. 6307–6312.
  * [16] A. Rajeswaran, V. Kumar, A. Gupta, J. Schulman, E. Todorov, and S. Levine, “Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,” _arXiv preprint arXiv:1709.10087_ , 2017.
  * [17] D. A. Pomerleau, “Efficient training of artificial neural networks for autonomous navigation,” _Neural Computation_ , vol. 3, no. 1, pp. 88–97, 1991.
  * [18] J. Ho and S. Ermon, “Generative adversarial imitation learning,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 4565–4573.
  * [19] A. Sendonaris and C. G. Dulac-Arnold, “Learning from demonstrations for real world reinforcement learning,” _arXiv preprint arXiv:1704.03732_ , 2017.
  * [20] S. Ross and D. Bagnell, “Efficient reductions for imitation learning,” in _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , 2010, pp. 661–668.
  * [21] A. Rai, R. Antonova, S. Song, W. Martin, H. Geyer, and C. G. Atkeson, “Bayesian optimization using domain knowledge on the atrias biped,” _arXiv preprint arXiv:1709.06047_ , 2017.
  * [22] M. H. Raibert, _Legged robots that balance_. MIT press, 1986.
  * [23] A. Wu and H. Geyer, “Highly robust running of articulated bipeds in unobserved terrain,” in _Intelligent Robots and Systems (IROS 2014), 2014 IEEE/RSJ International Conference on_. IEEE, 2014, pp. 2558–2565.
  * [24] C. Hubicki, J. Grimes, M. Jones, D. Renjewski, A. Spröwitz, A. Abate, and J. Hurst, “Atrias: Design and validation of a tether-free 3d-capable spring-mass bipedal robot,” _The International Journal of Robotics Research_ , vol. 35, no. 12, pp. 1497–1521, 2016.