11institutetext: Sharif University of Technology, Tehran, Iran
22institutetext: BmDeep Group<EMAIL_ADDRESS>
# Multi-scale Regional Attention Deeplab3+: Multiple Myeloma Plasma Cells
Segmentation in Microscopic Images

Afshin Bozorgpour 112∗2∗ Reza Azad 112∗2∗ Eman Showkatian 22 Alaa Sulaiman
2†2†

###### Abstract

Multiple myeloma cancer is a type of blood cancer that happens when the growth
of abnormal plasma cells becomes out of control in the bone marrow. There are
various ways to diagnose multiple myeloma in bone marrow such as complete
blood count test (CBC) or counting myeloma plasma cell in aspirate slide
images using manual visualization or through image processing technique. In
this work, an automatic deep learning method for the detection and
segmentation of multiple myeloma plasma cell have been explored. To this end,
a two-stage deep learning method is designed. In the first stage, the nucleus
detection network is utilized to extract each instance of a cell of interest.
The extracted instance is then fed to the multi-scale function to generate a
multi-scale representation. The objective of the multi-scale function is to
capture the shape variation and reduce the effect of object scale on the
cytoplasm segmentation network. The generated scales are then fed into a
pyramid of cytoplasm networks to learn the segmentation map in various scales.
On top of the cytoplasm segmentation network, we included a scale aggregation
function to refine and generate a final prediction. The proposed approach has
been evaluated on the SegPC2021 grand-challenge and ranked second on the final
test phase among all teams.

###### Keywords:

Myeloma Plasma Cell Segmentation Attention Deeplabv3+ Deep Learning SegPC2021
Grand Challenge.

††footnotetext:  Support: This work is fully supported by BmDeep Group.
${\dagger}$ To whom correspondence should be addressed: E-mail:
<EMAIL_ADDRESS>
$\ast$ These authors contributed equally to this work.

##  1 Introduction

Cancer happens when the cells start to grow out of control and spread to
healthy surrounding tissue. Myeloma, also known as multiple myeloma, is a type
of blood cancer that arises from plasma cells in the bone marrow [19, 12].
More specifically, bone marrow is a kind of soft tissue found inside some part
of larger bones in human body. Different types of blood cells such as red
blood cells, white blood cells, and platelets are made in the bone marrow
[13]. Plasma cells developed by the B lymphocytes (type of white blood cells)
form part of the body’s immune system. In order to fight the infections,
antibodies, also known as immunoglobulin, are produced by normal plasma cells.
In myeloma cancer plasma cells crow in the bone marrow in a way there is no
space for normal red cells, white cells, and platelets. Myeloma begins to
develop when the DNA is damaged or changed during the production of new plasma
cells. These abnormal plasma cells (myeloma cells) will spread in a different
part of bone marrow and produce more abnormal cells. Myeloma cells will
produce a large number of paraproteins (type of antibody) which are useless
and unable to fight the infections [6]. Unlike other cancers, myeloma will not
form a tumor or lump but it will lead to the accumulation of abnormal plasma
cells in the bone marrow and paraproteins in the body. Multiple myeloma is
referred to the situation when myeloma cancers affect multiple parts of the
body [1]. Since myeloma cell can be differentiated from normal plasma cells
based on histology and morphological features, it is a common method to
diagnose multiple myeloma cancer through the aspirate slide images [15, 18,
17]. In this method, at first, blood samples will be extracted from bone
marrow by using the injection of the needle onto the bone. The extracted blood
sample will be transferred to a slide and stained using hematoxylin and eosin.
Abnormal plasma cells will be detected and marked using manual microscopic
visualization (sample is shown in figure 1). Finally, based on the estimation
of the normal plasma cells in bone marrow, the presence or absence of myeloma
cancer will be concluded [16].

Figure 1: Some samples of myeloma plasma cells microscopy images [10], where
the myloma cancer cells are detected and highlighted with colored boundary.

Although manual inspection of stained slide images is a gold standard of
diagnosis of myeloma cancer, it is time-consuming and prone to inter and
intraobserver variation. These limitations could be compensated by the use of
advanced digital image processing techniques such as object detection and
segmentation. Automation of the abnormal plasma cell detection alongside
expert pathologist decision could lead to the reduction of diagnosis time and
workload of the pathologist. To this end, in this paper, we propose a deep
network that utilizes a pyramid of Attention Deeplabv3+ model in a regional-
based manner to segment each instance of a myeloma cancer cell. We utilize our
approach on the multiple mylomia plasma cell segmentation challenge which
provided by [11, 9, 8, 10]. Our contribution is summarized as follows:

  * •

Second ranking on the SegPC2021 challenge for multiple myeloma cancer cell
segmentation.

  * •

Regional base instance segmentation approach.

  * •

guiding cytoplasm segmentation network with additional nucleus mask as a
supervisory signal.

##  2 Related Work

Advanced image processing and machine learning methods such as image
classification, object detection, and segmentation could have promising
applications in various medical domain [3, 2, 7]. Detection and segmentation
of abnormal cells in microscopic images have been proposed by several
researchers in recent works. For instance, Vyshnav et al. used a deep
learning-based approach for multiple myeloma cancer detection in stained
microscopic images. They compared the performance of Mask R-CNN and U-Net in
segmentation and inferred that Mask R-CNN has a very better performance than
U-Net in myeloma cell segmentation [23]. Authors of [21] used convolutional
neural networks to classify the normal and abnormal plasma cells in stained
microscopic images. For the classification task, they used AlexNet [14] to
extract features from microscopic images and then used Support Vector Machine
(SVM) for the classifier. Their novelty was the preprocessing stage where they
used a median filter for each R, G, and B color channel (on the importance of
color space [5]) individually and linear contrast stretching for the color
enhancement. Then they compared their result for classification with state of
art techniques. They reported that their networks reached $89\%$ sensitivity
for the cell classification. Vuola et al. used Mask-RCNN and U-Net ensembled
for nuclei segmentation in microscopic images. They inferred that Mask-RCNN
and U-Net have similar results on the nucleus segmentation task. They reported
that U-Net has better performance in nucleus segmentation than Mask R-CNN in
the term of similarity index. On the other hand, the Mask R-CNN has better
performance in the term of precision assessment. Finally, they concluded that
an ensembled model improves the model performance in nucleus segmentation
[22]. Saeedizadeh et al. used a bottleneck algorithm, modified watershed, and
SVM for myeloma cell detection in microscopic images. At first, they separated
the white blood cell from red blood cells by use of the color normalization.
After that the used thresholding technique to separate the nucleus from the
cytoplasm and by the use of watershed and bottleneck algorithm they separated
the connected cells. Finally, by using the series of decision rules and the
use of an SVM classifier they achieved the sensitivity of $96.52\%$ and
precision of $95.28\%$ in recognition of myeloma cells [20]. Even though the
literature work gained promising results, their applicability in the multiple
myeloma cancer cell segmentation is limited due to the challenges such as
overlap between cytoplasm’s of instances, the fuzzy boundary of the cytoplasm,
and overlaying of one nucleus on another cytoplasm in microscopic images. To
mitigate these limitations, we propose to regional attention deep model to
segment each cell with precise attention.

##  3 Methodology

A general diagram of the proposed structure is depicted in figure 2. The
proposed methods consist of two stages: in the first stage, the nucleus
segmentation network extracts all the nucleus instances. Then each instance
fed into a multiscale cytoplasm segmentation network. This network utilizes
the Attention Deeplab3+ model to segment the cytoplasm area. In the next
subsections, we will elaborate on each part in more detail.

Figure 2: Proposed regional Attention Deeplabv3+ model for multiple myeloma
plasma cells segmentation. The proposed method applies a U-net structure to
learn the segmentation map for each nucleus instance then it utilizes multi-
scale attention deeplabv3+ model to generate the segmentation mask for
cytoplasm.

###  3.1 Nucleus Segmentation

In the proposed architecture instead of jointly learning the segmentation of
the nucleus and cytoplasm mask, we utilize a two-stage strategy. Our main
motivation is to use the detected nucleus instance as a supervisory signal for
the cytoplasm instance segmentation to deal with overlapped areas. In other
words, the man objective of the first stage is to extract all the possible
nucleus instances from the input image. Then each extracted nucleus instance
alongside the cropped image patch is fed to the multi-scale instance selection
function. The instance selection function is simply an image cropping function
with a predefined scale. We use multi-scale to deal with varying cytoplasm
scales. In figure 2, a sample of cropped nucleus instances with varying scale
sizes ($0.5$ to $3.0$) is demonstrated. To learn the nucleus segmentation map
we train a U-net model using a nucleus annotation mask. It is worthwhile to
mention that we include the predicted nucleus instance alongside the cropped
image as an input for the cytoplasm segmentation network. The goal of this
extra input is to guide the network for the object of interest.

###  3.2 Cytoplasm Segmentation

In a regular auto-encoder decoder structure the encoder network consists of
several convolutions blocks followed by pooling operations to encode the
object of interest in high-level representation space. In this structure, due
to the consecutive pooling operation, the spatial dimension of the network may
considerably decrease which can result in less discriminated representation
power for objects with varying scale. To mitigate this problem, the Deeplabv3+
model utilized an atrous convolution structure. The atrous convolution applies
a set of upsampled convolutional kernels to describe the object of interest in
higher receptive filed size. To further improve the representation power of
the Deeplab model, Azad et al. [4] proposed a two-level add-on attention
mechanism to extract more informative features from the atrous convolutions.
Where the first level attention mechanism scales the representation space to
highlight the more informative channels then the second attention mechanism
utilizes a 3D convolution kernel between each atrous scale to learn a robust
non-learn feature set. In this section, we use the Attention Deeplabv3+ model
to tackle the cytoplasm segmentation problem. Cytoplasm boundary has a high
overlap with the background area and it requires careful attention to
discriminate the cytoplasm boundary from the background area. In our
implementation, we fed the extracted nucleus area alongside the image batch to
the model to learn the instance segmentation mask. We also apply the image
histogram equalization method to normalize samples. Sample of the estimated
masks for the given nucleus instance is depicted in figure 3.

Input GT Mask Estimation Input GT Mask Estimation

Scale 2.6 Scale 1.0

Figure 3: Segmentation results of the proposed method for both nucleus and
cytoplasm area. The cropped image alongside the predicted nucleus mask is fed
to the cytoplasm network to generate the instance cytoplasm segmentation.

###  3.3 Aggregation function

Learning objects of interest in multi-scale fashion can produce a robust
segmentation mask. In this work, we apply the multi-scale technique on the
input level. Consequently, the model generates a multi-scale segmentation
mask. The main objective of this multi-scale technique is to tackle the
problem of cytoplasm boundary. More specifically, the cytoplasm boundary has a
non-rigid shape. If the boundary area is less then the model requires precise
attention around the nucleus to separate it from the background. On the other
hand, if the boundary is wast then the model needs to consider the big area to
separate its boundary from other instances. We solve this limitation by
defining several scales. Since the ultimate objective of the model is to
produce a single segmentation mask for each instance, we propose to use an
aggregation function to combine and select a single segmentation mask. The
aggregation function can use the output of all scales to generate a single
prediction (like non-maxima suppression), however, our experiment selection
approach produced better performance. To perform this operation, we simply
start from the lowest scale and calculate the relation between the detected
cytoplasm area and the nucleus area. If the ratio is higher than a threshold
value then the next scale is evaluated. The process goes through the next
scales until finding the appropriate condition.

###  3.4 Training Procedure

Our training procedure consists of two stages. In the first stage, we train
the U-net model to segment the nucleus from the input images. The training
process takes into account the training and validation set and learns the
nucleus mask. We train the model for 100 epochs using the Adam optimization
with a learning rate of $1e-4$. In the second phase, we trained each Attention
Deeplabv3+ model using the patches extracted from the input image alongside
the nucleus mask (resulted from the nucleus segmentation network). For each
scale, we train the model for 100 epochs using the Adam optimization with a
learning rate of $1e-5$. All training is done using cross-entropy loss on a
single GTX 1080 GPU.

###  3.5 Inference Procedure

The inference stage uses the trained models to generate the segmentation mask
for both nucleus and cytoplasm instances. In our inference, we use a fixed
number of scales (4 scales) for the test phase.

##  4 Results

The proposed method is evaluated on multiple myeloma cell segmentation grand
challenges which are provided by the SegPC 2021. The challenge data set
consists of a training set with 290 samples, validation and test sets with 200
and 277 samples respectively. All the samples are annotated by the pathologist
and instance base segmentation masks are provided for the object of interest
(myeloma plasma cells). We trained our model using the training and validation
set. During the competition time, we generated the segmentation mask for each
instance. The challenge leader-board compared each team using the MIOU metric,
where our method ranked second among all teams. Table 1 shows the comparison
results for the top five winning teams.

Table 1: Performance comparison on the final test phase for SegPC2021 challenge. Teams | Score (mIoU)
---|---
XLAB Insights | 0.9389
bmdeep | 0.9385
DSC-IITISM | 0.9382
507 | 0.9366
AIVIS | 0.9276

As shown in table 1, the proposed method outperformed most of the competitors
and achieved the second-best place with a small gap ($0.0004$) from the first
team. Figure 4 demonstrates some prediction results where the proposed method
estimated both nucleus and cytoplasm masks with high performance.

Input Ground-Truth Estimation

Figure 4: A sample of prediction results on SegPC2021 grand challenge.

As explained before the proposed method uses a multi-scale strategy to
generate a precise segmentation mask for the cytoplasm instances. In this
section, we will elaborate on the scale selection strategy and its effect on
the final performance. To this end, we have extracted the statistical scale
information from the training set. The histogram information is depicted in
figure 5.

Figure 5: Histogram of the area of cytoplasm to nucleus on the validation set.

According to figure 5, it is crystal clear that the ratio of cytoplasm area to
nucleus area is distributed in almost four different peaks. Thus, we select
four scales to generate a precise segmentation mask. It is worthwhile to
mention that we select the scale value a bit higher than the histogram peaks
(shown with red line on figure 5) to generate an image patch to cover the
appropriate receptive field size. In our experiment for the final test phase,
we selected 4 different scales as depicted in figure 2.

##  5 Conclusion

In this paper, we proposed a multi-scale regional Attention Deeplabv3+ model
for myeloma plasma cell segmentation. The proposed method utilized a U-net
model for nucleus instance segmentation. The segmented nucleus instance is
extracted from the input image and alongside the predicted nucleus mask fed
into a multi-scale cytoplasm detector network. The cytoplasm detector took
into account the strength of the Attention Deeplabv3+ model to segment each
cytoplasm instance. We further proposed an aggregation function to select the
more related scale to fulfill the prediction score. Evaluation results on the
final challenge phase demonstrated outstanding results.

Acknowledgements. This work was fully supported by the BmDeep group. All the
implementation code is available: https://github.com/bmdeep/SegPC2021

## References

  * [1] Alexanian, R., Dimopoulos, M.: The treatment of multiple myeloma. New England Journal of Medicine 330(7), 484–489 (1994)
  * [2] Asadi-Aghbolaghi, M., Azad, R., Fathy, M., Escalera, S.: Multi-level context gating of embedded collective knowledge for medical image segmentation. arXiv preprint arXiv:2003.05056 (2020)
  * [3] Azad, R., Asadi-Aghbolaghi, M., Fathy, M., Escalera, S.: Bi-directional convlstm u-net with densley connected convolutions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 0–0 (2019)
  * [4] Azad, R., Asadi-Aghbolaghi, M., Fathy, M., Escalera, S.: Attention deeplabv3+: Multi-level context attention mechanism for skin lesion segmentation. In: European Conference on Computer Vision. pp. 251–266. Springer (2020)
  * [5] Azad, R., Shayegh, H.R.: Novel and tuneable method for skin detection based on hybrid color space and color statistical features. arXiv preprint arXiv:1407.6506 (2014)
  * [6] Bird, J.M., Owen, R.G., D’Sa, S., Snowden, J.A., Pratt, G., Ashcroft, J., Yong, K., Cook, G., Feyler, S., Davies, F., et al.: Guidelines for the diagnosis and management of multiple myeloma 2011. British journal of haematology 154(1), 32–75 (2011)
  * [7] Feyjie, A.R., Azad, R., Pedersoli, M., Kauffman, C., Ayed, I.B., Dolz, J.: Semi-supervised few-shot learning for medical image segmentation. arXiv preprint arXiv:2003.08462 (2020)
  * [8] Gehlot, S., Gupta, A., Gupta, R.: Ednfc-net: Convolutional neural network with nested feature concatenation for nuclei-instance segmentation. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1389–1393. IEEE (2020)
  * [9] Gupta, A., Duggal, R., Gehlot, S., Gupta, R., Mangal, A., Kumar, L., Thakkar, N., Satpathy, D.: Gcti-sn: Geometry-inspired chemical and tissue invariant stain normalization of microscopic medical images. Medical Image Analysis 65, 101788 (2020)
  * [10] Gupta, A., Gupta, R., Gehlot, S., Goswami, S.: Segpc-2021: Segmentation of multiple myeloma plasma cells in microscopic images. IEEE Dataport 1(1), 1 (2021)
  * [11] Gupta, A., Mallick, P., Sharma, O., Gupta, R., Duggal, R.: Pcseg: Color model driven probabilistic multiphase level set based tool for plasma cell segmentation in multiple myeloma. PloS one 13(12), e0207908 (2018)
  * [12] Guyton, A.C., Hall, J.E.: of medical (2006)
  * [13] Hideshima, T., Mitsiades, C., Tonon, G., Richardson, P.G., Anderson, K.C.: Understanding multiple myeloma pathogenesis in the bone marrow to identify new therapeutic targets. Nature Reviews Cancer 7(8), 585–598 (2007)
  * [14] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25, 1097–1105 (2012)
  * [15] Kyle, R.A., Remstein, E.D., Therneau, T.M., Dispenzieri, A., Kurtin, P.J., Hodnefield, J.M., Larson, D.R., Plevak, M.F., Jelinek, D.F., Fonseca, R., et al.: Clinical course and prognosis of smoldering (asymptomatic) multiple myeloma. New England Journal of Medicine 356(25), 2582–2590 (2007)
  * [16] Minges Wols, H.A.: Plasma cells. e LS (2001)
  * [17] Nau, K.C., Lewis, W.D.: Multiple myeloma: diagnosis and treatment. American family physician 78(7), 853–859 (2008)
  * [18] Palumbo, A., Sezer, O., Kyle, R., Miguel, J., Orlowski, R., Moreau, P., Niesvizky, R., Morgan, G., Comenzo, R., Sonneveld, P., et al.: International myeloma working group guidelines for the management of multiple myeloma patients ineligible for standard high-dose chemotherapy with autologous stem cell transplantation. Leukemia 23(10), 1716–1730 (2009)
  * [19] Rajkumar, S.V., Dimopoulos, M.A., Palumbo, A., Blade, J., Merlini, G., Mateos, M.V., Kumar, S., Hillengass, J., Kastritis, E., Richardson, P., et al.: International myeloma working group updated criteria for the diagnosis of multiple myeloma. The lancet oncology 15(12), e538–e548 (2014)
  * [20] Saeedizadeh, Z., Mehri Dehnavi, A., Talebi, A., Rabbani, H., Sarrafzadeh, O., Vard, A.: Automatic recognition of myeloma cells in microscopic images using bottleneck algorithm, modified watershed and svm classifier. Journal of microscopy 261(1), 46–56 (2016)
  * [21] Tehsin, S., Zameer, S., Saif, S.: Myeloma cell detection in bone marrow aspiration using microscopic images. In: 2019 11th International Conference on Knowledge and Smart Technology (KST). pp. 57–61. IEEE (2019)
  * [22] Vuola, A.O., Akram, S.U., Kannala, J.: Mask-rcnn and u-net ensembled for nuclei segmentation. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). pp. 208–212. IEEE (2019)
  * [23] Vyshnav, M., Sowmya, V., Gopalakrishnan, E., VV, S.V., Menon, V.K., Soman, K.: Deep learning based approach for multiple myeloma detection. In: 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT). pp. 1–7. IEEE (2020)


# D-Flow: A Real Time Spatial Temporal Model for Target Area Segmentation

Wentao Lu, Claude Sammut

###### Abstract

Semantic segmentation has attracted a large amount of attention in recent
years. In robotics, segmentation can be used to identify a region of interest,
or _target area_. For example, in the RoboCup Standard Platform League (SPL),
segmentation separates the soccer field from the background and from players
on the field. For satellite or vehicle applications, it is often necessary to
find certain regions such as roads, bodies of water or kinds of terrain. In
this paper, we propose a novel approach to real-time target area segmentation
based on a newly designed spatial temporal network. The method operates under
domain constraints defined by both the robot’s hardware and its operating
environment . The proposed network is able to run in real-time, working within
the constraints of limited run time and computing power. This work is compared
against other real time segmentation methods on a dataset generated by a Nao
V6 humanoid robot simulating the RoboCup SPL competition. In this case, the
target area is defined as the artificial grass field. The method is also
tested on a maritime dataset collected by a moving vessel, where the aim is to
separate the ocean region from the rest of the image. This dataset
demonstrates that the proposed model can generalise to a variety of vision
problems.

Keywords— Robot Vision, Spatial-Temporal Learning, Semantic Segmentation

## Introduction

The aim of the work presented here is to efficiently segment regions of
interest in a robot’s vision system. Image segmentation must work within a set
of tight constraints. The computational resources available on-board a robot
are often very limited. Furthermore, conserving power is critical in mobile
robot, which often leads to trade-offs in computing power, making running time
a major concern.Dynamic lighting conditions significantly affect the robot’s
vision, hence many colour-based vision segmentation methods fail.

There have been many recent proposals for real time image segmentation.
However, most of them focus on static semantic segmentation and cannot
function effectively under the above constraints. Because of the focus on
static images, they do not take advantage of temporal information in a video
stream.

Assuming that robot vision can be treated as processing a sequence of images,
the model proposed in this paper tries to learn from both spatial and temporal
information based on a number of consecutive frames to predict the
segmentation of region of interest (ROI) of the final frame. Figure 1 presents
a structural overview of the proposed model, called _D-Flow_ because it has
two spatial temporal sub-nets to process the data flow in two colour spaces.
The basic recurrent unit is a newly designed cell where we introduce
convolution operations into a simplified gated recurrent unit known as the
Minimal Gated Unit (Zhou et al. 2016). This new convolutional recurrent cell
can perform spatial temporal feature extraction while significantly reducing
the number of parameters involved.

There are two main reasons for splitting the data flow into two separate sub-
nets. First, the model should be robust to changing lighting conditions.
Second, although the transform from RGB to YUV is linear and it can be done
with a single flow, a single flow architecture can result in a larger number
of parameters. Splitting the data into two flows acts as a heuristic to guide
each flow to emphasise a particular sub task, which we describe below.

In our first experiment, the SPL robot vision dataset is obtained from a Nao
V6 humanoid robot111https://www.softbankrobotics.com/emea/en/nao and the ROI
or _target area_ is defined as the soccer field as per the rules of the
RoboCup SPL competition. Our model and two other real time semantic
segmentation methods, Fast-SCNN (Poudel, Liwicki, and Cipolla 2019) and a
Fully Convolutional Segmentation Network (Long, Shelhamer, and Darrell 2015),
are trained on this dataset for comparison. The results are analysed, both
quantitatively and qualitatively.

The rest of this paper is organised as follows: first, we address related
work, discussing other spatial temporal models and semantic segmentation
methods. The design of the new model is described in the following section. We
then compare both the quantitative and qualitative results from the SPL
dataset with the other two baseline models along with another handcrafted
methods. Also, we apply the proposed model to a Maritime dataset to
demonstrate the generality of the method. We also perform a set of experiments
with different colour spaces to explain why YUV and RGB are chosen in our
model.

$RGB_{t-k}$$RGB_{t-k+1}$$RGB_{t}$…ConvMGUConvMGUConvMGU…ConvMGUConvMGUConvMGU…$YUV_{t-k}$$YUV_{t-k+1}$$YUV_{t}$…ConvMGUConvMGUConvMGU…ConvMGUConvMGUConvMGU…AddConv2dOutput
Figure 1: D-Flow Network Overview

## Related work

The two main components of this work are the Spatial Temporal Learning model
and the Vision Segmentation method. In this section, we discuss previous work
related to these two components.

### Spatial Temporal Learning Model

Spatial Temporal Learning models are capable of learning both spatial and
temporal information based on a dataset. One method for combining Spatial and
Temporal Leaning models is the introduction of both a convolution network
(LeCun et al. 1999) and a recurrent network into a single model. Previous work
has presented different alternatives to doing this. Donahue _et al._ (Donahue
et al. 2015) proposed a combined model where the image data are fed into the
CNN layers first to extract visual features that are then processed by a
sequence of LSTM (Gers, Schmidhuber, and Cummins 2000)(Hochreiter and
Schmidhuber 1997) units to encode temporal features. This is shown in Figure
2.

Figure 2: Long-term Recurrent Convolution Networks

Another similar approach was proposed by Tran _et al._ (Tran et al. 2015). The
authors describe a 3D convolutional neural network, shown in Figure 3, where
the third dimension of each convolution kernel is a convolution operation over
time. This model does not contain any recurrent units that require previous
states as an extra input to the process. Therefore, one of the major
advantages of this model is that its run time is faster than those models
based on recurrent networks.

Figure 3: 3D-Convolutional Neural Networks

Figure 4 shows a novel convolutional recurrent unit along with an encoder-
decoder structure proposed by Shi el al. (SHI et al. 2015), much like the
cascade architecture introduced by Donahue _et al_. The main innovation of
this model is that the authors have replaced the internal matrix
multiplication operation with a convolution operation that allows spatial
features to be extracted by the new recurrent unit. While the model is
successful in solving certain problems, there is a drawback when applying it
to real-time robotics problems, which is the relatively complex cell structure
and large number of parameters, since the base model is a LSTM.

Figure 4: ConvLSTM

### Image Segmentation

There are many recent models capable of performing image segmentation,
including FCN (Long, Shelhamer, and Darrell 2015), SegNet (Badrinarayanan,
Kendall, and Cipolla 2015) and DeepLabv3+ (Chen et al. 2018). They use an
object recognition network, usually a deep CNN structure, as the visual
feature encoder followed by a decoder network. These models demonstrate state
of the art performance on some image datasets, like COCO(Lin et al. 2015) or
PASCAL(Everingham et al. 2010). However, in our case, the only semantic class
in our data is a region of interest, which may have an arbitrary shape,
depending on the robot’s view pose. In other words, the target is not a group
of objects but a group of pixels that have similar colours. Furthermore, these
object recognition networks usually have a large number of parameters and very
complex structures, making them computationally expensive to run in real time
on limited hardware.

## Network Structure

In this section, we present a new convolutional recurrent unit, called
ConvMGU, and explain the design of both the unit and the D-Flow Network that
utilises it.

### Convolutional Minimal Gated Unit

Inspired by the MGU (Zhou et al. 2016) where the authors merge two gates in
the original LSTM design to reduce the number of parameters, we borrow this
idea to solve a similar problem when designing the new convolution recurrent
unit. Figure 5 shows the infrastructure of this unit.

$f_{t}=\sigma_{g}(W_{f}\ast x_{t}+U_{f}\ast h_{t-1}+b_{f})\\\ $ (1)

$h_{t}=(1-f_{t})\circ h_{t-1}+f_{t}\circ\sigma_{h}(W_{h}\ast
x_{t}+U_{h}\ast(f_{t}\circ h_{t-1})+b_{h})$ (2)

$W_{f}$$U_{f}$$\sigma_{g}$1-$\cdot$$\cdot$$W_{h}$$U_{h}$$\sigma_{h}$+$\cdot$$h_{t-j-1}$Hidden$x_{t-j}$Input
Frame j$x_{t-j-1}$ Figure 5: ConvMGU Unit

Equations 1 and 2 are modified from the original MGU paper where the internal
matrix multiplications are replaced with convolution operations as denoted by
$\ast$. $f_{t}$ is the output of the current step. It is used, along with the
hidden state $h_{t-1}$, coming from the last step, to calculate the hidden
state $h_{t}$ of the current step. These modifications introduce the ability
to extract spatial features while maintaining the major advantage of the
original MGU which is the relatively small number of parameters.

One further modification is shown in Figure 6. It stacks two layers of ConvMGU
and adds a shortcut later to form a ConvMGU block whose structure is
reminiscent of a standard Residual Block(He et al. 2016).

1x1CNN$x_{t-j}$$x_{t-j+1}$$x_{t}$ConvMGUConvMGU+ConvMGUConvMGU+ConvMGUConvMGU+…………
Figure 6: ConvMGU Block

The method proposed by Elsayed _et al_ (Elsayed, Maida, and Bayoumi 2018) is
used to estimate the number of parameters of the two layer ConvMGU stack,
ConvMGU Block and ConvLSTM stack as detailed in Equations 3-5.

$\displaystyle 2\ Layer\ ConvLSTM:2\times 4\times(m^{2}(\gamma+\kappa)+1)\cdot
n$ (3)

$\displaystyle Our\ Block:2\times(m^{2}(\gamma+\kappa)+1)\cdot
n+(f^{3}\cdot\gamma+1)\cdot n$ (4)

$\displaystyle Our\ 2\ Layer\ ConvMGU:2\times(m^{2}(\gamma+\kappa)+1)\cdot n$
(5)

For the purpose of analysis, the assigned values for each hyper parameter is
shown in Table 1.

| Meaning | Assigned Value
---|---|---
m | conv kernel size | 3
$\gamma$ | input channel | 3
$\kappa$ | number of feature maps | 40
n | output channel | 40
f | 3d conv kernel size | 3
Table 1: Unit Setting

The results demonstrate that the number of parameters of the proposed ConvMGU
block is 73% smaller in comparison with the original two layer ConvLSTM unit.

### D-Flow Network

The D-Flow Network is composed of two flows of ConvMGU units that encode
spatial temporal features derived from input vision frames. The following
convolution layer acts as a simple decoder to perform a one-step prediction of
the target area segmentation.

As mentioned previously, handling dynamic lighting conditions is a major
problems for mobile robots. The appearance of a colour can vary dramatically
when the robot operates in natural lighting conditions. This significantly
affects target area segmentation, where the target is defined as a group of
pixels that have similar colours. To overcome this problem efficiently, the
proposed two-flow network takes both YUV and RGB images as input to
efficiently search for the best weights. Figure 1 illustrates the proposed
network.

The whole network performs a single-step prediction that takes a sequence of
input frames and attempts to predict the segmentation of the current frame.
The YUV and RGB images are processed separately by stacks of ConvMGU to
extract various spatial temporal features. These features are then added
together to produce the final feature maps that the subsequent decoder can use
for prediction.

## Experiment

In this section we describe a set experiments to analyse the performance and
generalisation ability of our model.

### SPL Vision Dataset

For the first experiment, we used a dataset collected from a Nao V5/V6
humanoid robot from Softbank Robotics(Lu 2021). The dataset contains 20 video
captures where each video contains varying numbers of image sequences with
corresponding weak labels. The details of the labelled data are shown in the
Table 2. The raw resolution is 1280 by 960. The data were collected from
various locations and with different lighting conditions.

#frame | location | lighting condition | body motion | recording type
---|---|---|---|---
19 | Indoor(Germany) | Constant(low) | Static | Multi-testing
19 | Indoor(Germany) | Constant(low) | Static | Multi-testing
19 | Indoor(Germany) | Constant(low) | Static | Multi-testing
5 | Indoor(Germany) | Constant(high) | Static | Single-testing
6 | Indoor(Germany) | Constant(high) | Static | Single-testing
19 | Indoor(Germany) | Constant(low) | Dynamic | Single-testing
7 | Indoor(Germany) | Constant(low) | Dynamic | Game
259 | Indoor(UNSW) | Constant(medium) | Dynamic | Single-testing
199 | Outdoor(Sydney) | Night | Static | Single-testing
121 | Indoor(UNSW) | Uneven(medium) | Dynamic | Single-testing
271 | Indoor(UNSW) | Uneven(medium) | Dynamic | Multi-testing
113 | Indoor(UNSW) | Uneven(medium) | Dynamic | Multi-testing
68 | Outdoor(Sydney) | Night | Static | Single-testing
119 | Outdoor(Sydney) | Night | Static | Single-testing
298 | Indoor(Sydney) | Constant(low) | Dynamic | Multi-testing
251 | Outdoor(Sydney) | Morning(uneven) | Static | Single-testing
210 | Indoor(UNSW) | Extremely uneven | Dynamic | Single-testing
181 | Indoor(UNSW) | Extremely uneven | Dynamic | Single-testing
213 | Indoor(UNSW) | Extremely uneven | Dynamic | Multi-testing
188 | Indoor(UNSW) | Extremely uneven | Dynamic | Multi-testing

Table 2: Dataset Information

Figure 7 presents a pair of a raw images and a label that is used to identify
noisy target areas using white shapes. A labelling tool is used to manually
mark and generate the white shape that is considered to be the labelled target
area. The goal for this dataset is to train a model that can identify these
target areas with minimal false positives under dynamic lighting conditions.
We hypothesise that such a model may even be able to overcome issues
introduced by the weak labelling. Since it is a weak labelling system, the
lines and all other objects present within the target area are all given the
same label.

(a) Raw Image

(b) Label

Figure 7: A sample pair of SPL dataset

### Experiment Results

In this experiment we performed target area segmentation with Fast-SCNN, a
Fully Convolution Network and our model. Input images were down-sampled four
times to 320 by 240 for efficiency. All training was performed on a single
NVIDIA RTX 1060 and the inference time analyses were done on the same GPU. The
experiment was implemented using the Pytorch 1.7.1 (Paszke et al. 2017).

Figures 8 and 9 present the qualitative results for two test sequences
generated by the candidate methods with different loss functions. The third
set of results was generated by our model with Focal Loss (Lin et al. 2017)
while the others were trained with Binary Cross Entropy Loss.

As the images show, the FCN only learned positional patterns and failed to
generalise properly in different scenarios. Fast-SCNN learnt some out-of-
domain patterns, as it showed the ability to generalise through its
identification of the green area from the bottom part of the images.
Unfortunately, it tended to overfit based on the colour pattern since it
labelled all the green parts as the target area even though some of them were
simple light reflections. The left-most three images demonstrate the ability
of our model to outperform the other models in this task. All of them label
most parts of the target area with minimal outliers that can be easily removed
through post-processing. However, only our models were able to crop out random
objects in the target area. This demonstrates that our model can work
effectively with a weak labelling system which has noisy labels.

For a quantitative analysis, we used two different metrics to measure
segmentation performance. The Dice Coefficient (Dice 1945) is a statistic that
measures the similarity between two samples that we use in this experiment to
measure the model’s ability to learn label patterns. This score only measures
how well the model fits the labels, where the labels may be noisy. Therefore,
it does not measure the true ability for target area segmentation but acts as
a base check for learning outcomes. The silhouette score (Rousseeuw 1987) is
another metric that is a model-free measurement of the consistency in
segmentation. The inference time analysis shown was performed on a single
NVIDIA RTX 1060 GPU. The results are shown in Table 3.

| Dice Score | Silhouette score | Inference Time | #Params
---|---|---|---|---
Ours_small | 0.82 | 26% | 19 ms | 45K
Ours | 0.83 | 27% | 41 ms | 0.178M
FCN(VGG) | 0.85 | 24% | 31 ms | 32M
Fast-SCNN | 0.91 | 26% | 12ms | 1.1M

Table 3: Quantitative Results

(a) D-Flow_Small

(b) D-Flow

(c) D-Flow_Focal

(d) Fast_SCNN

(e) FCN(VGG)

(f) Raw

Figure 8: Comparison Experiment Result 1

(a) D-Flow_Small

(b) D-Flow

(c) D-Flow_Focal

(d) Fast_SCNN

(e) FCN(VGG)

(f) Raw

Figure 9: Comparison Experiment Result 2

(a) D-Flow_BCE-Loss

(b) Mean Adaptive-threshold

(c) Gaussian Adaptive-threshold

(d) Distance-transform threshold

(e) Raw Robot Camera Image

Figure 10: Experiment comparing against Handcrafted Methods Result 1

(a) D-Flow_BCE-Loss

(b) Mean Adaptive-threshold

(c) Gaussian Adaptive-threshold

(d) Distance-transform threshold

(e) Raw Robot Camera Image

Figure 11: Experiment comparing against Handcrafted Methods Result 2

### Comparison with Handcrafted Methods

In this section, we compare the proposed model with two handcrafted methods
used by the RoboCup team, rUNSWift, where both of them perform field
recognition (Brameld et al. 2018).

Adaptive-thresholding is an algorithm that groups pixels into binary classes
in a sliding window. As shown in Figures 10 and 11, two variants are actually
doing brightness-based clustering which is suitable for finding certain
objects. Unfortunately, these methods are not able to tell the difference
between the main part of the soccer field and the image background, where in
most cases, they are both shown as a darker colour.

A distance-transform is one step in the Watershed background
extraction(Roerdink and Meijster 2000). Since its purpose is to distinguish
between the foreground and background, it is not suitable for use in target
area segmentation. As shown, it marks white clusters inside the soccer field
as well as random bright pixels as foreground. Therefore, its output is a
mixture of a part of the target area and a part of the darker background.

### Experiments on Maritime Dataset

We tested our work on a maritime dataset to see how well the proposed method
performs in different domains. The Singapore Maritime Dataset(SMD) (Prasad et
al. 2017) was used in this experiment. It contains two different subsets. In
our experiment, we only used the one where the camera is mounted on a moving
vessel, which is similar to a mobile robot.

Dataset | Sub-set | #Video | #Frames/Video | #Total
---|---|---|---|---
SMD | On-Board | 11 | 300 | 3300

Table 4: SMD dataset

Table 4 gives the details of the subset of the data we used in this test. We
used 9 of the 11 videos as training data and the remaining 2 as test data. In
this task, we defined the target area as the ocean below horizon. Figure 12
shows a pair of sample raw frames and the corresponding label. Again, we used
a weak labelling system as we only put a rectangular mask as the target.

(a) Raw Image

(b) Label

Figure 12: a sample pair of SMD dataset

(a) Raw Image

(b) Prediction

(c) Annotated Image

Figure 13: Sample output for SMD dataset

Figure 13 shows a sample output for the test data. We can see that despite the
big difference in the ocean colour between the test data and the sample
training data, our model can still find the target region of the ocean below
the horizon with not many false positives.

### Experiment with Different Colour Spaces

Our model is composed of two sub-nets that process RGB and YUV formatted data
separately until the processed feature maps are combined for the final
prediction stage.

We conducted a series of experiments with the SMD dataset to understand the
effects of using different colour spaces and the network architecture. Table 5
in Appendix shows the setting for each experiment.

Figure 14 in the Appendix shows the learning curves for each experiment. We
see the RGB-YUV combination with the Double-Flow architecture achieves the
most stable performance and lowest validation loss at the 100th epoch. The
YUV-only model suffers from underfitting while HSV-only and the HSV-YUV
combination suffer from overfitting. The RGB-YUV(Y-Only) experiment is
intended to determine if the gray scale image has the same effect as a full
YUV image. The RGB-YUV(Y-Only) can achieve low validation loss at some epochs,
but the learning curve is quite unstable and tends to fluctuate dramatically
while achieving low validation loss. In conclusion, among all these different
colour space settings and network architectures, the RGB-YUV combination with
Double-Flow net is the best.

## Conclusion

We have proposed a new approach to real time robot vision target area
segmentation along with a new parameter reduced convolutional recurrent unit.
The proposed D-Flow Network demonstrates the ability to overcome problems
including dynamic lighting conditions. We have also created and published a
robot vision dataset that was obtained using a Nao V5/V6 robot. This dataset
can be used for a variety of robot vision problems.

##  Appendix A Learning Curves for Different Colour Space

Models | Colour for 1st flow | Colour for 2nd flow
---|---|---
Single-Flow | RGB | N/A
Single-Flow | HSV | N/A
Single-Flow | YUV | N/A
Double-Flow | RGB | YUV
Double-Flow | RGB | HSV
Double-Flow | HSV | YUV
Double-Flow | RGB | YUV(Y-Only)
Table 5: Experiment Settings

(a) HSV+RGB

(b) HSV+YUV

(c) RGB+YUV

(d) RGB+YUV(Y-Only)

(e) RGB

(f) HSV

(g) YUV

Figure 14: Learning Curves

## References

  * Badrinarayanan, Kendall, and Cipolla (2015) Badrinarayanan, V.; Kendall, A.; and Cipolla, R. 2015.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation.  _CoRR_ , abs/1511.00561.
  * Brameld et al. (2018) Brameld, K.; Hamersley, F.; Jones, E.; Kaur, T.; Li, L.; Lu, W.; Pagnucco, M.; Sammut, C.; Sheh, Q.; Schmidt, P.; et al. 2018.  RoboCup SPL 2018 rUNSWift Team Paper.
  * Chen et al. (2018) Chen, L.-C.; Zhu, Y.; Papandreou, G.; Schroff, F.; and Adam, H. 2018.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.  In _The European Conference on Computer Vision (ECCV)_.
  * Dice (1945) Dice, L. R. 1945.  Measures of the Amount of Ecologic Association Between Species.  _Ecology_ , 26(3): 297–302.
  * Donahue et al. (2015) Donahue, J.; Anne Hendricks, L.; Guadarrama, S.; Rohrbach, M.; Venugopalan, S.; Saenko, K.; and Darrell, T. 2015.  Long-Term Recurrent Convolutional Networks for Visual Recognition and Description.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Elsayed, Maida, and Bayoumi (2018) Elsayed, N.; Maida, A. S.; and Bayoumi, M. 2018.  Reduced-Gate Convolutional LSTM Using Predictive Coding for Spatiotemporal Prediction.  _CoRR_ , abs/1810.07251.
  * Everingham et al. (2010) Everingham, M.; Van Gool, L.; Williams, C. K. I.; Winn, J.; and Zisserman, A. 2010\.  The Pascal Visual Object Classes (VOC) Challenge.  _International Journal of Computer Vision_ , 88(2): 303–338.
  * Gers, Schmidhuber, and Cummins (2000) Gers, F. A.; Schmidhuber, J.; and Cummins, F. 2000.  Learning to Forget: Continual Prediction with LSTM.  _Neural Computation_ , 12(10): 2451–2471.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long Short-Term Memory.  _Neural Computation_ , 9(8): 1735–1780.
  * LeCun et al. (1999) LeCun, Y.; Haffner, P.; Bottou, L.; and Bengio, Y. 1999.  _Object Recognition with Gradient-Based Learning_ , 319–345.  Berlin, Heidelberg: Springer Berlin Heidelberg.  ISBN 978-3-540-46805-9.
  * Lin et al. (2017) Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; and Dollár, P. 2017.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , 2980–2988.
  * Lin et al. (2015) Lin, T.-Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C. L.; and Dollár, P. 2015.  Microsoft COCO: Common Objects in Context.  arXiv:1405.0312.
  * Long, Shelhamer, and Darrell (2015) Long, J.; Shelhamer, E.; and Darrell, T. 2015.  Fully Convolutional Networks for Semantic Segmentation.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Lu (2021) Lu, W. 2021.  The rUNSWift SPL Field Segmentation Dataset.  arXiv:2108.12809.
  * Paszke et al. (2017) Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017.  Automatic Differentiation in PyTorch.  In _NIPS Autodiff Workshop_.
  * Poudel, Liwicki, and Cipolla (2019) Poudel, R. P. K.; Liwicki, S.; and Cipolla, R. 2019.  Fast-SCNN: Fast Semantic Segmentation Network.  _CoRR_ , abs/1902.04502.
  * Prasad et al. (2017) Prasad, D. K.; Rajan, D.; Rachmawati, L.; Rajabally, E.; and Quek, C. 2017\.  Video Processing From Electro-Optical Sensors for Object Detection and Tracking in a Maritime Environment: A Survey.  _IEEE Transactions on Intelligent Transportation Systems_ , 18(8): 1993–2016.
  * Roerdink and Meijster (2000) Roerdink, J. B.; and Meijster, A. 2000.  The watershed transform: Definitions, algorithms and parallelization strategies.  _Fundamenta informaticae_ , 41(1, 2): 187–228.
  * Rousseeuw (1987) Rousseeuw, P. J. 1987.  Silhouettes: A graphical aid to the interpretation and validation of cluster analysis.  _Journal of Computational and Applied Mathematics_ , 20: 53 – 65\.
  * SHI et al. (2015) SHI, X.; Chen, Z.; Wang, H.; Yeung, D.-Y.; Wong, W.-k.; and WOO, W.-c. 2015.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting.  In Cortes, C.; Lawrence, N. D.; Lee, D. D.; Sugiyama, M.; and Garnett, R., eds., _Advances in Neural Information Processing Systems 28_ , 802–810. Curran Associates, Inc.
  * Tran et al. (2015) Tran, D.; Bourdev, L.; Fergus, R.; Torresani, L.; and Paluri, M. 2015.  Learning Spatiotemporal Features With 3D Convolutional Networks.  In _The IEEE International Conference on Computer Vision (ICCV)_.
  * Zhou et al. (2016) Zhou, G.-B.; Wu, J.; Zhang, C.-L.; and Zhou, Z.-H. 2016.  Minimal gated unit for recurrent neural networks.  _International Journal of Automation and Computing_ , 13(3): 226–234.


# Pre- and post-contact policy decomposition for non-prehensile manipulation
with zero-shot sim-to-real transfer

Minchan Kim1, Junhyek Han1, Jaehyung Kim1, and Beomjoon Kim1 1 Kim Jaechul
Graduate School of AI at KAIST, {minchan21, junhyek.han, kimjaehyung,
<EMAIL_ADDRESS>
###### Abstract

We present a system for non-prehensile manipulation that require a significant
number of contact mode transitions and the use of environmental contacts to
successfully manipulate an object to a target location. Our method is based on
deep reinforcement learning which, unlike state-of-the-art planning
algorithms, does not require apriori knowledge of the physical parameters of
the object or environment such as friction coefficients or centers of mass.
The planning time is reduced to the simple feed-forward prediction time on a
neural network. We propose a computational structure, action space design, and
curriculum learning scheme that facilitates efficient exploration and sim-to-
real transfer. In challenging real-world non-prehensile manipulation tasks, we
show that our method can generalize over different objects, and succeed even
for novel objects not seen during training. Project website:
https://sites.google.com/view/nonprenehsile-decomposition

##  I Introduction

Humans possess remarkable skills in non-prehensile manipulation techniques
such as pulling, pushing, dragging, and tumbling, which enables them to handle
objects that are hard or impossible to grasp. In contrast, most robots are
only capable of performing pick-and-place and fail in situations where
grasping is not an option due to physical or geometric limitations. Our
objective is to equip robots with the ability to manipulate objects even in
such circumstances, using a basic gripper and an RGB camera as shown in Figure
LABEL:fig:overall_illustration. This is a challenging problem that involves
planning a sequence of combined robot and object movements while accounting
for the contact interactions among an object, robot, and environment. Since
the robot is equipped with a simple gripper, it must leverage the gravity and
environment’s geometry to perform the task effectively.

The state-of-the-art algorithms for such problems are planning algorithms that
use tree search or trajectory optimization based on an analytical physics
model [1, 2, 3, 4, 5, 6]. However, their applicability in the real world is
limited by several factors. Firstly, their search space is extremely complex,
involving both discrete contact activations and continuous robot and object
motions, with different sub-manifolds defined by different motion constraints.
This makes planners too slow to use in practice. Secondly, they require
accurate analytical contact models of the world, which is non-trivial to
define as rigid-body dynamics are often insufficient [7]. Moreover, even if we
_can_ define the governing equation of contacts, extracting the physical
parameters for instantiating the equation, such as the center of mass or
friction coefficient, from high-dimensional RGB images is a challenging
problem.

Because of these limitations, they have only been applied in a controlled
setup where all necessary information about the object, such as inertial
parameters or pose, is known. To achieve reasonable planning time, they often
rely on strong assumptions about contact interactions, robot motions, and
transition dynamics, such as quasi-static dynamics, predefined contact modes
(slipping, static, rolling, etc.), and predefined contact locations in the
environment, object, and robot [3, 8, 1, 5].

We propose an alternative solution for non-prehensile manipulation that
utilizes deep reinforcement learning (RL). By directly learning the
representation that implicitly encodes the necessary information from sensory
data, our approach eliminates the need to extract difficult-to-observe object
inertial parameters. At the expense of offline training, the online
computation is reduced to feed-forward predictions from a neural network
rather than searching in a complex space. Notably, our approach does not make
assumptions about contact interactions and robot or object motions, making it
more versatile than traditional planning algorithms.

Like the recent approaches for contact-rich tasks such as in-hand manipulation
[9, 10, 11] and locomotion [12, 13], we use a simulator to train a policy and
then transfer it to the real world. In this scheme, there are two primary
challenges that we need to address for non-prehensile manipulation:
exploration and the sim-to-real gap. Exploration in non-prehensile
manipulation is challenging because unlike locomotion or in-hand manipulation
where the object or the ground is in close proximity to make contact, the
object is not in a position where we can directly make contact, especially in
the initial state. So, if we randomly explore, there is zero chance of
sampling an action that contacts the object [14]. The sim-to-real gap arises
from the inaccuracies in the physics engine and modeling. We found that the
hardware of the widely-used collaborative robots such as Franka Emika Panda is
based on industrial robots and has a large gear ratio with intricate joint
friction that is difficult to model accurately.

(a) EE-ABOVE

(b) EE-AT-RIGHT

Figure 1: (a) EE-ABOVE: The end-effector is placed above the object. (b) EE-
AT-RIGHT: The end-effector is placed on the right side of the object. The
black cube denotes the goal pose. The robot should contact the object on the
right to manipulate it to the goal, and initially contacting it on top of the
object makes exploration extremely challenging as it would have to break the
contract entirely and re-make it on the right.

Our primary contribution is the design of a computational structure, action
space, and learning curriculum that solves both of these challenges. For the
exploration problem, one naive approach would be to define a contact-inducing
reward that rewards the robot if its hand gets closer to the object. However,
for non-prehensile manipulation, not all contacts are equal: some initial
contacts are more crucial as it leads to more promising state space in the
post-contact phase, as shown in Figure 2. Unfortunately, such a reward would
make the robot contact the closest point on the object, and we empirically
found this approach too sensitive to the initial system state.

Instead, we make a key observation that in general, any non-prehensile
manipulation can be divided into two stages: the _pre-contact_ and _post-
contact_ stages. In the pre-contact stage, the robot primarily focuses on
finding a promising initial contact on the object that would lead to
successful manipulation. In the post-contact phase, the robot finds a sequence
of forces to apply to the object that successfully moves it to the target
location. In this latter stage, the robot will mostly be in contact with the
object — however, it might purposefully break a contact to exploit gravity and
environmental contact or simply change its contact point to exert a force from
a different direction.

Based on this observation, we propose a computational structure that
facilitates effective exploration, where we use two different policies for
each stage: pre-contact and post-contact policies, each having different
action spaces. The _pre-contact policy_ ’s action is defined by the point on
the object and the gripper that together defines the contact position, along
with the gripper’s orientation and gripper width at the contact. This
guarantees contact with every action and enables the robot to explore the
space of contacts more efficiently than say, using a joint torque or SE(3)
end-effector pose. We train this policy first in a simulator where we have
accurate object shape information and then transfer it to the real world by
using a student-teacher training scheme [15].

Because the pre-contact policy involves just position control, there is a
negligible sim-to-real gap. The design of the post-contact policy’s action
space, on the other hand, requires more consideration as it involves contact
interaction with the object. In locomotion and in-hand manipulation, the
typical approach is to learn a policy to generate the next target _joint
position_ , and use an analytical joint position controller [12, 9, 11]. On
the other hand, for our tasks with natural motion constraints111In the context
of hybrid position-force control and continual contacts, learning a policy
that generates the target _end-effector pose_ along with control gain
parameters for an operational space controller (OSC) has shown to have the
best sample efficiency among different action spaces [16].

Unfortunately, OSC requires an accurate robot model [17], but there are
several factors hindering accurate modeling; using a system identification
helps, but we found it to be insufficient. For instance, the wrist joint on
Panda has large friction in one direction but not so much in the opposite
direction, and such intricacy cannot be captured with the parameters available
in a typical physics simulator. Furthermore, since OSC uses a Cartesian space
error to compute joint torques unless the end-effector moves exactly the same
way for the given torque in simulation and the real world, there would be a
large joint position gap.

We instead propose a different action space. Just like in [16], we predict the
end-effector target pose. However, instead of predicting the gains for an OSC,
we predict the gains for each joint and use inverse kinematics to solve for
the joint position target. Then, we use a joint position controller to realize
that joint position using the predicted controller gains. We found that using
the joint position controller transfers much more effectively to the real
world than OSC because the controller errors are now in the joint space rather
than Cartesian space which you can correct by moving the respective joint.

Another factor we need to consider when transferring a policy to the real
world is the violation of joint velocity and torque limits. The real robot has
safety measures that prevent the robot to move above a certain speed and
contact force. To account for this, we can terminate the episode if the robot
violates them during learning, as in [10]. However, we found this makes
learning difficult as it explores too conservatively, and fails to learn a
policy in some of our domains. So, we introduce an action scaling curriculum
that allows the robot to explore aggressively at the beginning of the learning
but slowly decrease the maximum magnitude of the action so that the velocity
and torque limits are met. This facilitates both efficient exploration and
sim-to-real transfer.

As in previous works [9, 3, 12, 11], we make significant use of domain
randomization to close the sim-to-real gap. We found that, with an end-to-end
system, it is difficult to determine where the sim-to-real gap lies since the
system is not modularized. So we design a modular system where the perception
module takes a single RGB and outputs the 2D object key points. The policy
takes the 2D key points and the proprioceptive sensor data to produce an
action. We train our key point detector, pre-contact, and post-contact
policies entirely in simulation and transfer them to the real world.

We apply our method to three challenging non-prehensile manipulation problems
where environment or object geometry renders grasping impossible, as shown in
Figure LABEL:fig:overall_illustration. The robot needs to make use of both
intrinsic and extrinsic contacts to manipulate objects that are not directly
graspable. We show that not only can our method handle objects seen in
training, but also new objects with different friction coefficients, centers
of mass, and densities that are outside of the domain randomization range.

##  II Related work

###  II-A Planning through contacts

There are several methods for planning through contacts that can be applied to
our tasks. The first class of algorithms is optimization-based methods [5, 4,
18] which analytically models the robot kinematic and dynamics constraints,
such as joint limits, and contact constraints, such as friction cone
constraints, as constraints in an optimization problem, and uses gradient-
based techniques to find a trajectory. To handle the hybrid search space and
discontinuous dynamics, some methods smoothen the contact mode decisions [5]
or use complementarity constraints [4, 18]. While these algorithms can compute
impressive motions with many contact mode changes, they also tend to output
motions that are unrealistic because it is difficult to satisfy the
constraints and to accurately model contacts. To our knowledge, these methods
have scarcely been demonstrated on a real robot, if not never.

Another class of planning algorithms is based on graph search [19, 8, 1, 3,
20]. To handle discontinuous dynamics and hybrid search space, these methods
typically construct a graph where each node encodes the contact mode and
states of the object and robot, and each edge encodes the motion between two
states if the transition is feasible. To compute the motion on each edge,
either a pre-defined motion primitive [19] or an additional planner is used
that accounts for dynamics constraints imposed by the contact mode and state
[8, 3, 20, 1]. In general, these methods output more physically realistic
motions than optimization-based algorithms and have been demonstrated on real
robots [3, 18, 20]. However, they make strong assumptions such as quasi-static
assumption or a pre-defined set of primitives or contact modes, and are
limited to tasks that can be solved using simple motion with very few or no
contact mode changes.

Because all these algorithms must compute a plan in a hybrid search space with
discontinuous dynamics, they are too slow to use in practice. Furthermore,
except [20], which predicts whether a motion primitive and its parameters will
succeed from a segmented point cloud of the scene, all methods depend on the
assumption that physical parameters such as mass and friction coefficient are
known. In contrast, our approach suffers from neither of these problems. We
can compute the next action by making a feed-forward prediction using a neural
network, which usually takes on the order of milliseconds, and operate
directly using a high dimensional camera and proprioceptive sensor rather than
relying on accurate physical parameter knowledge.

###  II-B Reinforcement learning for contact-rich tasks

RL algorithms have been successfully demonstrated for in-hand manipulation
[21, 11, 22, 9, 10, 23] tasks. One key difference between these and our work
is that in in-hand manipulation, the object typically starts in close
proximity to the robot, whereas in our domain, we must solve the additional
problem of reaching and making contact with the object. With an exception to
[9], which has the object right beneath the hand, all systems begin with the
object in hand. Like in [9], we encourage contact between the object and the
finger using a reward. However, we found this to be insufficient as the
initial contact is more crucial in our domain, and use a pre-contact policy to
facilitate efficient exploration. Following [22], we use object key points to
represent their pose.

There are few works on RL-based non-prehensile manipulation [24, 25, 26, 27].
Unlike planning-based algorithms, these methods can handle high-dimensional
input data such as images, do not explicitly depend on physical parameters,
and have been demonstrated in the real world. However, these methods are
limited to planar pushing, whereas planning algorithms could synthesize
complex motions which incorporate multiple contact mode changes. Our method on
the other hand not only generates complex movements, but also operates on
high-dimensional sensory data without physical parameters.

The work by Zhou and Held [28] is close to ours in that it trains an agent to
grasp an initially ungraspable object, and shows generalization over a variety
of objects. One key difference from our work is that we focus on moving the
object to the goal pose, whereas their objective is to grasp the object.
Moreover, since they reward the agent to approach the target grasp, the motion
typically involves only a few mode changes. Finally, they use an OSC and must
operate the policy at a very low frequency to close the sim-to-real gap.
However, our method controls joint position directly and controls the robot at
a much higher frequency, enabling more dexterous motions.

##  III Method

TABLE I: The components of state space $\mathcal{S}$ of $\pi_{\text{post}}$. Component | Description
---|---
$q[t]\in\mathbb{R}^{9}$ | joint and finger positions at time step $t$
$\dot{q}[t]\in\mathbb{R}^{9}$ | joint and finger velocities at time step $t$
$u_{o}[t]\in\mathbb{R}^{2\times 8}$ | 2D object key points at time step $t$
$u_{g}\in\mathbb{R}^{2\times 8}$ | 2D goal object key points
$T_{E}[t]\in SE(3)$ | end-effector pose at time step $t$
$a_{post}[t-1]$ | post-contact policy’s action at time step $t-1$
Figure 2: Modules in our system. (Top left) In the simulation, we use the
ground-truth object and goal pose as inputs to $\pi_{\text{pre}}$, which
defines the initial robot configuration for the post-contact policy. (Bottom
left) In the real world, we use the output of the key point detector and
$\pi_{\text{pre}}^{S}$, the student pre-contact policy. (Right)
$\pi_{\text{post}}$ in simulation uses the ground-truth 2D key points along
with other inputs, while in the real world, it uses the output of the key
point detector.

We are given a single RGB camera, a manipulator with a simple gripper, and a
proprioceptive sensor. The target location of the object is defined using a
relative transform with respect to its initial pose. We use Franka Emika Panda
in this paper, but our method can be applied to other manipulators. Figure 2
shows how different modules in our system get trained and used in simulation
and the real world.

###  III-A Training $\pi_{\text{pre}}$ and $\pi_{\text{post}}$ in a simulator

The state space of $\pi_{\text{pre}}$ consists of $(T_{o},T_{g})\in
SE(3)\times SE(3)$ where $T_{o}$ and $T_{g}$ denote initial and goal object
poses respectively. Its action space consists of four components: the
orientation of the gripper, $R_{E}\in SO(3)$, its width, $l\in[0,0.04]$, a
point on the gripper, $c_{f}$, and a point on the object, $c_{o}$. The space
of $c_{f}$ is defined as the pre-defined points on the robot gripper, and the
space of $c_{o}$ is defined as points uniformly sampled from the surface of
the object. The desired end-effector position, denoted $p_{c}$, is defined by
matching $c_{f}$ and $c_{o}$. See Figure 3 for an illustration. We compute the
joint position at $(p_{c},R_{E})$ using inverse kinematics (IK) solver.

Figure 3: Illustration of how $\pi_{\text{pre}}$ makes the initial contact.

We define the 3D key points as the vertices of the 3D bounding box for the
target object. The state space of the post-contact policy $\pi_{post}$
consists of variables listed in Table I. Its action space consists of the end-
effector residual denoted with $\Delta T_{E}[t]\in SE(3)$, defined as the
difference between the target and the current end-effector pose, the
controller gain $k_{p}[t]$ and the damping ratio $\rho[t]$ for the joint
controller. The controller gain $k_{d}[t]$ is calculated by
$k_{d}[t]=\rho[t]\cdot\sqrt{k_{p}[t]}$. To execute an action, we use IK to
solve for the joint position target in the next step, and then run the joint
position controller with the controller gains.

We train both $\pi_{\text{pre}}$ and $\pi_{post}$ using Proximal Policy
Optimization (PPO) [29] using the reward function defined as

$\displaystyle r(s[t],a_{post}[t])$
$\displaystyle=\sum_{i=1}^{N_{k}}\frac{C_{1}}{\|x_{o}^{i}[t]-x_{g}^{i}\|_{2}+C_{1}}-\|k_{p}[t]\|_{2}$
$\displaystyle\mathrel{\phantom{=}}+C_{2}\mathbb{1}\left(d[t]<\bar{d}\;\text{and}\;\theta[t]<\bar{\theta}\right)$
$\displaystyle\mathrel{\phantom{=}}+\frac{C_{3}}{\|(p_{\mathrm{lf}}[t]+p_{\mathrm{rf}}[t])/2-p_{\mathrm{obj}}[t]\|_{2}}\
,$ (1)

where $C_{1},C_{2}$, and $C_{3}$ are positive constants,
$x_{o}^{i}[t]\in\mathbb{R}^{3}$ and $x_{g}^{i}\in\mathbb{R}^{3}$ are 3D
current and goal object key points, and $N_{k}$ is the number of key points on
the object. The first term increases as the object get closer to the goal,
where the distance is computed based on the 3D key points. The second term
regularizes the magnitude of $k_{p}$ to induce a compliant motion. The third
term is an extra reward for completing the task, in addition to matching the
key points for completing the job. Here, $d[t]$ is the distance between the
goal and current object position, and $\theta[t]$ is the distance between the
goal and current object orientation, measured using a quaternion difference
$\alpha[t]\alpha_{g}^{-1}$ between the current and goal object orientation,
denoted $\alpha[t]$ and $\alpha_{g}$ respectively. Typically we set $C_{2}\gg
C_{1}$. The last term induces the end-effector to be close to the object.
Here, $p_{\mathrm{obj}}$ is the position of the center of mass of the
collision mesh of the object, and $p_{\mathrm{lf}}$ and $p_{\mathrm{rf}}$ are
at the left and right finger joints respectively.

We train $\pi_{\text{pre}}$ and $\pi_{\text{post}}$ jointly. Given an
environment, we first create a problem by sampling a $(T_{o}[0],T_{g})$ pair.
We then use $\pi_{\text{pre}}$ to obtain its action, $a_{\text{pre}}$. If
$a_{\text{pre}}$ yields an end-effector pose that is in a collision or is
kinematically infeasible, we terminate the episode, receive $C_{4}<0$ as a
reward, and update $\pi_{\text{pre}}$ with this reward. Otherwise, we execute
$a_{\text{pre}}$ by setting the robot at joint configuration $q_{E}$ given by
the IK solver for $a_{\text{pre}}$. Then, we execute $\pi_{\text{post}}$,
updating it at every $H$ time steps until the episode terminates. In this
phase, an episode can terminate because (1) the maximum episode length, $L$,
has been reached, (2) the robot drops the object, or (3) the robot succeeds in
the task. Once the episode terminates, we take the sum of the rewards that we
have obtained during the episode and update $\pi_{\text{pre}}$. The process
repeats for a desired number of iterations.

###  III-B Sim-to-real transfer

####  III-B1 Transferring $\pi_{\text{pre}}$

Since the state and action of $\pi_{\text{pre}}$ depend on information
unavailable in the real world, we cannot use $\pi_{\text{pre}}$ directly. So,
we train a student policy, $\pi_{\text{pre}}^{S}$, using $\pi_{\text{pre}}$ as
the teacher policy [15]. The student policy’s action consists of $q_{E}\in
SE(3)$, the end-effector pose, and $l$. Its state consists of the 2D current
and goal object key points, $u_{o}[t]\in\mathbb{R}^{2\times 8}$, and
$u_{g}\in\mathbb{R}^{2\times 8}$. We generate the imitation learning data for
$\pi_{\text{pre}}^{S}$ by running a trained $\pi_{\text{pre}}$ in the
simulator. We convert actions of $\pi_{\text{pre}}$ to $q_{E}$ and $l$ pair
and use them as labels for $\pi_{\text{pre}}^{S}$. We also convert $T_{o}$ and
$T_{g}$ into $u_{o}[t]$ and $u_{g}$ by projecting the vertices of the bounding
box of the object to the image plane. We train $\pi_{\text{pre}}^{S}$ by
minimizing the mean squared error (MSE). To ensure the MSE loss makes sense on
the orientation, we represent the orientation of $q_{E}$ in the form of 6D
representation used in [30], which is just the first two columns of a rotation
matrix.

####  III-B2 Training and transferring a perception module

We train a key point detector using synthetic images in simulation and then
use domain randomization to transfer it to the real world. Our detector takes
a single RGB image as an input and outputs 2D object key points. We generate
training data for the key point detector by running trained $\pi_{\text{pre}}$
and $\pi_{\text{post}}$ in the simulator. At each time step, we take an RGB
image, a segmentation mask indicating whether each pixel belongs to the robot,
the object, the table, or the background, and ground-truth 2D key points
projected onto an image plane.

We adopt the key point detector network architecture from [31], which
generates $N_{k}$ heatmaps, one for each 2D key point, with the same size as
the input image. These heatmaps represent the probability of each 2D key
point’s existence at each pixel. To train the network, we use ground-truth
Gaussian heatmaps centered at the projected location on the image plane for
each key point, as in [31]. We train the key point detector by minimizing the
KL divergence between the predicted heatmaps and the ground-truth heatmaps. To
make the key point detector robust to changes in the background and the
environment when we transfer to the real world, we use momentum contrast
learning [32] which enables the key point detector to focus on the object
rather than its surroundings. We add InfoNCE term [33] to the KL divergence
loss. To augment the images, we replaced the RGB values of the pixels
belonging to the background and the table by referring to the segmentation
mask. With probability 0.5, we replaced the pixel RGB values with random RGB
colors, and with probability 0.5, we replaced them with textures used in [34].

####  III-B3 Correcting the errors in a robot model

To close the sim-to-real gap in the robot model, we perform system
identification on the joint dynamics parameters available in the simulator,
denoted $\alpha$, which includes joint friction, damping, and armature. To do
this, we first collect $N_{traj}$ trajectories with the length of
$\mathcal{T}$ from the real robot by setting one of the seven joints to follow
a sinusoidal joint position target of the form $q^{real}[t]=A\sin(\beta\gamma
t)$, while the remaining joints maintain their position. The amplitude $A$ and
frequency $\gamma$ of the trajectory are selected randomly from the hand-tuned
range where its maximum is proportional to the position and velocity limits of
each joint. We then optimize

$\min_{\alpha}\sum_{i=0}^{N_{traj}}(\sum_{t\in\mathcal{T}}(q_{i}^{real}[t]-q_{i}^{sim}[t;\alpha])^{2})^{\frac{1}{2}}$

for each joint using CMA-ES [35], where $q_{i}^{sim}[t;\alpha]$ is the joint
position that would result in the simulator when we use a joint position
controller to follow the same sinusoidal trajectory with dynamics parameters
set to $\alpha$. Since only one joint moves in each trajectory, we can treat
all joints separately, which simplifies the optimization problem.

####  III-B4 Meeting the joint limits

The real robot has joint velocity and torque limits. To satisfy them, we must
limit the magnitude of the end-effector residual at each time step, but this
hinders efficient exploration. We use a curriculum for the maximum end-
effector residual magnitude to handle this problem. Denote the target end-
effector residual magnitude that satisfies the joint limits as $\zeta^{*}$,
which is obtained from the real robot. We begin with a large initial limit
$\zeta_{o}>\zeta^{*}$, which gets reduced whenever the policy success rate
reaches 80%. The scale is reduced in a geometric sequence with ratio
$(\zeta^{*}/\zeta_{o})^{\frac{1}{N_{s}}}$ where $N_{s}$ is a hyperparameter
that determines the number of steps to reach the target residual magnitude
from the initial magnitude.

####  III-B5 Domain randomization

To make the policy more robust to domain difference and the noise when it is
transferred to the real robot, we further train our policy with domain
randomization (DR) after we train our policy with an action scaling
curriculum. Below is the list of properties we applied and the detailed ranges
are noted in Table II.

  * •

Physics properties: The friction of the table and robot and the mass of the
object are randomized. We also add random noise to the commanded torque from
the controller, to reflect the effects of real-world noise.

  * •

Robot model: We adaptively reduce the gap between minimum and maximum values
for the joint position range whenever the policy reached a success rate over a
threshold.

  * •

Perception: We add noise to the input of the policy to mimic the sensor noise
and error of the key point detector.

TABLE II: Domain randomization noise. $\mathcal{U}[min,max]$ denotes uniform distribution, and $\mathcal{N}[\mu,\sigma]$ denotes Normal distribution. Parameter | Range
---|---
Table friction | $\times\mathcal{U}[0.7,1.3]$
Robot end-effector surface friction | $\times\mathcal{U}[0.9,1.1]$
Object mass | $\times\mathcal{U}[0.7,1.3]$
Torque noise | $+\mathcal{N}[0.0,0.03]$
2D key point noise | $+\mathcal{N}[0.0,0.03]$
Sensor noise | $+\mathcal{N}[0.0,0.01]$

##  IV Experiments

###  IV-A Domain description and experiment setup

(a) Bump domain

(b) Card domain

(c) Bump domain

(d) Wall domain

Figure 4:  Learning curves for (a) comparing our method with the one that does
not use pre-contact policy in the bump domain, (b, c, d) comparing our method
to the one without scheduling in card, bump, and wall domains, respectively.
The sudden dips or temporary plateaus in the graphs are due to domain
randomization and residual scaling curriculum.

Domain | Controller | Object | Mass (g) | Surface | scenario 1 | scenario 2 | scenario 3 | Success rate
---|---|---|---|---|---|---|---|---
Card | Ours | 3D printed (default) | 8 | plastic | 5/5 | 3/5 | 5/5 | 0.87
OSC | 1/5 | 1/5 | 0/5 | 0.13
$\zeta^{*}$ | 5/5 | 2/5 | 4/5 | 0.73
Ours | 3D printed (wrapped) | 8 | vinyl | 4/5 | 1/5 | 2/5 | 0.47
Acrylic | 16* | acrylic | 3/5 | 3/5 | 5/5 | 0.73
Wood | 8 | wood | 5/5 | 4/5 | 5/5 | 0.93
Credit card | 5 | plastic | 5/5 | 5/5 | 4/5 | 0.93
Tissue${\dagger}$ | 1* | tissue | 4/5 | 5/5 | 5/5 | 0.93
Chocolate chip${\dagger}$ | 9 | vinyl | 3/5 | 4/5 | 5/5 | 0.87
Bump | Ours | 3D printed (default) | 146 | paper | 5/5 | 3/5 | 5/5 | 0.87
OSC | 0/5 | 0/5 | 1/5 | 0.07
Ours | 3D printed (wrapped) | 146 | vinyl | 4/5 | 2/5 | 5/5 | 0.73
3D printed (high density) | 223* | paper | 5/5 | 5/5 | 5/5 | 1.00
Wood | 381* | paper | 5/5 | 3/5 | 4/5 | 0.80
Painted wood | 381* | wood | 2/5 | 5/5 | 3/5 | 0.67
Sponge${\dagger}$ | 30* | paper | 2/5 | 5/5 | 3/5 | 0.67
Diaper box${\dagger}$ | 102 | paper | 2/5 | 4/5 | 2/5 | 0.53
Wall | Ours | 3D printed (default) | 38 | paper | 5/5 | 3/5 | 5/5 | 0.87
OSC | 0/5 | 0/5 | 0/5 | 0.00
Ours | 3D printed (wrapped) | 38 | vinyl | 5/5 | 3/5 | 5/5 | 0.87
3D printed (high density) | 62* | paper | 5/5 | 4/5 | 4/5 | 0.87
Wood | 97* | paper | 5/5 | 0/5 | 4/5 | 0.60
Painted wood | 97* | wood | 5/5 | 1/5 | 3/5 | 0.60
Sponge${\dagger}$ | 5* | paper | 5/5 | 5/5 | 5/5 | 1.00
Water tissue box${\dagger}$ | 138* | paper | 0/5 | 1/5 | 2/5 | 0.20

TABLE III: The real-world results in diverse objects for three domains.
${\dagger}$ means the object is non-rigid, and * means the mass of the object
is out of DR range. $\zeta^{*}$ means the policy is trained with the same
controller and architecture as our system except that it uses fixed residual
magnitude limit $\zeta^{*}$. Each scenario refers to different initial and
target object poses.

We have designed three domains to evaluate the performance of our method as
shown in Figure LABEL:fig:overall_illustration. The first domain is the card
domain, where for each training episode, we uniformly sample the initial and
the goal object positions on the table while its orientation along the z-axis
is uniformly sampled in the range of $[0,2\pi)$. The second domain is the
_bump domain_. The object is initially placed on the right side of the bump,
and the robot’s task is to push the object over the bump and reorient it to
match a goal position and orientation, which is uniformly sampled on top of
the bump or the opposite side of the table. The initial and the goal
orientations along the roll and pitch axis are uniformly sampled from the set
[0, $\pi/2$, $\pi$, $3\pi/2$], while that along the yaw axis is uniformly
sampled from $[0,2\pi)$. The third domain is the _wall domain_. The initial
pose of the object is fixed to the up-right pose near the wall. The goal pose
of the object is uniformly sampled on the left side of the top of the wall. We
do not try to match the orientation in this domain. During training, we fix
the density of the object. The default density of the object is 457.1/, 200/,
and 150/ for the card, bump, and wall domain respectively.

We use IssacGym [36] to train the policies, and use different policies for
different domains. In the simulator, we run $24,576$ environments
simultaneously. We use $C_{1}=0.02$, $C_{2}=1000$, $C_{3}=0.03$, $C_{4}=-100$,
$\bar{d}=0.01\unit{}$ and $\bar{\theta}=0.1\unit{}$ for the reward function.
We run $\pi_{post}$ to output actions at a frequency of 10Hz, while the
controller runs at a frequency of 100Hz. For the action scale schedule, we use
$N_{s}=10$, $\zeta_{o}=(0.06,0.1)$, and $\zeta^{*}=(0.02,0.03)$.

For the real-world experiment, we use a RealSense D435 camera to obtain RGB
images. We use RRT* [37] to plan a trajectory from the initial robot
configuration given by $\pi_{pre}^{S}$, and use Polymetis [38] to control the
robot. We run $\pi_{post}$ outputs its action at 10Hz, and the controller runs
at 500Hz. To address safety issues, we set the minimum damping ratio for
$\rho[t]$ at 0.5. We wrapped the gripper of the robot with a glove that has a
higher friction coefficient to match the setting of the simulation.

###  IV-B Results

In this paper, we make the following claims.

  * •

Claim 1: Jointly training $\pi_{\text{pre}}$ with $\pi_{\text{post}}$
facilitates more efficient exploration than using a single policy with
contact-inducing reward.

  * •

Claim 2: Residual curriculum enables efficient exploration and learning of a
policy that meets joint limits.

  * •

Claim 3: Inverse differential kinematics and joint position controller is
better suited for sim-to-real transfer on robots with large gear ratios, such
as Panda.

  * •

Claim 4: Our system can transfer to the real world and generalize to objects
that were not seen during training.

To support claim 1, we compare our method against the benchmark that does not
use $\pi_{pre}$ in the bump domain where we initialize the joint position of
the robot in two different ways, as shown in Figure 1(a). In both setups, the
robot needs to learn that it needs to contact the object on the right side to
push it over the bump. Figure 4(a) shows the results. We can see the effect of
using $\pi_{\text{pre}}$: our method reaches 95% success rate at around 6
billion steps, while the one without $\pi_{\text{pre}}$ and
$\pi_{\text{post}}$ decomposition fails to reach the same level of success
rate. The policy in EE-AT-RIGHT setup does a little better than the one in EE-
ABOVE, because the robot begins at a configuration near the right side of the
object. However, it is much less data-efficient than our method.

To support claim 2, We compare methods with and without end-effector residual
scaling during training. For the one without the end-effector residual, we fix
the residual magnitude limit to $\zeta^{*}$ and train the policy. Figures
4(b), 4(c), and 4(d) show the result in three domains. As the plot shows,
except for the card domain, the one without scheduling fails to learn the
task, demonstrating the importance of our scheduling scheme. For the card
domain, We further compare our policy with the policy without scheduling in
the real world. The policy without scheduling achieved 73% success rate, and
ours achieved 87% as shown in Table III. This shows that action scaling not
only helps with exploration, but also in closing the sim-to-real gap.

To support claim 3, we compare our method against the method in [16] where the
policy outputs the next end-effector pose and the gains for an OSC in the real
world. We use the same action scheduling and policy architecture as our system
but use a different action space and controller. In the simulation, the OSC-
based method achieves over 90% success rates in the card and bump domains and
over 80% in the wall domain in simulation. We then compare them in the real
world with the object that has the same physical parameters as the one used in
training. Table III shows the result. It shows that the OSC-based approach
does much worse in the real world and achieves 13%, 7%, and 0% success rates
in card, bump, and wall domains respectively, while ours achieve 87% success
rates in all three. This indicates that OSC-based action space has a larger
sim-to-real gap than our action space.

To support claim 4, we test our policy on 7 different objects in each domain.
The objects vary in their mass, surface friction, scale, and deformation. We
paint the objects so that it has the same visual appearance as the training
data. The list of the objects and the results are shown in Table III. For the
default object that has the same physical properties as the one we used in
training, we achieve 87% success rate, indicating that our system can transfer
to the real world. Furthermore, our method succeeds even for some objects
whose physical properties (ex. non-rigid, heavy, etc.) are outside of our
domain randomization range. The object with the lowest success rate is the
water tissue box, which is highly non-rigid.

##  V Limitation

Our current work is limited to fixed environments and objects, and must train
new policies if they change. The future work would involve generalizing across
different object shapes by incorporating shape information.

##  VI Acknowledgement

This work was supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea
government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School
Program(KAIST)), (No.2022-0-00311, Development of Goal-Oriented Reinforcement
Learning Techniques for Contact-Rich Robotic Manipulation of Everyday
Objects), (No. 2022-0-00612, Geometric and Physical Commonsense Reasoning
based Behavior Intelligence for Embodied AI), and Samsung Electronics.

## References

  * [1] G. Lee, T. Lozano-Perez, and L. P. Kaelbling, “Hierarchical Planning for Multi-Contact Non-Prehensile Manipulation,” in _International Conference on Intelligent Robots and Systems_ , 2015.
  * [2] J. Barry, T. Lozano-Peréz, and L. P. Kaelbling, “A Hierarchical Approach to Manipulation with Diverse Actions,” in _International Conference on Robotics and Automation_ , 2013.
  * [3] X. Cheng, E. Huang, Y. Hou, and M. T. Mason, “Contact Mode Guided Motion Planning for Quasidynamic Dexterous Manipulation in 3D,” in _International Conference on Robotics and Automation_ , 2022.
  * [4] M. Posa, C. Cantu, and R. Tedrake, “A Direct Method for Trajectory Optimization of Rigid Bodies Through Contact,” _The International Journal of Robotics Research_ , 2014.
  * [5] I. Mordatch, Z. Popović, and E. Todorov, “Contact-Invariant Optimization for Hand Manipulation,” in _ACM SIGGRAPH/Eurographics Symposium on Computer Animation_ , 2012.
  * [6] B. Aceituno-Cabezas and A. Rodriguez, “A Global Quasi-Dynamic Model for Contact-Trajectory Optimization,” in _Robotics: Science and Systems_ , 2020\.
  * [7] I. Kao, K. M. Lynch, and J. W. Burdick, “Contact Modeling and Manipulation,” in _Springer Handbook of Robotics_ , 2016.
  * [8] K. Miyazawa, Y. Maeda, and T. Arai, “Planning of Graspless Manipulation Based on Rapidly-Exploring Random Trees,” in _Assembly and Task Planning: From Nano to Macro Assembly and Manufacturing_ , 2005.
  * [9] T. Chen, J. Xu, and P. Agrawal, “A System for General In-Hand Object Re-Orientation,” in _Conference on Robot Learning_ , 2022.
  * [10] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal, “Visual Dexterity: In-Hand Dexterous Manipulation from Depth,” _arXiv_ , 2022.
  * [11] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, _et al._ , “Learning Dexterous In-Hand Manipulation,” _The International Journal of Robotics Research_ , 2020.
  * [12] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning Agile and Dynamic Motor Skills for Legged Robots,” _Science Robotics_ , 2019.
  * [13] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning Quadrupedal Locomotion over Challenging Terrain,” _Science Robotics_ , 2020\.
  * [14] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Perez, “Integrated Task and Motion Planning,” _Annual Review of Control, Robotics, and Autonomous Systems_ , 2021.
  * [15] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl, “Learning by Cheating,” in _Conference on Robot Learning_ , 2020.
  * [16] R. Martín-Martín, M. A. Lee, R. Gardner, S. Savarese, J. Bohg, and A. Garg, “Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks,” in _International Conference on Intelligent Robots and Systems_ , 2019.
  * [17] J. Nakanishi, R. Cory, M. Mistry, J. Peters, and S. Schaal, “Operational Space Control: A Theoretical and Empirical Comparison,” _The International Journal of Robotics Research_ , 2008.
  * [18] J. Moura, T. Stouraitis, and S. Vijayakumar, “Non-Prehensile Planar Manipulation via Trajectory Optimization with Complementarity Constraints,” in _International Conference on Robotics and Automation_ , 2022.
  * [19] C. Zito, R. Stolkin, M. Kopicki, and J. L. Wyatt, “Two-Level RRT Planning for Robotic Push Manipulation,” in _International Conference on Intelligent Robots and Systems_ , 2012.
  * [20] J. Liang, X. Cheng, and O. Kroemer, “Learning Preconditions of Hybrid Force-Velocity Controllers for Contact-Rich Manipulation,” in _Conference on Robot Learning_ , 2022.
  * [21] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, _et al._ , “Solving Rubik’s Cube with a Robot Hand,” _arXiv_ , 2019.
  * [22] A. Allshire, M. Mittal, V. Lodaya, V. Makoviychuk, D. Makoviichuk, F. Widmaier, M. Wüthrich, S. Bauer, A. Handa, and A. Garg, “Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World Trifinger,” _arXiv_ , 2021.
  * [23] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, _et al._ , “DeXtreme: Transfer of Agile In-Hand Manipulation from Simulation to Reality,” _arXiv_ , 2022.
  * [24] W. Yuan, J. A. Stork, D. Kragic, M. Y. Wang, and K. Hang, “Rearrangement with Nonprehensile Manipulation Using Deep Reinforcement Learning,” in _International Conference on Robotics and Automation_ , 2018.
  * [25] W. Yuan, K. Hang, D. Kragic, M. Y. Wang, and J. A. Stork, “End-to-End Nonprehensile Rearrangement with Deep Reinforcement Learning and Simulation-to-Reality Transfer,” _Robotics and Autonomous Systems_ , 2019\.
  * [26] K. Lowrey, S. Kolev, J. Dao, A. Rajeswaran, and E. Todorov, “Reinforcement Learning for Non-Prehensile Manipulation: Transfer from Simulation to Physical System,” in _International Conference on Simulation, Modeling, and Programming for Autonomous Robots_ , 2018.
  * [27] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization,” in _International Conference on Robotics and Automation_ , 2018.
  * [28] W. Zhou and D. Held, “Learning to Grasp the Ungraspable with Emergent Extrinsic Dexterity,” in _Conference on Robot Learning_ , 2022.
  * [29] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” _arXiv_ , 2017.
  * [30] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, “On the Continuity of Rotation Representations in Neural Networks,” in _Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [31] M. Vecerik, J.-B. Regli, O. Sushkov, D. Barker, R. Pevceviciute, T. Rothörl, R. Hadsell, L. Agapito, and J. Scholz, “S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency,” in _Conference on Robot Learning_ , 2021.
  * [32] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” in _Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [33] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” _arXiv_ , 2018.
  * [34] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, “VIMA: General Robot Manipulation with Multimodal Prompts,” _arXiv_ , 2022.
  * [35] N. Hansen, S. D. Müller, and P. Koumoutsakos, “Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES),” _Evolutionary Computation_ , 2003.
  * [36] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, _et al._ , “Isaac Gym: High Performance GPU-Based Physics Simulation for Robot Learning,” _arXiv_ , 2021.
  * [37] S. Karaman and E. Frazzoli, “Sampling-Based Algorithms for Optimal Motion Planning,” _The International Journal of Robotics Research_ , 2011.
  * [38] Lin, Yixin and Wang, Austin S. and Sutanto, Giovanni and Rai, Akshara and Meier, Franziska, “Polymetis,” 2021.